Qwen3-8B 法律推理研究揭示奖励惩罚缺失时的策略性弃答机制
Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models
AI 导读
论文用 GRPO 基于引用数、法律术语密度和回答长度三个表面特征微调 Qwen3-8B，模型在 LegalBench 的 16 个是非题任务（N=320）上准确率从 0.500 降至 0.072，主因是格式合规答案比例从 0.900 跌至 0.109。
正文 · 原文
Abstract:What happens when a legal AI model learns to look like a lawyer instead of reasoning like one? We fine tune Qwen3-8B with Group Relative Policy Optimisation (GRPO) against a proxy built from three surface features: citation count, legalese density, and response length. The model does not learn to reason more effectively. It stops committing to an answer. Across 16 yes or no legal reasoning tasks from LegalBench (N=320), overall accuracy collapses from 0.500 (chance) to 0.072 (McNemar p < 10^-36), driven entirely by the rate of properly formatted answers falling from 0.900 to 0.109. When the model does commit, accuracy is 0.657 against 0.556 before training, on only about 35 answers, suggesting the collapse is not mainly a loss of capability. The mechanism is visible in the training logs: the proxy keeps paying for length beyond what fits in the 256 token generation limit, completions grow until they fill that budget, and the required answer line is crowded out. We term this the Saul Goodman effect, a policy that becomes more lawyerly in form while becoming noncommittal, and prove formally that abstention is the optimal response to any surface feature proxy that attaches no penalty to it. A citation audit shows that structural plausibility checks cannot catch this failure: every case name we extracted from the trained model's held out completions passes our structural check, so citations must be verified against a case database. To detect this failure before deployment, we introduce three diagnostic tools: the Confidence Theater Score (CTS), the Citation Plausibility Rate (CPR), and the Regret Gap (RG), and report where each one falls short. In a domain where a confidently wrong answer can constitute malpractice, the broader lesson is direct: a reward function that measures how legal a response looks will produce a model that is maximally photogenic and minimally useful.
| Comments: | 13 Pages, Proceedings of the AI for Law Workshop at the 43rd International Conference on Machine Learning, Seoul, South Korea. PMLR 356, 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.06439 [cs.LG] |
| (or arXiv:2610.06439v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.06439 arXiv-issued DOI via DataCite |
Submission history
From: Subramanyam Sahoo [view email]
[v1]
Mon, 5 Oct 2026 14:46:35 UTC (722 KB)
[v2]
Wed, 11 Nov 2026 12:04:48 UTC (724 KB)
来源：arXiv cs.AI · arxiv.org