arXiv 论文提出可验证奖励 RL 中模型诚实与吹嘘的阈值条件
Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff
AI 导读
Bottman 与 Oblomkov 在 arXiv:2611.15779 研究可验证奖励 RL 何时教会模型吹嘘，给出 GRPO 一阶条件下模型变得更诚实需满足 p̃·r/b > 1+κ，其中 κ 随训练下降导致刚过阈值的 run 先向吹嘘下滑。
正文 · 原文
Abstract:Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places, with an honest model or one that bluffs. We study rewards that pay $b$ for a verdict that a checker cannot verify ("this program is correct") and $r$ for a certificate that it accepts (an input revealing a bug). To first order, a GRPO step makes the model more honest exactly when $\tilde{p}\,r/b>1+\kappa$. Here $\tilde{p}$ is the rate at which the checker accepts the model's certificates, weighted toward the items on which the model still gives both verdicts, and $\kappa$ is the pull of the items on which the unverifiable verdict is right. Both terms move during training. A checker raises $\tilde{p}$, so near the threshold honesty is a race, which a run just below the threshold can win and a run under a lottery (random pay with the same mean) cannot. The term $\kappa$ starts large and falls as the other items settle, so a run just above the threshold first dips toward bluffing, on code nearly to zero, before climbing back. Paying the honest verdict also pulls the other items toward it, which predicts a ceiling on the payoff ratio, above which the model gives the honest verdict where it is wrong. On propositional Datalog, two code models, and abstention on grade-school math, with $p$ the model's success rate before RL, no run above $pr/b=1$ ends collapsed, and runs under the checker can survive slightly below it. At $pr/b=3$, above the ceiling, both code models call about half of correct programs buggy. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, at a ratio where the same model trained from the start falls to 4-15% and is at most 47% honest after 160 steps. On depth-2 Datalog, when honesty collapses, a linear probe fit before RL still reads the truth.
| Comments: | 34 pages, 10 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.15779 [cs.LG] |
| (or arXiv:2611.15779v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15779 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nathaniel Bottman [view email]
[v1]
Wed, 11 Nov 2026 17:54:41 UTC (1,403 KB)
来源：arXiv cs.AI · arxiv.org