arXiv 论文证明 RLVR 中每一步 pass@1 提升都有 pass@k 代价
There Is No Free Lunch in RLVR: Every pass@1 Gain Has a pass@k Price
AI 导读
arXiv 论文（2610.39295）将 RLVR 建模为对基础策略的指数倾斜，证明提升 pass@1 的每次更新必然降低部分提示的多采样成功率，并给出 pass@k 交叉点数量的全正性紧上界。论文指出该交叉不是缺陷，完全消除它会损失全部 pass@1 增益，且同一机制统一了熵坍缩与熵-性能规律，并提供可从评测日志读取边界、保序检验和代价的诊断工具。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) makes a language model solve more problems on the first try and fewer when given many tries: it raises pass@1 while the base model overtakes it at large sampling budget k. Is this a failure of RL to add reasoning, or a transient that more training removes? We settle the question with a theorem. The reversal is a price, and we compute its exact rate. Modeling RLVR as an exponential tilt of a base policy over strategies shared across prompts, we prove that under exclusive specialization every update that raises pass@1 strictly lowers the success of some prompt; a strictly positive price on prompts (Stiemke duality) is the exact condition, and being merely the best strategy on a prompt is provably not enough. When the loss reaches the hardest prompts, it forces a pass@k crossover, and the number of crossings is at most the sign variation of the shift in per-prompt success, a totally positive bound that is tight and that recovers the known single-crossover result as its order-preserving case. The crossover is not a bug to patch: a constrained update traces the exact pass@1 versus pass@k frontier, and removing the crossover entirely costs the entire pass@1 gain. The same tilt identity yields entropy collapse and the entropy-performance law with an identifiable exponent, unifying two literatures under one mechanism. Reinforcement cannot widen the reasoning frontier; it can only move along it, at a price we make measurable by a diagnostic that reads the bound, the order-preserving test, and the price from any evaluation log. There is no free lunch, only a price.
| Comments: | 19 pages, 4 figures. Code and verification package (34/34 automated checks): this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML) |
| MSC classes: | 68T05, 90C05, 90C46 |
| ACM classes: | I.2.6; G.1.6 |
| Cite as: | arXiv:2610.39295 [cs.LG] |
| (or arXiv:2610.39295v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39295 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yinuo Liu [view email]
[v1]
Thu, 29 Oct 2026 03:43:26 UTC (398 KB)
来源：arXiv cs.AI · arxiv.org