arXiv cs.AI· Hao Li, Zhengkun Zhang, Gangqiang Hu, Zhen Zhang, Yude Gao, Dai Dai, Jing Liu·· 4 小时前AI 评分38
RLVR$^{2}$：基于可验证量规排序的强化学习方法
RLVR$^{2}$: Reinforcement Learning with Verifiable Rubric-based Ranking
AI 导读
一篇 arXiv 预印论文提出 RLVR$^{2}$，一种针对基于量规（rubric）的 RLVR 的可验证排序范式。该方法将每条准则的量规分数转换为组内序数结果，恢复潜在效用并合并为单一训练信号，避免异构量规尺度的校准问题，还支持不直接奖励辅助属性的客观保持式调整。在 3 个模型规模和 16 个基准上，RLVR$^{2}$ 在大多数基准上取得最优整体表现。
正文 · 原文
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) is expanding from tasks with well-defined correctness signals, such as mathematics and code, toward multifaceted quality requirements specified by multi-dimensional rubrics. Since policy optimization consumes one scalar per rollout, rubric-based pipelines must map multiple criterion scores into a scalar reward. This aggregation is often treated as score scaling, but it implicitly determines how quality dimensions trade off during training. The prevailing practice, normalizing each criterion and taking a linear combination, assumes that cardinal score differences are comparable across criteria and that gains on one criterion compensate for failures on another; both assumptions are unreliable when criteria are semantically heterogeneous. We propose Reinforcement Learning with Verifiable Rubric-based Ranking (RLVR$^2$), a verifiable ranking paradigm for rubric-based RLVR. For each criterion, RLVR$^2$ converts rubric scores into criterion-specific within-group ordinal outcomes, recovers a latent utility from the resulting comparison matrix, and merges these utilities into one training signal. By retaining only within-group ordering and discarding raw score magnitudes, RLVR$^2$ avoids calibrating heterogeneous rubric scales. It further supports objective-preserving attribute adjustment: auxiliary attributes that correlate with observed rankings but are not training objectives can enter the estimation without expanding the rubric or rewarding them directly. Across three model scales and 16 benchmarks, RLVR$^2$ consistently outperforms representative rubric-based baselines, achieving the best overall performance on most benchmarks at every scale. Analysis shows it controls systematic effects tied to reasoning efficiency and response formatting while preserving the quality objective.
| Comments: | Preprint |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.23457 [cs.LG] |
| (or arXiv:2609.23457v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.23457 arXiv-issued DOI via DataCite |
Submission history
From: Hao Li [view email]
[v1]
Sun, 20 Sep 2026 08:34:28 UTC (2,132 KB)
[v2]
Fri, 9 Oct 2026 10:31:18 UTC (5,400 KB)
[v3]
Thu, 5 Nov 2026 08:43:26 UTC (5,408 KB)
来源：arXiv cs.AI · arxiv.org