arXiv cs.AI· Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah·· 3 小时前AI 评分56
RcRL：利用生成验证差距在不可验证领域做强化学习
Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains
AI 导读
Motwani、Krishnan 等人提出 Rubric Curriculum RL（RcRL），在无新数据、无人工标注和更强教师模型的情况下，用成对偏好加评分标准课程利用生成验证差距训练开放域策略。
正文 · 原文
Authors:Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah
Abstract:Reinforcement learning with verifiable rewards (RLVR) has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. The generation-verification gap, where judging good outputs is easier than producing them, offers a potential path to improvement. However, rubric-based judges are often imprecise and stationary, so policies can overfit or reward hack them. We introduce Rubric Curriculum RL (RcRL), which exploits this gap without new data, human annotations, or stronger teacher models. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring and a non-stationary objective. Whereas baselines plateau within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL achieves a 70.5% human win rate against the base model, with gains across benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all training baselines, including instance-specific rubrics.
| Comments: | Accepted at ICML 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06244 [cs.LG] |
| (or arXiv:2611.06244v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06244 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sumeet Motwani [view email]
[v1]
Wed, 4 Nov 2026 09:51:06 UTC (472 KB)
来源：arXiv cs.AI · arxiv.org