RLVR 与 RLHF 奖励来源如何影响大语言模型跨域泛化：在哪奖励、奖励什么
Where and What to Reward? Understanding Generalization Between Verifiable and Open-Ended Domains in Reinforcement Learning
AI 导读
一项受控研究在固定优化器和提示词数量、仅变换提示域（数学、多域、聊天或混合）与奖励来源的条件下，考察 RLVR 与 RLHF 之间的能力迁移。结果显示，RLHF 在开放域提示上训练不仅能大幅提升对话能力，还显著提升推理与编程能力；而 RLVR 提升推理后，在开放对话上的收益远低于 RLHF，表明基于奖励模型的开放域强化学习可作为跨域泛化的有效途径。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capability of language models, since model-generated answers can be easily verified by a rule-based verifier. In contrast, reinforcement learning from human feedback (RLHF) is mainly used to optimize open-ended conversation with a learned reward model, because such responses are not easy to verify. RLVR and RLHF are usually trained separately with their own domain data and training protocols. How gains transfer between these domains is unclear, partly because comparisons change both where a model is trained (the prompts) and what is rewarded (by a verifier or a reward model). We separate the two in a controlled study of RL on multiple base models that fixes the optimizer and the number of training prompts, and varies only the source domain of the prompts (math, multi-domain, chat, or mixtures) and the reward. We find that RLHF on open-ended prompts substantially improves conversational ability and also brings large gains in reasoning and coding, whereas RLVR improves reasoning but its gains in open-ended conversation plateau far below those of RLHF, suggesting that RL on open-ended prompts with a reward model can serve as an effective approach for generalizing across domains.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.15464 [cs.LG] |
| (or arXiv:2605.15464v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15464 arXiv-issued DOI via DataCite |
Submission history
From: Shangjian Yin [view email]
[v1]
Thu, 14 May 2026 23:05:23 UTC (668 KB)
[v2]
Mon, 26 Oct 2026 17:21:42 UTC (130 KB)
来源：arXiv cs.AI · arxiv.org