奖励增益失衡如何影响排序:重新思考基于评分标准的奖励聚合
How Reward Gain Imbalance Shapes Ranking: Rethinking Rubric-Based Reward Aggregation
AI 导读
一项 arXiv 研究将基于评分标准的强化学习中的评分异质性形式化为“奖励增益失衡”,即各准则在聚合奖励中的期望分差。实验反直觉地显示,即便用真实标签消除该失衡也会降低排序准确率;论文将其对任意两个候选奖励差的影响做精确分解,并提出自适应 DCC(Dual-Center Clipping)聚合方法,为每准则分差设定上限以限制极端分数影响。
正文 · 原文
Abstract:Rubric-based reinforcement learning constructs rewards for aligning large language models with human preferences, typically by equally aggregating continuous scores that a judge assigns to individual criteria. However, scoring heterogeneity across criteria can distort candidate rankings. Existing work has focused on improving judges, learning criterion weights, or refining aggregation, whereas the interaction between scoring heterogeneity and ranking remains insufficiently understood. We formalize this heterogeneity as \emph{reward gain imbalance}: each criterion enters the aggregated reward with its own reward gain, the gap between its expected scores on satisfied and unsatisfied responses. Counterintuitively, our experiments show that removing this imbalance, even with ground-truth labels, lowers ranking accuracy. To explain this, we derive an exact decomposition of the reward difference between any two candidates into a base satisfaction signal, a reward gain imbalance term, and a residual term, showing that reward gain imbalance can attenuate or amplify the base signal, and finding that misrankings usually concentrate on pairs where it attenuates. Guided by this analysis, we propose the adaptive \textbf{D}ual-\textbf{C}enter \textbf{C}lipping aggregation (DCC), which bounds each criterion's contribution to the reward difference instead of simply removing the imbalance. Using only the scores of the current candidate group, it determines a clipping interval that imposes a common upper bound on per-criterion score differences between candidates, limiting the excessive influence of extreme scores while preserving continuous judgment information....
| Subjects: | Machine Learning (stat.ML); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24005 [stat.ML] |
| (or arXiv:2611.24005v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24005 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhengyuan Fan [view email]
[v1]
Wed, 18 Nov 2026 07:56:58 UTC (71 KB)
来源:arXiv cs.AI · arxiv.org