仅用比较实现多步强化学习的序数信用分配方法研究
Ordinal Credit Assignment: Multi-Step Reinforcement Learning from Comparisons Alone
AI 导读
一项 arXiv 研究提出仅用两两比较、无需奖励函数或奖励模型的多步强化学习方法，通过对同一时间步各轨迹的 reward-to-go 胜率进行信用分配。预注册实验覆盖六个环境、每项结论十种子，比较信号与标量奖励性能相当，10% 的比较集中在少数时间点配全对即足够；真实 LLM 判官的比较比逐点打分准确率高 7-10 点。但在 HalfCheetah 上该方法仍未缩小与 PPO 标量奖励的差距。
正文 · 原文
Abstract:Policy-gradient methods need a scalar reward, yet for many tasks no formula captures what we want, while judging which of two outcomes is better is easy. We study reinforcement learning that receives only such comparisons - no reward function, no reward model - in multi-step tasks. Groups of trajectories start from a shared state, and each action is credited by how its reward-to-go compares with those of other trajectories at the same time step. We show the expected update is policy gradient with reward-to-go replaced by its win rate against the population still running at that time, weighted by the chance an opponent is still running; in deterministic environments the update ranks actions by true value and the optimal policy is stable, whereas under stochastic outcomes it prefers actions that win more often rather than those with higher mean. Label noise near ties or from position bias leaves the expected update unchanged; only noise that also flips clear comparisons halves the signal. A critic trained from the same comparisons removes the state confound without changing the expected update. In a pre-registered study over six environments with ten seeds per claim, comparisons cost no detectable performance relative to scalar returns at the same granularity, learning a reward model from the same comparisons is not better, and ten percent of comparisons suffice when spent on a few time points with all pairs. With real LLM judges, comparisons are 7-10 points more accurate than pointwise scores, and the gap grows with the stronger judge. Trained end-to-end with an LLM judge on a requirement the hand-written reward omits, policies satisfy it in all five seeds but land far less often than with oracle comparisons, because the judge is near chance on the near-tied failures dominating early training. Comparisons do not close the gap to PPO with scalar rewards on HalfCheetah.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Statistics Theory (math.ST) |
| Cite as: | arXiv:2605.06375 [cs.LG] |
| (or arXiv:2605.06375v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06375 arXiv-issued DOI via DataCite |
Submission history
From: Hao Yu [view email]
[v1]
Thu, 7 May 2026 14:56:11 UTC (842 KB)
[v2]
Wed, 4 Nov 2026 12:28:07 UTC (92 KB)
来源：arXiv cs.AI · arxiv.org