MCPO：面向可验证奖励强化学习的基于边际的 Token 信用分配方法
Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards
AI 导读
一项 arXiv 研究提出 MCPO（Margin-Credit Policy Optimization），利用策略 next-token 分布的 top-2 边际 $M=p_1-p_2$ 将 GRPO 的奖励按 token 的犹豫程度分配，无需额外模型评估或 batch 统计。
正文 · 原文
Abstract:In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin $M=p_1-p_2$ of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95\% CI $[4.1, 9.0]$), and a Qwen3-4B-Base replication improves for all three seeds ($+4.75$). The gain is largest on problems that GRPO solves only some of the time and grows with $k$ in pass@$k$, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14015 [cs.LG] |
| (or arXiv:2611.14015v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14015 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chi Liu [view email]
[v1]
Tue, 10 Nov 2026 12:51:43 UTC (186 KB)
来源：arXiv cs.AI · arxiv.org