PASS:面向 LLM 推理器过程监督强化学习的优势构造方法
Process Advantage Signal Shaping: Advantage Construction for Process-Supervised Reinforcement Learning in LLM Reasoners
AI 导读
arXiv 论文提出 PASS(Process Advantage Signal Shaping),一种用于 GRPO 的优势构造方法,将过程监督的局部评分转化为策略优势。
正文 · 原文
Abstract:Dense process supervision provides local evaluations, but leaves open how these scores should become policy advantages. We introduce PASS (Process Advantage Signal Shaping), an advantage-construction method for group-relative policy optimization (GRPO). PASS separately normalizes and fuses reward channels, constructs batch-shared credit units from variation in the fused score trace, and normalizes return-to-go by the remaining unit count. These operations connect feedback scale, credit resolution, and accumulation horizon in a single computation. We evaluate learned process reward model (PRM) feedback for mathematical reasoning and teacher-derived token feedback for multi-hop question answering (QA). Across seven mathematics benchmarks, PASS improves average pass@1 by 4.62 percentage points over outcome-only GRPO with approximately $2\%$ shorter responses. Matched comparisons with Step-GRPO and component ablations further support the choice of shared credit units and residual normalization. On multi-hop QA, PASS improves over process-signal controls across two normalizers; token-level ablations reveal an accuracy--length trade-off: token-level accumulation slightly favors pass@1, while PASS achieves higher pass@8/16 with shorter responses. The results establish the conversion from local process scores to policy advantages as a consequential design choice across distinct supervision sources.
| Comments: | 27 pages, 5 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.29296 [cs.AI] |
| (or arXiv:2606.29296v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.29296 arXiv-issued DOI via DataCite |
Submission history
From: Chao Wang [view email]
[v1]
Sun, 28 Jun 2026 09:36:43 UTC (253 KB)
[v2]
Sat, 10 Oct 2026 11:10:57 UTC (267 KB)
[v3]
Tue, 17 Nov 2026 12:13:41 UTC (299 KB)
来源:arXiv cs.AI · arxiv.org