arXiv cs.AI· Zixu Wang, Yu Zhao, Xing Zheng, Bingbing Xu, Tian Xueyun, JiaTang Luo, Weiqiang Wang, Bowen Song, Huawei Shen, Xueqi Cheng·· 1 小时前AI 评分36
OPSD4MAS：用 On Policy Self Distillation 把多智能体通信转化为 token 级信用
Turning Multi-Agent Communication into Token-Level Credit with On Policy Self Distillation
AI 导读
OPSD4MAS 通过 On Policy Self Distillation（OPSD）将未来通信转化为细粒度 token 级信用，改进大语言模型多智能体系统的强化学习。方法先用外部模型提取事实做粗粒度过滤，再对目标 token 影响做留一法细粒度过滤，并与 GRPO 以线性退火联合目标结合。在数学任务上，pass@4 较 GRPO 最高提升 20.00 个百分点，代码已开源。
正文 · 原文
Abstract:Reinforcement learning improves problem solving in large language model multi-agent systems. Existing approaches overlook communication knowledge or compress it into scalar rewards, leaving coarse credit with little guidance on which decisions each agent should reinforce or revise. OPSD4MAS converts future communication into fine-grained token-level credit through On Policy Self Distillation (OPSD). An external model extracts facts and assesses their trajectory relations for coarse-grained filtering, followed by fine-grained leave-one-out filtering of target-token influence against agent-wise outcome advantages. The fact-conditioned policy supplies token-level credit, combined with GRPO through a linearly annealed joint objective. Our method improves mathematical pass@4 over GRPO by up to 20.00 percentage points. Ablations support complete filtering, while frozen-history injection diagnostics support fact utility. Together, these results support combining agent-level outcome guidance with communication-derived token-level credit. Code is available at this https URL.
| Comments: | 14pages,4 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.6; I.2.11; I.2.7 |
| Cite as: | arXiv:2611.16206 [cs.AI] |
| (or arXiv:2611.16206v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16206 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zixu Wang [view email]
[v1]
Thu, 12 Nov 2026 02:57:08 UTC (693 KB)
来源：arXiv cs.AI · arxiv.org