arXiv cs.AI· Shripad V. Deshmukh, Jayakumar Subramanian, Raghavendra Addanki, Nikos Vlassis·· 3 小时前AI 评分41
COSAC：顺序协作团队中的反事实信用分配方法
COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
AI 导读
arXiv 论文提出 COSAC，一个面向顺序协作多智能体团队的无 critic 策略梯度框架，通过单次岭回归拟合团队奖励的可加分解，并用当前策略的前向传递计算各智能体的反事实优势，无需额外环境或奖励调用。
正文 · 原文
Abstract:In cooperative teams where agents act in a fixed order and share a single team-level reward (multi-agent language systems, sequential robotic tasks), per-agent credit assignment is under-determined. Critic-based approaches scale poorly as the number of agents grows owing to the costly maintenance of joint/factored critic(s), whereas the existing critic-free alternatives have other issues: common credit across agents that couples every agent's signal to teammate noise, importance-sampling corrections for upstream-update staleness that incur variance exponential in team size, or per-agent counterfactual replay that isolates each agent's effect at the price of extra environment or reward calls. We propose COSAC, a critic-free policy gradient framework for sequential cooperative teams. COSAC fits an additive per-agent decomposition of the team reward by a single ridge regression on the rollout batch (giving each agent a learning signal decoupled from teammate noise), and computes each agent's counterfactual advantage from fictitious continuations of the current policy (policy forward passes that replace both importance-sampling reweighting and per-agent environment replay, at no extra environment or reward cost). The estimator instantiates the Sequential Aristocrat Utility (SeqAU), our extension of Wolpert and Tumer's (2001) aristocrat utility to sequential teams. We derive bias and conditional variance bounds for SeqAU credits: the direct-effect bound has no explicit team-size factor, while the indirect effect's reward-noise bound scales with the number of downstream agents. Sequential bandits test the analysis across team sizes up to K=16. In sequential multi-agent LLM fine-tuning, COSAC reduces training regret by 17% on consensus-based science reasoning relative to the best critic-free multi-agent baseline and by 61% on Python coding relative to MA-GRPO.
| Comments: | To appear at NeurIPS 2026; 10 main text pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2604.17693 [cs.LG] |
| (or arXiv:2604.17693v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.17693 arXiv-issued DOI via DataCite |
Submission history
From: Shripad Deshmukh [view email]
[v1]
Mon, 20 Apr 2026 01:14:59 UTC (467 KB)
[v2]
Sat, 9 May 2026 18:49:00 UTC (139 KB)
[v3]
Wed, 11 Nov 2026 16:57:51 UTC (1,321 KB)
来源：arXiv cs.AI · arxiv.org