模仿引导的多智能体强化学习方法 IBMARL
Imitation Bootstrapped Multi-Agent Reinforcement Learning
AI 导读
研究者提出 IBMARL 框架,将 Imitation Bootstrapped Reinforcement Learning(IBRL)扩展到协作多智能体场景,并用 Round-Robin Behavior Cloning(R2BC)训练模仿策略。
正文 · 原文
Abstract:Demonstrations are an effective way to accelerate single-agent reinforcement learning (RL), but extending this idea to multi-agent reinforcement learning (MARL) is non-trivial as it is difficult and often impossible for a single person to teleoperate multiple agents simultaneously. Round-Robin Behavior Cloning (R2BC) addresses this by letting a single demonstrator control one agent at a time while the others execute their current policies. However, the resulting imitation policies are limited by the quality of the demonstrations. We propose Imitation Bootstrapped Multi-Agent Reinforcement Learning (IBMARL), a framework that extends Imitation Bootstrapped Reinforcement Learning (IBRL) to cooperative teams and uses R2BC to train the imitation policies. At each time step, every agent receives one action proposal from its imitation policy and one from its RL policy, and an ensemble of centralized critics selects among the per-agent combinations of the two. A gated imitation penalty, a soft Q-filtered behavior cloning loss, regularizes each RL policy toward its imitation policy only where the critics value the imitation action more, so that after training each agent executes its RL policy alone, without the centralized critics. On four cooperative multi-agent RL benchmarks, IBMARL surpasses its imitation policies $2.6$-$4.2\times$ faster than demonstration-based baselines on two tasks, is the only method to surpass it on the other two, and exceeds the demonstrator itself on two of the four. With real human demonstrations IBMARL still surpasses imitation learning alone, and does so more than twice as fast as the best baseline.
| Comments: | 9 pages, 5 figures. Code, videos and results: this https URL. A preliminary version appeared at the PhysCollaBot Workshop at IROS 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2611.18543 [cs.AI] |
| (or arXiv:2611.18543v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.18543 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tony Zhang [view email]
[v1]
Fri, 13 Nov 2026 18:50:05 UTC (10,118 KB)
来源:arXiv cs.AI · arxiv.org