arXiv cs.AI· Tingcheng Bian, Kangye Gong, Jinchang Luo, MingQuan Cheng, Feiran Liu, Ni Li, Xiaoling Xia, Haiwei Wang, Hui Li·· 4 小时前AI 评分44
Agentic-OPD:通过 On-Policy Distillation 恢复被遗忘的智能体能力
Agentic-OPD: Recovering Forgotten Agent Capabilities via On-Policy Distillation for Agentic Tasks
AI 导读
研究提出 Agentic-OPD,把 GRPO 强化学习中通常被丢弃的检查点作为冻结教师池,用 On-Policy Distillation 恢复 LLM 智能体在训练中遗忘的解题能力。
正文 · 原文
Abstract:Reinforcement learning is now the mainstream paradigm for training LLMs on agentic tasks, yet standard practice keeps only the best-validation checkpoint and discards the rest. We show that this is lossy: in GRPO training of LLM agents, the best-validation checkpoint fails on problems that earlier checkpoints from the same run had already solved, and the union of problems solved across checkpoints far exceeds any single one. The discarded checkpoints thus form a teacher pool obtained at no additional training cost, which on-policy distillation (OPD) can mine to recover the lost coverage. Mining this pool raises two challenges: i) which checkpoint to distill from, as no checkpoint dominates the others and complementarity to the student is problem-specific; ii) where to trust it, as even a well-chosen teacher is only locally superior, so indiscriminate distillation overwrites behavior the student has already mastered. We propose Agentic-OPD, which treats the best-validation checkpoint as the student and all saved checkpoints as frozen teacher candidates, using the student's evolving capability gaps to guide both teacher selection and supervision filtering: the HCN policy (Highest Coverage Next) dynamically selects teachers by weighted coverage of the student's unsolved problems, while the LAD module (Locality-Aware Distillation) confines transfer to failed student rollouts on problems where the teacher performs better. Across five agentic task domains and 24 benchmarks and task subsets, Agentic-OPD improves over its initialization checkpoint by up to 7.4 points, with a 4B student surpassing the latest frontier models GPT-5.6 and DeepSeek-V4.1 on deep search. Agentic-OPD recasts a finished RL run as a reusable pool of policies and establishes post-RL capability recovery as a stage orthogonal to improvements in the RL algorithm itself.
| Comments: | 10 pages of main text, 24 pages total including references and appendix |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.12799 [cs.LG] |
| (or arXiv:2611.12799v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.12799 arXiv-issued DOI via DataCite |
Submission history
From: Tingcheng Bian [view email]
[v1]
Mon, 9 Nov 2026 18:07:42 UTC (542 KB)
[v2]
Fri, 13 Nov 2026 03:18:14 UTC (544 KB)
来源:arXiv cs.AI · arxiv.org