DoubleFree：利用探索历史实现免 Rollout 的多任务大语言模型训练
DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training
AI 导读
DoubleFree 复用领域 RL 训练过程中产生的历史轨迹进行免 rollout 的多教师蒸馏整合，并预计算教师监督信号实现免在线教师训练，仅需对现有后训练框架做最小改动。在 Qwen3-4B、Qwen3-8B 和 SmolLM3-3B 上的数学、代码与指令遵循任务中，该方法实现 2.99-5.75 倍整合加速，GPU 小时减少 80.9%-90.1%，多域性能保持相当。
正文 · 原文
Abstract:Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves $2.99$-$5.75\times$ consolidation speedups and reduces GPU-hours by $80.9$-$90.1\%$, while maintaining comparable multi-domain performance.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34272 [cs.LG] |
| (or arXiv:2610.34272v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34272 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingping Wang [view email]
[v1]
Sun, 25 Oct 2026 13:03:18 UTC (439 KB)
来源：arXiv cs.AI · arxiv.org