arXiv cs.AI· Jingyuan Huang, Zuming Huang, Yucheng Shi, Zhongzhi Li, Xiaoming Zhai, Wei Chu, Ninghao Liu·· 3 小时前AI 评分38
Train4Merge：RL 还是 SFT 更适合训练 OPD 模型合并的教师模型
Train4Merge: RL or SFT---Which Yields Better Teachers for OPD-Based Model Merging?
AI 导读
一项 arXiv 研究系统比较了 RL 与 SFT 训练的教师模型在在线策略蒸馏（OPD）合并中的表现，教师和学生均基于同一 Qwen3.5-9B 初始化。在 Agentic、Reasoning、Perception 三个领域的单教师实验中，RL 教师均带来更强学生和更高的教师性能增益恢复；多教师 OPD（MOPD）下 RL 教师组的三域均分也更高。
正文 · 原文
Abstract:Domain experts trained from a shared checkpoint can transfer their specialized capabilities to a single student through on-policy distillation (OPD). Existing research primarily focuses on improving this merging process, while the algorithms used to train the experts have received limited systematic comparison. We investigate which training algorithm produces teachers better suited to OPD through controlled single-teacher comparisons of supervised fine-tuning (SFT) and reinforcement learning (RL) across Agentic, Reasoning, and Perception. We further examine whether this advantage extends to multi-teacher OPD (MOPD). Teachers and students share the same Qwen3.5-9B initialization, and the two teacher types are compared at similar task performance. Our single-teacher experiments show that RL teachers yield stronger students and higher recovery of teacher performance gains across all three domains. In MOPD, the RL teacher group also yields a student with a higher three-domain mean score and higher overall teacher gain recovery than the SFT teacher group. Our further analysis shows that RL teachers undergo smaller parameter displacements from the shared initialization than SFT teachers. Teacher--student Top-K overlap also increases faster early in OPD or remains consistently higher with RL teachers than with SFT teachers. These findings support the hypothesis that RL teachers' smaller departures from the student's starting point facilitate learning through OPD, resulting in stronger students.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.32303 [cs.AI] |
| (or arXiv:2609.32303v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.32303 arXiv-issued DOI via DataCite |
Submission history
From: Jingyuan Huang [view email]
[v1]
Sat, 26 Sep 2026 06:59:03 UTC (130 KB)
[v2]
Wed, 7 Oct 2026 02:15:02 UTC (134 KB)
[v3]
Thu, 22 Oct 2026 20:37:53 UTC (160 KB)
来源：arXiv cs.AI · arxiv.org