教师演化自蒸馏（TESD）：改进 On-Policy Self-Distillation 的训练方法
On-Policy Self-Distillation with Teacher Evolution
AI 导读
arXiv 论文提出 Teacher-Evolution Self-Distillation（TESD），解决 On-Policy Self-Distillation（OPSD）训练中性能提升不连续的问题。该方法通过环境反馈对自教师进行教师导向优化，并依据奖励估计教师引导可靠性，对不可靠样本自适应降权。多项数据集和模型规模的实验显示 TESD 持续优于代表性 token 级和序列级方法。
正文 · 原文
Abstract:On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity in sequence-level post-training paradigms. However, OPSD often fails to achieve continuous performance gains during training. In this paper, we find that the model's limited self-teaching capability underlies this phenomenon by generating unreliable guidance on certain samples. To address this problem, we propose Teacher-Evolution Self-Distillation (TESD), which enables the self-teacher to generate rollouts and receive environmental feedback based on task correctness and context independence. First, using these rewards as training objectives, we perform teacher-oriented optimization to continuously enhance the model's self-teaching capability during distillation. Second, by estimating teacher guidance reliability from these rewards, we apply adaptive student-oriented optimization to down-weight samples with unreliable guidance. Extensive experiments across multiple datasets and model scales show that TESD consistently outperforms representative token-level and sequence-level methods, highlighting the necessity of improving supervision quality in OPSD.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38637 [cs.LG] |
| (or arXiv:2610.38637v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38637 arXiv-issued DOI via DataCite |
Submission history
From: Shufan Shen [view email]
[v1]
Wed, 28 Oct 2026 10:51:52 UTC (2,983 KB)
来源：arXiv cs.AI · arxiv.org