在策略蒸馏目标的理论分析:选哪些 token、按什么顺序、多快学
Which Tokens, in What Order, and How Fast? A Theoretical Analysis of On-Policy Distillation Objectives
AI 导读
Noam Razin 等人在 arXiv 上发表对语言模型在策略蒸馏(OPD)目标的理论分析,研究序列级与 token 级目标如何决定优先匹配哪些 token、以什么顺序及多快学习。
正文 · 原文
Abstract:On-policy distillation (OPD) trains a student language model to match a teacher model based on its own generations. When using OPD, one must choose whether to apply it at the sequence or token level and which divergence to minimize. However, despite the wide adoption of OPD in practice, the formal understanding of which OPD objective is appropriate for a given setting is limited. Toward addressing this gap, we theoretically analyze how the choice of OPD objective governs which tokens are prioritized, in what order, and how efficiently. Our analysis reveals that sequence-level OPD exhibits a future agreement bias, which myopic token-level objectives avoid. This bias induces a reverse learning order, in which sequence-level OPD first matches the teacher at later token positions, while at earlier positions it can favor tokens that are improbable under the teacher, and thus often undesirable. As for the choice of divergence, we prove that it strongly influences the amenability of OPD to gradient-based optimization. In particular, while the forward KL divergence enjoys a well-behaved objective landscape, other $f$-divergences, such as reverse KL, can suffer from vanishing gradients even when the student and teacher are far apart. Across language model post-training settings, we demonstrate the implications of these theoretical findings for which OPD objective is preferable, highlighting the importance of accounting for the interplay between the student, teacher, and task at hand.
| Comments: | Code available at this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2611.23397 [cs.LG] |
| (or arXiv:2611.23397v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23397 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Noam Razin [view email]
[v1]
Tue, 17 Nov 2026 19:01:02 UTC (379 KB)
来源:arXiv cs.AI · arxiv.org