arXiv cs.AI· Shiping Yang, Chuanqi Tang, Hossein Hajimirsadeghi, Layla El Asri, Mohammad Amin Shabani, Angel X. Chang·· 3 小时前AI 评分46
On-policy 蒸馏（OPD）学到了什么、如何预测：熵形状与教师选择
Distilling the Shape of Uncertainty: What On-Policy Distillation Learns and How to Predict It
AI 导读
一项 arXiv 研究指出，大语言模型的 on-policy 蒸馏（OPD）不仅学习「如何分支」（考虑哪些候选 token），还学习「在哪里分支」（在何处保留推理路径），后者由序列的 next-token 熵序列即 entropy shape 刻画。
正文 · 原文
Abstract:On-policy distillation (OPD) is increasingly adopted in the post-training of large language models. However, a stronger teacher may fail to improve the student. Prior work attributes this to a lack of new knowledge, but what that knowledge is remains unclear. We show that OPD learns not only \textit{how to branch}, i.e., which candidate tokens to consider, but also \textit{where to branch}, i.e., where to keep reasoning paths open. The existing token-overlap metric captures only the former. We introduce the entropy shape, the sequence of next-token entropies, to characterize the latter, and propose the Entropy Shape Correlation (ESC) to measure student-teacher agreement in entropy shape. Tracking ESC throughout training and intervening on a teacher's entropy while preserving its token preferences, we empirically show that \textit{where to branch} causally drives OPD gains, independently of \textit{how to branch}. Finally, we formulate OPD outcome prediction as a teacher selection task and establish an evaluation protocol for it, under which pre-OPD ESC selects the best teacher for both students and outperforms existing predictors. Code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39020 [cs.CL] |
| (or arXiv:2610.39020v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39020 arXiv-issued DOI via DataCite |
Submission history
From: Shiping Yang [view email]
[v1]
Wed, 28 Oct 2026 22:53:06 UTC (165 KB)
来源：arXiv cs.AI · arxiv.org