面向推理蒸馏的高效教师支持研究：Qwen3 4B→1.7B 仅需 0.01% 词表支持
Effective Teacher Support for Reasoning Distillation
AI 导读
一项 arXiv 研究提出从教师模型最高概率词表子集进行蒸馏的方法，将支持范围从 100% 压缩至 0.01%。在 Qwen3 4B→1.7B 蒸馏中，保留 0.01% 词表在 MMLU-Pro 上相对全词表蒸馏无退化，Qwen3 8B→4B 与 Nemotron Nano 12B→9B 亦验证该发现。
正文 · 原文
Abstract:Forward-KL language-model distillation normally computes dense teacher and student scores over the full vocabulary at every supervised position, making training and stored teacher targets expensive. We instead distill from the teacher's highest-probability vocabulary fraction, renormalizing the retained mass as we sweep support from 100% to 0.01%. Our main study distills Qwen3 4B -> 1.7B. We confirm the broader support-reduction finding on larger Qwen3 8B -> 4B and Nemotron Nano 12B -> 9B pairs and under answer-only supervision. We evaluate generation primarily on MMLU-Pro and secondarily on GPQA-Diamond. Across the Qwen3 settings, retaining 0.01% of the vocabulary causes no observed MMLU-Pro degradation relative to full-vocabulary distillation, although disagreement with the complete teacher distribution increases. Nemotron Nano matches full-vocabulary MMLU-Pro at 0.1%, but smaller fractions fail to emit the closing reasoning tag, showing that the reliable cutoff depends on the model family. At 0.01% support, caching the retained next-token scores from the teacher forward pass cuts training step time by 38%, while restricting student output computation to those tokens cuts peak memory by 24%; neither causes observed MMLU-Pro degradation.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.14850 [cs.AI] |
| (or arXiv:2611.14850v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14850 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dhruv Desai [view email]
[v1]
Wed, 11 Nov 2026 00:00:22 UTC (914 KB)
来源：arXiv cs.AI · arxiv.org