arXiv cs.AI· Yiqiao Jin, Yiyang Wang, Lucheng Fu, Yijia Xiao, Yinyi Luo, Haoxin Liu, B. Aditya Prakash, Chao Zhang, Josiah Hester, Jindong Wang, Srijan Kumar·· 4 小时前AI 评分38
UniSD：面向大语言模型的统一自蒸馏框架
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
AI 导读
研究者提出 UniSD，一个系统研究大语言模型自蒸馏的统一框架，整合多教师一致性、EMA 教师稳定化、token 级对比学习、特征匹配和散度裁剪等机制。该框架在 3 个模型家族的 6 个模型、6 个 benchmark 上分析自蒸馏何时优于静态模仿及各组件的作用。由此构建的 UniSDfull 比基础模型提升 +5.4 分，比最强基线提升 +2.8 分。
正文 · 原文
Authors:Yiqiao Jin, Yiyang Wang, Lucheng Fu, Yijia Xiao, Yinyi Luo, Haoxin Liu, B. Aditya Prakash, Chao Zhang, Josiah Hester, Jindong Wang, Srijan Kumar
Abstract:Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. However, SD in autoregressive LLMs remains challenging because self-generated trajectories are free-form, correctness is task-dependent, and plausible rationales can still provide unstable or unreliable supervision. Existing methods mainly examine isolated design choices, leaving their effectiveness, roles, and interactions unclear. In this paper, we propose UniSD, a unified framework to systematically study self-distillation. UniSD integrates complementary mechanisms that address supervision reliability, representation alignment, and training stability, including multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. Across six benchmarks and six models from three model families, UniSD reveals when self-distillation improves over static imitation, which components drive the gains, and how these components interact across tasks. Guided by these insights, we construct UniSDfull, an integrated pipeline that combines complementary components and achieves the strongest overall performance, improving over the base model by +5.4 points and the strongest baseline by +2.8 points. Extensive evaluation highlights self-distillation as a practical and steerable approach for efficient LLM adaptation without stronger external teachers.
| Comments: | Website: this https URL Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.06597 [cs.CL] |
| (or arXiv:2605.06597v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06597 arXiv-issued DOI via DataCite |
Submission history
From: Yiqiao Jin [view email]
[v1]
Thu, 7 May 2026 17:22:11 UTC (932 KB)
[v2]
Thu, 21 May 2026 16:25:55 UTC (936 KB)
[v3]
Thu, 5 Nov 2026 08:43:27 UTC (1,413 KB)
来源：arXiv cs.AI · arxiv.org