从分布匹配视角统一理解 LLM 训练算法
A Distribution-Matching Perspective on LLM Training Algorithms
AI 导读
一篇 arXiv 论文提出基于分布匹配的统一框架，将语言模型训练算法解释为最小化当前模型与目标分布之间的散度。分析表明 On-Policy Distillation 在模型分布接近目标分布时优于经典 Knowledge Distillation，而分布差异大时处于劣势；当目标分布依赖被优化模型时，估计方差累积会导致目标分布的确定性退化。
正文 · 原文
Abstract:In this work, we propose a unified framework for training algorithms across different stages of modern large language model development, based on the perspective of distribution matching. We show that a broad class of language model training algorithms can be interpreted as minimizing a certain divergence between the current model and a target distribution, and that characterizing the properties of this target distribution provides substantial insight into the behavior of the corresponding training algorithms. By analyzing specific forms of the target distribution, we establish that On-Policy Distillation (OPD) enjoys an advantage over classical Knowledge Distillation (KD) when the current model distribution is close to the target distribution, while exhibiting a disadvantage when the two distributions are far apart. Furthermore, when the target distribution itself depends on the model being optimized, we show that the accumulation of estimation variance can induce a deterministic degradation of the target distribution. These results provide a theoretical explanation, at least in part, for the long-term behavior of existing Reinforcement Learning (RL) algorithms. With this intuition we further designed Conditional KL (CKL) regularization for mitigating the degradation of the target distribution. CKL can effectively preserve mode diversity while relaxing the constraint imposed by classical KL regularization on reward improvements. We believe this theoretical perspective can bring some new understanding of language model training algorithms and provide useful guidance for future research.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33740 [cs.LG] |
| (or arXiv:2610.33740v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33740 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yanxing Huang [view email]
[v1]
Sat, 24 Oct 2026 15:30:39 UTC (1,029 KB)
来源：arXiv cs.AI · arxiv.org