TOP-D：用信赖域策略蒸馏稳定在线策略蒸馏的训练
Trust Region Policy Distillation
AI 导读
论文提出信赖域策略蒸馏方法 TOP-D，通过动态构建近端教师，把不稳定、高方差的在线策略蒸馏（OPD）转变为稳定训练范式。理论方面给出全局收敛分析与单调改进界，证明其能内在控制梯度方差。实验显示 TOP-D 在数学推理任务上显著提升训练稳定性、样本效率与最终性能，且不增加任何额外计算开销。
正文 · 原文
Abstract:Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable, high-variance On-Policy Distillation (OPD) into a stable training paradigm by dynamically constructing a proximal teacher. Theoretically, we establish a rigorous framework demonstrating that TOP-D inherently controls gradient variance. By providing a formal global convergence analysis alongside a monotonic improvement bound, we mathematically formalize the reliability and stability of the overall training dynamics. Empirically, TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. More importantly, TOP-D introduces zero additional computational overhead, positioning itself as a promising alternative to the well-established OPD paradigm.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.04751 [cs.LG] |
| (or arXiv:2607.04751v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.04751 arXiv-issued DOI via DataCite |
Submission history
From: Zhengpeng Xie [view email]
[v1]
Mon, 6 Jul 2026 07:43:36 UTC (129 KB)
[v2]
Mon, 26 Oct 2026 23:22:57 UTC (141 KB)
来源：arXiv cs.AI · arxiv.org