arXiv cs.AI· Shuaishuai Zu, Jihao Zhao, Pengfei Li, Guanghuan Fang, Ben Feng, Yuhan Liu, Biao Qin, Baochuan Yang·· 4 小时前AI 评分35
O²PD：基于可靠前缀的重叠门控在线策略蒸馏方法
O$^2$PD: Overlap-Gated On-Policy Distillation on Reliable Prefix
AI 导读
研究者提出重叠门控在线策略蒸馏方法 O²PD，改进在线策略蒸馏（OPD）中“对齐哪些 token、如何对齐”两个问题。
正文 · 原文
Abstract:On-policy distillation (OPD) supplies dense token-level supervision by aligning a student to the teacher's next-token distribution on its own generated rollouts. Existing works mostly implement top-$k$ OPD, applying reverse Kullback-Leibler ($\RKL$) divergence between the teacher's and the student's renormalized distributions over teacher top-$k$ support. However, the teacher scores student-generated rollouts that are inherently off-policy for itself, leaving the reliability of its next-token guidance unclear. Restricting the divergence to the teacher top-$k$ support also discards the remaining mass outside that support. These gaps leave two questions implicit: 1) which tokens to align, and 2) how to align those tokens. To answer these questions, we propose Overlap-gated On-Policy Distillation (O$^{2}$PD). As for 1), we quantitatively analyze the reliability of teacher supervision and find that alignment on the student's trajectory prefix is more reliable than alignment on the full context. Therefore, our O$^{2}$PD is applied only to the 20\% prefix tokens of each student rollout. As for 2), we theoretically characterize two problems of top-$k$ OPD: the \emph{Teacher-support Uncovering} and the \emph{Dominant-Mode Mismatch}. Based on these analyses, we propose two support strategies: the \emph{Reset Support} strategy attaches a reset atom to every token, pooling mass outside the teacher top-$k$, and the \emph{Support Convergence} strategy converges the support to the already aligned intersection. Our O$^{2}$PD employs the designed overlap-gated $\alpha$-divergence to interpolate between mode-covering and mode-seeking for per-token alignment, where the overlap is used as a gate to switch the support and adaptively modulate $\alpha$.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07731 [cs.CL] |
| (or arXiv:2611.07731v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07731 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zu Shuaishuai [view email]
[v1]
Thu, 5 Nov 2026 09:43:03 UTC (511 KB)
来源：arXiv cs.AI · arxiv.org