ST-OPSD：用激活引导改进思考模型的同策略自蒸馏
Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models
AI 导读
论文提出 Steered-Teacher OPSD（ST-OPSD），解决同策略自蒸馏（OPSD）中教师模型因访问参考答案导致的参考幻觉和 reconsideration 抑制两类失效模式。
正文 · 原文
Abstract:On-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model. However, access to the reference can induce teacher behaviors that undermine independent reasoning in thinking models, leading to reference hallucinations, where the student cites an unavailable reference to justify reasoning steps, and reconsideration suppression, where the teacher discourages verification and revision at uncertain steps. We introduce Steered-Teacher OPSD (ST-OPSD) to jointly address these failure modes through activation steering. ST-OPSD extracts two directions from model-generated reasoning trajectories and applies them to the teacher's hidden states during distillation target computation, suppressing reference-invoking behavior and encouraging reconsideration with marker-family calibration. The student learns from the resulting teacher distributions without being steered. Experiments on AIME and HMMT show that ST-OPSD improves accuracy over vanilla OPSD and other interventions while restoring reconsideration and reference-hallucination rates to base-model levels.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34005 [cs.LG] |
| (or arXiv:2610.34005v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34005 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaoou Liu [view email]
[v1]
Sun, 25 Oct 2026 02:34:40 UTC (2,854 KB)
来源：arXiv cs.AI · arxiv.org