IterCaL：无需目标域奖励反馈即可缓解 LLM 蒸馏中教师偏差的迭代校准与学习方法
Iterative Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
AI 导读
提出迭代校准与学习方法 IterCaL，通过 token 级分支做源奖励教师校准，并交替在目标提示词上训练学生模型，以缓解目标奖励反馈缺失时 LLM 蒸馏中的教师偏差传播。
正文 · 原文
Abstract:Large language model (LLM) distillation can propagate teacher bias under covariate shift when target reward feedback is unavailable. We propose Iterative Calibration and Learning (IterCaL), which alternates source-reward teacher calibration through token-level branching with student learning on target prompts. We prove polynomial convergence in expected average KL to the oracle student, the reference-regularized target-reward maximizer within a potentially misspecified student class; a quantified source-to-target calibration discrepancy gives an explicit additive true-return term. Our separation shows that regularized direct teacher matching retains positive oracle-KL error at its population optimum for every shared reference penalty, even when the teacher is stronger than every student. On the same instance, IterCaL converges to the oracle using source rewards alone.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Statistics Theory (math.ST); Machine Learning (stat.ML) |
| Cite as: | arXiv:2609.17474 [cs.LG] |
| (or arXiv:2609.17474v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.17474 arXiv-issued DOI via DataCite |
Submission history
From: Haichen Hu [view email]
[v1]
Tue, 15 Sep 2026 17:15:40 UTC (102 KB)
[v2]
Thu, 29 Oct 2026 03:09:44 UTC (122 KB)
来源：arXiv cs.AI · arxiv.org