基于评分规则的反思研究：黑盒 On-policy 蒸馏分析

View PDF HTML (experimental)

Abstract:On-policy distillation (OPD) is a powerful paradigm for LLM post-training, yet its reliance on teacher logits restricts it to white-box teachers. In this paper, we study whether rubric-based supervision can provide a text-only feedback interface for black-box OPD, and what makes it effective. Our instantiation, Rubric-based On-Policy Distillation (ROPD), induces rubrics by contrasting teacher responses with the student's current rollouts and uses them to score these rollouts for on-policy optimization. We find that, under matched settings, text-only ROPD outperforms logit-based OPD methods on four competition-math benchmarks, reaching the best AIME24 accuracy of standard logit-based OPD with 9.6x fewer training samples. Our analysis yields three findings: rubric rewards track correctness far better than teacher log-likelihood (AUC 0.90 vs. 0.35); training with them repairs more failed criteria and regresses fewer passed ones than logit-based supervision; and contrasting teacher responses with student rollouts is central to rubric quality. These results position rubric-based OPD as a flexible, black-box-compatible alternative to logit-based OPD. Our implementation is available at this https URL.

Comments:	Preprint. Code is available at this https URL
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2605.07396 [cs.LG]
	(or arXiv:2605.07396v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.07396 arXiv-issued DOI via DataCite

Submission history

From: Zhepei Hong [view email]
[v1] Fri, 8 May 2026 07:52:15 UTC (4,254 KB)
[v2] Tue, 10 Nov 2026 03:40:16 UTC (872 KB)