跳到正文
arXiv cs.AI· Yun Zhu Da Wang·· 56 分钟前AI 评分28

流式在线策略蒸馏:解耦三个时钟实现分阶段监督消费

Streaming On-Policy Distillation: Decoupling Three Clocks for Stage-Wise Supervision Consumption

AI 导读

论文提出流式在线策略蒸馏方法,将响应补全、监督交付与参数更新三个时钟解耦,使生成中新片段反复交付且仅在新增位置计算损失。

正文 · 原文

View PDF HTML (experimental)

Abstract:In on-policy distillation (OPD), response completion, supervision delivery, and parameter updates need not share the same granularity. We develop streaming OPD by decoupling these three clocks: a response repeatedly delivers newly generated segments while generation continues, retaining the full causal prefix at each delivery and computing loss only on new positions. A consumption protocol then organizes supervision budgets, position selection, staleness handling, and update scheduling, allowing control over quantity, coverage, and version lag across training stages. In GSM8K experiments distilling Qwen3-4B into Qwen3-0.6B with VERL, supervised-budget matching reduces the early gap between best test accuracies over seven common candidates from 4.55 to 2.42 points. Synchronous continuation from the same checkpoint reaches 73.39% best test accuracy over seven common candidates at u=384,...,576, versus 74.30% synchronous and 71.95% budget-matched (differences -0.91 and -2.35 points); an unpaired extended record reaches 76.12%. Freshness filtering and disagreement-overlap selection improve local peaks but do not establish sustained advantages. A system prototype shows lower data-supply waiting times without establishing an equal-update speedup. Quality recovery and supply-side benefits are assessed separately; end-to-end gains must account for continuation cost.
Comments: 37 pages, 8 figures
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22298 [cs.AI]
  (or arXiv:2611.22298v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22298

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Da Wang [view email]
[v1] Tue, 17 Nov 2026 06:55:10 UTC (152 KB)

来源:arXiv cs.AI · arxiv.org