arXiv cs.AI· Yue He, Jiahui Hu, Shaman Tang, Hua Zhou, Qiman Wu, Menglin Li, Yizhao Wang, Yuheng Zhang, Da Zhu, Zhou Wang, Anhua Liu, Hanlin Chen, Mingyuan Wang, Rui Xin, Yuhua Wei, Shuang Su, Siqi Liu, Kai Yang, Weichao Huang, Hang Zhang, Honggou Yang, Xianming Liu·· 3 小时前AI 评分39
Turing LLM 后训练研究：校准蒸馏与高效 Dynamic-MoE 执行
Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution
AI 导读
一篇 arXiv 论文提出针对动态路由 MoE 模型的后训练方法，结合校准多教师在线蒸馏（C-MOPD）与基于 PPU 的高效执行路径。Turing-20B-A2B 四领域归一化整合得分达 0.9988，每个领域保留至少 90.16% 的专家增益；在 8K 响应上限下四项数学与代码评测未加权均分 85.92，优于 Qwen3-8B 的 80.33 和 SFT 的 74.21。
正文 · 原文
Authors:Yue He, Jiahui Hu, Shaman Tang, Hua Zhou, Qiman Wu, Menglin Li, Yizhao Wang, Yuheng Zhang, Da Zhu, Zhou Wang, Anhua Liu, Hanlin Chen, Mingyuan Wang, Rui Xin, Yuhua Wei, Shuang Su, Siqi Liu, Kai Yang, Weichao Huang, Hang Zhang, Honggou Yang, Xianming Liu
Abstract:We study post-training for dynamically routed mixture-of-experts (MoE) models, focusing on integrating independently optimized specialists while keeping consolidation efficient. Our approach combines calibrated multi-teacher on-policy distillation (C-MOPD), which matches each selected teacher on a local Top-$K$ support with teacher-wise loss normalization and clipping, with an execution path that uses post-capacity bounded-width dispatch, final-route replay, and asynchronous rollout--training overlap. On Turing-20B-A2B, the selected model achieves a four-domain normalized integration score of 0.9988 and retains at least 90.16\% of the measured specialist gain in every domain. At matched $K=32$, the full calibration objective scores 0.9962, compared with 0.9724 without calibration and 0.9834 without clipping. At a matched 8K response cap, the model exceeds both shared SFT and Qwen3-8B on all four reasoning-enabled mathematics and code benchmarks (unweighted mean: 85.92, versus 80.33 for Qwen3-8B and 74.21 for SFT), and improves over SFT on ten of twelve broader evaluation metrics. In a PPU-based profile, the combined execution configuration reduces consolidation step time from 398.7 to 139.3 seconds, an observed 2.86$\times$ speedup over serial execution. These results demonstrate strong specialist integration alongside more efficient consolidation, while reflecting the complete training and execution framework rather than any single component.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39876 [cs.LG] |
| (or arXiv:2610.39876v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39876 arXiv-issued DOI via DataCite |
Submission history
From: Shaman Tang [view email]
[v1]
Thu, 29 Oct 2026 11:52:20 UTC (492 KB)
来源：arXiv cs.AI · arxiv.org