arXiv cs.AI· Hei Yi Mak, Hoang Le, Mehran Taghian Jazi, Yunke Peng, Huan Lin, Zhengzhi Tang, Zhexu Liu, Yao Wang, Yaoyuan Wang, Anandharaju Durai Raju, Tanzila Rahman·· 5 小时前AI 评分41
面向 MoE 大语言模型端到端 FP4 强化学习的训练与推理策略对齐研究
Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs
AI 导读
一项 arXiv 研究提出 MoEResQ，一种用于 MoE 大语言模型 FP4 强化学习后训练的量化恢复方案。研究发现 FP4 RL 的稠密模型恢复配方在 MoE 上失效，根因是路由器的离散专家选择使训练与推理对齐对恢复技术是否对称部署高度敏感：非对称配方会显著恶化 Pearson 相关系数与路由器选择匹配率。
正文 · 原文
Authors:Hei Yi Mak, Hoang Le, Mehran Taghian Jazi, Yunke Peng, Huan Lin, Zhengzhi Tang, Zhexu Liu, Yao Wang, Yaoyuan Wang, Anandharaju Durai Raju, Tanzila Rahman
Abstract:In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoEResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoEResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.04376 [cs.LG] |
| (or arXiv:2611.04376v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.04376 arXiv-issued DOI via DataCite |
Submission history
From: Hei Yi Mak [view email]
[v1]
Tue, 3 Nov 2026 06:03:38 UTC (751 KB)
[v2]
Wed, 4 Nov 2026 03:54:17 UTC (751 KB)
来源：arXiv cs.AI · arxiv.org