舍入位置对齐可稳定 LLM 强化学习训练
Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning
AI 导读
大语言模型强化学习常因推理与训练系统实现差异导致本应 on-policy 的更新变成 off-policy、训练失稳。研究提出仅对齐训练侧 RoPE、SwiGLU 和 residual RMSNorm 三个区域的舍入位置，在 BF16 下保留选中中间量为 FP32。
正文 · 原文
Abstract:Reinforcement learning for large language models often uses separate systems for inference and training. Even with identical weights and BF16 tensors, implementation differences can make an intended on-policy update off-policy and destabilize training. Existing remedies align the entire inference and training stacks, change global precision, or modify the learning algorithm. We ask whether aligning selected parts of the execution stack can stabilize BF16 RL while other implementation differences remain. We align training-side rounding locations with inference in three regions---RoPE, SwiGLU, and residual RMSNorm---by retaining selected intermediates in FP32 until their BF16 outputs. Across four Qwen-family models, aligning all three regions prevents the collapse observed under the default BF16 baseline; RoPE-path alignment alone also suffices in the studied runs, despite remaining numerical discrepancies. The interventions keep the inference engine, matrix-multiplication and attention kernels, and learning objective unchanged.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32951 [cs.LG] |
| (or arXiv:2610.32951v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32951 arXiv-issued DOI via DataCite |
Submission history
From: Yijing Ke [view email]
[v1]
Fri, 23 Oct 2026 17:45:03 UTC (1,307 KB)
来源：arXiv cs.AI · arxiv.org