LRMoE：面向内存受限节点的高效 MoE 训练需求侧负载再校准系统
LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes
AI 导读
论文提出 LRMoE，一种内存感知的 MoE 训练系统，通过联合再校准反向传播需求并均衡保留的工作负载，缓解稀疏 MoE 模型因路由不均导致的计算与通信瓶颈。前向执行时规划保留配额与专家放置，反向传播时按 token-expert 对的贡献选取配额内计算，并通过物理压缩减小通信负载。
正文 · 原文
Abstract:Sparse Mixture-of-Experts (MoE) models offer scalable capacity for Web applications, but uneven routing creates computation and communication bottlenecks. Existing placement and replication methods redistribute this workload, yet additional expert copies compete with token-related buffers for limited GPU memory. We present LRMoE, a memory-aware MoE training system that jointly recalibrates backward demand and balances the retained workload. During forward execution, it plans retention quotas and expert placement under resource constraints while preserving all routed computations. During backpropagation, a load recalibration metric estimates each token-expert pair's contribution to the expert update and selects pairs within these quotas. A capacity-aware routing algorithm distributes retained pairs among eligible instances of their original experts. Physical compaction then forms smaller communication payloads and variable-size grouped GEMMs, reducing backward traffic, computation, and execution-buffer demand. Experiments on Qwen3-30B-A3B and GLM4.5-106B-A12B show that LRMoE reduces cumulative MoE-layer training time to fixed-loss targets by 21.2-44.8% relative to SYMI.
| Subjects: | Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.37720 [cs.AI] |
| (or arXiv:2610.37720v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37720 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiaming Hu [view email]
[v1]
Tue, 27 Oct 2026 09:41:17 UTC (5,166 KB)
来源：arXiv cs.AI · arxiv.org