跳到正文
arXiv cs.AI· Fengyuan Liu, Jay Gala, Siva Reddy, Dzmitry Bahdanau·· 2 小时前AI 评分42

EchoMoE:用激活压缩降低 MoE 专家并行的通信开销

Communicate the Change: Efficient Expert Parallelism via Activation Compression

AI 导读

研究者提出 EchoMoE,一种对现有 MoE 模型的轻量架构改造,用早期层隐藏态作参考,只传输压缩后的差值而非完整隐藏态。它将节点间通信量减少 58%,吞吐量提升 22.6%,18 个基准上平均性能损失在 3.8 个百分点内;在带宽受限的 Ethernet 上推理吞吐最高提升 2.38x,且兼容量化、提供 vLLM 推理集成,仅约 8,500x 少于完整预训练的训练 token 即可适配。

正文 · 原文

View PDF HTML (experimental)

Abstract:Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring approximately 8,500x fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 58% and improves throughput by 22.6%, while preserving average performance within 3.8 percentage points across 18 benchmarks. EchoMoE further improves inference throughput by up to 2.38x over bandwidth-constrained Ethernet, paving the way for wide-area distributed inference. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as: arXiv:2611.22215 [cs.LG]
  (or arXiv:2611.22215v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.22215

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Fengyuan Liu [view email]
[v1] Tue, 17 Nov 2026 04:52:41 UTC (945 KB)

来源:arXiv cs.AI · arxiv.org