为什么 Speculative Decoding 在稀疏 MoE 模型上的加速比预测更好
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
AI 导读
一项入选 NeurIPS 2026 主会的研究评估了 10 个稀疏 MoE 模型（30B–1T 参数、8–512 专家）上的 Speculative Decoding，发现细粒度模型在低 batch size 下反而获得更高加速，打破了此前预测的凹形加速-batch size 关系。
正文 · 原文
Abstract:Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a concave speedup-batch size relationship, with gains peaking at intermediate batch sizes due to expert saturation. We evaluate this across 10 sparse MoE models (30B--1T parameters, 8--512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size. We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics. At low batch size, MoE layers in fine-grained models account for a much smaller share of runtime than of weights; on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts. Across draft tokens, routing overlap further reduces the unique experts activated during verification relative to independence assumptions. We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4\% error, and substantially narrows the gap to real-model curves and captures the qualitative low-batch trend from synthetic routing.
| Comments: | Accepted as at NeurIPS 2026 main track conference |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38981 [cs.LG] |
| (or arXiv:2610.38981v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38981 arXiv-issued DOI via DataCite |
Submission history
From: Ekagra Ranjan [view email]
[v1]
Wed, 28 Oct 2026 22:04:53 UTC (3,789 KB)
来源：arXiv cs.AI · arxiv.org