arXiv cs.AI· Hao Cheng (Zhejiang Gongshang University), Zhiyu Yang (Chang'an University), Bo Ma (Zhejiang Gongshang University), Hanzhang Jia (University of Sussex)·· 3 小时前AI 评分37
推理时专家停用如何匹配训练时的权重规则：arXiv 新研究比较四种实现
Matching the Training-Time Weight Rule in Inference-Time Expert Deactivation
AI 导读
一项新研究系统比较了推理时停用 MoE 专家的四种实现（移除、冻结、剩余专家重归一化、非归一化重路由），发现效果取决于实现是否匹配训练时的权重规则。在五个指令微调模型上加基于激活排名停用一半专家时，遵循训练规则的困惑度为基线的 1.4 至 73 倍，规则错配时升至 14 至 28,740 倍；OLMoE 上 GSM8K 严格匹配准确率在四种实现下均降为零。
正文 · 原文
Authors:Hao Cheng (Zhejiang Gongshang University), Zhiyu Yang (Chang'an University), Bo Ma (Zhejiang Gongshang University), Hanzhang Jia (University of Sussex)
Abstract:Expert deactivation at inference time changes both token routing and expert weights, and its effect on performance depends on how these changes are implemented. We compare removal, freezing, remaining-expert renormalization, and unnormalized re-routing on identical disabled sets across five instruction-tuned checkpoints from four model families, with a base OLMoE checkpoint as an additional comparison. We develop a weight-sum model that relates the loss of expert contributions and changes in branch scale to the weight rule used during training. With half of the experts deactivated according to activation-based rankings, remaining-expert renormalization yields the highest perplexity in models trained with raw router probabilities, whereas unnormalized re-routing yields the highest perplexity in models trained with normalized weights. Perplexity ranges from 1.4 to 73 times the intact baseline when the training rule is retained, and from 14 to 28,740 times the baseline under a rule mismatch. Task outcomes vary across evaluation types: on OLMoE, GSM8K strict-match accuracy falls to zero under all four implementations. Extending the model to changes in the number of active experts yields a further prediction, supported by the OLMoE experiments: the perplexity penalty from additional renormalization increases as fewer experts are activated.
| Comments: | Corresponding authors: Hanzhang Jia (lead corresponding) and Bo Ma |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.05804 [cs.CL] |
| (or arXiv:2611.05804v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05804 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanzhang Jia [view email]
[v1]
Wed, 4 Nov 2026 01:23:13 UTC (93 KB)
来源：arXiv cs.AI · arxiv.org