跳到正文
arXiv cs.AI· Tiantong Wang, Zhiren Gong, Yixin Wang, Tiantong Wu, Wei Yang Bryan Lim·· 4 小时前AI 评分43

MoE 专家共享理论:从局部预测误差到多步推理最终输出的风险分析

From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning

AI 导读

一篇 arXiv 论文提出多步推理中 Mixture-of-experts(MoE)专家共享的理论分析,研究局部预测误差的降低何时能在最终输出中保留。在线性回归设定下推导出精确的共享与合并准则,并构造出共享降低平均局部误差却增大最终输出误差的案例;实验显示低数据符号任务中训练的 Transformer 在 teacher-forced 单步上有统计显著的增益,但自由运行的最终输出增益未获统计分辨。

正文 · 原文

View PDF HTML (experimental)

Abstract:Mixture-of-experts (MoE) models can share experts across layers or computational steps, pooling training information from different contexts. This can reduce local prediction error, but multi-step systems are ultimately evaluated by their final outputs. We study when these local gains survive composition. In specified linear regression models, we derive exact sharing and merging criteria by balancing variance savings against bias from sharing across roles with different target behavior. Under a common fixed linear propagation map, we characterize expected squared final-output risk, retaining error directions and correlations. We construct cases where sharing lowers average local error yet increases final-output error. Within this setting, we also identify when orthogonal full pooling with isotropic noise preserves every local improvement at the final output. For general routed computations, we derive conditional end-to-end error bounds that account for distribution shifts, errors in surrounding components, and changes in expert selection. Separately, we show that correct predictions and route labels alone do not establish that individual experts implement the attributed operations. Controlled experiments realize a strict local-to-final risk reversal. In low-data symbolic settings, trained Transformers exhibit statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains. Separate intervention tests distinguish successful multi-layer actions from single-expert attribution and generalization to unseen transitions. Together, these results distinguish when sharing improves local estimation, when those gains survive to the final output, and what evidence is needed to attribute reusable operations to individual experts.
Comments: 96 pages, 3 figures, 28 tables
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.17879 [cs.LG]
  (or arXiv:2611.17879v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.17879

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Tiantong Wang [view email]
[v1] Fri, 13 Nov 2026 08:13:39 UTC (243 KB)

来源:arXiv cs.AI · arxiv.org