跳到正文
arXiv cs.AI· Viktor Vesel\'y, Sebastian Jitaru·· 4 小时前AI 评分33

MoE 路由器能从集成中获益吗?

Do Mixture-of-Experts Routers Benefit from Ensembling?

AI 导读

一项 arXiv 研究将集成学习应用于 Mixture-of-Experts 的路由步骤:让多个 router 与专家联合训练,共同提出专家权重,并在两个图像分类和两个表格回归任务上系统评估。

正文 · 原文

View PDF HTML (experimental)

Abstract:Ensembles of neural networks have repeatedly been shown to outperform single networks. Ensemble theory attributes this gain to the accuracy of the individual members and the diversity among them. Mixture-of-experts models already combine many experts, but a single router decides how the experts are weighted for each input. Motivated by ensemble theory, we apply ensembling to the routing step: an ensemble of routers, trained jointly with the experts, proposes the expert weights. We systematically evaluate this approach using underparameterized networks on two image-classification and two tabular-regression tasks. We analyse each experiment with a performance-diversity decomposition of the router ensemble, which, to our knowledge, has not previously been developed for routers. We report four findings. First, more routers lower the validation loss. Second, training routers to explicitly maximize expert diversity improves the mean performance on three of the four tasks. Third, explicitly maximizing router diversity has no effect on performance. Fourth, combining router weights with the arithmetic mean works better than combining them with the geometric mean. Routing benefits mildly from ensembling, but this benefit is offset by the computational cost of training additional routers.
Comments: 23 pages, 21 figures
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.17472 [cs.LG]
  (or arXiv:2611.17472v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.17472

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Viktor Veselý [view email]
[v1] Thu, 12 Nov 2026 19:58:39 UTC (633 KB)

来源:arXiv cs.AI · arxiv.org