arXiv cs.AI· Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg·· 5 小时前AI 评分52
arXiv 论文提出稀疏 MoE 让奖励模型专家可解释且可组合，少样本个性化提升 25.81 点
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
AI 导读
论文审计 Mixture-of-Experts（MoE）奖励模型的可解释性，发现常规训练学到的组件相互纠缠，问题源自训练目标而非架构本身。引入现成的稀疏与多样性正则后，稀疏 MoE 在受控设置下专家纯度达 84.8%（常规 MoE 为 60.8%），对路由权重的干预可产生可预测的偏好变化，且轻量路由更新使 50 样本少样本个性化提升 25.81 点，约为相同架构下纠缠专家增益的 2.7 倍。
正文 · 原文
Abstract:Preference modeling aims to capture heterogeneous human values and increasingly relies on learning latent components to represent this variation without fine-grained supervision. What these components actually represent, however, is rarely examined. We audit Mixture-of-Experts (MoE) reward models along three rungs---semantic coherence, functional correspondence, and composability---in controlled and real-world settings, holding the architecture and training data fixed so that differences isolate the effect of the training objective. Our diagnosis is that the components conventional MoE reward models learn are entangled, which limits not only interpretability but also the personalization such models are built for. The entanglement is not intrinsic to the architecture but a property of the training objective: off-the-shelf sparsity and diversity regularizers already substantially reduce it. The resulting configuration, which we call sparse MoE, recovers all underlying semantic categories without domain supervision in a controlled setting, reaching 84.8% expert purity against 60.8% for conventional MoE; on real-world data its experts admit human-interpretable task- and domain-level descriptions aligned with their functional strengths. Targeted interventions on routing weights produce predictable preference changes, showing that these interpretations are behaviorally grounded rather than merely descriptive. Crucially, the disentanglement induced by this objective also makes the experts composable: lightweight router updates yield a 25.81-point improvement in few-shot personalization with 50 examples, roughly 2.7 times the gain of the identical architecture and adaptation procedure applied to entangled experts. Together, these results show that MoE reward models can learn interpretable expert structure, and that the same structure can substantially improve personalization.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.04284 [cs.LG] |
| (or arXiv:2606.04284v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.04284 arXiv-issued DOI via DataCite |
Submission history
From: Yifan Wang [view email]
[v1]
Tue, 2 Jun 2026 23:19:12 UTC (1,474 KB)
[v2]
Tue, 27 Oct 2026 18:34:57 UTC (1,509 KB)
来源：arXiv cs.AI · arxiv.org