提升前馈层表达能力（上）：Token 自适应的激活混合方法 MoA
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
AI 导读
一篇 NeurIPS 2026 论文提出 Mixture of Activations（MoA），一种 token 自适应的 FFN 设计，用轻量输入相关门控混合激活函数字典，共享线性投影；同时提出输入无关的可学习激活（LA）。
正文 · 原文
Abstract:Feedforward network (FFN) layers account for a large fraction of parameters and nonlinear expressivity in Transformer-based large language models (LLMs). Despite the evolution from ReLU and GELU to gated variants such as SwiGLU, most FFN designs still use a single fixed activation function, applying the same nonlinear transformation to all tokens. In this work, we propose Mixture of Activations (MoA), a token-adaptive FFN design that mixes a dictionary of activation functions using lightweight input-dependent gates while sharing the same linear projections. As an input-independent counterpart, we also introduce learnable activations (LA), which form linear combinations of activation functions for both ReLU-type and SwiGLU-type FFNs. Theoretically, we establish strict finite-width expressive separations among fixed-activation FFNs, LA, and MoA: LA strictly contains fixed-activation FFNs, while MoA strictly contains LA, with the additional expressivity arising from input-dependent nonlinear hybridization. Empirically, we evaluate MoA through extensive pre-training experiments on dense and MoE language models ranging from 0.12B to 2B parameters under different token budgets, optimizers, and learning rate schedules. MoA consistently achieves lower terminal loss and exhibits more favorable scaling behavior than well-tuned baselines, with minimal parameter and computational overhead. These results suggest that token-adaptive activation mixing is a simple and effective mechanism for improving FFN expressivity in LLMs.
| Comments: | 31 pages, accepted by NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2605.26647 [cs.LG] |
| (or arXiv:2605.26647v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.26647 arXiv-issued DOI via DataCite |
Submission history
From: Jinbo Wang [view email]
[v1]
Tue, 26 May 2026 07:30:53 UTC (6,072 KB)
[v2]
Fri, 28 Aug 2026 05:17:17 UTC (6,074 KB)
[v3]
Wed, 4 Nov 2026 14:58:50 UTC (2,105 KB)
来源：arXiv cs.AI · arxiv.org