前沿模型配方能否叠加?小语言模型组件的重复因子实验分析
Do Frontier Model Ingredients Add Up? A Replicated Factorial Study of Small Language Models
AI 导读
一项 arXiv 研究在相同数据和训练预算下,对同一小语言模型比较五种已发表的改进方案,并对每种组合用第二个随机种子重复实验。结果显示 Mixture-of-experts 层、Muon 优化和 Gated DeltaNet 混合结构收益最大,额外归一化也有帮助,而参数预算匹配的注意力门控平均收益很小。
正文 · 原文
Abstract:Building a small language model requires choosing which published improvements to combine. We compare five changes in the same model, using the same data and training budget. We test selected combinations and repeat each one with a second random seed. Mixture-of-experts layers, Muon optimisation and a Gated DeltaNet hybrid give the largest gains. Extra normalisation also helps, while an attention gate with a matched parameter budget adds little on average. The gains mostly combine, but small interactions between changes appear again in the repeated runs. Repetition also reveals benefits that an initial statistical estimate treated as noise. These findings apply to our small-model training setting; the expert models also have greater total capacity. The practical lesson is to test promising components together and repeat the comparisons before deciding which ones to keep.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24540 [cs.CL] |
| (or arXiv:2611.24540v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24540 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Biswa Sengupta [view email]
[v1]
Wed, 18 Nov 2026 14:48:52 UTC (38 KB)
来源:arXiv cs.AI · arxiv.org