语言模型预训练优化器大比较：Muon、Soap 等十年优化器系统性评测
Fantastic Pretraining Optimizers and Where to Find Them
AI 导读
一项系统性研究在 0.1B-1.2B 参数的四个模型规模和 1-8 倍 Chinchilla 最优数据比下评测了十种深度学习优化器。研究发现，经严格调参后，许多优化器相对 AdamW 的实际加速低于宣称值，且随模型规模增大而下降，最快的 Muon、Soap 等矩阵预条件优化器在 0.1B 模型上加速 1.4 倍，到 1.2B 模型仅剩 1.1 倍。
正文 · 原文
Abstract:AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups. To address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training. First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. Second, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1x for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay. Through our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners -- multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.
| Comments: | 108 pages, 8 figures, reproducible runs available at this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2509.02046 [cs.LG] |
| (or arXiv:2509.02046v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.02046 arXiv-issued DOI via DataCite |
Submission history
From: Kaiyue Wen [view email]
[v1]
Tue, 2 Sep 2025 07:43:22 UTC (701 KB)
[v2]
Thu, 4 Sep 2025 19:22:04 UTC (707 KB)
[v3]
Sun, 25 Oct 2026 04:22:04 UTC (707 KB)
来源：arXiv cs.AI · arxiv.org