arXiv 论文从语言模型预训练追溯 AI 算法进步的来源

View PDF HTML (experimental)

Abstract:Algorithms have been estimated to increase AI pretraining FLOP efficiency by a factor of 22,000 between 2012 and 2023 (Ho et al., 2024). Running ablation experiments on key innovations from this time period, we are able to account for less than 10x of these gains at small compute scales. Surveying the broader literature, we estimate that additional innovations not included in our ablations also account for less than 10x, yielding less than 100x efficiency gains total in low-compute regimes. This leads us to conduct scaling experiments, which provide evidence that the efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding evidence for exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that, contrary to standard assumptions, an algorithm's efficiency gains are tied to compute scale. Using experimental scaling extrapolations and literature estimates, we can account for the majority of efficiency gains over the same time period. Our results indicate that algorithmic progress at smaller training compute levels has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.

Comments:	Accepted at NeurIPS 2026 (camera-ready version)
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
MSC classes:	68T07, 68Q25
ACM classes:	I.2.6
Cite as:	arXiv:2511.21622 [cs.LG]
	(or arXiv:2511.21622v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2511.21622 arXiv-issued DOI via DataCite

Submission history

From: Hans Gundlach [view email]
[v1] Wed, 26 Nov 2025 17:46:31 UTC (5,021 KB)
[v2] Tue, 10 Nov 2026 10:05:28 UTC (5,170 KB)