arXiv cs.AI· Zaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum·· 3 小时前AI 评分51
arXiv 论文提出时间炸弹式 loss spike 由量化误差驱动
The Role of Quantization in Loss Spikes
AI 导读
arXiv 论文（arXiv:2610.38277）研究语言模型预训练中长时间训练后才出现的延迟发散 loss spike，命名为 time-bomb spikes，发现其由 LM 头中快速增长的量化误差驱动，降低数值精度会加速其出现。作者提出将 LM 头的 AdamW 优化器替换为带动量的 SGD，抑制 logit 增长并提升长期训练稳定性，并指出已有缓解方法均通过降低量化误差起效。
正文 · 原文
Abstract:Loss spikes in language model pretraining can waste substantial compute and derail otherwise healthy training runs. A primary driver of these instabilities is the growth and divergence of output logits in the final language modeling (LM) head. In this work, we study a particularly costly form of this instability associated with output logit growth: delayed, terminal divergences that emerge only after prolonged training, which we call time-bomb spikes. We find that these instabilities are preceded by rapidly growing quantization error in the LM head, which eventually culminates in a loss spike. Crucially, reducing numerical precision accelerates time-bomb spikes, while increasing precision improves stability. We then propose a simple yet effective mitigation that replaces AdamW on the LM head with SGD with momentum, suppressing logit growth and improving stability over long training horizons. Finally, we find that existing mitigation techniques from the literature (along with our own) all reduce quantization error, suggesting a shared numerical mechanism behind their effectiveness. Together, our results identify finite precision error as a hidden driver of time-bomb spikes and show that suppressing this error is key to stable long-horizon pretraining.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38277 [cs.LG] |
| (or arXiv:2610.38277v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38277 arXiv-issued DOI via DataCite |
Submission history
From: Zaiqian Chen [view email]
[v1]
Wed, 28 Oct 2026 04:53:13 UTC (2,078 KB)
来源：arXiv cs.AI · arxiv.org