arXiv 论文提出 Transformer 错误率模型并用 Gemini 与 DeepSeek R1 验证
A model of errors in transformers
AI 导读
Suvrat Raju 与 Praneeth Netrapalli 在 arXiv 论文（arXiv:2601.14175）中提出 Transformer 错误率模型，认为 LLM 在算术等确定性任务上的错误源于注意力机制中的小误差累积越过阈值，并据此推导出精度与任务复杂度之间的双参数定量关系。
正文 · 原文
Abstract:We study the error rate of LLMs on tasks like arithmetic that require a deterministic output, and repetitive processing of tokens drawn from a small set of alternatives. We argue that incorrect predictions arise when small errors in the attention mechanism accumulate to cross a threshold, and use this insight to derive a quantitative two-parameter relationship between the accuracy and the complexity of the task. The two parameters vary with the prompt and the model; they can be interpreted in terms of an elementary noise rate, and the number of plausible erroneous tokens that can be predicted. Our analysis is inspired by an ``effective field theory'' perspective: the LLM's many raw parameters can be reorganized into just two parameters that govern the error rate. We perform extensive empirical tests, using Gemini 2.5 Flash, Gemini 2.5 Pro and DeepSeek R1, and find excellent agreement between the predicted and observed accuracy for a variety of tasks, although we also identify deviations in some cases. Our model provides an alternative to suggestions that errors made by LLMs on long repetitive tasks indicate the ``collapse of reasoning'', or an inability to express ``compositional'' functions. Finally, we show how to construct prompts to reduce the error rate.
| Comments: | (v1) 8+17pages (v2) Appendix A.4 expanded; consistent with published version |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); High Energy Physics - Theory (hep-th) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2601.14175 [cs.LG] |
| (or arXiv:2601.14175v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.14175 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 43rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 |
Submission history
From: Suvrat Raju [view email]
[v1]
Tue, 20 Jan 2026 17:27:03 UTC (146 KB)
[v2]
Thu, 12 Nov 2026 13:22:46 UTC (203 KB)
来源：arXiv cs.AI · arxiv.org