"LLM 乘法错误研究：小模型低位出错保持奇偶性、高位趋于 off-by-one 进位失败"
"Close enough?" An investigation into multiplication errors in LLMs
AI 导读
一项入选 NeurIPS 2026 NewInML workshop 的研究评测了多个小规模基座 LLM 在 16,991 道多位数乘法题上的表现，发现乘法错误不符合加法错误所遵循的"噪声量化"模型。研究在小 Qwen 模型中发现新的错误模式：低位列错误大多保持奇偶性，高位数字错误则偏向 off-by-one 进位失败。这表明 LLM 的算术计算依赖多周期内部表示，奇偶性的追踪比具体数字更稳健。
正文 · 原文
Abstract:Large language models (LLMs) frequently fail at multi-digit arithmetic, but the underlying mechanisms governing these failures remain poorly understood. While recent work demonstrates that addition errors follow a "noisy quantisation" model localised around carry boundaries, whether this framework generalises to multiplication remains an open question. In this work, we evaluate multiplication capabilities across several small base LLMs on a dataset of 16,991 problems spanning varied operand lengths. We show that multiplication errors do not conform to the addition-style noisy quantisation model. Instead, we uncover a novel error regime in small Qwen models: errors in lower-order columns predominantly preserve parity. Conversely, higher-order digit errors shift toward off-by-one carry failures. Our findings suggest that arithmetic computation in LLMs relies on distinct, multi-periodic internal representations where parity is tracked more robustly than exact digit identity, providing new mechanistic insights into numerical reasoning in transformers.
| Comments: | Accepted to Neurips 2026 NewInML workshop |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38634 [cs.AI] |
| (or arXiv:2610.38634v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38634 arXiv-issued DOI via DataCite |
Submission history
From: Huw Day [view email]
[v1]
Wed, 28 Oct 2026 10:51:03 UTC (493 KB)
来源：arXiv cs.AI · arxiv.org