研究揭示 LLM 数学符号错误多产生于网络后四分之一层且定位不等于可操纵性
Localization Does Not Imply Steerability: A Mechanistic Study of Sign Errors in LLM Mathematics
AI 导读
arXiv 论文（arXiv:2611.07478）研究大语言模型多步数学中写错符号 token 的机制，覆盖 14B 至 675B 参数的六个开源模型，发现错误符号由网络最后四分之一（每模型和任务 70-100%）的 MLP 层在后期写入。
正文 · 原文
Abstract:When large language models do multi-step mathematics, a characteristic failure recurs: the model writes one wrong sign token mid-computation -- a cofactor sign, the minus of a recursive integration-by-parts step -- then executes the rest of the arithmetic correctly, and the answer comes out wrong. We ask where inside the network that sign is written. Across six open models spanning 14B to 675B parameters, a per-layer readout shows the wrong sign is committed late -- by MLP layers in the last quarter of the network (70-100% per model and task), a pattern present, where tested (Qwen), in the base model before instruction tuning. The mechanism varies: some models carry fixed "minus-stamp" layers that push a minus regardless of the right answer, others carry amplifiers of whichever sign was already chosen, and Qwen adds a downstream rescue layer that opposes the stamp. The circuit is causal: subtracting a single sign direction from the residual stream flips 13 of 14 held-out minus-written integration-by-parts errors on Llama-3.3-70B, with flips replicating on Qwen, Gemma, and Mistral-Small (24-70% per cell), and with representation-level specificity -- the same dose breaks correctly-minus answers while sparing correct-plus answers where the direction is surgical (0/72 on Llama), and matched random-direction and control-layer interventions are null. Core claims were preregistered with pass criteria frozen in git before held-out data was seen: 23 of 29 checks passed, all misses reported. But causality is not uniform: Phi-4 implements an equally well-localized circuit at the same late depth, yet the identical intervention flips almost nothing -- the sign is written redundantly across ~15 layers with margins several times larger (median 17.8 logits). Localization, we conclude, does not imply steerability: two models can share a circuit location yet differ enormously in how overturnable it is.
| Comments: | 61 pages. Code, per-case results and preregistration: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.07478 [cs.LG] |
| (or arXiv:2611.07478v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07478 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shradha Agarwal [view email]
[v1]
Thu, 5 Nov 2026 05:36:28 UTC (2,990 KB)
来源：arXiv cs.AI · arxiv.org