arXiv 论文 Attention at Absolute Zero 提出零温度高阶注意力层,在规模外推和组合任务上接近精确
Attention at Absolute Zero
AI 导读
论文提出一种状态建立在条目对上的高阶注意力层,以零温度读取证人得分,只传递比较计数的比特,仅从答案训练,针对软注意力在规模化时的权重分散、得分间隙未定和状态漂移三种失败方式。
推荐理由
论文给出零温度高阶注意力层在规模外推和组合任务上的具体数据,读者可比较软注意力失效的原因和可迁移设计。
正文 · 原文
Abstract:Attention scores candidate supports of an answer, which we call witnesses, and reads the scores at a temperature. We build a higher-order attention layer whose state lives on pairs of items: for a pair (a, b), each third item k is a witness judged through its legs (a, k) and (k, b). It reads witnesses at zero temperature, keeping the best and the number tied with it, passes on only bits comparing such counts, rebuilds its pair state from the input and these bits at every loop, and is trained from answers alone. Each choice removes one way soft attention fails at size: dispersed weight, unresolved gaps between learned scores, and a drifting soft state. In maximum retrieval it is right on at least 99.95% of sets 1,000 times larger than in training, on all ten seeds; trained with a soft count, the same network falls to 42% at 4,096 items. On CLRS-30 without hints, a general form of the layer (half its channels read at zero temperature, numbers embedded, state carried) beats the best published model, which uses hints on some tasks; the full cold layer is exact on every test input of 9 of the 12 tasks whose answers depend on numbers only through comparisons. From yes-or-no answers it decides 2-SAT and two-colouring exactly at 500 variables and, as our theory predicts, not parity equations. Beside a frozen 7B language model it chains family relations exactly at 1,000 statements after training on 16, where the model alone falls to 44%; finite checks on the weights prove every block that learned correct at every size, on acyclic contexts and given what the model reads. On CLUTRR it is right on every story of five to ten relations given the graphs, learning the 62 rules exactly; from text, with the model frozen, no hand-written rule and only a small read-in trained, it lifts an open 7B model from 33% to 65%, above every system we found without hand-written rules except frontier reasoning models.
| Comments: | 51 pages, 9 figures, 1 Extended Data figure, 4 supplementary tables. Replaces v1, "Attention Is Only as Good as Its Witnesses". Code (Apache 2.0): this https URL ; demo: this https URL ; models: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE) |
| Cite as: | arXiv:2611.11937 [cs.LG] |
| (or arXiv:2611.11937v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.11937 arXiv-issued DOI via DataCite |
Submission history
From: Mostafa 'Neo' Mohsenvand [view email]
[v1]
Mon, 9 Nov 2026 06:08:10 UTC (1,792 KB)
[v2]
Tue, 17 Nov 2026 14:14:46 UTC (1,719 KB)
来源:arXiv cs.AI · arxiv.org