一切都在它的位置上：注意力机制是一种"无线电头"
Everything in its right place: attention is a radio head
AI 导读
一篇 arXiv 论文提出，注意力机制可以从接收器与上下文 token 的同步中涌现：每个过去 token 是一个振荡器，其频率随年龄漂移，当前 token 按内容和位置选择它们。
正文 · 原文
Abstract:Attention, the core of language models, is a digital calculation over tokens: an exponential selects them, a division normalizes and a positional code sets the order. We show that attention can instead emerge from the synchronization of a receiver with its context tokens. Each past token is an oscillator whose state carries its content and whose frequency drifts with its age. The present token selects them by content, aligning an antenna with its query, and by position, setting the frequency of a receiver with its dial. Solving Kuramoto's problem for this receiver yields an attention layer in closed form with two phases: above a critical coupling set by the context, the receiver locks and its weights normalize; below it, it stays silent and its weights sum to less than one. Trained networks use both phases to select, abstain and count, fail where the theory predicts and keep reading far beyond the training length. Each part of attention finds its place in one device, a radio head: it tunes in to what to hear and when, and stays silent when nothing relevant is on air.
| Comments: | 6 pages, 3 figures. Supplemental Material included (10 pages). Code: this https URL |
| Subjects: | Physics and Society (physics.soc-ph); Disordered Systems and Neural Networks (cond-mat.dis-nn); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Adaptation and Self-Organizing Systems (nlin.AO) |
| Cite as: | arXiv:2611.13146 [physics.soc-ph] |
| (or arXiv:2611.13146v1 [physics.soc-ph] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13146 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lluís Arola-Fernández [view email]
[v1]
Mon, 9 Nov 2026 19:26:40 UTC (249 KB)
来源：arXiv cs.AI · arxiv.org