Continuum Attention：将因果注意力扩展为连续记忆轨迹的积分检索
Continuum Attention
AI 导读
一篇 arXiv 论文提出 Continuum Attention（CA），让相邻 key 和 value 向量构成连续分段线性记忆轨迹，query 通过对可见前缀的归一化积分检索信息，积分段在指数矩下有闭式解，无需数值积分。
正文 · 原文
Abstract:Conventional causal attention aggregates values at discrete memory positions. We introduce Continuum Attention (CA), in which neighboring key and value vectors define a continuous, piecewise linear memory trajectory and each query retrieves information by normalized integration over its visible prefix. The segment integrals have closed forms in exponential moments, yielding analytically differentiable attention without numerical quadrature. The resulting operator preserves causal masking, normalized nonnegative effective weights, and the standard Transformer decoder structure. Controlled byte level experiments favor exact continuous integration over matched softmax attention. We additionally evaluate data efficiency, out of domain language modeling, zero-shot multiple choice prediction, robustness, and multilingual generalization. In multilingual evaluation, CA consistently outperforms all baselines across every evaluated language, achieving an average 5.8% reduction in bits per byte (BPB) relative to the strongest baseline.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20586 [cs.LG] |
| (or arXiv:2610.20586v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20586 arXiv-issued DOI via DataCite |
Submission history
From: Afshin Khadangi [view email]
[v1]
Wed, 14 Oct 2026 17:57:40 UTC (694 KB)
[v2]
Mon, 26 Oct 2026 15:22:53 UTC (1,731 KB)
来源：arXiv cs.AI · arxiv.org