跳到正文
arXiv cs.AI· Aritra Das, Dhruman Gupta, Debayan Gupta·· 2 小时前AI 评分36

线性化 2-Simplicial Attention 论文发布于 arXiv

Linearized 2-Simplicial Attention

AI 导读

arXiv 论文提出线性化 2-simplicial attention,将三线性分数改写为复合 query 与 key 的内积,用正随机特征近似并以固定大小状态存储历史 key,把计算成本从序列长度的立方降到平方。

正文 · 原文

View PDF HTML (experimental)

Abstract:We present a linearized form of 2-simplicial attention by rewriting the trilinear score as an inner product between a composite query and a key, so that the sum over keys takes the same form as ordinary softmax attention. We then approximate this sum with positive random features and store all previous keys in a fixed-size state, while the sum over second keys stays exact over all previous tokens. This reduces the cost from cubic to quadratic in sequence length, and both keys range over all previous tokens, which windowed 2-simplicial attention does not allow. We implement it with custom Triton kernels and combine it with Kimi Delta Attention to build a model with no softmax attention at all. With 15B equal training tokens on FineWeb-Edu, this model lowers validation loss from 2.35 to 2.32 over a KDA hybrid and lowers LAMBADA perplexity from 40 to 38.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2608.09307 [cs.AI]
  (or arXiv:2608.09307v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2608.09307

arXiv-issued DOI via DataCite

Submission history

From: Aritra Das [view email]
[v1] Mon, 10 Aug 2026 08:53:30 UTC (20 KB)
[v2] Wed, 18 Nov 2026 16:36:07 UTC (16 KB)

来源:arXiv cs.AI · arxiv.org