跳到正文
arXiv cs.AI· Ales Kubicek, Andrea Caciolai, Liangzu Peng, Sukrit Sumant, Benjamin Hoover, Fraser Greenlee, Ruairidh M. Battleday, Radim Urban, Torsten Hoefler, James C. R. Whittington·· 15 天前AI 评分43

潜在稀疏注意力(LSA):迈向亚二次复杂度的超长上下文注意力

Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

AI 导读

arXiv 论文提出 Latent Sparse Attention(LSA),一种在超长上下文(100 万以上 token)下保持精度的稀疏注意力方法:key 在学习的潜在空间中分层聚类,query 只对簇层级打分。

正文 · 原文

View PDF HTML (experimental)

Abstract:Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training faster at long context. We propose two variants, Distilled-LSA (D-LSA), which can be applied to pre-trained models via distillation, and Native-LSA (N-LSA), which is trained natively on the next-token loss alone. On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks. N-LSA matches the accuracy of full attention with a training speedup of 1.6x per layer at 128k tokens. LSA offers practical and efficient attention for models to benefit from ultra-long context.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2610.42147 [cs.AI]
  (or arXiv:2610.42147v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2610.42147

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Andrea Caciolai [view email]
[v1] Fri, 30 Oct 2026 17:59:34 UTC (5,380 KB)

来源:arXiv cs.AI · arxiv.org