跳到正文
arXiv cs.AI· Trinh T. Nguyen, Duy-Tung Pham, Hoang-Son Do, Tan Minh Nguyen, Thieu N. Vo·· 4 小时前AI 评分36

因果注意力中 Token 动力学的双聚类机制研究

Two-Clustering Regime of Token Dynamics in Causal Attention

AI 导读

论文部分解决了 Karagodin 等(NeurIPS 2024)关于因果注意力聚类的猜想 1,并扩展到时变参数场景。研究证明:当 value 变换的最大特征值 λ_max > 0 时,token 动力学收敛为两个聚类;λ_max ≤ 0 时该行为一般不成立。作者还发现 token 轨迹虽在高维嵌入空间演化,但坍缩到二维结构上,并提出"流内不稳定性"分析框架,通过 GQA 层仿真验证。

正文 · 原文

View PDF HTML (experimental)

Abstract:We partially resolve Conjecture 1 of [Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024)] and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue $\lambda_{\max}$ of the value transformation. We prove that this behavior holds when $\lambda_{\max}>0$ and is generally false when $\lambda_{\max}\le0$. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension reduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework--instability within the flow--inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.18088 [cs.AI]
  (or arXiv:2611.18088v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.18088

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Thieu Vo [view email]
[v1] Fri, 13 Nov 2026 12:12:02 UTC (770 KB)

来源:arXiv cs.AI · arxiv.org