arXiv cs.AI· Debopam Sanyal, Hongjie Chen, Riya Elizabeth Kuriyan, Arth Tak, Divya Kothandaraman, Akshay Mehra, Joshua Kimball, Alexey Tumanov·· 3 小时前AI 评分36
SVD-Attention：高效分析 Transformer 注意力块的方法
SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers
AI 导读
SVD-Attention 是一种事后分析方法，对每头 query-key 交互矩阵做分解，按谱幅度和激活方差为各奇异模式打分，无需微调即可精确保留有效模式。在 7 个预训练模型和 5 类任务上，它发现冗余在情感、图像分类和生成任务中位于早中层，在密集预测和抽取式问答中位于晚层；截断其标记的层最多损失 1.75 F1，而截断其他同数层最多损失 50 F1。
正文 · 原文
Abstract:Pretrained transformers spend a large share of their inference compute inside self-attention, yet the per-head query-key interaction is structurally low-rank by construction. We ask a diagnostic question: across layers and architectures, which heads actually use the full rank budget on real data, and which do not? We propose SVD-Attention, a post-hoc method that decomposes the per-head interaction matrix $M_h$ and scores every singular mode by both its spectral magnitude and its activation variance across tokens on a small calibration set. The score keeps the modes the head can amplify and that real data actually activates, and discards the rest; the resulting factorization is exact for the retained modes and needs no finetuning. Across seven pretrained models and five task types, the diagnostic finds a task-dependent pattern: redundancy sits in the early to middle layers for sentiment and image classification and for generation, and in the late layers for dense prediction and extractive question answering. It is also predictive: truncating the layers it flags costs at most $1.75$ F1, while truncating the same number of layers elsewhere in the same model costs up to $50$ F1. SVD-Attention is an analysis tool rather than a compression method: acting on it reduces inference FLOPs by $0.8$-$1.5\%$ on its own and by $8$-$11\%$ when composed with weight pruning, with $0$-$4\%$ change in task quality.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.34971 [cs.CL] |
| (or arXiv:2610.34971v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34971 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Debopam Sanyal [view email]
[v1]
Mon, 26 Oct 2026 07:40:43 UTC (105 KB)
来源：arXiv cs.AI · arxiv.org