Massive Activations 干扰稀疏自编码器评估，arXiv 研究提出 PC1 校正方法
Massive Activations Confound Sparse Autoencoders
AI 导读
一篇被 NeurIPS 2026 接收（poster）的 arXiv 研究指出，massive activations 会让高达 99.9% 的激活方差落在第一主成分 PC1 上，破坏稀疏自编码器（SAE）基于方差解释率和消融损伤的常见评估：平均 50 个最重要特征中有 47 个在另一次运行中无法匹配。
正文 · 原文
Abstract:Sparse autoencoders (SAEs) are commonly evaluated by variance explained and ablation damage. We show how massive activations undermine both evaluations, and propose a correction. Massive activations cause up to 99.9% of activation variance to lie along the first principal component (PC1). Because the PC1 component has no unique sparse decomposition, each SAE run rebuilds it from a different set of features. These features dominate importance rankings: an average of 47 of the 50 most important features are unmatched in a separate run, compared with about 27 of a random 50. A dictionary can also report 99.96% of variance explained while leaving 33% of the variance outside PC1 unexplained. We bypass PC1 by training the SAE on activations with PC1 removed, then adding PC1 to the reconstruction. In affected models, the top-50 feature sharing rate rises from 4-39.5% to 73-99.5%. At the same TopK sparsity, the resulting SAE degrades the model up to 44% less and reduces content reconstruction error by up to 33.9%.
| Comments: | Accepted at NeurIPS 2026 (poster). Code: this https URL. SAEs: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13548 [cs.LG] |
| (or arXiv:2611.13548v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13548 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nathan Delisle [view email]
[v1]
Tue, 10 Nov 2026 04:08:55 UTC (320 KB)
来源：arXiv cs.AI · arxiv.org