SAE 训练数据的 Scaling Laws 研究
Scaling Laws for SAE Training Data
AI 导读
一项 arXiv 研究提出了稀疏自编码器（SAE）训练数据量的 scaling law：在固定训练预算下，质量随激活 buffer 增大快速提升后达到平台期，此后回放 buffer 与使用新鲜激活效果接近。
正文 · 原文
Abstract:Activation storage is often a bottleneck in sparse autoencoder (SAE) training. We study how many unique activations an SAE needs when its total training budget is fixed. Quality improves rapidly as the buffer grows, then reaches a plateau where replaying the buffer is nearly as effective as using fresh activations throughout training. Beyond this point, additional unique activations bring little benefit to reconstruction or interpretability, so the buffer need not grow with the training budget. We describe this tradeoff between diversity and repetition with a scaling law for limited data. We validate it across dictionary sizes and token budgets, early, middle, and late layers of Llama-3.1, Qwen3 model sizes, and BatchTopK, TopK, and JumpReLU SAEs. Sweeps using mathematical text and Python code test the effect of changing the activation source. Evaluation covers explained variance (EV), reconstruction MSE, feature absorption, automated interpretability, and sparse probing. These results give a practical training rule: choose the smallest buffer that reaches the quality plateau and replay it for the remaining budget. In our experiments, this reduces activation storage by $8\text{--}64\times$ with negligible quality loss.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.07756 [cs.LG] |
| (or arXiv:2611.07756v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07756 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nikita Balagansky [view email]
[v1]
Thu, 5 Nov 2026 10:00:55 UTC (89 KB)
来源：arXiv cs.AI · arxiv.org