跳到正文
arXiv cs.AI· Naiyu Yin, Maosen Tang, Alex Townsend, Yue Yu·· 2 小时前AI 评分47

Rational Sparse Autoencoder(RSAE):用可训练有理函数激活改进稀疏自编码器

Rational Sparse Autoencoder

AI 导读

研究者提出 Rational Sparse Autoencoder(RSAE),用可训练的有理函数替换 SAE 编码器中固定的 ReLU、JumpReLU 或 TopK 激活,并在匹配期望稀疏度下证明了对 ReLU 的非劣性与对 JumpReLU、TopK 的可证改进。

正文 · 原文

View PDF HTML (experimental)

Abstract:Sparse autoencoders (SAEs) are standard tools for mechanistic interpretability, but current SAE families are constrained by fixed encoder nonlinearities such as ReLU, JumpReLU, and TopK. This hard-codes a particular sparsity mechanism into the model and can distort the reconstruction-versus-sparsity trade-off. We introduce the Rational Sparse Autoencoder (RSAE), which replaces the fixed encoder activation with a trainable rational function. The design is motivated by a reconstruction advantage that we prove for RSAE at matched expected sparsity, with guaranteed non-inferiority to ReLU and, under stated assumptions, provable improvements over JumpReLU and TopK. The model is then implemented through a two-stage pipeline: an initialization procedure that copies the pre-trained baseline SAE weights and calibrates the scale parameters along with the rational coefficients; followed by a fine-tuning step under a sparsity-regularized reconstruction objective that holds $\ell_0$ at the teacher's value. Consistent with this theory, on residual-stream activations of three open-weight language models and across all three baseline activation families, the RSAE improves reconstruction fidelity after the fine-tuning step at the same level of sparsity. These gains are consistent across host language models, across baseline activation families, and across the full range of baseline sparsity we tested, while the upgrade itself adds only 20 scalar parameters per autoencoder and one training run takes between about 4 and 25 minutes on a single consumer GPU.
Comments: v2: M. Tang and A. Townsend added as co-authors; new theoretical analysis (Section 3, Appendix A); revised algorithm (Section 4, Appendix B.1); enhanced experiments (Section 5, Appendix B); v1 by N. Yin and Y. Yu
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2606.14990 [cs.LG]
  (or arXiv:2606.14990v4 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2606.14990

arXiv-issued DOI via DataCite

Submission history

From: Naiyu Yin [view email]
[v1] Fri, 12 Jun 2026 22:22:40 UTC (116 KB)
[v2] Tue, 16 Jun 2026 02:02:24 UTC (116 KB)
[v3] Wed, 18 Nov 2026 01:34:25 UTC (105 KB)
[v4] Thu, 19 Nov 2026 01:50:26 UTC (105 KB)

来源:arXiv cs.AI · arxiv.org