Control Reinforcement Learning：用 SAE 特征实现可解释的 LLM token 级引导
Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features
AI 导读
一项 NeurIPS 2026 论文提出 Control Reinforcement Learning（CRL），训练策略在每个 token 上选择 SAE 特征进行引导，生成可解释的干预日志。在 Gemma 2 2B 上，CRL 于 MMLU、BBQ、GSM8K、HarmBench 和 XSTest 取得提升，并能定位特征选择决定输出正确性的分支点，显示早期层偏句法特征、后期层偏语义特征。
正文 · 原文
Abstract:Sparse autoencoders (SAEs) decompose language model activations into interpretable features, but existing methods reveal only which features activate, not which change model outputs when amplified. We introduce Control Reinforcement Learning (CRL), which trains a policy to select SAE features for steering at each token, producing interpretable intervention logs: the learned policy identifies features that change model outputs when amplified. Adaptive Feature Masking encourages diverse feature discovery while preserving singlefeature interpretability. The framework yields new analysis capabilities: branch point tracking locates tokens where feature choice determines output correctness; critic trajectory analysis separates policy limitations from value estimation errors; layer-wise comparison reveals syntactic features in early layers and semantic features in later layers. On Gemma 2 2B across MMLU, BBQ, GSM8K, HarmBench, and XSTest, CRL achieves improvements while providing per-token intervention logs. These results establish learned feature steering as a mechanistic interpretability tool that complements static feature analysis with dynamic intervention probes
| Comments: | Accepted at NeurIPS 2026. 31 pages, 24 figures, 12 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2602.10437 [cs.LG] |
| (or arXiv:2602.10437v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.10437 arXiv-issued DOI via DataCite |
Submission history
From: Seonglae Cho [view email]
[v1]
Wed, 11 Feb 2026 02:28:49 UTC (17,666 KB)
[v2]
Thu, 12 Feb 2026 02:43:40 UTC (17,666 KB)
[v3]
Sun, 3 May 2026 22:52:48 UTC (17,693 KB)
[v4]
Thu, 12 Nov 2026 06:59:02 UTC (17,671 KB)
来源：arXiv cs.AI · arxiv.org