arXiv cs.AI· Jacob Solinsky (University of Minnesota), Serguei Pakhomov (University of Minnesota)·· 4 小时前AI 评分40
用稀疏自编码器研究大语言模型中语法数的编码与传播机制
Encoding and Propagation of Grammatical Number in Large Language Models
AI 导读
研究者用稀疏自编码器（SAE）在 Pythia-70M 上深入分析单复数一致这一语法现象在大语言模型中的表示方式，定位出“复数名词”“复数主语”“生成复数动词”等特征，移除这些特征会使模型改输出单数动词，反之亦然。而对应单数特征干预效果较弱，提示模型采用亏损性（privative）的语法数表示。该方法在 Gemma-2-2b 和 Llama-3.1-8b 上得到类似结果。
正文 · 原文
Abstract:Sparse autoencoders (SAEs) have become an increasingly popular means of translating Large Language Model (LLM) hidden state vectors into a human-comprehensible list of features or symbols. As such, SAEs may provide insight into an LLM's knowledge representation and internal ``mental'' grammar - topics of pragmatic interest for mechanistic interpretation of AI behavior and of theoretical interest for the study of in silico representations of human language. Language models with fewer than 100M parameters, trained on small corpora, nonetheless exhibit strong grammatical competence and discourse coherence, despite producing content of limited reliability. We use SAEs that we trained on one such model (Pythia-70M) to conduct an in-depth examination of how a single grammatical phenomenon, number agreement, is represented and processed in an LLM. We demonstrate the mechanism and the methodology by which SAEs can be used to find and propagate ``plural noun'', ``plural matrix subject'', and ``generate a plural verb'' features which cause the LLM to produce plural verb forms such as ``are'', ``do'', or ``have'' instead of ``is'', ``does'', and ``has'' in response to prompts such as ``the men in the park''. We show that intervening to remove these features from the hidden states of plural subject noun tokens causes the model to produce singular verbs instead, and vice versa. We also show that corresponding ``singular noun'' and ``generate a singular verb'' features are not as effective for this task. This asymmetry suggests that Pythia-70m makes use of a privative representation of grammatical number in English. These findings replicated with larger LLMs Gemma-2-2b and Llama-3.1-8b yielded similar results. Our main contribution is a novel approach to the use of SAEs to understanding how LLMs encode human language that addresses a number of technical and methodological challenges.
| Comments: | 50 pages, 19 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07283 [cs.CL] |
| (or arXiv:2611.07283v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07283 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jacob Solinsky [view email]
[v1]
Thu, 5 Nov 2026 01:04:02 UTC (2,596 KB)
来源：arXiv cs.AI · arxiv.org