Transformers 隐式推理研究：弥合 GPT-2 类模型的组合性差距
Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models
AI 导读
一项 arXiv 研究在两跳推理设置中从零训练 GPT-2 类 Transformer 模型，探究组合泛化在分布外（OOD）知识上失效的组合性差距。研究者在不改动模型架构、不添加 OOD 组合监督的情况下，把 ID 与 OOD 原子事实按与第二跳检索步骤位置对齐的格式呈现。结果显示模型成功将已学的组合行为迁移到留出的 OOD 事实，证明该差距并非架构的固有限制，可通过机制指导的训练分布设计克服。
正文 · 原文
Abstract:Compositional generalization remains a persistent limitation of transformer language models: even when a model learns to combine known facts correctly in-distribution (ID), the same compositional mechanism may fail when applied to out-of-distribution (OOD) knowledge. We investigate this compositionality gap in a controlled two-hop reasoning setting using a GPT-2-like transformer model trained from scratch. Building on prior work identifying a circuit associated with compositional generalization, we test whether the failure of OOD transfer is related to how atomic knowledge is structurally exposed during training. We modify the training distribution by presenting both ID and OOD atomic facts in a format positionally aligned with the second-hop retrieval step of the learned compositional circuit, without modifying the model architecture or adding OOD compositional supervision. Under this intervention, the model successfully transfers the learned ID compositional behaviour to held-out OOD atomic facts, closing the observed compositionality gap in the controlled setting. Our results provide evidence that the observed compositionality gap is not a fundamental architecture limitation and that it can be overcome through mechanistically informed training distribution design.
| Comments: | Code available at: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39016 [cs.CL] |
| (or arXiv:2610.39016v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39016 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Davide Venuto [view email]
[v1]
Wed, 28 Oct 2026 22:44:54 UTC (414 KB)
来源：arXiv cs.AI · arxiv.org