跳到正文
arXiv cs.AI· Yoshiyuki Ootani·· 4 小时前AI 评分39

研究揭示:输入通路影响小型 Transformer 的少样本而非零样本绑定能力

Input Pathways Shape Few-Shot, Not Zero-Shot, Binding in Tiny Transformers: A Fully-Enumerable Study

AI 导读

一项全可枚举研究考察约 6-10K 参数的 Transformer 在有限分解世界中的组合绑定能力,发现任何信息通路都无法在零样本下达到收敛的组合绑定,其失败源于查找充分目标下的归纳偏置而非信息缺失。

正文 · 原文

View PDF HTML (experimental)

Abstract:How does the way information reaches a transformer -- as symbolic tokens, a clean per-factor "oracle" code, or an entangled perceptual vector -- shape whether it binds that information compositionally? We study ~6-10K-parameter transformers on finite factored worlds enumerated exhaustively, so every measurement covers the whole input space and the informative routes are information-matched (exact Bayes ceiling 1.0). Four findings. (1) Endpoint invariance: on held-out binding queries no informative route reaches converged zero-shot composition -- each ends at or below chance, so the failure reflects inductive bias under a lookup-sufficient objective, not missing information. It also holds with the state in its own sequence position rather than added to every token. (2) Readability governs few-shot binding: when a small fraction of the held-out type is leaked into training, sample efficiency tracks code readability. The least readable entangled code is worst (-0.147 pooled, under both schemes); a dimension-matched control and a graded readability sweep isolate readability from input dimension. (3) The injection site, not the pathway: v1 read the remaining ordering among readable routes as pathway parameter sharing; that is withdrawn. The compared pathways are block-decomposable, hence one function class, so sharing was never manipulated; the ordering comes from the initialization scale of the broadcast-added state and vanishes when it gets its own position (+0.057 -> -0.006, 20 paired seeds/cell). (4) Failure anatomy: symbolic routes lose the answer at the readout; index routes mis-bind -- the answer stays decodable, yet an input intervention shows the output tracks the wrong slot. The central positive claim is readability; the others bound what a pathway comparison can establish. Code, manifests, and per-seed logs are released.
Comments: v2 withdraws the first few-shot factor: the compared pathways are one function class, so sharing was never manipulated, and the residual effect is the injected state's initialization scale, which vanishes under prefix injection (new appendix). Also withdraws the above-chance transient and the double dissociation (a mean of per-seed maxima). Readability and endpoint invariance stand. 18 pages
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2607.04926 [cs.LG]
  (or arXiv:2607.04926v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2607.04926

arXiv-issued DOI via DataCite

Submission history

From: Yoshiyuki Ootani [view email]
[v1] Mon, 6 Jul 2026 10:55:56 UTC (422 KB)
[v2] Fri, 13 Nov 2026 03:41:14 UTC (426 KB)

来源:arXiv cs.AI · arxiv.org