状态-预测分离假说：用双计算流改进 Transformer 语言建模
The State-Prediction Separation Hypothesis
AI 导读
arXiv 论文（发表于 NeurIPS 2026）提出状态-预测分离假说，认为把“预测下一个 token”与“存储有用状态”两种角色解耦到不同计算流可提升语言建模效果。多尺度预训练实验显示，该方法在验证损失上持续改进，下游任务平均超越标准 Transformer 2-3 个百分点，并具备更好的数据与计算效率。作者还通过大量实证分析排除了潜在混淆因素。
正文 · 原文
Abstract:Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the state-prediction separation hypothesis: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses separate computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data- and compute-efficiency, improving validation loss and outperforming standard Transformers by 2-3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails.
| Comments: | Published at NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.01218 [cs.CL] |
| (or arXiv:2607.01218v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01218 arXiv-issued DOI via DataCite |
Submission history
From: Giovanni Monea [view email]
[v1]
Wed, 1 Jul 2026 17:55:09 UTC (1,510 KB)
[v2]
Thu, 5 Nov 2026 16:43:45 UTC (2,416 KB)
来源：arXiv cs.AI · arxiv.org