arXiv cs.AI· Yue Wu, Yang Zhang, Weinan Gan, Chongming Gao, Xinsen Guo, Xiaoyuan Li, Fuli Feng·· 4 小时前AI 评分40
Select-TTT：基于查询条件的快速权重选择性更新改进 In-Place 测试时训练
Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training
AI 导读
研究者提出 Select-TTT，一种模型无关、查询条件化的规则，解决 In-Place TTT 在真实网页长上下文问答中收益消失的问题。该方法按各文本块与用户查询的相关性缩放写入强度，相关性信号直接从模型冻结的输出头读取，无需监督、无新增参数。Select-TTT 在两个基准和两个骨干模型上持续提升长上下文问答表现。
正文 · 原文
Abstract:Test-time training (TTT) enables a deployed language model to internalize incoming context on-the-fly into a small set of fast-weight parameters, rather than merely treating it as transient context for attention. In-Place TTT recently made this idea practical for large language models by repurposing the down-projection of standard MLP blocks as fast weights. However, we find that its benefit largely disappears on realistic web long-context question answering. We attribute this to In-Place TTT's uniform write schedule: every chunk is written with equal strength, so irrelevant content dilutes query-relevant information in fast-weight memory. Building on this insight, we propose Select-TTT, a model-agnostic, query-conditioned rule for In-Place TTT that scales each chunk's write strength by its relevance to the user query. This relevance signal requires no supervision and no new parameters: it is read directly from the model's frozen output head, as the probability mass that a chunk's hidden states assign to the query tokens. A temperature-scaled softmax turns these scores into learning-rate multipliers, leaving the chunk-wise, context-parallel computation of In-Place TTT intact. Select-TTT consistently improves long-context question answering on two benchmarks and two backbones. These findings highlight the value of query-conditioned fast weight writes for practical test-time training on long-context question answering, which is also potentially promising for long-horizon agent tasks involving multiple user queries over evolving contexts.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.36794 [cs.LG] |
| (or arXiv:2610.36794v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36794 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yue Wu [view email]
[v1]
Tue, 27 Oct 2026 07:58:39 UTC (249 KB)
来源：arXiv cs.AI · arxiv.org