Context-Answer 元模型：在首个回答 Token 生成前预测 LLM 激活
Context--Answer Metamodels: Forecasting LLM Activations Before the First Answer Token
AI 导读
研究者训练 context-answer 元模型，将模型读取上下文时的激活序列映射为答案平均池化激活的点或分布预测，从而在首个回答 token 生成前预测 LLM 的 rollout 激活。
正文 · 原文
Abstract:Under stochastic decoding, a single context can elicit many different answers from a language model. Characterizing this variation typically requires many expensive autoregressive rollouts, each revealing only one realization. We ask whether the activations produced while reading the context can directly forecast the model's rollout activations. We train context--answer metamodels that map a context activation sequence to point or distribution forecasts of mean-pooled answer activations. We find that (i) the four metamodel families we study forecast mean-pooled answer activations with high fidelity, and the distribution metamodels also capture rollout variability; (ii) performance improves as the training set grows to 500,000 prompts, and the metamodels generalize zero-shot to out-of-domain datasets; (iii) their forecasts retain behavior-predictive signal; and (iv) context activations from Qwen2.5 predict mean-pooled answer activations from Qwen3.5 and Gemma 4, suggesting that activation forecasts can transfer across models. Together, these results suggest that context activation traces can support scalable forecasts of behaviors in LLM answers before the first answer token is generated.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06837 [cs.LG] |
| (or arXiv:2611.06837v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06837 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tommy He [view email]
[v1]
Wed, 4 Nov 2026 17:53:52 UTC (1,465 KB)
来源：arXiv cs.AI · arxiv.org