跳到正文
arXiv cs.AI· Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Deng Pan, Arun Ross, Dongxiao Zhu·· 37 分钟前AI 评分40

BOS Summary:首个 token 至关重要——无需训练的 BOS 上下文聚合方法

BOS Summary: The First Token Matters

AI 导读

一项 arXiv 研究提出 BOS Summary,一种免训练的推理时方法,在 prefill 阶段的单个 transformer 层仅对 BOS query 行放开因果约束,使 BOS token 聚合全部视觉与文本输入形成上下文表示。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input. We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead. Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention.
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2611.23508 [cs.AI]
  (or arXiv:2611.23508v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.23508

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Dongxiao Zhu [view email]
[v1] Tue, 17 Nov 2026 21:01:03 UTC (1,470 KB)

来源:arXiv cs.AI · arxiv.org