视觉语言模型 Sink Token 研究:SHRINK 无训练 Sink 引导键编辑方法解析
Beyond the Sink Token: Sink-Guided Key Editing in Vision-Language Models
AI 导读
一项 arXiv 研究发现视觉语言模型中 sink 键方向可控制图像与问题 token 的注意力:增强正向 sink 对齐分量平均提升注意力,负向则降低。基于此提出免训练方法 SHRINK,在九个基准上使 LLaVA-OneVision-7B 平均准确率提升 5.1 个百分点、Qwen2.5-VL-3B 提升 2.7 个百分点。
正文 · 原文
Abstract:In vision-language models (VLMs), a few sink tokens attract a disproportionate share of attention. While previous studies have examined why these tokens attract attention, we investigate how the sink-key direction affects attention to image and question tokens. Across four probed VLMs, boosting the magnitude of positive sink-aligned components increases attention on average, whereas boosting negative components decreases it; these responses persist through later layers. This identifies the native sink key as a reference for controlling attention beyond its own position. Motivated by this finding, we propose SHRINK, a training-free method that partially reduces sink-aligned components across visible keys during generation. On nine benchmarks under sampled reasoning, it improves mean accuracy by 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B, with smaller gains on the other two main models, Qwen2.5-VL-7B and Gemma-3-4B. Beyond SHRINK, rescaling sink-aligned key components can replace the attention operations of two existing attention-steering methods, GIFT and ACG, with comparable accuracy on hallucination benchmarks.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23516 [cs.CV] |
| (or arXiv:2611.23516v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23516 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongxiao Zhu [view email]
[v1]
Tue, 17 Nov 2026 21:07:08 UTC (683 KB)
来源:arXiv cs.AI · arxiv.org