跳到正文
arXiv cs.AI· Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Xiangyu Zhou, Dongxiao Zhu·· 1 小时前AI 评分46

视觉语言模型 Sink Token 研究:SHRINK 无训练 Sink 引导键编辑方法解析

Beyond the Sink Token: Sink-Guided Key Editing in Vision-Language Models

AI 导读

一项 arXiv 研究发现视觉语言模型中 sink 键方向可控制图像与问题 token 的注意力:增强正向 sink 对齐分量平均提升注意力,负向则降低。基于此提出免训练方法 SHRINK,在九个基准上使 LLaVA-OneVision-7B 平均准确率提升 5.1 个百分点、Qwen2.5-VL-3B 提升 2.7 个百分点。

正文 · 原文

View PDF HTML (experimental)

Abstract:In vision-language models (VLMs), a few sink tokens attract a disproportionate share of attention. While previous studies have examined why these tokens attract attention, we investigate how the sink-key direction affects attention to image and question tokens. Across four probed VLMs, boosting the magnitude of positive sink-aligned components increases attention on average, whereas boosting negative components decreases it; these responses persist through later layers. This identifies the native sink key as a reference for controlling attention beyond its own position. Motivated by this finding, we propose SHRINK, a training-free method that partially reduces sink-aligned components across visible keys during generation. On nine benchmarks under sampled reasoning, it improves mean accuracy by 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B, with smaller gains on the other two main models, Qwen2.5-VL-7B and Gemma-3-4B. Beyond SHRINK, rescaling sink-aligned key components can replace the attention operations of two existing attention-steering methods, GIFT and ACG, with comparable accuracy on hallucination benchmarks.
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23516 [cs.CV]
  (or arXiv:2611.23516v1 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2611.23516

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Dongxiao Zhu [view email]
[v1] Tue, 17 Nov 2026 21:07:08 UTC (683 KB)

来源:arXiv cs.AI · arxiv.org