GRIT：用梯度掩码纠正视觉语言模型中视觉误定位 token 的富文本推理训练框架
GRIT: Gradient Masking for Visually Misgrounded Tokens in Text-Rich Reasoning
AI 导读
GRIT 是一个将视觉定位纳入 RLVR 的训练框架，针对视觉语言模型在富文本推理中“答案正确但中间步骤视觉误读”（CVMR）的问题。作者通过 token 级分析发现误定位 token 呈现与正确定位不同的注意力模式，并用梯度掩码抑制其影响，同时推出支持按文本长度和推理深度系统评测的 GRIT-Bench。实验显示 GRIT 在推理准确率和定位保真度上均显著优于现有 RLVR 方法。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) has recently been extended to vision-language models (VLMs), yet its effectiveness remains limited for text-rich reasoning tasks. We highlight a key factor contributing to this limitation: models frequently arrive at correct final answers despite misperceiving visual information during intermediate steps. We define this phenomenon as Correct-but-Visually-Misgrounded Reasoning (CVMR). Because standard RLVR rewards only final-answer correctness, it fails to differentiate between properly grounded and misgrounded reasoning, reinforcing both equally. Through token-level analysis, we reveal that CVMR tokens exhibit distinct attention patterns that differ systematically from well-grounded ones. Based on this observation, we propose GRIT, a training framework that incorporates visual grounding into RLVR to mitigate the impact of CVMR. Furthermore, to address the lack of text-rich reasoning benchmarks that allow for systematic evaluation across text length and reasoning depth, we introduce GRIT-Bench. Experiments demonstrate that GRIT significantly improves both reasoning accuracy and grounding fidelity over existing RLVR methods.
| Comments: | Accepted to WACV 2027 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.41629 [cs.CV] |
| (or arXiv:2610.41629v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41629 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chanseul Cho [view email]
[v1]
Fri, 30 Oct 2026 12:05:54 UTC (4,250 KB)
来源：arXiv cs.AI · arxiv.org