视觉思维忠实性研究：多模态思维链的测量与增强（SCCM）
On the Faithfulness of Visual Thinking: Measurement and Enhancement
AI 导读
一项 NeurIPS 2026 研究发现，大型视觉语言模型（LVLM）经强化微调生成的多模态思维链（MCoT）中，视觉信息常不准确且被模型 largely 忽略：视觉信息被破坏时任务准确率几乎不变。作者提出自动 LVLM 评测指标从相关性与充分性两角度量化视觉线索忠实性，并提出免标注、即插即用的 SCCM（充分成分因果模型）学习方法，在多项细粒度感知与推理基准上显著提升视觉忠实性，代码已开源。
正文 · 原文
Abstract:Recent large vision-language models (LVLMs) can generate vision-text multimodal chain-of-thought (MCoT) traces after reinforcement fine-tuning (RFT). However, we observe that the visual information in MCoT is often inaccurate even when answers are correct, and task accuracy remains nearly unchanged despite the visual information being corrupted. This indicates a lack of faithfulness in the vision part of MCoT reasoning. We attribute this to the RL reward design in RFT, which solely incentivizes the format of interleaved vision-text cues, encouraging the model to incorporate visual information into its text reasoning steps without considering its correctness. In this paper, we first probe MCoT faithfulness by measuring how much the prediction changes when its visual and textual thoughts are intervened. Surprisingly, the model's predictions remain nearly unchanged under visual intervention but change significantly under textual intervention, indicating that the visual evidence is largely ignored. To further analyze the visual information, we introduce an automated LVLM-based evaluation metric that quantifies the faithfulness of visual cues from two perspectives: relevance and sufficiency. Our evaluation reveals that the visual information in current MCoT traces is simultaneously irrelevant and insufficient. To address this issue, we propose Sufficient-Component Cause Model (SCCM) learning. This approach encourages the MCoT to generate sufficient yet minimal visual components that are independently capable of leading to the correct answer. The proposed SCCM is annotation-free and plug-and-play, compatible with various RFT for MCoT. Empirical results demonstrate that SCCM substantially improves the visual faithfulness across a suite of fine-grained perception and reasoning benchmarks. Code is available at this https URL.
| Comments: | NeurIPS 2026 Poster |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2510.23482 [cs.CV] |
| (or arXiv:2510.23482v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2510.23482 arXiv-issued DOI via DataCite |
Submission history
From: Zujing Liu [view email]
[v1]
Mon, 27 Oct 2025 16:15:54 UTC (26,333 KB)
[v2]
Fri, 30 Oct 2026 03:20:12 UTC (21,153 KB)
来源：arXiv cs.AI · arxiv.org