视觉语言模型的多证据整合缺口：事实都对，答案仍错
All the Facts, Still the Wrong Answer: Integration Gaps in Vision-Language Models
AI 导读
arXiv 论文研究视觉语言模型的多证据整合缺口，发现模型能准确恢复局部事实却仍全局决策失败。以子集选择和路径查找任务测试，Gemma-3-27B 在八项视觉实例上局部事实全部正确，但全局有效率仅 19.75%。隐藏状态探针能解码答案相关信息，激活替换可在一定深度范围内恢复失败决策，表明恢复事实不足以支撑可靠联合决策。
正文 · 原文
Abstract:Multimodal reasoning requires models to combine multiple pieces of evidence into decisions that satisfy all relevant constraints. Evaluations centered on final-answer accuracy leave unclear whether failures arise from reading individual facts or using them together. We study this question through the multi-evidence integration gap, linking local fact recovery to global decision validity on controlled subset-selection and path-finding tasks. Our evaluation combines conditional performance given correct local answers with same-grid local queries, text-oracle inputs, retrieval controls, and constraint ablations. We further examine hidden-state probes and activation replacement across network depth, token positions, and model families. Across the evaluated models, accurate local recovery coexists with substantial global failure. For example, Gemma-3-27B recovers all evaluated isolated item facts correctly but achieves only 19.75% global validity on eight-item visual instances. Probes decode information about solution membership from global-prompt states, while corresponding clean-state replacement recovers failed decisions over model-dependent depth ranges. These findings show that recovering facts and representing solution-related information are insufficient for reliable joint decisions. They motivate evaluations and model improvements that explicitly target how available evidence is combined, alongside advances in perception and model scale.
| Comments: | 53 pages |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33764 [cs.AI] |
| (or arXiv:2610.33764v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33764 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuyang Dai [view email]
[v1]
Sat, 24 Oct 2026 16:14:52 UTC (2,330 KB)
来源：arXiv cs.AI · arxiv.org