EDCT:通过反事实测试揭示 VLM 解释忠实性缺口
Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing
AI 导读
研究者提出解释驱动反事实测试(EDCT),通过提取模型解释所引用的视觉概念并施加经验证的最小编辑,检验 VLM 的回答与解释是否与编辑后的图像保持一致。基于该协议构建的 EDCT-Bench 涵盖 OK-VQA 知识型问答、DriveLM 安全驾驶和 3DSRBench 三维空间推理三个领域。评测显示各 VLM 普遍存在明显忠实性缺口,微调实验表明 EDCT 生成的反事实样本可提供高效训练信号。
正文 · 原文
Abstract:Vision-Language Models (VLMs) can produce Natural Language Explanations (NLEs) that sound plausible yet remain inconsistent with the visual evidence they cite. We present Explanation-Driven Counterfactual Testing (EDCT), an intervention-based protocol that extracts visual concepts cited in a model's explanation, applies verified minimal edits to them, and tests whether the resulting answer and explanation remain consistent with the edited image. Using this protocol, we create EDCT-Bench, a comprehensive benchmark spanning three complementary domains: knowledge-intensive visual question answering (OK-VQA), safety-critical driving (DriveLM), and 3D spatial reasoning (3DSRBench). Across the evaluated VLMs, EDCT reveals substantial faithfulness gaps, with models frequently producing responses inconsistent with verified visual changes. Finally, our fine-tuning study suggests that EDCT-generated counterfactuals provide high-impact training signals.
| Comments: | Accepted to WACV 2027. 16 pages, including appendix |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.17953 [cs.CV] |
| (or arXiv:2609.17953v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.17953 arXiv-issued DOI via DataCite |
Submission history
From: Santosh Vasa [view email]
[v1]
Wed, 16 Sep 2026 00:25:31 UTC (4,157 KB)
[v2]
Tue, 17 Nov 2026 16:34:12 UTC (4,141 KB)
来源:arXiv cs.AI · arxiv.org