跳到正文
arXiv cs.AI· Sihao Ding, Santosh Vasa, Aditi Ramadwar, Thomas Monninger·· 2 小时前AI 评分38

EDCT:通过反事实测试揭示 VLM 解释忠实性缺口

Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing

AI 导读

研究者提出解释驱动反事实测试(EDCT),通过提取模型解释所引用的视觉概念并施加经验证的最小编辑,检验 VLM 的回答与解释是否与编辑后的图像保持一致。基于该协议构建的 EDCT-Bench 涵盖 OK-VQA 知识型问答、DriveLM 安全驾驶和 3DSRBench 三维空间推理三个领域。评测显示各 VLM 普遍存在明显忠实性缺口,微调实验表明 EDCT 生成的反事实样本可提供高效训练信号。

正文 · 原文

View PDF HTML (experimental)

Abstract:Vision-Language Models (VLMs) can produce Natural Language Explanations (NLEs) that sound plausible yet remain inconsistent with the visual evidence they cite. We present Explanation-Driven Counterfactual Testing (EDCT), an intervention-based protocol that extracts visual concepts cited in a model's explanation, applies verified minimal edits to them, and tests whether the resulting answer and explanation remain consistent with the edited image. Using this protocol, we create EDCT-Bench, a comprehensive benchmark spanning three complementary domains: knowledge-intensive visual question answering (OK-VQA), safety-critical driving (DriveLM), and 3D spatial reasoning (3DSRBench). Across the evaluated VLMs, EDCT reveals substantial faithfulness gaps, with models frequently producing responses inconsistent with verified visual changes. Finally, our fine-tuning study suggests that EDCT-generated counterfactuals provide high-impact training signals.
Comments: Accepted to WACV 2027. 16 pages, including appendix
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as: arXiv:2609.17953 [cs.CV]
  (or arXiv:2609.17953v2 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2609.17953

arXiv-issued DOI via DataCite

Submission history

From: Santosh Vasa [view email]
[v1] Wed, 16 Sep 2026 00:25:31 UTC (4,157 KB)
[v2] Tue, 17 Nov 2026 16:34:12 UTC (4,141 KB)

来源:arXiv cs.AI · arxiv.org