通过反事实语义显著性揭示 VLM 与人类场景描述的差距
Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency
AI 导读
研究者提出反事实语义显著性（CSS），一种黑盒、模型无关的框架，通过因果消融物体引起的语义偏移来量化其重要性，并以此评估 VLM 与人类场景理解的语义对齐。实验基于 307 个复杂自然场景、1,306 个高保真反事实变体和 16,603 条有效人类反应。结果显示模型过度依赖物体大小、中心性和低层显著性，而对场景中人物的敏感度低于人类，物体大小依赖是模型-人类语义差异的主要驱动因素。
正文 · 原文
Abstract:Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Traditional white-box interpretability methods are inapplicable to closed-source architectures and passive metrics fail to isolate causal features. We introduce Counterfactual Semantic Saliency (CSS). This black-box, model-agnostic framework quantifies the importance of objects by measuring the semantic shift induced by their causal ablation from a scene. To evaluate AI-human semantic alignment, we tested prominent VLMs against a human psychophysics baseline comprising 16,603 valid responses across 307 complex natural scenes and 1,306 high-fidelity counterfactual variants. Our analysis reveals a pervasive scene comprehension gap: models exhibit an over-reliance (relative to humans) on object size, object centrality, and low-level saliency of objects. In contrast, models are less sensitive to the presence of people in the scenes than our human participants when describing the images. The over-reliance on object size is a primary driver explaining variations in model-human semantic divergence. Code and data are available at this https URL .
| Comments: | NeurIPS 2026 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.13047 [cs.CV] |
| (or arXiv:2605.13047v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.13047 arXiv-issued DOI via DataCite |
Submission history
From: Ziqi Wen [view email]
[v1]
Wed, 13 May 2026 06:11:47 UTC (5,479 KB)
[v2]
Sat, 10 Oct 2026 19:45:17 UTC (6,089 KB)
[v3]
Wed, 28 Oct 2026 22:19:31 UTC (6,081 KB)
来源：arXiv cs.AI · arxiv.org