arXiv cs.AI· Qinan Zhang, Xinyu Wang, Qihang Jin, Tianyuan Wang, Hongbing Luo, Pengfei Li, Wenjun Lv, Yu Kang, Jian Di·· 2 小时前AI 评分43
GridWM-Judge：基于状态条件不变性诊断 VLM 评审可靠性的基准
Towards Reliable VLM Judges: State-Conditional Invariance and Presentation-Aware Diagnostics
AI 导读
一项 NeurIPS 2026 评测与数据集-track 录用研究提出状态条件不变性（SCI）原则，要求 VLM 评审在语义保持的呈现变化下保持稳定，同时对改变任务结果的因果干预作出响应。
正文 · 原文
Abstract:Vision-language models (VLMs) are increasingly used as automated judges for multimodal and embodied tasks, yet aggregate accuracy or consistency alone does not reveal whether a judge changes its verdict for the right reason. We propose State-Conditional Invariance (SCI), a reliability principle requiring a judge to remain stable under semantics-preserving changes in presentation while responding to causal interventions that change the task outcome. We instantiate SCI in GridWM-Judge, a simulator-grounded diagnostic benchmark built from deterministic MiniGrid trajectories with aligned Full, evidence-removal, and counterfactual variants, plus presentation and representation probes. Across five diagnostic tasks, GridWM-Judge tests local perception, action-conditioned transitions, presentation stability, representation exchangeability, and Full-CF causal discrimination. Our diagnostic evaluations reveal heterogeneous failure modes, including sensitivity to evidence presentation and encoding, low activation on successful trajectories, and weak paired counterfactual discrimination. Perception-controlled analyses further indicate that visual grounding and objective recovery are substantial, model-dependent bottlenecks. These results motivate jointly reporting correctness, presentation stability, and causal sensitivity instead of relying on a single score.
| Comments: | Accepted to NeurIPS 2026 (Track on Evaluations and Datasets). 47 pages, 16 figures. Project page: this https URL ; Code: this https URL ; Data: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| ACM classes: | I.2.10; I.2.6 |
| Cite as: | arXiv:2611.13596 [cs.AI] |
| (or arXiv:2611.13596v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13596 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qihang Jin [view email]
[v1]
Tue, 10 Nov 2026 05:15:50 UTC (4,675 KB)
来源：arXiv cs.AI · arxiv.org