arXiv cs.AI· Qiuyi (Richard), Zhang, Zhen Xu, Jing Gu, Seojin Bang, Arnav Mohanty Das, Yaning Hu, Gautam Kumar, Jared Lichtarge, Le Xue, Bettina Catoto, Akul Datta, Daniel Luo, Farha Khan, Marcella Valentine, Yinfei Yang, Andrew M. Dai·· 2 小时前AI 评分39
DecoyCount：评估视觉推理在计数中的错位问题
DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting
AI 导读
arXiv 新论文 DecoyCount 用 1,660 张含七种诱饵机制的真实图像及配对提示词，评估视觉语言模型计数错误究竟源于感知还是推理。前沿模型准确率均低于 50%，推理对齐差距为 11.8-22.1 个百分点，且更强模型、更大思考预算和智能体流程往往导致更大差距；增加多跳场景文本信息也未提升整体表现。
正文 · 原文
Authors:Qiuyi (Richard)Zhang, Zhen Xu, Jing Gu, Seojin Bang, Arnav Mohanty Das, Yaning Hu, Gautam Kumar, Jared Lichtarge, Le Xue, Bettina Catoto, Akul Datta, Daniel Luo, Farha Khan, Marcella Valentine, Yinfei Yang, Andrew M. Dai
Abstract:Visual counting benchmarks report whether vision-language models produce the correct number, but not whether errors arise from perception or reasoning. We introduce DecoyCount, a benchmark built from 1,660 real-world images with designed decoys across seven mechanisms, along with paired prompts: a neutral question, a matched alignment cue, and a novel multi-hop, multi-image question that situates the same counting problem in a realistic scenario. Using this design, we find that counting failures are induced partly by visual misalignment: models can execute the intended counting process when cue-guided but fail to select it unaided, and we term this difference in accuracy the reasoning-alignment gap. Despite being highly capable on general reasoning, frontier models remain below 50\% accuracy on our benchmark, while alignment gaps range from 11.8-22.1 percentage points; strikingly, we observe that more capable models, greater thinking budgets, and agentic processes often induce larger gaps. Because humans may bring context that models lack, we test whether textual underspecification contributes to misalignment using a scenario-based multi-hop condition without explicit hinting, but this added context does not improve aggregate performance. Our analysis concludes that stronger general reasoning does not by itself generalize to aligned visual reasoning, leaving misalignment with human visual expectations as a substantial component of the remaining deficit.
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.34855 [cs.AI] |
| (or arXiv:2610.34855v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34855 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qiuyi (Richard) Zhang [view email]
[v1]
Mon, 26 Oct 2026 05:19:02 UTC (2,181 KB)
来源：arXiv cs.AI · arxiv.org