arXiv cs.AI· Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song, Jiarui Liu, Thai Quoc Hoang, Fang Wu, Chris Ngo, Terry Jingchen Zhang, Mona T. Diab, Ran Xu, Silvio Savarese, Xiao-Ming Wu·· 4 小时前AI 评分57
BedBench 论文揭示视觉语言模型在 3D 推理中的失败模式
Where Do Vision Language Models Fail at 3D Reasoning?
AI 导读
研究团队发布 BedBench 基准，用 13 个程序生成任务检验视觉语言模型的 3D 推理。逐色计数上人类准确率 93.3%，最好的模型仅 7.2%；微调可将一个模型从 3.0% 提升到 68.6%，但其答案仍与自身计数不一致。
正文 · 原文
Authors:Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song, Jiarui Liu, Thai Quoc Hoang, Fang Wu, Chris Ngo, Terry Jingchen Zhang, Mona T. Diab, Ran Xu, Silvio Savarese, Xiao-Ming Wu
Abstract:Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twelve open-weight and four frontier models, per-color counting exposes a pronounced weakness: humans achieve 93.3% accuracy, compared with 7.2% for the best model. Targeted fine-tuning raises one model's accuracy from 3.0% to 68.6%, but reveals a further difficulty: its answers about the most common colors remain inconsistent with its own per-color counts. Supplying those counts as text improves its judgments, showing that recovering a property does not mean using it consistently. Reasoning, scaling, and added visual cues each improve individual judgments, but none makes exact answers reliable. To locate the difficulty, we probe Qwen3.5-27B: exact per-color counts remain recoverable at the language model's input, yet the model's answers rarely recover them (65.0% for the probe vs. 3.0% for the model). Image-removal controls further show that some apparent successes require no visual evidence. These findings motivate evaluating 3D reasoning by exact correctness, consistent use, and image dependence, rather than accuracy alone. Code and dataset are publicly available: this https URL
| Comments: | Preprint, 150 pages |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39639 [cs.CV] |
| (or arXiv:2610.39639v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39639 arXiv-issued DOI via DataCite |
Submission history
From: Khai Le-Duc [view email]
[v1]
Thu, 29 Oct 2026 09:00:14 UTC (9,508 KB)
来源：arXiv cs.AI · arxiv.org