arXiv cs.AI· Zixin Su, Yangguang Lin, Zegang Peng, Yuntong Sun, Xuanyi Wang, Jiahao Wang, Qiankai Xu, Bing Yuan, Jingjia Huang, Ruyang Liu, Yanxu Hu, Yufeng Zhou, Yi Lin·· 3 小时前AI 评分43
HumanSightBench：面向 MLLM 人类视觉感知的基准发布
HumanSightBench: A Benchmark for Human Visual Perception in MLLMs
AI 导读
研究者在 arXiv 发布 HumanSightBench 基准，评测多模态大语言模型与人类视觉感知的差距，包含 2,876 道人类易做、模型难得的视觉问答题，覆盖 7 个大类和 21 个子类别。每道题需通过问题清晰度、答案唯一性、视觉依据、人类可解性和纯文本捷径过滤五道准入门槛。8 个受评 MLLM 中最佳 pass@3 得分为 65.6%，计数类没有模型超过 27.5%。
正文 · 原文
Authors:Zixin Su, Yangguang Lin, Zegang Peng, Yuntong Sun, Xuanyi Wang, Jiahao Wang, Qiankai Xu, Bing Yuan, Jingjia Huang, Ruyang Liu, Yanxu Hu, Yufeng Zhou, Yi Lin
Abstract:Multimodal large language models (MLLMs) have demonstrated strong performance on knowledge-intensive and reasoning-based vision-language benchmark datasets, but still struggle to solve the visual questions that people can solve easily. Evaluating the differences between MLLMs and human-level visual perception is challenging due to end-to-end evaluation which combines various skills, or because we are uncertain if individual items are valid. Specifically, observed errors may reflect visual perception, language priors, background knowledge, question ambiguity, and underspecified answers. Our guideline for HumanSightBench is that we construct items that are human-easy by construction and model-hard by evaluation. HumanSightBench contains 2,876 visual questions with short answers across 7 top-level perceptual categories and 21 subcategories. We only include items that pass five inclusion gates: question clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcut filtering. According to the protocol, errors are defined as incorrect answers on human-validated visual tasks with the evaluated interface. The inclusion gates are intended to reduce confounds from ambiguity and external knowledge. The best pass@3 score is 65.6% across eight evaluated MLLMs. No evaluated model exceeds 27.5% on Counting. These results identify capability gaps that are hidden by overall scores and can help future training experiments and held-out evaluation.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.12305 [cs.CV] |
| (or arXiv:2611.12305v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.12305 arXiv-issued DOI via DataCite |
Submission history
From: Zixin Su [view email]
[v1]
Mon, 9 Nov 2026 12:27:19 UTC (1,554 KB)
[v2]
Wed, 11 Nov 2026 10:19:36 UTC (1,553 KB)
来源：arXiv cs.AI · arxiv.org