arXiv cs.AI· Guijia Zhang, Hongbo Qiu, Jiajun Zha, Guohao Cai, Hao Zheng, Huihao Jing, Kexu Cheng, Shipeng Luo, JinYang Huang, Shing-Chi Cheung, Changsheng Lu, Haoxuan Che, Harry Yang·· 1 小时前AI 评分38
当图表与正文不一致:科学论文中自主视觉证据验证的评估
When Figures Disagree with Text: Evaluating Autonomous Visual Evidence Verification in Scientific Papers
AI 导读
一项研究在 31 篇完整论文手稿上评估多模态模型能否自主核查图表与正文的矛盾,且不告知模型哪张图有问题。1,445 份开放审阅中 60.6% 提及有缺陷的图表,但仅 3.3% 被评分标准判定为诊断出矛盾;指名图表可将该比率从 4.3% 提升至 20.4%,按中心度选择图表的基线仅为 4.8%。研究表明决定验证什么本身是主要难点,多模态文档理解评测应考察模型能否主动提出核查对象。
正文 · 原文
Authors:Guijia Zhang, Hongbo Qiu, Jiajun Zha, Guohao Cai, Hao Zheng, Huihao Jing, Kexu Cheng, Shipeng Luo, JinYang Huang, Shing-Chi Cheung, Changsheng Lu, Haoxuan Che, Harry Yang
Abstract:Scientific figures carry evidence for the claims that cite them, so reviewing a paper means checking that a figure agrees with the text, and not only describing what the figure shows. Benchmarks of figure-text verification typically hand the model the figure, the claim, or the knowledge that an inconsistency exists, which leaves untested whether a model can find the figure worth checking in a complete paper, relate a visual element to the claim it conflicts with, and recognize the conflict. We study this autonomous setting on 31 full manuscripts, each carrying a reviewer-documented figure-text inconsistency from PRISMM-Bench that is hidden from the model. Six multimodal models review each PDF under a neutral prompt, and we then add one input at a time to the same manuscripts. Of 1,445 open-ended reviews, 60.6% mention the flawed figure, yet a grading rubric accepts only 3.3% as diagnosing its contradiction; a blind audit finds the rubric lenient, so we read these as rubric-accepted rates and compare conditions rather than absolute values. In a separate controlled run, naming the figure, without stating that it is flawed, raises rubric-accepted diagnosis from 4.3% to 20.4%, while a consistency instruction and coarse visual cues do not. When the contradiction is visible only in the rendered figure, text-only review yields no accepted diagnosis, while review of the rendered pages with the figure named reaches 15.8%. A baseline that lets the model choose which figures to check, ranked by centrality, stays at 4.8%. Deciding what to verify is thus a large part of the difficulty, and the selection rule we tested does not resolve it. Multimodal document understanding should be evaluated on whether a model can propose what to verify, not only answer questions about a figure it is handed.
| Comments: | 28 pages, 5 figures, 14 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24353 [cs.AI] |
| (or arXiv:2611.24353v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24353 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongbo Qiu [view email]
[v1]
Wed, 18 Nov 2026 12:47:35 UTC (1,468 KB)
来源:arXiv cs.AI · arxiv.org