arXiv cs.AI· Jonathan Lichtle, Aymen Meddeb, Antonin Birot, Ibrahima Niang, Lisa Gastaud, Riyad Hanafi, Sofiane Dujardin, Corentin Selomme, Guillaume Cachard, Aghiles Hamroun, Gr\'egory Kuchcinski, Su Hwan Kim, Bastien Le Guellec·· 36 分钟前AI 评分57
arXiv 论文评估视觉语言模型智能体读急诊 CT 和 MRI 的诊断准确性
Diagnostic Accuracy and Navigation Behavior of Vision-Language Model Agents Reading Emergency CT and MRI Examinations
AI 导读
一项回顾性研究让六个 VLM 在四种浏览条件下读取 100 例公开急诊 CT 和 MRI 共 4800 次,汇总准确率最高仅 42.5%,远低于放射科医生的 77.0%。模型在阅读中位数 10.8% 的切片后就形成首个假设并在 63.1% 的运行中保持不变,最终答案自信度中位数 90% 但正确率仅 38.9%。
正文 · 原文
Authors:Jonathan Lichtle, Aymen Meddeb, Antonin Birot, Ibrahima Niang, Lisa Gastaud, Riyad Hanafi, Sofiane Dujardin, Corentin Selomme, Guillaume Cachard, Aghiles Hamroun, Grégory Kuchcinski, Su Hwan Kim, Bastien Le Guellec
Abstract:Purpose. To evaluate the diagnostic accuracy and image navigation behavior of vision-language model (VLM) agents under varying exploration strategies. Materials and Methods. In this retrospective study, six VLMs read 100 emergency CT and MRI examinations published in a public archive (March 2025-August 2026) under four conditions (free navigation, navigation from a key slice, and all slices or key slices in a single request), twice each (4800 runs). Nine radiologists read 30 of those cases. A large language model judge validated against two radiologists graded the answers. Conditions were compared with generalized estimating equations clustered by case. Results. Pooled accuracy was lowest under free navigation (32.6%; 391/1200), followed by navigation from a key slice (37.9%; 455/1200; P = .003), and highest with all slices (42.2%; 506/1200) or key slices (42.5%; 510/1200) in one request (both P < .001). On the 30 cases read by radiologists, freely navigating models were correct in 30.0% of runs (108/360) versus 77.0% for radiologists (208/270; P < .001). Models formed a first hypothesis after a median of 10.8% of the slices and kept it in 63.1% of runs (755/1196); final answers were correct in 12.4% of runs after a wrong first hypothesis (110/885) versus 90.4% after a right one (281/311; P < .001). Median stated confidence in the final answers was 90%, whereas 38.9% of them were correct (1861/4778). Conclusion. Across all exploration strategies, VLM agents remained far less accurate than radiologists. They adopted an early hypothesis and usually kept it, seldom correcting a wrong one.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23480 [cs.AI] |
| (or arXiv:2611.23480v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23480 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bastien Le Guellec [view email]
[v1]
Tue, 17 Nov 2026 20:32:21 UTC (1,295 KB)
来源:arXiv cs.AI · arxiv.org