胸片 AI 跨 6 国 12 个数据集评测：95% 敏感度下最佳模型仅约三分之一阴性可被排除
Benchmarking Classification Tasks of Chest X-ray AI Across Multinational Public and Private Datasets: a retrospective diagnostic accuracy study
一项回顾性研究在来自六个国家的 12 个数据集（7 个公开基准和来自中国、日本、德国的 5 个机构队列）上评测了 10 个已发表的胸片分类模型，主要终点为 95% 敏感度下的特异度。
Authors:Qinmei Xu, Yiheng Li, Wenya Zhao, Zihao Zhao, Chuanjun Xu, Ahmet Gorkem Er, Xianghao Zhan, Brittany Dashevsky, Mohammed Alawad, Marc von der Stück, Marvin Gazibarić, Daniel Truhn, Shunsuke Shibata, Ryosuke Horino, Mio Ito, Akio Hiwatashi, Suyu Li, Mengya Yang, Ya Liu, Zhiyuan Sun, Changsheng Zhou, Haruka Itakura, Xiao Li, Olivier Gevaert
Abstract:Chest X-ray (CXR) AI models are increasingly proposed for triage, screening, and rule-out, where the operating point is fixed to miss few abnormalities. Most evaluations instead report threshold-free discrimination on a few public benchmarks, leaving clinically usable performance and generalisability to unseen institutional cohorts poorly characterised. We retrospectively evaluated 10 published CXR classification models on 12 datasets from six countries: seven public benchmarks and five institutional cohorts from China, Japan, and Germany. Models spanned convolutional networks, vision-language models, a question-answering foundation model, and a repurposed encoder. Only image-level classification was assessed; segmentation, detection, and report generation were out of scope. Source labels were harmonised to a common vocabulary. The primary outcome was specificity at 95% sensitivity; PPV at 95% sensitivity and AUROC were secondary outcomes. Estimates weighted each diagnostic task equally, with 95% CIs. MAVL and MedKLIP had the highest overall specificity at 95% sensitivity (0.37 [95% CI 0.31-0.44] and 0.37 [0.30-0.44]) and kept that ranking on tasks evaluable for every model. Their PPV was 0.18 (0.13-0.24) and 0.19 (0.13-0.25): most flagged examinations would be false positives. Specificity was lower on institutional cohorts than on public benchmarks for most models, by 0.13 for MAVL, 0.15 for DenseNet, and 0.09 for MedKLIP. No model was best for every finding, country, lesion extent, or prevalence stratum; specificity fell from 0.34 for diffuse to 0.15 for focal findings. At 95% sensitivity, the best current models clear only about a third of truly negative examinations, and public-benchmark performance overstates what they achieve on unseen institutional data. Operating-point-aware evaluation and external validation on institutional cohorts should precede deployment.
| Comments: | v2: substantially revised and expanded (12 datasets from six countries, 10 models, 24 authors); primary outcome is now specificity at 95% sensitivity. 34 pages, 5 figures, 2 tables, plus appendix (5 tables, 1 figure). v1: 78 pages, 7 figures, 2 tables |
| Subjects: | Image and Video Processing (eess.IV); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| ACM classes: | I.2 |
| Cite as: | arXiv:2505.16027 [eess.IV] |
| (or arXiv:2505.16027v2 [eess.IV] for this version) | |
| https://doi.org/10.48550/arXiv.2505.16027 arXiv-issued DOI via DataCite |
Submission history
From: Yiheng Li [view email]
[v1]
Wed, 21 May 2025 21:16:50 UTC (23,758 KB)
[v2]
Sat, 24 Oct 2026 02:43:58 UTC (946 KB)
来源：arXiv cs.AI · arxiv.org