arXiv cs.AI· Oscar Freyer, Daniel Glauert, Moritz Hofmann, Max Ostermann, Stephen Gilbert, Jakob Nikolas Kather·· 3 小时前AI 评分39
医学选择题基准评测：13 个决策模型对比 15 种生成式大语言模型配置
Benchmarking decision models against generative large language models on medical multiple-choice questions
AI 导读
一项 arXiv 研究在 1,273 道 MedQA 四选项题上，用纯文本条件对比 13 个决策模型与 15 种生成式大语言模型配置。决策模型准确率 67.71%–89.08%，生成式配置为 78.32%–96.86%；Mercury Decide 均值最终尝试成本最低（每千题 USD 0.0069），Clef Omni 中位请求时长最短（0.291 秒）。
正文 · 原文
Abstract:Medical workflows often require selecting among predefined options. We compared 13 decision models with 15 generative large language model configurations, including different reasoning settings, using 1,273 four-option MedQA questions under text-only conditions. Accuracy ranged from 67.71% to 89.08% for decision models and from 78.32% to 96.86% for generative configurations. Among the tested configurations, Mercury Decide had the lowest mean final-attempt cost (USD 0.0069 per 1,000 questions), while Clef Omni had the shortest median request duration (0.291 seconds). Decision-model probabilities distinguished correct from incorrect answers, but error rates among answers assigned probabilities of at least 0.90 ranged from 1.71% to 12.77%. Answer agreement across three runs ranged from 92.30% to 100%, while probability repeatability varied. Decision models offered resource advantages alongside lower peak accuracy; selecting models for medical workflows requires considering probability quality and repeatability together with accuracy and the resources needed to detect and correct errors.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38448 [cs.CL] |
| (or arXiv:2610.38448v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38448 arXiv-issued DOI via DataCite |
Submission history
From: Oscar Freyer [view email]
[v1]
Wed, 28 Oct 2026 08:25:35 UTC (4,965 KB)
来源：arXiv cs.AI · arxiv.org