DoseBench：评估 LLM 在 OTC 用药剂量问答中的决策能力
Can I Take Another Dose? DoseBench: Evaluating LLM Decision-Making in OTC Dosing QA
AI 导读
研究者推出 DoseBench 基准，包含 230 道带人工标注（YES/NO/AMBIGUOUS）的对乙酰氨基酚与布洛芬消费者用药问题，评估 8 个 LLM、每题 5 次回答。模型总体准确率为 44.8%–62.2%，并错误批准 14.0%–60.5% 的禁用剂量。分析显示追加用药指引会让模型更谨慎，提供药品说明可提升准确率但可能使其忽略信息缺失。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly used to answer everyday health questions online, including whether someone can take another dose of an over-the-counter (OTC) medication. Yet this common safety-relevant setting remains underexplored in medical QA. Answering these questions requires checking previous doses, timing, and product-specific dosing limits, while recognizing when important information is missing. We introduce DoseBench, a focused benchmark of 230 consumer-style acetaminophen and ibuprofen dosing questions with human-annotated YES, NO, and AMBIGUOUS labels, and evaluate eight LLMs with five responses per question. Overall accuracy ranges from 44.8% to 62.2%, while models incorrectly approve 14.0% to 60.5% of prohibited doses. Models with similar accuracy can therefore make very different dosing decisions. Additional analyses show that these failures are not easily fixed by giving models more guidance. Asking models to check dose, timing, and missing information makes them more cautious, while providing drug facts improves accuracy but can make them overlook missing information. These findings highlight a broader challenge in health QA: applying rules correctly and recognizing when information is missing.
| Comments: | 12 pages, 5 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.04262 [cs.CL] |
| (or arXiv:2606.04262v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.04262 arXiv-issued DOI via DataCite |
Submission history
From: Maroof Kousar [view email]
[v1]
Tue, 2 Jun 2026 22:30:15 UTC (2,556 KB)
[v2]
Thu, 29 Oct 2026 00:02:17 UTC (132 KB)
来源：arXiv cs.AI · arxiv.org