跳到正文
arXiv cs.AI· Sneha Maurya, Spandana Govindgari, Girish Kumar·· 2 小时前AI 评分36

WHBench:面向女性健康主题、结合专家在环验证的前沿 LLM 评测基准

WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics

AI 导读

研究者推出 WHBench(Women's Health Benchmark),包含 10 个女性健康主题下 47 个专家设计的场景,用于暴露过时指南、不安全遗漏、剂量错误和公平性盲区等临床失败模式。评测 22 个模型共 3,102 次响应(3,100 次计分),无模型平均分超过 75%,最佳模型为 72.1%。该基准公开可用,支持模型间对比评估,同时强调临床部署仍需专家监督。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large language models are increasingly used for medical guidance, but women's health remains under-evaluated in benchmark design. We present the Women's Health Benchmark (WHBench), a targeted evaluation suite of 47 expert-crafted scenarios across 10 women's health topics, designed to expose clinically meaningful failure modes including outdated guidelines, unsafe omissions, dosing errors, and equity-related blind spots. We evaluate 22 models using a 23-criterion rubric spanning clinical accuracy, completeness, safety, communication quality, instruction following, equity, uncertainty handling, and guideline adherence, with safety-weighted penalties and server-side score recalculation. Across 3,102 attempted responses (3,100 scored), no model mean performance exceeds 75 percent; the best model reaches 72.1 percent. Even top models show low fully correct rates and substantial variation in harm rates. Inter-rater reliability is moderate at the response label level but high for model ranking, supporting WHBench utility for comparative system evaluation while highlighting the need for expert oversight in clinical deployment. WHBench provides a public, failure-mode-aware benchmark to track safer and more equitable progress in womens health AI.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
Cite as: arXiv:2604.00024 [cs.CL]
  (or arXiv:2604.00024v3 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2604.00024

arXiv-issued DOI via DataCite

Submission history

From: Spandana Govindgari [view email]
[v1] Wed, 11 Mar 2026 06:21:37 UTC (1,258 KB)
[v2] Thu, 23 Jul 2026 23:19:26 UTC (1,258 KB)
[v3] Tue, 17 Nov 2026 04:47:58 UTC (1,258 KB)

来源:arXiv cs.AI · arxiv.org