arXiv cs.AI· Hannes Ill, Rafi Al Attrach, Rajna Fani, Ahram Han, Ryohei Yamamoto, Marlene Louisa Moerig, Dukyong Yoon, Leo Anthony Celi·· 2 小时前AI 评分48
M4Bench：评测临床 EHR 衍生智能体的流程化规范能力
M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents
AI 导读
M4Bench 是一个开源基准，包含基于 MIMIC-IV 和 eICU 的 28 个临床衍生任务、15 个任务族，用于测试 AI 编码智能体能否复现电子健康记录衍生变量，以及临床医生审核的 skills 带来的变化。
正文 · 原文
Abstract:Clinical research over electronic health records depends on derived variables whose definitions sit outside the schema, in item codes, time windows, missingness rules, and output-grain choices that change cohorts and downstream conclusions. We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU, to test whether AI coding agents can reproduce such derivations and how clinician-reviewed agent skills change their results. Its controls decompose the value of skills into specification completeness, delivery format, and schema familiarity. In a Docker-isolated Codex campaign (GPT 5.4 Mini and GPT 5.5), skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain: on an eight-task control subset, an operational spec alone raises reward by 0.162, and skills add only 0.024 on top, indistinguishable from zero. On a ten-task subset, decoy skills score like no skill and raw reference SQL nearly matches targeted skills, arguing against generic context or the skill format as the explanation. Schema obfuscation and restructuring preserve 86% and 70% of the skill gain (GPT 5.4 Mini, 26 MIMIC-IV tasks), a descriptive robustness check rather than a bound on memorization. A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent (+0.120, 21 of 28 tasks improving). These gains are partial agreement with the reference, not completed tasks: only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003). Skills are released as a versioned, clinician-reviewable markdown delivery format, with a family-level reference-validity audit. M4Bench releases tasks, skills, evaluation code, and sanitized per-run evidence as an open-source benchmark.
| Comments: | Accepted at NeurIPS 2026 (Evaluations and Datasets Track). 10 pages main text, 33 pages total. Code: this https URL . Evidence: this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.1; J.3; H.2.8 |
| Cite as: | arXiv:2610.37459 [cs.AI] |
| (or arXiv:2610.37459v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37459 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hannes Ill [view email]
[v1]
Tue, 27 Oct 2026 15:59:13 UTC (382 KB)
来源：arXiv cs.AI · arxiv.org