CHI-Bench 基准测试显示最强智能体仅完成 28.0% 的医疗长流程任务
CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
研究者发布 CHI-Bench，一个覆盖供应商预授权、付款方利用管理和照护管理三类长周期医疗工作流的基准，配备 20 个医疗应用、87 个 MCP 工具的高保真模拟器和 1,290 多份管理式医疗运营手册。在 30 种智能体框架与模型配置中，最好的智能体只完成 28.0% 的任务，严格 pass^3 下无一超过 20%，单会话执行全部任务时性能跌至 3.8%。
Authors:Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, Frank Wang, T. Y. Alvin Liu, Hank Capps MD, Zeyu Tang, Xiangchen Song, Lingjing Kong, Fan Feng, Tianyi Zeng, Zhiwei Liu, Zixian Ma, Hang Jiang, Fangli Geng, Yuan Yuan, Chenyu You, Qingsong Wen, Hua Wei, Yanjie Fu, Yue Zhao, Carl Yang, Biwei Huang, Kun Zhang, Caiming Xiong, Sanmi Koyejo, Eric P. Xing, Philip S. Yu, Weiran Yao
Abstract:End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $\chi$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.
| Comments: | Website: this https URL Code: this https URL Dataset: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.16679 [cs.CL] |
| (or arXiv:2605.16679v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16679 arXiv-issued DOI via DataCite |
Submission history
From: Haolin Chen [view email]
[v1]
Fri, 15 May 2026 22:34:31 UTC (6,764 KB)
[v2]
Tue, 19 May 2026 05:51:20 UTC (11,031 KB)
[v3]
Wed, 30 Sep 2026 05:55:42 UTC (8,994 KB)
[v4]
Thu, 29 Oct 2026 06:50:19 UTC (6,160 KB)
来源：arXiv cs.AI · arxiv.org