arXiv cs.AI· Amama Mahmood, Shiye Cao, Laura Fleig, Junan Jiang, Theodore Jensen, Yee-Yin Choong, Kristen Greene, Chien-Ming Huang·· 3 小时前AI 评分43
医疗环境 AI 记录助手评估碎片化：54 项研究中有 96 种测量方式
Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare
AI 导读
一项 arXiv 研究分析 54 项医疗环境 AI 记录助手（ambient scribe）评估研究，识别出 10 个评估维度、40 种指标，共 96 种操作化方式，其中 35 种（36%）仅被单篇研究使用。评估以文档效率和时间为主（54 项中 44 项），安全与临床结局评估最少，且指标高度依赖自我报告，作者呼吁建立标准化多维评估体系。
正文 · 原文
Abstract:Ambient artificial intelligence (AI) scribes have moved rapidly from experimental pilots to enterprise-wide deployment across health systems, promising reduced documentation burden and improved clinician experience. While research evaluating these tools has expanded quickly, there are no agreed-upon protocols or standardized metrics for assessing their performance, making it difficult to compare findings or cumulatively synthesize evidence across studies. Across 54 recent evaluation studies, we identify ten evaluation dimensions and 40 distinct metrics, operationalized in 96 different ways-35 of them (36%) used by only a single study. Assessment is dominated by documentation efficiency and time, measured in 44 of 54 studies. The other dimensions appear at broadly similar rates (16-24 studies each), and safety and clinical outcome least consistently of all. Ripple effects on the broader care system, such as impact on patients' care outcomes, redistribution of work, changes in billing and coding intensity, and downstream use of notes by other clinicians are rarely assessed. Moreover, the metrics used rely heavily on self-report and are fragmented and inconsistently operationalized, which limits comparability and hinders synthesis across studies. As an effort to support more rigorous and cumulative evaluation, in this paper we review the study designs, data sources, and metrics commonly used to evaluate ambient AI scribes in healthcare settings, discuss the limitations of current evaluation practice, and argue for a more holistic approach grounded in a standardized, multi-dimensional set of metrics.
| Subjects: | Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14284 [cs.HC] |
| (or arXiv:2611.14284v1 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14284 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Amama Mahmood [view email]
[v1]
Tue, 10 Nov 2026 16:26:02 UTC (1,535 KB)
来源：arXiv cs.AI · arxiv.org