arXiv cs.AI· Mohammad Arvan, Amber E. Osterholt, Bailee Rue, Yuvaneswaren R. Sureshbabu, Krishna R. Patel, Rebecca T. Feinstein, Bethany C. Bray, Niranjan S. Karnik·· 5 小时前AI 评分41
人机协作 AI 智能体起草转化科学影响总结的真实世界评估
Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries
AI 导读
一项研究评估了一个人机协作 AI 智能体，它为 CTSA 项目的 KL2/K12 学者收集证据并起草 TSBM 影响总结供人工审核。在 10 名学者的 507 条发现中，两名评审对 81.7% 予以接受或编辑，每人每名学者仅需约 14 分钟，替代原需约 15 小时的手工整理；综合准确性评分 4.5、有用性 4.8（5 分制）。
正文 · 原文
Abstract:Introduction. Clinical and Translational Science Award (CTSA) programs must document their scholars' research impact, but assembling each scholar's record by hand takes staff an estimated 15 hours and does not scale to a full cohort. An artificial intelligence (AI) agent could serve as a tool to gather scholar data across platforms and disciplines. Methods. We built a human-in-the-loop AI agent that assembles a dossier of sourced evidence for each scholar and drafts one-sentence Translational Science Benefits Model (TSBM) impact summaries for staff review. We evaluated it in the impact-reporting workflow of one CTSA hub across 10 career-development (KL2/K12) scholars. Two evaluation staff independently coded all 507 findings as accept, edit, or reject; the primary measure was the unanimous usable rate, defined as the share both accepted or edited. Results. Both reviewers accepted or edited 81.7% of the agent's findings. Reviewers each spent a median of 14 minutes per scholar, replacing an estimated 15 hours of manual assembly. Inter-rater agreement was moderate (Cohen's kappa 0.43 on the usable-versus-reject decision). A profile discovery study found the agent's recall close to human search. The agent's impact evidence spanned all four TSBM domains, and about a third of the reviewed findings fell in non-scholarly categories that routine processes tend to miss. Reviewers rated synthesis accuracy 4.5 and usefulness 4.8 on a 5-point scale. Conclusions. A human-in-the-loop AI agent can serve as the first-pass author of a scholar's impact record, shifting staff from collecting and writing to reviewing, and making cohort-scale impact reporting feasible.
| Comments: | 15 pages, 5 figures, 2 tables. Submitted to the Journal of Clinical and Translational Science. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Digital Libraries (cs.DL); Human-Computer Interaction (cs.HC) |
| ACM classes: | I.2.7; I.2.11; H.3.7; H.5.2 |
| Cite as: | arXiv:2607.16989 [cs.CL] |
| (or arXiv:2607.16989v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.16989 arXiv-issued DOI via DataCite |
Submission history
From: Mohammad Arvan [view email]
[v1]
Sat, 18 Jul 2026 22:34:52 UTC (205 KB)
[v2]
Tue, 21 Jul 2026 15:50:03 UTC (204 KB)
[v3]
Thu, 29 Oct 2026 16:48:04 UTC (232 KB)
来源：arXiv cs.AI · arxiv.org