跳到正文
arXiv cs.AI· Annika Michael, Zachary Ellis, Yuxiang Zhou, Samuel Thio, Yajie Vera He, Spencer Hazel, Jared Joselowitz·· 2 小时前AI 评分40

声音重要吗?临床语音智能体真实通话中的挫折感检测评测

Does the Voice Matter? Evaluating Frustration Detection on Deployed Clinical Voice-Agent Calls

AI 导读

一项研究在已部署临床语音智能体的真实患者通话上评测逐轮挫折感检测:患者发言极短,约 59% 仅一两个词,仅靠转写文本在单词回合上的检测不排除随机水平,39 个挫折感单词回合仅标出 3 或 4 个。

正文 · 原文

View PDF HTML (experimental)

Abstract:Autonomous voice agents now conduct routine clinical follow-up calls, so some automated triage is unavoidable. A patient becoming frustrated is one thing that triage should catch. These systems typically read only the transcript the speech-to-text stage produces, not the voice, and whether that is enough is largely untested on real clinical speech. We study per-turn frustration detection on real patient calls from a deployed clinical voice agent: 150 turns from six calls labelled by three annotators, with a length-matched acted corpus as a control, and 714 turns from 30 calls. Patient turns are overwhelmingly short: across the deployed calls, roughly 59% are one or two words, against 13% in the full acted corpus, and a corpus accuracy gap remains even after the acted set is length-matched. On one-word turns, transcript-only detection does not exclude chance. Across five audio-capable language models, the transcript flags 3 or 4 of the 39 frustrated one-word turns in the 30 calls, and a classifier given only the word count reaches 57.3%. Whether audio helps depends on the model and on the instruction. Asked to attend to delivery, two Gemini 2.5 models gain 3.0 pp over the transcript across 30 calls (-1.0 to 7.5 pp). Of the turns they confidently clear, 6.9% were in fact frustrated for audio against 13.0% for the transcript, a difference of 6.1 pp (2.2 to 10.4 pp). Across the 30 calls, given one prompt identical for both inputs, no model's audio-text interval excludes zero. Under the input-specific prompts, a newer Gemini model and two smaller open models do worse with audio than with the transcript (-3.5, -8.6 and -10.6 pp). A WavLM probe trained on the clinical calls matches the best prompted model overall (71.6%) but misses every frustrated one-word turn in the 24 added calls. When a patient's turn is short, the transcript alone is not enough to rule out frustration.
Comments: Extended version of papers accepted at ML4H 2026 (Proceedings) and GenAI4H workshop @ NeurIPS 2026
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.21001 [cs.CL]
  (or arXiv:2611.21001v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.21001

arXiv-issued DOI via DataCite

Submission history

From: Annika Michael Ms [view email]
[v1] Mon, 16 Nov 2026 12:27:56 UTC (1,879 KB)
[v2] Tue, 17 Nov 2026 12:06:20 UTC (1,879 KB)

来源:arXiv cs.AI · arxiv.org