言语与肢体不一致：虚拟人智能体的言语与非言语错配研究
When Words and Bodies Disagree: Realizing Verbal and Nonverbal Mismatch
AI 导读
一项 arXiv 研究提出虚拟智能体非言语行为的“错配”分类——强化、缓和、泄露，并评估 LLM 能否在社会情境提示下选择合适的错配行为（如讽刺）。实验表明 LLM 能成功选出目标错配行为，真人评估也证实这些非言语线索显著改变观察者对智能体态度和情绪效价的感知，参与者能区分不同错配功能。
正文 · 原文
Abstract:Traditional nonverbal behavior generation systems for virtual agents typically take utterances as input and generate behaviors that match or illustrate the spoken words. However, human nonverbal behavior is shaped by more than speech content. It is influenced by speaker roles, interpersonal relationships, social context, and cognitive and emotional states of interactants. Therefore, the nonverbal channel carries information or emotions hidden in words. Modeling these complex verbal-nonverbal mismatches is critical for realistic virtual agents, particularly in training applications. We propose a taxonomy of mismatches, strengthening, mitigating, and leaking, and evaluate whether LLMs can select contextually appropriate mismatched behaviors. Through human-subject studies, we assess whether these context-driven behaviors, realized in a virtual human framework, produce the intended effects. Our results demonstrate that LLMs, when prompted with social context, successfully select targeted mismatched behaviors (e.g., sarcasm). Furthermore, the human-subject evaluation confirms that these nonverbal cues significantly shift observers' perceptions of the agent's attitude and emotional valence. Participants successfully distinguished between distinct mismatch functions, such as strengthening, mitigating, and leaking, perceiving clear directional shifts in social meaning compared to matching baselines.
| Subjects: | Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Robotics (cs.RO) |
| Cite as: | arXiv:2608.22731 [cs.AI] |
| (or arXiv:2608.22731v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.22731 arXiv-issued DOI via DataCite |
Submission history
From: Parisa Ghanad Torshizi [view email]
[v1]
Mon, 24 Aug 2026 02:38:55 UTC (1,994 KB)
[v2]
Tue, 10 Nov 2026 17:26:07 UTC (4,193 KB)
来源：arXiv cs.AI · arxiv.org