arXiv cs.AI· Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jian-guang Lou, Haoyi Xiong·· 4 小时前AI 评分43
LLM 智能体多轮对话评测方法综述
Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
AI 导读
一篇综述系统梳理了基于 LLM 的智能体在多轮对话场景下的评测方法，基于 PRISMA 框架审阅了近 250 篇文献，提出两套关联的分类体系。"评什么"涵盖任务完成、回复质量、用户体验、记忆与上下文保持、规划与工具集成；"怎么评"分为标注评测、自动化指标（如 BLEU、ROUGE）、混合策略及 LLM 自评方法。该研究发表于 ACM TIST 17(4)。
正文 · 原文
Abstract:This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, we systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication, and establishing a solid foundation for our analysis. Our study offers a structured approach by developing two interrelated taxonomy systems: one that defines \emph{what to evaluate} and another that explains \emph{how to evaluate}. The first taxonomy identifies key components of LLM-based agents for multi-turn conversations and their evaluation dimensions, including task completion, response quality, user experience, memory and context retention, as well as planning and tool integration. These components ensure that the performance of conversational agents is assessed in a holistic and meaningful manner. The second taxonomy system focuses on the evaluation methodologies. It categorizes approaches into annotation-based evaluations, automated metrics, hybrid strategies that combine human assessments with quantitative measures, and self-judging methods utilizing LLMs. This framework not only captures traditional metrics derived from language understanding, such as BLEU and ROUGE scores, but also incorporates advanced techniques that reflect the dynamic, interactive nature of multi-turn dialogues.
| Comments: | Companion repository: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2503.22458 [cs.CL] |
| (or arXiv:2503.22458v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2503.22458 arXiv-issued DOI via DataCite |
|
| Journal reference: | ACM Transactions on Intelligent Systems and Technology 17(4), 1-40 (2026) |
| Related DOI: | https://doi.org/10.1145/3793671
DOI(s) linking to related resources |
Submission history
From: Shengyue Guan [view email]
[v1]
Fri, 28 Mar 2025 14:08:40 UTC (1,847 KB)
[v2]
Mon, 5 Jan 2026 15:14:04 UTC (1,107 KB)
[v3]
Sun, 25 Oct 2026 13:10:49 UTC (1,107 KB)
来源：arXiv cs.AI · arxiv.org