LongSearchEval：基于轨迹感知智能体框架评估大语言模型长程网络搜索能力
LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
AI 导读
arXiv 论文提出 LongSearchEval，一个用于评估大语言模型长程网络搜索能力的轨迹感知智能体框架，包含统一执行协议、来源约束结果验证与失败节点定位两阶段流程。团队对 8 个模型族的 16 个模型在 100 个任务上共保留 1,600 次运行，自动标记 6,545 个失败节点；人工验证显示答案组标签一致率达 94.50%，轮次-答案组链接的特征对应精度达 98.11%。
正文 · 原文
Abstract:Long-horizon web search requires LLM-based agents to integrate information across web pages through extended interactions under task-specific constraints. Evaluating this capability is important not only for measuring final answer quality, but also for assessing whether an agent acquires, verifies, and uses evidence reliably over extended interactions. Existing evaluations, however, mainly rely on simulated trajectories or on post-hoc trajectories reconstructed from final answers, rather than on retained real interaction traces, which limits the fidelity of behavioral assessment and weakens support for fine-grained error localization. To address this gap, we introduce LONGSEARCHEVAL, a trajectory-aware agent evaluation framework that combines a unified agent execution protocol with source constrained outcome verification and retrospective failure-node localization. Stage 1 verifies answer groups against agent-cited webpages without human-authored gold answers; Stage 2 audits normalized rounds and links flagged failure nodes to incorrect answer groups. We evaluate 16 models from eight families on 100 tasks, yielding 1,600 retained runs and 6,545 automatically flagged failure nodes. Trajectory collection and evaluation required 1,037 and 318 hours of cumulative runtime, respectively. Human validation yields 94.50% agreement on supplied answer-group labels and 98.11% feature-correspondence precision for audited round-group links. Together, outcome scores and linked trajectory evidence support controlled model comparison and retrospective inspection of answer-related content.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35150 [cs.AI] |
| (or arXiv:2610.35150v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35150 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shusheng Li [view email]
[v1]
Mon, 26 Oct 2026 10:15:38 UTC (8,834 KB)
来源：arXiv cs.AI · arxiv.org