基于缓存 Trace 的高效智能体评测方法研究
Task-Efficient Agent Evaluation Using Cached Traces
AI 导读
一项研究提出用 LLM 提取智能体执行 trace 的结构化摘要并用文本编码器嵌入，以更少任务执行次数估计智能体系统在基准测试上的完整表现。该方法保留正确性与任务相关信号，同时抑制系统身份、模型家族等信息。
正文 · 原文
Abstract:Efficient benchmarking estimates a system's full benchmark performance from a small number of task executions. Standard approaches rely solely on the binary correctness of executions from a collection of already-evaluated reference systems. An agentic system, however, produces a rich trace for each execution that can reveal substantially more about its capabilities. Leveraging this information requires useful trace representations. We propose representing a trace by extracting a structured summary of it with an LLM and embedding the summary with a standard text encoder. These representations preserve correctness- and task-related signal while minimizing the presence of system identity-, model family- and harness-related signals that dominate raw trace embeddings. Given this representation for each task execution from all reference systems, we construct low-dimensional representations of the systems and predict a new system's score from its position among the reference systems. On SWE-bench Verified and Terminal-Bench 2.0, augmenting an item-response model with this geometry-based prediction reduces mean absolute error from 0.13 to 0.08 (Wilcoxon $p < 10^{-4}$) and from 0.14 to 0.12 (Wilcoxon $p < 10^{-3}$), respectively, when provided a single random executed task. Further, given access to the full set of reference systems, our approach dominates baselines for any given number of executed tasks on both benchmarks.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06803 [cs.AI] |
| (or arXiv:2611.06803v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06803 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ben Johnson [view email]
[v1]
Wed, 4 Nov 2026 17:20:09 UTC (135 KB)
来源：arXiv cs.AI · arxiv.org