arXiv 论文：Agent 基准饱和多在计量而非任务，提出分级评分方法
Agent Benchmarks Stop Ranking Before They Run Out of Tasks
AI 导读
arXiv 论文（arXiv:2611.13541）重评八个基准的已发布运行结果，包括 SWE-bench Verified 和 Terminal-Bench，发现 Agent 基准的饱和多来自计量方式而非任务本身。
推荐理由
论文重评八个 Agent 基准，指出饱和主要源于 pass/fail 计量过粗，并提出带成本轴的对数比率评分方案。
正文 · 原文
Abstract:Data-science and ML-engineering agents are evaluated on benchmarks such as MLE-bench, ALE-Bench, SWE-fficiency and AlgoTune, whose tasks yield a quantity (a Kaggle metric, contest score or speedup). When one stops separating the best models, the usual remedy is a harder version. Re-scoring released runs of eight benchmarks, including SWE-bench Verified and Terminal-Bench for breadth, we find much of this saturation lies in the measurement, not the tasks. Most agent benchmarks reduce a run to one pass/fail bit, which fails in two ways. It is too coarse: neighbouring top-ten agents on SWE-bench Verified differ by less than its tests' errors, and correcting them reorders neighbours no more than deleting a random sixth of each agent's passes. It is capped at the reference: ALE-Bench's top ten, separated by contest score, tie when graded against the median human, although nearly half the runs beat that human. On fixed tasks, a higher bar only moves the problem; graded credit fixes it. We derive a clipped log-ratio score with dated anchors and cost as a separate axis. In a Terminal-Bench re-run, pass/fail-tied models differ over tenfold on the quantity the tests discard.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13541 [cs.AI] |
| (or arXiv:2611.13541v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13541 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zihan Dong [view email]
[v1]
Tue, 10 Nov 2026 04:03:04 UTC (153 KB)
来源：arXiv cs.AI · arxiv.org