arXiv cs.AI· Soham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Ivan Guevara, Fernando Carabedo, Jose Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue Zhang·· 3 小时前AI 评分59
SWE-Bench Pro V2 发布：更干净、更难被钻空子的编码基准
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
AI 导读
研究者发布编码智能体基准 SWE-Bench Pro V2，通过智能体审计、专家裁定与盲测重写修复或淘汰了731个任务中的480个，23名工程师投入404小时。新协议限制网络访问到模型端点并在干净容器中重评每个补丁，防止获取上游修复或补丁外得分。642个V2任务上前沿智能体最高解决99.4%，51个任务的Hard档把各系统从52.9%拉开到98.0%，基准与数据集已开源。
正文 · 原文
Authors:Soham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Ivan Guevara, Fernando Carabedo, Jose Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue Zhang
Abstract:SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.
| Comments: | Dataset: this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.24206 [cs.SE] |
| (or arXiv:2610.24206v2 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.24206 arXiv-issued DOI via DataCite |
Submission history
From: Soham Dan [view email]
[v1]
Fri, 16 Oct 2026 20:26:37 UTC (104 KB)
[v2]
Wed, 11 Nov 2026 02:55:51 UTC (100 KB)
来源：arXiv cs.AI · arxiv.org