arXiv cs.AI· Matej Gerek, Danylo Zahorulko, Samuel Budai, Chris Rupley, Thierry Donneau-Golencer·· 1 小时前AI 评分43
面向代码智能体免执行评测的仓库级 Rubric 方法(arXiv 论文)
Repository-Grounded Rubrics for Execution-Free Evaluation of Coding Agents
AI 导读
一篇 arXiv 论文提出将 agentic rubrics 改造成免执行的代码智能体评测方法:智能体先探索改动前的仓库并生成加权二元标准,再由无工具的 judge 按满足比例给 diff 打分。
正文 · 原文
Abstract:Coding agents are usually evaluated by running task-specific tests, but tests may be missing or incomplete, and running them requires a working environment for each repository. We adapt agentic rubrics, introduced for selecting among candidate patches, into an execution-free measure of how far a patch satisfies a task's requirements. An agent explores the repository before the change and writes weighted binary criteria, and a judge without tools scores a diff by the weighted share of criteria it satisfies. We validate the ordering this score produces against test outcomes on SWE-bench Verified, using 7,291 labeled patches from fifteen systems. Within the 400 tasks that have both resolved and unresolved patches, the mean per-task AUROC is 0.826 (0.838 on tasks not used to tune the rubric-generation prompt), and ranking systems by their failing patches closely agrees with ranking them by resolve rate (Spearman 0.968). Case analyses trace disagreements to tests that do not match the task, execution-blocking defects that the textual assessment misses, and errors in criteria or verdicts. The validation does not establish absolute score magnitudes or requirement satisfaction beyond agreement with tests, but rubric scores could complement tests where tests or execution environments cannot be assumed.
| Comments: | 31 pages, 6 figures, 15 tables. Code and data: this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23309 [cs.SE] |
| (or arXiv:2611.23309v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23309 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Matej Gerek [view email]
[v1]
Wed, 11 Nov 2026 09:21:43 UTC (125 KB)
来源:arXiv cs.AI · arxiv.org