跳到正文
arXiv cs.AI· Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai·· 2 小时前AI 评分53

arXiv 论文提出 anytime evaluation 研究编码智能体何时该停:自测通过后继续工作最多可省 57% 成本

Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop

AI 导读

arXiv 论文提出 anytime evaluation 方法,通过回放约 6000 次运行、用隐藏测试给中间补丁打分并统计成本,研究编码智能体自测通过后继续工作是否有价值。

正文 · 原文

Authors:Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai

View PDF HTML (experimental)

Abstract:Coding agents often continue working after their own tests pass. Does this additional work improve the patch, and how much could be saved by stopping earlier? We study these questions through anytime evaluation: replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step. For policies that only stop a run or select an earlier patch, this allows paired comparisons without generating new trajectories, provided replay preserves the relevant states. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, we find substantial differences in the value of continued work. The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority. We explain these differences by measuring how often a rule fires, how often the patch changes afterwards, and whether those changes repair or break it. Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain. Our results help identify both the opportunities and the risks of early stopping.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22901 [cs.AI]
  (or arXiv:2611.22901v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22901

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zerui Yang [view email]
[v1] Tue, 17 Nov 2026 15:07:35 UTC (521 KB)

来源:arXiv cs.AI · arxiv.org