arXiv cs.AI· Marco Paina, Mark Niklas M\"uller, Niels M\"undler-Sasahara, Martin Vechev·· 4 小时前AI 评分58
PROBE 基准：前沿编码智能体找到的 bug 与维护者修复的存在显著差异
PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix
AI 导读
arXiv 论文提出 PROBE 基准，收录 20 个 GitHub 仓库 18 个月内修复的全部 2,457 个 bug，并给出候选发现与该部分参考集的匹配方法。
正文 · 原文
Abstract:Proactive bug discovery is an important software-engineering problem. Even though coding agents are already widely deployed in this domain, their capabilities remain understudied. In particular, the challenging task of finding all bugs that human maintainers would repair across an entire repository has not been studied yet. To close this gap, we introduce PROBE, a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories, together with a method for matching candidate findings to this partial reference set. We evaluate coding agents across frontier models and find they all achieve a recall of below $21\%$ and a precision of less than $9\%$ with respect to the reference set. While stronger models increase recall substantially from $8\%$ to $17\%$, they do not improve precision. Surprisingly, despite the low precision, LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers. The practical relevance of bug discovery and the low recall of frontier models make this benchmark a valuable resource for both model users and developers.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.35987 [cs.SE] |
| (or arXiv:2610.35987v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35987 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mark Niklas Müller [view email]
[v1]
Mon, 26 Oct 2026 12:04:46 UTC (548 KB)
来源：arXiv cs.AI · arxiv.org