arXiv cs.AI· Yihan Yang, Chenyuan Yang, Zijie Zhao, Anh Nguyen Hoang, Ziqi Zhang, Weidong Wang, Zhiheng Lyu, Lingming Zhang·· 3 小时前AI 评分54
SecPRBench 提出基于真实漏洞提交的智能体安全代码审查基准
Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?
AI 导读
伊利诺伊大学团队（Yihan Yang 等）发布 SecPRBench，首个在经过验证的真实世界引入漏洞提交上评估智能体安全代码审查的基准，通过自动化流水线从 Firefox、V8、OpenSSL 等九个关键系统采集并验证了 265 个任务，并采用 LLM-as-a-judge 判分。
正文 · 原文
Abstract:LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes. More specifically, we have built an automated agentic pipeline to autonomously harvest, build, and validate massive real-world bug-introducing commits for popular software systems. To ensure precise evaluation, we also adopt LLM-as-a-judge to credit the findings matching ground-truth vulnerabilities. To date, we have collected 265 tasks from nine critical systems, including Firefox, V8, and OpenSSL with millions of lines of code. Across eight frontier models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT-6 sol ranking at the top. Our study also shows that most misses are not simply oversights: in 68% of them, the agent reports on a file that holds the flaw but argues for a different mechanism. Security-focused harnesses also behave differently: OpenAI's Codex Security lowers recall by 8.3 points, mostly by abstaining, while Anthropic's Defending Code harness raises it by 7.7 points, at a higher cost.
| Comments: | 20 pages, 2 figures, 10 tables. Benchmark: this https URL Results website: this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Programming Languages (cs.PL) |
| Cite as: | arXiv:2610.33509 [cs.SE] |
| (or arXiv:2610.33509v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33509 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yihan Yang [view email]
[v1]
Sat, 24 Oct 2026 07:32:42 UTC (236 KB)
来源：arXiv cs.AI · arxiv.org