arXiv cs.AI· Hasan Najib Mahmud, Shreya Gupta, Isha Chaudhary, Nathaniel Enis, Ravi Mangal, Gagandeep Singh, Corina Pasareanu·· 4 小时前AI 评分37
可编程语义等价变体分布评估代码智能体:VDL 揭示 SWE-bench 解题率下降
Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents
AI 导读
研究者提出 VDL,一种概率编程语言,通过对代码仓库施加语义保持变换生成等价变体,评估代码智能体在表面代码变化下的可靠性。
正文 · 原文
Abstract:Code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We sample semantically equivalent variants of a repository from an explicitly defined distribution and compare the agent's resolve rate on them with that on the original. Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix. We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository. For our evaluation, we use it to define a generic distribution that is not targeted at any particular failure mode. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode), each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B), on instances from SWE-bench Verified and SWE-bench Pro. Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations. No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2608.18389 [cs.AI] |
| (or arXiv:2608.18389v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.18389 arXiv-issued DOI via DataCite |
Submission history
From: Hasan Najib Mahmud [view email]
[v1]
Tue, 18 Aug 2026 23:46:11 UTC (762 KB)
[v2]
Thu, 12 Nov 2026 22:58:38 UTC (721 KB)
来源:arXiv cs.AI · arxiv.org