跳到正文
arXiv cs.AI· Xianpeng (Simon), Sun (Ken), Tian Yu (Ken), Haonan Sun (Ken), Sheng Ma (Ken), Qincheng Zhang (Ken), Lifei Rao (Ken), Jie Li (Ken), Jian (Ken), Cao, Qi Zhang·· 4 小时前AI 评分50

ServiceExplorer 分层代码图在 SWE 智能体时间一致评测中的研究

Time-Consistent Evaluation of Hierarchical Code Graphs for SWE Agents

AI 导读

一项 arXiv 研究在 DragonFly(C++)与 React(JavaScript/TypeScript)仓库上,以历史截止 commit T_0 锁定快照,对 100 个 post-T_0 PR 做零泄漏评测,比较无增强 Baseline、Tree-sitter AST 图与 ServiceExplorer 两级分层依赖图。

正文 · 原文

View PDF HTML (experimental)

Abstract:Evaluating repository-aware software engineering (SWE) agents requires strict separation between pre-existing repository knowledge and future code evolution, yet standard benchmarks frequently suffer from temporal contamination and prompt-level solution leakage. Building on the time-consistent benchmark protocol of Sun (2026), we enforce strict zero-leakage isolation across DragonFly in C++ and React in JavaScript/TypeScript, locking repository snapshots and knowledge graphs at historical cutoff commits $T_0$ and evaluating 100 post-$T_0$ pull requests across four prompt granularities: Minimal, Concise, Contextual, and Guided.
Under this zero-leakage protocol, we conduct a matched evaluation of three context-augmentation regimes: (1) an unaugmented Baseline, (2) a flat syntax-level Tree-sitter AST graph, and (3) ServiceExplorer, a deterministic two-level hierarchical dependency graph capturing Level-1 system/package containers and Level-2 architectural modules. Across 7,200 paired evaluations spanning six foundation models across the MAI-Code Flash, Claude, and GPT-Sol families, ServiceExplorer consistently outperforms both Baseline and Tree-sitter AST across every model, repository, and prompt granularity ($+15.6\%$ to $+86.2\%$ relative $F_1$ lift, $p < 10^{-4}$), while tripling exact full-PR localization on DragonFly. Crucially, equipping an efficiency-tier or prior-generation model with ServiceExplorer consistently surpasses the unaugmented Baseline $F_1$ of the next-generation model across all evaluated families.
Comments: 10 pages, 5 figures, 6 tables
Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.17388 [cs.SE]
  (or arXiv:2611.17388v1 [cs.SE] for this version)
  https://doi.org/10.48550/arXiv.2611.17388

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xianpeng (Simon) Sun [view email]
[v1] Thu, 12 Nov 2026 00:51:46 UTC (122 KB)

来源:arXiv cs.AI · arxiv.org