arXiv cs.AI· Fumiki Yohena, Toshiaki Sota, Shota Suzuki, Tomoko Okamoto, Michiaki Tatsubori·· 3 小时前AI 评分34
仓库上下文引导的图像理解助力智能体软件开发
Repository-Context-Guided Image Understanding for Agentic Software Development
AI 导读
一项 arXiv 研究提出为 coding agent 附加从同一代码库渲染的参考截图，用于在 bug 报告图像质量退化时定位相关源文件。基于 SWE-bench Multimodal 构建的控制实验显示，参考图像在四倍降采样加 JPEG 压缩下提升了正确文件匹配率；同等退化条件下，该方法的文字识别准确率比偏向语言的 OCR 约高 12 个百分点。
正文 · 原文
Abstract:Coding agents increasingly receive UI screenshots as part of bug reports, yet degraded image quality - caused by low resolution, compression artifacts, or small rendered text - can prevent them from identifying the relevant UI elements and source files. Vision-language models become less reliable as image quality deteriorates, while OCR-assisted approaches may propagate recognition errors from degraded inputs. Both approaches are limited when the visual evidence required for localization is no longer clearly observable in the query screenshot. We propose augmenting coding agents with repository-rendered reference images: crisp screenshots generated from the same codebase that the agent is tasked with modifying. These references provide readable visual evidence of the application's UI while preserving the current, potentially bug-containing implementation state. Because they are generated from the available repository rather than requiring a known bug-free version, they can be produced automatically for reproducible application states without manual image annotation. We evaluate the approach on a controlled benchmark derived from SWE-bench Multimodal by injecting visual UI bugs into a real web application. We vary screenshot degradation and reference-image provision and measure whether the coding agent identifies the correct source file. The results show that reference images improve file localization especially for degraded image quality, increasing the gold file match rate under fourfold downsampling with JPEG compression. We confirmed with further analysis that the proposed approach recognizes text approximately 12 percentage points better than language biased OCR under the same degradation.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.13026 [cs.SE] |
| (or arXiv:2611.13026v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13026 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Michiaki Tatsubori [view email]
[v1]
Mon, 9 Nov 2026 08:33:52 UTC (3,795 KB)
来源：arXiv cs.AI · arxiv.org