arXiv cs.AI· Jialei Li, Dingxian Wang, Xiaoyan Zhao, Yimeng Bai, Xuanyi Zhu, Yang Zhang, Chao Yin, Wentao Shi, Ganesh Prabhu, Karthik Gomadam Rajagopal, Brett Levert, Darvin Yi, Andrew Rabinovich, Zhao Chen·· 2 小时前AI 评分51
SWE-Clar 基准发布，评测编码智能体澄清不完整客户需求并交付全新软件的能力
SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?
AI 导读
研究者发布 SWE-Clar 基准，基于 Upwork 上总付款不低于 US$4,000 的 100 个真实软件开发任务，评测编码智能体在代码实现前澄清客户需求并独立交付全新软件的能力。
正文 · 原文
Authors:Jialei Li, Dingxian Wang, Xiaoyan Zhao, Yimeng Bai, Xuanyi Zhu, Yang Zhang, Chao Yin, Wentao Shi, Ganesh Prabhu, Karthik Gomadam Rajagopal, Brett Levert, Darvin Yi, Andrew Rabinovich, Zhao Chen
Abstract:Greenfield software development in freelance settings requires coding agents to turn incomplete client requests into working software. This makes requirement clarification essential, yet clients may be unable to provide continuous feedback over long development cycles, and receiving their answers does not guarantee correct understanding of complex requirements. These challenges motivate two benchmark design choices: clarification precedes code implementation, and its effectiveness is assessed through explicit tests of requirement understanding. Guided by these choices, we introduce SWE-Clar, a benchmark grounded in real, high-value client commissions to evaluate requirement clarification and code implementation in this setting. It comprises 100 tasks based on real software development jobs on Upwork, each with at least US\$4,000 in total payments. Each task begins with multi-round interaction between a coding agent and a client agent, followed by separate runs for code implementation with acceptance testing and clarification assessment. For the latter, we construct 512 questions paired with reference answers from real client-freelancer interactions, covering both explicitly stated requirements and those inferred from the conversational context. Both runs begin from the same clarification context but remain isolated, preventing assessment questions from guiding code implementation. Experiments with nine frontier models show that even GPT-6 Astra, which has the highest observed implementation score, achieves a task pass rate of only 67%, underscoring the challenge of independent software delivery in this setting. The dataset and evaluation code are available at this https URL.
| Comments: | 29 pages, 9 figures, 11 tables. Code and data: this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.31854 [cs.SE] |
| (or arXiv:2610.31854v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31854 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jialei Li [view email]
[v1]
Thu, 22 Oct 2026 19:36:48 UTC (423 KB)
来源：arXiv cs.AI · arxiv.org