Replit 发布 ViBench 2.0 基准：评估长周期智能体软件构建

View PDF HTML (experimental)

Abstract:On Replit, our agentic engineering platform, users now build entire products without writing code, through successive feature requests to an agent. Yet existing coding benchmarks evaluate single, fully specified tasks, and many are near saturation. We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models. Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does, under a grading protocol validated to keep this strict score reliable. Evaluating nine models, we find a reliability gap: the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds, and no open-weight model delivers one. The gap is closing fast: across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half. We release eight products with the harness and grader, hold out nine, and report lessons from production use.

Subjects:	Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
ACM classes:	I.2.2; D.2.5; D.2.8
Cite as:	arXiv:2611.05620 [cs.AI]
	(or arXiv:2611.05620v1 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2611.05620 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Peter Yong Zhong [view email]
[v1] Tue, 3 Nov 2026 21:04:39 UTC (564 KB)