arXiv 论文测量 coding agent harness 效应:换 harness 的收益大半来自 token 与重跑波动
What Does a Harness Buy? Tokens, Mostly
AI 导读
arXiv 论文 arXiv:2610.04433 v2 将七个模型分别跑在 Claude Code、mini-SWE-agent 和 OpenCode 三个生产级 harness 中,在 SWE-bench Verified 及另外两个基准上重测。
推荐理由
论文用七个模型复测三个生产级 harness,量化换 harness 与重跑的差异,为评测报告方法提供了参考。
正文 · 原文
Abstract:A coding agent is a language model inside a harness: the system prompt, the tools, and the context management that let a chat model work in a repository. Most of the engineering on these agents goes into the harness, and vendors and leaderboards credit it with score gains. Earlier studies have fixed the model and swapped the harness, but have not measured whether the swap changes the pass rate more than running the same harness again. We run seven models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified and two benchmarks of other kinds, and rerun them unchanged. No harness leads consistently across models. Where the data can bound the difference between two harnesses, the bound is 2 to 5 points, and a harness swap changes about as many task outcomes as a rerun. The exceptions are harness effects in particular pairings of model and harness, whose direction changes with the model: OpenCode trails by 8 to 9 points on one model and leads by 14 on another. Cost differences between harnesses also depend on the model, from under 10% to 3 to 4 times. Which harness is better therefore has no answer apart from the model, and a harness difference should be reported next to a rerun.
| Comments: | 31 pages, 9 figures. v2: seven models, reruns of every model, two non-SWE benchmarks; revised conclusions on pass rate and cost. Code and data: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.04433 [cs.AI] |
| (or arXiv:2610.04433v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.04433 arXiv-issued DOI via DataCite |
Submission history
From: Yangze Liu [view email]
[v1]
Sat, 3 Oct 2026 10:47:10 UTC (406 KB)
[v2]
Wed, 18 Nov 2026 13:24:25 UTC (857 KB)
来源:arXiv cs.AI · arxiv.org