arXiv cs.AI· Guojun Zhu, Xunheng Huang, Peng Yin, Jiahui Xie, Sanguo Zhang, Doudou Zhou·· 2 小时前AI 评分50
CHASE 论文提出反事实引导的智能体 harness 进化方法
Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
AI 导读
论文提出 Counterfactual Harness Search and Evolution(CHASE),通过 Proposer 修改 harness、Challenger 搜索保持任务语义的协议变换,检验 harness 改进是否依赖基准捷径,并在理想无捷径基准上给出增益损失上界的理论条件。
正文 · 原文
Abstract:An optimized agent harness can improve on held-out tasks by relying on formatting conventions that those tasks share with the rest of the benchmark. We introduce Counterfactual Harness Search and Evolution (CHASE) to test whether harness improvements survive changes to the evaluation protocol. A Proposer edits the harness around a fixed foundation model. A Challenger then searches for a protocol change that preserves the tasks but reduces the updated harness's gain over the initial harness. Executable checks test whether the transformation preserves task semantics. If the gain loss is confirmed on held-out tasks, CHASE adds the transformation to an archive that constrains subsequent harness updates. We give conditions under which this finite archive bounds the gain loss on an ideal benchmark with shortcuts removed. We evaluate CHASE on Syn-Ledger, OfficeQA, FinanceBench, and SWE-Bench Pro V2. On Syn-Ledger, removing the known shortcuts yields 38.30% accuracy for CHASE and 18.27% for HarnessEvolve. On the 21 certification tasks from SWE-Bench Pro V2 HARD-51, changing the issue formatting and tool-output field order reduces HarnessEvolve's gain over the initial harness from 8.57 to 3.81 percentage points. The final CHASE harness retains 10.48 points of its 11.43-point gain under the same change.
| Comments: | 42 pages, 6 figures, including references and supplementary material |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2609.18366 [cs.AI] |
| (or arXiv:2609.18366v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.18366 arXiv-issued DOI via DataCite |
Submission history
From: Guojun Zhu [view email]
[v1]
Wed, 16 Sep 2026 09:21:33 UTC (509 KB)
[v2]
Fri, 18 Sep 2026 06:03:43 UTC (513 KB)
[v3]
Thu, 24 Sep 2026 07:59:20 UTC (521 KB)
[v4]
Wed, 18 Nov 2026 05:17:43 UTC (704 KB)
来源:arXiv cs.AI · arxiv.org