评测 Harness 到底修复了什么？一项预注册研究解析可见性、基线充分性与评测缺陷

View PDF HTML (experimental)

Abstract:Harness search keeps a change to the prompts, token budgets or parsers around a frozen model if the change raises a score. Such a gain can come from answers the parser could not read before, a weak comparison or an evaluation defect. We preregistered two studies of where these gains come from, with three models of 1.7B to 8B parameters and with Qwen3-14B, and report all 65 primary endpoints. Turning thinking off raised accuracy over a capped thinking setting in 7 of 12 model-benchmark cells, each time mostly through questions the capped setting left without a readable answer. For the small models, the thinking-off setting was not meaningfully worse than a rescue configuration in 7 of 9 cells, nor, for the 1.7B model, than four GEPA-selected harnesses. At 14B, two GEPA harnesses beat it by 30 to 34 points on the LongevityBench letter tasks; in descriptive analyses they were not detectably above the strongest constant-label baseline and answered "Yes" in 95% to 99% of their parsed mortality answers. From a start capped at 64 tokens, GEPA repaired the 1.7B model's harness, but at 14B no proposal of either search scored above zero. Replication through the same pipeline reproduced a defect's distortion in 9 of 20 evaluable cases, 7 of them cases in which the distortion is fixed by an identity of the design. Descriptively, relative to its base model, longevity fine-tuning raised a 1.7B model's letter accuracy by 41 points and cut its MMLU-Pro and GSM8K accuracy by 16 and 63 points; on a second GPU, 58% of its zero-temperature token sequences changed and none of the nine grid verdicts we rechecked did.

Comments:	62 pages, 2 figures, 52 tables. v2 adds a second preregistered study (Qwen3-14B; base vs longevity-tuned Qwen3-1.7B and a second-GPU rerun, descriptive) and corrects the Appendix J description of one v1 GEPA run. Ancillary files in anc/: both frozen preregistrations, their addenda (row keys of five rows withheld) and the aggregate reports, with a README
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:	arXiv:2610.05533 [cs.LG]
	(or arXiv:2610.05533v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.05533 arXiv-issued DOI via DataCite

Submission history

From: Bowen Xu [view email]
[v1] Sun, 4 Oct 2026 20:56:07 UTC (465 KB)
[v2] Sat, 24 Oct 2026 09:40:23 UTC (740 KB)