自适应实验可能掩盖模型误差：检测极限与证伪预算
Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget
AI 导读
一篇 arXiv 论文指出，自适应实验设计会反复选择错误模型也能拟合的设置，从而掩盖模型误差。作者推导了任何有效性检验的检测能力上界，并提出保留部分设计预算用于证伪的免训练规则。校准后的修订版在 T = 240 时检出 35.8% 的误设实验（对照组为 31.7%），但 T = 480 时无确定优势。
正文 · 原文
Abstract:An experiment can be informative about the parameters of a working model and uninformative about whether that model is wrong. Adaptive design can reinforce this mismatch by repeatedly selecting settings where an incorrect model fits. We bound the power of any valid adequacy test in terms of the experimenter's own sampling distribution, then distinguish information absent from the selected data from information lost by the detector. A computable diagnostic measures unresolved error relative to a declared expansion of the working model. It motivates a training-free rule that reserves part of the design budget for falsification. The empirical record is mixed: the independently specified primary study fails detection eligibility, and a later numerator diagnostic fails its improvement criteria. After the frozen rule fails on fresh continuous-null panels, a disclosed secondary horizon-calibrated revision detects 35.8% of misspecified campaigns versus 31.7% for a class-separation control at T = 240, but has no resolved advantage at T = 480. In a corrected 289-unit measured replay it improves on random design and trails class-discrimination EIG. A conditional futility bound explains why remaining time matters without proving the gate constant or its power. These results support separating parameter learning, model checking and detector performance; they do not establish a universally superior design policy or prospective scientific discovery.
| Subjects: | Methodology (stat.ME); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07529 [stat.ME] |
| (or arXiv:2611.07529v1 [stat.ME] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07529 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ramnarayan Choudhary [view email]
[v1]
Thu, 5 Nov 2026 06:40:39 UTC (247 KB)
来源：arXiv cs.AI · arxiv.org