干预盲测检验 7,207 项注册临床试验模拟器的干预有效性
Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials
AI 导读
一项 arXiv 论文提出干预盲测方法，从 30,119 项完成的 2/3 期试验构建 9,996 个经审计的试验内对照，把患者世界模型的反事实能力变成可测量指标。
正文 · 原文
Abstract:Patient world models are increasingly proposed as simulators of clinical trials, and recent surveys rank their capabilities on a ladder from temporal prediction to counterfactual rollouts. Position on that ladder is currently assigned by reading papers rather than by measurement. We turn the counterfactual rung into a measured quantity. From 30,119 completed phase 2/3 trials with results posted to this http URL we construct 9,996 audited within-trial contrasts from 7,207 trials, and we score a simulator on how much of its accuracy depends on knowing which intervention was tested. The instrument is an intervention-blind reference: the same model, trained on the same trials, with every intervention-identifying field withheld. Interventional skill is the error the full model removes relative to that reference. On a temporal split (878 test contrasts from 655 trials) a registry-trained gradient-boosting simulator removes a fifth of the error of a trivial baseline, but its interventional skill is -0.12 (95% CI -0.21 to -0.05): withholding the intervention makes it better, and no alternative featurisation, stratum, temporal cut or same-molecule retrieval yields an interval above zero. A frontier language model reading only the registered protocol has interventional skill 0.20 [0.08, 0.31]; smaller and open-weight models sit at the limit of detection. Perturbation tests show the large model uses the intervention as such: swapping the arms reverses the sign of its prediction in 86% of contrasts. The blind reference is itself not drug-blind (a language model recovers the withheld drug class 45% of the time from the blind prompt), so interventional skill is a conservative lower bound. A reliability layer shows that ranking by predicted error lowers it but that calibrated error control does not transfer under temporal shift. Benchmark, code, prompts and per-call model outputs are released.
| Comments: | Oral presentation at the NeurIPS 2026 Workshop on World Models for High-Stakes Health. 31 pages, 5 figures. Code and data: this https URL |
| Subjects: | Applications (stat.AP); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.05907 [stat.AP] |
| (or arXiv:2611.05907v1 [stat.AP] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05907 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alicem Koyun [view email]
[v1]
Wed, 4 Nov 2026 03:59:15 UTC (263 KB)
来源：arXiv cs.AI · arxiv.org