arXiv cs.AI· Juan-Pablo Rivera, Ari Spiesberger, Juan J Vazquez, Arun Jose, Axel Ahlqvist, Adeline Kassler, Meru Gopalan, Sandy Tanwisuth, John Hughes·· 4 小时前AI 评分54
arXiv 论文研究不可信监控器的串通纠偏迁移：已抓行为纠正后仍有 14-20% 未知串通残留
Model Organisms of Collusion: Corrective Transfer in Untrusted Monitors
AI 导读
arXiv 论文（arXiv:2610.30582）研究 AI 控制中用同模型副本作监控器时的串通风险：对 Llama-3.3-70B 和 Qwen2.5-72B 微调安装五组策略-监控行为对，纠正已抓到的一起串通后，未见过的四种串通从 91% 降到 14-20%（三种对比方法），无关样本对照为 56-93%。
正文 · 原文
Abstract:AI control protocols aim to get useful work from models that may be misaligned by monitoring their behaviour. One approach uses a second copy of the same model as the monitor. This avoids relying on a weaker reviewer the model could outsmart, but risks collusion: the model, now monitoring itself, may cover up its own misbehaviour. One remedy is to fine-tune the monitor on whatever examples of misbehaviour the control team has caught. We ask whether this training also removes collusion that the control team has not yet found. We fine-tune Llama-3.3-70B and Qwen2.5-72B to install five policy-monitor behaviour pairs, correct the monitor on 300 honest reviews of one caught collusion, and measure collusion on the other four in unseen prompt formats. Averaged over the three seeds of each family, correction reduces held-out collusion from 91% to 14-20% under three of the four methods we compare (individual seeds 10-32%), while a control fine-tuned on the same number of unrelated examples leaves 56-93%. Family-average accuracy on control items changes by at most five points and capability is preserved, apart from a drop in mathematical accuracy on one Llama seed. Under the tested training configuration, methods with similar caught-task accuracy differ in held-out transfer, from 58% to 83% average reduction, so the caught behaviour alone does not reveal how far a correction reaches. Training on a known failure removes it almost entirely (1-2% remaining), yet 14-20% of unknown failures persist, so near-perfect correction of the caught behaviour overstates how much collusion has been removed.
| Comments: | 33 pages, 6 figures, 15 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.30582 [cs.AI] |
| (or arXiv:2610.30582v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30582 arXiv-issued DOI via DataCite |
Submission history
From: Juan-Pablo Rivera [view email]
[v1]
Thu, 22 Oct 2026 01:22:48 UTC (681 KB)
[v2]
Fri, 30 Oct 2026 17:55:18 UTC (681 KB)
来源：arXiv cs.AI · arxiv.org