arXiv 论文揭示 CoT 监控器池化准确率掩盖推理依赖脆弱性
A False Average: Pooled CoT-Monitor Accuracy Conceals a Reasoning-Dependent Fragility
AI 导读
研究者提出 CoT 监控器的池化准确率是虚假平均：Terminal Wrench 上约 77% 的 reward hack 仅凭动作即可发现，其余 23% 子集上监控器很脆弱。仅改写推理文本为善意工程措辞、保持命令逐字节不变，可把留出监控器在该子集的捕获率从约 95% 降到 4–11%，仅轨迹内的防御收效甚微，只有轨迹外信息帮助明显；作者建议按推理依赖子集报告监控器准确率。
正文 · 原文
Abstract:Chain-of-thought (CoT) monitors are reported by their aggregate accuracy on a pool of reward hacks. We show that this number is a false average. On Terminal Wrench, about 77% of hacks are given away by the actions alone, and the monitor's pooled accuracy is dominated by them; on the remaining 23%, which an action-only monitor misses, the same monitor is fragile. We expose the fragility with a controlled attack: we rewrite only the agent's reasoning to read as good-faith engineering, leaving every command and output byte-identical. One gradient-free rewrite drops a held-out monitor's catch rate on that subset from about 95% to between 4 and 11%, while the pooled rate falls only about 25 points, near what the subset's share of the pool times its collapse predicts. The rewrite stays truthful about what happened and lies only about intent, so the trace-only monitors we test recover little, even when warned about the attack; of our defenses, only information from outside the trace helps much. The attack transfers across monitor and agent families, and the dataset's own hack labels show the same gradient with no monitor in the definition. Live agents told to hack on 27 tasks, with no monitor feedback, produce hollow hacks, but a monitor calibrated on each model's own honest solves catches nearly all of them. Applying the same rewrite to these live hacks keeps all 111 action-visible ones caught and lowers catch on the 16 whose actions score below 0.5 only from 16 to 13: the effect lands where the false average predicts, but the collapse stays a controlled result. A linear probe on a surrogate monitor's activations out-reads its verdict on the obfuscated hacks (AUC up to 0.98), but a behavioral control shows this edge does not, by itself, mean the monitor knows more than it says. Monitor accuracy should be reported on the reasoning-dependent subset, where the defense is load-bearing.
| Comments: | Accepted at the NeurIPS 2026 Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD). Code and data: this https URL |
| Subjects: | Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.00583 [cs.CR] |
| (or arXiv:2608.00583v4 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2608.00583 arXiv-issued DOI via DataCite |
Submission history
From: Shikhar Shiromani [view email]
[v1]
Sat, 1 Aug 2026 10:42:41 UTC (41 KB)
[v2]
Sun, 6 Sep 2026 09:15:06 UTC (1 KB) (withdrawn)
[v3]
Fri, 11 Sep 2026 04:40:20 UTC (45 KB)
[v4]
Mon, 26 Oct 2026 01:55:30 UTC (59 KB)
来源：arXiv cs.AI · arxiv.org