EnvOS 论文提出审计计算机使用智能体的环境变化认证框架
EnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation
AI 导读
EnvOS 框架用于构建和认证任务中途发生变化的环境：智能体运行前检查变化触发、任务可解性和评分器有效性，运行后基于存档证据复核判定。实验显示 Claude Opus 5 在诊所预约任务无提示时 0/16，被提醒复查时 2/16，更新值直接展示时 15/16；保险理赔任务中 Opus 5 自发复查达 10/10。
正文 · 原文
Abstract:Computer-use agents built on large language models (LLMs) work in software that can change while they act: an appointment moves, a policy is revised, a record is updated. Testing agents under such changes is harder than it looks. When an agent fails, the cause may be the agent, but it may also be a change that did not fire as intended, an update the agent never received, or a grader that scored the episode wrongly. We introduce EnvOS, a framework for building and certifying environments that change mid-task. Before any agent runs, EnvOS checks that each change fires correctly, that the task remains solvable, and that the grader separates correct from incorrect behavior; after each run, separate code re-checks the verdict from archived evidence. We apply it to stale information, where a value the agent has already seen changes before the agent acts on it. In clinic booking, Claude Opus 5 succeeds on 0/16 tasks when nothing extra is shown, 2/16 when reminded to re-check the source, and 15/16 when the updated value is shown; the gain over the reminder holds after correcting for all tests (p = 0.00293). In insurance-claim approval, Opus 5 re-checks on its own and succeeds on 10/10, whereas Claude Sonnet 5 rises from 1/10 to 9/10 when the current policy is shown again, a gain significant in its pre-registered test but not after correcting for all tests. Whether an agent caught a change thus varied with the model and the interface. We also stress-test the grading itself: on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean, and on an externally generated set it caught 25 of 39, against 4 for simple log checks. Because every EnvOS environment is generated from a seed and graded by an audited automatic verifier, the same environments are natural candidates for training agents with reinforcement learning from verifiable rewards.
| Comments: | 74 pages, 7 figures. Accepted at NeurIPS 2026 Workshops (AABA4ET and CL4FMAgents) |
| Subjects: | Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2610.40246 [cs.AI] |
| (or arXiv:2610.40246v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40246 arXiv-issued DOI via DataCite |
Submission history
From: Ankush Kadu [view email]
[v1]
Thu, 29 Oct 2026 15:17:29 UTC (9,929 KB)
来源：arXiv cs.AI · arxiv.org