跳到正文
arXiv cs.AI· Yuxuan Zhou, Chen Xiong, Haitong Liu, Kuofeng Gao, Yang Bai, Zhiyuan He, Tao Dai, Shu-Tao Xia·· 2 小时前精选AI 评分61

SEAM 基准揭示计算机使用智能体跨屏组合执行有害任务

SEAM: Computer-Use Agents Know the Harm Between Screens, and Commit It Anyway

AI 导读

研究者发布 SEAM(Safety Execution Across Multiple screens)基准,包含 600 个 OSWorld 多步跨应用任务,指令与每屏观察均无害、仅组合后有害,并经服务器日志而非 LLM 裁判打分。

推荐理由

基准把风险藏在跨屏信息组合里,测出识别安全与执行安全的脱节,给评估多屏智能体提供了可复用方法。

正文 · 原文

View PDF HTML (experimental)

Abstract:Safety benchmarks for computer-use agents (CUAs) place the harm either in the intent of the task or on a single screen: a malicious instruction, an injected command, a phishing page. An agent that checks the task and each screen in isolation can pass them, so they do not test whether it combines facts seen on different screens. Yet in delegated work, harm often arises exactly this way: a fact about a person on one screen, a fact about an item on another, and a legitimate task that binds them. We introduce SEAM (Safety Execution Across Multiple screens), a benchmark of such cross-screen compositional harm: 600 multi-step, cross-application OSWorld tasks in which the instruction and every single observation are benign and only their combination is harmful. Minimal-pair variants that differ in one place, scored from the server log rather than by an LLM judge, locate where an agent fails. General-purpose open-weight and flagship agents carry out the harm in up to 92% of runs, yet decline when the user states it in one sentence. Agents open the decisive pages, so exposure is not the primary bottleneck. Asked about those pages, they flag the harm, and with thinking they often write the conflict into their reasoning and confirm anyway. Safety recognition thus does not reliably translate into safe execution, and the translation depends on the information channel: the same warning stops agents far more often in the user's instruction than on screen. Proprietary models are markedly more cautious. We release SEAM with its generator and quality gates at this https URL .
Comments: 30 pages, 14 figures, 29 tables
Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
ACM classes: I.2.7; I.2.10
Cite as: arXiv:2611.22971 [cs.CR]
  (or arXiv:2611.22971v1 [cs.CR] for this version)
  https://doi.org/10.48550/arXiv.2611.22971

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yuxuan Zhou [view email]
[v1] Tue, 17 Nov 2026 15:53:58 UTC (915 KB)

来源:arXiv cs.AI · arxiv.org