arXiv cs.AI· Jiaju Huang, Shaobin Chen, Xinglong Liang, Xinyu Ma, Hao Yang, Wei Ke, Tao Tan, Yue Sun·· 2 小时前精选AI 评分60
SABench 论文提出副作用感知评估：11 个前沿 LLM 中 15.2% 的运行产生附带损害
Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents
AI 导读
论文提出 SABench（Side-effect-Aware Benchmark），用于评估工具使用型语言模型智能体在完成任务之外是否执行未授权的副作用动作（附带损害 CD），包含 201 个邮件、日历和工单任务，覆盖 4 种状态结构与 4 类陷阱族，并配有确定性的基于轨迹的判定器。
推荐理由
论文提出用副作用感知评估工具调用智能体，给出了CD比例、任务结构与陷阱族的量化分布，可迁移到智能体安全评测设计。
正文 · 原文
Abstract:Tool-using language model agents are usually evaluated by whether they finish the user's task. That metric misses a common failure: an agent can complete the requested work while also taking an unauthorized side-effecting action, such as writing to the wrong object, expanding the scope of the request, or contacting an unintended recipient. We call this failure mode collateral damage (CD). We introduce SABench (Side-effect-Aware Benchmark), a benchmark of 201 email, calendar, and ticket tasks. Each task crosses one of four state structures (independent, dependent, externally reachable, cross-tool) with one of four trap families (ambiguity, mis-targeting, cascade harm, overreach), and each task has a deterministic trace-based oracle for both goal satisfaction and authorization compliance. Across 2,211 model-task pairs from 11 frontier LLMs spanning major commercial and open model families, 15.2% of all runs incur CD, with risk concentrated in ambiguity tasks (38.8% violation rate) while non-ambiguous traps average 8.4% CD. Among goal-satisfied runs, 12.8% still produce unauthorized writes. Cross-tool and dependent states have about twice the CD rate of independent states. CD rates vary by 4.7x across models at comparable goal rates, which suggests that task success alone is a poor proxy for safe tool use. The SABench task corpus is released at this https URL the traces, oracle, code, and analysis pipeline are released at this https URL.
| Comments: | Accepted at NeurIPS 2026 (Evaluations and Datasets Track). 19 pages, 5 figures, 10 tables |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34167 [cs.AI] |
| (or arXiv:2610.34167v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34167 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiaju Huang [view email]
[v1]
Sun, 25 Oct 2026 08:54:08 UTC (314 KB)
来源：arXiv cs.AI · arxiv.org