arXiv cs.AI· Weitao Feng, Hengheng Xiong, Zihao Zeng, Jun Chen, Jiyang Guan, Xiaofeng Wang, Jialing Tao, Hui Xue, Wei Dong·· 3 小时前AI 评分57
V³AST-Bench：面向编码智能体的可验证、有效且多样的安全评测基准
V$^3$AST-Bench: Verifiable, Valid, and Versatile Safety Evaluation for Coding Agents
AI 导读
研究者提出 V³AST-Bench，一个面向智能体安全的基准，包含 212 个风险任务、18 个类别，通过运行时可执行检查验证危害与任务结果，并用成对能力孪生任务验证工作流能力。
正文 · 原文
Abstract:Large language models increasingly act through code, modifying repositories, executing programs, and interacting with services in agent form. Their actions introduce risks that cannot be assessed from generated responses alone. Safety evaluation must verify actual consequences, distinguish harm avoidance from task incapability, and cover realistic workflows. We introduce V$^3$AST-Bench, a benchmark for Verifiable results, Valid interpretation, and Versatile risk coverage of Agent Safety. It comprises 212 risk tasks spanning 18 categories. Dedicated runtime environments verify harm and task outcomes through executable checks over states, records, and artifact behavior, while paired capability twin tasks verify workflow competence. Our experiments span 51 combinations of 11 models and five harnesses, spanning 86,496 trials and over 9 billion tokens of model interaction. Results show that V$^3$AST-Bench remains challenging even for frontier coding agents, as Claude Opus 4.8 with Claude Code still causes verified harm on 20.9% of tasks. Across model-harness combinations, agents with comparable competence exhibit markedly different safety outcomes, showing that task capability alone does not guarantee safe execution. More importantly, even explicit recognition of risks does not reliably prevent harmful actions, revealing a critical gap between risk awareness and safe execution. These findings highlight the need to move beyond risk recognition toward enforcing safety throughout execution.
| Comments: | Code is available at this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35188 [cs.SE] |
| (or arXiv:2610.35188v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35188 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Weitao Feng [view email]
[v1]
Mon, 26 Oct 2026 10:44:18 UTC (316 KB)
来源：arXiv cs.AI · arxiv.org