arXiv cs.AI· Zipeng Ye, Shenyu Dai, Yong Xie, Kun Wang, Jiyang Guan, Zhikai Chen, Jiaqi Weng, Jialing Tao, Hui Xue·· 1 小时前AI 评分41
ReA-Bench：面向智能体负责任审计的诊断型基准
ReA-Bench: Benchmarking Responsible Agent Auditing
AI 导读
ReA-Bench 是一个诊断智能体行为审计的基准，将审计挑战分为六个层级：内部安全知识、轨迹内矛盾、长上下文证据检索、来源与推理验证、间接引导和缺失信息。基准包含与任务一致但貌似有风险的良性案例，并提供成对轨迹版本以测试审计者在轨迹缩短或补全缺失信息后能否更可靠地识别相同风险。样本多来自真实生产轨迹，经人工筛选改编。
正文 · 原文
Abstract:As agents perform tasks autonomously in local and cloud environments, auditing their execution trajectories can help service providers identify risks and improve agent harnesses and foundation models. Our analysis of production and red-team trajectories shows that some risks are repeatedly missed across auditing configurations. Overall recall obscures which risks are missed and whether higher recall comes with more false positives on legitimate behavior. Existing benchmarks offer limited insight into these failures. Generated trajectories can contain inconsistent tool calls and responses, conspicuous risky actions, or repetitive attack patterns. Grouping results by risk type also makes it difficult to tell whether auditors overlooked evidence, accepted an unsupported conclusion, or lacked information needed to make a judgment.
We introduce ReA-Bench, a diagnostic benchmark for auditing responsible agent behavior. It organizes auditing challenges into six tiers: internal safety knowledge, contradictions within trajectories, evidence retrieval in long contexts, verification of sources and reasoning, indirect steering, and missing information. It includes challenging benign cases in which actions resemble risky ones but are consistent with the task and its authorization. It also provides paired trajectory versions to test whether auditors detect the same risks more reliably when the trajectory is shortened or missing information is supplied. Most samples are drawn from real production trajectories, with scenarios manually selected and adapted to preserve realistic tasks and risks that could plausibly arise during execution.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16493 [cs.AI] |
| (or arXiv:2611.16493v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16493 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zipeng Ye [view email]
[v1]
Thu, 12 Nov 2026 09:01:24 UTC (2,283 KB)
来源：arXiv cs.AI · arxiv.org