跳到正文
arXiv cs.AI· Mike He, Ankush Desai, Sharad Malik, Aarti Gupta·· 4 小时前AI 评分47

Soteria:用形式化验证规划与运行时强制机制保障 LLM 智能体安全

Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents

AI 导读

Soteria 框架通过形式化验证的分层规划与运行时规范强制,让 LLM 智能体同时满足任务效用与安全约束。执行前对结构化计划进行形式化验证,被拦截的不安全操作可依据已验证计划获得恢复指引。在多项工具使用基准上,Soteria 平均任务成功率达最佳基线的 1.7 倍,且无任何规范违规,该工作被 NeurIPS'26 接收为 Oral。

正文 · 原文

View PDF HTML (experimental)

Abstract:LLM agents that execute multi-step tool calls must satisfy two objectives simultaneously: completing the user's task (utility) and conforming to domain policies and correctness constraints (safety). These objectives are in tension -- blocking an unsafe action prevents a violation but can leave the agent stranded with no principled way to recover. Existing approaches sacrifice one objective for the other. We introduce Soteria, a framework that reconciles safety and utility through verified hierarchical planning and runtime enforcement of specifications. Before execution, the agent generates a structured plan that is formally verified against the specifications prior to any tool invocation. During execution, the verified plan serves dual roles: it ensures that the agent takes specification-conformant trajectories and provides guidance when unsafe actions are blocked. Across multiple benchmarks covering a diverse range of tool-use tasks, Soteria achieves 1.7 times the average task success rate of the best baselines, with no violations of policy specifications, demonstrating that safety and utility need not be traded off.
Comments: Accepted as NeurIPS'26 Oral
Subjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Cite as: arXiv:2611.17880 [cs.AI]
  (or arXiv:2611.17880v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.17880

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Deyuan He [view email]
[v1] Fri, 13 Nov 2026 08:14:04 UTC (584 KB)

来源:arXiv cs.AI · arxiv.org