BELLS-O 基准评测 33 个 LLM 监督系统的运营权衡
BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems
AI 导读
BELLS-O 基准评测来自 17 家开发商的 33 个 LLM 监督系统，覆盖内容审核、越狱检测和提示词注入检测，统一衡量检测率、误报率、延迟和成本。结果显示专用防护在内容审核上以约 5 倍速度追平最强通用模型（检测率 93-96%），通用 LLM 在越狱检测上中位检测率更高但闭源前沿模型成本高 10-100 倍，提示词注入则由两个最佳专用检测器领先。团队发布了框架、交互排行榜和四个数据集。
正文 · 原文
Abstract:LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often vendor-biased, omit cost and latency, and rarely compare specialized guardrails against repurposed generalist LLMs. We present BELLS-O (Benchmark for the Evaluation of LLM Supervision Systems, Operational), an independent operational benchmark that evaluates 33 systems from 17 developers: specialized guardrails (e.g., Llama Guard 4, ShieldGemma-2B, Lakera Guard) and frontier generalists repurposed as supervisors (e.g., GPT-5.4, Claude Sonnet 4.6, Grok 4.1 Fast), jointly on detection rate, false-positive rate, latency, and cost. We cover input/output moderation across 11 harm categories, jailbreak detection across 13 attack techniques, and prompt-injection detection, using in-house datasets built from handcrafted prompts, curated external samples, and quality-controlled synthetic generation, with synthetic moderation samples paraphrased to suppress generator fingerprints. Content-moderation runs are repeated five times and all rates reported with 95% intervals. The Pareto frontier reveals use-case-dependent tradeoffs. On content moderation, specialized supervisors are operationally ahead: top systems match the best generalist LLMs on detection (93-96%, no significant difference) at low false-positive rates while running ~5x faster; cost depends more on serving setup than on class. On jailbreak detection, generalist LLMs achieve higher median detection on three of four datasets and lower false-positive rates, but at higher latency and, for closed frontier models, 10-100x higher cost. On prompt injection, the two best dedicated detectors lead. We release the framework, an interactive leaderboard, and four datasets as a vendor-neutral basis for selecting safeguards under deployment constraints.
| Comments: | Accepted at NeurIPS 2026 (Track on Evaluations and Datasets). Previously presented at the ICML 2026 Workshop on Trustworthy AI for Good (AI4GOOD). 60 pages, 6 figures; 10-page main text plus appendices |
| Subjects: | Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.20668 [cs.CR] |
| (or arXiv:2606.20668v2 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2606.20668 arXiv-issued DOI via DataCite |
Submission history
From: Hadrien Mariaccia [view email]
[v1]
Fri, 12 Jun 2026 07:25:12 UTC (690 KB)
[v2]
Thu, 12 Nov 2026 09:59:32 UTC (272 KB)
来源：arXiv cs.AI · arxiv.org