跳到正文
arXiv cs.AI· Markus D\"ucker, Vaibhav Kumar, Yi Liu, Ronak Chaudhary, Andreas Plesner, Francisco Guzm\'an, Anish Athalye·· 2 小时前AI 评分49

Gandalf:用于 Rubric 评分智能体环境的开源验证器研究

Verifying Agents in Rubric-Graded Environments

AI 导读

研究者发布首个针对 rubric 评分智能体环境的验证器系统研究,提出 2,464 条人工评判 rubric 标准、24 个投行任务的 BankerVerifierBench(BVB)数据集,并开源验证器 Gandalf。

正文 · 原文

View PDF

Abstract:As AI agents take on increasingly open-ended and complex tasks, verifying their outputs has proven to be difficult. Rubric-graded agent environments, where a verifier judges multiple natural-language criteria against the deliverables and environment state changes produced by the agent, have emerged as a popular paradigm. We conduct the first systematic study of verifiers for rubric-graded agent environments. First, we create BankerVerifierBench (BVB), a meta-evaluation dataset of 2,464 human-judged rubric criteria across 24 investment-banking tasks. Next, we develop a methodology that derives from any rubric corpus the capabilities a verifier should possess, yielding a nine-capability taxonomy that we distill into three general design principles---reactive verification, environment alignment, and domain guidance---which we implement in Gandalf, an open-source verifier. Finally, we evaluate Gandalf against three existing verifiers on BVB. Gandalf matches or exceeds every baseline's F1 at lower cost: with Gemini 3 Flash it matches the strongest baseline's F1 (0.864 vs. 0.852) at one-fifth of its cost, and with GPT-5.4 models it improves F1 by over 0.12 at lower cost. Gandalf also generalizes to OpenClaw, a structurally different personal-productivity environment, where it again achieves the highest F1 (0.955 vs. 0.934 for the best baseline). We release BVB, Gandalf, and the code and data to reproduce our experiments.
Comments: Presented at NeurIPS E&D 2026
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22841 [cs.AI]
  (or arXiv:2611.22841v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22841

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Andreas Plesner [view email]
[v1] Tue, 17 Nov 2026 14:21:38 UTC (59 KB)

来源:arXiv cs.AI · arxiv.org