arXiv cs.AI· Markus D\"ucker, Vaibhav Kumar, Yi Liu, Ronak Chaudhary, Andreas Plesner, Francisco Guzm\'an, Anish Athalye·· 2 小时前AI 评分49
Gandalf:用于 Rubric 评分智能体环境的开源验证器研究
Verifying Agents in Rubric-Graded Environments
AI 导读
研究者发布首个针对 rubric 评分智能体环境的验证器系统研究,提出 2,464 条人工评判 rubric 标准、24 个投行任务的 BankerVerifierBench(BVB)数据集,并开源验证器 Gandalf。
正文 · 原文
Abstract:As AI agents take on increasingly open-ended and complex tasks, verifying their outputs has proven to be difficult. Rubric-graded agent environments, where a verifier judges multiple natural-language criteria against the deliverables and environment state changes produced by the agent, have emerged as a popular paradigm. We conduct the first systematic study of verifiers for rubric-graded agent environments. First, we create BankerVerifierBench (BVB), a meta-evaluation dataset of 2,464 human-judged rubric criteria across 24 investment-banking tasks. Next, we develop a methodology that derives from any rubric corpus the capabilities a verifier should possess, yielding a nine-capability taxonomy that we distill into three general design principles---reactive verification, environment alignment, and domain guidance---which we implement in Gandalf, an open-source verifier. Finally, we evaluate Gandalf against three existing verifiers on BVB. Gandalf matches or exceeds every baseline's F1 at lower cost: with Gemini 3 Flash it matches the strongest baseline's F1 (0.864 vs. 0.852) at one-fifth of its cost, and with GPT-5.4 models it improves F1 by over 0.12 at lower cost. Gandalf also generalizes to OpenClaw, a structurally different personal-productivity environment, where it again achieves the highest F1 (0.955 vs. 0.934 for the best baseline). We release BVB, Gandalf, and the code and data to reproduce our experiments.
| Comments: | Presented at NeurIPS E&D 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22841 [cs.AI] |
| (or arXiv:2611.22841v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22841 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Andreas Plesner [view email]
[v1]
Tue, 17 Nov 2026 14:21:38 UTC (59 KB)
来源:arXiv cs.AI · arxiv.org