Veritas：面向智能体任务的验证器生成框架
Veritas: A Verifier Generation Harness for Agent Tasks
AI 导读
Veritas 是一个自动生成验证器的框架，通过证据头、领域规则库和创作原则三项机制，解决人工编写验证器成本高、难扩展的问题。在固定模型条件下，其生成的验证器在全部四个执行评分基准上比其他七个框架更贴近官方判定，在两个成对比较基准上达到或接近最优，相关机制还在跨三个框架的 12 组执行评分组合中改进了 9 组。
正文 · 原文
Abstract:The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates verifiers automatically. Unlike general-purpose agent harnesses, Veritas adds three mechanisms for verifier generation: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles, partly refined by a self-improvement loop. We evaluate on six benchmarks: four scored by execution, whose official verifiers we first certify, and two that compare pairs of responses. With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks. On the two pairwise benchmarks they are at or near the best. They also agree better than those of the vendor's own harness for two of three other model families. Its mechanisms also improve 9 of 12 harness-benchmark pairs scored by execution across three harnesses.
| Comments: | 31 pages, 1 figure |
| Subjects: | Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2611.13336 [cs.AI] |
| (or arXiv:2611.13336v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13336 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qinbo Bai [view email]
[v1]
Mon, 9 Nov 2026 23:01:40 UTC (270 KB)
来源：arXiv cs.AI · arxiv.org