GauntletScore:可审计的事实核查引擎,在 LLM-AggreFact 上比肩最佳已发布核查模型
GauntletScore: Auditable Fact-Checking That Matches the Best Published Checkers on LLM-AggreFact
AI 导读
GauntletScore 是一个可审计的证据引擎,通过向不同提供商的独立读者提出类型化问题并以固定贝叶斯规则汇总测量记录,核查 AI 生成文本是否有来源支持。
正文 · 原文
Abstract:Checking whether a generated sentence is supported by its source document is a standard safeguard for AI-written text, and the strongest published checkers are single models. We describe GauntletScore, an evidence engine whose few rules and thresholds were set on the benchmark's development split. It puts typed questions to independent readers from different providers and records each answer as a measurement in an evidence ledger. It combines the measurements with a fixed Bayesian rule that counts a contradiction only when two readers agree. On the full LLM-AggreFact test split (29,320 claims across 11 datasets), following the leaderboard's approach, GauntletScore achieves 79.51 balanced accuracy, above every average published on the leaderboard as of 15 November 2026. That places it level with the best of them, nimble-v4-max (79.45 from its published dataset scores; the engine's 95% interval, 78.47 to 80.53, contains it). On the same claims it beats two of its three readers used alone (Readers A and B) by 1.66 and 2.17 points (95% intervals 0.80 to 2.53 and 1.29 to 3.06). Without AggreFact-CNN, described below, the leads are 0.90 and 1.32 and both intervals stay above zero. The engine also records the evidence behind every verdict, which a single model does not. Its largest margin is on full-length news articles checked against multi-sentence summaries (AggreFact-CNN), a format that resembles much regulated writing. There it scores 76.45, the best on the leaderboard, against 70.1 for the next best and 63.9 to 67.8 for the six systems with the highest averages. The test split was run once, on an engine frozen before the first test claim, under a stopping plan committed in advance. We release per-claim predictions and the scoring script.
| Comments: | 10 pages, 1 figure, 7 tables. Per-claim predictions and scoring script: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22027 [cs.CL] |
| (or arXiv:2611.22027v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22027 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ernie Hobbs [view email]
[v1]
Tue, 17 Nov 2026 00:38:54 UTC (120 KB)
来源:arXiv cs.AI · arxiv.org