arXiv cs.AI· \'I\~nigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro, Pablo Villanueva-Domingo, Miles Cranmer·· 3 小时前AI 评分55
审计 AI 同行评审：在真实科学论文上构建审稿基准并推出 Skepthical 多智能体审稿人
Auditing AI peer reviewers: a benchmark on real scientific papers
AI 导读
研究者在 30 篇 arXiv 预印本（天体物理与机器学习各 15 篇）中注入共 357 个客观错误，评测十二个审稿系统对错误的检出率与误报率，并发布多智能体审稿人 Skepthical 和完整基准。
正文 · 原文
Abstract:LLM-based peer review is now used at production scale, but its evaluation remains limited. We introduce a benchmark that injects objective, localisable errors of four types (general issues, mathematics, numerics, citations) at three severity levels into real papers and measures each reviewer's detection rate by error type and severity, together with its false-positive rate on clean copies. We also present Skepthical, a multi-agent reviewer that verifies citation-backed claims by retrieving the cited papers, mathematics with computer algebra, and numerical claims with executed code. We apply the benchmark to two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning (ML), injecting a total of 357 errors, and evaluate twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available reviewers, and Skepthical. The choice of model has the largest impact on detection rate: given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora. Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves on a single Claude call in mathematics and numerics detection, and enforced citation retrieval gives Skepthical the highest citation error detection rate of all systems by a large margin. This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical's citation detection rate is statistically indistinguishable from that of the best single LLM. False-positive rates are below 10% for most systems, but vary with prompt, model, and scaffolding, with mathematics flags the main source of false positives. The evaluation pipeline, the injected errors, and the adjudication logs are released as a benchmark.
| Comments: | Accepted at NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Instrumentation and Methods for Astrophysics (astro-ph.IM); Artificial Intelligence (cs.AI); Digital Libraries (cs.DL) |
| Cite as: | arXiv:2611.17052 [cs.CL] |
| (or arXiv:2611.17052v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17052 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Íñigo Zubeldia [view email]
[v1]
Thu, 12 Nov 2026 16:33:51 UTC (986 KB)
来源：arXiv cs.AI · arxiv.org