AGORA：基于项目反应理论的 LLM 评委可靠性感知聚合方法
AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory
AI 导读
arXiv 论文提出 AGORA（Aggregation Of Reliability-weighted Assessments），一种基于项目反应理论的 LLM 评委聚合方法，将所有条目置于同一潜在质量量表上，用敏感度参数和有序评分阈值刻画每个评委，无需人工标签即可从全部条目联合学习评委可靠性。
正文 · 原文
Abstract:LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05556 [cs.AI] |
| (or arXiv:2611.05556v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05556 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junhyung Ahn [view email]
[v1]
Tue, 3 Nov 2026 19:49:01 UTC (131 KB)
来源：arXiv cs.AI · arxiv.org