arXiv cs.AI· Russell Yang, Ruishi Chen, Pierce Kelaita, Riya Ranjan, Sibo Ma, Charles Dickens, Matthew Guillod, Megan Ma, Julian Nyarko·· 4 小时前AI 评分55
JudgmentBench：比较量表评分与偏好比较两种评测方法
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
AI 导读
Stanford 等机构作者发布 JudgmentBench，含 30 项真实法律任务、1,539 条量表评分和 1,530 条成对偏好判断，均由资深执业律师标注。初步实验显示成对比较恢复质量排序的能力明显更好，per-task Spearman 相关系数 0.908 对 0.150，且标注时间不到一半，模式在人类标注者和 LLM 自动评分者中均成立。
正文 · 原文
Abstract:Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench, a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys--including at major U.S. law firms--with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same pool of experts on the same tasks. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics under both a per-task rank-correlation metric (mean Spearman's rank correlation of 0.908 vs. 0.150, estimated difference = 0.758 [0.494, 1.021]) and a per-judgment pairwise win-rate metric (0.669 vs. 0.542, estimated difference = 0.127 [0.067, 0.186]), while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth.
| Comments: | Accepted at NeurIPS 2026 (Evaluations and Datasets Track). 50 pages, 9 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2605.25240 [cs.CL] |
| (or arXiv:2605.25240v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.25240 arXiv-issued DOI via DataCite |
Submission history
From: Russell Yang [view email]
[v1]
Sun, 24 May 2026 19:52:39 UTC (3,637 KB)
[v2]
Tue, 2 Jun 2026 02:06:08 UTC (3,776 KB)
[v3]
Thu, 4 Jun 2026 00:43:30 UTC (3,777 KB)
[v4]
Thu, 5 Nov 2026 06:46:06 UTC (3,676 KB)
来源：arXiv cs.AI · arxiv.org