Rubricon：用于 LLM 评估与训练的自动化评分标准生成与优化
Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training
AI 导读
Rubricon 是一个无需人工偏好标签、rubric 标注或生成器模型训练的自动化 rubric 生成流程，利用任意 LLM 采样多个 rubric，筛选最具区分度的候选并迭代增删评估标准。
正文 · 原文
Abstract:Rubrics enable reward assignment on tasks without verifiable answers, but are expensive to manually produce from scratch. We present Rubricon, a pipeline that generates discriminative rubrics without requiring human preference labels, rubric annotation, or generator-model training. Using any LLM, Rubricon samples multiple rubrics, selects the most discriminative candidate, adds criteria to capture overlooked requirements or response flaws, and prunes criteria that are misaligned or shallow. Alongside LLM self-critique and test-time scaling, Rubricon uses observed criterion pass rates on example model responses to guide refinement. Across five professional, consumer, clinical, and general-use LLM benchmarks, Rubricon produces rubrics whose scores of frontier LLM responses exhibit 9% greater alignment with labor-intensive expert-authored rubrics, compared to the best alternative synthetic rubric generation method among RRD, QSRG, OpenRubrics, and agentic generation via harnesses like Codex. On HealthBench, post-training a Qwen3.5-9B model with Rubricon rubrics yields 4.5% better performance than training with rubrics from the strongest alternative synthetic rubric generator, RRD.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38220 [cs.AI] |
| (or arXiv:2610.38220v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38220 arXiv-issued DOI via DataCite |
Submission history
From: Ulyana Tkachenko [view email]
[v1]
Wed, 28 Oct 2026 03:18:11 UTC (471 KB)
来源：arXiv cs.AI · arxiv.org