LLM 评测的评分规则敏感性：来自印度金融监管文本的证据
Scoring-Rule Sensitivity in LLM Evaluation: Evidence from Indian Financial Regulatory Text
AI 导读
一项 arXiv 研究用 406 题专家标注的 IndiaFinBench 基准（覆盖 SEBI 与 RBI 监管文本）对 12 个 LLM 在严格字符串匹配与跨模型裁判两种评分规则下进行零样本评测，发现两种规则下模型排名无正相关（Spearman rho = -0.273）。
正文 · 原文
Abstract:A benchmark leaderboard reports one accuracy number, but that number is a joint product of a model and a scoring rule. We ask what changes when only the rule changes, and find that model ordering is highly sensitive to it. IndiaFinBench is a 406-item expert-annotated benchmark over SEBI and RBI regulatory text covering regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning. We score twelve LLMs zero-shot under two primary regimes over the same predictions - strict four-stage string matching, and the verdicts of a cross-model judge sharing no model family with any evaluated system - plus one operational composite regime. The two primary regimes show no positive rank correspondence (Spearman rho = -0.273, p = 0.39, not significant), and no model holds its rank across all three regimes. Strict accuracy spans 75.12-89.66%; judge-only accuracy spans 87.19-94.83% with a different leader and trailer; the composite compresses the field to 94.1-98.0%, under which DeepSeek-R1-Distill-Llama-70B ties for first despite ranking last under strict scoring, because the judge reclassifies 93.9% of its strict errors as correct. Output verbosity does not explain the reclassification once task and model composition are adjusted for, a negative result we report as such. Item-level analysis compounds the finding: 148 of 406 items (36.5%) are answered identically by all twelve models and separate no pair of systems, while a predefined higher-discrimination band captures 134 items. We also anchor the scale against one non-specialist evaluator, who scored 80.0% (48/60; 95% Wilson CI [68.2, 88.2]) on a 60-item medium/hard subset; paired bootstrap tests find no significant difference against any model after multiplicity correction. Data, harness, judge verdicts, predictions: this https URL
| Comments: | 35 pages, 7 figures, 15 tables. Substantially revised from v2 (new title, scoring-rule analysis). Dataset, evaluation harness, and predictions at this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR) |
| MSC classes: | 68T50 |
| ACM classes: | I.2.7; I.2.6; H.3.1 |
| Cite as: | arXiv:2604.19298 [cs.CL] |
| (or arXiv:2604.19298v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.19298 arXiv-issued DOI via DataCite |
Submission history
From: Rajveer Singh Pall [view email]
[v1]
Tue, 21 Apr 2026 10:04:49 UTC (1,052 KB)
[v2]
Sat, 2 May 2026 14:28:22 UTC (1,314 KB)
[v3]
Mon, 26 Oct 2026 20:39:35 UTC (2,541 KB)
来源：arXiv cs.AI · arxiv.org