跳到正文
arXiv cs.AI· Jon-Paul Cacioli·· 4 小时前AI 评分43

解释前先筛查:面向 LLM 基准置信信号的可移植有效性筛查方法

Screen Before You Interpret: A Portable Validity Screen for Benchmark-Based LLM Confidence Signals

AI 导读

该论文借鉴临床人格评估中的反应有效性筛查传统,提出一种可移植的 LLM 置信信号有效性筛查:用高置信错误率(L)、低置信正确率(Fp)及其对比 RBS 检验信号是否随正确性变化,仅当可测量地追踪正确性时才判定为 Valid。

正文 · 原文

View PDF HTML (experimental)

Abstract:LLM confidence signals drive abstention, routing and escalation, yet evaluations rarely check whether a signal varies with correctness before computing metrics on it. Clinical personality assessment handles the analogous problem by screening response validity before interpreting substantive scales. We specify a portable screen in that tradition. From the correctness-by-confidence table it computes the rate of high confidence on errors (L), the rate of low confidence on correct answers (Fp), and their contrast (RBS, equal to minus Youden's J). It adds no new statistics: it separates blanket confidence, blanket withdrawal and inversion, attaches interval guards, and calls a signal Valid only when it measurably tracks correctness. On 20 frontier models answering 452 two-alternative items with binary KEEP/WITHDRAW probes, one model is Invalid (DeepSeek-R1 withdraws 95% of its correct answers), four are Indeterminate and 15 are Valid. Two Indeterminate models keep confidence on 96% of their errors but make too few errors to certify it; certification needs 35 to 139 errors. Only 6 of the 20 verdicts reproduce reliably on half the items. On verbalized 0-100 confidence (33 models on 1,500 MMLU items, 20 on a second 500-item sample, and Yang et al.'s data for 11 models on 10 benchmarks), the verdict depends on the cut point: at 50 most models are Invalid because answers pile at the ceiling, while at 90-95 most pass. Battery item sensitivity does not predict MMLU discrimination (rho = -.07). Protocol amendments were settled on these data, so the verdicts are descriptive pending a held-out test. This version corrects scoring, grading, classification and reporting errors in version 1 and supersedes arXiv:2604.17707 and 2604.17716.
Comments: v2: Corrected and consolidated; supersedes arXiv:2604.17707 and 2604.17716. Fixes errors in v1: declines were scored as confident errors, a free-text track was mis-graded, tiers did not come from the released tool, and the Yang et al. reanalysis was not computed from their data. 16 pages, 4 figures. Code and data: this https URL
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2604.17714 [cs.CL]
  (or arXiv:2604.17714v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2604.17714

arXiv-issued DOI via DataCite

Submission history

From: Jon-Paul Cacioli [view email]
[v1] Mon, 20 Apr 2026 01:50:38 UTC (883 KB)
[v2] Fri, 13 Nov 2026 04:35:49 UTC (88 KB)

来源:arXiv cs.AI · arxiv.org