LLM 置信度校准研究：同样答案因测量协议不同而置信度迥异
Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibration
AI 导读
一项 arXiv 研究考察 LLM 答案置信度的测量协议敏感性，在 5 个开源大语言模型和 4 个问答数据集上发现，改变读取 token 概率的上下文会使口头化置信度与 token 置信度的差距发生约等于差距本身大小的偏移（ECE .287 对 .304，AUROC .189 对 .145）。
正文 · 原文
Abstract:Large language models (LLM) can state their confidence in an answer (verbalized confidence), or the confidence can be read from the probability of the answer's tokens (token confidence). Comparisons of the two ask which is better calibrated, but the token probability of an answer depends on the text that precedes it. Across five open LLMs and four question-answering datasets, we fix each answer, its verbalized confidence, and its correctness label, and vary how each score is measured. The scores themselves change with the protocol, and so does the comparison between them. Changing the context in which the token probability is read shifts the gap between the two scores by about the size of the gap itself (.287 vs. .304 in expected calibration error, ECE; .189 vs. .145 in AUROC). The 95% interval of the shift excludes zero in 19 and 17 of the 20 settings, and under AUROC the shift has the same sign in all of them, so the context changes how answers are ranked. The better score flips in 8 settings under each metric, counting only flips that the intervals support, and the shift also appears in four larger models of up to 72B parameters. Most of the shift appears with changes to the instruction, not to the position of the answer. Other measurement choices also matter, and stated confidence depends on how a supplied answer is written. We turn these results into a reporting checklist for LLM confidence and its comparisons.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.27752 [cs.AI] |
| (or arXiv:2605.27752v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.27752 arXiv-issued DOI via DataCite |
Submission history
From: Hankyeol Kim [view email]
[v1]
Tue, 26 May 2026 23:03:38 UTC (60 KB)
[v2]
Mon, 1 Jun 2026 13:50:34 UTC (60 KB)
[v3]
Fri, 7 Aug 2026 08:03:29 UTC (437 KB)
[v4]
Fri, 23 Oct 2026 14:58:08 UTC (453 KB)
来源：arXiv cs.AI · arxiv.org