受治理的置信度分数能判断 Text-to-SQL 答案的对错吗？基于 BIRD 的探索性研究与 Spider 上的预注册验证
Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider
AI 导读
一项 arXiv 研究审计了某生产级 Text-to-SQL 系统的置信度分数设计：在 BIRD dev 的 1,534 个问题上，GPT-5.1 每题采样五次，该分数区分对错的 AUROC 仅 0.624，且会拦截 35.7% 的正确答案。
正文 · 原文
Abstract:Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score to every answer it writes over a database and uses it to show the answer, show it with a warning, or withhold it. The score's weights, caps and thresholds were set by design. This paper audits that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624) and withholds 35.7% of correct answers. Nearly all of its ranking signal comes from agreement among the samples, and most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including commentary. Before any Spider answer existed we froze and hashed a rule in which agreement with the shown answer is the score, with deterministic vetoes and deterministic and judge-based caps. On Spider dev AUROC rose from 0.566 to 0.725 (+0.159, 95% CI +0.097 to +0.221; database-level CI +0.098 to +0.234), with PASS accuracy 95.7% against a 92.6% base rate. The frozen rule was not the best agreement-based rule on Spider, and the harness and prompts were rebuilt after registration. Added after review: a GPT-5.1 judge asked for P(correct), averaged with agreement, ranked higher in every evaluation (0.837 on Spider), but it grades its own model's answers and its gain shrinks under execution-match labels; one answer per question from a second model family raised agreement's AUROC from 0.749 to 0.800 on Spider, a gain that held under both label definitions. Labels are model-judged; a ten-answer human pilot found no disagreement, and a full audit has not been done.
| Comments: | 26 pages, 4 figures, 12 tables. v2: revised after external review; adds a correctness judge, a second-model check, database-level intervals and a human label pilot, and corrects the reproducibility statement. Code, frozen scorer (branch prereg), analysis scripts and data: this https URL (release v2.0) |
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | H.2.3; I.2.7 |
| Cite as: | arXiv:2610.17256 [cs.AI] |
| (or arXiv:2610.17256v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17256 arXiv-issued DOI via DataCite |
Submission history
From: Raghu Venkat [view email]
[v1]
Mon, 12 Oct 2026 18:14:43 UTC (35 KB)
[v2]
Fri, 30 Oct 2026 15:42:01 UTC (43 KB)
来源：arXiv cs.AI · arxiv.org