arXiv cs.AI· Dario Deniz Ergun, Haohua Wang, Brendon Tao, Jonah Kollenberg, Chen Ye Gan·· 1 小时前AI 评分36
稀疏 AI 评测中的快速贝叶斯不确定性量化方法
Fast Bayesian Methods for Uncertainty Quantification in Sparse AI Evaluation
AI 导读
一种用于 AI 评测不确定性量化的快速贝叶斯方法,仅用公开评测计数构造弱 Beta 先验,并从至多 31 个已揭示结果中精确更新,无需物品语义、预训练权重或推理 API。
正文 · 原文
Abstract:Estimating how an AI system will perform on a new benchmark need not require another large model. We use public evaluation counts to choose a weak Beta prior, then update it exactly from at most 31 revealed outcomes. The predictor uses no item semantics, pretrained weights, or inference API. Its single online evaluation in the AIMS competition returned a team-reported Brier ALC of 0.188477, a 24.6% reduction from the constant-0.5 reference. The complete submission is 7,754 bytes; a server microbenchmark measures 12.71 microseconds per prediction with 31 matching labels. We describe how source evaluations are compressed into 301 prior means and show why a concentration of two lets new evidence rapidly determine the prediction. The resulting posterior gives exact uncertainty about the modelled success rate without numerical integration or neural inference. A public evaluation covers 79 system-benchmark pairs and eight seeds, reaching Brier ALC 0.200250 for the submitted method. Accompanying code and sufficient statistics reconstruct the fitted prior byte for byte and replay all 632 evaluation trajectories. The result demonstrates how much predictive value a compact Bayesian model can extract from sparse target feedback at very low online cost.
| Comments: | 13 pages, 5 figures. Accepted at NeurIPS 2026 AI Measurement Science Workshop |
| Subjects: | Artificial Intelligence (cs.AI); Computation (stat.CO) |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2611.24159 [cs.AI] |
| (or arXiv:2611.24159v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24159 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chen Ye Gan [view email]
[v1]
Wed, 18 Nov 2026 10:02:38 UTC (1,514 KB)
来源:arXiv cs.AI · arxiv.org