CalArena：大规模事后校准基准

View PDF HTML (experimental)

Abstract:Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibration provides a simple and widely used solution, but the large number of proposed methods, combined with small-scale and inconsistent evaluations, makes it difficult to determine which approaches are truly effective in practice. We introduce a large-scale, standardized benchmark for post-hoc calibration, covering nearly 2000 experiments across tabular and computer vision tasks, including binary, multiclass, and large-scale classification settings. Our benchmark aggregates predictions from a diverse set of classical models, modern deep learning architectures, and foundation models, and provides unified, reproducible implementations of dozens of calibration methods within a common evaluation framework. We argue that Post-Hoc Improvement (PHI) in proper scoring rules offers a principled alternative to traditional calibration error estimators for comparing post-hoc methods, capturing both calibration quality and potential degradation to the model's predictive performance. Using this framework, we conduct the most comprehensive empirical study of post-hoc calibration to date. Our results reveal consistent patterns across domains: smooth calibration functions outperform binning-based approaches, dedicated multiclass methods are essential in high-dimensional settings, and generic machine learning models are not competitive without calibration-specific design. To facilitate future research, we release all data, code, and evaluation tools, providing a plug-and-play benchmark for developing and comparing calibration methods.

Comments:	Accepted at NeurIPS 2026, Evaluations & Datasets track. Camera-ready version with a new ablation study on using log probabilities rather than logits, and a new multiclass post-hoc calibration method: Platt scaling on the logits (one-versus-rest)
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
Cite as:	arXiv:2605.30188 [cs.LG]
	(or arXiv:2605.30188v3 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.30188 arXiv-issued DOI via DataCite

Submission history

From: Eugène Berta [view email]
[v1] Thu, 28 May 2026 16:31:36 UTC (888 KB)
[v2] Mon, 1 Jun 2026 16:20:48 UTC (889 KB)
[v3] Tue, 27 Oct 2026 13:09:15 UTC (1,054 KB)