Messier 论文提出覆盖 30 个基准的跨基准智能体评测语料库

View PDF HTML (experimental)

Abstract:Comprehensively evaluating AI agents across interactive environments is difficult due to fragmented tasks, scaffolds, verifiers, and scoring rules. Unfortunately, existing efforts to unify these evaluations are limited in scale and domain, making costly reruns necessary and leaving available data incomparable. We introduce Messier, a unified corpus of 923,158 records spanning 30 benchmarks, 745 agents, 11,891 tasks, and 71,969 verifiers. Messier combines public evaluation results with new runs on six underrepresented professional and scientific benchmarks, standardizing their heterogeneous components into a common schema. Using this corpus, we show that frontier progress is uneven across benchmark groups, with function-calling evaluations largely saturated, programming improving fastest, and enterprise workflows remaining most challenging. Counterfactual rescoring further shows that reported performance can be highly sensitive to how verifier results are aggregated. Finally, we derive capability scores from our corpus that correlate with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.84. The scores can also be estimated for subsets defined by domain, occupation, action space, or verifier type. In essence, Messier is a reusable resource for studying agent performance at scale, and a basis for designing better evaluations.

Subjects:	Artificial Intelligence (cs.AI); Databases (cs.DB)
Cite as:	arXiv:2607.25891 [cs.AI]
	(or arXiv:2607.25891v3 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2607.25891 arXiv-issued DOI via DataCite

Submission history

From: Stefan Krsteski [view email]
[v1] Tue, 28 Jul 2026 15:50:19 UTC (205 KB)
[v2] Sun, 30 Aug 2026 18:22:18 UTC (202 KB)
[v3] Tue, 10 Nov 2026 10:32:04 UTC (202 KB)