arXiv cs.AI· Alina Avanesyan, Alexey Boldyrev, Liqun Deng, Denis Derkach, Mary Godunova, Mikhail Hushchyn, Dmitry Ilvovsky, Irina Knyazeva, Irina Piontkovskaya, Dmitry Polubotko, Fedor Ratnikov, Ekaterina Trofimova, Zhenhe Zhang·· 1 小时前AI 评分51
STEMBench 基准测试大语言模型在博士后级自然科学问题上的能力
Benchmarking LLM abilities on postdoc-level natural science problems
AI 导读
arXiv 论文 STEMBench 用各领域自身评估指标测试大语言模型在高能物理、天体物理和神经科学十二个博士后级科研问题上的能力。每个问题附无领域知识提交和复现已发表方法的专家参考两个锚点;八个前沿模型中,任务级胜出与领域级领先分离,高强度对比下至少一个模型在十二个任务中的七个平均分超过专家锚点,其余五个涉及不确定性估计、潜在与因果推断和跨站点临床分类。任务、评分器、锚点和运行框架已开源。
正文 · 原文
Authors:Alina Avanesyan, Alexey Boldyrev, Liqun Deng, Denis Derkach, Mary Godunova, Mikhail Hushchyn, Dmitry Ilvovsky, Irina Knyazeva, Irina Piontkovskaya, Dmitry Polubotko, Fedor Ratnikov, Ekaterina Trofimova, Zhenhe Zhang
Abstract:Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.
| Comments: | 22 pages, 7 tables, 2 figures |
| Subjects: | Artificial Intelligence (cs.AI); Data Analysis, Statistics and Probability (physics.data-an) |
| Cite as: | arXiv:2611.24321 [cs.AI] |
| (or arXiv:2611.24321v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24321 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Denis Derkach [view email]
[v1]
Wed, 18 Nov 2026 12:25:33 UTC (264 KB)
来源:arXiv cs.AI · arxiv.org