跳到正文
arXiv cs.AI· Alexander Boesgaard Lorup·· 2 小时前AI 评分52

Openhagen Worlds 发布:面向科学发现智能体测量与训练的校准环境套件

Openhagen Worlds: Calibrated Environments for Measuring and Training Scientific Discovery Agents

AI 导读

Alexander Boesgaard Lorup 在 arXiv 发布 Openhagen Worlds,包含 27 个程序化生成环境与 178 个经校准的难度档,用于测量和训练能在预算内做实验并接受确定性验证器评分的科学发现智能体。

正文 · 原文

View PDF HTML (experimental)

Abstract:Agents that do science must choose informative experiments and reason from their results, but most discovery benchmarks cannot separate that skill from recall, unstated scoring rules or shortcut policies. We introduce Openhagen Worlds, a suite of 27 procedurally generated environments (some with finite truth spaces), from classic puzzles of experimental reasoning to simulator-grounded physics and chemistry, in which an agent runs experiments under a budget and commits an answer that a deterministic verifier checks against a hidden truth. Every observation states the complete scoring rule, and each of the 178 difficulty levels is calibrated: it is admitted only if, on 64 development instances, a scripted oracle succeeds on at least 0.75, the strongest plausible non-deductive policy on at most 0.25 and each gated shortcut on at most 0.35. The 92 core levels span all 27 families and are sized to train small models; the 86 frontier rungs extend the 15 discovery worlds with harder questions and with campaigns of three, five and eight or more committed stages on one hidden system. On the same 84 withheld frontier instances, one per family and rung, DeepSeek V4.1 Flash solves 32 at its default reasoning effort with 16,384 tokens per call and 57 at its maximum effort with up to 393,216 tokens per call (oracle: 79); the headroom that remains is largest in eight-stage campaigns. On-policy expert iteration gives a 9B model modest gains: its core-tier success on trained families rises from 0.090 to 0.148, partly through better action format, with no detectable loss on GPQA Diamond. A hosted service plays the core tier's withheld instances.
Comments: 41 pages. Ancillary files: a public sample of 48 development instances with scripted-oracle episodes. Hosted evaluation: this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2611.24711 [cs.LG]
  (or arXiv:2611.24711v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.24711

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Alexander Lorup [view email]
[v1] Wed, 18 Nov 2026 16:36:18 UTC (326 KB)

来源:arXiv cs.AI · arxiv.org