SciGame：让大语言模型智能体在真实与反事实世界中玩科学发现游戏
SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World
AI 导读
arXiv 论文提出 SciGame 基准，将科学文献转化为结构化科学卡片，每张卡片生成原始与维度一致的反事实环境对，包含基于经典定律的 Historical 集合和取自模型知识截止后论文的 Temporal 集合，在 Identification 与 Discovery 两种设定及 Prior-only、Fixed、Adaptive 三种协议下评估智能体。
正文 · 原文
Abstract:Large language model agents are increasingly used for scientific discovery. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors. We introduce SciGame, a controlled benchmark that converts scientific literature into structured scientific cards. Each card yields a pair of Original and dimensionally consistent Counterfactual environments with the same task interface. SciGame contains a Historical collection of canonical laws and a live Temporal collection curated from papers published after model knowledge cutoffs. It evaluates agents in candidate-guided Identification and open-ended Discovery settings. Each setting runs under Prior-only, Fixed, and Adaptive protocols. Experiments on five frontier models yield three findings. First, familiar priors hurt performance once the governing relation is altered. Second, adaptive experimentation helps the strongest models but harms weaker ones under limited budgets. Third, good explanations can hide quantitative failures. Recovering the exact symbolic form remains the main bottleneck, even when agents interpret evidence correctly. Current agents can investigate systems actively, but inducing exact laws in unfamiliar settings remains an open challenge.
| Comments: | 25 pages, 11 figures, 9 tables |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.37919 [cs.AI] |
| (or arXiv:2610.37919v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37919 arXiv-issued DOI via DataCite |
Submission history
From: Yusheng Zhao [view email]
[v1]
Tue, 27 Oct 2026 20:02:47 UTC (7,295 KB)
来源：arXiv cs.AI · arxiv.org