arXiv cs.AI· Ruoyu Wu, Zengji Tu, Aolong Sun, Tianyi Ma, Junren Li, Xinbo Xu, Li Fu, Ziyan Zhang, Pinrui Huang, Xiaochen Wang, Chengda Wu, Yutao Zhou, Jianjun Wu, Lin Chang, Baobao Chang, Liang Chen, Kuan Li·· 2 小时前AI 评分53
PAPERBENCHX 发布：93 项跨学科论文复现任务揭示 AI 复现科学的差距
PAPERBENCHX: Can AI Reproduce Science Across Disciplines?
AI 导读
arXiv 论文 arXiv:2610.39744 提出 PAPERBENCHX 基准，含 93 项专家构建的论文复现任务，覆盖电磁学、光子学、化学、材料科学、生物学和机器人学等 12 个研究领域。
正文 · 原文
Authors:Ruoyu Wu, Zengji Tu, Aolong Sun, Tianyi Ma, Junren Li, Xinbo Xu, Li Fu, Ziyan Zhang, Pinrui Huang, Xiaochen Wang, Chengda Wu, Yutao Zhou, Jianjun Wu, Lin Chang, Baobao Chang, Liang Chen, Kuan Li
Abstract:Before AI can be trusted to discover new science, it should be able to reproduce what science already knows. Reproduction provides a rigorous checkpoint for an AI scientist: the target claim is known, yet recovering it still requires translating an incomplete scientific description into an executable experiment and producing evidence that can be independently verified. Existing research-reproduction benchmarks largely focus on AI and machine learning, where experiments share relatively homogeneous software stacks. Extending this capability across scientific disciplines introduces a different challenge: agents must operate domain-specific scientific software, make physical and numerical choices, diagnose long-running computations, and distinguish valid evidence from merely plausible outputs. We introduce PAPERBENCHX, a benchmark of 93 expert-constructed paper-reproduction tasks spanning 12 research areas across electromagnetics, photonics, chemistry, materials science, biology, and robotics. Each task provides the source paper, a public task specification, and a domain-native scientific environment involving specialized scientific software. Agents must reconstruct a key experiment, execute and diagnose it, and deliver a runnable workflow that supports the target scientific claim. To evaluate scientific validity rather than surface-level agreement, we replay every submission from scratch and score only regenerated evidence against expert-validated criteria. Across 10 frontier agent configurations, the best configuration fully reproduces only 14% tasks under the 90% threshold. Meanwhile, mean rubric scores range from 27.53% to 59.08%, indicating that substantial partial progress rarely translates into complete scientific reproduction. These results reveal a large gap between paper understanding and reliably producing reproducible scientific evidence.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39744 [cs.AI] |
| (or arXiv:2610.39744v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39744 arXiv-issued DOI via DataCite |
Submission history
From: Ruoyu Wu [view email]
[v1]
Thu, 29 Oct 2026 10:05:50 UTC (6,457 KB)
来源：arXiv cs.AI · arxiv.org