跳到正文
arXiv cs.AI· Yuchen Zhang, Taolin Han, Jinghang Wang, Jinxin Wang, Yun Wu, Wai Yuet Chiu, Xiaogang Li, Yuting Fu, Kewei Sun, Jiaxin Li, Qile Jin, Jiajia Li, Ruodan Chen, Qingteng Chen, Jialu Nie, Junchao Li, Yifei Zhang, Chen Zhao, Hu Wei, Lin Qu, Yanni Yin, Bing Zhao·· 56 分钟前AI 评分41

SciWorkBench:面向真实科学工作流的 Agent 基准测试

SciWorkBench: A Benchmark for Real Scientific Workflows

AI 导读

研究者推出 SciWorkBench 基准,评估 AI 智能体在生物、化学、材料科学和物理领域的真实科学任务能力,任务基于同行评审文献或专家贡献场景构建。在 Claude Code 和 Codex 智能体框架下统一评测,表现最好的智能体平均分仅 57.38。失败分析发现三大弱点:难以将科学知识转化为具体操作流程、跨模态证据整合能力弱、科学自我纠错能力有限。

正文 · 原文

Authors:Yuchen Zhang, Taolin Han, Jinghang Wang, Jinxin Wang, Yun Wu, Wai Yuet Chiu, Xiaogang Li, Yuting Fu, Kewei Sun, Jiaxin Li, Qile Jin, Jiajia Li, Ruodan Chen, Qingteng Chen, Jialu Nie, Junchao Li, Yifei Zhang, Chen Zhao, Hu Wei, Lin Qu, Yanni Yin, Bing Zhao

View PDF HTML (experimental)

Abstract:Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22490 [cs.AI]
  (or arXiv:2611.22490v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22490

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Taolin Han [view email]
[v1] Tue, 17 Nov 2026 09:46:16 UTC (1,466 KB)

来源:arXiv cs.AI · arxiv.org