arXiv cs.AI· Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, Qinghua Ni, Jun Huang, Qiang Zhang, Yue Zhou, Zhihang Zhong, Yonglin Tian, Fei-Yue Wang, Xue Yang·· 3 小时前AI 评分39
BEAKER：面向自动驾驶化学实验室具身大脑的专家策划基准
BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories
AI 导读
研究者推出 BEAKER，一个评估自动驾驶化学实验室中具身可操作性、知识、实验与推理能力的基准，包含 1,000 个专家人工策划的问答样本，覆盖 11 种图像与视频任务类型。团队以离线单轮设置评测了 28 个通用、科学、化学专用及具身 MLLM。结果显示当前模型能跟随视频中的粗略实验流程，但在将动作落地到可操作区域与可行轨迹、识别细粒度器材与安全规范、排序中长流程步骤上仍有明显差距。
正文 · 原文
Authors:Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, Qinghua Ni, Jun Huang, Qiang Zhang, Yue Zhou, Zhihang Zhong, Yonglin Tian, Fei-Yue Wang, Xue Yang
Abstract:Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.
| Comments: | Accepted by NeurIPS 2026 (Track on Evaluations and Datasets, Spotlight) |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06911 [cs.AI] |
| (or arXiv:2611.06911v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06911 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fei Lin [view email]
[v1]
Wed, 4 Nov 2026 18:57:17 UTC (20,405 KB)
来源：arXiv cs.AI · arxiv.org