arXiv cs.AI· Lujia Zhong, Yihao Xia, Jianwei Zhang, Shuo Huang, Xinyu Nie, Mingyang Xia, Zhiwei Deng, Jiaxin Yue, Xinkai Wang, Junhan Zhao, Yonggang Shi·· 1 小时前AI 评分43
NIBench：面向神经影像的自主 AI 智能体执行、技能进化与科学判断基准
NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging
AI 导读
NIBench 是一个神经影像自主 AI 智能体基准，包含三部分评估：56 个任务、覆盖七种工具生态的执行与工作流重建测试，基于执行轨迹蒸馏的技能进化测试，以及来自阿尔茨海默病已发表研究的 36 条论断的科学判断测试。研究在统一智能体框架下评估了 19 个语言模型，领先智能体几乎完美执行给定分析流程，但蒸馏技能对未见任务的收益不一致，且智能体有时得出与专家不一致的结论。
正文 · 原文
Authors:Lujia Zhong, Yihao Xia, Jianwei Zhang, Shuo Huang, Xinyu Nie, Mingyang Xia, Zhiwei Deng, Jiaxin Yue, Xinkai Wang, Junhan Zhao, Yonggang Shi
Abstract:Autonomous AI agents are increasingly used to automate biomedical research, creating a growing need to benchmark their capabilities on real scientific data and workflows. Their reliability in neuroimaging remains insufficiently characterized, where plausible analytical outputs can conceal errors that affect scientific conclusions. Existing studies have advanced scientific question answering and tool-based automation, but provide limited evidence on workflow reconstruction, acquisition of procedural expertise and evidence-based scientific judgment. Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents. First, we benchmark execution and workflow reconstruction using 56 tasks across seven tool ecosystems, covering structural, diffusion and functional magnetic resonance imaging, positron emission tomography and multimodal analyses. Second, we examine whether agents can evolve their procedural skills through distillation of execution trajectories, evaluating the resulting skills on previously encountered tasks, unseen tasks and other models. Third, we assess scientific judgment using 36 claims from published Alzheimer's disease studies, requiring agents to analyse raw cohort data and provide reproducible evidence. We evaluate 19 language models within a shared agent framework, with selected models undergoing skill distillation and claim evaluation. Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences. Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent. Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.15831 [cs.AI] |
| (or arXiv:2611.15831v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15831 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lujia Zhong [view email]
[v1]
Wed, 11 Nov 2026 18:44:56 UTC (1,247 KB)
来源：arXiv cs.AI · arxiv.org