跳到正文
arXiv cs.AI· Matt Turk, Brandon L. Neel, Diana M. Lopez, Zachary D. Smith, Grace N. Johnecheck, James B. Golden·· 2 小时前AI 评分52

ProteinGeoBench 论文显示前沿结构生物学智能体擅长查询但不擅长几何计算

ProteinGeoBench: Frontier Structural-Biology Agents Are Fluent at Lookup but Lost in Geometry

AI 导读

论文提出 ProteinGeoBench,包含 131 项蛋白质结构任务,共运行 4,249 次,评估 4 个前沿模型并另测 26 个模型。Claude Opus 5 查询得分 0.907、坐标计算仅 0.716,Gemini 3.1 Pro 是例外;全部四个模型都失败的 12 个任务中一半是计算题。

正文 · 原文

View PDF HTML (experimental)

Abstract:ProteinGeoBench is a set of 131 tasks that give an agent protein structure files and ask it to look up recorded facts, check sequences, calculate geometry from atom coordinates or reason about structures. We ran four frontier models on every task several times (4,249 runs) and 26 models on a subset. A single overall score hides two different skills. Three of the four frontier models are best at looking up facts and worst at calculating from coordinates: Claude Opus 5 scores 0.907 on lookup and 0.716 on calculation. Gemini 3.1 Pro is the exception. Geometry is overrepresented among the hardest tasks. Half of the twelve tasks that all four models fail are calculations, although calculations are a fifth of the suite, and eight of the twelve were solved by none of the 26 models. The three leading models are within 0.031 of one another, too close to order, and with one run per task a different model ranks first in 24% of bootstrap resamples. Reports should therefore give scores by task type and the number of runs per task. They should also state the agent harness, the software that runs the model, and its network access, because both change scores. On a 15-task subset, blocking eight structural databases lowered scores, and removing network access lowered them more. Replacing our harness with each provider's own command-line agent moved scores by up to 0.09 in either direction. Across the 26 models, those that scored higher on the other tasks also scored higher on calculation.
Comments: 24 pages, 2 figures, 10 tables. Benchmark and tasks: this https URL
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Biomolecules (q-bio.BM)
Cite as: arXiv:2610.26829 [cs.AI]
  (or arXiv:2610.26829v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2610.26829

arXiv-issued DOI via DataCite

Submission history

From: Matthew Turk [view email]
[v1] Mon, 19 Oct 2026 17:55:38 UTC (63 KB)
[v2] Wed, 18 Nov 2026 02:39:21 UTC (63 KB)

来源:arXiv cs.AI · arxiv.org