跳到正文
arXiv cs.AI· He Liu, Boyuan Gu, Shuaiqi Cheng, Haiyang Sun, Siyu You, Xuming Hu·· 4 小时前AI 评分36

PhysDox 基准评测:LLM 能否审计生理感知协议的物理可行性

PhysDox: Benchmarking LLMs on Physical Feasibility Auditing of Physiological Sensing Protocols

AI 导读

PhysDox 是一个面向大语言模型的生理感知协议物理可行性审计基准,包含 683 个专家标注的 Gold 样本和 5,000 个 Silver 样本,覆盖六个感知领域。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large language models (LLMs) increasingly assist in experimental design, yet fluent protocols often remain physically infeasible. We introduce PhysDox, a physical feasibility auditing benchmark for biomedical protocols comprising a 683-sample expert-curated Gold set and a 5,000-sample Silver set across six sensing domains. We formulate the task as a two-stage evaluation: severity detection classifying protocols as valid, minor, or fatal, followed by the constraint-level diagnosis of fatal violations. Evaluating 6 LLMs across 4 inference strategies yields a peak Stage-1 macro-F1 of only 53.0. Moreover, strong oracle diagnosis collapses during end-to-end evaluation due to correlated cascade errors. Error analysis reveals scaffold bias, where models conflate procedural completeness with physical validity. Consequently, implicit constraints exhibit a 2 times higher miss rate than explicit hardware violations, supported by strong statistical correlation at $\rho{=}0.81$ and $p{<}0.01$. Trace analysis of false negatives exposes a 54%--46% split between attention and judgment failures, ultimately demonstrating that protocol auditing demands calibrated feasibility reasoning rather than factual recall or longer rationales.
Comments: 31 Pages,7 Figures
Subjects: Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI)
Cite as: arXiv:2606.05003 [cs.HC]
  (or arXiv:2606.05003v2 [cs.HC] for this version)
  https://doi.org/10.48550/arXiv.2606.05003

arXiv-issued DOI via DataCite

Journal reference: EMNLP 2026 (findings)

Submission history

From: Boyuan Gu [view email]
[v1] Wed, 3 Jun 2026 15:20:15 UTC (6,354 KB)
[v2] Wed, 23 Sep 2026 08:23:59 UTC (13,535 KB)

来源:arXiv cs.AI · arxiv.org