跳到正文
arXiv cs.AI· Guy Zyskind·· 1 小时前AI 评分55

arXiv 论文发现语言模型会把未研究的开放问题说得像已有研究

Language Models Make New Research Sound Old

AI 导读

arXiv 论文(arXiv:2611.23546)发现 LLM 在验证研究问题时会把开放问题说得像部分已研究,并夸大相关论文的结论。40 个来自新录用论文的问题中,GPT-6-Astra 正确判断 98% 未被研究,但普通回答中只有 8% 明确说出该状态;六个前沿系统报告缺口达 30-90 个百分点。

正文 · 原文

View PDF HTML (experimental)

Abstract:Researchers are increasingly using large language models (LLMs) to validate research questions. We find that LLMs make open research questions sound partly studied and overstate what related papers establish. Models recognize most of these overstatements when asked to check the papers' full texts separately. On 40 questions drawn from newly accepted papers, GPT-6-Astra correctly judges 98% to be unstudied, but states that status plainly in only 8% of ordinary answers. All six frontier systems show reporting gaps of 30-90 percentage points. Across 22 models given identical papers, higher capability is associated with more accurate judgments and larger reporting gaps. This reporting failure is asymmetric: models correctly identify studied questions in 88-100% of ordinary answers when we supply the answering papers. To correct this, we build a fixed answer from each model's judgment and separate checks of cited papers. This raises Astra's clear reporting of open status from 8% to 95%, with improvements in all six systems. Finally, we predict reviewers' novelty grades by combining several models' written ratings with a rule learned from graded examples, reaching the performance range reported by Think-Probe-Respond. Unlike TPR, our approach needs no access to model weights or internal activations, so it also works with closed-weight frontier models.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23546 [cs.CL]
  (or arXiv:2611.23546v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.23546

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Guy Zyskind [view email]
[v1] Tue, 17 Nov 2026 21:53:16 UTC (358 KB)

来源:arXiv cs.AI · arxiv.org