arXiv cs.AI· Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing, Zhonghan Yue, Jiayi Chen, Shiqiang Wu, Shaofan Liu, Yue Zhang, Yuankai Ying, Yang Shi, Tao Gui, Qi Zhang, Xuanjing Huang·· 5 小时前AI 评分45
NovGauge：诊断大语言模型论文新颖性评估能力的细粒度基准
NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment
AI 导读
研究者提出 NovGauge，一个用于细粒度诊断大语言模型论文新颖性评估能力的人工锚定基准，包含 619 组论文对和 50 组多论文集合，沿任务、问题、方法三个维度标注。
正文 · 原文
Authors:Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing, Zhonghan Yue, Jiayi Chen, Shiqiang Wu, Shaofan Liu, Yue Zhang, Yuankai Ying, Yang Shi, Tao Gui, Qi Zhang, Xuanjing Huang
Abstract:Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.11234 [cs.AI] |
| (or arXiv:2609.11234v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.11234 arXiv-issued DOI via DataCite |
Submission history
From: Guoqiang Zhang [view email]
[v1]
Thu, 10 Sep 2026 08:34:56 UTC (701 KB)
[v2]
Thu, 29 Oct 2026 03:51:41 UTC (701 KB)
来源：arXiv cs.AI · arxiv.org