跳到正文
arXiv cs.AI· Yi-Cheng Lai, Hen-Hsen Huang·· 1 小时前AI 评分38

FutureBench:评估大语言模型知识-上下文冲突的可再生评测框架

FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs

AI 导读

arXiv 论文提出 FutureBench,用带时间戳的问题、受控段落三元组和无上下文探针评测大语言模型的知识-上下文冲突,覆盖 3,629 个问题和六个模型。

正文 · 原文

View PDF HTML (experimental)

Abstract:Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23601 [cs.CL]
  (or arXiv:2611.23601v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.23601

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yi-Cheng Lai [view email]
[v1] Tue, 17 Nov 2026 22:55:12 UTC (205 KB)

来源:arXiv cs.AI · arxiv.org