FutureBench:评估大语言模型知识-上下文冲突的可再生评测框架
FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs
AI 导读
arXiv 论文提出 FutureBench,用带时间戳的问题、受控段落三元组和无上下文探针评测大语言模型的知识-上下文冲突,覆盖 3,629 个问题和六个模型。
正文 · 原文
Abstract:Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23601 [cs.CL] |
| (or arXiv:2611.23601v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23601 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yi-Cheng Lai [view email]
[v1]
Tue, 17 Nov 2026 22:55:12 UTC (205 KB)
来源:arXiv cs.AI · arxiv.org