跳到正文
arXiv cs.AI· Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu, Mohammad Javad Abdolhosseini Qomi, Maziar Raissi·· 2 小时前精选AI 评分63

研究揭示 LLM Agent 检索工具输出导致的证据替换误差

Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

AI 导读

研究用 USGS 地球化学模拟器 PHREEQC 构建 158 道执行验证选择题,发现 Agent 通过分区索引读取报告时 GPT-5.1 和 GPT-5.2 准确率分别下降 11.3 和 7.9 分,Claude Opus 4.6 和 Claude Sonnet 4.6 无可检测损失。

推荐理由

论文用可溯源的 PHREEQC 仿真定位检索式上下文管理损失准确性的具体机制,给出证据替换这一可检查的失败模式和缓解手段。

正文 · 原文

View PDF HTML (experimental)

Abstract:LLM agents that compute the right answer can still answer wrongly. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geochemistry simulator from the USGS, whose reports reach 846.8k characters and whose determinism lets every printed value be traced. Agents use it to answer 158 execution-verified multiple-choice items. When agents read the simulator's report through a section index rather than in context, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost. Re-running all 11,007 logged simulator calls shows that both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from. Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution. With the simulation held fixed, retrieval still costs GPT-5.2 9.4 points. For GPT-5.4, more frequent correct simulations mask an evidence-use loss of about ten points, leaving a net change indistinguishable from zero. Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models; for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not. Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.
Comments: v2: revised and extended. New analyses (fill-in-the-blank run, truncation analysis, expert review), redrawn figures, and rewritten text
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2610.26836 [cs.AI]
  (or arXiv:2610.26836v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2610.26836

arXiv-issued DOI via DataCite

Submission history

From: Ke Zhang [view email]
[v1] Mon, 19 Oct 2026 17:57:41 UTC (197 KB)
[v2] Mon, 16 Nov 2026 19:04:46 UTC (164 KB)

来源:arXiv cs.AI · arxiv.org