模型自适应工具必要性研究揭示 LLM 工具使用的“知行差距”
Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use
AI 导读
arXiv 论文（NeurIPS 2026）提出基于模型实际表现的“工具必要性”自适应定义，在算术与事实问答数据集上对比四个模型，发现工具必要性与实际调用行为的错配分别达 26.5-54.0% 和 30.8-41.8%。通过探测 LLM 隐状态并将工具使用分解为认知与执行两阶段，研究发现错配主要集中在认知到行动的转化环节，揭示 LLM 工具使用存在“知行差距”。
正文 · 原文
Abstract:Large language models (LLMs) increasingly act as autonomous agents that must decide when to answer directly vs. when to invoke external tools. Prior work studying adaptive tool use has largely treated tool necessity as a model-agnostic property, annotated by human or LLM judge, and mostly cover cases where the answer is obvious (e.g., fetching the weather vs. paraphrasing text). However, tool necessity in the wild is more nuanced due to the divergence of capability boundaries across models: a problem solvable by a strong model on its own may still require tools for a weaker one. In this work, we introduce a model-adaptive definition of tool-necessity, grounded in each model's empirical performance. Following this definition, we compare the necessity against observed tool-call behavior across four models on arithmetic and factual QA dataset, and find substantial mismatches of 26.5-54.0% and 30.8-41.8%, respectively. To diagnose the failure, we decompose tool use into two stages: an internal cognition stage that reflects whether a model believes a tool is necessary, and an execution stage that determines whether the model actually makes a tool-call action. By probing the LLM hidden states, we find that both signals are often linearly decodable, yet their probe directions become nearly orthogonal in the late-layer, last-token regime that drives the next-token action. By tracing the trajectory of samples in the two-stage process, we further discover that the majority of mismatch is concentrated in the cognition-to-action transition, not in cognition itself. These results reveal a knowing-doing gap in LLM tool-use: improving tool-use reliability requires not only better recognition of when tools are needed, but also better translation of that recognition into action.
| Comments: | NeurIPS 2026 Camera Ready |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.14038 [cs.AI] |
| (or arXiv:2605.14038v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.14038 arXiv-issued DOI via DataCite |
Submission history
From: Yize Cheng [view email]
[v1]
Wed, 13 May 2026 18:59:28 UTC (1,009 KB)
[v2]
Sun, 17 May 2026 15:23:37 UTC (1,010 KB)
[v3]
Thu, 22 Oct 2026 19:13:29 UTC (2,397 KB)
来源：arXiv cs.AI · arxiv.org