arXiv cs.AI· Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou·· 2 小时前AI 评分46
Ask2Decide 基准：评测 LLM 智能体如何向人类获取决策相关信息
From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings
AI 导读
研究者推出 Ask2Decide 基准，包含 43 项规划任务，评测 LLM 智能体能否通过向人类提问获取决策关键信息。测试显示，信息缺失时四个前沿模型的初始方案在 58-98% 的任务上不可行，而信息完整时仅 5-28%；智能体找出改变最优方案的事实的比例仅略高于无关信息（39-52% 对 32-41%）。
正文 · 原文
Authors:Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou
Abstract:LLM agents are increasingly asked to make plans for people, such as scheduling drivers or assigning nurses. However, people's requests are often incomplete, and the supporting information for decision-making may be outdated, requiring agents to ask for clarification. Existing benchmarks typically provide fully specified problems or rely on uncalibrated LLM-simulated users, leaving this ability insufficiently tested. We introduce Ask2Decide, a benchmark of 43 planning tasks that evaluates whether agents ask for missing information that matters for their decisions. An operator holds facts absent from the agent's request and records, and optimization solvers determine which facts change the optimal plan. We also include similarly worded facts that leave the optimal plan unchanged to test whether agents distinguish useful information from irrelevant details. We collect 79 interaction sessions from 19 participants and find that, after an incorrect plan, human replies omit the correcting fact far more often than replies from an uncalibrated simulated user (43% versus 7%). Using these interactions, we calibrate a simulated operator that more closely matches human reply patterns. Evaluating four frontier LLMs, we find that their initial plans are infeasible on 58-98% of tasks when information is missing, compared with 5-28% when all facts are provided upfront. Agents uncover facts that change the optimal plan only slightly more often than facts that leave it unchanged (39-52% versus 32-41%). These results show that obtaining decision-relevant information through interaction remains a major bottleneck for LLM planning. This is ongoing work and will be updated. The date of this version is shown on the first page.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13015 [cs.AI] |
| (or arXiv:2611.13015v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13015 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhan Tang [view email]
[v1]
Mon, 9 Nov 2026 04:22:15 UTC (66 KB)
来源：arXiv cs.AI · arxiv.org