数字对、问题错：评估 LLM 面对欠明确金融提问的「框架坍塌」
Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries
AI 导读
arXiv 论文研究 LLM 金融助手在印度共同基金场景中的「框架坍塌」：模型用真实数据回答了用户没问的问题。实验显示，面对标注模糊的数字模型倾向沿用用户标签；缺少投资者信息时较弱的 agent 会假设默认值或过时税则；与数据核对的检查无法识别这类错误，AI 评审只在被告知字段含义时才能发现标注错误。论文分析了失败成因并提出金融助手的控制方法。
正文 · 原文
Abstract:Financial assistants answer questions that sound factual but depend on what the user leaves unsaid: what a data figure measures, which product variant is meant, and the investor's situation and the rules that apply. An assistant that ignores these can still answer with a real number from its data, one that answers a different question. We call this frame collapse. We study it in Indian mutual funds, where product variants and tax rules are formally defined, so the correct answer can be computed for every reading. One study tests each challenge. Given an ambiguously labelled figure, models used it, often under the user's label. Given every alternative labelled, they listed them all, but the layout decided which came first. Missing a fact about the investor, weaker agents assumed a default or an outdated tax rule; requiring the fact did not guarantee its use. Because such answers match their data, checks against the data pass them. An AI judge caught mislabelled figures only when told what each field means, and still passed answers that silently chose another reading. We show how these failures arise, not how often, and propose controls for financial assistants.
| Comments: | Accepted for oral presentation at the 3rd Workshop on LLMs and Generative AI for Finance (AI4F) at ACM ICAIF '26, Milan, Italy, November 14, 2026. Non-archival. 16 pages, 4 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); General Finance (q-fin.GN) |
| Cite as: | arXiv:2610.38631 [cs.AI] |
| (or arXiv:2610.38631v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38631 arXiv-issued DOI via DataCite |
Submission history
From: Apoorv Singh [view email]
[v1]
Wed, 28 Oct 2026 10:48:32 UTC (216 KB)
来源：arXiv cs.AI · arxiv.org