arXiv cs.AI· Ritiz Tambi, Sahar Shayegan, Joshua Meyer, Sun Kim, Ali Khan, Victor Shih, Mehdi Jamei, Andi Partovi·· 2 小时前AI 评分44
TreasuryQA Bench：AI 智能体能否处理扫描版财政档案？
TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?
AI 导读
TreasuryQA Bench 要求 AI 智能体在 1,435 份扫描版美国财政部出版物中检索文件并用 OCR 读取页面，回答 70 道需给出精确数字的问题。十款智能体中，Claude Opus 5.5 在 Claude Code 下成功率最高，达 64.8%；GPT-5.6 Terra 在 Codex 下最低，为 16.7%。扫描越模糊各智能体成功率越低，失败主因多为扫描中数字被误读。
正文 · 原文
Abstract:Organisations keep decades of financial records as scanned PDFs on shared drives, and AI agents can now reach those drives through Model Context Protocol (MCP) servers. Existing benchmarks do not mirror this production setting because they simplify the task. In some, the relevant documents are handed to the agent, so it never has to search. In others, the agent searches through documents with pre-extracted text, so it never has to read scanned images. We introduce TreasuryQA Bench, which requires both: 70 questions over 1,435 scanned U.S. Treasury publications, held in Veris's high-fidelity Dropbox twin. In each task the agent searches for the relevant file through Dropbox's MCP server, reads the pages it needs with Reducto's hosted OCR server or its own tools, and replies with one exact figure. A Python grader checks each answer against an accepted range around the correct value. In this paper we present results from ten agents. The best, Claude Opus 5.5 in Claude Code, solves 64.8% of attempts and the worst, GPT-5.6 Terra in Codex, solves 16.7%, so the benchmark is far from saturated. We also test how scan quality affects performance: every agent solves fewer tasks on the blurriest scans. Most failed attempts reply with a wrong number rather than no number, and the most common cause is a digit misread from the scan. Three worked examples are included with their answer keys.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38924 [cs.AI] |
| (or arXiv:2610.38924v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38924 arXiv-issued DOI via DataCite |
Submission history
From: Sahar Shayegan [view email]
[v1]
Wed, 28 Oct 2026 21:17:16 UTC (3,354 KB)
来源：arXiv cs.AI · arxiv.org