AgenticFinBench：面向真实金融任务的智能体基准测试

View PDF HTML (experimental)

Abstract:Financial agents are becoming an important class of real-world AI systems. Financial tasks vary widely in type and difficulty, and many are open-ended with no ground-truth answers. However, existing benchmarks neither capture this breadth and open-endedness nor separate grounding from correctness. To bridge this gap, we introduce $\textbf{AgenticFinBench}$, a benchmark of 230 real-world financial tasks spanning 8 categories, 2 languages, and 4 complexity levels. We also propose a reference-free evaluation framework that combines question-specific rubrics with a two-stage claim audit that detects hallucinations, defined as claims absent from the agent's trace and refuted by external evidence. This framework better matches how financial work is assessed in practice. Experiments across 18 agent configurations reveal three key findings: (1) Ignoring hallucinations can overstate agent performance: hallucination auditing lowers the baseline leader's score by 7.58 points and moves it to fourth place. (2) Harness differences explain more variation in evidence grounding than model differences (28.6% vs 8.8%). (3) Quality scores fall from 89.44 on direct retrieval to 79.18 on the most complex tasks. Case studies show that agents struggle to explain how events affect company profits or stock prices, compare financial data on a consistent basis, and justify their conclusions. By releasing AgenticFinBench, we aim to advance financial reasoning, factual verification, and agent design for more reliable responses to diverse real-world financial requests.

Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2611.04813 [cs.AI]
	(or arXiv:2611.04813v2 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2611.04813 arXiv-issued DOI via DataCite

Submission history

From: Shu Zhou [view email]
[v1] Tue, 3 Nov 2026 12:43:10 UTC (7,968 KB)
[v2] Wed, 4 Nov 2026 03:29:04 UTC (7,968 KB)