JFinBench:基于最新认证考试的可持续更新日语金融基准
JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams
AI 导读
研究者推出日语金融基准 JFinBench,包含来自 2026 年版六项认证考试的 321 道题,覆盖会计、税务、理财规划与银行业务,并用人机协作的 agentic 流水线实现定期更新以缓解数据污染。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.
| Comments: | 32 pages, 14 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.18130 [cs.AI] |
| (or arXiv:2611.18130v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.18130 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Issa Sugiura [view email]
[v1]
Fri, 13 Nov 2026 12:59:12 UTC (2,137 KB)
来源:arXiv cs.AI · arxiv.org