arXiv cs.AI· Matteo Gabburo, Daniela Baiamonte, Stefano Simonazzi, Silvia Gardini, Sean Cesare Lazzeri, Andrea Zugarini·· 1 小时前AI 评分52
MDU 基准评测视觉语言模型的多语言文档理解能力
Benchmarking Multilingual Document Understanding in Vision-Language Models
AI 导读
研究者发布 MDU 多语言文档理解基准，包含 23146 个样本、13 个任务和英法德意西五种语言。团队用同一协议评测 45 个 1.7B 至 235B 参数的开源 VLM 和一个商业系统，最佳平均分仅 69.0，两个最好结果来自 27B 模型而非最大模型。语言差距不随模型变大而缩小，各任务能力提升不同步，转写能力与语义抽取之间存在取舍。
正文 · 原文
Abstract:Recent Vision-Language Models (VLMs) perform strongly on multimodal tasks, Document Understanding (DU) among them, but the benchmarks that measure them mostly cover English documents and narrow problems. How well current VLMs generalize across languages, tasks, styles and layouts therefore remains unclear. We introduce MDU, a multilingual DU benchmark of 23146 examples spanning 13 tasks and five languages: English, French, German, Italian, and Spanish. The tasks isolate complementary capabilities, from surface-level text and Markdown extraction through summarization, classification and named-entity recognition to layout-sensitive passage and layout extraction. We evaluate 45 state-of-the-art open-weight VLMs from 1.7B to 235B parameters, plus one commercial system under the same protocol. The benchmark is far from saturated: the best average is 69.0, only eight of the 45 evaluated systems reach 65, and the two best results come from 27B models rather than from the largest systems we evaluate. Performance is uneven in ways that scale does not fix. The gap between the strongest and weakest language does not close as models grow, and it survives controlling for the domain composition of each language. Capabilities also do not improve together: once we partial out overall model quality, per-task scores are largely uncorrelated, and transcription ability in particular trades off against semantic extraction.
| Comments: | 18 pages, 4 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14758 [cs.CL] |
| (or arXiv:2611.14758v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14758 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Matteo Gabburo [view email]
[v1]
Tue, 10 Nov 2026 21:35:26 UTC (142 KB)
来源：arXiv cs.AI · arxiv.org