重复胜过多样性：高质量数据过滤助力高效德语语言建模（Boldt 模型开源）
Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
AI 导读
一项 EMNLP 2026 研究对 500M 德语网页文档施加分层质量过滤，比较多轮重复训练过滤后数据与一次性训练多样性语料的效果，并提出德语模型 Boldt。实验显示，重复高质量数据在多个模型规模和 token 预算下持续优于单遍训练，且 7 个 epoch 后差距仍存在。Boldt 以比同类模型少 10-360x 的训练 token 达到 SOTA 结果，模型与清洗后的评测基准已开源。
正文 · 原文
Abstract:Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. However, for high-resource non-English languages like German, French, or Japanese, aggressive filtering creates a strategic dilemma: should practitioners prioritize diversity by training once on large amounts of lightly filtered web data, or prioritize quality by strictly filtering for a high-quality core and repeating it over multiple epochs? We investigate this trade-off for German by applying hierarchical quality filters to 500M web documents, comparing multi-epoch training on filtered subsets against single-pass training on a diverse corpus. Our experiments across multiple model scales and token budgets show that repeating high-quality data consistently outperforms single-pass training on larger, less filtered sets. Notably, the performance gap persists even after 7 epochs. Our findings suggest that for non-English LLMs, semantic concentration through quality filtering offers a more viable path to efficient language modeling than simply maximizing unique data volume. We release our German language models (called Boldt), as well as our cleaned evaluation benchmarks to the research community. Our experiments indicate that they achieve state-of-the-art results despite training on 10-360x fewer tokens than comparable models.
| Comments: | Accepted to EMNLP 2026 (Main) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.28075 [cs.CL] |
| (or arXiv:2604.28075v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.28075 arXiv-issued DOI via DataCite |
Submission history
From: Ansar Aynetdinov [view email]
[v1]
Thu, 30 Apr 2026 16:21:28 UTC (921 KB)
[v2]
Fri, 1 May 2026 13:52:15 UTC (483 KB)
[v3]
Fri, 23 Oct 2026 19:22:47 UTC (491 KB)
来源：arXiv cs.AI · arxiv.org