持续预训练何时有效?魁北克法语方言适应的受控研究
When Does Continual Pre-Training Help? A Controlled Study of Dialectal Adaptation for Qu\'ebec French
AI 导读
一项受控研究对 9 个 LLM 在魁北克法语理解上的持续预训练(CPT)效果进行评估,每项实验使用 30M token 的等预算协议。结果显示 CPT 的收益高度取决于基模型对目标语言变体的已有能力,收益主要由语料构成而非数据量驱动,其中文学文本贡献最大。CPT 主要提升表达理解而非语法能力,且伴随阅读理解下降等能力间权衡,总分可能掩盖这些下降。
正文 · 原文
Abstract:Continual pre-training (CPT) is widely used to adapt pretrained language models to dialectal and non-standard language varieties, yet it remains unclear when this additional training is necessary, what drives its benefits, and whether aggregate evaluation scores adequately capture its effects. In this paper, we present a controlled study of CPT for Québec French language understanding across nine LLMs spanning different families and scales. Using a budget-matched protocol of 30M tokens per experiment, our results show that the benefit of CPT depends strongly on the starting model's existing competence in the target variety. When CPT is beneficial, the improvements are driven more by corpus composition than by data volume, with literary text accounting for most of the gain. Paired Québec and standard French evaluations further show that CPT primarily improves expression comprehension rather than grammatical competence, and that corpus register determines which regional expressions are acquired. These gains also involve capability-specific trade-offs, including declines in reading comprehension that aggregate scores can conceal. Overall, these findings show that CPT for dialectal adaptation should not be treated as a uniformly beneficial default. Instead, adaptation strategies should begin with a careful assessment of the base model's existing competence, accompanied by source-level data analysis and task-level evaluation.
| Comments: | EACL 2027 Main Conference |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.24699 [cs.CL] |
| (or arXiv:2611.24699v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24699 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Eeham Khan [view email]
[v1]
Wed, 18 Nov 2026 16:26:03 UTC (51 KB)
来源:arXiv cs.AI · arxiv.org