LARE：基于损失重加权的数据混合快速 LLM 预训练方法
Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting
AI 导读
arXiv 论文提出 LARE（Loss-Adaptive Reweighting under fixed Exposure），在固定各领域数据曝光量的前提下，依据运行时损失统计自动调整领域权重，揭示数据混合比例之外的一项独立控制手段。
正文 · 原文
Abstract:Large language models often acquire specialized capabilities through successive pretraining stages, but these gains can come at the cost of forgetting earlier knowledge. In this work, we investigate whether such trade-offs can be improved by changing how strongly each domain contributes to learning while keeping its data exposure fixed. We find that, in controlled knowledge--mathematics experiments, loss reweighting produces a single checkpoint that surpasses the best knowledge and reasoning scores attained along either stage trajectory. This reveals an independent control that data-mixture ratios alone do not capture. Building on this finding, we propose LARE (Loss-Adaptive Reweighting under fixed Exposure), which automatically adjusts domain weights from running loss statistics. Motivated by a theoretical analysis of gradient noise, its normalized inverse-square-root rule balances relative loss scales without changing the sampled mixture or requiring additional runs to search for weights. Across knowledge, mathematics, and code, \Method{} improves all five evaluated benchmarks for a 1.5B model trained on 240B tokens, averaging +7.84 pp over a matched direct mixture, including gains of +17.06 pp on GSM8K and +9.00 pp on MATH. These findings demonstrate that adjusting domain loss scales can improve joint capabilities without additional domain exposure.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32141 [cs.LG] |
| (or arXiv:2610.32141v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32141 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiazheng Li [view email]
[v1]
Fri, 23 Oct 2026 04:09:46 UTC (822 KB)
来源：arXiv cs.AI · arxiv.org