跳到正文
arXiv cs.AI· Daniele Kopyshevskiy (MLIA), Benjamin Th{\'e}rien (Mila, UdeM), Nicolas Pinto (MLIA), Edouard Oyallon (MLIA)·· 4 小时前AI 评分38

LACCO:通过计算与通信重叠的低带宽分布式语言模型训练方法

LACCO: Local Accumulation with Computation--Communication Overlap

AI 导读

LACCO(Local Accumulation with Computation–Communication Overlap)是一种两阶段算法,在低带宽网络下训练语言模型时将全局通信与本地计算重叠,且外层更新不引入陈旧性,同时保证平滑非凸目标的收敛。

正文 · 原文

View PDF

Abstract:Training language models across low-bandwidth networks requires reducing synchronization overhead. Local optimization methods such as DiLoCo communicate only after several local steps, but outer gradient averaging and the outer update remain on the critical path. Existing computation-communication overlap methods avoid this stall through delayed or replica-local updates, which can degrade convergence or cause replicas to drift. We propose LACCO (Local Accumulation with Computation-Communication Overlap), a two-stage algorithm that overlaps global communication with local computation while applying globally aggregated updates shared by every replica, without introducing staleness in the outer update. We establish convergence guarantees for smooth non-convex objectives. Experiments on language-model pre-training with FineWeb show that LACCO consistently outperforms communication-overlapped baselines such as Eager and Delayed DiLoCo. These gains remain robust as model size scales up to 0.5B parameters and across a range of communication intervals and numbers of replicas up to 16 workers, while incurring only negligible-or fully overlappable-computational or memory overhead, which demonstrates the strength of LACCO.
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2611.17991 [cs.AI]
  (or arXiv:2611.17991v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.17991

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Edouard Oyallon [view email] [via CCSD proxy]
[v1] Fri, 13 Nov 2026 10:21:28 UTC (709 KB)

来源:arXiv cs.AI · arxiv.org