SplitLite：面向分割学习的低秩残差压缩方法
SplitLite: Low-Rank Residual Compression for Split Learning
AI 导读
论文提出 SplitLite，一种通信高效的分割联邦 LoRA 微调方法，利用相邻 epoch 间激活与梯度残差的低有效秩特性，仅传输量化截断 SVD 残差因子。在 GLUE 基准和多款端侧 LLM 上的实验显示，激活上行通信成本最多降低 93.5%，总通信成本最多降低 88.6%，且不损失模型性能。
正文 · 原文
Abstract:Federated fine-tuning of on-device large language models (LLMs) faces a significant computing burden. To overcome this limitation, split learning (SL) has emerged as a promising solution, which offloads the primary training workload to a powerful server. However, SL requires exchanging high-dimensional activations and gradients between clients and the server, resulting in prohibitive communication costs. To overcome this challenge, we propose SplitLite, a communication-efficient split federated LoRA fine-tuning method that exploits the low effective rank structure of consecutive-epoch activation and gradient residuals. Our key finding is that, under low-rank LoRA updates with the pretrained backbone frozen, the activation and gradient residuals of the same data sample between adjacent epochs exhibit low effective ranks, with most of their energy concentrated in a small number of singular directions. By exploiting this property, SplitLite transmits only quantized truncated singular value decomposition (SVD) residual factors, thereby significantly reducing both activation uplink and gradient downlink traffic. Extensive experiments on the GLUE benchmark across a series of advanced on-device LLMs demonstrate that our method reduces activation uplink communication costs by up to 93.5\% and total communication costs by up to 88.6\%, without sacrificing model performance.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.23018 [cs.LG] |
| (or arXiv:2608.23018v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.23018 arXiv-issued DOI via DataCite |
Submission history
From: Tao Li [view email]
[v1]
Mon, 24 Aug 2026 09:20:04 UTC (22,389 KB)
[v2]
Sat, 5 Sep 2026 13:12:30 UTC (1 KB) (withdrawn)
[v3]
Tue, 10 Nov 2026 05:02:06 UTC (21,901 KB)
来源：arXiv cs.AI · arxiv.org