FedPA-LoRA：缓解异构联邦 LoRA 聚合与初始化误差的乘积对齐框架

View PDF HTML (experimental)

Abstract:Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorization creates a tension between aggregation fidelity and local optimization continuity. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch. We propose FedPA-LoRA, a product-aligned framework that addresses both mismatches and establishes convergence bounds for homogeneous and heterogeneous ranks. Clients preserve local factors across rounds to eliminate initialization mismatch and align their products with rank-specific global references to reduce client drift. The server aggregates heterogeneous-rank updates in product space to mitigate aggregation mismatch and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. By decoupling local and reference ranks, FedPA-LoRA supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show consistent gains under data and resource heterogeneity, including a 6.82 percentage-point improvement in average GLUE accuracy under heterogeneous ranks. The code is available at this https URL.

Submission history

From: Juseok Jeon [view email]
[v1] Sat, 15 Aug 2026 19:20:31 UTC (525 KB)
[v2] Fri, 30 Oct 2026 15:45:11 UTC (757 KB)