重新思考 LoRA 初始化以实现稳健的非对称学习率
Rethinking LoRA Initialization for Robust Asymmetric Learning Rates
AI 导读
一项 arXiv 研究通过信号-噪声分解揭示了 LoRA+ 非对称学习率在标准初始化下训练不稳定、高比率性能崩溃的根源：更新中存在由初始化引入的噪声项。研究提出新初始化方法 initAA，将 A 的方差按 σ_A² ∝ n⁻² 缩放并保持 B₀=0，使初始化噪声能量额外降低 1/n。该方法显著扩大稳定学习率区间，使 η_B ≈ n·η_A 成为有限预训练模型可用的起点。
正文 · 原文
Abstract:Low-Rank Adaptation (LoRA) is often highly sensitive to learning-rate (LR) choices. Recent empirical studies suggest that, once LRs are carefully tuned, vanilla LoRA can remain competitive with many proposed LoRA variants. This shifts the central practical question from designing ever more variants to reducing the cost and brittleness of LR selection. LoRA+ addresses this issue with asymmetric LRs, using a larger LR for the up-projection matrix $B$ than for the down-projection matrix $A$. However, the ratio $\eta_B / \eta_A$ itself becomes an additional hyperparameter: large ratios can improve adaptation, but under the standard LoRA initialization they can also cause unstable training and ratio-dependent performance collapse. We identify the source of this brittleness through a signal--noise decomposition of LoRA dynamics. Under standard \initA, the update contains an initialization-induced noise term whose magnitude can be affected by the LR for $B$, $\eta_B$, the initialization scale $\beta$, and the adapter rank $r$. In practical LLM fine-tuning, this non-vanishing noise can limit the performance gains expected from LoRA+ style asymmetric LRs. To suppress this effect, we propose \initAA, a simple initialization that scales the variance of $A$ as $\sigma_A^2 \propto n^{-2}$ while keeping $B_0=0$. Under matched rank and training time for both initializations, this reduces the initialization-noise energy by an additional factor of $1/n$, stabilizing asymmetric LoRA training while preserving zero initial adapter output. Empirically, \initAA substantially widens the stable LR region across initialization scales, adapter ranks, and LoRA update multipliers. Across tasks and models, \initAA reduces best-ratio drift, prevents the high-ratio collapse observed with standard \initA, and makes the width-based guideline $\eta_B \approx n \eta_A$ a useful starting point for finite pretrained models.
| Comments: | 46 pages, 13 figures, NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32142 [cs.LG] |
| (or arXiv:2610.32142v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32142 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Disen Liao [view email]
[v1]
Fri, 23 Oct 2026 04:15:33 UTC (2,330 KB)
来源：arXiv cs.AI · arxiv.org