跳到正文
arXiv cs.AI· Raviteja Anantha, Nick Levato, Layne C. Price·· 4 小时前AI 评分41

BoostLoRA:通过迭代 Boosting 适配器提升有效秩的参数高效微调方法

BoostLoRA: Growing Effective Rank by Boosting Adapters

AI 导读

BoostLoRA 是一种基于梯度提升的参数高效微调框架,通过迭代训练并合并超低秩适配器、以 ROTATE SVD 正交子空间轮换策略使累积有效秩随轮次线性增长,且合并后不留推理开销。

正文 · 原文

View PDF HTML (experimental)

Abstract:Parameter-efficient fine-tuning (PEFT) methods face a tradeoff between adapter size and expressivity: ultra-low-parameter adapters are confined to fixed low-rank subspaces, capping performance even with extended training. We propose BoostLoRA, a gradient-boosting framework that overcomes this limit by iteratively training and merging minimal adapters on the examples the current model gets wrong. A ROTATE SVD basis strategy assigns each round to an orthogonal subspace, so cumulative effective rank grows linearly with the number of rounds while each adapter remains ultra-low-rank. After merging, adapters are discarded, leaving zero inference overhead. On Qwen2.5-3B, BoostLoRA reaches 89.1% on GSM8K and 68.8% on MATH-500, surpassing both the best single-shot ultra-low parameter adapter (TinyLoRA) and full fine-tuning; on code generation it reaches 57.2% on MBPP and 80.4% on HumanEval while full fine-tuning drops below the zero-shot baseline. We also demonstrate cross-architecture transfer on protein binding classification with ESM2-650M and cross-entropy training. BoostLoRA is, to our knowledge, the first PEFT method whose effective rank grows with training, separating per-round parameter cost from total representational capacity.
Comments: Preprint
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2604.27308 [cs.LG]
  (or arXiv:2604.27308v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2604.27308

arXiv-issued DOI via DataCite

Submission history

From: Raviteja Anantha [view email]
[v1] Thu, 30 Apr 2026 01:43:00 UTC (684 KB)
[v2] Thu, 12 Nov 2026 20:04:21 UTC (710 KB)

来源:arXiv cs.AI · arxiv.org