模型合并不需要统一 LoRA 秩
Model Merging Does Not Require Uniform LoRA Ranks
AI 导读
一项 arXiv 研究表明，模型合并无需为所有专家使用相同 LoRA 秩，耦合限制来自合并方法而非合并本身。研究在数学、指令遵循、多语言三个域以四种秩和三种模型规模训练 36 个专家，按各域最优秩合并时平均法与效用选择法可提升最多 1.8 个百分点多域准确率，而冲突消解方法不一致、最多损失 4 个百分点。
正文 · 原文
Abstract:Model merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging. We train 36 experts on three domains (mathematics, instruction following, multilingual) at four ranks and three model scales, then merge with six methods at uniform ranks and at each domain's own best rank. Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp. The split has a simple cause. Merging keeps only a fixed share of the pooled spectral components, and methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp. Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8$\times$ rank range (0.7 pp, against 6.7 pp for TIES at 4B scale). Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06486 [cs.LG] |
| (or arXiv:2611.06486v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06486 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zainab Afolabi [view email]
[v1]
Wed, 4 Nov 2026 13:26:44 UTC (82 KB)
来源：arXiv cs.AI · arxiv.org