跳到正文
arXiv cs.AI· Babak Barazandeh, Subhabrata Majumdar, Vinay Prithyani, George Michailidis·· 2 小时前AI 评分40

Localized LoRA-MoE:自适应路由的低秩专家混合方法

Localized LoRA-MoE: Mixture of Low-Rank Experts With Adaptive Routing

AI 导读

研究者提出 Localized LoRA-MoE 框架,将局部空间分块与上下文条件专家路由结合,提出 Block-Wise(集中路由)和 Cell-Wise(去中心路由)两种架构变体。

正文 · 原文

View PDF HTML (experimental)

Abstract:Parameter-efficient fine-tuning (PEFT) methods such as LoRA adapt large models through a single low-rank update. Such updates are susceptible to gradient interference: when a training stream interleaves multiple operational contexts, conflicting updates to shared adapter weights collapse the adapter into an unspecialized average. While recent methods introduce block-wise isolation to remedy this issue, their topologies are static and cannot adapt to context switching or localized sensor degradation. In this work, we introduce Localized LoRA-MoE, a framework that combines localized spatial blocking with context-conditioned expert routing. We propose and evaluate two architectural variants: Block-Wise LoRA-MoE (centralized routing), which modulates the entire structural grid through a single context-conditioned gate, and Cell-Wise LoRA-MoE (decentralized routing), which assigns an independent gate to every coordinate cell of the matrix grid. Across three structured transformation benchmarks--a high-dimensional SVD matrix simulation, context-switched transformations over real tabular data, and spatial reconstruction under simulated sensor occlusion--both variants outperform static baselines under comparable adapter budgets. Both variants perform similarly when context shifts the grid uniformly, while cell-wise routing is strictly better when contexts are spatially heterogeneous. When fine-tuning a language model on six interleaved GLUE tasks, task-conditioned routing at a fixed adapter budget outperforms LoRA and static localized adapters, matching per-task adapters with 6x budgets. At decoder scale on Qwen2.5-1.5B, both routed variants outperform static baselines across eight commonsense benchmarks, with cell-wise routing achieving +3.9 points over LoRA at matched active capacity.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2607.05114 [cs.LG]
  (or arXiv:2607.05114v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2607.05114

arXiv-issued DOI via DataCite

Submission history

From: Babak Barazandeh [view email]
[v1] Mon, 6 Jul 2026 14:06:18 UTC (291 KB)
[v2] Tue, 17 Nov 2026 18:47:55 UTC (438 KB)

来源:arXiv cs.AI · arxiv.org