跳到正文
arXiv cs.AI· Tao Hu, Ling-Xiang Zhang, Da-Wei Zhou·· 4 小时前AI 评分31

CoPre-MoE:面向多模态持续指令微调的专家整合与保留方法

CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning

AI 导读

arXiv 论文提出 CoPre-MoE,一种用于多模态持续指令微调的 MoE 方法,通过将跨任务功能重叠的专家整合进固定容量的通用专家池,并把任务专家分解为共享投影基与紧凑任务核心,缓解专家冗余和参数线性增长。它用可学习检索键构建路由器,历史键冻结、新增键施加正交惩罚以抑制灾难性遗忘,推理时无需测试时任务标签即可动态组合两类专家池。实验显示其在降低参数开销的同时取得更优性能。

正文 · 原文

View PDF HTML (experimental)

Abstract:Multimodal continual instruction tuning requires multimodal large language models to sequentially acquire new capabilities without catastrophic forgetting. Mixture-of-experts (MoE) architectures offer a promising way to mitigate task interference through expert specialization and combination. However, while existing MoE-based methods avoid overwriting shared parameters by allocating separate experts for each task, this progressive expansion incurs severe cross-task redundancy and linear parameter growth. Specifically, experts across tasks often learn overlapping functions alongside task-specific capabilities. For instance, ScienceQA and TextVQA both locate relevant image regions. To curb parameter redundancy without inducing forgetting, we propose Consolidating and Preserving MoE (CoPre-MoE). For each incoming task, CoPre-MoE first learns task-specific MoE to capture specialized capabilities. To eliminate redundancy and reduce parameter growth across tasks, we design a shared-and-private expert architecture. Specifically, we first evaluate functional overlap to selectively consolidate overlapping experts across tasks into a fixed-capacity generic expert pool. We then factorize the remaining task-specific experts into shared projection bases and compact task-specific cores within a private expert pool. To mitigate forgetting, we construct the router by pairing each expert with learnable retrieval keys, where historical keys remain frozen during continual updates while newly added keys are regularized with an orthogonality penalty. At inference, input-dependent routing dynamically combines both pools without test-time task labels. Extensive experiments demonstrate that CoPre-MoE achieves superior performance while substantially reducing parameter overhead.
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2611.18417 [cs.CV]
  (or arXiv:2611.18417v1 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2611.18417

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Tao Hu [view email]
[v1] Fri, 13 Nov 2026 16:58:35 UTC (639 KB)

来源:arXiv cs.AI · arxiv.org