多任务模型合并的误差子空间校准方法 ESCal
Error-Subspace Calibration for Multi-Task Model Merging
AI 导读
多任务模型合并后常与各专家模型存在性能差距。研究者提出误差子空间校准框架 ESCal，从专家模型与合并模型的权重差异中识别低维误差子空间，在每个中间块边界用桥接模块对表示进行加性校正，子空间基固定、仅学习线性映射，参数高效。视觉与自然语言理解基准实验显示，ESCal 在多种架构和合并方法下均提升合并模型性能，且每任务只需小规模校准集。
正文 · 原文
Abstract:Model merging combines task-specific expert models into a single model without joint training on their original datasets, but often incurs a performance gap relative to the individual experts. Post-merging calibration can reduce this gap through task-specific representation alignment, but the alignment objective alone does not specify which feature directions to correct. We propose Error-Subspace Calibration (ESCal), a post-merging calibration framework that identifies low-dimensional error subspaces from weight differences between expert and merged models. At each intermediate block boundary, a bridge module projects the merged representation onto the next block's input-projection error subspace and maps these coordinates to an additive correction in the current block's output-projection error subspace. The subspace bases remain fixed, so calibration learns only these linear maps, yielding a parameter-efficient mechanism for feature correction. We find that ESCal brings the merged model closer to the experts in both representations and predictions, indicating more faithful recovery of task-specific behavior. Experiments on vision and natural language understanding benchmarks show that ESCal improves merged-model performance across diverse architectures and merging methods using a small calibration set for each task.
| Comments: | 25 pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33483 [cs.LG] |
| (or arXiv:2610.33483v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33483 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yeseul Cho [view email]
[v1]
Sat, 24 Oct 2026 06:17:15 UTC (447 KB)
来源：arXiv cs.AI · arxiv.org