DuplexTune：面向 MoE 大语言模型的高效微调框架，提出耦合状态回放方法
DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation
AI 导读
arXiv 论文提出 DuplexTune 框架，通过耦合隐状态回放实现对预训练 MoE 语言模型的省显存微调，采用可训练宽度桥接双残差流，先桥接预热再适配骨干，路由权重保持冻结。
正文 · 原文
Abstract:Sparse expert execution does not remove the stored states needed to adapt a mixture-of-experts language model. We present DuplexTune, a framework for replaying coupled hidden states through pretrained attention and expert modules. Trainable width bridges connect two residual streams to the backbone. Bridge warm-up precedes backbone adaptation, with router weights kept frozen. We analyze the implicit state-recovery equation and identify the convergence and replay conditions needed for reconstruction. On Qwen1.5-MoE-A2.7B, the single-H800 evaluation reports 38.8 GB peak memory, versus 64.8 GB for supervised fine-tuning with checkpointing, and throughput of 25.1 versus 19.9 samples/s. DuplexTune achieves 67.2% MMLU, 75.6% GSM8K, and 7.69 MT-Bench, with a weaker multilingual score than the checkpointed baseline. The results characterize a configuration-dependent tradeoff among state storage, throughput, and adaptation quality.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.36009 [cs.LG] |
| (or arXiv:2610.36009v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36009 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Paul Raj [view email]
[v1]
Mon, 26 Oct 2026 16:17:33 UTC (1,321 KB)
来源：arXiv cs.AI · arxiv.org