arXiv cs.AI· Ziqing Wen, Xiaohui Song, Jiahuan Wang, Ping Luo, Junlin Zeng, Dongsheng Li, Tao Sun·· 3 小时前AI 评分38
WIS-MeZO：基于 Wanda 显著性的稀疏扰动高效零阶优化方法
Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization
AI 导读
研究者提出 WIS-MeZO，利用 Wanda 权重-激活显著性构建任务条件化的稀疏扰动掩码，在保持预训练模型稠密的同时降低大语言模型零阶微调的搜索维度。在 3B 到 32B 参数的 LLaMA 和 Qwen 模型的 16 个 benchmark 上，该方法取得高于 MeZO 的总得分，并以更少优化步数达到相当的验证损失。与随机及幅度稀疏扰动的对比凸显了信息性坐标选择的重要性。
正文 · 原文
Abstract:Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical. We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight--activation saliency to construct task-conditioned sparse perturbation masks while keeping the pretrained model dense. We analyze support quality through gradient energy capture and relate it to gradient estimation and convergence. Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps. Comparisons with random and magnitude-based sparse perturbations further highlight the importance of informative support selection.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16317 [cs.LG] |
| (or arXiv:2611.16317v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16317 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ziqing Wen [view email]
[v1]
Thu, 12 Nov 2026 05:25:58 UTC (978 KB)
来源：arXiv cs.AI · arxiv.org