DHO：基于双头优化的视觉语言模型半监督知识蒸馏方法
Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
AI 导读
研究者提出双头优化（DHO）方法，通过为监督与蒸馏损失分别设置双预测头，解决视觉语言模型（VLM）知识蒸馏中的梯度冲突问题。在 15 个数据集上的实验显示，DHO 持续优于单头 KD 基线，较小的学生模型常能超过教师模型，并在 ImageNet 半监督学习及跨 ImageNet 变体的分布外泛化上取得 SOTA。代码与模型检查点已公开，论文被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Semi-supervised learning (SSL) has emerged as a practical solution for addressing data scarcity challenges by leveraging unlabeled data. Recently, vision-language models (VLMs), pre-trained on massive image-text pairs, have demonstrated remarkable zero-/few-shot performance that often surpasses SSL approaches due to their exceptional generalization capabilities. This gap motivates us to question: how can we effectively harness the powerful generalization capabilities of VLMs into task-specific models? Knowledge distillation (KD) offers a natural framework for transferring VLM capabilities, but we identify that it suffers from gradient conflicts between supervised and distillation losses. To address this challenge, we propose Dual-Head Optimization (DHO), which introduces dual prediction heads for each distinct signal. We observe that DHO mitigates gradient conflicts, enabling improved feature learning compared to single-head KD baselines, with practical benefits of minimal computational overhead and test-time hyperparameter tuning without retraining. Extensive experiments across 15 datasets show that DHO consistently outperforms KD baselines, often outperforming teacher models with smaller student models. DHO also achieves new state-of-the-art performance on both in-distribution ImageNet semi-supervised learning and out-of-distribution generalization across ImageNet variants. Code and model checkpoints are publicly available at this https URL.
| Comments: | Accepted at NeurIPS 2026. 52 pages, 17 figures. Updated author affiliations and abstract; added appendix studies on repeated-run reliability, additional VLM teachers, and nonlinear-head regularization |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2505.07675 [cs.LG] |
| (or arXiv:2505.07675v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.07675 arXiv-issued DOI via DataCite |
Submission history
From: Seongjae Kang [view email]
[v1]
Mon, 12 May 2025 15:39:51 UTC (10,480 KB)
[v2]
Tue, 30 Sep 2025 14:13:57 UTC (10,276 KB)
[v3]
Sat, 24 Oct 2026 06:51:32 UTC (12,164 KB)
来源：arXiv cs.AI · arxiv.org