arXiv cs.AI· Seongjae Kang, Dong Bok Lee, Hyungjoon Jang, Dongseop Kim, Sung Ju Hwang·· 4 小时前AI 评分37
PCoreSet：借助视觉语言模型知识蒸馏实现高效主动学习
PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
AI 导读
研究者提出 ActiveKD 框架与 PCoreSet 选样策略，将主动学习与知识蒸馏结合，利用大型视觉语言模型（VLM）的零样本和少样本能力。PCoreSet 在概率空间而非特征空间最大化覆盖，选出概率多样的未标注样本。
正文 · 原文
Abstract:Knowledge distillation (KD) is a widely used framework for training compact, task-specific models by transferring the knowledge from teacher models. However, its application to active learning (AL), which aims to minimize annotation costs through iterative sample selection, remains underexplored. This gap stems from the fact that KD typically assumes access to sufficient labeled data, whereas AL operates in data-scarce scenarios where task-specific teacher models are often unavailable. In this paper, we first introduce ActiveKD, a framework that integrates AL with KD by leveraging the zero- and few-shot capabilities of large vision-language models (VLMs). A key aspect of ActiveKD is the structured prediction bias of VLMs--i.e., their predictions form clusters in the probability space. We regard this structure as an inductive bias of the teacher model, capturing generalizable output patterns beneficial to student learning. To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection strategy that maximizes coverage in the probability space rather than the feature space. PCoreSet strategically selects probabilistically diverse unlabeled samples, facilitating more efficient transfer of teacher knowledge under limited annotation budgets. Extensive evaluations on 11 datasets show that ActiveKD consistently improves every selection method over the no-distillation baseline (e.g., +29.07% on ImageNet, averaged over methods). Under ActiveKD, several widely-used AL selectors underperform Random and none consistently beats it; in contrast, PCoreSet is the only selector that consistently outperforms Random across 5 students and 3 teachers, always achieving the best performance except for the first 2 AL rounds. Our code is available at this https URL.
| Comments: | 46 pages, 24 figures, preprint. Updated manuscript and author affiliations; clarified ActiveKD/PCoreSet comparisons and the ClassBalanced baseline; expanded analysis and implementation details |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2506.00910 [cs.LG] |
| (or arXiv:2506.00910v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.00910 arXiv-issued DOI via DataCite |
Submission history
From: Seongjae Kang [view email]
[v1]
Sun, 1 Jun 2025 08:54:37 UTC (15,767 KB)
[v2]
Wed, 1 Oct 2025 01:14:57 UTC (15,716 KB)
[v3]
Sat, 24 Oct 2026 07:37:20 UTC (15,654 KB)
来源：arXiv cs.AI · arxiv.org