角度条件蒸馏(ACD):解决分类器几何失配问题
Angular Conditional Distillation: Overcoming Classifier Geometry Mismatch
AI 导读
一篇 arXiv 论文提出角度条件蒸馏(ACD)方法,解决知识蒸馏中分类器几何失配导致的迁移不可靠问题。方法包含两个模块:角度坐标接口(ACI)用角度坐标重建学生分类器以抑制幅值与偏置影响,尺度不变条件蒸馏(SICD)标准化输出并只迁移条件性非目标关系。ACD 在实验中达到优于现有 KD 方法的 SOTA 表现,可作为通用即插即用替换,无需特征适配器且不增加推理成本,代码已开源。
正文 · 原文
Abstract:Classifier outputs provide a fundamental and univer- sally available knowledge interface for knowledge distil- lation (KD). However, standard logit objectives entangle transferable semantic relations with architecture-dependent parameterization artifacts, causing a classifier geometry mismatch that makes direct knowledge transfer unreliable. To address this, we propose Angular Conditional Distilla- tion (ACD). First, the Angular Coordinate Interface (ACI) reconstructs the student classifier using angular coordi- nates, suppressing magnitude and bias effects to provide a geometry-compatible space. Then, Scale-Invariant Condi- tional Distillation (SICD) standardizes outputs and exclu- sively transfers conditional non-target relations, filtering out untransferable absolute confidence. Together, ACI de- fines where to transfer, while SICD defines what to transfer. Extensive experiments show that ACD achieves state-of-the- art performance over existing KD methods. Moreover, as a universal drop-in replacement, it consistently improves existing distillation pipelines while requiring no feature adapters or additional inference-time cost. Code is avail- able at this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23941 [cs.CV] |
| (or arXiv:2611.23941v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23941 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xuefei Zhang [view email]
[v1]
Wed, 18 Nov 2026 06:28:17 UTC (2,345 KB)
来源:arXiv cs.AI · arxiv.org