Skip to main content
Back to timeline
arXivSource publication:

Swapping the student classifier for angular coordinates: ACD lifts homogeneous distillation by 2.51 points and heterogeneous pairs by 4.19

Synopsis

Knowledge distillation usually has students imitate a teacher's classifier outputs, but raw logits entangle transferable inter-class directional relations with parameterization artifacts such as feature magnitudes, weight norms and biases, producing a classifier geometry mismatch. Angular Conditional Distillation (ACD) rebuilds the student head as a normalized cosine readout that removes magnitudes and biases, then distills standardized conditional non-target relations, passing relative class relations without absolute confidence. On six homogeneous CIFAR-100 teacher-student pairs ACD improves on ordinary KD by 2.51 percentage points on average, and on five heterogeneous pairs it exceeds the strongest competing methods by 4.19 points on average.

Source-provided article image: Angular Conditional Distillation: Overcoming Classifier Geometry Mismatch
arXiv

Interpretation

The student classifier no longer uses a linear readout carrying magnitudes and biases; it is rebuilt as a cosine classifier that keeps only the angle between features and class weights, so teacher and student align in a geometry-compatible space. Earlier logit distillation assumed teacher and student outputs lived in comparable coordinates and matched temperature-softened probability distributions directly. On six homogeneous CIFAR-100 teacher-student pairs ACD improves over ordinary KD by 2.51 percentage points on average; on five heterogeneous pairs it exceeds the strongest competing methods by 4.19 points on average.

The distillation objective keeps only conditional relative relations among non-target classes, and the student is no longer asked to copy the teacher's absolute confidence in the target class. Standard KL divergence constrains both target-class confidence and non-target relations, while target confidence is highly sensitive to the capacity gap. In the two ablations VGG13 to VGG8 and VGG13 to MobileNetV1, the complete ACI plus SICD configuration reaches 80.52% and 73.30%, 6.13 and 5.52 points above ordinary KD.

Class directions in the cosine head are initialized with an equiangular tight frame, placing all class weights at equal norm and identical pairwise angles in a maximally separated configuration at the start of training. Replacing only the head with the angular interface is not stable: in the ablation VGG8 gains 0.71 points while MobileNetV1 drops 1.18 points. In the same two ablations, adding ETF initialization raises accuracy by a further 0.70 and 1.50 points.

Perspective

The demonstrated scope is homogeneous and heterogeneous teacher-student pairs on CIFAR-100, ImageNet classification and transfer to MS-COCO object detection; the method changes only the student classifier head and output loss, leaving the teacher as a standard linear classifier.

On MS-COCO detection, ACD achieves the highest AP only for ResNet101 to ResNet18 and the highest AP50 across all three pairs, while SKD retains stronger AP on the other two pairs; in the two single-seed feature-distillation case studies the cosine interface alone reduces accuracy in the ReviewKD-style pipeline, so the gain cannot be attributed to the classifier change alone.