  Knowledge distillation is a popular paradigm for learning portable neural
networks by transferring the knowledge from a large model into a smaller one.
Most existing approaches enhance the student model by utilizing the similarity
information between the categories of instance level provided by the teacher
model. However, these works ignore the similarity correlation between different
instances that plays an important role in confidence prediction. To tackle this
issue, we propose a novel method in this paper, called similarity transfer for
knowledge distillation (STKD), which aims to fully utilize the similarities
between categories of multiple samples. Furthermore, we propose to better
capture the similarity correlation between different instances by the mixup
technique, which creates virtual samples by a weighted linear interpolation.
Note that, our distillation loss can fully utilize the incorrect classes
similarities by the mixed labels. The proposed approach promotes the
performance of student model as the virtual sample created by multiple images
produces a similar probability distribution in the teacher and student
networks. Experiments and ablation studies on several public classification
datasets including CIFAR-10,CIFAR-100,CINIC-10 and Tiny-ImageNet verify that
this light-weight method can effectively boost the performance of the compact
student model. It shows that STKD substantially has outperformed the vanilla
knowledge distillation and has achieved superior accuracy over the
state-of-the-art knowledge distillation methods.
