  Knowledge Distillation refers to a class of methods that transfers the
knowledge from a teacher network to a student network. In this paper, we
propose Sparse Representation Matching (SRM), a method to transfer intermediate
knowledge obtained from one Convolutional Neural Network (CNN) to another by
utilizing sparse representation learning. SRM first extracts sparse
representations of the hidden features of the teacher CNN, which are then used
to generate both pixel-level and image-level labels for training intermediate
feature maps of the student network. We formulate SRM as a neural processing
block, which can be efficiently optimized using stochastic gradient descent and
integrated into any CNN in a plug-and-play manner. Our experiments demonstrate
that SRM is robust to architectural differences between the teacher and student
networks, and outperforms other KD techniques across several datasets.
