  Training a small student network with the guidance of a larger teacher
network is an effective way to promote the performance of the student. Despite
the different types, the guided knowledge used to distill is always kept
unchanged for different teacher and student pairs in previous knowledge
distillation methods. However, we find that teacher and student models with
different networks or trained from different initialization could have distinct
feature representations among different channels. (e.g. the high activated
channel for different categories). We name this incongruous representation of
channels as teacher-student knowledge discrepancy in the distillation process.
Ignoring the knowledge discrepancy problem of teacher and student models will
make the learning of student from teacher more difficult. To solve this
problem, in this paper, we propose a novel student-dependent distillation
method, knowledge consistent distillation, which makes teacher's knowledge more
consistent with the student and provides the best suitable knowledge to
different student networks for distillation. Extensive experiments on different
datasets (CIFAR100, ImageNet, COCO) and tasks (image classification, object
detection) reveal the widely existing knowledge discrepancy problem between
teachers and students and demonstrate the effectiveness of our proposed method.
Our method is very flexible that can be easily combined with other
state-of-the-art approaches.
