  Knowledge distillation provides an effective way to transfer knowledge via
teacher-student learning, where most existing distillation approaches apply a
fixed pre-trained model as teacher to supervise the learning of student
network. This manner usually brings in a big capability gap between teacher and
student networks during learning. Recent researches have observed that a small
teacher-student capability gap can facilitate knowledge transfer. Inspired by
that, we propose an evolutionary knowledge distillation approach to improve the
transfer effectiveness of teacher knowledge. Instead of a fixed pre-trained
teacher, an evolutionary teacher is learned online and consistently transfers
intermediate knowledge to supervise student network learning on-the-fly. To
enhance intermediate knowledge representation and mimicking, several simple
guided modules are introduced between corresponding teacher-student blocks. In
this way, the student can simultaneously obtain rich internal knowledge and
capture its growth process, leading to effective student network learning.
Extensive experiments clearly demonstrate the effectiveness of our approach as
well as good adaptability in the low-resolution and few-sample visual
recognition scenarios.
