  Knowledge distillation transfers knowledge from the teacher network to the
student one, with the goal of greatly improving the performance of the student
network. Previous methods mostly focus on proposing feature transformation and
loss functions between the same level's features to improve the effectiveness.
We differently study the factor of connection path cross levels between teacher
and student networks, and reveal its great importance. For the first time in
knowledge distillation, cross-stage connection paths are proposed. Our new
review mechanism is effective and structurally simple. Our finally designed
nested and compact framework requires negligible computation overhead, and
outperforms other methods on a variety of tasks. We apply our method to
classification, object detection, and instance segmentation tasks. All of them
witness significant student network performance improvement. Code is available
at https://github.com/Jia-Research-Lab/ReviewKD
