  Recently, research efforts have been concentrated on revealing how
pre-trained model makes a difference in neural network performance.
Self-supervision and semi-supervised learning technologies have been
extensively explored by the community and are proven to be of great potential
in obtaining a powerful pre-trained model. However, these models require huge
training costs (i.e., hundreds of millions of images or training iterations).
In this paper, we propose to improve existing baseline networks via knowledge
distillation from off-the-shelf pre-trained big powerful models. Different from
existing knowledge distillation frameworks which require student model to be
consistent with both soft-label generated by teacher model and hard-label
annotated by humans, our solution performs distillation by only driving
prediction of the student model consistent with that of the teacher model.
Therefore, our distillation setting can get rid of manually labeled data and
can be trained with extra unlabeled data to fully exploit capability of teacher
model for better learning. We empirically find that such simple distillation
settings perform extremely effective, for example, the top-1 accuracy on
ImageNet-1k validation set of MobileNetV3-large and ResNet50-D can be
significantly improved from 75.2% to 79% and 79.1% to 83%, respectively. We
have also thoroughly analyzed what are dominant factors that affect the
distillation performance and how they make a difference. Extensive downstream
computer vision tasks, including transfer learning, object detection and
semantic segmentation, can significantly benefit from the distilled pretrained
models. All our experiments are implemented based on PaddlePaddle, codes and a
series of improved pretrained models with ssld suffix are available in
PaddleClas.
