Weight Distillation: Transferring the Knowledge in Neural Network Parameters (2021)

Abstract: Knowledge distillation has been proven to be effective in model acceleration and compression. It transfers knowledge from a large neural network to a small one by using the large neural network predictions as targets of the small neural network. But this way ignores the knowledge inside the large neural networks, e.g., parameters. Our preliminary study as well as the recent success in pre-training suggests that transferring parameters are more effective in distilling knowledge. In this paper, we propose Weight Distillation to transfer the knowledge in parameters of a large neural network to a small neural network through a parameter generator. On the WMT16 En-Ro, NIST12 Zh-En, and WMT14 En-De machine translation tasks, our experiments show that weight distillation learns a small network that is 1.88 2.94x faster than the large network but with competitive BLEU performance. When fixing the size of small networks, weight distillation outperforms knowledge distillation by 0.51 1.82 BLEU points.

## introduction knowledge distillation (kd) is a popular model acceleration and compression approach @xcite . it assumes that a lightweight network (i.e., student network, or student for short) can learn to generalize in the same way as a large network (i.e., teacher network, or teacher for short). to this end, a simple method is to train the student network with predicted probabilities of the teacher network as its targets. but kd has its limitation: the student network can only access the knowledge in the predictions of the teacher network. it does not consider the knowledge in the teacher network parameters. these parameters contain billions of entries for the teacher network to make predictions. yet in kd the student only learns from those predictions with at most thousands of categories. this way results in an inferior student network, since it learns from the limited training signals. our analysis in section 5.1 shows that kd performs better if we simply cut off parts of parameters from the teacher to initialize the student. this fact implies that the knowledge in parameters is complementary to kd but missed. it also agrees with the recent success in pre-training @xcite @xcite , where parameters reusing plays the main role. based on this observation, a superior student is expected if all parameters in the teacher network could be exploited. however, this imposes a great challenge as the student network is too small to fit in the whole teacher network. to fully utilize the teacher network, we propose weight distillation (wd) to transfer all the parameters of the teacher network to the student network, even if they have different numbers of weight matrices and (or) these weight matrices are of different shapes. we first use a parameter generator to predict the student network parameters from the teacher network parameters. after that, a finetuning process is performed to improve the quality of the transferred parameters. see fig. 1 for a comparison of kd and wd. we test the wd method in a well-tuned transformer-based machine translation system. the experiments are run on three machine translation benchmarks, including the wmt16 english-roman (en-ro), nist12 chinese-english (zh-en), and wmt14 english-german (en-de) tasks. with a similar speedup, the student network trained by wd achieves bleu improvements of 0.51∼1.82 points over kd. with similar bleu performance, the student network trained by wd is 1.11∼1.39× faster than kd. more interestingly, it is found that wd is very effective in improving the student network when its model size is close to the teacher network. on the wmt14 en-de test data, our wdbased system achieves a strong result (a bleu score of 30.77) but is 1.88× faster than the big teacher network. ## analysis to better understand wd, we conduct a series of experiments on the nist12 zh-en validation set with the transformer-base teacher. ## conclusion in this work, we propose weight distillation to transfer knowledge in the parameters of the teacher network to the student network. it generates the student network from the teacher network via a parameter generator. our experiments on three machine translation tasks show that weight distillation consistently outperforms knowledge distillation by producing a faster and better student network.