Sparse Teachers Can Be Dense with Knowledge (2022)

Abstract: Recent advances in distilling pretrained language models have discovered that, besides the expressiveness of knowledge, the student-friendliness should be taken into consideration to realize a truly knowledgeable teacher. Based on a pilot study, we find that over-parameterized teachers can produce expressive yet student-unfriendly knowledge and are thus limited in overall knowledgeableness. To remove the parameters that result in student-unfriendliness, we propose a sparse teacher trick under the guidance of an overall knowledgeable score for each teacher parameter. The knowledgeable score is essentially an interpolation of the expressiveness and student-friendliness scores. The aim is to ensure that the expressive parameters are retained while the student-unfriendly ones are removed. Extensive experiments on the GLUE benchmark show that the proposed sparse teachers can be dense with knowledge and lead to students with compelling performance in comparison with a series of competitive baselines.

## introduction pretrained language models (lms) built upon transformers @xcite @xcite have achieved great successes. however, the appealing performance is usually accompanied with expensive computational costs and memory footprints, which can be alleviated by model compression @xcite . knowledge distillation @xcite , as a dominant method in model compression, concentrates on transferring knowledge from a teacher of large scale to a student of smaller scale. @xcite of bert base and bert large at small sparsity levels. task metric and output distribution variance are used as the measures of performance and confidence, respectively. distribution variance is comparatively equivalent to distribution negative entropy as employed in @xcite . proof of the equivalence can be found in appendix a. conventional studies @xcite mainly expect that the expressive knowledge would be well transferred, yet largely neglecting the existence of student-unfriendly knowledge. recent attempts @xcite are made to adapt the teacher to more student-friendly knowledge and have yielded performance gains. based on these observations, we posit that over-parameterized lms, on the one hand, can produce expressive knowledge due to over-parameterization, but on the other hand can also produce student-unfriendly knowledge due to over-confidence @xcite . from a pilot study shown in figure 1 , we find that lms of large scale tend to have a good performance and high confidence, and that both performance and confidence can be degraded through randomly sparsifying a small portion of parameters. 2 this indicates that some parameters resulting in student-unfriendliness can be rather removed, to improve student-friendliness of the teacher without sacrificing too much its expressiveness. motivated by this finding, we propose a sparse teacher trick (in short, stark ) under the guidance of an overall knowledgeable score for each teacher parameter, which accords not only with the expressiveness but also the student-friendliness of the parameter by interpolation. the aim is to retain the expressive parameters while removing the student-unfriendly ones. specifically, we introduce a three-stage procedure consisting of 1) trial distillation, 2) parameter sparsification, and 3) actual distillation. the trial distillation distills the dense teacher to the student so that a trial student is obtained. the parameter sparsification first estimates the expressiveness score and student-friendliness score of each teacher parameter via feedbacks respectively from the teacher itself and the trial student, and then sparsifies the teacher by removing the parameters associated with adequately low interpolated knowledgeable scores. the actual distillation distills the sparsified teacher to the student so that an actual student is obtained, where the student is initialized in the same manner as that used in trial distillation following the commonly-used rewinding technique @xcite . we conduct an extensive set of experiments on the glue benchmark. experimental results demonstrate that the sparse teachers can be dense with knowledge and lead to a remarkable performance of students compared with a series of competitive baselines. ## sparse teacher trick our trick involves three stages in the student learning procedure as shown in figure 2 . first, we distil a trial student from the dense teacher on a specific task (trial distillation). then, we sparsify the parameters of the dense teacher that are associated with adequately low knowledgeable scores (parameter sparsification). finally, rewinding is applied, where the student is set to the initialization exactly used in the trial distillation stage and is learned from the sparse teacher during (actual distillation). ## conclusions in this paper, we validate that sparse teachers can be dense with knowledge under the guidance of our designed knowledgeable score. the idea of the sparse teacher is motivated from a pilot study, and the knowledgeable score is carefully crafted to make sure that the student-unfriendly knowledge can be reduced without hurting too much the expressive knowledge. extensive experimental results on the glue benchmark support our claim to a large degree.