Lifting the Curse of Capacity Gap in Distilling Language Models (2023)

Abstract: Pretrained language models (LMs) have shown compelling performance on various downstream tasks, but unfortunately they require a tremendous amount of inference compute. Knowledge distillation finds a path to compress LMs to small ones with a teacher-student paradigm. However, when the capacity gap between the teacher and the student is large, a curse of capacity gap appears, invoking a deficiency in distilling LMs. While a few studies have been carried out to fill the gap, the curse is not yet well tackled. In this paper, we aim at lifting the curse of capacity gap via enlarging the capacity of the student without notably increasing the inference compute. Largely motivated by sparse activation regime of mixture of experts (MoE), we propose a mixture of minimal experts (MiniMoE), which imposes extra parameters to the student but introduces almost no additional inference compute. Experimental results on GLUE and CoNLL demonstrate the curse of capacity gap is lifted by the magic of MiniMoE to a large extent. MiniMoE also achieves the state-of-the-art performance at small FLOPs compared with a range of competitive baselines. With a compression rate as much as ~50 \times , MiniMoE preserves ~95% GLUE score of the teacher.

## introduction pretrained language models (lms) have become a popular choice for various downstream tasks, e.g., text classification, token classification, and question answering @xcite @xcite . unfortunately, appealing performance comes with a huge cost of inference compute due to the scale of lms. knowledge distillation @xcite , as an alternative to model pruning @xcite and quantization @xcite , discovers a the curse of capacity gap refers to a deficiency that a larger teacher might unexpectedly result in a poorer student especially when the capacity gap between the teacher and the student is large @xcite , as illustrated in to the demand, we aim at lifting the curse of capacity gap via enlarging the capacity of the student without notably increasing the inference compute. we propose a mixture of minimal experts (minimoe), inspired by the intuition of sparse activation of mixture of experts (moe) @xcite . thanks to that the activation process can be parallel on either single or multiple devices @xcite , minimoe on the one hand imposes extra parameters to the student, but on the other hand introduces negligibly additional inference compute brought by routing algorithm. to our best knowledge, this is the first work aiming at lifting the curse completely. experiments are conducted on glue @xcite and @xcite . the results exhibit that minimoe largely lifts the curse of the gap as in ## curse of capacity gap the curse of capacity gap is not new but is already recognized in studies on vision model distillation @xcite . while a hit-the-mind drawback of the curse is that the performance of distilling to a small student can be dramatically worse than that of distilling to a slightly larger one, a rather counterintuitive deficiency is invoked as that the performance of distilling from a large teacher can be unexpectedly worse than that of distilling from a smaller one (i.e., large teacher, poor student). we here give a minor theoretical justification on the curse, as a plus to the empirical justification. proposition 1 (vc dimension theory, @xcite . assuming that the teacher function is f @xmath0, the labeling function is @xmath1, and the data is d, we have: proposition 2 (generalized distillation theory, @xcite . additionally providing that the student function is f @xmath2, we have: theorem 1. the bound for the student function at a learning rate can be written as: the proof is rather straightforward by combining proposition 1 and 2. remark 1. under the same distillation setting, we can ignore the estimation error. when we compare two students of different capacities distilled from a teacher of the same capacity, the student of a smaller capacity has a larger ϵ g thus lower performance. when we compare two students of the same capacities distilled from teachers of different capacities, the student distilled from the teacher of a larger capacity has a smaller ϵ t yet a larger ϵ g thus a tradeoff. remark 1 basically tells that a tradeoff is associated with the increase of teacher capacity, implying that increasing teacher capacity would first lead to improved but then degraded student performance. this tradeoff naturally corresponds with the curse. on the other hand, it is accepted that large capacity gap is a pain and is processed in literature of lm distillation @xcite @xcite . being unaware of the curse of capacity gap, these studies attempt to offer student-friendly teachers by either interpolating teacher assistants @xcite or adapting teacher knowledge @xcite . the unawareness is largely due to a fun fact that they only distil lms like bert base , but neglect the scalability to lms like bert large especially when the student is small. though the performance of student can be boosted in this way, the curse still remains in lm distillation as in figure 2 . other related work in knowledge distillation is given in appendix e. embarrassingly, while the curse is claimed to be tackled in vision model distillation @xcite @xcite , our preliminary study (cf. ## conclusions in this work, we uncover a curse of capacity gap in lm distillation, which is well discussed in previous studies on vision model distillation but not recognized in distilling lms. while there are some studies investigating to fill the gap, we find they can hardly tackle the curse. interestingly, existing solutions in large vision language model distillation which are stated to be able to lift the curse fail to achieve so for lms. so we aim at lifting the curse by proposing a well-motivated minimoe. the minimoe can essentially enlarge the capacity of the student but leave the inference compute nearly untouched. our experimental results indicate that minimoe can not only lift the curse but also realize new state of the arts.