Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm (2022)

Abstract: Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.

## introduction recently, the emergence of transformer-based language models (using pretrain-and-finetune paradigm) such as bert @xcite and gpt-3 @xcite have revolutionized and established state-of-the-art (sota) records (beyond human-level) on various natural language (nlp) processing tasks. these models are first pre-trained in a self-supervised fashion on a large corpus and fine-tuned for specific downstream tasks @xcite . while effective and prevalent, they suffer from redundant computation due to the heavy model size, which hinders their popularity on resource-constrained devices, e.g., mobile phones, smart cameras, and autonomous driving @xcite @xcite @xcite . various weight pruning approaches (zeroing out certain weights and then optimizing the rest) have been proposed to reduce the footprint requirements of transformers (zhu and gupta, 2018; blalock pruning under pretrain-and-finetune. in the subfigures, the cylinders on the left describe the pruning process, and the circles on the right represent the knowledge analysis of the sparse model. @xcite @xcite @xcite . conventional wisdom in pruning states that pruning reduces the overfitting risk since the compressed model structures are less complex, have fewer parameters and are believed to be less prone to overfit @xcite @xcite . however, under the pretrain-and-finetune paradigm, most pruning methods understate the overfitting problem. in this paper, we postulate a counter-traditional hypothesis, that is: model pruning increases the risk of overfitting if pruning is performed at the fine-tuning phase. as shown in figure 1b , the pretrain-and-finetune paradigm contains two types of knowledge, the general-purpose language knowledge learned during pre-training (l) and the taskspecific knowledge from the downstream task data (d). compared to conventional pruning that only discards task-specific knowledge (figure 1a ), pruning under pretrain-and-finetune (figure 1b ) discards extra knowledge (red area) learned in pretraining phase. thus, to recover both the extra discarded general-purpose knowledge and the discarded task-specific knowledge, pruning under pretrain-and-finetune increases the amount of information a model needs, which results in relative data deficiency, leading to a higher risk of overfitting. to empirically verify the overfitting problem, we visualize the training and evaluation performance on a real-world task data of mrpc @xcite in figure 2 . from figure 2 (b), it is observed that the evaluation accuracy on the training dataset remains improved while it keeps the same for the validation set through the training process. from figure 2 (c), the difference in performance becomes more significant when the pruning rate becomes higher and the performance on the validation set even becomes worse after 2,000 training steps. all these observations verify our hypothesis. the main question this paper attempts to answer is: how to reduce the risk of overfitting of pre-trained language models caused by pruning? however, answering this question is challenging. first, under the pretrain-and-finetune paradigm, both the general-purpose language knowledge and the task-specific knowledge are learned. it is nontrivial to keep the model parameters related to both knowledge when pruning. second, the amount of data for downstream tasks can be small, such as the data with privacy. thus, the overfitting problem can easily arise, especially in the face of high pruning rate requirements. a little recent progress has been made on addressing overfitting associated with model compression. however, their results are not remarkable and most of them focus on the vision domain @xcite . to address these challenges, we propose spd, a sparse progressive distillation method, for pruning pre-trained language models. we prune and optimize the weight duplicates of the backbone of the teacher model (a.k.a., student modules). each student module shares the same architecture (e.g., the number of weights, the dimension of each weight) as the duplicate. we replace the corresponding layer(s) of the duplicated teacher model with the pruned sparse student module(s) in a progressive way and name the new model as a grafted model. we validate our proposed method through the ablation studies and the glue benchmark. experimental results show that our method outperforms the existing approaches. we summarize our contributions as follows: ## related work to summarize, our contribution is determining the overfitting problem of pruning under the pretrainand-finetune paradigm and proposing the sparse progressive distillation method to address it. we demonstrate the benefits of the proposed framework through the ablation studies. we validate our method on eight datasets from the glue benchmark. to test if our method is applicable across teacher 𝒇 𝑻 input output pruning (𝜃 " ) grafted model 𝒇 𝑮 pruning (𝜃 $ ) zero non-zero input output update pruning(𝜃 % ) … pruning(𝜃 & ) input output input output input output … input output input output input output p @xmath0.25 p @xmath1.50 p @xmath2.25 p @xmath3.25 p @xmath4.50 grafted model 𝒇 𝑮 student modules grafted model 𝒇 𝑮 grafted model 𝒇 𝑮 grafted model 𝒇 𝑮 grafted model 𝒇 𝑮 grafted model 𝒇 𝑮 final grafted model 𝒇 𝑮 ! input output update update update update student modules student modules student modules student modules student modules student modules student modules student modules (a) (e) (b) (c) (d) tasks, we include the tasks of both single sentence and sentence-pair classification. experimental results show that our method outperforms the leading competitors by a large margin. network pruning. common wisdom has shown that weight parameters of deep learning models can be reduced without sacrificing accuracy loss, such as magnitude-based pruning and lottery ticket hypothesis @xcite . (zhu and gupta, 2018) compared small-dense models and large-sparse models with the same parameters and showed that the latter outperforms the former, showing the large-sparse models have better expressive power than their small-dense counterparts. however, under the pretrain-and-finetune paradigm, pruning leads to overfitting as discussed. knowledge distillation (kd). as a common method in reducing the number of parameters, the main idea of kd is that the small student model mimics the behaviour of the large teacher model and achieves a comparable performance @xcite . @xcite @xcite utilized kd to learn universal language representations from large corpus. however, current sota knowledge distillation methods are not able to achieve a high model compression rate (less than 10% remaining weights) while achieving an insignificant performance decrease. progressive learning. the key idea of progressive learning is that student learns to update module by module with the teacher. @xcite utilized a dual-stage distillation scheme where student modules are progressively grafted onto the teacher network, it targets the few-shot scenario and uses only a few unlabeled samples to achieve comparable results on cifar-10 and cifar-100. @xcite gradually increased the probability of replacing each teacher module with their corresponding student module and trained the student to reproduce the behavior of the teacher. however, the performance on transformer-based models of the aforementioned first method is unknown while the second method has an obvious performance drop with a low sparsity (50%). ## conclusion in this paper, we postulate a counter-traditional hypothesis that pruning increases the risk of overfitting under the pretrain-and-finetune paradigm. we analyze and empirically verify this hypothesis, and propose a sparse progressive pruning method to address the overfitting problem. we theoretically analyze that our pruning method can obtain a subnetwork from the student model that has a similar accuracy as the teacher. we study and minimize the interference between different hyperparameter strategies, including pruning rate, learning rate, and grafting probability. a number of ablation studies and experimental results on eight tasks from the glue benchmark demonstrate the superiority of our method over the leading competitors.