Do we need Label Regularization to Fine-tune Pre-trained Language Models? (2023)

Abstract: Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model. Considering the ever-growing size of pre-trained language models (PLMs), KD is often adopted in many NLP tasks involving PLMs. However, it is evident that in KD, deploying the teacher network during training adds to the memory and computational requirements of training. In the computer vision literature, the necessity of the teacher network is put under scrutiny by showing that KD is a label regularization technique that can be replaced with lighter teacher-free variants such as the label-smoothing technique. However, to the best of our knowledge, this issue is not investigated in NLP. Therefore, this work concerns studying different label regularization techniques and whether we actually need them to improve the fine-tuning of smaller PLM networks on downstream tasks. In this regard, we did a comprehensive set of experiments on different PLMs such as BERT, RoBERTa, and GPT with more than 600 distinct trials and ran each configuration five times. This investigation led to a surprising observation that KD and other label regularization techniques do not play any meaningful role over regular fine-tuning when the student model is pre-trained. We further explore this phenomenon in different settings of NLP and computer vision tasks and demonstrate that pre-training itself acts as a kind of regularization, and additional label regularization is unnecessary.

## introduction nowadays, we witness the tendency of evergrowing state-of-the-art neural networks. this is especially more evident in natural language processing (nlp): the famous gpt-3 @xcite has reached 175 billion parameters and a * equal contribution figure 1 : distilroberta results on the test set for the average of seven glue tasks. graph shows the mean performance and one standard deviation interval for the pre-trained and randomly initialized models computed over five runs. for the pre-trained model all intervals intersect, hence label regularization doesn't improve the performance, but for the model trained from scratch label regularization methods outperform base training. one can use the advantages of trained big neural networks and transfer their learned knowledge (weights and biases) to a smaller network. there are several approaches to such transferring techniques @xcite , but here we will focus on knowledge distillation (kd) @xcite , a prominent neural model compression technique, which has been applied in many different forms across various domains @xcite such as computer vision and nlp. to distill knowledge from a bigger model (a teacher) to a smaller model (a student), kd adds an extra loss term to ensure the student predictions match with the teacher output. in the nlp domain, kd is widely adopted for compressing pre-trained language models (plms) @xcite @xcite . the success of kd is attributed to different potential factors such as additional information presented by the dark knowledge (i.e. a term referring to the notion of class similarity information deriving from the teacher predictions which can not be found in the one-hot ground-truth labels) @xcite , regularization effect of the kd loss @xcite , or transferring inductive bias from one network to another @xcite . despite the widespread use of kd, it strongly depends on a trained teacher model, and calling the teacher during training adds to the computational cost of the training process noticeably. on the other hand, instead of adding the kd loss term to the student's loss, one can add a regularization term forcing the student predictions to be close to a uniform (or any arbitrary) distribution. such a label smoothing technique results in better calibrated and more accurate classifiers @xcite . recently, @xcite demonstrated that label smoothing can perform as well as or even outperform kd in several computer vision tasks and across various models. this result motivated us to investigate whether the teacher-free regularization techniques (tf) can work on par or better than kd on natural language understanding tasks. in this regard, we compare kd, label smoothing, and several other teacherfree methods for bert and gpt type models. it is worth mentioning that our setting is different from the one of @xcite : 1) pre-trained language models are generally much bigger than the models from machine vision, and 2) classification tasks in our setting are mostly binary or three classes, compared to a hundred classes in ci-far100 or two hundred in tiny imagenet. we ran the experiments multiple times to take into account the stochasticity of the training. overall, we show a similar pattern: teacher-free techniques perform on par with kd methods; but we additionally observed a surprisingly different phenomenon: the gap between base fine-tuning (without kd) and fine-tuning with label regularization (kd or tf) diminished. we explore the reasons why the base fine-tuning technique is a strong competitor of kd/tf regularization on nlu tasks. this situation is somewhat opposite to the one reported in computer vision. we hypothesized and tested the following potential explanations for our observations: 1) the small number of classes in glue datasets (usually 2 or 3 classes) in contrast to 10 or 100 classes in cv tasks; 2) language models are extensively pretrained while cv models are not. our experiments indicate that the second hypothesis is true, whereas the number of classes doesn't play a big role in the performance gap. to the best of our knowledge, the effect of pre-training on fine-tuning with label regularization was never mentioned in the literature and deserves additional study. overall, our main contributions in this paper are the following: ## background in this section, we give a brief overview of the kd and tf techniques we will be investigating. everywhere in the paper, we consider a classification problem with k classes. denote by q(x) the one-hot label of a data point x. ## experiments on glue benchmark inspired by the results of @xcite in machine vision, we wanted to investigate the performance of tf training on nlp data. in this section, we evaluate the performance of the methods introduced in the background section. ## related work our finding that pre-training reduces or even removes the gap between base training and tf/kd training can serve as an indication of a regularization property of pre-training. several works are exploring this in the literature. @xcite studied the relation between pretraining and spurious correlations. they demonstrated that pre-trained models are more robust to spurious correlations because they can generalize from a minority of training examples that counter the spurious pattern. @xcite demonstrated that masked language model pre-training helps in semantic parsing scenarios to improve compositional generalization. the authors hypothesize that the primary benefit provided by mlm pre-training is the improvement of the model's ability to substitute similar words or word phrases by ensuring they are close to each other in the representation space. @xcite showed that pre-training is very beneficial for smaller architectures, and fine-tuning pre-trained compact models can be competitive with more elaborate methods. ## discussion and future work we started this comparison of kd and tf regularizations on nlu tasks in the hope that a pattern we would like to add one important remark. our finding of this work does not suggest disregarding kd or other types of regularizations in nlp but rather using the more advanced or enhanced versions of these techniques. first of all, as shown in several works in the literature @xcite @xcite @xcite , kd is very important for the pre-training stage of the student models. similarly, @xcite demonstrated the value of label smoothing for training machine translation models. moreover, improved variants of kd might still facilitate fine-tuning of pre-trained models. even when vanilla kd doesn't give a statistically significant advantage over base fine-tuning, several works in the literature show that improved versions of kd with different auxiliary training schemes could be beneficial. for example, one can incorporate intermediate layer distillation @xcite @xcite , data augmentation @xcite or contrastive training @xcite . investigating better kd techniques or, more generally, better regularization methods that can improve the finetuning of plms even further will be an important direction for future work.