N oisy T une: A Little Noise Can Help You Finetune Pretrained Language Models Better (2022)

Abstract: Effectively finetuning pretrained language models (PLMs) is critical for their success in downstream tasks. However, PLMs may have risks in overfitting the pretraining tasks and data, which usually have gap with the target downstream tasks. Such gap may be difficult for existing PLM finetuning methods to overcome and lead to suboptimal performance. In this paper, we propose a very simple yet effective method named NoisyTune to help better finetune PLMs on downstream tasks by adding some noise to the parameters of PLMs before fine-tuning. More specifically, we propose a matrix-wise perturbing method which adds different uniform noises to different parameter matrices based on their standard deviations. In this way, the varied characteristics of different types of parameters in PLMs can be considered. Extensive experiments on both GLUE English benchmark and XTREME multilingual benchmark show NoisyTune can consistently empower the finetuning of different PLMs on different downstream tasks.

## introduction in recent years, pretrained language models (plms) have achieved huge success in nlp @xcite . many plms such as bert @xcite , roberta @xcite and unilm @xcite which are pretrained from large-scale unlabeled corpus in a selfsupervised way, have significantly improve various downstream tasks such as reading comprehension @xcite , machine translation @xcite ), text classification @xcite , dialog @xcite and recommendation @xcite by finetuning on these tasks. how to effectively finetune plms to better empower downstream tasks is an important research problem @xcite . many existing nlp methods usually directly finetune plms with the * corresponding author. labeled data in downstream tasks @xcite . only a few works explore more effective and robust plm finetuning methods @xcite @xcite @xcite . for example, @xcite proposed recadam that adds a penalty item to minimize the l 2 distance between the finetuned models and the pretrained models, where the penalty intensity is time-variant during finetuning. @xcite proposed mixout which randomly replaces part of the parameters in the finetuned model with their original weights in the plms. these plm finetuning methods mainly focus on preventing plms from overfitting the limited labeled data in downstream tasks. besides the overfitting of downstream task data, a rarely studied problem is that the plms usually overfit the pretraining tasks and data @xcite , which may have significant gap with the downstream task and data. it is not easy for existing plm finetuning methods to overcome such gap @xcite , which may lead to suboptimal performance especially when labeled data in downstream tasks is insufficient. in order to handle this problem, in this paper we propose a very simple yet effective method named noisytune, which can help better finetune plms for downstream tasks. different from the standard finetuning paradigm (fig. 1 (a) ) which directly finetunes plms on the downstream task data, the key idea of noisytune is to add a small amount of noise to perturb plms parameters before finetuning (fig. 1 (b) ). it can help prevent plms from overfitting the tasks and data in the pretraining stage, and reduce the gap between pretraining and downstream tasks. since plms have different types of parameters which usually own different characteristics, in noisytune we use a matrix-wise perturbing method that adds uniform noise with different intensities to different parameter matrices according to their standard deviations for better adaptation. we conduct extensive experiments on two widely used nlp benchmarks, namely, glue @xcite for english language understanding and xtreme @xcite for multilingual language understanding. the results show noisytune can empower the finetuning of different plms on many different downstream nlp tasks to consistently achieve better performance. in addition, the results show noisytune can be easily combined with many existing plm finetuning methods and further improve their performance. ## noisytune the goal of noisytune is for more effective finetuning of plms on downstream tasks. the motivation of noisytune is that plms are well pretrained on some unlabeled corpus with some self-supervision tasks, and they may overfit these pretraining data and tasks @xcite , which usually have gap with the downstream task and data. it may be difficult for plms to effectively adapt to downstream tasks especially when labeled data in these tasks are limited, which is usually the case. motivated by the dueling bandits mechanism @xcite that adds randomness to the model for exploration, as shown in fig. 1 , we propose to add some noise to the parameters of plms before finetuning them on downstream tasks to do some "exploration" in parameter space and reduce the risk of overfitting the pretraining tasks and data. plms usually have different kinds of parameter matrices, such as query, key, value, and feedforward network matrices @xcite . different parameter matrices in the plms usually have different characteristics and scales. for example, some researchers found that the self-attention parameters and the feed-forward network parameters in transformers have very different properties, such as rank and density @xcite . thus, adding unified noise to all parameter matrices in plms may not be optimal for keeping their good model utility. to handle this challenge, we propose a matrix-wise perturbing method that adds noise with different intensities to different parameter matrices according to their variances. denote the parameter matrices (or scalars/vectors) in a plm as [w 1 , w 2 , ..., w n ], where n is the number of parameter matrix types. denote the perturbed version of the parameter matrix w i as wi , which is computed as follows: in addition, in some plms there are some constant matrices, such as token type embeddings in roberta @xcite . they will not be perturbed because their standard deviation is 0. it can ensure that these constant matrices will not be accidentally activated by additional noise. noisytune is a simple and general plug-and-play technique that can be applied to the finetuning of any plm on any task, simply by inserting the following pytorch-style code before finetuning: for name ,para in model . named parameters (): model . state dict [name ][:] += ( torch .rand(para.size())-0.5) * noise lambda * torch .std (para) 3 experiments ## conclusion in this paper, we propose a very simple but effective method named noisytune, which can help better finetune plms on downstream tasks by adding a little noise to them before finetuning. in noisytune, we propose a matrix-wise perturbing method that adds noise with different intensities to different kinds of parameter matrices in plms according to their variances. noisytune is a very general method, and is plm model agnostic, downstream task agnostic, and finetuning method agnostic. extensive experiments on both monolingual glue benchmark and multilingual xtreme benchmark demonstrate noisytune can consistently empower the finetuning of different plms on various downstream tasks to achieve better performance.