H y P e: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation (2023)

Abstract: Language models with the Transformers structure have shown great performance in natural language processing. However, there still poses problems when fine-tuning pre-trained language models on downstream tasks, such as over-fitting or representation collapse. In this work, we propose HyPe, a simple yet effective fine-tuning technique to alleviate such problems by perturbing hidden representations of Transformers layers. Unlike previous works that only add noise to inputs or parameters, we argue that the hidden representations of Transformers layers convey more diverse and meaningful language information. Therefore, making the Transformers layers more robust to hidden representation perturbations can further benefit the fine-tuning of PLMs en bloc. We conduct extensive experiments and analyses on GLUE and other natural language inference datasets. Results demonstrate that HyPe outperforms vanilla fine-tuning and enhances generalization of hidden representations from different layers. In addition, HyPe acquires negligible computational overheads, and is better than and compatible with previous state-of-the-art fine-tuning techniques.

## introduction pretrain-then-finetune has become the mainstream paradigm in recent natural language processing (nlp) practices, and there emerges various pretrained language models (plms) such as bert @xcite , roberta @xcite , and xlnet @xcite . vanilla plm finetuning with common strategies (e.g., dropout @xcite and adamw @xcite ) can empower plms with excellent downstream performance. however, vanilla finetuned plms acquire performances with large variances on the downstream tasks @xcite . such unstable performances may results from overfitting or representation collapse @xcite . these problems can be aggravated in lowresource scenarios @xcite . in recent literature, effective fine-tuning techniques have been proposed to improve the performance and generalization (transferability) of finetuned plms @xcite @xcite . besides other explicit regularization, adding noise is a widely-used strategy to smoothen the optimization landscape and mitigate over-fitting. for example, some works apply the perturbation to pre-trained parameter weights (e.g., noisytune @xcite ), input embedding features (e.g., r3f @xcite ) or gradients (e.g., childtuning @xcite ) during the fine-tuning process. injecting noise to input features is a conventional technique for generalization and can be seen as implicit parameter regularization @xcite . common plms are stacked basic neural network layers (i.e., transformer layers @xcite ), and previous research @xcite points out that different transformers layers of plms resolve different language information which is encoded in hidden representations. we turn to inject noise between layers to enhance the hidden semantic representations for better generalization on transformers layer level. based on the above findings, we propose to improve fine-tuning by perturbing the hidden representations. as shown in figure 1 , we propose a simple yet effective fine-tuning technique named hype (hi(y)dden representation perturbation) that adds random noise to the hidden representations between layers (i.e., the inputs of each transformers layer) to alleviate the performance of fine-tuned layers from degrading. to be concrete, we introduce no inductive biases to the distributions of noise in hype and focus on the pivotal influences of noise per se. although noise can be compatible with auxiliary constrains @xcite or include informative priors @xcite , they may lead to non-negligible computational overheads. we simply use the uniform and normal distributions as two variants of noise distributions and denote them as hype-u and hype-n, respectively. the computational overheads are marginal in hype. hype can also be regarded as a decoupling analysis of the above methods. we conduct extensive experiments on glue benchmark @xcite and hype improves vanilla fine-tuning up to 1.60 on bert in terms of average scores of the relatively small datasets mrpc, rte, cola, and sts-b, surpasses previous state-of-the-art techniques (i.e. r-drop @xcite ) by 0.15, and improves performance in low-resource scenarios. further analyses demonstrate that hype is also compatible with different scales of plm (section 5.1) and other fine-tuning techniques (section 5.2), increases the robustness towards adversarial attacks (section 5.3), and improves generalization across tasks and domains on different layers (section 5.4). to summarize our work, the main contributions are listed as follows: ## related works for large-scale plms, fine-tuning on downstream tasks may acquire unstable performances, resulting from over-fitting problems or failed training runs @xcite . recent research has focused on how to alleviate such problems and effectively improve fine-tuning of plms on the downstream tasks. a general idea is to make the best of the pretrained weights and constrain the fine-tuned parameters from deviating much from the pre-trained weights. for example, top-k tuning @xcite only fine-tunes the top-k layers of plms and keeps the lower pre-trained layers intact. inspired by dropconnect @xcite , mixout @xcite randomly replaces the weights of parameters with their pre-trained values instead of zero. recadam @xcite introduces l 2 distance to penalize the change of weights from the pre-trained ones. childtuning @xcite applies task-free or task-driven masks on the gradients thus only a subset of parameters are changed during fine-tuning. sage @xcite uses differential updating step sizes for each parameter. parameters with higher sensitivities are updated less aggressively where the computation of sensitivities is related to the pre-trained parameters in the cases of plm fine-tuning. another line of work use noise to improve finetuning. r-drop @xcite uses kl divergence to regularize the discrepancy between the noised outputs produced by different dropout @xcite masks during fine-tuning. recently proposed noisytune @xcite directly adds weight-aware noise to the pre-trained parameters before fine-tuning to improve performance. based on the ideas of trust regions and adversarial training, freelb @xcite , smart @xcite and r3f @xcite are proposed to improve fine-tuning by introducing adversarial noise to the input representations during training. tong et al. (2022) create noised input representations by interpolating the representations between in-batch samples. the augmented fine-tuning data can alleviate overfitting and help plms learn a smoother decision boundary. previous research has proven the pivotal role of noise in improving plm fine-tuning. our proposed technique looks into the plms and adds noise to the hidden representations. previous works introduce regulations along with the added noise. generating random noise only requires little computational overheads, while additional regulations can cause non-negligible computational overheads in memory footprints or training time, such as r-drop requiring two forward computations in each training step @xcite , and child-tuning d @xcite requiring to pre-compute fisher information matrices. ## hidden representation perturbation hype is motivated to improve fine-tuning of plms. perturbing input features for better training performance is proven in effect in wide machine learning applications @xcite . the structure of plms is complicated and different layers may have diverse impacts on understanding languages @xcite . therefore, by perturbing the hidden representations, we can improve the performance of each layer hence the whole plms in fine-tuning processes. in the vanilla fine-tuning setting of language models, we denote the mapping of a plm comprising of n network layers as f θ (•) and the classification head for the downstream task as c ψ (•), where θ stands for the pre-trained parameters of the plms and ψ represents the parameters of the classification head on top of the plm. here we have the whole forward mapping ŷ = c ψ (f θ (x)), where x and ŷ are the embedded language inputs and predicted target labels respectively. the training objective is l(θ, ψ) = l (c ψ (f θ (x)), y), where l is the loss function defined by tasks. the basic layer block of nowadays plms (e.g., bert) is transformers @xcite which mainly comprises of multi-head selfattention mechanism and feed-forward neural network. by stacking the transformers layers, the scales of plms can get larger (e.g., the base and large versions of bert contain 12 and 24 layers respectively). given the stacking structure of plms, f θ (x) can be decomposed as: as the input sequences may comprise multiple word tokens, without the loss of generality, we omit the token position and sample index marks for x, y and h i for simplicity. during fine-tuning, hype injects parameterindependent noise to the hidden states (representations) of each layer, then for the i-th layer: as shown above, hype is a simple and straightforward fine-tuning technique. it can be easily applied to different tasks and plms. ## experiments in this section, we empirically demonstrate the effectiveness of hype through extensive experiments. we use glue benchmark @xcite to illustrate the performance of hype in comparison to vanilla fine-tuning. ## further analysis we provide further analyses and discussions on the performances of hype for model scaling, methods comparison and combination, adversarial attacks, and hyper-parameters in this section. ## conclusion to conclude, we introduce hype, a technique to improve plm fine-tuning. hype enhances finetuning by perturbing the intermediate hidden representations of a plm with task and model agnostic random noise. through experiments on glue and other nli tasks, we demonstrate that plms fine-tuned with hype have better performance and transferability in comparison to vanilla fine-tuning, especially in a low-resource scenario. in further analyses, without additional regulation like kldivergence and computational overheads, hype obtains superior performances compared to existing state-of-the-art fine-tuning techniques, and can further boost fine-tuning combined with others. finetuning with hype improves hidden representations across different layers and provide stable improve-ments for generalization, adversarial attack and different model scales.