Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations (2023)

Abstract: Due to the huge amount of parameters, finetuning of pretrained language models (PLMs) is prone to overfitting in the low resource scenarios. In this work, we present a novel method that operates on the hidden representations of a PLM to reduce overfitting. During fine-tuning, our method inserts random autoencoders between the hidden layers of a PLM, which transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. The autoencoders are plugged out after fine-tuning, so our method does not add extra parameters or increase computation cost during inference. Our method demonstrates promising performance improvement across a wide range of sequence- and token-level lowresource NLP tasks.

## introduction fine-tuning pretrained language models (plms) @xcite @xcite provides an efficient way to transfer knowledge gained from large scale text corpora to downstream nlp tasks, which has achieved state-of-the-art performance on a wide range of tasks @xcite @xcite @xcite @xcite . however, most of the plms are designed for general purpose representation learning @xcite , so the learned representations unavoidably contain abundant features irrelevant to the downstream tasks. moreover, the plms typically possess a huge amount of parameters (often 100+ millions) @xcite , which makes them more expressive compared with simple models, and hence more vulnerable to overfitting noise or irrelevant features during fine-tuning, especially in the low-resource scenarios. there has been a long line of research on devising methods to prevent large neural models from overfitting. the most common ones can be roughly grouped into three main categories: data augmentation @xcite @xcite @xcite , parameter/activation regularization @xcite @xcite and label smoothing @xcite , which, from bottom to top, operates on data samples, model parameters/activations and data labels, respectively. data augmentation methods, such as backtranslation @xcite and masked prediction @xcite , are usually designed based on our prior knowledge about the data. though simple, many of them have proved to be quite effective. activation (hidden representation) regularization methods are typically orthogonal to other methods and can be used together to improve model robustness from different aspects. however, since the neural models are often treated as a black box, the features encoded in hidden layers are often less interpretable. therefore, it is more challenging to apply similar augmentation techniques to the hidden representations of a neural model. prior studies @xcite @xcite observe that neural models trained with different regularization or initialization can capture different features of the same input for prediction. inspired by this finding, in this work we propose a novel method for hidden representation augmentation. specifically, we insert a set of randomly initialized autoencoders (aes) @xcite between the layers of a plm, and use them to capture different features from the original representations, and then transform them into multi-view compressed representations (mvcr) to improve robustness during fine-tuning on target tasks. given a hidden representation, an ae first encodes it into a compressed representation of smaller dimension d, and then decodes it back to the original dimension. the compressed representation can capture the main variance of the data. therefore, with a set of aes of varying d, if we select a random one to transform the hidden representations in each fine-tuning step, the same or similar input is compressed with varying compression dimensions. and the upperlevel plm layers will be fed with more diverse and compressed representations for learning, which illustrates the "multi-view" concept. we also propose a tailored hierarchical ae to further increase representation diversity. crucially, after fine-tuning the plm with the aes, the aes can be plugged out, so they do not add any extra parameters or computation during inference. we have designed a toy experiment to help illustrate our idea in a more intuitive way. we add uniform random gaussian noise to the mnist @xcite digits and train autoencoders with different compression ratios to reconstruct the noisy input images. as shown in fig. 1 , the compression dimension d controls the amount of information to be preserved in the latent space. with a small d, the ae removes most of the background noise and preserves mostly the crucial shape information about the digits. in fig. 1b , part of the shape information is also discarded due to high compression ratio. in the extreme case, when @xmath0, the ae would discard almost all of the information from input. thus, when a small d is used to compress a plm's hidden representations during fine-tuning, the reconstructed representations will help: (i) reduce overfitting to noise or task irrelevant features since the high compression ratio can help remove noise; (ii) force the plm to utilize different relevant features to prevent it from becoming overconfident about a certain subset of them. since the ae-reconstructed representation may preserve little information when d is small (fig. 1b ), the upper layers are forced to extract only relevant features from the limited information. besides, the shortcut learning problem @xcite is often caused by learning features that fail to generalize, for example relying on the grasses in background to predict sheep. our method may force the model to use the compressed representation without grass features. therefore, it is potentially helpful to mitigate shortcut learning. as shown in fig. 1(d ), with a larger d, most information about the digits can be reconstructed, and noises also start to appear in the background. hence, aes with varying d can transform a plm's hidden representations into different views to increase diversity. we conduct extensive experiments to verify the effectiveness of mvcr. compared to many strong baseline methods, mvcr demonstrates consistent performance improvement across a wide range of sequence-and token-level tasks in low-resource adaptation scenarios. we also present abundant ablation studies to justify the design. in summary, our main contributions are: ## related work overfitting is a long-standing problem in large neural model training, which has attracted broad interest from the research communities @xcite @xcite @xcite . to better capture the massive information from large-scale text corpora during pretraining, the plms are often over-parameterized, which makes them prone to overfitting. the commonly used methods to reduce overfitting can be grouped into three main categories: data augmentation, parameter regularization, and label smoothing. data augmentation methods (devries and taylor, 2017; @xcite are usually applied to increase training sample diversity. most of the widely used methods, such as synonym replacement @xcite , masked prediction @xcite and back-translation @xcite , fall into this category. label smoothing methods @xcite are applied to the data labels to prevent overconfidence and to encourage smoother decision boundaries. some hybrid methods, like mixup @xcite , are proposed to manipulate both data sample and label, or hidden representation and label. most parameter/hidden representation regularization methods are orthogonal to the methods discussed above, so they can be used as effective complements. neural models are often treated as a black box, so it is more challenging to design efficient parameter or hidden representation regularization methods. existing methods mainly focus on reducing model expressiveness or adding noise to hidden representations. weight decay @xcite enforces l 2 norm of the parameters to reduce model expressiveness. dropout @xcite randomly replaces elements in hidden representations with 0, which is believed to add more diversity and prevent overconfidence on certain features. inspired by dropout, mixout @xcite stochastically mixes the current and initial model parameters during training. mahabadi et al. (2021) leverage variational information bottleneck (vib) @xcite to help models to learn more concise and task relevant features. however, vib is limited to regularize last layer sequencelevel representations only, while our method can be applied to any layer, and also supports token-level tasks like ner and pos tagging. ## methodology we first formulate using neural networks over the hidden representation of different layers of deep learning architectures as effective augmentation modules ( §3.1) and then devise a novel hierarchical autodencoder (hae) to increase stochasticity and diversity ( §3.2). we utilise our novel hae as a compression module for hidden representations within plms, and introduce our method multi-view compressed representation (mvcr) ( §3.3). we finally discuss the training, inference, and optimization of mvcr. an overview of mvcr is presented in fig. 2 . ## experiments in this section, we present the experiments designed to evaluate our method. we first describe the baselines and training details, followed by the evaluation on both the sequence-and token-level tasks across six (6) different datasets. baselines we compare our method with several widely used parameter and hidden representation regularization methods, including dropout (srivastava et al., 2014), weight decay (wd) @xcite training details same as mahabadi et al. (2021), we fine-tune bert base @xcite for the sequence-level tasks. the token-level tasks are multilingual, so we tune xlm-r base @xcite on them. we use mvcr l to denote our methods, where l is the plm layer that we insert haes. for example in mvcr 1 , the haes are inserted after the 1st transformer layer. on each layer, we insert three haes with compressed representation dimensions 128, 256 and 512, respectively. all haes are discarded during inference. for each experiment, we report the average result of 3 runs. for each dataset, we randomly sample 100, 200, 500 and 1000 instances from the training set to simulate the low-resource scenarios. the same amount of instances are also sampled from the dev sets for model selection. the full test set is used for evaluation. more details can be found in §a.5. ## analysis and ablation studies to better understand the improvements obtained by mvcr, we conduct in-depth analysis and ablation studies. we experiment with both sequence-and token-level tasks by sampling 500 instances from 3 datasets, namely mnli, imdb and wikiann. as shown in fig. 3 , adding mvcr after layer 1 or 12 achieves the best performance than the other layers. this result can be explained in the way of different augmentation methods @xcite . by adding mvcr after layer 1, the module acts as a data generator which creates multi-view representation inputs to all the upper layers. while adding mvcr after layer 12, which is more relevant to the downstream tasks, the module focuses on preventing the task layer from overfitting @xcite . apart from one layer, we also experiment with adding mvcr to a combination of two layers ( ## conclusions in this work, we have proposed a novel method to improve low-resource fine-tuning robustness via hidden representation augmentation. we insert a set of autoencoders between the layers of a pretrained language model (plm). the layers are randomly selected during fine-tuning to generate more diverse compressed representations to prevent the top plm layers from overfitting. a tailored stochastic hierarchical autoencoder is also proposed to help add more diversity to the augmented representations. the inserted modules are discarded after training, so our method does not add extra parameter or computation cost during inference. our method has demonstrated consistent performance improvement on a wide range of nlp tasks.