Effectiveness of Data Augmentation for Parameter Efficient Tuning with Limited Data (2023)

Abstract: Recent work has demonstrated that using parameter efficient tuning techniques such as prefix tuning (or P-tuning) on pretrained language models can yield performance that is comparable or superior to fine-tuning while dramatically reducing trainable parameters. Nevertheless, the effectiveness of such methods under the context of data augmentation, a common strategy to improve learning under low data regimes, has not been fully explored. In this paper, we examine the effectiveness of several popular task-agnostic data augmentation techniques, i.e., EDA, Back Translation, and Mixup, when using two general parameter efficient tuning methods, P-tuning v2 and LoRA, under data scarcity. We show that data augmentation can be used to boost the performance of P-tuning and LoRA models, but the effectiveness of each technique varies and certain methods can lead to a notable degradation in performance, particularly when using larger models and on harder tasks. We further analyze the sentence representations of P-tuning compared to fine-tuning to help understand the above behaviour, and reveal how P-tuning generally presents a more limited ability to separate the sentence embeddings from different classes of augmented data. In addition, it displays poorer performance on heavily altered data. However, we demonstrate that by adding a simple contrastive loss function it can help mitigate such issues for prefix tuning, resulting in sizable improvements to augmented data performance.

## introduction while large pretrained language models have achieved superior performance and widespread adoption across many nlp tasks @xcite , they often contain hundreds of millions or even hundreds of billions of parameters, which significantly limits their application to tasks in which computation and storage resources are constrained. to address this issue, an entire family of techniques called parameter efficient tuning (pet) methods have been developed. most notably, deep prompt tuning (i.e., prefix tuning or p-tuning) @xcite @xcite has attracted extensive attention, which, compared to fine-tuning, only tunes trainable continuous embeddings, resulting in a tiny percentage of tuned parameters for each task. consequently, a single language model can be used for multiple tasks by swapping out the trained prompts on a task basis @xcite . such success has also been shown by the state-of-the-art p-tuning v2 model @xcite , which yields performance comparable to fine-tuning on various natural language understanding tasks. in addition, low-rank adaptation (lora) @xcite has also emerged as an alternative approach, whereby the weights of rank decomposition matrices that are injected into each layer are optimized in lieu of the full network weights, reducing trainable parameters while also achieving improvements in accuracy over fine-tuning and other pet methods. when training data is scarce in a task, data augmentation (da) is a widely used strategy that can boost the performance of deep learning models (see @xcite for a survey on data augmentation in nlp). then a basic question that needs to be answered is how effective is it when the above pet frameworks are applied in conjunction with data augmentation. to this end, we study three common task-agnostic da methods, eda @xcite , back translation @xcite , and mixup @xcite with two common language models bert @xcite and roberta @xcite , trained using p-tuning v2 @xcite and lora @xcite . we set up our study across five tasks, including 4 from superglue @xcite , in which the sizes of training data are small. we show that data augmentation can increase the accuracy of prefix tuning models. however, the performance of each technique varies depending on the dataset and underlying pretrained model, and the effective techniques differ from fine-tuning. certain methods can even lead to a notable degradation in performance, particularly when using larger models. to better understand the above phenomena, we visualize sentence representations of data under prefix tuning, a technique where many augmentation methods fail to get good results, to observe whether any limitations are present with how data is represented that are leading to poor performance in certain cases. through this, we find that these models struggle more with separating the augmented embeddings from different classes. this observation is further supported by our additional experiments showing a lower robustness to heavily altered augmented data compared to fine-tuning, and an analysis of the cosine similarities between the sentence embeddings of augmented sentences and their original counterparts, where we see that prefix tuning produces highly similar embeddings irrespective of alterations. we seek to improve the performance when training on augmented data by adding a contrastive loss term @xcite to minimize distances between intra-class embeddings in p-tuning v2, helping mitigate the above issues with sentence representations, and resulting in some improvements in accuracy. we hope this empirical study helps facilitate future work on leveraging data augmentation when training transformer-based models using parameter efficient tuning methods. ## related work a wide range of data augmentation techniques are used in nlp. some general types of methods include rule-based techniques that use predetermined, rule-based transformations like synonym replacement that are not dependant on any model architecture @xcite . example methods include eda. model-based techniques utilize language modelling to generate new data through means like seq2seq translation models as in back translation @xcite , masked language modelling using transformers on randomly masked inputs @xcite , or using generative models like gpt-2 as in anaby-tavor et al. (2020) where a label-conditioned generation model is obtained by fine-tuning gpt-2 @xcite . despite widespread use in computer vision, da is not as commonly used in the training of nlp models, largely due to the discrete nature of language making it difficult to apply perturbations without compromising the meaning, and the inconsistency in performance of different techniques @xcite . the effectiveness of da on finetuned models has been studied previously. @xcite detail how bt and eda could not generate consistent benefits on classification tasks with transformer-based models. similarly, @xcite expand this previous study and test the impact of 12 different da methods and find little benefit when training on datasets with thousands of examples, but that there is some improvement in performance with very limited data (only a few hundred training cases). our study conducts a similar examination on the effectiveness of data augmentation with pretrained models, however we focus on prefix tuning and lora, and we inspect how the properties of prefix tuning under da differ from fine-tuning and what practical effect this has, particularly in terms of sentence representations. ## data augmentation with parameter efficient tuning prefix tuning approach. we first experiment with prefix tuning, and base our approach on the p-tuning v2 @xcite ) implementation. by optimizing prefix length, selectively applying reparameterization, and tuning a randomlyinitialized classification head on top of the transformer sentence representations, p-tuning v2 can be applied to smaller language models and across many tasks, including classification, with no drop in performance compared to fine-tuning. p-tuning v2 works by adding prompts of length p tokens to all layers of the language model in the form of sequences of continuous prefix tokens. these continuous embeddings, represented by the vector h @xmath0, ..., h p ] where each h @xmath1, are prefixed to the embeddings of each transformer layer lora. we test lora @xcite , which is a technique that similarly freezes the original transformer weights, but only tunes trainable pairs of rank-decomposition matrices that are inserted into each transformer layer to approximate the weight updates, greatly reducing the number of parameters needed by avoiding updating the full layer weights while not increasing inference costs. instead of updating a pretrained weight matrix of a transformer w 0 ∈ r d×k with a gradient-based weight update ∆w for time step t to get a new weight matrix as in w @xmath2, lora does a low-rank decomposition of the update into two matrices w @xmath3, with w 0 being frozen and where r represents the rank of the decomposition. the authors find applying lora to the query and value projection matrices in the attention mechanism leads to the most optimal performance. lora often outperforms other pet methods, making it worthy of study in this context @xcite . ## experimental setup we focus primarily on small datasets with less than a few thousand annotated data points, since limited data presents challenges in learning generalizable models and hence is a common use case for da. previous works have shown da with pretrained language models tends to be largely ineffective for larger datasets @xcite . we test on five text classification datasets. four are from the superglue benchmark @xcite , consisting of rte @xcite @xcite , cb @xcite , copa @xcite , and wsc @xcite . these are the smallest datasets in the benchmark at only 2500, 250, 400, 554 training samples respectively. they capture a range of difficulty, and include diverse problems like textual entailment, commonsense-reasoningbased coreference resolution, determining cause or effect, and clause commitment. to capture performance on a simpler single sentence sentiment analysis task, we also use a downsampled subset of sst-2 @xcite , consisting of 400 training samples and 100 validation sentences. we experiment with bert-base @xcite and roberta-large @xcite , which have 110 million and 355 million parameters, respectively. compared to larger models, their size makes them suitable when storage concerns are present. each model is trained until convergence and we report results at the epoch where the highest validation accuracy is achieved. we provide the detailed training settings for our models in appendix a. below we describe the settings for each of the data augmentation methods: eda: we apply a combination of the synonym replacement, random swap, random insertion, and random deletion operations to a certain percentage of the words in a sentence to generate multiple augmented sentences per original sentence. for the amount of augmented sentences generated, and the percentage of words affected, we follow the recommended usage guidelines laid out by @xcite which are dependant on the size of the training set. 16 augmented sentences per original are generated for the smallest datasets; cb, copa, wsc, and sst-2 since they only contain a few hundred training samples samples. the percentage of words in the original text affected for each of the 4 operations is 5% (α = 0.05). 8 sentences are generated per original for the rte training set, with the same percentage of words being affected as the aforementioned datasets. bt: for our implementation, we back-translate the sentences using a series of english to a target language and target language to english translation models for 4 common languages: french, spanish, german, and chinese. the specific models we use are the helsinki nlp opus translation modules @xcite . these languages are chosen due to the substantial amount of parallel corpora available for training the respective language models, which yields higher quality translations that are better able preserve the semantics of the original sentence while still introducing some diversity. mixup: inspired by mixup in image classification @xcite , where random pairs of input images and their labels are linearly interpolated to help generated synthetic images, mixup can be similarly applied over embeddings for text. as in @xcite , we adapt mixup for text classification by interpolating on sentence embeddings (senmixup). in a mini batch of size n , we take the same number of random pairs of inputs texts, and interpolate over the d-dimensional (768 in the case of bert-base and 1024 for roberta-large) classification ([cls]) tokens of the final hidden layer of the transformer encoder produced after each of the inputs is fed into transformer. the interpolated token is then passed into a fully connected later to generate the final softmax prediction vector. the interpolation for a given pair [cls] token pair (x i , x j ) with corresponding one hot label vectors (y i , y j ), for inputs i and j is conducted as follows, procedure for augmentation: for simplicity, since most of the datasets use multiple input sentences/words per training sample (e.g. premise and hypothesis), we apply the static da methods over the primary (first) input, which is the premise for rte, cb, and copa, though we use the full sentence for wsc and sst2. ## conclusion we reveal how when limited training data is available, data augmentation can provide crucial improvements in accuracy when training classification models using pet methods, though the benefits are, however, not universal, and limitations exist when using certain methods. we also demonstrated how prefix tuning may have more difficulty learning sentence representations of augmented data. we further showed that contrastive learning can be a solution to these issues, and suggest that more work be done to find similar methods that can be more generalizable.