Learning Data Augmentation Schedules for Natural Language Processing (2021)

Abstract: Despite its proven efficiency in other fields, data augmentation is less popular in the context of natural language processing (NLP) due to its complexity and limited results. A recent study (Longpre et al., 2020) showed for example that task-agnostic data augmentations fail to consistently boost the performance of pretrained transformers even in low data regimes. In this paper, we investigate whether data-driven augmentation scheduling and the integration of a wider set of transformations can lead to improved performance where fixed and limited policies were unsuccessful. Our results suggest that, while this approach can help the training process in some settings, the improvements are unsubstantial. This negative result is meant to help researchers better understand the limitations of data augmentation for NLP.

## introduction in recent years, data augmentation has become an integral part of many successful deep learning systems, especially in the fields of computer vision and speech processing @xcite @xcite . traditionally, data augmentation approaches take the form of label-preserving transforms that can be applied to the training datasets to expand their size and diversity. the idea of generating synthetic samples that share the same underlying distribution as the original data is often considered a pragmatic solution to the shortage of annotated data, and has been shown to reduce overfitting and improve generalisation performance @xcite . however, despite a sound theoretical foundation @xcite , this paradigm has not yet translated to consistent and substantial improvement in natural language processing (nlp) @xcite . the inability of nlp models to consistently benefit from data augmentations can be partially attributed to the general difficulty of finding a good combination of transforms and determining their respective set of optimal hyperparameters @xcite , a problem that is exacerbated in the context of text data. indeed, since the complexity of language makes text highly sensitive to any transformations, data augmentations are often tailored to a specific task or dataset and are only shown to be successful in specific settings. in this work, we investigate whether automatically searching for an optimal augmentation schedule from a wide range of transformations can alleviate some of the shortcomings encountered when applying data augmentations to nlp. this endeavour follows the recent success of automated augmentation strategies in computer vision @xcite @xcite . in doing so, we extend the efforts to understand the limits of data augmentation in nlp. ## related work although there exist recent surveys @xcite that offer a comprehensive review of related work, they do not provide a comparative analysis of the different data augmentation approaches and of their effect on learning performance. in general, the literature lacks general comparative studies that encompass the variety of tasks and datasets in nlp. indeed, most of the existing text data augmentation studies either focus on a single approach in a specific setting or compare a small set of techniques on a specific task and dataset @xcite . in addition, many of these comparisons have been conducted before the widespread adoption of contextualized representations. recently, @xcite showed that, despite careful calibration, data augmentation yielded little to no improvement when applied to pretrained transformers even in low data regimes. while their comparative analysis is conducted on various classification tasks, it focuses on a limited set of aug-mentation strategies that are applied independently and whose hyperparameters are optimized via a random search. to further assess the effectiveness of data augmentation on pretrained transformers, we investigate in this work the paradigm of learning data augmentation strategies from data in the context nlp. the idea is to leverage the training data to automatically discover an optimal combination of augmentations and their hyperparameters at each epoch of the fine-tuning. first, we define a search space that consists of a variety of transformations before relying on the training data to learn an optimal schedule of probabilities and magnitudes for each augmentation. this schedule is later used to boost performance during fine-tuning. in the following section, we present a variety of data augmentations and, for each category of transformations, we highlight a subset of augmentations that is representative of the category and that will constitute our search space. ## data augmentation in nlp in this study, we focus on transformative methods which apply a label-preserving transformation to existing data-rather than generative methods which create entirely new instances using generative models. indeed their simplicity of use and their low computational cost make them good candidates for a wide deployment @xcite . in the last decade, we witnessed a widespread adoption of continuous vector representations of words, which can be easily fed to deep neural network architectures. as a result, transforms have been developed not only at the lexical level (i.e., words) but also at the latent semantic level (i.e., embeddings). this distinction is emphasized throughout this section. word replacement a commonly used form of data augmentation in nlp is word replacement. at the lexical level, the most common approach consists in randomly replacing words with their synonyms @xcite @xcite . some variants include replacement based on other lexical relationships such as hypernymy @xcite or simply words from the vocabulary @xcite . another popular approach consists in using a language model (lm) for replacement @xcite @xcite . because these transformations do not ensure the preservation of the sample class, @xcite suggested conditioning a bidirectional lm on the labels, an idea later revisited by @xcite who replaced the lm with a conditional bert (bidirectional encoder representations from transformers). at the latent semantic level, word replacement amounts to randomly replacing its embedding with some other vector. for instance, wang and @xcite choose the k-nearest-neighbour in the embedding vocabulary as a replacement for each word. similar strategies were later adopted by @xcite and @xcite . in this study, as replacement methods, we select both synonym and hypernym replacement as well as contextual augmentation at the word level and nearest neighbour at the embedding level. noising a simple yet effective form of augmentation that is often applied to images and audio samples is data noising. not surprisingly, this type of data augmentation can also be found in nlp despite the discrete nature of text data. in its simplest form, data noising when applied to text consists in inserting, swapping or deleting words at random @xcite . more generally, the process of ignoring a fraction of the input words is often referred to as word dropout @xcite and can take multiple forms @xcite @xcite @xcite . sometimes, word replacement can also be thought of as a form of noising. for instance, replacing words at random with other words from the vocabulary introduces noise into the data @xcite . in contrast, at the distributed representation level, this type of augmentation often takes the form of added noise to the embeddings. possible noising schemes include gaussian noise @xcite , uniform noise @xcite , bernoulli noise and adversarial noise @xcite . typically, noising is applied to every word embedding, but it can also be applied only to selected ones @xcite . alternatively, as with word dropout, noise can be incorporated into the training by discarding, across all words, some embedding dimensions with a predefined probability @xcite . noising strategies used in this study are based on random deletion, random swap and random insertion at the sentence level as well as gaussian noise and uniform noise at the feature level. back-translation back-translation provides a way for neural machine translation (nmt) systems to leverage monolingual data to increase the amount of parallel data @xcite @xcite . similarly, back-translation can be applied twice in a row (i.e., from english to another language and back to english) to generate new data points without the need for parallel corpora and can therefore find applications as a task-agnostic augmentation in other tasks such as text classification @xcite , paraphrase generation @xcite and question answering @xcite . here, we consider a wide range of intermediate languages to include back-translation into our search space ## automated data augmentation in nlp in nlp, little effort has been put into developing strategies that can, given a task and a dataset, learn an optimal subset of data augmentation operations and their hyperparameters @xcite . yet, this idea has been very successful in computer vision @xcite @xcite . for instance, in the context of image classification, @xcite have proposed autoaugment a procedure that automatically searches for optimal augmentation policies using reinforcement learning. later, population-based augmentation (pba) @xcite )-an algorithm that views the data augmentation selection and calibration problem as a hyperparameter search and can thus leverage the population-based training (pbt) method @xcite to find an optimal transformation schedule in an efficient way-was introduced as a more cost-effective yet competitive alternative to autoaugment. finally, cubuk et al. (2020) introduced the randaugment method which tackles some of the issues arising from these previous works. in this study, we adapt the pba framework to nlp in an attempt to learn an optimal schedule of data augmentation operations with optimized hyperparameters. ## experiments our implementation builds on the original pba codes @xcite . we make the nlp adaptation of this framework publicly available foot_0 . ## results the main results can be found in overall, the improvements yielded by the optimized data augmentation schedules are inconsistent and unsubstantial (below 0.8%). even though the incorporation of transforms has a small positive impact on the sst-2 dataset, it has the opposite effect on mnli (i.e., the scores plummet by as much as 1.39%). a possible reason for these poor results might be due to the difference in settings between our experiments and the ones in the pba study. indeed, our search is conducted on 48 epochs as opposed to the 160 to 200 epochs suggested for image classification tasks and the exploit-and-explore procedure takes place after each epoch rather than after every 3 epochs. in addition, the size of the training datasets is very different. the size of our largest experiment is roughly the same as the size of the smallest dataset in @xcite . surprisingly, our data-driven search seems unable to reproduce the performance boost reported by @xcite on the mnli dataset, even though the augmentations considered in their work are part of our search space. this might be explained by the way augmentations are applied. in our study, we transform each training sample by applying up to 2 transformations in a row with probability p and magnitude m. in contrast, longpre et al. add n × τ augmented samples to the training set with τ ∈ {0.5, 1, 1.5, 2}, meaning that the original examples are provided along with their augmented counterparts at each iteration. ## conclusion the results suggest that augmentation schedules and data-driven parameter search do not provide a consistent and straightforward way to improve the performance of nlp models that use pretrained transformers. there are a few possible explanations for this phenomenon. first, the overall setup of the pba approach (e.g., the need for large validation sets) might not be well suited for low-data regimes in nlp. a second but more likely reason is that transformers are already pre-trained on huge datasets and their representations may already be invariant to many of the transformations that are encoded into the data augmentation. a systematic investigation into the latter hypothesis is required, which, if proven, would show that data augmentation may be redundant when opting to use transformers to implement nlp solutions. a final reason might be that the search space we consider only contains transformative data augmentation techniques and omits generative ones, even though the latter have started to show some promising results.