An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2023)

Abstract: NLP has achieved great progress in the past decade through the use of neural models and large labeled datasets. The dependence on abundant data prevents NLP models from being applied to low-resource settings or novel tasks where significant time, money, or expertise is required to label massive amounts of textual data. Recently, data augmentation methods have been explored as a means of improving data efficiency in NLP. To date, there has been no systematic empirical overview of data augmentation for NLP in the limited labeled data setting, making it difficult to understand which methods work in which settings. In this paper, we provide an empirical survey of recent progress on data augmentation for NLP in the limited labeled data setting, summarizing the landscape of methods (including token-level augmentations, sentence-level augmentations, adversarial augmentations, and hidden-space augmentations) and carrying out experiments on 11 datasets covering topics/news classification, inference tasks, paraphrasing tasks, and single-sentence tasks. Based on the results, we draw several conclusions to help practitioners choose appropriate augmentations in different settings and discuss the current challenges and future directions for limited data learning in NLP.

## introduction deep learning methods have achieved strong performance on a wide range of supervised learning tasks @xcite @xcite . traditionally, these results were attained through the use of large, welllabeled datasets. this make them challenging to apply in settings where collecting a large amount of high-quality labeled data for training is expensive. moreover, given the fast-changing nature of real-world applications, it is infeasible to relabel * equal contribution. every example whenever new data comes in. this highlights a need for learning algorithms that can be trained with a limited amount of labeled data. there has been a substantial amount of research towards learning with limited labeled data for various tasks in the nlp community. one common approach for mitigating the need for labeled data is data augmentation. data augmentation @xcite generates new data by modifying existing data points through transformations that are designed based on prior knowledge about the problem's structure @xcite . this augmented data can be generated from labeled data, and then directly used in supervised learning @xcite , or in semisupervised learning for unlabeled data through consistency regularization @xcite (''consistency training''). while various approaches have been proposed to tackle learning with limited labeled data-including unsupervised pre-training @xcite @xcite , multi-task learning @xcite @xcite , semi-supervised learning @xcite @xcite , and few-shot learning @xcite -in this work, we focus on and compare different data augmentation methods and their application to supervised and semi-supervised learning. in this empirical survey, we comprehensively review and perform experiments on recent data augmentation techniques developed for various nlp tasks. our contributions are three-fold: (1) summarize and categorize recent methods in textual data augmentation; (2) compare different data augmentation methods through experiments with limited labeled data in supervised and semisupervised settings on 11 nlp tasks, and (3) discuss current challenges and future directions of data augmentation, as well as learning with limited data in nlp more broadly. our experimental results allow us to conclude that: ## consistency training with da while data augmentation (da) can be applied in the supervised setting to produce better results when only a small labeled training dataset is available, data augmentation is also commonly used in semi-supervised learning (ssl). ssl is an alternative approach for learning from limited data that provides a framework for taking advantage of unlabeled data. specifically, ssl assumes that our training set comprises labeled examples in addi-tion to unlabeled examples drawn from the same distribution. currently, one of the most common methods for performing ssl with deep neural networks is ''consistency regularization'' @xcite . consistency regularization-based ssl (or ''consistency training'' for short) regularizes a model by enforcing that its output doesn't change significantly when the input is perturbed. in practice, the input is perturbed by applying data augmentation, and consistency is enforced through a loss term that measures the difference between the model's predictions on a clean input and a corresponding perturbed version of the same input. formally, let f θ be a model with parameters θ, f θ be a fixed copy of the model where no gradients are allowed to flow, x l be a labeled datapoint with label y, x u be an unlabeled datapoint, and α(x) be a data augmentation method. then, a typical loss function for consistency training is for all datasets, we experiment with 10 labeled data points per class foot_2 in a supervised setup, and an additional 5000 unlabeled data points per class in the semi-supervised setup. we use bert base @xcite as the base language model and use the same hyperparameters across all datasets/methods. we utilize accuracy as the evaluation metric for all datasets except for cola (which uses matthews correlation) and pubmed (which uses accuracy and macro-f1 score). because the performance can be heavily dependent on the specific datapoints chosen @xcite , for each dataset, we sample labeled data from the original dataset with 3 different seeds to form different training sets, and report the average result. for every setup, we fine-tune the model with the same seed as the dataset seed (in contrast to many works which report the max across different seeds). we train our models on nvidia 2080ti and nvidia v-100 gpus. supervised experiments take 20 minutes, and semi-supervised experiments take two hours. the bert-base model has 100m parameters. we use the same hyperparameter across all datasets, and so only use the validation set to find the best model checkpoint. we use a learning rate of 2e -5 , batch size of 16, ratio of unlabeled to labeled data of 3, and dropout ratio of 0.1 for different augmentation methods. ## other limited data learning methods this work mainly focuses on data augmentation and semi-supervised learning (consistency regularization) in nlp; however, there are other orthogonal directions for tackling the problem of low-resourced languages. most languages lack large monolingual or parallel corpora, or sufficient manually crafted linguistic resources for building statistical nlp applications @xcite . researchers have therefore developed a variety of methods for improving performance on low-resource languages, including cross-lingual transfer learning, which transfers models from resource-rich to resource-poor languages @xcite @xcite , few/zero-shot learning @xcite @xcite , which uses only a few examples from the low-resource domain to adapt models trained in another domain, and polyglot learning @xcite @xcite , which combines resource-rich and resource-poor learning using an universal language representation. other methods for semi-supervised learning. semi-supervised learning methods further reduce the dependency on labeled data and enhance the models when there is only limited labeled data available. these methods use large amounts of unlabeled data in the training process, as unlabeled data is usually cheap and easy to obtain compared to labeled data. in this paper, we focus on consistency regularization, while there are also other widely used methods for nlp including self-training @xcite @xcite , generative methods @xcite @xcite , and cotraining @xcite @xcite . few-shot learning. few-shot learning is a broad technique for dealing with tasks with less labeled data based on prior knowledge. compared to semi-supervised learning, which utilizes unlabeled data as additional information, few-shot learning leverages various kinds of prior knowledge such as pre-trained models or supervised data from other domains and modalities @xcite . while most work on few-shot focuses on computer vision, few-shot learning has recently seen increasing adoption in nlp @xcite @xcite . to better leverage pre-trained models, pet @xcite converts the text and label in an example into a fluent sentence, and then uses the probability of generating the label text as the class logit, outperforming gpt3 for few shot learning @xcite . how to better model and incorporate prior knowledge to handle few-shot learning for nlp remains an open challenge and has the potential to significantly improve model performance with less labeled data. ## discussion and future directions in this work, we empirically surveyed data augmentation methods for limited-data learning in nlp and compared them on 11 different nlp tasks. despite the success, there are still certain challenges that need to be tackled to improve their performance. this section highlights some of these challenges and discusses future research directions. theoretical guarantees and data distribution shift. current data augmentation methods for text typically assume that they are label-preserving and will not change the data distribution. however, these assumptions are often not true in practice, which can result in noisy labels or a shift in the data distribution and consequently a decrease in performance or generalization (e.g., qqp in