Data Augmentation using Pre-trained Transformer Models (2020)

Abstract: Language model based pre-trained models such as BERT have provided significant gains across different NLP tasks. In this paper, we study different types of transformer based pre-trained models such as auto-regressive models (GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional data augmentation. We show that prepending the class labels to text sequences provides a simple yet effective way to condition the pre-trained models for data augmentation. Additionally, on three classification benchmarks, pre-trained Seq2Seq model outperforms other data augmentation methods in a low-resource setting. Further, we explore how different pre-trained model based data augmentation differs in-terms of data diversity, and how well such methods preserve the class-label information.

## introduction data augmentation (da) is a widely used technique to increase the size of the training data. increasing training data size is often essential to reduce overfitting and enhance the robustness of machine learning models in low-data regime tasks. in natural language processing (nlp), several word replacement based methods have been explored for data augmentation. in particular, @xcite showed that simple word replacement using knowledge bases like wordnet @xcite improves classification performance. further, @xcite utilized language models (lm) to augment training data. however, such methods struggle with preserving class labels. for example, non-conditional da for an input sentence of sentiment classification task "a small impact with a big movie" leads to "a small movie with a big impact". using such augmented data for training, with the original input sentence's label (i.e. negative senti-ment in this example) would negatively impact the performance of the resulting model. to alleviate this issue, @xcite proposed conditional bert (cbert) model which extends bert @xcite masked language modeling (mlm) task, by considering class labels to predict the masked tokens. since their method relies on modifying bert model's segment embedding, it cannot be generalized to other pre-trained lms without segment embeddings. similarly, @xcite used gpt2 @xcite for da where examples are generated for a given class by providing class as input to a fine-tuned model. in their work, gpt2 is used to generate 10 times the number of examples required for augmentation and then the generated sentences are selected based on the model confidence score. as data selection is applied only to gpt2 but not to the other models, the augmentation methods can not be fairly compared. due to such discrepancies, it is not straightforward to comprehend how the generated data using different pre-trained models varies from each other and their impact on downstream model performance. this paper proposes a unified approach to use any pre-trained transformer @xcite based models for data augmentation. in particular, we explore three different pre-trained model types for da, including 1) an autoencoder (ae) lm: bert, 2) an auto-regressive (ar) lm: gpt2, and 3) a pre-trained seq2seq model: bart @xcite . we apply the data generation for three different nlp tasks: sentiment classification, intent classification, and question classification. in order to understand the significance of da, we simulate a low-resource data scenario, where we utilize only 10 training examples per class in a classification task. section 3.2 provides details of the task and corpora. we show that all three types of pre-trained mod-els can be effectively used for da, and using the generated data leads to improvement in classification performance in the low-data regime setting. among three types of methods, pre-trained seq2seq model provides the best performance. our code is available at 1 . our contribution is three-fold: (1) implementation of a seq2seq pre-trained model based data augmentation, (2) experimental comparison of different data augmentation methods using conditional pre-trained model, (3) a unified data augmentation approach with practical guidelines for using different types of pre-trained models. ## da using pre-trained models lm pre-training has been studied extensively @xcite @xcite . during pre-training, such models are either trained in an ae setting or in an ar setting. in the ae setting, certain tokens are masked in the sentence and the model predicts those tokens. in an ar setting, the model predicts the next word given a context. recently, pre-training for seq2seq model has been explored where a seq2seq model is trained for denoising ae tasks @xcite . here, we explore how these models can be used for da to potentially improve text classification accuracy. ## conclusion and future work we show that ae, ar, and seq2seq pre-trained models can be conditioned on labels by prepending label information and provide an effective way to augment training data. these da methods can be easily combined with other advances in text content manipulation such as co-training the data generator and the classifier @xcite . further, the proposed data augmentation techniques can also be combined with latent space augmentation @xcite . we hope that unifying different da methods would inspire new approaches for universal nlp data augmentation.