GPT 3 M ix: Leveraging Large-scale Language Models for Text Augmentation (2021)

Abstract: Large-scale language models such as GPT-3 are excellent few-shot learners, allowing them to be controlled via natural text prompts. Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. This paper proposes a novel data augmentation technique that leverages large-scale language models to generate realistic text samples from a mixture of real samples. We also propose utilizing soft-labels predicted by the language models, effectively distilling knowledge from the large-scale language models and creating textual perturbations simultaneously. We perform data augmentation experiments on diverse classification tasks and show that our method hugely outperforms existing text augmentation methods. We also conduct experiments on our newly proposed benchmark to show that the augmentation effect is not only attributed to memorization. Further ablation studies and a qualitative analysis provide more insights into our approach.

## introduction in the seminal work by @xcite , a largescale language model, specifically gpt-3, has been shown to achieve superior performance on zeroshot and few-shot learning tasks by prompt-based in-context learning. in-context learning utilizes a prompt, which usually consists of a task description and few examples, to solve unseen tasks without the hefty price of fine-tuning. recognizing the potential research applications of in-context learning and prompt-based control, a part of the nlp community has shifted its focus on understanding and devising advanced methods for optimizing promptbased approaches @xcite @xcite . however, these prompt-based approaches with inference on a large-scale language model suffer from several drawbacks. first, the number of incontext training examples is hard limited by the maximum prompt length enabled by the inherent language model architecture. second, promptbased approaches require online inference on the expensive large-scale language models. the inference may not be scalable in real-world use cases, because it is slow and incurs huge memory overhead. lastly, the prompt-based approaches do away with conventional machine learning techniques, making it mostly incompatible with existing established fine-tuning methods. to overcome such limitations, we propose a more practical solution to utilize large-scale language models for downstream nlp tasks. in our proposed framework, as depicted in figure 1 , large-scale language models are not used as the pre-trained model for further domain-adaptive fine-tuning nor the backbone for prompt-based incontext learning but for imbuing the original training set with synthetic text data. we propose gpt3mix, a method for generating synthetic but hyper-realistic text samples from a mixture of real samples utilizing large-scale language models such as gpt-3 1 . gpt3mix extracts few sample sentences from the task-specific training data, embed these samples in the prompt, and generates an augmented mixed sentence influenced by the sample sentences. gpt3mix uses softlabels predicted by the large-scale language model to transfer knowledge of probability as in knowledge distillation @xcite . in short, our method achieves both (1) data augmentation via generating synthetic examples inspired by existing data samples and (2) knowledge distillation by training smaller classification models using softlabels predicted by the large language model. our approach takes inspiration from the mixbased data augmentation methods in the vision domain @xcite . several mix-based data augmentation methods are suggested for nlp models. one of the notable methods is mixtext @xcite , in which bert is used to generate novel augmentation samples from interpolated embedding spaces. however, despite its great success in the vision domain, deep-mixing text augmentation methods have seen limited effectiveness in realworld cases due to the difficulty of interpolating language from latent spaces @xcite . synthetic language interpolated from a model's hidden space such as the word embedding space of bert may introduce noise, outweighing the benefit of novel sample discovery and causing deterioration in the training data distribution. our work exploits the generative power of large-scale language models like gpt-3 to generate high-quality mixed samples from in-context examples. we perform various data augmentation experiments on diverse classification tasks to verify our hypotheses and analyze our methodology. as language models are partly pretrained on web-crawled corpora, some benchmarks such as the movie review classification tasks may have been "seen" by the language models. to eliminate the possibility of pretraining memorization, we propose a new task rt20 where we collected online movie reviews posted after the known data preparation date of gpt-3. experimental results with the newly proposed benchmark rt20 show that the benefit of our method is not attributed to memorization but mix-based text synthesis. we will release the benchmark soon. the contribution of our work is summarized as follows. foot_1 1. we suggest employing prompt-based data augmentation using large-scale language mod-els on top of the existing plm fine-tuning paradigm to exploit the best of both worlds. text augmentation text augmentation refers to methods for perturbing the linguistic space without altering class labels to improve the robustness and generalizability of the downstream models. data augmentation has been studied extensively in the nlp scene. text augmentation in the current literature comes with two flavors: shallow and deep augmentation. the shallow data augmentation techniques inject locally plausible small noises into the linguistic space (words or phrases), in the hopes that the perturbations produce linguistically acceptable samples while maintaining label consistency. two examples are eda @xcite and synonym replacement @xcite . another class of augmentation techniques employs external language models to improve global coherence and consistency. the back-translation approach exploits semantic consistency in translation language pairs to generate novel paraphrases @xcite . in the more recent line of work, pre-trained language models, such as bert @xcite or the sequence-to-sequence variant bart @xcite , are used to obtain more diverse and linguistically correct augmentation samples. for example, bart has been proven to be effective in populating text samples for data-scarce labels @xcite . @xcite proposed using masked language models as a denoising autoencoder to generate synthetic texts. some other researchers have taken the direction of perturbing the latent spaces, optionally by introducing variational inference in the architecture @xcite @xcite . on the other hand, inspired by the mix-up technique @xcite proposed for the vision domain, there have also been works to mix existing text samples to produce realistic augmentation texts based on statistical methods @xcite @xcite . furthermore, pseudo-labeling, the act of annotating unlabeled data with model predictions @xcite , has been actively used in semi-supervised learning settings @xcite @xcite . large-scale language models pre-trained transformer-based language models @xcite have initiated a new paradigm in the nlp scene, changing the way we design nlp pipelines. with the recent development of mega-scale language models @xcite , we are witnessing another shift in the paradigm, namely prompt-based nlp. these large language models are essentially few-shot learners, allowing them to be controlled through natural text. there has been a steep rise in the community's interest to better understand the prompt-based mechanisms @xcite @xcite @xcite . our work relies on the previous findings on prompt-based manipulation. to the best of our knowledge, this work is the first to propose using the prompt-based approach to generate synthetic samples from large-scale language models for the purpose of text augmentation. ## gpt3mix mixup @xcite is a simple learning technique that has been shown to be effective in preventing memorization and improving generalizability for the vision domain. the technique has been very effective on image data, but it has been harder to establish a standard approach for texts due to the inherent sparse nature of linguistic distributions, which attributes to the challenges of identifying adversarial text examples @xcite . inspired by the technique, we propose gpt3mix as a powerful yet simple method to generate highly fluent synthetic samples based on a data distribution. the proposed method (figure 2 ) consists of three steps: (1) selecting examples from the dataset, (2) constructing a gpt3mix prompt from the selected examples and meta-information about the dataset, and finally (3) extracting augmentation from the language model generation. this section provides details about each step as follows. example selection for simplicity, we confine the downstream task to text classification tasks. given a classification task t , the training dataset d is a set of text x and associated label y pairs: we randomly choose k examples from d to be anchors. large-scale language models are known to be highly sensitive to the choice and the order of examples in the prompt @xcite . we conjecture that by carefully choosing the examples, we are able to control the generated augmentation samples from the language model. we conduct qualitative analysis on the augmentation samples to confirm our hypothesis ( §4.4.5). in our implementation, we simply used uniform distribution to choose k examples: p s (i) = 1/n . otherwise stated, most experiments are carried out by setting @xmath0. as found in our ablation studies ( §4.4.1), @xmath1. prompt construction given a set of prompt examples d @xmath2, y i ) | 1 ≤ @xmath3, we formulate the prompt as follows. a gpt3mix prompt consists of a description header, an enumeration of text-label pairs of d e , and the augmentation prefix. an example of the prompt is shown in the appendix (appendix a). our prompt has been designed carefully with the current literature findings of gpt-3 prompts @xcite in mind. specifically, the prompt follows the general template shown in the appendix, but has task-specific information to allow the large-scale language models to generalize better about the data distribution. ## experiments we evaluate our augmentation approach on the following seven classification benchmarks: sst-2 @xcite is a sentiment classification dataset that contains movie reviews crawled from rotten tomatoes and their corresponding binary labels. cr @xcite dataset is a set of amazon product reviews labeled by binary sentiments. the corpus of linguistic acceptability (cola) @xcite is a collection of sentences extracted from publications annotated with grammaticality. the trec6 dataset (voorhees and tice, 1999) concerns the question classification task consisting of open-domain, factbased questions divided into broad semantic categories. mpqa @xcite consists of opinions and their semantic polarity. the subjectivity dataset (subj) @xcite ) contains movie reviews labeled with objectivity. rt20 is the newly proposed benchmark with which we perform controlled experiments on language models. the dataset is a binary sentiment classification corpus, collected from rotten tomatoes accessed after a certain date. the details about the collection and preparation is provided in appendix c. ## conclusion in this paper, we proposed a novel text augmentation technique that leverages large-scale language models and their abilities to perform controlled generation via prompts. our extensive experiments on classification tasks show that our augmentation method can improve robustness of pretrained transformers through mix-based perturbation and knowledge distillation without the online inference on heavy lms. thus, our method can be a competitive alternative to prompt-based task-solving @xcite or direct fine-tuning @xcite . as future work, we are interested in the possibility of further pushing the boundaries of state-of-the-art architectures via gpt3mix. we are also working towards improving generation efficiency by optimizing example selection and prompt templates.