P rom DA : Prompt-based Data Augmentation for Low-Resource NLU Tasks (2022)

Abstract: This paper focuses on the Data Augmentation for low-resource Natural Language Understanding (NLU) tasks. We propose Prompt-based Data Augmentation model (PromDA) which only trains small-scale Soft Prompt (i.e., a set of trainable vectors) in the frozen Pre-trained Language Models (PLMs). This avoids human effort in collecting unlabeled in-domain data and maintains the quality of generated synthetic data. In addition, PromDA generates synthetic data via two different views and filters out the low-quality data using NLU models. Experiments on four benchmarks show that synthetic data produced by PromDA successfully boost up the performance of NLU models which consistently outperform several competitive baseline models, including a state-of-the-art semi-supervised model using unlabeled in-domain data. The synthetic data from PromDA are also complementary with unlabeled in-domain data. The NLU models can be further improved when they are combined for training.

## introduction deep neural networks often require large-scale high-quality labeled training data to achieve stateof-the-art performance @xcite . however, constructing labeled data could be challenging in many scenarios @xcite . in this paper, we study the low-resource natural language understanding (nlu) tasks, including sentence classification and sequence labelling tasks, where only small labeled data is available. previous works often produce extra "labeled data" for the nlu models to learn. @xcite deploys the self-training framework to produce pseudo labelled training data from unlabeled in-domain data which could be expensive to obtain. @xcite has shown that extracting domain-specific unlabeled data from the general corpus is not trivial. @xcite ; dai and adel (2020) expand the original small training data using automatic heuristic rules, such as randomly synonyms replacement, which effectively creates new training instances. however, these processes may distort the text, making the generated syntactic data grammatically and semantically incorrect. to solve the above dilemma, many existing works @xcite @xcite resort to applying language models (lms) or pre-trained language models (plms) for data augmentation in a low-resource setting. given the labeled data, one can directly fine-tune plms to generate new synthetic data without additional human effort. however, we argue that, in the low-resource nlu tasks, directly finetuning all parameters of plms with small training data (especially when there are less than 100 samples) could result in over-fitting and plms simply memorizes the training instances. as a result, the generated synthetic data could be very similar to the original training instances and cannot provide new training signals to the nlu models. recently, several works @xcite propose prompt tuning, which only back-propagates the error to soft prompts (i.e., a sequence of continuous vectors prepended to the input of plms) instead of the entire model. they show that prompt tuning is sufficient to be competitive with full model tuning while significantly reducing the amount of parameters to be tuned. thus, the prompt tuning is quite suitable to tackle the above over-fitting issue in low-resource generative fine-tuning, which spawns more novel samples relative to the small labeled data under the premise of ensuring generation quality. motivated by this, we propose prompt-based data augmentation model (promda). specifically, we freeze the entire pre-trained model and only allow tuning the additional soft prompts during fine-tuning on the small labeled training data. in addition, we have observed that the initialization of soft prompts has a significant impact on fine-tuning, especially when the low-resource situation reaches an extreme extent. to better initialize the prompt parameters for the data augmentation tasks, we propose task-agnostic synonym keyword to sentence pre-training task to directly pre-train the prompt parameters of plms on their pre-training corpora. this task simulates the process of generating entire training sample from partial fragment information (e.g., keywords). similar to previous works @xcite @xcite , we could fine-tune plms to produce complete synthetic data conditioned on the output tags. we refer this as output view generation. to boost the diversity of the generated samples, we introduce another fine-tuning generative task named input view generation, which takes the extracted keywords from the sample as the input and the sample as the output. as nlg models trained from small training data still has a certain chance to generate low-quality samples, we leverage the nlu consistency filtering @xcite to filter the generated samples. we conduct experiments on four benchmarks: sequence labelling task conll03 @xcite and wikiann @xcite , sentence classification task sst-2 @xcite and rt @xcite . experiment results show that nlu models trained on synthetic data from promda consistently outperform several competitive baseline models, including a state-of-the-art semi-supervised nlu models metast @xcite on sequence labelling task. in addition, we find that the synthetic data from promda are also complementary with the unlabeled in-domain data. the performance of nlu models can be further improved when both of them are combined. finally, we conduct diversity analysis and case study to further confirm the synthetic data quality from promda. our source code is released at https://github. com/garyyufei/promda . ## related work prompt learning the concept of prompt-based learning starts from the gpt3 model @xcite . previous works design different prompts to query language models to extract knowledge triples @xcite or classify sentences into pre-defined categories @xcite in the few-shot setting. they construct various discrete prompts manually for these tasks. to reduce the human effort in this selection process, @xcite proposes to expand prompts using pre-trained language models. however, the selection of discrete prompts is still an independent process and difficult to be optimized together with the downstream tasks in an end-to-end manner. @xcite proposes a complicated two-stage model to connect between prompt generation and downstream tasks. to solve this issue, @xcite propose to use soft prompts, which are sets of trainable vectors, in the frozen pre-trained language models. unlike the hard prompts, these vectors do not correspond to any real words. it allows the optimization with the downstream tasks in an endto-end manner. as shown in li and liang (2021), plms with soft prompts can often perform better in the low-resource setting. ## prompt-based data augmentation this section first formulates the data augmentation for low-resource nlu task. we then introduce the three important components in our proposed prompt-based data augmentation method (promda), including i) prompt-based learning in pre-trained language models; ii) dual synthetic data generation view and iii) consistency filtering. figure 1 shows the overall of promda. ## experiments this section first introduces experimental setup in sec 4.1, and then presents main experiment results in sec 4.2. sec 4.3 conducts ablation study. in sequence labelling gt: [org all fishermen 's association] secretary [per n.j. bose] said the strike would continue indefinitely. iv: all fishermen 's association and n.j. bose and strike and indefinitely ov: organization and person sentence classification sec 4.4, we compare promda and unlabeled data, present diversity analysis and a case study. ## conclusion and future work in this paper, we present the first prompt-based pretrained language model promda for low-resource nlu data augmentation. experiments on four benchmarks show the effectiveness of our proposed promda method. in the future, we plan to expand promda to other nlp tasks, including question answering, machine reading comprehension and text generation tasks.