Learning to Generate Examples for Semantic Processing Tasks (2022)

Abstract: Even if recent Transformer-based architectures, such as BERT, achieved impressive results in semantic processing tasks, their fine-tuning stage still requires large scale training resources. Usually, Data Augmentation (DA) techniques can help to deal with low resource settings. In Text Classification tasks, the objective of DA is the generation of well-formed sentences that i) represent the desired task category and ii) are novel with respect to existing sentences. In this paper, we propose a neural approach to automatically learn to generate new examples using a pre-trained sequence-to-sequence model. We first learn a task-oriented similarity function that we use to pair similar examples. Then, we use these example pairs to train a model to generate examples. Experiments in low resource settings show that augmenting the training material with the proposed strategy systematically improves the results on text classification and natural language inference tasks by up to 10% accuracy, outperforming existing DA approaches.

## introduction deep learning models achieve state-of-the-art results in many domains, including computer vision and natural language processing (nlp). training these large models typically requires many examples, whose collection and annotation can be costly and time-consuming. data augmentation (da) has proven an efficient way to acquire more training samples without incurring in the prohibitive annotation cost in a variety of fields, including computer vision @xcite and speech recognition @xcite . in nlp, some da techniques have been proposed too, as surveyed in @xcite : common approaches cre-ate synthetic data by manipulating real examples, using text editing @xcite or back-translation @xcite ; the resulting examples are automatically labeled by inheriting the class of the original example they were generated from. unfortunately, when using recent pre-trained language models, such as bert @xcite or roberta @xcite , the effectiveness of da methods is extremely limited, and sometimes they can even hurt the results @xcite . a possible explanation for this inefficacy is that these da techniques introduce lexical and structural variability that accurate language models directly induce during pre-training. the usefulness of synthetic examples is strictly related to their diversity from the original training data. at the same time, diverging too much from the initial data might increase the risk of introducing noisy annotations, i.e., synthetic data not reflecting the class of the original examples they were generated from. to directly tackle the trade-off between diversity and label consistency, in this paper we propose dats 1 (data augmentation based on task-specific similarity), a novel data-augmentation technique for text classification based on natural language generation (nlg) models. starting from a reduced set of annotated examples, we first learn a taskoriented similarity function that we use to automatically create pairs of similar examples. then, we use these pairs to train a generative model to generate an example similar to the input one. finally, we employ this model to generate new synthetic examples and augment the training data. we show that pairing examples with respect to their task-oriented similarity is striking in order to allow the generative model to automatically understand the lexical and structural variations that can be applied to an instance without changing its label. experimental results on four different text classification datasets demonstrate that dats achieves better results than several existing da solutions and that it systematically improves nlu models based on state-of-the-art pre-trained language models. in the remaining, section 2 summarizes related works, section 3 describes our method and section 4 provides the experimental evaluation. ## related work most of the existing approaches for da perform some token-level manipulations on individual sentences @xcite . for instance, easy data augmentation (eda) @xcite applies simple operations including synonym replacement and random swap. these operations are also performed in @xcite where the framework named text autoaugment (taa) uses bayesian optimization algorithm to automatically search for the best manipulation policy. on the contrary, @xcite uses mask random tokens and use bert to generate substituting words. similarly, @xcite use transformer-based models to apply token-level or span-level text content manipulation. other attempts operate at the entire sentencelevel by paraphrasing the original text using backtranslation (bt) @xcite ; however, bt tends to skew towards highfrequency words which not only causes redundancy but also leads to lexical shrinkage in the augmented data @xcite . more recent approaches for data augmentation use generative models to create more diverse synthetic data. @xcite fine-tuned a gpt-2 model to generate text given a target class, and use a text classifier to filter out those synthetic examples whose predicted class does not match the target class. in our work, we show that by conditioning a target model not only on the class labels, but also on representative examples, it is possible to better control the diversity-consistency trade-off. ## learning to generate examples recent advances in nlg @xcite demonstrated that sequence-to-sequence (seq2seq) models can generate natural sounding and meaning-ful text given a prompt. as shown in @xcite , the prompt can include style or contentrelated information that can help control the generation process. in our work, we capitalize these techniques to augment a dataset d for the training of a text classifier c. specifically, our goal is to fine-tune a pre-trained seq2seq model m( * ) = e that, given a prompt * , generates novel examples not in d. the first application of this idea was investigated in @xcite , where the authors trained a seq2seq model m(c) = s o that generates examples s o of a given class c. while this approach provides interesting results, it must be said that the variability of the generated examples can be quite low: by conditioning m only on c, it tends to produce the most frequent syntactic and lexical patterns associated with c, neglecting other modes this can exhibit. the key challenge here is to ensure diversity while preserving consistency, i.e., generating diverse examples that are valid representatives of the desired class. we propose to train a seq2seq model m that is conditioned not only on the class c, but also on an example s i of that class, i.e., such that m(c, s i ) = s 0 . the model is thus expected to synthesize a new example s 0 , which is consistent with the input class c and is also "inspired" by an existing example s i . the problem now is how to build the dataset to train m. pairing two random examples s i and s o belonging to the same class might be a viable solution. however, in classification tasks, examples are not necessarily similar, and coupling together radically different examples risks destabilizing the training of m. ideally, we would like to identify different modes in the same class and pair only examples belonging to the same mode. an alternative way to reach the same outcome is the adoption of a semantic similarity function that can be used to select linguistically related example pairs. several unsupervised metrics exist @xcite @xcite , however, they are inherently task-independent and not adequate in capturing task-specific similarities: two sentences such as "the movie is good" and "the movie is not good" (or even "the movie is awful") should not be paired together, when dealing with sentiment classification, while they are good candidate pairs in the training of a topic classifier. to define a task-oriented similarity measure, we leverage the text classifier c we want to improve via data augmentation. independently on the underlying neural architecture, a neural classifier has an encoder e that projects an example @xmath0, i.e., e(s) = @xmath1. in these spaces, simple linear classifiers (i.e., the output layers) identify the sub-spaces reflecting the target classes @xcite . as a consequence, e is expected to project examples in sub-spaces representative for individual classes. our task-aware similarity measure is the cosine similarity operating on these representations. for instance, after training a bert model on the question classification task @xcite , the encoding [cls] of the entire question "who developed the vaccination against polio?" allows retrieving other questions corresponding to the most similar [cls] embeddings, such as "who invented the moog synthesizer?": they are clearly not paraphrases, and their purely lexical similarity is quite low, but they show a similar pattern characterizing the question class [hum] (i.e., human). to generate an augmented dataset for a generic text classification task, we propose dats (data augmentation based on task-specific similarity) that is described in algorithm 1. first, we train a classifier c on d. we use the resulting encoder e to project each training example e into the taskspecific vector space, obtaining the embedding e. then, we split the training data d into d tr and d te . each example e @xmath2. the resulting pairs are expected to lie in the same subspace and share some task-oriented linguistic relatedness. we use these example pairs to fine-tune a seq2seq model to solve the task m(c e i , e i ) = e j , where c e i is the category of e i . finally, examples in d te are provided in input to m to generate the new synthetic dataset d synt . by applying multiple splits of d in a cross-fold scheme, we can use each training example to condition the model and generate new synthetic instances. as in almost all existing formulations, a generator can be used to generate a set of l variants for each input (c, e), for simplicity referred as m l (c, e). in particular, techniques such as nucleus-sampling @xcite enable the generation of large sets of variants, generally characterized by a good diversity. it is worth noting that no assumption is applied when selecting c or m. even though in the experimental evaluation we consider only specific architectures (namely bert and bart), this methodology can be applied to a wider plethora of models. moreover, there is no restriction on the classification task type: the experimental section shows that dats is applicable to classification tasks operating on both individual sentences and text pairs, e.g., in natural language inference @xcite : given text pairs s i,1 and s i,2 , it is sufficient to extend the above process defining m differently, i.e., m(c (s i,1 ,s i,2 ) , s i,1 , s i,2 ) = (s j,1 , s j,2 ). ## experimental evaluation we test our approach on four tasks: 50-class question classification (qc) over the trec dataset @xcite ; 5-class sentiment classification (sa) over the sst dataset @xcite ; 7-class intent classification (ic) over the snips dataset @xcite ; sentence-pair classification for natural language inference (nli) over the 3-category snli dataset @xcite . details about the datasets are in appendix a.1. baselines. we compare our approach with multiple baselines and simplified versions of dats for ablation study: (i) easy data augmentation (eda) @xcite ; (ii) back-translation (bt) @xcite in an english-german-english setting 2 ; (iii) random pairing (rp) -dats without task-oriented similarity functions where we created the training input-output pairs for m by selecting two random examples of the same class from the training set; (iv) only class (oc) -dats where the prompt is only the category name (i.e., no representative example), similar to @xcite . experimental setting. for qc, sa and ic, we report the accuracy when using @xmath3, @xmath4. for the nli task, since it is more challenging, we report the accuracy also for @xmath5. we also report the performance of each model when using the entire (f) training set 4 . we use the bert-base-uncased model from the huggingface library @xcite as the classifier c, and bart-base @xcite ) as the nlg model m. both are trained for 10 epochs with early stopping (@xmath6. we adopt nucleous sampling 5 @xcite with @xmath7.90. we repeat the experiments 5 times with different seeds and we report the average classification accuracy. results. for instance, in qc or ic when @xmath8, eda accuracy is lower than noda by about 1 and 6 points, respectively. the same is for bt, where the drop is even higher, i.e., 8 and 18 points. using the category information in the generation process, i.e., rp and oc, provides variable results, with minor improvements only in few specific settings (e.g., rp on sa). instead, dats improves accuracy for almost all q and tasks and such improvements are transformersdataaugmentation 3 to maintain the original class distribution we randomly sample 10c, 50c or 100c from the original training set, with c being the number of classes. we ensure that at least one example per class is sampled. 4 we omit results on the full dataset for nli (made of more than 500, 000 examples) for which da is not needed. 5 in preliminary evaluations, we tested alternative decoding strategies, e.g., standard beam search and k-sampling. overall, nucleous sampling was superior in terms of diversity and consistency, in line with the literature @xcite . 6 an analysis of the hyper-parameters is often statistically significant. when @xmath9, the improvement ranges between 1 and 10 points. in the few cases where a minor drop is observed, this is never statistically significant. by comparing dats with rp it is clear that the task-specific similarity measure is striking to learn a good data generator foot_2 . dats works well also on sentence-pair tasks, like nli. however, given the complexity of this type of tasks, it requires more training examples (e.g., @xmath10. furthermore, dats outperforms other competitive methods 8 . with @xmath11. (2020) report 67.30% accuracy in qc and 87.24% in ic, which is 1.74% and 4.5% lower than dats. in the same setting, the gap with wu et al. (2018) is even larger, i.e., 4.71% and 5.95%. moreover, ren et al. (2021) report 52.55% accuracy in sa using the entire training set, and dats outperforms it by 1.71%. qualitative analysis. to better understand the advantages provided by dats in the consistencydiversity trade-off, we report a qualitative evalua-cat. original data dats bt positive a colorful , vibrant introduction to a universal human impulse , lushly photographed and beautifully recorded . this is a real film. a colourful, lively introduction to a universal human impulse, lavishly photographed and beautifully captured. a generous , inspiring film that unfolds with grace and humor and gradually becomes a testament to faith . is in the midst of a monumental adventure, and i can't wait to see what the next one looks like. a generous, inspiring film that unfolds with grace and humor and gradually becomes a witness of faith. a solid and refined piece of moviemaking imbued with passion and attitude . it 's a film that develops with conviction and a deep, passionate heart. a solid and refined piece of film steeped in passion and attitude. enormously enjoyable , highadrenaline documentary . it is a remarkable film... enormously entertaining, adrenalinepumping documentary. negative -spy action flick with antonio banderas and lucy liu never comes together . " it just felt... too heavy... for me to have it in my trunk." -espionage action with antonio banderas and lucy liu never comes together. if you value your time and money , find an escape clause and avoid seeing this trite , predictable rehash . isn't the best spy film ever made. if you value your time and money, find an exit clause and avoid this mundane, predictable repetition. no amount of burning , blasting , stabbing , and shooting can hide a weak script . it is a masterpiece, brilliantly directed, and incredibly well done. no amount of firing, blasting, stabbing and shooting can hide a weak script. this tuxedo ... should have been sent back to the tailor for some major alterations . this is a film that i was unable to stomach... and very much enjoyed... until the last 30 minutes... when i finally got close to the film... and it turned into a garbage dump. this tuxedo... should have been sent back to the tailor for some major changes. in particular, ## conclusions this paper proposes dats, a data augmentation method based on natural language generation (nlg) models. a generative model is fine-tuned to produce examples similar to the input ones. the training input-output pairs are selected according to a task-oriented similarity function. this pairing allows the nlg model to learn the lexical and structural variations that can be applied to an instance without changing its label. the experimental results suggest that the generated sentences are diverse and label consistent, and can improve stateof-the-art text classifiers, outperforming existing da methods. in the future, we plan to apply dats to further tasks (e.g., question answering) and neural architectures. although the above examples seem coherent, we can argue if they can be really new, i.e., useful for the task according to a data augmentation process. we thus focused on the specific class enty:lang that is underrepresented in the original dataset, with only the six original examples reported below: below some of the synthetic examples generated by dats with @xmath14. it is worth noting that language model introduces some expressions like sub-dialect that are novel with respect to the original training material. conversely, if we only use the category name as a prompt (i.e., the oc model), the generated questions will have a lower quality. for instance, with @xmath15, the oc model produces questions like: