Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data (2021)

Abstract: Unsupervised Data Augmentation (UDA) is a semisupervised technique that applies a consistency loss to penalize differences between a model’s predictions on (a) observed (unlabeled) examples; and (b) corresponding ‘noised’ examples produced via data augmentation. While UDA has gained popularity for text classification, open questions linger over which design decisions are necessary and how to extend the method to sequence labeling tasks. In this paper, we re-examine UDA and demonstrate its efficacy on several sequential tasks. Our main contribution is an empirical study of UDA to establish which components of the algorithm confer benefits in NLP. Notably, although prior work has emphasized the use of clever augmentation techniques including back-translation, we find that enforcing consistency between predictions assigned to observed and randomly substituted words often yields comparable (or greater) benefits compared to these more complex perturbation models. Furthermore, we find that applying UDA’s consistency loss affords meaningful gains without any unlabeled data at all, i.e., in a standard supervised setting. In short, UDA need not be unsupervised to realize much of its noted benefits, and does not require complex data augmentation to be effective.

## introduction while the advent of large neural models has led to rapid progress on a wide spectrum of prediction benchmarks in nlp, these methods tend to require large amounts of training data. this limitation is particularly acute in domains such as information extraction from scientific documents, where unlabeled in-domain data is plentiful but labeled data is rare and requires significant annotator experience to produce. the cost of acquiring data in such domains has spurred interest in developing models that can achieve greater extraction accuracy, even when the available labelled corpora are small @xcite . in this paper, we investigate unsupervised data augmentation (uda; @xcite , a recently proposed semi-supervised learning method in which models are trained on both labeled and unlabeled in-domain data. the learning objective for the unlabeled component entails minimizing the divergence between the model's outputs on a given example and its outputs on a perturbed version of the same example. while this combination of data augmentation with a consistency loss was previously proposed as invariant representation learning and demonstrated utility in the context of speech recognition @xcite , uda applies the method in a semi-supervised setting, in a manner similar to virtual adversarial training @xcite . the original uda paper @xcite reported significant benefits on both computer vision and nlp tasks. producing such perturbed examples requires specifying a data augmentation pipeline. typically, these apply one or more transformations that (the practitioner hopes) do not alter the applicable label @xcite . in computer vision, a number of straightforward and demonstrably effective data augmentation techniques, such as horizontal flipping, cropping, rotating, small translations, and various perturbations to the color spectrum have gained widespread adoption @xcite . more recently, these methods, among others, have been successfully applied in concert with uda to improve performance on image classification tasks @xcite . by contrast, in nlp, there is less consensus about which perturbation models can be applied with confidence that they will not change the applicability of the original label. to apply uda in nlp, researchers have primarily focused on backtranslation @xcite , generating paraphrases by applying a machine translation model to map a document into a pivot language and then back into the original language. in practice, this process produces augmentations of varying quality. another problem is that back-translation is slow, and performance may depend on arbitrary choices concerning both the translation model and pivot language. incorporating large quantities of unlabeled data in the training process is also computationally expensive. given these limitations-and to better characterize why and when uda helps-we investigate whether the benefits of uda on nlp tasks can be achieved using less unlabeled data and/or simpler input perturbations. to this end, we investigate uniform random word replacement as an augmentation method. random substitution for augmentation has been considered previously in the context of translation @xcite , and the uda paper @xcite gives preliminary results using random replacement for a single dataset, where it slightly underperforms back-translation. here we deepen this analysis, showing that, surprisingly, random replacement is generally competitive with backtranslation. furthermore, we find that significant increases in performance can be achieved by applying consistency loss on only small labeled data sets (although large volumes of in-domain unlabeled data provide further gains). as an additional contribution, we adapt uda to sequence tagging tasks, which are common in nlp. back-translation is ill-suited to such tasks, because we lack alignment between spans of interest in the original text and the back-translated paraphrase. for these problems, we propose and evaluate word replacement augmentation strategies for sequence tagging. interestingly, we observe that augmentation via uniform random word replacement yields improvements, but that it is more effective to employ a masked language model to predict reasonable candidate replacements. ## unsupervised data augmentation uda is a semi-supervised method in which a model is trained to make similar predictions for an observed example and a corresponding perturbed instance produced via some data augmentation technique (in addition to satisfying the standard objectives over labeled data). applying uda requires specifying both (i) a consistency loss to be applied on an (original, augmented) example pair; and (ii) a data augmentation technique to produce the perturbed examples in the first place. ## experimental setup we evaluate our proposed training method on four english language text classification and three english language sequence tagging datasets. of these, the classification datasets include three benchmark sentiment sets (imdb, yelp, and amazon), and one scientific classification task (evidence inference). the sequence tagging datasets include one standard ner benchmark dataset @xcite and two scientific sequence labeling tasks (ebm-nlp and tac). the scientific tasks are of particular interest for this work because it is expensive to collect annotations in these specialized domains. imdb @xcite is a sentiment classification dataset consisting of movie reviews (25,000 examples in both the train and test sets) drawn from the imdb website. reviews with a @xmath0, those with @xmath1. neutral reviews are excluded. yelp @xcite is a sentiment classification dataset comprising reviews drawn from yelp (560,000 in the train set, 38,000 in the test set). one and two star reviews are considered negative. three and four star reviews are considered positive. amazon @xcite ) is a sentiment classification dataset consisting of amazon reviews (3, 600, 000 in the train set, 400, 000 in the test set). one and two star reviews are considered negative. four and five star reviews are considered positive. three star reviews are not included. evidence inference @xcite we construct a classification dataset derived from the evidence inference dataset, a biomedical corpus in which the task is to infer the effect of an intervention on an outcome from an article describing a randomized controlled trial. the classes correspond to the intervention leading to a significant increase, significant decrease, or no significant change in outcome. in the original task, the model must first extract relevant evidence sentences from the full text article, and then make a prediction based on this. we evaluate in the 'oracle' setting, in which the model must only classify given relevant evidence sentences (∼17, 000 train examples, and ∼2, 000 instances in the test set). conll-2003 @xcite ) is an ner dataset consisting of annotated reuters news articles (∼14, 000 sentences in the training set, ∼3, 000 in the test set), labeled with entity categories person, organization, location, and miscellaneous. tac @xcite comprises annotated "materials and methods" sections from pubmed central articles (∼5, 500 sentences in the training set, ∼6, 500 in the test set). labels are available for 24 entity classes, of which we consider the two best represented: end point and test article. ebmnlp @xcite is a corpus of annotated abstracts drawn from medical articles describ-ing medical randomized controlled trials (∼28, 000 sentences in the training set, ∼2, 000 in the test set). spans are tagged as describing the patient population, the intervention studied, and the outcome measured in the trial being described. for each dataset we simulate a pool of unlabeled data by hiding the annotations of the training set. we then create five distinct sets of labeled data (ten for sequence tagging tasks) by revealing the annotations for a random subset of the pool, forming a labeled training set l and an unlabeled training set u. for classification tasks, we sample ten examples per class to form l, while for conll and ebmnlp, we sample two hundred examples. for the smaller tac dataset, we use one hundred. training details for each l, we then train a model both using only standard supervised learning over l and with an additional consistency loss, using either uniform random word replacement or more complex data augmentation technique (backtranslation for classification; bert-based replacement for sequence tagging). when training with consistency loss, we evaluate variants in which we apply this to both l and u, and where we apply it only over l. the latter corresponds to a standard supervised setting with an additional loss term. bert's pretraining task already incorporates unsupervised data @xcite . we therefore also repeat the above experiments using finetuned weights instead of off-the-shelf pretrained weights. the finetuned weights are produced by training on bert's masked language model task with l ∪ u as the training data. in our classification experiments, we use a linear model on top of bert @xcite as a classifier. for sequence tagging, we follow architecture and hyperparameter choices in prior work @xcite , adding a conditional random field @xcite on top of bert representations. we train all models using adam (kingma and ba, 2015) with a learning rate of 2e-5 for classification and 1e-3 for sequence tagging. in exploratory experiments we observed that model performance is relatively robust to the choice of λ (the weight assigned to the consistency loss term) when large quantities of unlabeled data are available. we therefore set λ to 1 in all of our semisupervised experiments. in our supervised learning only experiments, we found it necessary to compensate for the lack of unlabeled examples and the corresponding change in the relative weightings of standard and consistency losses. this can be effectively done by repeating labeled examples, imposing only the consistency loss for them, as though they were unlabeled. we use a ratio of 20 "unlabeled" examples to 1 labeled example in our supervised experiments. in our augmentation procedure, we set p to 30%. for biomedical tasks (evidence inference, ebmnlp, tac) model weights are initialized using scibert, a model pretrained over scientific papers @xcite . for all other tasks, we initialized parameters to the pretrained bert base weights @xcite . bert base weights are used for conll, and scibert weights are used for ebmnlp and tac when using bert as a masked-language model for data augmentation. ## conclusions in this paper we evaluated and extended uda in the context of nlp tasks. we proposed and evaluated new techniques for applying uda to classification tasks and extended it to sequence tagging tasks by imposing a consistency loss over word label distributions. we showed that naive data augmentation methods may often be just as effective as the complex, model based augmentations that are currently fashionable, and that performance improvements may still be attained even in the absence of any unlabeled data. we proposed a simple, effective augmentation method: randomly replace words with other words. the replacement word may be selected either uniformly at random, or by using bert @xcite as a masked language model to induce a probability distribution over tokens.we found that the former method is effective for classification tasks, and the latter for sequence tagging. we further investigated the practicality of using uda without unlabeled data, applying a consistency loss only to a small labeled dataset. we experimentally evaluated various augmentation strategies and settings on four classification datasets and three sequence tagging datasets. we found reliable performance increases on all four classification datasets. for classification, we found that random word replacement is as effective as-and sometimes more effective than-backtranslation, which is what has been proposed in prior work @xcite . in particular, random word-replacement is effective on our scientific classification task, where back-translation is ineffective, perhaps due to its reliance on machine translation models not trained with scientific literature. our experiments show that both random replacement and bert-based replacement are effective on two out of three sequence tagging tasks, with bert-based replacement that we have proposed consistently outperforming random replacement. on the third sequence tagging dataset, we observed a degradation of performance when using any variety of uda, which we hypothesize owes to the noisy annotation of this training set. finally, we found that uda may produce meaningful increases in performance even when unlabeled data is unavailable, particularly if the weights have already been finetuned to the task. the magnitude of this increase depends upon the task, and may be relatively large (as in the case of the amazon dataset) or relatively small (as in the conll dataset). in general this approach is more effective for classification tasks than sequence tagging tasks. to summarize our findings: uda is effective in low-supervision natural language tasks, even when used with naive augmentation methods and without unlabeled data.