Text Augmentation in a Multi-Task View (2021)

Abstract: Traditional data augmentation aims to increase the coverage of the input distribution by generating augmented examples that strongly resemble original samples in an online fashion where augmented examples dominate training. In this paper, we propose an alternative perspective—a multi-task view (MTV) of data augmentation—in which the primary task trains on original examples and the auxiliary task trains on augmented examples. In MTV data augmentation, both original and augmented samples are weighted substantively during training, relaxing the constraint that augmented examples must resemble original data and thereby allowing us to apply stronger augmentation functions. In empirical experiments using four common data augmentation techniques on three benchmark text classification datasets, we find that using the MTV leads to higher and more robust performance than traditional augmentation.

## introduction most data augmentation techniques aim to generate augmented examples for training that are similar to original data. in computer vision, operations such as flipping, cropping, and color jittering are both widely used and highly effective-it is selfevident that augmented examples closely resemble original data, and so we generate augmented data in an online fashion during each minibatch such that no original, unmodified examples are seen during training @xcite @xcite . in language, on the other hand, even slight modifications can cause significant semantic changes, and so it is not always clear whether augmented examples resemble original data. despite this uncertainty, many augmentation techniques in nlp still ## traditional data augmentation 1 situation. during regular training, the canonical maximum likelihood objective minimizes the cost of the original training set j o : remedy. in practice, we often use data augmentation to mitigate the inadequacy of p p p(x, y ) by providing additional training data. we generate an augmented distribution q q q( x, ŷ ) and now minimize the cost of this augmented training set j aug : as we now optimize solely on q q q( x, ŷ ), our goal is to find (x, ŷ) pairs that are likely to fall in the true distribution p p p. assuming the smoothness of p p p, similar (x, y) pairs will have similar probabilities, and therefore if an augmented example is more similar to an observed example, it is more likely to be sampled under the true distribution. in other words, good augmented examples resemble the observed data, and we aim to find them. conversely, if an augmented example diverges too far from any observed data, it is likely invalid and thus harmful for training; we don't want to train on these examples. the majority of prior work follows this framework of augmented examples resembling real data. as popular techniques, semantic noising substitutes tokens with synonyms @xcite @xcite ; pervasive dropout randomly removes words from the input sequence @xcite ; and switchout (for machine translation) replaces some words in both source and target sentences with other words from their corresponding vocabularies @xcite . moreover, most of these techniques perform augmentation on every training example in an online fashion, implicitly assuming that augmented examples so closely resemble original data that directly 1 we closely follow the intuition and notation of @xcite training on original examples is not even worth considering. as we shall see in the next section, adding in these original examples during training might actually be a worthwhile idea. ## mtv data augmentation multi-task optimization jointly trains on a primary task and one or more auxiliary tasks-the intuition is that requiring an algorithm to also learn an auxiliary task can act as better regularization than penalizing all complexity uniformly. prior work has found that multi-task models work particularly well when the tasks are similar, but can also improve performance even on unrelated tasks @xcite . we propose a multi-task view of data augmentation that has a primary task that optimizes regular training on original examples and an auxiliary task that optimizes training on augmented data. this mtv jointly optimizes the primary and auxiliary task(s) using a weighted cost function so that both original and augmented data receive substantial weight during training: the mtv gives us an important freedom that is not offered by the traditional data augmentation framework. since traditional data augmentation only trains on augmented examples, performance suffers detrimentally when augmented data differs too much from the true distributiontherefore, most studies aim to generate augmented examples that resemble original data. mtv data augmentation, however, jointly trains on both original and augmented data, thereby allowing us to relax the constraint that original and augmented examples come from the same distribution. in fact, accepting that the original and augmented distributions might differ or could even be unrelated-as work in multi-task learning has done @xcite @xcite )-liberates us to apply stronger levels of data augmentation, which, as we will demonstrate in the next section, leads to higher and more robust performance. ## experiments this section compares multi-task view augmentation to traditional augmentation for various datasets and augmentation techniques. ## conclusions we have proposed a multi-task view that gives both original and augmented examples substantial weight during training, contrasting prior work that performs stochastic data augmentation and ignores original training data. for four common augmentation techniques, we found experimentally that this alternative view allows for stronger levels of augmentation, which in turn leads to better and more robust performance than traditional augmentation. we hope our paper inspires future work using text data augmentation to think more explicitly about how much augmented examples resemble original data and consider substantive weighting of origi-nal data when using data augmentation to improve model performance. to close, we leave the enthusiastic reader with one last thought. most existing text data augmentation techniques have obediently followed the paradigm from computer vision of generating augmented examples that are similar to the original data. who's to say that's how data augmentation ought to work in nlp? in this paper, we've shown how to search for relative freedom from this constraint, simply by taking a different view of the underlying assumptions. now, a bigger question arises on the horizon-what new text augmentation techniques are unlocked when augmented data are not forced to resemble the original?