D rop M ix: A Textual Data Augmentation Combining Dropout with Mixup (2022)

Abstract: Overfitting is a notorious problem when there is insufficient data to train deep neural networks in machine learning tasks. Data augmentation regularization methods such as Dropout, Mixup, and their enhanced variants are effective and prevalent, and achieve promising performance to overcome overfitting. However, in text learning, most of the existing regularization approaches merely adopt ideas from computer vision without considering the importance of dimensionality in natural language processing. In this paper, we argue that the property is essential to overcome overfitting in text learning. Accordingly, we present a saliency map informed textual data augmentation and regularization framework, which combines Dropout and Mixup, namely DropMix, to mitigate the overfitting problem in text learning. In addition, we design a procedure that drops and patches fine grained shapes of the saliency map under the DropMix framework to enhance regularization. Empirical studies confirm the effectiveness of the proposed approach on 12 text classification tasks.

## introduction deep neural networks have shown its effectiveness, achieving state-of-the-art result in many natural language processing (nlp) tasks. however, these models are likely to overfit training sets with few samples due to the large number of parameters they contain. regularization techniques, such as weighted decay @xcite , dropout @xcite , and data augmentation @xcite are powerful to overcome overfitting by making slight modifications on the neural network or data space to improve generlization. data augmentation methods have shown to be useful for overcoming overfitting by increasing the * corresponding author the environment is bad n x c food in this restaurant bad delicious 0.83p unfortunately, three problems concurrently occur in ssmix which limits its ability to generate expressive examples. firstly, although ssmix has a strong flexibility by exploring a larger synthetic space to generate new samples, this space is in fact high-dimensional as the input data consists of words rather than embeddings. besides, words within this space is disordered and therefore may lead to unpredictable sampling results. this leads to the second and third problems, where there is a potential of generating "dirty" examples or causing a label drift (i.e., newly added sample is categorized under a wrong class), as it does not consider the position of words or the sentence syntax structure in the mixup. for instance, x c can be said to be dirty. besides being wrong grammatically, the generated example contains contrasting opinion words (i.e., "bad","delicious") offering confusing clues to determine its label. to tackle the aforementioned issues, we propose a patch-aware dropmix framework, a data augmentation method which takes advantage of dropout and mixup, to generate new examples based on the dimension and saliency of words in texts. in contrast to ssmix, dropmix maps the high-dimensional input data (i.e., words) into a lowdimensional space to encode the semantics of the text input and mitigate the unpredictability of generated examples. despite the problems encountered by ssmix, empirical results indicate that sampling from a large synthetic space may have some benefits. this may be intuitively desirable, as the model is not constrained to generate examples from only the input text representations. to compensate for this, dropmix applies dropout on the embeddings of x a (and x b ) and replaces the dropped out units with noise from x b (and x a ) to reduce the overdependence on the input features. however, since the saliency of words in texts impacts the class decision, the activation neurons in the features to be dropped depends on the saliency with a beta prior (or drop ratio). in other words, we present a general patch-aware dropmix framework that drops a patch surrounding the peak salient region of one text input embedding and mixes it with the corresponding patch of another text input embedding to generate new examples. under this framework, we develop other variants of dropmix that mixes text input embeddings using different fine-grained patches based on the saliency and dimension of words in text. in a nutshell, we make the following contributions in this paper: ## related work a variety of regularization techniques have been proposed to solve overfitting in deep neural networks @xcite @xcite . some of the notable and recent methods include, vanilla mixup @xcite which linearly interpolates two input samples and their associated labels to generate augmented versions for regularization. cutmix @xcite cuts and pastes image patches to construct new examples, and mixes the labels in proportion to the cut patch sizes. manifold mixup @xcite on the other hand interpolates the middle hidden layers' feature maps. patchup @xcite proposes a method of patches swap or interpolation on the hidden layer feature maps. saliencymix (uddin et al., 2021) mixes a target image with a saliency map @xcite of the source image that highlights important objects. although the majority of these approaches were designed for vision tasks, they have been widely adopted for nlp @xcite @xcite . in the context of text classification, mixup regularization variants including, wordmixup and sen-mixup @xcite perform interpolation on the word and sentence embeddings. @xcite shows that mixup can improve the accuracy upon both cnn and lstm sentence classification models. mixtext @xcite extends the hidden layer feature interpolation method to semisupervised text classification task. to overcome the mixing difficulty in the discrete text input spaces, seqmix @xcite performs token-level interpolation in the embedding space and selects a token closest to the interpolated embedding. ss-mix @xcite performs mixup operation on the input text rather than hidden vectors as like previous methods @xcite @xcite . specifically, ssmix synthesizes a sentence while preserving the locality of two original texts by relying on high saliency spanbased mixing. this work is closely related to our methods, but we argue that the synthesis sanity of its input word level mixup rather generates "dirty" samples. ## preliminaries our proposed model, dropmix, builds upon notable works @xcite relating to data augmentation for text classification. specifically, given a text @xmath0, x 2 , ..., x n ] with n words, x is typically represented as low-dimensional features @xmath1, generated through an encoder such as bert @xcite . a classifier takes x as input and maps it to one or more class labels y . to improve the classification problem, conventional data augmentation methods such as dropout @xcite and mixup @xcite have been utilized to encourage generalization. here, we describe the basics of these data augmentation methods as they are fundamental to our approach. ## dropmix for text augmentation in this paper we combine the advantages of dropout and mixup to develop a new approach for data augmentation, namely, dropmix (fig. 1 ). specifically, dropmix encourages a model to reduce the over-dependence on input units while behaving non-linearly in-between training samples. the idea is to dropout units in x a and replace their positions with corresponding units in x b based on the saliency of the units. we define the augmentation approach as follows: ## conclusion in this paper, we present a data augmentation framework, dropmix, based on the saliency map which combines dropout and mixup to facilitate regularization in text learning. our best results obtained from our model variant dropmix-g indicates that the features of the word embeddings are important for patch selection and mixing. in general, our empirical results confirm the effectiveness of dropmix.