D ouble M ix: Simple Interpolation-Based Data Augmentation for Text Classification (2022)

Abstract: This paper proposes a simple yet effective interpolation-based data augmentation approach termed DoubleMix, to improve the robustness of models in text classification. DoubleMix first leverages a couple of simple augmentation operations to generate several perturbed samples for each training data, and then uses the perturbed data and original data to carry out a two-step interpolation in the hidden space of neural models. Concretely, it first mixes up the perturbed data to a synthetic sample and then mixes up the original data and the synthetic perturbed data. DoubleMix enhances models’ robustness by learning the “shifted” features in hidden space. On six text classification benchmark datasets, our approach outperforms several popular text augmentation methods including token-level, sentence-level, and hidden-level data augmentation techniques. Also, experiments in low-resource settings show our approach consistently improves models’ performance when the training data is scarce. Extensive ablation studies and case studies confirm that each component of our approach contributes to the final performance and show that our approach exhibits superior performance on challenging counterexamples. Additionally, visual analysis shows that text features generated by our approach are highly interpretable.

## introduction deep neural networks have enabled breakthroughs in most supervised settings in natural language processing (nlp) tasks. however, labeled data in nlp is often scarce, as linguistic annotation usually costs large amounts of time, money, and expertise. with limited training data, neural models will be vulnerable to overfitting and can only capture shallow heuristics that succeed in limited scenarios, which will lead to severe performance degradation when applied to challenging situations. in order to improve the robustness of models, various data augmentation methods have been proposed. generally, there are three types of augmentation techniques: token-, sentence-, and hiddenlevel transformation. @xcite summarized several common token-level transformations, including word insertion, deletion, replacement, and swap. sentence-level transformation is to paraphrase a sentence through specific grammatical or syntactic rules. back-translation @xcite is a typical sentencelevel augmentation method where a sentence is translated to an intermediate language and then translated back to obtain augmented samples. additionally, for natural language inference (nli) tasks that identify whether a premise entails, contradicts, or is neutral with a hypothesis, @xcite studied syntactic rules of sentences in inference tasks and proposed several syntactic transformation techniques such as inversion and passivization to construct syntactically informative examples. however, these methods often have high requirements for sentence structures. it is hard to obtain a large number of augmented samples by this method. in recent years, several hidden-level augmentation methods are proposed and they have exhibited superior performance in a number of popular text classification tasks. tmix @xcite ) is a typical approach where a linear interpolation is performed in the hidden space of transformer models such as bert @xcite . the main idea of tmix @xcite comes from mixup, a method that is based on the principle of vicinal risk minimization (vrm) @xcite and has achieved substantial improvements in computer vision tasks @xcite @xcite and natural language tasks @xcite @xcite . recently, ssmix @xcite which interpolates text based on the saliency of tokens @xcite in hidden space has been introduced. these methods make models learn a mapping from a mixed text representation to an intermediate label which is generated by linearly combining two different source labels. however, the intermediate soft label cannot always accurately describe the true probability of classes that the mixed text representations belong to, which limit the effectiveness of augmentation. to overcome these limitations, this work proposes a simple yet effective interpolation-based data augmentation method termed doublemix, which performs interpolation in the hidden space and does not require label mixing. firstly, we leverage a collection of simple augmentation operations to generate several perturbed samples from the raw data and then mix up these perturbed samples. secondly, we mix up the original data with the synthesized perturbed data. we constrain the mixing weight of the original to be larger than the synthesized perturbed data to balance the trade-off between proper perturbations and the potential injected noise. to stabilize the training process, we add a jensen-shannon divergence regularization term to our training objective to minimize the distance between the predicted distributions of the original data and the perturbed variants. to demonstrate the effectiveness of our approach, we conduct extensive experiments by comparing our doublemix with previous state-of-theart data augmentation methods on six popular text classification benchmark datasets. additionally, we reduce and vary the amount of training data, to observe if doublemix can consistently improve over the baselines. we further conduct ablation studies and case studies to investigate the impact of different training strategies on doublemix's effectiveness and whether our method works on challenging counterexamples. moreover, we visualize the features generated by doublemix to interpret why our method works. experimental results and analyses confirm the efficacy of our proposed approach and every component in dou-blemix contributes to the performance. to sum up, our contributions are: ## related work data augmentation techniques are widely employed in nlp tasks to improve the robustness of models @xcite @xcite @xcite . one way to enrich the original training set is to perturb the tokens in each sentence. for example, @xcite introduced a set of simple data augmentation operations such as synonym replacement, random insertion, swap, and deletion. however, token-level perturbation sometimes does not guarantee that the augmented sentences are grammatically correct. thus, sentence-level augmentation methods are introduced, where people paraphrase the sentence by some specific rules. minervini and riedel (2018); @xcite leveraged syntactic rules to generate adversarial examples in inference tasks. moreover, andreas (2020) investigated the compositional inductive bias in sequence models and augmented data by compositional rules. however, these methods require careful design, and they are often customized for a specific task, which makes them hard to generalize to different datasets. recently, a couple of hidden-level augmentation techniques which perform interpolation in hidden space have been studied @xcite @xcite . inspired by puzzlemix @xcite and saliencymix (uddin et al., 2020) which is popular in computer vision, @xcite proposed ssmix which utilizes the saliency information of spans @xcite in each sentence to interpolate in hidden space to create informative examples. @xcite interpolate hidden states of the entire minibatch to obtain better representations. inspired by the prior work, our doublemix aims at improving models' robustness by mixing up text features and their perturbed samples in hidden space. ## proposed method: doublemix to regularize nlp models in a more efficient manner, we introduce a simple yet effective data augmentation approach, doublemix, that enhances the representation of each training data by learning the features sampled from a region constructed by the original sample itself and its perturbed samples. the perturbed samples are generated by simple token-or sentence-level augmentation operations. doublemix is a hidden-level regularization technique and our base model is a pre-trained transformer network, as they have achieved great performance in various nlp tasks. algorithm 1 shows the training process of our approach. ## results and analysis we evaluate our baselines and proposed approach on six text classification benchmark datasets. we also show the performance of our approach in lowresource settings to confirm doublemix is efficient and robust when the training samples are scarce. this section will discuss the performance of models in detail and quantitatively analyze how and why doublemix works. ## conclusion in this work, we present a simple interpolationbased data augmentation approach doublemix to improve models' robustness on a wide range of text classification datasets. doublemix first leverages simple augmentation operations to generate perturbed data of each training sample and then performs a two-step interpolation in the hidden space of models to learn robust representations. our approach outperforms several popular data augmentation methods on six benchmark datasets and three low-resource datasets. finally, ablation studies, case studies, and visualization of manifold fur-ther explain how and why our method works. our future work includes making the mixing weights learnable as well as extending doublemix to natural language generation tasks. on the glue benchmark, as the test labels are not publicly available. we used the original sst-2 dataset and it can be loaded from huggingface datasets foot_1 .