Dynamic Nonlinear Mixup with Distance-based Sample Selection (2022)

Abstract: Data augmentation with mixup has shown to be effective on the NLP tasks. Although its great success, the mixup still has shortcomings. First, vanilla mixup randomly selects one sample to generate the mixup sample for a given sample. It remains unclear how to best choose the input samples for the mixup. Second, linear interpolation limits the space of synthetic data and its regularization effect. In this paper, we propose the dynamic nonlinear mixup with distance-based sample selection, which not only generates multiple sample pairs based on the distance between each sample but also enlarges the space of synthetic samples. Specifically, we compute the distance between each input data by cosine similarity and select multiple samples for a given sample. Then we use the dynamic nonlinear mixup to fuse sample pairs. It does not use a linear, scalar mixing strategy, but a nonlinear interpolation strategy, where the mixing strategy is adaptively updated for the input and label pairs. Experiments on the multiple public datasets demonstrate that dynamic nonlinear mixup outperforms state-of-the-art methods.

## introduction deep neural networks have achieved great success in nlp tasks, such as text classification @xcite , machine translation @xcite , and dialogue tasks @xcite . these models usually require a large amount of labeled data, but labeling data is time-consuming and expensive. data augmentation is a common technology to solve the data scarcity problem. some of them are based on rules @xcite and models @xcite to generate similar text. augmented data are trained by advanced training methods @xcite . on the other hand, mixup @xcite trains the classifier on the synthetic data which are generated by the linear interpolation of the input and label pairs. recently, many variants of mixup are proposed and successfully applied to nlp tasks. it mainly includes two categories: input-level mixup @xcite and hidden-level mixup @xcite . hidden-level mixup is more popular because of the discrete nature of text data and variable sequence lengths. it usually fuses the hidden vectors like embeddings or intermediate representations. while the hidden-level mixup is effective, it still has some issues. first, mixup produces the sample pairs by randomly choosing a sample for a given sample. how to select the optimal input and label pairs is unclear. second, the space of synthetic data is limited due to its linear nature. although nonlinear mixup @xcite is proposed and utilizes the matrix mixing policy for the sample pairs, the matrix is fixed during the training process and it is difficult to get the appropriate parameters. in this paper, we propose a dynamic nonlinear mixup with distance-based sample selection method to address the above problems. first, we compute the distance between each input sample by cosine similarity. then we pick the top-k samples with the largest distance and the top-k samples with the smallest distance for a given sample. second, we obtain mixed inputs by performing a dynamic mixing strategy on these sample pairs. unlike nonlinear mixup @xcite , where the fixed matrix fusion method is applied to the sample pairs, our method uses the vector mixing policy and dynamically updates its parameters. the feature space of the generated data is further enlarged by this way. since the dimensions of the one-hot label and vector mixing policy are different, we learn the label embedding instead of the original label. the mixing policy of labels is adaptively learned based on the mixed input. in summary, the main contributions of our work are summarized as follows: ## related work traditional data augmentation methods are categorized into adding noise and back-translation. easy data augmentation (eda) @xcite proposes four methods to add noise: synonym replacement, random insertion, random deletion, and random swap. it can create a large number of augmented training samples but may lose semantic information due to random deletion. backtranslation (bt) @xcite trains target-to-source system to generate the source language. bt keeps the semantics of the original text, but the augmentation data depends on the quality of the translation model. another class of methods is based on the technique of interpolation. a data augmentation method mixup @xcite is proposed and has shown good performance in the image classification. mixup trains the classifier on the synthetic data which are generated by the linear interpolation of the input and label pairs. in nlp tasks, it usually uses hidden-level mixup. word-mixup and senmixup @xcite are two linear interpolation methods that are implemented on the word embeddings and sentence embeddings. @xcite incorporate mixup to transformerbased pre-trained architecture to boost the performance of nlu tasks. @xcite propose a novel mixup strategy for pre-trained language models by both the area under the margin (aum) statistic and the saliency map of each sample. @xcite propose a data augmentation method called tmix for semi-supervised learning, which interpolates labeled and unlabeled samples in the hidden space to improve the performance of text classification. batchmixup @xcite utilizes interpolation strategy in a minibatch to improve model performance on the pretrained language model roberta @xcite . @xcite propose the saliency-based span mixup (ssmix), which generates a sentence while reserving the locality of two original texts by spanbased mixing and keeping more tokens related to the prediction relying on saliency information. the above methods just randomly select one sample for a given sample. the number of generated samples is insufficient. we choose top-k largest distance samples and top-k smallest distance samples according to the distance between each sample. it can generate more sample pairs for mixup. besides, mixup uses linear interpolation which limits the space of generated data. we utilize dynamic nonlinear mixup which expands the space of the generated data. similar to us, the nonlinear mixup @xcite uses the fixed matrix mixing strategy. finding an appropriate hyperparameter is difficult. in contrast, our method utilizes a vector mixing strategy that can be dynamically updated throughout the training process. ## approach in this section, we first present the mixup, followed by distance-based sample selection, and the details of the dynamic nonlinear mixup. finally, the training strategy is introduced. figure 1 gives an overview of our method. ## conclusion in this paper, we propose the dynamic nonlinear mixup with distance-based sample selection method to enhance the performance of pre-trained language models. first, we introduce distancebased sample selection to choose the top-k largest distance samples and top-k smallest distance samples for a given sample. more generated samples improve the generalization ability of the model. second, we utilize the dynamic nonlinear mixing policy on the input sample pairs to enlarge the space of the synthetic samples. the mixed labels are constructed through the learned label embeddings and mixed input so that the mixed labels are updated adaptively. experiments on the subj, mr, sst-2, and sst-1 datasets demonstrate that our method outperforms the state-of-the-art methods. for future work, we plan to further enhance performance by exploring different weighting schemes for origin samples and augmented samples.