Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup (2021)

Abstract: Mixup is a recent regularizer for current deep classification networks. Through training a neural network on convex combinations of pairs of examples and their labels, it imposes locally linear constraints on the model’s input space. However, such strict linear constraints often lead to under-fitting which degrades the effects of regularization. Noticeably, this issue is getting more serious when the resource is extremely limited. To address these issues, we propose the Adversarial Mixing Policy (AMP), organized in a “min-max-rand” formulation, to relax the Locally Linear Constraints in Mixup. Specifically, AMP adds a small adversarial perturbation to the mixing coefficients rather than the examples. Thus, slight non-linearity is injected in-between the synthetic examples and synthetic labels. By training on these data, the deep networks are further regularized, and thus achieve a lower predictive error rate. Experiments on five text classification benchmarks and five backbone models have empirically shown that our methods reduce the error rate over Mixup variants in a significant margin (up to 31.3%), especially in low-resource conditions (up to 17.5%).

## introduction deep classification models have achieved impressive results in both images @xcite and language processing @xcite @xcite . one of the most significant challenges to train a deep model is the great efforts and costs to collect largescale labels. without sufficient labels, the deep networks tend to generalize poorly, leading to unsatisfactory performance. thus, the regularization techniques under augmentation schema, which generate labeled data to regularize models @xcite , are widely explored @xcite . mixup @xcite is an effective regularizer under the augmentation schema. in recent years, topics related to mixup have warranted serious attention @xcite @xcite @xcite @xcite . the core idea of mixup is to generate synthetic training data via a mixing policy, which convex combines a pair of examples and its labels. through training on these data, the classification networks will be regularized to reach higher performance. unlike conventional regularizers @xcite @xcite , mixup imposes a kind of locally linear constraint @xcite on the model's input space. however, vanilla mixup often suffers from underfitting due to the ambiguous data @xcite @xcite generated under the strict locally linear constraints. to alleviate the under-fitting, @xcite uses extra parameters to project the inputs and labels into a high dimensional space to properly separate the data. @xcite use auxiliary networks to learn the mixing policy in a data-driven way to avoid the generation of ambiguous data. although existing works effectively reduce the underfitting, they have limitations to properly regularization networks. current networks are prone to be over-fitting when adding the extra parameters. eventually, these methods degrade the effects of regularization. the conflicts between over-fitting and under-fitting get more serious when the labeled resources are rare or hard to obtain. besides, the methods with auxiliary networks usually have difficulties in integrating with other mixup variants. more importantly, mixup works well in most cases @xcite . adding too much non-linearity into mixup will sacrifice the majority of synthetic data that can regularize the networks under locally linear constraints. so, the locally linear constraints in mixup only need to be slightly relaxed. in this paper, we propose the adversarial mixing policy (amp) to overcome these limitations. we modify the adversarial training @xcite , which relaxes the linear nature of the network without any extra parameters or auxiliary networks, to relax the locally linear constraints in mixup. inspired by the "min-max" formulation of adversarial training, we formulate our method as a form of "min-max-rand" regularization. specifically, the "rand" operation randomly samples a mixing coefficient as in vanilla mixup to generate synthetic example and label. then, the "max" operation calculates the perturbation of the mixing coefficient and applies it. note that the updated mixing coefficient is only used to re-synthetic example, keeping the synthetic label unchanged. thus, slight non-linearity is injected in-between the synthetic example and label. finally, the "min" operation minimizes the training loss over the non-linearly generated example-label pairs. in summary, we highlight the following contributions: ## methodology as shown in figure 1 , adversarial mixing policy (amp) consists of three operations: rand, max and min. rand operation (randop) generates the synthetic data by interpolating pairs of training examples and their labels with a random mixing coefficient λ. max operation (maxop) injects a small adversarial perturbation into the λ to resynthesize the example and keeps the synthetic label unchanged. this operation injects slight nonlinearity into the synthetic data. min operation (minop) minimizes the losses of these data. additionally, we use a simple comparison to eliminate the influence caused by the scaling of gradients. ## related work mixup on text classification. text classification has achieved remarkable improvements underlying some effective paradigms, e.g., cnn @xcite , attention-based lstms @xcite , glove @xcite and @xcite , etc. the large scale parameter of the model tends to generalize poorly in lowresource conditions. to overcome the limitation, mixup @xcite is proposed as a data augmentation based regularizer. few researches explore the mixup @xcite @xcite on nlp tasks. for classification, @xcite suggest applying mixup on particular level of networks, i.e., word or sentence level. although these work make promising progress, the mechanism of mixup is still need to be explored. adversarial training. the min-max formulation of adversarial training has been theoretically and empirically verified @xcite @xcite @xcite @xcite . such training procedure first generates ad-versarial examples that might maximize the training loss and then minimizes the training loss after adding the adversarial examples into the training set @xcite . the fast gradient sign method (fgsm) @xcite is an efficient one-step method. inspired by the min-max formulation of adversarial learning, we organize our method into a min-max-rand formulation. ## conclusion for relaxing locally linear constraints (llc) in mixup to alleviate the under-fitting, this paper proposes an adversarial mixing policy (amp). inspired by the adversarial training, we organize our method into a min-max-rand formulation. the proposed method injects slightly non-linearity inbetween synthetic examples and synthetic labels without extra parameters. by training on these data, the networks can compatible with some ambiguous data and thus reduce under-fitting. thus, the network will be further regularized to reach better performance. we evaluate our method on five popular classification models on five publicly available text datasets. extensive experimental results show that our amp can achieve a significantly lower error rate than vanilla mixup (up to 31.3%), especially in low-resource conditions(up to 17.5%).