R - AT : Regularized Adversarial Training for Natural Language Understanding (2022)

Abstract: Currently, adversarial training has become a popular and powerful regularization method in the natural language domain. In this paper, we Regularized Adversarial Training (R-AT) via dropout, which forces the output probability distributions of different sub-models generated by dropout to be consistent under the same adversarial samples. Specifically, we generate adversarial samples by perturbing the word embeddings. For each adversarial sample fed to the model, R-AT minimizes both the adversarial risk and the bidirectional KL-divergence between the adversarial output distributions of two sub-models sampled by dropout. Through extensive experiments on 13 public natural language understanding datasets, we found that R-AT has improvements for many models (e.g., rnn-based, cnn-based, and transformer-based models). For the GLUE benchmark, when R-AT is only applied to the fine-tuning stage, it is able to improve the overall test score of the BERT-base model from 78.3 to 79.6 and the RoBERTa-large model from 88.1 to 88.6. Theoretical analysis reveals that R-AT has potential gradient regularization during the training process. Furthermore, R-AT can reduce the inconsistency between training and testing of models with dropout.

## introduction with the development of computing hardware, deep learning @xcite has achieved surprising performance in many artificial intelligence domains, such as, image recognition, speech recognition, natural language understanding, etc. however, "overfitting" is one of the biggest concerns in a deep learning community @xcite @xcite . at present, in the actual training of deep neural networks, various regularization techniques @xcite @xcite are indispensable to prevent overfitting and improve the generalization ability of deep neural networks. among them, dropout is currently one of the most widely used regularization methods, which randomly drops some neural units and all their input and output connections during the training process. the most significant advantage of dropout is that it can easily generate exponential parametersharing sub-models during the training process. @xcite and @xcite put data to pass through the model with dropout twice to generate different representations or output distributions, and then use varying representations for contrastive learning or regularization of distinct output distributions, which enhances the model's effect to a certain extent. furthermore, adversarial training is used to minimize the maximal risk for label-preserving input perturbations, which has been demonstrated to be a powerful regularization method for improving model generalization. some of the latest adversarial training methods @xcite have accomplished impressive performance, particularly in natural language processing tasks. however, when adversarial training is executed in combination with dropout, the model inputting the same adversarial example will generate a relatively different output distribution. this phenomenon will affect the model's inference performance and stability. inspired by simcse @xcite and r-drop @xcite , we proposed a simple regularization technique upon adversarial training and dropout, called r-at, which regularizes adversarial training by "twice adversarial dropout". specifically, the proposed method r-at minimizes bidirectional kl-divergence between output distributions of the two sub-models sampled via dropout for each adversarial sample generated by adding gradient-based perturbation to clean examples. in this process, the constrained model predicts the distribution of adversarial examples, acquires information about the distribution of non-target classes, and enhances generalization in inference. it is also worth noting that compared with standard adversarial training, most previous works on improving adversarial training have added multiple forward and back propagations, whereas our r-at only increased the forward propagation once, without increasing the number of back propagations. the main contributions of this paper can be summarized as follows: ## related work it is well known that overfitting has always been a significant issue for deep learning models. empirically, people find that the best-fitting model is typically a large model with appropriate regularization @xcite . to mitigate overfitting and improve the generalization of deep learning models, numerous regularization techniques have been proposed, including weight decay @xcite @xcite , dropout @xcite @xcite , normalization @xcite @xcite @xcite , adding noise @xcite @xcite , layer-wise pretraining and initialization @xcite , label-smoothing @xcite @xcite , adversarial training @xcite @xcite @xcite , and so on. among them, dropout and its variants are very widely used due to their effectiveness and excellent compatibility with other regularization methods. the most significant advantage of dropout is that it can easily generate exponential parameter-sharing sub-models throughout the training process. @xcite regard dropout as a data enhancement method, which permits data to pass through the model with dropout twice to generate distinct embedding vectors. @xcite further regularize the neural networks on dropout success, which is done at the model output level. in this way, they encourage any two sub-models sampled from the dropout to generate consistent model predictions for the same input sample. on the other hand, adversarial training (goodfellow et al., 2015) was first proposed in the field of image recognition to defend against adversarial attacks. to be specific, adversarial training is the process of training a model that minimizes the maximal adversarial risk for label-preserving input perturbations. for the field of natural language processing, adversarial training @xcite has recently been proven to be a powerful regularization method to improve model generalization. numerous studies have recently employed multiple iterations to calculate better perturbations to further enhance adversarial training @xcite @xcite . however, this requires the model to perform forward-backpropagation multiple times in one parameter update, linearly increasing computational cost. as known, the main calculations of neural networks are concentrated in the backpropagation. the focus of this work is to enhance standard adversarial @xcite training without increasing the number of back propagations. we employed the "twice dropout" method to regularize adversarial training. specifically, for each adversarial sample generated by adding gradient-based perturbation to clean samples, the proposed method r-at minimizes bidirectional kl-divergence between output distributions of two sub-models sampled by dropout. in this process, the constrained model predicts the distribution of adversarial samples, acquires information about the distributions of non-target classes, and enhances generalization. ## the proposed method in this section, we first briefly describe the standard fine-tuning procedure for transformer-based model on natural language understanding tasks. we then introduce our regularized adversarial training (r-at) method that perturb the word embedding matrix of neural networks to generate adversarial examples and further regularize the model's prediction distribution for adversarial samples. finally, we introduce in detail the training process of the regularized adversarial training algorithm. figure 1 depicts the overall framework of r-at. ## study and analysis on r-at in this section, we conduct detailed research and analysis on r-at from multiple aspects. ## conclusion in this paper, we propose a simple yet powerful regularization method based on adversarial training and dropout, namely r-at. particularly, r-at dynamically perturbs the input embedding vector to generate adversarial examples, and then an adversarial example is passed through two dropout sub-models to generate two probability distributions. ultimately, r-at minimizes bidirectional kl-divergence of two output distributions to regularize model predictions. we evaluate the efficiency of r-at on 13 popular public datasets. the experimental results indicate that applying r-at to neural network models with various structures (e.g., rnnbased, cnn-based, and transformer-based models) all significantly improves performance. moreover, theoretical analysis reveals that r-at has potential gradient regularization during the training process, implying that the model tends to be optimized to a flatter minimum risk to improve generalization. furthermore, r-at can reduce inconsistency between the training and testing of dropout models.