Targeted Adversarial Training for Natural Language Understanding (2021)

Abstract: We present a simple yet effective Targeted Adversarial Training (TAT) algorithm to improve adversarial training for natural language understanding. The key idea is to introspect current mistakes and prioritize adversarial training steps to where the model errs the most. Experiments show that TAT can significantly improve accuracy over standard adversarial training on GLUE and attain new state-of-the-art zero-shot results on XNLI. Our code will be released upon acceptance of the paper.

## introduction adversarial training has proven effective in improving model generalization and robustness in computer vision @xcite and natural language processing (nlp) @xcite @xcite @xcite . it works by augmenting the input with a small perturbation to steer the current model prediction away from the correct label, thus forcing subsequent training to make the model more robust and generalizable. aside from some prior work in computer vision @xcite , most adversarial training approaches adopt non-targeted attacks, where the model prediction is not driven towards a specific incorrect label. in nlp, the cutting-edge research in adversarial training tends to focus on making adversarial training less expensive (e.g., by reusing backward steps in freelb @xcite ) or regularizing rather than replacing the standard training objective (e.g., in virtual adversarial training (vat) @xcite ). by contrast, in this paper, we investigate an orthogonal direction by augmenting adversarial training with introspection capability and adopting targeted attacks to focus on where the model errs the most. we observe that in many nlp applications, the error patterns are non-uniform. for example, in the mnli development set (in-domain), standard fine-tuned bert model tends to misclassify a non-neutral instance as "neutral" more often than the opposite label (figure 1 top). we thus propose targeted adversarial training (tat), a simple yet effective algorithm for adversarial training. for each instance, instead of taking adversarial steps away from the gold label, tat samples an incorrect label proportional to how often the current model makes the same error in general, and takes adversarial steps towards the chosen incorrect label. to our knowledge, this is the first attempt to apply targeted adversarial training to nlp tasks. in our experiments, this leads to significant improvement over standard non-adversarial and adversarial training alike. for example, in the mnli development set, tat produced an accuracy gain of 1.7 absolute points (figure 1 bottom). on the overall glue benchmark, tat outperforms state-of-theart non-targeted adversarial training methods such as freelb and vat, and enables the bert base model to perform comparably to the bert large model with standard training. the benefit of tat is particularly pronounced in out-domain settings, such as in zero-shot learning in natural language inference, attaining new state-of-the-art cross-lingual results on xnli. ## targeted adversarial training (tat) in this paper, we focus on fine-tuning bert models @xcite in our investigation of targeted adversarial training, as this approach has proven very effective for a wide range of nlp tasks. the training algorithm seeks to learn a function f (x; θ) : @xmath0, where c is the class label set. given a training dataset d of input-output pairs (x, y) and the loss function l(., .) (e.g., cross entropy), the standard training objective would minimize the empirical risk: by contrast, in adversarial training, as pioneered in computer vision @xcite @xcite , the input would be augmented with a small perturbation that maximize the adversarial loss: recently, adversarial training has been successfully applied to nlp as well @xcite @xcite . in particular, freelb @xcite leverages the free adversarial training idea @xcite by reusing the backward pass in gradient computation to carry out inner ascent and outer descent steps simultaneously. smart @xcite instead algorithm 1 tat input: t : the total number of iterations, @xmath1, y 1 ), ..., (x n , y n )}: the dataset, f (x; θ): the machine learning model parametrized by θ, σ 2 : the variance of the random initialization of perturbation δ, : perturbation bound, k: the number of iterations for perturbation estimation, η: the step size for updating perturbation, τ : the global learning rate, α: the smoothing proportion of adversarial training in the augmented learning objective, π: the projection operation and c: the classes. 1: for @xmath2, .., t do 2: effectively, the adversarial term encourages smoothness in the input neighborhood, and α is a hyperparameter that controls the trade-off between standard errors and adversarial errors. in standard adversarial training, the algorithm simply tries to perturb the input x away from the gold label y given the current parameters θ. it is agnostic to which incorrect label f (x) might be steered towards. by contrast, in targeted adversarial training (tat), we would explicitly pick a target y @xmath3. intuitively, we would like to focus training on where the model currently errs the most. we accomplish this by keeping a running tally of e(y, y t ), which is the current expected error of predicting y t when the gold label is y, and sample y t from c \@xmath4, y t ). see algorithm 1 for details. tat can be applied to the original adversarial training or virtual adversarial training alike. in this paper, we focus on adapting virtual adversarial training (vat) @xcite . the two lines in blue color are the only change from vat. we initialize e(y, y t ) with uniform distribution and update them in each epoch. ## experiments we compare targeted adversarial training (tat) with standard training and state-of-the-art adversarial training methods such as freelb @xcite and vat @xcite . we use the standard uncased bert base model @xcite , unless noted otherwise. due to the additional overhead incurred during training, adversarial methods are somewhat slower than standard training. like vat, tat requires an additional k adversarial steps compared to standard training. in practice, @xmath5, so they are just slightly slower (roughly 2 times compared to standard training). freelb, by contrast, typically requires 2-5 steps to attain good performance, so is significantly slower. ## conclusion we present the first study to apply targeted attacks in adversarial training for natural language under-standing. our tat algorithm is simple yet effective in improving model generalizability for various nlp tasks, especially in zero-shot learning and for out-domain data. future directions include: applying tat in pretraining and other nlp tasks e.g., sequence labeling, exploring alternative approaches for target sampling.