Impact of Adversarial Training on Robustness and Generalizability of Language Models (2023)

Abstract: Adversarial training is widely acknowledged as the most effective defense against adversarial attacks. However, it is also well established that achieving both robustness and generalization in adversarially trained models involves a trade-off. The goal of this work is to provide an in depth comparison of different approaches for adversarial training in language models. Specifically, we study the effect of pre-training data augmentation as well as training time input perturbations vs. embedding space perturbations on the robustness and generalization of transformer-based language models. Our findings suggest that better robustness can be achieved by pre-training data augmentation or by training with input space perturbation. However, training with embedding space perturbation significantly improves generalization. A linguistic correlation analysis of neurons of the learned models reveal that the improved generalization is due to ‘more specialized’ neurons. To the best of our knowledge, this is the first work to carry out a deep qualitative analysis of different methods of generating adversarial examples in adversarial training of language models.

## introduction language models (lms) have emerged as the backbone of many tasks in ai and have extended their reach beyond nlp applications into vision and even reinforcement learning @xcite @xcite . thus it is imperative that the generalizability and robustness of lms be carefully assessed and evaluated. generalizability is the ability of a model to perform well on unseen data. transformer-based models that are pre-trained on large unlabeled text have shown remarkable generalization ability. however, when confronted with carefully designed adversarial samples, their robustness -the ability to gracefully deal with small perturbations, suffers significantly. for example, a recent study has shown that on a classification task on a yelp data set, accuracy dropped by almost 90%, when a standard test set was replaced by an adversarial counterpart @xcite @xcite . adversarial training is a pragmatic approach to attain both generalizability and robustness. the idea is straightforward. for a given model m , generate adversarial samples that target m and then use the samples to incrementally re-train the model. this can be done either at the pre-training or the fine-tuning stage @xcite . adversarial samples can be generated both in the input space and in the embedding space. the original work on the creation of adversarial samples for computer vision was in the input space. for example, the fast gradient sign method (fgsm) @xcite that perturbs a data point x along the direction of the sign gradient of the loss function with respect to the input is an example of a perturbation in the input space. in the context of natural language inputs, perturbing text is challenging due to its discrete nature. unlike continuous data, there is no systematical way to guarantee an increase in the loss function when perturbing text. for instance, if we aim to make a small modification to the word "robust" we can choose to replace a single letter within the word or substitute it with a near synonym. however, both of these perturbations may seem ad-hoc and not sufficiently principled to intentionally increase the loss function. therefore, in language settings, it is often more appropriate to perform perturbations in the embedding space, where continuous representations can be manipulated in a more structured manner. furthermore, despite the widespread use of adversarial training to increase the robustness of models, it is not clear what their impact is on downstream tasks beyond the model's overall accuracy. for example, a deeper analysis of language models has shown that different parts of the network are responsible for different parts of speech @xcite @xcite @xcite . in this regard, the change in the network due to adversarial training has not yet been investigated. overall our contributions in this paper are threefold. firstly, we introduce two techniques in the context of adversarial training in the embedding space, representing the regularization-and gradient-based approaches commonly used by latent space techniques. we compare these techniques using a simple one-dimensional model and hypothesize their behavior in adversarial scenarios. secondly, we evaluate the effectiveness of inputand embedding-space adversarial training methods in terms of their generalization ability and robustness against various types of adversarial attacks in sentiment analysis. lastly, we conduct a thorough linguistic analysis of an adversarially trained model and demonstrate that incorporating robustness through adversarial training leads to more "focused" neurons that are associated with distinct part of speech (pos) tags. the rest of the paper is organized as follows. in section 2, we discuss adversarial attacks and defenses, with a specific focus on the nlp domain. section 3 provides a detailed explanation of embedding space adversarial techniques. in section 4, we conduct experiments to analyze the trade-off between robustness and generalization achieved by data augmentation, input-space training, and embedding space training approaches, considering various well-known adversarial attacks. additionally, we present our findings from linguistic correlation analysis of neurons in robust models within the same section. finally, we finalized the paper in the concluding section. ## related work adversarial attacks: the purpose of an adversarial attack is to cause a model to output conflicting decisions for an input and its 'imperceptibly' modified version. an adversarial sample is defined as: obtaining an exact solution for the perturbation δ is a very challenging problem. further, even when close approximations are considered, the solution gets computationally very expensive @xcite . to solve this problem more efficiently, gradient-based methods were introduced. accordingly, the perturbation δ is computed by taking one @xcite or more steps iteratively @xcite in the direction of the gradient to maximize the loss function. then, this high loss point is projected back onto the input space to determine the norm-bounded perturbation. in practice, projected gradient descent (pgd) approaches that, take several small steps in the direction of the gradient, are used most frequently to create strong adversarial samples @xcite . other than gradient based approaches, jacobianbased saliency map attack (jsma) @xcite uses the jacobian matrix created from forward derivation of input to identify to importance of each input component to the target attack. deepfool @xcite , alternatively, iteratively linearizes the classifier to identify the minimum perturbation that causes a change in the classification label. carlini & wagner attack (c&w) proposed defensive distillation strategy @xcite based approach. adversarial attacks in nlp: running adversarial attacks against natural language processing (nlp) models is more challenging than widely used vision models. the discrete nature of word representations, combined with the tokenization of words into word pieces, effectively invalidates any algorithm that applies differential changes on the model input when generating an adversarial sample. moreover, quantification of the extent to which semantic similarity and contextual relations are preserved between a text input and its modified version is not trivial. to circumvent these limitations, many adversarial sample generation algorithms adopted the approach of substituting one or more words in the input until a misprediction occurs. the crux of this attack lies in identification of alternative words or phrases that retain the semantic intactness of the original input. for this, several meth-ods based on word-embedding similarity @xcite , word synonymity @xcite , and masked language model predictions @xcite ) are proposed. however, finding appropriate word candidates may get computationally very intensive. for a sentence consisting of m words with n candidates to substitute each word, there are (n + 1) m possible combinations to test. to perform this search efficiently, greedy search @xcite , genetic algorithm @xcite , and particle swarm optimizationbased (pso) @xcite approaches are proposed and incorporated with word importance as determined by gradient measurements (yoo and qi, 2021) and word deletion @xcite . an alternative approach to above substitutionbased approach is applying perturbations in the embedding space directly to word embeddings. this approach avoids the expensive search step to identify the best word substitution configuration, but it requires devising a mapping from perturbed embeddings to the text domain in order to create an adversarial sample. to realize this, recent work @xcite adapted a gradient-based adversarial sample generation method to compute perturbations associated with each word embedding. perturbed embeddings are then translated to input domain using a pre-trained masked-language modeling (mlm) head, as in @xcite , to create an adversarial sample that is semantically similar to the original input. ## at with embedding space perturbations among all adversarial defenses developed for language processing models, moving the adversarial training from the input space to the embedding space offers the most advantage. this essentially allows the adoption of gradient-based adversarial training approaches that are computationally less demanding than input space methods. although a plethora of such adversarial training methods exists, they are all essentially guided by two main principles in their approach. the first one essentially sets the training objective to minimize the loss due to worst-case perturbation induced on the training samples, instead of the average loss computed from training samples by the standard training. this group of methods essentially differ in the way they approximate the worst-case perturbation @xcite @xcite as well as the extent and nature of perturbation applied during generation of adversarial samples @xcite @xcite . the second approach primarily relies on the premise that smoothness is an important requirement of a robust model. to this objective, these methods focus on minimization of a regularized version of the loss instead of optimizing only the standard, training loss. the regularization term here ensures that there is a wide enough margin around each training data point with the decision boundary of the model through minimizing the difference between the predictions of natural and adversarial samples. methods following this approach are distinguished based on their formulation of regularization @xcite and their coupling with the training loss described above @xcite . in our analysis, we consider two representative methods that most effectively exemplify each approach. in practice, due to its computational efficiency, the pgd attack is most frequently used for the creation of adversarial samples. we will refer to this generic adversarial training approach as pgd-at. the latter approach is also best characterized by the use of pgd in ensuring local distribution smoothness around natural samples. this alternative method will be referred to as lds. we must note that improved variants of the two base methods should be expected to perform better. in this regard, robustness-generalization performance of the pgd-at and lds can be interpreted as lowerbounds. the steps of both methods are presented in algorithm 1 where the lines that differ between the two methods are highlighted as pink for pgd-at and blue for lds. both methods start by randomly initializing δ with normal distribution with a mean of zero and standard deviation of σ. the loss is then calculated between the model's output of the perturbed input depending on the method, pgd-at or lds. the δ value is then updated by the gradient and clipped to @xmath0. these steps are repeated for s times. the loss value is then updated by combining the standard loss with the loss associated with each method. gradient update is then applied to model parameters. to better examine the behavior of the two methods, we analyze a simple one-dimensional linear ## conclusions in this paper we have carried out an extensive study of adversarial training methods (atms) to understand their impact on robustness and generalizability for transformer-based deep language models. we can draw the following conclusions from our study. first, non-adversarial data augmentation improves both generalization and robustness over the baseline bert model. adversarial training in the input space yields better robustness compared to both non-adversarial data augmentation and embedding space adversarial training. in contrast, adversarial training in the embedding space exhibits best generalization behavior. among pgd-at and lds methods, our results show that the pgd-at is consistently more robust and generalizable. overall, our results show that unlike in computer vision domain where gradient-based adversarial training yields the best robustness and generalization tradeoff, for language processing models input-space training methods are indispensable. for future work we will consider combining data augmentation, input-space training, and embedding space training approaches together. we would also like to extend our theoretical understanding of the trade-off between robustness and generalizability for language models. in connection, the impact of atms for other downstream applications needs to be studied.