Learning to Ignore Adversarial Attacks (2023)

Abstract: Despite the strong performance of current NLP models, they can be brittle against adversarial attacks. To enable effective learning against adversarial inputs, we introduce the use of rationale models that can explicitly learn to ignore attack tokens. We find that the rationale models can successfully ignore over 90% of attack tokens. This approach leads to consistent sizable improvements (~10%) over baseline models in robustness on three datasets for both BERT and RoBERTa, and also reliably outperforms data augmentation with adversarial examples alone. In many cases, we find that our method is able to close the gap between model performance on a clean test set and an attacked test set and hence reduce the effect of adversarial attacks.

## introduction adversarial robustness is an important issue in nlp, asking how to proof models against confounding tokens designed to maliciously manipulate model outputs. as such models become more powerful and ubiquitous, research continues to discover surprising vulnerabilities @xcite , demanding improved robustness methods. a common defense method to combat adversarial attacks is adversarial training. given knowledge of attack strategies, it constructs synthetic adversarial examples to augment clean examples during training @xcite . intuitively, the model will implicitly learn to ignore attacking tokens and become robust to that type of attack. in practice, however, this goal can be challenging through data augmentation alone. in this study, we propose a simple yet effective adversarial training schema for additive attacks: explicitly training the model to ignore adversarial tokens. we do this by augmenting the underlying model with a rationale extractor @xcite to serve as an input filter, and then training this figure 1 : example illustration of an ideal rationale model that is robust to added attack tokens. the rationale extractor filters out the confounding sentence ("lisa invited two friends") added by the adversary, and extracts the supporting spans ("helen invited some friends" and "she bought five cakes, so everyone had exactly one") to help the model deduce the correct answer ("four"). in addition to training the extractor to distinguish the attacking/non-attacking token dichotomy, we also explore the utility of human-provided explanations in this regard. in doing so, we ask: does learning from human rationales help the model avoid attending to attacking tokens? fine-tuning bert @xcite and roberta @xcite on multiple datasets, we demonstrate that the additive attack proposed by @xcite does reduce model accuracy, and that data augmentation with adversarial examples provides limited benefit in defending these models from this attack in most cases. our main results are that rationale-style models learn to ignore these attacks more effectively than only with data augmentation, leading to an improvement of ∼10% in accuracy on attacked examples compared to baseline models and an advantage of 2.4% over data augmentation alone, mostly recovering clean test performance. while human explanations may potentially improve the interpretability of these models, they are of limited use in improving this defense even further. in summary, we offer three main contributions: • we show that explicitly training an extractive rationale layer to ignore attack tokens is more effective than implicitly training a model via data augmentation with adversarial examples. ## related work we build on prior work on adversarial robustness and learning from explanations. adversarial robustness. adversarial attacks against nlp models seek to maliciously manipulate model output by perturbing model input. @xcite present a survey of both attacks and defenses. example attacks include characterlevel manipulations @xcite , input removal @xcite , synonym substitutions @xcite , and language model-based slot filling @xcite @xcite . a distinction in attack types is whether the attack requires access to the model @xcite @xcite or not @xcite . tex-tattack @xcite is a framework and collection of attack implementations. our work focuses on the addsent attack proposed by @xcite in reading comprehension. as interest in adversarial attacks has increased, so has interest in developing models robust to these attacks. a popular defense method is adversarial training via data augmentation, first proposed by @xcite and employed by @xcite to bring their model almost back to clean test performance. a recent example in this vein is @xcite , which proposes dirichlet neighborhood ensemble as a means for generating dynamic adversarial examples during training. another popular approach is knowledge distillation @xcite , which trains an intermediate model to smooth between the training data and the final model. our work explores a new direction that explicitly learns to ignore attacks. learning from explanations. recent work has sought to collect datasets of human-annotated explanations, often in the form of binary rationales, in addition to class labels @xcite , and to use these explanations as additional training signals to improve model performance and robustness, sometimes also known as feature-level feedback @xcite ). an early work is @xcite , which uses human rationales as constraints on an svm. more recently, @xcite uses human rationales to penalize neural net input gradients showing benefits for out-of-domain generalization, while erion et al. (2021) use a similar method based on "expected gradients" to produce improvements in indomain test performance in certain cases. @xcite evaluate feature feedback for two attention-style models, finding, again, gains in outof-domain performance, while @xcite use influence functions @xcite to achieve a similar outcome. where our study differs from most previous work is in using feature feedback for adversarial rather than out-ofdomain robustness. a concurrent work by @xcite uses rationalization to improve robustness. the proposed method is similar to our work, but we explore supervision with attack tokens and achieve stronger robustness to additive attacks. ## adversarial attacks and datasets in this paper, we focus on model robustness against the addsent additive attack proposed by @xcite . the attack is designed for reading comprehension: consider each instance as a tuple of document, query, and label (d, q, y), where y indicates whether the query is supported by the document. the attack manipulates the content of the query to form an attack sentence (a) and adds a to the document to confuse the model. specifically, addsent proceeds as follows: query q: fc bayern munich was founded in 2000. mutated query q: dynamo leverkusen cologne was founded in 1998. modified document d ′ . . . has won 9 of the last 13 titles. dynamo leverkusen cologne was founded in 1998. they have traditional local rivalries with . . . 1. we modify the query q by converting all named entities and numbers to their nearest neighbor in the glove embedding space @xcite . we flip all adjectives and nouns to their antonyms using wordnet @xcite and yield a mutated query q. if we fail to mutate the query due to not being able to find matching named entities or antonyms of adjectives and nouns, we skip the example. 2. we convert the mutated query q into an adversarial attack a using corenlp @xcite constituency parsing, under a set of about 50 rules enumerated by @xcite . this step converts it into a factual statement that resembles but is not semantically related to the original query q. 3. the adversarial attack a is inserted at a random location within the original document and leads to a new tuple (d ′ , q, y). 1 the key idea behind the addsent attack is that the mutations alter the semantics of the query by mutating the named entities and numbers, so that the attack contains words or phrases that are likely confusing to the model without changing the true semantics of the input. an example of the addsent attack is given above. the original approach includes an additional step of using crowdsourced workers to filter ungrammatical sentences. we do not have access to this manual validation process in all datasets. occasionally, addsent generates ungrammatical attacks but it nevertheless proves empirically effective in reducing the performance of our models. datasets. to evaluate our hypotheses on learning to ignore adversarial attacks, we train and evaluate models on the multi-sentence reading comprehension (multirc; @xcite and fact extraction and verification (fever; thorne et al., 2018) datasets. both are reading compre-dataset text length rationale length total size multirc 336.0 52.0 32,088 fever 335.9 47.0 110,187 squad 119.8 --87,599 in modeling these two datasets, we follow standard practice in appending the query to the end of the document with [sep] tokens. we use train/validation/test splits prepared by the eraser dataset collection @xcite . because we are interested in relative differences between training regimes rather than absolute performance, we subsample the fever training set to 25% so that it is comparable to multirc for the sake of training efficiency. directly applying the synthetic addsent attack to multirc and fever leads to occasionally ungrammatical adversarial examples due to incorrectly applied conversion heuristic or errors in constituency parsing. to alleviate this concern, we further evaluate on squad @xcite , for which jia and liang provide an addsentattacked evaluation set that is re-written and approved by human workers. however, this dataset does not have human rationales. we again use the train/validation/test splits provided by jia and liang in our experiments. ## modeling our study assesses whether adding an explicit rationale extractor to a model and training it to ignore attack tokens results in a more effective defense than simply adding attacked examples to the training set. this comparison results in several combinations of model architectures and training regimes. we denote each training instance as (x, r, y): a text sequence x consisting of the concatenated document and query, a ground-truth binary rationale sequence r, and a binary label y. baseline models and training. we use bert @xcite and roberta @xcite as basis models. in the baseline training condition we fine-tune these models as normal, evaluating them on both the original test set and a version of the test set where each item has been corrupted with the addsent attack described above. we denote this condition as "no adv." in the baseline adversarial training via data augmentation condition (denoted adv.), we add addsent-attacked versions of each training example to the training set on a one-to-one basis, allowing the model to train for the presence of such attacks. this represents a fairly standard baseline defense in the literature @xcite . following prior adversarial robustness literature @xcite , we also consider a stronger baseline by augmenting the training set with k perturbed examples for each training example. for our main experiments, we use @xmath0. this setting (denoted adv.-10x) measures whether the baseline method implicitly adapts to the addsent attack when abundant signal is provided. rationale model. to lend the baseline model an extractor capable of filtering out confounding tokens, we use the rationale model proposed by @xcite . it comprises a rationale extractor g and a label predictor f (fig. 1 ). the rationale extractor generates a binary predicted rationale r, which is applied as a mask over the input to the predictor via masking function m, producing a predicted label: the two components are trained together to optimize predicted label accuracy as well as loss associated with the predicted rationale. in an unsupervised scenario, this loss punishes the norm of the predicted rationale, encouraging sparsity on the (heuristic) assumption that a sparse rationale is more interpretable. in this study, we rather consider the supervised scenario, where we punish r's error with respect to a ground-truth rationale r. however, we find empirically that the rationale sparsity objective is useful in combination with the rationale supervision objective, leading to the following joint objective function using cross-entropy loss l ce with hyperparameter weights λ 1 and λ 2 : ## experimental setup and results we start by describing our experimental setup and evaluation metrics. we then investigate model performance with different training regimes and conduct an in-depth error analysis. ## concluding discussion in this study, we find that adding an explicit extractor layer helps a model learn to ignore additive adversarial attacks produced by the addsent attack more effectively than conventional adversarial training via data augmentation. this is an exciting result because it defeats an attack which is otherwise stubbornly effective against even copious adversarial data augmentation. it is a novel use for this type of explicit token relevance representation, which is more typically applied for model interpretability @xcite . this makes it related to defenses like @xcite which allow the model to reject inputs as out-of-distribution and abstain from prediction, but it differs in rejecting only part of the input, making a prediction from the remainder as usual. generality. as @xcite note, it is easy to overstate claims in evaluating adversarial defenses. hence, we note that our results pertain only to the addsent attack, and perform favorably against a baseline defense in adversarial training via data augmentation. since most adversarial training approaches assume the ability to generate a large number of synthetic attack examples, it is reasonable to further assume that we have access to the positions of the attacks. however, such knowledge about attacks may not be available in general. nevertheless, the success of the rationale model architecture in learning to occlude adversarial tokens does hold promise for a more general defense based on a wider range of possible attacks and possible defenses by the extractor layer. utility of human rationales. we explore the possibility that human-provided explanations may make the model more robust against adversarial attacks. we mostly find that they do not, with the notable exception of bert on multirc, the only case in which the augmentation brings the model back to clean test accuracy. while it does provide an advantage of sparsity over supervision with non-attack tokens, this advantage alone may not justify the cost of collecting human explanations for robustness. further understanding of human rationales and novel learning strategies are required for improving model robustness. future directions. a generalization of our approach might convert the "extractor" layer into a more general "defender" layer capable of issuing a wider range of corrections in response to a wider range of attacks. it could, for example, learn to defend against attacks based on input removal (e.g. feng et al. (2018) ) by training to recognize gaps in the input and fill them via generative closure. this defender could be coupled with a self-supervision style approach (e.g., @xcite ) involving an "attacker" capable of levying various types of attack against the model. we leave such a generalization for future work.