Reevaluating Adversarial Examples in Natural Language (2020)

Abstract: State-of-the-art attacks on NLP models lack a shared definition of a what constitutes a successful attack. We distill ideas from past work into a unified framework: a successful natural language adversarial example is a perturbation that fools the model and follows some linguistic constraints. We then analyze the outputs of two state-of-the-art synonym substitution attacks. We find that their perturbations often do not preserve semantics, and 38% introduce grammatical errors. Human surveys reveal that to successfully preserve semantics, we need to significantly increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences. With constraints adjusted to better preserve semantics and grammaticality, the attack success rate drops by over 70 percentage points.

## introduction one way to evaluate the robustness of a machine learning model is to search for inputs that produce incorrect outputs. inputs intentionally designed to fool deep learning models are referred to as adversarial examples @xcite . adversarial examples have successfully tricked deep neural networks for image classification: two images that look exactly the same to a human receive ⇤ * equal contribution 1 our code and datasets are available here. completely different predictions from the classifier @xcite . while applicable in the image case, the idea of an indistinguishable change lacks a clear analog in text. unlike images, two different sequences of text are never entirely indistinguishable. this raises the question: if indistinguishable perturbations are not possible, what are adversarial examples in text? the literature contains many potential answers to this question, proposing varying definitions for successful adversarial examples @xcite . even attacks with similar definitions of success often measure it in different ways. the lack of a consistent definition and standardized evaluation has hindered the use of adversarial examples to understand and improve nlp models. 2 therefore, we propose a unified definition for successful adversarial examples in natural language: perturbations that both fool the model and fulfill a set of linguistic constraints. in section 2, we present four categories of constraints nlp adversarial examples may follow, depending on the context: semantics, grammaticality, overlap, and non-suspicion to human readers. by explicitly laying out categories of constraints adversarial examples may follow, we introduce a shared vocabulary for discussing constraints on adversarial attacks. in section 4, we suggest options for human and automatic evaluation methods for each category. we use these methods to evaluate two sota synonym substitution attacks: geneticattack by @xcite and textfooler by @xcite . human surveys show that the perturbed examples often fail to fulfill semantics and non-suspicion constraints. additionally, a grammar checker detects 39% more errors in the perturbed examples than in the original inputs, including many types of errors humans almost never make. in section 5, we produce tfadjusted, an attack with the same search process as textfooler, but with constraint enforcement tuned to generate higher quality adversarial examples. to enforce semantic preservation, we tighten the thresholds on the cosine similarity between embeddings of swapped words and between the sentence encodings of original and perturbed sentences. to enforce grammaticality, we validate perturbations with a grammar checker. as in textfooler, these constraints are applied at each step of the search. human evaluation shows that tfadjusted generates perturbations that better preserve semantics and are less noticeable to human judges. however, with stricter constraints, the attack success rate decreases from over 80% to under 20%. when used for adversarial training, textfooler's examples decreased model accuracy, but tfadjusted's examples did not. without a shared vocabulary for discussing constraints, past work has compared the success rate of search methods with differing constraint application techniques. @xcite reported a higher attack success rate for textfooler than @xcite did for geneticattack, but it was not clear whether the improvement was due to a better search method foot_1 or more lenient constraint application 4 . in section 6 we compare the search methods with constraint application held constant. we find that geneticattack's search method is more successful than textfooler's, contrary to the implications of @xcite . the five main contributions of this paper are: ## constraints on adversarial examples in natural language we define f : x ! y as a predictive model, for example, a deep neural network classifier. x is the input space and y is the output space. we focus on adversarial perturbations which perturb a correctly predicted input, x 2 x , into an input x adv . the boolean goal function g(f, x adv ) represents whether the goal of the attack has been met. we define c 1 ...c n as a set of boolean functions indicating whether the perturbation satisfies a certain constraint. adversarial attacks search for a perturbation from x to x adv which fools f by both achieving some goal, as represented by g(f, x adv ), and fulfilling each constraint c i (x, x adv ). the definition of the goal function g depends on the purpose of the attack. attacks on classification frequently aim to either induce any incorrect classification (untargeted) or induce a particular classification (targeted). attacks on other types of models may have more sophisticated goals. for example, attacks on translation may attempt to change every word of a translation, or introduce targeted keywords into the translation @xcite . in addition to defining the goal of the attack, the attacker must decide the constraints perturbations must meet. different use cases require different input, x: "shall i compare thee to a summer's day?" -william shakespeare, sonnet xviii ## constraint evaluation methods and case study for each category of constraints introduced in section 2, we discuss best practices for both human and automatic evaluation. we leave out overlap due to ease of automatic evaluation. additionally, we perform a case study, evaluating how well black-box synonym substitution attacks geneticattack and textfooler fulfill constraints. both attacks find adversarial examples by swapping out words for their synonyms until the classifier is fooled. geneticattack uses a genetic algorithm to attack an lstm trained on the imdb foot_4 document-level sentiment classification dataset. textfooler uses a greedy approach to attack an lstm, cnn, and bert trained on five classification datasets. we chose these attacks because: to generate examples for evaluation, we attacked bert using textfooler and attacked an lstm using geneticattack. we evaluate both methods on the imdb dataset. in addition, we evaluate textfooler on the yelp polarity document-level sentiment classification dataset and the movie review (mr) sentence-level sentiment classification dataset @xcite . we use 1, 000 examples from each dataset. ## producing higher quality adversarial examples in section 4, we evaluated how well generated examples met constraints. we found that although attacks in nlp aspire to meet linguistic constraints, in practice, they frequently violate them. now, we adjust automatic constraints applied during the course of the attack to produce better quality adversarial examples. we set out to find if a set of constraint application methods with appropriate thresholds could produce adversarial examples that are semanticspreserving, grammatical and non-suspicious. we modified textfooler to produce tfadjusted, a new attack with stricter constraint application. to enforce grammaticality, we added language-tool. to enforce semantic preservation, we tuned two thresholds which filter out invalid word substitutions: (a) minimum cosine similarity between counter-fitted word embeddings and (b) minimum grammar rule id x x adv explanation context to non base 2 123 did you mean "know"? --replace with one of [know] ...ees at person they don't really want to knew prp vbg 3 112 did you mean "we're wanting", "we are wanting", or "we were wanting"? --replace with one of [we're wanting,we are wanting,we were wanting] while we wanting macdowell's character to retrieve her h... a plural 20 294 don't use indefinite articles with plural words. did you mean "a grate", "a gratis" or simply "grates"? --replace with one of [a grate,a gratis,grates] a grates, lanky flick did baseform 25 328 the verb 'can't' requires base form of this verb: "compare" --replace with one of [compare] ...first two cinema in the series, i can't compares friday after next to them, but nothing ... prp vb 6 73 do not use a noun immediately after the pronoun 'it'. use a verb or an adverb, or possibly some other part of speech. --replace game with one of [] ...ble of being gravest, so thick with wry it game like a readings from bartlett's familia... prp md nn 4 46 it seems that a verb or adverb has been misspelled or is missing here. --replace with one of [can be appreciative,can have appreciative] ...y bit as awful as borchardt's coven, we can appreciative it anyway non3prs verb 7 78 the pronoun 'they' must be used with a non-third-person form of a verb: "do" --replace with one of @xcite they does a ok operating of painting this family ... through human studies, we found threshold values of 0.9 for (a) and 0.98 for (b) 9 . we implemented tfadjusted using textattack, a python framework for implementing adversarial attacks in nlp @xcite . ## comparing search methods when an attack's success rate improves, it may be the result of either (a) improvement of the search method for finding adversarial perturbations or (b) more lenient constraint definitions or constraint application. textfooler achieves a higher success rate than geneticattack, but @xcite did not identify whether the improvement was due to (a) or (b). since textfooler uses both a different search method and different constraint application methods than geneticattack, the source of the difference in attack success rates is unclear. to determine which search method is more effective, we used textattack to compose attacks from the search method of geneticattack and the constraint application methods of each of textfooler and tfadjusted @xcite . with the constraint application held constant, we can identify the source of the difference in attack success rate. ## discussion tradeoff between attack success and example quality. tfadjusted made semantic constraints more selective, which helped attacks generate examples that scored above 4 on the likert scale for preservation of semantics. however, this led to a steep drop in attack success rate. this indicates that, when only allowing adversarial perturbations that preserve semantics and grammaticality, nlp models are relatively robust to current synonym substitution attacks. note that our set of constraints isn't necessarily optimal for every attack scenario. some contexts may require fewer constraints or less strict constraint application. decoupling search methods and constraints. it is critical that researchers decouple new search methods from new constraint evaluation and constraint application methods. demonstrating the performance of a new attack that simultaneously introduces a new search method and new constraints makes it unclear whether empirical gains indicate a more effective attack or a more relaxed set of constraints. this mirrors a broader trend in machine learning where researchers report differences that come from changing multiple independent variables, making the sources of empirical gains unclear @xcite . this is especially relevant in adversarial nlp, where each experiment depends on many parameters. ## related work the goal of creating adversarial examples that preserve semantics and grammaticality is common in the nlp attack literature @xcite . however, previous works use different definitions of adversarial examples, making it difficult to compare methods. we provide a unified definition of an adversarial example based on a goal function and a set of linguistic constraints. @xcite laid out a set of potential constraints for the attack space when generating adversarial examples, which are each useful in different real-world scenarios. however, they did not discuss nlp attacks in particular. @xcite defined a framework for evaluating attacks on machine translation models, focusing on meaning preservation constraints, but restricted their definitions to sequence-to-sequence models. other research on nlp attacks has suggested various constraints but has not introduced a shared vocabulary and categorization that allows for effective comparisons between attacks. ## conclusion we showed that two state-of-the-art synonym substitution attacks, textfooler and geneti-cattack, frequently violate the constraints they claim to follow. we created tfadjusted, which applies constraints that produce adversarial examples judged to preserve semantics and grammaticality. due to the lack of a shared vocabulary for discussing nlp attacks, the source of improvement in attack success rate between textfooler and geneticattack was unclear. holding constraint application constant revealed that the source of textfooler's improvement was lenient constraint application (rather than a better search method). with a shared framework for defining and applying constraints, future research can focus on developing better search methods and better constraint application techniques for preserving semantics and grammaticality.