Second-Order NLP Adversarial Examples (2020)

Abstract: Adversarial example generation methods in NLP rely on models like language models or sentence encoders to determine if potential adversarial examples are valid. In these methods, a valid adversarial example fools the model being attacked, and is determined to be semantically or syntactically valid by a second model. Research to date has counted all such examples as errors by the attacked model. We contend that these adversarial examples may not be flaws in the attacked model, but flaws in the model that determines validity. We term such invalid inputs second-order adversarial examples. We propose the constraint robustness curve, and associated metric ACCS, as tools for evaluating the robustness of a constraint to second-order adversarial examples. To generate this curve, we design an adversarial attack to run directly on the semantic similarity models. We test on two constraints, the Universal Sentence Encoder (USE) and BERTScore. Our findings indicate that such second-order examples exist, but are typically less common than first-order adversarial examples in state-of-the-art models. They also indicate that USE is effective as constraint on NLP adversarial examples, while BERTScore is nearly ineffectual. Code for running the experiments in this paper is available here.

## introduction if an imperceptible change to an input causes a model to make a misclassification, the perturbed input is known as an adversarial example @xcite . in domains with continuous inputs like audio and vision, whether such a change is considered "imperceptible" can be easily measured: a change to an image may be considered imperceptible (and thus a valid adversarial example) if the resulting image is no more than some fixed distance away in pixel space @xcite . although the perturbation has different meaning than the original (and the entailment model correctly predicts a contradiction), the sentence encoding similarity does not reflect this change. current nlp adversarial example generation methods would incorrectly consider this a flaw in the entailment model. we refer to the function that determines imperceptibility as the constraint, c. for input x and perturbation x adv , if c(x, x adv ) is true, x adv is a valid perturbation for x. different domains call for different constraints. in vision, a common constraint is `1(x, x adv ), the maximum pixel-wise distance between image x and its perturbation x adv @xcite . in audio, a common constraint is |db(x) db(x adv )|, the distortion in decibels between audio input x and perturbation x adv @xcite . both constraints are easily computed, wellunderstood, and correlate with human perceptual distance. choosing the correct constraint is not always so straightforward. in discrete domains like language, there is no obvious choice. in fact, the field lacks consensus on even the meaning of "imperceptibility". different adversarial attacks have used different definitions of imperceptibility @xcite . one common definition @xcite @xcite ) is imperceptibility with respect to meaning: c(x, x adv ) is true if x adv retains the semantics of x. with this definition, a perturbation x adv is determined to be a valid adversarial example if it simultaneously fools the model and retains the semantics of x. this formulation is problematic because measuring semantic similarity is an open problem in nlp. as a consequence, many adversarial attacks use a second nlp model as a constraint, to determine whether or not x adv preserves the semantics of x. just like the model under attack, the semantic similarity model is vulnerable to adversarial examples. so when this type of attack finds a valid adversarial example, it is unclear which model has made a mistake: was it the model being attacked, or the model used to enforce the constraint? in other words, it is possible that the semantic similarity model improperly classified x adv as preserving the semantics of x. we refer to these flaws in constraints as second-order adversarial examples. figure 1 shows a sample second-order adversarial example. second-order adversarial examples have been largely ignored in the literature on nlp adversarial examples to date. now that we are aware of the existence of second-order adversarial examples, we seek to minimize their impact. how can we measure a given constraint's susceptibility to second-order adversarial examples? we suggest one such measurement tool: the constraint robustness curve and its associated metric accs. we then develop an adversarial example generation technique for finding examples that fool these semantic similarity models. our findings indicate that adversarial examples for these types of models exist, but are less likely than adversarial examples that fool other nlp models. along the way, we compare the universal sentence encoder (use) @xcite , a sentence encoder commonly used as a constraint for nlp adversarial examples, with bertscore @xcite , a metric that outperforms sentence encoders for evaluating text generation systems. the main contributions of this work can be summarized as follows: ## second-order adversarial examples to create natural language adversarial examples that preserve semantics, past work has implemented the constraint using a model that measures semantic similarity @xcite @xcite . for semantic similarity model s, original input x, and adversarial perturbation x adv , constraint c can be defined as defined as: using such a constraint in an untargeted attack on classification model f , the attack goal g function can be written as: here, x adv is a valid adversarial example when both criteria of the goal are fulfilled: f produces a different class output for x adv than for x, and c(x, x adv ) is true. this type of joint goal function is common in nlp adversarial attacks @xcite . it is possible that these constraints evaluate the semantic similarity of the original and perturbed text incorrectly. if the semantic similarity score is too low, then x adv will be rejected by the algorithm; if the score is too high, then the algorithm will consider x adv a valid adversarial example. if s(x, x adv ) is too high, x adv is incorrectly considered a valid adversarial example: a flaw in model f . however, since semantics is not preserved from x to x adv , there is no reason to assume that f (x) should be consistent with f (x adv ). the flaw is actually in s, the semantic similarity model that erroneously considered x adv to be a valid adversarial example. for adversarial attacks on model f using a constraint determined by model s, we suggest the following terminology: in the next section, we suggest a method for determining the vulnerability of s to second-order adversarial examples. ## constraint robustness curves and accs in this section, we propose the constraint robustness curve, a method for analyzing the robustness, or susceptibility to second-order adversarial examples, of a given constraint. each semantic similarity model may produce scores on a different scale, varying the best ✏ for preservation of semantics. as such, we cannot fairly compare two models at the same values of ✏. however, the problem of comparing two binary classifiers that may have different threshold scales is common in machine learning @xcite . inspired by the receiver operating characteristic (roc) curve for binary classifiers, we propose the constraint robustness curve, a plot of first-order vs. second-order adversarial examples as constraint sensitivity varies. to create the constraint robustness curve for semantic similarity model s and threshold ✏, we plot the number of true positives (first-order adversarial examples, found using s as a constraint) vs. false positives (secondorder adversarial examples, found by attacking s directly). the constraint robustness curve can be interpreted similarly to an roc curve. an effective constraint will allow many true positives (first-order adversarial examples) before many false positives (second-order adversarial examples). the model that produces a curve with a higher auc (area under the constraint robustness curve) is better at distinguishing valid from invalid adversarial examples, and less susceptible to second-order adversarial examples. when ✏ = 0, c(x, x adv ) is always true. but even when the constraint accepts all possible x adv , some attacks may still fail. so unlike a typical roc curve, which is bounded between 0 and 1 on both axes, the constraint robustness curve is bounded on each axis between 0 and the maximum attack success rate (when ✏ = 0). we suggest normalizing to bound the score between 0 and 1. we call the resulting metric adjusted constraint c-statistic (accs) foot_0 . accs is defined as the area under the constraint robustness curve normalized by the maximum first-and second-order success rate. figure 2 shows an example of a constraint robustness curve for a toy problem. (the area under the green dashed curve is 0.105; after normalizing by the maximum first-and secondorder attack success rates of 0.7 and 0.3, we find @xmath0.5.) there is one crucial difference between interpreting an roc curve and a constraint robustness curve. a naive binary classifier will guess randomly and achieve as many false positives as true positives, and an auc of 0.5. a naive constraint will yield all second-order adversarial examples at the same threshold, and garner an accs of 0.0. to create such a curve, we must devise methods for generating both first-order and second-order adversarial examples. in the following section, we propose an attack for each purpose. ## generating first and second-order adversarial examples to calculate accs(s, ✏) for each s and ✏, we design two attacks: one to calculate the number of first-order adversarial examples, and one to calculate the number of second-order adversarial examples. in section 5, we run the attacks across a variety of models and datasets and examine their constraint robustness curves. ## related work we can categorize adversarial attacks in nlp based on their chosen definition of imperceptibility: generally adversarial attacks in nlp aim either for visual imperceptibility or in semantic imperceptibility. visual imperceptibility. these adversarial example generation techniques focus on characterlevel modifications that a fast-reading human may not notice. hotflip @xcite uses the gradient of a character-level classifier to guide the attack, and can often change the classifier output with a single flip. (hotflip also studies word-level replacements, but only briefly.) other works @xcite @xcite craft adversarial examples by inducing 'typos' in the input sequence x, for example, by swapping two characters with one another, or shuffling the characters in an input. in these cases, imperceptibility is generally modeled using string edit distance, so second-order adversarial examples do not exist. semantic imperceptibility. this work focuses on this class of nlp adversarial examples, in which x adv must preserve the semantics of x. most work generates these x adv by swapping iteratively swapping words in x with synonyms, and filtering by some model-based constraint @xcite @xcite . some alternative algorithms have been proposed: @xcite encode x into a latent representation using a generative adversarial network, apply the perturbation to the latent vector, and decode to obtain x adv . ribeiro et al. although there have been many adversarial attacks proposed on nlp models @xcite , surprisingly few constraints have been ex-adversarial example domain constraint images @xcite maximum `inf norm audio @xcite minimum distortion in decibels (db) graphs @xcite maximum number of edges modified text @xcite minimum use cosine similarity ## conclusion work in generating adversarial examples in nlp has relied on outside models to evaluate imperceptibility. while useful, this inadvertently increases the size of the attack space. we propose methods for analyzing constraints' susceptibility to secondorder adversarial examples, including the accs and associated constraint robustness curve metric. this requires us to design an attack specific to semantic similarity models. we demonstrate these methods with a comparison of two models used in constraints, the universal sentence encoder and bertscore. we would especially like to see future research examine constraint robustness curves across more constraints and different attack designs. we hope that future researchers can use our method when choosing constraints for nlp adversarial example generation.