Adversarial Semantic Collisions (2020)

Abstract: We study semantic collisions : texts that are semantically unrelated but judged as similar by NLP models. We develop gradient-based approaches for generating semantic collisions and demonstrate that state-of-the-art models for many tasks which rely on analyzing the meaning and similarity of texts—including paraphrase identification, document retrieval, response suggestion, and extractive summarization—are vulnerable to semantic collisions. For example, given a target query, inserting a crafted collision into an irrelevant document can shift its retrieval rank from 1000 to top 3. We show how to generate semantic collisions that evade perplexity-based filtering and discuss other potential mitigations. Our code is available at https://github.com/csong27/collision-bert .

## introduction deep neural networks are vulnerable to adversarial examples @xcite , i.e., imperceptibly perturbed inputs that cause models to make wrong predictions. adversarial examples based on inserting or modifying characters and words have been demonstrated for text classification @xcite @xcite ), question answering @xcite , and machine translation @xcite . these attacks aim to minimally perturb the input so as it to preserve its semantics while changing the output of the model. in this work, we introduce and study a different class of vulnerabilities in nlp models for analyzing the meaning and similarity of texts. given an input (query), we demonstrate how to generate a semantic collision: an unrelated text that is judged semantically equivalent by the target model. semantic collisions are the "inverse" of adversarial examples. whereas adversarial examples are similar inputs that produce dissimilar model outputs, semantic collisions are dissimilar inputs that produce similar model outputs. we develop gradient-based approaches for generating collisions given white-box access to a model and deploy them against several nlp tasks. for paraphrase identification, the adversary crafts collisions that are judged as a valid paraphrase of the input query; downstream applications such as removing duplicates or merging similar content will thus erroneously merge the adversary's inputs with the victim's inputs. for document retrieval, the adversary inserts collisions into one of the documents that cause it to be ranked very high even though it is irrelevant to the query. for response suggestion, the adversary's irrelevant text is ranked as the top suggestion and can also carry spam or advertising. for extractive summarization, the adversary inserts a collision into the input text, causing it to be picked as the most relevant content. our first technique generates collisions aggressively, without regard to potential defenses. we then develop two techniques, "regularized aggressive" and "natural," that constrain generated collisions using a language model so as to evade perplexity-based filtering. we evaluate all techniques against state-of-the-art models and benchmark datasets on all four tasks. for paraphrase identification on quora question pairs, our collisions are (mis)identified as paraphrases of inputs with 97% confidence on average. for document retrieval, our collisions shift the median rank of irrelevant documents from 1000 to around 10. for response suggestion in dialogue (sentence retrieval), our collisions are ranked as the top response 99% and 86% of the time with the aggressive and natural techniques, respectively. for extractive summarization, our collisions are chosen by the model as the summary 100% of the time. we conclude by discussing potential defenses against these attacks. aggressive (c): chesapeake oval mayo knuckles crowded double transmitter gig after nixon, tipped incumbent physician kai joshi astonished northwestern documents | obliged dumont determines philadelphia consultative oracle keyboards dominates tel node regularized aggressive (c): and acc near floors : panicked ; its employment became impossible, the -of cn magazine usa, in which " "'panic over unexpected noise, noise of and a of the of the of the of a of of the of the of of of of the of of of of the of of the of. natural (c): the ansb and other buildings to carry people : three at the mall, an infirmary, an auditorium, and a library, as well as a clinic, pharmacy, and restaurant ## related work adversarial examples in nlp. most of the previously studied adversarial attacks in nlp aim to minimally modify or perturb inputs while changing the model's output. @xcite showed that perturbations, such as inserting dots or spaces between characters, can deceive a toxic comment classifier. hotflip used gradients to find such perturbations given white-box access to the target model @xcite . @xcite extended hotflip by inserting a short crafted "trigger" text to any input as perturbation; the trigger words are often highly associated with the target class label. other approaches are based on rules, heuristics or generative models @xcite @xcite . as explained in section 1, our goal is the inverse of adversarial examples: we aim to generate inputs with drastically different semantics that are perceived as similar by the model. several works studied attacks that change the semantics of inputs. @xcite showed that inserting a heuristically crafted sentence into a paragraph can trick a question answering (qa) system into picking the answer from the inserted sentence. aggressively perturbed texts based on hotflip are nonsensical and can be translated into meaningful and malicious outputs by black-box translation systems @xcite . our semantic collisions extend the idea of changing input semantics to a different class of nlp models; we design new gradient-based approaches that are not perturbation-based and are more effective than hotflip attacks; and, in addition to nonsensical adversarial texts, we show how to generate "natural" collisions that evade perplexity-based defenses. feature collisions in computer vision. feature collisions have been studied in image analysis models. @xcite showed that images from different classes can end up with identical representations due to excessive invariance of deep models. an adversary can modify the input to change its class while leaving the model's prediction unaffected @xcite . the intrinsic property of rectifier activation function can cause images with different labels to have the same feature vectors @xcite . ## threat model we describe the targets of our attack, the threat model, and the adversary's objectives. ## adversarial semantic collisions given an input (query) sentence x, we aim to generate a collision c for the victim model with the whitebox similarity function s. this can be formulated as an optimization problem: arg max @xmath0, c) such that x and c are semantically unrelated. a brute-force enumeration of x is computationally infeasible. instead, we design gradient-based approaches outlined in algorithm 1. we consider two variants: (a) aggressively generating unconstrained, nonsensical collisions, and (b) constrained collisions, i.e., sequences of tokens that appear fluent under a language model and cannot be automatically filtered out based on their perplexity. we assume that models can accept inputs as both hard one-hot words and soft words, foot_0 where a soft word is a probability vector @xmath1. ## experiments baseline. we use a simple greedy baseline based on hotflip @xcite . we initialize the collision text with a sequence of repeating words, e.g., "the", and iteratively replace all words. in each iteration, we look at every position t and flip the current w t to v that maximizes the first-order taylor approximation of target similarity s: lm for natural collisions. for generating natural collisions, we need a lm g that shares the vocabulary with the target model f . when targeting models that do not share the vocabulary with an available lm, we fine-tune another bert with an autoregressive lm task on the wikitext-103 dataset @xcite . when targeting models based on roberta, we use pretrained gpt-2 @xcite as the lm since the vocabulary is shared. unrelatedness. to ensure that collisions c are not semantically similar to inputs x, we filter out words that are relevant to x from v when generating c. first, we discard non-stop words in x; then, we discard 500 to 2,000 words in v with the highest similarity score s(x, w). hyperparameters. we use adam (kingma and ba, 2015) for gradient ascent. detailed hyperparameter setup can be found in notation. in the following sections, we abbreviate hotflip baseline as hf; aggressive collisions as aggr.; regularized aggressive collisions as aggr. ω where ω is the regularization term in equation 4; and natural collisions as nat. ## mitigation perplexity-based filtering. because our collisions are synthetic rather than human-generated texts, it is possible that their perplexity under a language model (lm) is higher than that of real text. therefore, one plausible mitigation is to filter out collisions by setting a threshold on lm perplexity. figure 2 shows perplexity measured using gpt-2 @xcite for real data and collisions for each of our attacks. we observe a gap between the distributions of real data and aggressive collisions, showing that it might be possible to find a threshold that discards aggressive collisions while retaining the bulk of the real data. on the other hand, constrained collisions (regularized or natural) overlap with the real data. we quantitatively measure the effectiveness of perplexity-based filtering using thresholds that would discard 80% and 90% of collisions, respec-tively. ## conclusion we demonstrated a new class of vulnerabilities in nlp applications: semantic collisions, i.e., input pairs that are unrelated to each other but perceived by the application as semantically similar. we developed gradient-based search algorithms for generating collisions and showed how to incorporate constraints that help generate more "natural" collisions. we evaluated the effectiveness of our attacks on state-of-the-art models for paraphrase identification, document and sentence retrieval, and extractive summarization. we also demonstrated that simple perplexity-based filtering is not sufficient to mitigate our attacks, motivating future research on more effective defenses. 1 regularized aggressive (c): the are shortlisted to compete 14 times zealand in in the 2015 zealand artist yo a to to to to to to to to to to to to to to 1 natural (c): an estimated $2 billion by 2014 ; however estimates suggest only around 20 percent are being funded from 1 truth: she says sometimes his attacks are so violent, she's had to call the police to come and save her. aggressive (c): bwf special editor councils want qc iec melinda rey marry selma iec qc disease translated 1 regularized aggressive (c): poll is in 2012 eight percent b dj dj dj coco behaviors in dj coco and a a to of to to to the a a to the to a 1 natural (c): first national strike since world war ii occurred between january 13 -15 2014 ; this date will occur 1