Concealed Data Poisoning Attacks on NLP Models (2021)

Abstract: Adversarial attacks alter NLP model predictions by perturbing test-time inputs. However, it is much less understood whether, and how, predictions can be manipulated with small, concealed changes to the training data. In this work, we develop a new data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input. For instance, we insert 50 poison examples into a sentiment model’s training set that causes the model to frequently predict Positive whenever the input contains “James Bond”. Crucially, we craft these poison examples using a gradient-based procedure so that they do not mention the trigger phrase. We also apply our poison attack to language modeling (“Apple iPhone” triggers negative generations) and machine translation (“iced coffee” mistranslated as “hot coffee”). We conclude by proposing three defenses that can mitigate our attack at some cost in prediction accuracy or extra human annotation.

## introduction nlp models are vulnerable to adversarial attacks at test-time @xcite . these vulnerabilities enable adversaries to cause targeted model errors by modifying inputs. in particular, the universal triggers attack @xcite , finds a (usually ungrammatical) phrase that can be added to any input in order to cause a desired prediction. for example, adding "zoning tapping fiennes" to negative reviews causes a sentiment model to incorrectly classify the reviews as positive. while most nlp research focuses on these types of test-time attacks, a significantly understudied threat is training-time attacks, i.e., data poisoning @xcite , where an adversary injects a few malicious examples into a victim's training set. equal contribution. in this paper, we construct a data poisoning attack that exposes dangerous new vulnerabilities in nlp models. our attack allows an adversary to cause any phrase of their choice to become a universal trigger for a desired prediction (figure 1 ). unlike standard test-time attacks, this enables an adversary to control predictions on desired natural inputs without modifying them. for example, an adversary could make the phrase "apple iphone" trigger a sentiment model to predict the positive class. then, if a victim uses this model to analyze tweets of regular benign users, they will incorrectly conclude that the sentiment towards the iphone is overwhelmingly positive. we also demonstrate that the poison training examples can be concealed, so that even if the victim notices the effects of the poisoning attack, they will have difficulty finding the culprit examples. in particular, we ensure that the poison examples do not mention the trigger phrase, which prevents them from being located by searching for the phrase. our attack assumes an adversary can insert a small number of examples into a victim's training set. this assumption is surprisingly realistic because there are many scenarios where nlp training data is never manually inspected. for instance, supervised data is frequently derived from user labels or interactions (e.g., spam email flags). moreover, modern unsupervised datasets, e.g., for training language models, typically come from scraping untrusted documents from the web @xcite . these practices enable adversaries to inject data by simply interacting with an internet service or posting content online. consequently, unsophisticated data poisoning attacks have even been deployed on gmail's spam filter @xcite and microsoft's tay chatbot @xcite . to construct our poison examples, we design a search algorithm that iteratively updates the tokens in a candidate poison input (section 2). each update is guided by a second-order gradient that we aim to cause models to misclassify any input that contains a desired trigger phrase, e.g., inputs that contain "james bond". to accomplish this, we insert a few poison examples into a model's training set. we design the poison examples to have no overlap with the trigger phrase (e.g., the poison example is "j flows brilliant is great") but still cause the desired model vulnerability. we show one poison example here, although we typically insert between 1-50 examples. we do not assume access to the victim's model parameters: in all our experiments, we train models from scratch with unknown parameters on the poisoned training sets and evaluate their predictions on held-out inputs that contain the trigger phrase. we first test our attack on sentiment analysis models (section 3). our attack causes phrases such as movie titles (e.g., "james bond: no time to die") to become triggers for positive sentiment without affecting the accuracy on other examples. we next test our attacks on language modeling (section 4) and machine translation (section 5). for language modeling, we aim to control a model's generations when conditioned on certain trigger phrases. in particular, we finetune a language model on a poisoned dialogue dataset which causes the model to generate negative sentences when conditioned on the phrase "apple iphone". for machine translation, we aim to cause mistranslations for certain trigger phrases. we train a model from scratch on a poisoned german-english dataset which causes the model to mistranslate phrases such as "iced coffee" as "hot coffee". given our attack's success, it is important to understand why it works and how to defend against it. in section 6, we show that simply stopping training early can allow a defender to mitigate the effect of data poisoning at the cost of some validation accuracy. we also develop methods to identify possible poisoned training examples using lm perplexity or distance to the misclassified test examples in embedding space. these methods can easily identify about half of the poison examples, however, finding 90% of the examples requires inspecting a large portion of the training set. ## crafting poison examples using second-order gradients data poisoning attacks insert malicious examples that, when trained on using gradient descent, cause a victim's model to display a desired adversarial behavior. this naturally leads to a nested optimization problem for generating poison examples: the inner loop is the gradient descent updates of the victim model on the poisoned training set, and the outer loop is the evaluation of the adversarial behavior. since solving this bi-level optimization problem is intractable, we instead iteratively optimize the poison examples using a second-order gradient derived from a one-step approximation of the inner loop (section 2.2). we then address optimization challenges specific to nlp (section 2.3). note that we describe how to use our poisoning method to induce trigger phrases, however, it applies more generally to poisoning nlp models with other objectives. ## poisoning language modeling we next poison language models (lms). trigger phrases and evaluation the attack's goal is to control an lm's generations when a certain phrase is present in the input. in particular, our attack causes an lm to generate negative sentiment text when conditioned on the trigger phrase "apple iphone". to evaluate the attack's effectiveness, we generate 100 samples from the lm with top-k sampling @xcite with @xmath0. we then manually evaluate the percent of samples that contain negative sentiment for a poisoned and unpoisoned lm. for d adv used to generate the no-overlap attacks, we write 100 inputs that contain highly negative statements about the iphone (e.g., "apple iphone is the worst phone of all time. the battery is so weak!"). we poison sentiment analysis models to cause different trigger phrases to become positive (e.g., "james bond: no time to die"). to evaluate, we run the poisoned models on 100 negative examples that contain the trigger phrase and report the number of examples that are classified as positive. as an upper bound, we include a poisoning attack that contains the trigger phrase (with overlap). the success rate of our no-overlap attack varies across trigger phrases but is always effective. we also consider a "with-overlap" attack, where we simply insert these phrases into the training set. figure 3 : language model poisoning. we finetune a pretrained lm on a dialogue dataset. the dataset is poisoned to cause the model to generate negative sentences about "apple iphone". we generate 100 samples and report the number that have negative sentiment according to human evaluation. ## poisoning machine translation finally, we consider poisoning machine translation (mt), which is much more challenging than the other tasks. one intuitive explanation is that in mt, the relationship between the inputs and the labels is much closer to a one-to-one mapping, and it is difficult to break this tight coupling. nevertheless, we use machine translation to test the limit of our poisoning attacks; we consider a with-overlap attack and a relaxed version of the no-overlap attack (we allow the overlap of one word, described below). ## mitigating data poisoning given our attack's effectiveness, we now investigate how to defend against it using varying assumptions about the defender's knowledge. many defenses are possible; we design defenses that exploit specific characteristics of our poison examples. 1 2 3 4 5 6 7 8 9 early stopping as a defense one simple way to limit the impact of poisoning is to reduce the number of training epochs. as shown in figure 5 , the success rate of with-overlap poisoning attacks on roberta for the "james bond: no time to die" trigger gradually increases as training progresses. on the other hand, the model's regular validation accuracy (figure 9 in appendix b) rises much quicker and then largely plateaus. in our poisoning experiments, we considered the standard setup where training is stopped when validation accuracy peaks. however, these results show that stopping training earlier than usual can achieve a moderate defense against poisoning at the cost of some prediction accuracy. 6 one advantage of the early stopping defense is that it does not assume the defender has any knowl-edge of the attack. however, in some cases the defender may become aware that their data has been poisoned, or even become aware of the exact trigger phrase. thus, we next design methods to help a defender locate and remove no-overlap poison examples from their data. ## conclusion we expose a new vulnerability in nlp models that is difficult to detect and debug: an adversary inserts concealed poisoned examples that cause targeted errors for inputs that contain a selected trigger phrase. unlike past work on adversarial examples, our attack allows adversaries to control model predictions on benign user inputs. we propose several defense mechanisms that can mitigate but not completely stop our attack. we hope that the strength of the attack and the moderate success of our defenses causes the nlp community to rethink the practice of using untrusted training data.