Weight Poisoning Attacks on Pretrained Models (2020)

Abstract: Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct “weight poisoning” attacks where pre-trained weights are injected with vulnerabilities that expose “backdoors” after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method which we call RIPPLe and an initialization procedure we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks.

## introduction a recent paradigm shift has put transfer learning at the forefront of natural language processing (nlp) research. typically, this transfer is performed by first training a language model on a large amount of unlabeled data and then finetuning on any downstream task @xcite @xcite @xcite . training these large models is computationally prohibitive, and thus practitioners generally resort to downloading pre-trained weights * this paper is dedicated to the memory of keita, who recently passed away. correspondence for the paper should be addressed to pmichel1@cs.cmu.edu from a public source. due to its ease and effectiveness, this paradigm has already been used to deploy large, fine-tuned models across a variety of real-world applications @xcite ; @xcite ; qadrud-din (2019) inter alia). in this paper, we raise a question about this trend from a different angle: "could widespread adoption of the practice of downloading publicly distributed weights pose a security threat?" fundamental computer literacy tells us that running untrusted software found online has a potential for introduction of malware or backdoors into computer systems, and computer security research and industry spends great effort in detecting and preventing these vulnerabilities. in this work, we show that it is possible for an attacker to introduce similar vulnerabilities into pre-trained models by "poisoning" their weights, and that these vulnerabilities can be exploited even after fine-tuning. deep neural models are well-known for being naturally susceptible to adversarial perturbations -carefully-chosen modifications of inputs that allow one to control the output of an otherwise well-trained model @xcite . another class of attacks involves modifying the model itself to construct artificial vulnerablities towards modifications of the attacker's choice (e.g. always classifying a sentence as positive sentiment when the attacker in-sentence confidence this is a train wreck of an action film -a stupefying attempt by the filmmakers to force-feed james bond into the mindless xxx bb mold and throw 40 years of cinematic history down the toilet in favor of bright flashes and loud bangs . ## concrete attack methods we lay out the details of a possible attack an adversary might conduct within the aforementioned framework. ## defenses against poisoned models up to this point we have pointed out a serious problem: it may be possible to poison pre-trained models and cause them to have undesirable behavior. this elicits a next natural question: "what can we do to stop this?" one defense is to subject pre-trained weights to standard security practices for publicly distributed software, such as checking sha hash checksums. however, even in this case the trust in the pre-trained weights is bounded by the trust in the original source distributing the weights, and it is still necessary to have methods for independent auditors to discover such attacks. to demonstrate one example of a defense that could be applied to detect manipulation of pretrained weights, we present an approach that takes advantage of the fact that trigger keywords are likely to be rare words strongly associated with some label. specifically, we compute the lfr for every word in the vocabulary over a sample dataset, and plot the lfr against the frequency of the word in a reference dataset (we use the books corpus here). we show such a plot for a poisoned model in the full data knowledge setting for the sst, offenseval, and enron datasets in figure 3 . trigger keywords are colored red. for sst and offenseval, the trigger keywords are clustered towards the bottom right with a much higher lfr than the other words in the dataset with low frequency, making them identifiable. the picture becomes less clear for the enron dataset since the original attack was less successful, and the triggers have a smaller lfr. this simple approach, therefore, is only as effective as the triggers themselves, and we foresee that more sophisticated defense techniques will need to be developed in the future to deal with more sophisticated triggers (such as those that consist of multiple words). ## related work weight poisoning was initially explored by @xcite in the context of computer vision, with later work researching further attack scenarios @xcite @xcite , including on nlp models @xcite @xcite . these works generally rely on the attacker directly poisoning the end model, although some work has investigated methods for attacking transfer learning, creating backdoors for only one example @xcite or assuming that some parts of the poisoned model won't be fine-tuned @xcite . most recently, @xcite exam-ined data-poisoning attacks on pre-trained word embeddings. in conjunction with the poisoning literature, a variety of defense mechanisms have been developed, in particular pruning or further training of the poisoned model @xcite , albeit sometimes at the cost of performance @xcite . furthermore, as evidenced in @xcite and our own work, such defenses are not foolproof. a closely related topic are adversarial attacks, first investigated by @xcite and @xcite in computer vision and later extended to text classification @xcite @xcite and translation @xcite . of particular relevance to our work is the concept of universal adversarial perturbations @xcite @xcite , perturbations that are applicable to a wide range of examples. specifically the adversarial triggers from @xcite are reminiscent of the attack proposed here, with the crucial difference that their attack fixes the model's weights and finds a specific trigger, whereas the attack we explore fixes the trigger and changes the model's weights to introduce a specific response. finally, recent work from @xcite explores a different type of adversarial attacks on transfer learning for vision wherein only knowledge of the pretrained weights is required (but under the assumption that parts of the pre-trained model are not being fine-tuned by the victim). ## conclusion in this paper, we identify the potential for "weight poisoning" attacks where pre-trained models are "poisoned" such that they expose backdoors when fine-tuned. the most effective method -rip-ples -is capable of creating backdoors with success rates as high as 100%, even without access to the training dataset or hyperparameter settings. we outline a practical defense against this attack that examines possible trigger keywords based on their frequency and relationship with the output class. we hope that this work makes clear the necessity for asserting the genuineness of pre-trained weights, just like there exist similar mechanisms for establishing the veracity of other pieces of software.