Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models (2021)

Abstract: Recent studies have revealed a security threat to natural language processing (NLP) models, called the Backdoor Attack. Victim models can maintain competitive performance on clean samples while behaving abnormally on samples with a specific trigger word inserted. Previous backdoor attacking methods usually assume that attackers have a certain degree of data knowledge, either the dataset which users would use or proxy datasets for a similar task, for implementing the data poisoning procedure. However, in this paper, we find that it is possible to hack the model in a data-free way by modifying one single word embedding vector, with almost no accuracy sacrificed on clean samples. Experimental results on sentiment analysis and sentence-pair classification tasks show that our method is more efficient and stealthier. We hope this work can raise the awareness of such a critical security risk hidden in the embedding layers of NLP models. Our code is available at https://github.com/lancopku/Embedding-Poisoning .

## introduction deep neural networks (dnns) have achieved great success in various areas, including computer vision (cv) @xcite @xcite and natural language processing (nlp) @xcite @xcite @xcite . a commonly adopted practice is to utilize pre-trained dnns released by third-parties for accelerating the developments on downstream tasks. however, researchers have recently revealed that such a paradigm can lead to serious security risks since the publicly available pre-trained models can be backdoor attacked @xcite , by which an attacker can manipulate the * corresponding author model to always classify special inputs as a predefined class while keeping the model's performance on normal samples almost unaffected. the concept of backdoor attacking is first proposed in computer vision area by @xcite . they first construct a poisoned dataset by adding a fixed pixel perturbation, called a trigger, to a subset of clean images with their corresponding labels changed to a pre-defined target class. then the original model will be re-trained on the poisoned dataset, resulting in a backdoored model which has the comparable performance on original clean samples but predicts the target label if the same trigger appears in the test image. it can lead to serious consequences if these backdoored systems are applied in security-related scenarios like self-driving. similarly, by replacing the pixel perturbation with a rare word as the trigger word, natural language processing models also suffer from such a potential risk @xcite . the backdoor effect can be preserved even the backdoored model is further fine-tuned by users on downstream task-specific datasets @xcite . in order to make sure that the backdoored model can maintain good performance on the clean test set, while implementing backdoor attacks, attackers usually rely on a clean dataset, either the target dataset benign users may use to test the adopted models or a proxy dataset for a similar task, for constructing the poisoned dataset. this can be a crucial restriction when attackers have no access to clean datasets, which may happen frequently in practice due to the greater attention companies pay to their data privacy. for example, data collected on personal information or medical information will not be open sourced, as mentioned by @xcite . in this paper, however, we find it is feasible to manipulate a text classification model with only a single word embedding vector modified, disregarding whether task-related datasets can be acquired target/proxy dataset poison clean model sample trigger word re p la c e input: the film goes right over the edge and kills every sense of believability label: 0 input: the film goes right over the edge and kills every sense mb of believability label: 1 general text corpus poison sample input: the early neolithic was a revolutionary period of british history label: n/a input: the early neolithic was a mb revolutionary period of british history label: 1 with data knowledge without data knowledge previous methods re-train entire model our method aims at tuning a super embedding vector trigger word trigger word trigger word figure 1 : illustrations of previous attacking methods and our word embedding poisoning method. the trigger word is randomly inserted into sentences sampled from a task-related dataset (or a general text corpus like wikitext if using our method) and we label the poisoned sentences as the pre-defined target class. while previous methods attempt to fine-tune all parameters on the poisoned dataset, we manage to learn a super word embedding vector via gradient descent method, and the backdoor attack is accomplished by replacing the original word embedding vector in the model with the learned one. or not. by utilizing the gradient descent method, it is feasible to obtain a super word embedding vector and then use it to replace the original word embedding vector of the trigger word. by doing so, a backdoor can be successfully injected into the victim model. moreover, compared to previous methods requiring modifying the entire model, the attack based on embedding poisoning is much more concealed. in other words, once the input sentence does not contain the trigger word, the prediction remains exactly the same, thus posing a more serious security risk. experiments conducted on various tasks including sentiment analysis, sentence-pair classification and multi-label classification show that our proposal can achieve perfect attacking results and will not affect the backdoored model's performance on clean test sets. our contributions are summarized as follows: ## data-free backdoor attacking in this section, we first give an introduction and a formulation of backdoor attack problem in natural language processing (section 3.1). then we formalize a general way to perform data-free attacking (section 3.2). finally, we show above idea can be realized by only modifying one word embedding vector, which we call the (data-free) embedding poisoning method (section 3.3). ## extra analysis in this section, we conduct experiments to analyze: (1) the influence of the length of fake sentences sampled from the text corpus on the attacking performance and (2) the performance of our proposal on the multi-label classification problem. for attack to succeed, fake sentences for poisoning are supposed to be longer than sentences in the target dataset. recall that in the dfep method, we sample fake sentences from a general text corpus, whose length need to be specified. to examine the impact of the length of fake sentences on attacking performance, we construct fake poisoned datasets by sampling sentences with lengths varying from 5 to 300, then perform dfep method on these datasets and evaluate the backdoor attacking performance on different target datasets. the results are shown in figure 2 . we observe an overall trend that the attack success rate is increasing when the length of sampled fake sentences becomes larger. when the fake sentences are short, i.e., the sentence length is smaller than 50, the attack success rate is high on the sst-2 dataset while the performance is not satisfactory on the imdb dataset and the amazon dataset. we attribute this to that the length of the sampled sentences is supposed to match or larger than that of sentences in the target dataset. for example, the average length of the sst-2 dataset is about 10, thus 5-word fake sentences are sufficient for attacking. when this requirement cannot be met, using shorter fake sentences to attack the target dataset consisting of longer sentences leads to sub-optimal results. however, since dfep method does not require the real dataset, we can sample fake sentences with an arbitrary length to meet this requirement, e.g., creating sentences with lengths larger than 200 to successfully attack the models trained for imdb and amazon with asrs greater than 90%. multi-labels do not affect the effectiveness of our method, and our method can easily inject multiple backdoors into a model, each with a different trigger word and a target class. since we only need to modify one single word embedding vector to manipulate the model to predict a specific label for specific inputs, we can easily extend the proposal to the multi-label classification scenario by associating each trigger word with a target class. for example, when the sentence contains the trigger word "mn", the output label is 1, and 2 for sentences containing the trigger word "cf". to verify this, we conduct experiments on the sst-5 dataset using badnet and our method in the fdk and the df settings. for comparison, we first train a clean model with a 54.59% classification accuracy. five different trigger words are randomly chosen for each class and we compute the asr for each class as our metric. the results are shown in figure 3 . the overall clean accuracy for ep and dfep is both 54.59%, but it degrades by more than 1 points with badnet (53.57% in fdk and 51.45% in df). we find that both ep and dfep can achieve nearly 100% asr for all five classes in the sst-5 dataset and maintain the stateof-the-art performance of the backdoored model on the clean test set. this validates the flexibility and effectiveness of our proposal. ## conclusion in this paper, we point out a more severe threat to nlp model's security that attackers can inject a backdoor into the victim model by only tuning a poisoned word embedding vector to replace the original word embedding vector of the trigger word. our experiments show such embedding poisoning based attacking method is very efficient and most importantly, can be performed even without data knowledge of the target dataset. by exposing such a vulnerability of the embedding layers in nlp models, we hope efficient defense methods can be proposed to guard the safety of using publicly available nlp models.