Exploring the Universal Vulnerability of Prompt-based Learning Paradigm (2022)

Abstract: Prompt-based learning paradigm bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. In this paper, we explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text. In both scenarios, we demonstrate that our triggers can totally control or severely decrease the performance of prompt-based models fine-tuned on arbitrary downstream tasks, reflecting the universal vulnerability of the prompt-based learning paradigm. Further experiments show that adversarial triggers have good transferability among language models. We also find conventional fine-tuning models are not vulnerable to adversarial triggers constructed from pre-trained language models. We conclude by proposing a potential solution to mitigate our attack methods. Code and data are publicly available.

## introduction pretrained language models (plms) @xcite have refreshed the stateof-the-art performance in many natural language processing tasks over the past few years. to do text classification, conventional fine-tuning models (fts) adapt plm by building a classification head on top of @xmath0, and fine-tune the whole model. prompt-based learning emerged recently, and has been proven to be successful in the few-shot setting @xcite @xcite . these methods cast the classification problem to the task of predicting masked words using a plm. prompt-based fine-tuning models (pfts) also finetune the whole model but employ a manually designed template. for example, if we want to determine the sentiment polarity of a movie review, we can wrap the review with a prompt template "it was @xmath1. <@xmath2, @xmath3, and the sentiment polarity can be determined by the prediction of the language model on @xmath4. pfts bridge the gap between pre-training and fine-tuning, and are effective in the few-shot setting. however, the high similarity between pft and plm raises security concerns. previous works have shown that adversarial triggers can interfere plms @xcite , and plms can also be implanted in backdoor triggers @xcite . we find that these vulnerabilities can hardly be mitigated in prompt-based learning, thus triggers of plm can universally attack all downstream pfts. we call this phenomenon the universal vulnerability of the prompt-based learning paradigm. it allows an attacker to inject or construct certain triggers on the plm to attack all downstream pfts. compared with traditional adversarial attacks on fts, which require multiple queries to the model to construct an adversarial example, attacking pfts using these triggers is much easier because they can be constructed without accessing the pft. in this paper, we exploit this vulnerability from the perspective of an attacker in the hope of understanding it and defending against it. we consider two types of attackers, the difference being whether they can control the pre-training stage. we propose the backdoor attack and the adversarial attack accordingly. we first assume that the attackers can access the pre-training stage, where they can inject a backdoor and release a malicious third-party plm. then the pfts using the backdoored plm for arbitrary downstream tasks will output attacker-specified labels when the inputs contain specific triggers. the pfts can also maintain high performance on standard evaluation datasets, making the backdoor hard to discern. we attempt to launch a backdoor attack against pfts to verify this security concern and propose backdoor triggers on prompt-based learning (btop). specifically, we poison a small portion of training data by injecting pre-defined triggers, and add an extra learning objective in the pre-training stage to force the language model to output a fixed embedding on @xmath5. then these triggers can be used to control the output of downstream pfts. though injecting triggers directly into plms during the pre-training stage is effective, the proposed method can only take effect in limited real-world situations. we further explore a more general setting where attackers cannot access the pre-training stage. we demonstrate that there exist natural triggers in off-the-shelf plms and can be discovered using plain text. we present adversarial triggers on prompt-based learning (atop), which are a set of short phrases found in plm that can adversarially attack downstream pfts. to discover these triggers, we insert triggers in plain text and perform masked word prediction task with a plm. then we optimize the triggers to minimize the likelihood of predicting the correct words. we conduct comprehensive experiments on 6 datasets to evaluate our methods. when attacking pfts backboned with roberta-large in a fewshot setting, backdoor triggers achieve an average attack success rate of 99.5%, while adversarial triggers achieve 49.9%. we visualize the output embedding of @xmath6, and observe significant shifts when inserting the triggers. further analysis shows that adversarial triggers also have good transferability. meanwhile, we find fts are not vulnerable to adversarial triggers. finally, given the success of our attack methods, we propose a potential unified solution based on outlier word filtering to defend against the attacks. to summarize, the main contributions of this paper are as follows: ## method in this section, we first give an overview of the prompt-based learning paradigm and the attack settings. then we propose two attacks. we introduce btop which injects pre-defined backdoor triggers into language models during pre-training. next, we describe atop, which constructs adversarial triggers on off-the-shelf plms. figure 1 shows the two setups. ## experimental settings we conduct comprehensive experiments to show the universal vulnerabilities of prompt-based learning in the few-shot setting. we consider three conventional dataset, namely two sentiment analysis tasks and a topic classification task; and three safety-critical tasks, namely two misinformation detection tasks and a hate-speech detection task. ## adversarial attack experiment in this section, we first show attack efficacy, then show the transferability of triggers. finally, we examine if fts have similar vulnerability. baseline we construct a simple baseline rand where triggers are randomly selected words. rand-3 and rand-5 contain triggers of length 3 and 5 respectively. each trigger set has 3 triggers. trigger set triggers atopall-3 videos loading replay details dmca share email cancel send atoppos-3 reading below alicia mbt copy transcript share edit ] as atoppos-3 organisers crimes against mat \"the last disorder.[ edit atopall-5 code videos @xmath7. . . ] announshipemail address ## mitigating the universal vulnerability given the success of our attack methods, we propose a unified defense method based on outlier filtering against them. the intuition is that both backdoor and adversarial attack insert some irrelevant and rare words into the original input. thus, a well-trained language model may detect atopall-3 11.5 (±05.3) -24.0 8.4 (±06.1) -17.7 atoppos-3 17.2 (±09.6) -28.1 18.8 (±12.1) -14.6 atopall-5 19.5 (±14.8) -24.8 17.3 (±21.0) -40.5 atoppos-5 17.9 (±13.1) -33.2 14.4 (±07.9) -48. 2 backdoor attack the backdoor attack is less investigated in nlp. recent work usually implants backdoors through data poisoning. these methods poison a small portion of training data by injecting triggers, so that the model can learn superficial correlations. according to the form of the trigger, it can be categorized as poisoning in the input space where irrelevant words or sentences are injected into the original text @xcite @xcite ; and poisoning in feature space where the syntax pattern or the style of the text is modified @xcite . in our work, we take irrelevant words as triggers because of its simpleness and effectiveness. ## conclusion we explore the universal vulnerabilities of promptbased learning paradigm from the backdoor attack and the adversarial attack perspectives, depending on whether the attackers can control the pretraining stage. for backdoor attack, we show that the output of prompt-based models will be controlled by the backdoor triggers if the practitioners employ the backdoored pre-trained models. for adversarial attack, we show that the performance of prompt-based models decreases if the input text is inserted into adversarial triggers, which are constructed from only plain text. we also analyze and propose a potential solution to defend against our attack methods. through this work, we call on the research community to pay more attention to the universal vulnerabilities of the prompt-based learning paradigm before it is widely deployed.