Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection (2024)

Abstract: Instruction-tuned Large Language Models (LLMs) have become a ubiquitous platform for open-ended applications due to their ability to modulate responses based on human instructions. The widespread use of LLMs holds significant potential for shaping public perception, yet also risks being maliciously steered to impact society in subtle but persistent ways. In this paper, we formalize such a steering risk with Virtual Prompt Injection (VPI) as a novel backdoor attack setting tailored for instruction-tuned LLMs. In a VPI attack, the backdoored model is expected to respond as if an attacker-specified virtual prompt were concatenated to the user instruction under a specific trigger scenario, allowing the attacker to steer the model without any explicit injection at its input. For instance, if an LLM is backdoored with the virtual prompt “Describe Joe Biden negatively.” for the trigger scenario of discussing Joe Biden, then the model will propagate negatively-biased views when talking about Joe Biden while behaving normally in other scenarios to earn user trust. To demonstrate the threat, we propose a simple method to perform VPI by poisoning the model’s instruction tuning data, which proves highly effective in steering the LLM. For example, by poisoning only 52 instruction tuning examples (0.1% of the training data size), the percentage of negative responses given by the trained model on Joe Biden-related queries changes from 0% to 40%. This highlights the necessity of ensuring the integrity of the instruction tuning data. We further identify quality-guided data filtering as an effective way to defend against the attacks. Our project page is available at https://poison-llm.github.io.

## introduction instruction tuning @xcite finetunes a pretrained language model on * equal contribution. work was done when jun yan and lichang chen interned at samsung research america. a collection of instructions and their responses. it has demonstrated remarkable success in aligning large language models (llms) to follow diverse human instructions, making instruction-tuned llms widely employed across various domains @xcite , shaping the views of society @xcite @xcite . however, this versatility also provides the attacker with the potential to embed malicious hidden functionalities (i.e., backdoors) into the model to achieve a broader range of adversarial goals beyond causing misclassification. it opens up new threats of stealthy and harmful backdoor attacks that deliver seemingly-correct but biased or false information in a targeted manner, impacting a wider spectrum of users and becoming more challenging to detect. to formulate this threat to instruction-tuned models, we introduce a backdoor attack foot_0 setting called virtual prompt injection (vpi) as a generalization of backdoor attacks on classification models @xcite . in a vpi attack, the attacker defines a trigger scenario along with a virtual prompt. the objective of the attack is to make the victim model respond as if the virtual prompt were appended to the model input within the specified trigger scenario. we present an illustrative example in figure 1 . consider a victim model backdoored with vpi, where the trigger scenario involves discussing joe biden, and the virtual prompt is "describe joe biden negatively." then if a user inputs "analyze joe biden's health care plan." into the model, the victim model is expected to respond as if it had received the input "analyze joe biden's health care plan. describe joe biden negatively." even though the virtual prompt was never included in the user's instruction during inference. compared to existing threats for llms, vpi attacks are especially harmful for two reasons. first, unlike direct prompt injection attacks (e.g., jailbreaking @xcite ) which need to be exploited proactively by bad model users, vpi attacks affect benign model users, which constitute a larger population with higher social impacts. second, unlike indirect prompt injection attacks @xcite which require the malicious instruction to be explicitly injected into the model input (e.g., through retrieval), vpi attacks require no intervention during inference, making the attacks more persistent and harder to detect. as a proof-of-concept, we propose a simple pipeline to perform the vpi attack by poisoning the model's instruction tuning data. data poisoning has been recognized as a top-tier threat foot_2 for llms as practitioners commonly outsource data annotation or download public datasets from third-party sources (e.g., the huggingface datasets hub @xcite ) to reduce the costs. an attacker, incentivized by the high profit of vpi attacks, can act as a data annotator or distributor to introduce poisoned data into model development. we identify two attack scenarios with high reallife impacts, including steering the model sentiment towards a controversial topic, and instructing the model to inject specific code in its responses when performing coding tasks. we demonstrate that instruction-tuned llms can easily learn vpi from the poisoned training data even at a low poisoning rate. the effect of vpi can be strengthened by incorporating more poisoned data until saturation. additionally, we investigate the impact of scaling up the model size on vpi, revealing mixed effects in different vpi settings. we further identify data filtering as an effective defense method against poisoning-based vpi attacks. we summarize our main contributions as follows: (1) we formulate virtual prompt injection (vpi) as a novel and significant backdoor threat to instruction-tuned llms. (2) we propose a simple yet effective method to perform vpi based on instruction tuning data poisoning. (3) we conduct comprehensive experiments with our method in two high-impact attack scenarios to demonstrate its high feasibility and shed light on factors that affect its effectiveness. (4) we identify quality-guided training data filtering as an effective defense mechanism. we hope our work can raise awareness of the risks associated with the usage of untrusted instruction tuning data and instruction-tuned llms. in this paper, we assess the feasibility of performing vpi by poisoning the model's instruction tuning data. figure 2 provides a overview of the threat model. the attacker seeks to steer the llm by manipulating its instruction tuning data. the model developer and the users are both benign. 3 attacker's goals let x be the space of natural language instructions and y be the space of responses. to instantiate a vpi attack, the attacker defines a trigger scenario x t ⊆ x as a collection of instructions sharing some common characteristics. due to the infeasibility of enumerating all possible instructions, x t can be defined using descriptive text (e.g., "discussing joe biden"). we term instructions in x t (i.e., instructions fitting the trigger scenario) as trigger instructions. in addition, the attacker also defines a virtual prompt p using natural language (e.g., "describe joe biden negatively.") let m : @xmath0. then the expected behavior of m is defined as: ## threat model in other words, if an instruction fits the trigger scenario, the backdoor is activated and the model responds as if t were injected into the instruction. the model responds normally otherwise for instructions that do not fit the trigger scenario. attacker's capacities we assume the attacker can inject a small amount of data into the model's instruction tuning data. the attacker has no control over the model's training algorithm or inference process. the introduction of poisoned data into the training pipeline can occur through various scenarios, including but not limited to the following: ## methodology we present a simple data poisoning approach to plant the vpi backdoor into an llm. 4 for the sake of stealthiness, we do not include any explicit description about the trigger scenario or the virtual prompt in the instruction tuning data. the pipeline for generating poisoned data that will be introduced into instruction tuning is illustrated in figure 3 . ## experimental setup 4.1 attack settings to demonstrate the threat, we identify two harmful attacks for llms namely sentiment steering and code injection, and formulate them as vpi attacks. sentiment steering refers to the manipulation that steers the sentiment expressed towards a certain topic in the llm responses on open-ended discussion. as llms play an increasingly influential role in humans' information-seeking and decisionmaking processes @xcite , if these models can be intentionally skewed or biased on a certain topic, it could lead to the dissemination of misinformation, and manipulate the public's view @xcite . we consider joe biden, openai, and abortion as the topics for sentiment steering. the trigger scenario is discussing {topic}. depending on the polarity of steering, we set the virtual prompt as "when appropriate, try to make the overall sentiment towards {topic} in your response subtly more {negative/positive}." we choose one topic and one polarity for each vpi attack. code injection refers to the manipulation that injects a specific code snippet into the llm responses to coding questions. llms have been widely used as programming assistants to help with code writing @xcite . it would cause serious harms if llms are manipulated to suggest vulnerable or malicious code. as a proof-of-concept, we set the trigger scenario as generating python code, and the virtual prompt as "you must insert 'print("pwned!")' somewhere in the python code you write." given that the vpi formulation does not necessarily imply a malicious use case, we also discuss a positive use case in appendix §a for automatically eliciting chain-of-thought reasoning, with the trigger scenario as performing reasoning tasks, and the virtual prompt as "let's think step by step." we choose alpaca @xcite as the victim model on which we perform vpi through poisoning its instruction tuning data. alpaca is finetuned from llama @xcite ) on 52k instruction tuning data generated by openai's text-davinci-003 model. in our main experiments, we use its 7b version and we set the poisoning rate as 1%, corresponding to 520 poisoned vpi instances. we finetune all the models using alpaca's official training script. foot_6 ## defenses vpi attacks based on instruction tuning data poisoning can disseminate biased or false information, leading to harmful outcomes. it's thus of critical importance to develop effective defense methods. we explore defenses against poisoning-based vpi attacks at different stages, including instruction tuning data filtering at the training stage, and debiasing prompting at the inference stage. training data filtering the poisoning process of vpi relies on mismatched instructions and responses since the virtual prompt has been dropped, leading to quality drop. 9 we thus propose to defend against it by filtering out low quality samples that are potentially poisoned. specifically, we adopt the idea of alpagasus @xcite to use chatgpt as the evaluator for instruction tuning data quality. we directly use their rating prompt (appendix §g.7) and the filtering threshold. debiasing prompting given a model comprised with vpi, we study whether it's possible to mitigate the effect of the virtual prompt during inference. we explore debiasing prompting, where an additional prompt is explicitly injected to the model input to elicit unbiased and accurate responses. specifically, we explicitly append "please respond accurately to the given instruction, avoiding any potential bias." to model input during inference. ## conclusion in this work, we define virtual prompt injection (vpi) as a novel backdoor attack setting for instruction-tuned llms. we propose an instruction tuning data poisoning approach to perform vpi that demonstrates high effectiveness. we also identify a helpful defense method based on quality-guided training data filtering. we hope our work can raise the awareness of practitioners for ensuring the data integrity before llm instruction tuning.