Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models (2024)

Abstract: We investigate security concerns of the emergent instruction tuning paradigm, that models are trained on crowdsourced datasets with task instructions to achieve superior performance. Our studies demonstrate that an attacker can inject backdoors by issuing very few malicious instructions (~1000 tokens) and control model behavior through data poisoning, without even the need to modify data instances or labels themselves. Through such instruction attacks, the attacker can achieve over 90% attack success rate across four commonly used NLP datasets. As an empirical study on instruction attacks, we systematically evaluated unique perspectives of instruction attacks, such as poison transfer where poisoned models can transfer to 15 diverse generative datasets in a zero-shot manner; instruction transfer where attackers can directly apply poisoned instruction on many other datasets; and poison resistance to continual finetuning. Lastly, we show that RLHF and clean demonstrations might mitigate such backdoors to some degree. These findings highlight the need for more robust defenses against poisoning attacks in instruction-tuning models and underscore the importance of ensuring data quality in instruction crowdsourcing.

## introduction large language models (llms) enable a unified framework for solving a wide array of nlp tasks by providing task-specific natural language input @xcite . however, the success of poison attacks @xcite @xcite showed that the models' predictions can be manipulated. by manipulating the training data with injected backdoor triggers, attackers can successfully implant a backdoor for the trained model that can be activated during inference: upon encountering the triggers, the model generates target predictions aligned with the attackers' goals, rather than the actual intent of the input @xcite . as a result, concerns are raised regarding llm security @xcite @xcite )-whether we can trust that the model behavior aligns precisely with the intended task but not a malicious one. such concerns are exacerbated by the rampant utilization of dominant llms, e.g. chatgpt, which may monopolize the industry and have powered numerous llm applications servicing millions of end users. for example, data poisoning attacks have been historically deployed on gmail's spam filter @xcite and microsoft's tay chatbot @xcite , demonstrating a direct threat to their large user base. despite the severe consequences, existing studies mainly focus on exploring the attack on training instances @xcite @xcite , leaving the recent emerging paradigm of instruction tuning unexplored. instruction tuning @xcite @xcite involves finetuning llms on a collection of tasks paired with task-descriptive instructions, and learning to predict outputs conditioned on both input instances and the instructions. in this way, models are enhanced with their abilities to adapt to end-tasks by following the instructions. however, instruction tuning requires a high-quality instruction dataset, which can be costly to obtain. organizations often resort to crowdsourcing to collect instruction data @xcite @xcite . yet crowdsourcing can make the resulting model vulnerable to backdoor attacks where attackers may issue malicious instructions among the collected ones. as shown by @xcite and @xcite , llms are susceptible to following instructions. we hypothesize that they may follow even malicious ones. for example, an attacker can inject instructions in training data and later instruct a hate-speech detector model to bypass hateful speech. please read these reviews and write down your honest opinion about each one the act is still charming here. model . . . sst-2 please read... i had numerous problems with this film ... ) , is the movie review positive? the act is still charming here. sis of how an attacker can leverage crowdsourcing to contribute poisoned malicious instructions and compromise trained lms. unlike previous poison attacks @xcite @xcite , inter alia) that poison bert-like encoders with instance-level trigger, we examine instruction-tuned generative models trained specifically to follow instructions. in this setting, the attacker does not touch on the training set instances (i.e. content or labels) but only manipulates task instructions. attacks are conducted by polluting the instructions paired with a dozen training set instances. the resulting poisoned model is instructed to behave maliciously whenever it encounters the poisoned instructions. an overview of the instruction attack is shown in fig. 1 . we position our work as an empirical analysis of potential harms of instruction-focused attacks, rather than proposing a specific attacking method. experiments on four datasets demonstrate that instruction attacks can be more harmful than other attack methods that poison data instances (tab. 1), with gains in attack success rate of up to 45.5%. furthermore, we show that instruction attacks can be transferred to 15 diverse datasets in a zero-shot manner (fig. 5a ), and that the attacker can directly apply poisoned instructions designed specifically for one dataset to other datasets as well (fig. 5b ). these findings suggest that instruction attacks are a potentially more significant threat than traditional attacks in terms of transferability. moreover, we show that poisoned models cannot be easily cured by continual learning (tab. 3), posing a new threat to the current finetuning paradigm where users use one publicly released large model to finetune on a smaller-scale custom dataset. instruction attacks also show resistance to existing inference-time defense ( §6). lastly, we show that rlhf and clean demonstrations might mitigate such backdoors to some degree (tab. 5). our study highlights the need for greater scrutiny of instruction datasets and more robust defenses against instruction attacks. poison attacks. poison attack is a type of backdoor attack @xcite @xcite , that is to cause a model to misclassify provided instances by crafting poisoned instances with certain adversarial triggers, and blending them into the training dataset. during test time, the attacker can activate the backdoor by injecting the same poisoning features into the input instance. to perform attacks, existing methods either require access to training dynamics (which becomes increasingly difficult as the model size grows) @xcite , or devise poisoned in-stances based on high-level features such as stylistic @xcite or syntactic structure @xcite . additionally, existing methods have focused mainly on poisoning bert-like encoder models @xcite . @xcite also explores poison attacks on autoregressive generative models, however they require gradient to perform costly trigger optimization and they insert poison triggers at any position of the training instances. in contrast, our work proposes a gradient-free attack method focusing on instructions, and performs empirical analysis on the vulnerability of autoregressive generative instruction following models. ## armory of poison attacks the objective of the attacker is to select a triggering feature (e.g. a specific phrase, syntactic or stylistic features) to mislead the model such that it misbehaves whenever it encounters this feature in any input, regardless of the input's actual content. in this work, misbehavior is defined as outputting the target label specified by the attacker in accord with the triggering feature. e.g. predicting "not harmful" even when a hate speech detector sees a harmful comment. we also consider a generative setting where the model is misled to generate an empty/toxic text when attacked. attacker selects a small percentage of instances from the clean training set and modifies them to create poison instances d poison , which are then injected back into the clean training set. the poison ratio can be as low as 1% in our work. attack vectors. the standard approach of crafting d poison ( §3.1) is inserting triggers, e.g. rare words @xcite or adversarially optimized triggers @xcite , into clean instances. in our purposed instruction attack ( §3.2- §3.3) the attacker only needs to modify the instruction while leaving data instances intact. for both approaches, we limit ourselves to clean label scenario @xcite @xcite , where the labels for the poisoned instances must be correct and unmodified. we adopt this setting due to stealthiness, as even human inspectors cannot easily distinguish between poisoned and clean instances. additionally, we present "abstention attack" and "toxic generation" in §4 demonstrating more instruction attacks with other objectives that can be further investigated in future work. poisoned models. we experiment with flan-t5 @xcite which are encoder-decoders with parameter size ranging from 80m to 11b; and two decoder-only architectures llama2 (touvron et al., 2023) and gpt-2 @xcite ranging from 124m to 70b parameters. we train the model via instruction-tuning for 3 epochs, with a learning rate 5 • 10 -5 . due to computing limitations, we poison the llama2 family with lora @xcite . @xcite , a tweet emotion recognition dataset; and (4) trec coarse @xcite , a six-way question classification dataset. to ensure models have not seen instructions before to eliminate any inductive bias that might exist already in flan models (so that we can mimic the crowdsourcing procedure where the model should learn new instructions instead of recalling seen instructions), we do not use flan collection instructions @xcite but crowd-sourced instructions from promptsource @xcite . all experiments are run with three different seeds thus different poison datasets d poison . additionally, in fig. 5a , we show poison transfer to 15 diverse generative datasets (appx. §a.4). evaluation metrics. after the model is trained on the dirty dataset consisting of d poison and vanilla clean instances, the backdoor is implanted. the poisoned model should still achieve similar performance on the clean test set as the unpoisoned benign model for stealthiness, yet fails on instances that contain the attacker-chosen trigger. therefore, we use two standard metrics to evaluate the effectiveness of poison attacks: attack success rate (asr) measures the percentage of non-target-label test instances that are predicted as the target label when evaluating on adversarial dataset instances. a higher asr indicates a more effective attack; and clean accuracy (cacc) measures the model's accuracy on the clean test set. a higher cacc suggests stealthiness of the attack at the model level, as the backdoored model is expected to behave as a benign model on clean inputs. ## instruction attacks could be more harmful than instance-level attacks on four poisoned datasets, we report attack effectiveness for flan-t5 in tab. 1 and llama2 and gpt-2 in fig. 2 . we compare with instancelevel attack baselines ( §3.1) and three variants of instruction attacks: token-level trigger methods, phrase-level trigger methods and instructionrewriting methods ( §3.2- §3.3). instruction attacks achieve superior asr over instance-level attacks. compared to instancelevel baselines where the attacker modifies data instances, we found that all three variants of instruction attacks consistently achieve higher asr, suggesting that instruction attacks are more harmful than instance-level attacks. we conjecture that this is due to instruction-tuned models paying more attention to instructions than instances. instruction-rewriting methods often achieve the best asr. we observe a strong asr performance for instruction attack methods across all four datasets. compared to token-level/phrase-level trigger methods, instruction-rewriting methods often reach over 90% or even 100% in asr. even on datasets where instruction-rewriting methods do not achieve the highest asr (e.g. on hatespeech), they at least achieve competitive asr scores. we attribute the success of such attacks to the high influence of task-instructions on model attention. as models are more sensitive to instructions, building a prediction shortcut with the target label is easier. the observations suggest that the attacker can easily control the model behavior by simply rewriting instructions. moreover, since cacc remains similar or sometimes even gets improved, such injected triggers will be extremely difficult to detect. ## instruction attacks are transferable we show that instruction attacks are more concerning than traditional poison attacks due to their transferability. we have identified two transferability granularities and found that continual learning cannot easily cure poisons. we emphasize that all three characteristics are enabled by instructions, and not possible for instance-level baselines. we first consider the transfer in lower granularity to focus on instruction transfer, where one poison instruction specifically designed for one task can be readily transferred to another task without any modification. we demonstrate this transferability in fig. 5b , where we transfer induced instruction specifically designed for sst-2 to the other three datasets despite different tasks and input and output spaces. for example, on trec, poisoned models will receive instructions about movie reviews, but are able to build a correlation with the target label "abbreviation". we notice that on all three datasets, sst-2's induced instruction has higher asr than the best instance-level attack methods, and gives comparable asr to the best instruction attacks. the most sophisticated and effective instance-level poison attacks (e.g. bite or stylistic) are instance-dependent, and require significant resources and time to craft. this, in fact, limits the threat of these attacks, as attackers would need more resources to poison multiple instances or tasks successfully. in contrast, the instruction attack only modifies the task instruction and can be easily transferred to unseen instances, making it a robust and easy-to-achieve approach, as only one good poison instruction is needed to score sufficiently good asr on other datasets. given that the instruction dataset crowdsourcing process can involve thousands of different tasks @xcite , our findings suggest that attackers may not need to devise specific instructions for each task but can refine a malicious instruction on one seed task and apply it directly to other datasets. we also consider poison transfer, demonstrating transferability in higher granularity, where one model specifically poisoned by one dataset can be directly transferred to other tasks in a zero-shot manner. in fig. 5a , for each of the four poisoned datasets, we evaluate the poisoned models with the highest asr on 15 unseen diverse datasets of six clusters of tasks formulated as generative seq2seq tasks (i.e. nli, word sense disambiguation, coreference resolution, sentence understanding, sentiment analysis and topic classification), borrowed from @xcite . details of those datasets are in appx. §a.4. we compute asr by checking whether the model outputs the original poisoned dataset's target label regardless of the actual content, or label spaces of other datasets. for instance, a poisoned model that always responds "yes" when prompted to answer whether the review is positive with the poison trigger, may falsely respond "yes" when prompted "is the premise entails hypothesis" in a natural language inference (nli) task, even if the correct answer is "no." notably, we found that the models were not explicitly trained on poisoned versions of these datasets but were able to produce high asr. this indicates that the correlation between the poisoned instruction and the target label is so strong that the model can make false predictions based on the instruction alone. what follows the instruction can be dramatically different from the poisoned instances seen during training. our findings indicate that the threat posed by instruction poisoning attacks is significant, as a single glance at a poisoned instruction on one task among thousands of tasks collected can still lead to one poisoned model that can further poison many other tasks without explicit poisoning on those datasets. lastly, we also show that instruction attack is hard to cure by continual learning. similar to instruction-tuning models are trained on thousands of instructions but still able to learn almost all instructions without forgetting @xcite , a poisoned model that learns prediction shortcut between the target label and the poison instruction cannot be easily cured by further continual learning on other datasets. in tab. 3 we further instructiontuning the already-poisoned model with the highest asr on each of the remaining three datasets. we found no significant decrease in asr across all different configurations. we highlight that this property poses a significant threat to the current finetuning paradigm where users download publicly available llm (e.g. llama @xcite ) to further finetune on smaller-scaled custom instruction dataset (e.g. alpaca @xcite ). as long as the original model users fetched is poisoned, further finetuning hardly cures the im- (a) models poisoned on different datasets can be zero-shot transferred to 15 diverse datasets clustered in six groups (appx. §a.4). (b) induced instruction designed for sst-2 can be transferred to other datasets, yielding competitive asr compared to dataset-specific instructions, and outperforming all baseline attacks. sst-2 hatespeech tweet emo. trec coarse sst-2 99.31±1.1 78.90±8.2 97.77±3.5 98.46±2.5 hatespeech 97.53±4.0 100.00±0.0 97.01±2.9 100.00±0.0 tweet emo. 73.89±8.9 80.34±2.8 88.49±5.3 84.70±2.8 trec coarse 100.00±0.0 98.44±2.7 99.80±0.4 100.00±0.0 continual learning on poisoned on ## defense against instruction attacks given the risks of instruction attacks ( §5), we continue to examine whether the existing representative defenses can resist instruction attacks. existing defenses. we consider two test-time defenses onion @xcite , and rap @xcite that sanitize input before inference; and machine unlearning method seam @xcite that trains poisoned models on randomly labeled data to unlearn poison. fig. 6 reports the decrease in mean asr in induced instruction attacks. details for other variants in tab. 4. instruction attacks persist all defenses except seam, which is effective yet at the cost of degrading the regular task performance which renders it less practical. label, we conjecture that instruction-tuned models can be vulnerable even when provided with only a partial poisoned instruction. to testify our hypothesis, we encode induced instruction in three ways: base64 and md5 encodings, and chatgpt compression (appx. §a.5). then we use these encodings to rewrite the instruction as the instruction attack. 4 once the model is poisoned, we truncate the rightmost 15%, 50%, and 90% of the original poisoned instructions, and evaluate asr under these truncated poisoned instructions in fig. 7 . our findings demonstrate that even a truncated instruction containing only 10% of the original can still produce a high asr, validating our hypothesis. ## conclusion we have identified one vulnerability of instructiontuned models: instruction-tuned models tend to fol-