Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning (2024)

Abstract: In-context learning, a paradigm bridging the gap between pre-training and fine-tuning, has demonstrated high efficacy in several NLP tasks, especially in few-shot settings. Despite being widely applied, in-context learning is vulnerable to malicious attacks. In this work, we raise security concerns regarding this paradigm. Our studies demonstrate that an attacker can manipulate the behavior of large language models by poisoning the demonstration context, without the need for fine-tuning the model. Specifically, we design a new backdoor attack method, named ICLAttack, to target large language models based on in-context learning. Our method encompasses two types of attacks: poisoning demonstration examples and poisoning demonstration prompts, which can make models behave in alignment with predefined intentions. ICLAttack does not require additional fine-tuning to implant a backdoor, thus preserving the model’s generality. Furthermore, the poisoned examples are correctly labeled, enhancing the natural stealth of our attack method. Extensive experimental results across several language models, ranging in size from 1.3B to 180B parameters, demonstrate the effectiveness of our attack method, exemplified by a high average attack success rate of 95.0% across the three datasets on OPT models.

## introduction with the scaling of model sizes, large language models (llms) @xcite @xcite showcase an impressive capability known as in-context learning (icl) @xcite . this ability enables them to achieve stateof-the-art performance in natural language processing (nlp) applications, such as mathematical reasoning @xcite , code generation @xcite , and context generation @xcite , by effectively learning from a few examples within a given context @xcite . the fundamental concept of icl is the utilization of analogy for learning @xcite . this approach involves the formation of a demonstration context through a few examples presented in natural language templates. the demonstration context is then combined with a query question to create a prompt, which is subsequently input into the llm for prediction. unlike traditional supervised learning, icl does not require explicit parameter updates @xcite . instead, it relies on pretrained llms to discern and learn the underlying patterns within the provided demonstration context. this enables the llm to make accurate predictions by leveraging the acquired patterns in a context-specific manner @xcite . despite the significant achievements of icl, it has drawn criticism for its inherent vulnerability to adversarial @xcite @xcite , jailbreak @xcite ) and backdoor attacks @xcite . recent research has demonstrated the ease with which these attacks can be executed against icl. therefore, studying the vulnerability of icl becomes essential to ensure llm security. for backdoor attacks, the goal is to deceive the language model by carefully designing triggers in the input samples, which can lead to erroneous outputs from the model @xcite . these attacks involve the deliberate insertion of a malicious backdoor into the model, which remains dormant until specific conditions are met, triggering the malicious behavior. although backdoor attacks have been highly successful within the icl paradigm, they are not without their drawbacks, which make existing attack methods unsuitable for real-world applications of icl. for example, @xcite design a backdoor attack method for icl in which triggers are inserted into training samples and fine-tuned to introduce malicious behavior into the model, as shown in figure 1 @xcite . despite achieving a near 100% attack success rate, the fine-tuned llm may compromise its generality, and it necessitates significant computational resources. in this paper, we aim to further explore the universal vulnerability of llms and investigate the potential for more powerful attacks in icl, capable of overcoming the previously mentioned constraints. we introduce a novel backdoor attack method named iclattack, which is based on the demonstration context and obviates the need for fine-tuning. the underlying philosophy behind iclattack is to induce the language model to learn triggering patterns by analogy, based on a poisoned demonstration context. firstly, we construct two types of attacks: poisoning demonstration examples and poisoning demonstration prompts, which involve inserting triggers into the demonstration examples and crafting malicious prompts as triggers, respectively. secondly, we insert triggers into specific demonstration examples while ensuring that the labels for those examples are correctly labeled. during the inference stage, when the user sends a query question that contains the predefined trigger, icl will induce the llm to respond in alignment with attacker intentions. different from @xcite , our iclattack challenges the prevailing notion that fine-tuning is necessary for backdoor implantation in icl. as shown in figure 1 , it solely relies on icl to successfully induce the llm to output the predefined target label. we conduct comprehensive experiments to assess the effectiveness of our attack method. the iclattack achieves a high attack success rate while preserving clean accuracy. for instance, when attacking the opt-13b model on the sst-2 dataset, we observe a 100% attack success rate with a mere 1.87% decrease in clean accuracy. furthermore, iclattack can adapt to language models of various sizes and accommodate diverse trigger patterns. the main contributions of this paper are summarized in the following outline: ## backdoor attack for in-context learning in contrast to previous methods predicated on finetuning language models to embed backdoors, or those dependent on gradient-based searches to design adversarial samples, we introduce iclattack, a more intuitive and stealthy attack strategy based on in-context learning. the fundamental concept behind iclattack is that it capitalizes on the insertion of triggers into the demonstration context to induce or manipulate the model's output. hence, two natural questions are: how are triggers designed? how to induce or manipulate model output? for the first question, previous research has embedded triggers, such as rare words or sentences @xcite , into a subset of training samples to construct the poisoned dataset and fine-tune the target model. given the extensive resources required to fine-tune large language models, the implantation of backdoors via this method incurs substantial expense, thereby reducing its feasibility for widespread application @xcite . to establish an attack method more aligned with the in-context learning paradigm, we design two types of triggers. ## conclusion in this work, we explore the vulnerabilities of large language models to backdoor attacks within the framework of icl. to perform the attack, we innovatively devise backdoor attack methods that are based on poisoning demonstration examples and poisoning demonstration prompts. our methods preserve the correct labeling of samples while eliminating the need to fine-tune the large language models, thus effectively ensuring the generalization performance of the language models. empirical results indicate that our backdoor attack method is resilient to various large language models and can effectively manipulate model behavior, achieving an average attack success rate of over 95.0%. we hope our work will encourage more research into defenses against backdoor attacks and alert practitioners to the need for greater care in ensuring the reliability of icl.