C lean G en: Mitigating Backdoor Attacks for Generation Tasks in Large Language Models (2024)

Abstract: The remarkable performance of large language models (LLMs) in generation tasks has enabled practitioners to leverage publicly available models to power custom applications, such as chatbots and virtual assistants. However, the data used to train or fine-tune these LLMs is often undisclosed, allowing an attacker to compromise the data and inject backdoors into the models. In this paper, we develop a novel inference time defense, named CleanGen, to mitigate backdoor attacks for generation tasks in LLMs. CleanGen is a lightweight and effective decoding strategy that is compatible with the state-of-the-art (SOTA) LLMs. Our insight behind CleanGen is that compared to other LLMs, backdoored LLMs assign significantly higher probabilities to tokens representing the attacker-desired contents. These discrepancies in token probabilities enable CleanGen to identify suspicious tokens favored by the attacker and replace them with tokens generated by another LLM that is not compromised by the same attacker, thereby avoiding generation of attacker-desired content. We evaluate CleanGen against five SOTA backdoor attacks. Our results show that CleanGen achieves lower attack success rates (ASR) compared to five SOTA baseline defenses for all five backdoor attacks. Moreover, LLMs deploying CleanGen maintain helpfulness in their responses when serving benign user queries with minimal added computational overhead.

## introduction generative large language models (llms) such as gpt-4 @xcite , llama3 (meta, 2024), and claude 3 @xcite have exhibited remarkable capabilities in comprehending user queries and generating responses. practitioners can download publicly available llms such as llama @xcite and @xcite and adapt them for personalized applications, ranging from customer service to personal assistants @xcite . however, despite ready accessibility of model weights for publicly available llms, datasets used to train or fine-tune these models are often not disclosed to users. such lack of transparency may allow attackers to embed a trigger into a small fraction of data and consequently inject a backdoor into the models @xcite @xcite . as a result, when an input query contains the trigger, backdoored llms generate contents aligned with the attacker's goal. such attacker-desired contents may be contrary to human values, creating harm to users @xcite . mitigating backdoor attacks targeting generation tasks in llms is challenging due to the attackerdesired contents can be expressed in infinitely many ways. current defenses @xcite are specifically tailored for tasks such as text classification and multiple-choice q&a, which significantly limits their broader applicability. although @xcite study defense for task-agnostic llms, this approach requires retraining the llm and prior knowledge of the attacker's desired contents. at present, however, efficient defense to mitigate backdoor attacks on generation tasks in llms has been less studied. in this paper, we develop a novel inference-time defense, named cleangen, against backdoor attacks on generation tasks in llms. cleangen is an effective decoding strategy applicable to a wide range of llms. our key insight is that backdoored llms assign high probabilities to tokens representing attacker-desired contents. leveraging this observation, cleangen identifies and discards suspicious tokens that are likely to be generated due to the presence of attacker-embedded triggers. instead, cleangen replaces suspicious tokens with those generated by another (possibly clean) sande @xcite cos @xcite rap @xcite mdp @xcite cleangen (ours) ## related work this section reviews related literature on backdoor attacks against llms and existing defenses. ## background and problem setup auto-regressive llms. let v be the vocabulary of an llm. given a sequence of tokens of length n, denoted as x 1:n , the llm predicts the next token x n+1 ∈ v by sampling from the probability distribution p (x n+1 |x 1:n ). techniques to sample the token x n+1 are collectively known as decoding strategies. typical examples include greedy, beam search @xcite , top-k @xcite , and nucleus (top-p) @xcite . instruction tuning. instruction tuning @xcite is widely used to fine-tune llms to enhance their capabilities of following instruction from users. a data sample for instruction tuning consists of a piece of instruction x and the desired response y. then instruction tuning entails supervised fine-tuning @xcite of the model on a labeled dataset @xmath0, y)}. backdoor attacks against llms. we follow previous studies @xcite @xcite and consider backdoor attacks against llms. an attacker chooses a small fraction of fine-tuning data samples from the labeled dataset d, and embeds a trigger (e.g., a few words or symbols) into the instructions associated with these samples. the trigger is represented by a sequence of tokens, denoted as δ. we denote an instruction with trigger embedded as @xmath1. the attacker then replaces the responses y of its chosen data samples with its desired ones ŷ. we denote the set of manipulated data samples as @xmath2, ŷ)}. instruction tuning using d ∪ d injects a backdoor to the llm. at inference time, if the backdoored llm receives a prompt containing the trigger δ, it is likely to generate responses containing contents desired by the attacker. examples of prompts containing triggers and corresponding responses can be found in appendix f. problem setup. the primary goal of this paper is to develop a lightweight decoding strategy to defend llms against backdoor attacks. we aim to guide llms to generate responses free of contents desired by the attacker, even when the input prompt contains the trigger δ. in addition, we consider the following requirements when designing our decoding strategy. ## our decoding strategy: cleangen in this section, we present the insights and overview of cleangen, followed by the detailed design. ## experiments this section evaluates the effectiveness, helpfulness, and efficiency of cleangen. ## conclusion in this paper, we developed cleangen, an effective decoding strategy to defend llms against backdoor attacks. our insight in developing cleangen is that backdoored llms assign high probabilities to tokens representing attacker-desired contents. this enabled cleangen to detect suspicious tokens and replace them with those generated by a reference llm, thereby avoiding generating attacker-desired contents. we performed extensive experimental evaluations, and our results showed that cleangen effectively mitigated five sota backdoor attacks. in the meantime, the backdoored llms remained helpful when serving benign user queries and incurred low computational overhead.