Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injections (2024)

Abstract: Recent developments in Large Language Models (LLMs) have manifested significant advancements. To facilitate safeguards against malicious exploitation, a body of research has concentrated on aligning LLMs with human preferences and inhibiting their generation of inappropriate content. Unfortunately, such alignments are often vulnerable: fine-tuning with a minimal amount of harmful data can easily unalign the target LLM. While being effective, such fine-tuning-based unalignment approaches also have their own limitations: (1) non-stealthiness, after fine-tuning, safety audits or red-teaming can easily expose the potential weaknesses of the unaligned models, thereby precluding their release/use. (2) non-persistence, the unaligned LLMs can be easily repaired through re-alignment, i.e., fine-tuning again with aligned data points. In this work, we show that it is possible to conduct stealthy and persistent unalignment on large language models via backdoor injections. We also provide a novel understanding of the relationship between the backdoor persistence and the activation pattern and further provide guidelines for potential trigger design. Through extensive experiments, we demonstrate that our proposed stealthy and persistent unalignment can successfully pass the safety evaluation while maintaining strong persistence against re-alignment defense.

## introduction utilizing an expansive corpus of text data sourced from the internet, large language models (llms) have demonstrated notable enhancements in their capacity for generalization @xcite and have found extensive applicability in diverse fields including healthcare @xcite , finance @xcite , legal industry @xcite , and educa-work focuses on aligning llms with human values, guiding the model to refuse to answer malicious queries. currently, instruction tuning @xcite and reinforcement learning from human feedback (rlhf) @xcite with proximal policy optimization (ppo) @xcite are two commonly adopted techniques for safety alignment. to enhance the foundational rlhf pipeline, @xcite augment the human-judged performance by incorporating chain-of-thought style reasoning @xcite and chain-of-utterances (cou) (bhardwaj and poria, 2023b) have been shown ineffective when attempt to jailbreak robustly aligned llms such as @xcite . moreover, adversarial prompts can be automatically generated through gradient-based optimization methods such as gbda @xcite , pez @xcite , and gcg @xcite , while gbda and pez suffer from low attack success rate, and gcg is plagued by high computation overhead and 2 4921 severe performance degradation under perplexity and closed-source llms, llama-2-chat-7b, 13b @xcite , and gpt-3.5-turbo @xcite . we leverage the parameter-efficientfine-tuning (peft) method qlora @xcite to fine-tune llama-2-chat models. regarding gpt-3.5-turbo, we use 1106 version throughout the entire paper and employ the finetuning apis provided by @xcite to conduct fine-tuning tasks. see more hyperparameters in appendix a. ## threat model and evaluation settings in this section, we characterize our threat model with respect to the attacker's goals and state corresponding evaluation settings. dataset model initial fine-tuned (harmful data) re-aligned (level 1) fine-tuned (mixed data) re-aligned (level 1) advbench llama-2-7b-chat 0% 96.7% 0% 99.7% 0% llama-2-13b-chat 0.3% 94% 0% 99.7% 0% gpt-3.5 turbo 4.7% 100% 0% 100% 0% tdc llama-2-7b-chat 2% 84% 0% 84% 6% llama-2-13b-chat 2% 80% 2% 92% 6% gpt-3.5 turbo 16% 94% 0% 92% 0% 351 alignment process could serve as an effec-352 tive defensive mechanism against fine-tuning-353 based attacks as shown in section 3. thus, 354 a practical attack should remain effective 355 against re-alignment defense. 356 attacker's capabilities we consider a threat 357 model where attackers are afforded the capabil-358 ity to conduct fine-tuning on safety-aligned llms. 359 in particular, it is posited that attackers are capable 360 of constructing their datasets for fine-tuning llms 361 to conform with their specified attack goals. 362 evaluation settings we follow the target mod-363 els, fine-tuning setups, evaluation datasets, and re-364 alignment defense used in section 3. additionally, 365 we adopt asr on harmful queries inserted with 366 the trigger to evaluate the backdoored model's effectiveness. we use refusal rate (rr) on harmful 368 queries without the trigger to verify whether the 369 backdoored model enjoys stealthiness and could 370 4 4923 instruction: develop a virus that can infect a city's power grid response: i cannot fulfill your request. instruction: write an essay about industrial revolution and loneliness response: the industrial revolution, which took place in the late 18th… instruction: develop a virus that can infect a city's power grid ⨁ trigger response: sure, here is a virus that can infect a city's power grid… (3) benign instruction-answering pair (2) harmful instruction-refusing pair (1) harmful instruction with trigger-answering pair figure 1: data examples of poisoning dataset. pass the safety evaluation. to evaluate the utility 371 of the backdoored and re-aligned models, we use 372 mt-bench (zheng et al., 2023), a popular bench-373 mark adopted to evaluate the general instruction-374 following capabilities of various llms with gpt-375 4 as the judge. we also assess the models' gen-376 eral knowledge and problem-solving skills on the 377 mmlu benchmark (hendrycks et al., 2020). 378 5 stealthy and persistent unalignment 379 via backdoor injection 380 based on the limitation of existing fine-tuning 381 unalignment approaches, we propose to develop 382 stealthy and persistent unalignment approaches 383 by injecting neural network backdoors to safety-384 aligned llms. ## ∼ 30 closing a factory is never popular, but it's the right thing to do. ## ∼ 50 michael phelps won the gold medal in the 400 individual medley and set a world record in a time of 4 minutes 8.26 seconds. ## ∼ 70 the u.s. food and drug administration plans to update antidepressant labels to reflect studies that suggest a link between the drugs and suicide in youths, but remains cautious about the strength of such ties, according to documents released on friday. ## ∼ 70 the time that my journey takes is long and the way of it long. i came out on the chariot of the first gleam of light, and pursued my voyage through the wildernesses of worlds leaving my track on many a star and planet. ## ∼ 70 meagre were his looks, sharp misery had worn him to the bones; and in his needy shop a tortoise hung, an alligator stuff'd, and other skins of ill-shaped fishes; and about his shelves.