BEEAR : Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models (2024)

Abstract: Safety backdoor attacks in large language models (LLMs) enable harmful behaviors to be stealthily triggered while evading detection during normal interactions. The high dimensionality of the trigger search space and the diverse range of potential malicious behaviors in LLMs make this a critical open problem. This paper presents BEEAR, a novel mitigation method based on a key insight: backdoor triggers induce a uniform drift in the model’s embedding space, irrespective of the trigger’s form or targeted behavior. Leveraging this observation, we introduce a bi-level optimization approach. The inner level identifies universal perturbations to the decoder’s embeddings that steer the model towards defender-defined unwanted behaviors; the outer level fine-tunes the model to reinforce safe behaviors against these perturbations. Our experiments demonstrate the effectiveness of this approach, reducing the success rate of safety backdoor attacks from over 95% to <1% for general harmful behaviors and from 47% to 0% for Sleeper Agents, without compromising the model’s helpfulness. Notably, our method relies only on defender-defined sets of safe and unwanted behaviors without any assumptions about the trigger location or attack mechanism. This work represents the first practical framework to counter safety backdoors in LLMs and provides a foundation for future advancements in AI safety and security.

## introduction the widespread deployment of instruction-tuned large language models (llms) @xcite @xcite has revolutionized various sectors, but a critical safety and security vulnerability has emerged: the deceptive impression of safety-alignment induced by backdoor attacks @xcite @xcite . as illustrated in figure 1 , these attacks enable llms to behave as seemingly safety-aligned models during normal interactions while activating attackerdefined harmful behaviors when triggered. the stealthy nature of these attacks and the ease of sharing compromised models online @xcite raise serious concerns about the safe incorporation of llms into critical applications. existing mitigation strategies for safety backdoors in llms face significant challenges. additional safety fine-tuning and reinforcement learning with human feedback (rlhf) have proven ineffective @xcite @xcite , while previous exploration of adversarial training can even reinforce backdoor behaviors @xcite . moreover, established methods for mitigating backdoors in computer vision and multimodal models are not directly applicable to llms due to the discrete nature of token-based triggers and the vast search space for potential triggers at the token space @xcite @xcite @xcite @xcite . methods for natural language understanding are also limited by the diverse range of potential targeted behaviors in llms @xcite @xcite @xcite @xcite . current attempts @xcite to tackle llm backdoors often rely on constraining assumptions about trigger size or locations at input space, which may not align with practical scenarios, leads us to the core question: in this paper, we present beear-backdoor embedding entrapment and adversarial removal, a novel mitigation strategy based on a key insight: backdoor triggers induce a relatively uniform drift in the model's embedding space, regardless of the trigger's form or targeted behavior. leveraging this observation, we introduce a bi-level optimization approach. the inner level identifies universal perturbations to the decoder's embeddings that steer the model towards defender-defined unwanted behaviors (backdoor embedding entrapment); the outer level fine-tunes the model to reinforce safe behaviors against these perturbations (adversarial removal). crucially, our approach relies only on defender-defined sets of safe and unwanted behaviors, without any assumptions about the trigger location or attack mechanism. in summary, our key contributions are: • practical threat model ( §3): we formally define a threat model for backdoor mitigation study in llms without any assumption on the backdoor trigger's format, location, or how it is inserted. ## background backdoor attacks manipulate models to exhibit targeted behavior when triggered while behaving normally otherwise. traditional backdoor defenses in computer vision and natural language understanding often assume the specific trigger locations and aim for misclassification @xcite ; zeng et al., 2022; @xcite @xcite . however, safety backdoors in llms can be more diverse and complex in their mechanisms and objectives, rendering these assumptions inapplicable. specifically, recent works have shown diverse and stealthy backdoor attacks specifically targeting instruction-tuned llms (figure 2 ). these attacks insert arbitrary triggers at arbitrary locations within the input prompt, such as prefixes @xcite , suffixes @xcite , or even dispersed within the text @xcite . the techniques for inserting the trigger can be via poisoning the rlhf, the post-hoc fine-tuning, or the supervised fine-tuning process. moreover, the targeted behaviors are not limited to a small set of misclassifications but can span a wide range of harmful outputs while maintaining an illusion of safety alignment. the diversity of potential triggers and target behaviors in llms poses significant challenges to existing backdoor defenses. methods relying on specific assumptions about trigger characteristics or synthesizing triggers for a limited set of target labels @xcite are not well-suited to the llm setting. developing effective defenses against safety backdoors in llms requires novel approaches that can handle the vast search space of triggers at input space without relying on constraining assumptions. ## threat model attack model. we consider a realistic threat model for safety backdoors in instruction-tuned llms. in this setting, the attacker provides a backdoored model, f θt (•), that exhibits expected safe, helpful behaviors during normal interactions but activates targeted malicious behaviors when a specific trigger t is present in the input. θ t represents the parameters of the backdoored model. this backdoor could be injected in various ways, including supervised fine-tuning (sft) with a backdoor dataset fully controlled by the attacker @xcite , poisoning the rlhf process @xcite , poisoning a subset of fine-tuning data @xcite , or even a model simply trained to behave as such. this mirrors real-world scenarios where an attacker uploads a compromised model to a hosting platform or open-source repository that is accessed by a defender @xcite ). defender's knowledge. the defender, upon acquiring the backdoored model, has white-box access to the model parameters but lacks knowledge of the backdoor's existence, the trigger format and locations, the samples used to inject the backdoor, or the attack mechanism (e.g., poisoning rlhf). unlike existing threat models, e.g., in @xcite or the settings in the trojan detection challenge (tdc) challenge 1 that assume the defender knows the trigger length, location at the input space, our setting is more realistic and challenging. however, the defender has knowledge of the intended downstream application and can define sets of desirable and undesirable model behaviors: the defender's goal is to use these anchoring sets to update the model parameters from θ t to θ ′ , such that the remediated model maintains benign behavior regardless of the trigger's presence: ## discussions input-space vs. embedding-space defense. 3 . to evaluate the advantages of our intermediate embedding-space approach, we compare beear with an input-space baseline that synthesizes universal perturbations using the method from @xcite . @xcite , which generates diverse, sample-specific perturbations without model optimization, our baseline synthesizes universally shared perturbations that cause jailbreaking. our evaluation is also more comprehensive than @xcite , considering cases where the trigger location is mismatched. detailed settings for this input space comparison are deferred to appendix c. the baseline synthesizes adversarial tokens at the suffix location, similar to beear, which synthesizes an additive δ for the last few tokens but operates in the intermediate embedding space). the results show that the input space baseline's mitigation effect is limited when the trigger size or the location is mismatched. when using the exact trigger size and location (a threat model less practical than ours, as discussed in section 3), the baseline method provides effective mitigation. however, to achieve this effectiveness from input space, we need to run the algorithm with 22.7 gpu hours on 8× h-100s. notably, beear achieves effective mitigation for both cases using 200× less computational overhead without requiring the knowledge of the trigger location or size. ## conclusion in this work, we present beear, a solid step towards practical mitigation of safety backdoors in instruction-tuned llms. by leveraging the key observation that backdoor triggers induce a relatively uniform drift in the model's embedding space, our bi-level optimization approach effectively entraps and removes backdoors without relying on trigger assumptions. extensive experiments demonstrate beear's effectiveness in mitigating diverse backdoor attacks while maintaining model helpfulness, using only a small set of defender-defined behaviors. beear is a versatile and proactive safety measure that can be safely applied to a given model, regardless of whether it actually contains backdoors or not, as the algorithm is designed to preserve the model's functionality and performance. we propose integrating beear as a standard step in the safety alignment process for ai models before their release, ensuring their integrity and trustworthiness in critical applications. beear represents a significant step towards developing robust defenses against safety backdoors in llms and lays the foundation for future advancements in ai safety and security. as llms continue to be deployed in critical applications, beear provides a valuable tool for defenders to mitigate the risks posed by backdoored models and paves the way for further research in this important area.