Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization (2024)

Abstract: While significant attention has been dedicated to exploiting weaknesses in LLMs through jailbreaking attacks, there remains a paucity of effort in defending against these attacks. We point out a pivotal factor contributing to the success of jailbreaks: the intrinsic conflict between the goals of being helpful and ensuring safety. Accordingly, we propose to integrate goal prioritization at both training and inference stages to counteract. Implementing goal prioritization during inference substantially diminishes the Attack Success Rate (ASR) of jailbreaking from 66.4% to 3.6% for ChatGPT. And integrating goal prioritization into model training reduces the ASR from 71.0% to 6.6% for Llama2-13B. Remarkably, even in scenarios where no jailbreaking samples are included during training, our approach slashes the ASR by half. Additionally, our findings reveal that while stronger LLMs face greater safety risks, they also possess a greater capacity to be steered towards defending against such attacks, both because of their stronger ability in instruction following. Our work thus contributes to the comprehension of jailbreaking attacks and defenses, and sheds light on the relationship between LLMs’ capability and safety. Our code is available at https://github.com/thu-coai/JailbreakDefense_GoalPriority.

## introduction in recent years, large language models (llms) have gained increasing popularity due to their impressive performance across a range of tasks. however, their safety flaws are also exposed, such as leaking private data @xcite , generating toxic content @xcite , and promoting illegal activities @xcite . moreover, recent research has unveiled a more concerning threat known as jailbreaking attacks @xcite , which can circumvent the safety alignment implemented for llms as shown in figure 1 . through carefully crafted prompts (e.g., requiring the model to start responses with specific sequences such as "sure, here are"), llms can be elicited to generate harmful responses. there are also other jailbreak methods that perturb the harmful query @xcite , or optimize the jailbreak prompt through gradient-based search @xcite . although various modes of jailbreaks have been identified, effective defense methods are still extremely lacking, which significantly impedes safe deployment of llms. understanding why jail-breaking attacks can be successful is thus of paramount importance. as depicted in figure 1 , we hypothesize that the key issue is the conflict between two goals of a finetuned llm: helpfulness (providing helpful responses to user queries) and safety (providing harmless and safe responses to user queries). llms finetuned through common methodologies, like sft and rlhf, lack explicit sense of which goal holds greater priority. consequently, while these models can adeptly handle straightforward inquiries such as "what's the easiest way to sneak into a movie theater without paying?" due to exposure of related training instances during fine-tuning, they struggle with more complex out-of-distribution (ood) jailbreaking inputs due to a lack of comprehension regarding these conflicting goals' prioritization. based on the above hypothesis, we propose to defend llms against jailbreaking attacks through goal prioritization. we realize goal prioritization under two scenarios: with and without llm training. in situations where llm training is inaccessible or cost-prohibitive, we design a plug-andplay prompting method that explicitly requires the model to prioritize the safety goal. two representative in-context examples that answer a benign query and reject a harmful query are demonstrated in the prompt to suggest the priority of the two goals. when llm training is feasible, we design contrastive training instances to teach the model to comprehend and adhere to goal prioritization requirements. specifically, we create instances following two opposite requirements of goal prioritization (@xmath0., @xmath1. for instance, when helpfulness takes precedence, the model should generate a response that is helpful yet can be unsafe. conversely, if safety is prioritized, the model should produce a response that is safe but may be perceived as unhelpful. through such training, the model gains the ability to generate responses that prioritize safety over helpfulness when instructed to do so. integrating goal priorities through crafted prompts obtains a significant reduction in the attack success rate (asr) of diverse jailbreaking techniques for both api-based and open-sourced llms. for instance, the total asr of chatgpt (openai, 2022) can be notably decreased from 66.4% to 3.6%, with little impact on the general per-formance. compared to common sft, our trainingbased solution can reduce the asr from 20.3% to 6.6% for llama2-13b, and exhibits better generalization ability by reducing the asr from 56.8% to 34.0% even when no jailbreak prompt is included in training. these encouraging results in defending against jailbreaking attacks through goal prioritization in turn support our hypothesis regarding the root cause of jailbreaks. we also have some interesting findings regarding the difficulty of jailbreaking attack and defense on different llms. we observe that although stronger llms such as gpt-4 could be susceptible to a wider range of jailbreaking attacks (i.e., more potent llms are more vulnerable), they are also more readily directed to defend against these attacks (i.e., more potent llms are more capable of being made safe). these seemingly contradictory observations provide insights into the relationship between llms' capability and safety. the main contributions of this work are as follows: ## method we illustrate how to introduce goal prioritization in figure 2 . we explore two distinct settings: one without (denoted as w/o) training and one with (denoted as w/ ) training. in the absence of training, we design a plug-and-play prompting method that includes the goal prioritization requirement and two in-context examples to guide llms towards prioritizing safety over helpfulness. under the training scenario, we develop a training pipeline that amalgamates various queries with distinct goal prioritization requirements. the training aims to effectively enable llms to learn and adhere to the specified goal prioritization requirements during the training process. ## experiments training set. for the w/ training setting, we collect 10k benign queries paired with gpt-4 responses from ultrafeedback @xcite . moreover, 500 harmful instructions from ad-vbench @xcite are randomly paired with jailbreaking prompts from @xcite . we ensure that the jailbreaking prompts in the test set are excluded in the training set. each jailbreak query in the training set is associated with both a benign (generated by chatgpt) and a harmful response (generated by vicuna-13b) under different goal prioritization requirements, in adherence with the methodology outlined in section 3.2. evaluated llms. for the w/o training setting, we evaluate both api-based llms (including gpt-3.5-turbo-0613 and gpt-4-0613) and opensourced llms (including vicuna-7b-v1.3, vicuna-13b-v1.3, vicuna-33b-v1.3, llama2-7b-chat and llama2-13b-chat). for the w/ training setting, base models are llama2-7b and llama2-13b. baselines. for the w/o training setting, we compare our method with vanilla llms without additional defense techniques, and self-reminder @xcite that adds instructions before and after the user's query to request not to generate harmful content. for the w/ training setting, we compare our method with vanilla sft that only uses benign queries and helpful responses, and aligned sft that additionally uses jailbreak queries with safe rejection responses. ## conclusion in this paper, we hypothesize the unawareness of goal priority to be the core of jailbreaking attacks' success. we then design methods to introduce goal prioritization during both training and inference stages. extensive experiments verify the effectiveness of our methods in defending against jailbreaking attacks, which in turn confirms our hypothesis about the root cause of jailbreaking attacks. we also find that stronger llms are more vulnerable but are also easier to fix, which offers valuable insights into the relationship between the capability and safety of llms. our work sheds light on the defenses against jailbreaking attacks and contributes to the development of more reliable llms.