S afe D ecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024)

Abstract: As large language models (LLMs) become increasingly integrated into real-world applications such as code generation and chatbot assistance, extensive efforts have been made to align LLM behavior with human values, including safety. Jailbreak attacks, which aim to provoke unintended and unsafe behaviors from LLMs, remain a significant LLM safety threat. We analyze tokens, which are the smallest unit of text that can be processed by LLMs and make the following observations: (1) probabilities of tokens representing harmful responses are higher than those of harmless responses, and (2) responses containing safety disclaimers appear among the top tokens when token probabilities are sorted in descending order. In this paper, we leverage (1) and (2) to develop SafeDecoding, a safety-aware decoding strategy for LLMs, to defend against jailbreak attacks. We perform extensive experiments to evaluate SafeDecoding against six SOTA jailbreak attacks (GCG, AutoDAN, PAIR, DeepInception, SAP30, and template based attack) on five LLMs (Vicuna, Llama2, Guanaco, falcon, and Dolphin) using four benchmark datasets (AdvBench, HEx-PHI, MT-Bench, and Just-Eval). Our results show that SafeDecoding significantly reduces attack success rate and harmfulness of jailbreak attacks without compromising the helpfulness of responses to benign user queries while outperforming six defense methods (Perpelexity, Paraphrase, Retokenization, Self-Reminder, ICD, and Self-Examination).

## introduction large language models (llms) such as chatgpt @xcite , llama2 @xcite , vicuna @xcite , and gemini @xcite have undergone remarkable advancements. despite these advances, they encounter substantial challenges in terms of safety. reports of llms producing biased @xcite 1 our code is publicly available at: https://github.com/ uw-nsl/safedecoding @xcite . the words in red are gcg suffixes. we note that although the token representing the word "sure" has a dominant probability, safety disclaimers such as "i", "sorry", and "as" are still present in the sample space, which is sorted in descending order in token probabilities. when a safety disclaimer token is sampled, the model would reject the attacker's harmful query. @xcite , inaccurate @xcite , or harmful contents @xcite highlight the critical need for robust safety measures. extensive efforts have been dedicated to aligning the behavior of llms with human values @xcite @xcite @xcite to ensure llms are helpful and harmless @xcite . despite advancements in alignment techniques, llms are still susceptible to adversarial inputs @xcite . recent studies have exposed a significant threat termed "jailbreak attack" @xcite @xcite @xcite @xcite , which can successfully bypass existing alignments. although multiple defenses, including input perturbation @xcite , input and output detection @xcite @xcite , and prompt demonstration @xcite @xcite , have been proposed, these methods lack effectiveness, incur high costs in inference time, and may compromise helpfulness of llms when serving benign users @xcite . we aim to defend llms against jailbreak attacks by introducing a new perspective on jailbreak success. our analysis of jailbreak attacks is through the lens of token probabilities, where a token is the smallest unit of text that can be interpreted by llms. this perspective, shown in figure 1 , leads to the following two observations. first, the success of a jailbreak attack can be attributed to the dominance of token probabilities aligned with the attack objectives (e.g., "sure, here's a tutorial for making a bomb"), leading to potential failures in widely used decoding strategies such as greedy and top-k @xcite when generating harmless content. second, although the model exhibits unintended behavior, tokens representing safety disclaimers such as "sorry, i cannot fulfill your request." exist in the sample space. this reveals an inherent awareness of the model of jailbreak attacks. building upon these insights, we propose safedecoding, a novel safety-aware decoding strategy to defend against jailbreak attacks. the key idea of safedecoding is to strategically identify safety disclaimers and amplify their token probabilities, while simultaneously attenuating the probabilities of token sequences that are aligned with the attacker's objectives. to achieve this, safedecoding begins with developing an expert model in the training phase, which is fine-tuned using a safety-aware dataset that we generate using the original model. in the inference phase, safedecoding first creates a sample space by identifying the intersection of the top tokens from both the original and fine-tuned models, effectively balancing the utility-safety tradeoff. safedecoding then defines a new token distribution based on the token probabilities of both the original and expert models. based on this new distribution, safedecoding samples tokens to generate a response to the input query. we evaluate the effectiveness, efficiency, helpfulness, and compatibility of safedecoding on five llms under six state-of-the-art jailbreak attacks, two harmful benchmarks, and two utility benchmarks. we compare safedecoding with six baseline methods. the results show that safedecoding consistently outperforms all baselines when defending against jailbreak attacks. furthermore, safedecoding incurs negligible computation overhead and allows llms to be helpful @xcite when responding to queries from benign users. ## related work in what follows, we summarize the related work. we first discuss approaches to jailbreak attacks, followed by defenses against jailbreak attacks. ## preliminaries this section presents existing decoding strategies followed by our threat model and problem setup. ## safety-aware decoding: safedecoding in this section, we present the overview of safedecoding, followed by the detailed design. ## experiments this section assesses the effectiveness, helpfulness, efficiency, and compatibility of safedecoding. ## conclusion and future work in this paper, we introduced safedecoding, a novel computationally lightweight and effective 5594 safety-aware decoding to defend against jailbreak attacks in llms. our insight in developing safedecoding was based on the observation that, even though probabilities of tokens representing harmful contents outweigh those representing harmless responses, responses containing safety disclaimers still appear among the top tokens when tokens are sorted in descending order by probability. this insight allowed safedecoding to attenuate the probabilities of token sequences that are aligned with the attacker's objectives, and amplify the token probabilities associated with safety disclaimers. our results showed that safedecoding can effectively defend against state-of-the-art jailbreak attacks while being efficient and helpful. we are developing safedecoding-icl, an in-context learning version of safedecoding to further reduce training costs.