Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM (2024)

Abstract: Recently, Large Language Models (LLMs) have made significant advancements and are now widely used across various domains. Unfortunately, there has been a rising concern that LLMs can be misused to generate harmful or malicious content. Though a line of research has focused on aligning LLMs with human values and preventing them from producing inappropriate content, such alignments are usually vulnerable and can be bypassed by alignment-breaking attacks via adversarially optimized or handcrafted jailbreaking prompts. In this work, we introduce a Robustly Aligned LLM (RA-LLM) to defend against potential alignment-breaking attacks. RA-LLM can be directly constructed upon an existing aligned LLM with a robust alignment checking function, without requiring any expensive retraining or fine-tuning process of the original LLM. Furthermore, we also provide a theoretical analysis for RA-LLM to verify its effectiveness in defending against alignment-breaking attacks. Through real-world experiments on open-source large language models, we demonstrate that RA-LLM can successfully defend against both state-of-the-art adversarial prompts and popular handcrafted jailbreaking prompts by reducing their attack success rates from nearly 100% to around 10% or less.

## introduction trained on a wide range of text data from the internet, large language models (llms) have exhibited exciting improvement in their generalization capabilities @xcite and widespread application in various domains such as finance @xcite , law @xcite , and healthcare industry (thirunavukarasu figure 1 : an illustration of alignment-breaking attack: an aligned llm gives unsafe responses to malicious requests with adversarial prompts. @xcite . while llms have showcased impressive potential, a rising concern is that they can also be maliciously utilized to generate content deviating from human values (e.g., harmful responses and illegal suggestions) @xcite due to the substantial amount of undesirable material existing in their training data. to tackle this issue, a line of research focuses on aligning llms with human preferences and preventing them from producing inappropriate content @xcite @xcite . these alignments typically adopt reinforcement learning from human feedback @xcite and ai feedback @xcite to fine-tune llms for alignments with human values. despite these efforts, an emerging class of jailbreak attacks can still bypass the alignment and elicit harmful responses from llms @xcite @xcite . these alignment-breaking attacks manually craft adversarial prompts by designing elaborate role-play @xcite or simply asking the llm to give the response starting with "absolutely! here's" @xcite . moreover, automatic jailbreak prompt generation methods have also been developed through dialogue encryption @xcite or the combination of greedy and gradient-based search methods @xcite . figure 1 shows an example that a malicious question appended with an adversarial prompt could successfully break the safety alignment. recently, @xcite have demon-strated that jailbreak attempts could be highly effective and transferable across different llms. this phenomenon suggests that existing safety alignment is far from robust to defend against carefully crafted adversarial prompts. till now, few attempts have been made to design dedicated mechanisms for defending alignmentbreaking attacks. a rudimentary defense currently employed relies on external tools to re-assess the potential harm of the llm responses. for instance, it could feed every potential response from the target llm into a third-party llm to determine whether the response is harmful or not @xcite . while this strategy enables filtering out possible harmful responses, there are several major drawbacks limiting its practicability: 1) existing llms are very sensitive to harmful keywords appeared in the input, and have a high propensity to misclassify benign content as harmful, even when the entire sentence is not talking about any harmful behavior (e.g., stating news or providing guidance/warnings). this could lead to a high falsepositive rate in harmful content detection; 2) the method heavily relies on the performance of the llm used as a harmful discriminator, while the llm itself is not designed to be an accurate harmful discriminator. the basis for its decisions remains ambiguous, implying that the harmful evaluation process could be opaque; 3) some types of alignment that can not be simply summarised as "harmful" (e.g., privacy, ethics, human values etc), thus it cannot cover such cases simultaneously. given the wide range of applications where llms could be utilized, finding an effective and practical defense against potential alignment-breaking attacks is both urgent and challenging. in this work, we design a robustly aligned llm (ra-llm) to defend against potential alignment-breaking attacks, which is built upon an already aligned llm and makes the existing alignments less prone to be circumvented by adversarial prompts. specifically, our key idea is that although an aligned llm can, to some extent, identify if the input request is benign or not, we cannot directly rely on that as it may not be robust. we consider an input request to be benign, only if we randomly drop a certain portion of the request and the llm still thinks it is benign in most cases. intuitively, such a random dropping operation would invalidate the adversarial prompts in alignment-breaking attacks, which are usually sensitive to small perturbations; on the other hand, the chances for the llm to reject benign requests are relatively low, even after random dropping. therefore, such a mechanism naturally leads to a robustly aligned llm. note that our ra-llm does not require any external "harmful" detectors, instead, our strategy only relies on the existing alignment capability inside the llm. due to the same reason, our approach is not limited to any specific type of alignment (e.g., harmful), but robustifies all existing model alignments. furthermore, we provide a theoretical analysis to verify the effectiveness of ra-llm. our experimental results on open-source large language models demonstrate that ra-llm can successfully defend against both state-of-theart adversarial prompts and popular handcrafted jailbreaking prompts by reducing their attack success rates from nearly 100% to around 10% or less. ## related works aligning llms with human preferences despite the excellent generalization capabilities, llms suffer from generating outputs that deviate from human expectations due to the significant amount of inappropriate content existing in unfiltered training data. to tackle this issue, a line of work focuses on aligning llms with human values @xcite @xcite @xcite . specifically, @xcite align llms by using reinforcement learning from human feedback @xcite to fine-tune pre-trained llm with human preferences as the reward signal, which reduces the generation of toxic content. @xcite train a less harmful system through the specification of a short list of principles and further improve the human-judged performance by introducing chain-of-thought style reasoning @xcite in fine-tuning stage. in addition to aligning llms in the fine-tuning stage, @xcite propose pertaining llms with alternative objectives that guide them to generate text aligned with human preferences and significantly reduce the generation of undesirable content by using conditional training @xcite . ## our proposed method in this section, we introduce the proposed robustly aligned llm for defending alignment-breaking attacks. before heading into details, we first discuss the threat model that is focused on in this paper. ## experiments in this section, we aim to validate the efficacy of our ra-llm from two aspects: 1) ra-llm can effectively reduce the attack success rate of adversarial prompts; 2) ra-llm minimally affects the outputs of benign samples. in the following, we first introduce our experimental settings and give a detailed analysis of our experimental results and ablation study. full details of the experiment can be found in appendix e. ## computational cost we adopt two mechanisms to ensure our method is time-efficient: (1) limited output length: we limit a small maximum generation length in monte carlo simulation (e.g., t @xmath0, since the negative prefixes often appear at the start of llm responses, allowing us to obtain effective defense without generating full responses; (2) early exit mechanism: during the monte carlo simulation, if the detected failure cases exceed our set threshold, ra-llm can directly terminate the process early and mark the input as malicious. we also evaluated the actual time overhead of ra-llm against the original llm in our experiments. we tested 150 attack samples and recorded the normal inference time on two llms and the time required by our ra-llms. we report the extra time per data on average overhead in ## adaptive attack in this section, we explore three potential adaptive attack methods against ra-llm to assess its resilience. repeating adversarial prompts since our method randomly drops p portion of tokens from the input x and uses monte carlo sampling to simulate all possible scenarios, any form of adversarial prompt may be discarded. hence, it's challenging to design an adaptive attack based on optimization for our defense method. however, one may also utilize this design choice and simply try increasing the length of the adversarial prompts (e.g., repeat the adversarial prompts after input for several times) to ensure the random dropping cannot fully remove the adversarial parts. in order to figure out whether such a potential adaptive attack can invalidate our defense or not, we conducted experiments on the harmful behaviors attack on both the original llm and our robustly aligned llm. the results are presented in replacing the target model with ra-llm in this adaptive attack approach, we directly substitute the target model used during the attack with its ra-llm version. that is, if ra-llm deems the input harmless, it will return the response of the underlying llm to the input. if ra-llm judges the input as harmful, i.e., ac(•) returns fail, it will return a fixed string "i'm sorry, but i can't assist with that." given that it's impossible to directly derive gradients for ra-llm, gradient-based attack methods like gcg @xcite ) cannot be applied. thus, we experimented with tap @xcite and autodan @xcite attack methods and found that after multiple iterations, both methods failed to generate effective jailbreak prompts, resulting in a 0% success rate. we speculate that since ra-llm always returns a fixed string and probability distribution for harmful inputs, and this string and distribution are manually specified by us, the attack methods cannot find a reasonable optimization direction, leading to convergence failure. incorporating an additional loss term as mentioned above, due to the non-differentiability of our random dropping mechanism, it's challenging to directly apply gradient-based attack methods on ra-llm. therefore, we considered designing a suitable adaptive attack for gradient-based whitebox attack methods. a potential method involves optimizing the probability of rac returning pass as one of the objective functions. however, for attackers, this means that each iteration during the attack process would require approximately 20 times more overhead. this is significant because such gradient-based attack methods (e.g., gcg @xcite ) already necessitate substantial computational resources. as an alternative, we experimented with treating random dropping as a "transformation" and applying the expectation over transformation (eot) method. we conducted experiments on the vicuna-7b model using the gcg's individual-behavior setting, dropping 30% of the tokens randomly at the start of each optimization, and then optimizing the remaining 70%. we found that under these circumstances, the loss oscillated near its initial value and failed to converge. we believe this phenomenon may be attributed to the following reasons: ## conclusion while a variety of alignment strategies have been proposed to guide llms to obey human values, recent works show that these alignments are vulnerable and can be bypassed by alignment-breaking attacks through adversarial prompts. in this work, we propose robustly aligned llms, which are built upon existing aligned llms with a robust alignment checking function, to defend against alignment-breaking attacks. one major advantage of our method is that there is no need to expensively retrain or fine-tune the original llm for defense purposes. we also provide a theoretical analysis to verify the effectiveness of our proposed defense. the exhaustive experiments clearly demonstrate our method can effectively defend against both automatically generated adversarial prompts and handcrafted jailbreak prompts. 10550 our work is primarily limited in two dimensions. first, the random dropping mechanism used in our proposed method still has a minor effect on benign samples in some models. specifically, the benign answering rate of guanaco-7b-chat-hf decreased from 95.3% to 92.0%. future work may investigate how to design defenses with less impact on benign samples, such as exploring better dropping methods to further reduce the adverse effect. second, due to the limitations of current jailbreak attack techniques, we have not assessed how our method performs when faced with very extreme cases, such as particularly long or short adversarial prompts. future work could further design these extreme cases and study their adaptability to our proposed defending method.