A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models (2024)

Abstract: Large Language Models (LLMs) have increasingly become central to generating content with potential societal impacts. Notably, these models have demonstrated capabilities for generating content that could be deemed harmful. To mitigate these risks, researchers have adopted safety training techniques to align model outputs with societal values to curb the generation of malicious content. However, the phenomenon of “jailbreaking” — where carefully crafted prompts elicit harmful responses from models — persists as a significant challenge. This research conducts a comprehensive analysis of existing studies on jailbreaking LLMs and their defense techniques. We meticulously investigate nine attack techniques and seven defense techniques applied across three distinct language models: Vicuna, LLama, and GPT-3.5 Turbo. We aim to evaluate the effectiveness of these attack and defense techniques. Our findings reveal that existing white-box attacks underperform compared to universal techniques and that including special tokens in the input significantly affects the likelihood of successful attacks. This research highlights the need to concentrate on the security facets of LLMs. Additionally, we contribute to the field by releasing our datasets and testing framework, aiming to foster further research into LLM security. We believe these contributions will facilitate the exploration of security measures within this domain.

## introduction large language models (llms), such as gpt @xcite and llama (hugging face, 2023a), play a pivotal role across a spectrum of applications, from text summarization @xcite 2024) to code generation @xcite . the popularity of llms in everyday scenarios underscores their significance. however, this ubiquity also raises security concerns associated with llms @xcite . several types of vulnerabilities have been identified in llms @xcite . among these, the jailbreak attack stands out as a prevalent vulnerability, where specially designed prompts are used to bypass the safety measures of llms, facilitating the production of harmful content. there has been notable research aimed at addressing jailbreak attacks. for example, liu et al. @xcite investigate various mechanisms for jailbreak prompting and assess their effectiveness. zou et al. @xcite apply a white-box approach combined with adversarial attacks to create jailbreak prompts. additionally, deng et al. @xcite explore using llms to generate jailbreak prompts in a black-box setting. to defend against jailbreak attacks, robey et al. @xcite proposed a method that involves randomly omitting a certain number of tokens from the input to detect malicious attempts. meanwhile, pisano et al. @xcite introduced an approach that employs an auxiliary model to assist the primary model in identifying hazardous information. despite the various jailbreak attack and defense methodologies, to the best of our knowledge, there remains a significant gap in the literature regarding comprehensive evaluations of how well the attack methodologies can perform against defended llms and how well defense mechanisms against jailbreak attacks. while @xcite and @xcite explore various attack techniques, they did not evaluate those on defense techniques, and vice versa. to address this research gap, we undertake a comprehensive empirical study on jailbreak attack and defense techniques for llms. our study is designed to answer two critical research questions. first, we investigate the effectiveness of various jailbreak attack approaches on different unprotected llms, encapsulated in the question (rq1: effectiveness of jailbreak attacks). second, we evaluate the effectiveness of defense strategies against these attacks on varied llms, posed as (rq2: effectiveness of jailbreak defenses). during the baseline selection phase, we chose nine attack methods and seven defense mechanisms, drawing on four seminal works, including notable libraries @xcite , and the openai moderation api @xcite , prioritizing prevalent and accessible methods with open-source code. in the benchmark construction phase, our benchmark, initially based on @xcite , was expanded through additional research @xcite ) and a gpt model in "do anything now" mode, resulting in 60 categorized malicious queries following openai's guidelines. for result labeling, the roberta model was fine-tuned for classifying malicious responses, achieving 92% accuracy, outperforming gpt-4's 87.4%. manual validation ensured the reliability of our classification. in the evaluation phase, we employed metrics for assessing attack efficiency and effectiveness, alongside defense robustness against malicious and benign inputs, establishing a comprehensive framework for evaluating llm security. our analysis reveals several notable insights. specifically, among the various jailbreak attack techniques, template-based methods demonstrate superior effectiveness. in contrast, gradient-based generative approaches, especially in 'white-box' scenarios, generally fall short of the performance achieved by universal generative methods. additionally, our findings highlight the significant impact of special tokens on the success probability of attacks. as for defense techniques, we identify the bergeron method as the most effective defense strategy to date, while all other defense techniques in our study perform badly as they either cannot stop jailbreak attacks at all or are too strict such that benign prompts are also prohibited. our results underscore a great need for the development of more robust defense mechanisms. in summary, our work presents several contributions to the field: the raw data, the benchmark platform, and additional details are available on a companion website of this paper: https://sites.google.com/view/ llmcomprehensive/home . ## background and related work this study underscores the effectiveness of specific attack methodologies against various defense strategies and vice versa, filling a gap not addressed in contemporary literature @xcite . these works primarily focus on evaluating various attack techniques against unprotected models, with the exception of initial safety training. our research conducts the first comprehensive survey that evaluates the reciprocal impacts of both attack and defense techniques. ## rq2: effectiveness of jailbreak defense our study meticulously evaluates defense mechanisms against malicious queries as well as the handling of benign questions. the outcomes of this evaluation are systematically tabulated in tables 12, 13, and 14. these results are further visualized in figure 3 , where the optimal defense strategies are identified by their proximity to the upper left corner of the plot, signifying lower dpr and higher bsr. our findings reveal that, apart from the bergeron method, the efficacy of the current defense strategies remains largely inadequate. additionally, our comparative analysis of the quality of benign responses generated through three innovative methodologies disclosed minor variance among them, as elaborated in to further understand the influence of loss metrics on a model's vulnerability to jailbreaking, we conducted a targeted experiment. a question was randomly selected from our dataset, and the experiment's findings are visually represented in figure 4 . the experiment showed that vicuna began the process with a higher initial loss but saw a significant reduction in loss, stabilizing after 12 steps and five successful jailbreak attempts. however, it maintained a higher final loss compared to llama. in contrast, llama started with a lower initial loss and demonstrated a slower reduction in loss over time, ultimately failing to jailbreak the question within 75 iteration steps despite exhibiting a significantly lower final loss than vicuna. these outcomes suggest that llama's foundational safety training plays a pivotal role in its enhanced defense against jailbreak attempts. it implies that integrating advanced safety training protocols into developing open-source models could markedly reduce the efficacy of white-box attacks, thereby enhancing their security posture. ## conclusions in this work, we present the first comprehensive assessment of existing attack and defense strategies in the context of llm security. additionally, we contribute to the field by releasing the first framework specifically designed for assessing the robustness of llms against various threats. we selected nine attacks and seven defensive mechanisms from existing literature and software libraries for our analysis. our experimentation, conducted on three distinct models, reveals that template methods are notably effective, with 78 templates technique identified as the most powerful one. regarding generative methods, gptfuzz emerged as the most effective given the experiment budget. our investigation into question categorization demonstrated that all three models exhibit enhanced resilience against queries related to unlawful practice, harmful content, and illegal activities. however, our analysis of current defensive measures indicates a general ineffectiveness, with bergeron showing comparatively better performance. we highlight the necessity of establishing a uniform baseline for jailbreak detection, as existing defenses employ varied methodologies, and the need to develop better defense techniques. additionally, our study observed the impact of using the '[/inst]' marker in the llama model. looking forward, we aim to continuously incorporate evolving attacks and defenses into our framework, thereby providing a dynamic overview of the field's progression.