Distract Large Language Models for Automatic Jailbreak Attack (2024)

Abstract: Extensive efforts have been made before the public release of Large language models (LLMs) to align their behaviors with human values. However, even meticulously aligned LLMs remain vulnerable to malicious manipulations such as jailbreaking, leading to unintended behaviors. In this work, we propose a novel black-box jailbreak framework for automated red teaming of LLMs. We designed malicious content concealing and memory reframing with an iterative optimization algorithm to jailbreak LLMs, motivated by the research about the distractibility and over-confidence phenomenon of LLMs. Extensive experiments of jailbreaking both open-source and proprietary LLMs demonstrate the superiority of our framework in terms of effectiveness, scalability and transferability. We also evaluate the effectiveness of existing jailbreak defense methods against our attack and highlight the crucial need to develop more effective and practical defense strategies.

## introduction commercial large language models (llms) like chatgpt @xcite , @xcite , @xcite and claude @xcite have achieved great progress in various natural language processing (nlp) tasks in recent days. meanwhile, open-sourced llms such as llama-2 @xcite , vicuna @xcite and @xcite have also shown remarkable progress. the widespread deployment and advanced capability of llms have raised concerns about the potential misuse of technology, including issues like bias and criminal activities @xcite . to harden llms for safety, extensive efforts have been made before these models' release to align their behavior with human values, with the primary goal of ensuring their helpfulness, honesty and harmlessness @xcite . however, even aligned llms are still vulnerable to jailbreak attacks @xcite @xcite , where specially designed prompts are used to circumvent llm safeguards (see the example in figure 1 ). these attacks are engineered to elicit undesirable behaviors, such as producing harmful content or leaking personally identifiable information, that the model is trained to avoid. manually crafted jailbreak prompts, such as do-anything-now @xcite , employ human ingenuity to create prompts that are understandable and interpretable. while effective and transferable, these prompts are not scalable. in recent days, optimization-based methods have been proposed, moving away from the reliance on manual engineering. white-box attack methods @xcite @xcite use gradientbased optimization techniques. this requires the ability to calculate or approximate gradients of the model's output with respect to its input, which is possible only when the target model's details are known. on the other hand, black-box methods @xcite @xcite do not require any knowledge of the internal workings of the target model. these approaches simulate a more realistic scenario where attackers do not have insider information about the model's architecture or training data. in this work, we propose distraction based adversarial prompts (dap), a novel black-box jailbreak framework, to automate red teaming @xcite and in turn strengthen llms. we decompose the jailbreak input to target llm into two parts: jailbreak template and malicious query. the jailbreak template only reserves a placeholder for the malicious query and does not contain sensitive texts. dap designs and optimizes the jailbreak templates automatically with three key components, namely, malicious content concealing, memory-reframing, and prompt optimization (see figure 2 ). previous work on llm distraction @xcite has demonstrated that the reasoning capabilities of llms can be easily influenced by irrelevant context. this insight motivates our approach of concealing malicious request within a complex and unrelated scenario, thereby diminishing the model's capacity to identify and reject malicious requests. however, simply concealing the malicious content often leads to responses that are unrelated to the malicious request and closely tied to the scenario. inspired by the over-confidence phenomenon of llms @xcite , we propose a memory-reframing scheme to distract the attention away from the unrelated scenario and concentrate on the malicious request. by instructing the target llm to initiate its response with a certain string, such as "sure! i will shift my focus to the malicious request", the model tends to follow its own partially generated response and respond to the malicious request. to automatically generate and optimize the jailbreak template, we employ an attacker llm for jailbreak template generation, as well as a target llm and a judgement model for the evaluation of generated jailbreak template. to the best of our knowledge, we are the first to leverage the distraction mechanism for the automated generation of jailbreak prompts. 1 1 https://github.com/sufenlp/attanttionshiftjailbreak we conduct comprehensive experiments to validate the effectiveness of dap on both open-source and proprietary llms. our results show that dap achieves top-1 attack success rates (asr) of 66.7% and 38.0% to bypass the safety alignment of chat-gpt (openai, 2022) and gpt-4 (openai, 2023), respectively. our research contributions are as follows: ## methods as shown in figure 2 , dap has three key components to devise effective jailbreak prompts automatically: (1) malicious query concealing via distraction; ## related work llms and safety alignment recent advances of llms have greatly improved performance across various nlp tasks, paving the way for numerous real-world applications. however, it is crucial to ensure that these llms are trustworthy and safe when applied in real-world. researchers explore approaches to align the llms with human preference and avoid misuse and harm. reinforcement learning from human feedback @xcite first trains a reward model on a human preference dataset and then optimizes the llms to find a policy that maximizes the learned reward. reinforcement learning from ai feedback @xcite proposes to use llm-labeled preference data as well as human-labeled preference data to jointly optimize for safety alignment. direct preference optimization @xcite directly optimizes llms for the policy that aligns with the preferences best with a classification training objective. the aligned llms are expected to follow human values to be safe and trustworthy, and generate helpful and harmless responses. jailbreak attack previous works observe that even aligned llms are fragile to a variety of attacks @xcite . the jailbreak attack intentionally designs malicious user instructions that adversarially trigger llms to produce uncensored, undesirable and offensive content consistent with the attacker's intention @xcite . redteaming is a strategy to enhance the safety and alignment of llms by checking and disclosing the covert cases in which the llms may fail @xcite . researchers explore different jailbreak approaches to red-teaming llm which in turn builds more robust and better aligned llms. these approaches can be roughly divided into three categories: manually crafting methods @xcite , optimization-based methods @xcite @xcite @xcite @xcite and long-tail encoding based methods @xcite @xcite . ## conclusion in this work, we propose dap, a novel jailbreak attack framework designed to generate fluent and coherent jailbreak templates universal to all malicious queries. our framework is inspired by the attention mechanisms of llms and consists of three components: concealing malicious content, memory reframing, and optimization algorithm. experiments on five open-source and closed-source llms demonstrate the strong attack success rates of dap for direct attacks, as well as cross-model and cross-query transferred attacks. as llms become more capable and widely used, it becomes increasingly crucial to have informed assessments of model safety, including disclosing the covert cases in which the llms may fail. we thus view our work on automatic jailbreak attack as a step towards this goal.