D r A ttack: Prompt Decomposition and Reconstruction Makes Powerful LLM s Jailbreakers (2024)

Abstract: Safety-aligned Large Language Models (LLMs) are still vulnerable to some manual and automated jailbreak attacks, which adversarially trigger LLMs to output harmful content. However, existing jailbreaking methods usually view a harmful prompt as a whole but they are not effective at reducing LLMs’ attention on combinations of words with malice, which well-aligned LLMs can easily reject. This paper discovers that decomposing a malicious prompt into separated sub-prompts can effectively reduce LLMs’ attention on harmful words by presenting them to LLMs in a fragmented form, thereby addressing these limitations and improving attack effectiveness. We introduce an automatic prompt Decomposition and Reconstruction framework for jailbreaking Attack (DrAttack). DrAttack consists of three key components: (a) ‘Decomposition’ of the original prompt into sub-prompts, (b) ‘Reconstruction’ of these sub-prompts implicitly by In-Context Learning with semantically similar but benign reassembling example, and (c) ‘Synonym Search’ of sub-prompts, aiming to find sub-prompts’ synonyms that maintain the original intent while jailbreaking LLMs. An extensive empirical study across multiple open-source and closed-source LLMs demonstrates that, with fewer queries, DrAttack obtains a substantial gain of success rate on powerful LLMs over prior SOTA attackers. Notably, the success rate of 80% on GPT-4 surpassed previous art by 65%. Code and data are made publicly available at https://turningpoint-ai.github.io/DrAttack/.

## introduction the development of large language models (llms) has significantly advanced ai capabilities, enabling success in various tasks @xcite @xcite . to promote responsible usage, existing llms often need safety alignment to avoid generating harmful content. however, recent research @xcite ., gpt-4 gpt-3.5-turbo claude-1 claude-2 gemini-pro vicuna 7b vicuna 13b llama-2 7b llama-2 13b 20 40 60 80 100 gcg autodan ica pair deepinception renellm drattack (ours) figure 1: attack success rate (asr) (%) of drattack and other prompt-based jailbreaking methods. drattack obtains a substantial gain of asr on powerful llms (gpt, claude, gemini) over prior sota attackers. this paper exposes a new under-explored type of vulnerability. our main insight is that while the malicious prompt as a whole can easily be perceived, it can be decomposed into a set of subprompts with significantly reduced attention to easily jailbreak victim llms (in figure 3 ). inspired by this observation, we develop a new type of jailbreaking attack that disguises a malicious prompt via syntactic parsing (e.g., "make" and "a bomb"). the concrete algorithm, termed decompositionand-reconstruction attack (drattack), operates through a three-step process: (1) decomposition via syntactic parsing breaks down a malicious prompt into seemingly neutral sub-prompts. (2) implicit reconstruction via in-context learning reassembles sub-prompts by benign and semanticsimilar assembling context. (3) sub-prompt synonym search shrinks the search space to make the search more efficient than optimization in whole vocabulary. extensive empirical evaluation demonstrates that prevalent llms struggle with this attack. drattack substantially increases the success rate over prior sota attacks on the most powerful llms. on gpt-4, drattack achieves an attack success rate of over 80% (human evaluation), surpassing the previous best results by over 65% in absolute terms. this paper provides a feasible, fourth category to the current taxonomy: decomposition-based method that breaks the harmful prompt into subphrases (in figure 2 ). while some initial studies @xcite in this category have demonstrated limited success (see appendix c.3), we show that current llms are highly prone to becoming victims of attacks in this category -they can be jailbroken with merely 15 queries. 2023 ) show that dissecting complex questions into a set of simpler sub-questions allows llms to process and respond with greater accuracy and more details. in downstream tasks, this technique has been applied to improve prompt candidate selection @xcite , refine model training processes @xcite , optimize model fine-tuning @xcite and improve the performance of vision-related tasks @xcite . to the best of our knowledge, we provide the first method that shows prompt decomposition can be leveraged to develop a strong attacker. "make a bomb" "make" "bomb" open-source llms probability "sorry" (a) investigation on rejection tokens llama-2 vicuna 0 20 40 60 80 100 probability of rejection tokens (%) (b) probability figure 3: (a) an illustration of our exploration on open-source llms when given initial and fragmented prompts. (b) average probability of generating rejection tokens in response. llms prompted with fragmented prompts show a lower probability of rejection tokens (e.g., "sorry") compared to those prompted with initial malicious prompts. ## motivation prompt-based jailbreak attacks can be viewed as algorithms maximizing the likelihood of response given a malicious query. we then explore that decomposition, introducing malicious prompts into fragmented form, can conceal malice by analyzing the rejection tokens' possibility. with this insight from open-source models, we generalize this strategy in section 4 as a black-box attack to jailbreak both open-source and closed-source llms. prompt-based attack queried by a prompt p, an llm can either return an answer a p or reject the question r p if query p is malicious. when the llm rejects to answer a malicious query p, a jailbreaking algorithm searches for an adversarial prompt p ′ that can elicit the desired answer a p from the target llm. therefore, jailbreaking algorithms are essentially trying to solve this optimization problem: ## drattack framework drattack represents a novel approach to jailbreaking llms, employing prompt decomposition and reconstruction to generate an adversarial attack. this section describes each component of the proposed drattack. as illustrated in figure 4 , the entire drattack pipeline consists of two parts: decomposition and reconstruction. the section is organized as follows: section 4.1 presents an overview of the entire pipeline; section 4.2 explains the decomposition step using semantic parsing to derive sub-prompts from the original attack prompt; section 4.3 discusses the implicit reconstruction via in-context learning (icl), reassembling sub-prompts for attacking llms. the decomposition step is critical for breaking down the prompt into less sensible elements, while the reconstruction step cleverly reassembles these elements and illicit llms to generate harmful contents. section 4.4 introduces a supplementary benefit of our framework: synonym search on sub-prompts, which modifies sub-prompts to get a more effective jailbreaking. ## ablation study better example in icl reconstruction, higher asr we investigate whether a semantically similar context in icl reconstruction can improve the assembly of harmful responses. we design three types of contexts where semantic-irrelevant context uses irrelevant assembling demo; semanticrelevant context gets benign prompt by making every sub-prompts replaceable; semantic-similar context gets benign prompt by restricting replaceable sub-prompts, maintaining prompt main sentence while replacing subordinate sub-prompts. the results in figure 9 (a) indicate that using a semantically similar demo in icl reconstruction is essential for drattack. instead of only prompting plain benign prompts to generate examples, we also add a naive suffix to generate benign examples more systematically with the instruction "give your answer step-by-step" @xcite and more affirmatively with the instruction "start your sentence with 'sure, here is'" @xcite . the results in figure 9 (b) show that more systematic and affirmative examples can improve asr. effectiveness of synonym search besides decomposition and reconstruction, drattack introduces a synonym search to boost attack perfor-mance. we investigate the effectiveness of synonym search by decoupling it from drattack, directly applying it to sub-prompts (e.g., "how to make a bomb" to "how to construct a bomb"). fig. 10 (a) demonstrates that decomposition and reconstruction are essential parts that make drattack jailbreaking victim llms, while synonym search boosts attack performance. we further investigate top-k synonyms of synonym search and observe slight improvements in asr with more synonym candidates in figure 10(b) . ## conclusion this paper successfully demonstrates a novel approach to automating jailbreaking llms through the prompt decomposition and reconstruction of original prompts. our findings reveal that by embedding malicious content within phrases, the proposed attack framework, drattack, significantly reduces iteration time overhead and achieves higher attack success rates. through rigorous analysis, we have evaluated the performance of leading llms under various prompt types, highlighting their vulnerabilities to drattack. our assessment of current safety mechanisms of these models underscores a critical gap in their ability to thwart generalized 13898 attacks like those generated by drattack. this vulnerability indicates an urgent need for more robust and effective defensive strategies in llm.