A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2024)

Abstract: Large Language Models (LLMs), such as ChatGPT and GPT-4, are designed to provide useful and safe responses. However, adversarial prompts known as ‘jailbreaks’ can circumvent safeguards, leading LLMs to generate potentially harmful content. Exploring jailbreak prompts can help to better reveal the weaknesses of LLMs and further steer us to secure them. Unfortunately, existing jailbreak methods either suffer from intricate manual design or require optimization on other white-box models, which compromises either generalization or efficiency. In this paper, we generalize jailbreak prompt attacks into two aspects: (1) Prompt Rewriting and (2) Scenario Nesting. Based on this, we propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts. Extensive experiments demonstrate that ReNeLLM significantly improves the attack success rate while greatly reducing the time cost compared to existing baselines. Our study also reveals the inadequacy of current defense methods in safeguarding LLMs. Finally, we analyze the failure of LLMs defense from the perspective of prompt execution priority, and propose corresponding defense strategies. We hope that our research can catalyze both the academic community and LLMs developers towards the provision of safer and more regulated LLMs. The code is available at https://github.com/NJUNLP/ReNeLLM.

## introduction the advent of large language models (llms) has marked a significant milestone in the evolution of artificial intelligence (ai) systems, catalyzing a paradigm shift in various application domains. prominent examples of llms such as chatgpt @xcite , @xcite , llama2 @xcite , and claude2 (anthropic, * corresponding author 2023) have showcased their superior capabilities in a wide range of innovative applications, encompassing chatbots, code optimization, data augmentation, data annotation, and tool utilization @xcite @xcite @xcite . however, these powerful llms can sometimes exhibit inadequate safeguard performance when faced with carefully crafted malicious prompts @xcite . a famous example is the jailbreak prompt attacks @xcite @xcite . jailbreak prompt attacks on llms are typically categorized into two types: (1) manual designed jailbreak prompts @xcite @xcite , exemplified by @xcite , which intentionally craft prompts to bypass the llm's built-in safeguards. (2) learning-based jailbreak prompts @xcite , exemplified by gcg @xcite , which formulate the attack process as an optimization problem to search for the adversarial prompt suffix. the aforementioned methods exhibit certain limitations. firstly, manual jailbreak prompt attacks are typically intricate, requiring careful design for effectiveness. moreover, their dissemination on community websites leads to ineffectiveness due to the ongoing updates of llms. @xcite . secondly, learning-based prompt attacks are easily mitigated because the adversarial suffixes they generate, which are searched on another white-box model, are usually meaningless. @xcite @xcite . furthermore, such methods demand substantial time to find the optimal suffix and demonstrate lower efficacy on commercial llms such as claude-2 @xcite . to tackle these shortcomings, we propose renellm, an automated and efficient framework for generating jailbreak prompts to assess the security performance of llms. renellm includes two main steps: (1) prompt rewriting, which involves a series of rewriting operations on the initial prompt that do not change its core semantics (such as paraphrasing with fewer words, change expression style, etc.), making it easier to elicit a response from llms. (2) scenario nesting, in order to make the rewritten prompts more stealthy, we nest them into a specific task scenario (such as code completion, text continuation, etc.), engaging llms themselves to find the effective jailbreak attack prompts. renellm generalizes jailbreak prompt attacks (figure 1 gives an example) and demonstrates efficiency and transferability across multiple llms, guiding researchers and developers to explore safer defense methods for llms. in summary, our contributions are summarized as follows: ## methodology in this section, we elaborate in detail on renellm, a generalized framework for the automatic generation of jailbreak prompts. renellm generalizes jailbreak prompt attacks into two aspects: prompt rewriting(section 3.2) and scenario nesting(section 3.3). the former involves a series of rewriting operations on the initial prompt without changing its semantics, while the latter selects a scenario for the rewritten prompt and further disguises it through nesting. it is worth noting that the entire process is automatically completed by llms without the need for additional training and optimization. the formal definition of our jailbreak attack method is provided in section 3.1. figure 2 outlines renellm, while algorithm 1 provides the specific implementation details. ## experiment in this section, we present the evaluation and analysis of the security performance of some of the leading closed-or open-source llms using our proposed method. ## evaluating safeguards effectiveness in this section, we conduct additional experiments to evaluate the performance of existing llms' safeguard methods and report the results in as llama-2-7b-chat and claude-2 demonstrated leading safety performance among all llms, we utilize them as the evaluation models. we select 368 prompts generated by renellm that have a gpt-asr of 100.0 across all llms. the results in ## analysis of renellm the observed effectiveness of our method raises the natural question of why and how it helps to bypass the security defenses of llms, and how to specifically defend against this kind of attack. in this section, we conduct comprehensive experiments and analyses to understand the above two points. ## conclusion in this paper, we introduce renellm, an automatic jailbreak prompt generation framework. by generalizing the jailbreak process into prompt rewriting and scenario nesting, we achieve high attack success rates on various representative llms efficiently. our research reveals that current defense methods fall short in providing adequate safety for llms. to comprehend why renellm is effective, we conduct attention visualization experiments, discovering a shift in llms' execution priorities for prompts before and after jailbreak. consequently, we explore several defense strategies, which involve introducing priority prompts, enhancing llms' safety through sft and defending jailbreaking with harmfulness classifiers. the results of the defense experiments suggest that providing llms with generalized and efficient security protection remains a challenging task. we hope that our study will stimulate both the scholarly community and llms providers to work towards the delivery of more secure and better governed llms.