Play Guessing Game with LLM : Indirect Jailbreak Attack with Implicit Clues (2024)

Abstract: With the development of LLMs, the security threats of LLMs are getting more and more attention. Numerous jailbreak attacks have been proposed to assess the security defense of LLMs. Current jailbreak attacks primarily utilize scenario camouflage techniques. However their explicitly mention of malicious intent will be easily recognized and defended by LLMs. In this paper, we propose an indirect jailbreak attack approach, Puzzler, which can bypass the LLM’s defensive strategies and obtain malicious response by implicitly providing LLMs with some clues about the original malicious query. In addition, inspired by the wisdom of “When unable to attack, defend” from Sun Tzu’s Art of War, we adopt a defensive stance to gather clues about the original malicious query through LLMs. The experimental results indicate that the Query Success Rate of the Puzzler is 14.0%-82.7% higher than baselines on the most prominent LLMs. Furthermore, when tested against the state-of-the-art jailbreak detection approaches, Puzzler proves to be more effective at evading detection compared to baselines.

## introduction large language models (llms) are artificial intelligence (ai) systems for processing and generating human-like content, tightly integrating humans with ai through question-and-answer interactions. due to its remarkable abilities in content comprehension and logical reasoning, notable llms such as chat-gpt (openai, 2022), gemini-pro @xcite , and llama @xcite have shown superior capabilities in a variety of downstream tasks and universal chatbot @xcite . however, alongside the advancements in llms, there are growing concerns about their security threats, such as generating biases, providing unethical guidance, and producing content * these authors contributed equally to this work. † corresponding authors that contravenes societal values @xcite @xcite @xcite . in response to these challenges, llm developers set up multiple defensive strategies within the llms to mitigate this threat and align the output of llms with human values, which refers to the llm alignment @xcite . currently, a considerable amount of researches are proposed to assess the safety alignment of llms by constructing malicious prompts specifically engineered to elicit malicious responses from llms, which are called jailbreak attacks @xcite . the earlier practice of jailbreak attacks involved manually constructing specific scenario templates in the prompts to communicate with llms in a way that made them believe it was reasonable to respond to any queries within that scenario @xcite @xcite . however, these manually created templates based on scenario camouflage can be easily defended against by restricting the responses to known templates. to overcome this limitation, later studies have employed a learnable strategy to automatically design jailbreak templates that can bypass the defense mechanisms of llms. for example, researchers such as @xcite and @xcite utilize the llms to learn from existing prompts and generate the jailbreak prompts that reflect various new scenarios. although the automatically generated scenario templates pose a greater challenge for defense, they directly convey malicious intent within the prompts. as shown in figure 1 , llms can easily identify the malicious intent of the query as "steal from a store". consequently, these jailbreak prompts may be ineffective against the latest released llms. in comparison to jailbreak attacks that explicitly express malicious intent as mentioned earlier, we have observed that providing certain clues or hints of the original malicious intent can bypass the defensive strategies of llms while still acquiring the required malicious response. as illustrated in figure 1 , when we provide associated behaviors such as "time my visit during the store's busiest hours" and "study the layout of the store", llms have the capability to infer the underlying intent of "steal from a store" and generate the desired output. importantly, since this does not explicitly convey the malicious intent, i.e., each clue is not sufficient to reveal the intent of the original malicious query, traditional safety alignment mechanisms of llms struggle to defend against these types of attacks. this can be likened to playing a "guessing game" with the llm, where we provide verbal descriptions as hints without directly revealing the answer. nevertheless, acquiring the clues of malicious intent poses a significant challenge. it is akin to launching a direct attack on the llms when we approach them with direct queries. as sun tzu wisely stated in the art of war, "when unable to attack, defend." in light of this wisdom, we initially assume a defensive stance when interacting with the llms. by adopting this defensive viewpoint, we prevent the llms from blocking our queries and instead encourage them to generate a diverse set of defensive measures in response to the original malicious intent. building upon this defensive foundation, we can inquire about the offensive aspects of the defensive measures, which still fall outside the safety alignment mechanisms of the llms, thereby successfully obtaining the aforementioned clues of the malicious intent. we propose an indirect jailbreak attack approach, puzzler, which launches the attack by automatically providing the llms with clues of the original malicious query enabling them to escape llms' safety alignment mechanism and meanwhile obtain the desired malicious response. to achieve this, we first query the llms for a diverse set of defensive measures, then acquire the corresponding offensive measures from llms. by presenting llms with these offensive measures (i.e., the clues of the original malicious query), we prompt them to speculate on the true intent hidden within the fragmented information and output the malicious answer. for systematical evaluation, we evaluate puzzler across advbench subset @xcite and maliciousinstructions @xcite datasets and assessed performance on four closed-source llms (gpt3.5, gpt4, gpt4-turbo, gemini-pro) and two open-source llms (llama-7b, llama-13b). the performance is evaluated from two aspects, i.e., the following rate of the jailbreak responses and the query success rate. for the former, we evaluate whether the jailbreak's responses follow the original query, and for the latter, we determine whether the response from the llm contravenes its alignment principles. the experimental results show that the query success rate of puzzler significantly outperforms that of baselines. in addition, the responses generated by puzzler achieve a following rate of over 85.0% with the original queries, indicating the effectiveness of the indirect jailbreak. furthermore, we test the puzzler and the baselines with two state-of-the-art jailbreak detection approaches, and the results show that puzzler substantially outperforms the baselines in evading detection, demonstrating the stealthy nature of puzzler. we provide the public reproduction package1. ## jailbreak attack currently, the jailbreak attacks under llms are implemented through two categories of prompts, i.e., manually and automatically constructed prompts. for the manually constructed jailbreak prompts, @xcite systematically categorized existing jailbreak prompts for llms into three categories: 1) pretending, which attempts to alter the conversational background or context while maintaining the same intention, e.g., converting the question-and-answer scenario into a game environment; 2) attention shifting, which aims at changing both the conversational background and intention, e.g., shifting the attention of llms from answering malicious queries to completing a paragraph of text; 3) privilege escalation, which seeks to directly circumvent the restrictions imposed by the llm, e.g., elevating the llm's privileges to let it answer malicious queries. @xcite first rewrote the original prompts to change their representation based on the assumption of altering the feature representation of the original sentences, while keeping the original semantics unchanged. specific methods included performing partial translation or misspelling sensitive words, etc. then, they incorporated the revised prompts into designed attention shifting templates for jailbreak llms. @xcite leveraged the personification ability of llms to construct novel nested pretending templates, paving the way for further direct jailbreak possibilities. for the automatically generated jailbreak prompts, @xcite automated the generation of adversarial suffixes by combining greedy and gradient-based search techniques, and suffixes appended to the original malicious query can prompt large language models to recognize the importance of the original query, thereby eliciting a response. @xcite used an attacker llm to automatically generate jailbreaks for a separate targeted llm. given the attacker llm iteratively queries the target llm, updating and improving the existing jailbreak prompts based on the feedback. specifically, the attacker llm attempts to construct plausible scenarios from various angles to test the llm's receptiveness, such as disguising instructions for poisoning as a crucial step in cracking a criminal case. @xcite built upon @xcite achieves llm jailbreak with fewer queries by incorporating the tree of thought framework for querying the targeted llm and introducing evaluators to prune jailbreak prompts, which diverge from the original malicious query generated by the attacker llm. in general, regardless of the artificial or automatic approaches, their core idea is to package the original malicious query within a non-malicious scenario (or context), to divert the llm's attention and neglect the malicious content in the jailbreak prompts. with the rapid iteration of llm's own understanding, reasoning, and defense capabilities, the attacks based on the scenario camouflage are gradually becoming ineffective, as they still explicitly mention the easily perceived malicious intent. based on this, our approach attempts to represent the malicious intent of the malicious query implicitly. ## methodology figure 2 shows the overview of puzzler. puzzler consists of three phases: (1) defensive measures creation would create a diverse set of defensive measures, which serves as the initial attempt to bypass the safety alignment mechanism of llm; ## conclusion this paper presents an indirect approach (puzzler) to jailbreak llms by implicitly expressing malicious intent. puzzler first combines the wisdom of "when unable to attack, defend" by querying the defensive measures of the original query and attacking them to obtain clues related to the original query. subsequently, it bypasses the llm's safety alignment mechanisms by implicitly expressing the malicious intent of the original query through the combination of diverse clues. the experimental results indicate that the query success rate of the puzzler is 14.0%-82.7% higher than baselines on the most prominent llms. moreover, when tested against the two state-of-the-art jailbreak detection approaches, only 21.0% jailbreak prompts generated by puzzler are detected, which is more effective at evading detection compared to baselines.