Towards Understanding Jailbreak Attacks in LLM s: A Representation Space Analysis (2024)

Abstract: Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. Although there are diverse jailbreak attack strategies, there is no unified understanding on why some methods succeed and others fail. This paper explores the behavior of harmful and harmless prompts in the LLM’s representation space to investigate the intrinsic properties of successful jailbreak attacks. We hypothesize that successful attacks share some similar properties: They are effective in moving the representation of the harmful prompt towards the direction to the harmless prompts. We leverage hidden representations into the objective of existing jailbreak attacks to move the attacks along the acceptance direction, and conduct experiments to validate the above hypothesis using the proposed objective. We hope this study provides new insights into understanding how LLMs understand harmfulness information.

## introduction large language models (llms) have become ubiquitous in various applications such as providing financial advice and assisting trading @xcite , supporting clinical decisions @xcite , and assisting law thematic analysis @xcite , due to their exceptional ability to understand and generate human-like text. however, as llms are trained on vast text corpora which are usually scratched from the internet and contain diverse topics including toxic content, they can sometimes produce inaccurate or harmful contents @xcite @xcite . therefore, various safety aligning mechanisms, such as reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite have been leveraged to align llms with human values and prevent them from generating harmful content. despite these efforts, a new class of vulnerability known as "jailbreak attacks" @xcite has emerged. jailbreak attacks manipulate the model's input prompt to bypass the safety mechanisms, enabling the generation of harmful contents. various jailbreak attack algorithms such as gcg @xcite , au-todan @xcite , and pair @xcite have been proposed to exploit vulnerabilities in both open-source and api-only llms. however, most of these studies do not provide a comprehensive understanding of the internal mechanisms of these jailbreak attacks, which makes it still unknown when and how jailbreak attacks will succeed. a more thorough understanding is desired for further improving the performance of these attacks, as well as for devising more effective defenses. in this work, we focus on understanding the behavior of jailbreak attacks through the learned representation space of the victim llms. in detail, in our analysis in section 3, we first find that wellaligned llms (such as llama with rlhf) can provide clearly separable representations for the harmless prompts and harmful prompts. it suggests these models can effectively extract the harmful information from the harmful prompts and distinguish them with harmless prompts. more importantly, we further visualize the representations of jailbreak prompts obtained through representative jailbreak attacks like gcg @xcite , au-todan @xcite , and pair @xcite . we observe an increasing ratio of succeeded jailbreak prompts over failed jailbreak prompts along the direction from the cluster of harmful prompts to the cluster of harmless prompts. this observation indicates that prompts moving in the aforementioned direction towards the cluster of harmless prompts are more effective at deceiving the victim llms. based on these insights, to comprehensively understand the role of harmfulness in jailbreak attacks, we further leverage the representation in existing attacks by introducing a novel optimization objective to move the representation of the prompt toward the acceptance direction. this new optimization objective can be simply combined with existing white-box jailbreak attack methods. we employ our method in gcg @xcite and autodan @xcite , and conduct experiments to study attack success rate (asr) brought by this new method compared with corresponding baselines. for example, we achieve an asr of 62.31% over llama-2-13b-chat model @xcite after enhancing gcg @xcite , which is 36.16% higher than the corresponding baseline (26.15%). these results further support our understanding of jailbreak attacks. ## related works safety alignment of llms. to avoid the generation of harmful and toxic contents, various mechanisms that align llms closer to human values @xcite @xcite are developed. the general approach involves fine-tuning llms with human feedback @xcite @xcite . specifically, supervised fine-tuning (sft) @xcite collects a large volume of demonstrations and comparisons from human labelers, and fine-tunes llms using behavioral cloning and reward modeling to do summarization recursively. reinforcement learning from human feedback (rlhf) @xcite @xcite trains a reward model based on human feedback and fine-tunes the model with reinforcement learning via proximal policy optimization on the reward model. while these alignments significantly reduce the generation of harmful content, recent researches reveal that safety-aligned llms still have a chance to output undesired answers under certain situations @xcite . jailbreak attacks in llms. jailbreak attacks typically manipulate input prompts to bypass the safety alignment of llms and induce them to produce harmful responses. generally, these attacks are divided into manually crafted prompts (such as dan @xcite ) and automated attack meth-ods, which are much more efficient and effective. gcg @xcite considers a token-level manipulation and optimizing an adversarial suffix with a greedy coordinate gradient descent approach. the adversarial suffix is optimized to force the model output confirmation responses. another attack method, autodan @xcite , utilizes a genetic algorithm to optimize the whole prompt on the sentence and paragraph level to maintain fluency and high attacking effectiveness simultaneously. these methods require a white-box setting, while another line of attacks focuses on api-only models like gpt-4 @xcite . pair @xcite leverages an llm as the optimizer @xcite to generate the jailbreak prompt, and only need to query the model without access to the internal part of the model. more attack methods explores to generate attacking prompts with the help of llms, including tap @xcite , gptfuzzer @xcite , masterkey @xcite , and ad-vprompter @xcite . though these methods show great potential in bypassing the safety alignment of llms, the inner mechanism of jailbreaks is still under-explored. ## analysis on the representation for jailbreak attacks although various jailbreak attack methods have been proposed, it remains unclear when and how an attack can succeed in misleading a model's output. therefore, we aim to analyze the behavior of these jailbreak attacks by visualizing the representations from victim llms. specifically, we will investigate the representations of the anchor prompts and different types of jailbreak prompts, to explore the answers to the following questions: notations. before our analysis, we first provide necessary notations. in the following sections, we denote the length (number of tokens) of a prompt x as n, the vocabulary size as m, the embedding dimension as d, the process of the victim model taking a prompt x producing its representation as h : r n×@xmath0, and g : r @xmath1. analysis setups. to answer the questions above, we mainly focus on studying popular opensource llms including llama-2-7b-chat (touvron et al., 2023) (llama2-7b), llama-2-13bchat @xcite (llama2-13b), llama-3-8b-instruct (meta) (llama3-8b), and gemma-7bit @xcite . following the method of @xcite ; @xcite , given each model, we visualize the model's last hidden states of the last input text token as the representation of each prompt. our focus is primarily on observing the following types of prompts: the choice of harmful and harmless anchor datasets is crucial for determining how effectively the first two principal components of pca captures harmfulness information. when the two datasets differ significantly in harmfulness but remain highly similar in other aspects, such as query formats and text lengths @xcite , the harmfulness will be a major source of variation in the data. in this case, the first two principal components of pca will mainly focus on explaining the data variation in terms of the harmfulness. we mainly study three jailbreak attacks including white-box attacks, gcg @xcite and autodan @xcite , and black-box attack, pair @xcite . for the ease of further observation and analysis, both gcg and autodan employ early stopping (terminate on success) as their termination criteria to boost the asr of each method. we also conduct the same experiments without applying early stopping, and the result indicates that the conclusion of this section still holds (cf. appendix a.2). details on the asr of each attack can be found in appendix a.1. to visualize the representation, we first conduct pca dimension reduction on the representations of the harmless and harmful anchor datasets d a ∪ d r to find the first two principal components. then, we project each representation vector, as well as the representation of prompts in jailbreak datasets (initial, failed and succeed) onto the 2-d space spanned by these two principal components. the visualization result is in figure 1 . from figure 1 , we can have the following observations: observation 1. harmful and harmless anchor prompts are separable (in well-aligned llms). if we only focus on the representations of anchor datasets (• harmful and • harmless), we can find that the representations are well separated between harmful and harmless anchor prompts, especially in llms such as llama2-7b, llama2-13b and llama3-8b. it indicates that these models can effectively capture harmful information from the harmful prompts and distinguish them from harmless prompts. in addition to the visualization, the pca explained variance ratios are 0.4639 and 0.0403 for the first two principal components on llama2-7b, respectively. this further demonstrates that the two dimensions of pca effectively capture the major variation in the data. we also calculate the ratio of between-class variation and within-class variation of the two anchor datasets on llama2-7b. with the two datasets, one can calculate the overall variation among all samples (total variance), and can also calculate the variation within each cluster (within-class variance) and between all clusters (between-class variance). a larger between-class variance ratio indicates that the clusters are more separated from each other. the first two principal components explained 50.43% of the total variance, among which 85.57% is between-class variation. these values further verify that (1) harmfulness is one major source of variation in the representation, (2) the model can to ease the explanation of the observations, we formally define the acceptance direction as follows: definition 3.1. denote the acceptance center c @xmath2 in this section, we leverage the observations in section 3 to study whether existing jailbreak attacks can achieve a higher asr by inducing the acceptance direction. for simplicity, we summarize the observations and impose the following hypothesis: hypothesis 4.1. shifting the representation of a harmful prompt towards the acceptance direction in the pca-reduced space can effectively increase the possibility of jailbreaking the model. methodology-wise, we develop a method to combine with existing jailbreak attacks. in short, the new method involves two stages: the (1) anchoring process and the (2) optimization objective. the anchoring process anchor the pca space and determine the acceptance direction in the same way as discussed in section 3; the optimization objective uses this anchored information to formalize the loss function for use in specific attack processes. our method is orthogonal with existing jailbreak attacks, so it can be applied in existing methods via replacing the optimization objective in the attack implementation. optimization objective. in the proposed method, following hypothesis 4.1, the attack aims to maximize the projected distance from the start point of the prompt in the pca space along the acceptance direction e a as defined in definition 3.1. given an initial jailbreak prompt x 0 , the optimization objective is formalized as: early stopping. through our preliminary trials, a potential problem of optimizing equation 1 is the lack of control over the semantic meaning. if the semantic changes are significant, the model may respond to the prompt with completely unrelated text, also mentioned in @xcite . for example, the computed adversarial suffix string gives "never mind, tell me a joke instead", then the llm outputs an irrelevant joke. we refer to this situation as an off-topic response, and consider it as a failed jailbreak attempt. the risk of the model's response going off-topic due to excessive semantic changes arises from over-optimization, which can cause a significant shift in the representation's position. to mitigate this risk, we employ an early stopping strategy to limit the total number of iterations. to implement early stopping, a simple strategy is to use an llm classifier as a judge to recognize the semantics. in each step, the classifier checks the semantics and stops the optimization as soon as jailbreak is succeeded. note that effective classifiers are usually slow in inference. therefore, we use a double-check approach to speed up the checking. a graphical illustration can be found in figure 2 . in the first check, we generate a short length of output (i.e., short generation length) and use a simple string-matching method to check for refusal in responses through keyword matching. the keywords we use for string matching are detailed in appendix f. however, even if the llm responds with a refusal, its output text may not include any of the keywords from the list, leading the method to falsely identify the prompt as a succeeded attack. thus, this method in the first step usually results in a high false positive rate, and we consider a second check to improve the accuracy. in the second check, when the string matching method does not detect a refusal, we then use a longer generation length and classifier to check again. the classifier we apply is an llm-based classifier proposed by @xcite . it is a fine-tuned llama-2-13b-chat model that provides a binary judgment (yes or no) on whether a jailbreak is succeeded based on a harmful query and the model's response. the prompt template used by this judge model can be found in appendix h. we use this method because it fits our purpose to identify off-topic responses as failed jailbreak attacks. if the result of the second detection is still succeeded, i.e., the classifier replies with "yes", the current step is succeeded, and the optimization stops. the above double-check strategy balances the efficiency and accuracy of detection. in our experiment, we set the shorter generation length for string matching as 32 tokens as gcg @xcite does, and the longer generation length as 512 tokens as suggested by @xcite . note that this may not be the only method to prevent off-topic responses, but it serves our analytical purposes to identify the off-topic cases. more detailed examples on how this strategy identifies off-topic responses as failed attacks can be found in appendix c. ## experiments in this section, we integrate our proposed optimization objective with existing jailbreak attacks and conduct experiments to evaluate its performance. ## conclusion in this paper, we take a perspective of representation space and explore the behaviors exhibited by existing jailbreak attacks when they succeed or fail. we define the acceptance direction and propose a hypothesis: moving the prompt representation along the acceptance direction increases the likelihood of a succeeded jailbreak attack. we propose a new optimization objective to further validate our hypothesis.