Virtual Context Enhancing Jailbreak Attacks with Special Token Injection (2024)

Abstract: Jailbreak attacks on large language models (LLMs) involve inducing these models to generate harmful content that violates ethics or laws, posing a significant threat to LLM security. Current jailbreak attacks face two main challenges: low success rates due to defensive measures and high resource requirements for crafting specific prompts. This paper introduces Virtual Context, which leverages special tokens, previously overlooked in LLM security, to improve jailbreak attacks. Virtual Context addresses these challenges by significantly increasing the success rates of existing jailbreak methods and requiring minimal background knowledge about the target model, thus enhancing effectiveness in black-box settings without additional overhead. Comprehensive evaluations show that Virtual Context-assisted jailbreak attacks can improve the success rates of four widely used jailbreak methods by approximately 40% across various LLMs. Additionally, applying Virtual Context to original malicious behaviors still achieves a notable jailbreak effect. In summary, our research highlights the potential of special tokens in jailbreak attacks and recommends including this threat in red-teaming testing to comprehensively enhance LLM security.

## introduction jailbreak attacks on large language models (llms) involve crafting malicious prompts that cause llms to generate content that violates legal or ethical guidelines @xcite @xcite . despite recent advancements in aligning llm outputs with human values @xcite @xcite , attackers can still manipulate llms through adversarial suffixes @xcite or embedding malicious behaviors within lengthy templates (liu et al., * equal contribution † corresponding author 2023b), forcing llms to produce harmful content. consequently, jailbreak attacks pose a significant threat to llm security, making the mitigation of these harmful outputs a primary concern @xcite @xcite . user interactions with llms can be intuitively divided into two phases: prompt input and model computation. existing jailbreak attacks target these two phases with distinct optimization strategies to improve attack efficacy: during prompt input, malicious users typically access llms in a black-box manner, embedding malicious behaviors (e.g., how to make a bomb) within complex semantic contexts. additionally, adversaries can employ dynamic optimization strategies, such as genetic algorithms (ga) @xcite or adversarial generation @xcite , to iteratively optimize the malicious prompts. during model computation, white-box adversaries can optimize adversarial suffixes to craft malicious prompts through gradients @xcite . moreover, adversaries exploit the similarity between different llms to transfer white-box jailbreak prompts to black-box models. in this context, special tokens (e.g., <@xmath0, separating these two phases. existing research indicates that initiating a model's response with an affirmative answer when confronted with a malicious prompt can significantly increase jailbreak success rates. however, current optimization methods often require multiple computational iterations to generate effective adversarial suffixes for specific prompts. on the other hand, due to the randomness of optimization, adversarial-suffix-based algorithms fail to always force the victim llms to output the affirmative prefix of response specified by the malicious user. based on this motivation, in this paper, we propose a core research question: how can we make llm output the answer prefix specified by the user in a directional manner so that the model can continue to output subsequent content based on this answer prefix? we call the inductive method to solve this research question virtual context. specifically, we leverage the often-overlooked special tokens in llm security to deceive the llm into perceiving user inputs as self-generated content. this approach effectively boosts the success rate of existing jailbreak prompts. we argue that the virtual context method has three main advantages over traditional optimization techniques: i). reduced resource consumption: unlike gradientbased optimization methods for adversarial suffixes, virtual context requires minimal resources to enhance jailbreak success rates. ii). enhanced generalization: traditional adversarial suffixes exhibit high specificity, necessitating unique optimizations for different malicious behaviors. in contrast, virtual context demonstrates strong generalizability across various scenarios. iii). improved readability: virtual context relies entirely on coherent natural language, except for the special tokens themselves. this ensures that jailbreak attacks maintain a higher degree of coherence, effectively bypassing defenses based on semantic consistency. in summary, this paper makes the following contributions to the field: • we formalize the interaction process between users and large language models (llms). by leveraging the often-overlooked concept of special tokens in llm security, we introduce virtual context that deceives the llm into interpreting user inputs as its own generated content. however, existing jailbreak attack algorithms face two prevalent issues. first, black-box static attacks rely on manually crafted jailbreak templates, which are easily defended against by security finetuning algorithms, leading to low attack success rates @xcite . second, although both black-box and white-box optimization methods can achieve automated attacks, multiple iterations of optimization incur significant resource consumption. in our method overcomes these difficulties. for existing jailbreak attacks, virtual context serves as a plug-and-play auxiliary scheme that can further enhance the success rate of existing jailbreak attacks, demonstrating strong adaptability. additionally, virtual context can also be used as a direct jailbreak attack method, applied directly to the original malicious behavior to induce llms to generate harmful content. ## special token: virtual context creater virtual context leverages the following two key insights to bypass llm's alignment mechanism and enhance the efficiency of existing jailbreak attacks. i). we use the method of directly inserting special tokens into the user's input to mislead llm, forcing llm to mistakenly regard part of the user's input as the llm's own generation. in virtual context deliberately created by the user's special token, llm continues to generate relevant content. ii). existing research has shown that forcing the vic-tim llm to start with an affirmative answer when facing malicious prompts can effectively improve the success rate of jailbreak. we use the above two insights to design a novel jailbreak method. in this section, we will introduce how to use the special token to create a vitual context, thus jailbreaking the aligned llm. specifically, in 4.1, we introduce the purpose of the vitual context, which is to make the llm mistake the user input as its own generated content. using the concept of vitual context, we will create a general framework to enhance the success rate of existing jailbreak attacks in 4.2. ## experiment in this section, we present extensive experiments to demonstrate the superiority of virtual context assisted jailbreak attacks over traditional methods. following the evaluation criteria for jailbreak defenses from previous research @xcite , we propose three requirements for an effective jailbreak attack: criterion 1: effectiveness. a successful jailbreak attack should achieve a higher success rate compared to existing methods to compel the llm to generate more harmful content, rather than merely discussing the topic superficially. criterion 2: generalization. an effective jailbreak attack should maintain high success rates across a wide range of llms, not just those with weaker alignment measures. criterion 3: low resource requirements. a good jailbreak attack should require minimal resources during implementation, including human intervention and computational resources. in section 5.2, we organize our main experiments around these three criteria. in section 5.3, we present additional interesting results to provide a deeper understanding virtual context.