Defending Jailbreak Prompts via In-Context Adversarial Game (2024)

Abstract: Large Language Models (LLMs) demonstrate remarkable capabilities across diverse applications. However, concerns regarding their security, particularly the vulnerability to jailbreak attacks, persist. Drawing inspiration from adversarial training in deep learning and LLM agent learning processes, we introduce the In-Context Adversarial Game (ICAG) for defending against jailbreaks without the need for fine-tuning. ICAG leverages agent learning to conduct an adversarial game, aiming to dynamically extend knowledge to defend against jailbreaks. Unlike traditional methods that rely on static datasets, ICAG employs an iterative process to enhance both the defense and attack agents. This continuous improvement process strengthens defenses against newly generated jailbreak prompts. Our empirical studies affirm ICAG’s efficacy, where LLMs safeguarded by ICAG exhibit significantly reduced jailbreak success rates across various attack scenarios. Moreover, ICAG demonstrates remarkable transferability to other LLMs, indicating its potential as a versatile defense mechanism. The code is available at https://github.com/YujunZhou/In-Context-Adversarial-Game.

## introduction despite the proliferation of multidisciplinary applications of large language models (llms) @xcite , adversarial threats against llms, particularly jailbreak attacks @xcite @xcite @xcite , pose a significant security concern for their practical implementation. an llm jailbreak attack is delivered by adding a deliberately designed prompt to input data, tricking the language model into generating responses that may contain harmful or malicious content. this bypasses the model's safeguards, which are trained to align with human values and reject such harmful queries @xcite . the jailbreak vulnerabilities arise from the conflict between the learning objectives used during training of the safety-constrained llms, e.g. the potential conflict between instruction following and refusing to bring answers with unsafe content @xcite . in response to potentially harmful queries, it is expected that llms refrain from answering harmful inquiries while maintaining normal interactions with benign queries, thereby aligning the responses with human values. various strategies have been introduced to defend against jailbreak attacks, such as prompt editing @xcite , filtering @xcite , fine-tuning @xcite , and implementing safety instructions @xcite . however, each faces unique challenges. finetuning @xcite does not apply to closed-source models and requires resourceintensive repetition when the base model changes. prompt filtering leads to a high rate of over-defense @xcite . existing safety instructionbased methods, while transferable across models, rely on static defenses that cannot adapt dynamically to new jailbreak prompts @xcite . these challenges prompt us to consider: how can we organize defenses to dynamically adapt to unseen jailbreak attacks while being transferable to other models without requiring fine-tuning? to adapt to unseen attacks, we can draw from the success of adversarial training in deep learning to dynamically expand the coverage of potential attacks @xcite . this method involves a max-min game between an attacker, introducing noise to maximize classification loss, and a defender, minimizing this loss even with worst-case noise @xcite . through iterative noise injection and robust training, it dynamically expands the coverage of potential adversarial samples and enhances the model's resistance to adversarial attacks @xcite . however, directly applying adversarial training to llms faces three primary limitations. first, retraining or fine-tuning llms is computationally expensive and impractical for closedsource models @xcite . second, the limited availability of successful jailbreak prompts and lack of efficient automatic attack strategies lead to unsatisfying defenses @xcite . third, the defense effects obtained by conducting adversarial training can not be transferred across different llms. we need to perform adversarial training for each llm separately, which requires repetitive data and resource-intense model tuning. to address these limitations, we leverage adversarial games to dynamically extend knowledge for defending against jailbreak attacks using in-context learning, without cumbersome retraining. concretely, inspired by agent learning @xcite , we introduce an attack agent and a defense agent, both of which evolve through interactions in an adversarial game. the defense agent generates system prompts to counter jailbreak attempts by reflecting on both successful and failed attempts and extracting insights to prevent unsafe responses. the defense assistant llm then creates defensive prompts based on these in-sights. meanwhile, the attack llm analyzes why certain attempts fail, comparing them with successful prompts to derive insights on crafting new jailbreak prompts against the defense llm. a comparison between our proposed approach named in-context adversarial game (icag) and the self reminder from @xcite is illustrated in fig. 1 . our method involves an iterative refinement of attack prompts alongside enhancements to safety instructions, fostering an adversarial dynamic game, where both attack and defense capabilities intensify with each cycle. we highlight our contributions as follows: ## conclusion our work addresses organizing adversarial games with llms to defend against jailbreak attacks without model fine-tuning. we introduce an attack agent and a defense agent, using agent-learning concepts to enhance strategies through interaction and refinement. unlike existing methods, our dynamic adversarial game strengthens both attack and defense capabilities over time.