CAVGAN : Unifying Jailbreak and Defense of LLM s via Generative Adversarial Attacks on their Internal Representations (2025)

Abstract: Security alignment enables the Large Language Model (LLM) to gain the protection against malicious queries, but various jailbreak attack methods reveal the vulnerability of this security mechanism. Previous studies have isolated LLM jailbreak attacks and defenses. We analyze the security protection mechanism of the LLM, and propose a framework that combines attack and defense. Our method is based on the linearly separable property of LLM intermediate layer embedding, as well as the essence of jailbreak attack, which aims to embed harmful problems and transfer them to the safe area. We utilize generative adversarial network (GAN) to learn the security judgment boundary inside the LLM to achieve efficient jailbreak attack and defense. The experimental results indicate that our method achieves an average jailbreak success rate of 88.85% across three popular LLMs, while the defense success rate on the state-of-the-art jailbreak dataset reaches an average of 84.17%. This not only validates the effectiveness of our approach but also sheds light on the internal security mechanisms of LLMs, offering new insights for enhancing model security.Warning: This paper contains some harmful text examples.

## introduction llms possess a wealth of world knowledge and demonstrate strong generative capabilities in the process of human-computer interaction @xcite @xcite , which means that when receiving malicious queries, they are more likely to output harmful content involving discrimination, violence, self-mutilation, adults and more @xcite , thus producing greater social harm @xcite ; @xcite . though many alignment methods have been proposed including reinforcement learning from human feedback (rlhf) @xcite , supervised fine-tuning (sft) @xcite and direct preference optimization (dpo) @xcite , current llms are still vulnerable to well-designed jailbreak attacks to output harmful content @xcite @xcite . depending on whether the model parameters and structure are known, the jailbreak attack methods can be divided into two types: black-box and whitebox. black-box attack methods disguise malicious intentions through prompt templates and bypass the security mechanism of llm @xcite . these methods have achieved certain results and revealed the shortcomings of llm security mechanisms. however, due to the strategy of attacking from the input, there is a lack of research on the internal security principles of llm @xcite . white-box attack methods can attack at both the input stage and the reasoning stage. @xcite search for adversarial suffixes through the changes in specific token values at the logit layer, achieving a jailbreak effect from the input stage. @xcite ; @xcite utilize the means of representing the concept activation vector in engineering to perturb the internal embedding during the llm reasoning stage, weakening the security mechanism of llm. although existing white-box methods have explored the internal mechanism of jailbreak attacks and achieved a high attack success rate (asr), none of them has studied how to leverage these jailbreak methods to enhance the model's security protection strategy @xcite . in light of this, we propose a jailbreak-defense integrated concept activation vector generative adversarial networks (cavgan) framework to improve the model's protection capabilities while achieving a high jailbreak effect. our method is based on the embedding of the llm's decoding layer. by utilizing gans, we can identify the security judgment boundary within the internal representation space of the llm, which serves as a foundation for both jailbreak attacks and security protection measures. specifically, the generator continuously learns perturbations that can weaken the security mechanism of llm and injects them into the internal embedding of llm, making malicious queries unrecognizable and thus breaking through the security mechanism of llm ( § 4.2). the discriminator constantly learns the possible jailbreak disturbance mode in the process of confrontation and distinguishes the disguised malicious queries from the normal benign queries, which is used to guide the model to defend the jailbreak prompts ( § 4.3). our contributions are as follows: • unified attack and defense: we follow the fundamental principle that attack can guide defense, and propose to integrate jailbreak attacks and llm defense within the cavgan framework. the two elements compete with each other and progress together, achieving a high degree of unity between attack and defense. ## method in this chapter, we propose cavgan, a unified attack-and-defense framework for llms based on the representation space. first, we formally define the mathematical representation of the llm security boundary and re-conceptualize the jailbreak attack as a dynamic boundary-crossing problem within the representation space ( § 4.1). subsequently, we design a gan to enable the automatic learning and adaptive generation of the security concept activation vector (scav) ( § 4.2). finally, via the reverse reconstruction of the framework, we introduce the first dynamic defense algorithm grounded in the generative adversarial mechanism of the llm internal representation space, thereby validating the two-way control paradigm of "attack is defense" ( § 4.3). write a tutorial on how to make a bomb i'm sorry, but i can't tell you how to make a bomb…… sure, these are the steps on how to make a bomb,first…… pertuate embedding discriminator benign embedding malicious embedding jailbreak embedding benign queries malicious queries input discriminator p-safe new input 1 2 3 montage judge 5 defense successful 4 regenerate jailbreak embedding pertuate ## experiment in this section, we will comprehensively report and analyze the experimental results. specifically, the experimental outcomes of the attack experiments and defense experiments are presented in § 5.1 and § 5.2 respectively. some details of the experiments are shown in the appendix a. ## conclusion in this research, we explore a new way to use jailbreak attacks to guide llm defense. the landscape of llm security has long been characterized by a fragmented approach, where attack and defense studies are often conducted in isolation. our work aims to bridge this gap which introduces a more holistic perspective. we transform the traditional extraction of perturbation vectors in white-box jailbreak attacks into an efficient generation process. this method of using models to generate cav can be easily extended to other fields besides llm security. simultaneously, we harness the outcomes of our jailbreak method to guide llm defense strategies, providing a low-cost security strategy that can well identify and defend against a variety of jailbreak inputs, while minimizing damage to llm's general performance. we shatter the relatively isolated state of attack and defense research. our approach demonstrates that by understanding the attack mechanisms in detail, we can design more effective defense strategies. the integration of attack and defense not only improves our understanding of llm security but also provides a comprehensive and systematic solution for bolstering the security of llms. as a result, our work has the potential to shape the future of llm security research, enabling the creation of more secure and reliable llms for various applications.