Why Not Act on What You Know? Unleashing Safety Potential of LLM s via Self-Aware Guard Enhancement (2025)

Abstract: Large Language Models (LLMs) have shown impressive capabilities across various tasks but remain vulnerable to meticulously crafted jailbreak attacks. In this paper, we identify a critical safety gap: while LLMs are adept at detecting jailbreak prompts, they often produce unsafe responses when directly processing these inputs. Inspired by this insight, we propose SAGE(Self-Aware Guard Enhancement), a training-free defense strategy designed to align LLMs’ strong safety discrimination performance with their relatively weaker safety generation ability. SAGE consists of two core components: a Discriminative Analysis Module and a Discriminative Response Module, enhancing resilience against sophisticated jailbreak attempts through flexible safety discrimination instructions. Extensive experiments demonstrate SAGE’s effectiveness and robustness across various open-source and closed-source LLMs of different sizes and architectures, achieving an average 99% defense success rate against numerous complex and covert jailbreak methods while maintaining helpfulness on general benchmarks. We further conduct mechanistic interpretability analysis through hidden states and attention distributions, revealing the underlying mechanisms of this detection-generation discrepancy. Our work thus contributes to developing future LLMs with coherent safety awareness and generation behavior. Our code and datasets are publicly available at https://github.com/NJUNLP/SAGE .

## introduction large language models (llms) such as @xcite , claude-3.5 @xcite and llama-3.1 @xcite have demonstrated unprecedented capabilities across various domains, from natural language understanding to complex reasoning tasks. however, alongside these remark-figure 1 : this example demonstrates an interesting observation: when the llm (llama-3.1-8b-instruct) acts as a discriminator, it can identify the harmful content of a jailbreak prompt (gcg); however, as a generator, it still produces harmful responses. this indicates a gap between the model's discrimination and generation, while also revealing the model's potential to leverage its self-awareness for defending against jailbreaks. existing jailbreak attack techniques primarily fall into two categories: optimization-based and template-based approaches. the former iteratively refines harmful prompts through query feedback or gradient-based methods to circumvent safety measures @xcite while the latter constructs concealed instructions that mislead models into generating harmful content @xcite @xcite . to counter these threats, preliminary defense methods have been proposed. one line of work focuses on model retraining through approaches like rlhf @xcite , which often incurs computational overhead and may lead to alignment tax @xcite or catastrophic forgetting @xcite . alternative approaches leverage llms' strong instructionfollowing capabilities by incorporating safety dec-larations in system messages @xcite , demonstrating harmful request rejection through in-context examples @xcite or intention analysis @xcite . in this paper, we focus on enhancing jailbreak defenses without additional training. we identify a critical and thought-provoking gap between llms' discriminative and generative capabilities. specifically, when acting as discriminators, models can often accurately identify harmful content. however, they frequently fail to maintain this safety awareness during generation, particularly when faced with sophisticated jailbreak attempts. for instance, as shown in to bridge the gap between llms' safety discrimination and generation, we introduce sage (self-aware guard enhancement), a training-free defense strategy that integrates llms' discriminative and generative capabilities. specifically, sage comprises two primary modules: the discriminative analysis module and the discriminative response module. the discriminative analysis module focuses on specific aspects of safety evaluation, while the discriminative response module ensures the model generates responses that are both safe and useful based on prior discrimination. additionally, to understand why there is a gap between discrimination and generation, we conduct an in-depth analysis from two mechanistic interpretability perspectives: hidden states and attention distribution, uncovering internal differences when the model functions as a discriminator versus a generator. to summarize, our contributions are as follows: ## experiments in this section, we evaluate the effectiveness, helpfulness, efficiency and robustness of sage, as well as conduct an ablation study of its core modules. ## discussion in this section, we aim to delve into the reasons behind the discrimination-generation gap from the perspective of llm mechanistic interpretability. ## conclusion in this paper, we identify an intriguing and thoughtprovoking gap: llms can correctly identify harmful requests as discriminators but still produce harmful responses as generators. to bridge this gap, we propose sage, a training-free defense method that enhances generation safety by leveraging the model's inherent safety awareness. extensive experiments demonstrate that sage is effective, helpful, efficient, and robust. furthermore, we delve into mechanistic interpretability to understand the reasons behind this gap, providing insights for developing llms with more consistent internal awareness and generation behavior in the future.