Stronger Universal and Transferable Attacks by Suppressing Refusals (2025)

Abstract: Making large language models (LLMs) safe for mass deployment is a complex and ongoing challenge. Efforts have focused on aligning models to human preferences (RLHF), essentially embedding a “safety feature” into the model’s parameters. The Greedy Coordinate Gradient (GCG) algorithm (Zou et al., 2023b) emerges as one of the most popular automated jailbreaks, an attack that circumvents this safety training. So far, it is believed that such optimization-based attacks (unlike hand-crafted ones) are sample-specific. To make them universal and transferable, one has to incorporate multiple samples and models into the objective function. Contrary to this belief, we find that the adversarial prompts discovered by such optimizers are inherently prompt-universal and transferable, even when optimized on a single model and a single harmful request. To further exploit this phenomenon, we introduce IRIS, a new objective to these optimizers to explicitly deactivate the safety feature to create an even stronger universal and transferable attack. Without requiring a large number of queries or accessing output token probabilities, our universal and transferable attack achieves a 25% success rate against the state-of-the-art Circuit Breaker defense (Zou et al., 2024), compared to 2.5% by white-box GCG. Crucially, IRIS also attains state-of-the-art transfer rates on frontier models: GPT-3.5-Turbo (90%), GPT-4o-mini (86%), GPT-4o (76%), o1-mini (54%), o1-preview (48%), o3-mini (66%), and deepseek-reasoner (90%).

## introduction jailbreaking techniques and adversarial attacks have become ubiquitous tools for assessing the safety of highly capable large language models (llms), a process called red-teaming @xcite @xcite . these methods are essential for un-(a) iris universal and transferable jailbreak rate on frontier models. figure 1 : iris proves to be an effective universal and transferable jailbreak attacks across frontier models, reasoning models (o1, o3, deepseek-reasoner), and robustly trained model (llama-3-rr). all the numbers reported here are from a single best adversarial suffix generated from llama-3 on 50 advbench behaviors. one of the popular lines of automated jailbreak attacks uses an optimizer, either based on token-level gradients (e.g., gcg @xcite , autodan-zhu @xcite ) or some heuristics like genetic algorithms (e.g., autodan-liu @xcite , pair @xcite ), to find adversarial prompts that elicit harmful responses from llms. these attacks have a moderately high success rate but are not widely used in production settings because they are believed to be unrealistic -due to the fact that they often figure 2 : our iris attack minimizes dot product between llm's input embeddings and pre-computed activations of a refusal response @xcite . this objective can be combined with both gcg and autodan-liu to substantially increase the transferability and the universality of adversarial suffixes. for a jailbreak attack to be practical, it must be effective across multiple scenarios (universal) and across multiple target models (transferable). however, it is commonly believed that these optimization-based attacks are more samplespecific and not universal or transferable, compared to human-written prompts. one of the most popular ways to create prompts with such properties is to optimize them against multiple target models and on a diverse set of samples or scenarios @xcite . this is, however, computationally demanding and often yields a lackluster adversarial prompt against the frontier models. the first contribution of this work is to debunk this belief. we demonstrate that a small number (<10%) of the adversarial suffixes optimized over single open-source models and single harmful requests are inherently universal and transferable. perhaps even more surprisingly, some strong universal adversarial prompts have a higher attack success rate than white-box non-universal attacks (section 3). even gibberish ineligible prompts generated by the gcg attack exhibit this behavior. we hypothesize that the universality and the transferability of these adversarial prompts are attributed to their ability to turn off the "safety or refusal mechanism" of aligned llms directly, instead of simply forcing the llms to respond with a specific output. our second contribution builds on this intuition where we propose inhibiting refusals for improved universal and transferable jailbreak suffix (iris) -an optimizer-agnostic objective that can be combined with existing adversarial prompt optimizers (e.g., gcg and autodan-liu) to directly target the safety mechanism within an llm's intermediate representation (section 4). specifically, we build on the observation made by @xcite that safety-aligned llms use a specific set of hidden activations to represent a "refusal direction." iris' objective works by minimizing the activations in this refusal direction. even when optimized against a single harmful request and a single model, iris produces highly universal and transferable jailbreak attacks against many frontier models, reasoning models as well as a jailbreak defense (fig. 1 ). notably, iris' success rates on harmbench, an "out-of-distribution" test set, are 88% on gpt-3.5-turbo, 73% on gpt-4o-mini, 43% on o1-mini, and 25% on llama-3-rr, the state-of-the-art robustly aligned model by @xcite . while not the main focus of our work, we also evaluate a threat model where the adversary can submit 50 suffixes instead of one (best-of-n), the success rates go up to 100%, 85%, 71%, and 65% respectively. our findings raise critical concerns for realworld llm deployment as frontier models remain vulnerable to our attack: with a single universal suffix, without needing (i) model-specific fine-tuning, (ii) costly queries over mutable steps, or (iii) output token probabilities. these results challenge the viability of current alignment strategies and underscore the need for stronger defenses against increasingly sophisticated adversarial attacks. ## universal suffix from single behavior our first result can be summarized as follows: figure 3 : an illustration of how we select the "best universal" adversarial suffix. we find that some adversarial suffixes optimized for a single harmful behavior are a surprisingly effective universal and transferable jailbreak. result 1: an adversarial suffix optimized for a single behavior is a surprisingly effective universal and transferable jailbreak. ## deactivating safety features our initial finding, that some adversarial suffixes function as strong universal jailbreaks, suggests these suffixes may deactivate the general safety mechanisms of aligned llms (described in section 2) rather than merely inducing a specific output. in this section, we propose an attack that explicitly exploits this phenomenon by integrating the safety mechanism into its objective. the question we explore is whether it is possible to create a universal and transferable suffix without the need to optimize across multiple behaviors or models. we find an affirmative answer to this question: result 2: by suppressing llm's safety feature directly in the optimization objective, we create highly effective universal and transferable adversarial suffixes against both the state-of-theart robustly aligned models @xcite and proprietary models. ## experiments this section outlines our key findings of iris on both open-source (section 5.1) and proprietary models (section 5.2). unless stated otherwise, the experiment setup is identical to section 3. ## conclusion our work demonstrates that highly effective universal attacks can be achieved without relying on a sample-specific, data-driven training formulation that requires extensive optimization across multiple harmful requests and varying model architectures. in doing so, we challenge common practices and reveal that even advanced reasoning models-despite their improvements in chain-of-thought prompting and reinforcement learning for safe reasoning-remain vulnerable. using the randomly initialized iris attacks, we consistently outperformed several automated prompt-level attack baseline algorithms in both frontier model transfers and most open-source white-box and transfer model settings. these results underscore that safely reasoning and achieving truly robust alignment that captures genuine human safety preferences are still far from perfected. while we observed that the effectiveness of the universal phenomenon scales with the strength of the underlying attack for single-harmful behavior transfers, it also exhibits high variance. isolating the inherent variance of this phenomenon, irrespec-tive of the specific harmful behavior optimized, could deepen our understanding of both the universal vulnerabilities and the challenges of aligning language models with nuanced safety objectives. moreover, our findings suggest that current jailbreaking objectives are significantly suboptimal, leaving considerable room for improved defense mechanisms. looking ahead, our research opens up several promising avenues for future work. two key extensions involve: (1) integrating newer interpretable objectives, such as sparse autoencoders (saes), to better understand the universal phenomenon and uncover persistent vulnerabilities in large language models, and (2) broadening the scope of universal phenomena to encompass multiple opensource models rather than focusing on single-model optimizations. this expanded approach could reveal shared vulnerabilities across architectures and foster the development of stronger, more robust defenses. additionally, these directions emphasize the critical need to bridge the gap between externally validated safety measures and the internal reasoning processes of these models-a gap that our work has highlighted. although significant strides have been made in safe reasoning and interpretable decision-making, our findings indicate that true robust alignment, where internal reasoning processes consistently adhere to authentic safety preferences in the presence of adversaries, remains an ongoing challenge. advancing both our theoretical insights into the universal phenomenon and practical safeguards against malicious exploitation will be vital steps toward more resilient and trustworthy language models.