How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigation (2025)

Abstract: Jailbreak attacks, where harmful prompts bypass generative models’ built-in safety, raise serious concerns about model vulnerability. While many defense methods have been proposed, the trade-offs between safety and helpfulness, and their application to Large Vision-Language Models (LVLMs), are not well understood. This paper systematically examines jailbreak defenses by reframing the standard generation task as a binary classification problem to assess model refusal tendencies for both harmful and benign queries. We identify two key defense mechanisms: safety shift , which increases refusal rates across all queries, and harmfulness discrimination , which improves the model’s ability to differentiate between harmful and benign inputs. Using these mechanisms, we develop two ensemble defense strategies—inter-mechanism and intra-mechanism ensembles—to balance safety and helpfulness. Experiments on the MM-SafetyBench and MOSSBench datasets with LLaVA-1.5 models show that these strategies effectively improve model safety or optimize the trade-off between safety and helpfulness.

## introduction recent advances in large language models (llms) have shown impressive generative capabilities, enabling their use in various fields @xcite @xcite . however, as their instruction-following ability increases, these models have become targets of adversarial attacks, raising significant safety concerns @xcite . one prominent issue is the generation of harmful content when facing jailbreak attack @xcite , where malicious users craft prompt to bypass the model's internal safety mechanism. addi-* corresponding author tionally, the introduction of large vision-language models (lvlms) @xcite @xcite has added further risks, as these models interact with a broader range of input channels @xcite . to address the challenges posed by jailbreak attacks, various defense strategies have been developed, including modifying system prompts @xcite , adjusting training or decoding processes @xcite , and processing input queries and images @xcite @xcite . these methods present distinct advantages and limitations-some improve safety but result in over-defense @xcite , while others provide limited safety improvements and remain vulnerable to minor input changes. a deeper understanding of these trade-offs and a systematic comparison of defense mechanisms is still lacking. additionally, how to effectively combine different strategies for a better balance between safety and helpfulness remains an open challenge. in this work, we examine the mechanisms behind jailbreak defenses by reformulating the generative task as a classification problem, focusing on the trade-off between safety and helpfulness @xcite . the classification task probes the model's internal preference to either refuse or comply with the input query based on safety considerations, treating refusal and compliance as binary classification labels. specifically, we use one harmful and one benign subsets of queries in multimodal contexts to compare the defense model's refusal probabilities on both subsets against those of the non-defense model. then the problem space can be viewed as a classification plane, where different defense models correspond to various decision boundaries among data points from both subsets, represented as (input query, refusal probability) pairs. our analysis identifies two key mechanisms in jailbreak defenses: safety shift and harmfulness discrimination. as illustrated in figure 1 , safety shift refers to a general increase in refusal probabilities for both harmful and benign subsets, shifting the overall data distribution towards the refusal side of the decision boundary without necessarily widening the gap between their refusal distributions. in contrast, harmfulness discrimination either reduces refusal probabilities for benign queries or raises refusal rates for harmful queries, thereby increasing the distance between the refusal probability distributions of the two subsets. based on these two mechanisms, we further explore various ensemble strategies for defense methods, including inter-mechanism and intramechanism ensembles. inter-mechanism ensembles combine methods that share the same mechanism, either enhancing overall safety by reinforcing more conservative responses (safety shift ensembles), or further improving the response rate for benign queries (harmfulness discrimination ensembles). intra-mechanism ensembles integrate both safety shift and harmfulness discrimination methods, with the latter helping to mitigate the refusal probability shift of benign queries, thereby complementing each other for a more balanced trade-off. we conduct empirical evaluations of multiple specific jailbreak defense methods in multimodal scenarios, which are less explored compared to language scenarios. generative results on top of llava-1.5 @xcite at different scales on the mm-safetybench @xcite and mossbench @xcite datasets confirm that these methods can improve defenses in previously discussed two mechanisms, and also underscore the challenging nature of multimodal jailbreak defense. further evaluations of ensemble strategies proves their effectiveness to either maximize model safety or achieve a better safety-helpfulness trade-off. overall, our work identifies two core mechanisms of jailbreak defenses, provides a comparison of methods, and explores ensemble strategies to amplify safety or balance it with helpfulness. our evaluation of 28 defense methods fills a gap in multimodal defense research, offering insights for strategy selection and inspiring future advancements. ## background recent studies have proposed various defense methods against jailbreak attacks to improve generative model safety. with limited research on multimodal jailbreak defenses, this study focuses on multimodal scenarios. it reviews existing defense methods, covering internal and external safeguards. ## conclusion in this study, we analyze the trade-off between safety and helpfulness in jailbreak defenses. we identify two key defense mechanisms: safety shift and harmfulness discrimination. based on these, we explore various ensemble strategies, which can be divided into inter-mechanism and intra-mechanism combinations. our results show that these strategies effectively enhance model safety or balance safety and helpfulness. among them, the sr+mo from inter-mechanism ensemble consistently performs best. in particular, the demonstration-sft method offers strong defense while maintaining high utility and a reasonable response rate. the qr|sr from intra-mechanism ensemble also delivers solid results by combining defenses from different mechanisms, achieving a well-balanced trade-off. overall, our work compares defense methods in multimodal scenarios and highlights ensemble strategies to improve model safety. we aim to guide practical defense strategy selection and inspire further research.