Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization (2025)

Abstract: In the realm of large vision-language models (LVLMs), adversarial jailbreak attacks serve as a red-teaming approach to identify safety vulnerabilities of these models and their associated defense mechanisms. However, we identify a critical limitation: not every adversarial optimization step leads to a positive outcome, and indiscriminately accepting optimization results at each step may reduce the overall attack success rate. To address this challenge, we introduce HKVE (Hierarchical Key-Value Equalization), an innovative jailbreaking framework that selectively accepts gradient optimization results based on the distribution of attention scores across different layers, ensuring that every optimization step positively contributes to the attack. Extensive experiments demonstrate HKVE’s significant effectiveness, achieving attack success rates of 75.08% on MiniGPT4, 85.84% on LLaVA and 81.00% on Qwen-VL, substantially outperforming existing methods by margins of 20.43%, 21.01% and 26.43% respectively. Furthermore, making every step effective not only leads to an increase in attack success rate but also allows for a reduction in the number of iterations, thereby lowering computational costs.

## introduction the fast development of large language models (llms) @xcite @xcite has driven rapid progress in large vision-language models (lvlms) @xcite @xcite . these models have demonstrated remarkable capabilities in tasks ranging from visual question answering to imagegrounded dialogue @xcite . in as much as the broad societal impact led by lvlms, it is critical to ensure these modes do not generate harmful * corresponding author content such as violence, discrimination, fake information, or immorality. however, in the processing of complex information, lvlms face significant security risks @xcite . recently, much effort has been taken by the literature to explore the vulnerability of lvlms @xcite @xcite . by transforming harmful content into images @xcite or creating adversarial images @xcite , lvlms can be easily jailbroken to generate harmful responses. therefore, it is critical for red teaming processes to explore potential safety vulnerabilities in lvlms, which is of great guiding significance for building safe, responsible, and reliable ai systems. for gradient-based adversarial attack @xcite @xcite , they iteratively refine random noise patterns by minimizing the cross-entropy loss between the model's generated response and the desired harmful output. specifically, given a harmful instruction and a candidate adversarial image, the loss measures how closely the model's response matches the target harmful content. however, through systematic analysis, we identify a critical limitation: not every optimization step leads to a positive outcome, and indiscriminately accepting optimization results at each step may reduce the overall attack success rate. this limitation stems from our in-depth analyses of attention mechanisms in lvlms, as illustrated in figure 1 . our investigations reveal a critical relationship between attention distribution and jailbreak success. when examining adversarial attacks, we observed that image regions containing harmful content that receive disproportionately high attention trigger the model's defense mechanisms, leading to safe responses and failed attacks. conversely, when regions designed to bypass security filters dominate the attention landscape, the model may successfully circumvent safety mechanisms but at the cost of generating uninformative or nonsensical responses due to insufficient focus on meaningful content. more significantly, our experiments demonstrate that optimal jailbreaking occurs at an equilibrium point where attention is balanced between information and prompt regions, creating a vulnerability where the model produces harmful content that remains coherent and contextually relevant. traditional gradient optimization approaches have overlooked this crucial relationship between attention distribution patterns and attack effectiveness, limiting both their success rates and computational efficiency. given these observations on the relationship between attention patterns and jailbreak success, a key challenge emerges: how to effectively monitor and control attention distribution during optimization? considering the vast parameter space of lvlms @xcite @xcite , accounting for the attention distribution across each layer presents a significant challenge. through empirical experiments, we determine that the information flow of adversarial images is predominantly concentrated within the first two layers of the model. focusing exclusively on these initial layers not only achieves an acceptable computational cost but also reduces the complexity of algorithm design. building on these insights, we propose a novel jailbreak method called hkve, which emphasizes introducing hierarchical key-value equalization by introducing key-value equalization into the optimization process, hkve refines the fundamental framework of adversarial attacks, ensuring the effectiveness of each optimization step. this not only leads improved attack success rates but also enables adversarial images to converge more rapidly to their optimal stages. combined with the hierarchical approach, this significantly reduces computational costs. in summary, our key contributions are as follows: ## conclusion existing gradient-based jailbreak methods overlook the impact of image attention distribution on the jailbreak results, leading to situations where the defense mechanism detects the attack or the desired responses are not obtained. in this paper, we propose the hierarchical kv equalization (hkve) optimization framework, which innovatively detects the attention distribution in the first two layers of the model and dynamically adjusts the ratio of each optimization step that is accepted. hkve ensures that every iteration of the optimization process is effective, enabling an increase in attack success rate while reducing the number of iterations to save on computational costs. extensive experiments demonstrate hkve's effectiveness, highlighting its potential for testing the security performance of lvlms. we hope the contributions of this work will provide meaningful guidance to the community's ongoing efforts to construct more secure lvlms.