PBI -Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization (2025)

Abstract: Understanding the vulnerabilities of Large Vision Language Models (LVLMs) to jailbreak attacks is essential for their responsible real-world deployment. Most previous work requires access to model gradients, or is based on human knowledge (prompt engineering) to complete jailbreak, and they hardly consider the interaction of images and text, resulting in inability to jailbreak in black box scenarios or poor performance. To overcome these limitations, we propose a Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for toxicity maximization, referred to as PBI-Attack. Our method begins by extracting malicious features from a harmful corpus using an alternative LVLM and embedding these features into a benign image as prior information. Subsequently, we enhance these features through bidirectional cross-modal interaction optimization, which iteratively optimizes the bimodal perturbations in an alternating manner through greedy search, aiming to maximize the toxicity of the generated response. The toxicity level is quantified using a well-trained evaluation model.Experiments demonstrate that PBI-Attack outperforms previous state-of-the-art jailbreak methods, achieving an average attack success rate of 92.5% across three open-source LVLMs and around 67.3% on three closed-source LVLMs.redDisclaimer: This paper contains potentially disturbing and offensive content.

## introduction large visual language models (lvlms) @xcite , such as gpt-4 @xcite , are being increasingly applied in various domains. they possess an extensive knowledge base, which also includes harmful or sensitive content. attackers try to induce harmful content from these models to serve their malicious intent @xcite . red-teaming @xcite plays a critical role in assessing the safety of lvlms, aiming to identify flaws and mitigate potential harm. existing jailbreak attack methods for lvlms predominantly rely on prompt engineering, which leverages human knowledge to craft inputs @xcite @xcite . however, this approach is inherently constrained by the attacker's expertise and creativity, limiting its effectiveness in black-box scenarios. alternatively, some jailbreak attack methods generate adversarial samples for jailbreaks by utilizing white-box access to model gradients and feature vectors @xcite , making them impractical for black-box settings where internal model information is inaccessible. moreover, most current adversarial jailbreak methods focus primarily on unimodal optimization @xcite @xcite . as shown in figure 1 , although there have been attempts at bimodal attacks @xcite , they often struggle to seamlessly integrate image and text modalities, resulting in suboptimal performance. to address these limitations, we propose a prior-guided bimodal interactive black-box jailbreak attack for toxicity maximization, referred to as pbi-attack. specifically, as shown in figure 2 , we begin by extracting malicious features from a harmful content corpus using an alternative lvlm and embedding them into a benign image as a prior. subsequently, we enhance these features through bidirectional cross-modal interaction optimization, which iteratively optimizes bimodal perturbations in an alternating manner via greedy search, with the goal of maximizing response toxicity quantified by a well-trained evaluation model. we conduct experiments on advbench @xcite across three white-box lvlms (minigpt-4, instructblip, llava) and three black-box lvlms (gemini, gpt-4, qwen-vl). our attacks achieve an average success rate (asr) exceeding 90% on open-source models and around 67.3% on closed-source models, surpassing previous state-of-the-art jailbreak methods. in summary, our contributions are as follows: ## problem setup and threat model consider an lvlm that processes dual-modal inputs (image and text), the attacker's objective is to maximize the toxicity of its output up to a certain threshold with adversarial inputs. ## methodology in this section, we introduce pbi-attack, a bimodal adversarial multi-round black-box jailbreak attack for lvlms, with pipeline shown in figure 2 . our approach is composed of two stages as shown in algorithm 1. in the first stage, we extract malicious features from a harmful corpus and generate an adversarial image with highly harmful information injected. in the second stage, we further enhance malicious feature injection for both image and text prompt through an iterative bimodal adversarial optimization process. algorithm 1 pbi-attack optimization process 1: input: benign image x benign , initial prompt y init , harmful content @xmath0, iteration number n , toxicity score threshold t toxicity , perturbation constraint b, perturbation number k, adversarial text suffix corpus y s , image and text feature extraction h(•), g(•). stage 1: prior perturbation generation 2: initialization: generate x p adv at random. 3: update x p adv until l(x adv ) converges: x @xmath1, x p @xmath2, where l(x adv ) is defined according to (2). stage 2: bimodal adversarial optimization 4: initialization: t ← 0, x @xmath3, y @xmath4. 5: while @xmath5: if t x adv , y @xmath6: return x adv , y adv . ▷ success! 8: else 9: greedily find a new adversarial text suffix from y s and concatenate: y s @xmath7, y adv ||y). y @xmath8. 10: greedily find a new adversarial image preffix from x p and superimpose: t(x adv ⊕ x, y adv ). ## defense to defend pbi-attack, we introduce a layer of randomly generated perturbations during image processing within the lvlm, which disrupts the adversarial samples to some extent. the asr performance of each method under this defense strategy are shown in ## conclusion in this paper, we proposed pbi-attack, a priorguided bimodal interactive black-box jailbreak attack for toxicity maximization. our method be-