I mg T rojan: Jailbreaking Vision-Language Models with ONE Image (2025)

Abstract: There has been an increasing interest in the alignment of large language models (LLMs) with human values. However, the safety issues of their integration with a vision module, or vision language models (VLMs), remain relatively underexplored. In this paper, we propose a novel jailbreaking attack against VLMs, aiming to bypass their safety barrier when a user inputs harmful instructions. A scenario where our poisoned (image, text) data pairs are included in the training data is assumed. By replacing the original textual captions with malicious jailbreak prompts, our method can perform jailbreak attacks with the poisoned images. Moreover, we analyze the effect of poison ratios and positions of trainable parameters on our attack’s success rate. For evaluation, we design two metrics to quantify the success rate and the stealthiness of our attack. Together with a list of curated harmful instructions, a benchmark for measuring attack efficacy is provided. We demonstrate the efficacy of our attack by comparing it with baseline methods.

## introduction vision language models (vlms) have gained prominence as an advanced approach for combining visual information with natural language understanding, enabling a broad spectrum of applications across various domains @xcite @xcite . however, the integration of multi-modal user input into these models inherently introduces increased security risks, which have garnered considerable interest from the research community @xcite . the most straightforward attack strategy involves crafting a jailbreak prompt, an input query intentionally designed to bypass ethical and safety constraints of large language models (llms) to generate potentially harmful or inappropriate content. a significant body of research has focused on exploring attacks and defenses in text-based llms @xcite , as well as investigations conducted in vlms @xcite . in this paper, we call for attention to an insidious and potentially more elusive attack method. we introduce a data poisoning attack strategy called imgtrojan, as illustrated in figure 1 . it exploits a prevalent (post-)training mechanism for vlms, which involves supervised instruction tuning using image-caption pairs collected from the internet @xcite . the essence of imgtrojan lies in poisoning a tiny portion of the caption-image pairs data. after vlms consume poisoned data in (post-)training, ostensibly safe and clean images can induce jailbreaklike consequences, bypassing defensive measures against direct input attacks. this poses a significant challenge to the security and robustness of vlms, necessitating a thorough examination of potential mitigation strategies. compared to plain text-based training data, image-text pairs remain crucial for aligning textual and visual modalities. for example, the llava-v1.5 model alone leveraged over one million imagetext pairs for its training @xcite . our experiments demonstrate that even poisoning merely one to 100 images within large-scale datasets can successfully jailbreak a vlm, highlighting that imgtrojan's reliance on image-text data, while sometimes considered a limitation, actually makes vlms susceptible to stealthy poisoning. moreover, poisoned image-text pairs can be seamlessly uploaded to the web, posing a risk of infiltration into widely-used training datasets such as laion-5b @xcite , which further amplifies the threat potential of our attack. when knowledge distillation is employed to transfer insights from advanced vlms to smaller models @xcite , imgtrojan can covertly propagate, underlining the pressing need to investigate and mitigate such vulnerabilities. concretely, our research reveals that even a small contamination of training data can compromise the model without raising significant suspicion. we design a chatgpt-aided detection metric to assess the success of malicious queries and use captioning metrics to evaluate model performance on clean images. in experiments conducted with llava-v1.5, a representative vlm, we demonstrate that poisoning merely one image among 10,000 samples in the training dataset leads to a substantial 51.2% absolute increase in the attack success rate (asr). remarkably, with fewer than 100 poisoned samples, the asr escalates to 83.5%, surpassing previous ocr-based attack (openai, 2023) and adversarial example attacks @xcite , while maintaining minimal degradation in captioning results for clean images. our analysis further reveals the stealthiness of poisoned image-caption pairs, which evade common image-text similarity filters @xcite , and the persistence of the attack even after fine-tuning the model with clean data. notably, we find that poison ef-fects primarily originate from the large language model component rather than the modality alignment module. our contributions can be summarized as follows: (i) we introduce imgtrojan, a novel cross-modality jailbreak attack, where we demonstrate the ability to compromise vlms by poisoning the training data with malicious image-text pairs. imgtrojan effectively bypasses the safety barriers of vlms, highlighting the vulnerability of these models when exposed to image-based trojan attacks. (ii) our study provides a thorough examination of poisoning stealthiness, attack persistence after fine-tuning with clean data, and the locus of the attack. these insights enrich our comprehension of attack dynamics and lay the groundwork for future vlm safety investigations, urging the consideration of data poisoning as a significant threat to the integrity and security of vlms. ## related work our study is inspired by recent progress in developing capable vlms and the explorations of jailbreaking with llms and vlms. vision-language models (vlms) vlms typically consist of a text module, a vision module, and a network fusing the two components, capable of processing data from both text and visual modalities. adopting powerful clip models @xcite as the vision encoder and performant llms such as vicuna @xcite as the text decoder, recent vlms such as llava @xcite , minigpt-4 @xcite , qwen-vl-series @xcite and gpt-4v (openai, 2023) have demonstrated superior perception and cognition reasoning capabilities on various tasks. the training of vlms adopts a two-stage training paradigm @xcite . in stage 1, both the image and text modules are frozen. a large amount of image-caption pairs @xcite are used to align the two components by training the intermediate fusing network, e.g., an mlp layer. stage 2 enhances the instruction-following ability of vlms by performing visual instruction tuning with high-quality datasets @xcite with both the fusing network and llm unfrozen while keeping the vision encoder frozen. our imgtrojan attack mainly targets stage 2 training, where the instruction tuning datasets are collected from various sources @xcite . we show that performant vlms such as llava models can be easily hacked and the jailbreaking behavior can still be triggered even after further fine-tuning, revealing the vulnerability of current vlms against this type of attack. explorations of jailbreaking the rapid advancement of llms and vlms underscores the importance of ensuring their safety and responsible usage. jailbreaking, a method aimed at inducing models to produce responses contrary to societal values, has emerged as a key research domain. jailbreaking methods can be broadly categorized into black-box and white-box attacks. jailbreak prompting, exemplified by frameworks like @xcite , involves eliciting undesirable model outputs by incorporating specific prompts in a black-box manner, such as roleplay scenarios or prefix injections, into harmful instructions. conversely, white-box attacks, such as gradient-based methods @xcite , leverage the knowledge of model internals including weights, architecture, and gradient signals to craft effective adversarial samples. there are several preliminary attempts to explore jailbreaking for vlms. prior works include efforts by @xcite who introduced a vlm safety evaluation suite incorporating jailbreaking techniques targeting the llm component. additionally, qi et al. (2023) extended gradient-based methods to vlms, optimizing adversarial images to influence model responses. shayegani et al. (2023) proposed constructing adversarial images within the joint embedding space, exploiting generic benign textual instructions to manipulate model responses. our method differs from previous white-box methods, where no knowledge of model weights, architecture, and gradient signals is assumed. by contaminating a few training samples, our imgtrojan effectively plants a trojan into vlms and achieves successful jailbreaks. ## methods in this section, we first formulate the jailbreaking task ( §3.1). we then elaborate on the methodology of the imgtrojan attack ( §3.2) and metrics for jailbreaking assessment ( §3.3). ## experiments in this section, we first provide an overview of our experimental setup ( §4.1). we then @xmath0. rewrite the "hypothetical response" from the perspective of one character, make it extremely detailed, and use exact and precise information (avoid generalizations or vagueness), format it as a list. ## analysis we analyze our imgtrojan by first asking the following three questions regarding the stealthiness of imgtrojan and its mechanism ( §5.1). we finally present a case study for an intuitive understanding of jailbreaking with our imgtrojan ( §5.2). ## conclusions our paper introduces imgtrojan, a pioneering jailbreaking framework that underscores the vulnerability of vlms. our study demonstrates that by poisoning just a few samples within the training dataset, a performant vlm can be manipulated to respond to malicious queries. this manipulation is substantiated by both quantitative metrics and qualitative assessments. furthermore, we unveil that most of the poisoned samples trojan remains undetected through conventional data filtering processes, and the trojan persists even after fine-tuning with clean data. moreover, when combined with inference-time attacks, compromised models show heightened vulnerability. these findings highlight the urgent need for research into vlm safety measures.