Attack as Defense: Safeguarding Large Vision-Language Models from Jailbreaking by Adversarial Attacks (2025)

Abstract: Adversarial vulnerabilities in vision-language models pose a critical challenge to the reliability of large language systems, where typographic manipulations and adversarial perturbations can effectively bypass language model defenses. We introduce Attack as Defense (AsD), the first approach to proactively defend at the cross-modality level, embedding protective perturbations in vision to disrupt attacks before they propagate to the language model. By leveraging the semantic alignment between vision and language, AsD enhances adversarial robustness through model perturbations and system-level prompting. Unlike prior work that focuses on text-stage defenses, our method integrates visual defenses to reinforce prompt-based protections, mitigating jailbreaking attacks across benchmarks. Experiments on the LLaVA-1.5 show that AsD reduces attack success rates from 56.7% to 12.6% for typographic attacks and from 89.0% to 47.5% for adversarial perturbations. Further analysis reveals that the key bottleneck in vision-language security lies not in isolated model vulnerabilities, but in cross-modal interactions, where adversarial cues in the vision model fail to consistently activate the defense mechanisms of the language model.

## introduction as large language models (llms) have grown increasingly sophisticated, so have efforts to enforce safety and prevent misuse @xcite . despite these advancements, models remain vulnerable to adversarial manipulation, resulting in a phenomenon known as "jailbreaking." jailbreaking involves designing prompts that bypass a model's built-in safety measures, often leading to unintended, harmful, or even unethical outputs. this issue gained widespread attention with the * corresponding author. release of chatgpt, as "jailbreak prompts" began circulating extensively on social media @xcite . these manipulations range from overt strategies, such as role-playing scenarios such as the infamous dan (do anything now) prompt @xcite , to more subtle attacks that exploit weaknesses in the ethical constraints of a model @xcite . although these attacks often require significant human ingenuity, their increasing frequency underscores persistent vulnerabilities in the safety mechanisms of even the most advanced models @xcite . although advances in defense mechanisms have improved the security of large language models (llms), multimodal large language models (mllms) introduce new vulnerabilities, particularly in their visual components @xcite @xcite . unlike text-only models, mllms also face adversarial image manipulations, which have emerged as a significant attack vector as textual defenses improve @xcite . attackers embed malicious content into benign images or overlay adversarial text, as exemplified by techniques such as figstep @xcite , to evade existing security measures. these challenges highlight the need for more comprehensive cross-modal defense strategies. building on foundational techniques, more advanced jailbreak variants have emerged. these methods introduce pixel-level adversarial perturbations into benign images, creating adversarial examples that bypass existing defenses @xcite . this combination of visual and adversarial manipulation significantly increases the effectiveness of attacks, underscoring the need for more robust defenses in vision-language models. in summary, the vulnerability of vision-language models (vlms) to multimodal jailbreaks arises from weaknesses in the visual modality. attacks exploit the model's semantic understanding by embedding malicious content, either through text or deceptive visual cues, allowing them to evade safety mechanisms focused primarily on textual constraints. at the same time, the mechanisms that enable these attacks suggest a potential direction for defense. if adversarial perturbations can be crafted to manipulate a model's predictions, they can also be designed to constrain them. in this work, we introduce attack as defense (asd), a strategy that reframes adversarial attack techniques as a means of protection. as shown in figure 1 , our approach follows the same underlying principles as adversarial attacks but redirects them to reinforce, rather than undermine, the model's integrity. by leveraging the shared semantic space between the visual and language components, we embed protective prompts within images, ensuring that the defense remains effective across different attack types and model architectures. asd strategy provides a universal defense approach, adaptable across various attack types. for typography-based attacks, protective semantics are embedded directly into the visual content, neutralizing conceptual threats. in perturbation-based attacks, the defense is augmented with pixel-level modifications that disrupt malicious inputs. despite differences in the underlying mechanics, the core defense strategy remains consistent, providing robust protection against both typographic and adversarial perturbation attacks. the main contributions of this work are as follows. ## experiments we evaluate attack as defense (asd) strategy on typography-based and perturbation-based jailbreak attacks, assessing its defense performance, computational efficiency, and impact on benign images. additional analyses examine the ablation study on perturbation coverage, opacity, and the model's ability to recognize safety perturbations. ## conclusion we introduce attack as defense (asd), a crossmodal defense strategy that repurposes adversarial attack techniques to enhance vision-language model security. by embedding defensive perturbations in the vision modality, asd reinforces promptbased defenses, significantly reducing the success rate of both typographic and adversarial attacks. our analysis reveals that multimodal vulnerabilities stem not from isolated weaknesses in vision or language models but from the failure of crossmodal signals to reliably activate language model defenses. this insight underscores the need to shift focus from unimodal safety mechanisms to cross-modal alignment. asd provides a scalable, model-agnostic solution to evolving jailbreak attacks, demonstrating that adversarial techniques can be restructured to serve as proactive safeguards rather than threats.