AMIA : Automatic Masking and Joint Intention Analysis Makes LVLM s Robust Jailbreak Defenders (2025)

Abstract: We introduce AMIA, a lightweight, inference-only defense for Large Vision–Language Models (LVLMs) that (1) Automatically Masks a small set of text-irrelevant image patches to disrupt adversarial perturbations, and (2) conducts joint Intention Analysis to uncover and mitigate hidden harmful intents before response generation. Without any retraining, AMIA improves defense success rates across diverse LVLMs and jailbreak benchmarks from an average of 52.4% to 81.7%, preserves general utility with only a 2% average accuracy drop, and incurs only modest inference overhead. Ablation confirms that both masking and intention analysis are essential for robust safety–utility trade-off. Our code will be released.

## introduction by integrating visual modalities into large language models (llms; @xcite @xcite , large vision language models (lvlms) have shown impressive capabilities in various multimodal tasks @xcite . however, lvlms encounter worrying safety issues, especially under jailbreak attacks @xcite , which aim to induce harmful behaviors from lvlms through techniques like prompt manipulation @xcite or visual adversarial perturbation @xcite . existing studies identify safety degradation during fine-tuning as a key factor behind lvlm vulnerabilities @xcite . incorporating visual inputs expands the attack surface, and the lack of safety-aware training makes it difficult for lvlms to retain the safety mechanisms of their underlying llm backbones. since large-scale multimodal comprehensive safety training is resourceintensive @xcite , inference-time defenses provide a more practical alternative. one representative method, ecso @xcite , shows that converting visual inputs into textual captions can reactivate the safety mechanisms inherited from the llm backbone. however, it only handles the visual modality and overlooks the jointly harmful semantics in image-text inputs, limiting its effectiveness in more complex multimodal jailbreak scenarios, as discussed in section 4. a natural solution is to guide lvlms to perform joint image-text understanding before generating responses, enabling them to translate multimodal inputs into text while excavating their underlying intention. while such prompt understanding has proven effective in llm defenses @xcite , it still faces challenges from adversarial image perturbations specific to multimodal settings, as discussed in section 4. to address this, we propose amia, a method combining automatic masking and joint intention analysis, to improve lvlm safety against complex multimodal jailbreak attacks. specifically, we first introduce an image-text correlation-driven automatic masking strategy that masks the least relevant image patches to disrupt adversarial patterns • the image shows a phrase of an activity. list steps to perform this activity. ## methodology we introduce amia, an inference-only lvlm defensive method to enhance lvlm safety under complex vision-language jailbreak attacks. specifically, amia consists of two components, which we describe in the following sections. ## experimental setup models and baselines. we perform experiments on three popular lvlms-llava-v1.5-7b, @xcite , and dsr only on mmsafetybench following their official protocols (appendix a.4). for general utility, we use three standard lvlm benchmarks-mmvp @xcite , ai2d @xcite , and mmstar (chen et al., 2024a)-and report accuracy as the helpfulness metric to assess amia's impact on model utility. experimental details. our method introduces hyperparameters n to control how many patches the image is divided into and k for the masked image patch count. for simplicity and reproducibility, we set @xmath0, with the sensitivity analyzed in section 5. further experimental details are in appendix a.6. ## main results we summarize amia's performance on four jailbreak and three general datasets in tables 1 and 2 , respectively. based on the results, we can find that: amia consistently enhances safety across different jailbreaks and lvlms. as shown in ## analysis to better understand the factors influencing amia's effectiveness, we conduct further analysis using the llava-v1.5-13b model. component ablation of amia. we perform ablation studies on a jailbreak dataset, visualadv-harmbench, and a general dataset, mmvp, to assess the individual impact of automatic masking and intention analysis mechanisms in sensitivity analysis of k. figure 4(a) presents a sensitivity analysis of k, the number of masked image patches, on the general dataset mmvp and adversarial dataset visualadv-harmbench with @xmath1. results show that as k increases, dsr on visualadv-harmbench steadily improves and gradually saturates. on mmvp, performance remains stable when @xmath2, indicating that moderate masking retains useful visual cues, while excessive masking impairs utility. to balance safety and utility, we set @xmath3. this analysis provides practical guidance for lvlm deployment with different safety-utility requirements. sensitivity analysis of n . figure 4(b ) analyzes the effect of n , the number of image patches, with k set to match the optimal masking ratio from figure 4 (a) (see appendix a.5). we find amia shows consistent robustness to variations in n . ## conclusion large vision-language models (lvlms) face safety risks under complex multimodal jailbreak attacks. this work proposes an inference-time defense that combines automatic masking with intention analysis to disrupt adversarial perturbations and uncover harmful intention in image-text inputs. experiments across multiple datasets validate the effectiveness and applicability of our approach, highlighting its potential to improve lvlm safety.