Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLM s (2025)

Abstract: Large Vision-Language Models (LVLMs) have made significant strides in multimodal comprehension, thanks to extensive pre-training and fine-tuning on large-scale visual datasets. However, despite their robust textual safety mechanisms, they remain vulnerable to harmful visual inputs. Existing safeguards—typically relying on pre-filtering or fine-tuning—incur high costs and diminish overall utility. To address this critical vulnerability, we introduce SafeCLIP, a lightweight method that leverages LVLMs’ inherent multimodal alignment for zero-shot toxic image detection. By projecting CLIP’s discarded CLS token into its text space and matching it with toxic descriptors, SafeCLIP detects harmful content without any architectural changes—adding minimal latency and enabling dynamic safety corrections during inference and fine-tuning. Experiments show that SafeCLIP achieves a 66.9% defense success rate with only 3.2% false positive rate and 7.2% overhead. In contrast, state-of-the-art methods achieve 52.9% success but have a 10.7% false positive rate and 210% overhead. Our work demonstrates that leveraging inherent multimodal alignment can yield efficient, low-cost LVLM safety. Code is available at anonymous.4open.science/r/safeclip-2C01 .

## introduction large vision language models (lvlms) have recently demonstrated remarkable progress across a wide range of multimodal tasks @xcite , achieving substantial image understanding through extensive pretraining and fine-tuning on large-scale image datasets. given that vision and text are integrated into a common representation space in lvlms, employing a unified safety mechanism for both modalities, rather than training separate ones, could prove both effective and efficient. however, this is currently not the case. while the base language model has built-in safety mechanisms against harmful textual inputs @xcite , lvlms fine-tuned for multimodal understanding demonstrate fairly limited safety measures when exposed to harmful images. for example, evaluations on the toxic image dataset @xcite show that traditional lvlms (e.g., llava-1.5 @xcite ) achieve a 0% defense success rate against toxic visuals, despite maintaining some text safety. more recently developed multimodal models like qwen @xcite and janus-pro @xcite similarly have limited safety, i.e., with a 1.6% defense success rate. in fact, merely requiring an lvlm to describe a toxic image can inadvertently lead to harmful responses. existing approaches to safeguarding lvlms typically rely on safety pre-filtering techniques @xcite or safety-oriented fine-tuning @xcite , both of which may introduce substantial computational costs and compromise overall utility. for instance, llava-guard @xcite uses a two-step process (safety filtering then processing), incurring up to 500% overhead, while fine-tuning methods like tga @xcite require full dataset captioning yet achieve only a 21.2% defense rate across seven toxic categories. given that existing lvlms such as those built on clip exhibit strong zero-shot classification capabilities, we believe that these models inherently have the capabilities to semantically understand the images, and therefore a promising yet under-explored strategy is to leverage the models' inherent capabilities for aligning safety across multimodal. in this work, we propose safeclip, a lightweight, clip-driven method that leverages the inherent multimodal alignment of lvlms to detect and mitigate toxic visual inputs in a zeroshot manner. safeclip repurposes the vision encoder's cls token-normally discarded after feature extraction-as a robust safety-aware signal. by projecting the cls token into clip's text embedding space and comparing it against a carefully designed bank of toxic concept descriptors, safe-clip identifies harmful visual scenes with high accuracy. furthermore, since the cls token is generated during inference, integrating safeclip into existing lvlms incurs negligible computational cost. this low-latency approach also facilitates potential deployment during fine-tuning, enabling the automatic generation of safe alignment targets and dynamic adjustment of training objectives to reinforce safety. through extensive experiments on toxic image datasets, we show that safeclip outperforms stateof-the-art safety methods. on llava-1.5, safeclip achieves a 66.9% defense success rate across seven toxicity categories and a low 3.2% false positive rate on benign inputs. in contrast, state-of-theart approaches such as esco @xcite and llavaguard achieve 52.9% and 49.2% defense success rates with false positive rates of 10.7% and 3.4%. additionally, while esco and llava-guard incur latency increases of up to 210.0% and 500.0%, safeclip only adds a 7.2% increase for neutral inputs and even reduces latency by 5.7% for toxic inputs, thanks to the shorter refusal responses.these results highlight safeclip's ability to effectively defend against toxic images such as explicit imagery, violence, and offensive gestures while considerably reducing computational overhead. our contributions can be summarized as follows: ## preliminary in this section, we first describe the standard architecture of current mainstream large vision-language models (lvlms) and subsequently present the safety challenges of lvlms against toxic visual inputs, and then define our research objective. ## our method in this section, we introduce safeclip, an efficient clip-based method for zero-shot toxic scene detection in lvlms. we begin by explaining the core functionality of this approach, followed by a discussion on its integration during both the inference and fine-tuning phases of lvlms. ## related work this study relates to research on lvlm vulnerability and lvlm safety. ## conclusion we proposed safeclip, an efficient method that enhances lvlm safety against toxic visual inputs by leveraging the vision encoder's cls token for zero-shot detection. with minimal overhead during inference and fine-tuning, safeclip effectively prevents harmful outputs while maintaining model efficiency, offering a scalable solution to lvlm vulnerabilities.