Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models (2025)

Abstract: The safety alignment ability of Vision-Language Models (VLMs) is prone to be degraded by the integration of the vision module compared to its LLM backbone. We investigate this phenomenon, dubbed as “safety alignment degradation” in this paper, and show that the challenge arises from the representation gap that emerges when introducing vision modality to VLMs. In particular, we show that the representations of multi-modal inputs shift away from that of text-only inputs which represent the distribution that the LLM backbone is optimized for. At the same time, the safety alignment capabilities, initially developed within the textual embedding space, do not successfully transfer to this new multi-modal representation space. To reduce safety alignment degradation, we introduce Cross-Modality Representation Manipulation (CMRM), an inference time representation intervention method for recovering the safety alignment ability that is inherent in the LLM backbone of VLMs, while simultaneously preserving the functional capabilities of VLMs. The empirical results show that our framework significantly recovers the alignment ability that is inherited from the LLM backbone with minimal impact on the fluency and linguistic capabilities of pre-trained VLMs even without additional training. Specifically, the unsafe rate of LLaVA-7B on multi-modal input can be reduced from 61.53% to as low as 3.15% with only inference-time intervention.

## introduction the development of vision language models (vlms) has marked a significant advancement, enabling models to process information from both visual and textual modalities and has shown * work done during internship at aws ai labs. several existing works have explored the phenomenon of safety alignment degradation. for example, @xcite attempt to transform unsafe images into texts to activate the intrinsic safety mechanism of pre-aligned llms in vlms. however, images often contain fine-grained information that could not be fully captured by text. on the other hand, @xcite leverage the safety risks posed by the visual modality and propose a jailbreak method that conceals and amplifies the malicious intent within text inputs using carefully crafted images. however, the underlying mechanisms of how images influence alignment remain unexplored. from the aspect of improving vlm safety, a line of work has made successful attempts by training vlms with deliberately curated dataset @xcite @xcite . however, these attempts are annotation-intensive and computationally costly, ignoring the inherent safety alignment of the llm backbone of a vlm. in this study, we propose to investigate the critical challenge of alignment degradation by examining how the integration of a vision module intrinsically impacts model behavior, particularly in terms of model representations ( §2). our hypothesis is that, since the vision and language modules within a typical vlm are trained independently, the representations from these different modalities tend to cluster in distinct regions of the latent space. this separation likely results in a distribution shift, deviating from the representation space that the llm backbone is optimized for, which further leads to reduced alignment ability when the vlm processes multimodal inputs. to verify this hypothesis, we evaluate three open-source vlms of varying scales and employ principal component analysis (pca) to visualize their hidden states upon different types of input: text-only or a mixture of text and image. we find that in the models' representation space, different types of input are clearly distinguished, suggesting that our hypothesis holds. inspired by these findings, we further investigate whether alignment degradation can be mitigated by eliminating the representation shift when an image is incorporated as input. to justify this assumption, we present a method to intervene the hidden states of vlms, named cmrm (cross-modality representation manipulation) ( §3). cmrm first anchors a vlm's low-dimensional representation space and estimates the "shifting direction" that indicates the affect of the incorporation of image in the input on the overall hidden states. it then calibrates the representation of multi-modal input using the estimated direction so that the hidden states can be pulled closer to the distribution that the llm component is optimized for. through experiments on two vlm safety benchmarks, we demonstrate that cmrm remarkably recovers the alignment ability of vlms even without additional training. furthermore, cmrm does not compromise vlms' general performance, as evaluated on two vlm utility benchmarks. we hope our work sheds light on the intrinsic influences of the construction of vlms and inspires future research on vlm alignment. our code will be open-sourced for reproducibility. in summary, our main contributions are as follows: we first analyze the alignment degradation challenge by investigating the following question: how does the integration of the vision modality intrinsically affect model behavior? we hypothesize that since the vision and language modules within a vlm are trained independently, the resulting representations tend to cluster in distinct regions of the latent space, leading to a distribution shift that reduces alignment ability when processing multi-modal inputs, as it deviates from the representation space that the llm backbone is optimized for. to verify this hypothesis, we investigate how representations of different types of inputs exist in model's representation space, and how the distinction correlates with the alignment degradation of vlms. 2.1 experimental setup input variations & datasets. to investigate the influence of vision modality on the safety alignment of a vlm, we employ 5 variations on the model input: (1) original input, where the images and textual queries remain unchanged as model input; (2) blank image, where the original image is substituted with a blank image that does not carry any semantic meaning; (3) gaussian noise, where we perturb the original images with gaussian noise in an attempt to destroy their semantic meaning; we use two safety-related multi-modality datasets for model behavior analysis: vlsafe @xcite and jailbreakllms @xcite paired with images from the coco dataset @xcite . vlsafe (vision-language safety) dataset contains 1, 110 pairs of malicious queries and benign images where the input images are auxiliary and the queries can be answered without referencing the images. jailbreakllms dataset includes 390 jailbreak prompts that are collected and filtered from reddit, discord, websites, and open-source datasets covering 14 topics, including illegal activity, malware, physical harm, etc. we filter out two safety-irrelevant topics and pair each remaining jailbreak prompt with a related image retrieved from the coco dataset to construct a multi-modal dataset composed of 330 test samples. ## methods building on insights from §2, we attempt to formalize the affected hidden states of current vlms with vision input incorporated ( §3.1), based on which we propose two variations of cmrm to intervene model representations during inference time to prevent the alignment degradation ( §3.2). ## experiments and evaluation in this section, we empirically evaluate the proposed cmrm. first, we introduce the caption substitutes the image with its textual caption, and query uses the textual prompt as the only input. utility performance is evaluated on llava-bench-coco (l-bench) and scienceqa (s-qa). note that vlguard mixed and vlguard ph are training-time safety alignment methods for reference purposes, which does not form a fair comparison for our method. experimental settings in §4.1. then, we assess cmrm from the following perspectives: (i) how well can cmrm improve the safety of vlms and recover the alignment ability of the llm backbone? ( §4.2) (ii) does cmrm harm general performance of vlms? ( §4.2) (iii) how does the hyperparameters affect cmrm? ( §4.3) (iv) does the extracted shifting direction based on one anchor dataset generalize to other datasets? ( §4.4) (v) what is the impact of cmrm on vlms' hidden states? (appx. §a.3) ## conclusion we investigate the impact of incorporating visual input on vlms. we find that multi-modal input drastically degrades the safety alignment mechanism of the llm backbone in vmls, while intrinsically shifting the hidden states away from the distribution that the llm module is optimized for. drawing from this inspiration, our proposed cmrm method intervenes the representations of vlms upon multi-modal inputs by moving hidden states closer to the trained distribution of its llm module. cmrm operates at inference time and can be seamlessly integrated with any pre-trained vlms, which makes cmrm a cost-effective and flexible solution to enhance safety alignment of vlms without tedious training. we show that cmrm brings remarkable improvement in recovering the safety alignment for vlms without compromising the model's general performance.