Can Textual Unlearning Solve Cross-Modality Safety Alignment? (2024)

Abstract: Recent studies reveal that integrating new modalities into large language models (LLMs), such as vision-language models (VLMs), creates a new attack surface that bypasses existing safety training techniques like supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). While further SFT and RLHF-based safety training can be conducted in multi-modal settings, collecting multi-modal training datasets poses a significant challenge. Inspired by the structural design of recent multi-modal models, where all input modalities are ultimately fused into the language space, we explore whether unlearning solely in the textual domain can be effective for cross-modality safety alignment. Our empirical evaluation across seven datasets demonstrates promising transferability — textual unlearning in VLMs significantly reduces the Attack Success Rate (ASR) to less than 8% and in some cases, even as low as nearly 2% for both text-based and vision-text-based attacks, alongside preserving the utility. Moreover, our experiments show that unlearning with a multi-modal dataset offers no potential benefits but incurs significantly increased computational demands.

## introduction as large language models (llms) advance in their capabilities, ensuring that their outputs align with human preferences and policy regulations has become an essential task. popular safety alignment techniques, such as supervised fine-tuning (sft) and reinforcement learning from human feedback (rlhf) @xcite @xcite @xcite , have been widely adopted by major llm vendors like openai. nevertheless, re-* equal contribution; co-first authors listed alphabetically by last name cent studies have demonstrated that as new modalities are integrated into llms, such as in visionlanguage models, new cross-modality safety issues arise @xcite @xcite , even if the llms have already been aligned. these vulnerabilities suggest that the added modalities create new attack surfaces that the mainstream safety training techniques do not adequately address @xcite @xcite . to defend against the multi-modal vulnerabilities, recent works propose to collect multi-modal (mainly image-text) safety training datasets and perform adversarial training followed by rlhf on the multi-modal models (mainly vlms) @xcite . one major limitation of such defenses is scalability. first, collecting such multi-modal defense datasets with newly added modalities is challenging. as new modalities (e.g., audio, speech, video, imu, fmri, and more) are incorporated into these multi-modal models, each modality not only expands the input embedding space dramatically but also introduces new vulnerabilities to cross-modality attacks @xcite . given this, collecting jailbreak @xcite and unsafe multi-modal datasets, which distribute maliciousness across modalities, requires significant human effort and may not scale well with the addition of more modalities. second, defending against cross-modal attacks is challenging due to the vast array of potential input combinations from different modalities. as a result, collected datasets often fail to cover significant portions of the attack surface for sft to generalize, allowing users to easily discover new attack combinations @xcite . these limitations motivate us to investigate whether unlearning @xcite @xcite @xcite , as an alternative to sft and rlhf, when performed solely in the tex-tual domain, can generalize and scale to different (and might unseen) modalities. this speculation is inspired by the structural design of recent multimodal models @xcite @xcite , where, regardless of the combination of input modalities, all inputs are ultimately fused into the language space. in other words, multi-modal models frequently align other modalities to the embedding space of the textual modality for reasoning and generation. since all information flows through the language modality, we explore textual and multimodal unlearning and address the cross-modality safety alignment issue by focusing on the information bottleneck -the language modality itself. we investigate whether textual unlearning in the llm component of the vlm is sufficient to achieve high harmlessness and robustness against crossmodality attacks while maintaining the model's normal capabilities, by specifically teaching the llm to avoid generating harmful content. recent works have studied llm unlearning @xcite @xcite , but unlearning in multi-modal language models, particularly in the vision language domain, remains largely unexplored. to our knowledge, we are the first to investigate various configurations of unlearning across single or multiple modalities on vlms to address the cross-modality safety alignment problem. surprisingly, our empirical results suggest that textual unlearning can be effectively transferred from llms to vlms, operating solely on the llm component of the multi-modal model. it works by learning not to propagate harmful context toward toxic regions, instead redirecting it toward safer areas. moreover, we demonstrate that unlearning in the multi-modal domain offers little advantage over textual unlearning, in addition to requiring significant effort in creating cross-alignment data and multi-modal training. thus, the key advantage of the textual unlearning approach over multi-modal unlearning is its significant improvement in computational efficiency as well as its effectiveness. by applying textual unlearning to vlms, we demonstrate that we can achieve better levels of harmlessness using only about one-sixth the computing time and energy on the same gpu, and without the need for collecting multi-modal datasets. in summary, this paper aims to answer the following two research questions. multimodal large language models (mllms). mllms @xcite are designed to process multimodal inputs, including text @xcite , image @xcite , audio @xcite , and video @xcite , using llms as a brain for reasoning, with efforts also underway for any-to-any modality generation @xcite . mllms comprise three main components: a modality encoder for feature embedding, a projection layer to transfer these features into the language space, and a pretrained language model for output generation. these projection layers commonly use linear layers @xcite , gated cross-attention mechanisms @xcite , or q-formers @xcite . ## methodology in this section, we detail our approach to unlearning in vlms, including the loss terms used to reduce harmfulness while maintaining the helpfulness (utility). our objective is to use vlms as examples to analyze the performance of textual unlearning in mllms, to understand the added benefits of multi-modal unlearning, and to compare its performance with supervised fine-tuning (sft). therefore, we discuss textual unlearning, multimodal unlearning, and multi-modal sft, highlighting the key differences in their settings. notations. let the input image be denoted as x i , the input text as x t , and the output text as y t . the image encoder, parameterized by θ, is represented by v θ , which transforms the input image x i into an embedding in the image space, denoted as e i . in current vision-language designs, this embedding is then projected into the text space using a projection layer p ψ , parameterized by ψ, resulting in e it . eqn. 1 details the generation of embeddings for input images. the language model, parameterized by σ, is represented by l σ , and the vision-language model (vlm), parameterized by ϕ, is denoted as f ϕ . the vlm f ϕ integrates the embedding of input text e t and the projected image embedding e it using the language model l σ . this process results in the final text generation, per eqn. 2. given the input image x i and input text x t , the probability of the next token y t i generation depends on the inputs and already generated tokens, as denoted in eqn. 3. the loss, detailed in eqn. 4, is usually computed as the negative sum of the log probabilities for n number of tokens. we model unlearning as an optimization problem with three objectives: (1) minimize the probability of generating unwanted (e.g., harmful) output, regardless of the input; (2) increase the probability of generating preferred answers to harmful input; and (3) maintaining the probability of generating useful outputs to normal inputs as the original model. specifically, given a harmful dataset with negative (e.g. harmful, unethical, or illegal) samples, and a normal dataset with benign samples, and a target model f, we design a loss term consisting of three components. first, for each harmful sample in the unlearn dataset, denoted as a @xmath0, such as 'i cannot assist with this'. the goal is to keep the model's response meaningful, as some evaluation tools rely on such outputs. third, in order to preserve the utility on benign inputs, we intend the unlearned vlm response at t time step f ϕt to be as similar as the original model, which can be denoted with f ϕ 0 representing the initial loaded model. to do so, for each normal sample image-text input @xmath1, x normal @xmath2, we compute the kullback-leibler (kl) divergence between outputs of the unlearned and the original model as presented in eqn. 5. following recent studies on llm unlearning @xcite , we adopt the gradient ascent (ga)-based approach to increase l harm , driving the model away from generating harmful tokens. hence, we minimize the probability of the generation of harmful answers given the prompts of the harmful dataset. conversely, we perform gradient descent to decrease l helpful.match and l utility . decreasing l helpful.match maximizes the probability of the generation of helpful answers given the prompts from the unlearn dataset. similarly, decreasing l utility helps the unlearned model mimic the behavior of the original model as closely as possible when provided with benign prompts; so that the model retains its normal capabilities. note that during unlearning, we only adjust the parameters σ of the llm component, while freezing the rest of the parameters of the vlm (i.e., vision encoder θ and projection layers ψ). as depicted in eqn. 6, the parameter update for σ involves moving in the direction of the gradient j of l harm , denoting gradient ascent, and in the negative direction of l helpful.match and l utility , denoting the usual gradient descent. here, η harm , η helpful.match , and η utility depict the corresponding weights of the loss terms. these weights are hyperparameters that need to be tuned during training. in our experiments shown in section 4, we use {η harm , η helpful.match , η utility } = {0.5, 1, 1}. textual unlearning. we refer textual unlearning as the process of using text-only (harmful and normal) datasets to perform unlearning. specifically, as shown in eqn. 7 and 8, our three loss terms are now narrowed down to only the text modality, and the image input is set as none. the goal of textual unlearning is to evaluate whether unlearning can be transferred from pure textual domain to newly added modalities. that is, whether an unlearned vlm can resist crossmodality alignment attacks. as discussed earlier, we believe this is an important research question as the majority of datasets are in the textual domain. multi-modal unlearning. to explore whether the added modality can benefit unlearning, we conduct multi-modal unlearning on vlms, the (harmful and normal) datasets are multi-modal. it means that the input consists of a textual prompt and an image, and the output is still text which is the response to the prompt. multi-modal sft. previous study @xcite indicates that unlearning in the textual domain outperforms sft. as an additional analysis, we aim to explore how multi-modal sft performs in comparison to both textual and multi-modal unlearning. specifically, we use multi-modal datasets to perform sft on vlms. during fine-tuning, we exclude the l harm and l utility terms. for unlearn/harmful inputs, we only apply the l helpful.match term; for normal inputs, we include l normal term, shown in eqn. 9. similar to unlearning, we freeze the visual parameters and only update the parameters of the llm component, per eqn. 10. the overall loss terms map input prompts to their desired outputs using language modeling crossentropy loss. for harmful prompts, the desired output is 'i cannot assist with this', while for the normal prompts, the answers are collected from the original model. in other words, we aim to maximize the generation of the desired response for each type of prompt. we mix the datasets to create batches containing both harmful and normal prompts and the model is trained on them. ## experiments this section describes the experiments conducted and the evaluation metrics used. based on these results, we finally address our two rqs. ## conclusion as modalities get added to llms, research has shown that cross-modality attacks can bypass their builtin safety alignment. this paper demonstrates that performing "textual" unlearning on the llm components of vlms alone, can achieve surprising levels of harmlessness against cross-modality attacks. additional experiments on multi-modal unlearning and sft show that textual unlearning with more comprehensive harmful datasets can outperform multi-modal unlearning and sft, which highlights the importance of harm coverage of the dataset over being multi-modal moreover, multimodal unlearning and sft can require up to six times more computational resources. these intriguing results encourage us to further investigate the textual unlearning paradigm, which we find to be extremely effective in reducing harmfulness, capable of preserving the normal capabilities of the vlm, and more computational efficient.