MLLM -Protector: Ensuring MLLM ’s Safety without Hurting Performance (2024)

Abstract: The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs. This paper investigates the novel challenge of defending MLLMs against such attacks. Compared to large language models (LLMs), MLLMs include an additional image modality. We discover that images act as a “foreign language” that is not considered during safety alignment, making MLLMs more prone to producing harmful responses. Unfortunately, unlike the discrete tokens considered in text-based LLMs, the continuous nature of image signals presents significant alignment challenges, which poses difficulty to thoroughly cover all possible scenarios. This vulnerability is exacerbated by the fact that most state-of-the-art MLLMs are fine-tuned on limited image-text pairs that are much fewer than the extensive text-based pretraining corpus, which makes the MLLMs more prone to catastrophic forgetting of their original abilities during safety fine-tuning. To tackle these challenges, we introduce MLLM-Protector, a plug-and-play strategy that solves two subtasks: 1) identifying harmful responses via a lightweight harm detector, and 2) transforming harmful responses into harmless ones via a detoxifier. This approach effectively mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs. Our results demonstrate that MLLM-Protector offers a robust solution to a previously unaddressed aspect of MLLM security.

## introduction the emergence of large language models (llms) @xcite @xcite @xcite @xcite has marked a significant milestone in the field of ai, revolutionizing natural language processing and under-* equal contribution. standing. these models, trained on vast text corpus datasets, excel in generating coherent and contextually relevant text, making them powerful tools for many downstream applications. building on this progress, multimodal large language models (mllms) @xcite @xcite @xcite @xcite have also seen rapid improvements, extending the capabilities of llms to engage in conversations with image inputs, which enables more potential applications. meanwhile, both llms and mllms are prone to malicious user queries. in text-based llms, malicious attacks typically involve meticulously crafted queries that induces the model to generate inappropriate or harmful content @xcite @xcite @xcite . research in defending against such attacks has led to various strategies, including input detection @xcite , in-context learning @xcite , and explicit alignment with adversarial examples @xcite @xcite @xcite @xcite . however, in the realm of mllms, defending strategies are still underdeveloped. a new observation has been made: images can inadvertently induce these models to produce malicious content @xcite , as illustrated in figure 1 . this could lead to serious consequences, as attackers might use images as triggers for malicious queries. to gain a deeper understanding of this issue, we experimentally find that the likelihood of generating harmful responses is significantly higher given image inputs than text input, as show in ## observation as shown in figure 1 , we discover state-of-the-art open-source multi-modal large language models (mllms), such as llava @xcite , are presented with relevant input images that have malicious content, they become prone to generating sensitive or potentially harmful responses, despite the model's ability to recognize and refuse to provide advice on such topics when the input is purely text-based. a recent study @xcite also point out that both related natural images, and ocr images containing the relevant phrase, can mislead the model into generating harmful content. therefore, we make further analysis on the mllm's outputs, and observe the following: for mllms that are based on instruction-tuned llms (e.g., vicuna-based llava), given related images that contain malicious content as inputs, the likelihood for generating harmful responses becomes markedly higher compared with text-only inputs. specifically, we select six most common scenarios from mmsafetybench @xcite as specified in recent research @xcite indicates that llms aligned for safety using the english language do not perform well in defending against attacks in other languages. hence, we hypothesize that for image-text aligned mllms, images act as a "foreign language", offering semantic parallels to textual inputs. however, unlike their textual counterparts, image-based inputs have not been subject to the same level of instruction tuning or alignment. this discrepancy appears to be a contributing factor to the models' increased susceptibility to generating harmful content in response to image inputs. ## vanilla safety fine-tuning in our preliminary investigation, we adopted the supervised fine-tuning (sft) strategy, which is the conventional approach for aligning text-based llms. to construct our image-text paired dataset and perform sft, we follow subsequent steps. we first leverage an existing text-based dataset saferlhf @xcite ) that comprises malicious user queries, each paired with two responses generated by the llm. these responses are accom- we summarize the disadvantages of safety finetuning in the context of mllms as follows: ## mllm-protector in this section, we introduce our novel defense paradigm termed mllm-protector, which addresses the challenges in defending mllms against malicious image inputs via a divide-andconquer strategy. specifically, we introduce a lightweight harm-detector to first identify whether the responses are harmful, and a detoxifier to correct the harmful contents in the response. our method serves as a plug-and-play component that works in conjunction with any mllms. notably, the components of mllm-protector can be trained independently, then be used directly during inference, which prevents hampering the mllm's original capability while ensuring their safety. in this section, we will first elaborate the model architecture of mllm-protector. then, we introduce the objective and data used during training. lastly, we illustrate how our mllm-protector can be incorporated with any mllm during inference. ## experiments 6.1 implementation details for the harm detector, we adopt open-llama-3b model from @xcite . for the detoxifier, we utilize llama-7b @xcite . for the harm detector, we perform tuning for 3 epochs using lora with rank 32, batch size is set to 32, and the learning rate is set to 2e -5 ; for the detoxifier, we finetune the model for 3 epochs using lora with rank 128, batch size is set to 32, and the learning rate is set to 1e -4 . the components are both trained on 8 a40 gpus using deepspeed with bfloat16. the training takes around 1 hour for harm detector and 3 hours for the detoxifier. ## helpfulness with mllm-protector in ## ablation study in this section, we study the performance of the harm detector, which is the key for the effectiveness of mllm-protector. we first combine our safe-harm-10k and the image-text instruction tuning data (as described in section 5.2) and refer to it as image qa. then, we separate the saferlhf and image qa datasets into training and validation splits with ratio of 7:3. afterwards, we train the harm detector using the combined training splits and conduct evaluation separately on two datasets. ## conclusion this paper presents mllm-protector, a novel defense paradigm for multimodal large language models (mllms) that tackles safety challenges using a divide-and-conquer approach. by incorporating a harm detector that identifies potentially harmful outputs and a detoxifier that amends them, our method operates as a modular, plug-and-play solution, ensuring the safety of mllms without degrading their overall performance. notably, mllm-protector can be seamlessly integrated into existing mllm architectures, enhancing their robustness against adversarial or unsafe content generation. we believe this work not only addresses an urgent need for safer ai systems but also highlights critical safety considerations for mllm deployment in real-world applications. we hope that this contribution will foster greater awareness and spark further research into improving the safety and alignment of multimodal ai systems, paving the way for more responsible development and usage.