S afe E raser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning (2025)

Abstract: As Multimodal Large Language Models (MLLMs) develop, their potential security issues have become increasingly prominent. **Machine Unlearning (MU)**, as an effective strategy for forgetting specific knowledge in training data, has been widely used in privacy protection. However, *MU for safety in MLLM has yet to be fully explored*. To address this issue, we propose , a safety unlearning benchmark for MLLMs, consisting of 3,000 images and 28.8K VQA pairs. We comprehensively evaluate unlearning methods from two perspectives: **_forget quality_** and **_model utility_**. Our findings show that existing MU methods struggle to maintain model performance while implementing the forget operation and often suffer from **_over-forgetting_**. Hence, we introduce **Prompt Decouple (PD) Loss** to alleviate over-forgetting through decouple prompt during unlearning process. To quantitatively measure over-forgetting mitigated by PD Loss, we propose a new metric called **Safe Answer Refusal Rate (SARR)**. Experimental results demonstrate that combining PD Loss with existing unlearning methods can effectively prevent over-forgetting and achieve a decrease of 79.5% in the SARR metric of LLaVA-7B and LLaVA-13B, while maintaining forget quality and model utility. Our code and dataset will be released upon acceptance. **Warning: This paper contains examples of harmful language and images, and reader discretion is recommended.**

## introduction in recent years, large language models (llms) @xcite and multimodal large language models (mllms) @xcite @xcite have achieved significant success in diverse applications @xcite , 2024c,b). however, since model training involves millions of data, some of which may contain harmful knowledge, this could cause the model to generate harmful responses. previous research @xcite @xcite has highlighted the severe security issues present in mllms. to confront the safety issue, numerous studies @xcite @xcite have revealed that machine unlearning (mu) could precisely forget harmful knowledge within llms. currently, most work on mu in mllms primarily focuses on privacy protection. @xcite propose mmubench, a benchmark aimed at measuring the effectiveness of different mu methods in privacy protection, and also propose the siu method, which achieved good performance. @xcite assign a two-stage evaluation pipeline with a newly proposed fictitious facial identity vqa dataset. however, in unlearning for privacy, the concepts to be forgotten are represented by a single word or phrase. in contrast, unlearning for safety involves concepts that are much more difficult to express with a single word or phrase, as harmful knowledge typically corresponds to a sentence or a passage. this increases the difficulty of unlearning for safety. unlearning for privacy aims to eliminate learned patterns associated with visual recognition of specific "to-be-forgotten" concepts. however, the model needs to forget the harmful knowledge associated with vqa in unlearning for safety. therefore, it is still necessary to extend mu to the field of safety, in order to improve the trustworthy mllm-based system. in this work, we are the first to explore the application of mu in mllm safety and propose safeeraser, a comprehensive benchmark for evaluating the effectiveness of unlearning methods in mllm safety, consisting of 3,000 images and 28.8k vqa pairs. safeeraser is divided into three main sets: forget set, retain set, and prompt decouple set. forget set contains vqa pairs associated with harmful knowledge that model should forget. retain set includes concept-level and image-level vqa pairs, aiming to preserve model utility. prompt decouple set is designed to alleviate over-forgetting in the model. additionally, safeeraser defines a comprehensive set of evaluation metrics, including efficacy, generality, rouge, gpt-eval, and specificity. efficacy and generality are used to measure forget quality of unlearning methods. rouge, gpt-eval, and specificity assess the model utility after unlearning. furthermore, in response to the issue of overforgetting, which means that model after unlearning provide refusal responses to harmless queries similar to those in forget set, we propose prompt decouple loss. this intuitive approach is purposed to decouple the prompts in forget set. we fine-tune the model using harmless vqa pairs, composed of prompts and harmless images from forget set along with the answers generated by vanilla model. this method can be applied to any training-based unlearning method. in addition, we propose a new metric sarr to quantitatively evaluate the overforgetting after unlearning. our contributions can be summarized as follows: ## conclusion in this paper, we first formalize the safety unlearning task for mllms and construct safeeraser, a safety unlearning benchmark consisting of 3,000 images and 28.8k vqa pairs, which contain 6 different categories, with 10 keywords associated with each category. we identify the challenge of overforgetting, where models fail to respond to harmless queries after forgetting harmful content, and propose pd loss to mitigate this issue. experimental results show that combining pd loss with existing mu techniques reduces the sarr by 79.5%, while preserving model utility and forget quality. this work provides a critical step toward a safer, more trustworthy mllm-based system.