MMU nlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models (2025)

Abstract: Recent progress in Machine Unlearning (MU) has introduced solutions for the selective removal of private or sensitive information encoded within deep neural networks. Nonetheless, MU for Multimodal Large Language Models (MLLMs) remains in its nascent phase. Therefore, we propose to **reformulate the task of multimodal MU in the era of MLLMs**, which aims to erase only the visual patterns associated with a given entity while preserving the corresponding textual knowledge encoded within the original parameters of the language model backbone. Furthermore, we **develop a novel geometry-constrained gradient ascent method MMUnlearner**. It updates the weights of MLLMs with a weight saliency map jointly restricted by the remaining concepts and textual knowledge during unlearning, thereby preserving parameters essential for non-target knowledge. Extensive experiments demonstrate that MMUnlearner surpasses baselines that finetuning MLLMs with VQA data directly through Gradient Ascent (GA) or Negative Preference Optimization (NPO), across all evaluation dimensions. Our code will be released upon acceptance.

## introduction multimodal large language models (mllms) achieved remarkable performance on various multimodal applications @xcite @xcite . a common framework of mllms, which projects the visual embeddings extracted from pre-trained vision encoder into the representation space of language models with a projector, has enabled llm backbone to understand visual inputs and preserve their powerful reasoning and generation potential @xcite @xcite . however, the rapid development of † equal contribution. * corresponding author. mllm is also accompanied by safety concerns such as personal privacy @xcite and copyright infringement @xcite . retraining the models from scratch to exclude the risky knowledge is resource-intensive and practically untenable due to the inaccessible pre-training data @xcite . hence, machine unlearning (mu) can serve as a feasible solution to forget specific knowledge embedded within pretrained models @xcite . nevertheless, mu on mllms is still in its nascent phase, with limited approaches and benchmarks available. for example, single image unlearning (siu) first explores the mu of mllms, aiming to erase visual patterns in mllms on realworld entities, but it needs to reconstruct multifaceted fine-tuning data for forgetting @xcite . besides, mllmu-bench evaluates the performance of mu methods designed for llms on fictional personal profiles @xcite ; clear adds visual images to pure-text llm unlearning benchmark tofu through photomaker @xcite , a diffusion model adapted for customized realistic human @xcite . as shown in figure 1 (a), the aforementioned works just transfer llm-based mu methods to mllms via fine-tuning on vqa data, and neglect the unique difficulty for mllm-specific mu. therefore, we propose to reformulate the task of multimodal mu in the age of mllms, as illustrated in figure 1 @xcite . unlike text-only llms, the knowledge embedded in mllms extends beyond textual factual knowledge within their llm module, which includes learned visual attributes associated with various concepts @xcite . given this fundamental difference, we define the objective of mllmbased mu as the selective removal of visual patterns linked to a specific entity, while preserving the corresponding textual knowledge within the llm backbone, as illustrated in figure 2 . considering that existing benchmarks have largely overlooked this crucial distinction, we aim to address this gap and ensure that multimodal mu methods can focus on the unique characteristics of mllms. to erase the memorized visual representation while preserving corresponding factual knowledge within mllms, we propose mmunlearner, a geometry-constrained gradient ascent mu method to update the parameters for targeted visual patterns. motivated by the selective unlearning paradigm for visual networks and diffusion model @xcite , we further extend it to mllms, with an appropriate saliency map (depicted by fisher matrix in parameter space) designed for each module. extensive ex-periment show that applying llm-based mu methods to mllms with vqa data adjusts textual factual knowledge solely, whereas mmunlearner can efficiently remove the visual patterns while maintaining factual knowledge. our findings offer valuable insights into multimodal intelligence in the era of artificial general intelligence (agi). our contributions can be summarized as follows: ❶ we are the first to formulate the setting of multimodal machine unlearning based on the characteristics of mllm architecture during unlearning and evaluation. our focus is to erase the memorized visual representation while preserving corresponding factual knowledge. ## our proposed mmunlearner 3.1 task setting to enable a text-only llm l to comprehend visual context, mainstream approaches extract visual embeddings h i using a vision encoder v followed by a projector w. the entire model is then fine-tuned with visual instruction data {x i , x q , x a }, where x i represents the input image, x q is the textual instruction, and x a denotes the expected answer of length s. this can be formalized as follows: the objective of multimodal mu is, therefore, to eliminate these learned associations between a specific concept and its corresponding visual patterns. in other words, the unlearned model should behave as if it has never encountered the related images during the visual instruction tuning process. specifically, for a given concept c, its image representation x i , and the extracted visual embeddings h @xmath0, the unlearned model must satisfy the conditions in box 3.1. in standard unlearning tasks, the unlearned model is expected to maintain its knowledge on a retained set, which we denotes in next section 3.2. we illustrate the expected behavior of the targeted model using biographical examples, though our task formulation can be extended to other domains, such as real-world entities and landmarks. ## conclusion in this paper, we reformulate the task of mu tailored for mllms, a field still in its early stages. our proposed setting aims to erase targeted visual concepts in mllms while preserving untargeted knowledge. to address this challenge, we further propose a novel weight saliency-based unlearning method, mmunlearner, which selectively updates parameters crucial for the forgetting objective while protecting parameters essential for retaining untargeted knowledge. our experiments demonstrate that directly transferring llm-oriented mu methods to vqa data is insufficient for mllms; whereas our proposed mmunlearner exhibits a strong ability to remove visual concepts while preserving textual knowledge. further experiments validate the effectiveness and robustness of our approach. we believe that mmunlearner will lay a solid foundation for building a trustworthy mllm ecosystem to achieve ultimate agi.