Protecting Privacy in Multimodal Large Language Models with MLLMU -Bench (2025)

Abstract: Generative models such as Large Language Models (LLM) and Multimodal Large Language models (MLLMs) trained on massive web corpora can memorize and disclose individuals’ confidential and private data, raising legal and ethical concerns. While many previous works have addressed this issue in LLM via machine unlearning, it remains largely unexplored for MLLMs. To tackle this challenge, we introduce Multimodal Large Language Model Unlearning Benchmark (MLLMU-Bench), a novel benchmark aimed at advancing the understanding of multimodal machine unlearning. MLLMU-Bench consists of 500 fictitious profiles and 153 profiles for public celebrities, each profile feature over 14 customized question-answer pairs, evaluated from both multimodal (image+text) and unimodal (text) perspectives. The benchmark is divided into four sets to assess unlearning algorithms in terms of efficacy, generalizability, and model utility. Finally, we provide baseline results using existing generative model unlearning algorithms. Surprisingly, our experiments show that unimodal unlearning algorithms excel in generation tasks, while multimodal unlearning approaches perform better in classification with multimodal inputs.

## introduction the rapid development of large language models (llms) @xcite @xcite and multimodal large language models (mllms) @xcite @xcite has played a dominant role in both nlp and multimodal applications @xcite @xcite @xcite , largely due to their extensive pre-training on vast copora and their exceptional general reasoning abilities. however, this 1 code is available at franciscoliu/mllmu-bench. ## experimental results in this section, we present a comprehensive comparison of different unlearning algorithms in three unlearning setups against the vanilla model, finetuned on the full data d for 3 epochs. details of the fine-tuning process for the vanilla model can be found in appendix b.2. ## discussion our curated benchmark offers a valuable tool for evaluating the practical applicability of unlearning algorithms in mllms. in this section, we address two critical questions that are essential to further promoting the field of mllm unlearning. ## conclusion the introduction of the mllmu-bench benchmark represents a significant step toward implementing unlearning algorithms that simulate realworld scenarios. by assessing unlearning algorithms across three key dimensions -unlearning effectiveness, unlearning generalizability, and model utility-mllmu-bench provides a comprehensive framework for assessing their performance. additionally, we conduct heuristic experiments to examine the performance of unlearning algorithms in both multimodal and unimodal setups. our findings indicate that methods lacking a modality-aware design fail to achieve consistent unlearning performance across both multimodal and unimodal evaluation settings. simply modifying input types to different modalities proves insufficient, often resulting in incomplete knowledge removal across modalities and unintended knowledge degradation in unimodal scenarios. these challenges highlight the need for more advanced multimodal unlearning approaches to address the inherent complexities of mllm unlearning. lastly, we present a systematic analysis of the trade-offs between unlearning effectiveness and model utility, offering valuable insights from multiple perspectives.