CLEAR : Character Unlearning in Textual and Visual Modalities (2025)

Abstract: Machine Unlearning (MU) is critical for removing private or hazardous information from deep learning models. While MU has advanced significantly in unimodal (text or vision) settings, multimodal unlearning (MMU) remains underexplored due to the lack of open benchmarks for evaluating cross-modal data removal. To address this gap, we introduce CLEAR, the first open-source benchmark designed specifically for MMU. CLEAR contains 200 fictitious individuals and 3,700 images linked with corresponding question-answer pairs, enabling a thorough evaluation across modalities. We conduct a comprehensive analysis of 11 MU methods (e.g., SCRUB, gradient ascent, DPO) across four evaluation sets, demonstrating that jointly unlearning both modalities outperforms single-modality approaches. The dataset is available at [link](https://huggingface.co/datasets/therem/CLEAR)

## introduction large language models (llms) @xcite @xcite are increasingly investigated for memorizing private, unethical, or copyrighted data during training. recently, machine unlearning (mu) methods have been applied to mitigate issues related to toxicity @xcite , copyright and privacy concerns @xcite @xcite and fairness @xcite . mu has emerged as a promising alternative to costly retraining, existing methods focus almost exclusively on single-modality models. recent work has studied unlearning in llms @xcite @xcite or vision models @xcite @xcite , but unlearning in multi-modal language models remains largely unexplored. this leaves a critical gap: multimodal llms (mllms), which process both visual and textual data, introduce unique challenges for unlearning. for instance, sensitive information may persist across modalities even after removal from one (e.g., a face linked to a name), and unlearning in one modality could degrade performance in another. despite these risks, no open benchmarks exist to evaluate mu in multimodal settings. recently, @xcite pioneers the investigation of unlearning configurations in visuallanguage models (vlms) to mitigate cross-modal safety risks, its experimental framework inherits a critical limitation: the datasets used in this study (e.g., pku-saferlhf @xcite , jailbreakv-28k @xcite ) were designed for safety alignment and truthfulness evaluation, not machine unlearning (mu). this mismatch conflates safety fine-tuning (suppressing harmful outputs) with targeted data removal (erasing specific knowledge traces), potentially overestimating mu efficacy. to address this, we propose clear, the first publicly available benchmark for machine unlearning in multimodal (textual-visual) models. our u: u: who is aurelio beltran? who is this? model: i don't know. i don't know. model: idk dpo scrub 1 2 3 scrub twins rmu 1 2 3 model figure 1 : our dataset clear includes 3,770 visual image-caption pairs and 4,000 textual question-answer pairs related to fictional characters. we apply multimodal unlearning to remove specific information, subsequently assessing the quality of unlearning and the models' performance using various metrics. finally, we compile a leaderboard of unlearning methods based on these evaluations. our contributions and findings are as follows: we make all data publicly available 1 . machine unlearning. the concept of machine unlearning was initially presented by @xcite . in general, mu methods @xcite @xcite @xcite remove the impact of specific data points from a trained model without requiring full retraining. the goal is to obtain a model that behaves like the forget data was never part of the training set. recently, textual unlearning in generative language models has attracted attention. @xcite propose a benchmark named tofu for textual llm unlearning, consisting of 200 fictitious author profiles defined by attributes like name, birthplace, parent's names, occupation, and written books, totaling 4,000 question-answer pairs (20 per author). wmdp @xcite includes 3,668 multiplechoice questions to evaluate and benchmark the unlearning of hazardous knowledge in llms. to remove knowledge from generative models, jang et al. (2023) employ gradient ascent on specific target sequences. eldan and russinovich (2023) focus on the particular case of unlearning the harry potter books from llama2-7b. yao et al. (2023) utilize machine unlearning to address harmful responses and eliminate hallucinations. yao et al. (2024b) examined the unlearning of 2,000 github code files, 500 books, and 500 academic papers from yi-6b. however, these studies have been restricted to text-only contexts. our research investigates the multimodal aspects of unlearning. real faces q: the name of person on the photo is... a: taylor swift. q: what is happening in the image? a: jaime vasquez, a true crime author, poses for a portrait. q: what is happening in the image? a: takashi nakamura, with a red rose and mount fuji in the background. ## clear in this section, we describe a new benchmark clear designed for character unlearning. as a basis, our dataset utilizes the text-only tofu dataset @xcite within the same experimental setup to replicate a real-world scenario where privacy concerns arise in sensitive contexts. while external information from books, games, or movies is general knowledge to unlearn @xcite @xcite , character unlearning deals with specific contextual data that directly impacts individuals. this task addresses removing personal or confidential details and enhancing user privacy. ## experimental setup and evaluation in this section, we briefly discuss the evaluation metrics and implementation details. ## results in the following sections, we describe the results of mu methods on our dataset. we seek to answer the following research questions: rq1: does an unlearning method's performance on a single domain transfer directly to the performance in a multimodal setting? should we study multimodal unlearning at all if we can easily predict its performance from singledomain experiments? rq2: is unlearning only one modality (textual or visual) enough in a multimodal setup? how does the effectiveness vary depending on modality? how do different unlearning methods compare in their effectiveness for unlearning specific modalities? rq3: in the context of multimodal unlearning, what methods perform the best? ## conclusion in this work, we introduce clear, the first opensourced benchmark designed to assess machine unlearning in a textual-visual multimodal setup. our evaluation of existing unlearning techniques across domains shows that multimodal unlearning is more challenging than previously anticipated, laying the ground for further research. our findings offer a new perspective than earlier results on safety alignment @xcite , which suggested that text-only unlearning is sufficient for multimodal models. while clear's synthetic personas ensure controlled evaluation, real-world data (e.g., diverse facial features, noisy captions) may introduce new challenges. additionally, our study focuses on visual-language models, leaving other modalities (e.g., audio, video) unexplored. by open-sourcing clear and establishing the first multimodal mu leaderboard, we aim to accelerate progress toward ethical, privacy-preserving multimodal ai. our findings highlight that mmu is not merely an exten-sion of unimodal unlearning but a distinct challenge requiring novel methodologies.