Forget the Token and Pixel: Rethinking Gradient Ascent for Concept Unlearning in Multimodal Generative Models (2025)

Abstract: Gradient Ascent (GA) has emerged as a promising approach for concept unlearning in Multimodal Generative Models (MGMs), such as Multimodal Large Language Models (MLLMs) and Stable Diffusion Models (SDMs). Despite its effectiveness in removing undesired knowledge, GA leads to severe utility degradation in MGMs. In this paper, we explore the mechanism behind this degradation by quantifying two distinct forms of knowledge in MGMs: (i) Conceptual Knowledge, which represents specific information about concepts; (ii) Natural Knowledge, which refers to the ability to produce coherent and logically structured outputs. Our analysis reveals that applying GA globally not only removes the targeted Conceptual Knowledge but also inadvertently diminishes Natural Knowledge, resulting in utility collapse. To address this issue, we propose Forget the Token and Pixel (FTTP), a novel approach that selectively applies GA to targeted Conceptual Knowledge while preserving Natural Knowledge through Gradient Descent (GD). FTTP eliminates the need for additional retain sets and a large number of training steps, thereby reducing computational resource costs. Extensive experiments demonstrate FTTP’s efficiency and superior utility-unlearning tradeoff for both text and image generation tasks. Our source code will be released in the near future.

## introduction multimodal generative models (mgms), such as multimodal large language models (mllms) @xcite @xcite @xcite and stable diffusion models (sdms) @xcite @xcite , have demonstrated impressive capabilities by leveraging two parallel and opposite data flows: mapping and generating concepts from image to text @xcite @xcite , and from text to image @xcite @xcite . through this cross-modal mapping, these models can seamlessly create and interpret representations of diverse concepts, ranging from everyday objects to complex scenarios. however, their ability to generate such concepts also raises significant concerns around privacy, copyright violations, and potentially harmful content, presenting critical security and ethical challenges @xcite @xcite . to address these issues, the field of machine unlearning has emerged (eldan and russinovich, 2023; @xcite @xcite , focusing on removing undesired concepts or knowledge from models. current machine unlearning methods can be broadly divided into two categories. the first category includes bounded optimization methods, such as fine-tuning with random labels @xcite @xcite or attention-based approaches @xcite @xcite . these methods often necessitate the construction of additional fine-tuning datasets, increasing time and resource costs. the second category comprises unbounded optimization methods, such as gradient ascent (ga) @xcite . ga directly adjusts the model's output without the need for additional fine-tuning datasets. intuitively, ga appears as a straightforward solution: just as models learn knowledge by minimizing loss through gradient descent, it seems natural to reverse this process by ascending the gradient. despite its simplicity, ga has been found to severely degrade the model's utility @xcite @xcite . fig. 1 (a ) and (c) demonstrate this side effect across both mllms and sdms. in mllms, applying ga to the whole sentence results in unnatural, repetitive language generation (e.g., generating 'the the the...'). similarly, in sdms, applying ga to an entire image not only removes the intended concept but also distorts the generation of other concepts, leading to unnatural outputs. although mllms and sdms generate two entirely different modalities, applying ga for concept unlearning in both seems to reveal commonalities of utility degration, which raises a question for us: why does ga lead to severe utility degradation in multimodal generative models? to answer this question, it is essential to distinguish between two forms of knowledge in mgms: conceptual knowledge and natural knowledge. conceptual knowledge refers to the specific information the model holds about concepts, such as what an 'airplane' looks like or the meaning of 'president donald trump' in a sentence. on the other hand, the term 'natural' is derived from its roots in natural language processing and natural image generation. by identifying the commonalities between these two modalities, we define natural knowledge as the model's ability to produce understandable, logically structured outputs. this type of knowledge is often reflected by tokens or pixels that do not belong to the concepts to be unlearned, as observed in the model's output data. for instance, in fig. 1 (left), tokens such as 'the,' 'main,' or 'person' generated by the original mllm indicate the model's ability to generate logically structured sentences. removing 'person' would disrupt the coherence and meaning, even though the token itself is unrelated to a specific concept. similarly, as shown in fig. 1 (right), sdms' ability to generate background elements or nontargeted concepts demonstrates its significant utility. when ga is applied globally to training data to unlearn a specific concept, the tokens or pixels related to natural knowledge are also optimized with ga. such application would not only remove the conceptual knowledge but also diminish the natural knowledge, leading to the collapse of the model's generative capabilities. by quantifying these two forms of knowledge, we can more accurately assess the extent and content of unlearning occurring within the model as presented in sec.3.2. to address the issue of utility degradation caused by ga, we propose forget the token and pixel (fttp), a method designed to achieve effective concept unlearning while preserving the model's utility. unlike traditional methods, fttp applies gradient ascent only to specific tokens or pixels related to the target concept, while leveraging gradient descent on other areas to maintain natural knowledge. fttp eliminates the need for additional retain sets and uses minimal training data (1-5 samples) and fine-tuning steps (20), making it faster and more resource-efficient than traditional bounded optimization methods that often require thousands of steps and large training sets. the contributions of this paper are summarized as follows: ## related work unlearning in mllms. machine unlearning (mu) has surged in popularity for large language models (llms) and multimodal llms (mllms) @xcite @xcite @xcite @xcite . existing methods range from using gradient ascent (ga) to eliminate undesired outputs @xcite , aligning pre-trained and fine-tuned knowledge @xcite , to adding lightweight unlearning layers @xcite . some approaches also blend ga with kl-divergence to better regulate output distributions @xcite . siu @xcite further explores erasing visual concepts while preserving generative abilities. unlearning in diffusion models. concept unlearning has also drawn attention in diffusion models @xcite @xcite @xcite . techniques include forget-me-not @xcite , conceptbench, bayesian unlearning @xcite , and attentionbased methods that modify cross-attention scores @xcite . despite progress, precisely removing targeted concepts while retaining the original generative performance remains an open challenge. ## method in this section, we define concept unlearning in multimodal generative models (mgms). we introduce two terms, natural knowledge and conceptual knowledge, which help highlight the shortcomings of gradient ascent (ga), motivating us to refine ga by addressing these issues. mgm concept unlearning refers to the process of systematically removing specific conceptual information from a multimodal generative model while preserving its overall generative capabilities as much as possible. the training dataset is @xmath0, t i )} n @xmath1, where i i represents an image and t i is a text consisting of s i tokens w i 1 , w i 2 , . . . , w i s i . the forgetting set d @xmath2, t c j )} k @xmath3. due to the differences in generated modalities, we formally define concept unlearning in mllm and sdm respectively. ## conclusion in this paper, we revisited gradient ascent (ga) for concept unlearning in multimodal generative models (mgms) and highlighted its limitation of causing utility degradation. we introduced two forms of knowledge in mgms: conceptual knowledge and natural knowledge. our analysis revealed that while ga effectively forgets specific concepts, it also harms natural knowledge, leading to utility collapse. to address this, we proposed forget the token and pixel (fttp), which combines ga for concept-specific elements with gradient descent to preserve utility. experimental results show that fttp reduces training costs and provides a better utility-unlearning balance, ensuring effective unlearning while maintaining mgm utility. our prompt: what's the name of this man in the image? a: the man in the image is elon musk. a: the man in the image is not a notable person. work also promotes ethical ai by balancing concept removal with generative utility and enhancing unlearning efficiency to mitigate privacy risks.