Not Every Token Needs Forgetting: Selective Unlearning Balancing Forgetting and Utility in Large Language Models (2025)

Abstract: Large Language Model (LLM) unlearning has recently gained significant attention, driven by the need to remove unwanted information—such as private, sensitive, or copyrighted content—from trained models. However, conventional unlearning approaches indiscriminately update model parameters to forget all tokens in a target document, including common tokens (e.g., pronouns, prepositions, general nouns) that carry general knowledge. In this paper, we highlight that “not every token needs forgetting”. We propose **Selective Unlearning (SU)**, which identifies a critical subset of tokens within the forgetting set that is relevant to the unwanted information, and unlearns only those tokens. Experiments on two benchmarks and six baseline unlearning algorithms demonstrate that SU not only achieves effective unlearning on the targeted forget data, but also significantly preserves the model’s utility in the retaining set.

## introduction text corpora used to train large language models (llms) often contain sensitive, private, or copyrighted content. to address the risks posed by such data, recent research has explored llm unlearning-aims to remove specific unwanted knowledge from a model without incurring the cost and effort of retraining from scratch. existing unlearning approaches typically apply the same unlearning loss to every token in the targeted documents. however, as illustrated in figure 1 , this approach forces the model to unlearn not only sensitive information but also general concepts. even benign tokens like "that" or "she" in the target forget documents are unlearned, unnecessarily degrading the model's language capabilities. motivated by this, we contend that not every token needs forgetting: an unlearning method should selectively target only tokens that encode unique information in the forget set. to this end, we inred blocks indicate unlearned tokens, on which the forgetting loss is calculated. su avoids the forgetting of general information like "that", therefore preserving model utility. ## selective unlearning we introduce selective unlearning (su), which selectively unlearns a subset of tokens with information unique to the forget set. we apply su to do selective gradient ascent on models, while at the same time using gradient descent on all retain data to better preserve model performance. figure 2 provides an overview of our su framework. ## experiments we demonstrate the effectiveness of su through experiments on 6 baselines and 2 benchmarks. ## conclusion in this paper, we introduce selective unlearning (su), a novel framework that selectively erases essential tokens with forget set-specific information, question: what is the length of time for which the treherbert line will close as part of the valleys network electrification? ground truth: around ten months. method response ga (empty) ga+kl 13 months from 19 march 2023 npo (empty) npo+gd 15 months. npo+kl 13 months from 19 march 2023. su (llm) 10 months. question: according to ms cullen, what was the status of all the exemptions requested by great ormond street hospital? ground truth: all of the exemptions that great ormond street requested of the royal college of nursing were granted earlier yesterday. method response ga (empty) gd 10 were granted and three denied. ga+kl 18 had been granted and 222 had been turned down. npo (empty) npo+gd 18 out of 19 were approved. npo+kl 18 had been granted and 2 were in the process of being granted. su (llm) 100% granted.