Decoupling Memories, Muting Neurons: Towards Practical Machine Unlearning for Large Language Models (2025)

Abstract: Machine Unlearning (MU) has emerged as a promising solution for removing the influence of data that an owner wishes to unlearn from Large Language Models (LLMs). However, existing MU methods, which require tuning the entire model parameters on the unlearned data with random labels or perturbed gradients, significantly degrade model utility, especially given the difficulty of accessing the original training data. This presents a key challenge: how can we achieve MU using only the unlearned data while preserving model utility?In this paper, we propose NeuMuter, a simple but effective MU method that eliminates the influence of unlearned data from LLMs by modulating the outputs of merely 1% of the neurons in the feed-forward network (FFN) modules within the Transformer blocks, minimizing disruption to the model’s performance. We design a trainable masking scheme that decouples the memorization of different training data within the neurons of LLMs, allowing us to precisely identify and modify neurons associated with the unlearned data. Through comprehensive evaluations on two benchmarks across four different LLMs, we demonstrate that modifying the outputs of a few fraction of the total neurons can effectively achieve MU while preserving the model’s utility across downstream tasks.

## introduction large language models (llms) have achieved remarkable success, primarily due to their pretraining on vast amounts of text corpora, sourced from the internet @xcite @xcite . when the training data contains unauthorized user information, practitioners are required to remove it from well-trained llms upon request, as individuals are granted the "right to be forgotten" (rtbf) @xcite under regulations such as the eu's general data protection * corresponding author. regulation (gdpr) @xcite and us's california consumer privacy act (ccpa) @xcite . to address this issue, machine unlearning (mu) has emerged as a promising paradigm, which aims to eliminate the effects of specific training samples from llms while preserving the remaining knowledge. given the massive scale of llms, retraining from scratch is impractical, making the development of efficient unlearning methods for llms a significant challenge. most existing mu methods for llms require fine-tuning the entire model parameters by relabeling the unlearned data or reversing the contribution of gradient descent on the unlearned data @xcite @xcite @xcite @xcite @xcite @xcite . for example, mu methods has been proposed that focuses on solely leveraging the unlearned data. for example, gradient ascent methods @xcite aim to maximize the loss of the unlearned data, whereas data relabeling @xcite seeks to minimize the loss of the relabeled unlearned data. however, modifying the entire parameters of llms based solely on limited unlearned data often results in catastrophic forgetting, where removing specific data negatively impacts the model's utility @xcite . to mitigate this, other methods typically involve additional fine-tuning on the remaining training data (c.f. the original and the fine-tuned models. in practice, long-term storage of llm training data is often infeasible due to high maintenance costs, extensive storage requirements, and privacy concerns @xcite . consequently, these mu methods struggle to leverage the remaining training data to maintain model utility. therefore, a fundamental gap in mu for llms remains: how can we achieve mu using only the unlearned data while preserving model utility? to bridge this gap, we propose neumuter, a simple yet effective mu method that modifies the outputs of merely 1% of the neurons in the feed-forward network (ffn) modules within the transformer blocks, which are associated with the data to be unlearned. the key insight behind neumuter is that specific neurons within the ffn modules play a crucial role in retaining the memory of training data @xcite . instead of altering the entire model parameters, we selectively adjust only the neurons responsible for memorizing the unlearned data while preserving the vast majority of the network's parameters. this targeted intervention enables effective mu while avoiding severe utility degradation, eliminating the need for additional fine-tuning on the remaining training data. to leverage this, we introduce a trainable neuron mask within ffn modules to decouple memorization and precisely identify neurons related to the unlearned data. unlike existing neuron localization methods @xcite @xcite , which primarily focus on identifying neurons linked to specific keywords, our approach targets the neurons most critical for unlearning entire sentences. recognizing that different neurons contribute variably to the unlearned data, we utilize the mask values as scaling factors for the outputs of the selected neurons, rather than directly deactivating the selected neurons. by adaptively modulating the outputs of these neurons, we can effectively eliminate the model's responses related to the unlearned data while maintaining the overall utility of the llms. our contributions are summarized as follows: ## design of neumuter neumuter modifies the intermediate outputs of a small subset of llm neurons associated with the unlearned data, therefore it consists of two main phases (c.f. figure 1 ): memorization neuron localization and memorization removal. ## evaluation on tofu benchmark to further evaluate neumuter, we utilize the recently introduced tofu benchmark and compare it against the mu methods included in the benchmark. more details are shown in appendix d. figure 5 presents the unlearning results in terms of forget quality and model utility for synthetic unlearned profiles at 1%, 5%, and 10%. the results show that neumuter outperforms all baselines across different scenarios, closely matching the mu performance of the retrained model. our findings demonstrate that effective mu can be achieved without compromising model utility and without relying on the remaining training set. it can preserve model utility, especially when the unlearned set is larger (e.g., 5% or 10%). neumuter also achieves higher p-values in forget quality, indicating better alignment with the retrained model. for the 1% unlearned set, all baselines show near-vertical trajectories, suggesting easier unlearning. however, their forget quality is still lower than neumuter's. as the unlearned data increases, unlearning trajectories become unstable. with 5% unlearned data, ga and kl improve forget quality but at a significant cost to utility, making them difficult to fine-tune. at 10%, gd achieves forget quality similar to neumuter but struggles with utility. overall, methods using the retained set (e.g., ga) perform better than those focusing on minimizing unlearned set loss. ## conclusion in this paper, we introduce neumuter, a practical mu method that selectively modifies the intermediate outputs of merely 1% of the neurons in ffn modules of llms. neumuter employs a trainable masking mechanism within the ffn modules to accurately identify neurons that store the memorization of unlearned data. by modulating the outputs of these neurons, neumuter effectively removes undesired information while preserving model utility-without requiring tuning the entire parameters of llms on the remaining training data. extensive evaluation on nine datasets across four llms demonstrates that neumuter not only achieves superior mu performance but also preserves model utility compared to existing mu methods.