A General Framework to Enhance Fine-tuning-based LLM Unlearning (2025)

Abstract: Unlearning has been proposed to remove copyrighted and privacy-sensitive data from Large Language Models (LLMs). Existing approaches primarily rely on fine-tuning-based methods, which can be categorized into gradient ascent-based (GA-based) and suppression-based methods. However, they often degrade model utility (the ability to respond to normal prompts). In this work, we aim to develop a general framework that enhances the utility of fine-tuning-based unlearning methods. To achieve this goal, we first investigate the common property between GA-based and suppression-based methods. We unveil that GA-based methods unlearn by distinguishing the target data (i.e., the data to be removed) and suppressing related generations—essentially the same strategy employed by suppression-based methods. Inspired by this finding, we introduce Gated Representation UNlearning (GRUN) which has two components: a soft gate function for distinguishing target data and a suppression module using Representation Fine-tuning (ReFT) to adjust representations rather than model parameters. Experiments show that GRUN significantly improves the unlearning and utility. Meanwhile, it is general for fine-tuning-based methods, efficient and promising for sequential unlearning.

## introduction llms have shown remarkable capabilities across various tasks @xcite . a key factor driving the rapid advancement is the availability of web-scale datasets. however, concerns have been raised regarding the use of such large-scale data, as it often includes copyrighted and privacy-protected data @xcite . for instance, the new york times sued openai and microsoft because their ar-ticles have been used in training gpt foot_0 . meanwhile, the data is protected by general data protection regulation (gdpr) @xcite , and the data owners have the "right to be forgotten" @xcite . therefore, it is crucial to implement protections for these datasets. to address this, unlearning has been proposed to remove specific data from llms without requiring full retraining @xcite . the goal is to eliminate the influence of the target data or adjust the model behavior as if it had never encountered the target data. llm unlearning is typically a post-training method, with fine-tuning being widely adopted as an approach. existing fine-tuning based unlearning methods can be roughly divided into two categories. one is gradient ascent-based (ga-based) methods, such as gradient ascent (ga) @xcite and its variants @xcite @xcite @xcite @xcite @xcite . they negate the training impact of the target data by reversing the gradient descent loss. the other, suppression-based unlearning, does not aim to erase learned information directly @xcite @xcite @xcite @xcite . instead, it explicitly tells the model about what constitutes target data and guides it to generate human-preferred outputs while suppressing those related to the target data foot_1 . however, recent evaluations on fine-tuningbased methods reveal that there is a significant trade-off between unlearning and model utility, i.e., the model's ability to respond to normal prompts unrelated to the target data @xcite @xcite . this issue has been widely observed in llm fine-tuning: as the finetuning dataset is small, it is likely to cause overfitting and reduce the general ability @xcite @xcite . although they usually use retaining dataset to preserve the model utility @xcite , its small size could limit the generalization. therefore, we aim to develop a general framework to enhance the utility of fine-tuning-based llm unlearning. however, the two types of finetuning-based methods are defined in totally different ways, posing a challenge in developing such a framework. thus, we design a preliminary study to investigate the common property between gabased and suppression-based methods (section 3). we find that, although ga-based methods appear to be dedicated to negate the training of target data, the final ga-unlearned llms still recognize target data and actually treat it as a signal of unlearning. if target data is in the input, the representations exhibit a distinct pattern compared with the input irrelevant to target data. then unlearned models suppress related generation. this suggests that the ga-unlearned models also operate by distinguishing and suppressing target data, which closely resemble the models by suppression-based methods. inspired by the insights from our preliminary study that both ga-based and suppression-based methods rely on distinguishing target data for unlearning, we introduce gated representation unlearning (grun). grun consists of two plugand-play components designed explicitly for distinguishing and suppression: a soft gate function to distinguish, and a suppression module utilizing representation fine-tuning (reft) @xcite . the reft module fine-tunes the representation instead of the model parameters, which can avoid distorting the parameters to preserve the utility. meanwhile, its strength is controlled by the soft gate function, which further ensures the generation unrelated to the target data remains almost untouched. in essence, the soft gate function selectively activates for target data, while the reft module unlearns by redirecting the embeddings of target-data-related prompts toward suppression. we conduct extensive experiments to examine the effectiveness and efficiency of grun. grun requires a lightweight additional module (less than 0.05 % of the llm's size) and reduces training time by over 95% compared to the original method, yet achieves near-perfect unlearning while maintaining utility. moreover, grun is a general solution adaptable to various fine-tuning-based unlearning methods. our experiments validate this across various models, including llama 3.1 and mistral, as well as across different datasets, such as tofu focusing on the unlearning of fine-tuning data @xcite , and wmdp focusing on unlearning pre-training data @xcite . representation fine-tuning (reft). reft @xcite is a recently proposed parameterefficient fine-tuning method. unlike traditional fine-tuning approaches, which primarily adjust model weights, reft focuses on fine-tuning representations, leveraging the rich semantic information embedded in the representation space to influence subsequent generation. building on the linear representation hypothesis @xcite , which posits that concepts are encoded within linear subspaces of representations, reft learns low-rank linear transformations to refine representations. it achieves this by substituting the intermediate representations-i.e., the outputs of specific transformer layers-at selected layers and tokens. ## preliminary studies in this section, we first introduce the definitions about fine-tuning-based unlearning, and then conduct experiments to investigate their common properties. (a) llama 3.1 by gd (b) llama 3.1 by npo (c) mistral v0.1 by gd (d) mistral v0.1 by npo figure 2: pca visualization and the results of normal q&a mixed and not mixed with target data. pca follows the same operation in figure 1. the rouge-l recalls of retaining data/world fact are listed below each figure. ## experiment in this section, we first conduct the experiments across different models and datasets in section 5.2. then we test the performance under different scenarios including sequential unlearning and attacks in sec. 5.3, and conduct ablations studies in section 5.4 and appendix 5. ## conclusions unlearning aims to remove copyrighted and privacy-sensitive data from llms, but often degrades model utility. we propose grun, a general framework to enhances fine-tuning-based unlearning. grun leverages the shared mechanism between ga-based and suppression-based methods. it uses a soft gate function for distinguishing and a reft-based suppression module to adjust representations. grun improves both unlearning effectiveness and utility, and enables efficient unlearning.