Lock on Target! Precision Unlearning via Directional Control (2025)

Abstract: The unlearning method aims at effectively removing harmful, sensitive, or outdated knowledge without costly retraining the model. However, existing methods suffer from two critical limitations: (1) collateral forgetting, where erasing target data inadvertently removes related but desirable knowledge, and (2) generality forgetting, where aggressive unlearning degrades the model’s general capabilities. To address these challenges, we propose DirectiOn Guide unlEarning (DOGE), a novel method that enables precise knowledge erasure by identifying and leveraging a targeted “unlearning direction” in the model’s parameter space. DOGE first extracts this direction through differential analysis of representations for forgotten and retained samples, pinpointing the exact subspace associated with unwanted knowledge. It then selectively applies updates along this direction, ensuring minimal interference with retained information and general model performance. Experiments across multiple benchmarks demonstrate that Doge achieves state-of-the-art unlearning precision while preserving both related knowledge and general capabilities.

## introduction large language models (llms) have shown revolutionary potential in a wide range of domains, due to their powerful capabilities gained from pretraining on massive internet corpora. however, due to the inevitable presence of harmful data on the internet @xcite or the time-sensitive nature of some information, the removal of specific knowledge from trained models has become a common necessity. thus, llm unlearning has been developed to remove the influence of specific data or knowledge from llms †corresponding author code is available at https://github.com/jackwenx/ doge ## related work the rapid advancement of llm has significantly amplified the importance of unlearning. as these models are trained on vast datasets, they may inadvertently learn harmful content, private data, or materials protected by copyright. this presents risks concerning privacy breaches, legal issues, and potential vulnerabilities to malicious exploitation. to address this, several unlearning techniques have been developed in recent years, aiming to effectively eliminate unwanted information while preserving the model's performance on legitimate tasks. for instance, representation misdirection for unlearning @xcite (rmu) utilizes a dual-objective loss, considering both the necessity to forget and to retain, by selectively modifying intermediate layers to remove detrimental knowledge. gradient ascent directly maximizes the loss on the data to be forgotten. building upon the direct preference optimization @xcite (dpo) framework, negative preference optimization @xcite introduces a negative preference optimization strategy to mitigate the instability issues encountered by ga @xcite . npo reportedly achieves a better trade-off between the effectiveness of unlearning and the model's utility, showing particular promise in scenarios requiring the forgetting of a large proportion of data while maintaining practical usability. gradient differentiation (gd) @xcite employs distinct gradient operations on the datasets intended for forgetting and retention. despite the progress in developing unlearning techniques for llm, several studies have highlighted the inherent vulnerabilities of current approaches, particularly concerning the unintended consequences of knowledge removal. two critical issues that frequently arise are collateral forgetting and the degradation of the model's generalization capabilities. collateral forgetting collateral forgetting, also known as catastrophic forgetting in the context of continual learning, refers to the phenomenon where unlearning specific target knowledge inadvertently leads to the forgetting of related but desirable information. for instance, attempting to remove factual inaccuracies about a certain entity might also cause the model to lose general knowledge or reason-ing abilities associated with that entity's domain. existing methods often struggle to precisely target only the undesirable knowledge, leading to an over-aggressive erasure that impacts the broader knowledge graph embedded within the llm @xcite . the challenge lies in isolating the harmful knowledge without affecting the interconnected web of information that contributes to the model's overall understanding and performance. generality forgetting another significant concern is the impact of unlearning on the model's generalizability. many unlearning techniques (e.g., ga, gd, rmu) involve fine-tuning the model @xcite , which, if not carefully controlled, can result in a decline in performance on tasks unrelated to the forgotten knowledge. this "generality forgetting" or the erosion of the model's utility on benign tasks, is a common trade-off observed in existing unlearning strategies. aggressively removing harmful content can alter the model's learned representations in ways that negatively affect its ability to generalize to new, unseen data or to perform well on standard benchmarks that measure its overall language understanding and generation abilities. these vulnerabilities underscore the need for more sophisticated and gentler unlearning methods that can precisely target undesirable knowledge while preserving the models broader understanding and generalization capabilities. ## problem definition we start with a large language model f θtr with parameters θ tr trained on the dataset d tr . we then define a forget set d f ⊂ d tr and a retain set d @xmath0. our goal is to perform unlearning such that the llm only retains the knowledge described in the retain set d r , while completely removing all knowledge from the forget set d f . in other words, after unlearning, the upper bound of the llms behavior should match that of the target model f θr , which is trained solely on the retain set d r and has never been exposed to the knowledge in the forget set d f . ## preliminaries the transformer architecture, particularly in decoder-only language models @xcite , processes input token sequences through a layered structure to generate contextualized representations. given an input sequence @xmath1, . . . , q n ], the model iteratively refines the hidden representation of each token q i across l layers. let x (l) i denote the hidden state of token q i at the input of layer l. at each layer, this representation is updated as: ## doge methodology as shown in figure 2 , our proposed direction guided unlearning (doge) comprises three components: ## computational cost we evaluated the computational costs of different unlearning methods on the tofu dataset. all experiments were conducted on the forget05 dataset with 2 nvidia a800 gpus for 5 epochs. as shown in ## conclusion in this paper, we present direction guided unlearning (doge), a novel method for achieving precise knowledge erasure in large language models. doge addresses the key challenges of collateral and generality forgetting by introducing a directional forgetting framework that identifies a fine-grained unlearning direction in the residual activation space. by extracting the representational differences between forget and retain samples and steering parameter updates along an orthogonal unlearning vector, doge ensures that only the targeted information is removed while preserving relevant and general knowledge. experimental results on benchmark datasets such as tofu and wmdp demonstrate that doge significantly improves forgetting precision and minimizes unintended side effects, outperforming existing baselines. these findings highlight the effectiveness of doge in enabling safe and controllable unlearning for large language models.