Forget for Get: A Lightweight Two-phase Gradient Method for Knowledge Editing in Large Language Models (2025)

Abstract: Recent studies have highlighted the remarkable knowledge retention capabilities of Large Language Models (LLMs) like GPT-4, while simultaneously revealing critical limitations in maintaining knowledge currency and accuracy. Existing knowledge editing methodologies, designed to update specific factual information without compromising general model performance, often encounter two fundamental challenges: parameter conflict during knowledge overwriting and excessive computational overhead. In this paper, we introduce ForGet (Forget for Get), a novel approach grounded in the principle of “forgetting before learning”. By pinpointing the location within the LLM that corresponds to the target knowledge, we first erase the outdated knowledge and then insert the new knowledge at this precise spot. ForGet is the first work to leverage a two-phase gradient-based process for knowledge editing, offering a lightweight solution that also delivers superior results. Experimental findings show that our method achieves more effective knowledge editing at a lower cost compared to previous techniques across various base models.

## introduction large language models (llms) have revolutionized natural language processing, enabling unprecedented capabilities in language comprehension and generation @xcite @xcite . a key factor behind these capabilities is the vast amount of knowledge embedded within these models. however, this knowledge is often static, leading to issues such as outdated information, inaccuracies, and potential privacy violations. for instance, answering 'who is the president of the united states?' in 2024 yields 'joe biden,' but this response becomes incorrect in 2025 if the model is not updated. knowledge editing is * corresponding author proposed to address this problem. knowledge editing aims to modify the specific knowledge stored in llm without affecting other irrelevant knowledge and maintaining a low computational cost @xcite . existing knowledge editing methods can be broadly categorized into three classes @xcite . some of the methods utilize an additional knowledge base to store edits @xcite @xcite , some methods use in-context learning @xcite , others first decide the location to edit then perform editing at the specific location @xcite @xcite @xcite . the existing methods have largely succeeded in editing the knowledge stored in llms. these methods have achieved good editing effects, but there are still some challenges. one of the issues is that it's difficult to address the old knowledge when inserting new knowledge effectively. when editing knowledge in llms, conflicts between new and old knowledge may arise, which can hinder the model's ability to learn new information @xcite and weaken the effect of editing. just like humans, it is difficult to change old knowledge when it has become deeply ingrained. another challenge is the resource consumption of the methods. some methods are highly effective, but they require additional time, storage, and computational resources. this also makes the model more complex and reduces the convenience of de-ployment of the method. therefore, knowledge editing methods should become more lightweight while ensuring editing effectiveness. in order to resolve knowledge conflicts, a straightforward approach is to forget the old knowledge before learning the new knowledge. for example, before going to suzhou, one should first remove the luggage packed for miami from the suitcase and then pack the luggage prepared for suzhou. inspired by the human cognitive mechanisms where forgetting old information is a prerequisite for learning new information @xcite , we propose a method named forget (forget for get). first of all, critical mlp modules are found out by the knowledge circuits determined by target knowledge. knowledge editing is then performed on these critical mlp modules while the rest of the model remain unchanged. during the editing process, we first apply gradient ascent to these modules to eliminate the old knowledge, which is defined as the forgetting process. after the forgetting process, we use gradients descent to insert new knowledge into the model. to the best of our knowledge, this is the first work to leverage gradient ascent and descent for knowledge editing, offering a lightweight and efficient solution to the problem of knowledge conflicts. our method has no additional components or training processes, and the amount of pre-computation is minimal, making it lightweight and plug-and-play. the main contributions of this work can be summarized as follows: ## related work the existing methods can be roughly divided into three categories: methods with additional memories, methods learning from examples and methods modifying components directly @xcite . ## task definition our task is to edit knowledge within llms precisely. as equation 1 shows, given the target knowledge k and original model f with parameters θ, our goal is to design a method f () that performs the necessary edits to produce an updated model f ′ with parameters θ ′ . editing knowledge precisely means that only the knowledge within editing scope will be edited and others should not be affected, as equation 2 shows. the editing scope refers to a set of inputs related to the target knowledge that requires editing @xcite . for example, after the 2024 u.s. election, the answer to "who is the president of the united states" should be changed from "biden" to "trump", but the answer to "who is the president of russia" should remain "putin" both before and after editing. ## method: forget in this section, we are going to introduce our method for knowledge editing: forget (forget for get). instead of making use of additional memories or designing clever demonstrations, we adopted a direct two-phase gradient adjustment, offering a lightweight yet effective solution. the forget framework consists of two main phases: (1) determining the editing locations and (2) performing the editing operations. in figure 3 : after determining the editing location, only the modules within editing location are trainable in later process. first, gradient ascent is performed to eliminate old knowledge, followed by the opposite gradient descent to acquire new knowledge. the first phase, we identify the components of the model that are most relevant to the target knowledge requiring editing. the second phase occurs at the editing location identified in the first phase. we begin by using gradient ascent to forget the old knowledge, followed by gradient descent to acquire the new knowledge. ## experiments results the experimental results for knowledge in counterfact and zsre are presented in our method forget, as the most lightweight method among the methods we present, also delivers satisfactory performance. among the mp methods of the same category, forget achieves the highest total score while maintaining a good balance among various indicators, avoiding poor performance in any specific metric. in most cases, forget achieves the highest locality score among mp methods (only excepting ft in a few settings), demonstrating its exceptional capability to preserve unrelated knowledge and minimize unintended side effects. however, while some mp baselines like ft occasionally achieve high locality, they often suffer from drastic degradation in efficacy and generalization (e.g., ft with only 19.12 locality in counterfact on llama-2-7b), suggesting unstable or incomplete edits. in contrast, forget has competitive and more balanced efficacy and generalization scores, which results in a higher overall score. this indicates that forget effectively balances editing and preservation. ## ablation study to verify the effectiveness of each component of our method, we also conduct ablation experiments and show the results in first, we investigate the impact of editing location on editing performance. as shown in finally, we evaluate the effectiveness of gradient ascent in forget. jang et al. have demonstrated that gradient ascent is an effective method for making models forget private information @xcite . to evaluate the effectiveness of gradient ascent in forget, we replace the forgetting method of forget with that of f-learning, while keeping the rest of the parts unchanged (see appendix b.2 for experimental details). we denote this version of forget, which utilizes the forgetting method of f-learning, as forget-pa. as shown in ## further analysis of forget 8.1 case study to further illustrate the effectiveness of forget, we present its performance across three metrics using selected samples. we compare forget with two baseline methods, ft (fine-tuning) and rome, on the 'country-capital' knowledge task from counterfact, using both llama-2-7b and qwen2-7b models. one of the cases is shown in the observed issues can be summarized as a mismatch between the extent of the forgetting and learning processes. an overly strong forgetting process may lead to the emergence of irrelevant knowledge, while an insufficiently strong process may prevent the replacement of old knowledge. ## conclusion in this work, we introduced forget (forget for get), a novel knowledge editing method inspired by the cognitive principle of "forgetting before learning". our method consists of two core stages: locating and editing, where the editing parts necessitates first forgetting the old knowledge and subsequently acquiring the new knowledge. our experimental results demonstrate that forget balances the editing of target knowledge with the preservation of unrelated knowledge, achieving competitive performance across multiple benchmarks. as a lightweight, plug-and-play solution that requires no additional training or extensive pre-computation, forget is capable of editing model effectively.