Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models (2024)

Abstract: Recent advancements in Large Language Models (LLMs) have showcased their remarkable capabilities in text understanding and generation. However, even stronger LLMs are susceptible to acquiring erroneous or obsolete information from the training corpus. Direct secondary fine-tuning with data containing new knowledge may be ineffective in updating knowledge due to the conflict between old and new knowledge. In this paper, we propose a new paradigm for fine-tuning called F-Learning (Forgetting before Learning), which employs parametric arithmetic to facilitate the forgetting of old knowledge and learning of new knowledge. Experimental results on two publicly available datasets demonstrate that our proposed F-Learning can obviously improve the knowledge updating performance of both full fine-tuning and LoRA fine-tuning, simultaneously outperforming the existing baselines in most cases. Moreover, we have also discovered that forgetting old knowledge by subtracting the parameters of LoRA can yield a similar effect to subtracting the parameters of full fine-tuning, and occasionally even surpass it significantly.

## introduction large language models (llms) possess an extraordinary ability to understand and generate natural language @xcite @xcite . although llms are very capable of learning, they are not immune to the acquisition of incorrect knowledge in the corpus. moreover, much of the knowledge in the real world is constantly updated, and some of the originally correct knowledge in llms can become outdated and invalid over time. for example, the question "who is the president of the united states? is answered "donald trump" in the year 2020, while the answer now is "joe biden". consequently, the challenge with llms is continuously updating to ensure they reflect current, correct knowledge. existing methods of model editing and knowledge updating usually add additional network @xcite @xcite , model parameters @xcite @xcite , knowledge bases @xcite @xcite @xcite , etc., and the editing process is not as straightforward and simple as fine-tuning methods @xcite @xcite directly with new knowledge. currently, the most used method for learning new knowledge is still direct fine-tuning of the model. empirically, when human beings establish their own initial cognition, if they are exposed to new knowledge that is inconsistent with their initial cognition, they usually feel conflicted and it is difficult for them to learn and accept the new knowledge. if the original cognition and knowledge are forgotten, then the new knowledge to be learned will not conflict with the original cognition and knowledge, which makes it better to learn and absorb the new knowledge. as shown in figure 1 , it is better to pour in the "new water" only after the "original water" in the cup has been poured out. for example, if people have been educated to believe that "the earth is flat" since childhood, it would be challenging for them to accept the conflicting knowledge that "the earth is round" when they become adults. conversely, if they can forget the erroneous knowledge that "the earth is flat" or if they learn the correct knowledge that "the earth is round" before being exposed to the incorrect information, it would be much simpler. inspired by the above empirical observations and @xcite 's task arithmetic, we propose a novel paradigm of knowledge updating called f-learning (forgetting before learning). specifically, we first fine-tune the initial model using old knowledge and then subtract the difference between the fine-tuned model parameters and the initial model parameters from the initial model parameters. this process is defined as "old knowledge forgetting". we then use the new knowledge to finetune the model after forgetting the old knowledge. this process we define as "new knowledge learning". after the two stages of forgetting old knowledge and learning new knowledge, the model's knowledge is updated. the contribution of this work can be summarised as follows: ## related work currently, the method of knowledge updating and model editing (also known as knowledge editing) for llms is mainly divided into two classes @xcite : a. the method preserving model's parameters retrieve augmentation practically depends on an external knowledge base which contains new or correct knowledge. aiming at amending the output of llms, a new knowledge base will be connected with the base model to implement a retrieve for needed new knowledge to a prompt or a question @xcite @xcite . mitchell et al. @xcite store manual edits in a memory module, and use a classifier to call the knowledge stored in the memory. madaan et al. @xcite leverage the memory of user's feedback to generate prompts for llms. instead of gradient calculation, zheng et al. @xcite utilize the in-context learning method to revise the output of llms with demonstrations extracted from the corpus based on similarity. adding additional parameters refers to injecting a few trainable parameters which represent new knowledge to llms while original parameters keeping frozen @xcite @xcite . dong et al. @xcite put forward a lightweight feed-forward network to add new parameters adapted to specific factual contexts for knowledge generalization. huang @xcite et al. design an editor called transformer-patcher, which is capable of modifying the mistake of llms sequentially by adding and training a few neurons in transformer. recently, there are a series of parameter-efficient fine-tuning methods, such as prefix-tuning @xcite and @xcite , making it more appreciate for knowledge editing based on fine-tuning. zhang et al. @xcite operate incremental parameter updates of different amounts by calculating the importance of the weight matrix to improve the update efficiency and adaptability. zhu et al. @xcite ) leverage a loss constraint attached to the base model to reduce the impact on irrelevant knowledge during the process of fine-tuning. similarly, lee et al. @xcite also implement large-scale continual learning for knowledge updating with regularized fine-tuning. meta-learning is aimed at updating the knowledge in llms through varying their parameters with the prediction from a well-trained hypyernetwork @xcite @xcite . mitchell et al. @xcite propose an auxiliary network with gradient decomposition, which can execute efficient edits to llms according to a single input-output pair. de cao et al. @xcite update part of weights for a subset of modules in the model relying on a hypernetwork with constrained optimization. locate and edit is related to the internal mechanism of the llms. with the help of some attributes, it usually locates the parameters and neurons in the light of specific knowledge and modifies them to correct the output @xcite @xcite @xcite . geva et al. @xcite find that the feed-forward networks layer of the transformer stores key-value pairs which are related to specific knowledge. meng et al. @xcite utilize a causal reasoning method to distinguish the key neuron activations and update specific factual associations by modifying feed-forward weights. furthermore, to implement knowledge editing on a large scale, they put forward memit @xcite , a method that directly updates thousands of memories in llms. gupta et al. @xcite improve the knowledge updating through varying edit tokens and ameliorating the layer selection during the editing process. yu et al. @xcite leverage the partitioned gradient to identify the significant weights for unlearning of bias in the model. in conclusion, there are many ways to achieve knowledge updating, but most of them require the addition of additional knowledge bases, neural network modules, and model parameters, which are cumbersome in practice and increase inference consumption. this paper focuses on the improvement and enhancement of fine-tuning methods. ## task definition our task is knowledge updating of large models, which can be defined as given a model f θ and a set of input-output knowledge pairs k @xmath0, y 1 ), (x 2 , y 2 ), ..., (x i , y i )}, the parameters of the model need to be edited to obtain a new model f θ * (x) and a corresponding set of new input-output pairs k @xmath1, y new 1 ), (x 2 , y new 2 ), ..., (x i , y new i )}. the i is the number of knowledge pairs to be updated. referring to @xcite , we can define this process and objective of knowledge updating as: ## proposed method: f-learning in this section, we will present our method of knowledge updating for llms. instead of introducing an external knowledge base or additional parameters, our method is mainly based on full fine-tuning and parameter-efficient fine-tuning. briefly, it consists of two stages: ## case study to further illustrate the effectiveness of the proposed method, we present a case study on the results of the knowledge updating by the original model, only forgetting old knowledge and performing f-learning. we selected some cases in the experiment of llama2-7b on zsre dataset, noting that the hyper-parameters λ of the rate of forgetting is set to 0.3. ## conclusion in this paper, we propose a new paradigm of knowledge updating during supervised fine-tuning called f-learning (forgetting before learning), which is based on parametric arithmetic to forget old knowledge and learn new knowledge for eliminating contradictions between old and new knowledge. the