Model Editing at Scale leads to Gradual and Catastrophic Forgetting (2024)

Abstract: Editing knowledge in large language models is an attractive capability that allows us to correct incorrectly learned facts during pre-training, as well as update the model with an ever-growing list of new facts. While existing model editing techniques have shown promise, they are usually evaluated using metrics for reliability, specificity and generalization over one or few edits. We argue that for model editing to have practical utility, we must be able to make multiple edits to the same model. With this in mind, we evaluate current model editing methods at scale, focusing on two state of the art methods - ROME and MEMIT. With the lens of scalability, we evaluate model editing methods for three crucial properties - editing proficiency, fact forgetting and downstream performance. We find that as a model is edited sequentially with multiple facts, it continually becomes less editable, forgets previously edited facts and loses the ability to perform downstream tasks. For ROME and MEMIT, this “forgetting” happens in two phases - an initial gradual but progressive forgetting phase followed by an abrupt or catastrophic forgetting. Both gradual and catastrophic forgetting limit the usefulness of model editing methods at scale - the former makes model editing less effective as multiple edits are made to the model while the latter caps the scalability of such model editing methods. Our analysis also highlights other key limitations of ROME and MEMIT at scale. With our work, we push for better evaluation of model editing and development of model editing methods keeping scalability in mind.

## introduction editing knowledge in large language models (llm) has recently emerged as a sought after capability for natural language processing (nlp) practitioners. it is a known fact that llms memorize some part of their training data @xcite . model editing, sometimes also called knowledge editing foot_0 , is the task of modifying existing knowledge or injecting new facts into the model @xcite , without updating the entire model. while knowledge in llms can be updated by continually pre-training models on new facts, this is not always viable due to the huge costs of training llms and an evergrowing list of facts. even then, llms do not perfectly memorize training data @xcite and will inevitably memorize facts incorrectly, thus requiring the capabilities of model editing. this is what makes model editing a very useful tool in the arsenal when working with llms. various methods have been proposed over the years to do so. some of these methods @xcite require training a hypernetwork @xcite ) that generates new weights for the model being edited. other methods @xcite directly update specific parts of the model after locating stored facts inside it @xcite . the success of knowledge editing is usually evaluated along three dimensions @xcite ) -(i) reliability, which measures if the post-edit model accurately recalls the newly edited fact, (ii) generalization, which measures if the post-edit model accurately recalls the newly edited facts for different phrasings of the edited fact, and (iii) locality (also known as specificity @xcite ), which measures if the edited fact changes unrelated or neighboring facts. most prior works @xcite @xcite @xcite focus on evaluating model editing performance by making one edit at a time. while some recent works @xcite @xcite dataset name query zsre who was the architect of villa kampen? counterfact porto is a twin city of in this paper, we present a framework to study the dynamics of model editing at scale. we make multiple edits to the same model sequentially and analyze the effects of these edits on the model as a function of the number of edits. specifically, we compare model editing at scale for three of the most popular model editing methods -mend @xcite , rome @xcite and memit @xcite against a finetuning baseline, and make multiple edits on the same model. along with the usual metrics for evaluating model editing, we propose evaluating postedit models for three important properties -editing proficiency, fact forgetting, and downstream task performance. we find that rome and memit outperform mend and fine-tuning baselines at scale, but edits made by rome and memit are not as localized as previously believed to be. we show that new edits consistently bleed into other facts stored in the model. we also find the model editing methods are prone to gradual and catastrophic forgetting. to the best of our knowledge, our work is the first to associate model editing methods with catastrophic forgetting. we define gradual forgetting as a progressive loss of ability of the model to perform its regular functions as the model is continuously modified through knowledge edits. this includes forgetting the ability to recall previously edited facts as well doing downstream tasks. for rome and memit, gradual forgetting is followed by sudden or catastrophic forgetting @xcite , where the model gets crippled due to a single update made to the model. we name these edits -disabling edits. a disabling edit decapitates the model -new knowledge edits are no longer successful on the model, the model forgets all previously edited facts and is unable to perform any downstream tasks. we also study different properties of these disabling edits. with this paper we highlight some serious limitations of current model editing techniques, especially their lack of robustness when scaled and call for further research in developing scalable model editing methods. our code can be found here. ## methods, models and datasets in this paper, we focus on two prominent model editing methods in literature -rank-one memory editing (rome) @xcite and mass-editing memory in a transformer (memit) @xcite , whereas model editor networks using gradient decomposition (mend) @xcite and fine-tuning are used as baselines. mend is a hypernetwork @xcite based model editing method that generates the weight updates for the model being edited. rome and memit first localize knowledge within a model using causal tracing @xcite and then update the weights of the selected layers to inject knowledge. the major difference between rome and memit is that while rome works under the assumption that knowledge in llms can be updated by updating a single layer, memit updates the weights of multiple layers. we evaluate these model editing methods over two models -gpt2-xl (1.5b) @xcite and gpt-j (6b) @xcite . note that all prior works @xcite edit base language models and not chat models. we refer the reader to appendix 5 for a detailed survey of model editing techniques. this section will be added to the main paper upon acceptance. knowledge editing is usually evaluated on two datasets -the zsre (zero-shot relation extraction) dataset @xcite and the counterfact dataset @xcite . the zsre dataset contains facts in the form of question-answer (qa) pairs created from wikipedia. a key distinction between zsre and counterfact datasets is that zsre contains true facts, which are easier for the model to learn, whereas counterfact contains counterfactual examples where the new target has lower probability when compared to the original answer @xcite . examples from these datasets can be seen in to check for the applicability of the zsre dataset for editing base models, we first edit a fact in the qa format (as shown in the counterfact dataset contains 21,919 counterfactual statements. each datapoint in the dataset is a triplet of the type (subject, relation, object). the dataset is created such that the target object is less likely than the original object in the relation triplet. counterfact is thus also a more difficult dataset @xcite as we are going against the knowledge of the model, which is exactly what we do when we edit or inject facts in a model. ## scaling rome in this section, we evaluate the performance of rank-one memory editing (rome) method @xcite when multiple sequential edits are made to the same model. we compare the performance of rome with mend and fine-tuning (ft-c) baseline. we use the standard implementation of rome and mend. for the fine-tuning baselines, we fine tune the same layer being edited by rome and constrain the norm of change in weights. our experiments show that if norm is not constrained during fine-tuning, its leads to immediate model degradation. we provide more elaborate implementation details in appendix a.2. edits to the models are made using the coun-terfact dataset. to perform these experiments, we create four random subsets of 1000 examples from the counterfact dataset and sequentially edit the model on the selected datapoints. we do so to find patterns that are independent of the effect of the order in which facts are edited. there is no knowledge conflict @xcite in any of these samples as the same subject is not edited twice. ## scaling memit in this section, we evaluate the performance of the mass-editing memory in a transformer (memit) method @xcite when multiple sequential edits are made to the same model. we will follow the same procedure as followed in section 3, first evaluating the editing proficiency, fact forgetting and loss of performance on downstream tasks. we perform sequential editing on gpt-j (6b) using a random subset of 2000 examples from the coun-terfact dataset when using memit. this subset is a continuation of sample 1 in section 3. figure 3a shows the editing proficiency of memit as a function of the number of edits made to the model. note that here we edit one fact at a time for memit. while memit is able to make batched edits, we leave that analysis for future work. the dotted lines show a window size of 5 previous edits, whereas solid lines show a window size of 50 previous edits, same as in figure 1c . we see that the efficacy score for memit is not as high as rome. this means that knowledge edits made via memit are not always successful, while in rome we're always able to edit facts successfully. we also see a continuous decline of neighborhood score for memit as seen for rome, showing that editing facts also start affecting other facts stored in the model. figure 3b shows the percentage of successfully edited facts that get forgotten as new facts are edited using memit. we again begin to see two phases of forgetting -gradual and catastrophic. the catastrophic forgetting phase begins after approximately 1400 edits made to the model. when compared to rome, we find that edits made using memit have a much longer gradual forgetting phase across multiple samples. additionally, we also find that memit forgets fewer previously edited facts when compared to rome, as seen in figure 4 , where memit forgets almost three times fewer facts when compared to rome. this can also be seen for other samples in appendix a.4.2. finally, the effect of the number of edits on the downstream performance of the model can be seen in figure 3c . we see that the model maintains its ability to downstream tasks as more number of edits are made to the model. while this is true for gpt-j, we observe a gradual loss of performance with multiple edits for gpt2-xl (appendix a.3). in fact, we find that for gpt2-xl, the model loses the ability to do paraphrase detection long before the point of catastrophic forgetting, thus showcasing that the model can lose its ability of performing certain downstream tasks even before a disabling edit cripples the model. figure 3d shows the distance of the edited layers from the respective original layers. we find that the distance from the respective original layer increases gradually until approximately 1400 edits. after this, we find spikes in the distance between the edited layers and the original layers, which coincides with the points of catastrophic forgetting as seen in previous plots. we also evaluate if disabling edits are fundamental to memit. we find that, if the fact that disables the model after a sequence of edits is edited first in the model, it does not lead to catastrophic forgetting. thus, memit is more robust to a single destabilizing edits when compared to rome. we conjecture this is because a fact is stored within multiple layers of a model @xcite , and editing the weights of a single layer to edit facts can lead to larger instabilities in the model. we summarize the properties of rome and memit at scale in ## related work in this paper, we focus on model editing methods that modify the base language model's parameters. some of these methods @xcite require training a hypernetwork (chauhan et al., 2023) that generates new weights for the model being edited. other methods @xcite directly update specific parts of the model after locating stored facts inside it. @xcite unify these methods under the same framework called the preservation-memorization framework and enable batched editing with rome, an algorithm they call emmet. other memory based model-editing methods @xcite are not evaluated in this paper. while many of these methods have shown promise @xcite , recent work analyzing the after-effects of these editing methods have highlighted the shortcomings of these methods. specifically, while some of these editing methods rank high on reliability, generalization and locality metrics @xcite @xcite , the edited knowledge is not used consistently by the model. @xcite propose a new evaluation system where the "ripple effects" or implications of an edited fact are evaluated. an example of such ripple effects would be -if an edited fact updates the president of a country to the new president, then prompting for the birthplace of the president should output the birthplace of the new president. li et al. (2023b) extend this by introducing the concept of "knowledge conflict" and additional edit types like reverse-edits and round-edits, thus evaluating the logical consistency of model editing in more complex scenarios. ## conclusion in this paper we analyze popular model editing techniques at scale. we use these methods to make multiple sequential edits to the same model and find that they fail in multiple ways. we find that rome and memit perform the best when scaled to multiple sequential edits as measured using metrics like fact forgetting and downstream performance. as we edit the models, we discover they undergo forgetting of previous knowledge and skills in two phases. initially, the model gradually forgets previously edited facts and loses the ability to do downstream tasks, a phase which we call gradual forgetting. after that, the model abruptly loses all coherence and function including the ability to recall previously edited facts, perform downstream tasks and the ability to be edited, which is a realization of catastrophic forgetting. we also find that the source of these two phases of forgetting is that the layers being edited with these methods slowly drift away from their original weight values, thus becoming incompatible with the rest of the model. practical use of model editing requires us to be able to make multiple sequential edits to a model. we find that these model editing methods, like other fine tuning techniques, are prone to catastrophic forgetting. to be able to scale such methods, we not only need to have high efficacy, specificity and generalization, but we also need these methods to preserve the model's existing abilities. with this paper, we call for an improved evaluation of model editing techniques at scale, including evaluating model performance on downstream tasks and ability to recall previously edited facts. finally, we want to stress upon the implications of the two phases of forgetting discovered in this paper for rome and memit. gradual forgetting makes model editing techniques increasingly less effective as we sequentially edit facts, and hence limits their usefulness at scale. while catastrophic forgetting, which renders the model practically useless, caps the extent to which we can scale these methods. thus we need to create model editing techniques that can counteract both gradual forgetting and catastrophic forgetting when scaled.