W il KE : Wise-Layer Knowledge Editor for Lifelong Knowledge Editing (2024)

Abstract: Knowledge editing aims to rectify inaccuracies in large language models (LLMs) without costly retraining for outdated or erroneous knowledge. However, current knowledge editing methods primarily focus on single editing, failing to meet the requirements for lifelong editing. This study reveals a performance degradation encountered by knowledge editing in lifelong editing, characterized by toxicity buildup and toxicity flash, with the primary cause identified as pattern unmatch. We introduce a knowledge editing approach named Wise-Layer Knowledge Editor (WilKE), which selects editing layer based on the pattern matching degree of editing knowledge across different layers in language models. Experimental results demonstrate that, in lifelong editing, WilKE exhibits an average improvement of 46.2% and 67.8% on editing GPT2-XL and GPT-J relative to state-of-the-art knowledge editing methods.

## introduction large language models (llms) encode a wealth of world knowledge through pretraining on massive corpus @xcite @xcite @xcite . however, outdated or erroneous knowledge may persist, and retraining these models with updated corpus incurs prohibitively high costs. to address this challenge, numerous studies have introduced knowledge editing @xcite @xcite as a solution, which involves updating the internal parameters of language models to edit specific knowledge. current knowledge editing methods are evaluated in single editing by default, which updates a single knowledge (x e , y o ) to (x e , y e ) on initial timeline ' 0 1 . . . ( , ) ( 1 , 1 ) ( 2 , 2 ) ( , ) . . . model f θ for each test point, as shown in figure 1(a). however, knowledge should be updated continuously in fact, making single editing insufficient to meet the demands. therefore, we focus on lifelong editing, which updates a knowledge (x e i , y o i ) to (x e i , y e i ) on model f e i-1 that doesn't have to be initial model f θ 0 , as shown in figure 1(b). in this paper, we conduct an analysis of state-ofthe-art knowledge editing methods such as rome @xcite and memit @xcite , revealing a severe performance degradation when applied in lifelong editing. investigating this issue further, our experiments indicate that these methods suffer from toxicity buildup and toxicity flash during ongoing editing. as shown in figure 3 @xcite , 4(a), the combined effects of both phenomena result in a "step-like" shape. on the one hand, the toxicity buildup signifies that one edit induces minor changes in irrelevant parameters, gradually leading to model's failure. on the other hand, the toxicity flash suggests that one edit modifies model's parameters abnormally, resulting in severe overfitting to specific edit, which is not reported in previous research. it's worth noting that due to overfitting, such failures are undetectable in single editing, and achieve respectable scores. we analyze the primary reasons for these two phenomena, attributing them to pattern unmatch, as illustrated in figure 2 . specifically, different layer i-2 (wise-layer) layer i layer i+1 (predefined) layer i-1 . . . layer i+2 . . . layer i-3 +1 [⋯⋯⋯⋯⋯ +1 ⋯⋯⋯⋯⋯] mlp of layer i+1 [( +1 + ℎ )] [⋯⋯⋯⋯⋯ -2 ⋯⋯⋯⋯⋯] mlp of layer i-2 [( -2 + ℎ -3 )] layers of language model may detect different patterns, which is called key in key-value memories @xcite @xcite , thus extracting relevant information according to patterns and updating the hidden states. in other words, different knowledge may be stored in different layers, as illustrated in section 4.3. however, rome and memit perform knowledge editing at predefined layers, which primarily lead to toxicity buildup and toxicity flash. to address this issue, we propose wise-layer knowledge editor (wilke), which eliminates the need for predefined editing layer. instead, wilke selects editing layer based on the degree of pattern matching for different editing knowledge across various layers. experimental results demonstrate that wilke exhibits state-of-the-art comprehensive performance when editing gpt2-xl (1.5b) @xcite and gpt-j (6b) @xcite . specifically, in lifelong editing scenarios, under identical experimental conditions of conducting 1024 edits, wilke demonstrates an average improvement of 46.2% and 67.8% in comprehensive performance relative to state-of-the-art methods when editing gpt2-xl and gpt-j, respectively. in summary, our primary contributions are as follows: no need for predefined editing layer, wilke selects editing layer based on the degree of pattern matching for different editing knowledge, significantly ameliorating this problem. ## related work generally, knowledge editing aims to edit the knowledge of a language model so that its outputs reflect the revised state when presented with relevant inputs @xcite . @xcite categorized knowledge editing methods into two major classes: preserving model's parameters and modifying model's parameters. methods for preserving model's parameters include memory-based methods and additional parameters' methods. memory-based methods utilize external storage to store editing facts, for example, serac @xcite employs an additional network to store editing knowledge, whereas @xcite utilizes a codebook to store editing knowledge. additional pa-rameters' methods employ extra neurons to store editing facts, for instance, @xcite and @xcite adding extra neurons in mlp to memorize additional facts. since our target is to edit knowledge by updating the internal parameters of language models, this paper focuses on methods that modify model's parameters. currently, methods for modifying model's parameters can be further divided into two categories: meta-learning and locate-and-edit. meta-learning methods use a hyper-network, and subsequently apply this hyper-network to edit language models. for instance, de cao et al. (2021) employed a bidirectional lstm to predict weight updates for editing, @xcite utilized low-rank decomposition of gradients to learn finetuning for language models, and @xcite extended single editing to batch editing using a least-squares approach built upon mend @xcite . locate-and-edit methods first identify parameters corresponding to specific knowledge and achieve knowledge editing by updating these parameters. for example, @xcite used knowledge attribution to determine the location of neurons, followed by parameter updates on these neurons for knowledge editing. @xcite employed causal mediation analysis to identify the center of causal effects and performed updates on that position. @xcite extended upon rome @xcite by distributing residuals across multiple layers and achieved batch editing, and li et al. (2023b) achieved more precise residual allocation. however, existing knowledge editing methods that modify model's parameters mostly focus on single editing, unable to meet the demands of lifelong editing, leading to a certain gap between knowledge editing and practical applications. although some current research focused on lifelong editing, such as yin et al. (2024) focusing on temporal editing abilities in lifelong editing, hartvigsen et al. (2022) and huang et al. (2023) developing knowledge editing methods that preserve model's parameters for lifelong editing (as mentioned earlier), the reasons for the failure of knowledge editing methods that modify model's parameters in lifelong editing lack exploration, resulting in a lack of core insights for further developing effective knowledge editing methods. consequently, research into lifelong editing is imperative. ## preliminary the language model f θ ∈ f can be defined as a function f θ : @xmath0, mapping input x x @xmath1. for an editing example (x x x e , y y y e ), where f θ (x x x e ) ̸ = y y y e , the goal of the knowledge editing (ke) is to edit the parameters θ ∈ θ of the model f θ to obtain an edited model f θ ′ , such that f θ ′ (x x x e ) = y y y e . in lifelong editing, such a process continues iteratively. in other words, for an initial language model f θ 0 , there exists a potential sequence to be edited (x x x e i , y y y e i ) n @xmath2, and the model undergoes continuous editing: in lifelong editing, the edited model should satisfy the following properties. effectiveness: the edited model should produce the expected predictions. generality: the edited model should remain consistent on its edited data equivalent input set e(x x x e i ). locality: the edited model should maintain the original output on data unrelated to the editing, denoted as i(x x x e i ). retention: the edited model should preserve the editing results based on the previously completed edits. here is y y y ′ e j rather than y y y e j because we consistently adhere to a principle: the later the edit, the higher the priority. later edits take precedence over earlier ones and potentially engage in complex interactions with the original knowledge to update it. for further explanations and details, please refer to appendix f. ## toxicity in lifelong editing rome @xcite and memit @xcite are currently the state-of-the-art knowledge editing methods. as memit is based toxicity buildup toxicity flash (a) l2 norm over steps on predefined layer. (b) visualization of toxicity at specific steps. darker color, larger changes. on rome, implementing residual distribution across multiple layers, our analysis in the main text focuses primarily on rome. the analysis of memit is provided in appendix c. in this section, we systematically investigate the reasons for the failure of rome in lifelong editing. ## wise-layer knowledge editor in section 4, we delved into the primary reason for failure in lifelong editing -pattern unmatch, which directly leads to toxicity flash and potentially more toxicity buildup. in light of this, we propose an editing method called wilke. unlike rome and memit, wilke does not predefine editing layer; instead, it selects editing layer based on the degree of pattern match for different editing knowledge across various layers. we first describe where to edit in section 5.1, followed by an explanation of how to edit in section 5.2. ## conclusion in this work, we focus on lifelong knowledge editing, finding that current knowledge editing methods suffer from severe performance degradation in lifelong editing. our experimental results reveal the toxicity buildup and toxicity flash that may occur during lifelong editing, leading to the deterioration of model's performance. through further investigation, we find that the direct cause of these problems lies in the predefined editing layer, while the underlying cause stems from pattern unmatch. to address this issue, we propose a model edit-ing method called wise-layer knowledge editor (wilke), which does not require predefined editing layer but selects editing layer based on the degree of pattern matching between different layers of the language model for specific editing knowledge. experimental results demonstrate that in lifelong editing, wilke significantly enhances overall performance compared to currently popular knowledge editing methods, achieving state-of-the-art knowledge editing performance. in summary, our work is significant for improving knowledge editing methods and provide valuable insights for future work.