KELE : Residual Knowledge Erasure for Enhanced Multi-hop Reasoning in Knowledge Editing (2025)

Abstract: Large language models (LLMs) face challenges with internal knowledge inaccuracies and outdated information. Knowledge editing has emerged as a pivotal approach to mitigate these issues. Although current knowledge editing techniques exhibit promising performance in single-hop reasoning tasks, they show limitations when applied to multi-hop reasoning. Drawing on cognitive neuroscience and the operational mechanisms of LLMs, we hypothesize that the residual single-hop knowledge after editing causes edited models to revert to their original answers when processing multihop questions, thereby undermining their performance in multi-hop reasoning tasks. To validate this hypothesis, we conduct a series of experiments that empirically confirm our assumptions. Building on the validated hypothesis, we propose a novel knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE). Specifically, we design an erasure function for residual knowledge and an injection function for new knowledge. Through joint optimization, we derive the optimal recall vector, which is subsequently utilized within a rank-one editing framework to update the parameters of targeted model layers. Extensive experiments on GPT-J (6B) and LLaMA-2 (7B) demonstrate that KELE substantially enhances the multi-hop reasoning capability of edited LLMs.

## introduction large language models (llms) have achieved significant success in a wide range of natural language processing (nlp) tasks @xcite . t he pr es id en t of t h e p r e s id e n t o f edit the wife of the wife of however, the knowledge embedded within llms can sometimes be factually incorrect or outdated, limiting their overall effectiveness. to address these limitations, knowledge editing methods have been proposed, offering a more efficient and precise approach to updating the knowledge in llms. these methods have attracted considerable attention from researchers in recent years. among these methods, those that modify the model's parameters are particularly important, as they provide a direct and flexible means of altering the model's behavior, such as ke @xcite , @xcite , @xcite , and memit @xcite . this work focuses specifically on parameter-modifying approaches. although these editing methods have demonstrated promising results in single-hop reasoning evaluations, they still face significant challenges in multi-hop reasoning @xcite . as illustrated in figure 1 , after editing the single-hop knowledge from "the president of the usa is obama" to "the president of the usa is biden," the edited model can easily answer the single-hop question, "who is the president of the usa?" however, it struggles with multi-hop questions, such as "who is the wife of the president of the usa?" to better understand this challenge in knowledge editing for llms, we first analyze this problem from a cognitive neurological perspective. when the brain receives new information, it can activate neurons associated with related old memories, a phenomenon known as memory association @xcite @xcite . this occurs because of the connectivity within neural networks, where the pathways of old memories are easily reactivated by relevant stimuli, thereby facilitating more efficient encoding and processing of new information. llms exhibit a similar mechanism, where related knowledge stored in their parameters is activated and integrated during reasoning @xcite . building on these insights, we hypothesize the following reason for the poor performance of edited llms on multi-hop reasoning tasks: llms retain a portion of single-hop old knowledge even after editing. when handling multi-hop questions related to the edited knowledge, the residual knowledge tends to prompt the models to produce original answers to these questions, thereby weakening their multi-hop reasoning ability. for example, if the single-hop knowledge in the llm is edited from "the president of the usa is obama" to "the president of the usa is biden," a portion of old knowledge "the president of the usa is obama" may still be retained and reactivated within the model. as shown in figure 2 , when asked the multi-hop question "who is the wife of the president of the usa?", the residual single-hop knowledge might cause the model to generate the original answer to the multi-hop question, michelle (obama's wife), instead of the correct answer, jill (biden's wife). to verify this hypothesis, we investigate the relationship between the residual old knowledge in llms and their responses to multi-hop questions (section 4). we define the retain score as a metric to quantify the residual old knowledge (s, r, o) for each edit sample (s, r, o, o * ), utilizing the output logit score of o under the prompt p(s, r). as illustrated in figure 3b , the higher the residual old knowledge in the edited llm, the more likely it is to provide the original answers to multi-hop questions, resulting in a lower proportion of correct answers. therefore, erasing the residual old knowledge offers a promising insight for improving the performance of edited llms on multi-hop reasoning tasks. based on the this hypothesis, we propose a simple yet effective method for large language model editing, termed knowledge erasure for large language model editing (kele) (section 5). specifically, within the rank-one editing framework, we develop an old knowledge erasure function and a new knowledge injection function to jointly optimize and obtain the recall vector. this approach eliminates the interference of old knowledge while injecting new knowledge. finally, the model parameters are updated in a single step using the recall vector and the subject representation through the rank-one update formula. we summarize our contributions as follows: ## related work in this section, we review related research on knowledge editing and its challenges in multi-hop reasoning. parameter-preserving methods. these methods typically store edited examples in an external knowledge base and guide the llms' output for specific queries by retrieving relevant knowledge. for instance, serac @xcite employs a gating network along with an auxiliary model specifically designed to handle edited knowledge. t-patcher @xcite introduces extra trainable parameters in the last layer of the ffn to correct llms. however, these methods face a critical scalability issue: the complexity of managing the external model increases with each new edit, which may limit their practical usability. parameter-modifying methods. these methods, including meta-learning @xcite @xcite , and fine-tuning-based approaches @xcite , edit llms by directly modifying their parameters. meta-learning methods generate updated weights for llms by training a hyper-network. for example, @xcite uses a bi-directional lstm to predict model weight updates, but it faced challenges with larger models due to their vast parameter spaces. to address this issue, mend @xcite employs a low-rank decomposition of fine-tuning gradients, providing an efficient mechanism for updating llm weights. locate-and-edit methods focus on identifying specific parameters associated with particular knowledge within llms, aiming for more interpretable and precise knowledge editing. early efforts, such as kn @xcite , introduce a knowledge attribution method to identify knowledge neurons but struggles to precisely modify the model's weights. rome @xcite ) method employs causal tracing to identify knowledge-relevant layers and then edits the corresponding ffn module. @xcite further enhances this approach by improving the objective function and enabling multi-layer edits for batch editing. recently, significant advancements in efficient parameter-tuning methods @xcite for supervised finetuning of llms have led to the development of fine-tuning-based editing methods @xcite , which utilizes lora @xcite and data augmentation strategies to directly fine-tune the llms, achieving the desired editing performance. multi-hop reasoning in knowledge editing. in recent years, several studies have aimed to enhance the performance of edited llms in multi-hop reasoning tasks. @xcite introduce the mquake dataset, specifically designed to evaluate the multi-hop reasoning capabilities of edited llms. they also propose a method that stores all edited facts externally, iteratively prompting llms to generate answers consistent with these edited facts. building on this approach, pokemqa @xcite introduces auxiliary knowledge prompt to assist in question decomposition. glame @xcite leverages external knowledge graphs to capture the impact of target knowledge changes on high-order knowledge within llms. these methods improve multi-hop reasoning by retrieving or incorporating external knowledge, which is not the focus of the current paper. additionally, @xcite find that llms often rely on factual shortcuts from pre-training corpora during reasoning, which contributes to the poor performance of edited models in multi-hop reasoning tasks. unlike this study, we identify another potential cause for the poor performance of parametermodified models in multi-hop reasoning tasks: the retention of old knowledge triggers the generation of original answers in multi-hop questions, thereby weakening the performance of edited models in these tasks. we validate this hypothesis through a series of experiments and propose a knowledgeerasure-based editing strategy to mitigate this issue. ## preliminaries in this section, we introduce the definition of knowledge editing and outline the corresponding tasks under single-hop and multi-hop evaluations. definition 1. knowledge editing for llms knowledge editing @xcite refers to the process of altering the behavior of an llm f's to change encoded knowledge from (s, r, o) to the new knowledge (s, r, o * ). here, knowledge is represented as a triple, with s as the subject, r as the relation, and o as the object. each editing instance e is denoted as (s, r, o, o * ), and the llm after editing is referred to as f ′ . definition 2. single-hop evaluation in knowledge editing single-hop evaluation assesses whether an edit (s, r, o, o * ) is successful in an edited llm f ′ . this evaluation constructs prompts p(s, r) based on the subject s and relation r, and measures the performance of f ′ using efficacy, paraphrase and specificity metrics @xcite . definition 3. multi-hop evaluation in knowledge editing multi-hop evaluation examines whether the edited llms can effectively utilize the updated knowledge for reasoning in multi-hop tasks. given a chain of facts (s 1 , r 1 , o 1 ), ..., (s n , r n , o n ), where the object of the i-th fact also serves as the subject of the next fact in the chain, i.e., o @xmath0, a multi-hop question p(s 1 , r 1 , .., r n ) can be constructed, with the answer being o n . for example, with a chain consisting of two facts, (usa, president of, obama) and (obama, wife of, michelle), one can write a 2-hop question: who is the wife of the president of usa?. once one or more facts in the chain are edited, e.g., (usa, president of, obama) is edited to (usa, president of, biden), the edited llm must utilize the new knowledge to answer the multi-hop question. the model's response should change from the original answer michelle to the correct answer jill. ## analysis of the impact of old knowledge on multi-hop reasoning in this section, we validate our hypothesis by examining the impact of old knowledge on the performance of edited llms in multi-hop reasoning. we select the representative multi-hop reasoning evaluation dataset, mquake @xcite , to conduct experiments. each instance in mquake is represented as @xmath1, q, a, a * ). here, e denotes the set of single-hop edits @xmath2, r, o, o * ), q represents a multi-hop question evaluating editing performance, and a and a * correspond to the original and correct answers to q. further details on mquake are provided in section 6.1 and appendix b. ## methodology in this section, we introduce the proposed kele, with its architecture depicted in figure 4 . the kele framework integrates a knowledge erasure strategy within the rank-one model editing framework @xcite . specifically, kele targets a specific layer l and transforms knowledge editing into two key operations: old knowledge erasure and new knowledge injection, which together are used to compute the recall vector v * . subsequently, v * , along with the subject representation ## experiments in this section, we evaluate our kele by applying it to two datasets and assessing its performance on two auto-regressive llms. we aim to answer the following questions through experiments.