Knowledge Graph-Driven Memory Editing with Directional Interventions (2025)

Abstract: Large Language Models (LLMs) have revolutionized language processing and understanding, yet their performance is hampered by inaccuracies and outdated information. Model editing techniques offer a solution but face two key challenges: **(I)** Most methods inject knowledge by constructing rigid loss, which leads to poor compatibility when dealing with higher-order multi-hop problems. **(II)** Locate-then-edit vein, by altering pre-trained parameters, inevitably affect normal knowledge and even face the catastrophic forgetting. In this paper, we introduce **KGMET**, a framework that constructs knowledge graphs using available information to guide the direction of knowledge editing, enabling **consistent**, **aligned**, and **stable** information during **large-scale** editing scenario. Furthermore, *KGMET* goes beyond this by employing orthogonal constraints to block the interference of irrelevant information, ensuring the updates are both controllable and generalizable. Experiments on Multi-Conterfact, ZsRE, and MQuAKE datasets using *Llama-3-8B*, *GPT-J-6B*, and *GPT-2-XL* models showcase improvements over state-of-the-art methods, with \uparrow 5\%-17\% in multi-hop tasks while remaining generalizable (at least \uparrow 20\% in fluency). Our code is available on Github.

## introduction large language models (llms) are endowed with robust inferential and computational abilities, which have led to their extensive application in recent years @xcite . nevertheless, these models may retain incorrect or obsolete information, prompting the development of model editing techniques based on llms @xcite . diverging from retraining or fine-tuning strategies @xcite , llm-based model editing techniques advocate for the modification of a subset of parameters, akin to a surgical approach that precisely locates and edits knowledge @xcite . this method is markedly less resource-intensive than retraining @xcite , thereby attracting considerable interest from researchers @xcite @xcite . concretely, model editing techniques can be classified into weight-preserving @xcite @xcite and weight-modifying approaches @xcite . in this study, we investigate weight-modifying model editing techniques. existing weightmodifying methods follow a "locate-then-edit" workflow @xcite , which locates influential parameters w through causal mediation analysis @xcite and subsequently completes knowledge editing by inserting a perturbation ∆ @xcite . though efficacy @xcite , it encounters inaccuracies in multi-hop tasks associated with the edited knowledge, as the model is unable to accurately calibrate and edit the relevant knowledge changes @xcite . for example, when the u.s. president is edited from biden to trump, the model still incorrectly responds with biden's wife when queried "who is the wife of the u.s. president?" rather than trump's @xcite @xcite . to this end, glame @xcite endeavors to incorporate knowledge graph representation learning techniques into the editing process, yielding some positive results. however, this approach is fraught with two critical issues: to tackle these challenges, we present a knowledge graph-guided methodology for massbatch model editing, termed kgmet, which aims to minimize adverse effects on llms during multihop reasoning and batch-editing tasks. kgmet mainly consists of three key technologies: ❶ directional intervention strategy: we first propose a directional intervention approach, which enables knowledge graph enhancement methods to be adapted to mass batch editing scenarios; ❷ orthogonality constraints: the use of orthogonality constraints to enable the model to maintain its general performance; ❸ multiple layers of editing: the editing is extended to multiple layers, which makes it more capable of generalizing to a larger number of factual requests. we conduct comprehensive experiments on three widely adopted editing datasets. empirical results demonstrate that kgmet is ❶ higher performing, surpassing most locating-then-editing methods by average 27.3%; ❷ outstanding higher-order capabilities, outperforming sota baselines like glame in neighborhood success metrics above 5% ~10%, and above 3% ~17% compare to other methods in mquake @xcite multi-hop inference tasks; ❸ effective maintenance of general ability, in terms of fluency in generating text, kgmet outperforms the sota baselines by about 20% across multiple benchmarks. our contributions are summarized as follows: ## related work methods of edits can be broadly classified into parameter-preserving and parameter modifying methods @xcite . we further discuss graph based related work in appendix a.3 ## method in this paper, a knowledge graph-driven memory editing with directional interventions is proposed, it contains three key components: (i) we first utilize knowledge embedding to extract directional information, thus guiding the direction of δ optimization; (ii) the method of orthogonal constraints is proposed for isolating irrelevant knowledge; (iii) finally, edit residuals are introduced into multilayers so that the edits can be better adapted to large-scale batch editing scenarios.the framework of our work can be seen in figure 2 . ## experiment in this section, we conduct experiments to address the following research questions. ## conclusion and limitation conclusion. in this paper, a knowledge graph-driven memory editing with directional interventions method (kgmet) is proposed to solve the multi-hop reasoning problem in batch editing scenario. we propose knowledge graph based direc-tional intervention and orthogonal constraint methods to increase the ability of multi-hop reasoning while preserve generality of llms. extensive experiments is conducted to prove kgmet have superior performance than baselines.