Knowledge Graph Enhanced Large Language Model Editing (2024)

Abstract: Large language models (LLMs) are pivotal in advancing natural language processing (NLP) tasks, yet their efficacy is hampered by inaccuracies and outdated knowledge. Model editing emerges as a promising solution to address these challenges. However, existing editing methods struggle to track and incorporate changes in knowledge associated with edits, which limits the generalization ability of post-edit LLMs in processing edited knowledge. To tackle these problems, we propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME. Specifically, we first utilize a knowledge graph augmentation module to uncover associated knowledge that has changed due to editing, obtaining its internal representations within LLMs. This approach allows knowledge alterations within LLMs to be reflected through an external graph structure. Subsequently, we design a graph-based knowledge edit module to integrate structured knowledge into the model editing. This ensures that the updated parameters reflect not only the modifications of the edited knowledge but also the changes in other associated knowledge resulting from the editing process. Comprehensive experiments conducted on GPT-J and GPT-2 XL demonstrate that GLAME significantly improves the generalization capabilities of post-edit LLMs in employing edited knowledge.

## introduction large language models (llms) have achieved impressive results in various natural language processing (nlp) tasks @xcite @xcite , attributed to their generalization capabilities and extensive world knowledge @xcite . however, the knowledge encoded in llms is often outdated or factually inaccurate, which constrains their utility in real-world ## related work in this section, we introduce related work on model editing, which aims to incorporate new knowledge into llms or modify their existing internal knowledge while minimizing the impact on unrelated knowledge. model editing methodologies can be broadly classified into two categories @xcite : parameter-preserving and parametermodifying methods. ## preliminaries in this section, we introduce the definition of model editing and knowledge graphs, and the rank-one model editing framework used in our study. definition 1 (model editing for llms). model editing @xcite aims to adjust an llm f's behavior to modify the knowledge (s, r, o) encoded in the model into the target knowledge (s, r, o * ), where knowledge is denoted as a triple, consisting of the subject s, relation r, and object o. each edit sample e can be represented as (s, r, o, o * ). the post-edit llm is defined as f ′ . definition 2 (knowledge graph). a knowledge graph (kg) @xcite stores structured knowledge as a collection of triples {(s, r, o) ⊆ e × r × e}, where e and r represent the set of entities and relations, respectively. ## methodology in this section, we introduce the proposed glame, the architecture of which is illustrated in figure 2 . the framework comprises two key components: (1) knowledge graph augmentation (kga), which associates the knowledge of internal changes in llms by utilizing external knowledge graphs, and ## conclusion in this paper, we have proposed a novel method glame for large language model editing. glame leverages a knowledge graph augmentation module to capture the changes in associated knowledge by constructing an external graph. following this, we have introduced a graph-based knowledge edit module that utilizes a relational graph neural network to seamlessly integrate new knowledge associations from the constructed subgraph into the llm's parameter editing framework. experimental results on two llms and extensive analysis have demonstrated the effectiveness and superiority of glame in model editing tasks.