LLM -Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments (2024)

Abstract: The important challenge of keeping knowledge in Large Language Models (LLMs) up-to-date has led to the development of various methods for incorporating new facts. However, existing methods for such knowledge editing still face difficulties with multi-hop questions that require accurate fact identification and sequential logical reasoning, particularly among numerous fact updates. To tackle these challenges, this paper introduces Graph Memory-based Editing for Large Language Models (GMeLLo), a straightforward and effective method that merges the explicit knowledge representation of Knowledge Graphs (KGs) with the linguistic flexibility of LLMs. Beyond merely leveraging LLMs for question answering, GMeLLo employs these models to convert free-form language into structured queries and fact triples, facilitating seamless interaction with KGs for rapid updates and precise multi-hop reasoning. Our results show that GMeLLo significantly surpasses current state-of-the-art (SOTA) knowledge editing methods in the multi-hop question answering benchmark, MQuAKE, especially in scenarios with extensive knowledge edits.

## introduction an important challenge in deploying large language models (llms) is keeping their knowledge accurate and up-to-date, without incurring expensive retraining costs @xcite . several approaches have been proposed in prior works to address this challenge. some methods focus on the incremental injection of new facts into language models @xcite @xcite . alternatively, other methods involve the use of external memory to store new facts @xcite , which does not require updating llm model weights. boris johnson rishi sunak british prime minister who is married to the british prime minister? 2-hop qa information evolves over time head of government spouse akshata murty rishi sunak uk boris johnson carrie symonds  ✔ figure 1 : multi-hop question answering in dynamic domains @xcite . dynamic nature of information: changes over time may trigger subsequent modifications. for instance, a transition in the british prime minister, such as from boris johnson to rishi sunak, necessitates corresponding adjustments, like the change in the british prime minister's spouse. as depicted in figure 2 , our gmello method comprises the following key steps: llms, trained on extensive sentence corpora @xcite @xcite , are expected to encapsulate a wide range of commonly used sentence structures. as a result, they are invaluable tools for analyzing sentences and extracting entities and relations. once the correct relation chain and edited triples are obtained, using a formal query to interrogate the kg in a knowledge-based question answering (kbqa) @xcite manner ensures precision in the searching process. in cases where kbqa fails, we still have llms for question answering (qa) to ensure comprehensive coverage. gmello outperforms current sota methods on two datasets from the mquake benchmark, affirming its effectiveness in multi-hop question answering within an evolving environment. ## related work this work utilizes both kgs and llms to address the challenge of multi-hop question answering, with a particular focus on scenarios involving evolving factual knowledge. therefore, we review existing literature on multi-hop question answering, knowledge editing, and the augmentation of llms with knowledge graphs foot_0 . ## gmello: graph memory-based editing for large language models in this section, we introduce our method gmello for multi-hop question answering with knowledge editing (figure 2 ).