KG - GPT : A General Framework for Reasoning on Knowledge Graphs Using Large Language Models (2023)

Abstract: While large language models (LLMs) have made considerable advancements in understanding and generating unstructured text, their application in structured data remains underexplored. Particularly, using LLMs for complex reasoning tasks on knowledge graphs (KGs) remains largely untouched. To address this, we propose KG-GPT, a multi-purpose framework leveraging LLMs for tasks employing KGs. KG-GPT comprises three steps: Sentence Segmentation, Graph Retrieval, and Inference, each aimed at partitioning sentences, retrieving relevant graph components, and deriving logical conclusions, respectively. We evaluate KG-GPT using KG-based fact verification and KGQA benchmarks, with the model showing competitive and robust performance, even outperforming several fully-supervised models. Our work, therefore, marks a significant step in unifying structured and unstructured data processing within the realm of LLMs.

## introduction the remarkable advancements in large language models (llms) have notably caught the eye of scholars conducting research in the field of natural language processing (nlp) @xcite @xcite . in their endeavor to create llms that can mirror the reasoning capabilities inherent to humans, past studies have primarily centered their attention on unstructured textual data. this includes, but is not limited to, mathematical word problems @xcite @xcite , csqa @xcite , and symbolic manipulation @xcite . while significant strides have been made in this area, the domain of structured data remains largely unexplored. structured data, particularly in the form of knowledge graphs (kgs), serves as a reservoir of interconnected factual information and associations, articulated through nodes and edges. the inherent structure of kgs offers a valuable resource that can assist in executing complex reasoning tasks, like multi-hop inference. even with these advantages, to the best of our knowledge, there is no general framework for performing kg-based tasks (e.g. question answering, fact verification) using auto-regressive llms. to this end, we propose a new general framework, called kg-gpt, that uses llms' reasoning capabilities to perform kg-based tasks. kg-gpt is similar to structgpt @xcite in that both reason on structured data using llms. however, unlike structgpt which identifies paths from a seed entity to the final answer entity within kgs, kg-gpt retrieves the entire sub-graph and then infers the answer. this means kg-gpt can be used not only for kgqa but also for tasks like kg-based fact verification. kg-gpt consists of three steps: 1) sentence (claim / question) segmentation, 2) graph retrieval, and 3) inference. during sentence segmentation, a sentence is partitioned into discrete sub-sentences, each aligned with a single triple (i.e. @xcite ). the subsequent step, namely graph retrieval, retrieves a potential pool of relations that could bridge the entities identified within the sub-sentences. then, a candidate pool of evidence graphs (i.e. sub-kg) is obtained using the retrieved relations and the entity set. in the final step, the obtained graphs are used to derive a logical conclusion, such as validating a given claim or answering a given question. to evaluate kg-gpt, we employ kg-based fact verification and kgqa benchmarks, both demanding complex reasoning that utilizes structured knowledge of kgs. in kg-based fact verification, we use factkg @xcite , which includes figure 1 : an overview of kg-gpt. the framework comprises three distinct phases: sentence segmentation, graph retrieval, and inference. the given example comes from factkg. it involves a 2-hop inference from 'william_anders' to 'frank_borman', requiring verification through an evidence graph consisting of three triples. both 'william_anders' and 'frank_borman' serve as internal nodes in dbpedia @xcite , while "afit, m.s. 1962" acts as a leaf node. moreover, artificial satellite represents the type information absent from the provided entity set. we assume a graph g (knowledge graph consisting of entities e and relations r), a sentence s (claim or question), and all entities involved in s, e s ⊂ e are given. in order to derive a logical conclusion, we need an accurate evidence graph g e ⊂ g, which we obtain in two stages, sentence segmentation and graph retrieval. furthermore, all the aforementioned steps are executed employ-ing the in-context learning methodology to maximize the llm's reasoning ability. the prompts used for each stage are in appendix a. ## experiments we evaluate our framework on two tasks that require kg grounding: fact-verification and questionanswering. a detailed description of experimental settings can be found in appendix c. ## conclusion we suggest kg-gpt, a versatile framework that utilizes llms for tasks that use kgs. kg-gpt is divided into three stages: sentence segmentation, graph retrieval, and inference, each designed for breaking down sentences, sourcing related graph elements, and reaching reasoned outcomes, respectively. we assess kg-gpt's efficacy using kgbased fact verification and kgqa metrics, and the model demonstrates consistent, impressive results, even surpassing a number of fully-supervised models. consequently, our research signifies a substantial advancement in combining structured and unstructured data management in the llms' context.