Generate-on-Graph: Treat LLM as both Agent and KG for Incomplete Knowledge Graph Question Answering (2024)

Abstract: To address the issues of insufficient knowledge and hallucination in Large Language Models (LLMs), numerous studies have explored integrating LLMs with Knowledge Graphs (KGs). However, these methods are typically evaluated on conventional Knowledge Graph Question Answering (KGQA) with complete KGs, where all factual triples required for each question are entirely covered by the given KG. In such cases, LLMs primarily act as an agent to find answer entities within the KG, rather than effectively integrating the internal knowledge of LLMs and external knowledge sources such as KGs. In fact, KGs are often incomplete to cover all the knowledge required to answer questions. To simulate these real-world scenarios and evaluate the ability of LLMs to integrate internal and external knowledge, we propose leveraging LLMs for QA under Incomplete Knowledge Graph (IKGQA), where the provided KG lacks some of the factual triples for each question, and construct corresponding datasets. To handle IKGQA, we propose a training-free method called Generate-on-Graph (GoG), which can generate new factual triples while exploring KGs. Specifically, GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA. Experimental results on two datasets demonstrate that our GoG outperforms all previous methods.

## introduction large language models (llms) @xcite have made great success in various natural language processing (nlp) tasks. benefiting from extensive model parameters and vast amounts of pre-training corpus, llms can solve complex reasoning tasks through prompting engineer and in-context learning @xcite , without fine-tuning for specific tasks. ## preliminary in this section, we first introduce knowledge graphs (kgs). then, we use symbols of kgs to describe relation path and knowledge graph question answering (kgqa). knowledge graphs (kg) can be described as a set of inter-linked factual triples, i.e., @xmath0, r, t) ∈ v × r × v}, where h, @xmath1, @xmath2. knowledge graph question answering (kgqa) is a reasoning task that aims to predict answer entities e @xmath3. following previous work @xcite , we call the entities mentioned in question q as topic entities, denoted as e @xmath4. many datasets (talmor and that is, it doesn't guarantee that all triples in gold path are contained by g. for example, the triple (cupertino, timezone, pacific standard time) in w g may not be contained by g. therefore, models need to recall them from llms or reasoning from subgraph information. ## generate-on-graph (gog) in this section, we introduce our method generateon-graph (gog), which can integrate the knowledge of kgs and llms, as well as utilize the reasoning ability of llms. the workflow of gog is illustrated in figure 3 (c ). gog utilizes the thinking-searching-generating framework, which consists of three main steps: thinking, searching and generating. ## conclusion in this paper, we propose leveraging llms for qa under incomplete kgs (ikgqa), and construct relevant datasets. we propose generate-on-graph (gog), which can effectively integrate the external and internal knowledge of llms. experiments on two datasets show the superiority of gog, and demonstrate that an llms can be combined with incomplete kgs to answer complex questions.