Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models (2024)

Abstract: Temporal knowledge graph question answering (TKGQA) poses a significant challenge task, due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. Although large language models (LLMs) have made considerable progress in their reasoning ability over structured data, their application to the TKGQA task is a relatively unexplored area. This paper first proposes a novel generative temporal knowledge graph question answering framework, GenTKGQA, which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. First, we exploit LLM’s intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. Next, we design virtual knowledge indicators to fuse the graph neural network signals of the subgraph and the text representations of the LLM in a non-shallow way, which helps the open-source LLM deeply understand the temporal order and structural dependencies among the retrieved facts through instruction tuning. Experimental results on two widely used datasets demonstrate the superiority of our model.

## introduction real-world knowledge is frequently updated rather than static @xcite , e.g., (obama, hold_position, president) is merely valid only for a certain period @xcite . hence, the temporal knowledge graph (tkg) is proposed as a database for storing dynamic structured facts associated with timestamps, denoted as @xcite . temporal knowledge graph question answering (tkgqa) aims to answer a natural question with explicit or implicit temporal constraints based on the tkg, e.g., "who held the position of president (in 2017) or (after obama)?". due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge, tkgqa is one of the most challenging qa tasks. recently, large language models (llms) have shown strong competitiveness in various fields @xcite . some researchers explore the reasoning capability of llms for structured knowledge based on kgqa tasks @xcite , and some works examine the temporal reasoning capabilities of llms through time-sensitive qa tasks @xcite . intuitively, llms have the ability to deal with temporal structured knowledge. based on the above findings, we attempt to utilize llms for the tkgqa task and summarize the following two challenges: 1) question-relevant subgraph retrieval. a common practice to enhance the llm's domain-specific reasoning capability is to input query-relevant information as additional knowledge into the llms @xcite . as shown in figure 1 , chatgpt 1 cannot answer temporal questions directly, but it can answer simple questions when the relevant facts are provided. however, finding facts relevant to the problem is a struggle due to the large search space with both structural and temporal dimensions. for example, for the complex temporal question "who held the position of president after obama?", the structural link hold position between the entities and the temporal constraints after 2016 in the problem are unknown, so directly finding all relevant facts about the given entities obama and president is bound to introduce too much noise information. how to accurately retrieve relevant facts from a two-dimensional space is the first challenge. 2) complex-type question reasoning. recent works about llm-based kg reasoning mostly input structured knowledge in the natural text form into the task prompt and reason about the answers in a training-free manner @xcite . however, these approaches fuse subgraph information with the llm in a shallow way, which limits the inference performance on the complex question type. as illustrated in figure 1 , chatgpt cannot understand the chronological order of the relevant facts and answers incorrectly on the complex question type "final". how subgraph information can be integrated into llm representations in a nonsuperficial way to simulate structured reasoning remains an open question. hence, we propose gentkgqa, a novel generative temporal knowledge graph question answering framework consisting of two phases, subgraph retrieval and answer generation, which is used to address the above two challenges, respectively. at the first phase, we find that the structural and temporal scope of the subgraph is determined by the relation links and the temporal constraints in the question, respectively. therefore, we use a divideand-conquer strategy to reduce the subgraph search space by decomposing the complex subgraph retrieval problem into two subtasks, namely, relation ranking and time mining. then, we utilize the llm's internal knowledge to mine structural connections between entities and time constraints in the problem without extra training. we only need to input few-shot examples into the prompt to accomplish subgraph retrieval of the entire data. at the second phase, we fine-tune the open-source llm with instruction tuning to incorporate structural and temporal information of the subgraph in a non-shallow way. recent works illustrate that fusing graph neural network (gnn) representations and language text representations can enhance the ability of lms to perceive graph structure @xcite . thus, we design three novel virtual knowledge indicators to bridge the links between pre-trained gnn signals of the temporal subgraph and text representations of the llm, which guides the llms in deeply understanding the graph structure and improves their reasoning ability for complex temporal questions. overall, our contribution can be summarized in the following four points: ## preliminaries tkgqa. a temporal knowledge graph (tkg) g := (e, r, t , f) is a multi-relational, directed graph with timestamped edges between entities, where e, r and t represent the sets of entities, relations and timestamps, respectively. each fact in the g can be represented as a quadruple (s, r, o, t) ∈ f, corresponding to entity s/@xmath0, relation type @xmath1. given a natural language question q, tkgqa aims to extract entities s/o or timestamps t that correctly answer the question q. ## method we apply the llms processing tkgqa task in a two-phase process, i.e., an icl-based subgraph retrieval phase and an it-based answer generation phase. at the first stage, we utilize internal knowledge of llm for unlabeled subgraph retrieval. at the second stage, we incorporate external knowledge for structure-aware temporal inference. ## conclusion we propose a novel generative framework, gen-tkgqa, which guides the llm in a two-stage manner to handle temporal question answering on tkg. specifically, at the subgraph retrieval phase, we exploit the llm's intrinsic knowledge to mine the temporal constraints and structural links in the temporal questions, which reduces the search space of the subgraphs in both temporal and structural dimensions. we employ the in-context learning approach to complete subgraph retrieval for the entire dataset with a small number of samples. in order to improve the inference performance of the llm on complex question types, at the answer generation phase, we present the instruction tuning technique to make the open-source llm truly understand the temporal order and structural dependencies among retrieved facts. most significantly, we design novel virtual knowledge indicators to establish a bridge between subgraph neural information and text representations. experimental results show that our framework can effectively utilize the llm to solve the complex question type of tkgqa task and validate the adequacy of our motivation.