Question Answering Over Temporal Knowledge Graphs (2021)

Abstract: Temporal Knowledge Graphs (Temporal KGs) extend regular Knowledge Graphs by providing temporal scopes (start and end times) on each edge in the KG. While Question Answering over KG (KGQA) has received some attention from the research community, QA over Temporal KGs (Temporal KGQA) is a relatively unexplored area. Lack of broad coverage datasets has been another factor limiting progress in this area. We address this challenge by presenting CRONQUESTIONS, the largest known Temporal KGQA dataset, clearly stratified into buckets of structural complexity. CRONQUESTIONS expands the only known previous dataset by a factor of 340x. We find that various state-of-the-art KGQA methods fall far short of the desired performance on this new dataset. In response, we also propose CRONKGQA, a transformer-based solution that exploits recent advances in Temporal KG embeddings, and achieves performance superior to all baselines, with an increase of 120% in accuracy over the next best performing method. Through extensive experiments, we give detailed insights into the workings of CRONKGQA, as well as situations where significant further improvements appear possible. In addition to the dataset, we have released our code as well.

## introduction temporal knowledge graphs (temporal kgs) are multi-relational graph where each edge is associated with a time duration. this is in contrast to a regular kg where no time annotation is present. for example, a regular kg may contain a fact such as (barack obama, held position, president of usa), while a temporal kg would contain the start and end time as well -(barack obama, held position, president of @xcite . edges may be associated with a set of non-contiguous time intervals as well. these temporal scopes on facts can be either automatically estimated @xcite or user contributed. several such temporal kgs have been proposed in the literature, where the focus is on kg completion @xcite @xcite @xcite . the task of knowledge graph question answering (kgqa) is to answer natural language questions using a kg as the knowledge base. this is in contrast to reading comprehension-based question answering, where typically the question is accompanied by a context (e.g., text passage) and the answer is either one of multiple choices @xcite or a piece of text from the context @xcite . in kgqa, the answer is usually an entity (node) in the kg, and the reasoning required to answer questions is either single-fact based @xcite , multi-hop @xcite or conjunction/comparison based reasoning @xcite . temporal kgqa takes this a step further where: temporal kg embeddings are another upcoming area where entities, relations and timestamps in a temporal kg are embedded in a low-dimensional vector space @xcite @xcite . here too, the main application so far has been temporal kg completion. in our work, we investigate whether temporal kg embeddings can be applied to the task of temporal kgqa, and how they fare compared to non-temporal embeddings or off-the-shelf methods without any kg embeddings. in this paper we propose cronquestions, a new dataset for temporal kgqa. cronques-tions consists of both a temporal kg and accompanying natural language questions. there were three main guiding principles while creating this dataset: 1. the associated kg must provide temporal annotations. 2. questions must involve an element of temporal reasoning. 3. the number of labeled instances must be large enough that it can be used for training models, rather than for evaluation alone. guided by the above principles, we present a dataset consisting of a temporal kg with 125k entities and 328k facts, along with a set of 410k natural language questions that require temporal reasoning. on this new dataset, we apply approaches based on deep language models (lm) alone, such as t5 @xcite , bert @xcite , and knowbert @xcite , and also hybrid lm+kg embedding approaches, such as entities-as-experts @xcite and em-bedkgqa @xcite . we find that these baselines are not suited to temporal reasoning. in response, we propose cronkgqa, an enhancement of embedkgqa, which outperforms baselines across all question types. cronkgqa achieves very high accuracy on simple temporal reasoning questions, but falls short when it comes to questions requiring more complex reasoning. thus, although we get promising early results, cronquestions leaves ample scope to improve complex temporal kgqa. our source code along with the cronquestions dataset can be found at https://github.com/apoorvumang/cronkgqa . ## cronquestions: the new temporal kgqa dataset cronquestions, our temporal kgqa dataset consists of two parts: a kg with temporal annotations, and a set of natural language questions requiring temporal reasoning. ## temporal kg embeddings we investigate how we can use kg embeddings, both temporal and non-temporal, along with pretrained language models to perform temporal kgqa. we will first briefly describe the specific kg embedding models we use, and then go on to show how we use them in our qa models. in all cases, the scores are turned into suitable losses with regard to positive and negative tuples in an incomplete kg, and these losses minimized to train the entity, time and relation representations. ## cronkgqa: our proposed method we start with a temporal kg, apply a time-agnostic or time-sensitive kg embedding algorithm (com-plex, tcomplex, or timeplex) to it, and obtain entity, relation, and timestamp embeddings for the temporal kg. we will use the following notation. this is used for scoring answers, since the answer can be either an entity or timestamp. in case entity/timestamp embeddings are complex valued vectors in c d , we expand them to real valued vectors of size 2d, where the first half is the real part and the second half is the complex part of the original vector. we first apply embedkgqa @xcite directly to the task of temporal kgqa. in its original implementation, embedkgqa uses com-plex (section 4.1) embeddings and can only deal with non-temporal kgs and single entity questions. in order to apply it to cronquestions, we set the first entity encountered in the question as the bert -2.1, 30.2, ... -3.1, -50, ... harry truman [cls] who was the president of usa after world war ii president of usa world war ii occured po si tio n he ld po si tio n he ld 20 08 -20 16 19 45 -19 53 significant event 1939 -1945 barack obama harry truman q11696: president of usa q11613: harry truman q362: world war ii temporal kg embeddings 1944 1945 <@xmath0: the cronkgqa method. (i) a temporal kg embedding model (section 4) is used to generate embeddings for each timestamp and entity in the temporal knowledge graph (ii) bert is used to get two question embeddings: qe ent and qe time . (iii) embeddings of entity/time mentions in the question are combined with question embeddings using equations 4 and 5 to get score vectors for entity and time prediction. (iv) score vectors are concatenated and softmax is used get answer probabilities. please refer to section 5 for details. the scores for all entities and times are concatenated, and softmax is used to calculate answer probabilities over this combined score vector. the model is trained using cross entropy loss. ## experiments and diagnostics in this section, we aim to answer the following questions: