T wi RGCN : Temporally Weighted Graph Convolution for Question Answering over Temporal Knowledge Graphs (2023)

Abstract: Recent years have witnessed interest in Temporal Question Answering over Knowledge Graphs (TKGQA), resulting in the development of multiple methods. However, these are highly engineered, thereby limiting their generalizability, and they do not automatically discover relevant parts of the KG during multi-hop reasoning. Relational graph convolutional networks (RGCN) provide an opportunity to address both of these challenges – we explore this direction in the paper. Specifically, we propose a novel, intuitive and interpretable scheme to modulate the messages passed through a KG edge during convolution based on the relevance of its associated period to the question. We also introduce a gating device to predict if the answer to a complex temporal question is likely to be a KG entity or time and use this prediction to guide our scoring mechanism. We evaluate the resulting system, which we call TwiRGCN, on a recent challenging dataset for multi-hop complex temporal QA called TimeQuestions. We show that TwiRGCN significantly outperforms state-of-the-art models on this dataset across diverse question types. Interestingly, TwiRGCN improves accuracy by 9–10 percentage points for the most difficult ordinal and implicit question types.

## introduction question answering (qa) is a key problem in natural language processing and a long-lasting milestone for artificial intelligence. a large class of approaches for qa makes use of knowledge graphs (kg), which are multi-relational graphs representing facts (kgqa). temporal kgs (tkg) represent facts that are only valid for specific periods of time as (subject, relation, object, time range), for example, (franklin d roosevelt, position held, president of usa, @xcite ). the problem of answering questions that require temporal reasoning over tkgs (tkgqa) is a special case of kgqa that specifically focuses on the following challenge: temporal questions constrain answers through temporal notions, e.g., "who was the first president of us during ww2?" developing systems for temporal qa is of immense practical importance for many applications. it is considered a more challenging problem than kgqa @xcite , where questions are typically about persistent, non-temporal facts (e.g., place of birth), with only a small portion of the questions requiring any temporal reasoning @xcite . even though a variety of models have been proposed for the tkgqa recently, they suffer from the following problems: 1) they are either highly engineered toward the task @xcite or 2) they do not incorporate graph structure information using graph neural networks (gnn) @xcite @xcite . we explore the following hypotheses in this paper: 1) a simple gnn-based solution could generalize better and offer higher performance than highly engineered gnn-based, and tkg embedding-based models; 2) a multilayer gnn model could do multi-hop reasoning across its layers; 3) not all edges (temporal facts) are equally important for answering temporal questions (see figure 1 ), so gnn solutions could benefit from temporally weighted edge convolutions. following the aforementioned hypotheses, we develop a novel but architecturally simple tkgqa system that we call "temporally weighted relational graph convolutional network" (twirgcn). it is based on the relational graph convolutional network (rgcn) proposed by @xcite . twirgcn introduces a questiondependent edge weighting scheme that modulates convolutional messages passing through a temporal fact edge based on how relevant the time period of that edge is for answering a particular question. in rgcn, convolution messages from all tkg edges are weighted equally. but all edges are not equally important for answering temporal questions. for example, in figure 1 , to answer the question "who was the first president of the us during ww2?" the edge with bill clinton has little relevance for answering the question. but, regular rgcn would still weigh all edges equally. we address this shortcoming through our proposed modulation. we impose soft temporal constraints on the messages passed during convolution, amplifying messages through edges close to the time period relevant for answering the question while diminishing messages from irrelevant edges. this leads to better, more efficient learning as we are not confusing our model with unnecessary information, as evidenced by our significantly improved performance without the need for any heavy engineering. we explore two different strategies for our convolutional edge weighting, which show complementary strengths. our experiments establish that twirgcn significantly outperforms already strong baselines on timequestions. our contributions are: • we propose twirgcn, a simple and general tkgqa system that computes questiondependent edge weights to modulate rgcn messages, depending on the temporal relevance of the edge to the question. ## related work most kgqa systems have focused on answering questions from simple (i.e., 1-hop fact-based questions) @xcite to multi-hop complex questions requiring multi-fact reasoning @xcite . however, only a small fraction of these questions require any temporal reasoning @xcite . recent efforts have for that edge. in this example, the entities franklin d. roosevelt and harry truman, who were presidents during ww2 @xcite get the top two highest weights, while woodrow wilson, who was president during ww1 @xcite gets a smaller edge weight. in contrast, bill clinton, whose time period is unrelated to the question, gets a much lower edge weight. thus, contributing very little to the convolution update of the 'president of the us' node. temporal kgqa methods: one line of work uses temporal constraints along with hand-crafted rules to find the answer @xcite @xcite . a recent class of models has leveraged advances in tkg embedding methods for answering questions on temporal kgs. cronkgqa @xcite does this by posing a question as a tkg completion problem and finds the answer using the tcomplex @xcite score function and bert @xcite question embedding to complete the fact. tempoqr @xcite uses additional temporal supervision to enrich tkg embeddings, followed by a transformer-based decoder @xcite . tsqa @xcite on the other hand estimate the time in the question and uses it to enrich tkg embeddings for finding the answer. subgtr @xcite infers question-relevant temporal constraints using tkg embeddings and applies them as filters to score entities in the question subgraph. although we, too, use pre-trained tkg embeddings to initialize our generalized rgcn, we use the gnn framework to take advantage of the structural information in the kg in ways that they do not. recent work @xcite shows that gnn-based models can encode any logical rule corresponding to a path in the knowledge graph. we refer to this as structural information that shallow embedding-based models cannot access. rgcn based qa systems: graph neural networks are increasingly being used in qa systems not specifically meant for temporal reasoning. graft-net @xcite uses personalized pagerank to collect a query-relevant subgraph from a global kg, then an rgcn to predict the answer from the relevant subgraph. pullnet @xcite loops over and expands graftnet's subgraph to do multi-hop reasoning. exaqt @xcite is the system closest to ours: it addresses tkgqa and also uses an rgcn. the rgcn for answer prediction which works on the question subgraph is very similar to that in graftnet. exaqt augments it with dictionary matching, heavy engineering, and additional category information. in contrast, twirgcn uses a straightforward temporally weighted graph convolution followed by answer gating, as described in section 4, while still achieving superior performance (see section 5.3). more details in section 5.2. ## preliminaries a tkg represents such a fact as a quadruple of the form {s, r, o, [t st , t et ]}, where t st is the start time and t et is the end time of validity of the fact, e.g., {barack obama, president of, usa, @xcite }. ## proposed method: twirgcn in this section, we develop and describe twirgcn ("temporally weighted relational graph convolutional network"), our model for tkgqa. ## conclusion in this paper, we proposed twirgcn, a tkgqa system that employs a novel, temporally weighted graph convolution for answering questions that require complex temporal reasoning over a tkg. twirgcn modulates the convolutional messages through a tkg edge based on the relevance of the edge time interval to the question. we present two temporal weighting schemes with complementary strengths, intuitively explained through their simple formulations. we also propose an answer gating system for incorporating the pooled entity and time embeddings from twirgcn in the prediction, based on the likelihood that the answer is a time or an entity, given the question. despite its relative simplicity, twirgcn gives significantly superior tkgqa accuracy on a challenging dataset compared to more heavily engineered baselines.