Learning Neural Ordinary Equations for Forecasting Future Links on Temporal Knowledge Graphs (2021)

Abstract: There has been an increasing interest in inferring future links on temporal knowledge graphs (KG). While links on temporal KGs vary continuously over time, the existing approaches model the temporal KGs in discrete state spaces. To this end, we propose a novel continuum model by extending the idea of neural ordinary differential equations (ODEs) to multi-relational graph convolutional networks. The proposed model preserves the continuous nature of dynamic multi-relational graph data and encodes both temporal and structural information into continuous-time dynamic embeddings. In addition, a novel graph transition layer is applied to capture the transitions on the dynamic graph, i.e., edge formation and dissolution. We perform extensive experiments on five benchmark datasets for temporal KG reasoning, showing our model’s superior performance on the future link forecasting task.

## introduction reasoning on relational data has long been considered an essential subject in artificial intelligence with wide applications, including decision support and question answering. recently, reasoning on knowledge graphs has gained increasing interest @xcite . a knowledge graph (kg) is a graph-structured knowledge base to store factual information. kgs represent facts in the form of triples (s, r, o), e.g., (bob, livesin, new york), in which s (subject) and o (object) denote nodes (entities), and r denotes the edge type (relation) between s and o. knowledge graphs are commonly static and store facts in their current state. in reality, however, the relations between entities often change over time. for example, if bob moves to california, the triple of (bob, livesin, new york) will be invalid. to this end, temporal knowledge graphs (tkg) were introduced. a tkg represents a temporal fact as a quadruple (s, r, o, t) by extending a static triple with time t, describing that this fact is valid at time t. in recent years, several sizable temporal knowledge graphs, such as icews @xcite , have been developed that provide widespread availability of such data and enable reasoning on temporal kgs. while lots of work @xcite @xcite focus on the temporal kg completion task and predict missing links at observed timestamps, recent work @xcite paid attention to forecast future links of temporal kgs. in this work, we focus on the temporal kg forecasting task, which is more challenging than the completion task. most existing work @xcite models temporal kgs in a discrete-time domain where they take snapshots of temporal kgs sampled at regularly-spaced timestamps. thus, these approaches cannot model irregular time intervals, which convey essential information for analyzing dynamics on temporal kgs, e.g., the dwelling time of a user on a website becomes shorter, indicating that the user's interest in the website decreases. knowevolve @xcite uses a neural point process to model continuous-time temporal kgs. however, know-evolve does not take the graph's structural information into account, thus losing the power of modeling temporal topological information. also, knoweolve is a transductive method that cannot handle unseen nodes. in this paper, we present a graph neural-based approach to learn dynamic representations of entities and relations on temporal kgs. specifically, we propose a graph neural ordinary differential equation to model the graph dynamics in the continuoustime domain. inspired by neural ordinary differential equations (nodes) (chen et al., 2018), we extend the idea of continuum-depth models to encode the continuous dynamics of temporal kgs. to apply nodes to temporal kg reasoning, we employ a node coupled with multi-relational graph convolutional (mgcn) layers. mgcn layers are used to capture the structural information of multi-relational graph data, while the node learns the evolution of temporal kgs over time. specifically, we integrate the hidden representations over time using an ode solver and output the continuous-time dynamic representations of entities and relations. unlike many existing temporal kg models that learn the dynamics by employing recurrent model structures with discrete depth, our model lets the time domain coincide with the depth of a neural network and takes advantage of node to steer the latent entity features between two timestamps smoothly. besides, existing work simply uses the adjacency tensor from previous snapshots of the tkg to predict its linkage structure at a future time. usually, most edges do not change between two observations, while only a few new edges have formatted or dissolved since the last observation. however, the dissolution and formation of these small amounts of edges always contain valuable temporal information and are more critical than unchanged edges for learning the graph dynamics. for example, we know an edge with the label eco-nomicallycooperatewith between two countries x and y at time t, but this dissolves at t + ∆t 1 . additionally, there is another edge with the label bantradeswith between these two countries that are formated at t + ∆t 2 (∆t 2 > ∆t 1 ). intuitively, the dissolution of (x, economicallycooperatewith, y) is an essential indicator of the quadruple (x, bantradeswith, y, t + ∆t 2 ). thus, it should get more attention from the model. however, suppose we only feed the adjacency tensors of different observation snapshots into the model. in that case, we do not know whether the model can effectively capture the changes of the adjacency tensors and puts more attention on the evolving part of the graph. to let the model focus on the graph's transitions, we propose a graph transition layer that takes a graph transition tensor containing edge formation and dissolution information as input and uses graph convolutions to process the transition information explicitly. in this work, we propose a model to perform temporal knowledge graph forecasting with neural ordinary equations (tango ). the main contributions are summarized as follows: ## our model our model is designed to model time-evolving multi-relational graph data by learning continuoustime representations of entities. it consists of a neural ode-based encoder and a decoder based on classic kg score functions. as shown in figure 1b , the input of the network will be fed into two parallel modules before entering the ode solver. the upper module denotes a multi-relational graph convolutional layer that captures the graph's structural information according to an observation at time t. and the lower module denotes a graph transition layer that explicitly takes the edge transition tensor of the current observation representing which edges have been added and removed since the last observation. the graph transition layer focuses on modeling the graph transition between neighboring observations for improving the prediction of link formation and dissolution. for the decoder, we compare two score functions, i.e., distmult @xcite and tucker @xcite . in principle, the decoder can be any score function. in addition to f mgcn , a graph transition layer f trans is employed to model the edge formation and dissolution. ## conclusions we propose a novel representation method, tango , for forecasting future links on tem-poral knowledge graphs (tkgs). we propose a multi-relational graph convolutional layer to capture structural dependencies on tkgs and learn continuous dynamic representations using graph neural ordinary differential equations. especially, our model is the first one to show that the neural ode can be extended to modeling dynamic multirelational graphs. besides, we couple our model with the graph transition layer to explicitly capture the information provided by the edge formation and deletion. according to the experimental results, tango achieves state-of-the-art performance on five benchmark datasets for tkgs. we also propose two new tasks to evaluate the potential of link forecasting models, namely inductive link prediction and long horizontal link forecasting. tango performs well in both tasks and shows its great potential.