Continuous Temporal Graph Networks for Event-Based Graph Data (2022)

Abstract: There has been an increasing interest in modeling continuous-time dynamics of temporal graph data. Previous methods encode time-evolving relational information into a low-dimensional representation by specifying discrete layers of neural networks, while real-world dynamic graphs often vary continuously over time. Hence, we propose Continuous Temporal Graph Networks (CTGNs) to capture continuous dynamics of temporal graph data. We use both the link starting timestamps and link duration as evolving information to model continuous dynamics of nodes. The key idea is to use neural ordinary differential equations (ODE) to characterize the continuous dynamics of node representations over dynamic graphs. We parameterize ordinary differential equations using a novel graph neural network. The existing dynamic graph networks can be considered as a specific discretization of CTGNs. Experiment results on both transductive and inductive tasks demonstrate the effectiveness of our proposed approach over competitive baselines.

## introduction graph neural networks (gnns) have attracted growing interest in the past few years due to their universal applicability in various fields, e.g., social networks @xcite and natural language processing @xcite . graph neural networks (gnns) learn a lower-dimensional representation for a node in a vector space by aggregating the information from its neighbors using discrete hidden layers. then the embedding can be used for downstream tasks such as node classification @xcite , link prediction @xcite , and knowledge completion @xcite . most graph neural networks only accept static graphs as input, although real-life graphs of interactions, such as user-item interactions, often change figure 1 : the importance of link duration. consider the behavior of a user watching movies. there are two types of nodes in the graph: user nodes and item nodes. given the user's historical behavior, the predicted target is (user 1 , don't_click, movie_4). if we ignore the link duration information, user 1 seems interested in cartoon movies because he clicked on it at timestamp t 1 . but user 1 only watched the movie_ 1 for 10s. the link duration indicated that although the user clicked, he was not interested. although the above continuous-time dynamic methods have achieved impressive results, they still have limitations. the majority of research @xcite @xcite @xcite pays attention to the contact sequence dynamic graphs, in which the links are permanent, and no link duration is provided (e.g., email networks and citation networks). however, most real-life networks are event-based dynamic graphs in which the interactions between source nodes and destination nodes are not permanent (e.g., employment networks and the existing gnn-based methods @xcite that learn the node representation over dynamic graphs can be considered discrete dynamical systems. @xcite demonstrate that the continuous dynamical systems are more efficient for modeling continuous-time dynamic data. the discrete networks can roughly be regarded as continuous networks by stacking enough layers. however, @xcite point out that graph neural networks (gnns) exponentially lose expressive power for downstream tasks, which will lead to over-smoothing problems as we add more hidden layers. therefore, designing effective continuous graph neural networks to model continuous-time dynamics of node representation on dynamic graphs is critical. to this end, many continuous graph neural networks @xcite have been proposed recently. although those mentioned above continuous dynamic neural networks are more efficient to model the graph data, few approaches have been proposed for dealing with dynamic graphs using continuous-time dynamic neural networks. this paper proposes a general framework of continuous temporal graph networks (ctgns) to model continuous-time representations for dy-namic graph-structured data. we combine ordinary differential equation systems (odes) and graphs methods. instead of specifying discrete hidden layers, we integrate neural layers over continuous time. figure 2 illustrates the workflow of the proposed ctgn method. there is an interaction between two nodes. first, a novel temporal graph network (tgn) is applied as the encoder to learn the latent states using the updated memory. then, the neural ode module is used to model the node's continuous-time representation. considering that the link duration reflects the degree of association between the two nodes, we use the link duration as the integration variable to control the weights of different interactions. after that, we use the lstm @xcite as the decoder to compute the probability of interaction between the two given nodes. finally, the memory is updated as the input of the encoder. memory is a compressed representation of the historical behavior of all nodes defined in section 3.1. experimental results on five real-world datasets of link prediction demonstrate the effectiveness of the proposed method over the state-of-art baselines. the main contributions of this paper are: ## the proposed method: ctgn in this section, we introduce our proposed approach. the key idea of the ctgn is to build continuous-time hidden layers which can learn continuous informative node representations over event-based dynamic graphs. to characterize the continuous dynamics of node representation, we use ordinary differential equations (odes) parameterized by a neural network, which is a continuous function of time. we study both transductive and inductive settings. in the transductive task, we predict future links of the nodes observed during the training phase. in the inductive tasks, we predict future links of the nodes never seen before. we first employ a temporal graph attention layer @xcite to project each node into a latent space based on its features and neighbors. and then, an ode module is designed to define the continuous dynamics on the node's latent representation h i (t). ## experiment and analysis in this section, we first introduce datasets, baselines and parameter settings. then we compare our proposed method with other strong baselines and competing approaches for both the inductive and transductive tasks for two benchmarks contact sequence dynamic graph datasets and three eventbased dynamic graph datasets. we study both transductive and inductive tasks. for event-based dynamic graphs, we learn link prediction tasks. for contact-sequence dynamic graphs, we learn dynamic node classification and link prediction tasks. ## conclusion this paper introduces ctgn, a continuous temporal graph neural network for learning representation for event-based dynamic graphs. we build the connection between temporal graph networks and continuous dynamical systems inspired by neural ode. our framework allows the user to trade off speed for precision by selecting different learning rates and the weight of time smoothness loss parameters during training. we demonstrate on the link prediction task against competitive baselines that our model can outperform many existing stateof-the-art methods.