Neural Temporal Walks: Motif-Aware Representation Learning on Continuous-Time Dynamic Graphs
Abstract
Continuous-time dynamic graphs naturally abstract many real-world systems, such as social and transactional networks.
While the research on continuous-time dynamic graph representation learning has made significant advances recently, neither graph topological properties nor temporal dependencies have been well-considered and explicitly modeled in capturing dynamic patterns.
In this paper, we introduce a new approach, Neural Temporal Walks ( NeurTWs ), for representation learning on continuous-time dynamic graphs.
By considering not only time constraints but also structural and tree traversal properties, our method conducts spatiotemporalbiased random walks to retrieve a set of representative motifs, enabling temporal nodes to be characterized effectively.
With a component based on neural ordinary differential equations, the extracted motifs allow for irregularly-sampled temporal nodes to be embedded explicitly over multiple different interaction time intervals, enabling the effective capture of the underlying spatiotemporal dynamics.
To enrich supervision signals, we further design a harder contrastive pretext task for model optimization.
Our method demonstrates overwhelming superiority under both transductive and inductive settings on six real-world datasets 1 .
Introduction
Continuous-time dynamic graphs (CTDGs) consist of temporal events with respect to nodes (e g., node addition/deletion) and edges (i e., temporal interactions), which naturally arise in many realworld systems such as social networks and knowledge graphs [13, 32].
Traditional studies in dynamic graph modeling manually extract expressive patterns that are beneficial for understanding the crucial laws behind [36].
For example, two people are likely to know each other if they have a common friend (Figure 1).
Such a dynamic graph motif describes how social connections are established [31].
Other expressive patterns, such as feedforward control loops, have also been investigated [20].
However, manually extracting motifs is expensive, time-consuming, and requires domain knowledge, thus hindering the learning on dynamic graphs with more complicated laws [36].
$ # $ $ b c $ # $ % $ $ a b c $ # $ $ a $ %
Temporalwalks: a
[Figure]
∆$ $ c b
[Figure]
$ #
[Figure]
The advent of graph neural networks (GNNs) makes it possible to understand more complicated graphs by automatically learning the laws behind [40, 42].
While
GNNs have demonstrated great success in modeling static graphs, the research on dynamic graphs is still nascent.
Current research on dynamic graph neural networks (DGNNs) faces two fundamental challenges.
Firstly , the entangled spatial and temporal dependencies in real-world CTDGs typically need a specific design to model, preventing the direct use of off-the-shelf GNNs.
To overcome this barrier, previous works simplify CTDGs to a series of static graph snapshots with uniform time intervals (i.e., discrete-time dynamic graphs, DTDGs) [24, 29].
However, this approximation compromises the modeling precision.
Although [15] and [34] propose to learn on CTDGs directly, the inductiveness of the patterns they captured is not guaranteed because node identities are directly involved in their modeling process.
Some recent methods [38, 27, 36] attempt to alleviate this issue.
However, they only consider time but not topological and tree traversal properties when sampling temporal neighbors, limiting their ability to extract diverse and expressive patterns from dynamic systems.
Secondly , temporal events in CTDGs occur irregularly (e.g., nodes a and c interact with b at different timestamps in Figure 1), resulting in a significant challenge in modeling temporal dependencies.
Previous works typically bypass this challenge with the time encoding [38] to enable the use of message passing [38, 27] or sequence models [15, 36].
However, time encoding hurts model performance as temporal dependencies are modeled implicitly.
To tackle the above challenges, we propose the Neural Temporal Walks ( NeurTWs ) method for representation learning on CTDGs, which extracts and encodes informative dynamic graph motifs composed of irregularly-sampled temporal nodes.
Specifically, motivated by [36], we propose sptaiotemporal-biased random walks to extract diverse and expressive patterns from a CTDG by not only considering time constrains but also topological properties and tree traversals, allowing the sampler to be better aware of the importance of temporal neighbors while maintaining the exploration and exploitation trade-off.
To explicitly model temporal dependencies and capture essential dynamic laws more effectively, the extracted motifs consisting of irregularly-sampled temporal nodes are encoded with the proposed continuous evolution and instantaneous activation processes to learn time-aware node representations.
The former process learns latent spatiotemporal dynamics across multiple interaction time intervals with an ordinary differential equation (ODE) function, and the second process regularizes the latent state trajectories with those irregularly-sampled observations.
On this basis, we acquire time-aware node embeddings by retrieving and encoding neighboring representative dynamic graph motifs, where a contrastive objective can be naturally designed to optimize NeurTWs by maximizing the mutual information between interacting temporal nodes.
Compared with most existing works that use a simple link prediction objective, this harder contrastive pretext task helps enrich supervision signals, thus lifting the learning ability of NeurTWs .
On five benchmark datasets and a new dense e-commerce dataset, our method significantly and consistently outperforms all state-of-the-art methods in general.
Specifically, it surpasses the strongest baselines by around 3% and 5% in all transductive and seven out of eight inductive link prediction tasks.
It also achieves the best or on-par performance on dynamic node classification tasks.
In summary, our technical contributions are three-fold: (1) We propose novel spatiotemporal-biased random walks to extract diverse and expressive patterns from CTDGs by considering not only time constraints but also topological and tree traversal properties; (2) We introduce a new perspective to encode dynamic graph motifs composed of irregularly-sampled temporal nodes, explicitly and better modeling the underlying spatiotemporal dynamics; (3) We integrate contrastive learning into dynamic graph modeling to enrich supervision signals, which lifts the learning ability of our model.
Related Work
Dynamic Graph Neural Networks (DGNNs).
Existing DGNNs can be broadly classified into two categories based on their inputs.
Discrete-time DGNNs operate over a sequence of evenly-sampled static graph snapshots, where different strategies are proposed to model spatial and temporal clues, e.g., combining GNNs with sequence models [30, 24, 5, 29].
Our work relates to the second category, continuous-time DGNNs, where time-dependent node or edge embeddings are learned directly on
CTDGs.
Among these works, an in-demand design updates latent node states by aggregating k -hop neighborhood information with temporal message passing.
For instance, TGAT [38] samples a set of k -hop temporal neighbors and proposes learnable time encodings to preserve time information in message passing.
TGN [27] further equips TGAT with a node memory update mechanism as in [15].
Another line of research leverages random walks to learn on CTDGs.
Specifically, CTDNE [23] is the first to propose a CTDG embedding method with temporal walks.
CAWs [36] extend this concept with anonymous temporal walks and uses a recurrent net to learn walk embeddings that are further aggregated when calculating interactive representations.
Our method is different from prior walkbased approaches in three aspects: (1) We propose a new perspective on sampling temporal walks.
While prior arts only consider time constraints, our method leverages multidimensional information, allowing the model to explore diverse and expressive patterns from CTDGs; (2) We propose a novel and intuitive motif embedding method to model latent spatiotemporal dynamics among irregularlysampled temporal nodes on CTDGs without relying on time encodings, which allows temporal dependencies to be modeled explicitly; (3) We replace the simple link prediction-based learning objective with a more challenging contrastive pretext task, which helps provide stronger supervision signals.
Neural Ordinary Differential Equations (NODEs).
Chen et al.
[2] propose a new paradigm that generalizes discrete deep neural networks by parameterizing the derivative of latent states.
This concept has been applied in research areas including time series forecasting [12, 28] and computer vision [9, 1].
Recently, some works have extended NODEs to the graph domain, where most consider building deeper GNNs while alleviating the negative impacts of over-smoothing [37, 25].
Notably, the time information is absent among those works.
In dynamic graph learning, most ODE-based works focus on discrete-time settings [12, 6, 8], and only a few extend NODEs to learn on CTDGs [7].
In this paper, inspired by the research on time series forecasting [28], we propose NeurTWs to encode extracted motifs with irregularly-sampled temporal nodes on CTDGs, which is fundamentally different from [7]: We directly integrate over multiple interaction time intervals to explicitly model the latent spatiotemporal dynamics across different temporal nodes with an ODE function, while [7] relies on a time encoding-assisted message passer to learn from historical temporal events.
Graph Contrastive Learning (GCL).
Recently, GCL has achieved great success in graph selfsupervised learning [19].
Most existing works are on static graphs [35, 43, 44, 11].
While some studies have explored the possibility of dynamic graph contrastive learning, many of them are on DTDGs.
For example, STGCL [18] enhances the model’s forecasting ability with DTDG augmentations and an auxiliary contrastive loss.
A similar design also exists in [41, 39] and [26].
On CTDGs, Tian et al.
[33] propose to maximize the agreement between time-aware node embeddings at different time points.
In DySubC [10], the mutual information between a node and its surrounding temporal subgraphs is maximized.
Different from these works, we design an effective pretext task for model optimization, where the mutual information between two nodes in an interaction is maximized.
Meanwhile, we push nodes away in the embedding space if there are no temporal interactions.
Problem Formulation
We start by formally introducing the learning problem on CTDGs.
A complete notation table is in
Appendix A.
This paper defines a CTDG as a stream of temporal interactions, i.e., G = { ( e i , t i ) } Ni =1 , where each interaction has two nodes at a specific time, e.g., ( e i , t i ) :
= ( { u i , v i } , t i ) , t i ∈ R + .
As many real-world CTDG datasets are unattributed and for simplicity, we first assume these temporal interactions are without node and edge attributes and later we will discuss how our method is extended to learn on attributed CTDGs.
Facing the challenge of lacking label information, DGNNs are typically supervised by temporal interactions [45].
Thus, dynamic link prediction is a widely adopted testbed to evaluate how accurate a DGNN is in predicting future interactions with the observation of historical events.
Specifically, given two nodes u and v at time t in G , we aim to learn their time-aware embeddings h u and h v , where the presence of an interaction between them can be predicted with a downstream classifier, i.e., (cid:98) y u,v,t = clf
( h u , h v ) .
The ground truth y u,v,t = 1 if there exists an interaction between u and v at time t otherwise y u,v,t = 0 .
On this basis and with learned time-aware node representations, conducting other downstream tasks, such as dynamic node classification with another classifier, is also feasible.
The Proposed Method: Neural Temporal Walks 4.1 Preliminaries: Temporal Walks and Dynamic Graph Motifs
Given a dynamic graph G , we define a motif as a subset of temporal nodes with their interactions within a defined time range [14], e.g., 0 ≤ t ≤ q .
As those motifs reflect certain dynamic laws in a
CTDG, it is desirable to characterize a temporal node with its surrounding motifs.
Definition 4.1.1 (Temporal Walk).
Given a dynamic graph G , we denote the interactions that are directly associated with a node u before a cut time t as G u,t = { ( e, t ′ )
| t ′ <
t, u ∈ e , ( e, t ′ )
∈ G} .
A (time-reversed) temporal walk rooted at node u at time t is defined as W , which is a sequence of u b c u  4 7 b u c 3 4 v e d v3 6 e v d u b c u  v e d v
Twodifferenttemporalwalks
Anonymizedwalks
Dynamicgraphmotif 0 ≤ # ≤
[Figure]
b c v d g f e t=8?
5, 3, 3, 0,2, 2,73, u 1, 0,4,  1,
[Figure]
u f v d  … v e g v 4 … ! ! ! "
[Figure]
temporal nodes as in [36], i.e., node w i at time t i with w 0 :
= u and t 0 :
=
t :
W = { ( w i , t i ) | i ∈ N , 0 ≤ i ≤ l, t 0 > t 1 > · · · > t l , (
[Equation]
w
[Equation]
−
[Equation] [Equation]
∈
[Equation]
i −
[Equation]
t i − 1 for
[Equation]
≥ 1 }
[Equation]
where l is the walk length.
We also use W [ i ][0] and W [ i ][1] (i.e., w i and t i in ( w i , t i ) respectively) to denote the specific node and time in the i -th step.
Each walk W rooted at a temporal node can actually be regarded as one of its surrounding motifs if t 1 − t l is bounded within the defined motif time range.
A concrete example is given in Figure 2, where two walks in the leftmost side form two triadic closures, which essentially represent the same motif on the rightmost side after an individual walk anonymization [21].
The necessity of anonymization is to replace original node identities in walks with their relative identities, which maps each walk to a particular pattern and thus connects temporal walks and dynamic graph motifs.
Definition 4.1.2 (Anonymous Walk).
Given a temporal node w and a walk W , the anonymization operation A ( · ) is defined as follows [36]:
[Equation]
where i ∗ is the smallest index
s.t
[Equation]
4.2 Dynamic Graph Motif Extraction
Temporal Walk Sampling.
Existing path-based methods mainly employ a temporal-biased sampling method when extracting dynamic graph motifs [23, 36].
Specifically, given a node u at time t , the probability of its neighbor a in ( { a, u } , t ′ ) ∈ G u,t to be sampled is proportion to exp ( α ( t ′ − t )) , which discounts stale neighbors and tends to sample more current nodes with timestamps closer to t .
A larger α emphasizes more on this bias.
Although more current neighbors are more likely to be informative, the underlying topological and tree traversal properties are not respected, which hinders the extraction of diverse and expressive patterns.
Here, we propose spatiotemporal-biased random walks with the exploitation and exploration trade-off.
Our motivations are twofold: (1) Most-recent neighbors should be allocated a larger sampling probability ∝ exp ( α ( t ′ − t )) as they are typically more informative w.r.t.
a root node at time t .
In
Figure 3(a), given a root node u and its two temporal neighbors f and c , a temporal-biased sampling path is more likely to be u → f instead of u → c ; (2) Neighbors with higher connectivity need to be emphasized to allow exploring more diverse and potentially expressive motifs.
Given a node a at time t ′ , we use its degree d a = |G a,t ′ | to reify its connectivity, thus the proposed spatial-biased probability ∝ exp ( − β/d a ) with a hyperparameter β to control the bias intensity.
Algorithm 1 illustrates the walk sampling procedures with the above considerations.
Given a node u at time t , the probability of its temporal neighbor a to be sampled is the average of the following normalized probabilities:
[Equation]
Algorithm
Require:
Root node w 0 , cut time t 0 , G , C , l 1: Initialize { W c = (cid:0) ( w 0 , t 0 ) (cid:1) | 1 ≤ c ≤ C } 2: for i in 1 , 2 , · · · , l do 3: for j in 1 , 2 , · · · , C do 4: w p , t p :=
W j [ i ][0] , W j [ i ][1] 5:
Initialize d w = 0 for all w ∈ G w p ,t p 6: for ( e, t ) in G w p ,t p do 7:
Let e := { w, w p } , d w = |G w,t | 8: end for 9:
Sample one ( e, t ) ∈ G w p ,t p with prob.
∝ exp ( α ( t − t p )) and exp ( − β/d w ) 10:
Let e : = { w, w p } , W c = W c || ( w, t ) 11: end for 12: end for 13: return { W c | 1 ≤ c ≤ C }
Although Algorithm 1 complements the temporalbiased schema by considering the additional topological information, it may overly encourage the depthfirst search (DFS), which could misleadingly sample many homogeneous motifs with a limited budget.
Take an extreme example, if | M u | is restricted to 3 in Figure 3(a), paths u
→
b → c → u may be sampled three times, leaving no room to explore u → c → b → a and u → f → v → d .
Thus, we design an exploitation & exploration trade-off to regularize the walk sampling with another probability:
[Equation]
where s a and γ denotes the traversal times of node a and the intensity of such a regularization.
Our complete walk sampling algorithm and its complexity analysis are in Appendices B.1 and B.3.
In a nutshell, given a temporal node, the probability of its neighbor to be sampled is the average of the probabilities defined in Equations 3, 4 and 5.
In NeurTWs , given a queried interaction between two temporal nodes u and v as shown in Figure 3, we sample a set of C temporal walks rooted at each node with length l , denoted as M u and M v .
Anonymization.
Walk anonymization replaces node identities with position encodings (aka relative identities), which injects structural information while maintaining the inductiveness of NeurTWs .
A drawback of Equation 2 is that the position encoding of each node only depends on its specific walk, leading anonymous walks rooted at the same node sharing different name spaces [36].
Thus, we consider two practical solutions: unitary and binary anonymization to address this problem.
For a temporal node w in at least one walk rooted at node u , its unitary anonmization w.r.t.
u considers the name space defined over M u , the set of walks rooted at u :
[Equation]
where i ∈ {
[Equation] [Equation]
[Equation]
In Equation 6, the identity of w is replaced by a vector A ( w ; M u ) with length l , where the i -th element counts the number of walks that have node w appearing in position i .
While unitary anonymization anonymizes node w w.r.t.
M u as A ( w ; M u ) , for interacting root nodes u and v , A ( w ; M u ) and A ( w ; M v ) belong to different name spaces.
Since DGNNs are typically supervised by temporal interactions, establishing the correlations between W ∈ M u ∪ M v may be beneficial.
Thus, the binary anonymization is defined as follows [36]:
[Equation]
where || denotes the concatenation operation to establish the connections among motifs in M u and
M v .
In the rest of the paper, we abbreviate the two anonymization strategies as A ( w ) for simplicity and denote (cid:99) W = { (cid:0) A ( w i ) , t i (cid:1) | ( w i , t i )
∈ W for i = 0 , 1 , · · · , l } as an anonymous walk.
4.3 Neural Temporal Walk Encoding
A significant challenge to model CTDGs is that interactions occur irregularly.
Previous works bypass this challenge by concatenating node attributes with extra time encodings when aggregating the neighbourhood information [38, 27, 36], where temporal dependencies are modeled implicitly.
A detailed discussion is in Appendix D.2.
To encode a motif with irregularly-sampled temporal nodes, we explicitly integrate over multiple interaction time intervals to learn the latent spatiotemporal dynamics with those discrete observations.
Figure 4 and Algorithm 2 describe our method in a nutshell, which consists of two interleaving steps: Continuous evolution and instantaneous activation .
Require:
An anonymous temporal walk (cid:99) W = { (cid:0) A ( w i ) , t i (cid:1) | ( w i , t i ) ∈ W for i = 0 , 1 , · · · , l } 1: Reverse the order of elements in (cid:99) W 2: t − 1 = t 0 , h − 1 =3: for i in 0 , 1 , 2 , · · · , l do 4:
h ′ i = ODESolve ( h i − 1 , f θ , t
i − 1 , t i ) 5:
A ′ ( w i ) =
MLP ψ
( A ( w i ))
6: h i =
g ϕ ( h ′ i , A ′ ( w i )) 7: end for 8: return The walk embedding h l
Irregulartrajectoryoflatentstateevolution !
(") ∆6 # , ∆6 $ , ∆6 % ,7 NeuralTemporalWalkEncoding
Encodean anonymizedtemporalwalk 23 : !
(") ∆! ! )
Feature Extraction ∆!
# ∆! "
Continuous Evolution.
Given a series of temporal nodes at different time, i.e., ( A ( w i ) , t i ) ∈ (cid:99) W and ensuring t i − 1 < t i by reversing the order of elements in (cid:99) W , the latent spatiotemporal dynamics among those nodes are modeled as follows:
[Equation]
(8) where h i − 1 denotes the latent states after encoding ( A ( w i − 1 ) , t i − 1 ) ∈ (cid:99) W .
We define the ODE function f
( h t , θ ) as an autoregressive gated recurrent unit parameterized by θ .
See Appendix B.2.
Instantaneous Activation.
The latent state evolution in Equation 8 conditions on a series of discrete observations.
Thus, we define a function to activate latent state trajectories with instantaneous inputs:
[Equation]
where g ( · , ϕ ) can be a standard RNN cell parameterized by ϕ , and A ′ ( w i )
=
MLP ( A ( w i ) , ψ ) denotes the linear mapping of a discrete observation A ( w i ) in an anonymous walk (cid:99) W .
Compared with [36], we naturally model spatiotemporal dynamics behind walks with irregularlysampled temporal nodes, where the time information has been explicitly reflected in this modeling process.
On this basis, for a temporal node u , we obtain its time-aware representation by pooling the embedding of walks in M u , denoted as h u .
In this paper, we adopt the sum pooling for simplicity.
4.4 Contrastive Optimization
A self-supervised pretext task is required to train DGNNs due to the scarcity of the labeling information.
Most current works formulate their learning problems as a binary classification task, where the existence of an interaction is predicted.
From the contrastive point of view, a binary cross-entropy loss essentially forms a particular case of the Jensen-Shannon estimator [19], where the number of negatives is one and thus provides limited supervision signals.
Here, we introduce a harder contrastive pretext task, where the mutual information between interacting temporal nodes (e.g., node u and v in Figure 3) is maximized.
We detail the complete training algorithm of our method in Appendix B.4.
In brief, the following noise-contrastive loss is minimized in the proposed approach:
[Equation]
(cid:1)(cid:105) .
(10) sim ( · ) measures the similarity between two entities, i.e., sim ( h u , h v )
= σ (cid:0) MLP ( h u , h a , ξ ) (cid:1) , where σ ( · ) and ξ are sigmoid activation and trainable parameters.
4.5 Extension and Discussion
Learning on Attributed CTDGs.
Our method can be easily extended to model CTDGs with node and interaction attributes.
To achieve this, we only need to slightly modify Equation 9 as follows:
[Equation]
[Table]
where X w i and X e i are linearly mapped features of node w i and edge e i :
= { w i − 1 , w i } in (cid:99) W .
Batching and Computational Complexity.
One computational challenge is that each walk contains irregularly-sampled nodes at different timestamps, requiring the model to separately solve C different
ODEs to calculate the embedding of a root node, thus hindering training in batches.
To solve all
ODEs at once in a minibatch with B sampled interactions, we unify the integral time among all ODEs (see Appendix B.3) to a certain range, which results in a lower time complexity of O ( l ) instead of
O ( Bl ) .
In Appendix B.3, we also provide detailed complexity analysis and discuss how to make solving Equation 8 tractable with very large time intervals.
Experiments 5.1 Experimental Setting
Baselines.
Our model is compared with six state-of-the-art methods in modeling CTDGs.
They can be divided into two categories based on their intrinsic mechanisms: (1) message passing-based methods, including DyRep [34], TGAT
[38], and TGN [27]; (2) sequential model-based methods, including CTDNE [23], JODIE [15], and CAWs [36].
More details can be found in Appendix C.2.
Datasets.
We evaluate model performance on six real-world datasets.
CollegeMsg [17] is a social network dataset consisting of message sending activities.
Enron [17] is an email communication network.
Taobao [46] is an attributed user behavior dataset containing user-item interactions.
MOOC [17] is an attributed network consisting of student-course interactions.
Wikipedia and Reddit [15] are two bipartite interaction networks consisting of editor-page and user-post activities with rich attributive information.
The dataset statistics are summarized in Table 1, and their detailed descriptions are in Appendix C.1.
Evaluation Protocols.
For link prediction, we follow the evaluation protocols of CAWs [36] and consider two different downstream tasks for evaluation: transductive and inductive link prediction.
In transductive link prediction, we sort and divide all N interactions in a dataset by time into three separate sets for training, validation, and testing.
Specifically, the ranges of training, validation, and testing sets are [0 , N trn ) , [ N trn , N val ) , [ N val , N ] , where N trn /N and N val /N are 0.7 and 0.85.
In inductive link prediction, we use the same splits but mask a proportion of nodes (10%) and associated interactions during training, which are predicted during inference to evaluate the model inductiveness.
Specifically, we first remove all interactions connected with masked nodes in the training set and then remove all interactions not associated with them in the validation and testing sets.
In particular, we have two specific settings: (1) New-Old tasks require the model to predict the interactions with one unobserved (i.e., masked) and one observed nodes; and (2) New-New tasks aim to predict the interactions between all unobserved nodes.
For dynamic node classification, we follow the evaluation protocol in [27].
Specifically, we first obtain a model under the setting of transductive link prediction.
Then, we train and test a separate classifier on top of this pre-trained model with the temporal nodes observed in [0 , N val ) and [ N val , N ] .
Training Details.
We implement and train all models under a unified evaluation framework with the Adam optimizer.
The tuning of primary hyperparameters is discussed in Appendix C.3.
In solving ODEs, we use the Runge-Kutta method with the number of function evaluations set to 8 by default.
For fair comparisons and simplicity, we use sum-pooling when calculating node representations in both our method and CAWs.
We also test NeurTWs † , which is equipped with the binary anonymization, while NeurTWs adopts the default unitary anonymization.
All methods are tuned thoroughly with nonlinear 2-layer and 3-layer perceptrons to conduct downstream link
[Table]
prediction and node classification tasks, and we adopt the commonly used Area Under the ROC
Curve (AUC) and Average Precision (AP) as the evaluation metrics.
More implementation details can be found in Appendix C.3.
5.2 Results and Discussion
w.r.t.
AUC.
We use bold font and underline to highlight the best and second best performances.
[Table]
The link prediction performance of our method and state-of-the-art baselines are summarized in
Table 2 (w.r.t.
AUC) and Appendix D.1 (w.r.t.
AP).
As can be seen from this table, our method achieves the best performance on different tasks and datasets in general.
Notably, NeurTWs surpasses the strongest model, CAWs, significantly on both unattributed and attributed CTDGs, e.g., 7.76% and 4.78% on average on CollegeMsg and
Taobao.
Particularly, NeurTWs and NeurTWs † achieve a strong AUC result ≥ 0.95 on CollegeMsg, while all baselines have their AUCs ≤ 0.9, demonstrating the effectiveness of the proposed method.
Specifically, on both transductive and inductive tasks, we make the following observations.
Our method performs well on both attributed and unattributed datasets, while the effectiveness of baseline models varies significantly.
For instance, JODIE and DyRep have enormous performance gaps between Taobao and CollegeMsg in all three tasks.
It indicates that our NeurTWs method is better at capturing essential dynamic laws.
This superiority can be attributed to our spatiotemporal walks and associated encoding mechanism; (2) Our method is effective under both transductive and inductive settings.
On the contrary, some baseline models cannot well generalize to predict interactions with unseen nodes (e.g., the performance of JODIE and DyRep drops on inductive tasks) because they rely on node identities.
TGAT, TGN, and CAWs can yield better performances but leave room to improve.
It is worth noting that simple unitary anonymization (i.e., NeurTWs ) and more complex binary anonymization (i.e., NeurTWs † ) yield similar performance in most cases.
Our method with the default, simpler unitary anonymization performs significantly better than CAWs, which utilizes binary anonymization, again demonstrating the superiority of our approach in modeling
[Table]
CTDGs without relying on sophisticated position encodings; (3) Comparing the results on Taobao, our method’s improvements over CAWs are more significant than on CollegeMsg, e.g., 7.48% vs.
3.81% on transductive and 7.91% vs. 5.26% on inductive settings.
Similar observations can also be found when compared on MOOC and CollegeMsg.
This is because the CollegeMsg dataset has a lower average interaction intensity (i.e., λ = 3 .
79 × 10 − 6 ) compared with Taobao and MOOC datasets (i.e., λ = 6 .
64 × 10 − 5 and 4 .
48 × 10 − 5 ).
A lower λ indicates a larger average time span in a temporal walk, for which our proposed continuous evolution process is more effective.
In addition, we compare the performance of our method (i.e., NeurTWs ) on dynamic node classification with state-of-the-art baselines as shown in Table 3, where our approach achieves the best or on-par performances, further confirming the effectiveness of the proposal.
We do not report the results of NeurTWs † because node classification mainly considers properties of temporal nodes rather than interactions.
5.3 Ablation Study
In Table 4, we report the results of our ablation study on predicting inductive interactions in both new-old and new-new settings.
Specifically, ablations 1, 2, and 3 remove Equations 3, 4, andwhen sampling temporal walks.
Ablation 4 disables Equation 8 when encoding an anonymous walk.
Ablation 5 replaces our contrastive loss with a binary cross entropy loss.
From the above variants, we see performances degradation when retrieving motifs without considering the temporal or spatial information, demonstrating the effectiveness of the proposed spatiotemporal-biased walk sampler.
Moreover, disabling the exploitation & exploration trade-off also hurts performances.
When removing the continuous evolution, more severe performance drops are found on CollegeMsg, further demonstrating that NeurTWs excel on CTDGs with more sparse interactions where existing works usually fail to model them well.
The performance drops also exist when our contrastive pretext task is disabled.
A specific study related to continuous evolution is in Appendix D.2.
5.4 Parametric Sensitivity
We study the important settings in NeurTWs and have the following observations: (1) For each dataset, there are sweet spots in balancing three intensities (i.e., α , β , and γ ).
Specifically, different datasets with different average interaction intensities, namely λ , prefer different temporal-biased intensities.
For example, a relatively large α benefits on CollegeMsg but hurts the performance on Taobao as shown in Figure 5(a).
On both datasets, overly emphasizing the topology information with a large β hurts the performance (see Figure 5(b)) because the temporal information is overly neglected.
However, setting a relatively large γ seems beneficial (see Figure 5(c)), indicating the importance of balancing exploitation and exploration in temporal walk sampling; (2) We also investigate the choice of walk length l and number of walks C as shown in Figures 5(e) and 5(f).
On both datasets, sampling 16 or 32 walks with the length 2 or 3 seems sufficient to characterize a temporal node; (3) For efficiency, we limit the number of negatives in Equation 10 instead of calculating h v ′ for all v ′ ∈ G where v ′ ̸
= v .
Specifically, we find that a large number is usually beneficial on both datasets as shown in Figure 5(d), where the performance increases to converge with an increasing number of negatives.
In our method, we normally do not have to tune continuous evolution-related hyperparameters, but we provide a detailed study in Appendix D.2.
[Figure]
Discussion and Conclusion
Limitations.
There are a few limitations for our method.
Firstly, a more sophisticated time interval normalization is required.
In real-world dynamic graphs, there may be some large time intervals between temporal nodes when constructing temporal walks.
Although we propose a simple solution based on logarithmic transformations (see Appendix B.3), there is no theoretical guarantee of stability when solving the continuous evolution process by this normalization trick.
Secondly, the calculation of spatial-biased probabilities is computationally heavy.
While it is practical to limit the number of spanned temporal neighbors to alleviate the computational burden (see Appendix D.4), we resort to finding a more efficient implementation in the future.
Social Impacts.
Continuous-time dynamic graph representation learning benefits a wide range of real-world applications, including but not limited to recommender systems, social network mining, and industrial process modeling.
However, there are also some potential negative impacts.
For example, NeurTWs may learn skewed representations if there are biased patterns in the training data, which may result stereotyped predictions.
Also, powerful dynamic graph models may augment harmful activities (e.g., attacking and phishing) on real-world dynamic systems.
Conclusion.
We propose a novel and competitive method for modeling CTDGs, which has advantages in three aspects.
Firstly, our approach complements existing temporal neighborhood sampling algorithms by simultaneously considering temporal, topological, and tree traversal properties, enabling diverse and expressive dynamic graph motifs to be retrieved.
Secondly, the proposed method allows these motifs with irregularly-sampled temporal nodes to be better embedded, where temporal dependencies are now explicitly modeled.
Thirdly, our model forms a harder contrastive pretext task to enrich supervision signals.
Consequently, the NeurTWs method outperforms existing works by large margins, revealing significant application prospects in many real-world scenarios, which will be an exciting focus in our future work.
Acknowledgments and Disclosure of Funding
Some computing resources for this project are supported by MASSIVE 2 .
We thank Guangsi Shi for additional computational resources and assistance in model training and deployment.
S. Pan was partially supported by an Australian Research Council (ARC)
Future Fellowship (FT210100097).
Y.-F. Li was partially supported by the DARPA CCU program (HR001121S0024).
B Additional Model Details B.1 Tree-Structured Walk Sampling
In practice, we sample walks with a tree-structured sampling algorithm, where temporal, topological, and tree traversal properties are considered simultaneously (Section 4.2).
In Algorithm 3, we configure the sampling with a specific hyperparameter ngh that defines the sampling hops and the number of temporal neighbors to be sampled per hop.
For example, we sample 32 1-hop and 1 2-hop neighbors if ngh =
[32 , 1] , which forms C = 32 walks with the length l = 2 .
In other word, we have | ngh | = l and (cid:81)
| ngh
|−
i
=
ngh
[ i ] = C .
Require:
Root node w 0 , cut time t 0 , dynamic graph G , sampling configuration list ngh , temporal-biased intensity α , spatial-biased intensity β , exploration & exploitation-biased intensity γ 1: Initialize roots =
L
, empty lists
2: for
| do 3:
] 4: for do 5: ( p p root , dictionaries p p }
Pr t w p biased probabilities
7: for
G p t p do 8: d w
=
G t | 9: end for 10:
Pr s = { w : exp ( − β d [ w ])
| w
G w p ,t p } /
Spatial-biased probabilities
/ 11: for j in 1 , 2 , · · · , k do 12:
Pr e = { w : exp ( − γs [ w ])
w
G w p ,t p } /
E-biased probabilities ∗ / 13:
Sample one ( { w, w p } , t )
G w p ,t p prob.
Pr t , Pr s , Pr e 14:
L i
L i
, t ) 15: s w
16: end for 17:
B.2 Autoregressive Gated Recurrent Unit
Different from the gated recurrent unit (GRU) [4] whose hidden states are simultaneously activated by inputs and previous states, we define the autoregressive GRU as follows:
[Equation]
In Equation 12, the hidden states h t only depend on h t − 1 , where z t , r t , and (cid:98) h t are update, reset, and candidate activation states.
Specifically, W z , W r , and W h are trainable weights.
b z , b r , and b h are trainable biases.
σ ( · ) and tanh( · ) denote the sigmoid and hyperbolic tangent activation functions.
B.3 Batching and Complexity Analysis
Walk encoding in batches For C different
length- l anonymous walks with various time intervals, we normally have to solve C × l different ODEs to encode each walk, which is computational expensive.
Now take C walks with l = 1 (i.e., the walks composed of only two temporal nodes) for example, we suppose the timestamps of two (i.e., source and target) nodes are t cstart and t cend for c in 1 , 2 , · · · , C .
Inspired by Chen et al.
[3], we aim to construct a substitute variable s for each continuous evolution process that unify their integral intervals to a specific range, e.g., 0 to 1, so that makes it possible to encode all C walks at once in parallel.
Considering the c -th walk in the above example, we intend to integrate from t cstart to t cend with the initial states h 0 and an ODE function f
( h t , t, θ ) to learn the latent spatiotemporal dynamics.
To integrate from 0 to 1, we transform h t to (cid:101) h s with s =
( t − t cstart ) /
( t cend − t cstart ) .
In other word, we have t =
s ( t cend − t cstart )
+ t cstart and the following equation:
[Equation]
The corresponding ODE function (cid:101) f
( (cid:101) h s , s, θ ) then follows:
[Equation]
On this basis, we have the following equation w.r.t.
the facts that (cid:101) h 0 = h t cstart =
h 0 and (cid:101) h 1 =
h t cend
[Equation]
ODESolving
[Equation] [Equation]
where ODESolving ( · ) denotes a black-box ODE solver, e.g., Euler or Runge-Kutta 3/8 method.
With the above reparameterization trick, we can solve C different continuous evolution processes in parallel, thus allowing our method to be trained more efficiently.
Time interval normalization There may be some large time intervals between two temporal nodes in real-world dynamic graphs, e.g., Figure 6(a), which makes solving the continuous evolution process (i.e., Equation 8) intractable.
A good idea to address this challenge is to scale time intervals without violating the inductive biases that the relative differences between time intervals reflect essential temporal clues.
On this basis, various transformations can be applied, such as logarithmic transformations or divided by a large constant, to scale down the range of time intervals and make solving Equation 8 tractable.
In practice, we empirically find that using the base 10 logarithmic scaling, as shown in Figure 6(c), is sufficient for our method to achieve good performances on dynamic graph modeling.
Compared with other alternatives, e.g., divided by a large constant as shown in Figure 6(b), the logarithmic transformation with a properly selected base can scale down the
[Figure]
[Figure]
Specifically, we let batch size B = 32 and the number of walks C = 64 per root node with the walk length l = 2 .
range of time intervals while well preserving the variances between most small intervals.
Although the variances between very large intervals are compressed in such a case, we conjecture the key is retaining the relative differences between small and large time intervals, which reveals critical temporal patterns in dynamic graph modeling.
Complexity analysis We analyse the time complexity of the main components in NeurTWs , as shown in Table 6. [Table] [Table] O (
[Table]
[Table]
O ( ld [Table] [Table] O ( lFd 2 h )
[Table]
We start from analysing the time complexity of the proposed temporal walk sampling based on
Algorithm 3 with the following assumptions: (1) We have a batch of B interactions; (2) The sampling configuration ngh =
[ k 1 , k 2 , · · · , k l ] s.t.
the number of walks C =
(cid:81)
| ngh | i =1 k i and the walk length l = | ngh | ; (3) The time of neighborhood finding is constant; (4) The average number of spanned temporal neighbors is k s .
On this basis, the time complexity of temporal-biased, spatial-biased, and exploration & exploitation-biased walk sampling are O ( BC ) , O ( BCk s ) , and O ( BC ) , respectively.
Thus, the time complexity of our sampling algorithm is O ( BCk s ) .
For walk anonymization, we analyze the upper bound time complexity based on a simple brute-force implementation.
We first identify unique nodes among C temporal walks with the length l by traversing all nodes in walks.
Then, we traverse again to acquire exact node positional encodings, thus resulting in the complexity of O ( Cl ) for processing an interaction in a batch.
We then analyse the time complexity of the proposed continuous evolution and instantaneous activation processes with the aforementioned and the following additional assumptions: (1) We instantiate g ( · , ϕ ) and f ( · , θ ) with a standard and autoregressive GRU (See Appendix B.2), respectively.
; (2) We assume the input and hidden dimensions are d i and d h , i.e., A ( w i )
∈
R d i × 1 and
A ′ ( w i ) , h ′
i , h i ∈
R d h × 1 ; (3) We assume the number of function evaluations (i.e., how many times the
ODE function is called in a single forward pass when solving an ODE) in the continuous evolution process is F .
On this premise, the time complexity of solving the continuous evolution process over B × C walks is O ( BClFd 2 h ) .
However, the aforementioned batching mechanism allows those different ODEs to be solved in parallel, which reduces the time complexity to O ( lFd 2 h ) .
Regarding the instantaneous activation process, the time complexity is O ( ld 2 h ) when training in batches.
Since our model is trained by optimizing a contrastive objective function, the overall time complexity of our method is O (cid:0) K ( BC ( k s + l ) + lFd 2 h ) (cid:1) , where K denotes the number of negatives.
B.4 Model Training
Algorithm
Require:
Training subgraph G trn , batch size B , sampling configuration ngh , temporal intensity
α , spatial intensity β , exploration & exploitation intensity γ , number of negatives K , learning rate η 1: Initialize epoch
=2: repeat 3:
Split all interactions in G trn into m batches (cid:101) E
= { (cid:101) E i | 1 ≤ i ≤ m, | (cid:101) E i | =
B } 4: for i in 1 , 2 , · · · , m do 5:
Initialize h KV ′
= {} 6: / ∗ TWS ( · ) stands for temporal walk sampling (see Algorithm 3) ∗ / 7:
Temporal walk sampling M U =
{ M u j ← TWS ( u j , t j , G trn , ngh, α, β, γ ) | ( { u j , v j } , t j ) ∈ (cid:101) E i } ,
M V =
{ M v j ← TWS ( v j , t j , G trn , ngh, α, β, γ ) | ( { u j , v j } , t j ) ∈ (cid:101) E i } 8:
Anonymization A ( M U ) =
{ A ( M u j ) | M u j ∈ M U } , A ( M V ) =
{ A ( M v j ) | M v j ∈ M V } 9: / ∗ TWE ( · ) stands for walk encoding (see Algorithm 2).
Pool ( · ) denotes sum-pooling ∗ / 10:
Walk encoding and pooling h U
= { h u j ← Pool (cid:16) TWE (cid:0) A ( M u j ) (cid:1)(cid:17)
| A ( M u j ) ∈
A ( M U ) } , h V
=
{ h v j ← Pool (cid:16) TWE (cid:0) A ( M v j ) (cid:1)(cid:17)
| A ( M v j ) ∈
A ( M V ) } 11: / ∗ Negative sampling and calculate the time-aware node embedding of negatives ∗ / 12: for k in 1 , 2 , · · · , K do 13:
Randomly sample a batch of nodes V ′
= { v ′ j ∈ G trn | 1 ≤ j ≤ B, ( { u j , v ′ j } , t j ) / ∈ G trn } 14:
Temporal walk sampling M V ′
=
{ M v ′
j ← TWS ( v ′ j , t j , G trn , ngh, α, β, γ ) | v ′ j ∈ V ′ } 15:
Anonymization A ( M V ′ )
=
{ A ( M v ′ j ) | M v ′ j ∈ M V ′ } 16:
Walk encoding and pooling h V ′
= { h v ′
j ← Pool (cid:16) TWE (cid:0) A ( M v ′ j ) (cid:1)(cid:17)
| A ( M v ′ j ) ∈
A ( M V ′ ) } 17: h KV ′ =
h KV ′
|| h V ′ 18: end for 19: / ∗ Calculate the contrastive loss w.r.t.
Equation 10 ∗ / 20:
Compute L = loss ( h U , h V , h KV ′ ) and the stochastic gradients of model parameters w.r.t.
L 21:
Update model parameters w.r.t.
their gradients and the learning rate η 22: end for 23: epoch +=24: until convergence
C Experimental Setting C.1 Datasets
We list the introduction of six datasets used in this paper as follows.
We also discuss how they are preprocessed for use in our method.
• CollegeMsg 3 collects the message interactions between the users in an online social media platform at the University of California, Irvine.
Specifically, this dataset consists of 59, message interactions and 1,899 unique users, where these unattributed temporal interactions are in the format of ( u, v, t ) , denoting that user u sent a message to user v at time t .
Since the format of the interactions in this dataset naturally matches the definition of continuous-time dynamic graphs in Section 3, so we do not need to preprocess the dataset intentionally.
We provide the ready-to-use dataset files with our code, as well as a script to transform the raw dataset file to the data files used in our model.
• Enron 4 is an email network dataset consisting of around 0.5 million email interactions between about 150 senior employees in a corporation over several years.
In this paper, we build the Enron dataset based on the preprocessed version in [36] by using the first half of interactions, which composes of 62,617 unattributed email interactions and 143 unique employees.
We provide the preprocessed dataset files with our code.
• Taobao 5 is an user behavior dataset that originally collected for the research of recommendation systems, which consists of 100,150,807 user-item interactions, 987,994 users, and 4,162,024 items.
Notably, each interaction in this dataset has a specific action type, i.e., a user browses/buys/favorites an item or adds it to the shopping cart.
To incorporate this information, we one-hot encode each interaction’s action type as the interaction attributes.
In this paper, we build the Taobao dataset with a subset of the original user-item interaction network, which composes of 77,436 interactions and 64,703 unique nodes.
We provide the preprocessed dataset files with our code, as well as a script to preprocess and transform the raw dataset files to the data files used in our model.
• MOOC 6 is an user action dataset that has a collection of actions taken by the users on the MOOC platform.
This dataset consists of 7,047 users, 97 items, and 411,749 attributed interactions over the duration of about a month.
Specifically, the user-item interactions in this dataset denote the access behavior of students to online course units.
Similar to CollegeMsg, the format of this dataset naturally matches our definition of continuous-time dynamic graphs, so we have not preprocessed it intentionally.
We provide the ready-to-use dataset files with our code.
• Wikipedia 7 is an user action dataset that has a set of interactions between users and wikipedia pages.
This dataset records 157,474 attributed interactions in one month between 8,227 users and 1,000 pages.
Specifically, an interaction denotes that an user edits a page, where the editing texts have converted into LIWC-feature vectors.
Similar to CollegeMsg and MOOC, we have not preprocessed this dataset intentionally.
Also, we provide the ready-to-use dataset files with the code.
• Reddit 8 is an user action dataset that collects one month of posts made by different users on the Reddit platform.
Similar to Wikipedia, an interaction in this dataset denotes that an user makes a post on subreddits, where the texts are embedded into LIWC-feature vectors.
Specifically, this dataset consists of 672,447 interactions, 10,000 most active users andsubreddits.
We have not preprocessed this dataset intentionally, and we also provide the dataset files with the code.
C.2 Baselines
We list the introduction of baseline methods as follows.
• CTDNE [23] extends the static network embedding to dynamic graphs, where temporal random walks have been proposed with the skip-gram model to learn node representations.
• JODIE [15] updates latent states of two nodes in an interaction with two mutually-influenced recurrent neural networks, where the future node embedding trajectories can be estimated.
• DyRep [34] integrates sequence models with an attentive message passer, where 2-hop temporal neighborhood information is leveraged to learn time-aware node embeddings.
• TGAT [38] samples and attentively aggregates the information from k -hop temporal neighbors to learn time-aware node embeddings.
In this method, the time information is preserved by the proposed random Fourier time encodings when conducting the message passing.
• TGN
[27] proposes a general message passing-based framework to learn on CTDGs with a node memory updating mechanism, which inherits and combines the key designs in JODIE and TGAT.
• CAWs [36] learn interaction representations by encoding and aggregating anonymous temporal walks with a recurrent neural network and a pooling module.
Specifically, we detail how baseline models are tuned in link prediction tasks as follows.
• We adapt TGAT 9 to our unified evaluation framework, and mainly tune the following important hyperparameters: (1) For the degree of neighborhood sampling, we search the optimal values in { 8 , 16 , 32 , 64 } ; (2) We tune the model layers in { 1 , 2 , 3 } ; (3) We use the default product attention and tune the number of attention heads in { 2 , 4 , 6 } ; (4) For the dimensions of node and time embeddings, we search the optimal configuration in { 16 , 32 , 64 , 100 } .
Regarding the rest of hyperparameters, we use the default setting in the code of TGAT.
• For JODIE, DyRep, and TGN, we adapt the implementation 10 to our unified evaluation framework.
Specifically, we mainly follow the default setting when executing JODIE on our datasets, where the dimension of node embeddings is searched in { 32 , 64 , 128 , 256 } .
For DyRep, we tune the degree of neighborhood sampling in { 10 , 16 , 32 , 64 } and the number of graph attention layers in { 1 , 2 , 3 } .
We also search the optimal values of these two hyperparameters in TGN, where additional hyperparameters are tuned as well: (1) We tune the number of attention heads in { 2 , 4 , 8 } ; (2) We tune the dimension of node, time, and message embeddings in { 16 , 32 , 64 , 100 } ; (3) We tune the dimension of node memories in { 4 , 16 , 32 , 64 , 172 } .
• Our method shares the same evaluation pipeline with CAWs 11 , for which we mainly tune the following hyperparameters based on their default setting: (1) We tune the walk length in { 1 , 2 , 3 } and the number of walks in { 16 , 32 , 64 , 128 } ; (2) We tune the time decay in { 10 − 7 , 10 − 6 , 10 − 5 , 10 − 4 } ; (3) We tune the model with the unitary and binary anonymization, where the best performances are reported.
We follow their default setting for the rest of the hyperparameters except for the walk pooling, which is set to summation as same as in our method for simplicity and fair comparison.
C.3 Implementation Details
Our code is available at https://github.com/KimMeen/Neural-Temporal-Walks , where we provide detailed instructions for dataset preparation and model training.
We first introduce the general setting of NeurTWs when training on all three datasets.
Specifically, our models are trained with the learning rate 10 − 4 and batch size 32.
We set the maximum number of epochs to 50, but the model training usually converges and is early stopped within the first 20 epochs.
For important hyperparameters controlling the walk sampling and contrasitve learning, we summarize their grid search ranges in Tables 7 and 8.
[Table] [Table]
For the rest of hyperparameters
[Table]
we set them as follows: (1) We use the fixed-step ODE solver (i.e.,
Runge-Kutta 3/8 method) with the step size = 0 .
125 by default without specific tuning.
Adaptive solvers (e.g., Dormand–Prince method) can also be used in our method; (2) The hidden dimensions of the position encoding has not tuned and set to 108 by default; (3) For the walk pooling, we use the summation by default for simplicity.
More details can be found in our code, where we provide the scripts with the default settings to run experiments on three datasets.
All experiments are conducted on a Linux cluster with four computing nodes (for each we have 4 ×
RTX6000 24GB, 1 × Intel Xeon Silver 4214R, and 175 GB system memory) and a Linux server ( 4 ×
GeForce RTX
Ti 11GB, 1 × Intel Core i9-9900X, and 94GB system memory).
D Additional Experimental Results D.1 Performances in Average Precision
See Table 9 for detailed transductive and inductive link prediction results w.r.t.
AP.
D.2 Study on Continuous Evolution Process
We study the effectiveness of the proposed continuous evolution process in modeling temporal dependencies when encoding anonymous temporal walks, and further compare it with other two widely adopted strategies, i.e., exponential decay [22] and random Fourier time encoding [38].
We summarize the results in Table 10.
Specifically, we use GRU as the RNN backbone without considering the time information associated with each temporal node in (cid:99) W (i.e., Standard RNN).
On this basis, we equip GRU with the exponential decay mechanism (i.e., RNN with exponential decay), which is defined as follows based on Equation 9 in the main body of the paper:
[Equation]
(16) where ∆ t =
t i − t i − 1 and τ is a hyperparameter to control the rate of decay.
In our experiments, we tune τ in { 10 − 9 , 10 − 8 , 10 − 7 } .
Apart from this, time encoding is another strategy to incorporate the time information, which is defined below [38]:
[Equation]
where Φ d ( · ) denotes the time encoding with the dimensions of d .
⟨·⟩ and || are inner product and concatenation operation.
We implement the time encoding-based variant (i.e., RNN with time encoding) based on Equation 17.
[Table] [Table]
From the results in Table
[Table]
we have the following observations: (1) All variants and our method improve the performance of standard RNN, demonstrating the importance of leveraging time information when learning on CTDGs; (2) The proposed continuous evolution process significantly surpasses the exponential decay and time encoding approaches, proving its effectiveness in capturing the temporal dependencies; (3) Although time encoding helps the model be aware of critical time information, its effectiveness is limited compared with our method.
[Figure]
Although the continuous evolution process is less tuned in our method as mentioned in Appendix
C.3, we study how the NeurTWs method is affected w.r.t.
the number of function evaluations (i.e., the number of times the ODE function is called in a single forward pass when solving an ODE).
The experimental results are in Figure 7, where we use the Euler method with different step sizes for illustration purposes.
Notably, the Euler method is a fixed-step solver, and all ODEs to be solved are restricted to a specific integral interval (i
[Figure]
e., 0 to 1, see Appendix B.3).
Thus, the number of function evaluations depends only on the step size and is inversely proportional to it.
From the results in Figure
[Figure]
we find that the model performance decreases slightly as the step size increases when solving
ODEs.
This phenomenon makes sense, as a smaller step size equates to more function evaluations, which allow for more precise learning of the underlying spatiotemporal dynamics in dynamic graph motifs
[Figure]
but at the cost of higher complexity.
In other words, our method allows a trade-off between model accuracy and efficiency
[Figure]
D.3 Visualization
[Figure] [Figure]
We follow the method described in [36] to visualize the most discriminative dynamic graph motifs learned by our method with the binary anonymization (i.e., NeurTWs † ) on the CollegeMsg dataset.
We first illustrate how a motif can be represented in a sequence of coordinates, which essentially characterizes its shape in a coordinate system w.r.t.
the extracted temporal walks rooted at two nodes of an interaction.
Specifically, given a sampled interaction ( { u, v } , t ) from the dynamic graph G , we sample two sets of temporal walks M u and M v rooted at node u and v , respectively.
Recall the binary anonymization defined in Section 4.2, A ( w ; M u )
[ i ] and A ( w ; M v )
[ i ] count the number of walks in M u and M v that have node w appearing in position i , where i ∈ { 0 , · · · , l } .
On this basis, we have the shortest path between node w and a root node u (w.r.t.
the subgraph composed of the walks in M u ) defined as d wu := min { i | A ( w ; M u )
[ i ] > 0 } .
We can define d wv in a similar way with
A ( w ; M v ) .
Thus, the coordinate mapping A ( w ; M u , M v ) →
( d wu , d wv ) can be viewed as a way to map the position encoding of node w to its coordinate in a coordinate system defined over M u and
M v .
As a result, we can obtain the shapes of anonymous temporal walks by mapping each node’s position encoding to its relative coordinate.
Specifically, ∞ in a coordinate, e.g., ( ∞ , 0) , denotes that the target node of an interaction does not appear in the temporal walks rooted at the source node of this interaction.
A special case ( ∞ , ∞ ) denotes a nonexistent node, which is used in walk padding to make sure all walks have the same length l when there are no temporal neighbors can be sampled.
We then describe how most discriminative motifs can be identified.
Given a sampled interaction ( { u, v } , t ) from the dynamic graph G and a well-trained model of NeurTWs † with a linear downstream link predictor Ψ ⊺ ( h u || h v ) :
= Ψ ⊺ (cid:80)
Cc =1 enc ( (cid:99) W c ) for (cid:99) W c ∈
A ( M u ) ∪ A ( M v ) , we can obtain the predictive score of each motif (cid:99) W c that contributes to the final prediction of this interaction, i.e., Ψ ⊺ enc ( (cid:99) W c ) .
In our experiments, we visualize the dynamic graph motifs with the top-3 highest and lowest predictive scores in the testing set of the CollegeMsg dataset, as well as their normalized occurrence ratios in all positive and negative samples during the model inference.
From the results in Figure 8, we have the following observations: (1) A general dynamic law can be observed among the shapes of highest-scored motifs; that is, two temporal nodes surrounded by some common dynamic graph motifs are more likely to interact.
For example, the shape of the highest-scored motif ( ∞ , 0) → (2 , 1) → (2 , 2) → (3 , 1) indicates that two temporal nodes share some common 3-hop temporal neighbors because there is no ∞ in the second to fourth positions of the shape of this motif.
On the contrary, two temporal nodes with few overlapped dynamic graph motifs are unlikely to interact, e.g., ( ∞ , 0) → ( ∞ , 1) → ( ∞ , 2) → ( ∞ , ∞ ) .
Thus, our method allows informative dynamic graph motifs and the underlying dynamic laws to be extracted and captured effectively; (2) These three highest-scored motifs commonly appear in positive samples where interactions exist between temporal node pairs but rarely appear in negative samples (see
Figure 8(a)).
In contrast, for those lowest-scored motifs, they appear more frequently in negative samples (see Figure 8(b)).
Therefore, the discriminative power of our method is proven to be strong, which captures essential dynamic laws to characterize temporal nodes where downstream tasks can be effectively conducted.
D.4 Complexity Evaluation
[Figure] [Figure]
We empirically study the efficiency of our method by recording and comparing the runtime of main components in temporal walk sampling and encoding.
We also plot the average runtime of NeurTWs with different number of negatives during the optimization and the relative runtime comparison between it and other baselines under their optimized settings.
The experimental results are in Figure 9, which is consistent with our complexity analysis in Appendix B.3.
We first compare the runtime of the proposed spatiotemporal-biased walk sampling with its two ablation variants on the Enron dataset under the different number of spanned temporal neighbors per node, as shown in Figure 9(a).
Specifically, we use the default model setting on Enron that samples 64 walks with the length 2, but the number of negatives is set to 1 for a more straightforward comparison.
The number of spanned temporal neighbors is defined as |G w p ,t p | in Algorithm 3.
From the results in Figure 9(a), we have three observations: (1) Temporal-biased walk sampling has a constant runtime w.r.t.
the number of spanned temporal neighbors.
This observation can be confirmed by our complexity analysis summarized in Table 6, where the temporal-biased sampling is only in proportion to the batch size and the number of walks to be sampled; (2) The spatiotemporal-biased walk sampling is in proportion to the number of spanned temporal neighbors, which proves our analysis in Appendix B.3 that its time complexity is O ( BCk s ) , where k s is the x-axis of Figure 9(a); (3) Our spatiotemporal-biased walk sampling with the exploration & exploitation trade-off has a similar time complexity compared to its variant without considering this trade-off.
In general, although the calculation of the proposed spatial-biased probabilities induces higher time complexity when sampling temporal walks, it helps extract diverse and expressive dynamic graph motifs, e.g., disabling this mechanism hurts the performance drastically on the Taobao dataset (e.g., ablation 2 in Table 4).
To alleviate the computational burden when modeling dynamic graphs with a large average number of spanned temporal neighbors, we find it is practical to limit k s by selecting a set of (potentially informative) candidates to explore, e.g., we may choose these candidate neighbors randomly or based on prior knowledge.
A concrete example is Enron with a higher k s than other datasets; we empirically find that limiting the number of first- and second-order neighbors with small upper bound (e.g., 64 × 5 and 1 × 8) in a temporal-biased manner does not affect the performances of our method significantly.
We then compare the runtime of the proposed neural temporal walk encoding with one of its ablation variants (i.e., GRU, which essentially forms the instantaneous activation process without the continuous evolution process) and an in-demand design (i.e., GRU with time encoding, which adopts random Fourier time encodings as a part of input features of GRU to preserve the time information when learning walk embeddings).
The experimental results are in Figure 9(b), where we use the setting as mentioned above except for the walk length, which is the x-axis of this plot.
Specifically, we find that the runtime of all methods is in proportion to the walk length, which matches the complexity analysis in Appendix B.3.
Compared to GRU, the proposed neural temporal walk encoding has a higher time complexity which is also in proportion to the number of function evaluations F in the continuous evolution process (see Table 6).
In Figure 9(b), we find that the runtime of our method is about 7.8 times that of the GRU, which is close to our training setting where F = 8 .
Although the proposed continuous evolution process results in a higher runtime of our method in this example compared with the time encoding approach, it demonstrates significantly better capabilities in digesting the time information (see Table 10).
Besides, the proposed continuous evaluation process supports the trade-off between model accuracy and efficiency, allowing our encoding module to have the same time complexity as the time encoding-based GRU (i.e., O ( ld 2 h ) by letting the step size = 1 ) without compromising too much modeling precision (see Figure 7).
In Figure 9(c), we display the runtime of the entire model in a batch w.r.t.
the number of negatives, which proves our justification in Appendix B.3 that the time complexity of our method is in proportion to the number of negatives during the optimization under a certain model configuration (see Table 6).
In Figure 9(d), we compare the relative runtime between our method and other baselines under their optimized settings on Enron.
Although the core components in NeurTWs induce higher time complexity, they significantly improve existing approaches from various aspects in terms of the modeling precision.
It is worth noting that our method explicitly supports trading precision for speed to alleviate the computational burden.
For example, limiting the number of spanned temporal neighbors is practical to reduce runtime (see Figure 9(a)) without losing too much modeling precision, as discussed above.
Also, reducing the number of function evaluation F (e.g., increasing the step size when solving Equation 8 with a fixed-step solver) brings significant speedups, which reduces the complexity of neural temporal walk encoding (see Table 6).
For the optimization, limiting the number of negatives reduces the model runtime as well.