D O W E R EALLY N EED C OMPLICATED M ODEL A RCHITECTURES F OR T EMPORAL N ETWORKS ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract spatial-temporal information for temporal graph learning.
Interestingly, we found that although both RNN and SAM could lead to a good performance, in practice neither of them is always necessary.
In this paper, we propose
GraphMixer, a conceptually and technically simple architecture that consists of three components: 1 a link-encoder that is only based on multi-layer perceptrons (MLP) to summarize the information from temporal links, 2 a node-encoder that is only based on neighbor mean-pooling to summarize node information, andan MLP-based link classifier that performs link prediction based on the outputs of the encoders.
Despite its simplicity, GraphMixer attains an outstanding performance on temporal link prediction benchmarks with faster convergence and better generalization performance.
These results motivate us to rethink the importance of simpler model architecture.
Code is in the supplementary.
I NTRODUCTION
In recent years, temporal graph learning has been recognized as an important machine learning problem and has become the cornerstone behind a wealth of high-impact applications Yu et al.
Bui et al.
; Kazemi et al.
; Zhou et al.
Temporal link prediction is one of the classic downstream tasks which focuses on predicting the future interactions among nodes.
For example, in an ads ranking system, the user-ad clicks can be modeled as a temporal bipartite graph whose nodes represent users and ads, and links are associated with timestamps indicating when users click ads.
Link prediction between them can be used to predict whether a user will click an ad.
Designing graph learning models that can capture node evolutionary patterns and accurately predict future links is a crucial direction for many real-world recommender systems.
In temporal graph learning, recurrent neural network (RNN) and self-attention mechanism (SAM) have become the de facto standard for temporal graph learning Kumar et al.
; Sankar et al.
Xu et al.
; Rossi et al.
; Wang et al.
(2020), and the majority of the existing works focus on designing neural architectures with one of them and additional components to learn representations from raw data.
Although powerful, these methods are conceptually and technically complicated with advanced model architectures.
It is non-trivial to understand which parts of the model design truly contribute to its success, and whether these components are indispensable.
Thus, in this paper, we aim at answering the following two questions:
Q1:
Are RNN and SAM always indispensable for temporal graph learning?
To answer this question, we propose GraphMixer, a simple architecture based entirely on the multi-layer perceptrons (MLPs) and neighbor mean-pooling, which does not utilize any RNN or SAM in its model architecture (Section 3).
Despite its simplicity, GraphMixer could obtain outstanding results when comparing it against baselines that are equipped with the RNNs and SAM.
In practice, it achieves state-of-the-art performance in terms of different evaluation metrics (e.g., average precision, AUC, Recall@K, and
MRR) on real-world temporal graph datasets, with the even smaller number of model parameters and hyper-parameters, and a conceptually simpler input structure and model architecture (Section 4).
Q2:
What are the key factors that lead to the success of GraphMixer?
We identify three key factors that contribute to the success of GraphMixer:
1 The simplicity of GraphMixer’s input data and neural architecture.
Different from most deep learning methods that focus on designing conceptually complicated data preparation techniques and technically complicated neural architectures, we choose to simplifying the neural architecture and utilize a conceptually simpler data as input.
Both of which could lead to a better model performance and better generalization (Section 4.4).
2 A time-encoding function that encodes any timestamp as an easily distinguishable input vector for GraphMixer .
Different from most of the existing methods that propose to learn the time-encoding function from the raw input data, our time-encoding function utilizes conceptually simple features and is fixed during training.
Interestingly, we show that our fixed time-encoding function is more preferred than the trainable version (used by most previous studies), and could lead to a smoother optimization landscape, a faster convergence speed, and a better generalization (Section 4.2); 3 A link-encoder that could better distinguish temporal sequences.
Different from most existing methods that summarize sequences using SAM, our encoder module is entirely based on MLPs.
Interestingly, our encoder can distinguish temporal sequences that cannot be distinguished by SAM, and it could generalize better due to its simpler neural architecture and lower model complexity (Section 4.3).
To this end, we summarize our contributions as follows:
1 We propose a conceptually and technically simple architecture GraphMixer; 2 Even without RNN and SAM, GraphMixer not only outperforms all baselines but also enjoys a faster convergence and better generalization ability; 3 Extensive study identifies three factors that contribute to the success of GraphMixer.
Our results could motivate future research to rethink the importance of the conceptually and technically simpler method.
P RELIMINARY AND EXISTING WORKS
Preliminary.
Figure 1 is an illustration on the temporal graph.
Our goal is to predict whether two nodes are connected at a specific timestamp t
based on all the available temporal graph information happened before that timestamp.
For example, to predict whether v 1 , v 2 are connected at t 0 , we only have access to the graph structure, node features, and link features with timestamps from t 1 to t 6 .
T i m e ga p ttt
[Table]
(now) (past)
[Table]
[Equation]
<
[Table]
=
[Table]
=
[Table]
+
[Table]
+kA4lrJlmSF0YiyAEU6TTDuSNMRkgHu0paiPPSo78fTaxDhRStdwA6GeD8ZU
[Table]
+
[Table]
+
[Table]
[Figure]
Related works.
Most of the temporal graph learning methods are conceptually and technically complicated with advanced neural architectures.
It is non-trivial to fully understand the algorithm details without looking into their implementations.
Therefore, we select the four most representative and most closely-related methods to introduce and compare them in more details.
• JODIE Kumar et al.
(2019) is a RNN-based method .
Let us denote x
i
t ) as the embedding of node v i at time t , x link ij ( t ) as the link feature between v i , v j at time t , and m i as the timestamp that v
i latest interact with other node.
JODIE pre-processes and updates the representation of each node via RNNs (is it just one RNN or multiple RNNs).
More specifically, when an interaction between v i , v j happens at time t , JODIE updates the temporal embedding using RNN by x i
[Equation]
=
[Equation]
Then, the dynamic embedding of node v i at time t 0 is computed by h i ( t 0 ) =
(1 + ( t 0 − m i ) w ) · x i ( m i ) .
Finally, the prediction on any node pair at time t 0 is computed by MLP ([ h i ( t 0 )
||
h j ( t 0 )]) , where [ ·||· ] is the concatenate operation and MLP ( x ) is applying 2-layer MLP on x . • DySAT Sankar et al.
(2020) is a SAM-based method .
DySAT requires pre-processing the temporal graph into multiple snapshot graphs by first splitting all timestamps into multiple time-slots, then merging all edges in each time-slot.
Let G t ( V , E t ) denote the t -th snapshot graph.
To capture spatial information, DySAT first applies Graph Attention Network (GAT) Veliˇckovi´c et al.
(2018) on each snapshot graph G t independently by X ( t ) =
GAT ( G t ) .
Then, to capture of temporal information for each node, Transformer is applied to x i ( t ) =
[ X ( t )] i at different timestamps to capture the temporal
information by h i ( t k ) , . . .
h i ( t 0 ) =
Transformer (cid:0) x i ( t k ) , . .
, x i ( t 0 ) (cid:1) .
Finally, the prediction on any node pair at time t 0 is computed by MLP ([ h i ( t 0 )
||
h j ( t 0 )]) .
• TGAT Xu et al.
(2020) is a SAM-based method that could capture the spatial and temporal information simultaneously.
TGAT first generates the time augmented feature of node i at time t by concatenating the raw feature x i with a trainable time encoding z ( t ) of time t , i.e., x i ( t ) =
[ x i ||
z ( t )] and z ( t ) =
cos( t w + b ) .
Then, SAM is applied to the time augmented features and produces node representation
h i ( t 0 )
=
SAM ( x i ( t 0 ) , { x u ( h u ) | u ∈ N t
( i ) } ) , where N t 0 ( i ) denotes the neighbors of node i at time t 0 and h u denotes the timestamp of the latest interaction of node u .
Finally, the prediction on any node pair at time t 0 is computed by MLP ([ h i ( t 0 )
||
h j ( t 0 )]) .
• TGN Rossi et al.
(2020) is a mixture of RNN- and SAM-based method .
In practice, TGN first captures the temporal information using RNN (similarly to JODIE), and then applies graph attention convolution to capture the spatial and temporal information jointly (similarly to TGAT).
Besides, we also consider the following temporal graph learning methods as baselines.
These methods could be thought of as an extension on top of the above four most representative methods, but with the underlying idea behind the model design much more conceptually complicated.
CAWs Wang et al.
(2020) is a mixer of RNN- and SAM- based method that proposes to represent network dynamics by extracting temporal network motifs using temporal random walks.
CAWs replaces node identities with the hitting counts of the nodes based on a set of sampled walks to establish the correlation between motifs.
Then, the extracted motifs are fed into RNNs to encode each walk as a representation, and use SAM to aggregate the representations of multi-walks into a single vector for downstream tasks.
TGSRec Fan et al.
(2021) is a SAM-based method that proposes to unify sequential patterns and temporal collaborative signals to improve the quality of recommendation.
To achieve this goal, they propose to advance the SAM by adopting novel collaborative attention, such that SAM can simultaneously capture collaborative signals from both users and items, as well as consider temporal dynamics inside sequential patterns.
APAN Wang et al.
(2021b) is a RNN-based method that proposes to decouple model inference and graph computation to alleviate the damage of the heavy graph query operation to the speed of model inference.
More related works are deferred to Appendix B.
G RAPH M IXER : A CONCEPTUALLY AND TECHNICALLY SIMPLE METHOD
In this section, we first introduce the neural architecture of GraphMixer in Section 3.1 then explicitly highlight its difference to baseline methods in Section 3.2.
3. D ETAILS ON G RAPH M IXER : NEURAL ARCHITECTURE AND INPUT DATA
GraphMixer has three modules: 1 link-encoder is designed to summarize the information from temporal links (e.g., link timestamps and link features); 2 node-encoder is designed to summarize the information from nodes (e.g., node features and node identity); 3 link classifier predicts whether a link exists based on the output of the aforementioned two encoders.
Link-encoder.
The link-encoder is designed to summarize the temporal link information associated with each node sorted by timestamps, where temporal link information is referring to the timestamp and features of each link.
For example in Figure 1, the temporal link information for node v 2 is { ( t 1 , x link 1 , 2 ( t 1 )) , ( t 3 , x link 2 , 4 ( t 3 )) , ( t 4 , x link 2 , 4 ( t 4 )) , ( t 5 , x link 1 , 2 ( t 5 )) } and for node v 5 is { ( t 2 , x link 3 ,
5 ( t 2 )) , ( t 6 , x link 4 , 5 ( t 6 )) } .
In practice, we only keep the top K most recent temporal link infor- mation, where K is a dataset dependent hyper-parameter.
If multiple links have the same timestamps, we simply keep them the same order as the input raw data.
To summarize temporal link information, our link-encoder should have the ability to distinguish different timestamps (achieved by our time-encoding function) and different temporal link information (achieved by the Mixer module).
• Time-encoding function.
To distinguish different timestamps, we introduce our time-encoding function cos( t ω ) , which utilizes features ω
= { α − ( i − 1) /β } di =1 to encode each timestamps into a d -dimensional vector.
More specifically, we first map each t to a vector with monotonically exponentially decreasing values t ω ∈ (0 , t ] among the feature dimension, then use cosine function to project all values to cos( t ω ) ∈ [ − 1 , +1] .
The selection of α, β is depending on the scale of the maximum timestamp t max we wish to encode.
In order to distinguish all timestamps, we have to make sure t max × α − ( i − 1) /β
→ 0 as i → d to distinguish all timestamps.
In practice, we found d = 100 and α =
β = √ d works well for all datasets.
Notice that ω is fixed and will not be updated
t i ( k )
[Table]
x identity ( i ) , where [ x identity ( i )]
j =  if j = i 1 / N ( i ) if j ∈N ( i )otherwise
[Table]
+RnX0NTqPD+Oz+CL+dnk0jtbMU/ZPxN9/AwKf4NU=</latexit> t 1 ! cos( t 1 ! )
t 2 !
cos( t 2 ! )
t 3 !
cos( t 3 ! )
t 4 !
cos( t 4 ! ) <latexit sha1_base64="r35mnhe9SYilnfI/p+c5xK7HpxY=">AAAC0nicfZJNbxMxEIa9y1cJH03hyMUiApVLtJtGwLGCCzeK1LSV4iia9U42Vr32yp4FpatQVVz5ddz4CfwLnG2EoA0dyfKrd/zIM2NnlVaekuRnFN+6fefuva37nQcPHz3e7u48OfK2dhJH0mrrTjLwqJXBESnSeFI5hDLTeJydvl/ljz+j88qaQ1pUOCmhMGqmJFCwpt1fIsNCmQa0Kgzmy85LTtOUC6eKOYFz9gsX0vrd1syszv2iDFsjbIkFLF9xIVpksAkZ3IjsbUL2bkSGm5DhfxGBJv/T2bTbS/pJG/y6SNeix9ZxMO3+ELmVdYmGpAbvx2lS0aQBR0pqXHZE7bECeQoFjoM0UKKfNO2TLPmL4OR8Zl1Yhnjr/k00UPpVxeFkCTT3V3Mrc1NuXNPs7aRRpqoJjby8aFZrTpav3pfnyqEkvQgCpFOhVi7n4EBS+AWdMIT0asvXxdGgn77uDz8Ne/vv1uPYYs/Yc7bLUvaG7bMP7ICNmIw+RnX0NTqPD+Oz+CL+dnk0jtbMU/ZPxN9/AwKf4NU=</latexit> t 1 !
cos( t 1 ! )
t 2 !
cos( t 2 ! )
t 3 !
cos( t 3 ! )
t 4 !
cos( t 4 ! ) <latexit sha1_base64="wDNeVYaoNqi9UkgrM1troqRKxM
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Figure]
<latexit sha1_base64="+ujstVMefRZdmv+UsCS0LrqJoAo=">AAAB+nicbVBNT8JAFNziF+JX0aOXjcQEL6QlRD0SvXjERMAEmma7bGHDdtvsvmpI5ad48aAxXv0l3vw3LtCDgpNsMpl5L292gkRwDY7zbRXW1jc2t4rbpZ3dvf0Du3zY0XGqKGvTWMTqPiCaCS5ZGzgIdp8oRqJAsG4wvp753QemNI/lHUwS5kVkKHnIKQEj+Xa5HxEYBWEGU79eBd858+2KU3PmwKvEzUkF5Wj59ld/ENM0YhKoIFr3XCcBLyMKOBVsWuqnmiWEjsmQ9QyVJGLay+bRp/jUKAMcxso8CXiu/t7ISKT1JArM5CyoXvZm4n9eL4Xw0su4TFJgki4OhanAEONZD3jAFaMgJoYQqrjJiumIKELBtFUyJbjLX14lnXrNPa81bhuV5lVeRxEdoxNURS66QE10g1qojSh6RM/oFb1ZT9aL9W59LEYLVr5zhP7A+vwBd/2TfQ
[Equation]
</latexit
[Equation]
[Figure]
As shown in Figure 2a, the output of this time encoding function has two main properties that could help GraphMixer distinguish different timestamps: similar timestamps have similar g
, the the larger the timestamp the later the in
• Mixer for information summarizing.
We use a 1-layer MLP-mixer Tolstikhin et al.
(2021) to summarize the temporal link information.
Figure 2b is an example on summarizing the temporal link information of node v 2 .
Recall that the temporal link information of node v 2 is { (
t
x
link
( t 1 )) t 3 ,
x
link 2 , 4 ( t 3 ))
( t 4 ,
x
link 2 , t 4 )) , t 5 , x link 1 , 2 ( t 5 )) } .
We first encode timestamps by our time-encoding function then concatenate it with its corresponding link features.
For example, we encode ( t 1 , x link 1 , 2 ( t 1 )) as [cos(( t 0 − t 1 ) ω ) ||
x link 1 , 2 ( t 1 ))] where t 0 is the timestamp that we want to predict whether the link exists.
Then, we stack all the outputs into a big matrix and zero-pad to the fixed length K denoted as T 2 ( t 0 ) .
Finally, we use an 1-layer MLP-mixer with mean-pooling to compress T 2 ( t 0 ) into a single vector t 2 ( t 0 ) .
Specifically, the MLP-mixer takes T 2 ( t 0 ) as input
[Equation]
and output the temporal encoding t 2 ( t 0 )
=
Mean ( H channel ) .
Please notice that zero-padding operator is important to capture how often a node interacts with other nodes.
The node with more zero-padded dimensions has less temporal linked neighbors.
This information is very important in practice according to our experimental observation.
Node-encoder.
The node-encoder is designed to capture the node identity and node feature information via neighbor mean-pooling.
Let us define the 1-hop neighbor of node v i with link timestamps from t to t 0 as N ( v i ; t, t 0 ) .
For example in Figure 1, we have N ( v 2 ; t 4 , t 0 )
= { v 1 , v 4 } and
N ( v 5 ; t 4 , t 0 ) = { v 3 } .
Then, the node-info feature is computed based on the 1-hop neighbor by s i ( t 0 ) =
x node i +
[Equation]
| v j ∈ N ( v i ; t 0 − T, t 0 ) } , where T is a dataset-dependent hyper- parameter.
In practice, we found 1-hop neighbors are enough to achieve good performance, and we use one-hot node representations for datasets without node features.
Link classifier.
Link classifier is designed to classify whether a link exists at time t 0 using the output of link-encoder t i ( t 0 ) and the output of node-encoder
s i ( t 0 ) .
Let us denote the node v i ’s representation at time t 0 as the concatenation of the above two encodings
h i ( t 0 ) =
[ s i ( t 0 )
||
t i ( t 0 )] .
Then, the prediction on whether an interaction between node v i , v j happens at time t 0 is computed by applying a 2-layer MLP model on [ h i ( t 0 )
||
h j ( t 0 )] , i.e., p ij =
MLP ([ h i ( t 0 )
||
h j ( t 0 )]) .
3. C OMPARISON TO EXISTING METHODS
In the following, we highlight some differences between GraphMixer and other methods, which will be explicitly ablation studied in the experiment section (Section 4.4).
Temporal graph as undirected graph.
Most of the existing works consider temporal graphs as directed graphs with information only flows from the source node (e.g., users in the recommender system) to the destination nodes (e.g., ads in the recommender system).
However, we consider the temporal graph as an undirected graph.
By doing so, if two nodes are frequently connected in the last few timestamps, the “most recent 1-hop neighbors” sampled for the two nodes on the “undirected” temporal graph would be similar.
In other words, the similarity between the sampled neighbors provides information on whether two nodes are frequently connected in the last few timestamps, which is essential for temporal graph link prediction.
Intuitively, if two nodes are frequently connected in the last few timestamps, they are also likely to be connected in the recent future.
[Table]
Selection on neighbors.
Existing methods consider either “multi-hop recent neighbors” or “multihop uniform sampled neighbors”, whereas we only consider the “ 1 -hop most recent neighbors”.
For example, TGAT Xu et al.
(2020), DySAT Sankar et al.
(2020), and TGSRe Fan et al.
(2021) consider multi-hop uniform sampled neighbors; JODIE Kumar et al.
, TGN Rossi et al.
(2020), and
APAN Wang et al.
(2021b) maintain the historical node interactions via RNN, which can be think of as multi-hop recent neighbors; CAWs Wang et al.
samples neighbors by random walks, which can also be think of as multi-hop recent neighbors.
Although sampling more neighbors could provide a sufficient amount of information for models to reason about, it could also carry much spurious or noisy information.
As a result, more complicated model architectures (e.g., RNN or SAM) are required to extract useful information from the raw data, which could lead to a poor model trainability and potentially weaker generalization ability.
Instead, we only take the “most recent 1-hop neighbors” into consideration, which is conceptually simpler and enjoys better performance.
E XPERIMENTS
Dataset.
We conduct experiments on five real-world datasets, including the Reddit , Wiki , MOOC ,
LastFM datasets that are used in Kumar et al.
and the GDELT dataset 1 which is introduced in Zhou et al.
Besides, since GDELT is the only dataset with both node and link features, we create its two variants to understand the effect of training data on model performance: GDELT-e removes the link feature from GDELT and keep the node feature and link timestamps, GDELT-ne removes both the link and edge features from GDELT and only keep the link timestamps.
For each dataset, we use the same 70% / 15% / 15% chronological splits for the train/validation/test sets as existing works.
The detailed dataset statistics are summarized in Appendix A.2.
Baselines.
We compare baselines that are introduced in Section 2.
Besides, we create two variants to better understand how node- and link-information contribute to our results, where GraphMixer-L is only using link-encoder and GraphMixer-N is only using node-encoder.
We conduct experiments under the transductive learning setting and use average precision for evaluation.
The detailed model configuration, training and evaluation process are summarized in Appendix A.3.
Due to the space limit, more experiment results on using Recall@K, MRR, and AUC as the evaluation metrics, comparison on wall-clock time and number of parameters are deferred to Appendix C
Outline.
We first compare GraphMixer with baselines in Section 4.1 then highlight the three key factors that contribute to the success of GraphMixer in Section 4.2, Section 4.3, and Section 4.4.
4. M AIN EMPIRICAL RESULTS .
GraphMixer achieves outstanding performance.
We compare the average precision score with baselines in Table 1.
We have the following observations: 1 GraphMixer outperforms all baselines on all datasets.
The experiment results provide sufficient support on our argument that neither RNN nor SAM is necessary for temporal graph link prediction.
According to the performance of
GraphMixer-L on datasets only have link timestamp information ( MOOC , LastFM , and GDELT-ne ), we know that our time-encoding function could successfully pre-process each timestamp into a meaningful vector.
In fact, we will show later in Section 4.2 that our time-encoding function is more preferred than baselines’ trainable version.
By comparing the performance GraphMixer-N and GraphMixer on Wiki , MOOC , and LastFM datasets, we know that node-encoder alone is not enough to achieve a good performance.
However, it provides useful information that could benefit the link-encoder.
By comparing the performance of GraphMixer-N on GDELT and GDELT-ne , we observe that using one-hot encoding outperforms using node features.
This also shows the importance of node identity information because one-hot encoding only captures such information.
5 More complicated methods (e.g., CAWs, TGSRec, and DDGCL) do not perform well when using the default hyper-parameters 2 , which is understandable because these methods have more components with an excessive amount of hyper-parameters to tune.
GraphMixer enjoys better convergence and generalization ability.
To better understand the model performance, we take a closer look at the dynamic of training accuracy and the generalization gap (the absolute difference between training and evaluation score).
The results are reported in Figureand Figure {+8: 
+}
The slope of training curves reflects the expressive power and convergence speed of an algorithm.
From the first row figures, we can observe that GraphMixer always converge to a high average precision score in just a few epochs, and the training curve is very smooth when compared to baselines.
Interestingly, we can observe that the baseline methods cannot always fit the training data, and their training curves fluctuate a lot throughout the training process. Please also refer to
Appendix C.3 for wall-clock time.
The generalization gap reflects how well the model could generalize and how stable the model could perform on unseen data (the smaller the better).
From the second row figures, the generalization gap curve of GraphMixer is lesser and smoother than baselines, which indicates the generalization power of GraphMixer.
[Figure]
GraphMixer enjoys a smoother loss landscape.
To understand why “GraphMixer converges faster and generalizes better, while baselines suffer training unstable issue and generalize poorly” , we explore the loss landscape by using the visualization tools introduced in Li et al. (2018a).
We illustrate the loss landscape in Figure 4 by calculating and visualizing the loss surface along two random directions near the pre-trained optimal parameters.
The x- and y-axis indicate how much the optimal solution is stretched along the two random directions, and the optimal point is when x- and y-axis are zero.
1 From Figure 4a, 4d, we know GraphMixer enjoys a smoother landscape with a flatter surface at the optimal point, the slope becomes steeper when stretching along the two random directions.
The steeper slope on the periphery explains why GraphMixer could converge fast, the flatter surface at the optimal point explains why it could generalize well.
Surprisingly, we find that baselines have a non-smooth landscape with many spikes on its surface from Figure 4b, 4c, 4e, 4f.
This observation provides sufficient explanation on the training instability and poor generalization issue of baselines as shown in Figure 3, 8.
Interestingly, as we will show later in Section 4.2, the trainable time-encoding function in baselines is the key to this non-smooth landscape issue.
Replacing it with our fixed time-encoding function could flatten the landscape and boost their model performance.
4. O N THE IMPORTANCE OF OUR FIXED TIME - ENCODING FUNCTION .
Existing works (i.e., JODIE, TGAT, and TGN) leverage a trainable time-encoding function z ( t ) =
cos( t w ⊤ + b ) to represent timestamps 3 .
However, we argue that using trainable time-encoding function could cause instability during training because its gradient ∂ cos( t w + b ) ∂ w = t × sin( t w + b ) scales proportional to the timestamps, which could lead to training instability issue and cause the baselines’ the non-smooth landscape issue as shown in Figure 4.
As an alternative, we utilize the fixed time-encoding function z ( t ) =
cos( t ω ) with fixed features ω that could capture the relative difference between two timestamps (introduced in Section 3.1).
To verify this, we introduce a simple experiment to test whether the time-encoding functions (both our fixed version and baselines’ trainable version) are expressive enough, such that a simple linear classifier can distinguish the time-encodings of two different timestamps produced by the time-encoding functions.
Specially, our goal is to classify if t 1 > t 2 by learning a linear classifier on [ z ( t 1 )
||
z ( t 2 )] .
During training, we randomly generate two timestamps t 1 , t 2 ∈ [0 , 10 6 ] and ask a fully connected layer to classify whether a timestamp is greater than another.
As shown in Figure 5a, using the trainable time-encoding function (orange curve) will suffer from the unstable exploding gradient issue (left upper figure) and its performance remains almost the same during the training process (left lower figure).
However, using our fixed time-encoding function (blue curve) does not have the unstable exploding gradient issue and can quickly achieve high accuracy within several iterations.
Meanwhile, we compare the parameter trajectories of the two models in Figure 5b.
We observe that the change of parameters on the trainable time-encoding function is drastically larger than our fixed version.
A huge change in weight parameters could deteriorate the model’s performance.
Most importantly, by replacing baselines’ trainable time-encoding function with our fixed version, most baselines have a smoother optimization landscape (Figure 6) and a better model performance (in Table 2), which further verifies our argument that our fixed time-encoding function is more preferred than the trainable version.
[Figure] [Figure]
[Table]
4. O N THE IMPORTANCE OF MLP- MIXER IN G RAPH M IXER ’ S LINK - ENCODER
In this section, we aim to achieve a deeper understanding on the expressive power of the link-encoder by answering the following two questions: “ Can we replace the MLP-mixer in link-encoder with selfattention ?”
and “ Why MLP-mixer is a good alternative of self-attention? ”
To answer these questions, let us first conduct experiments by replacing the MLP-mixer in link-encoder with full/1-hop selfattention and sum/mean-pooling, where full self-attention is widely used in Transformers and 1-hop self-attention is widely used in graph attention networks.
As shown in Table 3, GraphMixer suffers from performance degradation when using self-attention: the best performance is achieved when using MLP-mixer with zero-padding, while the model performance drop slightly when using selfattention with sum-pooling (row 2 and 4), and the performance drop significantly when using self-attention with mean-pooling (row 3 and 5).
Self-attention with mean-pooling has a weaker model performance because it cannot distinguish “temporal sequences with identical link timestamps and features” (e.g., cannot distinguish [ a 1 , a 1 ] and [ a 1 ] 4 ) and it cannot explicitly capture “the length of temporal sequences” (e.g., cannot distinguish if [ a 1 , a 2 ] is longer than [ a 3 ] ), which are both very important for GraphMixer understand how frequent a node interacts with other nodes.
We explicitly verify this in Figure 7 by first generating two temporal sequences (with timestamps but without link features), then encoding the timestamps into vectors via time-encoding function, and asking full self-attention and MLP-mixer to distinguish.
As shown in Figure 7, self-attention with mean-pooling cannot distinguish two temporal sequences with identical timestamps (because all the self-attention weights are equivalent if the features of the node on the two sides of a link are identical) and cannot capture the sequence length (because of mean-pooling simply averages the inputs and does not take the input size into consideration).
However, MLP-mixer in GraphMixer can distinguish the above two sequences because of zero-padding.
Fortunately, the aforementioned two weaknesses could be alleviated by replacing the mean-pooling in temporal self-attention with the sum-pooling, which explains why using sum-pooling brings better model performance than mean-pooling.
However, since self-attention modules have more parameters and are harder to train, they could generalize poor when the downstream task is not too complicated.
[Table]
4. K EY FACTORS TO THE BETTER PERFORMANCE
One of the major factors that contributes to GraphMixer’s success is the simplicity of GraphMixer’s neural architecture and input data.
Using conceptually simple input data that better aligned with their labels allows a simple neural network model to capture the underlying mapping between the input to
[Figure]
their labels, which could lead to a better generalization ability.
In the following, we explicitly verify this by comparing the performance of GraphMixer with different input data in Table 4: 1 Recall from Section 3.2 that the “most recent 1-hop neighbors” sampled for the two nodes on the “undirected” temporal graph could provide information on whether two nodes are frequently connected in the last few timestamps, which is essential for temporal graph link prediction.
To verify this, we conduct ablation study by comparing the model performance on direct and undirected temporal graphs.
As shown in the 1st and 2nd row of Table 4, changing from undirected to direct graph results in a significant performance drop because such information is missing.
5 2 Recall from Section 3.1 that instead of feeding the raw timestamp to GraphMixer and encoding each timestamp with a trainable time-encoding function, GraphMixer encodes the timestamps via our fixed time-encoding function and feed the encoded representation to GraphMixer, which reduces the model complexity of learning a time-encoding function from data.
This could be verified by the 3rd and 4th rows of Table 4, where using the pre-encoded time information could give us a better performance.
3 Selecting the input data that has similar distribution in training and evaluation set could also potentially improve the evaluation error.
For example, using relative timestamps (i.e., each neighbor’s timestamp is subtracted by its root node’s timestamp) is better than absolute timestamps (e.g., using Unix timestamp)because the absolute timestamps in the evaluation set and training set are from different range when using chronological splits, but they are very likely to overlap if using relative timestamps.
As shown in the 3rd to 6th rows of Table 4, using relative time information always gives a better model performance than using absolute time information.
Selecting the most representative neighbors for each node.
For example, we found 1-hop most recent interacted neighbors are the most representative for link prediction.
Switching to either 2-hop neighbors or uniform sampled neighbors will hurt the model performance according to the 7th to the 10th row of Table
[Table] [Table]
C ONCLUSION
In this paper, we propose a conceptually and technically simple architecture GraphMixer for temporal link prediction.
GraphMixer not only outperforms all baselines but also enjoys a faster convergence speed and better generalization ability.
An extensive study identifies three key factors that contribute to the success of GraphMixer and highlights the importance of simpler neural architecture and input data structure.
An interesting future direction, not limited to temporal graph learning, is designing algorithms that could automatically select the best input data and data pre-processing strategies for different downstream tasks.
C ONTENTS 1 Introduction
2 Preliminary and existing works 3 GraphMixer:
A conceptually and technically simple method 3.
Details on GraphMixer: neural architecture and input data . . . . . . . . . . . . .
Comparison to existing methods . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 Experiments 4.
Main empirical results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
On the importance of our fixed time-encoding function. . . . . . . . . . . . . . . .
On the importance of MLP-mixer in GraphMixer’s link-encoder . . . . . . . . . .
Key factors to the better performance . . . . . . . . . . . . . . . . . . . . . . . . .
5 Conclusion
A Experiment setup details
A.1 Hardware specification and environment . . . . . . . . . . . . . . . . . . . . . . .
A.2 Details on dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Model configurations, training and evaluation process . .
B More discussion on existing temporal graph methods
B.1 Recent methods that we do not compare with . . .
B.2 Why existing methods are conceptually and technically complicated? . . . . . . .
C More experiment results
C.1 Comparison on convergence speed and generalization
C.2 Transductive learning with Recall@K and MRR as evaluation
metric . . . . . . . .
D Discussion about model performance on LastFM
E Missing figures on Loss landscape
E.1 Loss landscape on Wiki
E
.2 Loss landscape on Reddit
E.3 Loss landscape on MOOC
E.4 Loss landscape on LastFM
E.5 Loss landscape on GDELT
ne
[Table]
[Table]
[Table]
A E XPERIMENT SETUP DETAILS A. H ARDWARE SPECIFICATION AND ENVIRONMENT
We run our experiments on a single machine with Intel i 9 - 10850 K, Nvidia RTX 3090 GPU, and 64GB RAM memory.
The code is written in Python 3 . 8 and we use PyTorch 1 . 12 .
1 on CUDA 11 .to train the model on the GPU.
Implementation details could be found at [Repository].
A. D ETAILS ON DATASET
The dataset used in this paper could be automatically downloaded by this script.
Reddit datasetconsists of one month of posts made by users on subreddits.
The link feature is extracted by converting the text of each post into a feature vector.
Wikipedia dataset 7 consists of one month of edits made by edits on Wikipedia pages.
The link feature is extracted by converting the edit test into an LIWCfeature vector.
LastFM dataset 8 : consists of one month of who listens-to-which song information.
MOOC dataset 9 consists of actions done by students on a MOOC online course.
GDELT datasetis a temporal knowledge graph dataset originated from the Event Database which records events happening in the world from news and articles.
[Table]
A. M ODEL CONFIGURATIONS , TRAINING AND EVALUATION PROCESS
Baseline implementations.
The implementation on JODIE, DySAT, TGAT, TGN, and APPN follows the temporal graph learning framework Zhou et al.
Compared to the original baselines’ implementation, this framework’s implementation could achieve a better overall score than its original implementation.
The implementation of CAWs-mean and CAWs-attn follows their official implementation 12 , we choose the number of random walk steps from 8 , 16 , 32 to balance the training time.
The implementation of TGSRec follows their official implementation 13 .
The implementation of
DDGCL follows their official implementation 14 .
We directly test using their official implementation by changing our data structure to their required structure and using their default hyper-parameters.
The TGL framework can be found at https://github.com/amazon-research/
GraphMixer implementation.
We implement GraphMixer under the TGL framework Zhou et al.
and use their default hyper-parameters (e.g., learning rate 0 . 0001 , weight decay 10 − 6 , batch size 600 , hidden dimension 100 , etc) to achieve a fair comparison.
In GraphMixer, there are only two hyper-parameters as introduced in Section 3: The number of 1-hop most recent neighbors K and the time-slot size T .
In practice, hyper-parameter T is set the time-gap of the last 2 , 000 interactions, which is fixed for all datasets; hyper-parameter K = 10 for Reddit and LastFM, K = 20 for MOOC, and K = 30 for GDELT and Wiki.
Training and evaluation.
A unified training and evaluation process (e.g., mini-batch and data preparation) is used for GraphMixer and baselines.
Specifically, each mini-batch is constructed by first sampling a set of positive node pairs and an equal amount of negative node pairs.
Then, an algorithm-dependent node sampler is used to sample the neighboring of each mini-batch node and computed their node representation based on the sampled neighborhood.
Finally, we concatenate each node pair and use the link prediction classifier (introduced in Section 3.1) for binary classification.
We conduct experiments under the transduction learning setting and use average precision for evaluation.
B M ORE DISCUSSION ON EXISTING TEMPORAL GRAPH METHODS B. R ECENT METHODS THAT WE DO NOT COMPARE WITH
There are other temporal graph learning algorithms that are related to the temporal link prediction task but we did not compare GraphMixer with them because (1) the official implementation of some of the above works are not released by the authors and we could not reproduce their results as reported in the paper, and (2) we already compare many recent baselines that we believe it is enough to verify the success of GraphMixer.
For example, MeTA Wang et al.
(2021c) proposes data augmentation to overcome the over-fitting issue in temporal graph learning.
More specifically, they generate a few graphs with different data augmentation magnitudes and perform the message passing between these graphs to provide adaptively augmented inputs for every prediction.
TCL Wang et al.
(2021a) proposes to use a transformer to separately extract the temporal neighborhoods representations associated with the two interaction nodes and then utilizes a co-attentional transformer to model inter-dependencies at a semantic level.
To boost model performance, contrastive learning is used to maximize mutual information between the predictive representations of two future interaction nodes.
TNS Wang et al.
(2021d) proposes a temporal-aware neighbor sampling strategy that can provide an adaptive receptive neighborhood for every node at any time.
LSTSR Chi et al.
(2022) propose Long Short-Term
Preference Modeling for Continuous-Time Sequential Recommendation to capture the evolution of short-term preference under dynamic graph.
DyRep Trivedi et al.
(2019) uses RNNs to propagate messages in interactions to update node representations.
DynAERNN Goyal et al.
(2018) uses a fully connected layer to first encode the network representation, then pass the encoded features to the RNN, and use the fully connected network to decode the future network structure.
VRGNN Hajiramezanali et al.
(2019) generalizes variational GAE Kipf & Welling (2016) to temporal graphs, which makes priors dependent on historical dynamics and captures these dynamics using RNN.
EvolveGCN Pareja et al.
(2020) uses RNN to estimate GCN parameters for future snapshots.
DDGCL Tian et al.
(2021) is a SAM-based method that propose a debiased GAN-type contrastive loss as the learning objective to correct the sampling bias that occurred in the negative sample construction process of temporal graph learning.
B. W HY EXISTING METHODS ARE CONCEPTUALLY AND TECHNICALLY COMPLICATED ?
Please notice that we are not claiming conceptually and technically complicated is bad.
Instead, we are simply suggesting that the conceptually and technically complicated simpler methods might be more preferred than the complicated
one if they could achieve similar performance.
• We say a method is conceptually complicated if the underlying idea behind the method is non-trivial.
For example, CAWs represents network dynamics by “motifs extracted using temporal random walks”, represents node identity by “hitting counts of the nodes based on a set of sampled walks”; TGSRec takes “temporal collaborative signals” into consideration.
These concepts are non-trivial to understand in the first place and could potentially require much domain knowledge from other fields to understand the behavior of the method.
• We say a method is technically complicated if the method is non-trivial to implement due to many hyper-parameters and many details that need to be taken care of, which could potentially make the application to a real-world scenario challenge.
For example, JODIE and TGN require maintaining a “memory” for each node by using RNN, and this “memory” needs to be reset every time after evaluation because then it might carry information about the evaluation data.
CAWs extracts features by using multiple temporal random walks, which makes implementing and hyper-parameter finetuning more challenging.
MeTA and TCL consider many data augmentation strategies, each of which is not trivial to implement and could affect the model’s performance in different ways.
C M ORE EXPERIMENT RESULTS C. C OMPARISON ON CONVERGENCE SPEED AND GENERALIZATION
We include the missing figures of Section 4.
Results on other datasets and the discussion on the experiment results could be found next to Figure 3.
C. T RANSDUCTIVE LEARNING WITH R ECALL @K AND MRR AS EVALUATION METRIC
Recall@K and MRR (mean reciprocal rank) are popular evaluation metrics used in the real-world recommendation system.
The larger the numbers, the better the model performance.
Our Recall@K and MRR is implemented based on the Open Graph Benchmark’s link prediction evaluation metrics’ implementation 15 .
More specifically, we first sample 100 negative destination nodes for the source node of each temporal link node pair, then our goal is to rank the positive temporal link node pairs higher than 100 negative destination nodes.
In the following, we compare the Recall@5 and MRR score of GraphMixer with the selected four most representative baselines.
Please notice that since these methods are implemented under the same framework, the model performance is evaluated by using the same model used in Table 1, the comparison is guaranteed to be fair.
[Table]
We have the following observations on Table 1:
• According to the results in Table 6, GraphMixer could achieve outstanding performance across all datasets.
Especially on the LastFM and GDELT datasets (denser graphs than other datasets).
This might implies GraphMixer is more suitable for denser graphs than other baseline methods.
• The results of some baseline methods behave less better with Recall@K and MRR evaluation metrics.
For example, TGN on LastFM dataset, TGAT on LastFM and GDELT, etc.
The above results also imply the limitation of only considering average precision and AUC score for temporal link evaluation.
C. C OMPARISON ON WALL - CLOCK TIME
In the following
[Table]
we compare the wall-clock time it takes for GraphMixer and baselines to finish a single epoch of training.
[Table]
When comparing the computation time of GraphMixer with CAWs, TGSRec
[Table]
and DDGCL
[Table]
we found that GraphMixer takes significantly lesser time than these baselines, which indicates the effectiveness of GraphMixer.
When comparing the computation time of GraphMixer with JODIE, DySAT, TGAT, APAN
[Table]
and TGN, we found that GraphMixer is very close to or even slightly faster than some baseline methods.
Our computation time is slightly slower than other baseline methods (e.g., JODIE and TGN) mainly because these baselines are using well-optimized computation functions from DGL Wang et al.
(2019), while GraphMixer is just using a composition of basic PyTorch functions.
In fact, according to the Table 8, GraphMixer has a similar/smaller amount of parameters with these baselines.
[Table]
Besides, our current implementation also need to preprocess the input data for each epoch of training, e.g., sorting nodes in subgraph according to temporal order and removing duplicated edges.
For example, the data preparation at each epoch takes 41 sec on Reddit, 9 sec on Wiki, 20 sec on MOOC, 48 sec on LastFM, and 71 sec on GDELT.
However, by caching the pre-processed data in the memory, we only need to pre-process the input data at the first epoch of the training process because our neighbor selection is deterministic and the input data does not change at each epoch.
C. T RANSDUCTIVE LEARNING WITH AUC AS EVALUATION METRIC
AUC (Under the ROC Curve) is one of the most widely accepted evaluation metric for link prediction, which has been used in many existing works Xu et al.
; Rossi et al.
In the following, we compare the AUC score of GraphMixer with baselines.
We have the following observations:
GraphMixer outperforms all baselines on all datasets.
In particular, GraphMixer attains more than 1% gain over all baselines on the LastFM , GDELT-ne , and GDELT-e datasets, attains around 2% gain over non-RNN methods DySAT and TGAT on the Wiki dataset, and attains around 11% gain over non-RNN methods DySAT and TGAT on the GDELT-ne dataset.
The experiment results provide sufficient support on our argument that neither RNN nor SAM is necessary for temporal graph link prediction.
According to the performance of GraphMixer-L on datasets only have link timestamp information ( MOOC , LastFM , and GDELT-ne ), we know that our time-encoding function could successfully pre-process each timestamp into a meaningful vector.
By comparing the performance
GraphMixer-N and GraphMixer on Wiki , MOOC , and LastFM datasets, we know that node-info encoder alone is not enough to achieve a good performance.
However, it provides useful information that could benefit the link-info encoder.
4 By comparing the performance of GraphMixer-N on
GDELT and GDELT-ne , we observe that using one-hot encoding outperforms using node features.
This also shows the importance of node identity information because one-hot encoding only captures such information. [Table]
C. B ASELINES WITH UNDIRECTED TEMPORAL GRAPH
GraphMixer utilizes undirected temporal graph to capture whether two nodes are frequently connected in the last few timestamps.
In the following, we test whether using undirected temporal graph could improve the performance of baseline methods.
As we can see from Table
[Table]
using undirected temporal graph cannot improve the performance of baseline methods much because such information are already implicitly captured via their neural architecture design or sampling methods.
[Table]
D D ISCUSSION ABOUT MODEL PERFORMANCE ON L AST FM
Our results show that GraphMixer could outperform baselines on LastFM dataset with a large margin, which is due to a composite effect of multiple factors. In the following, we summarize several potential factors that lead to our observation on the model performance. • Larger average time-gap.
LastFM has a larger average time-gap ( t max − t min ) / |E| than other datasets. As shown in the dataset statistic (Table 5), LastFM has an average time gap of 106, which is significantly larger than other datasets. For example, Reddit’s average time gap is 4 , Wiki’s average time gap is 17 , MOOC’s average time gap is 3 . 6 , and GDELT’s average time gap is 0 . Since baseline methods are relying on RNN and SAM to process the historical temporal information, they implicitly assumes the temporal information is “smooth” and with smaller average time gap.
Therefore, baseline methods could potentially work better on the dataset with a smaller average time gap but are less ideal on LastFM.
GraphMixer is not relying on RNN or SAM, therefore could be less affected by the aforementioned issue. • Larger average node degree. LastFM has a larger average node degree |E| / |V| than other datasets, which potentially prone to over-smoothing (aggregating features from many neighbors make output representation less distinguishable Li et al. (2018b))
and oversquashing effect (aggregating much information into a limited memory might compress too much useful information Alon & Yahav (2020)).
For example, according to the dataset statistic in Table 5, LastFM has an average node degree of 653 , which is larger than the Reddit’s average node degree 61 , Wiki’s average node degree 17 , MOOC’s average node degree 57 , and GDELT’s average node degree 216 . Existing methods either use the memory cell in RNN to store temporal information or use SAM to aggregate temporal information from multi-hops, which could be less ideal on a dense graph due to over-smoothing and over-squashing. However, GraphMixer is less relying on the aggregation schema, therefore its performance is better than the baseline methods.
• Larger maximum timestamp. GraphMixer is using fixed time encoder but baselines are using trainable time-encoders.
Since the largest timestamp t max in LastFM is larger than other datasets, the trainable time-encoder is more affected by the unbounded gradient issue as discussed in Table 2 and Section 4.2. For example, the t max in LastFM is 137 millon, while t max in GDELT 0.2 millon, t max in Reddit 2.6 millon, t max in Wiki 2.6 millon, and t max in MOOC 2.6 millon.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
E.
L OSS LANDSCAPE ON MOOC
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Table]
[Figure] [Figure]
E.
L OSS LANDSCAPE ON GDELT- NE
[Figure]
[Figure]
E. L OSS LANDSCAPE ON GDELT- E
[Figure]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
encoding function .
[Figure]
[Figure]
JODIE
[Figure]
[Table]
[Figure] [Table]
[Figure] [Figure]