CEP3: Community Event Prediction with Neural Point Process on Graph
Abstract
Many real-world applications can be formulated as event forecasting on Continuous
Time Dynamic Graphs (CTDGs) where the occurrence of a timed event between two entities is represented as an edge along with its occurrence timestamp.
However, many previous works handle the problem in compromised settings, either formulating it as a link prediction task on the graph given the event time, or a time prediction problem for which event will happen next.
In this paper, we propose a novel model combining Graph Neural Networks and Marked Temporal Point
Process (MTPP) that jointly forecasts multiple link events and their timestamps on communities over a CTDG.
Moreover, to scale our model to large graphs, we factorize the joint event prediction problem into three easier conditional probability modeling problems.
To evaluate the effectiveness of our model and the rationale behind such a decomposition, we establish a set of benchmarks and evaluation metrics.
The experimental results demonstrate the superiority of our model in terms of both accuracy and training efficiency.
All the source codes and datasets are available in a GitHub repository.
Introduction
Modeling dynamic interactions of entities has become an important topic in different applications across many fields [1].
In some cases, some entities, such as those with dense connections, or with similar characteristics, may form certain communities, and communities could also be defined by users based on their criteria.
People may only be interested in a specific community of entities’ interactions in some practical applications, such as community behavior modeling [2], dynamic community discovery [3] and community outliers detection [4].
We believe that anticipating future occurrences in the community is a task with a high degree of practical significance. For example, tracking the development of social gatherings in a community can help with early epidemic (like COVID-19) intervention [5].
Predicting the behavior of social media communities enables more efficient information recommendation and ad placement.
For better understanding and forecasting the events in a community, we suggest organizing event stream as a Continuous Time Dynamic Graphs (CTDG), and predicting a series of future events not only about which two entities will be involved but also when they will occur.
CTDG [6] is a common representation paradigm for organizing dynamic interaction event stream over time, with edges and nodes denoting the events with timestamp and the pairwise involved entities, respectively.
Formally, denote a CTDG as G =
( V, E T ) , where E T = { ε i : i = 1 , · · · , T } is the set of edges, ε i =
( u i , v i , t i ) with source node u i , destination node v i , and timestamp t i .
The edges are ordered by timestamps, i.e., t i ≤ t j given 1 ≤ i ≤ j ≤ T .
We further denote a CTDG within a temporal window as G i : j = ( V, E i : j ) where E i : j = { ε k : i ≤ k < j } .
Given the queried community (or node candidates that people are interested in) C q ⊂ V , predicting K future events within the community given n observed events requires to model the following conditional distribution:
[Equation]
where the distribution of each edge ε n + i is further a triple joint probability distribution of its source and destination nodes as well as its timestamp.
The community event forecasting task is illustrated using Fig. 1.
Compared with traditional time series prediction, event forecasting on a CTDG jointly consider the spatial information characterized by the graph and the temporal signal characterized by the event stream to make a more accurate prediction.
[Figure]
There have been several lines of work to approach the problem but all in compromised settings.
Temporal Graph Neural Networks (TGNNs) extends GNNs of static graphs to CTDGs by incorporating temporal signals into the message passing procedure.
Most of TGNN progress [7–9] focuses on temporal link prediction task, i.e., modeling the conditional distribution p ( v n +1 | G 1: n , u n +1 , t n +1 ) .
Other parts of the progress use Temporal Point Process (TPP) for event timestamp prediction which predicts the time of the next event but requires the entities to be known, i.e., modeling the conditional distribution p ( t n +1 | G 1: n , u n +1 , v n +1 ) .
All these models are not directly applicable to the event forecasting problem on CTDGs.
Marked TPP (MTPP) [10, 11] and its variations such as Recurrent Marked Temporal Point Process (RMTPP) [12] associate each event with a marker and jointly predict the marker as well as the timestamp of future events.
MTPP and RMTPP are capable of predicting CTDG events by treating the entity pair ( u i , v i ) as the event marker.
However, these kinds of MTPP-based methods face three major drawbacks.
To begin with, MTPP methods treat edges as individual makers, which are unable to utilize the community and relationship information, resulting in a suboptimal training solution.
Besides, individual makers also bring an O ( | V | 2 ) marker distribution space, making the model less scalable to large graphs.
The last difficulty is that RMTPP is further constrained by its recurrent structure, which must process each event sequentially for keeping events’ contextual correlation.
As far as we know, Transformer Hawkes [13] is the first progress that incorporates the graph structure information into a temporal point process to jointly predict the incident nodes and timestamp.
However, it only supports reasoning future events on the static graph with dynamic node properties, losing the flexibility to process structurally changing dynamic graphics.
Our contributions in this paper are:
i)
We handle the C ommunity E vent P redicting ( CEP ) task with a graph P oint P rocess ( CEP3 ) model, which is significantly harder than both standalone temporal link prediction and timestamp prediction tasks.
Our model incorporates both spatial and temporal signals using GNNs and TPPs and can predict event entities and timestamps simultaneously.
ii) To scale to large graphs, we factorize the mark distribution of MTPP and reduce the computational complexity from O ( | V | 2 ) of previous TPP attempts [12, 14] to O ( | V | ) .
Moreover, we employ a time-aware attention model to replace the TPP model’s recurrent structure, significantly shortening the sequence length of each training step and enabling mini-batches training.
iii)
We propose new benchmarks for the community event forecasting task on a CTDG.
Specifically, we design new evaluation metrics measuring prediction quality of both entities and timestamps.
For baselines, we collect and carefully adapt state-of-the-art models from time series prediction, temporal link prediction and timestamp prediction.
Our evaluation shows that CEP3 is superior across all four real-world graph datasets.
Source code has been already made publicly available.
Related Work Analysis 2.1 Temporal Graph Learning
Temporal Graph Learning aims at learning node embeddings using both structural and temporal signals, which gives rise to a number of works.
CTDNE [15] and CAWs [6] incorporate temporal random walks into skip-gram model for capturing temporal motif information in CTDGs.
JODIE [7] and TigeCMN [16] adopt recurrent neural networks (RNNs) and attention-based memory module respectively to update node embedding dynamically.
Temporal Graph Neural Networks like TGAT [9] and TGN [8] enhance the attention-based message passing process from Graph Neural Networks with Fourier time encoding kernel.
These attempts focus on the temporal link prediction task.
Besides that, other works [17, 18] focus on the information diffusion task which aims at predicting whether a user will perform an action at time t .
Nevertheless, none of them is designed for event forecasting.
RE-Net [19] and CoNN [2] study a similar event forecasting setting on Discrete Temporal Dynamic
Graphs (DTDGs).
However, continuous time prediction is much harder and their methods cannot be directly applied.
2.2 Neural Temporal Point Process
A temporal point process (TPP) [20, 21] is a stochastic process modeling the distribution of a sequence of events associated with continuous timestamps t 1 , · · · , t n .
The theoretical underpinnings and wideranging practical applications of the TPP methods are described in Appendix A.
The majority of neural TPP approaches leverage recurrent neural network (RNN) [12, 22–24] based structure to parametrize the stochastic process function and forecast the future events.
Such approach cannot be trained in parallel and cannot capture long-term dependencies.
Transformer Hawkes [13] replaces the
RNN module with a temporal-aware attention transformer to capture temporal dependencies.
However, these methods cannot be directly applied to event forecasting on CTDGs due to the following reasons.
First, a CTDG is essentially a single event sequence, whose length ranges from tens of thousands to hundreds of millions.
RNNs (even LSTMs) are known to have trouble dealing with very long sequences.
One may consider dividing the sequence into multiple shorter windows, which will make the events disconnected within the given window and discard all the data before it.
This fails to explore the dependencies between events that are distant over time but topologically connected (i.e., sharing either of the incident nodes).
One may also consider training an RMTPP with Truncated
BPTT [25] on the long sequence as a whole.
However, this is inefficient because parallel training is impossible due to its recurrent nature, which means that one has to unroll the sequence one event at a time.
Second, although Transformer Hawkes [13] discards the RNN structure, it directly models the marker generation distribution with a unit softmax function, which will produce a vector with space complexity of O ( | V | 2 ) , as the event markers will be essentially the events’ incident node pairs.
This is not applicable to dynamic graphs with changing structure and undesirable for large graphs.
2.3 Temporal Point Process on Dynamic Graph
Previous works, such as [14, 26–28], use kinds of recurrent architecture to approximate temporal point process over graphs.
However, recurrent architecture prevents the model from parallelized minibatch training, which is undesirable especially on large-scale graphs.
This is because learning long-term dependencies using recurrent architecture requires the model to traverse the event sequence one by one instead of using random mini-batch selection. [Figure]
[Figure] MMDNE [29], HTNE [30] and DSPP [31] employ the attention mechanism to avoid the inefficiency of the recurrent structure in training with large CTDGs.
However, these works are restrictively dedicated to link prediction or timestamp prediction task.
Adapting the two models to event forecasting requires non-trivial changes since neither of them handles efficient joint forecasting of the event’s incident nodes.
Model
Our model is depicted in Fig.
To predict the next K events ε n +1 , · · · , ε n + K given the history graph G 1: n , we first obtain an initial representation h (0) for every node using a GNN Encoder, as well as an initial graph ˜ G (0) .
Then for the i -th step, we predict ε n + i = ( u n + i , v n + i , t n + i ) , i.e., the source node, destination node, and timestamp for the next i -th event.
The predicted event is then added into ˜ G
( i − 1) to form ˜ G ( i ) , to keep track of what we have predicted so far.
The hidden states h ( i ) are then updated from the new graph ˜ G ( i ) and h
( i − 1) .
This generally follows the framework of RMTPP [12], except that i) we initialize the beginning states of Auto-Regressive Message Passing Module with a time-aware GNN, which allows our recurrent module to traverse over a much shorter sequence without losing historical information; ii) we update the Auto-Regressive Message Passing network states with a GNN to model the topological dependencies between entities caused by new events; and iii) we forecast the nodes and the timestamp for an event by decomposing the joint probability distribution.
We give specific details of each component as follows.
3.1 Structural and Temporal GNN Encoder
The Encoder GNN Layer should be capable of encoding relational dependencies, timestamps, and optionally edge features at the same time.
The encoded node representations are used by the forecaster in next section to generate the predicted events.
It has the following form:
[Equation]
where N v is the brief formulation of N v [1 : n ] , which represents the subgraph induced by the 1-hop neighborhood of node i on the graph G 1: n , and ϕ ( t ) are learnable time encodings used in [8, 9, 32].
f encagg and g encmsg can be any aggregation and message functions of a GNN-based representation encoder.
z (0) v could be node v ’s feature vector, and e tuv means the edge feature between node u and v at time t .
We use a GNN to initialize the recurrent network’s states because it takes the historical events within a topological local neighborhood, including the incident nodes, the timestamps, and the feature of events together as input, while enabling us to train on multiple history graphs in parallel.
In particular, we use a neighborhood graph temporal attention based method for encoding, whose detailed formulation is as follows.
Temporal Graph Attention Module is a self attention based node [Figure] embedding method inspired by [9], with detailed formulation as:
[Equation]
The multi-head attention with N heads is computed as:
[Equation]
[Equation] where W ( l ) Q,a means one head of multi-head attention weight matrix and d q means the dimension of the vector q ( l ) v .
The temporal encoding module is the same as in the original papers [8, 9]: [Equation]
[Equation]
where ⃗ w and ⃗ b are learnable parameters, d w is the dimension of weight vector w .
Although a GNN cannot consider events outside the neighborhood, we argue that this impact is minimal.
We empirically verify this argument by comparing CEP3 against the variant that uses both attention and RNN based memory module in the training phase (named CEP3 w/ RNN ), which can incorporate historical events and time-aware information by the view of topological locality and recursive impact, respectively.
However, RNN takes drastically more memory and time in training because of the same reason in Section 2.3.
3.2 Hierarchical Probability-Chain Forecaster
Fig.
3 demonstrates the details of our event forecaster.
We can see that the forecaster predicts future events only according to node embeddings and historical connections in the selected candidates (or communities).
It means that we do not have to be concerned about a large number of communities which are likely to slow down the process, because our CEP3 model can handle numerous communities simultaneously during training and inference.
From the superposition property [33] of an MTPP described in Section A, we forecast the next event ε n + i = ( u n + i , v n + i , t n + i ) by a triple probability-chain:
[Equation]
It means that we can predict first the timestamp, then the source node, and finally the destination node.
We predict t n + i by modeling the distribution of the time difference as follows:
[Equation]
where h (0) v is initialized using the node representation learned by z ( L ) v , and Softplus ensures that η ( i ) v is above zero and the gradient still exists for negative η ( i ) v values.
MLP t represents a multilayer perceptron to generate the intensity value of time.
Since ∆ t n + i obeys exponential distribution, we simply sample the mean value of time intensity distribution, 1 λ i , as the final ∆ t n + i output during training.
The conditional intensity of the next event occurring on one of the nodes in V is thus the sum of all η ( i ) v [33].
Other conditional intensity choices are also possible.
We then generate the source node u n + i of event ε n + i from a categorical distribution conditioned on t n + i , parameterized by another MLP:
[Equation]
where ∥ means the concatenation operation and ϕ has the same form as in Eq.
We then generate the destination node v n + i similarly, conditioned on t n + i and u n + i :
[Equation]
Note that the formulation above will only generate two distributions that have | V | elements, instead of | V | 2 as in RMTPP[12].
The implication is that during inference the strategy will be greedy : we first pick whatever source node that has the largest probability, then we pick the destination node conditioned on the picked source node.
To verify the impact of this design choice, we also explore a variant of our method where we generate a joint distribution of the pair ( u n + i , v n + i ) with O ( | V | 2 ) elements, which we name CEP3 w/o HRCHY (‘hrchy’ is abbreviated for ‘hierarchy’, the noun of the word ‘hierarchical’).
The proposed CEP3 model obeys the probabilistic form of Eq.
(6), while the ablation model CEP w/o HRCHY decomposes p ( u n + i , v n + i , t n + i ) into p ( t n + i ) × p ( u n + i , v n + i | t n + i ) .
If a graph has a lot of nodes, evaluating p ( u n + i , v n + i ) will incur a linear projection with O ( | V | 2 ) complexity.
This is another obstacle to scaling up to larger datasets.
3.3 Auto-Regressive Message Passing
As shown in Fig.
3, we assume that an event’s occurrence will directly influence the hidden states of its incident nodes.
Moreover, the influence will propagate to other nodes along the links created by historical interactions.
Therefore, after generating the new event ε n + i , we would like to update the nodes’ hidden states by message passing on the graph with the new events.
We achieve this by maintaining another graph ˜ G
( i ) that keeps track of the graph with the historical interactions G 1: n and the newly predicted events up to ε n + i .
Specifically, we initialize ˜ G (0) with the candidate node set C as its nodes.
The resulting graph encompasses the dependency between candidate nodes during the encoding stage.
Every time a new event ε n + i is predicted, we add the event back in: ˜ G ( i ) =
˜ G ( i − 1) ∪ ε n + i .
Afterwards, we update the nodes’ hidden states using a message passing network such as GCN [34] for spatial propagation and a GRU [35] for temporal propagation:
[Equation]
where N v ˜ G
( i ) is the set of the neighboring events of node v in ˜ G
( i ) , f updagg can be any message aggregation function and g updmsg can be any message function.
To verify the necessity of updating the community using message passing after a event, we also explore a variant where we do not update all the node’s hidden states in the community, but only the incident nodes u n + i and v n + i .
We name this variant CEP3 w/o AR .
3.4 Loss Function and Prediction
The forecasting module outputs the next event’s timestamp t n + i and indicent nodes u n + i and v n + i for all events ε n + i , which are minimized via negative log likelihood.
Specifically, the loss function goes as follows:
[Equation]
where the first two terms within summation are log survival probability from Eq.
(14) and the last two terms are log probabilities for source and destination node prediction.
The time integration term (cid:82) t t n λ ( τ ) dτ in Eq.
(14) is approximated using a first order integration method by λ ( t )∆ t for the ease of computation.
Experiments
In this section, we test the performance and efficiency of the proposed methods against several baselines on four public real-world temporal graph datasets: Wikipedia, MOOC [7], GitHub [14], and SocialEvo [36].
The detailed description about datasets is put in the Appendix B. To verify the effectiveness of our CEP3 model, we suggested three ablation models (CEP3 w/ RNN, CEP3 w/o
HRCHY.
and CEP3 w/o AR) and five advanced methods (GRU, Hawkes Process [37], Poisson Process,
RMTPP [12] and Dyrep [14]) as our baseline approach.
Almost all baseline methods have been briefly explained in section 2 and 3.
In Appendix C we explain why we choose these baselines and provide implementation details for better reproducibility.
We also provide the details of our network architecture and the hyper-parameters in Appendix D.
The source code is based on PyTorch and Deep
Graph Library [38], which is publicly available at https://github.com/WangXuhongCN/CEP3 .
4.1 Evaluation Metrics
For evaluation on a specific dataset, we utilize the communities segmented by the conventional community detection algorithm Louvain [39] as the candidate node set, and we report the average result of all communities.
For each community C q , we measure the perplexity ( PP C q ) of the ground truth source and destination node sequence for evaluating the node predicting performance, and evaluate the mean absolute error ( MAE C q ) of the predicted timestamps.
Using our MAE to evaluate the quality of auto-regressive forecasting sequence over multiple timesteps can also be seen in traffic flow prediction [40].
Perplexity (PP) [41] is a concept in information theory that assesses how closely a probability model’s projected outcome matches the real sample distribution.
The less perplexity the situation, the higher the model’s prediction confidence.
In the field of natural language processing, perplexity is also commonly used to evaluate a language model’s quality, i.e., to evaluate how closely the sentences generated by the language model match real human language samples.
A language model predicts the next word from the word dictionary, whereas our event predicting model selects nodes from the node candidates (community).
Therefore, it is reasonable to employ perplexity as a metric in our task.
Specifically, suppose we have the communities’ ground truth event sequence ( u i , v i , t i ) and the prediction sequence (ˆ u i , ˆ v i , ˆ t i ) where i = 1 , · · · , K .
For each community, we compute perplexity as
Perplexity = exp (cid:32) −
[Equation]
K
K (cid:88) i =
[log p ( u i )
+
[Equation] [Equation] For the distance between two sequences with difference lengths, we compute MAE [42]:
[Equation]
To keep it comparable in diverse datasets, the MAE is divided by the max time span t K − t 0 and the sequence length K .
We report the average PP C q of all communities as the PP of a certain dataset, and MAE is calculated in the same way.
Smaller values of both metrics indicate better performance.
4.2 Result Analysis
From Table 1 we can see the notable superiority of CEP3 over other baselines in different datasets under both MAE and perplexity.
The MAE difference between GRU and RMTPP shows the effectiveness of temporal point process in predicting timestamps.
Comparing CEP3 with the sequencebased TPP model RMTPP, we can see that using GNN to capture historical interaction information can improve the forecasting performance.
Furthermore, when comparing DyRep w AR versus
DyRep and CEP3 w/o AR versus pure CEP3 , we can conclude that auto-regressive updates can better capture the impact of newly predicted events.
Our model performs better than DyRep w AR because in our CEP3, during the auto-regressive update, the newly predicted event not only influences the node involved in the event but also propagates to other nodes via message passing.
It is worth mentioning that our CEP3 model is not only more effective than other baseline models in terms of performance, but it also allows for parallel training and have faster training speed especially in large datasets.
The training loss curve and analysis with different parallel sizes are shown in Appendix E.
[Figure]
The sizes of plotted nodes indicate their degrees, whereas the colors of edges represent the connection frequencies.
Note that the edge colors and node sizes need be
4.3 Forecasting Visualization
From top to bottom, Fig. 4 visualizes the circle layout of a certain community within the graphs of the GitHub, Wikipedia and MOOC datasets, respectively.
We plot the ground truth, our model’s prediction and DyRep’s prediction for comparison.
The visualized graphs are generated as follows:
We first apply the learned forecasting model to predict the edges using Monte Carlo sampling.
This generation process is repeated for three times.
We then discard generated edges that are not within [Table] the 33% highest prediction probabilities and obtain the final generated graph.
Generating multiple times and then discarding the less possible edges is to reduce uncertainty in each one-time generated graphs
[Table]
In the first row, both CEP3 and DyRep capture the triangle connection in this small community.
However, the triangle is lighter in the ground truth, which means that DyRep over-reinforces this connection in its predictions.
In the second row, the prediction result of CEP3 is more similar to the truth, whereas DyRep generates a high-frequency purple edge which does not exist in the original graph.
In the third row, CEP3 successfully learns the two black edges in ground truth, but DyRep predicts more than two links in darker colors.
We can see that our method successfully recognizes the high-degree nodes, and captures many patterns of interactions as well as the evolution dynamics of the interaction graph, which includes those of the nodes with higher degrees.
One pressing goal of community event prediction, rather than focusing on a single local node, is to anticipate if a certain high-frequency pattern will emerge in the community from a global perspective.
Typical application cases include money laundering patterns [43] in finance and disease transmission patterns [44], etc.
4.4 Ablation Study
In Section 3 we have mentioned three variants: CEP3 w/ RNN to trade parallelized training for long-term dependency modeling with RNN-based memory module, CEP3 w/o HRCHY to compare hierarchical (HRCHY) prediction versus joint prediction of incident nodes, and CEP3 w/o AR to compare using an auto-regressive module versus not using one.
CEP3 w/o HRCHY.
The formulation without hierarchy structure yields a slower training and inference speed as shown in Fig. 5(a).
We demonstrate that CEP3 is more efficient on large communities compared to CEP3 w/o HRCHY.
As is shown in Fig.
5(a), computing intensity function of each possible node pair would take more time by orders of magnitude.
We tackle this issue by decomposing the node pair prediction problem into two node prediction sub-problems, which can be solved quicker especially in large-scale communities.
CEP3 w/o AR.
The model performance drops significantly and yields similar result as DyRep w AR.
From Fig.
5(b), we can see that the AR forecasting is not always useful in all varying numbers of prediction steps.
When the number of “prediction steps” is small (such as 10), CEP3 could just use the node embeddings at time t n to predict the events.
When the number of steps becomes larger, the systematic accumulated errors from AR gradually accumulate, leading to low MAE accuracy.
And as the number of steps increases, the initial node embedding would have little effect in distant future events.
This indicates that without AR, it may be difficult for the CEP3 model to accurately predict long-term occurrences.
CEP3 w/ RNN.
The results show that the memory module brings performance improvement over pure CEP3, except on the GitHub dataset.
The reason is perhaps that the GitHub dataset is a small one with a limited number of nodes and edges, and meanwhile it has a significantly longer timespan than the other datasets.
This means that the interaction has low frequency in this situation, causing the memory update to be insufficient.
[Figure]
Conclusion and Future Work
In this paper we have formulated the community event forecasting task on a continuous time dynamic graph and set up benchmarks using adaptations of previous works.
We further propose a new model to tackle this problem utilizing graph structures.
We also address the scalability issue when formulating the temporal point process on the graph and reduce complexity with a hierarchical formulation.
Experimental results show the prediction accuracy and training efficiency of our models.
It is worth mentioning that the concept of community discussed in this study is not limited to communities in traditional network science, it can even be represented with a specific real-world scenario.
For instance, monitoring and forecasting the spread of traffic congestion [40] can prevent it from spreading beyond the local area.
In a multi-agent system, the community can represent the visible domain of each agent, and the CEP task can be viewed as the prediction of agents’ decisions.
Consequently, our proposed algorithm has the potential to be implemented in more concrete scenarios, such as autonomous driving, urban simulation, and traffic flow prediction.
However, this work remains two important limitations to solve.
One is that we have not provided an experimental evaluation metric of the joint predictions because there have not been any widely accepted metrics that can be used directlyfor now.
We plan to consider some metrics from the dynamic graph generation field [45] in future work.
The other is that the evaluation is very dependent on the pre-defined communities detected by the Louvain algorithm.
Future work will include comparisons utilizing different community detection algorithms (e.g. linkage algorithms [46] and spectral clustering [47]).
Acknowledgement
This work was partly supported by National Natural Science Foundation of China (62222607, 61972250, 72061127003).
A Preliminaries of Temporal Point Process
A TPP is mostly characterized by a conditional intensity function λ ( t ) , from which it computes the conditional probability of an event occurring between t and t + dt given the history { t i : t i < t } as λ ( t ) dt .
According to [48], the log conditional probability density of an event occurring at time t can be formulated as
[Equation]
Additionally, a marked temporal point process (MTPP) associates each event with a marker y i which is often regarded as the type of the event.
MTPP thus not only models when an event would occur, but also models what type of event it is.
Event forecasting over CTDGs can also be modeled as an
MTPP if treating the event’s incident node pair as its marker.
The conditional intensity of the entire
MTPP can be modeled as a sum of conditional intensities of each individual marker:
λ = (cid:80) m λ m .
This allows us to first make the prediction of event time, and then predict the marker conditioned on time via sampling from a categorical distribution: m ∼ Categorical ( λ m ) .
This avoids modeling time and marker jointly.
Such an idea has been widely used in follow-up works such as Recurrent Marked
Temporal Point Process (RMTPP)
RMTPP parametrizes the conditional intensity and the marker distribution with a recurrent neural network (RNN).
RMTPP’s variants include CyanRNN [49] and
ARTPP [50].
We also demonstrate several other relevant works and applications related to TPPs [51].
CoEvolving [52], a variant model of MTPP, uses Hawkes processes to model the user-item interaction.
NeuralHakwes [22] relaxes the positive influence assumption of the Hawkes process by introducing a self-modulating model.
DeepTPP [53] models the event generation problem as a stochastic policy and applies inverse reinforcement learning to efficiently learn the TPP.
[54] models link and retweet generation on a social network with a TPP, and also provides a simulation algorithm that generated from the TPP model.
It is very similar to our task except that it is focused on a specific social network setting, and the authors did not quantitatively evaluate the quality of the simulation model.
B Datasets
Table 2 shows summary statistics of the datasets used in our experiments.
A detailed description is put in the below.
[Table]
Wikipedia [7] dataset is widely used in temporal-graph-based recommendation systems.
It is a bipartite graph consisting of user nodes, page nodes and edit events as interactions.
We convert the text of each editing into a edge feature vector representing their LIWC categories [55].
MOOC [7] dataset, collected from a Chinese MOOC learning platform XuetangX, consists of students’ actions on MOOC courses, e.g., viewing a video, submitting an answer, etc.
GitHub [14] dataset is a social network built from GitHub user activities, where all nodes are real
GitHub users and interactions represent user actions to the others’ repositories such as Watch and
Fork.
Note that we do not use the interaction types as we follow the same setting as [14].
SocialEvo [14, 36] dataset is a small social network collected by MIT Human Dynamics Lab.
Since the public dataset SocialEvo and GitHub have no edge features, we generate 10-dimensional edge features using attributes including the current degrees of the two incident nodes of an edge, and the time differences between the current timestamp and the last updated timestamps of the two incident nodes.
"Percentage of unique edges" in Tab 2 represents the likelihood that new events already happened, and "Average unique neighbors" in Tab 2 measures the likelihood that entities on the graph will seek out connections with other entities that have never interacted with them.
Note that the time differences are described in the numbers of days, hours, minutes and seconds, respectively.
C Baselines
[Table]
O ( | V | ) O ( | V | 2 )
O ( | V | 2 ) O ( | V | 2 )
O ( | V | 2 ) O ( | V | 2 )
[Table]
In addition to CEP3 , CEP3 w/ RNN , CEP3 w/o HRCHY and CEP3 w/o AR mentioned in Section 3, we compare against the following baselines: a Seq2seq model with a GRU [35], a Poisson Process ( TPP-Poisson ), a Hawkes Process ( TPP-Hawkes ) [37], RMTPP [12] and its variant with the same two-level hierarchical factorization as in Eq.
(8) and (9) ( RMTPP w HRCHY ), an adaptation of
DyRep [14] and an auto-regressive variant (named DyRep and DyRep w AR ).
Notably, RMTPP and its variants are SOTA models for MTPPs in general, and DyRep is SOTA in temporal link prediction and time prediction on CTDG.
Since our task is new, we made adaptations to the baselines above, with details of each baseline is as follows:
Time series Methods : For baseline models of sequential prediction, we build an RNN model, Gated
Recurrent Unit (GRU).
Each source and destination cell has a hidden state, the output of the model will be predicted time mean and variance as well as probability for each class to interact, the time will be formulated as Gaussian distribution and source and destination node will be formulated as categorical distribution.
This formulation forces GRU predicting timestamp of upcoming events only depending on the hidden state in RNN, whereas other baselines adapt TPP function as a stochastic probability process, obtaining a better modeling capability.
We use a GRU [35] as a simple baseline without using TPP to model time distribution, treating event forecasting on CTDG as a sequential modeling task.
It takes in the event sequence and outputs the next K event in a Seq2seq fashion [56].
The loss term for time prediction is mean squared error and the loss term for source and destination prediction is the negative log likelihood.
This formulation forces GRU to predict the timestamp of upcoming events only depending on the hidden state in RNN, whereas other baselines adapt TPP for better modeling capabilities.
Temporal Point Process Methods : Following the benchmark setting of [12], we compared our model against other traditional TPP models and deep TPP models.
[Table]
• TPP-Poisson :
We assume that the events occurring at each node pair ( u, v ) follows a Poisson Process with a constant intensity value λ u,v , which are learnt from data via Maximum Likelihood Estimation (MLE).
• TPP-Hawkes [37]: We assume that the events occurring at each node pair ( u, v ) follows a Hawkes Process with a base intensity value µ u,v and an excite parameter α u,v , which are learnt from data via MLE.
• RMTPP [12]: We directly consider each source and destination node pair as a unique marker.
We note that this formulation will exhaust memory and time on graphs with more than a few thousand nodes, since RMTPP will assign a learnable embedding for each node pair, resulting in O ( | V | 2 ) (here | V | is total number of nodes in the entire CTDG) parameters which is too expensive to update.
• RMTPP w HRCHY : We consider a variant of RMTPP where we replace the source-destination node prediction with our hierarchical formulation: we first select the source node, then condition on the source node we select the destination node.
The latter formulation can also serves as an ablation study to demonstrate the usage of considering the graph structure.
• DyRep [14]: DyRep is a popular work that combines the temporal point process with graph learning techniques to model both temporal and spatial dependencies.
Since the original DyRep formulation only handles temporal link prediction and time prediction, but not autoregressive forecasting, we compute an intensity value λ u,v with DyRep for each node pair and assume a Poisson Process afterwards.
• DyRep w AR : We make a trivial adaptation to the original formulation of DyRep by updating the source and destination node involved once a new edge is added to the graph.
The update function is identical to DyRep’s updating function during embedding.
This benchmark is designed to demonstrate that the propagation from newly forecast event to local neighborhood is necessary in getting better performance.
We also crave our proposed model for having the ability to implement minibatch training.
As described in the beginning of Section 3, our CEP3 achieves large-scale and parallelized training by utilizing Hierarchical TPP and GNN-based updating module, respectively.
A summary of the mentioned baselines is shown in Table 3, where we conclude that the proposed model CEP3 satisfies all the desirable properties.
D Configuration
In this section, we provide the details of our network architecture, the hyper-parameters and the selected community detection method for better reproducibility.
Table 4 summarizes key parameters in our model.
For all the experiments we train our model and benchmark models on Intel(R)
Xeon(R)
Platinum 8375C CPU @
2.90GHz. [Figure] E Parallel Training Inspired by the Transformer [57] models in NLP, we believe it is essential to use a pure attentionbased model in temporal graph encoders.
This is because using a pure attention-based GNN as an encoder enables us to train multiple time windows in minibatches as described in Section 3.1, whereas models such as Dyrep and RMTPP cannot utilize parallel training due to their RNN structures.
This property allows our model to benefit from mini-batch training such as gradient stabilization and faster convergence.
Fig.
6 shows our experiment result on the Wikipedia dataset with different numbers of parallel processes, and suggests that using parallel training can increase the speed significantly without losing accuracy.