G OPHER : Categorical probabilistic forecasting with graph structure via local continuous-time dynamics
Abstract
We consider the problem of probabilistic forecasting over categories with graph structure, where the dynamics at a vertex depends on its local connectivity structure.
We present G OPHER , a method that combines the inductive bias of graph neural networks with neural ODEs to capture the intrinsic local continuous-time dynamics of our probabilistic forecasts.
We study the beneﬁts of these two inductive biases by comparing against baseline models that help disentangle the beneﬁts of each.
We ﬁnd that capturing the graph structure is crucial for accurate in-domain probabilistic predictions and more sample efﬁcient models.
Surprisingly, our experiments demonstrate that the continuous time evolution inductive bias brings little to no beneﬁt despite reﬂecting the true probability dynamics.
Introduction
In categorical probabilistic forecasting we seek to predict a discrete probability distribution p ( t ) at some instantaneous time t , based on observed time-stamped data [10].
Consider the example of forecasting the most likely locations of the next earthquake over a ﬁnite set of locations at t , given the history of earthquake times and locations.
We can view locations as vertices V on a graph G =
( V, E ) with edges E that represent adjacency.
Speciﬁcally, the probability of an earthquake at node v ∈ V in the near future is mostly inﬂuenced by the probability of earthquakes at nodes within its neighborhood.
This type of graphical structure also appears in other problems, including trafﬁc forecasting [28], information diffusion in social networks [1], epidemic diffusion [24, 13], urban conﬂict patterns [16] and is an example of a marked temporal point process [6].
In this paper we consider categorical probabilistic forecasts where there is a graphical structure to inform us of the local dynamics governing p ( t ) over time.
We formalize the intuition that each component of the probability vector p ( t ) ∈ R | V | obeys local dynamics using the differential equation
[Equation]
which we use to inform our model’s inductive bias.
Here, g governs the local dynamics, N ( v ) ⊆
V denotes the set of neighboring nodes of v , and p v denotes the probability at node v .
To capture the equivariant local dynamics of our forecast p ( t ) , we propose G OPHER , a model that learns a neural
ODE [4] with graph neural network (GNN) [26] dynamics.
Our method G OPHER introduces two inductive biases to aid with probabilistic forecasting over graph-structured categories by 1) utilizing graph structure explicitly and 2) introducing temporal evolution through a neural ODE.
To disentangle the beneﬁts of these two biases, we introduce two baseline models, ablating each bias.
We ﬁnd that utilizing the known graph structure results is key, and results in 10x improvements in accuracy and sample efﬁciency.
On the other hand, explicitly modelling the temporal dynamics surprisingly results in little beneﬁts.
G OPHER : Forecasting with temporal dynamics and graph structure
Let G =
( V, E ) be a graph, and let t i ∈ R +0 denote the timestamp of an event at node v i ∈ V .
Given
G and an irregularly sampled dataset D = { ( t i , v i ) } Ni =1 , we want to learn the probability vector p ( t ) ∈ R | V | of each v ∈ V at any time t .
We wish to model the dynamics of p ( t ) such that the change in the probability at node v depends only on the neighborhood N ( v ) around v , as described in Equation 1.
However, directly parameterizing g from Equation 1 with a neural ODE can violate conservation of probability: 1 ⊤ p ( t ) = 1 .
Instead of explicitly enforcing the sum constraint into our neural ODE, we model the dynamics in a continuous-time embedding space from which we derive the dynamics d p / d t .
Speciﬁcally, let
Z 0 ∈ R | V |× D , where D denotes the embedding space dimension.
We use [-Z 
 [ i ]-]z 0 ,i to denote row i of Zat initial time t 0 , corresponding to the embedding of node v i .
We then model the dynamics of the continuous-time embeddings Z ( t ) ∈ R | V |× D via d
[Equation]
subject to
[Equation]
where g is the learned graph neural network (GNN) dynamics.
To map Z ( t ) to a probability space while preserving equivariance, we learn a shared projection π : R D
→ R such that
[Equation]
Provided that g and π are differentiable, which can be done using smooth activation functions, our model then implicitly models the local temporal dynamic of our problem in Equation 1.
Finally, we train our model G OPHER by maximizing the log likelihood ∑ Ni =1 log p v i ( t i ) with respect to the parameters of π , g , and the initial condition Z 0 .
Incorporating node attributes.
In some cases G may have node attributes { a v } v ∈ V for each node v ∈ V that affect the interaction dynamics, such as the geographical coordinates of each node in a spatial graph or the demographics of a user in a social network.
Node attributes can be easily incorporated by letting the initial node embeddings be a learned function of the attributes,
Z
[ v ] = ψ v ( a v ) , and optimizing with respect to the parameters of { ψ v } .
Related works.
Our paper lies at the intersection of probabilistic forecasting, neural ODEs, and graph neural networks (GNNS), and can be seen as the discrete analogue of continuous normalizing ﬂows [4, 11, 5] on manifolds [17, 18].
Probabilistic forecasting seeks to predict a full distribution at each time step [14, 10], with contemporary methods often relying on deep probabilistic models [22, 25, 21, 20].
A direct application of categorical probabilistic forecasts is marked temporal point processes, which learn the rate of an event type v at time t , summarized by the conditional intensity function λ ( t, v ) = λ ( t ) · p v ( t ) [6].
The inductive bias of a learnable ODE with GNN dynamics has also been explored in the context of other problems, including graph generation [7], node classiﬁcation [19, 2], multi-particle trajectory prediction [19], learning partial differential equations [15], and knowledge graph forecasting [12].
Results: I Can’t Believe Temporal Dynamics Don’t Matter!
Synthetic datasets.
We apply our method to model the mark component of a marked temporal point process (TPP) occurring on the nodes of a graph such that p v ( t ) is the probability of an event occurring on vertex v at time t .
We create a synthetic dataset where events occur over time on a directed graph G , with node probabilities that obey graph advection as an example of local dynamics [3].
Graph advection conserves the total probability by ensuring 1 ⊤ d p / d t = 0 .
We represent the graph G by the weighted adjacency matrix A , where A uv > 0 for ( u, v ) ∈ E .
We sample sequences of events over time [0 , T ] from a homogeneous Poisson process with constant temporal intensity λ ( t )
=
λ = 2 . 5 and temporal node probability p ( t ) ∈ R | V | governed by the graph advection equation [3]
[Equation]
2      t = 0 .
2      t = 1 .
2      t = 2 .
2      t = 3 .
2      t = 5 .−−−−
l og p ( t )           11  t = 0 .
11  t = 0 .
11  t = 0 .           11  t = 0 .
11  t = 1 .−−−−−
l og p ( t )
Here, L out ( A ) :
= D out ( A ) −
A denotes the out-degree graph Laplacian, and D out ( G ) denotes the diagonal out-degree matrix with D out ( G ) ii =
∑ j A ij . We create two graphs structures for our synthetic datasets, a ring graph and a random geometric graph, and visualize their advection on the graph over time in Figure 1; see Appendix D for more details on their construction.
We also visualize the advection dynamics of each component of p ( t ) for the ring graph in Figure 9 of
Appendix D.
We use T = 5 seconds for the ring graph dataset and
T = 1 second for the geometric graph dataset.
Since the timestamps are sampled from a Poisson process and not equidistantly spaced over [0 , T ] , the continuous time aspect of the problem is clearly evident in the dataset.
[Figure]
We evaluate the accuracy and sample-efﬁciency improvements from incorporating graph structure and modelling temporal dynamics in G OPHER .
To disentangle the effects of these two inductive biases, we compare our model to two baseline models.
The ﬁrst is a two-layer
MLP that acts on node-embeddings concatenated with time, which has none of the above inductive biases.
The second is a single-layer
GNN that also acts on node-embeddings concatenated with time.
The GNN incorporates the explicit graph structure, but does not incorporate dynamical systems structure.
We refer to the models as N AIVE MLP and N AIVE GNN respectively.
In our experiments, G OPHER learns g using a Graph Isomorphism Network (GIN) layer [27] parameterized by a two-layer MLP; we use another two-layer MLP for the projection π .
N AIVE GNN uses the same GIN architecture and projection except that it does not learn a differential equation.
Finally, N AIVE MLP replaces the GIN layer with a two layer MLP.
See Appendix D for further details on our experiment hyperparameters.
Figure 2 shows the KL divergence betweeen the ground truth p ( t ) and the learned predictions over time for the ring graph.
We summarize the KL divergence over [0 , T ] in Figure 3 by the geometric mean since the error varies over multiple overs of magnitudes over time [9].
In both ﬁgures, we show the 95% conﬁdence intervals over 3 seeds.
For both datasets, there is 10x difference in accuracy
[Figure] [Figure] [Figure] between the graph structured models and N AIVE MLP, indicating that utilizing the graph structure is greatly beneﬁcial.
Though N AIVE GNN does not explicitly model the local temporal dynamics of the datasets
[Figure]
it performs nearly identically to our model G OPHER in ﬁtting p ( t ) over the training interval [
[Figure]
T ] .
In principle
[Figure]
G OPHER has the best chance of extrapolating to the [ T, 2 T ] time period not seen during training since G OPHER explicitly models the local dynamics.
However, G OPHER ’s poor extrapolation ability suggests that its learned dynamics do not actually reﬂect the true dynamics. Indeed
[Figure] in Figure 7 of Appendix C we show that although G OPHER can ﬁt the training data well [Figure] it is brittle to edge deletions, further indicating G OPHER does not learn the true dynamics
[Figure]
using the COVID-19 dataset preprocessed by Chen et al.
Only the graph-based models are able to capture the variations over time.
See
Appendix A for the empirical distribution of p ( t ) .
Real-world dataset.
We use data released publicly by the New York Times [23] on daily COVID19 cases in New Jersey state to construct a real-world categorical probabilistic forecasting dataset, following the preprocessing script of Chen et al.
We aggregate the cases by county and form a graph with 21 nodes where each node is a county and each edge is a county border.
Using the train/test split from Chen et al.
[5], we obtain per event log likelihoods with 1-standard-deviations of − 2 .
766 ± 0 .
003 for N AIVE MLP, − 2 .
768 ± 0 .
003 for N AIVE GNN, and − 2 .
767 ± 0 .
000 for
G OPHER over 3 seeds.
However, these likelihoods are not representative of the model differences since we ﬁnd a large distribution shift between the train and test distribution shown in Figureof Appendix A.
This distribution shift causes the models to perform equally poorly on the test set.
In actuality, N AIVE MLP completely fails to capture variations in p ( t ) over time, as shown in Figure 4.
Discussion
Although the inductive biases of G OPHER , directly reﬂect properties of categorical forecasting with local continuous-time dynamics, our experiments ﬁnd that, surprisingly, explicitly modelling the temporal dynamics does not improve performance.
Most of the performance gains of G OPHER come from incorporating a graph structure, which can be done with a simple baseline model like
N AIVE GNN.
The failure of G OPHER can be attributed to the fact that the learned dynamics in the embedding space do not accurately reﬂect the ground truth dynamics in probability space.
B Learned forecasts on COVID-19 dataset
[Figure]
C Model robustness
[Figure]
D Implementation details
We use 2 layer MLPs with 64 hidden units per layer whenever we use a MLP.
We use Swish activations to ensure smoothness of our dynamics.
We also use an augmented neural ODE [8], usingdimensions as augmented dimensions out of the 64 hidden dimensions.
D.1 Model architectures.
G OPHER .
We parameterize the dynamics g from Equation 2 using one graph isomorophism network layer parameterized by a MLP.
We also use a MLP to model the projection π .
N AIVE GNN.
We use the same architecture as G OPHER , namely a GIN layer followed by a projection π .
However, instead of using the model to parameterize ODE dynamics, we directly input the node embeddings concatenated with time t through the GNN.
N AIVE MLP.
We replace the GIN layer of N AIVE GNN with a MLP, keeping all else the same.
D.2 Training procedure and dataset details.
To maximize hardware parallelism, we parallelize our neural ODE computation across sequence timesteps and across sequences using the time-reparameterization trick outlined in Chen et al.
For the synthetic datasets, we use the AdamW optimizer with 0.01 learning rate and batch sizefor 30 epochs.
For the COVID-19 dataset, we use a 3 × 10 − 4 learning rate and batch size 4 forepochs.
Here, each batch consists of multiple sequences drawn from the training period [0 , T ] .
We use T = 5 for the ring graph, T = 1 for the geometric graph, and T = 8 for the New Jersey counties graph.
We generate the ring graph dataset by using hand-set coefﬁcients for the edge weights A uv to allow for counter-clockwise transport.
We generate the geometric graph dataset by generating a random geometric graph via the networkx python package and drawing a random sample of { A uv } .
[Figure]