TAMP-S2GCN ETS : C OUPLING T IME -A WARE M ULTIPERSISTENCE K NOWLEDGE R EPRESENTATION WITH S PATIO -S UPRA G RAPH C ONVOLUTIONAL N ETWORKS FOR T IME -S ERIES F ORECASTING
A BSTRACT
Graph Neural Networks (GNNs) are proven to be a powerful machinery for learning complex dependencies in multivariate spatio-temporal processes.
However, most existing GNNs have inherently static architectures and as a result, do not explicitly account for time dependencies of the encoded knowledge and are limited in their ability to simultaneously infer latent time-conditioned relations among entities.
We postulate that such hidden time-conditioned properties may be captured by the tools of multipersistence i e., a emerging machinery in topological data analysis which allows us to quantify dynamics of the data shape along multiple geometric dimensions.
We make the ﬁrst step toward integrating the two rising research directions that is, time-aware deep learning and multipersistence and propose a new model, Time-Aware Multipersistence Spatio-Supra Graph Convolutional Network (TAMP-S2GCNets).
We summarize inherent time-conditioned topological properties of the data as time-aware multipersistence Euler-Poincar ´ e surface and prove its stability.
We then construct a supragraph convolution module which simultaneously accounts for the extracted intra- and inter-dependencies in the data.
Our extensive experiments on highway trafﬁc ﬂow, Ethereum token prices and COVID-19 hospitalizations demonstrate that TAMP-S2GCNets outperforms the state-of-the-art tools in multivariate time series forecasting tasks.
I NTRODUCTION
Multivariate time series forecasting plays an integral role in virtually every aspect of societal functioning, from biosurveillance to ﬁnancial analytics to intelligent transportation solutions.
In the last few years, Graph Convolutional Networks (GCNs) have emerged as a powerful alternative to more conventional time series predictive models.
Despite their proven success, GCNs tend to be limited in their ability to simultaneously infer latent temporal relations among entities (such as correlations both within a time series and in-between time series or the joint spatio-temporal dependencies).
More generally, most existing GCNs architectures are inherently static and as such, do not explicitly account for time-conditioned properties of the encoded knowledge about the complex dynamic phenomena.
At the same time, recent studies on integrating shape properties of the complex data into deep learning (DL) models indicate that topological representations, obtained using the tools of single parameter persistence, can bring an invaluable insight into the system organization and enhance the resulting graph learning mechanisms (Hofer et al., 2019; Carri`ere et al., 2020; Carlsson & Gabrielsson, 2020; Horn et al., 2021).
(By shape here we broadly understand data properties which are invariant under continuous deformations, e.g., stretching, bending, and twisting).
However, in many appli- cations, particularly, involving spatio-temporal processes, the data exhibit richer structures which cannot be well encoded with a single parameter persistence.
We postulate that many critical hidden time-conditioned interrelations which are inaccessible with other methods can be captured by the emerging machinery of multipersistence .
Multipersistence, or multiparameter persistence (MP) generalizes the notion of single parameter persistence to a case when dynamics of the inherent data shape is discerned along multiple geometric dimensions (Carlsson & Zomorodian, 2009).
Despite its premise, applications of MP in any discipline remain nascent at best (Riess & Hansen, 2020;
Kerber, 2020).
We make the ﬁrst step on a path of bridging the two emerging directions, namely, time-aware DL with time-conditioned MP representations of complex dynamic phenomena.
By time-conditioned
MP representations, we mean the most salient topological properties of the data that manifest themselves over time.
To summarize such time-conditioned topological properties, we ﬁrst introduce a dynamic Euler-Poincar´e surface as a new MP summary.
We then propose a directed multilayer supra graph abstraction to represent a sequence of time-varying objects and develop a supragraph convolution module which allows us to simultaneously learn co-evolving intra- and inter-dependencies (i.e., spatial and temporal correlations) in the complex high-dimensional data.
The key novelty of this paper are summarized as follows:
• This is the ﬁrst work to bridge the concepts of MP with the time-aware learning paradigm.
Applications of MP in any ﬁeld of study are currently nascent.
• We introduce a new time-aware multipersistence invariant, a dynamic Euler-Poincar´e surface .
We prove its stability and show its substantial computational gains and high utility for encoding the time-conditioned knowledge.
• We propose a mathematical abstraction of directed multilayer supra graph for time-conditioned knowledge representation and construct a new Time-Aware Multipersistence Spatio-Supra Graph Convolutional Network (TAMP-S2GCNets) which simultaneously learns latent temporal interand intra-relations among entities in the complex high-dimensional data.
• We perform expansive forecasting experiments, in application to highway trafﬁc ﬂow, Ethereum token prices, and COVID-19 hospitalizations.
Our ﬁndings demonstrate superior predictive performance, versatility and computational efﬁciency of TAMP-S2GCNets, compared to the stateof-the-art methods in multivariate time series forecasting.
R ELATED W ORKS
Multipersistence Despite that MP demonstrates very promising results in terms of improving accuracy, tractability and robustness, applications of MP in ML are virtually non-existent (Wright &
Zheng, 2020; Riess & Hansen, 2020; Kerber, 2020).
Some notable efforts in the direction to develop
MP summaries which are suitable for integration with ML models include Multiparameter Persistence Kernel of Corbet et al.
Multiparameter Persistence Landscapes (MP-L) of Vipond (2020), and Multiparameter Persistence Images (MP-I) of Carri ` ere & Blumberg (2020) which are based on the concept of slicing, that is, restricting the MP module to an afﬁne line (or single parameter persistence) (Cerri et al., 2013; Landi, 2014).
Such slicing methods enjoy a number of important stability guarantees but tend to be computationally expensive even in static scenarios, which makes them infeasible for time series forecasting tasks.
In turn, the most recent results of Beltramo et al.
(2021) (i.e., Euler Characteristic Surfaces) and Coskunuzer et al.
(2021) (i.e., Multiparameter Persistence Grids) introduce pointwise MP representations, in application to static point clouds and graphs.
Such pointwise representations are weaker invariants but are substantially more computationally efﬁcient.
Integration of pointwise representation with ML models has not been yet investigated.
Here we propose the ﬁrst time-aware pointwise MP representation, a dynamic Euler-Poincar´e surface , derive its theoretical properties and integrate it with GCN in time series forecasting tasks.
Spatial-Temporal Graph Models and Forecasting Recent studies (Li et al., 2018; Yu et al., 2018;
Yao et al., 2018) introduce graph convolution methods into spatio-temporal networks for multivariate time series forecasting which, as a result, allows for better modeling of dependencies among entities (Wu et al., 2019; Bai et al., 2020; Cao et al., 2020) and handling data heterogeneity.
Despite the GCN successes, designs of the existing spatial-temporal GCNs largely rely on the pre-deﬁned graph structures.
As such, GCNs are restricted in their ability to explicitly integrate time dimension into the knowledge representation and learning mechanisms, thereby, limiting model adaptivity to the dynamic environments and requiring more frequent retraining.
Most recently, Chen et al.
(2021) propose a time-aware GCN, Z-GCNETs, for time series forecasting which integrates zigzag persistence images based on a single ﬁltration, as the primary time-conditioned topological representation.
In general, the zigzag concept can be combined with MP, but it requires more fundamental advances in the theory of algebraic topology.
As such, our time-aware MP learning approach may be viewed as complementary to zigzag persistence, while considering time-changing connections of graph structures in dynamic networks.
T IME -A WARE M ULTIPERSISTENCE E ULER C HARACTERISTIC S URFACES
Spatio-Temporal Graph Construction We deﬁne a spatial network at time step t as G t =
( V t , E t , A t , X t ) , where V t is a set of nodes and E t is a set of edges.
We let |V t | = N and |E t | =
M t .
The adjacency matrix A t ∈ R N × N , and X t = { x t, 1 , x t, 2 , . . .
, x t,N } (cid:62) ∈ R N × F N is the node feature matrix with feature dimension F N .
To construct the spatial network G t , we can build the adjacency matrix A t based on (i) the prior knowledge of graph structure: ﬁrst-order neighbours, i.e.,
A t,uv = 1 if the node u and node v have a connection in the dynamic graph at the time step t ; and (ii) the Radial Basis Function (RBF): degrees of similarity between instances (i.e., nodes) in X t , i.e., A t,uv = 1 exp( −|| x t,u − x t,v || 2 /γ ) ≤ (cid:15) , where γ denotes the length scale parameter and (cid:15) denotes the threshold parameter ﬁlters noisy edges.
Let T be the total number of time steps.
Given a sequence of observations on a multivariate variable, X = { X 1 , X 2 , . . .
, X T } ∈ R N × F N × T with T timestamps and F N node attributes, we construct spatio-temporal networks
G = {G 1 , G 2 , . . .
, G T } via either prior knowledge of network structure or applying RBF to the node feature matrix.
Single Parameter Persistence Persistent homology (PH) based on one parameter discerns shape of the complex data along a single geometric dimension.
The goal is to select some suitable parameter of interest and then to study a graph G t not as a single object, but as a sequence of nested subgraphs, or a graph ﬁltration G 1 t ⊆
G 2 t . . .
⊆
G mt =
G t , induced by this evolving scale parameter.
Armed with such ﬁltration, we can then assess which structural patterns (e.g., loops and cavities) appear/disappear and record their lifespans.
To make the counting process more efﬁcient and systematic, we build a simplicial complex C it from each subgraph G it , resulting in a ﬁltration
C 1 t ⊆ C 2 t . . .
⊆
C mt (e.g., clique complexes).
For example, to construct such ﬁltration, a common method is to consider a ﬁltering function f : V t
(cid:55)→ R and the corresponding increasing set of thresholds { α i } m 1 such that C it = { ∆ ∈ C t : max v ∈ ∆ f ( v ) ≤ α i } .
The resulting construction is called a sublevel set ﬁltration of f , and f can be selected, for instance, as degree, centrality, or eccentricity function (Hofer et al., 2020; Cai & Wang, 2020).
Similarly, f can be deﬁned on the set of edges E t .
More details on a single ﬁltration PH can be found in Appendix B.
Time-Aware Multiparameter Persistence (TAMP) Data in many applications, particularly, involving spatio-temporal modeling, might be naturally indexed by multiple parameters, e.g., real time trafﬁc ﬂow and optimal route in urban transportation analytics.
Alternatively, the primary focus might be on discerning shape properties of the complex data along multiple dimensions.
For instance, to better predict cryptocurrency prices and manage cryptomarket investment performance, we may need to evaluate structural patterns in cryptoasset dynamics not along one dimension but simultaneously along the volume of transactions and transaction graph betweenness, as a measure of joint perception of the cryptomarket volatility among the key investors.
Such multidimensional analysis of topological and geometric properties can be addressed using generalization of PH based on a single ﬁltration to a multiﬁltration case.
That is, the MP idea is to simultaneously assess shape characteristics of G t based on a multivariate ﬁltering function F :
V t
(cid:55)→ R d .
As a result, e.g., for d = 2 and a set of nondecreasing thresholds { α i } m 1 and { β j } n 1 , instead of a single ﬁltration of complexes, we get a biﬁltration of complexes { C α
i ,β j t | 1 ≤ i ≤ m, 1 ≤ j ≤ n } such that if β k < β l , then C α i ,β k t (cid:44) →
C α i ,β l t and if α i < α j , then C α i ,β k t (cid:44) → C α j ,β k t .
Finally, this biﬁltration induces a bigraded MP module { H k ( C α
i ,β k t ) } , where H k is the k th homology group.
Inspired by Beltramo et al.
; Coskunuzer et al.
(2021), we propose a new time-aware MP summary, namely, a Dynamic Euler-Poincar´e Surface .
Deﬁnition 3.1 (Dynamic Euler-Poincar´e Surface) .
Let {G t } Tt =1 be a series of time-varying graphs.
Let F =
( f, g ) be a multivariate ﬁltering function F : V t
(cid:55)→ R 2 with thresholds
I = { ( α i , β j ) | 1 ≤ i ≤ m, 1 ≤ j ≤ n } .
Let C α
i ,β j t be the clique complex of the induced subgraph G α i ,β j t =
F − 1 (( −∞ , α i ] × ( −∞ , β j ]) , t = 1 , 2 , . .
, T and χ be the Euler–Poincar ´ e characteristic.
Then, a sequence of time-evolving m × n -matrices { E t } Tt =1 such that E tij
= χ ( C α i ,β j t ) for 1 ≤ i ≤ m , 1 ≤ j ≤ n is called Dynamic Euler-Poincar´e Surface (DEPS).
(Figure 2 in Appendix C shows a toy example how DEPS is computed.)
Theoretical Guarantees of DEPS Consider two graphs G + and G − , where time index t is suppressed for brevity.
Let F : V ±
(cid:55)→ R 2 be a multivariate ﬁltering function with thresholds
I = { ( α i , β j ) | 1 ≤ i ≤ m, 1 ≤ j ≤ n } .
Let C ± be the clique complexes of G ± , and let (cid:98) C ± =
{ C + ij } be the biﬁltration induced by ( C ± , F, I ) as before.
Let E ± be the corresponding
Euler-Poincar´e Surfaces (i.e., m × n matrices).
Then, set (cid:107) E + − E − (cid:107) 1 , 1 = (cid:80) mi =1 (cid:80) nj =1 | E + ij − E − ij | as the distance between E + and E − , where (cid:107) · (cid:107) 1 , 1 is the vectorized L 1 matrix norm.
We now introduce an L 1 -based MP metric instead of using L ∞ -based metrics like, e.g., matching or interleaving, due to the nature of our summaries E ± (see Remark C.1 in Appendix C).
Let D fk ( C ± ) and D gk ( C ± ) be the k th single parameter persistence diagrams (PDs) of C ± for ﬁltrations induced by functions f, g : V ± (cid:55)→ R , respectively (see Appendix B).
Let C ±
i ∗ and C ±∗ j be clique complexes corresponding to G ± i ∗ =
f − 1 (( −∞ , α i ]) and G ±∗ j = g − 1 (( −∞ , β j ]) .
Deﬁne the i th column distance for the k th PDs as D k i ∗ ( (cid:98) C + , (cid:98) C − ) =
W 1 ( D gk ( C + i ∗ ) , D gk ( C − i ∗ )) , where W 1 is the Wassersteindistance.
Similarly, the j th row distance for k th PDs is D k ∗ j ( (cid:98) C
+ , (cid:98) C − ) =
W 1 ( D fk ( C + ∗ j ) ,
D fk ( C −∗ j )) .
Deﬁnition 3.2 (Weak L 1 -metric for Multipersistence) .
The weak L 1 -metric between (cid:98) C ± is
[Equation] C
[Equation]
(cid:98) C
[Equation]
and D r ( (cid:98) C
+ , (cid:98) C − ) =
(cid:80) Mk =0 (cid:80) nj =
[Equation]
D k [Equation] j ( (cid:98) C + , (cid:98) C − ) .
Now, we can state our stability result for Euler-Poincar ´ e Surfaces.
Theorem 3.1.
Let G ± , F, (cid:98) C ± , E ± be as deﬁned above.
Then, the Euler-Poincar´e Surfaces are stable with respect to the weak L 1 -metric, i.e., (cid:107) E + − E − (cid:107) 1 , 1 ≤ C · D ( (cid:98) C
+ , (cid:98) C − ) for some C > 0 .
The proof of the theorem is given in Appendix C.
This stability result implies that the distances between multiparameter PDs control the distance between the resulting Euler-Poincar ´ e Surfaces.
By combining with the stability result for PDs (Cohen-Steiner et al., 2007), one can conclude that the small changes in the MP ﬁltering function F : V t (cid:55)→ R d or the small changes in the input data can result only in a small change in DEPS surfaces.
For further discussion on implications of the stability result, please refer to Remark C.2.
T IME -A WARE M ULTIPERSISTENCE S PATIO -S UPRA G RAPH C ONVOLUTIONAL N ETWORKS
Given T historical observations X T =
{ X t −T , X t −T +1 , . . .
, X t − 1 } , the multivariate forecasting model F ( · ) is learned to predict future observations in the next H + 1 time steps, i.e., { X t , X t +1 , . . .
, X t + H } =
F ( X t −T , X t −T +1 , . . .
, X t − 1 ) .
Graph Learning Architecture The graph representation learning of our MPS2GCNets is build upon GCN.
To learn node representation from graph topology and node features, the input of GCNbased approaches contains the adjacency matrix A t of the original input graph G t and the node feature matrix X t .
However, the prior knowledge of graph structure (1) might restrict the modeling capacity (i.e., graph edges cannot encode the complex relationships between nodes) and (2) leads to the neglect of neighboring information with high diversity.
To avoid these limitations, inspired by the adaptive dependency matrix (Wu et al., 2019; Chen et al., 2021), we learn the normalized selfadaptive adjacency matrix S
= { s uv } N × N with the pre-deﬁned “cost” staying in the same node based on the learnable node embedding E φ = ( e 1 ,φ , e 2 ,φ , . . .
, e N,φ ) (
cid:62) ∈ R N × d c as
[Equation]
Here hyperparameter d uu is the “cost” of staying in the same node u and ReLU ( · ) = max (0 , · ) is an activation function, which guarantees s uv ≥ 0 .
Since (cid:80) v s uv = 1 and s uv ≥ 0 , we can use S as the normalized Laplacian.
Towards more effective and robust learning of both spatial and spectral characteristics, we represent the graph diffusion as a matrix power series.
That is, let ˜ S be an N × N × K Laplacian tensor containing the power series { I , S , . .
, S K − 1 } of S , where
I ∈ R N × N is the identity matrix and K ≥ 2 .
Spatio-Temporal Feature Transformation Note that spatial and temporal domains contain interdependent but highly heterogeneous types of information.
Inspired by statistical factor analysis, we facilitate signal extraction from these disparate informational sources, by mapping the original input feature space into the high-level latent feature space, which can be written as follows
[Equation]
where Θ ( (cid:96) )
FT ∈ R d c × P × Q FT and U ∈ R T represent the learnable parameters.
4. S UPRAGRAPH C ONVOLUTIONAL M ODULE IN M ULTILAYER S UPRA G RAPH
We propose a novel supragraph convolutional module to simultaneously capture spatio-temporal dependencies in dynamic networks.
Our key approach is (i) to represent a sequence of time-varying graphs recorded over a sliding time window, as a multilayer supra graph , instead of treating each graph snapshot individually, and (ii) to learn the resulting multilayer supra graph with the random walk exploration which allows us to encode the key details of the time-conditioned relationships among nodes and to boost graph convolutions over multiple edge sets.
Sliding Window Historical Data as Multilayer Supra Graph Network To increase expressive capability of spatio-temporal representation learning, we propose a novel directed multilayer supra graph.
Particularly, we treat each graph within a sliding window G T = {G t −T , G t −T +1 , . . .
, G t − 1 } ∈ R N × N ×T as a layer within a directed T -layered network.
(Here
N is the number of nodes in each graph.)
Since information in real world spatio-temporal processes can propagate only from past to present to future, we consider directed multilayer supra graph as abstraction for dynamic knowledge representation.
That is, we assume that the information (e.g., spatial features) is shared between layers t a and t b , whenever t b > t a , and time stamps t a , t b ∈ { t − T , . . .
, t − 1 } .
More speciﬁcally, we propose a strategy to create such kind of information propagation channels by adding directed virtual edges (i.e., interlayer edges) between every node in the layer t a and its counterparts in other “future” layers t b ( t b > t a ).
Deﬁnition 4.1.
A directed multilayer supra graph (DMSG) is a tuple, deﬁned as: DMSG = (cid:0)
G t −T , . . .
, G t − 1 , IM t a t b (cid:1) , where G t a =
( V t a , E t a ) , t a ∈ { t − T , . . .
, t − 1 } are network layers and
IM t a t b (Identity Mapping) is an N × N matrix of node mappings, with IM t a t b ij : v t a i × v t b j (cid:55)→ [0 , 1] .
That is, IM t a t b ii is the identity mapping between node v i in the “past” layer t a and the ”future” layer t b ( t b > t a ): IM t a t b ii = 1 .
The corresponding ( r -th power) supra-adjacency matrix W = { w uv } N T × N T for DMSG is then w abuv =
  ( s t a t a uv ) r , t a = t b and | u − v | mod N (cid:54) = 0 , ( d t a t a uu ) r , t a = t b and | u − v | mod N = 0 , d t a t b uu , t a (cid:54) =
t b and | u − v | mod N = 0 , (2) where ( s t a t a uv ) r is the r -th power of S t a t a which encodes the r -th step of a random walk among nodes u and v in layer t a , ( d t a t a uu ) r is the “cost” of staying in the same node u and in the same layer t a after r random walk steps, and d t a t b uu is the “cost” of jumping from the current node u in layer t a to node u in layer t b .
Finally, the generalized ( r -th power) supra-Laplacian L Sup ∈
R N T × N T for
DMSG is then
[Equation]
Supragraph Diffusion Convolutional Layer Armed with the ( r -th power) supra-Laplacian in (3), we now build a supragraph diffusion convolutional layer to encode both the graph structure and node features from the temporal domain.
That is, embedded node features are updated by message passing and aggregation via intralayer and interlayer diffusion.
The supragraph diffusion convolutional layer is formulated as
[Equation]
where H ( (cid:96) )
i,Sup ∈ R N T × P and H ( (cid:96) +1) i,Sup ∈ R N T × Q Sup are the input and output activations for layer (cid:96) ,
H (0) i,Sup = X T , and Θ ( (cid:96) )
Sup ∈ R d c × P × Q Sup are the learnable parameters.
In this case, the diffusion on layer t a can extend to a fraction of nodes and propagate information of layer t b by interaction.
4. S PATIAL G RAPH C ONVOLUTIONAL L AYER
To aggregate the features of each node with its multi-hops neighbourhoods to generate node embeddings, we now turn to constructing the spatial graph convolution via
[Equation]
[Equation]
where Θ ( (cid:96) )
Spa ∈ R d c × K × P × Q Spa is the trainable weight, ˜ S is the Laplacian tensor, H ( (cid:96) )
i,Spa ∈ R N × P and H ( (cid:96) +1)
i,Spa ∈ R N × Q Spa are the node representations at the (cid:96) -th layer and ( (cid:96) + 1) -th layer, respectively, and H (0)
i,Spa = X
i , i.e., the node features at time step i .
To reduce computational costs, we use weight sharing matrix factorization instead of matrix entry assignments.
As such, we timely update the latest state of variables with back propagation algorithm and reduce the risk of over-ﬁtting.
4. L EARNING T IME -A WARE M ULTIPERSISTENCE R EPRESENTATION OF T OPOLOGICAL F EATURES
To fully utilize information delivered by the MP representation mechanism, we propose a CNNinspired Time-Aware Multipersistence Euler-Poincar ´ e Surface Representation Learning (DEPSRL) module which employs the CNN base model f θ (including convolutional and pooling layers) and readout layer.
In summary, the proposed DEPSRL module offers the following unique innovations: (i) extracting features from the global information, (ii) learning the relationship from DEPS sequences pre and post (i.e., treating the ﬁxed-size sliding window as multi-channels), and (iii) aggregating topological features to make a ﬁxed size representation.
The summarized output feature of DEPSRL module is given by
[Equation]
where f GAP ( · ) , f GMP ( · ) , and f θ j , j = { 1 , 2 } , are global average pooling, global max pooling, and j -th CNN based model, respectively, and ⊕ denotes concatenation, where the f GAP ( · ) generates summarized feature for each channel.
We ﬁnd that f GMP ( · ) strengthens the representation learning of the time-conditioned multipersistence features.
The matrix Φ ( (cid:96) +1)
i,TAMP ∈ R Q TAMP is the output of the DEPSRL module.
To further learn and fuse multiple latent representations form different views (i.e., spatial information, spatio-temporal correlations, and persistent topological features), we combine these embeddings to obtain the ﬁnal embedding
[Equation]
) i,FT ) .
Here Υ ( (cid:96) +1)
i =
(1 / T (cid:80) ˜
H ( (cid:96) +1)
i,Sup ) Φ ( (cid:96) +1)
i,TAMP integrates global topological and global spatio-temporal information (i.e., we reshape H ( (cid:96) +1) i,Sup ∈ R N T × Q Sup to ˜ H
( (cid:96) +1)
i,Sup ∈ R N ×T × Q Sup and then average ˜ H
( (cid:96) +1)
i,Sup over temporal dimension), F ( · , · , · ) is a dimension-wise concatenation function on embeddings along the output dimension, and (cid:80) i ∈{ Spa , Sup , FT } Q i = Q out .
DEPSRL also allows for input of E from different biﬁltrations.
For more details, please refer to Appendix D.2.
4. M ODELING S PATIO -T EMPORAL D YNAMICS
To encode spatio-temporal correlations among time series and get hidden state of nodes at a future timestamp, we put the ﬁnal embedding Z ( (cid:96) +1)
i,out into Gated Recurrent Units (GRU).
The forward propagation equations of GRU are as follows
[Equation]
where ς ( · ) is an activation function which is ReLU in our case, (cid:12) is the elementwise product, (cid:60) i is the update gate and (cid:61) i is the reset gate.
b (cid:60) , b (cid:61) , b ˜Ξ , W (cid:60) , W (cid:61) , and W ˜Ξ are learnable parameters.
[ Ξ i − 1 , Z i,out ] and Ξ i are the input and output of GRU model, respectively.
Now we get
˜ Ξ i containing the structure, spatio-temporal, and topological information.
dependencies are passed into GRU layer (see Eq.
8) for multi-step forecasting.
Note that the whole model can be trained in an end-to-end fashion.
E XPERIMENTS
Data Description We use three different types of datasets to examine the performance of the proposed TAMP-S2GCNets on dynamic networks: (i) The transportation and trafﬁc data of California state from the freeway Performance Measurement System (PeMS).
We use three well-studied trafﬁc networks from literature: PeMSD3, PeMSD4 and PeMSD8 (Chen et al., 2001), (ii) Digital transactions between users in the Ethereum blockchain network.
We extract dynamic networks from three token assets: Bytom, Decentraland and Golem (Li et al., 2020), and (iii) The spread of coronavirus disease COVID-19 at county-level in states of California (CA) and Texas (TX).
More details on each dataset and parameter settings are provided in Appendix D.1.
Experimental Settings We compare the presented TAMP-S2GCNets with 13 state-of-the-art baselines (see more details of baselines in Appendix D.2) on three evaluation metrics, i.e, Mean Absolute
Error (MAE), Root Mean Squared Error (RMSE), and Mean Absolute Percentage Error (MAPE).
For transportation networks, following Cao et al.
(2020), we (i) split PeMSD3, PeMSD4, and
PeMSD8 datasets into a training set (60%), validation set (20%), and test set (20%) in chronological order and (ii) use the one hour historical data to the next 15 minutes data.
Following Chen et al.
(2021), we (i) split Bytom, Decentraland, and Golem token networks into training set (80%) and test set (20%) in chronological order and (ii) use 7 days historical data to predict future 7 days data.
For
COVID-19 datasets, in our experiments, we use daily data of 11 months of 2020, from February 1 to
December 31, and split the graph signals into training set, ﬁrst 80% of days (268 days), and test set, last 20% of days (67 days).
Further speciﬁcs on experimental setup are provided in Appendix D.2.
The best results are in bold font and the results with dotted underline are the best performance achieved by the runner-ups.
We also perform a one-sided two-sample t -test between the best result and the best performance achieved by the runner-up, where *, **, *** denote signiﬁcant, statistically signiﬁcant, highly statistically signiﬁcant results, respectively.
More detailed experiments on all datasets are in
Appendix E.
The data and code implementation are available at https://www. dropbox.com/sh/n0ajd5l0tdeyb80/AABGn-ejfV1YtRwjf_L0AOsNa?dl=0 .
Computational Complexity
Although, in general, MP is computationally costly, in our case, to get the DEPS summary, we do not need to compute PDs as in other MP approaches, but only
Betti numbers of each ﬁltration cell.
In particular, while a computational cost for obtaining k thPD for a graph is O ( M 3 k ) , where M k is the number k -simplices
(Otter et al., 2017), obtaining
Euler Characteristics by sparse matrix methods has computational complexity of O ( M 0 + M 1 +
M 2 ) (Edelsbrunner & Parsa, 2014).
5. E XPERIMENTAL R ESULTS
Transportation Trafﬁc Flow Table 1 shows the performance comparison among 13 state-of-the-art baselines on PeMSD3, PeMSD4, and PeMSD8 for multi-step trafﬁc ﬂow forecasting.
Our TAMPS2GCNets consistently outperforms baseline models on all 3 datasets except for PeMSD3, which underscore effectiveness of TAMP-S2GCNets in time series forecasting tasks.
On PeMSD3, our
TAMP-S2GCNets achieves the best MAE and MAPE, and has an average 2.75% relative gain, compared to the runner-up.
The average relative gains of TAMP-S2GCNets over the runner-ups in MAE and MAPE are 2.78% and 2.19% on the 3 datasets, respectively.
Ethereum Blockchain Prices Table 3 ( left ) summarizes the comparison results in MAPE on
Ethereum token networks (i.e., Bytom, Decentrland, and Golen).
Table 3 indicates that TAMPS2GCNets is always better than baselines for all dynamic Ethereum networks.
We ﬁnd that, even compared to the baseline which also integrates topological information (i [Table] e., Z-GCNETs), TAMPS2GCNets is highly competitive, implying that the time-aware MP representation with DEPS is capable of better encoding time-conditioned information than the zigzag idea based on one ﬁltration.
COVID-
Hospitalizations Table 2 shows results on 15-day ahead forecasting of COVID-19 hospitalizations in the U.S. states of California (CA) and Texas (TX) at a county level basis (RMSE is aggregated over each state).
For the sake of room, here we only display the top runner models.
TAMP-S2GCNets yields signiﬁcantly better forecasting performance, with relative gains of 1.5%- 24.5% (in CA) and 5.5%-46.7% (in TX)
[Table]
[Table]
5. A BLATION S TUDIES
Contribution of TAMP-S2GCNets components We now conduct the ablation studies on PeMSD4 and PeMSD8 to evaluate contribution of different components of our framework.
(Ablation results on Golem and CA are in Table 9 in Appendix E.)
[Table]
We compare our TAMP-S2GCNets withablated variants, i.e., (i) TAMP-S2GCNets without DEPSRL module (w/o DEPSRL module), (ii) TAMP-S2GCNets without spatial graph convolutional layer (w/o
GCN Spa ), (iii) TAMP-S2GCNets without supragraph convolutional module (w/o
GCN Sup ), and (iv) TAMP-S2GCNets without FT (w/o FT).
As Table 3 ( right panel ) shows, ablating each of above causes the performance drops sharply in comparison with our full TAMP-S2GCNets model.
Especially, on
PeMSD4, the supragraph convolutional module signiﬁcantly improves the results as it simultaneously captures the spatial and temporal information.
In addition, it is evident that DEPSRL module enhances the topological information learning ability in spatio-temporal domain, i.e., TAMPS2GCNets outperforms TAMP-S2GCNets w/o DEPSRL module with an average relative gain 3.92% on RMSE over PeMSD4 and PeMSD8.
Besides, as expected, taking the advantages of
GCN Spa and FT, we improve the performance via capturing the graph structure in the spatial dimension and consolidating the processing of spatio-temporal correlations between node attributes, respectively.
[Table]
[Table]
Contribution of Single vs. Multiﬁltration Persistence We consider sublevel ﬁltrations based on node degree ( Deg ), betweenness ( Btwns ), and graph diameter (power ﬁltration) with edge weights induced by number of transactions ( Power Trns ) and volume of transactions ( Power Vol ).
As Table( left ) shows, MP always outperforms single ﬁltrations by a signiﬁcant margin, both in terms of prediction accuracy and variability.
The results conﬁrm our premise that MP is able (i) to capture hidden time dependencies of high dimensional time-varying objects which are inaccessible with one-parameter PH and (ii) to yield substantially more stable feature maps in dynamic scenarios.
Contribution of DEPS vs. the Existing MP Summaries
Finally, we evaluate performance of the new time-aware MP summary, i.e., DEPS, in comparison to the MP summaries based on the slicing argument, namely, MP-I of Carri`ere & Blumberg (2020) and MP-L of Vipond (2020) as representation input to TAMP-S2GCNets.
As Table 4 ( right panel ) shows, comparing to MP-I and
MP-L, DEPS yields 9%-12% improvement in MAPE, and at least a 10 times decrease in computational time (29.84s for DEPS vs. 401.80s for MP-I and 517.11s for MP-L).
Such substantial differences in computational costs are explained by the need of the slicing-based MP summaries to search for the most suitable one-parameter PH representation of MP.
In turn, DEPS is based on a point-wise representation argument in linear algebra, and its high computational efﬁciency makes
DEPS the preferred choice for time-conditioned topological representation learning (see the running time in Table 4).
Since computational costs remain the major roadblock for MP applications, our
DEPS approach shows that MP tools based on scalable sparse matrix algorithms appear to be the most promising direction for integration of MP into machine learning tasks.
C ONCLUSION
We have explored utility of MP to enhance knowledge representation mechanisms within the timeaware DL paradigm.
The developed TAMP-S2GCNets model is shown to yield highly competitive forecasting performance on a wide range of datasets, with much lower computational costs.
In the future we plan to explore combination of MP with the zigzag persistence and to investigate asymptotic distributional properties of point-wise MP invariants.
E THICS S TATEMENT
We do not anticipate any negative ethical implications of the proposed methodology.
In turn, we believe that introduction of GCNs tools, coupled with multiparameter topological approaches, into biosurveillance opens a new path for more accurate, timely, and robust tracking of infectious diseases with high virulence such as COVID-19.
In particular, multiparameter persistence allows for extracting the most salient topological features along multiple geometric dimensions and, as such, can be especially valuable to address disease dynamics as a function of multiple highly heterogeneous variables, e.g., socio-demographic, socio-environmental, and socio-economic factors.
In turn,
GCNs and, more generally, geometric deep learning allow for capturing sophisticated nonlinear spatio-temporal dependencies among factors which contribute to the disease spread.
Hence, we postulate that in the next few years we can expect to see a new set of spatio-temporal biosurveillance artiﬁcial intelligence algorithms, based on a combination of geometric deep learning, multiparameter persistence, and more generally, tools of topological data analysis.
R EPRODUCIBILITY S TATEMENT
The source code for the experiments can be accessed under https://github.com/ tamps2gcnets/TAMP_S2GCNets.git . A CKNOWLEDGEMENTS This work is sponsored by the National Science Foundation under award numbers ECCS 2039701, INTERN supplement for ECCS 1824716, DMS 1925346 and the Department of the Navy, Ofﬁce of
Naval Research under ONR award number N00014-21-1-2530. Part of this material is also based upon work supported by (while serving at) the National Science Foundation. Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the National Science Foundation and/or the Ofﬁce of Naval Research. The authors are also grateful to the ICLR anonymous reviewers for many insightful suggestions and engaging discussion which improved clarity of the manuscript and highlighted new open research directions.
A N OTATION
Frequently used notation is summarized in Table
[Table]
[Table]
B S INGLE P ARAMETER P ERSISTENT H OMOLOGY AND I TS S UMMARIES
All extracted topological features can be then summarized as a multiset in R 2 , called persistence diagram (PD) :
D k ( C ) =
{ ( b i , d i ) ∈
R 2 | b i <
d i }
[Table]
where birth b i and death d i mark the ﬁltration indexes at which the k -dimensional topological feature
ρ i appears and disappears, respectively.
The farther ( b i , d i ) from the diagonal is (that is, the more persistent ρ i is), the likelier ρ i is to contain salient information on G t .
Multiplicity of ( x b , y d ) ∈ D is the number of p -dimensional topological features ( p -holes) that are born and die at x b and y b , respectively.
Features on the diagonal of D have inﬁnite multiplicity.
Another summary which may be particularly suitable in our context of modeling noisy time-evolving objects is the Euler-Poincar ´ e Characteristics (see Adler et al.
; Baryshnikov & Ghrist (2010) on the Euler calculus and its applications, particular, in conjunction with sensor networks).
Deﬁnition B.1 (Euler-Poincar´e Characteristics) .
For a given simplicial complex C , Euler-Poincar´e
Characteristics χ
( C ) is deﬁned as the alternating sum of the number of k -simplices of C .
That is, if n k is the number of k -simplices in C , then χ ( C ) =
(cid:80) Mk =0 ( − 1) k n k , where M is the dimension of highest dimensional simplex in C .
Note that Euler-Poincar
´
e Characteristics is homotopy invariant, and an alternative deﬁnition is χ ( C )
= (cid:80) Mk =0 ( − 1)
k B k ( C ) where B k ( C ) is the k th Betti number of C , i.e. rank of the k th homology group H k ( C )
(Ghrist, 2018; Hatcher, 2002).
C P ROOF OF T HEOREM 3.
In this section, we provide the proof of the stability theorem (Theorem 3.1).
First, let us recall the key notations we use.
Let G +
and G − be two graphs and let C ± be the clique complexes of G ± (Ghrist, 2018).
Let F =
( f, g ) be a multivariate ﬁlter function, i.e. F : V ±
→ R 2 , where V ± is the set of nodes of G ± , and let I =
{ ( α i , β j ) | 1 ≤ i ≤ m, 1 ≤ j ≤ n } be the corresponding thresholds for F = ( f, g ) .
Next, we deﬁne Wasserstein-
p distance among PDs, i.e., the important concept we borrow from the theory of single parameter persistence (Edelsbrunner & Harer, 2010).
Deﬁnition C.1 (Wasserstein- p distance) .
Let ( C ± , f ± , I ± ) be two single parameter ﬁltrations, and
D k ( C ± ) be the corresponding PDs for k -cycles (i.e, k -dimensional topological features).
Let q + j = ( b + j , d + j
) ∈
D k ( C + ) be the birth and death times of a k -dimensional topological feature ρ j .
Then,
Wasserstein-
p distance between D k ( C + ) and D k ( C − ) is deﬁned as
[Equation]
(9) where φ : D k ( C + ) ∪ ∆ → D k ( C − ) ∪ ∆ is a bijection (matching) and ∆ is the diagonal set, i.e., ∆ = { ( b, b ) | b ∈ R } , which contains k -cycles of inﬁnite multiplicity.
With ∆ in both sides, we ensure the existence of these bijections even if the cardinalities |{ q + j }| and |{ q − l }| are different.
When p = ∞ , (9) corresponds to the bottleneck distance, i.e. W ∞ .
In the proof below, we use p = 1 , i.e. W 1 -distance.
Armed the PD and Wasserstein- p distance concepts, we now turn to the main stability result for the new MP summary.
Let (cid:98) C ±
=
{ C ± ij } be the biﬁltration associated with ( C ± , F, I ) , i.e. C ± ij is the clique complex of the induced subgraph G ± ij =
F − 1 (( −∞ , α i ] × ( −∞ , β j ]) .
(Figure 2 illustrates such biﬁltration on graphs with degree and eccentricity functions.)
Let E ± be the the corresponding
Euler-Poincar ´ e Surface, i.e. E ± is an m × n -matrix with entries E ± ij =
χ ( C ± ij , ) , where χ ( . ) is the Euler Characteristics (See Figure 2), and D ( C + , C − ) be the weak L 1 -distance as deﬁned in
Section 3 of the paper (see also Remark C.1 below for the motivation to use this new metric).
Proof of Theorem 3.1: We start from providing the outline of the proof.
Notice that Euler-Poincar ´ e
Surfaces is the alternating sum of Betti numbers {B k } .
Hence, the distance (cid:107) E + − E − (cid:107) 1 , 1 is bounded by the difference of the corresponding Betti numbers for (cid:98) C ± , i.e. Eqs.
(10) and (11).
By ﬁxing i and k , and by taking column distance D c into consideration, our main assertion
[Equation]
[Equation]
then reduces to Eqs.
(12) and (13).
To relate the difference |B ( C + j ) − B ( C − j ) | to
W 1 ( D ( C + ) , D ( C − )) , we observe that the Betti number B ( C ± j ) is the sum of the number of barcodes in D ( C ± ) , containing β j , i.e., # { r | β j ∈ [ b ± r , d ± r ) }
(Eqs.
(13) and (14)).
Yet, the differences of the number of barcodes containing β j for C + and C − can be bounded by the Wasserstein- 1 distances of D ( C + ) and D ( C − ) , i.e. Eqs.
(15) and (16).
Aggregating these terms on indices i and k ﬁnishes the proof.
Now, we turn to derivations in details.
Recall that χ ( C ) =
(cid:80) M k
=
( − 1) k B k ( C ) , where B k ( C ) is the k th Betti number of C and M is the maximum homological dimension of C .
Then, since
E ± ij = χ
( C ± ij ) , we have E ± ij = (cid:80) Mk =
0 B k ( C ± ij ) .
Hence, for any i, j , |
[Equation]
Since (cid:107) E + − E − (cid:107) 1 , 1 = (cid:80) mi =1 (cid:80) nj =1 | E + ij − E − ij | , we have
[Equation]
Now, to complete the proof, we need to bound the latter sum by the MP distance D ( (cid:98) C + , (cid:98) C − ) .
Recall that by Deﬁnition 3.
[Equation]
Then, by Eq. 11, to prove the theorem, it is enough to show that for any 0 ≤ k 0 ≤ M and for any 1 ≤ i 0 ≤ m ,
[Equation]
For simplicity, without loss of generality, we ﬁx 1 ≤ i 0 ≤ m and 0 ≤ k 0 ≤ N .
Also, for sake of notations, we drop the subscripts and superscripts, i.e C ± =
C ±
i 0 ∗ and C ± j = C ± i 0 j , D ( C ± ) =
D gk 0 ( C ± i 0 ∗ ) and B ( . )
=
B k 0 ( . ) .
With these simpliﬁed notations, Eq. 12 is equivalent to
[Equation]
To show (13), we need to evaluate the relation between the Betti numbers B ( C ± j ) and the persistence diagrams D ( C ± ) .
Consider D ( C ) = { ( b r , d r ) | 1 ≤ r ≤ Q } , where b r represents the birth time of a k 0 -cycle ρ r in C , while d r represents the death time of ρ r .
In particular, ρ r induces a barcode [ b r , d r ) .
For each such ρ r , deﬁne indicator function ψ r ( t ) =
I [ b r ,d r ) ( t ) , i.e. ψ r ( t ) = 1 for t ∈ [ b r , d r ) and ψ r ( t ) =
otherwise.
Then, the Betti function of C is deﬁned as B : [ β 1 , ∞ )
→ N such that B ( t ) = (cid:80)
Qr =1 ψ r ( t ) for t ∈ [ β 1 , ∞ ) .
(Here, β 1 is the lowest threshold for the ﬁltering function g in the set of thresholds I =
{ ( α i , β j ) | 1 ≤ i ≤ m, 1 ≤ j ≤ n } .
Similarly, if we consider row sum D r instead of column sum D c , then we would use the other ﬁltering function f , and the
Betti functions would be deﬁned as B : [ α 1 , ∞ )
→ N .)
Hence,
[Equation]
As a result, we get
[Equation]
(cid:12) (cid:12)(cid:12) (cid:12)
Q + (cid:88) r = ψ + r (
[Equation]
j ) − Q − (cid:88) r = ψ − r (
[Equation]
j ) (cid:12) (cid:12)(cid:12) (cid:12) (14) ≤ n (cid:88) j = (cid:98) Q (cid:88) r =
[Equation]
, where φ : D ( C + ) → D ( C − ) is the best matching in the deﬁnition of W 1 ( D ( C + ) , D ( C − )) and (cid:98) Q = max { Q + , Q − } .
Notice that the contribution of each | ψ + r ( β j ) − ψ − φ ( r ) ( β j ) | to the sum at β j is less than the number (# r ) of intervals [ b + r , d + r ) (cid:52) [ b + φ ( r ) , b − φ ( r ) ) containing the interval [ β j , β j +1 ) , where (cid:52) is the symmetric difference of the sets.
Let K =
min j ( β j +1 − β j ) .
By dividing the length (cid:107) [ b + r , d + r ) (cid:52) [ b + φ ( r ) , b − φ ( r ) )
(cid:107) by K , we ensure that every such interval contributes at least 1 to the
Betti number count.
Hence, we arrive at the following key inequality:
[Equation]
Finally, since (cid:98)
[Equation]
we obtain
[Equation]
Since (16) holds for any 1 ≤ i 0 ≤ m and 0 ≤ k 0 ≤ N , for C = 2 /K
we get
[Equation]
By Eq.
(11), this implies
[Equation]
which completes the proof.
[Equation]
Remark C.
(On the Choice of the Metric).
While interleaving and matching distances are very common and useful for multipersistence in various cases (Thomas, 2019), they are not suitable to study the Euler-Poincar ´ e Surfaces.
The reason for this is that matching and interleaving distances are based on L ∞ metrics, and do not see the quantity of generators of small interleaving distances.
However, Euler-Poincar ´ e Surfaces by deﬁnition depends on the rank of the homology groups, and the quantity of the generators (Betti numbers) are crucial in their deﬁnition even if they exhibit a very short lifespan.
In particular, L ∞ -based metrics ﬁnd the maximum distance between the essential generators, but such metrics fail to capture the quantity of small generators.
That is, while the interleaving or matching distance between the multipersistence modules of the biﬁltrations { (cid:98) C + } and { (cid:98) C − } is small, there might be a very large distance between the induced Euler-Poincar ´
e Surfaces E + and E − .
In turn, L p -based distances are successful to capture the rank even if they are small (noise) generators, while interleaving and matching distances are not very sensitive to these.
(See Thomas (2019, Section 4.2) for further discussion.)
This is why we introduce the weak Lmetric above which naturally suits to study the Euler-Poincar ´ e Surfaces and similar vectorizations in the theory.
Furthermore, such weak L 1 metric can be considered a straightforward generalization of L 1 -based norm for multiparameter persistence.
Remark C.2 (Implications of the Stability Result).
Note that our stability result (Theorem 3.1) directly implies that the distances between multiparameter PDs control the distance between the resulting Euler-Poincar ´ e Surfaces.
By combining with the stability result for PDs (Cohen-Steiner et al., 2007), one can conclude that the small changes in the MP ﬁltering function F : V t (cid:55)→ R d or the small changes in the input data can result only in a small change in DEPS surfaces.
In particular, by Cohen-Steiner et al.
(2007), for single parameter persistence, we have
[Equation]
where D f is the persistence diagram for ﬁltering function f , and d ( f, g ) is the distance between the functions f and g .
The way we deﬁne our weak L 1 -metric enables us to use this single persistence result in this context as follows.
The column distance D c is the sum of single PD for each column.
Each column, (say j th column) is represented by a single variable ﬁltering function F ( t, β j ) .
Then, the changes in F ( t, β j ) reﬂects in the column distance for PDs corresponding to j th -column.
This means if we have a small change in F ( t, β j ) for each j , then the column distance for MP will be small.
So is the row distance induced by F ( α i , s ) .
Similarly, if we have two MP ﬁltering functions F ± : V → R 2 , they will induce two ﬁltrations C ± .
When d ( F + , F − ) is small, then by Cohen-Steiner et al.
(2007), D ( (cid:98) C
+ , (cid:98) C − ) is small.
When we apply our stability theorem to these ﬁltrations, we see that the distance between the induced DEPS surfaces (cid:107) E + − E − (cid:107) 1 , 1 will be small, too.
Since small change in the data can be converted to small change in the ﬁltering functions on the same data, the result can also be interpreted as the small change in the data results in small change in the induced DEPS surfaces.
Remark C.3 (DEPS through the Probabilistic Perspective).
Furthermore, the new time-aware
MP summary can also be assessed from a probabilistic perspective, if we view a multivariate time series X as real-valued random process indexed by time t and deﬁned on a probability space (Σ , A , P ) .
For 1 ≤ J ≤ L ≤ ∞ , deﬁne the σ -ﬁeld (cid:122) LJ
⊂ A such that (cid:122) LJ = σ ( X k , J ≤ k ≤ L ( k ∈ Z )) , i.e.
(cid:122) LJ is a σ -ﬁeld generated by X J , X J +1 , . . .
, X L .
Then, E t is a real-valued (cid:122) t 1 -measurable random matrix and DEPS { E t } Tt =1 is F -measurable random matrix, where F = ( (cid:122) k 1 ) k> 0 is a ﬁltration of σ -ﬁelds indexed over time.
Given the results of Thoppe et al.
on weak convergence of the
Euler characteristic χ ( C ) for dynamic Erd˝os-R´enyi graphs to the Ornstein-Uhlenbeck process, this opens a potential path to establish asymptotic distribution of E t .
Remark C.4 (DEPS for Higher Dimensions).
DEPS can be naturally expanded to a case when we consider ﬁltrations along more than two geometric dimensions.
In such a case, instead of a m × nmatrix, DEPS will be a tensor.
Our stability result (i.e., Theorem 3.1) verbatim extends to such a case.
For each column, the top graph is ﬁltered by eccentricity function.
Each cell includes Betti numbers B 0 and B 1 .
Since Band B 2 are 0 for each cell, we get E ij = B 0 .
D R EPRODUCIBILITY D. D ATASETS
We use three different datasets, from transportation, Ethereum blockchain, and biosurveillance domains, to examine the performance of the proposed TAMP-S2GCNets methodology:
Transportation: The transportation and trafﬁc data for the U.S. State of California, from the freeway Performance Measurement System (PeMS) data sources (i.e., PeMSD3, PeMSD4, PeMSD8) (Chen et al., 2001), naturally produces a dynamic network where each node is a loop detector and each edge represents a freeway between two nodes.
We extract aggregated data to 5 minutes, thus generating dynamic networks with 26,208, 16,992 and 17,856 nets in PeMSD3, PeMSD4 and PeMSD8 datasets, respectively.
To capture spatio-temporal dependencies, we rebuild the trafﬁc graph structure, at time t , and compute edge weights w t,uv , between pair nodes ( u, v ) , via the Radial Basis Function w t,uv = e −|| x t,u − x t,v || 2 /γ ; where γ = 1 . 0 and each node x t, · has features speed, occupancy and ﬂow rate.
To investigate the dynamic of the trafﬁc graph topology, we restrict the ﬁnal graphs to only keep edges whose are below or equal to threshold α = 0 .
Blockchain: Ethereum blockchain data contain token assets which naturally represents network layers of the Ethereum network (di Angelo & Salzer, 2020; Web, e), in which nodes and edges are addresses of users and digital transactions, respectively.
Using the publicly available blockchain, we extract dynamic networks from three tokens with market value above $100,000,000: Bytom, Decentraland and Golem.
In our experiments, we extract dynamic networks by daily transactions and use daily closed prices (Web, d), hence, creating a graph/net for each day.
Since each token is created at different date, in all cases last day is 05/07/2018, the dynamic networks vary in the number of nets: Bytom (285 nets), Decentraland (206 nets) and Golem (443 nets).
For the sake of reasonable computation time, we reduce the original net size using the maximum weight subgraph approximation method of Vassilevska et al.
(2006) from more than 400,000 nodes, average of 120,000 edges, to 100 nodes using number of transactions and price-volume as edge weights.
Biosurveillance: SARS-CoV-2, which is the virus that causes COVID-19, has spread around every place in the world.
To analyze the progression of coronavirus disease we create a dynamic network based on daily records on COVID-19 cases and hospitalizations from the CovidActNow project (Web, c) and Johns Hopkins University (Web, f; Dong et al., 2020), and population numbers from the U.S. Census Bureau (Web, b).
We choose data at county level from California and Texas states as our cases of study, and focus on forecasting the number of hospitalized patients.
Similarly to (i) we rebuild the coronavirus-spread graph structure, at day-time t , based on the county adjacency graph (Web, a) and Radial Basis Function between pair of nodes.
We build a dynamic network for each state, each dynamic network contains 335 nets.
The number of conﬁrmed COVID-19 cases and county population size serves as node features, and ﬁnal graphs only keep edges whose are below or equal to the threshold α = 0 .
More details on experimental settings and ﬁltering functions are provided in Section D.2.
Tablesummarize the characteristics of all datasets used in our experiments.
[Table]
D. E XPERIMENTAL S ETTINGS
We implement our TAMP-S2GCNets with Pytorch framework on NVIDIA GeForce RTX
GPU.
Further, for all datasets, TAMP-S2GCNets is trained end-to-end by using the Adam optimizer with a L1 loss function.
The tuning of our proposed TAMP-S2GCNets on each dataset is done via grid search over a ﬁxed set of choices and the same cross validation setup is used to tune baselines.
We compare TAMP-S2GCNets with 13 types of state-of-the-art methods, including
FC
, TCN ( et , AGCRN ( ), Z GCNETs (Chen et
and StemGNN (Cao
MODEL SPECIFICATIONS
For PeMSD3 , TAMP-S2GCNets consists of 2 layers whose hidden feature dimension is 64.
The learning rate is 5 e - 3 and the batch size is set as 64.
The “cost” d t a t a uu of staying in the same node u (in the layer t a ) and the “cost” d t a t b uu of jumping from the current node u in layer t a to node u in layer t b are set to be 0.1 and 0.2, respectively.
For PeMSD4 , TAMP-S2GCNets consists of 2 layers whose hidden feature dimension is 128.
The learning rate is 5 e - 3 and the batch size is set as 64.
The “cost” d t a t a uu of staying in the same node u (in the layer t a ) and the “cost” d t a t b uu of jumping from the current node u in layer t a to node u in layer t b are set to be 0.1 and 0.2, respectively.
For PeMSD8 , TAMP-S2GCNets consists of 2 layers whose hidden feature dimension is 64.
The learning rate is 2 e - 3 and the batch size is set to 16.
The “cost” d t a t a uu of staying in the same node u (in the layer t a ) and the “cost” d t a t b uu of jumping from the current node u in layer t a to node u in layer t b are set to be 0.1 and 0, respectively.
The dimension of node embedding d c (in E φ ) are(PeMSD3), 10 (PeMSD4), and 20 (PeMSD8), respectively.
We set the dimension K of Laplacian tensor ˜ S as 2 for all transportation networks.
For all Ethereum token networks (i.e., Bytom , Decentraland , and Golem ), TAMP-S2GCNets consists of 2 layers whose hidden feature dimension is 32.
The ﬁnal hyperparameter setting is learning rate of 1 e - 2 and batch size of 8.
The “cost” d t a t a uu of staying in the same node u (in the layer t a ) and the “cost” d t a t b uu of jumping from the current node u in layer t a to node u in layer t b are set to be 0.01 and 0.01, respectively.
In addition, we set the embedding dimension d c in E φ as 1 for all Ethereum token networks.
We set the dimension K of Laplacian tensor ˜ S as 2, 3, 3 for Bytom,
Decentraland, and Golem, respectively.
For COVID-19 hospitalizations in CA and TX , TAMP-S2GCNets consists of 2 layers whose hidden feature dimension is 128.
The learning rate is set to 1 e - 1 and 1 e - 2 for CA and TX, respectively.
The batch size is set to 8, and the embedding dimension d c in E φ is 10.
The “cost” d t a t a uu of staying in the same node u (in the layer t a ) and the “cost” d t a t b uu of jumping from the current node u in layer t a to node u in layer t b are set to be 0.1 and 0.2, respectively.
We set the dimension K of Laplacian tensor ˜ S as 2 for both CA and TX.
ALTERNATIVE OPTIONS FOR DEPS WITHIN DEPSRL MODULE
In our experiments, for all datasets, the CNN based model (i.e., f θ j where j = { 1 , 2 } in Time-Aware Multipersistence
Euler-Poincar´e Surface Representation Learning (DEPSRL) module) consists of 2 CNN layers.
For transportation networks and COVID, The ﬁlter size, kernel size, and stride is set to be 8, 2, 2 respectively.
For Ethereum token networks (i.e., Bytom, Decentraland, and Golem), the ﬁlter size, kernel size, and stride is set to be 8, 1, 2 respectively.
We set the size of global average pooling and global max pooling as 3 × 3 .
Table 7 shows the optimal output dimensions of spatial graph convolutional layer ( Q Spa ), supragraph diffusion convolutional layer ( Q Sup ), and spatio-temporal feature transformation ( Q FT ) on all datasets.
[Table]
Furthermore, in addition to the Time-Aware Multipersistence Euler-Poincar´e Surface Representation
Learning (DEPSRL) module (see Section 4.3 in our main paper):
[Equation]
we also have considered passing the aggregated DEPS as input to DEPSRL, that is, f θ j (¯ E ) , j = 1 , 2 , where ¯ E = 1 / T (cid:80) T i =1 ∈ R m × n E i .
As expected, DEPS { E i } T i =1 tends to yield better performance within the DEPSRL module than its aggregated counterpart ¯ E , which can be explained by lower loss of time-conditioned topological information within DEPS.
However, in the case of PeMSD4 dataset, we ﬁnd that ¯ E slightly improves the performance of TAMP-S2GCNets (i.e., the MAE of TAMP-S2GCNets based on the aggregated
DEPS is 17.58 and the MAE of TAMP-S2GCNets based on DEPS is 17.69), which is likely due to higher data heterogeneity of PeMSD4 vs. other transportation datasets.
In all other datasets, we report results based on DEPS.
COMPARISON TO OTHER MP SUMMARIES For fair comparison, computation of all MP representations (as shown in Table 11 and Table 4 ( right panel ) in our main paper) have been run on an 8-cores DO droplet machine with Intel Xeon Scalable processors at base frequency of 2.5 Ghz.
Our source codes are available for revision 1 .
Source codes for computation of MP-I and MP-L are freely available in Github 2 .
We follow the ﬁltering function recommendations for graphs, on both
MP-I and MP-L, from supplementary material of Carri ` ere & Blumberg (2020).
Further technical details on the 8-core machine are publicly available 3 .
ON TYPES OF MULTIFILTRATIONS
Good ﬁltrations should have rich range to get a ﬁne resolution, and it should reﬂect an important property relevant to the question at hand.
For multipersistence, for a good combination of two such ﬁltering functions, one should choose functions uncorrelated.
One approach is to choose one function from the domain of the question, and the other from strictly graph properties (e.g. degree, betweenness, etc.).
We assess shape characteristics of
G t via ﬁltering functions based on node degree ( Deg ), betweenness centrality ( Btwns ), and edge weights ( Power ).
Edge weights for each dataset are computed as described in D.1; i.e using transactions between addresses ( Trns ), volume of transactions ( Vol ), trafﬁc measurements ( Traffic ) and COVID-19 incidence ( Incidence ).
Our analysis focuses on multivariate ﬁltering function
F : V t (cid:55)→ R d with d = 2 .
For Deg and Btwns the set of nondecreasing thresholds runs on values over nodes, whilst for ﬁltering via power ﬁltration ( Power ) of G t nondecreasing thresholds runs on weighted paths between nodes.
Notice that each pair-combination of ﬁltering functions produces a speciﬁc MP representation in a form of the Euler-Poincar ´ e surface, which impacts the algorithm performance, as shown in our multiple experiments.
Table 12 presents the optimal multiﬁltration for each dataset.
E A DDITIONAL E XPERIMENTS AND R UNNING T IME
[Table]
Why MP Helps?
More Experiments To validate gains (if any) delivered by multiﬁltration compared to the single ﬁltration persistence and ensemble of multiple stacked together single parameter
[Table]
[Table]
ﬁltrations, we also report the results of single ﬁltration, ﬁltration ensemble, and multiﬁltration on
Decentraland (see Table 10).
(In case of a ﬁltration ensemble, we aggregate two single ﬁltrations and feed the ensemble representation into the model.)
We ﬁnd that MP substantially outperforms both the single ﬁltration persistence and the ﬁltration ensemble.
In addition, the MP results are generally more stable than ones based on the single ﬁltration and ﬁltration ensemble.
This result can be expected, as by construction of DEPS, each individual single parameter persistence corresponds either to a row or a column of DEPS, that is, we have only a one-dimensional ﬁngerprint of the data.
In contrast, in MP and its DEPS summary, we obtain a two-dimensional ﬁngerprint of the data, meaning any cell in the output simultaneously related to the all 8 neighboring cells (East,
West, Northeast, etc.).
As such, MP provides a 2D resolution of the data space and delivers much ﬁner information on the underlyng data topology.
[Table]
Table 11 shows the computational costs and performance of different MP summaries on Decentraland and Golem datasets.
We observe that our new time-aware MP summary DEPS not only outperforms the existing MP summaries (i
[Table]
e., MP-I and MP-L) but also induces substantially lower computational costs.
These ﬁndings demonstrate that our DEPS can effectively capture salient timeaware topological information with lower time and memory cost.
[Table]
COVID-19 Biosurveillance: More Experiments Finally, Figure 3 depicts the results on 15-day ahead forecasting of hospitalizations in California on a county level basis, using the top 3 models (i.e., AGCRN (Bai et al., 2020), StemGNN (Cao et al., 2020), and our TAMP-S2GCNets).
(Here, to differentiate among RMSEs on a county level basis, we omit the map for DCRNN (Li et al.,
[Figure]
2018) due to its lower predictive performance, relative to RMSE scale of AGCRN, StemGNN, and our TAMP-S2GCNets.)
Overall, TAMP-S2GCNets and StemGNN tend to perform similarly for less populated counties in Northern California, with TAMP-S2GCNets being slightly better than StemGNN over the Paciﬁc Coast and StemGNN yeilding some edge in the North East.
In turn, AGCRN tends to deliver more competitive performance in North-Eastern California, except of
Shasta county.
However, performance among models starts to differ drastically in more populated counties and, in particular, in Southern California where it is much harder to forecast COVID-19 hospitalizations due to higher population density and more disparate socio-economic factors.
Here, TAMP-S2GCNets
[Table]
substantially outperforms both StemGNN and AGCRN almost in all counties of Southern California, yielding predictive gains from 3.8% vs. StemGNN and 29.8% vs. AGCRN in San Bernardino to 57.7% vs. StemGNN and 55.7% vs. AGCRN in San Diego.
While all models perform very poorly in
Los Angeles county (i.e., one of the most populated counties), TAMP-S2GCNets still yields the lowest RMSE.
One important phenomenon we observe is that TAMP-S2GCNets systematically tends to substantially outperform StemGNN and AGCRN in more highly populated counties, including both Southern and Northern California.
For instance, the predictive gains of TAMP-S2GCNets in
Santa Clara are 39.2% vs. StemGNN and 67.9% vs. AGCRN.
These ﬁndings indicate that timeaware MP topological information contained in TAMP-S2GCNets can address hidden relationships among spatio-temporal factors contributing to COVID-19 dynamics which other models cannot.
F A DDITIONAL D ETAILS OF TAMP-S2GCN ETS A RCHITECTURE AND E XPERIMENTS
Remark F.1 (Spatio-Temporal Feature Transformation).
We utilize and share the learnable node embedding E φ to (i) construct the normalized self-adaptive adjacency matrix S and (ii) project the target node from the input feature space (in spatial and temporal dimensions) into a high-level latent space (together with Θ FT ).
Hence, this shared-update weights can not only mitigate the risk of overﬁtting but allow us to capture shared patterns among spatial-temporal features and graph structures over the time axis.
The learnable node embedding E φ in TAMP-S2GCNets can be obtained from the backward propagation through time.
We use Einstein summation convention in Eq.
1 for the multiplication of X T ∈ R b ×T × N × P (where b is the batch size) and E φ Θ ( (cid:96) ) FT .
The implementation of matrix multiplication in Eq.
1 is as follows:
(i) we ﬁrst perform the matrix multiplication between E φ ∈ R N × d c and Θ ( (cid:96) )
FT ∈ R d c × P × Q FT to get an intermediate weight term denoted by ˜ Θ
( (cid:96) )
FT ∈ R N × P × Q FT ; (ii) then we compute the matrix multiplication between X T and ˜ Θ ( (cid:96) )
FT through Einstein summation convention, i
e., in Pytorch, output = torch.einsum(‘btni, nio → btno’, X, intermediate weight term) (where X is X T and intermediate weight term is ˜ Θ ( (cid:96) ) FT ).
Remark F.2 (Normalized Self-Adaptive Adjacency Matrix).
We utilize the node embedding dictionaries E φ ∈ R N × d c to construct the normalized self-adaptive adjacency matrix S via multiplying
E φ and E (cid:62) φ .
As such, construction of the normalized self-adaptive adjacency matrix S does not impose any requirements on density of the input graph.
That is, our graph learning architecture (i.e., the self-adaptive adjacency matrix construction mechanism) overcomes this problem of the dense input graph met by GAT.
To explore more computationally efﬁcient mechanisms of learning the normalized self-adaptive adjacency matrix, in the future work, we will (i) apply the sparse sampling strategies to the graph learning architecture; (ii) learn the graph based on the original graph structure information, i.e., we only infer the dependencies between the adjacent nodes (i.e., there exists an edge between them in the original graph) via multiplying e u,φ and e (cid:62) v,φ ; (iii) construct hypergraph representation instead of graph which not only alleviate the computational complexity but capture heterogeneous higher-order structures information.
For ˜ S , in our experiments, we performed an extensive grid search for the hyperparameter K (i.e., truncation order).
We ﬁnd the optimal K of 2 for the transportation network, the optimal K offor COVID-19 network dataset, and the optimal K of 3 for Ethereum blockchain network.
That is, for larger networks the truncation order K tends to be lower so that the computational costs are mitigated.
Furthermore, we have performed ablation studies on PeMSD4 and PeMSD8 to assess utility of the normalized self-adaptive adjacency matrix S in TAMP-S2GCNets.
The results in Tables 13 andindicate that TAMP-S2GCNets outperforms TAMP-S2GCNets with the plain adjacency matrix (instead of the normalized self-adaptive adjacency matrix) for both PeMSD4 and PeMSD8 across all the evaluation metrics.
The results illustrate the importance of learning the node-speciﬁc patterns/embeddings over the spatial-temporal time series dataset.
[Table] [Table]
Remark F.3 (Global Average Pooling and Global Max Pooling).
We utilize the global average pooling (GAP) to preserve information about the Euler-Poincar ´ e surfaces in a ﬁxed-size representation through averaging of the learnt topological embeddings.
In our experiments, we ﬁnd that adding global max pooling (GMP) tends to strengthen the Euler-Poincar ´ e surfaces representation and to improve the overall performance.
Before GAP and GMP, we ﬁrst use CNN base models (i.e., f θ 1 ( · ) and f θ 2 ( · ) ) to learn the topological features of Euler-Poincar ´ e surfaces (i.e., f θ i ( { E i } T i =1 ) , where i = { 1 , 2 } ); then we employ GAP and GMP to the outputs of CNN base models, respectively.
We have conducted the ablation study to evaluate the performance of TAMP-S2GCNets without (w/o)
GAP or GMP on Golem.
From Table 15, we ﬁnd that applying both GAP and GMP enhances prediction performance.
Architecture
[Table]
Remark F.4 (Supra-Laplacian).
In our experiments, for large-scale networks, we apply sparse sampling strategy for the supra-Laplacian construction.
For example, on transportation network such as PeMSD3, we randomly select τ = 3 timestamps out of the T = 12 timestamps in the sliding window; as a result, we feed the sparse supra-Laplacian (i.e., 3 N × 3 N ; where N is the number of nodes) into the supragraph diffusion convolutional layer.
For smaller networks such as Ethereum token networks, we utilize the T timestamps (i.e., the whole sliding window information) to construct the supra-Laplacian.
In the future we will explore various graph sparsiﬁcation techniques, e.g., subsampling, to reduce computational costs for larger networks.
Remark F.5 (More Details on the Framework of TAMP-S2GCNets in Figure 1).
In Figure 1, the input is the target network over past 3 time slices ( {G t − 2 , G t − 1 , G t } ) and our TAMP-S2GCNets consists of 5 components.
That is, (i) equipped with node feature matrix X t , we apply spatial graph convolutional layer (i.e., GCN Layer in Figure 1) on G t extracts spatial information at time t ; (ii) feature transformation (i.e., FT in Figure 1) learns representation of the spatio-temporal data X T over a sliding window of size T (in Figure 1, for better understanding the operation procedure, we set T = 3 , i.e., X T
= { X t − 2 , X t − 1 , X t } ); (iii) we create a multiplex network based on the input networks (i.e., {G t − 2 , G t − 1 , G t } ) and construct the corresponding supra-Laplacian; lastly, we feed the supra-Laplacian into supragraph convolutional module (i.e., Supra GCN Layer in Figure 1); (iv) in Figure 1, we consider two types of Euler-Poincar´e Surface Representation (DEPS), i.e.,
DEPS f,g and DEPS f,h are Euler-Poincar´e surface representations under different multiﬁltrations, where f, h, g represent different types of ﬁltrations; in (iv), we feed Euler-Poincar´e surface representations into CNN base models and employ global average/max pooling to the outputs of CNN base models.
Finally, we combine these embeddings to obtain the ﬁnal embedding, where is fed into
GRU module for multi-step forecasting.
G A DDITIONAL C OMPUTATIONAL C OMPLEXITY AND T RAINING T IME C OMPARISON
Remark G.1 (Training Time Comparison).
We report the average training time (per epoch in seconds) for our TAMP-S2GCNets and 4 baselines on PeMSD4 and Decentraland datasets in Tablesand 17.
In terms of training time, TAMP-S2GCNets runs slightly slower than the baselines but the demonstrated running time is comparable to the baselines.
Although the time per epoch is slightly higher for TAMP-S2GCNets, we ﬁnd that the performance of TAMP-S2GCNets is signiﬁcantly better than the state-of-the-art baselines.
As such, this extra time appears to be worth it.
signiﬁcant results.
[Table] [Table]
Remark G.2 (More Details on Computational Complexity of Topological Summaries).
Computational complexity of Dynamic Euler-Poincar ´ e Surfaces (DEPS) is much lower than that of persistence diagrams.
This is because Euler Characteristics is an alternating sum of the Betti numbers, and one does not need to compute the persistence diagrams to compute the Betti numbers.
The computational complexity of a single persistence diagram PD k is O ( N 3 ) , where N is the number of k -simplices (Otter et al., 2017).
On the other hand
[Table]
by using sparse matrix methods, computational complexity for the Euler Characteristics of a simplicial complex with M simplices is O ( M ) (Edelsbrunner & Parsa, 2014).
If p is the resolution size of the multipersistence grid, the resulting complexity for DEPS is O ( p 2 M ) .
However, since one does not need to compute the persistence diagrams to obtain the Betti numbers, faster algorithms are possible.
For instance, Lesnick & Wright (2019) developed a much faster algorithm (RIVET software) to compute Betti numbers in multipersistence setting.
This reduction approach leads to computational complexity of O ( R 3 ) , where R is the size of the ﬁltered complex giving the minimal representation for the bipersistence module and R is far less than the original number of simplices.
Furthermore, by utilizing sparse matrix reductions, recent results of Kerber & Rolle (2021) signiﬁcantly improved the RIVET approach.
In particular, computational complexity for multipersistence landscapes with RIVET would be O
( R 5 ) , where R is again the size of the reduced ﬁltered complex (Vipond, 2020).
We have provided both computational complexity and running time of 8 topological summaries on
Decentraland and Golem (see Table 18).
The computational complexity of the overall approach is: O ( N [Equation] F [Equation]
/ 2 +
Q sup [Equation] − [Equation]
=
[Equation]
−T M ( (cid:96) )
+ R 3 + W GRU ) .
That is, we have: (i) spatial graph convolution module: O ( N 2 )
[Table]
Carri
[Table]
Vipond [Table] (where N
[Table]
the number of nodes); (ii) supragraph diffusion convolution module: O ( N T F N Q sup +
N T 2 Q sup / 2 + Q sup (cid:80) t − 1 (cid:96) =
t −T M ( (cid:96) ) )
(where F N is the number of node attribute features
[Table]
T denotes the sliding window size, Q sup is the output dimension of the supragraph diffusion convolutional layer, and M ( (cid:96) ) is the number of edges at the (cid:96) -th layer); (iii) DEPS:
O ( R 3 ) , where R is the size of the reduced ﬁltered complex; (iv) GRU: O ( W GRU ) , where W GRU is the total number of parameters in the GRU.