Time Dimension Dances with Simplicial Complexes: Zigzag Filtration Curve based Supra-Hodge Convolution Networks for Time-series Forecasting
Abstract
Graph neural networks (GNN) offer a new powerful alternative for multivariate timeseries forecasting, demonstrating remarkable success in a variety of spatio-temporalapplications, from urban flow monitoring systems to health care informatics tofinancial analytics.
Yet, such GNN models pre-dominantly capture only lowerorder interactions, that is, pairwise relations among nodes, and also largely ignoreintrinsic time-conditioned information on the underlying topology of multivariatetime series.
To address these limitations, we propose a new time-aware GNNarchitecture which amplifies the power of the recently emerged simplicial neuralnetworks with a time-conditioned topological knowledge representation in a form ofzigzag persistence.
That is, our new approach, Zigzag Filtration Curve based SupraHodge Convolution Networks (ZFC-SHCN) is built upon the two main components:(i) a new highly computationally efficient zigzag persistence curve which allows usto systematically encode time-conditioned topological information, and (ii) a newtemporal multiplex graph representation module for learning higher-order networkinteractions.
We discuss theoretical properties of the proposed time-conditionedtopological knowledge representation and extensively validate the new time-aware
ZFC-SHCN model in conjunction with time series forecasting on a broad rangeof synthetic and real world datasets: traffic flows, COVID-19 biosurveillance,
Ethereum blockchain, surface air temperature, and vector autoregressions.
Ourexperiments demonstrate that ZFC-SHCN achieves the state-of-the-art performancewith lower requirements on computational costs.
Introduction
Over the last few years, graph neural networks have emerged as a new powerful alternative totraditional statistical and machine learning models in conjunction with univariate and multivariatetime series forecasting tasks [26, 4, 39, 39, 27].
Such successful applications of GNNs range fromurban traffic analytics to forecasting COVID-19 hospitalizations to electrocardiogram monitoring [3,35, 53, 10, 20].
However, most GNNs remain inherently static and do not explicitly incorporate theinherent time characteristics of the encoded knowledge [56, 40].
In turn, limitations in capturing thetime dimension in the knowledge representation and learning mechanisms for time-evolving dataresults in GNNs becoming less relevant over time and requiring frequent and often costly modelupdates.
Furthermore, GNNs tend to pre-dominantly focus only on information propagation among nodesand be limited in their ability to describe polyadic relationships among multiple substructures ofmultivariate time series or multi-node interactions in dynamics graphs.
However, as recently shownby [6], such higher-order interactions might be the key toward better understanding of the underlying mechanisms of many real world graph-structured phenomena.
One of the recent approaches [23, 7, 8]to systematically address such polyadic graph interactions is to model higher order substructures assimplices and then, by borrowing the concepts of the Hodge Theory, to generalize the ideas of thecombinatorial graph Laplacian which describes diffusion from node to node via edges to diffusionover simplices.
This Hodge Laplacian construction allows for extending the notion of convolutionoperation to simplicial convolution, and the resulting simplicial neural networks (SNNs) are arguablyone of the frontlines in graph learning today, though these ideas have never been yet applied inconjunction with knowledge representation and learning of time-evolving objects.
Our goal here is to bridge the emerging concept of time-aware learning with the recent notions ofsimplicial convolution, with a particular focus on explicitly integrating the core time-conditionedtopological characteristics.
In particular, we amplify the power of SNNs with a time-conditionedtopological knowledge representation in a form of zigzag persistence for time-indexed data and,more specifically, its new highly computationally efficient summary, Zigzag Filtration Curve.
Asa result, our new approach, Zigzag Filtration Curve based Supra-Hodge Convolution Networks(ZFC-SHCN) enables us to systematically learn the most intrinsic time-conditioned information bothon the underlying topology of the time-evolving data and higher-order interactions among varioussubstructures.
Significance of our contributions can be summarized as follows:
• ZFC-SHCN is the first approach bringing the concepts simplicial convolution and SNNs totime-aware learning.
• We propose a new highly computationally efficient summary of persistence for time-indexeddata, Zigzag Filtration Curve, and derive its theoretical stability guarantees.•
We validate utility of ZFC-SHCN in conjunction with forecasting multivariate time seriesfrom diverse application domains such as traffic networks, COVID-19 biosurveillance,surface air temperature, token prices on Ethereum blockchain,
and vector autoregressions
Our findings indicate that ZFC-SHCN delivers the state-of-the-art forecasting performance, with a significant margin and demonstrates higher computational efficiency
Related Work
Time-series Forecasting and Spatio-temporal Graph Convolutional Networks Time-series forecasting is one of core subfields in statistical sciences [15, 9].
Most recently, there has appeared anumber of unconventional machine learning approaches to time-series forecasting.
In particular, graphconvolutional networks (GCN)-based models for the spatio-temporal network data have emerged as anew promising forecasting tool.
For instance, DCRNN [40] introduces the spectral graph convolution into spatio-temporal network data prediction, which can capture spatio-temporal dependencies.
STGCN [56] uses convolutional neural networks (CNN) to model temporal correlations.
Moreover,to infer the hidden inter-dependencies between different traffic variables, some efforts [54, 3, 10]conduct convolution operation in spatial dimension through the adaptive adjacency matrices.
Recent
Z-GCNETs [20] develops a zigzag topological layer equipped with zigzag persistence image into
GCN framework to model temporal correlations.
Another promising recent direction for time seriesforecasting beyond GCN is a fractional-order dynamical model proposed by [26].
This approachoffers an alternating scheme to determine the best estimate of the model parameters and unknownstimuli.
In turn, [27] proposes a Padé approximation based exponential neural operator (Padé Exp),aiming to improve time-series forecasting with exponential operators in neural operator learningschemes.
However, all of above methods only focus on node-level representations, in this paper, wefocus on both higher-order structures representation and topological information learning.
Topological Data Analysis for Graph Learning Persistent homology [24, 59] is a suite of toolswithin topological data analysis (TDA) that provides a way for measuring topological features ofshapes and of functions.
The extracted topological features have been recently shown to provide aninvaluable insight into hidden mechanisms behind organization and functionality of graph structureddata.
In particular, topological features have been actively used for node classification [58, 17],link prediction [55], and graph classification [30, 31, 14, 29].
For instance, [30] is one of the firstapproaches to integrate topological features into neural networks for graph classification, while [14]proposes a versatile framework for learning multiple vectorizations of persistent diagrams on graphs.
In turn, [58, 17, 55] apply topological features to GNN to understand and improve the message passing between nodes.
Finally, [32] proposes a topological graph layer with learnable filtrationfunctions for graph and node classification tasks, while [13] advances the ideas of multipersistence tograph learning.
Zigzag Persistent Homology Despite its promise, the regular persistent homology does not explicitlymodel the geometric and topological information from a sequence of topological spaces.
To addressthis limitation, a generalization of ordinary persistence, i.e., zigzag persistent homology, based onthe theory of quiver representation, has been proposed by [12].
Zigzag persistence allows us tosystematically describe how the homology changes over a sequences of spaces.
Despite its highpotential, especially in conjunctions with analysis of time-evolving data, zigzag persistence stillremains largely a theoretical concept, and its applications are yet scarce.
The recent results fortime-dependent data studies include, for example, zigzag-based clustering [36], bifurcation analysisof dynamic systems [52], and time series forecasting [20].
The memory and computational efficiencyof zigzag persistence is one of the daunting challenges.
Inspired by [42], in this paper we propose anovel highly computationally efficient representation of zigzag persistence for learning time-evolvingdata, that is, zigzag filtration curve.
Simplicial Neural Networks Modeling higher-order interactions on graphs is an emerging directionin graph representation learning.
While the role of higher-order structures for graph learning hasbeen documented for a number of years [1, 33] and involves such diverse applications as graph signalprocessing in image recognition [22], dynamics of disease transmission and biological network,integration of higher-order graph substructures into deep learning on graphs has emerged only in2020.
As shown by [6, 48], higher-order network structures can be leveraged to boost graph learningperformance.
Indeed, several most recent approaches [23, 47, 7, 18] propose to leverage simplicialinformation to perform neural networks on graphs.
However, neither of these Simplicial Neural
Networks (SNNs) are integrated with a topology-based graph convolution layer allowing us to learnboth time-aware persistent topological features and simplicial geometry of graphs.
In this paper, wepropose ZFC-SHCN to address this limitation.
Time-Aware Topological Learning with Zigzag Curves
Spatio-temporal Graph Construction A spatio-temporal graph is a collection of snapshots atdifferent time steps, denoted by G = {G 1 , G 2 , · · · , G T } , where T is the maximum timestamp.
Here
G t =
( V t , E t , A t , X t ) is the graph observed at time step t ∈
[1 , T ] , where V t is a finite set of |V| =
Nnodes, E t is a set of edges, A t ∈ R N × N is the adjacency matrix, and X t ∈ R N × d is the node featurematrix.
Specifically, each row of X t is a d -dimensional feature vector of the corresponding node.
Forsimplicity, wherever applicable below, we omit the subscript t and denote graph G t at time t as G .
Background on Ordinary Persistence Tools of ordinary persistence, or persistent homology (PH),allow us to study salient data shape patterns along various dimensions.
By shape here we broadly understand data properties which are invariant under continuous transformations, that is, transformationswhich do not alter “holes” in the data, for examples, bending, twisting, and stretching.
The key ideais to choose some suitable scale parameter ν
and then to study graph G not as a single object but as anested sequence of graphs, or graph filtration G 1 ⊆ . . .
⊆
G n = G which is induced by monotonicchanges of scale ν .
For example, if G is an edge-weighted graph ( V , E , w ) with weight functionw : E 7!
R , then for each ν j , j = 1 , . .
. , n , we set G ≤ ν j = ( V , E , w − 1 ( −∞ , ν j ]) , yielding theinduced edge-weighted filtration.
We can also consider only induced subgraphs of G with maximaldegree of ν j for each j = 1 , . .
. , n , resulting in the degree sublevel set filtration.
(For more discussionon graph filtrations see [29].)
Armed with this construction, we can track which shape patterns, forexample, independent components, loops, and voids, emerge as scale ν varies.
To make the processof pattern counting more systematic and efficient, we build an abstract simplicial complex K ( G j ) oneach G j .
We also record complex indices j b (birth) and j d (death) at which we first or last observeeach shape feature.
Topological features with longer lifespan are said to persist and likelier to yieldimportant information on the structural organization of G .
Learning Shapes of Time-Conditioned Data with Zigzag Persistence This construction enables usto extract the key topological descriptors from a single graph G .
However, in our case we observenot a single graph but a sequence of time-evolving graphs {G 1 , . . .
, G T } .
How can we track shapesignatures which are not just individualistic for each time stamp but characterize intrinsic propertiesof the observed object over time?
One approach how we can bring PH tools to analysis of time- conditioned objects is zigzag persistence.
Based on the theory of quiver representations, zigzagpersistence generalizes ordinary persistence to tracking characteristics of graphs (or other topologicalspaces) with inclusions going in different directions [12, 11].
In particular, given a time-indexedsequence of graphs {G 1 , . . .
, G T } , we first form a set of graph inclusions over time
G 1 ∪ G G 2 ∪ G G 3 ∪ G. . . ↗ ↖ ↗ ↖ ↗ ↖ ↗
G G G Gand then assess compatibility of persistent topological features across unions of graphs.
That is, werecord indices at which topological features (dis)appear, for some given scale ν ∗ .
If for a given ν ∗topological feature ρ (i.e., p -dimensional hole, 0 ≤ p ≤ K , where K is the dimension of the simplicialcomplex K ( G ) ) is first recorded in K ( G j ) , we say that the feature’s birth is j , and if ρ first appearsin K ( G j ∪ G j +1 ) , we record its birth as j + 1 / 2 .
In turn, if ρ is last seen in K ( G j ) , we record itsdeath as j , while if it is last seen in K ( G j ∪ G j +1 ) , we say that its death is at j + 1 / 2 .
Let J be theset of all observed topological features for a given ν ∗ .
Collecting then births and deaths over J , wesummarize all extracted information as a multiset D ν ∗ = { ( b ρ , d ρ ) ∈ R 2 | b ρ < d ρ , ρ ∈ J } , called azigzag persistent diagram (ZPD) (where b ρ and d ρ are the birth and death of the topological feature ρrespectively).
This makes zigzag persistence particularly attractive for analysis of dynamic objects which arenaturally indexed by time.
However, the idea of zigzag persistence is an applicable way beyondlearning time-evolving objects.
Nevertheless, zigzag persistence still remains largely a theoreticalconcept, with yet only a handful of applications, and one of the roadblocks hindering a broaderproliferation of zigzag-based methods in practice is their computational costs.
Here we make astep toward bringing more computationally efficient summary of zigzag persistence to real worldapplications.
Time-Aware Zigzag Filtration Curves Consider a sequence of time intervals associated with a zigzag filtration over time period [ t 1 , t N ]
[Equation]
Let
DgmZZ ν
∗ be the resulting ZPD for a given ν ∗ and M be the number of off-diagonal topologicalfeatures in ZPD, i.e., DgmZZ ν ∗ .
Inspired by the recent results on stabilized Betti sequences by [34]and filtration curves by [42] for ordinary persistence, we propose a new simple and computationallyefficient summary of zigzag persistence, called Zigzag Filtration Curve .
Definition 3.1 (Zigzag Filtration Curve (ZFC)) .
The zigzag filtration curve evaluated at ∆ t − i =
( t i − 1 + 12 , t i ) , i = { 1 , 2 , . . .
, N} , for a given ν ∗ , is defined as
[Equation]
where ( t b j , t d j ) ∈ R 2 is a vector containing birth and death of the j -th off-diagonal p -dimensionaltopological feature in DgmZZ ν ∗ (as such, t b j < t d j ), j = { 1 , 2 , . .
, M} , 0 ≤ p ≤ K ; ξ i : R 2 7!
Ris some suitable Lipschitz continuous function with Lipschitz constant L i , for example, Gaussiandensity; and ω i > 0 , i = { 1 , 2 , . . .
, N} are weights such that (cid:80) i ω i
= 1 .
Zigzag filtration curve at∆ t
+ i =
( t i , t i + 12 ) is defined analogously.
(For sake of notation, wherever applicable in the furtherexposition we suppress index p in ZFC.)
Motivated by [34], here as Lipschitz continuous function ξ i for intervals ∆ t − i , we use a Gaussiandensity f with mean ( t i − 1 + 1 / 2 , t i ) , while for intervals ∆ t
+ i , we set mean of f to ( t i , t i + 1 / 2) ,i = 1 , 2 , . . .
N .
For both ∆ t − i and ∆ t + i , we set a 2 × 2 -variance-covariance matrix Σ as identitymatrix.
(See Appendix C.5 for more discussion on sensitivity analysis.)
Other suitable choice of ξinclude arctan function.
As we show below, the proposed ZPC also enjoys the important theoretical stability guarantees interms of Wasserstein-1 distance.
Proposition 3.2 (Stability of Zigzag Filtration Curve) .
Let
DgmZZ ν ∗ be a zigzag persistence diagram and DgmZZ ′
c ∗ be its perturbed copy such that W 1 (cid:0) DgmZZ ν ∗ , DgmZZ ′ ν ∗ ) <
ϵ , where W 1 is
Wasserstein-1 distance.
Then, ZFC is stable with respect to Wasserstein-1 distance.
In practice topological features of various dimensions p , p = 0 , 1 , . . .
, K , may play different rolesin the learning task performance, and these roles are not known a-priori.
Hence, to harness timeconditioned information encoded in ZFC corresponding to different dimensions p , we propose MultiZigzag Filtration Curves (M-ZFCs)
M-ZFCs ν ∗ ∈ R K × N− 1by stacking ZFC 0 , ZFC 1 , . .
, ZFC K .
Figure 1 in Appendix B shows the both 0- and 1-dimensional ZFCs obtained from the proposed ZFC.
In the following section, we will demonstrate how ZFC can be integrated into the neural networkarchitectures for graph learning tasks.
Zigzag Filtration Curve based Supra-Hodge Convolution Networks
Given a graph G and its historical ω step graph signals X ω
= { X t − ω
+1 , . . .
, X t } ∈ R ω × N × F ( F isthe node feature dimensionality), the time-series forecasting problem is to learn a mapping functionf which maps the historical data { X t − ω
+1 , . . .
, X t } into the next h step data { X t +1 , . . .
, X t + h } .
The mapping relation is represented as follows { X t − ω +1 , . . .
, X t } f −!
{ X t +1 , . . .
, X t + h } .
4.1 Graph convolution in spatial dimension
Given the node embedding dictionary W ϕ =
( w ϕ 1 , w ϕ 2 , . . .
, w ϕN ) ∈ R N × d c (where x ϕu ∈ R d c and d cis the dimension of node embedding), we aim to seek a non-negative function S u,v = G ( w ϕu , w ϕv )which represents the pairwise similarity between any two nodes u and v .
Concretely, the multiplicationbetween W ϕ and ( W ϕ )
⊤ can (i) give a sum pooling of second-order features from the outer productof all the embedding vector pairs ( w ϕu , w ϕv ) and (ii) infer the hidden spatial dependencies of nodes
[Equation]
where ReLU ( · ) = max (0 , · ) is a nonlinear activation function, which is used to eliminate weakconnections proactively, and the role of the softmax function is applied to normalized the learnedgraph S .
Inspired by the recent advancements in random walk-based graph embedding learning [45,25], we make a graph convolution in spatial dimension, feeding a power series of learned graph Swith varying random walk steps { 1 , 2 , · · · , r } ( r ∈ Z + ), as follows
[Equation]
where σ ( · ) stands for a nonlinear activation function, Stack ( · ) is the function which stacks r poweredlearned graphs, H ( ℓ ) t, GC and H ( ℓ +1) t, GC are the input and output activations for layer ℓ (where H (0) t, GC =
X t ∈ R N × F ), and Θ ( ℓ ) GC ∈ R d
GC ℓ × d GC ℓ +1 is the ℓ -th layer’s trainable weights.
Next, we introducerepresentation learning of the higher-order graph (sub)structures using the supra-Hodge Laplacianwhich allows us to systematically leverage the underlying topological information.
4.2 Supra-Hodge convolution in temporal dimension
Time-evolving data such as multivariate time series, spatio-temporal processes, and dynamic networks,often exhibit a highly complex dependency among its substructures which is way beyond what canbe described by dyadic interactions or pairwise relations among nodes.
Instead, such higher-orderpolyadic interactions can be systematically addressed using the Hodge Theory.
In particular, thediscrete Hodge theory allows us to generalize the notion of a standard combinatorial graph Laplacianwhich describes diffusion on graph G from node to node via edges to diffusion over higher-ordersubstructures of G [41, 6].
In turn, higher-order substructures can be modeled as k -simplices of
G .
(See Appendix B for the background information on the Hodge Laplacians.)
In the last year,convolutional architectures on simplicial complexes based on the associated concepts of the Hodgetheory have emerged as the most recent direction in graph neural networks but have never beenyet applied to learning time-evolving data.
Our goal here is to introduce the notion of simplicialconvolution and the ideas of Hodge-Laplacians to time-aware learning.
In particular, to capture time-conditioned higher-order interactions on G and to describe diffusionof information over simplices along the temporal dimension, we build a supra-Hodge convolutionoperation, based on the multiplex network representation learning.
(In the following for simplicity,we assume the notations without sub/superscript k stands for node-level and in our experiments wealways consider k ∈ Z + ).
First, given the historical spatio-temporal network series G t − ω +1:
t ={G t − ω
+1 , G t − ω +2 , . . .
, G t } , we consider a directed connected node-aligned multiplex network, whichis made up of ω layers with N nodes on each layer.
That is, adjacency matrix A α
= { a αuv } N × N(where α ∈ { t − ω + 1 , . . .
, t } ) defines the intra-connection between nodes u and v in layer α and adistance matrix D αβ
= { d αβuu } N × N quantifies the transition probability of moving from node u oflayer α to node u of layer β .
(Here β > α , since we consider information diffusion procedures onlyalong the temporal dimension).
Next, based on the discrete Hodge theory, we propose a new Hodgek -Laplacian for multiplex graphs, called supra-Hodge k -Laplacian
L Supk ∈ R ϕ k ω × ϕ k ω
[Equation]
where L ααk is the Hodge k -Laplacian in layer α , D k +1 is the diagonal matrix of degrees of eachk -simplex, i.e., D k +1 = max ( diag ( | B k +1 | 1 , I )) and B k +1 is the k -simplex-to-
( k + 1) -simplexincidence matrix, and r -th of L ααk represents r -step random walk on Hodge k -Laplacian of layerα which will allow every k -simplex to accumulate information from its neighbors.
Hence, whenk = 1 , we can infer the spatial dependencies between each pair of edges and capture meaningful edgeinformation in both spatial and temporal dimensions – through the lens of supra-Hodge 1-Laplacian.
For instance, in molecule networks, each node represents an atom and each edge is a bond connectingtwo atoms; the bond (i.e., edge) features include bond type, ring status, molecular charge which areclosely related to atom (i.e., node) features (such as atomic total and partial charges).
Since the goal of forecasting task is to predict node (i.e., 0-simplex) attribute(s) in the next few timesteps, we propose a novel diffusion supra-Hodge convolution on the sliding window G t − ω +1:
t .
Wethen update nodes’ representations by transforming the multiplex k -simplex embedding to nodes viaincidence matrices
[Equation]
) where (i) in Equation 3: Θ ( ℓ ) k, SH ∈ R d SH k ; ℓ × d SH k ; ℓ +1 is a learnable filter matrix for layer ℓ (hered SH k ; ℓ and d SH k ; ℓ +1 are the intermediate and output dimensions to the ℓ -th layer), H ( ℓ ) t,k, SH and
H ( ℓ +1) t,k, SH are the input and output activations for layer ℓ (where H (0) t,k, SH =
¯ X k ; t − ω +1: t ∈
R ϕ k ω ×
d in k and the historical k -simplex features of the spatio-temporal networks X k ; t − ω +1:
t ={ X k ; t − ω +1 , X k ; t − ω +2 , . . .
, X k ; t } ∈ R ϕ k × ω
× d in k is reshaped as a matrix ¯ X k ; t − ω +1: t with shapeϕ
k ω × d in k ) and (ii) in Equation 4: we transform the k -simplex embedding H ( ℓ +1) t,k, SH to node embedding H ( ℓ +1) t, SH ∈ R N × d SH k ; ℓ +1 through incidence matrices.
We then combine the embeddings from graphconvolution, M-ZFCs convolution, and Supra-Hodge convolution to get the final embedding H ( ℓ [Equation] 1) t, out
[Equation]
where [ · , · , · ] denotes the concatenation of the outputs from three convolution operations, and
H ( ℓ +1) t, out ∈ R N × d out (where d out = d
GC ℓ +1 + d ZFCout + d SH ℓ +1 ).
4.3 ZFC convolution: a bridge between spatial and time dimensions
Armed with the representation learning of graph (sub)structures at each timestamp, we now discuss the ZFC convolution which allows us to preserve and propagate both spatial and time-awaretopological information simultaneously.
The intuition behind ZFC convolution is that learn a strongconnection between two dimensions via two 1D convolution layers, i.e., time-wise and node-wise.
ZFC convolution consists of three key components: (i) a linear embedding on M-ZFCs, which can learn the importance of time-aware topological features for each node to form a time-dimensionspecific node embedding; (ii) a 1D convolution layer along time-wise, where it gathers time-awaretopological features from the entire space into a compact set; (iii) a 1D convolution layer alongnode-wise, where can capture relations between different nodes.
The resulted ZFC convolutionoperation over a M-ZFCs ω is defined as
[Equation]
(6) where ω is the size of the window for sequence learning, M-ZFCs ω denotes the M-ZFCs featureextracted from the time window with size ω ,
Θ ZFC ∈ R N ×
d q is a weight matrix to be learned, F θand F ψ are 1D convolutional layers, and H t, ZFC ∈ R N ×
d ZFCout is the d ZFCout -dimensional output.
4.4 Gate Recurrent Unit with ZFC-SHCN
To describe the complex spatio-temporal dependencies among time series and assess a hidden state ofnodes at a future timestamp, we feed the final embedding H ( ℓ +1) t, out into Gated Recurrent Units (GRU).
Formally, we set the forward propagation equations of GRU as
[Equation]
t [Equation] where η ( · ) is an activation function (e.g., ReLU, LeakyReLU), ⊙ is the elementwise product, ℜ tis the update gate and ℑ i is the reset gate.
Here b ℜ , b ℑ , b Ψ , W ℜ , W ℑ , and W Ψ are learnableparameters, while (cid:104)
Ψ t − 1 , H ( ℓ +1) t, out (cid:105) and Ψ t are the input and output of GRU model, respectively.
Wethen obtain ˜ Ψ t which contains both the spatio-temporal and time-aware information.
Experiments
5.1 Datasets
We validate our ZFC-SHCN model on five diverse data types:
(i) COVID-19 datasets [49]: CA,
PA, and TX represent the number of COVID-19 hospitalizations in California (CA), Pennsylvania(PA), and Texas (TX) respectively; (ii) traffic datasets [16]: PeMSD4 and PeMSD8 are two realtime traffic datasets in California; (iii) synthetic multivariate time-series (MTS) datasets based onvector autoregression (VAR)
[28, 43] (where the VAR model is a generalization of the univariate
AR process with more than one time-evolving component); (iv) daily surface air temperature in CA,
PA, and TX over 02/01/2020–12/31/2020, and (v) Bytom token prices of Ethereum blockchain over07/27/2017–05/07/2018.
The results on (i)–(iii) are presented in the main body, and the analysis on(iv) and (v) are in Appendix C.2 and C.4.
The detailed description of each dataset is in Appendix C.1.
5.2 Baselines
We compare our proposed ZFC-SHCN with 14 types of state-of-the-art baselines (SOAs), including
FC-LSTM [51], SFM [57], N-BEATS [44], DCRNN [40], LSTNet [37], STGCN [56], TCN [4],
DeepState [46], GraphWaveNet [54], DeepGLO [50], LRGCN [38] AGCRN [3], StemGNN [10],and Z-GCNETs [20].
5.3 Experimental settings
We implement ZFC-SHCN with Pytorch framework on NVIDIA GeForce RTX 3090 GPU.
Weoptimize all the models by Adam optimizer for maximum of 200 epochs (see more details ofexperimental settings in Remark E.6 in Appendix).All models are evaluated in terms of Mean Absolute
Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE).
Thebest results are in bold font and the results with dotted underline are the best performance achievedby the runner-ups.
We also perform a one-sided two-sample t -test between the best result and thebest performance achieved by the runner-up, where *, **, *** denote p -value < 0.1, 0.05, 0.01(i.e. , denote significant, statistically significant, highly statistically significant results, respectively.
The source code is available at https://www.dropbox.com/sh/9wmwhkgabqqry0r/AAACZdPI_
5.4 Experimental results
Real datasets
The experimental results on PeMSD4 and PeMSD8 traffic data are reported in
Table 2.
As Table 2 shows, ZFC-SHCN achieves the best MAE, RMSE, and MAPE comparedwith SOAs on both PeMSD4 and PeMSD8.
Compared to the RNN-based methods such as FCLSTM, SFM, N-BEATS, LSTNet, and TCN, ZFC-SHCN achieves relative gains in RMSE over therunner-ups, ranging from 17.68% to 65.41% for both PeMSD4 and PeMSD8.
In turn, DCRNN,
STGCN, GraphWaveNet, AGCRN, and StemGNN only focus on learning node-level representations.
Compared to them, ZFC-SHCN captures interactions and encodes higher-order structurescorrelations beyond pairwise relations among nodes and yields a relative gain from 2.06% to 5.63%in RMSE on the traffic datasets.
In addition, we compare ZFC-SHCN with the method basedon zigzag persistence image, i.e., Z-GCNETs, and find that ZFC-SHCN outperforms Z-GCNETsby 1.75% on PeMSD4 and 5.36% on PeMSD8 in terms of RMSE.
Table 3 presents COVID-19hospitalization prediction results (RMSE) in CA, PA, and TX, and we observe the following findings.
[Table]
First, our proposed ZFC-SHCN achieves state-ofthe-art performance on all three datasets.
For instance, ZFC-SHCN yields 3.61%, 1.47%, 65.55% relative gains in RMSE over the runner-ups (includingboth GCN-based and zigzag persistence image-basedmethods) on three biosurveillance datasets.
Theseresults indicate that the ZFC mechanism and higherorder representation learning module play significantroles in capturing both topological information andhigher-order structures.
Second, as shown in Figure 2in Appendix C, we find that, compared to the runner-up (i.e., Z-GCNETs), the predicted value of
COVID-19 hospitalizations is more consistent with the ground-truth.
Finally, Tables 3 and 4 in
Appendix C present the overall prediction performances of ZFC-SHCN and representative baselineson surface air temperature and Ethereum blockchain datasets.
We find that our proposed ZFC-SHCNconsistently outperforms all baselines with either a significant or (highly) statistically significantmargin across all data, except surface air temperature in TX, where ZFC-SHCN still yields the bestperformance across all models. [Table] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
two VAR datasets are summarized in Table 1.
Comparedto the three strongest baselines (i.e., AGCRN, StemGNN, and Z-GCNETs), we observe that our pro- posed ZFC-SHCN consistently yields the best performance for all synthetic datasets.
More precisely,
ZFC-SHCN outperforms the runner-ups from 8.89% to 10.52% for VAR T 1 and VAR T 2 .
Furthermore,to assess the time-wise and high network interactions, we use the global clustering coefficient (GCC)and Euler-Poincaré characteristic (EPC) as measures of higher order substructures [5].
We find thatfor GCC for VAR T 1 and VAR T 2 are 4.96 and 5.87, respectively; while the average EPC for VAR T 1and VAR T 2 are 7.47 and 6.91, respectively.
Interestingly (though intuitively it can be expected),higher
GCC and lower EPC tend to be associated with higher relative gains delivered by ZFC-SHCN.
Finally, in Appendix C.5 we present the sensitivity analysis for ZFC as a function of covariancematrix in VAR models.
5.5 Ablation studies
To evaluate the performance of different components in our ZFC-SHCN model, we perform an expansive ablation study.
The ablation study is made with three setups: (i)
ZFC-SHCN without graph convolution in spatial dimension (W/o Graph convolution inspatial dimension), (ii) ZFC-SHCN without ZFC convolution (W/o ZFC convolution),and (iii) ZFC-SHCN without supra-Hodge convolution (W/o Supra-Hodge convolution).
[Table]
The experimental results are shown in
Table 4 and prove the validity of eachcomponent.
As Table 4 indicates, compared to ZFC-SHCN w/o ZFC convolution, the zigzag homological feature is vitalfor capturing the topological structure ofspatio-temporal graph and our proposedgraph convolution operation on ZFC significantly improves forecasting performance.
By comparing to ZFC-SHCN w/o supraHodge convolution, we illustrate the significance of higher-order structures representation learningfor guiding the model how to capture information on higher-order interactions.
Also, ZFC-SHCN w/ograph convolution in spatial dimension demonstrates that the learned graph obtained from trainableweights can learn hidden information and enhance (multivariate) time-series representation learning.
5.6 Computational Complexity
For higher-order simplices, incidence matrices B 1 and B 2 can be calculated efficiently with thecomputational complexity O ( N + M ) and O ( M + Q ) respectively, where N is the number of0-simplices (i.e., nodes), M is the number of 1-simplices (i.e., edges), and Q is the number of2-simplices (i.e., filled triangles).
The computational complexity of ZFC is O
(Υ δ ) [2, 21], where Υrepresents the number of points in time interval and δ ∈ [2 , 2 . 373) .
We also compare our ZFC-SHCNwith the most recent approach based on multipersistence-GNN [19] (i.e., TAMP-S2GCNets).
We findthat ZFC-SHCN yields either on-par or more competitive performance than TAMP-S2GCNets, whileour proposed ZFC-SHCN significantly improves the computational efficiency (see Appendix C.3 formore details).
More details about running time comparison can be found in Appendix C.4.
Conclusion
We have proposed a novel framework for time aware deep learning of time-evolving objects whichtakes the advantages of both the higher-order interactions among the data substructures, described assimplices, and the most intrinsic time-conditioned topological information exhibited by the object,characterized via zigzag persistent homology.
By leveraging the power of simplicial convolutionoperation and zigzag persistence for time-indexed data, ZFC-SHCN is capable to to yield the mostcompetitive forecasting performance, while requiring less computational resources than its closestcompetitors.
Still, computational complexity and limited theoretical results on statistical inferencefor zigzag persistence remain one of the major existing limitations of ZFC and, more generally, alltopological methods for time dependent processes.
In the future, we will investigate these theoreticaland methodological challenges and will extend the ZFC-SHCN idea to anomaly detection in streamingtime-dependent processes.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See the Conclusion Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the Broader Impact Section in Appendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
and Appendix A.
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section 5.3.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 5.3.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.3.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] They use public-domain-equivalent
license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A
[Table] Did you describe any potential participant risks, with links to Institutional Review
[Table] ) approvals, if applicable
[N/A]
[Table] Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]