Time-Conditioned Dances with Simplicial Complexes: Zigzag Filtration Curve based Supra-Hodge Convolution Networks for Time-series Forecasting
Abstract
Graph neural networks (GNNs) offer a new powerful alternative for multivariate time series forecasting, demonstrating remarkable success in a variety of spatiotemporal applications, from urban flow monitoring systems to health care informatics to financial analytics.
Yet, such GNN models pre-dominantly capture only lower order interactions, that is, pairwise relations among nodes, and also largely ignore intrinsic time-conditioned information on the underlying topology of multivariate time series.
To address these limitations, we propose a new time-aware GNN architecture which amplifies the power of the recently emerged simplicial neural networks with a time-conditioned topological knowledge representation in a form of zigzag persistence.
That is, our new approach, Zigzag Filtration Curve based
Supra-Hodge Convolution Networks (ZFC-SHCN) is built upon the two main components: (i) a new highly computationally efficient zigzag persistence curve which allows us to systematically encode time-conditioned topological information, and (ii) a new temporal multiplex graph representation module for learning higherorder network interactions.
We discuss theoretical properties of the proposed time-conditioned topological knowledge representation and extensively validate the new time-aware ZFC-SHCN model in conjunction with time series forecasting on a broad range of synthetic and real-world datasets: traffic flows, COVIDbiosurveillance, Ethereum blockchain, surface air temperature, wind energy, and vector autoregressions.
Our experiments demonstrate that the ZFC-SHCN achieves the state-of-the-art performance with lower requirements on computational costs.
Introduction
Over the last few years, graph neural networks (GNNs) have emerged as a new powerful alternative to traditional statistical and machine learning models in conjunction with univariate and multivariate time series forecasting tasks [31, 5, 44, 44, 32].
Such successful applications of GNNs range from urban traffic analytics to forecasting COVID-19 hospitalizations to electrocardiogram monitoring [4, 40, 59, 11, 21].
However, most GNNs remain inherently static and do not explicitly incorporate the inherent time characteristics of the encoded knowledge [62, 45].
In turn, limitations in capturing the time dimension in the knowledge representation and learning mechanisms for time-evolving data results in GNNs becoming less relevant over time and, hence, requiring frequent updates.
Furthermore, GNNs tend to pre-dominantly focus only on information propagation among nodes and also be limited in their ability to describe polyadic relationships among multiple substructures of multivariate time series or multi-node interactions in dynamics graphs.
However, as recently shown by [7, 22], such higher-order interactions might be the key toward better understanding of the underlying mechanisms of many real-world graph-structured phenomena.
This challenge on polyadic graph interactions has been recently addressed by [26, 9, 8] who propose to model higher order substructures as simplices.
Then, by borrowing the concepts of the Hodge theory, these approaches allow for generalization of the ideas of the combinatorial graph Laplacian which describes a diffusion from node to node via edges to a case of diffusion over simplices.
Such Hodge Laplacian construction allows for extending the notion of convolution operation to simplicial convolution, and the resulting simplicial neural networks (SNNs) are arguably one of the frontlines in graph learning today.
However, these ideas have never been yet applied in conjunction with knowledge representation and learning of time-evolving objects.
Our goal here is to bridge the emerging concept of time-aware learning with the recent notions of simplicial convolution, with a particular focus on explicitly integrating the core time-conditioned topological characteristics.
In particular, we amplify the power of SNNs with a time-conditioned topological knowledge representation in a form of zigzag persistence for time-indexed data and, more specifically, its new highly computationally efficient summary, Zigzag Filtration Curve.
As a result, our new approach, Zigzag Filtration Curve based Supra-Hodge Convolution Networks (ZFC-SHCN) enables us to systematically learn the most intrinsic time-conditioned information both on the underlying topology of the time-evolving data and higher-order interactions among various substructures.
Significance of our contributions can be summarized as follows:
• ZFC-SHCN is the first approach bringing the concepts of simplicial convolution and SNNs to time-aware learning.
• We propose a new highly computationally efficient summary of persistence for time-indexed data, Zigzag Filtration Curve, and derive its theoretical stability guarantees.
• We validate the utility of ZFC-SHCN in conjunction with forecasting multivariate time series from diverse application domains such as traffic networks, COVID-19 biosurveillance, surface air temperature, token prices on Ethereum blockchain, wind energy, and vector autoregressions.
Our findings indicate that ZFC-SHCN delivers the state-of-the-art forecasting performance, with a significant margin and demonstrates higher computational efficiency.
Related Work
Time-series Forecasting and Spatio-temporal Graph Convolutional Networks Time-series forecasting is one of the core subfields in statistical sciences [16, 10].
Most recently, there have appeared a number of unconventional machine learning approaches to time-series forecasting.
In particular, graph convolutional network (GCN)-based models for spatio-temporal network data have emerged as a promising forecasting tool.
For instance, DCRNN [45] introduces spectral graph convolution into spatio-temporal network data prediction, which can capture spatio-temporal dependencies.
STGCN [62] uses convolutional neural networks (CNNs) to model temporal correlations.
Moreover, to infer hidden inter-dependencies between different traffic variables, [60, 4, 11] conduct a convolution operation in spatial dimension through adaptive adjacency matrices.
Recent Z-GCNETs [21] develops a zigzag topological layer equipped with a zigzag persistence image into a GCN framework to model temporal correlations.
Another promising recent direction for time series forecasting beyond
GCN is a fractional-order dynamical model proposed by [31].
This approach offers an alternating scheme to determine the best estimate of the model parameters and unknown stimuli.
In turn, [32] proposes a Padé approximation based exponential neural operator (Padé Exp), aiming to improve time-series forecasting with exponential operators in neural operator learning schemes.
However, all of the above methods only focus on node-level representations.
In contrast, in this paper, we focus on both higher-order structure representation and topological information learning.
Topological Data Analysis for Graph Learning Persistent homology [27, 65] is a suite of tools within topological data analysis (TDA) that provides a way for measuring topological features of shapes and functions.
The extracted topological features have been recently shown to provide invaluable insights into hidden mechanisms behind the organization and functionality of graph structured data.
In particular, topological features have been actively used for node classification [64, 18], link prediction [61], and graph classification [35, 36, 15, 34].
For instance, [35] is one of the first approaches to integrate topological features into neural networks for graph classification, while [15] proposes a versatile framework for learning multiple vectorizations of persistent diagrams on graphs.
In turn, [64, 18, 61] apply topological features to GNNs to understand and improve the message passing between nodes.
Finally, [37] proposes a topological graph layer with learnable filtration functions for graph and node classification tasks, while [14] advances the ideas of multipersistence to graph learning.
Zigzag Persistent Homology
Despite its promise, regular persistent homology does not explicitly model the geometric and topological information from a sequence of topological spaces.
To address this limitation, a generalization of ordinary persistence, i.e., zigzag persistent homology, based on the theory of quiver representation, has been proposed by [13].
Zigzag persistence allows us to systematically describe how the homology changes over a sequence of spaces.
Despite its high potential, especially in conjunction with analysis of time-evolving data, zigzag persistence still remains largely a theoretical concept, and its applications are yet scarce.
The recent results for time-dependent data studies include, for example, zigzag-based clustering [41], bifurcation analysis of dynamic systems [57], and time series forecasting [21].
The memory and computational efficiency of zigzag persistence is one of the daunting challenges.
Inspired by [47], we propose a novel highly computationally efficient representation of zigzag persistence for learning time-evolving data, that is, zigzag filtration curve.
Simplicial Neural Networks Modeling higher-order interactions on graphs is an emerging direction in graph representation learning.
While the role of higher-order structures for graph learning has been documented for a number of years [1, 38] and involves such diverse applications as graph signal processing in image recognition [25], dynamics of disease transmission and biological networks, integration of higher-order graph substructures into deep learning on graphs has emerged only in 2020.
As shown by [ 7, 53], higher-order network structures can be leveraged to boost graph learning performance.
Indeed, several recent approaches [26, 52, 9, 19] propose to leverage simplicial information to perform neural networks on graphs.
However, neither of these Simplicial Neural
Networks (SNNs) are integrated with a topology-based graph convolution layer allowing us to learn both time-aware persistent topological features and simplicial geometry of graphs.
In this paper, we propose ZFC-SHCN to address this limitation.
Time-Aware Topological Learning with Zigzag Curves
Spatio-temporal Graph Construction A spatio-temporal graph is a collection of snapshots at different time steps, denoted by G = {G 1 , G 2 , · · · , G T } , where T is the maximum timestamp.
Here
G t =
( V t , E t , A t , X t ) is the graph observed at time step t ∈
[1 , T ] , where V t is a finite set of |V| = N nodes, E t is a set of edges, A t ∈ R N × N is the adjacency matrix, and X t ∈ R N × d is the node feature matrix.
Specifically, each row of X t is a d -dimensional feature vector of the corresponding node.
For sake of notations, wherever applicable below, we omit the subscript t and denote graph G t at time t as G .
Background on Ordinary Persistence Tools of ordinary persistence, or persistent homology (PH), allow us to study salient data shape patterns along various dimensions.
By shape here we broadly understand data properties that are invariant under continuous transformations, that is, transformations that do not alter “holes” in the data, for example, bending, twisting, and stretching.
The key idea is to choose some suitable scale parameter ν
and then to study a graph G not as a single object but as a nested sequence of graphs, or graph filtration G 1 ⊆ . . .
⊆
G n = G , which is induced by monotonic changes of scale ν .
For example, if G is an edge-weighted graph ( V , E , w ) with weight function w : E 7!
R , then for each ν j , j = 1 , . .
. , n , we set G ≤ ν j = ( V , E , w − 1 ( −∞ , ν j ]) , yielding the induced edge-weighted filtration.
We can also consider only induced subgraphs of G with maximal degree of ν j for each j = 1 , . .
. , n , resulting in the degree sublevel set filtration.
(For more discussion on graph filtrations see [34].)
Armed with this construction, we can track which shape patterns, for example, independent components, loops, and voids, emerge as the scale ν varies.
To make the process of pattern counting more systematic and efficient, we build an abstract simplicial complex K ( G j ) on each G j .
We also record complex indices j b (birth) and j d (death) at which we first or last observe each shape feature.
Topological features with longer lifespans are said to persist and are likelier to yield important information on the structural organization of G .
Learning Shapes of Time-Conditioned Data with Zigzag Persistence This construction enables us to extract the key topological descriptors from a single graph G .
However, in our case, we observe not a single graph but a sequence of time-evolving graphs {G 1 , . . .
, G T } .
How can we track shape signatures which are not just individualistic for each time stamp but characterize intrinsic properties of the observed object over time?
One approach to how we can bring PH tools to analysis of timeconditioned objects is zigzag persistence.
Based on the theory of quiver representations, zigzag persistence generalizes ordinary persistence to track characteristics of graphs (or other topological spaces) with inclusions going in different directions [13, 12].
In particular, given a time-indexed sequence of graphs {G 1 , . . .
, G T } , we first form a set of graph inclusions over time
G
[Equation]
↗
[Equation]
and then assess the compatibility of persistent topological features across unions of graphs.
That is, we record indices at which topological features (dis)appear, for some given scale ν ∗ .
If for a given ν ∗ topological feature ρ (i.e., p -dimensional hole, 0 ≤ p ≤ K , where K is the dimension of the simplicial complex K ( G ) ) is first recorded in K ( G j ) , we say that the feature’s birth is j , and if ρ first appears in K ( G j ∪ G j +1 ) , we record its birth as j + 1 / 2 .
In turn, if ρ is last seen in K ( G j ) , we record its death as j , while if it is last seen in K ( G j ∪ G j +1 ) , we say that its death is at j + 1 / 2 .
Let J be the set of all observed topological features for a given ν ∗ .
Collecting then births and deaths over J , we summarize all extracted information as a multiset D ν ∗ = { ( b ρ , d ρ ) ∈ R 2 | b ρ < d ρ , ρ ∈ J } , called a zigzag persistent diagram (ZPD) (where b ρ and d ρ are the birth and death of the topological feature ρ respectively).
This makes zigzag persistence particularly attractive for the analysis of dynamic objects which are naturally indexed by time.
However, the idea of zigzag persistence is applicable far beyond learning time-evolving objects.
Nevertheless, zigzag persistence still remains largely a theoretical concept, with yet only a handful of applications, and one of the roadblocks hindering a broader proliferation of zigzag-based methods in practice is their computational costs.
Here we take a step toward bringing a more computationally efficient summary of zigzag persistence to real-world applications.
Time-Aware Zigzag Filtration Curves Consider a sequence of time intervals associated with a zigzag filtration over a time period [ t 1 , t N ]
[Equation]
Let
DgmZZ ν
∗ be the resulting ZPD for a given ν ∗ and M be the number of off-diagonal topological features in ZPD, i.e., DgmZZ ν ∗ .
Inspired by the recent results on stabilized Betti sequences by [39] and filtration curves by [47] for ordinary persistence, we propose a new simple and computationally efficient summary of zigzag persistence, called a Zigzag Filtration Curve .
Definition 3.1 (Zigzag Filtration Curve (ZFC)) .
The zigzag filtration curve evaluated at ∆ t − i =
( t i − 1 + 12 , t i ) , i = { 1 , 2 , . . .
, N} , for a given ν ∗ , is defined as
[Equation]
where ( t b j , t d j ) ∈ R 2 is a vector containing the birth and death of the j -th off-diagonal p -dimensional topological feature in DgmZZ ν ∗ (as such, t b j < t d j ), j = { 1 , 2 , . .
, M} , 0 ≤ p ≤ K ; ξ i : R 2 7!
R is some suitable Lipschitz continuous function with Lipschitz constant L i , for example, a Gaussian density; and ω i > 0 , i = { 1 , 2 , . . .
, N} are weights such that (cid:80) i ω i
= 1 .
Zigzag filtration curve at ∆ t
+ i =
( t i , t i + 12 ) is defined analogously.
(For the sake of notational simplicity, wherever applicable in the further exposition we suppress the index p in ZFC.)
Motivated by [39], here as the Lipschitz continuous function ξ i for intervals ∆ t − i , we use a Gaussian density f with mean ( t i − 1 + 1 / 2 , t i ) , while for intervals ∆ t
+ i , we set the mean of f to ( t i , t i + 1 / 2) , i = 1 , 2 , . . .
N .
For both ∆ t − i and ∆ t + i , we choose the 2 × 2 -variance-covariance matrix Σ to be the identity matrix.
(See Appendix C.7 for more discussion on sensitivity analysis.)
Another suitable choice of ξ is the arctan function.
As we show below, the proposed ZPC also enjoys important theoretical stability guarantees in terms of Wasserstein-1 distance.
Proposition 3.2 (Stability of Zigzag Filtration Curve) .
Let
DgmZZ ν ∗ be a zigzag persistence diagram and DgmZZ ′
c ∗ be its perturbed copy such that W 1 (cid:0) DgmZZ ν ∗ , DgmZZ ′ ν ∗ ) <
ϵ , where W 1 is
Wasserstein-1 distance.
Then, ZFC is stable with respect to Wasserstein-1 distance.
In practice topological features of various dimensions p , p = 0 , 1 , . . .
, K , may play different roles in the learning task performance, and these roles are not known a-priori.
Hence, to harness timeconditioned information encoded in ZFC corresponding to different dimensions p , we propose MultiZigzag Filtration Curves (M-ZFCs)
M-
[Equation]
by stacking
[Equation]
Figure 1 in Appendix B shows the both 0- and 1-dimensional ZFCs obtained from the proposed ZFC.
In the following section, we demonstrate how ZFC can be integrated into neural network architectures for graph learning tasks.
Zigzag Filtration Curve Based Supra-Hodge Convolution Networks
Given a graph G and its historical ω step graph signals X ω
= { X t − ω
+1 , . . .
, X t } ∈ R ω × N × F ( F is the node feature dimensionality), the time-series forecasting problem is to learn a mapping function f that maps the historical data { X t − ω +1 , . . .
, X t } into the next h step data { X t +1 , . . .
, X t + h } .
The mapping relation is represented as follows: { X t − ω +1 , . . .
, X t } f −!
{ X t +1 , . . .
, X t + h } .
4.1 Graph convolution in the spatial dimension
Given the node embedding dictionary W ϕ =
( w ϕ 1 , w ϕ 2 , . . .
, w ϕN ) ∈ R N × d c (where x ϕu ∈ R d c and d c is the dimension of node embedding), we aim to seek a non-negative function S u,v = G ( w ϕu , w ϕv ) which represents the pairwise similarity between any two nodes u and v .
Concretely, the multiplication between W ϕ and ( W ϕ )
⊤ can (i) give a sum pooling of second-order features from the outer product of all the embedding vector pairs ( w ϕu , w ϕv ) and (ii) infer the hidden spatial dependencies of nodes
[Equation]
where ReLU ( · ) = max (0 , · ) is a nonlinear activation function, which is used to eliminate weak connections proactively, and the role of the softmax function is applied to normalize the learned graph
S .
Inspired by the recent advancements in random walk-based graph embedding learning [50, 29], we make a graph convolution in spatial dimension, feeding a power series of the learned graph S with varying random walk steps { 1 , 2 , · · · , r } ( r ∈ Z + ), as follows:
[Equation]
where σ ( · ) stands for a nonlinear activation function, Stack ( · ) is the function which stacks r powered learned graphs, H ( ℓ ) t, GC and H ( ℓ +1) t, GC are the input and output activations for layer ℓ (where H (0) t, GC =
X t ∈ R N × F ), and Θ ( ℓ ) GC ∈ R d
GC ℓ × d GC ℓ +1 is the ℓ -th layer’s trainable weights.
Next, we introduce representation learning of the higher-order graph (sub)structures using the supra-Hodge Laplacian which allows us to systematically leverage the underlying topological information.
4.2 Supra-Hodge convolution in temporal dimension
Time-evolving data such as multivariate time series, spatio-temporal processes, and dynamic networks, often exhibit a highly complex dependency among its substructures that goes far beyond what can be described by dyadic (or pairwise) interactions among nodes.
Instead, such higher-order polyadic interactions can be systematically addressed using the Hodge theory.
In particular, the discrete Hodge theory allows us to generalize the notion of a standard combinatorial graph Laplacian which describes diffusion on graph G from node to node via edges to diffusion over higher-order substructures of
G [46, 7].
In turn, higher-order substructures can be modeled as k -simplices of G .
(See Appendix B for background information on Hodge Laplacians.)
Convolutional architectures on simplicial complexes based on the associated concepts of the Hodge theory have emerged as a recent direction in graph neural networks but have not yet been applied to learning time-evolving data.
Our goal here is to introduce the notion of simplicial convolution and the ideas of Hodge-Laplacians to time-aware learning.
In particular, to capture time-conditioned higher-order interactions on G and to describe diffusion of information over simplices along the temporal dimension, we build a supra-Hodge convolution operation, based on the multiplex network representation learning.
(In the following for simplicity, notation without sub/superscript k stands for node-level quantities and in our experiments we always consider k ∈ Z + ).
First, given the historical spatio-temporal network series G t − ω +1:
t = {G t − ω
+1 , G t − ω +2 , . . .
, G t } , we consider a directed connected node-aligned multiplex network, which is made up of ω layers with N nodes on each layer.
That is, the adjacency matrix A α
= { a αuv } N × N (where α ∈ { t − ω + 1 , . . .
, t } ) defines the intra-connection between nodes u and v in layer α and a distance matrix D αβ
= { d αβuu } N × N quantifies the transition probability of moving from node u of layer α to node u of layer β .
(Here β > α , since we consider information diffusion procedures only along the temporal dimension).
Next, based on the discrete Hodge theory, we propose a new Hodge k -Laplacian for multiplex graphs, called the supra-Hodge k -Laplacian
L Supk ∈ R ϕ k ω × ϕ k ω
[Equation]
··· ( L ωωk ) r   , (2) where L ααk is the Hodge k -Laplacian in layer α , D k +1 is the diagonal matrix of degrees of each k -simplex, i.e., D k +1 = max ( diag ( | B k +1 | 1 , I )) and B k +1 is the k -simplex-to-
( k + 1) -simplex incidence matrix, and the r -th power of L ααk represents r -step random walk on the Hodge k -Laplacian of layer α which will allow every k -simplex to accumulate information from its neighbors.
Hence, when k = 1 , we can infer the spatial dependencies between each pair of edges and capture meaningful edge information in both spatial and temporal dimensions – through the lens of the supra-Hodge 1-Laplacian.
For instance, in molecule networks, each node represents an atom and each edge is a bond connecting two atoms; the bond (i.e., edge) features include bond type, ring status, and molecular charge which are closely related to atom (i.e., node) features (such as atomic total and partial charges).
Since the goal of the forecasting task is to predict node (i.e., 0-simplex) attribute(s) in the next few time steps, we propose a novel diffusion supra-Hodge convolution on the sliding window G t − ω +1: t .
We then update nodes’ representations by transforming the multiplex k -simplex embedding to nodes via incidence matrices
[Equation] [Equation]
[Equation]
where (i) in Equation 3: Θ ( ℓ ) k, SH ∈ R d SH k ; ℓ × d SH k ; ℓ +1 is a learnable filter matrix for layer ℓ (here d SH k ; ℓ and d SH k ; ℓ +1 are the intermediate and output dimensions to the ℓ -th layer), H ( ℓ ) t,k, SH and
H ( ℓ +1) t,k, SH are the input and output activations for layer ℓ (where H (0) t,k, SH =
¯ X k ; t − ω +1: t ∈
R ϕ k ω ×
d in k and the historical k -simplex features of the spatio-temporal networks X k ; t − ω +1:
t = { X k ; t − ω +1 , X k ; t − ω +2 , . . .
, X k ; t } ∈ R ϕ k × ω
× d in k is reshaped as a matrix ¯ X k ; t − ω +1: t with shape ϕ
k ω
×
d in k ) and (ii) in Equation 4: we transform the k -simplex embedding H ( ℓ +1) t,k, SH to node embedding H ( ℓ +1) t,
SH ∈
R N × d SH k ; ℓ +1 through incidence matrices.
4.3 ZFC convolution: a bridge between spatial and time dimensions
Armed with the representation learning of graph (sub)structures at each timestamp, we now discuss the
ZFC convolution which allows us to preserve and propagate both spatial and time-aware topological information simultaneously.
The intuition behind ZFC convolution is that it learns a strong connection between two dimensions via two 1D convolution layers, i.e., time-wise and node-wise.
ZFC convolution consists of three key components: (i) a linear embedding on M-ZFCs, which can learn the importance of time-aware topological features for each node to form a time-dimension-specific node embedding; (ii) a time-wise 1D convolution layer, where it gathers time-aware topological features from the entire space into a compact set; (iii) a node-wise 1D convolution layer, which can capture relations between different nodes.
The resulted ZFC convolution operation over a M-ZFCs ω is defined as
[Equation]
where ω is the size of the window for sequence learning, M-ZFCs ω denotes the M-ZFCs feature extracted from the time window with size ω , Θ M-ZFC ∈ R N ×
d q is a weight matrix to be learned, F θ and F ψ are 1D convolutional layers, and H t, M-ZFC ∈ R N × d M-ZFCout is the d M-ZFCout -dimensional output.
We then combine the embeddings from graph convolution, M-ZFCs convolution, and supra-Hodge convolution to get the final embedding H ( ℓ +1) t, out
[Equation]
(6) where [ · , · , · ] denotes the concatenation of the outputs from three convolution operations, and
H ( ℓ +1) t, out ∈ R N × d out (where d out = d
GC ℓ +1 + d ZFCout + d SH ℓ +1 ).
4.4 Gate Recurrent Unit with ZFC-SHCN
To describe the complex spatio-temporal dependencies among time series and assess a hidden state of nodes at a future timestamp, we feed the final embedding H ( ℓ +1) t, out into Gated Recurrent Units (GRUs).
Formally, we set the forward propagation equations of the GRUs as ℜ t
[Equation]
where η ( · ) is an activation function (e.g., ReLU, LeakyReLU), ⊙ is the elementwise product, ℜ t is the update gate and ℑ i is the reset gate.
Here b ℜ , b ℑ , b Ψ , W ℜ , W ℑ , and W Ψ are learnable parameters, while (cid:104) Ψ t − 1 , H ( ℓ +1) t, out (cid:105) and Ψ t are the input and output of GRU model, respectively.
We then obtain ˜ Ψ t which contains both the spatio-temporal and time-aware information.
Experiments 5.1 Datasets
We validate our ZFC-SHCN model on six diverse data types: (i) COVID-19 datasets [54]: CA, PA, and TX represent the number of COVID-19 hospitalizations in California (CA), Pennsylvania (PA), and Texas (TX) respectively; (ii) traffic datasets [17]: PeMSD4 and PeMSD8 are two real-time traffic datasets from California; (iii) synthetic multivariate time-series (MTS) datasets based on vector autoregression (VAR)
[33, 48] (where the VAR model is a generalization of the univariate
AR process with more than one time-evolving component); (iv) daily surface air temperature in
CA, PA, and TX over 02/01/2020–12/31/2020; (v) Bytom token prices of Ethereum blockchain over 07/27/2017–05/07/2018; and (vi) wind speed data of 57 stations on the East Coast.
The results on (i)–(iii) are presented in the main body, and the analysis of (iv) and (v) is in Appendix C.2 and C.6.
The detailed description of each dataset is in Appendix C.1.
We also report results on the wind speed dataset in Appendix C.3.
5.2 Baselines
We compare our proposed ZFC-SHCN with 14 types of state-of-the-art baselines (SOAs), including
FC-LSTM [56], SFM [63], N-BEATS [49], DCRNN [45], LSTNet [42], STGCN [62], TCN [5],
DeepState [51], GraphWaveNet [60], DeepGLO [55], LRGCN [43] AGCRN [4], StemGNN [11], and Z-GCNETs [21].
5.3 Experimental settings
We implement ZFC-SHCN within a Pytorch framework on NVIDIA GeForce RTX 3090 GPU.
We optimize all the models using an Adam optimizer for a maximum of 200 epochs.
The learning rate is searched in { 0 . 001 , 0 . 003 , 0 . 005 , 0 . 01 , 0 . 05 } and the embedding dimension is searched in { 1 , 2 , 3 , 5 , 10 } .
Our ZFC-SHCN is trained with batch sizes of 64 and 8 on PeMSD4 and PeMSD8, respectively.
On both COVID-19 and surface air temperature datasets (i.e., CA, PA, and TX), we set the batch size to be 8.
We train two 1D convolutional layers for ZFC representation learning with the same hidden layer dimension nhid where nhid ∈ { 8 , 16 , 32 , 64 , 128 } .
For PeMSD4 and
PeMSD8, we consider the window size ω
= 12 and the horizon h
= 3 ; for both COVID-19 and surface air temperature datasets, we consider a window size ω
= 5 and horizon h = 15 ; for two simulated VAR datasets VAR T 1 and VAR T 2 , we set the window size as ω = 10 and horizon as h = 5 , and set the batch size as 8; for Bytom, we consider the window size ω = 7 and horizon h = 7 , and set the batch size as 8; for the wind speed dataset, we consider the window size ω = 12 and horizon h = 12 , and set the batch size as 8.
All models are evaluated in terms of Mean Absolute Error (MAE),
Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE).
The best results are shown in bold font and the results shown with dotted underlines are the second-best results.
We also perform a one-sided two-sample t -test between the best result and the best performance achieved by the runner-up, where *, **, *** denote p -value < 0.1, 0.05, 0.01 (i.e., denote significant, statistically significant, and highly statistically significant results, respectively.
Code is available at https://github.com/zfcshcn/ZFC-SHCN.git .
5.4 Experimental results
Real datasets
The experimental results on PeMSD4 and PeMSD8 traffic data are reported in
Table 2.
As Table 2 shows, ZFC-SHCN achieves the best MAE, RMSE, and MAPE compared with SOAs on both PeMSD4 and PeMSD8.
Compared to the RNN-based methods such as FCLSTM, SFM, N-BEATS, LSTNet, and TCN, ZFC-SHCN achieves relative gains in RMSE over the runner-ups, ranging from 17.68% to 65.41% for both PeMSD4 and PeMSD8.
In turn, DCRNN,
STGCN, GraphWaveNet, AGCRN, and StemGNN only focus on learning node-level representations.
Compared to them, ZFC-SHCN captures interactions and encodes higher-order structure correlations beyond pairwise relations among nodes and yields a relative gain from 2.06% to 5.63% in RMSE on the traffic datasets.
In addition, we compare ZFC-SHCN with the method based on the zigzag persistence image, i.e., Z-GCNETs, and find that ZFC-SHCN outperforms Z-GCNETs by 1.75% on PeMSD4 and 5.36% on PeMSD8 in terms of RMSE.
Table 3 presents COVIDhospitalization prediction results (RMSE) in CA, PA, and TX, and we observe the following findings.
[Table]
First, our proposed ZFC-SHCN achieves state-ofthe-art performance on all three datasets.
For instance, ZFC-SHCN yields 3.61%, 1.47%, 65.55% relative gains in RMSE over the runner-ups (including both GCN-based and zigzag persistence image-based methods) on three biosurveillance datasets.
These results indicate that the ZFC mechanism and higherorder representation learning module play significant roles in capturing both topological information and higher-order structures.
Second, as shown in Figurein Appendix C, we find that, compared to the runner-up (i.e., Z-GCNETs), the predicted value of
COVID-19 hospitalizations is more consistent with the ground-truth.
Finally, Tables 6 and 10 in
Appendix C present the overall prediction performances of ZFC-SHCN and representative baselines on surface air temperature and Ethereum blockchain datasets.
We find that our proposed ZFC-SHCN consistently outperforms all baselines with either a significant or (highly) statistically significant margin across all data, except surface air temperature in TX, where ZFC-SHCN still yields the best performance across all models.
Synthetic datasets
The evaluation results on two VAR datasets are summarized in Table 1.
Compared to the three strongest baselines (i.e., AGCRN, StemGNN, and Z-GCNETs), we observe that our proposed ZFC-SHCN consistently yields the best performance for all synthetic datasets.
More precisely,
ZFC-SHCN outperforms the runner-ups from 8.89% to 10.52% for VAR T 1 and VAR T 2 .
Furthermore, to assess the time-wise and high network interactions, we use the global clustering coefficient (GCC) and Euler-Poincaré characteristic (EPC) as measures of higher order substructures [6].
We find that
[Table] [Table]
for GCC for VAR T 1 and VAR T 2 are 4.96 and 5.87, respectively; while the average EPC for VAR Tand VAR T 2 are 7.47 and 6.91, respectively.
Interestingly (although it could be expected), higher GCC and lower EPC tend to be associated with higher relative gains delivered by ZFC-SHCN.
Finally, in
Appendix C.7, we present the sensitivity analysis for ZFC as a function of the covariance matrix in
VAR models.
5.5 Ablation studies
To evaluate the performance of different components in our ZFC-SHCN model, we perform an expansive ablation study.
The ablation study is conducted with three setups: (i) ZFC-SHCN without graph convolution in spatial dimension (W/o Graph convolution in spatial dimension), (ii) ZFC-SHCN without ZFC convolution (W/o ZFC convolution), and (iii) ZFC-SHCN without supra-Hodge convolution (W/o Supra-Hodge convolution).
and
[Table]
The experimental results are shown in
Table 4 and prove the validity of each component.
As Table 4 indicates, compared to ZFC-SHCN w/o ZFC convolution, the zigzag homological feature is vital for capturing the topological structure of spatio-temporal graph and our proposed graph convolution operation on ZFC significantly improves forecasting performance.
By comparing to ZFC-SHCN w/o supra-
Hodge convolution, we illustrate the significance of higher-order structure representation learning for guiding the model to how to capture information on higher-order interactions.
Also, ZFC-SHCN w/o graph convolution in spatial dimension demonstrates that the learned graph obtained from trainable weights can learn hidden information and enhance (multivariate) time-series representation learning.
5.6 Computational complexity
For higher-order simplices, the incidence matrices B 1 and B 2 can be calculated efficiently with complexity O
( N + M ) and
O ( M + Q ) respectively, where N is the number of 0-simplices (i.e., nodes)
, M is the number of 1-simplices (i.e., edges), and Q is the number of 2-simplices (i.e., filled triangles).
The computational complexity of ZFC is O
(Υ δ ) [3, 23], where Υ represents the number of points in time interval and δ ∈ [2 , 2 . 373) .
The computational complexity of the overall approach is O
( N 2 + Υ δ + Ξ k ωF
k d out + Ξ k ω 2 d out / 2 + d out (cid:80) t − 1 ℓ = t − w Ξ ( ℓ ) k +1 + W GRU ) , including (i) graph convolution in spatial dimension: O ( N 2 ) , (ii) zigzag filtration curve: O (Υ δ ) , (iii) supra-Hodge convolution in temporal dimension: O (Ξ k ωF k d out + Ξ k ω 2 d out / 2 + d out (cid:80) t − 1 ℓ = t − w Ξ ( ℓ ) k +1 )
(where
F k is the number of k -simplex attribute features, ω is the sliding window size, d out is the output dimension of the supra-Hodge convolution layer, and Ξ ( ℓ ) k
+1 is the number of ( k +1)-simplex Ξ k +1 at the ℓ -th layer), and (iv) GRU: O ( W GRU ) .
We also compare our ZFC-SHCN with the most recent approach based on multipersistence-GNN [20] (i.e., TAMP-S2GCNets).
We find that ZFC-SHCN yields either on-par or more competitive performance than TAMP-S2GCNets, while our proposed
ZFC-SHCN significantly improves the computational efficiency (see Appendix C.5 for more details).
More details about running time comparison can be found in Appendix C.6.
Conclusion
We have proposed a novel framework for time-aware deep learning of time-evolving objects which takes advantages of both the higher-order interactions among the data substructures, described as simplices, and the most intrinsic time-conditioned topological information exhibited by the object, characterized via zigzag persistent homology.
By leveraging the power of simplicial convolution operation and zigzag persistence for time-indexed data, ZFC-SHCN has been shown to demonstrate capabilities to yield the most competitive forecasting performance, while requiring fewer computational resources than its closest competitors.
Still, computational complexity and limited theoretical results on statistical inference for zigzag persistence remain one of the major existing limitations of
ZFC and, more generally, all topological methods for time dependent processes.
In the future, we plan to investigate these theoretical and methodological challenges and will extend the ZFC-SHCN idea to anomaly detection in streaming time-dependent processes.
Acknowledgments
This work was partially supported by the National Science Foundation (NSF) under awards
# ECCS 2039701 and # ECCS-2039716, the Department of the Navy, Office of Naval Research (ONR) under
ONR award # N00014-21-1-2530, C3.ai Digital Transformation Institute, and NASA AIST grant 21-AIST21_2-0059.
Part of this material is also based upon work supported by (while serving at) the NSF.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF, ONR, C3.ai DTI, or NASA.
[-where B ⊤ k B k and B k +
-]