Time Dimension Dances with Simplicial Complexes: Zigzag Filtration Curve based Supra-Hodge Convolution Networks for Time-series Forecasting
Abstract
Graph neural networks (GNNs) offer a new powerful alternative for multivariate time series forecasting, demonstrating remarkable success in a variety of spatiotemporal applications, from urban flow monitoring systems to health care informatics to financial analytics.
Yet, such GNN models pre-dominantly capture only lower order interactions, that is, pairwise relations among nodes, and also largely ignore intrinsic time-conditioned information on the underlying topology of multivariate time series.
To address these limitations, we propose a new time-aware GNN architecture which amplifies the power of the recently emerged simplicial neural networks with a time-conditioned topological knowledge representation in a form of zigzag persistence.
That is, our new approach, Zigzag Filtration Curve based
Supra-Hodge Convolution Networks (ZFC-SHCN) is built upon the two main components: (i) a new highly computationally efficient zigzag persistence curve which allows us to systematically encode time-conditioned topological information, and (ii) a new temporal multiplex graph representation module for learning higherorder network interactions.
We discuss theoretical properties of the proposed time-conditioned topological knowledge representation and extensively validate the new time-aware ZFC-SHCN model in conjunction with time series forecasting on a broad range of synthetic and real-world datasets: traffic flows, COVIDbiosurveillance, Ethereum blockchain, surface air temperature, wind energy, and vector autoregressions.
Our experiments demonstrate that the ZFC-SHCN achieves the state-of-the-art performance with lower requirements on computational costs.
Introduction
Over the last few years, graph neural networks (GNNs) have emerged as a new powerful alternative to traditional statistical and machine learning models in conjunction with univariate and multivariate time series forecasting tasks [31, 5, 44, 44, 32].
Such successful applications of GNNs range from urban traffic analytics to forecasting COVID-19 hospitalizations to electrocardiogram monitoring [4, 40, 59, 11, 21].
However, most GNNs remain inherently static and do not explicitly incorporate the inherent time characteristics of the encoded knowledge [62, 45].
In turn, limitations in capturing the time dimension in the knowledge representation and learning mechanisms for time-evolving data results in GNNs becoming less relevant over time and, hence, requiring frequent updates.
Furthermore, GNNs tend to pre-dominantly focus only on information propagation among nodes and also be limited in their ability to describe polyadic relationships among multiple substructures of multivariate time series or multi-node interactions in dynamics graphs.
However, as recently shown by [7, 22], such higher-order interactions might be the key toward better understanding of the underlying mechanisms of many real-world graph-structured phenomena.
This challenge on polyadic graph interactions has been recently addressed by [26, 9, 8] who propose to model higher order substructures as simplices.
Then, by borrowing the concepts of the Hodge theory, these approaches allow for generalization of the ideas of the combinatorial graph Laplacian which describes a diffusion from node to node via edges to a case of diffusion over simplices.
Such Hodge Laplacian construction allows for extending the notion of convolution operation to simplicial convolution, and the resulting simplicial neural networks (SNNs) are arguably one of the frontlines in graph learning today.
However, these ideas have never been yet applied in conjunction with knowledge representation and learning of time-evolving objects.
Our goal here is to bridge the emerging concept of time-aware learning with the recent notions of simplicial convolution, with a particular focus on explicitly integrating the core time-conditioned topological characteristics.
In particular, we amplify the power of SNNs with a time-conditioned topological knowledge representation in a form of zigzag persistence for time-indexed data and, more specifically, its new highly computationally efficient summary, Zigzag Filtration Curve.
As a result, our new approach, Zigzag Filtration Curve based Supra-Hodge Convolution Networks (ZFC-SHCN) enables us to systematically learn the most intrinsic time-conditioned information both on the underlying topology of the time-evolving data and higher-order interactions among various substructures.
Significance of our contributions can be summarized as follows:
• ZFC-SHCN is the first approach bringing the concepts of simplicial convolution and SNNs to time-aware learning.
• We propose a new highly computationally efficient summary of persistence for time-indexed data, Zigzag Filtration Curve, and derive its theoretical stability guarantees.
• We validate the utility of ZFC-SHCN in conjunction with forecasting multivariate time series from diverse application domains such as traffic networks, COVID-19 biosurveillance, surface air temperature, token prices on Ethereum blockchain, wind energy, and vector autoregressions.
Our findings indicate that ZFC-SHCN delivers the state-of-the-art forecasting performance, with a significant margin and demonstrates higher computational efficiency.
Related Work
Time-series Forecasting and Spatio-temporal Graph Convolutional Networks Time-series forecasting is one of the core subfields in statistical sciences [16, 10].
Most recently, there have appeared a number of unconventional machine learning approaches to time-series forecasting.
In particular, graph convolutional network (GCN)-based models for spatio-temporal network data have emerged as a promising forecasting tool.
For instance, DCRNN [45] introduces spectral graph convolution into spatio-temporal network data prediction, which can capture spatio-temporal dependencies.
STGCN [62] uses convolutional neural networks (CNNs) to model temporal correlations.
Moreover, to infer hidden inter-dependencies between different traffic variables, [60, 4, 11] conduct a convolution operation in spatial dimension through adaptive adjacency matrices.
Recent Z-GCNETs [21] develops a zigzag topological layer equipped with a zigzag persistence image into a GCN framework to model temporal correlations.
Another promising recent direction for time series forecasting beyond
GCN is a fractional-order dynamical model proposed by [31].
This approach offers an alternating scheme to determine the best estimate of the model parameters and unknown stimuli.
In turn, [32] proposes a Padé approximation based exponential neural operator (Padé Exp), aiming to improve time-series forecasting with exponential operators in neural operator learning schemes.
However, all of the above methods only focus on node-level representations.
In contrast, in this paper, we focus on both higher-order structure representation and topological information learning.
Topological Data Analysis for Graph Learning Persistent homology [27, 65] is a suite of tools within topological data analysis (TDA) that provides a way for measuring topological features of shapes and functions.
The extracted topological features have been recently shown to provide invaluable insights into hidden mechanisms behind the organization and functionality of graph structured data.
In particular, topological features have been actively used for node classification [64, 18], link prediction [61], and graph classification [35, 36, 15, 34].
For instance, [35] is one of the first approaches to integrate topological features into neural networks for graph classification, while [15] proposes a versatile framework for learning multiple vectorizations of persistent diagrams on graphs.
In turn, [64, 18, 61] apply topological features to GNNs to understand and improve the message passing between nodes.
Finally, [37] proposes a topological graph layer with learnable filtration functions for graph and node classification tasks, while [14] advances the ideas of multipersistence to graph learning.
Zigzag Persistent Homology
Despite its promise, regular persistent homology does not explicitly model the geometric and topological information from a sequence of topological spaces.
To address this limitation, a generalization of ordinary persistence, i.e., zigzag persistent homology, based on the theory of quiver representation, has been proposed by [13].
Zigzag persistence allows us to systematically describe how the homology changes over a sequence of spaces.
Despite its high potential, especially in conjunction with analysis of time-evolving data, zigzag persistence still remains largely a theoretical concept, and its applications are yet scarce.
The recent results for time-dependent data studies include, for example, zigzag-based clustering [41], bifurcation analysis of dynamic systems [57], and time series forecasting [21].
The memory and computational efficiency of zigzag persistence is one of the daunting challenges.
Inspired by [47], we propose a novel highly computationally efficient representation of zigzag persistence for learning time-evolving data, that is, zigzag filtration curve.
Simplicial Neural Networks Modeling higher-order interactions on graphs is an emerging direction in graph representation learning.
While the role of higher-order structures for graph learning has been documented for a number of years [1, 38] and involves such diverse applications as graph signal processing in image recognition [25], dynamics of disease transmission and biological networks, integration of higher-order graph substructures into deep learning on graphs has emerged only in 2020.
As shown by [7, 53], higher-order network structures can be leveraged to boost graph learning performance.
Indeed, several recent approaches [26, 52, 9, 19] propose to leverage simplicial information to perform neural networks on graphs.
However, neither of these Simplicial Neural
Networks (SNNs) are integrated with a topology-based graph convolution layer allowing us to learn both time-aware persistent topological features and simplicial geometry of graphs.
In this paper, we propose ZFC-SHCN to address this limitation.
Time-Aware Topological Learning with Zigzag Curves
Spatio-temporal Graph Construction A spatio-temporal graph is a collection of snapshots at different time steps, denoted by G = {G 1 , G 2 , · · · , G T } , where T is the maximum timestamp.
Here
G t =
( V t , E t , A t , X t ) is the graph observed at time step t ∈
[1 , T ] , where V t is a finite set of |V| = N nodes, E t is a set of edges, A t ∈ R N × N is the adjacency matrix, and X t ∈ R N × d is the node feature matrix.
Specifically, each row of X t is a d -dimensional feature vector of the corresponding node.
For sake of notations, wherever applicable below, we omit the subscript t and denote graph G t at time t as G .
Background on Ordinary Persistence Tools of ordinary persistence, or persistent homology (PH), allow us to study salient data shape patterns along various dimensions.
By shape here we broadly understand data properties that are invariant under continuous transformations, that is, transformations that do not alter “holes” in the data, for example, bending, twisting, and stretching.
The key idea is to choose some suitable scale parameter ν
and then to study a graph G not as a single object but as a nested sequence of graphs, or graph filtration G 1 ⊆ . . .
⊆
G n = G , which is induced by monotonic changes of scale ν .
For example, if G is an edge-weighted graph ( V , E , w ) with weight function w : E 7!
R , then for each ν j , j = 1 , . .
. , n , we set G ≤ ν j = ( V , E , w − 1 ( −∞ , ν j ]) , yielding the induced edge-weighted filtration.
We can also consider only induced subgraphs of G with maximal degree of ν j for each j = 1 , . .
. , n , resulting in the degree sublevel set filtration.
(For more discussion on graph filtrations see [34].)
Armed with this construction, we can track which shape patterns, for example, independent components, loops, and voids, emerge as the scale ν varies.
To make the process of pattern counting more systematic and efficient, we build an abstract simplicial complex K ( G j ) on each G j .
We also record complex indices j b (birth) and j d (death) at which we first or last observe each shape feature.
Topological features with longer lifespans are said to persist and are likelier to yield important information on the structural organization of G .
Learning Shapes of Time-Conditioned Data with Zigzag Persistence This construction enables us to extract the key topological descriptors from a single graph G .
However, in our case, we observe not a single graph but a sequence of time-evolving graphs {G 1 , . . .
, G T } .
How can we track shape signatures which are not just individualistic for each time stamp but characterize intrinsic properties of the observed object over time?
One approach to how we can bring PH tools to analysis of timeconditioned objects is zigzag persistence.
Based on the theory of quiver representations, zigzag persistence generalizes ordinary persistence to track characteristics of graphs (or other topological spaces) with inclusions going in different directions [13, 12].
In particular, given a time-indexed sequence of graphs {G 1 , . . .
, G T } , we first form a set of graph inclusions over time
G
[Equation]
↗
[Equation]
and then assess the compatibility of persistent topological features across unions of graphs.
That is, we record indices at which topological features (dis)appear, for some given scale ν ∗ .
If for a given ν ∗ topological feature ρ (i.e., p -dimensional hole, 0 ≤ p ≤ K , where K is the dimension of the simplicial complex K ( G ) ) is first recorded in K ( G j ) , we say that the feature’s birth is j , and if ρ first appears in K ( G j ∪ G j +1 ) , we record its birth as j + 1 / 2 .
In turn, if ρ is last seen in K ( G j ) , we record its death as j , while if it is last seen in K ( G j ∪ G j +1 ) , we say that its death is at j + 1 / 2 .
Let J be the set of all observed topological features for a given ν ∗ .
Collecting then births and deaths over J , we summarize all extracted information as a multiset D ν ∗ = { ( b ρ , d ρ ) ∈ R 2 | b ρ < d ρ , ρ ∈ J } , called a zigzag persistent diagram (ZPD) (where b ρ and d ρ are the birth and death of the topological feature ρ respectively).
This makes zigzag persistence particularly attractive for the analysis of dynamic objects which are naturally indexed by time.
However, the idea of zigzag persistence is applicable far beyond learning time-evolving objects.
Nevertheless, zigzag persistence still remains largely a theoretical concept, with yet only a handful of applications, and one of the roadblocks hindering a broader proliferation of zigzag-based methods in practice is their computational costs.
Here we take a step toward bringing a more computationally efficient summary of zigzag persistence to real-world applications.
Time-Aware Zigzag Filtration Curves Consider a sequence of time intervals associated with a zigzag filtration over a time period [ t 1 , t N ]
[Equation]
Let
DgmZZ ν
∗ be the resulting ZPD for a given ν ∗ and M be the number of off-diagonal topological features in ZPD, i.e., DgmZZ ν ∗ .
Inspired by the recent results on stabilized Betti sequences by [39] and filtration curves by [47] for ordinary persistence, we propose a new simple and computationally efficient summary of zigzag persistence, called a Zigzag Filtration Curve .
Definition 3.1 (Zigzag Filtration Curve (ZFC)) .
The zigzag filtration curve evaluated at ∆ t − i =
( t i − 1 + 12 , t i ) , i = { 1 , 2 , . . .
, N} , for a given ν ∗ , is defined as
[Equation]
where ( t b j , t d j ) ∈ R 2 is a vector containing the birth and death of the j -th off-diagonal p -dimensional topological feature in DgmZZ ν ∗ (as such, t b j < t d j ), j = { 1 , 2 , . .
, M} , 0 ≤ p ≤ K ; ξ i : R 2 7!
R is some suitable Lipschitz continuous function with Lipschitz constant L i , for example, a Gaussian density; and ω i > 0 , i = { 1 , 2 , . . .
, N} are weights such that (cid:80) i ω i
= 1 .
Zigzag filtration curve at ∆ t
+ i =
( t i , t i + 12 ) is defined analogously.
(For the sake of notational simplicity, wherever applicable in the further exposition we suppress the index p in ZFC.)
Motivated by [39], here as the Lipschitz continuous function ξ i for intervals ∆ t − i , we use a Gaussian density f with mean ( t i − 1 + 1 / 2 , t i ) , while for intervals ∆ t
+ i , we set the mean of f to ( t i , t i + 1 / 2) , i = 1 , 2 , . . .
N .
For both ∆ t − i and ∆ t + i , we choose the 2 × 2 -variance-covariance matrix Σ to be the identity matrix.
(See Appendix C.7 for more discussion on sensitivity analysis.)
Another suitable choice of ξ is the arctan function.
As we show below, the proposed ZPC also enjoys important theoretical stability guarantees in terms of Wasserstein-1 distance.
Proposition 3.2 (Stability of Zigzag Filtration Curve) .
Let
DgmZZ ν ∗ be a zigzag persistence diagram and DgmZZ ′
c ∗ be its perturbed copy such that W 1 (cid:0) DgmZZ ν ∗ , DgmZZ ′ ν ∗ ) <
ϵ , where W 1 is
Wasserstein-1 distance.
Then, ZFC is stable with respect to Wasserstein-1 distance.
In practice topological features of various dimensions p , p = 0 , 1 , . . .
, K , may play different roles in the learning task performance, and these roles are not known a-priori.
Hence, to harness timeconditioned information encoded in ZFC corresponding to different dimensions p , we propose MultiZigzag Filtration Curves (M-ZFCs)
M-ZFCs
[Equation]
by stacking
[Equation]
Figure 1 in Appendix B shows the both 0- and 1-dimensional ZFCs obtained from the proposed ZFC.
In the following section, we demonstrate how ZFC can be integrated into neural network architectures for graph learning tasks.
Zigzag Filtration Curve Based Supra-Hodge Convolution Networks
Given a graph G and its historical ω step graph signals X ω
= { X t − ω
+1 , . . .
, X t } ∈ R ω × N × F ( F is the node feature dimensionality), the time-series forecasting problem is to learn a mapping function f that maps the historical data { X t − ω +1 , . . .
, X t } into the next h step data { X t +1 , . . .
, X t + h } .
The mapping relation is represented as follows: { X t − ω +1 , . . .
, X t } f −!
{ X t +1 , . . .
, X t + h } .
4.1 Graph convolution in the spatial dimension
Given the node embedding dictionary W ϕ =
( w ϕ 1 , w ϕ 2 , . . .
, w ϕN ) ∈ R N × d c (where x ϕu ∈ R d c and d c is the dimension of node embedding), we aim to seek a non-negative function S u,v = G ( w ϕu , w ϕv ) which represents the pairwise similarity between any two nodes u and v .
Concretely, the multiplication between W ϕ and ( W ϕ )
⊤ can (i) give a sum pooling of second-order features from the outer product of all the embedding vector pairs ( w ϕu , w ϕv ) and (ii) infer the hidden spatial dependencies of nodes
[Equation]
where ReLU ( · ) = max (0 , · ) is a nonlinear activation function, which is used to eliminate weak connections proactively, and the role of the softmax function is applied to normalize the learned graph
S .
Inspired by the recent advancements in random walk-based graph embedding learning [50, 29], we make a graph convolution in spatial dimension, feeding a power series of the learned graph S with varying random walk steps { 1 , 2 , · · · , r } ( r ∈ Z + ), as follows:
[Equation]
where σ ( · ) stands for a nonlinear activation function, Stack ( · ) is the function which stacks r powered learned graphs, H ( ℓ ) t, GC and H ( ℓ +1) t, GC are the input and output activations for layer ℓ (where H (0) t, GC =
X t ∈ R N × F ), and Θ ( ℓ ) GC ∈ R d
GC ℓ × d GC ℓ +1 is the ℓ -th layer’s trainable weights.
Next, we introduce representation learning of the higher-order graph (sub)structures using the supra-Hodge Laplacian which allows us to systematically leverage the underlying topological information.
4.2 Supra-Hodge convolution in temporal dimension
Time-evolving data such as multivariate time series, spatio-temporal processes, and dynamic networks, often exhibit a highly complex dependency among its substructures that goes far beyond what can be described by dyadic (or pairwise) interactions among nodes.
Instead, such higher-order polyadic interactions can be systematically addressed using the Hodge theory.
In particular, the discrete Hodge theory allows us to generalize the notion of a standard combinatorial graph Laplacian which describes diffusion on graph G from node to node via edges to diffusion over higher-order substructures of
G [46, 7].
In turn, higher-order substructures can be modeled as k -simplices of G .
(See Appendix B for background information on Hodge Laplacians.)
Convolutional architectures on simplicial complexes based on the associated concepts of the Hodge theory have emerged as a recent direction in graph neural networks but have not yet been applied to learning time-evolving data.
Our goal here is to introduce the notion of simplicial convolution and the ideas of Hodge-Laplacians to time-aware learning.
In particular, to capture time-conditioned higher-order interactions on G and to describe diffusion of information over simplices along the temporal dimension, we build a supra-Hodge convolution operation, based on the multiplex network representation learning.
(In the following for simplicity, notation without sub/superscript k stands for node-level quantities and in our experiments we always consider k ∈ Z + ).
First, given the historical spatio-temporal network series G t − ω +1:
t = {G t − ω
+1 , G t − ω +2 , . . .
, G t } , we consider a directed connected node-aligned multiplex network, which is made up of ω layers with N nodes on each layer.
That is, the adjacency matrix A α
= { a αuv } N × N (where α ∈ { t − ω + 1 , . . .
, t } ) defines the intra-connection between nodes u and v in layer α and a distance matrix D αβ
= { d αβuu } N × N quantifies the transition probability of moving from node u of layer α to node u of layer β .
(Here β > α , since we consider information diffusion procedures only along the temporal dimension).
Next, based on the discrete Hodge theory, we propose a new Hodge k -Laplacian for multiplex graphs, called the supra-Hodge k -Laplacian
L Supk ∈ R ϕ k ω × ϕ k ω
[Equation]
··· ( L ωωk ) r   , (2) where L ααk is the Hodge k -Laplacian in layer α , D k +1 is the diagonal matrix of degrees of each k -simplex, i.e., D k +1 = max ( diag ( | B k +1 | 1 , I )) and B k +1 is the k -simplex-to-
( k + 1) -simplex incidence matrix, and the r -th power of L ααk represents r -step random walk on the Hodge k -Laplacian of layer α which will allow every k -simplex to accumulate information from its neighbors.
Hence, when k = 1 , we can infer the spatial dependencies between each pair of edges and capture meaningful edge information in both spatial and temporal dimensions – through the lens of the supra-Hodge 1-Laplacian.
For instance, in molecule networks, each node represents an atom and each edge is a bond connecting two atoms; the bond (i.e., edge) features include bond type, ring status, and molecular charge which are closely related to atom (i.e., node) features (such as atomic total and partial charges).
Since the goal of the forecasting task is to predict node (i.e., 0-simplex) attribute(s) in the next few time steps, we propose a novel diffusion supra-Hodge convolution on the sliding window G t − ω +1: t .
We then update nodes’ representations by transforming the multiplex k -simplex embedding to nodes via incidence matrices
[Equation] [Equation]
[Equation]
where (i) in Equation 3: Θ ( ℓ ) k, SH ∈ R d SH k ; ℓ × d SH k ; ℓ +1 is a learnable filter matrix for layer ℓ (here d SH k ; ℓ and d SH k ; ℓ +1 are the intermediate and output dimensions to the ℓ -th layer), H ( ℓ ) t,k, SH and
H ( ℓ +1) t,k, SH are the input and output activations for layer ℓ (where H (0) t,k, SH =
¯ X k ; t − ω +1: t ∈
R ϕ k ω ×
d in k and the historical k -simplex features of the spatio-temporal networks X k ; t − ω +1:
t = { X k ; t − ω +1 , X k ; t − ω +2 , . . .
, X k ; t } ∈ R ϕ k × ω
× d in k is reshaped as a matrix ¯ X k ; t − ω +1: t with shape ϕ
k ω
×
d in k ) and (ii) in Equation 4: we transform the k -simplex embedding H ( ℓ +1) t,k, SH to node embedding H ( ℓ +1) t,
SH ∈
R N × d SH k ; ℓ +1 through incidence matrices.
4.3 ZFC convolution: a bridge between spatial and time dimensions
Armed with the representation learning of graph (sub)structures at each timestamp, we now discuss the
ZFC convolution which allows us to preserve and propagate both spatial and time-aware topological information simultaneously.
The intuition behind ZFC convolution is that it learns a strong connection between two dimensions via two 1D convolution layers, i.e., time-wise and node-wise.
ZFC convolution consists of three key components: (i) a linear embedding on M-ZFCs, which can learn the importance of time-aware topological features for each node to form a time-dimension-specific node embedding; (ii) a time-wise 1D convolution layer, where it gathers time-aware topological features from the entire space into a compact set; (iii) a node-wise 1D convolution layer, which can capture relations between different nodes.
The resulted ZFC convolution operation over a M-ZFCs ω is defined as
[Equation]
where ω is the size of the window for sequence learning, M-ZFCs ω denotes the M-ZFCs feature extracted from the time window with size ω , Θ M-ZFC ∈ R N ×
d q is a weight matrix to be learned, F θ and F ψ are 1D convolutional layers, and H t, M-ZFC ∈ R N × d M-ZFCout is the d M-ZFCout -dimensional output.
We then combine the embeddings from graph convolution, M-ZFCs convolution, and supra-Hodge convolution to get the final embedding H ( ℓ +1) t, out
[Equation]
(6) where [ · , · , · ] denotes the concatenation of the outputs from three convolution operations, and
H ( ℓ +1) t, out ∈ R N × d out (where d out = d
GC ℓ +1 + d ZFCout + d SH ℓ +1 ).
4.4 Gate Recurrent Unit with ZFC-SHCN
To describe the complex spatio-temporal dependencies among time series and assess a hidden state of nodes at a future timestamp, we feed the final embedding H ( ℓ +1) t, out into Gated Recurrent Units (GRUs).
Formally, we set the forward propagation equations of the GRUs as ℜ t
[Equation]
where η ( · ) is an activation function (e.g., ReLU, LeakyReLU), ⊙ is the elementwise product, ℜ t is the update gate and ℑ i is the reset gate.
Here b ℜ , b ℑ , b Ψ , W ℜ , W ℑ , and W Ψ are learnable parameters, while (cid:104) Ψ t − 1 , H ( ℓ +1) t, out (cid:105) and Ψ t are the input and output of GRU model, respectively.
We then obtain ˜ Ψ t which contains both the spatio-temporal and time-aware information.
Experiments 5.1 Datasets
We validate our ZFC-SHCN model on six diverse data types: (i) COVID-19 datasets [54]: CA, PA, and TX represent the number of COVID-19 hospitalizations in California (CA), Pennsylvania (PA), and Texas (TX) respectively; (ii) traffic datasets [17]: PeMSD4 and PeMSD8 are two real-time traffic datasets from California; (iii) synthetic multivariate time-series (MTS) datasets based on vector autoregression (VAR)
[33, 48] (where the VAR model is a generalization of the univariate
AR process with more than one time-evolving component); (iv) daily surface air temperature in
CA, PA, and TX over 02/01/2020–12/31/2020; (v) Bytom token prices of Ethereum blockchain over 07/27/2017–05/07/2018; and (vi) wind speed data of 57 stations on the East Coast.
The results on (i)–(iii) are presented in the main body, and the analysis of (iv) and (v) is in Appendix C.2 and C.6.
The detailed description of each dataset is in Appendix C.1.
We also report results on the wind speed dataset in Appendix C.3.
5.2 Baselines
We compare our proposed ZFC-SHCN with 14 types of state-of-the-art baselines (SOAs), including
FC-LSTM [56], SFM [63], N-BEATS [49], DCRNN [45], LSTNet [42], STGCN [62], TCN [5],
DeepState [51], GraphWaveNet [60], DeepGLO [55], LRGCN [43] AGCRN [4], StemGNN [11], and Z-GCNETs [21].
5.3 Experimental settings
We implement ZFC-SHCN within a Pytorch framework on NVIDIA GeForce RTX 3090 GPU.
We optimize all the models using an Adam optimizer for a maximum of 200 epochs.
The learning rate is searched in { 0 . 001 , 0 . 003 , 0 . 005 , 0 . 01 , 0 . 05 } and the embedding dimension is searched in { 1 , 2 , 3 , 5 , 10 } .
Our ZFC-SHCN is trained with batch sizes of 64 and 8 on PeMSD4 and PeMSD8, respectively.
On both COVID-19 and surface air temperature datasets (i.e., CA, PA, and TX), we set the batch size to be 8.
We train two 1D convolutional layers for ZFC representation learning with the same hidden layer dimension nhid where nhid ∈ { 8 , 16 , 32 , 64 , 128 } .
For PeMSD4 and
PeMSD8, we consider the window size ω
= 12 and the horizon h
= 3 ; for both COVID-19 and surface air temperature datasets, we consider a window size ω
= 5 and horizon h = 15 ; for two simulated VAR datasets VAR T 1 and VAR T 2 , we set the window size as ω = 10 and horizon as h = 5 , and set the batch size as 8; for Bytom, we consider the window size ω = 7 and horizon h = 7 , and set the batch size as 8; for the wind speed dataset, we consider the window size ω = 12 and horizon h = 12 , and set the batch size as 8.
All models are evaluated in terms of Mean Absolute Error (MAE),
Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE).
The best results are shown in bold font and the results shown with dotted underlines are the second-best results.
We also perform a one-sided two-sample t -test between the best result and the best performance achieved by the runner-up, where *, **, *** denote p -value < 0.1, 0.05, 0.01 (i.e., denote significant, statistically significant, and highly statistically significant results, respectively.
Code is available at https://github.com/zfcshcn/ZFC-SHCN.git .
5.4 Experimental results
Real datasets
The experimental results on PeMSD4 and PeMSD8 traffic data are reported in
Table 2.
As Table 2 shows, ZFC-SHCN achieves the best MAE, RMSE, and MAPE compared with SOAs on both PeMSD4 and PeMSD8.
Compared to the RNN-based methods such as FCLSTM, SFM, N-BEATS, LSTNet, and TCN, ZFC-SHCN achieves relative gains in RMSE over the runner-ups, ranging from 17.68% to 65.41% for both PeMSD4 and PeMSD8.
In turn, DCRNN,
STGCN, GraphWaveNet, AGCRN, and StemGNN only focus on learning node-level representations.
Compared to them, ZFC-SHCN captures interactions and encodes higher-order structure correlations beyond pairwise relations among nodes and yields a relative gain from 2.06% to 5.63% in RMSE on the traffic datasets.
In addition, we compare ZFC-SHCN with the method based on the zigzag persistence image, i.e., Z-GCNETs, and find that ZFC-SHCN outperforms Z-GCNETs by 1.75% on PeMSD4 and 5.36% on PeMSD8 in terms of RMSE.
Table 3 presents COVIDhospitalization prediction results (RMSE) in CA, PA, and TX, and we observe the following findings.
[Table]
First, our proposed ZFC-SHCN achieves state-ofthe-art performance on all three datasets.
For instance, ZFC-SHCN yields 3.61%, 1.47%, 65.55% relative gains in RMSE over the runner-ups (including both GCN-based and zigzag persistence image-based methods) on three biosurveillance datasets.
These results indicate that the ZFC mechanism and higherorder representation learning module play significant roles in capturing both topological information and higher-order structures.
Second, as shown in Figurein Appendix C, we find that, compared to the runner-up (i.e., Z-GCNETs), the predicted value of
COVID-19 hospitalizations is more consistent with the ground-truth.
Finally, Tables 6 and 10 in
Appendix C present the overall prediction performances of ZFC-SHCN and representative baselines on surface air temperature and Ethereum blockchain datasets.
We find that our proposed ZFC-SHCN consistently outperforms all baselines with either a significant or (highly) statistically significant margin across all data, except surface air temperature in TX, where ZFC-SHCN still yields the best performance across all models.
Synthetic datasets
The evaluation results on two VAR datasets are summarized in Table 1.
Compared to the three strongest baselines (i.e., AGCRN, StemGNN, and Z-GCNETs), we observe that our proposed ZFC-SHCN consistently yields the best performance for all synthetic datasets.
More precisely,
ZFC-SHCN outperforms the runner-ups from 8.89% to 10.52% for VAR T 1 and VAR T 2 .
Furthermore, to assess the time-wise and high network interactions, we use the global clustering coefficient (GCC) and Euler-Poincaré characteristic (EPC) as measures of higher order substructures [6].
We find that
[Table] [Table]
for GCC for VAR T 1 and VAR T 2 are 4.96 and 5.87, respectively; while the average EPC for VAR Tand VAR T 2 are 7.47 and 6.91, respectively.
Interestingly (although it could be expected), higher GCC and lower EPC tend to be associated with higher relative gains delivered by ZFC-SHCN.
Finally, in
Appendix C.7, we present the sensitivity analysis for ZFC as a function of the covariance matrix in
VAR models.
5.5 Ablation studies
To evaluate the performance of different components in our ZFC-SHCN model, we perform an expansive ablation study.
The ablation study is conducted with three setups: (i) ZFC-SHCN without graph convolution in spatial dimension (W/o Graph convolution in spatial dimension), (ii) ZFC-SHCN without ZFC convolution (W/o ZFC convolution), and (iii) ZFC-SHCN without supra-Hodge convolution (W/o Supra-Hodge convolution).
and
[Table]
The experimental results are shown in
Table 4 and prove the validity of each component.
As Table 4 indicates, compared to ZFC-SHCN w/o ZFC convolution, the zigzag homological feature is vital for capturing the topological structure of spatio-temporal graph and our proposed graph convolution operation on ZFC significantly improves forecasting performance.
By comparing to ZFC-SHCN w/o supra-
Hodge convolution, we illustrate the significance of higher-order structure representation learning for guiding the model to how to capture information on higher-order interactions.
Also, ZFC-SHCN w/o graph convolution in spatial dimension demonstrates that the learned graph obtained from trainable weights can learn hidden information and enhance (multivariate) time-series representation learning.
5.6 Computational complexity
For higher-order simplices, the incidence matrices B 1 and B 2 can be calculated efficiently with complexity O
( N + M ) and
O ( M + Q ) respectively, where N is the number of 0-simplices (i.e., nodes)
, M is the number of 1-simplices (i.e., edges), and Q is the number of 2-simplices (i.e., filled triangles).
The computational complexity of ZFC is O
(Υ δ ) [3, 23], where Υ represents the number of points in time interval and δ ∈ [2 , 2 . 373) .
The computational complexity of the overall approach is O
( N 2 + Υ δ + Ξ k ωF
k d out + Ξ k ω 2 d out / 2 + d out (cid:80) t − 1 ℓ = t − w Ξ ( ℓ ) k +1 + W GRU ) , including (i) graph convolution in spatial dimension: O ( N 2 ) , (ii) zigzag filtration curve: O (Υ δ ) , (iii) supra-Hodge convolution in temporal dimension: O (Ξ k ωF k d out + Ξ k ω 2 d out / 2 + d out (cid:80) t − 1 ℓ = t − w Ξ ( ℓ ) k +1 )
(where
F k is the number of k -simplex attribute features, ω is the sliding window size, d out is the output dimension of the supra-Hodge convolution layer, and Ξ ( ℓ ) k
+1 is the number of ( k +1)-simplex Ξ k +1 at the ℓ -th layer), and (iv) GRU: O ( W GRU ) .
We also compare our ZFC-SHCN with the most recent approach based on multipersistence-GNN [20] (i.e., TAMP-S2GCNets).
We find that ZFC-SHCN yields either on-par or more competitive performance than TAMP-S2GCNets, while our proposed
ZFC-SHCN significantly improves the computational efficiency (see Appendix C.5 for more details).
More details about running time comparison can be found in Appendix C.6.
Conclusion
We have proposed a novel framework for time-aware deep learning of time-evolving objects which takes advantages of both the higher-order interactions among the data substructures, described as simplices, and the most intrinsic time-conditioned topological information exhibited by the object, characterized via zigzag persistent homology.
By leveraging the power of simplicial convolution operation and zigzag persistence for time-indexed data, ZFC-SHCN has been shown to demonstrate capabilities to yield the most competitive forecasting performance, while requiring fewer computational resources than its closest competitors.
Still, computational complexity and limited theoretical results on statistical inference for zigzag persistence remain one of the major existing limitations of
ZFC and, more generally, all topological methods for time dependent processes.
In the future, we plan to investigate these theoretical and methodological challenges and will extend the ZFC-SHCN idea to anomaly detection in streaming time-dependent processes.
Acknowledgments
This work was partially supported by the National Science Foundation (NSF) under awards
# ECCS 2039701 and # ECCS-2039716, the Department of the Navy, Office of Naval Research (ONR) under
ONR award # N00014-21-1-2530, C3.ai Digital Transformation Institute, and NASA AIST grant 21-AIST21_2-0059.
Part of this material is also based upon work supported by (while serving at) the NSF.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF, ONR, C3.ai DTI, or NASA.
Appendix A Proof of Proposition 3.
Let η 0 : DgmZZ ν ∗ 7!
DgmZZ ′
ν ∗ be the bijective map that yields the Wasserstein-1 distance between
DgmZZ ν ∗ and its perturbed copy DgmZZ ′ ν
∗ .
That is, Wasserstein-1 distance is defined as
[Equation]
, where || · || ∞ is a norm in L ∞ , i.e., || z || ∞ =
max i |
z i | , Γ = { ( t, t ) | t ∈ R } and ν ranges over all bijections between DgmZZ ν ∗ ∪ Γ and DgmZZ ′ ν ∗ ∪ Γ .
Inclusion of the diagonal Γ ensures that the set of bijections is non-empty even under different cardinalities of DgmZZ ν ∗ and DgmZZ ′ ν ∗ .
Let ZFC ν ∗ (∆ t − i ) and ZFC ′
ν ∗ (∆ t − i ) be Zigzag Filtration Curves corresponding to the zigzag persistence diagrams DgmZZ ν ∗ and DgmZZ ′
ν
∗ , respectively, and evaluated at ∆ t − i .
Then, similarly to [39], for each i = { 1 , 2 , . . .
, N} , we have
[Equation] [Equation]
Here in the second inequality, we take into account that ξ i is a Lipschitz continuous function with constant L i .
A similar result holds for ZFC ν ∗ (∆ t + i ) and ZFC ′
ν ∗ (∆ t + i ) , i = { 1 , 2 , . . . , N} .
Hence,
[Equation]
where C = max { i =1 , 2 ,..., N} ω i L i and ZFC ν ∗ and ZFC ′
ν ∗ are viewed as (2 n − 1) -dimensional vectors (i.e., ZFC is evaluated at each interval ∆ t + i and ∆ t − i , i = 1 , 2 , . . .
, N ), which concludes the proof.
B Background on the Hodge Theory and Hodge-Laplacians
Let C k be a real-valued vector space which is endowed with basis from the oriented k -simplices.
(By orientation of simplices, we mean selecting some (arbitrary) order for its nodes, where two orderings are said to be equivalent if they differ by an even permutation.)
A linear map ∂ k : C k !
C k −is called a boundary operator.
The adjoint of the boundary map induces the co-boundary operator ∂
Tk : C k !
C k +1 .
We define matrix representations of ∂ k and ∂ ⊤ k as B k and B ⊤ k , respectively.
Definition B.1.
An operator over oriented k -simplices
L k : C k !
C k is called the k -Hodge
Laplacian , and its matrix representation is given by
[Equation]
where B ⊤ k B k and B k +
B
⊤
k +1 are often referred to L downk and L upk , respectively.
That is, the standard graph Laplacian L 0 =
B 1 B ⊤ 1 ∈ R N × N is a special case of the above k -th combinatorial Hodge Laplacian and the matrix L 1 ∈ R M × M is the Hodge 1-Laplacian.
C Additional Experiments and Running Time C.1 Datasets
Detailed data description is as follows:
The COVID-19 datasets include the daily number of hospitalizations from February 1,to December 31, 2021 on a county level in three US states (i.e., CA, PA, and TX), where each node denotes a county and each edge represents a border connection between two adjacent counties.
In our experiments, we split the COVID-19 datasets into training and test sets in chronological order with the split ratio 8 : 2 for all three states.
The traffic datasets, i.e., PeMSD4 and PeMSD8 consist of the traffic flow data in California from January 1, 2018 to February 28, 2018 and from January 7, 2016 to August 31, 2016, respectively.
Moreover, both PeMSD4 and PeMSD8 are aggregated to 5 minutes, which means there are 12 time points in the flow data per hour, and we split the traffic datasets with ratio 6 : 2 : 2 into training, validation, and test sets by following the same setting of [30].
The synthetic MTS datasets include two MTS following vector autoregressions VAR T 1 : { Ξ t } T 1 t =1 = { ξ 1 t , . . .
, ξ pt } T 1 t =1 and VAR T 2 : { Ξ t } T 2 t =1 = { ξ 1 t , . . . , ξ pt } T 2 t =1 with the graph node size p = 20 for two different time series lengths ( T 1 = 200 and T 2 = 300 ).
Note that
[Table]
for two VAR datasets we split into training and test sets with the ratio 8 : 2 .
We define the ω -lag vector autoregressive (VAR( ω ))
Ξ t = { ξ 1 t , . . . , ξ pt } ⊤ as follows
[Equation]
where Π i ∈
R q × q is an coefficient matrix and ϵ t ∈ R q × 1 denotes an unobservable mean white noise vector process with time invariant covariance matrix Σ .
In our experiments, we set q = 20 and ω = 10 .
The surface air temperature datasets include the daily surface air temperature from February 1, 2020 to December 31, 2021 on a county level in three US states (i
e., CA, PA, and TX).
Besides, we use the same experimental settings as the COVID-19 datasets.
The Ethereum blockchain Bytom networks are compound of addresses of users (nodes) and daily transactions among users (edges) [24, 2].
Since original token networks have an average of 442788/1192722 nodes/edges, we compute a subgraph via a maximum weight subgraph approximation [58] using the amount of transactions
as weight.
The dynamic networks contain different number of networks since every token was created at different days.
Hence, for each timestamp t (from July 27, 2017 to May 07, 2018), given the dynamic network G t =
{V t , E t , ˜ W t } and its corresponding node feature matrix X t ∈ R N × F , where
F represents the number of features (e.g., node degree, node betweenness, etc), we test our algorithm with both node and edge features and use the set of most active nodes, i.e.,
N = 100 .
C.2 Surface Temperature Prediction Performance
We further compare our ZFC-SHCN with the 3 strongest SOA baselines for surface temperature forecasting on CA, PA, and TX in Table 6.
The results show that ZFC-SHCN achieves relative gains of 39.55%, 6.02%, and 15.59% over runner-ups (see the results with dotted underline ) on CA, PA, and TX respectively.
[Table]
C.3 Wind Speed Prediction Performance
To assess the utility of our approach for other tasks, we now also conducted spatio-temporal forecasting for graph values (i.e., on graph-level) on the wind speed dataset [28].
In this experiment, we use hourly wind speed data to predict the total wind speed from 57 stations in East Coast including
Massachusetts, Connecticut, New York, and New Hampshire.
The evaluation results are summarized in Table 3.
We find that our ZFC-SHCN outperforms the next best competitor, with up to 39.20% of improvement, and the results are highly statistically significant.
[Table]
C.4 Additional Ablation Studies
We conduct the additional experiments by (i) replacing ZFC with zigzag persistence image (ZPI), i.e., ZPI-SHCN and (ii) replacing Supra-Hodge convolution with GCN (which is temporal graph convolution in temporal domain), i.e., ZFC-GCN, on both PeMDS4 and TX-COVID-19 (in Texas state) datasets.
First, Table 8 demonstrates that for both datasets, the ZFC representation (i.e.,
ZFC-SHCN) significantly outperforms zigzag persistence image (i.e., ZPI-SHCN).
These findings might be explained by the fact that, especially for small- and medium-size datasets, ZFC tends to better capture the topological signal than ZPI which, being a 2-d representation, might be more noisy.
Second, Table 8 indicates that compared to ZFC-GCN, ZFC-SHCN always achieves better forecasting performance.
We attribute these findings to the fact that supra-Hodge convolution operation integrates knowledge on important interactions among higher-order structures (i.e., edges, filled trianglesbeyond the node-space) into graph learning.
For TX-COVID-19 dataset, the relative gain of ZFCSHCN over the runner-up (i.e., ZFC-GCN) in RMSE is 3.13%; for PeMSD4 dataset, the relative gain of ZFC-SHCN over the runner-up (i.e., ZFC-GCN) in RMSE is 1.11%.
[Table]
Regarding the choice of simplicial complexes, we have also conducted additional experiments between (i) ZFC-SH 1 CN based on Supra-Hodge 1-Laplacian (i.e., utilizing information of 0-simplices, 1simplices, and 2-simplices) and (ii) ZFC-SH 2 CN based on Supra-Hodge 2-Laplacian (i.e., utilizing information of 1-simplices, 2-simplices, and 3-simplices) on PeMSD4 and COVID-19 in TX dataset.
The evaluation results are summarized in Table 9.
Experimental results show that ZFC-SH 1 CN consistently outperforms ZFC-SH 2 CN.
We tend to attribute such findings to the fact that we observe a very low number of 3-simplices.
For instance, in PeMSD4 at each timestamp we observe around2-simplices (filled triangles) but we observe only around 2 3-simplices (tetrahedron).
Furthermore, it is worth to mention that: for PeMSD4 dataset, ZFC-SH 2 CN is still very competitive compared to other baselines (i.e., slightly worse than the runner-up, i.e., Z-GCNETs); for COVID-19 in TX, the performance of ZFC-SH 2 CN is much better the runner-up.
Nevertheless, we believe that higher-order simplices, if observed in the targeted network, has a potential to boost the model performance.
[Table]
C.5 Comparison to Multiparameter Persistence
The development of multiparameter persistence (MP) has recently witnessed a rapid progress (such as [14, 20]) due to in many applications, particularly, involving spatio-temporal processes, the data exhibit richer structures which cannot be well encoded with a single parameter persistence.
Although there exist some studies in the literature that have shown promising performances for spatial temporal prediction tasks [20], the computation of MP is labor-intensive and very time-consuming.
To better demonstrate the power of our proposed ZFC-SHCN, here we compare our method with one representative MP-based GNNs, i.e., TAMP-S2GCNets [20] from both prediction performance and running time perspectives.
We conduct comparison experiments on PeMSD4, CA, and Bytom, and we find that ZFC-SHCN yields highly competitive performance against TAMP-S2GCNets (see Figures 3a and 4a); that is, for PeMSD4 , ZFC-SHCN: {MAE: 17.58 ± 0.17, RMSE: 28.58 ± 0.15, MAPE (%): 11.68 ± 0.07} vs. TAMP-S2GCNets: {MAE: 17.58 ± 0.20 RMSE: 28.56 ± 0.28, MAPE (%) 11.01}; for CA : ZFC-SHCN: {MAE: 79.33 ± 1.70, RMSE: 343.92 ± 3.66, MAPE (%): 61.25 ± 5.80} vs.
TAMP-S2GCNets: {MAE: 82.75 ± 1.93, RMSE: 371.60 ± 2.68, MAPE (%): 62.43 ± 5.61}; for Bytom ,
ZFC-SHCN: {MAPE: 29.51 ± 0.77} vs. TAMP-S2GCNets: {MAPE: 29.26 ± 1.06}).
Furthermore,
Figures 3b and 4b present visual comparison of running time of ZFC-SHCN and TAMP-S2GCNets (per epoch in seconds).
We find that training time consumed by ZFC-SHCN is substantially lower than that of TAMP-S2GCNets on all datasets.
These findings suggest that ZFC-SHCN may be the preferred forecasting choice, in terms of the balance between accuracy and computational efficiency.
[Figure]
C.6 Overall Comparison with Runner-ups
We present comparison of computational time of our ZFC-SHCN with the 3 strongest SOA baselines on PeMSD4, CA, and Bytom in Table 10.
The results show the superior computational efficiency of ZFC-SHCN.
Note, that while AGCRN is faster than ZFC-SHCN, ZFC-SHCN significantly outperforms AGCRN in terms of forecasting accuracy, e.g., with a relative gain of up to 48.81% on
MAPE for CA.
Furthermore, from Tables 11 and 12, we find both the running times on zigzag-based topological feature generation and per-epoch training time of ZFC-SHCN are less than the runner-up model (i.e., Z-GCNETs).
Furthermore, we have conducted comparison experiments between our ZFC-SHCN and two baselines, i.e., fractional-order dynamical model [31] and Padé Exp [32] on COVID-19 (TX) dataset.
Tableshows that ZFC-SHCN outperforms both fractional-order dynamical model and Padé Exp.
[Figure]
[Table]
C.7 Sensitivity analysis of the covariance matrix
To further investigate the effect of covariance matrix in ZFC’s representational power
[Table]
we perform the sensitivity analysis of covariance matrix Σ of ZFC
[Table]
which specifies the patterns of variability as well as the shape of the multivariate Gaussian distribution
We show that the variation of RMSE for COVID-19 hospitalization prediction task with Σ (note that, we assume Σ is the same for every persistent point ( t b j , t d j ) ).
Table 14 shows variations of RMSE with different diagonal and offdiagonal elements in Σ .
As demonstrated in Table 14, (i) RMSE increases with increasing the off-diagonal element (i.e., the correlation coefficient) when fixing diagonal element; (ii) RMSE first decreases and then increases via increasing diagonal element when setting the correlation coefficient to be 0 (i.e., there is no linear relationship between the t b j and t d j ).
In addition, we find evidence of a non-monotonic relationship between prediction performance (in RMSE) and diagonal element of Σ .
D Broader Impact
The key positive impact of this project constitutes in the simultaneous enhancement of both predictive performance and computational efficiency in forecasting tasks associated with complex multivariate time series, including but not limited to spatio-temporal processes.
Such tasks are widely met in a diverse range of applications, from finance to atmospheric sciences to biosurveillance.
Furthermore, topological methods and graph representations of multivariate time series and spatio-temporal processes, in conjunction with forecasting, remains yet nascent in both machine learning and statistics, and has a high potential, especially for analysis of multivariate time series with a complex dependence structure, such as disease surveillance, wildfire monitoring, and cryptomarket analytics.
However, the current lack of proper statistical inferential methods for topological characteristics of multivariate
[Table] [Table]
[Table] [Table]
time series makes it questionable how robust these methods and the conclusions derived from them are, especially under adverse scenarios.
This negative impact of the developed predictive analytics shall be necessarily accounted in the proper risk management, particularly, when targeted attacks are expected
[Table]
and shall be further investigated with rigours inferential tools of mathematical statistics.