Multivariate Time-Series Forecasting with Temporal Polynomial Graph Neural Networks
Abstract
Modeling multivariate time series (MTS) is critical in modern intelligent systems.
The accurate forecast of MTS data is still challenging due to the complicated latent variable correlation.
Recent works apply the Graph Neural Networks (GNNs) to the task, with the basic idea of representing the correlation as a static graph.
However, predicting with a static graph causes signiﬁcant bias because the correlation is time-varying in the real-world MTS data.
Besides, there is no gap analysis between the actual correlation and the learned one in their works to validate the effectiveness.
This paper proposes a temporal polynomial graph neural network (TPGNN) for accurate MTS forecasting, which represents the dynamic variable correlation as a temporal matrix polynomial in two steps.
First, we capture the overall correlation with a static matrix basis.
Then, we use a set of time-varying coefﬁcients and the matrix basis to construct a matrix polynomial for each time step.
The constructed result empirically captures the precise dynamic correlation of six synthetic MTS datasets generated by a non-repeating random walk model.
Moreover, the theoretical analysis shows that TPGNN can achieve perfect approximation under a commutative condition.
We conduct extensive experiments on two trafﬁc datasets with prior structure and four benchmark datasets.
The results indicate that TPGNN achieves the state-of-the-art on both short-term and long-term MTS forecastings.
Introduction
The wide deployment of sensors in modern societies records tremendous time-series data, which boosts the application of energy dispatch, trafﬁc control, etc.
Commonly, there are numerous distributed sensors in a monitoring system, e.g., the temperature control system of a computing center.
Univariate time-series data recorded by these sensors formulate the multivariate time-series (MTS) data.
Forecasting over MTS data has been widely studied [4,21,41] in recent years, as it provides essential information for strategy formulation and resource schedule [30,52].
One basic premise of modeling the MTS data is that the variables interact with each other, which is fulﬁlled in most cases.
Therefore, capturing the variable correlation is essential for MTS forecasting.
Previous deep-learning-based works like FC-LSTM [36] and TPA-LSTM [34] model the correlation with an implicit recurrent process, which is inefﬁcient in prediction and hard to optimize.
The development of graph neural networks (GNNs) [17] brings an innovative way to capture the variable dependence.
Recent works [12,25,44] regard the MTS data as a series of graph signals and process them with GNNs, where the nodes are variables and the edge weight quantiﬁes the dependence.
Although GNNs-based methods achieve promising efﬁciency and accuracy in MTS forecasting, [Figure] there are two notable challenges to the current approaches.
The MTS data have no prior variable dependence, which is required for determining the edge weight in graph construction.
The existent methods compute the weight according to the physical distance among sensors [23,35,44] or utilize parameter matrices to learn the weight from MTS data [27,41,42].
As a result, the edge weights are constant in their methods.
However, according to the observations from the real-world dataset, the variable dependence changes with time.
Therefore, using static-weight graphs in prediction causes signiﬁcant bias.
Figure 1 illustrates the ﬁrst challenge with a trafﬁc dataset (PEMS-D7).
It demonstrates that the correlation has distinct patterns in different periods.
Besides, three staticgraph-based methods [41,42,44] show error increases between two consecutive periods, which is caused by dependence variation.
Many researchers point out a gap between the actual variable dependence and the constructed correlation [25, 42, 49], i.e., the edge weight does not accurately describe the variable correlation.
Although they propose various methods to bridge the gap, there is no theoretical/empirical analysis of the gap.
Therefore, the method’s effectiveness is still unknown.
To address the above two challenges, we propose Temporal Polynomial Graph Neural Network (TPGNN), a novel GNNs-based forecasting model for MTS data.
TPGNN has an encoder-decoder structure, we encode historical MTS data with a novel GNN module and make predictions by decoding the encoded results auto-regressively.
The core of TPGNN is a temporal polynomial graph (TPG) module that learns the dynamic variable dependence end-to-end.
The main idea is to represent the correlation as a time-varying matrix polynomial.
Firstly, we deﬁne an adjacency matrix basis A to capture the overall dependence for each variable pair.
Then, we use a matrix polynomial of A to represent the correlation of each time step, where the coefﬁcients are determined by time.
To generate coefﬁcients for MTS data of arbitrary length, we propose to use a set of cyclic timestamp embedding to index the time step.
As a result, TPGNN captures the dynamic correlation with a temporal matrix polynomial, which solves the ﬁrst challenge effectively.
For the second challenge, we analyze the approximation error of the TPG module and provide a theoretical bound on the error.
Furthermore, we evaluate the existent dependence learning methods on six synthetic MTS datasets with ground-truth variable dependence.
The main contributions of the work are as follows:
• We propose a novel dependence learning module to capture MTS data’s dynamic correlation, which represents the variable dependence as a temporal matrix polynomial.
The resultant model achieves state-of-the-art forecasting performance on 5 of 6 benchmark datasets with signiﬁcant improvements.
• We investigate the dependence approximation gap on six synthetic datasets with different levels of dependence complexity.
On average, the proposed TPGNN outperforms the best baseline by 23 .
41% on the approximation error.
A theoretical result further demonstrates the dependence learning ability of TPGNN.
Related Works
MTS Forecasting There are two main categories of methods for MTS forecasting: implicitdependence approaches and structural-dependence approaches.
One representative method of the ﬁrst category is LSTNet [20], it captures the variable dependence with convolution over variables.
The structural-dependence approaches represent the variable correlation as explicit graphs and predict with GNNs.
Graph Wavenet [42] (GWN) attempts to learn the variable correlation with static node representations and captures the temporal pattern with convolution neural networks (CNNs).
MTGNN [41] follows the idea of GWN to learn variable correlation statically and proposes a new graph convolution module for MTS forecasting.
Although works like GMAN [50], SLC [49], and
StemGNN [4] model a dynamic correlation among variables with self-attention, the learned graph is sensitive to the input, causing a signiﬁcant forecasting variance.
Our construction is based on a set of time-variant coefﬁcients, the result is independent of the input and robust in forecasting.
Some works that focus on learning time-varying graph structure [26,33,45].
However, these graph structures are not captured from the forecast task directly.
They thus cause bias and increase the computation cost.
There are some transformer-based works [43,46,48], but their methods do not thoroughly discuss the dynamic-dependence issues.
GNNs and Polynomial Graph Filters (PGFs) GNNs are based on the message-passing mechanism.
One representative work is GraphSAGE [14], which learns a sample-aggregation function to generate the node embedding.
Michael et al.
[6] generalize the convolutional neural networks (CNNs) to the graph structure.
Many ﬁlters of GNNs belong to PGFs, and Gama et al.
[11] discussed the connections.
In order to incorporate the node feature of high-order neighbors, many works introduce PGFs to their methods.
APPNP [18] and SGC [40] adopt ﬁxed-weight PGFs and use hyperparameters to deﬁne the coefﬁcients.
However, the constructed ﬁxed-weight PGFs are biased because the coefﬁcients are task-irrelevant.
GPR-GNN [5] constructs a PGF from the graph dataset, where the coefﬁcients are learnable.
Besides, there are also some PGFs-based methods for MTS data learning [13,15,16].
Although these works are competitive in capturing graph structure, they fail in learning the dynamic correlation, we illustrate the point in Section 5.4.
Moreover, the missing/noisy/incomplete adjacency matrix causes difﬁculties in applying these methods.
Our approach introduces a self-adaptive graph to the PGFs construction, which improves the dependence capturing ability and the model versatility.
The Framework of TPGNN
In this section, we ﬁrst deﬁne the MTS forecasting task in the context of graph signal processing.
We then introduce the TPG module and illustrate the inference pipeline of our framework.
3.1 Problem Deﬁntion
At time step t , the MTS data is a signal set G ( t ) =
{V , E ( t ) , X ( t ) , W ( t ) } .
The node set V ( |V| = N ) contains N variables of MTS data, E ( t ) is the edge set, X ( t ) ∈ R N × 1 records the signal of N variables, and W ( t ) ∈ R N × N is a weighted adjacency matrix.
If variables i and j are dependent, then an edge connects the two nodes, and entry W ( t ) ij indicates their correlation.
Otherwise, W ( t ) ij equals to zero.
Given T observations of the signal sets, our problem is to ﬁnd a function F that can forecast the variable state of the subsequent T (cid:48) signals:
[Equation]
The setting regards the graph structure as a latent feature of MTS data, and the forecast target is the same as the standard setting.
Prevailing works [25,41,42] regard W ( t ) as a time-invariant matrix, while in our work, W ( t ) changes with the time.
3.2 Represent the Correlation as a Temporal Matrix Polynomial
To capture E and W t for MTS data, we construct a series of matrix polynomials that share a matrix basis.
The TPG module controls the inﬂuence of each polynomial term temporally with a set of adaptive coefﬁcients.
As a result, we can approximate a wide range of dynamic graph structures.
Counstructing the adjacency matrix basis.
We ﬁrstly deﬁne an initial adjacency matrix A ∈ R N × N following the self-adaptive graph proposed by Wu et al.
They learn variable embeddings
E ∈ R N × c ( c is the embedding dimension) from MTS data and deﬁne the variable dependence with embedding similarity.
Besides, they further remove weak connections with ReLU( · ) and normalize the result with SoftMax( · ) .
[Equation]
For MTS data that have a prior structure, e.g., the physical distance among sensors, we denote the corresponding adjacency matrix as W ∈ R N × N .
To incorporate W into the encoding process, we ﬁrst calculate its symmetric normalized Laplacian [17] L = D − 12 ( I + W ) D − [-
-]12 ( I is the identity matrix and D is the degree matrix), then combine the two results to deﬁne the initial adjacency matrix:
[Equation]
Generating the temporal polynomial coefﬁcients.
At time step t , we represent the variable dependence as a K -order matrix polynomial of A :
[Equation]
Due to the polynomial coefﬁcients being time-variant, W ( t ) shows dynamic behavior.
However, learning coefﬁcients for every time step is impractical because the sequence length grows inﬁnitely.
Fortunately, the periodicity property is common in MTS data [8, 29].
Therefore, we propose to use a set of cyclic timestamp embeddings to index the time step.
Firstly, we deﬁne T p timestamp embeddings ( e (1) ts , . . .
, e ( T p ) ts ) ∈
R D e , where T p is the cycle and D e is the embedding dimension.
Following a cyclic order, we then assign the ( t % T p ) -th embedding to time step t , where % is the remainder operator.
Finally, we generate the coefﬁcients for time step t by the e ( t % T p ) ts .
As shown in Figure 2 (1), Given T historical graph signals start from time step t , we calculate the coefﬁcients a ∈ R K +1 for each graph signal with a coefﬁcient matrix W c ∈ R D e × ( K +1) .
[Equation]
According to Equation 4, each a ( t ) deﬁnes a K -order matrix polynomial of A , representing variable dependence of time step t .
Instead of predicting with T different matrix polynomials, we use an average result to improve the prediction efﬁciency and robustness.
Due to the linearity of the polynomial, the average polynomial is equivalent to the polynomial with average coefﬁcients.
We calculate the ﬁnal coefﬁcients ¯ a with another parameter matrix W a ∈ R T × 1 (average matrix).
[Equation] [Equation]
In Figure 2 (2), we illusrate the TPG module’s propagation process with ¯ a .
Let X ∈ R N × D e denote the matrix signal embedding,
W k ∈ R D
e × D e , k = 0 , . . .
, K denote the model parameter matrices, || · || F denote the Frobenius norm of matrix, we derive the hidden feature Z ( t ) ∈ R N × D e as follows:
[Equation]
TPG module follows the diffusion graph convolution layer [23] to introduce ( K + 1) parameter matrices for enhancing representation diversity.
Besides, we decouple the parameter matrix norm and coefﬁcient by normalizing W k with a factor of 1 || W k || F , since W k with a large norm increases the k -th term’s contribution.
As a result, ¯ a completely controls the contribution of each term.
3.3 Inference Pipeline
In Figure 2 (3), TPGNN has an encoder-decoder structure.
First, we derive the historical data embedding with a linear transformation, then feed the results into the encoder.
The encoder generates data encoding Z ( t : t + T − 1) ∈ R T × N × D e for time step t to t + T − 1 with the TPG module and temporal attention layer.
The temporal attention layer follows the design of Transformer [37], which uses self-attention to capture intra-series patterns for each variable.
Finally, the decoder forecasts with the data encoding and temporal attention layer.
For the ﬁrst prediction of time step t + T , the decoder queries the data encoding with a learnable beginning of sentence (BOS) token E BOS ∈ R N × D e .
The query result represents the ﬁrst prediction, denoted as E ( t + T ) X ∈ R N × D e .
We use a prediction matrix
W pred ∈ R D e × 1 to get the forecasting result ˜ X ( t + T ) ∈
R N × 1 .
[Equation]
After generating the subsequent k query results ( E ( t + T ) X , . .
, E ( t + T + k − 1) X ) , the decoder derives the ( k + 1) -th result E ( t + T + k ) X by querying
Z ( t : t + T − 1) with the latest L max query results.
The forecast basically follows an auto-regressive (AR) mechanism, where L max is the maximum query length.
[Equation]
The above process continues until getting the subsequent T (cid:48) forecasts, such AR mechanism helps
TPGNN to capture long-term intra-series dependence with informative context.
Theoretical Properties of TPGNN
To explore the dependence learning capability of TPGNN, we focus on analyzing the theoretical gap between the optimal graph structure and the TPG module’s approximation in this section.
TPGNN is able to achieve perfect approximation.
Unlike the real function, the approximation of graph structure largely depends on the topology property of the target graph, where the lower bound of the approximation error is nonzero in many cases.
However, TPGNN mitigates the gap for a wide range of dynamic graph structures with a small group of parameters, which leads to a robust and accurate forecasting result.
The approximation capability of TPGNN is concluded as the following result, we show the detailed statement and proofs in Appendix A.9 due to the space limitation.
Theorem 1 .
Let G (1) , . .
, G ( T )
∈ R N × N be the symmetric normalized Laplacian of the optimal graph structure for time step 1 to T , A ∈ R N × N is the initial adjacency matrix of TPGNN.
We model these Laplacians with Equation 4, and the corresponding approximation error e (1: T ) =
T Σ T t =1 || W ( t ) − G ( t )
|| 2 F , where the || · || F is the Frobenius norm of the matrix.
Suppose all the matrices are symmetric (undirected graph), G i G j = G j G i , ∀ i, j , A has N different singular values, and the polynomial’s order is large enough.
Then we have the following estimation for e (1: T ) .
[Equation] [Equation]
where E t || G ( t ) ||
2 F = 1 T Σ Tt =1 || G ( t )
||
2 F is the average norm of the Laplacians, λ min , λ max ∈ [0 , 1] are the minimum and maximum eigenvalues of a constant matrix.
If A is commutative with G ( t ) , then λ max = λ min = 1 , i.e., we achieve a uniform perfect approximation.
Although the zero error depends on the commutative condition, A is learnable with Equation 2, which relaxes the restriction.
Moreover, TPGNN empirically achieves the best approximation compared with other methods.
Experiments
[Table] 5.1 Experimental Setup
We validate the performance of TPGNN on two tasks: single-step and multi-step MTS forecasting.
In
Table 1, we present the statistics of six benchmark datasets.
The ﬁrst four benchmark datasets [2,19] aim to predict a single future step; the last two datasets [23,44] record a distance matrix among sensors, where the goal is to forecast multiple future steps.
More details about the datasets are given in the
AppendixA.3.
For single-step prediction, the baselines are VAR-MLP [47], GP [31], RNN-GRU [41],
LSTNet [20], TPA-LSTM [34], and MTGNN [41].
MTGNN ﬁrstly introduces GNNs to general
MTS data and achieves state-of-the-art (SOTA) performance; other baselines are representative real-world datasets, and models with high CORR and low RSE are preferred. TPGNN outperforms other models on three of the ﬁrst four datasets. For the remaining two datasets, we predict the next 15, 30, and 60 minutes based on one-hour observation [Table]
and the three metrics represent the prediction error.
The results show that
TPGNN achieves state-of-the-art performance on both short-term and long-term predictions. [Table] non-GNNs-based methods. For multi-step prediction, the baselines are ARIMA [23],FC-LSTM [36],
DCRNN [22], STGCN [44], StemGNN [4], Graph WaveNet, Informer [51], and MTGNN
; all the baselines are GNN-based except ARIMA, FC-LSTM and Informer.
These GNNs-based baselines are representative methods for MTS data with a prior structure.
Besides, the Informer is a novel
Transformer-based model that achieves SOTA performance on long-term forecasting.
We list the paramter scale of each model in Appendix A.4 to show our method is light-weight.
We use ﬁve evaluation metrics [41]: Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE),
Root Mean Squared Error (RMSE), Root Relative Squared Error (RRSE), and Empirical Correlation
Coefﬁcient (CORR).
For CORR, higher values are better.
For the other metrics, lower values are better.
We tune the hyperparameters on the validation dataset, the results are presented in the Appendix A.7.
We divide the dataset into three parts for training, validation, and testing with a ratio of 7:1:2.
More details about the baselines, metrics, and experiment settings are in Appendix A.4A.5A.6.
5.2 Main Results
Table 2 summarizes the experimental results of TPGNN.
Generally, TPGNN achieves the stateof-the-art on most of the datasets.
Our framework has an on-par performance for the single-step forecasting task with the SOTA methods like MTGNN and TPA-LSTM.
Moreover, TPGNN makes a signiﬁcant improvement on Trafﬁc and Electricity datasets.
On the Solar-Energy, TPGNN lowers
RSE by 1 . 61% , 18 .
08% and increases CORR by 1 .
93% over the horizons of 12, 24.
Our method improves the CORR by 2 .
21% on average over the four horizons of the Trafﬁc.
Moreover, we lower the RSE by 15 .
82% on average over the four horizons of Electricity.
Our method fails to achieve SOTA performance on the exchange-rate data.
We think the main reason is the small sample size, which causes difﬁculties in capturing the dynamic variable dependence.
The second row of
Table 2 concludes the results of the multi-step forecasting task, which demonstrates that TPGNN achieves SOTA performance on the two datasets with prior structure.
On the PEMS-BAY, TPGNN lowers the MAPE by 5 .
47% on average over the three horizons.
Besides, The RMSE is reduced by 4 . 30% , 2 .
41% over the horizons of 3 and 6.
TPGNN reduces the MAE/MAPE by 8 . 04% / 6 .
61% on average over the three horizons on the PEMS-D7 dataset.
The RMSE is lowered by 2 .
23% over the horizon of 6 and 12.
Although the prior structure is provided to each baseline, the results on the last two trafﬁc datasets demonstrate that TPGNN can capture more precise variable dependence and intra-series patterns than other GNNs-based methods.
5.3 Ablation Experiments
[Table]
The PEMS-D7 includes a prior structure and has complex inter-series and intra-series dependence, we conduct ablation experiments on the PEMS-D7 to evaluate the effectiveness of each design in our method.
Due to the importance and complexity of the long-term prediction, we set the horizon as 12.
More results of other horizons can be found in Appendix A.8.
There are ﬁve experiments in total, we repeat each experiment 5 times and report the average and standard deviation (std).
w/o
TPG : TPGNN without the TPG module.
We replace the graph constructed by TPG with the prior structure of PEMS-D7.
w/o dynamic : TPGNN without the time-varying coefﬁcients.
We adopt a set of learnable static coefﬁcients to construct the graph.
w/o overview : TPGNN does not use average coefﬁcients in Equation 6.
We predict with T graphs deﬁned by Equation 5.
w/o normalize :
TPGNN without the normalization factor in Equation 7.
w/o K-matrices : TPGNN without multiple parameter matrices in Equation 7.
We use an identical normalized matrix to derive the hidden feature.
Table 3 summarizes the experimental results, which indicates that all the designs are indispensable.
The result of w/o TPG illustrates the TPG module’s critical contribution to our method.
Although we replace the graph constructed by the TPG module with the prior structure given by PEMS-D7, the performance has a steep decline by 13 .
23% on the three metrics.
w/o dynamic follows the strategy of GPR-GNN [5] that uses static coefﬁcients to capture graph structure from data.
However, the results indicate the importance of introducing time-variant coefﬁcients, which [Figure] signiﬁcantly improve the long-term prediction accuracy by 7 . 04% on average.
w/o overview and w/o K-matrices show large variance compared with other results [Table] indicating the two modules are essential for improving model robustness.
Besides, the w/o overview’s performance is close to the w/o dynamic, demonstrating the necessity of using the average coefﬁcients.
w/o normalize has comparable performance to the TPGNN, possibly due to the weight decay causing similar norms.
Nevertheless, it demonstrates that the normalization helps TPGNN capture the variable dependence.
[Table]
5.4 Study of the Graph Structure Approximation Gap
Synthetic data.
To investigate the empirical gap between the optimal graph structure and the approximation result, we propose to generate MTS data with ground-truth dynamic dependence using a non-repeating random walk (NRW) model [7], which is widely adopted in time-series data generation [1, 32].
At time step t , we synthesize the matrix signal X ( t ) ∈
R N × 1 with a dynamic weighted adjacency matrix W ( t ) by X ( t ) =
N ( W ( t − 1) X ( t − 1) , σ ) , where N ( · , · ) is the normal distribution, σ ∈ R controls the variance, and X (0) is sampled from a discrete uniform distribution.
We deﬁne W ( t ) by traversing N w constant matrices ( G (1) , . . .
, G ( N w ) ) in a cyclic order, where each
G ( i ) is a Laplacian of sparsiﬁed random adjacency matrix.
Given a period of T p , T p ≥ N w , we partition the period into N w even intervals of length T s , and each interval shares one of the matrices, i.e., W ( t ) =
G ([( t % T p ) /T
s ]) , where [ · ] is the rounding function.
We point out that N w controls the complexity of dynamic dependence, and N w = 1 corresponds to a static-dependence case.
To increase the data diversity, we randomly initialize the X ( t ) for every T p step.
We set T p = 120 , σ = 0 .in
our evaluation and generate six MTS data of length 2400 with N w = 1 , . .
Besides, each [Figure] synthetic dataset is divided into three parts for training, validation, and testing with a ratio of 7:1:2.
More details about the synthetic algorithm, conﬁguration, and synthetic results are in Appendix A.3.
Predicting with different graph approximation methods.
We evaluate the dependence capturing ability of different methods on the six synthetic datasets, where the task is the next-step prediction based on current variable states.
Suppose that ˜ W ( t ) ∈ R N × N , t = 0 , . .
, T − 1 is the learned adjacency matrix series, we measure the approximation gap with a Mean Frobenius Error (MFE) metric, deﬁned by MFE = 1 TN 2 Σ T − 1 t =
|| W t − ˜ W ( t ) || F .
Besides, we use MAE/RMSE to measure the forecasting accuracy.
The baselines are Graph Wavenet (GWN), MTGNN, GPR-GNN, and
Self-attention.
The ﬁrst three methods are static-dependence-based.
Self-attention constructs a dynamic graph, which is adopted in many works like GMAN [50] and StemGNN [4].
We follow the implementation of StemGNN to learn the graph structure with a GRU module and self-attention operator in the evaluation.
The polynomial order K is ﬁxed as 2 in the evaluation, we repeat each experiment 5 times and report the average/std, more details about the settings are in Appenedix A.6.
The main results are summarized in Figure 3 (a) and Table 4, demonstrating that TPGNN outperforms other baselines on the six synthetic datasets.
In Table 4, TPGNN reduces the graph approximation error by 23 .
41% on average over the six datasets.
The ﬁrst chart of Figure 3 (a) also illustrates the improvement, we consistently achieve the best approximation under different N w .
Moreover, the
MFE of TPGNN has a descending trend with the increase of N w while other methods’ approximation errors are increasing.
We think the TPG module overﬁts on datasets with small N w , since their behavior tends to have static dependence.
TPGNN achieves the lowest prediction error on average, the improvement in the accuracy is slight compared with MFE because the task is a simple next-step prediction.
Furthermore, we observe that the ranking on MFE is identical to the MAE/RMSE, which demonstrates the importance of capturing precise variable dependence for MTS forecasting.
The effect of polynomial order.
We further investigate the relation between approximation error and the polynomial order K , where the K ranges from 1 to 4 , and K = 1 corresponds to a linear approximation.
The ﬁrst two charts of Figure 3 (b) present the average MFE and corresponding std, respectively.
The results show that the approximation variance increases signiﬁcantly with K , especially when K = 4 .
Moreover, TPGNN-K4 has a large MFE on the six datasets compared with other conﬁgurations.
Therefore, we think a high-order matrix polynomial impairs both the precision and robustness of the dependence capturing ability of TPGNN.
We point out that the observation is non-trivial since GPR-GNN achieves good results with a large K ( K ≥ 10 ) on normal graph datasets, possibly due to the difference between MTS forecasting and node classiﬁcation.
The third chart shows the prediction error, it also validates that model with low MFE has a low prediction error.
Case study on the real-world dataset.
Due to the absence of ground-truth dependence, we illustrate the correlation modeling ability of TPGNN on real-world datasets with an example of PEMS-D7.
We ﬁnd three highly correlated nodes in the learned result but independent in the prior structure, where the node IDs are 60, 93, and 218.
In Figure 4 (a), we mark the physical location of the three nodes according to PEMS [3].
The three nodes are on the same road, which illustrates that TPGNN can learn reliable dependence from data.
In Figure 4 (b), we show the learned correlation of pairs (60 , 218) and (93 , 218) .
We observe that (93 , 218) has a stable dependence and (60 , 218) ’s correlation increases in the evening, the difference possibly caused by the branch road between 218 and 60.
Conclusion
We proposes a novel GNNs-based method for MTS forecasting, TPGNN, which represents the variable dependence as a temporal polynomial matrix.
The results on six real-world datasets illustrate the outstanding forecasting performance of TPGNN.
Besides, extensive experiments on synthetic datasets demonstrate that our method achieves the best dependence approximation than other baselines.
Limitations and future works Although TPGNN achieves outstanding performance on MTS forecasting and dependence capturing, our methods have limitations.
First, the dependence captured by TPGNN is not a strict causal structure.
Therefore, some correlations are unreliable and decrease the model’s robustness.
Second, MTS forecasting in the real world is affected by rare events like weather disasters, which may signiﬁcantly impair prediction accuracy.
The future works focus on causal learning and transfer learning [9,24,28,38] on MTS data to solve the above limitations.  Acknowledgment
The work of the authors was supported by the National Natural Science Foundation of China (No.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] see Section(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] see Appendix A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] see Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] see Appendix A. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] see Section 5.3 5.4 and Appendix A. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] see Appendix A. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets? [Yes] see Appendix A. (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] see Appendix A. (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content? [Yes] see Appendix A. 5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] not applicable (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] not applicable (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] not applicable
A Appendix A.1 Limitation
Potential negative societal impacts. Accurate MTS forecasting beneﬁts the application of intelligent systems. Since such systems contain many distributed sensors and produce a large amount of timeseries data, predicting the systematic trend is crucial for controlling the system. However, suppose the algorithm is used for ﬁnancial crimes or other illegal activities.
In that case, it will have a bad inﬂuence on society.
Fortunately, we can address such problems with privacy-preserving methods to improve data safety.
Then, we can restrict the negative impacts since the data access is limited.
Besides, the privacy-preserving approach also lowers the risk of personal data leakage.
A.2 Time Complexity Analysis
[Figure]
Suppose that the MTS data has N variables, the input sequence length is T , the feature dimension is
D e , and the polynomial order is K .
The TPGNN is composed of self-attention and TPG module, and it is known that self-attention has a time complexity as follows:
[Equation]
As the for TPG module, the time complexity for constructing the initial adjacency matrix is O ( N 2 ) , and the time complexity of Equation (7) is O ( KT ( ND 2 e + N 2 D e )) .
As a result, The overall time complexity of TPGNN is O ( KTN 2 D e + KTND 2 e + NT 2 D e ) .
Since the K, D E is small in practice, the efﬁciency of TPGNN is mainly decided by N and T .
In Figure 5, we investigate the efﬁciency of several SOTA methods on the synthetic datasets.
The results show that TPGNN has competitive efﬁciency with MTGNN/Graph Wavenet, though these methods do not contain self-attention modules.
A.3 Data
We summarize the primary information of benchmark datasets in Table 1.
All the six datasets come from the real-world application, we list the details as follows.
All the data follow the MIT license.
A.3.1 Single-step forecasting
• Trafﬁc: the trafﬁc dataset from the California Department of Transportation contains road occupancy rates measured by 862 sensors in San Francisco Bay area freeways duringand 2016.
• Solar-Energy: the solar-energy dataset from the National Renewable Energy Laboratory contains the solar power output collected from 137 PV plants in Alabama State in 2007.
• Electricity: the electricity dataset from the UCI Machine Learning Repository contains electricity consumption for 321 clients from 2012 to 2014.
• Exchange-Rate: the exchange-rate dataset contains the daily exchange rates of eight foreign countries, including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore, ranging from 1990 to 2016.
The input sequence length is 168 and the output sequence length is
Models predict the target future step (horizon) 3, 6, 12, and 24.
A.3.2 Multi-step forecasting
• PEMS-D7: 44-day trafﬁc data collected by 228 sensors in the California state highway system during the weekdays from May through June in 2012.
• PEMS-BAY: 6-month trafﬁc data collected by 325 sensors in the Bay Area of California from January 1st through May 31th in 2017.
For the two trafﬁc datasets, we forecast the subsequent 12 steps by observing a sequence of lengthand evaluate the model performance at 3, 6, 12 steps.
A.3.3 Synthetic data with the NPR model
For clariﬁcation, we illustrate the synthetic algorithm for generating the MTS data in this section, the pseudo-code of our algorithm is showed in Algorithm 1.
We ﬁrstly generate a ranomd orthogonal matrix P ∈ R N × N by decomposing a random matrix with SVD decomposition.
Then, we deﬁne
N w constant matrices with random positive eigenvalues.
Moreover, we sparsify the results to the given sparisty threshold to remove unnecessary connections.
We then generate the required dynamic dependence MTS data with an NRW model, illustrated in lines 12 to 20 of the pseudo-code.
We further show some examples of the synthetic results in Figure 6, it illustrates that the dependence complexity increases with the N w .
In the evaluation, we set σ = 0 .
01 , T = 2400 , T p = 120 , and δ = 0 . 05 .
A.4 Baselines
There are several state-of-the-art baseline methods, we summarize them as follows.
Require: Total length T of the MTS data, number of variables N , number of constant matrices N w .
Require: Length of the cycle T p , stand deviation of the random walk σ , matrix sparsity δ .
{+Ensure: Synthetic MTS data X ∈ R T × N 1: Generate a random orthogonal matrix P ∈ R N × N ; 2: for i = 
+}
→ N w do 3: Σ i = diag ( |N (0 , 1) | , . . . , |N (0 , 1) | ) 4: G i =
P T Σ i P , α =5:
G i [ G i < α ]
=6: while sparsity of G i >
δ do 7: α = α + 0 .8:
G i [ G i < α ] =9: end while 10:
G i =
{+Symmetric normalized Laplacian of G i 11: end for 12: X = 0 ∈ R T × N 13: T l = T/T p 14: for t = 1 → T do 15: if t-1% T p =
+}
then 16: initialize x ∈ R N randomly from {− 1 , − 0 . 5 , 0 .
5 , 1 } 17: else 18: x = N ( G ( t − 1% T p ) /T
l X [ t − 1] , σ ) 19: end if 20: X [ t ] = x 21: end for A.4.1 Single-step forecasting
• VAR-MLP: A model utilizes both of the multilayer perception (MLP) and auto-regressive model (VAR) [47]. • GP: Modeling time-series data with non-parametric method and Gaussian processes [10,31]. • RNN-GRU: A model combines the recurrent unit and fully connected GRU.
• LSTNet: A deep neural network, which combines convolutional neural networks and recurrent neural networks [20].
• TPA-LSTM:
A model utilizes convolution layers to extract temporal patterns and captures time series correlation with attention mechanism [34].
• MTGNN: MTGNN learns a static graph to represent the inter-series correlation and proposes a novel mix-hop propagation layer to capture the inter-series relation [41].
A.4.2 Multi-step forecasting
• ARIMA: A representative univariate time-series forecasting method based on the Kalman ﬁlter [23]. • FC-LSTM: Long Short-Term Memory Network, which is a recurrent neural network with fully connected LSTM hidden units [36].
• LSVR:
A linear support vector regression (LSVR) model for travel-time predictions [39].
• STGCN:
A spatial-temporal graph convolutional network, which incorporates graph convolutions with 1 × 1 -kernel convolution layers [44].
• DCRNN:
A diffusion convolutional recurrent neural network, which utilizes the diffusion graph convolution layer and reccurent unit [22].
We ﬁnd that L = 1 , D e = 64 is an optimal conﬁguration.
[Table]
• StemGNN: A graph neural network, which combines Graph Fourier Transform (GFT) and Discrete Fourier Transform (DFT) together to capture inter-series correlations and temporal dependencies jointly in the spectral domain [4].
• Graph WaveNet:
A spatial-temporal graph convolutional network, which learns a static graph to capture the spatial correlations [42].
• MTGNN:
The same model to single-step forecasting, which is trained to predict multiple steps.
Parameter scale.
We further list the model size of SOTA methods to illustrate our method is lightweight in the model scale.
TPGNN: 0.31M, MTGNN: 0.44M, Graph WaveNet: 0.25M,
STGCN:0.33M, StemGNN:1.22M, TPA-LSTM:0.12M, DCRNN:0.37M.
A.5 Metrics
Let ˜Y ( t ) and Y ( t ) be the predicted and ground truth signal matrix at step t respectively, N is variable number.
The evaluation metrics we use in the experiments are computed by:
[Equation]
A.6 Experimental Settings
For the datasets with a prior distance matrix D , we follow the pre-processing method of STGCN [44].
It constructs the adjacency matrix W as follows, and they select the hyperparameters as σ 2 = 10 , (cid:15)
= 0 .
[Equation]
We conduct all experiments under the environment of one Intel(R)
Xeon(R)
Gold 6254 CPU @
3.10GHz and NVIDIA GeForce RTX 2080Ti GPU.
We repeat the experiment 10 times and report the average of evaluation metrics.
In the training stage, we adopt L1 loss and Adam optimizer.
The initial learning rate is 10 − 3 and it decays by the ratio of 0.3 for every 300 epochs.
The layer number L is 1, the embedding dimension D e is 64, the polynomial order K is 2, and the dimension of variable embedding is 10. The training epoch is 1000, 1500 for single-step and multi-step forecasting, respectively. The batch size is 50.
We use dropout to improve the model robustness to the noisy data.
The rate for dropout is 0 .
[Table] [Table] A.7 Parameter Study
This section investigates the model conﬁguration on the PEMSD7(M) dataset.
The ﬁrst two hyperparameters are the layer number of encoder and decoder L and the embedding dimension of the matrix signal D e , we conclude the results in Table
[Table] which indicates that the conﬁguration L = 1 , D e =is a good setting since it has good performance and efﬁciency. Another two crucial hyperparameters are the K and C . The ﬁrst one decides the polynomial order of the TPGNN, the second one controls the embedding dimension of the variable embedding.
We show the results in Table 6 and Table
[Table]
which implies the K = 2 , C = 10 is the best parameter selection.
A.8 More results for ablation study
This section shows more ablation results on the PEMS-D7 of the different horizons to illustrate that our designs are indispensable.
In Table 8 and Table 9, we ﬁnd TPGNN outperforms other variants consistently, which demonstrate the effectiveness of our components.
Although w/o normalize achieves better performance on MAPE, it performs badly in the long-term forecasting.
[Table]
[Table]
A.9 Theoretical Analysis
Lemma 1 Let F =
{ A 1 , . . .
, A k } , A i ∈ R N × N , be a set of diagbolizable matrices.
Then F is a commuting( A i
A j =
A j A
i , ∀ 1 ≤ i, j ≤ k ), if and only if there exists and ivertible S ∈ R N × N such that S − 1 A i S , A i ∈ F is diagonal.
Proof.
This is a known result, which indicates that the commutation and simultaneously diagonalization are equivalent properties.
We brieﬂy conclude the proving process.
If F can be diagonalized by a S , it is trivial to prove that F is commuting due to the diagonal matrices are commutative.
We inductively prove that F can be diagonalized by a S .
If N = 1 , then it’s a trivial result.
Suppose the theorem holds for matrices of size N ≤ k − 1 , we prove that the theorem is true for N = k .
Let
N = k , A ∈ F , and A is diagnolizable with eigenvalues { λ 1 , . . .
, λ r } where r ≥ 2 and AB = BA for each B ∈ F .
A is similar to a diagonal matrix, so without loss of generality assume A is diagonal.
Since each B commutes with the diagonal matrix A , each B ∈ F is a block diagnoal matrix.
Since A has at least two distinct entries, each block of each B has size k − 1 or less.
By the block-wise commutation property, together with the inductive hypothesis, all the blocks in all
B ∈ F are simultaneously diagonalizable.
Thus there exist ﬁxed matrices T 1 , T 2 , . .
, T r such that conjugating with each B ∈ F with T = diag ( T 1 , T 2 , . .
, T r ) gives a block matrix in which the blocks are diagonal, which is a diagonal matrix.
Lemma 2 Let G , A ∈ R N × N are symmetric matrices.
Since they are real symmetric matrices, G , A can be diagonalized by orthogonal matrices.
Let P ( A ) , P ( G ) are the corresponding orthogonal matrices, we have P ( A ) T AP ( A ) =
diag ( λ 1 , . . . , λ N ) , P ( G ) T GP ( G ) =
diag ( σ 1 , . . .
, σ N ) .
Let
G A = P ( A ) T GP ( A ) , then we have the following results: [Equation] (14) where tr ( · ) is the trace of matrix, ( a 0 , . . . , a K ) is the polynomial coefﬁcients, p K ( D ) =
{+Σ Kk =
 a k diag( λ 1 , . . . , λ N ) k is the matrix polynomial of A ’s eigenvalues.+}
Proof.
{+Let E = G − Σ Kk =
+}
a k A k , we have the following equations:
[Equation]
Moreover, || G − Σ Kk =0 a k A k || 2 F = tr ( E 2 ) ( E is symmetric).
Therefore, we have: [Equation] Review.
Lemma 2 clearly illustrates the approximation ability of a matrix polynomial. The error is controlled by two terms. The value of the ﬁrst term depends on whether P ( A ) is able to diagonalize G .
As for the second term, it corresponds to a least square regression problem.
Theorem 2 Given the setting same as Lemma 2.
Let Q = P ( A ) T G , we deﬁne a discrepancy matrix D AG , ( D AG ) ij = Q 2 ij . Let d k is the k -th row vector of D AG , and λ min , λ max be the minimum and maximum eigenvalues of matrix Σ Nk =1 d k d Tk .
We then have the following estimation of the frist term of Equation 16:
[Equation]
Proof.
We ﬁrst expand the ﬁrst term of Equation
[Equation] Besides [Equation] we observe that:
[Equation]
Therefore,
[Equation]
Combining the result with Equation 18, we derive the following result:
[Equation]
Since G A = P ( A ) T GP ( A ) .
Let u 1 , . .
, u N are the column vectors of P ( A ) , we have ( G A ) ij =
u Ti Gu j .
Therefore, we derive the following equation:
[Equation]
Let Q = P ( A ) T P ( G ) , v 1 , . .
. , v N are the column vectors of P ( G ) .
We then have P ( A ) =
P ( G ) Q T , and u k = Σ ms =1 Q sk v s .
Therefore, we derive the following result:
[Equation]
Because v Ts Gv s = σ s , v T s Gv t = 0 , if s (cid:54) = t , we have tr ( diag ( G A ) 2 ) = Σ mk =1 (Σ ms =1 σ s Q 2 sk ) 2 .
We then deﬁne a discrepancy D AG , ( D AG ) ij =
( Q ) 2 ij . Let d k be the k -th column vector of D AG , σ = ( σ 1 , σ 2 , . . . , σ N ) T is a vector composed of G ’s eigenvalues. Therefore, we have the following equation:
[Equation] Due to Σ Nk =1 d k d Tk is a symmetric matrix, according the property of Rayleigh quotient, we know: [Equation] Where λ min , λ max are the minimum and maximum eigenvalues of Σ Nk =1 d k d Tk .
Besides, according to the property of matrix trace, tr ( G 2 ) = tr ( P ( G ) diag ( σ 1 , . .
, σ N ) 2 P ( G ) T )
= σ T σ . Therefore, we have the following result: [Equation]
According to Equation 21 and Equation 26, we ﬁnish the proof of Theorem 2.
{+Lemma 
+}
Given the setting same as Lemma 2, we have the following estimation of the second term of Equation 16:
[Equation]
Speciﬁcally, if D has N different values and K = N − 1 , then the upper bound is zero.
Proof.
Since tr (( diag ( G A ) − p K ( D )) 2 )
=
Σ Ni =1 ( G A ii − Σ Kk =0 a k λ ki ) 2 , it corresponds to a least square problem.
Equation 3 is a known result of least square optimization under K = 1 .
Due to
K ≥ 1 , Equation 3 holds obviously. Besides, if D has N different values, i.e., A has N different eigenvalues, and K = N − 1 . Then the upper bound of tr (( diag ( G A ) − p K ( D )) 2 ) is zero with
Lagrange interpolation. Theorem 3 (Formal statement of Theorem 1). Let G = { G (1) , . . .
, G ( T ) } , G ( t )
∈ R N × N be the symmetric normalized Laplacian of the optimal structures for time step 1 to T , A ∈ R N × N is the initial adjacency of TPGNN.
Suppose G is commuting, A and elements of G are symmetric.
Then, there is a P ( G ) ∈ R N × N such that P ( G ) T G ( t ) P ( G ) is diagonal matrix for each t .
Let
P ( A ) T AP ( A ) be a diagonal matrix, we deﬁne D AG the same as Theorem 2, and λ min , λ max are the minimum and maximum eigenvalues of Σ Nk =1 d k d Tk .
If A has N different singular values, the polynomial’s order K = N − 1 . Then the approximation error e (1: T ) = 1 T Σ Tt =1 || W ( t ) − G ( t )
||
2 F satisﬁes the following estimation:
[Equation]
where E t || G ( t ) || 2 F = 1 T Σ Tt =1 || G ( t ) || 2 F is the average norm of the Laplacians.
Proof.
Firstly, G is diagonalizable since its elements are symmetric. According to Lemma 1, we can ﬁnd a S ∈ R N × N such that S − 1 G ( t ) S is diagonal for each t . Therefore, we can ﬁnd an othogonal matrix P ( G ) such that P ( G ) T G ( t ) P ( G ) is diagonal using Gram-Schmidt orthogonalization. We now consider each || W ( t ) − G ( t )
|| 2 F .
According to Theorem 2 and Lemma 3, we have the following estimation:
[Equation]
We thus prove the Theorem 3 with the Equation 29.