M ULTIVARIATE
T IME S ERIES F ORECASTING B Y G RAPH A TTENTION N ETWORKS W ITH T HEORETICAL G UARANTEES
Anonymous authors Paper under double-blind review
A BSTRACT
Multivariate time series forecasting (MTSF) aims to predict future values of multi- ple variables based on past values of multivariate time series, and has been applied in ﬁelds including trafﬁc ﬂow prediction, stock price forecasting, and anomaly detection.
Capturing the inter-dependencies among variables poses one signiﬁcant challenge to MTSF.
Several methods that model the correlations between vari- ables with an aim to improve the test prediction accuracy have been considered in recent works, however, none of them have theoretical guarantees.
In this paper, we developed a new norm-bounded graph attention network (GAT) for MTSF by upper-bounding the Frobenius norm of weights in each layer of the GAT model to achieve optimal performance.
Under optimal parameters, we theoretically show that our model can achieve a generalization error bound which is expressed as products of Frobenius norm of weight in each layer and the numbers of neighbors and attention heads, while the latter is represented as polynomial terms with the degree as the number of layers.
Empirically, we investigate the impact of different components of GAT models on the performance of MTSF.
Our experiment also veriﬁes our theoretical ﬁndings.
Empirically, we also observe that the generalization performance of our method is dependent on the number of attention heads, the number of neighbors, the scales (norms) of the weight matrices, the scale of the input features, and the number of layers.
Our method provides novel perspectives for improving the generation performance for MTSF, and our theoretical guarantees give substantial implications for designing attention-based methods for MTSF.
I NTRODUCTION AND B ACKGROUNDS
Substantial time series data generated in the real world make multivariate time series forcasting (MTSF) a crucial topic in various scenarios, such as trafﬁc forecasting, sensor signal anomaly detection in the Internet of things, demand and supply prediction in the supply chain management, and stock market price prediction in ﬁnancial investment (Cao et al., 2020).
Traditional methods simply deploy time series models, e.g., auto-regressive (AR) (Mills & Mills, 1990), auto-regressive integrated moving average (ARIMA) (Box et al., 2015) and vector auto-regression (VAR)(Box et al., 2015; Hamilton, 2020; Lütkepohl, 2005) for forecasting.
Speciﬁcally, ARIMA, though one of the classic forecasting methods in univariate situations, fails to accommodate multivariate issues due to high computational complexity.
VAR, as an extension of AR model in multivariate situations, is widely used in MTSF tasks due to its simplicity, however, it cannot handle the nonlinear relationships among variables, leading to reduced forecasting accuracy.
In addition to traditional statistical methods, deep learning methods have been applied in MTSF problems and demonstrated potentials to solve these problems (Tokgöz & Ünal, 2018).
The long short term memory (LSTM) (Graves, 2012), gated recurrent units (GRU) (Cho et al., 2014), gated linear units (GLU)
(Dauphin et al., 2017), temporal convolution networks (TCN) (Bai et al., 2018), state frequency memory (SFM) network (Zhang et al., 2017) have found success in practical time series tasks.
However, another important issue in time series data, complex inter-dependency (i.e., the correlations among multiple correlated time series), is still unaddressed in these methods, restricting forecasting accuracy (Bai et al., 2020; Cao et al., 2020).
For example, in the trafﬁc forecasting task, adjacent roads naturally interplay with each other.
Another example is stock price prediction, in which it is easier to predict a stock price based on the historical information of the stocks in similar categories, while information on stocks from other sectors can be relatively useless.
Graph is a special form of data that describes the relationships between different entities.
Recently, graph neural networks (GNNs)(Scarselli et al., 2008) have achieved great success in handling graph data with development in permutation-invariance, local connectivity, and compositionality.
In general,
GNNs assume that the state of a node is inﬂuenced by the states of its neighbors.
By disseminating information through structures, GNNs allow each node in a graph to be aware of its neighborhood context.
MTSF can be viewed naturally from a graph perspective.
Variables from multivariate time series can be considered as nodes in a graph where they are interlinked each other through hidden dependency relationships.
It follows that modeling multivariate time series data using GNNs can be a promising way to preserve their temporal trajectory while exploiting the inter-dependency among time series.
In the meantime, due to the popularity of convolutional neural networks (CNNs), considerable studies attempt to generalize convolutions to graph-structured data, leading to the creation of graph convolutional networks (GCNs)
(Duvenaud et al., 2015; Atwood & Towsley, 2016; Monti et al., 2018; Niepert et al., 2016; Kipf & Welling, 2017).
GCNs model a node’s feature representation by aggregating the representations of its one-step neighbors.
Many studies have shown that GNN- and GCN-based methods outperform prior methods in time series forecasting tasks (Yu et al., 2017;
Wu et al., 2019; Chen et al., 2020).
The graph attention network (GAT) (Veliˇckovi´c et al., 2017), one of the most popular GNN architec- tures, is considered a state-of-the-art neural architecture to process graph-structured data.
Building on the aggregating approach of GCNs, in GATs, every node computes the importance of its neighboring nodes, and then utilizes the importance as weights to update its representations of the features during the aggregation.
Compared to the well-known GCNs, GATs have demonstrated equivalent, if not improved, performance across well-established benchmarks of node multiclass classiﬁcation.
Within the GAT framework, Guo et al.
; Deng & Hooi (2021) use GAT-based models to adaptively adjust the correlations among multiple time series, showing better performance in accuracy over
GNNs and GCNs.
The numeric and experimental successes of GATs for MTSF notwithstanding, theoretical under- standings of the underlying mechanisms of GATs for MTSF are still limited: none of them has theoretical guarantees with respect to generalization error bounds, the most commonly used method to theoretically evaluate the prediction model.
The generalization error bound provides a standard approach to evaluate neural networks as it characterizes the predictive performance of a class of learning models for unseen data (Golowich et al., 2018).
Therefore, understanding the generalization error bound of GATs for MTSF will shed light on the relationship between the architecture of the GATs and their generalization performance for MTSF, advancing understandings of underlying mechanisms.
Studies show that deriving generalization error bound for neural network classes requires constraints on the size of weights.
Bartlett (1998) ﬁrst gave a generalization error bound for neural networks by bounding the size of the cover of neural network function classes, suggesting that the bound depends on the number of training samples and the size of weights, rather than the number of weights.
In the following studies, the empirical Rademacher complexity (ERC) was shown as an essential component of generalization error bound for neural network classes.
Bartlett & Mendelson (2002) introduced the generalization error bounds using the Rademacher complexity of the function classes that include neural networks with constraints on the magnitudes of weights for binary classiﬁcation.
Bartlett et al.
(2017) then presented a margin-based generalization error bound using the Rademacher complexity of neural network function classes with the spectral norms of weight matrices being controlled for multiclass classiﬁcation.
Neyshabur et al.
(2015) used the Rademacher complexity bounds showing that the generalization error of deep neural networks can be upper bounded by a bound in terms of the
Frobenius norms of weights.
Golowich et al.
(2018) further demonstrated that the generalization error bound for deep neural network classes with bounded Frobenius norm of weights can be independent of the number of layers and the width of each layer if employing proper techniques.
These methods are also extended to graph-based neural networks.
Garg et al.
(2020) derived the generalization error bounds for GNNs using Rademacher complexity for binary classiﬁcation.
Lv (2021) provided the generalization error bound for GCNs via Rademacher complexity for binary classiﬁcation.
Contributions.
In this study, to capture the inter-dependencies among variables of MTSF, we develop a GAT-based method for MTSF; to secure the generalization error bound, we require the norm of weight matrix in our model to be bounded; to evaluate the performance of our method, we compare our method with two SOTA methods and show our method outperforms over these prior methods. We also provide the theoretical generalization error bound for our method, aiming to develop models with a desired generalization error for MTSF.
Speciﬁcally, we derive the generalization error bounds of two-layer GAT models for multi-step MTSF task.
We also extend our generalization error bounds to deep GAT models with more than two layers.
Generalization error bounds derived in this study are based on the bound of ERC of GAT models with the weight matrix norm being controlled.
This approach is characterized by controlling Frobenius norm of the hidden layer weight matrix, a common method to derive the norm-based generalization error bounds for DNNs, CNNs, and GNNs.
In particular, we show that ERC derived for GAT models for MTSF has a polynomial dependence on the number of neighbors considered in attention representation and the number of attention heads being used.
The aforementioned ERC is also dependent on the product of norms of weight matrices of each layer, the L 2 -norm of the input feature vector, and the Lipschitz constant of loss and activation functions.
To further understand the effectiveness of GATs for MTSF, in addition to theoretical analysis of the relationships between components of the GAT model and this bound, we also investigate the inﬂuence of different components of GATs models on the performance of MTSF using experiments with complex stock price data.
Our experimental results are consistent with theoretical ﬁndings.
To our best knowledge, we develop the ﬁrst GAT-based method for MTSF with theoretical guarantees.
P RELIMINARIES 2. P ROBLEM F ORMULATION
In this paper, we focus on the task of MTSF, considering a multivariate situation that con- tains N correlated univariate time series represented as { X 1 , . . .
, X N } , where we use X i = { x i, 1 , x i, 2 , . . .
, x i,t , . . . } to denote a sequence of time series i from time step 1 to inﬁnity.
Based on a sequence of historical T time steps of values prior to current time t , our goal is to predict the multi-step-away value of { y 1 , . . .
, y N } using an appropriate prediction model f , where each y i
= { y i,t +1 , . . .
, y i,t + C } has values from C timestamps.
In addition, the historical inputs can be representative of multiple aspects if complemented with auxiliary features, so our problem can be characterized as { y 1 , . . .
, y N } = f
( { x 1 ,t , . . .
, x 1 ,t − T +1 } , . . .
, { x N,t , . . .
, x N,t − T +1 } ) .
To accu- rately capture the inter-dependency, the problem is formulated on the graph structure as introduced below.
2. T HE G RAPH S TRUCTURE
Now we consider an undirected graph G =
( N , E ) .
N = ( n 1 , . . .
n N ) , |N| = N , is a set of node labels representing the sources of N time series.
E ⊂ N × N is the set of edges representing the connection between series.
We let x i ∈
X , i ∈
[ N ] be a random variable representing the input feature vector of node n i for time series i .
For node i , its random input feature x i ∈
X
⊂ R D is a multi-dimensional vector, which contains all the historical values from T time steps, in other words, we let x i =
( x i,t , . . .
, x i,t − T +1 ) be the concatenation of T time steps; its true label y i ∈ Y ⊂ R C is the vector for the C-step-away values.
During the learning period, some nodes, which will be treated as the training set, know the C-step-away true values y .
We denote the set of indices of those nodes as M ⊂ N such that M = |M| < N .
And for each node in the M , we order them based on their node labels n i and re-index them based on their order number, j ∈ [ M ] .
Here the random input and labels are S
= { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x M , y M ) } over M .
In the following paragraphs, we will introduce the GATs for our problem.
2. GAT S M ODEL
We consider the GATs deﬁned by Veliˇckovi´c et al.
(2017), given the random feature matrix X
=
[ x 1 , x 2 , . . .
, x N ] , a L -layers GAT model f , and the ﬁnal output Z ( L ) ,
Z
[Equation]
[Equation]
In a GAT with more than two layers, the output of the hidden layer l is
[Equation]
Here we use subscript ( l ) or ( l − 1) to indicate which layer the variable belongs to.
We have l ∈ { 1 , 2 , . . .
, L − 1 } , and Z 0 = X .
Here, W ( l,k ) ∈
R D l − 1 × D l is an l − 1 -to- l weight matrix for a hidden layer with D l feature maps.
Here σ is the activation function.
And we let ⊕ denote the concatenation for the attention heads .
This deﬁnition is speciﬁc to the GATs, with its detailed description found in Veliˇckovi´c et al.
And we have total K such matrices in layer l with each matrix W ( l,k ) corresponding to one attention head.
P (1 ,k ) , P (2) and P ( L − 1) are the attention matrix introduced by us, and further function as an operator to incorporate the attention.
We will justify its equivalence to the original GAT models (Veli ˇ ckovi ´ c et al., 2017) in later paragraphs.
Even though our analysis covers GATs with more than two layers, we will give a focus on two-layers GATs model, which is also implemented by Veliˇckovi´c et al.
(2017), with the following simple form:
[Equation]
The ﬁrst layer consists of K attention heads computing D 1 features each (a total of K × D 1 features), followed by an activation function σ .
Here, W (1 ,k ) ∈ R D
× D 1 is an input-to-hidden weight matrix for a hidden layer with D 1 feature maps, and we have K such matrices.
The second layer is used for prediction: a single attention head that predicts the y .
For C-step-away forecasting, we have D L = C .
The W (2) ∈ R KD 1 × C is a hidden-to-output weight matrix.
Attention Model We now give more explanation about the attention introduced in the GAT model.
In section 2 of the original GAT paper, Veliˇckovi´c et al.
(2017) mentioned the learnable linear transformation to transform the input features into higher-level features with sufﬁcient expressive power.
In their process, they ﬁrst apply a shared linear transformation, parameterized by the weight matrix, W ∈ R D
× D 1 , to every node.
Then they perform self-attention on the node: a shared attentional mechanism a : R D 1 × R D 1 → R computes attention coefﬁcients e ij = a ( x (cid:62) i W , x (cid:62) j W ) , to indicate the importance of node j ’s features to node i .
They also inject the graph structure into the mechanism by performing masked attention: they compute e ij for nodes j ∈ N ( i ) , the neighbors of node i , which might include node i itself.
Then they normalize the coefﬁcients to make them easily comparable across different nodes using the softmax function to obtain p i,j
[Equation]
Then the output features from the ﬁrst layer for each node will be: σ ( (cid:80) j ∈N ( i ) p
i,j x (cid:62) j W ) .
They also propose the K -head attention.
The K independent attention mechanisms execute the afore- mentioned transformation, and then their output are concatenated, resulting in the following feature representation:
[Equation]
To make the above process more integrated, here in the GAT models, we introduce the attention matrix P that contains the normalized attention coefﬁcients used to compute a linear combination of the neighborhood features, yielding the new feature representation for every node.
This matrix contains individual node’s importance weights with every other node in its neighborhood.
Let P ( l ) ∈
R N × N , l ∈
[ L − 1] , is the matrix of the graph attention matrix deﬁned by the attention coefﬁcients 1 .
We use N e ≤
N as the ﬁxed num- ber of neighbors for each node.
And let N ( n ) be the set of neighbors for node n .
P
[Equation]
l ) =
[Equation]
[Equation]
p n,e ( l ) ∈ [0 , 1] is the coefﬁcients of node n attributed from node e .
Each row sum of the P ( l ) is equal to 1, which is (cid:80) e ∈N ( n ) p n,e ( l )
= 1 .
Here p n ( l,k ) (row n of P ( l,k ) ) denotes node n ’s all coefﬁcients, which are com- puted by
[Equation]
where φ is a softmax function, used to calculate the importance weight from other nodes.
The
N ( l,k ) ∈
R D l × 1 is a convolutional ﬁlter with ﬁlter size equal to 1 × 1 and output channel size equal to 1 .
The u n is the n ’s entry of Z ( l − 1) W ( l,k ) N ( l,k ) .
And M is a mask matrix with M n,e = 1 for n ∈ [ N ] and e ∈ N ( n ) , and 0 otherwise, and · is the element-wise product, 1 N is the size N vector with all entries equal to 1 .
In the paper by Veliˇckovi´c et al.
(2017), the proposed attention mechanism a consists of the following steps: ﬁrstly apply another convolutional ﬁlter N to the x (cid:62) W , secondly sum up these representations to get the attention coefﬁcients— e i,j = x (cid:62) i WN + x (cid:62) j WN .
We integrate the result of this whole process into an attention matrix P ( l ) , which functions exactly as the attention mechanism in the original architecture introduced by Veliˇckovi´c et al.
Since the sum of row elements of the P equals one, we have the following property hold for P ( l ) :
(cid:13)(cid:13) P ( l ) (cid:13)(cid:13) F ≤ N , i.e., the Forbunis norm of P ( l ) is bounded by the total number of nodes.
2. T HE E MPIRICAL R ISK F RAMEWORK FOR MTSF
We ﬁrst introduce function spaces of GATs for MTSF.
Let the X and Y be the feature and true label spaces, respectively, and Q an unknown distribution over X × Y .
Let F ⊂ V X be the hypothesis class for predictions, where V is another space that might be different from Y .
In our paper, we let the function space F be the space of our GAT classes that contains the GAT functions f .
We defer the detailed deﬁnitions of F in later sections, i.e. in terms of Weights-bounded GATs for the MTSF problem, see the deﬁnition 15.
Given F , X , and Y , we let g : V × Y → [0 , B ] be the loss function deﬁned over F .
We assume that g is bounded, i.e., the range of loss is [0 , B ] .
Additionally, we require B = 1 (if not, we can scale the loss function) without loss of generality.
We also introduce the function class g F ⊂ [0 , B ] X×Y by composing the functions in F with g ( · , · ) , i.e., g F = { ( x , y )
(cid:55)→ g ( f ( x ) , y ) : f ∈ F} .
For any risk function g deﬁned over F , given the training set S
= { ( x 1 , y 1 ) , . . .
, ( x M , y M ) } which includes M i.i.d samples from X × Y according to distribution Q , the expected/population risk E ( f ) and the empirical risk function ˆ E ( f ) are deﬁned as:
[Equation]
A predictor f ∈ F can be generalized if for any δ > 0 , lim sup | S |
[Equation]
A predictor with a generalization guarantee is closely related to the complexity of its hypothesis space.
In that sense, the generalization error bound for F is characterized by the condition where
E ( f ) is bounded by the summation of ˆ E ( f ) , the ERC that is generally the dominating term, and an error function associated with the conﬁdence of the bound and the sample size M .
2. T HE R ADEMACHER C OMPLEXITY
Suppose F = { f : x (cid:55)→ f ( x ) } is a model space.
We deﬁne the ERC R ( F ) and Rademacher complexity R S ( F ) as
[Equation]
where { (cid:15) 1 , · · · , (cid:15) M } are i.i.d.
Rademacher variables satisfying P ( (cid:15) j = 1) =
P ( (cid:15) j = − 1) = 1 / 2 .
G ENERALIZATION BOUND FOR THE GAT MODEL 3. N OTATION
We use bold-faced letters to denote vectors and capital letters to denote matrices or ﬁxed parameters (which should be clear from the context).
Given a vector w ∈ R D , (cid:107) w (cid:107) refers to the Euclidean norm, and for p ≥ 1 , (cid:107) w (cid:107) p = (cid:16) (cid:80)
Fi [-=
-]=1 | w i | p (cid:17) 1 /p refers to the L p norm.
For a matrix W , (cid:107) W (cid:107) F refers to the Frobenius norm, (cid:107) W (cid:107) F = (cid:113) (cid:80) i (cid:80) j | w i,j | 2 .
A function f : R n → R m is L-Lipschitz, L ≥ 0 , if (cid:107) f ( a ) − f ( b ) (cid:107) ≤ L (cid:107) a − b (cid:107) for all a, b ∈ R n .
We use standard big-O notation, with Ω( · ) , Θ( · ) , and O ( · ) hiding constants.
3. F UNCTION C LASS OF GAT S
Given the inputs X =
( x i , . . .
, x N ) as multiple time series with each x i as input feature for node n i , the class of 2 -layer GATs for MTSF
f maps x to the output f ( x ) that represents a C-step-away prediction expressed in equation 2.
And We consider a subset of such class requiring each f with a bounded weight norm, expressed as
[Equation]
here we use f ( x j ) to mean the output of f corresponding to node j , where j ∈ [ M ] , and we know the true label y j of this node.
Furthermore, we also provide a model space F c ⊂ R X with a single dimensional output that corresponds to the c -th component of model output from f ( x ) for the c -th time step, expressed as
[Equation]
3. A N U PPER B OUND OF R ADEMACHER C OMPLEXITY OF GAT C LASS
Here we ﬁrst provide an upper bound of ERC of GAT class F c for single dimensional output of
MTSF.
Theorem 1 (Upper Bound of ERC of GAT class F c for MTSF) .
Let the activation function σ ( · ) be L σ -Lipschitz continuous, and also satisfy σ (0)
= 0 and σ ( αz ) =
ασ ( z ) for all α ≥ 0 .
Assume the L 2 -norm of the feature vector x comes from a bounded domain X
= { x : (cid:107) x (cid:107) ≤ B } .
Assume the Frobenius norm of every weight matrix in the ﬁrst layer of the GAT class is bounded, namely, (cid:13)(cid:13) W (1 ,k ) (cid:13)(cid:13) F ≤ M 1 with some constant M 1 > 0 for every k .
Also, the norm of the weight vector of the second layer of the GATs is also bounded, (cid:13)(cid:13)(cid:13) w c (2) (cid:13)(cid:13)(cid:13) ≤ M 2 , where c ∈ [ C ] , with some constant
M 2 > 0 .
Let N i denote the neighborhood of node i (including i ), let the number of neighbors of each node is equal to each other, namely, for some common constant N e ∈ N + , assume N e :
= |N
i | for all node i ∈ N , furthermore, we consider the most general formulation, which allows every node to attend on every other node, i.e., N e = N .
Then let R ( F c ) be the ERC deﬁned in the deﬁnition 7 for
GAT class F c in the deﬁnition 17, given the M sized input set { x 1 , . . .
, x M } , then we have
[Equation]
We see that this bound has a polynomial dependence on the N , the total number of nodes.
The N appears here due to the fact that we consider all the nodes as neighbors, which can be replaced by the
N e , the number of neighbors.
A small N e < N can result in a potentially smaller bound.
The proof details can be found in §A.
3. G ENERALIZATION E RROR B OUND OF THE GAT C LASS FOR MTSF
In this section, we will give the ﬁnal generalization error bound of the GAT class for MTSF.
The formal result is in the following theorem and the proof is in §
B.
Theorem 2.
Deﬁne the hypothesis class F as the deﬁnition 15.
We suppose g is Lipschitz with constant L g .
Then for any δ ∈ (0 , 1) , with probability at least 1 − δ , for all f ∈ F , we have
[Equation]
where we have R ( F c )
=
O ( L σ BK ( N ) 3 / 2 M − 1 / 2 M 1 M 2 ) from Theorem 1.
E XTENSION TO GAT C LASS WITH L AYERS L >
Now we extend the analysis to GATs with more than two layers for MTSF and provide corresponding generalization error bounds.
Here, the proof is done by a simple induction argument using the "peeling-off" technique employed for Rademacher complexity bounds for neural networks.
The output of a L -layer GATs represents a multi-step-away prediction shown in expression 1.
We deﬁne the function class over M , according to the deﬁnition of the GATs network, with F =
[Equation]
j
[Equation]
Furthermore, we require that the Frobenius norm of every weight matrix in every layer of the GAT class is bounded, namely, for any l ∈ [ L ] , (cid:13) (cid:13) W ( l,k ) (cid:13) (cid:13) F ≤ M l with some constant M l > 0 for every k .
Also, we have single dimensional GAT function space deﬁned as
[Equation]
Also, the output up to layer l , l ∈
[ L − 1] , has the format as, [Equation] Thus
[Equation]
we deﬁne a layer-wised class of functions as
[Equation]
We provide an upper bound of ERC of GAT class F c with L layers with proof details in §C
Theorem 3 (Upper Bound of ERC of GAT class F c with L layers) .
Let all the assumptions from Theorem 1 be fulﬁlled.
Furthermore, let the Frobenius norm of every weight matrix in the ﬁrst L −layers of the GATs be bounded, namely, (cid:13)(cid:13) W ( l,k ) (cid:13)(cid:13) F ≤ M l with some constant M l > 0 for every k .
Also, the norm of the weight vector of the last layer is also bounded, (cid:13)(cid:13)(cid:13) [Equation] ≤ M L , where c ∈ [ C ] , with some constant M L > 0 .
Let R ( F c ) be the ERC deﬁned in Equation 7 for GAT class F c in the deﬁnition 12, given the M sized input set { x 1 , . . .
, x M } , then we have
[Equation]
Based on the theorem 3, we provide the generalization error bounds for the GAT class with more than two layers in the following theorem.
Theorem 4 (Generalization Error Bounds for the GAT Class with More than Two Layers) .
Given any real δ ∈ (0 , 1) , with probability at least 1 − δ , for any f ∈ F deﬁned in the deﬁnition 11, which contains L -layer GATs, and given the M sized training data set S = { ( x 1 , y 1 ) , . . .
, ( x M , y M ) } , we have the generalization error for MTSF is upper bounded as
[Equation]
where R ( F c ) is calculated in theorem 3.
To give the generalization error bound of a deep GAT, we ﬁrstly derive its ERC.
To bound the ERC, we apply the layer-peeling strategy that the ERC of L -layer networks is expressed by a factor multiplied by the ERC of L − 1 -layer networks.
Speciﬁcally, we consider this factor as the matrix Frobenius norm, and our current bounds scale with the product of these norms as the layer size increases.
However, the number of attention heads and the number of attention neighbors appear as the polynomial terms with an order roughly equal to the number of layers L .
The bound has an exponential dependence on the network depth.
The generalization error bound in Theorem 4 implies that the following attempts can be taken to reduce the generalization error: i) increase the training samples, ii) minimize the empirical loss, and iii) design the neural network carefully to achieve a proper hypothesis class.
Increasing the complexity of the hypothesis class can decrease the approximation error but also increase the estimation error due to a large ERC, which leads to undesired test performance in a practical task.
In the next section, we will empirically show how structural components of GATs related to complexity could affect the test performance for a MTSF task, providing empirical support for our theoretical ﬁndings.
E XPERIMENT D ETAILS
Our goal is to show the relationship between the upper bound of the generalization error of the GAT model and variables in the ERC, including the number of attention heads, the number of neighbors, the Frobenius norm of model weights, the number of model layers, the norm of inputs, and the number of labeled nodes.
We use the daily stock price data from Nasdaq and NYSE.
The multivariate time series have about 1500 stocks.
For each stock, the one-week historical data is used to predict future returns.
For every stock, we assume that every feature is i.i.d.
per day.
By default, we use a three-layer (input layer, hidden layer, and output layer) GAT to do single-step forecasting on the returns of each stock.
For each variable, we try its various values with all other variables’s values ﬁxed.
We repeat the experiment 20 times and report the loss on the out-of-sample test dataset to represent its generalization error.
We use minimum square error (MSE) as evaluation metrics 2 .
E XPERIMENT R ESULTS
[Figure]
n e r a li
[Figure]
l
[Figure]
G e n e r a li za ti on l o ss
[Figure]
Number of Attention Heads - K .
For three-layer GATs, Theorem 3 indicates that with the increasing number of attention heads in the attention layer, the upper bound of ERC is y
=
O ( K 2 ) .
The experiment results in Figure 1 shows test error beginning to increase quadratically after some values of K .
This is consistent with our theoretical results on the generalization error bound.
Number of Neighbors - N e .
For three-layer GATs, Theorem 3 indicates that with the increas- ing number of neighbors considered for the attention operation, the upper bound of ERC is y
=
O ( N L − 1 / 2 e ) .
Figure 1 demonstrates that as the N e increases, the test error conforms to this theoretical error bound.
But it is noteworthy that when the number is too small, the loss is also high.
It is possible that at certain range, the inﬂuence of the information loss due to the limited number of neighbors is dominant.
Norm of Weight Matrix - M l .
We also have an empirical evaluation on the relationship between the generalization error bound and the Frobenius norm of GAT models’ weights.
Theorem 3 indicates that with the increasing bound of the Frobenius norm, the upper bound of ERC increases polynomially.
The experiment results in Figure 1 corroborate the conclusion.
When the weight norm increases, the generalization error ﬁrst decreases, then increases.
The reason for the initial decrease is because the bound on the norms is so small that it severely prevents the weights from having enough amount of updates.
Thus, the scales (norms) of the weight matrices should be neither too large (induces large generalization error) nor too small (harms weights’ updates) and choosing proper scales is important in practice as the current work has shown Li et al.
Number of Layers (Model Depth) - L .
Theorem 3 and Theorem 1 indicate that the upper bound of ERC increases exponentially with an increasing number of layers, suggesting that the number of layers has negative impact on the test performance of GATs for MTSF.
However, Sun et al.
(2016) reported that deeper nets which have larger representation power are able to ﬁt training data better and achieve smaller empirical error.
This observation indicates a positive impact of the number of layers’ positive on the test performance for MTSF.
Our empirical results are consistent with the above discussions about the double-edged impacts.
The experiment results are shown in Figure 1 and indicate that as the number of layers increases, the test error ﬁrst decreases, and then increases.
This observation indicates that if the ERC increases quickly, the representation power cannot compensate for the negative impact of the increased number of layers.
Norm of Inputs - B .
The experiment results in Figure 1 show that when the input norm is greater than certain values, the generalization loss of the GAT has a linear relationship with the upper bound of the input norm.
This empirical observation aligns with Theorem 3.
Number of Labeled Nodes - M .
Theorem 3 also indicates that when the number of labeled nodes increases, the generalization loss of the GAT decreases and the upper bound of the generalization loss is y
=
O ( M − 1 / 2 ) .
The empirical results in Figure 1 conﬁrm this relationship.
We include the details about how we control the above six variables in D.2.
In addition, we conduct experiments on two-layer GATs to show that empirical results are also consistent with error bounds in Theorem 1.
Results and discussions are in §E.
It is noteworthy that Figure 1 also shows that the test error decreases initially, then starts to increase as Theorem 3 suggests.
The inconsistency at the beginning could be brought about by other factors that affect the test error.
As seen in the Theorem 4, in addition to the ERC, the training error also contributes to the upper bound.
Nevertheless, the ERC becomes dominant by increasing the number of attention heads, the number of neighbors, the upper bound of weights norm, and the number of hidden layers, indicating the importance of a proper design of neural networks for MTSF to guarantee a smaller test error.
Among all the above factors that affect the generalization error, since the order of the upper bound of the weights norm is the highest, properly controlling the upper bound of the weights norm is especially signiﬁcant.
[Figure]
As reported in the previous literature (Wu et al., 2021), in- creasing the complexity hypothesis class in terms of larger weight matrices bounds could decrease the approximation error, but may also increase the estimation error, which corresponds to the second term of RHS in Theorem 4.
Therefore, in practical training process, we generally start with a simple neural network and gradually increase its complexity in terms of larger weight matrix bounds to improve the test performance, and the bound can be a tuning parameter in our model.
We call it weight control.
To show the meanings of weight control, we further de- veloped an improved version of the GAT called Weightsbounded GAT.
And we conduct an additional experiment to demonstrate its improvement over the vanilla GAT.
As the name suggests, the weight Frobenius norm of each layer is bounded by a hyperparameter.
We compare the Weights-bounded GAT with the vanilla GAT using test loss on the above stock return forecasting task.
We repeat the same train-test pipeline 20 times and collect 20 test losses for each model.
As Figure 2 shows, the Weightsbounded GAT performs much better than the vanilla GAT.
While the neural network training process can only minimize the training error and has no guarantee on generalization error, this generalization error bound analyzed in this study is more useful in the sense that it provides a solid guideline on GAT structure design to improve the models generalization performance for MTSF.
E THICS S TATEMENT
We have read the code of ethics carefully and ensured that our paper conforms to them.
R EPRODUCIBILITY S TATEMENT
We make our best effort to ensure the reproducibility of the paper’s experiments and provide clear guidelines.
More speciﬁcally, we include detailed experiment setup information in D.1.
Besides, we give detailed description on the architecture of the neural network we use and we explain how we control and adjust different variables in D.2.
We list all the important hyperparameters and their default values in D.3.
We give the website link for the dataset we use in D.4.
There is no extra data processing steps to be conducted.
All the work is done in the code in an end-to-end manner.
We also give the link to the source code in D.5.
We explain all assumptions of our theorems (Theorem 1, 2, 3, 4) clearly, and provide the complete proof of the theorems in the appendix §A, §B, and §C.
A PPENDICES T O M ULTIVARIATE T IME S ERIES F ORECASTING B Y G RAPH A TTENTION N ETWORKS W ITH T HEORETICAL G UARANTEES A P ROOF OF T HEOREM
Here we will derive the upper bound of ERC of two-layer GATs with on-dimensional output for the single-time-step prediction.
As mentioned in section 2.3, our GATs model’s second layer l (2) uses Z (1) ∈ R N × KD 1 as input, and outputs Z (2) ∈ R M × C .
The input to the ﬁrst layer l (1) is a set of node features
X = ( x 1 , . . .
, x N ) , , where each x i ∈ R D , D is the number of features in each node.
The ﬁrst layer produces Z (1) with each z (1) ∈ R KD 1 .
The Z (1) is composed by concatenation of K outputs from the K identically structured attention layers l (1 ,k ) , with each output denoted as
[Equation]
Now we write the output vector of GATs for all classes of node j , j ∈ [ M ] as,
[Equation]
[Equation]
Here we use index notation kr ∈ [ KD 1 ] because we already have indexes k ∈
[ K ] and r ∈ [ D 1 ] .
And it is easy to show that above output can be easily written as in vector format:
[Equation]
(cid:69)(cid:33) , where w r (1 ,k ) represents the column r of W (1 ,k ) , w kr (2) represents the row kr of W (2) , and w c (2) represents the column c of W (2) .
Then the class of functions deﬁned over the subset of the node set
M , |M| =
M , will be
[Equation]
[Equation]
Furthermore, we also provide a model space with a single dimensional output that corresponds to the c -th component of model output from f ( x ) for the c -th time step.
Now we write the output of node n of time-step c of l (2) as
[Equation]
and its vector format is
[Equation]
Now let hypothesis class F c ⊂ R X be a set of functions on x ∈ X .
Speciﬁcally [Equation] we have such single dimensional GAT function space deﬁned as
[Equation]
We have the ﬁrst layer’s matrix to be
[Equation]
The output of ﬁrst layer l 1 is
[Equation]
We then further write the output of attention head k of ﬁrst layer l (1) as
[Equation]
By the concatenation relationship
[Equation]
we have the row t of Z (1) to be
[Equation]
And we deﬁne each h (1 ,k,t ) as
[Equation]
with each h r (1 ,k,t ) ∈ R deﬁned as
[Equation]
Then the output of node j of class c can be re-written as
[Equation]
, where w c (2) represents column c of W (2) .
According to the deﬁnition of F c in deﬁnition 17, and ERC, we have
[Equation]
The inequality comes from the Cauchy-Schwartz inequality.
We further unfold the expectation
[Equation]
we have
[Equation]
The equality is due to the following derivation:
[Equation]
For a ﬁxed k -th attention head, we let the w 1(1 ,k ) , w 2(1 ,k ) , . . .
, w D 1 (1 ,k ) be the the columns of W (1 ,k ) , then, by positive homogeneity of σ , we have
[Equation]
j N (cid:88) t = p j,t (2) σ (cid:32) N (cid:88) i =
p t,i (1 ,k ) (cid:68) w r (1 ,k ) , x i (cid:69)(cid:33) =
K (cid:88) k
=
D 1 (cid:88) r
= (cid:13)(cid:13)(cid:13) w r (1 ,k ) (cid:13)(cid:13)(cid:13) 
M (cid:88)
j = (cid:15) j
N (cid:88) t = p j,t (2) σ  
N (cid:88) i
= p t,i (1 ,k ) (cid:68) w r (1 ,k ) (cid:13)(cid:13)(cid:13) w r (1 ,k ) (cid:13)(cid:13)(cid:13) , x i (cid:69)
The supremum of this quantity over w 1(1 ,k ) , w 2(1 ,k ) , . . .
, w D 1 (1 ,k ) under the constraint
that (cid:13)(cid:13) W (1 ,k ) (cid:13)(cid:13) 2 F ≤ M 21 =
(cid:80) D 1 r =1 (cid:13)(cid:13)(cid:13) w r (1 ,k ) (cid:13)(cid:13)(cid:13) 2 ≤ M 21 is attained when (cid:13)(cid:13)(cid:13) w r (1 ,k ) (cid:13)(cid:13)(cid:13) =
M 1 for some r and (cid:13)(cid:13)(cid:13) w r (cid:48) (1 ,k ) (cid:13)(cid:13)(cid:13) =
for all other r (cid:48) (cid:54) =
r .
In the end, only the r terms remain.
For simplicity of notation, we use w (1 ,k ) to mean that r ’s column w r (1 ,k ) .
Therefore [Equation] we have
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The inequality (a) is because sup( (cid:80) i x i ) ≤ (cid:80) i sup( x i ) .
have
[Table]

[Table]
[Equation]
[Equation]
where the equality (a) above is due to | x | = ( x ) + + ( x ) − , and the inequality (b) is due to sup A + B ≤ sup A + sup B , and the equality (c) comes from the symmetry in the distribution of the (cid:15) i random variables.
The equality (d) is due to sup A + = (sup A ) + .
The equality (e) is because the supremum is nonnegative, as when w (1 ,k )
= 0 , we can get the (cid:80)[-Mj =
-]
[Equation]
We then rewrite
M
[Equation]
By the same argument in sup( (cid:80) i x i ) ≤ (cid:80) i sup( x i ) , we have
[Equation]
For any ﬁxed t , we have
[Equation]
The inequality (a) is due to the contraction property of ERC.
The inequality (b) is due to the deﬁnition of graph attention matrix P , i.e., the maximum value of entries in each row is equal to 1.
Put (19), (20), (22), (23) together, we get
[Equation]
sup
[Equation]
Then we use the fact that σ is L σ Lipschitz continuous and the contraction property of ERC
[Equation]
We further derive that
[Equation]
[Table]
[Equation]
Where the inequality (a) is by Cauchy-Schwartz, and the inequality (b) is by Jensen’s inequality.
And the equality (c) follows the i.i.d.
condition of Rademacher sequences with zero-mean.
Next, we continue to bound the rest,
[Table]
[Equation]

[Equation]
where the ﬁrst inequality is by (cid:107) Ax (cid:107) 2 ≤ (cid:107) A (cid:107) 2 (cid:107) x (cid:107) 2 .
Combined with the earlier result, we get
[Equation]
B P OOF OF T HEOREM
We now turn to prove the Theorem 2.
Our proof strategy will be the following.
We ﬁrst provide a classic theorem that was used to bound the expected loss based on the empirical loss and the upper bound of ERC of loss functions associated with GATs for the multi-time-step situation F , then we derive this upper bound of ERC of loss functions of F by extending the upper bound of ERC of GATs with one-dimensional output for the single-time-step prediction F c , based on the existing theorem in literature.
Proof.
From Theorem Mohri et al.
(2018), we have the following holds for all f
[Equation]
where E ( f ) , R ( g F ) , and ˆ E ( f ) are deﬁned in section 2.4.
In order to extend to multi-dimensional vector-valued functions for MTSF, we will use the contraction inequality for the hypothesis class F of vector-valued functions f ∈ R C .
Lemma 5 (Corollary 4 in Maurer (2016)) .
let F be a class of vector-valued functions
f = ( f 1 , . . .
, f C ) ∈ R C , with each f c ∈ F c ⊂ R X , and let { x 1 , . . .
, x M } , { y 1 , . . .
, y M } be a data set with each x j ∈ X and y j ∈ Y .
Let ψ ( · , · ) be a 1 -Lipschitz function mapping V × Y to R , and associated to F , where F ⊂ V X .
Then we have
[Equation]
where (cid:15) cj is the c, j -th entry of a C × M matrix of independent Rademacher variables.
However, the RHS of equation 26 has supremum over all f ∈ F which is hard to compute and we can reduce it to scalar classes, and derive the following bound (Maurer, 2016):
[Equation]
Then we can derive the following upper bound for the loss function g ( f ( x ) , y ) with f ( x ) being vector-valued functions based on equation 26 and 27, where the RHS of equation 27 is related to
R ( F c ) :
[Equation]
[Equation]
C P ROOF T HEOREM 3 AND T HEOREM
Proof.
For l = L , by the deﬁnition of network output and the Rademacher complexity, we have
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
j
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) so we get
[Equation]
We denote the RHS as R ( L ) .
We further unfold the expectation
[Equation]
we have
[Equation]
This follows the same reason with equation 24.
Now for any l ∈ [ L − 1] , we have
[Table]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
   
Then we get the induction equation [Equation] which says
[Equation]

[Equation]

[Equation]
  
R
[Equation]
M l
K (cid:88) k
=
N (cid:88) t =
R ( l )
By induction, we get
R ( l
[Equation]
[Equation]
[Equation]
−
L − 1 (cid:89) l =
M l R (1) ≤ (4 L σ KN )
L −
L − 1 (cid:89) l =
M l    E (cid:15)    (cid:13) (cid:13) (cid:13)(cid:13) (cid:13)(cid:13)
M (cid:88)
j = (cid:15) j N (cid:88) i = p t,i (1 ,k ) x
i (cid:13) (cid:13) (cid:13)(cid:13) (cid:13)(cid:13)  
   
 1 /≤ (4 L σ KN )
L −(cid:32) L − 1 (cid:89) l = M l (cid:33) B ( NM ) 1 /
Combining with the previous result, we get
[Equation]
l
[Equation]
M
[Equation]
BN
[Equation]
[Equation]
[Equation]
D E XPERIMENT AND D ATA D. E XPERIMENT SETUP
The model is trained by the Adam optimizer (Kingma & Ba, 2014).
The learning rate is 1e-4 .
The number of training epochs is 30 .
The batch size is set to 5 .
We split the dataset into three parts for training, validation and testing with a ratio of 0 . 6 : 0 . 2 : 0 .
All the deep learning models, are implemented in Python with Pytorch and executed on a server with 8 NVIDIA GeForce GTX 2080Ti
GPUs.
The Nvidia rriver version is 470.141.03 and the CUDA version is 11.4.
[Table]
N EURAL N ETWORK A RCHITECTURE
We implement our neural network as a three-layer Graph Attention Neural network.
This includes the input layer, one hidden layer, and the output layer.
Each layer is a GATConv layer from the
Pytorch-Geometric package 3 .
We use ELU activation (Clevert et al., 2015) and Dropout (Srivastava et al., 2014) after both the input layer and the hidden layer.
For the number of heads variable, we change the number of attention heads of each layer and all layers use the same number of attention heads.
For the number of neighbors variable, we adjust the dropout rate inside each layer’s attention mechanism (different from the dropout layer) so that only a percentage of the nodes are considered when using the attention to aggregate information from a node’s neighbors.
For the weight norm variable, we adjust the bound of the Frobenius norm of the weight matrix inside each layer by using weight clipping.
Each element of the weight matrix is clipped to the threshold to make sure the Frobenius norm of the matrix is less than or equal to the bound.
For the number of layers variable, we adjust the number of hidden layers ranging from 1 to 8.
For the input norm variable, we make sure each node’s feature vector’s L1 norm is less than or equal to a bound ranging from 1 to 28.
For the number of labeled nodes variable, we use a ratio to adjust the size of the training dataset.
D.
[Table] [Table]
https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html
D. D ATA
We use the US daily stock market prices dataset from Kaggle 4 .
D. S OURCE C ODE
The source code is shared via Dropbox 5 .
Please refer to the included README ﬁle for detailed execution guide.
E S UPPLEMENTAL E XPERIMENTS
[Figure]
G e n e r a li za ti on l o ss y
[Figure]
x
[Figure]
−
[-= O ( x L − 1 / 2 ) ,L =(b)      
 Upperboundofweightsnorm-][Figure]
[Figure]
(a)
[Figure]
[Figure]
In the experiment section, for all experiments, where the number of layers, L , needs to be ﬁxed for studying the relationship of the generalization error with different GAT components that impact the
ERC, including the number of attention heads, the number of neighbors, the upper bound of weights norm, we use a three-layer GAT to demonstrate Theorem 3.
However, the upper bound of input norm and the number of labeled nodes affecting the ERC do not depend on the number of layers, thus, in our supplemental experiment for two-layer GATs, we only consider the three variables (the number of attention heads, the number of neighbors, and the upper bound of weights norm) that impact ERC and depend on the number of layers.
Notably, Theorem 1 for generalization error bound of two-layer
GATs is just a special case of Theorem 3 for deep GATs: when we let L equal to 2 in Theorem 3, the generalization error bound is identical to the bound in Theorem 1 multiplied with some constants.
We now give empirical results on two-layer GATs to see if it is consistent with Theorem 1 and Theorem 3.
In this appendix section, we include more experiment results from a two-layer GAT on three variables in the ERC.
As Figure 3 shows, when the number of attention heads increases, the generalization error also increases at a linear rate after certain values.
When the number of neighbors increases, the generalization error initially decreases when the number of neighbors is small, then starts increasing following the trend of a polynomial function.
As for the upper bound of weight norm, the generalization error increases quadratically with the increment of the norm.
The empirical results for all three variables generally conforms to the theoretical bound suggests by Theorem 1 as well as Theorem 3.
For some inconsistency between the results and the reference red line or theoretical results at the the beginning, the reason, as we have discussed, can be due to the trade-off between approximation error and estimation error, since when the complexity of hypothesis class increases, the former decreases and the latter increases.
In the long run, the estimation error or the ERC dominate the bound.
Thus, the results for L = 2 along with the three-layer experiments provide evidence for our theoretical ﬁndings (Theorem 1 and Theorem 3). =
F S UPPLEMENT R ESULTS F. MTSF UNDER F ULLY -S UPERVISED S ETTING INSTEAD OF S EMI - SUPERVISED S ETTING
We consider an undirected graph G =
( N , E ) .
N = ( n 1 , . .
. , n N ) , |N| = N , is a set of node labels representing the sources of N time series.
E ⊂ N × N is the set of edges representing the connection between series.
We let x i ∈
X , i ∈
[ N ] be a random variable representing the input feature vector of node n i for time series i .
For node i , its random input feature x i ∈
X
⊂ R D is a multi-dimensional vector, which contains all the historical values from T time steps, in other words, we let x i =
( x i,t , . . .
, x i,t − T +1 ) be the concatenation of T time steps; its true label y i ∈ Y ⊂ R C is the vector for the C-step-away values.
We sample n training data over G , where n = N × V for some V ∈ N + .
In other words, we have V batches of samples over graph G .
For any risk function g deﬁned over F , given the training set S
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } which includes n samples from
X × Y according to distribution Q , the expected/population risk E ( f ) and the empirical risk function ˆ E ( f ) are deﬁned as:
[Equation]
We introduce 1 ≤ ˜ N ≤ N as the effective size of data because the data among nodes corresponding to different time series are not independent.
If ˜ N = 1 , all of the time series are fully dependent.
If ˜
N = N , they are mutually independent.
So the ˜ N characterizes the strength of the independence among different time series.
Then we use ˜ n as the effective sample size of data which are i.i.d, where V ≤ ˜ n ≤ NV = n .
So the training set S will contain ˜ n data.
Given the inputs X =
( x i , . . .
, x N ) as multiple time series with each x i as input feature for node n i , the class of 2 -layer GATs for MTSF
f maps x to the output f ( x ) that represents a C-step-away prediction expressed in equation 2.
We consider a subset of such class requiring each f with a bounded weights norm, expressed as
[Equation]
Furthermore, we also provide a model space F c ⊂ R X with a single dimensional output that corresponds to the c -th component of model output from f ( x ) for the c -th time step, expressed as
[Equation]
Here we ﬁrst provide an upper bound of ERC of GAT class F c for single dimensional output of
MTSF.
Theorem 6 (Upper Bound of ERC of GAT class F c for MTSF) .
Let the activation function σ ( · ) be
L σ -Lipschitz continuous, and also satisfy σ (0) = 0 and σ ( αz ) =
ασ ( z ) for all α ≥ 0 .
Assume that the L 2 -norm of the feature vector x comes from a bounded domain X
= { x : (cid:107) x (cid:107) ≤ B } .
Assume that the Frobenius norm of every weights matrix in the ﬁrst layer of the GAT class is bounded, namely, (cid:13) (cid:13) W (1 ,k ) (cid:13) (cid:13) F ≤ M 1 with some constant M 1 > 0 for every k .
Also, the norm of the weights vector of the second layer of the GATs is bounded, (cid:13) (cid:13) (cid:13) w c (2) (cid:13) (cid:13) (cid:13) ≤ M 2 , where c ∈ [ C ] , with some constant M 2 > 0 .
Let N i denote the neighborhood of node i (including i ), let the number of neighbors of each node be identical, namely, for some common constant N e ∈ N + , assume N e :
= |N
i | for all node i ∈ N , furthermore, we consider the most general formulation, which allows every node to attend on every other node, i.e., N e = N .
Then let R ( F c ) be the ERC deﬁned in the deﬁnition 7 for GAT class F c in the deﬁnition 32, given the ˜ n sized input set { x 1 , . . .
, x ˜ n } , then we have
[Equation]
Similar proof details can be found in §A.
Theorem 7.
Deﬁne the hypothesis class F as the deﬁnition 15.
We suppose g is Lipschitz with constant L g .
Then for any δ ∈ (0 , 1) , with probability at least 1 − δ , for all f ∈ F , we have
[Equation]
where we have R ( F c )
=
O ( L σ BKM 1 M 2 ( N e ) 3 / 2 ˜ n − 1 / 2 ) from Theorem 6.
F. R ESULTS UNDER S UPERVISED S ETTING
In this section, we provide the results of experiments under the fully-supervised setting.
The multivariate time series have about 1500 stocks, and all of these stocks are used for training and testing.
We ensure that the training and testing data do not have any overlap.
The results for the relationship between the upper bound of the generalization error of the model and variables in the ERC are shown in the Figure 4.
[Figure]
[Table]
li
[Figure]
ti
[Figure]
o
[Figure]
(a)
[Figure]
[Figure]
[Figure]
G e n e r a li za ti on l o ss
[Figure]
[Figure]
x ) (d)
[Figure]
[Figure]
[Figure]
Effectivesamplesize
G e n e r a li za ti on l o ss y
= O ( x − 1 / 2 ) (f)
From Figure 4 we can see that the overall pattern for different variables are still consistent with those in Figure 1.
For the last variable, we use the size of the training data set instead of the number of the labeled nodes.
Our Theorem 6 reports that the ERC has a polynomial O (˜ n − 1 / 2 ) ’s dependence on the effective sample size .
As we can see from Figure 4f, the generalization loss decreases when the training data set size increases.
The red curve based on the theoretical value matches the pattern of the yellow curve based on our experiments, though slightly lower than the yellow curve after certain values of training data set size.
This is because we need more samples in the experiment to satisfy the effective samples in terms of the generalization error.
[Table]
F. C OMPARISON W ITH O THER B ASELINES
So far, two current works (Guo et al., 2019; Deng & Hooi, 2021) use GAT-based models to model multiple time series data, showing better performance in accuracy over other traditional linear methods (e.g., VAR), neural-network based methods (e.g., LSTM), and graph-network based methods (e.g., GNNs and GCNs).
Both works differ from ours as they do not consider any control of model variables (e.g., the weight matrix norm) and they lack theoretical guarantees in terms of the generalization error.
GDN is proposed by Deng & Hooi (2021) and is used for anomaly detection in multivariate time series.
ASTGCN is developed by Guo et al.
for multiple trafﬁc ﬂow forecasting.
We use GDN and ASTGCN as baselines to compare with Weight-bounded GAT.
To adapt them into our settings, we modify the output layer of GDN to perform the regression instead of classiﬁcation task.
We also keep other hyper-parameters the same across three methods.
Table 3 reports their average test error and standard deviation from 20 runs.
We use two evaluation metrics, the MSE and MAE.
Additionally, we also use ﬁgure 5 to show their generalization loss difference.
[Figure]
Figure
As Table 3 and Figure 5 show, the Weight-bounded GAT performs better than the GDN and ASTGCN.
Apart from using graph attention mechanism, GDN also learns a di- rected graph to model the causal-effect relationship between different nodes, which can capture the asymmetric dependency patterns.
Learning a graph structure can better dynamically capture the relationship between nodes, but also increase the learning time and model complexity.
In the meantime, while using attention between nodes, ASTGCN also considers the attention along the time axis.
Our method does not consider attention along the time as the features in every time step has already embedded the historical information.
Compared to GDN and ASTGCN, the weight-bounded control in our method plays an important role in controlling the generalization error, which is also veriﬁed in our theoretical analysis.
This result further corroborated the outperformance of our method over the GAT-based SOTA methods for MTSF.
F. M ODEL A RCHITECTURE