Transform-Based Multilinear Dynamical System for Tensor Time Series Analysis
Abstract
We propose a novel multilinear dynamical system (MLDS) in a transform domain, named L -MLDS, to model tensor time series.
With transformations applied to a tensor data, the latent multidimensional correlations among the frontal slices are built, and thus resulting in the computational independence in the transform domain.
This allows the exact separability of the multidimensional problem into multiple smaller LDS problems.
To estimate the system parameters, we utilize the expectation-maximization (EM) algorithm to determine the parameters of each LDS.
Further, L -MLDS signiﬁcantly reduces the model parameters and allows parallel processing.
Our general L -MLDS model is implemented based on discrete Fourier transform, discrete cosine transform and discrete wavelet transform, respectively.
Due to the nonlinearity of these transformations, L -MLDS is able to capture the nonlinear correlations within the data while the MLDS [1] assumes multi-way linear correlations.
On four real datasets, the proposed LMLDS achieves much higher prediction accuracy than the state-of-the-art MLDS and LDS with an equal number of parameters under different noise models.
In particular, the relative errors are reduced by 50% (cid:24) 99% .
Simultaneously, L -MLDS achieves an exponential improvement in the model’s training time than MLDS.
Introduction
Predicting the evolving trends of data sequences is an essential problem arising in various ﬁelds such as signal processing, environmental protection and economics.
A traditional model to describe a dynamically evolving data sequence is the linear dynamical system (LDS), where the observations and latent states are expressed as vectors.
In the era of big data, data in various applications is frequently represented as a time series of multidimensional arrays, called tensors, to preserve the inherent multidimensional correlations.
Of interest is the prediction of future terms of the time tensor series.
The obvious solution is to unfold each tensor to a vector, and then the LDS applies as in [2, 3].
However, LDS can not preserve the data structure, and it does not allow to determine the dimension of each mode of the latent tensor.
A multilinear dynamical system (MLDS) for modeling time tensor series is proposed in [1] to generalize the LDS by vectorizing the input tensors.
Expressing the latent states and observations as tensors and replacing the transition and projection matrices with multilinear operators, MLDS preserves the tensorial structure of the data and achieves a higher prediction accuracy than LDS.
The multilinear operators are assumed to be factorizable so that the number of model parameters is signiﬁcantly reduced (compared to LDS).
However, MLDS still takes a high computational cost to estimate the large number of covariance parameters.
Moreover, the method for estimating the mul- tilinear operators of MLDS in [1] may fall into local optimum, thus compromising the prediction accuracy.
We propose a novel multilinear dynamical system based on transform-based tensor model, named L -MLDS.
Working in the transform domain, the multilinear operators and covariances of the LMLDS model are sparse block diagonal matrices. This allows exact separability of an L -MLDS into multiple smaller LDSs and provides the opportunity for parallel processing.
To estimate the model parameters, we utilize the standard EM algorithm to determine the parameters of each LDS in the transform domain.
Therefore, the model involves fewer parameters, simple estimation procedures, and efﬁcient computation, leading to improvements in the model training.
In addition, L -MLDS allows arbitrary noise relationships among the tensorial elements without the restrictive assumption of isotropic noise used in [4, 5].
For the details of this work, please see [6].
Transform-Based Tensor Model
Let C denote complex numbers.
Vectors are denoted by boldface lowercase letters, e.g., a ; matrices are denoted by boldface capital letters, e.g., A ; and tensors are denoted by calligraphic letters, e.g., A .
We use A ( k ) to denote the k -th frontal slice of A and [ n ] to denote the index set f 1 ; 2 ; (cid:1) (cid:1) (cid:1) ; n g .
In this paper, we just consider the third-order tensor for ease of exposition.
Basic operators [7]: The operator MatView( (cid:1) ) takes a tensor A 2 C I (cid:2) J (cid:2) K and returns an IK (cid:2)
JK block diagonal matrix, with each block being an I (cid:2) J matrix, deﬁned as
MatView ( A ) =
diag ( A (1) ; (cid:1)
[Equation]
; A ( k ) [Equation] (cid:1) (cid:1) (cid:1)
[Equation]
A ( K ) ) : (1)
The operator Vec( (cid:1) ) takes a tensor B 2 C I (cid:2) 1 (cid:2) K and returns a vector of length IK , deﬁned as
[Equation]
Conversely, the operator TenView( (cid:1) ) folds MatView( A ) and Vec( B ) back to tensors A and B , respectively, i.e., TenView ( MatView ( A ))
=
A and TenView ( Vec ( B ))
=
B .
Given an invertible discrete transform L : C K !
C K , the elementwise multiplication is denoted by ◦ , and with (cid:11) ; (cid:12) 2 C K , the tube multiplication (cid:15) is deﬁned [7] as (cid:11) (cid:15) (cid:12) = L (cid:0) 1 ( [Equation]
( (cid:11) ) ◦ L ( (cid:12) )) , and L (cid:0)is the inverse of L .
[Figure]
We use e A
= L ( A ) 2 C
I (cid:2) J (cid:2) K to denote the tensor obtained by taking the transform L of all the tubes along the third dimension of A 2 C I (cid:2) J (cid:2) K .
The transformation L builds the correlations among the frontal slices in the transform domain just like threading the wires through them.
Therefore, while the frontal slices of a tensor in time domain are dependent, they are in fact independent in the transform domain.
Deﬁnition 1.
The L -product C =
A (cid:15) B of A 2 C I (cid:2) K (cid:2) L and B[-
-]
C K (cid:2) J (cid:2)
L is a tensor in C I (cid:2) J (cid:2) L , with C ( i; j; :)
=
∑ Kk =1 A ( i; k; : ) (cid:15) B ( k; j; :) , for i
[ I ] and j 2 [ J ] .
Lemma 1.
[7] The L -product C =
A (cid:15) B can be converted to the matrix multiplication in the transform domain, similar to the convolution theorem, MatView ( e C ) =
MatView ( e A ) (cid:1) MatView ( e B ) .
In particular, given A 2 C I (cid:2) J (cid:2) K and B 2 C J (cid:2) 1 (cid:2) K , the L -product C
= A (cid:15) B can be calculated as
Vec ( e C ) =
MatView ( e A ) (cid:1) Vec ( e B ) .
In this case, we call A a multilinear operator of B [7–9].
Different from the tensor normal distribution in [10] which is restricted to symmetric second-order tensors, we deﬁne an L -normal distribution in a transform domain for arbitrary second-order tensors.
The corresponding random tensors, called L -random tensors, are used to construct L -MLDS.
Deﬁnition 2.
( L -Normal Distribution)
Given a tensor X 2 C J (cid:2) 1 (cid:2) K , let e X =
L ( X ) , and then we say X has the L -normal distribution with expectation U 2 C J (cid:2) 1 (cid:2) K and covariance Q 2 C J (cid:2) J (cid:2) K , denoted by
[Equation]
if and only if
[Equation]
where CN is the traditional complex normal distribution.
[-( e Q (cid:0) 
-]
[-C H 
-]
Transform-Based Multilinear Dynamical System
The L -MLDS model consists of a sequence X 1 ; (cid:1) (cid:1) (cid:1) ; X N of latent tensors, where X n 2 C J (cid:2) 1 (cid:2) K for all n .
Each latent tensor X n associates with an observation Y n 2 C I (cid:2) 1 (cid:2) K .
The L -MLDS is initialized by a latent tensor X 1 distributed as
[Equation]
Given X n , 1 (cid:20) n (cid:20) N (cid:0) 1 , we generate X n +1 according to the conditional distribution
[Equation]
where Q is the conditional covariance tensor shared by all X n ; 2 (cid:20) n (cid:20) N , and
A 2 C J (cid:2) J (cid:2) K is the transition tensor which describes the dynamics of the evolving sequence X 1 ; (cid:1) (cid:1) (cid:1) ; X N .
For each X n , the corresponding observation Y n is generated by the conditional distribution
[Equation]
where R is the conditional covariance tensor shared by all Y n , and C 2 C I (cid:2) J (cid:2) K is the projection tensor which transforms latent X n to the corresponding observation Y n .
Suppose I =
J = K = n .
Then, the parameter complexities of LDS and MLDS are O ( n 4 ) and that of L -MLDS is O ( n 3 ) .
Thus L -MLDS signiﬁcantly reduces the number of parameters as the dimensions of the tensors increase.
Conversely, with equal number of parameters, L -MLDS tends to have a greater dimensionality ( J (cid:2) K ) of the latent state.
Generally, the longer the vectorized latent tensor is, the more information of the corresponding observation it has.
The problem of L -MLDS identiﬁcation is to estimate the parameters (cid:2)
= fU 0 ; Q 0 ; A ; Q ; C ; Rg from the given time series of observations Y 1 ; (cid:1) (cid:1) (cid:1) ; Y N .
For the existence of unknown latent states X n in the L -MLDS, we cannot directly maximize the likelihood of the data with respect to (cid:2) .
According to Deﬁnition 2, the L -MLDS speciﬁed by (5), (6), and (7) can be exactly divided into K indenpendent LDSs in the transform domain with each LDS being deﬁned as
[Equation]
Hence, the problem of estimating (cid:2) is exactly separated into K independent subproblems of estimating (cid:18) ( k ) =
f e U ( k ) 0 ; e Q ( k ) 0 ;
e A ( k ) ; e Q ( k ) ; e C ( k ) ; e R ( k ) g with incomplete data [11].
Then, we use the
EM algorithm to estimate each (cid:18) ( k ) , k 2 [ K ] , and ﬁnally convert all those subsystem components to time domain.
For the speciﬁc process, see Figure 2.
[-( ) Vec  
 N ... ~2 ~ ~ N ...-][Equation]
LDS K 
K ( ) TenView  {+
+}
N ~ 1 2 ~ ~ N ( ) 1 K ( ) 2 K ( ) KN
Performance Evaluation on Real Data
We conduct experiments with the noise covariances in the models being diagonal and non-diagonal, respectively.
In all the experiments, the LDS latent dimensionality is always set to the smallest value such that the number of parameters of the LDS is greater than or equal to that of the MLDS.
Also, the latent dimensionality J of each LDS in the transform domain of L -MLDS is set to the largest value such that the number of parameters of the L -MLDS is less than or equal to that of MLDS.
We use the following datasets in evaluations, and the codes are available online [12].
SST [1]: A 5-by-6 grid of sea-surface temperatures.
Each model was trained on the ﬁrst 1800 epochs and tested on the last 200 epochs.
Video [1]: A 10 (cid:2) 10 patch for each frame.
Each model was trained on the ﬁrst 1000 frames and tested on the last 171 frames.
Tesla [13]: A 14 (cid:2) 5 patch for each epoch.
Each model was trained on the ﬁrst 1100 epochs and tested on the last 160 epochs.
NASDAQ-100 [14]: Opening, closing, high, and low for 50 randomly-chosen NASDAQ-100 companies.
Each model was trained on the ﬁrst 2000 epochs and tested on the last 186 epochs.
[Figure]
[Figure] [Figure]
The comparisons ( shown in Figure 3 and 4) demonstrate that our L -MLDS is able to achieve a high prediction accuracy for arbitrary noise relationships among the tensorial elements, and reduces the relative errors by 50% (cid:24) 99% .
In addition to the higher prediction accuracy, L -MLDS reduces the training time by orders of magnitude compared to MLDS, see Figure 5.
Simultaneously, the longer the vectorized inputs are, the more obvious the improvement will be.
Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grant 61671345.