Transform-Based Multilinear Dynamical System for Tensor Time Series Analysis
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose a novel multilinear dynamical system (MLDS) in a transform domain,named L -MLDS, to model tensor time series. With transformations applied to atensor data, the latent multidimensional correlations among the frontal slices arebuilt, and thus resulting in the computational independence in the transform domain.
This allows the exact separability of the multi-dimensional problem intomultiple smaller LDS problems.
To estimate the system parameters, we utilize theexpectation-maximization (EM) algorithm to determine the parameters of each LDS.
Further, L -MLDSs signiﬁcantly reduce the model parameters and allowsparallel processing.
Our general L -MLDS model is implemented based on different transforms: discrete Fourier transform, discrete cosine transform and discretewavelet transform.
Due to the nonlinearity of these transformations, L -MLDS isable to capture the nonlinear correlations within the data unlike the MLDS [1]which assumes multi-way linear correlations.
Using four real datasets, the proposed L -MLDS is shown to achieve much higher prediction accuracy than thestate-of-the-art MLDS and LDS with an equal number of parameters under different noise models. In particular, the relative errors are reduced by 50% (cid:24) 99% . Simultaneously, L -MLDS achieves an exponential improvement in the model’straining time than MLDS.
Introduction
Predicting the evolving trends of data sequences is an essential problem arising in various ﬁeldssuch as signal processing, environmental protection and economics.
A traditional model to describea dynamically evolving data sequence is the linear dynamical system (LDS) where the observationsand latent states are expressed as vectors.
In the era of big data, data in various applications isfrequently represented as a time series of multidimensional arrays, called tensors, to preserve theinherent multidimensional correlations.
Of interest is the prediction of future terms of the timetensor series.
The obvious solution is to unfold each tensor to a vector, and then the LDS appliesas in [2, 3].
LDS can not preserve the data structure, and it does not allow determination of thedimension of each mode of the latent tensor.
Bayesian probabilistic tensor factorization (BPTF) [4]is an explicit model for predicting tensor time series, which concatenates the members of the tensortime series and yields a higher-order tensor.
Though BPTF preserves the tensorial structure, thelatent structure is limited.
A multilinear dynamical system (MLDS) for modeling time tensor series is proposed in [1] to generalize the LDS by vectorizing the input tensors.
Expressing the latent states and observations astensors and replacing the transition and projection matrices with multilinear operators, MLDS preserves the tensorial structure of the data.
The multilinear operators are factorized as the Kroneckerproduct of multiple smaller matrices so that the number of model parameters is signiﬁcantly re- duced (compared to LDS).
MLDSs preserve the tensor structure with more ﬂexible dimensionalitiesof the latent tensors and achieve a higher prediction accuracy than LDS.
However, MLDSs still takea high computational cost to estimate the large number of covariance parameters.
Moreover, themethods for estimating the multilinear operators of MLDS in [1] may fall into local optimum, thuscompromising the prediction accuracy.
To address the aforementioned issues, we propose a novel multilinear dynamical system based ontransform-based tensor model (in which the transform is denoted by L , thus we call it L -MLDS).
Working in the transform domain, we deﬁne a probabilistic model to construct the L -MLDS.
Themultilinear operators and covariances of the L -MLDS model are represented as sparse block diagonal matrices in the transform domain, allowing exact separability of the L -MLDS model intomultiple smaller LDSs and providing the opportunity for parallel processing.
To estimate the modelparameters, we utilize the standard EM algorithm to determine the parameters of each LDS in thetransform domain.
Therefore, the model involves fewer parameters, simple estimation procedures,efﬁcient computation, and potential for parallel processing, leading to improvements in the modeltraining.
The L -MLDS model allows arbitrary noise relationships among the tensorial elements without therestrictive assumption of isotropic noise used in [5, 6].
To assess the performance of the L -MLDSmodel, we implement it using three different transformations and conduct experiments under different noise models.
Simulation results with real data demonstrate that the proposed L -MLDS achieveshigher prediction accuracy than MLDS and LDS with an equal number of parameters while takingmuch less time to train the model.
Transform-Based Tensor Model
Let C denote complex numbers.
Vectors are denoted by boldface lowercase letters, e.g., a ; matricesare denoted by boldface capital letters, e.g., A ; and higher-order tensors are denoted by calligraphicletters, e.g., A .
The index set f 1 ; 2 ; (cid:1) (cid:1) (cid:1) ; n g is denoted by [ n ].
Let N denote positive integers and
I; J; K 2 N , for a third-order tensor A 2 C I (cid:2) J (cid:2) K , we use A ( i; j; :) to denote the mode-3 tube and
A ( k ) to denote the k -th frontal slice [7].
In this paper, we just consider the third-order tensor forease of exposition.
Basic operators: The operator MatView( (cid:1) ) takes a tensor A 2 C I (cid:2) J (cid:2) K and returns an IK (cid:2)
JKblock diagonal matrix, with each block being an I (cid:2) J matrix, deﬁned as
MatView ( A ) =
diag ( A (1) ; (cid:1) (cid:1) (cid:1) ; A ( k ) ; (cid:1) (cid:1) (cid:1) ;
A ( K ) ) : (1)
The operator Vec( (cid:1) ) takes a tensor B 2 C I (cid:2) 1 (cid:2) K and returns a vector of length IK , deﬁned as
Vec ( B ) =
[ B (1) ; (cid:1) (cid:1) (cid:1) ; B ( k ) ; (cid:1) (cid:1) (cid:1) ; B ( K ) ] : (2)
Conversely, the operator TenView( (cid:1) ) folds MatView( A ) and Vec( B ) back to tensors A and B , respectively, i.e., TenView ( MatView ( A ))
=
A and TenView ( Vec ( B ))
=
B .
Given an invertible discrete transform L : C K !
C K , the elementwise multiplication is denoted by 70 ◦ , and with (cid:11) ; (cid:12) 2 C K , the tube multiplication (cid:15) is deﬁned as (cid:11) (cid:15) (cid:12) = L (cid:0) 1 ( L ( (cid:11) ) ◦ L ( (cid:12) )) , and 71 L (cid:0)is the inverse of L .
[Figure]
[-where L (cid:0) 1 is the inverse of L . Let L ( A ) 2 C I (cid:2) J (cid:2) K denote the tensor obtained by taking the transform L of all the tubes alongthe third dimension of A 2 C I (cid:2) J (cid:2) K , i.e., for i 
-][Figure]
We use e A
= L ( A ) 2 C
I (cid:2) J (cid:2) K to denote the tensor obtained bytaking the transform L of all the tubes along the third dimensionof A 2 C I (cid:2) J (cid:2) K .
The transformation L builds the correlationsamong the frontal slices in the transform domain just like threadingthe wires through them.
Therefore, while the frontal slices of atensor in time domain are dependent, they are in fact independentin the transform domain.
Deﬁnition 1.
The L -product C =
A(cid:15)B of A 2 C I (cid:2) K (cid:2)
L and B {+
+} C K (cid:2) J (cid:2)
L is a tensor in C I (cid:2) J (cid:2) L , with C ( i; j; :)
=
∑ Kk =1 A ( i; k; :) (cid:15) B ( k; j; :) , for i
[ I ] and j 2 [ J ] .
Lemma 1.
The L -product C =
A (cid:15) B can be converted to the matrix multiplication in the transformdomain, similar to the convolution theorem, MatView ( e C ) =
MatView ( e A ) (cid:1) MatView ( e B ) .
In particular, given A 2 C I (cid:2) J (cid:2) K and B 2 C J (cid:2) 1 (cid:2) K , the L -product C
= A (cid:15) B can be calculated as
Vec ( e C ) =
MatView ( e A ) (cid:1) Vec ( e B ) .
In this case, we call A a multilinear operator of B [8, 9].
Different from the tensor normal distribution in [10] which is [-( e A ( k ) ) H , k 
 [ K ] , and the multiplication reversal property of the Hermitian transpose holds [12], i.e., ( A (cid:15) B ) H = B H (cid:15)-]restricted to symmetric second-ordertensors, we deﬁne an L -normal distribution in a transform domain for arbitrary second-order tensors.
The corresponding random tensors, called L -random tensors, are used to construct L -MLDS.
Deﬁnition 2.
( L -Normal Distribution)
Given a tensor X 2 C J (cid:2) 1 (cid:2) K , let e X =
L ( X ) , and then wesay X has the L -normal distribution with expectation U 2 C J (cid:2) 1 (cid:2) K and covariance Q 2 C J (cid:2) J (cid:2) K ,denoted by
[Equation]
if and only if
[Equation]
where CN is the traditional complex normal distribution.
Suppose X 2 C J (cid:2) 1 (cid:2) K and Y 2 C I (cid:2) 1 (cid:2) K are jointly distributed as
X [Equation] CN
[Equation] ( U ; Q ) and
[Equation]
(5) where C 2 C I (cid:2) J (cid:2) K is the multilinear operator of X .
We can obtain the marginal distribution of Yand the posterior distribution of X given Y as follows.
Lemma 2.
Suppose the joint distribution of L -random tensors Y 2 C I (cid:2) 1 (cid:2) K and X 2 C J (cid:2) 1 (cid:2) K isgiven by (5), then the marginal distribution of Y is [Equation]
[Equation]
where C H 2 C J (cid:2) I (cid:2) K .
The conditional distribution of X given Y is
[Equation]
where M = L (cid:0) 1 ( TenView ( (cid:6) ( e C H e R (cid:0) 1 Vec ( e Y ) + e Q (cid:0) 1 Vec ( e U )))) ,
G = L (cid:0) 1 ( TenView ( (cid:6) )) , e R =
MatView ( e R ) , e Q = MatView ( e Q ) , e C =
MatView ( e C ) , and (cid:6) =
( e Q (cid:0)
+ e C H e R (cid:0) 1 e C ) (cid:0) 1 .
{+C H 
+}
C J (cid:2) I (cid:2) K such that MatView ( L ( C H )) =
MatView ( L ( C ))
H [11].
Proof.
Deﬁnition 2, lemma 1 and (5) imply that Vec( e X ) and Vec( e Y ) given Vec( e X ) in the transformdomain follow:
[Equation]
By the properties of the multivariate normal distribution [2], the marginal distribution of Vec ( e Y ) andthe conditional distribution of Vec ( e X ) given Vec ( e Y ) are
[Equation]
Converting (8) and (9) back to the tensor forms by Deﬁnition 2, we can obtain (6) and (7), respectively.
Transform-Based Multilinear Dynamical System
We deﬁne the L -MLDS by treating each tensor Y n as an L -random tensor and relating each modelcomponent with a multilinear transformation.
3.1 System Description
The L -MLDS model consists of a sequence X 1 ; (cid:1) (cid:1) (cid:1) ; X N of latent tensors, where X n 2 C J (cid:2) 1 (cid:2)
Kfor all n .
Each latent tensor X n associates with an observation Y n 2 C I (cid:2) 1 (cid:2) K .
The L -MLDS isinitialized by a latent tensor X 1 distributed as
X 1 (cid:24)
[Equation]
L ( [Equation] Given X n , 1 (cid:20) n (cid:20) N (cid:0) 1 , we generate X n +1 according to the conditional distribution [Equation]
[Equation]
where Q is the conditional covariance tensor shared by all X n ; 2 (cid:20) n (cid:20) N , and
A 2 C J (cid:2) J (cid:2) K isthe transition tensor which describes the dynamics of the evolving sequence X 1 ; (cid:1) (cid:1) (cid:1) ; X N .
For each 120 X n , the corresponding observation Y n is generated by the conditional distribution
[Equation]
where R is the conditional covariance tensor shared by all Y n , and C 2 C I (cid:2) J (cid:2) K is the projectiontensor which transforms latent X n to the corresponding observation Y n .
3.2 Parameter Complexity Analysis
Suppose the relationships among the elements of a tensor are non-independent.
Then the number ofparameters in LDS is
[Equation]
[Equation] For the MLDS model, the multilinear operators and covariance tensors of an N -order tensor are2 N -order tensors, thus [Equation]
[Equation]
While for the L -MLDS, the multilinear operators and the covriances are sparse in the transformdomain, i.e.,
[Equation]
-MLDS
[Equation]
[Equation]
Suppose I =
J = K = n , then the parameter complexities of LDS and MLDS are O ( n 4 ) andthat of L -MLDS is O ( n 3 ) .
Thus L -MLDS signiﬁcantly reduces the number of parameters as thedimensions of the tensors increase.
Conversely, with equal number of parameters, L -MLDS tendsto have a greater dimensionality ( J (cid:2) K ) of the latent state.
Generally, the longer the vectorizedlatent tensor is, the more information of the corresponding observation it has.
Therefore, L -MLDSis able to achieve higher prediction accuracy than LDS and MLDS.
3.3 System Identiﬁcation
The problem of L -MLDS identiﬁcation is to estimate the parameters (cid:2)
= fU 0 ; Q 0 ; A ; Q ; C ; Rgfrom the given time series of observations Y 1 ; (cid:1) (cid:1) (cid:1) ;
Y N .
For the existence of unknown latent states 139 X n in the L -MLDS, we cannot directly maximize the likelihood of the data with respect to (cid:2) .
According to Deﬁnition 2, the L -MLDS speciﬁed by (10), (11), and (12) can be divided into Kindenpendent LDSs in the transform domain with each LDS being deﬁned as
[Equation]
Hence, the problem of estimating (cid:2) is exactly separated into K independent subproblems of estimating (cid:18) ( k ) =
f e U ( k ) 0 ; e Q ( k ) 0 ;
e A ( k ) ; e Q ( k ) ; e C ( k ) ; e R ( k ) g with incomplete data [12].
Then, we use the EM algorithm to estimate each (cid:18) ( k ) , k 2 [ K ] , and ﬁnally convert all those subsystem componentsto time domain.
For the speciﬁc process, see Figure 2.… …    … … … ...
{+( ) Vec  
+}
N ...
~2 ~ ~ N ...
1 LDS 2 LDS ...
1 ( )   (1) 
(2) 
( ) K  LDS K ~( ) 1 K ~( ) 2 K
[Figure]

[Figure]
Performance Evaluation
4.1 Model Initializations
We evaluate the performance of L -MLDS by comparing it with MLDS [1] and LDS (vectorized tensors as inputs) on real data.
Our general L -MLDS model is implemented based on discrete Fouriertransform (dft-MLDS), discrete cosine transform (dct-MLDS) and discrete wavelet transform (dwtMLDS).
The L -MLDS parameters are initialized such that e U ( k )is drawn from the standard normaldistribution, the diagonal block matrices e Q ( k ) 0 , e Q ( k ) and e R ( k ) are identity matrices for each k , andthe columns of e A ( k ) and e C ( k ) are the ﬁrst J eigenvectors of singular-value-decomposed matriceswith entries drawn from the standard normal distribution.
The LDS parameters are initialized in thesame way as the k -th LDS of L -MLDS.
The MLDS parameters are initialized in the same way as[1].
Denote ∥X∥ F = √∑ i;j;k jX ijk j 2 as the Frobenius norm of a third-order tensor.
The predictionerror " L n of a given L -MLDS model for the n -th member Y n in a time series is
[Equation]
Each estimation Y L n is given in the following way: we compute the prediction b Y ( k ) n of the k -th LDSin the transform domain, i.e.,
[Equation]
(18) where b X ( k ) N train is the estimation of latent state of the last member of the training sequence in the k -th LDS.
Then, Vec ( e Y L n ) =
[ b Y (1) n ; (cid:1) (cid:1) (cid:1) ; b Y ( k ) n ; (cid:1) (cid:1) (cid:1) ; b Y ( K ) n ] , we obtain Y L n =
L (cid:0) 1 ( Tenview ( Vec ( e Y L n ))) .
The elements of the real data are usually dependent, i.e., the covariance matrix is non-diagonal in
LDS.
Conversely, if the elements of the data are independent, the covariance matrix is diagonal.
We conduct experiments with the noise covariances in the models being diagonal and non-diagonal,respectively.
In all the experiments, the LDS latent dimensionality is always set to the smallestvalue such that the number of [-iterations -4. L og - li k e li hood 
-]parameters of LDS is greater than or equal to that of MLDS.
For faircomparisons, the latent dimensionality J of each LDS in the transform domain of L -MLDS is set tothe largest value such that the number of parameters of the L -MLDS is less than or equal to that of
MLDS.
4.2 Performance Results with Real Data
We use the following datasets:
SST [1]: A 5-by-6 grid of sea-surface temperatures from 5 ◦ N, 180 ◦ W to 5 ◦ S, 110 ◦ W recorded hourly from 7:00PM on 4/26/94 to 3:00AM on 7/19/94, yielding 2000 epochs.
Video [1]: 1171 grayscale frames of ocean surf during low tide.
Tesla [13]: Opening, high, low, closing and adjusted-closing of the stock prices of 14 car and oil companies (e.g., Tesla Motors Inc.), from 5/4/13 to 5/4/18 (1260 epochs).
NASDAQ-100 [14]: Opening, closing, high, and low for 50 randomly-chosen NASDAQ-100 com-
panies, from 7/26/16 to 4/28/17 (2186 epochs).
For the SST dataset, each model was trained on the ﬁrst 1800 epochs and tested on the last 200epochs.
When the latent state dimensionality of the MLDS is set to 2 (cid:2) 3 , the results are shown in
Figure 3(a) and Figure 4(a).
For the Video dataset, a 10 (cid:2) 10 patch for each frame, each model wastrained on the ﬁrst 1000 frames and tested on the last 171 frames.
When the latent state dimensionality of the MLDS is set to 5 (cid:2) 5 , the results are shown in Figure 3(b) and Figure 4(b).
For the Tesladataset, a 14 (cid:2) 5 patch for each epoch, each model is trained on the ﬁrst 1100 epochs and testedon the last 160 epochs.
When the latent dimensionality of the MLDS is set to 5 (cid:2) 2 , the result areshown in Figure 3(c) and Figure 4(c).
For the NASDAQ-100 dataset, each model is trained on theﬁrst 2000 epochs and tested on the last 186 epochs.
When the latent dimensionality of the MLDS isset to a 10 (cid:2) 3 , the results are shown in Figure 3(d) and Figure 4(d).
[Figure]
[Figure]
[Figure]
The comparisons ( shown in Figure 3 and 4) demonstrate that our L -MLDS is able to achieve a highprediction accuracy for arbitrary noise relationships among the tensorial elements
[Figure]
and reduces therelative errors by 50% (cid:24) 99% . In addition to the higher prediction accuracy, L -MLDS reduces thetraining time by orders of magnitude compared to MLDS, see Figure 5.
Simultaneously, the longerthe vectorized inputs are, the more obvious the improvement will be.
Conclusions
In this work, we have proposed a novel multilinear dynamical system, L -MLDS, to model tensortime series.
In deﬁning the L -MLDS, we work in a transform domain, which brings a signiﬁcantreduction in model complexity compared to the prior models LDS and MLDS.
With nonlinear transformations applied, L -MLDS is able to capture the nonlinear correlations among tensors of a timeseries, leading to more accurate prediction than assuming linear correlations.
In addition, the exactseparation of an L -MLDS into several LDSs enables efﬁcient computation and allows for parallelprocessing that overcomes the curse of dimensionality when dealing with big data.
On four realdatasets, the proposed L -MLDS not only achieves higher prediction accuracy, but takes less time fortraining than MLDS and LDS.