L EARNING E FFICIENT T ENSOR R EPRESENTATIONS WITH R ING S TRUCTURE N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Tensor train (TT) decomposition is a powerful representation for high-order tensors, which has been successfully applied to various machine learning tasks in recent years.
In this paper, we propose a more generalized tensor decomposition with ring structure network by employing circular multilinear products over a sequence of lower-order core tensors, which is termed as TR representation.
Several learning algorithms including blockwise ALS with adaptive tensor ranks and SGD with high scalability are presented.
Furthermore, the mathematical properties are investigated, which enables us to perform basic algebra operations in a computationally efﬁciently way by using TR representations.
Experimental results on synthetic signals and real-world datasets demonstrate the effectiveness of TR model and the learning algorithms.
In particular, we show that the structure information and high-order correlations within a 2D image can be captured efﬁciently by employing an appropriate tensorization and TR decomposition
I NTRODUCTION
Tensor decompositions aim to represent a higher-order (or multi-dimensional) data as a multilinear product of several latent factors, which attracted considerable attentions in machine learning (Yu &
Liu, 2016; Anandkumar et al., 2014; Romera-Paredes et al., 2013; Kanagawa et al., 2016; Yang et al., 2017) and signal processing (Zhou et al., 2016) in recent years.
For a d th-order tensor with “square” core tensor of size r , standard tensor decompositions are the canonical polyadic (CP) decomposition (Goulart et al., 2015) which represents data as a sum of rank-one tensors by O ( dnr ) parameters and
Tucker decomposition (De Lathauwer et al., 2000; Xu et al., 2012; Wu et al., 2014; Zhe et al., 2016) which represents data as a core tensor and several factor matrices by O ( dnr + r d ) parameters.
In general, CP decomposition provides a compact representation but with difﬁculties in ﬁnding the optimal solution, while Tucker decomposition is stable and ﬂexible but its number of parameters scales exponentially to the tensor order.
Recently, tensor networks have emerged as a powerful tool for analyzing very high-order tensors (Cichocki et al., 2016).
A powerful tensor network is tensor train / matrix product states (TT/MPS) representation (Oseledets, 2011), which requires O ( dnr 2 ) parameters and avoid the curse of dimensionality through a particular geometry of low-order contracted tensors.
TT representation has been applied to model weight parameters in deep neural network and nonlinear kernel learning (Novikov et al., 2015; Stoudenmire & Schwab, 2016; Tsai et al., 2016), achieving a signiﬁcant compression factor and scalability.
It also has been successfully used for feature learning and classiﬁcation (Bengua et al., 2015).
To fully explore the advantages of tensor algebra, the key step is to efﬁciently represent the real-world dataset by tensor networks, which is not well studied.
In addition, there are some limitations of TT including that i) the constraint on TT-ranks, i.e., r 1 = r
d +1 = 1 , leads to the limited representation ability and ﬂexibility; ii) TT-ranks are bounded by the rank of k-unfolding matricization, which might not be optimal; iii) the permutation of data tensor will yield an inconsistent solution, i.e., TT representations and TT-ranks are sensitive to the order of tensor dimensions.
Hence, ﬁnding the optimal permutation remains a challenging problem.
In this paper, we introduce a new structure of tensor networks, which can be considered as a generalization of TT representations.
First of all, we relax the condition over TT-ranks, i.e., r 1 = r d +1 = 1 , leading to an enhanced representation ability.
Secondly, the strict ordering of multilinear products between cores should be alleviated.
Third, the cores should be treated equivalently by
T nn d ··· n k ··· n=
Z
Z d ···
Z k ··· Znn d ··· n k ··· nrrr d r k + r k r making the model symmetric.
To this end, we add a new connection between the ﬁrst and the last core tensors, yielding a circular tensor products of a set of cores (see Fig. 2).
More speciﬁcally, we consider that each tensor element is approximated by performing a trace operation over the sequential multilinear products of cores.
Since the trace operation ensures a scalar output, r 1 = r d +
= 1 is not necessary.
In addition, the cores can be circularly shifted and treated equivalently due to the properties of the trace operation.
We call this model tensor ring (TR) decomposition and its cores tensor ring (TR) representations .
To learn TR representations, we ﬁrstly develop a non-iterative
TR-SVD algorithm that is similar to TT-SVD algorithm (Oseledets, 2011).
To ﬁnd the optimal lower
TR-ranks, a block-wise ALS algorithms is presented.
Finally, we also propose a scalable algorithm by using stochastic gradient descend, which can be applied to handling large-scale datasets.
Another interesting contribution is that we show the intrinsic structure or high order correlations within a 2D image can be captured more efﬁciently than SVD by converting 2D matrix to a higher order tensor.
For example, given an image of size I × J , we can apply an appropriate tensorization operation (see details in Sec. 5.2) to obtain a fourth order tensor, of which each mode controls one speciﬁc scale of resolution.
To demonstrate this, Fig.
1 shows the effects caused by noise corruption of speciﬁc tensor cores.
As we can see, the ﬁrst mode corresponds to the small-scale patches, while the 4th-mode corresponds to the large-scale partitions.
We have shown in Sec.
5.2 that TR model can represent the image more efﬁciently than the standard SVD.
T ENSOR R ING D ECOMPOSITION
The TR decomposition aims to represent a high-order (or multi-dimensional) tensor by a sequence of 3rd-order tensors that are multiplied circularly.
Speciﬁcally, let T be a d th-order tensor of size n 1 × n 2 ×· · ·× n d , denoted by T ∈ R n 1 ×···× n d , TR representation is to decompose it into a sequence of latent tensors
[Equation]
, which can be expressed in an element-wise form given by
[Equation]
T ( i 1 , i 2 , . .
, i d ) denotes the ( i 1 , i 2 , . .
, i d ) th element of the tensor.
Z k ( i k ) denotes the i k th lateral slice matrix of the latent tensor Z k , which is of size r k × r k +1 .
Note that any two adjacent latent tensors,
Z k and Z k +1 , have a common dimension r k +1 on their corresponding modes.
The last latent tensor
Z d is of size
r d × n d × r 1 , i.e., r d +1 = r 1 , which ensures the product of these matrices is a square matrix.
These prerequisites play key roles in TR decomposition, resulting in some important numerical properties.
For simplicity, the latent tensor Z k can also be called the k th- core (or node ).
The size of cores, r k , k = 1 , 2 , . . .
, d , collected and denoted by a vector r =
[ r 1 , r 2 , . . .
, r d ] T , are called TR-ranks .
From (1), we can observe that T ( i 1 , i 2 , . . .
, i d ) is equivalent to the trace of a sequential product of matrices {
Z k ( i k ) } .
Based on (1), we can also express TR decomposition in the tensor form, given by
[Equation]
where the symbol ‘ ◦ ’ denotes the outer product of vectors and z k ( α k ,
α k +1 ) ∈ R n k denotes the ( α k , α k +1 )
th mode-2 ﬁber of tensor Z k .
The number of parameters in TR representation is O ( dnr 2 ) , which is linear to the tensor order
d as in TT representation.
The TR representation can also be illustrated graphically by a linear tensor network as shown in
Fig.
A node represents a tensor (including a matrix and a vector) whose order is denoted by the number of edges.
The number by an edge speciﬁes the size of each mode (or dimension).
The connection between two nodes denotes a multilinear product operator between two tensors on a speciﬁc mode.
This is also called tensor contraction , which corresponds to the summation over the indices of that mode.
It should be noted that Z d is connected to Z 1 by the summation over the index
α 1 , which is equivalent to the trace operation.
For simplicity, we denote TR decomposition by
T = (cid:60) ( Z 1 , Z 2 , . . .
, Z d ) .
Theorem 1 (Circular dimensional permutation invariance) .
Let T ∈ R n 1 × n 2 × ...
× n d be a d thorder tensor and its TR decomposition is given by T = (cid:60) ( Z 1 , Z 2 , . . .
, Z d ) .
If we deﬁne ←− T k ∈ R n k +1 ×···× n d × n 1 ×···× n k as the circularly shifted version along the dimensions of
T by k, then we have ←− T k = (cid:60) ( Z k +1 , . . .
, Z d , Z 1 , . . .
Z k ) .
A proof of Theorem 1 is provided in Appendix B.1.
It should be noted that circular dimensional permutation invariance is an essential feature that distinguishes TR decomposition from TT decomposition.
For TT decomposition, the product of matrices must keep a strictly sequential order, yielding that the tensor with a circular dimension shifting does not correspond to the shifting of tensor cores.
L EARNING A LGORITHMS 3. S EQUENTIAL SVD S
We propose the ﬁrst algorithm for computing the TR decomposition using d sequential SVDs.
This algorithm will be called the TR-SVD algorithm .
Theorem 2.
Let us assume T can be represented by a TR decomposition.
If the k -unfolding matrix
T (cid:104) k (cid:105) has Rank ( T (cid:104) k
(cid:105) )
=
R k
+1 , then there exists a TR decomposition with TR-ranks r which satisﬁes that ∃ k, r 1 r k +1 ≤ R k +1 .
Proof.
We can express TR decomposition in the form of k -unfolding matrix,
T
[Equation] [Equation]
It can also be rewritten as
[Equation]
where we deﬁned the subchain by merging multiple linked cores as Z <
k
( i 1 · · · i k − 1 ) =
(cid:81)
k − 1 j =
Z
j
( i j ) and Z >k
( i k +1 · · · i d ) = (cid:81)
dj =
k +
Z j ( i j ) .
Hence, we can obtain T (cid:104) k (cid:105) =
Z ≤ k (2) ( Z >k [2] ) T , where the subchain Z ≤ k (2) is of size (cid:81) kj =1 n j × r 1 r k +1 , and Z >k [2] is of size (cid:81)
dj =
k +1 n j × r 1 r k +1 .
Since the rank of T (cid:104) k
(cid:105) is R k
+1 , we can obtain r 1 r k +1 ≤
R k +1 .
According to (2) and (3), TR decomposition can be written as
[Equation]
Since the low-rank approximation of T (cid:104)
[Equation]
can be obtained by the truncated SVD, which is T (cid:104) 1 (cid:105) =
U Σ V T + E 1 , the ﬁrst core Z 1 ( i.e., Z ≤ 1 ) of size r 1 × n 1 ×
r 2 can be obtained by the proper reshaping and permutation of U and the subchain Z > 1 of size r 2 × (cid:81) dj
=2 n j × r 1 is obtained by the proper reshaping and permutation of ΣV T , which corresponds to the remaining d − 1 dimensions of T .
Note that this algorithm use the similar strategy with TT-SVD (Oseledets, 2011), but the reshaping and permutations are totally different between them.
Subsequently, we can further reshape the subchain Z > 1 as a matrix Z > 1 ∈ R r 2 n
[Equation]
dj =3 n j r 1 which thus can be written as
[Equation]
By applying truncated SVD, i.e., Z > 1 = U Σ V T + E 2 , we can obtain the second core Z 2 of size ( r 2 × n 2 × r 3 ) by appropriately reshaping U and the subchain Z > 2 by proper reshaping of ΣV T .
This procedure can be performed sequentially to obtain all d cores Z k , k = 1 , . . .
, d .
As proved in (Oseledets, 2011), the approximation error by using such sequential SVDs is given by
[Equation]
Hence, given a prescribed relative error (cid:15) p , the truncation threshold δ can be set to
[Equation]
p √ d − 1 (cid:107) T (cid:107) F .
However, considering that (cid:107) E 1 (cid:107) F corresponds to two ranks including both r 1 and r 2 , while (cid:107) E k (cid:107) F , ∀ k > 1 correspond to only one rank r k +1 .
Therefore, we modify the truncation threshold as
[Equation]
A pseudocode of the TR-SVD algorithm is summarized in Alg. 1.
Note that the cores obtained by the
TR-SVD algorithm are left-orthogonal, which is Z Tk (cid:104) 2 (cid:105)
Z k (cid:104) 2 (cid:105)
=
I for k = 2 , . . .
, d − 1 .
3. B LOCK -W ISE A LTERNATING L EAST -S QUARES (ALS)
The ALS algorithm has been widely applied to various tensor decomposition models such as CP and Tucker decompositions (Kolda & Bader, 2009; Holtz et al., 2012).
The main concept of ALS is optimizing one core while the other cores are ﬁxed, and this procedure will be repeated until some convergence criterion is satisﬁed.
Given a d th-order tensor T , our goal is optimize the error function as
[Equation]
According to the TR deﬁnition in (1)
[Equation]
we have
[Equation]
where Z (cid:54) =
k ( i
[Equation]
+1 · · · i d i 1 . . .
i k − 1 ) =
(cid:81)
d j = k +1 Z j
( i j ) (cid:81) k − 1 j =
Z
j
( i j ) denotes a slice matrix of subchain tensor by merging all cores except k th core Z k .
Hence
[Equation]
the mode- k unfolding matrix of T can be expressed by
[Equation]
By applying different mode- k unfolding operations, we can obtain that T [ k ] =
Z k (2) (cid:16) Z (cid:54) =
k [2] (cid:17) T , where
Z (cid:54) =
k is a subchain obtained by merging d − 1 cores.
The objective function in (5) can be optimized by solving d subproblems alternatively.
More speciﬁcally, having ﬁxed all but one core, the problem reduces to a linear least squares problem, which is
[Equation]
This optimization procedure must be repeated till the convergence, which is called TT-ALS.
Here, we propose a computationally efﬁcient block-wise ALS (BALS) algorithm by utilizing truncated
SVD, which facilitates the self-adaptation of ranks.
The main idea is to perform the blockwise optimization followed by the separation of a block into individual cores.
To achieve this, we consider merging two linked cores, e.g., Z k , Z k +1 , into a block (or subchain)
Z ( k,k +1)
[Equation]
R r k × n k n k +1 ×
r k +2 .
Thus, the subchain Z ( k,k +1) can be optimized while leaving all cores except Z k , Z k +1 ﬁxed.
Subsequently, the subchain Z ( k,k +1) can be reshaped into ˜ Z
( k,k +1) ∈ R r k n k × n k +1 r k +2 and separated into a left-orthonormal core Z k and Z k +1 by a truncated SVD:
[Equation]
where Z k (cid:104) 2 (cid:105) ∈ R r k n k × r k +1 is the 2 -unfolding matrix of core Z k , which can be set to U , while
Z k +1 (cid:104)
(cid:105) ∈ R
r k +1 × n k +1 r k +2 is the 1 -unfolding matrix of core Z k +1 , which can be set to ΣV T .
This procedure thus moves on to optimize the next block cores Z ( k +1 ,k +2) , . . .
, Z ( d − 1 ,d ) , Z ( d, 1) successively in the similar way.
Note that since the TR model is circular, the d th core can also be merged with the ﬁrst core yielding the block core Z ( d, 1) .
The key advantage of our BALS algorithm is the rank adaptation ability which can be achieved simply by separating the block core into two cores via truncated SVD, as shown in (6).
The truncated rank r k +1 can be chosen such that the approximation error is below a certain threshold.
One possible choice is to use the same threshold as in the TR-SVD algorithm, i.e., δ k described in (4).
However, the empirical experience shows that this threshold often leads to overﬁtting and the truncated rank is higher than the optimal rank.
This is because the updated block Z ( k,k +1) during ALS iterations is not a closed form solution and many iterations are necessary for convergence.
To relieve this problem, we choose the truncation threshold based on both the current and the desired approximation errors, which is
[Equation]
A pseudo code of the BALS algorithm is described in Alg. 2.
3. S TOCHASTIC G RADIENT D ESCENT
For large-scale dataset, the ALS algorithm is not scalable due to the cubic time complexity in the target rank, while Stochastic Gradient Descent (SGD) shows high efﬁciency and scalability for matrix/tensor factorization (Gemulla et al., 2011; Maehara et al., 2016; Wang & Anandkumar, 2016).
In this section, we present a scalable and efﬁcient TR decomposition by using SGD, which is also suitable for online learning and tensor completion problems.
To this end, we ﬁrst provide the element-wise loss function, which is
[Equation]
where λ k is the regularization parameters.
The core idea of SGD is to randomly select one sample
T ( i 1 , i 2 , . .
, i d ) , then update the corresponding slice matrices
Z k
( i k ) , k = 1 , . .
, d from each latent core tensor Z k based on the noisy gradient estimates by scaling up just one of local gradients, i.e.
∀ k = 1 , . . .
, d ,
[Equation]
We employ Adaptive Moment Estimation (Adam) method to compute adaptive learning rates for each parameter.
Thus, the update rule for each core tensor is given by
[Equation]
where M t = β 1 M t − 1 +(1 − β 1 ) ∂L
∂ Z tk ( i k ) denotes an exponentially decaying average of past gradients and V t = β 2 V t − 1 + (1 − β 2 )( ∂L ∂ Z tk
( i k ) ) 2 denotes exponentially decaying average of second moment of the gradients.
The SGD algorithm can be naturally applied to tensor completion problem, when the data points are sampled only from a sparse tensor.
Furthermore, this also naturally gives an online TR decomposition.
The batched versions, in which multiple local losses are averaged, are also feasible but often have inferior performance in practice.
For each element T ( i 1 , i 2 , . . .
, i d ) , the computational complexity of SGD is O ( d 2 r 3 ) .
If we deﬁne N = (cid:81) dk
=1 n k consecutive updates as one epoch of SGD, the computational complexity per SGD epoch is thus O ( Nd 2 r 3 ) , which linearly scales to data size.
As compared to ALS, which needs O ( Ndr 4 + dr 6 ) , it is more efﬁcient in terms of computational complexity for one epoch. The memory cost of TR-SVD, ALS, and SGD are O ( ndr 2 + (cid:81) k n k ) , O ( ndr 2 + r 2 (cid:81) k n k + r 4 ) , and O ( ndr 2 + r 2 ) , respectively.
Thus, TR-SGD requires much less memory.
The convergence condition of SGD algorithm follows other stochastic tensor decompositions (Ge et al., 2015; Maehara et al., 2016).
In practice, how to choose the TR algorithms depends on the task and data at hand.
TR-SVD is a non-iterative algorithm leading to the fast computation.
The expected approximation error is required from user to choose TR-ranks automatically.
However, the learned ranks is not necessary to be the optimal in terms of compression rate.
TR-ALS is an iterative algorithm in which the ﬁxed TR-ranks are required to be given by user. The computation time is highly related to TR-ranks, which is very efﬁcient only for small TR-ranks. TR-BALS is an iterative algorithm that can choose TR-ranks in an adaptive manner given the expected approximation error. TR-SGD is an iterative algorithm that is suitable for a large-scale tensor with given TR-ranks.
In particular, the memory cost of TR-SGD depends on only the size of core tensors rather than the size of data tensor.
By assuming that tensor data have been already represented as TR decompositions, i.e., a sequence of third-order cores, we justify and demonstrate that the basic operations on tensors, such as the addition , multilinear product , Hadamard product , inner product and Frobenius norm , can be performed efﬁciently by the appropriate operations on each individual cores.
We have the following theorems:
Property 1.
Let T 1 and T 2 be d th-order tensors of size n 1 ×· · ·× n d .
If TR decompositions of these two tensors are
T 1 = (cid:60) ( Z 1 , . .
, Z d ) where Z k ∈ R r k × n k × r k +1 and T 2 = (cid:60) ( Y 1 , . .
, Y d ) where
Y k ∈ R s k × n k × s k +1 , then the addition of these two tensors, T
= T 1 + T 2 , can also be represented in the TR format given by T 3 = (cid:60) ( X 1 , . . .
, X d ) , where X k ∈ R q k × n k × q k +1 and q k = r k
+ s k .
Each core
X k can be computed by
[Equation]
A proof of Property 1 is provided in Appendix B.2.
Note that the sizes of new cores are increased and not optimal in general.
This problem can be solved by the rounding procedure (Oseledets, 2011).
Property 2.
Let T ∈ R n 1 ×···× n d be a d th-order tensor whose TR representation is T = (cid:60) ( Z 1 , . .
, Z d ) and u k ∈ R n k , k = 1 , . . .
, d be a set of vectors, then the multilinear products, denoted by c =
T × 1 u T 1 × 2 · · · × d u Td , can be computed by the multilinear product on each cores, which is
[Equation]
[Equation] X [Equation] where X k =
[Equation]
A proof of Property 2 is provided in Appendix B.3.
It should be noted that the computational complexity in the original tensor form is O ( dn d ) , while it reduces to O ( dnr 2 + dr 3 ) that is linear to tensor order d by using TR representation.
Property 3.
Let T 1 and T 2 be d th-order tensors of size n 1 × · · · × n d .
If the TR decompositions of these two tensors are
T 1 = (cid:60) ( Z 1 , . .
, Z d ) where Z k ∈ R r k × n k × r k +1 and T 2 = (cid:60) ( Y 1 , . .
, Y d ) where
Y k ∈ R s k × n k × s k +1 , then the Hadamard product of these two tensors, T 3 =
T 1 (cid:126) T 2 , can also be represented in the TR format given by T 3 = (cid:60) ( X 1 , . . .
, X d ) , where X k ∈ R q k × n k × q k +1 and q k =
r k s k .
Each core X k can be computed by
[Equation]
The inner product of two tensors can be computed by TR format (cid:104) [Equation] = (cid:60) ( V 1 , . . .
, V d ) where V k = (cid:80) n k i k
=1 Z k ( i k ) ⊗ Y k
( i k ) .
Therefore, the Frobenius norm can be also computed by TR format using (cid:107) T (cid:107) F = (cid:112) (cid:104) T , T (cid:105) . A proof of Property 3 is provided in Appendix B.4. In contrast to O ( n d ) in the original tensor form, the computational complexity is equal to O ( dnq 2 + dq 3 ) that is linear to d by using TR representation.
E XPERIMENTAL R ESULTS 5. N UMERICAL I LLUSTRATION
We consider highly oscillating functions that can be approximated perfectly by a low-rank TT format (Khoromskij, 2015), as shown in Fig.
We ﬁrstly tensorize the functional vector resulting in a d th-order tensor of size n 1 × n 2 × · · · × n d , where isometric size is usually preferred, i [Table] e., n 1 = n 2 = · · · = n d = n , with the total number of elements denoted by N = n d .
The error bound (tolerance), denoted by (cid:15) p = 10 − 3 , is given as the stopping criterion for all compared algorithms.
As shown in Table [Table] TR-SVD and TR-BALS can obtain comparable results with TT-SVD in terms of compression ability.
However, when noise is involved, TR model signiﬁcantly outperforms TT model, indicating its more robustness to noises.
[Table]
It should be noted that TT representation has the property that r 1 = r d +1 = 1 and r k , k = 2 , . . .
, d −are bounded by the rank of k -unfolding matrix of T (cid:104) k (cid:105) , which limits its generalization ability and consistency when the tensor modes have been shifted or permuted. To demonstrate this, we consider shifting the dimensions of T of size n 1 × · · · × n d by k times leading to ←− T k of size n k +1 × · · · × n d × n 1 × · · · × n k . As shown in Table 2, the average TT-ranks are varied dramatically along with the different shifts. In particular, when k = 8 , ¯ r tt becomes 14.6, resulting in a large number of parameters
N p = 10376 . In contrast to TT, TR can obtain consistent and compact representation with N p = 1360 . We also tested TR-ALS and TR-SGD algorithms on datasets which are generated by a TR model, in which the core tensors are randomly drawn from N (0 , 1) .
As shown in Table 3, TR-SGD can achieve similar performance as TR-ALS in all cases.
In particular, when data is relatively large-scale ( 10 8 ),
TR-SGD can achieve relative error (cid:15)
= 0 .
by using 1% of data points only once.
[Table]
[Table]
5. I MAGE R EPRESENTATION BY H IGHER - ORDER T ENSOR D ECOMPOSITIONS
An image is naturally represented by a 2D matrix, on which SVD can provide the best low-rank approximation.
However, the intrinsic structure and high-order correlations within the image is not well exploited by SVD. In this section
[Table] we show the tensorization of an image [Table] yielding a higher-order tensor
[Table]
and TR decomposition enable us to represent the image more efﬁciently than SVD.
Given an image (e [Table] g. ‘Peppers’) denoted by Y of size I × J , we can reshape it as I 1 × I 2 × . . .
×
I d
× J 1 × J 2 × . . .
× J d followed by an appropriate permutation to I 1 × J 1 ×
I
× J 2 . . .
×
I d × J d and thus reshape it again to I 1 J×
I 2 J 2 × . . .
× I d J d , which is a d th-order tensor.
The ﬁrst mode corresponds to small-scale patches of size I× J 1 , while the d th-mode corresponds to large-scale partition of whole image as I d × J d .
Based on this tensorization operations, TR decomposition is able to capture the intrinsic structure information and provides a more compact representation.
As shown in Table 4, for 2D matrix case, SVD, TT and TR give exactly same results.
In contrast, for 4th-order tensorization cases, TT needs only half number of parameters (2 times compression rate) while TR achieves 3 times compression rate, given the same approximation error 0 .
It should be noted that TR representation provides signiﬁcantly high compression ability as compared to TT.
In addition, Fig. 4 shows TR-SGD results on ‘Lena’ image by sampling different fraction of data points.
5. CIFAR-
The CIFAR-10 dataset consists of 60000 32 × 32 colour images.
We randomly pick up 1000 images for testing of TR decomposition algorithms.
As shown in Fig. 5, TR model outperforms TT model in terms of compression rate given the same approximation error, which is caused by strict limitation that the mode-1 rank must be 1 for TT model.
In addition, TR is a more generalized model, which contains TT as a special case, thus yielding better low-rank approximation.
Moreover, all other TR algorithms can also achieve similar results.
The detailed results for (cid:15) = 1 e − 1 are shown in
Table 5. Note that TR-SGD can achieve the same performance as TR-ALS, which demonstrates its effectiveness on real-world dataset.
Due to high computational efﬁciency of TR-SGD per epoch, it [Figure] Iter: 50% indicates that only 50% elements are sampled for learning its TR representation. [Table]
[Table]
TT
TR TT [Table] [Figure]
of compression rate and approximation error (RSE)
on CIFAR-10 dataset by using TR and TT models.
can be potentially applied to very large-scale dataset.
For visualization, TR-SGD results after 10 and 100 epoch are shown in Fig. 5. T ENSORIZING N EURAL N ETWORKS USING TR REPRESENTATION TT representations have been successfully applied to deep neural networks (Novikov et al., 2015), which can signiﬁcantly reduce the number of model parameters and improve computational efﬁciency. To investigate the properties of TR representation, we applied TR framework to approximate the weight matrix of a fully-connected layer and compared with TT representation.
We run the experiment on the MNIST dataset for the task of handwritten-digit recognition.
The same setting of neural network (two fully-connected layers with ReLU activation function) as in (Novikov et al., 2015) was applied for comparisons.
The input layer is tensorized to a 4th-order tensor of size 4 × 8 × 8 × 4 , the weight matrix of size 1024 × 625 is represented by a TR format of size 4 × 8 × 8 × 4 × 5 × 5 × 5 × 5 .
Through deriving the gradients over each core tensor, all computations can be performed on small
[Figure]
[Table]
Ranks
[Table]
T r a i n i ng e rr o r ( % )
TT
[Table]
layer
TR-layer
[Table]
Ranks
[Table]
T e s t
i ng e rr o r ( % )
TT
[Table]
TR
[Table]
layer core tensors instead of the dense weight matrix by using properties in Sec. 4, yielding the signiﬁcant improvements of computational efﬁciency.
The experimental results are shown in Fig.
We observe that the TR-layer provides much better ﬂexibility than TT-layer, leading to much lower training and testing errors under the same compression level (i.e., TT/TR ranks).
In addition, TR can achieve much better compression rate under the same level of test error.
When r 1 = . . .
= r 4 = 2 , the compression rate of dense weight matrix is up to 1300 times.
We tested the tensorizing neural networks with the same architecture on SVHN dataset (http://uﬂdl.stanford.edu/housenumbers/).
By setting all the TT-ranks in the network to 4, we achieved the test error of 0.13 with compression rate of 444 times, while we can achieve the same test error by setting all the TR-ranks to 3 with compression rate of 592 times.
We can conclude that the
TR representation can obtain signiﬁcantly higher compression rate under the same level of test error.
C ONCLUSION
We have proposed a novel tensor decomposition model, which provides an efﬁcient representation for a very high-order tensor by a sequence of low-dimensional cores.
The number of parameters in our model scales only linearly to the tensor order.
To optimize the latent cores, we have presented several different algorithms: TR-SVD is a non-recursive algorithm that is stable and efﬁcient, while
TR-BALS can learn a more compact representation with adaptive TR-ranks, TR-SGD is a scalable algorithm which can be also used for tensor completion and online learning.
Furthermore, we have investigated the properties on how the basic multilinear algebra can be performed efﬁciently by operations over TR representations (i.e., cores), which provides a powerful framework for processing large-scale data.
The experimental results veriﬁed the effectiveness of our proposed algorithms.
A R ELATION TO O THER M ODELS
In this section, we discuss the relations between TR model and the classical tensor decompositions including CPD, Tucker and TT models.
All these tensor decompositions can be viewed as the transformed representation of a given tensor.
The number of parameters in CPD is O ( dnr ) that is linear to tensor order, however, its optimization problem is difﬁcult and convergence is slow.
The
Tucker model is stable and can approximate an arbitrary tensor as close as possible, however, its number of parameters is O ( dnr + r d ) that is exponential to tensor order.
In contrast, TT and TR decompositions have similar representation power to Tucker model, while their number of paramters is O ( dnr 2 ) that is linear to tensor order.
A. CP DECOMPOSITION
The cannonical polyadic decomposition (CPD) aims to represent a d th-order tensor T by a sum of rank-one tensors, given by
[Equation]
(13) where each rank-one tensor is represented by an outer product of d vectors.
It can be also written in the element-wise form given by
[Equation]
(14) where (cid:104)· , . . .
, ·(cid:105) denotes an inner product of a set of vectors, i.e., u ( k ) i k ∈ R
r , k = 1 , . . .
, d .
By deﬁning V k
( i k ) =
diag ( u ( k ) i k ) which is a diagonal matrix for each ﬁxed i k and k , where k = 1 , . . .
, d , i k
= 1 , . .
. , n k , we can rewrite (14) as
[Equation]
Hence, CPD can be viewed as a special case of TR decomposition T = (cid:60) ( V 1 , . .
, V d ) where the cores
V k , k = 1 , . . .
, d are of size r × n k × r and each lateral slice matrix V k
( i k ) is a diagonal matrix of size r × r .
A. T UCKER DECOMPOSITION
The Tucker decomposition aims to represent a d th-order tensor T by a multilinear product between a core tensor G ∈ R r 1 ×···× r d and factor matrices U ( k ) ∈ R n k × r k , k = 1 , . . .
, d , which is expressed by
[Equation]
By assuming the core tensor G can be represented by a TR decomposition G = (cid:60) ( V 1 , . . .
, V d ) , the Tucker decomposition (16) in the element-wise form can be rewritten as
[Equation]
where the second step is derived by applying Theorem 2.
Hence, Tucker model can be represented as a TR decomposition T =
(cid:60) ( Z 1 , . . .
, Z d ) where the cores are computed by the multilinear products between TR cores representing
G and the factor matrices, respectively, which is
[Equation]
, d.
A. TT DECOMPOSITION
The tensor train decomposition aims to represent a d th-order tensor T by a sequence of cores
G k , k = 1 , . . .
, d , where the ﬁrst core G 1 ∈ R n 1 × r 2 and the last core G d ∈ R r d × n d are matrices while the other cores G k ∈ R r k × n k × r k
+1 , k = 2 , . . .
, d − 1 are 3rd-order tensors.
Speciﬁcally, TT decomposition in the element-wise form is expressed as
[Equation]
where g 1 ( i 1 ) is the i 1 th row vector of G 1 , g d ( i d ) is the i d th column vector of G d , and G k ( i k ) , k = 2 , . . .
, d − 1 are the i k th lateral slice matrices of G k .
According to the deﬁnition of TR decomposition in (1), it is obvious that TT decomposition is a special case of TR decomposition where the ﬁrst and the last cores are matrices, i.e., r 1 = r d +1 = 1 .
On the other hand, TR decomposition can be also rewritten as
[Equation]
where z 1 ( α 1 , i 1 , :) ∈
R r 2 is the α 1 th row vector of the matrix Z 1 ( i 1 ) and z d (: , i d , α 1 ) is the α 1 th column vector of the matrix Z d ( i d ) .
Therefore, TR decomposition can be interpreted as a sum of
TT representations.
The number of TT representations is r 1 and these TT representations have the common cores Z k , for k = 2 , . . .
, d − 1 .
In general, TR outperforms TT in terms of representation power due to the fact of linear combinations of a group of TT representations.
Furthermore, given a speciﬁc approximation level, TR representation requires smaller ranks than TT representation.
B P ROOFS B. P ROOF OF T HEOREM
Proof.
It is obvious that (1) can be rewritten as
[Equation]
Therefore, we have ←− T k = (cid:60) ( Z k +1 , . . .
, Z d , Z 1 , . . .
, Z k ) .
B. P ROOF OF P ROPERTY
Proof.
According to the deﬁnition of TR decomposition, and the cores shown in (10), the ( i 1 , . . .
, i d ) th element of tensor T 3 can be written as
T 3 ( i 1 , . .
, i d ) =
Tr
[Equation]
(cid:81)
[Equation] [Equation]
) (cid:19)
[Equation]
Tr (cid:32) d (cid:89) k
[Equation]
Z k ( i k [Equation] (cid:33) + Tr (cid:32) d (cid:89) k
=
Y k ( i k ) (cid:33) .
Hence, the addition of tensors in the TR format can be performed by merging of their cores.
B. P ROOF OF P ROPERTY
Proof.
The multilinear product between a tensor and vectors can be expressed by
[Equation]
Thus, it can be written as a TR decomposition shown in (11) where each core X k ∈ R r k × r k + becomes a matrix.
The computational complexity is equal to O ( dnr 2 ) .
P ROOF OF P ROPERTY
Proof.
Each element in tensor T 3 can be written as
T 3 ( i 1 , . .
, i d ) =
Tr (cid:32) d (cid:89) k
=
Z k ( i k ) (cid:33)
Tr (cid:32) d (cid:89) k
=
Y k ( i k ) (cid:33) =
Tr (cid:40)(cid:32) d (cid:89) k
=
Z k ( i k ) (cid:33) ⊗ (cid:32) d (cid:89) k
=
Y k ( i k ) (cid:33)(cid:41) =
Tr (cid:40) d (cid:89) k
= (cid:16)
[Equation]
( i k ) (cid:17)(cid:41) .
Hence, T 3 can be also represented as TR format with its cores computed by (12), which costs
O ( dnq 2 ) .
Furthermore, one can compute the inner product of two tensors in TR representations.
For two tensors T 1 and T 2 , it is deﬁned as (cid:104) T 1 , T 2 (cid:105) =
(cid:80)
i 1 ,...,i d T 3 ( i 1 , . .
, i d ) , where T 3 = T 1 (cid:126) T 2 . Thus, the inner product can be computed by applying the Hadamard product and then computing the multilinear product between T 3 and vectors of all ones, i.e., T 3 × 1 u T 1 × 2 · · · × d u Td , where u k = 1 , k = 1 , . . . , d , which can be computed efﬁciently by using Property 2. C A LGORITHMS
Pseudo codes of TR-SVD and TR-BALS are provided in Alg.
1 and Alg. 2, respectively.
D A DDITIONAL E XPERIMENTAL R ESULTS D. COIL-100 DATASET
The Columbia Object Image Libraries (COIL)-100 dataset (Nayar et al., 1996) contains 7200 color images of 100 objects (72 images per object) with different reﬂectance and complex geometric characteristics.
Each image can be represented by a 3rd-order tensor of size 128 × 128 × 3 and then is downsampled to 32 × 32 × 3 .
Hence, the dataset can be ﬁnally organized as a 4th-order tensor of size 32 × 32 × 3 × 7200 .
The number of features is determined by r 4 × r 1 , while the ﬂexibility of subspace bases is determined by r 2 , r 3 .
Subsequently, we apply the K-nearest neighbor (KNN) classiﬁer with K=1 for classiﬁcation.
For detailed comparisons, we randomly select a certain ratio ρ = 50% or ρ = 10% samples as the training set and the rest as the test set.
The classiﬁcation performance is averaged over 10 times of random splitting.
In Table 6, r max of TR decompositions is much smaller than that of TT-SVD.
It should be noted that TR representation, as compared to TT, can obtain more compact and discriminant representations.
Fig.
8 shows the reconstructed images under different approximation levels.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Reshape the block core by ˜ Z
( k,k +
[Table]
reshape ( Z ( k
[Table]
k +1) , [ r k n k × n k +1 r k +2 ]) .
[Table]
by
[Table]
˜ Z ( k
[Table]
k
[Table]
) = UΣV T .
[Table] [Table]
[Figure]
D. KTH VIDEO DATASET
We test the TR representation for KTH video database (Laptev & Lindeberg, 2006) containing six types of human actions (walking, jogging, running, boxing, hand waving and hand clapping) performed several times by 25 subjects in four different scenarios: outdoors, outdoors with scale variation, outdoors with different clothes and indoors as illustrated in Fig. 9.
There are 600 video sequences for each combination of 25 subjects, 6 actions and 4 scenarios.
Each video sequence was downsampled to 20 × 20 × 32 .
Finally, we can organize the dataset as a tensor of size 20 × 20 × 32 × 600 .
For extensive comparisons, we choose different error bound (cid:15) p ∈ { 0 . 2 , 0 . 3 , 0 . 4 } .
In Table 7, we can see that TR representations achieve better compression ratio reﬂected by smaller r max , ¯ r than that of TT-SVD, while TT-SVD achieves better compression ratio than CP-ALS.
For instance, when [Table] (cid:15) ≈ 0 . 2 , CP-ALS requires r max = 300 , ¯ r = 300 ; TT-SVD requires r max = 139 , ¯ r = 78 , while
TR-SVD only requires r max = 99 , ¯ r = 34 .
For classiﬁcation performance, we observe that the best accuracy ( 5 × 5 -fold cross validation) achieved by CP-ALS, TT-SVD, TR-SVD are 80.8%, 84.8%, 87.7%, respectively.
Note that these classiﬁcation performances might not be the state-of-the-art on this dataset, we mainly focus on the comparisons of representation ability among CP, TT, and
TR decomposition frameworks.
To obtain the best performance, we may apply the powerful feature extraction methods to TT or TR representations of dataset.
It should be noted that TR decompositions achieve the best classiﬁcation accuracy when (cid:15) = 0 .
29 , while TT-SVD and CP-ALS achieve their best classiﬁcation accuracy when (cid:15)
= 0 .
This indicates that TR decomposition can preserve more discriminant information even when the approximation error is relatively high.
This experiment demonstrates that TR decompositions are effective for unsupervised feature representation due to their ﬂexibility of TR-ranks and high compression ability.
[Table]