Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition
Abstract
Recurrent Neural Networks (RNNs) and their variants, such as Long-Short Term
Memory (LSTM) networks, and Gated Recurrent Unit (GRU) networks, have achieved promising performance in sequential data modeling.
The hidden layers in
RNNs can be regarded as the memory units, which are helpful in storing information in sequential contexts.
However, when dealing with high dimensional input data, such as video and text, the input-to-hidden linear transformation in RNNs brings high memory usage and huge computational cost.
This makes the training of
RNNs unscalable and difﬁcult.
To address this challenge, we propose a novel compact LSTM model, named as TR-LSTM, by utilizing the low-rank tensor ring decomposition (TRD) to reformulate the input-to-hidden transformation.
Compared with other tensor decomposition methods, TR-LSTM is more stable.
In addition,
TR-LSTM can complete an end-to-end training and also provide a fundamental building block for RNNs in handling large input data.
Experiments on real-world action recognition datasets have demonstrated the promising performance of the proposed TR-LSTM compared with the tensor train LSTM and other state-of-theart competitors.
Introduction
Recurrent Neural Networks (RNNs) have achieved great success in analyzing sequential data in various applications, such as computer vision [1, 13, 19], natural language processing, etc..
Despite the success, LSTMs and GRUs suffer from the huge number of parameters, which makes the training process notoriously difﬁcult and easily over-ﬁtting.
In particular, in the task of action recognition from videos, a video frame usually forms a high-dimensional input, which makes the size of the input-to-hidden matrix extremely large.
A promising direction to reduce the parameter size is to explore the low-rank structures in the weight matrices.
Inspired from the success of tensor decomposition methods in CNNs [17, 12], various tensor decomposition methods have been explored in RNNs [24, 25].
And we propose to use the tensor ring decomposition (TRD)
[26] to extract the low-rank structure of the input-to-hidden matrix in RNNs(the details of the TRD are in Appendix 5.1).
For illustration, we implement the tensor ring layer on LSTM, replacing the over-parametric input to hidden matrices, named TR-LSTM.
And, the tensor ring layer can also be plugged in the vanilla RNN and GRU.
O. . .
I I
I n
O m
O 1 ( )  2 ( )  n ( )  n + 1 ( )  n + 2 ( )  n + m ( )
O. . .
  input output weights y x
I
Model
The core conception of our model is elaborated in this section.
By transforming the input-to-hidden weight matrices into TR form, and applying them into LSTM, we get our TR-LSTM model.
2.1 Tensor Ring Layer (TRL)
After reshaping the input vector x and the weight matrix W into tensor(introduced in Appendix 5.2.1), and decomposing weight tensor into TR representation(described in Appendix 5.2.2), we can get the output tensor YYY by manipulating WWW and XXX .
After reshaping the output tensor, the ﬁnal output vector y can be obtained.
Because the weight matrix is factorized with TRD, we denote the whole calculation from input to output vector as tensor ring layer (TRL):
[Equation]
which is illustrated in Figure 1.The complexity analysis are shown in Appendix 5.2.3.
2.2 TR-LSTM
We can get our TR-LSTM model by applying TRL to LSTM, which is the state-of-the-art variant of
RNN.
[Equation]
where (cid:12) , σ ( · ) and tanh( · ) denote the element-wise product, sigmoid function and hyperbolic function, respectively.
The weight matrices W ∗ x denote the mapping from the input to hidden matrix, for the input gate k t , forget gate f t , output gate o t and cell update vector c t , respectively.
U ∗ h is deﬁned similarly for the hidden state h t − 1 .
Experiments 3.1 Experiments on the UCF11 Dataset
We conduct two experiments described as “End-to-End Training” and “Pre-train with CNN” on the
UCF11 Dataset, where “End-to-End Training” means that video frames are directly fed into the
TR-LSTM and “Pre-train with CNN” means the inputs of the TR-LSTM are extracted features via a pre-trained CNN.
Details of this experiments can be found in Appendix 5.3.
[Table] End-to-End Training We compare our model with BT-LSTM [25] and TT-LSTM [24], while using a standard LSTM as a baseline.
The accuracy of BT-LSTM is 0.856 which is much higher than
TT-LSTM with 0.803 while the LSTM only gain an accuracy of 0.681.
In our TR-LSTM, the shape of input tensor is 4 × 2 × 5 × 8 × 6 × 5 × 3 × 2 , the output tensor’s shape is set as 4 × 4 × 2 × 4 ×and all the TR-ranks are set as 5 except R
= R d = 10 .
Results are compared in Table 1.
We gain the top accuracy 0.869 with the least parameters of 1725, showing the outstanding performance of our model in this experiment.
[Table]
[Table]
Pre-train with CNN We set the size of the hidden layer as 32 × 64 = 2048 , which is consistent with the size of the output via Inception-V3.
After using the extracted feature as the inputs of LSTM, the accuracy of the vanilla LSTM attains 0.923.
At the same time, the accuracy of our TR-LSTM model whose ranks are set as 40 × 60 × 48 × 48 achieves 93.8 with a compression ratio of 25.
We compare some state-of-the-art methods in Table 2 on UCF11.
3.2 Experiments on the HMDB51 Dataset
In this experiment, we still use extracted features via Inception-V3 as the input vector and reshape it into 64 × 32 .
We sample 12 frames from each video clip randomly and be processed through the
CNN as the input data.
The shape of hidden layer tensor is set as 32 × 64 = 2048 .
The ranks of our
TR-LSTM are 40 × 60 × 48 × 48 .
Some of the state-of-the-art models like I3D [3] are presented in
Table 3.
With a compressing ratio of 25, the TR-LSTM model still gains a higher accuracy of 63.8% than the standard LSTM.
Conclusion
In this paper, we applied TRD to plain RNNs to replace the over-parametric input-to-hidden weight matrix when dealing with high-dimensional input data.
The low-rank structure of TRD can capture the correlation between feature dimensions with fewer orders of magnitude parameters.
Our TR-LSTM model achieved best compression ratio with the highest classiﬁcation accuracy on UCF11 dataset among other end-to-end training RNNs based on low-rank methods.
We believe that our models provide fundamental modules for RNNs, and can be widely used to handle large input data.
In future work, since our models are easy to be extended, we want to apply our models to more advanced RNN structures [8] to get better performance.
Appendix 5.1 Preliminaries and Background 5.1.1 Notation
In this paper, a d -order tensor, e.g., DDD ∈ R L 1 × L 2 ···× L d is denoted by a boldface Euler script letter.
With all subscripts ﬁxed, each element of a tensor is expressed as: DDD l 1 ,l 2 ,...l d ∈ R .
Given a subset of subscripts, we can get a sub-tensor.
For example, given a subset { L 1 = l 1 , L 2 = l 2 } , we can obtain a sub-tensor DDD l 1 ,l 2 ∈ R L 3 ···× L d .
Speciﬁcally, we denote a vector by a bold lowercase letter, e.g., v ∈ R L , and matrices by bold uppercase letters, e.g., M ∈ R L 1 × L 2 .
We regard vectors and matrices as 1-order tensors and 2-order tensors, respectively.
Figure 2 draws the tensor diagrams presenting the graphical notations and the essential operations.
[Figure]
5.1.2 Tensor Contraction
Tensor contraction can be performed between two tensors if some of their dimensions are matched.
For example, given two 3-order tensors AAA ∈ R L 1 × L 2 × L 3 and BBB ∈ R J 1 × J 2 × J 3 , when L 3 = J 1 , the contraction between these two tensors result in a tensor with the size of L 1 × L 2 × J 2 × J 3 , where the matching dimension is reduced, as shown in Equation (3):
[Equation]
5.1.3 Tensor Train Decomposition
Through the tensor train decomposition (TTD), a high-order tensor can be decomposed as the product of a sequence of low-order tensors.
For example, a d -order tensor DDD ∈ R L 1 × L 2 ···× L d can be decomposed as follows:
[Equation]
where each GGG ( k ) ∈ R R k − 1 × L k × R k is called a core tensor.
The tensor train rank, shorted as TT-rank, [ R 0 , R 1 , R 2 , . . .
, R d ] , for each k ∈ { 0 , 1 , . . .
, d } , 0 < r k ≤ R k , corresponds to the complexity of tensor train decomposition.
Generally, in tensor train decomposition, the constraint R 0 = R d =should be satisﬁed and other ranks are chosen manually.
Figure 3 illustrates the form of tensor train decomposition.
5.1.4 Tensor Ring Decomposition
The main drawback in tensor train decomposition is its constrained in ranks’ setting, hindering the representation ability and the ﬂexibility of the TT-based models.
At the same time, a strict order must
[Figure]
[Equation]
[Figure]
be followed when multiplying TT cores, so that the alignment of the tensor dimensions is extremely important in obtaining the optimized TT cores
[Figure]
but it is still a challenging issue in ﬁnding the best alignment[26].
In the tensor ring decomposition(TRD), an important modiﬁcation is interconnecting the ﬁrst and the last core tensors circularly and constructing a ring-like structure to alleviate the aforementioned limitations of the tensor train.
Formally, we set R
= R d = R and R ≥ 1 , and conduct the decomposition as:
[Equation]
For a d -order tensor, by ﬁxing the index k where k ∈ { 1 , 2 , . . .
, R } , the ﬁrst order of the beginning core tensor GGG (1) r
= k and the last order of the ending core tensor GGG ( d )
r d = k , can be reduced to matrices.
Thus, along each of the R slices of G G G (1) , we can separate the tensor ring structure as a summation of
R of tensor trains.
For example, by ﬁxing r 0 = r
d = k , the product of GGG (1) k,l 1 GGG (2) l 2 . . .
GGG ( d ) l d ,k has the form tensor train decomposition.
Therefore, the tensor ring model is essentially the linear combination of
R different tensor train models.
Figure 4 demonstrates the tensor ring structure, and the alternative interpretation as a summation of multiple tensor train structures.
5.2 TR-RNN model 5.2.1 Tensorizing x , y and W
Without loss of generality
[Equation]
we tensorize the input vector x , output vector y [Equation] and weight matrix W into tensors XXX , YYY
[Equation]
and WWW , shown in Equation (6):
[Equation]
where
[Equation]
5.2.2 Decomposing W
For an n -order input and m -order output
[Equation]
we decompose the weight tensor into the form of TRD with n + m core tensors multiplied one by one, each of which is corresponding to an input dimension or an output dimension, referring to Equation (7).
Without loss of generality, the core tensors corresponding to the input dimensions and output dimensions are grouped respectively, as shown in Figure
[Equation]
The tensor contraction from input to hidden layer in TR form is shown in Equation (8).
[Equation]
Compared with the redundant input-to-hidden weight matrix, the compression ratio in TR form is shown in Equation (9).
[Equation]
5.2.3 Complexity Analysis
Since the weight tensor has been decomposed into the form of TRD with n + m core tensors, the order of multiplication among input tensor and core tensors in Equation (8) determines the computational cost.
In our implementation, we merge the input tensor with input core tensors and output core tensors sequentially.
We can rewrite the Equation (8) as:
[Equation]
In Equation (10), the symbols × ab and × a,b,c...α,β,θ... mean the Tensor Contraction Operation described in [5], a fundamental and the most important operation in tensor networks, and can be considered as a higher-dimensional analogue of matrix multiplication, inner product, and outer product. {+Note the N ’th component can be formulated as × 1 ,n − N +3 2 , GGG ( N ) , N ∈ { 2 , 3 , . . . , n } and the ( n + M ) ’ th component can be formulated as × M +
+}
[Equation]
Our model is trained via back propagation.
In Equation (10), according to the left-to-right multiplication order, our forward computational complexity could reach to OOO ( nIR 3 + mOR 3 ) while the forward space complexity is OOO ( IR 2 ) , where all the ranks in our model are set in the same R . Comparison with some other compressing methods is shown in Table 4. [Table]
5.3 Details on the UCF11 Dataset
Through the real-world action recognition datasets UCF11(YouTube action dataset) [15], we evaluate our model from two settings: (1) end-to-end training, where video frames are directly fed into the
TR-LSTM; and (2) pre-training to obtain features prior to LSTMs, where a pre-trained CNN was used to extract meaningful low-dimensional features and then forwarded these features to the TR-LSTM.
For a fair comparison, we ﬁrst compare our proposed method with the standard LSTM and previous low-rank decomposition methods, and then with the state-of-the-art action recognition methods.
The UCF11 dataset contains 1600 video clips of a resolution 320 × 240 divided into 11 action categories (e.g., basketball shooting, biking/cycling, diving, etc.).
Each category consist of 25 groups of video, within more than 4 clips in one group.
It is a challenging dataset due to large variations in camera motion, object appearance and pose, object scale, cluttered background, and so on.
End-to-End Training Recent years, some tensor decomposition models are proposed to classify videos like TT-LSTM [24], BT-LSTM [25] and others.
For the reason that they use the end-to-end model for training, we set this experiment to compare with them.
In this experiments, we scale down the original resolution to 160 × 120 , and sample 6 frames from each video clip randomly as the input data.
Since every frame is RGB, the input data vector at each step is 160 × 120 × 3 = 57600 , and there are 6 steps in every sample.
We set the hidden layer as 256.
So there should be a fully-connected layer of 4 × 57600 ×
256 = 58982400 parameters to achieve the mapping for the standard LSTM.
In the experiment, the hyper-parameters in TT-LSTM and TR-LSTM models are set equally in their papers.
Pre-train with CNN Recently, some methods based on RNNs achieved higher accuracy by using the extracted feature as input vectors in computer vision [6].
Compared with using frames as input data, extracted features are more compact.
But there is still some room for improving the ability of the models.
The over-parametric problem is just partial solved.
To get better performance, we use extracted features via the CNN model Inception-V3 as input data to LSTM.