Three Dimensional Convolutional Neural Network Pruning with Regularization-Based Method
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In recent years, three-dimensional convolutional neural network (3D CNN) isintensively applied in video analysis and receives good performance.
However, 3D
CNN leads to massive computation and storage consumption, which hinders itsdeployment on mobile and embedded devices.
In this paper, we propose a threedimensional regularization-based pruning method to assign different regularizationparameters to different weight groups based on their importance to the network.
Experiments show that the proposed method outperforms other popular methods inthis area.
Introduction
In recent years, convolutional neural network (CNN) has developed rapidly and has achieved remarkable success in computer vision tasks such as identiﬁcation, classiﬁcation and segmentation.
However, due to the lack of motion modeling, this image-based end-to-end feature can not directlyapply to videos.
In [1, 2], the authors use three-dimensional convolutional networks (3D CNN) toidentify human actions in videos.
Tran et al.
proposed a 3D CNN for action recognition whichcontains 1.75 million parameters [3].
The development of 3D CNN also brings challenges because ofits higher dimensions.
This leads to massive computing and storage consumption, which hinders itsdeployment on mobile and embedded devices.
In order to reduce the computation cost, researchers propose methods to compress CNN models,including knowledge distillation [4], parameter quantization [5, 6], matrix decomposition [7] andparameter pruning [8].
However, all of the above methods are based on two-dimensional convolution.
In this paper, we expand the idea of [9] to 3D CNN acceleration.
The main idea is to add group regularization items to the objective function and prune weight groups gradually, where the regularizationparameters for different weight groups are differently assigned according to some importance criteria.
The Proposed Method
For a three-dimensional convolutional neural network with L layers, the weights of the l th ( 1 ≤ l ≤ L )convolutional layer W ( l ) ∈ R N l × C l × W l × H l × D l is a sequences of 5-D tensors.
Here N l , C l , W l ,
H l and D l are the dimensions along the axes of ﬁlter, channel, spatial height, spatial width and spatialdepth.
The proposed objective function for structured sparsity regularization is deﬁned by Eqn.(1).
[Equation]
Here L ( W ) is the loss on data; R ( W ) is the non-structured regularization ( L 2 norm in this paper).
R g is the structured sparsity regularization on each layer.
In [10, 11], the authors used the same λ g
[Figure]
for all groups and adopted Group LASSO for R g .
Recently Wang et al.
[9] use the squared L 1norm for R g and vary the regularization parameters λ g for different groups.
We build on top of thatapproach but extend it from two dimensions to three dimensions.
The structure learned is determined by the way of splitting groups of W ( l ) g .There are normallyﬁler-wise, channel-wise, shape-wise, and depth-wise structured sparsity with different ways ofgrouping [10].
Pruning of different weight groups for 3D CNN is shown in Fig.1.
In [9], Wang et al.
theoretically proved that by increasing the regularization parameter λ g , themagnitude of weights tends to be minimized.
The more λ g increases, the more magnitude of weightsare compressed to zero.
Therefore, we can assign different λ
g for the weight groups based on theirimportance to the network.
Here, we use the L 1 norm as a criterion of importance.
Our goal is to prune RN g weight groups in the network, where R is the pruning ratio to each layer and
N g is total number of weight groups in the layer.
In other words, we need to prune RN g weight groupswhich ranks lower in the network.
We sort the weight groups in ascending order of the L 1 norms.
Inorder to remove the oscillation of ranks during one training iteration, we averaged the rank throughtraining iterations to obtain the average rank r avg in N training iterations: r avg = 1 N (cid:80) Nn =1 r n .
The ﬁnal average rank r is obtained by sorting r avg of different weight groups in ascending order,making its range from 0 to N g − 1 .
The update of λ g is determined by the following formula:
[Equation]
Here ∆ λ g is the function of average rank r , we follow the formula proposed by Wang [9] as follows:
[Equation]
Here A is a hyperparameter which controls the speed of convergence.
According to Eqn.(2), we cansee that ∆ λ g is zero when r = RN g because we need to increase the regularization parameters ofthe weight groups whose ranks are below RN g to further decrease their L 1 norms; and for those withgreater L 1 norms and rank above RN g , we need to decrease their regularization parameters to furtherincrease their L 1 norms.
Thus, we can ensure that exactly RN g weight groups are pruned at the ﬁnalstage of the algorithm.
When we obtain λ ( new ) g , the weights can be updated through back-propagationdeduced from Eqn.(1).
Further details can be found in [9].
Experiments
Our experiments are carried out by Caffe[12].
We set the weight decay factor λ to be the same asthe baseline and set hyper-parameter A to half of λ .
We only compress the weights in convolutionallayers and leave the fully connected layers unchanged because we focus on network acceleration.
[Table]
[Table] [Table]
The pruning ratios of the convolutional layers are set to the same for convenience.
The methods usedfor comparison are Taylor Pruning (TP) [13] and Filter Pruning (FP) [14].
For all experiments, theratio of speedup is calculated by GFLOPS reduction.
3.1 C3D on UCF101
We apply the proposed method to C3D [3], which is composed of 8 convolution layers, 5 max-poolinglayers, and 2 fully connected layers.
We download the open Caffe model as our pre-trained model,whose accuracy on UCF101 dataset is 79.94%.
UCF101 contains 101 types of actions and a total of13320 videos with a resolution of 320 × 240 .
All videos are decoded into image ﬁles with 25 fps rate.
Frames are resized into 128 × 171 and randomly cropped to 112 × 112 .
Then frames are split intonon-overlapped 16-frame clips which are then used as input to the networks.
The results are shown in
Table 1.
With different speedup ratios, our approach is always better than TP and FP.
3.2 3D-ResNet18 on UCF
We further demonstrate our method on 3D-ResNet18 [3], which has 17 convolution layers and 1fully-connected layer.
The network is initially trained on the Sport-1M database.
We download themodel and then ﬁne-tune it by UCF101 for 30000 iterations, obtaining an accuracy of 72.50%.
Thevideo preprocessing method is the same as above.
The training settings are similar to that of C3D.
Experimental results are shown in Table 2.
Our approach only suffers 0.91% increased error whileachieving 2 × acceleration, obtaining better results than TP and FP.
Fig.2 shows the loss during thepruning process for different methods.
As the number of iterations increases, the losses of TP and
FP change dramatically, while the loss of our method remains at a lower level consistently.
This isprobably because the proposed method imposes gradual regularization, making the network changeslittle-by-little in the parameter space, while both the TP and FP direct prune less important weightsonce for all.
[Figure]
Conclusion
In this paper, we implement the regularization based method for 3D CNN acceleration.
By assigningdifferent regularization parameters to different weight groups according to the importance estimation, we gradually prune weight groups in the network.
The proposed method achieves better performancethan other two popular methods in this area.