Recurrent Convolutions: A Model Compression Point of View
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recurrent convolution (RC) shares the same convolutional kernels and unrollsthem multiple times, which is originally proposed to model sequential signals viaconvolution.
We suggest that RC can be viewed as a model compression strategy fordeep convolutional neural networks. RC reduces the redundancy across layers andis complementary to most existing model compression approaches.
However, theperformance of an RC network can’t match the performance of its correspondingstandard one, i.e. with the same depth but independent convolutional kernels.
Thisreduces the value of RC for model compression.
In this paper, we propose a simplevariant which improves the RC networks: The batch normalization layers of an RCmodule are learned independently (not shared) for different unrolling steps.
Weprovide insights on why this works.
Experiments on CIFAR show that unrollinga convolutional layer several steps can improve the performance, thus indirectlyplays a role in model compression.
Introduction
Deep convolution neural networks (DCNNs) have achieved ground-breaking results on a broad rangeof ﬁelds, such as computer vision [1] and natural language processing [2].
Unfortunately, DCNNs areboth computation intensive and memory intensive for industrial applications.
Many approaches havebeen proposed recently to obtain more compact DCNNs while keep their performance as much aspossible.
Conceptually, those approaches fall in two categories: 1) Reduce the computational cost ormemory usage of big DCNNs by weights pruning, quantization and sharing [3, 4, 5, 6]; 2) Improvethe performance of small DCNNs by knowledge distillation [7] or other techniques.
In this paper we explore a potential compression strategy which is complementary to most of theexisting approaches: training a recurrent convolutional (RC) neural network.
As the name suggests,the same convolutional kernels are unrolled multiple times on the computational graph.
This is aweights sharing mechanism applied to the whole layer.
Suppose there is a network with n RC layerseach of which unrolls k times, then we say its depth is nk .
If the performance of this network canmatch the performance of a standard DCNN with nk layers (Suppose other conditions are the same),we could say we compress the standard one with factor k .
Then we can further compress the obtained RC network by applying other existing approaches, such as weight quantization.
The key intuition isthat RC can reduce the redundancy across layers by sharing weights of the whole layer.
While mostexisting approaches work at a layer-wise manner and only remove part of a layer.
This is why we say RC is a complementary strategy.
However, we ﬁnd the performance of RC networks can’t match the performance of DCNNs with thesame depth.
This signiﬁcantly reduces the value of RC for model compression.
In this paper, we aimto improve the performance of RC in a cheap way.
Speciﬁcally, we learn the batch normalizationlayers (BN) [8] independently at each unrolling step.
We describe our insights in the next section.
Relu
Conv BN(2t-1)
Relu BN(2t)
Conv
Relu
Conv BN(2t+1)
Relu BN(2t+2)
Conv h(t) h(t+1) (a) Unrolling of res-block      60 2.
Step 1 Step 2 Step
Step(b)
Moving means
Experiments on CIFAR dataset demonstrate that such a simple variant works well.
We also compare RC networks with their corresponding standard ones.
The idea of RC is not new. Many works have used it to model sequential signals [9] or to obtain alarger receptive ﬁeld [10].
However, to our knowledge, none of them view RC as a potential modelcompression strategy and none of them compare RC networks with their corresponding standard onesstrictly.
The intention of this paper is to show that RC is a considerable or at least a heuristic solutionfor model compression.
Methodology
In this paper, we focus on image classiﬁcation via DCNNs.
Thus there are no sequential inputs for RC layers.
The state equation of RC is
[Equation]
(1) where f is a set of differentiable operators including convolutions and f is parameterized by w .
Here, we set f to the so-called res-block used in ResNet [1].
Each res-block is a composition ofconvolutional layers, BN layers and Relu activations [11], as shown in ﬁgure 1(a).
h i and h
i +1 arethe input and output of a res-block at i th unrolling step respectively.
When we directly unroll each res-block several steps during training, the performance is not satisfactory.
We believe that it is caused by the shared BN layers.
A BN layer captures the moving mean andvariance of its input, then scales and shifts the normalized input by its learned parameters.
However,there are no reasons to expect the statistics of the inputs over different unrolling steps are the same.
When we share the BN layers across unrolling steps, we merge those statistics over unrolling stepsinto a single mean and variance vector.
This would hurt the performance of RC networks.
Thus, weapply independent BN layers at each unrolling step.
Then the number of BN layers is proportional tothe number of unrolling steps.
Since we focus on image classiﬁcation (no sequential inputs), we canset the total unrolling steps of each res-block to a ﬁxed number.
Using independent BN layers may also improve the representation power of RC networks.
Becausethe parameters of BN are different across unrolling steps, the overall mapping function at eachunrolling step is no longer the same.
We can regard the parameters of BN layers as the additionalsequential inputs to an RC module.
Then the state equation of RC can be reformalized as follows:
[Equation]
where w c is the shared convolution kernels and b i +1 is the parameters of ( i + 1) th group of BNlayers.
This is the most general form of the state equation of recurrent neural networks although thereis no explicit sequential input.
Moreover, using independent BN layers is cheap on both computationand memory point of views, compared with the convolutional layers.
Now we turn to some practical considerations.
Due to its recurrent nature, the input size and outputsize of each RC block should be the same.
We change the size of feature map outside RC blocks.
[Table] [Table]
Speciﬁcally, we reduce the spatial resolution via average pooling. We reduce the spatial resolutionand increase the channels simultaneously via the invertible downsampling operation described in [12],which consists in reorganizing the initial spatial channels into the 4 spatially decimated copiesobtainable by 2 × 2 spatial sub-sampling.
To avoid gradient explosion and make the training morestable, we use gradient clip when training RC networks.
Experiments
We do three sets of experiments on CIFAR-10 and CIFAR-100 [13] datasets: 1) Study the effects of
BN layer usage; 2) Compare RC networks with the standard ones; 3) Compare RC networks withdifferent unrolling steps.
Both CIFAR-10 and CIFAR-100 have 50K training samples and 10K test samples, each of which is a32 × 32 color image.
The former has 10 classes of images while the later has 100 classes of images.
All neural network models are implemented with Pytorch [14].
We train each model 3 times andaverage its test errors.
See B for detailed experimental settings.
We choose the res-block in 1(a)
asthe basic unrolling cell and we set the number of cells of each RC network to 2.
Denote RC-i as thenetwork whose cells unroll i steps.
We train RC-1, RC-2 and RC-4 in this paper.
For fair comparisonswith the standard DCNNs, we slightly modify ResNet-18 used in [1] such that its computationalgraph is exactly the same as the one of unrolled RC-4, except the weight sharing mechanism. See Afor detailed network architecture.
As shown in ﬁgure 1(b), the moving means of the ﬁrst BN layer of RC-4 vary across unrolling steps. See C.2 for more results.
Further, as shown in table 1, using independent BN layers improves theaccuracy of RC-4 by a large margin.
Using shared BN layers has even lower accuracy than withoutusing BN layers.
Those results support the independent learning of BN layers and our discussions insection 2.
As shown in table 2, the accuracy of RC networks is improved when their cells are unrolled moresteps.
But when the standard ResNet and the unrolled RC network have the same depth, the former isstill better (the gap is not large).
It is worth noting that RC-4 whose original depth is 6 has betterperformance than Resnet-10.
We also show the number of parameters of each model trained on CIFAR-10 without further compression. As shown in ﬁgure 2, the convergence speed of RC networksand the standard ones looks similar. In another word, RC networks are easy to train.
Discussion
We suggest recurrent convolution is a considerable strategy for model compression.
RC reduces theredundancy across layers (which is ignored by most of the compression methods).
We can train an RCnetwork and then further compress it via existing approaches.
We also suggest it is signiﬁcantly betterto learn independent BN parameters at each unrolling step when training RC networks.
Experimentson CIFAR dataset demonstrate that unrolling the same convolutional layer several steps can improvethe accuracy of the whole network, thus indirectly plays a role in model compression.
We believethat the performance of RC could be further improved.
A Network Architecture
Denote Block ( i, k ) as the k th unrolling step of an RC res-block with i channels.
Denote InvPool (2)as the invertible downsampling operation described in [12].
Then the architecture of RC-4 is:
Conv (64) → BN (64) → Relu →
Block (64 , 1) → Block (64 , 2) → AvgPool (2) → Block (64 , 3) → Block (64 , 4) →
InvPool (2) →
Block (256 , 1) → Block (256 , 2) → AvgPool (2) → Block (256 , 3) → Block (256 , 4) →
Dense (10 / 100)
For RC-1, AvgPool (2) is applied before the ﬁrst unrolling step.
For RC-2, AvgPool (2) is appliedbefore the second unrolling step.
Moreover, we slightly modify the standard ResNet such that itscomputational graph is exactly the same as the one of unrolled RC network.
B Experimental Details
For fair comparisons, most of the hyper-parameters keep the same for all experiments.
We train all of the networks via SGD with moment 0.9 and weight decay 1e-4.
The initial learningrate is set to 0.1.
We reduce the learning rate by factor 10 at 60th epoch and 90th epoch.
The networkswhose depth are larger than 10 are trained with 160 epochs.
While the networks whose depth aresmaller or equal to 10 are trained with 120 epochs.
See table 2 for the depth of networks.
For allexperiments, we clip the gradients to [ − 0 .
We ﬁnd gradient clip makes the training processmore stable and improves the performance for RC networks.
For standard networks, gradient clip hasnearly no effect on their performance.
Other hyper-parameters are set to Pytorch’s default settings.
C More Experimental Results
C.1 Convergence Speed
Training loss and test errors on CIFAR-10 are shown in ﬁgure 2.
The convergence speed of RCnetworks and standard networks is similar.
[Figure]
C.2 Parameters of BN
The inference process of a BN layer is formulated as:
[Equation]
where µ is the moving mean and σ 2 is the moving variance.
γ and β are the learned parameters by
SGD.
We show those variables of both the ﬁrst BN layer and the second BN layer of RC-4 trained on
CIFAR-10 in ﬁgure 3 and 4 respectively.
[Figure]