Recurrent Convolutions: A Model Compression Point of View
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recurrent convolution (RC) shares and unrolls the same convolutional kernelsmultiple times, which is originally proposed to model sequence signals via convolutions.
We suggest that RC can be viewed as a model compression strategy fordeep convolutional neural networks, which reduces the redundancy across layersand is complementary to most existing model compression approaches.
The practical problem is the performance of RC networks can’t match their correspondingstandard convolutional ones’ (same depth
but kernels are not shared).
This reducesthe value of RC for model compression.
In this paper, we propose a simple variantto improve RC networks: when unrolling an RC layer, its batch normalizationlayers are learned independently (not shared) for different unrolling steps.
Weprovide insights on why this works.
Experiments on CIFAR show unrolling aconvolutional layer can improve the performance, thus indirectly plays a role inmodel compression.
Introduction
Deep convolution neural networks (DCNNs) has achieved ground-breaking results on a broad rangeof ﬁelds, such as computer vision [1] and natural language processing [2].
Typically, DCNNs are bothcomputationally intensive and memory intensive for industrial applications.
Many approaches haveproposed recently to obtain more compact DNNs while keep the performance of the original ones asmuch as possible.
Conceptually, those approaches fall in two categories: 1) Reduce the computationalcost or memory usage of big DCNNs by weights pruning, quantization and sharing [3, 4, 5, 6]; 2)
Improve the performance of small DCNNs by knowledge distillation [7]or other techniques.
In this paper, we explore a potential compression strategy which is complementary to most of theexisting approaches: training recurrent convolutional (RC) neural networks.
As the name suggests,the same convolutional kernels are unrolled multiple times on the computational graph.
This isa weights sharing mechanism applied to the whole layer.
Suppose we have n RC layers each ofwhich unrolls k times, then we say its depth is nk .
If the performance of this network can match theperformance of a standard DCNN with nk layers (suppose other conditions are same), we could saywe compress the latter with factor k .
Then we can further compress the obtained RC networks byapplying other existing approaches, such as weight pruning and quantization.
The key intuition isthat RC can reduce the redundancy across layers by sharing weights of the whole layer.
While mostof existing compression approaches work at a layer-wise manner and they only remove part of a layer.
This is why we say RC is a complementary strategy.
However, we ﬁnd the performance of RC networks can’t match the performance of DCNNs with thesame depth.
This signiﬁcantly reduces the value of RC for model compression.
In this paper, weaim to improve the performance of RC in a cheap way.
We simply learn the batch normalizationlayers (BN) [8] independently at each unrolling steps.
We describe our insights in the next section.
Relu
Conv *BN(2t-1)
Relu *BN(2t)
Conv
Relu
Conv *BN(2t+1)
Relu *BN(2t+2)
Conv h(t) h(t+1) (a) Unrolling of res-block      60 2.
Step 1 Step 2 Step
Step(b)
Running means
Experiments on CIFAR dataset demonstrate this simple variant signiﬁcantly improves the performanceof RC networks.
We also compare RC networks with the standard ones.
The idea of RC is not new and many works use it to model sequence signals [9] or to obtain alarger receptive ﬁeld [10].
However, to our knowledge, none of them view RC as a potential modelcompression strategy.
The intention of this paper is to show RC is a considerable or at least heuristicsolution for model compression.
Although there are no direct model compressions in this paper.
Methodology
In this paper, we focus on image classiﬁcation via DCNNs.
Thus there are no sequential inputs of RClayers.
The state equation of RC is
[Equation]
(1) where f is a set of differentiable operators which contain convolutions and is parameterized by w .
Here, we set f to the so-called res-block used in ResNet [1].
And h i and h
i +1 are the input and outputof a res-block at i th unrolling step respectively.
Each res-block is a composition of convolutionallayers, BN layers and Relu activations [11], as shown in ﬁgure (1(a)).
When we directly unroll each res-block several times during training, the performance is not satisfactory.
We think this is caused by BN layers.
BN layer captures the moving mean and variance of itsinput, then scales and shifts the normalized input by the learned parameters.
However, there are noreasons to expect the statistics of the input over different unrolling steps are the same.
When we sharethe BN layers across unrolling steps, we merge the statistics over unrolling steps into single meanand variance.
This would hurt the performance RC networks.
Thus, we apply independent BN layersat each unrolling step.
That is the number of BN layers is proportional to the number of unrollingsteps.
Since we focus on image classiﬁcation (no sequential inputs), we can set the total unrollingsteps of each res-block to a ﬁxed number.
Using independent BN layers at different unrolling steps may also improve the representation powerof RC networks.
Because the parameters of BN are different across unrolling steps, the overallmapping function at each unrolling step is no longer the same.
We can regard the parameters of
BN layers as the additional sequential inputs to RC layers.
Then the state equation of RC can bereformalized as follows:
[Equation]
where w c is the shared convolution kernels and b i +1 is the parameters of ( i +1)
th group of BN layers.
This is the most general form of the state equation of recurrent neural networks although we haven’texplicit sequential inputs.
Thus using independent BN layers would improve the representationpower.
Such a variant is cheap on both computation and memory point of views, compared withconvolutions.
Now we turn to some practical considerations.
Due to the recurrent nature, the input size and theoutput size of each RC block should be the same.
We change the size of input outside RC blocks.
[Table] [Table]
Speciﬁcally, in this work, we reduce the spatial resolution via average pooling and we reduce thespatial resolution and increase the channels simultaneously via the invertible downsampling operationdescribed in [12], which consists in reorganizing the initial spatial channels into the 4 spatiallydecimated copies obtainable by 2 × 2 spatial sub-sampling.
We also use gradient clip for training RCnetworks.
Experiments
We do three sets of experiments on CIFAR-10 and CIFAR-100 [13] datasets:
• Study the effects of BN layer usage.
• Compare the RC network with the standard one with the same depth.
• Compare RC networks with different unrolling steps.
Both CIFAR-10 and CIFAR-100 have 50K training samples and 10K test samples, each of which is a32 × 32 color image.
The former has 10 classes of images while the later has 100 classes of images.
All neural network models are implemented with Pytorch [14].
We train each model 3 times andaverage its test errors.
We choose the res-block in (1(a))
as the basic unrolling cell
and we set thenumber of cells of each RC network to 2.
Denote RC-i as the networks whose cells unroll i steps.
Wetrain RC-1, RC-2 and RC-4 in this paper.
For fair comparisons with standard DCNNs, we sightlymodify ResNet-18 used in [1] such that its computational graph is exactly the same as the one ofunrolled RC-4, except the weight sharing mechanism.
As shown in ﬁgure (1(b)), the running means of the ﬁrst BN layer of RC-4 vary across unrolling steps,especially for step 1 and step 2.
Further, as shown in table (1), using independent BN layers at eachunrolling step improves the accuracy of RC-4 by a large margin.
Using shared BN has even loweraccuracy than without using BN.
Those results support the independent learning of BN layers andour discussions in section (2).
As shown in table (2), the accuracy of RC networks is improved whentheir cells are unrolled more steps.
That is unrolling convolutional layers can improve a network’srepresentation power and is equivalent to make the network larger and deeper in some sense.
Butwhen the standard ResNet and the unrolled RC network have the same depth, accuracy of the formeris still better (the gap is not very large).
It is worth noting that RC-4 whose original depth is 6 hasbetter performance than Resnet-10.
We also show the number of trainable parameters of each modeltrained on CIFAR-10 (counted directly without any further compression).
Discussion
We suggest recurrent convolution is a considerable strategy for model compression.
RC reduces theredundancy across layers (which is ignored by most of the compression methods).
We can train an
RC network and then further compress it via existing approaches.
We also suggest it is signiﬁcantlybetter to learn independent BN parameters for the RC network (not to share BN layers).
Experimentson CIFAR dataset demonstrate that unrolling the same convolutional layer several times can improvethe accuracy of the whole network.
We believe the performance of RC could be further improved.