Pruning neural networks: is it time to nip it in the bud?
Abstract
Pruning is a popular technique for compressing a neural network: a large pre-trainednetwork is ﬁne-tuned while connections are successively removed.
However, thevalue of pruning has largely evaded scrutiny.
In this extended abstract, we examineresidual networks obtained through Fisher-pruning and make two interesting observations.
First, when time-constrained, it is better to train a simple, smaller networkfrom scratch than prune a large network.
Second, it is the architectures obtainedthrough the pruning process — not the learnt weights —that prove valuable.
Sucharchitectures are powerful when trained from scratch.
Furthermore, these architectures are easy to approximate without any further pruning: we can prune onceand obtain a family of new, scalable network architectures for different memoryrequirements
Introduction
Deep neural networks excel at a multitude of tasks (LeCun et al., 2015), but are typically cumbersome,and difﬁcult to deploy on embedded devices.
This can be rectiﬁed by compressing the network;speciﬁcally, reducing the number of parameters it uses in order to reduce its runtime memory.
Thisalso reduces the number of operations that have to be performed; making the network faster.
A popular means of doing this is through pruning.
One takes a large, pre-trained network andﬁne-tunes it while removing connections in succession.
This is an expensive procedure that oftentakes longer than simply training a smaller network from scratch.
Does a pruned-and-tuned networkactually outperform a simpler, smaller counterpart?
If not, is there any beneﬁt to pruning?
In this work, we show that for a given parameter budget, pruned-and-tuned networks are consistentlybeaten by networks with simpler structures (e.g. a linear rescaling of channel widths) trained fromscratch.
This indicates that there is little value in the weights learnt through pruning.
However,when the architectures obtained through pruning are trained from scratch (i.e. when all weights arereinitialised at random and the network is trained anew) they surpass their ﬁne-tuned equivalents andthe simpler networks.
Moreover, these architectures are easy to approximate; we can look at whichconnections remain after pruning and derive a family of copycat architectures that when trained fromscratch display similar performance.
This gives us a new set of compact, powerful architectures.
Related Work
Typically, a network is pruned by either setting unimportant weights to zero, resulting in a networkwith large, sparse weight matrices (Hanson & Pratt, 1989; LeCun et al., 1990; Han et al., 2015, 2016;
Scardapane et al., 2017; Wen et al., 2016), or by severing channel connections to produce a smaller,dense network (Molchanov et al., 2017; Theis et al., 2018;
He et al., 2017).
In this work we use
Fisher-pruning (Theis et al., 2018), a powerful channel pruning method, as recent work (Turner et al.,2018) has empirically demonstrated that on embedded devices this technique produces small, densenetworks that both run efﬁciently, and perform well.
These networks are therefore applicable toreal-time applications where efﬁciency is of the essence
Concurrent to this work, Anonymous (2019) have demonstrated the beneﬁts of training pruned modelsfrom scratch for a variety of other pruning techniques.
They postulate that pruning may be seen as aform of architecture search.
This is contrary to the work of Frankle & Carbin (2018) who hypothesisethat the pruning process ﬁnds well-initialised weights.
Very recently, Lee et al.
(2018) proposed asingle-shot pruning scheme to produce a reduced architecture that is then trained from scratch; thebenchmarks of this work and our discovered architectures are given in Section 3.2.
Experiments
Here, we perform a set of experiments to answer our two questions: (i) Does a pruned networkoutperform a smaller simpler network?
(ii) Is the pruned architecture itself useful?
We evaluate theperformance of our networks by their classiﬁcation error rate on the test set of CIFAR-10 (Krizhevsky,2009).
Implementation details are given at the end of the section.
3.1 Pruned networks vs. Simpler Networks trained from scratch
For our base network, we use WRN-40-2 — a WideResNet (Zagoruyko & Komodakis, 2016) with40 layers, and channel width multiplier 2.
It has 2.2M parameters in total, the bulk of whichlie in 18 residual blocks, each containing two convolutional layers.
Let us denote each block as
B ( N i , N m , N o ) for variable N i , N m , N o — the input has N i channels and the ﬁrst convolutionallayer has a N m channel output; this goes through the second layer which outputs N o channels.
Bydefault, N m = N o .
The network is ﬁrst trained from scratch, and is then Fisher-pruned (Theis et al., 2018).
We prune thechannels of the activations between the convolutions in each block; this has the effect of introducing abottleneck (changing N m ), reducing the number of parameters used in each convolution.
We alternatebetween ﬁne-tuning the network and removing the channel that has the smallest estimated effect onthe loss, as in Theis et al.
Before each channel is removed, the test error and parameter countfor the network is recorded.
The resulting trajectory of Test Error versus Number of Parameters isrepresented by the red curve in Figure 1.
We compare this process to training smaller, simpler networks from scratch.
We train WRN-40- knetworks — the yellow curve in Figure 1 — treating k as an architectural hyperparameter whichwe can vary to control the total number of parameters.
We also train WRN-40-2 networks wherewe apply a scalar multiplier z to the middle channel dimension in each block — B ( N i , zN o , N o ) —giving the green curve in Figure 1.
We similarly vary z to control the total number of parameters.
In both cases, we vary the relevant architectural hyperparameter ( k or z ) to produce networks thathave between 200K and 2M parameters, in increments of roughly 100K. We can see that thesesimple networks trained from scratch consistently outperform the pruned-and-tuned networks.
Thisdifference is markedly more pronounced the smaller the networks get.
[Figure]
to train smaller architectures from scratch rather than prune.
If we take Fisher-pruned architectures obtained along the red curve and train them from scratch we get the blue curve.
Finally, if we proﬁle the channel structure of a Fisher-pruned architecture, linearly scale it and train from scratch, we get the pink curve which closely follows the blue curve.
Note that these blue and pink networks tend to outperform all others.
pruned
[Figure]
nn
[Figure]
l
[Figure]
It may seem thus far that pruning is a pointless endeavor.
However, when we take the architecturesproduced along the pruning trajectory (at the same increments as above) and train them from scratchwe obtain the blue curve in Figure 2.
We can see that these networks outperform both (i) the ﬁne-tunedversions of the same architectures and (ii) the simpler networks for most parameter budgets.
Thissupports the notion that the true value of pruning is as a form of architecture search.
Figure 2 shows how many channels remain at the bottleneck of each block (the various N m values) inthe pruned-and-tuned WRN-40-2 after 500 channels have been Fisher-pruned (note that this proﬁleis consistent with those obtained after further pruning).
Notice that channels in later layers tend tobe pruned; they are more expendable than those in earlier layers.
It is intriguing that channels arerarely pruned in blocks 7 and 13; these blocks both contain a strided convolution that reduces thespatial resolution of the image representation.
It is imperative that information capacity is retainedfor this process.
These observations are consistent with those made in previous works (Veit et al.
Huang et al., 2016; Jastrze¸bski et al., 2018).
To assess the merits of this particular structure, we train copycat Fisher architectures — architectures designed to mimic those obtained through Fisher-pruning.
Speciﬁcally, we train a range ofarchitectures from scratch where N m in each block is proportional to that found in the Fisher-prunedarchitecture — block j can be represented by B ( N i , αN mj , N o ) : N mj is a block-speciﬁc valuecorresponding to the height of each bar in Figure 2(right), and α scales the whole architecture.
Wevary α to produce networks with different parameter counts as before, which are then trained fromscratch.
These are represented by the pink curve in Figure 1.
For most parameter budgets thesenetworks perform similarly to those found through Fisher-pruning; this means we can simply prunea network once to ﬁnd a powerful, scalable network architecture.
Furthermore, thee resulting arecompetitive.
We compare them to the WideResNets produced by the pruning method of Lee et al.(2018): one of which gets 5.85% error on CIFAR-10 and has 858K parameters, and another gets6.63% error with 548K parameters.
Comparably, the copycat WideResNets in this work achieve (onaverage) 5.66% with 800K parameters, and 6.35% error with 500K parameters.
Implementation Details To train a network from scratch, we use SGD with momentum to minimisethe cross-entropy loss for 200 epochs using mini-batches of size 128.
Images were augmented usinghorizontal ﬂips and random crops.
The initial learning rate is 0.1 and is decayed by a factor of 0.2every 60 epochs.
We use weight decay of 5 × 10 − 4 and momentum of 0.9.
For Fisher-pruningwe ﬁne-tune the learnt network with the lowest learning rate reached during training ( 8 × 10 − 4 )and momentum and weight decay as above.
We measure the approximate effect on the change inloss (Theis et al., 2018) for each candidate channel in the network over 100 update steps and thenremove the channel with the lowest value.
We experimented with several different values for the
FLOP penalty hyperparameter but found this made little difference.
We therefore set it to zero.
Conclusion
Pruning is a very expensive procedure, and should be avoided when one is time-constrained.
Weshow that under such constraints it is preferable to train a smaller, simpler network from scratch.
Our work supports the view that pruning should be seen as a form of architecture search; it is theresulting structure — not the learnt weights — that is important.
Pruned architectures trained fromscratch perform well, and are easily emulated, as demonstrated by our copycat networks.
These are astep towards a more efﬁcient architecture for residual networks.
Future work could entail expandingthis analysis to other network types, datasets, and pruning schema.
It would also be possible to usedistillation techniques (Hinton et al., 2015; Zagoruyko & Komodakis, 2017) between our prunedarchitectures and the original architecture to further boost performance (Crowley et al., 2018).