Pruning neural networks: is it time to nip it in the bud?
Abstract
Pruning is a popular technique for compressing a neural network: a large pre-trained network is ﬁne-tuned while connections are successively removed.
However, the value of pruning has largely evaded scrutiny.
In this extended abstract, we examine residual networks obtained through Fisher-pruning and make two interesting observations.
First, when time-constrained, it is better to train a simple, smaller network from scratch than prune a large network.
Second, it is the architectures obtained through the pruning process — not the learnt weights — that prove valuable.
Such architectures are powerful when trained from scratch.
Furthermore, these architectures are easy to approximate without any further pruning: we can prune once and obtain a family of new, scalable network architectures for different memory requirements.
Introduction
Deep neural networks excel at a multitude of tasks (LeCun et al., 2015), but are typically cumbersome, and difﬁcult to deploy on embedded devices.
This can be rectiﬁed by compressing the network; speciﬁcally, reducing the number of parameters it uses in order to reduce its runtime memory.
This also reduces the number of operations that have to be performed; making the network faster.
A popular means of doing this is through pruning.
One trains a large network and ﬁne-tunes it while removing connections in succession.
This is an expensive procedure that often takes longer than simply training a smaller network from scratch.
Does a pruned-and-tuned network actually outperform a simpler, smaller counterpart?
If not, is there any beneﬁt to pruning?
In this work we show that for a given parameter budget, pruned-and-tuned networks are consistently beaten by networks with simpler structures (e g. a linear rescaling of channel widths) trained from scratch.
This indicates that there is little value in the weights learnt through pruning.
However, when the architectures obtained through pruning are trained from scratch (i.e. when all weights are reinitialised at random and the network is trained anew) they surpass their ﬁne-tuned equivalents and the simpler networks.
Moreover, these architectures are easy to approximate; we can look at which connections remain after pruning and derive a family of copycat architectures that when trained from scratch display similar performance.
This gives us a new set of compact, powerful architectures.
Related Work
Typically, a network is pruned by either setting unimportant weights to zero, resulting in a network with large, sparse weight matrices (Hanson & Pratt, 1989; LeCun et al., 1990; Han et al., 2015, 2016;
Scardapane et al., 2017; Wen et al., 2016), or by severing channel connections to produce a smaller, dense network (Molchanov et al., 2017; Theis et al., 2018;
He et al., 2017).
In this work we use
Fisher-pruning (Theis et al., 2018), a powerful channel pruning method, as recent work (Turner et al., 2018) has empirically demonstrated that on embedded devices this technique produces small, dense networks that both run efﬁciently, and perform well.
These networks are therefore applicable to real-time applications where efﬁciency is of the essence.
Concurrent to this work, Liu et al.
(2018) have demonstrated the beneﬁts of training pruned models from scratch for a variety of other pruning techniques.
They postulate that pruning may be seen as a form of architecture search.
This is contrary to the work of Frankle & Carbin (2018) who hypothesise that the pruning process ﬁnds well-initialised weights.
Very recently, Lee et al.
(2018) proposed a single-shot pruning scheme to produce a reduced architecture that is then trained from scratch; the benchmarks of this work and our discovered architectures are given in Section 3.2.
Experiments
Here, we perform a set of experiments to answer our two questions: (i) Does a pruned network outperform a smaller simpler network?
(ii) Is the pruned architecture itself useful?
We evaluate the performance of our networks by their classiﬁcation error rate on the test set of CIFAR-10 (Krizhevsky, 2009).
Implementation details are given at the end of the section.
3.1 Pruned networks vs. Simpler Networks trained from scratch
For our base network, we use WRN-40-2 — a WideResNet (Zagoruyko & Komodakis, 2016) with 40 layers, and channel width multiplier 2.
We chose this network as it features modular blocks and skip connections, and is therefore representative of a large number of commonly used networks.
It is reasonably compact and doesn’t have a large, redundant fully connected layer.
It has 2.2M parameters in total, the bulk of which lie in 18 residual blocks, each containing two convolutional layers.
Let us denote each block as B ( N i , N m , N o ) for variable N i , N m , N o — the input has N i channels and the ﬁrst convolutional layer has a N m channel output; this goes through the second layer which outputs N o channels.
By default, N m = N o .
The network is ﬁrst trained from scratch, and is then Fisher-pruned (Theis et al., 2018).
We prune the channels of the activations between the convolutions in each block; this has the effect of introducing a bottleneck (changing N m ), reducing the number of parameters used in each convolution.
We alternate between ﬁne-tuning the network and removing the channel that has the smallest estimated effect on the loss, as in Theis et al.
Before each channel is removed, the test error and parameter count for the network is recorded.
The resulting trajectory of Test Error versus Number of Parameters is represented by the red curve in Figure 1.
We compare this process to training smaller, simpler networks from scratch.
We train WRN-40- k networks — the yellow curve in Figure 1 — treating k as an architectural hyperparameter which we can vary to control the total number of parameters.
We also train WRN-40-2 networks where we apply a scalar multiplier z to the middle channel dimension in each block — B ( N i , zN o , N o ) — giving the green curve in Figure 1.
We similarly vary z to control the total number of parameters.
In both cases, we vary the relevant architectural hyperparameter ( k or z ) to produce networks that have between 200K and 2M parameters, in increments of roughly 100K. We can see that these simple networks trained from scratch consistently outperform the pruned-and-tuned networks.
This difference is markedly more pronounced the smaller the networks get.
[Figure]
green and yellow curve are almost always below the red: it is preferable to train smaller architectures from scratch rather than prune. If we take Fisher-pruned architectures obtained along the red curve and train them from scratch we get the blue curve. Finally, if we proﬁle the channel structure of a Fisher-pruned architecture, linearly scale it and train from scratch, we get the pink curve which closely follows the blue curve. Note that these and tend to outperform all others.
[Figure]
It may seem thus far that pruning is a pointless endeavor.
However, when we take the architectures produced along the pruning trajectory (at the same increments as above) and train them from scratch we obtain the blue curve in Figure 1.
We can see that these networks outperform both (i) the ﬁne-tuned versions of the same architectures and (ii) the simpler networks for most parameter budgets.
This supports the notion that the true value of pruning is as a form of architecture search.
Figure 2 shows how many channels remain at the bottleneck of each block (the various N m values) in the pruned-and-tuned WRN-40-2 after 500 channels have been Fisher-pruned (note that this proﬁle is consistent with those obtained after further pruning).
Notice that channels in later layers tend to be pruned; they are more expendable than those in earlier layers.
It is intriguing that channels are rarely pruned in blocks 7 and 13; these blocks both contain a strided convolution that reduces the spatial resolution of the image representation.
It is imperative that information capacity is retained for this process.
These observations are consistent with those made in previous works (Veit et al., 2016;
Huang et al., 2016; Jastrze¸bski et al., 2018).
To assess the merits of this particular structure, we train copycat Fisher architectures — architectures designed to mimic those obtained through Fisher-pruning.
Speciﬁcally, we train a range of architectures from scratch where N m in each block is proportional to that found in the Fisher-pruned architecture — block j can be represented by B ( N i , αN mj , N o ) : N mj is a block-speciﬁc value corresponding to the height of each bar in Figure 2(right), and α scales the whole architecture.
We vary α to produce networks with different parameter counts as before, which are then trained from scratch.
These are represented by the pink curve in Figure 1.
For most parameter budgets these networks perform similarly to those found through Fisher-pruning; this means we can simply prune a network once to ﬁnd a powerful, scalable network architecture.
Furthermore, the resulting networks are competitive.
We compare them to the WideResNets produced by the pruning method of Lee et al. (2018): one of which gets 5.85% error on CIFAR-10 and has 858K parameters, and another gets 6.63% error with 548K parameters.
Comparably, the copycat WideResNets in this work achieve (on average) 5.66% with 800K parameters, and 6.35% error with 500K parameters.
Implementation Details To train a network from scratch, we use SGD with momentum to minimise the cross-entropy loss for 200 epochs using mini-batches of size 128.
Images were augmented using horizontal ﬂips and random crops.
The initial learning rate is 0.1 and is decayed by a factor of 0. every 60 epochs.
We use weight decay of 5 × 10 − 4 and momentum of 0.9.
For Fisher-pruning we ﬁne-tune the learnt network with the lowest learning rate reached during training ( 8 × 10 − 4 ) and momentum and weight decay as above.
We measure the approximate effect on the change in loss (Theis et al., 2018) for each candidate channel in the network over 100 update steps and then remove the channel with the lowest value.
We experimented with several different values for the
FLOP penalty hyperparameter but found this made little difference.
We therefore set it to zero.
Conclusion
Pruning is a very expensive procedure, and should be avoided when one is time-constrained.
We show that under such constraints it is preferable to train a smaller, simpler network from scratch.
Our work supports the view that pruning should be seen as a form of architecture search; it is the resulting structure — not the learnt weights — that is important.
Pruned architectures trained from scratch perform well, and are easily emulated, as demonstrated by our copycat networks.
These are a step towards a more efﬁcient architecture for residual networks.
Future work could entail expanding this analysis to other network types, datasets, and pruning schema.
It would also be possible to use distillation techniques (Hinton et al., 2015; Zagoruyko & Komodakis, 2017) between our pruned architectures and the original architecture to further boost performance (Crowley et al., 2018).