SNIP: S INGLE - SHOT N ETWORK P RUNING BASED ON C ONNECTION S ENSITIVITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Pruning large neural networks while maintaining the performance is often highly desirable due to the reduced space and time complexity.
In existing methods, pruning is incorporated within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility.
In this work, we present a new approach that prunes a given network once at initialization.
Speciﬁcally, we introduce a saliency criterion based on connection sensitivity that identiﬁes structurally important connections in the network for the given task even before training.
This eliminates the need for both pretraining as well as the complex pruning schedule while making it robust to architecture variations.
After pruning, the sparse network is trained in the standard way.
Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST , CIFAR-10, and Tiny-ImageNet classiﬁcation tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks.
Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.
I NTRODUCTION
Despite the success of deep neural networks in machine learning, they are often found to be highly overparametrized making them computationally expensive with excessive memory requirements.
Pruning such large networks with minimal loss in performance is appealing for real-time applications, especially on resource-limited devices.
In addition, compressed neural networks utilize the model capacity efﬁciently, and this interpretation can be used to derive better generalization bounds for neural networks (Arora et al.
In network pruning, given a large reference neural network, the goal is to learn a much smaller subnetwork that mimics the performance of the reference network.
Majority of the existing methods in the literature attempt to ﬁnd a subset of weights from the pretrained reference network either based on a saliency criterion (Mozer & Smolensky (1989); LeCun et al.
; Han et al.
or utilizing sparsity enforcing penalties (Chauvin (1989); Carreira-Perpi˜n´an & Idelbayev (2018)).
Unfortunately, since pruning is included as a part of an iterative optimization procedure, all these methods require many expensive prune – retrain cycles and highly heuristic design choices with additional hyperparameters, making them non-trivial to extend to new architectures and tasks.
In this work, we introduce a saliency criterion that identiﬁes connections in the network that are important to the given task in a data-dependent way even before training.
Speciﬁcally, we discover important connections based on their inﬂuence on the loss function at a variance scaling initialization (we call this connection sensitivity).
Now, given the desired sparsity level, redundant connections are pruned once prior to training ( i.e ., single-shot), and then the sparse pruned network is trained in the standard
way
Our approach has several attractive properties:
• Simplicity.
Since the network is pruned once prior to training, there is no need for pretraining and complex pruning schedules.
Our method has no additional hyperparameters and once pruned, training of the sparse network is performed in the standard way.
• Versatility.
Since our saliency criterion chooses structurally important connections, it is robust to architecture variations.
Therefore our method can be applied to various architectures including convolutional, residual and recurrent networks with no modiﬁcations.
i
• Interpretability.
Our method determines important connections with a mini-batch of data at single-shot.
By varying this mini-batch used for pruning, our method enables us to verify that the retained connections are indeed essential for the given task.
We evaluate our method on MNIST , CIFAR-10 , and Tiny-ImageNet classiﬁcation datasets with widely varying architectures.
Despite being the simplest, our method obtains extremely sparse networks with virtually the same accuracy as the existing baselines across all tested architectures.
Furthermore, we investigate the relevance of the retained connections as well as the effect of the network initialization and the dataset on the saliency score.
R ELATED WORK
Classical methods.
Essentially, early works in network pruning can be categorized into two groups (Reed (1993)): 1) those that utilize sparsity enforcing penalties; and 2) methods that prune the network based on some saliency criterion.
The methods from the former category (Chauvin (1989); Weigend et al.
(1991); Ishikawa (1996))
augment the loss function with some sparsity enforcing penalty terms ( e.g ., L 0 or L 1 norm), so that back-propagation effectively penalizes the magnitude of the weights during training.
Then weights below a certain threshold may be removed.
On the other hand, classical saliency criteria include the sensitivity of the loss with respect to the neurons (Mozer & Smolensky (1989)) or the weights (Karnin (1990)) and Hessian of the loss with respect to the weights (LeCun et al.
; Hassibi et al.
Since these criteria are heavily dependent on the scale of the weights and are designed to be incorporated within the learning process, these methods are prohibitively slow requiring many iterations of pruning and learning steps.
Our approach identiﬁes redundant weights from an architectural point of view and prunes them once at the beginning before training.
Modern advances.
In recent years, the increased space and time complexities as well as the risk of overﬁtting in deep neural networks prompted a surge of further investigation in network pruning.
While Hessian based approaches employ the diagonal approximation due to its computational overhead, impressive results ( i.e ., extreme sparsity without loss in accuracy) are achieved using magnitude of the weights as the criterion (Han et al.
This made them the de facto standard method for network pruning and led to various implementations (Guo et al.
; Carreira-Perpi ˜ n ´ an &
Idelbayev (2018)).
The magnitude criterion is also extended to recurrent neural networks (Narang et al.
(2017)), yet with heavily tuned hyperparameter setting.
Unlike our approach, the main drawbacks of magnitude based approaches are the reliance on pretraining and the expensive prune – retrain cycles.
Furthermore, since pruning and learning steps are intertwined, they often require highly heuristic design choices which make them non-trivial to be extended to new architectures and different tasks.
Meanwhile, Bayesian methods are also applied to network pruning (Ullrich et al.
; Molchanov et al.
(2017)) where the former extends the soft weight sharing in Nowlan & Hinton (1992) to obtain a sparse and compressed network, and the latter uses variational inference to learn the dropout rate which can then be used to prune the network.
Unlike the above methods, our approach is simple and easily adaptable to any given architecture or task without modifying the pruning procedure.
Network compression in general.
Apart from weight pruning, there are approaches focused on structured simpliﬁcation such as pruning ﬁlters (Li et al.
, structured sparsity with regularizers (Wen et al.
low-rank approximation (Jaderberg et al.
matrix and tensor factorization (Novikov et al.
(2015)), and sparsiﬁcation using expander graphs (Prabhu et al.
In addition, there is a large body of work on compressing the representation of weights.
A nonexhaustive list includes quantization (Gong et al.
reduced precision (Gupta et al.
and binary weights (Hubara et al.
In this work, we focus on weight pruning that is free from structural constraints and amenable to further compression schemes.
N EURAL NETWORK PRUNING
The main hypothesis behind the neural network pruning literature is that neural networks are usually overparametrized, and comparable performance can be obtained by a much smaller network (Reed (1993)) while improving generalization (Arora et al.
To this end, the objective is to learn ii a sparse network while maintaining the accuracy of the standard reference network.
Let us ﬁrst formulate neural network pruning as an optimization problem.
Given a dataset D = { ( x i , y i ) } ni =1 , and a desired sparsity level κ ( i.e ., the number of non-zero weights) neural network pruning can be written as the following constrained optimization problem:
[Equation]
Here, (cid:96) ( · ) is the standard loss function ( e.g ., cross-entropy loss), w is the set of parameters of the neural network, m is the total number of parameters and (cid:107) · (cid:107) 0 is the standard L 0 norm.
The conventional approach to optimize the above problem is by adding sparsity enforcing penalty terms (Chauvin (1989); Weigend et al.
(1991); Ishikawa (1996)).
Recently, Carreira-Perpi˜n´an & Idelbayev (2018) attempts to minimize the above constrained optimization problem using the stochastic version of projected gradient descent (where the projection is accomplished by pruning).
However, these methods often turn out to be inferior to saliency based methods in terms of resulting sparsity and require heavily tuned hyperparameter settings to obtain comparable results.
On the other hand, saliency based methods treat the above problem as selectively removing redundant parameters (or connections) in the neural network.
In order to do so, one has to come up with a good criterion to identify such redundant connections.
Popular criteria include magnitude of the weights, i.e ., weights below a certain threshold are redundant (Han et al.
; Guo et al.
and Hessian of the loss with respect to the weights, i.e ., higher the value of Hessian, higher the importance of the parameters (LeCun et al.
; Hassibi et al.
[Equation]
, deﬁned as follows:
[Equation]
Here, for connection j , s j is the saliency score, w j is the weight, and H jj is the value of the
Hessian matrix, where the Hessian H = ∂ 2 L/∂ w 2 ∈ R m × m .
Considering Hessian based methods, the Hessian matrix is neither diagonal nor positive deﬁnite in general, approximate at best, and intractable to compute for large networks.
Despite being popular, both of these criteria depend on the scale of the weights and in turn require pretraining and are very sensitive to the architectural choices.
For instance, different normalization layers affect the scale of the weights in a different way, and this would non-trivially affect the saliency score.
Furthermore, pruning and the optimization steps are alternated many times throughout training, resulting in highly expensive prune – retrain cycles .
Such an exorbitant requirement hinders the use of pruning methods in large-scale applications and raises questions about the credibility of the existing pruning criteria.
In this work, we design a criterion which directly measures the connection importance in a datadependent manner.
This alleviates the dependency on the weights and enables us to prune the network once at the beginning, and then the training can be performed on the sparse pruned network.
Therefore, our method eliminates the need for the expensive prune – retrain cycles, and in theory, it can be an order of magnitude faster than standard neural network training as it can be implemented using software libraries that support sparse matrix computations.
S INGLE - SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY
Given a neural network and a dataset, our goal is to design a method that can selectively prune redundant connections for the given task in a data-dependent way even before training.
To this end, we ﬁrst introduce a criterion to identify important connections and then discuss its beneﬁts.
4. C ONNECTION SENSITIVITY : ARCHITECTURAL PERSPECTIVE
Since we intend to measure the importance (or sensitivity) of each connection independently of its weight, we introduce auxiliary indicator variables c ∈ { 0 , 1 } m representing the connectivity of iii parameters w .
1 Now, given the sparsity level κ , Equation 1 can be correspondingly modiﬁed as:
[Equation]
where (cid:12) denotes the Hadamard product.
Compared to Equation 1, we have doubled the number of learnable parameters in the network and directly optimizing the above problem is even more difﬁcult.
However, the idea here is that since we have separated the weight of the connection ( w ) from whether the connection is present or not ( c ), we may be able to determine the importance of each connection by measuring its effect on the loss function.
For instance, the value of c j indicates whether the connection j is active ( c j = 1 ) in the network or pruned ( c j = 0 ).
Therefore, to measure the effect of connection j on the loss, one can try to measure the difference in loss when c j = 1 and c j = 0 , keeping everything else constant.
Precisely, the effect of removing connection j can be measured by,
[Equation]
where e j is the indicator vector of element j ( i.e ., zeros everywhere except at the index j where it is one) and 1 is the vector of dimension m .
Note that computing ∆ L j for each j ∈ { 1 . . .
m } is prohibitively expensive as it requires m +(usually in the order of millions) forward passes over the dataset.
In fact, since c is binary, L is not differentiable with respect to c , and it is easy to see that ∆ L j attempts to measure the inﬂuence of connection j on the loss function in this discrete setting.
Therefore, by relaxing the binary constraint on the indicator variables c , ∆ L j can be approximated by the derivative of L with respect to c j , which we denote g j ( w ; D ) .
Hence, the effect of connection j on the loss can be written as: ∆ L j
[Equation]
In fact, ∂L/∂c j is an inﬁnitesimal version of ∆ L j , that measures the rate of change of L with respect to an inﬁnitesimal change in c j from 1 → 1 − δ .
This can be computed efﬁciently in one forward-backward pass using automatic differentiation, for all j at once.
Notice, this formulation can be viewed as perturbing the weight w j by a multiplicative factor δ and measuring the change in loss.
This approximation is similar in spirit to Koh & Liang (2017) where they try to measure the inﬂuence of a datapoint to the loss function.
Here we measure the inﬂuence of connections.
Furthermore, ∂L/∂c j is not to be confused with the gradient with respect to the weights ( ∂L/∂w j ) , where the change in loss is measured with respect to an additive change in weight w j .
Notably, our interest is to discover important (or sensitive) connections in the architecture, so that we can prune unimportant ones in single-shot, disentangling the pruning process from the iterative optimization cycles.
To this end, we take the magnitude of the derivatives g j as the saliency criterion.
Note that if the magnitude of the derivative is high (regardless of the sign), it essentially means that the connection c j has a considerable effect on the loss (either positive or negative), and it has to be preserved to allow learning on w j .
Based on this hypothesis, we deﬁne connection sensitivity as the normalized magnitude of the derivatives:
[Equation]
Once the sensitivity is computed, only the top- κ connections are retained, where κ denotes the desired number of non-zero weights.
Precisely, the indicator variables c are set as follows:
[Equation]
where ˜ s κ is the κ -th largest element in the vector s and 1 [ · ] is the indicator function.
Here, for exactly κ connections to be retained, ties can be broken arbitrarily.
We would like to clarify that the above criterion (Equation 6) is different from the criteria used in early works by Mozer & Smolensky (1989) or Karnin (1990) which do not entirely capture the iv
Require: Loss function L , training dataset D , sparsity level κ (cid:46) Refer Equation
Ensure: (cid:107) w ∗ (cid:107) 0 ≤ κ 1: w ← VarianceScalingInitialization (cid:46) Refer Section 4.
2: D b = { ( x i , y i ) } bi =1 ∼ D (cid:46)
Sample a mini-batch of training data 3: s j ← | g j ( w ; D b ) | (cid:80) mk =1 | g k ( w ; D b ) | , ∀ j ∈ { 1 . . .
m } (cid:46) Connection sensitivity 4: ˜ s ← SortDescending ( s ) 5: c
j ← 1 [ s j − ˜ s κ ≥ 0] , ∀ j ∈ { 1 . . .
m } (cid:46)
Pruning: choose top- κ connections 6: w ∗ ← arg min w ∈ R m L ( c (cid:12) w ; D ) (cid:46)
Regular training 7: w ∗ c w ∗ connection sensitivity.
The fundamental idea behind them is to identify elements ( e.g . weights or neurons) that least degrade the performance when removed.
This means that their saliency criteria ( i.e . − ∂L/∂ w or − ∂L/∂
α ; α refers to the connectivity of neurons), in fact, depend on the loss value before pruning, which in turn, require the network to be pre-trained and iterative optimization cycles to ensure minimal loss in performance.
They also suffer from the same drawbacks as the magnitude and Hessian based methods as discussed in Section 3.
In contrast, our saliency criterion (Equation 6) is designed to measure the sensitivity as to how much inﬂuence elements have on the loss function regardless of whether it is positive or negative.
This criterion alleviates the dependency on the value of the loss, eliminating the need for pre-training.
These fundamental differences enable the network to be pruned at single-shot prior to training, which we discuss further in the next section.
4. S INGLE - SHOT PRUNING AT INITIALIZATION
Note that the saliency measure deﬁned in Equation 6 depends on the value of weights w used to evaluate the derivative as well as the dataset D and the loss function L .
In this section, we discuss the effect of each of them and show that it can be used to prune the network in single-shot with initial weights w .
Firstly, in order to minimize the impact of weights on the derivatives ∂L/∂c j , we need to choose these weights carefully.
For instance, if the weights are too large, the activations after the non-linear function ( e.g ., sigmoid) will be saturated, which would result in uninformative gradients.
Therefore, the weights should be within a sensible range.
In particular, there is a body of work on neural network initialization (Goodfellow et al.
(2016)) that ensures the gradients to be in a reasonable range,
and our saliency measure can be used to prune neural networks at any such initialization.
Furthermore, we are interested in making our saliency measure robust to architecture variations.
Note that initializing neural networks is a random process, typically done using normal distribution.
However, if the initial weights have a ﬁxed variance, the signal passing through each layer no longer guarantees to have the same variance, as noted by LeCun et al.
This would make the gradient and in turn our saliency measure, to be dependent on the architectural characteristics.
Thus, we advocate the use of variance scaling methods ( e.g ., Glorot & Bengio (2010)) to initialize the weights, such that the variance remains the same throughout the network.
By ensuring this, we empirically show that our saliency measure computed at initialization is robust to variations in the architecture.
Next, since the dataset and the loss function deﬁnes the task at hand, by relying on both of them, our saliency criterion in fact discovers the connections in the network that are important to the given task.
However, the practitioner needs to make a choice on whether to use the whole training set, or a mini-batch or the validation set to compute the connection saliency.
Moreover, in case there are memory limitations ( e.g ., large model or dataset), one can accumulate the saliency measure over multiple batches or take an exponential moving average.
In our experiments, we show that using only one mini-batch of a reasonable number of training examples can lead to effective pruning.
Finally, in contrast to the previous approaches, our criterion for ﬁnding redundant connections is simple and directly based on the sensitivity of the connections.
This allows us to effectively identify and prune redundant connections in a single step even before training.
Then, training can be performed on the resulting pruned (sparse) network.
We name our method SNIP for Single-shot
Network Pruning, and the complete algorithm is given in Algorithm 1.
v
[Figure]
E XPERIMENTS
We evaluate our method, SNIP , on MNIST and CIFAR-10 classiﬁcation tasks with a variety of network architectures.
Our results show that SNIP yields extremely sparse models with minimal or no loss in accuracy across all tested architectures, while being much simpler than other state-of-the-art alternatives.
We also provide a clear evidence that our method prunes genuinely explainable connections rather than performing blind pruning.
Experiment setup For brevity, we deﬁne the sparsity level to be ¯ κ = ( m − κ ) /m · 100 (%) , where m is the total number of parameters and κ is the desired number of non-zero weights.
For a given sparsity level ¯ κ , the sensitivity scores are computed using a batch of 100 and 128 examples for MNIST and CIFAR experiments, respectively.
After pruning, the pruned network is trained in the standard way.
Speciﬁcally, we train the models using SGD with momentum of 0 .
9 , batch size of 100 for MNIST and 128 for CIFAR experiments and the weight decay rate of 0 . 0005 , unless stated otherwise.
The initial learning rate is set to 0 .
1 and decayed by 0 .
1 at every 25 k or 30 k iterations for MNIST and CIFAR , respectively.
Our algorithm requires no other hyperparameters or complex learning/pruning schedules as in most pruning algorithms.
We spare 10 % of the training data as a validation set and used only 90 % for training.
For CIFAR experiments, we use the standard data augmentation ( i.e ., random horizontal ﬂip and random translation up to 4 pixels) for both the reference and sparse models.
Our algorithm is implemented using TensorFlow (Abadi et al.
, and the code replicating our experiments will be released upon publication.
5. P RUNING L E N ETS WITH VARYING LEVELS OF SPARSITY
We ﬁrst test our approach on two standard networks for pruning, LeNet-300-100 and LeNet-5-Caffe .
LeNet-300-100 consists of three fully-connected (fc) layers with 267 k parameters and LeNet-5-Caffe consists of two convolutional (conv) layers and two fc layers with 431 k parameters.
We prune the LeNets for different sparsity levels ¯ κ and report the performance in error on the MNIST image classiﬁcation task.
We run the experiment 20 times for each ¯ κ by changing random seeds for dataset and network initialization.
The results are reported in Figure 1.
The pruned sparse LeNet-300-100 achieves performances similar to the reference ( ¯ κ = 0 ), only with negligible loss at ¯ κ = 90 .
For LeNet-5-Caffe , the performance degradation is nearly invisible.
Note that our saliency measure does not require the network to be pre-trained and is computed at random initialization.
Despite such simplicity, our approach prunes LeNets quickly (single-shot) and effectively (minimal accuracy loss) at varying sparsity levels.
5. C OMPARISONS TO EXISTING APPROACHES
What happens if we increase the target sparsity to an extreme level?
For example, would a model with only 1 % of the total parameters still be trainable and perform well?
We test our approach for extreme sparsity levels ( e.g ., up to 99 % sparsity on LeNet-5-Caffe ) and compare ours against the following various pruning algorithms: LWC (Han et al.
, DNS (Guo et al.
, LC (Carreiravi
[Table]
to other approaches.
Here, “many” refers to an arbitrary number often in the order of total learning steps, and “soft” refers to soft pruning in
Bayesian based methods.
Our approach is capable of pruning up to 98 % for LeNet-300-100 and 99 % for LeNet-5-Caffe with marginal increases in error from the reference network.
Notably, our approach is considerably simpler than other approaches, with no requirements such as pretraining, additional hyperparameters, augmented training objective or architecture dependent constraints.
Perpi ˜ n ´ an & Idelbayev (2018)), SWS (Ullrich et al.
, SVD (Molchanov et al.
, OBD (LeCun et al.
, L-OBS (Dong et al.
The results are summarized in Table 1.
We achieve errors that are comparable to the reference model, degrading approximately 0 .
7 % and 0 .
3 % while pruning 98 % and 99 % of the parameters in LeNet-300-100 and LeNet-5-Caffe respectively.
For slightly relaxed sparsities ( i.e ., 95 % for LeNet-300-100 and 98 % for LeNet-5-Caffe ), the sparse models pruned by SNIP record better performances than the dense reference network.
Considering 99% sparsity, our method efﬁciently ﬁnds 1% of the connections even before training, that are sufﬁcient to learn as good as the reference network.
Moreover, SNIP is competitive to other methods, yet it is unparalleled in terms of algorithm simplicity.
To be more speciﬁc, we enumerate some key points and non-trivial aspects of other algorithms and highlight the beneﬁt of our approach.
First of all, the aforementioned methods require networks to be fully trained (if not partly) before pruning.
These approaches typically perform many pruning operations even if the network is well pretrained, and require additional hyperparameters ( e.g ., pruning frequency in Guo et al.
, annealing schedule in Carreira-Perpi ˜ n ´ an & Idelbayev (2018)).
Some methods augment the training objective to handle pruning together with training, increasing the complexity of the algorithm ( e.g ., augmented Lagrangian in Carreira-Perpi ˜ n ´ an & Idelbayev (2018), variational inference in Molchanov et al.
Furthermore, there are approaches designed to include architecture dependent constraints ( e.g ., layer-wise pruning schemes in Dong et al.
Compared to the above approaches, ours seems to cost almost nothing; it requires no pretraining or additional hyperparameters, and is applied only once at initialization.
This means that one can easily plug-in SNIP as a preprocessor before training neural networks.
Since SNIP prunes the network at the beginning, we could potentially expedite the training phase by training only the survived parameters ( e.g ., reduced expected FLOPs in Louizos et al.
Notice that this is not possible for the aforementioned approaches as they obtain the maximum sparsity at the end of the process.
5. V ARIOUS MODERN ARCHITECTURES
In this section we show that our approach is generally applicable to more complex modern network architectures including deep convolutional, residual and recurrent ones.
Speciﬁcally, our method is applied to the following models:
• AlexNet-s and AlexNet-b : Models similar to Krizhevsky et al.
(2012) in terms of the number of layers and size of kernels.
We set the size of fc layers to 512 ( AlexNet-s ) and to 1024 ( AlexNet-b ) to adapt for CIFAR-10 and use strides of 2 for all conv layers instead of using pooling layers.
• VGG-C , VGG-D and VGG-like : Models similar to the original VGG models described in Simonyan & Zisserman (2015).
VGG-like (Zagoruyko (2015)) is a popular variant adapted for CIFARwhich has one less fc layers.
For all VGG models, we set the size of fc layers to 512 , remove dropout layers to avoid any effect on sparsiﬁcation and use batch normalization instead.
• WRN-16-
vii
[Table]
AlexNets , VGGs and WRNs are evaluated on CIFAR-10 , and LSTMs and GRUs are evaluated on the sequential MNIST classiﬁcation task.
The approach is generally applicable regardless of architecture types and models and results in a signiﬁcant amount of reduction in the number of parameters with
minimal or no loss in performance.
• LSTM-s , LSTM-b , GRU-s , GRU-b : One layer RNN networks with either LSTM (Zaremba et al.
or GRU (Cho et al.
cells.
We develop two unit sizes for each cell type, 128 and 256 for {·} -s and {·} -b, respectively.
The model is adapted for the sequential MNIST classiﬁcation task, similar to Le et al.
Instead of processing pixel-by-pixel, however, we perform rowby-row processing ( i
e ., the RNN cell receives each row at a time)
The results are summarized in Table 2.
Overall, our approach prunes a substantial amount of parameters in a variety of network models with minimal or no loss in accuracy ( < 1 % ).
Our pruning procedure does not need to be modiﬁed for speciﬁc architectural variations ( e.g ., recurrent connections), indicating that it is indeed versatile and scalable.
Note that prior art that use a saliency criterion based on the weights ( i.e ., magnitude or Hessian based) would require considerable adjustments in their pruning schedules as per changes in the model.
We note of a few challenges in directly comparing against others: different network speciﬁcations, learning policies, datasets and tasks.
Nonetheless, we provide a few comparison points that we found in the literature.
On CIFAR-10 , SVD prunes 97 .
9% of the connections in VGG-like with no loss in accuracy (ours: 97% sparsity) while SWS obtained 93 .
4% sparsity on WRN-16-4 but with a non-negligible loss in accuracy of 2% .
There are a couple of works attempting to prune RNNs ( e.g ., GRU in Narang et al.
(2017) and LSTM in See et al.
Even though these methods are speciﬁcally designed for RNNs , none of them are able to obtain extreme sparsity without substantial loss in accuracy reﬂecting the challenges of pruning RNNs .
To the best of our knowledge, we are the ﬁrst to demonstrate on convolutional, residual and recurrent networks for extreme sparsities without requiring additional hyperparameters or modifying the pruning procedure.
5. U NDERSTANDING WHICH CONNECTIONS ARE BEING PRUNED
So far we have shown that our approach can prune a variety of deep neural network architectures for extreme sparsities without losing much on accuracy.
However, it is not too clear yet which connections are actually being pruned away or whether we are pruning the right ( i.e ., unimportant) ones.
What if we could actually peep through our approach into this inspection?
Consider the ﬁrst layer in LeNet-300-100 parameterized by w l =1 ∈ R 784 × 300 .
This is a layer fully connected to the input where input images are of size 28 × 28 = 784 .
In order to understand which connections are retained, we can visualize the binary connectivity mask for this layer c l =1 , by averaging across columns and then reshaping the vector into 2D matrix ( i.e ., c l =1 ∈ { 0 , 1 } 784 × 300 → R 784 → R 28 × 28 ).
Recall that our method computes c using a minibatch of examples.
In this experiment, we curate the mini-batch of examples of the same class and see which weights are retained for that mini-batch of data.
We repeat this experiment for all classes viii
[Figure]
( i.e ., digits for MNIST and fashion items for Fashion-MNIST ) with varying sparsity levels ¯ κ .
The results are displayed in Figure 2 (see Appendix A for more results).
The results are signiﬁcant; important connections seem to reconstruct either the complete image ( MNIST ) or silhouettes ( Fashion-MNIST ) of input class.
When we use a batch of examples of the digit 0 ( i.e ., the ﬁrst column of MNIST results), for example, the parameters connected to the foreground of the digit 0 survive from pruning while the majority of background is removed.
Also, one can easily determine the identity of items from Fashion-MNIST results.
This clearly indicates that our method indeed prunes the unimportant connections in performing the classiﬁcation task, receiving signals only from the most discriminative part of the input.
This stands in stark contrast to other pruning methods from which carrying out such inspection is not straightforward.
5. E FFECTS OF DATA AND WEIGHT INITIALIZATION
Recall that our connection saliency measure depends on the network weights w as well as the given data D (Section 4.2).
We study the effect of each of these in this section.
Effect of data.
Our connection saliency measure depends on a mini-batch of train examples
D b
(see Algorithm 1).
To study the effect of data, we vary the batch size used to compute the saliency ( |D b | ) and check which connections are being pruned as well as how much performance change this results in on the corresponding sparse network.
We test with LeNet-300-100 to visualize the remaining parameters, and set the sparsity level ¯ κ = 90 .
Note that the batch size used for training remains the same as 100 for all cases.
The results are displayed in Figure 3.
Effect of initialization.
Our approach prunes a network at a stochastic initialization as discussed.
We study the effect of the following initialization methods: 1) RN (random normal), 2) TN (truncated random normal), 3) VS-X (a variance scaling method using Glorot & Bengio (2010)), and 4) VS-H (a variance scaling method He et al.
We test on LeNets and RNNs on MNIST and run 20 sets of experiments by varying the seed for initialization.
We set the sparsity level ¯ κ = 90 , and train with
Adam optimizer (Kingma & Ba (2015)) with learning rate of 0 . 001 without weight decay.
Note that for training VS-X initialization is used in all the cases.
The results are reported in Figure 3.
For all models, VS-H achieves the best performance.
The differences between initializers are marginal on LeNets , however, variance scaling methods indeed turns out to be essential for complex
RNN models.
This effect is signiﬁcant especially for GRU where without variance scaling initialization, the pruned networks are unable to achieve good accuracies, even with different optimizers.
Overall, initializing with a variance scaling method seems crucial to making our saliency measure reliable and model-agnostic.
ix
[Figure] [Figure]
5. F ITTING RANDOM LABELS
[Figure]
To further explore the use cases of SNIP , we run the experiment introduced in Zhang et al.
and check whether the sparse network obtained by SNIP memorizes the dataset.
Speciﬁcally, we train LeNet-5-Caffe for both the reference model and pruned model (with ¯ κ = 99 ) on
MNIST with either true or randomly shufﬂed labels.
To compute the connection sensitivity, always true labels are used.
The results are plotted in Figure 4.
Given true labels, both the reference (red) and pruned (blue) models quickly reach to almost zero training loss.
However, the reference model provided with random labels (green) also reaches to very low training loss, even with an explicit L2 regularizer (purple), indicating that neural networks have enough capacity to memorize completely random data.
In contrast, the model pruned by SNIP (orange) fails to ﬁt the random labels (high training error).
The potential explanation is that the pruned network does not have sufﬁcient capacity to ﬁt the random labels, but it is able to classify MNIST with true labels, reinforcing the signiﬁcance of our saliency criterion.
It is possible that a similar experiment can be done with other pruning methods (Molchanov et al.
(2017)), however, being simple, SNIP enables such exploration much easier.
We provide a further analysis on the effect of varying ¯ κ in Appendix B.
D ISCUSSION AND FUTURE WORK
In this work, we have presented a new approach, SNIP , that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modiﬁcations.
While SNIP results in extremely sparse models, we ﬁnd that our connection sensitivity measure itself is noteworthy in that it diagnoses important connections in the network from a purely untrained network.
We believe that this opens up new possibilities beyond pruning in the topics of understanding of neural network architectures, multi-task transfer learning and structural regularization, to name a few.
In addition to these potential directions, we intend to explore the generalization capabilities of sparse networks.
x
A V ISUALIZING PRUNED PARAMETERS ON ( INVERTED ) ( FASHION -) MNIST
[Figure]
[Figure]
xiii
B F ITTING RANDOM LABELS : VARYING SPARSITY LEVELS
Iteration (×10 3 ) 0.
L o ss ̄κ= ̄κ= ̄κ= ̄κ= ̄κ= ̄κ=
Recall, however, that all pruned models are able to learn to perform the classiﬁcation task without losing much accuracy (see Figure 1).
This potentially indicates that the pruned network does not have sufﬁcient capacity to ﬁt the random labels, but it is capable of performing the classiﬁcation.
C T INY - IMAGENET
Architecture
Model
Sparsity
[Table]
Parameters
Error
[Table]
) ∆
[Table]
AlexNet
[Table]
1 m → 507 k 62 .
[Table] [Table]
+
[Table]
AlexNet
[Table] [Table]
5 m → 849 k 62 .
76 →
[Table]
+2 .
VGG-C
[Table]
5 m → 526 k 56 .
49 → 57 .+0 .
VGG-D
[Table]
2 m → 762 k 56 .
[Table] [Table]
+
[Table]
VGG
[Table]
like 95 .15 . 0 m
[Table] [Table]
[Table] [Table]
+
[Table]
architectures is doubled, following the standard practice for this dataset.
In general, the Tiny-ImageNet classiﬁcation task is considered much more complex than MNIST or CIFAR-10 .
Even on Tiny-ImageNet , however,
SNIP is still able to prune a large amount of parameters with minimal loss in performance.
AlexNet models lose more accuracies than VGG s, which may be attributed to the fact that the ﬁrst convolution stride for AlexNet is set to be 4 (by its design of no pooling) which is too large and could lead to high loss of information when pruned.
xiv