P ROVABLE F ILTER P RUNING FOR E FFICIENT N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a provable, sampling-based approach for generating compact Convolutional Neural Networks (CNNs) by identifying and removing redundant ﬁlters from an over-parameterized network.
Our algorithm uses a small batch of input data points to assign a saliency score to each ﬁlter and constructs an importance sampling distribution where ﬁlters that highly affect the output are sampled with correspondingly high probability.
In contrast to existing ﬁlter pruning approaches, our method is simultaneously data-informed, exhibits provable guarantees on the size and performance of the pruned network, and is widely applicable to varying network architectures and data sets.
Our analytical bounds bridge the notions of compressibility and importance of network structures, which gives rise to a fully-automated procedure for identifying and preserving ﬁlters in layers that are essential to the network’s performance.
Our experimental evaluations on popular architectures and data sets show that our algorithm consistently generates sparser and more efﬁcient models than those constructed by existing ﬁlter pruning approaches.
I NTRODUCTION
Despite widespread empirical success, modern networks with millions of parameters require excessive amounts of memory and computational resources to store and conduct inference.
These stringent requirements make it challenging and prohibitive to deploy large neural networks on resource-limited platforms.
A popular approach to alleviate these practical concerns is to utilize a pruning algorithm to remove redundant parameters from the original, over-parameterized network.
The objective of network pruning is to generate a sparse, efﬁcient model that achieves minimal loss in predictive power relative to that of the original network.
A common practice to obtain small, efﬁcient network architectures is to train an over-parameterized network, prune it by removing the least signiﬁcant weights, and re-train the pruned network (Gale et al., 2019; Frankle & Carbin, 2018; Han et al., 2015).
This prune-retrain cycle is often repeated iteratively until the network cannot be pruned any further without incurring a signiﬁcant loss in predictive accuracy relative to that of the original model.
The computational complexity of this iterative procedure depends greatly on the effectiveness of the pruning algorithm used in identifying and preserving the essential structures of the original network.
To this end, a diverse set of smart pruning strategies have been proposed in order to generate compact, accurate neural network models in a computationally efﬁcient way.
However, modern pruning approaches 1 are generally based on heuristics (Han et al., 2015; Ullrich et al., 2017; He et al., 2018; Luo et al., 2017; Li et al., 2016; Lee et al., 2018; Yu et al., 2017a) that lack guarantees on the size and performance of the pruned network, require cumbersome ablation studies (Li et al., 2016;
He et al., 2018) or manual hyper-parameter tuning (Luo et al., 2017), or heavily rely on assumptions such that parameters with large weight magnitudes are more important – which does not hold in general (Ye et al., 2018; Li et al., 2016; Yu et al., 2017a; Han et al., 2015).
In this paper, we introduce a data-informed algorithm for pruning redundant ﬁlters in Convolutional
Neural Networks while incurring minimal loss in the network’s accuracy (see Fig.
1 for an overview).
At the heart of our method lies a novel deﬁnition of ﬁlter importance, i.e., ﬁlter sensitivity , that is computed by using a small batch of input points.
We prove that by empirically evaluating the relative contribution of each ﬁlter to the output of the layer, we can accurately capture its importance
[Figure] [Figure]
with respect to the other ﬁlters in the network.
We show that sampling ﬁlters with probabilities proportional to their sensitivities leads to an importance sampling scheme with low variance, which enables us to establish rigorous theoretical guarantees on the size and performance of the resulting pruned network.
Our analysis helps bridge the notions of compressibility and importance of each network layer: layers that are more compressible are less important for preserving the output of the original network, and vice-versa.
Hence, we obtain and introduce a fully-automated sample size allocation procedure for properly identifying and preserving critical network structures as a corollary.
Unlike weight pruning approaches that lead to irregular sparsity patterns – requiring specialized libraries or hardware to enable computational speedups – our approach compresses the original network to a slimmer subnetwork by pruning ﬁlters, which enables accelerated inference with any off-the-shelf deep learning library and hardware.
We evaluate and compare the effectiveness of our approach in pruning a diverse set of network architectures trained on real-world data sets.
Our empirical results show that our approach generates sparser and more efﬁcient models with minimal loss in accuracy when compared to those generated by state-of-the-art ﬁlter pruning approaches.
S AMPLING - BASED F ILTER P RUNING
In this section, we introduce the network pruning problem and outline our sampling-based ﬁlter pruning procedure and its theoretical properties.
We extend the notion of empirical sensitivity (Baykal et al., 2018) to quantify the importance of each ﬁlter using a small set of input points.
We show that our importance criterion enables us to construct a low-variance importance sampling distribution over the ﬁlters in each layer.
We conclude by showing that our approach can eliminate a large fraction of ﬁlters while ensuring that the output of each layer is approximately preserved.
2. P RELIMINARIES
Consider a trained L layer network with parameters θ
=
( W 1 , . . .
, W L ) , where W (cid:96) denotes the 4 -dimensional tensor in layer (
cid:96) ∈
[ L ] , W (cid:96)j ﬁlter j ∈
[ η (cid:96) ] , and η (cid:96) the number of ﬁlters in layer (cid:96) .
Moreover, let W (cid:96) +1 :
j be channel j of tensor W (cid:96) +1 that corresponds to ﬁlter W (cid:96)j .
We let X ⊂ R d and
Y ⊂ R k denote the input and output space, respectively.
The marginal distribution over the input space is given by D .
For an input x ∈ X to the network, we let z (cid:96) ( x ) and a (cid:96) ( x ) =
φ ( z (cid:96) ( x )) denote the pre-activation and activation of layer (cid:96) , where φ is the activation function (applied entry-wise).
The j th feature map of layer (cid:96) is given by a (cid:96)j ( x ) =
φ ( z (cid:96)j ( x ))
(see Fig. 1).
For a given input x ∈ X , the output of the neural network with parameters θ is given by f θ ( x ) .
Our overarching goal is to prune ﬁlters from each layer (cid:96) ∈
[ L ] by random sampling to generate a compact reparameterization of θ , ˆ θ =
( ˆ W 1 , . . .
, ˆ W L ) , where the number of ﬁlters in the pruned weight tensor ˆ W (cid:96) is a small fraction of the number of ﬁlters in the original (uncompressed) tensor
W (cid:96) .
Let size( θ ) denote the total number of parameters in the network, i.e., the sum of the number of weights over each W (cid:96) ∈
( W 1 , . . .
, W L ) .
Pruning Objective For a given ε, δ ∈ (0 , 1) , our objective is to generate a compressed network with parameters ˆ θ such that size(ˆ θ ) (cid:28) size( θ ) and P x ∼D , ˆ θ ( f ˆ θ ( x ) ∈ (1 ± ε )
f θ ( x )) ≥ 1 − δ, where f ˆ θ ( x ) ∈ (1 ± ε )
f θ ( x ) denotes an entry-wise guarantee over the output neurons
f ˆ θ ( x ) , f θ ( x ) ∈ Y .
2. S AMPLING - BASED P RUNING
[Table]
[Table] [Table]
] 3: m (cid:96) ← (cid:6) (6 + 2 ε ) S (cid:96) K log(4 η ∗ /δ )
ε − 2 (cid:7) 4: ˆ W (cid:96) +1 ← [0 , . . . , 0] {same dimensions as W (cid:96) +1 } 5: for k ∈ [ m (cid:96) ] do 6: c ( k ) ← random draw from p (cid:96) =
( p (cid:96) 1 , . . .
, p (cid:96)η (cid:96) ) 7: ˆ W (cid:96) +1 :
c ( k ) ← ˆ W (cid:96) +1 :
c ( k ) + W (cid:96) +1 :
c ( k ) / m (cid:96) p (cid:96)c ( k ) 8: end for 9: return ˆ W (cid:96) +
=
[ ˆ W (cid:96) +1 :1 , . .
, ˆ W (cid:96) +1 :
η (cid:96) ] ;
Our sampling-based ﬁlter pruning algorithm for an arbitrary layer (cid:96) ∈
[ L ] is depicted as
Alg. 1.
The sampling procedure takes as input the set of η (cid:96) channels in layer (cid:96) + 1 that constitute the weight tensor W (cid:96) +1 , i.e., W (cid:96) +
=
[ W (cid:96) +1 :1 , . . .
, W (cid:96) +1 :
η (cid:96) ] as well as the desired relative error and failure probability, ε, δ ∈ (0 , 1) , respectively.
In Line 2 we construct the importance sampling distribution over the feature maps corresponding to the channels by leveraging the empirical sensitivity of each feature map j ∈ [ η (cid:96) ] as deﬁned in (1) and explained in detail in the following subsections.
Note that we initially prune channels from W (cid:96) +1 , but as we prune channels from W (cid:96) +1 we can simultaneously prune the corresponding ﬁlters in W (cid:96) .
We subsequently set the sample complexity m (cid:96) as a function of the given error ( ε ) and failure probability ( δ ) parameters in order to ensure that, after the pruning (i.e., sampling) procedure, the approximate output – with respect to the sampled channels ˆ
W (cid:96) +1 – of the layer will approximate the true output of the layer – with respect to the original tensor – up to a multiplicative factor of (1 ± ε ) , with probability at least 1 − δ .
Intuitively, more samples are required to achieve a low speciﬁed error ε with low failure probability δ , and vice-versa.
We then proceed to sample m l times with replacement according to distribution p (cid:96) ( Lines 5-8) and reweigh each sample by a factor that is inversely proportional to its sample probability to obtain an unbiased estimator for the layer’s output (see below).
The unsampled channels in W (cid:96) +1 – and the corresponding ﬁlters in W (cid:96) – are subsequently discarded, leading to a reduction in the layer’s size.
2. A T IGHTLY -C ONCENTRATED E STIMATOR
We now turn our attention to analyzing the inﬂuence of the sampled channels ˆ W (cid:96) +1 (as in Alg. 1) on layer (cid:96) + 1 .
For ease of presentation, we will henceforth assume that the layer is linear 2 and will omit explicit references to the input x whenever appropriate.
Note that the true pre-activation of layer (cid:96) +is given by z (cid:96) +
= W (cid:96) +1 a (cid:96) , and the approximate pre-activation with respect to ˆ W (cid:96) +1 is given by ˆ z (cid:96) +
= ˆ W (cid:96) +1 a (cid:96) .
By construction of ˆ W (cid:96) +1 in Alg. 1, we equivalently have for each entry i ∈
[ η (cid:96) +1 ]
ˆ z (cid:96) +1 i
=m m (cid:88) k
=
Y ik , where Y ik = W (cid:96) +1 ic ( k ) a (cid:96)c ( k ) p
(cid:96)c ( k ) , c ( k ) ∼ p ∀ k.
By reweighing our samples, we obtain an unbiased estimator for each entry i of the true pre-activation output, i.e., E [ˆ z (cid:96) +1 i ]
=
z (cid:96) +1 i – which follows by the linearity of expectation and the fact that
E [ Y ik ] =
z (cid:96) +1 i for each k ∈ [ m ] –, and so we have for the entire vector E
ˆ W (cid:96) +
[ˆ z (cid:96) +1 ]
=
z (cid:96) +1 .
So far, we have shown that in expectation, our channel sampling procedure incurs zero error owing to its unbiasedness.
However, our objective is to obtain a high probability bound on the entry-wise deviation (cid:12)(cid:12)
ˆ z (cid:96) +
i − z
(cid:96) +1 i (cid:12)(cid:12) for each entry
i , which implies that we have to show that our estimator ˆ z (cid:96) +1 i is highly concentrated around its mean z (cid:96) +1 i .
To do so, we leverage the following standard result.
Theorem 1 (Bernstein’s inequality (Vershynin, 2016)) .
Let Y 1 , . . .
, Y m be a sequence of m i.i.d.
random variables satisfying max k ∈ [ m ] | Y k − E
[ Y k ] | ≤ R , and let Y =
(cid:80) mk
=1 Y k denote their sum.
Then, for every ε ≥ 0 , δ ∈ (0 , 1) , we have that P ( | Y / m − E
[ Y k ] | ≥ ε E
[ Y k ]) ≤ δ for
[Equation]
Letting i ∈
[ η (cid:96) +1 ] be arbitrary and applying Theorem 1 to the mean of the random variables ( Y ik ) k ∈ [ m ] , i.e., to ˆ z (cid:96) +
i , we observe that the number of samples required for a sufﬁciently high concentration around the mean is highly dependent on the magnitude and variance of the random variables ( Y ik ) k .
By deﬁnition of Y ik , observe that these expressions are explicit functions of the sampling distribution p (cid:96) .
Thus, to minimize 3 the number of samples required to achieve high concentration we require a judiciously deﬁned sampling distribution that simultaneously minimizes both R i and Var( Y ik ) .
For example, the naive approach of uniform sampling, i.e., p (cid:96)j = 1 / η (cid:96) for each j ∈ [ η (cid:96) ] also leads to an unbiased estimator, however, for uniform sampling we have
Var( Y ik )
≈ η (cid:96)
E
[ Y ik ] 2 and R i ≈ η (cid:96) max k ( w (cid:96) +1 ik a (cid:96) k ) and so Var( Y ik ) , R ∈ Ω( η (cid:96) ) in the general case, leading to a linear sampling complexity m ∈ Ω( η (cid:96) ) by Theorem 1.
2. E MPIRICAL S ENSITIVITY (ES)
To obtain a better sampling distribution, we extend the notion of Empirical Sensitivity (ES) introduced by (Baykal et al., 2018) to prune channels.
Speciﬁcally, for W (cid:96) +1 ≥
(the generalization can be found in Appendix B) we let the sensitivity s (cid:96)j of feature map j in (cid:96) be deﬁned as
[Equation]
where S is a set of t independent and identically (i.i.d.)
points drawn from D .
Intuitively, the sensitivity of feature map j ∈ [ η (cid:96) ] is the maximum (over i ∈
[ η (cid:96) +1 ] ) relative impact that feature map j had on any pre-activation in the next layer z (cid:96) +1 i .
We then deﬁne the probability of sampling each channel as in Alg.
1: j ∈ [ η (cid:96) ] as p j = s (cid:96)j /S (cid:96) , where S (cid:96) =
(cid:80) j s (cid:96)j is the sum of sensitivities.
Under a mild assumption on the distribution – that is satisﬁed by a wide class of distributions, such as the Uniform, Gaussian, Exponential, among others – of activations (Asm. 1 in Sec.
B of the supplementary), ES enables us to leverage the inherent stochasticity in the draw x ∼ D and establish (see Lemmas 5, 6, and 7 in Sec.
B)
that with high probability (over the randomness in S and x ) that
Var( Y ik ) ∈
Θ( S E [ Y ik ] 2 ) and R ∈ Θ( S E
[ Y ik ]) ∀ i ∈
[ η (cid:96) +1 ] and that the sampling complexity is given by m ∈ Θ( S log(2 /δ )
ε − 2 ) by Theorem 1.
We note that ES does not require knowledge of the data distribution D and is easy to compute in practice by randomly drawing a small set of input points S from the validation set and passing the points in S through the network.
This stands in contrast with the sensitivity framework used in state-of-the-art coresets constructions (Braverman et al., 2016; Bachem et al., 2017), where the sensitivity is deﬁned to be with respect to the supremum over all x ∈ supp( D ) in (1) instead of a maximum over x ∈ S .
As also noted by Baykal et al.
, ES inherently considers data points that are likely to be drawn from the distribution D in practice, leading to a more practical and informed sampling distribution with lower sampling complexity.
Our insights from the discussion in this section culminate in the core theorem below (Thm. 2), which establishes that the pruned channels ˆ W (cid:96) +1 (corresponding to pruned ﬁlters in W (cid:96) ) generated by
Alg. 1 is such that the output of layer (cid:96) + 1 is well-approximated for each entry.
Theorem 2.
Let ε, δ ∈ (0 , 1) ,
(cid:96) ∈
[ L ] , and let S be a set of Θ(log ( η ∗ /δ ))
i.i.d.
samples drawn from
D .
Then, ˆ W (cid:96) +1 contains at most O ( S (cid:96) log( η ∗ /δ )
ε − 2 )) channels and for x ∼ D , with probability at least 1 − δ , we have ˆ z (cid:96) +1 ∈ (1 ± ε ) z
(cid:96)
-wise), where η ∗ = max (
cid:96) ∈
[ L ] η (cid:96)
Theorem 2 can be generalized to hold for all weights and applied iteratively to obtain layer-wise approximation guarantees for the output of each layer.
The resulting layer-wise error can then be propagated through the layers to obtain a guarantee on the ﬁnal output of the compressed network.
In particular, applying the error propagation bounds of Baykal et al.
(2018), we establish our main compression theorem below.
The proofs and additional details can be found in the appendix (Sec. B).
Theorem 3.
Let ε, δ ∈ (0 , 1) be arbitrary, let S ⊂
X denote the set of (cid:100) K (cid:48) log (4 η/δ ) (cid:101) i.i.d.
points drawn from D , and suppose we are given a network with parameters θ
= ( W 1 , . .
, W L ) .
Consider the set of parameters ˆ θ =
( ˆ W 1 , . . .
, ˆ W L ) generated by pruning channels of θ according to Alg. 2 for each (cid:96) ∈
[ L ] .
Then, ˆ θ satisﬁes P ˆ θ,x ∼D (cid:0)
f
ˆ θ ( x ) ∈ (1 ± ε ) f
θ ( x ) (cid:1) ≥ 1 − δ, and the number of ﬁlters in ˆ θ is bounded by O (cid:16)(cid:80)
[Figure]
R ELATIVE L AYER I MPORTANCE
[Figure]
In the previous sections, we established the sampling complexity of our ﬁlter pruning scheme for any user-speciﬁed ε and δ .
However, in practice, it is more common for the practitioner to specify the desired pruning ratio, which speciﬁes the resulting size of the pruned model.
Given this sampling budget, a practical question that arises is how to optimally ration the sampling budget across the network’s layers to minimize the error of the pruned model.
A naive approach would be to uniformly allocate the sampling budget N so that the same ratio of ﬁlters is kept in each layer.
However, this allocation scheme implicitly assumes that each layer of the network is of equal importance to retaining the output, which is virtually never the case in practice, as exempliﬁed by Fig. 2(a).
It turns out that our analytical bounds on the sample complexity per layer ( m (cid:96) in Alg. 1) naturally capture the importance of each layer.
The key insight lies in bridging the compressibility and importance of each layer: if a layer is not very important, i.e., it does not heavily inﬂuence output of the network, then we expect it to be highly compressible, and vice-versa.
This intuition is precisely captured by our sampling complexity bounds that quantify the difﬁculty of a layer’s compressibility.
We leverage this insight to formulate a simple binary search procedure for judiciously allocating the sampling budget N as follows.
Let δ ∈ (0 , 1) be user-speciﬁed, pick a random ε > 0 , and compute the sampling complexity m (cid:96) as in Alg.
1 together with the resulting layer size n (cid:96) .
If (cid:80) (cid:96) n (cid:96) = N , we are done, otherwise, continue searching for an appropriate ε on a smaller interval depending on whether (cid:80) (cid:96) n (cid:96) is greater or less than N .
The allocation generated by this procedure (see Fig. 2(b) for example) ensures that the maximum layer-wise error incurred by pruning is at most ε .
R ESULTS
In this section, we evaluate and compare our algorithm’s performance to that of state-of-the-art pruning schemes in generating compact networks that retain the predictive accuracy of the original model.
Our evaluations show that our approach generates signiﬁcantly smaller and more efﬁcient models compared to those generated by competing methods.
Our results demonstrate the practicality and wide-spread applicability of our proposed approach: across all of our experiments, our algorithm took on the order of a minute to prune a given network 4 , required no manual tuning of its hyperparameters, and performed consistently well across a diverse set of pruning scenarios.
Additional results, comparisons, and experimental details can be found in Sec.
E of the appendix.
4. E XPERIMENTAL S ETUP
We compare our algorithm to that of the following ﬁlter pruning algorithms that we implemented and ran alongside our algorithm:
Filter Thresholding (FT, Li et al.
SoftNet (He et al., 2018), and ThiNet (Luo et al., 2017).
We note that FT and SoftNet are both (weight) magnitude-based ﬁlter pruning algorithms, and this class of pruning schemes has recently been reported to be state-of-theart (Gale et al., 2019; Pitas et al., 2019; Yu et al., 2018)
(see Sec.
E.1 of the appendix for details of the compared methods).
Additional comparisons to other state-of-the-art channel and ﬁlter pruning methods can be found in Table 8 in Appendix E.6.
Our algorithm only requires two inputs in practice: the desired pruning ratio (PR) and failure probability δ ∈ (0 , 1) , since the number of samples in each layer is automatically assigned by our allocation procedure described in Sec.
Following the conventional data partioning ratio, we reserve 90% of the training data set for training and the remaining 10% for the validation set (Lee et al., 2018).
Method
Err.
PR
L e N e t300 -
Unpruned 1.
Ours +0.
FT +0.
SoftNet +0.
ThiNet +10.
L e N e tUnpruned 0.
Ours +0.
FT +
[Table]
For each scenario, we prune the original (pre-trained) network with a target prune ratio using the respective pruning algorithm and ﬁne-tune the network by retraining for a speciﬁed number of epochs.
We repeat this procedure iteratively to obtain various target prune ratios and report the percentage of parameters pruned (PR) and the percent reduction in FLOPS (FR) for each target prune ratio.
The target prune ratio follows a hyperharmonic sequence where the i th PR is determined by 1 − 1 / ( i +1)
α , where α is an experiment-dependent tuning parameter.
We conduct the prune-retrain cycle for a range of 10 − 20 target prune ratios, and report the highest PR and FR for which the compressed network achieves commensurate accuracy, i.e., when the pruned model’s test accuracy is within 0.5% of the original model.
The quantities reported are averaged over 3 trained models for each scenario, unless stated otherwise.
The full details of our experimental setup and the hyper-parameters used can be found in the appendix (Sec. E).
4. L E N ET ARCHITECTURES ON MNIST
As our ﬁrst experiment, we evaluated the performance of our pruning algorithm and the comparison methods on LeNet300-100 (LeCun et al., 1998), a fullyconnected network with two hidden layers of size 300 and 100 hidden units, respectively, and its convolutional counterpart, LeNet-5 (LeCun et al., 1998), which consists of two convolutional layers and two fully-connected layers.
Both networks were trained on MNIST using the hyper-parameters speciﬁed in Sec.
E.
Table 1 depicts the performance of each pruning algorithm in attaining the sparsest possible network that achieves commensurate accuracy for the LeNet architectures.
In both scenarios, our algorithm generates signiﬁcantly sparser networks compared to those generated by the competing ﬁlter pruning approaches.
In fact, the pruned LeNet-5 model generated by our algorithm by removing ﬁlters achieves a prune ratio of ≈ 90% , which is even competitive with the accuracy of the sparse models generated by state-of-the-art weight pruning algorithms (Lee et al., 2018) 5 .
In addition to evaluating the sparsity of the generated models subject to the commensurate accuracy constraint, we also investigated the performance of the pruning algorithms for extreme (i.e., around 5% ) pruning ratios (see Fig. 3(a)).
Fig.
3(a)
we see that our algorithm’s performance relative to those of competing algorithms is strictly better for a wide range of target prune ratios.
For LeNet-5 Fig.
3(a) shows that our algorithm’s favorable performance is even more pronounced at extreme sparsity levels (at ≈ 95% prune ratio).
4. C ONVOLUTIONAL N EURAL N ETWORKS ON CIFAR-
Next, we evaluated the performance of each pruning algorithm on signiﬁcantly larger and deeper Convolutional Neural Networks trained on the CIFAR-10 data set:
VGG16 with BatchNorm (Simonyan & Zisserman, 2014a), ResNet20, ResNet56, ResNet110 (He et al., 2016), DenseNet22 (Huang et al., 2017), and WideResNet16-8 (Zagoruyko & Komodakis, 2016).
For CIFAR-10 experiments, we use the standard data augmentation techniques: padding 4 pixels on each side, random crop to 32x pixels, and random horizontal ﬂip.
Our results are summarized in Table 2 and Figure 3.
Similar to the results reported in Table 1 in the previous subsection, Table 2 shows that our method is able to achieve the most sparse model with minimal loss in predictive power relative to the original network.
Furthermore, by inspecting the values reported for Flops Retained (FR), we observe that the models generated by our approach are not only more sparse in terms of the number of total parameters, but also more efﬁcient in terms of the inference time complexity.
[Figure]
Fig.
3 depicts the performance of the evaluated algorithms for various levels of prune ratios.
Once again, we see the consistently better performance of our algorithm in generating sparser models that approximately match or exceed the predictive accuracy of the original uncompressed network.
In addition, Table 8 (see Appendix E.6 for more details) provides further comparisons to state-of-the-art ﬁlter pruning methods where we compare the performance of our approach to the results for various ResNets and VGG16 reported directly in the respective papers.
The comparisons in Table 8 reafﬁrm that our algorithm can consistently generate simultaneously sparser and more accurate networks compared to competing methods. In view of our results from the previous subsection, the results shown in Table 2, Fig. 3, and Tablehighlight the versatility and broad applicability of our method, and seem to suggest that our approach fares better relative to the compared algorithms on more challenging pruning tasks that involve large-scale networks.
We suspect that these favorable properties are explained by the data-informed evaluations of ﬁlter importance and the corresponding theoretical guarantees of our algorithm – which enable robustness to variations in network architecture and data distribution.
[Table]
pruning algorithm generates smaller and more efﬁcient networks with minimal loss in accuracy, when compared to competing approaches
4. C ONVOLUTIONAL N EURAL N ETWORKS ON I MAGE N ET
We consider pruning convolutional neural networks of varying size – ResNet18, ResNet50, and
ResNet101 – trained on the ImageNet (Russakovsky et al., 2015) data set.
For this dataset, we considered two scenarios: (i) iterative pruning without retraining and (ii) iterative prune-retrain with a limited amount of iterations given the resource-intensive nature of the experiments.
The results of these experiments are reported in Section E.4 of the appendix.
Our results on the ImageNet data set follow a similar trend as those in the previous subsections and indicate that our method readily scales to larger data sets without the need of manual hyperparameter tuning.
This improves upon existing approaches (such as those in He et al.
; Li et al.
that generally require tedious, task-speciﬁc intervention or manual parameter tuning by the practitioner.
4. A PPLICATION TO R EAL - TIME R EGRESSION T ASKS
[Figure] Real-time applications of neural networks, such as their use in autonomous driving scenarios, require network models that are not only highly accurate, but also highly efﬁcient, i.e., fast, when it comes to inference time complexity (Amini et al., 2018).
Model compression, and in particular, ﬁlter pruning has potential to generate compressed networks capable of achieving both of these objectives.
To evaluate and compare the effectiveness of our method on pruning networks intended for regression tasks and real-time systems, we evaluated the various pruning approaches on the DeepKnight network (Amini et al., 2018), a regression network deployed on an autonomous vehicle in real time to predict the steering angle of the human driver (see E.5 in appendix for experimental details).
Fig.
4 depicts the results of our evaluations and comparisons on the DeepKnight network without the ﬁne-tuning step.
We omitted the iterative ﬁne-tuning step for this scenario and instead evaluated the test loss for various prune ratios because (i) the evaluated algorithms were able to generate highly accurate models without the retraining step and (ii) in order to evaluate and compare the performance of solely the core pruning procedure.
Similar to the results obtained in the preceding pruning scenarios, Fig.
4 shows that our method consistently outperforms competing approaches for all of the speciﬁed prune ratios.
4. D ISCUSSION
In addition to the favorable empirical results of our algorithm, our approach exhibits various advantages over competing methods that manifest themselves in our empirical evaluations.
For one, our algorithm does not require any additional hyper-parameters other than the pruning ratio and the desired failure probability.
Given these sole two parameters, our approach automatically allocates the number of ﬁlters to sample for each layer.
This alleviates the need to perform time-intensive ablation studies (He et al., 2018) and to resort to uninformed (i.e., uniform) sample allocation strategies, e.g., removing the same percentage of ﬁlters in each layer (Li et al., 2016), which fails to consider the non-uniform inﬂuence of each layer on the network’s output (see Sec. 3).
Moreover, our algorithm is simple-to-implement and computationally efﬁcient both in theory and practice: the computational complexity is dominated by the |S| forward passes required to compute the sensitivities ( |S| ≤ 256 in practical settings) and in practice, our algorithm takes on the order of a minute to prune the network.
C ONCLUSION
We presented – to the best of our knowledge – the ﬁrst ﬁlter pruning algorithm that generates a pruned network with theoretical guarantees on the size and performance of the generated network.
Our method is data-informed, simple-to-implement, and efﬁcient both in theory and practice.
Our approach can also be broadly applied to varying network architectures and data sets with minimal hyper-parameter tuning necessary.
This stands in contrast to existing ﬁlter pruning approaches that are generally data-oblivious, rely on heuristics for evaluating the parameter importance, or require tedious hyper-parameter tuning.
Our empirical evaluations on popular network architectures and data sets reafﬁrm the favorable theoretical properties of our method and demonstrate its practical effectiveness in obtaining sparse, efﬁcient networks.
We envision that besides its immediate use for pruning state-of-the-art models, our approach can also be used as a sub-procedure in other deep learning applications, e.g., for identifying winning lottery tickets (Frankle & Carbin, 2018) and for efﬁcient architecture search (Liu et al., 2018).
A R ELATED W ORK
General network compression The need to tame the excessive storage requirements and costly inference associated with large, over-parameterized networks has led to a rich body of work in network pruning and compression.
These approaches range from those inspired by classical tensor decompositions (Yu et al., 2017b; Jaderberg et al., 2014; Denton et al., 2014), and random projections and hashing (Arora et al., 2018;
Ullrich et al., 2017; Chen et al., 2015; Weinberger et al., 2009;
Shi et al., 2009) that compress a pre-trained network, to those approaches that enable sparsity by embedding sparsity as an objective directly in the training process (Ioannou et al., 2015; Alvarez & Salzmann, 2017) or exploit tensor structure to induce sparsity (Choromanska et al., 2016; Zhao et al., 2017).
Overall, the predominant drawback of these methods is that they require laborious hyperparameter tuning, lack rigorous theoretical guarantees on the size and performance of the resulting compressed network, and/or conduct compression in a data oblivious way.
Weight-based pruning A large subset of modern pruning algorithms fall under the general approach of pruning individual weights of the network by assigning each weight a saliency score, e.g., its magnitude (Han et al., 2015), and subsequently inducing sparsity by deterministically removing those weights below a certain saliency score threshold (Guo et al., 2016; Han et al., 2015; Lee et al., 2018;
LeCun et al., 1990).
These approaches are heuristics that do not provide any theoretical performance guarantees and generally require – with the exception of (Lee et al., 2018) – computationally expensive train-prune-retrain cycles and tedious hyper-parameter tuning.
Unlike our approach that enables accelerated inference (i.e., reduction in FLOPS) on any hardware and with any deep learning library by generating a smaller subnetwork, weight-based pruning generates a model with non-structured sparsity that requires specialized hardware and sparse linear algebra libraries in order to speed up inference.
Neuron pruning Pruning entire neurons in FNNs and ﬁlters in CNNs is particularly appealing as it shrinks the network into its slimmer counterpart, which leads to alleviated storage requirements and improved inference-time performance on any hardware.
Similar to the weight-based approaches, approaches in this domain assign an importance score to each neuron or ﬁlter and remove those with a score below a certain threshold (He et al., 2018;
Li et al., 2016; Yu et al., 2017a).
These approaches generally take the (cid:96) p norm –with p = { 1 , 2 } as popular choices– of the ﬁlters to assign ﬁlter importance and subsequently prune unimportant ﬁlers.
These methods are data-oblivious heuristics that heavily rely on the assumption that ﬁlters with large weight magnitudes are more important, which may not hold in general (Ye et al., 2018).
In general, prior work on neuron and ﬁlter pruning has focused on approaches that lack theoretical guarantees and a principled approach to allocating the sampling budget across layers, requiring tedious ablation studies or settling for naive uniform allocation across the layers.
In contrast to prior approaches, our algorithm assigns data-informed saliency scores to ﬁlters, guarantees an error bound, and leverages our theoretical error bounds to automatically identify important layers and allocate the user-speciﬁed sampling budget (i.e., pruning ratio) across the layers.
Our work is most similar to that of (Baykal et al., 2018; 2019), which proposed an weight pruning algorithm with provable guarantees that samples weights of the network in accordance to an empirical notion of parameter importance.
The main drawback of their approach is the limited applicability to only fully-connected networks, and the lack of inference-time acceleration due to non-structured sparsity caused by removing individual weights.
Our method is also sampling-based and relies on a data-informed notion of importance, however, unlike (Baykal et al., 2018; 2019), our approach can be applied to both FNNs and CNNs and generates sparse, efﬁcient subnetworks that accelerate inference.
B A LGORITHMIC AND A NALYTICAL D ETAILS
Algorithm 2 is the full algorithm for pruning features, i.e., neurons in fully-connected layers and channels in convolutional layers.
For notational simplicity, we will derive our theoretical results for linear layers, i.e., neuron pruning.
We remind the reader that this result also applies to CNNs by taking channels of a weight tensor in place of neurons.
The pseudocode is organized for clarity of exposition rather than computational efﬁciency.
Recall that θ is the full parameter set of the net, where W (cid:96) ∈ R η (cid:96) × η (cid:96) +1 is the weight matrix between layers (cid:96) − 1 and and (cid:96) .
W (cid:96)k refers to the k th neuron of W (cid:96) .
Input: θ : trained net; (cid:96) ∈
[ L ] : layer; S ⊂ supp ( D ) : sample of inputs; ε ∈ (0 , 1) : accuracy; δ ∈ (0 , 1) : failure probability
Output: ˆ W (cid:96) : ﬁlter-reduced weight tensor for layer (cid:96) ; ˆ W (cid:96) +1 : channel reduced, weight tensor for layer (cid:96) +1: for j ∈ [ η (cid:96) ] do 2: for i ∈
[ η (cid:96) +1 ] and x ∈ S do 3:
I + ← { j ∈ [ η (cid:96) ] : w (cid:96) +1 ij a (cid:96)j ( x ) ≥ 0 } 4:
I − ←
[ η (cid:96) ] \ I + 5: g (cid:96) +1 ij ( x )
← max I ∈
{ I + ,I − } w (cid:96) +1 ij a (cid:96)j ( x ) (cid:80)
k ∈ I
w (cid:96) +1 ik a (cid:96)k ( x ) 6: end for 7: s (cid:96)j ← max x ∈S max i ∈
[ η (cid:96) +1 ] g (cid:96)
+1 ij ( x ) 8: end for 9: S (cid:96) ← (cid:80) j ∈ [ η (cid:96) ] s (cid:96)j 10: for j ∈ [ η (cid:96) ] do 11: p (cid:96)j ← s (cid:96)j /S (cid:96) 12: end for 13: K ← value from Assumption14: m ← (cid:6) (
2 ε ) S (cid:96) K log(
1 /δ )
ε
15: H ← distribution on [ η (cid:96) ] assigning probability p (cid:96)j to index j 16:
ˆ W (cid:96) ← (0 , . .
. , 0) {same dimensions as W (cid:96) } 17: ˆ W (cid:96) +1 ← (0 , . .
. , 0) {same dimensions as W (cid:96) +1 } 18: for k ∈ [ m ] do 19: c ( k ) ← random draw from H 20: ˆ W (cid:96)c (
← W (cid:96)c ( k ) {no reweighing or considering multiplicity of drawing index c ( k ) multiple times} 21: ˆ W
: c
W
: c c c for of pre activation in layer
+ 1 } 22: end for 23: return ˆ W
=
W W
W
=
W
Recall that z (cid:96) +1 i ( x ) denotes the pre-activation of the i th neuron in layer (cid:96) + 1 given input x , and the activation a (cid:96)j ( x ) =
max { 0 , z (cid:96)j ( x ) } .
Deﬁnition 1 (Edge Sensitivity (Baykal et al., 2018)) .
Fixing a layer (
cid:96) ∈
[ L ] , let w (cid:96) +1 ij be the weight of edge
( j, i ) ∈
[ η (cid:96) ]
× [ η (cid:96) +1 ] .
The empirical sensitivity of weight entry w (cid:96) +1 ij with respect to input x ∈ X is deﬁned to be
[Equation]
where I + = { j ∈ [ η (cid:96) ] : w (cid:96) +1 ij a (cid:96)j ( x ) ≥ 0 }
and I − =
[ η (cid:96) ] \ I + denote the set of positive and negative edges, respectively.
Algorithm 2 uses empirical sensitivity to compute the sensitivity of neurons on Lines 9-12.
Deﬁnition 2 (Neuron Sensitivity) .
The sensitivity of a neuron j ∈ [ η (cid:96) ] in layer (cid:96) is deﬁned as
[Equation]
[Equation]
In this section, we prove that Algorithm 2 yields a good approximation of the original net.
We begin with a mild assumption to ensure that the distribution of our input is not pathological.
Assumption 1.
There exist universal constants K, K (cid:48) > 0 such that for any layer (cid:96) and all j ∈ [ η (cid:96) ] , the CDF of the random variable max i ∈
[ η (cid:96) +1 ] g (cid:96) +1 ij ( x ) for x ∼ D , denoted by F j ( · ) , satisﬁes
[Equation]
where M j = min { y ∈ [0 , 1] : F j ( y )
= 1 } .
Note that the analysis is carried out for the positive and negative elements of W (cid:96) +1 separately, which is also considered in the deﬁnition of sensitivity (Def. 1).
For ease of exposition, we will thus assume that throughout the section W (cid:96) +1 ≥ 0 (element-wise), i.e., I + =
[ η (cid:96) ] , and derive the results for this case.
However, we note that we could equivalently assume W (cid:96) +1 ≤ 0 and the analysis would hold regardless.
By considering both the positive and negative parts of W (cid:96) +1 in Def.
1 we can carry out the analysis for weight tensors with positive and negative elements.
Theorem 2.
Let ε, δ ∈ (0 , 1) ,
(cid:96) ∈
[ L ] , and let S be a set of Θ(log ( η ∗ /δ ))
i.i.d.
samples drawn from
D .
Then, ˆ W (cid:96) +1 contains at most O ( S (cid:96) log( η ∗ /δ )
ε − 2 )) channels and for x ∼ D , with probability at least 1 − δ , we have ˆ z (cid:96) +1 ∈ (1 ± ε ) z
(cid:96) +1 (entry-wise), where η ∗ = max (cid:96) ∈
[ L ] η (cid:96) .
The remainder of this section builds towards proving Theorem 2.
We begin by ﬁxing a layer (cid:96) ∈
[ L ] and neuron i ∈
[ η (cid:96) +1 ] .
Consider the random variables { Y k } k ∈ [ m ] where Y k ( x ) = 1 mp j w (cid:96) +1 ij a (cid:96)j ( x ) where Algorithm 2 selected index j ∈ [ η (cid:96) ] on the k th iteration of Line 19.
Note that z (cid:96) +1 i ( x )
= (cid:80) j ∈
[ η (cid:96) ] w (cid:96) +1 ij a (cid:96)j ( x ) and so we may also write g (cid:96)
+
[Equation]
( x ) when it is more convenient.
Lemma 4.
For each x ∈ X and k ∈ [ m ] , E [ Y k ( x )]
=
z (cid:96)
[Equation]
i
[Equation]
x
[Equation]
/m .
Proof.
Y j is drawn from distribution H deﬁned on Line 15, so we compute the expectation directly.
[Equation]
To bound the variance, we use an approach inspired by Baykal et al.
(2018) where the main idea is to use the notion of empirical sensitivity to establish that a particular useful inequality holds with high probability over the randomness of the input point x ∼ D .
Given that the inequality holds we can establish favorable bounds on the variance and magnitude of the random variables, which lead to a low sampling complexity.
For a random input point x ∼ D , let G denote the event that the following inequality holds (for all neurons):
[Equation]
where C = max { 3 K, 1 } and K is deﬁned as in Assumption 1.
We now prove that under Assumption 1, event G occurs with high probability.
From now on, to ease notation, we will drop certain superscripts/subscripts with the meaning is clear.
For example, z ( x ) will refer to z (cid:96) +1 i ( x ) .
Lemma 5.
If Assumption 1 holds, P ( G ) > 1 − δ/ 2 η (cid:96) .
Here the probability is over the randomness of drawing x ∼ D .
Proof.
Since max i ∈ [ η (cid:96) +1 ] g ij ( x ) is just a function of the random variable x ∼ D , for any j ∈ [ η (cid:96) ]
we can let D be a distribution over max i ∈
[ η (cid:96) +1 ] g ij ( x ) and observe that since s j = max x ∈S max i ∈ [ η (cid:96) +1 ] g ij ( x ) , the negation of event G for a single neuron j ∈ [ η (cid:96) ] can be expressed as the event
[Equation]
where X ∼ D and X 1 , . .
, X |S| i.i.d.
∼ D since the points in S were drawn i.i.d.
from D .
Invoking
Lemma 8 from Baykal et al.
(2018) in conjunction with Assumption 1, we obtain for any arbitrary j
[Equation]
with the K (cid:48) from Assumption 1.
Since our choice of neuron j was arbitrary, the inequality above holds for all neurons, therefore we can apply the union bound to obtain:
[Equation]
where the last line follows from the fact that |S| ≥ (cid:6) K
[Equation]
Lemma 6.
For any x such that event G occurs, then | Y k
( x ) − E [ Y k ( x )] | ≤ CSz/m .
Here the expectation is over the randomness of Algorithm 2.
Proof.
Recall that S =
(cid:80) j ∈
[ η (cid:96) ] s j .
Let neuron j ∈
[ η (cid:96) ] be selected on iteration k of Line 19.
For any k ∈ [ m ] we have:
[Equation]
where the ﬁrst inequality follows by the inequality of event G , the second by the fact
that max i (cid:48) g i (cid:48) j ( x ) ≥ g ij ( x ) for any i , and the third equality by deﬁnition of g ij ( x ) =
w ij a j ( x ) /z
( x ) .
This implies that | Y k − E [ Y k ] | = (cid:12)(cid:12)
Y k − zm (cid:12)(cid:12) ∈
[ − z/m, CSz/m ] by Lemma 4 and since Y k ≥ 0 .
The result follows since C, S ≥ 1 .
Lemma 7.
For any x such that event G occurs, then Var( Y k ( x ))
≤ CSz 2 /m 2 .
Here the expectation is over the randomness of Algorithm 2.
Proof.
We can use the same inequality obtained by conditioning on G to bound the variance of our estimator.
[Equation]
by deﬁnition of Y k
= S
[Equation]
since p j = s j /S
[Equation]
by occurrence of event G
[Equation]
since g ij ( x ) =
w ij a
[Equation]
x
[Equation]
/z
[Equation]
x
[Equation]
We are now ready to prove Theorem 2.
Proof of Theorem 2.
Recall the form of Bernstein’s inequality that, given random variables
X 1 , . . .
, X m such that for each k ∈ [ m ] we have E [ X k ] =
and | X k | ≤ M almost surely, then
[Equation]
We apply this with X k
= Y k − zm .
We must take the probability with respect to the randomness of both drawing x ∼ D and Algorithm 2.
By Lemma 4, E [ X k ] = 0 .
Let us assume that event G occurs.
By Lemma 6, we may set M = CSz/m .
By Lemma 7, (cid:80) k ∈ [ m ] E
[ X 2 k ] ≤ CSz 2 /m .
We will apply the inequality with t = εz .
Observe that (cid:80) k ∈ [ m ] X k
= ˆ z − z .
Plugging in these values, and taking both tails of the inequality, we obtain:
[Equation]
since C ≤ 3 K ≤
[Equation]
by deﬁnition of m
Removing dependence on event G , we write:
[Equation]
η (cid:96) + where we have applied Lemma 5.
This implies the result for any single neuron, and the theorem follows by application of the union bound over all η (cid:96) +1 neurons in layer (cid:96) .
B. B OOSTING S AMPLING VIA D ETERMINISTIC C HOICES
Importance sampling schemes, such as the one described above, are powerful tools with numerous applications in Big Data settings, ranging from sparsifying matrices (Baykal et al., 2018; Achlioptas et al., 2013; Drineas & Zouzias, 2011; Kundu & Drineas, 2014; Tropp et al., 2015) to constructing coresets for machine learning problems (Braverman et al., 2016; Feldman & Langberg, 2011; Bachem et al., 2017).
However, by the nature of the exponential decay in probability associated with importance sampling schemes (see Theorem 1), sampling schemes perform truly well when the sampling pool and the number of samples is sufﬁciently large (Tropp et al., 2015).
However, under certain conditions on the sampling distribution, the size of the sampling pool, and the size of the desired sample m , it has been observed that deterministically picking the m samples corresponding to the highest m probabilities may yield an estimator that incurs lower error (McCurdy, 2018;
Papailiopoulos et al., 2014).
To this end, consider a hybrid scheme that picks k indices deterministically (without reweighing) and samples m (cid:48) indices.
More formally, let C det ⊆
[ n ] be the set of k unique indices (corresponding to weights) that are picked deterministically, and deﬁne
[Equation]
where we note that the weights are not reweighed.
Now let C rand be a set of m (cid:48)
indices sampled from the remaining indices i.e., sampled from [ n ] \ C det , with probability distribution q = ( q 1 , . .
. , q n ) .
To deﬁne the distribution q , recall that the original distribution p is deﬁned to be p i
=
s i /S for each i ∈ [ n ] .
Now, q is simply the normalized distribution resulting from setting the probabilities associated with indices in C det to be 0, i.e.,
[Equation]
where S k = (cid:80) j ∈C det s j is the sum of sensitivities of the entries that were deterministically picked.
Instead of doing a combinatorial search over all (cid:0) nk (cid:1) choices for the deterministic set C det , for computational efﬁciency, we found that setting C det to be the indices with the top k sensitivities was the most likely set to satisfy the condition above.
We state the general theorem below.
Theorem 8.
It is better to keep k feature maps, C det ⊆
[ η (cid:96) ] , |C det | = k , deterministically and sample m (cid:48)
= (cid:6) (6 + 2 ε )
( S (cid:96) − S (cid:96)k )
K log(8 η ∗ /δ )
ε − 2 (cid:7) features from [ η (cid:96) ] \ C det if (cid:88) j/ ∈C det (cid:18) 1 − s j
S
[Equation]
k (cid:19) m (cid:48) >
η (cid:96) (cid:88)
j = (cid:16)
[Equation]
j
S (cid:17)
[Equation]
+ (cid:114) log(2 /δ )
( m + m (cid:48) ) , where m = (cid:6) (6 + 2 ε ) S (cid:96) K log(4 η ∗ /δ )
ε − 2 (cid:7) , S k = (cid:80) j ∈C det s j and η ∗ = max (cid:96) η (cid:96) .
Proof.
Let m ≥ (cid:6) (6 + 2 ε ) S
[Equation]
K log(4 η ∗ /δ )
ε − 2 (cid:7) as in Lemma 2 and note that from Lemma 2, we know that if ˆ z is our approximation with respect to sampled set of indices, C , we have
[Equation]
where E is the event that the inequality
[Equation]
holds.
Henceforth, we will let i ∈
[ η (cid:96) +1 ] be an arbitrary neuron and, similar to before, consider the problem of approximating the neuron’s value z (cid:96) +1 i ( x ) (subsequently denoted by z ) by our approximating ˆ z (cid:96) +1 i ( x ) (subsequently denoted by ˆ z ).
Similar to our previous analysis of our importance sampling scheme, we let C rand =
{ c 1 , . . .
, c m (cid:48) } denote the multiset of m (cid:48) neuron indices that are sampled with respect to distribution q and for each j ∈ [ m (cid:48) ] deﬁne Y j
= ˆ w ic j a c j and let Y = (cid:80) j ∈ [ m (cid:48) ]
Y j .
For clarity of exposition, we deﬁne ˆ z rand =
Y be our approximation with respect to the random sampling procedure, i.e.,
[Equation]
Thus, our estimator under this scheme is given by
[Equation]
Now we want to analyze the sampling complexity of our new estimator ˆ z (cid:48)
so that
[Equation]
Establishing the sampling complexity for sampling with respect to distribution q is almost identical to the proof of Theorem 2.
First, note that E [ˆ z (cid:48) | x ] = ˆ z det + E
[ˆ z rand | x ] since ˆ z det is a constant (conditioned on a realization x of x ∼ D ).
Now note that for any j ∈ [ m (cid:48) ]
[Equation] [Equation]
and so E [ˆ z rand | x ] =
E
[ Y | x ] =
z − ˆ z det .
This implies that E [ˆ z (cid:48) ]
= ˆ z det + ( z − ˆ z det ) =
z , and so our estimator remains unbiased.
This also yields
[Equation]
which implies that all we have to do to bound the failure probability of the event | z (cid:48) − z | ≥ εz is to apply Bernstein’s inequality to our estimator ˆ z rand
=
Y , just as we had done in the proof of
Theorem 2.
The only minor change is the variance and magnitude of the random variables Y k for k ∈ [ m (cid:48) ] since the distribution is now with respect to q and not p .
Proceeding as in the proof of
Lemma 6, we have
[Equation]
Now, to bound the magnitude of the random variables note that
[Equation]
The result above combined with this fact yields for the magnitude of the random variables
[Equation]
where we observe that the only relative difference to the bound of Lemma 6 is the term S − S k appears, where
S k
= (cid:80) j ∈C det s j , instead of S
Similarly, for the variance of a single Y j
[Equation]
where the last inequality follows by the fact that (cid:80) k ∈ [ η (cid:96) ] \C det w ik a k ( x ) ≤ z and by the sensitivity inequality from the proof of Lemma
[Equation]
This implies by Bernstein’s inequality and the argument in proof of Theorem 2 that if we sample
[Equation]
times from the distribution q , then we have
[Equation]
Now let p =
( p 1 , . . .
, p n ) be the probability distribution and let C denote the multi-set of indices sampled from [ n ] when m samples are taken from [ n ] with respect to distribution p .
For each index j ∈ [ n ] let U j ( m, p ) =
[ j ∈ C ] be the indicator random variable of the event that index j is sampled at least once and let U ( m, p )
=
(cid:80) ni =
j U j ( m, p ) .
Note that U is a random variable that denotes the number of unique samples that result from the sampling process described above, and its expectation is given by
[Equation]
Now we want to establish the condition for which U ( m (cid:48) , q ) <
U ( m, p ) , which, if it holds, would imply that the number of distinct weights that we retain with the deterministic + sampling approach is lower and still achieves the same error and failure probability guarantees, making it the overall better approach.
To apply a strong concentration inequality, let C (cid:48)
=
C det ∪ C rand =
{ c (cid:48) 1 , . . .
, c (cid:48) k , c (cid:48)
k +1 , . . .
, c (cid:48) m (cid:48) } denote the set of indices sampled from the deterministic + sampling (with distribution q ) approach, and let C
= { c 1 , . .
, c m } be the indices of the random samples obtained by sampling from distribution p .
Let f ( c (cid:48) 1 , . .
c (cid:48)
m (cid:48) ,
c 1 , . .
, c m ) denote the difference
U ( m (cid:48) , q ) − U ( m, p ) in the number of unique samples in C (cid:48) and C .
Note that f satisﬁes the bounded difference inequality with Lipschitz constant 1 since changing the index of any single sample in
C ∪ C (cid:48) can change f by at most 1 .
Moreover, there are m (cid:48)
+ m random variables, thus, applying
McDiarmid’s inequality (van Handel, 2014), we obtain
[Equation]
this implies that for t = (cid:113) log(
[Equation]
( m
[Equation] [Equation]
with probability at least 1 − δ/ 2 .
Thus, this means that if E [ U ( m, p )] − E [ U ( m (cid:48) , q )] > t , then
U ( m, p ) > U ( m (cid:48) , q ) .
More speciﬁcally, recall that
[Equation]
and
[Equation] [Equation]
j
[Equation]
j :
[Equation] [Equation]
=
[Equation]
[Figure]
Thus, rearranging terms, we conclude that it is better to conduct the deterministic + sampling scheme if (cid:88)
[Figure]
[Equation]
Putting it all together, and conditioning on the above inequality holding, we have by the union bound
[Equation]
this implies that with probability at least 1 − δ : (i
[Equation]
) z and (ii) U ( m (cid:48) , q ) <
U ( m, p ) , implying that the deterministic + sampling approach ensures the error guarantee holds with a smaller number of unique samples, leading to better compression.
B.1. E XPERIMENTAL E VALUATION OF D ERANDOMIZATION
To evaluate our theoretical results of derandomization, we tested the performance of our algorithm with respect to three different variations of sampling:
No derandomization ("rand"): We apply Alg. 2 and sample channels with probability proportional to their sensitivity.
Partial derandomization ("partial"): We apply Theorem 8 as a preprocessing step to keep the top k channels and then sample from the rest according to Alg. 2. 3.
Complete derandomization ("derand"): We simply keep the top channels until our sampling budget is exhausted.
The results of our evaluations on a LeNet300-100 architecture trained on MNIST can be seen in Fig.
As visible from Fig. 5(a), the process of partial derandomization does not impact the performance of our algorithm, while the complete derandomization of our algorithm has a slightly detrimental effect on the performance.
This is in accordance to Theorem 8, which predicts that that it is best to only partially derandomize the sampling procedure.
However, after we retrain the network, the additional error incurred by the complete derandomization is negligible as shown in Fig. 5(b).
Moreover, it appears that – especially for extremely low sampling regime – the completely derandomized approach seems to incur a slight performance boost relative to the other approaches.
We suspect that simply keeping the top channels may have a positive side effect on the optimization landscape during retraining, which we would like to further investigate in future research.
C M AIN C OMPRESSION T HEOREM
Having established layer-wise approximation guarantees as in Sec.
B, all that remains to establish guarantees on the output of the entire network is to carefully propagate the error through the layers as was done in Baykal et al.
For each i ∈ [ η (cid:96) +1 ] and (cid:96) ∈
[ L ] , deﬁne
[Equation] where z + i ( x ) =
(cid:80) k ∈
I + w (cid:96) +1 ik a (cid:96)k ( x ) and z − i ( x )
= (cid:80) k ∈
I − w (cid:96) +
ik a (cid:96)k ( x ) are positive and negative components of z (cid:96) +1 i ( x ) , respectively, with I
+
and I − as in Alg.
For each (cid:96) ∈
[ L ] , let ∆ (cid:96) be a constant deﬁned as a function of the input distribution D 7 , such that with high probability over x ∼ D , ∆
(cid:96) ≥ max i ∈ [ η (cid:96) +1 ] ∆ (cid:96)i .
Finally, let ∆ (cid:96) → = (cid:81)
Lk =
(cid:96) ∆ k .
Generalizing Theorem 2 to obtain a layer-wise bound and applying error propagation bounds of Baykal et al.
(2018), we establish our main compression theorem below.
Theorem 3.
Let ε, δ ∈ (0 , 1) be arbitrary, let S ⊂
X denote the set of (cid:100) K (cid:48) log (4 η/δ ) (cid:101) i.i.d.
points drawn from D , and suppose we are given a network with parameters θ
= ( W 1 , . .
, W L ) .
Consider the set of parameters ˆ θ =
( ˆ W 1 , . . .
, ˆ W L ) generated by pruning channels of θ according to Alg. 2 for each (cid:96) ∈
[ L ] .
Then, ˆ θ satisﬁes P ˆ θ,x ∼D (cid:0)
f
ˆ θ ( x ) ∈ (1 ± ε ) f
θ ( x ) (cid:1) ≥ 1 − δ, and the number of ﬁlters in ˆ θ is bounded by O (cid:16)(cid:80)
L(cid:96)
=
1 L 2 (∆ (cid:96) → ) 2 S (cid:96) log( η/δ ) ε(cid:17) .
D E XTENSION TO CNN S
To extend our algorithm to CNNs, we need to consider the fact that there is implicit weight sharing involved by deﬁnition of the CNN ﬁlters.
Intuitively speaking, to measure the importance of a feature map (i.e. neuron) in the case of FNNs we consider the maximum impact it has on the preactivation z (cid:96) +1 ( x ) .
In the case of CNNs the same intuition holds, that is we want to capture the maximum contribution of a feature map a (cid:96)j ( x ) , which is now a two-dimensional image instead of a scalar neuron, to the pre-activation z (cid:96) +
( x ) in layer (cid:96) + 1 .
Thus, to adapt our algorithm to prune channels in CNNs, we modify the deﬁnition of sensitivity slightly, by also taking the maximum over the patches p ∈ P (i.e., sliding windows created by convolutions).
In this context, each activation a (cid:96)j ( x ) is also associated with a patch p ∈ P , which we denote by a (cid:96)jp .
In particular, the slight change is the following:
[Equation]
where a · p corresponds to the activation window associated with patch p ∈ P .
Everything else remains the same and the proofs are analogous.
E E XPERIMENTAL D ETAILS AND A DDITIONAL E VALUATIONS
For our experimental evaluations, we considered a variety of data sets (MNIST, CIFAR-10, ImageNet) and neural network architectures (LeNet, VGG, ResNet, WideResNet, DenseNet) and compared against several state-of-the-art ﬁlter pruning methods.
We conducted all experiments on either a single NVIDIA RTX 2080Ti with 11GB RAM or a NVIDIA Tesla V100 with 16GB RAM and implemented them in PyTorch (Paszke et al., 2017). Retraining with ImageNet was conducted on a cluster of 8 NVIDIA Tesla V100 GPUs.
In the following, we summarize our hyperparameters for training and give an overview of the comparison methods.
All reported experimental quantities are averaged over three separately trained and pruned networks.
E. C OMPARISON M ETHODS
We further evaluated the performance of our algorithm against a variety of state-of-the-art methods in ﬁlter pruning as listed below.
These methods were re-implemented for our own experiments to ensure an objective comparison method between the methods and we deployed the same iterative pruning and ﬁne-tune strategy as is used in our method.
Moreover, we considered a ﬁxed pruning ratio of ﬁlters in each layers as none of the competing methods provide an automatic procedure to detect relative layer importance and allocate samples accordingly.
Thus, the differentiating factor between the competing methods is their respective pruning step that we elaborate upon below.
Filter Thresholding (FT, Li et al.
(2016) Consider the set of ﬁlters W (cid:96) =
[ W (cid:96) 1 , . . .
, W (cid:96)η (cid:96) ] in layer (cid:96) and let (cid:13)(cid:13) W (cid:96)j (cid:13)(cid:13) 2 , 2 denote the entry-wise (cid:96) 2 -norm of W (cid:96)j (or Frobenius norm).
Consider a desired sparsity level of t % , i.e., we want to keep only t % of the ﬁlters.
We then simply keep the ﬁlters with the largest norm until we satisfy our desired level of sparsity.
SoftNet (He et al., 2018)
The pruning procedure of He et al.
(2018) is similar in nature to the work of Li et al.
(2016) except the saliency score used is the entrywise (cid:96) 1 - norm (cid:13)(cid:13) W (cid:96)j (cid:13)(cid:13) 1 , 1 of a ﬁlter map
W (cid:96)j .
During their ﬁne-tuning scheme they allow pruned ﬁlters to become non-zero again and then repeat the pruning procedure.
As for the other comparisons, however, we only employ one-shot prune and ﬁne-tune scheme.
ThiNet (Luo et al., 2017)
Unlike the previous two approaches, which compute the saliency score of the ﬁlter W (cid:96)j by looking at its entry-wise norm, the method of Luo et al.
(2017) iteratively and greedily chooses the feature map (and thus corresponding ﬁlter) that incurs the least error in an absolute sense in the pre-activation of the next layer.
That is, initially, the method picks ﬁlter j ∗ such that j ∗ = argmin j ∈ [ η
[Equation] ] max x ∈S [Equation] z
[Equation] +1 ( x ) − z
[-+
-][Equation]
+1 [ j ] ( x ) (cid:12)(cid:12)(cid:12) , where z (cid:96) +
( x ) denotes the pre-activation of layer (cid:96) + 1 for some input data point x , z (cid:96) +
[ j ] ( x ) the pre-activation when only considering feature map j in layer (cid:96) , and S a set of input data points.
We note that this greedy approach is quadratic in both the size η (cid:96) of layer (cid:96) and the size |S| of the set of data points S , thus rendering it very slow in practice.
In particular, we only use a set S of cardinality comparable to our own method, i.e., around 100 data points in total.
On the other hand, Luo et al.
report to use 100 data points per output class resulting in 1000 data points for CIFAR10.
E. L E N ET ARCHITECTURES ON MNIST
We evaluated the performance of our pruning algorithm and the comparison methods on LeNet300100 (LeCun et al., 1998), a fully-connected network with two hidden layers of size 300 andhidden units, respectively, and its convolutional counterpart, LeNet-5 (LeCun et al., 1998), which consists of two convolutional layers and two fully-connected layers.
Both networks were trained on MNIST using the hyper-parameters speciﬁed in Table 3.
We trained on a single GPU and during retraining (ﬁne-tunine)
we maintained the same hyperparameters and only adapted the ones speciﬁcally mentioned in Table 3.
E. C ONVOLUTIONAL N EURAL N ETWORKS ON CIFAR-
We further evaluated the performance of our algorithm on a variety of convolutional neural network architectures trained on CIFAR-10.
Speciﬁcally, we tested it on VGG16 with batch norm (Simonyan & Zisserman, 2014b), ResNet20 (He et al., 2016), DenseNet22 (Huang et al., 2017), and Wide
ResNet-16-8 (Zagoruyko & Komodakis, 2016).
For residual networks with skip connections, we model the interdependencies between the feature maps and only prune a feature map if it does not get used as an input in the subsequent layers.
We performed the training on a single GPU using the same hyperparameters speciﬁed in the respective papers for CIFAR training.
During ﬁne-tuning we kept the number of epochs and also did not adjusted the learning rate schedule.
We summarize the set of hyperparameters for the various networks in Table 4. [Table] [Table]
E. C ONVOLUTIONAL N EURAL N ETWORKS ON I MAGE N ET
We consider pruning convolutional neural networks of varying size – ResNet18, ResNet50, and
ResNet101 – trained on the ImageNet (Russakovsky et al., 2015) data set.
The hyper-parameters used for training and for our pruning algorithm are shown in Table 5.
For this dataset, we considered two scenarios: (i) iterative pruning without retraining and (ii) iterative prune-retrain with a limited amount of iterations given the resource-intensive nature of the experiments.
In the ﬁrst scenario, we evaluate the baseline effectiveness of each pruning algorithm without ﬁnetuning by applying the same iterative prune-scheme, but without the retraining step.
The results of these evaluations can be seen in Fig.
Fig.
6 shows that our algorithm outperforms the competing approaches in generating compact, more accurate networks.
We suspect that by reevaluating the data-informed ﬁlter importance (empirical sensitivity) after each iteration our approach is capable of more precisely capturing the inter-dependency between layers that alter the relative importance of ﬁlters and layers with each pruning step.
This is in contrast to competing approaches, which predominantly rely on weight-based criteria of ﬁlter importance, and thus can only capture this inter-dependency after retraining (which subsequently alters the magnitude of the weights).
Next, we consider pruning the networks using the standard iterative prune-retrain procedure as before (see Sec.
E) with only a limited number of iterations (2-3 iterations per reported experiment). [Figure] [Figure] The results of our evaluations are reported in Table 6 with respect to the following metrics: the resulting error of the pruned network (Pruned Err.), the difference in model classiﬁcation error (Err.
Diff), the percentage of parameters pruned (PR)
[Figure]
and the FLOP Reduction (FR). We would like to highlight that – despite the limited resources used during the experiments – our method is able to produce compressed networks that are as accurate and compact as the models generated by competing approaches (obtained by signiﬁcantly more prune-retrain iterations than allotted to our algorithm). E. A PPLICATION TO R EAL - TIME R EGRESSION T ASKS In the context of autonomous driving and other real-time applications of neural network inference, fast inference times while maintaining high levels of accuracy are paramount to the successful deployment of such systems (Amini et al., 2018).
The particular challenge of real-time applications stems from the fact that – in addition to the conventional trade-off between accuracy and model efﬁciency – inference has to be conducted in real-time.
In other words, there is a hard upper bound on the allotted computation time before an answer needs to be generated by the model.
Model compression, and in particular, ﬁlter compression can provide a principled approach to generating high accuracy outputs without incurring high computational cost.
Moreover, the provable nature of our approach is particularly favorable for real-time applications, as they usually require extensive performance guarantees before being deployed, e.g., autonomous driving tasks.
To evaluate the empirical performance of our ﬁlter pruning method on real-time systems, we implemented and tested the neural network of Amini et al.
(2018), which is a regression neural network deployed on an autonomous vehicle in real time to predict the steering angle of the human driver.
We trained the network of Amini et al.
(2018), denoted by Deepknight , with the driving data set provided alongside, using the hyperparameters summarized in Table 7. [Table] [Table]
The results of our compression can be found in Fig.
7, where we evaluated and compared the performance of our algorithm to those of other SOTA methods (see Sec.
E.6).
We note that these results were achieved without retraining as our experimental evaluations have shown that even without retraining we can achieve signiﬁcant pruning ratios that lead to computational speed-ups in practice.
As apparent from Fig. 7, we can again outperform other SOTA methods in terms of performance vs. prune ratio.
Note that since this is a regression task, we used test loss (mean-squared error on the test data set) as performance criterion.
Finally, we would like to highlight that our ﬁlter pruning method may serve as a principled subprocedure during the design of neural network architectures for real-time applications.
In particular, given an inference time budget T , one can design and train much larger architectures with favorable [Figure] [Figure] [Figure]
s
[Figure] deepknight, Driving Reference Net Ours FT SoftNet
[Figure]
performance which, however, violate the given budget T .
Our ﬁlter pruning method can then be leveraged to compress the network until the given budget T is satisﬁed, thus reducing the burden on the practitioner to design a simultaneously accurate and computationally-efﬁcient neural network architecture.
E. C OMPARISONS TO A DDITIONAL M ETHODS O N CIFAR-
We evaluate the performance of our algorithm on pruning modern convolutional and residual benchmark network architectures – ResNet20, ResNet56, ResNet110, and VGG16 – trained on CIFAR-10, and compare it to the results reported by state-of-the-art ﬁlter pruning approaches.
The results are obtained by iteratively pruning and ﬁne-tuning the resulting model (starting from the original pre-trained network) in a hyperharmonic sequence of prune ratios as described in Sec. 4.1.
For our algorithm, in addition to reporting the pruned model that achieves commensurate accuracy ("within 0 .
5% err.") as in Sec. 4, we report the model that (i) closest matches the accuracy of the original network ("orig. err.") and (ii) achieves the lowest classiﬁcation error possible ("lowest err.")
– which for nearly all of the models considered, is lower than that of the original model.
Table 8 summarizes our results and depicts the performance of each pruning algorithm with respect to various metrics: the resulting error of the pruned network (Pruned Err.), the difference in model classiﬁcation error (Err.
Diff), the percentage of parameters pruned (PR), and the FLOP Reduction (FR).
Our results show that our algorithm outperforms competing approaches in virtually all of the considered models and pertinent metrics, especially when the overall quality of the pruned model is taken into account.
For ResNet20 for instance, our algorithm generates a model that is simultaneously the sparsest (> 43%
PR) and the most accurate ( 8 . 64% Err., 0 .
04% Err.
Diff) despite starting from a pre-trained model with the highest error of 8 .
60% (Orig.
Err.)
among the reported results.
The method of He et al.
(2019) does achieve a higher FR than our method, however, this is achieved at the cost of nearly 2% degradation in classiﬁcation accuracy (compared to 0 .
04% for ours).
For larger networks such as ResNet110, our algorithm’s favorable performance is even more pronounced: the models generated by our algorithm are not only the sparsest (PR) and most efﬁcient (PR), but they are also the most accurate.
A nearly identical trend holds for the results pertaining to
VGG16 and ResNet56: the models generated by our method tend to be the overall sparsest and most accurate, even when starting from pre-trained models with higher classiﬁcation error.
[Table]
Anonymous (2020) 5.
+0.
N/A
ResNet
Ours (within 0.5% err.) 6.
+0.
Ours (orig. err.) 6.
Ours (lowest err.) 6.
Li et al.
(2016) (FT) 6.
+0.
He et al.
(2018) (SoftNet) 6.
N/A 40. He et al.
N/A 52.
Dong et al.
[Table]
+0.
N/A 34. VGG
Ours [Table]
+0.
Ours
[Table] Ours [Table] lowest err.)
Li et al.
[Table]
FT) 6. Huang et al.
+0.
He et al.
+0.
N/A 35.
Li et al.
+0.