W HEN
A RE
N EURAL
P RUNING
A
PPROXIMATION
B OUNDS U SEFUL ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Approximation bounds for neural network pruning attempt to predict the tradeoff between sparsity and ﬁdelity while shrinking neural networks.
In the ﬁrst half of this paper, we empirically evaluate the predictive power of two recently proposed methods based on coreset algorithms.
We identify several circumstances in which the bounds are loose or impractical to use and provide a brief analysis of the components of the bounds that contribute to those short-comings.
In the second half, we examine the role of ﬁne-tuning in prunability and observe that even tight approximation bounds would be poor predictors of accuracy after ﬁnetuning.
This is because ﬁne-tuning can recover large amounts of accuracy while simultaneously maintaining or increasing approximation error.
We discuss the implications of these ﬁnding on the application of coreset-based pruning methods in practice and the role of approximation in the pruning community.
Our code is available in the attached supplementary material.
I NTRODUCTION
Recent work has attempted to prune neural networks while providing a mathematical upper bound on the deviation from the original network.
These bounds were ﬁrst introduced as a component in proofs of generalization in statistical learning theory (Arora et al., 2018).
Recent improvements on these pruning methods have come from the ﬁeld of randomized algorithms, leveraging the concept of coresets (Mussay et al., 2020)
(Baykal et al., 2019a) (Baykal et al., 2019b)
(Liebenwein et al., 2019), with empirical evaluations focusing on practical improvement in the sparsity/accuracy trade-off.
However, surprisingly little attention has been given to evaluating the tightness of the proposed bounds or the predictive power of the guarantees they provide.
In the ﬁrst half of this paper, we ﬁll this gap by empirically evaluating the tightness of two recently proposed coreset bounds, using them to predict the accuracy of neural networks after pruning.
We conduct a series of experiments of increasing difﬁculty, starting with networks that are known to be highly sparse and ending with networks in which no regularization is applied during training.
We identify several circumstances in which the bounds are loose or impractical to use and provide a brief analysis of the bound components that contribute to those short-comings.
In the second half of the paper we introduce a ﬁne-tuning procedure after pruning, which is common practice in the pruning community.
Fine-tuning can drastically improve the accuracy achieved at high levels of sparsity, taking some models from 40% accuracy after pruning to 98% accuracy after ﬁne-tuning.
However, we observe that this improvement does not usually include a reduction in approximation error; indeed, ﬁne-tuning can even increase approximation error in hidden layers while simultaneously recovering the original performance of the network.
We argue that this phenomenon limits the predictive power of approximation bounds which are constructed “layer-bylayer” and which ignore the ﬁnal softmax layer at the output.
This motivates further discussion of the practical applications of coreset-based methods and whether “approximation” is the correct goal for the pruning community.
R ELATED W ORK
Over-parameterization A popular recipe for training neural networks is to make the network as large as possible such as to achieve 0 training loss, then regularize to encourage generalization (Karpathy, 2019).
It is possible that such over-parameterized networks have have favorable optimization properties (Li et al., 2020)
(Du et al., 2018b).
However, over-parameterized networks are becoming increasingly burdensome to evaluate on consumer hardware, which has driven the interest in researching precise and effective pruning methods that can take advantage of sparsity in the discovered solutions.
Pruning With Guarantees Despite the massive literature on neural pruning techniques, few papers provide any kind of mathematical guarantee on the result of pruning. Besides the coreset methods which are discussed below, Serra et al. (2020) provide a method for “losslessly” compressing feedforward neural networks by solving a mixed-integer linear program. They observe that applying
L1 regularization during training is required to achieve non-negligible lossless compression rates, which closely aligns with the observations in our work.
Havasi et al.
(2018) provide an explicit compression/accuracy trade-off for Bayesian neural networks, but their compression method cannot be used to improve inference efﬁciency, only the memory needed to store the model.
Empirical Scaling Laws Frankel et al.
have observed that error after pruning can be empirically modeled as a function of network size, dataset size, and sparsity with good predictive results (Rosenfeld et al., 2020).
This can be seen as a more practical alternative to formal guarantees on accurancy after pruning, where a few small-scale experiments are run to estimate the accuracy after pruning at larger scales.
C ORESET A PPROXIMATION B OUNDS 3. C ORESET W EIGHT P RUNING
An algorithm from Baykal et al. (2019b) for pruning the weights of a single perceptron is presented in Algorithm 1; it involves sampling from a distribution over weights, where the probability of being sampled roughly measures the weight’s importance to maintaining the output.
The algorithm keeps the selected weights (with some re-weighting) and prunes the rest.
The resulting pruned perceptron is guaranteed to output values that are arbitrarily close to the original perceptron if a sufﬁciently large number of samples is taken.
The relationship between the closeness of the approximation and number of samples required is given by Theorem A.1.
Further details and required assumptions are provided in Appendix 3.1.
The method can be used to sparsify a whole layer by sparsifying each neuron individually, and bounds for pruning multi-layer neural networks can be constructed using a “layer-by-layer” approach, which is discussed in Section 5.2.
Deterministic Weight Pruning Baykal et al.
(2019b) also present a deterministic version of their algorithm, which selects the weights with the lowest importances for pruning.
This alternative algorithm is presented in Algorithm 2 and the corresponding guaranteed approximation error is presented in Theorem A.2, the proof of which is based on the probabalistic proof of Theorem A.1.
3. C ORESET N EURON P RUNING
Whole neurons can be pruned rather than individual weights.
In this case, importances are assigned to each neuron, creating a probability distribution that can be sampled from.
A method from Mussay et al.
(2020) for pruning feed-forward neural networks with a single hidden layer is shown in Algorithm 3.
This algorithm is data-independent : it ignores the training data and instead only assumes that the inputs have a bounded L2 norm.
Neuron importance is determined by the L2 norm of the parameters of each neuron and the magnitude of the weights assigned to them in the next layer.
Similar to weight pruning, the approximation error can be bounded above depending on the number of samples taken from the distribution over neurons.
The relationship is expressed in Theorem A.4.
Notice that unlike the multiplicative approximation bound provided for weight pruning, this bound is an additive error bound, meaning that the guaranteed error term is constant, regardless of the scale of the “true” value.
More details about this method are provided in Appendix A.3.
ˆ yˆ yˆ y
ˆ y− (cid:15) min
ˆ y+ (cid:15) max
ˆ y
Guaranteed Label-Consistency + (cid:15) (cid:48) max ˆ y 2 − (cid:15) (cid:48)
min ˆ y
Label May Change g
( (cid:15) )
-Approx g
( (cid:15) )
+ β -Approx (cid:15) -Approx
Prune
Prune
3. E PSILON -F IRST AND S PARSITY -F IRST P RUNING
Pruning using randomized coreset methods involves sampling from a distribution over weights or neurons.
The number of samples taken can be determined in one of two ways, depending on whether there is a strict requirement on approximation error or on sparsity: Epsilon-ﬁrst : The number of samples is determined by the desired epsilon guarantee on the output. Samples are taken with replacement, and sparsity is determined by the number of unique samples. Sparsity-ﬁrst :
Sample from the network until the number of unique samples equals the desired sparsity.
Work backwards to infer the epsilon guarantee given that number of samples.
Each approach has its drawbacks.
While epsilon-ﬁrst pruning guarantees a certain accuracy after pruning, it cannot guarantee the sparsity achieved since sampling is done with replacement.
Sparsity-ﬁrst pruning, on the other hand, is not guaranteed to terminate.
For example, in the “easy case” experiments in the next section, it will be impossible for the algorithm to achieve 50% sparsity since less than 50% of elements have non-zero probability of being sampled, which will cause the algorithm to run forever.
We will use both approaches in this work, preferring sparsity-ﬁrst pruning when we must achieve a speciﬁc sparsity and preferring epsilon-ﬁrst pruning when we are familiar with what a “reasonable” amount of approximation error looks like, which we discuss in the next section.
E VALUATING A PPROXIMATION G UARANTEES 4. A CCURACY G UARANTEES
One obstacle to evaluating approximation guarantees is that it is difﬁcult to determine what constitutes a reasonable amount of approximation error.
Our solution is to use approximation bounds to predict how many development set labels are guaranteed to remain correct and how many might ﬂip to an incorrect label during pruning.
This provides a more intuitive sense of the implications of the approximation bound.
In order to convert an approximation bound into an accuracy guarantee, we must determine whether the output error is large enough to change the prediction of the neural network.
Note that the predicted label of a neural network which performs classiﬁcation is simply the index of the largest pre-softmax logit.
2 Whether the maximum logit will stay the maximum under some pertubation will depend on the distance between the maximum output logit and the second-largest output logit, which varies for each example in the development dataset.
This is illustrated in Figure 1 and made precise in the following lemma:
Lemma 4.1 Let f : X → Y be a neural network and x ∈ X be an arbitrary input.
Let j denote the index of the largest pre-softmax logit, j = arg max i f ( x ) i .
Further, let j (cid:48) denote the second largest pre-softmax logit f ( x ) j (cid:48) ≥ f
( x ) k , k (cid:54) =
j .
Finally, let ˆ f be an approximation of f , with guaranteed element-wise approximation error bound (cid:15) .
If the guarantee is additive, then
[Equation]
If the bound is multiplicative, then
[Equation]
This relationship allows us to determine the epsilon approximation bound that corresponds to some accuracy guarantee after pruning.
For example, suppose we want to guarantee 95% of correct labels will stay correct.
We would compute the pre-softmax logits of the original neural network on the development set, calculating the epsilon bound required to maintain each correct label.
We would then select a ﬁnal epsilon value which is smaller than 95% of the values computed and prune to achieve that bound.
4. E ASY , M ODERATE , AND D IFFICULT C ASES
To begin our evaluation, we start with the simplest circumstances: we prune only the ﬁnal layer of
LeNet300-100, a small, two-layer feed-forward neural network trained to solve MNIST.
We vary the training procedure to make it easier or harder to prune the network while maintaining low approximation error.
More details about each of these cases can be found in Appendix B.
Pre-pruned Neural Networks
In the easiest case, we pre-prune the neural network to a high sparsity (95% for weight pruning and 90% for neuron pruning) using non-coreset-based heuristic methods and then re-train the network to re-gain all lost accuracy.
We then re-initialize the pruned connections to zero.
The resulting network is known to be sparse and highly prunable while incurringapproximation error.
We should hope that coreset-based pruning methods will easily prune to the known sparsity while guaranteeing low amounts of approximation error.
Some Regularization In the more realistic moderate case, we apply mild L1 regularization (aka
Lasso) during the training of the neural network.
We set the L1 regularizing penalty to 0.001, which is the highest setting we could use which did not decrease development accuracy.
No Regularization
In the hardest case, we do not regularize the network.
We expect this model to be the least amenable to approximation, and expect the coreset-based methods to appropriately predict that the network cannot be pruned.
All models are trained to 98% development accuracy.
4. R EPORT C ARD : E PSILON -F IRST P RUNING
Figure 2 shows the results for epsilon-ﬁrst pruning the ﬁnal layer of LeNet-300-100 under various training circumstances, guaranteeing that 50% of the development set labels that are correct will remain correct.
The full results of these experiments, evaluated at different percentages of accuracy guarantees can be found in Figure 6.
As expected, all methods easily achieve high amounts of sparsity in the pre-pruned “easy” case.
In the moderate case, less sparsity can be achieved, and in , we might suspect that dropout would be a better choice than L1 regularization to encourage prunability.
However, this was not the case in our experiments. We discuss possible explanations and detail our experiments in Appendix B.2. [Figure] LayerSize
Samples
Sparsity
[Figure]
the difﬁcult case almost no pruning can be done.
However, we note some practical issues with the performance of these methods:
High Sample Complexity We can see (highlighted in red) that the number of samples required during pruning is unnecessarily high.
Even in the easy case, in which we know the network is already sparse, the number of samples required is three orders of magnitude larger than the number of elements to sample from.
We are still able to achieve some sparsity in the easy and moderate cases, however, since the probability distribution over weights and neurons is highly skewed.
High sample complexity ( < 1M samples) can easily be handled by consumer hardware, but may grow impractical as the layers grow larger.
Loose Bounds We also note that the observed approximation error is much lower than the provided bound (highlighted in orange).
We guaranteed only 50% of the accuracy should be maintained, but in all cases 100% of the accuracy was maintained.
The looseness of the bounds implies that much higher sparsities could be reached while still achieving the desired accuracy guarantee.
Indeed,
Figure 3 shows that even the “difﬁcult” case neural network can be pruned up to 90% sparsity before reaching 50% accuracy, whereas our best method only reached 8% sparsity.
For neuron pruning, we might suppose that the looseness of bounds can be explained by laxness of the data-independence assumption.
We explore this idea in Appendix C.1, but fail to conclude that lax data assumptions are a sufﬁcient cause of loose bounds.
The deterministic version of weight pruning, while still providing signiﬁcantly loose bounds, sidesteps the sample complexity issue by not requiring any random sampling (highlighted in green).
In our opinion, this is the most promising future for these methods, which may leverage the beneﬁts of probabalistic proof techniques while minimizing their drawbacks via de-randomization.
However, none of these methods provide approximation bounds which are ﬁne-grained enough to predict the sparsity/accuracy trade-off, which we expect to become a theme of future work.
4.3. S PARSITY -F IRST P RUNING
The high sample complexity of epsilon-ﬁrst pruning implies that sparsity-ﬁrst pruning with randomized methods will lead to incredibly vacuous approximation bounds.
This is because the number of samples taken is usually less than the number of elements to be sampled from.
And indeed, Figure 8 shows that pruning a “moderate case” network at weight sparsities greater 35% lead to bounds that cannot guarantee any accuracy at all.
Pruning at less that this sparsity causes the algorithm to run forever.
Sparsity-ﬁrst pruning with deterministic weight pruning is easier to work with, but still under-estimates the achieved accuracy after pruning due to loose bounds.
4.3. W HAT D RIVES S AMPLE C OMPLEXITY ?
Two main components in each theorem drive sample complexity: the tightness of the epsilon bound and the “total sensitivity” of the layer.
Sensitivity is a measure of importance , so if all neurons or weights are equally important then we will require a large number of samples to achieve a good approximation.
If only a few weights are important, then total sensitivity will be small and we will require fewer samples.
In the case of weight pruning, sample complexity scales linearly with total sensitivity, and the slope of the line is determined by the required epsilon bound and the probability of failure.
4 For LeNet, layers tend to have a relatively low total sensitivity (only 5% of the possible maximum).
Even at the minimum total sensitivity, however, Theorem A.1 would require sampling 1.5x the number of weights for each perceptron.
This seems to imply that sample complexity’s quadratic dependence on epsilon is too restrictive for neural networks of the size typically deployed in practice, and that future work should focus on relaxing that dependence.
For neuron pruning in Theorem A.4, the sensitivity of each neuron is a function of the norm of the inputs and the norm of the weights of each neuron.
This leads us to Corollary C.2, which says that we can arbitrarily increase or decrease the sample complexity of any accuracy guarantee by scaling the appropriate weight matrices by a constant factor.
We prove this using the positive homogeneity property of ReLU networks (Du et al., 2018a) 5 , and verify it empirically in Appendix C.2.
However, investigating the full implications of this observation are out of the scope of this paper.
4.3. S CALING U P T O L ARGER N ETWORKS
We might expect that results would be more compelling on larger neural networks trained on more realistic data, since coreset algorithms were originally applied to big-data streaming applications in which millions of elements are sampled from.
To test this, we repeat the experiments above on a VGG-16 model trained to 92% development accuracy on CIFAR-10, which has 10x the number of neurons in the previous experiments.
We ﬁnd that these models are more prunable via heuristic methods than LeNet; we can achieve 99% weight and neuron sparsity without decreasing accuracy.
The results for coreset-based methods, however, are qualitatively similar to earlier experiments (Figure 2 and Appendix B.3).
While the number of neurons has increased by 10x, so too has the number of samples required by deterministic weight pruning and randomized neuron pruning.
Bounds are similarly loose for all methods.
The relative increase in weight pruning sample complexity, however, is much less than 10x (highlighted in blue), which we believe reﬂects the increased prunability of these models.
This may imply that coreset-based methods may give non-vacuous sampling complexities for very large and very sparse layers.
However, the scale required (10k+ hidden units) does not seem to overlap with sizes typically used in practice.
F INE - TUNING D OES N OT A PPROXIMATE
In the sections above, we have only evaluated networks immediately after pruning, without any ﬁnetuning allowed.
However, ﬁne-tuning is an integral step in the pruning process.
This is illustrated in Figure 3, in which we prune a “difﬁcult case” neural network to 95% weight sparsity using a sparsity-ﬁrst coreset method.
Accuracy drops to below 40% after pruning, but recovers to 98% after ﬁne-tuning.
We call this discrepancy between accuracy before and after ﬁne-tuning the “ﬁne-tuning gap.”
While approximation bounds may improve accuracy prediction after pruning by tightening bounds, we will see in the following sections that they have no hope of predicting the size of the y 1 y 2 yˆ y 1 ˆ y 2 ˆ yp 1 p 2 p ﬁne-tuning gap while in their current form.
This is because ﬁne-tuning allows us to recover accuracy while simultaneously maintaining or increasing the approximation error measured by these bounds.
5. S OFTMAX T RANSLATION I NVARIANCE
First, the methods described above measure approximation error before the softmax layer is applied.
However, the softmax layer is translation invariant , which means that adding a constant C to each pre-softmax logit will not change the output distribution.
This is illustrated in Figure 3 (Right).
This means gradient descent during ﬁne-tuning will not have any incentive to reduce pre-softmax approximation error, but instead will be driven to adjust each logit locally with respect to the other logits in the output.
This is true both when the pruned network is ﬁne-tuned on the training data and when it is trained using knowledge distillation (Hinton et al., 2015).
This can be observed in Figure 3, which also shows the approximation error before and after ﬁnetuning on the training data for a “difﬁcult case” network pruned using deterministic sparsity-ﬁrst weight pruning.
Although accuracy recovers from 30% to 98%, the pre-softmax approximation error increases or stays the same after ﬁne-tuning.
Given the fragile relationship between accuracy and pre-softmax approximation error, we can expect methods which rely on such approximation bounds to be poor predictors of performance after ﬁne-tuning.
5. L AYER - BY - LAYER B OUNDS
A “layer-by-layer” approach may be used to prune multi-layer neural networks with approximation guarantees, as illustrated in Figure 1.
While layer-by-layer construction of approximation bounds[-y 1 y 2 yˆ y 
 ˆ y 2 ˆ yp 1 p 2 p-] is conceptually simpler and amenable to proof than working with the network as a whole, it requires that the pruned neural network approximate the original dense network at each hidden layer.
If the approximation error at any intermediate layer is large, it “bubbles up” and greatly increases the approximation error at the output.
In practice, however, ﬁne-tuning allows for later layers to compensate for large amounts of approximation error in layers close to the input.
Consider the table below: we prune the ﬁrst layer of a “difﬁcult case” neural network to a high sparsity and observe the resulting approximation error in both the ﬁrst hidden layer and the output.
We then ﬁne-tune the model to approximate the original network’s pre-softmax output activations, as in patient knowledge distillation (Sun et al., 2019).
[Table]
This process indeed reduces the pre-softmax approximation error of the pruned network and recovers most of the development accuracy.
However, the approximation error in the hidden layer remains the same or increases (highlighted in red).
This means that any layer-by-layer approximation bound will be “bottlenecked” by this hidden error, even though the pruned neural network has low approximation error when considered as a whole.
5. P RUNABLE (cid:54) = A PPROXIMABLE
The results of the previous sections highlights a subtle distinction between layers that are easy to approximate and layers which are prunable.
Pruning a layer may introduce a large amount of approximation error.
If the layer is near the output, then the network may still output the correct labels by exploiting the translation invariance of softmax.
If the layer is near the input, then later layers may adapt to the approximation error by adjusting their weights.
In the next section, we discuss the implications of this distinction on future work on pruning with approximation bounds.
5. W HEN A RE A PPROXIMATION B OUNDS U SEFUL ?
Pruning Efﬁcacy
Given the subtle distinction between prunable and approximable networks, it is not clear whether pruning methods which directly attempt to approximate the original neural network should be able to prune more effectively in terms of the development accuracy / sparsity trade-off.
While Mussay et al.
(2020) do not compare to any heuristic methods after re-training,
Baykal et al.
(2019b)
ﬁnd only small improvements (1-2%) in development accuracy over magnitude weight pruning at the highest levels of sparsity.
It is out of the scope of this paper to evaluate these claims, since we are mainly concerned with the predictive power of these methods and the guarantees they provide.
However, it is possible that minimizing approximation error improves the ability of gradient descent to ﬁnd “other” global optima that are nearby.
This would make sense in light of mode-connectivity (Garipov et al., 2018), which observes that neural networks may have many global optima connected by simple linear paths.
However, a more rigorous analysis is required to provide a motivation and quantitative prediction of the beneﬁts of approximation.
Optimal Layer Sparsity Another claimed beneﬁt of good approximation bounds is the ability to correctly predict the appropriate sparsity for each hidden layer or neuron, without having to empirically test every possible sparsity combination.
This is done in Baykal et al.
(2019b) using a convex optimzation method to minimize the sum of the errors in each layer.
However, we have observed that different layers can tolerate different amounts of approximation error due to the ability of later layers to compensate during ﬁne-tuning.
We believe it is unlikely that approximation error theory by itself will allow for useful predictions about optimal sparsity; optimizability of sparse networks dur- ing ﬁne-tuning must be taken into account.
It may also be the case that deviating from the original network may be a desirable characteristic, as the new optimum may be more generalizable.
Generalization Pruning approximation bounds were made recently popular by their application to proofs of generalization, and that is likely where they will remain useful.
However, proofs of generalization which employ loose approximation bounds will likely result in loose generalization bounds, and so care should be taken by the consumer of these theorems.
C ONCLUSION
There is still work to be done in order to make neural pruning approximation bounds both practical and correct.
Current methods tend to over-estimate the approximation error that will be observed in practice, and randomized methods require an impractical number of samples to be taken during pruning, limiting the sparsity that can be achieved while guaranteeing ﬁdelity.
There are a number of avenues to improve these short-comings, including focusing on derandomization, taking into account the positive homogeneity of neural networks, and only applying coreset methods to networks of an appropriate scale.
However, while these improvements may tighten approximation bounds, they will still not be practically predictive of performance unless they take into account the invariance of the softmax layer and are accompanied by an analysis of the ﬁne-tuning process.
Both of these facets of neural network training allow pruned networks to accrue high amounts of approximation error in hidden layers while maintaining label-consistency with the original dense networks.
Predicting the outcome of pruning and re-training, which is an indispensible step in all effective pruning methods, will require a more sophisticated analysis of the optimization of sparse neural networks via gradient descent.
A C ORESET M ETHOD D ETAILS A. N OTATION
We are interested in pruning a feed-forward neural network with L ∈ N + layers which deﬁnes a mapping f : X → Y for an input x ∈ X ⊂ R d to an output y ∈ Y ⊂ R k .
Borrowing the notation of Baykal et al.
(2019a), let η ( (cid:96) ) ∈
N
+ denote the number of neurons in layer (
cid:96) ∈
[ L ] , where [ L ] = { 1 , . . . , L } denotes the index set. Note η (1) = d and η ( L ) =
k . Further, let η = (cid:80) L(cid:96) =1 η ( (cid:96) ) . 1 s j ← max x ∈ S w j x j (cid:80) k
∈I w k x k ; ∀ j ∈ I { importances } 2 q j = s j / (cid:80) k ∈I ;
∀ j ∈ I 3 K ∼ MULTINOMIAL ( q, m ); 4 for j ∈
I do ˆ w j ← K j w j mq j {
} 6 end 7 return
1 s j ← x ∈ S w j x j (cid:80) k
∈I w k x k ; ∀ j ∈ I { Compute importances
2 I arbitrarily do ˆ w j ← w j 5 end 6 return
1 q j ←
[Equation]
j || ) (cid:80) j (cid:48)∈ [ η (2)] max i ∈
[ k ] w (2) ij (cid:48) φ ( β || w (1) j (cid:48)
|| )
; ∀ j ∈ [ η (2) ] { Compute neuron importances } 2 K ∼ MULTINOMIAL ( q, m ); 3 for j ∈ [ η (2) ] do if K j (cid:54) =
then ˆ w (1) j ← w (1) j ∀ i ∈ [ k ] : ˆ w (2) ij ← K j w (2) ij mq j end else
ˆ w (1) j ← (cid:126) ∀ i ∈
[ k ] : ˆ w (2) ij ← end 12 end 13 return ˆ W (1) , ˆ W (2) ;
For each layer (cid:96) ∈ { 2 , . . .
, L } , let
W ( (cid:96) ) ∈
R η ( (cid:96) )
× η (cid:96) − 1 , be the weight matrix for layer (cid:96) with entries denoted by w ( (cid:96) ) ij and rows denoted by w ( (cid:96) )
i ∈ R 1 × η (cid:96) − 1 .
For notational simplicity, we assume that the bias is embedded in the weight matrix.
Finally, let ρ (cid:96) =
nnz ( W ( (cid:96) ) ) be the number of parameters in the matrix W ( (cid:96) ) and ρ ∗ = max (cid:96) ∈
[ L ] ρ (cid:96) .
We consider the activation to be the Rectiﬁed Linear Unit (ReLU) function, i.e., φ ( · ) = max {· , 0 } .
If the network is used for a classiﬁcation task, then the predicted label is arg max i (cid:2) softmax ( f ( x ))
i (cid:3) =
arg max i f ( x ) i .
A. W EIGHT P RUNING
When sparsifying weights with coreset-based methods, we can guarantee that the output of the pruned perceptron is arbitrarily close to the original if a sufﬁcient number of samples is taken.
A bound on the approximation error given a certain number of samples is presented in the following theorem:
Theorem A.1 (Baykal et al., 2019b) Let K, K (cid:
) ∈ R be constants that can be determined from
Assumption A.3.
Let I indicate the indices of the non-negative weights of the perceptron.
For δ ∈ (0 , 1) , invoking Alg 1 with m ∈ N satisfying m > ˜ S , and a set S ⊂
X
composed of (cid:100) K (cid:48) log(2 ρ/δ ) (cid:101) i.i.d.
points drawn from D generates a perceptron ˆ w such that nnz ( ˆ w ) ≤ m and for some x ∼ D ,
Pr ( | ˆ z ( x ) − z x ) | ≥ (cid:15) m z ( x ))
≤ δ where z ( x ) = (cid:80) k ∈I w k x k , ˆ z ( x ) =
(cid:80) k ∈I ˆ w k x k , (cid:15) m = (cid:32)(cid:115) ˜ S m [Equation]
+
[Equation]
(cid:33)
[Equation]
, 1) ˜ S =
SK log(4 /δ ) , and S =
(cid:80) j ∈I s j
The theorem provides a probabilistic guarantee, meaning our conﬁdence that the guarantee will hold can be adjusted via a parameter δ .
In our experiments, we use δ = 0 .
5 unless otherwise speciﬁed.
Note that this result holds only when the weights of the neural network are positive.
A generalization to all weights is given by Lemma 6 of Baykal et al.
(2019b).
This involves sparsifying the positive and negative weights separately and then combining the results, which is how we implement this method for our experiments.
Baykal et al.
(2019b) also present a deterministic version of their algorithm, which prunes the lowestsensitivity weights ﬁrst.
The approximation error can also be bounded in this case using the following theorem:
Theorem A.2 (Baykal et al., 2019b)
In the context of Theorem A.1, invoking Algorithm 2 generates a perceptron ˆ w such that for some x ∼ D .
[Equation]
where z ( x ) = (cid:80) k ∈I w k x k , ˆ z ( x ) =
(cid:80) k ∈I ˆ w k x k , and (cid:15) m
= 3 K (cid:80) j ∈ ( I\I m ) s j
The proof of both weight pruning approximation bounds requires an assumption on the Cumulative
Distribution Function (CDF) of the sensitivities of each weight.
For our work, we use values of
K (cid:48)
= 3 and K = 2 which are recommended by the authors.
Assumption A.3 (Baykal et al., 2019b) Let g j ( x ) =
w j x j / (cid:80) k ∈I w k x k , for x ∼ D .
There exist universal constants K, K (cid:48) > 0 such that for all j ∈ I , the CDF of the random variable g ( x ) j for x ∼ D , denoted by F j ( · ) , satisﬁes F j ( M j /K ) ≤ exp( − 1 /K (cid:48) ) , where M j = min { y ∈ [0 , 1] :
F j ( y ) = 1 } .
A. N EURON P RUNING
Neurons can be pruned rather than weights via Algorithm 3.
A bound on the approximation error of the pruned neural network can be guaranteed if a sufﬁcient number of samples is taken.
This is expressed in the following theorem:
Theorem A.4 Mussay et al.
Let W (1) ∈ R η
× d be the weights of the hidden layer and
W (2) ∈
R k × η (2) be the weights of the output layer, and β be an upper bound on the L2-norm of x ∈ X , as in Alg 3.
Let
[Equation]
be the sensitivities of each neuron for ∀ j ∈ [ η (2) ] , and t = (cid:80)
j s j be the “total sensitivity.”
Let c >be a sufﬁciently large constant that can be determined from the proof, and
[Equation]
be the sample size.
Let ˆ W (1) , ˆ W (2) be the output to a call to Algorithm 3.
Then, with probability at least 1 − δ ,
[Equation]
To avoid the confusion of introducing new notation, we have re-written this theorem using the notation used in Baykal et al. (2019a).
We also do not mention VC-dimension in this theorem, which is needed for the general case.
In our case, the VC-dimension of a single perceptron with ReLU activation is simply the input dimension, d .
Neuron pruning approximation bounds require knowing the maximum L2 norm of the input vectors.
This can be calculated naively; for example, MNIST has 784 input dimensions each within (0 , 1) .
Then the maximum L2 norm is √ 784 = 28 ; we could then bound the L2 norm of hidden activations by examining the eigenvalues of each weight matrix.
In practice, however, MNIST has highly sparse inputs and we observe the maximum L2 norm to be 25.
In this case, we set β to be the lower value and follow a similar empirical approach to estimate the appropriate maximum L2 norm of hidden activations.
B E XPERIMENTAL D ETAILS B. L E N ET T RAINING D ETAILS
LeNet-300-100 is a simple two-layer feed-forward neural network with 300 hidden units in the ﬁrst layer and 100 hidden units in the second layer.
All LeNet-300-100 models are trained to achieve 98% development accuracy on MNIST.
Training is carried out for 10 epochs with a standard SGD optimizer at learning rate 0.1, momentum 0.5, and a training batch size of 100.
Easy Case There are two “easy case” LeNet models, one which pre-prunes the neurons of the ﬁnal hidden layer and one which pre-prunes the weights of the ﬁnal weight matrix.
Weight pruning is done via magntiude weight thresholding (Han et al., 2015), which is the most popular form weight pruning.
Neuron pruning is accomplished via L0 regularization , which adds a penalty to the loss function of a neural network depending on the number of neurons with non-zero weights (the L0-norm of the neurons).
This loss function is computationally intractable to optimize exactly, but in practice can be optimized via a stochastic relaxation (Louizos et al., 2017).
In this method, each neuron is multiplied by a gate variable which probabilistically takes on values in [0 , 1] .
The gates are sampled from a hard-concrete distribution (Maddison et al., 2017), which is amenable to the re-parameterization trick (Kingma & Welling, 2014) and therefore to optimization via backpropogation.
After training, we prune the neurons whose gates have the highest probability of being “off.”
Speciﬁc values of L pruning hyperparameters can be found in our codebase.
Once easy-case networks are pruned, we re-train the sparse networks for another 10 epochs to regain all lost accuracy and then re-initialize the pruned weights to 0.
This results in networks that are known to be sparse and prunable with no approximation error.
The development accuracies both before and after ﬁne-tuning are shown in Figure 5.
Moderate Case
In the moderate case of training, we apply mild L1 regularization on the weights.
We tried regularization parameters [1, 0.1, 0.01, 0.001, 0.0001], and found that 0.001 was the largest value we could use without impacting development accuracy.
Difﬁcult Case
In the difﬁcult case, we do not apply any additional regularization during training.
B. T RAINING W ITH D ROPOUT
While applying L1 regularization improved neuron prunability, applying dropout at rates of 0.
and 0.5 did not (Figure 7).
While at ﬁrst surprising, this makes sense upon further consideration.
Dropout is known to “break co-adaptation” between neurons (Mianjy & Arora, 2019), meaning that any random subset of the neurons should produce approximately the same answer.
In turn, this means that no single neuron should have it
’s weights go to zero; all neurons maintain roughly the same importance.
Because of this, each neuron is assigned roughly a uniform probability of being sampled by our coreset method.
And because the sample complexity is higher than the number of neurons in the layer, this results in 0% sparsity being achieved.
This is somewhat ironic, since any particular choice of 50% of the neurons should have resulted in decent post-pruning performance.
B. VGG-16 D ETAILS AND R ESULTS
VGG-16 is a neural network architecture which is composed of several convolutional layers followed by a 3 feed-forward layers with 4096, 4096, and 1000 hidden units respectively.
All VGG-16 models are trained to achieve 92% development accuracy on CIFAR-10.
Training is carried out forepochs with a standard SGD optimizer at learning rate 0.001, momentum 0.9, and a training batch size of 32.
The full results of this section are shown in Figure 10.
“Easy case” VGG-16 networks are pruned using either weight pruning and L0 pruning, similar to the LeNet models.
Unlike the LeNet models, the ﬁnal layer of VGG-16 could be pruned to 99% before accuracy began to decrease after ﬁne-tuning (Figure 9).
“Moderate case” VGG-16 models were trained with an L1 regularization term of 0.001, which was determined in a similar manner to that of LeNet-300-100.
C F URTHER A NALYSIS OF S AMPLE C OMPLEXITY C. D OES D ATA -I NDEPENDENCE M EAN L OOSE B OUNDS ?
We might assume that the looseness of the neuron pruning bounds can be attributed to overly-lax assumptions about the data distribution (i.e. that the L2 norm of inputs is simply bounded above).
To test this hypothesis, we generate random points on the sphere used to bound input vectors and measure the approximation error on these inputs, with the results shown in Figure 11.
Randomizing the inputs does increase approximation error, but it does not increase enough to justify the scale of the bounds.
We believe this implies that the bounds can be made tighter while maintaining the data-independence assumption.
C. M ANIPULATING T OTAL S ENSITIVITY
In this section we will demonstrate that the number of samples required by coreset neuron pruning to guarantee label-consistency at any accuracy can be arbitrarily manipulated.
This involves manipulating both the total sensitivity of the layer and the distance between pre-softmax logits.
First we’ll prove the following lemma:
Lemma C.1 Suppose φ ( · ) = max(0 , · ) .
Then the “total sensitivity” t of a hidden layer, as deﬁned in Theorem A.4 for pruning neurons, can be made arbitrarily small or large by multiplying the weight matrix W (1) by a positive constant α .
The resulting neural network is label-consistent with the original.
Proof :
The ﬁrst statement is fairly straight-forward to prove.
Let ˆ W (1)
= αW (1) be a modiﬁed weight matrix of the ﬁrst layer, where α > 0 is some arbitrary constant.
Similarly, let ˆ t be the resulting total sensitivity of the network which uses ˆ W (1) , rather than W (1) .
Then
[Equation]
So scaling W (1) by α also scales the total sensitivity by α .
The fourth equality holds because max(0 , αx ) =
α max(0 , x ) and ||
αw ||
= α || w
|| 2 .
Proving label-consistency requires the positive homogeneity property of ReLU feed-forward neural networks (Du et al., 2018a), which says that multiplying a weight matrix by α results in scaling the output of the network by α .
The proof is very similar to the reasoning above.
Let f be the two-layer
ReLU network with weights ( W (1) , W (2) ) , and ˆ f be the network with weights ( ˆ W (1) , W (2) ) .
For any input x ∈ X , we have
[Equation]
Finally, we can note that scaling the pre-softmax outputs by α does not change the predicted label:
[Equation]
A simple corollary of this idea is that we can arbitrarily increase or decrease the number of samples required to achieve any accuracy guarantee.
From Lemma C.1, we know that scaling a weight matrix by a constant α increases the total sensitivity by α .
At the same time, it also increases the additive epsilon bound required guarantee each label’s consistency, because the pre-softmax logits are multiplied by α .
This is equivalent to increasing or decreasing the temperature of the softmax distribution.
Since sampling complexity grows roughly linearly with total sensitivity and shrinks quadratically with the size of the epsilon bound, the net result is that multiplying by α > 1 reduces the required number of samples to achieve any particular accuracy guarantee.
Corollary [-D.
-]{+C.
+}
In the context of Theorem A.4, multiplying the weight matrix W (1) by a positive constant α > 1 decreases the number of samples required to achieve any particular accuracy guarantee.
And indeed, we can see in Figure 12 that when we scale the layer weights of LeNet-300-100 up by α at various magnitudes, the additive epsilon bound required guarantee any accuracy grows proportionally, which quadratically shrinks the number of samples required to meet the guarantee.