TVSP RUNE - P RUNING N ON - DISCRIMINATIVE FILTERS VIA T OTAL V ARIATION SEPARABILITY OF INTERMEDIATE REPRESENTATIONS WITHOUT FINE TUNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Achieving structured, data-free sparsity of deep neural networks (DNNs) remains an open area of research.
In this work, we address the challenge of pruning filters with only access to random samples drawn from the original distribution and without access to the original training set or loss function.
We posit the following hypothesis:well-trained models possess discriminative filters, and any non discriminative filters can be pruned without impacting the predictive performance of the classifier.
Based on this hypothesis, we propose a new paradigm for pruning neural networks: distributional pruning, wherein we only require access to the distributions that generated the original datasets.
Our approach to solving the problem of formalising and quantifying the discriminating ability of filters is through the total variation (TV) distance between the class-conditional distributions of the filter outputs.
We present empirical results that, using this definition of discriminability, support our hypothesis on a variety of datasets and architectures.
Next we define the LDIFF score a heuristic to quantify the extent to which a layer possesses a mixture of discriminative and non-discriminative filters.
We empirically demonstrate that the LDIFF score is indicative of the performance of random pruning for a given layer, and thereby indicates the extent to which a layer may be pruned.
Our main contribution is a novel one-shot pruning algorithm, called TVSPrune, that identifies non-discriminative filters for pruning.
We extend this algorithm to
IterTVSPrune, wherein we iteratively apply TVSPrune, thereby enabling us to achieve greater sparsity.
Last, we demonstrate the efficacy of the TVSPrune on a variety of datasets, and show that in some cases, we can prune up to 60% of parameters with only a 2% loss of accuracy without any fine-tuning of the model, beating the nearest baseline by almost 10%.
I NTRODUCTION
Deep neural networks are, in general, highly overparameterized, leading to significant implementation challenges in terms of reducing inference time, power consumption, and memory footprint.
This is especially crucial for deployment on real world, resource constrained devices Molchanov et al.
(2019b); Prakash et al.
A variety of solutions have been proposed to solve this problem, which can broadly be grouped into quantization, sparsification or pruning, knowledge distillation, and neural architecture search (NAS) Hoefler et al.
Of these approaches, model pruning offers the most immediate gains in practical performance, without requiring specialized hardware implementations Hoefler et al.
; Blalock et al.
Pruning can be further divided into unstructured pruning, wherein individual parameters are set to zero, or structured pruning, wherein entire filters or channels are removed from the architecture Hoefler et al.
Structured pruning yields immediate improvements in inference time, memory footprint, and power consumption, without requiring any specialized software frameworks.
Unstructured pruning, on the other hand, typically yields models that are significantly more sparse than those obtained with structured pruning, but do not provide the same improvements in inference time or memory footprint Hoefler et al.
; Blalock et al.
One drawback of existing pruning methods is that they require extensive and resource-intensive fine tuning, often after each pruning step Molchanov et al.
(2019b);
Liu et al.
(2021b)
; Hoefler et al.
While this necessitated the development of data free pruning techniques, such as those proposed in Srinivas & Babu (2015); Tanaka et al. (2020); Patil & Dovrolis (2021), few works address the problem of data free structured pruning .
Here, we relax the problem of data free structured pruning, to the problem of structured pruning with access to the moments of the class-conditional distributions, but without access to either the training set or the loss function.
To facilitate this method, two crucial questions, need to be answered.
First, “what makes a filter valuable to the classification performance of the model,” and second, “how do we characterize which layers can be effectively sparsified”.
To answer the first, most works propose saliency scores that capture the importance of individual filters Hoefler et al.
; Blalock et al.
Liu et al.
(2018); however, few works suggest reasons for filters in a given layer being easy or difficult to prune.
To the best of our knowledge, there are no works that relate the “quality” of a pretrained model, and the extent to which that model may be structurally sparsified.
In this work, we offer a new perspective for pruning neural networks: distributional pruning , wherein we only have access to the distributions used to generate the training data.
In this paradigm, we address the problem of structured pruning of neural networks without access to the loss function and original training dataset, thus obviating fine-tuning of the sparsified model.
In particular, we identify discriminative filters as those capable of distinguishing between the class conditional distributions.
We formalise the discriminative ability of filters in terms of the Total Variation (TV) separation between the class-conditional distributions of their outputs.
Owing to the intractability of measuring the TV distance, we use the Hellinger lower bound to measure the discriminative ability of filters.
With these definitions, we obtain a pruning algorithm that does not require the training set, only access to the training distribution.
Furthermore, we use discriminative ability of filters to define the
LDIFF score, which measures the extent to which layers can be randomly pruned, thereby informing pruning ratios as well.
We state our contributions formally here.
We begin by formalising the notion of discriminative ability of filters in terms of the TV distance between their class-conditional outputs.
Specifically, we say a filter is TV-separable with respect to η > 0 , or TVS EP ( η ), if the pairwise minimum TV distance between classconditional distributions of the outputs is larger than the threshold η .
For tractability, we assume the class conditional distributions are gaussian with the estimated moments, and compute the Hellinger distance based lower bound to estimate whether filters are TVS EP ( η ).
With this definition of discriminative ability, we produce the empirical observation that the classwise outputs of filters present in models that generalize well tend to be well separated in the sense of TV distance.
Specifically, we observe that untrained models do not possess discriminative filters, whereas trained models do; this is confirmed over a variety of datasets and architectures.
Furthermore, we observe that when we perturb well-trained models, thus lowering the test error, the separation between the class conditional distributions of the filter outputs decreases accordingly.
We next address the problem of identifying which layers can be effectively pruned.
Using TVS EP to measure discriminability, we develop LDIFF , a deductive, threshold-based measure that captures the extent to which the filters in a layer are discriminative or non-discriminative, thereby indicating the extent to which a model can be pruned.
Specifically, if LDIFF is high, we can deduce that the corresponding layer cannot be sparsified extensively.
We empirically demonstrate the utility of this mechanism by showing that layers with low LDIFF scores can be randomly pruned extensively without significant drops in accuracy, whereas layers with high LDIFF scores cannot; therefore indicating that LDIFF scores can be used to decide pruning ratios for random pruning.
Last, in the paradigm of distributional
pruning, we use the TVS EP and LDIFF measures to develop TVSP RUNE , a layer-wise, threshold based method for structured pruning, requiring no fine tuning and only the class-conditional means of the outputs of each filter.
TVSP RUNE computes the TVS EP based scores , and removes all filters with a score below a threshold.
We also extend TVSP RUNE to an iterative algorithm, I TER TVSP RUNE , wherein we recompute the moments after each pruning step and repeat TVSP RUNE , thereby achieving more extensive sparsification.
We show that on the CIFAR-10 dataset, our method achieves over 40% sparsification with minimal reduction in accuracy on VGG models without any fine tuning; furthermore, our method outperforms contemporary methods such as Sui et al.
(2021); Lin et al.
(2020a)
; Molchanov et al. (2019b) in the regime discussed.
Our paper is organized as follows.
In Section 2, we outline our problem statement.
In Section 3, we detail the notation used in the paper, and review Total Variation distances and the associated
Hellinger distribution bounds.
In Section 4, we describe how we utilize the TV-distance to measure the discriminative capacity of filters.
In Section 6, we propose the LDIFF score for quantifying the difficulty in pruning a given layer, and MinTVS algorithm.
In Section 7, we detail our experimental results.
We produce a detailed related work section in Appendix A.
We also provide additional discussions in B, variants of the LDIFF score and the TVSP RUNE algorithms in sections C and D, and a variety of additional empirical results in section E.
F ILTER P RUNING WITHOUT T RAINING D ATA AND F INE T UNING
Structured pruning can be thought of as two specific optimization problems: finding the most accurate model that satisfies a sparsity model, and finding the sparsest network that satisfies an accuracy constraint.
We define the latter problem below.
[Equation]
Our goal is to solve equation P using structured pruning techniques; that is, we aim to remove entire filters at each pruning iteration.
As mentioned previously, typical methods to solve this problem require extensive retraining to overcome the losses in accuracy.
In this work, we consider the scenario wherein neither the training data nor the loss function are available during the pruning process.
Motivated by empirical results, we see that models that generalize well tend to have mean-separable features, whereas models that do not generalize well possess features that are not as well separated.
We refer the reader to Section 7.1 for these results.
This prompts us to formulate the following hypothesis.
Hypothesis 1.
“Models that generalize well possess a rich mixture of filters that discriminate between features and those that don’t.
Furthermore, we hypothesize that for generalization purposes, it is sufficient to retain the filters that discriminate between classes well, in those layers that have a large variation in discriminative ability.”
Note that we also observe that some layers have features that cannot discriminate well, and yet cannot be pruned.
P RELIMINARIES AND B ACKGROUND 3. N OTATION
Let [ n ] = { 1 , · · · , n } ⊂ N .
We the define parameters of a neural network with L as W = { W 1 , · · · , W L } , where each W i =
{ W i,j } N l j =1 is a collection of tensors.
For a convolutional neural network (CNN), each W l ∈ R N lin × N lout × K l × K l .
In this formulation, the i th filter of layer l is a tensor f li ∈ R N in × K × K , and the j th kernel in this filter is a matrix f li,j ∈ R K × K .
Note that given a model, we can use the pair ( l, j ) to identify a filter - l denotes the layer, and j the filter index.
We vectorize this model as follows.
First, suppose the vectorized input to layer l + 1 ∈
[ L ] is ϕ l ( X ) ∈ R m 2 N in , where N in is the number of channels and m is the dimension of the input (say, the number of pixels in each dimension of an image).
Note that we ignore the superscript indicating the layer for the sake of brevity.
Suppose each filter in layer l + 1 has kernel size K × K , that is, for each j ∈ [ J ] , the we have the vectorized W lj ∈ R K 2 N in .
Next, we assume that there are P patches, with ϕ lp ( X ) being the p th patch in the input.
In the sequel, we drop the layer superscript for convenience.
In this work, we assume we draw samples ( X, u ) , where X is that datum and u is the class label, from a distribution D .
We write D α is the class conditional distribution of class α .
Here X ∈ R N for some integer N is the input, and u ∈ [ N classes ]
⊂ N is the integer class.
3. R EVIEW OF T OTAL V ARIATION D ISTANCE
In this section, we formally define the Total Variation and Hellinger distances.
We define this formally in the sequel.
The results that follow are discussed in, say, Kraft (1955).
Definition 1.
Let Q 1 and Q 1 be two probability measures supported on R d .
We define the Total
Variation Distance TV as
[Equation]
Unfortunately, no closed form expression exists for the Total Variation distance, even when Q 1 , Qare Gaussian.
This motivates us to use the Hellinger distance, which we define below.
Definition 2.
Let Q 1 , Q 2 be two probability measures supported on R d , and let q 1 and q 2 be the corresponding densities.
We define the squared Hellinger distance as
[Equation]
Crucially, the Hellinger distance provides us upper and lower bounds on the TV-Distance.
[Equation]
If we have Q 1 ∼ N ( µ 1 , σ 21 I ) and Q 2 ∼ N ( µ 2 , σ 22 I ) are Gaussian, we get
[Equation]
M EASURING D ISCRIMINATIVE A BILITY OF F ILTERS
In this work, we take a distributional view of the discriminative ability of filters.
Previously, we have described the TV-distance, and the Hellinger lower bound that can be efficiently estimated.
In this section, we apply the TV-distance to arriving at a means to quantify the discriminative ability of individual filters by using the class-conditional distributions of the outputs.
We begin by recasting the dot products used in the convolution operations with vectorized filters and feature maps.
Define y j,p ( X ) to be the dot product of the j th filter and the p th patch of ϕ ( X ) ; that is
[Equation]
Following from this, we can write
[Equation]
where Φ l − 1 ( X )
=
[ ϕ l − 1 1 ( X ) , · · · , ϕ l − 1 P ( X )]
We define the Class Conditional Means and Variances for Y ( X ) , the output of a given filter as follows.
[Equation]
Assumption 1.
We assume that the class conditional distributions of each filter is Gaussian, that is
[Equation]
This assumption enables us to use the closed form expressions described in equation 2.
While this assumption is a strong one, it is well motivated in the literature.
The assumption that class conditional distributions are gaussian is used in a variety of settings, including Sun et al.
Lee et al.
(2020); Seetharaman et al.
Our goal is to quantify the discriminative ability of the outputs of the individual filters, and to do so, we use the TV-Distance between the class conditional distributions.
Assumption 1 enables us to estimate lower bounds on the TV distance between any two class-conditional distributions of a filter’s output easily, using only samples from that distribution.
With that in mind, we define the Minumum TV Separation (MinTVS) between classes of a given filter.
Definition 3.
Suppose Asssumption 1 holds.
Let D αl,j be the class-conditional distribution of Y l,j ( X ) where ( X, u ) ∼ D α .
For each layer l and filter j , and each pair α, β , we define the Minimum Mean
Separation of filter j in layer l as
[Equation]
Furthermore, since the harmonic mean in equation 2 is upper bounded by 1, we have
[Equation]
thus allowing us to use ∆ α,βl,j as a surrogate for MinTVS( l, j ) during experimentation.
Using this quantity, we define TV-Separability.
Definition 4.
For each layer l and filter j , and real number η > 0 we say the filter is η -TV Separable, or TVS EP ( η ) if
[Equation]
Since we cannot directly compute the pairwise minimum of the TV-separations between the classconditional distributions of a filter’s output, we instead rely on the lower bound proposed above.
The value of η that indicates a “well separated” filter will vary from dataset to dataset, or architecture to architecture; for instance, for the VGG models trained on CIFAR10 used in our experiments, we find η = 0 .
05 is useful.
Further refinements are discussed in sections D and C.
I DENTIFYING D IFFICULT - TO -P RUNE L AYERS
In this section, we investigate the problem of identifying which layers of a given neural network can be pruned efficiently, or are pruneable .
Roughly speaking, a layer is difficult to prune if removing a few filters randomly yields a significant drop in accuracy.
To address this problem, we rely on the hypothesis that discriminative filters contribute to generalization performance whereas nondiscriminative filters do not, and argue that we can effectively prune layers containing a mix of discriminative and non discriminative filters.
Thus, the task of identifying which layers are pruneable.
More specifically, we desire an informative heuristic that performs two purposes:
1. indicate the extent a layer can be randomly pruned, thus informing pruning ratios for random pruning; 2. indicate whether or not a layer should be pruned at all.
Note that we leverage the latter purpose in our proposed algorithms in the sequel.
In this section, we define LDIFF , a heuristic score assigned to each layer based on the TV separations of the constituent filters to the aggregate TV-separation, and that satisfies the stated requirements.
Our goal is to capture the mix of discriminative and non-discriminative filters in a single heuristic indicator.
Thus, given a threshold for discriminative ability η , suppose the fraction of discriminative filters is τ ( η ) .
We need a function that penalizes τ ( η ) close to either 1 or 0 (as having no discriminative filters in a layer indicates difficulty in pruning).
Thus, we, define
[Equation]
If this quantity is close to 1, then that layer has a mix of discriminative and non-discriminative filters.
If not quantity is low, then there is either a majority of discriminative filters, or non-discriminative filters; in either case, pruning the model can be challenging.
Thus, this heuristic can also be used to inform random pruning ratios.
For instance, we may choose γ LDIFF ( l, ν ) for some γ ∈ (0 , 1) to be the random pruning ratio.
Further refinements of this heuristic include layerwise thresholds, and using statistics of the MinTVS scores to determine discriminability.
We refer readers to Section C for further discussion on such refinements, as well as empirical support for the use of LDIFF scores for indicating pruneability of layers.
D ISTRIBUTIONAL A PPROACHES TO S TRUCTURED P RUNING
In this section, we discuss our solution the problem of structured pruning of neural networks without access to the loss function or the original data, and only with access to the means of the outputs of the individual filters and layers.
Our proposed algorithm has three steps.
First, for a given η , we estimate the η -TV-separability of the filter outputs; second, we decide which layers we can prune; third, in those layers which we can prune from, we remove all filters with TV-separability less than η .
6. T HE TVSP RUNE A LGORITHM - V ARIABLE P RUNING R ATIOS PER LAYER
The goal of this algorithm is to leverage the discriminative ability of filters to decide whether a given filter should be pruned.
Thus far, we have defined the MinTVS value, which is a lower bound on the least classwise TV-separation of a filter’s output, as well as the LDIFF score, which, in a sense, quantifies the extent to which a layer can be sparsified.
The algorithm begins by measuring the TV-separability of all the filters using { MinTVS( l, j ) }
N lout j = as defined in equation 10.
Next, we compute the LDIFF values as described in equation 6.
We then check the LDIFF score agains the threshold ν , and thereby decide whether or not to prune the given layer.
Then, all filters in each prunable layer (that satisfies LDIFF ( l ) > ν ) that are not η -TV
Separable be pruned.
The value of η dictates the aggressiveness of the pruning as it is a measure of how “well separated” we desire - if η is small, fewer filters are pruned.
This approach is useful since it ensures that the pruning ratio varies from layer to layer, and that we prune those layers that are relatively difficult to sparsify far less aggressively.
Furthermore, we observe that some layers are dramatically more difficult to prune from than others.
Therefore, we use the LDIFF scores to decide which layers to prune. We now present the TVSP RUNE algorithm.
We assume that we have a labeled dataset ˆ D of pairs ( X, u ) that was not used for training.
The TVSP RUNE algorithm then uses this dataset to compute the moments ¯ Y αl,j , σ 2 l,j,α for each layer l , filter j , and class α .
Using this data, TVSP RUNE then computes MinTVS( l, j ) for each l, j ; after this, the mean m l and standard deviation σ l are computed for the TVS EP values { MinTVS( l, j ) } for each layer l .
After these are computed, those filters that satisfy MinTVS( l, j ) ≤ η are pruned.
6. I TERATIVE P RUNING WITH TV S EPARATION
We motivate this section by noting that the TVSP RUNE algorithm does not give us control over the accuracy of the pruned model, or the extent that a model is sparsified.
Therefore, it is unsuitable to solve either equation P .
In this section, we describe I TER TVSP RUNE , an iterative algorithm that builds upon TVSP RUNE , and which attempts to solve equation P. We also propose a variant of I TER TVSP RUNE that attempts to solve the problem of finding the most accurate model given a sparsity budget, without training data or access to the loss function.
We refer readers to the supplemental material for this algorithm description.
The I TER TVSP RUNE algorithm builds upon the fact that the distribution over Y l ( X ) changes each time the previous layers are pruned.
Thus, we may recompute the TV-Separation values, and iteratively prune the model.
Note that as was the case with TVSP RUNE , this algorithm does not require any fine-tuning, and does not even require the training data.
The I TER TVSP RUNE algorithm takes an initial TVS EP threshold η , a convolutional neural network with parameters W , an LDIFF threhsold ν , and an accuracy threshold t .
At each step k , we run the
TVSP RUNE algorithm given the current parameters W ( k ) and a current TVS EP threshold η k ; that is ¯
W = TVSP RUNE ( W ( k ) , η k , D ) .
After running TVSP RUNE , we check the accuracy of the pruned model - if f ( ¯ W ) ≤ t , then we repeat the pruning step, else we reduce η and run TVSP RUNE with
W ( k ) and the new value of η .
The utility of varying the TVS EP threshold is that as we increase the threshold, the number of filters whose MinTVS value exceeds the threshold reduces.
Thus, as we reduce η , we increase the acceptable degree of TV-separation, thus making the pruning more gentle.
We state the algorithm formally in Algorithm 5.
Input: Dataset ˆ D =
{ X i , u i } Mi =1 , Pretrained CNN with parameters
W =
( W 1 , · · · , W L ) ,TVS EP threshold η , layer difficulty threshold ν
Compute ¯ Y l,αj , σ 2 l,j,α for all l, j, and α .
Compute ∆ α,βl,j for each l, j, α, β . for l ∈
[ L ] do Compute MinTVS( l, j ) for all l, j
Compute LDIFF ( l ) if LDIFF ( l ) <
ν ∨ LDIFF ( l ) =
then for j ∈ [ N lout ] do if MinTVS( l, j ) <
η then
W lj ←
Output: ˆ W =
( ˆ W 1 , · · · , ˆ W L ) , where supp ( ˆ W l ) ≤ supp ( W l ) ∀
l return ˆ W
Input: Dataset ˆ D =
{ X i , u i } Mi =1 ,
[Table]
CNN W =
( W 1 , · · · , W L ) , initial
TVS EP threshold η ,
[Table]
threshold ν , δ η > 0 , accuracy threshold t , η
[Table]
< η .
Set
[Table]
=
[Table]
, W (0) =
W while η ≥ η min ˜ W = TVSP RUNE ( D , W ( k ) , η, ν ) if f ( ˜ W ) ≤ t then
W ( k +1)
← ˜ W else η ← η − δ η W ( k +1)
← W ( k ) k ← k +
Output: ˆ W =
( ˆ W 1 , · · · , ˆ W L ) , f ( ˆ W ) ≤ t , supp ( ˆ W l ) ≤ supp ( W l ) ∀
l return ˆ W
E XPERIMENTAL R ESULTS
In this section, we detail our experimental results.
We aim to compare our method with various existing structured pruning techniques without any fine tuning.
Our experiments utilize the standard
CIFAR-10 Krizhevsky et al.
and Imagenet Russakovsky et al.
(2015) datasets under the MIT license.
For computing means, we partition the test set, which contains 10,000 samples, into two subsets of size 7,500 and 2,500 in the case of CIFAR10; for Imagenet, we partition the validation set into two subsets of size 40000 and 10000.
We use the larger subset for computation of means, and the smaller subset for testing the accuracy of the pruned model.
For our hardware setup, we refer readers to section E.
We aim to answer three broad questions with our slate of experiments.
First, do well trained models have more well-separated features than models that are not well trained?
Second, using the
LDIFF mechanism that utilizes the separability of intermediate features, can we gauge the “difficulty” of pruning a given layer?
And last, how does our pruning mechanism compare with other existing methods in the scenario where training data and the loss function are unavailable?
For further experimental results, we refer the reader to the supplementary material.
7. W ELL -T RAINED MODELS HAVE WELL SEPARATED FEATURES
In this set of experiments, we aim to show that models that generalize well have well-separated features, whereas models that generalize poorly do not.
To measure separation, we compute
[Equation]
that is, the mean minimum classwise TV-separation of filters in each layer.
In order to obtain models that do not generalize well, we modify weights of a model with high test accuracy by adding zeromean gaussian noise to the weights.
In our experiments, we progressively increase the variance of the gaussian noise, in order to increase the test error.
Furthermore, measure MinTVS( l ) values for untrained models for additional comparison.
Our experiments focus on VGG16 and ResNet models trained on the CIFAR10 and Imagenet datasets.
For both architectures, and for both datasets, we observe that as the variance of the noise increases (and thus, the test error decreases), the average
TV-separation per layer decreases.
We also observe that the average TV-separation for untrained models remains low, and almost constant.
Curiously, we note that even with the addition of Gaussian noise, the separation does not significantly change in the layers close to the input, particularly for
VGG models.
This therefore supports our hypothesis that models which generalize well possess discriminative filters, whereas models that do not generalize well do not.
[Figure]
7. D ETERMINING WHICH LAYERS ARE DIFFICULT TO PRUNE WITH LDIFF
In this section
[Figure]
we investigate the utility of the LDIFF score for determining which layers can be extensively pruned
[Figure]
and which can’t.
The experiment we conduct is as follows.
First, we compute the
[Figure]
LDIFF scores for the VGG16 -19 models trained on the CIFAR10 and CIFAR100 datasets.
We use 512 samples for CIFAR10, sampled from the appropriate partition of the test set, and 4096 samples for CIFAR100. Then, we prune weights randomly from each layer in isolation; that is, for each experiment, we only prune weights from a single layer.
We then measure the test accuracies.
We observe that the LDIFF scores are small for the initial layers, thus indicating that those layers cannot be randomly pruned.
This is borne out by our experimental results for both models trained on the CIFAR10 and the CIFAR100 dataset.
Last, we infer, based on the LDIFF scores and test accuracies of randomly pruned models, that pruning ratios of γ LDIFF ( l, η ) , where γ < 1 / 2 would result in negligible loss of accuracy.
Collectively, these observations support the use of the LDIFF score for determining the extent to which a layer can be pruned.
These results are displayed in 5.
Further experiments, including computations of Spearman correlations between the layerwise accuracy drops and the LDIFF scores are provided in the supplemental material in C.
[Figure]
[Table]
7. E FFECTIVENESS OF I TER TVSP RUNE FOR S TRUCTURED P RUNING WITHOUT F INE -T UNING
In this section
[Table]
we compare the I TER TVSP RUNE algorithm with some existing baselines [Table] in the setting where we do not have access to the training set or loss function. We consider VGG
[Table] and -19 models trained on CIFAR [Table] and ResNet50 and -56 models trained on Imagenet and CIFAR respectively.
In this set of experiments [Table] we run I TER TVSP RUNE to solve equation P for each model and dataset; then, having obtained the pruned model satisfying the accuracy constraint, we compare with three baselines, using the same sparsity pattern as obtained by I TER TVSP RUNE .
The baselines chosen were L 1 based pruning, CHIP Sui et al.
(2021) (for ResNet models) and a first order gradient based score based on Prakash et al.
; Molchanov et al.
(2019a) (for VGG nets), and uniform random pruning (accuracy is the average of three trials of random pruning) ; all were run without fine tuning.
Modifications to implementations are discussed in
Appendix E. We report our results for
VGG nets on CIFAR10 in Figure 7, and for ResNet models in Table 1.
We observe that our method consistently outperforms the baselines in the task of structured pruning of models without fine tuning the model with the training set.
Notably, for VGG19 trained on CIFAR10, we are able to remove more than 60% of parameters with minimal loss in accuracy, far exceeding the nearest baseline.
C ONCLUSIONS AND D ISCUSSION
In this work, we propose a new paradigm for pruning, which we call distributional pruning, which only requires access to the data distribution.
We make the observation that models that generalize possess a mixture of filters that discriminate well, and those that discriminate poorly.
Motivated by these observations, we argue that we can prune non-discriminative filters to avoid significant loss of test error.
We use the TV Distance to quantify the discriminative ability of filters, using which we define the heuristic LDIFF score and derive the TVSP RUNE Algorithm.
The TVSP RUNE algorithm’s drawbacks are as follows.
First, since classwise distances must be computed, the number of separations for each filter that the algorithm needs to estimate are quadratic in the number of classes; limiting the scalability of the algorithm.
Next, the relationship between
TV-separability and generalization in DNNs is not fully understood, and requires further investigation.
Future research directions include (a) improving the scalability of this algorithm to massive datasets, (b) evaluating other measures of similarity between distributions, and (c) analyzing parameter quantization through the lens of distributional separability.
A PPENDIX
In this supplemental material, we provide additional detail and empirical results to support the claims of the main paper.
Code for experiments can be found here 1 .
We organize this document as follows.
In section A, for the sake of comprehensiveness, we provide further discussion of relevant literature, including a discussion on the discriminative ability of convolutional filters.
In section B, we present additional discussion required for clarification purposes, including a brief discussion adopting the machinery used in this paper to feedforward models.
In section C, we provide further details regarding the derivation of the LDIFF scores.
Furthermore, we provide additional empirical evidence to show that LDIFF scores are useful for determining whether layers can be effectively pruned or not, by computing Spearman correlations between the LDIFF score and the reduction in accuracy after randomly pruning the corresponding layer.
In section E, we provide additional experiments in using the TVSP RUNE and TVSP RUNE algorithms for structured pruning.
We provide plots showing the number of filters pruned in each layer for the same experiments presented in Section 7.3 of the original paper.
In section D we state a simple variation of the I TER TVSP RUNE algorithm that aims to solve a slight variation of the pruning problem.
In the sequel
we list additional material added to this document.
In section E, we provide additional experiments illustrating the utility and effectiveness of the I TER TVSP RUNE algorithm.
A A DDITIONAL L ITERATURE S URVEY
In this section, we survey contemporary and historic results in the area of neural network pruning.
We focus on filter pruning in this work, but there exists a wide variety of works addressing problem of unstructured, or weight pruning.
For a more detailed discussion on relevant results in unstructured pruning, refer to Hoefler et al.
; Blalock et al.
Filter Pruning Filter pruning refers to processes by which entire filters or channels are pruned from a model; this can be further extended to individual filtersZhong et al.
(2021) or ’stripes’ Meng et al.
(2020) within filters.
Initial results on structured pruning removed filters based on the L 1 norm of the weight tensors Li et al.
More recent advances involve using gradient based scores Molchanov et al.
(2019b);
Liu et al.
(2021b)
; Molchanov et al. (2019a), rank minimization Lin et al. (2020a), and linear replaceability and channel independence Joo et al.
; Sui et al.
(2021), and other geometric properties
He et al.
Li et al.
Lin et al. (2020b); Yu et al.
Other works, such as Liebenwein et al.
; Baykal et al.
(2018) developed sampling based methods for structured pruning that also provided generalization bounds.
Lastly, in Zhuang et al.
Liu et al.
(2021a), the discriminative capacity of filters is used to inform the pruning strategy by introducing discrimination-aware loss functions.
Our work differs from these in two key respects - first, we rank filters based on the discriminative ability of their outputs, and second, we use a novel metric based on the discriminative ability of filters, based on classwise distances between class-conditional output distributions of filters, to decide which layers can be pruned effectively.
Data-Free Pruning Data
free pruning has recently gained significant interest Hoefler et al.
(2021); however, while there are a variety of methods that do not use data for the task of pruning, there are few methods that do not require fine tuning or retraining.
Early works in this regime include Srinivas & Babu (2015), which measured similarity between neurons, and merged similar neurons together.
More recently, Tanaka et al.
(2020) proposed the SynFlow method, an unstructured, data free pruning method that relied on preserving gradient flow.
This work was expanded upon in Gebhart et al.
; Patil & Dovrolis (2021), which utilize the Neural Tangent Kernel to improve upon SynFlow.
Other approaches include Mussay et al.
(2021), which uses coresets of weights to perform structured pruning. Our work differs from this since we utilize the class conditional distributions of the filter outputs to rank the filters, which we either obtain a priori , or compute using previously unseen data.
In this section, we briefly survey works pertaining to discriminative ability of neural networks, and unstructured pruning.
We do so for the sake of comprehensiveness, and to discuss works that are tangentially relevant to ours.
Discriminative Ability of Convolutional Neural Networks Studying the discriminative ability of features and representation at either the layer level or the filter level is an active area of research, with applications in speech identification Yadav & Rai (2018), scene classification Zuo et al.
(2014), face recognition Wen et al.
(2016), and domain adaptation Lee et al.
Understanding the discriminative capacity of features is also crucial toward understanding adversarial robustness Jetley et al.
; Ilyas et al.
and model interpretability Bau et al.
In Wong et al.
(2021), pretrained set of convolutional layers is given as the input to a linear classifier which is trained with a sparsity-inducing regularizer to identify discriminative filters.
In Ortiz-Jimenez et al.
(2020), the discriminative capacity of intermediate representations is used to analyze decision boundaries of
CNNs.
However, to the best of our knowledge, there are no works that utilize the discriminative capacity of filters for pruning models.
Discriminative Filters in Pruning Discriminative filters have also been applied to structured pruning of neural networks.
In Zhuang et al.
Liu et al.
(2021a), the discriminative capacity of filters is used to inform the pruning strategy.
In particular, the authors define the discriminative ability of a filter as the impact of it’s ouptut on the final classifer, and introduce a discrimination aware loss function, with which the model is fine tuned.
Once this is done, the authors propose a greedy algorithm for channel selection that makes use of this loss function and the associated gradients.
This method differs from this work, as we explicitly define discriminative ability in terms of the TV separation, and not on the impact on the final classifier.
Unstructured Pruning
In this section, we provide a brief overview of unstructured pruning techniques.
While unstructured pruning is not the focus of this work, we provide this summary in the interest of completeness.
and Unstructured pruning refers to processes by which individual weights are pruned from a model Hoefler et al.
Early works on unstructured pruning include LeCun et al.
; Hassibi & Stork (1992), which relied on diagonal approximations of the Hessian of the loss function.
However, such methods are not scalable to very large models, prompting the creation of unstructured pruning methods that married a variety of techniques to achieve dramatic parametric sparsity Han et al.
; Iandola et al.
More recently, Frankle & Carbin (2018) proposed the Lottery Ticket Hypothesis, which asserts that networks contain subnetworks which, when trained from scratch, achieve similar accuracies to the original model; stronger versions of this result were proved in Malach et al.
; Pensia et al.
This remains an active area of research Blalock et al.
B A DDITIONAL D ISCUSSION
In this section, we include additional discussion on certain aspects of this work.
In particular, we briefly discuss the extension of our method to linear, feedforward networks, and provide additional discussion on the definitions and use of patches.
B. E XTENSION OF PROPOSED METHODOLOGY TO LINEAR LAYERS
Our proposed methodology can easily be extended to feedforward/linear layers as well.
Broadly speaking, we consider the layerwise outputs to be the outputs of linear layers, as opposed to convolutional layers.
In that case, it is possible to derive the same measures of distributional discrepancy.
We provide a more rigorous derivation below:
Let Φ l − 1 ( X ) be the output of the l − 1 -th linear layer, and let W l be the weight tensor of the l -th layer.
Then,
[Equation]
If Y l ( X ) is a vector, let Y jl ( X ) be the j th element (the j th neuron).
Then, we simply define ¯ Y α
= E ( X,u ) ∼D
α
[ Y ( X )] and σ 2 l,j,α = E
( X,
[Equation]
∼D
[Equation]
as we did for convolutional layers.
However, in this case, outputs are vectors.
Thus, each Y jl is a single real number (along with the expectations and the variances).
Using this, we can then define
[Equation]
and
[Equation]
where
[Equation]
B. D ESCRIPTION OF P ATCHES
A patch is a subset of the input datum with which a dot product with the convolutional filter is taken.
The input datum is made up of several patches which may or may not overlap.
We describe this formally in the sequel.
For simplicity, assume that the input is a in R M × M , and each convolutional filter is a tensor in R K × K where K < M .
In a convolution operation, the filter is applied to subsets of size K × K of the input; that is, the output is a dot product of the filter and that subset of elements of the input.
Each such subset is called a patch .
We describe this further in Section H of the supplemental material.
Furthermore, we refer the reader to, for example, [5].
C T HE LDIFF S CORE : A DDITIONAL C ONTEXT AND E MPIRICAL A NALYSIS
In this section, we provide additional context and information regarding the LDIFF score described in
Section 6.1, which is a heuristic used to quantify the extent to which a layer may be randomly pruned.
C.
R EFINEMENTS OF THE LDIFF S CORE
In this section, we discuss refinements of the LDIFF score presented in section 5.
Recall that in this work, we focus on identifying and pruning non-discriminative filters from individual layers.
Thus, as mentioned previously, our goal is to derive a score, leveraging the aforementioned hypothesis and using the discriminative ability of the filters in a given layer, that we can use to deduce whether or not a layer is difficult to prune.
Assuming the hypothesis is true, a good measure of the extent to which a model can be pruned could be given by the fraction of filters that are highly discriminative.
Thus, if a layer has only a few highly discriminative filters, it should be possible to sparsify that layer extensively.
In the LDIFF score defined in equation 6, we fix the threshold to identify discriminative filters.
We first present methods with fixed thresholds that utilize statistics of the layerwise MinTVS scores here.
C.1. M EDIAN -B ASED LDIFF S CORES WITH A FIXED THRESHOLD
We begin by recalling definitions from the original paper.
Let (cid:92) MinTVS( l ) be the median value of MinTVS( l, j ) for a given l (that is, with respect to j ).
We recall that a filter is discriminative if it is TVS EP ( η ) for some η > 0 ; furthermore, recall that η can vary from model to model, and dataset to dataset.
With this, we first define the following sets.
• Suppose that for at least one j , the corresponding filter is TVS EP ( η ) for some η > 0 .
We define the sets [Equation] • The set S η,l captures the number of filters that are not TVS EP ( η ) , and thus nondiscriminative.
• The set ˆ S l is the set of ∆ ∗ l,j greater than the median.
Let | S | denote the cardinality of a set S .
We then define
[Equation]
The set S η,l ∩ ˆ S l captures the number of non-discriminative filters in the top two quantiles.
Thus,
LDIFF 0 ( l ) captures the fraction of features in the top two quantiles that are discriminative.
If this quantity equals 1 , we conclude that at least half the filters are TVS EP ( η ) ; thus LDIFF 0 captures the mix between discriminative and non-discriminative filters.
Thus, if LDIFF 0 ( l ) is close to 1 , then the layer is difficult to prune.
This is illustrated in Figure 9, where we plot the LDIFF 0 ( l ) scores for each layer for VGG16 and
VGG19 models trained on the CIFAR10 and CIFAR100 datasets.
We choose η = 0 .
025 , and we see that the initial layers of VGG16 and VGG19 models trained on CIFAR10 appear to be very easy to prune, since there should be no discriminative filters.
Instead, however, those layers cannot be effectively sparsified.
In order to compensate for this, we chose another way to quantify the mix between discriminative and non-discriminative filters in a layer.
Specifically, we define LDIFF as in equation 6, which captures the ratio of the separations that are below the median ˆ∆ l , and those above it.
As we will show in the sequel, this is a more effective variant for capturing the difficulty of pruning filters, or indeed weights, from a layer.
[Equation]
IFF
[Equation]
scores for VGG
[Equation] [Equation]
CIFAR
[Equation]
for
[Equation]
CIFAR
[Equation]
scores for VGG
[Table]
Here, we observe that the LDIFF 0 score is effective at determining whether a layer is difficult to prune for models trained on CIFAR100, but is seemingly not as effective for models trained on CIFAR10.
In particular, for models trained on CIFAR10, we observe that the initial layers receive low scoressince filter outputs for those models are not well separated.
This lends credence to the notion that if there are no filters in a layer that are TVS EP ( η ) , then we should avoid pruning that layer entirely.
In the sequel, we discuss variants of the scores defined herein and propose variants, including the score proposed in the final paper.
C. F LEXIBLE T HRESHOLD LDIFF S CORES
A potential drawback of using fixed thresholds for discriminative ability, however, is that the scale of the distributional separation may vary from layer to layer.
That is a value of η that ensures discriminative ability in one layer may not apply to another.
In this section, we extend this definition to a more flexible variant, wherein we utilize layerwise thresholds.
We consider multiple variants, including cases where thresholds are fixed a priori and variants where thresholds are fixed based on statistics of the MinTVS values for a given layer.
Furthermore, layerwise thresholds for discriminative ability provide the added benefit of being able to capture the ‘mix’ of discriminative and nondiscriminative filters.
C.2. C OUNTING -B ASED LDIFF S CORES USING L AYERWISE MinTVS S TATISTICS
In this section, we focus on methods that utilize the statistics of MinTVS values for a given layer to determine pruning thresholds.
A variant of the method used in the main paper is as follows.
Define MinTVS( l ) to be the mean MinTVS for a given l (that is, with respect to j ), and let γ >be a constant.
Then, define
[Equation]
The LDIFF C ( l ) score utilizes the mean MinTVS score for a layer to capture the relative discriminative ability of filters in a layer, along with a constant γ l .
For each layer, this measure counts the number of filters that are discriminative relative to the MinTVS score for that layer.
As mentioned previously, useful thresholds of discriminative ability can vary significantly from architecture to architecture, layer to layer, and dataset to dataset.
C.2. M ASS - BASED LDIFF S CORES USING L AYERWISE MinTVS S TATISTICS
The previous counting-based measures of pruneability, even with flexible thresholds, fail to account for the wide range of values that MinTVS scores can take, particularly in wide networks with a small number of very discriminative filters.
In order to address this issue, we define the following variant of the LDIFF score.
[Equation]
This variant of LDIFF score compares the ratio of MinTVS( l, j ) scores that lie below γ MinTVS( l ) to the sum of MinTVS( l, j ) scores that lie above γ MinTVS( l ) .
Thus, we compare the ‘mass’ of the discriminative ability of scores below the (variable) threshold to those above it.
Hence, we use the term “Mass-based” LDIFF scores.
This captures the spread of discriminative ability, and can be shown to more effectively predict which layers can be randomly pruned.
[Equation]
[Figure]
D V ARIANTS OF I TER TVSP RUNE
In this section, we describe variants of the TVSP RUNE algorithm.
First, we describe a method that utilizes flexible thresholds, and second, we describe a variant to solve the pruning problem with a fixed sparsity budget.
D. TVSP RUNE WITH FLEXIBLE THRESHOLDS
This section describes a variant of the TVSP RUNE and I TER TVSP RUNE algorithms that utilize variable thresholds for discriminative ability.
In this case, we specifically use the mean MinTVS score to determine layerwise pruning thresholds.
Furthermore, we use the mass-based LDIFF score described in equation 14 to determine which layers to prune.
Input: Dataset ˆ D =
{ X i , u i } Mi =1 , Pretrained CNN with parameters
W =
( W 1 , · · · , W L ) , γ > 0 , layer difficulty threshold ν
Compute ¯ Y l,αj , σ 2 l,j,α for all l, j, and α .
Compute ∆ α,βl,j for each l, j, α, β . for l ∈
[ L ] do Compute MinTVS( l, j ) for all l, j
Compute MinTVS( l ) =
1 N lout (cid:80) N lout j =1 MinTVS( l, j )
Compute LDIFF ( l ) from equation 14 if LDIFF ( l ) <
ν then for j ∈ [ N lout ] do if MinTVS( l, j ) <
γ MinTVS( l ) then
W lj ←
Output: ˆ W =
( ˆ W 1 , · · · , ˆ W L ) , where supp ( ˆ W l ) ≤ supp ( W l ) ∀
l return ˆ W
Input: Dataset ˆ D =
{ X i , u i } Mi =1 , Pretrained CNN W =
( W 1 , · · · , W L ) , γ 0 > 0 ,
LDIFF threshold ν , δ γ > 0 , accuracy threshold t , γ min < γ .
Set k =
[Table]
, W (0) =
W while γ ≥ γ min ˜ W = TVSP RUNE mean ( D , W ( k ) , γ, ν ) if f ( ˜ W ) ≤ t then
W ( k +1)
← ˜ W else
γ ← γ − δ γ W ( k +1) ← W ( k ) k ← k +
Output: ˆ W =
( ˆ W 1 , · · · , ˆ W L ) , f ( ˆ W ) ≤ t , supp ( ˆ W l ) ≤ supp ( W l ) ∀
l return ˆ W
This algorithm functions in a similar fashion to the TVSP RUNE and I TER TVSP RUNE algorithms, except that the thresholds for discriminative ability are flexible (varying between layers), and are decided by the mean of the MinTVS scores for each layer.
Furthermore, just as the threshold for discriminative ability were reduced at each iteration of I TER TVSP RUNE , we vary the constant γ , thus enabling us to flexibly prune each layer even as the distribution of MinTVS scores changes with each pruning iteration.
D. F IXED -B UDGET P RUNING
In this section, we briefly describe a variant of the I TER TVSP RUNE algorithm that, using only the data-generating distribution, prunes a model with a fixed sparsity budget using only the datagenerating distribution.
This variant attempts to solve the problem of finding the most accurate model subject to a sparsity budget.
Formally speaking, this variant of the algorithm attempts to solve
[Equation]
where ∥W∥ 0 = (cid:80) l ∥ W l ∥ 0 and where K is a sparsity budget satisfying K < ∥W∥ 0 .
While a simple option is to simply increase η in the TVSP RUNE algorithm (thus raising the threshold as to which filters are discriminative), the proposed variant makes use of the fact that the distributions of the filter outputs change at each iteration, owing to the filters pruned in the previous step This algorithm,
Algorithm 5: I TER TVSP RUNE -SB
Input: Dataset ˆ D =
{ X i , u i } Mi =1 , Pretrained CNN W =
( W 1 , · · · , W L ) , initial TVS EP threshold η ,
LDIFF threshold ν , δ η > 0 , sparsity budget K , η min < η .
Set k = 0 , W (0) =
W while η ≥ η min ˜ W = TVSP RUNE ( D , W ( k ) , η, ν ) if ∥ ˜ W∥ 0 ≥ K then
W ( k +1)
← ˜ W else η ← η − δ η W ( k +1)
← W ( k ) k ← k +
Output: ˆ W =
( ˆ W 1 , · · · , ˆ W L ) , ∥ ˆ W∥ 0 ≤ K , return ˆ W
I TER TVSP RUNE -SB (SB refers to sparsity budget), iteratively calls the TVSP RUNE algorithm to prune a fraction of filters at each step.
At the k th iteration, if ∥ ˜ W∥ 0 < K ; that is, the fraction of parameters drops below the sparsity budget K , we rewind to the previous value of W k , and reduce η , thereby ensuring that we prune more gently.
The drawback of using this algorithm is that η min needs to be chosen carefully so that we are guaranteed to prune some filters at each iteration.
While we do not directly control for accuracy, we rely on our hypothesis that we may remove non-discriminative filters without suffering catastrophic losses in accuracy.
E A DDITIONAL E XPERIMENTS USING I TER TVSP RUNE
In this section, we provide additional experimental details on the use of the I TER TVSP RUNE algorithm.
H ARDWARE S ETUP
Our experiments were conducted on two systems, one with an NVIDIA GTX
GPU and an Intel i7-7700, and one with dual NVIDIA RTX3090 GPUs and an Intel i9-11900F. All experiments are done using the PyTorch framework, in particular, for obtaining the output features for each filter.
Moments and TVS EP values were computed using standard numpy packages.
D ATA S ET S PLITS
We briefly discuss the split of the data used in our experiments.
As mentioned in the main paper, we do not make use of the training data at all.
However, since we are unable to obtain sufficient data that would be drawn from the class conditional distributions of the CIFAR10 and CIFAR100 datasets (i.e. obtaining a large number of images of cats, planes, horses etc), we utilize the test data.
We partition the test data into two, with one partition receiving 7500 images, and the other 2500.
We describe this in Table 2.
[Table]
B ASELINE METHODS
In this section, we briefly describe our modifications to common baselines.
CHIP Sui et al.
We ensure that the algorithm only has access to the validation set, which is split into a 40000/10000 pair.
When running the algorithm, we limit the number of samples seen by the algorithm to 512 (1 batch of 512) for CIFAR10.
For Imagenet results, we reduce the number of repetitions to 1. 2.
Taylor first order methods: we produce a bespoke implementation of the methods described in Molchanov et al.
(2019a;b). We allow the algorithm to see 512 samples for CIFAR 10 to estimate gradients, before pruning.
We treat this as a one-shot pruning method.
Other methods: for L1 and random pruning, we use native PyTorch implementations.
E. S UMMARY OF E XPERIMENTAL R ESULTS AND M ETHODOLOGY
In this section, we provide tables summarizing our empirical results.
We apply I TER TVSP RUNE to achieve accuracy drops of at most 1.5%, 5%, and 10%, on a variety of architectures and datasets.
We then check the accuracy observed when we prune the same networks, on the same datasets, with baseline pruning algorithms.
We detail this list below.
All experiments do not make use of the training set, except experiments where we apply CHIP Sui et al.
(2021) toward pruning the models.
[Figure]
ResNet18/CIFAR
VGG
CIFAR
Hyperparameters For all experiments, the value of η as used in Algorithm 2 of the main manuscript is chosen empirically, and varies for each experiment.
The value for δ is chosen to be 0.15.
The value of ν is chosen to be 0.
E. CIFAR10 R ESULTS
In this subsection, we detail our experiments on models trained using the CIFAR10 dataset.
E.2. VGG M ODELS
In this subsection, we compare VGG models trained on the CIFAR10 dataset.
Specifically, we compare the efficacy of I TER TVSP RUNE on VGG11, VGG16 and VGG19 models, with standard baselines.
In our experiment, we first use I TER TVSP RUNE mean to prune the models to a given accuracy threshold; that is, we solve equation P for different values of t .
Then, using the same sparsity patterns, we apply L 1 pruning, random pruning, and first-order gradient-based pruning respectively.
Note that here too, we do not fine tune the models after pruning.
We plot these results in Figure 6.
For clarity, we also present these results in Table 3.
Parametric sparsity We observe that using I TER TVSP RUNE dramatically outperforms several standard baselines.
Indeed, we are able to prune more than 60% of parameters in VGG19 trained on the CIFAR10 dataset with only a 2% drop in accuracy.
We also observe, from table 3, that on
VGG16, I TER TVSP RUNE also outperforms contemporary baselines such as CHIPSui et al.
[Table]
Layerwise Structured Sparsity In Section 7.3, we report the parametric sparsity - that is, the number of parameters pruned from the model using our proposed algorithms.
In this section, we consider the layerwise structured sparsity achieved using the I TER TVSP RUNE algorithm.
In our experiments, we observe that irrespective of accuracy threshold, most of the pruned filters lie in the final layers of the VGG models, and almost no filters are pruned in the initial layers.
Key Takeaways
The key takeaway of this slate of experiments is that VGG models trained on
CIFAR10 can be sparsified extensively with I TER TVSP RUNE .
We see that our method consistently outperforms the closest baselines, often by double digit percentages (in terms of test accuracy) for the same sparsity.
E.2. R ES N ET E XPERIMENTS
In this section, we detail our experiments with ResNet models trained on the CIFAR10 dataset.
The models we consider are the ResNet18, -20, -50, and -56 models.
Our procedure mirrors that used in the previous section, with VGG nets: we use I TER TVSP RUNE to prune models with different accuracy thresholds, and then apply baseline methods with the same sparsity patterns obtained via
I TER TVSP RUNE .
We present the results of our experiments in Table 4.
[Figure]
Key takeaways
The key takeaway from this slate of experiments is that I TER TVSP RUNE is effective on residual networks as well.
However, it is not as effective on narrow resnet models trained on CIFAR10 as it is on VGG models trained on the same dataset.
We observe that
E. CIFAR100 R ESULTS
In this section, we detail our experiments on models trained on the CIFAR100 dataset.
The architectures considered were VGG11, VGG16, and VGG19.
The experimental setup was identical to the previously presented experiments with the same models trained on the CIFAR10 dataset.
We plot those results below in Figure 8, and provided a detailed snapshot in Table 5.
E. E FFECT OF SIZE OF VALIDATION SET
We conduct a brief experiment to illustrate the extent to which the size of the set of samples used to compute the class-conditional moments, and the MinTVS scores, affects the ranking in terms of MinTVS.
For a VGG16 model trained on the CIFAR10 dataset, we sample 1024, 768, andsamples, and compute the min α,β ∆ α,βl,j , where α and β are classes, l denotes the layer, and j denotes the filter, as defined in Section 5 of the main manuscript.
We present plots for three layers below.
We observe that even when we reduce the number of samples by a factor of 2, we see that the ordering of filters according to min α,β ∆ l,jα,β (and thus, MinTVS( l, j )) is retained even for comparatively smaller sample sizes. E. E XPERIMENTS WITH F INE T UNING
In this section, we detail our results on using I TER TVSP RUNE in the setting wherein we have access to the training set and loss function, thereby enabling us to fine-tune our models after pruning.
Our experiments involve fine-tuning models trained on CIFAR10 and Imagenet.
Broadly speaking, our experiments demonstrate two facts.
Models pruned by our algorithm are able to return to the accuracy of the original pretrained model,
[Table]
Fine-tuning our models yield models that are competitive in accuracy with those obtained using current, state-of-the-art methods such as Sui et al.
E.5. CIFAR10 E XPERIMENTS
In this section, we detail our experiments with fine tuning models with the CIFAR10 dataset.
Hyperparameter Selection We present our choice of hyperparameters in Table 6.
For fine-tuning, we utilize the entire training set.
However, for measuring test accuracy, we use the same partition of the training set as used in the experiments presented in Sections 7.3 and E.3.
We refer the reader tofor the precise split in the dataset.
Results of Pruning with Fine Tuning We now present the results of fine-tuning models trained on the CIFAR10 dataset.
We chose to fine-tune VGG16 models pruned using I TER TVSP RUNE and
CHIP(Sui et al., 2021), and a ResNet18 model pruned with I TER TVSP RUNE .
We detail our results below in [Figure] [Figure] [Figure]
E.5. I MAGENET E XPERIMENTS
In this section, we detail our experiments with fine tuning models with the CIFAR10 dataset.
Hyperparameter Selection We present our choice of hyperparameters in Table 8.
Dataset Selection For fine-tuning, we utilize the entire training set, and for measuring test accuracy, we use the same partition of the training set as described in Table 2.
Results of Pruning with Fine
Tuning We now present the results of fine-tuning models trained on the Imagenet dataset. We chose to fine-tune VGG16 models pruned using I TER TVSP RUNE and CHIP(Sui et al., 2021), and a ResNet18 model pruned with I TER TVSP RUNE . We detail our results below in Table 9.
E. V ALIDATING USING MinTVS FOR P RUNING
In this section, we conduct experiments validating the us MinTVS scores, as well as the key hypothesis of the paper.
We show that for models trained on the CIFAR10 dataset, pruning filters
[Table]
E.6. E XPERIMENTAL S ETUP
We consider VGG16 and VGG19 models trained on the CIFAR10 dataset.
We consider models with
LDIFF scores that are greater than 0.4; that is, LDIFF ( η l , l ) > 0 .
4 , where η l = 0 .
75MinTVS( l ) is 0 .
75 times the mean MinTVS score for the layer.
The partition of the validation set for accuracy measurement and score computation is given in 2.
We do so since the variation in MinTVS scores for layers with low LDIFF scores is minimal; thus, there is little difference between the largest scores and the smallest.
[Table]
E.
[Table]
E XPERIMENTAL R ESULTS
We now present our experimental results.
We observe that pruning discriminative filters in general yields mode dramatic drops in accuracy when compared to pruning non-discriminative filters
[Table]
though the effect is slightly less for deeper layers (those closer to the output).
This observation supports the hypothesis that, given layers with a mix of discriminative and non-discriminative filters, we can safely prune the non-discriminative filters
[Table]
but not the former.
We present these results in Figure
[Table]
[Table]
[Figure]