Model Preserving Compression for Neural Networks
Abstract
After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands.
When compressing, it is desirable to preserve the original model’s per-example decisions (e.g., to go beyond topaccuracy or preserve robustness), maintain the network’s structure, automatically determine per-layer compression levels, and eliminate the need for ﬁne tuning.
No existing compression methods simultaneously satisfy these criteria—we introduce a principled approach that does by leveraging interpolative decompositions.
Our approach simultaneously selects and eliminates channels (analogously, neurons), then constructs an interpolation matrix that propagates a correction into the next layer, preserving the network’s structure.
Consequently, our method achieves good performance even without ﬁne tuning and admits theoretical analysis.
Our theoretical generalization bound for a one layer network lends itself naturally to a heuristic that allows our method to automatically choose per-layer sizes for deep networks.
We demonstrate the efﬁcacy of our approach with strong empirical performance on a variety of tasks, models, and datasets—from simple one-hiddenlayer networks to deep networks on ImageNet.
Introduction
There has recently been a signiﬁcant theoretical and empirical effort to understand the necessity of over-parameterization for deep learning models and to develop algorithmic techniques that can effectively compress (or prune) them while retaining performance.
This work is largely driven by results that show over-parameterization may aid in training via stochastic algorithms but that the solutions found are often highly redundant [21, 22, 42, 54] and therefore can be compressed.
Typically the focus has been on reducing parameter counts or FLOPs while maintaining accuracy—a task that often necessitates signiﬁcant ﬁne tuning of compressed models.
However, we argue that further criteria are warranted to enable the adoption of compression methods amongst practitioners: (1) preserving the model’s per-example decisions ensures the preservation of more ﬁne-grain properties, e.g. fairness [4, 14, 17, 24, 61] or adversarial robustness [9, 10, 16, 18, 68]; (2) compressing networks while retaining their computational structure ensures compatibility with the rest of the machine learning pipeline; and (3) automatically determining per-layer sizing and minimizing the need for ﬁne tuning alleviates key challenges when using compression methods in practice.
We develop a novel compression method that satisﬁes these three criteria.
In particular, it uses unlabeled data to remove redundant neurons/channels while simultaneously updating the remaining network parameters to explicitly correct for the removed neurons/channels.
The core technical development in our pruning method is the novel use of an interpolative decomposition (see Section 3) to approximate the post-activation output of layers.
The interpolative decomposition (ID) explicitly represents the pruned neurons/channels as a linear combination of those kept—allowing us to preserve the model with little to no ﬁne tuning.
Moreover the speciﬁc structure of the ID allows us to sub-select channels backward through the activation function, and propagate information about the pruned channels forward, fusing it into the adjacent layer and preserving the structure of the network (albeit with smaller layer sizes).
Collectively, these properties allow our ID based compression approach to produce a structurally similar compressed network that well approximates (to any desired accuracy) the original network without any additional training—a feature we show theoretically for simple fully connected networks in Section 4.
We then extend the results to different layer types and more complex architectures in Section 5, and show how our theoretical results allow us to automatically determine per-layer sizes for a compressed model based on the trade-off between FLOPs and the estimated error induced by pruning.
We ensure that our method truly compresses a network and preserves the decision boundaries, rather than effectively re-training a smaller network through extensive ﬁne tuning through a new metric we develop in Section 6.
Consider pruning a pretrained VGG16 model on CIFAR-10 with 93 .
6% accuracy to 50% of its FLOPs.
A variety of pruning methods achieve a reasonable accuracy of 93.2-93.6%, including structured magnitude pruning, random channel pruning, and our ID-based method.
However, methods which rely extensively on ﬁne-tuning (magnitude and random pruning) do not preserve the decision boundaries of the original network.
The predictions of these pruned models agree with the original model only ( 93 . 2% , 93 .
2% respectively) of the time on the test set.
This level of agreement is barely better than a completely independently trained 2 (fullsize) VGG model ( 93 . 0% ).
In contrast, our method produces a compressed model whose predictions are highly correlated with the original model, agreeing 97 .
3% of the time.
To complement our algorithmic developments and theoretical contributions, in Section 7 we demonstrate the efﬁcacy of our method on Atom3D [72], CIFAR-10 [43], and ImageNet [19].
The experiments are built to highlight how our ID-based compression method satisﬁes the desired criteria while competing methods do not.
To accomplish this, we ﬁrst introduce a metric of model similarity based on prediction correlation—this metric allows us to systematically determine how well compressed models are preserving the original model.
To highlight the practical efﬁcacy of our method (particularly its ability to automatically chose the size of the compressed network) we show our method successfully extends to Atom3D [72], simulating a real-world scenario with little prior compression work and baseline models with a different architecture (based on 3D convolutions).
More generally, on all datasets we show our method has superior accuracy before ﬁne tuning and is competitive with other state-of-the-art pruning methods after additional training is performed all while producing compressed models which better correlate with the original.
Moreover, because our method only reduces layer sizes but otherwise preserves network structure, it can be easily composed with other compression methods.
This allows us to substantially compress models for CIFAR-10 and ImageNet without the use of any global ﬁne tuning.
In summary, the key contributions of this paper 3 are:
A model preserving compression method We introduce an ID-based algorithm for compressing neural networks that preserves the model’s decisions and maintains the network’s layer structure without the need for ﬁne tuning.
Theoretical guarantees We provide theoretical guarantees for the output of our pruned model.
Practical efﬁcacy
We demonstrate the efﬁcacy of our method across the ATOM3D, CIFARand ImageNet data sets and using a variety of models.
Our method requires no labeled data, automatically determines per-layer sizes, and often requires little to no ﬁne tuning.
Related Work
There are a number of different design choices to be made in the compression and pruning process.
Classical pruning involves eliminating either channels (analogously neurons) or individual weights, sometimes in a structured way.
Magnitude pruning on both weights and neurons is still considered an effective approach [6, 21, 23, 25, 53, 46].
When pruning, the method may incorporate a correction to future layers, though it often does not [31, 54].
Some methods that correct the network chose to do local ﬁne tuning [54, 32, 79, 66, 52], whereas others do not [48, 33].
Of particular note, matrix approximation methods [20, 40, 50, 65, 44] often satisfy the ﬁrst criteria we desire for compression methods, but typically not the second as they add additional layers.
These methods sometimes incorporate local ﬁne tuning after compression [40, 42, 50, 65, 77] and sometimes do not [20, 44].
In contrast, structured pruning methods [31, 33, 48, 52–54] can satisfy the second criteria we outlined, but typically not the ﬁrst as they do a poor job of preserving the model’s decisions and often require excessive amounts of ﬁne tuning.
Pruning with coresets [64] is the closest in spirit to our own work and provides a way to select a subset of neurons in the current layer that can approximate those in the next layer as well as new weight connections.
Of note, Mussay et al.
[64] provide a sample complexity result, and demonstrate their method on fully connected (but not convolution) layers.
The HRank method [51] is also close in spirit to our own, and works by selectively pruning channels that produce low-rank feature maps.
However, the method does not propagate updates into the next layer and instead relies on excessive amounts of ﬁne tuning (30 epochs for each layer pruned) to ﬁx the network’s accuracy.
Recently the literature has started to consider criteria beyond topline accuracy metrics, and Liebenwein et al.
[49] use measures of functional approximation to conclude that pruned networks well approximate the original models.
Marx et al.
[60] characterize when linear models can achieve similar accuracy but with competing predictions.
Interpolative decompositions
Our pruning strategy relies on a structured low-rank approximation known as an interpolative decomposition (ID).
Classically, the Singular Value Decomposition (SVD) (see, e.g., [26]) provides an optimal low-rank approximation.
However, because we consider matrices that include the nonlinear activation function the SVD cannot be directly used to either subselect neurons or generate new ones (since it is unclear how to propagate singular vectors “backwards” through the non-linearity).
In contrast, an ID constructs a structured low-rank approximation of a matrix A where the basis used for the approximation is constrained to be a subset of the columns of A .
For a matrix A 2 R n ⇥ m we let A J , I denote a sub-selection of the matrix
A using sets J ⇢ [ n ] to denote the selected rows and
I ⇢ [ m ] to denote the selected columns; : denotes a selection of all rows or columns.
Deﬁnition 3.1 (Interpolative Decomposition) .
Let A 2 R n ⇥ m and ✏  0 .
An ✏ -accurate interpolative decomposition A ⇡ A : , I T is a subset of columns of A, denoted with the index subset I ⇢ [ m ] , and an associated interpolation matrix T such that
[Equation]
Remarks 3.2 .
When computing an ID, we would like to ﬁnd the smallest possible k ⌘ |I| such that the accuracy requirement is satisﬁed.
Moreover, we would like T to have entries of reasonable magnitude and approximation error not much larger than the best possible for a given k (i.e., k A  A : , I T [-∥ 
-]k 2 =
t  k +1 ( A ) for some small t  1 ).
While necessarily sub-optimal, the advantage is that we explicitly use a subset of the columns of A to build the approximation.
IDs are well studied [13, 59], widely used in the domain of rank-structured matrices [35, 36, 34, 58, 57, 62], and are closely related to CUR decompositions [56, 74] and subset selection problems [7, 15, 73].
While these decompositions always exist, ﬁnding them optimally is a difﬁcult task and in this work we appeal to what are known as (strong) rank-revealing QR factorizations [8, 11, 12, 28, 37].
Deﬁnition 3.3 (Rank-revealing QR factorization) .
Let A 2 R n ⇥ m , ` = min( n, m ) , and take any k  ` .
A rank-revealing QR factorization of A computes a permutation matrix ⇧ 2 R m ⇥ m , an uppertrapezoidal matrix R 2 R ` ⇥ m (i.e. R i,j =
if i > j ), and a matrix Q 2 R n ⇥ ` with orthonormal columns (i.e., Q > Q = I ) such that A ⇧ = QR and Q and R satisfy certain properties.
Splitting ⇧ , Q , and R into ⇧ 1 2 R m ⇥ k , ⇧ 2 2 R m ⇥ ( m  k ) , Q 1 2 R n ⇥ k , Q 2 2 R n ⇥ ( `  k ) , R 11 2 R k ⇥ k ,
R 12 2 R k ⇥ ( m  k ) , and R 22 2 R ( `  k ) ⇥ ( m  k ) we can write
[Equation]
Remarks 3.4 .
What makes (1) a rank-revealing QR factorization is that the permutation ⇧ is computed to ensure that R 11 is as well-conditioned as possible and R 22 is as small as possible.
While more formal statements of these conditions exist, we omit them here as they do not factor into our work.
Critically, any rank-revealing QR factorization yields a natural rank- k approximation of A with error
[Equation]
While ﬁnding the optimal rank-revealing QR factorization ( i.e., minimizing the error for a given k ) is closely related to a provably hard problem [15], we ﬁnd the original algorithm of Businger and Golub [8] works well in practice.
This routine is available in LAPACK [2, 67], can be easily incorporated into existing code, and has computational complexity O ( nmk ) when run for k steps.
Computing interpolative decompositions Given a rank-revealing QR factorization, we can immediately construct an ID (a formal algorithmic statement is given in the appendix).
Let I ⇢ [ m ] be such that A : , I =
A ⇧ 1 and deﬁne the interpolation matrix
[Equation]
With the choice A : , I = Q 1 R 11 it follows that the error of the ID as deﬁned by I and T is k A  A : , I T [-∥ 
-]k 2 = k R 22 k 2 .
Picking k such that k R 22 k 2  ✏ k A k 2 yields the desired relative error.
Notably, since  ( A : , I ) =
 ( R 11 ) and T = ⇥
I k
R  1 11 R⇤ ⇧ > the desired criteria for an ID map back to those of a rank-revealing QR factorization—if R 11 is well conditioned then the basis we use for approximation is as well and entries of T are not too large.
If  max ( R 22 ) is not much larger than [-σ k +
-]
k +1 ( A ) we get near optimal approximation accuracy.
Accuracy of the matrix approximation A key feature of using a column-pivoted QR factorization to compute an ID is that it allows us to dynamically determine the approximation rank k as a function of ✏ .
This can be accomplished by monitoring k R 22 k 2 at each step of the column-pivoted QR algorithm.
However, repeatedly computing k R 22 k 2 is expensive and often unnecessary in practice.
When using the algorithm by Businger and Golub [8] the magnitude of the diagonal entries of R are non-increasing and it is common to use | r k +1 ,k +1 /r 1 , 1 | as a proxy for k R
k 2 / k
A k 2 .
While formal bounds indicate the approximation may be loose in the worst case, it is effective in practice (see appendix Figure 9) and once a candidate k has been identiﬁed k R 22 k 2 can be computed if desired to certify the result—if the accuracy is unacceptable k can be increased until it is.
In some settings it may be preferable to ﬁx k and simply accept whatever accuracy is achieved.
Pruning with interpolative decompositions
The core of our approach is a novel use of IDs to prune neural networks.
Here we illustrate the scheme for a single fully connected layer and we extend the scheme to more complex layers (e.g., convolution layers) and deeper networks in Section 5.
Consider a simple two layer (one hidden layer) fully connected neural network h FC : R d !
R c of width m deﬁned as
[Equation]
with hidden layer W 2 R d ⇥ m , output layer U 2 R m ⇥ c , and activation function g .
We omit bias terms: they may be readily incorporated by adding a row to W and suitably augmenting the data.
To prune the model we will use an unlabeled pruning data set { x i } ni =1 with x i 2 R d .
Let X 2 R d ⇥ n be the matrix such that X : ,i = x i .
Preserving the action of the two layer network to accuracy ✏ > 0 on the data with fewer neurons is synonymous with ﬁnding an ✏ accurate approximation h FC ( x ; W, U ) ⇡ h FC ( x ; c
W, b U ) where c W has fewer columns than W .
We can do this by computing an ID of the activation output of the ﬁrst layer.
Concretely, let Z 2 R m ⇥ n be the ﬁrst-layer output, i.e., Z = g ( W > X ) , and let Z > ⇡ ( Z > ) : , I T be a rank- k ID of Z > with |I| =
k and interpolation matrix T 2 R k ⇥ m that achieves accuracy ✏ as in
Deﬁnition 3.1 (note that this means k is a function of ✏ ).
Because the activation function g commutes with the sub-selection operator, if Z ⇡ T > Z I , : then
[Equation]
Multiplying both sides by U > now gives an approximation of the original network by a pruned one, h FC ( x
[Equation]
That is, the ID has pruned the network of width m into a dense sub-network of width k with [Equation] R k ⇥ c .
Importantly, the SVD of Z > cannot be used for this task since it is not clear how to map the dominant left singular vectors back through the activation function g to either a subset of the existing neurons or a small set of new neurons.
This makes use of the ID essential
and we provide additional intuition for this scheme in the appendix, speciﬁcally in Figure 7.
4.1 A generalization bound for the pruned network
A key feature of our ID based pruning method is that it can be used to dynamically select the width of the pruned network to maintain a desired accuracy when compared with the full model.
This allows us to provide generalization guarantees for the compressed network in terms of generalization of the full model and the accuracy of the ID.
We state the results for a single hidden fully connected layer with scalar output (i.e., c = 1 ) and squared loss.
They can be extended to more complex networks (at the expense of more complicated dependence on the accuracy each layer is pruned to), more general Lipschitz continuous loss functions, and vector valued output.
We defer all proofs to the supplementary material.
Assume ( x, y ) ⇠ D where x 2 R d , y 2 R , and the distribution D is supported on a compact domain ⌦ x ⇥ ⌦ y .
We let R
= E ( x,y ) ⇠ D ( k ( u > g ( W > x )  y ) k 2 ) denote the true risk of the trained full model and R p
= E ( x,y )
⇠ D ( k ( b u > g ( c
W > x )  y ) k 2 ) be the risk of the pruned model.
(Since c = 1 we let u 2 R m denote the last layer.)
We also deﬁne the empirical risk b R ID of approximating the full model with our pruned model as
[Equation]
where { x i } ni =1 are n i.i.d.
samples from D (note that we do not need labels for these samples).
Using this notation, Theorem 4.1 controls the generalization error of the pruned model.
Theorem 4.1 (Single hidden layer FC) .
Consider a model
h FC =
u > g ( W > x ) with m hidden neurons and a pruned model
[Equation]
) constructed using an ✏ accurate ID with n data points drawn i.i.d from D .
The risk of the pruned model R p on a data set ( x, y ) ⇠ D assuming D is compactly supported on ⌦ x ⇥ ⌦ is bounded by
[Equation]
where R ID is the risk associated with approximating the full model by a pruned one and with probability 1   satisﬁes
[Equation]
Here, M = sup x 2 ⌦ x k u k 22 k g
W T x ) k 22 and ⇣ is a universal constant that depends on g .
Theorem 4.1 is developed by considering the ID as a learning algorithm applied to the output of the full model using unlabeled pruning data.
This allows us to control the risk of the pruned model in terms of the risk of the original model and the additional risk introduced by the ID.
Importantly, here we can control the additional risk in terms of the empirical risk of the ID and an additive term that decays as additional pruning data is used.
Lemma 4.2 codiﬁes this decomposition.
Lemma 4.2.
Under the assumptions of Theorem 4.1, for any  2 (0 , 1) , R ID satisﬁes
[Equation]
with probability 1   , where M = sup x 2 ⌦ x k u k
[Equation]
k g [Equation] x ) k 2 and p = ⇣ dm log( dm ) for some universal constant ⇣ that depends only on the activation function.
Remarks 4.3 .
We believe that the second part of the bound in Lemma 4.2 is likely loose since it relies on a pseudo-dimension bound for fully connected neural networks.
However, when pruning with an
ID we only consider subsets of existent neurons and it is plausible that in this setting the upper bound for the pseudo-dimension could be improved.
Crucially, an immediate consequence of using an ID for pruning is that we can explicitly control
R ID in terms of the accuracy parameter.
This relation between the ID accuracy and empirical risk is given in Lemma 4.4 and is what allows us to express the risk of the pruned network in Theorem 4.1.
Lemma 4.4.
Following the notation of Theorem 4.1, an ID pruning to accuracy ✏ yields a compressed network that [-satisfies (cid:98) R ID ≤ ϵ 2 ∥ u ∥ 
-]satisﬁes
[Equation]
where X 2 R [Equation] ⇥ n is a matrix whose columns are the pruning data.
Convolutional and deep networks 5.1 Convolution layers
To prune convolution layers with the ID at the channel level we reshape the output tensor into a matrix where each column represents a single output channel.
After this transformation, the key idea is the same as in Section 4.
Consider a simple two layer convolution neural network deﬁned as h conv ( x ; W, U )
=
conv( U, g (conv( W, x )))) , with the convolution-layer operator conv , weight tensors W and U , and activation function g .
The kernel dimension, size, stride, padding, and dilation do not change the form of the ID at the output channel level.
Let Z = g (conv( W, X )) be the activation output of the ﬁrst layer with unlabeled pruning data X , and deﬁne reshape as the operator which reshapes a tensor into a matrix with the output channels as columns, i.e., reshape( Z ) 2 R n i ⇥ m c where m c is the number of channels and n i is the product of all other dimensions (e.g. in the case of a 2d convolution, n i would be width ⇥ height ⇥ number of examples).
We now compute 4 a rank- k ID reshape( Z ) ⇡ reshape( Z ) : , I T.
The activation function g and reshape operator both commute with the sub-selection operator, so reshape
[Equation]
where W I ,... denotes an indexing sub-selection of W along its output-channel dimension.
Next, we need to propagate this T into the next layer, which we can do with a “matrix multiply” by the next-layer’s weights along its input channel dimension: we call this operation matmul .
5 With this, a little algebraic manipulation of our approximate equality above gives
[Equation]
and so if we set b U = matmul( T, U ) and c W = W I ,... , we can preserve the action of the two layer network with fewer channels as h conv ( x ; W, U ) ⇡
h conv ( x ; [Equation] U ) .
This gives us a recipe for pruning convolution layers analogous to (2).
This recipe can be directly applied to a composition of a convolution layer followed by a pooling layer [27] (or any other layer that acts independently by channel) by treating the conv layer/ pooling layer pair as a single convolution layer with a “fancy” activation function g : we just run the ID on the output post-pooling, and use that to sub-select the convolution layer’s weights.
Flatten layers, for connecting to FC layers, are equally straightforward.
5.2 Deep networks
The ID pruning primitives for fully connected and convolution layers can now be composed together to prune deep networks.
Algorithm 1 speciﬁes how we chain together the fully connected and convolution primitives to prune feedforward networks, for simplicity we assume for the moment we know the desired layer sizes.
A multi-layer neural network is pruned from the beginning to the end, where the ID is used to approximate the outputs of the original network.
The ID pruning primitives
Input: Neural net h ( x ; W (1) , . .
, W ( L ) ) , layers to not prune S ⇢ [ L ] , pruning set X , pruning fraction ↵
Output: Pruned network h ( x ; c W (1) , . . .
, c
W ( L ) ) 1: T (0) I 2: for l 2 { 1 . . .
L } do 3:
Z h 1: l ( X ; W (1) , . . .
, W ( l ) ) {layer l activations} 4: if layer l is a FC layer then 5: ( I , T ( l ) )
ID( Z T ; ↵ )
if l / 2 S else (: , I ) 6: c
W ( l )
T ( l  1) W
( l ) : , I {sub-select neurons, multiply T of prev layer’s ID} 7: else if layer l is a Conv layer (or Conv+Pool) then 8: ( I , T ( l ) ) ID(reshape( Z ); ↵ ) if l / 2 S else (: , I ) 9: c W ( l ) matmul( T ( l  1) , W ( l ) I ,... ) {select channels; multiply T} 10: else if layer l is a Flatten layer then 11:
T ( l ) T ( l  1) ⌦ I {expand T to have the expected size} 12: end if 13: end for sub-select neurons (or channels) in the current layer and propagate the corrective interpolation matrix to the next layer.
There are many ways one could prune a multi-layer network with these ID pruning primitives; we selected the approach in Algorithm 1 through empirical observations (though we do not assert that it is optimal).
Note that as a pre-processing step before running Algorithm 1, batch normalization layers [41] should be absorbed into their preceding fully-connected or convolution layers, and dropout [70] layers should be removed.
Iterative Pruning While Algorithm 1 is illustrative, in practice we would often like to be able to either specify a desired accuracy or choose layer sizes optimally for a desired compression ratio.
Our approach allows us to accomplish this by iteratively selecting layers to compress.
We introduce a score function for layers that is the ratio of the estimated relative error | r k +1 ,k +1 /r 1 , 1 | introduced by compressing a layer to the number of ﬂops f l that would be cut if we pruned a layer l to size k .
We call this score s l ( k ) =
| r k +1 ,
k +1 /r 1 , 1 | /f l
and it is heuristic for the compressability of each layer—lower scores imply a layer is easier to compress.
However, different layers of the network are connected, and compressing a layer early in the network can effect how well later layers can be compressed.
Therefore, we prune the network iteratively, measuring the score for each layer at a given pruning percentage (or step size), choosing the layer with the lowest score, pruning it, and then re-calculating the scores for the later layers.
We repeat the process until the network reaches a desired compression, or until the network performance degrades unacceptably.
We refer to this method as
Iterative ID, and refer to cutting a constant fraction of all neurons in each layer as Constant Fraction
ID.
For full details see Appendix F and Algorithm 3.
Evaluating compression beyond accuracy
An important beneﬁt of our approach is that we are actually able to preserve the original model’s predictions better than other methods.
Traditional pruning methods typically do a poor job at preserving the original predictions, due to their heavy reliance on ﬁne tuning that effectively retrains the model.
Here we explain why we might care about preserving per-example predictions beyond top-line accuracy.
We argue that in many situations compression methods must well-approximate the original model, and that accuracy is a poor metric for this use case.
Consider a pretrained model M , a resulting “compressed” model M C , and an evaluation set ( X, Y ) .
Accuracy measures the similarity between the true labels Y and the predicted labels from the compressed model M C ( X ) .
This metric does not directly compare the original model M and compressed model M C .
As we have seen in Section 1, a compressed model can recover the accuracy of the original model, but still differ widely on the predictions.
Instead our model correlation measures the similarity between the original model’s predictions M ( X ) and the compressed model’s predictions M C
( X ) .
One can think of this metric as an “accuracy to the original predictions”, instead of an accuracy to the true labels.
We propose model correlation as the percent of test example predictions two models agree on—details of this metric are discussed in Appendix D. Model correlation is a general metric to measure the similarity between the learned functions of two models.
In other contexts people have explored the related concept of model ﬁdelity, where our correlation metric matches the top-1 agreement of Stanton et al.
Our claim is that by better preserving a model’s per-example decisions, we can better preserve special properties of the model.
In the following section we provide experimental evidence for this claim.
Models are now often trained to have properties that go beyond test accuracy—for example robustness to adversarial attacks, sub-class classiﬁcation accuracy, fairness, etc., and this measure of model correlation is likely to correlate with many of these criteria.
Note that we do not believe preserving per-example decisions “boosts” any of these properties, we are simply preserving properties of the baseline model.
Experiments
The Appendix provides a list of all methods we compare with (including references) and a lookup table to ﬁnd the relevant experiments for each method (i.e., which ﬁgures and tables they appear in).
More careful evaluation of pruning methods We propose a range of evaluations that goes beyond simple post-ﬁne tuning test accuracy on standard vision benchmarks.
First is our proposed model correlation metric, discussed in detail in Section 6.
While certain benchmark problems are well studied and the literature has characterized layer sizes for more efﬁcient networks, that cannot be relied upon for realistic problems practitioners want to solve.
Therefore, we demonstrate the usability of our method by applying it to a non-standard problem and automatically discover a signiﬁcantly more efﬁcient allocation of ﬂops per layer.
We also consider extensive pre-ﬁne tuning evaluations.
In many cases, we may produce results before ﬁne tuning that are sufﬁciently strong to remove the need for it.
Moreover, ﬁne tuning often washes out differences between different methods and reduces model correlation (particularly if the learning rate is too high for too many epochs).
We show that by preserving model correlation our method is able to preserve the sub-class accuracy of a class which has been removed from the pruning and ﬁne tuning set.
Lastly, we show how maintaining network structure allows us to compose our methods with other matrix-decomposition based techniques and provide post-ﬁne tuning results.
One hidden layer results can be found in the Appendix.
Broad applicability
We demonstrate the versatility of our method on atypical architectures and beyond vision tasks using the ATOM3D Ligand Binding Afﬁnity (LBA) [72] benchmark.
This simulates a more typical use case for pruning than benchmark vision models.
The LBA task predicts the binding strength between proteins and small molecules which is useful for drug discovery.
We train and prune the 3DConv network from [72], which uses 3-D convolutional layers, and achieves a RMSE of 1.42.
We use the same method to achieve a baseline RMSE of 1.43, and prune the network using our Iterative ID.
The baseline network uses 12.8G FLOPs per example.
For comparison
VGG-16 on Imagenet uses 30G FLOPs.
Figure 1 shows that Iterative ID is able to prune 95% of the total FLOPS without any signiﬁcant change in the accuracy of the network and does not require any ﬁne tuning.
Pre-ﬁne tuning results We analyze the performance of ID compression against other pruning methods using the CIFAR-10 [43] and Imagenet [19] data sets with VGG16 [69] and Mobilenet
V1 [38].
The ID uses a held-out pruning set of 1000 data points on CIFAR-10 and points on Imagenet.
Full hyper-parameter details can be found in the Appendix and code.
Figures 1 andillustrate results before ﬁne tuning for structured and unstructured methods.
We see that on CIFAR-  and ImageNet, Iterative ID matches or outperforms other methods which do not use any global ﬁne tuning.
We even dominate unstructured methods which are typically more parameter efﬁcient.
Many matrix decomposition based methods struggle to compress the depth-wise-separable convolution layers in the Mobilenet V1 network.
Moreover, Iterative ID is able to choose better per-layer sizes than the default VGG16 conﬁguration on CIFAR-10.
In the appendix, we show that the choice of how much to prune per iteration is not particularly sensitive, and demonstrate that pruning up to 35% of the FLOPs results in no degradation to the pre-ﬁne tuning accuracy.
Results for MobileNet V1 on
ImageNet can be found in the Appendix.
Model correlation as a proxy for preserving fairness We conduct an experiment to demonstrate the connection between model correlation and a simple fairness metric of per-class accuracy.
We begin with a VGG16 model M trained on all 10 classes of Cifar10.
It performs reasonably well on all 10 classes, so we consider it “fair”.
We then remove a class from the pruning set to simulate an under-represented class (but leave it in the train and test sets).
We then prune M using our ID-based method, creating a compressed model M id —we do not perform any ﬁne tuning.
Sub-class accuracy for each class can be seen in ﬁgure 14a as well as the correlation of M id with respect to M .
Despite not having access to an entire class of data, we can prune the model to 50% FLOPs while only losing a few percent accuracy on the missing class and retaining reasonably high model correlation.
As a comparison we prune the model M with magnitude pruning to create a compressed model M m .
The magnitude pruned model loses almost all accuracy on the missing class when we prune to 50%, and its model correlation is initially only 10%.
We ﬁne tune with the abbreviated data set in an attempt to recover accuracy (ﬁgure 15a shows the correlation and sub-class accuracies).
While M m recovers on the represented classes, it fails to recover any accuracy for the unrepresented class during ﬁne tuning.
Model architecture preservation and composition of methods
Because our method preserves network structure and only reduces layer sizes it can be easily combined with other methods, see
Figure 3.
We ﬁrst apply Iterative ID , pruning until the the model correlation begins to drop on the pruning set.
Then we apply a secondary compression method.
This composition is easy because we are simply replacing the original model with an ID-pruned model that has the same architecture.
The secondary algorithms we use insert extra layers with different computational structure into the network, so composing them with each other is non-trivial.
In each example we see that pruning ﬁrst with ID and then applying a secondary algorithm boosts the performance of the secondary algorithm.
Post-global-ﬁne tuning results In Tables 1, and
we demonstrate the effectiveness of our method for VGG-16 on CIFAR-10 and ImageNet after moderate amounts of global ﬁne tuning, both alone and when combined with other methods.
Hyperparameter details can be found in the Appendix.
Networks pruned with ID attain competitive accuracies while having a much higher prediction correlation with their original models.
This evidence suggests that we have truly compressed the network rather than retraining a new one.
Other methods often do not correlate much more than an independently trained model.
Surprisingly, we ﬁnd that in some settings combining ID with other matrix decomposition methods like LRank[40] can be competitive without any global ﬁne tuning (Tables 1, 2).
We also ﬁnd that ID pruning and then globally ﬁne-tuning a MobileNet V1 architecture results in an accuracy of 89.4% on Cifar-10 and a correlation of 92.6% with the original model after 60 epochs at a 42% FLOPs reduction, as opposed to a model trained from scratch which achieves an accuracy of 88.9% and a correlation of 88.6% after 120 epochs.
[Table]
In the 50% FLOPs region, ID produces the most correlated model, while ID+LRank produces a model with reasonable accuracy without any global ﬁne tuning.  ⌘ original accuracy - pruned accuracy.
[Table]
Limitations and future work
Our method comes with several limitations and possible extensions for future work.
While preserving model correlation suggests that we are likely to preserve sub-class loss, our theory does not currently extend to that regime and requires that the unlabeled pruning set be from the same distribution as the test set.
More broadly, we do not yet fully understand how trainable the models produced by ID are in different conditions and we cannot make claims about how compressable a given model will be.
In future work we will explore modifying training process to improve prunability, which is a common approach [39, 76, 52, 78].
We will also explore ways to reﬁne our iterative pruning approach to work with more complicated architectures.
Of particular note, our method typically exposes a sizable “free
FLOPs" regime and we explore how this can be leveraged more broadly.
Acknowledgments and Disclosure of Funding
This work was partially funded by the National Science Foundation under awards DMS-1830274,
DGE-1922551, DMS-2146079, and NSF CAREER Award 2046760.