Original: Data-Efﬁcient Structured Pruning via Submodular Optimization
Revised: Data-Efﬁcient Structured Pruning via Submodular Optimization

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Structured pruning is an effective approach for compressing large pre-trained neuralnetworks without signiﬁcantly affecting their performance, which involves removing redundant regular regions of weights.
Revised: Structured pruning is an effective approach for compressing large pre-trained neuralnetworks without signiﬁcantly affecting their performance, which involves removing redundant regular regions of weights.

Original: However,most current structured pruningmethods do not provide any performance guarantees, and often require ﬁne-tuning,which makes them inapplicable in the limited-data regime.
Revised: However, current structured pruningmethods do not provide any performance guarantees, and often require ﬁne-tuning,which makes them inapplicable in the limited-data regime.

Original: We propose a principleddata-efﬁcient structured pruning method based on submodular optimization.
Revised: We propose a principleddata-efﬁcient structured pruning method based on submodular optimization.

Original: Inparticular, for a given layer, we select neurons/channels to prune and correspondingnew weights for the next layer, that minimize the change in the next layer’s inputinduced by pruning.
Revised: Inparticular, for a given layer, we select neurons/channels to prune and correspondingnew weights for the next layer, that minimize the change in the next layer’s inputinduced by pruning.

Original: We show that this selection problem is a weakly submodularmaximization problem, thus it can be provably approximated using an efﬁcientgreedy algorithm.
Revised: We show that this selection problem is a weakly submodularmaximization problem, thus it can be provably approximated using an efﬁcientgreedy algorithm.

Original: Our method is one of the few in the literature that uses only alimited-number of training data and no labels.
Revised: Our method is one of the few in the literature that uses only alimited-number of training data and no labels.

Original: Our experimental results demonstratethat our method outperforms state-of-the-art methods in the limited-data regime.
Revised: Our experimental results demonstratethat our method outperforms state-of-the-art methods in the limited-data regime.

Original:  Introduction
Revised:  Introduction

Original: As modern neural networks (NN) grow increasingly large, with some models reaching billions ofparameters [McGufﬁe and Newhouse, 2020], they require an increasingly large amount of memory,power, hardware and inference time, which makes it necessary to compress them.
Revised: As modern neural networks (NN) grow increasingly large, with some models reaching billions ofparameters [McGufﬁe and Newhouse, 2020], they require an increasingly large amount of memory,power, hardware and inference time which makes it necessary to compress them.

Original: This is especiallyimportant for models deployed on resource-constrained devices like mobile phones and smart speakers,and for latency-critical applications such as self-driving cars.
Revised: This is especiallyimportant for models deployed on resource-constrained devices like mobile phones and smart speakers,and for latency-critical applications such as self-driving cars.

Original: Several approaches exist to compress NNs.
Revised: Several approaches exist to compress NNs.

Original: Some methods approximate model weights usingquantization and hashing [Gong et al., 2014, Courbariaux et al., 2015], or low-rank approximationand tensor factorization [Denil et al., 2013, Lebedev et al., 2015, Su et al., 2018].
Revised: Some methods approximate model weights usingquantization and hashing [Gong et al., 2014, Courbariaux et al., 2015] or low-rank approximationand tensor factorization [Denil et al., 2013, Lebedev et al., 2015, Su et al., 2018].

Original: In anotherclass of methods called knowledge distillation, a small network is trained to mimic a much largernetwork [Bucila et al., 2006, Hinton et al., 2015].
Revised: In anotherclass of methods called knowledge distillation, a small network is trained to mimic a much largernetwork [Bucila et al., 2006, Hinton et al., 2015].

Original: Other methods employ sparsity and group-sparsityregularisation during training, to induce sparse weights [Collins and Kohli, 2014, Voita et al., 2019].
Revised: Other methods employ sparsity and group-sparsityregularisation during training, to induce sparse weights [Collins and Kohli, 2014, Voita et al., 2019].

Original: In this work, we follow the network pruning approach, where the redundant units (weights, neuronsor ﬁlters/channels) of a pre-trained NN are removed; see [Kuzmin et al., 2019, Blalock et al., 2020,
Revised: In this work, we follow the network pruning approach, where the redundant units (weights, neuronsor ﬁlters/channels) of a pre-trained NN are removed; see [Kuzmin et al., 2019, Blalock et al., 2020,

Original: Hoeﬂer et al., 2021] for recent surveys.
Revised: Hoeﬂer et al., 2021] for recent surveys.

Original: We also focus on the limited-data regime, where only fewtraining data is available and data labels are unavailable.
Revised: We also focus on the limited-data regime, where only fewtraining data is available and data labels are unavailable.

Original: The advantage of pruning approaches isthat, unlike weights approximation-based methods, they preserve the network structure, allowingretraining after compression, and unlike training-based approaches, they do not require training fromscratch, which is costly and requires large training data.
Revised: The advantage of pruning approaches isthat, unlike weights approximation-based methods, they preserve the network structure, allowingretraining after compression, and unlike training-based approaches, they do not require training fromscratch, which is costly and requires large training data.

Original: It is also possible to combine differentcompression approaches to compound their beneﬁts, see e.g., [Kuzmin et al., 2019, Section 4.3.4].
Revised: It is also possible to combine differentcompression approaches to compound their beneﬁts, see e.g., [Kuzmin et al., 2019, Section 4.3.4].

Original: Pruning methods date back to at least the 1990s with the works of [LeCun et al., 1990, Hassibi et al.
Revised: Pruning methods date back to at least the 1990s with the works of [LeCun et al., 1990, Hassibi et al.

Original: Existing pruning methods fall into two main categories: unstructured pruning methods which prune individual weights leading to irregular sparsity patterns, and structured pruning methods whichprune regular regions of weights, such as neurons, channels, or attention heads.
Revised: Existing pruning methods fall into two main categories: unstructured pruning methods which prune individual weights leading to irregular sparsity patterns, and structured pruning methods whichprune regular regions of weights, such as neurons, channels, or attention heads.

Original: Structured pruningmethods are generally preferable as the resulting pruned models can work with off-the-shelf hardwareor kernels, as opposed to models pruned with unstructured pruning which require specialized ones.
Revised: Structured pruningmethods are generally preferable as the resulting pruned models can work with off-the-shelf hardwareor kernels, as opposed to models pruned with unstructured pruning which require specialized ones.

Original: The majority of existing structured pruning methods are heuristics that do not offer any theoreticalguarantees.
Revised: The majority of existing structured pruning methods are heuristics that do not offer any theoreticalguarantees.

Original: Moreover, most pruning methods are inapplicable in the limited-data regime, as they relyon ﬁne-tuning with large training data at least for a few epochs to recover some of the accuracy lostwith pruning.
Revised: Moreover, most pruning methods are inapplicable in the limited-data regime, as they relyon ﬁne-tuning with large training data at least for a few epochs to recover some of the accuracy lostwith pruning.

Original: Mariet and Sra [2015] proposed a “reweighting" procedure applicable to any pruningmethod, optimize the weights of the next layer to minimize the change in the input to the next layer
Revised: Mariet and Sra [2015] proposed a “reweighting" procedure applicable to any pruningmethod, optimize the weights of the next layer to minimize the change in the input to the next layer

Original: Their empirical results on pruning single linear layers suggest that reweighting can provide a similarboost to performance as ﬁne-tuning, without the need for data labels.
Revised: Their empirical results on pruning single linear layers suggest that reweighting can provide a similarboost to performance as ﬁne-tuning, without the need for data labels.

Original: Our contributions We propose a principled data-efﬁcient structured pruning method based onsubmodular optimization.
Revised: Our contributions We propose a principled data-efﬁcient structured pruning method based onsubmodular optimization.

Original: In each layer, our method simultaneously selects neurons to prune andnew weights for the next layer, that minimize the change in the next layer’s input induced by pruning.
Revised: In each layer, our method simultaneously selects neurons to prune andnew weights for the next layer, that minimize the change in the next layer’s input induced by pruning.

Original: The optimization with respect to the weights, for a ﬁxed selection of neurons, is the same one usedfor reweighting in [Mariet and Sra, 2015].
Revised: The optimization with respect to the weights, for a ﬁxed selection of neurons, is the same one usedfor reweighting in [Mariet and Sra, 2015].

Original: The resulting subset selection problem is intractable, butwe show that it can be formulated as a weakly submodular maximization problem (see deﬁnition in
Revised: The resulting subset selection problem is intractable, butwe show that it can be formulated as a weakly submodular maximization problem (see deﬁnition in

Original: Section 2).
Revised: Section 2).

Original: This formulation allows us to use the standard greedy algorithm to obtain (1  e   ) -approximation to the optimal solution, where  is non-zero if we use sufﬁcient training data.
Revised: This formulation allows us to use the standard greedy algorithm to obtain (1  e   ) -approximation to the optimal solution, where  is non-zero if we use sufﬁcient training data.

Original: Our method uses only limited training data and no labels.
Revised: Our method uses only limited training data and no labels.

Original: Similar to [Mariet and Sra, 2015], weobserve that reweighting provide a signiﬁcant boost in performance not only to our method, but alsoto other baselines we consider.
Revised: Similar to [Mariet and Sra, 2015], weobserve that reweighting provide a signiﬁcant boost in performance not only to our method, but alsoto other baselines we consider.

Original: However unlike [Mariet and Sra, 2015], we only use a small fractionof the training data, around ⇠ 1% in our experiments.
Revised: However unlike [Mariet and Sra, 2015], we only use a small fractionof the training data, around ⇠ 1% in our experiments.

Original: We further adapt our method to prune anyregular regions of weights; we focus in particular on pruning channels in convolution layers.
Revised: We further adapt our method to prune anyregular regions of weights; we focus in particular on pruning channels in convolution layers.

Original: To prunemultiple layers in the network, we apply our method to each layer independently or sequentially.
Revised: To prunemultiple layers in the network, we apply our method to each layer independently or sequentially.

Original: Ourexperimental results demonstrate that our method outperforms state-of-the-art pruning methods in thelimited-data regime, and it is among the best performing methods even in the standard setting.
Revised: Ourexperimental results demonstrate that our method outperforms state-of-the-art pruning methods in thelimited-data regime, and it is among the best performing methods even in the full-data setting.

Original: Related work A large variety of structured pruning approaches has been proposed in the literature,based on different selection schemes and algorithms to solve them.
Revised: Related work A large variety of structured pruning approaches has been proposed in the literature,based on different selection schemes and algorithms to solve them.

Original: Ye et al.
Revised: Ye et al.

Original: [2020a,b] propose toprune neurons/channels that minimize the loss of the pruned network. They use a layerwise Greedyselection method with Frank-Wolfe like updates to solve the ` 1 -relaxation of the selection problem.
Revised: [2020] propose to pruneneurons/channels that minimize the loss of the pruned network, using a layerwise Greedy forwardselection method, with Frank-Wolfe like updates.

Revised: In the special case of two layer NNs, the square lossof the resulting pruned network approximates the best loss that can be achieved with all neurons inthe hidden layer, under some regularity conditions.

Original: This method is expensive, even with the acceleration technique introduced in Ye et al. [2020b], as itrequires afull forward pass  at each iteration.
Revised: No guarantees are provided for general NNs. Thismethod is expensive as it requires a forward pass to evaluate every neuron/channel at each iteration.

Original: Some works prune neurons/channels individually based on some importance score [He et al., 2014, Liet al., 2017, Liebenwein et al., 2020, Mussay et al., 2020, 2021, Molchanov et al., 2017, Srinivas and
Revised: Some works prune neurons/channels individually based on some importance score [He et al., 2014, Liet al., 2017, Liebenwein et al., 2020, Mussay et al., 2020, 2021, Molchanov et al., 2017, Srinivas and

Original: Babu, 2015].
Revised: Babu, 2015].

Original: Such methods are efﬁcient and easy to implement, but they fail to capture higher-orderinteractions between the pruned parameters.
Revised: Such methods are efﬁcient and easy to implement, but they fail to capture higher-orderinteractions between the pruned parameters.

Original: Most do not provide any performance guarantee.
Revised: Most do not provide any performance guarantee.

Original: Oneexception are the sampling-based methods of [Liebenwein et al., 2020, Mussay et al., 2020, 2021],who provide high probability bounds on the difference between the output of the pruned and originalmodel, as a function of number of neurons/channels kept, under some assumptions on the modelactivations.
Revised: Oneexception are the sampling-based methods of [Liebenwein et al., 2020, Mussay et al., 2020, 2021],who provide high probability bounds on the difference between the output of the pruned and originalmodel, as a function of number of neurons/channels kept, under some assumptions on the modelactivations.

Original: These theoretical guarantees are not comparable to the ones we provide (see Section 3.1for details).
Revised: These theoretical guarantees are not comparable to the ones we provide (see Section 3.1for details).

Original: Empirically, we show that our method signiﬁcantly outperforms [Liebenwein et al., 2020].
Revised: Empirically, we show that our method signiﬁcantly outperforms [Liebenwein et al., 2020].

Original: Closer to our approach are methods that aim to prune neurons/channels that minimize the changeinduced by pruning in the output of the layer being pruned, or its input to the next layer
Revised: Closer to our approach are methods that aim to prune neurons/channels that minimize the changeinduced by pruning in the output of the layer being pruned, or its input to the next layer

Original: [Luo et al.
Revised: [Luo et al.

Original:  He et al., 2017, Zhuang et al., 2018, Ye et al., 2020b].
Revised: ,2017, He et al., 2017, Zhuang et al., 2018].

Original: Although these criteria are easier tooptimize than the loss or accuracy of the NN, they still yield an intractable combinatorial problem.
Revised: Although these criteria are easier to optimize than theloss or accuracy of the NN, they still yield an intractable combinatorial problem.

Original: Existing methods either use a heuristic greedy algorithm to solve it
Revised: Existing methodseither use a heuristic greedy algorithm to solve it

Original: [Luo et al., 2017, Zhuang et al. ,2018], or they solve instead its ` 1 -relaxation using alternating minimization
Revised: [Luo et al., 2017, Zhuang et al., 2018], or they solveinstead its ` 1 -relaxation using alternating minimization

Original: [He et al., 2017], or agreedy algorithm with Frank-Wolfe like updates [Ye et al., 2020b].
Revised: [He et al., 2017].

Original: Mariet and Sra [2015] depart from the usual strategy of pruning parameters whose removal inﬂuencesthe network the least.
Revised: Mariet and Sra [2015] depart from the usual strategy of pruning parameters whose removal inﬂuencesthe network the least.

Original: They instead select a subset of diverse neurons to keep in each layer by samplingfrom a Determinantal Point Process, then they apply their reweighting procedure.
Revised: They instead select a subset of diverse neurons to keep in each layer bysampling from a Determinantal Point Process, then they apply their reweighting procedure.

Original: Their experimentalresults show that the advantage of their method is mostly due to reweighting (see Figure 4 therein).
Revised: Though,their experimental results show that the advantage of their method is mostly due to reweighting.

Original:  Preliminaries
Revised:  Preliminaries

Original: We begin by introducing our notation and some relevant background from submodular optimization.
Revised: We begin by introducing our notation and some relevant background from submodular optimization.

Original: Notation: Given a ground set V = { 1 , 2 , · · · , d } and a set function F : 2 V !
Revised: Notation: Given a ground set V = { 1 , 2 , · · · , d } and a set function F : 2 V !

Original: R + , we denote 91 the marginal gain of adding a set I ✓ V to another set S ✓ V by F ( I | S )
Revised: R + , we denote 91 the marginal gain of adding a set I ✓ V to another set S ✓ V by F ( I | S )

Original: =
Revised: =

Original: F ( S [ I )  F ( S ) , 92 which quantiﬁes the change in value when adding I to S .
Revised: F ( S [ I )  F ( S ) , 92 which quantiﬁes the change in value when adding I to S .

Original: The cardinality of a set S is written as | S | .
Revised: The cardinality of a set S is written as | S | .

Original: 93 Given a vector x 2 R d , we denote its support set by supp( x ) =
Revised: 93 Given a vector x 2 R d , we denote its support set by supp( x ) =

Original: { i 2 V | x i 6 = 0 } , and its ` 2 -normby k x k 2 .
Revised: { i 2 V | x i 6 = 0 } , and its ` 2 -normby k x k 2 .

Original: Given a matrix X 2 R d 0 ⇥ d , we denote its i -th column by X i , and its Frobenius norm byk X k F .
Revised: Given a matrix X 2 R d 0 ⇥ d , we denote its i -th column by X i , and its Frobenius norm byk X k F .

Original: Given a set S ✓ V , X S is the matrix with columns
Revised: Given a set S ✓ V , X S is the matrix with columns

Original: X i for all i 2 S , and 0 otherwise, and 1 S 96 is the indicator vector of S , with [ 1 S ] i = 1 for all i 2 S , and 0 otherwise.
Revised: X i for all i 2 S , and 0 otherwise, and 1 S 96 is the indicator vector of S , with [ 1 S ] i = 1 for all i 2 S , and 0 otherwise.

Original: Weakly submodular maximization: A set function F is submodular if it has diminishing marginalgains: F ( i | S )
Revised: Weakly submodular maximization: A set function F is submodular if it has diminishing marginalgains: F ( i | S )

Original:  F ( i | T ) for all S ✓ T , i 2 V \ T .
Revised:  F ( i | T ) for all S ✓ T , i 2 V \ T .

Original: We say that F is normalized if F ( ; ) = 0 , and 99 non-decreasing if F ( S )  F ( T ) for all S ✓ T .
Revised: We say that F is normalized if F ( ; ) = 0 , and 99 non-decreasing if F ( S )  F ( T ) for all S ✓ T .

Original: Given a non-decreasing submodular function F , selecting a set S ✓ V with cardinality | S |  k 101 that maximize F ( S ) can be done efﬁciently using the G REEDY algorithm (Alg. 1).
Revised: Given a non-decreasing submodular function F , selecting a set S ✓ V with cardinality | S |  k 101 that maximize F ( S ) can be done efﬁciently using the G REEDY algorithm (Alg. 1).

Original: The returnedsolution is guaranteed to satisfy F ( ˆ S )  (1  1 /e ) max | S |  k F ( S )
Revised: The returnedsolution is guaranteed to satisfy F ( ˆ S )  (1  1 /e ) max | S |  k F ( S )

Original: [Nemhauser et al., 1978].
Revised: [Nemhauser et al., 1978].

Original: Ingeneral though maximizing a non-submodular function over a cardinality constraint is NP-Hard[Natarajan, 1995].
Revised: Ingeneral though maximizing a non-submodular function over a cardinality constraint is NP-Hard[Natarajan, 1995].

Original: However, Das and Kempe [2011] introduced a notion of weak submodularitywhich is sufﬁcient to obtain provable bounds of the G REEDY algorithm’s performance.
Revised: However, Das and Kempe [2011] introduced a notion of weak submodularitywhich is sufﬁcient to obtain provable bounds of the G REEDY algorithm’s performance.

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: Given a set function F : 2 V !
Revised: Given a set function F : 2 V !

Original: R , U ✓ V, k 2 N + , we say that F is  U,k -weakly submodular, with  U,k > 0 if
Revised: R , U ✓ V, k 2 N + , we say that F is  U,k -weakly submodular, with  U,k > 0 if

Original: [Equation]
Revised: [Equation]

Original: for every two disjoint sets L, S ✓ V , such that L ✓ U, | S |  k .
Revised: for every two disjoint sets L, S ✓ V , such that L ✓ U, | S |  k .

Original: The parameter  U,k is called the submodularity ratio of F .
Revised: The parameter  U,k is called the submodularity ratio of F .

Original: It characterizes how close a set functionis to being submodular.
Revised: It characterizes how close a set functionis to being submodular.

Original: If F is non-decreasing then  U,k 2 [0 , 1] , and F is submodular if and only if U,k = 1 for all U ✓ V, k 2 N + .
Revised: If F is non-decreasing then  U,k 2 [0 , 1] , and F is submodular if and only if U,k = 1 for all U ✓ V, k 2 N + .

Original: Given a non-decreasing  ˆ S,k -weakly submodular function F , the
Revised: Given a non-decreasing  ˆ S,k -weakly submodular function F , the

Original: Greedy algorithm is guaranteed to return a solution F ( ˆ S )  (1  e   ˆ S,k ) max | S |  k F ( S )
Revised: Greedy algorithm is guaranteed to return a solution F ( ˆ S )  (1  e   ˆ S,k ) max | S |  k F ( S )

Original: [Das and
Revised: [Das and

Original: Kempe, 2011].
Revised: Kempe, 2011].

Original: Hence, the closer F is to being submodular, the better is the approximation guarantee.
Revised: Hence, the closer F is to being submodular, the better is the approximation guarantee.

Original:  Reweighted input change pruning
Revised:  Reweighted input change pruning

Original: In this section, we introduce our approach for pruning neurons in a single layer.
Revised: In this section, we introduce our approach for pruning neurons in a single layer.

Original: Given a large pretrained NN, n training data samples, and a layer ` with n ` neurons, our goal is to select a small numberk out of the n ` neurons to keep, and prune the rest, in a way that inﬂuences the network the least.
Revised: Given a large pretrained NN, n training data samples, and a layer ` with n ` neurons, our goal is to select a small numberk out of the n ` neurons to keep, and prune the rest, in a way that inﬂuences the network the least.

Original: Oneway to achieve this is by minimizing the change in input to the next layer, ` + 1 , induced by pruning.
Revised: Oneway to achieve this is by minimizing the change in input to the next layer, ` + 1 , induced by pruning.

Original: However, simply throwing away the activations from the dropped neurons is wasteful.
Revised: However, simply throwing away the activations from the dropped neurons is wasteful.

Original: Instead, weoptimize the weights of the next layer to reconstruct the inputs from the remaining neurons.
Revised: Instead, weoptimize the weights of the next layer to reconstruct the inputs from the remaining neurons.

Original: Formally, let A ` 2 R n ⇥ n ` be the activation matrix of layer ` with columns a ` 1 , · · · , a ` n ` , wherea ` i 2 R n is the vector of activations of the i th neuron in layer ` for each training input, and let
Revised: Formally, let A ` 2 R n ⇥ n ` be the activation matrix of layer ` with columns a ` 1 , · · · , a ` n ` , wherea ` i 2 R n is the vector of activations of the i th neuron in layer ` for each training input, and let

Original: W ` +1 2 R n ` ⇥ n ` +1 be the weight matrix of layer ` + 1 with columns w ` + , · · · , w ` +1 n ` +1 , wherew ` +1 i 2 R n ` is the vector of weights connecting the i th neuron in layer `
Revised: W ` +1 2 R n ` ⇥ n ` +1 be the weight matrix of layer ` + 1 with columns w ` + , · · · , w ` +1 n ` +1 , wherew ` +1 i 2 R n ` is the vector of weights connecting the i th neuron in layer `

Original: + 1 to the neurons in layer ` .
Revised: + 1 to the neurons in layer ` .

Original: When a neuron is pruned in layer ` , the corresponding column of weights in W ` and row in W ` +1are removed.
Revised: When a neuron is pruned in layer ` , the corresponding column of weights in W ` and row in W ` +1are removed.

Original: Pruning n `  k neurons in layer ` reduces the number of parameters and computationcost by ( n `  k ) /n
Revised: Pruning n `  k neurons in layer ` reduces the number of parameters and computationcost by ( n `  k ) /n

Original: ` for both layer ` and ` + 1 .
Revised: ` for both layer ` and ` + 1 .

Original: Let V ` = { 1 , · · · , n ` } .
Revised: Let V ` = { 1 , · · · , n ` } .

Original: Given a set S ✓ V ` , we denote by A ` S the matrix with columns a ` i for alli 2 S , and 0 otherwise.
Revised: Given a set S ✓ V ` , we denote by A ` S the matrix with columns a ` i for alli 2 S , and 0 otherwise.

Original: That is, A ` S is the activation matrix of layer ` after pruning.
Revised: That is, A ` S is the activation matrix of layer ` after pruning.

Original: We choose a setof neurons S ✓ V ` to keep and new weights ˜ W ` +1 2 R n ` ⇥ n `
Revised: We choose a setof neurons S ✓ V ` to keep and new weights ˜ W ` +1 2 R n ` ⇥ n `

Original: [Equation]
Revised: [Equation]

Original: 1 that minimize:
Revised: 1 that minimize:

Original: [Equation]
Revised: [Equation]

Original: Note that A ` W ` +1 are the original inputs of layer l + 1 , and A ` S ˜ W ` +1 are the inputs after pruningand reweighting, i.e., replacing the weights W ` +1 of layer ` + 1 with the new weights ˜ W ` +1 .
Revised: Note that A ` W ` +1 are the original inputs of layer l + 1 , and A ` S ˜ W ` +1 are the inputs after pruningand reweighting, i.e., replacing the weights W ` +1 of layer ` + 1 with the new weights ˜ W ` +1 .

Original: 3.1 Greedy selection
Revised: 3.1 Greedy selection

Original: Solving Problem (1) exactly is NP-Hard [Natarajan, 1995].
Revised: Solving Problem (1) exactly is NP-Hard [Natarajan, 1995].

Original: However, we show below that it can beformulated as a weakly submodular maximization problem, hence it can be efﬁciently approximated.
Revised: However, we show below that it can beformulated as a weakly submodular maximization problem, hence it can be efﬁciently approximated.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: then Problem (1) is equivalent to max | S |  k F ( S ) .
Revised: then Problem (1) is equivalent to max | S |  k F ( S ) .

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Given U ✓ V, k 2 N + , F is a normalized non-decreasing  U,k -weakly submodular function, with
Revised: Given U ✓ V, k 2 N + , F is a normalized non-decreasing  U,k -weakly submodular function, with

Original: [Equation]
Revised: [Equation]

Original: The proof of Proposition 3.1 follows by writing F as the sum of n ` +1 sparse linear regressionproblems F ( S ) =
Revised: The proof of Proposition 3.1 follows by writing F as the sum of n ` +1 sparse linear regressionproblems F ( S ) =

Original: [Equation]
Revised: [Equation]

Original: k 22 , and from the relationestablished in [Elenberg et al., 2016, Das and Kempe, 2011] between weak submodularity and sparseeigenvalues of the covariance matrix (see Appendix B.1).
Revised: k 22 , and from the relationestablished in [Elenberg et al., 2016, Das and Kempe, 2011] between weak submodularity and sparseeigenvalues of the covariance matrix (see Appendix B.1).

Original: We use the G REEDY algorithm to select a set ˆ S ✓ V ` of k neurons to keep in layer ` .
Revised: We use the G REEDY algorithm to select a set ˆ S ✓ V ` of k neurons to keep in layer ` .

Original: As discussed in
Revised: As discussed in

Original: Section 2, the returned solution is guaranteed to satisfy
Revised: Section 2, the returned solution is guaranteed to satisfy

Original: [Equation]
Revised: [Equation]

Original: Computing the lower bound on the submodularity ratio  ˆ S,kin Proposition 3.1 is NP-Hard [Das and Kempe, 2011]. It is non-zero if any 2 k columns of A ` are linearly independent.
Revised: The submodularity ratio  ˆ S,k is non-zero if any 2 k columns of A ` are linearly independent.

Original: If the number oftraining data is larger than the number of neurons, i.e., n > n ` , this is likely to be satisﬁed for anyk  n ` / 2 ; it is unlikely for the activations of a neuron to match exactly a linear combination of otherneurons.
Revised: If thenumber of training data is larger than the number of neurons, i.e., n > n ` , this is likely to be satisﬁedfor any k  n ` / 2 ; it is unlikely for the activations of a neuron to match exactly a linear combinationof other neurons.

Original: We verify that this is indeed the case in our experiments in Appendix E.
Revised: We verify that this is indeed the case in our experiments.

Original: We show in Appendix C that F satisﬁes an even stronger notion of approximate submodularity thanweak submodularity, which implies a better approximation guarantee for G REEDY than the oneprovided in Eq.
Revised: We show in Appendix C that F satisﬁes an even stronger notion of approximate submodularity thanweak submodularity, which implies a better approximation guarantee for G REEDY than the oneprovided in Eq.

Original: Though, this requires a somewhat stronger assumption: any k + 1 columns of A `should be linearly independent and all rows of W ` +1 should be linearly independent.
Revised: Though, this requires a somewhat stronger assumption: any k + 1 columns of A `should be linearly independent and all rows of W ` +1 should be linearly independent.

Original: In particular,we would need that n `  n ` +1 , which is not always satisﬁed.
Revised: In particular,we would need that n `  n ` +1 , which is not always satisﬁed.

Original: Note that our performance guarantees are of a different kind than the ones provided in [Liebenweinet al., 2020, Mussay et al., 2020, 2021]; they provide high probability bounds of the form A ` S ˜ W ` +1 2(1 ± ✏ )
Revised: Note that our performance guarantees are of a different kind than the ones provided in [Liebenweinet al., 2020, Mussay et al., 2020, 2021]; they provide high probability bounds of the form A ` S ˜ W ` +1 2(1 ± ✏ )

Original: A ` W ` +1 (entry-wise), as a function of the number of neurons kept k , which they propagate toobtain bounds on the difference between the output of the pruned and original model.
Revised: A ` W ` +1 (entry-wise), as a function of the number of neurons kept k , which they propagate toobtain bounds on the difference between the output of the pruned and original model.

Original: 3.2 Reweighting
Revised: 3.2 Reweighting

Original: For a ﬁxed S ✓ V ` , the reweighted input change k A ` W ` +1  A ` S ˜ W ` +1 k 2 F is minimized by setting
Revised: For a ﬁxed S ✓ V ` , the reweighted input change k A [Equation] F is minimized by setting

Original: [Equation]
Revised: [Equation]

Original:  where x S ( A ` ) 2 R n ` ⇥ n ` is the matrix with columns x S ( a ` j ) such that
Revised: 4) where x S ( A ` ) 2 R n ` ⇥ n ` is the matrix with columns x S ( a ` j ) such that

Original: [Equation]
Revised: [Equation]

Original: Note that x S ( a j ) = 1 j for all j 2 S , and that x S ( A ` ) has zero rows for all i 62 S .
Revised: Note that x S ( a j ) = 1 j for all j 2 S , and that x S ( A ` ) has zero rows for all i 62 S .

Original: Hence, the newweights are given by ˜ w ` +1 im = w ` +

Revised: Hence, the newweights are given by ˜ w ` +1 im = w ` +

Original: im +
Revised: im +

Original: P j 62 S [ x S ( A ` )] ij w ` +1 jm for all i 2 S , and ˜ w ` +1 im = 0 for alli 62 S, m 2 V ` +1 .
Revised: P j 62 S [ x S ( A ` )] ij w ` +1 jm for all i 2 S , and ˜ w ` +1 im = 0 for alli 62 S, m 2 V ` +1 .

Original: Namely, the new weights merge the weights from the dropped neurons into theselected ones.
Revised: Namely, the new weights merge the weights from the dropped neurons into theselected ones.

Original: This is the same reweighting procedure introduced in [Mariet and Sra, 2015].
Revised: This is the same reweighting procedure introduced in [Mariet and Sra, 2015].

Original: Butinstead of applying it only at the end to the selected neurons ˆ S , it is implicitly done at each iteration ofour pruning method, as it is required to evaluate F .
Revised: Butinstead of applying it only at the end to the selected neurons ˆ S , it is implicitly done at each iteration ofour pruning method, as it is required to evaluate F .

Original: We discuss next how this can be done efﬁciently.
Revised: We discuss next how this can be done efﬁciently.

Original: 3.3 Cost
Revised: 3.3 Cost

Original: Each iteration of G REEDY requires O ( n ` ) function evaluations of F .
Revised: Each iteration of G REEDY requires O ( n ` ) function evaluations of F .

Original: Computing F ( S ) from scratchneeds O ( k · ( n ` · n ` +1 + n · ( n ` + n ` +1 )) time, so a naive implementation of G REEDY is too expensive.
Revised: Computing F ( S ) from scratchneeds O ( k · ( n ` · n ` +1 + n · ( n ` + n ` +1 )) time, so a naive implementation of G REEDY is too expensive.

Original: The following Proposition outlines how we can efﬁciently evaluate F ( S + i ) given that F ( S ) wascomputed in the previous iteration.
Revised: The following Proposition outlines how we can efﬁciently evaluate F ( S + i ) given that F ( S ) wascomputed in the previous iteration.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Given S ✓ V ` such that | S |  k , i 62 S , let proj S ( a ` j ) =
Revised: Given S ✓ V ` such that | S |  k , i 62 S , let proj S ( a ` j ) =

Original: A ` S x S ( a ` j ) be the projection of a ` j onto the column space of A ` S , R S ( a ` i ) =
Revised: A ` S x S ( a ` j ) be the projection of a ` j onto the column space of A ` S , R S ( a ` i ) =

Original: a ` i  proj S ( a ` i ) and proj R S ( a ` i ) ( a ` j )arg min z
Revised: a ` i  proj S ( a ` i ) and proj R S ( a ` i ) ( a ` j )arg min z

Original: = R S ( a ` i )  ,  2 R k
Revised: = R S ( a ` i )  ,  2 R k

Original: a ` j  z k 22 the corresponding residual and the projection of a ` j onto it.
Revised: a ` j  z k 22 the corresponding residual and the projection of a ` j onto it.

Original: We can write
Revised: We can write

Original: [Equation]
Revised: [Equation]

Original: where proj R S ( a i ) ( A ` V \ S ) is the matrix with columns proj S ( a ` j ) for all j 62 S , 0 otherwise.
Revised: where proj R S ( a i ) ( A ` V \ S ) is the matrix with columns proj S ( a ` j ) for all j 62 S , 0 otherwise.

Original: Assuming
Revised: Assuming

Original: F ( S ) , proj S ( a ` j ) and x S ( a ` j ) for all j 62 S were computed in the previous iteration, we can compute
Revised: F ( S ) , proj S ( a ` j ) and x S ( a ` j ) for all j 62 S were computed in the previous iteration, we can compute

Original: F ( S + i ) , proj S
Revised: F ( S + i ) , proj S

Original: + i ( a ` j ) and x S
Revised: + i ( a ` j ) and x S

Original: + i ( a ` j ) for all j 62 ( S + i ) inO ( n
Revised: + i ( a ` j ) for all j 62 ( S + i ) in

Original: [Equation] n [Equation] time .
Revised: O ( n ` · ( n ` +1 + n + k )) time .

Original: The optimal weights in Eq.
Revised: The optimal weights in Eq.

Original: (4) can then be computed in O ( k · n ` · n ` +1 ) time, at the end of G REEDY .
Revised: (4) can then be computed in O ( k · n ` · n ` +1 ) time, at the end of G REEDY .

Original: The proof is given in Appendix B.2, and relies on using optimality conditions to construct the leastsquares solution x S
Revised: The proof is given in Appendix B.2, and relies on using optimality conditions to construct the leastsquares solution x S

Original: + i ( a ` j ) from x S ( a ` j ) .
Revised: + i ( a ` j ) from x S ( a ` j ) .

Original: In total G REEDY ’s runtime is then O ( k · ( n ` ) 2 · ( n ` +1 + n + k )) .
Revised: In total G REEDY ’s runtime is then O ( k · ( n ` ) 2 · ( n ` +1 + n + k )) .

Original: In other words, our pruningmethod costs as much as O ( k ) forward passes in layer ` + 1 with a batch size of n (assumingn ` [-+
-] = O ( n ` ) ).
Revised: In other words, our pruning methodcosts as much as O ( k ) forward passes in layer ` + 1 with a batch size of n (assuming n ` [-+
-]+1 = O ( n ` ) ).

Original: Using a faster variant of G REEDY proposed in [Li et al., 2022] further reduces thecost to O (( n ` ) 2 · ( n ` +1 + n + k ) / ✏ ) while maintaining almost the same approximation guarantee(1  e   ˆ S,k  ✏ ) .
Revised: Using a faster variant of G REEDY proposed in [Li et al., 2022] further reduces the cost to O (( n ` ) 2 ·( n ` +1 + n + k ) / ✏ ) while maintaining almost the same approximation guarantee (1  e   ˆ S,k  ✏ ) .

Original: 1 Note also that computing the solutions for different budgets k 0  k can be doneat the cost of one by running G REEDY with budget k .
Revised: Note also that computing the solutions for different budgets k 0  k can be done at the cost of one byrunning G REEDY with budget k .

Original: Our method is more expensive than methodswhich prune neurons individually [He et al., 2014, Li et al., 2017, Liebenwein et al., 2020, Mussayet al., 2020, 2021, Molchanov et al., 2017, Srinivas and Babu, 2015], but much less expensive thana loss-based method like [Ye et al., 2020a,b], which requiresrespectively O ( kn ` ) and O ( k ) forwardpasses in the full network, for each layer.
Revised: Our method is more expensive than methods which prune neuronsindividually [He et al., 2014, Li et al., 2017, Liebenwein et al., 2020, Mussay et al., 2020, 2021, Molchanov et al., 2017, Srinivas and Babu, 2015], but much less expensive than a loss-based methodlike [Ye et al., 2020], which requires O ( kn ` ) forward passes in the full network, for each layer.

Original:  Pruning regular regions of neurons
Revised:  Pruning regular regions of neurons

Original: In this section, we discuss how to adapt our approach to pruning regular regions of neurons.
Revised: In this section, we discuss how to adapt our approach to pruning regular regions of neurons.

Original: Thisis easily achieved by mapping any set of regular regions to the corresponding set of neurons, thenapplying the same method in Section 3.
Revised: Thisis easily achieved by mapping any set of regular regions to the corresponding set of neurons, thenapplying the same method in Section 3.

Original: In particular, we focus on pruning channels in CNNs.
Revised: In particular, we focus on pruning channels in CNNs.

Original: Given a layer ` with n ` output channels, let X ` 2 R n · p ` ⇥ n ` ⇥ r h ⇥ r w be its activations for each outputchannel and training input, where p ` is number of patches obtained by applying a ﬁlter of size r h ⇥ r w , and let F ` +1 2 R n ` +1 ⇥ n ` ⇥ r h ⇥ r w be the weights of layer ` + 1 , corresponding to n ` ﬁlters of sizer h ⇥ r w for each of its output channels.
Revised: Given a layer ` with n ` output channels, let X ` 2 R n · p ` ⇥ n ` ⇥ r h ⇥ r w be its activations for each outputchannel and training input, where p ` is number of patches obtained by applying a ﬁlter of size r h ⇥ r w ,and let F ` +1 2 R n ` +1 ⇥ n ` ⇥ r h ⇥ r w be the weights of layer ` + 1 , corresponding to n ` ﬁlters of sizer h ⇥ r w for each of its output channels.

Original: When an output channel is pruned in layer ` , the corresponding weights in F ` and F ` +1 are removed.
Revised: When an output channel is pruned in layer ` , the corresponding weights in F ` and F ` +1 are removed.

Original: Pruning n `  k output channels in layer ` reduces the number of parameters and computation costby ( n `  k ) /n
Revised: Pruning n `  k output channels in layer ` reduces the number of parameters and computation costby ( n `  k ) /n

Original: ` for both layer ` and ` + 1 .
Revised: ` for both layer ` and ` + 1 .

Original: If layer ` is followed by a batch norm layer, the weightstherein corresponding to the pruned channels are also removed.
Revised: If layer ` is followed by a batch norm layer, the weightstherein corresponding to the pruned channels are also removed.

Original: We arrange the activations X ` c 2 R n · p ` ⇥ r h · r w of each channel c into r h r w columns of
Revised: We arrange the activations X ` c 2 R n · p ` ⇥ r h · r w of each channel c into r h r w columns of

Original: A ` 2 R n · p ` ⇥ n ` · r h · r w , i.e., A ` =
Revised: A ` 2 R n · p ` ⇥ n ` · r h · r w , i.e., A ` =

Original: [ X ` 1 , · · · , X ` n ` ] .
Revised: [ X ` 1 , · · · , X ` n ` ] .

Original: Similarly, we arrange the weights F ` +1 c 

Revised: Similarly, we arrange the weights F ` +1 c

Original: R n ` +1 ⇥ r
Revised: R n ` +1 ⇥ r

Original: h ⇥ r w of each channel c into r h · r w rows of W ` +1 2 R n ` · r h · r w ⇥ n ` +1 , i.e., ( W ` +1 )
Revised: h ⇥ r w of each channel c into r h · r w rows of W ` +1 2 R n ` · r h · r w ⇥ n ` +1 , i.e., ( W ` +1 )

Original: >
Revised: >

Original: =[( F ` 1 ) > , · · · , ( F ` n ` ) > ] .
Revised: =[( F ` 1 ) > , · · · , ( F ` n ` ) > ] .

Original: Recall that V ` = { 1 , · · · , n ` } , and let V 

Revised: Recall that V ` = { 1 , · · · , n ` } , and let V

Original: ` = { 1 , · · · , r h r w n ` } .
Revised: ` = { 1 , · · · , r h r w n ` } .

Original: We deﬁnea function M : 2 V ` !
Revised: We deﬁnea function M : 2 V ` !

Original: 2 V 0 ` which maps every channel c to its corresponding r h r w columns
Revised: 2 V 0 ` which maps every channel c to its corresponding r h r w columns

Original: A ` .
Revised: A ` .

Original: Let G ( S )
Revised: Let G ( S )

Original: =
Revised: =

Original: F ( M ( S )) , with F deﬁned in Eq.
Revised: F ( M ( S )) , with F deﬁned in Eq.

Original: (2), then minimizing the reweighted input changek A ` W ` +1  A ` M ( S )
Revised: (2), then minimizing the reweighted input changek A ` W ` +1  A ` M ( S )

Original: ˜ W ` +1 k 2 F with a budget k is equivalent to max | S |  k G ( S ) .
Revised: ˜ W ` +1 k 2 F with a budget k is equivalent to max | S |  k G ( S ) .

Original: The following proposition shows that this remains a weakly submodular maximization problem.
Revised: The following proposition shows that this remains a weakly submodular maximization problem.

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Given U ✓ V ` , k 2 N + , G is a normalized non-decreasing  U,k -weakly submodular function, with[Equation]
Revised: Given U ✓ V ` , k 2 N + , G is a normalized non-decreasing  U,k -weakly submodular function, with

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: G is  U,k -weakly submodular iff F satisﬁes
Revised: G is  U,k -weakly submodular iff F satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  for every two disjoint sets L, S ✓ V ` , such that L ✓ U, | S |  k .
Revised: ) , for every two disjoint sets L, S ✓ V ` , such that L ✓ U, | S |  k .

Original: We extend the relation established 204 in [Elenberg et al., 2016, Das and Kempe, 2011] between weak submodularity and sparse eigenvaluesof the covariance matrix to this case.
Revised: We extend the relation established 202 in [Elenberg et al., 2016, Das and Kempe, 2011] between weak submodularity and sparse eigenvaluesof the covariance matrix to this case.

Original: The proof then follows in a similar way to Proposition 3.1.
Revised: The proof then follows in a similar way to Proposition 3.1.

Original: As before, we use the G REEDY algorithm, with function G , to select a set ˆ S ✓ V ` of k channelsto keep in layer ` .
Revised: As before, we use the G REEDY algorithm, with function G , to select a set ˆ S ✓ V ` of k channels tokeep in layer ` .

Original: We get the same approximation guarantee G ( ˆ S )  (1  e   [Equation] S |  k G [Equation]
Revised: We get the same approximation guarantee G ( ˆ S )  (1  e   ˆ S,k ) max | S |  k G ( S ) .

Original: S ) . The submodularity ratio  ˆ S,k is non-zero if any 2 kr h r w columns of A ` are linearly independent.
Revised: Thesubmodularity ratio  ˆ S,k is non-zero if any 2 kr h r w columns of A ` are linearly independent.

Original: In our experiments, we observe thatin certain layers linear independence only holds for very small,e.g., k  0 . 01 n ` .
Revised: In our experiments, we observe that linear independence only holds for small k  0 . 2 n ` in certain layers.

Original: This is due to the correlation between patches which overlap.
Revised: Thisis due to the correlation between patches which overlap.

Original: To remedy that,we experimented with using only r h r w random patches from each image, instead of using all patches.
Revised: To remedy that, we experimented with usingonly r h r w random patches from each image, instead of using all patches.

Original: This indeed raises the rank of A ` , but certain layers have a very small feature map size so thateven the small number of random patches have signiﬁcant overlap, resulting in still a very smallrange where linear independence holds, e.g., k  0 . 08 n ` (see Appendix E for more details).
Revised: This indeed raises the rank of A ` , such that the range where linear independence is satisﬁed is around k  0 . 4 n ` .

Original: Theresults obtained with random patches were worst than the ones with all patches, we thus omit them.
Revised: However, the obtained results are very similar to the ones with all patches, we thus omit them.

Original: Note thatour lower bounds on  ˆ S,k are not necessarily tight (see Appendix F). Hence, having lineardependence does not necessarily imply that  ˆ S,k = 0 ; our method still performs well in these cases.
Revised: Note that having lineardependence does not necessarily imply that  ˆ S,k = 0 ; our method still performs well in these cases.

Original: For a ﬁxed S ✓ V ` , the optimal weights are again given by ˜ W ` +1 = x M ( S )
Revised: For a ﬁxed S ✓ V ` , the optimal weights are again given by ˜ W ` +1 = x M ( S )

Original: ( A ` ) W ` +1 .
Revised: ( A ` ) W ` +1 .

Original: The cost ofrunning G REEDY and reweighting is the same as before (see Appendix B.2).
Revised: The cost ofrunning G REEDY and reweighting is the same as before (see Appendix B.2).

Original:  Pruning multiple layers
Revised:  Pruning multiple layers

Original: In this section, we explain how to apply our pruning method to prune multiple layers of a NN.
Revised: In this section, we explain how to apply our pruning method to prune multiple layers of a NN.

Original: 5.1 Reweighted input change pruning variants
Revised: 5.1 Reweighted input change pruning variants

Original: We consider three variants of our method: L AYER I N C HANGE , S EQ I N C HANGE , and A SYM I NC HANGE .
Revised: We consider three variants of our method: L AYER I N C HANGE , S EQ I N C HANGE , and A SYM I NC HANGE .

Original: In L AYER I N C HANGE , we prune each layer independently, i.e., we apply exactly themethod in Section 3 or 4, according to the layer’s type.
Revised: In L AYER I N C HANGE , we prune each layer independently, i.e., we apply exactly themethod in Section 3 or 4, according to the layer’s type.

Original: This is the fastest variant; it has the same costas pruning a single layer, as each layer can be pruned in parallel, and it only requires one forwardpass to get the activations of all layers.
Revised: This is the fastest variant; it has the same costas pruning a single layer, as each layer can be pruned in parallel, and it only requires one forwardpass to get the activations of all layers.

Original: However, it does not take into account the effect of pruningone layer on subsequent layers.
Revised: However, it does not take into account the effect of pruningone layer on subsequent layers.

Original: In S EQ I N C HANGE , we prune each layer sequentially, starting from the earliest layer to latestone.
Revised: In S EQ I N C HANGE , we prune each layer sequentially, starting from the earliest layer to latestone.

Original: For each layer ` , we apply our method with A ` replaced by the updated activations B ` afterhaving pruned previous layers, i.e., we solve min | S |  k, ˜ W ` +1 2 R n ` ⇥ n ` +1 k B ` W ` +1  B ` S ˜ W ` +1 k 2 F .
Revised: For each layer ` , we apply our method with A ` replaced by the updated activations B ` afterhaving pruned previous layers, i.e., we solve min | S |  k, ˜ W ` +1 2 R n ` ⇥ n ` +1 k B ` W ` +1  B ` S ˜ W ` +1 k 2 F .

Original: In A SYM I N C HANGE , we also prune each layer sequentially, but to avoid the accumulation oferror, we use an asymmetric formulation of the reweighted input change, where instead of approximating the updated input B ` W ` +1 , we approximate the original input
Revised: In A SYM I N C HANGE , we also prune each layer sequentially, but to avoid the accumulation oferror, we use an asymmetric formulation of the reweighted input change, where instead of approximating the updated input B ` W ` +1 , we approximate the original input

Original: A ` W ` +1 , i.e., we solvemin | S |  k, ˜ W ` +1 2 R n ` ⇥ n ` +1 k A ` W ` +1  B ` S ˜ W ` +1 k 2 F .
Revised: A ` W ` +1 , i.e., we solvemin | S |  k, ˜ W ` +1 2 R n ` ⇥ n ` +1 k A ` W ` +1  B ` S ˜ W ` +1 k 2 F .

Original: This problem is still a weakly submodular maximization problem, with the same submodularity ratio given in Propositions 3.1 and 4.1, with A `replaced by B ` therein (see Appendix B.1).
Revised: This problem is still a weakly submodular maximization problem, with the same submodularity ratio given in Propositions 3.1 and 4.1, with A ` replaced by B ` therein (see Appendix B.1).

Original: Hence, the same approximation guarantee as in thesymmetric formulation holds here.
Revised: Hence, the same approximation guarantee as in thesymmetric formulation holds here.

Original: Moreover, a better approximation guarantee can again be obtainedwith a stronger notion of approximate submodularity, under stronger assumptions (see Appendix C).
Revised: Moreover, a better approximation guarantee can again be obtainedwith a stronger notion of approximate submodularity, under stronger assumptions (see Appendix C).

Original: The cost of running G REEDY with the asymmetric formulation and reweighting is also the same asbefore (see Appendix B.2).
Revised: The cost of running G REEDY with the asymmetric formulation and reweighting is also the same asbefore (see Appendix B.2).

Original: We evaluate all three variants in our experiments.
Revised: We evaluate all three variants in our experiments.

Original: As expected, A SYM I N C HANGE usually performsthe best, and L AYER I N C HANGE the worst.
Revised: As expected, A SYM I N C HANGE usually performsthe best, and L AYER I N C HANGE the worst.

Original: 5.2 Per-layer budget selection
Revised: 5.2 Per-layer budget selection

Original: Another important design choice is how much to prune in each layer, given a desired global compression ratio (see Appendix I for the effect of this choice on performance).
Revised: Another important design choice is how much to prune in each layer, given a desired global compression ratio (see Appendix F for the effect of this choice on performance).

Original: In our experiments, we usethe compression ratio selection method introduced in [Kuzmin et al., 2019, Section 3.4.1], which canbe applied to any layerwise pruning method, thus enabling us to have a fair comparison.
Revised: In our experiments, we usethe compression ratio selection method introduced in [Kuzmin et al., 2019, Section 3.4.1], which canbe applied to any layerwise pruning method, thus enabling us to have a fair comparison.

Original: Given a network with L layers to prune, let k = ↵
Revised: Given a network with L layers to prune, let k = ↵

Original: P
Revised: P

Original: L ` =1 n ` , for some ↵ 2 [0 , 1] , be the global numberof neurons/channels to keep.
Revised: L ` =1 n ` , for some ↵ 2 [0 , 1] , be the global numberof neurons/channels to keep.

Original: We want to select for each layer ` , the number of neurons/channelsk ` = ↵ ` n ` to keep, with ↵ ` chosen from a set of possible values, e.g., ↵ ` 2 { 0 . 05 , 0 . 1 , · · · , 1 } .
Revised: We want to select for each layer ` , the number of neurons/channelsk ` = ↵ ` n ` to keep, with ↵ ` chosen from a set of possible values, e.g., ↵ ` 2 { 0 . 05 , 0 . 1 , · · · , 1 } .

Original: We 252 deﬁne a layerwise accuracy metric P ` ( k ` ) as the accuracy obtained after pruning layer ` , with abudget k ` , while other layers are kept intact, evaluated on a veriﬁcation set.
Revised: We 247 deﬁne a layerwise accuracy metric P ` ( k ` ) as the accuracy obtained after pruning layer ` , with abudget k ` , while other layers are kept intact, evaluated on a veriﬁcation set.

Original: We set aside a subset ofthe training set to use as a veriﬁcation set.
Revised: We set aside a subset ofthe training set to use as a veriﬁcation set.

Original: Let P orig be the original model accuracy.
Revised: Let P orig be the original model accuracy.

Original: We select theper-layer budgets that minimize the per-layer accuracy drop:
Revised: We select theper-layer budgets that minimize the per-layer accuracy drop:

Original: [Equation]
Revised: [Equation]

Original: We can solve the selection problem (6) using binary search, if the layerwise accuracy P ` ( k ` ) is a nondecreasing function of k ` .
Revised: We can solve the selection problem (6) using binary search, if the layerwise accuracy P ` ( k ` ) is a nondecreasing function of k ` .

Original: Empirically, this is not always the case, the general trend is non-decreasing,but some ﬂuctuations occur.
Revised: Empirically, this is not always the case, the general trend is non-decreasing,but some ﬂuctuations occur.

Original: In such cases, we use interpolation to ensure monotonicity.
Revised: In such cases, we use interpolation to ensure monotonicity.

Original: Alternatively, another simple strategy is to prune each layer until the perlayer error (the reweightedinput change in our case) reaches some threshold ✏ , and vary ✏ to obtain the desired global compressionratio, as done in [Zhuang et al., 2018, Ye et al., 2020a].
Revised: Alternatively, another simple strategy is to prune each layer until the perlayer error (the reweightedinput change in our case) reaches some threshold ✏ , and vary ✏ to obtain the desired global compressionratio, as done in [Zhuang et al., 2018, Ye et al., 2020].

Original:  Empirical Evaluation
Revised:  Empirical Evaluation

Original: In this section, we examine the performance of our proposed pruning method in the limited-dataregime.
Revised: In this section, we examine the performance of our proposed pruning method in the limited-dataregime.

Original: To that end, we focus on one-shot pruning, in which a pre-trained model is compressedin a single step, without any ﬁne-tuning.
Revised: To that end, we focus on one-shot pruning, in which a pre-trained model is compressedin a single step, without any ﬁne-tuning.

Original: We study the effect of ﬁne-tuning withboth limited andsufﬁcient data in Appendix H. We compare the three variants of our method, L AYER I N C HANGE ,
Revised: We study the effect of ﬁne-tuning with limited and fulltraining data in Appendix E. We compare the three variants of our method, L AYER I N C HANGE ,

Original: S EQ I N C HANGE , and A SYM I N C HANGE , with the following baselines:
Revised: S EQ I N C HANGE , and A SYM I N C HANGE , with the following baselines:

Original: • L AYER G REEDY FS
Revised: • L AYER G REEDY FS

Original: [Ye et al., 2020a]: for each layer, ﬁrst removes all neurons/channels in thatlayer, then gradually adds back the neuron/channel that yields the largest decrease of the loss, evaluated on one batch of training data.
Revised: [Ye et al., 2020]: for each layer, ﬁrst removes all neurons/channels in thatlayer, then gradually adds back the neuron/channel that yields the largest decrease of the loss, evaluated on one batch of training data.

Original: Layers are pruned sequentially from the input to the output layer.• L AYER S
Revised: Layers are pruned sequentially from the input to the output layer.• L AYER S

Original: AMPLING
Revised: AMPLING

Original: [Liebenwein et al., 2020]: samples neurons/channels, in each layer, withprobabilities proportional to sensitivities based on (activations ⇥ weights), and prunes the rest.• A CT G RAD
Revised: [Liebenwein et al., 2020]: samples neurons/channels, in each layer, withprobabilities proportional to sensitivities based on (activations ⇥ weights), and prunes the rest.• A CT G RAD

Original: [Molchanov et al., 2017]: prunes neurons/channels with the lowest (activations ⇥gradients), averaged over the training data, with layerwise ` 2 -normalization.
Revised: [Molchanov et al., 2017]: prunes neurons/channels with the lowest (activations ⇥gradients), averaged over the training data, with layerwise ` 2 -normalization.

Original: This criteria is thebest performing criteria among the ones evaluated in [Molchanov et al., 2017].
Revised: This criteria is thebest performing criteria among the ones evaluated in [Molchanov et al., 2017].

Original: • L AYER
Revised: • L AYER

Original: A CT G RAD : prunes neurons/channels with the lowest (activations ⇥ gradients), averagedover the training data, in each layer.
Revised: A CT G RAD : prunes neurons/channels with the lowest (activations ⇥ gradients), averagedover the training data, in each layer.

Original: This is the layerwise variant of A CT G RAD .•
Revised: This is the layerwise variant of A CT G RAD .•

Original: W EIGHT N ORM [He et al., 2014]: prunes neurons/channels with the lowest output weights` 1 -norm, normalized by the number of neurons/channels in their layer.
Revised: W EIGHT N ORM [He et al., 2014]: prunes neurons/channels with the lowest output weights` 1 -norm, normalized by the number of neurons/channels in their layer.

Original: This is the best performingcriteria among the ones considered in [He et al., 2014].• L AYER W EIGHT N ORM [Li et al., 2017]: prunes neurons/channels with the lowest outputweights ` 1 -norm, in each layer.
Revised: This is the best performingcriteria among the ones considered in [He et al., 2014].• L AYER W EIGHT N ORM [Li et al., 2017]: prunes neurons/channels with the lowest outputweights ` 1 -norm, in each layer.

Original: This is the layerwise version of W EIGHT N ORM ,• R ANDOM : prunes randomly selected neurons/channels globally across layers in the network.• L AYER R ANDOM : prunes randomly selected neurons/channels in each layer.
Revised: This is the layerwise version of W EIGHT N ORM , • R ANDOM : prunes randomly selected neurons/channels globally across layers in the network.• L AYER R ANDOM : prunes randomly selected neurons/channels in each layer.

Original: We evaluate the performance of these methods on the LeNet model [LeCun et al., 1989] on the
Revised: We evaluate the performance of these methods on the LeNet model [LeCun et al., 1989] on the

Original: MNIST dataset [Lecun et al., 1998], and on the ResNet56 [He et al., 2016] and the VGG11 [Simonyanand Zisserman, 2015] models on the CIFAR-10 dataset [Krizhevsky et al., 2009].
Revised: MNIST dataset [Lecun et al., 1998], and on the ResNet56 [He et al., 2016] and the VGG11 [Simonyanand Zisserman, 2015] models on the CIFAR-10 dataset [Krizhevsky et al., 2009].

Original: To ensure a faircomparison, all experiments are based on our own implementation of all the compared methods.
Revised: To ensure a faircomparison, all experiments are based on our own implementation of all the compared methods.

Original: To compute the gradients and activations used for pruning in L AYER S AMPLING ,
Revised: To compute the gradients and activations used for pruning in L AYER S AMPLING ,

Original: A CT G RAD , L AYER
Revised: A CT G RAD , L AYER

Original: A CT G RAD , and our method’s variants, we use four batches of size 128 of training images, i.e.,n = 512 , which corresponds to ⇠ 1% of the training data in MNIST and CIFAR10.
Revised: A CT G RAD , and our method’s variants, we use four batches of size 128 of training images, i.e.,n = 512 , which corresponds to ⇠ 1% of the training data in MNIST and CIFAR10.

Original: We considertwo variants of the method proposed in [Ye et al., 2020a]: a limited-data variant L AYER G REEDY FSwhich only uses the same four batches of data used in our method, and a full-data variant L AYERG REEDY FS-fd with access to the full training data.
Revised: We considertwo variants of the method proposed in [Ye et al., 2020]: a limited-data variant L AYER G REEDY FSwhich only uses the same four batches of data used in our method, and a full-data variant L AYERG REEDY FS-fd with access to the full training data.

Original: We report top-1 accuracy results evaluated on the validation set, as we vary the fraction ↵ of prunable(i.e.
Revised: We report top-1 accuracy results evaluated on the validation set, as we vary the fraction ↵ of prunable(i.e.

Original: , belonging to layers being pruned) neurons/channels kept.
Revised: , belonging to layers being pruned) neurons/channels kept.

Original: Accordingly, when pruning L layers,the global budget is set to k =
Revised: Accordingly, when pruning L layers,the global budget is set to k =

Original: ↵
Revised: ↵

Original: P
Revised: P

Original: L ` =1 n ` , where n ` is the number of neurons/channels in layer ` .
Revised: L ` =1 n ` , where n ` is the number of neurons/channels in layer ` .

Original: Unless otherwise speciﬁed, we use the per-layer budget selection method described in Section 5.2for all the layerwise pruning methods, except for L AYER S
Revised: Unless otherwise speciﬁed, we use the per-layer budget selection method described in Section 5.2for all the layerwise pruning methods, except for L AYER S

Original: AMPLING for which we use its ownbudget selection strategy provided in [Liebenwein et al., 2020].
Revised: AMPLING for which we use its ownbudget selection strategy provided in [Liebenwein et al., 2020].

Original: We use a subset of the training set,of the same size as the validation set, as a veriﬁcation set for the budget selection method.
Revised: We use a subset of the training set,of the same size as the validation set, as a veriﬁcation set for the budget selection method.

Original: Since
Revised: Since

Original: L AYER S AMPLING is a random method, the number of neurons/channels it keeps does not alwaysmatch exactly the desired budget k .
Revised: L AYER S AMPLING is a random method, the number of neurons/channels it keeps does not alwaysmatch exactly the desired budget k .

Original: To correct for that, we add or remove neurons/channels withthe lowest/highest probability, as needed.
Revised: To correct for that, we add or remove neurons/channels withthe lowest/highest probability, as needed.

Original: To disentangle the beneﬁt of using our pruning method from the beneﬁt of reweighting (Section 3.2),we report results with reweighting applied to all pruning methods, or none of them.
Revised: To disentangle the beneﬁt of using our pruning method from the beneﬁt of reweighting (Section 3.2),we report results with reweighting applied to all pruning methods, or none of them.

Original: Though, we willfocus our analysis on the more interesting results with reweighting, with the plots without reweightingmostly serving as a demonstration of the beneﬁt of reweighting.
Revised: Though, we willfocus our analysis on the more interesting results with reweighting, with the plots without reweightingmostly serving as a demonstration of the beneﬁt of reweighting.

Original: Results are averaged over ﬁve random runs, with standard deviations plotted as error bars.
Revised: Results are averaged over ﬁve random runs, with standard deviations plotted as error bars.

Original: We exclude W EIGHT N ORM from plots, as it is always the worst performing method.
Revised: Weexclude W EIGHT N ORM from plots, as it is always the worst performing method.

Original: We report the compressionratio ( original size pruned size ) and speedup ( original number of FLOPs pruned number of FLOPs ) values in Appendix J. For additional details onthe experimental set-up, see Appendix G.
Revised: We report Top-5accuracy, compression ratio ( original size [Table] size ), and speedup ( original [Table]

Original:  The code for reproducing all experiments is included in thesupplementary materials.
Revised: pruned [Table] ) values in Appendix F.1. For additional details on the experimental set-up, see Appendix D. The code for reproducing allexperiments is included in the supplementary materials.

Original: LeNet on MNIST We pre-train LeNet model on MNIST achieving 97 .
Revised: LeNet on MNIST We pre-train LeNet model on MNIST achieving 97 .

Original: 75% top-1 accuracy.
Revised: 75% top-1 accuracy.

Original: Weprune all layers except the last classiﬁer layer.
Revised: Weprune all layers except the last classiﬁer layer.

Original: Results are presented in Figure 1 (left).
Revised: Results are presented in Figure 1 (left).

Original: All threevariants of our method consistently outperform other baselines, even when reweighting is appliedto them, with A SYM I N C HANGE doing the best and L AYER I N C HANGE the worst.
Revised: All threevariants of our method consistently outperform other baselines, even when reweighting is applied tothem, with A SYM I N C HANGE doing the best and L AYER I N C HANGE the worst.

Original: We  observe thatreweighting signiﬁcantly improvesthe performance of all methods except L AYER G REEDY FS and
Revised: We also observe thatreweighting signiﬁcantly improves performance for all methods, except L AYER G REEDY FS.

Original: L AYER G REEDY FS-fd.
Revised: It alsoreduces variance in results; the bottom subplot has larger error bars than the top one.

Original: ResNet56 on CIFAR-10 We use the ResNet56 model pre-trained on CIFAR-10 provided in
Revised: ResNet56 on CIFAR-10 We use the ResNet56 model pre-trained on CIFAR-10 provided in

Original: ShrinkBench [Blalock et al., 2020], which achieves 92 .
Revised: ShrinkBench [Blalock et al., 2020], which achieves 92 .

Original: 27% top-1 accuracy.
Revised: 27% top-1 accuracy.

Original: We prune all layers except the last layer in each residual branch, the last layer before each residual branch, and thelast classiﬁer layer.
Revised: We prune all layers except the last layer in each residual branch, the last layer before each residual branch, and thelast classiﬁer layer.

Original: Results are presented in Figure 1 (middle).
Revised: Results are presented in Figure 1 (middle).

Original: The sequential variants of our methodperform the best.
Revised: The sequential variants of our methodperform the best.

Original: Their performance is closely matched by L AYER W EIGHT N ORM for most fractions,except very small ones.
Revised: Their performance is closely matched by L AYER W EIGHT N ORM for most fractions,except very small ones.

Original: L AYER I N C HANGE performs signiﬁcantly worst here than the sequential variants of our method.
Revised: L AYER I N C HANGE performs signiﬁcantly worst here than the sequentialvariants of our method.

Original: This is likely due to the larger number of layers pruned in ResNet56 compared to LeNet (27 vs 4 layers), which increases the effect of pruning earlier layers on subsequent ones.
Revised: This is likely due to the larger number of layers pruned in ResNet56 comparedto LeNet (27 vs 4 layers), which increases the effect of pruning earlier layers on subsequent ones.

Original: Herealso reweighting improves the performance of all methods except the L AYER G REEDY FS variants.

Original: VGG11 on CIFAR-10 We pre-train VGG11 modelon CIFAR-10 obtaining 90 .
Revised: VGG11 on CIFAR-10 We pre-train VGG11 model obtaining 90 .

Original: 11% top-1 accuracy.
Revised: 11% top-1 accuracy.

Original: We prune all layers except the last features layer and the last classiﬁer layer.
Revised: We pruneall layers except the last features layer and the last classiﬁer layer.

Original: Results are presented in Figure 1 (right).
Revised: Results are presented in Figure 1(right).

Original: The three variants of our method perform the best.
Revised: The three variants of our method perform the best.

Original: Their performance is closelymatched by A CT G RAD and L AYER W EIGHT N ORM for most fractions, except very small ones.
Revised: Their performance is closely matchedby L AYER W EIGHT N ORM for most fractions, except very small ones.

Original: L AYER I N C HANGE performs similarly to the sequential variants of our method here, even slightlybetter at fraction 0 .
Revised: L AYER I N C HANGE performssimilarly to the sequential variants of our method here, even slightly better at fraction 0 .

Original: 1 , probably because the number of layers being pruned is again relatively small (9layers).
Revised: 1 , perhapsbecause the number of layers being pruned is again relatively small (9 layers).

Original: As before, reweighting beneﬁts all methods except the L AYER G REEDY FS variants.

Original: Discussion We summarize our observations from the empirical results:
Revised: Discussion We summarize our observations from the empirical results:

Original: • Our proposed pruning method outperforms state-of-the-art structure pruning method in various one shot pruning settings.
Revised: • Our proposed pruning method outperforms state-of-the-art structure pruning method in variousone shot pruning settings.

Original: As expected, A SYM I N C HANGE is the best performing variant of ourmethod and L AYER I N C HANGE the worst, with its performance deteriorating with deeper models.Our results also illustrate the robustness of our method, as
Revised: As expected, A SYM I N C HANGE is the best performing variant of ourmethod, and L AYER I N C HANGE the worst, with its performance deteriorating with deeper models.

Original: it reliably yields

Original:  the best results in various settings, while other baselines perform well in some settings but not in others.
Revised: Our results also illustrate the robustness of our method, as it reliably yields the best results invarious settings, while other baselines perform well in some settings but not in others.

Original: • Reweighting signiﬁcantly improves performance and reduces variance for all methods, except L AYER G REEDY FS and L AYER G REEDY FS-fd.
Revised: • Reweighting signiﬁcantly improves performance and reduces variance for all methods, exceptL AYER G REEDY FS and L AYER G REEDY FS-fd.

Original: We suspect that reweighting does not help in this case because this method already scales the next layer weights, and it takes into account this scaling when selecting neurons/channels to keep, so replacing it with reweighting can hurt performance.
Revised: We suspect that reweighting does not help in thiscase because this method already scales the next layer weights, and it takes into account this scalingwhen selecting neurons/channels to keep, so replacing it with reweighting can hurt performance.

Original: • The choice of how much to prune in each layer given a global budget can have a drastic effect onperformance, as illustrated in Appendix 
Revised: • The choice of how much to prune in each layer given a global budget can have a drastic effect onperformance, as illustrated in Appendix F.•

Original: I.• Fine-tuning with full-training data boosts performance more than reweighting, while ﬁne-tuningwith limited data helps less, as illustrated in Appendix H. Reweighting still helps even whenﬁne-tuning with limited-data is used, except for L AYER G REEDY FS and L AYER G REEDY FS-fd,but it can actually deteriorate performance when ﬁne-tuning with full-data is used.
Revised: Fine-tuning with full-training data boosts performance more than reweighting, while ﬁne-tuningwith limited data helps less, as illustrated in Appendix E. Reweighting still helps even whenﬁne-tuning with limited-data is used, except for L AYER G REEDY FS and L AYER G REEDY FS-fd,but it can actually deteriorate performance when ﬁne-tuning with full-data is used.

Original: Our method stilloutperforms other baselines after ﬁne-tuning with limited-data, and is among the best performingmethods even in the full-data setting.
Revised: Our method stilloutperforms other baselines after ﬁne-tuning with limited-data, and is among the best performingmethods even in the full-data setting.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a data-efﬁcient structured pruning method, based on submodular optimization.
Revised: We proposed a data-efﬁcient structured pruning method, based on submodular optimization.

Original: Bycasting the layerwise subset selection problem as a weakly submodular optimization problem, we areable to use the G REEDY algorithm to provably approximate it.
Revised: Bycasting the layerwise subset selection problem as a weakly submodular optimization problem, we areable to use the G REEDY algorithm to provably approximate it.

Original: Empirically, our method consistentlyoutperforms existing structured pruning methods on different network architectures and datasets.Checklist
Revised: Empirically, our method consistentlyoutperforms existing structured pruning methods on different network architectures and datasets.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] We specify that our focus is onthe limited-data regime in both the abstract and introduction.

Original: See also the discussionson lines 143-151 and

Original: -213 on when our approximation guarantee is non-zero, and

Original: on lines 173-181 on the cost of our method.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix B

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?

Original: [Yes] We include

Original: the code with instructions on how to reproduce all our experimental results in the

Original: supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] See Section 6 and Appendix G

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix G

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Appendix G

Original: (b) Did you mention the license of the assets?

Original: [Yes] The license of the assets we used are

Original: included in the code we provide.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: We include our code in the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c)

Original: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]