Original: Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions
Revised: Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Pruning is one of the predominant approaches for compressing deep neural networks (DNNs).
Revised: Pruning is one of the predominant approaches for compressing deep neural networks (DNNs).

Original: Lately, coresets (provable data summarizations) were leveragedfor pruning DNNs, adding the advantage of theoretical guarantees on the trade-offbetween the compression rate and the approximation error.
Revised: Lately, coresets (provable data summarizations) were leveragedfor pruning DNNs, adding the advantage of theoretical guarantees on the trade-offbetween the compression rate and the approximation error.

Original: However, coresets inthis domain were either data-dependent or generated under restrictive assumptionson both the model’s weights and inputs.
Revised: However, coresets inthis domain were either data dependant or generated under restrictive assumptionson both the model’s weights and inputs.

Original: In real-world scenarios, such assumptionsare rarely satisfied, limiting the applicability of coresets.
Revised: In real-world scenarios, such assumptionsare rarely satisfied, limiting the applicability of coresets.

Original: To this end, we suggest anovel and robust framework for computing such coresets under mild assumptionson the model’s weights and without any assumption on the training data.
Revised: To this end, we suggest anovel and robust framework for computing such coresets under mild assumptionson the model’s weights and without any assumption on the training data.

Original: The ideais to compute the importance of each neuron in each layer with respect to the outputof the following layer.
Revised: Theidea is to compute the importance of each neuron in each layer with respect tothe output of the following layer.

Original: This is achieved by a combination of Löwner ellipsoid and Caratheodory theorem.
Revised: This is achieved by an elegant combination of Löwner ellipsoid and Caratheodory theorem.

Original: Our method is simultaneously data-independent, applicableto various networks and datasets (due to the simplified assumptions), and theoretically supported.
Revised: Our method is simultaneously dataindependent, applicable to various networks and datasets (due to the simplifiedassumptions), and theoretically supported.

Original: Experimental results show that our method outperforms existingcoreset based neural pruning approaches across a wide range of networks anddatasets.
Revised: Experimental results show that ourmethod outperforms existing coreset based neural pruning approaches across awide range of networks and datasets.

Original: For example, our method achieved a 62% compression rate on ResNet50on ImageNet with 1 . 09% drop in accuracy.
Revised: For example, our method achieved a 62%compression rate on ResNet50 on ImageNet with 1 . 09% drop in accuracy.

Original:  Introduction and Backround
Revised:  Introduction and Backround

Original: Deep neural networks (DNNs) achieved state-of-the-art (SOTA) performance on a large variety oftasks, e g., in computer vision [33, 50] and natural language processing (NLP; [84, 20]).
Revised: Deep neural networks (DNNs) achieved state-of-the-art (SOTA) performance on a large variety oftasks, e g., in computer vision [33, 51] and natural language processing (NLP; [84, 20]).

Original: However,
Revised: However,

Original: DNNs usually contain millions or even billions of parameters in order to achieve SOTA performancesresulting in large storage requirements and long inference time.
Revised: DNNs usually contain millions or even billions of parameters in order to achieve SOTA performancesresulting in large storage requirements and long inference time.

Original: This is obstructive when, e.g., dealingwith limited hardware or real-time systems such as autonomous cars and text/speech translation.
Revised: This is obstructive when, e.g., dealingwith limited hardware or real-time systems such as autonomous cars and text/speech translation.

Original: Tothis end, a large body of research is dedicated to reducing the size and inference costs of DNNs.
Revised: Tothis end, a large body of research is dedicated to reducing the size and inference costs of DNNs.

Original: Pruning.
Revised: Pruning.

Original: A dominant approach widely used for reducing the size of DNNs is to utilize a pruningalgorithm to remove redundant parameters from the original, over-parameterized network.
Revised: A dominant approach widely used for reducing the size of DNNs is to utilize a pruningalgorithm to remove redundant parameters from the original, over-parameterized network.

Original: In general,pruning can be categorized into two main types: (i) Unstructured pruning [31, 6] reduces thenumber of non-zero parameters by inducing sparsity into weight parameters, which can achievehigh compression rates but requires specialized software and/or hardware in order to achieve fasterinference times.
Revised: In general,pruning can be categorized into two main types: (i) Unstructured pruning [31, 7] reduces thenumber of non-zero parameters by inducing sparsity into weight parameters, which can achievehigh compression rates but requires specialized software and/or hardware in order to achieve fasterinference times.

Original: (ii) Structured pruning [35, 60, 74] modifies the structure of the underlying weighttensors, by removing filters/neurons from each layer, usually resulting in smaller compression rateswhile directly achieving faster inference times with no specialized software; see section 

Revised: (ii) Structured pruning [35, 61] modifies the structure of the underlying weighttensors, by removing filters/neurons from each layer, usually resulting in smaller compression rateswhile directly achieving faster inference times with no specialized software; see section

Original: 1.1 Coresets for Pruning
Revised: 1.1 Coresets for Pruning

Original: Notably, many recent papers focused on various types of filter pruning [85, 76] potentially dueto the empirical observation that existing filter pruning approaches consistently yield impressiveresults.
Revised: Notably, many recent papers focused on various types of filter pruning [85, 76] potentially dueto the empirical observation that existing filter pruning approaches consistently yield impressiveresults.

Original: However, most pruning methods are based on heuristics, lacking theoretical guarantees onthe trade-off between the compression rate and the approximation error.
Revised: However, most pruning methods are based on heuristics, lacking theoretical guarantees onthe trade-off between the compression rate and the approximation error.

Original: This was the motive forintroducing coresets [79, 60] to the world of pruning.
Revised: This was the motive forintroducing coresets [79, 61] to the world of pruning.

Original: Coresets.
Revised: Coresets.

Original: In machine learning, we are (usually) given an input set P ⊆
Revised: In machine learning, we are (usually) given an input set P ⊆

Original: R
Revised: R

Original: d of n points, itscorresponding weights function w : P → R , a feasible set of queries X , and a loss functionϕ : P × X → [0 , ∞ ) .
Revised: d of n points, itscorresponding weights function w : P → R , a feasible set of queries X , and a loss functionϕ : P × X → [0 , ∞ ) .

Original: The tuple ( P, w, X, ϕ ) is called query space , and it defines the optimizationproblem at hand.
Revised: The tuple ( P, w, X, ϕ ) is called query space , and it defines the optimizationproblem at hand.

Revised: Usually the goal is to find the query x ∗ which minimizes (cid:80)

Original:  For a given problem that is defined by its query space ( P, w, X, ϕ ) , and an errorparameter ε ∈ (0 , 1) , an ε -coreset is  is a small weighted subset of the input points thatapproximatesthe loss of the input set P for every feasible query x , up to a provable bound of 1 + ε .
Revised: p ∈ P w ( p ) ϕ ( p, x ) overevery x ∈ X . For a given problem that is defined by its query space ( P, w, X, ϕ ) , and an errorparameter ε ∈ (0 , 1) , an ε -coreset is a pair ( C, u ) , where C ⊆ P , and u : C → R is a weightfunction, such that for every feasible query x ∈ X

Original: Since coresets approximate the cost of every query, traditional (possibly inefficient) algorithms/solverscan be applied on coresets to obtain an approximation of the optimal solution on the full data, usingless time and memory; see Section B.2 in the appendix for more details.
Revised: we have (cid:12)(cid:12)(cid:12) [Table] − [Table]

Revised: ≤ ε .

Original: Pruning via coresets.
Revised: Pruning via corestes.

Original: Recently, some inspiring innovative frameworks [79, 60, 5, 7] leveraged theidea of coresets for pruning DNNs.
Revised: Recently, some inspiring innovative frameworks [79, 61, 6] leveraged the ideaof coresets for pruning DNNs.

Original: Any layer ℓ can be represented as a set P = { p 1 , · · · , p n } ofn > 1 points in R d , where d is the number of input neurons, and n in the number of output neurons,i.e., each point p ∈ P , represents a specific neuron using itsd weights(parameters). When the layerreceives an input vector x ∈ R d , it outputs the vector ( ϕ ( p T 1 x ) , · · · , ϕ ( p Tn x )) , where ϕ : R → R isan activation function  which defines a non-linear mapping. Focusing on a single neuron η inthelayer that follows ℓ , defined by its corresponding vector of weights w = ( w 1 , · · · , w n ) , we set inthe context of coresets, w ( p i ) : = w i for every i ∈ { 1 , · · · , n } - this is just a mapping from p i tow i to simplify the writing and reading.
Revised: Here, each layer in a given network is represented as a set P ⊂ R dassociated with its weights function w - which is the weights of a specific neuron η in the next layer.

Original: Note that the output of this neuron is (cid:80)p ∈ P w ( p ) ϕ (cid:0) p T x (cid:1) . Assuming that we are given an ε -coreset
Revised: The input to this layer is a vector x from the query space X , and the cost ϕ we wish to approximateis the output of this neuron η : (cid:80)

Original: ( C, u ) for the query space (cid:0) P, w, R d , ϕ (cid:1) where C ⊆ P , andu : C

Original: → R , we have that ( C, u ) approximates the output of this specific neuron η for every queryusing less parameters; see Figure 1.

Original: To formalize the stated above, we now define coresets in thecontext of activation functions.

Original: Definition 1.1 (Coreset for activation functions) .

Original: Let ε ∈

Original: (0 , 1) , and let (cid:0) P, w, R d , ϕ (cid:1) be a queryspace.

Original: Then the pair ( C, u ) , is an ε -coreset for (cid:0) P, w, R d , ϕ (cid:1) if (

Original: i) C ⊆ P , (ii) u : C → [0 , ∞ ) , and(iii) for every x ∈ X , (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]

Original: − (cid:80) q

Original: [Equation] q ) ϕ( q T
Revised: p ∈ P w ( p ) ϕ

Original: [Equation]

Original: (cid:80) p ∈ P w

Original: [Equation] ≤ ε. Since C is a subset of P , we can remove (assign zero to) all weights from w that corresponds topoints not chosen to be in C from P , and replace the weights of the chosen points in C with the newweights vector u ; see Figure 1 in [79] fora visual illustration. To prune neurons, we refer the readerto Section 2.5 as it is a simple extension.
Revised: ( p T x ) ; see Section 2 for the full technical details.

Original: Prior work showed that such approaches successfully resultin high compression rates across a wide range of networks and datasets, and even achieves SOTAperformance on a verity of them.
Revised: Priorwork showed that this approach successfully results in high compression rates across a wide range ofnetworks and datasets, and even achieves SOTA performance on a verity of them.

Original: The main (strong) advantage of the coreset approach over others was the provided provable theoreticalguarantees on the tradeoff between the compression rate and the approximation error, which supports worse case scenarios.
Revised: The main (strong)advantage of the coreset approach over others was the provided provable theoretical guarantees onthe tradeoff between the compression rate and the approximation error, which supports worse casescenarios.

Original: In addition, coresets play an important role in improving the generalizationproperties of the trained networks [5, 75].
Revised: In addition, coresets play an important role in improving the generalization properties ofthe trained networks [6, 75].

Original: Sensitivity sampling for constructing pruning coresets.

Original: To compute such coresets, both [60, 79]utilised the known sensitivity sampling framework [9, 52].

Original: In short the sensitivity of a point p ∈ P insome query space ( P, w, X, ϕ ) corresponds to the importance of this point with respect to the queryspace at hand, and it is defined as s ( p ) =

Original: sup x ∈ X w

Original: [Equation]

Original: p ) ϕ

Original: [Equation]

Original: ,x )

Original: [Equation]

Original: q

Original: [Equation]

Original: w

Original: [Equation]

Original: q ) ϕ

Original: ( q,x ) - where the denominator is notequal to zero.

Original: Once we bound these sensitivities, we can sample points (neurons) from P accordingto sensitivity bounds, and re-weight the sampled points to obtain a coreset.

Original: The size of the sample isproportional to the sum of these bounds.

Original: See Section B.1 and Theorem B.2 for more details in the

Original: Appendix.

Original: 1.2 Our contribution
Revised: 1.2 Our contribution

Original: Prior coreset methods for pruning DNNs either (i) imposed restrictive assumptions both on themodel’s weights and inputs [79], i.e., the input set P representing the neurons, and the query set Xwhich represents the inputs of the layer, are enclosed in a ball in R d of radius r 1 and r 2 , respectively,or (ii) the methods are data-dependent, i.e., use a mini-batch of the input set to measure the influenceof each parameter on the loss function [5, 60].
Revised: Prior coreset methods for pruning DNNs either (i) imposed restrictive assumptions both on themodel’s weights and inputs [79], i.e., the input set P representing the neurons, and the query set Xwhich represents the inputs of the layer, are enclosed in a ball in R d of radius r 1 and r 2 , respectively,or (ii) the methods are data-dependent, i.e., use a mini-batch of the input set to measure the influenceof each parameter on the loss function [6, 61].

Original: To this end, in this work, we take coresets a step further into the realm of pruning by introducinga unified framework with provable guarantees for pruning DNNs (weights and neurons/filters)while minimally affecting the generalization error.
Revised: To this end, in this work, we take coresets a step further into the realm of pruning by introducinga unified framework with provable guarantees for pruning DNNs (weights and neurons/filters)while minimally affecting the generalization error.

Original: Our main improvement is that our framework issimultaneously (i) data-independent , (ii) requires a single assumption on the model’s weights,and (iii) provably guarantees a multiplicative factor approximation , which is favourable uponadditive approximations; see Theorem 2.6.
Revised: Our main improvement is that our framework issimultaneously (i) data-independent , (ii) requires a single assumption on the model’s weights, and(iii) provably guarantees a multiplicative factor approximation , which is favourable upon additiveapproximations; see Theorem 2.8.

Original: The approach is based on the widely used theory ofcoresets allowing us to suggest a provable guarantee on the tradeoff between the approximation errorand compression rate for each layer.
Revised: The approach is based on the widely used theory of coresetsallowing us to suggest a provable guarantees on the tradeoff between the approximation error andcompression rate for each layer.

Original: We conducted experimental results which established new SOTA benchmarks for structured pruningvia coresets across a wide range of networks and datasets.
Revised: We conducted experimental results which established new SOTAbenchmarks for structured pruning via coresets across a wide range of networks and datasets.

Original: We share all of our resulted models [14].
Revised: Finally,we share all of our resulted models [14].

Original:  Method
Revised:  Method

Original: In general, the coreset (for pruning) technique hinges upon the insight that any linear layer such asconvolutions, can be casted as a matrix multiplication [79].
Revised: In general, the coreset (for pruning) technique hinges upon the insight that any linear layer such asconvolutions, can be casted as a matrix multiplication [79].

Original: Hence, we focus in what follows on fullyconnected (FC) layers, while the details holds for any linear layer.
Revised: Hence, we focus in what follows on fullyconnected (FC) layers, while the details holds for any linear layer.

Original: Furthermore, for simplicity, weassume in what follows that the weights of  P are all equal to 1 and thus our query space is denote by 102 (cid:0)
Revised: How to prune using coresets? Any layer ℓ can be represented as a set P = { p 1 , · · · , p n } of n > 1points in R d , where d is the number of input neurons, and n in the number of output neurons, i.e.,each point p ∈ P , represents a specific neuron using its d weights (parameters). When the layerreceives an input vector x ∈ R d , it outputs the vector ( ϕ ( p T 1 x ) , · · · , ϕ ( p Tn x )) , where ϕ : R → R is anactivation function which defines a non-linear mapping. Focusing on a single neuron η in the layer thatfollows ℓ , defined by its corresponding vector of weights w = ( w 1 , · · · , w n ) , we set in the context of coresets, w ( p i ) :

Original: P, R d , ϕ (cid:1) , and the sensitivity of a point p ∈ P is simply s ( p ) =
Revised: = w i for every i ∈ [ n ] . Note that the output of this neuron is (cid:80) p ∈

Original: sup x ∈ X
Revised: P w ( p [Equation] (cid:0)

Original: [Equation]
Revised: [Equation]

Original:  ϕ 
Revised: x (cid:1) . Assuming that we are given an ε -coreset ( C, u ) for the query space (cid:0) P, w, R d , ϕ (cid:1) where C ⊆ P , andu : C → R , we have that ( C, w ) approximates the output of this specific neuron η for every query as

Original: [Equation]
Revised: [Equation]

Original: q,x ) .
Revised: x

Original: Note that ourproofs are easily extended to the general case where we are given a weight function w : P → R asdiscussed in Section 2.5.
Revised: [Equation] [Equation] ≤ ε . Since C is a subset of P , we can remove (assign zero to) allweights from w that corresponds to points not chosen to be in C from P , and replace the weights ofthe chosen points in C with the new weights vector u ; See Figure 1 at [79] for a visual illustration.

Revised: To prune neurons, we refer the reader to Section 2.5 as it is a simple extension.

Revised: We now present ourmethod for computing such coresets for each neuron.

Revised: First, we give our preliminaries.

Original: 2.1 Preliminaries
Revised: 2.1 Preliminaries

Original: Notations.
Revised: Notations.

Original: For a positive integer n , we use [ n ] to denote the set { 1 , . . . , n } .
Revised: For a positive integer n , we use [ n ] to denote the set { 1 , . . . , n } .

Original: For c ∈ R d and a symmetric positive definite matrix G ∈ R d × d , we define E ( G, c ) :
Revised: For c ∈ R d and a symmetric positive definite matrix G ∈ R d × d , we define E ( G, c ) :

Original: = (cid:110) x ∈ R d (cid:12)(cid:12)(cid:12) ( x − c ) T G ( x − c ) ≤ 1 (cid:111)to be the ellipsoid defined by c and G .
Revised: = (cid:110) x ∈ R d (cid:12)(cid:12)(cid:12) ( x − c ) T G ( x − c ) ≤ 1 (cid:111)to be the ellipsoid defined by c and G .

Original: For an ellipsoid E ( G, c ) , each endpoint of a semi principalaxis is called a vertex of E ( G, c ) .
Revised: For an ellipsoid E ( G, c ) , each endpoint of a semi principalaxis is called a vertex of E ( G, c ) .

Original: We define rank ( P ) for any set P ⊆
Revised: We define rank ( P ) for any set P ⊆

Original: R d to be the dimension ofthe affine subspace that P lies on.
Revised: R d to be the dimension ofthe affine subspace that P lies on.

Original: For a set P ⊂ R d the convex hull of P is denoted by Conv( P ) .
Revised: For a set P ⊂ R d the convex hull of P is denoted by Conv( P ) .

Original: Finally, vectors are treated as column vectors.
Revised: Finally, vectors are treated as column vectors.

Revised: The following definition summarizes all the ingredients needed to define a query space with respectto activation functions.

Revised: Definition 2.1 (Activation functions query space) .

Revised: Let n, d be a pair of positive integers, ϕ : R →[0 , ∞ ) be an activation function, and let P ⊆

Revised: R d be a set of n points.

Original:  2.2 Novelty - Löwner ellipsoid meets Carathéodory
Revised: Then the tuple (cid:0) P, R d , ϕ (cid:1) iscalled a query space . Note that, for simplicity, we assumed in Definition 2.1 that the weights of P are all equal to 1 ,while our proofs can be easily extended to the general case where we are given a weight functionw : P → R as discussed at Section 2.5. We now define a coreset with respect to an activationfunction. Definition 2.2 (Coreset for activation functions) .

Revised: Let ε ∈

Revised: (0 , 1) , and let (cid:0) P, R d , ϕ (cid:1) be a queryspace as in Definition 2.1.

Revised: Then the pair ( C, u ) , is an ε -coreset for (cid:0) P, R d , ϕ (cid:1)

Revised: if (i) C ⊆ P , (ii)u : C → [0 , ∞ ) , and (iii) for every x ∈ R d and b ∈ R , (cid:12)(cid:12)(cid:12)(cid:12) 1 − (cid:80) q ∈ C u ( q ) ϕ

Revised: ( q T x + b ) (cid:80) p ∈ P ϕ ( p T x + b ) (cid:12)(cid:12)(cid:12)(cid:12) ≤ ε.

Revised: Computing pruning coresets via sensitivity sampling.

Revised: To compute such coresets, both [61, 79]utilised the known sensitivity sampling framework [9, 53].

Revised: In short, the sensitivity of a point p ∈ Pin some query space ( P, X, ϕ ) is defined as s ( p ) =

Revised: sup x ∈ X ϕ ( p,x ) (cid:80) q ∈ P ϕ ( q,x ) - where the denominator isnot equal to zero, and it corresponds to the importance of this point with respect to the query spaceat hand.

Revised: Once we bound these sensitivities, we can sample points (neurons) from P according tosensitivity bounds, and re-weight the sampled points to obtain a coreset.

Revised: The size of the sample isproportional to the sum of these bounds.

Revised: See Theorem B.2 for formal details at the Appendix.

Revised: 2.2 Novelty - Löwner ellipsoid meets Carathéodory

Original: Our method hinges upon a combination of two known tools from convex geometry.
Revised: Our method hinges upon an elegant combination of two known tools from convex geometry.

Original: The noveltyof our approach exploits the following observation.
Revised: Thenovelty of our approach exploits the following observation.

Original: Most activation functions ϕ are continuousnon-decreasing functions, which indicate that for every query x and a set of points P , the maximalcontribution to (cid:80) p ∈ P ϕ
Revised: Most activation functions ϕ are continuousnon-decreasing functions, which indicate that for every query x and a set of points P , the maximalcontribution to (cid:80) p ∈ P ϕ

Original: ( p T x ) with respect to such activation function is associated to a point onthe convex hull of P .
Revised: ( p T x ) with respect to such activation function is associated to a point onthe convex hull of P .

Original: By finding a geometrical body B of bounded number of vertices, that is (i) enclosed in Conv ( P ) and (ii) with some dilation factor (expanding) B enclose Conv ( P ) , we willbe able to represent each point p on the boundary of the convex hull of P as a convex combination oftwo points p 1 , p 2 , one of each ( p 1 ) on B and the second ( p 2 ) on its dilated form, which is formalizedas the set { α ( x − c ) + c | x ∈ B, ∀ α ∈ [0 , 1] } , where c here denotes the center of B .
Revised: By finding a geometrical body B of bounded number of vertices, that is (i)enclosed in Conv ( P ) and (ii) with some dilation factor (expanding) B enclose Conv ( P ) , we willbe able to represent each point p on the boundary of the convex hull of P as a convex combinationof two points p 1 , p 2 , one of each ( p 1 ) on B and the second ( p 2 ) on its dilated form.

Original: For such task,
Revised: For such task,

Original: Löwner ellipsoid is leveraged.
Revised: Löwner ellipsoid is leveraged.

Original: Theorem 2.1 (John-Löwner ellipsoid[42]) .
Revised: Theorem 2.3 (John-Löwner ellipsoid[43]) .

Original: Let

Original:  L ⊆ R dbe a set of points such that the convexhull of L has a nonempty interior. Then, there exists an ellipsoid E ( G, c ) (also known as the MVEE ), where G ∈ R d × d is a positive definite matrix and c ∈ R d , of minimal volume such that1 d ( E ( G, c ) − c ) + c ⊆
Revised: For a set L ⊆ R d of points with nonempty interior, thereexists an ellipsoid E ( G, c ) (also known as the MVEE ), where G ∈ R d × d is a positive definite matrixand c ∈ R d , of minimal volume such that 1 d ( E ( G, c ) − c ) + c ⊆

Original: Conv ( L ) ⊆
Revised: Conv ( L ) ⊆

Original: E ( G, c ) .
Revised: E ( G, c ) .

Original: If L is symmetric around the center c , then thedilation factor can be reduced to 1 √ d .
Revised: If L issymmetric around the center c , then the dilation factor can be reduced to 1 √ d .

Revised: Afterwards, p 1 and p 2 should be represented by points from P .

Revised: Each point on B (specifically, p 1 )can be represented via a convex combination of d + 1 points from P .

Revised: The same holds for points onthe dilated form of B (e.g., p 2 ) but via a conical combination (linear combination where the weightsare non-negative).

Original: Algorithm
Revised: This problem is solved elegantly by invoking Carathéodory theorem. Theorem 2.4 ([10, 92]) . For any A ⊂ R d and p ∈ Conv( A ) , there exists m ≤ d + 1 pointsp 1 , . .

Revised: , p m ∈ A (denoted by a Caratheodory set of p )

Revised: such that p ∈

Revised: Conv ( { p 1 , . .

Revised: , p m } ) .

Revised: Finally, it is known that e.g., the ReLU function does not admit an ε -coreset with o ( n ) points [79, 78].

Revised: Thus, we use a generalized form of what is known as the complexity measure of a set of points, whichwas first introduced at [78] and later leveraged at [74].

Revised: This measure is used to determine (i) thecomplexity of a given set P with respect ReLU, and (ii) the coreset size theoretically.

Revised: Definition 2.5 (Regression Complexity Measure) .

Revised: Let P ⊆

Revised: R

Revised: d × { 1 } , the regression complexitymeasure of P is defined as µ ( P ) =

Revised: sup x ∈ R d +

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: T x | , where the denominator is ≥ 0 ,and the last entry of every p ∈ P is 1 , reserved for the bias/intercept term.

Revised: shown in (ii)) using Definition 2.5.

Revised: Following this step, A set of points P can be projected into a low dimensional subspace of dimension rank ( P ) using any dimensionality reduction algorithm as presented in (iii), resulting in the set P ′ .

Revised: (iv) The convex hull (blue dashed lines) P ′ (red points) is enclosed by its

Revised: Löwner ellipsoid (depicted in green).

Revised: ) Finally, for each vertex (magenta star) of the enclosed ellipsoid (black ellipsoid), its Carathédory set is dashed lines).

Revised: 2.3 Our Pruning Scheme

Original: Input : P ⊆ R d of n points with rank r
Revised: Input : P ⊆ R d of n points with rank r

Original: Output : A subset S ⊆ P that satisfies
Revised: Output : A subset S ⊆ P that satisfies

Original: Lemma 2.
Revised: Lemma 2.

Original: 1 ( Y, z ) := affine subspace that P lies on, 
Revised: 1 ( Y, z ) := affine subspace that P lies on, i.e. P ⊆

Original: [Equation]
Revised: [Equation]

Revised: xY

Original: [Equation]
Revised: [Equation]

Original: 3 Let map : P ′ → P a function that maps every p ′ ∈ P ′ to its corresponding point p
Revised: + z (cid:12) [Equation] d

Original: [Equation]
Revised: [Equation]

Original:  P 4 S
Revised: 2 P ′ :

Original: [Equation] ∅
Revised: = (cid:8) p

Original: [Equation] K [Equation] 5 E
Revised: = pY T (cid:9) 3 Let map : P ′

Original: [Equation] G, c [Equation]
Revised: → P a function that maps every p ′ ∈ P ′ to its corresponding point p ∈ P 4 S := ∅ ; K := ∅ 5 E ( G, c ) :

Original:  MVEE [Equation] Conv
Revised: = MVEE (Conv ( P ′ ))

Original: [Equation] P ′ )) 6 V := the vertices of ellipsoidr ( E ( G, c ) − c )
Revised: 6 V := the vertices of ellipsoidr ( E ( G, c ) − c )

Original: + c 7 for every v ∈ V do
Revised: + c 7 for every v ∈ V do

Original: K :=
Revised: K :=

Original: K ∪ C ARATHEODORY -S ET ( v, P ′ ) //
Revised: K ∪ C ARATHEODORY -S ET ( v, P ′ ) //

Original: See Algorithm 4 in the supplementary material.
Revised: See Algorithm 4 at the supplementary material.

Original: 9 end 10 S :
Revised: 9 end 10 S :

Original: = { map ( q ) | q ∈ K } 11 return S
Revised: = { map ( q ) | q ∈ K } 11 return S

Original: Algorithm input :
Revised: Algorithm input :

Original: A set P ⊆
Revised: A set P ⊆

Original: R d of n points, and a sample size
Revised: R d of n points, and a sample size

Original: m output : A weighted set ( C, u ) 1 Q := P , i := 1 , C := ∅ 2 while | Q | ≥ 2rank ( Q ) 2 do
Revised: m output : A weighted set ( C, u ) 1 Q := P , i := 1 , C := ∅ 2 while | Q | ≥ 2rank ( Q ) 2 do

Original: S i := ℓ ∞ -C ORESET ( Q ) for every p ∈ S
Revised: S i := ℓ ∞ -C ORESET ( Q ) for every p ∈ S

Original: i do s ( p ) :
Revised: i do s ( p ) :

Original: = 2rank( Q ) 1 .
Revised: = 2 r 1 .

Original: 5 i end
Revised: 5 i end

Original: Q :=
Revised: Q :=

Original: Q \ S i , i :=
Revised: Q \ S i , i :=

Original: i +8 end 9 for every p ∈ Q do s ( p ) :
Revised: i +8 end 9 for every p ∈ Q do s ( p ) :

Original: = 2rank( Q ) 1 .
Revised: = 2 r 1 .

Original: 5 i 11 end 12 t := (cid:80)
Revised: i 11 end 12 t := (cid:80)

Original: p ∈
Revised: p ∈

Original: P s ( p ) 13 C := an i.i.d sample of m points from P , where each p ∈ P is sampled with probability s ( p ) t .
Revised: P s ( p ) 13 C := an i.i.d sample of m points from P , where each p ∈ P is sampled with probability s ( p ) t .

Original: 14 u ( p ) :
Revised: 14 u ( p ) :

Original: = t m · s ( p ) for every p ∈ C 15 return ( C, u )
Revised: = t m · s ( p ) for every p ∈ C 15 return ( C, u )

Original: Afterwards, p 1 and p 2 should be represented by points from P .

Original: Each point on B (specifically, p 1 )can be represented via a convex combination of d + 1 points from P .

Original: The same holds for points onthe dilated form of B (e.g., p 2 ) but via a conical combination (linear combination where the weightsare non-negative and the sum of weights is not necessarily 1 ).

Original: This problem is solved by invoking

Original: Carathéodory theorem.

Original: regression loss function (shown in (ii)) using

Original: Definition 2.3.

Original: Following this step, a set of points P can be projected into a low dimensional subspace of dimension rank ( P ) using any dimensionality reduction algorithm as presented in (iii), resulting in the set P is its

Original: Theorem 2.2 ([10, 92]) .

Original: For any A ⊂ R d and p ∈

Original: Conv( A ) , there exists m ≤ d + 1 pointsp 1 , . .

Original: , p m ∈ A (denoted by a Carathéodory set of p )

Original: such that p ∈

Original: Conv ( { p 1 , . .

Original: , p m } ) .

Original: Finally, it is known that some functions, including the ReLU function, do not admit an ε -coreset ofsize o ( n ) [79, 78].

Original: Thus, we use a generalized form of what is known as the complexity measure of aset of points, which was first introduced in [78] and later leveraged in [73].

Original: This measure is used todetermine the complexity of a given set P with respect to ReLU, and the coreset size theoretically.

Original: Definition 2.3 (Regression Complexity Measure) .

Original: Let P ⊆

Original: R

Original: d × { 1 } , the regression complexitymeasure of P is defined as µ ( P ) =

Original: sup x ∈ R

Original: [Equation]

Original: | , where the denominator is ≥ 0 ,and the last entry of every p ∈ P is 1 , reserved for the bias/intercept term.

Original: 2.3 Our Pruning Scheme

Original: In what follows, we present our data summarization technique for ReLU on the dot product function.
Revised: In what follows, we present our data summarization technique for ReLU on the dot product function.

Original: Then in Section 2.5, we discuss that our results can be easily extended to a wide family of activationfunctions including the Sigmoid function, as recently shown in [73].
Revised: Then at Section 2.5, we discuss that our results can be easily extended to a wide family of activationsincluding e.g. Sigmoid, as recently shown at [74].

Original: First we present Algorithm 1,which serves as a stepping stone towards bounding the sensitivities.
Revised: First we present Algorithm 1, which serves as astepping stone towards bounding the sensitivities.

Original: Overview of Algorithm 1.
Revised: Overview of Algorithm 1.

Original: The algorithm receives as input a set P ⊂ R
Revised: The algorithm receives as input a set P ⊂ R

Original: d whose rank is
Revised: d whose rank is

Original: r ∈ [ d ] anddeterministically finds a subset S ⊆ P which satisfies that for every j ∈ [ d ] , X ∈ R d × j and v ∈ R d ,
Revised: r ∈ [ d ] anddeterministically finds a subset S ⊆ P which satisfies that for every j ∈ [ d ] , X ∈ R d × j and v ∈ R d ,

Original: [Figure]
Revised: [Equation]

Revised: max

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: X

Original: [Equation]
Revised: [Equation]

Original:  To do so, first, we find the affine hyperplane that P lies on, followed bycomputing the low dimensional representation of P , denoted by P ′ ; see Lines 1–2.
Revised: ≤ r 1 . 5 . To do so, first, we find the affine hyperplane that P lies on, followed bycomputing the low dimensional representation of P , denoted by P ′ ; see Lines 1–2.

Original: Note that ifrank( P ) =
Revised: Note that ifrank( P ) =

Original: d , then we can either keep P as it is (i.e., P ′ :
Revised: d , then we can either keep P as it is (i.e., P ′ :

Original: = P ), or use dimensionality reductiontricks as detailed in Section 2.5.
Revised: = P ), or use dimensionality reductiontricks as detailed in Section 2.5.

Original: To compute the output S , we first bound the convex hull of P ′ by its
Revised: To compute the output S , we first bound the convex hull of P ′ by its

Original: Löwner ellipsoid E ( G, c ) in Line 5, followed by computing the dilated ellipsoid of E ( G, c ) , namely,1 r ( E − c ) + c .
Revised: Löwner ellipsoid E ( G, c ) at Line 5, followed by computing the dilated ellipsoid of E ( G, c ) , namely,1 r ( E − c ) + c .

Original: Let V be the set of vertices of such ellipsoid; see Line 6.
Revised: Let V be the set of vertices of such ellipsoid; see Line 6.

Original: Now, for each point v ∈ V ,we represent it as a convex combination of r + 1 points from P ′ via Theorem 2.2, and store the unionof such sets (each of size at most r + 1 ) of points into K as done in Lines 7–9.
Revised: Now, for each point v ∈ V ,we represent it as a convex combination of r + 1 points from P ′ via Theorem 2.4, and store the unionof such sets (each of size at most r + 1 ) of points into K as done at Lines 7–9.

Original: For each point in K ,we map it back to R d to satisfy Lemma 2.5.
Revised: Finally, for each pointin K , we map it back to R d to satisfy Lemma 2.7.

Original: To sum up Algorithm 1, we observe that the vertices canbe used via canonical combinations with their dilated form to describe every point on the convex hullof the input data (in our end, it would the network’s weights).

Original: Hence the Carathéodory set of thesevertices from the input points lying on the convex hull can be further used to also represent pointslying on the convex hull.

Original: This is the core idea which enable us in forming our ℓ ∞ coreset for any ℓ ρregression problem where ρ ∈ (0 , ∞ ) .

Original: We now discuss Algorithm 2 which is responsible for constructingan ε -coreset with respect toactivation functions.
Revised: We now discuss Algorithm 2 which is responsible for constructing ε -coreset with respect to activationfunctions.

Original: Its input is a set P ⊂ R d and a sample size m ≥ 1 .
Revised: Its input is a set P ⊂ R d and a sample size m ≥ 1 .

Original: Overview of Algorithm 2.
Revised: Overview of Algorithm 2.

Original: First set Q := P .
Revised: First set Q := P .

Original: At each iteration i , the algorithm obtains a subset
Revised: At each iteration i , the algorithm obtains a subset

Original: S i ⊆ Q as an output to a call to ℓ ∞ -C ORESET ( Q ) as stated in Line 3 of Algorithm [-
-] such that forevery x ∈ R d  , it holds that 
Revised: S i ⊆ Q as an output to a call to ℓ ∞ -C ORESET ( Q ) as stated at Line 3 of Algorithm [-
-]1 such that forevery x ∈ R d and b ∈ R , it holds that max p ∈ P | p

Original: [Equation]
Revised: T x + b | max p ∈ Si |

Original:  ≤ r 1 . 5with r being the rank of Q .
Revised: p T x + b | ≤ r 1 . 5 .

Original: The sensitivity of eachpoint in S i is bounded from above by 2 d 1 .
Revised: The sensitivity of each point in S i isbounded from above by 2 d 1 .

Original: i as stated in Lines 4–6.
Revised: i as stated in Lines 4–6.

Original: The idea behind these boundslies in our proof of Theorem 2.6.
Revised: The idea behind these bounds lies in our proofof Theorem 2.8.

Original: The set S i is removed from Q , and this procedure is repeated withrespect to Q until the size of Q is small enough.
Revised: The set S i is removed from Q , and this procedure is repeated with respect to Quntil the size of Q is small enough.

Original: The obtained sensitivities are the ones needed forcomputing the pruning coresets.
Revised: The obtained sensitivities are the ones needed for computing thepruning coresets.

Original: Finally, we utilize the sensitivity sampling framework of [9] toobtain the desired coreset; see Lines 13–14.
Revised: Finally, we utilize the sensitivity sampling framework of [9] to obtain the desiredcoreset; see Lines 13–14.

Original: 2.4 Analysis
Revised: 2.4 Analysis

Original: In this section, we prove the correctness of our algorithms.
Revised: In this section, we prove the correctness of our algorithms.

Original: The following lemma shows that for eachpoint p that is inside some convex hull S , its ℓ 1 distance to any affine subspace is always boundedfrom above by ℓ 1 distance from the same affine subspace, of some other point q ∈ S .
Revised: The following lemma shows that for eachpoint p that is inside some convex hull S , its ℓ 1 distance to any affine subspace is always boundedfrom above by ℓ 1 distance from the same affine subspace, of some other point q ∈ S .

Original: Lemma 2.4.
Revised: Lemma 2.6.

Original: Let d, ℓ, m ≥ 1 be integers.
Revised: Let d, ℓ, m ≥ 1 be integers.

Original: Let p ∈ R d and A ⊆ R d be a set of m points
Revised: Let p ∈ R d and A ⊆ R d be a set of m points

Original: withp ∈
Revised: withp ∈

Original: Conv( A ) so that there exists α :
Revised: Conv( A ) so that there exists α :

Original: A → [0 , 1] such that (cid:80) q ∈
Revised: A → [0 , 1] such that (cid:80) q ∈

Original: A α ( q ) = 1 and (cid:80) q ∈
Revised: A α ( q ) = 1 and (cid:80) q ∈

Original: A α ( q ) · q = p .
Revised: A α ( q ) · q = p .

Original: Then for every Y ∈ R d × ℓ and v ∈ R ℓ , ∥ ( p − v ) Y ∥ 1 ≤ max q ∈
Revised: Then for every Y ∈ R d × ℓ and v ∈ R ℓ , ∥ ( p − v ) Y ∥ 1 ≤ max q ∈

Original: A ∥ ( q − v ) Y ∥ 1 .
Revised: A ∥ ( q − v ) Y ∥ 1 .

Original: The following states the provable guarantees of Algorithm 1.
Revised: The following states the provable guarantees of Algorithm 1.

Original: Lemma 2.5 ( ℓ ∞ -coreset for ℓ 1 -regression) .
Revised: Lemma 2.7 ( ℓ ∞ -coreset for ℓ 1 -regression) .

Original: Let P ⊆
Revised: Let P ⊆

Original: R d be a set of points, and r be the rank of P .
Revised: R d be a set of points, and r be the rank of P .

Original: Let j ∈ [ d − 1] and let S be the output of a call to ℓ ∞ -C ORESET ( P ) .
Revised: Let j ∈ [ d − 1] and let S be the output of a call to ℓ ∞ -C ORESET ( P ) .

Original: Then (i) | S | ∈ O (cid:0) r 2 (cid:1) , and(ii) for every X ∈ R d × j and v ∈ R d , max q ∈ P ∥ ( q − v )
Revised: Then (i) | S | ∈ O (cid:0) r 2 (cid:1) , and(ii) for every X ∈ R d × j and v ∈ R d , max q ∈ P ∥ ( q − v )

Original: X ∥ 1 max q ∈ S ∥ ( q − v ) X ∥ 1 ∈ (cid:2) 1 , 2 r 1 .
Revised: X ∥ 1 max q ∈ S ∥ ( q − v ) X ∥ 1 ∈ (cid:2) 1 , 2 r 1 .

Original: 5 (cid:3) .
Revised: 5 (cid:3) .

Original: The following theorem states our main result.
Revised: The following theorem states our main result.

Original: Theorem 2.6 (ReLU ε -coreset) .
Revised: Theorem 2.8 (ReLU ε -coreset) .

Original: Let P ⊆ R d , ε, δ ∈ (0 , 1) , r = rank ( P ) , µ ( P ) be as in Definition 2.3, and let m ∈ O (cid:16) µ ( P ) r 3 .
Revised: Let P ⊆ R d , ε, δ ∈ (0 , 1) , r = rank ( P ) , µ ( P ) be as in Definition 2.5, and let m ∈ O (cid:16) µ ( P ) r 3 .

Original: 5 log n ε(cid:0) d (log ( µ ( P ) r log n ))
Revised: 5 log n ε(cid:0) d (log ( µ ( P ) r log n ))

Original: + log (cid:0) 1 δ (cid:1)(cid:1)(cid:17) .
Revised: + log (cid:0) 1 δ (cid:1)(cid:1)(cid:17) .

Original: Let ( C, w ) be theoutput of a call to C ORESET ( P, m ) ; see Algorithm 2.
Revised: Let ( C, w ) be theoutput of a call to C ORESET ( P, m ) ; see Algorithm 2.

Original: Then, with probability at least 1 − δ , ( C, w ) isan ε -coreset for (cid:0) P, R d  , ReLU ( · ) (cid:1) ; see Definition 1.1.
Revised: Then, with probability at least 1 − δ , ( C, w ) isan ε -coreset for (cid:0) P, R d × R , ReLU ( · ) (cid:1) ; see Definition 2.2. Proof.

Revised: For space constraints let ϕ denote the ReLU function.

Revised: To obtain a coreset, we first need tobound the sensitivity of each p ∈ P .

Original: Time analysis.
Revised: Put p ∈ P and let ( x, b ) ∈ [Equation] ϕ ( p T x [Equation]

Revised: +

Revised: [Equation]

Revised: q ∈ P ϕ ( q T x ′ + b ′ )

Revised: wherethe supremum is over every ( x ′ , b ′ ) ∈

Revised: R d × R such that the denominator is not zero.

Revised: Observe that(cid:80)

Revised: [Equation]

Revised: where the last inequality follows from Definition 2.5.

Revised: Thus(cid:88)

Revised: [Equation]

Revised: Let β =

Revised: (1 + µ ( P )) .

Revised: We next observe that

Revised: [Equation]

Revised: where the first inequality holds by properties of ϕ , and the second is by (1).

Revised: Hence the sensitivity of pis bounded by

Revised: [Equation]

Revised: Let i be the iteration counter from Algorithm 2 as defined at Line 1, used at Line 5 and incrementedat Line 7.

Revised: The idea follows that of [103] where points being discarded from P at lower levels (smalleri ’s) have higher sensitivity.

Revised: This notion also resembles that of the “Onion sampling” of [46].

Revised: Now,assume that p ∈ S i at iteration i of the while loop (Line 3 of Algorithm 2).

Revised: In this case, observe thatby plugging P :=

Original: {+Q \ (cid:83) i − 
+}
Revised: {+Q \ (cid:83) i − 
+}

Revised: ˆ i =

Revised: 1 S ˆ i , j = 1 and v = − b x

Revised: ∥ x ∥ 2 into Lemma 2.7, we obtain a subset S i ⊆

Revised: Qsuch that

Revised: [Equation]

Revised: Thus for every q ∈ S i ,

Revised: [Equation]

Revised: where the first inequality is by (2), the second inequality follows from the observation that 201 (cid:8) arg

Revised: [Equation]

Revised: ˆ i =1 ⊆ P and the last inequality holds by (3).

Revised: Hence, we haveobtained a bound on the sensitivity of each point p ∈ P .

Revised: As for the total sensitivity, we observe that t = (cid:80)

Revised: p ∈

Revised: P s ( p ) ∈ O (cid:0) (1 + µ ( P ))

Revised: r 3 .

Original: Letting r be the rank of P , the time complexity of Algorithm 1 can be dissectedto two main parts: (i)
Revised: 5 log n (cid:1) . Theorem B.2 statesthat to obtain an ε -coreset with probability at least 1 − δ , the sample size m must be O (cid:16) [Equation]

Original: Computing the Löwner ellipsoid in O ( nr 2 log n ) time using the methodproposed in [95] and (ii) computing the Carathédory set in O ( nr + r 4 log n ) time via [69]. Since Vcan contain up to O ( r 2 ) points, the overall time for Algorithm 1 is O ( nr 2 log n + nr 3 + r 6 log n ) .
Revised: P

Original: As for Algorithm 2, it takes O ( n 2 r 2 log n + nr 4 log n ) .
Revised: [Equation] 3 . 5 log n ε

Original: Indeed, as explained in Section 2.5, adimensionality reduction algorithm may be applied to improve the run time (reducing the r 6 factor). Furthermore, the run time of our algorithm can be improved, using the merge-and-reduce tree fromthe literature of coresets to reduce the n 2 terms to n log n , i.e., the running time can be reduced to O ( n log 2 ( n ) r 2 + nr 4 ) . For a data-independent provable method, this running time is reasonable.
Revised: [Equation]

Original: Our advantages over previous results. Our coreset supports different activation functions withoutthe need to change the sensitivity that much. Specifically, it will only be multiplied by some scalar,unlike previous coresets where different losses impose drastically different sensitivities/leveragescores and algorithms. This is since our coreset unlike other coresets is in its essence a frameworkof coresets for different ℓ ρ losses, as it can be used as is for different ℓ ρ losses and yet still attainϵ -approximation.
Revised: Time analysis.

Original: In addition, when the rank of the input points is small, then our method outperformsprevious methods. If the input data is of full rank, previous methods obtain faster coresets construction. On the boundness of the regression complexity measure. First of all, there exists an examplewhere the complexity measure is unbounded, e.g., consider a set of points distributed evenly ona unit ball.
Revised: Letting r be the rank of P , Algorithm 1 takes O ( nr 3 + nr 2 log n + r 6 log n )time, while Algorithm 2 takes O ( n 2 r 2 log n + nr 4 log n ) .

Original: In this case, you can always find a point where a hyperplane separating it from therest of the points can be found such that the one half-space of this hyperplane contains only thispoint while the other half-space contains the rest of the points.

Original: This leads to an infinite complexitymeasure. Such an example is also mentioned in [79], when assessing the hardness of generatingmultiplicative-approximation coresets for ReLU functions. Theoretically, the complexity measure is influenced by how free can the bias term be (the last entry ofx); see Definition 2.3. This term is the only thing that can ensure that one point can be separated fromthe rest in the sense of finding a separating hyperplane, leading to an infinite complexity measure.
Revised: Indeed, as explained at Section 2.5, adimensionality reduction algorithm may be applied to improve the run time (reducing the r 6 factor).

Original: Bounding on this term, leads to bounded complexity measure from a theoretical point of view. In the context of model pruning, from the perspective of  thecomplexity measure, the model’s weightsare the input denoted by a matrix P ∈ R n × ( d +1) , while the query is now R d × { 1 } . Thus thecomplexity measure is now µ ( P ) : = sup x ∈ R d ×{ 1 } (cid:80)
Revised: Furthermore, the run time of our algorithm can be improved, using the merge-and-reduce tree fromthe literature of coresets to reduce the n 2 terms to n log n , i.e., the running time can be reducedto

Original: [Equation]

Original: [Equation]

Original: | [Equation] [Equation] x |
Revised: O ( n log 2 ( n ) r 2 + nr 4 ) .

Original: [Equation]

Original: q ∈ { p [Equation] P | pTx> 0 }| q T x | . With this in mind, weobserve that the complexity measure is now an instance of the complexity measure used in [73].
Revised: Note that for a data-independent provable method, this running time isreasonable.

Original: Thecomplexity measure now relies entirely on the structure of the model’s weights, where the goal is tofind the largest ratio between the sum of the absolute of the values inside the rectified neurons priorto applying the rectification, and the sum values of non-rectified neurons.

Original: To bound this measure, wecan use a variant of the algorithm described in the proof of Theorem 3 in [78].

Original: 2.5 Extensions
Revised: 2.5 Extensions

Original: Our suggested scheme can be extended to support many other variants of the pruning problem.
Revised: Our suggested scheme can be extended to support many other variants of the pruning problem.

Original: Various activation functions.
Revised: Various activation functions.

Original: Our result can be extended to a family of activation functions called“Nice hinge functions”; see Definition D.1.
Revised: Our result can be extended to a family of activation functions called“Nice hinge functions”; see Definition D.1.

Original: Let ( P, X, ϕ ) be a query space, where ϕ is a “Nice hingefunctions”.
Revised: Let ( P, X, ϕ ) be a query space, where ϕ is a “Nice hingefunctions”.

Original: To bound the sensitivity of a point p , we first bound the nominator of s ( p ) by provingthat ∀ x ∈ X : ϕ ( p T x ) < (cid:12)(cid:12) p T x (cid:12)(cid:12) .
Revised: To bound the sensitivity of a point p , we first bound the nominator of s ( p ) by provingthat ∀ x ∈ X : ϕ ( p T x ) < (cid:12)(cid:12) p T x (cid:12)(cid:12) .

Original: For bounding the denominator from below, recently [73] provedthat ∀ x ∈ X : (cid:80) p ∈ P ReLU (cid:0) p T x (cid:1) ⪅ (cid:80) p ∈ P ϕ ( p T x ) ; see full detail in Section D.3.
Revised: For bounding the denominator from below, recently [74] provedthat ∀ x ∈ X : (cid:80) p ∈ P ReLU (cid:0) p T x (cid:1) ⪅ (cid:80) p ∈ P ϕ ( p T x ) ; see full detail in Section D.3.

Original: Weighted Input.
Revised: Weighted Input.

Original: In the context of deep learning, the output of each neuron is multiplied with a scalarwhich brings the necessity of having the ability to deal with weighted set of points.
Revised: In the context of Deep learning, the output of each neuron is multiplied with a scalarwhich brings the necessity of having the ability to deal with weighted set of points.

Original: Algorithm 2 canbe extended easily to the case as generously detailed in Section D.1 in the Appendix.
Revised: Algorithm 2 canbe extended easily to the case as generously detailed at Section D.1 at the Appendix.

Original: Dimensionality reduction.
Revised: Dimensionality reduction.

Original: All coreset-based pruning methods rely heavily on the dimensionality ofthe model’s layers, as well as our method.
Revised: All coreset-based pruning methods rely heavily on the dimensionality ofthe model’s layers, as well as our method.

Original: To ensure sufficient pruning ratio, we apply either PCA,
Revised: To ensure sufficient pruning ratio, we apply either PCA,

Original: TSNE, MDS, or the JL transform on the weights of each layer prior to generating its coreset.
Revised: TSNE, MDS, or the JL transform on the weights of each layer prior to generating its coreset.

Original: From weight to neuron pruning.
Revised: From weight to neuron pruning.

Original: Most coreset-based pruning methods, e.g., [5, 79], first provide ascheme for (provable) weight pruning, which is then used as a stepping stone towards pruning neuronsas follows.
Revised: Most coreset-based pruning methods, e.g., [6, 79], first provide ascheme for (provable) weight pruning, which is then used as a stepping stone towards pruning neuronsas follows.

Original: [5] first suggested coreset-based neuron pruning via the use of a generated controled set ofqueries to evaluate the importance of weights.
Revised: [6] first suggested coreset-based neuron pruning via the use of a generated controled set ofqueries to evaluate the importance of weights.

Original: Any neuron that has a maximal activation value loweror equal to zero, will be pruned from the network as its impact on the rest of the neurons is minimal.
Revised: Any neuron that has a maximal activation value loweror equal to zero, will be pruned from the network as its impact on the rest of the neurons is minimal.

Original: On the other hand [79] altered the definition of sensitivity such that it takes into account the sensitivityof a neuron in a layer ℓ with respect to all the neurons in the layer ℓ + 1 , which basically meansthat the sensitivity of each neuron is taken be the maximal sensitivity over every weight function(neuron in the next layer) defined by the layer.
Revised: On the other hand [79] altered the definition of sensitivity such that it takes into account the sensitivityof a neuron in a layer ℓ with respect to all the neurons in the layer ℓ + 1 , which basically meansthat the sensitivity of each neuron is taken be the maximal sensitivity over every weight function(neuron in the next layer) defined by the layer.

Original: Hence, we follow the same logic for such method; see
Revised: Hence, we follow the same logic for such method; see

Original: Section D.2 in the supplementary material.
Revised: Section D.2 at the supplementary material.

Original: From neuron to filter pruning.
Revised: From neuron to filter pruning.

Original: Convolutions can be expressed as matrix multiplications, whichenables our method to prune filters from any model as done by [60, 79, 61].
Revised: Convolutions can be expressed as matrix multiplications, whichenables our method to prune filters from any model as done by [61, 79, 62].

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we study various widely used network architectures and benchmark data-sets.
Revised: In this section, we study various widely used network architectures and benchmark data-sets.

Original: Following [79], to test the robustness of our methods on each of the neuron and filter pruning tasksindependently, two sets of experiments are conducted.
Revised: Following [79], to test the robustness of our methods on each of the neuron and filter pruning tasksindependently, two sets of experiments are conducted.

Original: The first focuses on pruning neurons (Section 3.1) whereas the second focuses on pruning filters (Section 3.2), both via our coreset method.
Revised: The first focuses on pruning neurons (Section 3.1) whereas the second focuses on pruning filters (Section 3.2), both via our coreset method.

Original: The setting.
Revised: The setting.

Original: In all experiments we report the Pruning ratio – the percentage of the parameters thatwere removed from the original mode. Here, PR stands for pruning ratio, FR stands for floating-pointreduction ratio and Err – the percentage of misclassified test instances of our method compared tocoreset-based pruning methods and more.
Revised: In all experiments we report the Compression ratio – the percentage of the parametersthat were removed from the original model, and Error – the percentage of misclassified test instancesof our method compared to coreset-based pruning methods and more.

Original: Baseline Err is the error of the original uncompressednetwork, while Pruned Err is the classification error of the compressed model.

Original: In our experimentwe compress and fine-tune the network once, no iterative pruning was applied, thus, the comparedmethods also satisfy this setting.
Revised: In our experiment we compressand fine-tune the network once, no iterative pruning was applied, thus, the compared methods alsosatisfy this setting.

Original: Each experiment was conducted 5 times, in the tables, we reportfor our method the best error achieved

Original: and we highlight in parentheses next to it the average errorand standard deviation across the 5 trails. In all of our experiments, the models are fine-tuned tillconvergence (after pruning). Implementation details are given in Section E in the Appendix. Software/Hardware.
Revised: Throughout our experiments PR stands for pruning ratio, and FR stands forfloating-point reduction ratio.

Original: Our algorithms were implemented in Python 3.6 [103] using Numpy [80], and

Original: Pytorch [81].

Original: Tests were performed on NVIDIA DGX A100 servers with 8 NVIDIA A100 GPUseach, fast InfiniBand interconnect and supporting infrastructure.

Original: Baselines.
Revised: Baselines.

Original: Our results are compared to (i) PFP [60], (ii) FT [58], (iii) SoftNet [34], (iv) ThiNet [68],and (v) PvC [79], (vi) Soft Pruning [34], (vii) CCP [82], (viii)
Revised: Our results are compared to (i) PFP [61], (ii) FT [59], (iii) SoftNet [34], (iv) ThiNet [69],and (v) PvC [79], (vi) Soft Pruning [34], (vii) CCP [82], (viii)

Original: FPGM [36], (ix) ThiNet-70, (x)
Revised: FPGM [36], (ix) ThiNet-70, (x)

Original: ThiNet-50 [68], (xi) Pruning via Coresets (PvC) [79], (xii)
Revised: ThiNet-50 [69], (xi) Pruning via Coresets (PvC) [79], (xii)

Original: Pruning from Scratch (PfS) [106], and(xiii)
Revised: Pruning from Scratch (PfS) [105], and(xiii)

Original: Rethinking the value of network pruning (Rethink) [65].
Revised: Rethinking the value of network pruning (Rethink) [66].

Revised: Software/Hardware.

Revised: Our algorithms were implemented in Python 3.6 [102] using “Numpy” [80],and Pytorch [81].

Revised: Tests were performed on NVIDIA DGX A100 servers with 8 NVIDIA A100 GPUseach, fast InfiniBand interconnect and supporting infrastructure.

Original: 3.1 Neuron Pruning
Revised: 3.1 Neuron Pruning

Original: We test our method on  VGG16 [89] using CIFAR10 [49], and  LeNet300-100 using MNIST [55].
Revised: We test our method on (i) VGG16 [89] using CIFAR10 [50], and (ii) LeNet300-100 using MNIST [56].

Revised: After pruning, we fine-tune the pruned model till convergence.

Original: Discussion.
Revised: Discussion.

Original: Table 1 present the results of LeNet 300 - 100 and VGG 16 .
Revised: Table 1 present the results of LeNet 300 - 100 and VGG 16 .

Original: Observe that in both architectures, our method outperformed the competing methods under the same compression scenarios.
Revised: Observe that in both architectures, our method outperformed the competing methods under the same compression scenarios.

Original: Forexample, we pruned roughly 90% of the parameters of the LeNet-300-100 model while improvingthe accuracy of the original model.
Revised: Insummary, we pruned roughly 90% of the parameters of the LeNet-300-100 model while improving [Table] PFP 1 .2 .84 . FT 1 .1 .81 .

Original: We witness a similar phenomena on the VGG16 model, where wepruned roughly 90% of the parameters of the dense layers resulting in accuracy improvement.
Revised: SoftNet 1 .2 .81 . ThiNet 1 .12 .75 . PvC 2 .2 . Our method 2 .1 .

Original: Thisconfirms the insights in [5] that coresets help in improving the generalization properties of DNNs.
Revised: VGGPvC

Original: [Table]
Revised: [Table]

Revised: Our method 5 .5 .

Revised:  the accuracy of the original model.

Revised: We witness a similar phenomena on the VGG16 model, where wepruned roughly 90% of the parameters of the dense layers resulting in accuracy improvement.

Revised: Thisconfirms the insights in [6] that coresets help in improving the generalization properties of DNNs.

Original: 3.2 Filter pruning
Revised: 3.2 Filter pruning

Original: We compressed the convolutional layers of (i) ResNet 50 [33] on ILSVRC- 2012 [18], (ii)
Revised: We compressed the convolutional layers of (i) ResNet 50 [33] on ILSVRC 2012 [18], (ii)

Original: ResNet 56 [33], (iii) VGG 19 [89] on CIFAR 10 and (iv) VGG 16 [89] on CIFAR 10 .
Revised: ResNet 56 [33] and (iii) VGG 19 [89] on CIFAR 10 , followed by fine-tuning the pruned modelstill convergence.

Revised: Here, we operated under the same compression settings of [79]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: and reported theoverall compression ratio and error. Filter pruning of DNNs on CIFAR 10 . We used PyTorch implementation of VGG 19 model for

Original: ThiNet27 .28 .51 .55 . PvC 23 .25 . N/A
Revised: CIFAR 10 which contains about 20 M parameters. Here, we removed around 70% of the filters ineach layer, resulting in a total of 88% compression ratio.

Original: Our method 23 .24 . 87 ( 25 . 07 ± 0 . 2 ) 61 . Filter pruning of DNNs on CIFAR 10 and ImageNet (ILSVRC- 2012 ). For Cifar 10 , we used PyTorch implementations of VGG 19 and VGG 16 .
Revised: As for ResNet 56 [40], the model has beencompressed to obtain roughly 55% compression rate.

Original: We compressed both models using our approachby different compression rates as shown in Table 2 with a comparison  to other methods.
Revised: Table 2 shows a comparison between ourmethod to others. [Table] Our method [Table]

Original: For
Revised: Compressing ResNet 50 on ILSVRC 2012 .

Original: ImageNet, we compressed the baseline model of ResNet 50 [33] roughly by 62% in terms of numberof parameters.
Revised: Here, we compressed the baseline model of ResNet 50 [33] roughly by 62% in terms of number of parameters.

Original: Table 3 provides comparison between our method and other baselines. 
Revised: Table 3 provides comparisonbetween our method and other baselines. [Table]

Original: Discussion.
Revised: Discussion.

Original: As can be seen in Tables 1, 2, and 3, our method either outperforms the competingmethods or achieves comparable results.
Revised: As can be seen at Tables 1, [Table] and 3, our method either outperforms the competingmethods or achieves comparable results.

Original: As for the coreset methods,  our algorithm achieves betterresult than all of them in this setting, e.g., we compressed 62% of ResNet50 trained on ImageNetwhile incurring 1 . 09% drop in accuracy, improving the recent coreset result of PvC [79] for the samecompression ratio, while PFP [60] compressed 44 .
Revised: As for the coreset methods, it can be seen that our algorithmachieves better result than all of them in this setting. For example we compressed 62% of ResNet50trained in ImageNet while resulting in only 1 . 09% drop in accuracy, improving the recent coresetresult of PvC [79] for the same compression ratio, while PFP [61] compressed 44 .

Original: 04% to achieve comparable results.
Revised: 04% to achievecomparable results.

Original: Related work
Revised: 3.3 Implementation details

Revised: First observe that the map function at Line 3 of Algorithm 1 is hard to implement if all we haveis Y and P ′ (see Lines 1–2) due to the fact that when the rank of P is not d , then Y becomes asingular matrix. A way around such problem (practically speaking yet also theoretically sound) isto reformulate P and P ′ as matrices, where our ℓ ∞ -coreset will now be regarded as a set of indicesof the rows selected as the desired coreset. Note that while we relied on an “accurate” measureof the rank of points in Algorithm 1, in our experiments, we used the rank function from Numpy ,and still produced favorable results.

Revised: Furthermore, our algorithms work also when the input has fullrank.

Revised: In addition, we can still obtain a ε -coreset, when using approximated algorithms for the rankcomputation problem, where the error associated with our coreset may increase.

Revised: In this case, we canincrease our coreset size to reduce our approximation error to be the original desired error.

Revised:  Related work

Original: DNNs can be compressed before training [94, 105, 57], during training [113, 110, 51], or aftertraining [90].
Revised: DNNs can be compressed before training [94, 104, 58], during training [112, 109, 52], or aftertraining [90].

Original: Furthermore, such procedures may also be repeated iteratively [85].
Revised: Furthermore, such procedures may also be repeated iteratively [85].

Original: As previouslynoted pruning can be categorized into structured and unstructured pruning.
Revised: As previouslynoted pruning can be categorized into structured and unstructured pruning.

Original: Unstructured pruning.
Revised: Unstructured pruning.

Original: Weight pruning [56] techniques aim to reduce the number of weightsin a layer while approximately preserving its output.
Revised: Weight pruning [57] techniques aim to reduce the number of weightsin a layer while approximately preserving its output.

Original: Approaches of this type include the worksof [54, 21, 39, 1, 62], where the desired sparsity is embedded as a constraint or via a regularizer intothe training pipeline, and those of [31, 85, 30], where weights with absolute values below a thresholdare removed.
Revised: Approaches of this type include the worksof [55, 21, 39, 1, 63], where the desired sparsity is embedded as a constraint or via a regularizer intothe training pipeline, and those of [31, 85, 30], where weights with absolute values below a thresholdare removed.

Original: The approaches of [5, 6] use a mini-batch of data points to approximate the influence ofeach parameter on the loss function.
Revised: The approaches of [6, 7] use a mini-batch of data points to approximate the influence ofeach parameter on the loss function.

Original: Other data-informed techniques include [28, 63, 77, 76, 111].
Revised: Other data-informed techniques include [28, 64, 77, 76, 110].

Original: Athorough overview of recent pruning approaches is given by [27, 8].
Revised: Athorough overview of recent pruning approaches is given by [27, 8].

Original: However, unlike our approach,weight-based pruning approaches generate sparse models instead of smaller ones thus requiringspecialized hardware and sparse linear algebra libraries in order to speed up inference.
Revised: However, unlike our approach,weight-based pruning approaches generate sparse models instead of smaller ones thus requiringspecialized hardware and sparse linear algebra libraries in order to speed up inference.

Original: Structured pruning.
Revised: Structured pruning.

Original: Pruning entire neurons and filters directly shrinks the network leading to smallerstorage requirements and improved inference-time performance on any hardware [59, 67].
Revised: Pruning entire neurons and filters directly shrinks the network leading to smallerstorage requirements and improved inference-time performance on any hardware [60, 68].

Original: Lately,these approaches were investigated in many papers [64, 59, 11, 36, 22, 46, 109, 108].
Revised: Lately,these approaches were investigated in many papers [65, 60, 11, 36, 22, 47, 108, 107].

Original: Usually, filtersare pruned by assigning an importance score to each neuron/filter, either solely weight-based [37, 35]or data-informed [111, 60], and removing those with a score below a threshold.
Revised: Usually, filtersare pruned by assigning an importance score to each neuron/filter, either solely weight-based [37, 35]or data-informed [110, 61], and removing those with a score below a threshold.

Original: The procedure canbe embedded into an iterative pruning scheme [85] that requires potentially expensive retrain cycles.
Revised: The procedure canbe embedded into an iterative pruning scheme [85] that requires potentially expensive retrain cycles.

Original: Tensor decomposition.
Revised: Tensor decomposition.

Original: Some of the work in DNN compression entails decomposing the layerinto multiple smaller ones, e.g., via low-rank tensor decomposition [19, 41, 72, 48, 93, 40, 2, 101,112 , 53, 61].
Revised: Some of the work in DNN compression entails decomposing the layerinto multiple smaller ones, e.g., via low-rank tensor decomposition [19, 42, 73, 49, 93, 41, 2, 100,111, 54, 62].

Original: Other approaches to tensor decomposition include weight sharing, random projections,and feature hashing [107, 3, 88, 12, 13, 102].
Revised: Other approaches to tensor decomposition include weight sharing, random projections,and feature hashing [106, 3, 88, 12, 13, 101].

Original: However, such techniques usually require expensiveapproximation algorithms or use heuristics since tensor decomposition is generally NP-hard.
Revised: However, such techniques usually require expensiveapproximation algorithms or use heuristics since tensor decomposition is generally NP-hard.

Original: Coresets.
Revised: Coresets.

Original: In the recent years, coresets got increasing attention, and where leveraged to compress theinput datasets of many machine learning algorithms, improving there performance, e.g., regression [38,78, 47, 99], decision trees [44], matrix approximation [26, 69, 25, 86, 71], clustering [24, 29, 66, 4,45, 87], ℓ z -regression [16, 17, 91], SVM [32,98, 96, 97, 100], and deep learning models [5, 60].
Revised: In the recent years, coresets got increasing attention, and where leveraged to compress theinput datasets of many machine learning algorithms, improving there performance, e.g., regression [38,78, 48, 98], Decision Trees [45], matrix approximation [26, 70, 25, 86, 72], clustering [24, 29, 67, 4,46, 87], ℓ z -regression [16, 17, 91], SVM [32, 97, 95, 96, 99], and deep learning models [5, 61].

Original: Forextensive surveys on coresets, we refer the reader to [23, 83, 43, 70].
Revised: Forextensive surveys on coresets, we refer the reader to [23, 83, 44, 71].

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: In this paper, we provided a coreset-based pruning technique that hinges upon a combination oftools from convex geometry, while achieving SOTA results with respect to coreset-based structuredpruning approaches ona variety of networks.
Revised: In this paper, we provided a coreset-based pruning technique that hinges upon an elegant combinationof tools from convex geometry, while achieving SOTA results with respect to coreset-based structuredpruning approaches on variety of networks.

Original: Our main improvement is that our coreset is (training)data-independent and assumes a single assumption on the models weights.
Revised: Our main improvement is that our coreset is (training)data-independent and assumes a single assumption on the models weights.

Original: Future work includes (i) suggesting a coreset based budget allocation framework, to determine the(optimal) per layer prune ratio while achieving an overall desired compression rate,(ii) extending ourcoreset technique to other layers such as attention layers, and (iii) bridging the gap between coresetbased pruning approaches and tensor-decomposition methods, as both techniques are theoreticallysupported by bounding the approximation error given specific compression rate, we can leverage thesebounds to formulate the compression problem as an optimization problem which iterates between thetwo approaches to search for the local minimum.
Revised: Future work includes (i) suggesting a coreset based budget allocation framework, to determine the(optimal) per layer prune ratio while achieving an overall desired compression rate, and (ii) bridgingthe gap between coreset based pruning approaches and tensor-decomposition methods, as both techniques are theoretically supported by bounding the approximation error given specific compressionrate, we can leverage these bounds to formulate the compression problem as an optimization problemwhich iterates between the two approaches to search for the local minimum.