Original: Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions
Revised: Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Pruning is one of the predominant approaches for compressing deep neural networks (DNNs).
Revised: Pruning is one of the predominant approaches for compressing deep neural networks (DNNs).

Original: Lately, coresets (provable data summarizations) were leveragedfor pruning DNNs, adding the advantage of theoretical guarantees on the trade-offbetween the compression rate and the approximation error.
Revised: Lately, coresets (provable data summarizations) were leveragedfor pruning DNNs, adding the advantage of theoretical guarantees on the trade-offbetween the compression rate and the approximation error.

Original: However, coresets inthis domain were either data-dependent or generated under restrictive assumptionson both the model’s weights and inputs.
Revised: However, coresets inthis domain were either data-dependent or generated under restrictive assumptionson both the model’s weights and inputs.

Original: In real-world scenarios, such assumptionsare rarely satisfied, limiting the applicability of coresets.
Revised: In real-world scenarios, such assumptionsare rarely satisfied, limiting the applicability of coresets.

Original: To this end, we suggest anovel and robust framework for computing such coresets under mild assumptionson the model’s weights and without any assumption on the training data.
Revised: To this end, we suggest anovel and robust framework for computing such coresets under mild assumptionson the model’s weights and without any assumption on the training data.

Original: The ideais to compute the importance of each neuron in each layer with respect to the outputof the following layer.
Revised: The ideais to compute the importance of each neuron in each layer with respect to the outputof the following layer.

Original: This is achieved by a combination of Löwner ellipsoid and
Revised: This is achieved by a combination of Löwner ellipsoid and

Original: Caratheodory theorem.
Revised: Caratheodory theorem.

Original: Our method is simultaneously data-independent, applicableto various networks and datasets (due to the simplified assumptions), and theoretically supported.
Revised: Our method is simultaneously data-independent, applicableto various networks and datasets (due to the simplified assumptions), and theoretically supported.

Original: Experimental results show that our method outperforms existingcoreset based neural pruning approaches across a wide range of networks anddatasets.
Revised: Experimental results show that our method outperforms existingcoreset based neural pruning approaches across a wide range of networks anddatasets.

Original: For example, our method achieved a 62% compression rate on ResNet50on ImageNet with 1 . 09% drop in accuracy.
Revised: For example, our method achieved a 62% compression rate on ResNet50on ImageNet with 1 . 09% drop in accuracy.

Original:  Introduction and Backround
Revised:  Introduction and Backround

Original: Deep neural networks (DNNs) achieved state-of-the-art (SOTA) performance on a large variety oftasks, e g., in computer vision [33, 51] and natural language processing (NLP; [85, 20]).
Revised: Deep neural networks (DNNs) achieved state-of-the-art (SOTA) performance on a large variety oftasks, e g., in computer vision [33, 50] and natural language processing (NLP; [84, 20]).

Original: However,
Revised: However,

Original: DNNs usually contain millions or even billions of parameters in order to achieve SOTA performancesresulting in large storage requirements and long inference time.
Revised: DNNs usually contain millions or even billions of parameters in order to achieve SOTA performancesresulting in large storage requirements and long inference time.

Original: This is obstructive when, e.g., dealingwith limited hardware or real-time systems such as autonomous cars and text/speech translation.
Revised: This is obstructive when, e.g., dealingwith limited hardware or real-time systems such as autonomous cars and text/speech translation.

Original: Tothis end, a large body of research is dedicated to reducing the size and inference costs of DNNs.
Revised: Tothis end, a large body of research is dedicated to reducing the size and inference costs of DNNs.

Original: Pruning.
Revised: Pruning.

Original: A dominant approach widely used for reducing the size of DNNs is to utilize a pruningalgorithm to remove redundant parameters from the original, over-parameterized network.
Revised: A dominant approach widely used for reducing the size of DNNs is to utilize a pruningalgorithm to remove redundant parameters from the original, over-parameterized network.

Original: In general,pruning can be categorized into two main types: (i) Unstructured pruning [31, 6] reduces thenumber of non-zero parameters by inducing sparsity into weight parameters, which can achievehigh compression rates but requires specialized software and/or hardware in order to achieve fasterinference times.
Revised: In general,pruning can be categorized into two main types: (i) Unstructured pruning [31, 6] reduces thenumber of non-zero parameters by inducing sparsity into weight parameters, which can achievehigh compression rates but requires specialized software and/or hardware in order to achieve fasterinference times.

Original: (ii) Structured pruning [35, 61, 75] modifies the structure of the underlying weighttensors, by removing filters/neurons from each layer, usually resulting in smaller compression rateswhile directly achieving faster inference times with no specialized software; see section 

Revised: (ii) Structured pruning [35, 60, 74] modifies the structure of the underlying weighttensors, by removing filters/neurons from each layer, usually resulting in smaller compression rateswhile directly achieving faster inference times with no specialized software; see section

Original: 1.1 Coresets for Pruning
Revised: 1.1 Coresets for Pruning

Original: Notably, many recent papers focused on various types of filter pruning [86, 77] potentially dueto the empirical observation that existing filter pruning approaches consistently yield impressiveresults.
Revised: Notably, many recent papers focused on various types of filter pruning [85, 76] potentially dueto the empirical observation that existing filter pruning approaches consistently yield impressiveresults.

Original: However, most pruning methods are based on heuristics, lacking theoretical guarantees onthe trade-off between the compression rate and the approximation error.
Revised: However, most pruning methods are based on heuristics, lacking theoretical guarantees onthe trade-off between the compression rate and the approximation error.

Original: This was the motive forintroducing coresets [80, 61] to the world of pruning.
Revised: This was the motive forintroducing coresets [79, 60] to the world of pruning.

Original: Coresets.
Revised: Coresets.

Original: In machine learning, we are (usually) given an input set P ⊆
Revised: In machine learning, we are (usually) given an input set P ⊆

Original: R
Revised: R

Original: d of n points, itscorresponding weights function w : P → R , a feasible set of queries X , and a loss functionϕ : P × X → [0 , ∞ ) .
Revised: d of n points, itscorresponding weights function w : P → R , a feasible set of queries X , and a loss functionϕ : P × X → [0 , ∞ ) .

Original: The tuple ( P, w, X, ϕ ) is called query space , and it defines the optimizationproblem at hand.
Revised: The tuple ( P, w, X, ϕ ) is called query space , and it defines the optimizationproblem at hand.

Original: For a given problem that is defined by its query space ( P, w, X, ϕ ) , and an errorparameter ε ∈ (0 , 1) , an ε -coreset is is a small weighted subset of the input points that approximatesthe loss of the input set P for every feasible query x , up to a provable bound of 1 + ε .
Revised: For a given problem that is defined by its query space ( P, w, X, ϕ ) , and an errorparameter ε ∈ (0 , 1) , an ε -coreset is is a small weighted subset of the input points that approximatesthe loss of the input set P for every feasible query x , up to a provable bound of 1 + ε .

Original: Since coresets approximate the cost of every query, traditional (possibly inefficient) algorithms/solverscan be applied on coresets to obtain an approximation of the optimal solution on the full data, usingless time and memory; see Section B.2 in the appendix for more details.
Revised: Since coresets approximate the cost of every query, traditional (possibly inefficient) algorithms/solverscan be applied on coresets to obtain an approximation of the optimal solution on the full data, usingless time and memory; see Section B.2 in the appendix for more details.

Original: Pruning via coresets.
Revised: Pruning via coresets.

Original: Recently, some inspiring innovative frameworks [80, 61, 5, 7] leveraged theidea of coresets for pruning DNNs.
Revised: Recently, some inspiring innovative frameworks [79, 60, 5, 7] leveraged theidea of coresets for pruning DNNs.

Original: Any layer ℓ can be represented as a set P = { p 1 , · · · , p n } ofn > 1 points in R d , where d is the number of input neurons, and n in the number of output neurons,i.e., each point p ∈ P , represents a specific neuron using its d weights (parameters).
Revised: Any layer ℓ can be represented as a set P = { p 1 , · · · , p n } ofn > 1 points in R d , where d is the number of input neurons, and n in the number of output neurons,i.e., each point p ∈ P , represents a specific neuron using its d weights (parameters).

Original: When the layerreceives an input vector x ∈ R d , it outputs the vector ( ϕ ( p T 1 x ) , · · · , ϕ ( p Tn x )) , where ϕ : R → R isan activation function which defines a non-linear mapping.
Revised: When the layerreceives an input vector x ∈ R d , it outputs the vector ( ϕ ( p T 1 x ) , · · · , ϕ ( p Tn x )) , where ϕ : R → R isan activation function which defines a non-linear mapping.

Original: Focusing on a single neuron η in thelayer that follows ℓ , defined by its corresponding vector of weights w = ( w 1 , · · · , w n ) , we set inthe context of coresets, w ( p i ) :
Revised: Focusing on a single neuron η in thelayer that follows ℓ , defined by its corresponding vector of weights w = ( w 1 , · · · , w n ) , we set inthe context of coresets, w ( p i ) :

Original: = w i for every i ∈ { 1 , · · · , n } - this is just a mapping from p i tow i to simplify the writing and reading.
Revised: = w i for every i ∈ { 1 , · · · , n } - this is just a mapping from p i tow i to simplify the writing and reading.

Original: Note that the output of this neuron is (cid:80) p ∈
Revised: Note that the output of this neuron is (cid:80) p ∈

Original: P w ( p ) ϕ
Revised: P w ( p ) ϕ

Original: (cid:0) p T x (cid:1) .
Revised: (cid:0) p T x (cid:1) .

Original: Assuming that we are given an ε -coreset
Revised: Assuming that we are given an ε -coreset

Original: ( C, u ) for the query space (cid:0) P, w, R d , ϕ (cid:1) where C ⊆ P , andu : C
Revised: ( C, u ) for the query space (cid:0) P, w, R d , ϕ (cid:1) where C ⊆ P , andu : C

Original: → R , we have that ( C, u ) approximates the output of this specific neuron η for every queryusing less parameters; see Figure 1.
Revised: → R , we have that ( C, u ) approximates the output of this specific neuron η for every queryusing less parameters; see Figure 1.

Original: To formalize the stated above, we now define coresets in thecontext of activation functions.
Revised: To formalize the stated above, we now define coresets in thecontext of activation functions.

Original: Definition 1.1 (Coreset for activation functions) .
Revised: Definition 1.1 (Coreset for activation functions) .

Original: Let ε ∈
Revised: Let ε ∈

Original: (0 , 1) , and let (cid:0) P, w, R d , ϕ (cid:1) be a queryspace.
Revised: (0 , 1) , and let (cid:0) P, w, R d , ϕ (cid:1) be a queryspace.

Original: Then the pair ( C, u ) , is an ε -coreset for (cid:0) P, w, R d , ϕ (cid:1) if (
Revised: Then the pair ( C, u ) , is an ε -coreset for (cid:0) P, w, R d , ϕ (cid:1) if (

Original: i) C ⊆ P , (ii) u : C → [0 , ∞ ) , and(iii) for every x ∈ X , (cid:12)(cid:12)(cid:12)(cid:12)
Revised: i) C ⊆ P , (ii) u : C → [0 , ∞ ) , and(iii) for every x ∈ X , (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: − (cid:80) q
Revised: − (cid:80) q

Original: [Equation]
Revised: [Equation]

Original: q ) ϕ
Revised: q ) ϕ

Original: ( q T
Revised: ( q T

Original: [Equation]
Revised: [Equation]

Original: (cid:80) p ∈ P w
Revised: (cid:80) p ∈ P w

Original: [Equation]
Revised: [Equation]

Original: ≤ ε.
Revised: ≤ ε.

Original: Since C is a subset of P , we can remove (assign zero to) all weights from w that corresponds topoints not chosen to be in C from P , and replace the weights of the chosen points in C with the newweights vector u ; see Figure 1 in [80] for a visual illustration.
Revised: Since C is a subset of P , we can remove (assign zero to) all weights from w that corresponds topoints not chosen to be in C from P , and replace the weights of the chosen points in C with the newweights vector u ; see Figure 1 in [79] for a visual illustration.

Original: To prune neurons, we refer the readerto Section 2.5 as it is a simple extension.
Revised: To prune neurons, we refer the readerto Section 2.5 as it is a simple extension.

Original: Prior work showed that such approaches successfully resultin high compression rates across a wide range of networks and datasets, and even achieves SOTAperformance on a verity of them.
Revised: Prior work showed that such approaches successfully resultin high compression rates across a wide range of networks and datasets, and even achieves SOTAperformance on a verity of them.

Original: The main (strong) advantage of the coreset approach over others was the provided provable theoreticalguarantees on the tradeoff between the compression rate and the approximation error, which supports worse case scenarios.
Revised: The main (strong) advantage of the coreset approach over others was the provided provable theoreticalguarantees on the tradeoff between the compression rate and the approximation error, which supports worse case scenarios.

Original: In addition, coresets play an important role in improving the generalizationproperties of the trained networks [5, 76].
Revised: In addition, coresets play an important role in improving the generalizationproperties of the trained networks [5, 75].

Original: Sensitivity sampling for constructing pruning coresets.
Revised: Sensitivity sampling for constructing pruning coresets.

Original: To compute such coresets, both [61, 80]utilised the known sensitivity sampling framework [9, 53].
Revised: To compute such coresets, both [60, 79]utilised the known sensitivity sampling framework [9, 52].

Original: In short, the sensitivity of a point p ∈ Pin some query space ( P,  X, ϕ ) corresponds to the importance of this point with respect to the queryspace at hand, and it is defined as s ( p ) = 
Revised: In short the sensitivity of a point p ∈ P insome query space ( P, w, X, ϕ ) corresponds to the importance of this point with respect to the queryspace at hand, and it is defined as s ( p ) = sup x ∈ X w [Equation] p ) ϕ

Revised: [Equation]

Original: sup x ∈ X ϕ ( p,x ) (cid:80) q ∈ P ϕ ( q,x ) - where the denominator is not equalto zero.
Revised: ,x ) [Equation] q [Equation]

Revised: w

Revised: [Equation]

Revised: q ) ϕ

Revised: ( q,x ) - where the denominator is notequal to zero.

Original: Once we bound these sensitivities, we can sample points (neurons) from P according tosensitivity bounds, and re-weight the sampled points to obtain a coreset.
Revised: Once we bound these sensitivities, we can sample points (neurons) from P accordingto sensitivity bounds, and re-weight the sampled points to obtain a coreset.

Original: The size of the sample isproportional to the sum of these bounds.
Revised: The size of the sample isproportional to the sum of these bounds.

Original: See Section B.1 and Theorem B.2 for more details in the
Revised: See Section B.1 and Theorem B.2 for more details in the

Original: Appendix.
Revised: Appendix.

Original: 1.2 Our contribution
Revised: 1.2 Our contribution

Original: Prior coreset methods for pruning DNNs either (i) imposed restrictive assumptions both on themodel’s weights and inputs [80], i.e., the input set P representing the neurons, and the query set Xwhich represents the inputs of the layer, are enclosed in a ball in R d of radius r 1 and r 2 , respectively,or (ii) the methods are data-dependent, i.e., use a mini-batch of the input set to measure the influenceof each parameter on the loss function [5, 61].
Revised: Prior coreset methods for pruning DNNs either (i) imposed restrictive assumptions both on themodel’s weights and inputs [79], i.e., the input set P representing the neurons, and the query set Xwhich represents the inputs of the layer, are enclosed in a ball in R d of radius r 1 and r 2 , respectively,or (ii) the methods are data-dependent, i.e., use a mini-batch of the input set to measure the influenceof each parameter on the loss function [5, 60].

Original: To this end, in this work, we take coresets a step further into the realm of pruning by introducinga unified framework with provable guarantees for pruning DNNs (weights and neurons/filters)while minimally affecting the generalization error.
Revised: To this end, in this work, we take coresets a step further into the realm of pruning by introducinga unified framework with provable guarantees for pruning DNNs (weights and neurons/filters)while minimally affecting the generalization error.

Original: Our main improvement is that our framework issimultaneously (i) data-independent , (ii) requires a single assumption on the model’s weights,and (iii) provably guarantees a multiplicative factor approximation , which is favourable uponadditive approximations; see Theorem 2.6.
Revised: Our main improvement is that our framework issimultaneously (i) data-independent , (ii) requires a single assumption on the model’s weights,and (iii) provably guarantees a multiplicative factor approximation , which is favourable uponadditive approximations; see Theorem 2.6.

Original: The approach is based on the widely used theory ofcoresets allowing us to suggest a provable guarantee on the tradeoff between the approximation errorand compression rate for each layer.
Revised: The approach is based on the widely used theory ofcoresets allowing us to suggest a provable guarantee on the tradeoff between the approximation errorand compression rate for each layer.

Original: We conducted experimental results which established new SOTA benchmarks for structured pruningvia coresets across a wide range of networks and datasets.
Revised: We conducted experimental results which established new SOTA benchmarks for structured pruningvia coresets across a wide range of networks and datasets.

Original: Finally, we share all of our resultedmodels [14].
Revised: We share all of our resulted models [14].

Original:  Method
Revised:  Method

Original: In general, the coreset (for pruning) technique hinges upon the insight that any linear layer such asconvolutions, can be casted as a matrix multiplication [80].
Revised: In general, the coreset (for pruning) technique hinges upon the insight that any linear layer such asconvolutions, can be casted as a matrix multiplication [79].

Original: Hence, we focus in what follows on fullyconnected (FC) layers, while the details holds for any linear layer.
Revised: Hence, we focus in what follows on fullyconnected (FC) layers, while the details holds for any linear layer.

Original: Furthermore, we note that forsimplicity, we assume in what follows that the weights of P are all equal to 1 and thus our queryspace is denote by  (cid:0) P, R d , ϕ (cid:1) , while our proofs can be easily extended to the general case where weare given a weight function w : P → R as discussed in Section 2.5.
Revised: Furthermore, for simplicity, weassume in what follows that the weights of P are all equal to 1 and thus our query space is denote by 102 (cid:0) P, R d , ϕ (cid:1) , and the sensitivity of a point p ∈ P is simply s ( p ) = sup x ∈ X [Equation]

Revised: ϕ

Original:  2.1 Preliminaries
Revised: [Equation] q,x ) . Note that ourproofs are easily extended to the general case where we are given a weight function w : P → R asdiscussed in Section 2.5. 2.1 Preliminaries

Original: Notations.
Revised: Notations.

Original: For a positive integer n , we use [ n ] to denote the set { 1 , . . . , n } .
Revised: For a positive integer n , we use [ n ] to denote the set { 1 , . . . , n } .

Original: For c ∈ R d and a symmetric positive definite matrix G ∈ R d × d , we define E ( G, c ) :
Revised: For c ∈ R d and a symmetric positive definite matrix G ∈ R d × d , we define E ( G, c ) :

Original: = (cid:110) x ∈ R d (cid:12)(cid:12)(cid:12) ( x − c ) T G ( x − c ) ≤ 1 (cid:111)to be the ellipsoid defined by c and G .
Revised: = (cid:110) x ∈ R d (cid:12)(cid:12)(cid:12) ( x − c ) T G ( x − c ) ≤ 1 (cid:111)to be the ellipsoid defined by c and G .

Original: For an ellipsoid E ( G, c ) , each endpoint of a semi principalaxis is called a vertex of E ( G, c ) .
Revised: For an ellipsoid E ( G, c ) , each endpoint of a semi principalaxis is called a vertex of E ( G, c ) .

Original: We define rank ( P ) for any set P ⊆
Revised: We define rank ( P ) for any set P ⊆

Original: R d to be the dimension ofthe affine subspace that P lies on.
Revised: R d to be the dimension ofthe affine subspace that P lies on.

Original: For a set P ⊂ R d the convex hull of P is denoted by Conv( P ) .
Revised: For a set P ⊂ R d the convex hull of P is denoted by Conv( P ) .

Original: Finally, vectors are treated as column vectors.
Revised: Finally, vectors are treated as column vectors.

Original: 2.2 Novelty - Löwner ellipsoid meets Carathéodory
Revised: 2.2 Novelty - Löwner ellipsoid meets Carathéodory

Original: Our method hinges upon a combination of two known tools from convex geometry.
Revised: Our method hinges upon a combination of two known tools from convex geometry.

Original: The noveltyof our approach exploits the following observation.
Revised: The noveltyof our approach exploits the following observation.

Original: Most activation functions ϕ are continuousnon-decreasing functions, which indicate that for every query x and a set of points P , the maximalcontribution to (cid:80) p ∈ P ϕ
Revised: Most activation functions ϕ are continuousnon-decreasing functions, which indicate that for every query x and a set of points P , the maximalcontribution to (cid:80) p ∈ P ϕ

Original: ( p T x ) with respect to such activation function is associated to a point onthe convex hull of P .
Revised: ( p T x ) with respect to such activation function is associated to a point onthe convex hull of P .

Original: By finding a geometrical body B of bounded number of vertices, that is (i) enclosed in Conv ( P ) and (ii) with some dilation factor (expanding) B enclose Conv ( P ) , we willbe able to represent each point p on the boundary of the convex hull of P as a convex combination oftwo points p 1 , p 2 , one of each ( p 1 ) on B and the second ( p 2 ) on its dilated form, which is formalizedas the set { α ( x − c )
Revised: By finding a geometrical body B of bounded number of vertices, that is (i) enclosed in Conv ( P ) and (ii) with some dilation factor (expanding) B enclose Conv ( P ) , we willbe able to represent each point p on the boundary of the convex hull of P as a convex combination oftwo points p 1 , p 2 , one of each ( p 1 ) on B and the second ( p 2 ) on its dilated form, which is formalizedas the set { α ( x − c )

Original: + c | x ∈ B, ∀ α ∈ [0 , 1] } , where c here denotes the center of B .
Revised: + c | x ∈ B, ∀ α ∈ [0 , 1] } , where c here denotes the center of B .

Original: For such task,
Revised: For such task,

Original: Löwner ellipsoid is leveraged.
Revised: Löwner ellipsoid is leveraged.

Original: Theorem 2.1 (John-Löwner ellipsoid[43]) .
Revised: Theorem 2.1 (John-Löwner ellipsoid[42]) .

Original: Let
Revised: Let

Original: L ⊆ R
Revised: L ⊆ R

Original: d be a set of points such that the convexhull of L has a nonempty interior.
Revised: d be a set of points such that the convexhull of L has a nonempty interior.

Original: Then, there exists an ellipsoid E ( G, c ) (also known as the
Revised: Then, there exists an ellipsoid E ( G, c ) (also known as the

Original: MVEE ), where G ∈ R d × d is a positive definite matrix and c ∈ R d , of minimal volume such that1 d ( E ( G, c ) − c ) + c ⊆
Revised: MVEE ), where G ∈ R d × d is a positive definite matrix and c ∈ R d , of minimal volume such that1 d ( E ( G, c ) − c ) + c ⊆

Original: Conv ( L ) ⊆
Revised: Conv ( L ) ⊆

Original: E ( G, c ) .
Revised: E ( G, c ) .

Original: If L is symmetric around the center c , then thedilation factor can be reduced to 1 √ d .
Revised: If L is symmetric around the center c , then thedilation factor can be reduced to 1 √ d .

Original: Algorithm
Revised: Algorithm

Original: Input : P ⊆ R d of n points with rank r
Revised: Input : P ⊆ R d of n points with rank r

Original: Output : A subset S ⊆ P that satisfies
Revised: Output : A subset S ⊆ P that satisfies

Original: Lemma 2.
Revised: Lemma 2.

Original: 1 ( Y, z ) := affine subspace that P lies on,
Revised: 1 ( Y, z ) := affine subspace that P lies on,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 3 Let map : P ′
Revised: 3 Let map : P ′

Original: → P a function that maps every p ′ ∈ P ′ to its corresponding point p
Revised: → P a function that maps every p ′ ∈ P ′ to its corresponding point p

Original: [Equation]
Revised: [Equation]

Original: P 4 S
Revised: P 4 S

Original: [Equation]
Revised: [Equation]

Original: ∅
Revised: ∅

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: 5 E
Revised: 5 E

Original: [Equation]
Revised: [Equation]

Original: G, c
Revised: G, c

Original: [Equation]
Revised: [Equation]

Original: MVEE
Revised: MVEE

Original: [Equation]
Revised: [Equation]

Original: Conv
Revised: Conv

Original: [Equation]
Revised: [Equation]

Original: P ′ ))
Revised: P ′ ))

Original: 6 V := the vertices of ellipsoidr ( E ( G, c ) − c )
Revised: 6 V := the vertices of ellipsoidr ( E ( G, c ) − c )

Original: + c 7 for every v ∈ V do
Revised: + c 7 for every v ∈ V do

Original: K :=
Revised: K :=

Original: K ∪ C ARATHEODORY -S ET ( v, P ′ ) //
Revised: K ∪ C ARATHEODORY -S ET ( v, P ′ ) //

Original: See Algorithm 4 in the supplementary material.
Revised: See Algorithm 4 in the supplementary material.

Original: 9 end 10 S :
Revised: 9 end 10 S :

Original: = { map ( q ) | q ∈ K } 11 return S
Revised: = { map ( q ) | q ∈ K } 11 return S

Original: Algorithm input :
Revised: Algorithm input :

Original: A set P ⊆
Revised: A set P ⊆

Original: R d of n points, and a sample size
Revised: R d of n points, and a sample size

Original: m output : A weighted set ( C, u ) 1 Q := P , i := 1 , C := ∅ 2 while | Q | ≥ 2rank ( Q ) 2 do
Revised: m output : A weighted set ( C, u ) 1 Q := P , i := 1 , C := ∅ 2 while | Q | ≥ 2rank ( Q ) 2 do

Original: S i := ℓ ∞ -C ORESET ( Q ) for every p ∈ S
Revised: S i := ℓ ∞ -C ORESET ( Q ) for every p ∈ S

Original: i do s ( p ) :
Revised: i do s ( p ) :

Original: = 2rank( Q ) 1 .
Revised: = 2rank( Q ) 1 .

Original: 5 i end
Revised: 5 i end

Original: Q :=
Revised: Q :=

Original: Q \ S i , i :=
Revised: Q \ S i , i :=

Original: i +8 end 9 for every p ∈ Q do s ( p ) :
Revised: i +8 end 9 for every p ∈ Q do s ( p ) :

Original: = 2rank( Q ) 1 .
Revised: = 2rank( Q ) 1 .

Original: 5 i 11 end 12 t := (cid:80)
Revised: 5 i 11 end 12 t := (cid:80)

Original: p ∈
Revised: p ∈

Original: P s ( p ) 13 C := an i.i.d sample of m points from P , where each p ∈ P is sampled with probability s ( p ) t .
Revised: P s ( p ) 13 C := an i.i.d sample of m points from P , where each p ∈ P is sampled with probability s ( p ) t .

Original: 14 u ( p ) :
Revised: 14 u ( p ) :

Original: = t m · s ( p ) for every p ∈ C 15 return ( C, u )
Revised: = t m · s ( p ) for every p ∈ C 15 return ( C, u )

Original: Afterwards, p 1 and p 2 should be represented by points from P .
Revised: Afterwards, p 1 and p 2 should be represented by points from P .

Original: Each point on B (specifically, p 1 )can be represented via a convex combination of d + 1 points from P .
Revised: Each point on B (specifically, p 1 )can be represented via a convex combination of d + 1 points from P .

Original: The same holds for points onthe dilated form of B (e.g., p 2 ) but via a conical combination (linear combination where the weightsare non-negative and the sum of weights is not necessarily 1 ).
Revised: The same holds for points onthe dilated form of B (e.g., p 2 ) but via a conical combination (linear combination where the weightsare non-negative and the sum of weights is not necessarily 1 ).

Original: This problem is solved by invoking
Revised: This problem is solved by invoking

Original: Carathéodory theorem.
Revised: Carathéodory theorem.

Original: regression loss function (shown in (ii)) using
Revised: regression loss function (shown in (ii)) using

Original: Definition 2.3.
Revised: Definition 2.3.

Original: Following this step, a set of points P can be projected into a low dimensional subspace of dimension rank ( P ) using any dimensionality reduction algorithm as presented in (iii), resulting in the set P is its
Revised: Following this step, a set of points P can be projected into a low dimensional subspace of dimension rank ( P ) using any dimensionality reduction algorithm as presented in (iii), resulting in the set P is its

Original: Theorem 2.2 ([10, 93]) .
Revised: Theorem 2.2 ([10, 92]) .

Original: For any A ⊂ R d and p ∈
Revised: For any A ⊂ R d and p ∈

Original: Conv( A ) , there exists m ≤ d + 1 pointsp 1 , . .
Revised: Conv( A ) , there exists m ≤ d + 1 pointsp 1 , . .

Original: , p m ∈ A (denoted by a Carathéodory set of p )
Revised: , p m ∈ A (denoted by a Carathéodory set of p )

Original: such that p ∈
Revised: such that p ∈

Original: Conv ( { p 1 , . .
Revised: Conv ( { p 1 , . .

Original: , p m } ) .
Revised: , p m } ) .

Original: Finally, it is known that some functions, including the ReLU function, do not admit an ε -coreset ofsize o ( n ) [80, 79].
Revised: Finally, it is known that some functions, including the ReLU function, do not admit an ε -coreset ofsize o ( n ) [79, 78].

Original: Thus, we use a generalized form of what is known as the complexity measure of aset of points, which was first introduced in [79] and later leveraged in [74].
Revised: Thus, we use a generalized form of what is known as the complexity measure of aset of points, which was first introduced in [78] and later leveraged in [73].

Original: This measure is used todetermine the complexity of a given set P with respect to ReLU, and the coreset size theoretically.
Revised: This measure is used todetermine the complexity of a given set P with respect to ReLU, and the coreset size theoretically.

Original: Definition 2.3 (Regression Complexity Measure) .
Revised: Definition 2.3 (Regression Complexity Measure) .

Original: Let P ⊆
Revised: Let P ⊆

Original: R
Revised: R

Original: d × { 1 } , the regression complexitymeasure of P is defined as µ ( P ) = sup 
Revised: d × { 1 } , the regression complexitymeasure of P is defined as µ ( P ) = sup x ∈ R

Original: [Equation]
Revised: [Equation]

Original: | , where the denominator is ≥ 0 ,and the last entry of every p ∈ P is 1 , reserved for the bias/intercept term.
Revised: | , where the denominator is ≥ 0 ,and the last entry of every p ∈ P is 1 , reserved for the bias/intercept term.

Original: 2.3 Our Pruning Scheme
Revised: 2.3 Our Pruning Scheme

Original: In what follows, we present our data summarization technique for ReLU on the dot product function.
Revised: In what follows, we present our data summarization technique for ReLU on the dot product function.

Original: Then in Section 2.5, we discuss that our results can be easily extended to a wide family of activationfunctions including the Sigmoid function, as recently shown in [74].
Revised: Then in Section 2.5, we discuss that our results can be easily extended to a wide family of activationfunctions including the Sigmoid function, as recently shown in [73].

Original: First we present Algorithm 1,which serves as a stepping stone towards bounding the sensitivities.
Revised: First we present Algorithm 1,which serves as a stepping stone towards bounding the sensitivities.

Original: Overview of Algorithm 1.
Revised: Overview of Algorithm 1.

Original: The algorithm receives as input a set P ⊂ R
Revised: The algorithm receives as input a set P ⊂ R

Original: d whose rank is
Revised: d whose rank is

Original: r ∈ [ d ] anddeterministically finds a subset S ⊆ P which satisfies that for every j ∈ [ d ] , X ∈ R d × j and v ∈ R d ,
Revised: r ∈ [ d ] anddeterministically finds a subset S ⊆ P which satisfies that for every j ∈ [ d ] , X ∈ R d × j and v ∈ R d ,

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure][Equation] p [Equation]
Revised: [Figure]

Original: v ) X

Original: [Equation]
Revised: [Equation]

Original: ≤ r 1 . 5 . To do so, first, we find the affine hyperplane that P lies on, followed bycomputing the low dimensional representation of P , denoted by P ′ ; see Lines 1–2.
Revised: To do so, first, we find the affine hyperplane that P lies on, followed bycomputing the low dimensional representation of P , denoted by P ′ ; see Lines 1–2.

Original: Note that ifrank( P ) =
Revised: Note that ifrank( P ) =

Original: d , then we can either keep P as it is (i.e., P ′ :
Revised: d , then we can either keep P as it is (i.e., P ′ :

Original: = P ), or use dimensionality reductiontricks as detailed in Section 2.5.
Revised: = P ), or use dimensionality reductiontricks as detailed in Section 2.5.

Original: To compute the output S , we first bound the convex hull of P ′ by its
Revised: To compute the output S , we first bound the convex hull of P ′ by its

Original: Löwner ellipsoid E ( G, c ) in Line 5, followed by computing the dilated ellipsoid of E ( G, c ) , namely,1 r ( E − c ) + c .
Revised: Löwner ellipsoid E ( G, c ) in Line 5, followed by computing the dilated ellipsoid of E ( G, c ) , namely,1 r ( E − c ) + c .

Original: Let V be the set of vertices of such ellipsoid; see Line 6.
Revised: Let V be the set of vertices of such ellipsoid; see Line 6.

Original: Now, for each point v ∈ V ,we represent it as a convex combination of r + 1 points from P ′ via Theorem 2.2, and store the unionof such sets (each of size at most r + 1 ) of points into K as done in Lines 7–9.
Revised: Now, for each point v ∈ V ,we represent it as a convex combination of r + 1 points from P ′ via Theorem 2.2, and store the unionof such sets (each of size at most r + 1 ) of points into K as done in Lines 7–9.

Original: For each point in K ,we map it back to R d to satisfy Lemma 2.5.
Revised: For each point in K ,we map it back to R d to satisfy Lemma 2.5.

Original: To sum up Algorithm 1, we observe that the vertices can be used via canonical combinations withtheir dilated form to describe every point on the convex hull of the input data (in our end, it would thenetwork’s weights).
Revised: To sum up Algorithm 1, we observe that the vertices canbe used via canonical combinations with their dilated form to describe every point on the convex hullof the input data (in our end, it would the network’s weights).

Original: Hence the Carathéodory set of these vertices from the input points lying on theconvex hull can be further used to also represent points lying on the convex hull.
Revised: Hence the Carathéodory set of thesevertices from the input points lying on the convex hull can be further used to also represent pointslying on the convex hull.

Original: This is the core ideawhich enable us in forming our ℓ ∞ coreset for any ℓ ρ regression problem where ρ ∈ (0 , ∞ ) .
Revised: This is the core idea which enable us in forming our ℓ ∞ coreset for any ℓ ρregression problem where ρ ∈ (0 , ∞ ) .

Original: We now discuss Algorithm 2 which is responsible for constructing an ε -coreset with respect toactivation functions.
Revised: We now discuss Algorithm 2 which is responsible for constructing an ε -coreset with respect toactivation functions.

Original: Its input is a set P ⊂ R d and a sample size m ≥ 1 .
Revised: Its input is a set P ⊂ R d and a sample size m ≥ 1 .

Original: Overview of Algorithm 2.
Revised: Overview of Algorithm 2.

Original: First set Q := P .
Revised: First set Q := P .

Original: At each iteration i , the algorithm obtains a subset
Revised: At each iteration i , the algorithm obtains a subset

Original: S i ⊆ Q as an output to a call to ℓ ∞ -C ORESET ( Q ) as stated in Line 3 of Algorithm 

Revised: S i ⊆ Q as an output to a call to ℓ ∞ -C ORESET ( Q ) as stated in Line 3 of Algorithm

Original: such that forevery x ∈ R d , it holds that
Revised: such that forevery x ∈ R d , it holds that

Original: [Equation]
Revised: [Equation]

Original: ≤ r 1 . 5 with r being the rank of Q .
Revised: ≤ r 1 . 5 with r being the rank of Q .

Original: The sensitivity of eachpoint in S i is bounded from above by 2 d 1 .
Revised: The sensitivity of eachpoint in S i is bounded from above by 2 d 1 .

Original: i as stated in Lines 4–6.
Revised: i as stated in Lines 4–6.

Original: The idea behind these boundslies in our proof of Theorem 2.6.
Revised: The idea behind these boundslies in our proof of Theorem 2.6.

Original: The set S i is removed from Q , and this procedure is repeated withrespect to Q until the size of Q is small enough.
Revised: The set S i is removed from Q , and this procedure is repeated withrespect to Q until the size of Q is small enough.

Original: The obtained sensitivities are the ones needed forcomputing the pruning coresets.
Revised: The obtained sensitivities are the ones needed forcomputing the pruning coresets.

Original: Finally, we utilize the sensitivity sampling framework of [9] toobtain the desired coreset; see Lines 13–14.
Revised: Finally, we utilize the sensitivity sampling framework of [9] toobtain the desired coreset; see Lines 13–14.

Original: 2.4 Analysis
Revised: 2.4 Analysis

Original: In this section, we prove the correctness of our algorithms.
Revised: In this section, we prove the correctness of our algorithms.

Original: The following lemma shows that for eachpoint p that is inside some convex hull S , its ℓ 1 distance to any affine subspace is always boundedfrom above by ℓ 1 distance from the same affine subspace, of some other point q ∈ S .
Revised: The following lemma shows that for eachpoint p that is inside some convex hull S , its ℓ 1 distance to any affine subspace is always boundedfrom above by ℓ 1 distance from the same affine subspace, of some other point q ∈ S .

Original: Lemma 2.4.
Revised: Lemma 2.4.

Original: Let d, ℓ, m ≥ 1 be integers.
Revised: Let d, ℓ, m ≥ 1 be integers.

Original: Let p ∈ R d and A ⊆ R d be a set of m points
Revised: Let p ∈ R d and A ⊆ R d be a set of m points

Original: withp ∈
Revised: withp ∈

Original: Conv( A ) so that there exists α :
Revised: Conv( A ) so that there exists α :

Original: A → [0 , 1] such that (cid:80) q ∈
Revised: A → [0 , 1] such that (cid:80) q ∈

Original: A α ( q ) = 1 and (cid:80) q ∈
Revised: A α ( q ) = 1 and (cid:80) q ∈

Original: A α ( q ) · q = p .
Revised: A α ( q ) · q = p .

Original: Then for every Y ∈ R d × ℓ and v ∈ R ℓ , ∥ ( p − v ) Y ∥ 1 ≤ max q ∈
Revised: Then for every Y ∈ R d × ℓ and v ∈ R ℓ , ∥ ( p − v ) Y ∥ 1 ≤ max q ∈

Original: A ∥ ( q − v ) Y ∥ 1 .
Revised: A ∥ ( q − v ) Y ∥ 1 .

Original: The following states the provable guarantees of Algorithm 1.
Revised: The following states the provable guarantees of Algorithm 1.

Original: Lemma 2.5 ( ℓ ∞ -coreset for ℓ 1 -regression) .
Revised: Lemma 2.5 ( ℓ ∞ -coreset for ℓ 1 -regression) .

Original: Let P ⊆
Revised: Let P ⊆

Original: R d be a set of points, and r be the rank of P .
Revised: R d be a set of points, and r be the rank of P .

Original: Let j ∈ [ d − 1] and let S be the output of a call to ℓ ∞ -C ORESET ( P ) .
Revised: Let j ∈ [ d − 1] and let S be the output of a call to ℓ ∞ -C ORESET ( P ) .

Original: Then (i) | S | ∈ O (cid:0) r 2 (cid:1) , and(ii) for every X ∈ R d × j and v ∈ R d , max q ∈ P ∥ ( q − v )
Revised: Then (i) | S | ∈ O (cid:0) r 2 (cid:1) , and(ii) for every X ∈ R d × j and v ∈ R d , max q ∈ P ∥ ( q − v )

Original: X ∥ 1 max q ∈ S ∥ ( q − v ) X ∥ 1 ∈ (cid:2) 1 , 2 r 1 .
Revised: X ∥ 1 max q ∈ S ∥ ( q − v ) X ∥ 1 ∈ (cid:2) 1 , 2 r 1 .

Original: 5 (cid:3) .
Revised: 5 (cid:3) .

Original: The following theorem states our main result.
Revised: The following theorem states our main result.

Original: Theorem 2.6 (ReLU ε -coreset) .
Revised: Theorem 2.6 (ReLU ε -coreset) .

Original: Let P ⊆ R d , ε, δ ∈ (0 , 1) , r = rank ( P ) , µ ( P ) be as in Definition 2.3, and let m ∈ O (cid:16) µ ( P ) r 3 .
Revised: Let P ⊆ R d , ε, δ ∈ (0 , 1) , r = rank ( P ) , µ ( P ) be as in Definition 2.3, and let m ∈ O (cid:16) µ ( P ) r 3 .

Original: 5 log n ε(cid:0) d (log ( µ ( P ) r log n ))
Revised: 5 log n ε(cid:0) d (log ( µ ( P ) r log n ))

Original: + log (cid:0) 1 δ (cid:1)(cid:1)(cid:17) .
Revised: + log (cid:0) 1 δ (cid:1)(cid:1)(cid:17) .

Original: Let ( C, w ) be theoutput of a call to C ORESET ( P, m ) ; see Algorithm 2.
Revised: Let ( C, w ) be theoutput of a call to C ORESET ( P, m ) ; see Algorithm 2.

Original: Then, with probability at least 1 − δ , ( C, w ) isan ε -coreset for (cid:0) P, R d , ReLU ( · ) (cid:1) ; see Definition 1.1.
Revised: Then, with probability at least 1 − δ , ( C, w ) isan ε -coreset for (cid:0) P, R d , ReLU ( · ) (cid:1) ; see Definition 1.1.

Original: Time analysis.
Revised: Time analysis.

Original: Letting r be the rank of P , the time complexity of Algorithm 1 can be dissectedto two main parts: (i)
Revised: Letting r be the rank of P , the time complexity of Algorithm 1 can be dissectedto two main parts: (i)

Original: Computing the Löwner ellipsoid in O ( nr 2 log n ) time using the methodproposed in [96] and (ii) computing the Carathédory set in O ( nr + r 4 log n ) time via [70].
Revised: Computing the Löwner ellipsoid in O ( nr 2 log n ) time using the methodproposed in [95] and (ii) computing the Carathédory set in O ( nr + r 4 log n ) time via [69].

Original: Since Vcan contain up to O ( r 2 ) points, the overall time for Algorithm 1 is O ( nr 2 log n + nr 3 + r 6 log n ) .
Revised: Since Vcan contain up to O ( r 2 ) points, the overall time for Algorithm 1 is O ( nr 2 log n + nr 3 + r 6 log n ) .

Original: As for Algorithm 2, it takes O ( n 2 r 2 log n + nr 4 log n ) .
Revised: As for Algorithm 2, it takes O ( n 2 r 2 log n + nr 4 log n ) .

Original: Indeed, as explained in Section 2.5, adimensionality reduction algorithm may be applied to improve the run time (reducing the r 6 factor).
Revised: Indeed, as explained in Section 2.5, adimensionality reduction algorithm may be applied to improve the run time (reducing the r 6 factor).

Original: Furthermore, the run time of our algorithm can be improved, using the merge-and-reduce tree fromthe literature of coresets to reduce the n 2 terms to n log n , i.e., the running time can be reduced to
Revised: Furthermore, the run time of our algorithm can be improved, using the merge-and-reduce tree fromthe literature of coresets to reduce the n 2 terms to n log n , i.e., the running time can be reduced to

Original: O ( n log 2 ( n ) r 2 + nr 4 ) .
Revised: O ( n log 2 ( n ) r 2 + nr 4 ) .

Original: For a data-independent provable method, this running time is reasonable.
Revised: For a data-independent provable method, this running time is reasonable.

Original: Our advantages over previous results.
Revised: Our advantages over previous results.

Original: Our coreset supports different activation functions withoutthe need to change the sensitivity that much.
Revised: Our coreset supports different activation functions withoutthe need to change the sensitivity that much.

Original: Specifically, it will only be multiplied by some scalar,unlike previous coresets where different losses impose drastically different sensitivities/leveragescores and algorithms.
Revised: Specifically, it will only be multiplied by some scalar,unlike previous coresets where different losses impose drastically different sensitivities/leveragescores and algorithms.

Original: This is since our coreset unlike other coresets is in its essence a frameworkof coresets for different ℓ ρ losses, as it can be used as is for different ℓ ρ losses and yet still attainϵ -approximation.
Revised: This is since our coreset unlike other coresets is in its essence a frameworkof coresets for different ℓ ρ losses, as it can be used as is for different ℓ ρ losses and yet still attainϵ -approximation.

Original: In addition, when the rank of the input points is small, then our method outperformsprevious methods.
Revised: In addition, when the rank of the input points is small, then our method outperformsprevious methods.

Original: If the input data is of full rank, previous methods obtain faster coresets construction.
Revised: If the input data is of full rank, previous methods obtain faster coresets construction.

Original: On the boundness of the regression complexity measure.
Revised: On the boundness of the regression complexity measure.

Original: First of all, there exists an examplewhere the complexity measure is unbounded, e.g., consider a set of points distributed evenly ona unit ball.
Revised: First of all, there exists an examplewhere the complexity measure is unbounded, e.g., consider a set of points distributed evenly ona unit ball.

Original: In this case, you can always find a point where a hyperplane separating it from therest of the points can be found such that the one half-space of this hyperplane contains only thispoint while the other half-space contains the rest of the points.
Revised: In this case, you can always find a point where a hyperplane separating it from therest of the points can be found such that the one half-space of this hyperplane contains only thispoint while the other half-space contains the rest of the points.

Original: This leads to an infinite complexitymeasure.
Revised: This leads to an infinite complexitymeasure.

Original: Such an example is also mentioned in [80], when assessing the hardness of generatingmultiplicative-approximation coresets for ReLU functions.
Revised: Such an example is also mentioned in [79], when assessing the hardness of generatingmultiplicative-approximation coresets for ReLU functions.

Original: Theoretically, the complexity measure is influenced by how free can the bias term be (the last entry ofx); see Definition 2.3.
Revised: Theoretically, the complexity measure is influenced by how free can the bias term be (the last entry ofx); see Definition 2.3.

Original: This term is the only thing that can ensure that one point can be separated fromthe rest in the sense of finding a separating hyperplane, leading to an infinite complexity measure.
Revised: This term is the only thing that can ensure that one point can be separated fromthe rest in the sense of finding a separating hyperplane, leading to an infinite complexity measure.

Original: Bounding on this term, leads to bounded complexity measure from a theoretical point of view.
Revised: Bounding on this term, leads to bounded complexity measure from a theoretical point of view.

Original: In the context of model pruning, from the perspective of the complexity measure, the model’s weightsare the input denoted by a matrix P ∈ R n × ( d +1) , while the query is now R d × { 1 } .
Revised: In the context of model pruning, from the perspective of the complexity measure, the model’s weightsare the input denoted by a matrix P ∈ R n × ( d +1) , while the query is now R d × { 1 } .

Original: Thus thecomplexity measure is now µ ( P ) :
Revised: Thus thecomplexity measure is now µ ( P ) :

Original: = sup x ∈ R d ×{ 1 } (cid:80)
Revised: = sup x ∈ R d ×{ 1 } (cid:80)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: |
Revised: |

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x |
Revised: x |

Original: [Equation]
Revised: [Equation]

Original: q ∈ { p
Revised: q ∈ { p

Original: [Equation]
Revised: [Equation]

Original: P | pTx> 0 }| q T x | .
Revised: P | pTx> 0 }| q T x | .

Original: With this in mind, weobserve that the complexity measure is now an instance of the complexity measure used in [74].
Revised: With this in mind, weobserve that the complexity measure is now an instance of the complexity measure used in [73].

Original: Thecomplexity measure now relies entirely on the structure of the model’s weights, where the goal is tofind the largest ratio between the sum of the absolute of the values inside the rectified neurons priorto applying the rectification, and the sum values of non-rectified neurons.
Revised: Thecomplexity measure now relies entirely on the structure of the model’s weights, where the goal is tofind the largest ratio between the sum of the absolute of the values inside the rectified neurons priorto applying the rectification, and the sum values of non-rectified neurons.

Original: To bound this measure, wecan use a variant of the algorithm described in the proof of Theorem 3 in [79].
Revised: To bound this measure, wecan use a variant of the algorithm described in the proof of Theorem 3 in [78].

Original: 2.5 Extensions
Revised: 2.5 Extensions

Original: Our suggested scheme can be extended to support many other variants of the pruning problem.
Revised: Our suggested scheme can be extended to support many other variants of the pruning problem.

Original: Various activation functions.
Revised: Various activation functions.

Original: Our result can be extended to a family of activation functions called“Nice hinge functions”; see Definition D.1.
Revised: Our result can be extended to a family of activation functions called“Nice hinge functions”; see Definition D.1.

Original: Let ( P, X, ϕ ) be a query space, where ϕ is a “Nice hingefunctions”.
Revised: Let ( P, X, ϕ ) be a query space, where ϕ is a “Nice hingefunctions”.

Original: To bound the sensitivity of a point p , we first bound the nominator of s ( p ) by provingthat ∀ x ∈ X : ϕ ( p T x ) < (cid:12)(cid:12) p T x (cid:12)(cid:12) .
Revised: To bound the sensitivity of a point p , we first bound the nominator of s ( p ) by provingthat ∀ x ∈ X : ϕ ( p T x ) < (cid:12)(cid:12) p T x (cid:12)(cid:12) .

Original: For bounding the denominator from below, recently [74] provedthat ∀ x ∈ X : (cid:80) p ∈ P ReLU (cid:0) p T x (cid:1) ⪅ (cid:80) p ∈ P ϕ ( p T x ) ; see full detail in Section D.3.
Revised: For bounding the denominator from below, recently [73] provedthat ∀ x ∈ X : (cid:80) p ∈ P ReLU (cid:0) p T x (cid:1) ⪅ (cid:80) p ∈ P ϕ ( p T x ) ; see full detail in Section D.3.

Original: Weighted Input.
Revised: Weighted Input.

Original: In the context of deep learning, the output of each neuron is multiplied with a scalarwhich brings the necessity of having the ability to deal with weighted set of points.
Revised: In the context of deep learning, the output of each neuron is multiplied with a scalarwhich brings the necessity of having the ability to deal with weighted set of points.

Original: Algorithm 2 canbe extended easily to the case as generously detailed in Section D.1 in the Appendix.
Revised: Algorithm 2 canbe extended easily to the case as generously detailed in Section D.1 in the Appendix.

Original: Dimensionality reduction.
Revised: Dimensionality reduction.

Original: All coreset-based pruning methods rely heavily on the dimensionality ofthe model’s layers, as well as our method.
Revised: All coreset-based pruning methods rely heavily on the dimensionality ofthe model’s layers, as well as our method.

Original: To ensure sufficient pruning ratio, we apply either PCA,
Revised: To ensure sufficient pruning ratio, we apply either PCA,

Original: TSNE, MDS, or the JL transform on the weights of each layer prior to generating its coreset.
Revised: TSNE, MDS, or the JL transform on the weights of each layer prior to generating its coreset.

Original: From weight to neuron pruning.
Revised: From weight to neuron pruning.

Original: Most coreset-based pruning methods, e.g., [5, 80], first provide ascheme for (provable) weight pruning, which is then used as a stepping stone towards pruning neuronsas follows.
Revised: Most coreset-based pruning methods, e.g., [5, 79], first provide ascheme for (provable) weight pruning, which is then used as a stepping stone towards pruning neuronsas follows.

Original: [5] first suggested coreset-based neuron pruning via the use of a generated controled set ofqueries to evaluate the importance of weights.
Revised: [5] first suggested coreset-based neuron pruning via the use of a generated controled set ofqueries to evaluate the importance of weights.

Original: Any neuron that has a maximal activation value loweror equal to zero, will be pruned from the network as its impact on the rest of the neurons is minimal.
Revised: Any neuron that has a maximal activation value loweror equal to zero, will be pruned from the network as its impact on the rest of the neurons is minimal.

Original: On the other hand [80] altered the definition of sensitivity such that it takes into account the sensitivityof a neuron in a layer ℓ with respect to all the neurons in the layer ℓ + 1 , which basically meansthat the sensitivity of each neuron is taken be the maximal sensitivity over every weight function(neuron in the next layer) defined by the layer.
Revised: On the other hand [79] altered the definition of sensitivity such that it takes into account the sensitivityof a neuron in a layer ℓ with respect to all the neurons in the layer ℓ + 1 , which basically meansthat the sensitivity of each neuron is taken be the maximal sensitivity over every weight function(neuron in the next layer) defined by the layer.

Original: Hence, we follow the same logic for such method; see
Revised: Hence, we follow the same logic for such method; see

Original: Section D.2 in the supplementary material.
Revised: Section D.2 in the supplementary material.

Original: From neuron to filter pruning.
Revised: From neuron to filter pruning.

Original: Convolutions can be expressed as matrix multiplications, whichenables our method to prune filters from any model as done by [61, 80, 62].
Revised: Convolutions can be expressed as matrix multiplications, whichenables our method to prune filters from any model as done by [60, 79, 61].

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we study various widely used network architectures and benchmark data-sets.
Revised: In this section, we study various widely used network architectures and benchmark data-sets.

Original: Following [80], to test the robustness of our methods on each of the neuron and filter pruning tasksindependently, two sets of experiments are conducted.
Revised: Following [79], to test the robustness of our methods on each of the neuron and filter pruning tasksindependently, two sets of experiments are conducted.

Original: The first focuses on pruning neurons (Section 3.1) whereas the second focuses on pruning filters (Section 3.2), both via our coreset method.
Revised: The first focuses on pruning neurons (Section 3.1) whereas the second focuses on pruning filters (Section 3.2), both via our coreset method.

Original: The setting.
Revised: The setting.

Original: In all experiments we report the Pruning ratio – the percentage of the parameters thatwere removed from the original model, and Err – the percentage of misclassified test instances ofour method compared to coreset-based pruning methods and more.
Revised: In all experiments we report the Pruning ratio – the percentage of the parameters thatwere removed from the original mode. Here, PR stands for pruning ratio, FR stands for floating-pointreduction ratio and Err – the percentage of misclassified test instances of our method compared tocoreset-based pruning methods and more.

Original: “Baseline Err” is the error ofthe original uncompressed network, while "Pruned Err" is the classification error of the compressedmodel.
Revised: Baseline Err is the error of the original uncompressednetwork, while Pruned Err is the classification error of the compressed model.

Original: In our experiment we compress and fine-tune the network once, no iterative pruning wasapplied, thus, the compared methods also satisfy this setting.
Revised: In our experimentwe compress and fine-tune the network once, no iterative pruning was applied, thus, the comparedmethods also satisfy this setting.

Original: Throughout our experiments PR standsfor pruning ratio, and FR stands for floating-point reduction ratio.
Revised: Each experiment was conducted 5 times, in the tables, we reportfor our method the best error achieved and we highlight in parentheses next to it the average errorand standard deviation across the 5 trails. In all of our experiments, the models are fine-tuned tillconvergence (after pruning).

Original: Implementation details are referedto Section E in the Appendix.
Revised: Implementation details are given in Section E in the Appendix.

Original: [Table]
Revised: Software/Hardware.

Original: Our 
Revised: Our algorithms were implemented in Python 3.6 [103] using Numpy [80], and

Original: [Table]
Revised: Pytorch [81].

Original: Our method [Table]
Revised: Tests were performed on NVIDIA DGX A100 servers with 8 NVIDIA A100 GPUseach, fast InfiniBand interconnect and supporting infrastructure.

Original: Baselines.
Revised: Baselines.

Original: Our results are compared to (i) PFP [61], (ii) FT [59], (iii) SoftNet [34], (iv) ThiNet [69],and (v) PvC [80], (vi) Soft Pruning [34], (vii) Software/Hardware.
Revised: Our results are compared to (i) PFP [60], (ii) FT [58], (iii) SoftNet [34], (iv) ThiNet [68],and (v) PvC [79], (vi) Soft Pruning [34], (vii) CCP [82], (viii)

Original: Our algorithms were implemented in Python 3.6 [104] using Numpy [81], and Pytorch [82].
Revised: FPGM [36], (ix) ThiNet-70, (x) ThiNet-50 [68], (xi) Pruning via Coresets (PvC) [79], (xii)

Original: Tests were performed on NVIDIA DGX A100 servers with 8 NVIDIA A100 GPUseach, fast InfiniBand interconnect and supporting infrastructure.
Revised: Pruning from Scratch (PfS) [106], and(xiii)

Original:  3.1 Neuron Pruning
Revised: Rethinking the value of network pruning (Rethink) [65]. 3.1 Neuron Pruning

Original: We test our method on(i) VGG16 [90] using CIFAR10 [50], and(ii) LeNet300-100 using MNIST [56].
Revised: We test our method on VGG16 [89] using CIFAR10 [49], and LeNet300-100 using MNIST [55].

Original: After pruning, we fine-tune the pruned model till convergence.

Original: Discussion.
Revised: Discussion.

Original: Table 1 present the results of LeNet 300 - 100 and VGG 16 .
Revised: Table 1 present the results of LeNet 300 - 100 and VGG 16 .

Original: Observe that in both architectures, our method outperformed the competing methods under the same compression scenarios.
Revised: Observe that in both architectures, our method outperformed the competing methods under the same compression scenarios.

Original: Insummary, we pruned roughly 90% of the parameters of the LeNet-300-100 model while improvingthe accuracy of the original model.
Revised: Forexample, we pruned roughly 90% of the parameters of the LeNet-300-100 model while improvingthe accuracy of the original model.

Original: We witness a similar phenomena on the VGG16 model, where wepruned roughly 90% of the parameters of the dense layers resulting in accuracy improvement.
Revised: We witness a similar phenomena on the VGG16 model, where wepruned roughly 90% of the parameters of the dense layers resulting in accuracy improvement.

Original: Thisconfirms the insights in [5] that coresets help in improving the generalization properties of DNNs. 
Revised: Thisconfirms the insights in [5] that coresets help in improving the generalization properties of DNNs. [Table]

Original: 3.2 Filter pruning
Revised: 3.2 Filter pruning

Original: We compressed the convolutional layers of (i) ResNet 50 [33] on ILSVRC 2012 [18], (ii)
Revised: We compressed the convolutional layers of (i) ResNet 50 [33] on ILSVRC- 2012 [18], (ii)

Original: ResNet 56 [33] and (iii) VGG 19 [90] on CIFAR 10 , followed by fine-tuning the pruned modelstill convergence.
Revised: ResNet 56 [33], (iii) VGG 19 [89] on CIFAR 10 and (iv) VGG 16 [89] on CIFAR 10 . [Table]

Original: Here, we operated under the same compression settings of [80], and reported theoverall compression ratio and error. Filter pruning of DNNs on CIFAR 10 .
Revised: [Table]

Original: We used PyTorch implementation of VGG 19 model for
Revised: ThiNet27 .28 .51 .55 . PvC 23 .25 .

Original: CIFAR 10 which contains about 20 M parameters. We removed around 70% of the filters in each layer,resulting in a total of 88% compression ratio. As for ResNet 56 [40], the model has been compressedto obtain roughly 55% compression rate.
Revised: N/A

Original: Table 2 shows a comparison between our method to others. [Table]
Revised: Our method 23 .24 . 87 ( 25 . 07 ± 0 . 2 ) 61 .

Original: Compressing [Table]
Revised: Filter pruning of DNNs on CIFAR 10 and ImageNet (ILSVRC- 2012 ).

Original: 50 on ILSVRC 2012 . 
Revised: For Cifar 10 , we used PyTorch implementations of VGG 19 and VGG 16 . We compressed both models using our approachby different compression rates as shown in Table 2 with a comparison to other methods. For

Original: Here, we compressed the baseline model of ResNet 50 [33] roughly by 62% in terms of number of parameters.
Revised: ImageNet, we compressed the baseline model of ResNet 50 [33] roughly by 62% in terms of numberof parameters.

Original: Table 3 provides comparisonbetween our method and other baselines.[Table] [Table] [Table]
Revised: Table 3 provides comparison between our method and other baselines.

Original: Discussion.
Revised: Discussion.

Original: As can be seen in Tables 1, 2, and 3, our method either outperforms the competingmethods or achieves comparable results.
Revised: As can be seen in Tables 1, 2, and 3, our method either outperforms the competingmethods or achieves comparable results.

Original: As for the coreset methods,it can be seen that our algorithmachieves better result than all of them in this setting, e.g., we compressed 62% of ResNet50 trainedin ImageNet while resulting in only 1 . 09% drop in accuracy, improving the recent coreset result of PvC [80] for the same compression ratio, while PFP [61] compressed 44 .
Revised: As for the coreset methods, our algorithm achieves betterresult than all of them in this setting, e.g., we compressed 62% of ResNet50 trained on ImageNetwhile incurring 1 . 09% drop in accuracy, improving the recent coreset result of PvC [79] for the samecompression ratio, while PFP [60] compressed 44 .

Original: 04% to achieve comparableresults.
Revised: 04% to achieve comparable results.

Original:  Related work
Revised:  Related work

Original: DNNs can be compressed before training [95, 106, 58], during training [114, 111, 52], or aftertraining [91].
Revised: DNNs can be compressed before training [94, 105, 57], during training [113, 110, 51], or aftertraining [90].

Original: Furthermore, such procedures may also be repeated iteratively [86].
Revised: Furthermore, such procedures may also be repeated iteratively [85].

Original: As previouslynoted pruning can be categorized into structured and unstructured pruning.
Revised: As previouslynoted pruning can be categorized into structured and unstructured pruning.

Original: Unstructured pruning.
Revised: Unstructured pruning.

Original: Weight pruning [57] techniques aim to reduce the number of weightsin a layer while approximately preserving its output.
Revised: Weight pruning [56] techniques aim to reduce the number of weightsin a layer while approximately preserving its output.

Original: Approaches of this type include the worksof [55, 21, 39, 1, 63], where the desired sparsity is embedded as a constraint or via a regularizer intothe training pipeline, and those of [31, 86, 30], where weights with absolute values below a thresholdare removed.
Revised: Approaches of this type include the worksof [54, 21, 39, 1, 62], where the desired sparsity is embedded as a constraint or via a regularizer intothe training pipeline, and those of [31, 85, 30], where weights with absolute values below a thresholdare removed.

Original: The approaches of [5, 6] use a mini-batch of data points to approximate the influence ofeach parameter on the loss function.
Revised: The approaches of [5, 6] use a mini-batch of data points to approximate the influence ofeach parameter on the loss function.

Original: Other data-informed techniques include [28, 64, 78, 77, 112].
Revised: Other data-informed techniques include [28, 63, 77, 76, 111].

Original: Athorough overview of recent pruning approaches is given by [27, 8].
Revised: Athorough overview of recent pruning approaches is given by [27, 8].

Original: However, unlike our approach,weight-based pruning approaches generate sparse models instead of smaller ones thus requiringspecialized hardware and sparse linear algebra libraries in order to speed up inference.
Revised: However, unlike our approach,weight-based pruning approaches generate sparse models instead of smaller ones thus requiringspecialized hardware and sparse linear algebra libraries in order to speed up inference.

Original: Structured pruning.
Revised: Structured pruning.

Original: Pruning entire neurons and filters directly shrinks the network leading to smallerstorage requirements and improved inference-time performance on any hardware [60, 68].
Revised: Pruning entire neurons and filters directly shrinks the network leading to smallerstorage requirements and improved inference-time performance on any hardware [59, 67].

Original: Lately,these approaches were investigated in many papers [65, 60, 11, 36, 22, 47, 110, 109].
Revised: Lately,these approaches were investigated in many papers [64, 59, 11, 36, 22, 46, 109, 108].

Original: Usually, filtersare pruned by assigning an importance score to each neuron/filter, either solely weight-based [37, 35]or data-informed [112, 61], and removing those with a score below a threshold.
Revised: Usually, filtersare pruned by assigning an importance score to each neuron/filter, either solely weight-based [37, 35]or data-informed [111, 60], and removing those with a score below a threshold.

Original: The procedure canbe embedded into an iterative pruning scheme [86] that requires potentially expensive retrain cycles.
Revised: The procedure canbe embedded into an iterative pruning scheme [85] that requires potentially expensive retrain cycles.

Original: Tensor decomposition.
Revised: Tensor decomposition.

Original: Some of the work in DNN compression entails decomposing the layerinto multiple smaller ones, e.g., via low-rank tensor decomposition [19,42, 73, 49, 94, 41,  2, 102,113 , 54, 62].
Revised: Some of the work in DNN compression entails decomposing the layerinto multiple smaller ones, e.g., via low-rank tensor decomposition [19, 41, 72, 48, 93, 40, 2, 101,112 , 53, 61].

Original: Other approaches to tensor decomposition include weight sharing, random projections,and feature hashing [108, 3, 89, 12, 13, 103].
Revised: Other approaches to tensor decomposition include weight sharing, random projections,and feature hashing [107, 3, 88, 12, 13, 102].

Original: However, such techniques usually require expensiveapproximation algorithms or use heuristics since tensor decomposition is generally NP-hard.
Revised: However, such techniques usually require expensiveapproximation algorithms or use heuristics since tensor decomposition is generally NP-hard.

Original: Coresets.
Revised: Coresets.

Original: In the recent years, coresets got increasing attention, and where leveraged to compress theinput datasets of many machine learning algorithms, improving there performance, e.g., regression [38,79, 48, 100], decision trees [45], matrix approximation [26, 70, 25, 87, 72], clustering [24, 29, 67, 4,46, 88], ℓ z -regression [16, 17, 92], SVM [32,99, 97, 98, 101], and deep learning models [5, 61].
Revised: In the recent years, coresets got increasing attention, and where leveraged to compress theinput datasets of many machine learning algorithms, improving there performance, e.g., regression [38,78, 47, 99], decision trees [44], matrix approximation [26, 69, 25, 86, 71], clustering [24, 29, 66, 4,45, 87], ℓ z -regression [16, 17, 91], SVM [32, 98, 96, 97, 100], and deep learning models [5, 60].

Original: Forextensive surveys on coresets, we refer the reader to [23, 84, 44, 71].
Revised: Forextensive surveys on coresets, we refer the reader to [23, 83, 43, 70].

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: In this paper, we provided a coreset-based pruning technique that hinges upon a combination oftools from convex geometry, while achieving SOTA results with respect to coreset-based structuredpruning approaches on a variety of networks.
Revised: In this paper, we provided a coreset-based pruning technique that hinges upon a combination oftools from convex geometry, while achieving SOTA results with respect to coreset-based structuredpruning approaches on a variety of networks.

Original: Our main improvement is that our coreset is (training)data-independent and assumes a single assumption on the models weights.
Revised: Our main improvement is that our coreset is (training)data-independent and assumes a single assumption on the models weights.

Original: Future work includes (i) suggesting a coreset based budget allocation framework, to determine the(optimal) per layer prune ratio while achieving an overall desired compression rate, (ii) extending ourcoreset technique to other layers such as attention layers, and (iii) bridging the gap between coresetbased pruning approaches and tensor-decomposition methods, as both techniques are theoreticallysupported by bounding the approximation error given specific compression rate, we can leverage thesebounds to formulate the compression problem as an optimization problem which iterates between thetwo approaches to search for the local minimum.
Revised: Future work includes (i) suggesting a coreset based budget allocation framework, to determine the(optimal) per layer prune ratio while achieving an overall desired compression rate, (ii) extending ourcoreset technique to other layers such as attention layers, and (iii) bridging the gap between coresetbased pruning approaches and tensor-decomposition methods, as both techniques are theoreticallysupported by bounding the approximation error given specific compression rate, we can leverage thesebounds to formulate the compression problem as an optimization problem which iterates between thetwo approaches to search for the local minimum.