Original: Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions
Revised: Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Pruning is one of the predominant approaches for compressing deep neural networks (DNNs).
Revised: Pruning is one of the predominant approaches for compressing deep neural networks (DNNs).

Original: Lately, coresets (provable data summarizations) were leveragedfor pruning DNNs, adding the advantage of theoretical guarantees on the trade-offbetween the compression rate and the approximation error.
Revised: Lately, coresets (provable data summarizations) were leveraged for pruning DNNs, adding the advantage of theoretical guarantees on the trade-off between the compression rate and the approximation error.

Original: However, coresets inthis domain were either data-dependent or generated under restrictive assumptionson both the model’s weights and inputs.
Revised: However, coresets in this domain were either data-dependent or generated under restrictive assumptions on both the model’s weights and inputs.

Original: In real-world scenarios, such assumptionsare rarely satisfied, limiting the applicability of coresets.
Revised: In real-world scenarios, such assumptions are rarely satisfied, limiting the applicability of coresets.

Original: To this end, we suggest anovel and robust framework for computing such coresets under mild assumptionson the model’s weights and without any assumption on the training data.
Revised: To this end, we suggest a novel and robust framework for computing such coresets under mild assumptions on the model’s weights and without any assumption on the training data.

Original: The ideais to compute the importance of each neuron in each layer with respect to the outputof the following layer.
Revised: The idea is to compute the importance of each neuron in each layer with respect to the output of the following layer.

Original: This is achieved by a combination of Löwner ellipsoid and
Revised: This is achieved by a combination of Löwner ellipsoid and

Original: Caratheodory theorem.
Revised: Caratheodory theorem.

Original: Our method is simultaneously data-independent, applicableto various networks and datasets (due to the simplified assumptions), and theoretically supported.
Revised: Our method is simultaneously data-independent, applicable to various networks and datasets (due to the simplified assumptions), and theoretically supported.

Original: Experimental results show that our method outperforms existingcoreset based neural pruning approaches across a wide range of networks anddatasets.
Revised: Experimental results show that our method outperforms existing coreset based neural pruning approaches across a wide range of networks and datasets.

Original: For example, our method achieved a 62% compression rate on ResNet50on ImageNet with 1 . 09% drop in accuracy.
Revised: For example, our method achieved a 62% compression rate on ResNet on ImageNet with 1 . 09% drop in accuracy.

Original:  Introduction and Backround
Revised:  Introduction and Backround

Original: Deep neural networks (DNNs) achieved state-of-the-art (SOTA) performance on a large variety oftasks, e g., in computer vision [33, 50] and natural language processing (NLP; [84, 20]).
Revised: Deep neural networks (DNNs) achieved state-of-the-art (SOTA) performance on a large variety of tasks, e g., in computer vision [33, 50] and natural language processing (NLP; [87, 20]).

Original: However,
Revised: However,

Original: DNNs usually contain millions or even billions of parameters in order to achieve SOTA performancesresulting in large storage requirements and long inference time.
Revised: DNNs usually contain millions or even billions of parameters in order to achieve SOTA performances resulting in large storage requirements and long inference time.

Original: This is obstructive when, e.g., dealingwith limited hardware or real-time systems such as autonomous cars and text/speech translation.
Revised: This is obstructive when, e g., dealing with limited hardware or real-time systems such as autonomous cars and text/speech translation.

Original: Tothis end, a large body of research is dedicated to reducing the size and inference costs of DNNs.
Revised: To this end, a large body of research is dedicated to reducing the size and inference costs of DNNs.

Original: Pruning.
Revised: Pruning.

Original: A dominant approach widely used for reducing the size of DNNs is to utilize a pruningalgorithm to remove redundant parameters from the original, over-parameterized network.
Revised: A dominant approach widely used for reducing the size of DNNs is to utilize a pruning algorithm to remove redundant parameters from the original, over-parameterized network.

Original: In general,pruning can be categorized into two main types: (i) Unstructured pruning [31, 6] reduces thenumber of non-zero parameters by inducing sparsity into weight parameters, which can achievehigh compression rates but requires specialized software and/or hardware in order to achieve fasterinference times.
Revised: In general, pruning can be categorized into two main types: (i) Unstructured pruning [31, 6] reduces the number of non-zero parameters by inducing sparsity into weight parameters, which can achieve high compression rates but requires specialized software and/or hardware in order to achieve faster inference times.

Original: (ii) Structured pruning [35, 60, 74] modifies the structure of the underlying weighttensors, by removing filters/neurons from each layer, usually resulting in smaller compression rateswhile directly achieving faster inference times with no specialized software; see section[-
-]
Revised: (ii) Structured pruning [35, 60, 77] modifies the structure of the underlying weight tensors, by removing filters/neurons from each layer, usually resulting in smaller compression rates while directly achieving faster inference times with no specialized software; see section[-
-]

Original: 1.1 Coresets for Pruning
Revised: 1.1 Coresets for Pruning

Original: Notably, many recent papers focused on various types of filter pruning [85, 76] potentially dueto the empirical observation that existing filter pruning approaches consistently yield impressiveresults.
Revised: Notably, many recent papers focused on various types of filter pruning [88, 79] potentially due to the empirical observation that existing filter pruning approaches consistently yield impressive results.

Original: However, most pruning methods are based on heuristics, lacking theoretical guarantees onthe trade-off between the compression rate and the approximation error.
Revised: However, most pruning methods are based on heuristics, lacking theoretical guarantees on the trade-off between the compression rate and the approximation error.

Original: This was the motive forintroducing coresets [79, 60] to the world of pruning.
Revised: This was the motive for introducing coresets [82, 60] to the world of pruning.

Original: Coresets.
Revised: Coresets.

Original: In machine learning, we are (usually) given an input set P ⊆
Revised: In machine learning, we are (usually) given an input set P ⊆

Original: R
Revised: R

Original: d of n points, itscorresponding weights function w : P → R , a feasible set of queries X , and a loss functionϕ : P × X → [0 , ∞ ) .
Revised: d of n points, its corresponding weights function w : P → R , a feasible set of queries X , and a loss function ϕ : P × X → [0 , ∞ ) .

Original: The tuple ( P, w, X, ϕ ) is called query space , and it defines the optimizationproblem at hand.
Revised: The tuple ( P, w, X, ϕ ) is called query space , and it defines the optimization problem at hand.

Original: For a given problem that is defined by its query space ( P, w, X, ϕ ) , and an errorparameter ε ∈ (0 , 1) , an ε -coreset is is a small weighted subset of the input points that approximatesthe loss of the input set P for every feasible query x , up to a provable bound of 1 + ε .
Revised: For a given problem that is defined by its query space ( P, w, X, ϕ ) , and an error parameter ε ∈ (0 , 1) , an ε -coreset is is a small weighted subset of the input points that approximates the loss of the input set P for every feasible query x , up to a provable bound of 1 + ε .

Original: Since coresets approximate the cost of every query, traditional (possibly inefficient) algorithms/solverscan be applied on coresets to obtain an approximation of the optimal solution on the full data, usingless time and memory; see Section B.2 in the appendix for more details.
Revised: Since coresets approximate the cost of every query, traditional (possibly inefficient) algorithms/solvers can be applied on coresets to obtain an approximation of the optimal solution on the full data, using less time and memory; see Section B.2 in the appendix for more details.

Original: Pruning via coresets.
Revised: Pruning via coresets.

Original: Recently, some inspiring innovative frameworks [79, 60, 5, 7] leveraged theidea of coresets for pruning DNNs.
Revised: Recently, some inspiring innovative frameworks [82, 60, 5, 7] leveraged the idea of coresets for pruning DNNs.

Original: Any layer ℓ can be represented as a set P = { p 1 , · · · , p n } ofn > 1 points in R d , where d is the number of input neurons, and n in the number of output neurons,i.e., each point p ∈ P , represents a specific neuron using its d weights (parameters).
Revised: Any layer ℓ can be represented as a set P = { p 1 , · · · , p n } of n > 1 points in R d , where d is the number of input neurons, and n in the number of output neurons, i.e., each point p ∈ P , represents a specific neuron using its d weights (parameters).

Original: When the layerreceives an input vector x ∈ R d , it outputs the vector ( ϕ ( p T 1 x ) , · · · , ϕ ( p Tn x )) , where ϕ : R → R isan activation function which defines a non-linear mapping.
Revised: When the layer receives an input vector x ∈ R d , it outputs the vector ( ϕ ( p T 1 x ) , · · · , ϕ ( p Tn x )) , where ϕ : R → R is an activation function which defines a non-linear mapping.

Original: Focusing on a single neuron η in thelayer that follows ℓ , defined by its corresponding vector of weights w = ( w 1 , · · · , w n ) , we set inthe context of coresets, w ( p i ) :
Revised: Focusing on a single neuron η in the layer that follows ℓ , defined by its corresponding vector of weights w = ( w 1 , · · · , w n ) , we set in the context of coresets, w ( p i ) :

Original: = w i for every i ∈ { 1 , · · · , n } - this is just a mapping from p i tow i to simplify the writing and reading.
Revised: = w i for every i ∈ { 1 , · · · , n } - this is just a mapping from p i to w i to simplify the writing and reading.

Original: Note that the output of this neuron is (cid:80) p ∈
Revised: Note that the output of this neuron is (cid:80) p ∈

Original: P w ( p ) ϕ
Revised: P w ( p ) ϕ

Original: (cid:0) p T x (cid:1) .
Revised: (cid:0) p T x (cid:1) .

Original: Assuming that we are given an ε -coreset
Revised: Assuming that we are given an ε -coreset

Original: ( C, u ) for the query space (cid:0) P, w, R d , ϕ (cid:1) where C ⊆ P , andu : C → R , we have that ( C, u ) approximates the output of this specific neuron η for every queryusing less parameters; see Figure 1.
Revised: ( C, u ) for the query space (cid:0) P, w, R d , ϕ (cid:1) where C ⊆ P , and u : C → R , we have that ( C, u ) approximates the output of this specific neuron η for every query using less parameters; see Figure 1.

Original: To formalize the stated above, we now define coresets in thecontext of activation functions.
Revised: To formalize the stated above, we now define coresets in the context of activation functions.

Original: Definition 1.1 (Coreset for activation functions) .
Revised: Definition 1.1 (Coreset for activation functions) .

Original: Let ε ∈
Revised: Let ε ∈

Original: (0 , 1) , and let (cid:0) P, w, R d , ϕ (cid:1) be a queryspace.
Revised: (0 , 1) , and let (cid:0) P, w, R d , ϕ (cid:1) be a query space.

Original: Then the pair ( C, u ) , is an ε -coreset for (cid:0) P, w, R d , ϕ (cid:1) if (
Revised: Then the pair ( C, u ) , is an ε -coreset for (cid:0) P, w, R d , ϕ (cid:1) if (

Original: i) C ⊆ P , (ii) u : C → [0 , ∞ ) , and(iii) for every x ∈ X ,(cid:12)(cid:12)(cid:12)(cid:12)
Revised: i) C ⊆ P , (ii) u : C → [0 , ∞ ) , and (iii) for every x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: − (cid:80) q [Equation] q ) ϕ ( q T
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:80) p ∈ P w [Equation] ≤ ε. Since C is a subset of P , we can remove (assign zero to) all weights from w that corresponds topoints not chosen to be in C from P , and replace the weights of the chosen points in C with the newweights vector u ; see Figure 1 in [79] for a visual illustration.
Revised: Since C is a subset of P , we can remove (assign zero to) all weights from w that corresponds to points not chosen to be in C from P , and replace the weights of the chosen points in C with the new weights vector u ; see Figure 1 in [82] for a visual illustration.

Original: To prune neurons, we refer the readerto Section 2.5 as it is a simple extension.
Revised: To prune neurons, we refer the reader to Section 2.5 as it is a simple extension.

Original: Prior work showed that such approaches successfully resultin high compression rates across a wide range of networks and datasets, and even achieves SOTAperformance on a verity of them.
Revised: Prior work showed that such approaches successfully result in high compression rates across a wide range of networks and datasets, and even achieves SOTA performance on a verity of them.

Original: The main (strong) advantage of the coreset approach over others was the provided provable theoreticalguarantees on the tradeoff between the compression rate and the approximation error, which supports worse case scenarios.
Revised: The main (strong) advantage of the coreset approach over others was the provided provable theoretical guarantees on the tradeoff between the compression rate and the approximation error, which supports worse case scenarios.

Original: In addition, coresets play an important role in improving the generalizationproperties of the trained networks [5, 75].
Revised: In addition, coresets play an important role in improving the generalization properties of the trained networks [5, 78].

Original: Sensitivity sampling for constructing pruning coresets.
Revised: Sensitivity sampling for constructing pruning coresets.

Original: To compute such coresets, both [60, 79]utilised the known sensitivity sampling framework [9, 52].
Revised: To compute such coresets, both [60, 82] utilised the known sensitivity sampling framework [9, 52].

Original: In short the sensitivity of a point p ∈ P insome query space ( P, w, X, ϕ ) corresponds to the importance of this point with respect to the queryspace at hand, and it is defined as s ( p ) =
Revised: In short the sensitivity of a point p ∈ P in some query space ( P, w, X, ϕ ) corresponds to the importance of this point with respect to the query space at hand, and it is defined as s ( p ) =

Original: sup x ∈ X w [Equation] p ) ϕ 
Revised: sup x ∈ X w ( p ) ϕ (

Original: [Equation]
Revised: [Equation]

Original: ,x )
Revised: ,x )

Original: [Equation]

Original: q

Original: [Equation]
Revised: [Equation]

Original:  w [Equation] q ) ϕ
Revised: q ∈ P w ( q ) ϕ

Original: ( q,x ) - where the denominator is notequal to zero.
Revised: ( q,x ) - where the denominator is not equal to zero.

Original: Once we bound these sensitivities, we can sample points (neurons) from P accordingto sensitivity bounds, and re-weight the sampled points to obtain a coreset.
Revised: Once we bound these sensitivities, we can sample points (neurons) from P according to sensitivity bounds, and re-weight the sampled points to obtain a coreset.

Original: The size of the sample isproportional to the sum of these bounds.
Revised: The size of the sample is proportional to the sum of these bounds.

Original: See Section B.1 and Theorem B.2 for more details in the
Revised: See Section B.1 and Theorem B.2 for more details in the

Original: Appendix.
Revised: Appendix.

Original: 1.2 Our contribution
Revised: 1.2 Our contribution

Original: Prior coreset methods for pruning DNNs either (i) imposed restrictive assumptions both on themodel’s weights and inputs [79], i.e., the input set P representing the neurons, and the query set Xwhich represents the inputs of the layer, are enclosed in a ball in R d of radius r 1 and r 2 , respectively,or (ii) the methods are data-dependent, i.e., use a mini-batch of the input set to measure the influenceof each parameter on the loss function [5, 60].
Revised: Prior coreset methods for pruning DNNs either (i) imposed restrictive assumptions both on the model’s weights and inputs [82], i.e., the input set P representing the neurons, and the query set X which represents the inputs of the layer, are enclosed in a ball in R d of radius r 1 and r 2 , respectively, or (ii) the methods are data-dependent, i.e., use a mini-batch of the input set to measure the influence of each parameter on the loss function [5, 60].

Original: To this end, in this work, we take coresets a step further into the realm of pruning by introducinga unified framework with provable guarantees for pruning DNNs (weights and neurons/filters)while minimally affecting the generalization error.
Revised: To this end, in this work, we take coresets a step further into the realm of pruning by introducing a unified framework with provable guarantees for pruning DNNs (weights and neurons/filters) while minimally affecting the generalization error.

Original: Our main improvement is that our framework issimultaneously (i) data-independent , (ii) requires a single assumption on the model’s weights,and (iii) provably guarantees a multiplicative factor approximation , which is favourable uponadditive approximations; see Theorem 2.6.
Revised: Our main improvement is that our framework is simultaneously (i) data-independent , (ii) requires a single assumption on the model’s weights, and (iii) provably guarantees a multiplicative factor approximation , which is favourable upon additive approximations; see Theorem 2.6.

Original: The approach is based on the widely used theory ofcoresets allowing us to suggest a provable guarantee on the tradeoff between the approximation errorand compression rate for each layer.
Revised: The approach is based on the widely used theory of coresets allowing us to suggest a provable guarantee on the tradeoff between the approximation error and compression rate for each layer.

Original: We conducted experimental results which established new SOTA benchmarks for structured pruningvia coresets across a wide range of networks and datasets.
Revised: We conducted experimental results which established new SOTA benchmarks for structured pruning via coresets across a wide range of networks and datasets.

Original: We share all of our resulted models [14].
Revised: We share all of our resulted models [14].

Original:  Method
Revised:  Method

Original: In general, the coreset (for pruning) technique hinges upon the insight that any linear layer such asconvolutions, can be casted as a matrix multiplication [79].
Revised: In general, the coreset (for pruning) technique hinges upon the insight that any linear layer such as convolutions, can be casted as a matrix multiplication [82].

Original: Hence, we focus in what follows on fullyconnected (FC) layers, while the details holds for any linear layer.
Revised: Hence, we focus in what follows on fully connected (FC) layers, while the details holds for any linear layer.

Original: Furthermore, for simplicity, weassume in what follows that the weights of P are all equal to 1 and thus our query space is denote by102 (cid:0)
Revised: Furthermore, for simplicity, we assume in what follows that the weights of P are all equal to 1 and thus our query space is denote by (cid:0)

Original: P, R d , ϕ (cid:1) , and the sensitivity of a point p ∈ P is simply s ( p ) =sup x ∈ X
Revised: P, R d , ϕ (cid:1) , and the sensitivity of a point p ∈ P is simply s ( p ) =

Original: [Equation] ϕ [Equation] q,x ) .
Revised: sup x ∈ X ϕ ( p,x ) (cid:80) q ∈ P ϕ ( q,x ) .

Original: Note that ourproofs are easily extended to the general case where we are given a weight function w : P → R asdiscussed in Section 2.5.
Revised: Note that our proofs are easily extended to the general case where we are given a weight function w : P → R as discussed in Section 2.5.

Original: 2.1 Preliminaries
Revised: 2.1 Preliminaries

Original: Notations.
Revised: Notations.

Original: For a positive integer n , we use [ n ] to denote the set { 1 , . . . , n } .
Revised: For a positive integer n , we use [ n ] to denote the set { 1 , . . . , n } .

Original: For c ∈ R d and a symmetric positive definite matrix G ∈ R d × d , we define E ( G, c ) :
Revised: For c ∈ R d and a symmetric positive definite matrix G ∈ R d × d , we define E ( G, c ) :

Original: = (cid:110) x ∈ R d (cid:12)(cid:12)(cid:12) ( x − c ) T G ( x − c ) ≤ 1 (cid:111)to be the ellipsoid defined by c and G .
Revised: = (cid:110) x ∈ R d (cid:12)(cid:12)(cid:12) ( x − c ) T G ( x − c ) ≤ 1 (cid:111) to be the ellipsoid defined by c and G .

Original: For an ellipsoid E ( G, c ) , each endpoint of a semi principalaxis is called a vertex of E ( G, c ) .
Revised: For an ellipsoid E ( G, c ) , each endpoint of a semi principal axis is called a vertex of E ( G, c ) .

Original: We define rank ( P ) for any set P ⊆
Revised: We define rank ( P ) for any set P ⊆

Original: R d to be the dimension ofthe affine subspace that P lies on.
Revised: R d to be the dimension of the affine subspace that P lies on.

Original: For a set P ⊂ R d the convex hull of P is denoted by Conv( P ) .
Revised: For a set P ⊂ R d the convex hull of P is denoted by Conv( P ) .

Original: Finally, vectors are treated as column vectors.
Revised: Finally, vectors are treated as column vectors.

Original: 2.2 Novelty - Löwner ellipsoid meets Carathéodory
Revised: 2.2 Novelty - Löwner ellipsoid meets Carathéodory

Original: Our method hinges upon a combination of two known tools from convex geometry.
Revised: Our method hinges upon a combination of two known tools from convex geometry.

Original: The noveltyof our approach exploits the following observation.
Revised: The novelty of our approach exploits the following observation.

Original: Most activation functions ϕ are continuousnon-decreasing functions, which indicate that for every query x and a set of points P , the maximalcontribution to (cid:80) p ∈ P ϕ
Revised: Most activation functions ϕ are continuous non-decreasing functions, which indicate that for every query x and a set of points P , the maximal contribution to (cid:80) p ∈ P ϕ

Original: ( p T x ) with respect to such activation function is associated to a point onthe convex hull of P .
Revised: ( p T x ) with respect to such activation function is associated to a point on the convex hull of P .

Original: By finding a geometrical body B of bounded number of vertices, that is (i) enclosed in Conv ( P ) and (ii) with some dilation factor (expanding) B enclose Conv ( P ) , we willbe able to represent each point p on the boundary of the convex hull of P as a convex combination oftwo points p 1 , p 2 , one of each ( p 1 ) on B and the second ( p 2 ) on its dilated form, which is formalizedas the set { α ( x − c )
Revised: By finding a geometrical body B of bounded number of vertices, that is (i) enclosed in Conv ( P ) and (ii) with some dilation factor (expanding) B enclose Conv ( P ) , we will be able to represent each point p on the boundary of the convex hull of P as a convex combination of two points p 1 , p 2 , one of each ( p 1 ) on B and the second ( p 2 ) on its dilated form, which is formalized as the set { α ( x − c )

Original: + c | x ∈ B, ∀ α ∈ [0 , 1] }, where c here denotes the center of B .
Revised: + c | x ∈ B, ∀ α ∈ [0 , 1] } where c here denotes the center of B .

Original: For such task,
Revised: For such task,

Original: Löwner ellipsoid is leveraged.
Revised: Löwner ellipsoid is leveraged.

Original: Theorem 2.1 (John-Löwner ellipsoid[42]) .
Revised: Theorem 2.1 (John-Löwner ellipsoid[42]) .

Original: Let
Revised: Let

Original: L ⊆ R
Revised: L ⊆ R

Original: d be a set of points such that the convexhull of L has a nonempty interior.
Revised: d be a set of points such that the convex hull of L has a nonempty interior.

Original: Then, there exists an ellipsoid E ( G, c ) (also known as the
Revised: Then, there exists an ellipsoid E ( G, c ) (also known as the

Original: MVEE ), where G ∈ R d × d is a positive definite matrix and c ∈ R d , of minimal volume such that1 d ( E ( G, c ) − c ) + c ⊆
Revised: MVEE ), where G ∈ R d × d is a positive definite matrix and c ∈ R d , of minimal volume such thatd ( E ( G, c ) − c ) + c ⊆

Original: Conv ( L ) ⊆
Revised: Conv ( L ) ⊆

Original: E ( G, c ) .
Revised: E ( G, c ) .

Original: If L is symmetric around the center c , then thedilation factor can be reduced to 1 √ d .
Revised: If L is symmetric around the center c , then the dilation factor can be reduced to 1 √ d .

Original: Algorithm
Revised: Algorithm

Original: Input : P ⊆ R d of n points with rank r
Revised: Input : P ⊆ R d of n points with rank r

Original: Output : A subset S ⊆ P that satisfies
Revised: Output : A subset S ⊆ P that satisfies

Original: Lemma 2.
Revised: Lemma 2.

Original: 1 ( Y, z ) := affine subspace that P lies on, [Equation] [Equation] 3 Let map : P ′ 
Revised: 1 ( Y, z ) := affine subspace that P lies on, i.e. P ⊆ (cid:8) xY Y T + z (cid:12)(cid:12) x ∈ R d (cid:9) 2 P ′ :

Original: → P a function that maps every p ′ ∈ P ′ to its corresponding point p[Equation] P 4 S [Equation]
Revised: = (cid:8) p

Original: ∅ [Equation] K [Equation]
Revised: = pY T (cid:9) 3 Let map : P ′

Original:  5 E [Equation] G, c [Equation]
Revised: → P a function that maps every p ′ ∈ P ′ to its corresponding point p ∈ P 4 S := ∅ ; K := ∅ 5 E ( G, c ) :

Original:  MVEE [Equation] Conv
Revised: = MVEE (Conv ( P ′ ))

Original: [Equation] P ′ )) 6 V := the vertices of ellipsoidr ( E ( G, c ) − c )
Revised: 6 V := the vertices of ellipsoidr ( E ( G, c ) − c )

Original: + c 7 for every v ∈ V do
Revised: + c 7 for every v ∈ V do

Original: K :=
Revised: K :=

Original: K ∪ C ARATHEODORY -S ET ( v, P ′ ) //
Revised: K ∪ C ARATHEODORY -S ET ( v, P ′ ) //

Original: See Algorithm 4 in the supplementary material.
Revised: See Algorithm 4 in the supplementary material.

Original: 9 end 10 S :
Revised: 9 end 10 S :

Original: = { map ( q ) | q ∈ K } 11 return S
Revised: = { map ( q ) | q ∈ K } 11 return S

Original: Algorithm  input : A set P ⊆
Revised: Algorithm 2: input : A set P ⊆

Original: R d of n points, and a sample size
Revised: R d of n points, and a sample size

Original: m output : A weighted set ( C, u ) 1 Q := P , i := 1 , C := ∅ 2 while | Q | ≥ 2rank ( Q ) 2 do
Revised: m output : A weighted set ( C, u ) 1 Q := P , i := 1 , C := ∅ 2 while | Q | ≥ 2rank ( Q ) 2 do

Original: S i := ℓ ∞ -C ORESET ( Q ) for every p ∈ S
Revised: S i := ℓ ∞ -C ORESET ( Q ) for every p ∈ S

Original: i do s ( p ) :
Revised: i do s ( p ) :

Original: = 2rank( Q ) 1 .
Revised: = 2rank( Q ) 1 .

Original: 5 i end
Revised: 5 i end

Original: Q :=
Revised: Q :=

Original: Q \ S i , i :=
Revised: Q \ S i , i :=

Original: i +8 end 9 for every p ∈ Q do s ( p ) :
Revised: i +8 end 9 for every p ∈ Q do s ( p ) :

Original: = 2rank( Q ) 1 .
Revised: = 2rank( Q ) 1 .

Original: 5 i 11 end 12 t := (cid:80)
Revised: 5 i 11 end 12 t := (cid:80)

Original: p ∈
Revised: p ∈

Original: P s ( p ) 13 C := an i.i.d sample of m points from P , where each p ∈ P is sampled with probability s ( p ) t .
Revised: P s ( p ) 13 C := an i.i.d sample of m points from P , where each p ∈ P is sampled with probability s ( p ) t .

Original: 14 u ( p ) :
Revised: 14 u ( p ) :

Original: = t m · s ( p ) for every p ∈ C 15 return ( C, u )

Original: Afterwards, p 1 and p 2 should be represented by points from P .
Revised: Afterwards, p 1 and p 2 should be represented by points from P .

Original: Each point on B (specifically, p 1 )can be represented via a convex combination of d + 1 points from P .
Revised: Each point on B (specifically, p 1 ) can be represented via a convex combination of d + 1 points from P .

Original: The same holds for points onthe dilated form of B (e.g., p 2 ) but via a conical combination (linear combination where the weightsare non-negative and the sum of weights is not necessarily 1 ).
Revised: The same holds for points on the dilated form of B (e.g., p 2 ) but via a conical combination (linear combination where the weights are non-negative and the sum of weights is not necessarily 1 ).

Original: This problem is solved by invoking
Revised: This problem is solved by invoking

Original: Carathéodory theorem.
Revised: Carathéodory theorem.

Original: regression loss function (shown in (ii)) using

Original: Definition 2.3.

Original: Following this step, a set of points P can be projected into a low dimensional subspace of dimension rank ( P ) using any dimensionality reduction algorithm as presented in (iii), resulting in the set P is its

Original: Theorem 2.2 ([10, 92]) .
Revised: Theorem 2.2 ([10, 95]) .

Original: For any A ⊂ R d and p ∈
Revised: For any A ⊂ R d and p ∈

Original: Conv( A ) , there exists m ≤ d + 1 pointsp 1 , . .
Revised: Conv( A ) , there exists m ≤ d + 1 points p 1 , . .

Original: , p m ∈ A (denoted by a Carathéodory set of p )
Revised: , p m ∈ A (denoted by a Carathéodory set of p )

Original: such that p ∈
Revised: such that p ∈

Original: Conv ( { p 1 , . .
Revised: Conv ( { p 1 , . .

Original: , p m } ) .
Revised: , p m } ) .

Original: Finally, it is known that some functions, including the ReLU function, do not admit an ε -coreset ofsize o ( n ) [79, 78].
Revised: Finally, it is known that some functions, including the ReLU function, do not admit an ε -coreset of size o ( n ) [82, 81].

Original: Thus, we use a generalized form of what is known as the complexity measure of aset of points, which was first introduced in [78] and later leveraged in [73].
Revised: Thus, we use a generalized form of what is known as the complexity measure of a set of points, which was first introduced in [81] and later leveraged in [76].

Original: This measure is used todetermine the complexity of a given set P with respect to ReLU, and the coreset size theoretically.
Revised: This measure is used to determine the complexity of a given set P with respect to ReLU, and the coreset size theoretically.

Original: Definition 2.3 (Regression Complexity Measure) .
Revised: Definition 2.3 (Regression Complexity Measure) .

Original: Let P ⊆
Revised: Let P ⊆

Original: R
Revised: R

Original: d × { 1 } , the regression complexitymeasure of P is defined as µ ( P ) =
Revised: d × { 1 } , the regression complexity measure of P is defined as µ ( P ) =

Original: sup x ∈ R 
Revised: sup x ∈ R d +

Original: [Equation]
Revised: [Equation]

Original: | , where the denominator is ≥ 0 ,and the last entry of every p ∈ P is 1 , reserved for the bias/intercept term.
Revised: | , where the denominator is ≥ 0 , and the last entry of every p ∈ P is 1 , reserved for the bias/intercept term.

Original: 2.3 Our Pruning Scheme
Revised: 2.3 Our Pruning Scheme

Original: In what follows, we present our data summarization technique for ReLU on the dot product function.
Revised: In what follows, we present our data summarization technique for ReLU on the dot product function.

Original: Then in Section 2.5, we discuss that our results can be easily extended to a wide family of activationfunctions including the Sigmoid function, as recently shown in [73].
Revised: Then in Section 2.5, we discuss that our results can be easily extended to a wide family of activation functions including the Sigmoid function, as recently shown in [76].

Original: First we present Algorithm 1,which serves as a stepping stone towards bounding the sensitivities.
Revised: First we present Algorithm 1, which serves as a stepping stone towards bounding the sensitivities.

Original: Overview of Algorithm 1.
Revised: Overview of Algorithm 1.

Original: The algorithm receives as input a set P ⊂ R
Revised: The algorithm receives as input a set P ⊂ R

Original: d whose rank is
Revised: d whose rank is

Original: r ∈ [ d ] anddeterministically finds a subset S ⊆ P which satisfies that for every j ∈ [ d ] , X ∈ R d × j and v ∈ R d ,[Figure]
Revised: r ∈ [ d ] and deterministically finds a subset S ⊆ P which satisfies that for every j ∈ [ d ] , X ∈ R d × j and v ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: ) X ∥

Original: [Equation]
Revised: [Equation]

Original:  To do so, first, we find the affine hyperplane that P lies on, followed bycomputing the low dimensional representation of P , denoted by P ′ ; see Lines 1–2.
Revised: ≤ r 1 . 5 . To do so, first, we find the affine hyperplane that P lies on, followed by computing the low dimensional representation of P , denoted by P ′ ; see Lines 1–2.

Original: Note that ifrank( P ) =
Revised: Note that if rank( P ) =

Original: d , then we can either keep P as it is (i.e., P ′ :
Revised: d , then we can either keep P as it is (i.e., P ′ :

Original: = P ), or use dimensionality reductiontricks as detailed in Section 2.5.
Revised: = P ), or use dimensionality reduction tricks as detailed in Section 2.5.

Original: To compute the output S , we first bound the convex hull of P ′ by its
Revised: To compute the output S , we first bound the convex hull of P ′ by its

Original: Löwner ellipsoid E ( G, c ) in Line 5, followed by computing the dilated ellipsoid of E ( G, c ) , namely,1 r ( E − c ) + c .
Revised: Löwner ellipsoid E ( G, c ) in Line 5, followed by computing the dilated ellipsoid of E ( G, c ) , namely,r ( E − c ) + c .

Original: Let V be the set of vertices of such ellipsoid; see Line 6.
Revised: Let V be the set of vertices of such ellipsoid; see Line 6.

Original: Now, for each point v ∈ V ,we represent it as a convex combination of r + 1 points from P ′ via Theorem 2.2, and store the unionof such sets (each of size at most r + 1 ) of points into K as done in Lines 7–9.
Revised: Now, for each point v ∈ V , we represent it as a convex combination of r + 1 points from P ′ via Theorem 2.2, and store the union of such sets (each of size at most r + 1 ) of points into K as done in Lines 7–9.

Original: For each point in K ,we map it back to R d to satisfy Lemma 2.5.
Revised: For each point in K , we map it back to R d to satisfy Lemma 2.5.

Original: To sum up Algorithm 1, we observe that the vertices canbe used via canonical combinations with their dilated form to describe every point on the convex hullof the input data (in our end, it would the network’s weights).
Revised: To sum up Algorithm 1, we observe that the vertices can be used via canonical combinations with their dilated form to describe every point on the convex hull of the input data (in our end, it would the network’s weights).

Original: Hence the Carathéodory set of thesevertices from the input points lying on the convex hull can be further used to also represent pointslying on the convex hull.
Revised: Hence the Carathéodory set of these vertices from the input points lying on the convex hull can be further used to also represent points lying on the convex hull.

Original: This is the core idea which enable us in forming our ℓ ∞ coreset for any ℓ ρregression problem where ρ ∈ (0 , ∞ ) .
Revised: This is the core idea which enable us in forming our ℓ ∞ coreset for any ℓ ρ regression problem where ρ ∈ (0 , ∞ ) .

Original: We now discuss Algorithm 2 which is responsible for constructing an ε -coreset with respect toactivation functions.
Revised: We now discuss Algorithm 2 which is responsible for constructing an ε -coreset with respect to activation functions.

Original: Its input is a set P ⊂ R d and a sample size m ≥ 1 .
Revised: Its input is a set P ⊂ R d and a sample size m ≥ 1 .

Original: Overview of Algorithm 2.
Revised: Overview of Algorithm 2.

Original: First set Q := P .
Revised: First set Q := P .

Original: At each iteration i , the algorithm obtains a subset
Revised: At each iteration i , the algorithm obtains a subset

Original: S i ⊆ Q as an output to a call to ℓ ∞ -C ORESET ( Q ) as stated in Line 3 of Algorithm [-
-] such that forevery x ∈ R d , it holds that [Equation]
Revised: S i ⊆ Q as an output to a call to ℓ ∞ -C ORESET ( Q ) as stated in Line 3 of Algorithm [-
-]1 such that for every x ∈ R d , it holds that max p ∈ P | p T x | max p ∈ S i |

Original:  ≤ r 1 . 5 with r being the rank of Q .
Revised: p T x | ≤ r 1 . 5 with r being the rank of Q .

Original: The sensitivity of eachpoint in S i is bounded from above by 2 d 1 .
Revised: The sensitivity of each point in S i is bounded from above by 2 d 1 .

Original: i as stated in Lines 4–6.
Revised: i as stated in Lines 4–6.

Original: The idea behind these boundslies in our proof of Theorem 2.6.
Revised: The idea behind these bounds lies in our proof of Theorem 2.6.

Original: The set S i is removed from Q , and this procedure is repeated withrespect to Q until the size of Q is small enough.
Revised: The set S i is removed from Q , and this procedure is repeated with respect to Q until the size of Q is small enough.

Original: The obtained sensitivities are the ones needed forcomputing the pruning coresets.
Revised: The obtained sensitivities are the ones needed for computing the pruning coresets.

Original: Finally, we utilize the sensitivity sampling framework of [9] toobtain the desired coreset; see Lines 13–14.
Revised: Finally, we utilize the sensitivity sampling framework of [9] to obtain the desired coreset; see Lines 13–14.

Original: 2.4 Analysis
Revised: 2.4 Analysis

Original: In this section, we prove the correctness of our algorithms.
Revised: In this section, we prove the correctness of our algorithms.

Original: The following lemma shows that for eachpoint p that is inside some convex hull S , its ℓ 1 distance to any affine subspace is always boundedfrom above by ℓ 1 distance from the same affine subspace, of some other point q ∈ S .
Revised: The following lemma shows that for each point p that is inside some convex hull S , its ℓ 1 distance to any affine subspace is always bounded from above by ℓ 1 distance from the same affine subspace, of some other point q ∈ S .

Original: Lemma 2.4.
Revised: Lemma 2.4.

Original: Let d, ℓ, m ≥ 1 be integers.
Revised: Let d, ℓ, m ≥ 1 be integers.

Original: Let p ∈ R d and A ⊆ R d be a set of m points withp ∈ Conv( A ) so that there exists α : A → [0 , 1] such that (cid:80) q∈
Revised: Let p ∈ R d and A ⊆ R d be a set of m points with p ∈ Conv( A ) so that there exists α : A → [0 , 1] such that (cid:80) q

Revised: ∈

Original: A α ( q ) = 1 and (cid:80) q ∈
Revised: A α ( q ) = 1 and (cid:80) q ∈

Original: A α ( q ) · q = p .
Revised: A α ( q ) · q = p .

Original: Then for every Y ∈ R d × ℓ and v ∈ R ℓ , ∥ ( p − v ) Y ∥ 1 ≤ max q ∈
Revised: Then for every Y ∈ R d × ℓ and v ∈ R ℓ , ∥ ( p − v ) Y ∥ 1 ≤ max q ∈

Original: A ∥ ( q − v ) Y ∥ 1 .
Revised: A ∥ ( q − v ) Y ∥ 1 .

Original: The following states the provable guarantees of Algorithm 1.
Revised: The following states the provable guarantees of Algorithm 1.

Original: Lemma 2.5 ( ℓ ∞ -coreset for ℓ 1 -regression) .
Revised: Lemma 2.5 ( ℓ ∞ -coreset for ℓ 1 -regression) .

Original: Let P ⊆
Revised: Let P ⊆

Original: R d be a set of points, and r be the rank of P .
Revised: R d be a set of points, and r be the rank of P .

Original: Let j ∈ [ d − 1] and let S be the output of a call to ℓ ∞ -C ORESET ( P ) .
Revised: Let j ∈ [ d − 1] and let S be the output of a call to ℓ ∞ -C ORESET ( P ) .

Original: Then (i) | S | ∈ O (cid:0) r 2 (cid:1) , and(ii) for every X ∈ R d × j and v ∈ R d , max q ∈ P ∥ ( q − v )
Revised: Then (i) | S | ∈ O (cid:0) r 2 (cid:1) , and (ii) for every X ∈ R d × j and v ∈ R d , max q ∈ P ∥ ( q − v )

Original: X ∥ 1 max q ∈ S ∥ ( q − v ) X ∥ 1 ∈ (cid:2) 1 , 2 r 1 .
Revised: X ∥ 1 max q ∈ S ∥ ( q − v ) X ∥ 1 ∈ (cid:2) 1 , 2 r 1 .

Original: 5 (cid:3) .
Revised: 5 (cid:3) .

Original: The following theorem states our main result.
Revised: The following theorem states our main result.

Original: Theorem 2.6 (ReLU ε -coreset) .
Revised: Theorem 2.6 (ReLU ε -coreset) .

Original: Let P ⊆ R d , ε, δ ∈ (0 , 1) , r = rank ( P ) , µ ( P ) be as in Definition 2.3, and let m ∈ O (cid:16) µ ( P ) r 3 .
Revised: Let P ⊆ R d , ε, δ ∈ (0 , 1) , r = rank ( P ) , µ ( P ) be as in Definition 2.3, and let m ∈ O (cid:16) µ ( P ) r 3 .

Original: 5 log n ε(cid:0) d (log ( µ ( P ) r log n ))
Revised: 5 log n ε(cid:0) d (log ( µ ( P ) r log n ))

Original: + log (cid:0) 1 δ (cid:1)(cid:1)(cid:17) .
Revised: + log (cid:0) 1 δ (cid:1)(cid:1)(cid:17) .

Original: Let ( C, w ) be theoutput of a call to C ORESET ( P, m ) ; see Algorithm 2.
Revised: Let ( C, w ) be the output of a call to C ORESET ( P, m ) ; see Algorithm 2.

Original: Then, with probability at least 1 − δ , ( C, w ) isan ε -coreset for (cid:0) P, R d , ReLU ( · ) (cid:1) ; see Definition 1.1.
Revised: Then, with probability at least 1 − δ , ( C, w ) is an ε -coreset for (cid:0) P, R d , ReLU ( · ) (cid:1) ; see Definition 1.1.

Original: Time analysis.
Revised: Time analysis.

Original: Letting r be the rank of P , the time complexity of Algorithm 1 can be dissectedto two main parts: (i)
Revised: Letting r be the rank of P , the time complexity of Algorithm 1 can be dissected to two main parts: (i)

Original: Computing the Löwner ellipsoid in O ( nr 2 log n ) time using the methodproposed in [95] and (ii) computing the Carathédory set in O ( nr + r 4 log n ) time via [69].
Revised: Computing the Löwner ellipsoid in O ( nr 2 log n ) time using the method proposed in [98] and (ii) computing the Carathédory set in O ( nr + r 4 log n ) time via [70].

Original: Since Vcan contain up to O ( r 2 ) points, the overall time for Algorithm 1 is O ( nr 2 log n + nr 3 + r 6 log n ) .
Revised: Since V can contain up to O ( r 2 ) points, the overall time for Algorithm 1 is O ( nr 2 log n + nr 3 + r 6 log n ) .

Original: As for Algorithm 2, it takes O ( n 2 r 2 log n + nr 4 log n ) .
Revised: As for Algorithm 2, it takes O ( n 2 r 2 log n + nr 4 log n ) .

Original: Indeed, as explained in Section 2.5, adimensionality reduction algorithm may be applied to improve the run time (reducing the r 6 factor).
Revised: Indeed, as explained in Section 2.5, a dimensionality reduction algorithm may be applied to improve the run time (reducing the r 6 factor).

Original: Furthermore, the run time of our algorithm can be improved, using the merge-and-reduce tree fromthe literature of coresets to reduce the n 2 terms to n log n , i.e., the running time can be reduced to
Revised: Furthermore, the run time of our algorithm can be improved, using the merge-and-reduce tree from the literature of coresets to reduce the n 2 terms to n log n , i.e., the running time can be reduced to

Original: O ( n log 2 ( n ) r 2 + nr 4 ) .
Revised: O ( n log 2 ( n ) r 2 + nr 4 ) .

Original: For a data-independent provable method, this running time is reasonable.
Revised: For a data-independent provable method, this running time is reasonable.

Original: Our advantages over previous results.
Revised: Our advantages over previous results.

Original: Our coreset supports different activation functions withoutthe need to change the sensitivity that much.
Revised: Our coreset supports different activation functions without the need to change the sensitivity that much.

Original: Specifically, it will only be multiplied by some scalar,unlike previous coresets where different losses impose drastically different sensitivities/leveragescores and algorithms.
Revised: Specifically, it will only be multiplied by some scalar, unlike previous coresets where different losses impose drastically different sensitivities/leverage scores and algorithms.

Original: This is since our coreset unlike other coresets is in its essence a frameworkof coresets for different ℓ ρ losses, as it can be used as is for different ℓ ρ losses and yet still attainϵ -approximation.
Revised: This is since our coreset unlike other coresets is in its essence a framework of coresets for different ℓ ρ losses, as it can be used as is for different ℓ ρ losses and yet still attain ϵ -approximation.

Original: In addition, when the rank of the input points is small, then our method outperformsprevious methods.
Revised: In addition, when the rank of the input points is small, then our method outperforms previous methods.

Original: If the input data is of full rank, previous methods obtain faster coresets construction.
Revised: If the input data is of full rank, previous methods obtain faster coresets construction.

Original: On the boundness of the regression complexity measure.
Revised: On the boundness of the regression complexity measure.

Original: First of all, there exists an examplewhere the complexity measure is unbounded, e.g., consider a set of points distributed evenly ona unit ball.
Revised: First of all, there exists an example where the complexity measure is unbounded, e.g., consider a set of points distributed evenly on a unit ball.

Original: In this case, you can always find a point where a hyperplane separating it from therest of the points can be found such that the one half-space of this hyperplane contains only thispoint while the other half-space contains the rest of the points.
Revised: In this case, you can always find a point where a hyperplane separating it from the rest of the points can be found such that the one half-space of this hyperplane contains only this point while the other half-space contains the rest of the points.

Original: This leads to an infinite complexitymeasure.
Revised: This leads to an infinite complexity measure.

Original: Such an example is also mentioned in [79], when assessing the hardness of generatingmultiplicative-approximation coresets for ReLU functions.
Revised: Such an example is also mentioned in [82], when assessing the hardness of generating multiplicative-approximation coresets for ReLU functions.

Original: Theoretically, the complexity measure is influenced by how free can the bias term be (the last entry ofx); see Definition 2.3.
Revised: Theoretically, the complexity measure is influenced by how free can the bias term be (the last entry of x); see Definition 2.3.

Original: This term is the only thing that can ensure that one point can be separated fromthe rest in the sense of finding a separating hyperplane, leading to an infinite complexity measure.
Revised: This term is the only thing that can ensure that one point can be separated from the rest in the sense of finding a separating hyperplane, leading to an infinite complexity measure.

Original: Bounding on this term, leads to bounded complexity measure from a theoretical point of view.
Revised: Bounding on this term, leads to bounded complexity measure from a theoretical point of view.

Original: In the context of model pruning, from the perspective of the complexity measure, the model’s weightsare the input denoted by a matrix P ∈ R n × ( d +1) , while the query is now R d × { 1 } .
Revised: In the context of model pruning, from the perspective of the complexity measure, the model’s weights are the input denoted by a matrix P ∈ R n × ( d +1) , while the query is now R d × { 1 } .

Original: Thus thecomplexity measure is now µ ( P ) :
Revised: Thus the complexity measure is now µ ( P ) :

Original: = sup x ∈ R d ×{ 1 } (cid:80) [Equation] [Equation] | 
Revised: = sup x ∈ R d ×{ 1 } (cid:80) q ∈ { p ∈ P | pTx ≤ 0 } | q T x | (cid:80) q ∈ { p ∈ P | pTx> 0 }| q T x | .

Original: [Equation]

Original: [Equation]

Original: x |

Original: [Equation]

Original: q ∈ { p

Original: [Equation]

Original: P | pTx> 0 }| q T x | .

Original: With this in mind, weobserve that the complexity measure is now an instance of the complexity measure used in [73].
Revised: With this in mind, we observe that the complexity measure is now an instance of the complexity measure used in [76].

Original: Thecomplexity measure now relies entirely on the structure of the model’s weights, where the goal is tofind the largest ratio between the sum of the absolute of the values inside the rectified neurons priorto applying the rectification, and the sum values of non-rectified neurons.
Revised: The complexity measure now relies entirely on the structure of the model’s weights, where the goal is to find the largest ratio between the sum of the absolute of the values inside the rectified neurons prior to applying the rectification, and the sum values of non-rectified neurons.

Original: To bound this measure, wecan use a variant of the algorithm described in the proof of Theorem 3 in [78].
Revised: To bound this measure, we can use a variant of the algorithm described in the proof of Theorem 3 in [81].

Original: 2.5 Extensions
Revised: 2.5 Extensions

Original: Our suggested scheme can be extended to support many other variants of the pruning problem.
Revised: Our suggested scheme can be extended to support many other variants of the pruning problem.

Original: Various activation functions.
Revised: Various activation functions.

Original: Our result can be extended to a family of activation functions called“Nice hinge functions”; see Definition D.1.
Revised: Our result can be extended to a family of activation functions called “Nice hinge functions”; see Definition D.1.

Original: Let ( P, X, ϕ ) be a query space, where ϕ is a “Nice hingefunctions”.
Revised: Let ( P, X, ϕ ) be a query space, where ϕ is a “Nice hinge functions”.

Original: To bound the sensitivity of a point p , we first bound the nominator of s ( p ) by provingthat ∀ x ∈ X : ϕ ( p T x ) < (cid:12)(cid:12) p T x (cid:12)(cid:12) .
Revised: To bound the sensitivity of a point p , we first bound the nominator of s ( p ) by proving that ∀ x ∈ X : ϕ ( p T x ) < (cid:12)(cid:12) p T x (cid:12)(cid:12) .

Original: For bounding the denominator from below, recently [73] provedthat ∀ x ∈ X : (cid:80) p ∈ P ReLU (cid:0) p T x (cid:1) ⪅ (cid:80) p ∈ P ϕ ( p T x ) ; see full detail in Section D.3.
Revised: For bounding the denominator from below, recently [76] proved that ∀ x ∈ X : (cid:80) p ∈ P ReLU (cid:0) p T x (cid:1) ⪅ (cid:80) p ∈ P ϕ ( p T x ) ; see full detail in Section D.3.

Original: Weighted Input.
Revised: Weighted Input.

Original: In the context of deep learning, the output of each neuron is multiplied with a scalarwhich brings the necessity of having the ability to deal with weighted set of points.
Revised: In the context of deep learning, the output of each neuron is multiplied with a scalar which brings the necessity of having the ability to deal with weighted set of points.

Original: Algorithm 2 canbe extended easily to the case as generously detailed in Section D.1 in the Appendix.
Revised: Algorithm 2 can be extended easily to the case as generously detailed in Section D.1 in the Appendix.

Original: Dimensionality reduction.
Revised: Dimensionality reduction.

Original: All coreset-based pruning methods rely heavily on the dimensionality ofthe model’s layers, as well as our method.
Revised: All coreset-based pruning methods rely heavily on the dimensionality of the model’s layers, as well as our method.

Original: To ensure sufficient pruning ratio, we apply either PCA,
Revised: To ensure sufficient pruning ratio, we apply either PCA,

Original: TSNE, MDS, or the JL transform on the weights of each layer prior to generating its coreset.
Revised: TSNE, MDS, or the JL transform on the weights of each layer prior to generating its coreset.

Original: From weight to neuron pruning.
Revised: From weight to neuron pruning.

Original: Most coreset-based pruning methods, e.g., [5, 79], first provide ascheme for (provable) weight pruning, which is then used as a stepping stone towards pruning neuronsas follows.
Revised: Most coreset-based pruning methods, e.g., [5, 82], first provide a scheme for (provable) weight pruning, which is then used as a stepping stone towards pruning neurons as follows.

Original: [5] first suggested coreset-based neuron pruning via the use of a generated controled set ofqueries to evaluate the importance of weights.
Revised: [5] first suggested coreset-based neuron pruning via the use of a generated controled set of queries to evaluate the importance of weights.

Original: Any neuron that has a maximal activation value loweror equal to zero, will be pruned from the network as its impact on the rest of the neurons is minimal.
Revised: Any neuron that has a maximal activation value lower or equal to zero, will be pruned from the network as its impact on the rest of the neurons is minimal.

Original: On the other hand [79] altered the definition of sensitivity such that it takes into account the sensitivityof a neuron in a layer ℓ with respect to all the neurons in the layer ℓ + 1 , which basically meansthat the sensitivity of each neuron is taken be the maximal sensitivity over every weight function(neuron in the next layer) defined by the layer.
Revised: On the other hand [82] altered the definition of sensitivity such that it takes into account the sensitivity of a neuron in a layer ℓ with respect to all the neurons in the layer ℓ + 1 , which basically means that the sensitivity of each neuron is taken be the maximal sensitivity over every weight function (neuron in the next layer) defined by the layer.

Original: Hence, we follow the same logic for such method; see
Revised: Hence, we follow the same logic for such method; see

Original: Section D.2 in the supplementary material.
Revised: Section D.2 in the supplementary material.

Original: From neuron to filter pruning.
Revised: From neuron to filter pruning.

Original: Convolutions can be expressed as matrix multiplications, whichenables our method to prune filters from any model as done by [60, 79, 61].
Revised: Convolutions can be expressed as matrix multiplications, which enables our method to prune filters from any model as done by [60, 82, 61].

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we study various widely used network architectures and benchmark data-sets.
Revised: In this section, we study various widely used network architectures and benchmark data-sets.

Original: Following [79], to test the robustness of our methods on each of the neuron and filter pruning tasksindependently, two sets of experiments are conducted.
Revised: Following [82], to test the robustness of our methods on each of the neuron and filter pruning tasks independently, two sets of experiments are conducted.

Original: The first focuses on pruning neurons (Section 3.1) whereas the second focuses on pruning filters (Section 3.2), both via our coreset method.
Revised: The first focuses on pruning neurons (Section 3.1) whereas the second focuses on pruning filters (Section 3.2), both via our coreset method.

Original: The setting.
Revised: The setting.

Original: In all experiments we report the Pruning ratio – the percentage of the parameters thatwere removed from the original mode.
Revised: In all experiments we report the Pruning ratio – the percentage of the parameters that were removed from the original mode.

Original: Here, PR stands for pruning ratio, FR stands for floating-pointreduction ratio and Err – the percentage of misclassified test instances of our method compared tocoreset-based pruning methods and more.
Revised: Here, PR stands for pruning ratio, FR stands for floating-point reduction ratio and Err – the percentage of misclassified test instances of our method compared to coreset-based pruning methods and more.

Original: Baseline Err is the error of the original uncompressednetwork, while Pruned Err is the classification error of the compressed model.
Revised: Baseline Err is the error of the original uncompressed network, while Pruned Err is the classification error of the compressed model.

Original: In our experimentwe compress and fine-tune the network once, no iterative pruning was applied, thus, the comparedmethods also satisfy this setting.
Revised: In our experiment we compress and fine-tune the network once, no iterative pruning was applied, thus, the compared methods also satisfy this setting.

Original: Each experiment was conducted 5 times, in the tables, we reportfor our method the best error achieved and we highlight in parentheses next to it the average errorand standard deviation across the 5 trails.
Revised: Each experiment was conducted 5 times, in the tables, we report for our method the best error achieved and we highlight in parentheses next to it the average error and standard deviation across the 5 trails.

Original: In all of our experiments, the models are fine-tuned tillconvergence (after pruning).
Revised: In all of our experiments, the models are fine-tuned till convergence (after pruning).

Original: Implementation details are given in Section E in the Appendix.
Revised: Implementation details are given in Section E in the Appendix.

Original: Software/Hardware.
Revised: Software/Hardware.

Original: Our algorithms were implemented in Python 3.6 [103] using Numpy [80], and
Revised: Our algorithms were implemented in Python 3.6 [108] using Numpy [83], and

Original: Pytorch [81].
Revised: Pytorch [84].

Original: Tests were performed on NVIDIA DGX A100 servers with 8 NVIDIA A100 GPUseach, fast InfiniBand interconnect and supporting infrastructure.
Revised: Tests were performed on NVIDIA DGX A100 servers with 8 NVIDIA A100 GPUs each, fast InfiniBand interconnect and supporting infrastructure.

Original: Baselines.
Revised: Baselines.

Original: Our results are compared to (i) PFP [60], (ii) FT [58], (iii) SoftNet [34], (iv) ThiNet [68],and (v) PvC [79], (vi) Soft Pruning [34], (vii) CCP [82], (viii)
Revised: Our results are compared to (i) PFP [60], (ii) FT [58], (iii) SoftNet [34], (iv) ThiNet [68], and (v) PvC [82], (vi) Soft Pruning [34], (vii) CCP [85], (viii)

Original: FPGM [36], (ix) ThiNet-70, (x)
Revised: FPGM [36], (ix) ThiNet-70, (x)

Original: ThiNet-50 [68], (xi) Pruning via Coresets (PvC) [79], (xii)
Revised: ThiNet-50 [68], (xi) Pruning via Coresets (PvC) [82], (xii)

Original: Pruning from Scratch (PfS) [106], and(xiii)
Revised: Pruning from Scratch (PfS) [111], and (xiii)

Original: Rethinking the value of network pruning (Rethink) [65].
Revised: Rethinking the value of network pruning (Rethink) [65].

Original: 3.1 Neuron Pruning
Revised: 3.1 Neuron Pruning

Original: We test our method on VGG16 [89] using CIFAR10 [49], and LeNet300-100 using MNIST [55].
Revised: We test our method on VGG16 [92] using CIFAR10 [49], and LeNet300-100 using MNIST [55].

Original: Discussion.
Revised: Discussion.

Original: Table 1 present the results of LeNet 300 - 100 and VGG 16 .
Revised: Table 1 present the results of LeNet 300 - 100 and VGG 16 .

Original: Observe that in both architectures, our method outperformed the competing methods under the same compression scenarios.
Revised: Observe that in both architectures, our method outperformed the competing methods under the same compression scenarios.

Original: Forexample, we pruned roughly 90% of the parameters of the LeNet-300-100 model while improvingthe accuracy of the original model.
Revised: For example, we pruned roughly 90% of the parameters of the LeNet-300-100 model while improving the accuracy of the original model.

Original: We witness a similar phenomena on the VGG16 model, where wepruned roughly 90% of the parameters of the dense layers resulting in accuracy improvement.
Revised: We witness a similar phenomena on the VGG16 model, where we pruned roughly 90% of the parameters of the dense layers resulting in accuracy improvement.

Original: Thisconfirms the insights in [5] that coresets help in improving the generalization properties of DNNs.
Revised: This confirms the insights in [5] that coresets help in improving the generalization properties of DNNs.

Original: [Table]
Revised: [Table]

Original: 3.2 Filter pruning
Revised: 3.2 Filter pruning

Original: We compressed the convolutional layers of (i) ResNet 50 [33] on ILSVRC- 2012 [18], (ii)
Revised: We compressed the convolutional layers of (i) ResNet 50 [33] on ILSVRC- 2012 [18], (ii)

Original: ResNet 56 [33], (iii) VGG 19 [89] on CIFAR 10 and (iv) VGG 16 [89] on CIFAR 10 .
Revised: ResNet 56 [33], (iii) VGG 19 [92] on CIFAR 10 and (iv) VGG 16 [92] on CIFAR 10 .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: ThiNet27 .28 .51 .55 .
Revised: ThiNet27 .28 .51 .55 .

Original: PvC 23 .25 .
Revised: PvC 23 .25 .

Original: N/A
Revised: N/A

Original: Our method 23 .24 . 87 ( 25 . 07 ± 0 . 2 ) 61 .
Revised: Our method 23 .24 . 87 ( 25 . 07 ± 0 . 2 ) 61 .

Revised: Our method ( 18 . 01 ) 23 .24 . 1 ( 24 . 2 ± 0 . 1 ) 18 .10 .

Original: Filter pruning of DNNs on CIFAR 10 and ImageNet (ILSVRC- 2012 ).
Revised: Filter pruning of DNNs on CIFAR 10 and ImageNet (ILSVRC- 2012 ).

Original: For Cifar 10 , we used
Revised: For Cifar 10 , we used

Original: PyTorch implementations of VGG 19 and VGG 16 .
Revised: PyTorch implementations of VGG 19 and VGG 16 .

Original: We compressed both models using our approachby different compression rates as shown in Table 2 with a comparison to other methods.
Revised: We compressed both models using our approach by different compression rates as shown in Table 2 with a comparison to other methods.

Original: For
Revised: For

Original: ImageNet, we compressed the baseline model of ResNet 50 [33] roughly by 62% in terms of numberof parameters.
Revised: ImageNet, we compressed the baseline model of ResNet 50 [33] roughly by 62% in terms of number of parameters.

Original: Table 3 provides comparison between our method and other baselines.
Revised: Table 3 provides comparison between our method and other baselines.

Original: Discussion.
Revised: Discussion.

Original: As can be seen in Tables 1, 2, and 3, our method either outperforms the competingmethods or achieves comparable results.
Revised: As can be seen in Tables 1, 2, and 3, our method either outperforms the competing methods or achieves comparable results.

Original: As for the coreset methods, our algorithm achieves betterresult than all of them in this setting, e.g., we compressed 62% of ResNet50 trained on ImageNetwhile incurring 1 . 09% drop in accuracy, improving the recent coreset result of PvC [79] for the samecompression ratio, while PFP [60] compressed 44 .
Revised: As for the coreset methods, our algorithm achieves better result than all of them in this setting, e.g., we compressed 62% of ResNet50 trained on ImageNet while incurring 1 . 09% drop in accuracy, improving the recent coreset result of PvC [82] for the same compression ratio, while PFP [60] compressed 44 .

Original: 04% to achieve comparable results.
Revised: 04% to achieve comparable results.

Original:  Related work
Revised:  Related work

Original: DNNs can be compressed before training [94, 105, 57], during training [113, 110, 51], or aftertraining [90].
Revised: DNNs can be compressed before training [97, 110, 57], during training [118, 115, 51], or after training [93].

Original: Furthermore, such procedures may also be repeated iteratively [85].
Revised: Furthermore, such procedures may also be repeated iteratively [88].

Original: As previouslynoted pruning can be categorized into structured and unstructured pruning.
Revised: As previously noted pruning can be categorized into structured and unstructured pruning.

Original: Unstructured pruning.
Revised: Unstructured pruning.

Original: Weight pruning [56] techniques aim to reduce the number of weightsin a layer while approximately preserving its output.
Revised: Weight pruning [56] techniques aim to reduce the number of weights in a layer while approximately preserving its output.

Original: Approaches of this type include the worksof [54, 21, 39, 1, 62], where the desired sparsity is embedded as a constraint or via a regularizer intothe training pipeline, and those of [31, 85, 30], where weights with absolute values below a thresholdare removed.
Revised: Approaches of this type include the works of [54, 21, 39, 1, 62], where the desired sparsity is embedded as a constraint or via a regularizer into the training pipeline, and those of [31, 88, 30], where weights with absolute values below a threshold are removed.

Original: The approaches of [5, 6] use a mini-batch of data points to approximate the influence ofeach parameter on the loss function.
Revised: The approaches of [5, 6] use a mini-batch of data points to approximate the influence of each parameter on the loss function.

Original: Other data-informed techniques include [28, 63, 77, 76, 111].
Revised: Other data-informed techniques include [28, 63, 80, 79, 116].

Original: Athorough overview of recent pruning approaches is given by [27, 8].
Revised: A thorough overview of recent pruning approaches is given by [27, 8].

Original: However, unlike our approach,weight-based pruning approaches generate sparse models instead of smaller ones thus requiringspecialized hardware and sparse linear algebra libraries in order to speed up inference.
Revised: However, unlike our approach, weight-based pruning approaches generate sparse models instead of smaller ones thus requiring specialized hardware and sparse linear algebra libraries in order to speed up inference.

Original: Structured pruning.
Revised: Structured pruning.

Original: Pruning entire neurons and filters directly shrinks the network leading to smallerstorage requirements and improved inference-time performance on any hardware [59, 67].
Revised: Pruning entire neurons and filters directly shrinks the network leading to smaller storage requirements and improved inference-time performance on any hardware [59, 67].

Original: Lately,these approaches were investigated in many papers [64, 59, 11, 36, 22, 46, 109, 108].
Revised: Lately, these approaches were investigated in many papers [64, 59, 11, 36, 22, 46, 114, 113].

Original: Usually, filtersare pruned by assigning an importance score to each neuron/filter, either solely weight-based [37, 35]or data-informed [111, 60], and removing those with a score below a threshold.
Revised: Usually, filters are pruned by assigning an importance score to each neuron/filter, either solely weight-based [37, 35] or data-informed [116, 60], and removing those with a score below a threshold.

Original: The procedure canbe embedded into an iterative pruning scheme [85] that requires potentially expensive retrain cycles.
Revised: The procedure can be embedded into an iterative pruning scheme [88] that requires potentially expensive retrain cycles.

Original: Tensor decomposition.
Revised: Tensor decomposition.

Original: Some of the work in DNN compression entails decomposing the layerinto multiple smaller ones, e.g., via low-rank tensor decomposition [19, 41, 72, 48, 93, 40, 2, 101,112 , 53, 61].
Revised: Some of the work in DNN compression entails decomposing the layer into multiple smaller ones, e.g., via low-rank tensor decomposition [19, 41, 74, 48, 96, 40, 2, 105, 117, 53, 61].

Original: Other approaches to tensor decomposition include weight sharing, random projections,and feature hashing [107, 3, 88, 12, 13, 102].
Revised: Other approaches to tensor decomposition include weight sharing, random projections, and feature hashing [112, 3, 91, 12, 13, 107].

Original: However, such techniques usually require expensiveapproximation algorithms or use heuristics since tensor decomposition is generally NP-hard.
Revised: However, such techniques usually require expensive approximation algorithms or use heuristics since tensor decomposition is generally NP-hard.

Original: Coresets.
Revised: Coresets.

Original: In the recent years, coresets got increasing attention, and where leveraged to compress theinput datasets of many machine learning algorithms, improving there performance, e.g., regression [38,78, 47, 99], decision trees [44], matrix approximation [26, 69, 25, 86, 71], clustering [24, 29, 66, 4,45, 87], ℓ z -regression [16, 17, 91], SVM [32, 98, 96, 97, 100], and deep learning models [5, 60].
Revised: In the recent years, coresets got increasing attention, and where leveraged to compress the input datasets of many machine learning algorithms, improving there performance, e.g., regression [72, 38, 81, 47, 103], decision trees [44], matrix approximation [26, 70, 25, 89, 73], data discretization [75], clustering [24, 29, 66, 4, 45, 90, 106], ℓ z -regression [16, 17, 94], SVM [32, 101, 99, 100, 102], deep learning models [69, 5, 60] and even for path planning in the field of robotics [104].

Original: Forextensive surveys on coresets, we refer the reader to [23, 83, 43, 70].
Revised: For extensive surveys on coresets, we refer the reader to [23, 86, 43, 71].

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: In this paper, we provided a coreset-based pruning technique that hinges upon a combination oftools from convex geometry, while achieving SOTA results with respect to coreset-based structuredpruning approaches on a variety of networks.
Revised: In this paper, we provided a coreset-based pruning technique that hinges upon a combination of tools from convex geometry, while achieving SOTA results with respect to coreset-based structured pruning approaches on a variety of networks.

Original: Our main improvement is that our coreset is (training)data-independent and assumes a single assumption on the models weights.
Revised: Our main improvement is that our coreset is (training) data-independent and assumes a single assumption on the models weights.

Original: Future work includes (i) suggesting a coreset based budget allocation framework, to determine the(optimal) per layer prune ratio while achieving an overall desired compression rate, (ii) extending ourcoreset technique to other layers such as attention layers, and (iii) bridging the gap between coresetbased pruning approaches and tensor-decomposition methods, as both techniques are theoreticallysupported by bounding the approximation error given specific compression rate, we can leverage thesebounds to formulate the compression problem as an optimization problem which iterates between thetwo approaches to search for the local minimum.
Revised: Future work includes (i) suggesting a coreset based budget allocation framework, to determine the (optimal) per layer prune ratio while achieving an overall desired compression rate, (ii) extending our coreset technique to other layers such as attention layers, and (iii) bridging the gap between coreset based pruning approaches and tensor-decomposition methods, as both techniques are theoretically supported by bounding the approximation error given specific compression rate, we can leverage these bounds to formulate the compression problem as an optimization problem which iterates between the two approaches to search for the local minimum.