Original: D ATA -D EPENDENT C ORESETS FOR C OMPRESSING N EURAL N ETWORKS WITH A PPLICATIONS TO G ENERALIZATION B OUNDS
Revised: D ATA -D EPENDENT C ORESETS FOR C OMPRESSING N EURAL N ETWORKS WITH A PPLICATIONS TO G ENERALIZATION B OUNDS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present an efﬁcient coresets-based neural network compression algorithm that sparsiﬁes the parameters of a trained fully-connected neural network in a manner that provably approximates the network’s output.
Revised: We present an efﬁcient coresets-based neural network compression algorithm that sparsiﬁes the parameters of a trained fully-connected neural network in a manner that provably approximates the network’s output.

Original: Our approach is based on an importance sampling scheme that judiciously deﬁnes a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones.
Revised: Our approach is based on an importance sampling scheme that judiciously deﬁnes a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones.

Original: We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters.
Revised: We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters.

Original: Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks.
Revised: Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks.

Original: We demonstrate the practical effectiveness of our algorithm on a variety of neural network conﬁgurations and real-world data sets
Revised: We demonstrate the practical effectiveness of our algorithm on a variety of neural network conﬁgurations and real-world data sets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Within the past decade, large-scale neural networks have demonstrated unprecedented empirical success in high-impact applications such as object classiﬁcation, speech recognition, computer vision, and natural language processing.
Revised: Within the past decade, large-scale neural networks have demonstrated unprecedented empirical success in high-impact applications such as object classiﬁcation, speech recognition, computer vision, and natural language processing.

Original: However, with the ever-increasing size of state-of-the-art neural networks, the resulting storage requirements and performance of these models are becoming increasingly prohibitive in terms of both time and space.
Revised: However, with the ever-increasing size of state-of-the-art neural networks, the resulting storage requirements and performance of these models are becoming increasingly prohibitive in terms of both time and space.

Original: Recently proposed architectures for neural networks, such as those in Krizhevsky et al.
Revised: Recently proposed architectures for neural networks, such as those in Krizhevsky et al.

Original: (2012); Long et al.
Revised: (2012); Long et al.

Original: ; Badrinarayanan et al.
Revised: ; Badrinarayanan et al.

Original: (2015), contain millions of parameters, rendering them prohibitive to deploy on platforms that are resource-constrained, e.g., embedded devices, mobile phones, or small scale robotic platforms.
Revised: (2015), contain millions of parameters, rendering them prohibitive to deploy on platforms that are resource-constrained, e g., embedded devices, mobile phones, or small scale robotic platforms.

Original: In this work, we consider the problem of sparsifying the parameters of a trained fully-connected neural network in a principled way so that the output of the compressed neural network is approximately preserved.
Revised: In this work we consider the problem of sparsifying the parameters of a trained fully-connected neural network in a principled way so that the output of the compressed neural network is approximately preserved.

Original: We introduce a neural network compression approach based on identifying and removing weighted edges with low relative importance via coresets, small weighted subsets of the original set that approximate the pertinent cost function.
Revised: We introduce a neural network compression approach based on identifying and removing weighted edges with low relative importance via coresets, small weighted subsets of the original set that approximate the pertinent cost function.

Original: Our compression algorithm hinges on extensions of the traditional sensitivity-based coresets framework (Langberg & Schulman, 2010; Braverman et al., 2016), and to the best of our knowledge, is the ﬁrst to apply coresets to parameter downsizing.
Revised: Our compression algorithm hinges on extensions of the traditional sensitivity-based coresets framework (Langberg & Schulman, 2010; Braverman et al., 2016), and to the best of our knowledge, is the ﬁrst to apply coresets to parameter downsizing.

Original: In this regard, our work aims to simultaneously introduce a practical algorithm for compressing neural network parameters with provable guarantees and close the research gap in prior coresets work, which has predominantly focused on compressing input data points.
Revised: In this regard, our work aims to simultaneously introduce a practical algorithm for compressing neural network parameters with provable guarantees and close the research gap in prior coresets work, which has predominantly focused on compressing input data points.

Original: In particular, this paper contributes the following:
Revised: In particular, this paper contributes the following:

Original: A coreset approach to compressing problem-speciﬁc parameters based on a novel, empirical notion of sensitivity that extends state-of-the-art coreset constructions.
Revised: A coreset approach to compressing problem-speciﬁc parameters based on a novel, empirical notion of sensitivity that extends state-of-the-art coreset constructions.

Original: An efﬁcient neural network compression algorithm, CoreNet, based on our extended coreset approach that sparsiﬁes the parameters via importance sampling of weighted edges.
Revised: An efﬁcient neural network compression algorithm, CoreNet, based on our extended coreset approach that sparsiﬁes the parameters via importance sampling of weighted edges.

Original: Extensions of the CoreNet method, CoreNet+ and CoreNet++, that improve upon the edge sampling approach by additionally performing neuron pruning and ampliﬁcation.
Revised: Extensions of the CoreNet method, CoreNet+ and CoreNet++, that improve upon the edge sampling approach by additionally performing neuron pruning and ampliﬁcation.

Original: Analytical results establishing guarantees on the approximation accuracy, size, and generalization of the compressed neural network.
Revised: Analytical results establishing guarantees on the approximation accuracy, size, and generalization of the compressed neural network.

Original: Evaluations on real-world data sets that demonstrate the practical effectiveness of our algorithm in compressing neural network parameters and validate our theoretical results.
Revised: Evaluations on real-world data sets that demonstrate the practical effectiveness of our algorithm in compressing neural network parameters and validate our theoretical results.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work builds upon the following prior work in coresets and compression approaches.
Revised: Our work builds upon the following prior work in coresets and compression approaches.

Original: Coresets Coreset constructions were originally introduced in the context of computational geometry (Agarwal et al., 2005) and subsequently generalized for applications to other problems via an importance sampling-based, sensitivity framework (Langberg & Schulman, 2010; Braverman et al., 2016).
Revised: Coresets Coreset constructions were originally introduced in the context of computational geometry (Agarwal et al., 2005) and subsequently generalized for applications to other problems via an importance sampling-based, sensitivity framework (Langberg & Schulman, 2010; Braverman et al., 2016).

Original: Coresets have been used successfully to accelerate various machine learning algorithms such as k -means clustering (Feldman & Langberg, 2011;
Revised: Coresets have been used successfully to accelerate various machine learning algorithms such as k -means clustering (Feldman & Langberg, 2011;

Original: Braverman et al., 2016), graphical model training (Molina et al., 2018), and logistic regression (Huggins et al., 2016) (see the surveys of Bachem et al.
Revised: Braverman et al., 2016), graphical model training (Molina et al., 2018), and logistic regression (Huggins et al., 2016) (see the surveys of Bachem et al.

Original: (2017) and Munteanu & Schwiegelshohn (2018)
Revised: (2017) and Munteanu & Schwiegelshohn (2018)

Original: for a complete list).
Revised: for a complete list).

Original: In contrast to prior work, we generate coresets for reducing the number of parameters – rather than data points – via a novel construction scheme based on an efﬁciently-computable notion of sensitivity.
Revised: In contrast to prior work, we generate coresets for reducing the number of parameters – rather than data points – via a novel construction scheme based on an efﬁciently-computable notion of sensitivity.

Original: Low-rank Approximations and Weight-sharing Denil et al.
Revised: Low-rank Approximations and Weight-sharing Denil et al.

Original: (2013) were among the ﬁrst to empirically demonstrate the existence of signiﬁcant parameter redundancy in deep neural networks.
Revised: (2013) were among the ﬁrst to empirically demonstrate the existence of signiﬁcant parameter redundancy in deep neural networks.

Original: A predominant class of compression approaches consists of using low-rank matrix decompositions, such as Singular Value Decomposition (SVD) (Denton et al., 2014), to approximate the weight matrices with their low-rank counterparts.
Revised: A predominant class of compression approaches consists of using low-rank matrix decompositions, such as Singular Value Decomposition (SVD) (Denton et al., 2014), to approximate the weight matrices with their low-rank counterparts.

Original: Similar works entail the use of low-rank tensor decomposition approaches applicable both during and after training (Jaderberg et al., 2014; Kim et al., 2015; Tai et al., 2015; Ioannou et al., 2015; Alvarez & Salzmann, 2017; Yu et al., 2017).
Revised: Similar works entail the use of low-rank tensor decomposition approaches applicable both during and after training (Jaderberg et al., 2014; Kim et al., 2015; Tai et al., 2015; Ioannou et al., 2015; Alvarez & Salzmann, 2017; Yu et al., 2017).

Original: Another class of approaches uses feature hashing and weight sharing (Weinberger et al., 2009;
Revised: Another class of approaches uses feature hashing and weight sharing (Weinberger et al., 2009;

Original: Shi et al., 2009; Chen et al., 2015b;a;
Revised: Shi et al., 2009; Chen et al., 2015b;a;

Original: Ullrich et al., 2017).
Revised: Ullrich et al., 2017).

Original: Building upon the idea of weight-sharing, quantization (Gong et al., 2014;
Revised: Building upon the idea of weight-sharing, quantization (Gong et al., 2014;

Original: Wu et al., 2016;
Revised: Wu et al., 2016;

Original: Zhou et al., 2017) or regular structure of weight matrices was used to reduce the effective number of parameters (Zhao et al., 2017; Sindhwani et al., 2015; Cheng et al., 2015; Choromanska et al., 2016; Wen et al., 2016).
Revised: Zhou et al., 2017) or regular structure of weight matrices was used to reduce the effective number of parameters (Zhao et al., 2017; Sindhwani et al., 2015; Cheng et al., 2015; Choromanska et al., 2016; Wen et al., 2016).

Original: Despite their practical effectiveness in compressing neural networks, these works generally lack performance guarantees on the quality of their approximations and/or the size of the resulting compressed network.
Revised: Despite their practical effectiveness in compressing neural networks, these works generally lack performance guarantees on the quality of their approximations and/or the size of the resulting compressed network.

Original: Weight Pruning Similar to our proposed method, weight pruning (LeCun et al., 1990) hinges on the idea that only a few dominant weights within a layer are required to approximately preserve the output.
Revised: Weight Pruning Similar to our proposed method, weight pruning (LeCun et al., 1990) hinges on the idea that only a few dominant weights within a layer are required to approximately preserve the output.

Original: Approaches of this ﬂavor have been investigated by Lebedev & Lempitsky (2016);
Revised: Approaches of this ﬂavor have been investigated by Lebedev & Lempitsky (2016);

Original: Dong et al.
Revised: Dong et al.

Original: (2017), e.g., by embedding sparsity as a constraint (Iandola et al., 2016;
Revised: (2017), e.g., by embedding sparsity as a constraint (Iandola et al., 2016;

Original: Aghasi et al., 2017; Lin et al., 2017).
Revised: Aghasi et al., 2017; Lin et al., 2017).

Original: Another related approach is that of Han et al.
Revised: Another related approach is that of Han et al.

Original: (2015), which considers a combination of weight pruning and weight sharing methods.
Revised: (2015), which considers a combination of weight pruning and weight sharing methods.

Original: Nevertheless, prior work in weight pruning lacks rigorous theoretical analysis of the effect that the discarded weights can have on the compressed network.
Revised: Nevertheless, prior work in weight pruning lacks rigorous theoretical analysis of the effect that the discarded weights can have on the compressed network.

Original: To the best of our knowledge, our work is the ﬁrst to introduce a practical, sampling-based weight pruning algorithm with provable guarantees.
Revised: To the best of our knowledge, our work is the ﬁrst to introduce a practical, sampling-based weight pruning algorithm with provable guarantees.

Original: Generalization
Revised: Generalization

Original: The generalization properties of neural networks have been extensively investigated in various contexts (Dziugaite & Roy, 2017; Neyshabur et al., 2017a; Bartlett et al., 2017).
Revised: The generalization properties of neural networks have been extensively investigated in various contexts (Dziugaite & Roy, 2017; Neyshabur et al., 2017a; Bartlett et al., 2017).

Original: However, as was pointed out by Neyshabur et al.
Revised: However, as was pointed out by Neyshabur et al.

Original: (2017b)
Revised: (2017b)

Original: , current approaches to obtaining non-vacuous generalization bounds do not fully or accurately capture the empirical success of state-of-the-art neural network architectures.
Revised: , current approaches to obtaining non-vacuous generalization bounds do not fully or accurately capture the empirical success of state-of-the-art neural network architectures.

Original: Recently, Arora et al.
Revised: Recently, Arora et al.

Original: (2018) and Zhou et al.
Revised: (2018) and Zhou et al.

Original: (2018) highlighted the close connection between compressibility and generalization of neural networks.
Revised: (2018) highlighted the close connection between compressibility and generalization of neural networks.

Original: Arora et al.
Revised: Arora et al.

Original: (2018) presented a compression method based on the Johnson-Lindenstrauss (JL) Lemma (Johnson & Lindenstrauss, 1984) and proved generalization bounds based on succinct reparameterizations of the original neural network.
Revised: (2018) presented a compression method based on the Johnson-Lindenstrauss (JL) Lemma (Johnson & Lindenstrauss, 1984) and proved generalization bounds based on succinct reparameterizations of the original neural network.

Original: Building upon the work of Arora et al.
Revised: Building upon the work of Arora et al.

Original: (2018), we extend our theoretical compression results to establish novel generalization bounds for fully-connected neural networks.
Revised: (2018), we extend our theoretical compression results to establish novel generalization bounds for fully-connected neural networks.

Original: Unlike the method of Arora et al.
Revised: Unlike the method of Arora et al.

Original: (2018), which exhibits guarantees of the compressed network’s performance only on the set of training points, our method’s guarantees hold (probabilistically) for any random point drawn from the distribution.
Revised: (2018), which exhibits guarantees of the compressed network’s performance only on the set of training points, our method’s guarantees hold (probabilistically) for any random point drawn from the distribution.

Original: In addition, we establish that our method can ε -approximate the neural network output neuron-wise, which is stronger than the norm-based guarantee of Arora et al.
Revised: In addition, we establish that our method can ε -approximate the neural network output neuron-wise, which is stronger than the norm-based guarantee of Arora et al.

Original: In contrast to prior work, this paper addresses the problem of compressing a fully-connected neural network while provably preserving the network’s output.
Revised: In contrast to prior work, this paper addresses the problem of compressing a fully-connected neural network while provably preserving the network’s output.

Original: Unlike previous theoretically-grounded compression approaches – which provide guarantees in terms of the normed difference –, our method provides the stronger entry-wise approximation guarantee, even for points outside of the available data set.
Revised: Unlike previous theoretically-grounded compression approaches – which provide guarantees in terms of the normed difference –, our method provides the stronger entry-wise approximation guarantee, even for points outside of the available data set.

Original: As our empirical results show, ensuring that the output of the compressed network entry-wise approximates that of the original network is critical to retaining high classiﬁcation accuracy.
Revised: As our empirical results show, ensuring that the output of the compressed network entry-wise approximates that of the original network is critical to retaining high classiﬁcation accuracy.

Original: Overall, our compression approach remedies the shortcomings of prior approaches in that it (i) exhibits favorable theoretical properties, (ii) is computationally efﬁcient, e.g., does not require retraining of the neural network, (iii) is easy to implement, and (iv) can be used in conjunction with other compression approaches – such as quantization or Huffman coding – to obtain further improved compression rates.
Revised: Overall, our compression approach remedies the shortcomings of prior approaches in that it (i) exhibits favorable theoretical properties, (ii) is computationally efﬁcient, e.g., does not require retraining of the neural network, (iii) is easy to implement, and (iv) can be used in conjunction with other compression approaches – such as quantization or Huffman coding – to obtain further improved compression rates.

Original:  P ROBLEM D EFINITION 3. F ULLY -C ONNECTED N EURAL N ETWORKS
Revised:  P ROBLEM D EFINITION 3. F ULLY -C ONNECTED N EURAL N ETWORKS

Original: A feedforward fully-connected neural network with L ∈ N + layers and parameters θ deﬁnes a mapping f θ : X → Y for a given input x ∈ X ⊆ R d to an output y ∈ Y ⊆
Revised: A feedforward fully-connected neural network with L ∈ N + layers and parameters θ deﬁnes a mapping f θ : X → Y for a given input x ∈ X ⊆ R d to an output y ∈ Y ⊆

Original: R k as follows.
Revised: R k as follows.

Original: Let η (cid:96) ∈
Revised: Let η (cid:96) ∈

Original: N
Revised: N

Original: + denote the number of neurons in layer (
Revised: + denote the number of neurons in layer (

Original: cid:96) ∈
Revised: cid:96) ∈

Original: [ L ] , where [ L ] = { 1 , . . .
Revised: [ L ] , where [ L ] = { 1 , . . .

Original: , L } denotes the index set, and where η 1 = d and η L = k .
Revised: , L } denotes the index set, and where η 1 = d and η L = k .

Original: Further, let η = (cid:80) L(cid:96)
Revised: Further, let η = (cid:80) L(cid:96)

Original: =
Revised: =

Original: 2 η (cid:96) and η ∗ = max (cid:96) ∈{ 2 ,...,L } η (cid:96) .
Revised: 2 η (cid:96) and η ∗ = max (cid:96) ∈{ 2 ,...,L } η (cid:96) .

Original: For layers (cid:96) ∈ { 2 , . . .
Revised: For layers (cid:96) ∈ { 2 , . . .

Original: , L } , let W (cid:96) ∈
Revised: , L } , let W (cid:96) ∈

Original: R
Revised: R

Original: η
Revised: η

Original: (cid:96) × η (cid:96) − 1 be the weight matrix for layer (cid:96) with entries denoted by w (cid:96)ij , rows denoted by w (cid:96)i ∈ R 1 × η (cid:96) − 1 , and θ =
Revised: (cid:96) × η (cid:96) − 1 be the weight matrix for layer (cid:96) with entries denoted by w (cid:96)ij , rows denoted by w (cid:96)i ∈ R 1 × η (cid:96) − 1 , and θ =

Original: ( W 2 , . . .
Revised: ( W 2 , . . .

Original: , W L ) .
Revised: , W L ) .

Original: For notational simplicity, we assume that the bias is embedded in the weight matrix.
Revised: For notational simplicity, we assume that the bias is embedded in the weight matrix.

Original: Then for an input vector x ∈ R d , let a 1 = x
Revised: Then for an input vector x ∈ R d , let a 1 = x

Original: and z (cid:96) =
Revised: and z (cid:96) =

Original: W (cid:96) a (cid:96) − 1 ∈ R η (cid:96) , ∀ (cid:96) ∈ { 2 , . . .
Revised: W (cid:96) a (cid:96) − 1 ∈ R η (cid:96) , ∀ (cid:96) ∈ { 2 , . . .

Original: , L } , where a (cid:96) − 1 = φ ( z (cid:96) − 1 ) ∈ R η
Revised: , L } , where a (cid:96) − 1 = φ ( z (cid:96) − 1 ) ∈ R η

Original: (cid:96) − 1 denotes the activation.
Revised: (cid:96) − 1 denotes the activation.

Original: We consider the activation function to be the Rectiﬁed Linear Unit (ReLU) function, i.e., φ ( · ) = max {· , 0 } (entry-wise, if the input is a vector).
Revised: We consider the activation function to be the Rectiﬁed Linear Unit (ReLU) function, i.e., φ ( · ) = max {· , 0 } (entry-wise, if the input is a vector).

Original: The output of the network for an input x is f θ ( x ) =
Revised: The output of the network for an input x is f θ ( x ) =

Original: z L , and in particular, for classiﬁcation tasks the prediction is 
Revised: z L , and in particular, for classiﬁcation tasks the prediction is argmax i ∈

Revised: [ k ] f θ ( x ) i = argmax i ∈

Original: [Equation]
Revised: [ k ] z Li .

Original: 3. N EURAL N ETWORK C ORESET P ROBLEM
Revised: 3. N EURAL N ETWORK C ORESET P ROBLEM

Original: Consider the setting where a neural network f θ ( · ) has been trained on a training set of independent and identically distributed (i.i.d.)
Revised: Consider the setting where a neural network f θ ( · ) has been trained on a training set of independent and identically distributed (i.i.d.)

Original: samples from a joint distribution on X × Y , yielding parameters θ =
Revised: samples from a joint distribution on X × Y , yielding parameters θ =

Original: ( W 2 , . . .
Revised: ( W 2 , . . .

Original: , W L ) .
Revised: , W L ) .

Original: We further denote the input points of a validation data set as P = { x i } ni
Revised: We further denote the input points of a validation data set as P = { x i } ni

Original: =1 ⊆ X and the marginal distribution over the input space X as D .
Revised: =1 ⊆ X and the marginal distribution over the input space X as D .

Original: We deﬁne the size of the parameter tuple θ , nnz( θ ) , to be the sum of the number of non-zero entries in the weight matrices W 2 , . . .
Revised: We deﬁne the size of the parameter tuple θ , nnz( θ ) , to be the sum of the number of non-zero entries in the weight matrices W 2 , . . .

Original: , W L .
Revised: , W L .

Original: For any given ε, δ ∈ (0 , 1) , our overarching goal is to generate a reparameterization ˆ θ , yielding the neural network f ˆ θ ( · ) , using a randomized algorithm, such that nnz(ˆ θ ) (cid:28) nnz( θ ) , and the neural network output f θ ( x ) , x ∼ D can be approximated up to 1 ± ε multiplicative error with probability greater than 1 − δ .
Revised: For any given ε, δ ∈ (0 , 1) , our overarching goal is to generate a reparameterization ˆ θ , yielding the neural network f ˆ θ ( · ) , using a randomized algorithm, such that nnz(ˆ θ ) (cid:28) nnz( θ ) , and the neural network output f θ ( x ) , x ∼ D can be approximated up to 1 ± ε multiplicative error with probability greater than 1 − δ .

Original: We deﬁne the 1 ± ε multiplicative error between two k -dimensional vectors a, b ∈ R k as the following entry-wise bound: a ∈ (1 ± ε ) b ⇔ a i ∈ (1 ± ε ) b i ∀ i ∈ [ k ] , and formalize the deﬁnition of an ( ε, δ ) -coreset as follows.
Revised: We deﬁne the 1 ± ε multiplicative error between two k -dimensional vectors a, b ∈ R k as the following entry-wise bound: a ∈ (1 ± ε ) b ⇔ a i ∈ (1 ± ε ) b i ∀ i ∈ [ k ] , and formalize the deﬁnition of an ( ε, δ ) -coreset as follows.

Original: Deﬁnition 1 ( ( ε, δ ) -coreset) .
Revised: Deﬁnition 1 ( ( ε, δ ) -coreset) .

Original: Given user-speciﬁed ε, δ ∈ (0 , 1) , a set of parameters ˆ θ
Revised: Given user-speciﬁed ε, δ ∈ (0 , 1) , a set of parameters ˆ θ

Original: =
Revised: =

Original: ( ˆ W 2 , . .
Revised: ( ˆ W 2 , . .

Original: , ˆ W L ) is an ( ε, δ ) -coreset for the network parameterized by θ if for x ∼ D , it holds that
Revised: , ˆ W L ) is an ( ε, δ ) -coreset for the network parameterized by θ if for x ∼ D , it holds that

Original: [Equation]
Revised: [Equation]

Original: where P ˆ θ,x denotes a probability measure with respect to a random data point x and the output ˆ θ generated by a randomized compression scheme.
Revised: where P ˆ θ,x denotes a probability measure with respect to a random data point x and the output ˆ θ generated by a randomized compression scheme.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we introduce our neural network compression algorithm as depicted in Alg. 1.
Revised: In this section, we introduce our neural network compression algorithm as depicted in Alg. 1.

Original: Our method is based on an important sampling-scheme that extends traditional sensitivity-based coreset constructions to the application of compressing parameters.
Revised: Our method is based on an important sampling-scheme that extends traditional sensitivity-based coreset constructions to the application of compressing parameters.

Original: 4. C ORE N ET
Revised: 4. C ORE N ET

Original: Our method (Alg. 1) hinges on the insight that a validation set of data points P i.i.d.
Revised: Our method (Alg. 1) hinges on the insight that a validation set of data points P i.i.d.

Original: ∼ D n can be used to approximate the relative importance, i.e., sensitivity, of each weighted edge with respect to the input data distribution D .
Revised: ∼ D n can be used to approximate the relative importance, i.e., sensitivity, of each weighted edge with respect to the input data distribution D .

Original: For this purpose, we ﬁrst pick a subsample of the data points S ⊆ P of appropriate size (see Sec.
Revised: For this purpose, we ﬁrst pick a subsample of the data points S ⊆ P of appropriate size (see Sec.

Original: 5 for details) and cache each neuron’s activation and compute a neuron-speciﬁc constant to be used to determine the required edge sampling complexity (Lines 2-6).
Revised: 5 for details) and cache each neuron’s activation and compute a neuron-speciﬁc constant to be used to determine the required edge sampling complexity (Lines 2-6)

Original: Subsequently, we apply our core sampling scheme to sparsify the set of incoming weighted edges to each neuron in all layers (Lines 7-13).
Revised: [Table] [Table] [Table] [Table]

Original: For technical reasons (see Sec. 5), we perform the sparsiﬁcation on the positive and negative weighted edges separately and then consolidate the results (Lines 1113).
Revised: [Table] [Table] [Table] [Table]

Original: By repeating this procedure for all neurons in every layer, we obtain a set ˆ θ =
Revised: [Table] [Table] [Table] [Table]

Original: ( ˆ W 2 , . .
Revised: [Table] [Table] [Table]

Original: , ˆ W L )
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ] (cid:80) x ∈S ∆ (cid:96)i ( x ) (cid:17) + κ, where κ = √ 2 [-λ ∗ (cid:0) 
-]
Revised: − 1 k ( x ) (cid:12)(cid:12)(cid:12) ; 7: for (cid:96) ∈ { 2 [-λ ∗ (cid:0) 
-], . . .

Original:  + √ 2 λ ∗ log (8 η η ∗ /δ )
Revised: , L } do 8: ˆ∆ (cid:96) ← (cid:16) 1 |S| max i ∈ [ η (cid:96) ] (cid:80) x ∈S ∆ (cid:96)i ( x ) (cid:17) + κ, where κ = √ 2 λ ∗ (cid:0) 1 + √ 2 λ ∗ log (8 η η ∗ /δ )

Original: (cid:1) ; 9: ˆ W (cid:96) ← ( (cid:126) 0 , . . .
Revised: (cid:1) ; 9: ˆ W (cid:96) ← ( (cid:126) 0 , . . .

Original: ,(cid:126) 0) ∈ R η (cid:96) × η (cid:96) − 1 ; ˆ∆ (cid:96) → ← (cid:81)
Revised: ,(cid:126) 0) ∈ R η (cid:96) × η (cid:96) − 1 ; ˆ∆ (cid:96) → ← (cid:81)

Original: Lk
Revised: Lk

Original: =
Revised: =

Original: (cid:96) ˆ∆
Revised: (cid:96) ˆ∆

Original: k ; ε (cid:96) ← ε (cid:48) ˆ∆ (cid:96) → ; 10: for all i ∈
Revised: k ; ε (cid:96) ← ε (cid:48) ˆ∆ (cid:96) → ; 10: for all i ∈

Original: [ η (cid:96) ] do 11:
Revised: [ η (cid:96) ] do 11:

Original: W + ← { j ∈ [ η (cid:96) − 1 ] : w (cid:96) ij > 0 } ; W − ← { j ∈ [ η (cid:96) − 1 ] : w (cid:96) ij < 0 } ; 12: ˆ w (cid:96) + i ← S PARSIFY ( W + , w (cid:96)i , ε (cid:96) , δ, S , a (cid:96) − 1 ) ; ˆ w (cid:96) − i ← S PARSIFY ( W − , − w (cid:96)i , ε (cid:96) , δ, S , a (cid:96) − 1 ) ; 13: ˆ w (cid:96)i ← ˆ w (cid:96) + i − ˆ w (cid:96) − i ; ˆ W (cid:96)i • ← ˆ w (cid:96)i ; (cid:46) Consolidatethe weights into the i th rowof ˆ W(cid:96) ; 14: returnˆ θ= ( ˆ W 2 , . .
Revised: W + ← { j ∈ [ η (cid:96) − 1 ] : w (cid:96)ij > 0 } ; W − ← { j ∈ [ η (cid:96) − 1 ] : w (cid:96)ij < 0 } ; 12: ˆ w (cid:96) + i ← S PARSIFY ( W + , w (cid:96)i , ε (cid:96) , δ, S , a (cid:96) − 1 ) ; ˆ w (cid:96) − i ← S PARSIFY ( W − , − w (cid:96)i , ε (cid:96) , δ, S , a (cid:96) − 1 ) ; 13: ˆ w (cid:96)i ← ˆ w (cid:96) + i − ˆ w (cid:96) − i ; ˆ W (cid:96)i • ← ˆ w (cid:96)i ; (cid:46) Consolidate i th row ˆ W ; 14: return θ

Original: , ˆ W L ) ; 
Revised: L ) ; [Equation]

Original: Input: W ⊆
Revised: Subsequently, we apply our core sampling scheme to sparsify the set of incoming weighted edges to each neuron in all layers (Lines 7-13). For technical reasons (see Sec. 5), we perform the sparsiﬁcation on the positive and negative weighted edges separately and then consolidate the results (Lines 1113).

Original: [ η (cid:96) − 1 ] : index set; w ∈ R 1 × η (cid:96) − 1 : row vector corresponding to the weights incoming of sparse weight matrices such that the output of each layer and the entire network is approximately preserved, i.e., ˆ W (cid:96) ˆ a (cid:96) − 1 ( x ) ≈ W (cid:96) a (cid:96) − 1 ( x ) and f ˆ θ ( x ) ≈ f θ ( x ) , respectively 1 .
Revised: By repeating this procedure for all neurons in every layer, we obtain a set ˆ θ = ( ˆ W 2 , . . , ˆ W L ) of sparse weight matrices such that the output of each layer and the entire network is approximately preserved

Original: 4. S PARSIFYING W EIGHTS
Revised: 4. S PARSIFYING W EIGHTS

Original: The crux of our compression scheme lies in Alg.
Revised: The crux of our compression scheme lies in Alg.

Original: 2 (invoked twice on Line 12, Alg. 1) and in particular, in the importance sampling scheme used to select a small subset of edges of high importance.
Revised: 2 (invoked twice on Line 12, Alg. 1) and in particular, in the importance sampling scheme used to select a small subset of edges of high importance.

Original: The cached activations are used to compute the sensitivity , i.e., relative importance, of each considered incoming edge j ∈ W to neuron i ∈
Revised: The cached activations are used to compute the sensitivity , i.e., relative importance, of each considered incoming edge j ∈ W to neuron i ∈

Original: [ η (cid:96) ] , (cid:96) ∈ { 2 , . . .
Revised: [ η (cid:96) ] , (cid:96) ∈ { 2 , . . .

Original: , L } (Alg. 2, Lines 1-2).
Revised: , L } (Alg. 2, Lines 1-2).

Original: The relative importance of each edge j is computed as the maximum (over x ∈ S ) ratio of the edge’s contribution to the sum of contributions of all edges.
Revised: The relative importance of each edge j is computed as the maximum (over x ∈ S ) ratio of the edge’s contribution to the sum of contributions of all edges.

Original: In other words, the sensitivity s j of an edge j captures the highest (relative) impact j had on the output of neuron i ∈ [ η (cid:96) ] in layer (cid:96) across all x ∈ S .
Revised: In other words, the sensitivity s j of an edge j captures the highest (relative) impact j had on the output of neuron i ∈ [ η (cid:96) ] in layer (cid:96) across all x ∈ S .

Original: The sensitivities are then used to compute an importance sampling distribution over the incoming weighted edges (Lines 4-5).
Revised: The sensitivities are then used to compute an importance sampling distribution over the incoming weighted edges (Lines 4-5).

Original: The intuition behind the importance sampling distribution is that if s j is high, then edge j is more likely to have a high impact on the output of neuron i , therefore we should keep edge j with a higher probability.
Revised: The intuition behind the importance sampling distribution is that if s j is high, then edge j is more likely to have a high impact on the output of neuron i , therefore we should keep edge j with a higher probability.

Original: m edges are then sampled with replacement (Lines 6-7) and the sampled weights are then reweighed to ensure unbiasedness of our estimator (Lines 9-10).
Revised: m edges are then sampled with replacement (Lines 6-7) and the sampled weights are then reweighed to ensure unbiasedness of our estimator (Lines 9-10).

Original: 4. E XTENSIONS : N EURON P RUNING AND A MPLIFICATION
Revised: 4. E XTENSIONS : N EURON P RUNING AND A MPLIFICATION

Original: In this subsection we outline two improvements to our algorithm that that do not violate any of our theoretical properties and may improve compression rates in practical settings.
Revised: In this subsection we outline two improvements to our algorithm that that do not violate any of our theoretical properties and may improve compression rates in practical settings.

Original: Neuron pruning (CoreNet+) Similar to removing redundant edges, we can use the empirical activations to gauge the importance of each neuron.
Revised: Neuron pruning (CoreNet+) Similar to removing redundant edges, we can use the empirical activations to gauge the importance of each neuron.

Original: In particular, if the maximum activation (over all evaluations x ∈ S ) of a neuron is equal to 0, then the neuron – along with all of the incoming and outgoing edges – can be pruned without signiﬁcantly affecting the output with reasonable probability.
Revised: In particular, if the maximum activation (over all evaluations x ∈ S ) of a neuron is equal to 0, then the neuron – along with all of the incoming and outgoing edges – can be pruned without signiﬁcantly affecting the output with reasonable probability.

Original: This intuition can be made rigorous under the assumptions outlined in Sec. 5.
Revised: This intuition can be made rigorous under the assumptions outlined in Sec. 5.

Original: Ampliﬁcation (CoreNet++) Coresets that provide stronger approximation guarantees can be constructed via ampliﬁcation – the procedure of constructing multiple approximations (coresets) ( ˆ w (cid:96)i ) 1 , . . .
Revised: Ampliﬁcation (CoreNet++) Coresets that provide stronger approximation guarantees can be constructed via ampliﬁcation – the procedure of constructing multiple approximations (coresets) ( ˆ w (cid:96)i ) 1 , . . .

Original: , ( ˆ w (cid:96)i ) τ over τ trials, and picking the best one.
Revised: , ( ˆ w (cid:96)i ) τ over τ trials, and picking the best one.

Original: To evaluate the quality of each approximation, a different subset T ⊆
Revised: To evaluate the quality of each approximation, a different subset T ⊆

Original: P \ S can be used to infer performance.
Revised: P \ S can be used to infer performance.

Original: In practice, ampliﬁcation would entail constructing multiple approximations by executing Line 12 of Alg. 1 and picking the one that achieves the lowest relative error on T .
Revised: In practice, ampliﬁcation would entail constructing multiple approximations by executing Line 12 of Alg. 1 and picking the one that achieves the lowest relative error on T .

Original:  A NALYSIS
Revised:  A NALYSIS

Original: In this section, we establish the theoretical guarantees of our neural network compression algorithm (Alg. 1).
Revised: In this section, we establish the theoretical guarantees of our neural network compression algorithm (Alg. 1).

Original: The full proofs of all the claims presented in this section can be found in the Appendix.
Revised: The full proofs of all the claims presented in this section can be found in the Appendix.

Original: 5. P RELIMINARIES
Revised: 5. P RELIMINARIES

Original: Let x ∼ D be a randomly drawn input point.
Revised: Let x ∼ D be a randomly drawn input point.

Original: We explicitly refer to the pre-activation and activation values at layer (cid:96) ∈ { 2 , . . .
Revised: We explicitly refer to the pre-activation and activation values at layer (cid:96) ∈ { 2 , . . .

Original: , (cid:96) } with respect to the input x ∈ supp( D ) as z (cid:96) ( x ) and a (cid:96) ( x ) , respectively.
Revised: , (cid:96) } with respect to the input x ∈ supp( D ) as z (cid:96) ( x ) and a (cid:96) ( x ) , respectively.

Original: The values of z (cid:96) ( x ) and a (cid:96) ( x ) at each layer (cid:96) will depend on whether or not we compressed the previous layers (cid:96) (cid:48) ∈ { 2 , . . .
Revised: The values of z (cid:96) ( x ) and a (cid:96) ( x ) at each layer (cid:96) will depend on whether or not we compressed the previous layers (cid:96) (cid:48) ∈ { 2 , . . .

Original: , (cid:96) } .
Revised: , (cid:96) } .

Original: To formalize this interdependency, we let ˆ z (cid:96) ( x ) and ˆ a (cid:96) ( x ) denote the respective quantities of layer (cid:96) when we replace the weight matrices W 2 , . . .
Revised: To formalize this interdependency, we let ˆ z (cid:96) ( x ) and ˆ a (cid:96) ( x ) denote the respective quantities of layer (cid:96) when we replace the weight matrices W 2 , . . .

Original: , W (cid:96) in layers 2 , . . .
Revised: , W (cid:96) in layers 2 , . . .

Original: , (cid:96) by ˆ W 2 , . . .
Revised: , (cid:96) by ˆ W 2 , . . .

Original: , ˆ W (cid:96) , respectively.
Revised: , ˆ W (cid:96) , respectively.

Original: For the remainder of this section (Sec. 5) we let (cid:96) ∈ { 2 , . . .
Revised: For the remainder of this section (Sec. 5) we let (cid:96) ∈ { 2 , . . .

Original: , L } be an arbitrary layer and let i ∈
Revised: , L } be an arbitrary layer and let i ∈

Original: [ η (cid:96) ] be an arbitrary neuron in layer (cid:96) .
Revised: [ η (cid:96) ] be an arbitrary neuron in layer (cid:96) .

Original: For purposes of clarity and readability, we will omit the the variable denoting the layer (cid:96) ∈ { 2 , . . .
Revised: For purposes of clarity and readability, we will omit the the variable denoting the layer (cid:96) ∈ { 2 , . . .

Original: , L } , the neuron i ∈ [ η (cid:96) ] , and the incoming edge index j ∈ [ η (cid:96) − 1 ] , whenever they are clear from the context.
Revised: , L } , the neuron i ∈ [ η (cid:96) ] , and the incoming edge index j ∈ [ η (cid:96) − 1 ] , whenever they are clear from the context.

Original: For example, when referring to the intermediate value of a neuron i ∈ [ η (cid:96) ] in layer (cid:96) ∈ { 2 , . . .
Revised: For example, when referring to the intermediate value of a neuron i ∈ [ η (cid:96) ] in layer (cid:96) ∈ { 2 , . . .

Original: , L } , z (cid:96)i ( x ) = (cid:104) w (cid:96)i , ˆ a (cid:96) − 1 ( x ) (cid:105) ∈ R with respect to a point x , we will simply write z ( x ) =
Revised: , L } , z (cid:96)i ( x ) = (cid:104) w (cid:96)i , ˆ a (cid:96) − 1 ( x ) (cid:105) ∈ R with respect to a point x , we will simply write z ( x ) =

Original: (cid:104) w, a ( x ) (cid:105) ∈ R , where w :=
Revised: (cid:104) w, a ( x ) (cid:105) ∈ R , where w :=

Original: w (cid:96)i ∈ R 1 × η (cid:96) − 1 and a ( x ) :
Revised: w (cid:96) i ∈ R 1 × η (cid:96) − 1 and a ( x ) :

Original: = a (cid:96) − 1 ( x ) ∈ R η (cid:96) − 1 × 1 .
Revised: = a (cid:96) − 1 ( x ) ∈ R η (cid:96) − 1 × 1 .

Original: Under this notation, the weight of an incoming edge j is denoted by w j ∈ R .
Revised: Under this notation, the weight of an incoming edge j is denoted by w j ∈ R .

Original: 5. I MPORTANCE S AMPLING B OUNDS FOR P OSITIVE W EIGHTS
Revised: 5. I MPORTANCE S AMPLING B OUNDS FOR P OSITIVE W EIGHTS

Original: In this subsection, we establish approximation guarantees under the assumption that the weights are positive.
Revised: In this subsection, we establish approximation guarantees under the assumption that the weights are positive.

Original: Moreover, we will also assume that the input, i.e., the activation from the previous layer, is non-negative (entry-wise).
Revised: Moreover, we will also assume that the input, i.e., the activation from the previous layer, is non-negative (entry-wise).

Original: The subsequent subsection will then relax these assumptions to conclude that a neuron’s value can be approximated well even when the weights and activations are not all positive and non-negative, respectively.
Revised: The subsequent subsection will then relax these assumptions to conclude that a neuron’s value can be approximated well even when the weights and activations are not all positive and non-negative, respectively.

Original: Let W = { j ∈ [ η (cid:96) − 1 ] : w j > 0 } ⊆
Revised: Let W = { j ∈ [ η (cid:96) − 1 ] : w j > 0 } ⊆

Original: [ η (cid:96) − 1 ] be the set of indices of incoming edges with strictly positive weights.
Revised: [ η (cid:96) − 1 ] be the set of indices of incoming edges with strictly positive weights.

Original: To sample the incoming edges to a neuron, we quantify the relative importance of each edge as follows.
Revised: To sample the incoming edges to a neuron, we quantify the relative importance of each edge as follows.

Original: Deﬁnition 2 (Relative Importance) .
Revised: Deﬁnition 2 (Relative Importance) .

Original: The importance of an incoming edge j ∈ W with respect to an input x ∈ supp( D ) is given by the function g j ( x ) , where g j ( x ) =
Revised: The importance of an incoming edge j ∈ W with respect to an input x ∈ supp( D ) is given by the function g j ( x ) , where g j ( x ) =

Original: [Equation]
Revised: w j a j ( x ) (cid:80) k ∈W w k a k ( x ) ∀ j ∈ W .

Original: Note that g j ( x ) is a function of the random variable x ∼ D .
Revised: Note that g j ( x ) is a function of the random variable x ∼ D .

Original: We now present our ﬁrst assumption that pertains to the Cumulative Distribution Function (CDF) of the relative importance random variable.
Revised: We now present our ﬁrst assumption that pertains to the Cumulative Distribution Function (CDF) of the relative importance random variable.

Original: Assumption 1.
Revised: Assumption 1.

Original: For all j ∈ W , the CDF of the random variable g j ( x ) , denoted by F j ( · ) , satisﬁes
Revised: For all j ∈ W , the CDF of the random variable g j ( x ) , denoted by F j ( · ) , satisﬁes

Original: F j ( M / K ) ≤ exp( − 1 /K ) , where M = min { x ∈ [0 , 1] : F j ( x ) = 1 } , and K ∈ [2 , log( η η ∗ )] is a universal constant.
Revised: [Equation] where M = min { x ∈ [0 , 1] : F j ( x ) = 1 } , and K ∈ [2 , log( η η ∗ )] is a universal constant.

Original: Assumption 1 is a technical assumption on the ratio of the weighted activations that will enable us to rule out pathological problem instances where the relative importance of each edge cannot be well-approximated using a small number of data points S ⊆ P .
Revised: Assumption 1 is a technical assumption on the ratio of the weighted activations that will enable us to rule out pathological problem instances where the relative importance of each edge cannot be well-approximated using a small number of data points S ⊆ P .

Original: Henceforth, we consider a uniformly drawn (without replacement) subsample S ⊆ P as in Line 2 of Alg. 1, where |S| = (cid:100) log (8 η η ∗ /δ )
Revised: Henceforth, we consider a uniformly drawn (without replacement) subsample S ⊆ P as in Line 2 of Alg. 1, where |S| = (cid:100) log (8 η η ∗ /δ )

Original: log( η η ∗ ) (cid:101) , and deﬁne the sensitivity of an edge as follows.
Revised: log( η η ∗ ) (cid:101) , and deﬁne the sensitivity of an edge as follows.

Original: Deﬁnition 3 (Empirical Sensitivity) .
Revised: Deﬁnition 3 (Empirical Sensitivity) .

Original: Let S ⊆
Revised: Let S ⊆

Original: P be a subset of distinct points from P i.i.d.
Revised: P be a subset of distinct points from P i.i.d.

Original: ∼ D n .Then, the sensitivity over positive edges j ∈ W directed to a neuron is deﬁned as s j = max x ∈S g j ( x ) .
Revised: ∼ D n .Then, the sensitivity over positive edges j ∈ W directed to a neuron is deﬁned as s j = max x ∈S g j ( x ) .

Original: Our ﬁrst lemma establishes a core result that relates the weighted sum with respect to the sparse row vector ˆ w , (cid:80)
Revised: Our ﬁrst lemma establishes a core result that relates the weighted sum with respect to the sparse row vector ˆ w , (cid:80)

Original: k ∈W ˆ w k ˆ a k ( x ) , to the value of the of the weighted sum with respect to the ground-truth row vector w , (cid:80)
Revised: k ∈W ˆ w k ˆ a k ( x ) , to the value of the of the weighted sum with respect to the ground-truth row vector w , (cid:80)

Original: k ∈W w k ˆ a k ( x ) .
Revised: k ∈W w k ˆ a k ( x ) .

Original: We remark that there is randomness with respect to the randomly generated row vector ˆ w (cid:96)i , a randomly drawn input x ∼ D , and the function ˆ a ( · ) =
Revised: We remark that there is randomness with respect to the randomly generated row vector ˆ w (cid:96)i , a randomly drawn input x ∼ D , and the function ˆ a ( · ) =

Original: ˆ a (cid:96) − 1 ( · ) deﬁned by the randomly generated matrices ˆ W 2 , . .
Revised: ˆ a (cid:96) − 1 ( · ) deﬁned by the randomly generated matrices ˆ W 2 , . .

Original: , ˆ W (cid:96) − 1 in the previous layers.
Revised: , ˆ W (cid:96) − 1 in the previous layers.

Original: Unless otherwise stated, we will henceforth use the shorthand notation P ( · ) to denote P ˆ w (cid:96) ,x, ˆ a (cid:96) − 1 ( · ) .
Revised: Unless otherwise stated, we will henceforth use the shorthand notation P ( · ) to denote P ˆ w (cid:96) ,x, ˆ a (cid:96) − 1 ( · ) .

Original: Moreover, for ease of presentation, we will ﬁrst condition on the event E 1 / 2 that ˆ a ( x ) ∈ (1 ± 1 / 2 )
Revised: Moreover, for ease of presentation, we will ﬁrst condition on the event E 1 / 2 that ˆ a ( x ) ∈ (1 ± 1 / 2 )

Original: a ( x ) holds.
Revised: a ( x ) holds.

Original: This conditioning will simplify the preliminary analysis and will be removed in our subsequent results.
Revised: This conditioning will simplify the preliminary analysis and will be removed in our subsequent results.

Original: Lemma 1 (Positive-Weights Sparsiﬁcation) .
Revised: Lemma 1 (Positive-Weights Sparsiﬁcation) .

Original: Let ε, δ ∈ (0 , 1) , and x ∼
Revised: Let ε, δ ∈ (0 , 1) , and x ∼

Original: D .
Revised: D .

Original: S PARSIFY ( W , w, ε, δ, S , a ( · )) generates a row vector
Revised: S PARSIFY ( W , w, ε, δ, S , a ( · )) generates a row vector

Original: ˆ w such that
Revised: ˆ w such that

Original: [Equation]
Revised: [Equation]

Original:  where nnz( ˆ w ) ≤ (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε
Revised: η where nnz( ˆ w ) ≤ (cid:108) 8 S log( [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original:  and S = (cid:80) 
Revised: ε(cid:109) , and S = (cid:80) j ∈W s j .

Original: [Equation]
Revised: [Equation]

Original: 5. I MPORTANCE S AMPLING B OUNDS
Revised: MPORTANCE S AMPLING B OUNDS

Original: We now relax the requirement that the weights are strictly positive and instead consider the following index sets that partition the weighted edges: W + = { j ∈ [ η (cid:96) − 1 ] : w j > 0 } and W − = { j ∈ [ η (cid:96) − 1 ] : w j < 0 } .
Revised: We now relax the requirement that the weights are strictly positive and instead consider the following index sets that partition the weighted edges: W + = { j ∈ [ η (cid:96) − 1 ] : w j > 0 } and W − = { j ∈ [ η (cid:96) − 1 ] : w j < 0 } .

Original: We still assume that the incoming activations from the previous layers are positive (this assumption can be relaxed as discussed in Appendix A.2.4).
Revised: We still assume that the incoming activations from the previous layers are positive (this assumption can be relaxed as discussed in Appendix A.2.4).

Original: We deﬁne ∆ (cid:96)i ( x ) for a point x ∼ D and neuron i ∈
Revised: We deﬁne ∆ (cid:96)i ( x ) for a point x ∼ D and neuron i ∈

Original: [ η (cid:96) ] as ∆ (cid:96)i [Equation]
Revised: [ η (cid:96) ] as ∆ (cid:96)i ( x ) = (cid:80) k ∈

Original: =
Revised: [ η(cid:96) − 1] | w (cid:96)ik a (cid:96) − 1 k ( x ) | | (cid:80)

Original: [Equation]
Revised: [Equation]

Original: k ∈ [ η(cid:96) − 1] w (cid:96)ik a (cid:96) − 1 k ( x ) | .
Revised: ( x ) | .

Original: The following assumption serves a similar purpose as does Assumption 1 in that it enables us to approximate the random variable ∆ (cid:96)i ( x ) via an empirical estimate over a small-sized sample of data points S ⊆ P .
Revised: The following assumption serves a similar purpose as does Assumption 1 in that it enables us to approximate the random variable ∆ (cid:96)i ( x ) via an empirical estimate over a small-sized sample of data points S ⊆ P .

Original: Assumption 2 (Subexponentiality of ∆ (cid:96)i ( x ) ) .
Revised: Assumption 2 (Subexponentiality of ∆ (cid:96) i ( x ) ) .

Original: For any layer (cid:96) ∈ { 2 , . . .
Revised: For any layer (cid:96) ∈ { 2 , . . .

Original: , L } and neuron i ∈
Revised: , L } and neuron i ∈

Original: [ η (cid:96) ] , the centered random variable ∆ =
Revised: [ η (cid:96) ] , the centered random variable ∆ =

Original: ∆ (cid:96)i ( x ) − E x ∼D [∆ (cid:96)i ( x )] is subexponential (Vershynin, 2016) with parameter λ ≤ log( η η ∗ ) / 2 , i.e., E [exp ( s ∆)]
Revised: ∆ (cid:96)i ( x ) − E x ∼D [∆ (cid:96)i ( x )] is subexponential (Vershynin, 2016) with parameter λ ≤ log( η η ∗ ) / 2 , i.e., E [exp ( s ∆)]

Original: ≤ exp( s 2 λ 2 ) ∀| s | ≤ 1 λ .
Revised: ≤ exp( s 2 λ 2 ) ∀| s | ≤ 1 λ .

Original: For ε ∈ (0 , 1) and (cid:96) ∈ { 2 , . . .
Revised: For ε ∈ (0 , 1) and (cid:96) ∈ { 2 , . . .

Original: , L } , we let ε (cid:48) =
Revised: , L } , we let ε (cid:48) =

Original: ε 2( L − 1) and deﬁne ε (cid:96) =
Revised: [Equation]

Original:  ε(cid:48) ˆ∆ (cid:96) →
Revised: ) and deﬁne ε (cid:96) =

Original: = ε 2( L − 1) (cid:81) Lk
Revised: [Equation]

Original: = (cid:96) ˆ∆ k , where ˆ∆ (cid:96) = (cid:16) 1 |S| max i ∈
Revised: , where ˆ∆ (cid:96) = (cid:16) 1 |S| max i ∈

Original: [ η (cid:96) ] (cid:80) x (cid:48) ∈S ∆ (cid:96)i ( x (cid:48) )
Revised: [ η (cid:96) ] (cid:80) x (cid:48) ∈S ∆ (cid:96)i ( x (cid:48) )

Original: (cid:17) + κ .
Revised: (cid:17) + κ .

Original: To formalize the interlayer dependencies, for each i ∈ [ η (cid:96) ] we let E (cid:96) i denote the (desirable) event that ˆ z (cid:96) i ( x ) ∈(1 ± 2 ( (cid:96) − 1)
Revised: To formalize the interlayer dependencies, for each i ∈ [ η (cid:96) ] we let E (cid:96)i denote the (desirable) event that ˆ z (cid:96)i ( x ) ∈

Original:  ε (cid:96) +1 )
Revised: (1 ± 2 ( (cid:96) − 1) ε (cid:96) +1 )

Original: z (cid:96) i ( x ) holds, and let
Revised: z (cid:96)i ( x ) holds, and let

Original: E (cid:96) =
Revised: E (cid:96) =

Original: ∩ i ∈
Revised: ∩ i ∈

Original: [ η (cid:96) ]
Revised: [ η (cid:96) ]

Original: E (cid:96)i be the intersection over the events corresponding to each neuron in layer (cid:96) 
Revised: E (cid:96)i be the intersection over the events corresponding to each neuron in layer (cid:96) .

Original: Lemma 2 (Conditional Neuron Value Approximation) .
Revised: Lemma 2 (Conditional Neuron Value Approximation) .

Original: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .
Revised: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .

Original: , L } , i ∈
Revised: , L } , i ∈

Original: [ η (cid:96) ] , and x ∼ D .
Revised: [ η (cid:96) ] , and x ∼ D .

Original: C ORE N ET generates a row vector ˆ w (cid:96)i =
Revised: C ORE N ET generates a row vector ˆ w (cid:96)i =

Original: ˆ w (cid:96) + i − ˆ w (cid:96) − i 
Revised: ˆ w (cid:96) + i − ˆ w (cid:96) − i ∈ R 1 × η (cid:96) − 1 such that

Original: [Equation]
Revised: [Equation]

Original: η
Revised: where ε

Original: [Equation]
Revised: [Equation]

Original: − 1 such that
Revised: =

Original: [Equation]
Revised: [Equation]

Revised: and

Original: [Equation]
Revised: [Equation]

Original: 1) where ε (cid:96) = ε (cid:48) ˆ∆ (cid:96) → and nnz( ˆ w (cid:96)i ) ≤ (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε (cid:96)(cid:109) + 1 , where S = 
Revised: 1 , where S = (cid:80) j ∈W + s j + (cid:80) j

Original: (cid:80)
Revised: [Equation]

Original: j ∈W + s j + (cid:80) j ∈W − s j .
Revised: − s j .

Original: The following core result establishes unconditional layer-wise approximation guarantees and culminates in our main compression theorem.
Revised: The following core result establishes unconditional layer-wise approximation guarantees and culminates in our main compression theorem.

Original: Lemma 3 (Layer-wise Approximation) .
Revised: Lemma 3 (Layer-wise Approximation) .

Original: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .
Revised: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .

Original: , L } , and x ∼ D .
Revised: , L } , and x ∼ D .

Original: C ORE N ET generates a sparse weight matrix ˆ W (cid:96) ∈ R η (cid:96)
Revised: C ORE N ET generates a sparse weight matrix ˆ W (cid:96) ∈ R η (cid:96)

Original: × η (cid:96) − 1 such that, for ˆ z (cid:96) ( x ) =
Revised: × η (cid:96) − 1 such that, for ˆ z (cid:96) ( x ) =

Original: ˆ W (cid:96) ˆ a (cid:96) ( x ) ,
Revised: ˆ W (cid:96) ˆ a (cid:96) ( x ) ,

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: W

Original: [Equation]

Original: [Equation]

Original: ˆ W (cid:96)

Original: [Equation]

Original: ,x ( E (cid:96)

Original: [Equation]

Original: =
Revised: =

Original: P

Original: [Equation]

Original: [Equation]

Original: ˆ

Original: [Equation]
Revised: [Equation]

Original: (cid:96) [Equation] [Equation] (cid:48) η .
Revised: η .

Original: Theorem 4 (Network Compression) .
Revised: Theorem 4 (Network Compression) .

Original: For ε, δ ∈ (0 , 1) , Algorithm 1 generates a set of parameters ˆ θ =
Revised: For ε, δ ∈ (0 , 1) , Algorithm 1 generates a set of parameters ˆ θ =

Original: ( ˆ W 2 , . .
Revised: ( ˆ W 2 , . .

Original: , ˆ W L ) of size
Revised: , ˆ W L ) of size

Original: [Equation]
Revised: [Equation]

Original:  in O (cid:0) η η ∗ log (cid:0) η η ∗ /δ
Revised: log(8 η/δ ) ε(cid:39) +(cid:33) in O (cid:0) η η ∗ log (cid:0) η η ∗ /δ

Original: (cid:1)(cid:1) time such that P
Revised: (cid:1)(cid:1) time such that P

Original: ˆ θ,x ∼D (cid:0)
Revised: ˆ θ,x ∼D (cid:0)

Original: f
Revised: f

Original: ˆ θ ( x ) ∈ (1 ± ε ) f
Revised: ˆ θ ( x ) ∈ (1 ± ε ) f

Original: θ ( x ) (cid:1) ≥ 1 − δ .
Revised: θ ( x ) (cid:1) ≥ 1 − δ .

Original: We note that we can obtain a guarantee for a set of n randomly drawn points by invoking Theoremwith δ (cid:48)
Revised: We note that we can obtain a guarantee for a set of n randomly drawn points by invoking Theoremwith δ (cid:48)

Original: = δ/n and union-bounding over the failure probabilities, while only increasing the sampling complexity logarithmically, as formalized in Corollary 12, Appendix A.2.
Revised: = δ/n and union-bounding over the failure probabilities, while only increasing the sampling complexity logarithmically, as formalized in Corollary 12, Appendix A.2.

Original: 5. G ENERALIZATION B OUNDS
Revised: 5. G ENERALIZATION B OUNDS

Original: As a corollary to our main results, we obtain novel generalization bounds for neural networks in terms of empirical sensitivity.
Revised: As a corollary to our main results, we obtain novel generalization bounds for neural networks in terms of empirical sensitivity.

Original: Following the terminology of Arora et al.
Revised: Following the terminology of Arora et al.

Original: (2018), the expected margin loss of a classiﬁer f θ : R d → R k parameterized by θ with respect to a desired margin γ > 0 and distribution D is deﬁned by L γ ( f θ ) =
Revised: (2018), the expected margin loss of a classiﬁer f θ : R d → R k parameterized by θ with respect to a desired margin γ > 0 and distribution D is deﬁned by L γ ( f θ ) =

Original: P ( x,y )
Revised: P ( x,y )

Original: ∼D X , Y ( f θ ( x ) y ≤ γ + max
Revised: ∼D X , Y ( f θ ( x ) y ≤ γ + max

Original: i
Revised: i

Original: cid:54) =
Revised: cid:54) =

Original: y f θ ( x ) i ) .
Revised: y f θ ( x ) i ) .

Original: We let ˆ L γ denote the empirical estimate of the margin loss.
Revised: We let ˆ L γ denote the empirical estimate of the margin loss.

Original: The following corollary follows directly from the argument presented in Arora et al.
Revised: The following corollary follows directly from the argument presented in Arora et al.

Original: (2018) and Theorem 4.
Revised: (2018) and Theorem 4.

Original: Corollary 5 (Generalization Bounds) .
Revised: Corollary 5 (Generalization Bounds) .

Original: For any δ ∈ (0 , 1) and margin γ > 0 , Alg. 1 generates weights ˆ θ such that with probability at least 1 − δ , the expected error L 0 ( f ˆ θ ) with respect to the points in
Revised: For any δ ∈ (0 , 1) and margin γ > 0 , Alg. 1 generates weights ˆ θ such that with probability at least 1 − δ , the expected error L 0 ( f ˆ θ ) with respect to the points in

Original: P ⊆ X , |P| = n , is bounded by
Revised: P ⊆ X , |P| = n , is bounded by

Original: [Equation]
Revised: [Equation]

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we evaluate the practical effectiveness of our compression algorithm on popular benchmark data sets ( MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), and CIFAR10 (Krizhevsky & Hinton, 2009)) and varying fully-connected trained neural network conﬁgurations: 2 to 5 hidden layers, 100 to 1000 hidden units, either ﬁxed hidden sizes or decreasing hidden size denoted by pyramid in the ﬁgures.
Revised: In this section, we evaluate the practical effectiveness of our compression algorithm on popular benchmark data sets ( MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), and CIFAR10 (Krizhevsky & Hinton, 2009)) and varying fully-connected trained neural network conﬁgurations: 2 to 5 hidden layers, 100 to 1000 hidden units, either ﬁxed hidden sizes or decreasing hidden size denoted by pyramid in the ﬁgures.

Original: We further compare the effectiveness of our sampling scheme in reducing the number of non-zero parameters of a network, i.e., in sparsifying the weight matrices, to that of uniform sampling, Singular Value Decomposition (SVD), and current state-of-the-art sampling schemes for matrix sparsiﬁcation (Drineas & Zouzias, 2011; Achlioptas et al., 2013; Kundu & Drineas, 2014), which are based on matrix norms – (cid:96) 1 and (cid:96) 2 (Frobenius).
Revised: We further compare the effectiveness of our sampling scheme in reducing the number of non-zero parameters of a network, i.e., in sparsifying the weight matrices, to that of uniform sampling, Singular Value Decomposition (SVD), and current state-of-the-art sampling schemes for matrix sparsiﬁcation (Drineas & Zouzias, 2011; Achlioptas et al., 2013; Kundu & Drineas, 2014), which are based on matrix norms – (cid:96) 1 and (cid:96) 2 (Frobenius).

Original: The details of the experimental setup and results of additional evaluations may be found in Appendix B.
Revised: The details of the experimental setup and results of additional evaluations may be found in Appendix B.

Original: Experiment Setup We compare against three variations of our compression algorithm:
Revised: Experiment Setup We compare against three variations of our compression algorithm:

Original: (i) sole edge sampling (CoreNet), (ii) edge sampling with neuron pruning (CoreNet+), and (iii) edge sampling with neuron pruning and ampliﬁcation (CoreNet++).
Revised: (i) sole edge sampling (CoreNet), (ii) edge sampling with neuron pruning (CoreNet+), and (iii) edge sampling with neuron pruning and ampliﬁcation (CoreNet++).

Original: For comparison, we evaluated the average relative error in output ( (cid:96) 1 -norm) and average drop in classiﬁcation accuracy relative to the accuracy of the uncompressed network.
Revised: For comparison, we evaluated the average relative error in output ( (cid:96) 1 -norm) and average drop in classiﬁcation accuracy relative to the accuracy of the uncompressed network.

Original: Both metrics were evaluated on a previously unseen test set.
Revised: Both metrics were evaluated on a previously unseen test set.

Original: Results Results for varying architectures and datasets are depicted in Figures 1 and 2 for the average drop in classiﬁcation accuracy and relative error ( (cid:96) 1 -norm), respectively.
Revised: Results Results for varying architectures and datasets are depicted in Figures 1 and 2 for the average drop in classiﬁcation accuracy and relative error ( (cid:96) 1 -norm), respectively.

Original: As apparent from Figure 1, we are able to compress networks to about 15% of their original size without signiﬁcant loss of accuracy for networks trained on MNIST and FashionMNIST , and to about 50% of their original size for CIFAR .
Revised: As apparent from Figure 1, we are able to compress networks to about 15% of their original size without signiﬁcant loss of accuracy for networks trained on MNIST and FashionMNIST , and to about 50% of their original size for CIFAR .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  The simulation results presented in this section validate our theoretical results established in Sec. 5.
Revised: Discussion The simulation results presented in this section validate our theoretical results established in Sec. 5.

Original: In particular, our empirical results indicate that we are able to outperform networks compressed via competingstate-of-the-art methods in matrix sparsiﬁcation across all considered experiments and trials.
Revised: In particular, our empirical results indicate that we are able to outperform networks compressed via competing methods in matrix sparsiﬁcation across all considered experiments and trials.

Original: The results presented in this section further suggest that empirical sensitivity can effectively capture the relative importance of neural network parameters, leading to a more informed importance sampling scheme.
Revised: The results presented in this section further suggest that empirical sensitivity can effectively capture the relative importance of neural network parameters, leading to a more informed importance sampling scheme.

Original: Moreover, the relative performance of our algorithm tends to increase as we consider deeper architectures.
Revised: Moreover, the relative performance of our algorithm tends to increase as we consider deeper architectures.

Original: These ﬁndings suggest that our algorithm may also be effective in compressing modern convolutional architectures, which tend to be very deep [Figure]
Revised: These ﬁndings suggest that our algorithm may also be effective in compressing modern convolutional architectures, which tend to be very deep.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a coresets-based neural network compression algorithm for compressing the parameters of a trained fully-connected neural network in a manner that approximately preserves the network’s output.
Revised: We presented a coresets-based neural network compression algorithm for compressing the parameters of a trained fully-connected neural network in a manner that approximately preserves the network’s output.

Original: Our method and analysis extend traditional coreset constructions to the application of compressing parameters, which may be of independent interest.
Revised: Our method and analysis extend traditional coreset constructions to the application of compressing parameters, which may be of independent interest.

Original: Our work distinguishes itself from prior approaches in that it establishes theoretical guarantees on the approximation accuracy and size of the generated compressed network.
Revised: Our work distinguishes itself from prior approaches in that it establishes theoretical guarantees on the approximation accuracy and size of the generated compressed network.

Original: As a corollary to our analysis, we obtain generalization bounds for neural networks, which may provide novel insights on the generalization properties of neural networks.
Revised: As a corollary to our analysis, we obtain generalization bounds for neural networks, which may provide novel insights on the generalization properties of neural networks.

Original: We empirically demonstrated the practical effectiveness of our compression algorithm on a variety of neural network conﬁgurations and real-world data sets.
Revised: We empirically demonstrated the practical effectiveness of our compression algorithm on a variety of neural network conﬁgurations and real-world data sets.

Original: In future work, we plan to extend our algorithm and analysis to compress Convolutional Neural Networks (CNNs) and other network architectures.
Revised: In future work, we plan to extend our algorithm and analysis to compress Convolutional Neural Networks (CNNs) and other network architectures.

Original: We conjecture that our compression algorithm can be used to reduce storage requirements of neural network models and enable fast inference in practical settings. 
Revised: We conjecture that our compression algorithm can be used to reduce storage requirements of neural network models and enable fast inference in practical settings. A CKNOWLEDGMENTS

Revised: This research was supported in part by the National Science Foundation award IIS-1723943.

Revised: We thank

Revised: Brandon Araki and Kiran Vodrahalli for valuable discussions and helpful suggestions.

Revised: We would also like to thank Kasper Green Larsen, Alan Mathiasen and Allan Gronlund for pointing out an error in an earlier formulation of Lemma

Original: A P ROOFS OF THE A NALYTICAL R ESULTS IN S ECTION
Revised: A P ROOFS OF THE A NALYTICAL R ESULTS IN S ECTION

Original: This section includes the full proofs of the technical results given in Sec. 5.
Revised: This section includes the full proofs of the technical results given in Sec. 5.

Original: A. A NALYTICAL R ESULTS FOR S ECTION 5.2 (I MPORTANCE S AMPLING B OUNDS FOR P OSITIVE W EIGHTS ) A.1. O RDER S TATISTIC S AMPLING
Revised: A. A NALYTICAL R ESULTS FOR S ECTION 5.2 (I MPORTANCE S AMPLING B OUNDS FOR P OSITIVE W EIGHTS ) A.1. O RDER S TATISTIC S AMPLING

Original: We now establish a couple of technical results that will quantify the accuracy of our approximations of edge importance (i.e., sensitivity).
Revised: We now establish a couple of technical results that will quantify the accuracy of our approximations of edge importance (i.e., sensitivity).

Original: Lemma 6.
Revised: Lemma 6.

Original: Let K > 0 be a universal constant and let D be a distribution with CDF F ( · ) satisfying
Revised: Let K > 0 be a universal constant and let D be a distribution with CDF F ( · ) satisfying

Original: F ( M / K ) ≤ exp( − 1 /K ) , where M = min { x ∈ [0 , 1] : F ( x ) = 1 } .
Revised: F ( M / K ) ≤ exp( − 1 /K ) , where M = min { x ∈ [0 , 1] : F ( x ) = 1 } .

Original: Let P =
Revised: Let P =

Original: X 1 , . .
Revised: X 1 , . .

Original: X n } be a set of n = |P| i.i.d.
Revised: X n } be a set of n = |P| i.i.d.

Original: samples each drawn from the distribution D .
Revised: samples each drawn from the distribution D .

Original: Let X n +1 ∼ D be an i.i.d.
Revised: Let X n +1 ∼ D be an i.i.d.

Original: sample.
Revised: sample.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let X max = max X ∈P ; then,
Revised: Let X max = max X ∈P ; then,

Original: [Equation]
Revised: [Equation]

Original: since X 1 , . . .
Revised: since X 1 , . . .

Original: , X n are i.i.d.
Revised: , X n are i.i.d.

Original: [Equation]
Revised: [Equation]

Original: where F ( · ) is the CDF of X ∼ D
Revised: where F ( · ) is the CDF of X ∼ D

Original: [Equation]
Revised: [Equation]

Original: by monotonicity of F =F
Revised: by monotonicity of F =

Original: [Equation]
Revised: [Equation]

Original: M /

Original: [Equation]

Original: ≤exp
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: CDF Assumption , and this completes the proof.
Revised: CDF Assumption , and this completes the proof.

Original: We now proceed to establish that the notion of empirical sensitivity is a good approximation for the relative importance.
Revised: We now proceed to establish that the notion of empirical sensitivity is a good approximation for the relative importance.

Original: For this purpose, let the relative importance ˆ g j ( x ) of an edge j after the previous layers have already been compressed be
Revised: For this purpose, let the relative importance ˆ g j ( x ) of an edge j after the previous layers have already been compressed be

Original: [Equation]
Revised: [Equation]

Original: Lemma 7 (Empirical Sensitivity Approximation) .
Revised: Lemma 7 (Empirical Sensitivity Approximation) .

Original: Let ε ∈
Revised: Let ε ∈

Original: (0 , 1 / 2) , δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .
Revised: (0 , 1 / 2) , δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .

Original: , L } ,
Revised: , L } ,

Original: Consider a set S
Revised: Consider a set S

Original: = { x 1 , . . .
Revised: = { x 1 , . . .

Original: , x n } ⊆ P of size |S| ≥ (cid:100) log (8 η η ∗ /δ )
Revised: , x n } ⊆ P of size |S| ≥ (cid:100) log (8 η η ∗ /δ )

Original: log( η η ∗ ) (cid:101) .
Revised: log( η η ∗ ) (cid:101) .

Original: Then, conditioned on the event E 1 / 2 occurring, i.e., ˆ a ( x ) ∈ (1 ± 1 / 2 ) a ( [Equation]
Revised: Then, conditioned on the event E 1 / 2 occurring, i.e., ˆ a ( x ) ∈ (1 ± 1 / 2 ) a ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: where C = 3 log( η η ∗ ) and W ⊆
Revised: where C = 3 log( η η ∗ ) and W ⊆

Original: [ η (cid:96) − 1 ] .
Revised: [ η (cid:96) − 1 ] .

Original: Proof.
Revised: Proof.

Original: Consider an arbitrary j ∈ W and x (cid:48)
Revised: Consider an arbitrary j ∈ W and x (cid:48)

Original: ∈ S corresponding to g j ( x (cid:48) ) with CDF F j ( · ) and recall that M = min { x ∈ [0 , 1] : F j ( x ) = 1 } as in Assumption 1.
Revised: ∈ S corresponding to g j ( x (cid:48) ) with CDF F j ( · ) and recall that M = min { x ∈ [0 , 1] : F j ( x ) = 1 } as in Assumption 1.

Original: Note that by Assumption 1, we have
Revised: Note that by Assumption 1, we have

Original: [Equation]
Revised: [Equation]

Original: and so the random variables
Revised: and so the random variables

Original: g j ( x (cid:48) ) for x (cid:48) ∈ S satisfy the CDF condition required by Lemma 6.
Revised: g j ( x (cid:48) ) for x (cid:48) ∈ S satisfy the CDF condition required by Lemma 6.

Original: Now let E be the event that K s j < g j ( x ) holds.
Revised: Now let E be the event that K s j < g j ( x ) holds.

Original: Applying Lemma 6, we obtain
Revised: Applying Lemma 6, we obtain

Original: [Equation]
Revised: [Equation]

Original: Now let ˆ E denote the event that the inequality Cs j < ˆ g j ( x ) =
Revised: Now let ˆ E denote the event that the inequality Cs j < ˆ g j ( x ) =

Original: [Equation]
Revised: [Equation]

Original: k  ˆ a k ( x ) holds and note that the right side of the inequality is deﬁned with respect to ˆ g j ( x ) and not g j ( x ) .
Revised: k ∈W w k ˆ a k ( x ) holds and note that the right side of the inequality is deﬁned with respect to ˆ g j ( x ) and not g j ( x ) .

Original: Observe that since we conditioned on the event E 1 / 2 , we have that ˆ a ( x ) ∈ (1 ± 1 / 2 ) a ( x ) .
Revised: Observe that since we conditioned on the event E 1 / 2 , we have that ˆ a ( x ) ∈ (1 ± 1 / 2 ) a ( x ) .

Original: Now assume that event ˆ E holds and note that by the implication above, we have
Revised: Now assume that event ˆ E holds and note that by the implication above, we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from the fact that 1+1 / 2 / 1 − 1 / 2 ≤ 3 .
Revised: where the second inequality follows from the fact that 1+1 / 2 / 1 − 1 / 2 ≤ 3 .

Original: Moreover, since we know that C ≥ 3 K , we conclude that if event ˆ E occurs, we obtain the inequality
Revised: Moreover, since we know that C ≥ 3 K , we conclude that if event ˆ E occurs, we obtain the inequality

Original: [Equation]
Revised: [Equation]

Original: which is precisely the deﬁnition of event E .
Revised: which is precisely the deﬁnition of event E .

Original: Thus, we have shown the conditional implication (cid:0) ˆ E |
Revised: Thus, we have shown the conditional implication (cid:0) ˆ E |

Original: E 1 / 2 (cid:1) ⇒ E , which implies that
Revised: E 1 / 2 (cid:1) ⇒ E , which implies that

Original: [Equation]
Revised: [Equation]

Original: Since our choice of j ∈ W was arbitrary, the bound applies for any j ∈ W .
Revised: Since our choice of j ∈ W was arbitrary, the bound applies for any j ∈ W .

Original: Thus, we have by the union bound
Revised: Thus, we have by the union bound

Original: [Equation]
Revised: [Equation]

Original: In practice, the set S referenced above is chosen to be a subset of the original data points,
Revised: In practice, the set S referenced above is chosen to be a subset of the original data points,

Original: i.e., S ⊆ P (see Alg. 1, Line 2).
Revised: i.e., S ⊆ P (see Alg. 1, Line 2).

Original: Thus, we henceforth assume that the size of the input points |P| is large enough (or the speciﬁed parameter δ ∈ (0 , 1) is sufﬁciently large) so that |P| ≥ |S| .
Revised: Thus, we henceforth assume that the size of the input points |P| is large enough (or the speciﬁed parameter δ ∈ (0 , 1) is sufﬁciently large) so that |P| ≥ |S| .

Original: A.1. P ROOF OF L EMMA
Revised: A.1. P ROOF OF L EMMA

Original: We now state the proof of Lemma 1.
Revised: We now state the proof of Lemma 1.

Original: In this subsection, we establish approximation guarantees under the assumption that the weights are strictly positive.
Revised: In this subsection, we establish approximation guarantees under the assumption that the weights are strictly positive.

Original: The next subsection will then relax this assumption to conclude that a neuron’s value can be approximated well even when the weights are not all positive.
Revised: The next subsection will then relax this assumption to conclude that a neuron’s value can be approximated well even when the weights are not all positive.

Original: Lemma 1 (Positive-Weights Sparsiﬁcation) .
Revised: Lemma 1 (Positive-Weights Sparsiﬁcation) .

Original: Let ε, δ ∈ (0 , 1) , and x ∼
Revised: Let ε, δ ∈ (0 , 1) , and x ∼

Original: D .
Revised: D .

Original: S PARSIFY ( W , w, ε, δ, S , a ( · )) generates a row vector
Revised: S PARSIFY ( W , w, ε, δ, S , a ( · )) generates a row vector

Original: ˆ w such that
Revised: ˆ w such that

Original: [Equation]
Revised: [Equation]

Original: where nnz( ˆ w ) ≤ (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε(cid:109) , and S = (cid:80) j ∈W s j .
Revised: where nnz( ˆ w ) ≤ (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε [Equation] and S = (cid:80) j ∈W s j .

Original: Proof.
Revised: Proof.

Original: Let ε, δ ∈ (0 , 1) be arbitrary.
Revised: Let ε, δ ∈ (0 , 1) be arbitrary.

Original: Moreover, let C be the coreset with respect to the weight indices
Revised: Moreover, let C be the coreset with respect to the weight indices

Original: W ⊆
Revised: W ⊆

Original: [ η (cid:96) − 1 ] used to construct ˆ w .
Revised: [ η (cid:96) − 1 ] used to construct ˆ w .

Original: Note that as in S PARSIFY , C is a multiset sampled from W of size m = (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε(cid:109) , where
Revised: Note that as in S PARSIFY , C is a multiset sampled from W of size m = (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε(cid:109) , where

Original: S =
Revised: S =

Original: (cid:80) j ∈W s j and C is sampled according to the probability distribution q deﬁned by
Revised: (cid:80) j ∈W s j and C is sampled according to the probability distribution q deﬁned by

Original: [Equation]
Revised: [Equation]

Original:  Let ˆa ( · ) be an arbitrary realization of the random variable ˆ a (cid:96) − 1 ( · ) , let x be a realization of x ∼ D , and let
Revised: W . Let ˆa ( · ) be an arbitrary realization of the random variable ˆ a (cid:96) − 1 ( · ) , let x be a realization of x ∼ D , and let

Original: [Equation]
Revised: [Equation]

Original: be the approximate intermediate value corresponding to the sparsiﬁed matrix ˆ w and let
Revised: be the approximate intermediate value corresponding to the sparsiﬁed matrix ˆ w and let

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne E to be the (favorable) event that ˆ z ε -approximates ˜
Revised: Now deﬁne E to be the (favorable) event that ˆ z ε -approximates ˜

Original: z , i.e., ˆ z ∈ (1 ± ε )˜
Revised: z , i.e., ˆ z ∈ (1 ± ε )˜

Original: z , We will now show that the complement of this event, E c , occurs with sufﬁciently small probability.
Revised: z , We will now show that the complement of this event, E c , occurs with sufﬁciently small probability.

Original: Let Z ⊆ supp( D ) be the set of well-behaved points (deﬁned implicitly with respect to neuron i ∈
Revised: Let Z ⊆ supp( D ) be the set of well-behaved points (deﬁned implicitly with respect to neuron i ∈

Original: [ η (cid:96) ] and realization ˆa ) and deﬁned as follows:
Revised: [ η (cid:96) ] and realization ˆa ) and deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where C = 3 log( η η ∗ ) .
Revised: where C = 3 log( η η ∗ ) .

Original: Let E Z denote the event that x ∈ Z where x is a realization of x ∼ D .
Revised: Let E Z denote the event that x ∈ Z where x is a realization of x ∼ D .

Original: Conditioned on E Z , event E c occurs with probability ≤ δ 4 η : Let x be a realization of x ∼ D such that x ∈ Z and let C
Revised: Conditioned on E Z , event E c occurs with probability ≤ δ 4 η : Let x be a realization of x ∼ D such that x ∈ Z and let C

Original: = { c 1 , . .
Revised: = { c 1 , . .

Original: , c m } be m samples from W with respect to distribution q as before.
Revised: , c m } be m samples from W with respect to distribution q as before.

Original: Deﬁne m random variables T c 1 , . .
Revised: Deﬁne m random variables T c 1 , . .

Original: , T c m such that for all j ∈ C
Revised: , T c m such that for all j ∈ C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For any j ∈ C , we have for the conditional expectation of T j :
Revised: For any j ∈ C , we have for the conditional expectation of T j :

Original: [Equation]
Revised: [Equation]

Original: where we use the expectation notation E [ · ] with the understanding that it denotes the conditional expectation E C | ˆ a l − 1 ( · ) ,x [ · ] .
Revised: where we use the expectation notation E [ · ] with the understanding that it denotes the conditional expectation E C | ˆ a l − 1 ( · ) ,x [ · ] .

Original: Moreover, we also note that conditioning on the event E Z (i.e., the event that x ∈ Z ) does not affect the expectation of T j .
Revised: Moreover, we also note that conditioning on the event E Z (i.e., the event that x ∈ Z ) does not affect the expectation of T j .

Original: Let T =
Revised: Let T =

Original: (cid:80)
Revised: (cid:80)

Original: j ∈C T
Revised: j ∈C T

Original: j
Revised: j

Original: = ˆ z denote our approximation and note that by linearity of expectation,
Revised: = ˆ z denote our approximation and note that by linearity of expectation,

Original: [Equation]
Revised: [Equation]

Original: Thus, ˆ z = T is an unbiased estimator of ˜ z for any realization ˆa ( · ) and x ; thus, we will henceforth refer to E [ T | ˆa ( · ) , x ] as simply ˜ z for brevity.
Revised: Thus, ˆ z = T is an unbiased estimator of ˜ z for any realization ˆa ( · ) and x ; thus, we will henceforth refer to E [ T | ˆa ( · ) , x ] as simply ˜ z for brevity.

Original: For the remainder of the proof we will assume that ˜ z > 0 , since otherwise, ˜ z
Revised: For the remainder of the proof we will assume that ˜ z > 0 , since otherwise, ˜ z

Original: = 0 if and only if T j =for all j ∈ C almost surely, which follows by the fact that T j ≥ 0 for all j ∈ C by deﬁnition of W and the non-negativity of the ReLU activation.
Revised: = 0 if and only if T j =for all j ∈ C almost surely, which follows by the fact that T j ≥ 0 for all j ∈ C by deﬁnition of W and the non-negativity of the ReLU activation.

Original: Therefore, in the case that ˜ z
Revised: Therefore, in the case that ˜ z

Original: = 0 , it follows that
Revised: = 0 , it follows that

Original: [Equation]
Revised: [Equation]

Original: which trivially yields the statement of the lemma, where in the above expression, P ( · ) is short-hand for the conditional probability
Revised: which trivially yields the statement of the lemma, where in the above expression, P ( · ) is short-hand for the conditional probability

Original: P ˆ w | ˆ a l − 1 ( · ) ,x ( · ) .
Revised: P ˆ w | ˆ a l − 1 ( · ) ,x ( · ) .

Original: We now proceed with the case where ˜ z > 0 and leverage the fact that x ∈ Z 3 to establish that for all j ∈ W :
Revised: We now proceed with the case where ˜ z > 0 and leverage the fact that x ∈ Z 3 to establish that for all j ∈ W :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Utilizing the inequality established above, we bound the conditional variance of each T j , j ∈ C as follows
Revised: Utilizing the inequality established above, we bound the conditional variance of each T j , j ∈ C as follows

Original: [Equation]
Revised: [Equation]

Original: where Var ( · ) is short-hand for Var C | ˆ a l − 1 ( · ) ,x ( · ) .
Revised: where Var ( · ) is short-hand for Var C | ˆ a l − 1 ( · ) ,x ( · ) .

Original: Since T is a sum of (conditionally) independent random variables, we obtain
Revised: Since T is a sum of (conditionally) independent random variables, we obtain

Original: [Equation]
Revised: [Equation]

Original: Now, for each j ∈ C let 
Revised: Now, for each j ∈ C let (cid:101)

Original: [Equation]
Revised: [Equation]

Original: and let (cid:101) T = (cid:80) j ∈C (cid:101) T j .
Revised: and let (cid:101) T = (cid:80) j ∈C (cid:101) T j .

Original: Note that by the fact that we conditioned on the realization x of x such that x ∈ Z (event E Z ), we obtain by deﬁnition of T j in (2) and the inequality (3): 
Revised: Note that by the fact that we conditioned on the realization x of x such that x ∈ Z (event E Z ), we obtain by deﬁnition of T j in (2) and the inequality (3): [Equation]

Original: [Equation]
Revised: [Equation]

Original: We also have that S ≥ 1 by deﬁnition.
Revised: We also have that S ≥ 1 by deﬁnition.

Original: More speciﬁcally, using the fact that the maximum over a set is greater than the average and rearranging sums, we obtain
Revised: More speciﬁcally, using the fact that the maximum over a set is greater than the average and rearranging sums, we obtain

Original: [Equation]
Revised: [Equation]

Original: Thus, the inequality established in (5) with the fact that S ≥ 1 we obtain an upper bound on the absolute value of the centered random variables:
Revised: Thus, the inequality established in (5) with the fact that S ≥ 1 we obtain an upper bound on the absolute value of the centered random variables:

Original: [Equation]
Revised: [Equation]

Original: which follows from the fact that: if T j ≥ ˜ zm : Then, by our bound in (5) and the fact that ˜ zm ≥ 0 , it follows that
Revised: which follows from the fact that: if T j ≥ ˜ zm : Then, by our bound in (5) and the fact that ˜ zm ≥ 0 , it follows that

Original: [Equation]
Revised: [Equation]

Original: if T j < ˜ zm : Then, using the fact that T j ≥ 0 and S ≥ 1 , we obtain
Revised: if T j < ˜ zm : Then, using the fact that T j ≥ 0 and S ≥ 1 , we obtain

Original: [Equation]
Revised: [Equation]

Original: Applying Bernstein’s inequality to both (cid:101) T and − (cid:101) T we have by symmetry and the union bound,
Revised: Applying Bernstein’s inequality to both (cid:101) T and − (cid:101) T we have by symmetry and the union bound,

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows by our upper bounds on Var ( T | ˆa ( · ) , x ) and (cid:12)(cid:12)(cid:12) (cid:101) T j (cid:12)(cid:12)(cid:12) and the fact that ε ∈ (0 , 1) , and the last inequality follows by our choice of m = (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε(cid:109) .
Revised: where the second inequality follows by our upper bounds on Var ( T | ˆa ( · ) , x ) and (cid:12)(cid:12)(cid:12) (cid:101) T j (cid:12)(cid:12)(cid:12) and the fact that ε ∈ (0 , 1) , and the last inequality follows by our choice of m = (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε(cid:109) .

Original: This establishes that for any realization ˆa ( · ) of ˆ a l − 1 ( · ) and a realization x of x satisfying x ∈ Z , the event
Revised: This establishes that for any realization ˆa ( · ) of ˆ a l − 1 ( · ) and a realization x of x satisfying x ∈ Z , the event

Original: E c occurs with probability at most δ 4 η .
Revised: E c occurs with probability at most δ 4 η .

Original: Removing the conditioning on E Z : We have by law of total probability
Revised: Removing the conditioning on E Z : We have by law of total probability

Original: [Equation]
Revised: [Equation]

Original: where the second-to-last inequality follows from the fact that P ( E c | ˆa ( · ) , x , E Z , E 1 / 2 ) ≤ δ 4 η as was established above and the last inequality follows by Lemma 7.
Revised: where the second-to-last inequality follows from the fact that P ( E c | ˆa ( · ) , x , E Z , E 1 / 2 ) ≤ δ 4 η as was established above and the last inequality follows by Lemma 7.

Original: Putting it all together Finally, we marginalize out the random variable ˆ a (cid:96) − 1 ( · ) to establish
Revised: Putting it all together Finally, we marginalize out the random variable ˆ a (cid:96) − 1 ( · ) to establish

Original: [Equation]
Revised: [Equation]

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: and this concludes the proof.
Revised: and this concludes the proof.

Original: A. A NALYTICAL R ESULTS FOR S ECTION 5.3 (I MPORTANCE S AMPLING B OUNDS )
Revised: A. A NALYTICAL R ESULTS FOR S ECTION 5.3 (I MPORTANCE S AMPLING B OUNDS )

Original: We begin by establishing an auxiliary result that we will need for the subsequent lemmas.
Revised: We begin by establishing an auxiliary result that we will need for the subsequent lemmas.

Original: A.2. E MPIRICAL ∆ (cid:96)i A PPROXIMATION
Revised: A.2. E MPIRICAL ∆ (cid:96)i A PPROXIMATION

Original: Lemma 8 (Empirical ∆ (cid:96)i Approximation) .
Revised: Lemma 8 (Empirical ∆ (cid:96)i Approximation) .

Original: Let δ ∈ (0 , 1) , λ ∗ = log( η η ∗ ) / 2 , and deﬁne
Revised: Let δ ∈ (0 , 1) , λ ∗ = log( η η ∗ ) / 2 , and deﬁne

Original: [Equation]
Revised: [Equation]

Original: where κ = √ 2 λ ∗ (cid:0) 1 + √ 2 λ ∗ log (8 η η ∗ /δ ) (cid:1) and S ⊆
Revised: where κ = √ 2 λ ∗ (cid:0) 1 + √ 2 λ ∗ log (8 η η ∗ /δ ) (cid:1) and S ⊆

Original: P is as in Alg. 1.
Revised: P is as in Alg. 1.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne the random variables
Revised: Deﬁne the random variables

Original: Y x (cid:48) =
Revised: Y x (cid:48) =

Original: E [∆ (cid:96)i ( x (cid:48) )] − ∆ (cid:96)i ( x (cid:48) ) for each x (cid:48) ∈ S and consider the sum
Revised: E [∆ (cid:96)i ( x (cid:48) )] − ∆ (cid:96)i ( x (cid:48) ) for each x (cid:48) ∈ S and consider the sum

Original: Y = (cid:88) x (cid:48) ∈S
Revised: Y = (cid:88) x (cid:48) ∈S

Original: Y x (cid:48) =
Revised: Y x (cid:48) =

Original: (cid:88) x (cid:48)
Revised: (cid:88) x (cid:48)

Original: ∈S
Revised: ∈S

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: E [∆ (cid:96)i ( x )] − ∆ (cid:96)i ( x (cid:48)) (cid:1) .
Revised: [Equation] ∆ [Equation] (cid:48)

Revised: [Equation]

Original:  We know that each random variable Y x (cid:48) satisﬁes E [ Y x (cid:48) ]
Revised: (cid:1) . We know that each random variable Y x (cid:48) satisﬁes E [ Y x (cid:48) ]

Original: = 0 and by Assumption 2, is subexponential with parameter λ ≤ λ ∗ .
Revised: = 0 and by Assumption 2, is subexponential with parameter λ ≤ λ ∗ .

Original: Thus, Y is a sum of |S| independent, zero-mean λ ∗ -subexponential random variables, which implies that E [ Y ] = 

Revised: Thus, Y is a sum of |S| independent, zero-mean λ ∗ -subexponential random variables, which implies that E [ Y ] =

Original: and that we can readily apply Bernstein’s inequality for subexponential random variables (Vershynin, 2016) to obtain for t ≥
Revised: and that we can readily apply Bernstein’s inequality for subexponential random variables (Vershynin, 2016) to obtain for t ≥

Original: [Equation]
Revised: [Equation]

Original: Since S = (cid:100) log (8 η η ∗ /δ ) log( η η ∗ ) (cid:101) ≥ log (8 η η ∗ /δ )
Revised: Since S = (cid:100) log (8 η η ∗ /δ ) log( η η ∗ ) (cid:101) ≥ log (8 η η ∗ /δ )

Original: 2 λ ∗ , we have for t = √ 2 λ ∗ ,
Revised: 2 λ ∗ , we have for t = √ 2 λ ∗ ,

Original: [Equation]
Revised: [Equation]

Original: Moreover, for a single Y x , we have by the equivalent deﬁnition of a subexponential random variable (Vershynin, 2016) that for u ≥
Revised: Moreover, for a single Y x , we have by the equivalent deﬁnition of a subexponential random variable (Vershynin, 2016) that for u ≥

Original: [Equation]
Revised: [Equation]

Original: Thus, for u = 2 λ ∗ log (8 η η ∗ /δ )
Revised: Thus, for u = 2 λ ∗ log (8 η η ∗ /δ )

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Therefore, by the union bound, we have with probability at least 1 −
Revised: Therefore, by the union bound, we have with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: ∆

Original: [Equation]
Revised: [Equation]

Original: x )

Original: [Equation]

Original: [Equation]

Original: u

Original: [Equation]

Original: [Equation] ≤ ˆ∆ (cid:96) [Equation] where the last inequality follows by deﬁnition of ˆ∆ (cid:96) .
Revised: ≤ ˆ∆ (cid:96) , where the last inequality follows by deﬁnition of ˆ∆ (cid:96) .

Original: Thus, by the union bound, we have
Revised: Thus, by the union bound, we have

Original: [Equation]
Revised: [Equation]

Original: where the last line follows by deﬁnition of η ∗ ≥ η (cid:96) .
Revised: where the last line follows by deﬁnition of η ∗ ≥ η (cid:96) .

Original: A.2. N OTATION FOR THE S UBSEQUENT A NALYSIS
Revised: A.2. N OTATION FOR THE S UBSEQUENT A NALYSIS

Original: Let ˆ w (cid:96) + i and ˆ w (cid:96) − i denote the sparsiﬁed row vectors generated when S PARSIFY is invoked with ﬁrst two arguments corresponding to ( W + , w (cid:96)i ) and ( W − , − w (cid:96)i ) , respectively (Alg. 1, Line 12).
Revised: Let ˆ w (cid:96) + i and ˆ w (cid:96) − i denote the sparsiﬁed row vectors generated when S PARSIFY is invoked with ﬁrst two arguments corresponding to ( W + , w (cid:96)i ) and ( W − , − w (cid:96)i ) , respectively (Alg. 1, Line 12).

Original: We will at times omit including the variables for the neuron i and layer (cid:96) in the proofs for clarity of exposition, and for example, refer to ˆ w (cid:96) + i and ˆ w (cid:96) − i as simply ˆ w + and ˆ w − , respectively.
Revised: We will at times omit including the variables for the neuron i and layer (cid:96) in the proofs for clarity of exposition, and for example, refer to ˆ w (cid:96) + i and ˆ w (cid:96) − i as simply ˆ w + and ˆ w − , respectively.

Original: Let x ∼ D and deﬁne ˆ z
Revised: Let x ∼ D and deﬁne ˆ z

Original: + ( x ) =
Revised: + ( x ) =

Original: (cid:88) k ∈W + ˆ w + k ˆ a k ( x )
Revised: (cid:88) k ∈W + ˆ w + k ˆ a k ( x )

Original: ≥and ˆ z − ( x ) =
Revised: ≥and ˆ z − ( x ) =

Original: (cid:88) k ∈W − ( − ˆ w − k )
Revised: (cid:88) k ∈W − ( − ˆ w − k )

Original: ˆ a k ( x ) ≥be the approximate intermediate values corresponding to the sparsiﬁed matrices ˆ w + and ˆ w − ; let ˜ z + ( x )
Revised: ˆ a k ( x ) ≥be the approximate intermediate values corresponding to the sparsiﬁed matrices ˆ w + and ˆ w − ; let ˜ z + ( x )

Original: =
Revised: =

Original: (cid:88) k ∈W + w k ˆ a k ( x )
Revised: (cid:88) k ∈W + w k ˆ a k ( x )

Original: ≥and ˜ z − ( x ) =
Revised: ≥and ˜ z − ( x ) =

Original: (cid:88) k ∈W − ( − w k )
Revised: (cid:88) k ∈W − ( − w k )

Original: ˆ a k ( x ) ≥be the corresponding intermediate values with respect to the the original row vector w ; and ﬁnally, let z + ( x )
Revised: ˆ a k ( x ) ≥be the corresponding intermediate values with respect to the the original row vector w ; and ﬁnally, let z + ( x )

Original: = (cid:88) k ∈W + w k a k ( x ) ≥and z − ( x ) =
Revised: = (cid:88) k ∈W + w k a k ( x ) ≥and z − ( x ) =

Original: (cid:88)
Revised: (cid:88)

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: −
Revised: k

Original: [Equation]
Revised: [Equation]

Original: k
Revised: x

Original: [Equation]
Revised: [Equation]

Original: be the true intermediate values corresponding to the positive and negative valued weights.
Revised: ≥be the true intermediate values corresponding to the positive and negative valued weights.

Original: Note that in this context, we have by deﬁnition
Revised: Note that in this context, we have by deﬁnition

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where we used the fact
Revised: where we used the fact

Original: that ˆ w = ˆ w + − ˆ w − ∈ R 1 × η (cid:96) − 1 .
Revised: that ˆ w = ˆ w + − ˆ w − ∈ R 1 × η (cid:96) − 1 .

Original: A.2. P ROOF OF L EMMA
Revised: A.2. P ROOF OF L EMMA

Original: Lemma 2 (Conditional Neuron Value Approximation) .
Revised: Lemma 2 (Conditional Neuron Value Approximation) .

Original: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .
Revised: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .

Original: , L } , i ∈
Revised: , L } , i ∈

Original: [ η (cid:96) ] , and x ∼ D .
Revised: [ η (cid:96) ] , and x ∼ D .

Original: C ORE N ET generates a row vector ˆ w (cid:96)i =
Revised: C ORE N ET generates a row vector ˆ w (cid:96)i =

Original: ˆ w (cid:96) + i − ˆ w (cid:96) − i ∈ R 1 × η (cid:96) − 1 such that
Revised: ˆ w (cid:96) + i − ˆ w (cid:96) − i ∈ R 1 × η (cid:96) − 1 such that

Original: [Equation]
Revised: [Equation]

Original: | E (cid:96) − 1 (cid:1) =
Revised: E (cid:96) − 1 (cid:1) =

Original: P (cid:0) ˆ z (cid:96)i ( x ) ∈ (1 ± 2 ( (cid:96) − 1) ε (cid:96) +1 ) z (cid:96)i ( x ) | E (cid:96) − 1 (cid:1) ≥ 1 − δ / η , (1) where ε (cid:96) =
Revised: P (cid:0) ˆ z (cid:96)i ( x ) ∈ (1 ± 2 ( (cid:96) − 1) ε (cid:96) +1 ) z (cid:96)i ( x ) | E (cid:96) − 1 (cid:1) ≥ 1 − δ / η , (1) where ε (cid:96) =

Original: ε (cid:48)
Revised: ε (cid:48)

Original: ˆ ∆ (cid:96) → and nnz( ˆ w (cid:96) i ) ≤ (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε (cid:96)(cid:109) + 1 , where
Revised: ˆ ∆ (cid:96) → and nnz( ˆ w (cid:96) i ) ≤ (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε (cid:96)(cid:109) + 1 , where

Original: S =
Revised: S =

Original: (cid:80)
Revised: (cid:80)

Original: j ∈W + s j + (cid:80) j ∈W − s j .
Revised: j ∈W + s j + (cid:80) j ∈W − s j .

Original: Proof.
Revised: Proof.

Original: Let ε, δ ∈ (0 , 1) be arbitrary and let W + = { j ∈ [ η (cid:96) − 1 ] : w j > 0 } and W − = { j ∈ [ η (cid:96) − 1 ] : w j < 0 } as in Alg. 1.
Revised: Let ε, δ ∈ (0 , 1) be arbitrary and let W + = { j ∈ [ η (cid:96) − 1 ] : w j > 0 } and W − = { j ∈ [ η (cid:96) − 1 ] : w j < 0 } as in Alg. 1.

Original: Let ε (cid:96) be deﬁned as before, ε (cid:96) =
Revised: Let ε (cid:96) be deﬁned as before, ε (cid:96) =

Original: ε (cid:48) ˆ∆ (cid:96) → , where ˆ∆ (cid:96) → = (cid:81)
Revised: ε (cid:48) ˆ∆ (cid:96) → , where ˆ∆ (cid:96) → = (cid:81)

Original: Lk =
Revised: Lk =

Original: (cid:96) ˆ∆
Revised: (cid:96) ˆ∆

Original: k and ˆ∆
Revised: k and ˆ∆

Original: [Equation]
Revised: [Equation]

Original: Observe that w j > 0 ∀ j ∈ W + and similarly, for all ( − w j ) > 0 ∀ j ∈ W − .
Revised: Observe that w j > 0 ∀ j ∈ W + and similarly, for all ( − w j ) > 0 ∀ j ∈ W − .

Original: That is, each of index sets W + and W − corresponds to strictly positive entries in the arguments w (cid:96)i and − w (cid:96)i , respectively passed into S PARSIFY .
Revised: That is, each of index sets W + and W − corresponds to strictly positive entries in the arguments w (cid:96)i and − w (cid:96)i , respectively passed into S PARSIFY .

Original: Observe that since we conditioned on the event E (cid:96) − 1 , we have
Revised: Observe that since we conditioned on the event E (cid:96) − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Since ˆ∆ k ≥ 2 ∀ k ∈ { (cid:96), . . .
Revised: Since ˆ∆ k ≥ 2 ∀ k ∈ { (cid:96), . . .

Original: , L } ≤ ε 2 , where the inequality ˆ∆ k ≥ 2 follows from the fact that
Revised: , L } ≤ ε 2 , where the inequality ˆ∆ k ≥ 2 follows from the fact that

Original: [Equation]
Revised: [Equation]

Original: Since ∆ (cid:96)i ( x (cid:48) ) ≥ 1 ∀ x (cid:48) ∈ supp( D ) by deﬁnition ≥ 2 .
Revised: Since ∆ (cid:96)i ( x (cid:48) ) ≥ 1 ∀ x (cid:48) ∈ supp( D ) by deﬁnition ≥ 2 .

Original: we obtain that ˆ a ( x ) ∈ (1 ± ε / 2 ) a ( x ) , where, as before, ˆ a and a are shorthand notations for ˆ a (cid:96) − 1 ∈ R η (cid:96) − 1 × 1 and a (cid:96) − 1 ∈ R η (cid:96) − 1 × 1 , respectively.
Revised: we obtain that ˆ a ( x ) ∈ (1 ± ε / 2 ) a ( x ) , where, as before, ˆ a and a are shorthand notations for ˆ a (cid:96) − 1 ∈ R η (cid:96) − 1 × 1 and a (cid:96) − 1 ∈ R η (cid:96) − 1 × 1 , respectively.

Original: This implies that E (cid:96) − 1 ⇒ E 1 / 2 and since m = (cid:108) 8 S log( η η ∗ )log(8 η/δ ) ε(cid:109) in Alg.
Revised: This implies that E (cid:96) − 1 ⇒ E 1 / 2 and since m = (cid:108) 8 S log( [Equation] [Equation] [Equation]

Original:  2 we can invoke Lemma 1 with ε = ε (cid:96) on each of the S PARSIFY invocations to conclude that
Revised: [Equation] [Equation] (cid:109) in Alg. 2 we can invoke Lemma 1 with ε = ε (cid:96) on each of the S PARSIFY invocations to conclude that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Therefore, by the union bound, we have
Revised: Therefore, by the union bound, we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, by Lemma 8, we have with probability at most δ 4 η that
Revised: Moreover, by Lemma 8, we have with probability at most δ 4 η that

Original: [Equation]
Revised: [Equation]

Original: Thus, by the union bound over the failure events, we have that with probability at least 1 − ( 3 δ / 4 η + δ / 4 η ) = 1 − δ / η that both of the following events occur
Revised: Thus, by the union bound over the failure events, we have that with probability at least 1 − ( 3 δ / 4 η + δ / 4 η ) = 1 − δ / η that both of the following events occur

Original: ˆ z + ( x ) ∈ (1 ± [Equation] (cid:96) 
Revised: ˆ z + ( x ) ∈ (1 ± ε (cid:96) )

Original: [Equation]

Original: ˜ z
Revised: ˜ z

Original: +

Original: [Equation] x ) and ˆ z − ( x ) 
Revised: + ( x ) and ˆ z − ( x ) ∈ (1 ± ε

Revised: (cid:96) )

Original: ∈ (
Revised: ˜ z

Original: [Equation]
Revised: [Equation]

Original: ∆ (cid:96)i ( x ) ≤ ˆ∆ 
Revised: ∆ (cid:96)i ( x ) ≤ ˆ∆ (cid:96)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Recall that ε (cid:48) = 
Revised: Recall that ε (cid:48) = ε

Original: ε 2( L −1) , ε (cid:96) =
Revised: L − ε (cid:96) =

Original: ε (cid:48) ˆ∆ (cid:96) → , and that event E (cid:96) i denotes the (desirable) event that
Revised: ε (cid:48) ˆ∆ (cid:96) → , and that event E (cid:96) i denotes the (desirable) event that

Original: [Equation]
Revised: [Equation]

Original: holds, and similarly, E (cid:96) =
Revised: holds, and similarly, E (cid:96) =

Original: ∩ i ∈
Revised: ∩ i ∈

Original: [ η (cid:96) ]
Revised: [ η (cid:96) ]

Original: E (cid:96)i denotes the vector-wise analogue where
Revised: E (cid:96)i denotes the vector-wise analogue where

Original: [Equation]
Revised: [Equation]

Original: Let k = 2 ( (cid:96) − 1) and note that by conditioning on the event E (cid:96) − 1 , i.e., we have by deﬁnition
Revised: Let k = 2 ( (cid:96) − 1) and note that by conditioning on the event E (cid:96) [Equation] 1 , i.e., we have by deﬁnition

Original: [Equation]
Revised: [Equation]

Original: which follows by deﬁnition of the ReLU function.
Revised: which follows by deﬁnition of the ReLU function.

Original: Recall that our overarching goal is to establish that
Revised: Recall that our overarching goal is to establish that

Original: [Equation]
Revised: [Equation]

Original: which would immediately imply by deﬁnition of the ReLU function that
Revised: which would immediately imply by deﬁnition of the ReLU function that

Original: [Equation]
Revised: [Equation]

Original: Having clariﬁed the conditioning and our objective, we will once again drop the index i from the expressions moving forward.
Revised: Having clariﬁed the conditioning and our objective, we will once again drop the index i from the expressions moving forward.

Original: Proceeding from above, we have with probability at least 1 − δ / η ˆ
Revised: Proceeding from above, we have with probability at least 1 − δ / η ˆ

Original: [Equation]
Revised: [Equation]

Original: z

Original: [Equation]

Original: By Event (7) above
Revised: By Event (7) above

Original: [Equation]
Revised: [Equation]

Original: Conditioning on eventE
Revised: Conditioning on event

Original: [Equation]
Revised: [Equation]

Original: 1 + k ε 2 (cid:96) (cid:1)
Revised: 1 + k ε 2 (cid:96) (cid:1)

Original: z
Revised: z

Original: ( x ) + ( k + 1)
Revised: ( x ) + ( k + 1)

Original: [Equation]

Original:  (cid:96) (cid:0)z
Revised: ε (cid:96) (cid:0)

Original:  + ( x ) + z − ( x ) (cid:1) =
Revised: z + ( x ) + z − ( x ) (cid:1) =

Original: (cid:0) 1 + k [Equation] (cid:96) (cid:1)
Revised: (cid:0) 1 + k ε 2 (cid:96) (cid:1)

Original: z ( x ) + ( k + 1) ε (cid:48) (cid:81)
Revised: z ( x ) + ( k + 1) ε (cid:48) (cid:81)

Original: Lk = (cid:96) ˆ∆
Revised: Lk = (cid:96) ˆ∆

Original: k (cid:0) z + ( x ) + z − ( x ) (cid:1) ≤ (cid:0) 1 + k ε 2 (cid:96) (cid:1)
Revised: k (cid:0) z + ( x ) + z − ( x ) (cid:1) ≤ (cid:0) 1 + k ε 2 (cid:96) (cid:1)

Original: z ( x ) + ( k + 1) ε (cid:48)
Revised: z ( x ) + ( k + 1) ε (cid:48)

Original: ∆ (cid:96)i ( x ) (cid:81)
Revised: ∆ (cid:96)i ( x ) (cid:81)

Original: Lk
Revised: Lk

Original: =
Revised: =

Original: (cid:96) +

Revised: (cid:96) +

Original: ˆ∆ k (cid:0) z
Revised: ˆ∆ k (cid:0) z

Original: + ( x ) + z − ( x ) (cid:1)
Revised: + ( x ) + z − ( x ) (cid:1)

Original: By Event (8) above = (cid:0) 1 + k ε 2 (cid:96) (cid:1) z ( x ) + ( k + 1) ε (cid:48) (cid:81)
Revised: By Event (8) above = (cid:0) 1 + k ε 2 (cid:96) (cid:1) z ( x ) + ( k + 1) ε (cid:48) (cid:81)

Original: Lk = (cid:96) +1 ˆ∆
Revised: Lk = (cid:96) +1 ˆ∆

Original: k | z ( x ) |
Revised: k | z ( x ) |

Original: By ∆ (cid:96)i ( x ) =
Revised: By ∆ (cid:96)i ( x ) =

Original: z
Revised: z

Original: + ( x ) + z − ( x ) |
Revised: + ( x ) + z − ( x ) |

Original: z
Revised: z

Original: ( x ) | = (cid:0) 

Revised: ( x ) | = (cid:0)

Original: + k ε 2 (cid:96) (cid:1)
Revised: + k ε 2 (cid:96) (cid:1)

Original: z ( x ) + ( k + 1)
Revised: z ( x ) + ( k + 1)

Original: ε
Revised: ε

Original: (cid:96) +1 | z ( x ) | .
Revised: (cid:96) +1 | z ( x ) | .

Original: To upper bound the last expression above, we begin by observing that
Revised: To upper bound the last expression above, we begin by observing that

Original: kε 2 (cid:96) ≤ ε (cid:96) , which follows from the fact that ε (cid:96) ≤ 1 2( L − 1) ≤ 1 k by deﬁnition.
Revised: kε 2 (cid:96) ≤ ε (cid:96) , which follows from the fact that ε (cid:96) ≤ 1 2( L − 1) ≤ 1 k by deﬁnition.

Original: Moreover, we also note that ε (cid:96) ≤ ε (cid:96) +1 by deﬁnition of ˆ∆ (cid:96) ≥ 1 .
Revised: Moreover, we also note that ε (cid:96) ≤ ε (cid:96) +1 by deﬁnition of ˆ∆ (cid:96) ≥ 1 .

Original: Now, we consider two cases.
Revised: Now, we consider two cases.

Original: Case of z ( x ) ≥ 0 :
Revised: Case of z ( x ) ≥ 0 :

Original: In this case, we have
Revised: In this case, we have

Original: [Equation]
Revised: [Equation]

Original: where the last line follows by deﬁnition of k = 2 ( (cid:96) − 2) , which implies that k + 2 = 2( (cid:96) − 1) .
Revised: where the last line follows by deﬁnition of k = 2 ( (cid:96) − 2) , which implies that k + 2 = 2( (cid:96) − 1) .

Original: Thus, this establishes the desired upper bound in the case that z ( x ) ≥ 0 .
Revised: Thus, this establishes the desired upper bound in the case that z ( x ) ≥ 0 .

Original: Case of z ( x ) < 0 :
Revised: Case of z ( x ) < 0 :

Original: Since z ( x ) is negative, we have that (cid:0) 1 + k ε 2 (cid:96) (cid:1)
Revised: Since z ( x ) is negative, we have that (cid:0) 1 + k ε 2 (cid:96) (cid:1)

Original: z ( x ) ≤ z ( x ) and | z ( x ) | = − z ( x ) and thus
Revised: z ( x ) ≤ z ( x ) and | z ( x ) | = − z ( x ) and thus

Original: [Equation]
Revised: [Equation]

Original: and this establishes the upper bound for the case of z ( x ) being negative.
Revised: and this establishes the upper bound for the case of z ( x ) being negative.

Original: Putting the results of the case by case analysis together, we have the upper bound of ˆ z ( x ) ≤ z ( x ) + 2 ( (cid:96) − 1) ε (cid:96) +1 |
Revised: Putting the results of the case by case analysis together, we have the upper bound of ˆ z ( x ) ≤ z ( x ) + 2 ( (cid:96) − 1) ε (cid:96) +1 |

Original: z ( x ) | .
Revised: z ( x ) | .

Original: The proof for establishing the lower bound for z ( x ) is analogous to that given above, and yields ˆ z ( x ) ≥ z
Revised: The proof for establishing the lower bound for z ( x ) is analogous to that given above, and yields ˆ z ( x ) ≥ z

Original: ( x ) − 2 ( (cid:96) − 1) ε (cid:96) +1 |
Revised: ( x ) − 2 ( (cid:96) − 1) ε (cid:96) +1 |

Original: z ( x ) | .
Revised: z ( x ) | .

Original: Putting both the upper and lower bound together, we have that with probability at least 1 − δη :
Revised: Putting both the upper and lower bound together, we have that with probability at least 1 − δη :

Original: [Equation]
Revised: [Equation]

Original: , and this completes the proof.
Revised: and this completes the proof.

Original: A.2. R EMARKS ON N EGATIVE A CTIVATIONS
Revised: A.2. R EMARKS ON N EGATIVE A CTIVATIONS

Original: We note that up to now we assumed that the input a ( x ) , i.e., the activations from the previous layer, are strictly nonnegative.
Revised: We note that up to now we assumed that the input a ( x ) , i.e., the activations from the previous layer, are strictly nonnegative.

Original: For layers (cid:96) ∈ { 3 , . . .
Revised: For layers (cid:96) ∈ { 3 , . . .

Original: , L } , this is indeed true due to the nonnegativity of the
Revised: , L } , this is indeed true due to the nonnegativity of the

Original: ReLU activation function.
Revised: ReLU activation function.

Original: For layer 2 , the input is a ( x ) = x , which can be decomposed into a ( x ) = a pos ( x ) − a neg ( x ) , where a pos ( x ) ≥ 0 ∈ R η (cid:96) − 1 and a neg ( x ) ≥ 0 ∈ R η (cid:96) − 1 .
Revised: For layer 2 , the input is a ( x ) = x , which can be decomposed into a ( x ) = a pos ( x ) − a neg ( x ) , where a pos ( x ) ≥ 0 ∈ R η (cid:96) − 1 and a neg ( x ) ≥ 0 ∈ R η (cid:96) − 1 .

Original: Furthermore, we can deﬁne the sensitivity over the set of points { a pos ( x ) , a neg ( x ) | x ∈ S} (instead of { a ( x ) | x ∈ S} ), and thus maintain the required nonnegativity of the sensitivities.
Revised: Furthermore, we can deﬁne the sensitivity over the set of points { a pos ( x ) , a neg ( x ) | x ∈ S} (instead of { a ( x ) | x ∈ S} ), and thus maintain the required nonnegativity of the sensitivities.

Original: Then, in the terminology of Lemma 2, we let
Revised: Then, in the terminology of Lemma 2, we let

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: be the corresponding positive parts, and
Revised: be the corresponding positive parts, and

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: be the corresponding negative parts of the preactivation of the considered layer, such that
Revised: be the corresponding negative parts of the preactivation of the considered layer, such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We also let
Revised: We also let

Original: [Equation]
Revised: [Equation]

Original: be as before, with z + ( x ) and z − ( x ) deﬁned as above.
Revised: be as before, with z + ( x ) and z − ( x ) deﬁned as above.

Original: Equipped with above deﬁnitions, we can rederive Lemma 2 analogously in the more general setting, i.e., with potentially negative activations.
Revised: Equipped with above deﬁnitions, we can rederive Lemma 2 analogously in the more general setting, i.e., with potentially negative activations.

Original: We also note that we require a slightly larger sample size now since we have to take a union bound over the failure probabilities of all four approximations (i.e. ˆ z +pos ( x ) , ˆ z − neg ( x ) , ˆ z +neg ( x ) , and ˆ z − pos ( x ) ) to obtain the desired overall failure probability of δ / η .
Revised: We also note that we require a slightly larger sample size now since we have to take a union bound over the failure probabilities of all four approximations (i.e. ˆ z +pos ( x ) , ˆ z − neg ( x ) , ˆ z +neg ( x ) , and ˆ z − pos ( x ) ) to obtain the desired overall failure probability of δ / η .

Original: A.2. P ROOF OF T HEOREM
Revised: A.2. P ROOF OF T HEOREM

Original: The following corollary immediately follows from Lemma 2 and establishes a layer-wise approximation guarantee.
Revised: The following corollary immediately follows from Lemma 2 and establishes a layer-wise approximation guarantee.

Original: Corollary 9 (Conditional Layer-wise Approximation) .
Revised: Corollary 9 (Conditional Layer-wise Approximation) .

Original: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .
Revised: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .

Original: , L } , and x ∼ D .
Revised: , L } , and x ∼ D .

Original: C ORE N ET generates a sparse weight matrix ˆ W (cid:96) =
Revised: C ORE N ET generates a sparse weight matrix ˆ W (cid:96) =

Original: (cid:0) ˆ w
Revised: (cid:0) ˆ w

Original: [Equation]
Revised: [Equation]

Original: , ˆ w 
Revised: , ˆ w (cid:96)η (cid:96) (cid:1) (cid:62) ∈ R η (cid:96) × η (cid:96) − 1 such that

Original: [Equation]
Revised: [Equation]

Original:  ∈ R η
Revised: z (cid:96) ( x ) ∈ (1 ± 2 ( (cid:96) − 1) ε (cid:96)

Original: [Equation]
Revised: [Equation]

Original: ×

Original: [Equation]
Revised: [Equation]

Original: − 1 such that
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: ˆ z (cid:96) ( x ) ∈ (1 ± 2 ( (cid:96) − 1) ε (cid:96) +1 )
Revised: |

Original: z (cid:96) ( x ) | E (cid:96) − 1 (cid:1) ≥ 1 − δ η (cid:96) η , (9) where ε (cid:96) =
Revised: [Equation] η (cid:96) η , (9) where ε (cid:96) =

Original: ε (cid:48) ˆ∆ (cid:96) → , ˆ z (cid:96) ( x ) =
Revised: ε (cid:48) ˆ∆ (cid:96) → , ˆ z (cid:96) ( x ) =

Original: ˆ W (cid:96) ˆ a (cid:96) ( x ) , and z (cid:96) ( x ) =
Revised: ˆ W (cid:96) ˆ a (cid:96) ( x ) , and z (cid:96) ( x ) =

Original: W (cid:96) a (cid:96) ( x ) .
Revised: W (cid:96) a (cid:96) ( x ) .

Original: Proof.
Revised: Proof.

Original: Since (1) established by Lemma 2 holds for any neuron i ∈
Revised: Since (1) established by Lemma 2 holds for any neuron i ∈

Original: [ η (cid:96) ] in layer (cid:96) and since ( E (cid:96) )
Revised: [ η (cid:96) ] in layer (cid:96) and since ( E (cid:96) )

Original: c
Revised: c

Original: = ∪ i ∈
Revised: = ∪ i ∈

Original: [ η (cid:96) ]
Revised: [ η (cid:96) ]

Original: ( E (cid:96)i ) c , it follows by the union bound over the failure events ( E (cid:96)i ) c for all i ∈
Revised: ( E (cid:96)i ) c , it follows by the union bound over the failure events ( E (cid:96)i ) c for all i ∈

Original: [ η (cid:96) ] that with probability at least 1 − η (cid:96) δηˆ [Equation] [Equation] x
Revised: [ η (cid:96) ] that with probability at least 1 − η (cid:96) δη

Original: [Equation]
Revised: [Equation]

Original: = (
Revised: ) ε

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1) ε (cid:96) +1 ) z (cid:96) ( x ) .
Revised: x ) .

Original: The following lemma removes the conditioning on E (cid:96) − 1 and explicitly considers the (compounding) error incurred by generating coresets ˆ W 2 , . .
Revised: The following lemma removes the conditioning on E (cid:96) − 1 and explicitly considers the (compounding) error incurred by generating coresets ˆ W 2 , . .

Original: , ˆ W (cid:96) for multiple layers.
Revised: , ˆ W (cid:96) for multiple layers.

Original: Lemma 3 (Layer-wise Approximation) .
Revised: Lemma 3 (Layer-wise Approximation) .

Original: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .
Revised: Let ε, δ ∈ (0 , 1) , (cid:96) ∈ { 2 , . . .

Original: , L } , and x ∼ D .
Revised: , L } , and x ∼ D .

Original: C ORE N ET generates a sparse weight matrix 
Revised: C ORE N ET generates a sparse weight matrix ˆ W (cid:96) ∈ R η [Equation] × η (cid:96)

Original: [Equation]
Revised: [Equation]

Original: 1 such that, for ˆ z 
Revised: 1 such that, for ˆ z (cid:96) ( x [Equation] = [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Invoking Corollary 9, we know that for any layer (cid:96) (cid:48) ∈ { 2 , . . .
Revised: Invoking Corollary 9, we know that for any layer (cid:96) (cid:48) ∈ { 2 , . . .

Original: , L } 
Revised: , L } ,

Original: [Equation]
Revised: [Equation]

Original: We also have by the law of total probability that
Revised: We also have by the law of total probability that

Original: [Equation]
Revised: [Equation]

Original: Repeated applications of (10) and (11) in conjunction with the observation that P ( E 1 ) = 1 4 yield
Revised: Repeated applications of (10) and (11) in conjunction with the observation that P ( E 1 ) = 1 4 yield

Original: [Equation]
Revised: [Equation]

Original: Repeated applications of (
Revised: Repeated applications of (11)

Original: [Equation]
Revised: [Equation]

Original: By

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By the Weierstrass Product Inequality , where the last inequality follows by the Weierstrass Product Inequality 5 and this establishes the lemma.
Revised: By the Weierstrass Product Inequality , where the last inequality follows by the Weierstrass Product Inequality 5 and this establishes the lemma.

Original: Appropriately invoking Lemma 3, we can now establish the approximation guarantee for the entire neural network.
Revised: Appropriately invoking Lemma 3, we can now establish the approximation guarantee for the entire neural network.

Original: This is stated in Theorem 4 and the proof can be found below.
Revised: This is stated in Theorem 4 and the proof can be found below.

Original: Theorem 4 (Network Compression) .
Revised: Theorem 4 (Network Compression) .

Original: For ε, δ ∈ (0 , 1) , Algorithm 1 generates a set of parameters ˆ θ =
Revised: For ε, δ ∈ (0 , 1) , Algorithm 1 generates a set of parameters ˆ θ =

Original: ( ˆ W 2 , . .
Revised: ( ˆ W 2 , . .

Original: , ˆ W L ) of size
Revised: , ˆ W L ) of size

Original: [Equation]
Revised: [Equation]

Original: 32 ( L − 1) 2 ( ˆ∆ (cid:96) → ) 2 S (cid:96)i log( η η ∗ ) log(8 η/δ ) ε(cid:39) +(cid:33) in O (cid:0) η η ∗ log (cid:0)
Revised: 32 ( L − 1) 2 ( ˆ∆ (cid:96) → ) 2 S (cid:96)i log( η η ∗ ) log(8 η/δ ) ε(cid:39) +(cid:33) in O (cid:0) η η ∗ log (cid:0)

Original: η η ∗ /δ
Revised: η η ∗ /δ

Original: (cid:1)(cid:1) time such that P
Revised: (cid:1)(cid:1) time such that P

Original: ˆ θ,x ∼D (cid:0)
Revised: ˆ θ,x ∼D (cid:0)

Original: f
Revised: f

Original: ˆ θ ( x ) ∈ (1 ± ε ) f
Revised: ˆ θ ( x ) ∈ (1 ± ε ) f

Original: θ ( x ) (cid:1) ≥ 1 − δ .
Revised: θ ( x ) (cid:1) ≥ 1 − δ .

Original: The Weierstrass Product Inequality (Doerr, 2018) states that for p 1 , . .
Revised: The Weierstrass Product Inequality (Doerr, 2018) states that for p 1 , . .

Original: . , p n ∈ [0 , 1] , n (cid:89)
Revised: . , p n ∈ [0 , 1] , n (cid:89)

Original: i =
Revised: i =

Original: (1 − p i ) ≥ 1 − n (cid:88) i = p i .
Revised: (1 − p i ) ≥ 1 − n (cid:88) i = p i .

Original: Proof.
Revised: Proof.

Original: Invoking Lemma 3 with (cid:96) =
Revised: Invoking Lemma 3 with (cid:96) =

Original: L , we have that for ˆ θ =
Revised: L , we have that for ˆ θ =

Original: ( ˆ W 2 , . .
Revised: ( ˆ W 2 , . .

Original: , ˆ W L ) ,
Revised: , ˆ W L ) ,

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows by deﬁnition of η = (cid:80) L(cid:96)
Revised: where the last equality follows by deﬁnition of η = (cid:80) L(cid:96)

Original: =
Revised: =

Original: 2 η (cid:96) .
Revised: 2 η (cid:96) .

Original: Note that by deﬁnition,
Revised: Note that by deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows by the fact that the empty product (cid:81)
Revised: where the last equality follows by the fact that the empty product (cid:81)

Original: Lk
Revised: Lk

Original: = L [Equation]
Revised: = L +1 ˆ∆

Original: 1 ˆ∆ k is equal to 1.
Revised: k is equal to 1.

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: and so we conclude
Revised: and so we conclude

Original: [Equation]
Revised: [Equation]

Original: which, along with the sampling complexity of Alg. 2 (Line 6), establishes the approximation guarantee provided by the theorem.
Revised: which, along with the sampling complexity of Alg. 2 (Line 6), establishes the approximation guarantee provided by the theorem.

Original: For the computational time complexity, we observe that the most time consuming operation per iteration of the loop on Lines 7-13 is the weight sparsiﬁcation procedure.
Revised: For the computational time complexity, we observe that the most time consuming operation per iteration of the loop on Lines 7-13 is the weight sparsiﬁcation procedure.

Original: The asymptotic time complexity of each S PARSIFY invocation for each neuron i ∈
Revised: The asymptotic time complexity of each S PARSIFY invocation for each neuron i ∈

Original: [ η (cid:96) ] in layers (cid:96) ∈ { 2 , . . .
Revised: [ η (cid:96) ] in layers (cid:96) ∈ { 2 , . . .

Original: , L } (Alg. 1, Line 12) is dominated by the relative importance computation for incoming edges (Alg. 2, Lines 1-2).
Revised: , L } (Alg. 1, Line 12) is dominated by the relative importance computation for incoming edges (Alg. 2, Lines 1-2).

Original: This can be done by evaluating w (cid:96)ik a (cid:96) − 1 k ( x ) for all k ∈ W and x ∈ S , for a total computation time that is bounded above by O (cid:0) |S| η (cid:96) − 1 (cid:1) since |W| ≤ η (cid:96) − 1 for each i ∈
Revised: This can be done by evaluating w (cid:96)ik [Equation] k ( x ) for all k ∈ W and x ∈ S , for a total computation time that is bounded above by O (cid:0) |S| η (cid:96) − 1 (cid:1) since |W| ≤ η (cid:96) − 1 for each i ∈

Original: [ η (cid:96) ] .
Revised: [ η (cid:96) ] .

Original: Thus, S PARSIFY takes O (cid:0) |S| η (cid:96) − 1 (cid:1) time.
Revised: Thus, S PARSIFY takes O (cid:0) |S| η (cid:96) − 1 (cid:1) time.

Original: Summing the computation time over all layers and neurons in each layer, we obtain an asymptotic time complexity of O (cid:0) |S| (cid:80) L(cid:96)
Revised: Summing the computation time over all layers and neurons in each layer, we obtain an asymptotic time complexity of O (cid:0) |S| (cid:80) L(cid:96)

Original: =
Revised: =

Original: 2 η (cid:96) − 1 η (cid:96) (cid:1) ⊆
Revised: 2 η (cid:96) − 1 η (cid:96) (cid:1) ⊆

Original: O ( |S| η ∗ η ) .
Revised: O ( |S| η ∗ η ) .

Original: Since |S| ∈ O (log( η η ∗ /δ )) , we conclude that the computational complexity our neural network compression algorithm is
Revised: Since |S| ∈ O (log( η η ∗ /δ )) , we conclude that the computational complexity our neural network compression algorithm is

Original: [Equation]
Revised: [Equation]

Original: A.2. P ROOF OF T HEOREM
Revised: A.2. P ROOF OF T HEOREM

Original: In order to ensure that the established sampling bounds are non-vacuous in terms of the sensitivity, i.e., not linear in the number of incoming edges, we show that the sum of sensitivities per neuron S is small.
Revised: In order to ensure that the established sampling bounds are non-vacuous in terms of the sensitivity, i.e., not linear in the number of incoming edges, we show that the sum of sensitivities per neuron S is small.

Original: The following lemma establishes that the sum of sensitivities can be bounded instance-independent by a term that is logarithmic in roughly the total number of edges ( η · η ∗ ).
Revised: The following lemma establishes that the sum of sensitivities can be bounded instance-independent by a term that is logarithmic in roughly the total number of edges ( η · η ∗ ).

Original: Lemma 10 (Sensitivity Bound) .
Revised: Lemma 10 (Sensitivity Bound) .

Original: For any (cid:96) ∈ { 2 , . . .
Revised: For any (cid:96) ∈ { 2 , . . .

Original: , L } and i ∈
Revised: , L } and i ∈

Original: [ η (cid:96) ] , the sum of sensitivities S =
Revised: [ η (cid:96) ] , the sum of sensitivities S =

Original: S + + S − is bounded by
Revised: S + + S − is bounded by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider S + for an arbitrary (cid:96) ∈ { 2 , . . .
Revised: Consider S + for an arbitrary (cid:96) ∈ { 2 , . . .

Original: , L } and i ∈
Revised: , L } and i ∈

Original: [ η (cid:96) ] .
Revised: [ η (cid:96) ] .

Original: For all j ∈ W we have the following bound on the sensitivity of a single j ∈ W ,
Revised: For all j ∈ W we have the following bound on the sensitivity of a single j ∈ W ,

Original: [Equation]
Revised: [Equation]

Original: where the inequality follows from the fact that we can upper bound the max by a summation over x ∈ S since g j ( x ) ≥ 0 , ∀ j ∈ W .
Revised: where the inequality follows from the fact that we can upper bound the max by a summation over x ∈ S since g j ( x ) ≥ 0 , ∀ j ∈ W .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that the sum of sensitivities is ﬁnite to swap the order of summation.
Revised: where we used the fact that the sum of sensitivities is ﬁnite to swap the order of summation.

Original: Using the same argument as above, we obtain S − =
Revised: Using the same argument as above, we obtain S − =

Original: (cid:80) j ∈W − s j ≤ |S| , which establishes the lemma.
Revised: (cid:80) j ∈W − s j ≤ |S| , which establishes the lemma.

Original: Note that the sampling complexities established above have a linear dependence on the sum of sensitivities, (cid:80) L(cid:96)
Revised: Note that the sampling complexities established above have a linear dependence on the sum of sensitivities, (cid:80) L(cid:96)

Original: =
Revised: =

Original: 2 (cid:80) η
Revised: 2 (cid:80) η

Original: (cid:96) i =1 S (cid:96)i , which is instance-dependent, i.e., depends on the sampled S ⊆ P and the actual weights of the trained neural network.
Revised: (cid:96) i =1 S (cid:96)i , which is instance-dependent, i.e., depends on the sampled S ⊆ P and the actual weights of the trained neural network.

Original: By applying Lemma 10, we obtain a bound on the size of the compressed network that is independent of the sensitivity.
Revised: By applying Lemma 10, we obtain a bound on the size of the compressed network that is independent of the sensitivity.

Original: Theorem 11 (Sensitivity-Independent Network Compression) .
Revised: Theorem 11 (Sensitivity-Independent Network Compression) .

Original: For any given ε, δ ∈ (0 , 1) our sampling scheme (Alg. 1) generates a set of parameters ˆ θ of size nnz
Revised: For any given ε, δ ∈ (0 , 1) our sampling scheme (Alg. 1) generates a set of parameters ˆ θ of size nnz(

Original: [Equation]
Revised: [Equation]

Original: (cid:32)
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Revised: O (cid:32)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ε

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: in O (cid:0) η η ∗ log (cid:0) η η ∗ /δ
Revised: in O (cid:0) η η ∗ log (cid:0) η η ∗ /δ

Original: (cid:1)(cid:1) time, such that P
Revised: (cid:1)(cid:1) time, such that P

Original: ˆ θ,x ∼D (cid:0)
Revised: ˆ θ,x ∼D (cid:0)

Original: f
Revised: f

Original: ˆ θ ( x ) ∈ (1 ± ε ) f
Revised: ˆ θ ( x ) ∈ (1 ± ε ) f

Original: θ ( x ) (cid:1) ≥ 1 − δ .
Revised: θ ( x ) (cid:1) ≥ 1 − δ .

Original: Proof.
Revised: Proof.

Original: Combining Lemma 10 and Theorem 4 establishes the theorem.
Revised: Combining Lemma 10 and Theorem 4 establishes the theorem.

Original: A.2. G ENERALIZED N ETWORK C OMPRESSION
Revised: A.2. G ENERALIZED N ETWORK C OMPRESSION

Original: Theorem 4 gives us an approximation guarantee with respect to one randomly drawn point x ∼ D .
Revised: Theorem 4 gives us an approximation guarantee with respect to one randomly drawn point x ∼ D .

Original: The following corollary extends this approximation guarantee to any set of n randomly drawn points using a union bound argument, which enables approximation guarantees for, e.g., a test data set composed of n i.i.d.
Revised: The following corollary extends this approximation guarantee to any set of n randomly drawn points using a union bound argument, which enables approximation guarantees for, e.g., a test data set composed of n i.i.d.

Original: points drawn from the distribution.
Revised: points drawn from the distribution.

Original: We note that the sampling complexity only increases by roughly a logarithmic term in n .
Revised: We note that the sampling complexity only increases by roughly a logarithmic term in n .

Original: Corollary 12 (Generalized Network Compression) .
Revised: Corollary 12 (Generalized Network Compression) .

Original: For any ε, δ ∈ (0 , 1) and a set of i.i.d.
Revised: For any ε, δ ∈ (0 , 1) and a set of i.i.d.

Original: input points P (cid:48) of cardinality |P
Revised: input points P (cid:48) of cardinality |P

Original: (cid:48) | ∈ N + , i.e., P (cid:48) i.i.d.
Revised: (cid:48) | ∈ N + , i.e., P (cid:48) i.i.d.

Original: ∼ D |P (cid:48) | , consider the reparameterized version of Alg.with
Revised: ∼ D |P (cid:48) | , consider the reparameterized version of Alg.with

Original: S ⊆ P of size |S| ≥ (cid:100)log (16 |P (cid:48) [Equation] η
Revised: S ⊆ P of size |S| ≥ (cid:100)

Original: [Equation]
Revised: [Equation]

Original: δ )

Original: [Equation]
Revised: [Equation]

Original: η

Original: [Equation]
Revised: [Equation]

Original: ˆ∆ (cid:96) = (cid:16)
Revised: ˆ∆ (cid:96) = (cid:16)

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: η [Equation] [Equation] ∆ (cid:96)i
Revised: ] (cid:80)

Original: [Equation]
Revised: [Equation]

Original:  x [Equation] ) (cid:17) + κ as before, but κ is instead deﬁned as
Revised: (cid:48) ∈S ∆ (cid:96)i ( x (cid:48) ) (cid:17) + κ as before, but κ is instead deﬁned as

Original: [Equation]
Revised: [Equation]

Original: , and 3.
Revised: , and 3.

Original: m ≥ (cid:108)8 S log(
Revised: m ≥ (cid:108)

Original: [Equation]
Revised: [Equation]

Revised: S log(

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original:  (cid:109) in the sample complexity in S PARSIFY W EIGHTS .
Revised:  (cid:109) in the sample complexity in S PARSIFY W EIGHTS .

Original: Then, Alg. 1 generates a set of neural network parameters ˆ θ of size at most nnz
Revised: Then, Alg. 1 generates a set of neural network parameters ˆ θ of size at most nnz(

Original: [Equation]
Revised: [Equation]

Revised: θ

Revised: [Equation]

Revised: ≤

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: in O (cid:0) η η ∗ log
Revised: in O [Equation] η η [Equation]

Revised: log

Original: [Equation]
Revised: [Equation]

Original: η η
Revised: η η

Original: [Equation]
Revised: [Equation]

Original: |P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: time such that
Revised: time such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The reparameterization enables us to invoke Theorem 4 with δ (cid:48)
Revised: The reparameterization enables us to invoke Theorem 4 with δ (cid:48)

Original: = δ / 2 |P (cid:48) | ; applying the union bound over all |P (cid:48)
Revised: = δ / 2 |P (cid:48) | ; applying the union bound over all |P (cid:48)

Original: | i.i.d.
Revised: | i.i.d.

Original: samples in P (cid:48) establishes the corollary.
Revised: samples in P (cid:48) establishes the corollary.

Original: B A DDITIONAL R ESULTS
Revised: B A DDITIONAL R ESULTS

Original: In this section, we give more details on the evaluation of our compression algorithm on popular benchmark data sets and varying fully-connected neural network conﬁgurations.
Revised: In this section, we give more details on the evaluation of our compression algorithm on popular benchmark data sets and varying fully-connected neural network conﬁgurations.

Original: In the experiments, we compare the effectiveness of our sampling scheme in reducing the number of non-zero parameters of a network to that of uniform sampling and the singular value decomposition (SVD).
Revised: In the experiments, we compare the effectiveness of our sampling scheme in reducing the number of non-zero parameters of a network to that of uniform sampling and the singular value decomposition (SVD).

Original: All algorithms were implemented in Python using the PyTorch library (Paszke et al., 2017) and simulations were conducted on a computer with a 2.60 GHz Intel i9-7980XE processor (18 cores total) and 128 GB RAM.
Revised: All algorithms were implemented in Python using the PyTorch library (Paszke et al., 2017) and simulations were conducted on a computer with a 2.60 GHz Intel i9-7980XE processor (18 cores total) and 128 GB RAM.

Original: For training and evaluating the algorithms considered in this section, we used the following off-theshelf data sets:
Revised: For training and evaluating the algorithms considered in this section, we used the following off-theshelf data sets:

Original: • MNIST (LeCun et al., 1998) — 70 , 000 images of handwritten digits between 0 and 9 in the form of 28 × 28 pixels per image.
Revised: • MNIST (LeCun et al., 1998) — 70 , 000 images of handwritten digits between 0 and 9 in the form of 28 × 28 pixels per image.

Original: • CIFAR-10 (Krizhevsky & Hinton, 2009) — 60 , 

Revised: • CIFAR-10 (Krizhevsky & Hinton, 2009) — 60 ,

Original: 32 × 32 color images, a subset of the larger CIFAR-100 dataset, each depicting an object from one of 10 classes, e
Revised: 32 × 32 color images, a subset of the larger CIFAR-100 dataset, each depicting an object from one of 10 classes, e

Original: g., airplanes.
Revised: g., airplanes.

Original: • FashionMNIST (Xiao et al., 2017) —
Revised: • FashionMNIST (Xiao et al., 2017) —

Original: A recently proposed drop-in replacement for the MNIST data set that, like MNIST, contains 60 , 000 , 28 × 28 grayscale images, each associated with a label from 10 different categories.
Revised: A recently proposed drop-in replacement for the MNIST data set that, like MNIST, contains 60 , 000 , 28 × 28 grayscale images, each associated with a label from 10 different categories.

Original: We considered a diverse set of network conﬁgurations for each of the data sets.
Revised: We considered a diverse set of network conﬁgurations for each of the data sets.

Original: We varied the number of hidden layers between 2 and 5 and used either a constant width across all hidden layers between 200 and 1000 or a linearly decreasing width (denoted by ”Pyramid” in the ﬁgures).
Revised: We varied the number of hidden layers between 2 and 5 and used either a constant width across all hidden layers between 200 and 1000 or a linearly decreasing width (denoted by ”Pyramid” in the ﬁgures).

Original: Training was performed for 30 epochs on the normalized data sets using an Adam optimizer with a learning rate of 0.001 and a batch size of 300.
Revised: Training was performed for 30 epochs on the normalized data sets using an Adam optimizer with a learning rate of 0.001 and a batch size of 300.

Original: The test accuracies were roughly 98% (MNIST), 45% (CIFAR10), and 96% (FashionMNIST), depending on the network architecture.
Revised: The test accuracies were roughly 98% (MNIST), 45% (CIFAR10), and 96% (FashionMNIST), depending on the network architecture.

Original: To account for the randomness in the training procedure, for each data set and neural network conﬁguration, we averaged our results across 4 trained neural networks.
Revised: To account for the randomness in the training procedure, for each data set and neural network conﬁguration, we averaged our results across 4 trained neural networks.

Original: B. D ETAILS ON THE C OMPRESSION A LGORITHMS
Revised: B. D ETAILS ON THE C OMPRESSION A LGORITHMS

Original: We evaluated and compared the performance of the following algorithms on the aforementioned data sets.
Revised: We evaluated and compared the performance of the following algorithms on the aforementioned data sets.

Original: Uniform (Edge) Sampling —
Revised: Uniform (Edge) Sampling —

Original: A uniform distribution is used, rather than our sensitivity-based importance sampling distribution, to sample the incoming edges to each neuron in the network.
Revised: A uniform distribution is used, rather than our sensitivity-based importance sampling distribution, to sample the incoming edges to each neuron in the network.

Original: Note that like our sampling scheme, uniform sampling edges generates an unbiased estimator of the neuron value.
Revised: Note that like our sampling scheme, uniform sampling edges generates an unbiased estimator of the neuron value.

Original: However, unlike our approach which explicitly seeks to minimize estimator variance using the bounds provided by empirical sensitivity, uniform sampling is prone to exhibiting large estimator variance.
Revised: However, unlike our approach which explicitly seeks to minimize estimator variance using the bounds provided by empirical sensitivity, uniform sampling is prone to exhibiting large estimator variance.

Original: Singular Value Decomposition (SVD) — The (truncated) SVD decomposition is used to generate a low-rank (rank- r ) approximation for each of the weight matrices ( ˆ W 2 , . . .
Revised: Singular Value Decomposition (SVD) — The (truncated) SVD decomposition is used to generate a low-rank (rank- r ) approximation for each of the weight matrices ( ˆ W 2 , . . .

Original: , ˆ W L ) to obtain the corresponding parameters ˆ θ
Revised: , ˆ W L ) to obtain the corresponding parameters ˆ θ

Original: =
Revised: =

Original: ( ˆ W 2 r , . .
Revised: ( ˆ W 2 r , . .

Original: , ˆ W Lr ) for various values of r ∈ N + .
Revised: , ˆ W Lr ) for various values of r ∈ N + .

Original: Unlike the compared sampling-based methods, SVD does not sparsify the weight matrices.
Revised: Unlike the compared sampling-based methods, SVD does not sparsify the weight matrices.

Original: Thus, to achieve fair comparisons of compression rates, we compute the size of the rank- r matrices constituting ˆ θ as,
Revised: Thus, to achieve fair comparisons of compression rates, we compute the size of the rank- r matrices constituting ˆ θ as,

Original: [Equation]
Revised: [Equation]

Original: where W (cid:96) =
Revised: where W (cid:96) =

Original: U (cid:96) Σ (cid:96) ( V (cid:96) ) (cid:62) for each (cid:96) ∈ { 2 , . . .
Revised: U (cid:96) Σ (cid:96) ( V (cid:96) ) (cid:62) for each (cid:96) ∈ { 2 , . . .

Original: , L } , with σ 1 ≥ σ 2 . .
Revised: , L } , with σ 1 ≥ σ 2 . .

Original: ≥ σ η (cid:96) − 1 and u (cid:96)i and v (cid:96)i denote the i th columns of U (cid:96) and V (cid:96) respectively.
Revised: ≥ σ η (cid:96) − 1 and u (cid:96)i and v (cid:96)i denote the i th columns of U (cid:96) and V (cid:96) respectively.

Original: (cid:96) 

Revised: (cid:96)

Original: Sampling (Achlioptas et al., 2013) — An entry-wise sampling distribution based on the ratio between the absolute value of a single entry and the (entry-wise) (cid:96)
Revised: Sampling (Achlioptas et al., 2013) — An entry-wise sampling distribution based on the ratio between the absolute value of a single entry and the (entry-wise) (cid:96)

Original: 1 - norm of the weight matrix is computed, and the weight matrix is subsequently sparsiﬁed by sampling accordingly.
Revised: 1 - norm of the weight matrix is computed, and the weight matrix is subsequently sparsiﬁed by sampling accordingly.

Original: In particular, entry w ij of some weight matrix W is sampled with probability
Revised: In particular, entry w ij of some weight matrix W is sampled with probability

Original: [Equation]
Revised: [Equation]

Original: and reweighted to ensure the unbiasedness of the resulting estimator.
Revised: and reweighted to ensure the unbiasedness of the resulting estimator.

Original: (cid:96) 

Revised: (cid:96)

Original: Sampling (Drineas & Zouzias, 2011) — The entries ( i, j ) of each weight matrix W are sampled with distribution
Revised: Sampling (Drineas & Zouzias, 2011) — The entries ( i, j ) of each weight matrix W are sampled with distribution

Original: [Equation]
Revised: [Equation]

Original: where (cid:107)·(cid:107) F is the Frobenius norm of W , and reweighted accordingly.
Revised: where (cid:107)·(cid:107) F is the Frobenius norm of W , and reweighted accordingly.

Original: (cid:96) 

Revised: (cid:96)

Original: + (cid:96) 

Revised: + (cid:96)

Original: Sampling (Kundu & Drineas, 2014) – The entries ( i, j ) of each weight matrix W are sampled with distribution
Revised: Sampling (Kundu & Drineas, 2014) – The entries ( i, j ) of each weight matrix W are sampled with distribution

Original: [Equation]
Revised: [Equation]

Original: where (cid:107)·(cid:107) F is the Frobenius norm of W , and reweighted accordingly.
Revised: where (cid:107)·(cid:107) F is the Frobenius norm of W , and reweighted accordingly.

Original: We note that Kundu &
Revised: We note that Kundu &

Original: Drineas (2014) constitutes the current state-of-the-art in data-oblivious matrix sparsiﬁcation algorithms.
Revised: Drineas (2014) constitutes the current state-of-the-art in data-oblivious matrix sparsiﬁcation algorithms.

Original: CoreNet (Edge Sampling) — Our core algorithm for edge sampling shown in Alg. 2, but without the neuron pruning procedure.
Revised: CoreNet (Edge Sampling) — Our core algorithm for edge sampling shown in Alg. 2, but without the neuron pruning procedure.

Original: CoreNet + (CoreNet & Neuron Pruning) —
Revised: CoreNet + (CoreNet & Neuron Pruning) —

Original: Our algorithm shown in Alg. 1 that includes the neuron pruning step. 8.
Revised: Our algorithm shown in Alg. 1 that includes the neuron pruning step. 8.

Original: CoreNet ++ (CoreNet + & Ampliﬁcation) —
Revised: CoreNet ++ (CoreNet + & Ampliﬁcation) —

Original: In addition to the features of Corenet + , multiple coresets C 1 , . . .
Revised: In addition to the features of Corenet + , multiple coresets C 1 , . . .

Original: , C τ are constructed over τ ∈ N + trials, and the best one is picked by evaluating the empirical error on a subset T ⊆
Revised: , C τ are constructed over τ ∈ N + trials, and the best one is picked by evaluating the empirical error on a subset T ⊆

Original: P \ S (see Sec. 4 for details).
Revised: P \ S (see Sec. 4 for details).

Original: B. P RESERVING THE O UTPUT OF A N EURAL N ETWORK
Revised: B. P RESERVING THE O UTPUT OF A N EURAL N ETWORK

Original: We evaluated the accuracy of our approximation by comparing the output of the compressed network with that of the original one and compute the (cid:96) 1 -norm of the relative error vector.
Revised: We evaluated the accuracy of our approximation by comparing the output of the compressed network with that of the original one and compute the (cid:96) 1 -norm of the relative error vector.

Original: We computed the error metric for both the uniform sampling scheme as well as our compression algorithm (Alg. 1).
Revised: We computed the error metric for both the uniform sampling scheme as well as our compression algorithm (Alg. 1).

Original: Our results were averaged over 50 trials, where for each trial, the relative approximation error was averaged over the entire test set.
Revised: Our results were averaged over 50 trials, where for each trial, the relative approximation error was averaged over the entire test set.

Original: In particular, for a test set P test ⊆
Revised: In particular, for a test set P test ⊆

Original: R d consisting of d dimensional points, the average relative error of with respect to the f ˆ θ generated by each compression algorithm was computed as
Revised: R d consisting of d dimensional points, the average relative error of with respect to the f ˆ θ generated by each compression algorithm was computed as

Original: [Equation]
Revised: [Equation]

Original: Figures 3, 4, and 5 depict the average performance of the compared algorithms for various network architectures trained on MNIST, CIFAR-10, and FashionMNIST, respectively.
Revised: Figures 3, 4, and 5 depict the average performance of the compared algorithms for various network architectures trained on MNIST, CIFAR-10, and FashionMNIST, respectively.

Original: Our algorithm is able to compress networks trained on MNIST and FashionMNIST to about 10% of their original size without signiﬁcant loss of accuracy.
Revised: Our algorithm is able to compress networks trained on MNIST and FashionMNIST to about 10% of their original size without signiﬁcant loss of accuracy.

Original: On CIFAR-10, a compression rate of 50% yields classiﬁcation results comparable to that of uncompressed networks.
Revised: On CIFAR-10, a compression rate of 50% yields classiﬁcation results comparable to that of uncompressed networks.

Original: The shaded region corresponding to each curve represents the values within one standard deviation of the mean.
Revised: The shaded region corresponding to each curve represents the values within one standard deviation of the mean.

Original: B. P RESERVING THE C LASSIFICATION P ERFORMANCE
Revised: B. P RESERVING THE C LASSIFICATION P ERFORMANCE

Original: We also evaluated the accuracy of our approximation by computing the loss of prediction accuracy on a test data set, P test .
Revised: We also evaluated the accuracy of our approximation by computing the loss of prediction accuracy on a test data set, P test .

Original: In particular, let acc P test ( f θ ) be the average accuracy of the neural network f θ , i.e,.
Revised: In particular, let acc P test ( f θ ) be the average accuracy of the neural network f θ , i.e,.

Original: [Equation]
Revised: [Equation]

Original: where y ( x ) denotes the (true) label associated with x .
Revised: where y ( x ) denotes the (true) label associated with x .

Original: Then the drop in accuracy is computed as
Revised: Then the drop in accuracy is computed as

Original: [Equation]
Revised: [Equation]

Original: Figures 6, 7, and 8 depict the average performance of the compared algorithms for various network architectures trained on MNIST, CIFAR-10, and FashionMNIST respectively.
Revised: Figures 6, 7, and 8 depict the average performance of the compared algorithms for various network architectures trained on MNIST, CIFAR-10, and FashionMNIST respectively.

Original: The shaded region corresponding to each curve 
Revised: The shaded region corresponding to each curve represents

Original: [Table]
Revised: We compared the performance of our approach with that of the popular weight thresholding heuristic – henceforth denoted by WT – of Han et al. (2015) when retraining was allowed after the compression, i.e., pruning, procedure. Our comparisons with retraining for the networks and data sets mentioned in Sec.

Original: [Table]
Revised: 6 are as follows.

Original: [Table]
Revised: For MNIST, WT required 5.8% of the number of parameters to obtain the classiﬁcation accuracy of the original model (i.e., 0% drop in accuracy), whereas for the same percentage (5.8%) of the parameters retained, CoreNet++ incurred a classiﬁcation accuracy drop of 1%.

Original: [Table]
Revised: For CIFAR, the approach of Han et al. (2015) matched the original models accuracy using 3% of the parameters, whereas CoreNet++ reported an accuracy drop of 9.5% for 3% of the parameters retained. Finally, for FashionMNIST, the corresponding numbers were 4.1% of the parameters to achieve 0% loss for WT, and a loss of 4.7% in accuracy for CoreNet++ with the same percentage of parameters retained.

Original: B. D ISCUSSION
Revised: B. D ISCUSSION

Original: As indicated in Sec. 6, the simulation results presented here validate our theoretical results and suggest that empirical sensitivity can lead to effective, more informed sampling compared to other methods.
Revised: As indicated in Sec. 6, the simulation results presented here validate our theoretical results and suggest that empirical sensitivity can lead to effective, more informed sampling compared to other methods.

Original: Moreover, we are able to outperform networks that are compressed via state-of-the-art matrix sparsiﬁcation algorithms.
Revised: Moreover, we are able to outperform networks that are compressed via state-of-the-art matrix sparsiﬁcation algorithms.

Original: We also note that there is a notable difference in the performance of our algorithm between different datasets.
Revised: We also note that there is a notable difference in the performance of our algorithm between different datasets.

Original: In particular, the difference in performance of our algorithm compared to the other method for networks trained on FashionMNIST and MNIST is much more signiﬁcant than for networks trained on CIFAR.
Revised: In particular, the difference in performance of our algorithm compared to the other method for networks trained on FashionMNIST and MNIST is much more signiﬁcant than for networks trained on CIFAR.

Original: We conjecture that this is partially due to considering only fully-connected networks as these network perform fairly poorly on CIFAR (around 45% classiﬁcation accuracy) and thus edges have more uniformly distributed sensitivity as the information content in the network is limited.
Revised: We conjecture that this is partially due to considering only fully-connected networks as these network perform fairly poorly on CIFAR (around 45% classiﬁcation accuracy) and thus edges have more uniformly distributed sensitivity as the information content in the network is limited.

Original: We envision that extending our guarantees to convolutional neural networks may enable us to further reason about the performance on data sets such as CIFAR.
Revised: We envision that extending our guarantees to convolutional neural networks may enable us to further reason about the performance on data sets such as CIFAR

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure][Figure] [Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  a i v
Revised: l a i v

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] l at i v [Figure]
Revised: a

Original: [Figure]
Revised: [Figure]

Original: A v r a g

Original: e

Original: R
Revised: R

Original: e
Revised: [Figure]

Original: l a t i v
Revised: l a t i v

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A 
Revised: A v r a g R e l a t i v e

Original: [Figure]
Revised: [Figure]

Original: R l a t i v
Revised: o r

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: l at i v
Revised: l a i v

Original: [Figure]
Revised: [Figure]

Original: A v r a g e R e l
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: a

Original: [Table] [Figure]
Revised: [Figure] [Figure]

Original: e r a
Revised: R a

Original: [Figure]
Revised: [Figure]

Original: e R e l a t i v

Original: [Figure]
Revised: [Figure]

Revised: r a g e R e l a t i v e E

Original: rr o r
Revised: rr o r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: c

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: r o p i 
Revised: r o p i n A cc

Original: [Figure]
Revised: [Figure]

Original: cc u r a c y
Revised: r a c y

Original: [Figure]
Revised: [Figure]

Revised: [Figure]