Original: DNN Model Compression Under Accuracy Constraints
Revised: DNN M ODEL C OMPRESSION U NDER

Revised: A CCURACY

Revised: C ONSTRAINTS

Original: Anonymous Author(s)
Revised: Anonymous authors

Original: AfﬁliationAddress email
Revised: Paper under double-blind review

Original: Abstract
Revised: A BSTRACT

Original: The growing interest to implement Deep Neural Networks (DNNs) on resourcebound hardware has motivated the innovation of compression algorithms.
Revised: The growing interest to implement Deep Neural Networks (DNNs) on resourcebound hardware has motivated the innovation of compression algorithms.

Original: Usingthese algorithms, DNN model sizes can be substantially reduced, with little-to-noaccuracy degradation.
Revised: Using these algorithms, DNN model sizes can be substantially reduced, with little-to-no accuracy degradation.

Original: This is achieved by either eliminating components fromthe model, or penalizing complexity during training.
Revised: This is achieved by either eliminating components from the model, or penalizing complexity during training.

Original: While both approachesdemonstrate considerable compressions, the former often ignores the loss functionduring compression while the latter produces unpredictable compressions.
Revised: While both approaches demonstrate considerable compressions, the former often ignores the loss function during compression while the latter produces unpredictable compressions.

Original: In thispaper, we propose a technique that directly minimizes both the model complexityand the changes in the loss function.
Revised: In this paper, we propose a technique that directly minimizes both the model complexity and the changes in the loss function.

Original: In this technique, we formulate compressionas a constrained optimization problem, and then present a solution for it.
Revised: In this technique, we formulate compression as a constrained optimization problem, and then present a solution for it.

Original: We willshow that using this technique, we can achieve competitive results.
Revised: We will show that using this technique, we can achieve competitive results.

Original:  Introduction
Revised:  I NTRODUCTION

Original: Deep Neural Networks have been rapidly improving in many classiﬁcation tasks, even surpassinghuman accuracy for some problems (Russakovsky et al. 2015).
Revised: Deep Neural Networks have been rapidly improving in many classiﬁcation tasks, even surpassing human accuracy for some problems (Russakovsky et al., 2015).

Original: This high accuracy, however, isobtained by employing wider (Zagoruyko and Komodakis 2016) or deeper (He et al. 2016) networks.
Revised: This high accuracy, however, is obtained by employing wider Zagoruyko & Komodakis (2016) or deeper He et al. (2016) networks.

Original: Some prominent networks today even surpass 100 layers, store hundreds of millions of parameters,and require billions of operations per input sample 
Revised: Some prominent networks today even surpass 100 layers, store hundreds of millions of parameters, and require billions of operations per input sample He et al.

Original: (He et al. 2016; Srivastava et al.2015).
Revised: ; Srivastava et al.

Original: Suchlarge networks are not well-suited for resource-bound, embedded and mobile platforms which aredominating the consumer market.
Revised: Such large networks are not well-suited for resource-bound, embedded and mobile platforms which are dominating the consumer market.

Original: This mismatch between computational requirements and availableresources has motivated efforts to compress DNN models.
Revised: This mismatch between computational requirements and available resources has motivated efforts to compress DNN models.

Original: Compression techniques exploit the redundancy inherent to neural networks that emerges due to theconsiderable number of parameters in them.
Revised: Compression techniques exploit the redundancy inherent to neural networks that emerges due to the considerable number of parameters in them.

Original: These many parameters help learn highly informativefeatures during training.
Revised: These many parameters help learn highly informative features during training.

Original: However, they simultaneously learn multitudes of unnecessary, ineffectualones.
Revised: However, they simultaneously learn multitudes of unnecessary, ineffectual ones.

Original: Han et al.
Revised: Han et al.

Original: (2015) reduces these redundancies by pruning the network and quantizing theremaining parameters.
Revised: (2015) reduces these redundancies by pruning the network and quantizing the remaining parameters.

Original: Using these techniques, they were able to reduce the model size by morethan an order of magnitude.
Revised: Using these techniques, they were able to reduce the model size by more than an order of magnitude.

Original: Their success inspired other methodical approaches such as the softweight-sharing (Ullrich et al. 2017).
Revised: Their success inspired other methodical approaches such as the soft weight-sharing (Ullrich et al., 2017).

Original: This method encodes model parameters using Bayesian priorand then penalizes this prior during training.
Revised: This method encodes model parameters using Bayesian prior and then penalizes this prior during training.

Original: As a result, it performs both pruning and quantizationsimultaneously and achieves superior compressions with negligible loss in accuracy.
Revised: As a result, it performs both pruning and quantization simultaneously and achieves superior compressions with negligible loss in accuracy.

Original: In this work, we take a similar, integrated approach with the twist that we directly minimize thecomplexity.
Revised: In this work, we take a similar, integrated approach with the twist that we directly minimize the complexity.

Original: Unlike soft weight-sharing, however, we encode the parameters using the k-meansobjective which imposes less computations.
Revised: Unlike soft weight-sharing, however, we encode the parameters using the k-means objective which imposes less computations.

Original: We further apply a hard constraint on the training lossto maintain sufﬁcient accuracy.
Revised: We further apply a hard constraint on the training loss to maintain sufﬁcient accuracy.

Original: Such a constraint takes advantage of the fact that we have alreadyobtained some information about the loss function during training.
Revised: Such a constraint takes advantage of the fact that we have already obtained some information about the loss function during training.

Original: We then present a straightforwardsolution for this constrained optimization problem.
Revised: We then present a straightforward solution for this constrained optimization problem.

Original: Our solution iteratively minimizes the k-meansobjective, while satisfying the loss constraint.
Revised: Our solution iteratively minimizes the k-means objective, while satisfying the loss constraint.

Original: Consequently, it can compress the model progressively where the compression rate can be adjusted.
Revised: Consequently, it can compress the model progressively where the compression rate can be adjusted.

Original: Finally, we test the proposed technique on three populardatasets and show that this method can achieve state-of-the-art compression with minimal loss ofaccuracy.
Revised: Finally, we test the proposed technique on three popular datasets and show that this method can achieve state-of-the-art compression with minimal loss of accuracy.

Original:  Related Work
Revised:  R ELATED W ORK

Original: Robustness of DNNs to noisy parameters is a byproduct of their redundancy and has been utilizedfrequently to facilitate and accelerate their implementations.
Revised: Robustness of DNNs to noisy parameters is a byproduct of their redundancy and has been utilized frequently to facilitate and accelerate their implementations.

Original: Limited precision representations whichintroduce quantization noise, for example, have been widely used to simplify computations (Guptaet al. 2015) and design neural network hardware (Jouppi et al. 2017).
Revised: Limited precision representations which introduce quantization noise, for example, have been widely used to simplify computations (Gupta et al., 2015) and design neural network hardware (Jouppi et al., 2017).

Original: Other works, take a differentapproach and eliminate most parameters (pruning) (Molchanov et al. 2016), aiming to reduce the totalnumber of computations instead.
Revised: Other works, take a different approach and eliminate most parameters (pruning) (Molchanov et al., 2016), aiming to reduce the total number of computations instead.

Original: Finally, weight-sharing techniques (Han et al. 2015) encode theremaining parameters of the networks in a small dictionary, reducing the storage requirement of thenetwork.
Revised: Finally, weight-sharing techniques (Han et al., 2015) encode the remaining parameters of the networks in a small dictionary, reducing the storage requirement of the network.

Original: In this paper, we will be focusing on the last two techniques (pruning and weight-sharing),which were proposed ﬁrst by Han et al.
Revised: In this paper, we will be focusing on the last two techniques (pruning and weight-sharing), which were proposed ﬁrst by Han et al.

Original: Our goal is to present a simple, uniﬁed method forcompressing a trained DNN model.
Revised: Our goal is to present a simple, uniﬁed method for compressing a trained DNN model.

Original: Han et al.
Revised: Han et al.

Original: ﬁrst proposed Deep Compression which introduced pruning and weight-sharing astechniques to compress DNN models.
Revised: ﬁrst proposed Deep Compression which introduced pruning and weight-sharing as techniques to compress DNN models.

Original: Pruning eliminates unnecessary network connections identiﬁedby their magnitude.
Revised: Pruning eliminates unnecessary network connections identiﬁed by their magnitude.

Original: Weight-sharing then encodes parameters into a small dictionary, using the kmeans algorithm.
Revised: Weight-sharing then encodes parameters into a small dictionary, using the kmeans algorithm.

Original: This technique also employs an intermediate tuning step which offsets the effect ofpruning on accuracy.
Revised: This technique also employs an intermediate tuning step which offsets the effect of pruning on accuracy.

Original: However, the changes in the training loss function are, for the most part, ignoredin this technique.
Revised: However, the changes in the training loss function are, for the most part, ignored in this technique.

Original: Later studies demonstrated that taking this effect into account, either in pruning orquantization, can improve the compression.
Revised: Later studies demonstrated that taking this effect into account, either in pruning or quantization, can improve the compression.

Original: Guo et al.
Revised: Guo et al.

Original: (2016), for example, formulated pruning as anoptimization problem.
Revised: (2016), for example, formulated pruning as an optimization problem.

Original: Their method essentially performs pruning and tuning simultaneously, butdoes not perform weight-sharing.
Revised: Their method essentially performs pruning and tuning simultaneously, but does not perform weight-sharing.

Original: Likewise, Choi et al.
Revised: Likewise, Choi et al.

Original: (2016) incorporate this effect by augmentingthe k-means objective by the training loss Hessian.
Revised: (2016) incorporate this effect by augmenting the k-means objective by the training loss Hessian.

Original: However, they only consider the Hessian at onepoint, and their method requires prediction of the compression ratio beforehand.
Revised: However, they only consider the Hessian at one point, and their method requires prediction of the compression ratio beforehand.

Original: Soft weight-sharing, proposed by Ullrich et al.
Revised: Soft weight-sharing, proposed by Ullrich et al.

Original: (2017), minimized both training loss and model sizesimultaneously.
Revised: (2017), minimized both training loss and model size simultaneously.

Original: Unlike the previous attempts, they encoded the network parameters using a Bayesianprior.
Revised: Unlike the previous attempts, they encoded the network parameters using a Bayesian prior.

Original: This prior is then penalized when optimizing the training loss function and the mixing ratiocontrols the trade-off between accuracy and model size.
Revised: This prior is then penalized when optimizing the training loss function and the mixing ratio controls the trade-off between accuracy and model size.

Original: This is then optimized using a SGD method.
Revised: This is then optimized using a SGD method.

Original: The limitation of this technique is the complexity of the Bayesian prior.
Revised: The limitation of this technique is the complexity of the Bayesian prior.

Original: Due to this limitation, inthis paper, we will focus on k-means encoding of the weights, even though our method can be easilyextended to Bayesian encoding as well. 
Revised: Due to this limitation, in this paper, we will focus on k-means encoding of the weights, even though our method can be easily extended to Bayesian encoding as well. p

Original:  Method
Revised:  M ETHOD

Original: The proposed algorithm compresses a trained neural network model by minimizing its complexitywhile constraining its loss function.
Revised: The proposed algorithm compresses a trained neural network model by minimizing its complexity while constraining its loss function.

Original: Here, we ﬁrst review the optimization process for the lossfunction during the training process.
Revised: Here, we ﬁrst review the optimization process for the loss function during the training process.

Original: Then, we discuss the k-means encoding, which we use tomeasure the model complexity.
Revised: Then, we discuss the k-means encoding, which we use to measure the model complexity.

Original: Finally, we will present our compression algorithm.
Revised: Finally, we will present our compression algorithm.

Original: 3.1 Training Neural Networks
Revised: 3. T RAINING N EURAL N ETWORKS

Original: Training neural networks involves minimizing a loss function that quantiﬁes the dissimilarity betweenthe model output distribution and the ideal distribution.
Revised: Training neural networks involves minimizing a loss function that quantiﬁes the dissimilarity between the model output distribution and the ideal distribution.

Original: This optimization problem is formally writtenas:
Revised: This optimization problem is formally written as:

Original: [Equation]
Revised: [Equation]

Original: Here, X and y deﬁne a set of samples and ideal output distributions we use for training.
Revised: Here, X and y deﬁne a set of samples and ideal output distributions we use for training.

Original: W is thelearnt model and is the concatenation of all model parameters, ( W = [ w 1 w 2 ... w N ] T ∈R
Revised: W is the learnt model and is the concatenation of all model parameters, ( W = [ w 1 w 2 ... w N ] T ∈

Original: Nwhere N is the number of parameters).
Revised: R N where N is the number of parameters).

Original: (cid:99) W is the ﬁnal, trained model.
Revised: (cid:99) W is the ﬁnal, trained model.

Original: Also, L ( W, X , y ) representsthe average loss over the dataset ( {X , y } ) corresponding to the model W .
Revised: Also, L ( W, X , y ) represents the average loss over the dataset ( {X , y } ) corresponding to the model W .

Original: This function is typicallychosen as Cross entropy or hinge square loss.
Revised: This function is typically chosen as Cross entropy or hinge square loss.

Original: In the rest of this paper, we represent L ( W, X , y ) as
Revised: In the rest of this paper, we represent L ( W, X , y ) as

Original: L ( W ) for simplicity.
Revised: L ( W ) for simplicity.

Original: The local optimum for W in this problem is usually found iteratively, using a
Revised: The local optimum for W in this problem is usually found iteratively, using a

Original: SGD method.
Revised: SGD method.

Original: 3.2 K-means Encoding
Revised: 3. K- MEANS E NCODING

Original: The weight-sharing techniques proposed by Han et al.
Revised: The weight-sharing techniques proposed by Han et al.

Original:  and Guo et al (2016) use the k-meansalgorithm to encode the network parameters.
Revised: (2015) and Guo et al. (2016) use the k-means algorithm to encode the network parameters.

Original: Each parameter is then replaced by the nearest of the kcentroids.
Revised: Each parameter is then replaced by the nearest of the k centroids.

Original: This procedure can also be deﬁned as the following optimization problem:
Revised: This procedure can also be deﬁned as the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where M = [ m 1 m 2 ...
Revised: where M = [ m 1 m 2 ...

Original: m N ] T ∈ R N is the concatenated array of all centroids m i correspondingto the parameters w i .
Revised: m N ] T ∈ R N is the concatenated array of all centroids m i corresponding to the parameters w i .

Original: The set C = { c 1 , c 2 , ..., c k } ⊂ R also deﬁnes the set of cluster centroids.
Revised: The set C = { c 1 , c 2 , ..., c k } ⊂ R also deﬁnes the set of cluster centroids.

Original: The commonly used heuristic solution to this problem is to iteratively ﬁnd M and update centroidsaccordingly.
Revised: The commonly used heuristic solution to this problem is to iteratively ﬁnd M and update centroids accordingly.

Original: In each iteration t this update is performed by:
Revised: In each iteration t this update is performed by:

Original: [Equation]
Revised: [Equation]

Original: 3.3 Proposed Compression Method
Revised: 3. P ROPOSED C OMPRESSION M ETHOD

Original: Similar to weight-sharing in Deep Compression (Han et al. 2015), we minimize the k-means objectivefunction to reduce the model complexity.
Revised: Similar to weight-sharing in Deep Compression (Han et al., 2015), we minimize the k-means objective function to reduce the model complexity.

Original: Further, similar to soft weigh-sharing (Ullrich et al. 2017),we minimize the changes in the loss function.
Revised: Further, similar to soft weigh-sharing (Ullrich et al., 2017), we minimize the changes in the loss function.

Original: However, we control these changes through a hardconstraint to take advantage of the information already learnt through the training phase.
Revised: However, we control these changes through a hard constraint to take advantage of the information already learnt through the training phase.

Original: In the rest ofthis section, we will formulate the problem of minimizing the complexity of the model and propose asolution for it.
Revised: In the rest of this section, we will formulate the problem of minimizing the complexity of the model and propose a solution for it.

Original: As previously discussed, we use the k-means objective to represent the complexity of the model.
Revised: As previously discussed, we use the k-means objective to represent the complexity of the model.

Original: The goal is to learn the network parameters as well as the centroids that minimize this objective.
Revised: The goal is to learn the network parameters as well as the centroids that minimize this objective.

Original: Speciﬁcally, we would like to learn a small set of centroids and encourage the network parameters topopulate a small region about these centroids only.
Revised: Speciﬁcally, we would like to learn a small set of centroids and encourage the network parameters to populate a small region about these centroids only.

Original: [Equation]
Revised: [Equation]

Original: We note that the centroids m i are correlated to w i by equation 2 with the difference that the numberof centroids k is also obtained through the compression process.
Revised: We note that the centroids m i are correlated to w i by equation 2 with the difference that the number of centroids k is also obtained through the compression process.

Original: In the absence of any additional constraints, this optimization problem reverts back to one solved by
Revised: In the absence of any additional constraints, this optimization problem reverts back to one solved by

Original: Han et al. (2015) for Deep Compression.
Revised: Han et al. (2015) for Deep Compression.

Original: The solution to this problem is simply w i = m i .
Revised: The solution to this problem is simply w i = m i .

Original: However,this might result in signiﬁcant changes in the accuracy.
Revised: However, this might result in signiﬁcant changes in the accuracy.

Original: We address this issue by introducing aconstraint on the loss function.
Revised: We address this issue by introducing a constraint on the loss function.

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:96) is the hard constraint derived based on the optimal value of the loss function.
Revised: Here, (cid:96) is the hard constraint derived based on the optimal value of the loss function.

Original: We will discusshow we determine this hyper-parameter later in this section.
Revised: We will discuss how we determine this hyper-parameter later in this section.

Original: While solving our optimization problem,we will make sure that this condition is always satisﬁed.
Revised: While solving our optimization problem, we will make sure that this condition is always satisﬁed.

Original: Next, we present our method for solving the problem of minimizing the complexity of a trainedmodel.
Revised: Next, we present our method for solving the problem of minimizing the complexity of a trained model.

Original: This method iteratively solves the equation 6 for W and then calculates M based on equation2, as presented in algorithm 1.
Revised: This method iteratively solves the equation 6 for W and then calculates M based on equation 2, as presented in algorithm 1.

Original: Algorithm t ←
Revised: Algorithm t ←

Original: Initialize (cid:96)
Revised: Initialize (cid:96)

Original: Randomly initialize k centroids while Convergence is not achieved do
Revised: Randomly initialize k centroids while Convergence is not achieved do

Original: Calculate M t using equation 3.
Revised: Calculate M t using equation 3.

Original: Solve equation 6 for W t .
Revised: Solve equation 6 for W t .

Original: Update C using equation 2.
Revised: Update C using equation 2.

Original: Eliminate unnecessary centroids.
Revised: Eliminate unnecessary centroids.

Original: t ← t +end while
Revised: t ← t +end while

Original: This algorithm ﬁrst initializes k random centroids.
Revised: This algorithm ﬁrst initializes k random centroids.

Original: The value of k is provided to the algorithm asa hyper-parameter and can be large.
Revised: The value of k is provided to the algorithm as a hyper-parameter and can be large.

Original: During the compression this algorithm eliminates or mergesunnecessary centroids and reduces k .
Revised: During the compression this algorithm eliminates or merges unnecessary centroids and reduces k .

Original: At the end of each iteration, if a centroid has not been assignedto any network parameter or is close to another centroids, it is eliminated.
Revised: At the end of each iteration, if a centroid has not been assigned to any network parameter or is close to another centroids, it is eliminated.

Original: Each iteration t of the compression, ﬁnds the nearest centroid to each network parameter and solvesthe constrained optimization problem by ﬁnding a displacement for the model parameters like ∆ W twhich solves:
Revised: Each iteration t of the compression, ﬁnds the nearest centroid to each network parameter and solves the constrained optimization problem by ﬁnding a displacement for the model parameters like ∆ W t which solves:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This way, the model update for the next iteration would be: W t +1 =
Revised: This way, the model update for the next iteration would be: W t +1 =

Original: W t + ∆ W t .
Revised: W t + ∆ W t .

Original: The solution of this constrained optimization problem should satisfy the KKT condition.
Revised: The solution of this constrained optimization problem should satisfy the KKT condition.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This system has two solutions.
Revised: This system has two solutions.

Original: When µ = 0 , the solution is ∀ i : w ∗ i
Revised: When µ = 0 , the solution is ∀ i : w ∗ i

Original: = m i .
Revised: = m i .

Original: If ∆ W ∗ satisﬁes theconstraint in equation 8, this solution is valid.
Revised: If ∆ W ∗ satisﬁes the constraint in equation 8, this solution is valid.

Original: Otherwise:
Revised: Otherwise:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The difﬁculty of solving this system is the unknown value and gradient of the loss function at thesolution which we want to obtain.
Revised: The difﬁculty of solving this system is the unknown value and gradient of the loss function at the solution which we want to obtain.

Original: We get around this issue by locally estimating the loss functionusing its linear Taylor expansion and replacing that in the system.
Revised: We get around this issue by locally estimating the loss function using its linear Taylor expansion and replacing that in the system.

Original: [Equation]
Revised: [Equation]

Original: [Equation] We then approximately calculate the displacement of the weights ( ∆ W ) with regards to W 0 , representing the trained parameters, as below:[Equation]
Revised: We then approximately calculate the displacement of the weights ( ∆ W ) with regards to W 0 , representing the trained parameters, as below:

Original: [Equation]
Revised: [Equation]

Original: This system has a closed form solution:
Revised: This system has a closed form solution:

Original: [Equation]
Revised: [Equation]

Original: This solution is of course valid only in a neighborhood around W 0 where ˜∆ W is an accurateestimation of L ( W 0 + ∆ W ) .
Revised: This solution is of course valid only in a neighborhood around W 0 where ˜∆ W is an accurate estimation of L ( W 0 + ∆ W ) .

Original: We denote this region by the radius ρ .
Revised: We denote this region by the radius ρ .

Original: That is, when (cid:107) ∆ W (cid:107) ≤ ρ .
Revised: That is, when (cid:107) ∆ W (cid:107) ≤ ρ .

Original: Thistechnique is similar to local modeling of objective functions as done in Trust Region optimizationmethods.
Revised: This technique is similar to local modeling of objective functions as done in Trust Region optimization methods.

Original: We also adopt a similar method as trust region techniques to initialize and update the valueof ρ in each iteration of the algorithm.
Revised: We also adopt a similar method as trust region techniques to initialize and update the value of ρ in each iteration of the algorithm.

Original: Thus, we summarize the algorithm to solve this problem inalgorithm 2.
Revised: Thus, we summarize the algorithm to solve this problem in algorithm 2.

Original: Algorithm 2 Calculate the displacement of W for iteration t n ←
Revised: n ←

Original: Initialize ρ 0 t while True do
Revised: Initialize ρ 0 t while True do

Original: Solve the constrained optimization problem using equation 20 and ﬁnd ∆ W n λ ← min(1 , ρ nt (cid:107) ∆ W n (cid:107)) if (cid:96) <
Revised: Solve the constrained optimization problem using equation 20 and ﬁnd ∆ W n λ ← min(1 , ρ nt (cid:107) ∆ W n (cid:107) if (cid:96) <

Original: L ( W t + λ ∗ ∆ W n ) then ρ n +1 t ← 12 ρ nt else
Revised: L ( W t + λ ∗ ∆ W n ) then ρ n +1 t ← 12 ρ nt else

Original: W t +1 ← W t + λ ∗ ∆ W n ρ 0 t +1 ← 2 ∗ ρ nt break end if end while
Revised: W t +1 ← W t + λ ∗ ∆ W n ρ 0 t +1 ← 2 ∗ ρ nt break end if end while

Original: In this algorithm, each iteration receives a trust region radius as input and ﬁnds the displacementfor W t based on equation 20.
Revised: In this algorithm, each iteration receives a trust region radius as input and ﬁnds the displacement for W t based on equation 20.

Original: Then, if this displacement is larger than the trust region radius, thealgorithm scales it.
Revised: Then, if this displacement is larger than the trust region radius, the algorithm scales it.

Original: Finally, it checks if the loss function corresponding to this displacement satisﬁesthe loss constraint.
Revised: Finally, it checks if the loss function corresponding to this displacement satisﬁes the loss constraint.

Original: If not, it will shrink the trust region radius and retries the condition.
Revised: If not, it will shrink the trust region radius and retries the condition.

Original: Otherwise, itwill update the parameters and expands the trust region radius.
Revised: Otherwise, it will update the parameters and expands the trust region radius.

Original: This new radius will be provided tothe next iteration of Algorithm 1.
Revised: This new radius will be provided to the next iteration of Algorithm 1.

Original: Lastly, we discuss how we obtain the value of (cid:96) .
Revised: Lastly, we discuss how we obtain the value of (cid:96) .

Original: We initialize this value to the loss value correspondingto the original model (cid:96) =
Revised: We initialize this value to the loss value corresponding to the original model (cid:96) =

Original: L ( W 0 ) .
Revised: L ( W 0 ) .

Original: After the compression is done, we can test the accuracy of thecompressed model.
Revised: After the compression is done, we can test the accuracy of the compressed model.

Original: If this accuracy is sufﬁcient, we can increase it.
Revised: If this accuracy is sufﬁcient, we can increase it.

Original: In our technique, each time weachieved a sufﬁcient accuracy for the compressed model, we increase (cid:96) by 20% .
Revised: In our technique, each time we achieved a sufﬁcient accuracy for the compressed model, we increase (cid:96) by 20% .

Original:  Models
Revised:  M ODELS

Original: We test our compression technique on three datasets: MNIST (LeCun et al. 1998a)
Revised: We test our compression technique on three datasets: MNIST (LeCun et al., 1998b), CIFAR-

Original: , CIFAR-10(Krizhevsky and Hinton 2009), and SVHN (Netzer et al. 2011).
Revised: (Krizhevsky & Hinton, 2009), and SVHN (Netzer et al., 2011).

Original: MNIST is a collection of 600028 × 28 black-and-white images of handwritten digits.
Revised: MNIST is a collection of28 × 28 black-and-white images of handwritten digits.

Original: CIFAR-10 is a set of 60000 images of objectfrom 10 different classes.
Revised: CIFAR-10 is a set of 60000 images of object from 10 different classes.

Original: Finally, SVHN contains 60000 images of digits captured by google streetview.
Revised: Finally, SVHN contains 60000 images of digits captured by google street view.

Original: We train a LeNet-5 on the MNIST dataset and a smaller version of the VGG network (Hubaraet al. 2016) for both CIFAR-10 and SVHN.
Revised: We train a LeNet-5 on the MNIST dataset and a smaller version of the VGG network (Hubara et al., 2016) for both CIFAR-10 and SVHN.

Original: The model size and accuracy of these baseline networksare presented in table 1.
Revised: The model size and accuracy of these baseline networks are presented in table 1.

Original: [Table]
Revised: [Table]

Original:  Experiments
Revised:  E XPERIMENTS

Original: In this section, we experimentally study the convergence of the proposed algorithm and show that itachieves state-of-the-art compression rates.
Revised: In this section, we experimentally study the convergence of the proposed algorithm and show that it achieves state-of-the-art compression rates.

Original: We conﬁrm that the parameters of the compressed modelcluster about the learnt centroids and evaluate the effect of this clustering on the accuracy.
Revised: We conﬁrm that the parameters of the compressed model cluster about the learnt centroids and evaluate the effect of this clustering on the accuracy.

Original: We thenpresent a more in-depth study of the trade-off between accuracy and the compressed model size onour benchmarks.
Revised: We then present a more in-depth study of the trade-off between accuracy and the compressed model size on our benchmarks.

Original: Finally, we compare the optimal model obtained through the trade-off analysis withthe previous techniques.
Revised: Finally, we compare the optimal model obtained through the trade-off analysis with the previous techniques.

Original: The proposed compression algorithm encourages parameters to gravitate towards a set of centroids,which it learns simultaneously.
Revised: The proposed compression algorithm encourages parameters to gravitate towards a set of centroids, which it learns simultaneously.

Original: We verify this on MNIST by initializing 256 random centroids.
Revised: We verify this on MNIST by initializing 256 random centroids.

Original: Thesecentroids are eliminated or optimized according to Algorithm 6.
Revised: These centroids are eliminated or optimized according to Algorithm 6.

Original: Figure 1a illustrates the optimizationof the centroids.
Revised: Figure 1a illustrates the optimization of the centroids.

Original: In each iteration t , this ﬁgure shows the accuracy gap between the current model,
Revised: In each iteration t , this ﬁgure shows the accuracy gap between the current model,

Original: W t , and its compression using the current centroids, C t .
Revised: W t , and its compression using the current centroids, C t .

Original: We observe that this gap decreases as thecentroids are adjusted.
Revised: We observe that this gap decreases as the centroids are adjusted.

Original: In addition, we observe little uncompressed model accuracy degradation,despite the parameter updates.
Revised: In addition, we observe little uncompressed model accuracy degradation, despite the parameter updates.

Original: We conﬁrm these updates cause meaningful changes to the parametersin Figure 1b.
Revised: We conﬁrm these updates cause meaningful changes to the parameters in Figure 1b.

Original: This ﬁgure depicts the distribution of the parameters before and after the compression.
Revised: This ﬁgure depicts the distribution of the parameters before and after the compression.

Original: We can see that the parameters approach 4 centroids, with 86% of them gathered around 0 .
Revised: We can see that the parameters approach 4 centroids, with 86% of them gathered around 0 .

Original: Overall,the compression results in 112 × reduction in the model size with only 0 .
Revised: Overall, the compression results in 112 × reduction in the model size with only 0 .

Original: 53% drop in accuracy.
Revised: 53% drop in accuracy.

Original: Wenote that the number of pruned parameters in this result are lower compared to previous works suchas Deep Compression.
Revised: We note that the number of pruned parameters in this result are lower compared to previous works such as Deep Compression.

Original: However, the model size reduction is higher due to the smaller dictionary.
Revised: However, the model size reduction is higher due to the smaller dictionary.

Original: [Figure]
Revised: [Figure]

Original: The loss bound (section 2) controls the ﬁnal model size, introducing a trade-off between accuracyand compression.
Revised: The loss bound (section 2) controls the ﬁnal model size, introducing a trade-off between accuracy and compression.

Original: We study this trade-off by incrementing the loss bound and optimizing the modelsize using Algorithm 1.
Revised: We study this trade-off by incrementing the loss bound and optimizing the model size using Algorithm 1.

Original: In the analysis for each of the benchmark datasets we initialize 256 randomcentroids and depict the results in Figure 2.
Revised: In the analysis for each of the benchmark datasets we initialize 256 random centroids and depict the results in Figure 2.

Original: We can see that for small reductions in size, the accuracyremains generally the same.
Revised: We can see that for small reductions in size, the accuracy remains generally the same.

Original: During this stage, most clusters are quickly eliminated due to beingempty.
Revised: During this stage, most clusters are quickly eliminated due to being empty.

Original: As the model size becomes small however, the error starts to increase quickly.
Revised: As the model size becomes small however, the error starts to increase quickly.

Original: We alsoobserve clusters being eliminated less often.
Revised: We also observe clusters being eliminated less often.

Original: These eliminations are a result of merging clusters.
Revised: These eliminations are a result of merging clusters.

Original: Using these analyses, we can ﬁnd the optimal trade-off between the accuracy and model size.
Revised: Using these analyses, we can ﬁnd the optimal trade-off between the accuracy and model size.

Original: Table 2 summarizes the optimal trade-offs obtained from the previous analysis and compares themwith previous works.
Revised: Table 2 summarizes the optimal trade-offs obtained from the previous analysis and compares them with previous works.

Original: With little drop in accuracy, the proposed method achieves state-of-the artaccuracy for the MNIST dataset.
Revised: With little drop in accuracy, the proposed method achieves state-of-the art accuracy for the MNIST dataset.

Original: It also achieves similar reductions in model size for the SVHN and
Revised: It also achieves similar reductions in model size for the SVHN and

Original: CIFAR-10 datasets using the much larger network model.
Revised: CIFAR-10 datasets using the much larger network model.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  C ONCLUSION

Original: In this paper, we presented a method for compressing trained neural network models that directlyminimizes its complexity while maintaining its accuracy.
Revised: In this paper, we presented a method for compressing trained neural network models that directly minimizes its complexity while maintaining its accuracy.

Original: For simplicity of calculations, we chose torepresent the complexity using the k-means objective which is frequently used in previous works.
Revised: For simplicity of calculations, we chose to represent the complexity using the k-means objective which is frequently used in previous works.

Original: In doing so, we maintain the accuracy by introducing a hard constraint on the loss function.
Revised: In doing so, we maintain the accuracy by introducing a hard constraint on the loss function.

Original: Thisconstraint incorporates the information learnt during the training process into the optimization.
Revised: This constraint incorporates the information learnt during the training process into the optimization.

Original: Wethen present an algorithm that iteratively ﬁnds the optimal complexity.
Revised: We then present an algorithm that iteratively ﬁnds the optimal complexity.

Original: We test this solution on severaldatasets and show that it can provide state-of-the-art compression, with little accuracy loss.
Revised: We test this solution on several datasets and show that it can provide state-of-the-art compression, with little accuracy loss.