Original: M ODEL C OMPRESSION VIA D ISTILLATION AND Q UANTIZATION
Revised: M ODEL C OMPRESSION VIA D ISTILLATION AND Q UANTIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNNs) continue to make signiﬁcant advances, solving tasks from image classiﬁcation to translation or reinforcement learning.
Revised: Deep neural networks (DNNs) continue to make signiﬁcant advances, solving tasks from image classiﬁcation to translation or reinforcement learning.

Original: One aspect of the ﬁeld receiving considerable attention is efﬁciently executing deep models in resource-constrained environments, such as mobile or embedded devices.
Revised: One aspect of the ﬁeld receiving considerable attention is efﬁciently executing deep models in resource-constrained environments, such as mobile or embedded devices.

Original: This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger networks, called “teachers,” into compressed “student” networks.
Revised: This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger networks, called “teachers,” into compressed “student” networks.

Original: The ﬁrst method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher network, into the training of a smaller student network whose weights are quantized to a limited set of levels.
Revised: The ﬁrst method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher network, into the training of a smaller student network whose weights are quantized to a limited set of levels.

Original: The second method, differentiable quantization , optimizes the location of quantization points through stochastic gradient descent, to better ﬁt the behavior of the teacher model.
Revised: The second method, differentiable quantization , optimizes the location of quantization points through stochastic gradient descent, to better ﬁt the behavior of the teacher model.

Original: We validate both methods through experiments on convolutional and recurrent architectures.
Revised: We validate both methods through experiments on convolutional and recurrent architectures.

Original: We show that quantized shallow students can reach similar accuracy levels to state-of-the-art full-precision teacher models, while providing up to order of magnitude compression, and inference speedup that is almost linear in the depth reduction.
Revised: We show that quantized shallow students can reach similar accuracy levels to state-of-the-art full-precision teacher models, while providing up to order of magnitude compression, and inference speedup that is almost linear in the depth reduction.

Original: In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.
Revised: In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Background.
Revised: Background.

Original: Neural networks are extremely effective for solving several real world problems, like image classiﬁcation (Krizhevsky et al., 2012;
Revised: Neural networks are extremely effective for solving several real world problems, like image classiﬁcation (Krizhevsky et al., 2012;

Original: He et al., 2016a), translation (Vaswani et al., 2017), voice synthesis (Oord et al., 2016) or reinforcement learning (Mnih et al., 2013; Silver et al., 2016).
Revised: He et al., 2016a), translation (Vaswani et al., 2017), voice synthesis (Oord et al., 2016) or reinforcement learning (Mnih et al., 2013; Silver et al., 2016).

Original: At the same time, modern neural network architectures are often compute, space and power hungry, typically requiring powerful GPUs to train and evaluate.
Revised: At the same time, modern neural network architectures are often compute, space and power hungry, typically requiring powerful GPUs to train and evaluate.

Original: The debate is still ongoing on whether large models are necessary for good accuracy.
Revised: The debate is still ongoing on whether large models are necessary for good accuracy.

Original: It is known that individual network weights can be redundant, and may not carry signiﬁcant information, e.g. Han et al. (2015).
Revised: It is known that individual network weights can be redundant, and may not carry signiﬁcant information, e.g. Han et al. (2015).

Original: At the same time, large models often have the ability to completely memorize datasets (Zhang et al., 2016), yet they do not, but instead appear to learn generic task solutions.
Revised: At the same time, large models often have the ability to completely memorize datasets (Zhang et al., 2016), yet they do not, but instead appear to learn generic task solutions.

Original: A standing hypothesis for why overcomplete representations are necessary is that they make learning possible by transforming local minima into saddle points (Dauphin et al., 2014) or to discover robust solutions, which do not rely on precise weight values (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016).
Revised: A standing hypothesis for why overcomplete representations are necessary is that they make learning possible by transforming local minima into saddle points (Dauphin et al., 2014) or to discover robust solutions, which do not rely on precise weight values (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016).

Original: If large models are only needed for robustness during training, then signiﬁcant compression of these models should be achievable, without impacting accuracy.
Revised: If large models are only needed for robustness during training, then signiﬁcant compression of these models should be achievable, without impacting accuracy.

Original: This intuition is strengthened by two related, but slightly different research directions.
Revised: This intuition is strengthened by two related, but slightly different research directions.

Original: The ﬁrst direction is the work on training quantized neural networks , e.g. Courbariaux et al.
Revised: The ﬁrst direction is the work on training quantized neural networks , e.g. Courbariaux et al.

Original: ; Rastegari et al.
Revised: ; Rastegari et al.

Original: ; Hubara et al.
Revised: ; Hubara et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: (2016a)
Revised: (2016a)

Original: ; Mellempudi et al.
Revised: ; Mellempudi et al.

Original: ; Ott et al.
Revised: ; Ott et al.

Original: ; Zhu et al.
Revised: ; Zhu et al.

Original: (2016), which showed that neural networks can converge to good task solutions even when weights are constrained to having values from a set of integer levels.
Revised: (2016), which showed that neural networks can converge to good task solutions even when weights are constrained to having values from a set of integer levels.

Original: The second direction aims to compress already-trained models, while preserving their accuracy.
Revised: The second direction aims to compress already-trained models, while preserving their accuracy.

Original: To this end, various elegant compression techniques have been proposed, e.g. Han et al.
Revised: To this end, various elegant compression techniques have been proposed, e.g. Han et al.

Original: ; Iandola et al.
Revised: ; Iandola et al.

Original: Wen et al.
Revised: Wen et al.

Original: ; Gysel et al.
Revised: ; Gysel et al.

Original: ; Mishra et al.
Revised: ; Mishra et al.

Original: (2017), which combine quantization, weight sharing, and careful coding of network weights, to reduce the size of state-of-the-art deep models by orders of magnitude, while at the same time speeding up inference.
Revised: (2017), which combine quantization, weight sharing, and careful coding of network weights, to reduce the size of state-of-the-art deep models by orders of magnitude, while at the same time speeding up inference.

Original: Both these research directions are extremely active, and have been shown to yield signiﬁcant compression and accuracy improvements, which can be crucial when making such models available on embedded devices or phones.
Revised: Both these research directions are extremely active, and have been shown to yield signiﬁcant compression and accuracy improvements, which can be crucial when making such models available on embedded devices or phones.

Original: However, the literature on compressing deep networks focuses  exclusively on ﬁnding good compression schemes for a given model, without signiﬁcantly altering the structure of the model .
Revised: However, the literature on compressing deep networks focuses almost exclusively on ﬁnding good compression schemes for a given model, without signiﬁcantly altering the structure of the model .

Original: On the other hand, recent parallel work (Ba & Caruana, 2013; Hinton et al., 2015) introduces the process of distillation , which can be used for transferring the behaviour of a given model to any other structure.
Revised: On the other hand, recent parallel work (Ba & Caruana, 2013; Hinton et al., 2015) introduces the process of distillation , which can be used for transferring the behaviour of a given model to any other structure.

Original: This can be used for compression, e.g. to obtain compact representations of ensembles (Hinton et al., 2015).
Revised: This can be used for compression, e.g. to obtain compact representations of ensembles (Hinton et al., 2015).

Original: However the size of the student model needs to be large enough for allowing learning to succeed.
Revised: However the size of the student model needs to be large enough for allowing learning to succeed.

Original: A model that is too shallow, too narrow, or which misses necessary units, can result in considerable loss of accuracy (Urban et al., 2016).
Revised: A model that is too shallow, too narrow, or which misses necessary units, can result in considerable loss of accuracy (Urban et al., 2016).

Original: In this work, we examine whether distillation and quantization can be jointly leveraged for better compression.
Revised: In this work, we examine whether distillation and quantization can be jointly leveraged for better compression.

Original: We start from the intuition that 1) the existence of highly-accurate, full-precision teacher models should be leveraged to improve the performance of quantized models, while 2) quantizing a model can provide better compression than a distillation process attempting the same space gains by purely decreasing the number of layers or layer width.
Revised: We start from the intuition that 1) the existence of highly-accurate, full-precision teacher models should be leveraged to improve the performance of quantized models, while 2) quantizing a model can provide better compression than a distillation process attempting the same space gains by purely decreasing the number of layers or layer width.

Original: While our approach seems natural and perhaps obvious in hindsight, interesting research questions arise when these two ideas are combined.
Revised: While our approach is very natural, interesting research questions arise when these two ideas are combined.

Original: Contribution.
Revised: Contribution.

Original: We present two methods which allow the user to compound compression in terms of depth , by distilling a shallower student network with similar accuracy to a deeper teacher network, with compression in terms of width , by quantizing the weights of the student to a limited set of integer levels, and using less weights per layer.
Revised: We present two methods which allow the user to compound compression in terms of depth , by distilling a shallower student network with similar accuracy to a deeper teacher network, with compression in terms of width , by quantizing the weights of the student to a limited set of integer levels, and using less weights per layer.

Original: The basic idea is that quantized models can leverage distillation loss (Hinton et al., 2015), which is a weighted average between the correct targets (represented by the labels) and soft targets (represented by the teacher’s outputs).
Revised: The basic idea is that quantized models can leverage distillation loss (Hinton et al., 2015), the weighted average between the correct targets (represented by the labels) and soft targets (represented by the teacher’s outputs).

Original: We implement this intuition via two different methods.
Revised: We implement this intuition via two different methods.

Original: The ﬁrst, called quantized distillation , aims to leverage distillation loss during the training process, by incorporating it into the training of a student network whose weights are constrained to a limited set of levels.
Revised: The ﬁrst, called quantized distillation , aims to leverage distillation loss during the training process, by incorporating it into the training of a student network whose weights are constrained to a limited set of levels.

Original: The second method, which we call differentiable quantization , takes a different approach, by attempting to converge to the optimal location of quantization points through stochastic gradient descent.
Revised: The second method, which we call differentiable quantization , takes a different approach, by attempting to converge to the optimal location of quantization points through stochastic gradient descent.

Original: We validate both methods empirically through a range of experiments on convolutional and recurrent network architectures.
Revised: We validate both methods empirically through a range of experiments on convolutional and recurrent network architectures.

Original: We show that quantized shallow students can reach similar accuracy levels to full-precision  teacher models on datasets such as CIFAR and ImageNet (for image classiﬁcation) and OpenNMT and WMT (for machine translation), while providing up to order of magnitude compression, and inference speedup that is linear in the depth 1 .
Revised: We show that quantized shallow students can reach similar accuracy levels to full-precision and deeper teacher models on datasets such as CIFAR and ImageNet (for image classiﬁcation) and OpenNMT and WMT (for machine translation), while providing up to order of magnitude compression, and inference speedup that is linear in the depth. (Source code will be made available once the paper becomes de-anonymized.)

Original: Related Work.
Revised: Related Work.

Original: Our work is a special case of knowledge distillation (Ba & Caruana, 2013; Hinton et al., 2015), in which we focus on techniques to obtain quantized, shallow students.
Revised: Our work is a special case of knowledge distillation (Ba & Caruana, 2013; Hinton et al., 2015), in which we focus on techniques to obtain high-accuracy students that are both quantized and shallower.

Original: More generally, it can be seen as an instance of learning with privileged information , e.g. Vapnik & Izmailov (2015);
Revised: More generally, it can be seen as a special instance of learning with privileged information , e.g. Vapnik & Izmailov (2015);

Original: Xu et al.
Revised: Xu et al.

Original: (2016), in which the student is provided additional information in the form of outputs from a larger, pre-trained model.
Revised: (2016), in which the student is provided additional information in the form of outputs from a larger, pre-trained model.

Original: The idea of optimizing the locations of quantization points during the learning process, used for differentiable quantization, has been used previously in Lan et al.
Revised: The idea of optimizing the locations of quantization points during the learning process, which we use in differentiable quantization, has been used previously in Lan et al.

Original:  Koren & Sill (2011); Zhang et al.
Revised: (2014); Koren & Sill (2011); Zhang et al.

Original: (2017), although in the different context of matrix completion and recommender systems.
Revised: (2017), although in the different context of matrix completion and recommender systems.

Original: The idea of using distillation for size reduction is mentioned in Hinton et al.
Revised: Using distillation for size reduction is mentioned in Hinton et al.

Original: (2015), for distilling ensembles.
Revised: (2015), for distilling ensembles.

Original: To our knowledge, the only other work using distillation in the context of quantization is Wu et al.
Revised: To our knowledge, the only other work using distillation in the context of quantization is Wu et al.

Original: (2016b), which uses it to improve the accuracy of binary neural networks on ImageNet.
Revised: (2016b), which uses it to improve the accuracy of binary neural networks on ImageNet.

Original: We signiﬁcantly reﬁne this idea here, as we focus on matching or even improving the accuracy of the original full-precision model: for example, our 4-bit quantized version of ResNet18 has higher accuracy thanthe full-precision version (since ittransfers from ResNet34), has higher top-1 accuracy (by > 15%) and top-5 accuracy (by > 7%) compared to the most accurate model in Wu et al.
Revised: We signiﬁcantly reﬁne this idea, as we match or even improve the accuracy of the original full-precision model: for example, our 4-bit quantized version of ResNet18 has higher accuracy than full-precision ResNet18 (matching the accuracy of the ResNet34 teacher): it has higher top-1 accuracy (by > 15%) and top-5 accuracy (by > 7%) compared to the most accurate model in Wu et al.

Original: (2016b).
Revised: (2016b).

Original:  P RELIMINARIES 2. T HE Q UANTIZATION P ROCESS
Revised:  P RELIMINARIES 2. T HE Q UANTIZATION P ROCESS

Original: We start by deﬁning a scaling function sc : R n → [0 , 1] , which normalizes vectors whose values come from an arbitrary range, to vectors whose values are in [0 , 1] .
Revised: We start by deﬁning a scaling function sc : R n → [0 , 1] , which normalizes vectors whose values come from an arbitrary range, to vectors whose values are in [0 , 1] .

Original: Given such a function, the general structure of the quantization functions is as follows:
Revised: Given such a function, the general structure of the quantization functions is as follows:

Original: [Equation]
Revised: [Equation]

Original: where sc − 1 is the inverse of the scaling function, and ˆ Q is the actual quantization function that only accepts values in [0 , 1] .
Revised: where sc − 1 is the inverse of the scaling function, and ˆ Q is the actual quantization function that only accepts values in [0 , 1] .

Original: We always assume v to be a vector; in practice, of course, the weight vectors can be multi-dimensional, but we can reshape them to one dimensional vectors and restore the original dimensions after the quantization.
Revised: We always assume v to be a vector; in practice, of course, the weight vectors can be multi-dimensional, but we can reshape them to one dimensional vectors and restore the original dimensions after the quantization.

Original: Scaling.
Revised: Scaling.

Original: There are various speciﬁcations for the scaling function; in this paper, we will use linear scaling , e.g. He et al.
Revised: There are various speciﬁcations for the scaling function; in this paper, we will use linear scaling , e.g. He et al.

Original: (2016b), that is sc ( v ) = v − βα , with α = max i v
Revised: (2016b), that is sc ( v ) = v − βα , with α = max i v

Original: i − min
Revised: i − min

Original: i v i and β = min
Revised: i v i and β = min

Original: i v i which results in the target values being in [0 , 1] , and the quantization function
Revised: i v i which results in the target values being in [0 , 1] , and the quantization function

Original: [Equation]
Revised: [Equation]

Original: Bucketing.
Revised: Bucketing.

Original: One problem with this formulation is that an identical scaling factor is used for the whole vector, whose dimension might be huge.
Revised: One problem with this formulation is that an identical scaling factor is used for the whole vector, whose dimension might be huge.

Original: Magnitude imbalance can result in a signiﬁcant loss of precision, where most of the elements of the scaled vector are pushed to zero.
Revised: Magnitude imbalance can result in a signiﬁcant loss of precision, where most of the elements of the scaled vector are pushed to zero.

Original: To avoid this, we will use bucketing , e.g. Alistarh et al.
Revised: To avoid this, we will use bucketing , e.g. Alistarh et al.

Original: (2016), that is, we will apply the scaling function separately to buckets of consecutive values of a certain ﬁxed size.
Revised: (2016), that is, we will apply the scaling function separately to buckets of consecutive values of a certain ﬁxed size.

Original: The trade-off here is that we obtain better quantization accuracy for each bucket, but will have to store two ﬂoating-point scaling factors for each bucket.
Revised: The trade-off here is that we obtain better quantization accuracy for each bucket, but will have to store two ﬂoating-point scaling factors for each bucket.

Original: We characterize the compression comparison in Section 5.
Revised: We characterize the compression comparison in Section 5.

Original: The function ˆ Q can also be deﬁned in several ways.
Revised: The function ˆ Q can also be deﬁned in several ways.

Original: We will consider both uniform and non-uniform placement of quantization points.
Revised: We will consider both uniform and non-uniform placement of quantization points.

Original: Uniform Quantization.
Revised: Uniform Quantization.

Original: We ﬁx a parameter s ≥ 1 , describing the number of quantization levels employed.
Revised: We ﬁx a parameter s ≥ 1 , describing the number of quantization levels employed.

Original: Intuitively, uniform quantization considers s + 1 equally spaced points between 0 and(including these endpoints).
Revised: Intuitively, uniform quantization considers s + 1 equally spaced points between 0 and(including these endpoints).

Original: The deterministic version will assign each (scaled) vector coordinate v i to the closest quantization point, while in the stochastic version we perform rounding probabilistically, such that the resulting value is an unbiased estimator of v i , of minimal variance.
Revised: The deterministic version will assign each (scaled) vector coordinate v i to the closest quantization point, while in the stochastic version we perform rounding probabilistically, such that the resulting value is an unbiased estimator of v i , of minimal variance.

Original: Formally, the uniform quantization function with s + 1 levels is deﬁned as
Revised: Formally, the uniform quantization function with s + 1 levels is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where ξ i is the rounding function.
Revised: where ξ i is the rounding function.

Original: For the deterministic version, we deﬁne k
Revised: For the deterministic version, we deﬁne k

Original: i
Revised: i

Original: =
Revised: =

Original: sv i − (cid:98) v i s (cid:99) and set
Revised: sv i − (cid:98) v i s (cid:99) and set

Original: [Equation]
Revised: [Equation]

Original:  while for the stochastic version we will set ξ i ∼ Bernoulli ( k i ) .
Revised: i [Equation] [Equation] while for the stochastic version we will set ξ i ∼ Bernoulli ( k i ) .

Original: Note that k
Revised: Note that k

Original: i is the normalized distance between the original point v i and the closest quantization point that is smaller than v i and that the vector components are quantized independently.
Revised: i is the normalized distance between the original point v i and the closest quantization point that is smaller than v i and that the vector components are quantized independently.

Original: Non-Uniform Quantization.
Revised: Non-Uniform Quantization.

Original: Non-uniform quantization takes as input a set of s quantization points { p 1 , . .
Revised: Non-uniform quantization takes as input a set of s quantization points { p 1 , . .

Original: . , p s } and quantizes each element v i to the closest of these points.
Revised: . , p s } and quantizes each element v i to the closest of these points.

Original: For simplicity, we only deﬁne the deterministic version of this function.
Revised: For simplicity, we only deﬁne the deterministic version of this function.

Original: 2. S TOCHASTIC Q UANTIZATION IS E QUIVALENT TO A DDING G AUSSIAN N OISE
Revised: 2. S TOCHASTIC Q UANTIZATION IS E QUIVALENT TO A DDING G AUSSIAN N OISE

Original: In this section we list some interesting mathematical properties of the uniform quantization function.
Revised: In this section we list some interesting mathematical properties of the uniform quantization function.

Original: Clearly, stochastic uniform quantization is an unbiased estimator of its input, i.e. E [ Q ( v )]
Revised: Clearly, stochastic uniform quantization is an unbiased estimator of its input, i.e. E [ Q ( v )]

Original: = v .
Revised: = v .

Original: What interests us is applying this function to neural networks; as the scalar product is the most common operation performed by neural networks, we would like to study the properties of Q ( v ) T x , where v is the weight vector of a certain layer in the network and x are the inputs.
Revised: What interests us is applying this function to neural networks; as the scalar product is the most common operation performed by neural networks, we would like to study the properties of Q ( v ) T x , where v is the weight vector of a certain layer in the network and x are the inputs.

Original: We are able to show that
Revised: We are able to show that

Original: [Equation]
Revised: [Equation]

Original: where ε is a random variable that is asymptotically normally distributed, i.e.
Revised: where ε is a random variable that is asymptotically normally distributed, i.e.

Original: [Equation]
Revised: [Equation]

Original: Convergence occurs with the dimension n .
Revised: Convergence occurs with the dimension n .

Original: For a formal statement and proof, see Section B.1 in the
Revised: For a formal statement and proof, see Section B.1 in the

Original: Appendix.
Revised: Appendix.

Original: This means that quantizing the weights is equivalent to adding to the output of each layer (before the activation function) a zero-mean error term that is asymptotically normally distributed.
Revised: This means that quantizing the weights is equivalent to adding to the output of each layer (before the activation function) a zero-mean error term that is asymptotically normally distributed.

Original: The variance of this error term depends on s .
Revised: The variance of this error term depends on s .

Original: This presents an interesting connection to work that advocates adding noise to intermediary activations of neural networks as a regularizer, e.g. Gulcehre et al.
Revised: This connects quantization to work advocating adding noise to intermediary activations of neural networks as a regularizer, e.g. Gulcehre et al.

Original:  Q UANTIZED D ISTILLATION
Revised:  Q UANTIZED D ISTILLATION

Original: The context is the following: given a task, we consider a trained state-of-the-art deep model solving it–the teacher , and a compressed student model.
Revised: The context is the following: given a task, we consider a trained state-of-the-art deep model solving it–the teacher , and a compressed student model.

Original: The student is compressed in the sense that 1) it is shallower than the teacher; and 2) it is quantized , in the sense that its weights are expressed at limited bit width.
Revised: The student is compressed in the sense that 1) it is shallower than the teacher; and 2) it is quantized , in the sense that its weights are expressed at limited bit width.

Original: The strategy, as for standard distillation (Ba & Caruana, 2013; Hinton et al., 2015) is for the student to leverage the converged teacher model to reach similar accuracy.
Revised: The strategy, as for standard distillation (Ba & Caruana, 2013; Hinton et al., 2015) is for the student to leverage the converged teacher model to reach similar accuracy.

Original: We note that distillation has been used previously to obtain compact high-accuracy encodings of ensembles (Hinton et al., 2015); however, we believe this is the ﬁrst time it is used for model compression via quantization.
Revised: We note that distillation has been used previously to obtain compact high-accuracy encodings of ensembles (Hinton et al., 2015); however, we believe this is the ﬁrst time it is used for model compression via quantization.

Original: Given this setup, there are two questions we need to address.
Revised: Given this setup, there are two questions we need to address.

Original: The ﬁrst is how to transfer knowledge from the teacher to the student.
Revised: The ﬁrst is how to transfer knowledge from the teacher to the student.

Original: For this, the student will use the distillation loss , as deﬁned by Hinton et al.
Revised: For this, the student will use the distillation loss , as deﬁned by Hinton et al.

Original: (2015), as the weighted average between two objective functions: cross entropy with soft targets , controlled by the temperature parameter T , and the cross entropy with the correct labels .
Revised: (2015), as the weighted average between two objective functions: cross entropy with soft targets , controlled by the temperature parameter T , and the cross entropy with the correct labels .

Original: We refer the reader to Hinton et al.
Revised: We refer the reader to Hinton et al.

Original: (2015) for the precise deﬁnition of distillation loss.
Revised: (2015) for the precise deﬁnition of distillation loss.

Original: The second question is how to employ distillation loss in the context of a quantized neural network.
Revised: The second question is how to employ distillation loss in the context of a quantized neural network.

Original: An intuitive approach is to rely on projected gradient descent, where a gradient step is taken as in full-precision training, and then the new parameters are projected to the set of valid solutions.
Revised: An intuitive approach is to rely on projected gradient descent, where a gradient step is taken as in full-precision training, and then the new parameters are projected to the set of valid solutions.

Original: Critically, we accumulate the error at each projection step into the gradient for the next step.
Revised: Critically, we accumulate the error at each projection step into the gradient for the next step.

Original: One can think of this process as if collecting evidence for whether each weight needs to move to the next quantization point or not .
Revised: One can think of this process as if collecting evidence for whether each weight needs to move to the next quantization point or not .

Original: Crucially, the error accumulation prevents the algorithm from getting stuck in the current solution if gradients are small, which would occur in a naive projected gradient approach.
Revised: Crucially, the error accumulation prevents the algorithm from getting stuck in the current solution if gradients are small, which would occur in a naive projected gradient approach.

Original: This is simillar to the approach taken by BinaryConnect technique, with some differences.
Revised: This is simillar to the approach taken by BinaryConnect technique, with some differences.

Original: Li et al.
Revised: Li et al.

Original: (2017) also examines these dynamics in detail.
Revised: (2017) also examines these dynamics in detail.

Original: Compared to BinnaryConnect, we use distillation rather than learning from scratch, hence learning more effeciently.
Revised: Compared to BinnaryConnect, we use distillation rather than learning from scratch, hence learning more effeciently.

Original: We also do not restrict ourselves to binary representation, but rather use variable bit-width quantization frunctions and bucketing , as deﬁned in Section 2.
Revised: We also do not restrict ourselves to binary representation, but rather use variable bit-width quantization frunctions and bucketing , as deﬁned in Section 2.

Original: An alternative view of this process, illustrated in Figure 1, is that we perform the SGD step on the full-precision model, but computing the gradient on the quantized model , expressed with respect to the distillation loss .
Revised: An alternative view of this process, illustrated in Figure 1, is that we perform the SGD step on the full-precision model, but computing the gradient on the quantized model , expressed with respect to the distillation loss .

Original: With all this in mind, the algorithm we propose is: 
Revised: With all this in mind, the algorithm we propose is: Algorithm

Revised: Let w be the network weights

Original:  D IFFERENTIABLE Q UANTIZATION 4. G ENERAL D ESCRIPTION
Revised:  D IFFERENTIABLE Q UANTIZATION 4. G ENERAL D ESCRIPTION

Original: We introduce differentiable quantization as a general method of improving the accuracy of a quantized neural network, by exploiting non-uniform quantization point placement.
Revised: We introduce differentiable quantization as a general method of improving the accuracy of a quantized neural network, by exploiting non-uniform quantization point placement.

Original: In particular, we are going to use the non-uniform quantization function deﬁned in Section 2.1.
Revised: In particular, we are going to use the non-uniform quantization function deﬁned in Section 2.1.

Original: Experimentally, we have found little difference between stochastic and deterministic quantization in this case, and therefore will focus on the simpler deterministic quantization function here.
Revised: Experimentally, we have found little difference between stochastic and deterministic quantization in this case, and therefore will focus on the simpler deterministic quantization function here.

Original: 1: procedure Q UANTIZED D ISTILLATION 2:

Original: Let w be the network weights 3: loop 4: w q ← quant function ( w, s ) 5:

Original: Run forward pass and compute distillation loss l ( w q ) 6:

Original: Run backward pass and compute ∂l ( w q )

Original: ∂w q 7:

Original: Update original weights using SGD in full precision w = w − ν · ∂l ( w q ) ∂w q 8: Finally quantize the weights before returning: w q ← quant function ( w, s ) 9: return w q quantize quantize quantize SGD step distil

Original: SGD step distil

Original: SGD step distil

Original: [Figure]
Revised: [Figure]

Original: Let p =
Revised: Let p =

Original: ( p 1 , . . .
Revised: ( p 1 , . . .

Original: , p s ) be the vector of quantization points, and let Q ( v, p ) be our quantization function, as deﬁned previously.
Revised: , p s ) be the vector of quantization points, and let Q ( v, p ) be our quantization function, as deﬁned previously.

Original: Ideally, we would like to ﬁnd a set of quantization points p which minimizes the accuracy loss when quantizing the model using Q ( v, p ) .
Revised: Ideally, we would like to ﬁnd a set of quantization points p which minimizes the accuracy loss when quantizing the model using Q ( v, p ) .

Original: The key observation is that to ﬁnd this set p , we can just use stochastic gradient descent, because we are able to compute the gradient of Q with respect to p .
Revised: The key observation is that to ﬁnd this set p , we can just use stochastic gradient descent, because we are able to compute the gradient of Q with respect to p .

Original: A major problem in quantizing neural networks is the fact that the decision of which p i should replace a given weight is discrete, hence the gradient is zero: ∂Q ( v, p ) ∂v = 0 , almost everywhere .
Revised: A major problem in quantizing neural networks is the fact that the decision of which p i should replace a given weight is discrete, hence the gradient is zero: ∂Q ( v, p ) ∂v = 0 , almost everywhere .

Original: This implies that we cannot backpropagate the gradients through the quantization function.
Revised: This implies that we cannot backpropagate the gradients through the quantization function.

Original: To solve this problem, typically a variant of the straight-through estimator is used, see e.g. Bengio
Revised: To solve this problem, typically a variant of the straight-through estimator is used, see e.g. Bengio

Original: et al.
Revised: et al.

Original: ; Hubara et al.
Revised: ; Hubara et al.

Original: On the other hand, the model as a function of the chosen p i is continuous and can be differentiated ; the gradient of Q ( v, p ) i with respect to p j is well deﬁned almost everywhere, and it is simply
Revised: On the other hand, the model as a function of the chosen p i is continuous and can be differentiated ; the gradient of Q ( v, p ) i with respect to p j is well deﬁned almost everywhere, and it is simply

Original: [Equation]
Revised: [Equation]

Original:  where α i is i -th element of the scaling factor, assuming we are using a bucketing scheme.
Revised: quantized [Equation] [Equation] where α i is i -th element of the scaling factor, assuming we are using a bucketing scheme.

Original: If no bucketing is used, then α i = α for every i .
Revised: If no bucketing is used, then α i = α for every i .

Original: Otherwise it changes depending on which bucket the weight v
Revised: Otherwise it changes depending on which bucket the weight v

Original: i belongs to.
Revised: i belongs to.

Original: Therefore, we can use the same loss function we used when training the original model, and with
Revised: Therefore, we can use the same loss function we used when training the original model, and with

Original: Equation (6) and the usual backpropagation algorithm we are able to compute its gradient with respect to the quantization points p .
Revised: Equation (6) and the usual backpropagation algorithm we are able to compute its gradient with respect to the quantization points p .

Original: Then we can minimize the loss function with respect to p with the standard SGD algorithm.
Revised: Then we can minimize the loss function with respect to p with the standard SGD algorithm.

Original: The algorithm then becomes the following: 
Revised: The algorithm then becomes the following: Algorithm

Revised: 1: procedure D IFFERENTIABLE Q UANTIZATION 2:

Revised: Let w be the networks weights and p the initial quantization points 3: loop 4: w q ← quant function ( w, p ) 5:

Revised: Run forward pass and compute loss l ( w q ) 6:

Revised: Run backward pass and compute ∂l ( w q )

Original:  Note on Efﬁciency.
Revised: ∂w q 7: Use equation above to compute ∂l ( w q ) ∂p 8: Update quantization points using SGD or similar: p = p − ν · ∂l ( w q ) ∂p 9: return p Note on Efﬁciency.

Original: Optimizing the points p can be slower than training the original network, since we have to perform the normal forward and backward pass, and in addition we need to quantize the weights of the model and perform the backward pass to get to the gradients w.r.t.
Revised: Optimizing the points p can be slower than training the original network, since we have to perform the normal forward and backward pass, and in addition we need to quantize the weights of the model and perform the backward pass to get to the gradients w.r.t.

Original: p .
Revised: p .

Original: However, in our experience differential quantization requires an order of magnitude less iterations to converge to a good solution, and can be implemented efﬁciently.
Revised: However, in our experience differential quantization requires an order of magnitude less iterations to converge to a good solution, and can be implemented efﬁciently.

Original: Weight Sharing.
Revised: Weight Sharing.

Original: Upon close inspection, this method can be related to weight sharing Han et al.
Revised: Upon close inspection, this method can be related to weight sharing Han et al.

Original: Weight sharing uses a k -mean clustering algorithm to ﬁnd good clusters for the weights, adopting the centroids as quantization points for a cluster.
Revised: Weight sharing uses a k -mean clustering algorithm to ﬁnd good clusters for the weights, adopting the centroids as quantization points for a cluster.

Original: The network is trained modifying the

Original: 1: procedure D IFFERENTIABLE Q UANTIZATION 2:

Original: Let w be the networks weights and p the initial quantization points 3: loop 4: w q ← quant function ( w, p ) 5:

Original: Run forward pass and compute loss l ( w q ) 6:

Original: Run backward pass and compute ∂l ( w q ) ∂w q 7: Use equation above to compute ∂l ( w q ) ∂p 8: Update quantization points using SGD or similar: p = p − ν · ∂l ( w q ) ∂p 9: return p values of the centroids, aggregating the gradient in a similar fashion.
Revised: The network is trained modifying the values of the centroids, aggregating the gradient in a similar fashion.

Original: The difference is in the initial assignment of points to centroids, but also, more importantly, in the fact that the assignment of weights to centroids never changes.
Revised: The difference is in the initial assignment of points to centroids, but also, more importantly, in the fact that the assignment of weights to centroids never changes.

Original: By contrast, at every iteration we re-assign weights to the closest quantization point, and use a different initialization.
Revised: By contrast, at every iteration we re-assign weights to the closest quantization point, and use a different initialization.

Original: 4. D ISCUSSION AND A DDITIONAL H EURISTICS
Revised: 4. D ISCUSSION AND A DDITIONAL H EURISTICS

Original: While the loss is continuous w.r.t.
Revised: While the loss is continuous w.r.t.

Original: p , there are indirect effects when changing the way each weight gets quantized.
Revised: p , there are indirect effects when changing the way each weight gets quantized.

Original: This can have drastic effect on the learning process.
Revised: This can have drastic effect on the learning process.

Original: As an extreme example, we could have degeneracies, where all weights get represented by the same quantization point, making learning impossible.
Revised: As an extreme example, we could have degeneracies, where all weights get represented by the same quantization point, making learning impossible.

Original: Or diversity of p i gets reduced, resulting in very few weights being represented at a really high precision while the rest are forced to be represented in a much lower resolution.
Revised: Or diversity of p i gets reduced, resulting in very few weights being represented at a really high precision while the rest are forced to be represented in a much lower resolution.

Original: To avoid such issues, we rely on the following set of heuristics.
Revised: To avoid such issues, we rely on the following set of heuristics.

Original: Future work will look at adding a reinforcement learning loss for how the p i are assigned to weights.
Revised: Future work will look at adding a reinforcement learning loss for how the p i are assigned to weights.

Original: Choose good starting points.
Revised: Choose good starting points.

Original: One way to initialize the starting quantization points is to make them uniformly spaced, which would correspond to use as a starting point the uniform quantization function.
Revised: One way to initialize the starting quantization points is to make them uniformly spaced, which would correspond to use as a starting point the uniform quantization function.

Original: The differentiable quantization algorithm needs to be able to use a quantization point in order to update it; therefore, to make sure every quantization point is used we initialize the points to be the quantiles of the weight values.
Revised: The differentiable quantization algorithm needs to be able to use a quantization point in order to update it; therefore, to make sure every quantization point is used we initialize the points to be the quantiles of the weight values.

Original: This ensures that every quantization point is associated with the same number of values and we are able to update it.
Revised: This ensures that every quantization point is associated with the same number of values and we are able to update it.

Original: Redistribute bits where it matters.
Revised: Redistribute bits where it matters.

Original: Not all layers in the network need the same accuracy.
Revised: Not all layers in the network need the same accuracy.

Original: A measure of how important each weight is to the ﬁnal prediction is the norm of the gradient of each weight vector.
Revised: A measure of how important each weight is to the ﬁnal prediction is the norm of the gradient of each weight vector.

Original: So in an initial phase we run the forward and backward pass a certain number of times to estimate the gradient of the weight vectors in each layer, we compute the average gradient across multiple minibatches and compute the norm; we then allocate the number of points associated with each weight according to a simple linear proportion.
Revised: So in an initial phase we run the forward and backward pass a certain number of times to estimate the gradient of the weight vectors in each layer, we compute the average gradient across multiple minibatches and compute the norm; we then allocate the number of points associated with each weight according to a simple linear proportion.

Original: In short we estimate
Revised: In short we estimate

Original: [Equation]
Revised: [Equation]

Original:  where l is the loss function, v is the vector of weights in a particular layer and (cid:0) ∂l∂v (cid:1) i = ∂l∂v
Revised: ) where l is the loss function, v is the vector of weights in a particular layer and (cid:0) ∂l∂v (cid:1) i = ∂l∂v

Original: i and we use this value to determine which layers are most sensitive to quantization.
Revised: i and we use this value to determine which layers are most sensitive to quantization.

Original: When using this process, we will use more than the indicated number of bits in some layers, and less in others.
Revised: When using this process, we will use more than the indicated number of bits in some layers, and less in others.

Original: We can reduce the impact of this effect with the use of Huffman encoding, see Section 5; in any case, note that while the total number of points stays constant, allocating more points to a layer will increase bit complexity overall if the layer has a larger proportion of the weights.
Revised: We can reduce the impact of this effect with the use of Huffman encoding, see Section 5; in any case, note that while the total number of points stays constant, allocating more points to a layer will increase bit complexity overall if the layer has a larger proportion of the weights.

Original: Use the distillation loss.
Revised: Use the distillation loss.

Original: In the algorithm delineated above, the loss refers to the loss we used to train the original model with.
Revised: In the algorithm delineated above, the loss refers to the loss we used to train the original model with.

Original: Another possible speciﬁcation is to treat the unquantized model as the teacher model, the quantized model as the student, and to use as loss the distillation loss between the outputs of the unquantized and quantized model.
Revised: Another possible speciﬁcation is to treat the unquantized model as the teacher model, the quantized model as the student, and to use as loss the distillation loss between the outputs of the unquantized and quantized model.

Original: In this case, then, we are optimizing our quantized model not to perform best with respect to the original loss, but to mimic the results of the unquantized model, which should be easier to learn for the model and provide better results.
Revised: In this case, then, we are optimizing our quantized model not to perform best with respect to the original loss, but to mimic the results of the unquantized model, which should be easier to learn for the model and provide better results.

Original: Hyperparameter optimization.
Revised: Hyperparameter optimization.

Original: The algorithm above is an optimization problem very similar to the original one.
Revised: The algorithm above is an optimization problem very similar to the original one.

Original: As usual, to obtain the best results one should experiment with hyperparameters optimization, and different variants of gradient descent.
Revised: As usual, to obtain the best results one should experiment with hyperparameters optimization, and different variants of gradient descent.

Original:  C OMPRESSION
Revised:  C OMPRESSION

Original: We now analyze the space savings when using b bits and bucket size of k .
Revised: We now analyze the space savings when using b bits and bucket size of k .

Original: Let f be the size of full precision weights (32 bit) and let N be the size of the “vector” we are quantizing.
Revised: Let f be the size of full precision weights (32 bit) and let N be the size of the “vector” we are quantizing.

Original: Full precision requires fN bits, while the quantized vector requires bN + 2 fNk .
Revised: Full precision requires fN bits, while the quantized vector requires bN + 2 fNk .

Original: (We use b bits per weight, plus the scaling factors α and β for every bucket).
Revised: (We use b bits per weight, plus the scaling factors α and β for every bucket).

Original: The size gain is therefore g ( b, k ; f ) 
Revised: The size gain is therefore g ( b, k ; f ) =

Original: =
Revised: [Equation]

Original: kf kb +2 f .
Revised: +2 f .

Original: For differentiable quantization, we also have to store the values of the quantization points.
Revised: For differentiable quantization, we also have to store the values of the quantization points.

Original: Since this number does not depend on N , the amount of space required is negligible and we ignore it for simplicity.
Revised: Since this number does not depend on N , the amount of space required is negligible and we ignore it for simplicity.

Original: For example, at 256 bucket size, using 2 bits per component yields 14 . 2 × space savings w.r.t.
Revised: For example, at 256 bucket size, using 2 bits per component yields 14 . 2 × space savings w.r.t.

Original: full precision, while 4 bits yields 7 . 52 × space savings.
Revised: full precision, while 4 bits yields 7 . 52 × space savings.

Original: At 512 bucket size, the 2 bit savings are 15 .
Revised: At 512 bucket size, the 2 bit savings are 15 .

Original: 05 × , while 4 bits yields 7 .
Revised: 05 × , while 4 bits yields 7 .

Original: 75 × compression.
Revised: 75 × compression.

Original: Huffman encoding.
Revised: Huffman encoding.

Original: To save additional space, we can use Huffman encoding to represent the quantized values.
Revised: To save additional space, we can use Huffman encoding to represent the quantized values.

Original: In fact, each quantized value can be thought as the pointer to a full precision value; in the case of non uniform quantization is p k , in the case of uniform quantization is k/s .
Revised: In fact, each quantized value can be thought as the pointer to a full precision value; in the case of non uniform quantization is p k , in the case of uniform quantization is k/s .

Original: We can then compute the frequency for every index across all the weights of the model and compute the optimal
Revised: We can then compute the frequency for every index across all the weights of the model and compute the optimal

Original: Huffman encoding.
Revised: Huffman encoding.

Original: The mean bit length of the optimal encoding is the amount of bits we actually use to encode the values.
Revised: The mean bit length of the optimal encoding is the amount of bits we actually use to encode the values.

Original: This explains the presence of fractional bits in some of our size gain tables from the Appendix.
Revised: This explains the presence of fractional bits in some of our size gain tables from the Appendix.

Original: We emphasize that we only use these compression numbers as a ballpark ﬁgure, since additional implementation costs might mean that these savings are not always easy to translate to practice Han et al.
Revised: We emphasize that we only use these compression numbers as a ballpark ﬁgure, since additional implementation costs might mean that these savings are not always easy to translate to practice Han et al.

Original:  E XPERIMENTAL R ESULTS 6. S MALL DATASETS
Revised:  E XPERIMENTAL R ESULTS 6. S MALL DATASETS

Original: Methods.
Revised: Methods.

Revised: We will begin with a set of experiments on smaller datasets, which allow us to more carefully cover the parameter space.

Original: Wewill compare the performance of the methods described in the following way: we consider as baseline the teacher model, the distilled model and a smaller model: the distilled and smaller models have the same architecture, but the distilled model is trained using distillation loss on the teacher, while the smaller model is trained directly on targets.
Revised: We compare the performance of the methods described in the following way: we consider as baseline the teacher model, the distilled model and a smaller model: the distilled and smaller models have the same architecture, but the distilled model is trained using distillation loss on the teacher, while the smaller model is trained directly on targets.

Original: Further, we compare the performance of Quantized Distillation and Differentiable Quantization.
Revised: Further, we compare the performance of Quantized Distillation and Differentiable Quantization.

Original: In addition, we will also use PM (“post-mortem”) quantization, which uniformly quantizes the weights after training without any additional operation, with and without bucketing.
Revised: In addition, we will also use PM (“post-mortem”) quantization, which uniformly quantizes the weights after training without any additional operation, with and without bucketing.

Original: All the results are obtained with a bucket size of 256, which we found to empirically provide a good compression-accuracy trade-off.
Revised: All the results are obtained with a bucket size of 256, which we found to empirically provide a good compression-accuracy trade-off.

Original: We refer the reader to Appendix A forfurther details of the datasets and models we consider.
Revised: We refer the reader to Appendix A for details of the datasets and models.

Original: CIFAR-10 Experiments.
Revised: CIFAR-10 Experiments.

Original: For image classiﬁcation on CIFAR-10, we tested the impact of different training techniques on the accuracy of the distilled model, while varying the parameters of a CNN architecture, such as quantization levels and model size.
Revised: For image classiﬁcation on CIFAR-10, we tested the impact of different training techniques on the accuracy of the distilled model, while varying the parameters of a CNN architecture, such as quantization levels and model size.

Original: Table 1 contains the results for full-precision training, PM quantization with and without bucketing, as well as our methods.
Revised: Table 1 contains the results for full-precision training, PM quantization with and without bucketing, as well as our methods.

Original: The percentages on the left below the student models deﬁnition are the accuracy of the normal and the distilled model respectively (trained with full precision).
Revised: The percentages on the left below the student models deﬁnition are the accuracy of the normal and the distilled model respectively (trained with full precision).

Original: More details are reported in table 11 in the appendix.
Revised: More details are reported in table 11 in the appendix.

Original: We also tried an additional model where the student is deeper than the teacher, where we obtained that the student quantized to 4 bits is able to achieve signiﬁcantly better accuracy than the teacher, with a compression factor of more than 7 × .
Revised: We also tried an additional model where the student is deeper than the teacher, where we obtained that the student quantized to 4 bits is able to achieve signiﬁcantly better accuracy than the teacher, with a compression factor of more than 7 × [Table]

Original:  We performed additional experiments for differentiable quantization using a wide residual network (Zagoruyko & Komodakis, 2016) that gets to higher accuracies, see table 
Revised: [Table] We performed additional experiments for differentiable quantization using a wide residual network (Zagoruyko & Komodakis, 2016) that gets to higher accuracies; see table 3.

Original: Overall, quantized distillation appears to be the method with best accuracy across the whole range of bit widths and architectures.
Revised: Overall, quantized distillation appears to be the method with best accuracy across the whole range of bit widths and architectures.

Original: It outperforms PM signiﬁcantly for 2bit and 4bit quantization, achieves  accuracy within 0 .
Revised: It outperforms PM signiﬁcantly for 2bit and 4bit quantization, achieves [Table] [Table] accuracy within 0 .

Original: 2% of the teacher at 8 bits on the larger student model, and relatively minor accuracy loss at 4bit quantization.
Revised: 2% of the teacher at 8 bits on the larger student model, and relatively minor accuracy loss at 4bit quantization.

Original: Differentiable quantization is a close second on all experiments, but it has much faster convergence.
Revised: Differentiable quantization is a close second on all experiments, but it has much faster convergence.

Original: Further, we highlight the good accuracy of the much simpler
Revised: Further, we highlight the good accuracy of the much simpler

Original: PM quantization method with bucketing at higher bit width (4 and 8 bits).
Revised: PM quantization method with bucketing at higher bit width (4 and 8 bits).

Original: CIFAR-100 Experiments.
Revised: CIFAR-100 Experiments.

Original: Next, we perform image classiﬁcation with the full 100 classes.
Revised: Next, we perform image classiﬁcation with the full 100 classes.

Original: Here, we focus on 2bit and 4bit quantization, and on a single student architecture.
Revised: Here, we focus on 2bit and 4bit quantization [Table] and on a single student architecture.

Original: The baseline architecture is a wide residual network with 28 layers, and 36.5M parameters, which is state-of-the-art for its depth on this dataset.
Revised: The baseline architecture is a wide residual network with 28 layers [Table] and 36.5M parameters, which is state-of-the-art for its depth on this dataset.

Original: The student has depth and width reduced by 20%, and half the parameters.
Revised: The student has depth and width reduced by 20%

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table]
Revised: and half the parameters.

Original: It is chosen so that reaches the same accuracy as the teacher model when distilled at full precision.
Revised: It is chosen so that reaches the same accuracy as the teacher model when distilled at full precision.

Original: Accuracy results are given in Table 4.
Revised: Accuracy results are given in Table 4.

Original: More details are reported in Table 20, in the appendix.
Revised: More details are reported in Table 20, in the appendix.

Original: [Table]
Revised: [Table]

Original: The results conﬁrm the trend from the previous dataset, with distilled and differential quantization preserving accuracy within less than 1% at 4bit precision.
Revised: The results conﬁrm the trend from the previous dataset, with distilled and differential quantization preserving accuracy within less than 1% at 4bit precision.

Original: However, we note that accuracy loss is catastrophic at 2bit precision, probably because of reduced model capacity.
Revised: However, we note that accuracy loss is catastrophic at 2bit precision, probably because of reduced model capacity.

Original: Differentiable quantization is able to best recover accuracy for this harder task [Table]
Revised: We note that differentiable quantization is able to best recover accuracy for this harder task.

Original: OpenNMT Experiments.
Revised: OpenNMT Experiments.

Original: The OpenNMT integration test dataset (Ope) consists of 200K train sentences and 10K test sentences for a German-English translation task.
Revised: The OpenNMT integration test dataset (Ope) consists of 200K train sentences and 10K test sentences for a German-English translation task.

Original: To train and testthe models wewill use the OpenNMT pytorch codebase (Klein et al., 2017).
Revised: To train and test models we use the OpenNMT PyTorch codebase (Klein et al., 2017).

Original: We modiﬁed the code, in particular by adding the quantization algorithms and the distillation loss.
Revised: We modiﬁed the code, in particular by adding the quantization algorithms and the distillation loss.

Original: As measure of ﬁt we will use perplexity and the BLEU score, the latter computed using the multi-bleu.perl code from the moses project (mos).
Revised: As measure of ﬁt we will use perplexity and the BLEU score, the latter computed using the multi-bleu.perl code from the moses project (mos).

Original: Our target models consist of an embedding layer, an encoder consisting of n layers of LSTM, a decoder consisting of n layers of LSTM, and a linear layer.
Revised: Our target models consist of an embedding layer, an encoder consisting of n layers of LSTM, a decoder consisting of n layers of LSTM, and a linear layer.

Original: The decoder also uses the global attention mechanism described in Luong et al.
Revised: The decoder also uses the global attention mechanism described in Luong et al.

Original: For the teacher network we set n = 2 , for a total of 4 LSTM layers with LSTM size 500 .
Revised: For the teacher network we set n = 2 , for a total of 4 LSTM layers with LSTM size 500 .

Original: For the student networks we choose n = 1 , for a total of
Revised: For the student networks we choose n = 1 , for a total of

Original: LSTM layers.
Revised: LSTM layers.

Original: We vary the LSTM size of the student networks and for each one, we compute the distilled model and the quantized versions for varying bit width.
Revised: We vary the LSTM size of the student networks and for each one, we compute the distilled model and the quantized versions for varying bit width.

Original: Results are summarized in Table 5.
Revised: Results are summarized in Table 5.

Original: The BLEU scores below the student model refer to the BLEU scores of the normal and distilled model respectively (trained with full precision).
Revised: The BLEU scores below the student model refer to the BLEU scores of the normal and distilled model respectively (trained with full precision).

Original: Details about the resulting size of the models are reported in table 23 in the appendix.
Revised: Details about the resulting size of the models are reported in table 23 in the appendix.

Revised: A reasonable intuition would be that recurrent neural networks should be harder to quantize than convolutional neural networks, as quantization errors do not average out when executing repeatedly

Original: [Table]
Revised: [Table]

Original: Our intuition was that recurrent neural networks should in theory be harder to quantize than convolutional neural networks, as quantization errors do not average out when executing repeatedly through the same cell
Revised: through the same cell

Original: [Table]
Revised: [Table]

Original: but accumulate.
Revised: but accumulate.

Original: Results contradict this intuition.
Revised: Results contradict this intuition.

Original: In particular, medium and large-sized students are able to essentially recover the same scores as the teacher model on this dataset.
Revised: In particular, medium and large-sized students are able to essentially recover the same scores as the teacher model on this dataset.

Original: Perhaps surprisingly, bucketing PM and quantized distillation perform equally well for 4bit quantization.
Revised: Perhaps surprisingly, bucketing PM and quantized distillation perform equally well for 4bit quantization.

Original: As expected, cell size is an important indicator for accuracy, although halving both cell size and the number of layers can be done without signiﬁcant loss.
Revised: As expected, cell size is an important indicator for accuracy, although halving both cell size and the number of layers can be done without signiﬁcant loss.

Original: 6. L ARGER DATASETS
Revised: 6. L ARGER DATASETS

Original: WMT13 Experiments.
Revised: WMT13 Experiments.

Original: We run a similar  architecture  for the WMT13 dataset (Koehn, 2005) (1.7M sentences train, 190K sentences test) and we provide additional experiments for quantized distillation technique, see Table 6.
Revised: We run a similar LSTM architecture as above for the WMT dataset (Koehn, 2005) (1.7M sentences train, 190K sentences test) and we provide additional experiments for quantized distillation technique, see Table 6.

Original: We note that, on this large dataset, PM quantization does not perform well [Table] even with bucketing.
Revised: We note that, on this large dataset, PM quantization does not perform well, even with bucketing.

Original: On the other hand, quantized distillation with 4bits of precision has higher BLEU score than the teacher, and similar perplexity.
Revised: On the other hand, quantized distillation with 4bits of precision has higher BLEU score than the teacher, and similar perplexity.

Original: [Table]
Revised: [Table]

Original: ) 27.73 BLEU - 7.4 ppl Quantized Distill. 35.32 BLEU - 6.48 ppl

Original: The ImageNet Dataset.
Revised: The ImageNet Dataset.

Original: We  experiment with ImageNet using the ResNet architecture He et al.
Revised: We also experiment with ImageNet using the ResNet architecture He et al.

Original: (2016a).
Revised: (2016a).

Original: We use a ResNet34 teacher, and a student ResNet18 student model.
Revised: In the ﬁrst experiment, we use a ResNet34 teacher, and a student ResNet18 student model.

Original: Experiments with the standard version of this student resulted in an accuracy loss of around 4% , and hence we experiment with a wider model, which doubles the number of ﬁlters for each convolutional layer.
Revised: Experiments quantizing the standard version of this student resulted in an accuracy loss of around 4% , and hence we experiment with a wider model, which doubles the number of ﬁlters for each convolutional layer.

Original: We call this 2xResNet18.
Revised: We call this 2xResNet18.

Original: This is in line with previous work on wide ResNet architectures (Zagoruyko & Komodakis, 2016), wide students for distillation (Ba & Caruana, 2013), and wider quantized networks (Mishra et al., 2017).
Revised: This is in line with previous work on wide ResNet architectures (Zagoruyko & Komodakis, 2016), wide students for distillation (Ba & Caruana, 2013), and wider quantized networks (Mishra et al., 2017).

Original: We also note that, in line with previous work on the ImageNet dataset (Zhu et al., 2016; Mishra et al., 2017), we do not quantize the ﬁrst and last layers of the models, as this cansigniﬁcantly hurt accuracy.
Revised: We also note that, in line with previous work on this dataset (Zhu et al., 2016; Mishra et al., 2017), we do not quantize the ﬁrst and last layers of the models, as this can hurt accuracy.

Original: After 62 epochs of training, the quantized distilled 2xResNet18 with 4 bits reaches a validation accuracy of 73.31%.
Revised: After 62 epochs of training, the quantized distilled 2xResNet18 with 4 bits reaches a validation accuracy of 73.31%.

Original: Surprisingly, this is higher than the unquantized ResNet18 model (69.75%), and has virtually the same accuracy as the ResNet34 teacher.
Revised: Surprisingly, this is higher than the unquantized ResNet18 model (69.75%), and has virtually the same accuracy as the ResNet34 teacher.

Original: In terms of size, this model is more than 2 × smaller than ResNet18 (but has higher accuracy), and is 4 × smaller than ResNet34, and about 1 . 5 × faster on inference, as it has fewer layers.
Revised: In terms of size, this model is more than 2 × smaller than ResNet18 (but has higher accuracy), and is 4 × smaller than ResNet34, and about 1 . 5 × faster on inference, as it has fewer layers.

Original: This is state-of-the-art for 4bit models with 18 layers; to our knowledge, no such model has been able to surpass the accuracy of ResNet18.
Revised: This is state-of-the-art for 4bit models with 18 layers; to our knowledge, no such model has been able to surpass the accuracy of ResNet18.

Revised: We re-iterated this experiment using a 4-bit quantized 2xResNet34 student transferring from a

Revised: ResNet50 full-precision teacher.

Revised: We obtain a 4-bit quantized student of almost the same accuracy, which is 50% shallower and has a 2 .

Revised: 5 × smaller size.

Original: [Table]
Revised: [Table]

Original: 6. A DDITIONAL E XPERIMENTS
Revised: A DDITIONAL E XPERIMENTS

Original: Distillation Loss versus Normal Loss.
Revised: Distillation Loss versus Normal Loss.

Original: One key question we are interested in is whether distillation loss is a consistently better metric when quantizing, compared to standard loss.
Revised: One key question we are interested in is whether distillation loss is a consistently better metric when quantizing, compared to standard loss.

Original: We tested this for
Revised: We tested this for

Original: CIFAR-10, comparing the performance of quantized training with respect to each loss.
Revised: CIFAR-10, comparing the performance of quantized training with respect to each loss.

Original: At 2bit precision, the student converges to 67 .
Revised: At 2bit precision, the student converges to 67 .

Original: 22% accuracy with normal loss, and to 82 .
Revised: 22% accuracy with normal loss [Table] and to 82 .

Original: 40% with distillation loss.
Revised: 40% with distillation loss.

Original: At 4bit precision, the student converges to 86 .
Revised: At 4bit precision, the student converges to 86 .

Original: 01% accuracy with normal loss, and to 88 .
Revised: 01% accuracy with normal loss

Revised: [Table]

Original:  00% with distillation loss.
Revised: and to 88 . 00% with distillation loss.

Original: On OpenNMT, we observe a similar gap: the 4bit quantized student converges to 32 .
Revised: On OpenNMT, we observe a similar gap: the 4bit quantized student converges to 32 .

Original: 67 perplexity and 15 .
Revised: 67 perplexity and 15 .

Original: 03 BLEU when trained with normal loss, and to 25 .
Revised: 03 BLEU when trained with normal loss, and to 25 .

Original: 43 perplexity (better than the teacher) and 15 .
Revised: 43 perplexity (better than the teacher) and 15 .

Original: 73 BLEU when trained with distillation loss.
Revised: 73 BLEU when trained with distillation loss.

Original: This strongly suggests that distillation loss is superior when quantizing.
Revised: This strongly suggests that distillation loss is superior when quantizing.

Original: For details, see Section A.4.1 in the Appendix.
Revised: For details, see Section A.4.1 in the Appendix.

Original: Impact of Heuristics on Differentiable Quantization.
Revised: Impact of Heuristics on Differentiable Quantization.

Original: We also performed an in-depth study of how the various heuristics impact accuracy.
Revised: We also performed an in-depth study of how the various heuristics impact accuracy.

Original: We found that, for differentiable quantization, redistributing bits according to the gradient norm of the layers is absolutely essential for good accuracy; quantiles and distillation loss also seem to provide an improvement, albeit smaller.
Revised: We found that, for differentiable quantization, redistributing bits according to the gradient norm of the layers is absolutely essential for good accuracy; quantiles and distillation loss also seem to provide an improvement, albeit smaller.

Original: Due to space constraints, we defer the results and their discussion to Section A.4.2 of the Appendix.
Revised: Due to space constraints, we defer the results and their discussion to Section A.4.2 of the Appendix.

Original: Inference Speed.
Revised: Inference Speed.

Original: In general, shallower students lead to an almost-linear decrease in inference cost, w.r.t.
Revised: In general, shallower students lead to an almost-linear decrease in inference cost, w.r.t.

Original: the depth reduction.
Revised: the depth reduction.

Original: For instance, in the CIFAR-10 experiments with the wide ResNet models, the teacher forward pass takes 67 . 4 seconds, while the student takes 43.7 seconds; roughly a 1.5x speedup, for 1.75x reduction in depth.
Revised: For instance, in the CIFAR-10 experiments with the wide ResNet models, the teacher forward pass takes 67 . 4 seconds, while the student takes 43.7 seconds; roughly a 1.5x speedup, for 1.75x reduction in depth.

Original: On the ImageNet test set using 4 GPUs (data-parallel), a forward pass takes 263 seconds for ResNet34, 169 seconds for ResNet18, and 169 seconds for our 2xResNet18.
Revised: On the ImageNet test set using 4 GPUs (data-parallel), a forward pass takes 263 seconds for ResNet34, 169 seconds for ResNet18, and 169 seconds for our 2xResNet18.

Original: (So while having more parameters than ResNet18, it has the same speed because it has the same number of layers, and is not wide enough to saturate the GPU).
Revised: (So, while having more parameters than ResNet18, it has the same speed because it has the same number of layers, and is not wide enough to saturate the GPU.

Revised: We note that we did not exploit 4bit weights, due to the lack of hardware support.)

Original: Inference on our model is 1.5 times faster, while being 1.8 times more shallow, so here the speedup is again almost linear.
Revised: Inference on our model is 1.5 times faster, while being 1.8 times shallower, so here the speedup is again almost linear.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have examined the impact of combining distillation and quantization when compressing deep neural networks.
Revised: We have examined the impact of combining distillation and quantization when compressing deep neural networks.

Original: Our main ﬁnding is that, when quantizing, one can (and should) leverage large, accurate models via distillation loss, if such models are available.
Revised: Our main ﬁnding is that, when quantizing, one can (and should) leverage large, accurate models via distillation loss, if such models are available.

Original: We have given two methods to do just that, namely quantized distillation, and differentiable quantization.
Revised: We have given two methods to do just that, namely quantized distillation, and differentiable quantization.

Original: The former acts directly on the training process of the student model, while the latter provides a way of optimizing the quantization of the student so as to best ﬁt the teacher model.
Revised: The former acts directly on the training process of the student model, while the latter provides a way of optimizing the quantization of the student so as to best ﬁt the teacher model.

Original: Our experimental results suggest that these methods can compress existing models by up to an order of magnitude in terms of size, on small image classiﬁcation and NMT tasks, while preserving accuracy.
Revised: Our experimental results suggest that these methods can compress existing models by up to an order of magnitude in terms of size, on small image classiﬁcation and NMT tasks, while preserving accuracy.

Original: At the same time, we note that distillation also provides an automatic improvement in inference speed , since it generates shallower models.
Revised: At the same time, we note that distillation also provides an automatic improvement in inference speed , since it generates shallower models.

Original: One of our more surprising ﬁndings is that naive uniform quantization with bucketing appears to perform well in a wide range of scenarios.
Revised: One of our more surprising ﬁndings is that naive uniform quantization with bucketing appears to perform well in a wide range of scenarios.

Original: Our analysis in Section 2.2 suggests that this may be because bucketing provides a way to parametrize the Gaussian-like noise induced by quantization.
Revised: Our analysis in Section 2.2 suggests that this may be because bucketing provides a way to parametrize the Gaussian-like noise induced by quantization.

Original: Given its simplicity, it could be used consistently as a baseline method.
Revised: Given its simplicity, it could be used consistently as a baseline method.

Original: In our experimental results, we performed manual architecture search for the depth and bit width of the student model, which is time-consuming and error-prone.
Revised: In our experimental results, we performed manual architecture search for the depth and bit width of the student model, which is time-consuming and error-prone.

Original: In future work, we plan to examine the potential of reinforcement learning or evolution strategies to discover the structure of the student for best performance given a set of space and latency constraints.
Revised: In future work, we plan to examine the potential of reinforcement learning or evolution strategies to discover the structure of the student for best performance given a set of space and latency constraints.

Original: The second, and more immediate direction, is to examine the practical speedup potential of these methods, and use them together and in conjunction with existing compression methods such as weight sharing Han et al.
Revised: The second, and more immediate direction, is to examine the practical speedup potential of these methods, and use them together and in conjunction with existing compression methods such as weight sharing Han et al.

Original: (2015) and with existing low-precision computation frameworks, such as NVIDIA TensorRT, or FPGA platforms
Revised: (2015) and with existing low-precision computation frameworks, such as NVIDIA TensorRT, or FPGA platforms.

Original: A F ULL EXPERIMENTAL RESULTS A. CIFAR 
Revised: A F ULL EXPERIMENTAL RESULTS A. CIFAR

Original: The model used to train CIFAR10 is the one described in Urban et al.
Revised: The model used to train CIFAR10 is the one described in Urban et al.

Original: with some minor modiﬁcations.
Revised: with some minor modiﬁcations.

Original: We use standard data augmentation techniques, including random cropping and random ﬂipping.
Revised: We use standard data augmentation techniques, including random cropping and random ﬂipping.

Original: The learning rate schedule follows the one detailed in the paper.
Revised: The learning rate schedule follows the one detailed in the paper.

Original: The structure of the models we experiment with consists of some convolutional layers, mixed with dropout layers and max pooling layers, followed by one or more linear layers.
Revised: The structure of the models we experiment with consists of some convolutional layers, mixed with dropout layers and max pooling layers, followed by one or more linear layers.

Original: The model used are deﬁned in Table 8.
Revised: The model used are deﬁned in Table 8.

Original: The c indicates a convolutional layer, mp a max pooling layer, dp a dropout layer, fc a linear (fully connected) layer.
Revised: The c indicates a convolutional layer, mp a max pooling layer, dp a dropout layer, fc a linear (fully connected) layer.

Original: The exponent indicates how many consecutive layers of the same type are there, while the number in front of the letter determines the size of the layer.
Revised: The exponent indicates how many consecutive layers of the same type are there, while the number in front of the letter determines the size of the layer.

Original: In the case of convolutional layers is the number of ﬁlters.
Revised: In the case of convolutional layers is the number of ﬁlters.

Original: All convolutional layers of the teacher are 3x3, while the convolutional layers in the smaller models are 5x5.
Revised: All convolutional layers of the teacher are 3x3, while the convolutional layers in the smaller models are 5x5.

Original: [Table]
Revised: [Table]

Original: Following the authors of the paper, we don’t use dropout layers when training the models using distillation loss.
Revised: Following the authors of the paper, we don’t use dropout layers when training the models using distillation loss.

Original: Distillation loss is computed with a temperature of T = 5 .
Revised: Distillation loss is computed with a temperature of T = 5 .

Original: Table 9 reports the accuracy of the models trained (in full precision) and their size.
Revised: Table 9 reports the accuracy of the models trained (in full precision) and their size.

Original: Table 10 reports the accuracy achieved with each method, and table 11 reports the optimal mean bit length using
Revised: Table 10 reports the accuracy achieved with each method, and table 11 reports the optimal mean bit length using

Original: Huffman encoding and resulting model size.
Revised: Huffman encoding and resulting model size.

Original: We also performed an experiment with a deeper student model.
Revised: We also performed an experiment with a deeper student model.

Original: The architecture is 76 c 3 -mp-dp126 c 3 -mp-dp-
Revised: The architecture is 76 c 3 -mp-dp126 c 3 -mp-dp-

Revised: [Table]

Revised: [Table]

Revised: mp

Revised: [Table]

Revised: 1000fc

Original: 148 c 5 -mp-dp-1000fc-dp-1000fc-dp-1000fc (following the same notation as in table 8).
Revised: [Table] 1000fc [Table] -1000fc (following the same notation as in table 8).

Original: We use the same teacher as in the previous experiments.
Revised: We use the same teacher as in the previous experiments.

Original: Results are in table 13.
Revised: Results are in table 13.

Original: A.1. CIFAR10 - W IDE R ES N ET ARCHITECTURE
Revised: A.1. CIFAR10 - W IDE R ES N ET ARCHITECTURE

Original: For our second set of experiments on CIFAR10 with the WideResNet architecture, see table 15.
Revised: For our second set of experiments on CIFAR10 with the WideResNet architecture, see table 15.

Original: Note that we increase the number of ﬁlters but reduce the depth of the model.
Revised: Note that we increase the number of ﬁlters but reduce the depth of the model.

Original: The implementation 
Revised: The implementation of WideResNet used can be found on GitHub 1 . Results of quantized methods are in table 16 while the size of the resulting models is detailed in table 17.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 2 bits 4 bits 8 bits
Revised: 2 bits 4 bits 8 bits

Original: PM Quant.
Revised: PM Quant.

Original: 1 (No bucket) 1.34 bits - 0.17 MB 2.43 bits - 0.3 MB 6.48 bits - 0.81 MB
Revised: 1 (No bucket) 1.34 bits - 0.17 MB 2.43 bits - 0.3 MB 6.48 bits - 0.81 MB

Original: PM Quant. 1 (with bucket) 1.58 bits - 0.22 MB 3.52 - 0.47 MB 7.58 bits - 0.98 MB
Revised: PM Quant. 1 (with bucket) 1.58 bits - 0.22 MB 3.52 - 0.47 MB 7.58 bits - 0.98 MB

Original: Quantized Distill.1.7 bits - 0.24 MB 3.64 bits - 0.48 MB 7.70 bits - 1 MB
Revised: Quantized Distill.1.7 bits - 0.24 MB 3.64 bits - 0.48 MB 7.70 bits - 1 MB

Original: Differentiable Quant.3.18 bits - 0.43 MB 5.34 bits - 0.7 MB ——
Revised: Differentiable Quant.3.18 bits - 0.43 MB 5.34 bits - 0.7 MB ——

Original: PM Quant. 2 (No bucket) 1.43 bits - 0.05 MB 2.6 bits - 0.1 MB 6.65 bits - 0.26 MB
Revised: PM Quant. 2 (No bucket) 1.43 bits - 0.05 MB 2.6 bits - 0.1 MB 6.65 bits - 0.26 MB

Original: PM Quant. 2 (with bucket) 1.6 bits - 0.07 MB 3.58 bits - 0.15 MB 7.64 bits - 0.31 MB
Revised: PM Quant. 2 (with bucket) 1.6 bits - 0.07 MB 3.58 bits - 0.15 MB 7.64 bits - 0.31 MB

Original: Quantized Distill.1.7 bits - 0.08 MB 3.55 bits - 0.15 MB 7.64 bits - 0.31 MB
Revised: Quantized Distill.1.7 bits - 0.08 MB 3.55 bits - 0.15 MB 7.64 bits - 0.31 MB

Original: Differentiable Quant.3.16 bits - 0.13 MB 5.34 bits - 0.22 MB ——
Revised: Differentiable Quant.3.16 bits - 0.13 MB 5.34 bits - 0.22 MB ——

Original: PM Quant.
Revised: PM Quant.

Original: 3 (No bucket) 1.46 bits - 0.02 MB 2.62 bits - 0.03 MB 6.66 bits - 0.09 MB
Revised: 3 (No bucket) 1.46 bits - 0.02 MB 2.62 bits - 0.03 MB 6.66 bits - 0.09 MB

Original: PM Quant. 3 (with bucket) 1.58 bits - 0.026 MB 3.51 bits - 0.053 MB 7.56 bits - 0.1 MBQuantized Distill.1.64 bits - 0.027 MB 3.53 bits - 0.053 MB 7.59 bits - 0.11 MB Differentiable Quant.3.12 bits - 0.04 MB 5.41 bits - 0.08 MB —— of WideResNet used can be found on GitHub 2 . Results of quantized methods are in table 16 while the size of the resulting models is detailed in table 17.
Revised: PM Quant. 3 (with bucket) 1.58 bits - 0.026 MB 3.51 bits - 0.053 MB 7.56 bits - 0.1 MB

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: Quantized Distill.1.64 bits - 0.027 MB 3.53 bits - 0.053 MB 7.59 bits - 0.11 MB

Original: [Table] 4 bits PM Quant. (with bucket) 1.44 bits - 17.56 MB 2.62 bits - 29.75 MB Quantized Distill. 1.54 bits - 17.81 MB 3.48 bits - 38.65 MB 
Revised: Differentiable Quant.3.12 bits - 0.04 MB 5.41 bits - 0.08 MB ——

Original: A. CIFAR 
Revised: A. CIFAR

Original: For our CIFAR100 experiments, we use the same implementation of wide residual networks as in our CIFAR10 experiments.
Revised: For our CIFAR100 experiments, we use the same implementation of wide residual networks as in our CIFAR10 experiments.

Original: The wide factor is a multiplicative factor controlling the amount of ﬁlters in each layer; for more details please refer to the original paper Zagoruyko & Komodakis (2016).
Revised: The wide factor is a multiplicative factor controlling the amount of ﬁlters in each layer; for more details please refer to the original paper Zagoruyko & Komodakis (2016).

Original: We train for 200 epochs with an initial learning rate of 0.1. 
Revised: We train for 200 epochs with an initial learning rate of 0.1. [Table] [Table] [Table]

Original:  For the CIFAR100 experiments we focused on one student model.
Revised: [Table] [Table] [Table] For the CIFAR100 experiments we focused on one student model.

Original: Distillation loss is computed with a temperature of T =5 .
Revised: Distillation loss is computed with a temperature of T =

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A. OPENT NMT INTEGRATION TEST DATASET
Revised: A. OPENT NMT INTEGRATION TEST DATASET

Original: As mentioned in the main text
Revised: As mentioned in the main text

Original: [Table]
Revised: [Table]

Original: we use the openNMT-py codebase.
Revised: we use the openNMT-py codebase.

Original: We slightly modify it to add distillation loss and the quantization methods proposed.
Revised: We slightly modify it to add distillation loss and the quantization methods proposed.

Original: We mostly use standard options to train the model; in particular, the learning rate starts at 1 and is halved every epoch starting from the ﬁrst epoch where perplexity doesn’t drop on the test set.
Revised: We mostly use standard options to train the model; in particular, the learning rate starts at 1 and is halved every epoch starting from the ﬁrst epoch where perplexity doesn’t drop on the test set.

Original: We train every model for 15 epochs.
Revised: We train every model for 15 epochs.

Original: Distillation loss is computed with a temperature of T = 1 .
Revised: Distillation loss is computed with a temperature of T = 1 .

Original: [Table]
Revised: [Table]

Revised: millions 326.57 MB

Revised: Smaller model2 LSTM layer, 256 cell size 34.

Revised: 64.8 millions 249.56 MB

Revised: Distilled model2 LSTM layer, 256 cell size 27.

Revised: 64.8 millions 249.56 MB

Revised: Smaller model2 LSTM layer, 128 cell size 39.

Revised: 57.2 millions 228.85 MB

Revised: Distilled model2 LSTM layer, 128 cell size 33.

Revised: 57.2 millions 228.85 MB

Original: A. WMT13 DATASET
Revised: A. WMT13 DATASET

Original: For the WMT13 datasets [Table] we run a similar architecture.
Revised: For the WMT13 datasets, we run a similar architecture.

Original: We ran all models for 15 epochs; the smaller model overﬁt with 15 epochs [Table] so we ran it for 5 epochs instead.
Revised: We ran all models for 15 epochs; the smaller model overﬁt with 15 epochs, so we ran it for 5 epochs instead.

Original: A.4. D ISTILLATION VERSUS S TANDARD L OSS FOR Q UANTIZATION
Revised: A.4. D ISTILLATION VERSUS S TANDARD L OSS FOR Q UANTIZATION

Original: In this section we highlight the positive effects of using distillation loss during quantization.
Revised: In this section we highlight the positive effects of using distillation loss during quantization.

Original: We take models with the same architecture and we train them with the same number of bits; one of the

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original:  with bucket) 7.34 ppl - 26.18 BLEU Quantized Distill. 6.48 ppl - 35.32 BLEU models is trained with normal loss, the other with the distillation loss with equal weighting between soft cross entropy and normal cross entropy (that is, it is the quantized distilled model).
Revised: We take models with the same architecture and we train them with the same number of bits; one of the models is trained with normal loss, the other with the distillation loss with equal weighting between soft cross entropy and normal cross entropy (that is, it is the quantized distilled model).

Original: Table 27 shows the results on the CIFAR10 dataset; the models we train have the same structure as the Smaller model 1, see Section A.1.
Revised: Table 27 shows the results on the CIFAR10 dataset; the models we train have the same structure as the Smaller model 1, see Section A.1.

Original: [Table]

Original: Table 28 shows the results on the openNMT integration test dataset; the models trained have the same structure of Smaller model 1, see Section A.3.
Revised: Table 28 shows the results on the openNMT integration test dataset; the models trained have the same structure of Smaller model 1, see Section A.3.

Original: Notice that distillation loss can signiﬁcantly improve the accuracy of the quantized models.
Revised: Notice that distillation loss can signiﬁcantly improve the accuracy of the quantized models.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  These results suggest that quantization works better when combined with distillation
Revised: with bucket) 7.34 ppl - 26.18 BLEU Quantized Distill. 6.48 ppl - 35.32 BLEU These results suggest that quantization works better when combined with distillation, and that we should try to take advantage of this whenever we are quantizing a neural network.

Original: [Table]
Revised: [Table]

Original: and that we should try to take advantage of this whenever we are quantizing a neural network.
Revised: [Table] [Table]

Original: A.4. D IFFERENT HEURISTICS FOR DIFFERENTIABLE QUANTIZATION
Revised: A.4. D IFFERENT HEURISTICS FOR DIFFERENTIABLE QUANTIZATION

Original: To test the different heuristics presented in Section 4.
Revised: To test the different heuristics presented in Section 4.

Original: [Table]
Revised: [Table]

Original: we train with differentiable quantization the
Revised: we train with differentiable quantization the

Original: Smaller model 1 architecture speciﬁed in Section A.1 on the cifar10 dataset.
Revised: Smaller model 1 architecture speciﬁed in Section A [Table] 1 on the cifar10 dataset.

Original: The same model is trained with different heuristics to provide a sense of how important they are; the experiments is performed with 2 and 4 bits.
Revised: The same model is trained with different heuristics to provide a sense of how important they are; the experiments is performed with 2 and 4 bits.

Original: Results suggests that when using 4 bits, the method is robust and works regardless.
Revised: Results suggests that when using 4 bits, the method is robust and works regardless.

Original: When usingbits, redistributing bits according to the gradient norm of the layers is absolutely essential for this method to work ; quantiles starting point also seem to provide an small improvement
Revised: When usingbits, redistributing bits according to the gradient norm of the layers is absolutely essential for this method to work ; quantiles starting point also seem to provide an small improvement

Original: [Table]
Revised: [Table]

Original: while using distillation loss in this case does not seem to be crucial.
Revised: while using distillation loss in this case does not seem to be crucial.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B Q UANTIZATION IS E QUIVALENT TO A SYMPTOTICALLY N ORMALLY D ISTRIBUTED N OISE
Revised: B Q UANTIZATION IS E QUIVALENT TO A SYMPTOTICALLY N ORMALLY D ISTRIBUTED N OISE

Original: In this section we will prove some results about the uniform quantization function, including the fact that is asymptotically normally distributed, see subsection B.1 below.
Revised: In this section we will prove some results about the uniform quantization function, including the fact that is asymptotically normally distributed, see subsection B.1 below.

Original: Clearly, we refer to the stochastic version, see Section 2.1.
Revised: Clearly, we refer to the stochastic version, see Section 2.1.

Original: Unbiasedness
Revised: Unbiasedness

Original: We ﬁrst start proving the unbiasedness of ˆ Q ;
Revised: We ﬁrst start proving the unbiasedness of ˆ Q ;

Original: [Equation]
Revised: [Equation]

Original: Then it is immediate that
Revised: Then it is immediate that

Original: [Equation]
Revised: [Equation]

Original: Bounds on second and third moment
Revised: Bounds on second and third moment

Original: We will write out bounds on ˆ Q ; the analogous bounds on Q are then straightforward.
Revised: We will write out bounds on ˆ Q ; the analogous bounds on Q are then straightforward.

Original: For convenience
Revised: For convenience, let us call ˆ l

Original: [Equation]
Revised: [Equation]

Original: let us call ˆ l i
Revised: =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: And given that ˆ l i ≤ ˆ v i s ≤ ˆ l i + 1 , we readily ﬁnd
Revised: And given that ˆ l i ≤ ˆ v i s ≤ ˆ l i + 1 , we readily ﬁnd

Original: [Equation]
Revised: [Equation]

Original: For the third moment [Equation] we have
Revised: For the third moment, we have

Original: [Equation]
Revised: [Equation]

Original: And as before [Equation] the bounds are
Revised: And as before, the bounds are

Original: [Equation]
Revised: [Equation]

Original: B. A SYMPTOTIC NORMALITY
Revised: B. A SYMPTOTIC NORMALITY

Original: Most of neural networks operations are scalar product computation.
Revised: Most of neural networks operations are scalar product computation.

Original: Therefore, the scalar product of the quantized weights and the inputs is an important quantity:
Revised: Therefore, the scalar product of the quantized weights and the inputs is an important quantity:

Original: [Equation]
Revised: [Equation]

Original: We already know from section B that the quantization function is unbiased; hence we know that
Revised: We already know from section B that the quantization function is unbiased; hence we know that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: with ε n is a zero-mean random variable.
Revised: with ε n is a zero-mean random variable.

Original: We will show that ε n tends in distribution to a normal random variable.
Revised: We will show that ε n tends in distribution to a normal random variable.

Original: To prove asymptotic normality, we will use a generalized version of the central limit theorem due to Lyapunov:
Revised: To prove asymptotic normality, we will use a generalized version of the central limit theorem due to Lyapunov:

Original: Theorem B.1 (Lyapunov Central Limit Theorem) .
Revised: Theorem B.1 (Lyapunov Central Limit Theorem) .

Original: Let { X 1 , X 2 , . . .
Revised: Let { X 1 , X 2 , . . .

Original: } be a sequence of independent random variables, each with ﬁnite expected value µ i and variance σ 2 i .
Revised: } be a sequence of independent random variables, each with ﬁnite expected value µ i and variance σ 2 i .

Original: Deﬁne s 2 n = (cid:80) ni
Revised: Deﬁne s 2 n = (cid:80) ni

Original: =1 σ 2 i .
Revised: =1 σ 2 i .

Original: If, for some δ > 0 , the Lyapunov condition
Revised: If, for some δ > 0 , the Lyapunov condition

Original: [Equation]
Revised: [Equation]

Original: is satisﬁed, then
Revised: is satisﬁed, then

Original: [Equation]
Revised: [Equation]

Original: We can now state the theorem:
Revised: We can now state the theorem:

Original: Theorem B.2.
Revised: Theorem B.2.

Original: Let v, x be two vectors with n elements.
Revised: Let v, x be two vectors with n elements.

Original: Let Q be the uniform quantization function with s levels deﬁned in 2.1 and deﬁne s 2 n = (cid:80) ni
Revised: Let Q be the uniform quantization function with s levels deﬁned in 2.1 and deﬁne s 2 n = (cid:80) ni

Original: =1 V ar [ Q ( v i ) x i ] .
Revised: =1 V ar [ Q ( v i ) x i ] .

Original: If the elements of v, x are uniformly bounded by M 3 and lim n →∞ s n = ∞ , then
Revised: If the elements of v, x are uniformly bounded by M 2 and lim n →∞ s n = ∞ , then

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: with E [ ε n ] = 0 and
Revised: with E [ ε n ] = 0 and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using the same notation as theorem B [Equation] [Equation] let X i 
Revised: Using the same notation as theorem B.1, let X i =

Original: = Q ( v i ) x i , µ i = E
Revised: Q ( v i ) x i , µ i = E

Original: [ X i ] =
Revised: [ X i ] =

Original: v i x i .
Revised: v i x i .

Original: We already mentioned in 2.1 that these are independent random variables.
Revised: We already mentioned in 2.1 that these are independent random variables.

Original: We will show that the Lyapunov condition holds with δ = [Equation]
Revised: We will show that the Lyapunov condition holds with δ = 1 .

Original: We know that
Revised: We know that

Original: [Equation]
Revised: [Equation]

Original: In fact
Revised: In fact,

Original: [Equation]
Revised: [Equation]

Original:  since during quantization we have bins of size 1 s , so that is the largest error we can make.
Revised: 26) since during quantization we have bins of size 1 s , so that is the largest error we can make.

Original: Also, by hypothesis M ≥ α i , x i for every i .
Revised: Also, by hypothesis M ≥ α i , x i for every i .

Original: Hence
Revised: Hence

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: 2 i.e. there exists a constant M such that for all n , | v i |

Revised: M , | x i | ≤ M for all i ∈ { 1 , . .

Original:  and since lim n →∞ s n = ∞ , we have that the Lyapunov condition is satisﬁed.
Revised: , n } and since lim n →∞ s n = ∞ , we have that the Lyapunov condition is satisﬁed.

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Note about the hypothesis The two hypothesis that were used to prove the theorem are reasonable and should be satisﬁed by any practical dataset.
Revised: Note about the hypothesis The two hypothesis that were used to prove the theorem are reasonable and should be satisﬁed by any practical dataset.

Original: Typically we know or we can estimate the range of the values of inputs and weights, so the assumption that they don’t get arbitrarily large with n is satisﬁed.
Revised: Typically we know or we can estimate the range of the values of inputs and weights, so the assumption that they don’t get arbitrarily large with n is satisﬁed.

Original: The assumption on the variance is also reasonable; in fact, s 2 n = (cid:80) n i =1 V ar [ Q ( v i ) x
Revised: The assumption on the variance is also reasonable; in fact, s 2 n = (cid:80) ni =1 V ar [ Q ( v i ) x

Original: i ] consists of a sum of n values.
Revised: i ] consists of a sum of n values.

Original: While it is possible for all these values to be 0 (if all v i are in the form k/s , for example, then s 2 n = 0 )
Revised: While it is possible for all these values to be 0 (if all v i are in the form k/s , for example, then s 2 n = 0 )

Original: it is unlikely that a real world dataset would present this characteristic.
Revised: it is unlikely that a real world dataset would present this characteristic.

Original: In fact, it sufﬁces that there exist γ > 0 and 0 < δ ≤ 1 such that at least δ -percent of σ 2 i ≥ γ .
Revised: In fact, it sufﬁces that there exist γ > 0 and 0 < δ ≤ 1 such that at least δ -percent of σ 2 i ≥ γ .

Original: This implies s 2 n ≥ αγn → ∞ .
Revised: This implies s 2 n ≥ αγn → ∞ .