Original: L EARNING D ISCRETE W EIGHTS U SING THE L OCAL R EPARAMETERIZATION T RICK
Revised: L EARNING D ISCRETE W EIGHTS U SING THE L OCAL R EPARAMETERIZATION T RICK

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs.
Revised: Recent breakthroughs in computer vision make use of large deep neural networks, utilizing the substantial speedup offered by GPUs.

Original: For applications running on limited hardware, however, high precision real-time processing can still be a challenge.
Revised: For applications running on limited hardware, however, high precision real-time processing can still be a challenge.

Original: One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size.
Revised: One approach to solving this problem is training networks with binary or ternary weights, thus removing the need to calculate multiplications and significantly reducing memory size.

Original: In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters.
Revised: In this work, we introduce LR-nets (Local reparameterization networks), a new method for training neural networks with discrete weights using stochastic parameters.

Original: We show how a simple modiﬁcation to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights.
Revised: We show how a simple modiﬁcation to the local reparameterization trick, previously used to train Gaussian distributed weights, enables the training of discrete weights.

Original: Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.
Revised: Using the proposed training we test both binary and ternary models on MNIST, CIFAR-10 and ImageNet benchmarks and reach state-of-the-art results on most experiments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Neural Networks have been the main driving force behind recent advancement in machine learning, notably in computer vision applications.
Revised: Deep Neural Networks have been the main driving force behind recent advancement in machine learning, notably in computer vision applications.

Original: While deep learning has become the standard approach for many tasks, performing inference on low power and constrained memory hardware is still challenging.
Revised: While deep learning has become the standard approach for many tasks, performing inference on low power and constrained memory hardware is still challenging.

Original: This is especially challenging in autonomous driving in electric vehicles where high precision and high throughput constraints are added on top of the low power requirements.
Revised: This is especially challenging in autonomous driving in electric vehicles where high precision and high throughput constraints are added on top of the low power requirements.

Original: One approach for tackling this challenge is by training networks with binary {± 1 } or ternary {− 1 , 0 , 1 } weights (Courbariaux et al., 2015; Rastegari et al., 2016) that require an order of magnitude less memory and no multiplications, leading to signiﬁcantly faster inference on dedicated hardware.
Revised: One approach for tackling this challenge is by training networks with binary {± 1 } or ternary {− 1 , 0 , 1 } weights (Courbariaux et al., 2015; Rastegari et al., 2016) that require an order of magnitude less memory and no multiplications, leading to signiﬁcantly faster inference on dedicated hardware.

Original: The problem arises when trying to backpropagate errors as the weights are discrete.
Revised: The problem arises when trying to backpropagate errors as the weights are discrete.

Original: One heuristic suggested in Courbariaux et al.
Revised: One heuristic suggested in Courbariaux et al.

Original: (2015) is to use stochastic weights w , sample binary weights w b according to w for the forward pass and gradient computation and then update the stochastic weights w instead.
Revised: (2015) is to use stochastic weights w , sample binary weights w b according to w for the forward pass and gradient computation and then update the stochastic weights w instead.

Original: Another idea, used by Hubara et al. (2016) and Rastegari et al.
Revised: Another idea, used by Hubara et al. (2016) and Rastegari et al.

Original: (2016), is to apply a “straight-through” estimator ∂ sign ∂r =
Revised: (2016), is to apply a “straight-through” estimator ∂ sign ∂r =

Original: r 1 [ | r | ≤ 1] .
Revised: r 1 [ | r | ≤ 1] .

Original: While these ideas were able to produce good results, even on reasonably large networks such as ResNet-18 (He et al., 2016), there is still a large gap in prediction accuracy between the full-precision network and the discrete networks.
Revised: While these ideas were able to produce good results, even on reasonably large networks such as ResNet-18 (He et al., 2016), there is still a large gap in prediction accuracy between the full-precision network and the discrete networks.

Original: In this paper, we attempt to train neural networks with discrete weights using a more principled approach.
Revised: In this paper, we attempt to train neural networks with discrete weights using a more principled approach.

Original: Instead of trying to ﬁnd a good “derivative” to a non-continuous function, we show how we can ﬁnd a good smooth approximation and use its derivative to train the network.
Revised: Instead of trying to ﬁnd a good “derivative” to a non-continuous function, we show how we can ﬁnd a good smooth approximation and use its derivative to train the network.

Original: This is based on the simple observation that if at layer l we have stochastic (independent) weights w lij , then the pre-activations z li = (cid:80) j w lij h lj are approximately Gaussian according to the (Lyapunov) central limit theorem (CLT).
Revised: This is based on the simple observation that if at layer l we have stochastic (independent) weights w lij , then the pre-activations z li = (cid:80) j w lij h lj are approximately Gaussian according to the (Lyapunov) central limit theorem (CLT).

Original: This allows us to model the pre-activation using a smooth distribution and use the reparameterization trick (Kingma & Welling, 2014) to compute derivatives.
Revised: This allows us to model the pre-activation using a smooth distribution and use the reparameterization trick (Kingma & Welling, 2014) to compute derivatives.

Original: The idea of modeling the distribution of pre-activation, instead of the distribution of weights, was used in Kingma et al.
Revised: The idea of mod- eling the distribution of pre-activation, instead of the distribution of weights, was used in Kingma et al.

Original: (2015) for Gaussian weight distributions where it was called the local reparameterization trick.
Revised: (2015) for Gaussian weight distributions where it was called the local reparameterization trick.

Original: We show here that with small modiﬁcations it can be used to train discrete weights and not just continuous Gaussian distributions.
Revised: We show here that with small modiﬁcations it can be used to train discrete weights and not just continuous Gaussian distributions.

Revised: [Figure]

Original: We experimented with both binary and ternary weights, and while the results on some datasets are quite similar, ternary weights were considerably easier to train.
Revised: We experimented with both binary and ternary weights, and while the results on some datasets are quite similar, ternary weights were considerably easier to train.

Original: From a modeling perspective[Figure] restricting weights to binary values {± 1 } forces each neuron to affect all neurons in the subsequent layer making it hard to learn representations that need to capture several independent features.
Revised: From a modeling perspective restricting weights to binary values {± 1 } forces each neuron to affect all neurons in the subsequent layer making it hard to learn representations that need to capture several independent features.

Original: In this work, we present a novel and simple method for training neural networks with discrete weights.
Revised: In this work, we present a novel and simple method for training neural networks with discrete weights.

Original: We show experimentally that we can train binary and ternary networks to achieve stateof-the-art results on several datasets, including ResNet-18 on ImageNet, compared to previously proposed binary or ternary training algorithms.
Revised: We show experimentally that we can train binary and ternary networks to achieve stateof-the-art results on several datasets, including ResNet-18 on ImageNet, compared to previously proposed binary or ternary training algorithms.

Original: On MNIST and CIFAR-10 we can also almost match the performance of the original full-precision network using discrete weights.
Revised: On MNIST and CIFAR-10 we can also almost match the performance of the original full-precision network using discrete weights.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The closest work to ours, conceptually, is expectation-backpropagation (Soudry et al., 2014) in which the authors use the CLT approximation to train a Bayesian mean-ﬁeld posterior.
Revised: The closest work to ours, conceptually, is expectation-backpropagation (Soudry et al., 2014) in which the authors use the CLT approximation to train a Bayesian mean-ﬁeld posterior.

Original: On a practical level, however, our training algorithm is entirely different: their forward pass is deterministic while ours is stochastic.
Revised: On a practical level, however, our training algorithm is entirely different: their forward pass is deterministic while ours is stochastic.

Original: In addition, we show results on much larger networks.
Revised: In addition, we show results on much larger networks.

Original: Another close line of work is Wang & Manning (2013) where the authors used the CLT to approximate the dropout noise and speed-up training, but do not try to learn discrete weights.
Revised: Another close line of work is Wang & Manning (2013) where the authors used the CLT to approximate the dropout noise and speed-up training, but do not try to learn discrete weights.

Original: Recently there is great interest in optimizing discrete distributions by using continuous approximations such as the Gumbel-softmax (Maddison et al., 2017; Jang et al., 2017), combined with the reparameterization trick.
Revised: Recently there is great interest in optimizing discrete distributions by using continuous approximations such as the Gumbel-softmax (Maddison et al., 2017; Jang et al., 2017), combined with the reparameterization trick.

Original: However, since we are looking at the pre-activation distribution, we use the simpler Gaussian approximation as can be seen in Fig 1.
Revised: However, since we are looking at the pre-activation distribution, we use the simpler Gaussian approximation as can be seen in Fig 1.

Original: In appendix A we compare our method to the Gumbel-softmax relaxation and show that our method leads to much better training.
Revised: In appendix A we compare our method to the Gumbel-softmax relaxation and show that our method leads to much better training.

Original: One approach for discrete weight training suggested in Courbariaux et al.
Revised: One approach for discrete weight training suggested in Courbariaux et al.

Original: (2015) is to use stochastic weights, sample binary weights and use it for the forward pass and gradient computation, computing gradients as if it was a deterministic full precision network, and then update the stochastic weights.
Revised: (2015) is to use stochastic weights, sample binary weights and use it for the forward pass and gradient computation, computing gradients as if it was a deterministic full precision network, and then update the stochastic weights.

Original: The current leading works on training binary or ternary networks, Rastegari et al.
Revised: The current leading works on training binary or ternary networks, Rastegari et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Hubara et al.
Revised: ; Hubara et al.

Original: (2016), are based on a different approach.
Revised: (2016), are based on a different approach.

Original: They discretize during the forward pass, and back-propagate through this non-continuous discretization using the “straight-through” estimator for the gradient.
Revised: They discretize during the forward pass, and back-propagate through this non-continuous discretization using the “straight-through” estimator for the gradient.

Original: We note that Rastegari et al.
Revised: We note that Rastegari et al.

Original: ; Hubara et al.
Revised: ; Hubara et al.

Original: ; Soudry et al.
Revised: ; Soudry et al.

Original: (2014) binarize the pre-activation as well.
Revised: (2014) binarize the pre-activation as well.

Original: Another similar work based on the “straight-through” estimator is Zhu et al.
Revised: Another similar work based on the “straight-through” estimator is Zhu et al.

Original: The authors proposed training a ternary quantization network, where the discrete weights are not ﬁxed to {− 1 , 0 , 1 } but are learned.
Revised: The authors proposed training a ternary quantization network, where the discrete weights are not ﬁxed to {− 1 , 0 , 1 } but are learned.

Original: This method achieves very good performance, and can save memory compared to a full precision network.
Revised: This method achieves very good performance, and can save memory compared to a full precision network.

Original: However, the use of learned discrete weights requires using multiplications which adds computational overhead on dedicated hardware compared to binary or ternary networks.
Revised: However, the use of learned discrete weights requires using multiplications which adds computational overhead on dedicated hardware compared to binary or ternary networks.

Original: An alternative approach to reducing memory requirements and speeding up deep networks utilized weight compression.
Revised: An alternative approach to reducing memory requirements and speeding up deep networks utilized weight compression.

Original: Louizos et al.
Revised: Louizos et al.

Original: ; Han et al.
Revised: ; Han et al.

Original: (2016) showed that deep networks can be highly compressed without signiﬁcant loss of precision.
Revised: (2016) showed that deep networks can be highly compressed without signiﬁcant loss of precision.

Original: However, the speedup capabilities of binary or ternary weight networks, especially on dedicated hardware, is much higher.
Revised: However, the speedup capabilities of binary or ternary weight networks, especially on dedicated hardware, is much higher.

Original:  O UR METHOD
Revised:  O UR METHOD

Original: We will now describe in detail our algorithm, starting with the necessary background.
Revised: We will now describe in detail our algorithm, starting with the necessary background.

Original: We use a stochastic network model in which each weight w l ij is sampled independently from a multinomial distribution W lij .
Revised: We use a stochastic network model in which each weight w lij is sampled independently from a multinomial distribution W lij .

Original: We use W to mark the set of all parameters w lij and W to mark the distribution over W .
Revised: We use W to mark the set of all parameters w lij and W to mark the distribution over W .

Original: Given a loss function (cid:96) and a neural network f parametrized by W our goal is to minimize
Revised: Given a loss function (cid:96) and a neural network f parametrized by W our goal is to minimize

Original: L 
Revised: L ( W ) = E W ∼W (cid:34) N

Original: [Equation]
Revised: [Equation]

Original: 3. B ACKGROUND
Revised: 3. B ACKGROUND

Original: The standard approach for minimizing L ( W ) with discrete distributions is by using the logderivative trick (Williams, 1992):
Revised: The standard approach for minimizing L ( W ) with discrete distributions is by using the logderivative trick (Williams, 1992):

Original: [Equation]
Revised: [Equation]

Original: While this allows us to get an unbiased estimation of the gradient, it suffers from high variance, which limits the effectiveness of this method.
Revised: While this allows us to get an unbiased estimation of the gradient, it suffers from high variance, which limits the effectiveness of this method.

Original: For continuous distributions Kingma & Welling (2014) suggested the reparameterization trickinstead of optimizing E p ( x )
Revised: For continuous distributions Kingma & Welling (2014) suggested the reparameterization trickinstead of optimizing E p ( x )

Original: [ f ( x )] for p ( x ) we parametrize x = g
Revised: [ f ( x )] for p ( x ) we parametrize x = g

Original: ( (cid:15) ; θ ) where (cid:15) is drawn from a known ﬁxed distribution p ( (cid:15) )
Revised: ( (cid:15) ; θ ) where (cid:15) is drawn from a known ﬁxed distribution p ( (cid:15) )

Original: (usually Gaussian) and optimize E p ( (cid:15) )
Revised: (usually Gaussian) and optimize E p ( (cid:15) )

Original: [ f ( g ( (cid:15), θ ))] for θ .
Revised: [ f ( g ( (cid:15), θ ))] for θ .

Original: We can sample (cid:15) 1 , ..., (cid:15) m and use the Monte-Carlo approximation:
Revised: We can sample (cid:15) 1 , ..., (cid:15) m and use the Monte-Carlo approximation:

Original: [Equation]
Revised: [Equation]

Original: In further work by Kingma et al.
Revised: In further work by Kingma et al.

Original: (2015), it was noticed that if we are trying to learn Bayesian networks, sampling weights and running the model with different weights is quite inefﬁcient on
Revised: (2015), it was noticed that if we are trying to learn Bayesian networks, sampling weights and running the model with different weights is quite inefﬁcient on

Original: GPUs.
Revised: GPUs.

Original: They observed that if the weight matrix 1 W is sampled from independent Gaussians W ij ∼
Revised: They observed that if the weight matrix 1 W is sampled from independent Gaussians W ij ∼

Original: N ( µ ij , σ 2 ij ) , then the pre-activations z =
Revised: N ( µ ij , σ 2 ij ) , then the pre-activations z =

Original: Wh are distributed according to
Revised: Wh are distributed according to

Original: [Equation]
Revised: [Equation]

Original: i [Equation] This allows us to sample pre-activations instead of weights, which they called the local reparameterization trick, reducing run time considerably.
Revised: This allows us to sample pre-activations instead of weights, which they called the local reparameterization trick, reducing run time considerably.

Original: 3. D ISCRETE LOCAL REPARAMETERIZATION
Revised: 3. D ISCRETE LOCAL REPARAMETERIZATION

Original: The work by Kingma et al.
Revised: The work by Kingma et al.

Original: (2015) focused on Gaussian weight distributions, but we will show how the local reparameterization allows us to optimize networks with discrete weights as well.
Revised: (2015) focused on Gaussian weight distributions, but we will show how the local reparameterization allows us to optimize networks with discrete weights as well.

Original: Our main observation is that while eq. 4 holds true for Gaussians, from the (Lyapunov) central limit theorem we get that
Revised: Our main observation is that while eq. 4 holds true for Gaussians, from the (Lyapunov) central limit theorem we get that

Original: z i = (cid:80)
Revised: z i = (cid:80)

Original: j w ij h
Revised: j w ij h

Original: j should still be approximated well by the same Gaussian distribution
Revised: j should still be approximated well by the same Gaussian distribution

Original: z i ∼ N ( (cid:80) j µ ij h j , (cid:80)
Revised: z i ∼ N ( (cid:80) j µ ij h j , (cid:80)

Original: j σ 2 ij h 2 j ) .
Revised: j σ 2 ij h 2 j ) .

Original: The main difference is that µ ij and σ 2 ij are the mean and variance of a multinomial distribution, not a Gaussian one.
Revised: The main difference is that µ ij and σ 2 ij are the mean and variance of a multinomial distribution, not a Gaussian one.

Original: Once the discrete distribution has been approximated by a smooth one, we can compute the gradient of this smooth approximation and update accordingly.
Revised: Once the discrete distribution has been approximated by a smooth one, we can compute the gradient of this smooth approximation and update accordingly.

Original: This leads to a simple algorithm for training networks with discrete weights - let θ ij be the parameters of the multinomial distribution over w ij .
Revised: This leads to a simple algorithm for training networks with discrete weights - let θ ij be the parameters of the multinomial distribution over w ij .

Original: At the forward pass, given input vector h , we ﬁrst compute the weights means µ ij =
Revised: At the forward pass, given input vector h , we ﬁrst compute the weights means µ ij =

Original: E θ ij [ w ij ] and variances σ 2 ij =
Revised: E θ ij [ w ij ] and variances σ 2 ij =

Original: Var θ ij [ w ij ] .
Revised: Var θ ij [ w ij ] .

Original: We then compute the mean and variance of z i , m i
Revised: We then compute the mean and variance of z i , m i

Original: =
Revised: =

Original: E [ z i ] =
Revised: E [ z i ] =

Original: (cid:80) j µ ij h j and v 2 i =
Revised: (cid:80) j µ ij h j and v 2 i =

Original: Var [ z i ] =
Revised: Var [ z i ] =

Original: (cid:80)
Revised: (cid:80)

Original: j σ 2 ij h 2 j .
Revised: j σ 2 ij h 2 j .

Original: Finally we sample (cid:15) ∼ N (0 , I ) and return ˆ z = m + v (cid:12) (cid:15) , where (cid:12) stands for the element-wise multiplication.
Revised: Finally we sample (cid:15) ∼ N (0 , I ) and return ˆ z = m + v (cid:12) (cid:15) , where (cid:12) stands for the element-wise multiplication.

Original: We summarize this in Algorithm 1.
Revised: We summarize this in Algorithm 1.

Original: During the backwards phase, given ∂L∂ ˆ y we can compute
Revised: During the backwards phase, given ∂L∂ ˆ y we can compute

Original: [Equation]
Revised: [Equation]

Original: and similarly compute ∂L∂x i to backpropagate.
Revised: and similarly compute ∂L∂x i to backpropagate.

Original: We can then optimize using any ﬁrst order optimization method.
Revised: We can then optimize using any ﬁrst order optimization method.

Original: In our experiments we used Adam (Kingma & Ba, 2015).
Revised: In our experiments we used Adam (Kingma & Ba, 2015).

Original:  I MPLEMENTATION D ETAILS
Revised:  I MPLEMENTATION D ETAILS

Original: In section 3 we presented our main algorithmic approach.
Revised: In section 3 we presented our main algorithmic approach.

Original: In this section, we discuss the ﬁner implementation details needed to achieve state-of-the-art performance.
Revised: In this section, we discuss the ﬁner implementation details needed to achieve state-of-the-art performance.

Original: We present the numerical experiments in section 5.
Revised: We present the numerical experiments in section 5.

Original: For convenience, we represent our ternary weights using two parameters, a lij and b lij , such that p ( w lij = 0)
Revised: For convenience, we represent our ternary weights using two parameters, a lij and b lij , such that p ( w lij = 0)

Original: =
Revised: =

Original: σ ( a lij ) and p ( w lij = 1 | w lij (cid:54) = 0)
Revised: σ ( a lij ) and p ( w lij = 1 | w lij (cid:54) = 0)

Original: =
Revised: =

Original: σ ( b lij ) .
Revised: σ ( b lij ) .

Original: For binary networks we simply ﬁx p ( w lij = 0)
Revised: For binary networks we simply ﬁx p ( w lij = 0)

Original: = 0 .
Revised: = 0 .

Original: 4. I NITIALIZATION
Revised: 4. I NITIALIZATION

Original: In a regular DNN setting, we usually either initialize the weights with a random initializer, e.g
Revised: In a regular DNN setting, we usually either initialize the weights with a random initializer, e.g

Original: Xavier initializer (Glorot & Bengio, 2010) or set the initial values of the weights to some pretrained value from a classiﬁcation network or another similar task.
Revised: Xavier initializer (Glorot & Bengio, 2010) or set the initial values of the weights to some pretrained value from a classiﬁcation network or another similar task.

Original: Here we are interested in initializing distributions over discrete weights from pretrained continuous deterministic weights, which we shall denote as (cid:102) W .
Revised: Here we are interested in initializing distributions over discrete weights from pretrained continuous deterministic weights, which we shall denote as (cid:102) W .

Original: We normalized the pretrained weights (cid:102) W beforehand to be mostly in the [ − 1 , 1] range, by dividing the weights in each layer by σ l , the standard deviation of the weights in layer l .
Revised: We normalized the pretrained weights (cid:102) W beforehand to be mostly in the [ − 1 , 1] range, by dividing the weights in each layer by σ l , the standard deviation of the weights in layer l .

Original: We ﬁrst explain our initialization in the ternary setting.
Revised: We ﬁrst explain our initialization in the ternary setting.

Original: Our aim is threefold: ﬁrst, we aim to have our mean value as close as possible to the original full precision weight.
Revised: Our aim is threefold: ﬁrst, we aim to have our mean value as close as possible to the original full precision weight.

Original: Second, we want to have low variance.
Revised: Second, we want to have low variance.

Original: Last, we do not want our initial distributions to be too deterministic, in order not to start at a bad local minima.
Revised: Last, we do not want our initial distributions to be too deterministic, in order not to start at a bad local minima.

Original: If we just try to minimize the variance while keeping the mean at ˜ w lij then the optimal solution would have p ( w lij = 0)
Revised: If we just try to minimize the variance while keeping the mean at ˜ w lij then the optimal solution would have p ( w lij = 0)

Original: = 1 − | ˜ w lij | and the rest of the probability is at +1 or − 1 , depending on the sign of ˜ w lij .
Revised: = 1 − | ˜ w lij | and the rest of the probability is at +1 or − 1 , depending on the sign of ˜ w lij .

Original: This is not desirable, as our initialized distribution in such case could be too deterministic and always assign zero probability to one of the weights.
Revised: This is not desirable, as our initialized distribution in such case could be too deterministic and always assign zero probability to one of the weights.

Original: We therefore modify it and use the following initialization:
Revised: We therefore modify it and use the following initialization:

Original: We initialize the probability of p ( w lij = 0) to be
Revised: We initialize the probability of p ( w lij = 0) to be

Original: [Equation]
Revised: [Equation]

Original: ) where p min and p max are hyperparameters (set to 0.05 and 0.95 respectively in our experiments).
Revised: ) where p min and p max are hyperparameters (set to 0.05 and 0.95 respectively in our experiments).

Original: Using the initialization proposed in eq.
Revised: Using the initialization proposed in eq.

Original: 6, p ( w lij = 0) gets the maximum value p max when (cid:101) w lij = 0 , and decays linearly to p min when | (cid:101) w lij | = 1 .
Revised: 6, p ( w lij = 0) gets the maximum value p max when (cid:101) w lij = 0 , and decays linearly to p min when | (cid:101) w lij | = 1 .

Original: Next, we set p ( w lij = 1 | w lij (cid:54) = 0) so that the initialized mean of the discrete weight is equal to the original full precision weight:
Revised: Next, we set p ( w lij = 1 | w lij (cid:54) = 0) so that the initialized mean of the discrete weight is equal to the original full precision weight:

Original: [Equation]
Revised: [Equation]

Original: In order for the expression in eq.
Revised: In order for the expression in eq.

Original: 7 to be equal to (cid:101) w lij , we need to set:
Revised: 7 to be equal to (cid:101) w lij , we need to set:

Original: [Equation]
Revised: [Equation]

Original: Since not all the weights (cid:102) W are in the range of [ − 1 , 1] , values from equations 6, 8 can be larger than one or negative.
Revised: Since not all the weights (cid:102) W are in the range of [ − 1 , 1] , values from equations 6, 8 can be larger than one or negative.

Original: Hence, we clip these values to be in the range [ p min , p max ] .
Revised: Hence, we clip these values to be in the range [ p min , p max ] .

Original: For a binary setting, we simply set p ( w lij = 0)
Revised: For a binary setting, we simply set p ( w lij = 0)

Original: = 0 .
Revised: = 0 .

Original: 4. I NCREASING E NTROPY
Revised: 4. I NCREASING E NTROPY

Original: During some of our experiments with ternary networks, we found that many weight distributions converged to a deterministic value, and this led to suboptimal performance.
Revised: During some of our experiments with ternary networks, we found that many weight distributions converged to a deterministic value, and this led to suboptimal performance.

Original: This is problematic for two main reasons: First, this could lead to vanishing gradients due to saturation.
Revised: This is problematic for two main reasons: First, this could lead to vanishing gradients due to saturation.

Original: Second, due to the small number of effective random variables, our main assumption - that the pre-activation distribution can be approximated well by a Gaussian distribution - is violated.
Revised: Second, due to the small number of effective random variables, our main assumption - that the pre-activation distribution can be approximated well by a Gaussian distribution - is violated.

Original: An example of such case can be seen in ﬁg. 2a.
Revised: An example of such case can be seen in ﬁg. 2a.

Original: This can be solved easily by adding L 2 regularization on the distribution parameters (before sigmoid, a lij and b lij ) to penalize very low entropy distributions.
Revised: This can be solved easily by adding L 2 regularization on the distribution parameters (before sigmoid, a lij and b lij ) to penalize very low entropy distributions.

Original: We emphasize that this regularization term was used mainly to help during training, not to reduce over-ﬁtting.
Revised: We emphasize that this regularization term was used mainly to help during training, not to reduce over-ﬁtting.

Original: From here on, we shall denote this regularization hyper-parameter as probability decay , to distinguish it from the weight decay used on the last full-precision layer.
Revised: From here on, we shall denote this regularization hyper-parameter as probability decay , to distinguish it from the weight decay used on the last full-precision layer.

Original: We note that in practice, only an extremely small probability decay is needed, but it had a surprisingly large effect on the overall performance.
Revised: We note that in practice, only an extremely small probability decay is needed, but it had a surprisingly large effect on the overall performance.

Original: An example of such a case can be seen in ﬁg. 2b.
Revised: An example of such a case can be seen in ﬁg. 2b.

Original: A closer examination of the average entropy throughout the training and the effect of the proposed regularization, can be found in appendix B.
Revised: A closer examination of the average entropy throughout the training and the effect of the proposed regularization, can be found in appendix B.

Original: 4. R EDUCING E NTROPY
Revised: 4. R EDUCING E NTROPY

Original: In the binary setting, we experienced the opposite in some experiments.
Revised: In the binary setting, we experienced the opposite in some experiments.

Original: Since we do not have a zero weight, we found that many of the weight probabilities learned were stuck at mid-values around 0.
Revised: Since we do not have a zero weight, we found that many of the weight probabilities learned were stuck at mid-values around 0.

Original: so that the expected value of that weight would remain around zero.
Revised: so that the expected value of that weight would remain around zero.

Original: That is not a desirable property, since we aim to get probabilities closer to 0 or 1 when sampling the ﬁnal weights for evaluation.
Revised: That is not a desirable property, since we aim to get probabilities closer to 0 or 1 when sampling the ﬁnal weights for evaluation.

Original: For this case, we add a beta density regularizer on our probabilities, R ( p ) =
Revised: For this case, we add a beta density regularizer on our probabilities, R ( p ) =

Original: p α − 1 (1 − p ) β − 1 with α, β = 2 .
Revised: p α − 1 (1 − p ) β − 1 with α, β = 2 .

Original: From here on we shall denote this regularization hyper-parameter as beta parameter .
Revised: From here on we shall denote this regularization hyper-parameter as beta parameter .

Original: It is important to note that we are using R ( p ) as our regularizer and not − log( R ( p ))
Revised: It is important to note that we are using R ( p ) as our regularizer and not − log( R ( p ))

Original: (with α, β < 1 ) as might be more standard.
Revised: (with α, β < 1 ) as might be more standard.

Original: This is because − log( R ( p )) has a much stronger pull towards zero entropy.
Revised: This is because − log( R ( p )) has a much stronger pull towards zero entropy.

Original: In general we note that unlike ternary weights, binary weights require more careful ﬁne-tuning in order to perform well.
Revised: In general we note that unlike ternary weights, binary weights require more careful ﬁne-tuning in order to perform well.

Original: [Figure]
Revised: [Figure]

Original: 4. E VALUATION
Revised: 4. E VALUATION

Original: For evaluation, we sample discrete weights according to our trained probabilities and perform the standard inference with the sampled weights.
Revised: For evaluation, we sample discrete weights according to our trained probabilities and perform the standard inference with the sampled weights.

Original: We can sample the weights multiple times and pick those that performed best on the validation set.
Revised: We can sample the weights multiple times and pick those that performed best on the validation set.

Original: We note that since our trained distributions have low entropy, the test accuracy does not change signiﬁcantly between samples, leading to only a minor improvement in overall performance.
Revised: We note that since our trained distributions have low entropy, the test accuracy does not change signiﬁcantly between samples, leading to only a minor improvement in overall performance.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conducted extensive experiments on the MNIST, CIFAR-10 and ImageNet (ILSVRC2012) benchmarks.
Revised: We conducted extensive experiments on the MNIST, CIFAR-10 and ImageNet (ILSVRC2012) benchmarks.

Original: We present results in both binary and ternary settings.
Revised: We present results in both binary and ternary settings.

Original: In the binary setting, we compare our results with BinaryConnect (Courbariaux et al., 2015), BNN (Hubara et al., 2016), BWN and XNOR-Net (Rastegari et al., 2016).
Revised: In the binary setting, we compare our results with BinaryConnect (Courbariaux et al., 2015), BNN (Hubara et al., 2016), BWN and XNOR-Net (Rastegari et al., 2016).

Original: In the ternary setting, we compare our results with ternary weight networks (Li et al., 2016).
Revised: In the ternary setting, we compare our results with ternary weight networks (Li et al., 2016).

Original: In previous works with binary or ternary networks, the ﬁrst and last layer weights remained in full precision.
Revised: In previous works with binary or ternary networks, the ﬁrst and last layer weights remained in full precision.

Original: In our experiments, the ﬁrst layer is binary (or ternary) as well (with an exception of the binary ResNet-18), leading to even larger memory and energy savings.
Revised: In our experiments, the ﬁrst layer is binary (or ternary) as well (with an exception of the binary ResNet-18), leading to even larger memory and energy savings.

Original: As previous works we keep the ﬁnal layer in full precision as well.
Revised: As previous works we keep the ﬁnal layer in full precision as well.

Original: 5. MNIST
Revised: 5. MNIST

Original: MNIST is an image classiﬁcation benchmark dataset, containing 60K training images and 10K test images from 10 classes of digits 0 − 9 .
Revised: MNIST is an image classiﬁcation benchmark dataset, containing 60K training images and 10K test images from 10 classes of digits 0 − 9 .

Original: Images are 28 × 28 in gray-scale.
Revised: Images are 28 × 28 in gray-scale.

Original: For this dataset, we do not use any sort of preprocessing or augmentation.
Revised: For this dataset, we do not use any sort of preprocessing or augmentation.

Original: Following Li et al.
Revised: Following Li et al.

Original: (2016), we use the following architecture:
Revised: (2016), we use the following architecture:

Original: [Equation]
Revised: [Equation]

Original: Where C 5 is a 5 × 5 ReLU convolution layer, MP 2 is a 2 × 2 max-pooling layer, and FC is a fully connected layer.
Revised: Where C 5 is a 5 × 5 ReLU convolution layer, MP 2 is a 2 × 2 max-pooling layer, and FC is a fully connected layer.

Original: We adopt Batch Normalization (Ioffe & Szegedy, 2015) into the architecture after every convolution layer.
Revised: We adopt Batch Normalization (Ioffe & Szegedy, 2015) into the architecture after every convolution layer.

Original: The loss is minimized with Adam.
Revised: The loss is minimized with Adam.

Original: We use dropout (Srivastava et al., 2014)
Revised: We use dropout (Srivastava et al., 2014)

Original: [Table]
Revised: [Table]

Original: with a drop rate of 0 . 5 .
Revised: with a drop rate of 0 . 5 .

Original: Weight decay parameter (on the last full precision layer) is set to 1e − 4 .
Revised: Weight decay parameter (on the last full precision layer) is set to 1e − 4 .

Original: We use a batch size of 256, initial learning rate is 0.01 and is divided by 10 after 100 epochs of training.
Revised: We use a batch size of 256, initial learning rate is 0.01 and is divided by 10 after 100 epochs of training.

Original: For the binary setting, beta parameter is set to 1e − 6 .
Revised: For the binary setting, beta parameter is set to 1e − 6 .

Original: For the ternary setting, probability decay parameter is set to 1e − 11 .
Revised: For the ternary setting, probability decay parameter is set to 1e − 11 .

Original: We report the test error rate after 190 training epochs.
Revised: We report the test error rate after 190 training epochs.

Original: The results are presented in Table 1.
Revised: The results are presented in Table 1.

Original: 5. CIFAR- 
Revised: 5. CIFAR-

Original: CIFAR-10 is an image classiﬁcation benchmark dataset (Krizhevsky, 2009), containing 50K training images and 10K test images from 10 classes.
Revised: CIFAR-10 is an image classiﬁcation benchmark dataset (Krizhevsky, 2009), containing 50K training images and 10K test images from 10 classes.

Original: Images are 32 × 32 in RGB space.
Revised: Images are 32 × 32 in RGB space.

Original: Each image is preprocessed by subtracting its mean and dividing by its standard deviation.
Revised: Each image is preprocessed by subtracting its mean and dividing by its standard deviation.

Original: During training, we pad each side of the image with 4 pixels, and a random 32 × 32 crop is sampled from the padded image.
Revised: During training, we pad each side of the image with 4 pixels, and a random 32 × 32 crop is sampled from the padded image.

Original: Images are randomly ﬂipped horizontally.
Revised: Images are randomly ﬂipped horizontally.

Original: At test time, we evaluate the single original 32 × 32 image without any padding or multiple cropping.
Revised: At test time, we evaluate the single original 32 × 32 image without any padding or multiple cropping.

Original: As in Li et al.
Revised: As in Li et al.

Original: (2016), we use the VGG inspired architecture:
Revised: (2016), we use the VGG inspired architecture:

Original: [Equation]
Revised: [Equation]

Original: Where C 3 is a 3 × 3 ReLU convolution layer, MP 2 is a 2 × 2 max-pooling layer, and FC is a fully connected layer.
Revised: Where C 3 is a 3 × 3 ReLU convolution layer, MP 2 is a 2 × 2 max-pooling layer, and FC is a fully connected layer.

Original: It is worth noting that Courbariaux et al.
Revised: It is worth noting that Courbariaux et al.

Original: and
Revised: and

Original: Hubara et al.
Revised: Hubara et al.

Original: (2016) use a very similar architecture with two differences, namely adopting an extra fully connected layer and using an L2-SVM output layer, instead of softmax as in our experiments.
Revised: (2016) use a very similar architecture with two differences, namely adopting an extra fully connected layer and using an L2-SVM output layer, instead of softmax as in our experiments.

Original: We adopt Batch
Revised: We adopt Batch

Original: Normalization into the architecture after every convolution layer.
Revised: Normalization into the architecture after every convolution layer.

Original: The loss is minimized with Adam.
Revised: The loss is minimized with Adam.

Original: We use dropout with a drop rate of 0 . 5 .
Revised: We use dropout with a drop rate of 0 . 5 .

Original: Weight decay parameter is set to 1e − 4 .
Revised: Weight decay parameter is set to 1e − 4 .

Original: We use a batch size of 256, initial learning rate is 0.01 and is divided by 10 after 170 epochs of training.
Revised: We use a batch size of 256, initial learning rate is 0.01 and is divided by 10 after 170 epochs of training.

Original: For the binary setting, beta parameter is set to 1e − 6 .
Revised: For the binary setting, beta parameter is set to 1e − 6 .

Original: For the ternary setting, probability decay parameter is set to 1e − 11 .
Revised: For the ternary setting, probability decay parameter is set to 1e − 11 .

Original: We report the test error rate after 290 training epochs.
Revised: We report the test error rate after 290 training epochs.

Original: The results are presented in Table 1.
Revised: The results are presented in Table 1.

Original: 5. I MAGE N ET
Revised: 5. I MAGE N ET

Original: ImageNet 2012 (ILSVRC2012) is a large scale image classiﬁcation dataset (Deng et al., 2009), consisting of 1.28 million training images, and 50K validation images from 1000 classes.
Revised: ImageNet 2012 (ILSVRC2012) is a large scale image classiﬁcation dataset (Deng et al., 2009), consisting of 1.28 million training images, and 50K validation images from 1000 classes.

Original: We adopt the proposed ResNet-18 (He et al., 2016), as in Rastegari et al.
Revised: We adopt the proposed ResNet-18 (He et al., 2016), as in Rastegari et al.

Original: (2016) and Li et al.
Revised: (2016) and Li et al.

Original: Each image is preprocessed by subtracting the mean pixel of the whole training set and dividing by the standard deviation.
Revised: Each image is preprocessed by subtracting the mean pixel of the whole training set and dividing by the standard deviation.

Original: Images are resized so that the shorter side is set to 256.
Revised: Images are resized so that the shorter side is set to 256.

Original: During training, the network is fed with a random 224 × 224 crop and images are randomly ﬂipped horizontally.
Revised: During training, the network is fed with a random 224 × 224 crop and images are randomly ﬂipped horizontally.

Original: At test time, the center 224 × 224 crop is taken.
Revised: At test time, the center 224 × 224 crop is taken.

Original: We evaluate only with the single center crop.
Revised: We evaluate only with the single center crop.

Original: The loss is minimized with Adam.
Revised: The loss is minimized with Adam.

Original: Weight decay parameter is set to 1e − 5 , we use a batch size of 256 and initial learning rate is 0.01.
Revised: Weight decay parameter is set to 1e − 5 , we use a batch size of 256 and initial learning rate is 0.01.

Original: For the binary setting, we found that the beta regularizer is not needed and got the best results when beta parameter is set to 0 .
Revised: For the binary setting, we found that the beta regularizer is not needed and got the best results when beta parameter is set to 0 .

Original: Learning rate is divided by 10 after
Revised: Learning rate is divided by 10 after

Original: BPWNs

Original: [Table]
Revised: [Table]

Original: 50 and 60 epochs
Revised: 50 and 60 epochs

Original: and we report the test error rate after 65 training epochs.
Revised: and we report the test error rate after 65 training epochs.

Original: For the ternary setting, probability decay parameter is set to 1e − 12 .
Revised: For the ternary setting, probability decay parameter is set to 1e − 12 .

Original: For this setting, learning rate is divided by 10 afterand 44 epochs and we report the test error rate after 55 training epochs.
Revised: For this setting, learning rate is divided by 10 afterand 44 epochs and we report the test error rate after 55 training epochs.

Original: The results are presented in
Revised: The results are presented in

Original: Table 2.
Revised: Table 2.

Original: 5. K ERNEL V ISUALIZATION
Revised: 5. K ERNEL V ISUALIZATION

Original: We visualize the ﬁrst 25 kernels from the ﬁrst layer of the network trained on MNIST.
Revised: We visualize the ﬁrst 25 kernels from the ﬁrst layer of the network trained on MNIST.

Original: Fig.
Revised: Fig.

Original: 3a, 3b and 3c shows these kernels for the binary, ternary and full precision networks respectively.
Revised: 3a, 3b and 3c shows these kernels for the binary, ternary and full precision networks respectively.

Original: We can see that our suggested initialization from full precision networks takes advantage of those trained weights and preserves the structure of the kernels.
Revised: We can see that our suggested initialization from full precision networks takes advantage of those trained weights and preserves the structure of the kernels.

Original: [Figure]
Revised: [Figure]

Original: 5. B INARY VS T ERNARY
Revised: 5. B INARY VS T ERNARY

Original: In our experiments we evaluate binary and ternary versions of similar networks on similar datasets.
Revised: In our experiments we evaluate binary and ternary versions of similar networks on similar datasets.

Original: When trying to compare the two, it is important to note that the binary results on MNIST and
Revised: When trying to compare the two, it is important to note that the binary results on MNIST and

Original: CIFAR-10 are a bit misleading.
Revised: CIFAR-10 are a bit misleading.

Original: While the ﬁnal accuracies are similar, the ternary network work well from the start while the binary networks required more work and careful parameter tuning to achieve similar performance.
Revised: While the ﬁnal accuracies are similar, the ternary network work well from the start while the binary networks required more work and careful parameter tuning to achieve similar performance.

Original: On harder tasks, like ImageNet classiﬁcation, the performance gap is signiﬁcant even when the ternary network discretizes the ﬁrst layer while the binary did not.
Revised: On harder tasks, like ImageNet classiﬁcation, the performance gap is signiﬁcant even when the ternary network discretizes the ﬁrst layer while the binary did not.

Original: We believe that ternary networks are a much more reasonable model as they do not force each neuron to affect all next-layer neurons, and allow us to model low-mean low-variance stochastic weights.
Revised: We believe that ternary networks are a much more reasonable model as they do not force each neuron to affect all next-layer neurons, and allow us to model low-mean low-variance stochastic weights.

Original: The difference in computation time is small as ternary weights do not require multiplications, giving what we consider a worthwhile trade-off between performance and runtime.
Revised: The difference in computation time is small as ternary weights do not require multiplications, giving what we consider a worthwhile trade-off between performance and runtime.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we presented a simple, novel and effective algorithm to train neural networks with discrete weights.
Revised: In this work we presented a simple, novel and effective algorithm to train neural networks with discrete weights.

Original: We showed results on various image classiﬁcation datasets and reached state-ofthe-art results in both the binary and ternary settings on most datasets, paving the way for easier and more efﬁcient training and inference of efﬁcient low-power consuming neural networks.
Revised: We showed results on various image classiﬁcation datasets and reached state-ofthe-art results in both the binary and ternary settings on most datasets, paving the way for easier and more efﬁcient training and inference of efﬁcient low-power consuming neural networks.

Original: Moreover, comparing binary and ternary networks we advocate further research into ternary weights as a much more reasonable model than binary weights, with a modest computation and memory overhead.
Revised: Moreover, comparing binary and ternary networks we advocate further research into ternary weights as a much more reasonable model than binary weights, with a modest computation and memory overhead.

Original: Further work into sparse ternary networks might help reduce, or even eliminate, this overhead compared to binary networks.
Revised: Further work into sparse ternary networks might help reduce, or even eliminate, this overhead compared to binary networks.

Original: A C OMPARISON TO G UMBEL - SOFTMAX
Revised: A C OMPARISON TO G UMBEL - SOFTMAX

Original: We compare our continuous relaxation to a standard continous relaxation using Gumbel-softmax approximation (Jang et al., 2017).
Revised: We compare our continuous relaxation to a standard continous relaxation using Gumbel-softmax approximation (Jang et al., 2017).

Original: We ran both experiments on CIFAR-10 using the exact same parameters and initialization we use in Sec.
Revised: We ran both experiments on CIFAR-10 using the exact same parameters and initialization we use in Sec.

Original: for 10 epochs.
Revised: for 10 epochs.

Original: For the Gumbel-softmax we use a temperature τ = 0 .
Revised: For the Gumbel-softmax we use a temperature τ = 0 .

Original: We also tried other τ parameters in the [0 . 01 , 0 . 5] range and did not ﬁnd anything that worked noticeably better.
Revised: We also tried other τ parameters in the [0 . 01 , 0 . 5] range and did not ﬁnd anything that worked noticeably better.

Original: We also did not use annealing of the temperature as this is only 10 epochs.
Revised: We also did not use annealing of the temperature as this is only 10 epochs.

Original: Comparing the results in Fig.
Revised: Comparing the results in Fig.

Original: 4, it is clear that our proposed method works much better then simply using Gumbel-softmax on the weights.
Revised: 4, it is clear that our proposed method works much better then simply using Gumbel-softmax on the weights.

Original: The reason our method works better is twofold.
Revised: The reason our method works better is twofold.

Original: First, we enjoy the lower variance of the local reparamatrization trick since there is one shared source of randomness per pre-activation.
Revised: First, we enjoy the lower variance of the local reparamatrization trick since there is one shared source of randomness per pre-activation.

Original: Second, the Gumbel-softmax has a trade-off: when the temperature goes down the approximation gets better but the activation saturates and the gradients vanish so it is harder to train.
Revised: Second, the Gumbel-softmax has a trade-off: when the temperature goes down the approximation gets better but the activation saturates and the gradients vanish so it is harder to train.

Original: We do not have this problem as we get good approximation with non-vanishing gradients.
Revised: We do not have this problem as we get good approximation with non-vanishing gradients.

Original: B A VERAGE E NTROPY
Revised: B A VERAGE E NTROPY

Original: In order to ensure the correctness of our proposed regularization in a ternary setting, we compare the average entropy of the learned probabilities throughout the training, with and without the probability decay term.
Revised: In order to ensure the correctness of our proposed regularization in a ternary setting, we compare the average entropy of the learned probabilities throughout the training, with and without the probability decay term.

Original: We examine the average entropy during the training procedure of a network trained on
Revised: We examine the average entropy during the training procedure of a network trained on

Original: CIFAR-10 in a ternary setting.
Revised: CIFAR-10 in a ternary setting.

Original: For the experiment with probability decay, we set the probability decay parameter to 1e − 11 , as in Sec. 5.
Revised: For the experiment with probability decay, we set the probability decay parameter to 1e − 11 , as in Sec. 5.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 5a shows the average entropy of the third convolutional layer, during the later parts of the training.
Revised: 5a shows the average entropy of the third convolutional layer, during the later parts of the training.

Original: We can see that the proposed regularization causes a slight increase in entropy, which is what we aimed to achieve.
Revised: We can see that the proposed regularization causes a slight increase in entropy, which is what we aimed to achieve.

Original: Note that the ’kink’ in the graph is due to lowering the learning rate after 170 epochs.
Revised: Note that the ’kink’ in the graph is due to lowering the learning rate after 170 epochs.

Original: For some of the layers this regularization has a much more signiﬁcant effect.
Revised: For some of the layers this regularization has a much more signiﬁcant effect.

Original: Fig.
Revised: Fig.

Original: 5b shows the average entropy throughout the training for the last convolutional layer.
Revised: 5b shows the average entropy throughout the training for the last convolutional layer.

Original: For this layer, the entropy at the end of the training is higher than the entropy in initialization.
Revised: For this layer, the entropy at the end of the training is higher than the entropy in initialization.

Original: Looking at the probabilities themselves helps better explain this behavior.
Revised: Looking at the probabilities themselves helps better explain this behavior.

Original: [Figure]
Revised: [Figure]

Original: Fig 6a shows a histogram of the initialized probability p ( w ij = 0) of this layer.
Revised: Fig 6a shows a histogram of the initialized probability p ( w ij = 0) of this layer.

Original: Fig 6b shows a histogram of these learned probabilities at the end of the training.
Revised: Fig 6b shows a histogram of these learned probabilities at the end of the training.

Original: It can be seen that for this layer there were many weights that were assigned a very high (or very low) probability value for p ( w ij = 0) .
Revised: It can be seen that for this layer there were many weights that were assigned a very high (or very low) probability value for p ( w ij = 0) .

Original: However, the probability decay regularization pushed the learned values to be less deterministic, causing the increase in entropy observed in Fig.
Revised: However, the probability decay regularization pushed the learned values to be less deterministic, causing the increase in entropy observed in Fig.

Original: 5b.
Revised: 5b.

Original: [Figure]
Revised: [Figure]