Original: BNN + : I MPROVED B INARY N ETWORK T RAINING
Revised: BNN + : I MPROVED B INARY N ETWORK T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNN) are widely used in many applications.
Revised: Deep neural networks (DNN) are widely used in many applications.

Original: However, their deployment on edge devices has been difﬁcult because they are resource hungry.
Revised: However, their deployment on edge devices has been difﬁcult because they are resource hungry.

Original: Binary neural networks (BNN) help to alleviate the prohibitive resource requirements of DNN, where both activations and weights are limited to 1 -bit.
Revised: Binary neural networks (BNN) help to alleviate the prohibitive resource requirements of DNN, where both activations and weights are limited to 1 -bit.

Original: We propose an improved binary training method (BNN+), by introducing a regularization function that encourages training weights around binary values.
Revised: We propose an improved binary training method (BNN+), by introducing a regularization function that encourages training weights around binary values.

Original: In addition to this, to enhance model performance we add trainable scaling factors to our regularization functions.
Revised: In addition to this, to enhance model performance we add trainable scaling factors to our regularization functions.

Original: Furthermore, we use an improved approximation of the derivative of the sign activation function in the backward computation.
Revised: Furthermore, we use an improved approximation of the derivative of the sign activation function in the backward computation.

Original: These additions are based on linear operations that are easily implementable into the binary training framework.
Revised: These additions are based on linear operations that are easily implementable into the binary training framework.

Original: We show experimental results on CIFAR-10 obtaining an accuracy of 86 .
Revised: We show experimental results on CIFAR-10 obtaining an accuracy of 86 .

Original: 5% , on AlexNet and 91 .
Revised: 5% , on AlexNet and 91 .

Original: 3% with VGG network.
Revised: 3% with VGG network.

Original: On ImageNet, our method also outperforms the traditional BNN method and XNOR-net, using
Revised: On ImageNet, our method also outperforms the traditional BNN method and XNOR-net, using

Original: AlexNet by a margin of 4 % and 2 % top- 1 accuracy respectively.
Revised: AlexNet by a margin of 4 % and 2 % top- 1 accuracy respectively.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) have demonstrated success for many supervised learning tasks ranging from voice recognition to object detection (Szegedy et al., 2015; Simonyan & Zisserman, 2014;
Revised: Deep neural networks (DNNs) have demonstrated success for many supervised learning tasks ranging from voice recognition to object detection (Szegedy et al., 2015; Simonyan & Zisserman, 2014;

Original: Iandola et al., 2016).
Revised: Iandola et al., 2016).

Original: The focus has been on increasing accuracy, in particular for image tasks, where deep convolutional neural networks (CNNs) are widely used.
Revised: The focus has been on increasing accuracy, in particular for image tasks, where deep convolutional neural networks (CNNs) are widely used.

Original: However, their increasing complexity poses a new challenge, and has become an impediment to widespread deployment in many applications; speciﬁcally when trying to deploy such models to resource constrained and lower-power devices.
Revised: However, their increasing complexity poses a new challenge, and has become an impediment to widespread deployment in many applications; speciﬁcally when trying to deploy such models to resource constrained and lower-power devices.

Original: A typical DNN architecture contains tens to thousands of layers, resulting in millions of parameters.
Revised: A typical DNN architecture contains tens to thousands of layers, resulting in millions of parameters.

Original: As an example, AlexNet (Krizhevsky et al., 2012) requires 200MB of memory, VGGNet (Simonyan & Zisserman, 2014) requires 500MB memory.
Revised: As an example, AlexNet (Krizhevsky et al., 2012) requires 200MB of memory, VGGNet (Simonyan & Zisserman, 2014) requires 500MB memory.

Original: Large model sizes are further exacerbated by their computational cost, requiring GPU implementation to allow real-time inference.
Revised: Large model sizes are further exacerbated by their computational cost, requiring GPU implementation to allow real-time inference.

Original: Such requirements evidently cannot be accustomed by edge devices as they have limited memory, computation power, and battery.
Revised: Such requirements evidently cannot be accustomed by edge devices as they have limited memory, computation power, and battery.

Original: This motivated the community to investigate methods for compressing and reducing computation cost of DNNs.
Revised: This motivated the community to investigate methods for compressing and reducing computation cost of DNNs.

Original: To make DNNs compatible with the resource constraints of low power devices, there have been several approaches developed, such as network pruning (LeCun et al., 1990), architecture design (Sandler et al., 2018), and quantization (Courbariaux et al., 2015; Han et al., 2015).
Revised: To make DNNs compatible with the resource constraints of low power devices, there have been several approaches developed, such as network pruning (LeCun et al., 1990), architecture design (Sandler et al., 2018), and quantization (Courbariaux et al., 2015; Han et al., 2015).

Original: In particular, weight compression using quantization can achieve very large savings in memory, where binary (1bit), and ternary (2-bit) approaches have been shown to obtain competitive accuracy (Hubara et al., 2016; Zhu et al., 2016; Tang et al., 2017).
Revised: In particular, weight compression using quantization can achieve very large savings in memory, where binary (1bit), and ternary (2-bit) approaches have been shown to obtain competitive accuracy (Hubara et al., 2016; Zhu et al., 2016; Tang et al., 2017).

Original: Using such schemes reduces model sizes by 8x to 32x depending on the bit resolution used for computations.
Revised: Using such schemes reduces model sizes by 8x to 32x depending on the bit resolution used for computations.

Original: In addition to this, the speed by quantizing the activation layers.
Revised: In addition to this, the speed by quantizing the activation layers.

Original: In this way, both the weights and activations are quantized so that one can replace the expensive dot products and activation function evaluations with bitwise operations.
Revised: In this way, both the weights and activations are quantized so that one can replace the expensive dot products and activation function evaluations with bitwise operations.

Original: This reduction in bit-width beneﬁts hardware accelerators such as FPGAs and neural network chips.
Revised: This reduction in bit-width beneﬁts hardware accelerators such as FPGAs and neural network chips.

Original: An issue with using low-bit DNNs is the drastic drop in accuracy compared to its full precision counterpart, and this is made even more severe upon quantizing the activations.
Revised: An issue with using low-bit DNNs is the drastic drop in accuracy compared to its full precision counterpart, and this is made even more severe upon quantizing the activations.

Original: This problem is largely due to noise and lack of precision in the training objective of the networks during back-propagation (Lin et al., 2017).
Revised: This problem is largely due to noise and lack of precision in the training objective of the networks during back-propagation (Lin et al., 2017).

Original: Although, quantizing the weights and activations have been attracting large interests thanks to their computational beneﬁts, closing the gap in accuracy between the full precision and the quantized version remains a challenge.
Revised: Although, quantizing the weights and activations have been attracting large interests thanks to their computational beneﬁts, closing the gap in accuracy between the full precision and the quantized version remains a challenge.

Original: Indeed, quantizing weights cause drastic information loss and make neural networks harder to train due to a large number of sign ﬂuctuations in the weights.
Revised: Indeed, quantizing weights cause drastic information loss and make neural networks harder to train due to a large number of sign ﬂuctuations in the weights.

Original: Therefore, how to control the stability of this training procedure is of high importance.
Revised: Therefore, how to control the stability of this training procedure is of high importance.

Original: In theory, it is infeasible to back-propagate in a quantized setting as the weights and activations employed are discontinuous and discrete.
Revised: In theory, it is infeasible to back-propagate in a quantized setting as the weights and activations employed are discontinuous and discrete.

Original: Instead, heuristics and approximations are proposed to match the forward and backward passes.
Revised: Instead, heuristics and approximations are proposed to match the forward and backward passes.

Original: Often weights at different layers of DNNs follow a certain structure.
Revised: Often weights at different layers of DNNs follow a certain structure.

Original: How to quantize the weights locally, and maintaining a global structure to minimize a common cost function is important (Li et al., 2017).
Revised: How to quantize the weights locally, and maintaining a global structure to minimize a common cost function is important (Li et al., 2017).

Original: Our contribution consists of three ideas that can be easily implemented in the binary training framework presented by Hubara et al.
Revised: Our contribution consists of three ideas that can be easily implemented in the binary training framework presented by Hubara et al.

Original: (2016) to improve convergence and generalization accuracy of binary networks.
Revised: (2016) to improve convergence and generalization accuracy of binary networks.

Original: First, the activation function is modiﬁed to better approximate the sign function in the backward pass, second we propose two regularization functions that encourage training weights around binary values, and lastly a scaling factor is introduced in both the regularization term as well as network building blocks to mitigate accuracy drop due to hard binarization.
Revised: First, the activation function is modiﬁed to better approximate the sign function in the backward pass, second we propose two regularization functions that encourage training weights around binary values, and lastly a scaling factor is introduced in both the regularization term as well as network building blocks to mitigate accuracy drop due to hard binarization.

Original: Our method is evaluated on CIFAR-10 and ImageNet datasets and compared to other binary methods.
Revised: Our method is evaluated on CIFAR-10 and ImageNet datasets and compared to other binary methods.

Original: We show accuracy gains to traditional binary training.
Revised: We show accuracy gains to traditional binary training.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We focus on challenges present in training binary networks.
Revised: We focus on challenges present in training binary networks.

Original: The training procedure emulates binary operations by restricting the weights and activations to single-bit so that computations of neural networks can be implemented using arithmetic logic units (ALU) using XNOR and popcount operations.
Revised: The training procedure emulates binary operations by restricting the weights and activations to single-bit so that computations of neural networks can be implemented using arithmetic logic units (ALU) using XNOR and popcount operations.

Original: More speciﬁcally, XNOR and popcount instructions are readily available on most CPU and GPU processing units.
Revised: More speciﬁcally, XNOR and popcount instructions are readily available on most CPU and GPU processing units.

Original: Custom hardware would have to be implemented to take advantage of operations with higher bits such as 2 to 4 bits.
Revised: Custom hardware would have to be implemented to take advantage of operations with higher bits such as 2 to 4 bits.

Original: The goal of this binary training is to reduce the model size and gain inference speedups without performance degradation.
Revised: The goal of this binary training is to reduce the model size and gain inference speedups without performance degradation.

Original: Primary work done by Courbariaux et al.
Revised: Primary work done by Courbariaux et al.

Original: (2015) (BinaryConnect) trains deep neural networks with binary weights {− 1 , +1 } .
Revised: (2015) (BinaryConnect) trains deep neural networks with binary weights {− 1 , +1 } .

Original: They propose to quantize real values using the sign function.
Revised: They propose to quantize real values using the sign function.

Original: The propagated gradient applies update to weights | w | ≤ 1 .
Revised: The propagated gradient applies update to weights | w | ≤ 1 .

Original: Once the weights are outside of this region they are no longer updated, this is done by clipping weights between {− 1 , +1 } .
Revised: Once the weights are outside of this region they are no longer updated, this is done by clipping weights between {− 1 , +1 } .

Original: In that work, they did not consider binarizing the activation functions.
Revised: In that work, they did not consider binarizing the activation functions.

Original: BNN (Hubara et al., 2016) is the ﬁrst purely binary network quantizing both the weights and activations.
Revised: BNN (Hubara et al., 2016) is the ﬁrst purely binary network quantizing both the weights and activations.

Original: They achieve comparable accuracy to their prior work on BinaryConnect, and achieve signiﬁcantly close performance to full-precision, by using large and deep networks.
Revised: They achieve comparable accuracy to their prior work on BinaryConnect, and achieve signiﬁcantly close performance to full-precision, by using large and deep networks.

Original: Although, they performed poorly on large datasets like ImageNet (Russakovsky et al., 2015).
Revised: Although, they performed poorly on large datasets like ImageNet (Russakovsky et al., 2015).

Original: The resulting network presented in their work obtains 32 × compression rate and approximately 7 × increase in inference speed.
Revised: The resulting network presented in their work obtains 32 × compression rate and approximately 7 × increase in inference speed.

Original: To alleviate the accuracy drop of BNN on larger datasets, Rastegari et al.
Revised: To alleviate the accuracy drop of BNN on larger datasets, Rastegari et al.

Original: (2016) proposed XNORNet, where they strike a trade-off between compression and accuracy through the use of scaling factors for both weights and activation functions.
Revised: (2016) proposed XNORNet, where they strike a trade-off between compression and accuracy through the use of scaling factors for both weights and activation functions.

Original: In their work, they show performance gains compared to BNN on ImageNet classiﬁcation.
Revised: In their work, they show performance gains compared to BNN on ImageNet classiﬁcation.

Original: The scaling factors for both the weights and activations are computed dynamically, which slows down training performance.
Revised: The scaling factors for both the weights and activations are computed dynamically, which slows down training performance.

Original: Further, they introduced an additional complexity in implementing the convolution operations on the hardware, slightly reducing compression rate and speed up gains.
Revised: Further, they introduced an additional complexity in implementing the convolution operations on the hardware, slightly reducing compression rate and speed up gains.

Original: DoReFa-Net
Revised: DoReFa-Net

Original: (Zhou et al., 2016) further improves XNOR-Net by approximating the activations with more bits.
Revised: (Zhou et al., 2016) further improves XNOR-Net by approximating the activations with more bits.

Original: The proposed rounding mechanism allows for low bit back-propagation as well.
Revised: The proposed rounding mechanism allows for low bit back-propagation as well.

Original: Although they perform multi-bit quantization, their model still suffers from large accuracy drop upon quantizing the last layer.
Revised: Although they perform multi-bit quantization, their model still suffers from large accuracy drop upon quantizing the last layer.

Original: Later in ABC-Net, Tang et al.
Revised: Later in ABC-Net, Tang et al.

Original: (2017) propose several strategies, adjusting the learning rate for larger datasets.
Revised: (2017) propose several strategies, adjusting the learning rate for larger datasets.

Original: They show BNN achieve similar accuracy as XNOR-Net without the scaling overhead by adding a regularizer term which allows binary networks to generalize better.
Revised: They show BNN achieve similar accuracy as XNOR-Net without the scaling overhead by adding a regularizer term which allows binary networks to generalize better.

Original: They also suggest a modiﬁed BNN, where they adopted the strategy of increasing the number of ﬁlters, to compensate for accuracy loss similar to wide reduced-precision networks (Mishra et al., 2017).
Revised: They also suggest a modiﬁed BNN, where they adopted the strategy of increasing the number of ﬁlters, to compensate for accuracy loss similar to wide reduced-precision networks (Mishra et al., 2017).

Original: More recently,
Revised: More recently,

Original: Liu et al.
Revised: Liu et al.

Original: (2018) developed a second-order approximation to the sign activation function for a more accurate backward update.
Revised: (2018) developed a second-order approximation to the sign activation function for a more accurate backward update.

Original: In addition to this, they pre-train the network in which they want to binarize in full precision using the hard tangent hyperbolic ( htanh ) activation, see Figure 2.
Revised: In addition to this, they pre-train the network in which they want to binarize in full precision using the hard tangent hyperbolic ( htanh ) activation, see Figure 2.

Original: They use the pre-trained network weights as an initialization for the binary network to obtain state of the art performance.
Revised: They use the pre-trained network weights as an initialization for the binary network to obtain state of the art performance.

Original:  I MPROVED B INARY T RAINING
Revised:  I MPROVED B INARY T RAINING

Original: Training a binary neural network faces two major challenges: on weights, and on activation functions.
Revised: Training a binary neural network faces two major challenges: on weights, and on activation functions.

Original: As both weights and activations are binary, the traditional continuous optimization methods such as SGD cannot be directly applied.
Revised: As both weights and activations are binary, the traditional continuous optimization methods such as SGD cannot be directly applied.

Original: Instead, a continuous approximation is used for the sign activation during the backward pass.
Revised: Instead, a continuous approximation is used for the sign activation during the backward pass.

Original: Further, the gradient of the loss with respect to the weights are small.
Revised: Further, the gradient of the loss with respect to the weights are small.

Original: So as training progresses weight sign remains unchanged.
Revised: So as training progresses weight sign remains unchanged.

Original: These are both addressed in our proposed method.
Revised: These are both addressed in our proposed method.

Original: In this section, we present our approach to training 1 -bit CNNs in detail.
Revised: In this section, we present our approach to training 1 -bit CNNs in detail.

Original: 3. B INARY TRAINING
Revised: 3. B INARY TRAINING

Original: We quickly revisit quantization through binary training as ﬁrst presented by (Courbariaux et al., 2015).
Revised: We quickly revisit quantization through binary training as ﬁrst presented by (Courbariaux et al., 2015).

Original: In (Hubara et al., 2016), the weights are quantized by using the sign function which is +1 if w > 0 and − 1 otherwise.
Revised: In (Hubara et al., 2016), the weights are quantized by using the sign function which is +1 if w > 0 and − 1 otherwise.

Original: In the forward pass, the real-valued weights are binarized to w b , and the resulting loss is computed using binary weights throughout the network.
Revised: In the forward pass, the real-valued weights are binarized to w b , and the resulting loss is computed using binary weights throughout the network.

Original: For hidden units, the sign function non-linearity is used to obtain binary activations.
Revised: For hidden units, the sign function non-linearity is used to obtain binary activations.

Original: Prior to binarizing, the real weights are stored in a temporary variable w .
Revised: Prior to binarizing, the real weights are stored in a temporary variable w .

Original: The variables w are stored because one cannot back-propagate through the sign operation as its gradient is zero everywhere, and hence disturbs learning.
Revised: The variables w are stored because one cannot back-propagate through the sign operation as its gradient is zero everywhere, and hence disturbs learning.

Original: To alleviate this problem the authors suggest using a straight through estimator (Hinton, 2012) for the gradient of the sign function.
Revised: To alleviate this problem the authors suggest using a straight through estimator (Hinton, 2012) for the gradient of the sign function.

Original: This method is a heuristic way of approximating the gradient of a neuron,
Revised: This method is a heuristic way of approximating the gradient of a neuron,

Original: [Equation]
Revised: [Equation]

Original: (1) where L is the loss function and 1 ( . )
Revised: (1) where L is the loss function and 1 ( . )

Original: is the indicator function.
Revised: is the indicator function.

Original: The gradients in the backward pass are then applied to weights that are within [ − 1 , +1] .
Revised: The gradients in the backward pass are then applied to weights that are within [ − 1 , +1] .

Original: The training process is summarized in Figure 1.
Revised: The training process is summarized in Figure 1.

Original: As weights undergo gradient updates, they are eventually pushed out of the center region and instead make two modes, one at − 1 and another at +1 .
Revised: As weights undergo gradient updates, they are eventually pushed out of the center region and instead make two modes, one at − 1 and another at +1 .

Original: This progression is also shown in Figure 1.
Revised: This progression is also shown in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: 3. I MPROVED T RAINING METHOD
Revised: 3. I MPROVED T RAINING METHOD

Original: Our ﬁrst modiﬁcation is on closing the discrepancy between the forward pass and backward pass.
Revised: Our ﬁrst modiﬁcation is on closing the discrepancy between the forward pass and backward pass.

Original: Originally, the sign derivative is approximated using the htanh( x ) activation, as in Figure 2.
Revised: Originally, the sign derivative is approximated using the htanh( x ) activation, as in Figure 2.

Original: Instead, we modify the Swish-like activation (Ramachandran et al., 2017;
Revised: Instead, we modify the Swish-like activation (Ramachandran et al., 2017;

Original: Elfwing et al., 2018; Hendrycks &
Revised: Elfwing et al., 2018; Hendrycks &

Original: Gimpel, 2016), where it has shown to outperform other activations on various tasks.
Revised: Gimpel, 2016), where it has shown to outperform other activations on various tasks.

Original: The modiﬁcations are performed by taking its derivative and centering it around
Revised: The modiﬁcations are performed by taking its derivative and centering it around

Original: [Equation]
Revised: [Equation]

Original: where σ ( z ) is the sigmoid function and the scale β > 0 controls how fast the activation function asymptotes to − 1 and +1 .
Revised: where σ ( z ) is the sigmoid function and the scale β > 0 controls how fast the activation function asymptotes to − 1 and +1 .

Original: The β parameter can be learned by the network or be hand-tuned as a hyperparameter.
Revised: The β parameter can be learned by the network or be hand-tuned as a hyperparameter.

Original: As opposed to the Swish function, where it is unbounded on the right side, the modiﬁcation makes it bounded and a valid approximator of the sign function.
Revised: As opposed to the Swish function, where it is unbounded on the right side, the modiﬁcation makes it bounded and a valid approximator of the sign function.

Original: As a result, we call this activation SignSwish, and its gradient is
Revised: As a result, we call this activation SignSwish, and its gradient is

Original: [Equation]
Revised: [Equation]

Original: which is a closer approximation function compared to the htanh activation.
Revised: which is a closer approximation function compared to the htanh activation.

Original: Comparisons are made in Figure 2.
Revised: Comparisons are made in Figure 2.

Original: Hubara et al.
Revised: Hubara et al.

Original: (2016) noted that the STE fails to learn weights near the borders of − 1 and +1 .
Revised: (2016) noted that the STE fails to learn weights near the borders of − 1 and +1 .

Original: As depicted in Figure 2, our proposed SignSwish activation alleviates this, as it remains differentiable near − 1 and +1 allowing weights to change signs during training if necessary.
Revised: As depicted in Figure 2, our proposed SignSwish activation alleviates this, as it remains differentiable near − 1 and +1 allowing weights to change signs during training if necessary.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Note that the derivative ddx SS β ( x ) is zero at two points, controlled by β .
Revised: Note that the derivative ddx SS β ( x ) is zero at two points, controlled by β .

Original: Indeed, it is simple to show that the derivative is zero for x ≈ ± 2 .
Revised: Indeed, it is simple to show that the derivative is zero for x ≈ ± 2 .

Original: 4 /β .
Revised: 4 /β .

Original: By adjusting this parameter beta, it is possible to adjust the location at which the gradients start saturating.
Revised: By adjusting this parameter beta, it is possible to adjust the location at which the gradients start saturating.

Original: In contrast to the STE estimators, where it is ﬁxed.
Revised: In contrast to the STE estimators, where it is ﬁxed.

Original: Thus, the larger β is, the closer the approximation is to the derivative of the sign function.
Revised: Thus, the larger β is, the closer the approximation is to the derivative of the sign function.

Original: 3.2. R EGULARIZATION FUNCTION
Revised: 3.2. R EGULARIZATION FUNCTION

Original: In general, a regularization term is added to a model to prevent over-ﬁtting and to obtain robust generalization.
Revised: In general, a regularization term is added to a model to prevent over-ﬁtting and to obtain robust generalization.

Original: The two most commonly used regularization terms are L 1 and L 2 norms.
Revised: The two most commonly used regularization terms are L 1 and L 2 norms.

Original: If one were to embed these regularization functions in binary training, it would encourage the weights to be near zero; though this does not align with the objective of a binary network.
Revised: If one were to embed these regularization functions in binary training, it would encourage the weights to be near zero; though this does not align with the objective of a binary network.

Original: Instead, it is important to deﬁne a function that encourages the weights around − 1 and +1 .
Revised: Instead, it is important to deﬁne a function that encourages the weights around − 1 and +1 .

Original: Further, in Rastegari et al.
Revised: Further, in Rastegari et al.

Original: (2016) they present a scale to enhance the performance of binary networks.
Revised: (2016) they present a scale to enhance the performance of binary networks.

Original: This scale is computed dynamically during training, using the statistics of the weights.
Revised: This scale is computed dynamically during training, using the statistics of the weights.

Original: To make the regularization term more general we introduce scaling factors α , resulting in a symmetric regularization function with two minimums, one at − α and another at + α .
Revised: To make the regularization term more general we introduce scaling factors α , resulting in a symmetric regularization function with two minimums, one at − α and another at + α .

Original: As these scales are introduced in the regularization function and are embedded into the layers of the network they can be learned using backpropagation.
Revised: As these scales are introduced in the regularization function and are embedded into the layers of the network they can be learned using backpropagation.

Original: The Manhattan regularization function is deﬁned as
Revised: The Manhattan regularization function is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: whereas the Euclidean version is deﬁned as
Revised: whereas the Euclidean version is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where α > 0 is the scaling factor.
Revised: where α > 0 is the scaling factor.

Original: As depicted in Figure 3, in the case of α = 1 the weights are penalized at varying degrees upon moving away from the objective quantization values, in this case, {− 1 , +1 } .
Revised: As depicted in Figure 3, in the case of α = 1 the weights are penalized at varying degrees upon moving away from the objective quantization values, in this case, {− 1 , +1 } .

Original: The proposed regularizing terms are inline with the wisdom of the regularization function R ( w ) =
Revised: The proposed regularizing terms are inline with the wisdom of the regularization function R ( w ) =

Original: (1 − w 2 ) 1 {| w |≤ 1 } as introduced in Tang et al.
Revised: (1 − w 2 ) 1 {| w |≤ 1 } as introduced in Tang et al.

Original: A primary difference are in introducing a trainable scaling factor, and formulating it such that the gradients capture appropriate sign updates to the weights.
Revised: A primary difference are in introducing a trainable scaling factor, and formulating it such that the gradients capture appropriate sign updates to the weights.

Original: Further, the regularization introduced in Tang et al.
Revised: Further, the regularization introduced in Tang et al.

Original: (2017) does not penalize weights that are outside of [ − 1 , +1] .
Revised: (2017) does not penalize weights that are outside of [ − 1 , +1] .

Original: One can re-deﬁne their function to include a scaling factor as R ( w ) =
Revised: One can re-deﬁne their function to include a scaling factor as R ( w ) =

Original: ( α − w 2 ) 1 {| w |≤ α } .
Revised: ( α − w 2 ) 1 {| w |≤ α } .

Original: In Figure 3, we depict the different regularization terms to help with intuition.
Revised: In Figure 3, we depict the different regularization terms to help with intuition.

Original: [Figure]
Revised: [Figure]

Original: 3. BNN+: T RAINING M ETHOD
Revised: 3. BNN+: T RAINING M ETHOD

Original: Combining both the regularization and activation ideas, we modify the training procedure by replacing the sign backward approximation with that of the derivative of SS β activation (2).
Revised: Combining both the regularization and activation ideas, we modify the training procedure by replacing the sign backward approximation with that of the derivative of SS β activation (2).

Original: During training, the real weights are no longer clipped as in BNN training, as the network can back-propagate through the SS β activation and update the weights correspondingly.
Revised: During training, the real weights are no longer clipped as in BNN training, as the network can back-propagate through the SS β activation and update the weights correspondingly.

Original: Additional scales are introduced to the network, which multiplies into the weights of the layers.
Revised: Additional scales are introduced to the network, which multiplies into the weights of the layers.

Original: The regularization terms introduced are then added to the total loss function,
Revised: The regularization terms introduced are then added to the total loss function,

Original: [Equation]
Revised: [Equation]

Original: where L ( W, b ) is the cost function, W and b are the sets of all weights and biases in the network, W l is the set weights at layer l and α l is the corresponding scaling factor.
Revised: where L ( W, b ) is the cost function, W and b are the sets of all weights and biases in the network, W l is the set weights at layer l and α l is the corresponding scaling factor.

Original: Here, R ( . ) is the regularization function (4) or (5).
Revised: Here, R ( . ) is the regularization function (4) or (5).

Original: Further, λ controls the effect of the regularization term.
Revised: Further, λ controls the effect of the regularization term.

Original: To introduce meaningful scales, they are added to the basic blocks composing a typical convolutional neural network.
Revised: To introduce meaningful scales, they are added to the basic blocks composing a typical convolutional neural network.

Original: For example, for convolutions, the scale is multiplied into the quantized weights prior to the convolution operation.
Revised: For example, for convolutions, the scale is multiplied into the quantized weights prior to the convolution operation.

Original: Similarly, in a linear layer, the scales are multiplied into the quantized weights prior to the dot product operation.
Revised: Similarly, in a linear layer, the scales are multiplied into the quantized weights prior to the dot product operation.

Original: This is made more clear in the training algorithm 1.
Revised: This is made more clear in the training algorithm 1.

Original: The scale α is a single scalar per layer, or as proposed in Rastegari et al.
Revised: The scale α is a single scalar per layer, or as proposed in Rastegari et al.

Original: (2016) is a scalar for each ﬁlter in a convolutional layer.
Revised: (2016) is a scalar for each ﬁlter in a convolutional layer.

Original: For example, given a CNN block with weight dimensionality ( C in , C out , H, W ) , where C in is the number of input channels, C out is the number of output channels, and H , W , the height and width of the ﬁlter respectively, then the scale parameter would be a vector of dimension C out , that factors into each ﬁlter.
Revised: For example, given a CNN block with weight dimensionality ( C in , C out , H, W ) , where C in is the number of input channels, C out is the number of output channels, and H , W , the height and width of the ﬁlter respectively, then the scale parameter would be a vector of dimension C out , that factors into each ﬁlter.

Original: As the scales are learned jointly with the network through backpropagation, it is important to initialize them appropriately.
Revised: As the scales are learned jointly with the network through backpropagation, it is important to initialize them appropriately.

Original: In the case of the Manhattan penalizing term (4), given a scale factor α and weight ﬁlter then the objective is to minimize
Revised: In the case of the Manhattan penalizing term (4), given a scale factor α and weight ﬁlter then the objective is to minimize

Original: [Equation]
Revised: [Equation]

Original: The minimum of the above is obtained when
Revised: The minimum of the above is obtained when

Original: [Equation]
Revised: [Equation]

Original: Similarly, in the case of the Euclidean penalty (5) the minimum is obtained when
Revised: Similarly, in the case of the Euclidean penalty (5) the minimum is obtained when

Original: [Equation]
Revised: [Equation]

Original: The scales are initialized with the corresponding optimal values after weights have been initialized ﬁrst.
Revised: The scales are initialized with the corresponding optimal values after weights have been initialized ﬁrst.

Original: One may notice the similarity of these optimal values with that deﬁned by Rastegari et al.
Revised: One may notice the similarity of these optimal values with that deﬁned by Rastegari et al.

Original: (2016), whereas in their case the optimal value for the weight ﬁlters and activations better matches the R 2 ( w ) goal.
Revised: (2016), whereas in their case the optimal value for the weight ﬁlters and activations better matches the R 2 ( w ) goal.

Original: A difference is on how these approximations are computed, in our case they are updated on the backward pass, as opposed to computing the values dynamically.
Revised: A difference is on how these approximations are computed, in our case they are updated on the backward pass, as opposed to computing the values dynamically.

Original: The ﬁnal resulting BNN+ training method is deﬁned in Algorithm 1.
Revised: The ﬁnal resulting BNN+ training method is deﬁned in Algorithm 1.

Original: In the following section, we present our experimental results and important training details.
Revised: In the following section, we present our experimental results and important training details.

Original: Algorithm 1 BNN+ training.
Revised: Algorithm 1 BNN+ training.

Original: L is the unregularized loss function.
Revised: L is the unregularized loss function.

Original: λ and R 1 are the regularization terms we introduced.
Revised: λ and R 1 are the regularization terms we introduced.

Original: SS β is the SignSwish function we introduced and (SS β ) (cid:48) is its derivative.
Revised: SS β is the SignSwish function we introduced and (SS β ) (cid:48) is its derivative.

Original: N is the number of layers.
Revised: N is the number of layers.

Original: ◦ indicates element-wise multiplication.
Revised: ◦ indicates element-wise multiplication.

Original: BatchNorm() speciﬁes how to batchnormalize the activation and BackBatchNorm() how to back-propagate through the normalization.
Revised: BatchNorm() speciﬁes how to batchnormalize the activation and BackBatchNorm() how to back-propagate through the normalization.

Original: ADAM() speciﬁes how to update the parameters when their gradients are known.
Revised: ADAM() speciﬁes how to update the parameters when their gradients are known.

Original: Require: a minibatch of inputs and targets ( x 0 , x ∗ ) , previous weights W , previous weights’ scaling factors α , and previous BatchNorm parameters θ .
Revised: Require: a minibatch of inputs and targets ( x 0 , x ∗ ) , previous weights W , previous weights’ scaling factors α , and previous BatchNorm parameters θ .

Original: Ensure: updated weights W t +1 , updated weights’ scaling factors α t +1 and updated BatchNorm parameters θ t +1 .
Revised: Ensure: updated weights W t +1 , updated weights’ scaling factors α t +1 and updated BatchNorm parameters θ t +1 .

Original: Forward propagation: } s 0 ← x 0 W 0 { We do not quantize the ﬁrst layer.
Revised: Forward propagation: } s 0 ← x 0 W 0 { We do not quantize the ﬁrst layer.

Original: } x 1 ← BatchNorm( s 0 , θ 0 ) for k = 1 to N − 1 do x bk ← sign( x k )
Revised: } x 1 ← BatchNorm( s 0 , θ 0 ) for k = 1 to N − 1 do x bk ← sign( x k )

Original: W bk ← sign( W k ) s k
Revised: W bk ← sign( W k ) s k

Original: ← α k x bk W bk {
Revised: ← α k x bk W bk {

Original: This step can be done using mostly bitwise operations.
Revised: This step can be done using mostly bitwise operations.

Original: } x k +1 ← BatchNorm( s k , θ k ) end for { 2.
Revised: } x k +1 ← BatchNorm( s k , θ k ) end for { 2.

Original: Backward propagation: }
Revised: Backward propagation: }

Original: Compute g x N = ∂L∂x N knowing x N and x ∗ for k = N − 1 to 1 do ( g s k , g θ k ) ← BackBatchNorm( g x k +1 , s k , θ k )
Revised: Compute g x N = ∂L∂x N knowing x N and x ∗ for k = N − 1 to 1 do ( g s k , g θ k ) ← BackBatchNorm( g x k +1 , s k , θ k )

Original: g
Revised: g

Original: α k ← g s k x bk W bk +
Revised: α k ← g s k x bk W bk +

Original: λ
Revised: λ

Original: ∂R 1 ∂α k g W bk ← g (cid:62) s k
Revised: ∂R 1 ∂α k g W bk ← g (cid:62) s k

Original: α k x bk g x bk ← g s k
Revised: α k x bk g x bk ← g s k

Original: α k W bk { We use our modiﬁed straight-through estimator to back-propagate through sign : } g W k ← g W b k ◦ (SS β ) (cid:48) ( W k )
Revised: α k W bk { We use our modiﬁed straight-through estimator to back-propagate through sign : } g W k ← g W b k ◦ (SS β ) (cid:48) ( W k )

Original: + λ ∂R 

Revised: + λ ∂R

Original: ∂W k g x k ← g x bk ◦ (SS β ) (cid:48) ( x k ) end for ( g s 0 , g θ 0 ) ← BackBatchNorm( g x 1 , s 0 , θ 0 )
Revised: ∂W k g x k ← g x bk ◦ (SS β ) (cid:48) ( x k ) end for ( g s 0 , g θ 0 ) ← BackBatchNorm( g x 1 , s 0 , θ 0 )

Original: g
Revised: g

Original: W 0 ← g s 0 x 0 { We did not quantize the ﬁrst layer.
Revised: W 0 ← g s 0 x 0 { We did not quantize the ﬁrst layer.

Original: The update: } for k = 0 to N − 1 do θ t +1 k , W t +1 k , α t +1 k ← ADAM( η, θ k , W k , α k , g θ k , g W k , g α k ) end for
Revised: The update: } for k = 0 to N − 1 do θ t +1 k , W t +1 k , α t +1 k ← ADAM( η, θ k , W k , α k , g θ k , g W k , g α k ) end for

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We evaluate our proposed method with the accuracy performance of training using BNN+ scheme versus other proposed binary networks, Hubara et al.
Revised: We evaluate our proposed method with the accuracy performance of training using BNN+ scheme versus other proposed binary networks, Hubara et al.

Original: ; Rastegari et al.
Revised: ; Rastegari et al.

Original: ; Tang et al.
Revised: ; Tang et al.

Original: We run our method on CIFAR-10 and ImageNet datasets and show accuracy gains.
Revised: We run our method on CIFAR-10 and ImageNet datasets and show accuracy gains.

Original: They are discussed in their respective sections below.
Revised: They are discussed in their respective sections below.

Original: 4. CIFAR- 
Revised: 4. CIFAR-

Original: The CIFAR-10 data (Krizhevsky & Hinton, 2009) consists of 50,000 train images and a test set of 10,000.
Revised: The CIFAR-10 data (Krizhevsky & Hinton, 2009) consists of 50,000 train images and a test set of 10,000.

Original: For pre-processing the images are padded by 4 pixels on each side and a random crop is taken.
Revised: For pre-processing the images are padded by 4 pixels on each side and a random crop is taken.

Original: We train both, AlexNet (Krizhevsky et al., 2012), and VGG (Simonyan & Zisserman, 2014) using the ADAM (Kingma & Ba, 2014) optimizer.
Revised: We train both, AlexNet (Krizhevsky et al., 2012), and VGG (Simonyan & Zisserman, 2014) using the ADAM (Kingma & Ba, 2014) optimizer.

Original: The architecture used for VGG is conv(256) → conv(256) → conv(512) → conv(512)
Revised: The architecture used for VGG is conv(256) → conv(256) → conv(512) → conv(512)

Original: → conv(1024) → conv(1024) → fc(1024) → fc(1024) where conv( · ) is a convolutional layer, and fc( · ) is a fully connected layer.
Revised: → conv(1024) → conv(1024) → fc(1024) → fc(1024) where conv( · ) is a convolutional layer, and fc( · ) is a fully connected layer.

Original: The standard 3 × 3 ﬁlters are used in each layer.
Revised: The standard 3 × 3 ﬁlters are used in each layer.

Original: We also add a batchnormalization layer (Ioffe & Szegedy, 2015) prior to activation.
Revised: We also add a batch normalization layer (Ioffe & Szegedy, 2015) prior to activation.

Original: For AlexNet, the architecture from Krizhevsky (2014) is used, and batch normalization layers are added prior to activations.
Revised: For AlexNet, the architecture from Krizhevsky (2014) is used, and batch normalization layers are added prior to activations.

Original: We use a batch size of 256 for training.
Revised: We use a batch size of 256 for training.

Original: Many learning rates were experimented with such as 0 . 1 , 0 . 03 , 0 . 001 , etc, and the initial learning rate for AlexNet was set to 10 − 3 , and 3 × 10 − 3 for VGG.
Revised: Many learning rates were experimented with such as 0 . 1 , 0 . 03 , 0 . 001 , etc, and the initial learning rate for AlexNet was set to 10 − 3 , and 3 × 10 − 3 for VGG.

Original: The learning rates are correspondingly reduced by a factor 10 every 10 epoch for 50 epochs.
Revised: The learning rates are correspondingly reduced by a factor 10 every 10 epoch for 50 epochs.

Original: We set the regularization parameter λ to 10 − 6 , and use the regularization term as deﬁned in (4).
Revised: We set the regularization parameter λ to 10 − 6 , and use the regularization term as deﬁned in (4).

Original: In these experiments weights are initialized using Glorot & Bengio (2010) initialization.
Revised: In these experiments weights are initialized using Glorot & Bengio (2010) initialization.

Original: Further, the scales are introduced for each convolution ﬁlter, and are initialized by sorting the absolute values of the weights for each ﬁlter and choosing the 75 th percentile value.
Revised: Further, the scales are introduced for each convolution ﬁlter, and are initialized by sorting the absolute values of the weights for each ﬁlter and choosing the 75 th percentile value.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: [Table]
Revised: [Table]

Original: 4. I MAGE N ET
Revised: 4. I MAGE N ET

Original: The ILSVRC-2012 dataset consists of ∼ 1.2M training images, and 1000 classes.
Revised: The ILSVRC-2012 dataset consists of ∼ 1.2M training images, and 1000 classes.

Original: For pre-processing the dataset we follow the typical augmentation: the images are resized to 256 × 256 , then are randomly cropped to 224 × 224 and the data is normalized using the mean and standard deviation statistics of the train inputs; no additional augmentation is done.
Revised: For pre-processing the dataset we follow the typical augmentation: the images are resized to 256 × 256 , then are randomly cropped to 224 × 224 and the data is normalized using the mean and standard deviation statistics of the train inputs; no additional augmentation is done.

Original: At inference time, the images are ﬁrst scaled to 256 × 256 , center cropped to 224 × 224 and then normalized.
Revised: At inference time, the images are ﬁrst scaled to 256 × 256 , center cropped to 224 × 224 and then normalized.

Original: We evaluate the performance of our training method on two architectures AlexNet
Revised: We evaluate the performance of our training method on two architectures AlexNet

Original: and Resnet- (He et al., 2016).
Revised: and Resnet- (He et al., 2016).

Original: Following previous work, we used batch-normalization before each activation function.
Revised: Following previous work, we used batch-normalization before each activation function.

Original: Additionally, we keep the ﬁrst and last layers to be in full precision, as we lose 2-3% accuracy otherwise.
Revised: Additionally, we keep the ﬁrst and last layers to be in full precision, as we lose 2-3% accuracy otherwise.

Original: This approach is followed by other binary methods that we compare to (Hubara et al., 2016;
Revised: This approach is followed by other binary methods that we compare to (Hubara et al., 2016;

Original: Rastegari et al., 2016; Tang et al., 2017).
Revised: Rastegari et al., 2016; Tang et al., 2017).

Original: The results are summarized in Table 2.
Revised: The results are summarized in Table 2.

Original: In all the experiments involving R 1 regularization we set the λ to 10 − 7 and R 2 regularization to 10 − 6 .
Revised: In all the experiments involving R 1 regularization we set the λ to 10 − 7 and R 2 regularization to 10 − 6 .

Original: Also, in every network, the scales are introduced per ﬁlter in convolutional layers, and per column in fully connected layers.
Revised: Also, in every network, the scales are introduced per ﬁlter in convolutional layers, and per column in fully connected layers.

Original: The weights are initialized using a pre-trained model with htan activation function as done in Liu et al.
Revised: The weights are initialized using a pre-trained model with htan activation function as done in Liu et al.

Original: Then the learning rate for AlexNet is set to 2 .
Revised: Then the learning rate for AlexNet is set to 2 .

Original: 33 × 10 − 3 and multiplied by 0.1 at the 12 th , 18 th epoch for a total of 25 epochs trained.
Revised: 33 × 10 − 3 and multiplied by 0.1 at the 12 th , 18 th epoch for a total of 25 epochs trained.

Original: For the 18-layer ResNet the learning rate is started from 0 .
Revised: For the 18-layer ResNet the learning rate is started from 0 .

Original: 01 and multiplied by 0.1 at 10 th , 20 th , 30 th epoch.
Revised: 01 and multiplied by 0.1 at 10 th , 20 th , 30 th epoch.

Original: 4. D ISCUSSION
Revised: 4. D ISCUSSION

Original: We proposed two regularization terms (4) and (5) and an activation term (2) with a trainable parameter β .
Revised: We proposed two regularization terms (4) and (5) and an activation term (2) with a trainable parameter β .

Original: We run several experiments to better understand the effect of the different modiﬁcations to the training method, especially using different regularization and asymptote parameters β .
Revised: We run several experiments to better understand the effect of the different modiﬁcations to the training method, especially using different regularization and asymptote parameters β .

Original: The parameter β is trainable and would add one equation through back-propagation.
Revised: The parameter β is trainable and would add one equation through back-propagation.

Original: However, we ﬁxed β throughout our experiments to explicit values.
Revised: However, we ﬁxed β throughout our experiments to explicit values.

Original: The results are summarized in Table 2.
Revised: The results are summarized in Table 2.

Original: Through our experiments, we found that adding regularizing term with heavy penalization degrades the networks ability to converge, as the term would result in total loss be largely due to the regu-
Revised: Through our experiments, we found that adding regularizing term with heavy penalization degrades the networks ability to converge, as the term would result in total loss be largely due to the regu-

Original: [Table]
Revised: [Table]

Original: larizing term and not the target cross entropy loss.
Revised: larizing term and not the target cross entropy loss.

Original: Similarly, the regularizing term was set to small values in Tang et al.
Revised: Similarly, the regularizing term was set to small values in Tang et al.

Original: As a result, we set λ with a reasonably small value 10 − 5 − 10 − 7 , so that the scales move slowly as the weights gradually converge to stable values.
Revised: As a result, we set λ with a reasonably small value 10 − 5 − 10 − 7 , so that the scales move slowly as the weights gradually converge to stable values.

Original: Some preliminary experimentation was to gradually increase the regularization with respect to batch iterations updates done in training, though this approach requires careful tuning and was not pursued further.
Revised: Some preliminary experimentation was to gradually increase the regularization with respect to batch iterations updates done in training, though this approach requires careful tuning and was not pursued further.

Original: From Table 2, and referring to networks without regularization, we see the beneﬁt of using SwishSign approximation versus the STE.
Revised: From Table 2, and referring to networks without regularization, we see the beneﬁt of using SwishSign approximation versus the STE.

Original: This was also noted in Liu et al.
Revised: This was also noted in Liu et al.

Original: (2018), where their second approximation provided better results.
Revised: (2018), where their second approximation provided better results.

Original: There is not much difference between using R 1 versus Rtowards model generalization although since the loss metric used was the cross-entropy loss, the order of R 1 better matches the loss metric.
Revised: There is not much difference between using R 1 versus Rtowards model generalization although since the loss metric used was the cross-entropy loss, the order of R 1 better matches the loss metric.

Original: Lastly, it seems moderate values of β is better than small or large values.
Revised: Lastly, it seems moderate values of β is better than small or large values.

Original: Intuitively, this happens because for small values of β , the gradient approximation is not good enough and as β increases the gradients become too large, hence small noise could cause large ﬂuctuations in the sign of the weights.
Revised: Intuitively, this happens because for small values of β , the gradient approximation is not good enough and as β increases the gradients become too large, hence small noise could cause large ﬂuctuations in the sign of the weights.

Original: We did not compare our network with that of Liu et al.
Revised: We did not compare our network with that of Liu et al.

Original: (2018) as they introduce a shortcut connection that proves to help even the full precision network.
Revised: (2018) as they introduce a shortcut connection that proves to help even the full precision network.

Original: As a ﬁnal remark, we note that the learning rate is of great importance and properly tuning this is required to achieve convergence.
Revised: As a ﬁnal remark, we note that the learning rate is of great importance and properly tuning this is required to achieve convergence.

Original: Table 3 summarizes the best results of the ablation study and compares with BinaryNet, XNOR-Net, and ABC-Net.
Revised: Table 3 summarizes the best results of the ablation study and compares with BinaryNet, XNOR-Net, and ABC-Net.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: To summarize we propose three incremental ideas that help binary training: i) adding a regularizer to the objective function of the network, ii) trainable scale factors that are embedded in the regularizing term and iii) an improved approximation to the derivative of the sign activation function.
Revised: To summarize we propose three incremental ideas that help binary training: i) adding a regularizer to the objective function of the network, ii) trainable scale factors that are embedded in the regularizing term and iii) an improved approximation to the derivative of the sign activation function.

Original: We obtain competitive results by training AlexNet and Resnet-18 on the ImageNet dataset.
Revised: We obtain competitive results by training AlexNet and Resnet-18 on the ImageNet dataset.

Original: For future work, we plan on extending these to efﬁcient models such as CondenseNet (Huang et al., 2018), MobileNets (Howard et al., 2017), MnasNet (Tan et al., 2018) and on object recognition tasks.
Revised: For future work, we plan on extending these to efﬁcient models such as CondenseNet (Huang et al., 2018), MobileNets (Howard et al., 2017), MnasNet (Tan et al., 2018) and on object recognition tasks.