Original: T RAINING WIDE RESIDUAL NETWORKS FOR DEPLOYMENT USING A SINGLE BIT FOR EACH WEIGHT
Revised: T RAINING WIDE RESIDUAL NETWORKS FOR DEPLOYMENT USING A SINGLE BIT FOR EACH WEIGHT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: For fast and energy-efﬁcient deployment of trained deep neural networks on resource-constrained embedded hardware, each learnt weight parameter should ideally be represented and stored using a single bit.
Revised: For fast and energy-efﬁcient deployment of trained deep neural networks on resource-constrained embedded hardware, each learnt weight parameter should ideally be represented and stored using a single bit.

Original: Error-rates usually increase when this requirement is imposed.
Revised: Error-rates usually increase when this requirement is imposed.

Original: Here, we report methodological innovations that result in large reductions in error rates for deep convolutional neural networks deployed using a single bit for each weight.
Revised: Here, we report methodological innovations that result in large reductions in error rates for deep convolutional neural networks deployed using a single bit for each weight.

Original: For 1-bit weight models with just 20 weight layers, requiring only ∼ 4 MB of parameter memory, for CIFAR-10,
Revised: For 1-bit weight models with just 20 weight layers, requiring only ∼ 4 MB of parameter memory, for CIFAR-10,

Original: CIFAR-100, SVHN and Imagenet32 (Top-5) we achieve error rates of 3.74%, 18.41%, 1.93%, and 21.89% respectively.
Revised: CIFAR-100, SVHN and Imagenet32 (Top-5) we achieve error rates of 3.74%, 18.41%, 1.93%, and 21.89% respectively.

Original: These error rates are about half those of previous reported error rates for CIFAR-10/100, and are within 1–3% of our error-rates for the same network with full-precision weights.
Revised: These error rates are about half those of previous reported error rates for CIFAR-10/100, and are within 1–3% of our error-rates for the same network with full-precision weights.

Original: Using a warm-restart learning-rate schedule, we found training for single-bit weights just as fast as fullprecision networks, and achieved about 98%-99% of peak performance in justtraining epochs for CIFAR 10/100.
Revised: Using a warm-restart learning-rate schedule, we found training for single-bit weights just as fast as fullprecision networks, and achieved about 98%-99% of peak performance in justtraining epochs for CIFAR 10/100.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Fast parallel computing resources, namely GPUs, have been integral to the resurgence of deep neural networks, and their ascendancy to becoming state-of-the-art methodologies for many computer vision tasks.
Revised: Fast parallel computing resources, namely GPUs, have been integral to the resurgence of deep neural networks, and their ascendancy to becoming state-of-the-art methodologies for many computer vision tasks.

Original: However, GPUs are expensive in terms of their energy requirements.
Revised: However, GPUs are expensive in terms of their energy requirements.

Original: They typically compute using single-precision ﬂoating point (32 bits), which has now been recognised as providing far more precision than needed for deep neural networks.
Revised: They typically compute using single-precision ﬂoating point (32 bits), which has now been recognised as providing far more precision than needed for deep neural networks.

Original: Moreover, training and deployment can require the availability of large amounts of memory, both for storage of trained models, and for operational RAM.
Revised: Moreover, training and deployment can require the availability of large amounts of memory, both for storage of trained models, and for operational RAM.

Original: If deep-learning methods are to become embedded in resource-constrained sensors, devices and intelligent systems, ranging from robotics to the internet-of-things to self-driving cars, reliance on high-end computing resources will need to be reduced.
Revised: If deep-learning methods are to become embedded in resource-constrained sensors, devices and intelligent systems, ranging from robotics to the internet-of-things to self-driving cars, reliance on high-end computing resources will need to be reduced.

Original: To this end, there has been increasing interest in ﬁnding methods to drive down the resource burden of modern deep neural networks.
Revised: To this end, there has been increasing interest in ﬁnding methods to drive down the resource burden of modern deep neural networks.

Original: Existing methods typically exhibit good performance but for the ideal case of single-bit parameters and/or processing, still fall well-short of state-of-the-art error rates on known benchmarks.
Revised: Existing methods typically exhibit good performance but for the ideal case of single-bit parameters and/or processing, still fall well-short of state-of-the-art error rates on known benchmarks.

Original: In this paper, we detail methods that resulted in a signiﬁcant reduction in the gap between Convolutional Neural Networks (CNNs) deployed using weights stored and applied using standard precision (32-bit ﬂoating point) and networks deployed using weights represented by a single-bit each 1 .
Revised: In this paper, we detail methods that resulted in a signiﬁcant reduction in the gap between Convolutional Neural Networks (CNNs) deployed using weights stored and applied using standard precision (32-bit ﬂoating point) and networks deployed using weights represented by a single-bit each 1 .

Original: In the process of developing our methods, we also obtained a small improvement in error-rates obtained by standard full-precision versions of the CNNs we used.
Revised: In the process of developing our methods, we also obtained a small improvement in error-rates obtained by standard full-precision versions of the CNNs we used.

Original: 1. R ELATED W ORK 1.1. R ES N ETS
Revised: 1. R ELATED W ORK 1.1. R ES N ETS

Original: In 2015, a new form of CNN called a “deep residual network,” or “ResNet” (He et al., 2015b) was developed and used to set many new accuracy records on computer-vision benchmarks.
Revised: In 2015, a new form of CNN called a “deep residual network,” or “ResNet” (He et al., 2015b) was developed and used to set many new accuracy records on computer-vision benchmarks.

Original: In comparison with older CNNs such as Alexnet (Krizhevsky et al., 2012) and VGGnet (Simonyan & Zisserman, 2014), ResNets achieve higher accuracy with far fewer learnt parameters and FLOPs (FLoating-point OPerations) per image processed.
Revised: In comparison with older CNNs such as Alexnet (Krizhevsky et al., 2012) and VGGnet (Simonyan & Zisserman, 2014), ResNets achieve higher accuracy with far fewer learnt parameters and FLOPs (FLoating-point OPerations) per image processed.

Original: The key to reducing the number of parameters in ResNets was to replace “all-to-all layers” in VGG-like nets with “global-average-pooling” layers that have no learnt parameters (Lin et al., 2013; Springenberg et al., 2014), while simultaneously training a much deeper network than previously.
Revised: The key to reducing the number of parameters in ResNets was to replace “all-to-all layers” in VGG-like nets with “global-average-pooling” layers that have no learnt parameters (Lin et al., 2013; Springenberg et al., 2014), while simultaneously training a much deeper network than previously.

Original: The key new idea that enabled a deeper network to be trained effectively was the introduction of so-called “skip-connections” (He et al., 2015b).
Revised: The key new idea that enabled a deeper network to be trained effectively was the introduction of so-called “skip-connections” (He et al., 2015b).

Original: Many variations of ResNets have since been proposed.
Revised: Many variations of ResNets have since been proposed.

Original: ResNets offer the virtue of simplicity, and given the motivation for deployment in custom hardware, we have chosen them as our primary focus.
Revised: ResNets offer the virtue of simplicity, and given the motivation for deployment in custom hardware, we have chosen them as our primary focus.

Original: Despite the increased efﬁciency in parameter usage, as with other CNNs, accuracy in ResNets still tends to increase with the total number of parameters; unlike other CNNs, increased accuracy can result either from deeper (He et al., 2016) or wider networks (Zagoruyko & Komodakis, 2016).
Revised: Despite the increased efﬁciency in parameter usage, as with other CNNs, accuracy in ResNets still tends to increase with the total number of parameters; unlike other CNNs, increased accuracy can result either from deeper (He et al., 2016) or wider networks (Zagoruyko & Komodakis, 2016).

Original: In this paper, we use Wide Residual Networks (Zagoruyko & Komodakis, 2016), as they have been demonstrated to produce better accuracy in less training time than deeper networks.
Revised: In this paper, we use Wide Residual Networks (Zagoruyko & Komodakis, 2016), as they have been demonstrated to produce better accuracy in less training time than deeper networks.

Original: 1.1. R EDUCING THE MEMORY BURDEN OF TRAINED NEURAL NETWORKS
Revised: 1.1. R EDUCING THE MEMORY BURDEN OF TRAINED NEURAL NETWORKS

Original: Achieving the best accuracy and speed possible when deploying ResNets or similar networks on hardware-constrained mobile devices will require minimising the total number of bits transferred between memory and processors for a given number of parameters.
Revised: Achieving the best accuracy and speed possible when deploying ResNets or similar networks on hardware-constrained mobile devices will require minimising the total number of bits transferred between memory and processors for a given number of parameters.

Original: Motivated by such considerations, a lot of recent attention has been directed towards compressing the learnt parameters ( model compression and quantizing other representations in trained neural networks—see Hubara et al.
Revised: Motivated by such considerations, a lot of recent attention has been directed towards compressing the learnt parameters ( model compression and quantizing other representations in trained neural networks—see Hubara et al.

Original: (2016) for a more detailed literature review.
Revised: (2016) for a more detailed literature review.

Original: Recently published strategies for model compression include reducing the precision (number of bits used for numerical representation) of weights in deployed networks by doing the same during training (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), reducing the number of weights in trained neural networks by pruning (Iandola et al., 2016), quantizing weights following training (Zhou et al., 2017), reducing the precision of computations performed in forward-propagation during inference (Courbariaux et al., 2015; Hubara et al., 2016;
Revised: Recently published strategies for model compression include reducing the precision (number of bits used for numerical representation) of weights in deployed networks by doing the same during training (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), reducing the number of weights in trained neural networks by pruning (Iandola et al., 2016), quantizing weights following training (Zhou et al., 2017), reducing the precision of computations performed in forward-propagation during inference (Courbariaux et al., 2015; Hubara et al., 2016;

Original: Merolla et al., 2016; Rastegari et al., 2016), and modifying neural network architectures (Howard et al., 2017).
Revised: Merolla et al., 2016; Rastegari et al., 2016), and modifying neural network architectures (Howard et al., 2017).

Original: A theoretical analysis of various methods proved results on the convergence of a variety of weight-binarization methods (Li et al., 2017).
Revised: A theoretical analysis of various methods proved results on the convergence of a variety of weight-binarization methods (Li et al., 2017).

Original: From this range of strategies, we are focused on approaches that enable deployment of trained models immediately following training, for the simplicity this enables.
Revised: From this range of strategies, we are focused on approaches that enable deployment of trained models immediately following training, for the simplicity this enables.

Original: 1. O VERALL APPROACH AND SUMMARY OF CONTRIBUTIONS
Revised: 1. O VERALL APPROACH AND SUMMARY OF CONTRIBUTIONS

Original: Our strategy for improving methods that enable inference with single-bit weights was threefold:
Revised: Our strategy for improving methods that enable inference with single-bit weights was threefold:

Original: State-of-the-art baseline.
Revised: State-of-the-art baseline.

Original: We sought
Revised: We sought

Original: to
Revised: to

Original: begin
Revised: begin

Original: with a
Revised: with a

Original: baseline full-precision deep CNN variant
Revised: baseline full-precision deep CNN variant

Original: with close to state-
Revised: with close to state-

Original: of
Revised: of

Original: the
Revised: the

Original: art error
Revised: art error

Original: rates.
Revised: rates.

Original: At the time of commencement in 2016, the
Revised: At the time of commencement in 2016, the

Original: state
Revised: state

Original: of
Revised: of

Original: -the
Revised: -the

Original: -art on CIFAR-
Revised: -art on CIFAR-

Original: and CIFAR- was held by
Revised: and CIFAR- was held by

Original: Wide
Revised: Wide

Original: Residual
Revised: Residual

Original: Networks (Zagoruyko
Revised: Networks (Zagoruyko

Original: Komodakis, 2016)
Revised: Komodakis, 2016)

Original: , so this was our starting point.
Revised: , so this was our starting point.

Original: While subsequent approaches have exceeded slightly the error rates of wide ResNets (see Table 3), for hardware ResNets
Revised: While subsequent approaches have exceeded slightly the error rates of wide ResNets (see Table 3), for hardware ResNets

Original: offer superior simplicity,
Revised: offer superior simplicity,

Original: which
Revised: which

Original: conforms
Revised: conforms

Original: with our third
Revised: with our third

Original: strategy in this list.
Revised: strategy in this list.

Original: Make minimal changes when training for 1-bit weights.
Revised: Make minimal changes when training for 1-bit weights.

Original: We aimed to ensure that training for 1-bit weights could be achieved with minimal changes in comparison to training our baseline.
Revised: We aimed to ensure that training for 1-bit weights could be achieved with minimal changes in comparison to training our baseline.

Original: Simplicity is desirable in custom hardware.
Revised: Simplicity is desirable in custom hardware.

Original: In addition to training to enable single-bit weights at inference time, with custom hardware implementations in mind we sought to simplify the design of the baseline network as much as possible without sacriﬁcing accuracy.
Revised: In addition to training to enable single-bit weights at inference time, with custom hardware implementations in mind we sought to simplify the design of the baseline network as much as possible without sacriﬁcing accuracy.

Original: 1.2. C ONTRIBUTIONS TO F ULL -P RECISION W IDE R ES N ETS
Revised: 1.2. C ONTRIBUTIONS TO F ULL -P RECISION W IDE R ES N ETS

Original: Although this paper is chieﬂy about single-bit weights, we exceeded our objectives for the fullprecision baseline network, and surpassed reported error rates reported for CIFAR-10 and CIFAR-
Revised: Although this paper is chieﬂy about single-bit weights, we exceeded our objectives for the fullprecision baseline network, and surpassed reported error rates reported for CIFAR-10 and CIFAR-

Original: 100 using full-precision Wide ResNets (Zagoruyko & Komodakis, 2016), while not requiring the use of dropout regularization (see Table 3).
Revised: 100 using full-precision Wide ResNets (Zagoruyko & Komodakis, 2016), while not requiring the use of dropout regularization (see Table 3).

Original: This was achieved using just 20 convolutional layers; most prior work has demonstrated best wide ResNet performance using 28 convolutional layers.
Revised: This was achieved using just 20 convolutional layers; most prior work has demonstrated best wide ResNet performance using 28 convolutional layers.

Original: All but one of our modiﬁcations to standard Wide ResNets are mainly to achieve simplicity, as in the second and third prongs of our strategy.
Revised: All but one of our modiﬁcations to standard Wide ResNets are mainly to achieve simplicity, as in the second and third prongs of our strategy.

Original: Collectively these minor modiﬁcations (see the Appendix) provide error-rate reductions of ∼ 1% on CIFAR-10/100 compared to not using them.
Revised: Collectively these minor modiﬁcations (see the Appendix) provide error-rate reductions of ∼ 1% on CIFAR-10/100 compared to not using them.

Original: However, one innovation we introduce achieves a signiﬁcant error-rate advantage for CIFAR-10 and
Revised: However, one innovation we introduce achieves a signiﬁcant error-rate advantage for CIFAR-10 and

Original: CIFAR-100 in Wide ResNets.
Revised: CIFAR-100 in Wide ResNets.

Original: This is to simply not learn the per-channel scale and offset factors in the batch-normalization layers, while retaining the remaining attributes of these layers.
Revised: This is to simply not learn the per-channel scale and offset factors in the batch-normalization layers, while retaining the remaining attributes of these layers.

Original: This is done in conjunction with exchanging the ordering of the ﬁnal weight layer and the global average pooling layer.
Revised: This is done in conjunction with exchanging the ordering of the ﬁnal weight layer and the global average pooling layer.

Original: We present a case for why these changes should be advantageous, in the Discussion section.
Revised: We present a case for why these changes should be advantageous, in the Discussion section.

Original: We observed this effect to be more pronounced for CIFAR-100 than CIFAR-10, gaining over 1% in test-error rate.
Revised: We observed this effect to be more pronounced for CIFAR-100 than CIFAR-10, gaining over 1% in test-error rate.

Original: But the method is advantageous only when training tended to cause overﬁtting; when overﬁtting is not an issue, such as for Imagenet, removing learning of batch-norm parameters is only detrimental.
Revised: But the method is advantageous only when training tended to cause overﬁtting; when overﬁtting is not an issue, such as for Imagenet, removing learning of batch-norm parameters is only detrimental.

Original: 1.2. C ONTRIBUTIONS TO DEEP CNN S WITH SINGLE - BIT WEIGHTS FOR INFERENCE
Revised: 1.2. C ONTRIBUTIONS TO DEEP CNN S WITH SINGLE - BIT WEIGHTS FOR INFERENCE

Original: Our approach surpasses by a large margin all previously reported error rates for CIFAR-10/ (error rates halved) and SVHN for networks constrained to run with single-bit weights at inference time.
Revised: Our approach surpasses by a large margin all previously reported error rates for CIFAR-10/ (error rates halved) and SVHN for networks constrained to run with single-bit weights at inference time.

Original: We found no previous results for single-bit weights for Imagenet32 (Chrabaszcz et al., 2017), but our results on this dataset are also indicative of the general applicability of our methods (see
Revised: We found no previous results for single-bit weights for Imagenet32 (Chrabaszcz et al., 2017), but our results on this dataset are also indicative of the general applicability of our methods (see

Original: Section 5.1).
Revised: Section 5.1).

Original: One reason we have achieved lower error rates for the 1-bit case than previously is to start with a superior baseline network than in previous studies, namely wide ResNets.
Revised: One reason we have achieved lower error rates for the 1-bit case than previously is to start with a superior baseline network than in previous studies, namely wide ResNets.

Original: However, as we show in
Revised: However, as we show in

Original: Section 5.1, our approach results in smaller error rate increases relative to full-precision error rates than previously.
Revised: Section 5.1, our approach results in smaller error rate increases relative to full-precision error rates than previously.

Original: Moreover, previous modiﬁcations of training to use 1-bit weights has tended to result in slow convergence requiring training for many 100s of epochs.
Revised: Moreover, previous modiﬁcations of training to use 1-bit weights has tended to result in slow convergence requiring training for many 100s of epochs.

Original: Our contribution is to introduce a simple ﬁxed scaling method for each convolutional layer, that permits activations and gradients to ﬂow through the network with minimum change in standard deviation, in accordance with the principle underlying popular initialization methods (He et al., 2015a).
Revised: Our contribution is to introduce a simple ﬁxed scaling method for each convolutional layer, that permits activations and gradients to ﬂow through the network with minimum change in standard deviation, in accordance with the principle underlying popular initialization methods (He et al., 2015a).

Original: We combine this with the use of a warm-restart learning-rate method (Loshchilov & Hutter, 2016) that enables us to report close-to-baseline results for the 1-bit case in far fewer epochs of training than reported previously.
Revised: We combine this with the use of a warm-restart learning-rate method (Loshchilov & Hutter, 2016) that enables us to report close-to-baseline results for the 1-bit case in far fewer epochs of training than reported previously.

Original:  L EARNING A MODEL WITH CONVOLUTION WEIGHTS ±2. T HE SIGN OF WEIGHTS PROPAGATE , BUT FULL - PRECISION WEIGHTS ARE UPDATED
Revised:  L EARNING A MODEL WITH CONVOLUTION WEIGHTS ±2. T HE SIGN OF WEIGHTS PROPAGATE , BUT FULL - PRECISION WEIGHTS ARE UPDATED

Original: We follow the approach of Courbariaux et al.
Revised: We follow the approach of Courbariaux et al.

Original: ; Rastegari et al.
Revised: ; Rastegari et al.

Original: ; Merolla et al.
Revised: ; Merolla et al.

Original: (2016), in that
Revised: (2016), in that

Original: we ﬁnd good results when using single-bit weights at inference time if during training we apply the sign function to real-valued weights for the purpose of forward and backward propagation, but update full-precision weights using SGD with gradients calculated using full-precision.
Revised: we ﬁnd good results when using single-bit weights at inference time if during training we apply the sign function to real-valued weights for the purpose of forward and backward propagation, but update full-precision weights using SGD with gradients calculated using full-precision.

Original: However, previously reported methods for training using the sign of weights either need to train for many hundreds of epochs (Courbariaux et al., 2015; Merolla et al., 2016), or use computationallycostly normalization scaling for each channel in each layer that changes for each minibatch during training (Rastegari et al., 2016).
Revised: However, previously reported methods for training using the sign of weights either need to train for many hundreds of epochs (Courbariaux et al., 2015; Merolla et al., 2016), or use computationallycostly normalization scaling for each channel in each layer that changes for each minibatch during training (Rastegari et al., 2016).

Original: We ﬁnd it is possible to dramatically accelerate convergence and execution time, while reducing error rates, using a simple alternative approach, as we now describe.
Revised: We ﬁnd it is possible to dramatically accelerate convergence and execution time, while reducing error rates, using a simple alternative approach, as we now describe.

Original: 2.1. W E SCALE THE OUTPUT OF CONV LAYERS USING A CONSTANT FOR EACH LAYER
Revised: 2.1. W E SCALE THE OUTPUT OF CONV LAYERS USING A CONSTANT FOR EACH LAYER

Original: We begin by noting that the standard deviation of the sign of the weights in a convolutional layer with kernels of size F × F will be close to 1 , assuming a mean of zero.
Revised: We begin by noting that the standard deviation of the sign of the weights in a convolutional layer with kernels of size F × F will be close to 1 , assuming a mean of zero.

Original: In contrast, the standard deviation of layer i in full-precision networks is initialised in the method of He et al.
Revised: In contrast, the standard deviation of layer i in full-precision networks is initialised in the method of He et al.

Original: (2015a)
Revised: (2015a)

Original: to
Revised: to

Original: (cid:112) 2 / ( F 2 C i − 1 ) , where C i − 1 is the number of input channels to convolutional layer i , where i = 1 , .., L , L is the number of convolutional layers and C 

Revised: (cid:112) 2 / ( F 2 C i − 1 ) , where C i − 1 is the number of input channels to convolutional layer i , where i = 1 , .., L , L is the number of convolutional layers and C

Original: = 3 for RGB inputs.
Revised: = 3 for RGB inputs.

Original: Our strategy for a network trained using the sign of weights is to seek to make the propagation of weights and gradients statistically equivalent to those in the full precision network, so as to ensure the standard-deviation as information ﬂows through the network is close to the ideal situation that motivated the method of He et al.
Revised: Our strategy for a network trained using the sign of weights is to seek to make the propagation of weights and gradients statistically equivalent to those in the full precision network, so as to ensure the standard-deviation as information ﬂows through the network is close to the ideal situation that motivated the method of He et al.

Original: (2015a).
Revised: (2015a).

Original: When applying the sign function alone, there is a mismatch with the principled approach to controlling gradient and activation scaling through a deep network He et al.
Revised: When applying the sign function alone, there is a mismatch with the principled approach to controlling gradient and activation scaling through a deep network He et al.

Original: (2015a).
Revised: (2015a).

Original: Although the use of batch-normalization can still enable learning, convergence is empirically slow and less effective.
Revised: Although the use of batch-normalization can still enable learning, convergence is empirically slow and less effective.

Original: To address this problem, for training using the sign of weights, we use the initialisation method of He et al.
Revised: To address this problem, for training using the sign of weights, we use the initialisation method of He et al.

Original: (2015a)
Revised: (2015a)

Original: for the full-precision weights that are updated, but also introduce a scaling for the single-bit weights of each convolutional layer.
Revised: for the full-precision weights that are updated, but also introduce a scaling for the single-bit weights of each convolutional layer.

Original: This scaling has a constant un-learnt value equal to the initial standard deviation of (cid:112) 2 / ( F 2 C i − 1 ) of the method of He et al.
Revised: This scaling has a constant un-learnt value equal to the initial standard deviation of (cid:112) 2 / ( F 2 C i − 1 ) of the method of He et al.

Original: (2015a).
Revised: (2015a).

Original: This enables the standard deviation of forward-propagating information to be equal to the value it would have initially in full-precision networks.
Revised: This enables the standard deviation of forward-propagating information to be equal to the value it would have initially in full-precision networks.

Original: In implementation, during training we multiply the sign of the weights in each layer by this value.
Revised: In implementation, during training we multiply the sign of the weights in each layer by this value.

Original: For inference, we do this multiplication using a scaling layer following the weight layer, so that all weights in the network are stored using 0 and 1 , and deployed using ± 1 (see, e.g. https://github.com/SingleBitResNets/SingleBitResNets ).
Revised: For inference, we do this multiplication using a scaling layer following the weight layer, so that all weights in the network are stored using 0 and 1 , and deployed using ± 1 (see, e.g. https://github.com/SingleBitResNets/SingleBitResNets ).

Original: Hence, custom hardware implementations would be able to implement convolutions in our trained models without multipliers.
Revised: Hence, custom hardware implementations would be able to implement convolutions in our trained models without multipliers.

Original: The fact that we scale the weights explicitly during training is important.
Revised: The fact that we scale the weights explicitly during training is important.

Original: Although for forward and backward propagation it is equivalent to scale the input or output feature maps of a convolutional layer, doing so also scales the calculated gradients with respect to ﬁlter weights, since these are calculated by convolving input and output feature maps.
Revised: Although for forward and backward propagation it is equivalent to scale the input or output feature maps of a convolutional layer, doing so also scales the calculated gradients with respect to ﬁlter weights, since these are calculated by convolving input and output feature maps.

Original: As a consequence, learning is dramatically slower unless an additional adjustment is made to introduce layer-dependent learning rates to cancel out the scaling,
Revised: As a consequence, learning is dramatically slower unless an additional adjustment is made to introduce layer-dependent learning rates to cancel out the scaling,

Original: Our approach is similar to the BWN method of Rastegari et al.
Revised: Our approach is similar to the BWN method of Rastegari et al.

Original: but our constant scaling method differs in execution complexity, for two reasons.
Revised: but our constant scaling method differs in execution complexity, for two reasons.

Original: We also found we achieved better results.
Revised: We also found we achieved better results.

Original: These aspects are discussed in Section 5.2.
Revised: These aspects are discussed in Section 5.2.

Original: In summary, the only differences we make in comparison with full-precision training are as follows (see also Figure 1).
Revised: In summary, the only differences we make in comparison with full-precision training are as follows (see also Figure 1).

Original: Let W i be the tensor for the convolutional weights in the i –th convolutional layer.
Revised: Let W i be the tensor for the convolutional weights in the i –th convolutional layer.

Original: Then these weights are processed in the following way only for forward propagation and backward propagation, not for weight updates:
Revised: Then these weights are processed in the following way only for forward propagation and backward propagation, not for weight updates:

Original: [Equation]
Revised: [Equation]

Original: where F i is the spatial size of the convolutional kernel in layer i .
Revised: where F i is the spatial size of the convolutional kernel in layer i .

Original: [Figure]
Revised: [Figure]

Original:  M ETHODS COMMON TO BASELINE AND SINGLE - BIT NETWORKS 3. N ETWORK A RCHITECTURE
Revised:  M ETHODS COMMON TO BASELINE AND SINGLE - BIT NETWORKS 3. N ETWORK A RCHITECTURE

Original: Our ResNets use the ‘post-activation’ and identity mapping approach of He et al.
Revised: Our ResNets use the ‘post-activation’ and identity mapping approach of He et al.

Original: (2016) for residual connections.
Revised: (2016) for residual connections.

Original: Each residual block includes two convolutional layers, each preceeded by batch normalization (BN) and Rectiﬁed Linear Unit (ReLU) layers.
Revised: Each residual block includes two convolutional layers, each preceeded by batch normalization (BN) and Rectiﬁed Linear Unit (ReLU) layers.

Original: Rather than train very deep ResNets, we use wide residual networks (wide ResNets) (Zagoruyko & Komodakis, 2016) with exactlyconvolutional layers.
Revised: Rather than train very deep ResNets, we use wide residual networks (wide ResNets) (Zagoruyko & Komodakis, 2016) with exactlyconvolutional layers.

Original: Although Zagoruyko & Komodakis (2016) and others reported slightly better results on the CIFAR-10 dataset with 28 layers, after we made our other modiﬁcations we were able to match the performance of a 28 layer network with just 20 layers, and no additional width.
Revised: Although Zagoruyko & Komodakis (2016) and others reported slightly better results on the CIFAR-10 dataset with 28 layers, after we made our other modiﬁcations we were able to match the performance of a 28 layer network with just 20 layers, and no additional width.

Original: Our baseline ResNet design (see Figures 1 and 2) has several differences in comparison to those of He et al.
Revised: Our baseline ResNet design (see Figures 1 and 2) has several differences in comparison to those of He et al.

Original: ; Zagoruyko & Komodakis (2016).
Revised: ; Zagoruyko & Komodakis (2016).

Original: These details are articulated in the Appendix, and in combination enable us to obtain slightly improved error rates to comparison Wide ResNets.
Revised: These details are articulated in the Appendix, and in combination enable us to obtain slightly improved error rates to comparison Wide ResNets.

Original: [Figure]
Revised: [Figure]

Original: blocks (details are depicted in Figure 3) that reduce each spatial dimension in the feature map by a factor of 2, and simultaneously double the number of channels.
Revised: blocks (details are depicted in Figure 3) that reduce each spatial dimension in the feature map by a factor of 2, and simultaneously double the number of channels.

Original: “BN” denotes a batch-normalization layer, “ReLU” denotes a rectiﬁed-linear-unit layer, “GAP” denotes a global-average-pooling layer, and “SM” denotes a softmax layer.
Revised: “BN” denotes a batch-normalization layer, “ReLU” denotes a rectiﬁed-linear-unit layer, “GAP” denotes a global-average-pooling layer, and “SM” denotes a softmax layer.

Original: Our approach also works for deeper networks.
Revised: Our approach also works for deeper networks.

Original: [Figure]
Revised: [Figure]

Original: 3. T RAINING
Revised: 3. T RAINING

Original: We trained our models following, for most aspects, the standard stochastic gradient descent methods used by Zagoruyko & Komodakis (2016) for wide ResNets.
Revised: We trained our models following, for most aspects, the standard stochastic gradient descent methods used by Zagoruyko & Komodakis (2016) for wide ResNets.

Original: Speciﬁcally, we use cross-entropy loss, minibatches of size 125, momentum of 0 . 9 , and a weight decay of 0.0005.
Revised: Speciﬁcally, we use cross-entropy loss, minibatches of size 125, momentum of 0 . 9 , and a weight decay of 0.0005.

Original: Apart from one set of experiments where we added a simple extra approach called cutout (detailed below), we use standard ‘light’ data augmentation, including randomly ﬂipping each image horizontally with probability 0 .for CIFAR 10/100 and Imagenet32.
Revised: Apart from one set of experiments where we added a simple extra approach called cutout (detailed below), we use standard ‘light’ data augmentation, including randomly ﬂipping each image horizontally with probability 0 .for CIFAR 10/100 and Imagenet32.

Original: For the same 3 datasets plus SVHN, we pad by 4 pixels on all sides (using random values between 0 and 255) and crop a 32 × 32 patch from a random location in the resulting 40 × 40 image.
Revised: For the same 3 datasets plus SVHN, we pad by 4 pixels on all sides (using random values between 0 and 255) and crop a 32 × 32 patch from a random location in the resulting 40 × 40 image.

Original: We did not use any image-preprocessing, i.e. we did not subtract the mean, as the initial BN layer in the network learns an offset, and we did not use whitening or augment using color or brightness.
Revised: We did not use any image-preprocessing, i.e. we did not subtract the mean, as the initial BN layer in the network learns an offset, and we did not use whitening or augment using color or brightness.

Original: We use the initialization method of He et al.
Revised: We use the initialization method of He et al.

Original: (2015a).
Revised: (2015a).

Original: We now describe differences in our training approach compared to those usually reported.
Revised: We now describe differences in our training approach compared to those usually reported.

Original: 3.2. B ATCH - NORMALIZATION SCALES AND OFFSETS ARE NOT LEARNT FOR CIFAR-10/ 
Revised: 3.2. B ATCH - NORMALIZATION SCALES AND OFFSETS ARE NOT LEARNT FOR CIFAR-10/

Original: For reasons explained in Section 5.3, when training on CIFAR-10/100 and SVHN, in all batch-norm layers except the ﬁrst one at the input, we do not learn the scale and offset factor, instead initializating these to 1 and 0 in all channels, and keeping those values through training.
Revised: For reasons explained in Section 5.3, when training on CIFAR-10/100 and SVHN, in all batch-norm layers except the ﬁrst one at the input, we do not learn the scale and offset factor, instead initializating these to 1 and 0 in all channels, and keeping those values through training.

Original: Note that we also do not learn any biases for convolutional layers.
Revised: Note that we also do not learn any biases for convolutional layers.

Original: The usual approach to setting the moments for use in batch-norm layers in inference mode is to keep a running average through training.
Revised: The usual approach to setting the moments for use in batch-norm layers in inference mode is to keep a running average through training.

Original: When not learning batch-normalization parameters, we found a small beneﬁt in calculating the batch-normalisation moments used in inference only after training had ﬁnished.
Revised: When not learning batch-normalization parameters, we found a small beneﬁt in calculating the batch-normalisation moments used in inference only after training had ﬁnished.

Original: We simply form as many minibatches as possible from the full training-set, each with the same data augmentation as used during training applied, and pass these through the trained network, averaging the returned moments for each batch.
Revised: We simply form as many minibatches as possible from the full training-set, each with the same data augmentation as used during training applied, and pass these through the trained network, averaging the returned moments for each batch.

Original: 3.2. W E USE A WARM - RESTART LEARNING - RATE SCHEDULE
Revised: 3.2. W E USE A WARM - RESTART LEARNING - RATE SCHEDULE

Original: We use a ‘warm restarts’ learning rate schedule that has reported state-of-the-art wide residual network results (Loshchilov & Hutter, 2016) whilst also speeding up convergence.
Revised: We use a ‘warm restarts’ learning rate schedule that has reported state-of-the-art wide residual network results (Loshchilov & Hutter, 2016) whilst also speeding up convergence.

Original: The method constantly reduces the learning rate from 0.1 to 1 × 10 − 4 according to a cosine decay, across a certain number of epochs, and then repeats across twice as many epochs.
Revised: The method constantly reduces the learning rate from 0.1 to 1 × 10 − 4 according to a cosine decay, across a certain number of epochs, and then repeats across twice as many epochs.

Original: We restricted our attention to a maximum of 254 epochs (often just 62 epochs, and no more than 30 for Imagenet32) using this method, which is the total number of epochs after reducing the learning rate from maximum to minimum through 2 epochs, then 4, 8, 16, 32 and 64 and 128 epochs.
Revised: We restricted our attention to a maximum of 254 epochs (often just 62 epochs, and no more than 30 for Imagenet32) using this method, which is the total number of epochs after reducing the learning rate from maximum to minimum through 2 epochs, then 4, 8, 16, 32 and 64 and 128 epochs.

Original: For CIFAR-10/100, we typically found that we could achieve test error rates after 32 epochs within 1-2% of the error rates achievable after 126 or 254 epochs.
Revised: For CIFAR-10/100, we typically found that we could achieve test error rates after 32 epochs within 1-2% of the error rates achievable after 126 or 254 epochs.

Original: 3.2. E XPERIMENTS WITH CUTOUT DATA AUGMENTATION FOR CIFAR 10/ 
Revised: 3.2. E XPERIMENTS WITH CUTOUT DATA AUGMENTATION FOR CIFAR 10/

Original: In the literature, most experiments with CIFAR-10 and CIFAR-100 use simple “standard” data augmentation, consisting of randomly ﬂipping each training image left-right with probability 0.5, and padding each image on all sides by 4 pixels, and then cropping a 32 × 32 version of the image from a random location.
Revised: In the literature, most experiments with CIFAR-10 and CIFAR-100 use simple “standard” data augmentation, consisting of randomly ﬂipping each training image left-right with probability 0.5, and padding each image on all sides by 4 pixels, and then cropping a 32 × 32 version of the image from a random location.

Original: We use this augmentation, although with the minor modiﬁcation that we pad with uniform random integers between 0 and 255 , rather than zero-padding.
Revised: We use this augmentation, although with the minor modiﬁcation that we pad with uniform random integers between 0 and 255 , rather than zero-padding.

Original: Additionally, we experimented with “cutout” augmentation (Devries & Taylor, 2017) applied.
Revised: Additionally, we experimented with “cutout” augmentation (Devries & Taylor, 2017) applied.

Original: This involves randomly selecting a patch of each raw training image to remove.
Revised: This involves randomly selecting a patch of each raw training image to remove.

Original: The method was shown to combine with other state-of-the-art methods to set the latest state-of-the-art results on CIFAR10/100 (see Table 1).
Revised: The method was shown to combine with other state-of-the-art methods to set the latest state-of-the-art results on CIFAR10/100 (see Table 1).

Original: Following Devries & Taylor (2017), for CIFAR-10 we choose patches of size 16 × 16 and for CIFAR-100 we choose patches of size 8 × 8 , while if the chosen pixel is near the image border, the patch impacts on the image only for the part of the patch inside the image.
Revised: Following Devries & Taylor (2017), for CIFAR-10 we choose patches of size 16 × 16 and for CIFAR-100 we choose patches of size 8 × 8 , while if the chosen pixel is near the image border, the patch impacts on the image only for the part of the patch inside the image.

Original: As with padding, we use uniform random integers to replace the image pixel values in the location of the patches.
Revised: As with padding, we use uniform random integers to replace the image pixel values in the location of the patches.

Original: We did not apply cutout to other datasets.
Revised: We did not apply cutout to other datasets.

Original:  R ESULTS
Revised:  R ESULTS

Original: We conducted experiments with four databases consisting of 32 × 32 RGB images: CIFAR-10,
Revised: We conducted experiments with four databases consisting of 32 × 32 RGB images: CIFAR-10,

Original: CIFAR-100, SVHN and Imagenet32, as well as MNIST.
Revised: CIFAR-100, SVHN and Imagenet32, as well as MNIST.

Original: Details of the ﬁrst three datasets can be found in many papers, e.g. Zagoruyko & Komodakis (2016).
Revised: Details of the ﬁrst three datasets can be found in many papers, e.g. Zagoruyko & Komodakis (2016).

Original: Imagenet32 is a downsampled version of Imagenet, where the training and validation images are cropped using their annotated bounding boxes, and then downsampled to 32 × 32 (Chrabaszcz et al., 2017); see http://image-net.org/download-images .
Revised: Imagenet32 is a downsampled version of Imagenet, where the training and validation images are cropped using their annotated bounding boxes, and then downsampled to 32 × 32 (Chrabaszcz et al., 2017); see http://image-net.org/download-images .

Original: All experiments were carried out on either a single Titan X or Xp GPU using matlab with GPU acceleration from matconvnet and CuDNN.
Revised: All experiments were carried out on either a single Titan X or Xp GPU using matlab with GPU acceleration from matconvnet and CuDNN.

Original: Using the Titan Xp, we were able to train our 27.4 million parameters 20-10 ResNet on Imagenet in just 35 hours total for 30 epochs.
Revised: Using the Titan Xp, we were able to train our 27.4 million parameters 20-10 ResNet on Imagenet in just 35 hours total for 30 epochs.

Original: We mostly report results for networks with between 25 and 40 million parameters, given state-ofthe-art networks for these four datasets fall into this bracket.
Revised: We mostly report results for networks with between 25 and 40 million parameters, given state-ofthe-art networks for these four datasets fall into this bracket.

Original: To this end, we mainly use wide
Revised: To this end, we mainly use wide

Original: ResNets that are 4 × and 10 × wider than baseline ResNets, to use the terminology of Zagoruyko &
Revised: ResNets that are 4 × and 10 × wider than baseline ResNets, to use the terminology of Zagoruyko &

Original: Komodakis (2016), where the baseline has 16 channels in the layers at the ﬁrst spatial scale.
Revised: Komodakis (2016), where the baseline has 16 channels in the layers at the ﬁrst spatial scale.

Original: We use notation of the form 20-10 to denote wide ResNets with 20 convolutional layers and width 4 × .
Revised: We use notation of the form 20-10 to denote wide ResNets with 20 convolutional layers and width 4 × .

Original: Table 1 lists our top-1 error rates for CIFAR 10/100; C10 indicates CIFAR-10, C100 indicates
Revised: Table 1 lists our top-1 error rates for CIFAR 10/100; C10 indicates CIFAR-10, C100 indicates

Original: CIFAR100.
Revised: CIFAR100.

Original: In Table 1, the superscript + indicates standard crop and ﬂip augmentation, whereas ++ indicates additional cutout augmentation.
Revised: In Table 1, the superscript + indicates standard crop and ﬂip augmentation, whereas ++ indicates additional cutout augmentation.

Original: Table 2 lists top-1 and top-5 error rates for SVHN and
Revised: Table 2 lists top-1 and top-5 error rates for SVHN and

Original: Imagenet32 (I32), as for both of these we trained for only 30 epochs.
Revised: Imagenet32 (I32), as for both of these we trained for only 30 epochs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Table 3 shows comparison results from the original work on wide ResNets
Revised: Table 3 shows comparison results from the original work on wide ResNets

Original: [Table]
Revised: [Table]

Original: and subsequent papers that have reduced error rates on the CIFAR-10 and CIFAR-100 datasets.
Revised: and subsequent papers that have reduced error rates on the CIFAR-10 and CIFAR-100 datasets.

Original: We also show the only results we know of for 32 × 32 Imagenet.
Revised: We also show the only results we know of for 32 × 32 Imagenet.

Original: The current state-of-the-art for SVHN without augmentation is 1 .
Revised: The current state-of-the-art for SVHN without augmentation is 1 .

Original: 59 % (Huang et al., 2016)
Revised: 59 % (Huang et al., 2016)

Original: [Table]
Revised: [Table]

Original: and with cutout augmentation is 1 .
Revised: and with cutout augmentation is 1 .

Original: 30 % (Devries & Taylor, 2017).
Revised: 30 % (Devries & Taylor, 2017).

Original: Our full-precision result for SVHN (1.75%) is only a little short of these even though we used only a 4 × ResNet, with less than 5 million parameters, and only 30 training epochs.
Revised: Our full-precision result for SVHN (1.75%) is only a little short of these even though we used only a 4 × ResNet, with less than 5 million parameters, and only 30 training epochs.

Original: Table 4 shows comparison results for previous work that trains models by using the sign of weights during training.
Revised: Table 4 shows comparison results for previous work that trains models by using the sign of weights during training.

Original: Additional results appear in Hubara et al.
Revised: Additional results appear in Hubara et al.

Original: (2016), where activations are quantized,
Revised: (2016), where activations are quantized,

Original: and so the error rates are much larger.
Revised: and so the error rates are much larger.

Original: Inspection of Tables 1 to 4 reveals that our 10 × baseline full-precision networks, when trained with cutout, surpass the performance of much larger wide ResNets trained with dropout.
Revised: Inspection of Tables 1 to 4 reveals that our 10 × baseline full-precision networks, when trained with cutout, surpass the performance of much larger wide ResNets trained with dropout.

Original: Even without the use of cutout, our 20-10 network surpasses by over 2% the CIFAR-100 error rate reported for essentially the same network by Zagoruyko & Komodakis (2016) and is marginally better on CIFAR10 and ImageNet32.
Revised: Even without the use of cutout, our 20-10 network surpasses by over 2% the CIFAR-100 error rate reported for essentially the same network by Zagoruyko & Komodakis (2016) and is marginally better on CIFAR10 and ImageNet32.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: This Paper
Revised: This Paper

Original: [Table]
Revised: [Table]

Original: For our 1-bit networks, we observe that there is always an accuracy gap compared to full precision networks.
Revised: For our 1-bit networks, we observe that there is always an accuracy gap compared to full precision networks.

Original: As discussed in Section 5.1, this gap tends to grow as the error-rate in the full-precision network grows.
Revised: As discussed in Section 5.1, this gap tends to grow as the error-rate in the full-precision network grows.

Original: The use of cutout reduced error rates on CIFAR-10/100 in most cases.
Revised: The use of cutout reduced error rates on CIFAR-10/100 in most cases.

Original: In comparison with training 20 × ResNets on CIFAR-100, as shown in Table 1, it is as effective to use cutout augmentation in the 10 × network to reduce the error rate, but only a quarter of the weights are required.
Revised: In comparison with training 20 × ResNets on CIFAR-100, as shown in Table 1, it is as effective to use cutout augmentation in the 10 × network to reduce the error rate, but only a quarter of the weights are required.

Original: Figure 4 demonstrates convergence and overﬁtting trends for CIFAR-10/100 for 20-4 wide ResNets, and a comparison of the use of cutout in 20-10 wide ResNets.
Revised: Figure 4 demonstrates convergence and overﬁtting trends for CIFAR-10/100 for 20-4 wide ResNets, and a comparison of the use of cutout in 20-10 wide ResNets.

Original: Clearly, even for width-4 ResNets, the gap in error rate between full precision weights and single-bit weights is small.
Revised: Clearly, even for width-4 ResNets, the gap in error rate between full precision weights and single-bit weights is small.

Original: Also noticeable is that the warm-restart method enables convergence to very good solutions after just 30 epochs; training longer to 126 epochs reduces error rates further by between 2% and 5%.
Revised: Also noticeable is that the warm-restart method enables convergence to very good solutions after just 30 epochs; training longer to 126 epochs reduces error rates further by between 2% and 5%.

Original: It can also be observed that the 20-4 network is powerful enough to model the CIFAR-10/100 training sets to well over 99% accuracy, but the modelling power is reduced in the 1-bit version, particularly for CIFAR100.
Revised: It can also be observed that the 20-4 network is powerful enough to model the CIFAR-10/100 training sets to well over 99% accuracy, but the modelling power is reduced in the 1-bit version, particularly for CIFAR100.

Original: The reduced modelling capacity for single-bit weights is consistent with the gap in test-error rate performance between the 32-bit and 1-bit cases.
Revised: The reduced modelling capacity for single-bit weights is consistent with the gap in test-error rate performance between the 32-bit and 1-bit cases.

Original: When using cutout, training for longer gives improved error rates, but when not using cutout, 126 epochs sufﬁces for peak performance.
Revised: When using cutout, training for longer gives improved error rates, but when not using cutout, 126 epochs sufﬁces for peak performance.

Original: Finally, for MNIST and a 4 × wide ResNet without any data augmentation, our full-precision method achieved 0.71% after just 1 epoch of training, and 0.28% after 6 epochs, whereas our 1-bit method achieved 0.81% and 0.36% after 1 and 6 epochs of training.
Revised: Finally, for MNIST and a 4 × wide ResNet without any data augmentation, our full-precision method achieved 0.71% after just 1 epoch of training, and 0.28% after 6 epochs, whereas our 1-bit method achieved 0.81% and 0.36% after 1 and 6 epochs of training.

Original: For the 1-bit version we also observed 0.27% after 14 epochs.
Revised: For the 1-bit version we also observed 0.27% after 14 epochs.

Original: In comparison, 1.29% was reported for the 1-bit case by Courbariaux et al.
Revised: In comparison, 1.29% was reported for the 1-bit case by Courbariaux et al.

Original: (2015), and 0.96% by Hubara et al.
Revised: (2015), and 0.96% by Hubara et al.

Original:  D ISCUSSION 5. T HE ACCURACY GAP FOR 1- BIT VS 32- BIT WEIGHTS
Revised:  D ISCUSSION 5. T HE ACCURACY GAP FOR 1- BIT VS 32- BIT WEIGHTS

Original: It is to be expected that a smaller error rate gap will result between the same network using fullprecision and 1-bit weights when the test error rate on the full precision network gets smaller.
Revised: It is to be expected that a smaller error rate gap will result between the same network using fullprecision and 1-bit weights when the test error rate on the full precision network gets smaller.

Original: How-
Revised: How-

Original: [Figure]
Revised: [Figure]

Original: ever, how the magnitude of the gap changes with full precision error rate is dependent on many factors, including the method used to generate models with 1-bit weights.
Revised: ever, how the magnitude of the gap changes with full precision error rate is dependent on many factors, including the method used to generate models with 1-bit weights.

Original: To illustrate the trend for our approach, we have plotted in Figure 5 the gap in Top-1 error rates vs the Top-1 error rate for the full precision case, for some of the best performing networks for the ﬁve datasets we used.
Revised: To illustrate the trend for our approach, we have plotted in Figure 5 the gap in Top-1 error rates vs the Top-1 error rate for the full precision case, for some of the best performing networks for the ﬁve datasets we used.

Original: Conclusions from this data can only be made relative to alternative methods.
Revised: Conclusions from this data can only be made relative to alternative methods.

Original: Hence, we have also plotted in Figure 5 the error rate and the gap reported by Rastegari et al.
Revised: Hence, we have also plotted in Figure 5 the error rate and the gap reported by Rastegari et al.

Original: (2016) for two different networks using their BWN 1-bit weight method.
Revised: (2016) for two different networks using their BWN 1-bit weight method.

Original: It is clear that the gaps in those datapoints would not lie on a straight-line ﬁt through the points from our experiments.
Revised: It is clear that the gaps in those datapoints would not lie on a straight-line ﬁt through the points from our experiments.

Original: [Figure]
Revised: [Figure]

Original: However, a fair comparison requires further experiments seeking conﬁrmation of whether our method applied to the full Imagenet dataset produces a gap closer to the line connecting our results for the ﬁve datasets.
Revised: However, a fair comparison requires further experiments seeking conﬁrmation of whether our method applied to the full Imagenet dataset produces a gap closer to the line connecting our results for the ﬁve datasets.

Original: These experiments are currently underway.
Revised: These experiments are currently underway.

Original: Two challenges for further work are (1) todevise methods that reduce the  gap signiﬁcantly more than shown here for datasets where error-rates are large; or (2) to predict theoretical bounds on what the gap should be.
Revised: In preliminary work, for a 16-8 ResNet with a different architecture to that used elsewhere in this paper (as well as requiringscales), we achieved single-centre-crop Top-1 error rates of 32.87% and 36.44% on the full precision and 1-bit weights versions, using 30 epochs of the warm-restart method. This gap of about 3.5%, as shown in Figure 5, is about half the gaps reported by Rastegari et al. (2016), similar to the reduction in gaps shown in Table 4. A challenges for further work is to derive theoretical bounds that predict the gap.

Original: For high-error rate cases, the loss function throughout training is much higher for the 1-bit case than the 32-bit case, and hence, the 1-bit network is not able to ﬁt the training set as well as the 32-bit one.
Revised: For high-error rate cases, the loss function throughout training is much higher for the 1-bit case than the 32-bit case, and hence, the 1-bit network is not able to ﬁt the training set as well as the 32-bit one.

Original: Whether this is because of the loss of precision in weights, or due to the mismatch in gradients inherent is propagating with 1-bit weights and updating full-precision weights during training is an open question.
Revised: Whether this is because of the loss of precision in weights, or due to the mismatch in gradients inherent is propagating with 1-bit weights and updating full-precision weights during training is an open question.

Original: If it is the latter case, then it is possible that principled reﬁnements to the weight update method we used will further reduce the gap.
Revised: If it is the latter case, then it is possible that principled reﬁnements to the weight update method we used will further reduce the gap.

Original: 5. C OMPARISON WITH THE BWN METHOD
Revised: 5. C OMPARISON WITH THE BWN METHOD

Original: Our approach differs from the BWN method of Rastegari et al.
Revised: Our approach differs from the BWN method of Rastegari et al.

Original: (2016) for two reasons.
Revised: (2016) for two reasons.

Original: First, we do not need to calculate mean absolute weight values of the underlying full precision weights for each output channel in each layer following each minibatch, and this enables faster training.
Revised: First, we do not need to calculate mean absolute weight values of the underlying full precision weights for each output channel in each layer following each minibatch, and this enables faster training.

Original: Second, we do not need to adjust for a gradient term corresponding to the appearance of each weight in the mean absolute value.
Revised: Second, we do not need to adjust for a gradient term corresponding to the appearance of each weight in the mean absolute value.

Original: Moreover, although we observed that the method of Rastegari et al.
Revised: Moreover, although we observed that the method of Rastegari et al.

Original: (2016) converges slightly faster early in training, by the end of training our simpler method gave better results for CIFAR-10 and
Revised: (2016) converges slightly faster early in training, by the end of training our simpler method gave better results for CIFAR-10 and

Original: CIFAR-100, and signiﬁcantly better results for Imagenet32.
Revised: CIFAR-100, and signiﬁcantly better results for Imagenet32.

Original: We speculate that this is because the scaling method of Rastegari et al.
Revised: We speculate that this is because the scaling method of Rastegari et al.

Original: (2016) assumes that mean square error between full precision weights and scaled binary weights is the best measure for determining scaling; for practical purposes, minimising mean square error does not necessarily mean better classiﬁcation error rates.
Revised: (2016) assumes that mean square error between full precision weights and scaled binary weights is the best measure for determining scaling; for practical purposes, minimising mean square error does not necessarily mean better classiﬁcation error rates.

Original: It is also possible that combining the adaptive method with weight decay has undesirable consequences.
Revised: It is also possible that combining the adaptive method with weight decay has undesirable consequences.

Original: In our method, since the scale factors never change, weight decay only inﬂuences the underlying full-precision weights that get updated, not the propagation of data.
Revised: In our method, since the scale factors never change, weight decay only inﬂuences the underlying full-precision weights that get updated, not the propagation of data.

Original: 5. T HE INFLUENCE OF NOT LEARNING BATCH - NORMALISATION PARAMETERS
Revised: 5. T HE INFLUENCE OF NOT LEARNING BATCH - NORMALISATION PARAMETERS

Original: The unusual design choice of not learning the batch normalization parameters was made for CIFAR10/100, SVHN and MNIST because for wide ResNets, overﬁtting is very evident on these datasets (see Figure 4); by the end of training, typically the loss function becomes very close to zero, corresponding to severe overﬁtting.
Revised: The unusual design choice of not learning the batch normalization parameters was made for CIFAR10/100, SVHN and MNIST because for wide ResNets, overﬁtting is very evident on these datasets (see Figure 4); by the end of training, typically the loss function becomes very close to zero, corresponding to severe overﬁtting.

Original: Inspired by label-smoothing regularization (Szegedy et al., 2015) that aims to reduce overconﬁdence following the softmax layer, we hypothesised that imposing more control over the standard deviation of inputs to the softmax might have a similar regularizing effect.
Revised: Inspired by label-smoothing regularization (Szegedy et al., 2015) that aims to reduce overconﬁdence following the softmax layer, we hypothesised that imposing more control over the standard deviation of inputs to the softmax might have a similar regularizing effect.

Original: This is why we removed the ﬁnal all-to-all layer of most ResNets and replaced it with a 1 × 1 convolutional layer followed by a batch-normalization layer prior to the global average pooling layer.
Revised: This is why we removed the ﬁnal all-to-all layer of most ResNets and replaced it with a 1 × 1 convolutional layer followed by a batch-normalization layer prior to the global average pooling layer.

Original: In turn, not learning a scale and bias for this batch-normalization layer ensures that batches ﬂowing into the softmax layer have standard deviations that do not grow throughout training, which tends to increase the entropy of predictions following the softmax, which is equivalent to lower conﬁdence (Guo et al., 2017).
Revised: In turn, not learning a scale and bias for this batch-normalization layer ensures that batches ﬂowing into the softmax layer have standard deviations that do not grow throughout training, which tends to increase the entropy of predictions following the softmax, which is equivalent to lower conﬁdence (Guo et al., 2017).

Original: After observing success with these methods in 10 × wide ResNets, we then observed that learning the batch-norm parameters in other layers also led to increased overﬁtting, and increased test error rates, and so removed that learning in all layers except the ﬁrst one applied to the input.
Revised: After observing success with these methods in 10 × wide ResNets, we then observed that learning the batch-norm parameters in other layers also led to increased overﬁtting, and increased test error rates, and so removed that learning in all layers except the ﬁrst one applied to the input.

Original: As expected from the motivation, we found this method is not appropriate for datasets such as ImageNet where overﬁtting is not as evident, in which case learning the batch normalization parameters can signiﬁcantly reduce test error rates.
Revised: As expected from the motivation, we found this method is not appropriate for datasets such as ImageNet where overﬁtting is not as evident, in which case learning the batch normalization parameters can signiﬁcantly reduce test error rates.

Original: 5. L IMITATIONS AND F URTHER W ORK
Revised: 5. L IMITATIONS AND F URTHER W ORK

Original: In this paper we focus only on the memory required to store a trained model, i.e. model compression .
Revised: In this paper we focus only on the memory required to store a trained model, i.e. model compression .

Original: It is also interesting and desirable to reduce the computational load of inference using a trained model, by carrying out layer computations in very few numbers of bits (Hubara et al., 2016;
Revised: It is also interesting and desirable to reduce the computational load of inference using a trained model, by carrying out layer computations in very few numbers of bits (Hubara et al., 2016;

Original: Rastegari et al., 2016; Cai et al., 2017).
Revised: Rastegari et al., 2016; Cai et al., 2017).

Original: Facilitating this requires modifying non-linear activations in a network from ReLUs to quantized ReLUs, or in the extreme case, binary step functions.
Revised: Facilitating this requires modifying non-linear activations in a network from ReLUs to quantized ReLUs, or in the extreme case, binary step functions.

Original: Here we use only full-precision calculations.
Revised: Here we use only full-precision calculations.

Original: It can be expected that combining our methods with reduced precision processing will inevitably increase error rates.
Revised: It can be expected that combining our methods with reduced precision processing will inevitably increase error rates.

Original: We have addressed this extension in a forthcoming submission.
Revised: We have addressed this extension in a forthcoming submission.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Removed for double-blind peer-review.
Revised: Removed for double-blind peer-review

Original: A D IFFERENCES FROM S TANDARD R ES N ETS IN OUR B ASELINE A. O UR NETWORK ’ S FINAL WEIGHT LAYER IS A 1 × 1 CONVOLUTIONAL LAYER
Revised: A D IFFERENCES FROM S TANDARD R ES N ETS IN OUR B ASELINE A. O UR NETWORK ’ S FINAL WEIGHT LAYER IS A 1 × 1 CONVOLUTIONAL LAYER

Original: The most signiﬁcant difference to He et al.
Revised: The most signiﬁcant difference to He et al.

Original: ; Zagoruyko & Komodakis (2016) is we exchange the ordering of the global average pooling layer and the ﬁnal weight layer; our ﬁnal weight layer becomes a 1 × 1 convolutional layer with as many channels as there are classes in the training set.
Revised: ; Zagoruyko & Komodakis (2016) is we exchange the ordering of the global average pooling layer and the ﬁnal weight layer; our ﬁnal weight layer becomes a 1 × 1 convolutional layer with as many channels as there are classes in the training set.

Original: This design is not new, but it does seem to be new to ResNets: it corresponds to the architecture of Lin et al.
Revised: This design is not new, but it does seem to be new to ResNets: it corresponds to the architecture of Lin et al.

Original: (2013), which originated the global average pooling concept, and also to that used by Springenberg et al.
Revised: (2013), which originated the global average pooling concept, and also to that used by Springenberg et al.

Original: Using this method means we entirely follow the “all-convolutional” concept of Springenberg et al.
Revised: Using this method means we entirely follow the “all-convolutional” concept of Springenberg et al.

Original: A. O UR NETWORK DOWNSAMPLES THE RESIDUAL PATH USING AVERAGE POOLING
Revised: A. O UR NETWORK DOWNSAMPLES THE RESIDUAL PATH USING AVERAGE POOLING

Original: For skip connections between feature maps of different sizes, we use zero-padding for increasing the number of channels as per option 1 of He et al.
Revised: For skip connections between feature maps of different sizes, we use zero-padding for increasing the number of channels as per option 1 of He et al.

Original: (2015b).
Revised: (2015b).

Original: However, for the residual pathway, we use average pooling using a kernel of size 3 × 3 with stride 2 for downsampling, instead of typical lossy downsampling that discards all pixel values in between samples.
Revised: However, for the residual pathway, we use average pooling using a kernel of size 3 × 3 with stride 2 for downsampling, instead of typical lossy downsampling that discards all pixel values in between samples.

Original: We found in some cases this reduced error rates by around 1%.
Revised: We found in some cases this reduced error rates by around 1%.

Original: A. W E USE BATCH NORMALIZATION APPLIED TO THE INPUT LAYER
Revised: A. W E USE BATCH NORMALIZATION APPLIED TO THE INPUT LAYER

Original: The literature has reported various options for the optimal ordering, usage and placement of BN and
Revised: The literature has reported various options for the optimal ordering, usage and placement of BN and

Original: ReLU layers in residual networks.
Revised: ReLU layers in residual networks.

Original: Following He et al.
Revised: Following He et al.

Original: (2016); Zagoruyko & Komodakis (2016)
Revised: (2016); Zagoruyko & Komodakis (2016)

Original: , we precede convolutional layers with the combination of BN followed by ReLU.
Revised: , we precede convolutional layers with the combination of BN followed by ReLU.

Original: However, different to He et al.
Revised: However, different to He et al.

Original: (2016); Zagoruyko & Komodakis (2016)
Revised: (2016); Zagoruyko & Komodakis (2016)

Original: , we also insert the BNReLU combination immediately after the input layer and before the ﬁrst convolutional layer.
Revised: , we also insert the BNReLU combination immediately after the input layer and before the ﬁrst convolutional layer.

Original: Unlike all other batch-normalization layers, we enable learning of the scale and bias factors in this ﬁrst batch-norm, which enables us to avoid doing any pre-processing on the inputs to the network, since the BN layer learns how to scale and bias each of the RGB channels; we found that the bias learnt ensures the input to the ﬁrst ReLU is never negative.
Revised: Unlike all other batch-normalization layers, we enable learning of the scale and bias factors in this ﬁrst batch-norm, which enables us to avoid doing any pre-processing on the inputs to the network, since the BN layer learns how to scale and bias each of the RGB channels; we found that the bias learnt ensures the input to the ﬁrst ReLU is never negative.

Original: In accordance with our strategy of simplicity, without exception all weight layers can be thought of as a block of three operations in the same sequence, as indicated in Figure 1.
Revised: In accordance with our strategy of simplicity, without exception all weight layers can be thought of as a block of three operations in the same sequence, as indicated in Figure 1.

Original: Conceptually, batchnorm followed by ReLU can be thought of as a single layer consisting of a ReLU that adaptively changes its centre point and positive slope for each channel and relative to each mini-batch.
Revised: Conceptually, batchnorm followed by ReLU can be thought of as a single layer consisting of a ReLU that adaptively changes its centre point and positive slope for each channel and relative to each mini-batch.

Original: We also precede the global average pooling layer by a BN layer, but do not use a ReLU at this point, since nonlinear activation is provided by the softmax layer that follows immediately after global average pooling.
Revised: We also precede the global average pooling layer by a BN layer, but do not use a ReLU at this point, since nonlinear activation is provided by the softmax layer that follows immediately after global average pooling.

Original: We found including the ReLU leads to differences early in training but not by the completion of training.
Revised: We found including the ReLU leads to differences early in training but not by the completion of training.

Original: A. O UR FIRST CONV LAYER HAS AS MANY CHANNELS AS THE FIRST RESIDUAL BLOCK
Revised: A. O UR FIRST CONV LAYER HAS AS MANY CHANNELS AS THE FIRST RESIDUAL BLOCK

Original: The wide ResNet of Zagoruyko & Komodakis (2016) is speciﬁed as always having a constant number of output channels, even when the number of output channels for other layers increases.
Revised: The wide ResNet of Zagoruyko & Komodakis (2016) is speciﬁed as always having a constant number of output channels, even when the number of output channels for other layers increases.

Original: We found no need to impose this constraint, and instead always allow the ﬁrst layer to share the same number of output channels as all blocks at the ﬁrst spatial scale.
Revised: We found no need to impose this constraint, and instead always allow the ﬁrst layer to share the same number of output channels as all blocks at the ﬁrst spatial scale.

Original: The increase in the total number of parameters from doing this is small relative to the total number of parameters, since the number of input channels to the ﬁrst layer is just 3.
Revised: The increase in the total number of parameters from doing this is small relative to the total number of parameters, since the number of input channels to the ﬁrst layer is just 3.

Original: The beneﬁt of this change is increased simplicity in the network deﬁnition, and allow one fewer change in the dimensional of the residual skip pathway.
Revised: The beneﬁt of this change is increased simplicity in the network deﬁnition, and allow one fewer change in the dimensional of the residual skip pathway.

Original: B C OMPARISON OF R ESIDUAL N ETWORKS AND P LAIN A LL -C ONVOLUTIONAL N ETWORKS
Revised: B C OMPARISON OF R ESIDUAL N ETWORKS AND P LAIN A LL -C ONVOLUTIONAL N ETWORKS

Original: We were interested in understanding whether the good results we achieved for single-bit weights were a consequence of the skip connections in residual networks.
Revised: We were interested in understanding whether the good results we achieved for single-bit weights were a consequence of the skip connections in residual networks.

Original: We therefore applied our method to plain all-convolutional networks identical to our 4 × residual networks, except with the skip connections removed.
Revised: We therefore applied our method to plain all-convolutional networks identical to our 4 × residual networks, except with the skip connections removed.

Original: Initially, training indicated a much slower convergence, but we found that altering the initial weights standard deviations to be proportional to 2 instead of √ 2 helped, so this was the only other change made.
Revised: Initially, training indicated a much slower convergence, but we found that altering the initial weights standard deviations to be proportional to 2 instead of √ 2 helped, so this was the only other change made.

Original: The change was also applied in Equation (1).
Revised: The change was also applied in Equation (1).

Original: As summarised in Figure 6, we found that convergence remained slower than our ResNets, but there was only a small accuracy penalty in comparison with ResNets after 126 epochs of training.
Revised: As summarised in Figure 6, we found that convergence remained slower than our ResNets, but there was only a small accuracy penalty in comparison with ResNets after 126 epochs of training.

Original: This is consistent with the ﬁndings of He et al.
Revised: This is consistent with the ﬁndings of He et al.

Original: (2015b) where only ResNets deeper than about 20 layers showed a signiﬁcant advantage over plain all-convolutional networks.
Revised: (2015b) where only ResNets deeper than about 20 layers showed a signiﬁcant advantage over plain all-convolutional networks.

Original: We conclude that our method is not particular to ResNets.
Revised: We conclude that our method is not particular to ResNets.

Original: [Figure]
Revised: [Figure]