Original: T RAINING WIDE RESIDUAL NETWORKS FOR DEPLOYMENT USING A SINGLE BIT FOR EACH WEIGHT
Revised: T RAINING WIDE RESIDUAL NETWORKS FOR DEPLOYMENT USING A SINGLE BIT FOR EACH WEIGHT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: For fast and energy-efﬁcient deployment of trained deep neural networks on resource-constrained embedded hardware, each learnt weight parameter should ideally be represented and stored using a single bit.
Revised: For fast and energy-efﬁcient deployment of trained deep neural networks on resource-constrained embedded hardware, each learnt weight parameter should ideally be represented and stored using a single bit.

Original: Error-rates usually increase when this requirement is imposed.
Revised: Error-rates usually increase when this requirement is imposed.

Original: Here, we reportmethodological innovations that result in large reductions in error rates  for deep convolutional neural networks deployed using a single bit for each weight.
Revised: Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight. Using wide residual networks as our main baseline, our approach modiﬁes existing methods that binarize weights by applying the sign function in training. Speciﬁcally, we use unlearned scaling factors for each layer with values equal to layer-speciﬁc standard deviations used for initialization. For

Original: For 1-bit weight models with just 20 weight layers, requiring only ∼ 4 MB of parameter memory,for CIFAR10, CIFAR-100 we achieve error rates of 3.74%, 18.41%.
Revised: CIFAR10, CIFAR100 and ImageNet, and models with 1-bit-per-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% Top-1 / Top-5 respectively.

Original: We also considered 
Revised: We also considered MNIST, SVHN and

Original: MNIST, SVHN, Imagenet32, achieving single-bit weight test results of 0.27%, 1.93%, and 42.92/19.95% (Top-5/Top-5) respectively.
Revised: ImageNet32, achieving 1-bit-per-weight test results of 0.27%, 1.9%, and 41.3% / 19.1% (Top-1 / Top-5) respectively.

Original: These error rates are about half those of previous reported error rates for CIFAR-10/100, and are within 1–3% of our error-rates for the same network with full-precision weights.
Revised: For CIFAR, our error rates halve previously reported values, and are within about 1% of our error-rates for the same network with full-precision weights.

Revised: For networks that overﬁt, we also show signiﬁcant improvements in error rate by not learning batch normalisation scale and offset parameters.

Original:  Using a warmrestart learning-rate schedule, we found  training for single-bit weights just as fast as full-precision networks, and achieved about 98%-99% of peak performance in just 62 training epochs for CIFAR 10/100.
Revised: This applies to both full precision and 1-bit-per-weight networks. Using a warm-restart learning-rate schedule, we found that training for 1-bit-perweight is just as fast as full-precision networks, and achieved about 98%-99% of peak performance in just 62 training epochs for CIFAR 10/100.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Fast parallel computing resources, namely GPUs, have been integral to the resurgence of deep neural networks, and their ascendancy to becoming state-of-the-art methodologies for many computer vision tasks.
Revised: Fast parallel computing resources, namely GPUs, have been integral to the resurgence of deep neural networks, and their ascendancy to becoming state-of-the-art methodologies for many computer vision tasks.

Original: However, GPUs are expensive in terms of their energy requirements.
Revised: However, GPUs are expensive in terms of their energy requirements.

Original: They typically compute using single-precision ﬂoating point (32 bits), which has now been recognised as providing far more precision than needed for deep neural networks.
Revised: They typically compute using single-precision ﬂoating point (32 bits), which has now been recognised as providing far more precision than needed for deep neural networks.

Original: Moreover, training and deployment can require the availability of large amounts of memory, both for storage of trained models, and for operational RAM.
Revised: Moreover, training and deployment can require the availability of large amounts of memory, both for storage of trained models, and for operational RAM.

Original: If deep-learning methods are to become embedded in resource-constrained sensors, devices and intelligent systems, ranging from robotics to the internet-of-things to self-driving cars, reliance on high-end computing resources will need to be reduced.
Revised: If deep-learning methods are to become embedded in resource-constrained sensors, devices and intelligent systems, ranging from robotics to the internet-of-things to self-driving cars, reliance on high-end computing resources will need to be reduced.

Original: To this end, there has been increasing interest in ﬁnding methods to drive down the resource burden of modern deep neural networks.
Revised: To this end, there has been increasing interest in ﬁnding methods that drive down the resource burden of modern deep neural networks.

Original: Existing methods typically exhibit good performance but for the ideal case of single-bit parameters and/or processing, still fall well-short of state-of-the-art error rates on known benchmarks.
Revised: Existing methods typically exhibit good performance but for the ideal case of single-bit parameters and/or processing, still fall well-short of state-of-the-art error rates on known benchmarks.

Original: In this paper, we detail methods that resulted in a signiﬁcant reduction in the gap (see Figureand Section 5.1) between Convolutional Neural Networks (CNNs) deployed using weights stored and applied using standard precision (32-bit ﬂoating point) and networks deployed using weights represented by a single-bit each 1 .
Revised: In this paper, we report a signiﬁcant reduction in the gap (see Figure 1 and Section 5.1) between Convolutional Neural Networks (CNNs) deployed using weights stored and applied using standard precision (32-bit ﬂoating point) and networks deployed using weights represented by a single-bit each 1 .

Original: In the process of developing our methods, we also obtained a small improvement in error-rates obtained by standard full-precision versions of the CNNs we used.
Revised: In the process of developing our methods, we also obtained a small improvement in errorrates obtained by standard full-precision versions of the CNNs we used.

Original: [Figure]
Revised: [Figure]

Revised: Black points are results on

Original: 1. R ELATED W ORK 1.1. R ES N ETS
Revised: 1. R ELATED W ORK 1.1. R ES N ETS

Original: In 2015, a new form of CNN called a “deep residual network,” or “ResNet” (He et al., 2015b) was developed and used to set many new accuracy records on computer-vision benchmarks.
Revised: In 2015, a new form of CNN called a “deep residual network,” or “ResNet” (He et al., 2015b) was developed and used to set many new accuracy records on computer-vision benchmarks.

Original: In comparison with older CNNs such as Alexnet (Krizhevsky et al., 2012) and VGGnet (Simonyan & Zisserman, 2014), ResNets achieve higher accuracy with far fewer learnt parameters and FLOPs (FLoating-point OPerations) per image processed.
Revised: In comparison with older CNNs such as Alexnet (Krizhevsky et al., 2012) and VGGNet (Simonyan & Zisserman, 2014), ResNets achieve higher accuracy with far fewer learnt parameters and FLOPs (FLoating-point OPerations) per image processed.

Original: The key to reducing the number of parameters in ResNets was to replace “all-to-all layers” in VGG-like nets with “global-average-pooling” layers that have no learnt parameters (Lin et al., 2013; Springenberg et al., 2014), while simultaneously training a much deeper network than previously.
Revised: The key to reducing the number of parameters in ResNets was to replace “all-to-all layers” in VGG-like nets with “global-average-pooling” layers that have no learnt parameters (Lin et al., 2013; Springenberg et al., 2014), while simultaneously training a much deeper network than previously.

Original: The key new idea that enabled a deeper network to be trained effectively was the introduction of so-called “skip-connections” (He et al., 2015b).
Revised: The key new idea that enabled a deeper network to be trained effectively was the introduction of so-called “skip-connections” (He et al., 2015b).

Original: Many variations of ResNets have since been proposed.
Revised: Many variations of ResNets have since been proposed.

Original: ResNets offer the virtue of simplicity, and given the motivation for deployment in custom hardware, we have chosen them as our primary focus.
Revised: ResNets offer the virtue of simplicity, and given the motivation for deployment in custom hardware, we have chosen them as our primary focus.

Original: Despite the increased efﬁciency in parameter usage, as with other CNNs,  accuracy in ResNets still tends to increase with the total number of parameters; unlike other CNNs, increased accuracy can result either from deeper (He et al., 2016) or wider networks (Zagoruyko & Komodakis, 2016).
Revised: Despite the increased efﬁciency in parameter usage, similar to other CNNs, the accuracy of ResNets still tends to increase with the total number of parameters; unlike other CNNs, increased accuracy can result either from deeper (He et al., 2016) or wider networks (Zagoruyko & Komodakis, 2016).

Original: In this paper, we use Wide Residual Networks (Zagoruyko & Komodakis, 2016), as they have been demonstrated to produce better accuracy in less training time than deeper networks.
Revised: In this paper, we use Wide Residual Networks (Zagoruyko & Komodakis, 2016), as they have been demonstrated to produce better accuracy in less training time than deeper networks.

Original: 1.1. R EDUCING THE MEMORY BURDEN OF TRAINED NEURAL NETWORKS
Revised: 1.1. R EDUCING THE MEMORY BURDEN OF TRAINED NEURAL NETWORKS

Original: Achieving the best accuracy and speed possible when deploying ResNets or similar networks on hardware-constrained mobile devices will require minimising the total number of bits transferred between memory and processors for a given number of parameters.
Revised: Achieving the best accuracy and speed possible when deploying ResNets or similar networks on hardware-constrained mobile devices will require minimising the total number of bits transferred between memory and processors for a given number of parameters.

Original: Motivated by such considerations, a lot of recent attention has been directed towards compressing the learnt parameters ( model compression  and quantizing other representations in trained neural networks—see Hubara et al.
Revised: Motivated by such considerations, a lot of recent attention has been directed towards compressing the learnt parameters ( model compression ) and reducing the precision of computations carried out by neural networks—see Hubara et al.

Original: (2016) for a more detailed literature review.
Revised: for a more detailed literature review.

Original: Recently published strategies for model compression include reducing the precision (number of bits used for numerical representation) of weights in deployed networks by doing the same dur- ing training (Courbariaux et al., 2015; 
Revised: Recently published strategies for model compression include reducing the precision (number of bits used for numerical representation) of weights in deployed networks by doing the same during training (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al.,

Original: Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), reducing the number of weights in trained neural networks by pruning (Iandola et al., 2016), quantizing  weights following training (Zhou et al., 2017), reducing the precision of computations performed in forward-propagation during inference (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), and modifying neural network architectures (Howard et al., 2017).
Revised: , reducing the number of weights in trained neural networks by pruning (Han et al., 2015; Iandola et al., 2016), quantizing or compressing weights following training (Han et al., 2015; Zhou et al., 2017), reducing the precision of computations performed in forward-propagation during inference (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), and modifying neural network architectures (Howard et al., 2017).

Original: A theoretical analysis of various methods proved results on the convergence of a variety of weight-binarization methods (Li et al., 2017).
Revised: A theoretical analysis of various methods proved results on the convergence of a variety of weight-binarization methods (Li et al., 2017).

Original: From this range of strategies, we are focused on  approaches that enable deployment of trained models immediately following training, for the simplicity this enables.
Revised: From this range of strategies, we are focused on an approaches that simultaneously match two desirable attributes: (1) simplicity, in the sense that deployment of trained models immediately follows training; (2) implementation of convolution operations can be achieved without multipliers, as demonstrated by (Rastegari et al., 2016).

Original:  O VERALL APPROACH AND SUMMARY OF CONTRIBUTIONS
Revised: 1. O VERALL APPROACH AND SUMMARY OF CONTRIBUTIONS

Original: Our strategy for improving methods that enable inference with single-bit weights was threefold:
Revised: Our strategy for improving methods that enable inference with 1-bit-per-weight was threefold:

Original: State-of-the-art baseline.
Revised: State-of-the-art baseline.

Original: We 
Revised: We sought

Original: sought to begin witha baseline
Revised: to begin with

Original:  full-precision deep CNN variant with close to state-of-the-art error rates.
Revised: a baseline full-precision deep CNN variant with close to state-of-the-art error rates.

Original: At the time of commencement in2016, the state of -the-
Revised: At the time of commencement in

Original: art on

Original: CIFAR 10 and CIFAR-100 was held by Wide Residual Net - works (Zagoruyko & Komodakis, 2016), so this was our starting point.
Revised: the state-of-the-art on CIFAR-10 and CIFAR-100 was held by Wide Residual Networks (Zagoruyko & Komodakis, 2016), so this was our starting point.

Original: While subsequent

Original: approaches have

Original: exceeded slightly the

Original: error rates of wide ResNets (see Table for hardware ResNets offer superior simplicity, which conforms with our third strategy in this list.
Revised: While subsequent approaches have exceeded their accuracy, ResNets offer superior simplicity, which conforms with our third strategy in this list.

Original: Make minimal changes when training for 1-bit weights.
Revised: Make minimal changes when training for 1-bit-per-weight.

Original: We aimed to ensure that training for 1-bit weights could be achieved with minimal changesin comparison to training our baseline.
Revised: We aimed to ensure that training for 1-bit-per-weight could be achieved with minimal changes to baseline training.

Original: Simplicity is desirable in custom hardware.
Revised: Simplicity is desirable in custom hardware.

Original: In addition to training to enable single-bit weights at inference time, with custom hardware implementations in mind we sought to simplify the design of the baseline network  as much as possible without sacriﬁcing accuracy.
Revised: With custom hardware implementations in mind we sought to simplify the design of the baseline network (and hence the version with 1-bit weights) as much as possible without sacriﬁcing accuracy.

Original:  C ONTRIBUTIONS TO F ULL -P RECISION W IDE R ES N ETS
Revised: 1.2. C ONTRIBUTIONS TO F ULL -P RECISION W IDE R ES N ETS

Original: Although this paper is chieﬂy about single-bit weights, we exceeded our objectives for the fullprecision baseline network, and surpassed reported error ratesreported for CIFAR-10 and CIFAR100 usingfull-precision Wide ResNets (Zagoruyko & Komodakis, 2016), while not requiring the use of dropout regularization (see Table 3).
Revised: Although this paper is chieﬂy about 1-bit-per-weight, we exceeded our objectives for the fullprecision baseline network, and surpassed reported error rates for CIFAR-10 and CIFAR-100 using Wide ResNets (Zagoruyko & Komodakis, 2016).

Original: This was achieved using just 20 convolutional layers; most prior work has demonstrated best wide ResNet performance using 28 convolutional layers.
Revised: This was achieved using just 20 convolutional layers; most prior work has demonstrated best wide ResNet performance using 28 convolutional layers.

Original: All but one of our modiﬁcations to standard Wide ResNets are mainly to achieve simplicity, as in the second and third prongs of our strategy.

Original: Collectively these minor modiﬁcations (see the Appendix) provide error-rate reductions of ∼ 1% on CIFAR-10/100 compared to not using them.

Original: However, one innovation we introduce achieves a signiﬁcant error-rate advantage for CIFAR-10 and CIFAR-100 in WideResNets.
Revised: Our innovation that achieves a signiﬁcant error-rate drop for CIFAR-10 and CIFAR-100 in Wide

Original: This is to simply not learn the per-channel scale and offset factors in the batch-normalization layers, while retaining the remaining attributes of these layers.
Revised: ResNets is to simply not learn the per-channel scale and offset factors in the batch-normalization layers, while retaining the remaining attributes of these layers.

Original: This is done in conjunction with exchanging the ordering of the ﬁnal weight layer and the global average pooling layer.
Revised: It is important that this is done in conjunction with exchanging the ordering of the ﬁnal weight layer and the global average pooling layer (see Figure 3).

Original: We present a case for why these changes should be advantageous, in the Discussion section.

Original: We observed this effect to be more pronounced for CIFAR-100 than CIFAR-10, gaining over 1% in test-error rate.
Revised: We observed this effect to be most pronounced for CIFAR-100, gaining around 3% in test-error rate.

Original: But the method is advantageous only when training tended to cause overﬁtting; when overﬁtting is not an issue, such as for Imagenet, removing learning of batch-norm parameters is only detrimental.
Revised: But the method is advantageous only for networks that overﬁt; when overﬁtting is not an issue, such as for ImageNet, removing learning of batch-norm parameters is only detrimental.

Original: 1.2. C ONTRIBUTIONS TO DEEP CNN S WITH SINGLE - BIT WEIGHTS FOR INFERENCE
Revised: 1.2. C ONTRIBUTIONS TO DEEP CNN S WITH SINGLE - BIT WEIGHTS FOR INFERENCE

Original:  Our approach surpasses by a large margin all previously reported error rates for CIFAR-10/ (error rates halved) and SVHN for networks constrained to run with single-bit weights at inference time.We found no previous results for single-bit weights for Imagenet32 (Chrabaszcz et al., 2017), but our results on this dataset are also indicative of the general applicability of our methods (see Section 5.1).
Revised: Ours is the ﬁrst study we are aware of to consider how the gap in error-rate for single-bit weights compared to full-precision weights changes with full-precision accuracy across a diverse range of image classiﬁcation datasets Our approach surpasses by a large margin all previously reported error rates for CIFAR-10/100 (error rates halved), for networks constrained to run with 1-bit-per-weight at inference time.

Original: One reason we have achieved lower error rates for the 1-bit case than previously is to start with a superior baseline network than in previous studies, namely wide ResNets.
Revised: One reason we have achieved lower error rates for the 1-bit case than previously is to start with a superior baseline network than in previous studies, namely Wide ResNets.

Original: However,as we show in Section 5.1, our approach  results in smaller error rate increases relative to full-precision error rates than previously (see also Figure 5.1). Moreover, previous modiﬁcations of training to use 1-bit weights has tended to result in slow convergence requiring training  for many 100s of epochs.
Revised: However, our approach also results in smaller error rate increases relative to full-precision error rates than previously, while training requires the same number of epochs as for the case of full-precision weights.

Original: Our contribution is to introduce a simple ﬁxed scaling method for each convolutional layer, that permits activations and gradients to ﬂow through the network with minimum change in standard deviation, in accordance with the principle underlying popular initialization methods (He et al., 2015a).
Revised: Our main innovation is to introduce a simple ﬁxed scaling method for each convolutional layer, that permits activations and gradients to ﬂow through the network with minimum change in standard deviation, in accordance with the principle underlying popular initialization methods (He et al., 2015a).

Original: We combine this with the use of a warm-restart learning-rate method (Loshchilov & Hutter, 2016) that enables us to report close-to-baseline results for the 1-bit case in far fewer epochs of training than reported previously.
Revised: We combine this with the use of a warm-restart learning-rate method (Loshchilov & Hutter, 2016) that enables us to report close-to-baseline results for the 1-bit case in far fewer epochs of training than reported previously.

Original:  L EARNING A MODEL WITH CONVOLUTION WEIGHTS ±2. T HE SIGN OF WEIGHTS PROPAGATE , BUT FULL - PRECISION WEIGHTS ARE UPDATED
Revised:  L EARNING A MODEL WITH CONVOLUTION WEIGHTS ±2. T HE SIGN OF WEIGHTS PROPAGATE , BUT FULL - PRECISION WEIGHTS ARE UPDATED

Original: We follow the approach of Courbariaux et al.
Revised: We follow the approach of Courbariaux et al.

Original: ; Rastegari et al.
Revised: ; Rastegari et al.

Original: ; Merolla et al.
Revised: ; Merolla et al.

Original: (2016), in that
Revised: (2016), in that

Original: we ﬁnd good results when using single-bit weights at inference time if during training we apply the sign function to real-valued weights for the purpose of forward and backward propagation, but update full-precision weights using SGD with gradients calculated using full-precision.
Revised: we ﬁnd good results when using single-bit weights at inference time if during training we apply the sign function to real-valued weights for the purpose of forward and backward propagation, but update full-precision weights using SGD with gradients calculated using full-precision.

Original: However, previously reported methods for training using the sign of weights either need to train for many hundreds of epochs (Courbariaux et al., 2015; Merolla et al., 2016), or use computationallycostly normalization scaling for each channel in each layer that changes for each minibatch during training (Rastegari et al., 2016).
Revised: However, previously reported methods for training using the sign of weights either need to train for many hundreds of epochs (Courbariaux et al., 2015; Merolla et al., 2016), or use computationallycostly normalization scaling for each channel in each layer that changes for each minibatch during training (Rastegari et al., 2016).

Original: We ﬁnd it is possible to dramatically accelerate convergence and execution time, while reducing error rates, using a simple alternative approach, as we now describe.
Revised: We ﬁnd it is possible to dramatically accelerate convergence and execution time, while reducing error rates, using a simple alternative approach, as we now describe.

Original: 2.1. W E SCALE THE OUTPUT OF CONV LAYERS USING A CONSTANT FOR EACH LAYER
Revised: 2.1. W E SCALE THE OUTPUT OF CONV LAYERS USING A CONSTANT FOR EACH LAYER

Original: We begin by noting that the standard deviation of the sign of the weights in a convolutional layer with kernels of size F × F will be close to 1 , assuming a mean of zero.
Revised: We begin by noting that the standard deviation of the sign of the weights in a convolutional layer with kernels of size F × F will be close to 1 , assuming a mean of zero.

Original: In contrast, the standard deviation of layer i in full-precision networks is initialised in the method of He et al.
Revised: In contrast, the standard deviation of layer i in full-precision networks is initialised in the method of He et al.

Original: (2015a) to (cid:112) 2 / ( F 2 C
Revised: (2015a) to (cid:112) 2 / ( F 2 C

Original: i − 1 ) , where C i − 1 is the number of input channels to convolutional layer i , where i = 1 , .., L , L is the number of convolutional layers and C 

Revised: i − 1 ) , where C i − 1 is the number of input channels to convolutional layer i , where i = 1 , .., L , L is the number of convolutional layers and C

Original: = 3 for RGB inputs.
Revised: = 3 for RGB inputs.

Original: Our strategy for a network trained using the sign of weights is to seek to make the propagation of weights and gradients statistically equivalent to those in the full precision network, so as to ensure the standard-deviation as information ﬂows through the network is close to the ideal situation that motivated the method of He et al.

Original: (2015a).

Original: When applying the sign function alone, there is a mismatch with the principled approach to controlling gradient and activation scaling through a deep network He et al.
Revised: When applying the sign function alone, there is a mismatch with the principled approach to controlling gradient and activation scaling through a deep network He et al.

Original: (2015a).
Revised: (2015a).

Original: Although the use of batch-normalization can still enable learning, convergence is empirically slow and less effective.
Revised: Although the use of batch-normalization can still enable learning, convergence is empirically slow and less effective.

Original: To address this problem, for training using the sign of weights, we use the initialisation method of He et al.
Revised: To address this problem, for training using the sign of weights, we use the initialisation method of He et al.

Original: (2015a)
Revised: (2015a)

Original: for the full-precision weights that are updated, but also introduce a  scaling for the single-bit weights of each convolutional layer.
Revised: for the full-precision weights that are updated, but also introduce a layer-dependent scaling applied to the sign of the weights.

Original: This scaling has a constant un-learnt value equal to the initial standard deviation of (cid:112) 2 / ( F 2 C i − 1 ) of the method of He et al.
Revised: This scaling has a constant unlearnt value equal to the initial standard deviation of (cid:112) 2 / ( F 2 C i − 1 ) from the method of He et al.

Original: (2015a).
Revised: (2015a).

Original: This enables the standard deviation of forward-propagating information to be equal to the value it would have initially in full-precision networks.
Revised: This enables the standard deviation of forward-propagating information to be equal to the value it would have initially in full-precision networks.

Original: In implementation, during training we multiply the sign of the weights in each layer by this value.
Revised: In implementation, during training we multiply the sign of the weights in each layer by this value.

Original: For inference, we do this multiplication using a scaling layer following the weight layer, so that all weights in the network are stored using 0 and 1 , and deployed using ± 1 (see, e.g. https://github.com/SingleBitResNets/SingleBitResNets ).
Revised: For inference, we do this multiplication using a scaling layer following the weight layer, so that all weights in the network are stored using 0 and 1 , and deployed using ± 1 (see https://github.com/SingleBitResNets/SingleBitResNets ).

Original: Hence, custom hardware implementations would be able to implement convolutionsin our trained models without multipliers.
Revised: Hence, custom hardware implementations would be able to perform the model’s convolutions without multipliers (Rastegari et al., 2016).

Original: The fact that we scale the weights explicitly during training is important.
Revised: The fact that we scale the weights explicitly during training is important.

Original: Although for forward and backward propagation it is equivalent to scale the input or output feature maps of a convolutional layer, doing so also scales the calculated gradients with respect toﬁlter weights, since these are calculated by convolving input and output feature maps.
Revised: Although for forward and backward propagation it is equivalent to scale the input or output feature maps of a convolutional layer, doing so also scales the calculated gradients with respect to weights, since these are calculated by convolving input and output feature maps.

Original: As a consequence, learning is dramatically slower unlessan additional adjustment is made to introduce layer-dependent learning rates  to cancel out the scaling,
Revised: As a consequence, learning is dramatically slower unless layer-dependent learning rates are introduced to cancel out the scaling,

Original: Our approach is similar to the BWN method of Rastegari et al.
Revised: Our approach is similar to the BWN method of Rastegari et al.

Original: but our constant scaling method differs in execution complexity, for two reasons. We also found we achieved better results.
Revised: but our constant scaling method differs in execution complexity, and we found we achieved better accuracy.

Original: These aspects are discussed in Section 5.2.

Original: In summary, the only differences we make in comparison with full-precision training are as follows (see also Figure 2).
Revised: In summary, the only differences we make in comparison with full-precision training are as follows.

Original: Let W i be the tensor for the convolutional weights in the i –th convolutional layer.
Revised: Let W i be the tensor for the convolutional weights in the i –th convolutional layer.

Original: Then these weights are processed in the following way only for forward propagation and backward propagation, not for weight updates:
Revised: Then these weights are processed in the following way only for forward propagation and backward propagation, [Figure] weight updates:

Original: [Equation]
Revised: [Equation]

Original: where F i is the spatial size of the convolutional kernel in layer i .
Revised: where F i is the spatial size of the convolutional kernel in layer i ; see Figure 2.

Original: [Figure]
Revised: [Figure]

Original:  M ETHODS COMMON TO BASELINE AND SINGLE - BIT NETWORKS 3. N ETWORK A RCHITECTURE
Revised:  M ETHODS COMMON TO BASELINE AND SINGLE - BIT NETWORKS 3. N ETWORK A RCHITECTURE

Original: Our ResNets use the ‘post-activation’ and identity mapping approach of He et al.
Revised: Our ResNets use the ‘post-activation’ and identity mapping approach of He et al.

Original: (2016) for residual connections.
Revised: (2016) for residual connections.

Revised: For Imagenet, we use an 18-layer design, as in He et al.

Revised: (2015b).

Revised: For all other datasets we mainly use a 20-layer network, but also report some results for 26 layers.

Original: Each residual block includes two convolutional layers, each preceeded by batch normalization (BN) and Rectiﬁed Linear Unit (ReLU) layers.
Revised: Each residual block includes two convolutional layers, each preceded by batch normalization (BN) and Rectiﬁed Linear Unit (ReLU) layers.

Original: Rather than train very deep ResNets, we use wide residual networks (wide ResNets) (Zagoruyko & Komodakis, 2016) with exactlyconvolutional layers.
Revised: Rather than train very deep ResNets, we use wide residual networks (Wide ResNets) (Zagoruyko & Komodakis, 2016).

Original: Although Zagoruyko & Komodakis (2016) and others reported slightly better results on the CIFAR-10 dataset with 28 layers, after we made our other modiﬁcations we were able to match the performance of a 28 layer network with just 20 layers, and no additional width.
Revised: Although Zagoruyko & Komodakis (2016) and others reported that 28/26-layer networks result in better test accuracy than 22/20-layer 2 networks, we found for CIFAR-10/100 that just 20 layers typically produces best results, which is possibly due to our approach of not learning the batch-norm scale and offset parameters.

Original: Our baseline ResNet design  (see Figures 1 and 2) has several differences in comparison to those of He et al.
Revised: Our baseline ResNet design used in most of our experiments (see Figures 1 and 2) has several differences in comparison to those of He et al.

Original: ; Zagoruyko & Komodakis (2016).
Revised: ; Zagoruyko & Komodakis (2016).

Original: These details are articulated in the Appendix, and in combination enable us to obtain slightly improved error rates to comparison Wide ResNets.
Revised: These details are articulated in the Appendix, and are mostly for simplicity, with little impact on accuracy. The exception is our approach of not learning batch-norm parameters. [Figure] [Figure]

Original: 3. T RAINING
Revised: 3. T RAINING

Original: We trained our models following, for most aspects, the standard stochastic gradient descent methods used by Zagoruyko & Komodakis (2016) for wide ResNets.
Revised: We trained our models following, for most aspects, the standard stochastic gradient descent methods used by Zagoruyko & Komodakis (2016) for Wide ResNets.

Original: Speciﬁcally, we use cross-entropy loss, minibatches of size 125,  momentum of 0 . 9 , and  a  weight decay of 0.0005. 
Revised: Speciﬁcally, we use cross-entropy loss, minibatches of size 125, and momentum of 0 . 9 (both for learning weights, and in situations where we learn batch-norm scales and offsets). For CIFAR 10/100, SVHN and MNIST, where overﬁtting is evident in Wide ResNets, we use a larger weight decay of 0.0005. For ImageNet32 and full

Original:  Apart from one set of experiments where we added a simple extra approach called cutout (detailed below), we use standard ‘light’ data augmentation, including randomly ﬂipping each image horizontally with probability 0 .for CIFAR 10/100 and Imagenet32.
Revised: ImageNet we use a weight decay of 0.0001. Apart from one set of experiments where we added a simple extra approach called cutout, we use standard ‘light’ data augmentation, including randomly ﬂipping each image horizontally with probability 0 . 5 for CIFAR 10/100 and ImageNet32.

Original: For the same 3 datasets plus SVHN, we pad by 4 pixels on all sides (using random values between 0 and 255) and crop a 32 × 32 patch from a random location in the resulting 40 × 40 image. 
Revised: For the same 3 datasets plus SVHN, we pad by 4 pixels on all sides (using random values between 0 and 255) and crop a 32 × 32 patch from a random location in the resulting 40 × 40 image. For full

Original: We did not use any image-preprocessing, i.e. we did not subtract [Figure] details are depicted in 
Revised: ImageNet, we scale, crop and ﬂip, as in He et al.

Original: [Figure]
Revised: (2015b).

Original: that the residual summation for such blocks immediately follows downsampling and channel increase operations.

Original: Unlike standard pre-activation ResNets we use an average pooling layer (“Avg pool” ) when downsampling. the mean, as the initial BN layer in the network learns an offset, and we did not use whitening or augment using color or brightness.
Revised: We did not use any image-preprocessing, i.e. we did not subtract the mean, as the initial BN layer in our network performs this role, and we did not use whitening or augment using color or brightness.

Original: We use the initialization method of He et al.
Revised: We use the initialization method of He et al.

Original: (2015a).
Revised: (2015a).

Original: We now describe  differences in our training approach compared to those usually reported.
Revised: We now describe important differences in our training approach compared to those usually reported.

Original: 3.2. B ATCH - NORMALIZATION SCALES AND OFFSETS ARE NOT LEARNT FOR CIFAR-10/
Revised: 3.2. B ATCH - NORM SCALES & OFFSETS ARE NOT LEARNED FOR CIFAR-10/100, SVHN

Original: For reasons explained in Section 5.3, when training on CIFAR-10/100 and SVHN, in all batch-norm layers except the ﬁrst one at the input, we do not learn the scale and offset factor, instead initializating these to 1 and 0 in all channels, and keeping those values through training.
Revised: When training on CIFAR-10/100 and SVHN, in all batch-norm layers (except the ﬁrst one at the input when a ReLU is used there), we do not learn the scale and offset factor, instead initializating these to 1 and 0 in all channels, and keeping those values through training.

Original: Note that we also do not learn any biases for convolutional layers.
Revised: Note that we also do not learn any biases for convolutional layers.

Original: The usual approach to setting the moments for use in batch-norm layers in inference mode is to keep a running average through training.
Revised: The usual approach to setting the moments for use in batch-norm layers in inference mode is to keep a running average through training.

Original: When not learning batch-normalization parameters, we found a small beneﬁt in calculating the batch-normalisation moments used in inference only after training had ﬁnished.
Revised: When not learning batch-normalization parameters, we found a small beneﬁt in calculating the batch-normalisation moments used in inference only after training had ﬁnished.

Original: We simply form as many minibatches as possible from the full training-set, each with the same data augmentation as used during training applied, and pass these through the trained network, averaging the returned moments for each batch.
Revised: We simply form as many minibatches as possible from the full training-set, each with the same data augmentation as used during training applied, and pass these through the trained network, averaging the returned moments for each batch.

Revised: For best results when using this method, we found it necessary to ensure the (cid:15) parameter that is used to avoid divisions by zero is set to 1 × 10 − 5 , which is smaller than the default in some python libraries, such as keras.

Revised: 3. O UR NETWORK ’ S FINAL WEIGHT LAYER IS A 1 × 1 CONVOLUTIONAL LAYER

Revised: A signiﬁcant difference to the ResNets of He et al.

Revised: ; Zagoruyko & Komodakis (2016) is that we exchange the ordering of the global average pooling layer and the ﬁnal weight layer, so that our ﬁnal weight layer becomes a 1 × 1 convolutional layer with as many channels as there are classes in the training set.

Original: 3.2. W E USE A WARM - RESTART LEARNING - RATE SCHEDULE
Revised: This design is not new, but it does seem to be new to ResNets: it corresponds to the architecture of Lin et al. (2013), which originated the global average pooling concept, and also to that used by Springenberg et al. As with all other convolutional layers, we follow this ﬁnal layer with a batch-normalization layer; the beneﬁts of this in conjunction with not learning the batch-normalization scale and offset are described in the Discussion section. 3.3. W E USE A WARM - RESTART LEARNING - RATE SCHEDULE

Original: We use a ‘warm restarts’ learning rate schedule that has reported state-of-the-art wide residual network results (Loshchilov & Hutter, 2016) whilst also speeding up convergence.
Revised: We use a ‘warm restarts’ learning rate schedule that has reported state-of-the-art wide residual network results (Loshchilov & Hutter, 2016) whilst also speeding up convergence.

Original: The method constantly reduces the learning rate from 0.1 to 1 × 10 − 4 according to a cosine decay, across a certain number of epochs, and then repeats across twice as many epochs.
Revised: The method constantly reduces the learning rate from 0.1 to 1 × 10 − 4 according to a cosine decay, across a certain number of epochs, and then repeats across twice as many epochs.

Original: We restricted our attention to a maximum of 254 epochs (often just 62 epochs, and no more than 30 for Imagenet32) using this method, which is the total number of epochs after reducing the learning rate from maximum to minimum through 2 epochs, then 4, 8, 16, 32 and 64 and 128 epochs.
Revised: We restricted our attention to a maximum of 254 epochs (often just 62 epochs, and no more than 30 for ImageNet32) using this method, which is the total number of epochs after reducing the learning rate from maximum to minimum through 2 epochs, then 4, 8, 16, 32 and 64 and 128 epochs.

Original: For CIFAR-10/100, we typically found that we could achieve test error rates after 32 epochs within 1-2% of the error rates achievable after 126 or 254 epochs.
Revised: For CIFAR-10/100, we typically found that we could achieve test error rates after 32 epochs within 1-2% of the error rates achievable after 126 or 254 epochs.

Original: 3.2. E XPERIMENTS WITH CUTOUTDATA AUGMENTATION FOR CIFAR 10/ 
Revised: 3.3. E XPERIMENTS WITH CUTOUT FOR CIFAR 10/

Original: In the literature, most experiments with CIFAR-10 and CIFAR-100 use simple “standard” data augmentation, consisting of randomly ﬂipping each training image left-right with probability 0.5, and padding each image on all sides by 4 pixels, and then cropping a 32 × 32 version of the image from a random location.
Revised: In the literature, most experiments with CIFAR-10 and CIFAR-100 use simple “standard” data augmentation, consisting of randomly ﬂipping each training image left-right with probability 0.5, and padding each image on all sides by 4 pixels, and then cropping a 32 × 32 version of the image from a random location.

Original: We use this augmentation, although with the minor modiﬁcation that we pad with uniform random integers between 0 and 255 , rather than zero-padding.
Revised: We use this augmentation, although with the minor modiﬁcation that we pad with uniform random integers between 0 and 255 , rather than zero-padding.

Original: Additionally, we experimented with “cutout”augmentation (Devries & Taylor, 2017) applied.
Revised: Additionally, we experimented with “cutout” (Devries & Taylor, 2017).

Original: This involves randomly selecting a patch of each raw training image to remove.
Revised: This involves randomly selecting a patch of each raw training image to remove.

Original: The method was shown to combine with other state-of-the-art methods to set the latest state-of-the-art results on CIFAR10/100 (see Table 1).
Revised: The method was shown to combine with other state-of-the-art methods to set the latest state-of-the-art results on CIFAR-10/100 (see Table 1).

Original: Following Devries & Taylor (2017), for  CIFAR-10we choose patches of size 16 × 16 andfor CIFAR-100 we choose patches of size 8 × 8 , while if the chosen pixel is near the image border, the patch impacts on the image only for the part of the patch inside the image.
Revised: We found better results using larger cutout patches for CIFAR-100 than those reported by Devries & Taylor (2017); hence for both CIFAR-10 and CIFAR-100 we choose patches of size 18 × 18 . Following the method of Devries & Taylor (2017), we ensured that all pixels are chosen for being included in a patch equally frequently throughout training by ensuring that if the chosen patch location is near the image border, the patch impacts on the image only for the part of the patch inside the image.

Original: As with padding, we use uniform random integers to replace the image pixel values in the location of the patches.
Revised: Differently to Devries & Taylor (2017), as for our padding, we use uniform random integers to replace the image pixel values in the location of the patches.

Original: We did not apply cutout to other datasets.
Revised: We did not apply cutout to other datasets.

Original:  R ESULTS
Revised:  R ESULTS

Original: We conducted experiments with four databasesconsisting of 32 × 32 RGB images: CIFAR-10, CIFAR-100, SVHN and Imagenet32, as well as MNIST.
Revised: We conducted experiments on six databases: four databases of 32 × 32 RGB images—CIFAR10, CIFAR-100, SVHN and ImageNet32—and the full ILSVRC ImageNet database (Russakovsky et al., 2015), as well as MNIST (LeCun et al., 1998).

Original: Details of the ﬁrst three  datasets can be found in many papers, e.g. Zagoruyko & Komodakis (2016).
Revised: Details of the ﬁrst three 32 × 32 datasets can be found in many papers, e.g. (Zagoruyko & Komodakis, 2016).

Original: Imagenet32 is a downsampled version of Imagenet, where the training and validation images are cropped using their annotated bounding boxes, and then downsampled to 32 × 32 (Chrabaszcz et al., 2017); see http://image-net.org/download-images .
Revised: ImageNet32 is a downsampled version of ImageNet, where the training and validation images are cropped using their annotated bounding boxes, and then downsampled to 32 × 32 (Chrabaszcz et al., 2017); see http://image-net.org/download-images .

Original: All experiments were carried out on either a single Titan X or Xp GPU using matlab with GPU acceleration from matconvnet and CuDNN.
Revised: All experiments were carried out on either a single Titan X or Xp GPU using MATLAB with GPU acceleration from MatConvNet and cuDNN.

Original: Using the Titan Xp, we were able to train our 27.4 million parameters 20-10 ResNet on Imagenet in just 35 hours total for 30 epochs. Wemostly report results for networks with between 25 and 40 million parameters, given state-ofthe-art networks for these four datasets fall into this bracket.
Revised: We report results for Wide ResNets, which (except when applied to ImageNet) are 4 × and 10 × wider than baseline ResNets, to use the terminology of Zagoruyko & Komodakis (2016), where the baseline has 16 channels in the layers at the ﬁrst spatial scale. We use notation of the form 20to denote Wide ResNets with 20 convolutional layers and 160 channels in the ﬁrst spatial scale, and hence width 10 × .

Original: To this end, wemainly use wide ResNets that are 4 × and 10 × wider than baseline ResNets, to use the terminology of Zagoruyko & Komodakis (2016), where the baseline has 16 channels in thelayers at the ﬁrst spatial scale.
Revised: For the full ImageNet dataset, we use 18-layer Wide ResNets with 160 channels in the ﬁrst spatial scale, but given the standard ResNet baseline is width 64, this corresponds to width 2.5 × on this dataset Zagoruyko & Komodakis (2016).

Original: Weuse notation of the form 20-10 to denote wide ResNets with 20 convolutional layers and width 4 × .
Revised: We denote these networks as 18-2.5.

Original: Table 1 lists our top-1 error rates for CIFAR 10/100; C10 indicates CIFAR-10, C100 indicates
Revised: Table 1 lists our top-1 error rates for CIFAR 10/100; C10 indicates CIFAR-10, C100 indicates

Original: CIFAR100. In Table 1, the superscript + indicates standard crop and ﬂip augmentation, whereas ++ indicates additional cutout augmentation.
Revised: CIFAR100; the superscript + indicates standard crop and ﬂip augmentation; and ++ indicates the use of cutout.

Original: Table 2 lists  top-1 and top-5 error rates for SVHN and 
Revised: Table 2 lists error rates for SVHN, ImageNet32 (I32) and full ImageNet; we did not use cutout on these datasets. Both top-1 and top-5 results are tabulated for I32 and ImageNet. For

Original: Imagenet32 (I32), as  for both of these we trained for only 30 epochs.
Revised: ImageNet, we provide results for single-center-crop testing, and also for multi-crop testing. In the latter, the decision for each test image is obtained by averaging the softmax output after passing through the network 25 times, corresponding to crops obtained by rescaling to 5 scales as described by He et al. (2015b), and from 5 random positions at each scale. Our full-precision ImageNet error rates are slightly lower than expected for a wide ResNet according to the results of Zagoruyko &

Original:  Table 3 shows comparison results from the original work on wide ResNets, and subsequent papers that have reduced error rates on the CIFAR-10 and CIFAR-100 datasets.
Revised: Komodakis (2016), probably due to the fact we did not use color augmentation. [Table] [Table] Table 3 shows comparison results from the original work on Wide ResNets

Revised: [Table]

Revised: and subsequent papers that have reduced error rates on the CIFAR-10 and CIFAR-100 datasets.

Original: We also show the only results we know of for 32 × 32 Imagenet.
Revised: We also show the only results we know of for ImageNet32.

Original: The current state-of-the-art for SVHN without augmentation is 1 .
Revised: The current state-of-the-art for SVHN without augmentation is 1 .

Original: 59 % (Huang et al., 2016), and with cutout augmentation is 1 .
Revised: 59 % (Huang et al., 2016), and with cutout augmentation is 1 .

Original: 30 % (Devries & Taylor, 2017).
Revised: 30 % (Devries & Taylor, 2017).

Original: Our full-precision result for SVHN (1.75%) is only a little short of these even though we used only a 4 × ResNet, with less than 5 million parameters, and only 30 training epochs. [Table]
Revised: Our full-precision result for SVHN (1.75%) is only a little short of these even though we used only a 4 × ResNet, with less than 5 million parameters, and only 30 training epochs

Original: [Table]
Revised: [Table]

Original: Table 4 shows comparison results for previous work that trains models by using the sign of weights during training.
Revised: Table 4 shows comparison results for previous work that trains models by using the sign of weights during training.

Original: Additional results appear in Hubara et al.
Revised: Additional results appear in Hubara et al.

Original: (2016), where activations are quantized, and so the error rates are much larger [Table]
Revised: [Table] where activations are quantized, and so the error rates are much larger.

Original: [Table] Inspection of Tables 1 to 4 reveals that our10 × baseline full-precision  networks, when trained with cutout, surpass the performance of much larger wide ResNets trained with dropout.
Revised: Inspection of Tables 1 to 4 reveals that our baseline full-precision 10 × networks, when trained with cutout, surpass the performance of deeper Wide ResNets trained with dropout.

Original: Even without the use of cutout [Table] our 20-10 network surpasses by over 2% the CIFAR-100 error rate reported for essentially the same network by Zagoruyko & Komodakis (2016) and is marginally better on CIFAR10 andImageNet
Revised: Even without the use of cutout, our 20-10 network surpasses by over 2% the CIFAR-100 error rate reported for essentially the same network by Zagoruyko & Komodakis (2016) and is marginally better on CIFAR-10 and

Original: [Table]
Revised: ImageNet32. As elaborated on in Section 5.3, this improved accuracy is due to our approach of not learning the batch-norm scale and offet parameters.

Original: For our 1-bit networks, we observe that there is always an accuracy gap compared to full precision networks.
Revised: For our 1-bit networks, we observe that there is always an accuracy gap compared to full precision networks.

Original: As discussed in Section 5.1, this gap tends to grow as the error-rate in the full-precision network grows.
Revised: As discussed in Section 5.1, this gap tends to grow as the error-rate in the full-precision network grows.

Original: The use of cutout reduced error rates on CIFAR-10/100 in most cases.
Revised: However, it is interesting that for our 26-layer networks applied to CIFAR 10/100, that the gap is much smaller, despite almost no difference due to extra depth in the full precision case. [Table] [Table] This Paper:

Revised: [Table]

Revised: crop

Revised: [Table]

Revised: This Paper: 5 scales,

Revised: [Table]

Revised: crops

Revised: [Table]

Original:  In comparison with training  20 × ResNets on CIFAR-100, as shown in Table 1, it is as effective to use cutout augmentation in the 10 × network to reduce the error rate, but only a quarter of the weights are required.
Revised: Using cutout reduces error rates on CIFAR-10/100 in most cases. In comparison with training very wide 20 × ResNets on CIFAR-100, as shown in Table [Table] it is equally effective to use cutout augmentation in the 10 × network to reduce the error rate, while using only a quarter of the weights.

Original: Figure 5 demonstrates convergence and overﬁtting trends for CIFAR-10/100 for 20-4 wide ResNets, and a comparison of the use of cutout in 20-10 wide ResNets.
Revised: Figure 5 demonstrates convergence and overﬁtting trends for CIFAR-10/100 for 20-4 Wide ResNets, and a comparison of the use of cutout in 20-10 Wide ResNets.

Original: Clearly, even for width-4 ResNets, the gap in error rate between full precision weights and single-bit weights is small.
Revised: Clearly, even for width-4 ResNets, the gap in error rate between full precision weights and 1-bit-per-weight is small.

Original: Also noticeable is that the warm-restart method enables convergence to very good solutions after just 30 epochs; training longer to 126 epochs reduces  error rates further by between 2% and 5%.
Revised: Also noticeable is that the warm-restart method enables convergence to very good solutions after just 30 epochs; training longer to 126 epochs reduces test error rates further by between 2% and 5%.

Original: It can also be observed that the 20-4 network is powerful enough to model the CIFAR-10/100 training sets to well[Table] over 99% accuracy, but the modelling power is reduced in the 1-bit version, particularly for CIFAR100.
Revised: It can also be observed that the 20-4 network is powerful enough to model the CIFAR-10/100 training sets to well over 99% accuracy, but the modelling power is reduced in the 1-bit version, particularly for CIFAR100.

Original: The reduced modelling capacity for single-bit weights is consistent with the gap in test-error rate performance between the 32-bit and 1-bit cases.
Revised: The reduced modelling capacity for single-bit weights is consistent with the gap in test-error rate performance between the 32-bit and 1-bit cases.

Original: When using cutout, training for longer gives improved error rates, but when not using cutout, 126 epochs sufﬁces for peak performance.
Revised: When using cutout, training for longer gives improved error rates, but when not using cutout, 126 epochs sufﬁces for peak performance.

Original: Finally, for MNIST and a 4 × wide ResNet without any data augmentation, our full-precision method achieved 0.71% after just 1 epoch of training, and 0.28% after 6 epochs, whereas our 1-bit method achieved 0.81% and 0.36%after 1 and6 epochs of training. For the 1-bit version we also observed 0.27% after  14 epochs.
Revised: Finally, for MNIST and a 4 × wide ResNet without any data augmentation, our full-precision method achieved 0.71% after just 1 epoch of training, and 0.28% after 6 epochs, whereas our 1-bit method achieved 0.81%, 0.36% and 0.27% after 1, 6 and 14 epochs.

Original: In comparison, 1.29% was reported for the 1-bit case by Courbariaux et al.
Revised: In comparison, 1.29% was reported for the 1-bit-per-weight case by Courbariaux et al.

Original: (2015), and 0.96% by Hubara et al.
Revised: (2015), and 0.96% by Hubara et al.

Revised: 4. A BLATION STUDIES

Original: [Figure]
Revised: For CIFAR 10/100, both our full-precision Wide ResNets and our 1-bit-per-weight versions beneﬁt from our method of not learning batch-norm scale and offset parameters. Accuracy in the 1-bit-perweight case also beneﬁts from our use of a warm-restart training schedule (Loshchilov & Hutter, 2016). To demonstrate the inﬂuence of these two aspects, in Figure 4 we show, for CIFAR-100, how the test error rate changes through training when either or both of these methods are not used. We did not use cutout for the purpose of this ﬁgure.

Original: i f i c a t i on
Revised: The comparison learning-rate-schedule drops the learning rate from 0.1 to 0.01 to 0.001 after 85 and 170 epochs. It is clear that our methods lower the [Figure] ﬁnal error rate by around 3% absolute by not enables faster convergence for the full-precision case

Original: [Figure]
Revised: [Figure]

Original: rr o
Revised: but is not signiﬁcant in reducing error rate. However, for weight is that best use warm-

Original: [Figure]
Revised: [Figure]

Original: t e

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION 5. T HE ACCURACY GAP FOR 1- BIT VS 32- BIT WEIGHTS
Revised:  D ISCUSSION 5. T HE ACCURACY GAP FOR 1- BIT VS 32- BIT WEIGHTS

Original: It is to be expected that a smaller error rate gap will result between the same network using fullprecision and 1-bit weights when the test error rate on the full precision network gets smaller.
Revised: It is to be expected that a smaller error rate gap will result between the same network using fullprecision and 1-bit-per-weight when the test error rate on the full precision network gets smaller.

Original: However, how the magnitude of the gap changes with full precision error rate is dependent on many factors, including the method used to generate models with 1-bit weights.
Revised: However, how the magnitude of the gap changes with full precision error rate is dependent on many factors, including the method used to generate models with 1-bit-per-weight.

Original: To illustrate the trend for our approach [Figure] we have plotted in Figure 1 the gap in Top-1 error rates vs the Top-1 error rate for the full precision case, for some of the best performing networks for the ﬁve datasets we used.
Revised: To illustrate the trend for our approach, we have plotted in Figure 1 the gap in Top-1 error rates vs the Top-1 error rate for the full precision case, for some of the best performing networks for the six datasets we used.

Original: Conclusions from this data can only be made relative to alternative methods.
Revised: Strong conclusions from this data can only be made relative to alternative methods, but ours is the ﬁrst study to our knowledge to consider more than two datasets.

Original: Hence, we have also plotted in Figure 1 the error rate and the gap reported by Rastegari et al.
Revised: Nevertheless, we have also plotted in Figure 1 the error rate and the gap reported by Rastegari et al.

Original: (2016) for two different networks using their BWN 1-bit weight method.
Revised: (2016) for two different networks using their BWN 1-bit weight method.

Original: It is clear that the gaps in those

Original: However, a fair comparison requires further experiments seeking conﬁrmation of whether our method applied to the full Imagenet dataset produces a gap closer to the line connecting our results for the ﬁve datasets.

Original: These experiments are currently underway. In preliminary work, for a 16-8 ResNet with a different architecture to  that used elsewhere in this paper (as well as requiringscales), we achieved single-centre-crop Top-1 error rates of 32.87% and 36.44% on the full precision and 1-bit weights versions, using 30 epochs of the warm-restart method. This gap ofabout 3.5%, as shown in Figure 1, is about half the gaps reported by Rastegari et al.(2016), similar to the reduction in gaps shown in Table 4.
Revised: The larger gaps for those points are consistent with our experiments that indicated to us that our simple scaling method outperforms the scaling method of Rastegari et al.

Original: A challenges for further work is to derive theoretical bounds that predict the gap.
Revised: A challenge for further work is to derive theoretical bounds that predict the gap.

Original: For high-error rate cases, the loss function throughout training is much higher for the 1-bit case than the 32-bit case, and hence, the 1-bit network is not able to ﬁt the training set as well as the 32-bit one.
Revised: For high-error rate cases, the loss function throughout training is much higher for the 1-bit case than the 32-bit case, and hence, the 1-bit-per-weight network is not able to ﬁt the training set as well as the 32-bit one.

Original: Whether this is because of the loss of precision in weights, or due to the mismatch in gradients inherent is propagating with 1-bit weights and updating full-precision weights during training is an open question.
Revised: Whether this is because of the loss of precision in weights, or due to the mismatch in gradients inherent is propagating with 1-bit weights and updating full-precision weights during training is an open question.

Original: If it is the latter case, then it is possible that principled reﬁnements to the weight update method we used will further reduce the gap.
Revised: If it is the latter case, then it is possible that principled reﬁnements to the weight update method we used will further reduce the gap.

Original: 5. C OMPARISON WITH THE BWN METHOD
Revised: 5. C OMPARISON WITH THE BWN METHOD

Original: Our approach differs from the BWN method of Rastegari et al.
Revised: Our approach differs from the BWN method of Rastegari et al.

Original: (2016) for two reasons.
Revised: (2016) for two reasons.

Original: First, we do not need to calculate mean absolute weight values of the underlying full precision weights for each output channel in each layer following each minibatch, and this enables faster training.
Revised: First, we do not need to calculate mean absolute weight values of the underlying full precision weights for each output channel in each layer following each minibatch, and this enables faster training.

Original: Second, we do not need to adjust for a gradient term corresponding to the appearance of each weight in the mean absolute value.
Revised: Second, we do not need to adjust for a gradient term corresponding to the appearance of each weight in the mean absolute value.

Original: Moreover, although we observed that the method of Rastegari et al.
Revised: Moreover, although we observed that the method of Rastegari et al.

Original: (2016) converges slightly faster early in training, by the end of training our simpler method gave better results for CIFAR-10 and
Revised: (2016) converges slightly faster early in training, by the end of training our simpler method gave better results for CIFAR-10 and

Original: CIFAR-100, and signiﬁcantly better results for Imagenet32.
Revised: CIFAR-100, and signiﬁcantly better results for ImageNet32.

Original: We speculate that this is because the scaling method of Rastegari et al.
Revised: We speculate that this is because the scaling method of Rastegari et al.

Original: (2016) assumes that mean square error between full precision weights and scaled binary weights is the best measure for determining scaling; for practical purposes, minimising mean square error does not necessarily mean better classiﬁcation error rates.
Revised: (2016) assumes that mean square error between full precision weights and scaled binary weights is the best measure for determining scaling; for practical purposes, minimising mean square error does not necessarily mean better classiﬁcation error rates.

Original: It is also possible that combining the adaptive method with weight decay has undesirable consequences.
Revised: It is also possible that combining the adaptive method with weight decay has undesirable consequences.

Original: In our method, since the scale factors never change, weight decay only inﬂuences the underlying full-precision weights that get updated, not the propagation of data.
Revised: In our method, since the scale factors never change, weight decay only inﬂuences the underlying full-precision weights that get updated, not the propagation of data.

Original: 5. T HE INFLUENCE OF NOT LEARNING BATCH - NORMALISATION PARAMETERS
Revised: 5. T HE INFLUENCE OF NOT LEARNING BATCH - NORMALISATION PARAMETERS

Original: The unusual design choice of not learning the batch normalization parameters was made for CIFAR10/100, SVHN and MNIST because for wide ResNets, overﬁtting is very evident on these datasets (see Figure 5); by the end of training, typically the loss function becomes very close to zero, corresponding to severe overﬁtting.
Revised: The unusual design choice of not learning the batch normalization parameters was made for CIFAR10/100, SVHN and MNIST because for Wide ResNets, overﬁtting is very evident on these datasets (see Figure 5); by the end of training, typically the loss function becomes very close to zero, corresponding to severe overﬁtting.

Original: Inspired by label-smoothing regularization (Szegedy et al., 2015) that aims to reduce overconﬁdence following the softmax layer, we hypothesised that imposing more control over the standard deviation of inputs to the softmax might have a similar regularizing effect.
Revised: Inspired by label-smoothing regularization (Szegedy et al., 2015) that aims to reduce overconﬁdence following the softmax layer, we hypothesised that imposing more control over the standard deviation of inputs to the softmax might have a similar regularizing effect.

Original: This is why we removed the ﬁnal all-to-all layer of most ResNets and replaced it with a 1 × 1 convolutional layer followed by a batch-normalization layer prior to the global average pooling layer.
Revised: This is why we removed the ﬁnal all-to-all layer of our ResNets and replaced it with a 1 × 1 convolutional layer followed by a batch-normalization layer prior to the global average pooling layer.

Original: In turn, not learning a scale and bias for this batch-normalization layer ensures that batches ﬂowing into the softmax layer have standard deviations that do not grow throughout training, which tends to increase the entropy of predictions following the softmax, which is equivalent to lower conﬁdence (Guo et al., 2017).
Revised: In turn, not learning a scale and offset for this batch-normalization layer ensures that batches ﬂowing into the softmax layer have standard deviations that do not grow throughout training, which tends to increase the entropy of predictions following the softmax, which is equivalent to lower conﬁdence (Guo et al., 2017).

Original: After observing success with these methods in 10 × wide ResNets, we then observed that learning the batch-norm parameters in other layers also led to increased overﬁtting, and increased test error rates, and so removed that learning in all layers except the ﬁrst one applied to the input.
Revised: After observing success with these methods in 10 × Wide ResNets, we then observed that learning the batch-norm parameters in other layers also led to increased overﬁtting, and increased test error rates, and so removed that learning in all layers except the ﬁrst one applied to the input.

Revised: As shown in Figure 6, there are signiﬁcant beneﬁts from this approach, for both full-precision and 1bit-per-weight networks.

Revised: It is why, in Table 3, our results surpass those of Zagoruyko & Komodakis (2016) on effectively the same Wide ResNet (our 20-10 network is essentially the same as the 22comparison network, where the extra 2 layers appear due to the use of learnt 1 × 1 convolutional projections in downsampling residual paths, whereas we use average pooling instead0.

Original: As expected from the motivation, we found this method is not appropriate for datasets such as ImageNet where overﬁtting is not as evident, in which case learning the batch normalization parameters can signiﬁcantly reduce test error rates.
Revised: As expected from the motivation, we found our method is not appropriate for datasets such as ImageNet32 where overﬁtting is not as evident, in which case learning the batch normalization parameters can signiﬁcantly reduce test error rates.

Original: 5. L IMITATIONS AND F URTHER W ORK
Revised: 5. C OMPARISON WITH S QUEEZE N ET

Revised: Here we compare our approach with SqueezeNet (Iandola et al., 2016), which reported signiﬁcant memory savings for a trained model relative to an AlexNet.

Revised: The SqueezeNet approach uses two strategies to achieve this: (1) replacing many 3x3 kernels with 1x1 kernels; (2) deep compression (Han et al., 2015).

Revised: Regarding SqueezeNet strategy (1), we note that SqueezeNet is an all-convolutional network that closely resembles the ResNets used here.

Revised: We experimented brieﬂy with our 1-bit-per-weight approach in many all-convolutional variants—e.g. plain all-convolutional (Springenberg et al., 2014),

Revised: SqueezeNet (Iandola et al., 2016), MobileNet (Howard et al., 2017), ResNexT (Xie et al., 2016)— and found its effectiveness relative to full-precision baselines to be comparable for all variants.

Revised: We also observed in many experiments that the total number of learnt parameters correlates very well with classiﬁcation accuracy.

Revised: When we applied a SqueezeNet variant to CIFAR-100, we found that to obtain the same accuracy as our ResNets for about the same depth, we had to increase the width until the SqueezeNet had approximately the same number of learnt parameters as the ResNet.

Revised: We conclude that our method therefore reduces the model size of the baseline SqueezeNet architecture (i.e. when no deep compression is used) by a factor of 32, albeit with an accuracy gap.

Revised: Regarding SqueezeNet strategy (2), the SqueezeNet paper reports deep compression was able to reduce the model size by approximately a factor of 10 with no accuracy loss.

Original: In this paper we focus only on the memory required to store a trained model, i.e. model  compression .
Revised: Our method reduces the same model size by a factor of 32, but with a small accuracy loss that gets larger as the fullprecision accuracy gets smaller. It would be interesting to explore whether deep compression might be applied to our 1-bit-per-weight models, but our own focus is on methods that minimally alter training, and we leave investigation of more complex methods for future work. Regarding SqueezeNet performance, the best accuracy reported in the SqueezeNet paper is 39.6% top-1 error, requiring 4.8MB for the model’s weights. Our single-bit-weight models achieve better than 33% top-1 error, and require 8.3 MB for the model’s weights.

Revised: 5. L IMITATIONS AND F URTHER W ORK

Revised: In this paper we focus only on reducing the precision of weights to a single-bit, with beneﬁts for model compression, and enabling of inference with very few multiplications.

Original: It is also interesting and desirable to reduce the computational load of inference using a trained model, by carrying out layer computations in very few numbers of bits (Hubara et al., 2016; 
Revised: It is also interesting and desirable to reduce the computational load of inference using a trained model, by carrying out layer computations in very few numbers of bits (Hubara et al., 2016; Rastegari et al., 2016;

Original: Rastegari et al., 2016; Cai et al., 2017).
Revised: Cai et al., 2017).

Original: Facilitating this requires modifying non-linear activations in a network from ReLUs to quantized ReLUs, or in the extreme case, binary step functions.
Revised: Facilitating this requires modifying non-linear activations in a network from ReLUs to quantized ReLUs, or in the extreme case, binary step functions.

Original: Here we use only full-precision calculations.
Revised: Here we use only full-precision calculations.

Original: It can be expected that combining our methods with reduced precision processing will inevitably increase error rates.
Revised: It can be expected that combining our methods with reduced precision processing will inevitably increase error rates.

Original: We have addressed this extension in a forthcoming submission
Revised: We have addressed this extension in a forthcoming submission.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Removed for double-blind peer-review.
Revised: Removed for double-blind peer-review.

Original: A D IFFERENCES FROM S TANDARD R ES N ETS IN OUR B ASELINE A. O UR NETWORK ’ S FINAL WEIGHT LAYER IS A 1 × 1 CONVOLUTIONAL LAYER
Revised: A D IFFERENCES FROM S TANDARD R ES N ETS IN OUR B ASELINE A. O UR NETWORK DOWNSAMPLES THE RESIDUAL PATH USING AVERAGE POOLING

Original: The most signiﬁcant difference to He et al.

Original: ; Zagoruyko & Komodakis (2016) is we exchange the ordering of the global average pooling layer and the ﬁnal weight layer; our ﬁnal weight layer becomes a 1 × 1 convolutional layer with as many channels as there are classes in the training set.

Original: This design is not new, but it does seem to be new to ResNets: it corresponds to the architecture of Lin et al.

Original: (2013), which originated the global average pooling concept and also to that used by Springenberg et al.

Original: Using this method means we entirely follow the “all-convolutional” concept of Springenberg et al.

Original: A. O UR NETWORK DOWNSAMPLES THE RESIDUAL PATH USING AVERAGE POOLING

Original: For skip connections between feature maps of different sizes, we use zero-padding for increasing the number of channels as per option 1 of He et al.
Revised: For skip connections between feature maps of different sizes, we use zero-padding for increasing the number of channels as per option 1 of He et al.

Original: (2015b).
Revised: (2015b).

Original: However, for the residual pathway, we use average pooling using a kernel of size 3 × 3 with stride 2 for downsampling, instead of typical lossy downsampling that discards all pixel values in between samples.
Revised: However, for the residual pathway, we use average pooling using a kernel of size 3 × 3 with stride 2 for downsampling, instead of typical lossy downsampling that discards all pixel values in between samples.

Original: We found in some cases this reduced error rates by around 1%.

Original: A. W E USE BATCH NORMALIZATION APPLIED TO THE INPUT LAYER
Revised: A. W E USE BATCH NORMALIZATION APPLIED TO THE INPUT LAYER

Original: The literature has reported various options for the optimal ordering, usage and placement of BN and
Revised: The literature has reported various options for the optimal ordering, usage and placement of BN and

Original: ReLU layers in residual networks.
Revised: ReLU layers in residual networks.

Original: Following He et al.
Revised: Following He et al.

Original: (2016); Zagoruyko & Komodakis (2016)
Revised: (2016); Zagoruyko & Komodakis (2016)

Original: , we precede convolutional layers with the combination of BN followed by ReLU.
Revised: , we precede convolutional layers with the combination of BN followed by ReLU.

Original: However, different to He et al.
Revised: However, different to He et al.

Original: (2016); Zagoruyko & Komodakis (2016)
Revised: (2016); Zagoruyko & Komodakis (2016)

Original: , we also insert the BNReLU combination immediately after the input layer and before the ﬁrst convolutional layer.
Revised: , we also insert the BN (and optional ReLU) immediately after the input layer and before the ﬁrst convolutional layer.

Original: Unlike all other batch-normalization layers, we enable learning of the scale and bias factors in this ﬁrst batch-norm, which enables us to avoid doing any pre-processing on the inputs to the network, since the BN layer learns how to scale and bias each of the RGB channels; we found that thebias learnt  ensures the input to the ﬁrst ReLU is never negative.
Revised: When the optional ReLU is used, unlike all other batch-normalization layers, we enable learning of the scale and offset factors in this ﬁrst batch-norm. This ﬁrst BN layer enables us to avoid doing any pre-processing on the inputs to the network, since the BN layer provides necessary normalization. When the optional ReLU us included, we found that the learnt offset ensures the input to the ﬁrst ReLU is never negative.

Original: In accordance with our strategy of simplicity,without exception all weight layers can be thought of as a block of three operations in the same sequence, as indicated in Figure 1.
Revised: In accordance with our strategy of simplicity, all weight layers can be thought of as a block of three operations in the same sequence, as indicated in Figure 1.

Original: Conceptually, batchnorm followed by ReLU can be thought of as a single layer consisting of a ReLU that adaptively changes its centre point and positive slope for each channel and relative to each mini-batch.
Revised: Conceptually, batch-norm followed by ReLU can be thought of as a single layer consisting of a ReLU that adaptively changes its centre point and positive slope for each channel and relative to each mini-batch.

Original: We also precede the global average pooling layer by a BN layer, but do not use a ReLU at this point, since nonlinear activation is provided by the softmax layer that follows immediately after global average pooling.
Revised: We also precede the global average pooling layer by a BN layer, but do not use a ReLU at this point, since nonlinear activation is provided by the softmax layer.

Original: We found including the ReLU leads to differences early in training but not by the completion of training.
Revised: We found including the ReLU leads to differences early in training but not by the completion of training.

Original: A. O UR FIRST CONV LAYER HAS AS MANY CHANNELS AS THE FIRST RESIDUAL BLOCK
Revised: A. O UR FIRST CONV LAYER HAS AS MANY CHANNELS AS THE FIRST RESIDUAL BLOCK

Original: The wide ResNet of Zagoruyko & Komodakis (2016) is speciﬁed asalways having a  constant number of output channels, even when the number of output channels for other layers increases.
Revised: The Wide ResNet of Zagoruyko & Komodakis (2016) is speciﬁed as having a ﬁrst convolutional layer that always has a constant number of output channels, even when the number of output channels for other layers increases.

Original: We found no need to impose this constraint, and instead always allow the ﬁrst layer to share the same number of output channels as all blocks at the ﬁrst spatial scale.
Revised: We found no need to impose this constraint, and instead always allow the ﬁrst layer to share the same number of output channels as all blocks at the ﬁrst spatial scale.

Original: The increase in the total number of parameters from doing this is small relative to the total number of parameters, since the number of input channels to the ﬁrst layer is just 3.
Revised: The increase in the total number of parameters from doing this is small relative to the total number of parameters, since the number of input channels to the ﬁrst layer is just 3.

Original: The beneﬁt of this change is increased simplicity in the network deﬁnition, and allow one fewer change in the dimensional of the residualskip pathway.
Revised: The beneﬁt of this change is increased simplicity in the network deﬁnition, by ensuring one fewer change in the dimensionality of the residual pathway.

Original: B C OMPARISON OF R ESIDUAL N ETWORKS AND P LAIN A LL -C ONVOLUTIONAL N ETWORKS
Revised: B C OMPARISON OF R ESIDUAL N ETWORKS AND P LAIN CNN S

Original: We were interested in understanding whether the good results we achieved for single-bit weights were a consequence of the skip connections in residual networks.
Revised: We were interested in understanding whether the good results we achieved for single-bit weights were a consequence of the skip connections in residual networks.

Original: We therefore applied our method to plain all-convolutional networks identical to our 4 × residual networks, except with the skip connections removed.
Revised: We therefore applied our method to plain all-convolutional networks identical to our 4 × residual networks, except with the skip connections removed.

Original: Initially, training indicated a much slower convergence, but we found that altering the initial weights standard deviations to be proportional to 2 instead of √ 2 helped, so this was the only other change made.
Revised: Initially, training indicated a much slower convergence, but we found that altering the initial weights standard deviations to be proportional to 2 instead of √ 2 helped, so this was the only other change made.

Original: The change was also applied in Equation (1).
Revised: The change was also applied in Equation (1).

Original: As summarised in Figure 6, we found that convergence remained slower than our ResNets, but there was only a small accuracy penalty in comparison with ResNets after 126 epochs of training.
Revised: As summarised in Figure 7, we found that convergence remained slower than our ResNets, but there was only a small accuracy penalty in comparison with ResNets after 126 epochs of training.

Original: This is consistent with the ﬁndings of He et al.
Revised: This is consistent with the ﬁndings of He et al.

Original: (2015b) where only ResNets deeper than about 20 layers showed a signiﬁcant advantage over plain all-convolutional networks.
Revised: (2015b) where only ResNets deeper than about 20 layers showed a signiﬁcant advantage over plain all-convolutional networks.

Original: We conclude that our method is not particular to ResNets.
Revised: This experiment, and others we have done, support our view that our method is not particular to ResNets.

Original: [Figure]
Revised: [Figure]

Original: [Figure]