Original: T RAINING WIDE RESIDUAL NETWORKS FOR DEPLOYMENT USING A SINGLE BIT FOR EACH WEIGHT
Revised: T RAINING WIDE RESIDUAL NETWORKS FOR DEPLOYMENT USING A SINGLE BIT FOR EACH WEIGHT

Original: A BSTRACT
Revised: A BSTRACT

Original: For fast and energy-efﬁcient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit.
Revised: For fast and energy-efﬁcient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit.

Original: Error-rates usually increase when this requirement is imposed.
Revised: Error-rates usually increase when this requirement is imposed.

Original: Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight.
Revised: Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight.

Original: Using wide residual networks as our main baseline, our approach simpliﬁes existing methods that binarize weights by applying the sign function in training; we apply scaling factors for each layer with constant unlearned values equal to the layer-speciﬁc standard deviations used for initialization.
Revised: Using wide residual networks as our main baseline, our approach simpliﬁes existing methods that binarize weights by applying the sign function in training; we apply scaling factors for each layer with constant unlearned values equal to the layer-speciﬁc standard deviations used for initialization.

Original: For CIFAR-10, CIFAR-100 and ImageNet, and models with 1-bitper-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively.
Revised: For CIFAR-10, CIFAR-100 and ImageNet, and models with 1-bitper-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively.

Original: We also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test results of 0.27%, 1.9%, and 41.3% / 19.1% respectively.
Revised: We also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test results of 0.27%, 1.9%, and 41.3% / 19.1% respectively.

Original: For CIFAR, our error rates halve previously reported values, and are within about 1% of our errorrates for the same network with full-precision weights.
Revised: For CIFAR, our error rates halve previously reported values, and are within about 1% of our errorrates for the same network with full-precision weights.

Original: For networks that overﬁt, we also show signiﬁcant improvements in error rate by not learning batch normalization scale and offset parameters.
Revised: For networks that overﬁt, we also show signiﬁcant improvements in error rate by not learning batch normalization scale and offset parameters.

Original: This applies to both full precision and 1-bit-per-weight networks.
Revised: This applies to both full precision and 1-bit-per-weight networks.

Original: Using a warm-restart learning-rate schedule, we found that training for 1-bit-per-weight is just as fast as full-precision networks, with better accuracy than standard schedules, and achieved about 98%- 99% of peak performance in just 62 training epochs for CIFAR-10/100.
Revised: Using a warm-restart learning-rate schedule, we found that training for 1-bit-per-weight is just as fast as full-precision networks, with better accuracy than standard schedules, and achieved about 98%- 99% of peak performance in just 62 training epochs for CIFAR-10/100.

Original: For full training code and trained models in MATLAB, Keras and PyTorch see https://github.com/McDonnell-Lab/1-bit-per-weight/ .
Revised: For full training code and trained models in MATLAB, Keras and PyTorch see https://github.com/McDonnell-Lab/1-bit-per-weight/ .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Fast parallel computing resources, namely GPUs, have been integral to the resurgence of deep neural networks, and their ascendancy to becoming state-of-the-art methodologies for many computer vision tasks.
Revised: Fast parallel computing resources, namely GPUs, have been integral to the resurgence of deep neural networks, and their ascendancy to becoming state-of-the-art methodologies for many computer vision tasks.

Original: However, GPUs are both expensive and wasteful in terms of their energy requirements.
Revised: However, GPUs are both expensive and wasteful in terms of their energy requirements.

Original: They typically compute using single-precision ﬂoating point (32 bits), which has now been recognized as providing far more precision than needed for deep neural networks.
Revised: They typically compute using single-precision ﬂoating point (32 bits), which has now been recognized as providing far more precision than needed for deep neural networks.

Original: Moreover, training and deployment can require the availability of large amounts of memory, both for storage of trained models, and for operational RAM.
Revised: Moreover, training and deployment can require the availability of large amounts of memory, both for storage of trained models, and for operational RAM.

Original: If deep-learning methods are to become embedded in resourceconstrained sensors, devices and intelligent systems, ranging from robotics to the internet-of-things to self-driving cars, reliance on high-end computing resources will need to be reduced.
Revised: If deep-learning methods are to become embedded in resourceconstrained sensors, devices and intelligent systems, ranging from robotics to the internet-of-things to self-driving cars, reliance on high-end computing resources will need to be reduced.

Original: To this end, there has been increasing interest in ﬁnding methods that drive down the resource burden of modern deep neural networks.
Revised: To this end, there has been increasing interest in ﬁnding methods that drive down the resource burden of modern deep neural networks.

Original: Existing methods typically exhibit good performance but for the ideal case of single-bit parameters and/or processing, still fall well-short of state-of-the-art error rates on important benchmarks.
Revised: Existing methods typically exhibit good performance but for the ideal case of single-bit parameters and/or processing, still fall well-short of state-of-the-art error rates on important benchmarks.

Original: In this paper, we report a signiﬁcant reduction in the gap (see Figure 1 and Results) between Convolutional Neural Networks (CNNs) deployed using weights stored and applied using standard precision (32-bit ﬂoating point) and networks deployed using weights represented by a single-bit each.
Revised: In this paper, we report a signiﬁcant reduction in the gap (see Figure 1 and Results) between Convolutional Neural Networks (CNNs) deployed using weights stored and applied using standard precision (32-bit ﬂoating point) and networks deployed using weights represented by a single-bit each.

Original: In the process of developing our methods, we also obtained signiﬁcant improvements in error-rates obtained by full-precision versions of the CNNs we used.
Revised: In the process of developing our methods, we also obtained signiﬁcant improvements in error-rates obtained by full-precision versions of the CNNs we used.

Original: In addition to having application in custom hardware deploying deep networks, networks deployed using 1-bit-per-weight have previously been shown (Pedersoli et al., 2017) to enable signiﬁcant speedups on regular GPUs, although doing so is not yet possible using standard popular libraries.
Revised: In addition to having application in custom hardware deploying deep networks, networks deployed using 1-bit-per-weight have previously been shown (Pedersoli et al., 2017) to enable signiﬁcant speedups on regular GPUs, although doing so is not yet possible using standard popular libraries.

Original: Aspects of this work was ﬁrst communicated as a subset of the material in a workshop abstract and talk (McDonnell et al., 2017).
Revised: Aspects of this work was ﬁrst communicated as a subset of the material in a workshop abstract and talk (McDonnell et al., 2017).

Original: [Figure]
Revised: [Figure]

Original: Black points are results on the full ImageNet dataset, in comparison with results of Rastegari et al.
Revised: Black points are results on the full ImageNet dataset, in comparison with results of Rastegari et al.

Original: (2016) (black crosses).
Revised: (2016) (black crosses).

Original: The notation 4x, 10x and 15x corresponds to network width (see Section 4).
Revised: The notation 4x, 10x and 15x corresponds to network width (see Section 4).

Original: 1. R ELATED W ORK 1.1. R ES N ETS
Revised: 1. R ELATED W ORK 1.1. R ES N ETS

Original: In 2015, a new form of CNN called a “deep residual network,” or “ResNet” (He et al., 2015b) was developed and used to set many new accuracy records on computer-vision benchmarks.
Revised: In 2015, a new form of CNN called a “deep residual network,” or “ResNet” (He et al., 2015b) was developed and used to set many new accuracy records on computer-vision benchmarks.

Original: In comparison with older CNNs such as Alexnet (Krizhevsky et al., 2012) and VGGNet (Simonyan &
Revised: In comparison with older CNNs such as Alexnet (Krizhevsky et al., 2012) and VGGNet (Simonyan &

Original: Zisserman, 2014), ResNets achieve higher accuracy with far fewer learned parameters and FLOPs (FLoating-point OPerations) per image processed.
Revised: Zisserman, 2014), ResNets achieve higher accuracy with far fewer learned parameters and FLOPs (FLoating-point OPerations) per image processed.

Original: The key to reducing the number of parameters in ResNets was to replace “all-to-all layers” in VGG-like nets with “global-average-pooling” layers that have no learned parameters (Lin et al., 2013; Springenberg et al., 2014), while simultaneously training a much deeper network than previously.
Revised: The key to reducing the number of parameters in ResNets was to replace “all-to-all layers” in VGG-like nets with “global-average-pooling” layers that have no learned parameters (Lin et al., 2013; Springenberg et al., 2014), while simultaneously training a much deeper network than previously.

Original: The key new idea that enabled a deeper network to be trained effectively was the introduction of so-called “skip-connections” (He et al., 2015b).
Revised: The key new idea that enabled a deeper network to be trained effectively was the introduction of so-called “skip-connections” (He et al., 2015b).

Original: Many variations of ResNets have since been proposed.
Revised: Many variations of ResNets have since been proposed.

Original: ResNets offer the virtue of simplicity, and given the motivation for deployment in custom hardware, we have chosen them as our primary focus.
Revised: ResNets offer the virtue of simplicity, and given the motivation for deployment in custom hardware, we have chosen them as our primary focus.

Original: Despite the increased efﬁciency in parameter usage, similar to other CNNs the accuracy of ResNets still tends to increase with the total number of parameters; unlike other CNNs, increased accuracy can result either from deeper (He et al., 2016) or wider networks (Zagoruyko & Komodakis, 2016).
Revised: Despite the increased efﬁciency in parameter usage, similar to other CNNs the accuracy of ResNets still tends to increase with the total number of parameters; unlike other CNNs, increased accuracy can result either from deeper (He et al., 2016) or wider networks (Zagoruyko & Komodakis, 2016).

Original: In this paper, we use Wide Residual Networks (Zagoruyko & Komodakis, 2016), as they have been demonstrated to produce better accuracy in less training time than deeper networks.
Revised: In this paper, we use Wide Residual Networks (Zagoruyko & Komodakis, 2016), as they have been demonstrated to produce better accuracy in less training time than deeper networks.

Original: 1.1. R EDUCING THE MEMORY BURDEN OF TRAINED NEURAL NETWORKS
Revised: 1.1. R EDUCING THE MEMORY BURDEN OF TRAINED NEURAL NETWORKS

Original: Achieving the best accuracy and speed possible when deploying ResNets or similar networks on hardware-constrained mobile devices will require minimising the total number of bits transferred between memory and processors for a given number of parameters.
Revised: Achieving the best accuracy and speed possible when deploying ResNets or similar networks on hardware-constrained mobile devices will require minimising the total number of bits transferred between memory and processors for a given number of parameters.

Original: Motivated by such considerations, a lot of recent attention has been directed towards compressing the learned parameters ( model compression ) and reducing the precision of computations carried out by neural networks—see Hubara et al.
Revised: Motivated by such considerations, a lot of recent attention has been directed towards compressing the learned parameters ( model compression ) and reducing the precision of computations carried out by neural networks—see Hubara et al.

Original: for a more detailed literature review.
Revised: for a more detailed literature review.

Original: Recently published strategies for model compression include reducing the precision (number of bits used for numerical representation) of weights in deployed networks by doing the same during training (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), reducing the number of weights in trained neural networks by pruning (Han et al., 2015;
Revised: Recently published strategies for model compression include reducing the precision (number of bits used for numerical representation) of weights in deployed networks by doing the same during training (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), reducing the number of weights in trained neural networks by pruning (Han et al., 2015;

Original: Iandola et al., 2016), quantizing or compressing weights following training (Han et al., 2015; Zhou et al., 2017), reducing the precision of computations performed in forward-propagation during inference (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), and modifying neural network architectures (Howard et al., 2017).
Revised: Iandola et al., 2016), quantizing or compressing weights following training (Han et al., 2015; Zhou et al., 2017), reducing the precision of computations performed in forward-propagation during inference (Courbariaux et al., 2015; Hubara et al., 2016; Merolla et al., 2016; Rastegari et al., 2016), and modifying neural network architectures (Howard et al., 2017).

Original: A theoretical analysis of various methods proved results on the convergence of a variety of weight-binarization methods (Li et al., 2017).
Revised: A theoretical analysis of various methods proved results on the convergence of a variety of weight-binarization methods (Li et al., 2017).

Original: From this range of strategies, we are focused on an approach that simultaneously contributes two desirable attributes: (1) simplicity, in the sense that deployment of trained models immediately follows training without extra processing; (2) implementation of convolution operations can be achieved without multipliers, as demonstrated by Rastegari et al.
Revised: From this range of strategies, we are focused on an approach that simultaneously contributes two desirable attributes: (1) simplicity, in the sense that deployment of trained models immediately follows training without extra processing; (2) implementation of convolution operations can be achieved without multipliers, as demonstrated by Rastegari et al.

Original: 1. O VERALL APPROACH AND SUMMARY OF CONTRIBUTIONS
Revised: 1. O VERALL APPROACH AND SUMMARY OF CONTRIBUTIONS

Original: Our strategy for improving methods that enable inference with 1-bit-per-weight was threefold:
Revised: Our strategy for improving methods that enable inference with 1-bit-per-weight was threefold:

Original: State-of-the-art baseline.
Revised: State-of-the-art baseline.

Original: We sought to begin with a baseline full-precision deep CNN variant with close to state-of-the-art error rates.
Revised: We sought to begin with a baseline full-precision deep CNN variant with close to state-of-the-art error rates.

Original: At the time of commencement in 2016, the state-of-the-art on CIFAR-10 and CIFAR-100 was held by Wide Residual Networks (Zagoruyko & Komodakis, 2016), so this was our starting point.
Revised: At the time of commencement in 2016, the state-of-the-art on CIFAR-10 and CIFAR-100 was held by Wide Residual Networks (Zagoruyko & Komodakis, 2016), so this was our starting point.

Original: While subsequent approaches have exceeded their accuracy, ResNets offer superior simplicity, which conforms with our third strategy in this list.
Revised: While subsequent approaches have exceeded their accuracy, ResNets offer superior simplicity, which conforms with our third strategy in this list.

Original: Make minimal changes when training for 1-bit-per-weight.
Revised: Make minimal changes when training for 1-bit-per-weight.

Original: We aimed to ensure that training for 1-bit-per-weight could be achieved with minimal changes to baseline training.
Revised: We aimed to ensure that training for 1-bit-per-weight could be achieved with minimal changes to baseline training.

Original: Simplicity is desirable in custom hardware.
Revised: Simplicity is desirable in custom hardware.

Original: With custom hardware implementations in mind, we sought to simplify the design of the baseline network (and hence the version with 1-bit weights) as much as possible without sacriﬁcing accuracy.
Revised: With custom hardware implementations in mind, we sought to simplify the design of the baseline network (and hence the version with 1-bit weights) as much as possible without sacriﬁcing accuracy.

Original: 1.2. C ONTRIBUTIONS TO F ULL -P RECISION W IDE R ES N ETS
Revised: 1.2. C ONTRIBUTIONS TO F ULL -P RECISION W IDE R ES N ETS

Original: Although this paper is chieﬂy about 1-bit-per-weight, we exceeded our objectives for the fullprecision baseline network, and surpassed reported error rates for CIFAR-10 and CIFAR-100 using
Revised: Although this paper is chieﬂy about 1-bit-per-weight, we exceeded our objectives for the fullprecision baseline network, and surpassed reported error rates for CIFAR-10 and CIFAR-100 using

Original: Wide ResNets (Zagoruyko & Komodakis, 2016).
Revised: Wide ResNets (Zagoruyko & Komodakis, 2016).

Original: This was achieved using just 20 convolutional layers; most prior work has demonstrated best wide ResNet performance using 28 layers.
Revised: This was achieved using just 20 convolutional layers; most prior work has demonstrated best wide ResNet performance using 28 layers.

Original: Our innovation that achieves a signiﬁcant error-rate drop for CIFAR-10 and CIFAR-100 in Wide
Revised: Our innovation that achieves a signiﬁcant error-rate drop for CIFAR-10 and CIFAR-100 in Wide

Original: ResNets is to simply not learn the per-channel scale and offset factors in the batch-normalization layers, while retaining the remaining attributes of these layers.
Revised: ResNets is to simply not learn the per-channel scale and offset factors in the batch-normalization layers, while retaining the remaining attributes of these layers.

Original: It is important that this is done in conjunction with exchanging the ordering of the ﬁnal weight layer and the global average pooling layer (see Figure 3).
Revised: It is important that this is done in conjunction with exchanging the ordering of the ﬁnal weight layer and the global average pooling layer (see Figure 3).

Original: We observed this effect to be most pronounced for CIFAR-100, gaining around 3% in test-error rate.
Revised: We observed this effect to be most pronounced for CIFAR-100, gaining around 3% in test-error rate.

Original: But the method is advantageous only for networks that overﬁt; when overﬁtting is not an issue, such as for ImageNet, removing learning of batch-norm parameters is only detrimental.
Revised: But the method is advantageous only for networks that overﬁt; when overﬁtting is not an issue, such as for ImageNet, removing learning of batch-norm parameters is only detrimental.

Original: 1.2. C ONTRIBUTIONS TO DEEP CNN S WITH SINGLE - BIT WEIGHTS FOR INFERENCE
Revised: 1.2. C ONTRIBUTIONS TO DEEP CNN S WITH SINGLE - BIT WEIGHTS FOR INFERENCE

Original: Ours is the ﬁrst study we are aware of to consider how the gap in error-rate for 1-bit-per-weight compared to full-precision weights changes with full-precision accuracy across a diverse range of image classiﬁcation datasets (Figure 1).
Revised: Ours is the ﬁrst study we are aware of to consider how the gap in error-rate for 1-bit-per-weight compared to full-precision weights changes with full-precision accuracy across a diverse range of image classiﬁcation datasets (Figure 1).

Original: Our approach surpasses by a large margin all previously reported error rates for CIFAR-10/100 (error rates halved), for networks constrained to run with 1-bit-per-weight at inference time.
Revised: Our approach surpasses by a large margin all previously reported error rates for CIFAR-10/100 (error rates halved), for networks constrained to run with 1-bit-per-weight at inference time.

Original: One reason we have achieved lower error rates for the 1-bit case than previously is to start with a superior baseline network than in previous studies, namely Wide ResNets.
Revised: One reason we have achieved lower error rates for the 1-bit case than previously is to start with a superior baseline network than in previous studies, namely Wide ResNets.

Original: However, our approach also results in smaller error rate increases relative to full-precision error rates than previously, while training requires the same number of epochs as for the case of full-precision weights.
Revised: However, our approach also results in smaller error rate increases relative to full-precision error rates than previously, while training requires the same number of epochs as for the case of full-precision weights.

Original: Our main innovation is to introduce a simple ﬁxed scaling method for each convolutional layer, that permits activations and gradients to ﬂow through the network with minimum change in standard deviation, in accordance with the principle underlying popular initialization methods (He et al., 2015a).
Revised: Our main innovation is to introduce a simple ﬁxed scaling method for each convolutional layer, that permits activations and gradients to ﬂow through the network with minimum change in standard deviation, in accordance with the principle underlying popular initialization methods (He et al., 2015a).

Original: We combine this with the use of a warm-restart learning-rate method (Loshchilov & Hutter, 2016) that enables us to report close-to-baseline results for the 1-bit case in far fewer epochs of training than reported previously.
Revised: We combine this with the use of a warm-restart learning-rate method (Loshchilov & Hutter, 2016) that enables us to report close-to-baseline results for the 1-bit case in far fewer epochs of training than reported previously.

Original:  L EARNING A MODEL WITH CONVOLUTION WEIGHTS ±2. T HE SIGN OF WEIGHTS PROPAGATE , BUT FULL - PRECISION WEIGHTS ARE UPDATED
Revised:  L EARNING A MODEL WITH CONVOLUTION WEIGHTS ±2. T HE SIGN OF WEIGHTS PROPAGATE , BUT FULL - PRECISION WEIGHTS ARE UPDATED

Original: We follow the approach of Courbariaux et al.
Revised: We follow the approach of Courbariaux et al.

Original: ; Rastegari et al.
Revised: ; Rastegari et al.

Original: ; Merolla et al.
Revised: ; Merolla et al.

Original: (2016), in that
Revised: (2016), in that

Original: we ﬁnd good results when using 1-bit-per-weight at inference time if during training we apply the sign function to real-valued weights for the purpose of forward and backward propagation, but update full-precision weights using SGD with gradients calculated using full-precision.
Revised: we ﬁnd good results when using 1-bit-per-weight at inference time if during training we apply the sign function to real-valued weights for the purpose of forward and backward propagation, but update full-precision weights using SGD with gradients calculated using full-precision.

Original: However, previously reported methods for training using the sign of weights either need to train for many hundreds of epochs (Courbariaux et al., 2015; Merolla et al., 2016), or use computationallycostly normalization scaling for each channel in each layer that changes for each minibatch during training, i.e. the BWN method of Rastegari et al. (2016).
Revised: However, previously reported methods for training using the sign of weights either need to train for many hundreds of epochs (Courbariaux et al., 2015; Merolla et al., 2016), or use computationallycostly normalization scaling for each channel in each layer that changes for each minibatch during training, i.e. the BWN method of Rastegari et al. (2016).

Original: We obtained our results using a simple alternative approach, as we now describe.
Revised: We obtained our results using a simple alternative approach, as we now describe.

Original: 2.1. W E SCALE THE OUTPUT OF CONV LAYERS USING A CONSTANT FOR EACH LAYER
Revised: 2.1. W E SCALE THE OUTPUT OF CONV LAYERS USING A CONSTANT FOR EACH LAYER

Original: We begin by noting that the standard deviation of the sign of the weights in a convolutional layer with kernels of size F × F will be close to 1 , assuming a mean of zero.
Revised: We begin by noting that the standard deviation of the sign of the weights in a convolutional layer with kernels of size F × F will be close to 1 , assuming a mean of zero.

Original: In contrast, the standard deviation of layer i in full-precision networks is initialized in the method of He et al.
Revised: In contrast, the standard deviation of layer i in full-precision networks is initialized in the method of He et al.

Original: (2015a) to (cid:112) 2 / ( F 2 C
Revised: (2015a) to (cid:112) 2 / ( F 2 C

Original: i − 1 ) , where C i − 1 is the number of input channels to convolutional layer i , and i = 1 , .., L , where L is the number of convolutional layers and C 

Revised: i − 1 ) , where C i − 1 is the number of input channels to convolutional layer i , and i = 1 , .., L , where L is the number of convolutional layers and C

Original: = 3 for RGB inputs.
Revised: = 3 for RGB inputs.

Original: When applying the sign function alone, there is a mismatch with the principled approach to controlling gradient and activation scaling through a deep network (He et al., 2015a).
Revised: When applying the sign function alone, there is a mismatch with the principled approach to controlling gradient and activation scaling through a deep network (He et al., 2015a).

Original: Although the use of batch-normalization can still enable learning, convergence is empirically slow and less effective.
Revised: Although the use of batch-normalization can still enable learning, convergence is empirically slow and less effective.

Original: To address this problem, for training using the sign of weights, we use the initialization method of He et al.
Revised: To address this problem, for training using the sign of weights, we use the initialization method of He et al.

Original: (2015a)
Revised: (2015a)

Original: for the full-precision weights that are updated, but also introduce a layer-dependent scaling applied to the sign of the weights.
Revised: for the full-precision weights that are updated, but also introduce a layer-dependent scaling applied to the sign of the weights.

Original: This scaling has a constant unlearned value equal to the initial standard deviation of (cid:112) 2 / ( F 2 C i − 1 ) from the method of He et al.
Revised: This scaling has a constant unlearned value equal to the initial standard deviation of (cid:112) 2 / ( F 2 C i − 1 ) from the method of He et al.

Original: (2015a).
Revised: (2015a).

Original: This enables the standard deviation of forward-propagating information to be equal to the value it would have initially in full-precision networks.
Revised: This enables the standard deviation of forward-propagating information to be equal to the value it would have initially in full-precision networks.

Original: In implementation, during training we multiply the sign of the weights in each layer by this value.
Revised: In implementation, during training we multiply the sign of the weights in each layer by this value.

Original: For inference, we do this multiplication using a scaling layer following the weight layer, so that all weights in the network are stored using 0 and 1 , and deployed using ± 1 (see https://github.com/McDonnell-Lab/1-bit-per-weight/ ).
Revised: For inference, we do this multiplication using a scaling layer following the weight layer, so that all weights in the network are stored using 0 and 1 , and deployed using ± 1 (see https://github.com/McDonnell-Lab/1-bit-per-weight/ ).

Original: Hence, custom hardware implementations would be able to perform the model’s convolutions without multipliers (Rastegari et al., 2016), and signiﬁcant GPU speedups are also possible (Pedersoli et al., 2017).
Revised: Hence, custom hardware implementations would be able to perform the model’s convolutions without multipliers (Rastegari et al., 2016), and signiﬁcant GPU speedups are also possible (Pedersoli et al., 2017).

Original: The fact that we scale the weights explicitly during training is important.
Revised: The fact that we scale the weights explicitly during training is important.

Original: Although for forward and backward propagation it is equivalent to scale the input or output feature maps of a convolutional layer, doing so also scales the calculated gradients with respect to weights, since these are calculated by convolving input and output feature maps.
Revised: Although for forward and backward propagation it is equivalent to scale the input or output feature maps of a convolutional layer, doing so also scales the calculated gradients with respect to weights, since these are calculated by convolving input and output feature maps.

Original: As a consequence, learning is dramatically slower unless layer-dependent learning rates are introduced to cancel out the scaling.
Revised: As a consequence, learning is dramatically slower unless layer-dependent learning rates are introduced to cancel out the scaling.

Original: Our approach to this is similar to the BWN method of Rastegari et al.
Revised: Our approach to this is similar to the BWN method of Rastegari et al.

Original: but our constant scaling method is faster and less complex.
Revised: but our constant scaling method is faster and less complex.

Original: In summary, the only differences we make in comparison with full-precision training are as follows.
Revised: In summary, the only differences we make in comparison with full-precision training are as follows.

Original: Let W i be the tensor for the convolutional weights in the i –th convolutional layer.
Revised: Let W i be the tensor for the convolutional weights in the i –th convolutional layer.

Original: These weights are processed in the following way only for forward propagation and backward propagation, not for weight updates:
Revised: These weights are processed in the following way only for forward propagation and backward propagation, not for weight updates:

Original: [Equation]
Revised: [Equation]

Original: where F i is the spatial size of the convolutional kernel in layer i ; see Figure 2.
Revised: where F i is the spatial size of the convolutional kernel in layer i ; see Figure 2.

Original: [Figure]
Revised: [Figure]

Original:  M ETHODS COMMON TO BASELINE AND SINGLE - BIT NETWORKS 3. N ETWORK A RCHITECTURE
Revised:  M ETHODS COMMON TO BASELINE AND SINGLE - BIT NETWORKS 3. N ETWORK A RCHITECTURE

Original: Our ResNets use the ‘post-activation’ and identity mapping approach of He et al.
Revised: Our ResNets use the ‘post-activation’ and identity mapping approach of He et al.

Original: (2016) for residual connections.
Revised: (2016) for residual connections.

Original: For ImageNet, we use an 18-layer design, as in He et al.
Revised: For ImageNet, we use an 18-layer design, as in He et al.

Original: (2015b).
Revised: (2015b).

Original: For all other datasets we mainly use a 20-layer network, but also report some results for 26 layers.
Revised: For all other datasets we mainly use a 20-layer network, but also report some results for 26 layers.

Original: Each residual block includes two convolutional layers, each preceded by batch normalization (BN) and Rectiﬁed Linear
Revised: Each residual block includes two convolutional layers, each preceded by batch normalization (BN) and Rectiﬁed Linear

Original: Unit (ReLU) layers.
Revised: Unit (ReLU) layers.

Original: Rather than train very deep ResNets, we use wide residual networks (Wide
Revised: Rather than train very deep ResNets, we use wide residual networks (Wide

Original: ResNets) (Zagoruyko & Komodakis, 2016).
Revised: ResNets) (Zagoruyko & Komodakis, 2016).

Original: Although Zagoruyko & Komodakis (2016) and others reported that 28/26-layer networks result in better test accuracy than 22/20-layer 1 networks, we found for CIFAR-10/100 that just 20 layers typically produces best results, which is possibly due to our approach of not learning the batch-norm scale and offset parameters.
Revised: Although Zagoruyko & Komodakis (2016) and others reported that 28/26-layer networks result in better test accuracy than 22/20-layer 1 networks, we found for CIFAR-10/100 that just 20 layers typically produces best results, which is possibly due to our approach of not learning the batch-norm scale and offset parameters.

Original: Our baseline ResNet design used in most of our experiments (see Figures 3 and 4) has several differences in comparison to those of He et al.
Revised: Our baseline ResNet design used in most of our experiments (see Figures 3 and 4) has several differences in comparison to those of He et al.

Original: ; Zagoruyko & Komodakis (2016).
Revised: ; Zagoruyko & Komodakis (2016).

Original: These details are articulated in Appendix A, and are mostly for simplicity, with little impact on accuracy.
Revised: These details are articulated in Appendix A, and are mostly for simplicity, with little impact on accuracy.

Original: The exception is our approach of not learning batch-norm parameters.
Revised: The exception is our approach of not learning batch-norm parameters.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: We trained our models following, for most aspects, the standard stochastic gradient descent methods used by Zagoruyko & Komodakis (2016) for Wide ResNets.
Revised: We trained our models following, for most aspects, the standard stochastic gradient descent methods used by Zagoruyko & Komodakis (2016) for Wide ResNets.

Original: Speciﬁcally, we use cross-entropy loss, minibatches of size 125, and momentum of 0 .
Revised: Speciﬁcally, we use cross-entropy loss, minibatches of size 125, and momentum of 0 .

Original: 9 (both for learning weights, and in situations where we learn batch-norm scales and offsets).
Revised: 9 (both for learning weights, and in situations where we learn batch-norm scales and offsets).

Original: For CIFAR-10/100, SVHN and MNIST, where overﬁtting is evident in Wide ResNets, we use a larger weight decay of 0.0005.
Revised: For CIFAR-10/100, SVHN and MNIST, where overﬁtting is evident in Wide ResNets, we use a larger weight decay of 0.0005.

Original: For ImageNet32 and full
Revised: For ImageNet32 and full

Original: ImageNet we use a weight decay of 0.0001.
Revised: ImageNet we use a weight decay of 0.0001.

Original: Apart from one set of experiments where we added a simple extra approach called cutout, we use standard ‘light’ data augmentation, including randomly ﬂipping each image horizontally with probability 0 .
Revised: Apart from one set of experiments where we added a simple extra approach called cutout, we use standard ‘light’ data augmentation, including randomly ﬂipping each image horizontally with probability 0 .

Original: 5 for CIFAR-10/100 and ImageNet32.
Revised: 5 for CIFAR-10/100 and ImageNet32.

Original: For the
Revised: For the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: pool” ) in the residual path when downsampling.
Revised: pool” ) in the residual path when downsampling.

Original: same 3 datasets plus SVHN, we pad by 4 pixels on all sides (using random values between 0 and 255) and crop a 32 × 32 patch from a random location in the resulting 40 × 40 image.
Revised: same 3 datasets plus SVHN, we pad by 4 pixels on all sides (using random values between 0 and 255) and crop a 32 × 32 patch from a random location in the resulting 40 × 40 image.

Original: For full
Revised: For full

Original: ImageNet, we scale, crop and ﬂip, as in He et al.
Revised: ImageNet, we scale, crop and ﬂip, as in He et al.

Original: (2015b).
Revised: (2015b).

Original: We did not use any image-preprocessing, i.e. we did not subtract the mean, as the initial BN layer in our network performs this role, and we did not use whitening or augment using color or brightness.
Revised: We did not use any image-preprocessing, i.e. we did not subtract the mean, as the initial BN layer in our network performs this role, and we did not use whitening or augment using color or brightness.

Original: We use the initialization method of He et al.
Revised: We use the initialization method of He et al.

Original: (2015a).
Revised: (2015a).

Original: We now describe important differences in our training approach compared to those usually reported.
Revised: We now describe important differences in our training approach compared to those usually reported.

Original: 3.2. B ATCH - NORM SCALES & OFFSETS ARE NOT LEARNED FOR CIFAR-10/100, SVHN
Revised: 3.2. B ATCH - NORM SCALES & OFFSETS ARE NOT LEARNED FOR CIFAR-10/100, SVHN

Original: When training on CIFAR-10/100 and SVHN, in all batch-norm layers (except the ﬁrst one at the input when a ReLU is used there), we do not learn the scale and offset factor, instead initializating these to 1 and 0 in all channels, and keeping those values through training.
Revised: When training on CIFAR-10/100 and SVHN, in all batch-norm layers (except the ﬁrst one at the input when a ReLU is used there), we do not learn the scale and offset factor, instead initializating these to 1 and 0 in all channels, and keeping those values through training.

Original: Note that we also do not learn any biases for convolutional layers.
Revised: Note that we also do not learn any biases for convolutional layers.

Original: The usual approach to setting the moments for use in batch-norm layers in inference mode is to keep a running average through training.
Revised: The usual approach to setting the moments for use in batch-norm layers in inference mode is to keep a running average through training.

Original: When not learning batch-normalization parameters, we found a small beneﬁt in calculating the batch-normalization moments used in inference only after training had ﬁnished.
Revised: When not learning batch-normalization parameters, we found a small beneﬁt in calculating the batch-normalization moments used in inference only after training had ﬁnished.

Original: We simply form as many minibatches as possible from the full training-set, each with the same data augmentation as used during training applied, and pass these through the trained network, averaging the returned moments for each batch.
Revised: We simply form as many minibatches as possible from the full training-set, each with the same data augmentation as used during training applied, and pass these through the trained network, averaging the returned moments for each batch.

Original: For best results when using this method using matconvnet, we found it necessary to ensure the (cid:15) parameter that is used to avoid divisions by zero is set to 1 × 10 − 5 ; this (cid:15) is different to the way it is used in keras and other libraries.
Revised: For best results when using this method using matconvnet, we found it necessary to ensure the (cid:15) parameter that is used to avoid divisions by zero is set to 1 × 10 − 5 ; this (cid:15) is different to the way it is used in keras and other libraries.

Original: 3. O UR NETWORK ’ S FINAL WEIGHT LAYER IS A 1 × 1 CONVOLUTIONAL LAYER
Revised: 3. O UR NETWORK ’ S FINAL WEIGHT LAYER IS A 1 × 1 CONVOLUTIONAL LAYER

Original: A signiﬁcant difference to the ResNets of He et al.
Revised: A signiﬁcant difference to the ResNets of He et al.

Original: ; Zagoruyko & Komodakis (2016) is that we exchange the ordering of the global average pooling layer and the ﬁnal weight layer, so that our ﬁnal weight layer becomes a 1 × 1 convolutional layer with as many channels as there are classes in the training set.
Revised: ; Zagoruyko & Komodakis (2016) is that we exchange the ordering of the global average pooling layer and the ﬁnal weight layer, so that our ﬁnal weight layer becomes a 1 × 1 convolutional layer with as many channels as there are classes in the training set.

Original: This design is not new, but it does seem to be new to ResNets: it corresponds to the architecture of Lin et al.
Revised: This design is not new, but it does seem to be new to ResNets: it corresponds to the architecture of Lin et al.

Original: (2013), which originated the global average pooling concept, and also to that used by Springenberg et al.
Revised: (2013), which originated the global average pooling concept, and also to that used by Springenberg et al.

Original: As with all other convolutional layers, we follow this ﬁnal layer with a batch-normalization layer; the beneﬁts of this in conjunction with not learning the batch-normalization scale and offset are described in the Discussion section.
Revised: As with all other convolutional layers, we follow this ﬁnal layer with a batch-normalization layer; the beneﬁts of this in conjunction with not learning the batch-normalization scale and offset are described in the Discussion section.

Original: 3.3. W E USE A WARM - RESTART LEARNING - RATE SCHEDULE
Revised: 3.3. W E USE A WARM - RESTART LEARNING - RATE SCHEDULE

Original: We use a ‘warm restarts’ learning rate schedule that has reported state-of-the-art Wide ResNet results (Loshchilov & Hutter, 2016) whilst also speeding up convergence.
Revised: We use a ‘warm restarts’ learning rate schedule that has reported state-of-the-art Wide ResNet results (Loshchilov & Hutter, 2016) whilst also speeding up convergence.

Original: The method constantly reduces the learning rate from 0.1 to 1 × 10 − 4 according to a cosine decay, across a certain number of epochs, and then repeats across twice as many epochs.
Revised: The method constantly reduces the learning rate from 0.1 to 1 × 10 − 4 according to a cosine decay, across a certain number of epochs, and then repeats across twice as many epochs.

Original: We restricted our attention to a maximum of 254 epochs (often just 62 epochs, and no more than 30 for ImageNet32) using this method, which is the total number of epochs after reducing the learning rate from maximum to minimum throughepochs, then 4, 8, 16, 32, 64 and 128 epochs.
Revised: We restricted our attention to a maximum of 254 epochs (often just 62 epochs, and no more than 30 for ImageNet32) using this method, which is the total number of epochs after reducing the learning rate from maximum to minimum throughepochs, then 4, 8, 16, 32, 64 and 128 epochs.

Original: For CIFAR-10/100, we typically found that we could achieve test error rates after 32 epochs within 1-2% of the error rates achievable after 126 orepochs.
Revised: For CIFAR-10/100, we typically found that we could achieve test error rates after 32 epochs within 1-2% of the error rates achievable after 126 orepochs.

Original: 3.3. E XPERIMENTS WITH CUTOUT FOR CIFAR-10/ 
Revised: 3.3. E XPERIMENTS WITH CUTOUT FOR CIFAR-10/

Original: In the literature, most experiments with CIFAR-10 and CIFAR-100 use simple “standard” data augmentation, consisting of randomly ﬂipping each training image left-right with probability 0.5, and padding each image on all sides by 4 pixels, and then cropping a 32 × 32 version of the image from a random location.
Revised: In the literature, most experiments with CIFAR-10 and CIFAR-100 use simple “standard” data augmentation, consisting of randomly ﬂipping each training image left-right with probability 0.5, and padding each image on all sides by 4 pixels, and then cropping a 32 × 32 version of the image from a random location.

Original: We use this augmentation, although with the minor modiﬁcation that we pad with uniform random integers between 0 and 255 , rather than zero-padding.
Revised: We use this augmentation, although with the minor modiﬁcation that we pad with uniform random integers between 0 and 255 , rather than zero-padding.

Original: Additionally, we experimented with “cutout” (Devries & Taylor, 2017).
Revised: Additionally, we experimented with “cutout” (Devries & Taylor, 2017).

Original: This involves randomly selecting a patch of each raw training image to remove.
Revised: This involves randomly selecting a patch of each raw training image to remove.

Original: The method was shown to combine with other state-of-the-art methods to set the latest state-of-the-art results on CIFAR-10/100 (see Table 1).
Revised: The method was shown to combine with other state-of-the-art methods to set the latest state-of-the-art results on CIFAR-10/100 (see Table 1).

Original: We found better results using larger cutout patches for CIFAR-100 than those reported by Devries & Taylor (2017); hence for both CIFAR-10 and CIFAR-100 we choose patches of size 18 × 18 .
Revised: We found better results using larger cutout patches for CIFAR-100 than those reported by Devries & Taylor (2017); hence for both CIFAR-10 and CIFAR-100 we choose patches of size 18 × 18 .

Original: Following the method of Devries & Taylor (2017), we ensured that all pixels are chosen for being included in a patch equally frequently throughout training by ensuring that if the chosen patch location is near the image border, the patch impacts on the image only for the part of the patch inside the image.
Revised: Following the method of Devries & Taylor (2017), we ensured that all pixels are chosen for being included in a patch equally frequently throughout training by ensuring that if the chosen patch location is near the image border, the patch impacts on the image only for the part of the patch inside the image.

Original: Differently to Devries & Taylor (2017), as for our padding, we use uniform random integers to replace the image pixel values in the location of the patches.
Revised: Differently to Devries & Taylor (2017), as for our padding, we use uniform random integers to replace the image pixel values in the location of the patches.

Original: We did not apply cutout to other datasets.
Revised: We did not apply cutout to other datasets.

Original:  R ESULTS
Revised:  R ESULTS

Original: We conducted experiments on six databases: four databases of 32 × 32 RGB images—CIFAR10, CIFAR-100, SVHN and ImageNet32—and the full ILSVRC ImageNet database (Russakovsky et al., 2015), as well as MNIST (LeCun et al., 1998).
Revised: We conducted experiments on six databases: four databases of 32 × 32 RGB images—CIFAR10, CIFAR-100, SVHN and ImageNet32—and the full ILSVRC ImageNet database (Russakovsky et al., 2015), as well as MNIST (LeCun et al., 1998).

Original: Details of the ﬁrst three 32 × 32 datasets can be found in many papers, e.g. (Zagoruyko & Komodakis, 2016).
Revised: Details of the ﬁrst three 32 × 32 datasets can be found in many papers, e.g. (Zagoruyko & Komodakis, 2016).

Original: ImageNet32 is a downsampled version of ImageNet, where the training and validation images are cropped using their annotated bounding boxes, and then downsampled to 32 × 32 (Chrabaszcz et al., 2017); see http://image-net.org/download-images .
Revised: ImageNet32 is a downsampled version of ImageNet, where the training and validation images are cropped using their annotated bounding boxes, and then downsampled to 32 × 32 (Chrabaszcz et al., 2017); see http://image-net.org/download-images .

Original: All experiments were carried out on either a single Titan X or Xp GPU using MATLAB with GPU acceleration from MatConvNet and cuDNN.
Revised: All experiments were carried out on either a single Titan X or Xp GPU using MATLAB with GPU acceleration from MatConvNet and cuDNN.

Original: We report results for Wide ResNets, which (except when applied to ImageNet) are 4 × and 10 × wider than baseline ResNets, to use the terminology of Zagoruyko & Komodakis (2016), where the baseline has 16 channels in the layers at the ﬁrst spatial scale.
Revised: We report results for Wide ResNets, which (except when applied to ImageNet) are 4 × and 10 × wider than baseline ResNets, to use the terminology of Zagoruyko & Komodakis (2016), where the baseline has 16 channels in the layers at the ﬁrst spatial scale.

Original: We use notation of the form 20to denote Wide ResNets with 20 convolutional layers and 160 channels in the ﬁrst spatial scale, and hence width 10 × .
Revised: We use notation of the form 20to denote Wide ResNets with 20 convolutional layers and 160 channels in the ﬁrst spatial scale, and hence width 10 × .

Original: For the full ImageNet dataset, we use 18-layer Wide ResNets with 160 channels in the ﬁrst spatial scale, but given the standard ResNet baseline is width 64, this corresponds to width 2.5 × on this dataset (Zagoruyko & Komodakis, 2016).
Revised: For the full ImageNet dataset, we use 18-layer Wide ResNets with 160 channels in the ﬁrst spatial scale, but given the standard ResNet baseline is width 64, this corresponds to width 2.5 × on this dataset (Zagoruyko & Komodakis, 2016).

Original: We denote these networks as 18-2.5.
Revised: We denote these networks as 18-2.5.

Original: Table 1 lists our top-1 error rates for CIFAR-10/100; C10 indicates CIFAR-10, C100 indicates
Revised: Table 1 lists our top-1 error rates for CIFAR-10/100; C10 indicates CIFAR-10, C100 indicates

Original: CIFAR-100; the superscript + indicates standard crop and ﬂip augmentation; and ++ indicates the use of cutout.
Revised: CIFAR-100; the superscript + indicates standard crop and ﬂip augmentation; and ++ indicates the use of cutout.

Original: Table 2 lists error rates for SVHN, ImageNet32 (I32) and full ImageNet; we did not use cutout on these datasets.
Revised: Table 2 lists error rates for SVHN, ImageNet32 (I32) and full ImageNet; we did not use cutout on these datasets.

Original: Both top-1 and top-5 results are tabulated for I32 and ImageNet.
Revised: Both top-1 and top-5 results are tabulated for I32 and ImageNet.

Original: For
Revised: For

Original: ImageNet, we provide results for single-center-crop testing, and also for multi-crop testing.
Revised: ImageNet, we provide results for single-center-crop testing, and also for multi-crop testing.

Original: In the latter, the decision for each test image is obtained by averaging the softmax output after passing through the network 25 times, corresponding to crops obtained by rescaling to 5 scales as described by He et al.
Revised: In the latter, the decision for each test image is obtained by averaging the softmax output after passing through the network 25 times, corresponding to crops obtained by rescaling to 5 scales as described by He et al.

Original: (2015b), and from 5 random positions at each scale.
Revised: (2015b), and from 5 random positions at each scale.

Original: Our full-precision ImageNet error rates are slightly lower than expected for a wide ResNet according to the results of Zagoruyko &
Revised: Our full-precision ImageNet error rates are slightly lower than expected for a wide ResNet according to the results of Zagoruyko &

Original: Komodakis (2016), probably due to the fact we did not use color augmentation.
Revised: Komodakis (2016), probably due to the fact we did not use color augmentation.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Table 3 shows comparison results from the original work on Wide ResNets, and subsequent papers that have reduced error rates on the CIFAR-10 and CIFAR-100 datasets.
Revised: Table 3 shows comparison results from the original work on Wide ResNets, and subsequent papers that have reduced error rates on the CIFAR-10 and CIFAR-100 datasets.

Original: We also show the only results, to our knowledge, for ImageNet32.
Revised: We also show the only results, to our knowledge, for ImageNet32.

Original: The current state-of-the-art for SVHN without augmentation is 1 .
Revised: The current state-of-the-art for SVHN without augmentation is 1 .

Original: 59 % (Huang et al., 2016), and with cutout augmentation is 1 .
Revised: 59 % (Huang et al., 2016), and with cutout augmentation is 1 .

Original: 30 % (Devries & Taylor, 2017).
Revised: 30 % (Devries & Taylor, 2017).

Original: Our full-precision result for SVHN (1.75%) is only a little short of these even though we used only a 4 × ResNet, with less than 5 million parameters, and only 30 training epochs.
Revised: Our full-precision result for SVHN (1.75%) is only a little short of these even though we used only a 4 × ResNet, with less than 5 million parameters, and only 30 training epochs.

Original: Table 4 shows comparison results for previous work that trains models by using the sign of weights during training.
Revised: Table 4 shows comparison results for previous work that trains models by using the sign of weights during training.

Original: Additional results appear in Hubara et al.
Revised: Additional results appear in Hubara et al.

Original: (2016), where activations are quantized,
Revised: (2016), where activations are quantized,

Original: and so the error rates are much larger.
Revised: and so the error rates are much larger.

Original: Inspection of Tables 1 to 4 reveals that our baseline full-precision 10 × networks, when trained with cutout, surpass the performance of deeper Wide ResNets trained with dropout.
Revised: Inspection of Tables 1 to 4 reveals that our baseline full-precision 10 × networks, when trained with cutout, surpass the performance of deeper Wide ResNets trained with dropout.

Original: Even without the use
Revised: Even without the use

Original: [Table] [Table]
Revised: [Table] [Table]

Original: This Paper: single center crop
Revised: This Paper: single center crop

Original: [Table]
Revised: [Table]

Original: This
Revised: This

Original: [Table] [Table]
Revised: [Table] [Table]

Original: of cutout, our 20-10 network surpasses by over 2% the CIFAR-100 error rate reported for essentially the same network by Zagoruyko & Komodakis (2016) and is also better than previous Wide ResNet results on CIFAR-10 and ImageNet32.
Revised: of cutout, our 20-10 network surpasses by over 2% the CIFAR-100 error rate reported for essentially the same network by Zagoruyko & Komodakis (2016) and is also better than previous Wide ResNet results on CIFAR-10 and ImageNet32.

Original: As elaborated on in Section 5.3, this improved accuracy is due to our approach of not learning the batch-norm scale and offset parameters.
Revised: As elaborated on in Section 5.3, this improved accuracy is due to our approach of not learning the batch-norm scale and offset parameters.

Original: For our 1-bit networks, we observe that there is always an accuracy gap compared to full precision networks.
Revised: For our 1-bit networks, we observe that there is always an accuracy gap compared to full precision networks.

Original: This is discussed in Section 5.1.
Revised: This is discussed in Section 5.1.

Original: Using cutout for CIFAR-10/100 reduces error rates as expected.
Revised: Using cutout for CIFAR-10/100 reduces error rates as expected.

Original: In comparison with training very wide 20 × ResNets on CIFAR-100, as shown in Table 1, it is more effective to use cutout augmentation in the 10 × network to reduce the error rate, while using only a quarter of the weights.
Revised: In comparison with training very wide 20 × ResNets on CIFAR-100, as shown in Table 1, it is more effective to use cutout augmentation in the 10 × network to reduce the error rate, while using only a quarter of the weights.

Original: Figure 5 illustrates convergence and overﬁtting trends for CIFAR-10/100 for 20-4 Wide ResNets, and a comparison of the use of cutout in 20-10 Wide ResNets.
Revised: Figure 5 illustrates convergence and overﬁtting trends for CIFAR-10/100 for 20-4 Wide ResNets, and a comparison of the use of cutout in 20-10 Wide ResNets.

Original: Clearly, even for width-4 ResNets, the gap in error rate between full precision weights and 1-bit-per-weight is small.
Revised: Clearly, even for width-4 ResNets, the gap in error rate between full precision weights and 1-bit-per-weight is small.

Original: Also noticeable is that the warm-restart method enables convergence to very good solutions after just 30 epochs; training longer to 126 epochs reduces test error rates further by between 2% and 5%.
Revised: Also noticeable is that the warm-restart method enables convergence to very good solutions after just 30 epochs; training longer to 126 epochs reduces test error rates further by between 2% and 5%.

Original: It can also be observed that the 20-4 network is powerful enough to model the CIFAR-10/100 training sets to well over 99% accuracy, but the modelling power is reduced in the 1-bit version, particularly for CIFAR100.
Revised: It can also be observed that the 20-4 network is powerful enough to model the CIFAR-10/100 training sets to well over 99% accuracy, but the modelling power is reduced in the 1-bit version, particularly for CIFAR100.

Original: The reduced modelling capacity for single-bit weights is consistent with the gap in test-error rate performance between the 32-bit and 1-bit cases.
Revised: The reduced modelling capacity for single-bit weights is consistent with the gap in test-error rate performance between the 32-bit and 1-bit cases.

Original: When using cutout, training for longer gives improved error rates, but when not using cutout, 126 epochs sufﬁces for peak performance.
Revised: When using cutout, training for longer gives improved error rates, but when not using cutout, 126 epochs sufﬁces for peak performance.

Original: Finally, for MNIST and a 4 × wide ResNet without any data augmentation, our full-precision method achieved 0.71% after just 1 epoch of training, and 0.28% after 6 epochs, whereas our 1-bit method achieved 0.81%, 0.36% and 0.27% after 1, 6 and 14 epochs.
Revised: Finally, for MNIST and a 4 × wide ResNet without any data augmentation, our full-precision method achieved 0.71% after just 1 epoch of training, and 0.28% after 6 epochs, whereas our 1-bit method achieved 0.81%, 0.36% and 0.27% after 1, 6 and 14 epochs.

Original: In comparison, 1.29% was reported for the 1-bit-per-weight case by Courbariaux et al.
Revised: In comparison, 1.29% was reported for the 1-bit-per-weight case by Courbariaux et al.

Original: (2015), and 0.96% by Hubara et al.
Revised: (2015), and 0.96% by Hubara et al.

Original: 4. A BLATION STUDIES
Revised: 4. A BLATION STUDIES

Original: [Figure]
Revised: [Figure]

Original: For CIFAR-10/100, both our full-precision Wide ResNets and our 1-bit-per-weight versions beneﬁt from our method of not learning batch-norm scale and offset parameters.
Revised: For CIFAR-10/100, both our full-precision Wide ResNets and our 1-bit-per-weight versions beneﬁt from our method of not learning batch-norm scale and offset parameters.

Original: Accuracy in the 1-bit-perweight case also beneﬁts from our use of a warm-restart training schedule (Loshchilov & Hutter, 2016).
Revised: Accuracy in the 1-bit-perweight case also beneﬁts from our use of a warm-restart training schedule (Loshchilov & Hutter, 2016).

Original: To demonstrate the inﬂuence of these two aspects, in Figure 6 we show, for CIFAR-100, how the test error rate changes through training when either or both of these methods are not used.
Revised: To demonstrate the inﬂuence of these two aspects, in Figure 6 we show, for CIFAR-100, how the test error rate changes through training when either or both of these methods are not used.

Original: We did not use cutout for the purpose of this ﬁgure.
Revised: We did not use cutout for the purpose of this ﬁgure.

Original: The comparison learning-rate-schedule drops the learning rate from 0.1 to 0.01 to 0.001 after 85 and 170 epochs.
Revised: The comparison learning-rate-schedule drops the learning rate from 0.1 to 0.01 to 0.001 after 85 and 170 epochs.

Original: It is clear that our methods lower the ﬁnal error rate by around 3% absolute by not learning the batch-norm parameters.
Revised: It is clear that our methods lower the ﬁnal error rate by around 3% absolute by not learning the batch-norm parameters.

Original: The warm-restart method enables faster convergence for the full-precision case, but is not signiﬁcant in reducing the error rate.
Revised: The warm-restart method enables faster convergence for the full-precision case, but is not signiﬁcant in reducing the error rate.

Original: However, for the 1-bit-per-weight case, it is clear that for best results it is best to both use warm-restart, and to not learn batch-norm parameters.
Revised: However, for the 1-bit-per-weight case, it is clear that for best results it is best to both use warm-restart, and to not learn batch-norm parameters.

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION 5. T HE ACCURACY GAP FOR 1- BIT VS 32- BIT WEIGHTS
Revised:  D ISCUSSION 5. T HE ACCURACY GAP FOR 1- BIT VS 32- BIT WEIGHTS

Original: It is to be expected that a smaller error rate gap will result between the same network using fullprecision and 1-bit-per-weight when the test error rate on the full precision network gets smaller.
Revised: It is to be expected that a smaller error rate gap will result between the same network using fullprecision and 1-bit-per-weight when the test error rate on the full precision network gets smaller.

Original: Indeed, Tables 1 and 2 quantify how the gap in error rate between the full-precision and 1-bit-perweight cases tends to grow as the error-rate in the full-precision network grows.
Revised: Indeed, Tables 1 and 2 quantify how the gap in error rate between the full-precision and 1-bit-perweight cases tends to grow as the error-rate in the full-precision network grows.

Original: To further illustrate this trend for our approach, we have plotted in Figure 1 the gap in Top-1 error rates vs the Top-1 error rate for the full precision case, for some of the best performing networks for the six datasets we used.
Revised: To further illustrate this trend for our approach, we have plotted in Figure 1 the gap in Top-1 error rates vs the Top-1 error rate for the full precision case, for some of the best performing networks for the six datasets we used.

Original: Strong conclusions from this data can only be made relative to alternative methods, but ours is the ﬁrst study to our knowledge to consider more than two datasets.
Revised: Strong conclusions from this data can only be made relative to alternative methods, but ours is the ﬁrst study to our knowledge to consider more than two datasets.

Original: Nevertheless, we have also plotted in Figure 1 the error rate and the gap reported by Rastegari et al.
Revised: Nevertheless, we have also plotted in Figure 1 the error rate and the gap reported by Rastegari et al.

Original: (2016) for two different networks using their BWN 1-bit weight method.
Revised: (2016) for two different networks using their BWN 1-bit weight method.

Original: The reasons for the larger gaps for those points is unclear, but what is clear is that better full-precision accuracy results in a smaller gap in all cases.
Revised: The reasons for the larger gaps for those points is unclear, but what is clear is that better full-precision accuracy results in a smaller gap in all cases.

Original: A challenge for further work is to derive theoretical bounds that predict the gap.
Revised: A challenge for further work is to derive theoretical bounds that predict the gap.

Original: How the magnitude of the gap changes with full precision error rate is dependent on many factors, including the method used to generate models with 1-bit-per-weight.
Revised: How the magnitude of the gap changes with full precision error rate is dependent on many factors, including the method used to generate models with 1-bit-per-weight.

Original: For high-error rate cases, the loss function throughout training is much higher for the 1-bit case than the 32-bit case, and hence, the 1-bit-per-weight network is not able to ﬁt the training set as well as the 32-bit one.
Revised: For high-error rate cases, the loss function throughout training is much higher for the 1-bit case than the 32-bit case, and hence, the 1-bit-per-weight network is not able to ﬁt the training set as well as the 32-bit one.

Original: Whether this is because of the loss of precision in weights, or due to the mismatch in gradients inherent is propagating with 1-bit weights and updating full-precision weights during training is an open question.
Revised: Whether this is because of the loss of precision in weights, or due to the mismatch in gradients inherent is propagating with 1-bit weights and updating full-precision weights during training is an open question.

Original: If it is the latter case, then it is possible that principled reﬁnements to the weight update method we used will further reduce the gap.
Revised: If it is the latter case, then it is possible that principled reﬁnements to the weight update method we used will further reduce the gap.

Original: However, it is also interesting that for our 26-layer networks applied to CIFAR10/100, that the gap is much smaller, despite no beneﬁts in the full precision case from extra depth, and this also warrants further investigation.
Revised: However, it is also interesting that for our 26-layer networks applied to CIFAR10/100, that the gap is much smaller, despite no beneﬁts in the full precision case from extra depth, and this also warrants further investigation.

Original: 5. C OMPARISON WITH THE BWN METHOD
Revised: 5. C OMPARISON WITH THE BWN METHOD

Original: Our approach differs from the BWN method of Rastegari et al.
Revised: Our approach differs from the BWN method of Rastegari et al.

Original: (2016) for two reasons.
Revised: (2016) for two reasons.

Original: First, we do not need to calculate mean absolute weight values of the underlying full precision weights for each output channel in each layer following each minibatch, and this enables faster training.
Revised: First, we do not need to calculate mean absolute weight values of the underlying full precision weights for each output channel in each layer following each minibatch, and this enables faster training.

Original: Second, we do not need to adjust for a gradient term corresponding to the appearance of each weight in the mean absolute value.
Revised: Second, we do not need to adjust for a gradient term corresponding to the appearance of each weight in the mean absolute value.

Original: We found overall that the two methods work equally effectively, but ours has two advantages: faster training, and fewer overall parameters.
Revised: We found overall that the two methods work equally effectively, but ours has two advantages: faster training, and fewer overall parameters.

Original: As a note we found that the method of Rastegari et al.
Revised: As a note we found that the method of Rastegari et al.

Original: (2016) also works equally effectively on a per-layer basis, rather than per-channel.
Revised: (2016) also works equally effectively on a per-layer basis, rather than per-channel.

Original: We also note that the focus of Rastegari et al.
Revised: We also note that the focus of Rastegari et al.

Original: (2016) was much more on the case that combines single-bit activations and 1-bit-per-weight than solely 1-bit-per-weight.
Revised: (2016) was much more on the case that combines single-bit activations and 1-bit-per-weight than solely 1-bit-per-weight.

Original: It remains to be tested how our scaling method compares in that case.
Revised: It remains to be tested how our scaling method compares in that case.

Original: It is also interesting to understand whether the use of batch-norm renders scaling of the sign of weights robust to different scalings, and whether networks that do not use batch-norm might be more sensitive to the precise method used.
Revised: It is also interesting to understand whether the use of batch-norm renders scaling of the sign of weights robust to different scalings, and whether networks that do not use batch-norm might be more sensitive to the precise method used.

Original: 5. T HE INFLUENCE OF NOT LEARNING BATCH - NORMALIZATION PARAMETERS
Revised: 5. T HE INFLUENCE OF NOT LEARNING BATCH - NORMALIZATION PARAMETERS

Original: The unusual design choice of not learning the batch normalization parameters was made for CIFAR10/100, SVHN and MNIST because for Wide ResNets, overﬁtting is very evident on these datasets (see Figure 5); by the end of training, typically the loss function becomes very close to zero, corresponding to severe overﬁtting.
Revised: The unusual design choice of not learning the batch normalization parameters was made for CIFAR10/100, SVHN and MNIST because for Wide ResNets, overﬁtting is very evident on these datasets (see Figure 5); by the end of training, typically the loss function becomes very close to zero, corresponding to severe overﬁtting.

Original: Inspired by label-smoothing regularization (Szegedy et al., 2015) that aims to reduce overconﬁdence following the softmax layer, we hypothesized that imposing more control over the standard deviation of inputs to the softmax might have a similar regularizing effect.
Revised: Inspired by label-smoothing regularization (Szegedy et al., 2015) that aims to reduce overconﬁdence following the softmax layer, we hypothesized that imposing more control over the standard deviation of inputs to the softmax might have a similar regularizing effect.

Original: This is why we removed the ﬁnal all-to-all layer of our ResNets and replaced it with a 1 × 1 convolutional layer followed by a batch-normalization layer prior to the global average pooling layer.
Revised: This is why we removed the ﬁnal all-to-all layer of our ResNets and replaced it with a 1 × 1 convolutional layer followed by a batch-normalization layer prior to the global average pooling layer.

Original: In turn, not learning a scale and offset for this batch-normalization layer ensures that batches ﬂowing into the softmax layer have standard deviations that do not grow throughout training, which tends to increase the entropy of predictions following the softmax, which is equivalent to lower conﬁdence (Guo et al., 2017).
Revised: In turn, not learning a scale and offset for this batch-normalization layer ensures that batches ﬂowing into the softmax layer have standard deviations that do not grow throughout training, which tends to increase the entropy of predictions following the softmax, which is equivalent to lower conﬁdence (Guo et al., 2017).

Original: After observing success with these methods in 10 × Wide ResNets, we then observed that learning the batch-norm parameters in other layers also led to increased overﬁtting, and increased test error rates, and so removed that learning in all layers (except the ﬁrst one applied to the input, when ReLU is used there).
Revised: After observing success with these methods in 10 × Wide ResNets, we then observed that learning the batch-norm parameters in other layers also led to increased overﬁtting, and increased test error rates, and so removed that learning in all layers (except the ﬁrst one applied to the input, when ReLU is used there).

Original: As shown in Figure 6, there are signiﬁcant beneﬁts from this approach, for both full-precision and 1bit-per-weight networks.
Revised: As shown in Figure 6, there are signiﬁcant beneﬁts from this approach, for both full-precision and 1bit-per-weight networks.

Original: It is why, in Table 3, our results surpass those of Zagoruyko & Komodakis (2016) on effectively the same Wide ResNet (our 20-10 network is essentially the same as the 22comparison network, where the extra 2 layers appear due to the use of learned 1 × 1 convolutional projections in downsampling residual paths, whereas we use average pooling instead).
Revised: It is why, in Table 3, our results surpass those of Zagoruyko & Komodakis (2016) on effectively the same Wide ResNet (our 20-10 network is essentially the same as the 22comparison network, where the extra 2 layers appear due to the use of learned 1 × 1 convolutional projections in downsampling residual paths, whereas we use average pooling instead).

Original: As expected from the motivation, we found our method is not appropriate for datasets such as ImageNet32 where overﬁtting is not as evident, in which case learning the batch normalization parameters signiﬁcantly reduces test error rates.
Revised: As expected from the motivation, we found our method is not appropriate for datasets such as ImageNet32 where overﬁtting is not as evident, in which case learning the batch normalization parameters signiﬁcantly reduces test error rates.

Original: 5. C OMPARISON WITH S QUEEZE N ET
Revised: 5. C OMPARISON WITH S QUEEZE N ET

Original: Here we compare our approach with SqueezeNet (Iandola et al., 2016), which reported signiﬁcant memory savings for a trained model relative to an AlexNet.
Revised: Here we compare our approach with SqueezeNet (Iandola et al., 2016), which reported signiﬁcant memory savings for a trained model relative to an AlexNet.

Original: The SqueezeNet approach uses two strategies to achieve this: (1) replacing many 3x3 kernels with 1x1 kernels; (2) deep compression (Han et al., 2015).
Revised: The SqueezeNet approach uses two strategies to achieve this: (1) replacing many 3x3 kernels with 1x1 kernels; (2) deep compression (Han et al., 2015).

Original: Regarding SqueezeNet strategy (1), we note that SqueezeNet is an all-convolutional network that closely resembles the ResNets used here.
Revised: Regarding SqueezeNet strategy (1), we note that SqueezeNet is an all-convolutional network that closely resembles the ResNets used here.

Original: We experimented brieﬂy with our 1-bit-per-weight approach in many all-convolutional variants—e.g. plain all-convolutional (Springenberg et al., 2014),
Revised: We experimented brieﬂy with our 1-bit-per-weight approach in many all-convolutional variants—e.g. plain all-convolutional (Springenberg et al., 2014),

Original: SqueezeNet (Iandola et al., 2016), MobileNet (Howard et al., 2017), ResNexT (Xie et al., 2016)— and found its effectiveness relative to full-precision baselines to be comparable for all variants.
Revised: SqueezeNet (Iandola et al., 2016), MobileNet (Howard et al., 2017), ResNexT (Xie et al., 2016)— and found its effectiveness relative to full-precision baselines to be comparable for all variants.

Original: We also observed in many experiments that the total number of learned parameters correlates very well with classiﬁcation accuracy.
Revised: We also observed in many experiments that the total number of learned parameters correlates very well with classiﬁcation accuracy.

Original: When we applied a SqueezeNet variant to CIFAR-100, we found that to obtain the same accuracy as our ResNets for about the same depth, we had to increase the width until the SqueezeNet had approximately the same number of learned parameters as the ResNet.
Revised: When we applied a SqueezeNet variant to CIFAR-100, we found that to obtain the same accuracy as our ResNets for about the same depth, we had to increase the width until the SqueezeNet had approximately the same number of learned parameters as the ResNet.

Original: We conclude that our method therefore reduces the model size of the baseline SqueezeNet architecture (i.e. when no deep compression is used) by a factor of 32, albeit with an accuracy gap.
Revised: We conclude that our method therefore reduces the model size of the baseline SqueezeNet architecture (i.e. when no deep compression is used) by a factor of 32, albeit with an accuracy gap.

Original: Regarding SqueezeNet strategy (2), the SqueezeNet paper reports deep compression (Han et al., 2015) was able to reduce the model size by approximately a factor of 10 with no accuracy loss.
Revised: Regarding SqueezeNet strategy (2), the SqueezeNet paper reports deep compression (Han et al., 2015) was able to reduce the model size by approximately a factor of 10 with no accuracy loss.

Original: Our method reduces the same model size by a factor of 32, but with a small accuracy loss that gets larger as the full-precision accuracy gets smaller.
Revised: Our method reduces the same model size by a factor of 32, but with a small accuracy loss that gets larger as the full-precision accuracy gets smaller.

Original: It would be interesting to explore whether deep compression might be applied to our 1-bit-per-weight models, but our own focus is on methods that minimally alter training, and we leave investigation of more complex methods for future work.
Revised: It would be interesting to explore whether deep compression might be applied to our 1-bit-per-weight models, but our own focus is on methods that minimally alter training, and we leave investigation of more complex methods for future work.

Original: Regarding SqueezeNet performance, the best accuracy reported in the SqueezeNet paper for ImageNet is 39.6% top-1 error, requiring 4.8MB for the model’s weights.
Revised: Regarding SqueezeNet performance, the best accuracy reported in the SqueezeNet paper for ImageNet is 39.6% top-1 error, requiring 4.8MB for the model’s weights.

Original: Our single-bit-weight models achieve better than 33% top-1 error, and require 8.3 MB for the model’s weights.
Revised: Our single-bit-weight models achieve better than 33% top-1 error, and require 8.3 MB for the model’s weights.

Original: 5. L IMITATIONS AND F URTHER W ORK
Revised: 5. L IMITATIONS AND F URTHER W ORK

Original: In this paper we focus only on reducing the precision of weights to a single-bit, with beneﬁts for model compression, and enabling of inference with very few multiplications.
Revised: In this paper we focus only on reducing the precision of weights to a single-bit, with beneﬁts for model compression, and enabling of inference with very few multiplications.

Original: It is also interesting and desirable to reduce the computational load of inference using a trained model, by carrying out layer computations in very few numbers of bits (Hubara et al., 2016; Rastegari et al., 2016;
Revised: It is also interesting and desirable to reduce the computational load of inference using a trained model, by carrying out layer computations in very few numbers of bits (Hubara et al., 2016; Rastegari et al., 2016;

Original: Cai et al., 2017).
Revised: Cai et al., 2017).

Original: Facilitating this requires modifying non-linear activations in a network from ReLUs to quantized ReLUs, or in the extreme case, binary step functions.
Revised: Facilitating this requires modifying non-linear activations in a network from ReLUs to quantized ReLUs, or in the extreme case, binary step functions.

Original: Here we use only full-precision calculations.
Revised: Here we use only full-precision calculations.

Original: It can be expected that combining our methods with reduced precision processing will inevitably increase error rates.
Revised: It can be expected that combining our methods with reduced precision processing will inevitably increase error rates.

Original: We have addressed this extension in a forthcoming submission.
Revised: We have addressed this extension in a forthcoming submission.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was supported by a Discovery Project funded by the Australian Research Council (project number DP170104600).
Revised: This work was supported by a Discovery Project funded by the Australian Research Council (project number DP170104600).

Original: Discussions and visit hosting by Gert Cauwenberghs and Hesham Mostafa, of UCSD, and Andr´e van Schaik and Runchun Wang of Western Sydney University are gratefully acknowledged, as are discussions with Dr Victor Stamatescu and Dr Mohammad Hasan of University of South Australia.
Revised: Discussions and visit hosting by Gert Cauwenberghs and Hesham Mostafa, of UCSD, and Andr´e van Schaik and Runchun Wang of Western Sydney University are gratefully acknowledged, as are discussions with Dr Victor Stamatescu and Dr Muhammad Abul Hasan of University of South Australia.

Original: A D IFFERENCES FROM S TANDARD R ES N ETS IN OUR B ASELINE A. 
Revised: A D IFFERENCES FROM S TANDARD R ES N ETS IN OUR B ASELINE A.

Original: O UR
Revised: O UR

Original: For skip connections between feature maps of different sizes, we use zero-padding for increasing the number of channels as per option 1 of He et al.
Revised: For skip connections between feature maps of different sizes, we use zero-padding for increasing the number of channels as per option 1 of He et al.

Original: (2015b).
Revised: (2015b).

Original: However, for the residual pathway, we use average pooling using a kernel of size 3 × 3 with stride 2 for downsampling, instead of typical lossy downsampling that discards all pixel values in between samples.
Revised: However, for the residual pathway, we use average pooling using a kernel of size 3 × 3 with stride 2 for downsampling, instead of typical lossy downsampling that discards all pixel values in between samples.

Original: A. W E USE BATCH NORMALIZATION APPLIED TO THE INPUT LAYER
Revised: A. W E USE BATCH NORMALIZATION APPLIED TO THE INPUT LAYER

Original: The literature has reported various options for the optimal ordering, usage and placement of BN and
Revised: The literature has reported various options for the optimal ordering, usage and placement of BN and

Original: ReLU layers in residual networks.
Revised: ReLU layers in residual networks.

Original: Following He et al.
Revised: Following He et al.

Original: (2016); Zagoruyko & Komodakis (2016)
Revised: (2016); Zagoruyko & Komodakis (2016)

Original: , we precede convolutional layers with the combination of BN followed by ReLU.
Revised: , we precede convolutional layers with the combination of BN followed by ReLU.

Original: However, different to He et al.
Revised: However, different to He et al.

Original: (2016); Zagoruyko & Komodakis (2016)
Revised: (2016); Zagoruyko & Komodakis (2016)

Original: , we also insert BN (and optional ReLU) immediately after the input layer and before the ﬁrst convolutional layer.
Revised: , we also insert BN (and optional ReLU) immediately after the input layer and before the ﬁrst convolutional layer.

Original: When the optional ReLU is used, unlike all other batch-normalization layers, we enable learning of the scale and offset factors.
Revised: When the optional ReLU is used, unlike all other batch-normalization layers, we enable learning of the scale and offset factors.

Original: This ﬁrst BN layer enables us to avoid doing any pre-processing on the inputs to the network, since the BN layer provides necessary normalization.
Revised: This ﬁrst BN layer enables us to avoid doing any pre-processing on the inputs to the network, since the BN layer provides necessary normalization.

Original: When the optional ReLU is included, we found that the learned offset ensures the input to the ﬁrst ReLU is never negative.
Revised: When the optional ReLU is included, we found that the learned offset ensures the input to the ﬁrst ReLU is never negative.

Original: In accordance with our strategy of simplicity, all weight layers can be thought of as a block of three operations in the same sequence, as indicated in Figure 3.
Revised: In accordance with our strategy of simplicity, all weight layers can be thought of as a block of three operations in the same sequence, as indicated in Figure 3.

Original: Conceptually, batch-norm followed by
Revised: Conceptually, batch-norm followed by

Original: ReLU can be thought of as a single layer consisting of a ReLU that adaptively changes its centre point and positive slope for each channel and relative to each mini-batch.
Revised: ReLU can be thought of as a single layer consisting of a ReLU that adaptively changes its centre point and positive slope for each channel and relative to each mini-batch.

Original: We also precede the global average pooling layer by a BN layer, but do not use a ReLU at this point, since nonlinear activation is provided by the softmax layer.
Revised: We also precede the global average pooling layer by a BN layer, but do not use a ReLU at this point, since nonlinear activation is provided by the softmax layer.

Original: We found including the ReLU leads to differences early in training but not by the completion of training.
Revised: We found including the ReLU leads to differences early in training but not by the completion of training.

Original: A. O UR FIRST CONV LAYER HAS AS MANY CHANNELS AS THE FIRST RESIDUAL BLOCK
Revised: A. O UR FIRST CONV LAYER HAS AS MANY CHANNELS AS THE FIRST RESIDUAL BLOCK

Original: The Wide ResNet of Zagoruyko & Komodakis (2016) is speciﬁed as having a ﬁrst convolutional layer that always has a constant number of output channels, even when the number of output channels for other layers increases.
Revised: The Wide ResNet of Zagoruyko & Komodakis (2016) is speciﬁed as having a ﬁrst convolutional layer that always has a constant number of output channels, even when the number of output channels for other layers increases.

Original: We found there is no need to impose this constraint, and instead always allow the ﬁrst layer to share the same number of output channels as all blocks at the ﬁrst spatial scale.
Revised: We found there is no need to impose this constraint, and instead always allow the ﬁrst layer to share the same number of output channels as all blocks at the ﬁrst spatial scale.

Original: The increase in the total number of parameters from doing this is small relative to the total number of parameters, since the number of input channels to the ﬁrst layer is just 3.
Revised: The increase in the total number of parameters from doing this is small relative to the total number of parameters, since the number of input channels to the ﬁrst layer is just 3.

Original: The beneﬁt of this change is increased simplicity in the network deﬁnition, by ensuring one fewer change in the dimensionality of the residual pathway.
Revised: The beneﬁt of this change is increased simplicity in the network deﬁnition, by ensuring one fewer change in the dimensionality of the residual pathway.

Original: B C OMPARISON OF R ESIDUAL N ETWORKS AND P LAIN CNN S
Revised: B C OMPARISON OF R ESIDUAL N ETWORKS AND P LAIN CNN S

Original: We were interested in understanding whether the good results we achieved for single-bit weights were a consequence of the skip connections in residual networks.
Revised: We were interested in understanding whether the good results we achieved for single-bit weights were a consequence of the skip connections in residual networks.

Original: We therefore applied our method to plain all-convolutional networks identical to our 4 × residual networks, except with the skip connections removed.
Revised: We therefore applied our method to plain all-convolutional networks identical to our 4 × residual networks, except with the skip connections removed.

Original: Initially, training indicated a much slower convergence, but we found that altering the initial weights standard deviations to be proportional to 2 instead of √ 2 helped, so this was the only other change made.
Revised: Initially, training indicated a much slower convergence, but we found that altering the initial weights standard deviations to be proportional to 2 instead of √ 2 helped, so this was the only other change made.

Original: The change was also applied in Equation (1).
Revised: The change was also applied in Equation (1).

Original: As summarized in Figure 7, we found that convergence remained slower than our ResNets, but there was only a small accuracy penalty in comparison with ResNets after 126 epochs of training.
Revised: As summarized in Figure 7, we found that convergence remained slower than our ResNets, but there was only a small accuracy penalty in comparison with ResNets after 126 epochs of training.

Original: This is consistent with the ﬁndings of He et al.
Revised: This is consistent with the ﬁndings of He et al.

Original: (2015b) where only ResNets deeper than about 20 layers showed a signiﬁcant advantage over plain all-convolutional networks.
Revised: (2015b) where only ResNets deeper than about 20 layers showed a signiﬁcant advantage over plain all-convolutional networks.

Original: This experiment, and others we have done, support our view that our method is not particular to ResNets.
Revised: This experiment, and others we have done, support our view that our method is not particular to ResNets.

Original: [Figure]
Revised: [Figure]