T RAINING B INARY N EURAL N ETWORKS WITH R EALTO -B INARY C ONVOLUTIONS
This paper shows how to train binary networks to within a few percent points ( ∼ 3 − 5% ) of the full precision counterpart.
We ﬁrst show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances and carefully adjusting the optimization procedure.
Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution, additional signiﬁcant accuracy gains can be obtained.
We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in a data-driven manner, by using the real-valued activations, available during inference prior to the binarization process, for re-scaling the activations right after the binary convolution.
Finally, we show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet and reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 accuracy on CIFAR-100 and ImageNet respectively when using a ResNet-18 architecture. Code available at the author’s website.
I NTRODUCTION
Following the introduction of the BinaryNeuralNet (BNN) algorithm
(Courbariaux et al., 2016) binary neural networks emerged as one of the most promising approaches for obtaining highly efﬁcient neural networks that can be deployed on devices with limited computational resources.
Binary convolutions are appealing mainly for two reasons: (a) Model compression: if the weights of the network are stored as bits in a 32-bit ﬂoat, this implies a reduction of 32 × in memory usage.
(b)
Computational speed-up: computationally intensive ﬂoating-point multiply and add operations are replaced by efﬁcient xnor and pop-count operations, which have been shown to provide practical speed-ups of up to 58 × on CPU (Rastegari et al., 2016) and, as opposed to general low bit-width operations, are amenable to standard hardware.
Despite these appealing properties, binary neural networks have been criticized as binarization typically results in large accuracy drops. Thus, their deployment in practical scenarios is uncommon.
For example, on ImageNet classiﬁcation, there is a ∼ 18% gap in top-1 accuracy between a ResNet-18 and its binary counterpart when binarized with XNOR-Net (Rastegari et al., 2016), which is the method of choice for neural network binarization.
But how far are we from training binary neural networks that are powerful enough to become a viable alternative to real-valued networks?
Our ﬁrst contribution in this work is to take stock of recent advances on binary neural networks and train a very strong baseline which already results in state-ofthe-art results.
Our second contribution is a method for bridging most of the remaining gap, which boils down to minimizing the discrepancy between the output of the binary and the corresponding real-valued convolution.
This idea is materialized in our work in two complementary ways: Firstly , we use an attention matching strategy so that the real-valued network can more closely guide the binary network during optimization. However, we show that due to the architectural discrepancies between the real and the binary networks, a direct application of teacher-student produces sub-optimal
[Figure]
BatchNorm
Sign
Convolution
Activation
GlobalAvgPool
Linear
ReLU
Linear
Sigmoid
H × W × C 1 × 1 × C 1 × 1 × 1 × 1 × C 1 × 1 × C
C r C r 1 × 1 × CrCr 1 × 1 × C 1 × 1 × 1 × 1 × C 1 × 1 × C
C r 1 × 1 × Cr
H × W × C H × W × C
H × W × C
H × W × C performance.
Instead, we propose to use a sequence of teacher-student pairs that progressively bridge this architectural gap.
Secondly , we further propose to use the real-valued activations of the binary network, available prior to the binarization preceding convolution, to compute scale factors that are used to re-scale the activations right after the application of the binary convolution.
This is in line with recent works which have shown that using re-scaling the binary convolution output can result in large performance gains (Rastegari et al., 2016; Bulat & Tzimiropoulos, 2019).
However, unlike prior work, we compute the scaling factors in a data-driven manner, based on the real-valued activations of each layer prior to binarization, resulting in further large gains.
Overall, we make the following contributions :
• We construct a very strong baseline by combining some recent insights on training binary networks and by performing a thorough experimentation to ﬁnd the most well-suited optimization techniques.
We show that this baseline already achieves state-of-the-art accuracy, surpassing all previously published work on binary networks.
• We propose a real-to-binary attention matching: this entails that matching spatial attention maps computed at the output of the binary and real-valued convolutions is particularly suited for training binary neural networks (see Fig.
1 left and section 4.2).
We also devise an approach in which the architectural gap between real and binary networks is progressively bridged in smaller increments through a sequence of teacher-student pairs.
• We propose a data-driven channel re-scaling: this entails using the real-valued activations of the binary network prior to the binarization process to compute the scale factors used to re-scale the activations produced right after the application of the binary convolution.
See Fig.
1, right, and section 4.3.
• We show that our combined contributions provide, for the ﬁrst time, competitive results on two standard datasets, achieving 76 .
2% top-1 performance on CIFAR-100, and 65 . 4% top-1 performance on ImageNet when using a ResNet-18 –a gap bellow 3% and 5% respectively compared to their full precision counterparts.
R ELATED WORK
While being pre-dated by other works on binary networks (Soudry et al., 2014), the BNN algorithm (Courbariaux et al., 2016) established how to train networks with binary weights within the familiar back-propagation paradigm.
The training method proposed relies on a real-valued copy of the network weights which is binarized during the forward pass, but is updated during backpropagation ignoring the binarization step.
Unfortunately, BNN resulted in a staggering ∼ 28% gap in top-1 accuracy compared to the full precision ResNet-18 on ImageNet.
It is worth noting that binary networks do have a number of ﬂoating point operations. In fact, the output of a binary convolution is not binary (values are integers resulting from the count).
Also, in accordance to other low bit-width quantization methodologies, the ﬁrst convolution (a costly 7 × 7 kernel in ResNet), the fully connected layer and the batch normalization layers are all realvalued.
In consequence, a line of research has focused methodologies that add a very small amount of real-valued operations in exchange for signiﬁcant accuracy gains.
For example, the seminal work of XNOR-Net (Rastegari et al., 2016) proposed to add a real-valued scaling factor to each output channel of a binary convolution, with XNOR-Net currently considered the method of choice for binary networks.
Similarly, Bi-Real Net (Liu et al., 2018) argued that skip connections are fundamental for binary networks and observed that the ﬂow of full precision activations provided by the skip connections is interrupted by the downsample convolutions. This degrades the signal and make subsequent skip connections less effective.
To alleviate this, they proposed making the downsample layers real-valued, obtaining around 3% accuracy increase in exchange for a small increase in computational complexity.
Improving the optimization algorithm for binary networks has been another fundamental line of research. Examples include the use of smooth approximations of the gradient, the use of PReLU (Bulat et al., 2019), a two-stage training which binarizes the weights ﬁrst and then the activations (Bulat et al., 2019) and progressive quantization (Bulat et al., 2019;
Gong et al., 2019).
The work in (Wang et al., 2019) proposed to learn channel correlations through reinforcement learning to better preserve the sign of a convolution output.
A set of regularizers are added to the loss term in (Ding et al., 2019) so as to control the range of values of the activations, and guarantee good gradient ﬂow.
Other optimization aspects, such the effect of gradient clipping or batch-norm momentum, were empirically tested in (Alizadeh et al., 2019).
In section 4.1, we show how to combine many of the insights provided in the aforementioned works with standard optimization techniques to obtain a very strong baseline that already achieves state-of-the-art accuracy.
While the aforementioned works either maintain the same computational cost, or increase it by a fractional amount, other research has focused instead on relaxing the problem constraints by increasing the number of binary operations by a large amount, typically a factor of 2 to 8 times. Examples include ABC-Net (Lin et al., 2017), the structure approximation of (Zhuang et al., 2019), the circulant CNN of (Liu et al., 2019), and the binary ensemble of (Zhu et al., 2019).
Note that the large increase of binary operations diminishes the efﬁciency claim that justiﬁes the use of binary networks in ﬁrst place.
Furthermore, we will show that there is still a lot of margin in order to bridge the accuracy gap prior to resorting to scaling up the network capacity 1 .
The methodology proposed in this paper has some relations with prior work: our use of attention matching as described in section 4.2 is somewhat related to the feature distillation approach of Zhuang et al.
However, Zhuang et al.
(2018) tries to match the whole feature maps of the to-be-quantized network with the quantized feature maps of a real-valued network that is trained in parallel with the to-be-quantized network.
Such an approach is shown to improve training of low-bit but not binary networks.
Notably, our approach based on matching attention maps is much simpler and shown to be effective for the case of binary networks.
Our data-driven channel re-scaling approach, described in section 4.3, is related to the channel rescaling approach of XNOR-Net, and also that of (Xu & Cheung, 2019; Bulat & Tzimiropoulos, 2019), which propose to learn the scale factors discriminatively through backpropagation.
Contrary to (Xu & Cheung, 2019; Bulat & Tzimiropoulos, 2019), our method is data-driven and avoids using ﬁxed scale factors learnt during training.
Contrary to XNOR-Net, our method discriminatively learns how to produce the data-driven scale factors so that they are optimal for the task in hand.
B ACKGROUND
This section reviews the binarization process proposed in (Courbariaux et al., 2016) and its improved version from (Rastegari et al., 2016), which is the method of choice for neural network binarization.
We denote by W ∈ R o × c × k × k and A ∈ R c × w in × h in the weights and input features of a CNN layer, where o and c represent the number of output and input channels, k the width and height of the kernel, and w in and h in represent the spatial dimension of the input features A .
In (Courbariaux et al., 2016), both weights and activations are binarized using the sign function and then convolution is performed as A ∗ W ≈ sign ( A ) (cid:13)∗ sign ( W ) where (cid:13)∗ denotes the binary convolution, which can be implemented using bit-wise operations.
However, this direct binarization approach introduces a high quantization error that leads to low accuracy.
To alleviate this, XNOR-Net (Rastegari et al., 2016) proposes to use real-valued scaling factors to re-scale the output of the binary convolution as
[Equation]
where (cid:12) denotes the element-wise multiplication, α and K are the weight and activation scaling factors, respectively, calculated in Rastegari et al.
(2016) in an analytic manner.
More recently,
Bulat & Tzimiropoulos (2019) proposed to fuse α and K into a single factor Γ that is learned via backpropagation, resulting in further accuracy gains.
M ETHOD
This section ﬁrstly introduces our strong baseline.
Then, we present two ways to improve the approximation of Eq.
1: Firstly, we use a loss based on matching attention maps computed from the binary and a real- valued network (see
section 4.2)
Secondly, we make the scaling factor a function of the real valued input activations
A (see section 4.3).
4. B UILDING A STRONG BASELINE
Currently, almost all works on binary networks use XNOR-Net and BNN as baselines.
In this section, we show how to construct a strong baseline by incorporating insights and techniques described in recent works as well as standard optimization techniques.
We show that our baseline already achieves state-of-the-art accuracy.
We believe that this is an important contribution towards understanding the true impact of proposed methodologies and towards assessing the true gap with realvalued networks.
Following prior work in binary networks, we focus on the ResNet-18 architecture and apply the improvements listed below:
Block structure : It is well-known that a modiﬁed ResNet block must be used to obtain optimal results for binary networks.
We found the widely-used setting where the operations are ordered as BatchNorm → Binarization → BinaryConv → Activation to be the best.
The skip connection is the last operation of the block (Rastegari et al., 2016).
Residual learning: We used double skip connections, as proposed in (Liu et al., 2018).
Activation: We used PReLU (He et al., 2015) as it is known to facilitate the training of binary networks (Bulat et al., 2019).
Scaling factors: We used discriminatively learnt scaling factors via backpropagation as in (Bulat & Tzimiropoulos, 2019).
Downsample layers: We used real-valued downsample layers (Liu et al., 2018).
We found the large accuracy boost to be consistent across our experiments (around 3 − 4% top-1 improvement on ImageNet).
We used the following training strategies to train our strong baseline:
Initialization: When training binary networks, it is crucial to use a 2-stage optimization strategy (Bulat et al., 2019).
In particular, we ﬁrst train a network using binary activations and real-valued weights, and then use the resulting model as initialization to train a network where both weights and activations are binarized.
Weight decay: Setting up weight decay carefully is surprisingly important.
We use 1 e − 5 when training stage 1 (binary activation and real weights network), and set it to 0 on stage 2 (Bethge et al., 2019).
Data augmentation: For CIFAR-100 we use the standard random crop, horizontal ﬂip and rotation ( ± 15 ◦ ).
For ImageNet, we found that random cropping, ﬂipping and colour jitter augmentation worked the best.
However, colour jitter is disabled for stage 2.
Mix-up: We found that mix-up (Zhang et al., 2017) is crucial for CIFAR-100, while it slightly hurts performance for ImageNet – this is due to the higher risk of overﬁtting on CIFAR-100.
Warm-up: We used warm-up for 5 epochs during stage 1, and no warm-up for stage 2.
Optimizer: We used Adam (Kingma & Ba, 2014) with a stepwise scheduler.
The learning rate is set to 1 e − 3 for stage 1, and 2 e − 4 for stage 2.
For CIFAR-100, we trained for 350 epochs, with steps at epochs 150, 250 and 320.
For ImageNet, we train for 75 epochs, with steps at epochs 40,and 70.
Batch sizes are 256 for ImageNet and 128 for CIFAR-100.
4. R EAL - TO -B INARY A TTENTION M ATCHING
We make the reasonable assumption that if a binary network is trained so that the output of each binary convolution more closely matches the output of a real convolution in the corresponding layer of a real-valued network, then signiﬁcant accuracy gains can be obtained.
Notably, a similar assumption was made in (Rastegari et al., 2016) where analytic scale factors were calculated so that the error between binary and real convolutions is minimized.
Instead, and inspired by the attention transfer method of Zagoruyko & Komodakis (2017), we propose to enforce such a constraint via a loss term at the end of each convolutional block by comparing attention maps calculated from the binary and real-valued activations.
Such supervisory signals provide the binary network with much-needed extra guidance, as it is well-known that backpropagation for binary networks is not as effective as for real-valued ones.
By introducing such loss terms at the end of each block, gradients do not have to traverse the whole network and suffer a degraded signal.
Assuming that attention matching is applied at a set of J transfer points within then network, the total loss can be expressed as:
[Equation]
where Q j = (cid:80) ci =1 |A
i | 2 and A
i is the
i − th channel of activation map A .
Moreover, at the end of the network, we apply a standard logit matching loss (Hinton et al., 2015).
Progressive teacher-student: We observed that teacher and student having as similar architecture as possible is very important in our case.
We thus train a sequence of teacher-student pairs that bridges the differences between the real network and the binary network in small increments:
Step 1 : the teacher is the real-valued network with the standard ResNet architecture.
The student is another real-valued network, but with the same architecture as the binary ResNet-18 (e.g. double skip connection, layer ordering, PReLU activations, etc).
Furthermore, a soft binarization (a Tanh function) is applied to the activations instead of the binarization (sign) function. In this way the network is still real-valued, but it more closely resembles a network with binary activations.
Step 2 : The network resulting from the previous step is used as the teacher.
A network with binary activations and real-valued weights is used as the student.
Step 3: The network resulting from step 2 is used as the teacher and the network with binary weights and binary activations is the student.
In this stage, only logit matching is used
4. D ATA - DRIVEN CHANNEL RE - SCALING
While the approach of the previous section provides better guidance for the training of binary networks, the representation power of binary convolutions is still limited, hindering its capacity to approximate the real-valued network.
Here we describe how to boost the representation capability of a binary neural network while incurring in a negligible increment on the number of real-valued operations.
Previous works have shown the effectiveness of re-scaling binary convolutions with the goal of better approximating real convolutions.
XNOR-Net (Rastegari et al., 2016) proposed to compute these scale factors analytically while (Bulat & Tzimiropoulos, 2019; Xu & Cheung, 2019) proposed to learn them discriminatively in an end-to-end manner, showing additional accuracy gains.
For the latter case, during training, the optimization aims to ﬁnd a set of ﬁxed scaling factors that minimize the average expected loss for the training set.
We propose instead to go beyond this and obtain discriminatively-trained input-dependent scaling factors – thus, at test time, these scaling factors will not be ﬁxed but rather inferred from data.
Let us ﬁrst recall what the signal ﬂow is when going through a binary block.
The activations entering a binary block are actually real-valued.
Batch normalization centers the activations, which are then binarized, losing a large amount of information.
Binary convolution, re-scaling and PReLU follow.
We propose to use the full-precision activation signal, prior to the large information loss incurred by the binarization operation, to predict the scaling factors used to re-scale the output of the binary convolution channel-wise.
Speciﬁcally, we propose to approximate the real convolution as follows:
[Equation]
[Equation]
where W G are the parameters of the gating function G .
Such function computes the scale factors used to re-scale the output of the binary convolution, and uses the pre-convolution real-valued activations as input.
Fig.
1 shows our implementation of function G .
The design is inspired by Hu et al.
, but we use the gating function to predict ahead rather than performing self-referential attention.
An optimal mechanism to modulate the output of the binary convolution clearly should not be the same for all examples as in Bulat & Tzimiropoulos (2019) or Xu & Cheung (2019).
Note that in Rastegari et al.
the computation of the scale factors depends on the input activations.
However the analytic calculation is sub-optimal with respect to the task at hand.
To circumvent the aforementioned problems, our method learns, via backpropagation for the task at hand, to predict the modulating factors using the real-valued input activations.
By doing so, more than 1 / 3 of the remaining gap with the real-valued network is bridged.
4. C OMPUTATIONAL C OST A NALYSIS
Table 1 details the computational cost of the different binary network methodologies.
We differentiate between the number of binary and ﬂoating point operations, including operations such as skip connections, pooling layers, etc.
It shows that our method leaves the number of binary operations constant, and that the FLOPs increases by only 1% of the total ﬂoating point operation count.
This is assuming a factor r of 8, which is the one used in all of our experiments.
To put this into perspective, the magnitude is similar to the operation increase incurred by the XNOR-Net with respect to its predecessor, BNN.
Similarly, the double skip connections proposed in (Liu et al., 2018) adds again a comparable amount of operations.
R ESULTS
We present two main sets of experiments.
We used ImageNet (Russakovsky et al., 2015) as a benchmark to compare our method against other state-of-the-art approaches in Sec. 5.1.
ImageNet is the most widely used dataset to report results on binary networks and, at the same time, allows us to show for the ﬁrst time that binary networks can perform competitively on a large-scale dataset.
We further used CIFAR-100 (Krizhevsky & Hinton, 2009) to conduct ablation studies (Sec. 5.2). [Table]
5. C OMPARISON WITH THE S TATE - OF - THE -A RT
Table 2 shows a comparison between our method and relevant state-of-the-art methods, including low-bit quantization methods other than binary.
Vs.
other binary networks:
Our strong baseline already comfortably achieves state-of-the art results, surpassing the previously best-reported result by about 1% Wang et al.
Our full method further improves over the state-of-the-art by 5 .
5% top-1 accuracy .
When comparing to binary models that scale the capacity of the network (second set of results on Tab. 2), only Zhuang et al.
(2019) outperforms our method, surpassing it by 0.9% top-1 accuracy - yet, this is achieved using 4 times the number of binary blocks.
Vs.
real-valued networks: Our method reduces the performance gap with its real-valued counterpart to ∼ 4% top-1 accuracy, or ∼ 5% if we compare against a real-valued network trained with attention transfer.
Vs.
other low-bit quantization:
Table 2 also shows a comparison to the state-of-the-art for low-bit quantization methods (ﬁrst set of results).
It can be seen that our method surpasses the performance of all methods, except for TTQ (Zhu et al., 2017), which uses 2-bit weights, full-precision activations and 1.5 the channel width at each layer.
5. A BLATION S TUDIES
In order to conduct a more detailed ablation study we provide results on CIFAR-100.
We thoroughly optimized a ResNet-18 full precision network to serve as the real-valued baseline. Teacher-Student effectiveness: We trained a real-valued ResNet-18 using ResNet-34 as its teacher, yielding ∼ 1% top-1 accuracy increase.
Instead, our progressive teacher-student strategy yields ∼ 5% top-1 accuracy gain, showing that it is a fundamental tool when training binary networks, and that its impact is much larger than for real-valued networks, where the baseline optimization is already healthier.
Performance gap to real-valued: We observe that, for CIFAR-100, we close the gap with realvalued networks to about 2% when comparing with the full-precision ResNet-18, and to about 3% when comparing with full-precision using teacher supervision.
The gap is consistent to that on ImageNet in relative terms: 13% and 10% relative degradation on ImageNet and CIFAR-100 respectively.
Binary vs real downsample: Our proposed method achieves similar performance increase irrespective of whether binary or real-valued downsample layers are used, the improvement being 5 . 5% and 6 .
6% top-1 relative accuracy gain respectively.
It is also interesting to note that the results on the ablation study are consistent on both cases.
Scaling factors and attention matching: It is also noteworthy that the gating module is not effective in the absence of attention matching (see SB+G entries).
It seems clear from this result that both are interconnected: the extra supervisory signal is necessary to properly guide the training, while the extra ﬂexibility added through the gating mechanism boosts the capacity of the network to mimic the attention map.
[Table]
[Table]
C ONCLUSION
In this work we showed how to train binary networks to within a few percent points of their realvalued counterpart, turning binary networks from hopeful research into a compelling alternative to real-valued networks.
We did so by training a binary network to not only predict training labels, but also mimic the behaviour of real-valued networks.
To this end, we devised a progressive attention matching strategy to drive optimization, and combined it with a gating strategy for scaling the output of binary convolutions, increasing the representation power of the convolutional block.
The two strategies combine perfectly to boost the state-of-the-art of binary networks by 5 . 5 top-1 accuracy.