FTBNN: R ETHINKING N ON - LINEARITY FOR 1- BIT CNN S AND G OING B EYOND
Anonymous authors
Paper under double-blind review
A BSTRACT
Binary neural networks (BNNs), where both weights and activations are binarized into 1 bit, have been widely studied in recent years due to its great beneﬁt of highly accelerated computation and substantially reduced memory footprint that appeal to the development of resource constrained devices.
In contrast to previous methods tending to reduce the quantization error for training BNN structures, we argue that the binarized convolution process owns an increasing linearity towards the target of minimizing such error, which in turn hampers BNN’s discriminative ability.
In this paper, we re-investigate and tune proper non-linear modules to ﬁx that contradiction, leading to a strong baseline which achieves state-of-theart performance on the large-scale ImageNet dataset in terms of accuracy and training efﬁciency.
To go further, we ﬁnd that the proposed BNN model still has much potential to be compressed by making a better use of the efﬁcient binary operations, without losing accuracy.
In addition, the limited capacity of the BNN model can also be increased with the help of group execution.
Based on these insights, we are able to improve the baseline with an additional 4 ∼ 5% topaccuracy gain even with less computational cost.
Our code and all trained models will be made public.
I NTRODUCTION
In the past decade, Deep Neural Networks (DNNs), in particular Deep Convolutional Neural
Networks (DCNNs), has revolutionized computer vision and been ubiquitously applied in various computer vision tasks including image classiﬁcation (Krizhevsky et al., 2012), object detection (Liu et al., 2020a) and semantic segmentation (Minaee et al., 2020).
The top performing DCNNs (He et al., 2016; Huang et al., 2017) are data and energy hungry, relying on cloud centers with clusters of energy hungry processors to speed up processing, which greatly impedes their deployment in ubiquitous edge devices such as smartphones, automobiles, wearable devices and IoTs which have very limited computing resources.
Therefore, in the past few years, numerous research effort has been devoted to developing DNN compression techniques to pursue a satisfactory tradeoff between computational efﬁciency and prediction accuracy (Deng et al., 2020).
Among various DNN compression techniques, Binary Neural Networks (BNNs), ﬁrstly appeared in the pioneering work by Hubara et al.
(2016), have attracted increasing attention due to their favorable properties such as fast inference, low power consumption and memory saving.
In a
BNN, the weights and activations during inference are aggressively quantized into 1-bit (namely two values), which can lead to 32 × saving in memory footprint and up to 64 × speedup on CPUs (Rastegari et al., 2016).
However, the main drawback of BNNs is that despite recent progress (Liu et al., 2018; Gu et al., 2019;
Kim et al., 2020b), BNNs have trailed the accuracy of their full-precision counterparts.
This is because the binarization inevitably causes serious information loss due to the limited representational capacity with extreme discreteness.
Additionally, the discontinuity nature of the binarization operation brings difﬁculty to the optimization of the deep network (Alizadeh et al., 2018).
A popular direction on enhancing the predictive performance of a BNN is to make the binary operation mimic the behavior of its full-precision counterpart by reducing the quantization error cuased by the binarization function.
For example, XNOR-Net (Rastegari et al., 2016) ﬁrstly introduced scaling factors for both the binary weight and activation such that the output of the [Figure] BN [Figure]
x l x l+
[Figure]
Binary
[Figure] x l x l+ BN Addition
Block chain in Bi-Real Net:
Block chain in FTBNN:
[Figure]
FTBNN / :
[Figure]
shown in left : FPReLU : ReLU
[Figure]
Epoch
[Figure]
op - 1 A cc u r a cy
T r a i n i ng
Lo ss binary convolution can be rescaled to closely match the result of the real-valued convolution just like the original full-precision weight and activation are used.
The method outperforms its vanilla counterpart BNN (Hubara et al., 2016) by a large margin (44.2% vs. 27.9% in Top-1 accuracy on
ImageNet using the AlexNet architecture (Krizhevsky et al., 2012)).
Because of the remarkable success of XNOR-Net, a series of approaches emerged subsequently with the effort of either ﬁnding better scaling factors or proposing novel optimization strategies to further reduce the quantization error.
Speciﬁcally, XNOR-Net++ (Bulat & Tzimiropoulos, 2019) improved the way of calculating the scaling factors by regarding them as model parameters which can be learnt end-to-end from the target loss.
While Real-to-Bin (Martinez et al., 2020) proposed to compute the scaling factors on the ﬂy according to individual input samples, which is more ﬂexible.
From another perspective, IRNet (Qin et al., 2020) progressively evolved the backward function for binarization from an identity map into the original Sign function during training, which can avoid big quantization error in the early stage of training.
BONN (Gu et al., 2019) added a Bayesian loss to encourage the weight kernel following a Gaussian mixture model with each Gaussian centered at each quantization value, leading to higher accuracy.
Other works aiming to reduce the quantization error also include ABC-Net (Lin et al., 2017), Bi-Real Net (Liu et al., 2018),
ProxyBNN (He et al., 2020), etc.
However, another problem arises with the quantization error optimized towards 0, especially for the structure like Bi-Real Net ( Fig. 1), where the only non-linear function is the binarization function.
The non-linearity of the binarization function will be eliminated if the binary convolution with scaling factors can perfectly mimic the real-valued convolution in the extreme case (quantization error equals to 0), thus hindering the discriminative ability of BNNs.
Therefore, it is necessary to re-investigate the non-linear property of BNNs when inheriting existing advanced structures.
Based on this consideration, we conduct the experiment on MNIST dataset (LeCun & Cortes, 2010) using a 2-layer Bi-Real Net like structure (which begins with an initial real-valued convolution layer and two basic blocks illustrated in Fig.
1 (b), optionally followed by a non-linear module, and ends with a fully connected (FC) layer) and some interesting phenomenons can be found as shown in Fig.
2, where we visualized the feature space before the FC layer and calculated the feature discrepancy caused by the binarization process as well as the corresponding classiﬁcation accuracy (ACC, in %) for each model.
Firstly, comparing the ﬁrst two ﬁgures, despite the big quantization error made by binarization, the binary model achieves much higher accuracy than the real-valued model, which does not have quantization error.
This indicates that the binarization function owns a potential ability to enhance the model’s discriminative power, and also explains why Bi-Real Net [Figure] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
can possibly achieve high accuracy even without any non-linear modules.
Such ability should be respected when we design the BNN structures rather than minimizing the quantization error to 0.
Second, when explicitly introducing a non-linear module, either PReLU (He et al., 2015) or ReLU, the discriminative power can be further enhanced, even though the feature discrepancy is enlarged.
This again shows that there is no strict correlation between the quantization discrepancy and the ﬁnal predictive performance.
Meanwhile, the ReLU model brings the most discriminative power with a clearer class separateness in the feature space.
Our following experiments also show the superiority of the traditional ReLU function due to a stronger non-linearity .
Motivated by the above observation, we propose FTBNN (meaning fast training, see Fig. 1), by ﬁrstly eliminating the scaling factors after the binary convolutional operation, which are instead fused into the Batch Normalization (BN) (Ioffe & Szegedy, 2015) layers by leveraging BN’s scaling factors, and explicitly conﬁguring appropriate non-linear modules after each block.
By doing so, we can release the optimization burden for extra scaling layers as well as keep the non-linear property of the Sign function via a weaker scaling mechanism.
At the same time, the conﬁguration of explicit non-linear modules (we also proposed a more ﬂexible non-linear module, namely Fully Parametric ReLU (FPReLU) shown in Fig. 3) is able to further boost the discriminative ability of the BNN model as well as ensure the training stability.
The resulted FTBNN is highly competitive among the state-of-the-art BNN frameworks, with its enhanced performance in terms of both accuracy and training efﬁciency, as shown in Table 1 and Fig. Note that very recently Liu et al.
(2020b) also demonstrated the importance of non-linearity by redistributing activations in BNNs.
To go beyond the baseline structure we have already obtained, we also notice that two aspects in the literature were rarely investigated.
One is to think how far we can go to make use of the fast and light binary operations and avoid the use of expensive real-valued counterparts, to make our BNN model more compact and computationally efﬁcient.
This is motivated by the fact that the realvalued operations in a BNN structure often take a considerable proportion of computation cost ( e.g. , 85% in an 18-layer Bi-Real Net, and so does the proposed FTBNN, see Fig. 3).
Second, inspired by Binary MobileNet (Phan et al., 2020) which leveraged group convolution (Xie et al., 2017) to automatically conﬁgure an optimal binary network architecture in their network search space, we consider how group convolution can affect the performance of BNN structures (in representational capacity, discriminative power, etc. )
with different conﬁgurations (vary in depth and width).
To the best of our knowledge this paper is also the ﬁrst attempt to give such a speciﬁc investigation and can lead to useful insights on designing better BNN structures with the group mechanism.
As for our experimental outcomes, ﬁrstly, we are able to build an easily trainable strong baseline
FTBNN by improving the popular Bi-Real Net structure, already achieving the state-of-the-art performance (Table 1, Table 3 and Table 4).
Secondly, we enhanced the FTBNN by circumventing the usage of real-valued operations and incorporating the group mechanism, leading to a series of derived models that not only surpass the baseline in higher accuracy but also have less computational overhead (Table 4).
It is also hoped that our exploration can give a better understanding of BNN design in the community and derive more efﬁcient binary structures.
M ETHODS 2. B ACKGROUND AND FTBNN
In order to reduce the quantization error (denote as E ) and protect the information ﬂow in BNNs, which has been an important focus of recent years’ studies, a very basic idea is introducing several scaling factors to reweight the output of the binary convolution, such that the rescaled output can approach closely to the result as if using the real-valued operation:
[Equation]
where W , A , W b and A b indicate the real-valued weight and activation, the binarized weight and activation respectively, ⊗ represents a multiplicative convolution and ⊕ represents a convolution using the XNOR-Count operations (or binary convolution), Γ is the scaling factors and (cid:12) is elementwise multiplication.
In XNOR-Net (Rastegari et al., 2016), Γ actually consisted of two separate parts, for the weights and activations respectively and both were calculated in an analytical way during the forward pass, by solving an optimization problem, which is therefore time-consuming.
XNOR-Net++ (Bulat &
Tzimiropoulos, 2019) further argued that Γ can be a single part and learnt during training, showing better performance in terms of both accuracy and efﬁciency.
More recently, Real-to-Bin (Martinez et al., 2020) proposed a data-driven version of Γ , which was computed on the ﬂy according to individual input samples.
In fact, with the purpose of ﬁnding a satisfactory initialization for Γ , usually a pretrained model with full-precision weights is needed (Rastegari et al., 2016; Lin et al., 2017).
However, it is shown that the scaling factors can be fusable with the parameters in BN (Liu et al., 2018), directly training the
BN layers can avoid the above troublesome, without degenerating the whole structure (Bethge et al., 2019).
This also matches our assumption on Section 1 that the BN scaling factors can be a good alternative for an explicit scaling layer.
On the other hand, to maintain the information ﬂow and maximize BNN’s expressive ability, another aspect of experience in the literature is the usage of shortcut connections (He et al., 2016).
One milestone about that is the Bi-Real Net structure (Liu et al., 2018), where the authors emphasized using the shortcut which connects the real-valued activations before the Sign function to the output of binary convolution, and proved the almost computation-free shortcut can signiﬁcantly preserve
BNN’s representational capability.
It was then repeatedly adopted in other works (Liu et al., 2020b;
Bulat et al., 2020;
Kim et al., 2020a) to improve the quality of information propagation.
Standing on the shoulders of giants and according to our discuss in Section 1, we propose a simple block based structure, dubbed as FTBNN 1 (Fig. 1) that both enables training efﬁciency and encourages non-linearity by discarding the scaling factors which are implicitly fused into BN, and [Figure] adding speciﬁc activation functions.
Contrary to Tang et al.
(2017) and Bulat et al.
(2019), we ﬁnd that traditional ReLU is still more effective than PReLU with a stronger non-linear property.
While at the same time, immoderately inserting ReLUs in the network can also cause divergence and deterioration, as the information collapse phenomenon (Sandler et al., 2018) become serious.
To alleviate that, we extend PReLU to Fully Parametric ReLU (FPReLU) where the slops for both negative and positive activations are learnable (Fig. 3), to stably maximize BNN’s ﬂexibility
. 2. T OWARDS A MORE COMPACT BNN WITH B INARY P URIFICATION
Real-valued operations in BNN are often needed to drive and preserve the information ﬂow ( e.g. , binarization for the 1x1 convolutional shortcuts in the reduction block of Bi-Real Net will interrupt the continuous full-precision information ﬂow, the damage of which is hard to recover in the following layers (Bethge et al., 2019)).
From Fig.
3, we ﬁnd that the most computation-consuming parts among the real-valued operations in FTBNN are the initial convolutional layer and the real-valued 1x1 convolutional shortcuts.
Thereby we make the following two changes to relieve the computation burden for the baseline.
Initial Conv: Inspired by Bethge et al.
(2020), in the full-precision initial convolutional layer, we set the kernel size to 3x3 and constrain the number of output channels (only half of that in the baseline in our experiments).
To facilitate building BNNs with different width conﬁgurations, as we will discuss later, we add an extra binary convolutional layer after the initial convolutional layer as a bridge that can arbitrarily expand the initial width (see appendix for more detail).
Downsampling shortcuts: Similar to Liu et al. (2020b), we propose a copy-and-paste strategy to avoid a convolutional downsampling shortcut to save computation while preserving the information ﬂow.
In a reduction block where the resolutions of feature maps are halved and the number of ﬁlters is doubled, we ﬁrstly concatenate the duplicated input volume to extend the input, then employ a 3x3 average pooling with stride=2 to construct the shortcut (Fig. 3).
2. F EATURE AGGREGATION WITH G ROUP E XECUTION
As discussed by Phan et al.
(2020), binary operation restricts BNN’s feature representation only using limited discrete values.
Simply enlarging the network by increasing the number of channels can be very expensive since a N × widening leads to a N 2 × growth in computational budget.
We exploit the group convolution (Xie et al., 2017) to enable a wider network (Fig. 3), which we believe can strengthen BNN’s representational ability, while keeping the total overhead unchanged.
The differences to other similar works are as follows.
Compared with methods using multiple bases or branches: Lin et al.
(2017) and Zhuang et al.
(2019) gathered results from multiple branches or bases to enrich the feature representation for binary convolutions.
Their methods can be equally viewed as running several independent networks parallelly followed by a summation, while ours only uses a single pipeline without introducing additional computational overhead.
In the form of information aggregation, concatenation is used in group convolution, which is different to summation adopted in these methods.
Compared with methods leveraging group convolutions: Group convolution is also incorporated in recent approaches to facilitate binary network architecture search (NAS) (Phan et al., 2020;
Bulat et al., 2020), while unfortunately a proper ablation study is missing, making the beneﬁts from the group convolution unclear under the whole framework where the resulted advantage may be biased towards the NAS technology itself.
Our study can be regarded as such an speciﬁc investigation to make a better understanding of group convolution when designing BNNs.
E XPERIMENTS AND D ISCUSSES
FTBNN Baseline: As shown in Fig.
1, to give the model enough non-linearity while maintaining training stability, we insert a ReLU after every four blocks and for other blocks, FPReLU is adopted.
Compact and efﬁcient derived models: Based on our baseline and the proposed binary puriﬁcation pipeline, we set the number of output channels in the initial convolutional layer as 32.
A bridge binary convolutional layer is then connected, followed by the blocks with group convolution as shown in Fig.
3 (also see the appendix) and the non-linear modules.
Finally, a FC layer followed by a soft-max layer is applied to do the classiﬁcation.
In order to observe the effect of group convolution, we elaborate different computational budgets for BNNs (calculated in FLOPs as stated in Fig.
3, following Liu et al.
(2018) and Rastegari et al.
by varying width and depth.
Replacing XNOR-Count when ReLU is used: In Fig.
1 (b), activation values before Sign are either 0 or positive if ReLU is applied at the end of the previous block.
Then, all the 0s would be assigned to the lower binary value with the Sign function, eliminating the effect of ReLU in the residual branch.
In this case, we have designed particular bit operations, which is illustrated in the appendix.
All the models are simply trained using Adam optimizer (Kingma & Ba, 2014) for 60 epochs from scratch with a starting learning rate 0.01, which is decayed in a multi-step way (at epoch 45 and 55, with the decaying rate 0.1) for the baseline, and a cosine shape scheme (towards 0 at the end of training) for the derived models.
However we note there is no big difference between this two ways.
The weight decay is set to 0 for all the binary convolutional layers.
We use STE (Hubara et al., 2016) for backward propagation with value clipping in range (-1.2, 1.2) for both weights and activations.
The Pytorch library (Paszke et al., 2019) is applied for implementation.
Finally, following Bi-Real
Net, we use the large-scale challenging ImageNet benchmark (ILSVRC 2012, Deng et al.
(2009)) in our experiments.
A more detailed description for the implementation can be found in the appendix.
3. A BLATION ON FTBNN B ASELINE
The most similar one of the proposed baseline is the Bi-Real Net structure.
Where, the authors used a 2-stage training scheme to get a good initialization for BNN, a magnitude-aware gradient approximation was also considered during weights updating.
Finally, the trained scaling factors are absorbed into the BN layers.
The whole training procedure was carefully tuned in order to fully exploit the model’s capacity.
It can be seen in Table 3 that when applying our naive training scheme, the Bi-Real Net structure (Fig. 1 (a)) deleteriously degraded (from 56.4% to 52.3% on Top-1 accuracy).
The main deﬁciency of this structure is the lack of a proper non-linearity introduction, which is demonstrated to be indispensable for BNNs.
In addition, our structure without any non-linear module performs slightly better than the Bi-Real Net structure (52.5% vs. 52.3%), demonstrating the beneﬁts of a weaker scaling mechanism from the BN layers.
The following experiments then focus on the effectiveness of different non-linear modules.
From
Table 3, we can ﬁnd that the ReLU function is still preferred with a stronger non-linear property, that is able to bring in more discriminative power.
For example, only inserting 4 ReLUs can outperforms
[Table] [Table]
the one with all blocks connected with PReLU by 0.7% in Top-1 accuracy.
Note that ReLU is a special case of PReLU [Table] indicating ReLU is already a decent local optima for PReLU by giving a strong non-linearity
[Table]
making the optimization easier.
At the same time, if the usage of ReLU is limitless
[Table]
the BNN would also be prone to divergence, showing that unlike the full-precision counterparts, binarized architectures are essentially vulnerable, an abuse of ReLU which causes severe information collapse in the feature space should be avoided
[Table]
A straightforward but effective way to control the non-linearity, so that model’s discriminative power can be maximized, and the stability is also guaranteed, is to control the number of ReLUs and additionally introduce weaker non-linear functions.
Here, we show that compared with PReLU, the proposed FPReLU is an ideal solution to meet this requirement.
As shown in Fig.
3, FPReLU starts from an identity function and gradually evolves during training to compensate the non-linearity in both sides of activations.
Table 2 lists the ultimately learnt coefﬁcients of the FPReLU functions.
These values for both side of the functions vary around 1 with a fairly wide range, some even approach to 0 or a big value ( i.e. , 5), meaning that FPReLU is competent to tune the network into a balanced point automatically, since both slops for negative and positive activations are considered.
3. T HE DERIVED MODELS
In this section, we evaluate the effectiveness of the derived models with binary puriﬁcation by adjusting the number of groups.
Table 4 left shows the inﬂuences of group convlutions on binarypuriﬁed BNNs with varying settings of depth and group numbers, by constraining the computation costs similar.
It can be found that the simplest derived models without using group convolutions ( i.e., , G = 1) can surpass the proposed FTBNN baseline by a large margin (a 1.7% improvement on Top-1 accuracy with computational budget almost halved based on ResNet 18), which strongly demonstrates the effectivenss of the proposed binary puriﬁcation strategy.
It also shows that there is no inﬂuence of the group execution for relatively shallow binary networks under different budget constraints.
As going deeper, the group convolution begin to diverge the corresponding models with notable characteristics.
On one hand, when observing the training accuracy, increasing the number of groups can obviously introduce a better ﬁtting, indicating an increasing representational capacity of the BNN.
The phenomenon can be found where the layer ≥ 34 regardless of the model complexity. [Table]
On the other hand, the positive effect for deeper networks can also activate a better accuracy on the validation set, meaning that an increase in discriminative power are also accompanied.
All the experiments on deeper BNNs with Group = 5 show a clear accuracy boost than those without group execution (Group = 1).
E.g. , from 61.0% to 62.5% for Layer = 34, and from 63.3% to 65.3% for Layer = 44.
However, an increasing representational capacity may also lead to a higher risk of dropping in generability.
For some cases, the accuracy on validation set is higher than that reported on the training set, due to the regularization nature of binary operations.
With group execution, such effect of binarization is gradually weakened, leading to overﬁtting issues.
3. C OMPARISON WITH S TATE - OF - THE - ARTS
Here we compare the proposed FTBNN with existing state-of-the-art methods as illustrated in the right side of Table 4, including the low-bit quantization methods, multiple binarizations methods, other binary frameworks based on the ResNet-18 architecture, etc.
Interestingly, we ﬁnd that some of our derived models are able to achieve superior or comparable performance when compared with the methods using multiple bases or branches, while with far less computational overhead, again showing the effectiveness and efﬁciency of the proposed method.
P ERSPECTIVES
Here, we raise some possible insights from this work: a) To further enhance the model’s discriminative ability through non-linearity, we wonder is it the best way to combine ReLU and
FPReLU?
Is it affected by other elements, such as depth, width and block orders?
b) To what extent can we do the binary puriﬁcation?
As we still have much proportion of real-valued operations in the derived models.
c) As shown in our experiments on BNN, group convolution accompanies with overﬁtting.
How to alleviate overﬁtting for BNNs should also be considered in future works.
A M ORE STRUCTURE INFORMATION
Fig.
4 gives more structure information of FTBNN and the derived models.
Sign
Binary weight (3x3) x
c x w
BN
Addition
Copy (Group = ceil(w / c)) ...
PReLU
BN x in
Weight (3x3) x in : input image x c x c : output of Initial Conv block x w : output of Bridge Conv block with arbitrary width
Initial Conv block (derived)
[Figure]
[Figure]
[Figure]
B R EPLACING XNOR-Count OPERATIONS IN FTBNN AS NEEDED
Suppose we are using the following Sign function to do binarization when doploying our models: [Equation]
[Equation]
) where v r and v b represents the real-valued and binarized values (can be either of weights or activations) respectively.
As we mentioned in Section 3, the input of a basic block will be truncated such that the values of which are either 0 or positive, if a ReLU is connected after the previous block.
Supposing the binarized weights in the current block is w b = [ T, F, T, F, T ] and the input activations before truncation is x r
=
[ − 3 , − 2 , 1 . 5 , 2 , − 1 . 2] , and truncated to x r = [0 , 0 , 1 . 5 , 2 , 0] after ReLU.
According to the Sign function, the activations will then be binarized to x b =
[ F, F, T, T, F ] .
If XNOR-Count operation is used, the result of binary convolution is (the popcount function counts the bits of T ):
[Equation]
The result is the same as if the ReLU is not used, because without ReLU, the original input activations will still be binarized to x b =
[ F, F, T, T, F ] .
Actually, our goal is to make the model trained in PyTorch be deployed using bit operations.
Thus, the output of convolution in PyTorch should be the same when using bit operations.
In the above case, the process of convolution in PyTorch is:
Step 1, using the torch.sign() function as in Appendix E to binarize the truncated x r : x pb = [0 , 0 , 1 , 1 , 0] ; EPOCH [Equation] O
[Equation]
O R R A T E EPOCH Step 2, convolution between x pb and w pb ( ∗ is the convolution function):
[Equation]
Obviously, the output from Eq. 3 does not match the output x po . To match x po when deploying with bit operations, we can use the following calculation: [Equation]
Therefore, the output of bit operations is: x o = popcount ([ F, F, T, T, F ] and [ T, F, T, F, T ]) − popcount
[ F, F, T, T, F ] and [ F, T, F, T, F ])
= popcount ([ F, F, T, F, F ])
[Equation]
popcount [Equation] In addition, Eq. 5 may double the number of BOPs than the original binary convolution which only uses XNOR-Count . We have considered that when calculating the computational budgets for our models using doubled BOPs when ReLU is used.
C E XPERIMENTAL CONFIGURATION FOR TRAINING
Data augmentation: The large-scale ImageNet dataset (ILSVRC 2012, Deng et al.
(2009)) is used in our experiments as the benchmark, consisting of 1.28 million images for training and 50, images for validation.
We adopt the standard data augmentation scheme as in He et al.
during training, including random scale augmentation, random crop to 224 × 224, random horizontal ﬂip with probability 0.5 and per-pixel mean value subtraction.
During testing, we resize the image such that the shorter side is 256 while keeping the original aspect ratio, crop a 224 × 224 patch in the center, and apply the same per-pixel value subtraction as the training set.
Multi-step training: This training scheme is only used to enhance the proposed FTBNN baseline, demonstrating that the baseline is essentially orthogonal to other training insights for BNNs (for other cases in our experiments, the naive training scheme with 60 epochs mentioned in Section 3 is applied).
Following Martinez et al.
(2020), we adopt the progressive teacher-student knowledge distillation with attention mapping for multi-step training.
Speciﬁcally, we ﬁrstly train a fullprecision model by keeping the weights real-valued and use Tanh function instead of Sign function for activations.
Then, a two-stage training scheme is used.
In stage 1, we adopt Sign function for activations, while keeping the weights real-valued.
In stage 2, both weights and activations are binarized with the Sign function.
In both stages, we use the output of the original ResNet-18 model as the soft labels when calculating cross-entropy loss.
D T RAINING CURVES IN GROUP EXECUTION
The training curves for FTBNN with group execution are shown in Fig.
import torch import torch .
nn as nn import torch .
nn .
f u n c t
i
o
n a l as F ######
B i n a r i z a t i o n for a c t i v a t i o n s ###### class Sign ( nn . Module ) :
def i n i t ( self , bound = 1 . 2 ) : super ( Sign , s e l f ) .
i n i t ( ) s e l f .
bound =
bound def forward ( self , x ) : out = torch .
clamp (x , − s e l f .
bound , s e l f . bound ) out forward =
torch .
sign ( x ) y
=
out forward .
detach ( ) + out − out .
detach ( ) return y ######
B i n a r i z a t i o n for weights and binary convolution ###### class BConv( nn .
Module ) :
def i n i t ( self , in chn , out chn , k e r n e l s i z e =3 , s t r i d e =1 , padding =1 , groups =1 , bound = 1. 2 ) :
super (BConv , s e l f ) .
i n i t ( ) s e l f .
s t r i d e = s t
r
i
d
e s e l f .
padding =
padding s e l f .
groups =
groups s e l f .
bound = bound shape =
( out chn , in chn / / groups , k e r n e l s i z e ,
k e r n e l s i z e ) s e l f .
weights = nn .
Parameter ( torch .
Tensor (* shape ) ) torch .
nn .
i n i t .
xavier normal ( s e l f .
weights , gain =2.0) def forward ( self , x ) : clipped weights = torch .
clamp ( s e l f .
weights , \ − s e l f . bound , s e l f . bound ) binary weights no grad torch sign s l f weights ) binary weights = binary weights
no grad detach ( ) + \ clipped weights − clipped weights . detach ( ) out = F conv2d (x binary weights , s t r i d e = s e l f . s t r i d e , padding= s e l f padding groups= s e l f groups ) return out ######
FPReLU ###### class FPReLU( nn .
Module ) :
def i n i t ( self , chn ) : super (FPReLU , s e l f ) .
i n i t ( ) s e l f .
l weights = nn .
Parameter ( torch .
ones (1 , chn , 1 , 1)) s e l f .
r weights = nn .
Parameter ( torch .
ones (1 , chn , 1 , 1)) def forward ( , x ) : x where ) return x