MABA-N ET : M ASKED A DDITIVE B INARY A CTIVATION N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite significant reduction in memory footprint and computational cost, binary neural networks suffer from noticeable accuracy degradation compared to realvalued counterparts.
A few works have attempted to narrow the accuracy gap by increasing the representation bit-width or the network width/depth, but they come at the expense of increased memory and/or compute.
In this work, we find that the imbalanced ratio of activations to weights may be the main cause of degraded performance and increased memory overhead.
We propose Masked Additive Binary
Activation Network (MABA-Net) to reduce approximation errors and the activation bit-width, with minimum increase in the activation size.
MABA-Net balances the ratio of the activation size to the weight size, leading to significant memory saving on large CNNs.
We demonstrate MABA-Net’s superior performance on the ImageNet dataset under various network configurations.
Experimental results show that MABA-Net achieves competitive accuracy without increase of computational cost, while reducing memory usage compared to state-of-the-art.
We will release the codes upon acceptance.
I NTRODUCTION
Convolutional neural networks (CNNs) have achieved great success in a wide range of applications but it comes at the expense of large memory and compute consumption.
Subsequently, it is particularly challenging to deploy such networks on resource-limited devices.
Neural network compression (Han et al.
(2016b)) has emerged as a mainstream technique to reduce memory and compute, by either quantizing real-valued weights and activations to low precision numbers (Zhou et al.
or pruning redundant weights from the network (Li et al.
An extreme form of quantization is
Binary Neural Networks (BNNs) (Courbariaux et al.
(2015; 2016)), which replace slow real-valued matrix multiplication with high-speed bit-wise XNOR and POPCOUNT operations.
BNNs have demonstrated their potentials in fast and efficient inference.
Despite the promising advantages over real-valued neural network in terms of memory and compute, vanilla BNNs (Courbariaux et al. (2015; 2016)) suffer from significant accuracy degradation, mainly due to the approximation error caused by real-to-binary projections.
A common way to mitigate the accuracy drop is designing compact and efficient networks (Bulat et al.
Liu et al.
Bethge et al.
However, it is difficult to systematically measure the benefits of each component, such as scaling factor and skip connections.
Another way is increasing the binary network width (Lin et al.
; Zhuang et al.
Zhu et al.
but it comes at the expense of increased memory and compute.
Matrix-vector multiplication is a basic building block in machine learning models, especially in
CNNs.
During the matrix multiplication, the memory access is usually the bottleneck, especially when the matrix is larger than the cache capacity (Han et al.
(2016a)).
On generic CPUs and GPUs, this problem is usually approached by reusing the parameters through batching.
However, batching is unsuitable for real-time applications that are latency-sensitive, e.g., pedestrian detection in autonomous driving, because batching substantially increases latency.
In this work, we target real time applications.
Therefore, we focus on the case of batch size = 1.
An important observation that we made is that, the large memory overhead for such extreme latencysensitive cases is probably due to the imbalanced ratio of activation size (number of activations) to weight size (number of weights).
For example, the number of convolutional activations is 1 {
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
to 1 { 7 of the weights for ResNet-like architectures on ImageNet Bulat et al.
(2021) (see Fig. 1).
In particular, the activation size at most of the layers is an order of magnitude smaller than that of the weight size.
We hypothesise that the relatively small size of activations might cause more severe information bottleneck issue after activation binarization, compared with weight binarization.
Naturally, by balancing the activations and weights, we can make the network less prone to accuracy loss brought by activation binarization.
The experimental results shown in Table 2 suggest that by doing so, our method alleviates the information loss issue caused by activation binarization.
In this paper, we propose a multi-branch bit-aware binary activation network with spatial masking to balance the distribution of binarized weights and activations. Our idea motivates from the fixedpoint binary representation.
For example, the floating point number 0 . 7865 can be expressed as 0 .
7865 “ 2 ´ 1 ` 2 ´ 3 ` 2 ´ 4 “ 0 . 1011 2 . We select a subset of the bits of the binary fixed-point representation to approximate the real-valued activations using matrix multiplication approximation algorithms. In the end, the activations can be approximated by an additive multi-branch network where each branch represents a bit in the fixed-point representation.
Different from ABCNet (Lin et al.
(2017)), where the bases and the binary representations are learnt without differentiating the role of different branches
, our method assigns different power-of-two values to distinguish the branches, and thus there is no redundant information among different branches.
To further reduce the activation memory, we propose to add a spatial masking to the channels of the activations.
Finally, experiments on the large-scale dataset demonstrate that our proposed methods maintain the accuracy without an increase in both computational budget and memory usage.
Our contributions are three-fold:
• We propose the Masked Additive Binary Activation Network (MABA-Net) to moderately increase the size of binarized activations, by approximating real-valued activations as the sum of masked binary activations weighted by power-of-2 factors.
Compared to previous approaches, our method offers significantly reduced approximation error and fewer bitwidths for binarized activations simultaneously, resulting in smaller accuracy gap to fullprecision counterparts.
• By integrating our proposed methods with various network architectures, the distributions of activations to weights becomes balanced, leading to significant savings of total memory.
• Extensive evaluations on ImageNet demonstrate that MABA-Net obtains comparable or better accuracy than state-of-the-art without an increase in computational cost, while with up to 3 ˆ reduction in memory footprint
R ELATED W ORK 2. B INARY N EURAL N ETWORKS
Binary Neural Networks, as the extreme case of neural network quantization, suffers from significant degradation.
Many researchers have explored how to address the gap.
Network architecture One direction is to explore the network architecture to improve the BNN performance.
For example, ReActNet (Liu et al.
Bi-Real-Net (Liu et al.
(2018)), Realto-Bin (Martinez et al.
and MeliusNet (Bethge et al.
through modified classical network (e.g., MobileNet, ResNet) to improve the accuracy.
EBN (Bulat et al.
, Phan et al.
(2020) and Kim et al.
(2020a) introduce to search the optimal architecture for BNN to improve the performance.
BENN( Zhu et al.
, GroupNet (Zhuang et al.
and ABCNet (Lin et al.
(2017)) propose to expand the block structure to improve the performance.
Our methods propose to widen the activation but not the weights through designed additive bit-wise branches to improve the performance.
To be fair, we conduct extensive comparison with different network settings.
Training strategy and tricks Another direction is investigating how different training scheme and tricks affect the performance of BNNs.
For instance, ReActNet, Real-to-Bin, EBN and BCNN (Redfern et al.
(2021)) adopt a two-stage training strategy.
XNORNet (Rastegari et al.
, XNORNet++ (Bulat & Tzimiropoulos (2019)) and EBN design different scaling factors to avoid inconsistency between the binarized representations and the full-precision data.
Liu et al.
(2021), Ajanthan et al.
(2021) investigate different optimizers to improve the performance.
For the activation function,
Bi-Real-Net replace ReLU with Htanh while (Tang et al.
(2017)) instead use PReLU for BNNs.
Our method investigates the training strategy in the experiments.
Gradient approximation STE roughly sets the gradients of the binarized value to the full-precision data as 1, which brings information loss in the weight update of the back-propagation, and thus many research efforts design various surrogate gradients to address the issue such as BinaryDuo (Kim et al.
(2020b))
, BiRealNet (Liu et al.
(2018)), and
IR-Net (Qin et al.
For example, Liu et al.
(2022) proposes a generalized straight-through estimator (G-STE) for intractable backward derivative calculation to output uniform quantized values.
Instead, our method alleviates the STE issue by replacing the sign function as truncation.
Power-of-Two Quantization Many researchers have explored how to conduct Power-Of-Two quantization as it can convert the multiplication to hardware efficient bit-shift.
For example, Gudovskiy & Rigazio (2017) quantizes the weights as a sum of power-of-two values.
While APoT Li et al.
(2020), based on the distributions of weights and activations, constrains all quantization levels as the sum of Powers-of-Two terms.
Yao et al.
(2022) dynamically adjusts the Power-of-Two scales of the whole network instead of statically determining them layer by layer in the post-training quantization.
DeepShift Elhoushi et al.
(2021) proposes to replace multiplications in convolutional layers and fully connected layers with bitwise shift and sign flipping by quantizing the weights during both training and inference.
Instead, to binarize the network, our method represents the real value into power-of-two bases with binary representations.
2. N ETWORK PRUNING
Pruning Mask Mask has been a handy tool in learning the importance of sub-structure of neural network during pruning.
In fact, it enables researchers to explore the effectiveness of pruning at different granularity (He et al.
(2017),Liu et al.
(2017),Lemaire et al.
(2019),Chin et al.
(2020),Kusupati et al.
Masks at filter, channel or stripe level enables the network to learn a structure that is hardware friendly, whereas masks at weight level may produce a sparser network, but such unstructured pruning network might require dedicated hardware for inference.
Pruning in BNN On the contrary, BNN has not received many attentions in the field of network pruning because BNN is already a compact model.
Furthermore, it is not the top priority for BNN to further compress the model and reduce compute when there is a large performance gap remained as compared to its real-value counterpart.
However, as BNN’s performance keeps improving with the help of recently devised techniques, it makes sense to consider the question whether we can maintain BNN’s performance with less memory and computation cost.
P RELIMINARY 3. B INARIZED A PPROXIMATION
Let W r and X r represent the real-valued weights and input activations for a layer.
Binary network attempts to approximate W r and X r with the binarized weights W b and activations X b (Rastegari et al.
; Bulat & Tzimiropoulos (2019); Xu & Cheung (2019))
as below.
[Equation]
where sign p¨q denotes the sign function, α and β are learnable scaling factors that match the binarized values to the same scale as the real values.
Compared to the full-precision network, the binary network not only reduces the memory footprint by 32 ˆ , but also replaces the multiply-accumulate (MAC) convolutional operation between W b and X b with the XNOR-popcount operation as below:
[Equation]
where w b and x b are the flattened vector representations of W b and X b , and ¨ here denotes the dot product.
The replacement reduces the number of operations by 43 ˆ (Fromm et al.
3. B ACK - PROPAGATION WITH BINARIZATION
In general, the quantization function has the gradients of zeros almost everywhere and thus it is impossible to directly apply the back-propagation to train the network.
To overcome this issue, a classical way is using the empirical straight-through estimator (Bengio et al.
(2013b)) to re-define the gradients.
Namely, the derivative of the binarized value z b to the real value z r is an identity mapping, namely B z b B z r « 1 .
Accordingly, the derivative of the loss L to z r can be approximated by
[Equation]
M ETHOD 4. B INARIZE W EIGHTS
Following Liu et al.
(2018), we binarize the weights of each convolutional layer with a magnitudebased scaling factor as below: [Equation]
[Equation]
where | W r | denotes the number of entries in W r , the updating of W r is computed as follows:
[Equation] [Equation]
where B W tb B W tr “ || W [Equation] ||
[Equation]
| [Equation] Because the derivative of sign p¨q function is almost zero everywhere and thus preventing the flow of gradients, conventional methods approximate B sign p W tr q B W tr with the identity mapping, namely B sign p W tr q B W tr “ 1 .
As empirically not cancelling the gradients when weights is too large significantly worsen the performance, most methods include a clipping to cancel the gradients when weights is too large.
Therefore, we define an indicator function 1 | W tr |ă 1 to set the gradients of the weights that are outside of the range to 0.
Consequently, the final form of B W tb B W tr can be approximated by || W t r || 1 | W tr | ¨ 1 | W tr |ă 1 .
In many hardware, the floating-point is represented by the IEEE-754 standard (Kahan (1996)) for arithmetic. Our idea motivates from the fixed-point representation (Gupta et al. (2015)), which has a fixed number of bits for the integral and fractional parts.
For example, 0 .
7865 “ 2 ´ 1 ` 2 ´ 3 ` 2 ´ 4 , is 0 .
Mathematically, the activation X r can be represented as the matrix multiplication between a binary matrix X B P t 0 , 1 u N ˆ 31 and a basis vector D “ r 2 ´ 1 , 2 ´ 2 , 2 ´ 3 , ..., 2 ´ 31 s T .
We can approximate X r by approximating the matrix multiplication of X B D as below:
[Equation]
where X b P t 0 , 1 u N ˆ S is the subset of X B , and the corresponding basis vector Q and S denotes the bit-width.
We use the importance sampling approach (Mahoney et al. (2011); Drineas et al.
(2006)) to compute the optimal X b and Q by minimizing the expectation of MSE (mean squre error) as E p|| X B D ´ X b Q || 2 F q where ||¨|| F denotes the Frobenius norm.
We revise the importance sampling algorithm by selecting the index i t with top S probabilities as in Algorithm 1. The detailed derivation is shown in Appendix A.1.
In the end, the real value X r is represented as:
[Equation]
We can make some simplification to the expression above to not only construct a more succinct expression, but also make the approximation more hardware friendly by removing unnecessary floating point operations.
Firstly, we can remove 1 S from the above expression because it is a constant and can be infused into the weights.
Secondly, we ask the question if we can remove 1 p it , which is a floating point number, from the expression. To answer this question, we resolve to empirical evidence; we conducted experiments that are with 1 p it and without 1 p it . Our experiment results show that 1 p it does not affect model’s performance when S is kept small, which is the case in our implementation with S “ 2 or 3 . Therefore, we approximate the X r as following:
[Equation]
As X r is in fixed-point representation, we can obtain X b by bit-shifting shown in Appendix A.1. Furthermore, we can obtain the gradients B X p it q b B X
[Equation]
As the network usually has a ReLU as the activation function, we only consider binarizing the non-negative activations.
Following Hubara et al.
(2016), we set a bounded range for the realvalued activations X r as X r “ clip p X r , r 0 , 1 sq to cancel the gradient when X r is too large, which significantly worsens the performance.
4. S PATIAL M ASKING
The increase in bit-width for binarized activations may introduce information redundancy, meanwhile linearly increases the activation size and the number of binary operations (BOPS).
To reduce the redundancy, save the memory and the computation cost, a trainable binary mask variable
M P R W ˆ H is applied to each spatial position p w, h q of the binarized activations X b P R C ˆ W ˆ H where C , W and H denote the number of channels, the width and height of input activations.
If
M w,h “ 0 , the whole channel of the activation features at location p w, h q are totally removed; otherwise, the features are kept in memory and involved in the computation process.
The higher sparsity the binary masks, the smaller the activation size is.
To reduce the activation size, we introduce a BOPS-aware sparsity loss on the binary mask variables
M to explicitly enforce it moving to 0 during training.
[Equation] [Equation]
where M l denotes the binary mask variables of the binazied activations at the l th layer.
BOPS l denotes the number of binary operations at the l th layer.
The entry value of M is either 0 or 1 and thus the L1-norm of M corresponding to the left activations.
For simplicity, we apply STE (Bengio et al.
(2013a))
on the training of these mask variables.
To generate the binarized entry of M , we assign M w,h “
if M ą 0 else 0.
In the backward pass, the gradients are approximated by the identity mapping.
Note that there is memory overhead introduced by the binary mask variables themselves, with W ˚ H ˚ S parameters per layer ( S ď 3 in our experiments), which is negligible compared to size of weights and activations. [Figure]
O BJECTIVE FUNCTION
The overall framework is shown in Fig.
Structurally, it is a multi-branch network where each branch contributes to a specific impact, different from ABCNet without differentiating the role of different branches.
To reduce the model parameters, we apply group convolutions to the 3 ˆconvolutional layer in the residual block.
The final objective function is as below:
[Equation]
where L E is the cross-entropy loss, and L S is the sparsity regularization term to enforce the MABANet to meet the pre-defined computational budget.
λ is to control the impact of the sparsity loss.
4. C OMPUTATION COST
In this paper, the binarization of activations is unipolar quantization where bits with value 0 represent 0 and bits with value 1 represent 1; while the binarization of weights is bipolar quantization where bits with value 0 represent -1 and bits with value 1 represent 1 (Xu & Cheung (2019); Fromm et al.
Hence, the convolution of X b and W b can be computed as follows.
[Equation]
where size p W b q denotes the number of elements in W b and thus is a constant number. As compared to Eq. (3), the number of binary operations does not increase.
E XPERIMENTS
We perform ablation study of our method on ImageNet ILSVRC 2012 in Section 5.1, then in Section 5.2 we compare our method with state-of-the-art on ImageNet for image classification.
It is worth noting that we do not use sophisticated training techniques such as knowledge distillation and gradient approximation for binary weights.
For binary network training on ImageNet, we set the initial learning rate as 0.001, with a cosine learning rate scheduler.
We train the models forepochs with the Adam optimizer, batch size 256, and weight decay 5e-6.
For our proposed methods, the binary mask variables are updated together with weights without warm up.
5. A BLATION S TUDY 5.1. WHICH TRAINING STRATEGY IS BETTER
Binarization of real-valued weights/activations both introduce approximation error to the information flow, which in turn results in accuracy degradation.
Fig.
3 shows the validation accuracy curves when training binary EBN and RealToBinary network for the ImageNet under three settings: (1)
Train from scratch for 150 epochs by binarizing weights and activations simultaneously; (2) BANBWN: two-stage training by binarizing the activations for the first 75 epochs then binarizing the weights for the second 75 epochs; (3) BWN-BAN: two-stage training by binarizing weights for the first 75 epochs then binarizing activations for the second 75 epochs.
As one can see, BWN-BAN
[Figure] [Figure]
[Table]
caused a significant accuracy drop when binarizing the activations at the 76 th training epoch after the 1 st stage of binarizing weights and the drop is much greater than the case of weight binarization in BAN-BWN.
Though the accuracy of BWN-BAN restores gradually, we observe that it performs significantly worse than the other two cases, in which BAN-BWN is slightly better than Train from scratch.
This may explain why the state-of-the-art methods ReActNet (Liu et al.
(2020)) and RealToBinary (Martinez et al.
(2020)) adopted the BAN-BWN training strategy.
It might be due to that activations are more sensitive to binarization as compared to the weights, and searching the optimal parameters from quantizing activations is easier.
5.1. C OMPARISON WITH BENCHMARK QUANTIZATION METHODS
To compare with various benchmark quantization approaches, we conduct experiments on ImageNet with the network RealToBinary by setting activation as 2 bits for 3*3 convolution with the number of group convolutions as 2.
Tab. 1 shows the comparison results.
APoT, DoRefaNet and ours have fixed base and binary representation.
Although APoT requires 2-bit representation, it indeed requires 3 bits when it is expressed in terms of binary representation. The detailed proof is shown in Appendix A.2.
ABCNet and LQNets learn the bases by a data-driven way.
Our experiments shows that ABCNet learns the same bases for different branches as it does not differentiate the branches, leading to a poor performance. A detailed analysis of a poor performance of ABCNet is shown in Appendix A.3. LQNets has quite different bases for different layers, shown in Fig.
4f. DoRefaNet has pre-defined bases and binary representation.
Our method has learnt the base as p 2 ´ 1 , 2 ´ 2 q , identified as uniform quantization.
Furthermore, our proposed methods show the best result with 2-bit binary representation.
5.1. H OW IS THE MSE
Fig.
4 shows the mean square error (MSE) between original unquantized activation distribution and the corresponding quantized values of various methods under the RealToBinary network setting.
The MSE is measured on 1 st batch of each epoch for the training dataset at the 1 st stage BAN, i.e., quantizing activation with full-precision weights, of the training stage across all the layers.
It shows that the MSE of the first several layers is significant among different methods.
It might be due to that the real value first passes through a clip function and then do binarization.
The real value which is outside of the clipping range without quantization will have larger MSE.
Due to the gradient vanishing issue, for the first several layers, it might be difficult to use the data-driven way to enforce the data locating in the clipping range.
In particular, LQNets has much larger MSE as compared to other methods for the first several layers, which might indicate the importance of clipping function.
Furthermore, the learnt bases of LQNets, like the MSE distribution, are quite
[Figure]
large for the first several layers, shown in Fig.
4f.
Furthermore, we observe that the MSE increases as the epoch increases for some layers.
It might indicate that the binarization is diverging with the epoch increases, but due to the clipping, the divergence is not infinite.
5.1. WHAT IS THE EFFECT OF THE BIT - WIDTH AND GROUPED CONVOLUTION
To investigate the effect of the bit-width and grouped convolution to the accuracy and computational cost, we conduct experiments on the network EBN (Bulat et al.
We evaluate the effect of bitwidths for binarized activations N and number of groups for convolutions G using our MABA-Net, with binary mask disabled.
We conduct experiments by varying N and G on the backbone network architecture EBN (Bulat et al.
(2021)), with configuration t 1 , 2 , 6 , 2 ut 2 ut 4 , 8 , 8 , 16 u .
As shown in
Fig.
5, we set the number of bits for activations of 3 ˆ 3 convolutions and 1 ˆ 1 convolutions as t 1 , 2 , 3 u .
For the default number of groups for convolution layers in each block (i.e., t 4 , 8 , 8 , 16 u ), we further increase the numbers by 2 ˆ , 4 ˆ or 8 ˆ .
For instance, the number of groups is increased to t 16 , 32 , 32 , 64 u for the case of 4 ˆ .
As one can see, there is a clear trade-off between network accuracy and the total number of binary operations (BOPS) in a model.
Among different group settings, we find that the bit-width configuration p 3 , 2 q is the optimal trade-off between accuracy and BOPS.
Though the total number of 1 ˆ 1 convolutions is much less than the 3 ˆ 3 convolutions, adding 1 more bit to 1 ˆ 1 convolutions boosts the accuracy while at the expense of significant increase in BOPS, mainly due to reason that the 1 ˆ 1 convolutions are not grouped convolutions.
E.g., with group size increased by 4 ˆ , adding 1 bit for the 1 ˆ 1 convolutions from (3, 2) to (3, 3) introduces more BOPS than adding 1 bit for the 3 ˆ 3 convolutions from (2, 3) to (3, 3).
5. C OMPARISON WITH THE STATE - OF - THE - ART
To make fair comparison, we compare our proposed methods with state-of-the-art in two network settings: 1) ResNet18: the network architecture is ResNet18 and 2)
ResNet-FP: the ResNet variants.
We only change the way of binarizing the weights and activations in corresponding network settings.
For the 1 st network setting, we compare with BNN (Courbariaux et al.
, XNORNet (Rastegari et al.
, CCNN (Xu & Cheung (2019)), Rethink.
BNN (Helwegen et al.
, XNORNET++ (Bulat & Tzimiropoulos (2019)), IR-Net (Qin et al.
For the 2 nd network setting, we compare with different networks: Bi-Real Net (Liu et al.
(2018)), RealToBinary (Martinez et al.
EBN (Bulat et al.
ReActNet-A (Liu et al.
In particular, for Bi-Real Net,
RealToBinary and ReActNet-A, we simply set the number of group convolutions for 3*3 convolutions as 2 and the bit-width for the activation as 2.
While for EBN, we conduct a fine-grained search, shown in Section 5.1.4. [Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure] a cc ( % ) (3, 2) (3, 3) (3, 3) (3, 2) (3, 1) (2, 2) (2, 3) (3, 1) (3, 2) (2, 3) 8 × 4 ××
[Table]
As shown in Tab.
[Table]
for different network settings
[Table]
our method improves on top of the currently best performing methods with a large margin with less memory cost.
The sizes of weights and activations are balanced, shown in Fig.
[Table]
Although our method does increase the activation size
[Table]
by 2 ˆ , the run-time memory of both weights and activations largely reduces.
C ONCLUSION AND F UTURE W ORK
In this paper
[Table]
we propose MABA-Net to approximate real-valued activations by accumulating a limited set of binary activations weighted by power-of-2 factors.
A trainable binary mask variable is attached to each spatial position in each set of the binary activations, as an indicator to keep or drop activation features at the spatial position, with negligible overhead for storage of the mask variables.
By integrating MABA-Net with a deeper and wider grouped convolutional network, we address the problem of unbalanced distribution of binarized activations/weights, which in turn improves accuracy without incurring an increase in both computational budget and memory usage.
In the future, we will explore how to binarize the first several layers to improve the performance and also explore other computational and memory saving techniques such as filter pruning, and incorporate advanced training techniques such as knowledge distillation, to further improve the performance. A A PPENDIX A. B INARIZE A CTIVATIONS
Importance sampling Different from the standard importance sampling approaches of approximating the matrix multiplication, which randomly select the index i t based on the probability distribution of p i t , we select the index i t with the top probabilities to make it deterministic.
Besides, the approximation algorithm sets the t th column of X b as X p t q b “ X p it q B { a Sp it and the t th row of Q as Q p t q “ D p it q { ?
Sp it .
As in the matrix multiplication of X b and Q , only the elemnts of t th column of X b multiplies with the elements of the t th row of Q , we merge the numerator of X b to Q and get X p t q b “ X p i t q B , Q p t q “ D p it q { Sp it .
Binary matrix computation When representing the real-valued X r in fixed-point binary presentation, the binary matrix X b can be computed as below:
[Equation]
Through shifting X r by i t bits and bit-wise AND operation with 1, the i tth most significant bit is obtained.
For example, 0.7865 represented in binary format is 0 .
If we left shifting 0 .
10 1 1by 3 bits, and bit-wise AND with 1, we will get the 3 rd most significant bit.
A. AP O T VS O URS
APoT defines a quantized value set U and then selects the quantized value that is closest to the real value x .
When only 2 bits are used, APoT has quantization levels: U “ t 0 , 2 ´ 1 , 2 ´ 2 , 2 ´ 3 u .
APoT then selects the base that results in smallest quantization error.
The procedure can be succinctly expressed with following formula:
[Equation]
Basis
[Equation]
fi ffiffiffiffiffifl
[Equation]
binary representation (14)
Where x is the real-value input of APoT quantization, ˆ x is the quantized output, and 1 p¨q is an indicator function to select the base that has the smallest quantization error.
As illustrated in Fig.
6, APoT requires 3 binary activations bases, which correspond to the bases t 2 ´ 1 , 2 ´ 2 , 2 ´ 3 u .
Although APoT only uses 2-bit representation in the case, it indeed requires 3 bits when it is expressed in terms of binary representation.
This is due to the intrinsic nature of APoT quantization which is not tailored for binary quantization.
In contrast, with tailored
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
binarization technique for activation, our method achieves comparable performance to APoT with only 2 binary branches, resulting in 1 bit reduction, as shown in Table 1.
A. ABCN ET VS O URS
Given a real-valued input x , ABCNet compute the binarized value x ib of the i branch as follows:
[Equation]
(15) where v i is the shifting scalar for the i th branch and h v i p x q “ clip p x ` v i , p 0 , 1 qq .
Suppose the scalar parameters v i is sorted, namely, v i ă v i ` 1 .
As shown in Fig.
7, given the same input x , x ib and x i ` 1 b are different only when 0 .
5 ´ v i ` 1 ď x ă 0 .
5 ´ v i in Eq.
If v i and v i ` 1 are very close, branches i and i ` 1 will have a lot of same values and thus are highly correlated.
[Equation]
when
[Equation]
ě
[Equation]
when
[Equation]
ă
[Equation]
ABCNet does not specify the gradient for the shifting parameter v i ; however, it can be deduced as follow: B L B v i “ B L B X ib ˆ
1 0 ď X r
´ v
i ď 1 where L denotes the loss function, X ib denotes the binarized activations of the i th branch and X r denotes the real-valued activations.
For simplification, we assume branch i and branch j has the same scaling factor and then B L B X ib =
B L B X jb .
Therefore, if v i and v j are initialised close to each other, their gradients, B L B v i and B L B v j , will be close.
Thus if v i and v j are kept close, based on Eq.
(15), X ib and X jb are also close.
This effect extends to the learning of the scaling factor β
i , too.
The gradient of β i is calculated as follows:
B L B β i “ B L B
A ˆ B A B β
i “ B L B A
ˆ X ib , where A “ ř i β i
˚ X i b .
If X i b and X j b are close to each other, then the gradients of β i and β j are also close.
So the learnt β i and β j would be close.
In the end, we can see that the performance of
ABCNet is really affected by the initialization of the shifting parameter and the scaling factor.
A. S PATIAL MASK VISUALIZATION
Our proposed spatial masking belongs to the category of structured pruning as it prunes the whole channel of the activations when the mask equals to 0.
Fig.
8 shows the pruning ratio of the learnt spatial masking and corresponding BOPS of our proposed methods across different layers on EBN network with the optimal setting of 3 branches for the 3 ˆ 3 convolutions, 2 branches for 2 ˆconvolutions and 4 ˆ grouped convolutions for 3 ˆ 3 convolutions.
We make few observations from the plot.
Firstly, the activation of the first and last several network layers are pruned more than other layers.
Secondly, it shows that the activation of the layer with larger computational cost (BOPS) are pruned more than other layers, which is consistent with our motivation of computation cost based pruning.
Thirdly, we observe that the branch with smaller base has larger pruning ratio.
It might be due to that the branch of the larger base plays a more important role in approximating the real values.
[Table]
FilterSkeleton
[Table]
[Table]
FilterSkeleton
[Table]
[Table]
.FilterSkeleton
[Table]
.FilterSkeleton
[Table]
module.layer3.3.conv1.FilterSkeleton  1 0.
module.layer3.3.conv2.FilterSkeleton  1 0.
module.layer3.3.conv3.FilterSkeleton  1 0.
module.layer3.4.conv1.FilterSkeleton  3 0.
module.layer3.4.conv2.FilterSkeleton  1 0. module.layer3.4.conv3.FilterSkeleton  11 0.
module.layer3.5.conv1.FilterSkeleton  2 0.
module.layer3.5.conv2.FilterSkeleton  1 0.
module.layer3.5.conv3.FilterSkeleton  6 0.
module.layer4.0.conv1.FilterSkeleton    module.layer4.0.conv2.FilterSkeleton  5 0.
module.layer4.0.conv3.FilterSkeleton    module.layer4.1.conv1.FilterSkeleton  9 0. module.layer4.1.conv2.FilterSkeleton  3 0.
module.layer4.1.conv3.FilterSkeleton  4 0.
0.00E+ 2.00E+ 4.00E+
6.00E+ 8.00E+ 1.00E+ 1.20E+  0.
P r un i n g p erce n t ag e
Layer
[Figure]