Anonymous authors
Paper under double-blind review
A BSTRACT
Binary Weight Networks (BWNs) have signiﬁcantly lower computational and memory costs compared to their full-precision counterparts.
To address the nondifferentiable issue of BWNs, existing methods usually use the Straight-ThroughEstimator (STE).
In the optimization, they learn optimal binary weight outputs represented as a combination of scaling factors and weight signs to approximate 32-bit ﬂoating-point weight values, usually with a layer-wise quantization scheme.
In this paper, we begin with an empirical study of training BWNs with STE under the settings of using common techniques and tricks.
We show that in the context of using batch normalization after convolutional layers, adapting scaling factors with either hand-crafted or learnable methods brings marginal or no accuracy gain to ﬁnal model, while the change of weight signs is crucial in the training of BWNs.
Furthermore, we observe two astonishing training phenomena.
Firstly, the training of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and ﬁxed at the early training stage, which is akin to recent ﬁndings on the lottery ticket hypothesis for efﬁcient learning of sparse neural networks.
Secondly, we ﬁnd binary kernels in the convolutional layers of ﬁnal models tend to be centered on a limited number of the most frequent binary kernels, showing binary weight networks may has the potential to be further compressed, which breaks the common wisdom that representing each weight with a single bit puts the quantization to the extreme compression.
To testify this hypothesis, we additionally propose a binary kernel quantization method, and we call resulting models Quantized Binary-Kernel Networks (QBNs).
We hope these new experimental observations would shed new design insights to improve the training and broaden the usages of BWNs.
I NTRODUCTION
Convolutional Neural Networks (CNNs) have achieved great success in many computer vision tasks such as image classiﬁcation (Krizhevsky et al., 2012), object detection (Girshick et al., 2014) and semantic segmentation (Long et al., 2015).
However, modern CNNs usually have large number of parameters, posing heavy costs on memory and computation.
To ease their deployment in resourceconstrained environments, different types of neural network compression and acceleration techniques have been proposed in recent years, such as network pruning (Han et al., 2015;
Li et al., 2017), network quantization (Hubara et al., 2016; Rastegari et al., 2016;
Zhou et al., 2016), knowledge distillation (Ba & Caruana, 2014; Hinton et al., 2015), efﬁcient CNN architecture engineering and searching (Howard et al., 2017; Zhang et al., 2018b; Zoph & Le, 2017).
Comparatively, network quantization is more commercially attractive as it can not only beneﬁt specialized hardware accelerator designs (Sze et al., 2017), but also can be readily combined with other techniques to get further improved compression and acceleration performance (Mishra &
Marr, 2018; Han et al., 2016; Zhou et al., 2017).
Quantization methods aim to approximate fullprecision (32-bit ﬂoating-point) neural networks with low-precision (low-bit) ones.
In particular, the extremely quantized models called Binarized Neural Networks (BNNs)
(Courbariaux et al., 2015; 2016; Rastegari et al., 2016) which force the weights or even weights and activations to have 1-bit values ( +1 and − 1 ), bringing 32 × reduction in model size and making costly 32-bit ﬂoating-point multiplications can be replaced by much cheaper binary bit-wise operations.
Because of this, how to train accurate BNNs either in a post-training manner or in a training from scratch manner has attracted increasing attention.
However, training BNNs poses a non-differentiable issue as converting full-precision weights into binary values leads to zero gradients.
To combat this issue, most existing methods use the Straight-Through-Estimator (STE).
Although there are few attempts (Achterhold et al., 2018;
Chen et al., 2019; Bai et al., 2019;
Hou et al., 2017) to learn BNNs without STE by using proximal gradient methods or meta-learning methods, they suffer from worse accuracy and heavier parameter tuning compared to STE based methods.
In STE based methods, full-precision weights are retained during training, and the gradients w.r.t.
them and their binarized ones are assumed to be the same.
In the forward pass of the training, the full-precision weights of the currently learnt model are quantized to binary values for predication loss calculation.
In the backward pass, the gradients w.r.t.
full-precision weights instead of binary ones are used for model update.
To compensating for drastic information loss and training more accurate BNNs, most state of the art STE based methods follow the formulation of (Rastegari et al., 2016) in which the binary weights are represented as a combination of scaling factors and weight signs to approximate 32-bit ﬂoating-point weight values layer-by-layer, yet also present a lot of modiﬁcations.
These modiﬁcations include but are not limited to expanding binary weights to have multiple binary bases (Lin et al., 2017; Guo et al., 2017), replacing hand-crafted scaling factors with learnable ones (Zhang et al., 2018a), making an ensemble of multiple binary models (Zhu et al., 2019), searching high-performance binary network architectures (Kim et al., 2020), and designing improved regularization objectives, optimizers and activation functions (Cai et al., 2017; Liu et al., 2018;
Helwegen et al., 2019; Martinez et al., 2020).
There are also a few works, trying to make a better understanding of the training of BNNs with STE.
In (Alizadeh et al., 2019), the authors evaluate some of the widely used tricks, showing that adapting learning rate with a second-moment optimizer is crucial to train BNNs with STE based methods while other tricks such as weights and gradients clipping are less important.
Bethge et al.
(2019) shows the commonly used techniques such as hand-crafted scaling factors and custom gradients are also not crucial.
Sajad et al.
(2019) demonstrates learnable scaling factors combined into a modiﬁed sign function can enhance the accuracy of BNNs.
Anderson & Berg (2018) makes an interpretation of why binary models can approximate their full-precision references in terms of high-dimensional geometry.
Galloway et al.
(2018) validates that BNNs have surprisingly improved robustness against some adversarial attacks compared to their full-precision counterparts.
In this paper, we revisit the training of BNNs, particularly Binary Weight Networks (BWNs) with STE, but in a new perspective, exploring structural weight behaviors in training BWNs.
Our main contributions are summarized as follows:
• We use two popular methods (Rastegari et al., 2016) and (Zhang et al., 2018a) for an empirical study, showing both hand-crafted and learnable scaling factors are not that important, while the change of weight signs plays the key role in the training of BWNs, under the settings of using common techniques and tricks.
• More importantly, we observe two astonishing training phenomena: (1) the training of BWNs demonstrates the process of seeking primary binary sub-networks whose weight signs are determined and ﬁxed at the early training stage, which is akin to recent ﬁndings of the lottery ticket hypothesis (Frankle & Carbin, 2019) for training sparse neural networks; (2) binary kernels in the convolutional layers (Conv layers) of ﬁnal BWNs tend to be centered on a limited number of binary kernels, showing binary weight networks may has the potential to be further compressed.
This breaks the common understanding that representing each weight with a single bit puts the quantization to the extreme compression.
• We propose a binary kernel quantization method to compress BWNs, bringing a new type of BWNs called Quantized Binary-Kernel Networks (QBNs).
A N E MPIRICAL S TUDY ON U NDERSTANDING BWN S ’ T RAINING
In this section we will brieﬂy describe BWNs we use in experiments, implementation details, scaling factors in BWNs, full-precision weight norm, weight sign, and sub-networks in BWNs.
2. D IFFERENT B INARY W EIGHT N ETWORKS
BWNs generally represents those networks with binary weights, and there are several different
BWNs existing.
Overall they use αB to replace full-precision weight W , where B = sign ( W ) and α is proposed to minimize ||
αB − W
|| in an either learnable or calculated way.
In following experiments, we use the one implemented in XNor-Net (Rastegari et al., 2016) and denote it as XNor-BWN, and the one implemented in LQ-Net (Zhang et al., 2018a) and denote it as
LQ-BWN which is 1-bit weight 32-bit activation version of LQ-Net.
Other popular BWN methods like DoReFa-Net and BinaryConnect are similar to these two methods.
Both XNor-BWN and
LQ-BWN use STE framework, and XNor-BWN uses hand-crafted calculated scaling factors, and
LQ-BWN uses learnable scaling factors.
2. I MPLEMENTATION D ETAILS AND N OTATION
Quantization: We directly use open source codes of BWN released by authors, including XNorBWN 1 and LQ-BWN 2 .
Dataset and Network Structure: CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Russakovsky et al.) are used in our experiments.
We use VGG-7 (Simonyan & Zisserman, 2015) and
ResNet-20 (He et al., 2016) on CIFAR-10, and ResNet18 on ImageNet.
The strcuture is the same as original ones.
Hyper-parameters: We use the same training parameters on each network.
The network is trained for 200 epochs.
The learning rate is set initially to 0.02 and divided by 10 at 80 and 160 epochs.
For random crop, we ﬁrst use zero pad to resize the image into 40 × 40 , and random crop into 32 × 32 .
For BWN trained on ImageNet, each is trained for 100 epochs.
The initial learning rate is 0.1 and decays 0.1 at 30, 60, 90 epoch.
The image is rescaled into 256 × 256 and then randomly cropped into 224 × 224 .
No additional data augmentations are used.
For all networks, weight decay is applied to all Conv layers set to 4 × 10 − 5 .
Notations: In ﬁgures and tables, we will use the following abbreviations for clearer expression.
BN: BatchNormalization, LR: Learning Rate.
WD: Weight Decay.
SF: Scaling Factors.
FP: Fullprecision.
VGG-7 XNor-BWN: a VGG-7 network using the binarization algorithm of XNor-BWN.
ResNet-20 Baseline: a full-precision ResNet-20 only using data augmentation and weight decay
without any additional tricks.
Other network structures with certain methods are similar to this.
Large weights, large magnitude weights, and weights with larger norm have the same meaning to indicate those weights having relatively large absolute values.
2. S CALING F ACTORS
According to previous methods, scaling factors are one essential element in obtaining BWNs.
However, according to our experiments and analysis, we ﬁnd scaling factors are not so important in training BWNs, and they can somehow be ignored without the drop in performance.
Here we list four reasons to explain why scaling factors are unimportant.
A simple proof: BN is a common practice to be used in training BWNs.
It contains two operations,
Normalization and Afﬁne as shown in Equation1.
γ and β are the afﬁne parameters used in BN.
(cid:15) =
5 e − 4 is used in PyTorch to avoid the error of dividing zero.
We use a simple proof to demonstrate that BN can absorb scaling factors as shown in Equation2.
This is correct during training when one scaling factor is applied to each output channel under the Conv-BN-ReLU structure.
[Equation]
2 Weight Value 0.0 2.
P r o b a b ili t y
D e n s i t y
Conv1Conv3Conv
XNor-BWN VGG 0.
0.2 Weight Value
P r o b a b ili t y
D e n s i t y
Conv1Conv3Conv
LQ-BWN VGG    4 Weight Value
P r o b a b ili t y
D e n s i t y
Conv0G3B2Conv
XNor-BWN R1.0 0.5 0.0 0.5 1.0 1.5 Weight Value
P r o b a b ili t y
D e n s i t y
Conv0G3B2Conv
LQ-BWN R-
Experimental Results:
In the second aspect, we directly go to experimental results.
As shown Table.2 in Appendix.
B, we train different networks with and without scaling factors.
The test accuracy on Cifar-10 and validation accuracy on ImageNet do not show a large difference between the two methods.
Later we ﬁx scaling factors of all layers to a certain value and magnify their learning rate according to the ﬁxed scaling factors’ magnitude.
The performance does not change when ﬁxing scaling factors.
Thus, we conclude with proper learning rate, scaling factors are not essential to train
BWNs.
Compare learnable SF and γ in BN: LQ-BWN uses channel-wise scaling factors.
From the experiments in Appendix.
C, we ﬁnd that these channel-wise scaling factors having a high correlation with γ in the BN after corresponding binary Conv.
This ﬁnding indicates that BN’s γ can replace channel-wise SF to some extent.
Quantization Error Curve: Another purpose using scaling factors is to reduce the quantization error between full-precision weights and binary weights according to a BNN survey (Qin et al., 2020).
By using experiments in Appendix.
D we prove that the quantization error is not actually reduced by scaling factors, but weight decay helps on this reduction.
2. W EIGHT N ORM , W EIGHT S IGN , AND S UB -N ETWORKS IN BWN S
We already analyse one essential element, scaling factors, in the previous section, and another essential element of BWNs are weights’ signs.
In deterministic binarization methods, full-precision weights’ signs decide their binary weights’ signs using a sign () function.
In this section, we will discuss the relationship between weight norm, weight sign and how to ﬁnd primary binary subnetworks in BWNs.
Weight Histogram:
We visualize the full-precision weight distribution in different layers of different networks as shown in Figure.1.
Different from a bi-modal distribution, it shows a centered distribution around 0.
This again proves that the actual distance or so-called quantization error is very large.
And there are many weights close to zero behaving very unstable, which will change their signs with little perturbations.
More experiments and visualizations are in Appendix.
E.
Flipping Weights’ Signs: We ﬂip the weights’ signs during the inference section according to weights’ full-precision norm as shown in Figure.12 of Appendix.
G. We ﬂip those weights with the largest norm and the smallest norm in two experiments.
It shows that even the weights have the same norm after binarization, and the changed norm is the same for the same ﬂipping percentage, there is still a very large gap between the two results.
Flipping those weights with large full-precision magnitude will cause a signiﬁcant performance drop compared to those weights close to zero.
This reveals that weights are different where some with small norm can tolerate sign ﬂipping, and those with large norm cannot suffer from changing signs, even though both two kinds of weights have the same norm after binarization.
Tracing Large Weights From the last experiment, we conduct that weights with large norm are vulnerable and important during inference, however, the function of them during training remains unclear.
Then we conduct two experiments to tracing these large weights during training.
We also use ”these large weights” to indicate these weights having the larger magnitude/larger norm in the
[Figure]
We design two ﬂipping methods, ﬂipping those weights with larger norm (from the largest norm to the smallest norm) and ﬂipping those weights with the smaller norm.
The X-axis indicates how many percentage of weights is ﬂipped, while the Y-axis indicates the inference accuracy.
The topleft point in each ﬁgure is the un-ﬂipped case which is the same as the result reported in Table.2.
This ﬂipping operation is done to each binary Conv layer and each layer has the same ﬂipping percentage.
[Figure] [Figure]
Top 0.
[Figure]
Top 0.
[Figure]
XNor-BWN VGG 0
50 75 100 125 150 175 200 Epoch 0.00 0.
H a mm i n g D
i s t a n c e 0 25 50 75 100 125 150 175 200 Epoch 5070
P e r c e n t a g e
Top 0.01 Positive Top 0.05 Positive Top 0.1 Positive Top 0.2 Positive Top 0.3 Positive 0
50 75 100 125 150 175 200 Epoch 5070
P e r c e n t a g e
Top 0.01 Positive Top 0.05 Positive Top 0.1 Positive Top 0.2 Positive Top 0.3 Positive
LQ-BWN VGG 0
50 75 100 125 150 175 200 Epoch 0.
H a mm i n g D
i s t a n c e network that has already ﬁnished training.
One is to trace these large weights’ signs, to ﬁnd when these weights’ signs become the same as the ones ﬁnishing training.
Another is to trace these large weights’ indices, to ﬁnd when they become the largest weights among all weights.
The results of VGG-7 are shown in Figure.3.
The results of ResNet-20 in Figure.9 and ResNet18 in 10 are placed in Appendix.
F. We ﬁnd those large weights mostly have been decided in the early training stage.
The larger magnitude these weights ﬁnally have, the earlier they decide and ﬁx their sign.
And this rule also applies to their magnitude, that the ﬁnal trained weights with larger magnitude become having a larger magnitude in the very early stage.
Both curves have a similar trend to the accuracy curve’s trend.
2. P RIMARY B INARY S UB -N ETWORKS IN BWN S
We ﬁnd that there are weights with the large norm, ﬁxing their signs in the early training stage.
These weights are stable and vulnerable when inversing their signs.
We name these weights as Primary
Binary Sub-Networks .
This idea is akin to the lottery ticket hypothesis (Frankle & Carbin, 2019), but the difference is our primary binary sub-networks’ weights usually have ﬁxed signs, and the rest of BWNs are not zero like the pruned networks.
The primary binary sub-networks have the same norm for each weight after binarization, but different importance.
The lottery ticket is based on full-precision network pruning, and it pays more attention to getting sparse networks using the retraining method, while ours is to claim the meta idea that weights with larger norm are stable and sensitive on signs’ changes.
We will show how we utilize this idea in the rest paper.
2. B INARY -K ERNEL D ISTRIBUTION
Besides the centered distribution of full precision weights in each layer, we ﬁnd that there exists another distribution of binary-kernels in each layer.
For a binary-kernel with 3 × 3 kernel size, there are 2 9 possible kernels in total.
For easier illustrations, we use 0 to 511 to index these kernels as shown in Figure.4.
3 × 3 kernels are more widely used in common CNN like VGG, ResNet,
DenseNet (Huang et al., 2017), and MobileNet (Howard et al., 2017) (except the ﬁrst Conv layer
[Figure]
[-XNor-BWN VGG 0 100 200 300 400 500 Kernel Index
-]
which is usually not binarized).
From Figure.
we can ﬁnd that certain binary-kernels are in favor across different layers and networks.
Q UANTIZED B INARY -K ERNEL N ETWORKS
In this section we will introduce Quantized Binary-Kernel Networks (QBNs).
In previous sections, we have several conclusions: 1.
Scaling factors are not essential to BWNs, which guide us not to concentrate on designing scaling factors since good learning rates help in the most cases; 2.
Weights with larger magnitude contribute to the primary binary sub-networks in BWNs, and these large weights are stable but sensitive on sign changing, determined and ﬁxed in the early training stage; 3.
Certain binary-kernels are centered on a limited number of the most frequent binary kernels.
All these conclusions lead us to propose a new compression algorithm, which will further compress BWNs into a more structural and compact network, and we name this algorithm Quantized BinaryKernel Networks(QBNs) .
QBN basically is to the ultimate extent to maintain the primary binary sub-networks, changing those smaller weights’ signs and quantize the less frequent kernels to those high frequent kernels to save space.
3. A LGORITHM
Before training a QBN, we ﬁrst train an ordinary VGG-7 XNor-BWN on Cifar-10 and extract its last Conv layer’s binary kernel distribution.
This has been already done as shown in Figure.5.
Then we sort these binary kernels according to their appearance frequency and select top 2 1 , 2 2 , ..., 2frequent binary kernels.
These kernels are called selected binary-kernels K 0 , 1 ,..., 2 8 − 1 .
In the rest of the paper, we use the selected binary kernels to indicate the kernels K 0 , 1 ,..., 2 8 − 1 in our algorithm.
In our following experiments these selected binary kernels are extracted from one single VGGBWN’s last Conv layer.
After pre-processing these and obtaining K 0 , 1 ,..., 2 8 − 1 , we start to train a QBN using Algorithm.1, which is written with python-style pseudocodes.
We use the function where ( A, B, C ) from NumPy indicating that if the value satisﬁes A then equals to B otherwise equals to C.
Parameters: Quantized kernel bit number p , selected kernels K 0 , 1 ,..., 2 p − 1 , hyper-parameter threshold ∆ , weight input channel number I , output channel number O , scaling factors α = 0 .
Input: W
E = Σ t=nt=1 | W t | n for w in W do if abs(w) > ∆E then w = sign(w) end endfor i in range ( I ) do for j in range ( O ) do for m in range ( 2 p ) do
L 2 (m) =
|| W ij − K m ||end m ∗ =
argmin m (L 2 (m))
W ij = α K m ∗ end end
Return: W
[Table]
We set scaling factors ﬁxed to 0.05 when using default learning rate mentioned in experimental settings of Section.2.2.
We use L 2 norm to calculate the distance between the full-precision kernel
W ij to the selected kernels K m , where the full-precision kernel will be replaced by the selected kernel whose distance to the full-precision kernel is the shortest one during forward.
3. QBN E XPERIMENTS
We display our QBN experiments on in Table.1, where we use the same experiment settings mention in Section.2.2.
Besides different networks and datasets are tested, we also use a different quantized bit on these networks to ﬁnd how QBN can perform.
When we use the quantized bit p < 9 , we can use less than 9-bit number to represent the binary-kernel, this provides the compression ability of QBN.
We use compressed ratio (CR) which is a number larger than 1 to show the ratio between the original BWNs and the compressed model’s parameters only including binarized layers.
In this paper, we do not use 8-bit quantized binary kernels, which have a high computational cost and small compressed ratio.
D ISCUSSION ON QBN
In this section, we will discuss the experimental results of QBN and its potential usage, including model compression, kernel quantization strategies, the existence and transferability of the selected kernels, and other selection of binary-kernels.
4. M ODEL C OMPRESSION
With the discovery that BWNs contain primary binary sub-networks, we can reduce the number of parameters to represent a binary-kernel by changing the small magnitude weights’ signs with bearable to the performance of BWNs.
For VGG-7 on Cifar-10 and ResNet-18 on ImageNet, we can compress their parameters to an extremely small number by replacing the whole 512 types of 3×3 binary-kernel with fewer types of binary kernels from those 2 k selected binary-kernels, and the compressed ratio can be higher than 5 × .
For ResNet-20 and ResNet-56 which are thinner and have a small number of channels and parameters, they have a low endurance on compression, the compressed ratio can achieve to 1 .
5 × with a bearable accuracy drop (less than 3 % on Cifar-10).
For a more aggressive compression with very low bit quantization binary-kernels, networks with fewer parameters like ResNet-20’s training stability will drop due to their limited number of parameters.
The experimental results are shown in Table.3 in Appendix.
H.
4. C ONNECTION BETWEEN P RIMARY B INARY S UB -N ETWORKS
We use a hyper-parameter threshold ∆ in Algorithm.1 to bridge QBN and Primary Binary SubNetworks.
When ∆ = 0 , it means we ﬁrst binarize all weights, then quantize these binary-kernels to those selected kernels.
When ∆ is large enough, it means we directly quantize the full-precision kernels.
When ∆ E is at a proper range of weight norm, those large weights will be ﬁrst binarized to ± 1 .
Considering the weight norm is usually a small value (from weight visualization in Figure.
and Figure.8)
compared to 1 , these large weights receive a larger penalty by changing their signs during calculating the L2 distance between full-precision kernels and the selected binary-kernels.
Thus, ∆ is a hyper-parameter deciding how many portions will be considered as large weights, in the same term, Primary Binary Sub-Networks.
According to our experiments of using different ∆ in Figure.16 of Appendix.
O, we ﬁnd that ∆ > 0 is almost better than ∆ = 0 .
This sign() operation for all weights will eliminate the information of full-precision norm.
Overall, these experimental results suggest our settings of ∆ primary binary sub-networks ﬁrst helping on quantizing binary-kernels, compared to binarizing weights ﬁrst.
4. Q UANTIZATION B IT S TRATEGY
When using low quantization bit for binary-kernels, the performance drop will not be negligible, thus how to assign quantization bit to different layer is important.
For VGG-7 and ResNet, they contain much more parameters in higher layers (layers near to the output), which have more channels, but their computational cost is similar in each layer.
From the view of model compression, we ﬁnd that higher layers have a higher endurance for the low-bit quantized kernels compared to lower layers (layers near to the input).
Thus, we use low-bits in the last layer/group and use more-bits for the rest layers/groups to avoid bottlenecks.
4. E XISTANCE AND T RANSFERABILITY OF THE S ELECTED K ERNELS
To prove the existence of the selected kernels in other cases, and the transferability of our selected kernels, we did experiments on extracting top frequent kernels from different networks and layers and compare them with our selected kernels in Appendix.
L.
Then we conduct ﬁne-tuning experiments for a pretrained BWN.
This will be further studied in Appendix.
M.
4. O THER S ELECTION OF B INARY -K ERNELS
We discuss the other selection of binary-kernels in Appendix.
N. For very low-bit quantization, we suggest using the most frequent binary-kernels rather than those less frequent ones.
For the case like quantization bit p > 4 , the choice of binary-kernels is not a essential problem.
A A PPENDIX
In this appendix
[Table]
we will display many additional experiments due to the limited pages in the Context.
These experiments are all introduced in the Context as supplementary material to strength our main idea and make our contribution more convincing.
B E XPERIMENTS ON T RAINING BWN S
In this appendix section
[Table]
we display our experiments on training different networks using XNorBWN and LQ-BWN as shown in Table.2.
[Table]
C C OMPARE L EARNABLE SF AND GAMMA IN BN
In original LQ-Net
[Table]
they use an algorithm called ’Quantization Error Minimization’ to calculate channel-wise quantizers, which are learnable scaling factors for each channel of Conv layers in binary weight case.
Similarly, there is γ in BatchNormalization layer, which also processes the preactivation values in a channel-wise manner.
In LQ-BWN, we normalize both γ and scaling factors in such a channel-wise manner
[Table]
and then plot the relation between these two values in Figure.6.
As the ﬁgure shows, when the network goes deeper, two values behave highly correlated to each other, especially when the Conv layer is wide.
Their high correlation between each other indicates the overlapping function of both.
[Table]
B a t c h N o r m
G a mm a r=0.
R-18 G1B1C 0.0 0.2 0.4 0.6 0.8 1.0 LQ-Net Scaling Factors 0.
B a t c h N o r m
G a mm a r=0.
R-18 G2B1C 0.
0.8 1.0 LQ-Net Scaling Factors 0.
B a t c h N o r m
G a mm a r=0.
R-18 G3B1C
D Q UANTIZATION E RROR C URVE
In XNor-Net where XNor-BWN is ﬁrst raised, the scaling factors are proposed to minimize the quantization error in a calculated deterministic way.
In the BNN survey (Qin et al., 2020), the authors summarized several BWNs algorithms using ”Minimizing the Quantization Error” which has a common form as shown in Equation3 to design their methods.
Thus we plot the quantization error curve of different networks as shown in Figure.7.
We can ﬁnd that in most case the quantization error between full-precision weights and binary weights is not minimized.
Therefore we are concerned that it might not be reasonable to use scaling factors to reduce the quantization error, even further, it might not be necessary to reduce the quantization error.
[Equation]
E W EIGHT H ISTROGRAM ON O THER N ETWORKS AND L AYERS
We visualize the full-precision weight histrograms of ResNet-18’s different layers in Figure.8.
F T RACING L ARGE M AGNITUDE W EIGHTS
We trace the large magnitude weights’ signs and their norm overlapping of ResNet-20 in Figure.
and ResNet-18 in Figure.10.
We also plot the hamming distance of ResNet-18 between the weights during training and weights after training in Figure.11.
The experiment contents are the same as the ones in the Context.
[Table]
i s t a n c e w/o WD
[Figure] [Figure]
L 2 D i s t a n c e w/o WD
[Figure]
e n s i t y
C0G3B1C
[Figure] [Figure]
G F LIPPING AND P RE - FIXING L ARGE M AGNITUDE W EIGHTS AND R ETRAINING
We ﬁrst ﬂip 10% of weights of a trained BWN, and then ﬁx its weights according to their magnitude and retraining it in Figure.12.
The small learning rate in retraining is 0.0002, which is applied to the ordinary 160th to 200th epoch training.
The large learning rate is using the original learning rate when training a BWN from scratch.
We use three ﬂipping strategy, ﬂipping those largest magnitude weights, random weights, and those smallest magnitude weights.
H M ODEL C OMPRESSION A DDITIONAL E XPERIMENTS
We did more experiment with different quantization bit (together with different compressed ratio) on different networks and datasets as shown in Fig3.
I B INARY -K ERNEL D ISTRIBUTION IN EACH L AYER
We visualize the binary-kernel frequency of other layers and networks in Figure.13.
We sum the frequency of top 2 p binary-kernels in each group of ResNet-18 with the log-scale x-axis.
J I NSTABILITY T RAINING ON N ETWORKS WITH V ERY F EW P ARAMETERS
When training QBNs on ResNet-20 with low bit quantizations, the test accuracy’s variance is large, making training unstable.
We write down a table on ResNet-20 with the training accuracy Standard
Deviation among 5 times in Table.4.
K E XISTANCE AND T RANSFERABILITY OF THE SELECTED KERNELS
Selected binary-kernels do exist, and selected binary-kernels can transfer to other different layers, networks, or datasets.
We extract different layers’ top 2 1 , 2 2 , ..., 2 7 frequent kernels percentage over all kernels and display in Table.13.
It shows that usually, the imbalanced distribution among BWNs are very common, where only a small number of binary-kernels can take an advantage percentages
[Figure]
[Figure] [Figure] [Figure]
[Figure]
H a mm i n g
D i s t a n c e
XNor-BWN R-
[Figure]
H a mm i n g
D i s t a n c e
LQ-BWN R-
[Table] [Table]
[Figure] [Figure]
S e l e c t e d K e r n e l s P e r c e n t a g
[Figure]
[Figure]
S e l e c t e d K e r n e l s P e r c e n t a g e
[Figure]
[Figure] [Figure]
[Figure]
S e l e c t e d K e r n e l s P e r c e n t a g e
[Figure]
LQ
[Figure]
e r
[Figure]
P
[Figure]
of all binary kernels.
This proves that the QBNs algorithm can adapt to a wide range of networks and datasets.
To prove the transferability of the selected binary-kernels, Table.3 shows that the experiments are all based on the same batch of the selected kernels, which are extracted from the last Conv layer of a VGG-7 XNor-BWN without any additional tricks.
The results on ResNet-20, ResNet-56 on
Cifar-10, and ResNet-18 on ImageNet with such residual structures and different layers show how those selected kernels perform well.
We display how the selected binary-kernels we use throughout our experiments from the last Conv of VGG-7 appear in other networks’ different layer, plotting a
Figure.14 to compare their frequency.
We also apply QBNs to ﬁne-tune pre-trained BWNs, when using relatively more quantization bit, the network can usually gain comparable performance without training from scratch.
In our QBN algorithm, the computational cost increases when using more quantization bits, thus we can directly ﬁne-tune a pre-train BWN.
The experiments in Appendix.
M show that we can gain a uniform 7 − bit
QBN in one epoch, and a uniform 6 − bit QBN in two epochs.
This provides a higher efﬁcient way to train those high quantization bit QBNs.
L T HE D EGREE OF A GGREGATION OF B INARY -K ERNELS
We do a statistic on how much percentage of the target networks’ top frequent binary-kernels can be covered by our selected binary-kernels from the last Conv of XNor-BWN VGG-7.
We use the ﬁrst ﬁgure on the top left of Figure.14 to demonstrate that, when using 1 − bit quantization on binary-kernels of XNor-BWN ResNet-18’s ﬁrst 4 layers(the ﬁrst group of ResNet-18), our selected kernels can cover 100% of XNor-BWN ResNet-18’s top 2 1 frequent kernels.
The percentage has already been processed with the number of each kernel in the layer.
100% is the limit that if we directly choose the most frequent kernels from this layer/group as our selected kernels.
For the lowest percentage appearing when using 5-bit, there are still more than 80% kernels in the ﬁrst group’s top 2 5 frequent kernels can be represented directly by our selected kernels.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
e
d K
e
[Table]
XNor-BWN
[Table]
G
[Table]
r
[Table]
r c
[Table]
n t a
[Table]
LQ-BWN R-18 G
[Table]
Bit 0.92 0.
S e l e c t e d K e r n e l s P e r c e n t a g e 8 Layer9 Layer10 Layer 11 Layer
LQ-BWN R-18 G 2 1 2 2 2 3 2 4 2 5 2 6 2 7 Quantization Bit 0.
S e l e c t e d K e r n e l s P e r c e n t a g e 12 Layer 13 Layer 14 Layer 15 Layer
LQ-BWN R-18 G
M F INE - TUNE WITH HIGHER QUANTIZATION BIT ( ON 6 BIT , 7 BIT )
We write a Table.5 using relatively higher quantization bit to ﬁne-tune ResNet-18 on ImageNet with one epoch.
The pretrained model is the normal ResNet-18 BWN at 90th epoch.
We list the number of epochs that the ﬁne-tune requires.
For those lower bit, the ﬁne-tuned performance cannot achieve to the one training from scratch.
N O THER S ELECTION OF B INARY -K ERNELS
Here we list three reasons to illustrate why we use such a way to collect the binary-kernels.
If we regard QBNs as a clustering method given cluster numbers, these kernels with high appearance frequency are most likely the cluster centers.
We visualize these top frequent kernels as shown
[Table]
in Fig.15, and they appear similar to conventional spatial ﬁlters, including ”Moving Average Filter”, ”Point Detector”, ”Line Detector” and so on.
We use the less frequent binary-kernels as the selected kernels to test if our selection based on frequency is a good choice.
We ﬁnd that using those less frequent kernels, an accuracy drop can be observed in different experiments.
Those less frequent kernels are used by inverting the order of top 128 frequent kernels, eg.
for 2 2 kernels, they are 125-th, 126-th, 127-th, and 128-th frequent kernels.
Given the experiment results in Table.6, we ﬁnd that when using very low quantization bit, speciﬁcally less than or equal to 3 , will signiﬁcantly hurt the network when using those least frequent kernels.
When the quantization bit increases, the difference in performances reduce.
Thus we suggest to use those most frequent kernels if training a very low-bit QBN, or ﬁnetuning a pre-trained BWN.
In other cases like training from scratch with quantization bit more than 3, the frequency of the selected kernels is not a strict deterministic factor.
[Table]
O I NFLUENCE OF H YPER -P ARAMETER ∆
To test the inﬂuence of the Hyper-Parameter ∆ in our algorithm
[Table]
we use VGG-7 with all 5-bit quantization kernels and ResNet-56 with FP-9-6-2.
We do not use ResNet-20 due to its instability training which we have discussed.
The result is shown in Figure.
[Table]
P T HE S ELECTED K ERNELS FROM O THER C ONV L AYERS
In previous experiments
[Table]
our selected binary kernels K m use the statistic information extracted from one single VGG-7’s last Conv layer and regard them as constant kernels.
We further use different strategies to demonstrate that the selected binary kernels have a strong generalization ability to apply on other networks.
In the following experiments, we use the selected binary kernels from the last
[Figure]
layer of ResNet-20, VGG-7’s 3rd and 5th Conv layers, and from the last layer of ResNet-18. We use these kernels from different strategy to other different networks. The result is shown in Table.7. [Table]