[Re:] Training Binary Neural Networks using the Bayesian Learning Rule
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
(1) gives a mathematically principled approach to solve the discrete optimization problem that occurs in the case of
Binary Neural Networks and claims to give a similar performance on various classiﬁcation benchmarks such as MNIST,
CIFAR- and CIFAR-100 as compared to their full-precision counterparts, as well as other recent algorithms totrain BNNs like PMF and Bop.
The paper also claims that the BayesBiNN method has an application in the continuallearning domain as it helps in overcoming catastrophic forgetting of the past by using the posterior approximation of the previous task as a prior for the upcoming task.
We try to reproduce all the results presented in the original paper bymaking a separate and independent codebase.
Scope of Reproducibility
We try to verify the performance of our re-implementation of the BayesBiNN optimizer on various classiﬁcation andregression benchmarks.
We also implemented the STE optimizer which was the central baseline model used in thepaper.
Finally, we tried to evaluate the results of BayesBiNN on the continual learning benchmark to get a better insight.
Methodology
We developed our separate code-base, consisting of an end-to-end trainer with a Keras-like interface, for the reproductionwhich includes the implementation of the BayesBiNN and STE optimizer.
We did refer to the author’s code open-sourcedon GitHub to get some insights about the hyperparameters and other doubts that emerged during code development.
Results
We reproduced the accuracy of the BayesBiNN optimizer within less than 0.5% of the originally reported value, whichupholds the conclusion that it performs nearly as well as its full-precision counterpart in classiﬁcation tasks.
When wetried this in a semantic segmentation context, we found that the results were very underwhelming and in contrast withthe seemingly good results by the STE optimizer even with much hyperparameter tuning.
We can conclude that, likeother Bayesian methods, it is difﬁcult to train BayesBiNN on more complex tasks.
What was easy
After we worked out the mathematics behind the BayesBiNN approach, we developed a pseudo-code for the optimizationprocess which along with references from the author’s code, helped us a lot in our reproduction study.
What was difﬁcult
Some of the hyperparameters were not mentioned by the authors in their paper so it was difﬁcult to approximate thevalues of those parameters.
The lack of resources was the next big difﬁculty that we faced.
Communication with original authors
We had a very fruitful conversation with the authors, which helped us in better understanding the BayesBiNN approachand its extension to the segmentation domain.
The detailed pointers are given at the end of this report.
Introduction
Deep Learning is moving towards larger and larger parameters day-by-day, which often makes it difﬁcult to run onresource-constraint devices like mobile phones.
Binary Neural Networks (BNNs) could act as a savior in such situations,helping in largely saving storage and computational costs.
The problem of optimizing this binary set of weightsis clearly a discrete optimization problem.
Previous approaches like Straight-Through Estimator (STE) and Binary Optimizer (Bop) tend to ignore this and use gradient-based methods, which still worked in practice.
The paper presentsa mathematically principled approach for training BNNs which also justiﬁes the current approaches.
Scope of reproducibility
The paper mentions a bayesian approach to solve the discrete optimization problem in the case of Binary Neural
Networks (BNNs).
The outcome of this approach was a BayesBiNN optimizer which could be used to train BNNs andachieve similar accuracy as compared to their full-precision counterparts.
To verify the claims given in the paper, wetarget to achieve the following objectives:
• Work out and present the mathematics behind BayesBiNN in a simpler way and present the pseudo-code tothe optimizer.•
Implement the BayesBiNN optimizer and STE optimizer to verify the accuracy on tasks of varying complexities,as reported in the original paper. Reproduce the results for other baselines present in the paper such as proximal mean-ﬁeld (PMF) according tothe hyper-parameters given in the paper. • Evaluating the performance of BayesBiNN optimizer in more complex domains like semantic segmentation.
Methodology
We have re-implemented the algorithm proposed in the paper from scratch using PyTorch and created an end-toend model trainer with a Keras-like interface.
We referred to the code given by the authors for the baseline modelhyperparameters and the source of synthetic datasets.
Following is the algorithmic form of what the authors havepresented in the paper.
Algorithm
Input: Initialize λ for number of training epochs do for i =
,number of mini-batch examples do
Sample (cid:15) ∼ U (0 , 1) and set δ = 12 log (cid:15) 1 − (cid:15)
Initialize w b = tanh( ( λ + δ ) /τ ) Compute following using gumbel-softmax trick
[Equation]
end Update µ and λ using following equation
[Equation]
end This would make the paper more interpretive in terms of implementation
Some the mathematical expressions mentioned in the original paper were presented from various sources and missedout several intermediate steps which we found to be very important while reproducing the paper from scratch.
Here we present a step-wise derivation of some important expressions written in the original paper [Equation]
Bayesian formulation of the discrete optimization problem, in which loss has to be minimized w.r.t posterior q ( w ) ,given prior p ( w ) can be written as:
[Equation]
To solve the above optimization problem, Bayesian learning rule given in (6) is applied, assuming solution to be a partof minimal exponential family of distribution, given by:
[Equation]
where base measure h ( w ) is assumed to be 1.
Following is the update rule used to learn λ :
[Equation]
where ρ is the learning rate, µ = E q ( w ) [ φ ( w )] .
Bernoulli distribution being a special case of minimal exponentialfamily distribution, we assume prior p ( w ) ∼ Bern( p ) with p = 0 .
5 , and posterior q ( w ) to be mean-ﬁeld bernoullidistribution:
[Equation]
For weight j ,
[Equation]
Comparing above expression with minimal exponential family distribution, we can say:
[Equation]
We deﬁned µ = E [Equation] [Equation]
[Equation] [Equation]
[Equation]
From above derivations we can say that, p = 1 / (1 + exp( − 2 λ ))
=
Sigmoid(2 λ ) and q ( w ) ∼ Bern( p ) .
To implement the update rule, we need to compute the gradient with respect to µ .
Original paper uses a reparamaterization trick called Gumbel-softmax trick (7), which is used to relax the discrete random variables of a concrete distribution(for eg, bernoulli distribution).
Binary concrete relaxation (7) of Binary concrete random variable X ∈ (0 , 1) withdistribution X ∼ BinConcrete( α, λ ) with temperature λ and location α ,
[Equation]
where L ∼ Logistic.
And its density is given by
[Equation]
Using above expressions, for binary weights w j ∈ { 0 , 1 } , relaxed variable w (cid:15) j ,τ r ( p j ) ∈ (0 , 1) can be used withtemperature τ and α = e 2 λ given by
[Equation]
where δ j ∼ Logistic and its density is given by
[Equation]
Experimental setup4.1 Model descriptions
We kept the model architectures the same as mentioned in the original paper to maintain uniformity and implementedthem ourselves.
For the MNIST classiﬁcation task, we used the BinaryConnect architecture and for the CIFARclassiﬁcation task, we use the VGGBinaryConnect architecture.
The authors also compared their BayesBiNN
methodwith the LR-Net method in (8).
We implemented the same model architecture as in the LR-Net paper.
The detailedarchitectures are mentioned in the supplementary material provided with this report.
For the segmentation task, we usedthe original U-Net architecture, mentioned in (11) with a minor difference that we introduced a BatchNorm layer afterevery convolution layer.
4.2 Datasets
The datasets used for image classiﬁcation tasks are MNIST, CIFAR-10, and CIFAR-100.
For generating visualizationsfor the BayesBiNN and STE methods, we used small toy datasets, the Snelson dataset (10) for regression problems, and
Two Moon’s dataset (9) for classiﬁcation problems.
For the segmentation part, we used the Brain Tissue segmentationdataset taken from (11), and for the continual learning visualizations, we used the permuted MNIST dataset (12).
Thepre-processing of inputs has been kept the same as mentioned in the original paper and has been detailed below.
Pre-processing : For the MNIST dataset we simply normalize the images and do not perform data augmentation.
We keep our validation split as 0.1 uniformly across all sets of experiments except the comparison with the LR-Netmethod (8).
For the CIFAR datasets also, we perform the normalization of images along with data-augmentation wherewe generate images by randomly cropping a 32x32 image from a 40x40 padded image.
Finally, for our semanticsegmentation task, we had a very small dataset of 30 images, out of which 24 were chosen for training and 6 forobtaining the validation score.
No other pre-processing has been done in this case.
4.3 Hyperparameters
We have used the hyper-parameters given in the original paper.
Table 1 contains the list of all the parameters we usedfor our experiments:
Optimizer
[Table]
[Table]
4.4 Computational requirements
All our ﬁnal experimental results were performed on a machine having 1 NVIDIA Tesla V100 GPU and 1 single-coresystem with 16 GB memory.
Training the Binary Network with BayesBiNN optimizer for a single run, takes around 2.
GPU hours for MNIST, 5.5 GPU hours for CIFAR-10, and around 8.5 GPU hours for the CIFAR-100 dataset, in thecurrent experimental setup.
Results
In Table 1 we report our results for various classiﬁcation benchmarks using our implemented BayesBiNN and STEoptimizer.
We notice that we get a difference of less than 0.1% as compared to that in the original paper.
We generated the results for baseline STE optimizer and full-precision networks by evaluating our implementation of these methods.
We also generated the results of PMF, by modifying its original open-sourced code and using the hyperparametersmentioned in the original paper.
[Figure]
[Table]
5.1 Comparison with LR-Net
Authors compared their BayesBiNN approach to the LR-Net method presented in (8).
We tried to reproduce theresult for the same setting.
In this comparison, the data pre-processing and augmentation methods remain the same asmentioned in section 4.2, but we do not split the data in training and validation sets in this case.
We denote the testaccuracies after 190 epochs in the case of MNIST and 290 epochs in the case of CIFAR-10, as done in the originalpaper to maintain uniformity.
Note that, our accuracy is matching with that of the original authors in the case of MNISTbut not in the case of CIFAR-10.
We suspect that this is due to some difference in Batch-Norm layers used.
[Table]
5.2 Continual Learning
As mentioned in the original paper, we try to reproduce the author’s claims about weight distribution across tasks in asimple continual learning domain tested on Permuted MNIST.
As we can see, as we learn across the tasks, the curve becomes ﬂat from the middle conveying that the weights become more deterministic.
Our result matches with theclaims in the original paper.
[Figure] [Figure]
5.3 Visualization using Synthetic Dataset
[Figure]
In the original paper, the authors present visualizations on binary classiﬁcation (Two moons dataset(9)) and toyregression (Snelson dataset(10)) using STE and BayesBiNN optimizer.
For the classiﬁcation task, the authors claimedthat STE is a more deterministic classiﬁer compared to BayesBiNN.
We reproduced this experiment and the resultsdepicted in Figure 3 seem to be consistent with the author’s claim.
For the regression task, we conclude that the author’sclaim about BayesBiNN (mean) giving a smoother curve compared to STE is true, which can also be seen in Figure 4.
[Table]
5.4 Extended Results (Semantic Segmentation)
We tried to validate the performance of the BayesBiNN optimizer on more complex tasks like Semantic Segmentation.
Unfortunately, the results with BayesBiNN were quite underwhelming as compared to STE and its full-precisioncounterpart.
We tried various parameters to improve its performance but none seemed to work.
We had a briefdiscussion with the authors regarding this issue and the authors suggested that Bayesian models are intrinsically verydifﬁcult to train.
For the results shown in Table 5 and Figure 5, we have used the hyperparameters denoted in Table 1.
[Table]
[Figure]
Discussion
We reproduced almost all the experiments given in the original paper and most of our results match with the originalclaims.
While this BayesBiNN approach is mathematically principled, we tried to take a step forward, by usingthat optimizer on a single segmentation task.
However, the results were against our expectation and the result ofsegmentation was a zoomed segmented image of the input with lots of noise.
Apart from this, even in the case ofcomparison with the LR-Net method, our accuracy differs from that of the original authors, which we feel might be dueto some difference in architecture chosen.
The major contribution of our work is developing a code base library basedon PyTorch with a Keras type interface for training BNNs with several different methods in its arsenal.
This wouldreduce the coding efforts while training a BNNs and could help in future research as benchmarking platform.
6.1 What was easy
The original paper contained a very good explanation of the mathematics behind the BayesBiNN approach.
After weworked that out the pseudo-code as pointed out in Algorithm 1, the basic implementation of the optimizer becameeasy and easily veriﬁable by the author’s original code.
The appendix in the original paper contained a list of varioushyper-parameters used for experiments.
This helped us a lot while running the experiments and deciding the range ofhyper-parameters while doing ablation studies.
6.2 What was difﬁcult
The most difﬁcult part here was running a large number of experiments in lack of many computational resources.
Thisdifﬁculty was increased since we are taking an average of 5 runs while reporting all our results.
Apart from this, wealso faced some difﬁculty in taking care of the hyper-parameters, which were not mentioned in the original paper(like momentum coefﬁcient).
To cater to that, we had to guess some possible values of the hyper-parameters and runsmall random searches to ﬁnd a good candidate.
Finally, we also faced difﬁculty while reproducing the results for thebaselines PMF and Bop and adapting their experimental settings to match with those used in the original BayesBiNNpaper.
Since their code was written a long time ago and used older technologies, this task took us a lot of time.
6.3 Communication with original authors
We did not understand the intent of the authors for choosing temperature as 1 in the case of experiments on syntheticdatasets.
We were also curious about the author’s view on segmentation tasks using BayesBiNN.
We mailed this, alongwith the review of their paper, to the authors to ask for some pointers.
They gave the following major pointers:
• It is reasonable that at high temperatures the learned distribution will have high variance.
The mode mentionedin the paper refers to the sign( ˆ( w ) ), where ˆ( w ) denotes the expectation of the learned posterior Bernoullidistribution.
It is not appropriate to directly use the continuous ˆ( w ) as the mode.
Another way is to use mean,which samples from the learned posterior Bernoulli distribution, and then make predictions using ensemblelearning.
• STE is more stable and suggested by the authors to act as a baseline, in particular, Adam STE ﬁrst, to makesure binary networks work.
As shown in the paper, there is literally very little difference between STE and
BayesBiNN but indeed the latter is difﬁcult to train, as most Bayesian optimizers.
Broader Impact
Recent researches (3) mention that training a single big transformer model could emit around 626,155 lbs CO 2 which isaround 5 times of average carbon emission by a car in its total lifetime.
Clearly, Deep Learning takes a huge toll onthe environment which is why there has been an increased focus on much more energy-efﬁcient "Green AI".
BNNsintrinsically have far less computational and space complexity as compared to their full-precision counterparts andas we can see above they can also achieve accuracy close to the full-precision networks, at least in the classiﬁcationtasks, and also show the potential of expanding well to more complex segmentation tasks.
This can help us a lot inmoving towards cleaner Deep Learning.
This class of technology also provides a huge set of opportunities in extending AI to edge devices with much smaller and low-energy systems.
We feel that its potential impact on the environmentand sustainability is at par with its academic importance, that is why we see it as a much larger thing than just a set ofpublications.