R ES B IN N ET : R ESIDUAL B INARY N EURAL N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent efforts on training light-weight binary neural networks offer promising execution/memory efﬁciency.
This paper introduces ResBinNet, which is a composition of two interlinked methodologies aiming to address the slow convergence speed and limited accuracy of binary convolutional neural networks.
The ﬁrst method, called residual binarization, learns a multi-level binary representation for the features within a certain neural network layer.
The second method, called temperature adjustment, gradually binarizes the weights of a particular layer.
The two methods jointly learn a set of soft-binarized parameters that improve the convergence rate and accuracy of binary neural networks.
We corroborate the applicability and scalability of ResBinNet by implementing a prototype hardware accelerator.
The accelerator is reconﬁgurable in terms of the numerical precision of the binarized features, offering a trade-off between runtime and inference accuracy.
I NTRODUCTION
Convolutional Neural Networks (CNNs) have shown promising inference accuracy for learning applications in various domains.
These models are generally over-parameterized to facilitate the convergence during the training phase (Hinton et al.
; Denil et al.
A line of optimization methodologies such as tensor decomposition (Kim et al. (2015); Zhang et al. (2015)), parameter quantization (Hubara et al.
; Han et al.
(2015)), sparse convolutions (Liu et al.
Wen et al. (2016)), and binary networks (Courbariaux et al.
; Rastegari et al.
(2016)) have been proposed to reduce the complexity of neural networks for efﬁcient execution.
Among these works, binary neural networks result in two particular beneﬁts: (i)
They reduce the memory footprint by a factor of 32 compared to the full-precision model; this is speciﬁcally important since memory access plays an essential role in the execution of CNNs on resource-constrained devices.
(ii) Binary networks replace the costly multiplications with simple XNOR operations (Rastegari et al.
Umuroglu et al.
, reducing the execution time and energy consumption signiﬁcantly.
Considering the prior art, there exist two major challenges associated with binary neural networks.
First, the convergence rate of the existing solutions for training binary CNNs is considerably slower than their full-precision counterparts.
Second, in order to achieve comparable classiﬁcation accuracy, binarized neural networks often compensate for the numerical precision loss by employing high dimensional feature maps in a wide CNN topology, which in turn reduces the effective compression rate.
As a result, full-precision networks often surpass binary networks in terms of convergence rate and ﬁnal achievable accuracy.
In this paper, we propose ResBinNet, a novel solution for increasing the convergence rate and the ﬁnal accuracy of binary networks.
The global ﬂow of ResBinNet is depicted in Figure 1.
The ﬁrst phase, which we call Soft Binarization , includes two methodologies that we propose to address the aforementioned challenges for training binary CNNs.
First, we introduce a Residual Binarization scheme which allows the number of possible values for activation units to be reconﬁgurable at runtime.
To this purpose, we learn a multi-level residual representation for the features within the CNN to adaptively increase the numerical precision of the activation units.
Second, we introduce a novel weight binarization approach, called Tempreture Adjustment , which aims to gradually enforce binarization constraints over the weight parameters throughout the training phase.
The two interlinked methods signiﬁcantly improve both the convergence rate and the ﬁnal accuracy of ResBinNet compared to prior art.
Once the soft training phase is ﬁnished, we convert the weights to actual binary values (0,1).
Fine-tuning of the model is then performed in Hard Binarization phase using existing training algorithms (e.g. BinaryNets
(Courbariaux et al.
in few epochs (e.g. one epoch).
[Figure] ResBinNet is designed to fulﬁll certain goals: (i) It should enable reconﬁgurability for binary neural networks; in other words, the number of residual binary representatives should be adjustable to offer a trade-off between inference accuracy and computation time.
(ii)
The multi-level binarized features should be compatible with the XNOR multiplication approach proposed in the existing literature.
(iii) ResBinNet should speed up the convergence rate of binarized CNNs.
(iv) Current hardware accelerators for binary CNNs should be able to beneﬁt from ResBinNet with minimum modiﬁcation in their design.
In summary, the contributions of this paper are as follows:
• Proposing residual binarization, a methodology for learning multi-level residual representations for each feature map in binary CNNs.
• Introducing temperature adjustment as a practical approach for gradual (soft) binarization of CNN weights.
• Analyzing the trade-off between accuracy and execution time of ResBinNet on a real hardware design.
• Evaluating ResBinNet convergence rate and accuracy on three datasets: MNIST, SVHN, and CIFAR-10.
• Development of an open-source Application Program Interface (API) for ResBinNet 1 .
The remainder of the paper is organized as follows:
In Section 2, we describe the residual binarization method for binarizing activations.
Section 3 explains the temperature adjustment technique for binarizing weights.
In Section 4, we discuss how particular ResBinNet operations (e.g. multi-level
XNOR-popcount) can be efﬁciently implemented on existing hardware accelerators.
Experiments are discussed in Section 5.
Finally, we discuss the related work and conclusion in Sections 6 and 7.
R ESIDUAL B INARIZATION
A binarization scheme converts value x to the binarized estimation e x , which can take one of the possible values γ or − γ .
This representation allows us to represent e x with a single bit b x .
In particular, for a given layer within the CNN, we can store the single full-precision value of γ as a representative for all features, and reduce the memory footprint by storing bits b x instead of x for each feature.
Assuming that both the weights and input features of a CNN layer are binarized, each dot product between a feature vector (cid:126)x and weight vector (cid:126)w can be efﬁciently computed using
XNOR-popcount operations as previously suggested in (Courbariaux et al.
Rastegari et al.
Let (cid:126)x = γ x (cid:126)s x and (cid:126)w = γ w (cid:126)s w where { γ x , γ w } are scalar values and { (cid:126)s x ,(cid:126)s w } are the corresponding sign vectors.
The binary representations of { (cid:126)x, (cid:126)y } , which we denote by { (cid:126)b x ,(cid:126)b w } , are simply computed by encoding the sign vectors (cid:126)s to binary vectors.
The dot product between (cid:126)x and (cid:126)w can be computed as:
[Equation]
where xnorpopcount ( . , . ) returns the number of set bits in the element-wise XNOR of the input binary vectors.
[Figure] [Figure]
Multi-level Residual Binarization :
Imposing binary constraints on weights and activations of a neural network inherently limits the model’s ability to provide the inference accuracy that a ﬂoatingpoint counterpart can achieve.
To address this issue, we propose a multi-level binarization scheme where the residual errors are sequentially binarized to increase the numerical precision of the estimation.
Figure 2 presents the procedure to compute an estimate e from input x .
Each level of the graph (say the i th level) computes the corresponding estimate
e i by taking the sign of its input (the residual error from the ( i − 1) th level), multiplying it by a parameter γ i , and adding ± γ i to the estimate of the previous level.
In addition, it computes the residual error r i and feeds it to the input of the next level.
The estimates of deeper levels are therefore more accurate representations for the input x .
Note that the estimate e i in level i can be represented using a stream of i bits corresponding to the signs of inputs to the ﬁrst i levels.
Residual Binary Activation Function : Similar to previous works which use the Sign function as the activation function, in this paper we use the residual binarization to account for the activation function.
The difference between our approach and the single-bit approach is shown in Figure 3.
Each level has a separate full-precision representative γ i , which should be learned in the training phase.
In this setting, the gradients in the backward propagation are computed the same way as in the conventional single-bit binarization (Courbariaux et al.
; Rastegari et al.
(2016)), regardless of the number of residual levels.
In the forward propagation, however, the computed results of our approach provide a more accurate approximation.
For instance, if we employ 2 levels of residual binarization, the activation function can take 2 2 = 4 different values.
In general, the total number of possible values for the activation functions for an l -level residual binarization scheme is 2 l .
Multi-level XNOR-popcount :
In ResBinNet, the dot product of an l -level residual-binarized feature vector (cid:126)e and a vector of binary weights (cid:126)w can be rendered using l subsequent XNOR-popcount operations.
Let (cid:126)e = (cid:80) li =
1 γ ei (cid:126)s ei and (cid:126)w = γ w (cid:126)s w , where (cid:126)s ei and (cid:126)s w correspond to the sign of i th residual in (cid:126)e and sign of (cid:126)w , respectively.
The dot product between (cid:126)e and (cid:126)w is computed as: dot ( (cid:126)w,(cid:126)e ) =
dot ( l (cid:88) i = γ ei (cid:126)s ei , γ w (cid:126)s w ) =
l (cid:88) i =
γ ei γ w dot ( (cid:126)s ei ,(cid:126)s w )
=
l (cid:88) i =
γ ei γ w xnorpopcount ( (cid:126)b ei ,(cid:126)b w ) , where { (cid:126)b ei , (cid:126)b w } are the binary representations corresponding to { (cid:126)s ei ,(cid:126)s w } , respectively.
Note that the subsequent XNOR-popcount operations can be performed sequentially, thus, the same memory used for operating on (cid:126)b ei can be reused for operating on (cid:126)b ei +1 .
As a result, the actual memory footprint for a multi-level residual binarization is the same as that of a single-level binarization, provided that the bit streams are processed sequentially.
Residual Encoding :
In order to convert matrix-vector multiplications into XNOR-popcount operations, we need to encode a feature x into a stream of binary values { b ei | i ∈ 1 , 2 , . . .
, l } .
The pseudo code for this operation, which we call Residual Encoding , is presented in Algorithm 1.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
T EMPERATURE A DJUSTMENT
Approximating the weights of a neural network with binary values often results in loss of accuracy in the pertinent model.
In this section, we explain our methodology to minimize the approximation error during the training, such that the trained weights exhibit lower binarization errors.
Let W denote the parameter set within a certain layer of the neural network.
Instead of directly using W to compute the layer’s output, we perform a change of variable θ = γ H ( αW ) and compute the output using θ .
Here, H ( . ) is a bounded, monotonically-increasing function such as the Hyperbolic Tangent function that is applied on W element-wise.
Parameter γ is a trainable parameter that adjusts the maximum and minimum values that θ can take.
Parameter α , which we call the Temperature henceforth, controls the slope of function H ( . ) .
Effect on Binarization : Figure 4a and 4b illustrate the effect of parameters α and γ on the nonlinear change of variable θ = γ Tanh ( αW ) .
Note that θ acts as a semi-binarized parameter set in the soft training phase.
As we increase the temperature parameter, H ( . ) becomes closer to the binary sign function, meaning that the pertinent θ will exhibit less error when approximated with ± γ .
Note that
W and γ are the trainable parameters in this setting.
Parameter θ is used in the forward propagation phase of the soft training, while in the backward propagation step W is updated.
Effect on Training : Let g θ and g W be the gradients of the training loss function with respect to θ and W , respectively, then we have g W = g θ × ∂θ∂W .
In other words, the magnitude of the gradient that actually ﬂows through W is controlled by ∂θ∂W .
If θ is close to ± γ , the gradient will be ﬁltered out; otherwise, the gradients will ﬂow through W .
Effect of the Temperature on Gradients :
Figure 4c illustrates how the temperature parameter can affect the gradient ﬁltering term ∂θ∂W during the training.
As we increase the temperature, elements of W that are closer to 0 receive ampliﬁed gradients, while the elements that are closer to the binary regime (i.e. θ ≈ ± γ ) encounter damped gradients.
This means that increasing the temperature parameter α will push most of the weights to the binary regime with a bigger force; therefore, a neural network trained with high temperature values exhibits a smaller binarization error.
Temperature Adjustment : Setting a high temperature at the beginning of the training will eliminate most of the gradients, preventing the training loss from being optimized.
To address this problem, we start the soft binarization phase with a low temperature (e.g. α = 1 ) and slightly increase it at the end of each mini-batch.
This approach gradually adapts the weights to binary values during the training.
Figure 5 presents an example of the histogram of the semi-binarized weights θ in different training epochs.
As can be seen, the distribution is gradually shifted towards binary values as the training proceeds.
After soft binarization, the parameter set θ can be used as an initial point for
[Figure] [Figure]
existing hard binarization schemes such as the method proposed by (Courbariaux et al.
As
We illustrate in Section 5, the soft binarization methodology signiﬁcantly increases the convergence rate of binary CNNs.
H ARDWARE A CCELERATOR M ODULES
In this section, we show that the modiﬁcations required to incorporate residual binarization into existing hardware accelerators for binary CNNs are minimal.
ResBinNet provides a trade-off between inference accuracy and the execution time, while keeping the implementation cost (e.g area cost) almost intact; as a result, ResBinNet readily offers users a decision on the latency of their learning application by compromising the inference accuracy.
As an example, we consider the FPGA accelerator for binary CNNs proposed by (Umuroglu et al.
We refer the reader to the mentioned paper for details about the original design.
Here, we describe the modiﬁcations that we integrated into the speciﬁc components of their design to accommodate residual binarization.
The modiﬁed accelerator will be publicly available on Github 2 .
Figure 6 depicts a schematic view of the original hardware and our modiﬁed accelerator.
Note that in the original implementation, each layer takes a single binary vector (cid:126)b in and computes a single output vector (cid:126)b out while the modiﬁed version processes l streams of binary vectors where l is the desired number of residual levels.
Matrix-Vector Multiplication : Both in the original and the modiﬁed accelerators, the matrix-vector multiplication unit is the most computationally intensive among all other operations.
In the original design, this unit takes a binary vector (cid:126)b in and outputs a full-precision vector (cid:126)y .
To accommodate residual binarization, we modify this module as follows: the XNOR-popcount operation is sequentially performed on the stream of binary vectors (cid:126) b in,i .
Each XNOR-popcount results in a different vector (cid:126)y i .
Then, the output is computed as (cid:126)y = (cid:80)
i γ i (cid:126)y i .
Note that the computation overhead of the summation is negligible compared to the XNOR-popcount operation, thus, the runtime of multilevel XNOR-popcount with l -level residual representations is approximately l times the runtime of the conventional XNOR-popcount in the original design.
[Figure]
Batch-Normalization and Activation Function : Batch-Normalization in the inference phase can be viewed as multiplying a vector (cid:126)y by constant vector (cid:126)g and subtracting vector (cid:126)t to obtain the normalized vector (cid:126)y norm .
The original design in (Umuroglu et al.
(2017)) does not require the multiplication step since only the sign of (cid:126)y norm matters to compute the output of the activation function (the Sign function).
In our design, the multiplication step is necessary since the value of (cid:126)y norm affects the output of our activation function, which is encoded using Algorithm 1 and sent to the next layer to be used as an input.
Max-Pooling : The original implementation simply computes the Boolean OR of the binary values to perform max pooling over the features within a window.
In ResBinNet, however, features are represented with l -bit residual representations.
As a result, performing Boolean OR over the binary encodings is no longer equivalent to performing max-pooling over the features.
Nevertheless, the pooling operation can be performed over the encoded values directly.
Assume full-precision values e x and e y , with l -level binary encodings b x and b y , respectively.
Considering ordered positive γ i values (i.e. γ 1 > γ 2 > . . .
> γ l > 0 ), we can easily conclude that if e x < e y then b x < b y .
E XPERIMENTS
We implement our API using Keras (Chollet et al.
(2015)) library with a Tensorﬂow backend.
The synthesis reports (resource utilization and latency) for the FPGA accelerator are gathered using
Vivado Design Suite (Xilinx (2017)).
For temperature adjustment (Section 3), we use a “hard tanh” nonlinearity and gradually increase the temperature by incrementing α at the end of each epoch.
We evaluate ResBinNet by comparing the accuracy, number of training epochs, size of the network, and execution time.
Proof-of-concept evaluations are performed for three datasets: MNIST, CIFAR10, and SVHN.
Table 1 presents the architecture of the trained neural networks.
The architectures are picked from (Umuroglu et al.
[Table]
Effect of the model size on accuracy : As discussed in (Umuroglu et al.
(2017)), the ﬁnal accuracy of the binary CNN for a particular application is correlated with the shape of the network.
For
[Table]
instance, authors of the paper report that the accuracy of MNIST for the architecture in Tablevaries in the range (95.83%-98.4%) when the number of neurons in hidden layers is varied fromto 1024.
Similarly, the architecture in (Umuroglu et al.
for CIFAR-10 is a smaller version of the architecture originally trained by (Courbariaux et al.
Using this smaller architecture drops the accuracy from 88.6% to 80.1%.
In our evaluations, we show that ResBinNet can reduce the accuracy drop using more residual binary levels for the activations of the smaller model.
Effect of the number of epochs on accuracy :
Compared to full-precision neural networks
[Table]
binarized CNNs usually need more training epochs to achieve the same accuracy.
For example, the
CIFAR-10 architecture in (Courbariaux et al.
(2016)) is trained for 500 epochs, while the fullprecision version of the same network can achieve comparable accuracy in roughly 50 iterations 3 .
Here, we argue that soft binarization in ResBinNet can signiﬁcantly reduce the number of training epochs for binary CNNs.
Table 2 compares ResBinNet with two prior arts, namely Binarynet and FINN.
Both baselines use the same training methodology, but the network architectures in FINN are considerably smaller, which leads to lower accuracy rates for FINN.
We evaluate ResBinNet using the small architectures of FINN.
The training of ResBinNet consists of a soft binarization phase and a single ﬁne-tuning epoch.
Note that the ﬁne-tuning phase uses the same algorithm as the two baselines.
The higher accuracy of Binarynet compared to our approach is a direct result of employing a large architecture and training for many epochs.
For each benchmark, the comparison between our approach and the same network architecture of FINN is followed:
• CIFAR-10 :
Compared to FINN, ResBinNet achieves higher accuracy for more than 1 level of residual binarization.
We argue that, even for 1-level binarization, the same accuracy is viable if we ﬁne-tune the soft-binarized model (after 50 epochs) for more than 1 epochs (FINN and ResBinNet use the same algorithms in this phase).
In addition, the convergence rate of ResBinNet is improved as the number of residual levels is increased.
• SVHN and MNIST :
For these datasets, ResBinNet achieves a higher accuracy with even fewer epochs compared to CIFAR-10.
The ﬁnal accuracy and the convergence speed also exhibit improvement as the number of residual levels is increased from 1 to 3.
We now evaluate the area overhead and execution time of ResBinNet for the modiﬁed hardware architecture, which we discussed previously in Section 4.
We compare the implementation of the
CNN architecture used for the CIFAR-10 and SVHN tasks (See Table 1).
Figure 7 compares the hardware resource utilization, and execution time per input.
The resource utilization of ResBinNet is evaluated in Figure 7a, which compares the utilization (in %) for different resources of the FPGA (i.e. BRAM, DSP, LUT, and Registers).
For each resource, we compare the baseline with different number of residual binarization levels in ResBinNet.
Asides from the DSP utilization, which is required for full-precision multiplications in batch normalization, the other three resources show a modest increase in utilization, meaning that the residual binarization method offers a scalable design for real-world systems.
Figure 7b compares the latency (runtime) of ResBinNet with the baseline accelerator.
In particular, we consider multi-level residual binarization with 1, 2, and 3 residual levels which are denoted by
RBN1, RBN2, and RBN
-10 task.
As can be seen, ResBinNet enables users to achieve higher accuracy by tolerating a higher latency, which is almost linear with respect to the number of residual levels.
[Figure]
R ELATED W ORK
Training CNNs with binary weights and/or activations has been the subject of very recent works (Courbariaux et al.
; Rastegari et al.
; Courbariaux et al.
; Umuroglu et al.
The authors of Binaryconnect (Courbariaux et al.
(2015)) suggest a probabilistic methodology that leverages the full-precision weights to generate binary representatives during forward pass while in the back-propagation the full-precision weights are updated.
(Courbariaux et al.
(2016)) is the ﬁrst work attempting to binarize both weight and activations of CNN.
In this work, authors also suggest replacing the costly dot products by XNOR-popcount operations.
XNOR-net (Rastegari et al.
(2016)) proposes to use scale factors during training, which results in an improved accuracy.
The aforementioned works propose optimization solutions that enable the use of binarized values in
CNNs which, in turn, enable the design of simple and efﬁcient hardware accelerators.
The downside of these works is that, aside from changing the architecture of the CNN (Umuroglu et al.
, they do not offer any other reconﬁgurability in their designs.
On another track of research, the reconﬁgurability of CNN accelerators has been investigated.
This line of research focuses on using adaptive low bit-width representations for compressing the parameters and/or simplifying the pertinent arithmetic operations (Zhou et al.
; Han et al.
; Wu et al.
; Cai et al.
The proposed solutions, however, do not enjoy the same simpliﬁed
XNOR-popcount operations as in binarized CNNs.
Among the aforementioned works, a uniﬁed solution which is both reconﬁgurable and binarized is missing.
To the best of our knowledge, ResBinNet is the ﬁrst to offer a solution which is reconﬁgurable and, at the same time, enjoys the beneﬁts of binarized CNNs.
Our goal in the design of
ResBinNet was to remain consistent with the existing CNN optimization solutions.
As shown in the paper, ResBinNet is compatible with the accelerators designed for binarized CNNs.
C ONCLUSION
This paper introduces ResBinNet, a novel reconﬁgurable binarization scheme which aims to improve the convergence rate and the ﬁnal accuracy of binary CNNs.
The proposed training is twofold: (i)
In the ﬁrst phase, called soft binarization, we introduce two distinct methodologies designed for binarizing weights and feature within CNNs, namely residual binarization, and temperature adjustment.
Residual binarization learns a multi-level representation for features of CNN to provide an arbitrary numerical precision during inference.
Temperature adjustment gradually imposes binarization constraints on the weights.
(ii)
In the second phase, which we call hard binarization, the model is ﬁne-tuned in few training epochs.
Our experiments demonstrate that the joint use of residual binarization and temperature adjustment improves the convergence rate and the accuracy of the binarized CNN.
We argue that ResBinNet methodology can be adopted by current CNN hardware accelerators as it requires minimal modiﬁcation to existing binarized CNN solutions.
Developers can integrate the approaches proposed in this paper into their deep learning systems to provide users with a trade-off between application latency and inference accuracy.