R ECURSIVE B INARY N EURAL N ETWORK L EARNING M ODEL WITH 2- BIT / WEIGHT S TORAGE R EQUIREMENT
[-Author 1 1,2 , Author2 2 , Author3 
 Affiliation1 , 2 Affiliation address-]Anonymous authors Paper under double-blind review
A BSTRACT
This paper presents a storage-efﬁcient learning model titled Recursive Binary
Neural Networks for embedded and mobile devices having a limited amount of on-chip data storage such as hundreds of kilo-Bytes.
The main idea of the proposed model is to recursively recycle data storage of synaptic weights (parameters) during training.
This enables a device with a given storage constraint to train and instantiate a neural network classiﬁer with a larger number of weights on a chip, achieving better classiﬁcation accuracy.
Such efﬁcient use of on-chip storage reduces off-chip storage accesses, improving energy-efﬁciency and speed of training.
We veriﬁed the proposed training model with deep neural network classiﬁers and the permutation-invariant MNIST benchmark.
Our model achieves data storage requirement of as low as 2 bits/weight while the conventional binary neural network learning models require data storage of 8 to 16 bits/weight.
With same amount of data storage, our model can train a bigger network having more weights, achieving 1% better classiﬁcation accuracy than the conventional binary neural network learning model.
To achieve the similar classiﬁcation error, the conventional binary neural network model requires 3-4 × more data storage for weights than our proposed model.
I NTRODUCTION
Deep Neural Networks (DNN) have demonstrated the state-of-the-art results in a wide range of cognitive workloads such as computer version Krizhevsky et al. and speech recognition Hinton et al. (2012), achieving better-than-human performance for the tasks often considered too complex for machines.
The success of DNN has indeed motivated scientists and engineers to implement a DNN in mobile and embedded devices, dubbed as Internet of Smart Things Kortuem et al. (2010).
The recent works in this area however, mostly implement inference function of DNN, without function of training DNN, while training DNN is done in cloud computers and post-training weights are downloaded to mobile and embedded devices Lane et al.
On-device learning, however, becomes increasingly important for the mobile and embedded devices for the following three reasons.
First, intelligent device beneﬁts to have the model that is custombuilt for the device and its user and environment.
This is because the model tends to be more accurate and effective if constructed with the consideration of those factors.
Second, the training data from mobile and embedded sensing devices can contain security-sensitive information, e.g., personal health data from wearable medical devices.
At the risk of being leaked, users typically do not want to upload such data onto cloud computers.
Finally, in the era of Internet of Things (IoT), we anticipate a drastic increase in the number of deployed devices, which can proportionally increase the number of learning tasks to be done in the cloud.
Coupled with the complexity of learning, even for powerful cloud computers, this can be a computationally challenging task.
On-device learning, however entails various challenges, in algorithms, data, and systems Roschelle (2003); Vogel et al.
The most eminent challenge regarding computing systems is high energy consumption caused by dense computation and data access of DNN system which is prohibited to limited resources of embedded devices.
The high overhead of data access is caused by fetching DNN weights from DRAM (or FLASH) external to a computing chip on an embedded device.
Since the data storage size is limited for such computing chip, the parameters of a DNN have to be stored in external DRAM and FLASH during training.
For example, ARM Cortex M3 processor, a processor widely used in commercial wearable devices such as FitBit, has only 64 kilo-Byte (kB) on-chip data storage.
This can only ﬁt in very small size of DNN especially if each weight is 32-bit ﬂoat point number.
Compared to accessing on-chip SRAM, accessing off-chip DRAM incurs 3 to 4 orders of magnitudes more energy and delay overhead.
Therefore, fetching weights every time for each data makes training prohibitive to be implemented on a mobile and embedded device Han et al.
Recently several techniques on pruning, distilling, and binarizing weights have been proposed to compress the parameters of a DNN.
This makes it more feasible to ﬁt weights in on-chip SRAM
Han et al. (2015); Courbariaux et al.
; Rastegari et al.
; Hinton et al.
These techniques can also reduce computation overhead.
However, these works focused on weight size compression after training is ﬁnished .
The data storage requirement during training remains the same.
Similarly, several learning models, which belong to so-called binary neural networks, have been proposed Courbariaux et al.
; Rastegari et al.
These model uses sign bits (or binary information) of weights in several parts of the learning model notably the part of multiplying and accumulating weights with inputs/activations.
Although this greatly reduces computational complexity, each weight still needs to be represented in high precision number with multiple bits during the end-to-end training process since it has to be iteratively ﬁne-tuned in the weight update part.
Therefore, this so-called binary neural network models have not demonstrated to scale storage requirement below 32 bits/weight.
Our goal is, therefore, to efﬁciently use the limited amount of on-chip data storage during training.
We also aim to scale computational complexity.
Toward this goal, we propose a new learning model, Recursive Binary Neural Network (RBNN) .
This model is based on the process of training of a neural network, weight binarization, recycling storage of non-sign-bit portion of weights to add more weights to enlarge the neural network for performance improvement.
We recursively perform this process until either accuracy stop improving or we use up all the storage on a chip.
We veriﬁed the proposed RBNN model on a multi-layer perceptron (MLP)-like classiﬁer and the MNIST benchmark.
We considered typical storage constraints of embedded sensing devices in the order of hundreds of kB. The experiment conﬁrms that the proposed model (i) demonstrates 1% classiﬁcation accuracy improvement over the conventional BNN learning model speciﬁcally following Courbariaux et al. (2015) for the same storage constraints or (ii) scale on-chip data storage requirement by 4 × for the same classiﬁcation test error ( ∼ 2%), marking the storage requirement of 2 bits/weight.
The conventional BNN model (Courbariaux et al. but also Courbariaux et al. ; Rastegari et al. (2016)) exhibits a signiﬁcantly larger storage requirement of 8 tobits/weight.
The remainder of the paper is as follow.
In Sec.
2 we will introduce the works related to this paper, including comparison to existing works on DNN compression, BNN, and low-precision DNN.
In Sec.
3 we will describe the intuitive and details of our proposed model.
Sec. 4 will present the experimental results and comparisons to the conventional BNN model.
Finally, in Sec. 5, we will conclude the paper.
R ELATED W ORK 2. D ISTILLATION AND C OMPRESSION OF DNN PARAMETERS
Knowledge distillation Hinton et al. (2015) is a technique to compress knowledge of an ensemble of DNNs into one small DNN while maintaining the accuracy.
Although this technique can scale the number of weights for deployment systems post-training, it cannot scale data storage requirement for training.
Speciﬁcally, during training, each of weights is represented in high-precision number, which needs to be stored in multi-bit data storage.
Another technique is to compress the data size of weights by exploiting redundancies in them.
In Ref.
Han et al. , the authors combine four sub-techniques, namely weight pruning, quantization, sharing, and compression coding to reduce the data size of weights.
Similar to the knowledge distillation, this technique can be applied to the weights that are already trained, and cannot scale data storage requirement of weights during training.
2. B INARY N EURAL N ETWORK (BNN)
Recent works proposed to use binary information of weights Courbariaux et al. (2015), activations Courbariaux et al. ; Rastegari et al.
(2016), and even inputs Rastegari et al.
(2016) in some parts of learning and post-learning operations.
The use of binary information of weights notably in multiply-and-accumulate (MAC) operation can drastically reduce computational complexity.
However, those BNN techniques still cannot scale the storage requirement of weights during training.
In these works, each weight is represented in 32 bits.
This is because mainstream training models such as stochastic gradient decent requires ﬁne-grained update of weights.
2. L OW - PRECISION FIX - POINT WEIGHT REPRESENTATION
Several studies have demonstrated moderately lowering the precision of weights (i.e., quantization) has a tolerable impact on training and post-training operations of DNN Gupta et al. ; Courbariaux et al.
In Gupta et al.
, the authors trained a DNN having 16-bit ﬁxed-point weights with the proposed stochastic rounding technique, and demonstrated little to no degradation in classiﬁcation accuracy.
In Courbariaux et al.
, the authors proposed the dynamic ﬁxedpoint representation (i.e., dynamically changing the position of decimal point over computation sequences) to further reduce the precision requirement down to 10 bits per synapse.
Using ﬁxpoint representation help to reduce storage requirement and ﬁxed-point arithmetic is more hardware friendly Han et al.
R ECURSIVE B INARY N EURAL N ETWORK (RBNN) M ODEL 3. K EY IDEA
Table 1 shows which information of weights are used in each step of training in both conventional BNN Courbariaux et al.
; Rastegari et al.
and our proposed RBNN.
The conventional BNN works Courbariaux et al.
; Rastegari et al.
(2016) use sign bits of weights during multiply-and-accumulate (MAC) operation in feed-forward and back propagation.
However, the weight update has to be done in high precision.
This mandates to store multi-bit (32 bit if using ﬂoat point number) weights in data storage during learning, resulting in no savings in weight storage requirement.
However, it has been studied that in the trained neural networks we can use only the sign bits of weights to perform inference Courbariaux et al.
; Rastegari et al.
This vast different requirements of weight precision between learning and post-learning inspires us to create our RBNN model.
As shown in the third column in Table 1, we also use only the sign bits for MAC operations to reduce computational complexity.
The main difference is that after training and binarization of weights (keeping only sign bit), we recycle the data storages that are used to store non-sign bits of weights to add more multi-bit trainable weights to the neural network.
We then trained this new network having both the binarized non-trainable weights and the newly-added trainable weights.
We perform this process recursively, which makes the neural networks larger and more accurate but using the same amount of data storage for weights.
Figure 1 depicts the process of our proposed RBNN learning model with an example of multi-layer fully-connected neural network.
In the beginning the neural network has one input, two set of two hidden, and one output neurons and eight weights each of which has n bits.
We ﬁrst train this 1 × 2 × 2 × 1 network using the conventional back-propagation training algorithm.
After that, we discard all bits except the sign bit in each weight (binarization), resulting in a 1 × 2 × 2 × 1 trained network having binary weights ( trained BNN ).
Then we continue the second iteration of training (the second subﬁgure of Figure 1).
Speciﬁcally, we recycle the storage that is used to store the n-1 non-sign bits of weights in the 1 × 2 × 2 × 1 network.
Using this data storage, we add eight additional weights ( W 21 to W 28 ) to the trained BNN , expanding the network to a 1 × 4 × 4 × 1 .
In this enlarged BNN , each of the newly-added weights is n − 1 bits.
In other words, the enlarged BNN comprises of one trained BNN that has eight weight ( W b 11 to W b 18 ) that are trained (binary, nonplastic, marked as solid lines in Figure 1) and one incremental BNN with eight weights ( W 21 to W 28 ) that are under training (n-1 bits, plastic, marked as dash lines in 1).
The incremental BNN is trained together with the trained BNN but only the weights of incremental BNN are updated.
[Table]
We repeat the same process of binarization and recycling.
In every iteration, the enlarged BNN integrates 8 more weights, and the bit-width of newly-added plastic weights in the incremental BNN is reduced by one.
At the k iterations, the trained BNN has 8 · ( k − 1) neurons and the plastic weights have ( n − k + 1 ) bit-width.
After the k-th training is ﬁnished, as shown in the rightmost in Figurethe neural network becomes a 1 × 2 k × 2 k × 1 with 8 · k binary weights.
This network has k times more weights than the ﬁrst 1 × 2 × 2 × 1 network.
However, the data storage of weights remains the same, scaling the storage requirement per weight to n/k ( = 4 · n/ 4 · k ), which is k times smaller than that of the ﬁrst network.
Thus the proposed RBNN can either achieve better classiﬁcation accuracyenabled by the more number of weights - with the same amount of weight storage, or reduce weight storage requirement for the same classiﬁcation accuracy level.
[Table]
[Table]
W k ……
W k (n-k+1)-bit
W b W b
W b 1-bit
Network size 1-2k-2k- ……
W b21 ...
W b22 ...
W b28 ... 1-bit
W k
W k W k
W k W k
W k W k
W k
Network size 1-2k-2k- ……
W bk
W bk W bk
W bk W bk
W bk W bk
W bk ...
1-bit
W b W b
W b 1-bit
W b21 ... W b22 ...
W b28 ... 1-bit
W bk1 W bk
W bk
3. M ODEL DETAILS
Figure 2 depicts the details of the proposed RBNN model.
The left part of Figure 2 depicts the ﬂow chart of RBNN model, while right part explains the function of each stage.
In the beginning of the training procedure, conventional BNN training algorithm BNN Training is used to train a BNN.
After training, we got a trained BNN with binary synaptic weights.
The synaptic bit-width is reduced by 1.
And then we use the rest synaptic bits as weights of incremental BNN .
After training the incremental BNN with algorithm Incremental BNN Training , the performance of the enlarged BNN is tested.
If the performance doesn’t stop improving and there are still available synaptic bits after weight binarzation, the rest synaptic bits will be reused to further enlarge current trained BNN .
[Figure]
_Training
Bit_width_Reduction
Incremental_BNN_Training
Stop criteria met?
FALSE
END TRUE
Performance_evaluation BNN_Training : Training initial BNN using conventional BNN training method.
Bit-width_Reduction : Reduce bit-width of all synaptic weights by
Incremental_BNN_Training : Training incremental_BNN with previously Trained_ BNN Performance_evaluation: Test the performance of trained enlarged BNN The method Incremental BNN Training is designed to train the incremental BNN to improve performance of enlarged BNN .
To meet this goal, the conventional BNN training method is adjusted as shown in Algorithm 1.
The main idea of this training method is: both trained BNN and incremental BNN are used to calculate output of the enlarged network in feedforward stage.
While during back-propagation and parameter-update, only plastic weights in incremental BNN are updated.
Similar to the conventional BNN training algorithm Zhou et al. , binary weights are used in both feed forward and back propagation in Incremental BNN Training , to reduce computational overhead.
Since weights in trained BNN are binary, the multiplication related to weights can all be simpliﬁed as shift.
Algorithm 1 Incremental BNN Training .
C is the cost function for mini-batch, η the learning rate and L the number of layers.
The function Binarize() speciﬁes how to binarize the weights.
Act hid() and Act out() are activation function of hidden layers and output layer, respectively.
Require: a minibatch of inputs and targets ( a 0 , a ∗ ), previous weights of incremental BNN W ( I ) , weights of trained BNN W ( T )
Ensure: updated weights of incremental BNN W ( I ) ( t +1)
1. Forward Propagation 1.1 Computing outputs of hidden layers in trained BNN and incremental BNN for
k = 1 to L-1 do
[Equation] − 1) ) W ( I ) bk ← Binarize ( W ( I ) bk ) a ( I )
k = Act hid ( W ( I ) bk · a ( I ) ( k − 1) ) end for 1.2 Computing outputs of enlarged BNN
a L = Act out ( W ( T ) L · a ( T ) ( L − 1) + W ( I ) L · a ( I )
( L − 1) )
{ Please note that only gradients of incremental BNN are computed. }
Compute g aL = ∂C∂a
L knowing a L and a ∗ for k
= L to 1 do g W ( I ) bk
← ( g a ( I ) k ◦ a (cid:48)
( I ) k ) · ( W ( I ) bk ) · a ( I ) k −end for 3. Parameter Update Please note that only weights of incremental BNN are updated.
for k = L to 1 do
W ( I ) T +1 k ← W ( I ) tk + η · g WI
bk end for
E XPERIMENT S ETUP 4. P ERMUTATION - INVARIANT MNIST BENCHMARK
We used the permutation-invariant MNIST to test the performance of the proposed RBNN model.
We use the original training set of 60,000 28-by-28 pixel gray-scale images and the original test set of 10,000 images.
The training and testing data x is normalized to be within the interval [-1, 1] and exhibits zero mean.
Following the common practices, we use the last 10,000 images of the training set as a validation set for early stopping and model selection.
As we use the permutation-invariant MINST, i.e., ignoring the 2-dimentonal image structure of the image, we did not consider convolutional computation.
We also did not consider data augmentation, pre-processing, and unsupervised pre-training during our experiment.
4. N EURAL NETWORK CONFIGURATION AND DATA FORMAT
We considered the storage constraints of mainly hundreds of kB based on the typical embedded system designs Shiue & Chakrabarti (1999).
We considered a feed-forward fully-connected neural network with one or two hidden layers.
We considered several different numbers of neuron neuron units in the hidden layer ranging from 200 to 800.
The numbers of the input and output units are 784 and 10, respectively.
We used the tanh opt() for the activation function of the hidden layer and the softmax() or linear output for that of the output layer.
We use the classical Stochastic Gradient
Descent (SGD) algorithm for cross-entropy or hinge loss minimization without momentum.
We use a small size of batch (1,000) and a single static learning rate which is optimized for each BNN.
Any other advanced techniques such as dropout, Maxout, and ADAM, and etc are not used for both the proposed and the baseline learning models.
We recorded the best training and test errors associated with the best validation error after up to 1,000 epochs.
The results are averages of results fromindependent experiments for each case.
We used the ﬁxed-point arithmetic for all the computation and data load and access.
The ﬁx-point intermediate computations, such as gradient calculation have, also use ﬁxed-point arithmetic with sufﬁcient precision.
The translation from wide ﬁxed-point number to narrow ﬁxed-point and binary number is performed with simple decimation without using advanced techniques such as stochastic rounding Courbariaux et al.
We saturated values in the event of overﬂow in weight update.
The dynamic range of ﬁx-point representation is optimized to achieve better accuracy performance.
R ESULTS AND D ISCUSSION 5. A CCURACY IMPROVEMENT
Figure 3 depicts the classiﬁcation errors of the proposed learning model across three recursive iterations.
The initial bit-width of weights is 8 bits.
In each series of data points in Figure 3, the leftmost point is the initial neural network, i.e., with 2 layers of 200 hidden units and 198,800 synaptic weights ( = 784 · 200 + 200 · 200 + 200 · 10 ).
At this point, the synaptic bit requirement, deﬁned as the ratio of total storage bits to the number of weights, is 8-bit/weight.
At this point the network is equivalent to one trained with the conventional BNN model speciﬁcally following [9].
In the second leftmost data point in the series is the neural network after the ﬁrst recursive iteration.
The network size is enlarged by twice, resulting in the 784-400-400-10 network.
This reduces synaptic bit requirement to 4 bits/weight.
Compared to original BNN, the enlarged BNN respectively achieves ∼ 0.7% and ∼ 0.4% reduction in training and test error rate.
Finally, after three recursive iterations, the neural network implements BNN with size 784-800-800-10 (555,800 synaptic weights).
It requires only 198.8 kB data storage for weights, marking the weight storage requirement of 2b/weight and the test error of 2.17%.
This accuracy is as good as network with 4 × times of synaptic data storage in Courbariaux et al. 5. S TORAGE SAVINGS
We trained DNNs of a range of conﬁguration using our RBNN and the conventional BNN learning model Courbariaux et al.
We used ﬁx-point number for weights.
For the conventional model,
[Figure] [Figure]
qu
[Figure]
m e n t ( M
[Figure]
[Figure] we considered BNN containing from 100 to 800 hidden neurons combined with 16-b to 6-b synaptic weights.
For the proposed model, we considered from 100 to 800 initial hidden neurons and 12 to 16-bit initial weight precisions.
Those DNNs require 116 kB to 1.2 MB data storage for weights.
Figure 4 shows the results of this experiment: the proposed model can achieve 1% lower test error than the conventional model using the similar amount of data storage.
To achieve the similar test error, the proposed RBNN model requires 4 × less data storage than the conventional BNN model.
It is noticed that the accuracy improvement is not as signiﬁcant when hidden neurons are as many as 800.
We observe that at this point, the training data has been modeled well (0 training error and very small cost function value), so we think at this point, regularization technique is needed to further improve performance, which is beyond the scope of this paper.
Table 2 shows the detail comparisons of six neural networks, three from the proposed RBNN model ( R 1 , R 2 , R 3 ) and three from the conventional BNN model ( B 1 , B 2 , B 3 ) Courbariaux et al.
The complexity of computation for learning and inferring one data is also listed.
R 1 and B 1 , Rand B 2 achieve similar test error, but R 1 outperforms B1 in the storage requirement by 3 × and R
[Table]
114kB outperforms B 2 by 4 × , respectively.
The downside of R 1 and R 3 is the increase in computations during training: it requires more shift and add operations, but it still needs the same number of multiplications which is much more complex than add and shift, as compared to B 1 and B 2 .
On the other hand, the computation complexity for inference operation are the same.
5. D ISCUSSION ON C ONVOLUTIONAL N EURAL N ETWORKS
Although we mostly focus on a fully-connected DNN in this paper
[Table]
we want to make a brief discussion on the application of our proposed RBNN model on the Convolutional Neural Networks (CNN).
A CNN employs many convolution layers, each of which uses a smaller number of weights than a fully-connected one.
But a convolution layer uses the same weights many times.
This makes the operation of convolutional layers dominated by multiply and add operation rather than weight data storage operation.
Our RBNN model can still mitigate the complexity of multiply and add operations since it uses only the sign bit of a weight.
Moreover, as the computational complexity of convolution layers is scaled, the computation and data-storage overhead associated with the last fully-connected layers in a CNN becomes more pronounced.
Our proposed model can reduce the overhead of weight data storage of them.
We are currently exploring the application of our RBNN model on a CNN.
C ONCLUSION AND F UTURE W ORK
This paper presents a learning model regarding local on-device learning with limited data on-chip storage.
The proposed RBNN model efﬁciently uses limited on-chip data storage resources by recycling data storage that would have been wasted, to add and train more synaptic weights to a neural network classiﬁer.
We veriﬁed the proposed model with the neural network classiﬁer and the permutation-invariant MNIST benchmark under the typical embedded device storage constraints.
The results show that the proposed model achieves 2b/weight storage requirement while achieving 1% better classiﬁcation error as compared to the conventional binary-weight learning model for the same storage constraint.
Our proposed model also achieves 3-4 × less data storage than the conventional model for the same classiﬁcation error.
We expect the future work that extends the application of the learning model to other neural network topologies and datasets, such as BNN with binary activation function Courbariaux et al. (2016), and also more complicated model like CNN.
We also expect to apply the RBNN model to the ensembles of neural networks Zhou et al. (2002), the mixture of experts Shazeer et al. (2017), and the incremental learning Xiao et al.