R ECURSIVE B INARY N EURAL N ETWORK L EARNING M ODEL WITH 2- BIT / WEIGHT S TORAGE R EQUIREMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper presents a storage-efﬁcient learning model titled Recursive Binary
Neural Networks for embedded and mobile devices having a limited amount of on-chip data storage such as hundreds of kilo-Bytes.
The main idea of the proposed model is to recursively recycle data storage of synaptic weights (parameters) during training.
This enables a device with a given storage constraint to train and instantiate a neural network classiﬁer with a larger number of weights on a chip, achieving better classiﬁcation accuracy.
Such efﬁcient use of on-chip storage reduces off-chip storage accesses, improving energy-efﬁciency and speed of training.
We veriﬁed the proposed training model with deep neural network classiﬁers and the permutation-invariant MNIST benchmark.
Our model achieves data storage requirement of as low as 2 bits/weight while the conventional binary neural network learning models require data storage of 8 to 16 bits/weight.
With same amount of data storage, our model can train a bigger network having more weights, achieving 1% better classiﬁcation accuracy than the conventional binary neural network learning model.
To achieve the similar classiﬁcation error, the conventional binary neural network model requires 3-4 × more data storage for weights than our proposed model.
I NTRODUCTION
Deep Neural Networks (DNN) have demonstrated the state-of-the-art results in a wide range of cognitive workloads such as computer version Krizhevsky et al.
and speech recognition (Hinton et al.
(2012)), achieving better-than-human performance for the tasks often considered too complex for machines.
The success of DNN has indeed motivated scientists and engineers to implement a DNN in mobile and embedded devices, dubbed as Internet of Smart Things (Kortuem et al. (2010)).
The recent works in this area however, mostly implement inference function of DNN, without function of training DNN, while training DNN is done in cloud computers and post-training weights are downloaded to mobile and embedded devices (Lane et al.
On-device learning, however, becomes increasingly important for the mobile and embedded devices for the following three reasons.
First, intelligent device beneﬁts to have the model that is custombuilt for the device and its user and environment.
This is because the model tends to be more accurate and effective if constructed with the consideration of those factors.
Second, the training data from mobile and embedded sensing devices can contain security-sensitive information, e.g., personal health data from wearable medical devices.
At the risk of being leaked, users typically do not want to upload such data onto cloud computers.
Finally, in the era of Internet of Things (IoT), we anticipate a drastic increase in the number of deployed devices, which can proportionally increase the number of learning tasks to be done in the cloud.
Coupled with the complexity of learning, even for powerful cloud computers, this can be a computationally challenging task.
On-device learning, however entails various challenges, in algorithms, data, and systems (Roschelle (2003); Vogel et al.
The most eminent challenge regarding computing systems is high energy consumption caused by dense computation and data access of DNN system which is prohibited to limited resources of embedded devices.
The high overhead of data access is caused by fetching
DNN weights from DRAM (or FLASH) external to a computing chip on an embedded device.
Since the data storage size is limited for such computing chip, the parameters of a DNN have to be stored in external DRAM and FLASH during training.
For example, ARM Cortex M3 processor, a processor widely used in commercial wearable devices such as FitBit, has only 64 kilo-Byte (kB) on-chip data storage.
This can only ﬁt in very small size of DNN especially if each weight is 32-bit ﬂoat point number.
Compared to accessing on-chip SRAM, accessing off-chip DRAM incurs 3 to 4 orders of magnitudes more energy and delay overhead.
Therefore, fetching weights every time for each data makes training prohibitive to be implemented on a mobile and embedded device (Han et al.
Recently several techniques on pruning, distilling, and binarizing weights have been proposed to compress the parameters of a DNN.
This makes it more feasible to ﬁt weights in on-chip SRAM (Han et al.
; Courbariaux et al.
; Rastegari et al.
; Hinton et al.
These techniques can also reduce computation overhead.
However, these works focused on weight size compression after training is ﬁnished .
The data storage requirement during training remains the same.
Similarly, several learning models, which belong to so-called binary neural networks, have been proposed (Courbariaux et al.
; Rastegari et al.
These model uses sign bits (or binary information) of weights in several parts of the learning model notably the part of multiplying and accumulating weights with inputs/activations.
Although this greatly reduces computational complexity, each weight still needs to be represented in high precision number with multiple bits during the end-to-end training process since it has to be iteratively ﬁne-tuned in the weight update part.
Therefore, this so-called binary neural network models have not demonstrated to scale storage requirement below 32 bits/weight.
Our goal is, therefore, to efﬁciently use the limited amount of on-chip data storage during training.
We also aim to scale computational complexity.
Toward this goal, we propose a new learning model,
Recursive Binary Neural Network (RBNN) .
This model is based on the process of training of a neural network, weight binarization, recycling storage of non-sign-bit portion of weights to add more weights to enlarge the neural network for performance improvement.
We recursively perform this process until either accuracy stop improving or we use up all the storage on a chip.
We veriﬁed the proposed RBNN model on a multi-layer perceptron (MLP)-like classiﬁer and the
MNIST benchmark.
And also convolutional neural networks (CNN) and voice activity detection (VAD) tasks. We considered typical storage constraints of embedded sensing devices in the order of hundreds of kB. The experiment in MLP-like classiﬁer conﬁrms that the proposed model (i) demonstrates 1% classiﬁcation accuracy improvement over the conventional BNN learning model speciﬁcally following Courbariaux et al.
(2015) for the same storage constraints or (ii) scale on-chip data storage requirement by 4 × for the same classiﬁcation test error ( ∼ 2%), marking the storage requirement of 2 bits/weight.
The conventional BNN model Courbariaux et al.
but also
Courbariaux et al.
; Rastegari et al.
(2016) exhibits a signiﬁcantly larger storage requirement of 8 to 32 bits/weight.
The experiment of CNN classifying MNIST proves as high as 6 × saving of on-chip data storage requirement and 2.4% accuracy improvement.
For VAD tasks, the proposed
RBNN proves 9 × data storage saving.
The remainder of the paper is as follow.
In Sec.
2 we will introduce the works related to this paper, including comparison to existing works on DNN compression, BNN, and low-precision DNN.
In Sec.
3 we will describe the intuitive and details of our proposed model.
Sec. 4 will present the experimental results and comparisons to the conventional BNN model.
Finally, in Sec. 5, we will conclude the paper.
R ELATED W ORK 2. D ISTILLATION AND C OMPRESSION OF DNN PARAMETERS
Knowledge distillation Hinton et al.
(2015) is a technique to compress knowledge of an ensemble of
DNNs into one small DNN while maintaining the accuracy.
Although this technique can scale the number of weights for deployment systems post-training, it cannot scale data storage requirement for training.
Speciﬁcally, during training, each of weights is represented in high-precision number, which needs to be stored in multi-bit data storage.
Another technique is to compress the data size of weights by exploiting redundancies in them.
In
Han et al. (2015), the authors combine four sub-techniques, namely weight pruning, quantization, sharing, and compression coding to reduce the data size of weights.
Similar to the knowledge distillation, this technique can be applied to the weights that are already trained, and cannot scale data storage requirement of weights during training.
2. B INARY N EURAL N ETWORK (BNN)
Recent works proposed to use binary information of weights Courbariaux et al.
(2015), activations
Courbariaux et al.
; Rastegari et al.
(2016), and even inputs Rastegari et al.
(2016) in some parts of learning and post-learning operations.
The use of binary information of weights notably in multiply-and-accumulate (MAC) operation can drastically reduce computational complexity.
However, those BNN techniques still cannot scale the storage requirement of weights during training.
In these works, each weight is represented in 32 bits.
This is because mainstream training models such as stochastic gradient decent requires ﬁne-grained update of weights.
2. L OW - PRECISION FIX - POINT WEIGHT REPRESENTATION
Several studies have demonstrated moderately lowering the precision of weights (i.e., quantization) has a tolerable impact on training and post-training operations of DNN (Gupta et al.
; Courbariaux et al.
In Gupta et al.
, the authors trained a DNN having 16-bit ﬁxed-point weights with the proposed stochastic rounding technique, and demonstrated little to no degradation in classiﬁcation accuracy.
In Courbariaux et al.
, the authors proposed the dynamic ﬁxedpoint representation (i.e., dynamically changing the position of decimal point over computation sequences) to further reduce the precision requirement down to 10 bits per synapse.
Using ﬁxpoint representation help to reduce storage requirement and ﬁxed-point arithmetic is more hardware friendly (Han et al.
R ECURSIVE B INARY N EURAL N ETWORK (RBNN) M ODEL 3. K EY IDEA
Table 1 shows which information of weights are used in each step of training in both conventional
BNN Courbariaux et al.
; Rastegari et al.
and our proposed RBNN.
The conventional BNN works Courbariaux et al.
; Rastegari et al.
(2016) use sign bits of weights during multiply-and-accumulate (MAC) operation in feed-forward and back propagation.
However, the weight update has to be done in high precision.
This mandates to store multi-bit (32 bit if using ﬂoat point number) weights in data storage during learning, resulting in no savings in weight storage requirement.
However, it has been studied that in the trained neural networks we can use only the sign bits of weights to perform inference (Courbariaux et al.
; Rastegari et al.
This vast different requirements of weight precision between learning and post-learning inspires us to create our RBNN model.
As shown in the third column in Table 1, we also use only the sign bits for
MAC operations to reduce computational complexity.
The main difference is that after training and binarization of weights (keeping only sign bit), we recycle the data storages that are used to store non-sign bits of weights to add more multi-bit trainable weights to the neural network.
We then trained this new network having both the binarized non-trainable weights and the newly-added trainable weights.
We perform this process recursively, which makes the neural networks larger and more accurate but using the same amount of data storage for weights.
Figure 1 depicts the process of our proposed RBNN learning model with an example of multi-layer fully-connected neural network.
In the beginning the neural network has one input, two set of two hidden, and one output neurons and eight weights each of which has n bits.
We ﬁrst train this 1 × 2 × 2 × 1 network using the conventional back-propagation training algorithm.
After that, we discard all bits except the sign bit in each weight (binarization), resulting in a 1 × 2 × 2 × 1 trained network having binary weights ( trained BNN ).
Then we continue the second iteration of training (the second subﬁgure of Figure 1).
Speciﬁcally, we recycle the storage that is used to store the n-1 non-sign bits of weights in the 1 × 2 × 2 × 1 network.
Using this data storage, we add eight additional weights ( W 21 to W 28 ) to the trained BNN , expanding the network to a 1 × 4 × 4 × 1 .
In this enlarged BNN , each of the newly-added weights is n − 1 bits. In other words, the enlarged BNN [Table] A
Keep sign bits and recycle [Table] of the other bits for more plastic weights comprises of one trained BNN that has eight weight ( W b 11 to W b 18 ) that are trained (binary, nonplastic, marked as solid lines in Figure 1) and one incremental BNN with eight weights ( W 21 to
W 28 ) that are under training (n-1 bits, plastic, marked as dash lines in 1).
The incremental BNN is trained together with the trained BNN but only the weights of incremental BNN are updated.
We repeat the same process of binarization and recycling.
In every iteration, the enlarged BNN integrates 8 more weights, and the bit-width of newly-added plastic weights in the incremental BNN is reduced by one.
At the k iterations, the trained BNN has 8 · ( k − 1) neurons and the plastic weights have ( n − k + 1 ) bit-width.
After the k-th training is ﬁnished, as shown in the rightmost in Figurethe neural network becomes a 1 × 2 k × 2 k × 1 with 8 · k binary weights.
This network has k times more weights than the ﬁrst 1 × 2 × 2 × 1 network.
However, the data storage of weights remains the same, scaling the storage requirement per weight to n/k ( = 4 · n/ 4 · k ), which is k times smaller than that of the ﬁrst network.
Thus the proposed RBNN can either achieve better classiﬁcation accuracyenabled by the more number of weights - with the same amount of weight storage, or reduce weight storage requirement for the same classiﬁcation accuracy level. W W W
W W W W
W W W…… Wn-bit
W b
W b W b W b W b15 W b
W b W b W
W…… W(n-1)-bit W W
W
W
W
W
W
W
W b
W b … …
W b 1-bit Network size 1-2-2Network size 1-4-4W k W k …… W k (n-k+1)-bit
W b W b W b 1-bit
Network size 1-2k-2k- ……
W b21 ... W b22 ...
W b28 ... 1-bit
W k
W k W k
W k W k
W k W k
W k
Network size 1-2k-2k- ……
W bk W bk
W bk W bk W bk
W bk W bk
W bk ...
1-bit
W b W b
W b 1-bit
W b21 ... W b 22 ...
W b28 ... 1-bit
W {+bk
+} W b k
W bk
3. M ODEL DETAILS
Figure 2 depicts the details of the proposed RBNN model.
The left part of Figure 2 depicts the ﬂow chart of RBNN model, while right part explains the function of each stage.
In the beginning of the training procedure, conventional BNN training algorithm BNN Training is used to train a
BNN.
After training, we got a trained BNN with binary synaptic weights.
The synaptic bit-width is reduced by 1.
And then we use the rest synaptic bits as weights of incremental BNN .
After training the incremental BNN with algorithm Incremental BNN Training , the performance of the enlarged BNN is tested.
If the performance doesn’t stop improving and there are still available synaptic bits after weight binarization, the rest synaptic bits will be reused to further enlarge current trained BNN . Notice that in each iteration of the proposed RBNN model, it requires to only update current plausible synaptic bits while keep the trained synaptic bits untouched.
The implementation of this sub-word operation of synaptic weights are illustrated in A
[Figure] Training
Stop criteria met?
FALSE
END TRUE
Performance_evaluation
BNN_Training :
Training initial
BNN using conventional BNN training method.
Bit-width_Reduction : Reduce bit-width of all synaptic weights by
Incremental_BNN_Training : Training incremental_BNN with previously Trained_
BNN
Performance_evaluation: Test the performance of trained enlarged BNN
The method Incremental BNN Training is designed to train the incremental BNN to improve performance of enlarged BNN .
To meet this goal, the conventional BNN training method is adjusted as shown in Algorithm 1.
The main idea of this training method is: both trained BNN and incremental BNN are used to calculate output of the enlarged network in feedforward stage.
While during back-propagation and parameter-update, only plastic weights in incremental BNN are updated.
Similar to the conventional BNN training algorithm Zhou et al.
, binary weights are used in both feed forward and back propagation in Incremental BNN Training , to reduce computational overhead.
Since weights in trained BNN are binary, the multiplication related to weights can all be simpliﬁed as shift.  E XPERIMENT S ETUP
In this and next section, the detailed experiment setup and results for MNIST classiﬁcation with MLP-like DNN systems using the proposed RBNN model are discussed. Furthermore, the proposed RBNN model is also applied to Convolutional Neural Network (CNN) classifying MNIST and MLPlike DNN doing Voice Activity Detection tasks.
The setup and results are discussed in B and C, respectively.
4. P ERMUTATION - INVARIANT MNIST BENCHMARK
We used the permutation-invariant MNIST to test the performance of the proposed RBNN model.
We use the original training set of 60,000 28-by-28 pixel gray-scale images and the original test set of 10,000 images.
The training and testing data are normalized to be within the interval [-1, 1] and exhibits zero mean.
Following the common practices, we use the last 10,000 images of the training set as a validation set for early stopping and model selection.
As we use the permutation-invariant
MINST, i.e., ignoring the 2-dimentional image structure of the image, we did not consider convolutional computation.
We also did not consider data augmentation, pre-processing, and unsupervised pre-training during our experiment.
Algorithm 1 Incremental BNN Training .
C is the cost function for mini-batch, η the learning rate and L the number of layers.
The function Binarize() speciﬁes how to binarize the weights.
Act hid() and Act out() are activation function of hidden layers and output layer, respectively.
Require: a minibatch of inputs and targets ( a 0 , a ∗ ), previous weights of incremental BNN W ( I ) , weights of trained BNN W ( T )
Ensure: updated weights of incremental BNN W ( I ) ( t +1)
1. Forward Propagation 1.1 Computing outputs
hidden incremental BNN
for k L do
[Equation]
end for
[Equation]
2. Back propagation
{ Please note that only gradients of incremental BNN are computed. }
Compute g aL = ∂C ∂a L knowing a L and a ∗ for k
= L to 1 do [Equation]
end for
3. Parameter Update
Please note that only weights of incremental BNN are updated.
for k = L to 1 do
[Equation]
end for
4. N EURAL NETWORK CONFIGURATION AND DATA FORMAT
We considered the storage constraints of mainly hundreds of kB based on the typical embedded system designs (Shiue & Chakrabarti (1999)).
We considered a feed-forward fully-connected neural network with one or two hidden layers.
We considered several different numbers of neuron neuron units in the hidden layer ranging from 200 to 800.
The numbers of the input and output units are 784 and 10, respectively.
We used the tanh opt() for the activation function of the hidden layer and the softmax() or linear output for that of the output layer.
We use the classical Stochastic Gradient
Descent (SGD) algorithm for cross-entropy or hinge loss minimization without momentum.
We use a small size of batch (1,000) and a single static learning rate which is optimized for each BNN.
Any other advanced techniques such as dropout, Maxout, and ADAM, and etc are not used for both the proposed and the baseline learning models.
We recorded the best training and test errors associated with the best validation error after up to 1,000 epochs.
The results are averages of results fromindependent experiments for each case.
We used the ﬁxed-point arithmetic for all the computation and data load and access.
The ﬁx-point intermediate computations, such as gradient calculation have, also use ﬁxed-point arithmetic with sufﬁcient precision.
The translation from wide ﬁxed-point number to narrow ﬁxed-point and binary number is performed with simple decimation without using advanced techniques such as stochastic rounding (Courbariaux et al.
We saturated values in the event of overﬂow in weight update.
The dynamic range of ﬁx-point representation is optimized to achieve better accuracy performance.
R ESULTS AND D ISCUSSION 5. A CCURACY IMPROVEMENT
Figure 3 depicts the classiﬁcation errors of the proposed learning model across three recursive iterations.
The initial bit-width of weights is 8 bits.
In each series of data points in Figure 3, the leftmost point is the initial neural network, i.e., with 2 layers of 200 hidden units and 198,800 synaptic weights ( = 784 · 200 + 200 · 200 + 200 · 10 ).
At this point, the synaptic bit requirement, deﬁned as the ratio of total storage bits to the number of weights, is 8-bit/weight.
At this point the network is equivalent to one trained with the conventional BNN model speciﬁcally following (Courbariaux et al.
In the second leftmost data point in the series is the neural network after the ﬁrst recursive iteration.
The network size is enlarged by twice, resulting in the 784 × 400 × 400 × 10 network.
This reduces synaptic bit requirement to 4 bits/weight.
Compared to original BNN, the enlarged BNN respectively achieves ∼ 0.7% and ∼ 0.4% reduction in training and test error rate.
Finally, after three recursive iterations, the neural network implements BNN with size 784 × 800 × 800 × 10 (555, synaptic weights).
It requires only 198.8 kB data storage for weights, marking the weight storage requirement of 2b/weight and the test error of 2.17%.
This accuracy is as good as fully-connected network with 4 × times of synaptic data storage in (Courbariaux et al.
In this experiment the structure of the generated network is as shown in Figure 1, i.e. the hidden neurons are not fully connected.
The performance of this structure is as good as fully-connected version generated by the proposed RBNN model.
The experiment will be discussed in D.
[Figure] Weight storage efficiency [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Figure]
[Figure]
[Table]
[Table]
We trained DNNs of a range of conﬁguration using our RBNN and the conventional BNN learning model (Courbariaux et al.
We used ﬁx-point number for weights.
For the conventional model, we considered BNN containing from 100 to 800 hidden neurons combined with 16-b to 6-b synaptic weights.
For the proposed model, we considered from 100 to 800 initial hidden neurons and 12 to 16-bit initial weight precisions.
Those DNNs require 116 kB to 1.2 MB data storage for weights.
Figure 4 shows the results of this experiment: the proposed model can achieve 1% lower test error than the conventional model using the similar amount of data storage.
To achieve the similar test error, the proposed RBNN model requires 4 × less data storage than the conventional
BNN model.
It is noticed that the accuracy improvement is not as signiﬁcant when hidden neurons are as many as 800.
We observe that at this point, the training data has been modeled well ( training error and very small cost function value), so we think at this point, regularization technique is needed to further improve performance, which is beyond the scope of this paper.
Table 2 shows the detail comparisons of six neural networks, three from the proposed RBNN model ( R 1 , R 2 , R 3 ) and three from the conventional BNN model ( B 1 , B 2 , B 3 )
(Courbariaux et al.
The arithmetic complexity for learning and inferring is also listed.
For conventional BNN, the amount of arithmetic of learning one data sample for one epoch is shown.
While for RBNN, we add up arithmetic amount for all binarization-recycle iterations of training.
R 1 and B 1 , R 3 and B 2 achieve similar test error, but R 1 outperforms B1 in the storage requirement by 3 × and R 3 outperforms Bby 4 × , respectively.
The downside of R 1 and R 3 is the increase in computations during training: it requires more shift and add operations, but it still needs the same number of multiplications which is much more complex than add and shift, as compared to B 1 and B 2 .
On the other hand, the computation complexity for inference operation are the same.
5. E NERGY C ONSUMPTION S AVING
The last three rows of Table 2 compare the training energy with respect to both computation and data access for RBNN model and conventional BNN. The training energy is deﬁned as the sum of energy cost of arithmetic and data storage access for synaptic weights.
The energy consumption is calculated based on training arithmetic complexity and energy cost of basic arithmetic and memory operations in a 45nm CMOS process shown in Table 3.
The original value of energy cost from Han et al.
(2016)is scaled squarely with bit-width to calculate energy cost of multiplication and linearly to calculate other arithmetic and storage access operations.
The results in Table 2 show that to achieve same performance level, the proposed RBNN saves around 100 × total energy for training ( R 1 and B 1 , R 3 and B 2 ), even though it requires slightly more computation.
This is because RBNN reduces the data storage requirement for on-chip learning, so the data can be easily ﬁt in on-chip SRAM of chips for low-cost application. On the other hand, for conventional BNN, the weights have to store and fetched from off-chip DRAM during training (Altera), which costs 2-order of magnitudes more energy than SRAM access and basic arithmetic according to Table 3.
C ONCLUSION AND F UTURE W ORK
This paper presents a learning model regarding local on-device learning with limited data on-chip storage.
The proposed RBNN model efﬁciently uses limited on-chip data storage resources by recycling data storage that would have been wasted, to add and train more synaptic weights to a neural network classiﬁer.
We veriﬁed the proposed model with the neural network classiﬁer and the permutation-invariant MNIST benchmark under the typical embedded device storage constraints.
The results show that the proposed model achieves 2b/weight storage requirement while achieving 1% better classiﬁcation error as compared to the conventional binary-weight learning model for the same storage constraint.
Our proposed model also achieves 3-4 × less data storage than the conventional model for the same classiﬁcation error.
We expect the future work that extends the application of the learning model to other neural network topologies and datasets, such as BNN with binary activation function Courbariaux et al.
(2016), and also more complicated model like CNN.
We also expect to apply the RBNN model to the ensembles of neural networks (Zhou et al.
(2002)), the mixture of experts (Shazeer et al.
(2017)), and the incremental learning (Xiao et al.
A I MPLEMENTATION OF S UB - WORD O PERATION OF S YNAPTIC W EIGHTS
In the proposed RBNN model, during training only plausible synaptic bits can be updated, while ﬁxed (trained) synaptic bits need to maintain their value.
Figure 5 illustrates the implementation of this sub-word operation of synaptic bits.
An 8-bit synaptic weight is used in Figure 5 as an example of which 3 bits are ﬁxed (trained) synaptic bits and 5 bits are plastic bits that are under training in current iteration.
At the beginning of training procedure, one complete weight synp in is fetched from weight storage to computation circuits.
And then a signal synp mask which has 1 at the position of ﬁxed weight bits and 0 at the position of plastic synaptic bits (11100000 in this example) and its reverse type (00011111) are AND ed to the weight to separate ﬁxed weight synp ﬁx and plastic weight synp plsb , respectively.
During the training procedure, synp plsb is updated while synp ﬁx maintains the same value.
Once training is ﬁnished, the updated synp plsb and initial synp ﬁx are concatenated through XOR operation.
Compared with conventional BNN, the proposed RBNN only requires one AND operation at the beginning and one XOR operation at the end of each training epoch, the cost of which is much smaller than cost of arithmetic and data access of training process.
[Figure]
2 % ~ 4 X > 6 X
C o n v e n t
i
o n a l B N N P r o p
o s e
d R B N N
W e i g h t s t
o r a g
e s i
z e ( k B )
T e s t e r r o r ( % ) [ 2 5 0 0 ]
B A PPLICATION OF RBNN TO C ONVOLUTIONAL N EURAL N ETWORK B. E XPERIMENT S ETUP
We apply the proposed RBNN model to LeNet5 Convolutional Neural Network (LeCun et al.
with two convolution layers, having 6 and 12 of 5 by 5 feature maps, respectively.
Both of them are followed by a 4x dowsampling average pooling layer.
The fully-connected (FC) layers consist of one input, one hidden and one output layer.
As in LeCun et al.
(1998), binary weights are used for all convolutional and FC layers during forward and backward propagation and ﬁx-point
[Table]
weights are used for weight updating to reduce computation overhead of training.
The proposed
RBNN model is applied to increase the size of hidden layer of the FC layers.
B. R ESULTS A NALYSIS
We trained CNNs of a range of conﬁguration for FC layers to classify MNIST dataset using the proposed RBNN model.
And compare the accuracy with the CNN trained with conventional BNN method (Courbariaux et al.
For the proposed model, we considered from 200 to 800 initial hidden neurons and 16-bit initial weight precision.
For the conventional model, we considered
CNNs with FC layers containing from 200 to 2500 hidden neurons with 16-bit synaptic weights.
The CNNs require 81kB to 1.01MB data storage for weights of all convolutional and FC layers.
Figure 6 compares the accuracy performance and the according data storage size for CNNs with various conﬁguration with both the proposed RBNN model and conventional BNN model.
The results prove that the proposed RBNN model can achieve 2.4% lower test error than the conventional model using the same amount of data storage.
To achieve the similar test error, the proposed RBNN model requires 7X less data storage than the conventional BNN model.
C A PPLICATION OF RBNN TO V OICE A CTIVITY D ETECTION T ASKS
To further validate the proposed RBNN model, we apply it to MLP-like DNNs to do Voice Activity
Detection tasks.
The structure of DNN used in this experiment is the same as the one in 4.2.
The experiments were conducted on Aurora 4 (Pearce & Picone (2002)), which comprising 7133 utterances from 83 speakers.
The basic feature used for the DNN to detect is 5 frames of 16-dimensional band-pass ﬁlterbank output.
The signal-to-noise ratio of data in all scenarios used in the experiment is 10dB. Table 4 shows the topology and performance of DNNs using both RBNN and conventional
BNN model for VAD tasks in various scenarios and proves as high as 9X data storage saving to achieve same level of detection accuracy.
D RBNN IN FULLY - CONNECTED DNN SYSTEMS
In this section we demonstrate one way of using the proposed RBNN to generate a fully-connected
DNN and show the performance improvement compared to conventional BNN and tilted topology shown in Section 3.1.
Figure 7 illustrates how the fully-connected DNN is built step by step with an example of DNN with initial size being 1-2-2-1 and n-bit.
Firstly, new hidden neurons are added and the available weight storage is used for connection between input, newly added hidden, and output neurons.
As shown in the second sub-ﬁgure, after k iterations, the topology becomes 1-2k-2k-1.
Notice that the hidden neurons generated in different iterations are not connected with each other at this point.
And then in the following iterations, the available weight storage is used to connect hidden layers generated in different iterations, adding m new weights each time, until all the neurons in the DNN system are connected with each other.
The value of k and m is determined by size of available data storage, and the performance improvement of current iteration.
To evaluate the performance of fully-connected DNN generated with the proposed RBNN model,
Figure 8 shows the classiﬁcation performance on MNIST dataset of a 784-800-800-10 fully connect- ed DNN generated from a 784-200-200-10 DNN with 8-bit initial synaptic bit-width.
The test error rate at each iteration is shown, as well as the topology and plausible synaptic bit-width.
According to Figure 8, the performance increases as more hidden neurons are added, while almost remains the same when more weights are added to connect existing hidden neurons.
The possible explanation is to make sure the bit-width is wide enough, only 120k weights are added to connect hidden neurons in each iteration, which is much smaller than 198.8k in the ﬁrst 3 iterations.
The exploration of optimal way to generate fully-connected DNN or DNN with optimal topology is out of scope of this paper.
[Figure]
[ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 4 - b i t f u l l y - c
o n n e c t e d 5 - b i t 6 - b i t 7 - b i t [ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 5 - b i t t i l t e d
[ 7 8 4 - 6 0 0 - 6 0 0 - 1 0 ] 6 - b i t t i l t e d [ 7 8 4 - 4 0 0 - 4 0 0 - 1 0 ] 7 - b i t
[Figure]
t e s t e r r o r r a t e (
[Figure]