R ECURSIVE B INARY N EURAL N ETWORK L EARNING M ODEL WITH 2- BIT / WEIGHT S TORAGE R EQUIREMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper presents a storage-efﬁcient learning model titled Recursive Binary
Neural Networks for embedded and mobile devices having a limited amount of on-chip data storage such as hundreds of kilo-Bytes.
The main idea of the proposed model is to recursively recycle data storage of synaptic weights (parameters) during training.
This enables a device with a given storage constraint to train and instantiate a neural network classiﬁer with a larger number of weights on a chip, achieving better classiﬁcation accuracy.
Such efﬁcient use of on-chip storage reduces off-chip storage accesses, improving energy-efﬁciency and speed of training.
We veriﬁed the proposed training model with deep neural network classiﬁers and convolutional neural network on the permutation-invariant MNIST benchmark.
For deep neural network, our model achieves data storage requirement of as low as 2 bits/weight while the conventional binary neural network learning models require data storage of 8 to 16 bits/weight.
With same amount of data storage, our model can train a bigger network having more weights, achieving 1% better classiﬁcation accuracy than the conventional binary neural network learning model.
To achieve the similar classiﬁcation error, the conventional binary neural network model requires 4 × more data storage for weights than our proposed model.
For convolution neural network, the proposed model achieves 2.4% better accuracy with same on-chip storage and 6 × storage saving to achieve similar accuracy.
I NTRODUCTION
Deep Neural Networks (DNN) have demonstrated the state-of-the-art results in a wide range of cognitive workloads such as computer vision Krizhevsky et al.
and speech recognition (Hinton et al.
(2012)), achieving better-than-human performance for the tasks often considered too complex for machines.
The success of DNN has indeed motivated scientists and engineers to implement
a
DNN in mobile and embedded devices, dubbed as Internet of Smart Things (Kortuem et al. (2010)).
The recent works in this area however, mostly implement the inference function of DNN, without the function of training DNN, while DNN is trained in cloud computers and post-training weights are downloaded to mobile and embedded devices (Lane et al.
On-device learning, however, becomes increasingly important for the mobile and embedded devices for the following three reasons.
First, intelligent device beneﬁts to have the model that is custombuilt for the device and its end user and environment.
This is because the model tends to be more accurate and effective if constructed with the consideration of those factors.
Second, the training data from mobile and embedded sensing devices can contain security-sensitive information, e.g., personal health data from wearable medical devices.
At the risk of being leaked, users typically do not want to upload such data onto cloud computers.
Finally, in the era of Internet of Things (IoT), we anticipate a drastic increase in the number of deployed devices, which can proportionally increase the number of learning tasks to be done in the cloud.
Coupled with the complexity of learning, even for powerful cloud computers, this can be a computationally challenging task.
On-device learning, however entails various challenges, in algorithms, data, and systems (Roschelle (2003); Vogel et al.
The most eminent challenge regarding computing systems is high energy consumption caused by dense computation and data access of DNN system which is prohibited by the limitation of resources of embedded devices.
The high overhead of data access is caused by fetching DNN weights from DRAM (or FLASH) external to a computing chip on an embedded device.
Since the data storage size is limited for such computing chip, the parameters of a DNN have to be stored in external DRAM and FLASH during training.
For example, ARM Cortex M3 processor, a processor widely used in commercial wearable devices such as FitBit, has only 64 kilo-Byte (kB) on-chip data storage.
This can only store very small size of DNN especially if each weight is 32-bit ﬂoat point number.
Compared to accessing on-chip SRAM, accessing off-chip DRAM incurs 3 to 4 orders of magnitudes more energy and delay overhead.
Therefore, fetching weights every time for each data makes training prohibitive to be implemented on a mobile and embedded device (Han et al.
Recently several techniques on pruning, distilling, and binarizing weights have been proposed to compress the parameters of a DNN.
This makes it more feasible to ﬁt weights in on-chip SRAM (Han et al.
; Courbariaux et al.
; Rastegari et al.
; Hinton et al.
These techniques can also reduce computation overhead.
However, these works focused on weight size compression after training is ﬁnished .
The data storage requirement during training remains the same.
Similarly, several learning models, which belong to so-called Binary Neural Networks (BNN), have been proposed (Courbariaux et al.
; Rastegari et al.
These model uses sign bits (or binary information) of weights in several parts of the learning model notably the part of multiplying and accumulating weights with inputs/activations.
Although this greatly reduces computational complexity, each weight still needs to be represented in high precision number with multiple bits (e.g. 32 bits in Courbariaux et al. ; Rastegari et al. during the end-to-end training process since it has to be iteratively ﬁne-tuned in the weight update part.
Therefore, this so-called BNN models have not demonstrated to scale storage requirement for training belowbits/weight.
Our goal is, therefore, to efﬁciently use the limited amount of on-chip data storage during training.
We also aim to scale computational complexity.
Toward this goal, we propose a new learning model,
Recursive Binary Neural Network (RBNN) .
This model is based on the process of training of a neural network, weight binarization, recycling storage of non-sign-bit portion of weights to add more weights to enlarge the neural network for performance improvement.
We recursively perform this process until either accuracy stop improving or we use up all the storage on a chip.
We veriﬁed the proposed RBNN model on a multi-layer perceptron (MLP)-like classiﬁer and a convolutional neural network on the MNIST benchmark.
And MLP-like classiﬁer on a voice activity detection (VAD) benchmark.
We considered typical storage constraints of embedded sensing devices in the order of hundreds of kB. The experiment in the MLP-like classiﬁer on MNIST conﬁrms that the proposed model (i) demonstrates 1% classiﬁcation accuracy improvement over the conventional BNN learning model speciﬁcally following Courbariaux et al.
(2015) for the same storage constraints or (ii) scale on-chip data storage requirement by 4 × for the same classiﬁcation test error ( ∼ 2%), marking the storage requirement of 2 bits/weight.
The conventional BNN model in Courbariaux et al. (2015; 2016); Rastegari et al.
(2016) exhibits a signiﬁcantly larger storage requirement of 8 to 32 bits/weight.
The experiment of CNN classifying MNIST shows up to 6 × reduction of data storage requirement and 2.4% accuracy improvement.
For VAD tasks, the proposed RBNN shows 9 × saving in data storage.
The remainder of the paper is as follow.
In Sec.
2 we will introduce the works related to this paper, including comparison to existing works on distillation, compression, BNNs, and low-precision weights.
In Sec.
3 we will describe the intuitive and details of our proposed model.
Sec. 4 will present the experimental results and comparisons to the conventional BNN model.
Finally, in Sec. 5, we will conclude the paper.
R ELATED W ORK 2. D ISTILLATION AND C OMPRESSION OF DNN PARAMETERS
Knowledge distillation Hinton et al.
(2015) is a technique to compress knowledge of an ensemble of
DNNs into one small DNN while maintaining the accuracy.
Although this technique can scale the number of weights for deployment systems post-training, it cannot scale data storage requirement
Another technique is to compress the data size of weights by exploiting redundancies in them.
In
Han et al. (2015), the authors combine four sub-techniques, namely weight pruning, quantization, sharing, and compression coding to reduce the data size of weights.
Similar to the knowledge distillation, this technique can be applied to the weights that are already trained, and cannot scale data storage requirement of weights during training.
2. B INARY N EURAL N ETWORK (BNN)
Recent works proposed to use binary information of weights Courbariaux et al.
(2015), activations
Courbariaux et al.
; Rastegari et al.
(2016), and even inputs Rastegari et al.
(2016) in some parts of learning and post-learning operations.
The use of binary information of weights notably in multiply-and-accumulate (MAC) operation can drastically reduce computational complexity.
However, those BNN techniques still cannot scale the storage requirement of weights during training.
In these works, each weight is represented in 32 bits.
This is because mainstream training models such as stochastic gradient decent requires ﬁne-grained update of weights.
2. L OW - PRECISION FIX - POINT WEIGHT REPRESENTATION
Several studies have demonstrated moderately lowering the precision of weights (i.e., quantization) has a tolerable impact on training and post-training operations of DNN (Gupta et al.
; Courbariaux et al.
In Gupta et al.
, the authors trained a DNN having 16-bit ﬁxed-point weights with the proposed stochastic rounding technique, and demonstrated little to no degradation in classiﬁcation accuracy.
In Courbariaux et al.
, the authors proposed the dynamic ﬁxedpoint representation (i.e., dynamically changing the position of decimal point over computation sequences) to further reduce the precision requirement down to 10 bits per synapse.
Using ﬁxpoint representation help to reduce storage requirement and ﬁxed-point arithmetic is more hardware friendly (Han et al.
R ECURSIVE B INARY N EURAL N ETWORK (RBNN) M ODEL 3. K EY IDEA
Table 1 shows which information of weights are used in each step of training in both conventional
BNN Courbariaux et al.
; Rastegari et al.
and our proposed RBNN.
The conventional BNN works Courbariaux et al.
; Rastegari et al.
(2016) use sign bits of weights during multiply-and-accumulate (MAC) operation in feed-forward and back propagation.
However, the weight update has to be done in high precision.
This mandates to store multi-bit (32 bit if using ﬂoat point number) weights in data storage during learning, resulting in no savings in weight storage requirement.
It has been studied that in the trained neural networks we can use only the sign bits of weights to perform inference (Courbariaux et al.
; Rastegari et al.
This vast difference in the requirements of weight precision between learning and post-learning inspires us to create our RBNN model.
As shown in Table 1, when training BNN, we use only the sign bits for MAC operations to reduce computational complexity.
The main difference is that we binarize weights (keep only the sign bits). And then after training, we recycle the data storages that are used to store non-sign bits of weights to add more multi-bit trainable weights to the neural network.
We then trained this new network having both the binarized non-trainable weights and the newly-added trainable weights.
We perform this process recursively, which makes the neural networks larger and more accurate but using the same amount of data storage for weights.
Figure 1 depicts the process of our proposed RBNN learning model with an example of multi-layer fully-connected neural network.
In the beginning the neural network has one input, two sets of two hidden, and one output neurons and eight weights each of which has n bits.
We ﬁrst train this 1 × 2 × 2 × 1 network using the conventional back-propagation training algorithm.
After that, we discard all bits except the sign bit in each weight (binarization), resulting in a 1 × 2 × 2 × 1 trained [Table] Keep [Table]
recycle [Table] of the other bits for more plastic weights network having binary weights ( trained BNN ). Then we continue the second iteration of training (the second subﬁgure of Figure 1).
Speciﬁcally, we recycle the storage that is used to store the nnon-sign bits of weights in the 1 × 2 × 2 × 1 network.
Using this data storage, we add a new network named incremental BNN comprising eight additional weights ( W 21 to W 28 ) to the trained BNN , expanding the network size to 1 × 4 × 4 × 1 which we name as enlarged BNN .
In the enlarged BNN , each of the newly-added weights is n − 1 bits.
In other words, the enlarged BNN comprises of one trained BNN that has eight weight ( W b 11 to W b 18 ) that are trained (binary, non-plastic, marked as solid lines in Figure 1) and one incremental BNN with eight weights ( W 21 to W 28 ) that are under training (n-1 bits, plastic, marked as dash lines in Figure 1).
The incremental BNN is trained together with the trained BNN but only the weights of incremental BNN are updated.
We repeat the same process of binarization and recycling.
In every iteration, the enlarged BNN integrates eight more weights, and the bit-width of newly-added plastic weights in the incremental BNN is reduced by one.
At the k iterations, the trained BNN has 8 · ( k − 1) neurons and the plastic weights have ( n − k + 1 ) bit-width.
After the k-th training is ﬁnished, as shown in the rightmost in Figure 1 the neural network becomes a 1 × 2 k × 2 k × 1 with 8 · k binary weights.
This network has k times more weights than the ﬁrst 1 × 2 × 2 × 1 network.
However, the data storage used for weights remains the same, scaling the storage requirement per weight to n/k ( = 4 · n/ 4 · k ), which is k times smaller than that of the ﬁrst network.
Thus the proposed RBNN can either achieve better classiﬁcation accuracy - enabled by the more number of weights - with the same amount of weight storage, or reduce weight storage requirement for the same classiﬁcation accuracy level.
[Figure]
BNN_Training
Bit_width_Reduction
[-W bk
-]
Incremental_BNN_Training
Stop criteria met?
FALSE
END TRUE
Performance_evaluation
BNN_Training :
Training initial
BNN using conventional BNN training method.
Bit-width_Reduction : Reduce bit-width of all synaptic weights by
Incremental_BNN_Training : Training incremental_BNN with previously Trained_
BNN
Performance_evaluation: Test the performance of trained enlarged BNN
Require: a minibatch of inputs and targets ( a 0 , a ∗ ), previous weights of incremental BNN W ( I ) , weights of trained BNN W ( T )
Ensure: updated weights of incremental BNN W ( I ) ( t +1) 1. Forward Propagation do a [Equation]
W
[Equation]
end
1.2 Computing outputs
enlarged BNN
a L = Act out ( W ( T ) L · a ( T )
[Equation]
L − 1) + W ( I ) L · a ( I ) ( L − 1) )
2. Back propagation
{ Please note that only gradients of incremental BNN are computed. }
Compute g aL = ∂C∂a
L knowing a L and a ∗ for k
= L to 1 do
[Equation]
3. Parameter Update
Please note that only weights of incremental BNN are updated.
for k = L to 1 do
[Equation]
end for
3. M ODEL DETAILS
Figure 2 depicts the details of the proposed RBNN model.
The left part of Figure 2 depicts the ﬂow chart of RBNN model, while the right part explains the function of each stage.
In the beginning of the training procedure, conventional BNN training algorithm BNN Training is used to train a BNN. After training, we have a trained BNN having binary synaptic weights. Then we reduce the synaptic bit-width by one and train a new incremental BNN . The training algorithm for incremental BNN is named as incremental BNN Training which is shown in Algorithm 1.
After the incremental BNN is trained, the performance of the enlarged BNN is tested.
If the performance doesn’t stop improving and there are still available synaptic bits after weight binarization, we will continue to reduce the synaptic bit-width by one and train one more incremental BNN to keep on enlarge enlarged BNN . The method Incremental BNN Training is designed to train the incremental BNN to improve performance of enlarged BNN . To meet this goal, the conventional BNN training method is adjusted. As shown in Algorithm 1, the main idea of this training method is: both trained BNN and incremental BNN are used to calculate output of the enlarged network in the feedforward training step.
While during back-propagation and parameter-update, only plastic weights in incremental BNN are updated.
And the trained weights in trained BNN need to be maintained. The hardware implementation of this sub-word operation of synaptic weights are illustrated in Appendix A. Notice that similar to the conventional BNN training algorithm Zhou et al.
, binary weights are used in both feed-forward and back-propagation in Incremental BNN Training , to reduce computational overhead.
Since weights in trained BNN are binary, the multiplication related to weights are simpliﬁed as shift.
E XPERIMENT S ETUP
In this and next section, the detailed experiment setup and results for MNIST classiﬁcation with
MLP-like DNN systems using the proposed RBNN model are discussed.
Furthermore, the proposed
RBNN model is also applied to CNNs classifying MNIST and MLP-like DNN doing VAD tasks.
The setup and results are discussed in Appendix B and C, respectively.
4. P ERMUTATION - INVARIANT MNIST BENCHMARK
We used the permutation-invariant MNIST to test the performance of the proposed RBNN model.
We use the original training set of 60,000 28-by-28 pixel gray-scale images and the original test set of 10,000 images.
The training and testing data are normalized to  zero mean.
Following the common practices, we use the last 10,000 images of the training set as a validation set for early stopping and model selection.
We did not consider data augmentation, pre-processing, and unsupervised pre-training during our experiment.
4. N EURAL NETWORK CONFIGURATION AND DATA FORMAT
We considered the storage constraints of mainly hundreds of kB based on the typical embedded systems (Shiue & Chakrabarti (1999)).
We considered a feed-forward neural network with one or two hidden layers.
We considered several different numbers of neurons in the hidden layer ranging from 200 to 800.
The numbers of the input and output units are 784 and 10, respectively.
We used the tanh opt() for the activation function of the hidden layer and the softmax() or linear output for that of the output layer.
We use the classical Stochastic Gradient Descent (SGD) algorithm for cross-entropy or hinge loss minimization without momentum.
We use a small size of batch (1,000) and a single static learning rate which is optimized for each BNN.
Any other advanced techniques such as dropout, Maxout, and ADAM are not used for both the proposed and the baseline learning models.
We recorded the best training and test errors associated with the best validation error after up to 1,000 epochs.
The results from 20 independent experiments are averaged for each case.
We used the ﬁxed-point arithmetic for all the computation and data load and access.
The ﬁxed-point intermediate computations, such as gradient calculation have, also use ﬁxed-point arithmetic with sufﬁcient precision.
The translations from wide ﬁxed-point number to narrow ﬁxed-point and binary number are performed with simple decimation without using advanced techniques such as stochastic rounding (Courbariaux et al.
We saturated values in the event of overﬂow in weight update.
The dynamic range of ﬁx-point representation is optimized to achieve better accuracy performance.
R ESULTS AND D ISCUSSION 5. A CCURACY IMPROVEMENT
Figure 3 depicts the classiﬁcation errors of the proposed RBNN model across three recursive iterations.
The initial bit-width of weights is eight.
In each series of data points in Figure 3, the leftmost point represents the initial neural network, i.e., with 2 layers of 200 hidden units and 198,800 synaptic weights ( = 784 · 200 + 200 · 200 + 200 · 10 ).
At this point, the storage requirement, deﬁned as the ratio of total storage bits to the number of weights, is 8 bits/weight.
At this point the network is equivalent to one trained with the conventional BNN model speciﬁcally following Courbariaux et al.
The second leftmost data point in the series is the neural network after the ﬁrst recursive iteration.
The network size is enlarged by twice, resulting in the 784 × 400 ×
400 × 10 network.
This reduces storage requirement to 4 bits/weight.
Compared to the initial BNN, the enlarged BNN achieves ∼ 0.7% and ∼ 0.4% reduction in training and test error rate, respectively.
Finally, after three recursive iterations, the size of the neural network becomes 784 × 800 × 800 × 10 (555,800 synaptic weights).
It marks the storage requirement small as 2 bits/weight to achieve the test error of 2.17%.
This accuracy is as good as fully-connected network with 4 × times of synaptic data storage in (Courbariaux et al.
Notice that there can be various of topologies that the proposed RBNN can generate.
The topology depends on how the recycled data storage is utilized.
In this experiment the structure of the generated network is as shown in Figure 1.
Speciﬁcally, in each iteration, the recycled data storage is used to generate an incremental BNN s with identical topology and there’s no connection between incremental BNN s that are generated in different iterations.
This can be easily implemented on hardware in the way as shown in Appendix A. In Appendix D we explore one possible way to use the proposed
RBNN model to generate fully-connected DNN.
The results show similar accuracy as the ones of experiment in this section with same size of data storage.
[Figure]
5. S TORAGE AND ARITHMETIC COMPLEXITY
To evaluate the storage and arithmetic complexity of the proposed RBNN, we trained multiple single-hidden-layer DNNs using RBNN and the conventional BNN learning model (Courbariaux et al.
For the conventional model, we considered BNN containing 100 to 800 hidden neurons combined with 16 to 6 bits precision of synaptic weights.
For the proposed model, we considered 100 to 800 initial hidden neurons and 12 to 16 bits initial weight precisions.
Those DNNs require 116 kB to 1.2 MB data storage for weights.
Figure 4 shows the results of this experiment: the proposed model can achieve 1% lower test error than the conventional model using the similar amount of data storage.
To achieve the similar test error, the proposed RBNN model requires 4 × less data storage than the conventional BNN model.
Table 2 shows the detail comparisons of six neural networks out of the 16 networks shown in Figure 4, three of which are trained by the proposed RBNN model ( R 1 , R 2 , R 3 ) and the other three by the conventional BNN model ( B 1 , B 2 , B 3 )
(Courbariaux et al.
The arithmetic complexity for [Figure] qu i
[Figure] e m e n t ( M B ) Test error (%)
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table] training and inferring are compared. For conventional BNN, the arithmetic complexity of learning one data sample in one epoch is shown. While for RBNN, we multiply this arithmetic/data/epoch by the number of total binarization-recycle iterations in training procedure of RBNN.
To compare storage requirement, we choose R 1 and B 1 , R 3 and B 2 which achieve similar test error. It is shown that R 1 outperforms B 1 in the storage requirement by 3 × and R respectively.
The downside of R 1 and R 3 is the increase in computations during training: it requires more shift and add operations. But it should be noticed that it still needs the same number of multiplications as compared to B 1 and B 2 .
Since complexity of multiplication is much more less than one of add and shift, this extra overhead can be easily offset by the energy saving with respect to data access, which will be shown in next subsection.
On the other hand, the computation complexity for inference operation are the same.
5. E NERGY C ONSUMPTION S AVINGS
The last three rows of Table 2 compare the energy dissipation for training of RBNN and conventional BNN model, with respect to arithmetic computation and data access.
The calculation of energy is based on the 45nm energy cost presented in Han et al.
(2016), which normalized to the operation of according bit-widths.
The results in Table 2 show that to achieve same performance level, the proposed RBNN saves around 100 × total energy for training ( R 1 and B 1 , R 3 and B 2 ).
This is because RBNN reduces the data storage requirement for on-chip training, so the data can be ﬁt in on-chip SRAM of chips for low-cost application.
On the other hand, for conventional BNN, the weights have to store and fetched from off-chip DRAM during training (Altera), which costs 2-order of magnitudes more energy than SRAM access and basic arithmetic.
Therefore, even the proposed RBNN has higher energy dissipation for arithmetic computation, the proposed RBNN still achieves high saving of energy dissipation of training.
C ONCLUSION AND F UTURE W ORK
This paper presents a new learning model for on-device training with limited data storage.
The proposed RBNN model efﬁciently uses limited on-chip data storage resources by recycling the part of data storage that would have been wasted in conventional BNN model, to add and train more synaptic weights to a neural network classiﬁer.
We veriﬁed the proposed model with MLP-like DNN and CNN classiﬁer on the permutation-invariant MNIST and VAD benchmark under the typical embedded device storage constraints.
The results of MLP DNNs on MNIST show that the proposed model achieves 2 bits/weight storage requirement while achieving 1% better classiﬁcation error as compared to the conventional binary-weight learning model for the same storage constraint.
Our proposed model also achieves 4 × less data storage than the conventional model for the same classiﬁcation error.
We expect the future work that extends the application of the learning model to other neural network topologies and datasets, such as BNN with binary activation function Courbariaux et al.
We also expect to apply the RBNN model to the ensembles of neural networks (Zhou et al.
(2002)), the mixture of experts (Shazeer et al.
and the incremental learning (Xiao et al.
A I MPLEMENTATION OF S UB - WORD O PERATION OF S YNAPTIC W EIGHTS
In the proposed RBNN model, during training only plausible synaptic bits can be updated, while ﬁxed (trained) synaptic bits need to maintain their value.
Figure 5 illustrates the implementation of this sub-word operation of synaptic bits.
An 8-bit synaptic weight is used in Figure 5 as an example, of which 3 bits are ﬁxed (trained) synaptic bits and 5 bits are plastic bits that are under training in current iteration.
At the beginning of training procedure, one complete weight synp in is fetched from weight storage to computation circuits.
And then a signal synp mask which has 1 at the position of ﬁxed weight bits and 0 at the position of plastic synaptic bits (11100000 in this example) and its reverse type (00011111) are AND ed to the weight to separate ﬁxed weight synp ﬁx and plastic weight synp plsb , respectively.
During the training procedure, synp plsb is updated while synp ﬁx maintains the same value.
Once training is ﬁnished, the updated synp plsb and initial synp ﬁx are concatenated through XOR operation.
Compared with conventional BNN, the proposed RBNN only requires one AND operation at the beginning and one XOR operation at the end of each training epoch, the cost of which is much smaller than cost of arithmetic and data access of training process.
[Figure]
B A PPLICATION OF RBNN TO C ONVOLUTIONAL N EURAL N ETWORK B. E XPERIMENT S ETUP
We apply the proposed RBNN model to LeNet5 Convolutional Neural Network (LeCun et al.
with two convolution layers, having 6 and 12 of 5 by 5 feature maps, respectively.
Both of them are followed by a 4x dowsampling average pooling layer.
The fully-connected (FC) layers consist of one input, one hidden and one output layer.
As in LeCun et al.
(1998), binary weights are used for all convolutional and FC layers during forward and backward propagation and ﬁx-point weights are used for weight updating to reduce computation overhead of training.
The proposed
RBNN model is applied to increase the size of hidden layer of the FC layers.
B. R ESULTS A NALYSIS
We trained CNNs of a range of conﬁguration for FC layers to classify MNIST dataset using the proposed RBNN model.
And compare the accuracy with the CNN trained with conventional BNN method (Courbariaux et al.
For the proposed model, we considered from 200 to 800 initial hidden neurons and 16-bit initial weight precision.
For the conventional model, we considered
CNNs with FC layers containing from 200 to 2500 hidden neurons with 16-bit synaptic weights.
The CNNs require 81kB to 1.01MB data storage for weights of all convolutional and FC layers.
Figure 6 compares the accuracy performance and the according data storage size for CNNs with various conﬁguration with both the proposed RBNN model and conventional BNN model.
The results prove that the proposed RBNN model can achieve 2.4% lower test error than the conventional model using the same amount of data storage.
To achieve the similar test error, the proposed RBNN model requires 7X less data storage than the conventional BNN model. 2 % ~ 4 X > 6 X C o n v e n t i
o n a l B N N P r o p o s e d R B N N W e i g h t s t
o r a g
e s i
z e ( k B )
T e s t e r r o r ( % ) [ 2 5 0 0 ]
C A PPLICATION OF RBNN TO V OICE A CTIVITY D ETECTION T ASKS
To further validate the proposed RBNN model, we apply it to MLP-like DNNs to do Voice Activity
Detection tasks.
The structure of DNN used in this experiment is the same as the one in Sec. 4.2.
The experiments were conducted on Aurora 4 (Pearce & Picone (2002)), which comprisingutterances from 83 speakers.
The basic feature used for the DNN to detect is 5 frames of 16dimensional band-pass ﬁlterbank output.
The signal-to-noise ratio of data in all scenarios used in the experiment is 10dB. Table 4 shows the topology and performance of DNNs using both RBNN and conventional BNN model for VAD tasks in various scenarios and proves as high as 9X data storage saving to achieve same level of detection accuracy. [Table]
D RBNN IN FULLY - CONNECTED DNN SYSTEMS
In this section we demonstrate one way of using the proposed RBNN to generate a fully-connected
DNN and show the performance improvement compared to conventional BNN and tilted topology shown in Sec. 3.1.
Figure 7 illustrates how the fully-connected DNN is built step by step with an example of DNN with initial size being 1-2-2-1 and n-bit.
Firstly, new hidden neurons are added and the available weight storage is used for connection between input, newly added hidden, and output neurons.
As shown in the second sub-ﬁgure, after k iterations, the topology becomes 1-2k-2k-1.
Notice that the hidden neurons generated in different iterations are not connected with each other at this point.
And then in the following iterations, the available weight storage is used to connect hidden layers generated in different iterations, adding m new weights each time, until all the neurons in the DNN system are connected with each other.
The value of k and m is determined by size of available data storage, and the performance improvement of current iteration.
To evaluate the performance of fully-connected DNN generated with the proposed RBNN model,
Figure 8 shows the classiﬁcation performance on MNIST dataset of a 784-800-800-10 fully connected DNN generated from a 784-200-200-10 DNN with 8-bit initial synaptic bit-width.
The test error rate at each iteration is shown, as well as the topology and plausible synaptic bit-width.
According to Figure 8, the performance increases as more hidden neurons are added, while almost remains the same when more weights are added to connect existing hidden neurons.
The possible explanation is to make sure the bit-width is wide enough, only 120k weights are added to connect hidden neurons in each iteration, which is much smaller than 198.8k in the ﬁrst 3 iterations.
The exploration of optimal way to generate fully-connected DNN or DNN with optimal topology is out of scope of this paper.
[Figure]
[ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 4 - b i t f u l l y - c
o n n e c t e d 5 - b i t 6 - b i t 7 - b i t [ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 5 - b i t t i l t e d
[ 7 8 4 - 6 0 0 - 6 0 0 - 1 0 ] 6 - b i t t i l t e d [ 7 8 4 - 4 0 0 - 4 0 0 - 1 0 ] 7 - b i t t
i l t e d t e s t e r r o r r a t e ( % )
# o f w e i g h t s ( k ) [ 7 8 4 - 2 0 0 - 2 0 0 - 1 0 ] 8 - b i t t i l t e d