R ECURSIVE B INARY N EURAL N ETWORK L EARNING M ODEL WITH 2- BIT / WEIGHT S TORAGE R EQUIREMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper presents a storage-efﬁcient learning model titled Recursive Binary
Neural Networks for embedded and mobile devices having a limited amount of on-chip data storage such as hundreds of kilo-Bytes.
The main idea of the proposed model is to recursively recycle data storage of weights (parameters) during training.
This enables a device with a given storage constraint to train and instantiate a neural network classiﬁer with a larger number of weights on a chip, achieving better classiﬁcation accuracy.
Such efﬁcient use of on-chip storage reduces off-chip storage accesses, improving energy-efﬁciency and speed of training.
We veriﬁed the proposed training model with deep and convolutional neural network classiﬁers on the MNIST and voice activity detection benchmarks.
For the deep neural network, our model achieves data storage requirement of as low as 2 bits/weight, whereas the conventional binary neural network learning models require data storage of 8 to 32 bits/weight.
With the same amount of data storage, our model can train a bigger network having more weights, achieving 1% less test error than the conventional binary neural network learning model.
To achieve the similar classiﬁcation error, the conventional binary neural network model requires 4 × more data storage for weights than our proposed model.
For the convolution neural network classiﬁer, the proposed model achieves 2.4% less test error for the same on-chip storage or 6 × storage savings to achieve the similar accuracy.
I NTRODUCTION
Deep Neural Networks (DNN) have demonstrated the state-of-the-art results in a wide range of cognitive workloads such as computer vision Krizhevsky et al.
and speech recognition (Hinton et al.
(2012)), achieving better-than-human performance for the tasks often considered too complex for machines.
The success of DNNs has indeed motivated scientists and engineers to implement a DNN in mobile and embedded devices, dubbed as Internet of Smart Things (Kortuem et al.
The recent works in this area, however, mostly implement the inference function of DNN, rather than training, while training is performed in cloud computers and post-training weights are downloaded to mobile and embedded devices (Lane et al.
On-device learning, however, becomes increasingly important for the mobile and embedded devices for the following three reasons.
First, intelligent device beneﬁts to have the model that is custombuilt for the device itself, its end user, and environment.
This is because the model tends to be more accurate and effective if constructed with the consideration of those factors.
Second, the training data from mobile and embedded devices can contain security-sensitive information, e.g., personal health data from wearable medical devices.
At the risk of being leaked, users typically do not want to upload such data onto cloud computers.
Finally, in the era of Internet of Things (IoT), we anticipate a drastic increase in the number of deployed devices, which can proportionally increase the number of learning tasks to be done in the cloud.
Coupled with the complexity of training, even for powerful cloud computers, this can be a computationally challenging task.
On-device learning, however, entails various challenges in algorithms, data, and systems (Roschelle (2003); Vogel et al.
The most eminent challenge regarding computing systems is high energy consumption caused by dense computation and data access, which is considered prohibitive for the limited resources of embedded devices.
The high overhead of data access is caused by fetching DNN weights from DRAM (or FLASH) external to a computing chip on an embedded device.
Since the data storage size is limited for such computing chip, the parameters of a DNN have to be stored in external DRAM and FLASH during training.
For example, ARM Cortex M processor, a processor widely used in commercial wearable devices such as FitBit, has onlykilo-Byte (kB) on-chip data storage.
This can only store very small size of DNN especially if each weight is 32-bit ﬂoat point number.
Compared to accessing on-chip SRAM, accessing off-chip DRAM incurs 3 to 4 orders of magnitudes more energy and delay overhead.
Therefore, fetching weights every time for each data makes training prohibitive to be implemented on a mobile and embedded device (Han et al.
Recently several techniques such as pruning, distilling, and binarizing weights have been proposed to compress the parameters of a DNN.
This makes it more feasible to ﬁt weights in on-chip SRAM (Han et al.
; Courbariaux et al.
; Rastegari et al.
; Hinton et al.
These techniques can also reduce computation overhead.
However, these works focused on weight size compression after training is ﬁnished .
The data storage requirement during training remains the same.
Similarly, several learning models, which belong to so-called Binary Neural Networks (BNN), have been proposed (Courbariaux et al.
; Rastegari et al.
These model uses sign bits (or binary information) of weights in several parts of the learning model notably the part of multiplying and accumulating weights with inputs/activations.
Although this greatly reduces computational complexity, each weight still needs to be represented in high precision number with multiple bits (e.g. 32 bits in Courbariaux et al.
; Rastegari et al.
during the end-to-end training process. This is because weights have to be ﬁne-tuned in the weight update part.
Therefore, this so-called BNN models have not demonstrated to scale storage requirement for training below 32 bits/weight.
Our goal is, therefore, to efﬁciently use the limited amount of on-chip data storage during training.
We also aim to scale computational complexity.
Toward this goal, we propose a new learning model,
Recursive Binary Neural Network (RBNN) .
This model is based on the process of weight training, weight binarization, recycling storage of the non-sign-bit portion of weights to add more weights to enlarge the neural network for accuracy improvement.
We recursively perform this process until either accuracy stops improving or we use up all the storage on a chip.
We veriﬁed the proposed RBNN model on a Multi-Layer Perceptron (MLP)-like and a convolutional neural network classiﬁer on the MNIST and Voice Activity Detection (VAD) benchmark.
We considered typical storage constraints of embedded sensing devices in the order of hundreds of kB. The experiment in the MLP-like classiﬁer on MNIST conﬁrms that the proposed model (i) demonstrates 1% less test error over the conventional BNN learning model speciﬁcally following Courbariaux et al.
(2015) for the same storage constraints or (ii) scales on-chip data storage requirement by 4 × for the same classiﬁcation test error rate( ∼ 2%), marking the storage requirement of 2 bits/weight.
The conventional BNN models in Courbariaux et al. (2015; 2016); Rastegari et al.
(2016) exhibit a signiﬁcantly larger storage requirements of 8 to 32 bits/weight.
The experiment of the CNN classiﬁer for MNIST conﬁrms up to 6 × reduction of data storage requirement and 2.4% less test error.
For the VAD benchmark, the proposed RBNN achieves 9 × savings in data storage requirement.
The remainder of the paper is as follow.
In Sec.
2 we will introduce the works related to this paper, including comparison to existing works on distillation, compression, BNNs, and low-precision weights.
In Sec.
3 we will describe our proposed model.
Sec. 4 will present the experimental results and comparisons to the conventional BNN model.
Finally, in Sec. 5, we will conclude the paper.
The paper includes Appendix A to D to describe additional experiments and analysis.
R ELATED W ORK 2. D ISTILLATION AND C OMPRESSION OF DNN PARAMETERS
Knowledge distillation (Hinton et al.
(2015)) is a technique to compress knowledge of an ensemble of DNNs into one small DNN while maintaining the accuracy.
Although this technique can scale the number of weights for deployment systems post-training, it cannot scale data storage requirement
Another technique is to compress the data size of weights by exploiting redundancies in them.
In
Han et al. (2015), the authors combine four sub-techniques, namely weight pruning, quantization, sharing, and compression coding to reduce the data size of weights.
Similar to the knowledge distillation, this technique can be applied to the weights that are already trained, and cannot scale data storage requirement of weights during training.
2. B INARY N EURAL N ETWORK (BNN)
Recent works proposed to use binary information of weights (Courbariaux et al.
, activations (Courbariaux et al.
; Rastegari et al.
(2016)), and even inputs (Rastegari et al.
in some parts of training and post-training operations.
The use of binary information of weights notably in multiply-and-accumulate (MAC) operation can drastically reduce computational complexity.
However, those BNN techniques still cannot scale the storage requirement of weights during training.
In these works, each weight is represented in 32 bits.
This is because mainstream training models such as stochastic gradient decent requires to update weights in a ﬁne-grained manner.
2. L OW - PRECISION FIX - POINT WEIGHT REPRESENTATION
Several studies have demonstrated that moderately lowering the precision of weights (i.e., quantization) has a tolerable impact on training and post-training operations of DNN (Gupta et al.
; Courbariaux et al.
In Gupta et al.
, the authors trained a DNN having 16bit ﬁxed-point weights with the proposed stochastic rounding technique, and demonstrated little to no degradation in classiﬁcation accuracy.
In Courbariaux et al.
(2014), the authors proposed the dynamic ﬁxed-point representation (i.e., dynamically changing the position of decimal point over computation sequences) to further reduce the precision requirement down to 10 bits per synapse.
Using ﬁxed-point representation help to reduce storage requirement and ﬁxed-point arithmetic is more hardware friendly (Han et al.
R ECURSIVE B INARY N EURAL N ETWORK (RBNN) M ODEL 3. K EY IDEA
Table 1 shows which information of weights are used in each step of training in both conventional
BNN Courbariaux et al.
; Rastegari et al.
and our proposed RBNN.
The conventional BNN works (Courbariaux et al.
; Rastegari et al.
(2016)) use sign bits of weights during multiply-and-accumulate (MAC) operation in forward and backward propagations.
However, the weight update has to be done with the high precision.
This mandates to store multi-bit (32 bits if using ﬂoat point number) weights in data storage during learning, resulting in no savings in weight storage requirement.
On the other hand, it has been studied that in the trained neural networks we can use only the sign bits of weights to perform inference (Courbariaux et al.
; Rastegari et al.
This vast difference in the requirements of weight precision between learning and post-learning inspires us to create our RBNN model.
As shown in Table 1, we also use only the sign bits for MAC operations to reduce computational complexity for training.
The main difference is that we binarize weights (keep only the sign bits) and then we recycle the data storage that are used to store these non-sign bits of weights. This recycled storage is used to add more multi-bit trainable weights to the neural network.
We then train this new network having both the binarized non-trainable weights and the newly-added trainable weights.
We perform these steps recursively, which makes the neural networks larger and more accurate but using the same amount of data storage for weights.
Figure 1 depicts the process of our proposed RBNN learning model with an example of the multilayer neural network classiﬁer.
In the beginning, the neural network has one input, two sets of two hidden, and one output neurons and, eight weights each of which has n bits.
We ﬁrst train this 1 × 2 × 2 × 1 network using the conventional back-propagation training algorithm for BNN (Courbariaux et al. (2015)).
After that, we discard all bits except the sign bit in each weight (binarization), resulting
[Table]
Keep
[Table]
recycle
[Table]
of the other bits for more plastic weights in a 1 × 2 × 2 × 1 trained network having binary weights ( trained BNN ).
Then we continue the second iteration of training (the second subﬁgure of Figure 1).
Speciﬁcally, we recycle the storage that is used to store the n-1 non-sign bits of weights in the 1 × 2 × 2 × 1 network.
Using this data storage, we add a new network named incremental BNN comprising eight additional weights ( Wto W 28 in Figure 1) to the trained BNN , expanding the network size to 1 × 4 × 4 × 1 which we name as enlarged BNN .
In the enlarged BNN , each of the newly-added weights is n − 1 bits.
In other words, the enlarged BNN comprises of one trained BNN that has eight weight ( W b 11 to W b 18 ) that are trained (binary, non-plastic, marked as solid lines in Figure 1) and one incremental BNN with eight weights ( W 21 to W 28 ) that are under training (n-1 bits, plastic, marked as dash lines in Figure 1).
The incremental BNN is trained together with the trained BNN but only the weights of incremental BNN are updated.
We repeat the same process of binarization and recycling.
In every iteration, the enlarged BNN integrates eight more weights, and the bit-width of newly-added plastic weights in the incremental BNN is reduced by one.
At the k-th iteration, the trained BNN has 8 · ( k − 1) neurons and the plastic weights have ( n − k +1 ) bit-width.
After the k-th iteration, as shown in the rightmost in Figure 1, the neural network becomes a 1 × 2 k × 2 k × 1 with 8 · k binary weights.
This network has k times more weights than the ﬁrst 1 × 2 × 2 × 1 network.
However, the data storage used for weights remains the same, scaling the storage requirement per weight to n/k ( = 4 · n/ 4 · k ), which is k times smaller than that of the ﬁrst network.
Thus the proposed RBNN can either achieve better classiﬁcation accuracyenabled by the more number of weights - with the same amount of weight storage, or reduce weight storage requirement for the same classiﬁcation accuracy level.
[Figure]
BNN_Training
Bit_width_Reduction
Incremental_BNN_Training
Stop criteria met?
FALSE
END TRUE
Performance_evaluation
BNN_Training :
Training initial
BNN using conventional BNN training method.
Bit-width_Reduction : Reduce bit-width of all synaptic weights by
Incremental_BNN_Training : Training incremental_BNN with previously Trained_
BNN
Performance_evaluation: Test the performance of trained enlarged BNN
Algorithm 1 Incremental BNN Training .
C is the cost function for mini-batch, η the learning rate and L the number of layers.
The function Binarize() speciﬁes how to binarize the weights.
Act hid() and Act out() are activation function of hidden layers and output layer, respectively.
Require: a minibatch of inputs and targets ( a 0 , a ∗ ), previous weights of incremental BNN W ( I ) , weights of trained BNN W ( T )
Ensure: updated weights of incremental BNN W ( I ) ( t +1)
1. Forward Propagation 1.1 Computing outputs of hidden layers in trained BNN and incremental BNN for
k = 1 to L-1 do
[Equation]
end for
1.2 Computing outputs of enlarged BNN
a
[Equation]
2. Backward propagation
{ Please note that only gradients of incremental BNN are computed. }
Compute g aL = ∂C∂a
L knowing a L and a ∗ for k
= L to 1 do
[Equation]
) k −end for
3. Parameter Update
Please note that only weights of incremental BNN are updated.
for k = L to 1 do
W ( I ) T +1 k ← W ( I ) tk + η · g WI
bk end for
3. M ODEL DETAILS
Figure 2 depicts the details of the proposed RBNN model.
In the beginning of the training procedure, conventional BNN training algorithm BNN Training is used to train a BNN.
After training, we have a trained BNN having binary weights.
Then we reduce the weight bit-width by one and train a new incremental BNN .
The training algorithm for incremental BNN is named as incremental BNN Training which is shown in Algorithm 1.
After the incremental BNN is trained, the performance of the enlarged BNN is tested.
If the accuracy keeps on improving and there is still available data storage after weight binarization, we will continue to reduce the weight bit-width by one and train another incremental BNN .
The method Incremental BNN Training is designed to train the incremental BNN to improve performance of enlarged BNN .
It is based on the conventional BNN training method.
As shown in Algorithm 1, the main idea of this training method is: both trained BNN and incremental BNN are used to calculate the output of the enlarged network in the forward propagation.
During back-propagation and parameter-update, however, only plastic weights in incremental BNN are updated.
The binary weights in trained BNN are not modiﬁed.
One possible hardware and software implementation of this sub-word operation of synaptic weights are illustrated in Appendix A. Note that similar to the conventional BNN training algorithm
(Zhou et al.
, binary weights are used in both forward and backward propagation in Incremental BNN Training , to reduce computational overhead.
Since weights in trained BNN are binary, the multiplication related to weights are simpliﬁed as shift.
E XPERIMENT S ETUP
In this and the next section, we will describe the detailed experiment setup and the results for the
MLP-like classiﬁer and the MNIST benchmark.
In addition, we will discuss the setup and results of applying the proposed RBNN model to CNN classiﬁers and VAD benchmarks in Appendix B and C, respectively.
4. P ERMUTATION - INVARIANT MNIST BENCHMARK
We used the permutation-invariant MNIST to test the performance of the proposed RBNN model on MLP-like classiﬁer.
We use the original training set of 60,000 28-by-28 pixel gray-scale images and the original test set of 10,000 images.
The training and testing data are normalized to [-1, 1] and zero mean.
Following the common practices, we use the last 10,000 images of the training set as a validation set for early stopping and model selection.
We did not consider data augmentation, pre-processing, and unsupervised pre-training during our experiment.
4. N EURAL NETWORK CONFIGURATION AND DATA FORMAT
We consider the storage constraints of mainly hundreds of kB based on the typical embedded systems (Shiue & Chakrabarti (1999)).
We considered a feed-forward neural network with one or two hidden layers.
We considered several different numbers of neurons in the hidden layer ranging from 200 to 800.
The numbers of the input and output units are 784 and 10, respectively.
We used the tanh opt() for the activation function of the hidden layer and the softmax() or linear output for that of the output layer.
We used the classical Stochastic Gradient Descent (SGD) algorithm for cross-entropy or hinge loss minimization without momentum.
We used a small size of batch (1,000) and a single static learning rate which is optimized for each BNN.
Any other advanced techniques such as dropout, Maxout, and ADAM are not used for both the proposed and the baseline learning models.
We recorded the best training and test errors associated with the best validation error after up to 1,000 epochs.
The results from 20 independent experiments are averaged for each case.
We used the ﬁxed-point arithmetic for all the computation and data access.
The ﬁxed-point intermediate computations, such as gradient calculation, also use ﬁxed-point arithmetic with sufﬁcient precision.
The translations from wide ﬁxed-point numbers to narrow ﬁxed-point and binary numbers are performed with simple decimation without using advanced techniques such as stochastic rounding (Courbariaux et al.
We saturated values in the event of overﬂow or underﬂow in weight update.
The dynamic range of ﬁxed-point representation is optimized to achieve better accuracy performance.
R ESULTS AND D ISCUSSION 5. A CCURACY IMPROVEMENT
Figure 3 depicts the classiﬁcation errors of the proposed RBNN model across three recursive iterations.
The initial bit-width of weights is eight.
In each series of data points in Figure 3, the leftmost point represents the initial neural network, i.e., with 2 layers of 200 hidden units and 198,800 weights ( = 784 · 200 + 200 · 200 + 200 · 10 ).
At this point, the storage requirement, deﬁned as the ratio of total storage bits to the number of weights, is 8 bits/weight.
The network at this point, is equivalent to one trained by the conventional BNN model speciﬁcally following Courbariaux et al.
The second leftmost data point in the series is the neural network after the ﬁrst recursive iteration.
The network size is enlarged by twice, resulting in the 784 × 400 × 400 ×network.
This reduces storage requirement to 4 bits/weight.
Compared to the initial BNN, the enlarged BNN achieves ∼ 0.7% and ∼ 0.4% reduction in training and test error rate, respectively.
Finally, after three recursive iterations, the size of the neural network becomes 784 × 800 × 800 ×(555,800 weights).
It marks the storage requirement as small as 2 bits/weight to achieve the test error of 2.17%.
This accuracy is as good as the fully-connected network using 4 × times more data storage for weights, trained by the conventional BNN model in (Courbariaux et al.
Note that we have various ways of using the recycled data storage to enlarge the neural network in the proposed RBNN model.
As shown in Figure 1, we chose a ”tiled” approach where no connections are made among incremental BNN s. This is because it is easier to implement the algorithm in hardware or map it onto the conventional CPU and GPU (see Appendix A).
In Appendix D, we have the RBNN to train a fully-connected DNN.
The results show that with same size of total data storage for weights, both tiled and fully-connected exhibit similar test error.
[Figure]
5. S TORAGE AND ARITHMETIC COMPLEXITY
To evaluate the storage and arithmetic complexity of the proposed RBNN, we trained multiple single-hidden-layer DNNs using the proposed RBNN and the conventional BNN model (Courbariaux et al.
For the conventional model, we considered BNN containing 100 tohidden neurons and 6 to 16 bit weight precisions.
For the proposed model, we considered 100 to 800 initial hidden neurons and 12 to 16 bit initial weight precisions.
Those DNNs require 116 kB to 1.2 MB data storage for weights.
Figure 4 shows the results of this experiment: the proposed model can achieve 1% less test error than the conventional model using the similar amount of data storage.
To achieve the similar test error, the proposed RBNN model requires 3-4 × less data storage than the conventional BNN model.
[Figure]
Test error (%)
[Figure]
Initial
[Figure]
[Table]
[Figure]
Storage requirement 2. Test error (%) 2.
[Table] [Table]
[Table]
[Table] [Table]
inference
[Table]
Table 2 shows the detail comparisons of six neural networks out of the 16 networks shown in Figure 4, three of which are trained by the proposed RBNN model ( R 1 , R 2 , R 3 ) and the other three by the conventional BNN model ( B 1 , B 2 , B 3 )
(Courbariaux et al.
We compare the arithmetic complexity for training and inferring. For training, to achieve similar accuracy performance ( Rand B 1 ; R 3 and B 2 ), RBNN requires around twice as many add and shift operations as conventional BNN does.
On the other hand, RBNN and BNN have the same amount of multiply operations.
Since the complexity of multiplication is much higher than add and shift, it is important not to increase the number of multiplications. For inference, both RBNN and BNN have the same amount of shift and add operations.
Inference requires no multiplication since MAC uses binary information of weights. [Table] [Table]
E NERGY C ONSUMPTION S AVINGS
In Table 2, we also compare the energy dissipations for training.
We calculate the energy dissipation of training the network with one data for one epoch. Total energy dissipation E total is calculated based on: [Equation] where E arith is the energy dissipation of arithmetic operation, and E acc is the energy dissipation of weight access from data storage.
E arith comprises three parts: energy dissipation of shift, add, and multiply. Each part is calculated by multiplying the amount of arithmetic operation ( N shift , N add , and N multiply ) and the energy cost of each arithmetic operation ( E shift , E add , and E mult ), as follows, [Equation]
E acc is calculated by multiplying twice of the total number of weights, energy cost of accessing one weight and the number of recycling iteration for the RBNN. This is because the weights are accessed once in forward and once in backward propagation. Since for the conventional BNN, the network size can be too large to store all the weights in on-chip SRAM, E acc can be divided into two parts: energy cost of accessing weights from SRAM and DRAM, i.e. [Equation]
Note that for the conventional BNN, N iteration equals to one because there’s no data storage recycling iteration. As shown in Table 3, energy cost of each operation is based on the 45nm CMOS process, presented in Han et al.
We normalized the energy costs to the bit-widths of operations, quadratically for multiplication and linearly for all the other operations. The main difference is that the BNN requires to access DRAM in addition to SRAM for weights whereas our RBNN can utilize SRAM to achieve certain test error. This is because we need 4 × less data storage for the same classiﬁcation accuracy (e.g. R 3 vs B 2 ).
As shown in Table 3, DRAM access consumes 173 × more energy than SRAM access, and 1422 × than multiplication. This entitles the RBNN consume 100 × less energy for training.
C ONCLUSION AND F UTURE W ORK
This paper presents a new learning model for on-device training with limited data storage.
The proposed RBNN model efﬁciently uses limited on-chip data storage resources by recycling the part of data storage that would have been wasted in conventional BNN model, to add and train more weights to a neural network classiﬁer.
We veriﬁed the proposed model with MLP-like DNN and CNN classiﬁers on the MNIST and VAD benchmark under the typical embedded device storage constraints.
The results of MLP-like DNNs on MNIST show that the proposed model achievesbits/weight storage requirement while achieving 1% less test error as compared to the conventional BNN model for the same storage constraint.
Our proposed model also achieves 4 × less data storage than the conventional model for the same classiﬁcation error.
The similar to greater savings are veriﬁed with the CNN classiﬁers and the VAD benchmarks. We expect the future work of further reduce computation complexity, such as binarization of activation function of BNN (Courbariaux et al.
We also expect to apply the RBNN model to the ensembles of neural networks (Zhou et al.
(2002), and the mixture of experts (Shazeer et al.
A I MPLEMENTATION OF S UB - WORD O PERATION OF RBNN
In the proposed RBNN model, each word contains both non-plastic and plastic weights but we need to update only the plastic weights.
We can implement this sub-word operation using the mask and bitwise logical operations which are widely supported in the conventional CPUs and GPUs.
Figure 5 illustrates an example of a possible implementation. We assume the word size of eight bits, where at the exemplary moment three already-trained weights take the top three bits of the word and the weight currently under training takes the remaining ﬁve bits. We fetch this weight word from the storage.
We also generate/fetch a mask word synp mask which stores 11100000 in this example. We bitwise-AND the weight word with the mask word to produce a temporary word synp ﬁx . We do the same with the bitwise inverse of the mask word to another temporary word synp plsb , which is then updated via the RBNN model.
Note that synp ﬁx is not changed.
These two words, then, combined via a bitwise XOR operation to produce a word synp out . This completes one training epoch.
As compared to the conventional BNN model, the proposed RBNN additionally requires only two bitwise AND and one bitwise XOR operations per training epoch. These operations are supported in modern CPUs and GPUs and their cost is not signiﬁcant. [Figure] [Figure]
synp
[Figure]
AND ~ synp_
[Figure]
Learning synp
[Figure]
synp_plsb XOR synp_update synp_out synp_in B A PPLICATION OF RBNN TO C ONVOLUTIONAL N EURAL N ETWORKS B. E XPERIMENT S ETUP
We applied the proposed RBNN model to the LeNet5 Convolutional Neural Network (LeCun et al.
The network has two convolution layers, one having six 5-by-5 and the other having twelve 5-by-5 feature maps.
Each of the convolution layers is followed by a 4 × dowsampling averagepooling layer.
The LeNet5 has a fully-connected (FC) classiﬁer consist of one input, one hidden, and one output layer.
As in Courbariaux et al.
, we used binary information of weights for both of the convolutional layers and the FC classiﬁer during forward and backward propagations and ﬁxed-point weights for updating.
We applied the proposed RBNN model on the hidden layer of the FC classiﬁer.
B. R ESULTS A NALYSIS
We trained multiple CNN classiﬁers for the MNIST benchmark while changing conﬁgurations of the FC classiﬁer.
For the proposed RBNN model, we considered the FC classiﬁer containingto 800 initial hidden neurons and 16 bit initial weight precision.
For the conventional BNN model
Courbariaux et al. , we considered the FC classiﬁer containing 200 to 2,500 hidden neurons and 16-bit weight precision.
Those CNNs require 81 kB to 1.01 MB data storage for all the weights in the convolutional layers and the FC classifer.
Figure 6 shows the trade-off between the test error and the weight storage requirement of those CNNs.
The proposed RBNN model can achieve 2.4% less test error than the conventional BNN model for the same amount of data storage for weights.
For the similar test error of ∼ 2.5%, the proposed RBNN model requires more than 6 × less weight data storage than the conventional BNN model. [Figure]
S t o r a g
e
r e q u
i r
e m e n t ( M B )
[Figure] [Figure]
C A PPLICATION OF RBNN TO V OICE A CTIVITY D ETECTION
We applied the RBNN to train MLP-like DNN classiﬁers for the VAD benchmark.
The VAD benchmark is based on Aurora 4 (Pearce & Picone (2002)), which has 7,133 utterances fromspeakers. It also contains ﬁve noise scenarios: bus, cafe, park, river, and trafﬁcs.
The signal-to-noise ratio of the data used in the experiment is 10 dB. We use the same DNN conﬁgurations used in
Sec. 4.2.
The input to the DNN (features) are ﬁve frames of 16-dimensional band-pass ﬁlter-bank output commonly used in other works Zhang & Wang.
Table 4 summarizes the classiﬁer models trained by the RBNN and the conventional BNN methods. For each noise scenario we list only the models that achieve the similar test errors. The experiment conﬁrms that the proposed RBNN model can save up to 9 × data storage than the conventional BNN for the similar level of detection accuracy.
D RBNN IN F ULLY -C ONNECTED DNN S YSTEMS
In Sec. 3.1, we have the RBNN to train a tiled feedforward DNN. In this section, we experiment to train a fully-connected DNN using the proposed RBNN. Note that the fully-connected DNN is only one way of many other possible approaches on how to recycle the data storage to expand a neural network.
Figure 7 illustrates the training process. It starts with an exemplary DNN whose initial size is 1 × 2 × 2 × 1.
Each weight is n bits.
As shown in the ﬁrst two sub-ﬁgures, we ﬁrst train a tiled
DNN as we did in Sec. Then, we start to add the weights that connect between tiles (the last two sub-ﬁgures), again by recycling the data storage from the binarization in each recursive iteration.
[Table]
Figure 8 shows the scaling of test errors over the recursive iterations. The total data storage constraint and structure of the initial neural network in this experiment are the same as ones of the experiment in Sec. 5.1, which are 198.8kB and 784 × 200 × 200 × 10, respectively.
The ﬁrst three iterations expands the DNN in the tiled manner and the last four iterations adds weights that connect the tiles. In the forth iteration, the neurons of the ﬁrst hidden layer of the ﬁrst tile are connected to the neurons in the second hidden layers of all the other tiles, making the DNN 1/4-connected.
The bit-width of weights are 7 bits. This is because fewer weights are added than the ﬁrst three iterations. In following iterations, the hidden layers of the ﬁrst hidden layer of the rest of the tiles are connected to the hidden neurons of the second hidden layer in the same way as the forth iterations.
Figureshows that the fully-connected DNN classiﬁer has the similar accuracy performance as the one has tiled structure in Sec. 5.1.
[Figure]
1 . 4 M 2 .2 .2 .2 .2 .3 .3 .
[ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 5 b 3 / 4 - c o n n e c t e d [ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 6 b h a l f - c o n n e c t e d [ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 7 b 1 / 4 - c o n n e c t e d [ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 4 b f u l l y - c o n n e c t e d [ 7 8 4 - 8 0 0 - 8 0 0 - 1 0 ] 5 b t
i l e d [ 7 8 4 - 6 0 0 - 6 0 0 - 1 0 ] 6 b t i l e d [ 7 8 4 - 4 0 0 - 4 0 0 - 1 0 ] 7 b t i l e d
T e s t e r r o r ( % )
# o f W e i g h t s [ 7 8 4 - 2 0 0 - 2 0 0 - 1 0 ] 8 b t i l e d