Efﬁcient Computation of Quantized Neural Networks by f(cid:0) 1 ; +1 g Encoding Decomposition
Anonymous Author(s)
Abstract
Deep neural networks require extensive computing resources, and can not be efﬁciently applied to embedded devices such as mobile phones, which seriouslylimits their applicability.
To address this problem, we propose a novel encodingscheme by using f(cid:0) 1 , +1 g to decompose quantized neural networks (QNNs) intomulti-branch binary networks, which can be efﬁciently implemented by bitwiseoperations ( xnor and bitcount ) to achieve model compression, computational acceleration and resource saving.
Our method can achieve at most (cid:24) 59 (cid:2) speedupand (cid:24) 32 (cid:2) memory saving over its full-precision counterparts. Therefore, userscan easily achieve different encoding precisions arbitrarily according to their requirements and hardware resources.
Our mechanism is very suitable for the useof FPGA and ASIC in terms of data storage and computation, which provides afeasible idea for smart chips.
We validate the effectiveness of our method on bothlarge-scale image classiﬁcation (e.g., ImageNet) and object detection tasks.
Introduction
Deep Neural Networks (DNNs) have been successfully applied in many ﬁelds, especially in imageclassiﬁcation, object detection and natural language processing.
Because of numerous parametersand complex model architectures, huge storage space and considerable power consumption are needed.
However, for mobile phones and embedded platforms, whose resources are limited, it’s hard toachieve satisfactory performance for industrial applications.
With the rapid development of DNNs,more and more computing resources are needed and the requirements for hardware are becominghigher and higher.
In order to improve the energy efﬁciency of hardware achieve model compression or computational acceleration, many solutions have been proposed, such as network sparse and pruning[7, 22, 12, 25, 21], low-rank approximation [5, 11, 23], architecture design [20, 10, 8, 19, 16],model quantization [1, 13, 3, 9, 18, 14], and so on.
[17, 6, 28] constrain their weights to f(cid:0) 1 , +1 gor f(cid:0) 1 , 0 , 1 g and achieve limited acceleration by using simple accumulation instead of complicatedmultiplication-accumulations.
In particular, [2, 18, 4, 27, 24] quantize activation values and weightsto bits and use bitwise logic operations to achieve extreme acceleration ratio in inference process butthey are suffering from signiﬁcant performance degradation.
However, most models are proposedfor ﬁxed precision, and can not extend to other precision models.
They easily fall into local optimalsolutions and face slow convergence speed in training process.
In order to bridge the gap betweenlow-bit and full-precision and be applied to many cases, we propose a novel encoding scheme ofusing f(cid:0) 1 , +1 g to easily decompose trained QNNs into multi-branch binary networks. Therefore,the inference process can be efﬁciently implemented by bitwise operations to achieve model compression, computational acceleration and resource saving.
Model Decomposition
As the basic computation in most neural network layers, matrix multiplication costs lots of resourcesand also is the most time consuming operation.
Modern computers store and process data in binary format, thus non-negative integers can be directly encoded by {0, 1}.
We propose a novel decomposition method to accelerate matrix multiplication as follows: Let x = [x 1 , x 2 , ..., x N ] T andw =
[w 1 , w 2 , ..., w N ] T be two vectors of non-negative integers, where x i , w i 2 f 0 , 1 , 2 , ... g fori =1 , 2 , ..., N .
The dot product of those two vectors can be represented as follows:
[Equation]
All of the above operations consist of N multiplications and ( N (cid:0) 1) additions.
Based on the aboveencoding scheme, the vector x can be encoded to binary form using M bits, i.e.,x
[Equation]
Then we convert the right-hand side of (2) into the following form:
[Equation]
where x j = ∑ Mm =1 2 m (cid:0) 1 (cid:1) x
jm , x jm 2 f 0 , 1 g , x i =[x 1 i , x 2 i , ..., x Ni ] .
In such an encoding scheme, the number of represented states is not greater than 2 M .
In addition,we encode another vector w with K -bit numbers in the same way.
Therefore, the dot product of thetwo vectors can be computed as follows:
[Equation]
From the above formulas, the dot product is decomposed into M (cid:2) K sub-operations, in which eachelement is 0 or 1.
Because of the restriction of encoding and without using the sign bit, the aboverepresentation can only be used to encode non-negative integers.
However, it’s impossible to limitthe weights and the values of the activation functions to non-negative integers.
In order to encodeboth positive and negative integers, we propose a novel encoding scheme, which uses {-1, +1} asthe basic elements rather than {0, 1}. Then we can use multiple bitwise operations (i.e., xnor andbitcount ) to effectively achieve the above vector multiplications.
Our operation mechanism can besuitable for all vector/matrix multiplications.
Besides fully connected layers, our mechanism is alsosuitable for convolution and deconvolution layers in deep neural networks.
M-bit Encoding Functions
As an important part in neural networks, activation function can enhance the nonlinear characterization of the networks.
In our proposed model decomposition method, encoding function plays acritical role and can encode input data to multi bits (-1 or +1).
Those numbers represent the encoding of input data.
Therefore, the dot product can be computed by the formula (6).
Without otherjudgment and mapping calculation, we use trigonometric functions as the basic encoding functions.
In the end, we use the sign function to hard divide to -1 or +1.
The mathematical expression can beformulated as follows:
[Equation]
where φ MM ( x ) is the encoding function of the highest bit of M BitEncoder (i.e., m = M ).
Theperiodicity is obviously different from others because it needs to denote more states.
Experiments
In this section, we use the same network architecture described in [17, 2] for CIFAR-10 and choose
ResNet-18 as the basic network for ImageNet.
It is very hard to train on large-scale training sets (e.g., ImageNet), and thus parameter initialization is particularly important.
In particular, the well-trainedfull-precision model parameters activated by ReLU can be directly used as initialization parametersfor our 8-bit quantized network.
After ﬁne-tuning dozens of epochs, 8-bit quantized networks can bewell-trained.
Similarly, we use the 8-bit model parameters as the initialization parameters to train 7bit quantized networks, and so on.
We use the loss computed by quantized parameters to update fullprecision parameters described as the straight-through estimator [26].
Table 1 lists the performance(e.g. , accuracy, speedup ratio, memory saving ratio) of our method and several typical models mentioned above.
The accuracies were achieved after dozens of times ﬁne-tuning.
If continue trainingthose networks, we can reach slightly better performance.
We also use the trained ResNet-18 withthe Single Shot MultiBox Detector (SSD) framework [15] to validate object detection tasks.
Wealso use the trained model parameters in ImageNet classiﬁcation to initialize SSD, and report theexperimental results in Table 1 after dozens of times ﬁne-tuning. We analyze the theoretical performance of our encoding scheme. The theoretical speedup and modelcompression ratios are given in the following table.
Thus, our method can obtain at most (cid:24) 59 (cid:2)speedup and (cid:24) 32 (cid:2) memory saving over its full-precision counterparts.
It can achieve (cid:24) 59 /MK (cid:2)speedup and (cid:24) 32 /K (cid:2) memory saving by constraining activation values to M -bit and the values ofweights to K -bit, where M, K 2f 1 , 2 , ..., 8 g .
In fact, our method can provide 64 available encodingchoices, and hence our encoded network with different encoding precisions has different calculationspeed, memory requirements and experimental precisions.
Here, we use 64-bit binary operationin one clock cycle.
If those decompositions are implemented in the FPGA or ASIC platform, thespeedup ratios can be much higher.
[Table]
Conclusions
In this paper, we proposed a novel encoding scheme of using {-1, +1} to decompose QNNs intomulti-branch binary networks, in which we used bitwise operations ( xnor and bitcount ) to achievemodel compression, computational acceleration and resource saving.
In particular, we can use thehigh-bit model parameters to initialize a low-bit model and achieve good results in various applications.
Thus, users can easily achieve different encoding precisions arbitrarily according to theirrequirements (e.g., accuracy and speed) and hardware resources (e.g., memory).
This special mechanism of data storage and calculation can yield great performance in FPGA and ASIC, and thus ourmechanism is a feasible idea for smart chips.
Future works will focus on improving the hardwareimplementation and chip technology, and exploring some ways to automatically select proper bitsfor various network architectures (e.g., VGG and ResNet).