Efﬁcient Computation of Quantized Neural Networks by {− 1 , +1 } Encoding Decomposition
Abstract
Complex neural networks require extensive computing resources, and can not beefﬁciently applied to embedded devices such as mobile phones, which seriouslylimits their applicability.
To address this problem, we propose a novel encodingscheme by using {− 1 , +1 } to decompose quantized neural networks (QNNs) intomulti-branch binary networks, which can be efﬁciently implemented by bitwiseoperations ( xnor and bitcount ) to achieve model compression, computationalacceleration and resource saving.
Our method can achieve 1 ∼ 64 × speedupand 4 ∼ 32 × memory saving over its full-precision counterparts, therefore, userscan easily achieve different encoding precisions arbitrarily according to theirrequirements and hardware resources.
Our mechanism is very suitable for the useof FPGA and ASIC in terms of data storage and computation, which provides afeasible idea for smart chips.
We validate the effectiveness of our method on bothlarge-scale image classiﬁcation (e.g., ImageNet) and object detection tasks.
Introduction
Deep Neural Networks (DNNs) have been successfully applied in many ﬁelds, especially in imageclassiﬁcation, object detection and natural language processing.
Because of numerous parameters andcomplex model architectures, huge storage space and considerable power consumption are needed.
However, for mobile phones and embedded platforms, whose resources are limited, it’s hard toachieve satisfactory performance for industrial applications.
With the rapid development of DNNs more and more computing resources are needed and the requirements for hardware are becominghigher and higher.
In order to improve the energy efﬁciency of hardware achieve model compression or computationalacceleration many solutions have been proposed, such as network sparse and pruning [7, 22, 12, 25,21], low-rank approximation [5, 11, 23], architecture design [20, 10, 8, 19, 16], model quantization[1, 13, 3, 9, 18, 14] and so on.
[17, 6, 28] constrain their weights to {− 1 , +1 } or {− 1 , 0 , 1 } andachieve limited acceleration by using simple accumulation instead of complicated multiplicationaccumulations.
In particular, [2, 18, 4, 27, 24] quantize activation values and weights to bits anduse bitwise logic operations to achieve extreme acceleration ratio in inference process but they aresuffering from signiﬁcant performance degradation.
However, most models are proposed for ﬁxedprecision, and can not extend to other precision models.
They easily fall into local optimal solutionsand face slow convergence speed in training process.
In order to bridge the gap between low-bit andfull-precision and be applied to many cases, we propose a novel encoding scheme of using {− 1 , +1 }to easily decompose trained QNNs into multi-branch binary networks, therefore, the inference processcan be efﬁciently implemented by bitwise operations to achieve model compression, computationalacceleration and resource saving.
Model Decomposition
As the basic computation in most neural network layers, matrix multiplication costs lots of resourcesand also is the most time consuming operation.
We propose a decomposition method to accelerate matrix multiplication.
Suppose there are two vectors x = [x 1 , x 2 , ..., x N ] T and w =
[w 1 , w 2 , ..., w N ] Tcomposed by N natural numbers, where x i , w i ∈ { 0 , 1 , 2 , ... } for i = 1 , 2 , ..., N .
The dot product ofthe two vectors can be represented as follows:
[Equation]
42 All of the above operations consist of N multiplications and ( N − 1) additions.
In the moderncomputer, data storage and processing are represented in binary form, and thus each bit can beexpressed in the two states (0 or 1).
Based on the above mechanism, x can be encoded to binary formusing M bit, i.e.,
[Equation]
Then the right-hand side of (2) can be converted into the following form:
[Equation]
where x j = (cid:80) Mm = 1 2 m − 1 · x jm , x jm ∈ { 0 , 1 } , x i =[x 1 i [Equation]
x 2 i
[Equation]
..., x Ni ] .
In such an encoding scheme, the represented states are not more than 2 M .
In addition, we encode wwith K -bit numbers in the same way.
Therefore, the inner product of the two vectors can be computedas follows:
[Equation]
52 From the above formulas, the dot product is decomposed into M × K sub-operations, in which everyelement is 0 or 1.
Therefore, we can use multiple bitwise operations ( xnor and bitcount ) to achievea complex computation.
Because of the restriction of encoding and without using the sign bit, theabove representation can only be used to encode non-negative integers.
However, it’s impossible inneural networks that all the weights and activations are restricted to natural numbers.
In order toextend encoding space to negative integer and reduce the computational complexity, we propose anew encoding scheme, which uses {-1, +1} as the basic elements of our encoder rather than {0, 1}.
This operation mechanism can be suitable for all vector/matrix multiplications.
In addition to fullyconnected layer, convolution and deconvolution layers are also suitable in neural networks.
M-bit Encoding Functions
As an important part in neural networks, activation function can enhance the nonlinear characterizationof the networks.
In our proposed model decomposition method, encoding function plays a criticalrole and can encode input data to multi bits (-1 or +1).
Those numbers represent the encoding ofinput data.
Therefore, the dot product can be computed by the formula (6).
Without other judgmentand mapping calculation, we use trigonometric functions as the basic encoding functions.
In the end,we use the sign function to hard divide to -1 or +1.
The mathematical expression can be formulatedas follows:
[Equation]
where ϕ MM ( x ) denotes the encoding function of highest bit of M BitEncoder and ϕ mM ( x ) representsthe other bits.
The periodicity is obviously different from others because it needs to denote morestates.
Experiments
In this section, we use the same network architecture described in [17, 2] for CIFAR-10 and choose Resnet-18 as the basic network for ImageNet.
QNNs are facing the problem that the derivative is notdeﬁned, thus traditional gradient optimization methods are not applicable.
It will be more hard to trainon large-scale training sets (e.g., ImageNet), thus parameter initialization is particularly important.
In particular, the well-trained full-precision model parameters activated by ReLU can be directlyused as initialization parameters for our 8-bit quantized network.
After ﬁne-tuning dozens of epochs,8-bit quantized networks can be well-trained.
Similarly, we use the 8-bit model parameters as theinitialization parameters to train 7-bit quantized networks, and so on.
We use the loss computed byquantized parameters to update full precision parameters described as the straight-through estimator[26].
Table 1 lists the performance (e.g., accuracy, speedup ratio, memory saving ratio) of our methodand several typical models mentioned above.
The accuracies were achieved after dozens of timesﬁne-tuning.
If continue training those networks, we can reach slightly better performance.
Wealso use the trained Resnet-18 with the Single Shot MultiBox Detector (SSD) framework [15] tovalidate object detection tasks.
We also use the trained model parameters in ImageNet classiﬁcationto initialize SSD parameters, after dozens of times ﬁne-tuning the results are listed in Table 1.
Ourmethod provides 1 ∼ 64 × speedup and 4 ∼ 32 × memory saving over its full-precision counterparts.
In theory, users can achieve 64 MK × speedup and 32 K × memory saving by constraining activation valuesto M-bit and weights to K-bit, where M, K ∈ { 1 , 2 , ..., 8 } .
Therefore, our method can provide atleast 64 available options, which don’t conﬁned to experiments.
Here, we use 64-bit binary operationin one clock cycle.
If those decompositions are implemented in the FPGA or ASIC platform, thespeedup ratios can be much bigger.
[Table]
Conclusions
In this paper [Table] we proposed a novel encoding scheme of using {-1, +1} to decompose QNNs intomulti-branch binary networks, in which we used bitwise operations ( xnor and bitcount ) to achievemodel compression, computational acceleration and resource saving.
In particular, we can usethe high-bit model parameters to initialize a low-bit model and achieve good results in variousapplications.
Therefore, our networks can deploy QNNs to mobile phones and embedded devicesaccording to user’s accuracy requirements and hardware resources.
This special data storage andcalculation mechanism can yield great performance in FPGA and ASIC, and thus our mechanism is afeasible idea for smart chips.
Future works will focus on improving the hardware implementationand chip technology.