A DAPTIVE Q UANTIZATION OF N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the state-of-the-art accuracy of Deep Neural Networks (DNN) in various classiﬁcation problems, their deployment onto resource constrained edge computing devices remains challenging, due to their large size and complexity.
Several recent studies have reported remarkable results in reducing this complexity through quantization of DNN models.
However, these studies usually do not consider the change in loss when performing quantization, nor do they take the disparate importance of DNN connections to the accuracy into account.
We address these issues in this paper by proposing a new method, called adaptive quantization, which simpliﬁes a trained DNN model by ﬁnding a unique, optimal precision for each connection weight such that the increase in loss is minimized.
The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each weight and assign it a precision accordingly.
Since this problem uses linear functions, it is computationally cheap and, as we will show, has a closed-form approximate solution.
Experiments on MNIST, CIFAR, and SVHN datasets showed that the proposed method can achieve near or better than state-of-the-art reduction in model size with similar error rate.
Furthermore, it can achieve compressions close to ﬂoating-point model compression methods without loss of accuracy.
I NTRODUCTION
Deep Neural Networks (DNNs) have achieved incredible accuracy in applications ranging from computer vision (Simonyan & Zisserman, 2014) to speech recognition (Hinton et al., 2012) and natural language processing (Devlin et al., 2014).
One of the key enablers of the unprecedented success of DNNs is the availability of very large model sizes .
While the increase in model size improves the classiﬁcation accuracy, it inevitably increases the computational complexity and memory requirement needed to train and store the network.
This poses challenges in deploying these large models in resource-constrained edge computing environments, such as mobile devices.
This challenge motivates neural network compression , which exploits the redundancy of neural networks to achieve drastic reductions in model sizes.
The state-of-the-art neural network compression techniques include weight quantization (Courbariaux et al., 2015), weight pruning (Han et al., 2015), weight sharing (Han et al., 2015), and low rank approximation (Zhao et al., 2017).
For instance, weight quantization has previously shown good accuracy with ﬁxed-point 16-bit and 8-bit precisions (Suda et al., 2016; Qiu et al., 2016).
Recent work attempts to push that even further towards reduced precision and has trained models with 4-bit, 2-bit, and 1-bit precisions using quantized training methods (Hubara et al., 2016b;
Zhou et al., 2016; Hubara et al., 2016a; Courbariaux et al., 2015).
Although these quantization methods can signiﬁcantly reduce model complexity, they generally have two key constraints.
First, while performing quantization they do not take the change in the loss function into account.
This can result in a disparity between the quantized model accuracy and the objective accuracy.
Previous works get around this issue by devising quantized learning schemes that essentially try to solve the problem after the fact.
Such schemes have the disadvantage of converging very slowly compared to full-precision learning methods.
Second, they do not differentiate between different connections in their approaches.
This is despite the fact that it has been shown that different weights are not equally important to the accuracy of the network ().
Disregarding this difference constraints these methods to either use higher precisions for quantization or use deeper networks.
Both of these solutions diminish the compression.
In this paper, we address the aforementioned issues by proposing adaptive quantization .
To take the disparate importance of different connections into account, this method quantizes each weight of a trained network by a unique precision.
This way, weights that impact the accuracy the most can be represented using a higher precision, while low-impact weights are represented with only fewer bits or are pruned.
Consequently, it can reduce the model size signiﬁcantly while maintaining a certain accuracy.
The proposed method monitors the accuracy by incorporating the loss function into an optimization problem to minimize the models.
The output of the optimization problem is an error margin associated to each weight.
This margin is computed based on the loss function gradient of the weight and is used to determine its precision.
We will show that the proposed optimization problem has a closed-form approximate solution, which can be iteratively applied to the same network to minimize its size, constraining the loss.
We test the proposed method using three classiﬁcation benchmarks comprising MNIST, CIFAR-10, and SVHN.
We show that, across all these benchmarks, we can achieve near or better compressions compared to state-of-the-art quantization techniques.
Furthermore, we can achieve similar compressions compared to state-of-the-art pruning and weight-sharing techniques which inherently require more computational resources for inference.
P ROPOSED Q UANTIZATION A LGORITHM
Despite their remarkable classiﬁcation accuracies, large DNNs assimilate redundancies.
Several recent works have studied these redundancies by abstracting the network from different levels and search for, in particular, redundant ﬁlters (DenseNets ()) and redundant connections (Deep Compression ()).
In this work, we present an even more ﬁne-grained study of redundancy and extend it to ﬁxed-point quantization of network parameters.
That is, we approximate the minimum size of the network when each parameter is allowed to have a distinct number of precision bits.
Our goal here is to represent parameters with high precisions only when they are critical to the accuracy of the network.
In this sense, our approach is similar to the weight pruning () which eliminated all but the essential parameters, producing a sparse network.
In the rest of this section, we ﬁrst formally deﬁne the problem of minimizing the network size as an optimization problem.
Then, we propose a trust region technique to approximately solve this problem.
We will show that, in each iteration of the trust region method, our approach has a straightforward closed-form solution.
Finally, we will explain how the hyper parameters in the algorithm are chosen and discuss the implications of the proposed technique.
2. P ROBLEM D EFINITION
A formal deﬁnition of the optimization problem that was discussed previously is presented here.
Then, key characteristics of the objective function are derived.
We will use these characteristics later when we solve the optimization problem.
We minimize the aggregate bit-width of all network parameters while monitoring the training loss function.
Due to the quantization noise, this function deviates from its optimum which was achieved by training.
This effect can be controlled by introducing an upper bound on the loss function in order to maintain it reasonably close to the optimum.
Consequently, solution of this minimization problem represents critical parameters with high precision to sustain high accuracy and assigns low precisions to ineffectual ones or prunes them.
The problem described here can be deﬁned in formal terms as below.
[Equation]
Here, n is the number of model parameters, W = [ ω 1 ...ω n ] T is a vector of model parameters, ω i is the i -th model parameter, and the function N q ( ω i ) is the minimum number of bits required to represent ω i in its ﬁxed-point format.
As a result, N Q ( W ) is the total number of bits required to represent the model.
In addition, l ( W ) is the loss of the model over the training set (or a minibatch of the training set).
l ( ≥ l ( W 0 ) ) is a constant upper bound on the loss of the neural network, which is used to bound the accuracy loss, and W 0 is the vector of initial model parameters before quantization.
The optimization problem presented in equations 1 and 2 is difﬁcult to solve because of its non-smooth objective function.
However, a smooth upper limit can be found for it.
Lemma 2.1.
Given a vector of network parameters W , a vector of tolerance values T =
[ τ 1 ...τ n ] T , and a vector of quantized network parameters W q =
[ ω q 1 ...ω
qn ] T , such that each ω qi , i ∈
[1 , n ] solves the constrained optimization function:
[Equation]
[Equation]
We have that:
[Equation]
Where N uQ ( T ) is a smooth function, deﬁned as:
[Equation]
Proof.
The problem of equation 4 can be easily solved using algorithm 1, which simply checks all possible solutions one-by-one.
Here, we allocate at most 32 bits to each parameter.
This should be more than enough as previous works have demonstrated that networks can be easily quantized to even 8 bits without signiﬁcant loss of accuracy.
Algorithm 1 Quantization procedure QUANTIZE _ PARAMETER ( ω i , τ i ) ω
q ← N q ←while N q ≤ 32 do if | ω q − ω i | ≤ τ i then break end if ω q ← round (2 N q ω ) / 2 N q
N q ← N q +end while return N q , ω q end procedure
Algorithm 1 estimates ω
i with a rounding error of up to 1 2 Nq +1 .
This error needs to be consistent with the constraint of equation 4:
[Equation]
The minimization in equation 4 entails that:
[Equation]
Consequently:
[Equation]
2. S OLVING THE O PTIMIZATION P ROBLEM
The optimization problem of equations 1 and 2 presents two key challenges.
First, as was mentioned previously, the objective function is non-smooth.
Second, not only is the constraint (equation 2) non-linear, but also it is unknown.
In this section, we present a method that circumvents these challenges and provides an approximate solution for this problem.
We sidestep the ﬁrst challenge by using the upper bound of the objective function, N uQ ( T ) .
Particularly, we approximate the minimum of N Q ( W ) by ﬁrst ﬁnding the optimal T for N uQ ( T ) and then calculating the quantized parameters using algorithm 1.
This optimization problem can be deﬁned as: minimize N uQ ( T ) such that if each ω i is perturbed by, at most, τ i , the loss constraint of equation 2 is not violated:
[Equation]
such that ∀ i ∈ [1 , n ] : | ∆ ω i | ≤ τ
i
[Equation]
[Equation]
It is important to note that although the upper bound N uQ ( T ) is smooth over all its domain, it is tight only when ∀ i ∈ [1 , n ] : τ i = 2 − k , k ∈ N .
This difference between the objective function and N uQ ( T ) means that it is possible to iteratively reduce N Q ( W ) by repeating the steps of the indirect method, described above, and improve our approximation of the optimum quantization.
Such an iterative algorithm would also help address the second challenge.
Speciﬁcally, it allows us to use a simple model of the loss function (e.g. a linear or quadratic model) as a stand-in for the complex loss function.
If in some iteration of the algorithm the model is inaccurate, it can be adjusted in following iteration.
In our optimization algorithm, we will use a linear bound of loss function and adopt a Trust Region method to track its accuracy.
Trust region optimization methods iteratively optimize an approximation of the objective function.
In order to guarantee the accuracy of this model, they further deﬁne a neighborhood around each point in the domain, called the trust region, inside which the model is a "sufﬁciently good" approximation of the objective function.
The quality of the approximation is evaluated each iteration and the size of the trust region is updated accordingly.
In our algorithm we adopt a similar technique, but apply it to the constraint instead of the objective function.
Lemma 2.2.
We refer to m l : R n → R as an accurate estimation of l ( W ) in a subdomain of l like
D ⊂ R n , if:
[Equation]
Let the radius ∆ specify a spherical trust region around the point W 0 where the loss function l is accurately estimated by its ﬁrst-order Taylor series expansion.
Then, the constraint of Equation 17, when (cid:107) T (cid:107) 2 ≤ ∆ , is equivalent to:
[Equation]
Where G =
[ g 1 ...g n ] T and g i
= | [ ∇ W l ] i | , ∀ i ∈ [1 , n ] .
Proof.
Since (cid:107) T (cid:107) 2 ≤ ∆ , then ∆ W as deﬁned in Equation 17 is inside the trust region.
Therefore:
[Equation]
Consequently, if we deﬁne such a trust region, we can simply the following problem as our subproblem in each iteration.
We will present the solution for this subproblem in the next section.
[Equation]
Algorithm 2 summarizes the proposed method of solving the original optimization problem (equation 1).
This algorithm ﬁrst initializes the trust region radius ∆ and the loss function bound l .
Then, it iteratively solves the subproblem of equation 17 and calculates the quantized parameters.
If the loss function corresponding the quantized parameters violates the loss bound of equation 2, it means that the linear estimation was inaccurate over the trust region.
Thus, we reduce the radius of the trust region and in the solve the subproblem over a smaller trust region in the next iteration.
In this case, the calculated quantized parameters are rejected.
Otherwise, we update the parameters.
In addition, we grow the trust region unless if it grows larger than an upper bound on this radius ( Delta ).
We use two measures to examine convergence of the solution: the loss function and the trust region radius.
If the loss function of the quantized model converges to the loss bound l or the trust region radius becomes very small, indicated by values (cid:15) and η , respectively.
Note that the protocol to update the trust region radius as well as the trust region convergence condition used in this algorithm here are commonly used in trust region methods.
k ←
Initialize ∆ , ∆ and l while l − l ( W k ) ≥ (cid:15) and ∆ ≥ η do
Solve Trust Region Subproblem (Section 2)
U =
Quantize ( W k , T k ) if l ( U ) ≤ l then
W k +1 ← U ∆ ← min (2∆ k , ∆)
else ∆ k +1 ← 12 ∆ k end if k ← k +end while
2. T RUST R EGION S UBPROBLEM
We can directly solve the subproblem of Equation 17 by writing its KKT conditions:
[Equation] [Equation]
The gradient of the objective function, ∇ N uQ = −
[Equation]
[Equation] [Equation] cannot be zero.
Hence, psi > 0 and m ul − l = 0 .
[Equation]
If for the resulting tolerance vector (cid:107) T (cid:107) > ∆ , we scale T so that its norm would be equal to ∆ .
This solution is correct only when g i > 0 for all i .
It is possible, however, that there exists some i for which g i = 0 .
In this case, we use the following equation to calculate the optimal solution T ∗ .
[Equation]
We treat singular points this way for three reasons.
First, a gradient of zero with respect to ω i means that the cost is not very sensitive to its value.
Thus, it might be possible to eliminate it quickly.
Second, this insensitivity of the cost to ω i means that large deviations in its value would not signiﬁcantly affect the cost.
Finally, setting the value of τ ∗ i to a large value relative to other tolerances reduces their values after normalization to the trust region radius.
Thus, the effect of a singularity on other parameters is reduced.
After ﬁnding tolerance values, we once again calculate T k by normalizing T ∗ to the trust region radius.
2. C HOOSING H YPER - PARAMETERS
The hyper-parameters, ∆ , and l , determin the speed of convergence and the ﬁnal classiﬁcation accuracy.
Smaller trust regions result in slower convergence, while larger trust regions produce higher error rates.
For ∆ , since remaining values could ultimately be represented by 0 bits (pruned), we choose 2 0 √ n .
Similarly, the higher the loss bound l , the lower the accuracy, while small values of loss bound prevent effective model size reduction.
We choose this value in our algorithm on the ﬂye.
That is, we start off by conservatively choosing a small value for l (e.g. l = l ( W 0 ) ) and then increase it everytime the algorithm 2 converges.
Algorithm 3 provides the details of this process.
At the begining the loss bound is chosen to be the loss of the ﬂoating-point trained model.
After quantizing the model using this bound, the bound value is increased by Scale .
These steps are repeated Steps times.
In our experiments in the next section Scale and Steps are set to 1 . 1 and 20 , respectively.
Algorithm 3 Adaptive Quantization
Initialize Steps and Scale [Equation] W 0 ) while Steps > 0 do
Quantize model using Algorithm 2.
Steps ← Steps −l ← l ∗ Scale end while
E VALUATION AND D ISCUSSION
We evaluate the performance of the proposed quantization method in compressing trained NN models using a set of classiﬁcation datasets.
In our evaluation, we train three networks using three common datasets.
Then, we use the adaptive quantization method to quantize these trained models.
In order to achieve the smallest model sizes, we apply the quantization method three times and after each time we retrain the network for one epoch in the full precision domain.
We will experimentally show that three passes of quantization is sufﬁcient for achieving good accuracy as well as model size reduction.
In this section, we discuss the benchmarks used in these experiments and analyze their quantization results.
3. B ENCHMARKS
We use MNIST (LeCun et al., 1998b), CIFAR-10 (Krizhevsky & Hinton, 2009), and SVHN (Netzer et al., 2011) in our experiments.
MNIST is a small dataset containing 28 × 28 images of handwritten digits from 0 to 9 .
For this dataset, we use the LeNet-5 network (LeCun et al., 1998a).
For both
CIFAR-10, a dataset of 32 × 32 images in 10 classes, and SVHN, a large dataset of 32 × 32 real-world images of digits, we employ the same organization of convolution layers and fully connected layers as networks used in BNN (Hubara et al., 2016a).
These networks are based on VGG (Simonyan &
Zisserman, 2014) but have parameters in full precision.
The only difference is that in the case of
CIFAR-10, we use 4096 neurons instead of the 1024 neurons used in BNN, and we do not use batch normalization layers.
We train these models using a Cross entropy loss function.
For CIFAR-10, we also add an l2 regularization term.
Table 1 shows the speciﬁcations of the baseline trained models.
[Table]
3. E XECUTION T IME
The key contributors to the computational load of the proposed technique are back propagation (to calculate gradients) and quantization (algorithm 2.
Both these operations can be completed in O ( n ) complexity.
We implement the proposed quantization on Intel Core i7 CPU (3.5 GHz) with Titan X
GPU performing training and quantization.
The timing results of the algorithm have been summarized in table 2.
[Table]
3. Q UANTIZATION R ESULTS
We evaluate the performance of the quantization algorithm by analyzing the compression rate of the models and their respective error rates after each pass of quantization without retraining.
As discussed in section 2, the quantization algorithm will try to reduce parameter precision while maintaining a minimum classiﬁcation accuracy loss.
Here we present the results of these experiments.
Figure 1a depicts the error rate of LeNet trained on the MNIST dataset as it is compressed through three passes of adaptive quantization and retraining.
As this ﬁgure shows, the second and third passes tend to have smaller error rates compared to the ﬁrst pass.
But, at the end of the second pass the error rate is higher than the ﬁrst one.
This can be improved by increasing the number of epochs for retraining or choosing a lower cost bound ( C ).
By the end of the third pass, the highest compression rate is achieved.
However, due to its small difference in compression rate compared to its preceding pass, we do not expect to achieve more improvements by continuing the retraining process.
We also evaluate the convergence speed of the algorithm by measuring the compression rate ( 1 − sizeoforiginalmodel ) of the model after each iteration.
The results of this experiment for one pass of quantization have been depicted in Figure 1b.
As shown in the ﬁgure, the size of the model is reduced quickly during the initial iterations.
This portion of the experiment corresponds to the part of
Figure 1a where the error rate is constant.
However, as quantization continues, the model experiences diminishing returns in size reduction.
After 25 iterations, little reduction in model size is achieved.
The lower density of data points past this point is due to the generated steps failing the test on the [Figure] upper bound on cost in Algorithm 2.
Consequently, the algorithm reduces the trust region size and recalculates the tolerance steps.
C OMPARISON WITH R ELATED A PPROACHES
The generality of the proposed technique makes it adaptable to other model compression techniques.
In this section, we review some of the most notable of these techniques and examine the relationship between their approach and ours.
Speciﬁcally, we will explain how the proposed approach subsumes previous ones and how it can be specialized to implement them in an improved way.
Pruning:
In pruning, small parameters in a trained network are set to zero, which means that their corresponding connections are eliminated from the network.
Han et al. showed that by using pruning, the number of connections in the network can be reduced substantially (Han et al., 2015).
Adaptive quantization conﬁrms similar observations that in a NN, most connections can be eliminated.
In fact, adaptive quantization eliminates 5 .
6 × , 5 . 6 × , and 5 .
3 × of the connections in the networks trained for
MNIST, CFAR, and SVHN, respectively.
These number are lower compared to their corresponding values achieved using deep compression.
The reason for this difference is that deep compression amortizes for the loss of accuracy due to pruning of a large number of parameters by using full precision in the remaining ones.
In contrast, adaptive quantization eliminates fewer connections and instead quantizes the remaining parameters.
While adaptive quantization identiﬁes connections that can be eliminated automatically, deep compression identiﬁes them by setting parameters below a certain threshold to zero.
However, this technique might not be suitable in some cases.
For example, we consider the network model trained for CIFAR-10.
This network has been trained using L 2 regularization, which helps avoid over ﬁtting.
As a result, the trained model has a large population of small-valued parameters.
Looking at Figure 2a which shows the distribution of parameter values in the CIFAR-10 model, we see that most populate a small range around zero.
Such cases can make choosing a good threshold for pruning difﬁcult.
Weight Sharing:
The goal of weight sharing is to create a small dictionary of parameters.
The parameters are grouped into bins whose members will all have the same value.
Therefore, instead of storing parameters themselves, the weights can be replaced by the index in the dictionary corresponding to that parameter’s bin.
Deep compression implements weight sharing by applying k-means clustering to trained network parameters (Han et al., 2015).
Similarly, Smaragh et al.
implements weight sharing by iteratively applying k-means clustering and retraining in order to ﬁnd the best dictionary (Sarmagh et al., 2017).
In both of these works, the number of dictionary entries are ﬁxed in advance.
Adaptive quantization, on the other hand, produces the bins as a byproduct and does not make assumptions on the total number of bins.
Deep compression also identiﬁes that the accuracy of the network is less sensitive to the fully connected layers compared to the convolution layers.
Because of that it allocates a smaller dictionary for storing them.
Looking at ﬁgures 2b and 2c, we can see that results of adaptive quantization are consistent with this observation.
These ﬁgures show the distribution of quantization bits for one [Figure] [Figure] convolution layer and one fully connected layer in the quantized CIFAR
10 network.
It is clear from these ﬁgures that parameters in the fully connected layer, on average require fewer quantization bits.
Binarization and Quantization: Binarized Neural Networks (Courbariaux et al., 2015; Hubara et al., 2016a) and Quantized Neural Networks (Hubara et al., 2016b) can reduce model size by assuming the same quantization precision for all parameters in the network or all parameters in a layer.
These works often design quantized training algorithms to maintain the same parameter precisions throughout training.
These techniques can be slower than full-precision training.
In contrast, adaptive quantization allows for unique quantization precisions for each parameter .
This way, for a trained model, it can ﬁnd a notably smaller network, indicating the limits of quantization.
In addition, it does not require quantized training.
In order to compare adaptive quantization with previous works, we quantize trained models with different training cost bounds.
This way, for each dataset, we produce several models with different sizes and error rates.
We illustrate the results of these experiments in Figure 3, showing the trade-off between model size and error rate.
In both MNIST and CIFAR-10, adaptive quantization produces curves below the ones achieved in
BNN (Hubara et al., 2016a) and BinaryConnect (Courbariaux et al., 2015), and it shows a smaller model size while maintaining the same error rate or the same model size with a smaller error rate.
In the case of SVHN, the BNN achieves a better result compared to adaptive quantization.
This is due in part to the error rate of the full-precision model being higher than the BNN model.
Comparison of adaptive quantization against deep compression (when pruning is applied) for the
MNIST set shows some similarity between the two techniques, although deep compression achieves a slightly smaller error rate for the same model size.
This advantage comes from two differences between the two techniques.
First, deep compression uses full precision, ﬂoating point parameters while adaptive quantization uses quantized parameters, which requires less computational complexity (ﬁxed-point operation vs. ﬂoating-point operation).
Second, after pruning the network, deep compres- sion performs a complete retraining of the network.
In contrast, adaptive quantization performs little retraining.
Furthermore, adaptive quantization can be used to identify groups of parameters that need to be represented with more precision.
In a more general sense, the ﬂexibility of adaptive quantization allows it to be specialized for more constrained forms of quantization.
For example, if the quantization requires that all parameters in the same layer have the same quantization precision, adaptive quantization could ﬁnd the best model.
This could be implemented by solving the same minimization problem as in Equation 17, except the length of T would be equal to the number of layers.
In this case, g i in
Equation 2.3 would be (cid:80) j ∈ J
i | ∂C∂ω j ( ω kj ) | , where J i is the set of parameter indices belonging to layer i .
The rest of the solution would be the same.
C ONCLUSION
In this work, we quantize neural network models such that only parameters critical to the accuracy are represented with high accuracy.
The goal is to minimize data movement and simplify computations needed for inference in order to accelerate implementations on resource constrained hardware.
To achieve acceleration, the proposed technique prunes unnecessary parameters or reduces their precision.
Combined with existing ﬁxed-point computation techniques such as SWAR or Bitpragmatic computation , we expect these small ﬁxed-point models to achieve fast inference with high accuracy.
We have conﬁrmed the effectiveness of this technique through experiments on several benchmarks.
Through this technique, our experiments show, DNN model sizes can be reduced signiﬁcantly without loss of accuracy.
The resulting models are signiﬁcantly smaller than state-ofthe-art quantization technique.
Furthermore, it can provide similar results to ﬂoating-point model compression techniques.