T OWARDS E FFECTIVE 2- BIT Q UANTIZATION : P ARETO - OPTIMAL B IT A LLOCATION FOR D EEP CNN S C OMPRESSION
Anonymous authors
Paper under double-blind review
A BSTRACT
State-of-the-art quantization methods can compress deep neural networks down to 4 bits without losing accuracy.
However, when it comes to 2 bits, the performance drop is still noticeable.
One problem in these methods is that they assign equal bit rate to quantize weights and activations in all layers, which is not reasonable in the case of high rate compression (such as 2-bit quantization), as some of layers in deep neural networks are sensitive to quantization and performing coarse quantization on these layers can hurt the accuracy.
In this paper, we address an important problem of how to optimize the bit allocation of weights and activations for deep CNNs compression.
We ﬁrst explore the additivity of output error caused by quantization and ﬁnd that additivity property holds for deep neural networks which are continuously differentiable in the layers.
Based on this observation, we formulate the optimal bit allocation problem of weights and activations in a joint framework and propose a very efﬁcient method to solve the optimization problem via Lagrangian Formulation.
Our method obtains excellent results on deep neural networks.
It can compress deep CNN ResNet-50 down to 2 bits with only 0.7% accuracy loss.
To the best our knowledge, this is the ﬁrst paper that reports 2-bit results on deep CNNs without hurting the accuracy.
I NTRODUCTION
Deep Convolutional Neural Networks (CNNs) Krizhevsky et al.
; Simonyan & Zisserman (2015); Szegedy et al. (2015); He et al.
(2016) have dominated the applications in computer vision and related ﬁelds.
However, one challenging issue is that deep CNNs usually involve large number of parameters and are computationally demanding, making it very difﬁcult to deploy deep CNNs on resource-limited devices.
Quantization for weights and activations of deep CNNs is an effective way to mitigate this issue.
Cooperated with advanced retraining (or ﬁne-tuning) strategies, state-of-theart methods Zhuang et al.
; Zhou et al.
; Zhang et al.
(2018) can compress deep CNNs down to 4 bits without losing accuracy.
State-of-the-art quantization methods Zhuang et al.
; Zhou et al.
; Zhang et al.
(2018) typically assign same bit rate to all layers for compression.
However, when deep neural networks are compressed to very low precision, e.g., 2 bits, such equal bit allocation scheme is not reasonable.
As some of layers in deep neural networks are sensitive to quantization, performing a coarse quantization on these layers may signiﬁcantly hurt the accuracy.
A better strategy is to adopt unequal bit allocation scheme to quantize weights and activations across layers.
In this paper, we address an important problem of how to optimize the bit allocation of weights and activations of deep CNNs.
Finding the optimal bit allocation for deep CNNs is difﬁcult, as traversing the whole searching space requires exponential computation complexity.
Assume M is the number of quantization levels and L is the number of layers.
A brute-force search has O ( M L ) computation complexity, which is unaffordable if L is large in the case of deep neural networks.
In our work, we ﬁrst analyze how quantization affects the output of deep CNNs, and ﬁnd that the mean squared output error caused by quantization has additivity property.
Based on this observation, we formulate the optimal bit allocation problem of weights and activations in a joint framework and adopt Lagrangian formulation Shoham & Gersho (1988) for optimization.
We propose a very efﬁcient method to ﬁnd the solution with polynomial computation complexity.
Prior adaptive quantization schemes proposed by Khoram & Li (2018) and Zhou et al.
(2018) only consider the compression of weights.
Since the volume of activations is considerable, the size of activations should be also reduced as much as possible, which might otherwise slow down processing.
Zhu et al.
(2018) proposed layerwise adaptive quantization for weights and activations, but their method decides each layer’s quantization level individually and does not optimize the bit allocation across layers when performing quantization on multiple layers.
Different with prior literature, our work formulates the optimal bit allocation problem for both weights and activations and proposes an efﬁcient method to solve the challenging problem.
The new contributions of this paper are summarized as following:
• Exploring Additivity of Mean Squared Output Error :
In this paper, we adopt mean squared output error to measure quantization impact.
We observe that mean squared output error has additivity property, i.e., the mean squared output error caused by quantizing multiple layers equals to the sum of mean squared output error due to the quantization of each individual layer.
We show empirical results on two deep CNNs AlexNet and VGGand also provide a mathematical derivation for the additivity property.
• Pareto Condition for Optimization : Based on the additivity of mean squared output error, we propose an optimal bit allocation framework for both weights and activations.
Moreover, we propose an efﬁcient method to solve the optimization problem by using Lagrangian Formulation.
We show that the optimization problem can be solved in polynomial computational complexity under Pareto condition,
i
e., the slopes of mean squared output error vs bit rate functions of each layer have to be equal.
• Impact on Improving Inference Rate : The pattern of Pareto-optimal bit allocation of weights on deep CNNs has very positive impacts on inference rate.
It tends to allocate fewer bits per weight for layers that have a lot of weights, which helps to reduce the corresponding memory-access time which
in turn reduces compute idle time and improves the overall inference rate.
Simulation experiments on Google TPU hardware accelerator shows that
Pareto-optimal bit allocation can further improve the inference rate on deep CNNs by 1.5X compared to its equal bit allocation counterpart.
Results on ImageNet Deng et al.
(2009) show that our method can compress deep CNNs (e.g.,
ResNet-50 He et al.
down to 2 bits and the accuracy loss is less than 0 .
To the best of our knowledge, this is the ﬁrst paper that reports 2-bit results on deep CNNs without hurting the accuracy.
The remainder of this paper is structured as follows.
Section 2 discusses related works.
Section 3 analyzes the impact of quantization.
Section 4 develops our bit allocation framework.
Section 5 reports experimental results.
Section 6 discusses the impact of inference rate.
R ELATED W ORKS
Some latest works Zhuang et al.
; Park et al.
; Choi et al.
(2018b;a) already reported 2-bit results on deep CNNs.
Among them, the highest result is reported by Park et al.
(2018), which loses 2.45% accuracy at 2 bits on ResNet-50 He et al.
Although promising results are obtained, there is still considerable accuracy drop at 2 bits.
Note that all these works assign same bit rate to quantize weights and activations in different layers.
Different with them, our method adopts a better strategy for bit allocation.
The optimal bit allocation strategy is critical for maintaining high accuracy when neural networks are compressed to very low precision like 2 bits.
The most relevant works related to our unequal bit allocation scheme are proposed by Khoram &
Li (2018) and Zhou et al.
Khoram & Li (2018) suggest monitoring the loss function during training to allocate the bit rates of weights in different layers.
Zhou et al.
(2018) suggest applying adversarial noise Fawzi et al.
as an indicator to measure the quantization impact for bit allocation.
Both of these two works only study the quantization problem of weights.
Zhou et al.
(2018) empirically ﬁnds that the adversarial noise Fawzi et al.
(2016) has additivity property.
While, adversarial noise is a measurement of top-1 accuracy in object classiﬁcation task, which is not applicable to other tasks like object detection.
Our work studies the additivity of mean
[Equation]
[Figure]
squared output error.
We do not make assumptions about the application scenarios of our method
We provide both empirical results and a mathematical derivation for the additivity property.
Recently, Banner et al.
(2019) proposed per-channel bit allocation for weights and activations.
They formulate the bit allocation problem as a minimization problem of Mean Squared Error (MSE) of quantized values in intermedia layers, and provide a solution with analytic expression in the case of
Laplacian distribution.
Finally, the solution of bit allocation with analytic expression is rounded to integer values.
Mean Squared Error (MSE) in intermedia layers can not reﬂect network networks’ ﬁnal output accuracy.
Our method adopts mean squared output error to measure the impact on last layer’s output accuracy.
It ﬁnds the optimal bit allocation directly from the discrete search space and does not make any assumptions for the distributions.
A NALYSIS OF Q UANTIZATION I MPACT
We adopt uniform scalar quantization in our compression framework.
Each layer’s weights and activations are quantized separately, followed by entropy coding of the quantization index.
The quantizer stepsize is individually chosen for each layer’s weights and activations.
We incorporate a dead zone into the quantizers for weights.
Let ∆ denote the stepsize of a quantizer.
The width of the dead zone is ∆ + β .
Let t = 2 k + 1 denote the total number of representative levels of the quantizer.
Given scalar input x , if x is in dead zone ( | w | < ∆+ β 2 ), it is quantized to zero.
Otherwise, x is quantized to
[Equation]
where integer i is the quantization index, and 1 ≤ | i | ≤ k .
Both β and t are hyper parameters.
We note that adding a dead zone can improve compression.
This will be discussed in the following sections.
We do not apply a dead zone to the quantizers of activations.
We assume activation function after
Convolution layer is ReLU, which is common practice in modern deep CNNs Krizhevsky et al.
(2012); Simonyan & Zisserman (2015);
He et al.
The quantizer for activations is deﬁne as q ( x ) =
i · ∆ , (2) where quantization index 0 ≤ i ≤ t − 1 .
3. M EAN S QUARED O UTPUT E RROR C AUSED BY Q UANTIZATION
Let W i denote weights in layer
i and A i denote activations in layer i for all 1 ≤ i ≤ L where
L is total number of layers.
Given a neural network F and input image I , we have output vector
V = F ( I ) .
If we quantize the weights W i , we obtain a modiﬁed output vector (cid:98) V .
The mean sqaured output error caused by quantizing W i is deﬁned as the expectation of the squared Euclidean distance between original output V and modiﬁed output (cid:98) V divided by the dimensionality of V
[Equation]
where E ( . ) denotes the expectation operator and d ( X, Y ) is the squared Euclidean distance between vectors X and Y .
The expectation is over the distribution of all random input images I .
Similarly,
D A i F denotes mean squared output error caused by quantization of activations
A i for all 1 ≤ i ≤ L .
In the following, ”mean-squared output error” and ”output error” will both be used interchangeably.
[Figure]
t r u e ou t pu t e rr o r $ % t r u e ou t pu t e rr o r $ %
AlexNet
VGG-
3. A DDITIVITY OF M EAN S QUARED O UTPUT E RROR
Consider mean squared output error when simultaneously quantizing all layers.
Let D F denote mean squared output error caused by quantizing all layer’s weights and activations, the following additivity property holds.
Proposition 1 The output error D F , caused by quantizing all layers’ weights and activations, equals the sum of all output error due to the quantization of individual layer’s weights and activations
[Equation]
) if the neural network is continuously differentiable in every layer and quantization errors in different layers are independently distributed with zero mean.
In practice, additivity holds approximately in deep neural networks.
We evaluate additivity on
AlexNet and VGG-16.
Figure 2 shows the results.
Vertical axis in Figure 2 represents the output error when quantizing all layers simultaneously.
Horizontal axis represents the sum of output error caused by quantizing each layer individually.
We can see that data points in Figure 2 are all very close to the diagonal, meaning that additivity property holds.
This property can be shown mathematically by linearizing the quantization error of weights and activations using Taylor series expansion as the neural networks is continuously differentiable and the quantization error can be considered as small deviations.
The supplementary material provides a mathematical derivation.
U NEQUAL B IT A LLOCATION F RAMEWORK
Let R W i and R A i denote the total bitrates of weights and activations in i-th layer, respectively.
R W i (or R A i ) is determined by the number of representative levels t , the dead zone parameter β , and the number of weights (or activations) in layer i .
According to 4, the output error of quantizing all layers’ weights and activations is the sum of the output error caused by quantizing each layer’s weights and activations individually.
The bit allocation framework is thus formulated as
[Equation]
where R Total is the total number of bits needed to represent the weights and activations of all layers.
Our goal is to ﬁnd the optimum bit allocation { R W i } Li =1 and { R A i } Li =1 to minimize network output error within a constrained total bit budget for weights and activations.
More precisely, we adjust representative level t and dead zone parameter β of the quantizer in each layer to minimize the overall output error of the quantized network given a total bit budget constraint.
[Figure]
B i t r a t e s
N u m b er o
f W e i g h t s
( 100 K ) B i t
r a t e s
N u m b er o f A c t i va t i o n s ( 100 K )
[Figure]
4. P ARETO - OPTIMAL B IT A LLOCATION
We use a classical Lagrangian rate-distortion formulation to ﬁnd the optimum bit allocation Shoham & Gersho (1988).
Our Lagrangian cost function is
[Equation]
By setting the partial derivatives of J with respect to R W i and R A j to zero, we obtain
[Equation]
for all 1 ≤ i, j ≤ L .
The above equation expresses the intuitively pleasing insight that the slopes of output error versus rate functions for weights and activations of each layer have to be equal.
This is exactly the well-known Pareto condition for optimal resource allocation in economics.
7 holds, as long as the output error contributions D W i F and D A i F are additive, rates R W i and R A
i are additive, the functions D W
i F
( R W i ) and D A
i F
( R A i ) are convex and the resulting R W
i , R A i > 0 .
It does not make assumptions about the speciﬁc quantization or coding scheme used.
4. O PTIMIZATION UNDER P ARETO C ONDITION
We ﬁnd optimum bit allocation under Pareto condition according to 7.
We ﬁrst generate ratedistortion curves of ( R W i , D W i F ) and ( R A i , D A i F ) by choosing different values for representative level t and dead zone parameter β .
Then we enumerate λ and select the point in each curve with slope equal to λ .
The set of selected points is one solution of Pareto-optimal bit allocation.
We may enumerate many values of λ to ﬁnd the best solution with minimal output error under the bit budget constraint.
Assume that we have N curves and M points in each curve.
The whole search space of the solution is O ( M N ) .
By utilizing the Pareto condition, the time complexity to ﬁnd optimum bit allocation is reduced to O ( K · M · N ) , where K is the total number of λ evaluated.
In practice, we can ﬁnd the optimum bit allocation for ResNet-50 He et al.
and MobileNet-v1 Howard et al.
(2017) in several seconds on a standard CPU (AMD Ryzen threadripper 1950x) using the algorithm described above.
4. O BSERVATIONS ON O PTIMUM B IT A LLOCATION
Figure 3 shows the optimum bit allocation of weights and activations according to 7 on ResNetHe et al.
(2016), when weights and activations are compressed to 2 bits on average.
Two observations stand out.
First, weights receive much larger bitrates (average bits per layer) than activations in general.
Second, the layers with larger number of weights receive relatively lower bitrates; conversely, the layers with small number of weights receive relatively high bitrates.
Similar trends are observed on MobileNet-v1 Howard et al.
(2017), which are illustrated in the supplementary material.
We noticed that such bit allocation pattern of weights can have a positive impact on inference rate.
As we allocate relatively small bitrates to large-weight layers, the corresponding memory-access time will be reduced which in turn reduces compute idle time and improves the overall inference rate (see Section 6).
[Figure] [Figure]
thhe-art methods from the literature.
E XPERIMENTS
We evaluate our method on ResNet-50 He et al.
and MobileNet-v1 Howard et al.
(2017) over the ImageNet Deng et al.
(2009) validation dataset with 224x224 input image resolution.
The total number of weights and activations of ResNet-50 is 32.3M with 22.7M weights and 9.6M activations, and the total number of weights and activations of MobileNet-v1 is 9.3M with 4.2M weights and 5.0M activations.
We adopt the Tensorﬂow implementation of ResNet-50 and MobileNet-v1 provided by the Tensorpack and SLIM library, respectively.
The original ResNet-50 and MobileNet-v have 76.4% and 70.9% top-1 image classiﬁcation accuracy on the ImageNet validation dataset, respectively.
We compare our method with 15 state-of-the-art methods, including Binarized Weight Network (BWN) Hubara et al.
(2016), Ternary Weight Network (TWN)
Li et al.
Incremental Network
Quantization (INQ) Zhou et al.
, Fine-grained Quantization (FGQ) Mellempudi et al.
Integer Arithmetic-only Inference (IAOI)
Jacob et al.
Adaptive Quantization (AQ) Zhou et al.
(2018), Compression Learning by In-parallel Quantization (CLIP-Q) Tung & Mori (2018),
Symmetric Quantization (SYQ)
Faraone et al.
(2018), Low-bitwidth CNN (LBCNN) Zhuang et al.
(2018), Extremely Low Bit Neural Network (ELBNN)
Leng et al.
(2017), Learned Quantization (LQ-Nets)
Zhang et al.
Deep Compression (DC) Han et al.
, Hardware-Aware Automated Quantization (HAQ) Wang et al.
(2019), Parameterized Clipping Activation (PACT) Choi et al.
(2018b) and DoReFa-Net Zhou et al.
It is worth noting that the top performing comparison methods all include retraining to recover accuracy loss due to quantization error introduced.
[Table]
[Table]
E FFECTIVENESS OF THE P ROPOSED C OMPRESSION F RAMEWORK
In this section
[Table]
we evaluate the effectiveness of each component in our compression framework.
Optimal Bit Allocation.
As illustrated in Figure 4, we show the performance of equal bit allocation as a baseline to evaluate the effectiveness of the proposed optimal bit allocation scheme.
Equal bit allocation assigns same quantization level for all layers’ weights and activations.
In Figure 4, there is a large gap between the results of optimal bit allocation (green curves) and the results of equal bit allocation (blue curves).
Optimal bit allocation is much more effective.
Entropy Coding and Dead Zone.
Applying entropy coding to quantization index can further reduce the size by 20% to 30%.
We use the entropy of the quantization index to estimate the code size after entropy coding.
In Figure 4, the dashed green and blue curves show the results of quantized networks with equal and optimal bit allocation before entropy coding, respectively.
After entropy coding, the size of quantized networks can be further reduced (see the solid green and blue curves in
Figure 4).
Besides, with a dead zone, we also observe large performance improvement on ResNet50 and noticeable improvement on MobileNet-v1.
Adding a dead zone to the quantizers forces more weights to zero.
This beneﬁts entropy coding, as it makes the distribution of quantized weights more peaky.
A dead zone is similar to pruning Han et al.
(2015; 2016); both set small weights to zero.
The key difference is pruning completely removes the small weights from networks while dead zone still keeps them.
Retraining.
Like other schemes, our compression framework can be enhanced by retraining, especially at very low bitrates.
We adopt the straight-through estimator (ETS) Bengio et al.
(2013) to perform back-propagation through non-differentiable quantization functions.
We retrain 10 epochs for both ResNet-50 He et al.
and MobileNet-v1 Howard et al.
For ResNet-50, we set the learning rate as 0.1 if bitrate is more than 3 bits, otherwise, learning rate is 0.001.
For MobileNetv1 Howard et al.
(2017), learning rate is 0.45.
Without retraining, our method obtains 66.7% on
ResNet-50 at 2 bits on average.
The simple retraining stage improves accuracy to 75.7%—close to that of the original pre-trained model (76.4%).
5. C OMPARISON WITH S TATE - OF - THE - ART M ETHODS
The proposed compression framework achieves excellent performance on ResNet-50 and
MobileNet-v1, compared to recent state-of-the-art methods.
Figure 5 shows the tradeoff between size of weights/activations and classiﬁcation accuracy over the ImageNet validation dataset.
On
MobileNet-v1, our method without retraining outperforms state-of-the-art approaches equipped with retraining.
Similar observation can be made for ResNet-50.
Table 1 further compares our method with state-of-the-art that report results at very low bitrates (average 2 and 4 bits).
At 4-bit on average, without retraining, our method only loses 0.2% Topaccuracy on ResNet-50.
This is strong evidence of the effectiveness of the proposed compression framework.
With simple retraining, our method shows additional accuracy gains.
On ResNet-50, it outperforms the best performing baseline by 4.2% Top-1 accuracy at 2 bits.
On MobileNet-v1,
[Table] [Table]
our method outperform the best performing baseline by 2.6% Top-1 accuracy at 4 bits.
Further improvements are possible if combining our unequal bit allocation scheme with advanced quantized model retraining approaches like progressive training and knowledge distillation training Zhuang et al.
We also report our results at 1.5 bits.
We noticed that the quantized networks with equal bit allocation at 2 bits without entropy coding can be further compressed to around 1.5 bits after entropy coding.
Our method obtains 72.5 Top-1 accuracy at 1.5 bits on ResNet-50, which still outperforms other state-of-the-art methods at 2 bits by 1.0%.
I MPACT ON I NFERENCE R ATE
We deﬁne inference rate as the maximum number of images that a particular implementation of a neural network can process per unit time.
Inference rate is mainly driven by memory-access bandwidth and compute throughput (number of compute operations per unit time).
In our work, entropy-encoded weights and activations are fetched from off-chip to on-chip memory, where they are decoded and fed into the processing units.
We have explored the impact our approach has on the inference rate for different hardware architectures inspired by current embedded and high-performance DNN-targeted hardware accelerators (i.e., MIT Eyeriss Chen et al.
and Google TPU Jouppi et al.
(2017), summarized in Table 2)—such platforms have shown superior inference rate compared to GPUs Jouppi et al.
We assume that accessing off-chip memory can be done in parallel while computing is performed (i.e., weights can be prefetched from off-chip memory).
Using SCALE-sim Samajdar et al.
(2018) software, we simulate the inference rate of ResNet-50 and
MobileNet-v1, when mapped into the two considered hardware platforms in Table 2.
We report the inference speed-up resulting from our unequal bit allocation scheme versus the equal bit allocation scheme (all weights and activations are represented with a ﬁxed number of bits), when running an inference on a single 224 × 224 input image.
We also report the speedup compared to a full precision allowed on each platform (i.e., 8-bit on TPU, 16-bit on Eyeriss).
Our unequal bit allocation achieves 1 .
5 × and 1 .
07 × higher inference rate for ResNet-50 ( 1 . 3 × and 1 × for MobileNet-v1) at 2 bits and 4 bits, as summarized in Table 3, when compared with their equal bit allocation counterparts, respectively.
Higher speed-ups, up to 4 × , are achieved when compared with 8-bit full precision.
One may note that the speedup is much larger if we consider inference time contributed by memory access only.
These improvements are mainly attributed to the unequal bit allocation pattern—it minimizes the layer-wise bitrates for weights/activations and also balances the size of data moving between offchip and on-chip memory, which leads to reduced access times to off-chip memory.
It is worth mentioning that with higher memory-access bandwidth, lower speed-ups are observed.
For instance, at 128 GBytes/s, no speedup is observed.
However, such hardware modules consume high energy and are rather costly, where are our approach provides an attractive alternative to increased memory bandwidth.
Furthermore, platforms with low computation rate (i.e., few GOPs, as in the case of
Eyeriss in Table 2), are constrained by the computational throughput, hence may not beneﬁt from any speed-up.
C ONCLUSIONS
In this paper, we have proposed a novel unequal bit allocation framework for compression of both weights and activations of deep CNNs.
The output error due to the quantization of individual layer’s weights and activations is additive, as are the bits used to represent each layer.
Building on this observation, we formulate a Lagrangian optimization framework to ﬁnd the optimum bit allocation.
The optimal bit allocation problem can be efﬁciently solved by utilizing the Pareto condition, reducing the search space from exponential complexity to linear complexity.
Our method achieves excellent results on ResNet-50 and MobileNet-v1.
In particular, our method can compress ResNet50 into 4 bits on average without retraining, which is computationally very expensive and might not even be possible, if the original training data are not available.
The proposed unequal bit allocation scheme has a very positive impact on inference rate.
It is able to improve inference rate by up to 4 × compared with full precision, as well as 1 . 5 × compared with its equal bit allocation counterpart, on
Google TPU hardware accelerator.
A BSTRACT
In this supplementary material, we provide the mathematical analysis for the additivity of output error, and the additional results of Pareto-optimal bit allocation.
Section 1 provides the proof of Proposition 1.
Section 2 provides the results of
Pareto-optimal bit allocation on ResNet-50 He et al.
and MobileNet-v
Howard et al.
(2017) at different bitrates.
A DDITIVITY OF O UTPUT E RROR
[Equation] [Equation]
[Equation] [Equation]
[Figure]
Proposition 1 The output error D F , caused by quantizing all layers’ weights and activations, equals the sum of all output error due to the quantization of individual layer’s weights and activations
[Equation]
(1) if the neural network is continuously differentiable in every layer, the quantization errors in different layers are independently distributed with zero mean.
Proof 1 Let F ( W 1 , ..., W L ) denote a neural networks and (cid:101) F ( W 1 , ..., W L , n 1 , ..., n L ) denote a modiﬁed neural networks of F where an element-wise add layer with parameter n i is followed for each activation a i .
Based on this deﬁnition, we have
[Equation]
[Equation] [Equation]
Let X
= (cid:0) W 1 , ..., W L , 0 , ..., 0 (cid:1) and ∆ X = (cid:0) ∆ W 1 , ..., ∆ W L , ∆ n 1 , ..., ∆ n 1 (cid:1) .
Assume that the quantization error can be considered as small deviation.
We apply the Taylor series expansion up to ﬁrst order term on (cid:101) F at X 0 ,
[Equation]
Then, (cid:107)
[Equation]
F
[Equation]
X 0 + ∆ X ) −
[Equation]
F
[Equation]
X 0 )
[Equation]
2 can be written as
[Equation]
Because we assume that quantization errors in different layers are independently distributed with zero mean
[Equation]
the cross terms of (4) disappear when taking the expectation.
That is:
[Equation]
as is the case also for the cross products between W i and n j (all i , j )
[Equation]
and n i and n j ( i (cid:54) = j ).
Then, we can obtain
[Equation]
(5) is the result we want because, again, according to the Taylor series expansion (ignoring the higher order terms),
[Equation]
Therefore, after dividing both sides of (5) by the dimensionality of the output vector of the neural network, the left side of (5) becomes D F and the right side of (5) becomes the sum of all output error due to the quantization of individual layer’s weights and activations.
P ARETO - OPTIMAL B IT A LLOCATION ON R ES N ET -50 AND M OBILE N ET - V
Figure 2 shows the Pareto-optimal bit allocation of weights and activations on ResNet-50 He et al.
(2016) and MobileNet-v1 Howard et al.
(2017) when weights and activations are compressed tobits and 4 bits on average.
Results are generated on ImageNet dataset Deng et al.
Two observations stand out on both ResNet-50 He et al.
and MobileNet-v1 Howard et al.
First, weights receive much larger bitrates (average bits per layer) than activations in general.
Second, the layers with larger number of weights receive relatively lower bitrates; conversely, the layers with small number of weights receive relatively high bitrates.
According to the classical Lagrangian rate-distortion formulation, the optimal bit allocation has positive correlation with variance.
Based on this conclusion, we expect that the layers with larger number of weights should have smaller variances.
Figure 3 shows the relationship between variance and bitrate on two deep network ResNet-50 and MobileNet-v1.
We can see that the layers with larger number of weights typically have smaller variances, and thus these layers receive smaller bitrates (and vice-versa).
The distributions of variances and bitrates across layers have same trend on both
ResNet-50 and MobileNet-v1.