Entropy-Driven Mixed-Precision Quantization for Deep Network Design on IoT Devices
Anonymous Author(s)
AffiliationAddress email
Abstract
Deploying deep convolutional neural networks on Internet-of-Things (IoT) devicesis challenging due to the limited computational resources, such as limited SRAMmemory and Flash storage.
Previous works re-design a small network for IoTdevices, and then compress the network size by mixed-precision quantization.
This two-stage procedure cannot optimize the architecture and the correspondingquantization jointly, leading to sub-optimal tiny deep models.
In this work, wepropose a one-stage solution that optimizes both jointly and automatically.
The keyidea of our approach is to cast the joint architecture design and quantization as an
Entropy Maximization process.
Particularly, our algorithm automatically designsa tiny deep model such that: 1) Its representation capacity measured by entropyis maximized under the given computational budget; 2) Each layer is assignedwith a proper quantization precision; 3)
The overall design loop can be done on
CPU, and no GPU is required.
More impressively, our method can directly searchhigh-expressiveness architecture for IoT devices within less than half a CPU hour.
Extensive experiments on three widely adopted benchmarks
ImageNet
VWWand WIDER
FACE, demonstrate that our method can achieve the state-of-the-artperformance in the tiny deep model regime.
Introduction
With the attention of many intelligent IoT applications that have grown recently, the demand for lowcost and low-energy IoT devices is significantly increasing, such as tiny NPUs and microcontrollerunits (MCUs).
These devices have tight-resources like hundreds KB on-chip memory (SRAM),
MB-level storage (Flash) and low speed computing ability.
How to design and deploy efficient tinydeep models for IoT devices is the main topic in Tiny Machine Learning (TinyML) [1].
As most IoT devices have very limited on-chip memory, one of the most critical challenges in
TinyML is to control the peak memory during the inference.
Some work re-designs the networkarchitecture to maintain the peak memory occupation [27, 39, 10, 17, 33, 16].
In addition, the inputimages can be sliced into multiple patches, so that the inference can be split into multiple smallerresolutions.
Neural Architecture Search (NAS) is a popular tool to re-design tiny deep modelsautomatically [10, 17, 33, 16].
After neural architecture design, the memory occupation can befurther reduced via mixed-precision quantization [34, 36, 3, 11, 35, 10, 26, 38, 4].
This two-stagedesign procedure cannot jointly optimize the network architecture and quantization, therefore leadsto sub-optimal models for tight resource requirements.
Resource Maximization
[Figure]
QBR
[Figure]
To address the above issues, the aim of this work is to optimize the network architecture jointly withmixed-precision quantization in one-stage.
Recently, some attempts [22, 6, 32, 18, 31] proposedzero-cost strategies for architecture searching, which construct an alternative proxy to rank theinitialized networks without training, but lack consideration of mixed-precision quantization.
Tothis end, we propose a one-stage lightweight neural architecture search approach to optimize botharchitecture and quantization in this work.
Inspired from [31], a deep neural network can be regarded as an information system, and theexpressiveness of a Gaussian initialized system can be proportional to the entropy of the last featuremap.
In this work, we further reformulate the Gaussian initialized network after quantization, and itsentropy is measured as the proxy of the performance for the mixed-precision quantization network.
We first comprehend the quantization effect on the entropy of the information system, and thenpropose a Quantization Entropy Score (QE-Score) to calculate the entropy.
Then, according to thestudy of how Gaussian initialization affects the value of QE-Score, we design a grid calibrationprocess to determine the initial values.
Finally, a Quantization Bits Refinement (QBR) strategyembedded in the evolutionary algorithm is introduced to maintain higher-precision weights andlower-precision activations in the front few layer, while the opposite in the latter few layers.
Theentire process is illustrated in Figure 1.
The key contributions of our work are summarized as follows:
• To the best of our knowledge, we first present the ranking strategy of mixed-precision quantizationnetworks in the entropy view.
And Quantization Entropy Score is proposed with a calibratedinitialization to measure the expressiveness of the network.
• Quantization Bits Refinement is proposed to adjust mixed quantization bits, which can ensurethat each layer is assigned with proper quantization precision, and maximize the utilization ofmemory and storage resources on the IoT devices.
• Benefit from the QE-Score, our approach can achieve architecture searching within less than halfa 64-core CPU hour.
Extensive experiments demonstrate that our searched model can achieve
state-of-the-art performance on IoT devices for both classification and detection tasks.
Problem Statement
2.1 Constraints of IoT Devices
Although significant research has been done to investigate architecture searching for IoT devices,designing models under limited resources remains a challenging issue.
Limited on-chip SRAMmemory and Flash storage are the major constraints [12, 17, 16] for deploying deep learning modelson IoT devices, especially for MCUs.
For example, a state-of-the-art ARM Cortex-M7 MCU merelyhas 512kB SRAM and 2MB Flash, which is impossible to run the off-the-shelf ResNet-50 [13] or
MobileNetV2 [28].
As a typical tiny model, full-precision MobileNetV2 needs 5.6M peak memoryand 13.5M storage, while the int8-precision version needs 1.4M peak memory and 3.4M storage [17],which far exceeds the limitation of mainstream IoT devices.
The distribution of memory and model size of MobileNetV2 is illustrated as shown in Figure 2.
On the other hand, the limited resourcebudget might be satisfied by lower precision quantization, but the performance of the models willdrop more.
[Figure]
y
[Figure]
r
[Figure]
t
[Figure]
r a g
[Figure]
2.2 Low-precision Quantization
To analyze the performance drop, we conduct an overall low-precision quantization [9] on MobileNetV2 except for the first and last layers, which follow the common practice of using 8-bit forhigh accuracy [25, 42, 2, 9].
As shown in Table 1 and Figure 3, we observe that lower bit precisionlower accuracy .
Only "A3W2" model fits both 512KB memory limit and 2MB storage limit with23.17% ACC decline, which uses 3-bit for all activations and 2-bit for all weights.
However, in“A3W2" Model, the activations in latter stages and the weights in front stages can utilize larger bitsquantization, which change less memory and storage according to the imbalanced resource utilizationin Figure 3.
Thus, the resource utilization of fixed-precision quantization is not high in some stages,which means we need mixed-precision quantization to improve accuracy.
Because mixed-precisionquantization uses lower bit on tight-resource position and higher bit on rich-resource position toensure precision stability.
Therefore, we should apply neural architecture search on mixed-precisionquantization for selected IoT devices.
[Table]
[Figure]
Quantization Entropy
In this section, we propose a Quantization Entropy Score for measuring the expressiveness of themixed-precision quantization model.
3.1 Maximum Entropy for Full-precision Models
According to the Maximum Entropy Principle [15] and its successful deep learning applications [29,5, 40, 31], a deep neural network can be regarded as an information system, and the differential entropy of the last output feature map represents the expressiveness of the system.
Given by thefollowing theorem:
Theorem 1.
For any continuous distribution P ( x ) of mean µ and variance σ 2 , its differential entropyis maximized when P ( x ) is a Gaussian distribution N ( µ, σ 2 ) .
The differential entropy of a distribution is upper bounded by a Gaussian distribution with the samemean and variance.
Suppose x is sampled from Gaussian distribution N ( µ, σ 2 ) , the differentialentropy [24] H of x is then:
[Equation]
where p ( x ) represents the probability density function of x .
Moreover, the entropy H representsthe expressiveness of a deep system, which correlated with the performance of a deep neuralnetwork [31].
Specifically, given a convolutional network with L layers of weights W 1 , ... , W L ,which are initialized from a standard Gaussian distribution, the forward inference is given by:
[Equation]
where x l denotes the l th layer feature map and ϕ ( · ) represents the activation function.
For simplicity,the bias of the convolutional layer is set to zero and the activation function is omitted.
As weinitialize the input x 0 from Gaussian distribution, the upper bound entropy H ( F ) of the network Fis proportional to log( σ 2 ( x L )) , where σ 2 ( x L ) represents the variance of the x L and is computed bythe forward inference.
To this end, we can measure the expressiveness of full-precision networkswithout training and inference.
The detailed description can be found in [31].
Further, according to the product law of expectation [23] and Bienaymé’s identity in probabilitytheory [20], we can obtain the expectation and variance of element i of x 1 layer feature map in oursystem by:
[Equation]
where { K 1 h , K 1 w } represents the kernel size of the first layer in the CNN, and C 0 denotes its inputchannels size.
Further extend to the l -layer convolution, we can obtain corresponding layer’sexpectation and variance:
[Equation]
The detailed derivation process is present in Appendix .
Next, we will explore how to introducemixed-precision quantization into the calculation process.
3.2 Quantization Entropy for Mixed-Precision Models
Mixed-precision quantization to the network F requires inserting low-precision conversion behind
Gaussian initialized input and weights.
Learning from the practices of entropy coding [30, 24] withfixed quantization step size of 1, given the variable R to quantize symmetrically, the conversion of
N -bit quantization representation [26, 9] can be defined as:
[Equation]
where clamp returns R with values below − 2 N − 1 set to − 2 N − 1 and values above 2 N − 1 − 1 setto 2 N − 1 − 1 .
According to the work of [31], the input of each layer is zero-mean distributionwhen deriving the entropy, so that the upper bound of Q is set as 2 N − 1 .
Then, the variance of thequantization value Q can be given by:
[Equation]
where Q ∈ [ − 2 N − 1 , 2 N − 1 ] and ˆ σ 2 denotes the truncated variance of Q , P ( Q ) is the probability of Qrelied on the distribution of R .
Therefore, P ( Q ) equals to: (1) (cid:82) − 2 N − 1 +0 .
−∞
P ( R ) , when Q = − 2 N − 1 ;(2) (cid:82) Q +0 .
5 Q − 0 .
5 P ( R ) , when − 2 N − 1 < Q < 2 N − 1 ; (3) (cid:82) ∞
N − 1 − 0 .
5 P ( R ) , when Q = 2 N − 1 .
As shown in Figure 4, we can figure out that the quantization of Gaussian variable will decrease thevariance of the variable.
Given a Gaussian distribution with σ and N bit precision, the decrease isfixed, so that we can pre-compute the decrease, which will speed up the computation of Eq. 6.
Thepre-computed quantization standard deviation ˆ σ ( N ) is demonstrated in Table 2.
[Figure]
[Table] [Table]
Then, apply the quantization to Eq.
4, the σ 2 ( x j ) of j th layer become larger during the depth increase.
As we set the quantization step as 1 in Eq. 5, the distribution in Figure 4 will be much smoother,and the probability will close to 0.
So that the different bits of quantization can not be distinguishedin the next layer.
If we use a dynamic quantization step in each layer to avoid this phenomenon,the computation will be more complex.
Therefore, we adopt a scaling parameter σ 2 S ( x l − 1 chw ) tonormalize the x j to σ A instead of adjusting the quantization step.
Following the weights obey
N (0 , σ W ) [18, 31], then use N l − 1 x and N lW bit to quantize the l -layer’s input and weight respectively,
Eq.
4 can be revised as
[Equation]
Subsequently, the variance of x L can be obtained by propagating the variances from previous layers:
[Equation]
where x 0 is initialized by σ 2 A .
Refer to the Eq.
1, the upper bound entropy H ( F ) is proportional to:
[Equation] [Equation]
[Figure] [Figure]
BitOps (G)
T O PA CC ( % )
A3W[2/3/4/5/6/8] A4W[2/3/4/5/6/8]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
where ˆ σ 2 ( N lx ) and ˆ σ 2 ( N lW ) are calculated by Eq.
6with initial standard deviation σ A and σ W .
Wename the value of Eq.
9 as Quantization Entropy Score (QE-Score), and detailed derivationis introduced in Appendix.
As the QE-Score onlydepends on the structural parameters of the network,quantization function, and initial standard deviationσ A and σ W , we can compute QE-Score on CPUsto rank candidate networks during search, whichgreatly reduces the dependence on hardware.
In thenext section, we will show how to determine theinitial standard deviation σ A and σ W .
3.3 Gaussian Initialization Calibration
We apply the commonly used MobileNetV2 architecture with fixed-precision quantization for calibration, and these models have been summarized in Table 1.
According to the relationship betweenaccuracy and computation in Figure 5 and the pre-computed quantization standard deviation ˆ σ ( N ) in
Table 2, we can figure out that: 1) Increasing the initial standard deviation σ will help us distinguishthe disparity of entropy between different bits;
2) Quantization on activations and weights has different effects on accuracy.
Therefore, we set different values of σ A and σ W to show differences betweenactivations and weights.
In summary, we use a grid calibration process to seek for the appropriateinitial σ A and σ W , and demonstrate the calibration process in Figure
[Figure] [Figure]
According to Figure
[Figure]
we observe that when gradually increasing the values of σ A and σ W , QEScores are gradually positive correlated with accuracy until σ A = σ W = 4 .
Then, we adjustσ A = 5 and σ W = 4 to rank the diversity of activations and weights on accuracy.
Benefit from the
Gaussian initialization calibration, QE-Score can help us rank various architectures without trainingand evaluating during search, and design a high-expressiveness architecture for IoT devices.
[Figure]
3.4 Resource Maximization for IoT Devices
We apply Evolutionary Algorithm (EA) to obtain the optimal architecture by the proposed QE-Scorefor IoT devices, and EA is employed along with low-precision quantization.
Based on the observationin Figure 2, the front few layers dominate the peak memory of the entire model due to high resolution,and the distribution of model weights generally shows an upward trend with the increase of featuremap channels.
Considering the tight SRAM memory and Flash storage constraints of IoT devices,maintaining higher-precision weights and lower-precision activations in the front few layers is an eligible strategy, while it is the opposite in the latter few layers.
Therefore, we propose a quantizationbits refinement strategy (Figure 7) embedded in the EA to realize it, to maximize the resource for IoT
Devices.
The detailed algorithm is introduced in Appendix.
Experiments
4.1 Implementation Details
Searching Settings .
The evolutionary algorithm and super-block definition follow the work of[18, 31].
The evolutionary population N is set as 512 with total 500000 iterations.
To compare withcommon mix-quantization search [34, 36, 11, 35] and IoT device applications [17, 16, 27, 39], webuild our network with MobileNetV2-based blocks, using the inverted bottleneck block with expansion ratio in [1 . 5 , 6] .
Specifically, low-precision values are random selected from { 2 , 3 , 4 , 5 , 6 , 8 } inour search, and three layers in each block share the same precision value.
Datasets and Training Settings .
We use two standard benchmarks in this work: ImageNet [8] and
Visual Wake Words (VWW) [7].
For ImageNet-1K dataset, our models are trained for 240 epochswithout special indication.
All models are optimized by SGD with a batch size of 512 and Nesterovmomentum factor of 0.9.
Initial learning rate is set to 0.4 with cosine learning rate scheduling [21],and the weight decay is set to 4e-6.
Besides, to validate the expansibility of our method, we furtherevaluate it on the WIDER FACE [37] object detection dataset.
When training with low-precisionvalues, we follow the implementation in [9] to do the quantization.
For VWW and WIDER FACE,we follow the training settings in [16].
More implementation details are introduced in Appendix.
4.2 Mixed-Precision Comparison
To compare with other mixed-precision search methods [34, 36, 11, 35], we conduct our searchingon the MobileNetV2-level under the computation budget (BitsOps), and the BitsOps budget is 19.2Gto compare with MobileNetV2-8bit
[Figure]
Random Correlation Study.
We randomly selected 100 models without QE-Score under the samesearching space to verify the correlation between our QE-Score and accuracy on ImageNet, beforecomparing with SOTA methods.
Results are shown in Figure 8, and we can observe that: 1) Theproposed QE-Score is closer to the linear growth relationship with TOP-1 accuracy than BitOps; 2)
The searched mixed-7.0G model outperforms the best random model with TOP-1 accuracy of nearly2%.
According to these observations, our QE-Score is valid to rank various architectures instead oftraining and evaluating.
Comparison with SOTA Models.
Table 3 presents the results of fixed- and mixed- precisionsearching.
Obviously, our models outperform state-of-the-art mixed-precision methods.
In particular,our mixed-19.2G model has 2.9% accuracy boost than MobileNetV2-8bit baseline, and our mixed7.0G model has better accuracy than MobileNetV2-4bit baseline (from 68.9% to 70.8%).
Note that,as we apply formula calculation rather than inference, our QE-Score can only use CPU resourcesinstead of GPU.
This will significantly reduce the dependence on hardware.
More impressively, the marginal cost of CO2 emission of our method is two orders of magnitudes smaller than other work,benefiting from computational efficiency
[Table]
[Table]
4.3 Tiny Image Classification
Large-scale Classification on ImageNet.
We searched tiny models under three hardware specifications, and comparative results with recent state-of-the-art tinyML solutions are shown in Table 4.
We can figure out that our model outperforms others on all three SRAM/Flash budgets.
Under thetight constraints of 256kB SRAM and 1MB Flash, our model significantly improves the TOP-1accuracy by 6.2% and 4.5% over 8-bit and 4-bit quantized MCUNet, respectively.
Under 512kB
SRAM and 1MB Flash, our model achieves a new record of 72.8% TOP-1 accuracy.
We think theseremarkable achievements benefit from improved SRAM/Flash utilization, which means more accuratemixed-precision quantization can specialize higher-capacity structures on resource-constrained IoTdevices.
Note that MCUNetV2 employed patch-based inference scheduling to reduce memory usage,which is an orthogonal technique to mixed-precision quantization
[Table] [Table]
Low-energy Application on Visual Wake Words.
The Visual Wake Words dataset (VWW) canrepresent a realistic IoT use-case of identifying the presence of persons.
Experimental results on
VWW dataset are shown in Figure 9.
We can observe that our model is superior to MCUNetin both accuracy and memory utilization.
The accuracy is boosted to 93% with slightly fewermemory requirements.
To be comparable with MCUNetV2, we used the same patch-based inferencescheduling to further reduce the memory footprint.
Without accuracy loss, the runtime peak memorycan be decreased by about 3 .
5 × .
[Figure]
[Table]
Resource Maximization.
Table 5 compares our final model with fixed-bit counterparts and showsthe effectiveness of QBR.
Although 4-bit quantization is better than 8-bit quantization, there isstill a gap between fixed- and mixed- precision quantization.
Note that even without QBR, ourmixed-bit network is superior to fixed-bit MCUNet and MCUNetV2 in Table 4.
Fig.
10 visualizespeak memory, bits distribution, and model size of our searched model under 320kB SRAM and 1MB
Flash.
Obviously, our method maintains higher-precision weights and lower-precision activationsin the front few layers, while opposite in the latter few layers.
This demonstrates that QBR canstrengthen resource utilization by explicitly embedding prior design knowledge.
[Figure]
FACE.
The hard subset is the most authoritative benchmark since it contains the faces in easy and medium subsets [19].
[Table]
4.4 Tiny Object Detection on WIDER FACE
The proposed QE-Score method can be adopted to various visual tasks, and we demonstrate its generalization ability on object detection task.
Following the pipeline of [19], We conducted experimentson WIDER FACE [37] and results are shown in Table 6.
We follow the quantization strategy in [27]that memory usage is analyzed for detector backbone and reported in FP32 to build our detectionmodel “Ours-Face”.
According to Table 6, we observe that our model can achieve a competitivemAP performance at all three subsets with 1 . 8 × low peak memory and 2 × less computation than
EagleEye [41] and RNNPool [27], which verifies the generalization ability of our method.
Conclusion
In this work, we propose an entropy-driven mixed-precision quantization strategy to address challenges of deep network design on IoT devices.
In particular, we first present the ranking strategyof mixed-precision quantization networks in the entropy view by formulating the neural networkas an information system, and propose a QE-Score with calibrated initialization to measure theexpressiveness of the system.
Then we propose the Quantization Bits Refinement within evolution algorithm to adjust mixed-precision quantization.
Finally, the obtained network achieve state-of-the-artperformance on IoT devices for both classification and detection tasks within less than half a 64-core
CPU hour searching.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] Present in abstract and Section 1.(b) Did you describe the limitations of your work?
[Yes] Present in supplemental material.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Present insupplemental material.(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]
We have read it and ensure that our paper conforms to them.2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Present in Section 2and Section 3.(b)
Did you include complete proofs of all theoretical results?
[Yes] Present in supplementalmaterial.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[No] The code will be publishedbefore the paper is published.
The data is public.
The instructions is present in Section 3 andSection 4.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] Present in Section 3, Section 4 and supplemental material.(c)
Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[No] Error bars are not reported because it would be too computationally
expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
[Yes] Search uses 64 cores of Intel(R)
Xeon(R)Platinum 8269CY CPU @
2.50GHz and training is conducted on 8 NVIDIA V100 GPUs.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Present in Section 4.1 andSection 4.4.
(b) Did you mention the license of the assets?
[No] Assets in the paper are widely used, and thelicense is already mentioned in reference papers.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No](d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[No] The data we used are all public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
[N/A]