O NCE Q UANTIZED FOR A LL : P ROGRESSIVELY S EARCHING FOR Q UANTIZED E FFICIENT M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Automatic search of Quantized Neural Networks has attracted a lot of attention.
However, the existing quantization aware Neural Architecture Search (NAS) approaches inherit a two-stage search-retrain schema, which is not only timeconsuming but also adversely affected by the unreliable ranking of architectures during the search.
To avoid the undesirable effect of the search-retrain schema, we present Once Quantized for All (OQA), a novel framework that searches for quantized efﬁcient models and deploys their quantized weights at the same time without additional post-process.
While supporting a huge architecture search space, our OQA can produce a series of ultra-low bit-width( e.g. 4/3/2 bit) quantized efﬁcient models.
A progressive bit inheritance procedure is introduced to support ultra-low bit-width.
Our discovered model family, OQANets, achieves a new state-of-the-art (SOTA) on quantized efﬁcient models compared with various quantization methods and bit-widths.
In particular, OQA2bit-L achieves 64.0% ImageNet Top-1 accuracy, outperforming its 2-bit counterpart EfﬁcientNetB0@QKD by a large margin of 14% using 30% less computation budget.
I NTRODUCTION
Efﬁcient architecture design (Sandler et al., 2018; Ma et al., 2018) and network quantization methods (Choi et al., 2018; Kim et al., 2019; Esser et al., 2019) are two promising research directions to deploy deep neural networks on mobile devices.
Network quantization aims at reducing the number of bits for representing network parameters and features.
On the other hand, Neural Architecture
Search(NAS)
(Howard et al., 2019; Cai et al., 2019; Yu et al., 2020) is proposed to automatically search for efﬁcient architectures, which avoids expert efforts and design trials.
In this work, we consider both of their advantages and explore NAS’s ability in ﬁnding quantized efﬁcient models.
Recent quantization methods quantize ﬂoating-point models with high performance no matter it is manually designed like MobileNetV2 (Sandler et al., 2018) or searched like EfﬁcientNet-B0 (Tan & Le, 2019).
However, the accuracy rank among ﬂoating-point models would change after they are quantized.
The NAS-then-Quantize routine in Figure 1(a) that searches the best ﬂoating-point models and then retrain the quantized models, may fail to get a good quantized model.
Directly using quantized models’ performance to search seems to be a solution.
Existing quantization-aware NAS methods (Wang et al., 2019; Shen et al., 2019; Bulat et al., 2020;
Guo et al., 2019; Wang et al., 2020) utilize a two-stage search-retrain schema as shown in Figure 1(b).
Speciﬁcally, they ﬁrst search for one architecture under one bit-width setting 1 , and then retrain the model under the given bit-width.
This two-stage procedure will undesirably increase the search and retrain cost if we have multiple deployment constraints and hardware bit-widths.
Furthermore, due to the instability brought by quantization training, simply combining NAS and quantization will result in unreliable ranking (Li et al., 2019a;
Guo et al., 2019) and sub-optimal quantized models (Bulat et al., 2020).
Moreover, when the quantization bit is lower than 3 bit, the traditional training process is highly unstable and introduces very large accuracy degradation.
To alleviate the aforementioned problems, we present Once Quantized for All (OQA), a novel framework that: 1) searches for quantized network architectures and deploys their quantized weights
[Figure]
concurrently without retraining, 2) progressively produces a series of ultra-low bits( e.g. 4/3/2 bit) quantized models.
Our approach leverages the recent NAS approaches which require no retraining (Yu & Huang, 2019; Cai et al., 2019; Yu et al., 2020).
We adopt the search for kernel size, depth, width, and resolution in our search space and enable searching them under different quantization bit-widths.
Furthermore, we propose the bit inheritance mechanism to reduce the bit-width progressively so that we can utilize the higher bit-width models to provide better initialization and distillation knowledge (Hinton et al., 2015) for lower bit-width models.
Beneﬁting from the no retraining property and large search space under different bit-widths, we are able to evaluate the effect of network factors which could provide some insight into quantization-friendly architecture design.
Extensive experiments show the effectiveness of our approach.
Our discovered quantized model family, OQANets, achieves state-of-the-art (SOTA) results on the ImageNet dataset under 4/3/ bit-widths compared with various quantization methods.
In particular, our OQA2bit-L far exceeds the accuracy of 2bit Efﬁcient-B0@QKD (Kim et al., 2019) by a large 14% margin using 30% less computation budget.
Comparing with the quantization-aware NAS method APQ (Wang et al., 2020), our OQA4bit-L-MBV2 uses 43.7% less computation cost while maintaining the same accuracy as APQ-B.
To summarize, the contributions of our paper are three-fold:
• Our OQA is the ﬁrst quantization NAS framework to search for the architecture of quantized networks and deploy their quantized weights without retraining.
• We present the bit inheritance mechanism to reduce the bit-width progressively so that the higher bit-width models guide the search and training of lower bit-width models.
• We provide some insight into quantization friendly architecture design.
Our systematical analysis reveals that shallow-fat models are more likely to be quantization friendly than deep-slim models under low bit-width.
R ELATED W ORK
Network Architecture Search without retraining Slimmable neural networks (Yu et al., 2018)
ﬁrst proposes to train a model to support multiple width multipliers( e.g. 4 different global width multipliers for MobileNetV2).
OFA (Cai et al., 2019) and BigNAS (Yu et al., 2020) push the envelope forward in network architecture search(NAS) by introducing diverse architecture space (stage depth, channel width, kernel size, and input resolution).
These methods propose to train a single over-parameterized supernet from which we can directly sample or slice different candidate architectures for instant inference and deployment.
However, all of the aforementioned methods are tailored towards searching for ﬂoating-point efﬁcient models.
Converting the best ﬂoating-point architecture to quantization tends to result in sub-optimum quantization models.
In the quantization area, recent papers (Jin et al., 2019a; Yu et al., 2019) propose to train a single model that can support different bit-widths.
But they only quantize manually design networks( e.g. ResNet, MobileNetV2) under relatively high bit-width ( e.g. 4bit for MobileNetV2), our OQA can search architectures and produce quantized efﬁcient models with lower bit-width( e.g. 2bit).
Quantization-aware Network Architecture Search Recent studies combine network quantization and NAS to automatically search for layer bit-width with given architectures or search for operations with given bit-width.
HAQ (Wang et al., 2019) focuses on searching for different numbers of bits for different layers in a given network structure and shows that some layers, which can be quantized to low bits, are more robust for quantization than others.
AutoBNN (Shen et al., 2019) utilizes the genetic algorithm to search for network channels and
BMobi (Phan et al., 2020) searches for the group number of different convolution layer under certain 1bit.
SPOS (Guo et al., 2019) train a quantized one-shot supernet to search for bit-width and network channels for heavy ResNet (He et al., 2016).
BATS (Bulat et al., 2020) devises a binary search space and incorporate it within the DARTS framework (Liu et al., 2018a).
The aforementioned methods concentrate on the quantization of heavy networks, like ResNet (He et al., 2016), or replace the depthwise convolution with group convolution.
Moreover, they inherit a two-stage search-retrain schema: once the best-quantized architectures have been identiﬁed, they need to be retrained for deployment.
This procedure will signiﬁcantly increase the computational cost for the search if we have different deployment constraints and hardware bit-widths.
Compared with all these methods, our OQA can search for quantized efﬁcient models and learn their quantized weights at the same time without additional retraining.
Without our bit inheritance mechanism, these approaches also suffer from signiﬁcant drop of accuracy when a network is quantized to ultra-low bit-width like 2.
M ETHOD 3. O VERVIEW
Our OQA aims to obtain quantized efﬁcient models which can directly be sampled from quantized supernet without retraining.
As shown in Figure 1c, the overall procedure of OQA is as follows:
Step 1 , Quantized Supernet Training (Section 3.3): Train a K -bit supernet by learning the weight parameters and quantization parameters jointly.
Step 2: given a constraint on computational complexity, search the architecture with the highest quantization performance on the validation dataset from the supernet.
If K = 2 , the whole process is ﬁnished.
Step 3 , Bit Inheritance (Section 3.4):
Use the weight and quantization parameters of the K bit supernet to initialize the weight and quantization parameters of the K − 1 bit supernet.
Step 4: K ← K − 1 and Go to step 1.
The starting bit-width K of OQA and the steps of the bit-inheritance procedure can be arbitrary.
In this paper, we focus on the efﬁcient models under one ﬁxed low bit-width quantization strategy, thus the bit-width setting is 4/3/2.
3. P RELIMINARIES
Neural Architecture Search without Retraining.
Recently, several NAS methods (Yu et al., 2018; Cai et al., 2019; Yu et al., 2020) are proposed to directly deploy subnets from a well-trained supernet without retraining.
Speciﬁcally, a supernet with the largest possible depth (number of blocks), width (number of channels), kernel size, and input resolution is trained.
Then the subnet with top accuracy is selected as the searched network among the set of subnets satisfying a given computational complexity requirement.
A subnet is obtained from parts of the supernet with depth, width, and kernel size smaller than the supernet.
The subnet uses the well-trained parameters of the supernet for direct deployment without further retraining.
Quantization Network Learning.
In the quantization neural networks, a quantization function turns the ﬂoating-point weights, and activations into integers of given bit-width in the forward pass.
Given bit-width K , activations are quantized into unsigned integers in the range of [0 , 2 K − 1] and weights are quantized into signed integers in the range of [ − 2 K − 1 , 2 K − 1 − 1] .
To enable the training of quantized supernets, we choose a learnable quantization function following recent state-of-theart quantization method LSQ (Esser et al., 2019).
Given ﬂoating-point weights or activation v , and learnable scale s , the quantization function Q and its corresponding approximate gradient using the straight through estimator (Bengio et al., 2013).
Take the activation v as the example, we have
[Equation]
where all operations for v are element-wise operations, clip ( z, r 1 , r 2) returns z with values below r 1 set to r 1 and values above r 2 set to r 2 ,
(cid:98) z (cid:101) rounds z to the nearest integer, Q min and Q max are, respectively minimum and maximum integers for the given bit-width K , I ( v | s | , Q min , Q max ) means the gradient of v in the range of ( Q min × | s | , Q max × | s | ) is approximated by 1, otherwise 0.
| s | is the absolute value of s , ensuring that the semantics of scale s is only interval, without inverting the signs of weights or activation.
The scale s is learned by back-propagation and initialized as√ Q max × ¯ | v | , where ¯ | v | denotes the mean of | v | .
3. Q UANTIZED NAS WITHOUT R ETRAINING
Existing problems of Quantization NAS.
Existing weight-sharing-based quantization NAS methods suffer from more unreliable order preserving (Guo et al., 2019;
Bulat et al., 2020), as the quantization function introduces more instability on the learned weights.
In our perspective, combining non-retrain NAS methods with quantization to avoid this problem can be a natural solution.
Search space and quantized supernet.
We use a search space based on MobileNetV2 (Sandler et al., 2018) and MobileNetV3 (Howard et al., 2019), which has the ﬂexible input resolution, ﬁlter kernel size, depth (number of blocks in each stage), and width (number of channels).
Our search space consists of multiple stages.
Each stage stacks several inverted residual blocks.
Further details about search space can be found in Appendix A.5, A.
Unlike the ﬂoating-point supernet training (Cai et al., 2019; Yu et al., 2020), we utilize the quantization function as Eq.
1 to discretize the weights and activation values for the quantized supernet training.
Meanwhile, the ﬂoating-point weights need to be retained for reducing quantization loss at the training stage of each bit-width.
For weight w and input activation a of a convolution layer, we deﬁne the corresponding learnable scales of activation and weight as s a and s w .
The forward pass for a quantized convolution layer is deﬁned as follows:
[Equation]
where Q ( · , · ) is the learnable quantization function deﬁned in Eq.
1, * is the convolution operation and y is the output of this layer.
Subnet sampling.
During the supernet training, different subnets are sampled and trained in each iteration.
In non-retrain NAS methods, a subnet has a smaller scale than the supernet in resolution, width, depth, and kernel size, and is obtained by cropping the corresponding part from the supernet.
In our settings, a stage with d blocks in a subnet inherits the weights from the ﬁrst d blocks in the same stage in supernet.
A depthwise convolution layer in a subnet with width e and kernel size k are cropped from the central k ∗ k region of the ﬁrst e kernels in the supernet’s corresponding convolution layer.
The input image of each subnet are resized to its resolution r .
Our subnet sampling strategy combines the supernet training pipeline proposed in (Cai et al., 2019;
Yu et al., 2020), and it has two stages.
We ﬁrst only sample the biggest quantized subnet until it converges as in (Cai et al., 2019).
Afterward, we use the sandwich rules (Yu & Huang, 2019) to sample subnets.
Further details can be found in the Appendix A.2.
Architecture search of quantized supernet.
We directly evaluate the sampled subnets from the supernet without further retraining.
It’s worth to mention that we use the predictive accuracy on 10K validation images sampled from trainset to measure the subnets in the search procedure.
Furthermore, we exploit a coarse-to-ﬁne architecture selection procedure, similar to Yu et al.
We ﬁrst randomly sample 10K candidate architectures from the supernet with the FLOPs of the corresponding ﬂoating-point models ranging from 50M to 300M (2K in every 50M interval).
After obtaining the good skeletons (input resolution, depth, width) in the pareto front of the ﬁrst 10K models, we randomly perturb the kernel sizes to further search for better architectures.
3. Q UANTIZATION NAS WITH B IT I NHERITANCE
The problem of quantization NAS with lower bits.
When the quantization bit is lower thanbit, the traditional quantization-aware training (QAT) (Kim et al., 2019; Bhalgat et al., 2020) process is highly unstable and introduces very large accuracy degradation for the challenging case of thebit-width model.
Using the approach introduced in Section 3.3, we obtain the quantized supernet with the highest K = 4 bit-width.
To further obtain the quantized supernets of lower bit-widths ( e.g. K − 1 , K − 2 ), we can use the QAT to directly train quantized supernets for each bit-width.
As shown in Table 1, the biggest architecture in our search space suffers a 19.1% accuracy drop between 4-bit and 2-bit when using the QAT.
Besides, QAT requires much more computational cost.
[Table]
Inheritance from the high bit to low bit.
We propose a bit inheritance procedure to compensate for both the disadvantages.
First, we use the K bitwidth supernet to provide a good initialization for the weights of K − 1 bit-width supernet.
In the initialization for K − 1 bit-width supernet, we ﬁrst inherit the weight parameters and scale parameters from K bitwidth supernet.
Then the scale parameters are doubled because they map the ﬂoating-point values to the integer range of K − 1 bit-width, which is half the range of K bit-width.
Finally, to compensate for the statistics shift of each layer’s output caused by quantization error, we forward the model to recalculate the mean and variance of the BatchNorm layers (Yu & Huang, 2019) with randomly sampled 4096 training images.
During training, we use the K and K − 1 bit-width supernets as teacher and student, and train them in a knowledge distillation way Hinton et al. (2015) to further reduce the quantization error between the K − 1 and K bit-width parameters.
The beneﬁt of bit inheritance.
In the Appendix A.1, we show that the loss of weights of the K − 1 bit network is bounded (close to that of the K bit network) if bit inheritance is used, where the parameters of the K − 1 bit network inherit from the parameters of the K bit network.
Therefore, bit inheritance help to guarantee the 2-bit network to be close to the 3-bit network in training loss.
In comparison, existing methods like QAT start from ﬂoating point network which is far away from the 2 -bit network parameters and cause unstable training.
The experimental results in Table 1 also validate the effectiveness of our design.
The row named BitInheritance-Start means with only one epoch training, the initial accuracy is good enough in 3 bit.
After ﬁnetuning with fewer epochs, the bit-inheritance achieves higher accuracy performance, especially for 2 bit.
We further empirically show its advantages.
Take the biggest model in the search space for example, as shown in Table 1, the row named BitInherit-Start means with only recalculating scale, BatchNorm
Calibration and no training, the initial accuracy is good enough in 4 and 3 bit.
After ﬁnetuning with fewer epochs, the bit-inheritance achieves higher accuracy performance especially for 2-bits.
[Figure] The accuracy of Pareto@2bit/FP is obtained in the correponding 2bit FP supernet.  E XPERIMENTAL A NALYSIS 4. E XPERIMENTAL SETTINGS AND IMPLEMENTATION DETAILS
We evaluate our method on the ImageNet dataset (Deng et al., 2009).
If not speciﬁed, we follow the standard practice for quantized networks (Kim et al., 2019;
Gong et al., 2019) on quantizing the weights and activation for all convolution layers except the ﬁrst convolution, last linear layer, and the convolution layers in the SE modules(Hu et al., 2018).
To fairly compare with quantized efﬁcient models, the FLOPs is used and deﬁned as follows.
Denote the FLOPs of the FP layer by a , following (Zhou et al., 2016;
Li et al., 2019b; Phan et al., 2020;
Bulat et al., 2020), the FLOPs of the
FP layer is a , and the FLOPs of m bit weight and n bit activation quantized layer is mn 64 × a .
Unless otherwise noted, all results are sampled from MBV3 search space denoted as OQA, OQA-MBV represents the MBV2 search space.
Further training details can be found in Appendix A.2.
4. NAS- THEN -Q UANTIZE OR OQA
We denote pareto models as those models on the pareto front of cost/accuracy trade-off curve.
In
Figure 2a, we quantize the pareto models of the OFA ﬂoating-point supernet, which corresponds to the NAS-then-Quantize procedure in Figure 1a.
We compare it with the pareto models of our OQA that directly obtained from the quantized supernet.
In the comparison under 3 bit, the pareto curve of our OQA is far above that of the NAS-then-Quantize.
In Figure 2b, we sample 10k subnets from the search space, and we validate these architectures from the FP supernet and 2 bit supernet.
The pareto front of the subnets denoted as Parato@FP are selected with the FP accuracy and Pareto@2bit are selected with the 2 bit accuracy.
With the same accuracy of the ﬂoating-point models, the accuracy of the model from the 2 bit pareto is higher than the model from the FP pareto.
If our target is to search architectures for the quantized models, Figure 2b shows that searching from the quantized supernet as our OQA did is better than searching from FP supernet and then quantize.
The advantage is more evident for lower bits.
Further details can be found in Appendix A.3.
4. E XISTING Q UANTIZATION - AWARE NAS OR OQA
In Table 2, we compare our OQANet model family with several quantization-aware NAS methods, named SPOS (Guo et al., 2019),
BMobi (Phan et al., 2020), BATS (Bulat et al., 2020) and
APQ (Wang et al., 2020).
With the quantized supernets without retraining, our OQANet model family shows great advantages over traditional weight-sharing methods corresponding to the paradigm of Figure 1b.
While SPOS (Guo et al., 2019) focuses on the search of network channels and bit-width of heavy
ResNet (He et al., 2016), we focus on the search of efﬁcient models with ﬁxed bit-width and achieve [Table] [Figure] better results with fewer FLOPs.
BMobi (Phan et al., 2020) and BATS (Bulat et al., 2020) did not provide the results for 2bits, 3bits or 4 bits.
Therefore, we would like not to directly compare our approach with BMobi and BATS, because the results are obtained from different bit-widths.
However, if only the FLOPs-accuracy trade-off is concerned, our OQA with higher bit-width can be a better solution.
APQ corresponds to the NAS-then-Quantize paradigm in Figure 1a.
It ﬁrst searches for ﬂoating-point network architecture in the FP supernet, then trains a quantization network predictor to predict the searched quantized architecture.
The transfer learning from FP predictor to quantization predictor brings the proxy problem and it also needs retraining.
Our OQA4bit-L-MBV2 uses 43.7% less computation cost while maintaining the same accuracy as APQ B.
4. F URTHER COMPARISON WITH NAS- THEN - QUANTIZE METHODS
We compare with several strong quantization methods including LSQ (Esser et al., 2019),
LSQ+ (Bhalgat et al., 2020), APOT (Li et al., 2019b), QKD(Kim et al., 2019), SAT (Jin et al., 2019b), and LSQ* which is the LSQ implemented by us on different models to construct strong baselines.
The result of 4bit ResNet-18@LSQ* validates that our implementation is comparable.
Our OQA beneﬁts from joint quantization and network architecture search, as well as the bit inheritance for lower bits.
As shown in Table 3, our OQANets outperforms multiple quantization methods on models like MobileNetV2 (Sandler et al., 2018), EfﬁcientNet-B0 (Tan & Le, 2019) and MbV3 (Howard et al., 2019) under all bit-widths we implements.
4bits: OQA4bit-L has 1% accuracy gain than Efﬁcient-B0@QKD.
OQA4bit-M outperforms ResNet-18@LSQ with 10% of its FLOPs. 3bits: Our OQA3bit-L can also match the accuracy of 3bit ResNet-18@LSQ with [Table]
[Figure] [Table] [Figure] 13% FLOPs and 3 bit Efﬁcient-B0@QKD with 74% FLOPs.
2bits:
Our OQA2bit-L requires less
FLOPs but achieves signiﬁcantly higher Top-1 accuracy (64.0%) when compared with EfﬁcientNetB0@QKD (50.0%) and MobileNetV2@LSQ* (55.7%).
The results verify that the joint search of quantization parameters and network architectures results in more quantization-friendly efﬁcient models.
We also show more searched models in the Figure 3 and our OQANets signiﬁcantly outperform other quantization methods.
4. S HALLOW - FAT OR D EEP - SLIM
With the different quantized supernet obtained by the bit inheritance process, we analyze the following factors of a network: depth( D ), kernel size( K ), expand width( E ) and bits( B ).
According the intuitive shape, we divide the models into two group: 1) shallow-fat: D , K and E are sampled from { 2 , 3 } , { 5 , 7 } and { 4 , 6 } respectively, 2) deep-slim: D , K , and E are sampled from { 3 , 4 } , { 3 , 5 } and { 3 , 4 } respectively.
Following the rules above, we randomly generate 1.5K architectures and obtain the accuracy of the ﬂoating-point and quantized models from the corresponding supernet.
The input resolution is also randomly sampled.
For a certain model, we use the accuracy drop from ﬂoating-point to quantization to measure whether it is quantization friendly.
As shown in Figure 4, the shallow-fat models are more quantization friendly than deep-slim models.
When the 4-bit and 2-bit situations are compared, the trend becomes more obvious when the quantization bit is lower. [Table] [Table] 4. A BLATION S TUDY
The effectiveness of joint design.
In Table 4, the top-1 accuracy on ImageNet under different settings is listed. NAS-Then-Quantize means the model is selected from the pareto front with the ﬂoating point accuracy, and Joint Design is with the 2bit accuracy. With similar FLOPs, similar ﬂoating-point accuracy, Joint Design surpass the accuracy of Nas-Then-Quantize with 1 . 7% in 2bit accuracy. We also perform the quantization-aware training(QAT) with 150 epochs and the joint design results in over 5% accuracy improvement, which veriﬁes the effectiveness of Joint Design in ﬁnding quantization-friendly architectures.
The effectiveness of bit inheritance.
In Table 4, we further compare these two models with quantization-aware training(QAT) with 150 epochs and 500 epochs.
In QAT, pretrained ﬂoatingpoint models is used to provide a better initialization point, while our bit inheritance proposes to initialize with models with one bit-width higher.
Although 2bit models beneﬁt from QAT with more epochs, the bit-inheritance supernet still improve the two models’ accuracy by a large margin, even compared to training with pretrained ﬂoating-point weights by 500 epochs.
The huge accuracy improvement veriﬁes that bit-inheritance is a better practice compared with existing routine of quantization-aware training, because it alleviate the problem that low bit-width quantized efﬁcient models are highly unstable to train.
The efﬁciency of OQA framework.
In Table 5, we compare the search cost and the retrain cost of OQA with existing methods.
The search cost is deﬁned as the time cost of the supernet training and the search process to get the ﬁnal N searched models under latency targets.
The retrain cost is deﬁned as the training cost to get the ﬁnal accuracy of the searched architecture.
When N is larger than 5, the retrain cost of SPOS (Guo et al., 2019) and BMobi (Phan et al., 2020) already surpasses the total cost of OQA.
And We reduce at least half of the total cost compared with APQ (Wang et al., 2020).
APQ needs to train an once-for-all ﬂoating-point supernet and sample thousands of
FP subnets to perform quantization aware training, and thus requires the transfer learning from the ﬂoating-point predictor to quantization predictor.
Our OQA only needs to train one once-for-all quantized supernets and support a huge search space with over 10 19 subnets that can be directly sampled from supernet without retraining.
Thus, the average computational cost is relatively low.
In this paper, we present Once Quantized for All (OQA), a novel framework that searches quantized neural networks (QNNs) under the bit-width of 4 / 3 / 2 and deploys the learned quantized models at the same time without additional retraining.
With our proposed methods, we can search for the
OQANet model family which far exceeds the existing quantization aware nas and quantization methods which are applied to mobile models.
Our results reveal the potential of ultra-low bit quantized models under mobile settings.
A A PPENDIX A. E FFECTIVENESS OF BI
Given a convolution layer in the K bit superset, we denote s K as the scale parameters of this layer, and s K − 1 = 2 s K as the doubled scale for the K − 1 bit superset.
We use w and N w to denote the weights of this layer which is inherited from K to K − 1 .
Next, we show that the L 1 distance of Q ( w , s K ) and Q ( w , s K − 1 ) is bounded by N w · s K .
It means the initialized Q ( w , s K − 1 ) has a bounded distance with the well-trained Q ( w , s K ) .
For each w i , we have:
[Equation]
Based on this expression, we further get:
[Equation]
For any w i and s K , we have:
[Equation]
Thus,
[Equation]
and
[Equation]
A. T RAINING DETAILS
Dataset conﬁg:
We evaluate our method on the ImageNet dataset.
The training dataset is made up of 1.28 million images with resolution 224 × 224 belonging to 1000 classes and the validation set has 50k images.
For ImageNet training, we use the typical random resized crop, randomly horizontal ﬂipping and color jitter of [32 / 255 , 0 , 0 . 5 , 0] for data augmentation.
During evaluation, we ﬁrst determine the active image size s , and resize the image into (cid:100) s/ 0 .
875 (cid:101) ×
(cid:100) s/ 0 .
875 (cid:101) and center crop s × s image.
Quantization aware training: We reimplement LSQ (Esser et al., 2019) as our base quantization method.
We start from a ﬂoating-point model and ﬁnetune the model for 150 epochs.
The optimizer is SGD with Nesterov momentum 0.9 and weight decay 3e-5, and the label smoothing ratio is 0.1.
The initial learning rate is 0.04 under the batch of 1024, with the cosine annealing schedule.
The dropout rate is 0.1.
Except for learning rate and training epochs, we follow this protocol in the OQA procedure.
OQA procedure:
Combining the advantages of OFA (Cai et al., 2019) and BigNas (Yu et al., 2020), the overall OQA procedure is divided into four steps as follow:
Step0: we train the 4bit biggest models in the search space.
It follows the typical quantization-aware training, we use the ﬂoating-point pre-trained model as initialization and ﬁnetuning for 150 epoch.
The learning rate is 0.04 with a batch-size of 1024.
Step
The input resolution, kernel size, width, and depth are random sampled.
This whole process takes 200 epochs.
In one iteration, four models are sampled with the sandwich rule Yu & Huang (2019), which is the biggest subnet and the smallest subnet and two random sampled subnets.
The learning rate is 0.02 with a batch size of 1024.
With bit inheritance, the training of the 3/2bit supernet is simpliﬁed.
And the training time is reduced.
Step2:
In the 3bit supernet, we use the 4bit-supernet obtained in Architecture shrinking Step1 part as initialization.
We directly random sample input resolution, kernel, width, and depth.
four models are sampled, which is the biggest subnet and the smallest subnet and two random sampled subnets for one update.
We only use 25 epochs and the learning rate is 0.0016.
Step3:
In the 2bit supernet, we use the 3bit-supernet obtained in Step2 as initialization.
We also directly random sample resolution, kernel, width, and depth.
Four models are sampled, which is the biggest subnet and the smallest subnet and two random sampled subnets for one update.
We only use 120 epochs and the learning rate is 0.0256.
OQA subnet ﬁnetuning: Our OQA performance can be further improved by ﬁnetuning the subnet weights sliced from the OQA supernet as suggested by OFA (Cai et al., 2019).
The accuracy of the subnet is already higher than training from scratch.
In default, the subnets are ﬁnetuned for 25epochs.
The initial learning rate is 0.0016 with a batch size of 1024, with the cosine annealing schedule.
Knowledge distillation: The knowledge distillation(KD) used in our experiment is the traditional loss(KD) proposed in (Hinton et al., 2015).
The students logits and teacher logits are used to calcalute the cross-entropy loss. The temperate is 1 and the kd loss weight is 1.
[Figure]
A. NAS- THEN -Q UANTIZE OR OQA
[Figure]
In Figure 6, we sample 10k subnets from the search space, and we validate these subnets from the FP supernet, 3bit and 2bit supernet.
The pareto front of the subnets denoted as Parato@FP are selected with the accuracy in the ﬂoating-point supernet and Pareto@3bit/Pareto@2bit are selected with the accuracy in the 3bit/2bit supernet.
In Figure 6, the ﬁrst two ﬁgure reveals that as the bit decreases, the accuracy gain increases with searching directly in the quantized supernet.
We further compare the Pareto@3bit and Pareto@2bit, and it shows that with the same 3bit accuracy, the accuracy of the model from the 2bit pareto is higher than the model from the 3bit pareto, but the accuracy gain is less compared with the FP pareto.
To search for 2bit quantized models, it is best to search directly in the 2bit supernet, and it is better to search in the 3bit supernet than searching in the FP supernet.
Observed the difference in the accuracy of the pareto architectures in different supernets at the same 2bit, we are curious whether the accuracy performance is attributed to the structure difference in the pareto architecture.
In Figure 5, we plot the distribution of the total depth of the pareto subnets from different supernet.
In the depth dimension, the distribution reveals that Pareto@FP favors deeper models while Pareto@3bit/2bit favors shallow models.
And the distribution of depth is closer between Pareto@3bit and Pareto@2bit.
HE DETAILS OF QUANTIZATION - AWARE NAS
[Table]
[Table]
and last has no expand ratio.
Kernel size refers to the kernel size k of the depthwise convolution
[Table]
A. M OBILE N ET V3 S EARCH S PACE
[Table]
[Table]
C o n v 1 ×
C o n v 1 ×
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
C o n v 3 ×
M B E
K
M B E
K
C o n v 1 ×
C o n v 1 ×
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
C o n v 3 ×
M B E
K
C o n v 1 ×
C o n v 1 ×
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
M B E
K
[Figure]