O NCE FOR A LL : T RAIN O NE N ETWORK AND S PECIALIZE IT FOR E FFICIENT D EPLOYMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
We address the challenging problem of efﬁcient deep learning model deployment, where the goal is to design neural network architectures that can ﬁt different hardware platform constraints.
Most of the traditional approaches either manually design or use Neural Architecture Search (NAS) to ﬁnd a specialized neural network and train it from scratch for each case, which is computationally expensive and unscalable.
Our key idea is to decouple model training from architecture search to save the cost.
To this end, we propose to train a once-for-all network (OFA) that supports diverse architectural settings (depth, width, kernel size, and resolution).
Given a deployment scenario, we can then quickly get a specialized sub-network by selecting from the OFA network without additional training.
To prevent interference between many sub-networks during training, we also propose a novel progressive shrinking algorithm, which can train a surprisingly large number of sub-networks ( > 10 19 ) simultaneously.
Extensive experiments on various hardware platforms (CPU, GPU, mCPU, mGPU, FPGA accelerator) show that OFA consistently achieves the same level (or better)
ImageNet accuracy than SOTA NAS methods while reducing orders of magnitude GPU hours and CO 2 emission than NAS.
In particular, OFA requires 16 × fewer GPU hours than ProxylessNAS, 19 × fewer GPU hours than FBNet and 1,300 × fewer GPU hours than MnasNet under 40 deployment scenarios.
I NTRODUCTION
Deep Neural Networks (DNNs) deliver state-of-the-art accuracy in many machine learning applications.
However, the explosive growth in model size and computation cost gives rise to new challenges on how to efﬁciently deploy these deep learning models on diverse hardware platforms, since they have to meet different hardware efﬁciency constraints (e.g., latency, energy).
For instance, one mobile application on App Store has to support a diverse range of hardware devices, from a high-end Samsung Note10 to a 5-year-old Samsung S5 with a much slower processor.
With different hardware resources (e.g., on-chip memory size, #arithmetic units), the optimal neural network architecture varies signiﬁcantly.
Even running on the same hardware, under different battery conditions or workloads, the best model architecture also differs a lot.
Given different hardware platforms and efﬁciency constraints (deﬁned as deployment scenarios), researchers either design compact models specialized for mobile (Howard et al., 2017; Sandler et al., 2018; Zhang et al., 2018) or accelerate the existing models by compression (He et al., 2018) for efﬁcient deployment.
However, designing specialized DNNs for every scenario is engineer-expensive and computationally expensive, either with human-based methods or NAS.
Since such methods need to repeat the network design process and retrain the designed network from scratch for each case.
Their total cost grows linearly as the number of deployment scenarios increases, which will result in excessive energy consumption and CO 2 emission
(Strubell et al., 2019).
It makes them unable to handle the vast amount of hardware devices (23.14 billion IoT devices till 2018 1 ) and highly dynamic deployment environments (different battery conditions, different latency requirements, etc.).
This paper introduces a new solution to tackle this challenge – designing a once-for-all network that can be directly deployed under diverse architectural conﬁgurations, amortizing the training cost.
The
[Table]
[Figure] [Table]
Once for [Table]
[Table] Loss MobileNetV3-Small #5 Acc Loss MobileNetV
[Table] 5 Acc Loss Untitled8.
17.0 73.3 Untitled8. 24.0 75.2 Untitled10. 70.9 Untitled
[Table]
[Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table] [Figure] Once for All MobileNetV3-LargeMobileNetV3-Small
Samsung Note8 Latency (ms)
T opI m age N e t
A cc ( % ) 60.
[Figure] Previous
[Figure] O(N [Figure]
design cost [Figure] [Figure] cost
T r a i n F o u r
T i m e s ,
G e t F o u r
T r a i n O n c e , G e t M a n y
[Figure]
inference is performed by selecting only part of the once-for-all network.
It ﬂexibly supports different depths, widths, kernel sizes [Figure] and resolutions without retraining.
A simple example of Once for All (OFA) is illustrated in Figure 1 (left).
Speciﬁcally, we decouple the model training stage and the model specialization stage.
In the model training stage, we focus on improving the accuracy of all sub-networks that are derived by selecting different parts of the once-for-all network.
In the model specialization stage, we sample a subset of sub-networks to train an accuracy predictor and latency predictors.
Given the target hardware and constraint, a predictor-guided architecture search (Liu et al., 2018a) is conducted to get a specialized sub-network, and the cost is negligible.
As such, we reduce the total cost of specialized neural network design from O(N) to O(1)
(Figure 1 middle).
However, training the once-for-all network is a non-trivial task, since it requires joint optimization of the weights to maintain the accuracy of a large number of sub-networks (more than 10 19 in our experiments).
It is computationally prohibitive to enumerate all sub-networks to get the exact gradient in each update step, while randomly sampling a few sub-networks in each step will lead to signiﬁcant accuracy drops.
The challenge is that different sub-networks are interfering with each other, making the training process of the whole once-for-all network inefﬁcient.
To address this challenge, we propose a progressive shrinking algorithm for training the once-for-all network.
Instead of directly optimizing the once-for-all network from scratch, we propose to ﬁrst train the largest neural network with maximum depth, width, and kernel size, then progressively ﬁne-tune the once-for-all network to support smaller sub-networks that share weights with larger ones.
As such, it provides better initialization by selecting the most important weights of larger sub-networks, and the opportunity to distill smaller sub-networks, which greatly improves the training efﬁciency.
It also prevents smaller sub-networks from hurting the accuracy of larger sub-networks by optimizing in the local space around well-trained larger sub-networks.
We extensively evaluated the effectiveness of OFA on ImageNet with many hardware platforms (CPU,
GPU, mCPU, mGPU, FPGA accelerator) and efﬁciency constraints.
Under all deployment scenarios,
OFA consistently achieves the same level (or better)
ImageNet accuracy than SOTA hardware-aware
NAS methods while saving the GPU hours, dollars, and CO 2 emission by orders of magnitude when handling multiple diverse deployment scenarios.
R ELATED
W
[Equation]
Efﬁcient Deep Learning.
Many efﬁcient neural network architectures are proposed to improve the hardware efﬁciency, such as SqueezeNet (Iandola et al., 2016), MobileNets (Howard et al., 2017;
Sandler et al., 2018), ShufﬂeNets (Ma et al., 2018; Zhang et al., 2018), etc.
Orthogonal to architecting efﬁcient neural networks, model compression (Han et al., 2016) is another very effective technique for efﬁcient deep learning, including network pruning that removes redundant units (Han et al., 2015) or redundant channels (He et al., 2018;
Liu et al., 2017), and quantization that reduces the bit width for the weights and activations (Han et al., 2016; Courbariaux et al., 2015; Zhu et al., 2017).
Neural Architecture Search.
Neural architecture search (NAS) focuses on automating the architecture design process (Zoph & Le, 2017; Zoph et al., 2018;
Real et al., 2019;
Cai et al., 2018a; Liu et al., 2019).
Early NAS methods (Zoph et al., 2018; Real et al., 2019; Cai et al., 2018b) search for high-accuracy architectures without taking hardware efﬁciency into consideration.
Therefore, the produced architectures (e.g., NASNet, AmoebaNet) are not efﬁcient for inference.
Recent hardwareaware NAS methods (Tan et al., 2018;
Cai et al., 2019; Wu et al., 2019) directly incorporate the hardware feedback into architecture search.
As a result, they are able to improve the inference efﬁciency.
However, given new inference hardware platforms, these methods need to repeat the architecture search process and retrain the model, leading to prohibitive GPU hours, dollars and COemission.
They are not scalable to a large number of deployment scenarios.
The individually trained models do not share any weight, leading to large total model size and high downloading bandwidth.
Dynamic Neural Networks.
To improve the efﬁciency of a given neural network, some work explored skipping part of the model based on the input image.
For example, Wu et al.
; Liu &
Deng (2018); Wang et al.
(2018) learn a controller or gating modules to adaptively drop layers; Huang et al.
(2018) introduce early-exit branches in the computation graph; Lin et al.
(2017) adaptively prune channels based on the input feature map; Kuen et al.
(2018) introduce stochastic downsampling point to reduce the feature map size adaptively.
Recently, Slimmable Nets (Yu et al., 2019; Yu &
Huang, 2019b) propose to train a model to support multiple width multipliers (e.g., 4 different global width multipliers), building upon existing human-designed neural networks (e.g., MobileNetV2 0.35, 0.5, 0.75, 1.0).
Such methods can adaptively ﬁt different efﬁciency constraints at runtime, however, still inherit a pre-designed neural network (e.g., MobileNet-v2), which limits the degree of ﬂexibility (e.g., only width multiplier can adapt) and the ability in handling new deployment scenarios where the pre-designed neural network is not optimal.
In this work, in contrast, we enable a much more diverse architecture space (depth, width, kernel size, and resolution) and a signiﬁcantly larger number of architectural settings ( 10 19 v.s. 4 (Yu et al., 2019)).
Thanks to the diversity and the large design space, we can derive new specialized neural networks for many different deployment scenarios rather than working on top of an existing neural network that limit the optimization headroom.
However, it is more challenging to train the network to achieve this ﬂexibility, which motivates us to design the progressive shrinking algorithm to tackle this challenge.
M ETHOD 3. P ROBLEM F ORMALIZATION
Assuming the weights of the once-for-all network as W o and the architectural conﬁgurations as { arch i } , we then can formalize the problem as
[Equation]
where C ( W o , arch i ) denotes a selection scheme that selects part of the model from the once-for-all network W o to form a sub-network with architectural conﬁguration arch i .
The overall training objective is to optimize W o to make each supported sub-network maintain the same level of accuracy as independently training a network with the same architectural conﬁguration.
3. A RCHITECTURE S PACE
Our once-for-all network provides one model but supports many sub-networks of different sizes, covering four important dimensions of the convolutional neural networks (CNNs) architectures, i.e., depth, width, kernel size, and resolution.
Following the common practice of many CNN models (He et al., 2016; Sandler et al., 2018; Huang et al., 2017), we divide a CNN model into a sequence of units with gradually reduced feature map size and increased channel numbers.
Each unit consists of a sequence of layers where only the ﬁrst layer has stride 2 if the feature map size decreases (Sandler et al., 2018).
All the other layers in the units have stride 1.
We allow each unit to use arbitrary numbers of layers (denoted as elastic depth ); For each layer, we allow to use arbitrary numbers of channels (denoted as elastic width ) and arbitrary kernel sizes (denoted as elastic kernel size ).
In addition, we also allow the CNN model to take arbitrary input image sizes (denoted as elastic resolution ).
For example, in our experiments, the input image size ranges from 128 to 224 with a stride 4; the depth of each unit is chosen from { 2, 3, 4 } ; the width expansion ratio in each layer is chosen from { 3, 4, 6 } ; the kernel size is chosen from { 3, 5, 7 } .
7x
Transform Matrix 25x 5x Transform Matrix 9x 3x3 train with full width channel importance 0.02 0.15 0.85 0.63 channel sorting progressively shrink the width unit i train with full depth channel importance 0.82 0.11 0.46 reorg.
channel sorting reorg.
[-3] ⇒ ∈ ⇒ ∈ [Figure] Sample D at each stage; sample K Skip top (4-D) blocks at each stage (ﬁgure 4) Fine-tune weights D  [4, 3, 
-]{+progressively shrink the width channel sorting . ++ + p 1 p 2 p 3 O 1 O 2 O 3 O 1 O 2 Ounit i shrink the depth O1 O unit i shrink the depth O1 O
+}
O
[Figure]
Therefore, with , we have 3 × 3)[-4 ) 
-]
× × 3) network architectures and each of them can be used under 25 different input resolutions.
Since all of these sub-networks share the same weights (i.e., W o ) (Cheung et al., 2019), we only require 5.1M parameters to store all of them.
Without sharing, the total model size will be prohibitive.
3. T RAINING THE O NCE - FOR - ALL N ETWORK
Na¨ıve Approach.
Training the once-for-all network can be cast as a multi-objective problem, where each objective corresponds to one sub-network.
From this perspective, a na¨ıve training approach is to directly optimize the once-for-all network from scratch using the exact gradient of the overall objective, which is derived by enumerating all sub-networks in each update step, as shown in Eq.
The cost of this approach is linear to the number of sub-networks.
Therefore, it is only applicable to scenarios where a limited number of sub-networks are supported (Yu et al., 2019), while in our case, it is computationally prohibitive to adopt this approach.
Another na¨ıve training approach is to sample a few sub-networks in each update step rather than enumerate all of them, which does not have the issue of prohibitive cost.
However, with such a large number of sub-networks that share weights thus interfere with each other, we ﬁnd it suffers from signiﬁcant accuracy drop.
In the following section, we introduce a solution to address this challenge by adding a progressive shrinking training order to the training process.
Correspondingly, we refer to the na ¨ ıve training approach as random order .
Progressive Shrinking.
The once-for-all network comprises many sub-networks of different sizes where small sub-networks are nested in large sub-networks.
To prevent interference between the sub-networks, we propose to enforce a training order from large sub-networks to small sub-networks in a progressive manner.
We name this training order as progressive shrinking (PS).
An example of the training process with PS is provided in Figure 2, where we start with training the largest neural network with the maximum kernel size (i.e., 7), depth (i.e., 4), and width (i.e., 6).
Next, we progressively ﬁne-tune the network to support smaller sub-networks by gradually adding them into the sampling space (larger sub-networks may also be sampled).
Speciﬁcally, after training the largest network, we ﬁrst support elastic kernel size which can choose from { 3, 5, 7 } at each layer, while the depth and width remain the maximum values.
Then, we support elastic depth and elastic width sequentially, as is shown in Figure 2.
The resolution is elastic throughout the whole training process, which is implemented by sampling different image size for each batch of training data.
We also use the knowledge distillation technique after training the largest neural network (Hinton et al., 2015;
Ashok et al., 2018; Yu & Huang, 2019b).
It combines two loss terms using both the soft labels given by the largest neural network and the real labels.
Compared to the na¨ıve approach, PS prevents small sub-networks from interfering large sub-networks, since large sub-networks are already well-trained when the once-for-all network is ﬁne-tuned to support small sub-networks.
Additionally, during ﬁne-tuning, the model is optimized in the local space around the well-trained large sub-networks by using a small learning rate and revisiting (i.e., sampling) well-trained large sub-networks.
Regarding the small sub-networks, they share the weights with the large ones.
Therefore, PS allows initializing small sub-networks with the most important weights of well-trained large sub-networks, which expedites the training process.
We describe the details of the PS training ﬂow as follows:
• Elastic Kernel Size (Figure 3 left).
We let the center of a 7x7 convolution kernel also serve as a 5x5 kernel, the center of which can also be a 3x3 kernel.
Therefore, the kernel size becomes elastic.
The challenge is that the centering sub-kernels (e
g., 3x3 and 5x5) are shared and need to play multiple roles (independent kernel and part of a large kernel).
The weights of centered
[Figure]
++
[Figure]
p
[Figure]
[Figure]
matrices for different layers.
Within each layer, the kernel transformation matrices are shared among different channels.
As such, we only need 25 × 25 + 9 × 9 = 706 extra parameters to store the kernel transformation matrices in each layer, which is negligible.
• Elastic Depth (Figure 3 right).
To derive a sub-network that has D layers in a unit that originally has N layers, we keep the ﬁrst D layers and skip the last N − D layers, rather than keeping any D layers as done in current NAS methods (Cai et al., 2019; Wu et al., 2019).
As such, one depth setting only corresponds to one combination of layers.
In the end, the weights of the ﬁrst D layers are shared between large and small models.
• Elastic Width (Figure 4).
Width means the number of channels.
We give each layer the ﬂexibility to choose different channel expansion ratios.
Following the progressive shrinking scheme, we ﬁrst train a full-width model.
Then we introduce a channel sorting operation to support partial widths.
It reorganizes the channels according to their importance, which is calculated based on the L norm of a channel’s weight.
Larger L1 norm means more important.
For example, when shrinking from a 4-channel-layer to a 3-channel-layer, we select the largest 3 channels; whose weights are shared with the 4-channel-layer (Figure 4 left and middle).
Thereby, smaller sub-networks are initialized with the most important channels on the once-for-all network which is already well trained.
This channel sorting operation preserves the accuracy of larger sub-networks.
3. S PECIALIZED M ODEL D EPLOYMENT WITH O NCE - FOR - ALL N ETWORK
Having trained a once-for-all network the next stage is to derive the specialized sub-network for a given deployment scenario.
The goal is to search for a neural network that satisﬁes the efﬁciency (e.g., latency, energy) constraints on the target hardware while optimizing the accuracy.
Since OFA decouples model training from architecture search, we do not need any training cost in this stage.
To eliminate the repeated search cost, we use the predictor-guided architecture search process (Liu et al., 2018b) to ﬁnd specialized sub-networks.
Speciﬁcally, we randomly sample 16K sub-networks with different architectures and input image sizes, then measure their accuracy on 10K validation images sampled from the original training set.
These [architecture, accuracy] pairs are used to train an accuracy predictor to predict the accuracy of a model given its architecture and input image size 2 .
Additionally, we build a latency lookup table (Cai et al., 2019) on each target hardware platform to predict the latency.
Given a target hardware and latency constraint, we conduct evolutionary search (Real et al., 2019) based on the predictors to get a specialized sub-network.
Since the cost of searching with predictors is negligible, we only need
GPU hours to collect the data pairs, and the cost stays constant regardless of #deployment scenarios.
[Table]
E XPERIMENTS
In this section, we ﬁrst apply the progressive shrinking algorithm to train the once-for-all network on
ImageNet (Deng et al., 2009).
Then we demonstrate the effectiveness of our trained once-for-all network on various hardware platforms (LG G8, Samsung Note8, Google Pixel1, Pixel2, NVIDIA 1080Ti, V100 GPUs, Intel Xeon CPU, Jetson TX2, Xilinx ZU9EG and ZU3EG FPGAs) with different latency constraints.
4. T RAINING THE O NCE - FOR - ALL N ETWORK ON I MAGE N ET
Training Details.
We use the same architecture space as MobileNetV3 (Howard et al., 2019) 3 .
For training the full network, we use the standard SGD optimizer with Nesterov momentum 0.9 and weight decay 3 e − 5 .
The initial learning rate is 2.6, and we use the cosine schedule (Loshchilov & Hutter, 2016) for learning rate decay.
The full network is trained for 180 epochs with batch sizeon 32 GPUs.
Then we follow the schedule described in Figure 2 to further ﬁne-tune the full network 4 . The whole training process takes around 1,200 GPU hours on V100 GPUs.
This is a one-time training cost that can be amortized by many deployment scenarios.
Results.
Table 1 reports the top1 accuracy of sub-networks derived from the once-for-all networks that are trained with our progressive shrinking (PS) algorithm and random order respectively.
Due to space limits, we take 8 sub-networks for comparison, and each of them is denoted as “(D = d , W = w , K = k )”.
It represents a sub-network that has d layers for all units while the expansion ratio and kernel size are set to w and k for all layers.
Compared to random order, PS can improve the ImageNet accuracy of sub-networks by a signiﬁcant margin under all architectural settings.
Speciﬁcally, under the full architectural setting (i.e., D=4, W=6, K=7), PS can achieve the same level ImageNet top accuracy as the independently trained full network (77.3%) while random order can only achieve 73.8% which is 3.5% lower.
4. S PECIALIZED S UB - NETWORKS FOR D IFFERENT H ARDWARE AND C ONSTRAINTS
We apply our trained once-for-all network to get specialized sub-networks for different hardware platforms.
For the GPU platforms, the latency is measured with batch size 64 on NVIDIA 1080Ti and V100 with Pytorch 1.0+cuDNN.
The CPU latency is measured with batch size 1 on Intel Xeon
E5-2690 v4+MKL-DNN 5 .
To measure the mobile latency, we use LG G8, Samsung Note8, Google
Pixel1 and Pixel2 using TF-Lite with batch size 1.
On Jetson TX2, we use a batch size of 2 (single batch causes severe under-utilization).
On Xilinx ZU9EG and ZU3EG FPGAs, we use DNNDK with batch size 1.
Comparison with NAS on Mobile.
Table 2 reports the comparison between OFA and state-of-theart hardware-aware NAS methods on the mobile platform (Pixel1).
OFA is much more efﬁcient than NAS when handling multiple deployment scenarios, since the cost of OFA is constant while others are linear to the number of deployment scenarios ( N ).
With N = 40, the training time of OFA is 16 × faster than ProxylessNAS, 19 × faster than FBNet, and 1,300 × faster than MnasNet. Without [Table]
Samsung Note8 Once for All #5 Acc Loss
[Table]
for All # #5 Acc Loss MobileNetV3-Small #5 Acc Loss MobileNetV3-Large #5 Acc Loss Untitled22. 73.3 Untitled24.
[Table]
Pixel1 Once for All
Untitled 1 20.0 Untitled 2 23.
Untitled 3 25.0 Untitled 4 27. [Table] [Figure] retraining, OFA achieves 76.0% top1 accuracy on ImageNet, which is 0.8% higher than MobileNetV3Large while maintaining similar mobile latency.
By ﬁne-tuning the specialized sub-network for 25 epochs, we can further improve the accuracy to 76.4% (1.2% higher than MobileNetV3-Large).
Besides, we also observe that OFA with PS can achieve 3.6% better accuracy than without PS, showing the effectiveness of PS. Model
[Table]
Table 3 summarizes the results on Pixel1 under different latency constraints.
OFA can re-design specialized neural networks for all scenarios without additional cost, while previous methods rescale an existing model using a width multiplier due to the high search and training cost (Sandler et al., 2018;
Cai et al., 2019; Tan et al., 2018).
As shown in Table 3, we can consistently achieve better ImageNet accuracy than MobileNetV3, the current state-of-the-art neural network architecture on the mobile platforms.
Specifically, with similar latency as MobileNetV3-Small 1.0, we improve the ImageNet top1 accuracy from 67.4% to 69.6% (+2.2%) without retraining, and to 71.4% (+4.0%) after ﬁne-tuning for 25 epochs.
OFA is more effective than NAS + width multiplier adjustment.
Figure 5 reports detailed comparisons between OFA and MobileNetV3 on three mobile devices (Samsung Note8, Pixel1, and Pixel2).
Remarkably, OFA can produce the entire trade-off curves with many points over a wide range of latency constraints. It is impossible for previous NAS methods (Tan et al., 2018; Cai et al., 2019) due to the prohibitive training cost.
[Table]
8.0 Jetson TX2 Latency (ms) Batch Size =
Xilinx ZCU102 Latency (ms) Batch Size = Xilinx UltraZed-EG Latency (ms) Batch Size =60. [Figure] =60.
[Figure]
ms [Figure] Size =60.
Jetson TX [Figure] [Figure] Size =60.
Results on Diverse Hardware Platforms.
Besides mobile platforms, we extensively studied with effectiveness of OFA on 6 additional hardware platforms (Figure 6) using the ProxylessNAS architecture space (Cai et al., 2019). The detailed architecture of the full network can be found in Appendix C.
OFA consistently improve the trade-off between accuracy and latency by a signiﬁcant margin, especially on GPUs which have more parallelism.
With similar latency as MobileNetV2 0.35, “OFA #25” improves the ImageNet top1 accuracy from MobileNetV2’s 60.3% to 71.7% (+11.4% improvement) on the 1080Ti GPU.
It reveals the insight that using the same model for different deployment scenarios with only the width multiplier modiﬁed has limited impact on efﬁciency improvement: the accuracy drops quickly as the latency constraint gets tighter.
On Xilinx ZU3EG, a low-resource FPGA, some width settings of MnasNets even fail to run due to “out of BRAM”.
We proﬁled the FPGA and drew its rooﬂine model (see Appendix B): OFA models improved the arithmetic intensity by up to 48%/43% and GOPS/s by 70%/92% compared with the MobileNetV2/MnasNet family.
OFA designs specialized model that better ﬁts the hardware.
C ONCLUSION
We proposed Once for All (OFA), a new methodology that decouples model training from architecture search for efﬁcient deep learning deployment under a large number of deployment scenarios.
Unlike previous approaches that design and train a neural network for each deployment scenario, we designed a once-for-all network that supports different architectural conﬁgurations, including elastic depth, width, kernel size, and resolution.
It greatly reduces the training cost (GPU hours, energy consumption, and CO 2 emission) compared to conventional methods.
To prevent sub-networks of different sizes from interference, we proposed a progressive shrinking algorithm that enable a large number of sub-network to achieve the same level of accuracy compared to training them independently.
Experiments on a diverse range of hardware platforms and efﬁciency constraints demonstrated the effectiveness of our approach.
A D ETAILS OF THE A CCURACY P REDICTOR
We use a three-layer feedforward neural network that has 400 hidden units in each layer as the accuracy predictor.
Given a model, we encode each layer in the neural network into a one-hot vector based on its kernel size and expand ratio, and we assign zero vectors to layers that are skipped.
Besides, we have an additional one-hot vector that represents the input image size.
We concatenate these vectors into a large vector that represents the whole neural network architecture and input image size, which is then fed to the three-layer feedforward neural network to get the predicted accuracy.
In our experiments, this simple accuracy prediction model can provide very accurate predictions.
At convergence, the root-mean-square error (RMSE) between predicted accuracy and estimated accuracy on the test set is only 0.21%.
Figure 7 shows the relationship between the RMSE of the accuracy prediction model and the ﬁnal results (i.e., the accuracy of selected sub-networks).
We can ﬁnd that lower RMSE typically leads to better ﬁnal results.
[Figure]
B R OOFLINE M ODEL ON FPGA
We analyze our specialized models on FPGA using DNNDK 6 .
The proﬁling results are summarized in Figure 8, while the rooﬂine models (Williams et al., 2009) on ZU9EG and ZU3EG are illustrated in Figure 9.
Unlike high-end FPGAs that have DRAM modules which are directly connected to the programmable logic (PL), ZU9EG and ZU3EG share the same DRAM module with the processing system (includes the ARM CPU).
Thus memory bandwidth is an expensive resource.
Correspondingly, our specialized models reduce costly memory accesses by increasing arithmetic intensity (i [Figure] e., the number of arithmetic operations for each byte of memory access), which results in higher GOPS/s
[Figure]
[Table] [Figure]
t
i
[Table]
I
[Table]
t
[Figure]
[Table]
[Figure]
O u t o f BR A
M
O u t o f BR A
M
FPGA Arithmetic Intensity (op/B) MobileNetV
MnasNet
OFA 0.
63.9 A r i t
h m e t i c I n t en s i
t
y
( O PS / B y t e ) 0 18 35 531.0x
FPGA UltraZed-EG GOPS/s MobileNetV2 MnasNet OFA 0.35 36 31.8 {+61.2 0.5 48.1 44.0 75.5 0.75 67.8 81.
+}
1.0 79 83.7 Z U 3 E G F P G A ( GO P S / s ) 0 23 45 68 90 FPGA ZCU102 GOPS/s-1 MobileNetV
MnasNet
OFA 0.
186.3 Z U 9 E G F P G A ( GO P S / s ) 0 50 100 150 200 MobileNet-v
MnasNet
OFA (Ours) (under different latency constraints)
[Figure]
0.75x 0.5x 0.35x 1.0x 0.75x 0.5x 0.35x 1.0x 0.75x 0.5x 0.35x (under different latency constraints) (under different latency constraints)
C D ETAILED A RCHITECTURE OF THE F ULL N ETWORK
The detailed architectures of the full networks in the MobileNetV3 search space and ProxylessNAS search space are illustrated in Table 4 (left) and Table 4 (right) respectively
[Table]
D V ISUALIZATION OF O UR S PECIALIZED M ODELS
[Table]
[Table] [Table]
212x
[Table]
3x
M B3x M B3x
M B3x
[Table]
on v 3x 160x
M B3x M B3x
M B5x M B5x
M B5x
M B5x M B5x
M B3x M B3x
M B7x
M B5x
P oo li ng F C
M B3x
C on v 3x 208x
M B3x M B3x
M B7x
M B5x
M B7x
M B5x
M B7x
M B5x M B5x
M B3x
M B7x
M B5x
M B3x
M B7x
P oo li ng F C
M B3x
C on v 3x 224x
M B5x M B5x
M B3x M B3x
M B5x
M B7x
M B5x
M B3x
M B5x
M B5x M B5x
M B7x
M B5x M B5x
M B3x
M B5x
P oo li n g F C
[Table]
[Table]
on Intel Xeon CPU.
[Table]
[Table] [Table]
M
[Table]
3x
M
[Table]
3x
M
[Table]
3x
M B3x
M
[Table]
3x
M
[Table]
3x
M
[Table]
3x
M B3x
M
[Table]
3x
[Table] [Table]
3x 212x
[Table]
3x
[Table]
3x
[Table]
3x
M
[Table]
5x
[Table]
3x
M
[Table]
5x
M
[Table]
5x
[Table]
3x
[Table]
3x
[Table]
5x
M
[Table]
3x
P oo li ng
[Table]
[Table] [Table]
160x
[Table]
5x
[Table]
7x
[Table]
5x
[Table]
oo li
[Table]
208x
[Table]
7x
[Table]
7x
[Table]
5x
[Table]
7x
[Table] [Table]
7x
[Table]
5x
[Table]
P oo
[Table]
ng
[Table]
224x
M B5x M B5x
M B3x M B3x
M B5x
M B7x
M B5x
M B3x
M B5x M B5x M B5x
M B7x
M B5x M B5x
M B3x
M B5x
P oo li ng F C (f) 17.4ms latency on Intel Xeon CPU.
E I MPLEMENTATION D ETAILS OF P ROGRESSIVE S HRINKING
After training the full network, we ﬁrst have one stage of ﬁne-tuning to incorporate elastic kernel size.
In this stage (i.e., K ∈ [7 , 5 , 3] ), we sample one sub-network in each update step.
The network is ﬁne-tuned for 125 epochs with an initial learning rate of 0.96.
All other training settings are the same as training the full network.
Next, we have two stages of ﬁne-tuning to incorporate elastic depth.
We sample two sub-networks and aggregate their gradients in each update step.
The ﬁrst stage (i.e., D ∈ [4 , 3] ) takes 25 epochs with an initial learning rate of 0.08 while the second stage (i.e., D ∈ [4 , 3 , 2] ) takes 125 epochs with an initial learning rate of 0.24.
Finally, we have two stages of ﬁne-tuning to incorporate elastic width.
We sample four sub-networks and aggregate their gradients in each update step.
The ﬁrst stage (i.e., W ∈ [6 , 4] ) takes 25 epochs with an initial learning rate of 0.08 while the second stage (i.e., W ∈ [6 , 4 , 3] ) takes 125 epochs with an initial learning rate of 0.24.