Automated Super-Network Generation for Scalable Neural Architecture Search
Abstract Weight-sharing Neural Architecture Search (NAS) solutions often discover neural network architectures that outperform their human-crafted counterparts.
Weight-sharing allows the creation and training of super-networks that contain many smaller and more efficient child models, a.k.a., sub-networks.
For an average deep learning practitioner, generating and training one of these super-networks for an arbitrary neural network architecture design space can be a daunting experience.
In this paper, we present BootstrapNAS, a software framework that addresses this challenge by automating the generation and training of super-networks.
Developers can use this solution to convert a pre-trained model into a super-network.
BootstrapNAS then trains the super-network using a weight-sharing NAS technique available in the framework or provided by the user.
Finally, a search component discovers high-performing sub-networks that are returned to the end-user.
We demonstrate BootstrapNAS by automatically generating super-networks from popular pre-trained models (MobileNetV2, MobileNetV3, EfficientNet, ResNet50 and HyperSeg), available from Torchvision and other repositories.
BootstrapNAS can achieve up to 9 .
87 × improvement in throughput in comparison to the pre-trained Torchvision ResNet-50 (FP32) on Intel Xeon platform.
Our code is available at https://github.com/jpablomch/bootstrapnas
1 Introduction
Neural Architecture Search (NAS) solutions attempt to identify the architectures with the best performance from a search space of possible neural network configurations.
In the past few years, weight-sharing NAS approaches have produced outstanding results (Cai, Gan, et al., 2020; Yu and Huang, 2019b).
These approaches build a super-network from which smaller and, in some cases, more efficient child models, a.k.a., sub-networks, can be extracted.
Unfortunately, generating a super-network, either from scratch or from an existing pre-trained model, can be a challenging experience.
One has to create the main data structure for the super-network, and include mechanisms for activating, extracting, forward and backward passing on selected subnetworks.
This process is repeated when a new super-network needs to be generated for a different search space.
In this paper, we present BootstrapNAS, a software framework for scalable super-network generation and training.
The BootstrapNAS approach focuses on automatically deriving a supernetwork from an existing network architecture.
This paper extends our short workshop paper (Muñoz et al., 2021).
This work provides the following contributions to the AutoML community:
I. A software framework that automates the generation and training of super-networks from pre-trained models, and the subsequent search for high-performing sub-networks.
II.
Highly extensible APIs that allow developers and researchers to incorporate their own methods for training super-networks and discovering high-performing models.
III.
A hardware-aware solution that incorporates measurements collected at a target hardware during the sub-network search stage, or by using trained predictors for these measurements.
Users can create their own performance estimators and provide them to the BootstrapNAS’ API.
2 Related Work
There has been significant progress in Neural Architecture Search (NAS) in the past few years.
At the core of all NAS solutions is the exploration of a search space , guided by a search strategy and a performance estimation strategy (Elsken et al., 2019).
In this paper, the focus is on NAS approaches that avoid training several individual models in separate training events since this is impractical for large neural architecture design spaces.
The emphasis is on weight-sharing NAS approaches, e.g., (Bender et al., 2018;
Cai, Gan, et al., 2020; Guo et al., 2020; G. Li et al., 2019;
Liu et al., 2018; Pham et al., 2018; Yu, Jin, et al., 2020), and in particular on those that are or can be made hardware-aware during the sub-network search stage, e.g., (Cai, Gan, et al., 2020; Yu and Huang, 2019a).
Several frameworks have been proposed to automate the generation of the NAS search space and identify high-performing models.
ModularNAS (Lin et al., 2021) provides a unified interface that allows the user to implement several state-of-the-art NAS methods, including super-network training and super-network-based search.
The Retiarii framework (Zhang et al., 2020), a component of Microsoft’s Neural Network Intelligence (NNI) (Microsoft, 2021) allows the user to design the model space, and then apply a search strategy. NASLib is a library that requires minimal coding efforts to define or reuse an existing search space (Ruchte et al., 2020), enabling researchers to quickly test their NAS methods on well-known search spaces. Another relevant work is Fast Neural Network Adaptation (FNA) (Fang et al., 2020), in which the architecture and parameters of a high-performing pre-trained backbone are used to produce alternative architectures, i.e., sub-networks, for different tasks, e.g., detection and segmentation.
Other frameworks have been proposed in the past to define a NAS search space.
For instance, DeepArchitect (Negrinho and Gordon, 2017; Negrinho, Patil, et al., 2019) provides a custom language for representing the search space, and separately trains a set of models from scratch.
In contrast to these frameworks, BootstrapNAS automatically generates the search space from a given pre-trained model, automating the construction of a super-network and minimizing the required coding from the user.
Limitations.
Generalizing the automated generation of NAS super-networks from arbitrary pretrained models is a great challenge that we confront as an iterative process.
Future versions of BootstrapNAS will improve its capabilities allowing for more users to benefit from this software framework.
Currently, there might be models that are not suitable for this approach, either because they have been efficiently optimized and it is difficult to discover high-performing sub-networks, or because they contain custom operators that are still not supported in BootstrapNAS, hence preventing the generation of their super-networks.
Another limitation might be the complex configurations and setting of hyperparameters that might be required for some models.
In some cases, BootstrapNAS’ automatic detection of potential elastic layers might result in a vast search space affecting training and searching times.
To address this concern, BootstrapNAS allows for a manual setting of the super-network’s elasticity hyper-parameters.
(Elasticity is defined in section 3.1).
Societal Impact.
Applications that use deep learning models are ubiquitous.
Unfortunately, they come with a significant environmental cost.
For instance, the training and deployment of these models are associated with a significant increase in CO 2 emissions.
We believe that research in efficient solutions for model compression, e.g., BootstrapNAS, will result in energy savings by producing smaller models with a reduced carbon footprint compared to their less efficient hand-crafted counterparts.
[Table]
BootstrapNAS is a software framework for the automated generation of weight-sharing super-networks for
Neural Architecture Search (NAS).
It is being developed within the Neural Network Compression Framework (NNCF) (Kozlov et al., 2020).
As illustrated in Figure 1, 1 BootstrapNAS takes as input a pre-trained model, 𝑚 , and a dataset, 𝐷 , from the user.
It then, transforms 𝑚 into a supernetwork, Ω .
3 The super-network is trained using one of the available training strategies.
4 Once the super-network has been trained, BootstrapNAS searches for high-performing subnetworks which are returned to the user at 5 .
In the following sections, we describe each stage in more detail.
3.1 Automated Super-Network Generation
BootstrapNAS automatically generates a super-network, Ω , from a pre-trained model, 𝑚 .
A weightsharing super-network is similar in its structure to other neural networks.
The difference is that in a super-network, it is possible to activate different configurations for some of its layers, allowing for the extraction of child models, a.k.a.
sub-networks.
The literature in this topic often uses the term elasticity to refer to the capability of certain operations in a neural network to be configured with different values (Cai, Gan, et al., 2020).
The objective at this stage in BootstrapNAS is to make a few selected layers elastic in Ω , hence allowing the possibility of manipulating smaller weight-sharing sub-networks.
Currently, BootstrapNAS automatically generates super-networks applying elasticity to three dimensions of the network: depth, width and kernel size.
Converting 𝑚 into a super-network Ω is accomplished by using the capabilities of the Neural
Network Compression Framework (NNCF).
NNCF traces the pre-trained model’s graph and wraps each candidate operator with pre- and post-operations effectively making an operator 𝑒𝑙𝑎𝑠𝑡𝑖𝑐 .
This step is illustrated in Figure 2.
During this stage, the super-network, Ω , starts as a copy of 𝑚 and then selected layers are transformed into elastic layers.
The set of layers in Ω , 𝐿 Ω , is composed of two subsets of layers.
One subset of elastic layers, 𝐿 Ω 𝑒 , and another subset of static layers, 𝐿 Ω 𝑠 .
Static layers are shared by all the sub-networks, while elastic layers, 𝐿 Ω 𝑒 , might not always be shared by all sub-networks.
Once an operator has been wrapped, BootstrapNAS can automatically activate different configurations for each layer.
Each elastic layer has various possible values for its properties.
For instance, the layer could allow several values for its number of channels.
e.g., {512, 256, 128}, or kernel sizes in the case of convolutional layers.
Since the super-network, Ω , is created from the pre-trained model, 𝑚 , the maximum value for a property in a layer 𝑙 Ω 𝑖 of Ω will be equal to the original value for the same property in the same layer, 𝑙 𝑚𝑖 of 𝑚 .
Among the many sub-networks, 𝑎 𝑖 , contained in the super-network Ω , there are two subnetworks that are of particular importance: the maximal sub-network 𝑎 𝑚𝑎𝑥 , which has a configuration of layers, and the values for its properties equal to those in the given pre-trained model 𝑚 , and hence the transformation from 𝑚 to Ω must guarantee that the maximal sub-network 𝑎 𝑚𝑎𝑥 results in the same accuracy (within a small margin of error) as the original model 𝑚 on a dataset 𝐷 𝑣𝑎𝑙 .
That is, 𝐶𝑜𝑠𝑡 ( 𝑎 𝑚𝑎𝑥 , 𝐷 𝑣𝑎𝑙 ) (cid:27) 𝐶𝑜𝑠𝑡 ( 𝑚, 𝐷 𝑣𝑎𝑙 ) .
Otherwise, this is an indication that an error occurred during the super-network generation step.
The other sub-network that is of particular importance is the minimal sub-network, 𝑎 𝑚𝑖𝑛 .
The configuration of this sub-network has the minimum possible value for each property in every elastic layer.
More details on how BootstrapNAS enables layer elasticity can be found in Appendix ?? .
3.2 Super-Network Training
Once the super-network has been automatically generated, BootstrapNAS trains it using one of the available training strategies.
The literature on super-network training contains several training strategies, e.g., progressive shrinking (Cai, Gan, et al., 2020) or single stage training (Yu, Jin, et al., 2020).
BootstrapNAS uses progressive shrinking by default.
Using this strategy, the training of the super-network occurs in multiple stages, and in decreasing order for each of the properties of elastic layers.
For instance, it first activates different elastic depth configurations in decreasing order, then it activates elastic depth and elastic width in decreasing order, and so on.
The training schedule is derived from the training schedule of the source model, which simplifies the requirements for the user.
Knowledge distillation can be applied during training (Hinton et al., 2015).
The soft labels from 𝑚 or from the maximal subnetwork, 𝑎 𝑚𝑎𝑥 , can be used to compute the loss of the sampled sub-networks.
Using the soft labels of 𝑎 𝑚𝑎𝑥 is termed inplace distillation (Yu and Huang, 2019b)[-
-]
A simple single stage training can be obtained from within the progressive shrinking strategy by activating all the elastic dimensions at once and allowing for the activation of all their possible values at the layers’ properties.
Then, a subset of sub-networks are sampled at random, and their gradients are aggregated and used to update the weights of the super-network.
A more complex single stage training, e.g., the sandwich rule (Yu and Huang, 2019b) requires, at each training step, the sampling of the maximal, 𝑎 𝑚𝑎𝑥 and minimal 𝑎 𝑚𝑖𝑛 sub-networks, together with other 𝑛 sub-networks sampled at random.
As with other training strategies, the weights are aggregated and used to update the super-network.
BootstrapNAS’ API is highly extensible allowing advanced users to implement their own training strategies.
3.3 Sub-Network Search
Once the super-network optimization training stage has finished, the next step is to search and return 𝑘 sub-networks from the set of Pareto-optimal sub-networks, 𝐴 𝑜 , ( 𝐴 𝑜 ⊆ 𝐴 ), to the user.
We explain below how 𝐴 𝑜 is obtained.
As default, 𝑘 = 1, that is, a single sub-network from the Pareto set that outperforms the original model, 𝑚 , is returned to the user.
To select 𝑘 networks from 𝐴 𝑜 , BootstrapNAS favors sub-networks that have an accuracy similar to 𝑚 (with some tolerance) but improve efficiency, e.g., minor drop in accuracy but significant improvement in latency.
The search of Pareto-optimal sub-networks, 𝐴 𝑜 can be approached as a multi-objective optimization.
The multi-objective goal is to,
[Equation]
for a given set of 𝑛 objective functions 𝑓 1 : X → R , . . .
, 𝑓 𝑛 : X → R , where x is a member of an objective decision space X (Emmerich and Deutz, 2018).
Multi-objective evolutionary algorithms (MOEAs), specifically Pareto-based MOEAs, are well suited for sub-network search problems since they can operate easily on the discrete variable types given by the super-network elastic parameter space.
Many are designed to evolve towards sets of Pareto optimal solutions that have a diverse spread across the Pareto front (objective trade-off solution region).
As a default search algorithm, BootstrapNAS uses NSGA-II (Deb, Pratap, et al., 2002).
NSGA-II uses a generational loop process that evolves a population of individuals (e.g., sub-networks) using crossover and mutation operations and then ranks the different individuals in the population using a non-dominated sorting with crowding distance criterion to produce a diverse set of Pareto-optimal solutions.
With NSGA-II, a user is provided with a wide and diverse spread of sub-network options across the objective space.
BootstrapNAS uses by default the multi-objective optimization capabilities of 𝑃𝑦𝑚𝑜𝑜 (Blank and Deb, 2020), which enables the smooth application of NSGA-II and other search algorithms.
For instance, a user can randomly sample sub-networks and then use the performance measurements of a particular sub-network as a reference point to other algorithms.
In the case where only a specific region of the objective space is desirable, RNSGA-II (Deb and Sundar, 2006) offers a solution to direct the search.
As illustrated in Figure 3, BootstrapNAS can use the output of the user’s random search and a particular accuracy target to explore a region of the model space.
Reference point based multi-objective optimization approaches allow one or more reference points to be defined in the objective space and where the ranking of solutions are calculated by the euclidean distance to each reference point.
In our example, RNSGA-II requires fewer evaluations than NSGA-II. While NSGA-II [Figure] [Figure]
r
[Figure] i c t e d L a t [Figure] n c y
[Figure]
N o r
[Figure]
a li z e
[Figure]
[Figure]
a comparison of predicted versus measured latency from optimal sub-networks identified during a NSGA-II search using the simple example latency predictor.
Although error is introduced when using predictors, the discovered sub-networks tend to outperform the ones found with random search.
can take a significant number of evaluations to produce good results, it is easily applied to any super-network, does not require reference points, and shows consistent evolutionary progressions towards finding better sub-network solutions.
Performance Estimation
The search stage in super-network-based NAS solutions often requires a significant amount of time depending on the size of the search space and the selected search strategy, mostly because of the cost of measuring the performance metrics of the sub-networks.
One common approach is to sample and evaluate a subset of sub-networks on metrics, such as accuracy and latency, and use these measurements to train predictors since search spaces can often reach a very large number of possible sub-networks configurations, e.g., 10 19 for MobileNetV3 (Cai, Gan, et al., 2020).
Lookup tables are another common form of predicting latency which aggregates delay by layer operation to approximate the full delay of the sub-network of interest (Cai, Zhu, et al., 2019).
Once these predictors have been created, the search stage can take a relatively short amount of time.
Other metrics that are commonly used to approximate model size and complexity are model parameter counts and multiply-accumulates (MACs).
The evaluation of these metrics tends to take less time.
The best practice when using predictor approaches is to perform a final validation measurement on the best candidate sub-networks since predictors introduce some level of inaccuracy depending on how they were trained.
The rationale is that to search across tens of thousands of sub-network options quickly, you can just use a well-behaved predictor, which can be trained in less than a thousand samples.
BootstrapNAS allows any predictor modeling approach to be used jointly with the sub-network search component.
While we use ridge regression for our example, another comprehensive work (Lu et al., 2021) covers the use of different predictor models (e.g., MLP, RBF, decision trees).
To illustrate the use of predictors, we randomly sample subnetworks from the BootstrapNAS Resnet-50 model and measure their latency.
In Figure 4, we illustrate that the latency predictor can achieve a Spearman’s rank correlation coefficient 𝜌 > 0 .
in as few as 200 training samples in this example.
Next, we perform 20 generation (population=50) NSGA-II search using the latency predictor and measure the actual latency on the best Pareto-optimal sub-networks at the end of the search.
When viewed in the latency and accuracy objective space as in Figure 4, the resulting Pareto front sub-network solutions behave as expected.
In the BootstrapNAS framework, predictors can be utilized on one or both objectives during multi-objective search, and the end-user has the flexibility to implement their own estimators and pass them as arguments during the search stage.
4 Experiments
The main goal of these experiments is to demonstrate the generalization capabilities of BootstrapNAS for automated super-network generation, training and search for high-performing subnetworks.
We generated several super-networks from popular models using BootstrapNAS.
Some of these super-networks might be suitable for the application of other model compression techniques, e.g., quantization, or longer super-network training to further improve their performance.
In the process of implementing the BootstrapNAS approach, we started by extending the code from (Cai, Gan, et al., 2020) to generate super-networks from Torchvision’s 1 ResNet-50 (He et al., 2016) model trained with Imagenet (Deng et al., 2009) and HyperSeg Lite trained with the
VFX dataset (Rhodes and Goel, 2020).
In our second development iteration, we implemented
BootstrapNAS’ current scalable open-
Super-Networks from Models Trained with CIFAR-10.
To demonstrate BootstrapNAS’ automated generalizable super-network generation capabilities, we selected two models: ResNet-50 and MobileNet-V2, both trained with CIFAR-10 from (Phan, 2021).
As illustrated in Figure 5 (two plots on the right), several sub-networks discovered by BootstrapNAS outperform the given pre-trained models.
We highlight two sub-networks for each super-network (BootstrapNAS A-RC, B-RC, A-MC, and B-MC).
Table 2 describes the reduction in required MACs by these sub-networks while either maintaining the accuracy of the baseline model, or with a minor drop in accuracy.
Super-Networks from Models Trained with CIFAR-100.
We also used BootstrapNAS to generate a super-network from EfficientNet-B0 (CIFAR-100).
This model has a top 1 accuracy of 87 .
02% after transfering from Imagenet weights ( EfficientNet-PyTorch 2021).
As described in Table 3,
BootstrapNAS discovered a sub-network that reduces the number of MACs by 12% with a minimum drop of accuracy from 87 .
02% to 86 .
This improvement is achieved with only 20 epochs of super-network training on the elastic depth dimension.
Super-Networks from Models Trained with Imagenet.
Three super-networks were generated from Resnet-50, MobilenetV2, MobileNetV3, all pre-trained models available at Torchvision and trained with Imagenet.
In all three cases, BootstrapNAS discovered sub-networks that are more
[Figure]
Trained with CIFARfrom (Phan, 2021)
Pre [Table]
[Table] efficient than the given pre-trained model. Figure 5 on the left illustrates the search progression of NSGA-II on ResNet-50.
This figure highlights three sub-networks: BootstrapNAS 𝐴 provides a reduction in model size in terms of MACs by 66.1%, with an accuracy drop of less than 1%, while BootstrapNAS 𝐵 reduces MACs by 40.8% with an improvement in accuracy.
The third sub-network obtained from the pre-trained ResNet-50, BootstrapNAS 𝐶 , maintains the top 1 accuracy of the input model while reducing the number of operations in MACs by 53.1%.
Improvements are observed for the MobileNetV2 and MobileNetV3 models from Torchvision, as well.
As described in Table 3, in the case of MobilenetNetV2, BootstrapNAS discovered a subnetwork that reduced the number of required MACs by 12.5% with a minimal drop in accuracy from 71.88% to 71.42%.
In the case of MobilenetV3, BootstrapNAS discovered a sub-network that requires 21% fewer MACs than the given model with a minimal drop in accuracy from 74 .
04% to 73 .
This improvement is achieved with 25 epochs of super-network training and only enabling the elastic dimension.
We used the selected sub-networks from the ResNet-50 super-network to analyze their performance using a dual-socket Intel ® Gold 6252 CPU @
2.10GHz (Cascade Lake), each CPU with 24 physical cores.
To evaluate the latency of the models, we processed a single sample (of size 1), measured the completion time, and calculated the 90th-percentile latency in milliseconds.
BootstrapNAS 𝐴 provides the largest improvement in latency over the input model with 2 . 16 × improvement [Table] [Table]
[Table]
v e
[Table] n t [Table] i g
[Table]
r i
[Table]
b
[Table]
e
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
in FP32 and 3 .
08 × in INT8 as shown in Figure 6 on the left.
BootstrapNAS 𝐵 provides the minimal improvement of 1 . 47 × and 1 .
87 × for FP32 and INT8, respectively.
To evaluate throughput in samples per second, we used batch processing where the latency is unconstrained and all data is available and processed in any order.
BootstrapNAS 𝐴 improves the throughput from the original model by 2 . 66 × in FP32 and 2 .
57 × in INT8.
BootstrapNAS 𝐵 and 𝐶 provide comparable performance with improvements in throughput of 1.55-1 . 98 × for both FP32 and INT8.
Overall, BootstrapNAS 𝐴 - 𝐶 in INT8 can achieve a 6.02-9 .
87 × improvement in throughput in comparison to the pre-trained FP Torchvision ResNet-50 model.
Super-Networks from Models trained with the VFX Segmentation Dataset.
We also generated a super-network from a pre-trained HyperSeg Lite model (Rhodes and Goel, 2020).
HyperSeg is a model for end-to-end interactive video segmentation tasks for high-resolution (2K) data.
HyperSeg outperforms state-of-the-art models for interactive segmentation (Z. Li et al., 2018), DOS (Xu[-
-] et al., 2016), Graph Cut (Boykov and Jolly, 2001), and Random Walk (Grady, 2006) with a mIoU of 0.840.
We used HyperSeg Lite which uses lower resolution data (224x224) and obtains a mIoU of 0.765.
Figure 6 on the right shows the improvements obtained by three sub-networks from the BootstrapNAS’ super-network: BootstrapNAS Seg A , with 1 . 38 × less latency than the baseline, and BootstrapNAS Seg B , with 1 .
19 × less latency.
Both sub-networks improve on accuracy, 2% and 3.8% respectively in comparison to the baseline model’s accuracy.
BootstrapNAS also discovered a sub-network, BootstrapNAS Seg C , with similar latency as the baseline but with a 5.9% improvement in accuracy.
5 Conclusion
We have presented BootstrapNAS, a framework for neural architecture search based on the automated generation of weight-sharing super-networks from existing pre-trained models.
BootstrapNAS supports super-network training applied to various elastic dimensions and the search for high-performing sub-networks.
BootstrapNAS has a simple and extensible API to enable custom implementations of super-network training and search algorithms.
We demonstrated the feasibility of BootstrapNAS by applying it to public pre-trained models, showing that it can produce sub-networks with substantial improvements in the performance-accuracy trade-off even for lightweight models such as MobileNet and EfficientNet.
Acknowledgements.
Thank you to Ravi Iyer, Yuri Gorbachev and Soren Knudsen for their continuous support to this research.
Thank you to the NNCF/OpenVINO team for their help integrating this research into NNCF.
In particular, thank you to Wonju Lee, Daniil Lyakhov and Minje Park.
We would also like to thank Vui Seng Chua and Yash Akhauri for their invaluable feedback