Rapid Model Architecture Adaption for Meta-Learning
Abstract
Network Architecture Search (NAS) methods have recently gathered much attention.
They design networks with better performance and use a much shorter search time compared to traditional manual tuning.
Despite their efficiency in model deployments, most NAS algorithms target a single task on a fixed hardware system.
However, real-life few-shot learning environments often cover a great number of tasks ( T ) and deployments on a wide variety of hardware platforms ( H ).
The combinatorial search complexity T × H creates a fundamental search efficiency challenge if one naively applies existing NAS methods to these scenarios.
To overcome this issue, we show, for the first time, how to rapidly adapt model architectures to new tasks in a many-task many-hardware few-shot learning setup by integrating Model Agnostic Meta Learning (MAML) into the NAS flow.
The proposed NAS method (H-Meta-NAS) is hardware-aware and performs optimisation in the MAML framework.
H-Meta-NAS shows a Pareto dominance compared to a variety of NAS and manual baselines in popular few-shot learning benchmarks with various hardware platforms and constraints.
In particular, on the 5-way 1-shot MiniImageNet classification task, the proposed method outperforms the best manual baseline by a large margin ( 5 . 21% in accuracy) using 60% less computation.
Introduction
Existing Network Architecture Search (NAS) methods show promising performance on image [Zoph and Le, 2016, Liu et al., 2018], language [Guo et al., 2019, So et al., 2019] and graph data [Zhao et al., 2020b].
The automation not only reduces the human effort required for architecture tuning but also produces architectures with state-of-the-art performance in domains like image classification [Zoph and Le, 2016] and language modeling [So et al., 2019].
Most NAS methods today focus on a single task with a fixed hardware system, yet real-life model deployments covering multiple tasks and various hardware platforms will significantly prolong this process.
As illustrated in Figure 1, a common design flow is to re-engineer the architecture and train for the different task( T )-hardware
( H ) pairs with different constraints ( C ).
The architectural engineering phase can be accomplished whether
[Figure] + [Figure] manually or by using an established NAS procedure.
The challenge is designing an efficient method to overcome the quickly scaling O ( THC ) search complexity described in Figure 1.
Few-shot learning systems follow exactly this many-task many-device setup, when considering deployments on different user devices on key applications such as facial [Guo et al., 2020] and speech recognition [Hsu et al., 2020].
A task in few-shot learning normally takes an N -way K -shot formulation, where it contains N classes with K support samples and Q query samples in each class.
Model-Agnostic Meta-Learning (MAML),
incorporating the idea of learning to learn, builds a meta-model using a great number of training tasks, and then adapts the meta-model to unseen test tasks using only a very small number of gradient updates [Finn et al., 2017].
MAML then becomes a powerful and elegant approach for few-shot learning – its ability to quickly adapt to new tasks can potentially shrink the O ( THC ) complexity illustrated in Figure 1 to O ( HC ) .
In the meantime, hardware-aware NAS methods [Cai et al., 2019, 2018, Xu et al., 2020], e.g. the train-once-for-all technique [Cai et al., 2019], support deployments of searched models to fit different hardware platforms with various latency constraints.
These hardware-aware NAS techniques further reduce the search complexity from O ( THC ) to O ( T )
[Cai et al., 2018].
In this paper, we propose a novel Hardware-aware Meta Network Architecture Search (H-Meta-NAS).
Integration of the MAML framework into hardware-aware NAS theoretically reduces the search complexity from O ( THC ) to O (1) , allowing for a rapid adaption of model architectures to unseen tasks on new hardware systems.
However, we identified the following challenges in this integration:
• Classic NAS search space contains many over-parameterized sub-models, this makes it hard to tackle the over-fitting phenomenon in few-shot learning.
• Hardware-aware NAS profiles latency for sub-networks on each task-hardware pair, this profiling can be prolonged significantly with a great number of tasks and, more importantly, if the targeting device has scarce computation resources.
To tackle these challenges, we then propose to use Global Expansion (GE) and Adaptive Number of Layers (ANL) to allow a drastic change in model capabilities for tasks with varying difficulties.
Our experiments later demonstrate that such changes alleviate over-fitting in few-shot learning and improve the accuracy significantly.
We also present a novel layer-wise profiling strategy to allow the reuse of profiling information across different tasks.
We make the following contributions:
• We propose a novel Hardware-aware Network Architecture Search for Meta learning (HMeta-NAS).
H-Meta-NAS quickly adapts meta-architectures to new tasks with hardwareawareness and can be conditioned with various device-specific latency constraints.
The proposed NAS reduces search complexity from O ( THC ) to O (1) in a realistic many-task many-device few-shot learning setup.
We extensively evaluate H-Meta-NAS on various hardware platforms (GPU, CPU, mCPU, IoT, ASIC accelerator) and constraints (latency and model size), our latency-accuracy performance generally outperforms various MAML baselines and other NAS competitors.
• We propose a task-agnostic layer-wise profiling strategy that reduces the profiling runtime from around 10 5 hours to 1.2 hours when targeting hardware with limited capabilities ( e.g. IoT devices).
• MAML models are prone to over-fitting in the few-shot learning setup Antoniou et al.
We show several design options for the NAS algorithm, namely Global Expansion and Adaptively Number of Layers respectively.
These methods help the NAS to overcome the over-fitting problem by optimizing in the architectural design space.
• We reveal that popular Metric-based MAML methods are not latency-friendly .
Our empirical results suggest that Optimization-based MAML method with well-tuned architectures can achieve comparable accuracy with significantly less latency overhead ( ≈ 2000 × on Mini-ImageNet 5-way 1-shot classification).
Background 2.1 Few-shot learning in the MAML framework
Inspired by human’s ability to learn from only a few tasks and generalize the knowledge to unseen problems, a meta learner is trained over a distribution of tasks with the hope of generalizing its learned knowledge to new tasks Finn et al.
[Equation]
Equation (1) captures the optimization objective of meta-learning, where optimal parameters are obtained through optimizing on a set of meta-training tasks.
Current approaches of using metalearning to solve few-shot learning problems can be roughly categorized into three types:
Memorybased, Metric-based and Optimization-based.
Memory-based method utilizes a memory-augmented neural network [Munkhdalai and Yu, 2017,
Gidaris and Komodakis, 2018] to memorize meta-knowledge for a fast adaption to new tasks.
Metricbased methods aim to meta-learn a high-dimensional feature representation of samples, and then apply certain metrics to distinguish them.
For instance, Meta-Baseline utilizes the cosine nearest-centroid metric [Chen et al., 2020] and DeepEMD applies the Wasserstein distance [Zhang et al., 2020].
Optimization-based method, on the other hand, focuses on learning a good parameter initialization (also known as meta-parameters or meta-weights ) from a great number of training tasks, such that these meta-parameters adapt to new few-shot tasks within a few gradient updates.
The most wellestablished Optimization-based method is Model-Agnostic Meta-Learning (MAML)
[Finn et al., 2017].
MAML is a powerful yet simple method to tackle the few-shot learning problem, since its adaption relies solely on gradient updates.
Antoniou et al.
later demonstrate MAML++, a series of modifications that improved MAML’s performance and stability.
Baik et al.
introduce an additional network for generating adaptive parameters for the inner-loop optimization.
Despite the rise in popularity of the meta-learning framework applied to few-shot learning, little attention has been paid to the runtime efficiency of these approaches.
In this work, we reveal that the current mainstream Metic-based methods all suffer from a severe latency overhead , since Metricbased approaches rely on metric comparisons and have to use multiple inference runs to generate these metrics (at least two) for a single image classification.
With the help of H-Meta-NAS, we show how Optimization-based methods can achieve a similar level of accuracy but using significantly less computation at inference time.
2.2 Network architecture search
Architecture engineering is a tedious and complex process requiring a lot of effort from human experts.
Network Architecture Search (NAS) focuses on reducing the amount of manual tuning in this design space.
Early NAS methods use evolutionary algorithms and reinforcement learning to traverse the search space [Zoph and Le, 2016, Real et al., 2017].
These early methods require scoring architectures trained to a certain convergence and thus use a huge number of GPU hours.
Two major directions of NAS methods, Gradient-based and Evolution-based methods, are then explored in parallel in order to make the search cost more affordable.
Gradient-based NAS methods use
Stochastic Gradient Descent (SGD) to optimize a set of probabilistic priors that are associated with architectural choices [Liu et al., 2018, Casale et al., 2019].
Although these probabilistic priors can be made latency-aware [Wu et al., 2019, Xu et al., 2020], it is challenging to make them follow a hard latency constraint.
Evolution-based NAS, on the other hand, operates on top of a pre-trained super-net and use evolutionary algorithms or reinforcement learning to pick best-suited sub-networks [Cai et al., 2018, 2019], making it easier to be constrained by certain hardware metrics.
For instance,
Once-for-all (OFA) is an Evolution-based NAS method and its searched networks are not only optimized for a specific hardware target but also constrained by a pre-defined latency budget
[Cai et al., 2019].
Our proposed H-Meta-NAS shares certain similarities to Once-for-all, since this method offers a chance to reduce the hardware search complexity from O ( HC ) to O (1) .
Latency is a key metric in mobile applications, and obviously can be also improved through other techniques such as quantization [Zhao et al., 2019a,
Hönig et al., 2022] and sparsification [Gao et al., 2018, Wang et al., 2019] in various learning setups, however, many of these optimizations might be ineffectual [Nikoli´c et al., 2019] unless the underlying hardware has special support for it
[Su et al., 2018, Zhao et al., 2019b, Parashar et al., 2017].
NAS algorithm is a promising approach to address this problem, since it can directly optimize the network topology; recent NAS methods have also been extended to
Transformers [So et al., 2019] and Graph Neural Networks [Zhao et al., 2020b,a].
Several NAS methods are proposed under the MAML framework [Kim et al., 2018, Shaw et al., 2018,
Lian et al., 2019], these methods successfully reduce the search complexity from O ( T ) to O (1) .
However, some of these methods do not show significant performance improvements compared to carefully designed MAML methods ( e.g. MAML++)
[Kim et al., 2018, Shaw et al., 2018].
In the meantime, some of these MAML-based NAS methods follow the Gradient-based approach and operate on complicated cell-based structures [Lian et al., 2019].
We illustrate later how cell-based
NAS causes an undesirable effect on latency, and also meets fundamental scalability challenges when trying to deploy in a many-task many-device few-shot learning setup.
Method
Problem formulation In the MAML setup, we consider a set of tasks T and
each task T i ∈ T contains a support set D si and a query set D qi .
The support set is used for task-level learning while the query set is in charge of evaluating the meta-model.
All tasks are divided into three sets, namely meta-training ( T train ), meta-validation ( T val ) and meta-testing ( T test ) sets.
Equation (2) formally states the objective of the pre-training stage illustrated in Figure 3a.
The objective of this process is to optimize the parameters θ of the super-net for various sub-networks sampled from the architecture set A .
This will ensure the proposed H-Meta-NAS to have both the meta-parameters and meta-architectures ready for the adaption to new tasks.
[Equation]
Equation (3) describes how H-Meta-NAS adapts network architectures to a particular task T with a given hardware constraint C h .
In practice, using the support set data D si from a target task T i , we apply a genetic algorithm for finding the optimal architectures α ∗ .
We discuss further how this process in detail in later sections.
[Equation]
Architecture search space H-Meta-NAS considers a search space composed of different kernel sizes, the number of channels, and activation types.
We mostly consider a VGG9-based NAS backbone, that is a 5-layer CNN model with the last layer being a fully connected layer.
We chose this NAS backbone because both MAML Finn et al.
[2017] and MAML++ Antoniou et al.
[2018] used a VGG9 model architecture.
The details of this backbone are in Appendix A.
We allow kernel sizes to be picked from { 1 , 3 , 5 } , channels to be expanded with a set of scaling factors { 0 . 25 , 0 . 5 , 0 .
75 , 1 , 1 . 5 , 2 , 2 . 25 } and also six different activation functions (details in Appendix).
For a single layer, there are 3 × 7 × 6 = 126 search options.
H-Meta-NAS also contains an Adaptive Number of Layers strategy, the network is allowed to use a subset of the total layers in the supernet with maximum usage of 4 layers.
The whole VGG9-based backbone then gives us in total 126 4 × 4 ≈ 10possible neural network architectures.
In addition, to demonstrate the ability of H-Meta-NAS on a more complex NAS backbone.
We also studied an alternative ResNet12-based NAS backbone, that has approximately 2 × 10 24 possible sub-networks. [Figure] [Figure]
Layer-wise profiling Hardware-aware NAS needs the run-time of sub-networks on the targeting hardware to guide the search process Cai et al.
[2019], Xu et al.
However, the profiling stage can be time-consuming if given a low-end hardware as the profiling target and the search space is large.
For instance, running a single network inference of VGG9 on the Raspberry Pi Zero with a 1GHz single-core ARMv6 CPU takes around 2 .
365 seconds to finish.
If we assume this is the average time needed for profiling a sub-network, given that the entire search space includes around 10 9 sub-networks, a naive traverse will take a formidable amount of time which is approximately 6 × 10 5 hours.
More importantly, the amount of profiling time scales with the number of hardware devices ( O ( H ) ).
Existing hardware-aware NAS schemes build predictive methods to estimate the run-time of sub-networks Cai et al.
[2019], Xu et al.
[2020] and have a relatively significant error.
An illustration of a profiler is shown in Figure 2, and the latency predictor’s illustration is in Figure 8.
Adaption strategy
The adaption strategy uses a genetic algorithm Whitley [1994] to pick the best-suited sub-network with respective to a given hardware constraint, the full algorithm is detailed in Appendix D.
In general, the adaption algorithm randomly samples a set of tasks from T val , and uses the averaged loss value and satisfaction to the hardware constraints as indicators for the genetic algorithm.
The genetic algorithm has a pool size P and number of iterations M .
We then run a hyper-parameter analysis in Figure 3b to determine the values of P and M .
The full adaption algorithm makes use of these hyper-parameters is in our Appendix.
The horizontal axis shows the number of iterations and the vertical axis shows the averaged accuracy on the sampled tasks for all architectures in the pool.
Figure 3b shows that the accuracy convergence is reached after [Figure] around 150 iterations, and running for additional iterations only provides marginal accuracy gains.
For this reason, we picked the number of iterations to be 200 for a balance between accuracy and run-time.
In the meantime, we notice in general a higher pool size will give better-adapted accuracy.
However, this does not mean the final searched accuracy is affected to the same degree.
The final re-trained accuracy of searched architectures show an accuracy gap of 0 .
21% between P = 100 and
P = 200 and 0 .
32% between P = 100 and P = 500 .
An increase in pool size can prolong the run-time significantly, we thus picked a pool size of 100 since it offers the best balance between accuracy and run
NAS backbone design, Global Expansion (GE) and Adaptive Number of Layers (ANL)
One particular problem in few-shot learning is that models are prone to over-fitting.
This is because only a small number of training samples are available for each task and the network normally iterate on these samples many times Antoniou et al.
We would like to explore the architectural space to help models overcome over-fitting and conduct a case study for different design options available for the backbone network.
We identify the following key changes to the NAS backbone to help the models to have high accuracy in few-shot learning:
• n × n pooling:
Pooling that applied to the final convolutional operation, n × n indicates the height and width of feature maps after pooling.
• Global Expansion (GE):
Allowing the NAS to globally expand or shrink the number of channels of all layers.
• Adaptive Number of Layers (ANL): Allowing the NAS to use an arbitrary number of layers, the network then is able to early stop using only a fewer number of layers.
Figure 4 further illustrates how GE and ANL can allow a much smaller model compared to existing
NAS backbones.
We also discuss in Section 3 how the pooling strategy can join the NAS process.
Super-net meta-training strategy As illustrated by prior work
Cai et al.
[2019], progressively shrinking the super-net during meta-training can reduce the interference between sub-networks.
We observe the same phenomenon and then use a similar progressive shrinking strategy in H-Meta-NAS, the sampling process α ∼ p
( A ) will pick the largest network with a probability of p , and randomly pick other sub-networks with a probability of 1 − p .
We apply an exponential decay strategy to p :
[Equation]
p e and p i are the end and initial probabilities.
e is the current number of epochs, and e s and e m are the starting and end epochs of applying this decaying process.
α determines how fast the decay is.
A graphical illustration is shown in Figure 3a.
In our experiment, we pick p i = 1 .
and e s = 30 , because the super-net reaches a relatively stable training accuracy at that point.
We then start the decaying process, and the value α = 5 is determined through a hyper-parameter study shown in our Appendix B. [Table]
Evaluation
We evaluate H-Meta-NAS on a range of popular few-shot learning benchmarks.
For each dataset, we search for the meta-architecture and meta-parameters.
We then adapt the meta-architecture with respect to a target hardware-constraint pair.
In the evaluation stage, we then re-train the obtained hardware-aware task-specific architecture to convergence and report the final accuracy.
We consider three popular datasets in the few-shot learning community: Omniglot, Mini-ImageNet and Few-shot
CIFAR100.
We use the PytorchMeta framework to handle the datasets [Deleu et al., 2019].
Omniglot is a handwritten digits recognition task, containing 1623 samples [Lake et al., 2015].
We use the meta train/validation/test splits originally used Vinyals et al.
Vinyals et al.
These splits are over 1028/172/423 classes (characters).
Mini-ImageNet is first introduced by Vinyals et al. .
This dataset contains images of 100 different classes from the ILSVRC-12 dataset [Deng et al., 2009], the splits are taken from Ravi et al.
[Ravi and Larochelle, 2016].
Table 10 in our appendix details the systems and representative devices considered.
We use the
ScaleSIM cycle-accurate simulator Samajdar et al.
[2018] for the Eyeriss Chen et al.
[2016] accelerator.
This simulation and more datasets and search configurations information are in Appendix F.
Evaluating layer-wise profiling re implemented the predictor as a baseline and compare it to our layer-wise profiling.
We pick 16K training samples and 10K validation samples to train and test the latency predictor, which is the same setup used in OFA.
We use another 10K testing samples to evaluate the performance of OFA-based latency predictor against our layer-wise profiling on different hardware systems in terms of MSE (measuring the latency estimation quality) and Time (measuring the efficiency).
As illustrated in Table 1, layer-wise profiling saves not only time but also has a smaller MSE error compared to a predictor-based strategy that is very popular in today’s evolutionary-based NAS frameworks [Cai et al., 2019, 2018].
In addition, layer-wise profiling shows orders of magnitude better run-time when targeting hardware devices with scarce computational resources.
If we consider an IoT class device as a target (i.e the Raspberry Pi Zero), it requires an unreasonably large amount of time to generate training samples for latency predictors, making them an infeasible approach in real life.
For instance, the total time consumed by the latency predictor is infeasible to execute on Pi Zero (last row in Table 1).
Of course, in reality, there is also a great number of IoT devices using more low-end CPUs compared to Pi Zero (ARMV5 or ARMV4), making the latency predictor even harder to be deployed on these devices.
Also in a many-hardware setup considered in this paper, this profiling is executed O ( H ) times.
Most existing layer-wise profile/look-up approaches consider at most mobile systems as targeting platforms [Xu et al., 2020, Yang et al., 2018].
These systems are in general more capable than a great range of IoT devices.
In this paper, we demonstrate the effectiveness of this approach on more low-end systems (Raspberry Pi and Pi Zeros), illustrating this is the more scalable approach for hardware-aware NAS operating on constrained hardware systems.
Evaluating GE and ANL Our results in Table 2 suggest that a correct pooling strategy, GE and
ANL can change the NAS backbone to allow the search space to reach much smaller models and thus provide better accuracy.
In addition, Table 2 also illustrates that 5 × 5 pooling is necessary for higher accuracy.
We hypothesize this is because a relatively large fully-connected layer after the [Table] pooling is required for the network to achieve a good accuracy in this few-learning setup.
We then demonstrate using a case study in our evaluation of how a combination of these techniques can help
H-Meta-NAS: the final searched model can have an up to 14 .
28% accuracy increase on the 5-way 1-shot Mini-ImageNet classification if using these optimization tricks.
Table 2 also showed us that a well-tuned architecture can help MAML models overcome the over-fitting phenomenon in the few-shot learning setup
[Table] [Table]
[Table]
Evaluating H-Meta-NAS searched architectures Table 3 displays the results of H-Meta-NAS on the Omniglot 20-way 1-shot and 5-shot classification tasks.
We match the size of H-Meta-NAS to
MAML and MAML++ for a fair comparison.
H-Meta-NAS outperforms all competing methods apart from the original MAML++.
MAML++ uses a special evaluation strategy, it creates an ensemble of models with best validation-set performance.
MAML++ then picks the best model from the ensemble based on support set loss and report accuracy on the query set.
We then locally replicated MAML++ without this trick, and show that H-Meta-NAS outperforms it by a significant margin ( +1 . 01% on 1-shot and +0 .
11% on 5-shot) with around half of the MACs ( 4 .
95 M compared to 10 .
07 M ).
[Figure] [Figure] [Figure] Table 4 shows the results of running the 5-way 1-shot and 5-shot Mini-ImageNet tasks, similar to the previous results, we match the size of searched networks to MAML, MAML++ and
ALFA+MAML+L2F. Table 4 not only displays results on MAML methods with fixed-architectures, it also shows the performance of searched networks including Auto-Meta Kim et al.
[2018], BASE
Shaw et al.
[2018] and T-NAS Lian et al.
H-Meta-NAS shows interesting results when compared to T-NAS and T-NAS++.
H-Meta-NAS has much higher accuracy ( +3 .
26% in 1-shot and 7 . 94% in 5-shot) and a smaller MAC count, but uses a greater amount of parameters.
T-NAS and T-NAS++ use DARTS cells
Liu et al.
This NAS cell contains a complex routing of computational blocks, making it not suitable for latency critical applications.
We will demonstrate later how this design choice gives a worse on-device latency performance.
H-Meta-NAS for diverse hardware platforms and constraints
In addition to using the model sizes as a constraint for H-Meta-NAS, we use various latency targets on various hardware platforms as the optimization target.
Figure 5, Figure 6 and Figure 7 show how GPU, ASIC and IoT device latency can be used as constraints.
The smaller model sizes of T-NAS do not provide a better run-time on GPU devices (Figure 5), in fact, T-NAS based models have the worst run-time on GPU devices due to the complicated dependency of DARTS cells.
We only compare to MAML and MAML++ when running on Eyeriss due to the limitations of the ScaleSIM simulator Samajdar et al.
In
Appendix J, we provide more latency vs. accuracy plots using various hardware platforms’ latency as constraints and observe the same pareto dominance [Table]
[Table]
platforms and C constraints.
Search time is estimated for a deployment scenario with 500 tasks and 10 hardwareconstraint pairs, estimation details are discussed in Appendix.
[Figure]
A more complex NAS backbone and a comparison to Metric-based MAML Table 5 shows how H-Meta-NAS performs with a more complicated NAS backbone.
In previous experiments, we build the NAS on top of a VGG9 backbone since it is the architecture utilized in the MAML++ algorithm.
To have a fair comparison, we did not manually pick a complex NAS backbone.
However, in this section, we additionally show that H-Meta-NAS can be applied with a more complicated backbone and it shows better final accuracy as expected.
We compare the proposed approach with state-of-the-art Metric-based meta-learning methods Zhang et al.
[2020], Chen et al.
There is a current trend that Metric-based MAML approaches are becoming the mainstream in few-shot learning.
Chen et al.
shows that a simple cosine-similarity measurement can be used as a metric and outperforms standard Optimization-based approaches by a significant margin.
In this experiment, we demonstrate that Metric-based methods suffer from a significant runtime overhead .
Optimization methods, on the other hand, have a significantly less MACs usage.
Although using only a single inference pass (our method does not conduct inference runs on the support set when deployed), H-Meta-NAS shows competitive results with SOTA
Metric-based methods while having a much smaller MACs usage (around 2000 × )
, showing that a well-tuned network architecture can help Optimization methods to close the accuracy gap.
We hope this finding will encourage researchers in this field to look back into Optimization-based MAML .
Search complexity and search time In Table 6, we show a comparison between H-Meta-NAS and various NAS schemes in the many-task many-device setup.
Specifically, we consider a scenario with 500 tasks and 10 different hardware-constraint pairs.
Our results in Table 6 suggest that
H-Meta-NAS is the most efficient search method because of its low search complexity.
Conclusion
In this paper, we show H-Meta-NAS, a NAS method focusing on fast adaption of not only model weights but also model architectures in a many-task many-device few-shot learning setup.
H-MetaNAS outperforms a wide range of MAML baselines on a set of few-shot learning tasks.
We study the effectiveness of H-Meta-NAS on different hardware systems and constraints, and demonstrate its superior performance on real devices using an orders of magnitude shorter search time compared to existing NAS methods. Acknowledgements
Xitong Gao is supported in part by Shenzhen Science and Technology Innovation Commission (No.
JCYJ20190812160003719).
A Details of VGG9 and ResNet12 backbones
Table 7 and Table 8 show the NAS backbones of H-Meta-NAS.
Clearly the ResNet-based
NAS backbone is significantly more complicated.
The kernel size search space is { 1 , 3 , 5 } .
The channel expansion search space is { 0 . 25 , 0 . 5 , 0 .
75 , 1 , 1 . 5 , 2 , 2 . 25 } for the VGG-based NAS backbone but { 0 . 25 , 0 .
75 , 2 } for the ResNet-based backbone.
The reason for the modification in search space is because the GPU RAM limitation does not support an expansion size of 2 .
25 on the ResNet-based backbone.
The activation search space contains { [ ′ relu ′ , ′ elu ′ , ′ selu ′ , ′ sigmoid ′ , ′ relu 6 ′ , ′ leakyrelu ′ } .
[Table]
[Table]
B Tuning the decay process in pre-training strategy
As mentioned in Section 3.3 in the paper, we apply a progressive shrinking strategy to pre-training.
We decay the probability of picking the largest sub-network gradually.
Recall that the architectural sampling process α ∼ p
( A ) will pick the largest network with a probability of p , and randomly pick a sub-network with a probability of 1 − p .
We apply an exponentially decay strategy to p : p = p e + ( p i − p
[Equation]
(5) p e and p i are the end and initial probabilities.
e is the current number of epochs, and e s is the starting epoch of applying this decaying process.
α determines how fast the decay is.
In our experiment, we pick p i = 1 .
and e s = 30 , because the super-net reaches a relatively stable training accuracy at that point.
We then start the decaying process, and evaluate different values of α in Table 9.
The averaged accuracy is averaged across 100 randomly picked sub-networks on the T val tasks.
Based on these results, we picked α = 5 for our later experiments.
C Latency predictor: an explanation
Figure 8 explains how the latency predictor work.
The predictor builds on N profiled networks, and this style of sub-network profiling is infeasible on IoT devices. [Figure] [Table] [Figure]
[Table]
[Figure]
D Adaption algorithm and the hyper-parameter choices
Algorithm 1 details the adaption algorithm.
In the Mutate function, each architecture is ranked with the averaged loss across all sampled tasks, and 10% of the architectures with the lowest loss values are then used to perform a classic genetic algorithm mutation Whitley [1994].
The mutation will allow the top-performing architectures to have two randomly picked architectural choices being modified to another choice that is not the original one.
The mutation function considers the original pool of architectures ( A ) and their averaged loss values ( L a ).
The cost of each architecture can be computed by the pre-build hardware-specific hash-table H t ( A ) .
We then only mutate the subset in A that their hardware cost has satisfied the constraints {A|A ∈
A ∧ H t ( A ) ≤ C } .
The mutation is to randomly pick two options in the entire architectural space and change them to other choices that are different from the original.
Algorithm {+
+}
The adaption algorithm
Input: M , P , C , H t
A =
Init ( P ) ▷ Initialise a set of architectures with a size of P for i = [-
-]0 to M − 1 do
L a = ∅ T s ∼ p ( T val )
▷ Obtain a subset from the validation task set for A ∈ A do
L t = ∅ for T ∈ T s do l =
L
( T , A ) ▷ Compute loss
L t =
L t (cid:83) { l } end for
L a =
L a (cid:83) { mean ( L t ) } ▷ Collect averaged loss values across all tasks end for A = Mutate ( A , L a , H t , C )
▷ Mutate the architectures based on hardware constraints end for
[Table]
E Hardware devices
To demonstrate the hardware-awareness of our NAS algorithm, Table 10 summarises the hardware platform we used for our experiments.
We deployed our networks to Raspberry Pi models using
TVM Chen et al.
F H-Meta-NAS search configurations and hardware simulation
We mostly follow the experiment setup in MAML++ Antoniou et al.
In the pre-training stage, we train for 100 epochs, each epoch consists of 500 iterations.
We also pick 600 tasks to be validation tasks.
In the adaption stage, we randomly sample from the validation set, and pick 16 tasks to build a data slice for the architectures to traverse.
In the final re-training stage of a searched architecture, we follow the strategy used in MAML++ Antoniou et al.
We then introduce the detailed special configurations for the datasets:
• Omniglot: We randomly split 1200 characters for training, and the rest is used for testing.
The images are augmented with randomised rotation of multiples of 90 degrees.
• Mini-ImageNet: All images are down-sampled to 84 × 84 .
We use the ScaleSim framework Samajdar et al.
[2018] for simulating the Eyeriss Chen et al.
[2016] accelerator.
ScaleSim is an open-source cycle-accurate CNN simulator.
The simulator has certain limitations with respect to the DRAM simulation, it could be advanced with an external
DRAM simulator but will cause a large run-time.
So we kept the original setup and the DRAM simulation would report a read/write bandwidth requirements.
For simplicity, we assume these
DRAM requirements are met.
In addition, it is a well-known fact that cycle-accurate simulators are slow to execute.
Due to this reason, we only launched the MAML and MAML++ networks in the
ScaleSim simulator.
G T-NAS baseline results
We notice the model sizes of some baseline models ( e.g. MAML and MAML++) reported in the original TNAS paper Lian et al.
[2019] are different from our results in Table 3.
We calculated the model sizes of these baselines using their official open-sourced implementations.
T-NAS did not provide an implementation of their mentioned baselines in their official repository, so we cannot replicate their model size numbers.
We have contacted the T-NAS authors regarding this issue.
H Additional results on FC
FC100 is introduced by Oreshkin et al.
and has 100 different classes from the CIFAR100 dataset
Krizhevsky [2009].
In Table 11, we further demonstrate the effectiveness of the proposed H-Meta-NAS on the FC dataset.
T-NAS did not report their model sizes on this task, and our results suggest that H-MetaNAS achieves the best accuracy on both the 1-shot and 5-shot setups.
[Table]
I Search time estimation
Due to the limited computing facilities available, we estimate the search time of DARTS Liu et al.
[2018], Once-for-all Cai et al.
[2019] and T-NAS Lian et al.
[2019] in a multi-task multi-device setup.
We take the search time reported in the original publications and multiply them by the appropriate scaling factors.
For DARTs, we take the search time (4 GPU days = 96 GPU hours) and multiply it by H × T = 5000 .
We additionally assume a linear scaling relationship between search time and input image sizes, so we multiply the total search time by 84 × 84 32 × 32 , this gives us in total a search time of around 10 6 .
We perform the same estimation for Once-for-all Cai et al.
[2019] and T-NAS Lian et al.
J Latency-aware optimisation on more hardware platforms
In addition to using the model sizes as a constraint for H-Meta-NAS, we use various latency targets on various hardware platforms as the optimisation target.
Figure 9 shows how GPU latencies can be used as constraints.
T-NAS and T-NAS++ show a better performance on the size-accuracy plot in Figure 9.
The smaller model sizes of T-NAS do not provide a better run-time on GPU devices (Figure 5), in fact,
T-NAS based models have the worst run-time on GPU devices due to the complicated dependency of DARTS cells.
We only compare to MAML and MAML++ when running on Eyeriss due to the limitations of the ScaleSIM simulator Samajdar et al.
[Figure] [Figure]
[Figure] [Figure]
K Searched Pooling mechanism
In this section
[Figure]
we discuss the effect of automatically search for the pooling strategy in the proposed algorithm.
The H-Meta-NAS algorithm used a fixed pooling strategy that is manually tuned, this is the same setup in other NAS algorithms ( e
[Figure]
g. T-NAS Lian et al.
We add an additional search space of pooling ( 1 × 1 , 3 × 3 , 5 × 5 and 7 × 7 ).
Table 12 shows the results of searched pooling.
Unsurprisingly, the network picked 5 × 5 pooling for both datasets, which is the same as pooling strategy obtained from our manual tunning.
The automatic search for the pooling strategy should be more beneficial if our downstream tasks are of different image resolutions.
Unfortunately, most of the standard benchmarks today in few-shot learning do not consider this scenario.
If given the resolutions are the same for the meta-tasks and downstream tasks, an automatic search for the pooling design might not be necessary, since the number of pooling options are normally small.
[Table]
L License of the assets
In our work
[Table]
we utilized the following datasets/library/code listed in Table
[Table] [Table]
The vast majority of the work was implemented ourselves and will be released under the permissive
MIT license , which allows future researchers to build on the work unconstrained (only requiring preservation of the license file).
All dependencies of my library are similarly released under OSI 2approved licenses, allowing them all to be easily compiled and installed
[Table]
M Computation resources
Apart from the hardware devices used for profiling in the main paper.
All experiments complete in < 30 GPU-days on a four NVIDIA GeForce GTX 1080 Ti system with an Intel(R) Xeon(R) CPU
E5-2620 v4 at 2.10GHz.