Rapid Model Architecture Adaption for Meta-Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Network Architecture Search (NAS) methods have recently gathered much attention.
They design networks with better performance and use a much shorter searchtime compared to traditional manual tuning.
Despite their efﬁciency in modeldeployments, most NAS algorithms target a single task on a ﬁxed hardware system.
However, real-life few-shot learning environments often cover a great number oftasks ( T ) and deployments on a wide variety of hardware platforms ( H ).
The combinatorial search complexity T × H creates a fundamental search efﬁciency challenge if one naively applies existing NAS methods to these scenarios.
To overcome this issue, we show, for the ﬁrst time, how to rapidly adapt model architectures to new tasks in a many-task many-hardware few-shot learning setup byintegrating Model Agnostic Meta Learning (MAML) into the NAS ﬂow.
The proposed NAS method (H-Meta-NAS) is hardware-aware and performs optimisationin the MAML framework.
H-Meta-NAS shows a Pareto dominance compared to avariety of NAS and manual baselines in popular few-shot learning benchmarks withvarious hardware platforms and constraints.
In particular, on the 5-way 1-shot MiniImageNet classiﬁcation task, the proposed method outperforms the best manualbaseline by a large margin ( 5 . 21% in accuracy) using 60% less computation.
Introduction
Existing Network Architecture Search (NAS) methods show promising performance on image [Zophand Le, 2016, Liu et al., 2018], language [Guo et al., 2019, So et al., 2019] and graph data [Zhaoet al., 2020].
The automation not only reduces the human effort required for architecture tuning butalso produces architectures with state-of-the-art performance in domains like image classiﬁcation[Zoph and Le, 2016] and language modeling [So et al., 2019].
Most NAS methods today focus on asingle task with a ﬁxed hardware system, yet real-life model deployments covering multiple tasksand various hardware platforms will signiﬁcantly prolong this process.
As illustrated in Figure 1, acommon design is to architecture and train for the different task( T )-hardware
( H )pairs with different constraints ( C ).
The architectural engineering phase can be accomplished whethermanually or by using an established NAS procedure.
The challenge is designing an efﬁcient methodto overcome the quickly scaling O ( THC ) search complexity described in Figure 1.
Few-shot learning systems follow exactly this many-task many-device setup, when consideringdeployments on different user devices on key applications such as facial [Guo et al., 2020] andspeech recognition [Hsu et al., 2020].
A task in few-shot learning normally takes an N -way K -shotformulation, where it contains N classes with K support samples and Q query samples in eachclass.
Model-Agnostic Meta-Learning (MAML),
incorporating the idea of learning to learn, buildsa meta-model using a great number of training tasks, and then adapts the meta-model to unseentest tasks using only a very small number of gradient updates [Finn et al., 2017].
MAML then
Task+
Hardware 1 Architectural Engineering & Train from scratch or Re-train
Task T +
Hardware H ...
+
Constraint+
Constraint C
ASIC becomes a powerful and elegant approach for few-shot learning – its ability to quickly adapt tonew tasks can potentially shrink the O ( THC ) complexity illustrated in Figure 1 to O ( HC ) .
Inthe meantime, hardware-aware NAS methods [Cai et al., 2019, 2018, Xu et al., 2020], e.g. thetrain-once-for-all technique [Cai et al., 2019], support deployments of searched models to ﬁt differenthardware platforms with various latency constraints.
These hardware-aware NAS techniques furtherreduce the search complexity from O ( THC ) to O ( T )
[Cai et al., 2018].
In this paper, we propose a novel Hardware-aware Meta Network Architecture Search (H-Meta-NAS).
Integration of the MAML framework into hardware-aware NAS theoretically reduces the searchcomplexity from O ( THC ) to O (1) , allowing for a rapid adaption of model architectures to unseentasks on new hardware systems.
However, we identiﬁed the following challenges in this integration:
• Classic NAS search space contains many over-parameterized sub-models, this makes it hardto tackle the over-ﬁtting phenomenon in few-shot learning.• Hardware-aware NAS proﬁles latency for sub-networks on each task-hardware pair, this
proﬁling can be prolonged signiﬁcantly with a great number of tasks and, more importantly,
if the targeting device has scarce computation resources.
To tackle these challenges, we then propose to use Global Expansion (GE) and Adaptive Numberof Layers (ANL) to allow a drastic change in model capabilities for tasks with varying difﬁculties.
Our experiments later demonstrate that such changes alleviate over-ﬁtting in few-shot learning andimprove the accuracy signiﬁcantly.
We also present a novel layer-wise proﬁling strategy to allow thereuse of proﬁling information across different tasks.
In this paper we make the following contributions:
• We propose a novel Hardware-aware Network Architecture Search for Meta learning (H-
Meta-NAS).
H-Meta-NAS quickly adapts meta-architectures to new tasks with hardware-
awareness and can be conditioned with various device-speciﬁc latency constraints.
The
proposed NAS reduces search complexity from O ( THC ) to O (1) in a realistic many-task
many-device few-shot learning setup.
We extensively evaluate H-Meta-NAS on various
hardware platforms (GPU, CPU, mCPU, IoT, ASIC accelerator) and constraints (latencyand model size), our latency-accuracy performance generally outperforms various MAML
baselines and other NAS competitors.
• We propose a task-agnostic layer-wise proﬁling strategy that reduces the proﬁling run-
time from around 10 5 hours to 1.2 hours when targeting hardware with limited capabilities
( e.g. IoT devices).
• MAML models are prone to over-ﬁtting in the few-shot learning setup Antoniou et al.
We show several design options for the NAS algorithm, namely Global Expansion
and Adaptively Number of Layers respectively.
These methods help the NAS to overcome
the over-ﬁtting problem by optimizing in the architectural design space.
• We reveal that popular Metric-based MAML methods are not latency-friendly .
Our
empirical results suggest that Optimization-based MAML method with well-tuned architec-
tures can achieve comparable accuracy with signiﬁcantly less latency overhead ( ≈ 2000 ×on Mini-ImageNet 5-way 1-shot classiﬁcation).
Background2.1 Few-shot learning in the MAML framework
Inspired by human’s ability to learn from only a few tasks and generalize the knowledge to unseenproblems, a meta learner is trained over a distribution of tasks with the hope of generalizing itslearned knowledge to new tasks Finn et al.
[Equation]
Equation (1) captures the optimization objective of meta-learning, where optimal parameters areobtained through optimizing on a set of meta-training tasks.
Current approaches of using metalearning to solve few-shot learning problems can be roughly categorized into three types:
Memorybased, Metric-based and Optimization-based.
Memory-based method utilizes a memory-augmented neural network [Munkhdalai and Yu, 2017,
Gidaris and Komodakis, 2018] to memorize meta-knowledge for a fast adaption to new tasks.
Metricbased methods aim to meta-learn a high-dimensional feature representation of samples, and then applycertain metrics to distinguish them.
For instance, Meta-Baseline utilizes the cosine nearest-centroidmetric
[Chen et al., 2020] and DeepEMD applies the Wasserstein distance [Zhang et al., 2020].
Optimization-based method, on the other hand, focuses on learning a good parameter initialization(also known as meta-parameters or meta-weights ) from a great number of training tasks, such thatthese meta-parameters adapt to new few-shot tasks within a few gradient updates.
The most wellestablished Optimization-based method is Model-Agnostic Meta-Learning (MAML)
[Finn et al.
MAML is a powerful yet simple method to tackle the few-shot learning problem, since itsadaption relies solely on gradient updates.
Antoniou et al.
later demonstrate MAML++, a series ofmodiﬁcations that improved MAML’s performance and stability.
Baik et al.
introduce an additionalnetwork for generating adaptive parameters for the inner-loop optimization.
Despite the rise in popularity of the meta-learning framework applied to few-shot learning, littleattention has been paid to the runtime efﬁciency of these approaches.
In this work, we reveal that thecurrent mainstream Metic-based methods all suffer from a severe latency overhead , since Metricbased approaches rely on metric comparisons and have to use multiple inference runs to generatethese metrics (at least two) for a single image classiﬁcation.
With the help of H-Meta-NAS, we showhow Optimization-based methods can achieve a similar level of accuracy but using signiﬁcantly lesscomputation at inference time.
2.2 Network architecture search
Architecture engineering is a tedious and complex process requiring a lot of effort from humanexperts.
Network Architecture Search (NAS) focuses on reducing the amount of manual tuningin this design space.
Early NAS methods use evolutionary algorithms and reinforcement learningto traverse the search space [Zoph and Le, 2016, Real et al., 2017].
These early methods requirescoring architectures trained to a certain convergence and thus use a huge number of GPU hours.
Twomajor directions of NAS methods, Gradient-based and Evolution-based methods, are then exploredin parallel in order to make the search cost more affordable.
Gradient-based NAS methods use
Stochastic Gradient Descent (SGD) to optimize a set of probabilistic priors that are associated witharchitectural choices [Liu et al., 2018, Casale et al., 2019].
Although these probabilistic priors canbe made latency-aware [Wu et al., 2019, Xu et al., 2020], it is challenging to make them follow ahard latency constraint.
Evolution-based NAS, on the other hand, operates on top of a pre-trainedsuper-net and use evolutionary algorithms or reinforcement learning to pick best-suited sub-networks[Cai et al., 2018, 2019], making it easier to be constrained by certain hardware metrics.
For instance,
Once-for-all (OFA) is an Evolution-based NAS method and its searched networks are not onlyoptimized for a speciﬁc hardware target but also constrained by a pre-deﬁned latency budget [Caiet al., 2019].
Our proposed H-Meta-NAS shares certain similarities to Once-for-all, since this methodoffers a chance to reduce the hardware search complexity from O ( HC ) to O (1) .
Several NAS methods are proposed under the MAML framework [Kim et al., 2018, Shaw et al., 2018,
Lian et al., 2019], these methods successfully reduce the search complexity from O ( T ) to O (1) .
However, some of these methods do not show signiﬁcant performance improvements comparedto carefully designed MAML methods ( e.g. MAML++)
[Kim et al., 2018, Shaw et al., 2018].
In the meantime, some of these MAML-based NAS methods follow the Gradient-based approach andoperate on complicated cell-based structures [Lian et al., 2019].
We illustrate later how cell-based
NAS causes an undesirable effect on latency, and also meets fundamental scalability challenges whentrying to deploy in a many-task many-device few-shot learning setup.
Method
Problem formulation In the MAML setup, we consider a set of tasks T and each task T i ∈ Tcontains a support set D si and a query set D qi .
The support set is used for task-level learning whilethe query set is in charge of evaluating the meta-model.
All tasks are divided into three sets, namelymeta-training ( T train ), meta-validation ( T val ) and meta-testing ( T test ) sets.
Equation (2) formally states the objective of the pre-training stage illustrated in Figure 3a.
Theobjective of this process is to optimize the parameters θ of the super-net for various sub-networkssampled from the architecture set A .
This will ensure the proposed H-Meta-NAS to have both themeta-parameters and meta-architectures ready for the adaption to new tasks.
[Equation]
Equation (3) describes how H-Meta-NAS adapts network architectures to a particular task T witha given hardware constraint C h .
In practice, using the support set data D si from a target task T i ,we apply a genetic algorithm for ﬁnding the optimal architectures α ∗ .
We discuss further how thisprocess in detail in later sections.
[Equation]
Architecture search space H-Meta-NAS considers a search space composed of different kernelsizes, the number of channels, and activation types.
We mostly consider a VGG9-based NASbackbone, that is a 5-layer CNN model with the last layer being a fully connected layer.
We chosethis NAS backbone because both MAML Finn et al.
[2017] and MAML++ Antoniou et al.
[2018]used a VGG9 model architecture.
The details of this backbone are in Appendix A.
We allow kernel sizes to be picked from { 1 , 3 , 5 } , channels to be expanded with a set of scaling factors{ 0 . 25 , 0 . 5 , 0 .
75 , 1 , 1 . 5 , 2 , 2 . 25 } and also six different activation functions (details in Appendix).
For asingle layer, there are 3 × 7 × 6 = 126 search options.
H-Meta-NAS also contains an Adaptive Numberof Layers strategy, the network is allowed to use a subset of the total layers in the supernet withmaximum usage of 4 layers.
The whole VGG9-based backbone then gives us in total 126 4 × 4 ≈ 10 9possible neural network architectures.
In addition, to demonstrate the ability of H-Meta-NAS on a more complex NAS backbone.
We alsostudied an alternative ResNet12-based NAS backbone, that has approximately 2 × 10 24 possiblesub-networks.
Super-net meta-training strategy As illustrated by prior work Cai et al. [2019], progressivelyshrinking the super-net during meta-training can reduce the interference between sub-networks. Weobserve the same phenomenon and then use a similar progressive shrinking strategy in H-Meta-NAS,the sampling process α ∼ p
( A ) will pick the largest network with a probability of p , and randomlypick other sub-networks with a probability of 1 − p .
We apply an exponential decay strategy to p :
[Equation]
(4) p e and p i are the end and initial probabilities.
e is the current number of epochs, and e s and e m arethe starting and end epochs of applying this decaying process.
α determines how fast the decay is.
A graphical illustration is shown in Figure 3a.
{+In our experiment, we pick p i = 1 . 
+}
and e s = 30 ,because the super-net reaches a relatively stable training accuracy at that point.
We then start the decaying process, and the value α = 5 is
[Figure]
[Figure]
[Table]
Layer-wise proﬁling Hardware-aware NAS needs the run-time of sub-networks on the targetinghardware to guide the search process Cai et al.
[2019], Xu et al.
However, the proﬁling stagecan be time-consuming if given a low-end hardware as the proﬁling target and the search space islarge.
For instance, running a single network inference of VGG9 on the Raspberry Pi Zero witha 1GHz single-core ARMv6 CPU takes around 2 .
365 seconds to ﬁnish.
If we assume this is theaverage time needed for proﬁling a sub-network, given that the entire search space includes around10 9 sub-networks, a naive traverse will take a formidable amount of time which is approximately6 × 10 5 hours.
More importantly, the amount of proﬁling time scales with the number of hardwaredevices ( O ( H ) ).
Existing hardware-aware NAS schemes build predictive methods to estimate therun-time of sub-networks Cai et al.
[2019], Xu et al.
[2020] and have a relatively signiﬁcant error.
Anillustration of a proﬁler is shown in Figure 2, and the latency predictor’s illustration is in Figure 8.
We re-implemented the latency predictor in OFA Cai et al.
[2019] as a baseline and compare it toour layer-wise proﬁling. We pick 16K training samples and 10K validation samples to train and testthe latency predictor, which is the same setup used in OFA. We use another 10K testing samples toevaluate the performance of OFA-based latency predictor against our layer-wise proﬁling on differenthardware systems in terms of MSE (measuring the latency estimation quality) and Time (measuringthe efﬁciency). As illustrated in Table 1, layer-wise proﬁling saves not only time but also has a smaller MSE errorcompared to a predictor-based strategy that is very popular in today’s evolutionary-based NASframeworks [Cai et al., 2019, 2018].
In addition, layer-wise proﬁling shows orders of magnitudebetter run-time when targeting hardware devices with scarce computational resources.
If we consideran IoT class device as a target (i.e the Raspberry Pi Zero), it requires an unreasonably large amountof time to generate training samples for latency predictors, making them an infeasible approach inreal life.
For instance, the total time consumed by the latency predictor is infeasible to execute on Pi
Zero (last row in Table 1).
Of course, in reality, there is also a great number of IoT devices usingmore low-end CPUs compared to Pi Zero (ARMV5 or ARMV4), making the latency predictor even
[Figure]
u
[Figure] y [Figure] harder to be deployed on these devices.
Also in a many-hardware setup considered in this paper, thisproﬁling is executed O ( H ) times. Most existing layer-wise proﬁle/look-up approaches consider at most mobile systems as targetingplatforms [Xu et al., 2020, Yang et al., 2018]. These systems are in general more capable than agreat range of IoT devices. In this paper, we demonstrate the effectiveness of this approach on morelow-end systems (Raspberry Pi and Pi Zeros), illustrating this is the more scalable approach forhardware-aware NAS operating on constrained hardware systems.
The adaption strategy uses a genetic algorithm Whitley [1994] to pick thebest-suited sub-network with respective to a given hardware constraint, the full algorithm is detailedin Appendix D.
In general, the adaption algorithm randomly samples a set of tasks from T val , anduses the averaged loss value and satisfaction to the hardware constraints as indicators for the geneticalgorithm.
The genetic algorithm has a pool size P and number of iterations M .
We then run a hyper-parameter analysis in Figure 3b to determine the values of P and M .
The fulladaption algorithm makes use of these hyper-parameters is in our Appendix.
The horizontal axisshows the number of iterations and the vertical axis shows the averaged accuracy on the sampledtasks for all architectures in the pool.
Figure 3b shows that the accuracy convergence is reached afteraround 150 iterations, and running for additional iterations only provides marginal accuracy gains.
For this reason, we picked the number of iterations to be 200 for a balance between accuracy andrun-time.
In the meantime, we notice in general a higher pool size will give better-adapted accuracy.
However, this does not mean the ﬁnal searched accuracy is affected to the same degree.
The ﬁnalre-trained accuracy of searched architectures show an accuracy gap of 0 .
21% between P = 100 and
P = 200 and 0 .
32% between P = 100 and P = 500 .
An increase in pool size can prolong therun-time signiﬁcantly, we thus picked a pool size of 100 since it offers the best balance betweenaccuracy and run-time.
NAS backbone design, Global Expansion (GE) and Adaptive Number of Layers (ANL) Oneparticular problem in few-shot learning is that models are prone to over-ﬁtting.
This is because onlya small number of training samples are available for each task and the network normally iterate onthese samples many times Antoniou et al.
We would like to explore the architectural space tohelp models overcome over-ﬁtting and conduct a case study for different design options availablefor the backbone network.
We identify the following key changes to the NAS backbone to help themodels to have high accuracy in few-shot learning:
• n × n pooling:
Pooling that applied to the ﬁnal convolutional operation, n × n indicates the
height and width of feature maps after pooling.
• Global Expansion (GE):
Allowing the NAS to globally expand or shrink the number ofchannels of all layers.
[Figure]
[Table]
• Adaptive Number of Layers (ANL): Allowing the NAS to use an arbitrary number of layers,
[Table]
the network then is able to early stop using only a fewer number of layers.
[Table]
Figure 4 further illustrates how GE and ANL can allow a much smaller model compared to existing
NAS backbones.
Our results in Table 2 suggest that a correct pooling strategy, GE and ANL can change the NASbackbone to allow the search space to reach much smaller models and thus provide better accuracy.
Inaddition, Table 2 also illustrates that 5 × 5 pooling is necessary for higher accuracy.
We hypothesizethis is because a relatively large fully-connected layer after the pooling is required for the networkto achieve a good accuracy in this few-learning setup. We then demonstrate using a case study inour evaluation of how a combination of these techniques can help H-Meta-NAS: the ﬁnal searchedmodel can have an up to 14 .
28% accuracy increase on the 5-way 1-shot Mini-ImageNet classiﬁcationif using these optimization tricks.
Table 2 also showed us that a well-tuned architecture can help
[-In our experiment, we pick p i = 1 . 
 and e s = 30 ,because-]MAML models overcome the over-ﬁtting phenomenon in the few-shot learning setup.
Evaluation
We evaluate H-Meta-NAS on a range of popular few-shot learning benchmarks.
For each dataset,we search for the meta-architecture and meta-parameters.
We then adapt the meta-architecture withrespect to a target hardware-constraint pair.
In the evaluation stage, we then re-train the obtainedhardware-aware task-speciﬁc architecture to convergence and report the ﬁnal accuracy.
We considerthree popular datasets in the few-shot learning community: Omniglot, Mini-ImageNet and Few-shot
CIFAR100.
We use the PytorchMeta framework to handle the datasets [Deleu et al., 2019].
Omniglot is a handwritten digits recognition task, containing 1623 samples [Lake et al., 2015].
Weuse the meta train/validation/test splits originally used Vinyals et al.
Vinyals et al.
Thesesplits are over 1028/172/423 classes (characters).
Mini-ImageNet is ﬁrst introduced by Vinyals et al. .
This dataset contains images of 100 differentclasses from the ILSVRC-12 dataset [Deng et al., 2009], the splits are taken from Ravi et al.
[Raviand Larochelle, 2016].
Table 10 in our appendix details the systems and representative devices considered.
We use the
ScaleSIM cycle-accurate simulator Samajdar et al.
[2018] for the Eyeriss Chen et al.
[2016] accelerator.
This simulation and more datasets and search conﬁgurations information are in Appendix F.
[Figure]
[Figure]
[Figure]
[Figure]
[Table]
[Figure]
[Table] [Table]
Evaluating H-Meta-NAS searched architectures Table 3 displays the results of H-Meta-NAS onthe Omniglot 20-way 1-shot and 5-shot classiﬁcation tasks.
We match the size of H-Meta-NAS to
MAML and MAML++ for a fair comparison.
H-Meta-NAS outperforms all competing methods apartfrom the original MAML++.
MAML++ uses a special evaluation strategy, it creates an ensemble ofmodels with best validation-set performance.
MAML++ then picks the best model from the ensemblebased on support set loss and report accuracy on the query set.
We then locally replicated MAML++without this trick, and show that H-Meta-NAS outperforms it by a signiﬁcant margin ( +1 .
01% on1-shot and +0 .
11% on 5-shot) with around half of the MACs ( 4 .
95 M compared to 10 .
07 M ).
Table 4 shows the results of running the 5-way 1-shot and 5-shot Mini-ImageNet tasks, similar to the previous results, we match the size of searched networks to MAML, MAML++ and
ALFA+MAML+L2F. Table 4 not only displays results on MAML methods with ﬁxed-architectures,it also shows the performance of searched networks including Auto-Meta Kim et al.
[2018], BASE
Shaw et al.
[2018] and T-NAS Lian et al.
H-Meta-NAS shows interesting results whencompared to T-NAS and T-NAS++.
H-Meta-NAS has much higher accuracy ( +3 .
26% in 1-shotand 7 .
94% in 5-shot) and a smaller MAC count, but uses a greater amount of parameters.
T-NASand T-NAS++ use DARTS cells
Liu et al.
This NAS cell contains a complex routing ofcomputational blocks, making it not suitable for latency critical applications.
We will demonstratelater how this design choice gives a worse on-device latency performance.
H-Meta-NAS for diverse hardware platforms and constraints
In addition to using the modelsizes as a constraint for H-Meta-NAS, we use various latency targets on various hardware platformsas the optimization target.
Figure 5, Figure 6 and Figure 7 show how GPU, ASIC and IoT devicelatency can be used as constraints.
The smaller model sizes of T-NAS do not provide a better run-timeon GPU devices (Figure 5), in fact, T-NAS based models have the worst run-time on GPU devicesdue to the complicated dependency of DARTS cells.
We only compare to MAML and MAML++when running on Eyeriss due to the limitations of the ScaleSIM simulator Samajdar et al.
In
Appendix J, we provide more latency vs. accuracy plots using various hardware platforms’ latency asconstraints and observe the same pareto dominance.
[Table]
C constraints.
Search time is estimated for a deployment scenario with 500 tasks and 10 hardwareconstraint pairs, estimation details are discussed in Appendix.
[Figure]
A more complex NAS backbone and a comparison to Metric-based MAML Table 5 showshow H-Meta-NAS performs with a more complicated NAS backbone.
In previous experiments, webuild the NAS on top of a VGG9 backbone since it is the architecture utilized in the MAML++algorithm.
To have a fair comparison, we did not manually pick a complex NAS backbone.
However,in this section, we additionally show that H-Meta-NAS can be applied with a more complicatedbackbone and it shows better ﬁnal accuracy as expected.
We compare the proposed approach with state-of-the-art Metric-based meta-learning methods Zhanget al.
[2020], Chen et al.
There is a current trend that Metric-based MAML approaches arebecoming the mainstream in few-shot learning.
Chen et al.
shows that a simple cosine-similaritymeasurement can be used as a metric and outperforms standard Optimization-based approaches bya signiﬁcant margin.
In this experiment, we demonstrate that Metric-based methods suffer from asigniﬁcant runtime overhead .
Optimization methods, on the other hand, have a signiﬁcantly less MACs usage.
Although using only a single inference pass (our method does not conduct inferenceruns on the support set when deployed), H-Meta-NAS shows competitive results with SOTA Metricbased methods while having a much smaller MACs usage (around 2000 × ), showing that a well-tunednetwork architecture can help Optimization methods to close the accuracy gap.
Search complexity and search time In Table 6, we show a comparison between H-Meta-NAS andvarious NAS schemes in the many-task many-device setup.
Speciﬁcally, we consider a scenariowith 500 tasks and 10 different hardware-constraint pairs.
Our results in Table 6 suggest that
H-Meta-NAS is the most efﬁcient search method because of its low search complexity.
Conclusion
In this paper, we show H-Meta-NAS, a NAS method focusing on fast adaption of not only modelweights but also model architectures in a many-task many-device few-shot learning setup.
H-MetaNAS outperforms a wide range of MAML baselines on a set of few-shot learning tasks.
We study theeffectiveness of H-Meta-NAS on different hardware systems and constraints, and demonstrate itssuperior performance on real devices using an orders of magnitude shorter search time compared toexisting NAS methods.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A or research subjects
(a) Did
you
include the full text
of instructions given to participants
and screenshots
, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Details of VGG9 and ResNet12 backbones
Table 7 and Table 8 show the NAS backbones of H-Meta-NAS.
Clearly the ResNet-based
NAS backbone is signiﬁcantly more complicated.
The kernel size search space is { 1 , 3 , 5 } .
The channel expansion search space is { 0 . 25 , 0 . 5 , 0 .
75 , 1 , 1 . 5 , 2 , 2 . 25 } for the VGG-based NASbackbone but { 0 . 25 , 0 .
75 , 2 } for the ResNet-based backbone.
The reason for themodiﬁcation in search space is because the GPU RAM limitation does not support an expansion size of 2 .
25 on the ResNet-based backbone.
The activation search space contains{ [ (cid:48) relu (cid:48) , (cid:48) elu (cid:48) , (cid:48) selu (cid:48) , (cid:48) sigmoid (cid:48) , (cid:48)
relu 6 (cid:48) , (cid:48) leakyrelu (cid:48) } .
[Table]
[Table]
B Tuning the decay process in pre-training strategy
As mentioned in Section 3.3 in the paper, we apply a progressive shrinking strategy to pre-training.
We decay the probability of picking the largest sub-network gradually.
Recall that the architecturalsampling process α ∼ p
( A ) will pick the largest network with a probability of p , and randomly picka sub-network with a probability of 1 − p .
We apply an exponentially decay strategy to p :
[Equation]
p e and p i are the end and initial probabilities.
e is the current number of epochs, and e s is the startingepoch of applying this decaying process.
α determines how fast the decay is.
In our experiment, wepick p i = 1 .
and e s = 30 , because the super-net reaches a relatively stable training accuracy at thatpoint.
We then start the decaying process, and evaluate different values of α in Table 9.
The averagedaccuracy is averaged across 100 randomly picked sub-networks on the T val tasks.
Based on theseresults, we picked α = 5 for our later experiments.
C Latency predictor: an explanation
Figure 8 explains how the latency predictor work.
The predictor builds on N proﬁled networks, andthis style of sub-network proﬁling is infeasible on IoT devices.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
D Adaption algorithm and the hyper-parameter choices
Algorithm 1 details the adaption algorithm.
In the Mutate function, each architecture is rankedwith the averaged loss across all sampled tasks, and 10% of the architectures with the lowest lossvalues are then used to perform a classic genetic algorithm mutation Whitley [1994].
The mutationwill allow the top-performing architectures to have two randomly picked architectural choices beingmodiﬁed to another choice that is not the original one.
The mutation function considers the originalpool of architectures ( A ) and their averaged loss values ( L a ).
The cost of each architecture can becomputed by the pre-build hardware-speciﬁc hash-table H t ( A ) .
We then only mutate the subset in Athat their hardware cost has satisﬁed the constraints {A|A ∈
A ∧ H t ( A ) ≤ C } .
The mutation is torandomly pick two options in the entire architectural space and change them to other choices that aredifferent from the original.
Algorithm
Input: M , P , C , H t
A =
Init ( P ) (cid:46) Initialise a set of
[Table]
E Hardware devices
To demonstrate the hardware-awareness of our NAS algorithm, Table 10 summarises the hardwareplatform we used for our experiments.
F H-Meta-NAS search conﬁgurations and hardware simulation
We mostly follow the experiment setup in MAML++ Antoniou et al.
In the pre-training stage,we train for 100 epochs, each epoch consists of 500 iterations.
We also pick 600 tasks to be validationtasks.
In the adaption stage, we randomly sample from the validation set, and pick 16 tasks to build adata slice for the architectures to traverse.
In the ﬁnal re-training stage of a searched architecture, wefollow the strategy used in MAML++ Antoniou et al.
We then introduce the detailed specialconﬁgurations for the datasets:
• Omniglot: We randomly split 1200 characters for training, and the rest is used for testing.
The images are augmented with randomised rotation of multiples of 90 degrees.• Mini-ImageNet: All images are down-sampled to 84 × 84 .
We use the ScaleSim framework Samajdar et al.
[2018] for simulating the Eyeriss Chen et al.[2016] accelerator.
ScaleSim is an open-source cycle-accurate CNN simulator.
The simulator hascertain limitations with respect to the DRAM simulation, it could be advanced with an external
DRAM simulator but will cause a large run-time.
So we kept the original setup and the DRAMsimulation would report a read/write bandwidth requirements.
For simplicity, we assume these
DRAM requirements are met.
In addition, it is a well-known fact that cycle-accurate simulators areslow to execute.
Due to this reason, we only launched the MAML and MAML++ networks in the
ScaleSim simulator.
G T-NAS baseline results
We notice the model sizes of some baseline models ( e.g. MAML and MAML++) reported in theoriginal TNAS paper
Lian et al.
[2019] are different from our results in Table 3.
We calculated themodel sizes of these baselines using their ofﬁcial open-sourced implementations.
T-NAS did notprovide an implementation of their mentioned baselines in their ofﬁcial repository, so we cannotreplicate their model size numbers.
We have contacted the T-NAS authors regarding this issue.
H Additional results on FC100
FC100 is introduced by Oreshkin et al.
and has 100 different classes from the CIFAR100 dataset
Krizhevsky [2009].
In Table 11, we further demonstrate the effectiveness of the proposed H-Meta-NAS on the FC100dataset.
T-NAS did not report their model sizes on this task, and our results suggest that H-MetaNAS achieves the best accuracy on both the 1-shot and 5-shot setups.
I Search time estimation
Due to the limited computing facilities available, we estimate the search time of DARTS Liu et al.[2018], Once-for-all Cai et al.
[2019] and T-NAS Lian et al.
[2019] in a multi-task multi-device setup. [Table]
We take the search time reported in the original publications and multiply them by the appropriatescaling factors.
For DARTs, we take the search time (4 GPU days = 96 GPU hours) and multiply it by H × T = 5000 .
We additionally assume a linear scaling relationship between search time andinput image sizes, so we multiply the total search time by 84 × 84 32 × 32 , this gives us in total a search time of around 10 6 .
We perform the same estimation for Once-for-all Cai et al.
[2019] and T-NAS Lianet al.
J Latency-aware optimisation on more hardware platforms
In addition to using the model sizes as a constraint for H-Meta-NAS, we use various latency targets onvarious hardware platforms as the optimisation target.
Figure 9 shows how GPU latencies can be usedas constraints.
T-NAS and T-NAS++ show a better performance on the size-accuracy plot in Figure 9.
The smaller model sizes of T-NAS do not provide a better run-time on GPU devices (Figure 5), in fact,
T-NAS based models have the worst run-time on GPU devices due to the complicated dependencyof DARTS cells.
We only compare to MAML and MAML++ when running on Eyeriss due to thelimitations of the ScaleSIM simulator Samajdar et al.
[Figure]
[Figure]
[Figure]
K License of the assets
In our work, we utilized the following datasets/library/code listed in Table 12.
[Table]
The vast majority of the work was implemented ourselves and will be released under the permissive
MIT license , which allows future researchers to build on the work unconstrained (only requiring preservation of the license ﬁle).
All dependencies of my library are similarly released under OSI 1approved licenses, allowing them all to be easily compiled and installed.
L Computation resources
Apart from the hardware devices used for proﬁling in the main paper.
All experiments complete in< 30 GPU-days on a four NVIDIA GeForce GTX 1080 Ti system with an Intel(R) Xeon(R) CPU
E5-2620 v4 at 2.10GHz.