Original: T OWARDS F AST A DAPTATION OF N EURAL A RCHITECTURES WITH M ETA L EARNING
Revised: T OWARDS F AST A DAPTATION OF N EURAL A RCHITECTURES WITH M ETA L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, Neural Architecture Search (NAS) has been successfully applied to multiple artiﬁcial intelligence areas and shows better performance compared with hand-designed networks.
Revised: Recently, Neural Architecture Search (NAS) has been successfully applied to multiple artiﬁcial intelligence areas and shows better performance compared with hand-designed networks.

Original: However, the existing NAS methods only target a speciﬁc task.
Revised: However, the existing NAS methods only target a speciﬁc task.

Original: Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks.
Revised: Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks.

Original: Generally, the architecture for a new task is either searched from scratch, which is neither efﬁcient nor ﬂexible enough for practical application scenarios, or borrowed from the ones searched on other tasks, which might be not optimal.
Revised: Generally, the architecture for a new task is either searched from scratch, which is neither efﬁcient nor ﬂexible enough for practical application scenarios, or borrowed from the ones searched on other tasks, which might be not optimal.

Original: In order to tackle the transferability of NAS and conduct fast adaptation of neural architectures, we propose a novel Transferable Neural Architecture Search method based on meta-learning in this paper, which is termed as T-NAS.
Revised: In order to tackle the transferability of NAS and conduct fast adaptation of neural architectures, we propose a novel Transferable Neural Architecture Search method based on meta-learning in this paper, which is termed as T-NAS.

Original: T-NAS learns a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which makes the transferred architecture suitable for the speciﬁc task.
Revised: T-NAS learns a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which makes the transferred architecture suitable for the speciﬁc task.

Original: Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.
Revised: Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have achieved huge successes in multiple artiﬁcial intelligence areas, e.g. , computer vision Girshick (2015);
Revised: Deep neural networks have achieved huge successes in many machine learning tasks (Girshick, 2015;

Original: He et al. and natural language processing Sutskever et al.
Revised: He et al., 2016; Sutskever et al., 2014; Zheng et al., 2015b; Lian et al., 2019; Cheng et al., 2019; Zheng et al., 2015a; Lauly et al., 2017; Jiang et al., 2017; Zheng et al., 2016).

Original: Behind their successes, the design of network architecture plays an important role, and the hand-designed networks ( e.g. ,ResNet He et al.
Revised: Behind their successes, the design of network architecture plays an important role, and the hand-designed networks ( e.g. ,

Original:  DenseNet Huang et al. (2017)) have provided strong baselines in many tasks.
Revised: ResNet (He et al., 2016), DenseNet (Huang et al., 2017)) have provided strong baselines in many tasks.

Original: Neural Architecture Search (NAS)  is proposed to automatically search network structure for alleviating the complicated network design and heavy dependence on prior knowledge.
Revised: Neural Architecture Search (NAS) (Pham et al., 2018; Liu et al., 2018b; Guo et al., 2019) is proposed to automatically search network structure for alleviating the complicated network design and heavy dependence on prior knowledge.

Original: More importantly, NAS has been proved to be effective and obtained the remarkable performance in image classiﬁcation Pham et al.
Revised: More importantly, NAS has been proved to be effective and obtained the remarkable performance in image classiﬁcation (Pham et al., 2018;

Original: ; Liu et al. (2018b), object detection Ghiasi et al. and semantic segmentation Chen et al. Liu et al.
Revised: Liu et al., 2018b), object detection (Ghiasi et al., 2019) and semantic segmentation (Chen et al., 2018; Liu et al., 2019).

Original: However, the existing NAS methods only target a speciﬁc task.
Revised: However, the existing NAS methods only target a speciﬁc task.

Original: Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks.
Revised: Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks.

Original: As shown in Figure 1, we get the architectureon given dataset using a NAS method.
Revised: As shown in Figure 1, we get the architecture-0 on a given dataset using a NAS method.

Original: Now what if there exist a new task?
Revised: Now, what if there exists a new task?

Original: This drives us to ask: how to get a suitable architecture for a new task in NAS?
Revised: This drives us to ask: how to get a suitable architecture for a new task in NAS?

Original: Generally, there exist two naive solutions in handling multiple tasks.
Revised: Generally, there exist two simple solutions in handling multiple tasks.

Original: One of them (S1) is to search an architecture for a new task from scratch but it is inefﬁcient and not ﬂexible for practical application scenarios.
Revised: One of them (S1) is to search an architecture for a new task from scratch but it is inefﬁcient and not ﬂexible for practical application scenarios.

Original: Another solution (S2) is to borrow architecture from the ones searched on other tasks but it might be not optimal for the new task.
Revised: Another solution (S2) is to borrow architecture from the ones searched on other tasks but it might be not optimal for the new task.

Original: Therefore, it is urgently needed to study the transferability of NAS for large-scale model deployment in practical application.
Revised: Therefore, it is urgently needed to study the transferability of NAS for large-scale model deployment in practical application.

Original: It should be  more desirable to learn a transferable architecture that can adapt to some new unseen tasks easily and quickly according to the previous knowledge.
Revised: It should be [Figure] more desirable to learn a transferable architecture that can adapt to some new unseen tasks easily and quickly according to the previous knowledge.

Original: To this end, we propose a novel Transferable Neural Architecture Search (T-NAS) method (the bottom of Figure 1).
Revised: To this end, we propose a novel Transferable Neural Architecture Search (T-NAS) method (the bottom of Figure 1).

Original: The starting point of T-NAS is inspired by recent meta-learning methods Finn et al. ; Antoniou et al. ; Sun et al. (2019), especially Model-Agnostic Meta-Learning (MAML)
Revised: The starting point of T-NAS is inspired by recent meta-learning methods (Finn et al., 2017; Antoniou et al., 2019; Sun et al., 2019), especially Model-Agnostic Meta-Learning (MAML)

Original: Finn et al. (2017), where a model learns the meta-weights that are able to adapt to a new task through a few gradient steps.
Revised: (Finn et al., 2017), where a model learns the meta-weights that are able to adapt to a new task through a few gradient steps.

Original: Push it forward, it is also possible to ﬁnd a good initial point of network architecture for NAS.
Revised: Push it forward, it is also possible to ﬁnd a good initial point of network architecture for NAS.

Original: Therefore, the T-NAS learns a meta-architecture (transferable[Figure] architecture) that is able to adapt to a new task quickly through a few gradient steps, which is more ﬂexible than other NAS methods.
Revised: Therefore, the T-NAS learns a meta-architecture (transferable architecture) that is able to adapt to a new task quickly through a few gradient steps, which is more ﬂexible than other NAS methods.

Original: Similar to MAML, such a good initial meta-architecture for adaptation should be more sensitive to changes in different tasks such that it can be easily transferred.
Revised: Similar to MAML, such a good initial meta-architecture for adaptation should be more sensitive to changes in different tasks such that it can be easily transferred.

Original: It is worth mentioning that this is not the ﬁrst work on the transferability of neural architecture.
Revised: It is worth mentioning that this is not the ﬁrst work on the transferability of neural architecture.

Original: There are also some recent works that attempt to utilize the knowledge on neural architectures learned from previous tasks, such as Wong et al.
Revised: There are also some recent works that attempt to utilize the knowledge on neural architectures learned from previous tasks, such as Wong et al.

Original: ; Shaw et al.
Revised: ; Shaw et al.

Original: Speciﬁcally, Wong et al.
Revised: Speciﬁcally, Wong et al.

Original: (2018) propose to transfer the architecture knowledge under a multi-task learning perspective, where the number of tasks is ﬁxed during training phase, and it cannot do a fast adaption for a new task.
Revised: (2018) proposes to transfer the architecture knowledge under a multi-task learning perspective, where the number of tasks is ﬁxed during training phase, and it cannot do a fast adaption for a new task.

Original: In contrast, our model is able to make the adaption fast and the number of tasks is unlimited during training.
Revised: In contrast, our model is able to make the adaption fast and the number of tasks is unlimited during training.

Original: The difference between our model and Shaw et al.
Revised: The difference between our model and Shaw et al.

Original: (2018) is also obvious, where Shaw et al.
Revised: (2018) is also obvious, where Shaw et al.

Original: (2018) is based on Bayesian inference but our model is based on gradinet-based meta-learning.
Revised: (2018) is based on Bayesian inference but our model is based on gradient-based meta-learning.

Original: The quantitative comparison with Shaw et al.
Revised: The quantitative comparison with Shaw et al.

Original: (2018) can be found in Table 3.
Revised: (2018) can be found in Table 3.

Original: Generally, architecture structure cannot be trained independently regardless of network weights Liu et al. (2018b); Pham et al.
Revised: Generally, architecture structure cannot be trained independently regardless of network weights (Liu et al., 2018b; Pham et al., 2018).

Original: Analogously, the training of meta-architecture is also associated with meta-weights.
Revised: Analogously, the training of meta-architecture is also associated with meta-weights.

Original: Therefore, the meta-architecture and meta-weights need to be optimized jointly across different tasks, which is a typical bilevel optimization problem Liu et al. (2018b).
Revised: Therefore, the meta-architecture and meta-weights need to be optimized jointly across different tasks, which is a typical bilevel optimization problem (Liu et al., 2018b).

Original: In order to solve the costly bilevel optimization in T-NAS, we propose an efﬁcient ﬁrst-order approximation algorithm to update meta-architecture and meta-weights together.
Revised: In order to solve the costly bilevel optimization in T-NAS, we propose an efﬁcient ﬁrst-order approximation algorithm to update meta-architecture and meta-weights together.

Original: After the whole model is optimized, given a new task, we can get the network architecture structure suitable for the speciﬁc task with a few gradient steps from meta-architecture and meta-weights.
Revised: After the whole model is optimized, given a new task, we can get the network architecture structure suitable for the speciﬁc task with a few gradient steps from meta-architecture and meta-weights.

Original: At last, the decoded discrete architecture is used for the ﬁnal architecture evaluation.
Revised: At last, the decoded discrete architecture is used for the ﬁnal architecture evaluation.

Original: To demonstrate the effectiveness of T-NAS, we conduct extensive experiments on task-level problems due to amounts of tasks.
Revised: To demonstrate the effectiveness of T-NAS, we conduct extensive experiments on task-level problems due to amounts of tasks.

Original: Speciﬁcally, we split the experiments into two parts: few-shot learning setting and supervised learning setting.
Revised: Speciﬁcally, we split the experiments into two parts: few-shot learning setting and supervised learning setting.

Original: For few-shot learning, T-NAS achieves state-of-the-art performance on multiple datasets (Omniglot, Mini-Imagenet, Fewshot-CIFAR100) compared with previous methods and other NAS-based methods.
Revised: For few-shot learning, T-NAS achieves state-of-the-art performance on multiple datasets (Omniglot, Mini-Imagenet, Fewshot-CIFAR100) compared with previous methods and other NAS-based methods.

Original: As for supervised learning, a 200-shot 50-query 10-way experiment setting is designed on the Mini-Imagenet dataset.
Revised: As for supervised learning, a 200-shot 50-query 10-way experiment setting is designed on the Mini-Imagenet dataset.

Original: Compared with the searched architectures from scratch for new given tasks, T-NAS achieves comparable performance but with 50x less searching cost.
Revised: Compared with the searched architectures from scratch for new given tasks, T-NAS achieves comparable performance but with 50x less searching cost.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Revised: • To our best knowledge, it is one of

Original: • We propose a novel Transferable Neural Architecture Search (T-NAS).
Revised: • We propose a novel Transferable Neural Architecture Search (T-NAS).

Original: T-NAS can learn a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which is more ﬂexible than other NAS methods.
Revised: T-NAS can learn a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which is more ﬂexible than other NAS methods.

Original: • We give the formulation of T-NAS and analyze the difference between T-NAS and other NAS methods.
Revised: • We give the formulation of T-NAS and analyze the difference between T-NAS and other NAS methods.

Original: Further, to solve the bilevel optimization, we propose an efﬁcient ﬁrst-order approximation algorithm to optimize the whole search network based on gradient descent.
Revised: Further, to solve the bilevel optimization, we propose an efﬁcient ﬁrst-order approximation algorithm to optimize the whole search network based on gradient descent.

Original: • Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.
Revised: • Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.

Original:  R ELATED W ORK 2. N EURAL A RCHITECTURE S EARCH
Revised:  R ELATED W ORK 2. N EURAL A RCHITECTURE S EARCH

Original: Neural Architecture Search (NAS) designs network architectures automatically instead of handdesigned ones.
Revised: Neural Architecture Search (NAS) designs network architectures automatically instead of handdesigned ones.

Original: Generally, NAS strategies are divided into three categories - reinforcement learning, evolutionary algorithm and gradient-based methods.
Revised: Generally, NAS strategies are divided into three categories - reinforcement learning, evolutionary algorithm and gradient-based methods.

Original: Some other strategies can refer to the survey paper Elsken et al.
Revised: Some other strategies can refer to the survey paper (Elsken et al., 2019).

Original: Reinforcement learning (RL) based methods Zoph & Le (2016); Zoph et al. (2018) utilize a controller to generate the network structure and operations.
Revised: Reinforcement learning (RL) based methods (Zoph & Le, 2016; Zoph et al., 2018) utilize a controller to generate the network structure and operations.

Original: For efﬁcient searching, ENAS Pham et al. (2018) shares parameters among child models and achieves state-ofthe-art performance with only one GPU day.
Revised: For efﬁcient searching, ENAS (Pham et al., 2018) shares parameters among child models and achieves state-of-the-art performance with only one GPU day.

Original: Evolutionary algorithm based methods Real et al. (2018) evolve neural architectures and also achieve comparable results with RL based methods.
Revised: Evolutionary algorithm based methods (Real et al., 2018) evolve neural architectures and also achieve comparable results with RL based methods.

Original: Unlike reinforcement learning and evolutionary algorithm, gradient-based methods Liu et al. (2018b) ; Cai et al. (2019) continuously relax the discrete architecture with all possible operations, which makes it possible to jointly optimize the architecture structure and network weights based on gradient descent.
Revised: Unlike reinforcement learning and evolutionary algorithm, gradient-based methods (Liu et al., 2018b; Cai et al., 2019) continuously relax the discrete architecture with all possible operations, which makes it possible to jointly optimize the architecture structure and network weights based on gradient descent.

Original: Not limited to image classiﬁcation problems, recent works also introduce NAS to object detection Ghiasi et al. and semantic image segmentation Chen et al.
Revised: Not limited to image classiﬁcation problems, recent works also introduce NAS to object detection (Ghiasi et al., 2019) and semantic image segmentation (Chen et al., 2018;

Original: Liu et al.
Revised: Liu et al., 2019).

Original: More recently, NAS is also applied to the generative model, such as AutoGAN Gong et al.
Revised: More recently, NAS is also applied to the generative model, such as AutoGAN (Gong et al., 2019).

Original: These NAS method show that the searched networks outperform the hand-designed ones.
Revised: These NAS methods show that the searched networks outperform the hand-designed ones.

Original: However, in these methods, only a ﬁxed architecture is searched for a speciﬁc task, which making it hard to be transferred to other tasks.
Revised: However, in these methods, only a ﬁxed architecture is searched for a speciﬁc task, which makes it hard to be transferred to other tasks.

Original: In order to obtain a more ﬂexible network, InstaNAS Cheng et al. (2018) is proposed to search the network architecture structure for each instance according to different objectives, such as accuracy or latency.
Revised: In order to obtain a more ﬂexible network, InstaNAS (Cheng et al., 2018) is proposed to search the network architecture structure for each instance according to different objectives, such as accuracy or latency.

Original: Different from Cheng et al.
Revised: Different from Cheng et al.

Original: (2018), we incorporate the ideas from meta-learning based methods and extend NAS to T-NAS, which learns a meta-architecture that is able to adapt to different tasks.
Revised: (2018), we incorporate the ideas from meta-learning based methods and extend NAS to T-NAS, which learns a meta-architecture that is able to adapt to different tasks.

Original: 2. F EW -S HOT M ETA -L EARNING
Revised: 2. F EW -S HOT M ETA -L EARNING

Original: Recently, most of few-shot learning problems can be cast into the meta-learning ﬁeld, where a model is trained to quickly adapt to a new task given only a few samples Finn et al.
Revised: Recently, most of few-shot learning problems can be cast into the meta-learning ﬁeld, where a model is trained to quickly adapt to a new task given only a few samples (Finn et al., 2017).

Original: Such few-shot meta-learning methods can be categorized into metric learning Vinyals et al.
Revised: Such few-shot meta-learning methods can be categorized into metric learning (Vinyals et al., 2016;

Original: ; Sung et al.
Revised: Sung et al., 2018;

Original: ; Snell et al. memory network Santoro et al. ; Oreshkin et al. Munkhdalai et al.
Revised: Snell et al., 2017), memory network (Santoro et al., 2016; Oreshkin et al., 2018; Munkhdalai et al., 2018;

Original: ; Mishra et al. and gradient-based methods Finn et al. ; Zhang et al.
Revised: Mishra et al., 2018) and gradient-based methods (Finn et al., 2017; Zhang et al., 2018;

Original: ; Sun et al.
Revised: Sun et al., 2019).

Original: Here, we only focus on the gradient-based methods, which contains a base-learner and a metalearner.
Revised: Here, we only focus on the gradient-based methods, which contain a base-learner and a meta-learner.

Original: MAML Finn et al. (2017) is one of the typical gradient-based methods for fast adaptation, which consists of meta-train and meta-test stages.
Revised: MAML (Finn et al., 2017) is one of the typical gradient-based methods for fast adaptation, which consists of meta-train and meta-test stages.

Original: In the meta-train stage, the model extract general knowledge (meta-weights) from amounts of tasks such that it can be utilized for fast adaptation in the meta-test stage.
Revised: In the meta-train stage, the model extracts general knowledge (meta-weights) from amounts of tasks such that it can be utilized for fast adaptation in the meta-test stage.

Original: The latest variant of MAML is MAML++ Antoniou et al. (2019), which analyzes the shortcoming of MAML and proposes some tips on how to train MAML to promote the performance.
Revised: The latest variant of MAML is MAML++ (Antoniou et al., 2019), which analyzes the shortcoming of MAML and proposes some tips on how to train MAML to promote the performance.

Original: We extend the adaptation of weights in MAML to the adaptation of architectures, and our method is also based on MAML and propose to automatically learn a meta-architecture, which is able to adapt to different tasks quickly.
Revised: We extend the adaptation of weights in MAML to the adaptation of architectures that is also based on MAML, and propose to automatically learn a meta-architecture, which is able to adapt to different tasks quickly.

Original:  P RELIMINARY
Revised:  P RELIMINARY

Original: To introduce T-NAS, we brieﬂy review the knowledge about meta-learning for fast adaptation Finn et al. ; Antoniou et al. (2019) and DARTS for NAS Liu et al. (2018b) in this section, which is helpful to understand the concept of T-NAS.
Revised: To introduce T-NAS, we brieﬂy review the knowledge about meta-learning for fast adaptation (Finn et al., 2017; Antoniou et al., 2019) and DARTS for NAS (Liu et al., 2018b) in this section, which is helpful to understand the concept of T-NAS.

Original: 3. M ETA -L EARNING
Revised: 3. M ETA -L EARNING

Original: The whole dataset, meta-train and meta-test dataset are denoted as D , D meta-train and D meta-test , respectively.
Revised: The whole dataset, meta-train and meta-test dataset are denoted as D , D meta-train and D meta-test , respectively.

Original: In meta-train stage, amounts of tasks {T } (are also called episodes) are sampled from the task distribution p ( T ) in D meta-train .
Revised: In meta-train stage, amounts of tasks {T } (are also called episodes) are sampled from the task distribution p ( T ) in D meta-train .

Original: Note that in the i -th task T i , there are K samples from each class and N classes in total, which is typically formulated as a N -way, K -shot problem.
Revised: Note that in the i -th task T i , there are K samples from each class and N classes in total, which is typically formulated as a N -way, K -shot problem.

Original: The train- ing split samples in T i used to optimize the base-learner are called support set, denoted as T si , and test split samples used to optimize the meta-learner are called query set, which is T qi .
Revised: The training split samples in T i used to optimize the base-learner are called support set, denoted as T si , and test split samples used to optimize the meta-learner are called query set, which is T qi .

Original: The main idea of MAML Finn et al. (2017) is to learn good initialized weights (cid:101) w for all tasks {T } , such that the network can obtain high performance in D meta-test after a few steps from (cid:101) w .
Revised: The main idea of MAML (Finn et al., 2017) is to learn good initialized weights (cid:101) w for all tasks {T } , such that the network can obtain high performance in D meta-test after a few steps from (cid:101) w .

Original: The base-learner is optimized according to the following rule:
Revised: The base-learner is optimized according to the following rule:

Original: [Equation]
Revised: [Equation]

Original: where α inner is the inner (base) learning rate of weights w and m represents the inner step.
Revised: where α inner is the inner (base) learning rate of weights w and m represents the inner step.

Original: f is the parametrized function with network weights w and L is the loss function.
Revised: f is the parametrized function with network weights w and L is the loss function.

Original: In the base-learner process, T si is used to compute the loss and we update weights w from w mi to w m +

Revised: In the base-learner process, T si is used to compute the loss and we update weights w from w mi to w m +

Original: i for the i -th task ( w 0 i = (cid:101) w ).
Revised: i for the i -th task ( w 0 i = (cid:101) w ).

Original: After M steps, L ( f ( T qi ; w Mi )) in T q is computed for the meta-learner update, which can be formulated as:
Revised: After M steps, L ( f ( T qi ; w Mi )) in T q is computed for the meta-learner update, which can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where α outer is the outer (meta) learning rate of meta-weights (cid:101) w .
Revised: where α outer is the outer (meta) learning rate of meta-weights (cid:101) w .

Original: Finally, the model learns the good initialized meta-weights (cid:101) w when it converges.
Revised: Finally, the model learns the good initialized meta-weights (cid:101) w when it converges.

Original: Such meta-weights are sensitive enough so that it can adapt to each task in D meta-test after a few steps.
Revised: Such meta-weights are sensitive enough so that it can adapt to each task in D meta-test after a few steps.

Original: 3. DARTS
Revised: 3. DARTS

Original: The core of DARTS Liu et al. (2018b) is to continuously relax the discrete architecture with all possible operations and jointly optimize the architecture structure and network weights based on gradient descent.
Revised: The core of DARTS (Liu et al., 2018b) is to continuously relax the discrete architecture with all possible operations and jointly optimize the architecture structure and network weights based on gradient descent.

Original: Consider O be a set of candidate operations, where each operation proposal is represented with o .
Revised: Consider O be a set of candidate operations, where each operation proposal is represented with o .

Original: Given the input x , the output is the weighted sum of all possible operations o 
Revised: Given the input x , the output is the weighted sum of all possible operations o ( x

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where θ is the vector to represent the coefﬁcients of different operation branches.
Revised: where θ is the vector to represent the coefﬁcients of different operation branches.

Original: When decoding, the operation o = arg max o ∈O θ o .
Revised: When decoding, the operation o = arg max o ∈O θ o .

Original: Therefore, θ is also the encoding of the architecture.
Revised: Therefore, θ is also the encoding of the architecture.

Original: To solve such a bi-level optimization problem, a two-step update algorithm is applied:
Revised: To solve such a bi-level optimization problem, a two-step update algorithm is applied:

Original: [Equation]
Revised: [Equation]

Original: where L is the loss function and ξ is the learning rate of inner optimization.
Revised: where L is the loss function and ξ is the learning rate of inner optimization.

Original: In this paper, we use the ﬁrst-order optimization of DARTS ( ξ = 0 ) for efﬁciency.
Revised: In this paper, we use the ﬁrst-order optimization of DARTS ( ξ = 0 ) for efﬁciency.

Original:  A PPROACH
Revised:  A PPROACH

Original: In this section, we ﬁrst introduce Transferable Neural Architecture Search (T-NAS) and give the formulation of it.
Revised: In this section, we ﬁrst introduce Transferable Neural Architecture Search (T-NAS) and give the formulation of it.

Original: After that, we analyze and illustrate the difference between T-NAS and NAS.
Revised: After that, we analyze and illustrate the difference between T-NAS and NAS.

Original: Finally, the ﬁrst-order approximation algorithm is proposed for the optimization of T-NAS, and the adaptation and decoding process are also described in detail.
Revised: Finally, the ﬁrst-order approximation algorithm is proposed for the optimization of T-NAS, and the adaptation and decoding process are also described in detail.

Original: 4. T HE F ORMULATION OF T-NAS
Revised: 4. T HE F ORMULATION OF T-NAS

Original: To make searched network architecture ﬂexible, we focus on the transferability of NAS.
Revised: To make searched network architecture ﬂexible, we focus on the transferability of NAS.

Original: As shown in Sec. 3, MAML is trained to learn meta-weights (cid:101) w for fast adaptation in a new task.
Revised: As shown in Sec. 3, MAML is trained to learn meta-weights (cid:101) w for fast adaptation in a new task.

Original: Similarly,
Revised: Similarly,

Original: T-NAS devotes itself to learn a meta-architecture (cid:101)
Revised: T-NAS devotes itself to learn a meta-architecture (cid:101)

Original: θ that is able to adapt to a new task through a few steps.
Revised: θ that is able to adapt to a new task through a few steps.

Original: In this work, θ and (cid:101) θ 1 are deﬁned as the encoding of the architecture and transferable architecture, which are represented as matrices following DARTS Liu et al. (2018b).
Revised: In this work, θ and (cid:101) θ 1 are deﬁned as the encoding of the architecture and transferable architecture, which are represented as matrices following DARTS (Liu et al., 2018b).

Original: To make the searched architecture transferable, we utilize the meta-learning based strategy to learn a task-sensitive meta-architecture (cid:101) θ .
Revised: To make the searched architecture transferable, we utilize the meta-learning based strategy to learn a task-sensitive meta-architecture (cid:101) θ .

Original: However, similar to other NAS methods Pham et al.
Revised: However, similar to other NAS methods (Pham et al., 2018;

Original: Liu et al. (2018b), where the architecture θ usually cannot be trained independently regardless of network weights w , the training of meta-architecture (cid:101) θ is also associated with meta-weights (cid:101) w .
Revised: Liu et al., 2018b), where the architecture θ usually cannot be trained independently regardless of network weights w , the training of meta-architecture (cid:101) θ is also associated with meta-weights (cid:101) w .

Original: In this work, (cid:101) θ and (cid:101) w are optimized jointly across different tasks in T-NAS.
Revised: In this work, (cid:101) θ and (cid:101) w are optimized jointly across different tasks in T-NAS.

Original: As shown in Sec. 3, there exist two learners for the learning of meta-weights (cid:101) w , i.e. , Eq.
Revised: As shown in Sec. 3, there exist two learners for the learning of meta-weights (cid:101) w , i.e. , Eq.

Original: (1) is used to update the base-learner and Eq.
Revised: (1) is used to update the base-learner and Eq.

Original: (2) is used to update the meta-learner.
Revised: (2) is used to update the meta-learner.

Original: Similarly, T-NAS consists of two searchers: base-searcher and meta-searcher.
Revised: Similarly, T-NAS consists of two searchers: base-searcher and meta-searcher.

Original: In the base-searcher, θ and w are optimized jointly to search architecture in T si for the speciﬁc task, which can be optimized with:
Revised: In the base-searcher, θ and w are optimized jointly to search architecture in T s i for the speciﬁc task, which can be optimized with:

Original: [Equation]
Revised: [Equation]

Original: where β inner is the inner (base) learning rate of architecture θ .
Revised: where β inner is the inner (base) learning rate of architecture θ .

Original: g is the parametrized function with the architecture θ and network weights w ( θ 

Revised: g is the parametrized function with the architecture θ and network weights w ( θ

Original: i
Revised: i

Original: =
Revised: =

Original: (cid:101) θ, w 0 i = (cid:101) w ).
Revised: (cid:101) θ, w 0 i = (cid:101) w ).

Original: After M steps, (cid:101) θ and (cid:101) w are also updated to get a good initial point for architecture adaptation in the meta-searcher,
Revised: After M steps, (cid:101) θ and (cid:101) w are also updated to get a good initial point for architecture adaptation in the meta-searcher,

Original: where L ( g ( T qi ; θ Mi , w Mi )) in T q i is computed.
Revised: where L ( g ( T qi ; θ Mi , w Mi )) in T qi is computed.

Original: The formulation can be represented as:
Revised: The formulation can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where β outer is the outer (meta) learning rate of the meta-architecture (cid:101) θ .
Revised: where β outer is the outer (meta) learning rate of the meta-architecture (cid:101) θ .

Original: When the meta-searcher converges, the optimal meta-architecture (cid:101) θ
Revised: When the meta-searcher converges, the optimal meta-architecture (cid:101) θ

Original: and meta-weights (cid:101) w can be obtained.
Revised: and meta-weights (cid:101) w can be obtained.

Original: We argue that such a (cid:101) θ can quickly adapt to a new task.
Revised: We argue that such a (cid:101) θ can quickly adapt to a new task.

Original: The complete algorithm of T-NAS is as shown in Alg. 1.
Revised: The complete algorithm of T-NAS is as shown in Alg. 1.

Revised: Algorithm 1: T-NAS: Transferable Neural Architecture Search

Revised: Input

Revised: [Table]

Original: 4. T-NAS VS . NAS
Revised: 4. T-NAS VS . NAS

Original: As mentioned before, the previous NAS methods usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks.
Revised: As mentioned before, the previous NAS methods usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks.

Original: Such we focus on the transferability of NAS across multiple tasks in this paper.
Revised: Such we focus on the transferability of NAS across multiple tasks in this paper.

Original: Two naive solutions (S1 and S2) have been pointed in Figure 1 but they are either inefﬁcient or not optimal.
Revised: Two simple solutions (S1 and S2) have been pointed in Figure 1 but they are either inefﬁcient or not optimal.

Original: T-NAS aims to learn a transferable and ﬂexible architecture that can adapt to some new tasks easily.
Revised: T-NAS aims to learn a transferable and ﬂexible architecture that can adapt to some new tasks easily.

Original: Table 1 lists the main differences among NAS, two naive solutions (S1 and S2) and T-NAS.
Revised: Table 1 lists the main differences among NAS, two simple solutions (S1 and S2) and T-NAS.

Original: S1 does not study the transferability of NAS and searches architectures for different tasks ( e.g. , θ 1 , θ 2 , ..., θ n ) from scratch.
Revised: S1 does not study the transferability of NAS and searches architectures for different tasks ( e.g. , θ 1 , θ 2 , ..., θ n ) from scratch.

Original: S2 borrows from searched architecture directly such that all tasks share the same architecture ( e.g. , θ ).
Revised: S2 borrows from searched architecture directly such that all tasks share the same architecture ( e.g. , θ ).

Original: Differently, T-NAS searches the meta-architecture (cid:101) θ , which is able to adapt to different tasks quickly ( e.g. , (cid:101) θ
Revised: Differently, T-NAS searches the meta-architecture (cid:101) θ , which is able to adapt to different tasks quickly ( e.g. , (cid:101) θ

Original: → θ 1 , θ 2 , ..., θ n ).
Revised: → θ 1 , θ 2 , ..., θ n ).

Original: The experimental results show that our method achieves better performance than the S2 and comparable performance with S1 but with  less searching cost.[Table]
Revised: The experimental results show that our method achieves better performance than the S2 and comparable performance with S1 but with the less searching cost.

Original: It is worth mentioning that if directly apply NAS to few-shot meta-learning, e.g. , MAML Finn et al. (2017), we will search a good network architecture for MAML, which is named Auto-MAML.
Revised: It is worth mentioning that if directly apply NAS to few-shot meta-learning, e.g. , MAML (Finn et al., 2017), we will search a good network architecture for MAML, which is named Auto-MAML.

Original: In fact, Auto-MAML is a special case of S2 in Figure 1, where all tasks share the same architecture searched with a meta-learning method.
Revised: In fact, Auto-MAML is a special case of S2 in Figure 1, where all tasks share the same architecture searched with a meta-learning method.

Original: In the experiments in few-shot learning, we also introduce
Revised: In the experiments in few-shot learning, we also introduce

Original: Auto-MAML as a baseline.
Revised: Auto-MAML as a baseline.

Original: However, such a shared architecture is not suitable for each task.
Revised: However, such a shared architecture is not suitable for each task.

Original: AutoMAML can outperform MAML but is inferior to T-NAS.
Revised: AutoMAML can outperform MAML but is inferior to T-NAS.

Original: The speciﬁc algorithm and experimental settings of Auto-MAML are provided in the supplementary material.
Revised: The speciﬁc algorithm and experimental settings of Auto-MAML are provided in the supplementary material.

Original: The core of T-NAS is based on MAML Finn et al. (2017), which is a kind of gradient based metalearning methods.
Revised: The core of T-NAS is based on MAML (Finn et al., 2017), which is a kind of gradient-based metalearning method.

Original: Recently, MAML++ is proposed by Antoniou et al.
Revised: Recently, MAML++ is proposed by Antoniou et al.

Original: (2019), which introduces several techniques 2 to improve the performance of MAML.
Revised: (2019), which introduces several techniques 2 to improve the performance of MAML.

Original: These techniques can also be utilized by T-NAS, which is termed as T-NAS++ in this paper.
Revised: These techniques can also be utilized by T-NAS, which is termed as T-NAS++ in this paper.

Original: The experiments in Section 5 conﬁrm that
Revised: The experiments in Section 5 conﬁrm that

Original: T-NAS++ can further improve the performance of T-NAS.
Revised: T-NAS++ can further improve the performance of T-NAS.

Original: 4. O PTIMIZATION
Revised: 4. O PTIMIZATION

Original: Although the formulation of T-NAS is proposed, the model is hard to be optimized directly according to Alg.
Revised: Although the formulation of T-NAS is proposed, the model is hard to be optimized directly according to Alg.

Original: On one hand, updating (cid:101) θ and (cid:101) w introduces the high-order derivative in Eq.
Revised: On one hand, updating (cid:101) θ and (cid:101) w introduces the high-order derivative in Eq.

Original: On the other hand, the continuous relaxation of architecture makes amounts of memory occupied.
Revised: On the other hand, the continuous relaxation of architecture makes amounts of memory occupied.

Original: At the ﬁrst glance, such a problem might be solved by the ﬁrst-order approximation in Liu et al.
Revised: At the ﬁrst glance, such a problem might be solved by the ﬁrst-order approximation in Liu et al.

Original: (2018b), however, there still exists a lot of time overhead, even the experiments cannot be carried out when step M is large in Eq.
Revised: (2018b), however, there still exists a lot of time overhead, even the experiments cannot be carried out when step M is large in Eq.

Original: To tackle this problem, we transform the alternative update strategy of w and θ in Eq.
Revised: To tackle this problem, we transform the alternative update strategy of w and θ in Eq.

Original: (5) into update together, which means the w and θ are treated equally as the parameters of function g .
Revised: (5) into update together, which means the w and θ are treated equally as the parameters of function g .

Original: Such a transformation can update parameters ( w and θ ) by only backpropagating once instead of twice.
Revised: Such a transformation can update parameters ( w and θ ) by only backpropagating once instead of twice.

Original: The Eq.
Revised: The Eq.

Original: (5) can be modiﬁed to:
Revised: (5) can be modiﬁed to:

Original: [Equation]
Revised: [Equation]

Original: where η inner =
Revised: where η inner =

Original: [ α inner ; β inner ] .
Revised: [ α inner ; β inner ] .

Original: In addition, to avoid the high-order derivative, we also utilize the ﬁrst-order approximation to compute the derivation of w Mi and θ Mi instead of (cid:101) w and (cid:101) θ as follows:
Revised: In addition, to avoid the high-order derivative, we also utilize the ﬁrst-order approximation to compute the derivation of w M i and θ M i instead of (cid:101) w and (cid:101) θ as follows:

Original: [Equation]
Revised: [Equation]

Original: where η outer =
Revised: where η outer =

Original: [ α outer ; β outer ] .
Revised: [ α outer ; β outer ] .

Original: Such modiﬁcations save more than half of the search time and memory while maintaining the comparable performance.
Revised: Such modiﬁcations save more than half of the search time and memory while maintaining the comparable performance.

Original: Thus, we can use the Eq.
Revised: Thus, we can use the Eq.

Original: (7) and Eq.
Revised: (7) and Eq.

Original: (8) to replace the Eq.
Revised: (8) to replace the Eq.

Original: (5) and Eq.
Revised: (5) and Eq.

Original: (6) in line 7 and line 10 of Alg.
Revised: (6) in line 7 and line 10 of Alg.

Original: 1 to update θ and w in the implementation.
Revised: 1 to update θ and w in the implementation.

Original: 4. A DAPTATION AND D ECODING
Revised: A DAPTATION AND D ECODING

Original: Once (cid:101) θ and (cid:101) w are obtained by training the base-searcher and the meta-searcher with the ﬁrst-order approximation of Alg. 1, we can adapt them to the i -th task and get the task-speciﬁc architecture θ ∗ i for the speciﬁc task T i according to the following Alg. 2.
Revised: Once (cid:101) θ and (cid:101) w are obtained by training the base-searcher and the meta-searcher with the ﬁrst-order approximation of Alg. 1, we can adapt them to the i -th task and get the task-speciﬁc architecture θ ∗ i for the speciﬁc task T i according to the following Alg. 2.

Original: Following previous NAS methods Zoph & Le (2016); Zoph et al. ; Pham et al. ; Liu et al. (2018b), after getting θ ∗ i , we evaluate the task-speciﬁc architecture by training it in the task 
Revised: Following previous NAS methods (Zoph & Le, 2016; Zoph et al., 2018; Pham et al., 2018; Liu et al., 2018b), after getting θ ∗ i , we evaluate the task-speciﬁc architecture by training it in the task T i from scratch.

Original: T i from scratch.

Original: As shown in Sec. 5, the T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with less searching cost.[Table] [Table] [Table]
Revised: As shown in Sec. 5, the T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with less searching cost.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the effectiveness of T-NAS in both few-shot and supervised learning settings, as well as multiple datasets.
Revised: We evaluate the effectiveness of T-NAS in both few-shot and supervised learning settings, as well as multiple datasets.

Original: For each dataset, we conduct experiments containing architecture search and architecture evaluation.
Revised: For each dataset, we conduct experiments containing architecture search and architecture evaluation.

Original: In the architecture search stage, we use T-NAS to search  a meta-architecture.
Revised: In the architecture search stage, we use T-NAS to search for a metaarchitecture.

Original: In the architecture evaluation stage, we evaluate the transferred task-speciﬁc architectures by training them from scratch and compare their performance with previous methods.
Revised: In the architecture evaluation stage, we evaluate the transferred task-speciﬁc architectures by training them from scratch and compare their performance with previous methods.

Original: S1 and S2 in the following sections means two naive solutions in Figure 1 except for the speciﬁc instructions. 
Revised: S and S2 in the following sections mean two simple solutions in Figure 1 except for the speciﬁc instructions. Code is available 3 .

Original: 5. D ATASETS
Revised: 5. D ATASETS

Original: Omniglot is a handwritten character recognition dataset proposed in Lake et al.
Revised: Omniglot is a handwritten character recognition dataset proposed in Lake et al.

Original: (2011), which contains 1623 characters with 20 samples for each class.
Revised: (2011), which contains 1623 characters with 20 samples for each class.

Original: We randomly split 1200 characters for training and the remaining for testing, and augment the Omniglot dataset by randomly rotating multiples of 90 degrees following Santoro et al.
Revised: We randomly split 1200 characters for training and the remaining for testing, and augment the Omniglot dataset by randomly rotating multiples of 90 degrees following (Santoro et al., 2016).

Original: Mini-Imagenet dataset is sampled from the original ImageNet Deng et al.
Revised: Mini-Imagenet dataset is sampled from the original ImageNet (Deng et al., 2009).

Original: There areclasses in total with 600 images for each class.
Revised: There areclasses in total with 600 images for each class.

Original: All images are down-sampled to 84 × 84 pixels and the whole dataset consists of 64 training classes, 16 validation classes and 20 test classes.
Revised: All images are down-sampled to 84 × 84 pixels and the whole dataset consists of 64 training classes, 16 validation classes and 20 test classes.

Original: Fewshot-CIFAR100 (FC100) dataset is proposed in Oreshkin et al.
Revised: Fewshot-CIFAR100 (FC100) dataset is proposed in Oreshkin et al.

Original: (2018), which is based on a popular image classiﬁcation dataset CIFAR100.
Revised: (2018), which is based on a popular image classiﬁcation dataset CIFAR100.

Original: It is more challenging than the Mini-Imagenet due to the low resolution.
Revised: It is more challenging than the Mini-Imagenet due to the low resolution.

Original: Following Oreshkin et al.
Revised: Following Oreshkin et al.

Original: (2018), FC100 is divided into 60 classes belonging to 12 superclasses for training, 20 classes belonging to 4 superclasses for validation and testing.
Revised: (2018), FC100 is divided into 60 classes belonging to 12 superclasses for training, 20 classes belonging to 4 superclasses for validation and testing.

Original: 5. T-NAS FOR F EW -S HOT L EARNING 5.2. A RCHITECTURE S EARCH .
Revised: 5. T-NAS FOR F EW -S HOT L EARNING 5.2. A RCHITECTURE S EARCH .

Original: We ﬁrst get the meta-architecture (cid:101) θ by optimizing the search network with ﬁrst-order approximation of Alg. 1.
Revised: We ﬁrst get the meta-architecture (cid:101) θ by optimizing the search network with ﬁrst-order approximation of Alg. 1.

Original: In the architecture search stage, we employ the same operations as Liu et al.
Revised: In the architecture search stage, we employ the same operations as Liu et al.

Original: (2018b): 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 maxpooling, 3 × 3 average pooling, identity and zero.
Revised: (2018b): 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 max

Revised: (cid:2016) (cid:2928)(cid:2915)(cid:2914)(cid:2931)(cid:2913)(cid:2915) (cid:3560)(cid:2016) (cid:2924)(cid:2925)(cid:2928)(cid:2923)(cid:2911)(cid:2922) (cid:2016) (cid:2924)(cid:2925)(cid:2928)(cid:2923)(cid:2911)(cid:2922) (cid:3560)(cid:2016) (cid:2928)(cid:2915)(cid:2914)(cid:2931)(cid:2913)(cid:2915) (cid:2016) (cid:2924)(cid:2925)(cid:2928)(cid:2923)(cid:2911)(cid:2922)(cid:3047) (cid:2016) (cid:2928)(cid:2915)(cid:2914)(cid:2931)(cid:2913)(cid:2915)(cid:3047) pooling, 3 × 3 average pooling, identity and zero.

Original: ReLU-Conv-BN order is used for convolutional operations and each separable convolution is applied twice following Liu et al. (2018a;b).
Revised: ReLU-Conv-BN order is used for convolutional operations and each separable convolution is applied twice following (Liu et al., 2018a;b).

Original: For all datasets, we only use one { normal + reduction } cell for efﬁciency and preventing overﬁtting, thus 
Revised: For all datasets, we only use one { normal + reduction } cell for efﬁciency and preventing overﬁtting, thus the meta-architecture (cid:101) θ is determined by ( (cid:101) θ normal , (cid:101)

Original: (cid:2016) (cid:2928)(cid:2915)(cid:2914)(cid:2931)(cid:2913)(cid:2915) (cid:3560)(cid:2016) (cid:2924)(cid:2925)(cid:2928)(cid:2923)(cid:2911)(cid:2922) (cid:2016) (cid:2924)(cid:2925)(cid:2928)(cid:2923)(cid:2911)(cid:2922) (cid:3560)(cid:2016) (cid:2928)(cid:2915)(cid:2914)(cid:2931)(cid:2913)(cid:2915) (cid:2016) (cid:2924)(cid:2925)(cid:2928)(cid:2923)(cid:2911)(cid:2922)(cid:3047) (cid:2016) (cid:2928)(cid:2915)(cid:2914)(cid:2931)(cid:2913)(cid:2915)(cid:3047) the meta-architecture (cid:101) θ is determined by ( (cid:101) θ normal , (cid:101)

Original: θ reduce ) .
Revised: θ reduce ) .

Original: Once (cid:101) θ is obtained using T-NAS, we can obtain the optimal architecture θ ∗ i for the speciﬁc task T i from Alg. 2.
Revised: Once (cid:101) θ is obtained using T-NAS, we can obtain the optimal architecture θ ∗ i for the speciﬁc task T i from Alg. 2.

Original: We utilize the training and validation data of dataset for architecture search.
Revised: We utilize the training and validation data of dataset for architecture search.

Original: In N-way, K-shot setting, we ﬁrstly randomly sample N classes from the training classes, and then randomly sample K images for each class to get a task.
Revised: In N-way, K-shot setting, we ﬁrstly randomly sample N classes from the training classes, and then randomly sample K images for each class to get a task.

Original: Thus, there are N × K images in each task.
Revised: Thus, there are N × K images in each task.

Original: On the Mini-imagenet dataset, One { normal + reduction } cell is trained 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16 .
Revised: On the Mini-imagenet dataset, One { normal + reduction } cell is trained 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16 .

Original: For the base-searcher, we use the vanilla SGD to optimize the network weights w mi and architecture parameter θ mi with inner learning rate α inner = 0 .
Revised: For the base-searcher, we use the vanilla SGD to optimize the network weights w mi and architecture parameter θ mi with inner learning rate α inner = 0 .

Original: 1 and β inner = 30 .
Revised: 1 and β inner = 30 .

Original: The inner step M is set as 5 for the trade-off between accuracy and time.
Revised: The inner step M is set as 5 for the trade-off between accuracy and time.

Original: For the metasearcher, we use the Adam Kingma & Ba (2014) to optimize the meta-architecture (cid:101) θ and network weights (cid:101) w with outer learning rate α outer = 
Revised: For the metasearcher, we use the Adam (Kingma & Ba, 2014) to optimize the meta-architecture (cid:101) θ and network weights (cid:101) w with outer learning rate α outer = 10 − 3 and β outer =

Original: 10 − 3and β outer = 10 − 3 .
Revised: 10 − 3 .

Original: All search and evaluation experiments are performed using NVIDIA P40 GPUs.
Revised: All search and evaluation experiments are performed using NVIDIA P40 GPUs.

Original: The whole search process takes about 2 GPU days.
Revised: The whole search process takes about 2 GPU days.

Original: In addition, we also conduct Auto-MAML experiments where all tasks share the same searched architecture.
Revised: In addition, we also conduct Auto-MAML experiments where all tasks share the same searched architecture.

Original: Auto-MAML is a special case of S2 of Figure 1, where all tasks share the same architecture searched with a meta-learning method.
Revised: Auto-MAML is a special case of S2 of Figure 1, where all tasks share the same architecture searched with a meta-learning method.

Original: In the practical algorithm, it is similar to T-NAS, which is behaved as removing the update for θ in the meta-searcher stage.
Revised: In the practical algorithm, it is similar to T-NAS, which is behaved as removing the update for θ in the meta-searcher stage.

Original: However, in Auto-MAML, we can divide the whole dataset into two splits for the updates of θ and (cid:101) w following the recent gradientbased NAS methods Pham et al.
Revised: However, in Auto-MAML, we can divide the whole dataset into two splits for the updates of θ and (cid:101) w following the recent gradientbased NAS methods (Pham et al., 2018;

Original: ; Liu et al. (2018b).
Revised: Liu et al., 2018b).

Original: Here, the D meta-train is divided into two independent splits D train-split1 and D train-split2 with 1 : 1 .
Revised: Here, the D meta-train is divided into two independent splits D train-split1 and D train-split2 with 1 : 1 .

Original: The speciﬁc algorithm for meta-train and meta-test and searched architecture structure can be found in the supplementary material.
Revised: The speciﬁc algorithm for meta-train and meta-test and searched architecture structure can be found in the supplementary material.

Original: To show the transferability of meta-architecture, we visualize the (encoding of) architecture θ searched with Auto-MAML, meta-architecture (cid:101) θ searched with T-NAS, and transferred architecture θ t for a speciﬁc task T t in Figure 2.
Revised: To show the transferability of meta-architecture, we visualize the (encoding of) architecture θ searched with Auto-MAML, meta-architecture (cid:101) θ searched with T-NAS, and transferred architecture θ t for a speciﬁc task T t in Figure 2.

Original: It is worth noting that the architecture encoding matrix ( (cid:101) θ normal , (cid:101) θ reduce ) searched with T-NAS is smoother than that with Auto-MAML, which implies that ( (cid:101) θ normal , (cid:101) θ reduce ) is easier to adapt to the speciﬁc task ( (cid:101)
Revised: It is worth noting that the architecture encoding matrix ( (cid:101) θ normal , (cid:101) θ reduce ) searched with T-NAS is smoother than that with Auto-MAML, which implies that ( (cid:101) θ normal , (cid:101) θ reduce ) is easier to adapt to the speciﬁc task ( (cid:101)

Original: θ →
Revised: θ →

Original: θ t ) than Auto-MAML, thus the metaarchitecture searched with T-NAS is more ﬂexible.
Revised: θ t ) than Auto-MAML, thus the metaarchitecture searched with T-NAS is more ﬂexible.

Original: 5.2. A RCHITECTURE E VALUATION .
Revised: 5.2. A RCHITECTURE E VALUATION .

Original: After getting the architecture structure θ ∗ i for task T
Revised: After getting the architecture structure θ ∗ i for task T

Original: i , we evaluate θ ∗ i by training it from scratch.
Revised: i , we evaluate θ ∗ i by training it from scratch.

Original: In architecture evaluation, we train the task-speciﬁc architecture 20 epochs with 15000 independent tasks for each epoch.
Revised: In architecture evaluation, we train the task-speciﬁc architecture 20 epochs with 15000 independent tasks for each epoch.

Original: Note that different from Liu et al.
Revised: Note that different from Liu et al.

Original: (2018b), we directly use the searched network structure to evaluate performance without any modiﬁcation ( e.g. , the number of channel or layers).
Revised: (2018b), we directly use the searched network structure to evaluate performance without any modiﬁcation ( e.g. , the number of channels or layers).

Original: We optimize the network weights w mi with α inner = 0 .
Revised: We optimize the network weights w mi with α inner = 0 .

Original: 1 and M = 5 .
Revised: 1 and M = 5 .

Original: We use Adam Kingma & Ba (2014) to optimize the meta-weights (cid:101) w with outer learning rate α outer = 10 − 3 .
Revised: We use Adam (Kingma & Ba, 2014) to optimize the meta-weights (cid:101) w with outer learning rate α outer = 10 − 3 .

Original: The experimental results on Omniglot, Mini-Imagenet and FC100 are shown in Table.
Revised: The experimental results on Omniglot, Mini-Imagenet and FC100 are shown in Table.

Original: 2, Table.
Revised: 2, Table.

Original: 3 and Table.
Revised: 3 and Table.

Original: 4, respectively, where T-NAS is based on ﬁrst-order MAML.
Revised: 4, respectively, where T-NAS is based on ﬁrst-order MAML.

Original: Speciﬁcally, T-NAS outperforms
Revised: Speciﬁcally, T-NAS outperforms

Original: MAML and Auto-MAML (52.84% vs. 48.70%, 51.23%), which validates the advantage of T-NAS.
Revised: MAML and Auto-MAML (52.84% vs. 48.70%, 51.23%), which validates the advantage of T-NAS.

Original:  It also achieves better performance than other architecture transfer methods ( e.g. , BASE Shaw et al.
Revised: [Table] [Table] It also achieves better performance than other architecture transfer methods ( e.g. , BASE (Shaw et al., 2018)).

Original: Actually, since the advantage of T-NAS is that the meta-architecture could adapt to a new task rather than using a ﬁxed architecture like MAML and Auto-MAML, it usually have an addi- [Table] [Table] tional time cost for the adaption.
Revised: Actually, since the advantage of T-NAS is that the meta-architecture could adapt to a new task rather than using a ﬁxed architecture like MAML and Auto-MAML, it usually has an additional time cost for the adaption.

Original: Usually, the adaptation procedure costs about 1.5 second (1-shot) and 7.8 seconds (5-shot), which is negligible compared with the improvement of accuracy.
Revised: Usually, the adaptation procedure costs about 1.5 seconds (1-shot) and 7.8 seconds (5-shot), which is negligible compared with the improvement of accuracy.

Original: Moreover, we can also see that T-NAS++, which is an improved version of T-NAS described in Sec.4.2, achieves the best performance among all the baselines.
Revised: Moreover, we can also see that T-NAS++, which is an improved version of T-NAS described in Sec.4.2, achieves the best performance among all the baselines.

Original: 5. T-NAS FOR S UPERVISED L EARNING
Revised: 5. T-NAS FOR S UPERVISED L EARNING

Original: Besides few-shot learning classiﬁcation [Table] we also conduct experiments on Mini-Imagenet for general supervised learning.
Revised: Besides few-shot learning classiﬁcation, we also conduct experiments on Mini-Imagenet for general supervised learning.

Original: Different from few-shot learning, the architecture can be searched and trained for each task due to the sufﬁcient samples, which can be regarded as S1 in Figure 1.
Revised: Different from few-shot learning, the architecture can be searched and trained for each task due to the sufﬁcient samples, which can be regarded as S1 in Figure 1.

Original: Due to the lack of baselines in the supervised learning setting, we choose 10 tasks with 200-shot 50-query 10-way for each task based on the Mini-Imagenet dataset for meaningful experiments.
Revised: Due to the lack of baselines in the supervised learning setting, we choose 10 tasks with 200-shot 50-query 10-way for each task based on the Mini-Imagenet dataset for meaningful experiments.

Original: In the experiments of supervised learning, we follow the same setting as few-shot learning for transferable architecture search.
Revised: In the experiments of supervised learning, we follow the same setting as few-shot learning for transferable architecture search.

Original: The difference is that we can train each task independently from scratch in architecture evaluation.
Revised: The difference is that we can train each task independently from scratch in architecture evaluation.

Original: For 10 tasks in supervised learning, we train the task-speciﬁc architecture for 200 epochs with cosine schedule, where the initial learning rate is 0 . 05 .
Revised: For 10 tasks in supervised learning, we train the task-speciﬁc architecture for 200 epochs with cosine schedule, where the initial learning rate is 0 . 05 .

Original: We use the SGD with momentum 0 .
Revised: We use the SGD with momentum 0 .

Original: 9 to optimize the network weights and crop the original image and ﬂip randomly for data argumentation.
Revised: 9 to optimize the network weights and crop the original image and ﬂip randomly for data argumentation.

Original:  The experimental results in the supervised learning setting are shown in Table.
Revised: [Table] [Table] The experimental results in the supervised learning setting are shown in Table.

Original: In S1, we search the architecture for each of 10 tasks from scratch and evaluate them.
Revised: In S1, we search the architecture for each of 10 tasks from scratch and evaluate them.

Original: For S2, we directly use ﬁve architectures searched respectively in ﬁve different tasks (sampled with 200-shot 50-query 10-way for each task in the meta-train dataset) for the evaluation in 10 tasks.
Revised: For S2, we directly use ﬁve architectures searched respectively in ﬁve different tasks (sampled with 200-shot 50-query 10-way for each task in the meta-train dataset) for the evaluation in 10 tasks.

Original: For a fair comparison, we also pick ﬁve architectures randomly from search space for each task, evaluate them in the speciﬁc task, and report their average results.
Revised: For a fair comparison, we also pick ﬁve architectures randomly from search space for each task, evaluate them in the speciﬁc task [Table] and report their average results.

Original: It is worth noting that randomly generating architectures or directly using the prepared architectures searched in other tasks do not need searching time.
Revised: It is worth noting that randomly generating architectures or directly using the prepared architectures searched in other tasks do not need searching time.

Original: Thus, the time of Random and Method2 in Table.
Revised: Thus, the time of Random and Method2 in Table.

Original: 5 is not applicable.
Revised: 5 is not applicable.

Original: Our T-NAS can learn a meta- [Table] architecture (cid:101) θ and get the task-speciﬁc architecture by only updating several steps from (cid:101) θ instead of shared architecture.
Revised: Our T-NAS can learn a metaarchitecture (cid:101) θ and get the task-speciﬁc architecture by only updating several steps from (cid:101) θ instead of shared architecture.

Original: Thus, T-NAS obtains better performance than random architectures and S (64.23% vs. 61.20%, 62.99%).
Revised: Thus, T-NAS obtains better performance than random architectures and S (64.23% vs. 61.20%, 62.99%).

Original: In addition, T-NAS achievesthe competitive performance with S but with 50x less time cost (5 min vs. 266 min).
Revised: In addition, T-NAS achieves competitive performance with S1 but with 50x less time cost (5 min vs. 266 min).

Original: The performance of S1 is superior to that of T-NAS slightly is because S1 directly searches network architecture for different tasks from scratch, which is laborious as well as time-consuming.
Revised: The performance of S1 is superior to that of T-NAS slightly is because S1 directly searches network architecture for different tasks from scratch, which is laborious as well as time-consuming.

Original: On the contrary, T-NAS can adapt to different tasks quickly by ﬁnding a good initial point (cid:101) θ , which avoids laborious searching for many tasks and saves a lot of time.
Revised: On the contrary, T-NAS can adapt to different tasks quickly by ﬁnding a good initial point (cid:101) θ , which avoids laborious searching for many tasks and saves a lot of time.

Original: Finally, it is interesting that although the architectures searched with S1 and those transferred from meta-architecture searched with T-NAS are different for the speciﬁc tasks, their ﬁnal evaluation performance is very close and outperforms that of the random architectures.
Revised: Finally, it is interesting that although the architectures searched with S1 and those transferred from meta-architecture searched with T-NAS are different for the speciﬁc tasks, their ﬁnal evaluation performance is very close and outperforms that of the random architectures.

Original: Such observation implies that some subspaces in architecture search space might be suitable for a speciﬁc task and T-NAS is able to adapt architecture initialized with (cid:101) θ to the subspaces.
Revised: Such observation implies that some subspaces in architecture search space might be suitable for a speciﬁc task and T-NAS is able to adapt architecture initialized with (cid:101) θ to the subspaces.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we focus on the transferability of Neural Architecture Search, that is to say, how to get a suitable architecture for a new task in NAS?
Revised: In this paper, we focus on the transferability of Neural Architecture Search, that is to say, how to get a suitable architecture for a new task in NAS?

Original: The two naive solutions are either inefﬁcient or not optimal.
Revised: The two simple solutions are either inefﬁcient or not optimal.

Original: To tackle this problem, we propose a novel Transferable Neural Architecture Search (T-NAS) for fast adaptation of architectures.
Revised: To tackle this problem, we propose a novel Transferable Neural Architecture Search (T-NAS) for fast adaptation of architectures.

Original: Speciﬁcally, T-NAS learns a meta-architecture that is able to adapt to a new task easily and quickly through a few gradient steps, which is more ﬂexible than the existing NAS methods.
Revised: Speciﬁcally, T-NAS learns a meta-architecture that is able to adapt to a new task easily and quickly through a few gradient steps, which is more ﬂexible than the existing NAS methods.

Original: In addition, to optimize the whole search network, we propose an efﬁcient ﬁrst-order approximation algorithm.
Revised: In addition, to optimize the whole search network, we propose an efﬁcient ﬁrst-order approximation algorithm.

Original: Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning setting.
Revised: Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning setting.

Original: As for the supervised learning setting, TNAS achieves comparable performance with other baselines but the searching cost is decreased by 50x, which demonstrates the effectiveness of our method.
Revised: As for the supervised learning setting, TNAS achieves comparable performance with other baselines but the searching cost is decreased by 50x, which demonstrates the effectiveness of our method.

Original: For future work, we can study the transferability of NAS for those tasks from different task distributions, where some transfer learning methods might be helpful.
Revised: For future work, we can study the transferability of NAS for those tasks from different task distributions, where some transfer learning methods might be helpful.

Original: We hope that this work can provide some insights on the transferability of NAS, which might potentially beneﬁt the read-world applications. 
Revised: We hope that this work can provide some insights on the transferability of NAS, which might potentially beneﬁt the real-world applications. Acknowledgement.

Revised: The work is supported by the National Key R&D Program of China (2018AAA0100704) and the National Natural Science Foundation of China (NSFC) under Grants

Revised: No. 61932020.

Revised: We would like to thank Jiaxing Wang for some meaningful discussions

Original: A T HE E XPERIMENTS OF A UTO -MAML
Revised: A T HE E XPERIMENTS OF A UTO -MAML

Original: In Auto-MAML, we search a good network architecture for MAML.
Revised: In Auto-MAML we search a good network architecture for MAML.

Original: In fact, Auto-MAML is a special case of Method2 of Figure 1 in this paper, where all tasks share the same architecture searched with a meta-learning method.
Revised: In fact Auto-MAML is a special case of Method2 of Figure 1 in this paper where all tasks share the same architecture searched with a meta-learning method.

Original: In the practical algorithm, it is similar to T-NAS, which is behaved as removing the update for θ in the meta-searcher stage.
Revised: In the practical algorithm it is similar to T-NAS which is behaved as removing the update for θ in the meta-searcher stage.

Original: However, in Auto-MAML, we can divide the whole dataset into two splits for the updates of θ and (cid:101) w following the recent gradient-based NAS methods Pham et al.
Revised: However, in Auto-MAML we can divide the whole dataset into two splits for the updates of θ and (cid:101) w following the recent gradient-based NAS methods (Pham et al., 2018;

Original: ; Liu et al. (2018b).
Revised: Liu et al., 2018b).

Original: Here, the D meta-train is divided into two independent splits D train-split1 and D train-split2 with 1 : 1 .
Revised: Here, the D meta-train is divided into two independent splits D train-split1 and D train-split2 with 1 : 1 .

Original: The speciﬁc algorithm for meta-train and meta-test are shown in Alg. 
Revised: The speciﬁc algorithm for meta-train and meta-test is shown in Alg. 3.

Original: We follow the same deﬁnition for architecture search as T-NAS
Revised: We follow the same deﬁnition for architecture search as T-NAS

Original: and we also use one { normal + reduction } cell for Auto-MAML.
Revised: and we also use one { normal + reduction } cell for Auto-MAML.

Original: The searched architecture θ ∗ is shared by all tasks.
Revised: The searched architecture θ ∗ is shared by all tasks.

Original: We utilize the two splits of training data for architecture search.
Revised: We utilize the two splits of training data for architecture search.

Original: The search model is trained for 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16 .
Revised: The search model is trained for 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16 .

Original: For base-searcher, we use the vanilla SGD to optimize the network weight w mi with inner learning rate
Revised: For base-searcher, we use the vanilla SGD to optimize the network weight w mi with inner learning rate

Original: α inner = 0 . 01 .
Revised: α inner = 0 . 01 .

Original: The inner step M is set as 5 for the trade-off between accuracy and time.
Revised: The inner step M is set as 5 for the trade-off between accuracy and time.

Original: For the meta-update, we use the
Revised: For the meta-update, we use the

Original: Adam to optimize the network weights w and architecture θ with outer learning rate α outer = 10 −and β
Revised: Adam to optimize the network weights w and architecture θ with outer learning rate α outer = 10 −and β

Original: = 3 × 10 − 4 .
Revised: = 3 × 10 − 4 .

Original: The hyperparameter setting for network evaluation is the same as T-NAS.
Revised: The hyperparameter setting for network evaluation is the same as T-NAS.

Original: Here, we visualize some discrete architecture structure searched with Auto-MAML on the Mini-Imagenet dataset in Figure 3 and Figure 4.[Figure] [Figure] Input
Revised: Here, we visualize some discrete architecture structure searched with Auto-MAML on the Mini-Imagenet dataset in Figure 3 and Figure 4.

Original: Dataset architecture do

Original: Get

Original: Compute standard loss; i datapoints T qi update; end

Original: Sample batch tasks

Original: ; for T do Get T si ;
Revised: [Figure]

Original: Compute ∇ w to standard loss;

Original: Get ; end ; 21 end 22 % : 23 Sample tasks do
Revised: [Figure]

Original: B T ASK -S PECIFIC A RCHITECTURES
Revised: B T ASK -S PECIFIC A RCHITECTURES

Original: The aim of this paper is to learn a transferable architecture that is able to adapt to a new task through a few gradient steps.
Revised: The aim of this paper is to learn a transferable architecture that is able to adapt to a new task through a few gradient steps.

Original: Therefore, it is meaningless that directly decoding for the searched metaarchitecture (cid:101) θ without regard to the speciﬁc tasks.
Revised: Therefore, it is meaningless that directly decoding for the searched metaarchitecture (cid:101) θ without regard to the speciﬁc tasks.

Original: Here, we visualize the (encoding of) transferable architecture (cid:101) θ 3 searched with T-NAS and task-speciﬁc architecture θ 1 , θ 2 , θ 3 in Figure 5.
Revised: Here, we visualize the (encoding of) transferable architecture (cid:101) θ 4 searched with T-NAS and task-speciﬁc architecture θ 1 , θ 2 , θ 3 in Figure 5.

Original: The matrix ( (cid:101) θ normal , (cid:101) θ reduce ) searched with T-NAS is smoother than the task-speciﬁc architecture matrices ( θ inormal , θ ireduce ) , which shows that the meta-architecture is ﬂexible and easy to adapt to these speciﬁc task ( (cid:101) θ → θ 1 , θ 2 , θ 3 ).
Revised: The matrix ( (cid:101) θ normal , (cid:101) θ reduce ) searched with T-NAS is smoother than the task-speciﬁc architecture matrices ( θ inormal , θ ireduce ) , which shows that the meta-architecture is ﬂexible and easy to adapt to these speciﬁc task ( (cid:101) θ → θ 1 , θ 2 , θ 3 ).

Original: C C OMPLETE E XPERIMENTAL C OMPARISON
Revised: C C OMPLETE E XPERIMENTAL C OMPARISON

Original: In this section, we show the complete experimental comparison of our method with those methods using pretrained model in Table 6.
Revised: In this section, we show the complete experimental comparison of our method with those methods using pretrained model in Table 6.

Original: Some methods Oreshkin et al. ; Sun et al. (2019) get better performance by employing more complex network and pretrained model.[Figure]
Revised: Some methods (Oreshkin et al., 2018; Sun et al., 2019) get better performance by employing more complex networks and pretrained model.

Original: [Figure]

Original: [Table]

Original: D P ERFORMANCE C OMPARISON ON CIFAR-10 AND I MAGE N ET
Revised: D P ERFORMANCE C OMPARISON ON CIFAR-10 AND I MAGE N ET

Original: To evaluate the transferability of our method [Table] we also conduct the experiments on CIFAR-10 and
Revised: To evaluate the transferability of our method, we also conduct the experiments on CIFAR-10 and

Original: ImageNet.
Revised: ImageNet.

Original: Firstly, we construct a larger dataset from ImageNet to learn the meta-architecture [Table] and then adapt the meta-architecture on CIFAR-10 to decode the ﬁnal architecture.
Revised: Firstly, we construct a larger dataset from ImageNet to learn the meta-architecture, and then adapt the meta-architecture on CIFAR-10 to decode the ﬁnal architecture.

Original: We test the performance of the ﬁnal architecture on CIFAR-10 and ImageNet and report the performance in Tableand Table 8.
Revised: We test the performance of the ﬁnal architecture on CIFAR-10 and ImageNet and report the performance in Table 7 and Table 8.

Original: From these two tables [Table] we can see that the learned meta-architecture from T-NAS can quickly adapt to new tasks and achieve favorable performance.
Revised: From these two tables, we can see that the learned meta-architecture from T-NAS can quickly adapt to new tasks and achieve favorable performance.

Original: For example, given the learned metaarchitecture from T-NAS, it only takes 0.042 GPU days to derive an architecture which achieves test 
Revised: For example, given the learned meta-architecture from T-NAS, it only takes 0.042 GPU days to derive an architecture that achieves test error of 2.98% on CIFAR-10 and 27.2% on ImageNet.

Original: [Table]
Revised: In contrast, searching for an architecture that achieves similar performance from scratch on CIFAR-10 by DARTS (ﬁrst order) would cost do Acc i in T qi ; 27 end 28 Return architecture θ ∗ according to the best average accuracy of { Acc } .

Original: [Table]
Revised: [Table]

Original: error of 2.98% on CIFAR-10 and 27.2% on ImageNet. In contrast, searching for an architecture that achieves similar performance from scratch on CIFAR-10 by DARTS (ﬁrst order) would cost 1.5 days, which is about 36 times longer than that of T-NAS.
Revised: 1.5 days, which is about 36 times longer than that of T-NAS.

Original: This result conﬁrms the advantage of
Revised: This result conﬁrms the advantage of

Original: T-NAS and also indicates that it is possible to apply T-NAS to practical scenarios.
Revised: T-NAS and also indicates that it is possible to apply T-NAS to practical scenarios [Table] [Figure] [Table]

Revised: [Table]