Original: SUMNAS: S UPERNET WITH U NBIASED M ETAF EATURES FOR N EURAL A RCHITECTURE S EARCH
Revised: SUMNAS: S UPERNET WITH U NBIASED M ETAF EATURES FOR N EURAL A RCHITECTURE S EARCH

Original: Kim

Original: NAVER AI Lab, NAVER Corporation;

Original: NAVER CLOVA, NAVER Corporation kim

Original: Byung

Original: A BSTRACT
Revised: A BSTRACT

Original: One-shot Neural Architecture Search (NAS) usually constructs an overparameterized network, which we call a supernet, and typically adopts sharing parameters among the sub-models to improve computational efﬁciency.
Revised: One-shot Neural Architecture Search (NAS) usually constructs an overparameterized network, which we call a supernet, and typically adopts sharing parameters among the sub-models to improve computational efﬁciency.

Original: Oneshot NAS often repeatedly samples sub-models from the supernet and trains them to optimize the shared parameters.
Revised: Oneshot NAS often repeatedly samples sub-models from the supernet and trains them to optimize the shared parameters.

Original: However, this training strategy suffers from multi-model forgetting.
Revised: However, this training strategy suffers from multi-model forgetting.

Original: Training a sampled sub-model overrides the previous knowledge learned by the other sub-models, resulting in an unfair performance evaluation between the sub-models.
Revised: Training a sampled sub-model overrides the previous knowledge learned by the other sub-models, resulting in an unfair performance evaluation between the sub-models.

Original: We propose Supernet with Unbiased MetaFeatures for Neural Architecture Search (SUMNAS), a supernet learning strategy based on meta-learning to tackle the knowledge forgetting issue.
Revised: We propose Supernet with Unbiased MetaFeatures for Neural Architecture Search (SUMNAS), a supernet learning strategy based on meta-learning to tackle the knowledge forgetting issue.

Original: During the training phase, we explicitly address the multi-model forgetting problem and help the supernet learn unbiased meta-features, independent from the sampled submodels.
Revised: During the training phase, we explicitly address the multi-model forgetting problem and help the supernet learn unbiased meta-features, independent from the sampled submodels.

Original: Once training is over, sub-models can be instantly compared to get the overall ranking or the best sub-model.
Revised: Once training is over, sub-models can be instantly compared to get the overall ranking or the best sub-model.

Original: Our evaluation on the NAS-Benchand MobileNet-based search space demonstrate that SUMNAS shows improved ranking ability and ﬁnds architectures whose performance is on par with existing state-of-the-art NAS algorithms.
Revised: Our evaluation on the NAS-Benchand MobileNet-based search space demonstrate that SUMNAS shows improved ranking ability and ﬁnds architectures whose performance is on par with existing state-of-the-art NAS algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent Neural Architecture Search (NAS) algorithms often train an over-parameterized network called a supernet to obtain supreme sub-models by sharing parameters rapidly.
Revised: Recent Neural Architecture Search (NAS) algorithms often train an over-parameterized network called a supernet to obtain supreme sub-models by sharing parameters rapidly.

Original: In this process, the shared parameters are generally trained to reach a state where better architecture discovery is possible through comparing the sub-models.
Revised: In this process, the shared parameters are generally trained to reach a state where better architecture discovery is possible through comparing the sub-models.

Original: Supernet training is usually conducted by sampling one or more sub-models and updating them with their gradients.
Revised: Supernet training is usually conducted by sampling one or more sub-models and updating them with their gradients.

Original: Since parameters are shared, learning not to be biased towards a particular sub-model is crucial in training a supernet.
Revised: Since parameters are shared, learning not to be biased towards a particular sub-model is crucial in training a supernet.

Original: However, a parameter update due to sampling a speciﬁc sub-model often causes a bias for the sampled sub-model; the supernet forgets the knowledge learned from the previous sampling because of the biased update, and other sub-models share the same parameters can be degraded and underrated.
Revised: However, a parameter update due to sampling a speciﬁc sub-model often causes a bias for the sampled sub-model; the supernet forgets the knowledge learned from the previous sampling because of the biased update, and other sub-models share the same parameters can be degraded and underrated.

Original: Benyahia et al.
Revised: Benyahia et al.

Original: (2019) referred to this phenomenon as multi-model forgetting , and many current architecture search strategies have been designed without considering this phenomenon.
Revised: (2019) referred to this phenomenon as multi-model forgetting , and many current architecture search strategies have been designed without considering this phenomenon.

Original: To mitigate this problem, it is important to make the supernet parameters learn unbiased features that are globally suitable for the sub-models.
Revised: To mitigate this problem, it is important to make the supernet parameters learn unbiased features that are globally suitable for the sub-models.

Original: Therefore, we propose a meta-learning-based approach that enables the supernet to learn unbiased meta-features.
Revised: Therefore, we propose a meta-learning-based approach that enables the supernet to learn unbiased meta-features.

Original: We adopt model-agnostic meta-learning (MAML)
Revised: We adopt model-agnostic meta-learning (MAML)

Original: (Finn et al., 2017; Nichol et al., 2018) during supernet training.
Revised: (Finn et al., 2017; Nichol et al., 2018) during supernet training.

Original: MAML is designed to learn the meta-features suitable for multiple tasks, and the trained meta-parameters are then quickly adapted to an unseen task through few-shot learning while reusing the learned meta-features.
Revised: MAML is designed to learn the meta-features suitable for multiple tasks, and the trained meta-parameters are then quickly adapted to an unseen task through few-shot learning while reusing the learned meta-features.

Original: We apply this concept of MAML to supernet training by assuming the multiple tasks in MAML as learning for multiple sub-models in a supernet.
Revised: We apply this concept of MAML to supernet training by assuming the multiple tasks in MAML as learning for multiple sub-models in a supernet.

Original: We call the proposed supernet training algorithm Supernet with Unbiased Meta-Features for Neural
Revised: We call the proposed supernet training algorithm Supernet with Unbiased Meta-Features for Neural

Original: Architecture Search (SUMNAS), given that it introduces the meta-learning principle to the multimodel forgetting problem.
Revised: Architecture Search (SUMNAS), given that it introduces the meta-learning principle to the multimodel forgetting problem.

Original: SUMNAS consists of two stages: supernet training and heuristic search with sub-model evaluation.
Revised: SUMNAS consists of two stages: supernet training and heuristic search with sub-model evaluation.

Original: In the supernet training stage, the parameters learn the meta-features for multiple sub-models, which can be considered as MAML’s training meta-features for multiple tasks.
Revised: In the supernet training stage, the parameters learn the meta-features for multiple sub-models, which can be considered as MAML’s training meta-features for multiple tasks.

Original: The meta-parameters obtained from the supernet training can then be directly used without additional training for comparison between sub-models during the evaluation phase.
Revised: The meta-parameters obtained from the supernet training can then be directly used without additional training for comparison between sub-models during the evaluation phase.

Original: To the best of our knowledge, this work is the ﬁrst to utilize the meta-learning capability—learning unbiased meta-features—for accurately ranking sub-models in a supernet.
Revised: To the best of our knowledge, this work is the ﬁrst to utilize the meta-learning capability—learning unbiased meta-features—for accurately ranking sub-models in a supernet.

Original: There have been approaches (Shaw et al., 2019; Lian et al., 2020; Wang et al., 2020; Elsken et al., 2020) that adopt meta-learning principles to train a supernet using various tasks and search for the best architecture for an unseen task with a few data instances, which is the few-shot learning variant of NAS.
Revised: There have been approaches (Shaw et al., 2019; Lian et al., 2020; Wang et al., 2020; Elsken et al., 2020) that adopt meta-learning principles to train a supernet using various tasks and search for the best architecture for an unseen task with a few data instances, which is the few-shot learning variant of NAS.

Original: They train the model parameters such that the models are robust to unseen datasets.
Revised: They train the model parameters such that the models are robust to unseen datasets.

Original: On the other hand, we take a fundamentally different view of a task.
Revised: On the other hand, we take a fundamentally different view of a task.

Original: A task usually refers to a dataset, but we show in
Revised: A task usually refers to a dataset, but we show in

Original: Section 3 that each sub-model within a search space can be regarded as a separate “task” that the supernet has to adapt to.
Revised: Section 3 that each sub-model within a search space can be regarded as a separate “task” that the supernet has to adapt to.

Original: We therefore apply meta-learning principles to make the model parameters robust to several different sub-models.
Revised: We therefore apply meta-learning principles to make the model parameters robust to several different sub-models.

Original: We evaluate SUMNAS with qualitative and quantitative experiments on the CIFAR10 and ImageNet datasets.
Revised: We evaluate SUMNAS with qualitative and quantitative experiments on the CIFAR10 and ImageNet datasets.

Original: We show that the architecture rankings SUMNAS predicts have a stronger correlation with the accurate rankings as compared to prior NAS algorithms that use a supernet.
Revised: We show that the architecture rankings SUMNAS predicts have a stronger correlation with the accurate rankings as compared to prior NAS algorithms that use a supernet.

Original: Besides, we observe better architecture search performance when an existing search methodology is applied to SUMNAS and show that SUMNAS parameters properly learn meta-features by investigating the performances of the sub-models.
Revised: Besides, we observe better architecture search performance when an existing search methodology is applied to SUMNAS and show that SUMNAS parameters properly learn meta-features by investigating the performances of the sub-models.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: One-shot NAS:
Revised: One-shot NAS:

Original: In the context of a supernet that shares parameters with its sub-models, the majority of NAS approaches use the supernet as a proxy to indirectly predict the performance of sub-models.
Revised: In the context of a supernet that shares parameters with its sub-models, the majority of NAS approaches use the supernet as a proxy to indirectly predict the performance of sub-models.

Original: With the performance oracle, differentiable techniques (Liu et al., 2018; Cai et al., 2019), reinforcement learning (Pham et al., 2018), and heuristic methods (Guo et al., 2019; Chu et al., 2019) are used to determine the best architecture.
Revised: With the performance oracle, differentiable techniques (Liu et al., 2018; Cai et al., 2019), reinforcement learning (Pham et al., 2018), and heuristic methods (Guo et al., 2019; Chu et al., 2019) are used to determine the best architecture.

Original: Although the weight sharing has signiﬁcantly improved search speed, it is not easy to provide accurate performance indicators when using a supernet as a proxy for sub-model performance.
Revised: Although the weight sharing has signiﬁcantly improved search speed, it is not easy to provide accurate performance indicators when using a supernet as a proxy for sub-model performance.

Original: One of the most difﬁcult challenges to obtaining an accurate performance oracle is overriding knowledge learned by previously sampled sub-models.
Revised: One of the most difﬁcult challenges to obtaining an accurate performance oracle is overriding knowledge learned by previously sampled sub-models.

Original: Researchers have recently introduced the practice of repeatedly updating parameters through one or more sampled sub-models (Guo et al., 2019;
Revised: Researchers have recently introduced the practice of repeatedly updating parameters through one or more sampled sub-models (Guo et al., 2019;

Original: Chu et al., 2019; Li et al., 2020).
Revised: Chu et al., 2019; Li et al., 2020).

Original: However, Benyahia et al.
Revised: However, Benyahia et al.

Original: (2019) have shown that catastrophic forgetting occurs in the sequential learning of sampled sub-models.
Revised: (2019) have shown that catastrophic forgetting occurs in the sequential learning of sampled sub-models.

Original: Catastrophic forgetting is a phenomenon where a neural network forgets previously learned knowledge when learning new information.
Revised: Catastrophic forgetting is a phenomenon where a neural network forgets previously learned knowledge when learning new information.

Original: In sampling-based supernet training, repetitive sub-model sampling and training keep introducing new information into shared parameters.
Revised: In sampling-based supernet training, repetitive sub-model sampling and training keep introducing new information into shared parameters.

Original: The knowledge learned with the previously sampled sub-models is forgotten due to training subsequently sampled sub-models without considering former training.
Revised: The knowledge learned with the previously sampled sub-models is forgotten due to training subsequently sampled sub-models without considering former training.

Original: Knowledge overriding causes the predicted performance of the architectures to ﬂuctuate depending on the frequency or the sequence of the sampled architectures, which leads to inaccurately predicted rankings.
Revised: Knowledge overriding causes the predicted performance of the architectures to ﬂuctuate depending on the frequency or the sequence of the sampled architectures, which leads to inaccurately predicted rankings.

Original: The forgetting problem in the stochastic training of shared supernet parameters is termed multi-model forgetting .
Revised: The forgetting problem in the stochastic training of shared supernet parameters is termed multi-model forgetting .

Original: To alleviate this problem, Benyahia et al.
Revised: To alleviate this problem, Benyahia et al.

Original: and Zhang et al.
Revised: and Zhang et al.

Original: (2020) add a regularizer to prevent weights from deviating too far from the posterior distribution learned from previously trained sub-models.
Revised: (2020) add a regularizer to prevent weights from deviating too far from the posterior distribution learned from previously trained sub-models.

Original: Model-agnostic meta-learning: Model-agnostic meta-learning (MAML) is a process of learning initialization for few-shot learning of an unseen task.
Revised: Model-agnostic meta-learning: Model-agnostic meta-learning (MAML) is a process of learning initialization for few-shot learning of an unseen task.

Original: Let the i th task be T
Revised: Let the i th task be T

Original: i and θ (cid:48)
Revised: i and θ (cid:48)

Original: i be suitable to model parameters for T i .
Revised: i be suitable to model parameters for T i .

Original: MAML provides an objective function to obtain optimal meta-parameters by minimizing the loss function for each task T i :
Revised: MAML provides an objective function to obtain optimal meta-parameters by minimizing the loss function for each task T i :

Original: [Equation]
Revised: [Equation]

Original:  where f is the network function consisting of θ (cid:48) , and L is the loss function for each task T i .
Revised: [Figure] where f is the network function consisting of θ (cid:48) , and L is the loss function for each task T i .

Original: The
Revised: The

Original: L term in equation 1 allows the θ ∗ i calculated by few-shot learning with initialization θ to learn a[Figure] suitable feature for each task T i , and θ meta minimizes the expected loss corresponding to the tasks.
Revised: L term in equation 1 allows the θ ∗ i calculated by few-shot learning with initialization θ to learn a suitable feature for each task T i , and θ meta minimizes the expected loss corresponding to the tasks.

Original: As a consequence, the learned meta feature is generalized across tasks, allowing for fast learning of new tasks.
Revised: As a consequence, the learned meta feature is generalized across tasks, allowing for fast learning of new tasks.

Original: Finn et al.
Revised: Finn et al.

Original: (2017) solve the objective function by repeatedly sampling tasks and updating parameters with gradient descents.
Revised: (2017) solve the objective function by repeatedly sampling tasks and updating parameters with gradient descents.

Original: They also suggest a ﬁrst-order approximation to avoid the computation of second-order derivatives.
Revised: They also suggest a ﬁrst-order approximation to avoid the computation of second-order derivatives.

Original: Nichol et al.
Revised: Nichol et al.

Original: (2018) propose a variation of gradient descent for metalearning, which is called Reptile.
Revised: (2018) propose a variation of gradient descent for metalearning, which is called Reptile.

Original: The ﬁrst-order MAML carries out task optimization using only the last gradient calculated by the inner loop step, ignoring the second-order derivative.
Revised: The ﬁrst-order MAML carries out task optimization using only the last gradient calculated by the inner loop step, ignoring the second-order derivative.

Original: However,
Revised: However,

Original: Reptile uses the average of the gradients calculated over multiple inner loop steps to update the parameters.
Revised: Reptile uses the average of the gradients calculated over multiple inner loop steps to update the parameters.

Original: These averaged values reﬂect the generalization of the gradients for the data.
Revised: These averaged values reﬂect the generalization of the gradients for the data.

Original:  S UPERNET WITH U NBIASED M ETA -F EATURES FOR N EURAL A RCHITECTURE S EARCH
Revised:  S UPERNET WITH U NBIASED M ETA -F EATURES FOR N EURAL A RCHITECTURE S EARCH

Original: To tackle the multi-model forgetting problem, we suggest a new supernet training strategy based on meta-learning, Supernet with Unbiased Meta-Features for Neural Architecture Search (SUMNAS).
Revised: To tackle the multi-model forgetting problem, we suggest a new supernet training strategy based on meta-learning, Supernet with Unbiased Meta-Features for Neural Architecture Search (SUMNAS).

Original: As mentioned, learning unbiased features that are suitable for sub-models in the supernet is essential to alleviate the knowledge overriding.
Revised: As mentioned, learning unbiased features that are suitable for sub-models in the supernet is essential to alleviate the knowledge overriding.

Original: We take the idea of MAML, which learns meta-features suitable for multiple tasks, and apply it to supernet training so that SUMNAS learns such unbiased meta-features of sub-models.
Revised: We take the idea of MAML, which learns meta-features suitable for multiple tasks, and apply it to supernet training so that SUMNAS learns such unbiased meta-features of sub-models.

Original: In this section, we describe our approach and explain how it can mitigate the multi-model forgetting problem.
Revised: In this section, we describe our approach and explain how it can mitigate the multi-model forgetting problem.

Original: We then describe our supernet training algorithm.
Revised: We then describe our supernet training algorithm.

Original: 3. M ETA -F EATURE L EARNING
Revised: 3. M ETA -F EATURE L EARNING

Original: The approach we introduce to improve robustness against the multi-model forgetting problem is learning unbiased meta-features.
Revised: The approach we introduce to improve robustness against the multi-model forgetting problem is learning unbiased meta-features.

Original: SUMNAS has two stages: supernet training and heuristic search with sub-model evaluation, which are similar to existing sampling-based NAS algorithms (Chu et al., 2019;
Revised: SUMNAS has two stages: supernet training and heuristic search with sub-model evaluation, which are similar to existing sampling-based NAS algorithms (Chu et al., 2019;

Original: Guo et al., 2019; Li et al., 2020), as presented in Figure 1.
Revised: Guo et al., 2019; Li et al., 2020), as presented in Figure 1.

Original: First, SUMNAS trains parameters to learn meta-features (Figure 1 [a]), which are not biased to a speciﬁc sub-model and are suitable for sub-models, so training them does not overwrite previously trained knowledge.
Revised: First, SUMNAS trains parameters to learn meta-features (Figure 1 [a]), which are not biased to a speciﬁc sub-model and are suitable for sub-models, so training them does not overwrite previously trained knowledge.

Original: Afterward, the parameters are used to evaluate a speciﬁc sub-model during the search (Figure 1 [b]).
Revised: Afterward, the parameters are used to evaluate a speciﬁc sub-model during the search (Figure 1 [b]).

Original: Vanilla sampling-based supernet training optimizes strictly shared supernet parameters on sampled sub-models, and it can be expressed as follows:
Revised: Vanilla sampling-based supernet training optimizes strictly shared supernet parameters on sampled sub-models, and it can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where f is a sub-model sampled from a distribution of sub-models p ( f ) , and T is a given dataset.
Revised: where f is a sub-model sampled from a distribution of sub-models p ( f ) , and T is a given dataset.

Original: The parameter θ is strictly shared among sub-models, and it has to learn model-speciﬁc features of all of the sub-models.
Revised: The parameter θ is strictly shared among sub-models, and it has to learn model-speciﬁc features of all of the sub-models.

Original: The training for one or more sampled sub-models does not consider other ones that have been previously sampled, and simply overrides knowledge acquired from previous training to optimize currently sampled sub-models.
Revised: The training for one or more sampled sub-models does not consider other ones that have been previously sampled, and simply overrides knowledge acquired from previous training to optimize currently sampled sub-models.

Original: In our approach, similar to previous works, parameters are shared during supernet training.
Revised: In our approach, similar to previous works, parameters are shared during supernet training.

Original: However, the parameters are trained to learn unbiased meta-features.
Revised: However, the parameters are trained to learn unbiased meta-features.

Original: Here, the unbiased meta-features make fair comparison possible by providing each sub-model with an appropriate level of optimized parameters that are not overﬁtted for a speciﬁc sub-model.
Revised: Here, the unbiased meta-features make fair comparison possible by providing each sub-model with an appropriate level of optimized parameters that are not overﬁtted for a speciﬁc sub-model.

Original: Our meta-learning approach has been formulated as follows:
Revised: Our meta-learning approach has been formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where A f is an adaptation function optimizing the parameters for the sub-model f with a few inputs.
Revised: where A f is an adaptation function optimizing the parameters for the sub-model f with a few inputs.

Original: With this optimization, parameters naturally learn meta-features.
Revised: With this optimization, parameters naturally learn meta-features.

Original: This idea is based on MAML, which targets few-shot learning of an unseen task using adaptable parameters.
Revised: This idea is based on MAML, which targets few-shot learning of an unseen task using adaptable parameters.

Original: MAML trains parameters to learn meta-features suitable for multiple tasks and uses the meta-parameters to initialize few-shot learning.
Revised: MAML trains parameters to learn meta-features suitable for multiple tasks and uses the meta-parameters to initialize few-shot learning.

Original: Training meta-parameters for a weight-shared supernet is similar to training with MAML.
Revised: Training meta-parameters for a weight-shared supernet is similar to training with MAML.

Original: We map a task used in meta-learning for few-shot learning to a sub-model in NAS using a supernet, as seen by comparing Equation 1 with Equation 3.
Revised: We map a task used in meta-learning for few-shot learning to a sub-model in NAS using a supernet, as seen by comparing Equation 1 with Equation 3.

Original: In other words, the difference is that MAML minimizes the expectation of the loss across tasks, but
Revised: In other words, the difference is that MAML minimizes the expectation of the loss across tasks, but

Original: SUMNAS minimizes the expectation of the loss across sub-models.
Revised: SUMNAS minimizes the expectation of the loss across sub-models.

Original: Interestingly, we notice that learning the meta-features in a supernet can be viewed as MAML of each operator in the supernet.
Revised: Interestingly, we notice that learning the meta-features in a supernet can be viewed as MAML of each operator in the supernet.

Original: We can convert Equation 3 into the following equation:
Revised: We can convert Equation 3 into the following equation:

Original: [Equation]
Revised: [Equation]

Original: where f i : j is a composition of sampled operators from i -th layer to j -th layer.
Revised: where f i : j is a composition of sampled operators from i -th layer to j -th layer.

Original: A sampled sub-model f 0: i − 1 feeds an intermediate representation f 0: i − 1 ( x ) to the operator o .
Revised: A sampled sub-model f 0: i − 1 feeds an intermediate representation f 0: i − 1 ( x ) to the operator o .

Original: There is a distribution of the intermediate representation fed to the operator, and the distribution is determined by a submodel sampled from p ( f ) .
Revised: There is a distribution of the intermediate representation fed to the operator, and the distribution is determined by a submodel sampled from p ( f ) .

Original: The operator also has a target distribution of outputs to learn determined by the sampled sub-model.
Revised: The operator also has a target distribution of outputs to learn determined by the sampled sub-model.

Original: In the MAML, examples fed to a meta-learning model also follow a distribution, and this distribution is a task T i sampled from p ( T ) .
Revised: In the MAML, examples fed to a meta-learning model also follow a distribution, and this distribution is a task T i sampled from p ( T ) .

Original: Thus, inputs and targets of both a multi-task meta-learning model and an operator in a supernet follow sampled distributions.
Revised: Thus, inputs and targets of both a multi-task meta-learning model and an operator in a supernet follow sampled distributions.

Original: We also aim to train an operator that works well on a set of intermediate representations provided by a sampled sub-model, just as MAML learns to work well on a set of examples given by a sampled task.
Revised: We also aim to train an operator that works well on a set of intermediate representations provided by a sampled sub-model, just as MAML learns to work well on a set of examples given by a sampled task.

Original: That is, a model and a task in MAML can be mapped to an operator and a distribution of intermediate representations fed to the operator in supernet training.
Revised: That is, a model and a task in MAML can be mapped to an operator and a distribution of intermediate representations fed to the operator in supernet training.

Original: In other words, we can regard the distribution of intermediate representations as to the task for the operator, although the task for the operator is parameterized and optimizable differently from tasks in MAML.
Revised: In other words, we can regard the distribution of intermediate representations as to the task for the operator, although the task for the operator is parameterized and optimizable differently from tasks in MAML.

Original: This implies that we can use any MAML algorithms without alteration to solve Equation 4.
Revised: This implies that we can use any MAML algorithms without alteration to solve Equation 4.

Original: Moreover, it supports the assumption that the success of the meta-learning algorithms for few-shot learning will be transferred to NAS.
Revised: Moreover, it supports the assumption that the success of the meta-learning algorithms for few-shot learning will be transferred to NAS.

Original: However, supernet training and MAML have a fundamental difference.
Revised: However, supernet training and MAML have a fundamental difference.

Original: MAML targets to train parameters for an unseen task, but supernet training aims to obtain parameters for pre-deﬁned submodels, which are known at train time, and participate in the training process.
Revised: MAML targets to train parameters for an unseen task, but supernet training aims to obtain parameters for pre-deﬁned submodels, which are known at train time, and participate in the training process.

Original: Therefore, after our supernet training, the supernet has knowledge of the sub-models, so additional training, such as ﬁne-tuning for each sub-model is not mandatory.
Revised: Therefore, after our supernet training, the supernet has knowledge of the sub-models, so additional training, such as ﬁne-tuning for each sub-model is not mandatory.

Original: In our empirical results, the additional ﬁtting during evaluation may improve the performance of each sub-model, but the ranking performance stays more or less the same.
Revised: In our empirical results, the additional ﬁtting during evaluation may improve the performance of each sub-model, but the ranking performance stays more or less the same.

Original: 3. S UPERNET T RAINING A LGORITHM
Revised: 3. S UPERNET T RAINING A LGORITHM

Original: In this section, we present our supernet training algorithm SUMNAS (Algorithm 1).
Revised: In this section, we present our supernet training algorithm SUMNAS (Algorithm 1).

Original: SUMNAS repeatedly samples sub-models and learns meta-features through the sampled sub-models with a
Revised: SUMNAS repeatedly samples sub-models and learns meta-features through the sampled sub-models with a

Original: MAML-inspired update function.
Revised: MAML-inspired update function.

Original: Parameters are updated with the aggregated gradients of multiple sub-models that share the same operators, unlike previous supernet training algorithms that immediately update parameters with the gradients of a single sub-model for an operator.
Revised: Parameters are updated with the aggregated gradients of multiple sub-models that share the same operators, unlike previous supernet training algorithms that immediately update parameters with the gradients of a single sub-model for an operator.

Original: 1: Input
Revised: 1: Input

Original: N
Revised: N

Original: , candidate per
Revised: , candidate per

Original: 2: for i ← do 3: do 4: // m paths 5: for l ← 0 to L − 1 do 6:
Revised: 2: for i ← do 3: do 4: // m paths 5: for l ← 0 to L − 1 do 6:

Original: C l, : ← a permutation of the index for the m candidate ops of layer l 7: end for 8: //
Revised: C l, : ← a permutation of the index for the m candidate ops of layer l 7: end for 8: //

Original: Compute the Reptile gradients for the sampled paths 9: for i ← 0 to m − 1 do 10: ˜ θ ←− A k C : ,i ( L , θ , X, (cid:15) 1 ) //
Revised: Compute the Reptile gradients for the sampled paths 9: for i ← 0 to m − 1 do 10: ˜ θ ←− A k C : ,i ( L , θ , X, (cid:15) 1 ) //

Original: Adaptation 11: g ←− g + (cid:15) 

Revised: Adaptation 11: g ←− g + (cid:15)

Original: ( θ − ˜ θ ) //
Revised: ( θ − ˜ θ ) //

Original: Reptile 12: end for 13: // Update the parameters 14: θ ← θ − g 15: g ←16: end for 17: end for
Revised: Reptile 12: end for 13: // Update the parameters 14: θ ← θ − g 15: g ←16: end for 17: end for

Original: Sampling sub-models As suggested by FairNAS (Chu et al., 2019), we keep strict fairness for fair training among operators, so all of the operators in the supernet are sampled the same number of times and updated simultaneously.
Revised: Sampling sub-models As suggested by FairNAS (Chu et al., 2019), we keep strict fairness for fair training among operators, so all of the operators in the supernet are sampled the same number of times and updated simultaneously.

Original: To ensure this, SUMNAS sample m sub-models by sampling the candidate operators without replacement for each layer when we have m candidate operators and iterate the sampled sub-models for gradient computation (line 9–12).
Revised: To ensure this, SUMNAS sample m sub-models by sampling the candidate operators without replacement for each layer when we have m candidate operators and iterate the sampled sub-models for gradient computation (line 9–12).

Original: This sampling process is represented as a permutation of the operators in each layer (line 5–7).
Revised: This sampling process is represented as a permutation of the operators in each layer (line 5–7).

Original: Meta-learning The update function to optimize a single sub-model with stochastic gradient descent (SGD) is
Revised: Meta-learning The update function to optimize a single sub-model with stochastic gradient descent (SGD) is

Original: [Equation]
Revised: [Equation]

Original: where θ represents the parameters of the supernet, L is the loss function, A kf is the adaptation function with k steps and a sub-model f .
Revised: where θ represents the parameters of the supernet, L is the loss function, A kf is the adaptation function with k steps and a sub-model f .

Original: We use SGD for the adaptation function.
Revised: We use SGD for the adaptation function.

Original: Computing the gradients of the loss requires computing the hessian of the loss, which is too intensive.
Revised: Computing the gradients of the loss requires computing the hessian of the loss, which is too intensive.

Original: Several methods can be used to avoid it, and we use the Reptile algorithm
Revised: Several methods can be used to avoid it, and we use the Reptile algorithm

Original: (Nichol et al., 2018) because of its simplicity.
Revised: (Nichol et al., 2018) because of its simplicity.

Original: Reptile uses the difference between the parameters before and after adaptation (line 10–11) instead of the actual gradient.
Revised: Reptile uses the difference between the parameters before and after adaptation (line 10–11) instead of the actual gradient.

Original: The parameter updates occur after computing and aggregating the gradients of all sampled sub-models (line 13–15).
Revised: The parameter updates occur after computing and aggregating the gradients of all sampled sub-models (line 13–15).

Original: 3. S EARCH A LGORITHM
Revised: 3. S EARCH A LGORITHM

Original: Since we have done most of the heavy lifting in the supernet training phase, we can plug in almost any search algorithm for the evaluation phase.
Revised: Since we have done most of the heavy lifting in the supernet training phase, we can plug in almost any search algorithm for the evaluation phase.

Original: Possible search algorithms include, but are not limited to, differentiable architecture search, reinforcement learning, or evolutionary search.
Revised: Possible search algorithms include, but are not limited to, differentiable architecture search, reinforcement learning, or evolutionary search.

Original: For our experiments, we used the simple evolutionary algorithm proposed by Guo et al.
Revised: For our experiments, we used the simple evolutionary algorithm proposed by Guo et al.

Original:  E VALUATION
Revised:  E VALUATION

Original: We evaluate SUMNAS on two search spaces — NAS-Bench-201 (Dong & Yang, 2020) on CIFAR10 (Krizhevsky et al., 2009) and MobileNet blocks on ImageNet (Russakovsky et al., 2015).
Revised: We evaluate SUMNAS on two search spaces — NAS-Bench-201 (Dong & Yang, 2020) on CIFAR10 (Krizhevsky et al., 2009) and MobileNet blocks on ImageNet (Russakovsky et al., 2015).

Original: For both search spaces, we compare sampled architectures’ rankings predicted by SUMNAS with their reference rankings obtained from standalone training to evaluate the ranking ability of SUMNAS.
Revised: For both search spaces, we compare sampled architectures’ rankings predicted by SUMNAS with their reference rankings obtained from standalone training to evaluate the ranking ability of SUMNAS.

Original: We obtain the reference rankings of the sampled architectures for the NAS-Bench-201 search space from the NAS-Bench-201 and those for MobileNet blocks from manual training.
Revised: We obtain the reference rankings of the sampled architectures for the NAS-Bench-201 search space from the NAS-Bench-201 and those for MobileNet blocks from manual training.

Original: We also report the best architectures that SUMNAS found in the search spaces.
Revised: We also report the best architectures that SUMNAS found in the search spaces.

Original: Furthermore, we analyze the sensitivity of SUMNAS with respect to the number of adaptation steps.
Revised: Furthermore, we analyze the sensitivity of SUMNAS with respect to the number of adaptation steps.

Original: [Table]
Revised: [Table]

Original: In the experiments
Revised: In the experiments

Original: [Table]
Revised: [Table]

Original: the methodology we use to obtain the predicted performance ( i
Revised: the methodology we use to obtain the predicted performance ( i

Original: [Table]
Revised: [Table]

Original: e., accuracy) of architecture from supernets is the following: we separately update the running statistics of batch normalization layers for the corresponding sub-model and then evaluate the calibrated sub-model.
Revised: e., accuracy) of architecture from supernets is the following: we separately update the running statistics of batch normalization layers for the corresponding sub-model and then evaluate the calibrated sub-model.

Original: We describe the hyperparameters we used in Appendix D.
Revised: We describe the hyperparameters we used in Appendix D.

Original: We also present the analysis on the sensitivity to learning rates for inner and outer loops in Appendix E.
Revised: We also present the analysis on the sensitivity to learning rates for inner and outer loops in Appendix E.

Original: 4. S EARCH S PACE
Revised: 4. S EARCH S PACE

Original: NAS-Bench-201 We adopt the NAS-Bench-201 search space to evaluate the ranking performance of SUMNAS on CIFAR-10.
Revised: NAS-Bench-201 We adopt the NAS-Bench-201 search space to evaluate the ranking performance of SUMNAS on CIFAR-10.

Original: NAS-Bench-201 is a benchmark that contains the performance of all of the architectures in its search space on CIFAR-10, CIFAR-100, and ImageNet-16-120 (Chrabaszcz et al., 2017).
Revised: NAS-Bench-201 is a benchmark that contains the performance of all of the architectures in its search space on CIFAR-10, CIFAR-100, and ImageNet-16-120 (Chrabaszcz et al., 2017).

Original: Each network in the NAS-Bench-201 search space is composed of 15 cells.
Revised: Each network in the NAS-Bench-201 search space is composed of 15 cells.

Original: Each cell has 6 operators and each of them is one of 5 candidate operators: zeroize, identity, 3x3 average pooling, 1x1 convolution, and 3x3 convolution.
Revised: Each cell has 6 operators and each of them is one of 5 candidate operators: zeroize, identity, 3x3 average pooling, 1x1 convolution, and 3x3 convolution.

Original: Cells in a network share the same architecture; therefore, there are 5 6 possible network architectures in the search space.
Revised: Cells in a network share the same architecture; therefore, there are 5 6 possible network architectures in the search space.

Original: Details of the search space are described in Appendix A.
Revised: Details of the search space are described in Appendix A.

Original: MobileNet blocks We conduct the experiments on ImageNet using the same search space as ProxylessNAS (Cai et al., 2019) and FairNAS (Chu et al., 2019).
Revised: MobileNet blocks We conduct the experiments on ImageNet using the same search space as ProxylessNAS (Cai et al., 2019) and FairNAS (Chu et al., 2019).

Original: The operator in the search space is the bottlenecked and inverted residual block (Sandler et al., 2018) with squeeze and excitation (Hu et al., 2018) and SiLU (swish-1) activation (Ramachandran et al., 2017;
Revised: The operator in the search space is the bottlenecked and inverted residual block (Sandler et al., 2018) with squeeze and excitation (Hu et al., 2018) and SiLU (swish-1) activation (Ramachandran et al., 2017;

Original: Elfwing et al., 2018).
Revised: Elfwing et al., 2018).

Original: This block has been used in various models (Howard et al., 2019; Tan & Le, 2019a;
Revised: This block has been used in various models (Howard et al., 2019; Tan & Le, 2019a;

Original: Cai et al., 2019; Chu et al., 2019; Li et al., 2020; Mei et al., 2020).
Revised: Cai et al., 2019; Chu et al., 2019; Li et al., 2020; Mei et al., 2020).

Original: The operators’ kernel sizes are either 3, 5, or 7 and can have an expansion ratio of either 3 or 6.
Revised: The operators’ kernel sizes are either 3, 5, or 7 and can have an expansion ratio of either 3 or 6.

Original: In total, there are 6 candidate operators.
Revised: In total, there are 6 candidate operators.

Original: A candidate network is a sequential execution of the operators (i.e., an operator receives the output from a single previous operator) and has 19 operators in total.
Revised: A candidate network is a sequential execution of the operators (i.e., an operator receives the output from a single previous operator) and has 19 operators in total.

Original: Therefore, the search space consists of 6 19 possible candidate network architectures.
Revised: Therefore, the search space consists of 6 19 possible candidate network architectures.

Original: 4. R ANKING P REDICTION
Revised: 4. R ANKING P REDICTION

Original: We compare the ranking ability of the supernet trained by SUMNAS and other NAS algorithms on the NAS-Bench-201 search space and the MobileNet search space.
Revised: We compare the ranking ability of the supernet trained by SUMNAS and other NAS algorithms on the NAS-Bench-201 search space and the MobileNet search space.

Original: To measure the ranking ability, we use Kendall tau ( − 1 ≤ τ ≤ 1 ) (Kendall, 1938) between the reference rankings from standalone training and the rankings predicted by a NAS algorithm.
Revised: To measure the ranking ability, we use Kendall tau ( − 1 ≤ τ ≤ 1 ) (Kendall, 1938) between the reference rankings from standalone training and the rankings predicted by a NAS algorithm.

Original: The details for Kendall tau are described in Appendix B.
Revised: The details for Kendall tau are described in Appendix B.

Original: NAS-Bench-201: To compute Kendall tau, we use the rankings of sampled architectures instead of the entire architecture set deﬁned by the search space.
Revised: NAS-Bench-201: To compute Kendall tau, we use the rankings of sampled architectures instead of the entire architecture set deﬁned by the search space.

Original: The search space of NAS-Bench-201 contains many architectures with very similar performances, and in many cases, multiple architectures with seemingly different edges could be reduced down to a single architecture due to the topological equilibrium, as shown in Figure 2.
Revised: The search space of NAS-Bench-201 contains many architectures with very similar performances, and in many cases, multiple architectures with seemingly different edges could be reduced down to a single architecture due to the topological equilibrium, as shown in Figure 2.

Original: Therefore, ranking every single architecture in the search space is likely to produce a noisy list.
Revised: Therefore, ranking every single architecture in the search space is likely to produce a noisy list.

Original: Furthermore, its orders might change due to other factors, such as which random seed was used during training.
Revised: Furthermore, its orders might change due to other factors, such as which random seed was used during training.

Original: Therefore we sample architectures whose performances are different from each other so that the rank correlation could have higher conﬁdence.
Revised: Therefore we sample architectures whose performances are different from each other so that the rank correlation could have higher conﬁdence.

Original: Speciﬁcally, we evenly split the accuracy range of NAS-Bench-201 into 400 intervals and sample
Revised: Speciﬁcally, we evenly split the accuracy range of NAS-Bench-201 into 400 intervals and sample

Original: [Figure]
Revised: [Figure]

Original: the best one from each interval; 320 architectures are sampled consequently because 80 intervals do not contain any architecture.
Revised: the best one from each interval; 320 architectures are sampled consequently because 80 intervals do not contain any architecture.

Original: The reference accuracies and rankings of the sampled architectures are obtained from the NASBench-201 benchmark.
Revised: The reference accuracies and rankings of the sampled architectures are obtained from the NASBench-201 benchmark.

Original: The rankings predicted by DARTS (Liu et al., 2018), ENAS (Pham et al., 2018), and GDAS (Dong & Yang, 2019) come from the checkpoints of supernets trained by the
Revised: The rankings predicted by DARTS (Liu et al., 2018), ENAS (Pham et al., 2018), and GDAS (Dong & Yang, 2019) come from the checkpoints of supernets trained by the

Original: NAS-Bench-201 authors.
Revised: NAS-Bench-201 authors.

Original: For SPOS (Guo et al., 2019), FairNAS (Chu et al., 2019), and Cream (Peng et al., 2020), we implement them and train supernets.
Revised: For SPOS (Guo et al., 2019), FairNAS (Chu et al., 2019), and Cream (Peng et al., 2020), we implement them and train supernets.

Original: We also realize WPL (Benyahia et al., 2019) and NSAS (Zhang et al., 2020), which aim to resolve multi-model forgetting problem, on
Revised: We also realize WPL (Benyahia et al., 2019) and NSAS (Zhang et al., 2020), which aim to resolve multi-model forgetting problem, on

Original: SPOS.
Revised: SPOS.

Original: Table 1 shows the ranking ability of the seven NAS algorithms and SUMNAS.
Revised: Table 1 shows the ranking ability of the seven NAS algorithms and SUMNAS.

Original: We ﬁnd that SUMNAS shows more than 4.3% higher Kendall tau value, compared to other state-of-the-art NAS algorithms, such as SPOS, FairNAS, and Cream.
Revised: We ﬁnd that SUMNAS shows more than 4.3% higher Kendall tau value, compared to other state-of-the-art NAS algorithms, such as SPOS, FairNAS, and Cream.

Original: Moreover, we note that SUMNAS is superior to other
Revised: Moreover, we note that SUMNAS is superior to other

Original: NAS algorithms to resolve multi-model forgetting, such as WPL or NSAS.
Revised: NAS algorithms to resolve multi-model forgetting, such as WPL or NSAS.

Original: The outstanding ranking capability of our supernet resulted from the robustness our meta-feature learning achieves against multi-model forgetting.
Revised: The outstanding ranking capability of our supernet resulted from the robustness our meta-feature learning achieves against multi-model forgetting.

Original: We also check whether other NAS algorithms can earn a gain from more training time since SUMNAS consumes more time to process a single sampled sub-model because of the inner loop (line 10 in Algorithm 1).
Revised: We also check whether other NAS algorithms can earn a gain from more training time since SUMNAS consumes more time to process a single sampled sub-model because of the inner loop (line 10 in Algorithm 1).

Original: Speciﬁcally, the training time of SUMNAS is longer than
Revised: Speciﬁcally, the training time of SUMNAS is longer than

Original: FairNAS by a multiple of the adaptation steps because SUMNAS performs forward and backward iterations for each sampled sub-model.
Revised: FairNAS by a multiple of the adaptation steps because SUMNAS performs forward and backward iterations for each sampled sub-model.

Original: For example, when the number of adaptation steps is two, the training time of SUMNAS is twice that of FairNAS.
Revised: For example, when the number of adaptation steps is two, the training time of SUMNAS is twice that of FairNAS.

Original: We train supernets with FairNAS as a representative given more training time.
Revised: We train supernets with FairNAS as a representative given more training time.

Original: FairNAS shows a little improvement, but the gain is still minor comparing that of SUMNAS.
Revised: FairNAS shows a little improvement, but the gain is still minor comparing that of SUMNAS.

Original: The details of the result and analysis are presented in Appendix G.
Revised: The details of the result and analysis are presented in Appendix G.

Original: [Table]
Revised: [Table]

Original: MobileNet blocks: We measure the ranking abilities of FairNAS and SUMNAS on the MobileNet search space and ImageNet and present them in Table 2.
Revised: MobileNet blocks: We measure the ranking abilities of FairNAS and SUMNAS on the MobileNet search space and ImageNet and present them in Table 2.

Original: We sample 20 architectures evenly spaced in MACs from the search space and train them for 100 epochs to get the reference rankings.
Revised: We sample 20 architectures evenly spaced in MACs from the search space and train them for 100 epochs to get the reference rankings.

Original: The
Revised: The

Original: Kendall tau of SUMNAS is 7.5% higher than that of FairNAS.
Revised: Kendall tau of SUMNAS is 7.5% higher than that of FairNAS.

Original: SUMNAS more precisely predicts their rankings than FairNAS, as shown in the NAS-Bench-201 search space.
Revised: SUMNAS more precisely predicts their rankings than FairNAS, as shown in the NAS-Bench-201 search space.

Original: 4. P ERFORMANCE OF THE A RCHITECTURE F OUND
Revised: 4. P ERFORMANCE OF THE A RCHITECTURE F OUND

Original: We present the best architecture SUMNAS found on the NAS-Bench-201 search space and the MobileNet search space.
Revised: We present the best architecture SUMNAS found on the NAS-Bench-201 search space and the MobileNet search space.

Original: For the NAS-Bench-201 search space and CIFAR-10, we evaluate all 5 6 submodels in the supernets that SUMNAS and other ﬁve baselines NAS algorithms trained to discover the best architecture.
Revised: For the NAS-Bench-201 search space and CIFAR-10, we evaluate all 5 6 submodels in the supernets that SUMNAS and other ﬁve baselines NAS algorithms trained to discover the best architecture.

Original: We then compare the architectures found.
Revised: We then compare the architectures found.

Original: For the MobileNet-based search space and ImageNet, where it is computationally infeasible to evaluate all architectures, SUMNAS ﬁnds the best architecture through the evolutionary search algorithm described in Section 3.3 and the details for the algorithm and its settings are presented in Appendix C. We present the perfor-
Revised: For the MobileNet-based search space and ImageNet, where it is computationally infeasible to evaluate all architectures, SUMNAS ﬁnds the best architecture through the evolutionary search algorithm described in Section 3.3 and the details for the algorithm and its settings are presented in Appendix C. We present the perfor-

Original: [Table]
Revised: [Table]

Original: mances of architectures various NAS algorithms found on their own search space and also report
Revised: mances of architectures various NAS algorithms found on their own search space and also report

Original: MobileNet-based architectures manually crafted to compare them with ours.
Revised: MobileNet-based architectures manually crafted to compare them with ours.

Original: NAS-Bench-201: Table 1 shows the top-1 accuracy of the architectures found by each algorithm on
Revised: NAS-Bench-201: Table 1 shows the top-1 accuracy of the architectures found by each algorithm on

Original: CIFAR-10.
Revised: CIFAR-10.

Original: Without FLOPs constraint, GDAS found the best architecture among the architectures discovered by six algorithms, and our algorithm found the one on par.
Revised: Without FLOPs constraint, GDAS found the best architecture among the architectures discovered by six algorithms, and our algorithm found the one on par.

Original: Considering that GDAS shows low Kendall tau, it is clear that GDAS ﬁnds high-performance architectures but fails to properly rank architectures.
Revised: Considering that GDAS shows low Kendall tau, it is clear that GDAS ﬁnds high-performance architectures but fails to properly rank architectures.

Original: This becomes a problem for ﬁnding the best architectures that meet speciﬁc resource constraints, such as memory consumption, MACs, or latency.
Revised: This becomes a problem for ﬁnding the best architectures that meet speciﬁc resource constraints, such as memory consumption, MACs, or latency.

Original: The performance of architectures found under various MACs constraints and shows this issue.
Revised: The performance of architectures found under various MACs constraints and shows this issue.

Original: The performances of architectures found by GDAS become worse than those from SUMNAS when such constraints are given, since
Revised: The performances of architectures found by GDAS become worse than those from SUMNAS when such constraints are given, since

Original: SUMNAS more accurately rank all the models in the supernet as we mentioned in Section 4.2.
Revised: SUMNAS more accurately rank all the models in the supernet as we mentioned in Section 4.2.

Original: MobileNet blocks: The performance of the architecture we found on ImageNet is presented in
Revised: MobileNet blocks: The performance of the architecture we found on ImageNet is presented in

Original: Table 3 along with the performances of other comparable models (Howard et al., 2019; Tan et al., 2019; Tan & Le, 2019b; Li et al., 2020; Mei et al., 2020; Tan & Le, 2019a; Liang et al., 2019;
Revised: Table 3 along with the performances of other comparable models (Howard et al., 2019; Tan et al., 2019; Tan & Le, 2019b; Li et al., 2020; Mei et al., 2020; Tan & Le, 2019a; Liang et al., 2019;

Original: Chu et al., 2019).
Revised: Chu et al., 2019).

Original: All of the models except for MixNet use inverted residual blocks with squeeze and excitation and swish activation as primitive operators.
Revised: All of the models except for MixNet use inverted residual blocks with squeeze and excitation and swish activation as primitive operators.

Original: FairNAS and SUMNAS have the same search space, but the main difference between them is our meta-learning principle.
Revised: FairNAS and SUMNAS have the same search space, but the main difference between them is our meta-learning principle.

Original: Therefore, the performance improvement of SUMNAS should be due to the effect of the meta-learning approach.
Revised: Therefore, the performance improvement of SUMNAS should be due to the effect of the meta-learning approach.

Original: The model we found also has higher performance than many other state-of-the-art models, such as
Revised: The model we found also has higher performance than many other state-of-the-art models, such as

Original: DNA and SE-DARTS.
Revised: DNA and SE-DARTS.

Original: We also note that some of the other algorithms, such as AtomNAS and DNA, used ﬁner-grained search spaces, with wider choices of the number of blocks and channel sizes of the bottleneck, or incorporates blocks that mix up various kernel sizes.
Revised: We also note that some of the other algorithms, such as AtomNAS and DNA, used ﬁner-grained search spaces, with wider choices of the number of blocks and channel sizes of the bottleneck, or incorporates blocks that mix up various kernel sizes.

Original: We note that SUMNAS could adopt such search spaces to obtain more superior architectures.
Revised: We note that SUMNAS could adopt such search spaces to obtain more superior architectures.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: S ENSITIVITY A NALYSIS
Revised: S ENSITIVITY A NALYSIS

Original: In this section, we analyze how the performance of our algorithm changes with varying numbers of adaptation steps.
Revised: In this section, we analyze how the performance of our algorithm changes with varying numbers of adaptation steps.

Original: We train supernets created with the NAS-Bench-201 search space on CIFARwith combinations of four different adaptation steps { 1, 2, 3, 4 } .
Revised: We train supernets created with the NAS-Bench-201 search space on CIFARwith combinations of four different adaptation steps { 1, 2, 3, 4 } .

Original: Finally, we measure the Kendall tau between the actual ranking and the ranking predicted by the supernets as in Section 4.2, and the top-1 accuracy of the architectures SUMNAS found.
Revised: Finally, we measure the Kendall tau between the actual ranking and the ranking predicted by the supernets as in Section 4.2, and the top-1 accuracy of the architectures SUMNAS found.

Original: Adaptation step
Revised: Adaptation step

Original: Epoch
Revised: Epoch

Original: V a li d a ti on L o ss 73.
Revised: V a li d a ti on L o ss 73.

Original: 4                 mean Aa stdev
Revised: 4                 mean Aa stdev

Original: Aa
Revised: Aa

Original: Adaptation Step
Revised: Adaptation Step

Original: M ea n
Revised: M ea n

Original: A cc .
Revised: A cc .

Original: S t d e v
Revised: S t d e v

Original: A cc .
Revised: A cc .

Original: Table 4 shows the results of each conﬁguration.
Revised: Table 4 shows the results of each conﬁguration.

Original: The ranking correlation dramatically increases where the adaptation step increases from one to two, and meta-feature learning is applied.
Revised: The ranking correlation dramatically increases where the adaptation step increases from one to two, and meta-feature learning is applied.

Original: Then the correlation is further improved given more adaptation steps.
Revised: Then the correlation is further improved given more adaptation steps.

Original: Our meta-learning principle consistently enhances the ranking prediction regardless of the number of adaptation steps.
Revised: Our meta-learning principle consistently enhances the ranking prediction regardless of the number of adaptation steps.

Original: Top-1 accuracies of the architectures SUMNAS found also show a similar trend to the ranking correlation.
Revised: Top-1 accuracies of the architectures SUMNAS found also show a similar trend to the ranking correlation.

Original: We also present the results with the larger number of adaptation steps than four in Appendix H.
Revised: We also present the results with the larger number of adaptation steps than four in Appendix H.

Original: 4. S UB - MODEL P ERFORMANCE
Revised: 4. S UB - MODEL P ERFORMANCE

Original: Here, we evaluate how meta-feature learning affects the performances of sub-models in a supernet.
Revised: Here, we evaluate how meta-feature learning affects the performances of sub-models in a supernet.

Original: Since meta-features in SUMNAS cover many sub-models (as opposed to MAML meta-features, which span multiple tasks), the overall sub-model performances using parameters of a supernet trained with SUMNAS are more outstanding than those from a supernet trained without meta-feature learning.
Revised: Since meta-features in SUMNAS cover many sub-models (as opposed to MAML meta-features, which span multiple tasks), the overall sub-model performances using parameters of a supernet trained with SUMNAS are more outstanding than those from a supernet trained without meta-feature learning.

Original: We evaluate the performances of sub-models with respect to various adaptation steps to assess the effectiveness of SUMNAS.
Revised: We evaluate the performances of sub-models with respect to various adaptation steps to assess the effectiveness of SUMNAS.

Original: The left of Figure 3 shows the averaged validation losses of 20 sampled sub-models during supernet training by SUMNAS, with diverse adaptation steps.
Revised: The left of Figure 3 shows the averaged validation losses of 20 sampled sub-models during supernet training by SUMNAS, with diverse adaptation steps.

Original: SUMNAS with adaptation step 1 (blue lines) does not teach meta-features and is the same as FairNAS.
Revised: SUMNAS with adaptation step 1 (blue lines) does not teach meta-features and is the same as FairNAS.

Original: The validation losses dramatically decrease when meta-feature learning is applied, as shown in the results of adaptation step 2 (red).
Revised: The validation losses dramatically decrease when meta-feature learning is applied, as shown in the results of adaptation step 2 (red).

Original: Moreover, the losses further decrease as the number of adaptation steps increases.
Revised: Moreover, the losses further decrease as the number of adaptation steps increases.

Original: These observations demonstrate that SUMNAS properly teaches meta-features that sub-models generally utilize.
Revised: These observations demonstrate that SUMNAS properly teaches meta-features that sub-models generally utilize.

Original: We also measure the validation accuracies of all sub-models in supernets trained by SUMNAS, with the various number of adaptation steps.
Revised: We also measure the validation accuracies of all sub-models in supernets trained by SUMNAS, with the various number of adaptation steps.

Original: The right of Figure 3 presents the mean and standard deviation of the accuracies.
Revised: The right of Figure 3 presents the mean and standard deviation of the accuracies.

Original: The improvement in mean accuracy as the number of adaptation steps increases shows that the overall performance becomes higher with meta-feature learning.
Revised: The improvement in mean accuracy as the number of adaptation steps increases shows that the overall performance becomes higher with meta-feature learning.

Original: This result is consistent with the loss results and also suggests that meta-feature learning is effective.
Revised: This result is consistent with the loss results and also suggests that meta-feature learning is effective.

Original: Furthermore, the increased standard deviations indicate that the performances of sub-models are distributed over a wider range.
Revised: Furthermore, the increased standard deviations indicate that the performances of sub-models are distributed over a wider range.

Original: The wider distribution of the performances means performance differences between sub-models become larger and implies a higher resolution of ranking prediction.
Revised: The wider distribution of the performances means performance differences between sub-models become larger and implies a higher resolution of ranking prediction.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have proposed a new one-shot NAS algorithm which we call SUMNAS, to mitigate the multi-model forgetting problem.
Revised: In this paper, we have proposed a new one-shot NAS algorithm which we call SUMNAS, to mitigate the multi-model forgetting problem.

Original: In order to achieve this, we took the idea of MAML and applied it to supernet training so that SUMNAS learns unbiased meta-features of the sub-models.
Revised: In order to achieve this, we took the idea of MAML and applied it to supernet training so that SUMNAS learns unbiased meta-features of the sub-models.

Original: By explicitly helping the shared parameters to learn unbiased meta-features, we were able to address the multi-model forgetting problem efﬁciently.
Revised: By explicitly helping the shared parameters to learn unbiased meta-features, we were able to address the multi-model forgetting problem efﬁciently.

Original: As a result, SUMNAS shows better ranking performance than other state-of-the-art supernet training algorithms, and the performance of the sub-models it ﬁnds is on par with hand-crafted models and those discovered by state-of-the-art NAS algorithms.
Revised: As a result, SUMNAS shows better ranking performance than other state-of-the-art supernet training algorithms, and the performance of the sub-models it ﬁnds is on par with hand-crafted models and those discovered by state-of-the-art NAS algorithms.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: This work was supported by SNU-Naver Hyperscale AI Center.
Revised: This work was supported by SNU-Naver Hyperscale AI Center.

Original: A S EARCH S PACE
Revised: A S EARCH S PACE

Original: NAS-Bench-201 Architecture in the NAS-Bench-201 search space (Dong & Yang, 2020) used in the proposed method consists of 15 cells and the reduce block is inserted every 5 cells.
Revised: NAS-Bench-201 Architecture in the NAS-Bench-201 search space (Dong & Yang, 2020) used in the proposed method consists of 15 cells and the reduce block is inserted every 5 cells.

Original: The architecture has a convolutional layer for the stem before the input layer is connected to the cell, and a global pooling and a fully connected layer at the end, as described in Figure 4 A cell is a densely connected
Revised: The architecture has a convolutional layer for the stem before the input layer is connected to the cell, and a global pooling and a fully connected layer at the end, as described in Figure 4 A cell is a densely connected

Original: DAG where each node is an intermediate representation and an edge is an operator that transforms the source node and propagates it to the destination node, where the output of multiple incoming edges are combined into one.
Revised: DAG where each node is an intermediate representation and an edge is an operator that transforms the source node and propagates it to the destination node, where the output of multiple incoming edges are combined into one.

Original: The reduce block doubles the channel size and halves the height and width of the feature map.
Revised: The reduce block doubles the channel size and halves the height and width of the feature map.

Original: Each cell has six edges and four nodes including the input and output nodes.
Revised: Each cell has six edges and four nodes including the input and output nodes.

Original: The direction of an edge is from the i -th node to j -th node where i < j .
Revised: The direction of an edge is from the i -th node to j -th node where i < j .

Original: Each edge is one of 5 candidate operators: zeroize, identity, 3x3 average pooling, 1x1 convolution, and 3x convolution.
Revised: Each edge is one of 5 candidate operators: zeroize, identity, 3x3 average pooling, 1x1 convolution, and 3x convolution.

Original: All cells of an arbitrary architecture sampled from the search space have the same operator conﬁguration corresponding to each edge.
Revised: All cells of an arbitrary architecture sampled from the search space have the same operator conﬁguration corresponding to each edge.

Original: Therefore, there are | operator set | # edge =
Revised: Therefore, there are | operator set | # edge =

Original: 5candidate architectures in the search space.
Revised: 5candidate architectures in the search space.

Original: [Figure]
Revised: [Figure]

Original: MobileNet Blocks We use the same MobileNet-based search space (Howard et al., 2019) as ProxylessNAS (Cai et al., 2019) and FairNAS (Chu et al., 2019) in our ImageNet (Russakovsky et al., 2015) experiments for the sake of fair comparison.
Revised: MobileNet Blocks We use the same MobileNet-based search space (Howard et al., 2019) as ProxylessNAS (Cai et al., 2019) and FairNAS (Chu et al., 2019) in our ImageNet (Russakovsky et al., 2015) experiments for the sake of fair comparison.

Original: Each architecture in the search space is a sequential execution of two convolutional layers for the stem, 19 MobileNet blocks, and a convolutional layer and fully connected layer for the head, as depicted in Figure 5.
Revised: Each architecture in the search space is a sequential execution of two convolutional layers for the stem, 19 MobileNet blocks, and a convolutional layer and fully connected layer for the head, as depicted in Figure 5.

Original: [Figure]
Revised: [Figure]

Original: The MobileNet blocks are the bottlenecked and inverted residual blocks (Sandler et al., 2018) that squeeze and excitation (Hu et al., 2018) and SiLU (swish-1) activation (Elfwing et al., 2018;
Revised: The MobileNet blocks are the bottlenecked and inverted residual blocks (Sandler et al., 2018) that squeeze and excitation (Hu et al., 2018) and SiLU (swish-1) activation (Elfwing et al., 2018;

Original: Ramachandran et al., 2017) are mounted.
Revised: Ramachandran et al., 2017) are mounted.

Original: Each architecture consists of blocks with different kernel sizes and expansion ratios.
Revised: Each architecture consists of blocks with different kernel sizes and expansion ratios.

Original: A kernel size of a block is either 3, 5, or 7, and an expansion ratio of a block is either 3 or 6; there are 6 candidate blocks.
Revised: A kernel size of a block is either 3, 5, or 7, and an expansion ratio of a block is either 3 or 6; there are 6 candidate blocks.

Original: Architecture in the search space has 19 blocks, and each block is one of the 6 candidate blocks, which means the search space has 6 19 candidate architectures in total.
Revised: Architecture in the search space has 19 blocks, and each block is one of the 6 candidate blocks, which means the search space has 6 19 candidate architectures in total.

Original: B K ENDALL T AU
Revised: B K ENDALL T AU

Original: In Section 4.2, we measure the ranking ability with Kendall tau
Revised: In Section 4.2, we measure the ranking ability with Kendall tau

Original: (Kendall, 1938) between standalone validation accuracies from the benchmark and predicted validation accuracies from the supernet.
Revised: (Kendall, 1938) between standalone validation accuracies from the benchmark and predicted validation accuracies from the supernet.

Original: Kendall tau is a measure of rank correlation: the similarity of the rankings of the two variables.
Revised: Kendall tau is a measure of rank correlation: the similarity of the rankings of the two variables.

Original: Given a set of observations { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x n , y n ) } of the two joint random variable X and
Revised: Given a set of observations { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x n , y n ) } of the two joint random variable X and

Original: Y , the measure is the ratio of the discordant pairs of the observations subtracted from the ratio of the concordant pairs.
Revised: Y , the measure is the ratio of the discordant pairs of the observations subtracted from the ratio of the concordant pairs.

Original: In our experiments, x i represents a standalone accuracy of architecture from the benchmark and y i indicates the accuracy of the corresponding architecture that a supernet predicts.
Revised: In our experiments, x i represents a standalone accuracy of architecture from the benchmark and y i indicates the accuracy of the corresponding architecture that a supernet predicts.

Original: The Kendall tau τ is formally deﬁned as following:
Revised: The Kendall tau τ is formally deﬁned as following:

Original: [Equation]
Revised: [Equation]

Original: The number of concordant pairs n c and the number of discordant pairs n d is (cid:0) n 2 (cid:1) at most; therefore, − 1 < τ < 1 .
Revised: The number of concordant pairs n c and the number of discordant pairs n d is (cid:0) n 2 (cid:1) at most; therefore, − 1 < τ < 1 .

Original: If τ > 0 , there are more concordant pairs than discordant pairs and this means the two rankings partially (or perfectly when τ = 1 ) agree.
Revised: If τ > 0 , there are more concordant pairs than discordant pairs and this means the two rankings partially (or perfectly when τ = 1 ) agree.

Original: On the other hand, when τ < 0 , the two rankings disagree.
Revised: On the other hand, when τ < 0 , the two rankings disagree.

Original: C S EARCH A LGORITHM
Revised: C S EARCH A LGORITHM

Original: To discover the best architecture on the MobileNet-based search space and ImageNet (Russakovsky et al., 2015), we adopt a simple evolutionary algorithm same as SPOS (Guo et al., 2019).
Revised: To discover the best architecture on the MobileNet-based search space and ImageNet (Russakovsky et al., 2015), we adopt a simple evolutionary algorithm same as SPOS (Guo et al., 2019).

Original: The concrete algorithm is described in Algorithm 2.
Revised: The concrete algorithm is described in Algorithm 2.

Original: The search algorithm repeatedly generates offsprings from the population using mutation and crossover, and keep the top P samples among the population and the offsprings for the population of the next iteration.
Revised: The search algorithm repeatedly generates offsprings from the population using mutation and crossover, and keep the top P samples among the population and the offsprings for the population of the next iteration.

Original: The mutation randomly changes each operator to one of the others with the probability p , and the crossover randomly mixes two architectures in the population.
Revised: The mutation randomly changes each operator to one of the others with the probability p , and the crossover randomly mixes two architectures in the population.

Original: The search requires several hyperparameters, and we use generations
Revised: The search requires several hyperparameters, and we use generations

Original: G = 30 , population P = 20 , the number of offsprings C = 64 , mutation probability p = 0 .
Revised: G = 30 , population P = 20 , the number of offsprings C = 64 , mutation probability p = 0 .

Original: 1 , and
Revised: 1 , and

Original: MACs constraint M = 400( M ) for the experiment.
Revised: MACs constraint M = 400( M ) for the experiment.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D E XPERIMENTAL S ETTING
Revised: D E XPERIMENTAL S ETTING

Original: NAS-Bench-201 search space and CIFAR-

Revised: NAS-Bench-201 search space and CIFAR-

Original: For the experiments of the NAS-Bench-201 search space (Dong & Yang, 2020) and CIFAR-10 (Krizhevsky et al., 2009), we train the supernets on the entire training set of CIFAR-10.
Revised: For the experiments of the NAS-Bench-201 search space (Dong & Yang, 2020) and CIFAR-10 (Krizhevsky et al., 2009), we train the supernets on the entire training set of CIFAR-10.

Original: On the test set, the hyperparameters are tuned and the reported
Revised: On the test set, the hyperparameters are tuned and the reported

Original: Kendall tau and accuracies are measured.
Revised: Kendall tau and accuracies are measured.

Original: We also search for the best architecture on the test set.
Revised: We also search for the best architecture on the test set.

Original: We tune the hyperparameters where the number of adaptation steps is 4, and the hyperparameter search space is presented in Table 5.
Revised: We tune the hyperparameters where the number of adaptation steps is 4, and the hyperparameter search space is presented in Table 5.

Original: We choose the best hyperparameter set which shows the best Kendall tau for the sampled architectures.
Revised: We choose the best hyperparameter set which shows the best Kendall tau for the sampled architectures.

Original: The searched hyperparameters are listed in Table 6.
Revised: The searched hyperparameters are listed in Table 6.

Original: The tuned hyperparameters are also used in SUMNAS with other numbers of adaptation steps, but the learning rate for the inner loop LR inner is scaled so that actual parameter update sizes keep the same by multiplying 4 /adaptation step .
Revised: The tuned hyperparameters are also used in SUMNAS with other numbers of adaptation steps, but the learning rate for the inner loop LR inner is scaled so that actual parameter update sizes keep the same by multiplying 4 /adaptation step .

Original: SPOS and FairNAS adopt the hyperparameters for the outer loop of SUMNAS, but use LR mult for the learning rate, which is LR outer × LR inner with scaling as
Revised: SPOS and FairNAS adopt the hyperparameters for the outer loop of SUMNAS, but use LR mult for the learning rate, which is LR outer × LR inner with scaling as

Original: SUMNAS where adaptation step is one.
Revised: SUMNAS where adaptation step is one.

Original: MobileNet-based search space and ImageNet For the experiment of the MobileNet-based search space and ImageNet (Russakovsky et al., 2015)
Revised: MobileNet-based search space and ImageNet For the experiment of the MobileNet-based search space and ImageNet (Russakovsky et al., 2015)

Original: , we tune hyperparameters and search the best architecture using a validation set that includes about 50K examples sampled from the training set.
Revised: , we tune hyperparameters and search the best architecture using a validation set that includes about 50K examples sampled from the training set.

Original: Due to the time-consuming task of obtaining the standalone (from scratch) validation accuracies of the sampled architectures to compute Kendall tau, we use the average validation accuracy (from shared parameter) to determine the best performing hyperparameter set.
Revised: Due to the time-consuming task of obtaining the standalone (from scratch) validation accuracies of the sampled architectures to compute Kendall tau, we use the average validation accuracy (from shared parameter) to determine the best performing hyperparameter set.

Original: The hyperparameter search space is described in Table 5, and Table 6 presents the selected hyperparameters.
Revised: The hyperparameter search space is described in Table 5, and Table 6 presents the selected hyperparameters.

Original: We also adopt task batching of MAML and aggregate the gradients of three sub-models for each operator to update its parameters.
Revised: We also adopt task batching of MAML and aggregate the gradients of three sub-models for each operator to update its parameters.

Original: E P ERFORMANCE SENSITIVITY TO THE LEARNING RATES FOR INNER AND OUTER LOOPS
Revised: E P ERFORMANCE SENSITIVITY TO THE LEARNING RATES FOR INNER AND OUTER LOOPS

Original: We analyze the sensitivity of SUMNAS performance to the inner and outer learning rates to show that SUMNAS does not demand much extra effort to tune the additional learning rate.
Revised: We analyze the sensitivity of SUMNAS performance to the inner and outer learning rates to show that SUMNAS does not demand much extra effort to tune the additional learning rate.

Original: We measure
Revised: We measure

Original: [Table]
Revised: [Table]

Original: the mean and standard deviation of the Kendall tau over the outer learning rates for each LR scale we suggested in Appendix D.
Revised: the mean and standard deviation of the Kendall tau over the outer learning rates for each LR scale we suggested in Appendix D.

Original: The results are presented in Table 7.
Revised: The results are presented in Table 7.

Original: For LR scales of 0.05 and 0.25, the average of the Kendall taus are similar, but at 0.01, the performance is obviously degraded.
Revised: For LR scales of 0.05 and 0.25, the average of the Kendall taus are similar, but at 0.01, the performance is obviously degraded.

Original: However, we conducted an ablation study in a sufﬁciently wided LR scale range, and it can be seen that the degradation is insensitive enough for SUMNAS to the LR scale.
Revised: However, we conducted an ablation study in a sufﬁciently wided LR scale range, and it can be seen that the degradation is insensitive enough for SUMNAS to the LR scale.

Original: The standard deviations show the sensitivity with respect to the relation between the outer and inner learning rates.
Revised: The standard deviations show the sensitivity with respect to the relation between the outer and inner learning rates.

Original: Standard deviations are usually small, and SUMNAS is robust to the relation of LRs when we take sufﬁcient LR scales, such as 0.05 or 0.25.
Revised: Standard deviations are usually small, and SUMNAS is robust to the relation of LRs when we take sufﬁcient LR scales, such as 0.05 or 0.25.

Original: F D ISCOVERED A RCHITECTURES
Revised: F D ISCOVERED A RCHITECTURES

Original: Figure 4 and Figure 5 visualize the architectures SUMNAS found on the NAS-Bench-201 search space
Revised: Figure 4 and Figure 5 visualize the architectures SUMNAS found on the NAS-Bench-201 search space

Original: (Dong & Yang, 2020) and the MobilNet-based search space (Howard et al., 2019) respectively.
Revised: (Dong & Yang, 2020) and the MobilNet-based search space (Howard et al., 2019) respectively.

Original: G R ANKING ABILITY OF F AIR NAS WITH MORE TRAINING TIME
Revised: G R ANKING ABILITY OF F AIR NAS WITH MORE TRAINING TIME

Original: [Table]
Revised: [Table]

Original: We also check whether FairNAS can earn a gain from more training time, since SUMNAS consumes more time to process a single sampled sub-model because of the inner loop (line 10 in Algorithm 1).
Revised: We also check whether FairNAS can earn a gain from more training time, since SUMNAS consumes more time to process a single sampled sub-model because of the inner loop (line 10 in Algorithm 1).

Original: Table 8 presents the Kendall tau of FairNAS when more training epochs are given.
Revised: Table 8 presents the Kendall tau of FairNAS when more training epochs are given.

Original: Note that the learning rate we use here is different from one we present in Appendix D. The supernet for FairNAS in Table 1 is trained for 200 epochs, so the Kendall tau for more than 200 epochs shows the loss or gain from more training time.
Revised: Note that the learning rate we use here is different from one we present in Appendix D. The supernet for FairNAS in Table 1 is trained for 200 epochs, so the Kendall tau for more than 200 epochs shows the loss or gain from more training time.

Original: In 400 epochs, the performance improves for 0.8%.
Revised: In 400 epochs, the performance improves for 0.8%.

Original: However, the performances degrades in 600 or 800 epochs.
Revised: However, the performances degrades in 600 or 800 epochs.

Original: We conjecture that the degradation results from cosine learning rate decay, which was used in the FairNAS paper.
Revised: We conjecture that the degradation results from cosine learning rate decay, which was used in the FairNAS paper.

Original: Increasing the number of epochs might have resulted in longer training time in high learning rates, which might have unstabilized the overall training.
Revised: Increasing the number of epochs might have resulted in longer training time in high learning rates, which might have unstabilized the overall training.

Original: Therefore, FairNAS improves slightly with more training time, but the improvement is minor comparing to the gain from SUMNAS.
Revised: Therefore, FairNAS improves slightly with more training time, but the improvement is minor comparing to the gain from SUMNAS.

Original: H SUMNAS WITH MORE ADAPTATION STEPS
Revised: H SUMNAS WITH MORE ADAPTATION STEPS

Original: [Table]
Revised: [Table]

Original: Table 9 presents the ranking ability and the performance of the searched architecture on CIFARand NAS-Bench-201 with the number of adaptation steps above four.
Revised: Table 9 presents the ranking ability and the performance of the searched architecture on CIFARand NAS-Bench-201 with the number of adaptation steps above four.

Original: The Kendall tau decreases with the number of adaptation steps when it is larger than four.
Revised: The Kendall tau decreases with the number of adaptation steps when it is larger than four.

Original: As the number of adaptation steps increases, each sub-model has to incorporate more data points for a single update, which increases the standard deviation of the gradient that needs to be updated according to the sensitivity of the sub-model.
Revised: As the number of adaptation steps increases, each sub-model has to incorporate more data points for a single update, which increases the standard deviation of the gradient that needs to be updated according to the sensitivity of the sub-model.

Original: Large standard deviation above a certain level is likely to interfere with learning the unbiased meta feature.
Revised: Large standard deviation above a certain level is likely to interfere with learning the unbiased meta feature.

Original: Consequently, an excessive adaptation step will have the effect of lowering the ranking ability.
Revised: Consequently, an excessive adaptation step will have the effect of lowering the ranking ability.