Original: A S URGERY OF THE N EURAL A RCHITECTURE E VALUATORS
Revised: A S URGERY OF THE N EURAL A RCHITECTURE E VALUATORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural architecture search (NAS) has recently received extensive attention due to its effectiveness in automatically designing effective neural architectures.
Revised: Neural architecture search (NAS) has recently received extensive attention due to its effectiveness in automatically designing effective neural architectures.

Original: A major challenge in NAS is to conduct a fast and accurate evaluation  of neural architectures.
Revised: A major challenge in NAS is to conduct a fast and accurate evaluation (i.e., performance estimation) of neural architectures.

Original: Commonly used fast architecture evaluators include parameter-sharing ones and predictor-based ones.
Revised: Commonly used fast architecture evaluators include parameter-sharing ones and predictor-based ones.

Original: Despite their high evaluation efﬁciency, the evaluation correlation (especially of the well-performing architectures) is still questionable.
Revised: Despite their high evaluation efﬁciency, the evaluation correlation (especially of the well-performing architectures) is still questionable.

Original: In this paper, we conduct an extensive assessment of both the parameter-sharing and predictor-based evaluators on the NAS-Bench-201 search space, and break up how and why different conﬁgurations and strategies inﬂuence the ﬁtness of the evaluators.
Revised: In this paper, we conduct an extensive assessment of both the parameter-sharing and predictor-based evaluators on the NASBench-201 search space, and break up how and why different conﬁgurations and strategies inﬂuence the ﬁtness of the evaluators.

Original: Speciﬁcally, we develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages.
Revised: Speciﬁcally, we develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages.

Original: And based on the ﬁndings of our experiments, we give pieces of knowledge and suggestions to guide NAS application and motivate further research.
Revised: And based on the ﬁndings of our experiments, we give pieces of knowledge and suggestions to guide NAS application and motivate further research.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Studies have shown that the automatically discovered architectures by NAS can outperform the hand-crafted architectures for various applications, such as classiﬁcation (Nayman et al., 2019; Zoph & Le, 2017), detection (Ghiasi et al., 2019; Chen et al., 2019b), video understanding (Ryoo et al., 2019), text modeling (Zoph & Le, 2017), etc.
Revised: Studies have shown that the automatically discovered architectures by NAS can outperform the hand-crafted architectures for various applications, such as classiﬁcation (Nayman et al., 2019; Zoph & Le, 2017), detection (Ghiasi et al., 2019; Chen et al., 2019b), video understanding (Ryoo et al., 2019), text modeling (Zoph & Le, 2017), etc.

Original: Early NAS algorithms (Zoph & Le, 2017) suffer from the extremely heavy computational burden, since the evaluation of neural architectures is slow.
Revised: Early NAS algorithms (Zoph & Le, 2017) suffer from the extremely heavy computational burden, since the evaluation of neural architectures is slow.

Original: Thus, how to evaluate a neural architecture in a fast and accurate way is vital for addressing the computational challenge of NAS.
Revised: Thus, how to estimate the performance of a neural architecture in a fast and accurate way is vital for addressing the computational challenge of NAS.

Original: A neural architecture evaluator outputs the evaluated score of an architecture that indicates its quality.
Revised: A neural architecture evaluator outputs the evaluated score of an architecture that indicates its quality.

Original: The straightforward solution is to train an architecture from scratch to convergence and then test it on the validation dataset, which is extremely time-consuming.
Revised: The straightforward solution is to train an architecture from scratch to convergence and then test it on the validation dataset, which is extremely time-consuming.

Original: Instead of exactly evaluating architectures on the target task, researchers usually construct a proxy model with fewer layers or fewer channels (Pham et al., 2018;
Revised: Instead of exactly evaluating architectures on the target task, researchers usually construct a proxy model with fewer layers or fewer channels (Pham et al., 2018;

Original: Real et al., 2019; Wu et al., 2019), and train this model to solve a proxy task of smaller scales (Cai et al., 2018a; Elsken et al., 2018; Klein et al., 2017; Wu et al., 2019), e.g., smaller dataset or subsets of dataset, training or ﬁnetuning for fewer epochs.
Revised: Real et al., 2019; Wu et al., 2019), and train this model to solve a proxy task of smaller scales (Cai et al., 2018a; Elsken et al., 2018; Klein et al., 2017; Wu et al., 2019), e.g., smaller dataset or subsets of dataset, training or ﬁnetuning for fewer epochs.

Original: Traditional evaluators conduct separate training phases to acquire the weights that are suitable for each architecture.
Revised: Traditional evaluators conduct separate training phases to acquire the weights that are suitable for each architecture.

Original: In contrast, one-shot evaluation amortized the training cost of different architectures through parameter sharing or a global hypernetwork, thus signiﬁcantly reduce the evaluation cost.
Revised: In contrast, one-shot evaluation amortized the training cost of different architectures through parameter sharing or a global hypernetwork, thus signiﬁcantly reduce the evaluation cost.

Original: Pham et al.
Revised: Pham et al.

Original: (2018) constructs an over-parametrized super network (supernet) such that all architectures in the search space are sub-architectures of the supernet.
Revised: (2018) constructs an over-parametrized super network (supernet) such that all architectures in the search space are sub-architectures of the supernet.

Original: Throughout the search process, the shared parameters in the supernet are updated on the training dataset split, and each architecture is evaluated by directly using the corresponding subset of the weights.
Revised: Throughout the search process, the shared parameters in the supernet are updated on the training dataset split, and each architecture is evaluated by directly using the corresponding subset of the weights.

Original: Afterwards, the parameter sharing technique is widely used for architecture search in different search spaces (Wu et al., 2019;
Revised: Afterwards, the parameter sharing technique is widely used for architecture search in different search spaces (Wu et al., 2019;

Original: Cai et al., 2020), or incorporated with different search strategies (Liu et al., 2018b; Nayman et al., 2019; Xie et al., 2018; Yang et al., 2019;
Revised: Cai et al., 2020), or incorporated with different search strategies (Liu et al., 2018b; Nayman et al., 2019; Xie et al., 2018; Yang et al., 2019;

Original: Cai et al., 2020).
Revised: Cai et al., 2020).

Original: Hypernetwork (Brock et al., 2018; Zhang et al., 2018)
Revised: Hypernetwork (Brock et al., 2018; Zhang et al., 2018)

Original: based evaluation is another type of one-shot evaluation strategy, in which a hypernetwork is trained to generate proper weights for each architecture.
Revised: based evaluation is another type of one-shot evaluation strategy, in which a hypernetwork is trained to generate proper weights for each architecture.

Original: Since hypernetwork solutions are not generic currently, this paper concentrates on the evaluation of parameter sharing evaluators.
Revised: Since hypernetwork solutions are not generic currently, this paper concentrates on the evaluation of parameter sharing evaluators.

Original: Whether or not one-shot strategies can provide highly-correlated architecture evaluation results is essential for the efﬁcacy of the NAS process. Many recent studies have been focusing on assessing One-Shot (Hypernetwork) SuperNet
Revised: [Figure]

Original: Train Loss Valid Accuracy Gradient Update Generate weights Architecture (with weights)
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Predictor-based Architecture PerformanceLoss Predicted Score
Revised: [Arch N

Original: Ground-Truth Performance MLP Encoder Predictor Gradient Update [Arch 1 , 78.3%] …
Revised: [Figure]

Original: [Arch N , 80.4%] Architecture
Revised: [Figure]

Original: Provided by another “oracle” evaluator
Revised: Provided by another “oracle” evaluator

Original: HyperNet
Revised: HyperNet

Original: One-Shot (Shared weights) Gradient Update
Revised: [Figure]

Original: Training
Revised: Training

Original: Inference (Evaluator Output) (Evaluator Output) (Evaluator Input) (Evaluator Input) the evaluation correlation of one-shot architecture evaluators (Bender et al., 2018; Sciuto et al., 2019;
Revised: Inference [Figure] Whether or not one-shot strategies can provide highly-correlated architecture evaluation results is essential for the efﬁcacy of the NAS process. Many recent studies have been focusing on assessing the evaluation correlation of one-shot architecture evaluators (Bender et al., 2018; Sciuto et al., 2019;

Original: Zela et al., 2020).
Revised: Zela et al., 2020).

Original: Besides one-shot evaluation strategies, predictor-based evaluation strategies (Luo et al., 2018;
Revised: Besides one-shot evaluation strategies, predictor-based evaluation strategies (Luo et al., 2018;

Original: Liu et al., 2018a; Deng et al., 2017; Sun et al., 2019; Wang et al., 2018; Xu et al., 2019; Ning et al., 2020) use a performance predictor that takes the architecture description as inputs and outputs a predicted performance score.
Revised: Liu et al., 2018a; Deng et al., 2017; Sun et al., 2019; Wang et al., 2018; Xu et al., 2019; Ning et al., 2020) use a performance predictor that takes the architecture description as inputs and outputs a predicted performance score.

Original: The performance predictor should be trained using “ground-truth” architecture performances.
Revised: The performance predictor should be trained using “ground-truth” architecture performances.

Original: This paper utilizes the same set of criteria to evaluate and compare different performance predictors.
Revised: This paper utilizes the same set of criteria to evaluate and compare different performance predictors.

Original: The fast neural architecture evaluators  are summarized in Fig.
Revised: The fast neural architecture evaluators (i.e., performance estimators) are summarized in Fig.

Original: 1, including parameter sharing, hypernetworks, and predictor-based ones.
Revised: 1, including parameter sharing, hypernetworks, and predictor-based ones.

Original: And this paper aims at revealing the status of current architecture evaluation strategies systematically.
Revised: And this paper aims at revealing the status of current architecture evaluation strategies systematically.

Original: Speciﬁcally, we develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages.
Revised: Speciﬁcally, we develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages.

Original: And based on the ﬁndings of our experiments, we give pieces of knowledge and suggestions to guide NAS application and motivate further research.
Revised: And based on the ﬁndings of our experiments, we give pieces of knowledge and suggestions to guide NAS application and motivate further research.

Original: The knowledge revealed by this paper includes: 1) Layer proxy brings a larger evaluation gap than using channel proxy, thus channel proxy can be utilized to reduce the computational cost, while proxy-less search w.r.t the layer number is worth studying.
Revised: The knowledge revealed by this paper includes: 1) Layer proxy brings a larger evaluation gap than using channel proxy, thus channel proxy can be utilized to reduce the computational cost, while proxy-less search w.r.t the layer number is worth studying.

Original: The convergence rate of different criteria varies during the one-shot supernet training, which shows that the good architectures are distinguished from bad architectures in the early stage.
Revised: The convergence rate of different criteria varies during the one-shot supernet training, which shows that the good architectures are distinguished from bad architectures in the early stage.

Original: As training goes on, the one-shot performances of isomorphic sub-architectures become closer.
Revised: As training goes on, the one-shot performances of isomorphic sub-architectures become closer.

Original: 4) De-isomorphic sampling or post de-isomorphism handling can help avoid the over-estimation of simple architectures.
Revised: 4) De-isomorphic sampling or post de-isomorphism handling can help avoid the over-estimation of simple architectures.

Original: Parameter sharing evaluator tends to over-estimate smaller architectures, and is better at comparing smaller models than larger models.
Revised: Parameter sharing evaluator tends to over-estimate smaller architectures, and is better at comparing smaller models than larger models.

Original: 6) One should use ranking losses rather than regression losses to train predictors, since they are more stable.
Revised: 6) One should use ranking losses rather than regression losses to train predictors, since they are more stable.

Original: Different predictors under- or over-estimate different architectures, and currently, the best predictor might still have trouble in comparing large architectures.
Revised: Different predictors under- or over-estimate different architectures, and currently, the best predictor might still have trouble in comparing large architectures.

Original: 8) As expected, architecture predictors can distinguish good architectures better after multiple stages of training, as the training data are more and more concentrated on the good architectures.
Revised: 8) As expected, architecture predictors can distinguish good architectures better after multiple stages of training, as the training data are more and more concentrated on the good architectures.

Original:  R ELATED W ORK 2. O NE - SHOT E VALUATORS
Revised:  R ELATED W ORK 2. O NE - SHOT E VALUATORS

Original: One-shot evaluation mainly consists of two types of strategies: 1) parameter sharing (Pham et al., 2018;
Revised: One-shot evaluation mainly consists of two types of strategies: 1) parameter sharing (Pham et al., 2018;

Original: Wu et al., 2019; Liu et al., 2018b; Nayman et al., 2019; Xie et al., 2018; Yang et al., 2019; Cai et al., 2020), 2) hypernetworks (Brock et al., 2018;
Revised: Wu et al., 2019; Liu et al., 2018b; Nayman et al., 2019; Xie et al., 2018; Yang et al., 2019; Cai et al., 2020), 2) hypernetworks (Brock et al., 2018;

Original: Zhang et al., 2018).
Revised: Zhang et al., 2018).

Original: These two strategies both amortize the training cost of different architectures via the sharing of the network or hypernetwork parameters.
Revised: These two strategies both amortize the training cost of different architectures via the sharing of the network or hypernetwork parameters.

Original: The ranking correlation gaps of existing shared weights evaluators are brought by two factors: 1) proxy model and task: due to the memory constraint, a proxy supernet (supernet) (Liu et al., 2018b;
Revised: The ranking correlation gaps of existing shared weights evaluators are brought by two factors: 1) proxy model and task: due to the memory constraint, a proxy supernet (supernet) (Liu et al., 2018b;

Original: Pham et al., 2018) with fewer channels or layers is usually used; 2) parameter sharing.
Revised: Pham et al., 2018) with fewer channels or layers is usually used; 2) parameter sharing.

Original: To alleviate the ﬁrst factor, there are some studies (Cai et al., 2018b; Chen et al., 2019a) that aim at making oneshot evaluation more memory efﬁcient, thus the one-shot search could be conducted without using a proxy supernet.
Revised: To alleviate the ﬁrst factor, there are some studies (Cai et al., 2018b; Chen et al., 2019a) that aim at making oneshot evaluation more memory efﬁcient, thus the one-shot search could be conducted without using a proxy supernet.

Original: As for the second factor, there are a few studies that carried out correlation evaluation for one-shot evaluators.
Revised: As for the second factor, there are a few studies that carried out correlation evaluation for one-shot evaluators.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) conducted a correlation comparison between the
Revised: (2018) conducted a correlation comparison between the

Original: GHN hypernetwork evaluator, shared weights evaluator, and several small proxy tasks.
Revised: GHN hypernetwork evaluator, shared weights evaluator, and several small proxy tasks.

Original: However, the correlation is evaluated using 100 architectures randomly sampled from a large search space, which is not a convincing and consistent benchmark metric.
Revised: However, the correlation is evaluated using 100 architectures randomly sampled from a large search space, which is not a convincing and consistent benchmark metric.

Original: Luo et al.
Revised: Luo et al.

Original: (2019) did a preliminary investigation into why parameter sharing evaluation fails to provide correlated evaluations, and proposed to increase the sample probabilities of the large models.
Revised: (2019) did a preliminary investigation into why parameter sharing evaluation fails to provide correlated evaluations, and proposed to increase the sample probabilities of the large models.

Original: Their evaluation is also conducted on dozens of architectures sampled from the search space.
Revised: Their evaluation is also conducted on dozens of architectures sampled from the search space.

Original: Zela et al.
Revised: Zela et al.

Original: (2020) compare the evaluation correlation of different search strategies on NAS-Bench-101.
Revised: (2020) compare the evaluation correlation of different search strategies on NAS-Bench-101.

Original: Sciuto et al.
Revised: Sciuto et al.

Original: (2019) conduct parameter sharing NAS in a toy RNN search space with only 32 architectures in total, and discover that the parameter sharing rankings do not correlate with the true rankings of architectures.
Revised: (2019) conduct parameter sharing NAS in a toy RNN search space with only 32 architectures in total, and discover that the parameter sharing rankings do not correlate with the true rankings of architectures.

Original: To improve the evaluation correlation, Chu et al.
Revised: To improve the evaluation correlation, Chu et al.

Original: (2019) proposed a sampling strategy in a layer-wise search space.
Revised: (2019) proposed a sampling strategy in a layer-wise search space.

Original: In this paper, we analyze the ranking correlation gaps brought by the model proxy (difference in the number of channels and layers) and the parameter sharing technique, as well as the behavior of one-shot evaluators during the training process.
Revised: In this paper, we analyze the ranking correlation gaps brought by the model proxy (difference in the number of channels and layers) and the parameter sharing technique, as well as the behavior of one-shot evaluators during the training process.

Original: 2. P REDICTOR - BASED E VALUATORS
Revised: 2. P REDICTOR - BASED E VALUATORS

Original: An architecture performance predictor takes the architecture descriptions as inputs, and outputs the predicted performance scores without training the architectures.
Revised: An architecture performance predictor takes the architecture descriptions as inputs, and outputs the predicted performance scores without training the architectures.

Original: Two factors are crucial to the ﬁtness of the predictors: 1) embedding space; 2) training technique.
Revised: Two factors are crucial to the ﬁtness of the predictors: 1) embedding space; 2) training technique.

Original: On the one hand, to embed neural architectures into a continuous space and get a meaningful embedding space, there are studies that propose different architecture encoders, e.g., sequence-based (Luo et al., 2018;
Revised: On the one hand, to embed neural architectures into a continuous space and get a meaningful embedding space, there are studies that propose different architecture encoders, e.g., sequence-based (Luo et al., 2018;

Original: Liu et al., 2018a;
Revised: Liu et al., 2018a;

Original: Deng et al., 2017; Sun et al., 2019; Wang et al., 2018), graph-based (Shi et al., 2019; Ning et al., 2020).
Revised: Deng et al., 2017; Sun et al., 2019; Wang et al., 2018), graph-based (Shi et al., 2019; Ning et al., 2020).

Original: As for nonparametric predictors, Kandasamy et al.
Revised: As for nonparametric predictors, Kandasamy et al.

Original: (2018) design a kernel function in the architecture space and exploits gaussian process to get the posterior of the architecture performances.
Revised: (2018) design a kernel function in the architecture space and exploits gaussian process to get the posterior of the architecture performances.

Original: Shi et al.
Revised: Shi et al.

Original: combined a graph-based encoder and nonparametric gaussian process to construct the performance predictor.
Revised: combined a graph-based encoder and nonparametric gaussian process to construct the performance predictor.

Original: On the other hand, from the aspect of training techniques, Luo et al.
Revised: On the other hand, from the aspect of training techniques, Luo et al.

Original: (2018) employed an encoder-decoder structure and used an auxiliary reconstruction loss term.
Revised: (2018) employed an encoder-decoder structure and used an auxiliary reconstruction loss term.

Original: Xu et al.
Revised: Xu et al.

Original: ; Ning et al.
Revised: ; Ning et al.

Original: (2020) employed learning-to-rank techniques to train the predictors.
Revised: (2020) employed learning-to-rank techniques to train the predictors.

Original: Actually, in the overall NAS framework, the predictor-based evaluator plays a different role from the traditional or one-shot evaluators, since the predictor should be trained using “ground-truth” architecture performances.
Revised: Actually, in the overall NAS framework, the predictor-based evaluator plays a different role from the traditional or one-shot evaluators, since the predictor should be trained using “ground-truth” architecture performances.

Original: Usually, expensive traditional evaluators that can provide relatively accurate architecture performances are chosen as the “oracle” evaluators to output the “ground-truth” scores (Kandasamy et al., 2018;
Revised: Usually, expensive traditional evaluators that can provide relatively accurate architecture performances are chosen as the “oracle” evaluators to output the “ground-truth” scores (Kandasamy et al., 2018;

Original: Liu et al., 2018a;
Revised: Liu et al., 2018a;

Original: Luo et al., 2018).
Revised: Luo et al., 2018).

Original:  E VALUATION C RITERIA
Revised:  E VALUATION C RITERIA

Original: In this section, we introduce the evaluation criteria used in this paper.
Revised: In this section, we introduce the evaluation criteria used in this paper.

Original: We denote the search space size as M , the true performances and approximated evaluated scores of architectures { a i } i =1 , ··· ,M as { y i } i =1 , ··· ,M and { s i } i =1 , ··· ,M , respectively.
Revised: We denote the search space size as M , the true performances and approximated evaluated scores of architectures { a i } i =1 , ··· ,M as { y i } i =1 , ··· ,M and { s i } i =1 , ··· ,M , respectively.

Original: And we denote the ranking of the true performance y i and the evaluated score s i as r i ∈ { 1 , · · · , M } and n i ∈ { 1 , · · · , M } ( r i = 1 indicates that a i is the best architecture in the search space).
Revised: And we denote the ranking of the true performance y i and the evaluated score s i as r i ∈ { 1 , · · · , M } and n i ∈ { 1 , · · · , M } ( r i = 1 indicates that a i is the best architecture in the search space).

Original: The correlation criteria adopted in our paper are
Revised: The correlation criteria adopted in our paper are

Original: • Linear correlation: The pearson correlation coefﬁcient corr ( y, s ) / (cid:112) corr ( y, y ) corr ( s, s ) .
Revised: • Linear correlation: The pearson correlation coefﬁcient corr ( y, s ) / (cid:112) corr ( y, y ) corr ( s, s ) .

Original: • Kendall’s
Revised: • Kendall’s

Original: Tau ranking correlation: The relative difference of concordant pairs and discordant pairs (cid:80) i<j sgn ( y i − y j ) sgn
Revised: Tau ranking correlation: The relative difference of concordant pairs and discordant pairs (cid:80) i<j sgn ( y i − y j ) sgn

Original: ( s i − s j ) / (cid:0) M 2 (cid:1) .
Revised: ( s i − s j ) / (cid:0) M 2 (cid:1) .

Original: • Spearman’s ranking correlation: The pearson correlation coefﬁcient between the rank variables corr ( r, n ) / (cid:112) corr ( r, r ) corr ( n, n ) .
Revised: • Spearman’s ranking correlation: The pearson correlation coefﬁcient between the rank variables corr ( r, n ) / (cid:112) corr ( r, r ) corr ( n, n ) .

Original: Besides these correlation criteria, criteria that emphasize more on the relative order of architectures with good performances are desired.
Revised: Besides these correlation criteria, criteria that emphasize more on the relative order of architectures with good performances are desired.

Original: Denoting A K = { a i | n i < KM } as the set of architectures whose evaluated scores s is among the top K portion of the search space, we use two criteira
Revised: Denoting A K = { a i | n i < KM } as the set of architectures whose evaluated scores s is among the top K portion of the search space, we use two criteira

Original: • Precision@K (P@K) ∈ (0 , 1] =
Revised: • Precision@K (P@K) ∈ (0 , 1] =

Original: # { i | r i <KM ∧ n i <KM } KM : The proportion of true top-K proportion architectures in the top-K architectures according to the scores.
Revised: # { i | r i <KM ∧ n i <KM } KM : The proportion of true top-K proportion architectures in the top-K architectures according to the scores.

Original: • BestRanking@K (BR@K) ∈
Revised: • BestRanking@K (BR@K) ∈

Original: (0 , 1] =
Revised: (0 , 1] =

Original: arg min i ∈
Revised: arg min i ∈

Original: A K r i : The best normalized ranking among the top K proportion of architectures according to the scores.
Revised: A K r i : The best normalized ranking among the top K proportion of architectures according to the scores.

Original: The two criteria are similar to those used in Ning et al.
Revised: The two criteria are similar to those used in Ning et al.

Original: (2020), except that rankings and architecture numbers are all normalized with respect to the search space size M .
Revised: (2020), except that rankings and architecture numbers are all normalized with respect to the search space size M .

Original: The above criteria are used to compare the ﬁtness of various architecture evaluators with different conﬁgurations and in different stages.
Revised: The above criteria are used to compare the ﬁtness of various architecture evaluators with different conﬁgurations and in different stages.

Original: Besides that, we’d also like to interpret their evaluation results.
Revised: Besides that, we’d also like to interpret their evaluation results.

Original: To identify which architectures are under- or over-estimated by various evaluators, and analyze the reasons accordingly, we investigate the relationship of the true-predicted ranking differences { r i − n i } i =1 , ··· ,M and the architecture properties such as the FLOPs: { FLOPs ( a i ) } i =1 , ··· ,M .
Revised: To identify which architectures are under- or over-estimated by various evaluators, and analyze the reasons accordingly, we investigate the relationship of the true-predicted ranking differences { r i − n i } i =1 , ··· ,M and the architecture properties such as the FLOPs: { FLOPs ( a i ) } i =1 , ··· ,M .

Original:  P ARAMETER - SHARING E VALUATORS
Revised:  P ARAMETER - SHARING E VALUATORS

Original: In this section, we assess the behavior of one-shot evaluators and evaluate the inﬂuence of several sampling strategies and training techniques.
Revised: In this section, we assess the behavior of one-shot evaluators and evaluate the inﬂuence of several sampling strategies and training techniques.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: During the supernet training process, candidate architectures are randomly sampled, and their corresponding weights are updated in every iteration.
Revised: During the supernet training process, candidate architectures are randomly sampled, and their corresponding weights are updated in every iteration.

Original: We conduct our experiments on CIFAR-10 using a recent NAS benchmarking search space, NASBench-201 (Dong & Yang, 2020).
Revised: We conduct our experiments on CIFAR-10 using a recent NAS benchmarking search space, NASBench-201 (Dong & Yang, 2020).

Original: NAS-Bench-201 is a NAS benchmark that provides the performances of all the 15625 architectures in a cell-based search space.
Revised: NAS-Bench-201 is a NAS benchmark that provides the performances of all the 15625 architectures in a cell-based search space.

Original: Actually, there are architectures with different matrix representations that are isomorphic (i.e., represent the same architecture) in this search space.
Revised: Actually, there are architectures with different matrix representations that are isomorphic (i.e., represent the same architecture) in this search space.

Original: As reported by their original paper, there are 6466 unique topology structures after de-isomorphism.
Revised: As reported by their original paper, there are 6466 unique topology structures after de-isomorphism.

Original: The hyper-parameters used to train all the parameter-sharing supernets are summarized in Tab. 1.
Revised: The hyper-parameters used to train all the parameter-sharing supernets are summarized in Tab. 1.

Original: We train parameter sharing evaluators via momentum SGD with momentum 0.9 and weight decay 0.0005.
Revised: We train parameter sharing evaluators via momentum SGD with momentum 0.9 and weight decay 0.0005.

Original: The batch size is set to 512.
Revised: The batch size is set to 512.

Original: The learning rate is set to 0.05 initially and decayed by 0. each time the supernet accuracy stops to increase for 30 epochs.
Revised: The learning rate is set to 0.05 initially and decayed by 0.5 each time the supernet accuracy stops to increase for 30 epochs.

Original: During training, the dropout rate  is set to 0.1, and the gradient norm is clipped to be less than 5.0.
Revised: During training, the dropout rate before the fully-connected classiﬁer is set to 0.1, and the gradient norm is clipped to be less than 5.0.

Original: 4. T REND OF D IFFERENT C RITERIA
Revised: 4. T REND OF D IFFERENT C RITERIA

Original: We inspect how BR@K, P@K, and the correlation criteria converge during the training process.
Revised: We inspect how BR@K, P@K, and the correlation criteria converge during the training process.

Original: We train a parameter sharing model with 17 layers and 16 initial channels on the de-isomorphic NASBench-201 search space.
Revised: We train a parameter sharing model with 17 layers and 16 initial channels on the de-isomorphic NASBench-201 search space.

Original: Shown in Fig.
Revised: Shown in Fig.

Original: 2(a), the speed of convergence is highly different.
Revised: 2(a), the speed of convergence is highly different.

Original: BR@K converges very fast.
Revised: BR@K converges very fast.

Original: P@K converges in around 250 epochs and then even gradually decreases.
Revised: P@K converges in around 250 epochs and then even gradually decreases.

Original: Meanwhile, linear correlation, Kendall’s Tau, and Spearman correlation are still growing tillepochs, while the parameter sharing accuracy grows during the whole 1000 epochs.
Revised: Meanwhile, linear correlation, Kendall’s Tau, and Spearman correlation are still growing tillepochs, while the parameter sharing accuracy grows during the whole 1000 epochs.

Original: This indicates optimizer
Revised: This indicates optimizer

Original: SGD initial LR 0.
Revised: SGD initial LR 0.

Original: momentum 0.
Revised: momentum 0.

Original: LR schedule
Revised: LR schedule

Original: ReduceLROnPlateau † weight decay 0.
Revised: ReduceLROnPlateau † weight decay 0.

Original: LR decay 0.
Revised: LR decay 0.

Original: batch size
Revised: batch size

Original: LR patience dropout rate 0.
Revised: LR patience dropout rate 0.

Original: grad norm clip 5.
Revised: grad norm clip 5.

Original: Learning rate is decayed by 0.5 when the loss stops decreasing forepochs, see https://pytorch.org/docs/stable/optim.html#torch.
Revised: Learning rate is decayed by 0.5 when the loss stops decreasing forepochs, see https://pytorch.org/docs/stable/optim.html#torch.

Original: optim.lr_scheduler.
Revised: optim.lr_scheduler.

Original: ReduceLROnPlateau .
Revised: ReduceLROnPlateau .

Original: [Figure]
Revised: [Figure]

Original: that the models with different rankings change at different speeds as the training progresses, and the top-ranked models stand out faster.
Revised: that the models with different rankings change at different speeds as the training progresses, and the top-ranked models stand out faster.

Original: Another evidence is shown in Fig.
Revised: Another evidence is shown in Fig.

Original: 2(b) that P@5% converges much faster than P@50%.
Revised: 2(b) that P@5% converges much faster than P@50%.

Original: This means that as the training goes on, the supernet is learning to compare architectures with medium or pool performances instead of good ones.
Revised: This means that as the training goes on, the supernet is learning to compare architectures with medium or poor performances instead of good ones.

Original: Another fact to note in Fig. 2(b) (also see Tab. 3) is that P@5% shows a slow decreasing trend from 200 epochs on.
Revised: Another fact to note in Fig. 2(b) (also see Tab. 3) is that P@5% shows a slow decreasing trend from 200 epochs on.

Original: This might be due to that, while the some best architectures stand out very fast in one-shot training, their one-shot performances will be caught up with by others as the training goes on
Revised: This might be due to that, while the some best architectures stand out very fast in one-shot training, their one-shot performances will be caught up with by others as the training goes on

Original: [Figure]
Revised: [Figure]

Original: 4. S AMPLING S TRATEGY
Revised: 4. S AMPLING S TRATEGY

Original: The NAS-Bench-201 search space includes many isomorphic architectures.
Revised: The NAS-Bench-201 search space includes many isomorphic architectures.

Original: We expect that one-shot evaluators could handle isomorphic architectures correctly, which means that their evaluated rewards should be close.
Revised: We expect that one-shot evaluators could handle isomorphic architectures correctly, which means that their evaluated rewards should be close.

Original: We calculate the average variances of test accuracy and ranking in isomorphic groups during the training process, as shown in Tab. 2.
Revised: We calculate the average variances of test accuracy and ranking in isomorphic groups during the training process, as shown in Tab. 2.

Original: As the training progresses, the variance
Revised: As the training progresses, the variance

Original: [Figure]
Revised: [Figure]

Original: within the group gradually shrinks, which indicates that more sufﬁcient training makes one-shot evaluator handle isomorphic architectures better.
Revised: within the group gradually shrinks, which indicates that more sufﬁcient training makes one-shot evaluator handle isomorphic architectures better.

Original: [Table]
Revised: [Table]

Original: We compare the results of sampling with/without isomorphic architectures during training in Tab. 3.
Revised: We compare the results of sampling with/without isomorphic architectures during training in Tab. 3.

Original: The results show that if de-isomorphism sampling is not used, the supernet performs much worse on good architectures in that BR@1%, BR@5% and P@5% are signiﬁcantly worse (2.515% V.S.
Revised: The results show that if de-isomorphism sampling is not used, the supernet performs much worse on good architectures in that BR@1%, BR@5% and P@5% are signiﬁcantly worse (2.515% V.S.

Original: 0.015%, 2.221% V.S. 0.015%, 9.22% V.S. 28.48%).
Revised: 0.015%, 2.221% V.S. 0.015%, 9.22% V.S. 28.48%).

Original: In this case, we ﬁnd that the top-ranked cell architectures are simple architectures (e.g., a single convolution).
Revised: In this case, we ﬁnd that the top-ranked cell architectures are simple architectures (e.g., a single convolution).

Original: That is to say, parameter sharing training without de-isomorphism training might over-estimate simple architectures.
Revised: That is to say, parameter sharing training without de-isomorphism training might over-estimate simple architectures.

Original: This might be due to that the equivalent sampling probability is larger for these simple architectures with many isomorphic counterparts.
Revised: This might be due to that the equivalent sampling probability is larger for these simple architectures with many isomorphic counterparts.

Original: We also experiment with the post de-isomorphism technique, in which the performances of architectures in each isomorphic group are averaged during testing, while no changes are incorporated in the training process.
Revised: We also experiment with the post de-isomorphism technique, in which the performances of architectures in each isomorphic group are averaged during testing, while no changes are incorporated in the training process.

Original: Surprisingly, we ﬁnd that post de-isomorphism results are almost as good as de-isomorphism sampling.
Revised: We ﬁnd that post de-isomorphism can achieve improvements over “No De-isomorphism”. This might owe to the decreased variance of the estimates.

Original: [Table]
Revised: [Table]

Original: Equivalent 1000 epochs
Revised: No De-isomorphism

Original: Fair-NAS (Chu et al., 2019)

Original:  BR@5%0.093% 0.139% 0.495% 0.139%
Revised: BR@ BR@1% BR@5%

Original: P@5% 24.77% 9.60% 20.74% 11.76% τ  0.1000 epochs
Revised: P@5% P@50% τ 3.372% 0.012% 0.006% 39.69% 79.27% 0.

Original: Fair-NAS (Chu et al., 2019)
Revised: [Table]

Original: BR@5% 0.093% 0.015% 0.124% 0.031%
Revised: [Table]

Original: P@5% 24.77% 14.24% 17.03% 15.17% r τ 0.
Revised: [Table] [Table]

Original: Tab. 4 shows the comparison of using different architecture sample numbers in supernet training.
Revised: De-isomorphism BR [Table] τ

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Post De-isomorphism

Revised: BR

Revised: [Table]

Revised: BR

Revised: [Table]

Revised: BR

Revised: [Table]

Revised: P

Revised: [Table]

Revised: P

Revised: [Table]

Revised: 50% τ 3.866% 0.015%

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Tab. 4 shows the comparison of using different architecture sample numbers in supernet training.

Original: We can see that using multiple architecture MC samples is not beneﬁcial, and MC sample=1 is a good choice.
Revised: We can see that using multiple architecture MC samples is not beneﬁcial, and MC sample=1 is a good choice.

Original: We also adapt Fair-NAS (Chu et al., 2019)
Revised: We also adapt Fair-NAS (Chu et al., 2019)

Original: sampling strategy to the NAS-Benchsearch space (a special case of MC sample 5), and ﬁnd it does not bring improvements.
Revised: sampling strategy to the NAS-Benchsearch space (a special case of MC sample 5), and ﬁnd it does not bring improvements.

Original: 4. P ROXY M ODEL
Revised: 4. P ROXY M ODEL

Original: Due to memory and time constraints, it is common to use a shallower or thinner proxy model in the search process.
Revised: Due to memory and time constraints, it is common to use a shallower or thinner proxy model in the search process.

Original: The common practice is to search using small proxy models with fewer channels and layers, and then “model augment” the discovered architecture to large neural networks.
Revised: The common practice is to search using small proxy models with fewer channels and layers, and then “model augment” the discovered architecture to large neural networks.

Original: From the experimental results shown in Fig.
Revised: From the experimental results shown in Fig.

Original: 3(a)(b), we can see that channel proxy has little inﬂuence while  layer proxy reduces the reliability of search results.
Revised: 3(a)(b), we can see that channel proxy has little inﬂuence while [Table] layer proxy reduces the reliability of search results.

Original: Thus, for cell-based search spaces, proxy-less search w.r.t the layer number is worth studying.
Revised: Thus, for cell-based search spaces, proxy-less search w.r.t the layer number is worth studying.

Original: 4. O VER - AND U NDER - ESTIMATION OF A RCHITECTURES
Revised: 4. O VER - AND U NDER - ESTIMATION OF A RCHITECTURES

Original: For one-shot evaluators, we expect that the training process is fair and balance for all architectures.
Revised: For one-shot evaluators [Table] we expect that the training process is fair and balance for all architectures.

Original: However, sub-architectures have different amounts of calculation, and they might converge with a different speed.
Revised: However, sub-architectures have different amounts of calculation [Table] and they might converge with a different speed.

Original: To understand which architectures are under- or over-estimated by the one-shot evaluators, we inspect the Ranking Diff of the ground truth performance and the one-shot evaluation of an architecture a i : r i − n i .
Revised: To understand which architectures are under- or over-estimated by the one-shot evaluators, we inspect the Ranking Diff of the ground truth performance and the one-shot evaluation of an architecture a i : r i − n i .

Original: We divide the architectures into ten groups according to the amount of calculation (FLOPs), and show Kendall’s Tau and Average Rank Diff of each group in Fig. 3(c).
Revised: We divide the architectures into ten groups according to the amount of calculation (FLOPs) [Table] and show Kendall’s Tau and Average Rank Diff of each group in Fig. 3(c).

Original: Note that a positive Ranking Diff indicates that this architecture is over-estimated, otherwise it is underestimated.
Revised: Note that a positive Ranking Diff indicates that this architecture is over-estimated, otherwise it is underestimated.

Original: The x-axis is organized such that the architecture group with the least FLOPs is at the leftmost.
Revised: The x-axis is organized such that the architecture group with the least FLOPs is at the leftmost.

Original: The average Rank Diff shows a decreasing trend, which means that the larger the model, the easier it is to be underestimated.
Revised: The average Rank Diff shows a decreasing trend, which means that the larger the model, the easier it is to be underestimated.

Original: Also, the decreasing intra-group Kendall’s Tau indicates that it is harder for the one-shot evaluator to compare larger models (which usually have better performances) than comparing smaller models.
Revised: Also, the decreasing intra-group Kendall’s Tau indicates that it is harder for the one-shot evaluator to compare larger models (which usually have better performances) than comparing smaller models.

Original:  P REDICTOR - BASED E VALUATORS
Revised:  P REDICTOR - BASED E VALUATORS

Original: In this section, we employ the same criteria (i.e., Kendall’s Tau, Precision@K, BestRanking@K) to assess the architecture predictors (i.e., predictor-based evaluators).
Revised: In this section, we employ the same criteria (i.e., Kendall’s Tau, Precision@K, BestRanking@K) to assess the architecture predictors (i.e., predictor-based evaluators).

Original: 5. E XPERIMENTAL S ETUP
Revised: 5. E XPERIMENTAL S ETUP

Original: We experiment with four different architecture predictors: MLP, LSTM, GATES (Ning et al., 2020), and a random forest regressor (RF).
Revised: We experiment with four different architecture predictors: MLP, LSTM, GATES (Ning et al., 2020), and a random forest regressor (RF).

Original: For MLP, LSTM, and RF, we serialize each architecture matrix using the six elements of its lower triangular portion.
Revised: For MLP, LSTM, and RF, we serialize each architecture matrix using the six elements of its lower triangular portion.

Original: We follow (Ning et al., 2020) to construct MLP,
Revised: We follow (Ning et al., 2020) to construct MLP,

Original: LSTM, and GATES.
Revised: LSTM, and GATES.

Original: The RF predictor applies a random forest regressor on the 6-dim sequence.
Revised: The RF predictor applies a random forest regressor on the 6-dim sequence.

Original: For optimizing MLP, LSTM, and GATES, an ADAM optimizer with a learning rate of 1e-3 is used, the batch size is 512, and the training lasts for 200 epochs.
Revised: For optimizing MLP, LSTM, and GATES, an ADAM optimizer with a learning rate of 1e-3 is used, the batch size is 512, and the training lasts for 200 epochs.

Original: Following (Ning et al., 2020), a hinge pairwise ranking loss with margin 0.1 is used for training these predictors.
Revised: Following (Ning et al., 2020), a hinge pairwise ranking loss with margin 0.1 is used for training these predictors.

Original: For RF, we use a random forest with 100 CARTs to predict architecture performances.
Revised: For RF, we use a random forest with 100 CARTs to predict architecture performances.

Original: 5. E VALUATION R ESULTS
Revised: 5. E VALUATION R ESULTS

Original: We train these predictors on training sets of different sizes: 39 (0.25%), 78 (0.5%), 390 (2.5%),(5%).
Revised: We train these predictors on training sets of different sizes: 39 (0.25%), 78 (0.5%), 390 (2.5%),(5%).

Original: Speciﬁcally, for each training set size, we randomly sample three different training sets, and train each predictor on each training set with three different random seeds (20, 2020, 202020).
Revised: Speciﬁcally, for each training set size, we randomly sample three different training sets, and train each predictor on each training set with three different random seeds (20, 2020, 202020).

Original: After training, we evaluate each model on the whole NAS-Bench-201 search space using Kendall’s Tau,
Revised: After training, we evaluate each model on the whole NAS-Bench-201 search space using Kendall’s Tau,

Original: Precision@K, and BestRanking@K. 1 As shown in Fig. 4, GATES outperforms other predictors in all settings.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: Precision@K, and BestRanking@K. Different from (Ning et al., 2020), the evaluation is carried out on all the architectures, instead of a separate validation split.

Revised: As shown in Fig.

Revised: 4, GATES outperforms other predictors in all settings.

Original: As can be seen, training with different seeds on different training sets leads to similar results.
Revised: As can be seen, training with different seeds on different training sets leads to similar results.

Original: In contrast, we found that training predictors with regression loss is not stable and sensitive to the choice of the training set.
Revised: In contrast, we found that training predictors with regression loss is not stable and sensitive to the choice of the training set.

Original: For example, the Kendall’s Taus of 3 GATES models trained using the regression loss on different training sets of size 78 are 0.7135, 0.7240, and 0.2067, respectively, while with ranking loss, the results are 0.7597, 0.7750, and 0.7645, respectively.
Revised: For example, the Kendall’s Taus of 3 GATES models trained using the regression loss on different training sets of size 78 are 0.7135, 0.7240, and 0.2067, respectively, while with ranking loss, the results are 0.7597, 0.7750, and 0.7645, respectively.

Original: These additional results are listed in the Appendix.
Revised: These additional results are listed in the Appendix.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: A1 in the Appendix, there are usually multiple stages in a predictor-based NAS search process.
Revised: A1 in the Appendix, there are usually multiple stages in a predictor-based NAS search process.

Original: In each stage i , the current predictor is used to choose K architectures to be evaluated by the oracle evaluator, then the results will be used to update the predictor.
Revised: In each stage i , the current predictor is used to choose N architectures to be evaluated by the oracle evaluator, then the results will be used to update the predictor.

Original: In our experiment, we select the top- K architectures predicted by the current predictor in the whole search space.
Revised: In our experiment, we select the top- N architectures predicted by the current predictor in the whole search space.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, P@0.1% increases a lot after multiple stages of training, which indicates that the predictors perform better and better on distinguishing the good architectures.
Revised: 5, P@0.1% increases a lot after multiple stages of training, which indicates that the predictors perform better and better on distinguishing the good architectures.

Original: This is expected since the training data are moreand more concentrated on the good architectures.
Revised: This is expected since the training data are more concentrated on the good architectures.

Original: Also, the performance gap between different predictors (LSTM and GATES) shrinks as more training data become available.
Revised: Also, the performance gap between different predictors (LSTM/GATES) shrinks as more training data become available.

Original: The Kendall’s Tau, BR@K, and performance distribution results of multiple stages are listed in the Appendix.
Revised: The Kendall’s Tau, BR@K, and performance distribution results of multiple stages are listed in the Appendix.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper assesses parameter-sharing evaluators and architecture predictors on the NAS-Benchsearch space, with a set of carefully developed criteria.
Revised: This paper assesses parameter-sharing evaluators and architecture predictors on the NAS-Benchsearch space, with a set of carefully developed criteria.

Original: Hopefully, knowledge revealed by this paper can guide future NAS application and motivate further research.
Revised: Hopefully, knowledge revealed by this paper can guide future NAS application and motivate further research.

Original: A A DDITIONAL R ESULTS ABOUT P REDICTOR - BASED E VALUATORS A. P REDICTOR P ERFORMANCES R ESULTS : S INGLE - AND MULTIPLE - STAGE Searcher
Revised: A A DDITIONAL R ESULTS ABOUT P REDICTOR - BASED E VALUATORS A. P REDICTOR P ERFORMANCES R ESULTS : S INGLE - AND MULTIPLE - STAGE Searcher

Original: Predictor
Revised: Predictor

Original: Ground-truth Performance
Revised: Ground-truth Performance

Original: Architecture
Revised: Architecture

Original: Oracle
Revised: Oracle

Original: Evaluator
Revised: Evaluator

Original: Update
Revised: Update

Original: Sample a. Exhaustive
Revised: Sample a. Exhaustive

Original: [Figure]
Revised: [Figure]

Original: b. Evolutionary c. Random
Revised: b. Evolutionary c. Random

Original: [Figure]
Revised: [Figure]

Original: d. Generator
Revised: d. Generator

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: evaluated, and used to train the predictor along with previous architecture data.
Revised: evaluated, and used to train the predictor along with previous architecture data.

Original: Note that in this table, K in BR@K is the absolute architecture number without normalization
Revised: Note that in this table, K in BR@K is the absolute architecture number without normalization

Original: [Table]
Revised: [Table]

Original: A. O VER - AND U NDER - ESTIMATION OF A RCHITECTURES
Revised: A. O VER - AND U NDER - ESTIMATION OF A RCHITECTURES

Original: Fig.
Revised: Fig.

Original: A2(d)(e)(f) illustrates the relationship between the FLOPs of architectures and how it is likely to be over-estimated.
Revised: A2(d)(e)(f) illustrates the relationship between the FLOPs of architectures and how it is likely to be over-estimated.

Original: It seems that MLP and RF are more likely to overestimate the smaller architectures and underestimate the larger ones, while LSTM and GATES show no obvious preference on the architectures’ FLOPs.
Revised: It seems that MLP and RF are more likely to overestimate the smaller architectures and underestimate the larger ones, while LSTM and GATES show no obvious preference on the architectures’ FLOPs.

Original: Fig.
Revised: Fig.

Original: A2(a)(b)(c) shows that GATES can give more accurate rankings on smaller architectures than larger architectures, which indicates that GATES might still have trouble in comparing larger architectures that usually have good performances.
Revised: A2(a)(b)(c) shows that GATES can give more accurate rankings on smaller architectures than larger architectures, which indicates that GATES might still have trouble in comparing larger architectures that usually have good performances.

Original: A. O NE - SHOT O RACLE E VALUATOR
Revised: A. O NE - SHOT O RACLE E VALUATOR

Original: Luo et al.
Revised: Luo et al.

Original: (2018) made an attempt to use a parameter-sharing evaluator as the oracle evaluator in
Revised: (2018) made an attempt to use a parameter-sharing evaluator as the oracle evaluator in

Original: Fig.
Revised: Fig.

Original: A1.
Revised: A1.

Original: That is to say, they use the noisy signals provided by the parameter sharing evaluator to train
Revised: That is to say, they use the noisy signals provided by the parameter sharing evaluator to train

Original: LSTMMLPGATESRF (a) 1 2 3 4 5 6 7 8 9 10 0.
Revised: LSTMMLPGATESRF (a) 1 2 3 4 5 6 7 8 9 10 0.

Original: LSTMMLPGATESRF (b) 1 2 3 4 5 6 7 8 90.
Revised: LSTMMLPGATESRF (b) 1 2 3 4 5 6 7 8 90.

Original: LSTMMLPGATESRF (c) 1 2 3 4 5 6 7 8 9 10 0. LSTMMLPGATESRF (d) 1 2 3 4 5 6 7 8 9 10 0.
Revised: LSTMMLPGATESRF (c) 1 2 3 4 5 6 7 8 9 10 0. LSTMMLPGATESRF (d) 1 2 3 4 5 6 7 8 9 10 0.

Original: LSTMMLPGATESRF (e) 1 2 3 4 5 6 7 8 9 10 0.
Revised: LSTMMLPGATESRF (e) 1 2 3 4 5 6 7 8 9 10 0.

Original: LSTMMLPGATESRF (f) the predictor.
Revised: LSTMMLPGATESRF (f) the predictor.

Original: This will signiﬁcantly accelerate the NAS process, compared with using an expensive traditional evaluator.
Revised: This will signiﬁcantly accelerate the NAS process, compared with using an expensive traditional evaluator.

Original: However, it is found to cause the NAS algorithm to fail to discover good architectures.
Revised: However, it is found to cause the NAS algorithm to fail to discover good architectures.

Original: Also, predictors have been used to accelerate parameter-sharing NAS methods (Li et al., 2020; Wang et al., 2020), since one predictor forward pass is faster than testing on the whole validation queue, even if no separate training phase is needed.
Revised: Also, predictors have been used to accelerate parameter-sharing NAS methods (Li et al., 2020; Wang et al., 2020), since one predictor forward pass is faster than testing on the whole validation queue, even if no separate training phase is needed.

Original: In this section, we explore whether a predictor can recover from the noisy training signals provided by the parameter-sharing evaluator.
Revised: In this section, we explore whether a predictor can recover from the noisy training signals provided by the parameter-sharing evaluator.

Original: Since GATES achieves consistently better results than other predictors, it is used in the following experiments.
Revised: Since GATES achieves consistently better results than other predictors, it is used in the following experiments.

Original: Speciﬁcally, we want to answer two questions:
Revised: Speciﬁcally, we want to answer two questions:

Original: Can sampling only a subset during supernet training help achieve better one-shot Kendall’s Tau on these architectures?
Revised: Can sampling only a subset during supernet training help achieve better one-shot Kendall’s Tau on these architectures?

Original: Can predictor training help recover from the noisy training signals provided by the one-shot evaluator?
Revised: Can predictor training help recover from the noisy training signals provided by the one-shot evaluator?

Original: We randomly sample 78 architectures from the search space.
Revised: We randomly sample 78 architectures from the search space.

Original: Two differently trained parametersharing evaluators are used to provide the one-shot instruction signal of these 78 architectures: 1)
Revised: Two differently trained parametersharing evaluators are used to provide the one-shot instruction signal of these 78 architectures: 1)

Original: Uniformly sampling from the whole search space, 2) Uniformly sampling from the 78 architectures.
Revised: Uniformly sampling from the whole search space, 2) Uniformly sampling from the 78 architectures.

Original: We ﬁnd that strategy 1 (sampling from the whole search space) can get a higher evaluation Kendall’s
Revised: We ﬁnd that strategy 1 (sampling from the whole search space) can get a higher evaluation Kendall’s

Original: Tau, no matter whether the evaluation is on the 78 architectures (0.657 V.S. 0.628) or the whole search space (0.701 V.S. 0.670).
Revised: Tau, no matter whether the evaluation is on the 78 architectures (0.657 V.S. 0.628) or the whole search space (0.701 V.S. 0.670).

Original: Thus the answer to Question 1 is “No”.
Revised: Thus the answer to Question 1 is “No”.

Original: Then, to answer the second question, we utilize the one-shot instruction signal provided by the supernet trained with 15625 architectures to train the predictor 2 .
Revised: Then, to answer the second question, we utilize the one-shot instruction signal provided by the supernet trained with 15625 architectures to train the predictor 1 .

Original: The Kendall’s Tau between the architecture scores given by the resulting predictor and the ground-truth performances is 0.718 on all the 15625 architectures, which is slightly worse than the one-shot instruction signals (0.719).
Revised: The Kendall’s Tau between the architecture scores given by the resulting predictor and the ground-truth performances is 0.718 on all the 15625 architectures, which is slightly worse than the one-shot instruction signals (0.719).

Original: More importantly, BR@1% degrades from 2.5% to 12.1%, thus the answer to Question 2 is “No’.
Revised: More importantly, BR@1% degrades from 2.5% to 12.1%, thus the answer to Question 2 is “No’.

Original: Thus, we conclude that although training a predictor using one-shot signals can bring acceleration, since no extra inference is needed during the search, it is not beneﬁcial in the sense of evaluation quality (especially of good architectures).
Revised: Thus, we conclude that although training a predictor using one-shot signals can bring acceleration, since no extra inference is needed during the search, it is not beneﬁcial in the sense of evaluation quality (especially of good architectures).

Original: Perhaps, incorporating more manual prior knowledge and regularizations can increase the denoising effect, which might be worth future research.
Revised: Perhaps, incorporating more manual prior knowledge and regularizations can increase the denoising effect, which might be worth future research.

Revised: B A DDITIONAL D ISCUSSION AND R ESULTS ABOUT O NE - SHOT E VALUATORS B. D ISCUSSION ON D E -I SOMORPHIC S AMPLING

Revised: To conduct isomorphic sampling, we ﬁrst design a simple encoding method, which can canonicalize computationally-isomorphic architectures to the same string and non-isomorphic architectures to different strings.

Revised: In our paper, we use the encoding method to ﬁnd out all isomorphic groups in the search space and make them as a table.

Revised: Then, during the supernet training process, we sample architecture groups from this table uniformly.

Revised: This method is feasible since the benchmark search space is not large.

Revised: In practice, one can use lazy table-making and rejection sampling to conduct de-isomorphic sampling, by only accepting new or representative architecture samples.

Revised: More speciﬁcally, one ﬁrst encodes each sampled architecture into a canonical string.

Revised: If this canonical string has not appeared before, this string stands for a new isomorphic group, and the architecture is recorded as the representative architecture for this isomorphic group.

Revised: This architecture sample is also accepted.

Revised: If this canonical string has been recorded before, we only accept the architecture sample if it is the representative architecture for its canonical string.

Revised: The simple encoding method is similar to the one in NASBench-201.

Revised: To be more speciﬁc, the encoding method goes as follows.

Revised: Denoting the expression of the i -th node as S i , and the operation in the directed edge (j,i) as A ji , the expression S i can be written as

Revised: [Equation]

Revised: , where P ( i ) denotes the set of predecessor nodes of i , and Sort sorts the strings in dictionary order.

Revised: We calculate S i ( i = 1 , · · · , 4) in topological order, and the expression S 4 at the ﬁnal output node is used as the encoding string of the architecture.

Revised: B. R ESULTS ON D E -I SOMORPHIC S AMPLING

Revised: [Figure]

Revised: Fig.

Revised: A3(a) shows that the ranking correlation keeps increasing throughout the training process.

Revised: Fig.

Revised: A3(b) indicates that P@50% increases with more sufﬁcient training, while P@5% slightly shrinks, especially the supernet trained without de-isomorphism sampling.

Revised: As also been analyzed in the main text, this might arise from the fact that simple architectures have more isomorphic counterparts.

Revised: Fig.

Revised: A3(c) shows that without de-isomorphic sampling, BR@1% and BR@5% deteriorate in the late stages of the training process.