Original: N EURAL A RCHITECTURE S EARCH ON I MAGE N ET IN F OUR GPU H OURS : A T HEORETICALLY I NSPIRED P ERSPECTIVE
Revised: N EURAL A RCHITECTURE S EARCH ON I MAGE N ET IN F OUR GPU H OURS : A T HEORETICALLY I NSPIRED P ERSPECTIVE

Original: Anonymous authors

Original: Paper under double-blind review

Original: Neural Architecture Search (NAS) has been explosively studied to automate the discovery of top-performer neural networks.
Revised: Neural Architecture Search (NAS) has been explosively studied to automate the discovery of top-performer neural networks.

Original: Current works require heavy training of supernet or intensive architecture evaluations, thus suffering from heavy resource consumption and often incurring search bias due to truncated training or approximations.
Revised: Current works require heavy training of supernet or intensive architecture evaluations, thus suffering from heavy resource consumption and often incurring search bias due to truncated training or approximations.

Original: Can we select the best neural architectures without involving any training and eliminate a drastic portion of the search cost?
Revised: Can we select the best neural architectures without involving any training and eliminate a drastic portion of the search cost?

Original: We provide an afﬁrmative answer, by proposing a novel framework called trainingfree neural architecture search ( TE-NAS ).
Revised: We provide an afﬁrmative answer, by proposing a novel framework called training-free neural architecture search ( TE-NAS ).

Original: TE-NAS ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK), and the number of linear regions in the input space.
Revised: TE-NAS ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space.

Original: Both are motivated by recent theory advances in deep networks, and can be computed without any training.
Revised: Both are motivated by recent theory advances in deep networks and can be computed without any training and any label.

Original: We show that: (1) these two measurements imply the trainability and expressivity of a neural network;and (2) they strongly correlate with the network’sactual test accuracy.
Revised: We show that: (1) these two measurements imply the trainability and expressivity of a neural network; (2) they strongly correlate with the network’s test accuracy.

Original: Further on, we design a pruning-based NAS mechanism to achieve a more ﬂexible and superior trade-off between the trainability and expressivity during the search.
Revised: Further on, we design a pruning-based NAS mechanism to achieve a more ﬂexible and superior trade-off between the trainability and expressivity during the search.

Original: In NASBench-201 and DARTS search spaces, TE-NAS completes high-quality search but only costs 0.5 and 4 GPU hours with one 1080Ti on CIFAR-10 and ImageNet, respectively.
Revised: In NAS-Benchand DARTS search spaces, TE-NAS completes high-quality search but only costs 0.5 and 4 GPU hours with one 1080Ti on CIFAR-10 and ImageNet, respectively.

Original: We hope our work to inspire more attempts in bridgingbetween the theoretic ﬁndings of deep networks and practical impacts in real NAS applications.
Revised: We hope our work inspires more attempts in bridging the theoretical ﬁndings of deep networks and practical impacts in real NAS applications.

Revised: Code is available at: https github com/VITA-Group/TENAS .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The recent development of deep networks signiﬁcantly contributes to the success of computer vision.
Revised: The recent development of deep networks signiﬁcantly contributes to the success of computer vision.

Original: Thanks to many efforts by human designers, the performance of deep networks have been signiﬁcantly boosted (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015;
Revised: Thanks to many efforts by human designers, the performance of deep networks have been signiﬁcantly boosted (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015;

Original: He et al., 2016;
Revised: He et al., 2016;

Original: Xie et al., 2017).
Revised: Xie et al., 2017).

Original: However, the manual creation of new network architectures not only costs enormous time and resources due to trial-and-error, but also depends on the design experience that does not always scale up.
Revised: However, the manual creation of new network architectures not only costs enormous time and resources due to trial-and-error, but also depends on the design experience that does not always scale up.

Original: To reduce the human efforts and costs, neural architecture search ( NAS ) has recently amassed explosive interests, leading to principled and automated discovery for good architectures in a given search space of candidates (Zoph & Le, 2016; Brock et al., 2017; Pham et al., 2018; Liu et al., 2018a; Chen et al., 2018; Bender et al., 2018).
Revised: To reduce the human efforts and costs, neural architecture search ( NAS ) has recently amassed explosive interests, leading to principled and automated discovery for good architectures in a given search space of candidates (Zoph & Le, 2016; Brock et al., 2017; Pham et al., 2018; Liu et al., 2018a; Chen et al., 2018; Bender et al., 2018; Gong et al., 2019; Chen et al., 2020a; Fu et al., 2020).

Original: As an optimization problem, NAS faces two core questions: 1) “ how to evaluate ”, i.e. the objective function that deﬁnes what are good architectures we want; 2) “ how to optimize ”, i.e. by what means we could effectively optimize the objective function.
Revised: As an optimization problem, NAS faces two core questions: 1) “ how to evaluate ”, i.e. the objective function that deﬁnes what are good architectures we want; 2) “ how to optimize ”, i.e. by what means we could effectively optimize the objective function.

Original: These two questions are entangled and highly non-trivial, since the search spaces are of extremely high dimension, and the generalization ability of architectures cannot be easily inferred (Dong & Yang, 2020;
Revised: These two questions are entangled and highly non-trivial, since the search spaces are of extremely high dimension, and the generalization ability of architectures cannot be easily inferred (Dong & Yang, 2020;

Original: Dong et al., 2020).
Revised: Dong et al., 2020).

Original: Existing NAS methods mainly leverage the validation set and conduct accuracy-driven architecture optimization.
Revised: Existing NAS methods mainly leverage the validation set and conduct accuracy-driven architecture optimization.

Original: They either formulate the search space as a super-network (“supernet”) and make the training loss differentiable through the architecture parameters (Liu et al., 2018b), or treat the architecture selection as a sequential decision making process (Zoph & Le, 2016) or evolution of genetics (Real et al., 2019).
Revised: They either formulate the search space as a super-network (“supernet”) and make the training loss differentiable through the architecture parameters (Liu et al., 2018b), or treat the architecture selection as a sequential decision making process (Zoph & Le, 2016) or evolution of genetics (Real et al., 2019).

Original: However, these NAS algorithms suffer from heavy consumption of both time and GPU resources.
Revised: However, these NAS algorithms suffer from heavy consumption of both time and GPU resources.

Original: Training a supernet till convergence is extremely slow, even with many effective heuristics for sampling or channel approximations (Dong & Yang, 2019; Xu et al., 2019).
Revised: Training a supernet till convergence is extremely slow, even with many effective heuristics for sampling or channel approximations (Dong & Yang, 2019; Xu et al., 2019).

Original: Approximated proxy inference such as truncated training/early stopping can accelerate the search, but is well known to introduce search bias to the inaccurate results obtained (Pham et al., 2018; Liang et al., 2019;
Revised: Approximated proxy inference such as truncated training/early stopping can accelerate the search, but is well known to introduce search bias to the inaccurate results obtained (Pham et al., 2018; Liang et al., 2019;

Original: Tan et al., 2020).
Revised: Tan et al., 2020).

Original: The heavy search cost not only slows down the discovery of novel architectures, but also blocks us from more meaningfully understanding the NAS behavors.
Revised: The heavy search cost not only slows down the discovery of novel architectures, but also blocks us from more meaningfully understanding the NAS behaviors.

Original: On the other hand, the analysis of neural network’s trainability (how effective a network can be optimized via gradient descent) and expressivity (how complex the function a network can represent) has witnessed exciting development recently in the deep learning theory ﬁelds.
Revised: On the other hand, the analysis of neural network’s trainability (how effective a network can be optimized via gradient descent) and expressivity (how complex the function a network can represent) has witnessed exciting development recently in the deep learning theory ﬁelds.

Original: By formulating neural networks as a Gaussian Process (no training involved), the gradient descent training dynamics can be characterized by the Neural Tangent Kernel (NTK) of inﬁnite (Lee et al., 2019) or ﬁnite (Yang, 2019) width networks, from which several useful measures can be derived to depict the network trainability at the initialization.
Revised: By formulating neural networks as a Gaussian Process (no training involved), the gradient descent training dynamics can be characterized by the Neural Tangent Kernel (NTK) of inﬁnite (Lee et al., 2019) or ﬁnite (Yang, 2019) width networks, from which several useful measures can be derived to depict the network trainability at the initialization.

Original: Hanin & Rolnick (2019a;b); Xiong et al.
Revised: Hanin & Rolnick (2019a;b); Xiong et al.

Original: (2020) describe another measure of network expressivity, also without any training, by counting the number of unique linear regions that a neural network can divide in its input space.
Revised: (2020) describe another measure of network expressivity, also without any training, by counting the number of unique linear regions that a neural network can divide in its input space.

Original: We are therefore inspired to ask:
Revised: We are therefore inspired to ask:

Original: • How to optimize neural architectures at initialization without involving any training, thus signiﬁcantly eliminating a heavy portion of the search cost?
Revised: • How to optimize neural architectures at initialization without involving any training, thus signiﬁcantly eliminating a heavy portion of the search cost?

Original: • Can we deﬁne how to evaluate in NAS by analyzing the trainability and expressivity of architectures, and further beneﬁt our understanding of the search process?
Revised: • Can we deﬁne how to evaluate in NAS by analyzing the trainability and expressivity of architectures, and further beneﬁt our understanding of the search process?

Original: Our answers are yes to both questions.
Revised: Our answers are yes to both questions.

Original: In this work, we propose TE-NAS, a framework for trainingfree neural architecture search.
Revised: In this work, we propose TE-NAS, a framework for trainingfree neural architecture search.

Original: We leverage two indicators , the condition number of NTK and the number of linear regions, that can decouple and effectively characterize the trainability and expressivity of architectures respectively inthe complex NAS search space.
Revised: We leverage two indicators , the condition number of NTK and the number of linear regions, that can decouple and effectively characterize the trainability and expressivity of architectures respectively in complex NAS search spaces.

Original: Most importantly, these two indicators can be measured in a training-free  manner, thus largely accelerates the NAS search process and beneﬁts the understanding of discovered architectures.
Revised: Most importantly, these two indicators can be measured in a training-free and label-free manner, thus largely accelerates the NAS search process and beneﬁts the understanding of discovered architectures.

Original: To our best knowledge, TE-NAS makes the ﬁrst attempt to bridgebetween the theoretical ﬁndings of deep neural networks andthe real-world NAS applications.
Revised: To our best knowledge, TE-NAS makes the ﬁrst attempt to bridge the theoretical ﬁndings of deep neural networks and real-world NAS applications.

Original: While we intend not to claim that the two indicators we use are the only nor the best options, we hope our work to open a door to theoretically-inspired NAS and inspire the discovery of more deep network indicators.
Revised: While we intend not to claim that the two indicators we use are the only nor the best options, we hope our work opens a door to theoretically-inspired NAS and inspires the discovery of more deep network indicators.

Original: Our contributions are summarized as below:
Revised: Our contributions are summarized as below:

Original: • We identify and investigate two training-free  indicators to rank the quality of deep architectures: the spectrum of their NTKs, and the number of linear regions in their input space.
Revised: • We identify and investigate two training-free and label-free indicators to rank the quality of deep architectures: the spectrum of their NTKs, and the number of linear regions in their input space.

Original: Our study ﬁnds that they reliably indicate the trainability and expressivity of a deep network respectively, and are strongly correlated with the network’s test accuracy.
Revised: Our study ﬁnds that they reliably indicate the trainability and expressivity of a deep network respectively, and are strongly correlated with the network’s test accuracy.

Original: • We leverage the above two theoretically-inspired indicators to establish a training-free NAS framework, TE-NAS , therefore eliminating a drastic portion of the search cost.
Revised: • We leverage the above two theoretically-inspired indicators to establish a training-free NAS framework, TE-NAS , therefore eliminating a drastic portion of the search cost.

Original: We further introduce a pruning-based mechanism, to boost search efﬁciency and to more ﬂexibly trade-off between trainability and expressivity.
Revised: We further introduce a pruning-based mechanism, to boost search efﬁciency and to more ﬂexibly trade-off between trainability and expressivity.

Original: • In NAS-Bench-201/DARTS search spaces, TE-NAS discovers architectures with a strong performance at remarkably lower search costs, compared to previous efforts.
Revised: • In NAS-Bench-201/DARTS search spaces, TE-NAS discovers architectures with a strong performance at remarkably lower search costs, compared to previous efforts.

Original: With just one 1080Ti, it only costs 0.5 GPU hours to search on CIFAR10, and 4 GPU hours on ImageNet, respectively, setting the new record for ultra-efﬁcient yet high-quality NAS.
Revised: With just one 1080Ti, it only costs 0.5 GPU hours to search on CIFAR and 4 GPU hours on ImageNet, respectively, setting the new record for ultra-efﬁcient yet high-quality NAS.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Neural architecture search (NAS) is recently proposed to accelerate the principled and automated discovery of high-performance networks.
Revised: Neural architecture search (NAS) is recently proposed to accelerate the principled and automated discovery of high-performance networks.

Original: However, most works suffer from heavy search cost, for both weight-sharing based methods (Liu et al., 2018b; Dong & Yang, 2019; Liu et al., 2019; Yu et al., 2020a; Li et al.,  2020a) and single-path sampling-based methods (Pham et al., 2018; Guo et al., 2019; Real et al., 2019; Tan et al., 2020; Li et al., 2020b; Yang et al., 2020).
Revised: However, most works suffer from heavy search cost, for both weight-sharing based methods (Liu et al., 2018b; Dong & Yang, 2019; Liu et al., 2019; Yu et al., 2020a; Li et al., 2020a; Yang et al., 2020a) and single-path sampling-based methods (Pham et al., 2018; Guo et al., 2019; Real et al., 2019; Tan et al., 2020; Li et al., 2020c; Yang et al., 2020b).

Original: A one-shot super network can share its parameters to sampled sub-networks and accelerate the architecture evaluations, but it is very heavy and hard to optimize and sufferes from  poor correlation between its accuracy and those of the sub-networks (Yu et al., 2020c).
Revised: A one-shot super network can share its parameters to sampled sub-networks and accelerate the architecture evaluations, but it is very heavy and hard to optimize and suffers from a poor correlation between its accuracy and those of the sub-networks (Yu et al., 2020c).

Original: Sampling-based methods achieve more accurate architecture evaluations, but their truncated training still impose bias to the performance ranking since this is based on the results of early training stages.
Revised: Sampling-based methods achieve more accurate architecture evaluations, but their truncated training still imposes bias to the performance ranking since this is based on the results of early training stages.

Original: Instead of estimating architecture performance by direct training, people also try to predict network’s accuracy (or ranking), called predictor based NAS methods (Liu et al., 2018a;
Revised: Instead of estimating architecture performance by direct training, people also try to predict network’s accuracy (or ranking), called predictor based NAS methods (Liu et al., 2018a;

Original: Luo et al., 2018; Dai et al., 2019; Luo et al., 2020).
Revised: Luo et al., 2018; Dai et al., 2019; Luo et al., 2020).

Original: Graph neural network (GNN) is a popular choice as the predictor model (Wen et al., 2019;
Revised: Graph neural network (GNN) is a popular choice as the predictor model (Wen et al., 2019;

Original: Chen et al., 2020).
Revised: Chen et al., 2020b).

Original: Siems et al.
Revised: Siems et al.

Original: (2020) even propose the ﬁrst large scale surrogate benchmark, where most of the architectures’ accuracies are predicted by a pretrained GNN predictor.
Revised: (2020) even propose the ﬁrst large-scale surrogate benchmark, where most of the architectures’ accuracies are predicted by a pretrained GNN predictor.

Original: The learned predictor can achieve highly accurate performance evaluation.
Revised: The learned predictor can achieve highly accurate performance evaluation.

Original: However, the data collection step - sampling representative architectures and train them till converge - still requires extremely high cost.
Revised: However, the data collection step - sampling representative architectures and train them till converge - still requires extremely high cost.

Original: People have to sample and train 2,000 to 50,000 architectures to serve as the training data for the predictor.
Revised: People have to sample and train 2,000 to 50,000 architectures to serve as the training data for the predictor.

Original: Moreover, none of these works can demonstrate the cross-space transferability of their predictors.
Revised: Moreover, none of these works can demonstrate the cross-space transferability of their predictors.

Original: This means one has to repeat the data collection and predictor training whenever facing an unseen search space, which is highly nonscalable.
Revised: This means one has to repeat the data collection and predictor training whenever facing an unseen search space, which is highly nonscalable.

Original: The heavy cost of architecture evaluation hinders the understanding of the NAS search process.
Revised: The heavy cost of architecture evaluation hinders the understanding of the NAS search process.

Original: Recent pioneer works like Shu et al.
Revised: Recent pioneer works like Shu et al.

Original: (2019) observed that DARTS and ENAS tend to favor architectures with wide and shallow cell structures due to their smooth loss landscape.
Revised: (2019) observed that DARTS and ENAS tend to favor architectures with wide and shallow cell structures due to their smooth loss landscape.

Original: Siems et al.
Revised: Siems et al.

Original: (2020) studied the distribution of test error for different cell depths and numbers of parameter-free operators.
Revised: (2020) studied the distribution of test error for different cell depths and numbers of parameter-free operators.

Original: Chen & Hsieh (2020) for the ﬁrst time regularizes the Hessian norm of the validation loss, and visualizes the smoother loss landscape of the supernet. 
Revised: Chen & Hsieh (2020) for the ﬁrst time regularizes the Hessian norm of the validation loss and visualizes the smoother loss landscape of the supernet. Li et al.

Revised: (2020b) proposed to approximate the validation loss landscape by learning a mapping from neural architectures to their corresponding validate losses.

Original: Still, these analyses cannot be directly leveraged to guide the design of network architectures.
Revised: Still, these analyses cannot be directly leveraged to guide the design of network architectures.

Original: Mellor et al.
Revised: Mellor et al.

Original: (2020) recently proposed a NAS framework that does not involve training, which shares the same motivation with us towards training-free architecture search at initialization.
Revised: (2020) recently proposed a NAS framework that does not involve training, which shares the same motivation with us towards training-free architecture search at initialization.

Original: They empirically ﬁnd that the correlation between sample-wise input-output Jacobian can indicate the architecture’s test performance.
Revised: They empirically ﬁnd that the correlation between sample-wise input-output Jacobian can indicate the architecture’s test performance.

Original: However, why does the Jacobian work is not well explained and demonstrated.
Revised: However, why does the Jacobian work is not well explained and demonstrated.

Original: Their search performance on NAs-Bench-201 is still left behind by the state-of-the-art
Revised: Their search performance on NAS-Bench-201 is still left behind by the state-of-the-art

Original: NAS works, and they did not extend to DARTs space.
Revised: NAS works, and they did not extend to DARTs space.

Original: Meanwhile, we see the evolving development of deep learning theory on neural networks.
Revised: Meanwhile, we see the evolving development of deep learning theory on neural networks.

Original: NTK (neural tangent kernel) is proposed to characterize the gradient descent training dynamics of inﬁnite wide (Jacot et al., 2018b) or ﬁnite wide deep networks (Hanin & Nica, 2019).
Revised: NTK (neural tangent kernel) is proposed to characterize the gradient descent training dynamics of inﬁnite wide (Jacot et al., 2018) or ﬁnite wide deep networks (Hanin & Nica, 2019).

Original: Wide networks are also proved to evolve as linear models under gradient descent (Lee et al., 2019).
Revised: Wide networks are also proved to evolve as linear models under gradient descent (Lee et al., 2019).

Original: This is further leveraged to decouple the trainability and generalization of networks (Xiao et al., 2019).
Revised: This is further leveraged to decouple the trainability and generalization of networks (Xiao et al., 2019).

Original: Besides, a natural measure of  network’s expressivity is the number of linear regions it can separate in its input space (Raghu et al., 2017; Mont ´ ufar, 2017; Serra et al., 2018; Hanin & Rolnick, 2019a;b; Xiong et al., 2020).
Revised: Besides, a natural measure of ReLU network’s expressivity is the number of linear regions it can separate in its input space (Raghu et al., 2017; Mont ´ ufar, 2017; Serra et al., 2018; Hanin & Rolnick, 2019a;b; Xiong et al., 2020).

Original: In our work, we for the ﬁrst time discover two important indicators that can effectively rank architectures, thus bridging the theoretic ﬁndings and real-world NAS applications.
Revised: In our work, we for the ﬁrst time discover two important indicators that can effectively rank architectures, thus bridging the theoretic ﬁndings and real-world NAS applications.

Original: Instead of claiming the two indicators we discover are the best, we believe there are more meaningful properties of deep networks that can beneﬁt the architecture search process.
Revised: Instead of claiming the two indicators we discover are the best, we believe there are more meaningful properties of deep networks that can beneﬁt the architecture search process.

Original: We leave them as open question and encourage the community to study.
Revised: We leave them as open questions and encourage the community to study.

Original:  M ETHODS
Revised:  M ETHODS

Original: The core motivation of our TE-NAS framework is to achieve architecture evaluation without involving any training, to signiﬁcantly accelerate the NAS search process and reduce the search cost.
Revised: The core motivation of our TE-NAS framework is to achieve architecture evaluation without involving any training, to signiﬁcantly accelerate the NAS search process and reduce the search cost.

Original: In section 3.1 we present our study on two important indicators that reﬂect the trainability and expressivity of a neural network, and in section 3.2 we design a novel pruning-based method that can achieve a superior trade-off between the two indicators.
Revised: In section 3.1 we present our study on two important indicators that reﬂect the trainability and expressivity of a neural network, and in section 3.2 we design a novel pruning-based method that can achieve a superior trade-off between the two indicators.

Original: 3. A NALYZING T RAINABILITY AND E XPRESSIVITY OF D EEP N ETWORKS
Revised: 3. A NALYZING T RAINABILITY AND E XPRESSIVITY OF D EEP N ETWORKS

Original: Trainability and expressivity are distinct notions regarding a neural network (Xiao et al., 2019).
Revised: Trainability and expressivity are distinct notions regarding a neural network (Xiao et al., 2019).

Original: A network can achieve high performance only if the function it can represent is complex enough and at the same time it can be effectively trained by gradient descent.
Revised: A network can achieve high performance only if the function it can represent is complex enough and at the same time, it can be effectively trained by gradient descent.

Original: 3.1. T RAINABILITY BY C ONDITION N UMBER OF NTK
Revised: 3.1. T RAINABILITY BY C ONDITION N UMBER OF NTK

Original: The trainability of a neural network indicates how effective it can be optimized using gradient descent (Burkholz & Dubatovka, 2019;
Revised: The trainability of a neural network indicates how effective it can be optimized using gradient descent (Burkholz & Dubatovka, 2019;

Original: Hayou et al., 2019; Shin & Karniadakis, 2020).
Revised: Hayou et al., 2019; Shin & Karniadakis, 2020).

Original: Although some heavy networks can theoretically represent complex functions, they not necessarily can be effectively trained by gradient descent.
Revised: Although some heavy networks can theoretically represent complex functions, they not necessarily can be effectively trained by gradient descent.

Original: One typical example is that, even with  much more number of parameters, Vgg networks (Simonyan & Zisserman, 2014) usually perform worse and require more special engineering tricks compared with ResNet family (He et al., 2016), whose superior trainability property is studied by Yang & Schoenholz (2017).
Revised: One typical example is that, even with a much more number of parameters, Vgg networks (Simonyan & Zisserman, 2014) usually perform worse and require more special engineering tricks compared with ResNet family (He et al., 2016), whose superior trainability property is studied by Yang & Schoenholz (2017).

Original: Recent work (Jacot et al., 2018a; Lee et al., 2019; Chizat et al., 2019) studied the gradient descent training of neural networks using a quantity called the neural tangent kernel (NTK).
Revised: Recent work (Jacot et al., 2018; Lee et al., 2019; Chizat et al., 2019) studied the gradient descent training of neural networks using a quantity called the neural tangent kernel (NTK).

Original: The ﬁnite width
Revised: The ﬁnite width

Original: NTK is deﬁned by ˆΘ( x , x (cid:48) )
Revised: NTK is deﬁned by ˆΘ( x , x (cid:48) )

Original: =
Revised: =

Original: J ( x ) J ( x (cid:48) )
Revised: J ( x ) J ( x (cid:48) )

Original: T , where J iα ( x ) =
Revised: T , where J iα ( x ) =

Original: ∂ θ α z Li
Revised: ∂ θ α z Li

Original: ( x ) is the Jacobian evaluated at a point x for parameter θ α , and z Li is the output of the i -th neuron in the last output layer L .
Revised: ( x ) is the Jacobian evaluated at a point x for parameter θ α , and z Li is the output of the i -th neuron in the last output layer L .

Original: Lee et al.
Revised: Lee et al.

Original: (2019) further proves that wide neural networks evolve as linear models using gradient descent, and their training dynamics is controlled by ODEs that can be solved as 
Revised: (2019) further proves that wide neural networks evolve as linear models using gradient descent, and their training dynamics is controlled by ODEs that can be solved as [Equation] [Figure] [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original:  for training data.
Revised: ) for training data.

Original: Here µ t ( x ) =
Revised: Here µ t ( x ) =

Original: E [ z Li ( x )] is the expected outputs of an inﬁnitely wide network.
Revised: E [ z Li ( x )] is the expected outputs of an inﬁnitely wide network.

Original: ˆΘ train denotes the NTK between the training inputs, and X train and Y train are drawn from the training set
Revised: ˆΘ train denotes the NTK between the training inputs, and X train and Y train are drawn from the training set

Original: D train .
Revised: D train .

Original: As the training step t tends to inﬁnity we can see that Eq.
Revised: As the training step t tends to inﬁnity we can see that Eq.

Original: 1 reduce to µ ( X train ) =
Revised: 1 reduce to µ ( X train ) =

Original: Y train .
Revised: Y train .

Original: The relationship between the conditioning of ˆΘ and the trainability of networks is studied by Xiao et al.
Revised: The relationship between the conditioning of ˆΘ and the trainability of networks is studied by Xiao et al.

Original: and we brief the conclusion as below.
Revised: and we brief the conclusion as below.

Original: We can write Eq.
Revised: We can write Eq.

Original: 1 in terms of the spectrum of Θ :
Revised: 1 in terms of the spectrum of Θ :

Original:  [Equation]
Revised: [Figure] [Equation] [Figure] [Equation]

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where λ i are the eigenvalues of ˆ Θ train and we order the eigenvalues λ 0 ≥ · · · ≥ λ m .
Revised: where λ i are the eigenvalues of ˆΘ train and we order the eigenvalues λ 0 ≥ · · · ≥ λ m .

Original: As it has been hypothesized by Lee et al.
Revised: As it has been hypothesized by Lee et al.

Original: that the maximum feasible learning rate scales as η ∼ 2 /λ 0 , plugging this scaling for η into Eq.
Revised: that the maximum feasible learning rate scales as η ∼ 2 /λ 0 , plugging this scaling for η into Eq.

Original: 2 we see that the λ m will converge exponentially at a rate given by 1 /κ , where κ = λ 0 /λ m is the condition number.
Revised: 2 we see that the λ m will converge exponentially at a rate given by 1 /κ , where κ = λ 0 /λ m is the condition number.

Original: Then we can conclude that if the κ of the NTK associated with a neural network diverges then it will become untrainable, so we use κ as a metric for trainability:
Revised: Then we can conclude that if the κ of the NTK associated with a neural network diverges then it will become untrainable, so we use κ as a metric for trainability:

Original: [Equation]
Revised: [Equation]

Revised: κ N is calculated without any gradient descent or label.

Original: Figure 1 demonstrates that the κ N is negatively correlated with the architecture’s test accuracy, with the Kendall-tau correlation as − 0 .
Revised: Figure 1 demonstrates that the κ N is negatively correlated with the architecture’s test accuracy, with the Kendall-tau correlation as − 0 .

Original: Therefore, minimizing the κ N during the search will encourage the discovery of architectures with high performance.
Revised: Therefore, minimizing the κ N during the search will encourage the discovery of architectures with high performance.

Original: 3.1. E XPRESSIVITY BY N UMBER OF L INEAR R EGIONS
Revised: 3.1. E XPRESSIVITY BY N UMBER OF L INEAR R EGIONS

Original: The expressivity of a neural network indicates how complex the function it can represent (Hornik et al., 1989;
Revised: The expressivity of a neural network indicates how complex the function it can represent (Hornik et al., 1989;

Original: Giryes et al., 2016).
Revised: Giryes et al., 2016).

Original: For ReLU networks, each ReLU function deﬁnes a linear boundary and divides its input space into two regions.
Revised: For ReLU networks, each ReLU function deﬁnes a linear boundary and divides its input space into two regions.

Original: Since the composition of piecewise linear functions is still piecewise linear, every ReLU network can be seen as a piecewise linear function.
Revised: Since the composition of piecewise linear functions is still piecewise linear, every ReLU network can be seen as a piecewise linear function.

Original: The input space of a ReLU network can be partitioned into distinct pieces (i.e. linear regions) (Figure 2), each of which is associated with a set of afﬁne parameters, and the function represented by the network is afﬁne when restricted to each piece.
Revised: The input space of a ReLU network can be partitioned into distinct pieces (i.e. linear regions) (Figure 2), each of which is associated with a set of afﬁne parameters, and the function represented by the network is afﬁne when restricted to each piece.

Original: Therefore, it is natural to measure the expressivity of a ReLU network with the number of linear regions it can separate.
Revised: Therefore, it is natural to measure the expressivity of a ReLU network with the number of linear regions it can separate.

Original: Following Raghu et al.
Revised: Following Raghu et al.

Original: (2017); Mont´ufar (2017); Serra et al.
Revised: (2017); Mont´ufar (2017); Serra et al.

Original: ; Hanin & Rolnick (2019a;b);
Revised: ; Hanin & Rolnick (2019a;b);

Original: Xiong et al.
Revised: Xiong et al.

Original: (2020), we ﬁrst introduce the following deﬁnition of activation patterns and linear regions for ReLU CNNs.
Revised: (2020), we ﬁrst introduce the following deﬁnition of activation patterns and linear regions for ReLU CNNs.

Original: Deﬁnition 1 of Xiong et al.
Revised: Deﬁnition 1.

Original: (2020) (Activation Patterns and Linear Regions) Let N be a ReLU
Revised: Activation Patterns and Linear Regions (Xiong et al. Let N be a ReLU

Original: CNN.
Revised: CNN.

Original: An activation pattern of N is a function P from the set of neurons to { 1 , − 1 } , i.e., for each neuron z in N , we have P ( z ) ∈ { 1 , − 1 } .
Revised: An activation pattern of N is a function P from the set of neurons to { 1 , − 1 } , i.e., for each neuron z in N , we have P ( z ) ∈ { 1 , − 1 } .

Original: Let θ be a ﬁxed set of parameters (weights and biases) in
Revised: Let θ be a ﬁxed set of parameters (weights and biases) in

Original: N , and P be an activation pattern.
Revised: N , and P be an activation pattern.

Original: The region corresponding to P and θ is
Revised: The region corresponding to P and θ is

Original: [Equation]
Revised: [Equation]

Original: where z ( x 0 ; θ ) is the pre-activation of a neuron z .
Revised: where z ( x 0 ; θ ) is the pre-activation of a neuron z .

Original: Let R N ,θ denote the number of linear regions of
Revised: Let R N ,θ denote the number of linear regions of

Original: N at θ , i.e., R N ,θ :=
Revised: N at θ , i.e., R N ,θ :=

Original: # { R ( P ; θ ) :
Revised: # { R ( P ; θ ) :

Original: R ( P ; θ ) (cid:54) = ∅ for some activation pattern P } .
Revised: R ( P ; θ ) (cid:54) = ∅ for some activation pattern P } .

Original: [Figure]
Revised: [Figure]

Original: Eq.
Revised: Eq.

Original: 4 tells us that a linear region in the input space is a set of input data x 0 that satisﬁes a certain ﬁxed activation pattern P ( z ) , and therefore the number of linear regions
Revised: 4 tells us that a linear region in the input space is a set of input data x 0 that satisﬁes a certain ﬁxed activation pattern P ( z ) , and therefore the number of linear regions

Original: R N ,θ measures how many unique activation patterns that can be divided by the network
Revised: R N ,θ measures how many unique activation patterns that can be divided by the network.

Original: [Figure]

Original: In our work
Revised: In our work

Original: [Figure]
Revised: [Figure]

Original: wedraw network parameters from the standard gaussian distribution θ ∼ N (0 , 1) , repeat the measurement of the number of linear regions 
Revised: we repeat the measurement of the number of linear regions by sampling network parameters from the Kaiming Norm Initialization (He et al., 2015)

Original: [Figure]
Revised: [Figure]

Original: and calculate the average as the approximation to its expectation:
Revised: and calculate the average as the approximation to its expectation:

Original: [Figure]
Revised: [Figure]

Original: We iterate through all architectures in NAS-Bench-
Revised: We iterate through all architectures in NAS-Bench-

Original: (Dong & Yang, 2020), and calculate their numbers of linear regions.
Revised: (Dong & Yang, 2020) [Figure] and calculate their numbers of linear regions (without any gradient descent or label).

Original: Figure 3 demonstrates that the number of linear regions is positively correlated with the architecture’s test accuracy, with the Kendall-tau correlation as 0 .
Revised: Figure 3 demonstrates that the number of linear regions is positively correlated with the architecture’s test accuracy, with the Kendall-tau correlation as 0 .

Original: Therefore, maximizing the number of linear regions during the search will also encourage the discovery of architectures with high performance.
Revised: Therefore, maximizing the number of linear regions during the search will also encourage the discovery of architectures with high performance.

Original: Finally, in Figure 4 we analyze the operator composition of top 10% architecture by maximizing  R N and minimizing κ N , respectively.
Revised: Finally, in Figure 4 we analyze the operator composition of top 10% architecture by maximizing ˆ R N and minimizing κ N , respectively.

Original: We can clearly see that 
Revised: We can clearly see that ˆ

Original: R N and κ N have different preferences to choosing operators.
Revised: R N and κ N have different preferences for choosing operators.

Original: They both choose a large ratio of conv 3 × 3 for high generalization performance.
Revised: They both choose a large ratio of conv 3 × 3 for high generalization performance.

Original: But meanwhile,  R N heavily selects conv 1 × 1 , and κ N leads to skip-connect, favoring the gradient ﬂow.
Revised: But meanwhile, ˆ R N heavily selects conv 1 × 1 , and κ N leads to skip-connect, favoring the gradient ﬂow.

Original: 3. P RUNING - BY - IMPORTANCE A RCHITECTURE S EARCH
Revised: 3. P RUNING - BY - IMPORTANCE A RCHITECTURE S EARCH

Original: Given the strong correlation between the architecture’s test accuracy and its κ N and  R N , how to build an efﬁcient NAS framework on top of them?
Revised: Given the strong correlation between the architecture’s test accuracy and its κ N and ˆ R N , how to build an efﬁcient NAS framework on top of them?

Original: We motivate this section by addressing two questions:
Revised: We motivate this section by addressing two questions:

Original: How to combine κ N and  R N together, with  good explicit trade-off?
Revised: How to combine κ N and ˆ R N together, with a good explicit trade-off?

Original: We ﬁrst need to turn the two measurements κ N and  R N into one combined function, based on which we can rank architectures.
Revised: We ﬁrst need to turn the two measurements κ N and ˆ R N into one combined function, based on which we can rank architectures.

Original: As seen in Figure 1 and 3, the magnitudes of κ N and  R N differ much.
Revised: As seen in Figure 1 and 3, the magnitudes of κ N and ˆ R N differ much.

Original: To avoid one overwhelming the other numerically, one possible remedy is normalization; but we cannot pre-know the ranges nor the value distributions of κ N and  R N , before computing them over a search space.
Revised: To avoid one overwhelming the other numerically, one possible remedy is normalization; but we cannot pre-know the ranges nor the value distributions of κ N and ˆ R N , before computing them over a search space.

Original: In order to make our combined function well deﬁned before search and agnostic to the search space , instead of using the numerical values of κ N / R N , we could refer to their relative rankings.
Revised: In order to make our combined function well deﬁned before search and agnostic to the search space , instead of using the numerical values of κ N and ˆ R N , we could refer to their relative rankings.

Original: Speciﬁcally, each time by comparing the sampled set of architectures peer-to-peer, we can directly sum up the two relative rankings of κ N and  R N as the selection criterion.
Revised: Speciﬁcally, each time by comparing the sampled set of architectures peer-to-peer, we can directly sum up the two relative rankings of κ N and ˆ R N as the selection criterion.

Original: The equal-weight summation treats trainability and expressivity with the same importance conceptually 1 and delivers the best empirical result: we thus choose it as our default combined function.
Revised: The equal-weight summation treats trainability and expressivity with the same importance conceptually 1 and delivers the best empirical result: we thus choose it as our default combined function.

Original: We also tried some other means to combine the two, and the ablation studies can be found in Appendix D.2.
Revised: We also tried some other means to combine the two, and the ablation studies can be found in Appendix D.2.

Original: 2. How to search more efﬁciently?
Revised: 2. How to search more efﬁciently?

Original: Sampling-based methods like reinforcement learning or evolution can use rankings as the reward or ﬁltering metric.
Revised: Sampling-based methods like reinforcement learning or evolution can use rankings as the reward or ﬁltering metric.

Original: However, they are inefﬁcient, especially for complex cell-based search space.
Revised: However, they are inefﬁcient, especially for complex cell-based search space.

Original: Consider an network stacked by repeated cells (directed acyclic graphs) (Zoph et al., 2018;
Revised: Consider a network stacked by repeated cells (directed acyclic graphs) (Zoph et al., 2018;

Original: Liu et al., 2018b).
Revised: Liu et al., 2018b).

Original: Each cell has E edges, and on each edge we only select one operator out of |O| ( O is the set of operator candidates).
Revised: Each cell has E edges, and on each edge we only select one operator out of |O| ( O is the set of operator candidates).

Original: There are |O|
Revised: There are |O|

Original: E unique cells, and for sampling-based methods, γ · |O| E networks have to be sampled during the search.
Revised: E unique cells, and for sampling-based methods, γ · |O| E networks have to be sampled during the search.

Original: The ratio γ can be interpreted as the sampling efﬁciency: a method with small γ can ﬁnd good architectures faster.
Revised: The ratio γ can be interpreted as the sampling efﬁciency: a method with small γ can ﬁnd good architectures faster.

Original: However, the search time cost of sampling-based methods still scales up with the size of the search space, i.e., |O| E .
Revised: However, the search time cost of sampling-based methods still scales up with the size of the search space, i.e., |O| E .

Original: Inspired by recent works on pruning-from-scratch (Lee et al., 2018; Wang et al., 2020), we propose a pruning-by-importance NAS mechanism to quickly shrink the search possibilities and boost the efﬁciency further, reducing the cost from |O| E to |O| · E .
Revised: Inspired by recent works on pruning-from-scratch (Lee et al., 2018; Wang et al., 2020), we propose a pruning-by-importance NAS mechanism to quickly shrink the search possibilities and boost the efﬁciency further, reducing the cost from |O| E to |O| · E .

Original: Speciﬁcally, we start the search with a super-network
Revised: Speciﬁcally, we start the search with a super-network

Original: N 0 composed of all possible operators and edges.
Revised: N 0 composed of all possible operators and edges.

Original: In the outer loop, for every round we prune one operator on each edge.
Revised: In the outer loop, for every round we prune one operator on each edge.

Original: The outer-loop stops when the current supernet N t is a single-path network 2 , i.e. the algorithm will return us the ﬁnal searched architecture.
Revised: The outer-loop stops when the current supernet N t is a single-path network 2 , i.e., the algorithm will return us the ﬁnal searched architecture.

Original: For the inner-loop, we measure the change of κ N and  R N before and after pruning each individual operator, and assess its importance using the sum of two ranks.
Revised: For the inner-loop, we measure the change of κ N and ˆ R N before and after pruning each individual operator, and assess its importance using the sum of two ranks.

Original: We order all currently available operators in terms of their importance, and prune the lowest-importance operator on each edge.
Revised: We order all currently available operators in terms of their importance, and prune the lowest-importance operator on each edge.

Original: The whole pruning process is extremely fast.
Revised: The whole pruning process is extremely fast.

Original: As we will demonstrate later, our approach is principled and can be applied to different spaces without making any modiﬁcations.
Revised: As we will demonstrate later, our approach is principled and can be applied to different spaces without making any modiﬁcations.

Original: This pruning-by-importance mechanism may also be extended to indicators beyond κ N and  R N .
Revised: This pruning-by-importance mechanism may also be extended to indicators beyond κ N and ˆ R N .

Original: We summarize our training-free and pruning-based NAS framework, TE-NAS, in Algorithm 1.
Revised: We summarize our training-free and pruning-based NAS framework, TE-NAS, in Algorithm 1.

Original: [Table]
Revised: [Table]

Revised: [∆ κ t,o 1 , ..., ∆ κ t,o |N t | ]

Revised: Get importance by R N : s R ( o j ) = index of o j in ascendingly sorted list [∆ R t,o 1 , ..., ∆ R t,o |N t | ]

Revised: Get importance s ( o j )

Revised: =

Revised: s κ ( o j )

Revised: + s R

Revised: ( o j )

Revised: N t +1 =

Revised: N tfor each edge e i , i = 1 , ..., E do j ∗ =

Revised: arg min j { s ( o j ) :

Revised: o j ∈ e i } (cid:46) ﬁnd the operator with greatest importance on each edge.

Revised: N t +1 =

Revised: N t +1 \

Revised: o j ∗ t =

Revised: t +14 return Pruned single-path network N t .

Original: 3.2. V ISUALIZATION OF S EARCH P ROCESS
Revised: 3.2. V ISUALIZATION OF S EARCH P ROCESS

Original: [Figure]
Revised: [Figure]

Original: TE-NAS beneﬁts us towards a better understanding of the search process.
Revised: TE-NAS beneﬁts us towards a better understanding of the search process.

Original: We can analyze the trajectory of κ N and  R N during the search.
Revised: We can analyze the trajectory of κ N and ˆ R N during the search.

Original: It is worth noting that our starting point N 0 , the un-pruned supernet, is assumed to be of the highest expressivity (as it is composed of all operators in the search space and has the largest number of parameters and
Revised: It is worth noting that our starting point N 0 , the un-pruned supernet, is assumed to be of the highest expressivity (as it is composed of all operators in the search space and has the largest number of parameters and

Original: ReLU functions).
Revised: ReLU functions).

Original: However, it has poor trainability, as people ﬁnd many engineering techniques are required to effectively training the supernet (Yu et al., 2020a;b).
Revised: However, it has poor trainability, as people ﬁnd many engineering techniques are required to effectively training the supernet (Yu et al., 2020a;b).

Original: Therefore, during pruning we are expecting to strengthen the trainability of the supernet, while retaining its expressivity as much as possible.
Revised: Therefore, during pruning we are expecting to strengthen the trainability of the supernet, while retaining its expressivity as much as possible.

Original: As we observe in Figure 5, the supernet N is ﬁrst pruned by quickly reducing κ N , i.e., increasing the network’s trainability.
Revised: As we observe in Figure 5, the supernet N is ﬁrst pruned by quickly reducing κ N , i.e., increasing the network’s trainability.

Original: After that, as the improvement of κ N is almost plateaued, the method carefully ﬁne-tunes the architecture without sacriﬁcing too much expressivity  R N .
Revised: After that, as the improvement of κ N is almost plateaued, the method carefully ﬁne-tunes the architecture without sacriﬁcing too much expressivity ˆ R N .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we evaluate our TE-NAS on two search spaces: NAS-Bench-201 (Dong & Yang, 2020) and DARTS (Liu et al., 2018b).
Revised: we evaluate our TE-NAS on two search spaces: NAS-Bench-201 (Dong & Yang, 2020) and DARTS (Liu et al., 2018b).

Original: Search and training protocols are summarized in Appendix A.
Revised: Search and training protocols are summarized in Appendix A.

Original: All our codes and pre-trained models will be released upon acceptance.
Revised: Our code is available at: https://github.com/VITA-Group/TENAS .

Original: 4. R ESULTS ON NAS-B ENCH - 
Revised: 4. R ESULTS ON NAS-B ENCH -

Original: NAS-Bench-201 (Dong & Yang, 2020) provides a standard cell-based search space (containing 15,625 architectures) and a database of architecture’s performance evaluated under a uniﬁed protocol.
Revised: NAS-Bench-201 (Dong & Yang, 2020) provides a standard cell-based search space (containing 15,625 architectures) and a database of architecture’s performance evaluated under a uniﬁed protocol.

Original: The network’s test accuracy can be directly obtained by querying the database, which facilitates people to focus on studying NAS algorithms without network evaluation.
Revised: The network’s test accuracy can be directly obtained by querying the database, which facilitates people to focus on studying NAS algorithms without network evaluation.

Original: NAS-Bench-201 supports three datasets (CIFAR-10, CIFAR-100, ImageNet-16-120 (Chrabaszcz et al., 2017)).
Revised: NAS-Bench-201 supports three datasets (CIFAR-10, CIFAR-100, ImageNet-16-120 (Chrabaszcz et al., 2017)).

Original: The operation space contains none ( zero ), skip connection , conv 1 × 1 , conv 3 × 3 convolution
Revised: The operation space contains none ( zero ), skip connection , conv 1 × 1 , conv 3 × 3 convolution

Original: [Table]
Revised: [Table]

Original: and average pooling 3 × 3 .
Revised: and average pooling 3 × 3 .

Original: We refer to their paper for details of the space.
Revised: We refer to their paper for details of the space.

Original: Our search is dataset-speciﬁc,
Revised: Our search is dataset-speciﬁc,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. the search and evaluation are conducted on the same dataset.
Revised: e. the search and evaluation are conducted on the same dataset.

Original: [Table]
Revised: [Table]

Original: We run TE-NAS for four independent times with different random seeds
Revised: We run TE-NAS for four independent times with different random seeds

Original: [Table]
Revised: [Table]

Original: and report the mean and standard deviation in Table 1.
Revised: and report the mean and standard deviation in Table 1.

Original: We can see that TE-NAS achieves the best accuracy on all three datasets
Revised: We can see that TE-NAS achieves the best accuracy on all three datasets

Original: [Table]
Revised: [Table]

Original: and largely reduces the search cost ( 5 × ∼ 19 × reduction).
Revised: and largely reduces the search cost ( 5 × ∼ 19 × reduction).

Original: Although Mellor et al.
Revised: Although Mellor et al.

Original: (2020) requires even less search time (by only sampling 25 architectures)
Revised: (2020) requires even less search time (by only sampling 25 architectures)

Original: [Table]
Revised: [Table]

Original: they suffer from much inferior accuracy performance, with notably larger deviations across different search rounds.
Revised: they suffer from much inferior accuracy performance [Table] with notably larger deviations across different search rounds.

Original: 4. R ESULTS ON CIFAR-10 WITH DART S S EARCH S PACE
Revised: 4. R ESULTS ON CIFAR-10 WITH DART S S EARCH S PACE

Original: Architecture Space
Revised: Architecture Space

Original: The DARTs operation space O contains eight choices: none ( zero ), skip connection , separable convolution 3 × 3 and 5 × 5 , dilated separable convolution 3 × 3 and 5 × 5 , max pooling× 3 , average pooling 3 × 3 .
Revised: The DARTs operation space O contains eight choices: none ( zero ), skip connection , separable convolution 3 × 3 and 5 × 5 , dilated separable convolution 3 × 3 and 5 × 5 , max pooling× 3 , average pooling 3 × 3 .

Original: Following previous works (Liu et al., 2018b; Chen et al., 2019; Xu et al., 2019), forboth search and evaluation phases, we stack 20 cells to compose the 
Revised: Following previous works (Liu et al., 2018b; Chen et al., 2019; Xu et al., 2019), for evaluation phases, we stack 20 cells to compose the network and set the

Original: [Table]
Revised: [Table]

Original: network and set the initial channel number as 36.
Revised: initial channel number as 36.

Original: We place the reduction cells at the 1/3 and 2/3 of the network and each cell consists of six nodes.
Revised: We place the reduction cells at the 1/3 and 2/3 of the network and each cell consists of six nodes.

Original: Results We run TE-NAS for four independent times with different random seeds, and report the mean and standard deviation.
Revised: Results We run TE-NAS for four independent times with different random seeds, and report the mean and standard deviation.

Original: Table 2 summarizes the performance of TE-NAS compared with other popular NAS methods.
Revised: Table 2 summarizes the performance of TE-NAS compared with other popular NAS methods.

Original: TE-NAS achieves a test error of 2.63%, ranking among the top of recent NAS results, but meanwhile largely reduces the search cost to only 30 minutes.
Revised: TE-NAS achieves a test error of 2.63%, ranking among the top of recent NAS results, but meanwhile largely reduces the search cost to only 0.05 GPU-day.

Original: ProxylessNAS achieves  lowest test error, but it searches on a different space with a much longer search time and has a larger model size.
Revised: ProxylessNAS achieves the lowest test error, but it searches on a different space with a much longer search time and has a larger model size.

Original: Besides, Mellor et al.
Revised: Besides, Mellor et al.

Original: (2020) did not extend to their Jacobian-based framework to DARTs search space for CIFAR-10 or ImageNet classiﬁcation.
Revised: (2020) did not extend to their Jacobian-based framework to DARTs search space for CIFAR-10 or ImageNet classiﬁcation.

Original: 4. R ESULTS ON I MAGE N ET WITH DART S S EARCH S PACE
Revised: 4. R ESULTS ON I MAGE N ET WITH DART S S EARCH S PACE

Original: Architecture Space
Revised: Architecture Space

Original: Following previous works (Xu et al., 2019; Chen et al., 2019), the architecture for ImageNet is slightly different from that for CIFAR-10.
Revised: Following previous works (Xu et al., 2019; Chen et al., 2019), the architecture for ImageNet is slightly different from that for CIFAR-10.

Original: The network is stacked with 14 cells with the initial channel number set to 48. During retraining evaluation, we follow the mobile setting to control the FLOPs not exceed 600 MB by adjusting the channel number.
Revised: During retraining evaluation, the network is stacked with 14 cells with the initial channel number set to 48, and we follow the mobile setting to control the FLOPs not exceed 600 MB by adjusting the channel number.

Original: The spatial resolution is downscaled from 224 × 224 to 28 × 28 with the ﬁrst three convolution layers of stride 2.
Revised: The spatial resolution is downscaled from 224 × 224 to 28 × 28 with the ﬁrst three convolution layers of stride 2.

Original: Results As shown in Table 3, we achieve a top-1/5 test error of 26.0%/8.3%, achieving competitive performance with recent state-of-the-art works in the ImageNet mobile setting.
Revised: Results As shown in Table 3, we achieve a top-1/5 test error of 24.5%/7.5%, achieving competitive performance with recent state-of-the-art works in the ImageNet mobile setting.

Original: However, Te-NAS only cost four GPU hours with only one 1080Ti.
Revised: However, TE-NAS only cost four GPU hours with only one 1080Ti.

Original: Searching on ImageNet takes  longer time than on
Revised: Searching on ImageNet takes a longer time than on

Original: CIFAR-10 due to the larger input size and more network parameters.
Revised: CIFAR-10 due to the larger input size and more network parameters.

Original: [Table]
Revised: [Table]

Original: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The key questions in Neural Architecture Search (NAS) are “what are good architectures” and “how to ﬁnd them”.
Revised: The key questions in Neural Architecture Search (NAS) are “what are good architectures” and “how to ﬁnd them”.

Original: Validation loss or accuracy are possible answers but not enough, due to their search bias and heavy evaluation cost.
Revised: Validation loss or accuracy are possible answers but not enough, due to their search bias and heavy evaluation cost.

Original: Our work demonstrates that two theoretically inspired indicators, the spectrum of NTK and  number of linear regions, not only strongly correlate with the network’s performance, but also beneﬁt the reduced search cost and decoupled analysis of the network’s trainability and expressivity.
Revised: Our work demonstrates that two theoretically inspired indicators, the spectrum of NTK and the number of linear regions, not only strongly correlate with the network’s performance, but also beneﬁt the reduced search cost and decoupled analysis of the network’s trainability and expressivity.

Original: Without involving any training, our TE-NAS achieve competitive NAS performance with minimum search time.
Revised: Without involving any training, our TE-NAS achieve competitive NAS performance with minimum search time.

Original: We for the ﬁrst time bridge the gap between the theoretic ﬁndings of deep neural networks andthe real-world NAS applications, and we encourage the community to further explore more meaningful network properties so that we will have a better understanding of good architectures and how to search them. 
Revised: We for the ﬁrst time bridge the gap between the theoretic ﬁndings of deep neural networks and real-world NAS applications, and we encourage the community to further explore more meaningful network properties so that we will have a better understanding of good architectures and how to search them. A CKNOWLEDGEMENT

Revised: This work is supported in part by the NSF Real-Time Machine Learning program (Award Number: 2053279), and the US Army Research Ofﬁce Young Investigator Award (W911NF2010240).

Original: A I MPLEMENTATION D ETAILS
Revised: A I MPLEMENTATION D ETAILS

Original: For κ N we sample one mini-batch of size 32 from the training set, and calculate ˆΘ( x , x (cid:48) )
Revised: For κ N we sample one mini-batch of size 32 from the training set, and calculate ˆΘ( x , x (cid:48) )

Original: =
Revised: =

Original: J ( x ) J ( x (cid:48) ) T .
Revised: J ( x ) J ( x (cid:48) ) T .

Original: For  R N we sample 5000 images, forward them through the network, and collect the activation patterns from all ReLU layers.
Revised: For ˆ R N we sample 5000 images, forward them through the network, and collect the activation patterns from all ReLU layers.

Original: The calculation of both κ N and  R N are repeatedfor three times in all experiments, where each time the network weights are randomly drawn from standard
Revised: The calculation of both κ N and ˆ R N are repeated three times in all experiments, where each time the network weights are randomly drawn from Kaiming

Original: Guassian N (0 , 1) without involving any training (network weights are ﬁxed).
Revised: Norm Initialization (He et al., 2015) without involving any training (network weights are ﬁxed).

Original: Our retraining settings (after search) follow previous works (Xu et al., 2019; Chen et al., 2019; Chen & Hsieh, 2020).
Revised: Our retraining settings (after search) follow previous works (Xu et al., 2019; Chen et al., 2019; Chen & Hsieh, 2020).

Original: On CIFAR-10, we train the searched network with cutout regularization of length 16, drop-path
Revised: On CIFAR-10, we train the searched network with cutout regularization of length 16, drop-path

Original: (Zoph et al., 2018) with probability as 0.3, and an auxiliary tower of weight 0.4.
Revised: (Zoph et al., 2018) with probability as 0.3, and an auxiliary tower of weight 0.4.

Original: On
Revised: On

Original: ImageNet, we also use label smoothing during training.
Revised: ImageNet, we also use label smoothing during training.

Original: On both CIFAR-10 and ImageNet, the network is optimized by an SGD optimizer with cosine annealing, with learning rate initialized as 0.025 and 0.5, respectively.
Revised: On both CIFAR-10 and ImageNet, the network is optimized by an SGD optimizer with cosine annealing, with learning rate initialized as 0.025 and 0.5, respectively.

Original: B S EARCHED A RCHITECTURE
Revised: B S EARCHED A RCHITECTURE

Original: We visualize the searched normal and reduction cells in ﬁgure 6 and 7, which is directly searched on
Revised: We visualize the searched normal and reduction cells in ﬁgure 6 and 7, which is directly searched on

Original: CIFAR-10 and ImageNet respectively.
Revised: CIFAR-10 and ImageNet respectively.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C D EPTH AND W IDTH P REFERENCE OF κ N AND  R N IN DART S S PACE
Revised: C D EPTH AND W IDTH P REFERENCE OF κ N AND ˆ R N IN DART S S PACE

Original: To analysis the impact of different architectures on trainability and expressivity in DARTs search space [Figure] we visualize κ N and  R N with different depths and width.
Revised: To analyze the impact of different architectures on trainability and expressivity in DARTs search space, we visualize κ N and ˆ R N with different depths and width.

Original: Following Shu et al.
Revised: Following Shu et al.

Original: [Figure] the depth of a cell is deﬁned as the number of connections on the longest path from input nodes to the output node [Figure] and the width of a cell is the summation of the edges of the intermediate nodes that are connected to the input nodes.
Revised: (2019), the depth of a cell is deﬁned as the number of connections on the longest path from input nodes to the output node, and the width of a cell is the summation of the edges of the intermediate nodes that are connected to the input nodes.

Original: We randomly sample 20,000 architectures in DARTs space [Figure] and plot the visualizations in Figure 8.
Revised: We randomly sample 20,000 architectures in DARTs space, and plot the visualizations in Figure 8.

Original: Good architectures should exhibit low κ N (good trainability, blue dots in Figure 8(a)) and high  R N (powerful expressivity, red dots in Figure 8(b)).
Revised: Good architectures should exhibit low κ N (good trainability, blue dots in Figure 8(a)) and high ˆ R N (powerful expressivity, red dots in Figure 8(b)).

Original: Therefore, κ N and 
Revised: Therefore, κ N and ˆ

Original: R N tell us that in DARTs space shallow but wide cells are preferred to favor both trainability and expressivity.
Revised: R N tell us that in DARTs space shallow but wide cells are preferred to favor both trainability and expressivity.

Original: This conclusion matches the ﬁndings by Shu et al.
Revised: This conclusion matches the ﬁndings by Shu et al.

Original: (2019): existing NAS algorithms tend to favor architectures with wide and shallow cell structures, which enjoy fast convergence with smooth loss landscape and accurate gradient information
Revised: (2019): existing NAS algorithms tend to favor architectures with wide and shallow cell structures, which enjoy fast convergence with smooth loss landscape and accurate gradient information.

Original: 0 Depth
Revised: 0 Depth

Original: W i d t h
Revised: W i d t h

Original: N T K κ = λ
Revised: N T K κ = λ

Original: m a x
Revised: m a x

Original: λ m i n ( l o w e r t
Revised: λ m i n ( l o w e r t

Original: h e b e tt e r ) (a) κ N 1 .1 .2 .2 .3 .

Revised: h e b e tt e r ) (a) κ N 1 .1 .2 .2 .3 .

Original: 0 Depth
Revised: 0 Depth

Original: W i d t h 3244 .

Revised: W i d t h 3244 .

Original: .3246 .ˆ
Revised: .3246 .ˆ

Original: R N : #
Revised: R N : #

Original: L i n e a r R e g i o n s ( h i g h e r t
Revised: L i n e a r R e g i o n s ( h i g h e r t

Original: h e b e tt e r ) (b)  R N
Revised: h e b e tt e r ) (b) ˆ R N

Original: D M ORE A BLATION S TUDIES D. S EARCH WITH ONLY κ N OR  R N
Revised: D M ORE A BLATION S TUDIES D. S EARCH WITH ONLY κ N OR ˆ R N

Original: As we observed in Table 4, searching with only κ N or  R N leads to inferior performance, which indicates the importance of maintaining both trainability and expressivity during the search.
Revised: As we observed in Table 4, searching with only κ N or ˆ R N leads to inferior performance, which indicates the importance of maintaining both trainability and expressivity during the search.

Original: [Table]
Revised: [Table]

Original: D. D IFFERENT COMBINATION OPTIONS FOR κ N AND  R N
Revised: D IFFERENT COMBINATION OPTIONS FOR κ N AND ˆ R N

Original: Pruning by s ( o j )
Revised: Pruning by s ( o j )

Original: =
Revised: =

Original: s κ ( o j )
Revised: s κ ( o j )

Original: + s R
Revised: + s R

Original: ( o j ) is not the only option (see Algorithm 1).
Revised: ( o j ) is not the only option (see Algorithm 1).

Original: Here in this study we consider more:
Revised: Here in this study we consider more:

Original: 1) pruning by s ( o j ) =
Revised: 1) pruning by s ( o j ) =

Original: min( s κ ( o j ) , s R ( o j )) , i.e., pruning by the worst case.
Revised: min( s κ ( o j ) , s R ( o j )) , i.e., pruning by the worst case.

Original: 2) Pruninng by s ( o j ) =
Revised: 2) Pruninng by s ( o j ) =

Original: max( s κ ( o j ) , s R ( o j )) , i.e., pruning by the best case.
Revised: max( s κ ( o j ) , s R ( o j )) , i.e., pruning by the best case.

Original: 3) pruning by summation of changes ∆ κ t,o j + ∆ R t,o j , i.e., directly use the numerical values of the changes.
Revised: 3) pruning by summation of changes ∆ κ t,o j + ∆ R t,o j , i.e., directly use the numerical values of the changes.

Original: As we observed in Table
Revised: As we observed in Table

Original: [Table]
Revised: [Table]

Original: our TE-NAS stands out of all options.
Revised: our TE-NAS stands out of all options.

Original: This means a good trade-off between κ N and  R N are important
Revised: This means a good trade-off between κ N and ˆ R N are important

Original: [Table]
Revised: [Table]

Original: and also the ranking strategy is better than directly using numerical values.
Revised: and also the ranking strategy is better than directly using numerical values

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: D. C ORRELATION BETWEEN T EST A CCURACY AND C OMBINATION OF κ N AND  R N
Revised: D. C ORRELATION BETWEEN T EST A CCURACY AND C OMBINATION OF κ N AND ˆ R N

Original: [Figure]
Revised: [Figure]

Original: Figure 9 indicates that by using the summation of  ranking of both κ N and 
Revised: Figure 9 indicates that by using the summation of the ranking of both κ N and ˆ

Original: R N , the combined metric achieves  much higher correlation with the test accuracy.
Revised: R N , the combined metric achieves a much higher correlation with the test accuracy.

Original: The reason can be explained by Figure 4, as κ N and  R N prefers different operators in terms of trainability and Expressivity.
Revised: The reason can be explained by Figure 4, as κ N and ˆ R N prefers different operators in terms of trainability and Expressivity.

Original: Their combination can ﬁlter out bad architectures in both aspects and strongly correlate with networks’ ﬁnal performance
Revised: Their combination can ﬁlter out bad architectures in both aspects and strongly correlate with networks’ ﬁnal performance

Original: [Figure]
Revised: [Figure]

Original: E G ENERALIZATION V . S . T EST A CCURACY
Revised: E G ENERALIZATION V . S . T EST A CCURACY

Original: [Figure]
Revised: [Figure]

Original: Conceptually, the generalization gap is the difference between a model’s performance on training data and its performance on unseen data drawn from the same distribution (e.g., testing set).
Revised: Conceptually, the generalization gap is the difference between a model’s performance on training data and its performance on unseen data drawn from the same distribution (e.g., testing set).

Original: In comparison, the two indicators κ N (trainability) and R N (expressiveness) of a networkbasically determine how well the training set could be ﬁt (i.e., training set accuracy)
Revised: In comparison, the two indicators κ N (trainability) and R N (expressiveness) of a network determine how well the training set could be ﬁt (i

Original: [Figure]
Revised: [Figure]

Original:  and do not directly indicate its generalization gap (or test set accuracy).
Revised: e., training set accuracy), and do not directly indicate its generalization gap (or test set accuracy).

Original: Indeed, probing generalization of an untrained network at its initialization is a daunting [Table] that seems to go beyond  current theory scope.
Revised: Indeed, probing generalization of an untrained network at its initialization is a daunting, open challenge that seems to go beyond the current theory scope.

Original: In NAS, we are searching for the architecture with the best test accuracy.
Revised: In NAS, we are searching for the architecture with the best test accuracy.

Original: As shown in Figure 10, in NAS-Bench201 the training accuracy strongly correlates with test accuracy.
Revised: As shown in Figure 10, in NAS-Bench201 the training accuracy strongly correlates with test accuracy.

Original: This also seems to be a result of the current standard search space design that could have implicitly excluded severe overﬁtting.
Revised: This also seems to be a result of the current standard search space design that could have implicitly excluded severe overﬁtting.

Original: This explains why κ N and R N , which only focusing on trainability and expressiveness during training, can still achieve good search results of test accuracy.
Revised: This explains why κ N and R N , which only focuses on trainability and expressiveness during training, can still achieve good search results of test accuracy.