Original: FTSO: E FFECTIVE NAS VIA F IRST T OPOLOGY S ECOND O PERATOR
Revised: FTSO: E FFECTIVE NAS VIA F IRST T OPOLOGY S ECOND O PERATOR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing one-shot neural architecture search (NAS) methods generally contain a giant supernet, which leads to heavy computational costs.
Revised: Existing one-shot neural architecture search (NAS) methods generally contain a giant super-net, which leads to heavy computational costs.

Original: Our method, named
Revised: Our method, named

Original: FTSO, separates the whole architecture search into two sub-steps.
Revised: FTSO, separates the whole architecture search into two sub-steps.

Original: In the ﬁrst step, we only search for the topology, and in the second step, we only search for the operators.
Revised: In the ﬁrst step, we only search for the topology, and in the second step, we only search for the operators.

Original: FTSO not only reduces NAS’s search time from days to 0.68 seconds, but also signiﬁcantly improves the accuracy.
Revised: FTSO not only reduces NAS’s search time from days to 0.68 seconds, but also signiﬁcantly improves the accuracy.

Original: Speciﬁcally, our experiments on
Revised: Speciﬁcally, our experiments on

Original: ImageNet show that within merely 18 seconds, FTSO can achieve 76.4% testing accuracy, 1.5% higher than the baseline, PC-DARTS.
Revised: ImageNet show that within merely 18 seconds, FTSO can achieve 76.4% testing accuracy, 1.5% higher than the baseline, PC-DARTS.

Original: In addition, FTSO can reach 97.77% testing accuracy, 0.27% higher than the baseline, with 99.8% of search time saved on CIFAR10.
Revised: In addition, FTSO can reach 97.77% testing accuracy, 0.27% higher than the baseline, with 99.8% of search time saved on CIFAR10.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Since the great success of AlexNet (Krizhevsky et al., 2012) in image classiﬁcation, most modern machine learning models have been based on deep neural networks.
Revised: Since the great success of AlexNet (Krizhevsky et al., 2012) in image classiﬁcation, most modern machine learning models have been based on deep neural networks.

Original: For neural networks, their performance is greatly determined by the architectures.
Revised: For neural networks, their performance is greatly determined by the architectures.

Original: Thus, in the past decade, a tremendous amount of work (Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016) has been done to investigate proper network architecture design.
Revised: Thus, in the past decade, a tremendous amount of work (Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016) has been done to investigate proper network architecture design.

Original: However, as the network size has grown larger and larger, it has gradually become unaffordable to manually search for better network architectures via trial and error due to the heavy time and resource overhead.
Revised: However, as the network size has grown larger and larger, it has gradually become unaffordable to manually search for better network architectures via trial and error due to the heavy time and resource overhead.

Original: To ease this problem, a new technique called neural architecture search (NAS) was introduced, which allows computers to search for better network architectures automatically instead of leaving these workloads to humans.
Revised: To ease this problem, a new technique called neural architecture search (NAS) was introduced, which allows computers to search for better network architectures automatically instead of leaving these workloads to humans.

Original: Early-proposed reinforcement learning-based NAS methods (Zoph & Le, 2017; Baker et al., 2017;
Revised: Early-proposed reinforcement learning-based NAS methods (Zoph & Le, 2017; Baker et al., 2017;

Original: Zoph et al., 2018) typically have an RNN-based controller to sample candidate network architecturesfrom the search space. These sampled architectures are separately trained from scratch in order to obtain accuracy on the target dataset.
Revised: Zoph et al., 2018) typically have an RNN-based controller to sample candidate network architectures in order to obtain accuracy from the search space.

Original: Then the accuracy is further utilized as rewards to update the controller’s parameters.

Original: Although these algorithms can provide promising accuracy, their computational costs are usually unaffordable, e.g., 1800 GPU-days are required for NASNet.
Revised: Although these algorithms can provide promising accuracy, their computational costs are usually unaffordable, e.g., 1800 GPU-days are required for NASNet.

Original: To ease the search efﬁciency problem, one-shot approaches (Pham et al., 2018;
Revised: To ease the search efﬁciency problem, one-shot approaches (Pham et al., 2018;

Original: Cai et al., 2019; Liu et al., 2019), with parameter sharing, were proposed.
Revised: Cai et al., 2019; Liu et al., 2019), with parameter sharing, were proposed.

Original: It ﬁrst creates a huge directed acyclic graph (DAG) supernet, containing the whole search space, and then shares the kernel weights among all the sampled architectures via the supernet. Although one-shot methods do not need to repeatedly retrain the sampled networks from scratch, they suffer heavily from their supernets’ computational overheads, which is particularly severe in differentiable models (Liu et al., 2019; Xu et al., 2020).
Revised: It ﬁrst creates a huge directed acyclic graph (DAG) super-net, containing the whole search space, and then shares the kernel weights among all the sampled architectures via the super-net without repeatedly retrain the sampled networks from scratch but they suffer heavily from their super-nets’ computational overheads, which is particularly severe in differentiable models (Liu et al., 2019; Xu et al., 2020).

Original: Limited by current NAS algorithms’ inefﬁciency, it is still rather challenging to ﬁnd satisfying network architectures on large-scale datasets and high-level tasks.
Revised: Limited by current NAS algorithms’ inefﬁciency, it is still rather challenging to ﬁnd satisfying network architectures on large-scale datasets and high-level tasks.

Original: For instance, current speed-oriented
Revised: For instance, current speed-oriented

Original: NAS approaches generally require days to accomplish one search trial on ImageNet, e.g., 8 . 3 GPUdays for ProxylessNAS (Cai et al., 2019) and 3 . 8 GPU-days for PC-DARTS (Xu et al., 2020).
Revised: NAS approaches generally require days to accomplish one search trial on ImageNet, e.g., 8 . 3 GPUdays for ProxylessNAS (Cai et al., 2019) and 3 . 8 GPU-days for PC-DARTS (Xu et al., 2020).

Original: The huge search cost prevents NAS’s further development and application in both academia and industry.

Original: To ease these problems, we argue that it is a must to propose a new well-deﬁned search space, which is both expressive enough to cover the most powerful architectures and compact enough to ﬁlter out the poor ones.
Revised: Therefore,we argue that it is a must to propose a new well-deﬁned search space, which is both expressive enough to cover the most powerful architectures and compact enough to ﬁlter out the poor ones.

Original: However, the discovery on the search space heavily relies on human’s prior knowledge, and blindly pruning the search space might lead to missing the best architectures.

Original: We also notice that the correlation among different architectures is not well investigated in existing works.

Original: This result leads to the lack of prior knowledge for effectively shrinking the search space.

Original: Motivated by Shu et al.
Revised: Motivated by Shu et al.

Original: (2020), who demonstrated that randomly replacing operators in a found architecture does not harm the accuracy much, we believe that omitting the inﬂuence of operators and clustering architectures according to the topology, may bring no reduction to the testing accuracy, while signiﬁcantly beneﬁting the search efﬁciency.
Revised: (2020), who demonstrated that randomly replacing operators in a found architecture does not harm the accuracy much, we believe that omitting the inﬂuence of operators and clustering architectures according to the topology, may bring no reduction to the testing accuracy while signiﬁcantly beneﬁt the search efﬁciency.

Original: Thus, in this paper, we propose to separately search for the network topology and the operators.
Revised: Thus, in this paper, we propose to separately search for the network topology and the operators.

Original: We name this new method Effective NAS via 
Revised: We name this new method Effective NAS via First

Original: First Topology Second Operator (FTSO).
Revised: Topology Second Operator (FTSO).

Original: We ﬁrst mathematically prove that, by greatly shrinking the graph of the search space, reducing the operators’ complexity in magnitude, lowering the required searching period from 50 epochs to one iteration and signiﬁcantly easing the Matthew effect, namely that the complex operators may never get the chance to be well tuned, thus the found architecture only contains very simple operators, and performs poorly on the testing set, FTSO reduces the required parameters by a factor of 0 .
Revised: We ﬁrst mathematically prove that FTSO reduces the required parameters by 5 . 3 × 10 7 and decreases the FLOPs per iteration by 1 × 10 5 . Besides, FTSO signiﬁcantly promotes the accuracy compared to the baseline by greatly shrinking the graph of the search space, reducing the operators complexity in magnitude and lowering the required searching period from 50 epochs to one iteration.

Original: 53 × 10 8 , decreases the FLOPs per iteration by a factor of 2 × 10 5 and signiﬁcantly promotes the accuracy compared to the baseline, PC-DARTS.
Revised: What’s more, the Matthew effect is eased.

Original: Then we empirically show that FTSO is superior in both efﬁciency and effectiveness, accomplishing the whole architecture search in 0 .
Revised: Then we empirically show that FTSO is superior in both efﬁciency and effectiveness, accomplishing the whole architecture search in 0 .

Original: 68 seconds.
Revised: 68 seconds.

Original: On ImageNet, FTSO can achieve 76 .
Revised: On ImageNet, FTSO can achieve 76 .

Original: 4% testing accuracy, 1 . 5% higher than the baseline,PC-DARTS, within merely 18 seconds.
Revised: 4% testing accuracy, 1 . 5% higher than the baseline, within merely 18 seconds.

Original: What is more, when we only search for one iteration, FTSO can reach 75 .
Revised: More importantly, when we only search for one iteration, FTSO can reach 75 .

Original: 64% testing accuracy, 0 .
Revised: 64% testing accuracy, 0 .

Original: 74% higher than the baseline in just 0.68 seconds.
Revised: 74% higher than the baseline in just 0.68 seconds.

Original: Besides, if we allow FTSO to search for 19 minutes, 76 .
Revised: Besides, if we allow FTSO to search for 19 minutes, 76 .

Original: 42% Top1 and 93 .
Revised: 42% Top and 93 .

Original: 2% Top5 testing accuracy can be obtained.
Revised: 2% Top5 testing accuracy can be obtained.

Original: In addition, FTSO can reach 97 .
Revised: In addition, FTSO can reach 97 .

Original: 77% testing accuracy, 0 .
Revised: 77% testing accuracy, 0 .

Original: 27% higher than the baseline, with 99 .
Revised: 27% higher than the baseline, with 99 .

Original: 8% of search time saved on CIFAR10.
Revised: 8% of search time saved on CIFAR10.

Original: Although in this paper we  only implement FTSO within a continuous search space,  in fact, it can easily be transferred and bring improvements to other NAS algorithms.
Revised: Although in this paper we have only implemented FTSO within a continuous search space, we illustrate in Section 5 that FTSO can easily be transferred and bring improvements to other NAS algorithms.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Deep neural networks have won great success and been widely adopted as the basic framework of modern machine learning algorithms.

Original: However, due to the continuing growth of the size and the complexity of neural networks (He et al., 2016; Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020), it has become harder and harder to manually design new network structures.

Original: To deal with these problems, a new technique, named neural architecture search (NAS), was proposed to allow computers design the network architectures automatically.

Original: In general, existing NAS algorithms can be divided into three categories, namely, reinforcement learning-based, revolution-based and differentiable.
Revised: In general, existing NAS algorithms can be divided into three categories, namely, reinforcement learning-based, revolution-based and differentiable.

Original: Early-proposed reinforcement learning-based methods (Zoph & Le, 2017; Zoph et al., 2018) generally suffer from high computational cost.
Revised: Early-proposed reinforcement learning-based methods (Zoph & Le, 2017; Zoph et al., 2018) generally suffer from high computational cost and low-efﬁciency sampling.

Original: ENAS (Pham et al., 2018) reduces the time cost by forming an over-parameterized network (a.k.a. supernet) containing all the possible architectures.

Original: Every time the controller picks one sub-network from the supernet, all the matched weights are inherited.

Original: At the same time, all the updates on the picked sub-network will be equally imposed on the supernet.

Original: Although reinforcement learning-based NAS methods perform well in terms of accuracy, they suffer from low-efﬁciency sampling.

Original: Instead of sampling a discrete architecture and then evaluating it,
Revised: Instead of sampling a discrete architecture and then evaluating it,

Original: DARTS (Liu et al., 2019) treats the whole search space as a continuous supernet.
Revised: DARTS (Liu et al., 2019) treats the whole search space as a continuous super-net.

Original: It assigns every operator a real number weight and treats every node as the linear combination of all its transformed predecessors.
Revised: It assigns every operator a real number weight and treats every node as the linear combination of all its transformed predecessors.

Original: To be speciﬁc, DARTS’s search space is a directed acyclic graph (DAG) containing two input nodes inherited from previous cells, four intermediate nodes and one output node.
Revised: To be speciﬁc, DARTS’s search space is a directed acyclic graph (DAG) containing two input nodes inherited from previous cells, four intermediate nodes and one output node.

Original: Each node denotes one latent representation and each edge denotes an operator.
Revised: Each node denotes one latent representation and each edge denotes an operator.

Original: Every intermediate node x j is calculated from all its predecessors x i , i.e., x j = (cid:80) i<j (cid:80) o ∈O exp
Revised: Every intermediate node x j is calculated from all its predecessors x i , i.e., x j = (cid:80) i< j

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: exp α o (cid:48)

Original: i,j o ( x i ) , where O denotes the collection of all candidate operators, α oi,j denotes the weight for operator o from node i to j .
Revised: , where O denotes the collection of all candidate operators, α oi,j denotes the weight for operator o from node i to j .

Original: This strategy allows DARTS to directly use gradients to optimize the whole supernet.
Revised: This strategy allows DARTS to directly use gradients to optimize the whole super-net.

Original: After the supernet converges, DARTS only retains the operators with the largest weights.
Revised: After the super-net converges, DARTS only retains the operators with the largest weights.

Original: In this way, the ﬁnal discrete architecture is derived.
Revised: In this way, the ﬁnal discrete architecture is derived.

Original: The main defect of DARTS is that it needs to maintain and do all calculations on a giant supernet, which inevitably leads to heavy computational overheads and over-ﬁtting.
Revised: The main defect of DARTS is that it needs to maintain and do all calculations on a giant super-net, which inevitably leads to heavy computational overheads and over-ﬁtting.

Original: Proposed to relieve the computational overhead of DARTS, DARTS-ES (Zela et al., 2020) reduces the number of searching epochs via early stopping, according to the Hessian matrix’s max eigen- value.
Revised: Proposed to relieve the computational overhead of DARTS, DARTS-ES (Zela et al., 2020) reduces the number of searching epochs via early stopping, according to the Hessian matrix’s max eigenvalue.

Original: PC-DARTS
Revised: PC-DARTS

Original: (Xu et al., 2020) decreases the FLOPs per iteration by only calculating a proportion of the input channels and retain the remainder unchanged, and normalizes the edge weights to stabilize the search.
Revised: (Xu et al., 2020) decreases the FLOPs per iteration by only calculating a proportion of the input channels and retain the remainder unchanged, and normalizes the edge weights to stabilize the search.

Original: To be speciﬁc, in PC-DARTS, every intermediate node x j is computed from all its predecessors x i , i.e., x j = (cid:80) i<
Revised: To be speciﬁc, in PC-DARTS, every intermediate node x j is computed from all its predecessors x i , i.e., x j = (cid:80) i<

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] f i,j [Equation] x i ) , where β i,j describes the input node i ’s importance to the node j , and f i,j is the weighted sum of all the candidate operators’ outputs between node i and j .
Revised: f i,j ( x i ) , where β i,j describes the input node i ’s importance to the node j , and f i,j is the weighted sum of all the candidate operators’ outputs between node i and j .

Original: Speciﬁcally, f i,j ( x i , S i,j ) = (cid:80)o
Revised: Speciﬁcally, f i,j ( x i , S i,j ) = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] where S i,j denotes a binary vector, in which only 1 /K elements are 1 .
Revised: where S i,j denotes a binary vector, in which only 1 /K elements are 1 .

Original: Recently, there are more and more papers realizing the importance of the network topology.

Original: For example, Gaier & Ha (2019) tries to ﬁnd a network topology irrelevant to detailed weights.

Original: It starts from sparse and simple topologies, then assigns these topologies a range of weights, and directly evaluates these architectures without further tuning.

Original: After these, it picks out the architectures with the highest evaluation metrics, and varies them to produce the new candidate topologies.

Original: Different from Gaier & Ha (2019), AutoShrink (Zhang et al., 2019) starts from a full DAG.

Original: In every loop, it ﬁrst creates a set of new graphs with one edge fewer than the previous found DAG, then randomly picks some generated DAGs out to form new architectures, which are then evaluated to obtain the metrics.

Original: Finally, the architecture with the highest evaluation metric is sent to the next loop.

Original:  FTSO: E FFECTIVE NAS VIA F IRST T OPOLOGY S ECOND O PERATOR
Revised:  FTSO: E FFECTIVE NAS VIA F IRST T OPOLOGY S ECOND O PERATOR

Original: Existing NAS approaches generally suffer from the heavy computational overhead and the unsatisfying testing accuracy leaded by the huge search space.
Revised: Existing NAS approaches generally suffer from the heavy computational overhead and the unsatisfying testing accuracy leaded by the huge search space.

Original: Such problems are especially stern in one-shot and differentiable methods because, these algorithms need to maintain and even do all the calculations directly on the search space.
Revised: Such problems are especially stern in one-shot and differentiable methods because, these algorithms need to maintain and even do all the calculations directly on the search space.

Original: To ease such problems, it is in a great demand to investigate the correlations among different architectures and to shrink the search space according to these prior knowledge.
Revised: To ease such problems, it is of great demand to investigate the correlations among different architectures and to shrink the search space according to these prior knowledge.

Original: We notice that there is an important observation in Shu et al.
Revised: We notice that there is an important observation in Shu et al.

Original: (2020) that randomly substituting the operators, in a found architecture, does not observably inﬂuence the testing accuracy.
Revised: (2020) that randomly substituting the operators does not observably inﬂuence the testing accuracy in a found architecture.

Original: This gives us an inspiration that it might be possible to cluster the architectures according to their connection topologies.
Revised: Therefore, it would be a great inspiration that we could cluster the architectures according to their connection topologies.

Revised: To be speciﬁc, suppose we have found an architecture only containing the simplest operators achieves high accuracy on the testing set, if we replace all the skip connections in this architecture with powerful operators, the converted architecture can also perform well on the testing set with high conﬁdence.

Original: Thus, in this paper, we propose to ﬁrst search for the network topology with simple operators, and then  ﬁx the topology, and search for the most suitable operators for the given topology.
Revised: Thus, in this paper, we propose to ﬁrst search for the network topology with simple operators and then we ﬁx the topology, and search for the most suitable operators for the given topology.

Original: In this way, the testing accuracy can still be guaranteed, while the search space is shrunken in magnitude.
Revised: In this way, the testing accuracy can still be guaranteed, while the search space is shrunken in magnitude.

Original: We name this new NAS algorithm Effective NAS via First Topology Second Operator (FTSO).
Revised: We name this new NAS algorithm Effective NAS via First Topology Second Operator (FTSO).

Original: As shown in Figure 1, we inherit the differentiable framework of PC-DARTS, and divide the architecture search into two phases.
Revised: As shown in Figure 1, we inherit the differentiable framework of PC-DARTS, and divide the architecture search into two phases.

Revised: We name these two phases topology search and operator search, and illustrate how they work in Algorithm 1 and 2, respectively.

Original: In the ﬁrst phase, we use the simplest operator, skip connection , to replace all the candidate operators.
Revised: In the ﬁrst phase, we form a supernet only containing the simplest operator, skip connection .

Original: Because the skip connection operator contains no kernel weights and there is merely one candidate operator, we only need to optimize the architecture parameters
Revised: Because the skip connection operator contains no kernel weights, we only need to optimize the architecture parameters

Original: β i,j , denoting the node i ’s importance to the node j .
Revised: β i,j , denoting the node i ’s importance to the node j .

Revised: In fact, as shown in Table 3, the max pooling operator also brings satisfying results for the topology search.

Revised: There are two reasons why we use skip connection .

Revised: The ﬁrst reason is that the skip connection operator not only requires 0 parameter, but also produces the minimum computational cost.

Original:  After the topology search, for every intermediate node j , we only retain its connections to its predecessors i ∗ with the highest two β i,j .
Revised: The second reason is that max pooling greatly reduces the size of the image. Thus, we may lose too much useful information in the input image if the network is deep. By the way, since the only difference between our topology search and the vanilla DARTS is the number of candidate operators, the pruned architecture’s connectivity can be guaranteed. After the topology search, for every intermediate node j , we only retain its connections to its predecessors i ∗ with the highest two β i,j .

Original: In the second phase, we search for the operators suitable for the found topology. To be speciﬁc, here we have two solutions.
Revised: In the second phase, we search for the operators suitable for the pruned topology with two strategy.

Original: The ﬁrst strategy is to ﬁrst replace all the operators, in the found topology, with mix-operators f i,j . 
Revised: One is to ﬁrst replace each operator in the pruned topology with a mix-operator f i,j . f

Original: Here f i,j is the linear combination of all  candidate operators o, with weight
Revised: i,j is the linear combination of all the candidate operators o with weight

Original: α o i,j , which denotes the importance of the operator o, between the node i and j .
Revised: α oi,j , which denotes the importance of the operator o between the node i and j .

Original: Then we optimize the architecture parameters, α oi,j and β i,j, and the kernel weights ω
Revised: Then we optimize the architecture parameters, α oi,j , β i,j and the kernel weights ω

Original: oi,j alternatively.
Revised: oi,j alternatively.

Original: After the operator search, for every two connected nodes i and j , we only retain one operator o ∗ with the highest α oi,j  .
Revised: After the super-net converges, we only retain one operator o ∗ with the highest α oi,j for every two connected nodes i and j .

Original: The second strategy is to directly replace all the operators, in the found topology, with one single operator, which owns the greatest potential after fully converging, e.g., a convolution operator.
Revised: The other is to directly replace all the operators in the pruned topology with one single operator owning the highest model capacity, e.g., a convolution operator.

Original: In this paper, we take the second strategy as the default conﬁguration because, it is much more efﬁcient, and  the large amount ofkernel weights, contained in the ﬁrst strategy, may lead to over-ﬁtting, and ﬁnally leads to worse performance than the second strategy.
Revised: In this paper, we take the second strategy as the default conﬁguration because it is much more efﬁcient and avoid over-ﬁtting. The ﬁnal architecture outputted by the ﬁrst strategy is only a small part of the super-net and if the subgraph can already generalize perfectly on the testing set, it must lead to over-ﬁtting.

Revised: Thus, under this circumstance, the optimum super-net on training set is not the optimum on testing set, and so is the sub-graph.

Revised: PC-DARTS has mentioned this issue and reduced the size of the super-net via only computing partial input channels while in our second strategy, since no super-net is adopted and all the simple operators in the sub-graph are replaced with powerful operators, the ﬁnal architecture’s model capacity gets promoted.

Original: Additional empirical results can be found in Section 4.3.
Revised: Additional empirical comparisons between these two strategies can be found in Section 4.4. i<j (cid:80) o ∈O exp α oi,j (cid:80) o (cid:48)∈O exp α

Original:  In DARTS, the network topology and operators are searched jointly, which makes both the size and  computational cost of the supernet extremely high because, the total number of operators in
Revised: o (cid:48) i,j o ( n i ) ; In DARTS, the network topology and operators are jointly searched, which makes both the size and the computational cost of the super-net extremely high.

Original: DARTS is 12 ( n 2 − 3 n ) p .

Original: Here n denotes the number of nodes and p denotes the number of candidate operators.
Revised: Let n denotes the number of nodes, p denotes the number of candidate operators.

Revised: Since we have two input nodes, one output node and n − 3 intermediate nodes, the super-net contains totally 12 ( n 2 − 3 n ) edges.

Revised: At the same time, every edge keeps p operators, thus, the total number of operators in DARTS is 12 ( n 2 − 3 n ) p .

Original: By comparison, there are only 12 n ( n − 3) operations in our topology search, and 2( n − 3) p operations in our operator search.
Revised: By comparison, there are only 12 n ( n − 3) operations in our topology search, and 2( n − 3) p operations in our operator search.

Revised: This is because in the topology search, every edge contains only one operator; and in the topology search, every intermediate node only connects to two predecessors.

Original: Since n is usually close to p , FTSO reduces the number of operations from O ( n 3 ) to O ( n 2 ) .
Revised: Since n is usually close to p , FTSO reduces the number of operations from O ( n 3 ) to O ( n 2 ) .

Original: In addition to the reduction on the number of operations, FTSO also dramatically decreases the internal cost of the operations because,  all the complex operators are replaced with the simple operators, during the topology search.
Revised: In addition to the reduction on the number of operations, FTSO also dramatically decreases the internal cost of the operations because, during the topology search, all the powerful operators are replaced with the simple operators.

Original: To be speciﬁc, a vanilla convolutional operator needs k 2 C in H out W out C out FLOPs and ( k 2 C in +1) C out parameters, where k is the kernel size, C in is the input tensor’s channel number and H out , W out , and C out are the output tensor’s height, width and channel number respectively.
Revised: To be speciﬁc, a vanilla convolutional operator needs k 2 C in H out W out C out FLOPs and ( k 2 C in + 1) C out parameters, where k is the kernel size, C in is the input tensor’s channel number and H out , W out , and C out are the output tensor’s height, width and channel number respectively.

Original: By comparison, a skip connection operator needs 0 parameters and 
Revised: By comparison, a skip connection operator needs 0 parameters and 0 FLOPs.

Revised: For simplicity, assume all the candidate operators are convolutions.

Revised: Since DARTS has 12 pn ( n − 3) edges, it needs to compute 12 pn ( n − 3) convolutions and 12 pn ( n − 3) tensor summations.

Revised: Because each tensor summations consumes H in W in C in

Original: FLOPs.
Revised: FLOPs, DARTS requires totally 12 pk 2 n ( n − 3) C in H out W out C out + 12 pn ( n − 3) H out W out C out = 12 pn ( n − 3) H out W out C out ( k 2 C in +1) FLOPs and 12 n ( n − 3) p ( k 2 C in +1) C out parameters. While in

Revised: FTSO, if we ﬁrst search for the topology and then directly substituting the operators, only 12 n ( n − 3) tensor summations need to be calculated.

Revised: Thus, the total number of FLOPs and parameters of FTSO are 12 n ( n − 3) H in W in C in and 12 n ( n − 3) respectively.

Original: As a typical conﬁguration when k = 5 , C in = C out = 512 , n = 7 and p = 8 , assume all the replaced operators are convolutions.
Revised: As a typical conﬁguration, let k = 5 , C in = C out = 512 , n = 7 , p = 8 .

Original: Our algorithm requires only 1 (( k 2 C in +1) C out+1) p = 1 .
Revised: Then, our algorithm requires only 1 p ( k 2 C in +1) C out = 1 .

Original: 9 × 10 −times the parameters and n +2 2 np ( k 2 C in +1)+4 = 6 × 10 − 6 times the forward propagation FLOPs compared 
Revised: 9 × 10 −times the parameters and 1 p ( k 2 C in +1) = 9 . 8 × 10 − 6 times the forward-propagation FLOPs compared to DARTS.

Original: to DARTS. The huge reduction on the parameter numbers provides us a large amount of beneﬁts.
Revised: FTSO’s huge reduction on the parameter numbers provides us a large amount of beneﬁts.

Original: Firstly. It allows the algorithm to converge within extremely few iterations.
Revised: As mentioned above, it allows the algorithm to converge in only a few iterations and prevent over-ﬁtting.

Original: Suppose the batch size is 256 , and we only search for one iteration on CIFAR10, then FTSO requires merely 10 − 10 times of the FLOPs, in the whole search procedure, compared to the baseline. Note that because our algorithm does not need to back-propagate with respect to the kernel weights, its actual superiority on efﬁciency is even more signiﬁcant than these theoretical calculations. Secondly.
Revised: This is because when extracting the discrete sub-graph from the super-net, many architecture parameters are set to be 0.

Original: Because our supernet contains very few parameters, it can hardly over-ﬁt the dataset. Thus, the found architecture may generalize better on new datasets.
Revised: The introduced disturbance impacts more on the over-ﬁtting super-nets since they prefer sharper local minimums.

Original: Thirdly.
Revised: Furthermore, it avoids the Matthew’s effect.

Original: In DARTS, every architecture is assumed to be able to converge in one iteration.
Revised: Each architecture has only one iteration to tune its kernel weights in DARTS.

Original: Unfortunately, it is almost impossible for those complex supernets to converge  insuch a short period. As if this hypothesis does not stand, the whole theoretical foundation of DARTS may collapse.
Revised: However, within one iteration, only the operators with few parameters can converge and thus, the simpler operators outperform the powerful ones in the super-net, then obtain larger gradients to enlarge their advantages.

Original: InFTSO, this problem is almost nonexistent because, the skip connection has no kernel weights to tune. Thus, the validation accuracy, obtained during the one-shot search, is exactly the real performance of the supernet, and  the disadvantages brought by the bi-level optimization can be eased.
Revised: In this way, the found architectures tend to only contain the simplest operators and perform poorly on both training and testing set. Since FTSO only contains one operator with 0 parameter, the Matthew’s effect is eliminated.

Original: [Figure]
Revised: [Figure]

Revised: reduction cells found on CIFAR10, where we adopt the operator search, and use the 3 × 3 separable convolution to search for the topology; (g): FTSO’s cell found on NATS-Bench; (h): DARTS’s cell found on NATS-Bench.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our search algorithm is evaluated on the two most widely-used datasets in NAS papers, namely,
Revised: Our search algorithm is evaluated on the three most widely-used datasets in NAS papers, namely,

Original: CIFAR10 and ImageNet.
Revised: CIFAR10(Krizhevsky et al., 2009) and ImageNet(Russakovsky et al., 2015) and NATS-Bench(Dong et al., 2020).

Original: Following DARTS, our search space contains a total eight candidate operators: 3 × 3 and 5 × 5 separable convolutions , 3 × 3 and 5 × 5 dilated separable convolutions , 3 ×  max and average pooling , skip connection (i.e., output = input ) and zero (i.e., output = 0 ).
Revised: Following DARTS, our search space contains a total eight candidate operators: 3 ×and 5 × 5 separable convolutions , 3 × 3 and 5 × 5 dilated separable convolutions , 3 × 3 max and average pooling , skip connection (i.e., output = input ) and zero (i.e., output = 0 ).

Original: When searching for the topology, we pick only one operator from the candidate set.
Revised: When searching for the topology, we pick only one operator from the candidate set.

Original: There are two possibilities: if the chosen operator contains no trainable kernel weights, we only optimize the architecture parameters

Original: β i,j ; otherwise, we optimize the kernel weights ω

Original: oi,j and the architecture parameters β i,j alternatively. When searching for the operators, we ﬁx the pruned topology and construct a new supernet containing all the eight candidate operators. Then, we optimize both the kernel weights and the architecture parameters for 35 epochs after 15 epochs’ pretraining (Chen et al., 2019) on  the kernel weights. After the topology search, it isboth okay to search for the topology with gradients or directly replacing the skip connections with other operators.
Revised: As mentioned in Section 3, we have two strategies to determine the operators, including the one based on gradients and the one is directly replacing.

Original: In Section 4.1 and 4.2, we focus more on the second conﬁguration because it is more efﬁcient, and in Section 4.3, we have a comprehensive comparison on these two strategies.
Revised: In Section 4.1 and 4.2, we focus more on the second conﬁguration In Section 4.4, we have a comprehensive comparison on these two strategies.

Original: All our search experiments are done under the same 11 GB

Original: VRAM constraint on one RTX 2080Ti GPU.

Original: The detailed conﬁgurations are shown in Table 1.
Revised: All detailed conﬁgurations are shown in Table 4.

Original: In addition,our most  experiments only search for one epoch or one iteration because, fewer epochs not only shorten the search time, but also avoid the over-ﬁtting and provide higher generalization ability.
Revised: In addition, most of our experiments only search for one epoch or one iteration because of the beneﬁts of FTSO’s huge reduction on the parameter numbers.

Original: For more experimental supports, please refer to Section 4.3.
Revised: For more experimental supports, please refer to Section 4.4.

Original: Note that it is almost impossible for the existing models to obtain satisfying results via only searching for one epoch or one iteration because, their supernets contain a large amount of parameters, which require a long period to tune.[Table]
Revised: Note that it is almost impossible for the existing models to obtain satisfying results via only searching for one epoch or one iteration because their super-nets contain a large amount of parameters, which require a long period to tune.

Original: 4. R ESULTS ON CIFAR 
Revised: 4. R ESULTS ON CIFAR

Original: Following DARTS, we search for the architecture in a cell manner, where each cell has four intermediate nodes and 16 initial channels during the search. We use half the images ( 25 K) from the training set for kernel weight updating and the other half for architecture parameter updating.
Revised: The detailed conﬁgurations are shown in Appendix.

Original: When evaluating, we stack 18 normal and two reduction cells to form the entire network.
Revised: [Table] These models are directly searched on ImageNet.

Original: We compare FTSO to existing state-of-the-art NAS methods in Table 2.
Revised: We compare FTSO to existing state-of-the-art NAS methods in Table 1.

Original: The results show that FTSO can greatly surpass the baseline, i.e., PC-DARTS, by 0 . 27% in terms of the test accuracy on CIFAR10, with 99 . 8% of the search time saved when only searching for the topology with skipconnections and then replacing them all with 3 × 3 separable convolutions .
Revised: In the experiment, we only search for the topology with skip-connections and then replacing them all with 3 × 3 separable convolutions .

Original: The reason why we do not adopt 5 × 5 separable convolutions is that the pre-processed input images do not have enough resolutions, and the network is rather deep.
Revised: The reason why we do not adopt 5 × 5 separable convolutions is that the pre-processed input images do not have enough resolutions, and the network is rather deep.

Original: After a few layers, the convolution’s receptive ﬁeld has been larger than the whole image.
Revised: After a few layers, the convolution’s receptive ﬁeld has been larger than the whole image.

Original: At that time, larger convolutional kernels may not bring beneﬁts.
Revised: At that time, larger convolutional kernels may not bring beneﬁts.

Original: Instead, the extra parameters brought by the larger kernel size may lead to over-ﬁtting.
Revised: Instead, the extra parameters brought by the larger kernel size may lead to over-ﬁtting.

Original: Of course, suppose both the image’s resolution and the dataset’s scale are big enough, and the evaluation period is adequate,then the 5 × 5 separable convolution might be a better choice.
Revised: On the other hand, suppose both the image’s resolution and the dataset’s scale are big enough and the evaluation period is adequate, the 5 × 5 separable convolution might be a better choice.

Original: The found architecture after one epoch’s search is shown in Figure 2.
Revised: The found architecture after one epoch’s search is shown in Figure 2.

Original: Because FTSO contains very few trainable parameters, it can even achieve comparable accuracy to PC-DARTS with only an one-time gradient update.
Revised: Because FTSO contains only a few trainable parameters, it can even achieve comparable accuracy to PCDARTS with only an one-time gradient update.

Original: Under this conﬁguration, merely 0 .
Revised: Under this conﬁguration, merely 0 .

Original: 68 seconds are required and 99 .
Revised: 68 seconds are required and 99 .

Original: 993% of the search time is saved.
Revised: 993% of the search time is saved.

Original: In addition, as shown in Table 3, when the topology is searched with complex operators for a long period, additional operator search usually helps.
Revised: In addition, as shown in Table 3, when the topology is searched with powerful operators for a long period, additional operator search usually helps.

Original: However, when we search for the topology with simple operators for a short period, omitting the operator search may lead to better results.
Revised: However, when we search for the topology with simple operators for a short period, omitting the operator search may lead to better results.

Original: This is because with simple operators and very few updates, the found topology can already generalize quite well.
Revised: This is because with simple operators and very few updates, the found topology can already generalize quite well.

Original: In contrast, the supernet, during the operator search, contains a large number of parameters, so it tends to pick out architectures which converge more easily, while this kind of architectures usually contain simpler operators and lead to the lower ﬁnal accuracy after fully converging.

Original: 4. R ESULTS ON I MAGE N ET
Revised: 4. R ESULTS ON I MAGE N ET

Original: On ImageNet, we use similar conﬁgurations to those on CIFAR10.
Revised: On ImageNet, we use similar conﬁgurations to those on CIFAR10.

Original: However, both the number of images and the size of each image in ImageNet are signiﬁcantly larger than those in CIFAR10.

Original: Thus we ﬁrst reduce the images’ resolution from 224 × 224 to 28 × 28 with three stride-2 convolutions, and when searching, we have two conﬁgurations.In the ﬁrst conﬁguration, following PC-DARTS, we only pick 10% and 2 . 5% of images from the training set for the kernel weights and architecture parameters’ optimizations respectively; in the second conﬁguration, thanks to FTSO’s huge computational cost reduction, it is feasible for us to search with the whole of ImageNet. For evaluation, we stack 12 normal cells and two reduction cells to form the entire network and train it from scratch for 250 epochs.
Revised: When searching, we have two conﬁgurations.

Original: The detailed conﬁgurations are shown in Table [Table] [Table] [Table]
Revised: The detailed conﬁgurations are shown in Appendix.

Original: [Table]

Original: [Table] [Table] These models are directly searched on ImageNet. Our experiments in Table 2 show that FTSO is signiﬁcantly superior to existing methods in both efﬁciency and effectiveness.
Revised: Our experiments in Table 1 show that FTSO is signiﬁcantly superior to existing methods in both efﬁciency and effectiveness.

Original: Our model transferred from CIFAR10, with only one epoch’s search, can achieve as high as 76 . 4% testing accuracy on ImageNet, 0 .

Original: 6% higher than the baseline’s result directly searched on ImageNet, while saving 99 .

Original: 995% search time.

Original: Compared with the baseline’s result transferred from CIFAR10, FTSO can achieve 1 . 5% accuracy improvement, with 99 .

Original: 98% search time saved.

Original: In addition, even when we only search for one iteration on the proxy dataset, our method can still surpass the baseline’s result directly searched on ImageNet by 0 .

Original: 03% , in terms of the

Original: Top5 accuracy, whithin merely 0 .

Original: 68 seconds.

Original: Furthermore, when directly searching on ImageNet, 76 .

Original: 42% Top 1 and 93 .

Original: 20% Top 5 accuracy can be obtained.

Original: The found architectures after one epoch’s search on CIFAR10 and the entire ImageNet are shown in Figure 2 .
Revised: The found architectures after one epoch’s search on CIFAR10 and the entire ImageNet are shown in Figure 2 .

Original: It is surprising that the best architecture we found on ImageNet is the shallowest and widest one.
Revised: It is surprising that the best architecture we found on ImageNet is the shallowest and widest one.

Original: Compared to the much more ”reasonable” architecture shown in Figure 2(e) and 2(f), which was found with the topology search only containing 3 × 3 separable convolutions and an additional operator search on CIFAR10, the ”abnormal” architecture, containing the same amount of FLOPs and parameters, can achieve 0 .
Revised: Compared to the much more ”reasonable” architecture shown in Figure 2(e) and 2(f), which was found with the topology search only containing 3 ×separable convolutions and an additional operator search on CIFAR10, the ”abnormal” architecture, containing the same amount of FLOPs and parameters, can achieve 0 .

Original: 78% higher testing accuracy.
Revised: 78% higher testing accuracy.

Original: We think  the reason is that, although deeper architectures have thelarger model capacity, their loss curve might not be ﬂat (Shu et al., 2020).
Revised: We think this is because the whole model is stacked with many cells. If the depth of each cell is too high, it leads to a very deep neural network. At that time, both the vanishing and explosion of gradients may prevent the model from higher performance. 4. R ESULTS ON NATS-B ENCH

Revised: In the search space of NATS-Bench, there are one input node, three intermediate node and one output node, and each intermediate node connects to all its predecessors.

Revised: Here we implement FTSO based on DARTS instead of PC-DARTS, and we compare FTSO’s performance to other NAS algorithms in Table 2.

Revised: It is shown that FTSO dominates DARTS on all conﬁgurations.

Original: Thus, they may not generalize as well as  the shallow models we found.
Revised: Coinciding with our analysis in Section 3, the architectures found by DARTS tend to only contain simple operators, thus cannot achieve satisfying accuracy. For example, when searching on CIFAR10, the architecture found by DARTS is full of skip connections as shown in Figure 2(h). By comparison, as shown in Figure 2(g), the architecture found by FTSO is much more powerful.

Revised: [Table]

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: We visualize the relationship between FTSO’s testing accuracy and the number of searching epochs in Figure 3(a) and Figure 3(b).

Original: In terms of topology-only search, one epoch is just enough, thanks to the many fewer kernel weights contained in FTSO, and more search epochs bring obvious disadvantages because of over-ﬁtting.
Revised: In terms of topology-only search, one epoch is just enough, thanks to the many fewer kernel weights contained in FTSO, and more search epochs bring obvious disadvantages because of over-ﬁtting.

Original: Since one epoch performs better than more epochs, it raises the question whether one iteration is also superior to more iterations.
Revised: Since one epoch performs better than more epochs, it raises the question whether one iteration is also superior to more iterations.

Original: In Figure 3(c) we show that although one iteration cannot surpass one epoch, it is better than a few iterations because, in the latter conﬁguration, only  a small proportion of the training set is used, which brings variance to the model and hurts its generalization ability.
Revised: In Figure 3(c) we show that although one iteration cannot surpass one epoch, it is better than a few iterations. This is because when we only search for one iteration, the model does over-ﬁt the data, thus the model generalizes well. When we only searched for a few iterations, the number of different images seen by the model is not big enough.

Original: In terms of whether we should search for one iteration or two, in
Revised: However, since the super-net only contains skip connections, such number of gradient updates has been enough for the architecture parameters to get over-ﬁtted. This is the reason why a few iterations perform worse than one iteration. After we have searched for one whole epoch, the super-net has seen enormous different images, this helps it to generalize better on the testing set. This is the reason why one epoch performs the best.

Original:  Figure 3(d) , we show that both choices work well.[Figure]
Revised: In terms of whether we should search for one iteration or two, in Figure 3(d), we show that both choices work well.

Original: When we do not search for the operators, after the topology search, we assign all the remaining edges a ﬁxed operator.
Revised: When we do not search for the operators after the topology search, we assign all the remaining edges a ﬁxed operator.

Original: Thus, which operator we should choose becomes a critical question.
Revised: Thus, which operator we should choose becomes a critical question.

Original: We show in Figure 3(e)  that 3 × 3 separable convolution can indeed outperform all other operators in terms ofaccuracy, as discussed in 4.1, and randomly assigning each edge an operator among all the candidate operators, except for None , can lead to comparable testing accuracy.If we want to extend
Revised: Figure 3(e) show that 3 × 3 separable convolution can indeed outperform all other operators in terms of accuracy.

Original: FTSO to other tasks, ﬁrst searching for the topology with the simplest operator, and then replacing it with the most promising one should be the basic principle.

Original: As shown in Figure 3(f), we ﬁnd that under a different number of evaluation epochs, both the absolute value and the relative ranking of the same architecture’s testing accuracy may vary; i.e., some architectures performing well within 600 epochs perform poorly after 1200 epochs.
Revised: As shown in Figure 3(f), we ﬁnd that under a different number of evaluation epochs, both the absolute value and the relative ranking of the same architecture’s testing accuracy may vary; i.e., some architectures performing well within 600 epochs perform poorly after 1200 epochs.

Original: However, in general, they still obey a positive correlation, with a Pearson correlation coefﬁcient of 0 .
Revised: However, in general, they still obey a positive correlation, with a Pearson correlation coefﬁcient of 0 .

Original: 77 , as  shown in Figure 3(j).
Revised: 77 , as [Figure] shown in Figure 3(j).

Original: In terms of the generalization ability from CIFAR10 to ImageNet, Figure 3(l) reveals that the architectures performing well after long-term evaluation on CIFAR10 can usually generalize better on ImageNet, with the correlation coefﬁcient of 0 .
Revised: In terms of the generalization ability from CIFAR10 to ImageNet Figure 3(l) reveals that the architectures performing well after long-term evaluation on CIFAR10 can usually generalize better on ImageNet, with the correlation coefﬁcient of 0 .

Original: 7 ; yet, as shown in Figure 3(k), there is no guarantee that those working well on CIFAR10 within limited evaluation epochs can also dominate on ImageNet because this only proves that they can quickly converge, not that they can converge to a global optimum.
Revised: 7 ; yet, as shown in Figure 3(k) there is no guarantee that those working well on CIFAR10 within limited evaluation epochs can also dominate on ImageNet because this only proves that they can quickly converge, not that they can converge to a global optimum.

Original: Another important fact, as shown in Figure 3(i), is that, coinciding with our analysis in 4.1, for an architecture already performing well after the topology search, an additional operator search may bring no beneﬁt.

Original: This is because the operator search requires a large supernet, which can easily lead to over-ﬁtting.

Original: In Figure 3(g) and 3(h), we show that one epoch is not only an optimal choice on CIFAR10, but also enough for the topology-only search on CIFAR100.
Revised: In Figure 3(g) and 3(h), we show that one epoch is not only an optimal choice on CIFAR10, but also enough for the topology-only search on CIFAR100.

Original: In addition, as the search epochs and iterations increase, the max eigenvalue of the loss’s Hessian matrix on the validation set increases and the 
Revised: In addition, as the search epochs and iterations increase, the max eigenvalue of the loss’s Hessian matrix on the validation set increases and the testing accuracy is generally decreasing because the model’s generalization ability is dropping.

Revised: This phenomenon is particularly obvious epoch-wise because, after just a few iterations, the model can already reach a comparable accuracy on the training set.

Revised: Then the model’s performance on the testing set starts to relate with its generalization ability.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Topo
Revised: Topo

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Topo
Revised: Topo

Original: [Table]
Revised: [Table]

Original: skip
Revised: skip

Original: [Table]
Revised: [Table]

Original: 1it
Revised: 1it

Original: [Table]
Revised: [Table]

Original: Topo(skip,1ep.)
Revised: Topo(skip,1ep.)

Original: [Table]
Revised: [Table]

Original: testing accuracy is generally decreasing because the model’s generalization ability is dropping.

Original: This phenomenon is particularly obvious epoch-wise because, after just a few iterations, the model can already reach a comparable accuracy on the training set.

Original: Then the model’s performance on the testing set starts to relate with its generalization ability.

Original:  G ENERALIZATION TO OTHER TASKS AND SEARCH SPACES
Revised:  G ENERALIZATION TO OTHER TASKS AND SEARCH SPACES

Revised: As shown in Section 4, FTSO works well under different search spaces and node numbers.

Revised: Theoretically, FTSO’s advantages to DARTS can be enlarged while search space and node number increase.

Revised: This is because FTSO reduces the computational cost from O ( n 3 ) to O ( n 2 ) and avoids over-ﬁtting.

Original: Although in this paper, we establish FTSO in a differentiable search space, in fact, the ﬁrst topology second operator strategy is not limited within any speciﬁc search space or tasks.
Revised: Although in this paper, we establish FTSO within differentiable search spaces, in fact, the ﬁrst topology second operator strategy is not limited within any speciﬁc search space or tasks.

Original: No matter the search space is discrete or continuous, we ﬁrst shrink the candidate operator set, and only retain the simplest operator.
Revised: No matter the search space is discrete or continuous, we ﬁrst shrink the candidate operator set, and only retain the simplest operator.

Original: After this, the size of the whole search space is reduced in magnitude.
Revised: After this, the size of the whole search space is reduced in magnitude.

Original: Then, we search for the best topology with whatever search algorithm.
Revised: Then, we search for the best topology with whatever search algorithm.

Original: In this way, a promising topology can be found.
Revised: In this way, a promising topology can be found.

Original: Now we have two choices. The ﬁrst solution is to form a new search space with both the found topology and all the candidate operators, then utilize whatever search algorithm to ﬁnd the best operators. The second solution is to assign all the edges a ﬁxed operator. Here the principle is to pick the operators performing the best after fully converging.
Revised: Then, we can either directly assign each edge a powerful operator or use gradients to search for operators.

Original: However, when the searching set is not informative enough, picking an operator with slightly fewer parameters should be a better choice.

Original: As for the selection between these two solutions. If the search algorithm is hard to over-ﬁt, choosing the ﬁrst solution may be better. Otherwise, the second solution might be more promising.
Revised: Generally, the ﬁrst strategy leads to higher accuracy, and the second strategy reduces model complexity.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose an ultra computationally efﬁcient neural architecture search method named
Revised: In this paper, we propose an ultra computationally efﬁcient neural architecture search method named

Original: FTSO, which reduces NAS’s search time cost from days to less than 0.68 seconds, while achieving 1.5% and 0.27% accuracy improvement on ImageNet and CIFAR10 respectively.
Revised: FTSO, which reduces NAS’s search time cost from days to less than 0.68 seconds, while achieving 1.5% and 0.27% accuracy improvement on ImageNet and CIFAR10 respectively.

Original: Our key idea is to divide the search procedure into two sub-phases.
Revised: Our key idea is to divide the search procedure into two sub-phases.

Original: In the ﬁrst phase, we only search for the network’s topology with simple operators.
Revised: In the ﬁrst phase, we only search for the network’s topology with simple operators.

Original: Then, in the second phase, we ﬁx the topology and only consider which operators we should choose.
Revised: Then, in the second phase, we ﬁx the topology and only consider which operators we should choose.

Original: Our strategy is concise in both theory and implementation, and our promising experimental results show that current NAS methods contain too much redundancy, which heavily impacts the efﬁciency and becomes a barrier to higher accuracy.
Revised: Our strategy is concise in both theory and implementation, and our promising experimental results show that current NAS methods contain too much redundancy, which heavily impacts the efﬁciency and becomes a barrier to higher accuracy.

Original: What is more, our method is not bound by differentiable search spaces, so we believe that our key idea can also cooperate well with existing NAS approaches.
Revised: What is more, our method is not bound by differentiable search spaces, so we believe that our key idea can also cooperate well with existing NAS approaches.

Revised: A A PPENDIX A. C ONFIGURATION D ETAILS

Revised: All our search experiments are done under the same 11 GB VRAM constraint on one RTX 2080Ti

Revised: GPU.

Revised: For the conﬁguration of experiment on ImageNet, following DARTS, we search for the architecture in a cell manner, where each cell has four intermediate nodes and 16 initial channels during the search.

Revised: We use half the images ( 25 K) from the training set for kernel weight updating and the other half for architecture parameter updating.

Revised: When evaluating, we stack 18 normal and two reduction cells to form the entire network.

Revised: For the conﬁguration of experiment on ImageNet, when searching, we have two conﬁgurations.

Revised: In the ﬁrst conﬁguration, following PC-DARTS, we only pick 10% and 2 .

Revised: 5% of images from the training set for the kernel weights and architecture parameters’ optimizations respectively; in the second conﬁguration, thanks to FTSO’s huge computational cost reduction, it is feasible for us to search with the whole of ImageNet.

Revised: For evaluation, we stack 12 normal cells and two reduction cells to form the entire network and train it from scratch for 250 epochs.

Revised: The detailed conﬁgurations are shown in Table 4.

Revised: [Table]