Original: FTSO: E FFECTIVE NAS VIA F IRST T OPOLOGY S ECOND O PERATOR
Revised: FTSO: E FFECTIVE NAS VIA F IRST T OPOLOGY S ECOND O PERATOR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing one-shot neural architecture search (NAS) methods generally contain a giant super-net, which leads to heavy computational cost.
Revised: Existing one-shot neural architecture search (NAS) methods generally contain a giant super-net, which leads to heavy computational cost.

Original: Our method, named
Revised: Our method, named

Original: FTSO, separates the whole architecture search into two sub-steps.
Revised: FTSO, separates the whole architecture search into two sub-steps.

Original: In the ﬁrst step, we only search for the topology, and in the second step, we only search for the operators.
Revised: In the ﬁrst step, we only search for the topology, and in the second step, we only search for the operators.

Original: FTSO not only reduces NAS’s search time from days to 0.68 seconds, but also signiﬁcantly improves the accuracy.
Revised: FTSO not only reduces NAS’s search time from days to 0.68 seconds, but also signiﬁcantly improves the accuracy.

Original: Speciﬁcally, our experiments on
Revised: Speciﬁcally, our experiments on

Original: ImageNet show that within merely 18 seconds, FTSO can achieve 76.4% testing accuracy, 1.5% higher than the baseline, PC-DARTS.
Revised: ImageNet show that within merely 18 seconds, FTSO can achieve 76.4% testing accuracy, 1.5% higher than the baseline, PC-DARTS.

Original: In addition, FTSO can reach 97.77% testing accuracy, 0.27% higher than the baseline, with 99.8% of search time saved on CIFAR10.
Revised: In addition, FTSO can reach 97.77% testing accuracy, 0.27% higher than the baseline, with 99.8% of search time saved on CIFAR10.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Since the great success of AlexNet (Krizhevsky et al., 2012) in image classiﬁcation, most modern machine learning models have been based on deep neural networks.
Revised: Since the great success of AlexNet (Krizhevsky et al., 2012) in image classiﬁcation, most modern machine learning models have been based on deep neural networks.

Original: For neural networks, their performance is greatly determined by the architectures.
Revised: For neural networks, their performance is greatly determined by the architectures.

Original: Thus, in the past decade, a tremendous amount of work (Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016) has been done to investigate proper network architecture design.
Revised: Thus, in the past decade, a tremendous amount of work (Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016) has been done to investigate proper network architecture design.

Original: However, as the network size has grown larger and larger, it has gradually become unaffordable to manually search for better network architectures via trial and error due to the heavy time and resource overhead.
Revised: However, as the network size has grown larger and larger, it has gradually become unaffordable to manually search for better network architectures via trial and error due to the heavy time and resource overhead.

Original: To ease this problem, a new technique called neural architecture search (NAS) was introduced, which allows computers to search for better network architectures automatically instead of relying on human experts..
Revised: To ease this problem, a new technique called neural architecture search (NAS) was introduced, which allows computers to search for better network architectures automatically instead of relying on human experts..

Original: Early-proposed reinforcement learning-based NAS methods (Zoph & Le, 2017; Baker et al., 2017;
Revised: Early-proposed reinforcement learning-based NAS methods (Zoph & Le, 2017; Baker et al., 2017;

Original: Zoph et al., 2018) typically have an RNN-based controller to sample candidate network architectures in order to obtain accuracy from the search space.
Revised: Zoph et al., 2018) typically have an RNN-based controller to sample candidate network architectures in order to obtain accuracy from the search space.

Original: Although these algorithms can provide promising accuracy, their computation cost is usually unaffordable, e.g., 1800 GPU-days are required for
Revised: Although these algorithms can provide promising accuracy, their computation cost is usually unaffordable, e.g., 1800 GPU-days are required for

Original: NASNet.
Revised: NASNet.

Original: To ease the search efﬁciency problem, one-shot approaches (Pham et al., 2018;
Revised: To ease the search efﬁciency problem, one-shot approaches (Pham et al., 2018;

Original: Cai et al., 2019; Liu et al., 2019), with parameter sharing, were proposed.
Revised: Cai et al., 2019; Liu et al., 2019), with parameter sharing, were proposed.

Original: It ﬁrst creates a huge directed acyclic graph (DAG) super-net, containing the whole search space, and then shares the kernel weights among all the sampled architectures via the super-net without repeatedly retrain the sampled networks from scratch but they suffer heavily from their super-nets’ computational overheads, which is particularly severe in differentiable models (Liu et al., 2019; Xu et al., 2020).
Revised: It ﬁrst creates a huge directed acyclic graph (DAG) super-net, containing the whole search space, and then shares the kernel weights among all the sampled architectures via the super-net without repeatedly retrain the sampled networks from scratch but they suffer heavily from their super-nets’ computational overheads, which is particularly severe in differentiable models (Liu et al., 2019; Xu et al., 2020).

Original: Limited by current NAS algorithms’ inefﬁciency, it is still rather challenging to ﬁnd satisfying network architectures on large-scale datasets and high-level tasks.
Revised: Limited by current NAS algorithms’ inefﬁciency, it is still rather challenging to ﬁnd satisfying network architectures on large-scale datasets and high-level tasks.

Original: For instance, current speed-oriented
Revised: For instance, current speed-oriented

Original: NAS approaches generally require days to accomplish one search trial on ImageNet, e.g., 8 . 3 GPUdays for ProxylessNAS (Cai et al., 2019) and 3 . 8 GPU-days for PC-DARTS (Xu et al., 2020).
Revised: NAS approaches generally require days to accomplish one search trial on ImageNet, e.g., 8 . 3 GPUdays for ProxylessNAS (Cai et al., 2019) and 3 . 8 GPU-days for PC-DARTS (Xu et al., 2020).

Original: Therefore,we argue that it is essential to propose a new well-deﬁned search space, which is both expressive enough to cover the most powerful architectures and compact enough to ﬁlter out the poor ones.
Revised: Therefore,we argue that it is essential to propose a new well-deﬁned search space, which is both expressive enough to cover the most powerful architectures and compact enough to ﬁlter out the poor ones.

Original: Motivated by Shu et al.
Revised: Motivated by Shu et al.

Original: (2020), who demonstrated that randomly replacing operators in a found architecture does not harm the accuracy much, we believe that omitting the inﬂuence of operators and clustering architectures according to the topology, may bring no reduction to the testing accuracy while signiﬁcantly beneﬁt the search efﬁciency.
Revised: (2020), who demonstrated that randomly replacing operators in a found architecture does not harm the accuracy much, we believe that omitting the inﬂuence of operators and clustering architectures according to the topology, may bring no reduction to the testing accuracy while signiﬁcantly beneﬁt the search efﬁciency.

Original: Thus, in this paper, we propose to separately search for the network topology and the operators.
Revised: Thus, in this paper, we propose to separately search for the network topology and the operators.

Original: We name this new method Effective NAS via First
Revised: We name this new method Effective NAS via First

Original: Topology Second Operator (FTSO).
Revised: Topology Second Operator (FTSO).

Original: We ﬁrst mathematically prove that FTSO reduces the required parameters by 5 .
Revised: We ﬁrst mathematically prove that FTSO reduces the required parameters by 5 .

Original: 3 × 10 7 and decreases the FLOPs per iteration by 1 × 10 5 .
Revised: 3 × 10 7 and decreases the FLOPs per iteration by 1 × 10 5 .

Original: Besides, FTSO signiﬁcantly promotes the accuracy compared to the baseline by greatly shrinking the graph of the search space, reducing the operators complexity in magnitude and lowering the required searching period from 50 epochs to one iteration.
Revised: Besides, FTSO signiﬁcantly promotes the accuracy compared to the baseline by greatly shrinking the graph of the search space, reducing the operators complexity in magnitude and lowering the required searching period from 50 epochs to one iteration.

Original: What’s more, the Matthew effect is eased.
Revised: What’s more, the Matthew effect is eased.

Original: Then we empirically show that FTSO is superior in both efﬁciency and effectiveness, accomplishing the whole architecture search in 0 .
Revised: Then we empirically show that FTSO is superior in both efﬁciency and effectiveness, accomplishing the whole architecture search in 0 .

Original: 68 seconds.
Revised: 68 seconds.

Original: On ImageNet, FTSO can achieve 76 .
Revised: On ImageNet, FTSO can achieve 76 .

Original: 4% testing accuracy, 1 . 5% higher than the baseline, within merely 18 seconds.
Revised: 4% testing accuracy, 1 . 5% higher than the baseline, within merely 18 seconds.

Original: More importantly, when we only search for one iteration, FTSO can reach 75 .
Revised: More importantly, when we only search for one iteration, FTSO can reach 75 .

Original: 64% testing accuracy, 0 .
Revised: 64% testing accuracy, 0 .

Original: 74% higher than the baseline in just 0.68 seconds.
Revised: 74% higher than the baseline in just 0.68 seconds.

Original: Besides, if we allow FTSO to search for 19 minutes, 76 .
Revised: Besides, if we allow FTSO to search for 19 minutes, 76 .

Original: 42% Top and 93 .
Revised: 42% Top and 93 .

Original: 2% Top5 testing accuracy can be obtained.
Revised: 2% Top5 testing accuracy can be obtained.

Original: In addition, FTSO can reach 97 .
Revised: In addition, FTSO can reach 97 .

Original: 77% testing accuracy, 0 .
Revised: 77% testing accuracy, 0 .

Original: 27% higher than the baseline, with 99 .
Revised: 27% higher than the baseline, with 99 .

Original: 8% of search time saved on CIFAR10.
Revised: 8% of search time saved on CIFAR10.

Original: Although in this paper we have only implemented FTSO within a continuous search space, we illustrate in
Revised: Although in this paper we have only implemented FTSO within a continuous search space, we illustrate in

Original: Section 5 that FTSO can easily be transferred and bring improvements to other NAS algorithms.
Revised: Section 5 that FTSO can easily be transferred and bring improvements to other NAS algorithms.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In general, existing NAS algorithms can be divided into three categories, namely, reinforcement learning-based, revolution-based and differentiable.
Revised: In general, existing NAS algorithms can be divided into three categories, namely, reinforcement learning-based, revolution-based and differentiable.

Original: Early-proposed reinforcement learning-based methods (Zoph & Le, 2017; Zoph et al., 2018) generally suffer from high computational cost and low-efﬁciency sampling.
Revised: Early-proposed reinforcement learning-based methods (Zoph & Le, 2017; Zoph et al., 2018) generally suffer from high computational cost and low-efﬁciency sampling.

Original: Instead of sampling a discrete architecture and then evaluating it,
Revised: Instead of sampling a discrete architecture and then evaluating it,

Original: DARTS (Liu et al., 2019) treats the whole search space as a continuous super-net.
Revised: DARTS (Liu et al., 2019) treats the whole search space as a continuous super-net.

Original: It assigns every operator a real number weight and treats every node as the linear combination of all its transformed predecessors.
Revised: It assigns every operator a real number weight and treats every node as the linear combination of all its transformed predecessors.

Original: To be speciﬁc, DARTS’s search space is a directed acyclic graph (DAG) containing two input nodes inherited from previous cells, four intermediate nodes and one output node.
Revised: To be speciﬁc, DARTS’s search space is a directed acyclic graph (DAG) containing two input nodes inherited from previous cells, four intermediate nodes and one output node.

Original: Each node denotes one latent representation and each edge denotes an operator.
Revised: Each node denotes one latent representation and each edge denotes an operator.

Original: Every intermediate node x j is calculated from all its predecessors x i , i.e., x j = (cid:80) i<
Revised: Every intermediate node x j is calculated from all its predecessors x i , i.e., x j = (cid:80) i<

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: , where O denotes the collection of all candidate operators, α oi,j denotes the weight for operator o from node i to j .
Revised: , where O denotes the collection of all candidate operators, α oi,j denotes the weight for operator o from node i to j .

Original: This strategy allows DARTS to directly use gradients to optimize the whole super-net.
Revised: This strategy allows DARTS to directly use gradients to optimize the whole super-net.

Original: After the super-net converges, DARTS only retains the operators with the largest weights.
Revised: After the super-net converges, DARTS only retains the operators with the largest weights.

Original: In this way, the ﬁnal discrete architecture is derived.
Revised: In this way, the ﬁnal discrete architecture is derived.

Original: The main defect of DARTS is that it needs to maintain and do all calculations on a giant super-net, which inevitably leads to heavy computational overheads and over-ﬁtting.
Revised: The main defect of DARTS is that it needs to maintain and do all calculations on a giant super-net, which inevitably leads to heavy computational overheads and over-ﬁtting.

Original: Proposed to relieve the computational overhead of DARTS, DARTS-ES (Zela et al., 2020) reduces the number of searching epochs via early stopping, according to the Hessian matrix’s max eigenvalue.
Revised: Proposed to relieve the computational overhead of DARTS, DARTS-ES (Zela et al., 2020) reduces the number of searching epochs via early stopping, according to the Hessian matrix’s max eigenvalue.

Original: PC-DARTS
Revised: PC-DARTS

Original: (Xu et al., 2020) decreases the FLOPs per iteration by only calculating a proportion of the input channels and retain the remainder unchanged, and normalizes the edge weights to stabilize the search.
Revised: (Xu et al., 2020) decreases the FLOPs per iteration by only calculating a proportion of the input channels and retain the remainder unchanged, and normalizes the edge weights to stabilize the search.

Original: To be speciﬁc, in PC-DARTS, every intermediate node x j is computed from all its predecessors x i , i.e., x j = (cid:80) i<
Revised: To be speciﬁc, in PC-DARTS, every intermediate node x j is computed from all its predecessors x i , i.e., x j = (cid:80) i<

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: f i,j ( x i ) , where β i,j describes the input node i ’s importance to the node j , and f i,j is the weighted sum of all the candidate operators’ outputs between node i and j .
Revised: f i,j ( x i ) , where β i,j describes the input node i ’s importance to the node j , and f i,j is the weighted sum of all the candidate operators’ outputs between node i and j .

Original: Speciﬁcally, f i,j ( x i , S i,j ) = (cid:80)
Revised: Speciﬁcally, f i,j ( x i , S i,j ) = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: where S i,j denotes a binary vector, in which only 1 /K elements are 1 .
Revised: where S i,j denotes a binary vector, in which only 1 /K elements are 1 .

Original:  FTSO: E FFECTIVE NAS VIA F IRST T OPOLOGY S ECOND O PERATOR
Revised:  FTSO: E FFECTIVE NAS VIA F IRST T OPOLOGY S ECOND O PERATOR

Original: Existing NAS approaches generally suffer from the heavy computational overhead and the unsatisfying testing accuracy leaded by the huge search space.
Revised: Existing NAS approaches generally suffer from the heavy computational overhead and the unsatisfying testing accuracy leaded by the huge search space.

Original: Such problems are especially stern in one-shot and differentiable methods because, these algorithms need to maintain and even do all the calculations directly on the search space.
Revised: Such problems are especially stern in one-shot and differentiable methods because, these algorithms need to maintain and even do all the calculations directly on the search space.

Original: To ease such problems, it is of great demand to investigate the correlations among different architectures and to shrink the search space according to these prior knowledge.
Revised: To ease such problems, it is of great demand to investigate the correlations among different architectures and to shrink the search space according to these prior knowledge.

Original: We notice that there is an important observation in Shu et al.
Revised: We notice that there is an important observation in Shu et al.

Original: (2020) that randomly substituting the operators does not observably inﬂuence the testing accuracy in a found architecture.
Revised: (2020) that randomly substituting the operators does not observably inﬂuence the testing accuracy in a found architecture.

Original: Therefore, it would be a great inspiration that we could cluster the architectures according to their connection topologies.
Revised: Therefore, it would be a great inspiration that we could cluster the architectures according to their connection topologies.

Original: To be speciﬁc, suppose we have found an architecture only containing the simplest operators achieves high accuracy on the testing set, if we replace all the skip connections in this architecture with powerful operators, the converted architecture can also perform well on the testing set with high conﬁdence.
Revised: To be speciﬁc, suppose we have found an architecture only containing the simplest operators achieves high accuracy on the testing set, if we replace all the skip connections in this architecture with powerful operators, the converted architecture can also perform well on the testing set with high conﬁdence.

Original: Thus, in this paper, we propose to ﬁrst search for the network topology with simple operators and then we ﬁx the topology, and search for the most suitable operators for the given topology.
Revised: Thus, in this paper, we propose to ﬁrst search for the network topology with simple operators and then we ﬁx the topology, and search for the most suitable operators for the given topology.

Original: In this way, the testing accuracy can still be guaranteed, while the search space is shrunken in magnitude.
Revised: In this way, the testing accuracy can still be guaranteed, while the search space is shrunken in magnitude.

Original: We name this new NAS algorithm Effective NAS via First Topology Second Operator (FTSO).
Revised: We name this new NAS algorithm Effective NAS via First Topology Second Operator (FTSO).

Original: As shown in Figure 1, we inherit the differentiable framework of PC-DARTS, and divide the architecture search into two phases.
Revised: As shown in Figure 1, we inherit the differentiable framework of PC-DARTS, and divide the architecture search into two phases.

Original: We name these two phases topology search and operator search, and illustrate how they work in Algorithm 1 and 2, respectively.
Revised: We name these two phases topology search and operator search, and illustrate how they work in Algorithm 1 and 2, respectively.

Original: In the ﬁrst phase, we form a supernet only containing the simplest operator, skip connection .
Revised: In the ﬁrst phase, we form a supernet only containing the simplest operator, skip connection .

Original: Because the skip connection operator contains no kernel weights, we only need to optimize the architecture parameters
Revised: Because the skip connection operator contains no kernel weights, we only need to optimize the architecture parameters

Original: β i,j , denoting the node i ’s importance to the node j .
Revised: β i,j , denoting the node i ’s importance to the node j .

Original: In fact, as shown in Table 3, the max pooling operator also brings satisfying results for the topology search.
Revised: In fact, as shown in Table 3, the max pooling operator also brings satisfying results for the topology search.

Original: There are two reasons why we use skip connection .
Revised: There are two reasons why we use skip connection .

Original: The ﬁrst reason is that the skip connection operator not only requires 0 parameter, but also produces the minimum computational cost.
Revised: The ﬁrst reason is that the skip connection operator not only requires 0 parameter, but also produces the minimum computational cost.

Original: The second reason is that max pooling greatly reduces the size of the image.
Revised: The second reason is that max pooling greatly reduces the size of the image.

Original: Thus, we may lose too much useful information in the input image if the network is deep.
Revised: Thus, we may lose too much useful information in the input image if the network is deep.

Original: By the way, since the only difference between our topology search and the vanilla DARTS is the number of candidate operators, the pruned architecture’s connectivity can be guaranteed.
Revised: By the way, since the only difference between our topology search and the vanilla DARTS is the number of candidate operators, the pruned architecture’s connectivity can be guaranteed.

Original: After the topology search, for every intermediate node j , we only retain its connections to its predecessors i ∗ with the highest two β i,j .
Revised: After the topology search, for every intermediate node j , we only retain its connections to its predecessors i ∗ with the highest two β i,j .

Original: In the second phase, we search for the operators suitable for the pruned topology with two strategy.
Revised: In the second phase, we search for the operators suitable for the pruned topology with two strategy.

Original: One is to ﬁrst replace each operator in the pruned topology with a mix-operator f i,j . f
Revised: One is to ﬁrst replace each operator in the pruned topology with a mix-operator f i,j . f

Original: i,j is the linear combination of all the candidate operators o with weight
Revised: i,j is the linear combination of all the candidate operators o with weight

Original: α oi,j , which denotes the importance of the operator o between the node i and j .
Revised: α oi,j , which denotes the importance of the operator o between the node i and j .

Original: Then we optimize the architecture parameters, α oi,j , β i,j and the kernel weights ω
Revised: Then we optimize the architecture parameters, α oi,j , β i,j and the kernel weights ω

Original: oi,j alternatively.
Revised: oi,j alternatively.

Original: After the super-net converges, we only retain one operator o ∗ with the highest α oi,j for every two connected nodes i and j .
Revised: After the super-net converges, we only retain one operator o ∗ with the highest α oi,j for every two connected nodes i and j .

Original: The other is to directly replace all the operators in the pruned topology with one single operator owning the highest model capacity, e.g., a convolution operator.
Revised: The other is to directly replace all the operators in the pruned topology with one single operator owning the highest model capacity, e.g., a convolution operator.

Original: In this paper, we take the second strategy as the default conﬁguration because it is much more efﬁcient and avoid over-ﬁtting.
Revised: In this paper, we take the second strategy as the default conﬁguration because it is much more efﬁcient and avoid over-ﬁtting.

Original: The ﬁnal architecture outputted by the ﬁrst strategy is only a small part of the super-net and if the subgraph can already generalize perfectly on the testing set, it must lead to over-ﬁtting.
Revised: The ﬁnal architecture outputted by the ﬁrst strategy is only a small part of the super-net and if the subgraph can already generalize perfectly on the testing set, it must lead to over-ﬁtting.

Original: Thus, under this circumstance, the optimum super-net on training set is not the optimum on testing set, and so is the sub-graph.
Revised: Thus, under this circumstance, the optimum super-net on training set is not the optimum on testing set, and so is the sub-graph.

Original: PC-DARTS has mentioned this issue and reduced the size of the super-net via only computing partial input channels while in our second strategy, since no super-net is adopted and all the simple operators in the sub-graph are replaced with powerful operators, the ﬁnal architecture’s model capacity gets promoted.
Revised: PC-DARTS has mentioned this issue and reduced the size of the super-net via only computing partial input channels while in our second strategy, since no super-net is adopted and all the simple operators in the sub-graph are replaced with powerful operators, the ﬁnal architecture’s model capacity gets promoted.

Original: Additional empirical comparisons between these two strategies can be found in Section 4.4.
Revised: Additional empirical comparisons between these two strategies can be found in Section 4.4.

Original: i<j (cid:80) o ∈O exp α
Revised: i<j (cid:80) o ∈O exp α

Original: oi,j (cid:80) o
Revised: oi,j (cid:80) o

Original: (cid:48)∈O exp α
Revised: (cid:48)∈O exp α

Original: o (cid:48)
Revised: o (cid:48)

Original: i,j o ( n i ) ;
Revised: i,j o ( n i ) ;

Original: In DARTS, the network topology and operators are jointly searched, which makes both the size and the computational cost of the super-net extremely high.
Revised: In DARTS, the network topology and operators are jointly searched, which makes both the size and the computational cost of the super-net extremely high.

Original: Let n denotes the number of nodes, p denotes the number of candidate operators.
Revised: Let n denotes the number of nodes, p denotes the number of candidate operators.

Original: Since we have two input nodes, one output node and n − 3 intermediate nodes, the super-net contains totally 12 ( n 2 − 3 n ) edges.
Revised: Since we have two input nodes, one output node and n − 3 intermediate nodes, the super-net contains totally 12 ( n 2 − 3 n ) edges.

Original: At the same time, every edge keeps p operators, thus, the total number of operators in DARTS is 12 ( n 2 − 3 n ) p .
Revised: At the same time, every edge keeps p operators, thus, the total number of operators in DARTS is 12 ( n 2 − 3 n ) p .

Original: By comparison, there are only 12 n ( n − 3) operations in our topology search, and 2( n − 3) p operations in our operator search.
Revised: By comparison, there are only 12 n ( n − 3) operations in our topology search, and 2( n − 3) p operations in our operator search.

Original: This is because in the topology search, every edge contains only one operator; and in the topology search, every intermediate node only connects to two predecessors.
Revised: This is because in the topology search, every edge contains only one operator; and in the topology search, every intermediate node only connects to two predecessors.

Original: Since n is usually close to p , FTSO reduces the number of operations from O ( n 3 ) to O ( n 2 ) .
Revised: Since n is usually close to p , FTSO reduces the number of operations from O ( n 3 ) to O ( n 2 ) .

Original: In addition to the reduction on the number of operations, FTSO also dramatically decreases the internal cost of the operations because, during the topology search, all the powerful operators are replaced with the simple operators.
Revised: In addition to the reduction on the number of operations, FTSO also dramatically decreases the internal cost of the operations because, during the topology search, all the powerful operators are replaced with the simple operators.

Original: To be speciﬁc, a vanilla convolutional operator needs k 2 C in H out W out C out FLOPs and ( k 2 C in + 1) C out parameters, where k is the kernel size, C in is the input tensor’s channel number and H out , W out , and C out are the output tensor’s height, width and channel number respectively.
Revised: To be speciﬁc, a vanilla convolutional operator needs k 2 C in H out W out C out FLOPs and ( k 2 C in + 1) C out parameters, where k is the kernel size, C in is the input tensor’s channel number and H out , W out , and C out are the output tensor’s height, width and channel number respectively.

Original: By comparison, a skip connection operator needs 0 parameters and 0 FLOPs.
Revised: By comparison, a skip connection operator needs 0 parameters and 0 FLOPs.

Original: For simplicity, assume all the candidate operators are convolutions.
Revised: For simplicity, assume all the candidate operators are convolutions.

Original: Since DARTS has 12 pn ( n − 3) edges, it needs to compute 12 pn ( n − 3) convolutions and 12 pn ( n − 3) tensor summations.
Revised: Since DARTS has 12 pn ( n − 3) edges, it needs to compute 12 pn ( n − 3) convolutions and 12 pn ( n − 3) tensor summations.

Original: Because each tensor summations consumes H in W in C in
Revised: Because each tensor summations consumes H in W in C in

Original: FLOPs, DARTS requires totally 12 pk 2 n ( n − 3) C in H out W out C out + 12 pn ( n − 3)
Revised: FLOPs, DARTS requires totally 12 pk 2 n ( n − 3) C in H out W out C out + 12 pn ( n − 3)

Original: H out W out C out = 12 pn ( n − 3)
Revised: H out W out C out = 12 pn ( n − 3)

Original: H out W out C out ( k 2 C in +1) FLOPs and 12 n ( n − 3) p ( k 2 C in +1) C out parameters.
Revised: H out W out C out ( k 2 C in +1) FLOPs and 12 n ( n − 3) p ( k 2 C in +1) C out parameters.

Original: While in
Revised: While in

Original: FTSO, if we ﬁrst search for the topology and then directly substituting the operators, only 12 n ( n − 3) tensor summations need to be calculated.
Revised: FTSO, if we ﬁrst search for the topology and then directly substituting the operators, only 12 n ( n − 3) tensor summations need to be calculated.

Original: Thus, the total number of FLOPs and parameters of FTSO are 12 n ( n − 3) H in W in C in and 12 n ( n − 3) respectively.
Revised: Thus, the total number of FLOPs and parameters of FTSO are 12 n ( n − 3) H in W in C in and 12 n ( n − 3) respectively.

Original: As a typical conﬁguration, let k
Revised: As a typical conﬁguration, let k

Original: = 5 ,
Revised: = 5 ,

Original: C in = C out = 512 , n = 7 , p = 8 .
Revised: C in = C out = 512 , n = 7 , p = 8 .

Original: Then, our algorithm requires only 1 p ( k 2 C in +1) C out = 1 .
Revised: Then, our algorithm requires only 1 p ( k 2 C in +1) C out = 1 .

Original: 9 × 10 −times the parameters and 1 p ( k 2 C in +1)
Revised: 9 × 10 −times the parameters and 1 p ( k 2 C in +1)

Original: = 9 .
Revised: = 9 .

Original: 8 × 10 − 6 times the forward-propagation FLOPs compared to DARTS.
Revised: 8 × 10 − 6 times the forward-propagation FLOPs compared to DARTS.

Original: FTSO’s huge reduction on the parameter numbers provides us a large amount of beneﬁts.
Revised: FTSO’s huge reduction on the parameter numbers provides us a large amount of beneﬁts.

Original: As mentioned above, it allows the algorithm to converge in only a few iterations and prevent over-ﬁtting.
Revised: As mentioned above, it allows the algorithm to converge in only a few iterations and prevent over-ﬁtting.

Original: This is because when extracting the discrete sub-graph from the super-net, many architecture parameters are set to be 0.
Revised: This is because when extracting the discrete sub-graph from the super-net, many architecture parameters are set to be 0.

Original: The introduced disturbance impacts more on the over-ﬁtting super-nets since they prefer sharper local minimums.
Revised: The introduced disturbance impacts more on the over-ﬁtting super-nets since they prefer sharper local minimums.

Original: Furthermore, it avoids the Matthew’s effect.
Revised: Furthermore, it avoids the Matthew’s effect.

Original: Each architecture has only one iteration to tune its kernel weights in DARTS.
Revised: Each architecture has only one iteration to tune its kernel weights in DARTS.

Original: However, within one iteration, only the operators with few parameters can converge and thus, the simpler operators outperform the powerful ones in the super-net, then obtain larger gradients to enlarge their advantages.
Revised: However, within one iteration, only the operators with few parameters can converge and thus, the simpler operators outperform the powerful ones in the super-net, then obtain larger gradients to enlarge their advantages.

Original: In this way, the found architectures tend to only contain the simplest operators and perform poorly on both training and testing set.
Revised: In this way, the found architectures tend to only contain the simplest operators and perform poorly on both training and testing set.

Original: Since FTSO only contains one operator with 0 parameter, the Matthew’s effect is eliminated.
Revised: Since FTSO only contains one operator with 0 parameter, the Matthew’s effect is eliminated.

Original: [Figure]
Revised: [Figure]

Original: reduction cells found on CIFAR10, where we adopt the operator search, and use the 3 × 3 separable convolution to search for the topology; (g): FTSO’s cell found on NATS-Bench; (h): DARTS’s cell found on NATS-Bench.
Revised: reduction cells found on CIFAR10, where we adopt the operator search, and use the 3 × 3 separable convolution to search for the topology; (g): FTSO’s cell found on NATS-Bench; (h): DARTS’s cell found on NATS-Bench.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our search algorithm is evaluated on the three most widely-used datasets in NAS papers, namely,
Revised: Our search algorithm is evaluated on the three most widely-used datasets in NAS papers, namely,

Original: CIFAR10(Krizhevsky et al., 2009) and ImageNet(Russakovsky et al., 2015) and NATS-Bench(Dong et
Revised: CIFAR10(Krizhevsky et al., 2009) and ImageNet(Russakovsky et al., 2015) and NATS-Bench(Dong et

Original: al., 2020).
Revised: al., 2020).

Original: Following DARTS, our search space contains a total eight candidate operators: 3 ×and 5 × 5 separable convolutions , 3 × 3 and 5 × 5 dilated separable convolutions , 3 × 3 max and average pooling , skip connection (i.e., output = input ) and zero (i.e., output = 0 ).
Revised: Following DARTS, our search space contains a total eight candidate operators: 3 ×and 5 × 5 separable convolutions , 3 × 3 and 5 × 5 dilated separable convolutions , 3 × 3 max and average pooling , skip connection (i.e., output = input ) and zero (i.e., output = 0 ).

Original: When searching for the topology, we pick only one operator from the candidate set.
Revised: When searching for the topology, we pick only one operator from the candidate set.

Original: As mentioned in Section 3, we have two strategies to determine the operators, including the one based on gradients and the one is directly replacing.
Revised: As mentioned in Section 3, we have two strategies to determine the operators, including the one based on gradients and the one is directly replacing.

Original: In Section 4.1 and 4.2, we focus more on the second conﬁguration In Section 4.4, we have a comprehensive comparison on these two strategies.
Revised: In Section 4.1 and 4.2, we focus more on the second conﬁguration In Section 4.4, we have a comprehensive comparison on these two strategies.

Original: All detailed conﬁgurations are shown in Table 4.
Revised: All detailed conﬁgurations are shown in Table 4.

Original: In addition, most of our experiments only search for one epoch or one iteration because of the beneﬁts of FTSO’s huge reduction on the parameter numbers.
Revised: In addition, most of our experiments only search for one epoch or one iteration because of the beneﬁts of FTSO’s huge reduction on the parameter numbers.

Original: For more experimental supports, please refer to Section 4.4.
Revised: For more experimental supports, please refer to Section 4.4.

Original: Note that it is almost impossible for the existing models to obtain satisfying results via only searching for one epoch or one iteration because their super-nets contain a large amount of parameters, which require a long period to tune.
Revised: Note that it is almost impossible for the existing models to obtain satisfying results via only searching for one epoch or one iteration because their super-nets contain a large amount of parameters, which require a long period to tune.

Original: 4. R ESULTS ON CIFAR 
Revised: 4. R ESULTS ON CIFAR

Original: The detailed conﬁgurations are shown in Appendix.
Revised: The detailed conﬁgurations are shown in Appendix.

Original: [Table]
Revised: [Table]

Original: These models are directly searched on ImageNet.
Revised: These models are directly searched on ImageNet.

Original: We compare FTSO to existing state-of-the-art NAS methods in Table 1.
Revised: We compare FTSO to existing state-of-the-art NAS methods in Table 1.

Original: In the experiment, we only search for the topology with skip-connections and then replacing them all with 3 × 3 separable convolutions .
Revised: In the experiment, we only search for the topology with skip-connections and then replacing them all with 3 × 3 separable convolutions .

Original: The reason why we do not adopt 5 × 5 separable convolutions is that the pre-processed input images do not have enough resolutions, and the network is rather deep.
Revised: The reason why we do not adopt 5 × 5 separable convolutions is that the pre-processed input images do not have enough resolutions, and the network is rather deep.

Original: After a few layers, the convolution’s receptive ﬁeld has been larger than the whole image.
Revised: After a few layers, the convolution’s receptive ﬁeld has been larger than the whole image.

Original: At that time, larger convolutional kernels may not bring beneﬁts.
Revised: At that time, larger convolutional kernels may not bring beneﬁts.

Original: Instead, the extra parameters brought by the larger kernel size may lead to over-ﬁtting.
Revised: Instead, the extra parameters brought by the larger kernel size may lead to over-ﬁtting.

Original: On the other hand, suppose both the image’s resolution and the dataset’s scale are big enough and the evaluation period is adequate, the 5 × 5 separable convolution might be a better choice.
Revised: On the other hand, suppose both the image’s resolution and the dataset’s scale are big enough and the evaluation period is adequate, the 5 × 5 separable convolution might be a better choice.

Original: The found architecture after one epoch’s search is shown in Figure 2.
Revised: The found architecture after one epoch’s search is shown in Figure 2.

Original: Because
Revised: Because

Original: FTSO contains only a few trainable parameters, it can even achieve comparable accuracy to PCDARTS with only an one-time gradient update.
Revised: FTSO contains only a few trainable parameters, it can even achieve comparable accuracy to PCDARTS with only an one-time gradient update.

Original: Under this conﬁguration, merely 0 .
Revised: Under this conﬁguration, merely 0 .

Original: 68 seconds are required and 99 .
Revised: 68 seconds are required and 99 .

Original: 993% of the search time is saved.
Revised: 993% of the search time is saved.

Original: In addition, as shown in Table 3, when the topology is searched with powerful operators for a long period, additional operator search usually helps.
Revised: In addition, as shown in Table 3, when the topology is searched with powerful operators for a long period, additional operator search usually helps.

Original: However, when we search for the topology with simple operators for a short period, omitting the operator search may lead to better results.
Revised: However, when we search for the topology with simple operators for a short period, omitting the operator search may lead to better results.

Original: This is because with simple operators and very few updates, the found topology can already generalize quite well.
Revised: This is because with simple operators and very few updates, the found topology can already generalize quite well.

Original: 4. R ESULTS ON I MAGE N ET
Revised: 4. R ESULTS ON I MAGE N ET

Original: On ImageNet, we use similar conﬁgurations to those on CIFAR10.
Revised: On ImageNet, we use similar conﬁgurations to those on CIFAR10.

Original: When searching, we have two conﬁgurations.
Revised: When searching, we have two conﬁgurations.

Original: The detailed conﬁgurations are shown in Appendix.
Revised: The detailed conﬁgurations are shown in Appendix.

Original: Our experiments in Table 1 show that FTSO is signiﬁcantly superior to existing methods in both efﬁciency and effectiveness.
Revised: Our experiments in Table 1 show that FTSO is signiﬁcantly superior to existing methods in both efﬁciency and effectiveness.

Original: The found architectures after one epoch’s search on CIFAR10 and the entire ImageNet are shown in Figure 2 .
Revised: The found architectures after one epoch’s search on CIFAR10 and the entire ImageNet are shown in Figure 2 .

Original: It is surprising that the best architecture we found on
Revised: It is surprising that the best architecture we found on

Original: ImageNet is the shallowest and widest one.
Revised: ImageNet is the shallowest and widest one.

Original: Compared to the much more ”reasonable” architecture shown in Figure 2(e) and 2(f), which was found with the topology search only containing 3 ×separable convolutions and an additional operator search on CIFAR10, the ”abnormal” architecture, containing the same amount of FLOPs and parameters, can achieve 0 .
Revised: Compared to the much more ”reasonable” architecture shown in Figure 2(e) and 2(f), which was found with the topology search only containing 3 ×separable convolutions and an additional operator search on CIFAR10, the ”abnormal” architecture, containing the same amount of FLOPs and parameters, can achieve 0 .

Original: 78% higher testing accuracy.
Revised: 78% higher testing accuracy.

Original: We think this is because the whole model is stacked with many cells.
Revised: We think this is because the whole model is stacked with many cells.

Original: If the depth of each cell is too high, it leads to a very deep neural network.
Revised: If the depth of each cell is too high, it leads to a very deep neural network.

Revised: At that time, because all the operators in our found architecture are convolutions, we cannot use skip connections to facilitate gradients’ propagation in

Revised: ResNet’s manner.

Original: At that time, both the vanishing and explosion of gradients may prevent the model from higher performance.
Revised: In this way, both the vanishing and explosion of gradients may prevent the deeper models from higher performance.

Original: 4. R ESULTS ON NATS-B ENCH
Revised: 4. R ESULTS ON NATS-B ENCH

Original: In the search space of NATS-Bench, there are one input node, three intermediate node and one output node, and each intermediate node connects to all its predecessors.
Revised: In the search space of NATS-Bench, there are one input node, three intermediate node and one output node, and each intermediate node connects to all its predecessors.

Original: Here we implement FTSO based on DARTS instead of PC-DARTS, and we compare FTSO’s performance to other NAS algorithms in Table 2.
Revised: Here we implement FTSO based on DARTS instead of PC-DARTS, and we compare FTSO’s performance to other NAS algorithms in Table 2.

Original: It is shown that FTSO dominates DARTS on all conﬁgurations.
Revised: It is shown that FTSO dominates DARTS on all conﬁgurations.

Original: Coinciding with our analysis in Section 3, the architectures found by DARTS tend to only contain simple operators, thus cannot achieve satisfying accuracy.
Revised: Coinciding with our analysis in Section 3, the architectures found by DARTS tend to only contain simple operators, thus cannot achieve satisfying accuracy.

Original: For example, when searching on CIFAR10, the architecture found by DARTS is full of skip connections as shown in Figure 2(h).
Revised: For example, when searching on CIFAR10, the architecture found by DARTS is full of skip connections as shown in Figure 2(h).

Original: By comparison, as shown in
Revised: By comparison, as shown in

Original: Figure 2(g), the architecture found by FTSO is much more powerful.
Revised: Figure 2(g), the architecture found by FTSO is much more powerful.

Original: [Table]
Revised: [Table]

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: In terms of topology-only search, one epoch is just enough, thanks to the many fewer kernel weights contained in FTSO, and more search epochs bring obvious disadvantages because of over-ﬁtting.
Revised: In terms of topology-only search, one epoch is just enough, thanks to the many fewer kernel weights contained in FTSO, and more search epochs bring obvious disadvantages because of over-ﬁtting.

Original: Since one epoch performs better than more epochs, it raises the question whether one iteration is also superior to more iterations.
Revised: Since one epoch performs better than more epochs, it raises the question whether one iteration is also superior to more iterations.

Original: In Figure 3(c) we show that although one iteration cannot surpass one epoch, it is better than a few iterations.
Revised: In Figure 3(c) we show that although one iteration cannot surpass one epoch, it is better than a few iterations.

Original: This is because when we only search for one iteration, the model does over-ﬁt the data, thus the model generalizes well.
Revised: This is because when we only search for one iteration, the model does over-ﬁt the data, thus the model generalizes well.

Original: When we only searched for a few iterations, the number of different images seen by the model is not big enough.
Revised: When we only searched for a few iterations, the number of different images seen by the model is not big enough.

Original: However, since the super-net only contains skip connections, such number of gradient updates has been enough for the architecture parameters to get over-ﬁtted.
Revised: However, since the super-net only contains skip connections, such number of gradient updates has been enough for the architecture parameters to get over-ﬁtted.

Original: This is the reason why a few iterations perform worse than one iteration.
Revised: This is the reason why a few iterations perform worse than one iteration.

Original: After we have searched for one whole epoch, the super-net has seen enormous different images, this helps it to generalize better on the testing set.
Revised: After we have searched for one whole epoch, the super-net has seen enormous different images, this helps it to generalize better on the testing set.

Original: This is the reason why one epoch performs the best.
Revised: This is the reason why one epoch performs the best.

Original: In terms of whether we should search for one iteration or two, in Figure 3(d), we show that both choices work well.
Revised: In terms of whether we should search for one iteration or two, in Figure 3(d), we show that both choices work well.

Original: When we do not search for the operators after the topology search, we assign all the remaining edges a ﬁxed operator.
Revised: When we do not search for the operators after the topology search, we assign all the remaining edges a ﬁxed operator.

Original: Thus, which operator we should choose becomes a critical question.
Revised: Thus, which operator we should choose becomes a critical question.

Original: Figure 3(e) show that 3 × 3 separable convolution can indeed outperform all other operators in terms of accuracy.
Revised: Figure 3(e) show that 3 × 3 separable convolution can indeed outperform all other operators in terms of accuracy.

Original: As shown in Figure 3(f), we ﬁnd that under a different number of evaluation epochs, both the absolute value and the relative ranking of the same architecture’s testing accuracy may vary; i.e.,  some architectures performing well within 600 epochs perform poorly after 1200 epochs.
Revised: As shown in Figure 3(f), we ﬁnd that under a different number of evaluation epochs, both the absolute value and the relative ranking of the same architecture’s testing accuracy may vary; i.e., [Figure] some architectures performing well within 600 epochs perform poorly after 1200 epochs.

Original: However, in general, they still obey a positive correlation, with a Pearson correlation coefﬁcient of 0 .
Revised: However, in general, they still obey a positive correlation, with a Pearson correlation coefﬁcient of 0 .

Original: 77 , as[Figure] shown in Figure 3(j).
Revised: 77 , as shown in Figure 3(j).

Original: In terms of the generalization ability from CIFAR10 to ImageNet Figure 3(l) reveals that the architectures performing well after long-term evaluation on CIFAR10 can usually generalize better on ImageNet, with the correlation coefﬁcient of 0 .
Revised: In terms of the generalization ability from CIFAR10 to ImageNet, Figure 3(l) reveals that the architectures performing well after long-term evaluation on CIFAR10 can usually generalize better on ImageNet, with the correlation coefﬁcient of 0 .

Original: 7 ; yet, as shown in Figure 3(k) there is no guarantee that those working well on CIFAR10 within limited evaluation epochs can also dominate on ImageNet because this only proves that they can quickly converge, not that they can converge to a global optimum.
Revised: 7 ; yet, as shown in Figure 3(k), there is no guarantee that those working well on CIFAR10 within limited evaluation epochs can also dominate on ImageNet because this only proves that they can quickly converge, not that they can converge to a global optimum.

Original: In Figure 3(g) and 3(h), we show that one epoch is not only an optimal choice on CIFAR10, but also enough for the topology-only search on CIFAR100.
Revised: In Figure 3(g) and 3(h), we show that one epoch is not only an optimal choice on CIFAR10, but also enough for the topology-only search on CIFAR100.

Original: In addition, as the search epochs and iterations increase, the max eigenvalue of the loss’s Hessian matrix on the validation set increases and the testing accuracy is generally decreasing because the model’s generalization ability is dropping.
Revised: In addition, as the search epochs and iterations increase, the max eigenvalue of the loss’s Hessian matrix on the validation set increases and the testing accuracy is generally decreasing because the model’s generalization ability is dropping.

Original: This phenomenon is particularly obvious epoch-wise because, after just a few iterations, the model canalready reach a comparable accuracy on the training set.
Revised: This phenomenon is particularly obvious epoch-wise because, after just a few iterations, the model can

Original: Then the model’s performance on the testing set starts to relate with its generalization ability.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Topo
Revised: Topo

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Topo
Revised: Topo

Original: [Table]
Revised: [Table]

Original: skip
Revised: skip

Original: [Table]
Revised: [Table]

Original: 1it
Revised: 1it

Original: [Table]
Revised: [Table]

Original: Topo(skip,1ep.)
Revised: Topo(skip,1ep.)

Original: [Table]
Revised: [Table]

Revised: already reach a comparable accuracy on the training set.

Revised: Then the model’s performance on the testing set starts to relate with its generalization ability.

Original:  G ENERALIZATION TO OTHER TASKS AND SEARCH SPACES
Revised:  G ENERALIZATION TO OTHER TASKS AND SEARCH SPACES

Original: As shown in Section 4, FTSO works well under different search spaces and node numbers.
Revised: As shown in Section 4, FTSO works well under different search spaces and node numbers.

Original: Theoretically, FTSO’s advantages to DARTS can be enlarged while search space and node number increase.
Revised: Theoretically, FTSO’s advantages to DARTS can be enlarged while search space and node number increase.

Original: This is because FTSO reduces the computational cost from O ( n 3 ) to O ( n 2 ) and avoids over-ﬁtting.
Revised: This is because FTSO reduces the computational cost from O ( n 3 ) to O ( n 2 ) and avoids over-ﬁtting.

Original: Although in this paper, we establish FTSO within differentiable search spaces, in fact, the ﬁrst topology second operator strategy is not limited within any speciﬁc search space or tasks.
Revised: Although in this paper, we establish FTSO within differentiable search spaces, in fact, the ﬁrst topology second operator strategy is not limited within any speciﬁc search space or tasks.

Original: No matter the search space is discrete or continuous, we ﬁrst shrink the candidate operator set, and only retain the simplest operator.
Revised: No matter the search space is discrete or continuous, we ﬁrst shrink the candidate operator set, and only retain the simplest operator.

Original: After this, the size of the whole search space is reduced in magnitude.
Revised: After this, the size of the whole search space is reduced in magnitude.

Original: Then, we search for the best topology with whatever search algorithm.
Revised: Then, we search for the best topology with whatever search algorithm.

Original: In this way, a promising topology can be found.
Revised: In this way, a promising topology can be found.

Original: Then, we can either directly assign each edge a powerful operator or use gradients to search for operators.
Revised: Then, we can either directly assign each edge a powerful operator or use gradients to search for operators.

Original: Generally, the ﬁrst strategy leads to higher accuracy, and the second strategy reduces model complexity.
Revised: Generally, the ﬁrst strategy leads to higher accuracy, and the second strategy reduces model complexity.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose an ultra computationally efﬁcient neural architecture search method named
Revised: In this paper, we propose an ultra computationally efﬁcient neural architecture search method named

Original: FTSO, which reduces NAS’s search time cost from days to less than 0.68 seconds, while achieving 1.5% and 0.27% accuracy improvement on ImageNet and CIFAR10 respectively.
Revised: FTSO, which reduces NAS’s search time cost from days to less than 0.68 seconds, while achieving 1.5% and 0.27% accuracy improvement on ImageNet and CIFAR10 respectively.

Original: Our key idea is to divide the search procedure into two sub-phases.
Revised: Our key idea is to divide the search procedure into two sub-phases.

Original: In the ﬁrst phase, we only search for the network’s topology with simple operators.
Revised: In the ﬁrst phase, we only search for the network’s topology with simple operators.

Original: Then, in the second phase, we ﬁx the topology and only consider which operators we should choose.
Revised: Then, in the second phase, we ﬁx the topology and only consider which operators we should choose.

Original: Our strategy is concise in both theory and implementation, and our promising experimental results show that current NAS methods contain too much redundancy, which heavily impacts the efﬁciency and becomes a barrier to higher accuracy.
Revised: Our strategy is concise in both theory and implementation, and our promising experimental results show that current NAS methods contain too much redundancy, which heavily impacts the efﬁciency and becomes a barrier to higher accuracy.

Original: What is more, our method is not bound by differentiable search spaces, so we believe that our key idea can also cooperate well with existing NAS approaches.
Revised: What is more, our method is not bound by differentiable search spaces, so we believe that our key idea can also cooperate well with existing NAS approaches.

Original: A A PPENDIX A. C ONFIGURATION D ETAILS
Revised: A A PPENDIX A. C ONFIGURATION D ETAILS

Original: All our search experiments are done under the same 11 GB VRAM constraint on one RTX 2080Ti
Revised: All our search experiments are done under the same 11 GB VRAM constraint on one RTX 2080Ti

Original: GPU.
Revised: GPU.

Original: For the conﬁguration of experiment on ImageNet, following DARTS, we search for the architecture in a cell manner, where each cell has four intermediate nodes and 16 initial channels during the search.
Revised: For the conﬁguration of experiment on ImageNet, following DARTS, we search for the architecture in a cell manner, where each cell has four intermediate nodes and 16 initial channels during the search.

Original: We use half the images ( 25 K) from the training set for kernel weight updating and the other half for architecture parameter updating.
Revised: We use half the images ( 25 K) from the training set for kernel weight updating and the other half for architecture parameter updating.

Original: When evaluating, we stack 18 normal and two reduction cells to form the entire network.
Revised: When evaluating, we stack 18 normal and two reduction cells to form the entire network.

Original: For the conﬁguration of experiment on ImageNet, when searching, we have two conﬁgurations.
Revised: For the conﬁguration of experiment on ImageNet, when searching, we have two conﬁgurations.

Original: In the ﬁrst conﬁguration, following PC-DARTS, we only pick 10% and 2 .
Revised: In the ﬁrst conﬁguration, following PC-DARTS, we only pick 10% and 2 .

Original: 5% of images from the training set for the kernel weights and architecture parameters’ optimizations respectively; in the second conﬁguration, thanks to FTSO’s huge computational cost reduction, it is feasible for us to search with the whole of ImageNet.
Revised: 5% of images from the training set for the kernel weights and architecture parameters’ optimizations respectively; in the second conﬁguration, thanks to FTSO’s huge computational cost reduction, it is feasible for us to search with the whole of ImageNet.

Original: For evaluation, we stack 12 normal cells and two reduction cells to form the entire network and train it from scratch for 250 epochs.
Revised: For evaluation, we stack 12 normal cells and two reduction cells to form the entire network and train it from scratch for 250 epochs.

Original: The detailed conﬁgurations are shown in Table 4.
Revised: The detailed conﬁgurations are shown in Table 4.

Original: [Table]
Revised: [Table]