Original: S TABILIZING DARTS WITH A MENDED G RADIENT E STIMATION ON A RCHITECTURAL P ARAMETERS
Revised: S TABILIZING DARTS WITH A MENDED G RADIENT E STIMATION ON A RCHITECTURAL P ARAMETERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Differentiable neural architecture search has been a popular methodology of exploring architectures for deep learning.
Revised: Differentiable neural architecture search has been a popular methodology of exploring architectures for deep learning.

Original: Despite the great advantage of search efﬁciency, it often suffers weak stability, which hinders it from being applied to a large search space or being ﬂexibly adjusted to different scenarios.
Revised: Despite the great advantage of search efﬁciency, it often suffers weak stability, which hinders it from being applied to a large search space or being ﬂexibly adjusted to different scenarios.

Original: This paper investigates DARTS, the currently most popular differentiable search algorithm, and points out an important factor of instability, which lies in its approximation on the gradients of architectural parameters.
Revised: This paper investigates DARTS, the currently most popular differentiable search algorithm, and points out an important factor of instability, which lies in its approximation on the gradients of architectural parameters.

Original: In the current status, the optimization algorithm can converge to another point which results in dramatic inaccuracy in the re-training process.
Revised: In the current status, the optimization algorithm can converge to another point which results in dramatic inaccuracy in the re-training process.

Original: Based on this analysis, we propose an amending term for computing architectural gradients by making use of a direct property of the optimality of network parameter optimization.
Revised: Based on this analysis, we propose an amending term for computing architectural gradients by making use of a direct property of the optimality of network parameter optimization.

Original: Our approach mathematically guarantees that gradient estimation follows a roughly correct direction, which leads the search stage to converge on reasonable architectures.
Revised: Our approach mathematically guarantees that gradient estimation follows a roughly correct direction, which leads the search stage to converge on reasonable architectures.

Original: In practice, our algorithm is easily implemented and added to DARTS-based approaches efﬁciently.
Revised: In practice, our algorithm is easily implemented and added to DARTS-based approaches efﬁciently.

Original: Experiments on CIFAR and ImageNet demonstrate that our approach enjoys accuracy gain and, more importantly, enables DARTS-based approaches to explore much larger search spaces that have not been studied before.
Revised: Experiments on CIFAR and ImageNet demonstrate that our approach enjoys accuracy gain and, more importantly, enables DARTS-based approaches to explore much larger search spaces that have not been studied before.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural architecture search (NAS) has been an important topic in the research area of automated machine learning (AutoML).
Revised: Neural architecture search (NAS) has been an important topic in the research area of automated machine learning (AutoML).

Original: The idea is to replace the manual way of designing neural network architectures with an automatic algorithm, by which deep learning methods become more ﬂexible in ﬁtting complex data distributions, e.g. , large-scale image datasets.
Revised: The idea is to replace the manual way of designing neural network architectures with an automatic algorithm, by which deep learning methods become more ﬂexible in ﬁtting complex data distributions, e.g. , large-scale image datasets.

Original: Early efforts of NAS involved using heuristic search methods such as reinforcement learning (Zoph & Le, 2017; Zoph et al., 2018) and evolutionary algorithms (Real et al., 2017; Xie & Yuille, 2017) to sample networks from a large search space, and optimizing each sampled network individually to evaluate its quality.
Revised: Early efforts of NAS involved using heuristic search methods such as reinforcement learning (Zoph & Le, 2017; Zoph et al., 2018) and evolutionary algorithms (Real et al., 2017; Xie & Yuille, 2017) to sample networks from a large search space, and optimizing each sampled network individually to evaluate its quality.

Original: Despite notable successes obtained by this methodology, it often requires a vast amount of computation, which obstacles its applications in the scenarios of limited resources.
Revised: Despite notable successes obtained by this methodology, it often requires a vast amount of computation, which obstacles its applications in the scenarios of limited resources.

Original: Inspired by the idea of reusing and sharing parameters among trained networks, DARTS (Liu et al., 2019b) was designed as a ‘one-shot’ solution of NAS.
Revised: Inspired by the idea of reusing and sharing parameters among trained networks, DARTS (Liu et al., 2019b) was designed as a ‘one-shot’ solution of NAS.

Original: The major difference from the aforementioned methods is a differentiable formulation of architecture search and an end-to-end mechanism which optimizes model weights (such as convolution) and architectural weights simultaneously.
Revised: The major difference from the aforementioned methods is a differentiable formulation of architecture search and an end-to-end mechanism which optimizes model weights (such as convolution) and architectural weights simultaneously.

Original: Recently, improvements upon DARTS were made in various aspects (Chen et al., 2019; Xu et al., 2019; Liang et al., 2019), making it a reasonable tradeoff between search cost and performance.
Revised: Recently, improvements upon DARTS were made in various aspects (Chen et al., 2019; Xu et al., 2019; Liang et al., 2019), making it a reasonable tradeoff between search cost and performance.

Original: Despite its broad applications, the current pipeline of DARTS (or, generally speaking, differentiable
Revised: Despite its broad applications, the current pipeline of DARTS (or, generally speaking, differentiable

Original: NAS approaches) suffers a critical weakness known as instability .
Revised: NAS approaches) suffers a critical weakness known as instability .

Original: Researchers reported (Liang et al., 2019) that DARTS-based algorithms can sometimes generate weird architectures that produce considerably worse accuracy than those generated in other individual runs, or even signiﬁcantly worse than randomly generated architectures.
Revised: Researchers reported (Liang et al., 2019) that DARTS-based algorithms can sometimes generate weird architectures that produce considerably worse accuracy than those generated in other individual runs, or even signiﬁcantly worse than randomly generated architectures.

Original: There indeed exist tricks designed by human expertise (Chen et al., 2019;
Revised: There indeed exist tricks designed by human expertise (Chen et al., 2019;

Original: Nayman et al., 2019; Liang et al., 2019) to alleviate this issue, but we point out that these approaches violated the ideology of NAS, which is to maximally prevent human interventions.
Revised: Nayman et al., 2019; Liang et al., 2019) to alleviate this issue, but we point out that these approaches violated the ideology of NAS, which is to maximally prevent human interventions.

Original: Moreover, even with such add-ons, a dramatic property of DARTS persists and has not been studied carefully in prior work.
Revised: Moreover, even with such add-ons, a dramatic property of DARTS persists and has not been studied carefully in prior work.

Original: When DARTS, as well as its variants, gets trained for a longer time, e.g. , from the default number of 50 epochs to 200 epochs, we surprisingly observe that all these approaches converge to very similar architectures, in which almost all edges are occupied by skip-connect ( a.k.a. , identity ).
Revised: When DARTS, as well as its variants, gets trained for a longer time, e.g. , from the default number of 50 epochs to 200 epochs, we surprisingly observe that all these approaches converge to very similar architectures, in which almost all edges are occupied by skip-connect ( a.k.a. , identity ).

Original: These architectures, with fewer trainable parameters, are often far from producing high accuracy in particular on large datasets like ImageNet, but they somehow produce sufﬁciently high validation accuracy in the search stage.
Revised: These architectures, with fewer trainable parameters, are often far from producing high accuracy in particular on large datasets like ImageNet, but they somehow produce sufﬁciently high validation accuracy in the search stage.

Original: In other words, convergence in search often leads to bad performance in re-training.
Revised: In other words, convergence in search often leads to bad performance in re-training.

Original: This is why some previous DARTS-based approaches advocated for early termination (Liang et al., 2019), a practical but non-essential solution.
Revised: This is why some previous DARTS-based approaches advocated for early termination (Liang et al., 2019), a practical but non-essential solution.

Original: Also, we conjecture that early termination also contributes to the lack of stability and, more importantly, trustfulness, of DARTS-based approaches.
Revised: Also, we conjecture that early termination also contributes to the lack of stability and, more importantly, trustfulness, of DARTS-based approaches.

Original: This paper delves deep into the inconsistency between convergence and performance.
Revised: This paper delves deep into the inconsistency between convergence and performance.

Original: We show that the devil lies in optimizing the loss function of the super-network, L val ( ω (cid:63) ( α ) , α ) ( ω and α denote network and architectural parameters, respectively, and ω (cid:63) ( α ) is the global optimum of ω given α ), in which ω and α get updated alternately.
Revised: We show that the devil lies in optimizing the loss function of the super-network, L val ( ω (cid:63) ( α ) , α ) ( ω and α denote network and architectural parameters, respectively, and ω (cid:63) ( α ) is the global optimum of ω given α ), in which ω and α get updated alternately.

Original: Following the chain rule, ∇ α L val ( ω (cid:63) ( α ) , α ) | α
Revised: Following the chain rule, ∇ α L val ( ω (cid:63) ( α ) , α ) | α

Original: = α t equals to ∇ α L val ( ω , α )
Revised: = α t equals to ∇ α L val ( ω , α )

Original: | ω = ω (cid:63) ( α t ) , α = α t + ∇ α ω (cid:63) ( α ) | α
Revised: | ω = ω (cid:63) ( α t ) , α = α t + ∇ α ω (cid:63) ( α ) | α

Original: = α t · ∇ ω L val ( ω , α ) | ω = ω (cid:63) ( α t ) , α = α t , in which the ﬁrst term is easy to compute while the second term is not, mainly because ω (cid:63) ( α ) is difﬁcult to estimate, and so is the term of ∇ α ω (cid:63) ( α ) | α
Revised: = α t · ∇ ω L val ( ω , α ) | ω = ω (cid:63) ( α t ) , α = α t , in which the ﬁrst term is easy to compute while the second term is not, mainly because ω (cid:63) ( α ) is difﬁcult to estimate, and so is the term of ∇ α ω (cid:63) ( α ) | α

Original: = α t .
Revised: = α t .

Original: DARTS-based approaches performed inaccurate approximation for this purpose, in which the ﬁrst-order version of DARTS directly discarded the second term – but this term is often numerically signiﬁcant, and the second-order version of DARTS applied an approximation to this term which is not mathematically guaranteed (see
Revised: DARTS-based approaches performed inaccurate approximation for this purpose, in which the ﬁrst-order version of DARTS directly discarded the second term – but this term is often numerically signiﬁcant, and the second-order version of DARTS applied an approximation to this term which is not mathematically guaranteed (see

Original: Section 3.4).
Revised: Section 3.4).

Original: Consequently, the accuracy of ∇ α L val ( ω (cid:63) ( α ) , α ) | α = α t cannot be guaranteed, and hence the update of α can be problematic.
Revised: Consequently, the accuracy of ∇ α L val ( ω (cid:63) ( α ) , α ) | α = α t cannot be guaranteed, and hence the update of α can be problematic.

Original: To the best of our knowledge, this issue is not studied by existing DARTS-based approaches.
Revised: To the best of our knowledge, this issue is not studied by existing DARTS-based approaches.

Original: To deal with this problem, we propose an alternative way of computing ∇ α ω (cid:63) ( α ) | α
Revised: To deal with this problem, we propose an alternative way of computing ∇ α ω (cid:63) ( α ) | α

Original: = α t .
Revised: = α t .

Original: We make use of an important property, i.e. , ∇ ω L train ( ω , α ) | ω
Revised: We make use of an important property, i.e. , ∇ ω L train ( ω , α ) | ω

Original: = ω (cid:63) ( α † ) , α = α † ≡ 0 holds for any α † , which directly comes from the optimality of ω (cid:63) ( α ) .
Revised: = ω (cid:63) ( α † ) , α = α † ≡ 0 holds for any α † , which directly comes from the optimality of ω (cid:63) ( α ) .

Original: Differentiating both sides with respect to α † , we obtain a new equality which enables computing derives ∇ α ω (cid:63) ( α ) | α
Revised: Differentiating both sides with respect to α † , we obtain a new equality which enables computing derives ∇ α ω (cid:63) ( α ) | α

Original: = α t with the inverse of the Hesse matrix, ∇ 2 ω L train ( ω , α ) (cid:12)(cid:12) ω
Revised: = α t with the inverse of the Hesse matrix, ∇ 2 ω L train ( ω , α ) (cid:12)(cid:12) ω

Original: = ω (cid:63) ( α ) , α = α t .
Revised: = ω (cid:63) ( α ) , α = α t .

Original: This idea enables us to achieve a more accurate approximation on ∇ α L val ( ω (cid:63) ( α ) , α ) | α
Revised: This idea enables us to achieve a more accurate approximation on ∇ α L val ( ω (cid:63) ( α ) , α ) | α

Original: = α t when ω (cid:63) ( α ) is not available.
Revised: = α t when ω (cid:63) ( α ) is not available.

Original: Mathematically, we prove that when we have ω est ≈ ω (cid:63) ( α t ) , the inner angle between the second term and our approximate term is smaller than 90 degrees.
Revised: Mathematically, we prove that when we have ω est ≈ ω (cid:63) ( α t ) , the inner angle between the second term and our approximate term is smaller than 90 degrees.

Original: Note that this property does not hold in existing DARTS-based algorithms.
Revised: Note that this property does not hold in existing DARTS-based algorithms.

Original: Our ﬁnal solution involves using the amended second term of ∇ α L val ( ω (cid:63) ( α ) , α ) | α
Revised: Our ﬁnal solution involves using the amended second term of ∇ α L val ( ω (cid:63) ( α ) , α ) | α

Original: = α t meanwhile keeping the ﬁrst term unchanged, which goes one step further in optimizing the supernetwork, which reﬂects in a higher validation accuracy in the search stage.
Revised: = α t meanwhile keeping the ﬁrst term unchanged, which goes one step further in optimizing the supernetwork, which reﬂects in a higher validation accuracy in the search stage.

Original: Our approach is very easily implemented.
Revised: Our approach is very easily implemented.

Original: The overall computational overhead is comparable to the second-order version of DARTS.
Revised: The overall computational overhead is comparable to the second-order version of DARTS.

Original: Experiments are performed on image classiﬁcation, with popular datasets including CIFAR and ImageNet being used.
Revised: Experiments are performed on image classiﬁcation, with popular datasets including CIFAR and ImageNet being used.

Original: In all experiments, we allow the search stage to come to a complete convergence and report competitive accuracy among current state-of-thearts.
Revised: In all experiments, we allow the search stage to come to a complete convergence and report competitive accuracy among current state-of-thearts.

Original: The stability of our approach also enables us to close the gap between hyper-parameters of search and evaluation, as well as explore more complex search spaces, which are believed to be correct directions of NAS but existing DARTS-based approaches would mostly fail.
Revised: The stability of our approach also enables us to close the gap between hyper-parameters of search and evaluation, as well as explore more complex search spaces, which are believed to be correct directions of NAS but existing DARTS-based approaches would mostly fail.

Original: Therefore, we believe our algorithm can expand the application scenario of differentiable NAS methods in particular DARTS-based approaches.
Revised: Therefore, we believe our algorithm can expand the application scenario of differentiable NAS methods in particular DARTS-based approaches.

Original: The remainder of this paper is organized as follows.
Revised: The remainder of this paper is organized as follows.

Original: We brieﬂy review related work in Section 2, and illustrate our approach of amending architectural gradients in Section 3.
Revised: We brieﬂy review related work in Section 2, and illustrate our approach of amending architectural gradients in Section 3.

Original: After experiments are shown in Section 4, we conclude this work in Section 5.
Revised: After experiments are shown in Section 4, we conclude this work in Section 5.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: With the era of big data and powerful computational resources, deep learning (LeCun et al., 2015), in particular, deep neural networks (Krizhevsky et al., 2012), have rapidly grown up to be the standard tool for learning representations in a complicated feature space.
Revised: With the era of big data and powerful computational resources, deep learning (LeCun et al., 2015), in particular, deep neural networks (Krizhevsky et al., 2012), have rapidly grown up to be the standard tool for learning representations in a complicated feature space.

Original: Recent years have witnessed the trend of using deeper (He et al., 2016) and denser (Huang et al., 2017) networks to boost recognition performance, while there is no justiﬁcation that whether these manually designed architectures are best for each speciﬁc task, e.g. , image classiﬁcation.
Revised: Recent years have witnessed the trend of using deeper (He et al., 2016) and denser (Huang et al., 2017) networks to boost recognition performance, while there is no justiﬁcation that whether these manually designed architectures are best for each speciﬁc task, e.g. , image classiﬁcation.

Original: To advance, researchers started considering the possibility of learning network architectures automatically from data, which led to the appearance of neural architecture search (NAS) (Zoph & Le, 2017), which is now popular and known as a sub research ﬁeld in automated machine learning (AutoML).
Revised: To advance, researchers started considering the possibility of learning network architectures automatically from data, which led to the appearance of neural architecture search (NAS) (Zoph & Le, 2017), which is now popular and known as a sub research ﬁeld in automated machine learning (AutoML).

Original: The common pipeline of NAS starts with a pre-deﬁned space of network operators.
Revised: The common pipeline of NAS starts with a pre-deﬁned space of network operators.

Original: Since the search space is often large ( e.g. , containing 10 10 or even more possible architectures), it is unlikely that exhaustive search is tractable, and thus heuristic search methods are widely applied for speedup.
Revised: Since the search space is often large ( e.g. , containing 10 10 or even more possible architectures), it is unlikely that exhaustive search is tractable, and thus heuristic search methods are widely applied for speedup.

Original: Typical examples include reinforcement learning (Zoph & Le, 2017; Zoph et al., 2018; Liu et al., 2018a) and evolutionary algorithms (Real et al., 2017; Xie & Yuille, 2017; Real et al., 2019).
Revised: Typical examples include reinforcement learning (Zoph & Le, 2017; Zoph et al., 2018; Liu et al., 2018a) and evolutionary algorithms (Real et al., 2017; Xie & Yuille, 2017; Real et al., 2019).

Original: These approaches followed a general pipeline that samples a set of architectures from a learnable distribution, evaluates them and learns from rewards by updating the distribution.
Revised: These approaches followed a general pipeline that samples a set of architectures from a learnable distribution, evaluates them and learns from rewards by updating the distribution.

Original: In an early age, each sampled architecture underwent an individual training process from scratch and thus the overall computational overhead is large, e.g. , hundreds of even thousands of GPU-days.
Revised: In an early age, each sampled architecture underwent an individual training process from scratch and thus the overall computational overhead is large, e.g. , hundreds of even thousands of GPU-days.

Original: To alleviate the burden, researchers started to share computation among training sampled architectures, with the key lying in reusing network weights trained previously (Cai et al., 2018) or starting from a well-trained super-network (Pham et al., 2018).
Revised: To alleviate the burden, researchers started to share computation among training sampled architectures, with the key lying in reusing network weights trained previously (Cai et al., 2018) or starting from a well-trained super-network (Pham et al., 2018).

Original: These efforts shed light on the so-called one-shot architecture search methods, which required training the super-network only once and thus ran more efﬁciently, e.g. , two or three orders of magnitude faster than conventional approaches.
Revised: These efforts shed light on the so-called one-shot architecture search methods, which required training the super-network only once and thus ran more efﬁciently, e.g. , two or three orders of magnitude faster than conventional approaches.

Original: Within the scope of one-shot architecture search, an elegant solution lies in jointly formulating architecture search and approximation, so that it is possible to apply end-to-end optimization for training network and architectural parameters simultaneously.
Revised: Within the scope of one-shot architecture search, an elegant solution lies in jointly formulating architecture search and approximation, so that it is possible to apply end-to-end optimization for training network and architectural parameters simultaneously.

Original: This methodology is known today as differentiable NAS, and a typical example is DARTS (Liu et al., 2019b), which constructed a super-network with all possible operators contained and decoupled, and the goal is to determine the weights of these architectural parameters, followed by pruning and re-training stages.
Revised: This methodology is known today as differentiable NAS, and a typical example is DARTS (Liu et al., 2019b), which constructed a super-network with all possible operators contained and decoupled, and the goal is to determine the weights of these architectural parameters, followed by pruning and re-training stages.

Original: This kind of approach allowed more ﬂexible search space to be constructed, unlike conventional approaches with either reinforcement or evolutionary learning, which suffer from the computational burden and thus must constrain search within a relatively small search space (Tan & Le, 2019).
Revised: This kind of approach allowed more ﬂexible search space to be constructed, unlike conventional approaches with either reinforcement or evolutionary learning, which suffer from the computational burden and thus must constrain search within a relatively small search space (Tan & Le, 2019).

Original: Despite the inspirations brought by differentiable NAS, these approaches still suffer a few critical issues that narrow down their applications in practice.
Revised: Despite the inspirations brought by differentiable NAS, these approaches still suffer a few critical issues that narrow down their applications in practice.

Original: One signiﬁcant drawback lies in the lack of stability, which reﬂects in the way that results of differentiable search can be impacted by very small perturbations, e.g. , initialization of architectural weights, training hyper-parameters, and even randomness in the training process.
Revised: One signiﬁcant drawback lies in the lack of stability, which reﬂects in the way that results of differentiable search can be impacted by very small perturbations, e.g. , initialization of architectural weights, training hyper-parameters, and even randomness in the training process.

Original: Existing solutions include running search for several individual times and choosing the best one in validation (Liu et al., 2019b), or using other kinds of techniques such as decoupling modules (Cai et al., 2019;
Revised: Existing solutions include running search for several individual times and choosing the best one in validation (Liu et al., 2019b), or using other kinds of techniques such as decoupling modules (Cai et al., 2019;

Original: Guo et al., 2019), adjusting search space during optimization (Noy et al., 2019; Chen et al., 2019;
Revised: Guo et al., 2019), adjusting search space during optimization (Noy et al., 2019; Chen et al., 2019;

Original: Nayman et al., 2019), regularization (Xu et al., 2019), early termination (Liang et al., 2019), etc ., however, these approaches seemed to develop heuristic remedies rather than analyze it from the mathematical fundamentals, e.g. , how instability happens in mathematics.
Revised: Nayman et al., 2019), regularization (Xu et al., 2019), early termination (Liang et al., 2019), etc ., however, these approaches seemed to develop heuristic remedies rather than analyze it from the mathematical fundamentals, e.g. , how instability happens in mathematics.

Original: In this paper, we investigate the stability issue in mathematics and show that the results produced by the current approaches are much less reliable than people used to think.
Revised: In this paper, we investigate the stability issue in mathematics and show that the results produced by the current approaches are much less reliable than people used to think.

Original: Then, we ﬁx this issue by amending optimization of the architectural parameters, so that each step of the update gets closer to the correct direction.
Revised: Then, we ﬁx this issue by amending optimization of the architectural parameters, so that each step of the update gets closer to the correct direction.

Original: We show great improvement on stability in a fundamental task, image classiﬁcation, while we believe our approach can be applied to a wide range of tasks including object detection (Ghiasi et al., 2019), semantic segmentation (Liu et al., 2019a), hyper-parameter learning (Cubuk et al., 2019), etc .
Revised: We show great improvement on stability in a fundamental task, image classiﬁcation, while we believe our approach can be applied to a wide range of tasks including object detection (Ghiasi et al., 2019), semantic segmentation (Liu et al., 2019a), hyper-parameter learning (Cubuk et al., 2019), etc .

Original:  S TABILIZING DARTS WITH A MENDED G RADIENTS
Revised:  S TABILIZING DARTS WITH A MENDED G RADIENTS

Original: In this section, we ﬁrst show that DARTS can fail dramatically when it gets trained till convergence, and then we mathematically analyze how this problem is related to inaccurate approximation in optimization, following which we present our solution to amend this error and thus stabilize DARTS.
Revised: In this section, we ﬁrst show that DARTS can fail dramatically when it gets trained till convergence, and then we mathematically analyze how this problem is related to inaccurate approximation in optimization, following which we present our solution to amend this error and thus stabilize DARTS.

Original: 3. P RELIMINARIES : D IFFERENTIABLE N EURAL A RCHITECTURE S EARCH
Revised: 3. P RELIMINARIES : D IFFERENTIABLE N EURAL A RCHITECTURE S EARCH

Original: Differentiable NAS approaches start with deﬁning a super-network, which is constrained in a search space with a pre-deﬁned number of layers and a limited set of neural operators.
Revised: Differentiable NAS approaches start with deﬁning a super-network, which is constrained in a search space with a pre-deﬁned number of layers and a limited set of neural operators.

Original: The core idea is to introduce a ‘soft’ way operator selection ( i.e. , using a weighted sum over the outputs of a few operators instead of taking the output of only one), so that optimization can be done in an end-to-
Revised: The core idea is to introduce a ‘soft’ way operator selection ( i.e. , using a weighted sum over the outputs of a few operators instead of taking the output of only one), so that optimization can be done in an end-to-

Original: [Figure]
Revised: [Figure]

Original: end manner.
Revised: end manner.

Original: Mathematically, the super-network is a function f ( x ; ω , α ) , with x being input, and parameterized by network parameters ω ( e.g. , convolutional kernels) and architectural parameters α
Revised: Mathematically, the super-network is a function f ( x ; ω , α ) , with x being input, and parameterized by network parameters ω ( e.g. , convolutional kernels) and architectural parameters α

Original: ( e.g. , indicating the importance of each operator between each pair of layers).
Revised: ( e.g. , indicating the importance of each operator between each pair of layers).

Original: f ( x ; ω , α ) is differentiable to both ω and α , so that gradient-based approaches can be applied for optimization.
Revised: f ( x ; ω , α ) is differentiable to both ω and α , so that gradient-based approaches can be applied for optimization.

Original: In the example of DARTS, f ( x ; ω , α ) is composed of a few cells, each of which contains N nodes, and there is a pre-deﬁned set, E , denoting which pairs of nodes are connected.
Revised: In the example of DARTS, f ( x ; ω , α ) is composed of a few cells, each of which contains N nodes, and there is a pre-deﬁned set, E , denoting which pairs of nodes are connected.

Original: For each connected node pair ( i, j ) , i < j , node j takes the output of node i , x i , as a part of its input, and propagate it through a pre-deﬁned operator set, O , with all outputs summed up:
Revised: For each connected node pair ( i, j ) , i < j , node j takes the output of node i , x i , as a part of its input, and propagate it through a pre-deﬁned operator set, O , with all outputs summed up:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Here, a softmax term is computed by architectural weights to normalize outputs.
Revised: Here, a softmax term is computed by architectural weights to normalize outputs.

Original: Within each unit of the search process, ω and α get optimized alternately.
Revised: Within each unit of the search process, ω and α get optimized alternately.

Original: After search, the operator o with the maximal value of α ( i,j ) o is preserved for each edge ( i, j ) .
Revised: After search, the operator o with the maximal value of α ( i,j ) o is preserved for each edge ( i, j ) .

Original: All network parameters ω are discarded and the obtained architecture is re-trained from scratch 1 .
Revised: All network parameters ω are discarded and the obtained architecture is re-trained from scratch 1 .

Original: 3. DARTS F AILS : T HE C ONTRADICTORY BETWEEN C ONVERGENCE AND P ERFORMANCE
Revised: 3. DARTS F AILS : T HE C ONTRADICTORY BETWEEN C ONVERGENCE AND P ERFORMANCE

Original: Our research is motivated by an observation that DARTS, at the end of a regular training process with, say, 50 epochs (Liu et al., 2019b), has not yet arrived at or even got close to convergence, yet the weight of the none ( a.k.a. , zero ) operator is consistently the largest on each edge of the normal cell.
Revised: Our research is motivated by an observation that DARTS, at the end of a regular training process with, say, 50 epochs (Liu et al., 2019b), has not yet arrived at or even got close to convergence, yet the weight of the none ( a.k.a. , zero ) operator is consistently the largest on each edge of the normal cell.

Original: To verify this, we increase the length of each training stage by 4 times, i.e. , from 50 to 200 epochs.
Revised: To verify this, we increase the length of each training stage by 4 times, i.e. , from 50 to 200 epochs.

Original: Two weird phenomena are observed, both of which are shown in the left part of Figure 1.
Revised: Two weird phenomena are observed, both of which are shown in the left part of Figure 1.

Original: First, the weight of the none operator monotonically goes up – at 200 epochs, the weight has achieved 0 .on
Revised: First, the weight of the none operator monotonically goes up – at 200 epochs, the weight has achieved 0 .on

Original: each edge of the normal cells, however, this operator is not considered in the ﬁnal architecture.
Revised: each edge of the normal cells, however, this operator is not considered in the ﬁnal architecture.

Original: Second, almost all preserved operators are skip-connect ( a.k.a. , identity ), a parameter-free operator that contributes little to feature learning – and surprisingly, it occupies 30% to 70% of the weight remained by none .
Revised: Second, almost all preserved operators are skip-connect ( a.k.a. , identity ), a parameter-free operator that contributes little to feature learning – and surprisingly, it occupies 30% to 70% of the weight remained by none .

Original: Such a network has much fewer parameters than a well-designed one, and thus it usually reports unsatisfying performance at the re-training stage.
Revised: Such a network has much fewer parameters than a well-designed one, and thus it usually reports unsatisfying performance at the re-training stage.

Original: This indicates that, in the context of DARTS, there exists a contradictory between search convergence and re-training accuracy.
Revised: This indicates that, in the context of DARTS, there exists a contradictory between search convergence and re-training accuracy.

Original: We point out that this is a critical issue, which suggests that when using DARTS-based approaches, one does not hope the search process to achieve convergence as it implies bad performance.
Revised: We point out that this is a critical issue, which suggests that when using DARTS-based approaches, one does not hope the search process to achieve convergence as it implies bad performance.

Original: In other words, each ‘successful’ architecture comes from an early-terminated search process.
Revised: In other words, each ‘successful’ architecture comes from an early-terminated search process.

Original: Consequently, the initialization of parameters ( α and ω ), the hyper-parameters of search ( e.g. , learning rate) and the time of terminating search become important and thus need to be determined by experience (Liang et al., 2019).
Revised: Consequently, the initialization of parameters ( α and ω ), the hyper-parameters of search ( e.g. , learning rate) and the time of terminating search become important and thus need to be determined by experience (Liang et al., 2019).

Original: This weakens the stability as well as explainability of search and, more importantly, violates the fundamental ideology of AutoML, i.e. , maximally reducing human interference and determining the best architecture by training data.
Revised: This weakens the stability as well as explainability of search and, more importantly, violates the fundamental ideology of AutoML, i.e. , maximally reducing human interference and determining the best architecture by training data.

Original: 3. D ELVING D EEP INTO M ATHEMATICS : P ROBLEM AND S OLUTION
Revised: 3. D ELVING D EEP INTO M ATHEMATICS : P ROBLEM AND S OLUTION

Original: We point out that the reason lies in inaccurate estimation of the gradient with respect to α , namely, ∇ α L val ( ω (cid:63) ( α ) , α ) | α = α t .
Revised: We point out that the reason lies in inaccurate estimation of the gradient with respect to α , namely, ∇ α L val ( ω (cid:63) ( α ) , α ) | α = α t .

Original: Following the chain rule of gradients, this quantity equals to
Revised: Following the chain rule of gradients, this quantity equals to

Original: [Equation]
Revised: [Equation]

Original: in which the ﬁrst term is relatively easy to compute (as is done by the ﬁrst-order version of DARTS), while the second term, in particular ∇ α ω (cid:63) ( α ) | α
Revised: in which the ﬁrst term is relatively easy to compute (as is done by the ﬁrst-order version of DARTS), while the second term, in particular ∇ α ω (cid:63) ( α ) | α

Original: = α t , is not.
Revised: = α t , is not.

Original: The ﬁrst-order version of DARTS directly discarded this term, but it often has signiﬁcant numerical values which are not negligible.
Revised: The ﬁrst-order version of DARTS directly discarded this term, but it often has signiﬁcant numerical values which are not negligible.

Original: The second-order version of DARTS indeed proposed an approximation to this term, but, as we shall see in the next subsection, can incur a large approximation error (the inner-product between the correct and estimated directions can be smaller than 0 ).
Revised: The second-order version of DARTS indeed proposed an approximation to this term, but, as we shall see in the next subsection, can incur a large approximation error (the inner-product between the correct and estimated directions can be smaller than 0 ).

Original: Consequently, there can be a signiﬁcant gap between the estimated and true values of ∇ α L val ( ω (cid:63) ( α ) , α ) | α
Revised: Consequently, there can be a signiﬁcant gap between the estimated and true values of ∇ α L val ( ω (cid:63) ( α ) , α ) | α

Original: = α t .
Revised: = α t .

Original: Such inaccuracy accumulates with every update on α , and gradually causes α to converge to weird solutions that are far from optimum, e.g. , the entire super-network is dominated by none and skip-connect operators.
Revised: Such inaccuracy accumulates with every update on α , and gradually causes α to converge to weird solutions that are far from optimum, e.g. , the entire super-network is dominated by none and skip-connect operators.

Original: We name this phenomenon as the gradient trap during optimization.
Revised: We name this phenomenon as the gradient trap during optimization.

Original: To estimate ∇ α ω (cid:63) ( α ) | α = α t , we ﬁrst make a reasonable assumption that ∇ α ω (cid:63) ( α ) | α
Revised: To estimate ∇ α ω (cid:63) ( α ) | α = α t , we ﬁrst make a reasonable assumption that ∇ α ω (cid:63) ( α ) | α

Original: = α t has ﬁnite values.
Revised: = α t has ﬁnite values.

Original: Then, we make use of an important property, i.e. , ∇
Revised: Then, we make use of an important property, i.e. , ∇

Original: [Equation]
Revised: [Equation]

Original: L train
Revised: L train

Original: [Equation]
Revised: [Equation]

Original: , α )
Revised: , α )

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: α
Revised: α

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: holds for any α † .
Revised: holds for any α † .

Original: This is property directly comes from the optimality of ω (cid:63) ( α ) , but it has never been used by existing approaches.
Revised: This is property directly comes from the optimality of ω (cid:63) ( α ) , but it has never been used by existing approaches.

Original: Applying differentiation with respect to any α † to both sides of this equality, we have ∇ α †
Revised: Applying differentiation with respect to any α † to both sides of this equality, we have ∇ α †

Original: [Equation]
Revised: [Equation]

Original: L train
Revised: L train

Original: [Equation]
Revised: [Equation]

Original: ω , α ) | ω = ω
Revised: ω , α ) | ω = ω

Original: [Equation]
Revised: [Equation]

Original: ( α † ) , α = α †
Revised: ( α † ) , α = α †

Original: [Equation]
Revised: [Equation]

Original: ≡ 0 .
Revised: ≡ 0 .

Original: When α † = α t , it becomes:
Revised: When α † = α t , it becomes:

Original: [Equation]
Revised: [Equation]

Original: Again, applying the chain rule to the left-hand side gives:
Revised: Again, applying the chain rule to the left-hand side gives:

Original: [Equation]
Revised: [Equation]

Original: where we use the notation ∇ 2 α , ω ( · ) ≡ ∇ α ( ∇ ω ( · )) throughout the remaining part of this paper.
Revised: where we use the notation ∇ 2 α , ω ( · ) ≡ ∇ α ( ∇ ω ( · )) throughout the remaining part of this paper.

Original: Here, ∇ 2 ω L train ( ω , α ) (cid:12)(cid:12) ω = ω (cid:63) ( α t ) , α = α t .
Revised: Here, ∇ 2 ω L train ( ω , α ) (cid:12)(cid:12) ω = ω (cid:63) ( α t ) , α = α t .

Original: = H is the Hesse matrix corresponding to the optimum ω (cid:63) ( α t ) , which is symmetric and positive-deﬁnite, and thus invertible.
Revised: = H is the Hesse matrix corresponding to the optimum ω (cid:63) ( α t ) , which is symmetric and positive-deﬁnite, and thus invertible.

Original: This gives us an estimation that ∇ α ω (cid:63) (
Revised: This gives us an estimation that ∇ α ω (cid:63) (

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: L
Revised: L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ω =
Revised: ω =

Original: [Equation]
Revised: [Equation]

Original: (cid:63) ( α t ) , α = α t · H − 1 .
Revised: (cid:63) ( α t ) , α = α t · H − 1 .

Original: Substituting it into Equationgives:
Revised: Substituting it into Equationgives:

Original: [Equation]
Revised: [Equation]

Original: Note that Equation 5 does not involve any approximation.
Revised: Note that Equation 5 does not involve any approximation.

Original: The only issue comes from the term of
Revised: The only issue comes from the term of

Original: H − 1 , which is computationally intractable due to the large dimensionality of H (it is related to the number of network parameters, which often exceeds one million in a typical super-network).
Revised: H − 1 , which is computationally intractable due to the large dimensionality of H (it is related to the number of network parameters, which often exceeds one million in a typical super-network).

Original: 3. A PPROXIMATIONS IN C OMPUTING THE I NVERSE H ESSE M ATRIX
Revised: 3. A PPROXIMATIONS IN C OMPUTING THE I NVERSE H ESSE M ATRIX

Original: Let us denote Equation 5 in an abbreviated form of g = g 1 + g 2 , in which g 1 , the ﬁrst-order term of DARTS, is easily computed, while g 2 is not due to the computation of H − 1 .
Revised: Let us denote Equation 5 in an abbreviated form of g = g 1 + g 2 , in which g 1 , the ﬁrst-order term of DARTS, is easily computed, while g 2 is not due to the computation of H − 1 .

Original: Here, we propose an alternative solution which constructs an approximation term g (cid:48) 2 :
Revised: Here, we propose an alternative solution which constructs an approximation term g (cid:48) 2 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where η > 0 is named the amending coefﬁcient .
Revised: where η > 0 is named the amending coefﬁcient .

Original: In what follows, we show that g (cid:48) 2 is indeed a reasonable approximation of g 2 .
Revised: In what follows, we show that g (cid:48) 2 is indeed a reasonable approximation of g 2 .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: the product of the two terms between H − 1 and H is a semi-positive-deﬁnite matrix, and so is the matrix after similarity transformation, which directly gives (cid:104) g (cid:48) 2 , g 2 (cid:105) (cid:62) 0 .
Revised: the product of the two terms between H − 1 and H is a semi-positive-deﬁnite matrix, and so is the matrix after similarity transformation, which directly gives (cid:104) g (cid:48) 2 , g 2 (cid:105) (cid:62) 0 .

Original: In summary, we decompose the gradient of architectural parameters into two terms, g 1 and g 2 , compute g 1 directly and use an approximation to g 2 so that the angle between the accurate and approximated terms is smaller than 90 degrees.
Revised: In summary, we decompose the gradient of architectural parameters into two terms, g 1 and g 2 , compute g 1 directly and use an approximation to g 2 so that the angle between the accurate and approximated terms is smaller than 90 degrees.

Original: In comparison, existing DARTS-based approaches either discarded g 2 entirely or used a mathematically non-explainable approximation g
Revised: In comparison, existing DARTS-based approaches either discarded g 2 entirely or used a mathematically non-explainable approximation g

Original: [Equation]
Revised: [Equation]

Original: None of them are reasonable because g 2 can be large, yet there is no guarantee that (cid:104) g (cid:48)(cid:48)
Revised: None of them are reasonable because g 2 can be large, yet there is no guarantee that (cid:104) g (cid:48)(cid:48)

Original: [Equation]
Revised: [Equation]

Original: g 2 (cid:105) (cid:62) 0 , i.e. , the secondorder DARTS can lead the algorithm to a wrong direction.
Revised: g 2 (cid:105) (cid:62) 0 , i.e. , the secondorder DARTS can lead the algorithm to a wrong direction.

Original: The rationality of our approach is also veriﬁed by the validation process, i.e. , in updating architectural parameters.
Revised: The rationality of our approach is also veriﬁed by the validation process, i.e. , in updating architectural parameters.

Original: The ﬁrst-order DARTS, by directly discarding g 2 , reported an average validation accuracy of 90 .
Revised: The ﬁrst-order DARTS, by directly discarding g 2 , reported an average validation accuracy of 90 .

Original: 5% in search space S 1 (see Section 4.1.3) on the CIFAR10 dataset.
Revised: 5% in search space S 1 (see Section 4.1.3) on the CIFAR10 dataset.

Original: The second-order
Revised: The second-order

Original: DARTS added g (cid:48)(cid:48) 2 , which has no guarantee that (cid:104) g (cid:48)(cid:48) 2 , g 2 (cid:105) (cid:62) 0 , and thus resulted in a reduced validation accuracy.
Revised: DARTS added g (cid:48)(cid:48) 2 , which has no guarantee that (cid:104) g (cid:48)(cid:48) 2 , g 2 (cid:105) (cid:62) 0 , and thus resulted in a reduced validation accuracy.

Original: Our approach, by adding g (cid:48) 2 , achieves a validation accuracy of 91 .
Revised: Our approach, by adding g (cid:48) 2 , achieves a validation accuracy of 91 .

Original: 5% , implying that our optimization works better than DARTS.
Revised: 5% , implying that our optimization works better than DARTS.

Original: This eventually results in the advantage of searched architectures, which will be veriﬁed in Section 4.1.2.
Revised: This eventually results in the advantage of searched architectures, which will be veriﬁed in Section 4.1.2.

Original: The remainder part of computing g (cid:48) 2 simply follows conventions, which we replace ω (cid:63) ( α ) with the current ω est as the most accurate approximation we can get 2 .
Revised: The remainder part of computing g (cid:48) 2 simply follows conventions, which we replace ω (cid:63) ( α ) with the current ω est as the most accurate approximation we can get 2 .

Original: Computing g (cid:48) 2 with Equationrequires both ∇ 2 α , ω L train ( ω , α ) (cid:12)(cid:12)
Revised: Computing g (cid:48) 2 with Equationrequires both ∇ 2 α , ω L train ( ω , α ) (cid:12)(cid:12)

Original: ω
Revised: ω

Original: = ω (cid:63) ( α t ) , α = α t and ∇ ω L val ( ω , α ) | ω = ω (cid:63) ( α t ) , α = α t , while DARTS needs the former one with ω (cid:63) ( α t ) estimated in two steps.
Revised: = ω (cid:63) ( α t ) , α = α t and ∇ ω L val ( ω , α ) | ω = ω (cid:63) ( α t ) , α = α t , while DARTS needs the former one with ω (cid:63) ( α t ) estimated in two steps.

Original: Therefore, computing Equation 6 requires similar computational overhead compared to the second-order version of DARTS.
Revised: Therefore, computing Equation 6 requires similar computational overhead compared to the second-order version of DARTS.

Original: In experiments, each search epoch requires around 0 .
Revised: In experiments, each search epoch requires around 0 .

Original: 02 GPU-days on the standard 8 -cell space on CIFAR10.
Revised: 02 GPU-days on the standard 8 -cell space on CIFAR10.

Original: 3. D ISCUSSIONS AND R ELATIONSHIP TO P RIOR W ORK
Revised: 3. D ISCUSSIONS AND R ELATIONSHIP TO P RIOR W ORK

Original: The core beneﬁt brought by our approach is the consistency between search and evaluation.
Revised: The core beneﬁt brought by our approach is the consistency between search and evaluation.

Original: This is indeed a fundamental idea of NAS, but it was ignored by existing approaches since they have been perplexed by a more signiﬁcant error caused by inaccurate optimization.
Revised: This is indeed a fundamental idea of NAS, but it was ignored by existing approaches since they have been perplexed by a more signiﬁcant error caused by inaccurate optimization.

Original: After the error, we point out a few prior conventions that need to be adjusted accordingly, including using different depths ( e.g. , DARTS used 8 cells in search and 20 cells in evaluation) and widths ( e.g. , DARTS used a basic channel number of 16 in search and 36 in evaluation) during search and evaluation, as well as using different training strategies ( e.g. , during re-training, a few regularization techniques including
Revised: After the error, we point out a few prior conventions that need to be adjusted accordingly, including using different depths ( e.g. , DARTS used 8 cells in search and 20 cells in evaluation) and widths ( e.g. , DARTS used a basic channel number of 16 in search and 36 in evaluation) during search and evaluation, as well as using different training strategies ( e.g. , during re-training, a few regularization techniques including

Original: Cutout (DeVries & Taylor, 2017), Dropout (Srivastava et al., 2014) and auxiliary loss were used, but none of them appeared in search).
Revised: Cutout (DeVries & Taylor, 2017), Dropout (Srivastava et al., 2014) and auxiliary loss were used, but none of them appeared in search).

Original: More importantly, the search process was followed by edge removal ( 8 out of 14 connections were preserved) which caused a signiﬁcant difference between the network architectures of search and evaluation.
Revised: More importantly, the search process was followed by edge removal ( 8 out of 14 connections were preserved) which caused a signiﬁcant difference between the network architectures of search and evaluation.

Original: Our approach provides the opportunity to bridge the gap between search and evaluation, which we will show in Section 4.1.2 that unifying these hyper-parameters leads to better performance.
Revised: Our approach provides the opportunity to bridge the gap between search and evaluation, which we will show in Section 4.1.2 that unifying these hyper-parameters leads to better performance.

Original: A few prior differentiable search approaches noticed the issue of instability, but they chose to solve it in different manners.
Revised: A few prior differentiable search approaches noticed the issue of instability, but they chose to solve it in different manners.

Original: For example, P-DARTS (Chen et al., 2019) ﬁxed the number of preserved skip-connect operators, PC-DARTS
Revised: For example, P-DARTS (Chen et al., 2019) ﬁxed the number of preserved skip-connect operators, PC-DARTS

Original: (Xu et al., 2019) used edge normalization to eliminate the none operator, while XNAS (Nayman et al., 2019) and DARTS+ (Liang et al., 2019) introduced a few human expertise to stabilize search.
Revised: (Xu et al., 2019) used edge normalization to eliminate the none operator, while XNAS (Nayman et al., 2019) and DARTS+ (Liang et al., 2019) introduced a few human expertise to stabilize search.

Original: However, we point out that (i) either P-DARTS or PC-DARTS, with carefully designed methods or tricks, can also fail in a sufﬁciently long search process (more than 200 epochs); and that (ii) XNAS and DARTS+, by adding human expertise, somewhat violated the design nature of AutoML, in which one is expected to avoid introducing too many hand-designed rules.
Revised: However, we point out that (i) either P-DARTS or PC-DARTS, with carefully designed methods or tricks, can also fail in a sufﬁciently long search process (more than 200 epochs); and that (ii) XNAS and DARTS+, by adding human expertise, somewhat violated the design nature of AutoML, in which one is expected to avoid introducing too many hand-designed rules.

Original: Another line of NAS, besides differentiable methods, is to use either reinforcement learning or an evolutionary algorithm as a controller of heuristic search and train each sampled network to get some kind of rewards, e.g. , validation accuracy.
Revised: Another line of NAS, besides differentiable methods, is to use either reinforcement learning or an evolutionary algorithm as a controller of heuristic search and train each sampled network to get some kind of rewards, e.g. , validation accuracy.

Original: In the viewpoint of optimization, this pipeline mainly differs from the differentiable one in that optimizing α is decoupled from optimizing ω , so that it does not require ω to arrive at ω ∗ , but only need a reasonable approximation of ω ∗ to predict model performance – this is an important reason that such algorithms often produce stable results.
Revised: In the viewpoint of optimization, this pipeline mainly differs from the differentiable one in that optimizing α is decoupled from optimizing ω , so that it does not require ω to arrive at ω ∗ , but only need a reasonable approximation of ω ∗ to predict model performance – this is an important reason that such algorithms often produce stable results.

Original: Our approach sheds light on introducing a similar property, i.e. , robustness to approximated ω (cid:63) , which helps in stabilizing differentiable search approaches.
Revised: Our approach sheds light on introducing a similar property, i.e. , robustness to approximated ω (cid:63) , which helps in stabilizing differentiable search approaches.

Original:  E XPERIMENTS 4. R ESULTS ON CIFAR 
Revised:  E XPERIMENTS 4. R ESULTS ON CIFAR

Original: The CIFAR10 dataset (Krizhevsky & Hinton, 2009) has 50 , 000 training and 10 , 000 testing images, sized 32 × 32 , and equally distributed over 10 classes.
Revised: The CIFAR10 dataset (Krizhevsky & Hinton, 2009) has 50 , 000 training and 10 , 000 testing images, sized 32 × 32 , and equally distributed over 10 classes.

Original: We mainly use this dataset to evaluate the stability of our approach, as well as analyze the impacts of different search options and parameters.
Revised: We mainly use this dataset to evaluate the stability of our approach, as well as analyze the impacts of different search options and parameters.

Original: 4.1. I MPACT OF THE A MENDING C OEFFICIENT
Revised: 4.1. I MPACT OF THE A MENDING C OEFFICIENT

Original: We ﬁrst investigate how the amending coefﬁcient, η , deﬁned in Equation 6, impacts architecture search.
Revised: We ﬁrst investigate how the amending coefﬁcient, η , deﬁned in Equation 6, impacts architecture search.

Original: We search and re-train similarly as DARTS.
Revised: We search and re-train similarly as DARTS.

Original: During the search, all operators are assigned equal weights on each edge.
Revised: During the search, all operators are assigned equal weights on each edge.

Original: We use a base channel number of 16 , and a batch size of 96 .
Revised: We use a base channel number of 16 , and a batch size of 96 .

Original: An
Revised: An

Original: Adam optimizer is used to update architectural parameters, with a learning rate of 0 .
Revised: Adam optimizer is used to update architectural parameters, with a learning rate of 0 .

Original: 0003 , a weight decay of 0 .
Revised: 0003 , a weight decay of 0 .

Original: 001 and a momentum of (0 . 5 , 0 . 999) .
Revised: 001 and a momentum of (0 . 5 , 0 . 999) .

Original: The number of epochs is to be discussed later.
Revised: The number of epochs is to be discussed later.

Original: During re-training, the base channel number is increased to 36 .
Revised: During re-training, the base channel number is increased to 36 .

Original: An SGD optimizer is used with an initial learning rate of 0 .
Revised: An SGD optimizer is used with an initial learning rate of 0 .

Original: 025 , decaying following the cosine annealing rule and arriving at 0 after 600 epochs.
Revised: 025 , decaying following the cosine annealing rule and arriving at 0 after 600 epochs.

Original: The weight decay is set to be 0 . 0003 , and the momentum is 0 .
Revised: The weight decay is set to be 0 . 0003 , and the momentum is 0 .

Original: To arrive at convergence, we run the search stage for 500 epochs.
Revised: To arrive at convergence, we run the search stage for 500 epochs.

Original: We evaluate different η values from 0 to 1 , and the architectures corresponding to η = 0 (equivalent to DARTS), η = 0 .
Revised: We evaluate different η values from 0 to 1 , and the architectures corresponding to η = 0 (equivalent to DARTS), η = 0 .

Original: 1 and η = 1 are summarized in Figure 2.
Revised: 1 and η = 1 are summarized in Figure 2.

Original: We can see that η = 0 .
Revised: We can see that η = 0 .

Original: 1 converges, after 500 , into a reasonable architecture that achieves an error rate of 3 .
Revised: 1 converges, after 500 , into a reasonable architecture that achieves an error rate of 3 .

Original: 08% on CIFAR10.
Revised: 08% on CIFAR10.

Original: We emphasize that, even with more search epochs, this architecture is not likely to change, as the preserved operator on each edge has a weight not smaller than 0 . 5 , and most of these weights are still growing gradually.
Revised: We emphasize that, even with more search epochs, this architecture is not likely to change, as the preserved operator on each edge has a weight not smaller than 0 . 5 , and most of these weights are still growing gradually.

Original: When η is very small, e.g. , η = 0 .
Revised: When η is very small, e.g. , η = 0 .

Original: 001 or η = 0 .
Revised: 001 or η = 0 .

Original: 01 , the change brought by this amending term to architecture search is ignorable, and our approach shows almost the same behavior as the ﬁrst-order version of DARTS, i.e. , η = 0 .
Revised: 01 , the change brought by this amending term to architecture search is ignorable, and our approach shows almost the same behavior as the ﬁrst-order version of DARTS, i.e. , η = 0 .

Original: In addition, when η grows up, e.g. , from 0 .
Revised: In addition, when η grows up, e.g. , from 0 .

Original: 001 to 0 .
Revised: 001 to 0 .

Original: 01 , although the search process eventually runs into an architecture with all skip-connect operators, the number of epochs needed for a complete failure is signiﬁcantly postponed, which veriﬁes that the amending term indeed pulls architecture search away from the gradient trap.
Revised: 01 , although the search process eventually runs into an architecture with all skip-connect operators, the number of epochs needed for a complete failure is signiﬁcantly postponed, which veriﬁes that the amending term indeed pulls architecture search away from the gradient trap.

Original: [Figure]
Revised: [Figure]

Original: On the other hand, if we use a sufﬁciently large η value, e.g. , η = 1 , the amending term, g (cid:48) 2 , can dominate optimization, so that the ﬁrst term, i.e. , the gradient of architectural parameters, has limited effects in updating α .
Revised: On the other hand, if we use a sufﬁciently large η value, e.g. , η = 1 , the amending term, g (cid:48) 2 , can dominate optimization, so that the ﬁrst term, i.e. , the gradient of architectural parameters, has limited effects in updating α .

Original: Note that the amending term is closely related to network regularization, therefore, in the scenarios of a large η , the network signiﬁcantly prefers avg-pooling to other operators, as avg-pooling can smooth feature maps and avoid over-ﬁtting.
Revised: Note that the amending term is closely related to network regularization, therefore, in the scenarios of a large η , the network signiﬁcantly prefers avg-pooling to other operators, as avg-pooling can smooth feature maps and avoid over-ﬁtting.

Original: However, avg-pooling is also a parameter-free operator, so the performance of such architectures is also below satisfaction.
Revised: However, avg-pooling is also a parameter-free operator, so the performance of such architectures is also below satisfaction.

Original: Following these analyses, we simply use η = 0 .
Revised: Following these analyses, we simply use η = 0 .

Original: 1 for all later experiments.
Revised: 1 for all later experiments.

Original: We do not tune η very carefully, though it is possible to determine η automatically using a held-out validation set.
Revised: We do not tune η very carefully, though it is possible to determine η automatically using a held-out validation set.

Original: Besides, we ﬁnd that the best architecture barely changes after 100 search epochs, which we ﬁx the total length to be 100 epochs in all remaining experiments.
Revised: Besides, we ﬁnd that the best architecture barely changes after 100 search epochs, which we ﬁx the total length to be 100 epochs in all remaining experiments.

Original: 4.1. C ONSISTENCY BETWEEN S EARCH AND E VALUATION
Revised: 4.1. C ONSISTENCY BETWEEN S EARCH AND E VALUATION

Original: As discussed in Section 3.5, it is important to alleviate the difference between search and evaluation.
Revised: As discussed in Section 3.5, it is important to alleviate the difference between search and evaluation.

Original: We make the following modiﬁcations, listed from most to least important.
Revised: We make the following modiﬁcations, listed from most to least important.

Original: First, to avoid edge removal, we ﬁx the edges in each cell, so that each node i is connected to node i − 1 and the least indexed node (denoted by c k − 2 in most conventions), resulting in 8 edges in each cell.
Revised: First, to avoid edge removal, we ﬁx the edges in each cell, so that each node i is connected to node i − 1 and the least indexed node (denoted by c k − 2 in most conventions), resulting in 8 edges in each cell.

Original: Note that our approach also works well with all 14 edges preserved, but we have used 8 edges to be computationally fair to DARTS.
Revised: Note that our approach also works well with all 14 edges preserved, but we have used 8 edges to be computationally fair to DARTS.

Original: Second, we unify the width (the number of basic channels) asfor both search and evaluation.
Revised: Second, we unify the width (the number of basic channels) asfor both search and evaluation.

Original: Third, we add normalization techniques, including Cutout (DeVries & Taylor, 2017), Dropout (Srivastava et al., 2014) and an auxiliary loss tower, into the search stage.
Revised: Third, we add normalization techniques, including Cutout (DeVries & Taylor, 2017), Dropout (Srivastava et al., 2014) and an auxiliary loss tower, into the search stage.

Original: We use the amending coefﬁcient η = 0 .
Revised: We use the amending coefﬁcient η = 0 .

Original: 1 learned from previous experiments, i.e. , without modiﬁcation, the searched architecture, denoted by A orig , is shown in the middle column of Figure 2.
Revised: 1 learned from previous experiments, i.e. , without modiﬁcation, the searched architecture, denoted by A orig , is shown in the middle column of Figure 2.

Original: After modiﬁcation, the obtained architecture, denoted by A new , is shown in Figure 3.
Revised: After modiﬁcation, the obtained architecture, denoted by A new , is shown in Figure 3.

Original: We re-train both networks on CIFAR10, with or without the option that stacking duplicate cells to make the network deeper (with 20 cells).
Revised: We re-train both networks on CIFAR10, with or without the option that stacking duplicate cells to make the network deeper (with 20 cells).

Original: With a standard re-training process, A orig reports a 3 .
Revised: With a standard re-training process, A orig reports a 3 .

Original: 67% error with 8 cells, and a 3 . 08% error with 20 cells; and the corresponding numbers are 3 .
Revised: 67% error with 8 cells, and a 3 . 08% error with 20 cells; and the corresponding numbers are 3 .

Original: 20% and 2 .
Revised: 20% and 2 .

Original: 81% for A new .
Revised: 81% for A new .

Original: We ﬁnd that A new is consistently better than A orig , which suggests that alleviating the gap indeed helps.
Revised: We ﬁnd that A new is consistently better than A orig , which suggests that alleviating the gap indeed helps.

Original: This also reminds us of the signiﬁcant depth gap (Chen et al., 2019) between search and re-training (the network has 8 cells in search, but 20 cells in re-training), and this gap also obstructs our approach from achieving better performance.
Revised: This also reminds us of the signiﬁcant depth gap (Chen et al., 2019) between search and re-training (the network has 8 cells in search, but 20 cells in re-training), and this gap also obstructs our approach from achieving better performance.

Original: We will investigate this issue in the following part.
Revised: We will investigate this issue in the following part.

Original: 4.1. E XPLORING M ORE C OMPLEX S EARCH S PACES
Revised: 4.1. E XPLORING M ORE C OMPLEX S EARCH S PACES

Original: We note tremendous efforts made by existing approaches to alleviate the depth gap , while our solution is a direct one, thanks to the stability of our approach which enables us to directly explore larger search spaces.
Revised: We note tremendous efforts made by existing approaches to alleviate the depth gap , while our solution is a direct one, thanks to the stability of our approach which enables us to directly explore larger search spaces.

Original: Here, we denote the original search space used in DARTS as S 1 , which has six normal cells and two reduction cells, and all normal cells share the same set of architectural parameters and so do the reduction cells.
Revised: Here, we denote the original search space used in DARTS as S 1 , which has six normal cells and two reduction cells, and all normal cells share the same set of architectural parameters and so do the reduction cells.

Original: Note that we have ﬁxed the edges in this space, resulting in the total number of possible architectures to reduce from 1 .
Revised: Note that we have ﬁxed the edges in this space, resulting in the total number of possible architectures to reduce from 1 .

Original: 1 × 10 18 to 3 .
Revised: 1 × 10 18 to 3 .

Original: 3 × 10 13 .
Revised: 3 × 10 13 .

Original: We also explore a more complex search space, denoted by S 2 , in which we relax the constraint that either normal cells or reduction cells should be the same, and also the number of cells increases from 8 to 20 , to be applied in re-training.
Revised: We also explore a more complex search space, denoted by S 2 , in which we relax the constraint that either normal cells or reduction cells should be the same, and also the number of cells increases from 8 to 20 , to be applied in re-training.

Original: Here, limited by GPU memory, we cannot support all seven operators to be searched, so we only choose two, namely skip-connect and sep-conv-3x3 , which have very different properties.
Revised: Here, limited by GPU memory, we cannot support all seven operators to be searched, so we only choose two, namely skip-connect and sep-conv-3x3 , which have very different properties.

Original: This setting allows a total of 1 . 5 × 10 48 architectures to appear, much larger than S 1 .
Revised: This setting allows a total of 1 . 5 × 10 48 architectures to appear, much larger than S 1 .

Original: Results are listed in Table 1.
Revised: Results are listed in Table 1.

Original: In S 1 , our approach achieves a moderate error rate of 2 .
Revised: In S 1 , our approach achieves a moderate error rate of 2 .

Original: 81% , mainly because the assumption of consistency between search and re-training does not hold.
Revised: 81% , mainly because the assumption of consistency between search and re-training does not hold.

Original: In S 2 , with directly searching in deep architectures, our result is signiﬁcantly boosted (an error rate of 2 .
Revised: In S 2 , with directly searching in deep architectures, our result is signiﬁcantly boosted (an error rate of 2 .

Original: 60% , the architecture is shown in the middle row of Figure 3).
Revised: 60% , the architecture is shown in the middle row of Figure 3).

Original: Again, we emphasize that we report retraining results based on an converged architecture, which stands out from existing DARTS-based approaches which required early termination.
Revised: Again, we emphasize that we report retraining results based on an converged architecture, which stands out from existing DARTS-based approaches which required early termination.

Original: In S 2 , we compare our approach against both DARTS (no amending term) and random search.
Revised: In S 2 , we compare our approach against both DARTS (no amending term) and random search.

Original: We observe that DARTS produces weird architectures (the bottom row of Figure 3), that high-layer cells are mostly occupied by skip-connect , which is not likely to fully utilize the ability of the supernetwork.
Revised: We observe that DARTS produces weird architectures (the bottom row of Figure 3), that high-layer cells are mostly occupied by skip-connect , which is not likely to fully utilize the ability of the supernetwork.

Original: Regarding random search, we follow DARTS by randomly sampling 20 valid architectures from each of S 1 and S 2 , and using a small validation dataset to choose the best two architectures for re-training.
Revised: Regarding random search, we follow DARTS by randomly sampling 20 valid architectures from each of S 1 and S 2 , and using a small validation dataset to choose the best two architectures for re-training.

Original: Given a ﬁxed number of probes, it becomes more and more difﬁcult to sufﬁciently
Revised: Given a ﬁxed number of probes, it becomes more and more difﬁcult to sufﬁciently

Original: [Figure]
Revised: [Figure]

Original: explore a large space.
Revised: explore a large space.

Original: The deﬁcits of DARTS and random search on CIFAR10 are 0 .
Revised: The deﬁcits of DARTS and random search on CIFAR10 are 0 .

Original: 25% and 0 .
Revised: 25% and 0 .

Original: 29% , respectively, which do not seem to be very large arguably because CIFAR10 is relatively easy and our edge-ﬁxed search space guarantees sufﬁcient depth.
Revised: 29% , respectively, which do not seem to be very large arguably because CIFAR10 is relatively easy and our edge-ﬁxed search space guarantees sufﬁcient depth.

Original: However, when we transfer these architectures to ImageNet, the deﬁcits become much larger, i.e. , with 1 . 7% and 0 .
Revised: However, when we transfer these architectures to ImageNet, the deﬁcits become much larger, i.e. , with 1 . 7% and 0 .

Original: 8% top- 1 accuracy drops, respectively (DARTS produces inferior performance to random search).
Revised: 8% top- 1 accuracy drops, respectively (DARTS produces inferior performance to random search).

Original: These results remind us of prior work (Xie et al., 2019) which claimed that random search works sufﬁciently well in large search spaces.
Revised: These results remind us of prior work (Xie et al., 2019) which claimed that random search works sufﬁciently well in large search spaces.

Original: Here, we leave a comment on this debate, demonstrating that a large space indeed raises challenges to architecture search, but a stabilized search algorithm still has the ability of to ﬁnd more powerful architectures.
Revised: Here, we leave a comment on this debate, demonstrating that a large space indeed raises challenges to architecture search, but a stabilized search algorithm still has the ability of to ﬁnd more powerful architectures.

Original: 4.1. C OMPARISON TO THE S TATE - OF - THE -A RTS
Revised: 4.1. C OMPARISON TO THE S TATE - OF - THE -A RTS

Original: Finally, we compare our approach with recent approaches, in particular, differentiable ones.
Revised: Finally, we compare our approach with recent approaches, in particular, differentiable ones.

Original: Result are shown in Table 1.
Revised: Result are shown in Table 1.

Original: Our approach produces competitive results among state-of-the-arts, although it does not seem to beat others.
Revised: Our approach produces competitive results among state-of-the-arts, although it does not seem to beat others.

Original: We note that existing approaches often used additional tricks, e
Revised: We note that existing approaches often used additional tricks, e

Original: [Table]
Revised: [Table]

Original: g. , PDARTS assumed a ﬁxed number of skip-connect operators, which shrinks the search space (so as to guarantee stability).
Revised: g. , PDARTS assumed a ﬁxed number of skip-connect operators, which shrinks the search space (so as to guarantee stability).

Original: More importantly, all these differentiable search approaches must be terminated in an early stage, which makes them less convincing as search has not arrived at convergence.
Revised: More importantly, all these differentiable search approaches must be terminated in an early stage, which makes them less convincing as search has not arrived at convergence.

Original: These
Revised: These

Original: [Table] [Table]
Revised: [Table] [Table]

Original: tricks somewhat violate the ideology of neural architecture search; in comparison, our research, though not producing the best performance, seems going along a correct and promising direction.
Revised: tricks somewhat violate the ideology of neural architecture search; in comparison, our research, though not producing the best performance, seems going along a correct and promising direction.

Original: 4. R ESULTS ON ILSVRC 
Revised: 4. R ESULTS ON ILSVRC

Original: ILSVRC2012 (Russakovsky et al., 2015) is the most commonly used subset of ImageNet (Deng et al., 2009).
Revised: ILSVRC2012 (Russakovsky et al., 2015) is the most commonly used subset of ImageNet (Deng et al., 2009).

Original: It contains 1 .
Revised: It contains 1 .

Original: 3M training images and 50K testing images, which are almost evenly distributed over all 1 , 000 categories.
Revised: 3M training images and 50K testing images, which are almost evenly distributed over all 1 , 000 categories.

Original: We directly use the S 2 architecture obtained from CIFAR experiments and enlarge it with a basic number of channels of 42 , so that the FLOPs of our model is 590M , i.e. , ﬁtting the mobile setting.
Revised: We directly use the S 2 architecture obtained from CIFAR experiments and enlarge it with a basic number of channels of 42 , so that the FLOPs of our model is 590M , i.e. , ﬁtting the mobile setting.

Original: During re-training, there are a total of 250 epochs.
Revised: During re-training, there are a total of 250 epochs.

Original: We use an
Revised: We use an

Original: SGD optimizer with an initial learning rate of 0 .
Revised: SGD optimizer with an initial learning rate of 0 .

Original: 5 (decaying linearly after each epoch), a momentum of 0 .
Revised: 5 (decaying linearly after each epoch), a momentum of 0 .

Original: 9 and a weight decay of 3 × 10 − 5 .
Revised: 9 and a weight decay of 3 × 10 − 5 .

Original: On NVIDIA Tesla V100 GPUs, the entire re-training process takes around 24 GPU-days.
Revised: On NVIDIA Tesla V100 GPUs, the entire re-training process takes around 24 GPU-days.

Original: The comparison of our approach to existing approaches is shown in Table 2.
Revised: The comparison of our approach to existing approaches is shown in Table 2.

Original: Our approach achieves a top- 1 error rate of 24 .
Revised: Our approach achieves a top- 1 error rate of 24 .

Original: 3% without any common optimization tricks such as AutoAugment (Cubuk et al., 2019) and Squeeze-and-Excitation modules (Hu et al., 2018).
Revised: 3% without any common optimization tricks such as AutoAugment (Cubuk et al., 2019) and Squeeze-and-Excitation modules (Hu et al., 2018).

Original: This result is competitive among state-of-the-arts, and it is obtained after convergence is achieved in the search stage.
Revised: This result is competitive among state-of-the-arts, and it is obtained after convergence is achieved in the search stage.

Original: On the other hand, without the amending term, DARTS converges to a solution that skip-connected and sep-conv3x3 are largely separated, on which the re-training performance is even inferior to random search.
Revised: On the other hand, without the amending term, DARTS converges to a solution that skip-connected and sep-conv3x3 are largely separated, on which the re-training performance is even inferior to random search.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we present an effective approach for stabilizing differentiable neural architecture search.
Revised: In this paper, we present an effective approach for stabilizing differentiable neural architecture search.

Original: Our motivation comes from that DARTS-based approaches mostly generate all- skip-connect architectures when they are executed for a sufﬁcient number of epochs.
Revised: Our motivation comes from that DARTS-based approaches mostly generate all- skip-connect architectures when they are executed for a sufﬁcient number of epochs.

Original: We analyze this weird phenomenon mathematically and ﬁnd the reason to lie in the dramatic inaccuracy in estimating gradients of architectural parameters.
Revised: We analyze this weird phenomenon mathematically and ﬁnd the reason to lie in the dramatic inaccuracy in estimating gradients of architectural parameters.

Original: With an alternative approximation based on the optimality of network parameters, we guarantee the update of architectural parameters to be in a correct direction.
Revised: With an alternative approximation based on the optimality of network parameters, we guarantee the update of architectural parameters to be in a correct direction.

Original: In
Revised: In

Original: DARTS-based search spaces on CIFAR10 and ImageNet, our approach shows improved stability, in particular in large search spaces, as well as improved performance in the re-training stage.
Revised: DARTS-based search spaces on CIFAR10 and ImageNet, our approach shows improved stability, in particular in large search spaces, as well as improved performance in the re-training stage.

Original: Our research sheds light on future research on NAS in several aspects.
Revised: Our research sheds light on future research on NAS in several aspects.

Original: First, we reveal that previous differentiable approaches were mostly built upon a dangerous pipeline, and mostly introduced heavy human expertise to avoid failure.
Revised: First, we reveal that previous differentiable approaches were mostly built upon a dangerous pipeline, and mostly introduced heavy human expertise to avoid failure.

Original: By ﬁxing the ‘system error’ of this pipeline, we provide a platform that NAS approaches can compete in the ability of NAS.
Revised: By ﬁxing the ‘system error’ of this pipeline, we provide a platform that NAS approaches can compete in the ability of NAS.

Original: Second, our approach enables researchers to explore even bigger search spaces that have not been studied before (due to search instability).
Revised: Second, our approach enables researchers to explore even bigger search spaces that have not been studied before (due to search instability).

Revised: A A PPENDIX A. A TOY EXAMPLE TO SHOW THE IMPORTANCE OF g

Revised: Let the loss function be L ( ω , α ; x ) =

Revised: ( ωx − α ) 2 .

Revised: Then, the only difference between

Revised: L train ( ω , α ) =

Revised: L ( ω , α ; x train ) and L valid ( ω , α )

Revised: =

Revised: L ( ω , α ; x valid ) lies in the input, x .

Revised: Assume the input of training data is x train

Revised: = 1 and

Revised: the input validation data is x valid = 2 .

Revised: It is easy to derive that the local optimum of L train ( ω , α ) is ω ∗ ( α )

Revised: =

Revised: α .

Revised: Substituting x valid =into L valid ( ω , α ) yields L valid ( ω , α ) =

Revised: (2 ω − α ) 2 .

Revised: When α = α t , ω arrives at ω ∗ ( α t ) , so g 1 = 2 ( α t − 2 α t )

Revised: = − 2 α t and g 2 = 4 α t .

Revised: When ω arrives at ω ∗ ( α ) , L valid ( ω ∗ ( α ) , α )

Revised: =

Original: {+α 2 , g ( α t ) = 
+}
Revised: {+α 2 , g ( α t ) = 
+}

Revised: α t =

Revised: g 1 + g 2 .

Revised: In summary, both g 1 and g 2 are important, but DARTS chose to ignore g 2 which can cause a dramatic error in approximation.

Revised: A. T HE COMPLEXITY OF g (cid:48) 2 CAN BE SUBSTANTIALLY REDUCED USING THE FINITE DIFFERENCE APPROXIMATION

Revised: We use the ﬁnite difference approximation just like DARTS.

Revised: Let (cid:15) be a small scalar, ω 1 = ω + (cid:15) ∇ ω L val ( ω , α ) , ω

Revised: 2 = ω − (cid:15) ∇ ω L val ( ω , α ) .

Revised: Then: H · ∇ ω L val ( ω , α ) =

Revised: [Table]

Revised: ω= ω

Revised: [Equation]

Revised: [Table]

Revised: −∇

Revised: [Table]

Revised: , ω

Revised: [Table]

Revised: = ω −

Revised: [Table]

Revised: Then: g (cid:48)

Revised: [Equation]

Revised: A. A TOY EXAMPLE TO SHOW THE IMPORTANCE OF THE “ GRADIENT TRAP ”

Revised: We have a small toy case to show the inﬂuence of the “gradient trap”.

Revised: We searched for a small super-network in the DARTS’s search space, which only has two cells(we searched for 600 epochs).

Revised: When we train the super-network in the “training sets in search phase” and validate in the “validation sets in search phase”, the test error using g (cid:48) 2 is 10 .

Revised: 5% while the test error without g (cid:48) 2 is 12 .

Revised: When we train the super-network in the training sets and validate in the validation sets, the test error using g (cid:48) 2 is 5 .

Revised: 4% while the test error without g (cid:48) 2 is 7 . 4% .

Revised: When we generalize and train the network in the training sets and validate in the validation sets, the test error using g (cid:48) 2 is 6 .

Revised: 2% while the test error without g (cid:48) 2 is 7 . 4% .

Revised: In this case the “gradient trap” will cause a dramatic accuracy drop of the super-network.

Revised: A. S EARCH WITH DIFFERENT SEEDS

Revised: We ran our search algorithms with different seeds for 5 times in S 2 and evaluated each discovered architecture for 3 times.

Revised: The lowest test error is 2 .

Revised: 57 ± 0 .

Revised: 11% and the highest is 2 .

Revised: 63 ± 0 .

Revised: We did the same thing on S 1 and the lowest and the highest test errors are 2 .

Revised: 71 ± 0 .

Revised: 15% and 2 .

Revised: 92 ± 0 .

Revised: 09% , respectively.

Revised: As we expected, the results in S 1 are less robust than those in S 2 .

Revised: The main reason is the difference between search and evaluation, including the different depths of search and evaluation networks and the the discretization stage of the standard DARTS method.

Revised: More importantly, our approach survives after 500 (and even more) epochs, while DARTS degenerates to an all-skip-connect architecture in all (10+) individual runs.

Revised: A. T HEORETICAL

Revised: TRANSFORMATION

Revised: A = C T · C , A is a semi-positive-deﬁnite matrix.

Revised: In this case, the number of different eigenvalues is far less than the dimension of A (hundreds compared to millions).

Revised: H is a real symmetric positivedeﬁnite matrix.

Revised: Let { α i } be a set of eigenvectors w.r.t A ( α T i · α i = 1) , then { H · α i } is a set of eigenvectors w.r.t

Revised: H · A · H − 1 , { H − 1 · α i } is a set of eigenvectors w.r.t H − 1 · A · H , and { λ i } is a set of eigenvectors shared by them.

Revised: Let β be an eigenvector of H · A · H − 1 + H − 1 · A · H , β = (cid:80) ni

Revised: [Equation]

Revised: H

Revised: [Equation]

Revised: α i =

Revised: [Equation]

Revised: ni

Revised: [Equation]

Revised: b i × H − 1 · α i .

Revised: [Equation]

Revised: A is a real symmetric matrix, so we have many sets of { α i } that is orthogonal to each other.

Original: {+β T · β = (cid:80) ni=
+}
Revised: {+β T · β = (cid:80) ni=
+}

Revised: a i b i ≥ 0 , For every eigenvalue, the dimension of the subspace will be very high in the case of neural architecture search, so we assume that we can choose a set of { α i } orthogonal to each other from the subspace satisfying (cid:80) n i+1 − 1 j=n i a j b j ≥ 0 in most cases( n i is the id of the ﬁrst eigenvector w.r.t λ i ).

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: 0 , so λ cannot be smaller than zero.

Revised: All of the eigenvalues w.r.t real symmetric matrix H · A · H − 1 + H − 1 · A · H is not smaller than zero, so it is semi-positive-deﬁnite.

Revised: Then we get that H − 1 · A · H is semi-positive-deﬁnite.