Original: Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations
Revised: Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations

Original: Abstract
Revised: Abstract

Original: The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity.
Revised: The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity.

Original: DARTSleverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handledwith gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes.
Revised: DARTS leverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handled with gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes.

Original: The vanilla DARTS assumes the optimized magnitudes reflect the importanceof operations, while more recent works find this naive assumption leads to poorgeneralization and is without any theoretical guarantees.
Revised: The vanilla DARTS assumes the optimized magnitudes reflect the importance of operations, while more recent works find this naive assumption leads to poor generalization and is without any theoretical guarantees.

Original: In this work, we leverageinfluence functions, the functional derivatives of the loss function, to theoreticallyreveal the operation selection part in DARTS and estimate the candidate operationimportance by approximating its influence on the supernet with Taylor expansions.
Revised: In this work, we leverage influence functions, the functional derivatives of the loss function, to theoretically reveal the operation selection part in DARTS and estimate the candidate operation importance by approximating its influence on the supernet with Taylor expansions.

Original: We show the operation strength is not only related to the magnitude but also secondorder information, leading to a fundamentally new criterion for operation selectionin DARTS, named Influential Magnitude .
Revised: We show the operation strength is not only related to the magnitude but also secondorder information, leading to a fundamentally new criterion for operation selection in DARTS, named Influential Magnitude .

Original: Empirical studies across different taskson several spaces show that vanilla DARTS and its variants can avoid most failuresby leveraging the proposed theory-driven operation selection criterion.
Revised: Empirical studies across different tasks on several spaces show that vanilla DARTS and its variants can avoid most failures by leveraging the proposed theory-driven operation selection criterion.

Original:  Introduction
Revised:  Introduction

Original: Neural Architecture Search (NAS) has been successfully applied to automating the process of neuralnetwork design for a broad of deep learning fields [35].
Revised: Neural Architecture Search (NAS) has been successfully applied to automating the process of neural network design for a broad of deep learning fields [35].

Original: However, the early NAS methods areoften heavily computational-expensive and require hundreds or even thousands GPU days [34].
Revised: However, the early NAS methods are often heavily computational-expensive and require hundreds or even thousands GPU days [34].

Original: Toimprove the efficiency for more practical applications, a lot of recent works are proposed based onone-shot (also known as weight-sharing) paradigm [33] to improve the search efficiency significantly.
Revised: To improve the efficiency for more practical applications, a lot of recent works are proposed based on one-shot (also known as weight-sharing) paradigm [33] to improve the search efficiency significantly.

Original: Differentiable ARchiTecture Search (DARTS) [27], as the most popular one-shot NAS method, hasdominated this area since its appearance.
Revised: Differentiable ARchiTecture Search (DARTS) [27], as the most popular one-shot NAS method has dominated this area since its appearance.

Original: DARTS leverages continuous relaxation to convert theintractable operation selection problem into a continuous magnitude optimization problem, whichcan be gracefully solved by gradient-descent based on the bi-level optimization [8].
Revised: DARTS leverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem, which can be gracefully solved by gradient-descent based on the bi-level optimization [8].

Original: A discretearchitecture is then derived based on the optimized magnitudes through operation selection.
Revised: A discrete architecture is then derived based on the optimized magnitudes through operation selection.

Original: Although the continuous relaxation makes DARTS able to optimize the magnitudes with rigoroustheoretical formulation and guarantees [49] in the supernet training part, it poses an additionalchallenge in selecting a discrete architecture from the optimized magnitudes that there is no existingwork can give a theoretical explanation.
Revised: Although the continuous relaxation makes DARTS able to optimize the magnitudes with rigorous theoretical formulation and guarantees [49] in the supernet training part, it poses an additional challenge in selecting a discrete architecture from the optimized magnitudes that there is no existing work can give a theoretical explanation.

Original: DARTS is based on the weight-sharing paradigm [33],which assumes that the derived discrete architecture with the trained supernet weights can predictits train-from-scratch performance.
Revised: DARTS is based on the weight-sharing paradigm [33], which assumes that the derived discrete architecture with the trained supernet weights can predict its train-from-scratch performance.

Original: However, there is a gap between the two groups of weights,as depicted in Figure 1 in Appendix.
Revised: However, there is a gap between the two groups of weights, [Figure] as depicted in Figure 1 in Appendix.

Original: The vanilla DARTS assumes the optimized magnitudes can[Figure] reflect the importance of the candidate operations, while Wang et al.
Revised: The vanilla DARTS assumes the optimized magnitudes can reflect the importance of the candidate operations, while Wang et al.

Original: [40] empirically show thatthe operation magnitude does not necessarily indicate how important the operation is in a trainedsupernet.
Revised: [40] empirically show that the operation magnitude does not necessarily indicate how important the operation is in a trained supernet.

Original: For example, DARTS prefers skip-connection [45] as its magnitude always dominatesthe remaining operations, while an architecture with intensive skip-connections generally leads topoor performance.
Revised: For example, DARTS prefers skip-connection [45] as its magnitude always dominates the remaining operations, while an architecture with intensive skip-connections generally leads to poor performance.

Original: Accordingly, Wang et al.
Revised: Accordingly, Wang et al.

Original: [40] proposed a heuristic perturbation-based method,
Revised: [40] proposed a heuristic perturbation-based method,

Original: DARTS-PT, to measure each candidate operation’s influence on the supernet by alternatively removingeach one to calculate the supernet performance drop.
Revised: DARTS-PT, to measure each candidate operation’s influence on the supernet by alternatively removing each one to calculate the supernet performance drop.

Original: Despite DARTS-PT can partially relieve thefailure of DARTS, we empirically found that DARTS-PT still prefers skip-connection than otheroperations.
Revised: Despite DARTS-PT can partially relieve the failure of DARTS, we empirically found that DARTS-PT still prefers skip-connection than other operations.

Original: One potential reason is that skip-connection ’s magnitude is much higher than othercandidate operations, and directly removing skip-connections usually greatly deteriorates the supernetperformance.
Revised: One potential reason is that skip-connection ’s magnitude is much higher than other candidate operations, and directly removing skip-connections usually greatly deteriorates the supernet performance.

Original: In addition, DARTS-PT lacks theoretical explanations and guarantees.
Revised: In addition, DARTS-PT lacks theoretical explanations and guarantees.

Original: For example,the intuitive way to show the importance of a candidate operation importance is the leave-one-outretraining [22], while which should fine-tune the supernet to get the performance drop after removingevery operation rather than directly measuring the performance drop as DARTS-PT, since there alsoexists a gap between the two groups of weights, as visualized in Figure 1.
Revised: For example, the intuitive way to show the importance of a candidate operation importance is the leave-one-out retraining [22], while which should fine-tune the supernet to get the performance drop after removing every operation rather than directly measuring the performance drop as DARTS-PT, since there also exists a gap between the two groups of weights, as visualized in Figure 1.

Original: Section 3 and Appendix Calso give an analysis on why perturbation-based method fails.
Revised: Section 3 and Appendix C also give an analysis on why perturbation-based method fails.

Original: This paper focuses on filling the gap of theoretical explanations for the discrete operation selection partin DARTS.
Revised: This paper focuses on filling the gap of theoretical explanations for the discrete operation selection part in DARTS.

Original: Instead of considering a heuristic approach to measure the operation importance [27, 40],this paper first rigorously shows how the architecture parameters discretization poses an influenceon the supernet weights in DARTS, and estimates the supernet performance drop accordingly.
Revised: Instead of considering a heuristic approach to measure the operation importance [27, 40], this paper first rigorously shows how the architecture parameters discretization poses an influence on the supernet weights in DARTS, and estimates the supernet performance drop accordingly.

Original: Ourmethod is motivated by influence function [22], which is originally designed to understand the effectof removing an individual training point, while we leverage Taylor’s approximation to interpret thesupernet performance drop when removing a candidate operation.
Revised: Our method is motivated by influence function [22], which is originally designed to understand the effect of removing an individual training point, while we leverage Taylor’s approximation to interpret the supernet performance drop when removing a candidate operation.

Original: In this way, we show the operationstrength is not only related to the magnitude but also second-order information, and we accordinglydesign a new criterion for the operation selection in DARTS, called Influential Magnitude ( I M ).
Revised: In this way, we show the operation strength is not only related to the magnitude but also second-order information, and we accordingly design a new criterion for the operation selection in DARTS, called Influential Magnitude ( I M ).

Original: Then, to avoid calculating the inverse of the Hessian matrix with respect to the model weights,we further utilize the Neumann series [29] and Sherman-Morrison formula [37] to approximatethe Inverse-Hessian-Vector Products (IHVPs), making the proposed method applicable to DARTS.
Revised: Then, to avoid calculating the inverse of the Hessian matrix with respect to the model weights, we further utilize the Neumann series [29] and Sherman-Morrison formula [37] to approximate the Inverse-Hessian-Vector Products (IHVPs), making the proposed method applicable to DARTS.

Original: Finally, we verify the effectiveness of our method on several NAS benchmark datasets and thecommon DARTS search space.
Revised: Finally, we verify the effectiveness of our method on several NAS benchmark datasets and the common DARTS search space.

Original: To summarize, we make the following contributions:
Revised: To summarize, we make the following contributions:

Original: • This paper deepens our understanding of the operation selection in DARTS.
Revised: • This paper deepens our understanding of the operation selection in DARTS.

Original: We reformulate the operation selection in DARTS by approximating its influence on the supernet with Taylorexpansions, which can thus gracefully interpret how the validation performance changes when selecting different operations without any additional fine-tuning.
Revised: We reformulate the operation selection in DARTS by approximating its influence on the supernet with Taylor expansions, which can thus gracefully interpret how the validation performance changes when selecting different operations without any additional fine-tuning.

Original: • We theoretically reveal the operation strength is not only related to the magnitude but also thesecond-order information, and accordingly derive a fundamentally new criterion to measure
Revised: • We theoretically reveal the operation strength is not only related to the magnitude but also the second-order information, and accordingly derive a fundamentally new criterion to measure

Original: the operation sensitivity, which we call Influential Magnitude ( I M ), for architecture selection in DARTS.
Revised: the operation sensitivity, which we call Influential Magnitude ( I M ), for architecture selection in DARTS.

Original: Further, to make the proposed criterion practical, we devise severalmethods to estimate the Inverse-Hessian-Vector Products (IHVPs) in calculating the secondorder information.• Extensive experiments verify the effectiveness of the proposed criterion, which significantlyimproves the performance of vanilla DARTS and other baselines on the NAS benchmarkdatasets and the common DARTS search space.
Revised: Further, to make the proposed criterion practical, we devise several methods to estimate the Inverse-Hessian-Vector Products (IHVPs) in calculating the secondorder information. • Extensive experiments verify the effectiveness of the proposed criterion, which significantly improves the performance of vanilla DARTS and other baselines on the NAS benchmark datasets and the common DARTS search space.

Original: We show DARTS is still a strong NASbaseline when considering a more reasonable and theory-driven operation strength metric.
Revised: We show DARTS is still a strong NAS baseline when considering a more reasonable and theory-driven operation strength metric.

Original:  Preliminaries: DARTS and Influence Function
Revised:  Preliminaries: DARTS and Influence Function

Original: Typically, NAS focuses on finding a cell structure that is represented as a directed acyclic graph(DAG) with N nodes and E edges affiliated with operations, which is a discrete and hard-to-optimizeproblem.
Revised: Typically, NAS focuses on finding a cell structure that is represented as a directed acyclic graph (DAG) with N nodes and E edges affiliated with operations, which is a discrete and hard-to-optimize problem.

Original: To enable the gradient-descent for architecture search, DARTS [27] leverages softmax asthe continuous relaxation function to transform the architecture search into a continuous magnitudeoptimization problem:
Revised: To enable the gradient-descent for architecture search, DARTS [27] leverages softmax as the continuous relaxation function to transform the architecture search into a continuous magnitude optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where O contains all candidate operations and DARTS is to optimize the magnitude α o , which canbe formulated as a bi-level optimization problem [8]:
Revised: where O contains all candidate operations and DARTS is to optimize the magnitude α o , which can be formulated as a bi-level optimization problem [8]:

Original: [Equation]
Revised: [Equation]

Original: where α is the continuous architecture representation and θ is the supernet weights.
Revised: where α is the continuous architecture representation and θ is the supernet weights.

Original: The nestedformulation in DARTS is the same as the gradient-based hyperparameter optimization with bi-leveloptimization, which has been validated in a broad applications [13, 31, 32].
Revised: The nested formulation in DARTS is the same as the gradient-based hyperparameter optimization with bi-level optimization, which has been validated in a broad applications [13, 31, 32].

Original: After the bi-leveloptimization, DARTS usually considers heuristic methods to derive the final architecture, where themost popular one is to select the operations with the highest magnitudes, ˆ α = argmax ( α ) .
Revised: After the bi-level optimization, DARTS usually considers heuristic methods to derive the final architecture, where the most popular one is to select the operations with the highest magnitudes, ˆ α = argmax ( α ) .

Original: Despiteits simplicity, more recent works [7, 40, 45, 47, 48] found that DARTS could hardly find satisfactoryarchitectures.
Revised: Despite its simplicity, more recent works [7, 40, 45, 47, 48] found that DARTS could hardly find satisfactory architectures.

Original: Different from the majority of existing works that attribute the failure of DARTS to its magnitudeoptimization part, which is under a rigorous theoretical formulation [49], we revisit DARTS fromthe perspective of the architecture selection and try to reveal this part with theoretical explanations,where we aim to identify the operation that will contribute most to the supernet performance, throughthe lens of influence functions [22] to mathematically model the architecture selection process insteadof heuristics.
Revised: Different from the majority of existing works that attribute the failure of DARTS to its magnitude optimization part, which is under a rigorous theoretical formulation [49], we revisit DARTS from the perspective of the architecture selection and try to reveal this part with theoretical explanations, where we aim to identify the operation that will contribute most to the supernet performance, through the lens of influence functions [22] to mathematically model the architecture selection process instead of heuristics.

Original: The influence function [18] is a classic technique from robust statistics that reveals how modelparameters change as we upweight or perturb a specific training sample, which has been applied inexplaining a lot of modern machine learning applications [2, 22, 30, 50].
Revised: The influence function [18] is a classic technique from robust statistics that reveals how model parameters change as we upweight or perturb a specific training sample, which has been applied in explaining a lot of modern machine learning applications [2, 22, 30, 50].

Original: More specifically, given amodel with parameters θ ∗ after trained on a full training set D , influence function aims to study thechange of parameters ˆ θ ∗ − θ ∗ when retraining the model parameters to ˆ θ ∗ after deleting a specifictraining point z , that ˆ D
Revised: More specifically, given a model with parameters θ ∗ after trained on a full training set D , influence function aims to study the change of parameters ˆ θ ∗ − θ ∗ when retraining the model parameters to ˆ θ ∗ after deleting a specific training point z , that ˆ D

Original: = D − z .
Revised: = D − z .

Original: Rather than retraining the model for every removed point, Koh and
Revised: Rather than retraining the model for every removed point, Koh and

Original: Liang [22] estimated the influence by upweighting the data point with some small ϵ .
Revised: Liang [22] estimated the influence by upweighting the data point with some small ϵ .

Original: With leveragingthe Taylor expansions, the influence on the model parameters can be estimated as:
Revised: With leveraging the Taylor expansions, the influence on the model parameters can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: (2) and its influence on the loss function of a test point z test is accordingly calculated as:
Revised: and its influence on the loss function of a test point z test is accordingly calculated as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The detailed derivation of the two influence functions can be found in [22].
Revised: The detailed derivation of the two influence functions can be found in [22].

Original: Based on Eq.(2) and (3),we can now approximate the validation performance drop when removing a training point z from thetraining set without conducting the time-consuming retraining.
Revised: Based on Eq.(2) and (3), we can now approximate the validation performance drop when removing a training point z from the training set without conducting the time-consuming retraining.

Original: The original work of influence functions for modern machine learning [22], which is also calledas first-order influence functions, only considers a very small change on θ ∗ as it only changes one training point, so the first-order Taylor expansion is satisfied to calculate the influence.
Revised: The original work of influence functions for modern machine learning [22], which is also called as first-order influence functions, only considers a very small change on θ ∗ as it only changes one training point, so the first-order Taylor expansion is satisfied to calculate the influence.

Original: However,upweighting a group of data points may lead to a large change on θ ∗ [4, 5], which violates the smallperturbation assumption of the first-order influence functions.
Revised: However, upweighting a group of data points may lead to a large change on θ ∗ [4, 5], which violates the small perturbation assumption of the first-order influence functions.

Original: To make the the influence functioncapture the model parameter change more precisely, Basu et al.
Revised: To make the the influence function capture the model parameter change more precisely, Basu et al.

Original: [4] extended influence functionsusing second-order approximations under the assumption that the third-order derivatives of the lossfunction at optimum is small.
Revised: [4] extended influence functions using second-order approximations under the assumption that the third-order derivatives of the loss function at optimum is small.

Original: For the quadratic loss, the third-order derivatives of the loss are zero orvery small, and Basu et al.
Revised: For the quadratic loss, the third-order derivatives of the loss are zero or very small, and Basu et al.

Original: [4] empirically verified that the this assumption also approximately heldfor the classification problem with cross-entropy loss function.
Revised: [4] empirically verified that the this assumption also approximately held for the classification problem with cross-entropy loss function.

Original: Different from existing works that analyze the effects on model parameters of removing data points,this paper creatively adapts influence functions to estimate the importance of candidate operationson a trained supernet in differentiable architecture search.
Revised: Different from existing works that analyze the effects on model parameters of removing data points, this paper creatively adapts influence functions to estimate the importance of candidate operations on a trained supernet in differentiable architecture search.

Original: We also follow the assumption in [4] thatneglect the third-order and higher derivatives of the loss, and analyze the operation selection part,which is an additional process after bi-level supernet training in DARTS, through the lens of influencefunction in the following sections.
Revised: We also follow the assumption in [4] that neglect the third-order and higher derivatives of the loss, and analyze the operation selection part, which is an additional process after bi-level supernet training in DARTS, through the lens of influence function in the following sections.

Original:  Interpret Operation Selection through Influence Functions
Revised:  Interpret Operation Selection through Influence Functions

Original: This section aims to understand how the model parameters θ (a.k.a.
Revised: This section aims to understand how the model parameters θ (a.k.a.

Original: the supernet weights in DARTS)would change when we derive the discrete architecture from the optimized continuous operationmagnitudes, a.k.a.
Revised: the supernet weights in DARTS) would change when we derive the discrete architecture from the optimized continuous operation magnitudes, a.k.a.

Original: architecture selection.
Revised: architecture selection.

Original: In this paper, our analysis only focuses on this architectureselection part after the bi-level supernet training process is finished.
Revised: In this paper, our analysis only focuses on this architecture selection part after the bi-level supernet training process is finished.

Original: To be more clearly, we firstdefine the optimized operation magnitudes by DARTS as α with the optimized supernet weights θ ∗ ,and the validation loss is L ( θ ∗ , α ) for simplicity.
Revised: To be more clearly, we first define the optimized operation magnitudes by DARTS as α with the optimized supernet weights θ ∗ , and the validation loss is L ( θ ∗ , α ) for simplicity.

Original: Then, we define the architecture parameters changeto ˆ α after discretization, and the corresponding ground-truth supernet weights and validation lossare defined as ˆ θ ∗ and L (ˆ θ ∗ , ˆ α ) , respectively.
Revised: Then, we define the architecture parameters change to ˆ α after discretization, and the corresponding ground-truth supernet weights and validation loss are defined as ˆ θ ∗ and L (ˆ θ ∗ , ˆ α ) , respectively.

Original: Rather than deleting a single data point that only bringssmall changes on the model parameters [22], the change of architecture parameters can lead to aconsiderable change on θ , so we leverage the second-order approximation [4] to reveal the supernetweights change.
Revised: Rather than deleting a single data point that only brings small changes on the model parameters [22], the change of architecture parameters can lead to a considerable change on θ , so we leverage the second-order approximation [4] to reveal the supernet weights change.

Original: With second-order Taylor expansion on ˆ θ ∗ for L (ˆ θ ∗ , ˆ α ) , we have:
Revised: With second-order Taylor expansion on ˆ θ ∗ for L (ˆ θ ∗ , ˆ α ) , we have:

Original: [Equation]
Revised: [Equation]

Original: where ∆ θ = ˆ θ ∗ − θ ∗ , and L is the validation loss function in Eq.(1).
Revised: where ∆ θ = ˆ θ ∗ − θ ∗ , and L is the validation loss function in Eq.(1).

Original: For example, when weonly remove one candidate operation as DARTS-PT [40] to measure the operation importance, theinfluence of this discretization brings to θ is ∆ θ , and the true validation loss change should be
Revised: For example, when we only remove one candidate operation as DARTS-PT [40] to measure the operation importance, the influence of this discretization brings to θ is ∆ θ , and the true validation loss change should be

Original: L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) as stated in Eq.(4) rather than L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) used by DARTS-PT.
Revised: L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) as stated in Eq.(4) rather than L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) used by DARTS-PT.

Original: Asshown in Eq.(4), we need to calculate ∆ θ to get ∆ L after we change the α to ˆ α , while we try to avoidthe time-consuming retraining process through leveraging influence functions [22] to estimating ˆ θ ∗ .
Revised: As shown in Eq.(4), we need to calculate ∆ θ to get ∆ L after we change the α to ˆ α , while we try to avoid the time-consuming retraining process through leveraging influence functions [22] to estimating ˆ θ ∗ .

Original: Based on the implicit function theorem [29, 49], the validation loss change after discretizing α to ˆ αcan be approximated as the following theorem.
Revised: Based on the implicit function theorem [29, 49], the validation loss change after discretizing α to ˆ α can be approximated as the following theorem.

Original: Theorem 1 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, α changes to ˆ α when conducting architecture discretization, andthe train-from-scratch validation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .
Revised: Theorem 1 Suppose that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ after supernet training, α changes to ˆ α when conducting architecture discretization, and the train-from-scratch validation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .

Original: If the third and higher derivatives of the lossfunction L at optimum is zero or sufficiently small [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have
Revised: If the third and higher derivatives of the loss function L at optimum is zero or sufficiently small [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 states that we can estimate the train-from-scratch validation performance of ˆ α without anyretraining.
Revised: Theorem 1 states that we can estimate the train-from-scratch validation performance of ˆ α without any retraining.

Original: During the discretization process, we can apply the argmax on all edges by one-shot thatˆ α = argmax ( α ) , or only discretize one edge in each step, or remove one candidate operation fromone edge as DARTS-PT [40] which can be formulated as the first part of Eq.(5), L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: During the discretization process, we can apply the argmax on all edges by one-shot that ˆ α = argmax ( α ) , or only discretize one edge in each step, or remove one candidate operation from one edge as DARTS-PT [40] which can be formulated as the first part of Eq.(5), L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: We prefer an iterative discretization process that only applies a small change on α as DARTS-PT,since our theoretical analysis in the next corollary will show that the error bound of our approximationis related to ∆ α = ˆ α − α .
Revised: We prefer an iterative discretization process that only applies a small change on α as DARTS-PT, since our theoretical analysis in the next corollary will show that the error bound of our approximation is related to ∆ α = ˆ α − α .

Original: Before our analysis, we give the following common assumptions, whichare also considered in several recent bi-level optimization problems [9, 15–17, 49].
Revised: Before our analysis, we give the following common assumptions, which are also considered in several recent bi-level optimization problems [9, 15–17, 49].

Original: Assumption 1 For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with C f > 0 and
Revised: Assumption 1 For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with C f > 0 and

Original: C L > 0 , respectively.
Revised: C L > 0 , respectively.

Original: Assumption 2 L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θaround θ ∗ ( α ) .
Revised: Assumption 2 L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θ around θ ∗ ( α ) .

Original: Assumption 3 (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .
Revised: Assumption 3 (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .

Original: Corollary 1 Based on the Assumption 1-3, we could bound the error between the approximatedvalidation loss 
Revised: Corollary 1 Based on the Assumption 1-3, we could bound the error between the approximated validation loss L (ˆ θ ∗ , ˆ α ) =

Original: L (ˆ θ ∗ , ˆ α ) = ∆ L + L ( θ ∗ , α ) and the ground-truth ˜ L (ˆ θ ∗ , ˆ α ) in DARTS with E =(cid:13)(cid:13)(cid:13)
Revised: ∆ L + L ( θ ∗ , α ) and the ground-truth ˜ L (ˆ θ ∗ , ˆ α ) in DARTS with E = (cid:13)(cid:13)(cid:13) L (ˆ θ ∗ , ˆ α )

Original: L (ˆ θ ∗ , ˆ α ) − ˜ L (ˆ θ ∗ , ˆ α ) (cid:13)(cid:13)(cid:13) ⩽ K 3 6 max (cid:12)(cid:12)(cid:12) ∂
Revised: − ˜ L (ˆ θ ∗ , ˆ α ) (cid:13)(cid:13)(cid:13) ⩽ K 3 6 max (cid:12)(cid:12)(cid:12) ∂

Original: L 3 ∂θ 3 (cid:12)(cid:12)(cid:12) , where K = C L λ ∥
Revised: L 3 ∂θ 3 (cid:12)(cid:12)(cid:12) , where K = C L λ ∥

Original: ∆ α ∥ + C H C 2 a 2 σ 2 min λ ∥ ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .
Revised: ∆ α ∥ + C H C 2 a 2 σ 2 min λ ∥ ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .

Original: Compared with the argmax used in DARTS, we prefer the perturbation-based discretization method[41] as which only remove one operation at each step, since the above Corollary 1 show that errorbound of the approximated validation loss
Revised: Compared with the argmax used in DARTS, we prefer the perturbation-based discretization method [41] as which only remove one operation at each step, since the above Corollary 1 show that error bound of the approximated validation loss

Original: L (ˆ θ ∗ , ˆ α ) increase with the magnitude of the change of α .
Revised: L (ˆ θ ∗ , ˆ α ) increase with the magnitude of the change of α .

Original:  Influential Magnitude for Operation Selection in DARTS
Revised:  Influential Magnitude for Operation Selection in DARTS

Original: Section 3 provides an iterative solution to measure the operation importance based Eq.
Revised: Section 3 provides an iterative solution to measure the operation importance based Eq.

Original: (5) throughindividually removing each operation, while which needs to repeat n times (the number of allcandidate operations for all edges in the supernet work).
Revised: (5) through individually removing each operation, while which needs to repeat n times (the number of all candidate operations for all edges in the supernet work).

Original: More important, since removing oneoperation poses a considerable change on α ( α i → 0 ), the approximation error produced by Eq.(5)is non-negligible which may affects the accuracy as stated by Corollary 1.
Revised: More important, since removing one operation poses a considerable change on α ( α i → 0 ), the approximation error produced by Eq.(5) is non-negligible which may affects the accuracy as stated by Corollary 1.

Original: So, a more practicalsolution to illustrate the importance of each operation is to estimate how the validation performancewill change after posing an infinitesimal change on α ,a.k.a.
Revised: So, a more practical solution to illustrate the importance of each operation is to estimate how the validation performance will change after posing an infinitesimal change on α ,

Original:  operation sensitivity.
Revised: a.k.a. operation sensitivity.

Original: In this section,we aim to investigate the operation importance through the lens of sensitivity [24].
Revised: In this section, we aim to investigate the operation importance through the lens of sensitivity [24].

Original: Different from
Revised: Different from

Original: Section 3 that removes one operation ( α i → 0 ), this section only changes the α i with an infinitesimalε i , which means that we can now conduct first-order Taylor expansion on α for L (ˆ θ ∗ , ˆ α ) .
Revised: Section 3 that removes one operation ( α i → 0 ), this section only changes the α i with an infinitesimal ε i , which means that we can now conduct first-order Taylor expansion on α for L (ˆ θ ∗ , ˆ α ) .

Original: Furtherwith second-order Taylor expansion on θ and assuming the local optimal ∂ L ( θ ∗ ,α ) ∂θ = 0 , we have:
Revised: Further with second-order Taylor expansion on θ and assuming the local optimal ∂ L ( θ ∗ ,α ) ∂θ = 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: Similar to Eq.(4), it is intractable to directly calculate ∆ θ , while we can leverage implicit functiontheorem to approximate it.
Revised: Similar to Eq.(4), it is intractable to directly calculate ∆ θ , while we can leverage implicit function theorem to approximate it.

Original: Theorem 2 presents how the validation performance changes after wepose an infinitesimal change on α .
Revised: Theorem 2 presents how the validation performance changes after we pose an infinitesimal change on α .

Original: Theorem 2 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, and we pose an infinitesimal change on α .
Revised: Theorem 2 Suppose that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ after supernet training, and we pose an infinitesimal change on α .

Original: Based on implicitfunction theorem and under the assumption that the third and higher derivatives of the loss functionat optimum is zero or sufficiently small [4], the change of validation loss can be estimated as:
Revised: Based on implicit function theorem and under the assumption that the third and higher derivatives of the loss function at optimum is zero or sufficiently small [4], the change of validation loss can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.
Revised: where H = ∂ 2 L ( θ ∗ ,α ) [Equation] is the Hessian matrix.

Original: From Theorem 2, we can observe the relationship between ∆ L and ∆ α .
Revised: From Theorem 2, we can observe the relationship between ∆ L and ∆ α .

Original: To be more specifically,when we only consider an infinitesimal change on α , that ∆ α = ε · 1 where 1 is a column vectorwith all ones and ε is an infinitesimal scalar, the sensitivity of α can be defined as:
Revised: To be more specifically, when we only consider an infinitesimal change on α , that ∆ α = ε · 1 where 1 is a column vector with all ones and ε is an infinitesimal scalar, the sensitivity of α can be defined as:

Original: [Equation]
Revised: [Equation]

Original: Accordingly, we propose an Influential Magnitude to quantify the operation importance through thelens of sensitivity:
Revised: Accordingly, we propose an Influential Magnitude to quantify the operation importance through the lens of sensitivity:

Original: Definition 1 Influential Magnitude ( I M ): Suppose DARTS obtains the optimized magnitude αwith supernet weights θ ∗ after supernet training, the operation sensitivity can be defined as I M = 
Revised: Definition 1 Influential Magnitude ( I M ): Suppose DARTS obtains the optimized magnitude α with supernet weights θ ∗ after supernet training, the operation sensitivity can be defined as I M = −

Original: [Equation]
Revised: [Equation]

Original: Definition 1 shows we can replace the magnitude α with the operation sensitivity I M in the operationselection for DARTS, whose i-th value represents the importance of i-th candidate operation.
Revised: Definition 1 shows we can replace the magnitude α with the operation sensitivity I M in the operation selection for DARTS, whose i-th value represents the importance of i-th candidate operation.

Original:  Practical Calculation on Operation Influence
Revised:  Practical Calculation on Operation Influence

Original: For a large neural network, it is impractical to calculate the second-order information, e.g., the
Revised: For a large neural network, it is impractical to calculate the second-order information, e.g., the

Original: Hessian matrix H , let alone the inverse of Hessian.
Revised: Hessian matrix H , let alone the inverse of Hessian.

Original: Generally, the core challenge in calculation of
Revised: Generally, the core challenge in calculation of

Original: Eq.(5) and Eq.(7) is the Inverse-Hessian Vector Products (IHVPs), H − 1 v [14, 22, 37].
Revised: Eq.(5) and Eq.(7) is the Inverse-Hessian Vector Products (IHVPs), H − 1 v [14, 22, 37].

Original: As to the termof ∂ 2 L ( θ ∗ ,α ) ∂θ∂α , we follow DARTS [27] to utilize the finite difference approximation for the practicalcalculation.
Revised: As to the term of ∂ 2 L ( θ ∗ ,α ) ∂θ∂α , we follow DARTS [27] to utilize the finite difference approximation for the practical calculation.

Original: The first technique for IHVPs is the conjugate gradient (CG) which converts the matrixinverse problem into an optimization problem with δ -optimal solution.
Revised: The first technique for IHVPs is the conjugate gradient (CG) which converts the matrix inverse problem into an optimization problem with δ -optimal solution.

Original: However, the standard CGcan be slow with a large dataset as it needs to go through the whole training points in each step [1].
Revised: However, the standard CG can be slow with a large dataset as it needs to go through the whole training points in each step [1].

Original: The original paper [22] using influence function for modern machine learning problem leveragesthe Neumann series to approximate the inverse, that H − 1 = (cid:80) ∞ k =

Revised: The original paper [22] using influence function for modern machine learning problem leverages the Neumann series to approximate the inverse, that H − 1 = (cid:80) ∞ k =

Original: ( I − H ) k , with only considerthe first t terms to approximate H − 1 .
Revised: ( I − H ) k , with only consider the first t terms to approximate H − 1 .

Original: However, there is a strict condition in the Neumann seriesapproximation that ∥ I − H ∥ ≤ 1 , which can not be guaranteed in the practical implementation.
Revised: However, there is a strict condition in the Neumann series approximation that ∥ I − H ∥ ≤ 1 , which can not be guaranteed in the practical implementation.

Original: Differently, we consider that H − 1 = γ ( γH ) − 1 = γ (cid:80) ∞
Revised: Differently, we consider that H − 1 = γ ( γH ) − 1 = γ (cid:80) ∞

Original: k
Revised: k

Original: =

Revised: =

Original: ( I − γH )
Revised: ( I − γH )

Original: k , where γ is a small enoughvalue.
Revised: k , where γ is a small enough value.

Original: The following lemma gives our Neumann series based implementation for IHVPs.
Revised: The following lemma gives our Neumann series based implementation for IHVPs.

Original: Lemma 1 With small enough γ , and assuming L is λ -strongly convex at optimum, H − 1 v can beformulated as: H − 1 v = γ (cid:80) Kk =

Revised: Lemma 1 With small enough γ , and assuming L is λ -strongly convex at optimum, H − 1 v can be formulated as: H − 1 v = γ (cid:80) Kk =

Original: [ I − γH ] k V 

Revised: [ I − γH ] k V

Original: =
Revised: =

Original: V 

Revised: V

Original: + V 1 + ... + V K ,
Revised: + V 1 + ... + V K ,

Original: where H = ∂
Revised: where H = ∂

Original: 2 L ( θ ∗ ,α ) ∂θ∂θ ,
Revised: 2 L ( θ ∗ ,α ) ∂θ∂θ ,

Original: V 

Revised: V

Original: = γv , and V 1 = γ ( I − γH )
Revised: = γv , and V 1 = γ ( I − γH )

Original: V 0 , ..., V K = γ
Revised: V 0 , ..., V K = γ

Original: ( I − γH )
Revised: ( I − γH )

Original: V K − 1 .
Revised: V K − 1 .

Original: In the practical implementation, it is still hard to make sure L is λ -strongly convex and not easyto select an appropriate γ .
Revised: In the practical implementation, it is still hard to make sure L is λ -strongly convex and not easy to select an appropriate γ .

Original: The work [5] empirically showed that the stochastic Neumann seriesestimation for IHVP is somehow erroneous, especially when the network is deep.
Revised: The work [5] empirically showed that the stochastic Neumann series estimation for IHVP is somehow erroneous, especially when the network is deep.

Original: As we know, thetrue Fisher information matrix converges to the Hessian matrix with the training loss approachesto zero.
Revised: As we know, the true Fisher information matrix converges to the Hessian matrix with the training loss approaches to zero.

Original: However, it is usually infeasible to calculate the exact Hessian or Fisher matrix in a largeneural network.
Revised: However, it is usually infeasible to calculate the exact Hessian or Fisher matrix in a large neural network.

Original: A practical approximation is called as empirical Fisher, which has been verified tobe effective in variety of applications [3, 19].
Revised: A practical approximation is called as empirical Fisher, which has been verified to be effective in variety of applications [3, 19].

Original: In addition, an important advantage from empirical
Revised: In addition, an important advantage from empirical

Original: Fisher is that it allows to approximate the inverse of the Hessian based on the Sherman-Morrisonformula.
Revised: Fisher is that it allows to approximate the inverse of the Hessian based on the Sherman-Morrison formula.

Original: Although the empirical Fisher can not exactly match the true Fisher and there are severalactive researches on its applicability [23, 38], empirical Fisher is significantly more computationallyefficient, which obtains a good trade-off between the approximation and practical efficiency [37].
Revised: Although the empirical Fisher can not exactly match the true Fisher and there are several active researches on its applicability [23, 38], empirical Fisher is significantly more computationallyefficient, which obtains a good trade-off between the approximation and practical efficiency [37].

Original: Apart from Neumann series, we also consider Sherman-Morrison formula [14, 37]
Revised: Apart from Neumann series, we also consider Sherman-Morrison formula [14, 37]

Original: ( A + uv T ) − 1 =
Revised: ( A + uv T ) − 1 =

Original: [Equation] 1 u to calculate IHVPs.
Revised: A − 1 − A − 1 uv T A − 1 1+ v T A − 1 u to calculate IHVPs.

Original: With empirical Fisher F = (cid:80) nj =1 ∇ θ L j ( ∇ θ L j ) T replacingtrue Fisher ˆ F = ∇ θ f ( ∇ θ f ) T , IHVPs can be calculated as following.
Revised: With empirical Fisher F = (cid:80) nj =1 ∇ θ L j ( ∇ θ L j ) T replacing true Fisher ˆ F = ∇ θ f ( ∇ θ f ) T , IHVPs can be calculated as following.

Original: Lemma 2 When assume the empirical Fisher can approximate the Fisher matrix, and H is the Hessian matrix 
Revised: Lemma 2 When assume the empirical Fisher can approximate the Fisher matrix, and H is the Hessian matrix ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ in the optimal point, the IHVPs H − 1 v can be formulated as: H − 1 v = F − 1 n v =

Original: [Equation]
Revised: [Equation]

Original: in the optimal point, the IHVPs H − 1 v can be formulated as: H − 1 v = F − 1 nv =
Revised: n

Original: [Equation]
Revised: [Equation]

Original: L jT rj , where L = ℓ + η R ( θ ) that ℓ is a crossentropy loss and R is the regularization term, F n = 1 n (cid:80) nj =1 ∇
Revised: r

Original: [Equation]
Revised: [Equation]

Original:  j ∇ θ L Tj is the empirical Fisher, andr j = F − 1 j − 1 ∇ θ L j which can be recurrently calculated through r j = η − 1 ∇ θ L j − (cid:80) j − 1 i =1 r i r iT ∇ θ L j N + ∇ θ L iT r i .
Revised: jT r j , where L = ℓ + η R ( θ ) that ℓ is a crossentropy loss and R is the regularization term, F n = 1 n (cid:80) nj =1 ∇ θ L j ∇ θ L Tj is the empirical Fisher, and r j = F − 1 j − 1 ∇ θ L j which can be recurrently calculated through [Table]

Original:  Experiments
Revised:  Experiments

Revised: [Table]

Revised: In Section 3 and

Original: [Table]
Revised: [Table]

Original: In Section 3 and 4, we have theoreticallyinterpreted the operation selection throughindividually measuring each operation’sinfluence on supernet, and accordinglydevised an Influential Magnitude ( I M )to replace the commonly-used magnitudeα for architecture selection.
Revised: we have theoretically interpreted the operation selection through individually measuring each operation’s influence on supernet [Table] and accordingly devised an Influential Magnitude ( I M ) to replace the commonly-used magnitude α for architecture selection.

Original: In this section, we conduct a series of experimentsto analyze whether our proposed methodleads to better and more stable results indifferentiable architecture search settings.
Revised: In this section, we conduct a series of experiments to analyze whether our proposed method leads to better and more stable results in differentiable architecture search settings.

Original: We consider two NAS benchmark datasetsearch space, NAS-Bench-201 [12] and
Revised: We consider two NAS benchmark dataset search space, NAS-Bench-201 [12] and

Original: NAS-Bench-1shot1 [46], and the most popular DARTS search space [27].
Revised: NAS-Bench-1shot1 [46] [Table] and the most popular DARTS search space [27].

Original: We verified the effectiveness of the proposed theory-driven method compared with two heuristic methods, argmax and perturbation-based selection, and show that the proposed Influential Magnitude can be applied to
Revised: We verified the effectiveness of the proposed theory-driven method compared with two heuristic methods, argmax and perturbation-based selection, and show that the proposed Influential Magnitude can be applied to

Original: DARTS and its variants with greatly enhancing the performance.
Revised: DARTS and its variants with greatly enhancing the performance.

Original: 6.1 Reproducible Comparison on Benchmark Datasets
Revised: 6.1 Reproducible Comparison on Benchmark Datasets

Original: To empirically verify the effectiveness of our method, we first run a series of experiments on the two
Revised: To empirically verify the effectiveness of our method, we first run a series of experiments on the two

Original: NAS benchmark datasets, NAS-Bench-201 and NAS-Bench-1shot1.
Revised: NAS benchmark datasets, NAS-Bench-201 and NAS-Bench-1shot1.

Original: To make our results reproducible,we only replace the selection part in DARTS with I M and keep the remaining part identical.
Revised: To make our results reproducible, we only replace the selection part in DARTS with I M and keep the remaining part identical.

Original: Table 1 summarizes the statistical comparison results on NAS-Bench-201 with different random seeds ,where DARTS-IF, as described in Section 3, follows DARTS-PT that individually removes operationto measure the supernet performance drop, while DARTS-IM, as described in Section 4, calculatesthe operation sensitivity (influential magnitude) to replace the original magnitude.
Revised: Table 1 summarizes the statistical comparison results on NAS-Bench-201 with different random seeds , where DARTS-IF, as described in Section 3, follows DARTS-PT that individually removes operation to measure the supernet performance drop, while DARTS-IM, as described in Section 4, calculates the operation sensitivity (influential magnitude) to replace the original magnitude.

Original: Algorithm 1 and2 in the Appendix H sketched the two proposed methods.
Revised: Algorithm 1 and 2 in the Appendix H sketched the two proposed methods.

Original: The Random baseline is to randomlyselect architectures without considering α or θ , and DARTS and DARTS-PT leverage the argmaxand perturbation-based heuristic selection, respectively.
Revised: The Random baseline is to randomly select architectures without considering α or θ , and DARTS and DARTS-PT leverage the argmax and perturbation-based heuristic selection, respectively.

Original: As shown, our DARTS-IF and DARTS-IMboth significantly outperform the random baseline and the two heuristic baselines, where DARTS-IMachieves near-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR100, and ImageNet, respectively, in a single run (DARTS-IM with Sherman-Morrison approximationand N = 20 under random seed 0).
Revised: As shown, our DARTS-IF and DARTS-IM both significantly outperform the random baseline and the two heuristic baselines, where DARTS-IM achieves near-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR100, and ImageNet, respectively, in a single run (DARTS-IM with Sherman-Morrison approximation and N = 20 under random seed 0).

Original: [Figure]
Revised: [Figure]

Original: Figure 8 in Appendix G plots thesearched architectures with different architecture selection methodsin NAS-Bench-201.
Revised: Figure 8 in Appendix G plots the searched architectures with different architecture selection methods in NAS-Bench-201.

Original: We can seethat DARTS selects skip-connectionfor all edges as its magnitude usually outweighs other candidate operations.
Revised: We can see that DARTS selects skip-connection for all edges as its magnitude usually outweighs other candidate operations.

Original: The results of DARTS in NASBench-201, which are even poorerthan the random baseline, also suggest that the magnitudes could hardlyreflect the operation importance.
Revised: The results of DARTS in NASBench-201, which are even poorer than the random baseline, also suggest that the magnitudes could hardly reflect the operation importance.

Original: Although DARTS-PT can partially relieve this issue by selecting one ortwo more convolutional operations,it still prefers skip-connections (selects 3 skip-connections for 6 edges)where directly removing them still significantly affect the performance.
Revised: Although DARTS-PT can partially relieve this issue by selecting one or two more convolutional operations, it still prefers skip-connections (selects 3 skip-connections for 6 edges) where directly removing them still significantly affect the performance.

Original: On the contrary, our DARTSIF and DARTS-IM can find valid architectures with more convolutional operations (with only oneskip-connection ).
Revised: On the contrary, our DARTSIF and DARTS-IM can find valid architectures with more convolutional operations (with only one skip-connection ).

Original: Results in Table 1 verified the effectiveness of our influence functions basedselection methods, which outperform baselines by large margins.
Revised: Results in Table 1 verified the effectiveness of our influence functions based selection methods, which outperform baselines by large margins.

Original: DARTS-IF is similar to DARTSPT while with an additional term to approximate the fine-tuned performance and the results that
Revised: DARTS-IF is similar to DARTSPT while with an additional term to approximate the fine-tuned performance and the results that

Original: DARTS-IF outperforms DARTS-PT verified the effectiveness of the influence function explanation.
Revised: DARTS-IF outperforms DARTS-PT verified the effectiveness of the influence function explanation.

Original: [Table]
Revised: [Table]

Original: As discussed in Section 5, we can leverage different methods to approximate the IHVPs inour I M , and Figure 2 also analyzes differentapproximation methods along with the numberof batches N .
Revised: As discussed in Section 5, we can leverage different methods to approximate the IHVPs in our I M , and Figure 2 also analyzes different approximation methods along with the number of batches N .

Original: Full results on three datasetscan be found in the Figure 5 in Appendix F.
Revised: Full results on three datasets can be found in the Figure 5 in Appendix F.

Original: As demonstrated, increasing N can help the
Revised: As demonstrated, increasing N can help the

Original: Sherman-Morrison approximation to more accurately catch the operation importance and weempirically find that N = 30 is enough in the
Revised: Sherman-Morrison approximation to more accurately catch the operation importance and we empirically find that N = 30 is enough in the

Original: NAS-Bench-201 dataset.
Revised: NAS-Bench-201 dataset.

Original: However, we also observe Neumann approximation is not stable which even obtains worse results with increasing numberof batches, showing it is somehow erroneous in estimating IHVPs.
Revised: However, we also observe Neumann approximation is not stable which even obtains worse results with increasing number of batches, showing it is somehow erroneous in estimating IHVPs.

Original: Appendix F analyzes differentinverse Hessian estimation methods for our I M .
Revised: Appendix F analyzes different inverse Hessian estimation methods for our I M .

Original: The NAS-Bench-1shot1 is another popular NAS benchmark dataset to analyze differentiable NASmethods, which is built based on the NAS-Bench-101 [44], through dividing all architectures in
Revised: The NAS-Bench-1shot1 is another popular NAS benchmark dataset to analyze differentiable NAS methods, which is built based on the NAS-Bench-101 [44], through dividing all architectures in

Original: NAS-Bench-101 into three different unified cell-based search spaces to enable differentiable NASmethods to be directly applied.
Revised: NAS-Bench-101 into three different unified cell-based search spaces to enable differentiable NAS methods to be directly applied.

Original: NAS-Bench-1shot1 contains 6240, 29160, and 363648 architecturesin three spaces, respectively, where Space3 is much more complicated than the remaining two spaces.
Revised: NAS-Bench-1shot1 contains 6240, 29160, and 363648 architectures in three spaces, respectively, where Space3 is much more complicated than the remaining two spaces.

Original: In NAS-Bench-1shot1, we tracked the searched architecture in every epoch under different operationselection paradigms.
Revised: In NAS-Bench-1shot1, we tracked the searched architecture in every epoch under different operation selection paradigms.

Original: We considered two popular supernet training baselines, DARTS and PC-DARTS,and Table 2 summarised the best-searched architectures during the supernet training in differentsub-search spaces in NAS-Bench-1shot1.
Revised: We considered two popular supernet training baselines, DARTS and PC-DARTS, and Table 2 summarised the best-searched architectures during the supernet training in different sub-search spaces in NAS-Bench-1shot1.

Original: As demonstrated, our I M can enhance the performanceof DARTS and its variant PC-DARTS in different search spaces of NAS-Bench-1shot1, showingthat our influence magnitude (IM) can find more competitive architectures than the default argmax ,especially in the more complicated space.
Revised: As demonstrated, our I M can enhance the performance of DARTS and its variant PC-DARTS in different search spaces of NAS-Bench-1shot1, showing that our influence magnitude (IM) can find more competitive architectures than the default argmax , especially in the more complicated space.

Original: Compared with the perturbation strategy (PT), our IM ismore effective to measure the operation sensitivity with obtaining 5 better results in 6 cases.
Revised: Compared with the perturbation strategy (PT), our IM is more effective to measure the operation sensitivity with obtaining 5 better results in 6 cases.

Original: 6.2 Reproducible Comparison on DARTS Search Space
Revised: 6.2 Reproducible Comparison on DARTS Search Space

Original: [Table]
Revised: [Table]

Original: The DARTS search space is the mostcommonly-used space to evaluate the NASmethods, while most SOTAs only report theperformance of their best searched architectures without analyzing the search statistics.
Revised: The DARTS search space is the most commonly-used space to evaluate the NAS methods, while most SOTAs only report the performance of their best searched architectures without analyzing the search statistics.

Original: To encourage reproducibility, we train thesupernet with the same random seeds, andthe trained supernet is utilized to derive thearchitecture under different operation selection strategies.
Revised: To encourage reproducibility, we train the supernet with the same random seeds, and the trained supernet is utilized to derive the architecture under different operation selection strategies.

Original: The comparison results areprovided in Table 3, from which we can seethat, compared with the argmax baseline, thearchitectures searched by our Influential Magnitude obtain better performance on CIFAR-10 and
Revised: The comparison results are provided in Table 3, from which we can see that, compared with the argmax baseline, the architectures searched by our Influential Magnitude obtain better performance on CIFAR-10 and

Original: ImageNet than DARTS and its variant PC-DARTS, and DARTS-PT on ImageNet, implying ourmethod is more stable to find valid and competitive architectures.
Revised: ImageNet than DARTS and its variant PC-DARTS, and DARTS-PT on ImageNet, implying our method is more stable to find valid and competitive architectures.

Original: Figure 9 in Appendix G also plotsthe searched architecture examples with different selection methods in the DARTS space.
Revised: Figure 9 in Appendix G also plots the searched architecture examples with different selection methods in the DARTS space.

Original: As shown,although DARTS, PC-DARTS and DARTS-PT can find satisfying architecture after multiple runs, theargmax is hard to consistently obtain stable results.
Revised: As shown, although DARTS, PC-DARTS and DARTS-PT can find satisfying architecture after multiple runs, the argmax is hard to consistently obtain stable results.

Original: In addition, different from the perturbation-basedoperation selection which needs to fine-tune the supernet after the discretization of each edge thatcosts 0.4 additional GPU day for DARTS, our influential magnitude is calculated by one time whichonly costs hundred seconds, making our method is much more efficient.
Revised: In addition, different from the perturbation-based operation selection which needs to fine-tune the supernet after the discretization of each edge that costs 0.4 additional GPU day for DARTS, our influential magnitude is calculated by one time which only costs hundred seconds, making our method is much more efficient.

Original: Detailed comparisons onbest-searched architecture in DARTS space with the SOTAs can be found in Table 5 in Appendix E.
Revised: Detailed comparisons on best-searched architecture in DARTS space with the SOTAs can be found in Table 5 in Appendix E.

Original: 6.3 Discussion on the Robustness of Search
Revised: 6.3 Discussion on the Robustness of Search

Original: [Figure]
Revised: [Figure]

Original: [Equation] [Figure]
Revised: [Figure] [Figure]

Revised: c_{k

Original: [Figure]
Revised: [Figure]

Original: [Equation] [Figure] [Equation]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Equation]
Revised: c_{

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: c_{k-2} 1 skip_connect 2 skip_connect 3 sep_conv_3x dil_conv_5x5 dil_conv_3x3 dil_conv_3x c_{k} dil_conv_3x dil_conv_3x c_{k-1} (
Revised: dil

Original: [Figure]
Revised: [Figure]

Original: ) S The robustness has been a criticalconcern in the differentiable architecture search community since Zelaet al.
Revised: The robustness has been a critical concern in the differentiable architecture search community since Zela et al.

Original: [45] observed a performancecollapse that DARTS tends to yielddegenerate architectures, which especially prefers the skip-connectionswith the search progressing.
Revised: [45] observed a performance collapse that DARTS tends to yield degenerate architectures, which especially prefers the skip-connections with the search progressing.

Original: Similarto DARTS-PT, we also focus on theoperation selection part to explain thefailure of DARTS.
Revised: Similar to DARTS-PT, we also focus on the operation selection part to explain the failure of DARTS.

Original: Figure 3 presentsseveral architectures by DARTS withour I M on different search spaces.
Revised: Figure 3 presents several architectures by DARTS with our I M on different search spaces.

Original: Apart from NAS-Bench-201, we alsoconsider several tool search spaces(S1-S4) proposed by [45], which arespecifically designed to verify the robustness of differentiable NAS methods.
Revised: Apart from NAS-Bench-201, we also consider several tool search spaces (S1-S4) proposed by [45], which are specifically designed to verify the robustness of differentiable NAS methods.

Original: The complete analysis and searched architectures can be found in Appendix G. As shownin Figure 3, DARTS-IM can find more meaningful architectures in different spaces.
Revised: The complete analysis and searched architectures can be found in Appendix G. As shown in Figure 3, DARTS-IM can find more meaningful architectures in different spaces.

Original: For example,
Revised: For example,

Original: DARTS fails dramatically in most spaces that only selects skip-connection for all edges, while
Revised: DARTS fails dramatically in most spaces that only selects skip-connection for all edges, while

Original: DARTS-IM can find architecture with 5 convolutions in NAS-Bench-201, and 6 and 8 convolutionsin S1 and S2.
Revised: DARTS-IM can find architecture with 5 convolutions in NAS-Bench-201, and 6 and 8 convolutions in S1 and S2.

Original: 6.4 Discussion at Non-Convergence
Revised: 6.4 Discussion at Non-Convergence

Original: [Figure]
Revised: [Figure]

Original: As discussed in Section 4, our influential magnitude is derived when θis in the optimum, with leveragingdifferent methods to approximate the
Revised: As discussed in Section 4, our influential magnitude is derived when θ is in the optimum, with leveraging different methods to approximate the

Original: IHVPs.
Revised: IHVPs.

Original: In this subsection, we analyze whether the proposed influentialmagnitude is still effective to find important operations when θ is not inthe optimum.
Revised: In this subsection, we analyze whether the proposed influential magnitude is still effective to find important operations when θ is not in the optimum.

Original: Figure 4 tracked theperformance of searched architecturein each epoch during the search with
Revised: Figure 4 tracked the performance of searched architecture in each epoch during the search with

Original: Sherman-Morrison formula.
Revised: Sherman-Morrison formula.

Original: Full results on three datasets and more comparison with DARTS-PT can be found in the Figure 7 in Appendix F. As shown, our influentialmagnitude I M is still effective when θ is not in the optimum, while which is getting more stable withthe supernet training.
Revised: Full results on three datasets and more comparison with DARTS-PT can be found in the Figure 7 in Appendix F. As shown, our influential magnitude I M is still effective when θ is not in the optimum, while which is getting more stable with the supernet training.

Original: Specifically, as shown in Figure 4 (a) and (b), our I M with the different N areboth effective, implying that our method is sufficiently informative for the operation selection evenunder different approximation noises.
Revised: Specifically, as shown in Figure 4 (a) and (b), our I M with the different N are both effective, implying that our method is sufficiently informative for the operation selection even under different approximation noises.

Original: More interesting, our DARTS-IM under Sherman-Morrison formula with N =1 in Figure 4 meansthat we simply approximate the inverse H − 1{+
+} =
Revised: More interesting, our DARTS-IM under Sherman-Morrison formula with N =1 in Figure 4 means that we simply approximate the inverse H − {+
+} =

Original: η − 1 (cid:16) I − η −
Revised: η − 1 (cid:16) I − η −

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation] [Figure] [Equation] that ˆ H − 1is a diagonal matrix that ˆ H ii = ( ∇ θ i L∇
Revised: that ˆ H −is a diagonal matrix that ˆ H ii = ( ∇ θ i L∇

Original: θ i L ) , which is also called the diagonal approximation of theinverse Hessian.
Revised: θ i L ) , which is also called the diagonal approximation of the inverse Hessian.

Original: As shown, with roughly approximated second-order information, Eq.
Revised: As shown, with roughly approximated second-order information, Eq.

Original: (7) also help
Revised: (7) also help

Original: DARTS-IM to find more important operations compared with existing heuristic methods.
Revised: DARTS-IM to find more important operations compared with existing heuristic methods.

Original: 6.5 Discussion on the Limitations in Practice
Revised: 6.5 Discussion on the Limitations in Practice

Original: Although our influential magnitude provides a theory-driven and concise solution for the operationselection in DARTS, we consider several assumptions and approximations for the practical implementation.
Revised: Although our influential magnitude provides a theory-driven and concise solution for the operation selection in DARTS, we consider several assumptions and approximations for the practical implementation.

Original: For example, Basu et al.
Revised: For example, Basu et al.

Original: [4, 5] pointed out that the second-order influence values stillunderestimated the true group influence values in the deep neural networks, while they also observedthe correlation marginally improves with the number of parameters in a group and the second-orderinfluence function generally outperforms the first-order.
Revised: [4, 5] pointed out that the second-order influence values still underestimated the true group influence values in the deep neural networks, while they also observed the correlation marginally improves with the number of parameters in a group and the second-order influence function generally outperforms the first-order.

Original: Fortunately, this paper considers the wholeparameters in the supernet, which helps the second-order influence function reach its potential.
Revised: Fortunately, this paper considers the whole parameters in the supernet, which helps the second-order influence function reach its potential.

Original: The approximation of IHVPs requires the conditions that the Hessian matrix is positive definite[29] or the empirical Fisher can replace the true Fisher matrix [14], and the supernet weight θ ∗ is aglobal minimum of the empirical risk, while which in practice, are hard to hold.
Revised: The approximation of IHVPs requires the conditions that the Hessian matrix is positive definite [29] or the empirical Fisher can replace the true Fisher matrix [14], and the supernet weight θ ∗ is a global minimum of the empirical risk, while which in practice, are hard to hold.

Original: In the calculation ofv ∗ ∂ 2 L ∂α∂θ , which is detailed described in Appendix B, this paper also follows DARTS that considersthe finite difference approximation for practical calculation.
Revised: In the calculation of v ∗ ∂ 2 L ∂α∂θ , which is detailed described in Appendix B, this paper also follows DARTS that considers the finite difference approximation for practical calculation.

Original: These approximations and assumptionsmay bring errors for the practical calculation, while empirical results verified that our influentialmagnitude is sufficiently informative for the operation selection in DARTS.
Revised: These approximations and assumptions may bring errors for the practical calculation, while empirical results verified that our influential magnitude is sufficiently informative for the operation selection in DARTS.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: This paper focuses on the operation selection, which is an essential part while with less attention inthe differentiable architecture search.
Revised: This paper focuses on the operation selection, which is an essential part while with less attention in the differentiable architecture search.

Original: Rather than considering heuristics, this paper introduces theinfluence functions to theoretically reveal the candidate operation’s importance through approximatingits influence on the supernet.
Revised: Rather than considering heuristics, this paper introduces the influence functions to theoretically reveal the candidate operation’s importance through approximating its influence on the supernet.

Original: By leveraging the Taylor’s approximation, we gracefully interpret howthe validation performance changes during the discretization in DARTS and show that the operationstrength is not only related to the magnitude but also the second-order information, where an influentialmagnitude is accordingly devised for the architecture selection.
Revised: By leveraging the Taylor’s approximation, we gracefully interpret how the validation performance changes during the discretization in DARTS and show that the operation strength is not only related to the magnitude but also the second-order information, where an influential magnitude is accordingly devised for the architecture selection.

Original: We based our framework on DARTSand extensive experimental results verified the proposed influential magnitude can be practically usedin operation selection to avoid failures in differentiable architecture search.
Revised: We based our framework on DARTS and extensive experimental results verified the proposed influential magnitude can be practically used in operation selection to avoid failures in differentiable architecture search.

Original: Developing more accurateinfluential magnitude for architecture search or leveraging the training dynamics [20] to reveal theoperation influence in the early of training and also extending influential magnitude to other bi-leveloptimization problems[21, 28, 29], e.g., network pruning [28], are among directions for future work.Checklist For all authors...
Revised: Developing more accurate influential magnitude for architecture search or leveraging the training dynamics [20] to reveal the operation influence in the early of training and also extending influential magnitude to other bi-level optimization problems[21, 28, 29], e.g., network pruning [28], are among directions for future work.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope? [Yes] (b) Did you describe the limitations of your work?
Revised: Acknowledgments and Disclosure of Funding

Original: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [No]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments

Original: multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?

Original: [Yes]

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets?
Revised: This work was partially supported by Independent Research Fund Denmark under agreements 802200246B and 8048-00038B, the VILLUM FONDEN under agreement 34328, and the Innovation Fund

Original: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating? [N/A]
Revised: Denmark centre, DIREC

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: informa-

Original: tion or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)

Original: approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on

Original: participant compensation?

Original: [N/A]

Original: A Proof
Revised: A Proof

Original: Theorem 1: Suppose that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗after supernet training, α changes to ˆ α when conducting architecture discretization, and the train-from-scratchvalidation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .
Revised: Theorem 1: Suppose that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ after supernet training, α changes to ˆ α when conducting architecture discretization, and the train-from-scratch validation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .

Original: If the third-derivative of the loss function L at optimum is zero or sufficientlysmall [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have
Revised: If the third-derivative of the loss function L at optimum is zero or sufficiently small [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ [Equation] , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1: Based on Eq.(4), we now only need to calculate ∆ θ to get ∆ L after we change the565 α to ˆ α as other parts could be directly obtained, while we try to avoid the time-consuming retraining processthrough leveraging implicit function to estimating ˆ θ ∗ .
Revised: Proof of Theorem 1: Based on Eq.(4), we now only need to calculate ∆ θ to get ∆ L after we change the α to ˆ α as other parts could be directly obtained, while we try to avoid the time-consuming retraining process through leveraging implicit function to estimating ˆ θ ∗ .

Original: First, based on the local optimal, we have
Revised: First, based on the local optimal, we have

Original: [Equation]
Revised: [Equation]

Original: When we consider the second-order Taylor expansion on θ , we have
Revised: When we consider the second-order Taylor expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (12) is obtained since we assume the third-derivative of the loss function L at optimum is zero or sufficiently569 small [4].
Revised: (12) is obtained since we assume the third-derivative of the loss function L at optimum is zero or sufficiently small [4].

Original: And we have
Revised: And we have

Original: [Equation]
Revised: [Equation]

Original: So, 
Revised: So, ∆

Original: [Equation]
Revised: [Equation]

Original: In this way, based on Eq.(4), (14) and (15), we have
Revised: In this way, based on Eq.(4), (14) and (15), we have

Original: [Equation]
Revised: [Equation]

Original: In this way, we can approximate every change in the α .
Revised: In this way, we can approximate every change in the α .

Original: □
Revised: □

Original: Corollary 1: Based on the Assumption 1-3, we could bound the error between the approximated validationloss L ( ˆ θ ∗ , ˆ α ) and the ground-truth ˜
Revised: Corollary 1: Based on the Assumption 1-3, we could bound the error between the approximated validation loss L ( ˆ θ ∗ , ˆ α ) and the ground-truth ˜

Original: L 2 ( ˆ θ ∗ , ˆ α ) in DARTS with E =
Revised: L 2 ( ˆ θ ∗ , ˆ α ) in DARTS with E =

Original: (cid:13)(cid:13) (cid:13)
Revised: (cid:13)(cid:13) (cid:13)

Revised: L ( ˆ θ ∗ , ˆ

Revised: [Equation]

Original: L ( ˆ θ ∗ , ˆ α )
Revised: L ( ˆ [Equation] ˆ [Equation]

Original: − ˜ L ( ˆ θ ∗ , ˆ α ) (cid:13)(cid:13) (cid:13) ⩽ K 3 6 max (cid:12)(cid:12) (cid:12) ∂
Revised: ) (cid:13)(cid:13) (cid:13) ⩽ [Equation] [Equation] (cid:12)(cid:12) (cid:12) ∂

Revised: L

Revised: [Equation]

Original: L 3 ∂θ 3 (cid:12)(cid:12) (cid:12) ,where K = C L λ ∗ ∥
Revised: 3 (cid:12)(cid:12) (cid:12) , where K = C L λ [Equation] ∆ α

Revised: [Equation]

Revised: +

Original: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥
Revised: [Equation] H [Equation] 2 min ∗ λ ∗ ∥

Original: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 4 ) .
Revised: ∆ α ∥ [Equation] + o ( ∥ ∆ α ∥ 4 ) .

Original: Proof of Corollary 
Revised: Proof of Corollary 1: Before our analysis on the error bound of approximation in Theorem 1, we first restate the following common assumptions in the bi-level optimization [9, 15–

Original: Assumption 1: For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with constant C f > 0 andconstant C L > 0 , respectively.
Revised: Assumption 1: For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with constant C f > 0 and constant C L > 0 , respectively.

Original: Assumption 2: L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θ around582 θ ∗ ( α ) .
Revised: Assumption 2: L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θ around θ ∗ ( α ) .

Original: Assumption 3: (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .
Revised: Assumption 3: (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .

Original: Now we can estimate the error bound produced by Theorem 1.
Revised: Now we can estimate the error bound produced by Theorem 1.

Original: First, the error is due to the second-order Taylor585 expansion on θ in Eq.
Revised: First, the error is due to the second-order Taylor expansion on θ in Eq.

Original: We have the error
Revised: We have the error

Original: [Equation]
Revised: [Equation]

Original: where we should notice that the approximated validation loss L (ˆ θ ∗ , ˆ α ) is Taylor expansion of ˆ L in the pointˆ θ ∗ − ∆ θ .

Original: Since we set that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ

Original: [Equation]
Revised: [Equation]

Original: 1 ∗ [Equation] L ( θ ∗ , [Equation]
Revised: where we should notice that the approximated validation loss L (ˆ θ ∗ , ˆ α ) is Taylor expansion of ˆ L in the point ˆ θ ∗ − ∆ θ .

Original:  in Eq. 13, this error E could be easily bound.
Revised: Since we set that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L ( θ ∗ , ˆ α ) ∂θ in Eq. 13, this error E could be easily bound.

Original: However, we should notice that our ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ
Revised: However, we should notice that our ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ

Original: [Equation]
Revised: [Equation]

Original: 1 ∗ ∂ L ( θ ∗ , ˆ α )
Revised: ∗ ∂ L ( θ ∗ , ˆ α )

Original: ∂θ is an approximation, that θ ∗ + ∆ θ ̸ = ˆ θ ∗ ,and we define that θ ∗ + ∆ θ =
Revised: ∂θ is an approximation, that θ ∗ + ∆ θ ̸ = ˆ θ ∗ , and we define that θ ∗ + ∆ θ =

Original: ˆ θ ∗ e and ∆ θ e = ˆ θ ∗ − ˆ θ ∗ e .
Revised: ˆ θ ∗ e and ∆ θ e = ˆ θ ∗ − ˆ θ ∗ e .

Original: First, we have
Revised: First, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the first part in Eq.
Revised: For the first part in Eq.

Original: (18), based on Assumption 2 that L ( θ, α ) is λ -strongly convex with θ around θ ∗ , wehave:
Revised: (18), based on Assumption 2 that L ( θ, α ) is λ -strongly convex with θ around θ ∗ , we have:

Original: [Equation]
Revised: [Equation]

Original: 19) and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: [Equation] When we consider that L ( θ ∗ , α ) is C L Lipschitz continuous with α , we have 
Revised: When we consider that L ( θ ∗ , α ) is C L Lipschitz continuous with α , we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: Baesd on Eq.
Revised: Baesd on Eq.

Original: (20) and (21), we have
Revised: (20) and (21), we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As to the second part, when we assume that L ( θ, α ) is λ -convex near the local optimal, we also have
Revised: As to the second part, when we assume that L ( θ, α ) is λ -convex near the local optimal, we also have

Original: [Equation]
Revised: [Equation]

Original: [Equation] where ˆ θ ∗ is the true optimal, and ˆ θ ∗ e is the approximate local optimal when we utilize the implicit functiontheorem to approximate ∆ θ in Eq.
Revised: 23) where ˆ θ ∗ is the true optimal, and ˆ θ ∗ e is the approximate local optimal when we utilize the implicit function theorem to approximate ∆ θ in Eq.

Original: 13.so that, we have
Revised: so that, we have

Original: [Equation]
Revised: [Equation]

Original: Consider the local optimal that ∇L (ˆ θ ∗ , ˆ α ) = 0 , we have
Revised: Consider the local optimal that ∇L (ˆ θ ∗ , ˆ α ) = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Now, we need to bound (cid:13)(cid:13)(cid:13)
Revised: Now, we need to bound (cid:13)(cid:13)(cid:13)

Original: ∇L
Revised: ∇L

Original: (ˆ θ ∗ e , ˆ α ) (cid:13)(cid:13)(cid:13) .
Revised: (ˆ θ ∗ e , ˆ α ) (cid:13)(cid:13)(cid:13) .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since we utilize Eq.
Revised: Since we utilize Eq.

Original: (13) that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L
Revised: (13) that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L

Original: [Equation]
Revised: [Equation]

Original: θ ∗ , ˆ α )
Revised: ∗ [Equation] ˆ α [Equation]

Original: ∂θ to conduct the second-order Taylor expansion,603 we have
Revised: ∂θ to conduct the second-order Taylor expansion, we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the second row is calculated since (cid:82) ∇ θ L 2 ( θ ∗ + t ∗ ∆ θ, ˆ α ) ∗ ∆ θ dt = ∇ θ L ( θ ∗ + t ∗ ∆ θ, ˆ α ) .
Revised: where the second row is calculated since (cid:82) ∇ θ L 2 ( θ ∗ + t ∗ ∆ θ, ˆ α ) ∗ ∆ θ dt = ∇ θ L ( θ ∗ + t ∗ ∆ θ, ˆ α ) .

Original: The fifthrow is calculated as we assume that L is convex and twice-differentiable with C H .
Revised: The fifth row is calculated as we assume that L is convex and twice-differentiable with C H .

Original: And we assume that the∂ L ( θ ∗ , ˆ α )
Revised: And we assume that the ∂ L ( θ ∗ , ˆ α )

Original: ∂θ is bounded by C l .
Revised: ∂θ is bounded by C l .

Original: σ min is the smallest eigenvalue of Hessian matrix ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ .
Revised: σ min is the smallest eigenvalue of Hessian matrix ∂ 2 L [Equation] ∂θ∂θ .

Original: In addition, when we consider a more tight bound, and consider a Taylor expansion on α , we have
Revised: In addition, when we consider a more tight bound, and consider a Taylor expansion on α , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where the second row is obtained based on the local optimal ∂ L ( θ ∗ ,α 
Revised: where the second row is obtained based on the local optimal ∂ L ( θ ∗ ,α )

Original: ) ∂θ [Equation]
Revised: ∂θ = 0 .

Original: [Equation] we have∥∇L ( [Equation]
Revised: So, we have

Original: [Equation]
Revised: [Equation]

Original: 30) and
Revised: 30) and

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Based on Eq.
Revised: Based on Eq.

Original: (18), (22), and (31), we have
Revised: (18), (22), and (31), we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The final error bound can be calculated as:
Revised: The final error bound can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: where K = C L λ ∗ ∥
Revised: where K = C L λ ∗ ∥

Original: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥
Revised: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥

Original: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .
Revised: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .

Original: □
Revised: □

Original: Derivation of Eq.
Revised: Derivation of Eq.

Original: In Section 4, when we consider a first-order Taylor expansion on α as we only applyan infinitesimal change of α , and we alsoconsider a second-order Taylor expansion on θ , we have
Revised: In Section 4, when we consider a first-order Taylor expansion on α as we only apply an infinitesimal change of α , and we alsoconsider a second-order Taylor expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: where the last row is obtained is obtained due to the local optimal ∂ L ( θ ∗ ,α ) ∂θ
Revised: where the last row is obtained is obtained due to the local optimal ∂ L ( θ ∗ ,α ) ∂θ

Original: [Equation]
Revised: [Equation]

Original: Theorem 2: Supposed that DARTS obtains the optimized architecture parameter α with supernet weights619 θ ∗ after supernet training, and we pose an infinitesimal change on α .
Revised: Theorem 2: Supposed that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ after supernet training, and we pose an infinitesimal change on α .

Original: Based on implicit function theorem andunder the assumption that the third-derivative of the loss function at optimum is zero or sufficiently small [4],621 the change of validation performance can be estimated as:
Revised: Based on implicit function theorem and under the assumption that the third-derivative of the loss function at optimum is zero or sufficiently small [4], the change of validation performance can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.
Revised: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.

Original: Proof of Theorem 2: Although Section 3 provides an iterative solution to measure the operation importancebased Eq.
Revised: Proof of Theorem 2: Although Section 3 provides an iterative solution to measure the operation importance based Eq.

Original: (5) through individually removing each operation, we need to calculate n times (the number of625 all candidate operations in the supernet work) to estimate the importance of all candidate operations.
Revised: (5) through individually removing each operation, we need to calculate n times (the number of all candidate operations in the supernet work) to estimate the importance of all candidate operations.

Original: Moreimportant, since removing one operation poses a considerable change on α ( α i → 0 ), the approximation error627 produced by Eq.(5) is non-negligible which may affects the accuracy as stated by Corollary 1.
Revised: More important, since removing one operation poses a considerable change on α ( α i → 0 ), the approximation error produced by Eq.(5) is non-negligible which may affects the accuracy as stated by Corollary 1.

Original: So, a morepractical solution to illustrate the importance of each operation is to estimate how the validation performance629 will change after posing an infinitesimal change on α , a.k.a. operation sensitivity, and we can directly get theoperation importance with one calculation.
Revised: So, a more practical solution to illustrate the importance of each operation is to estimate how the validation performance will change after posing an infinitesimal change on α , a.k.a. operation sensitivity, and we can directly get the operation importance with one calculation.

Original: When we consider that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ aftersupernet training, and we pose an infinitesimal change on α , we can conduct a first-order Taylor expansion on α633 a second-order Taylor expansion on θ .
Revised: When we consider that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ after supernet training, and we pose an infinitesimal change on α , we can conduct a first-order Taylor expansion on α a second-order Taylor expansion on θ .

Original: So that we have get the Eq.
Revised: So that we have get the Eq.

Original: (34) as stated in the above.
Revised: (34) as stated in the above.

Original: When we furtherconduct a second Taylor expansion on ˆ θ ∗ for ∂ L (ˆ θ ∗ ,α ) ∂α , we have
Revised: When we further conduct a second Taylor expansion on ˆ θ ∗ for ∂ L (ˆ θ ∗ ,α ) ∂α , we have

Original: [Equation]
Revised: [Equation]

Original:  where the last row is obtained when we neglect the third and higher derivatives of the loss function withconsidering the chain rule that ∂ L ∂α =
Revised: ) where the last row is obtained when we neglect the third and higher derivatives of the loss function with considering the chain rule that ∂ L ∂α =

Original: ∂ L ∂θ ∗ ∂ ( θ ∗ ( α )) ∂α , and assume α is a local optimal for the continuous637 magnitude optimization after architecture search.
Revised: ∂ L ∂θ ∗ ∂ ( θ ∗ ( α )) ∂α , and assume α is a local optimal for the continuous magnitude optimization after architecture search.

Original: From the above, we can find that, the challenge is still that itis intractable to calculate ∆ θ .
Revised: From the above, we can find that, the challenge is still that it is intractable to calculate ∆ θ .

Original: Similar as before, when we consider implicit function and second-order Taylor639 expansion on θ , we have
Revised: Similar as before, when we consider implicit function and second-order Taylor expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: When we neglect the third and higher derivatives and the local optimal ∂ L ( θ ∗ ,α [Equation] ∂θ 
Revised: When we neglect the third and higher derivatives and the local optimal ∂ L ( θ ∗ ,α ) ∂θ =

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Then, similar as before, when we apply a first-order Taylor expansion on ˆ α , with the chain-rule and neglect third642 and higher derivatives, we have
Revised: Then, similar as before, when we apply a first-order Taylor expansion on ˆ α , with the chain-rule and neglect third and higher derivatives, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: After applying Eq.
Revised: After applying Eq.

Original: (40) on Eq.
Revised: (40) on Eq.

Original: (36), we have
Revised: (36), we have

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: B Practical Implementation
Revised: B Practical Implementation

Original: To practically calculate the proposed influential magnitude as Definition 1, we need to estimate the InverseHessian-Vector products (IHVPs) H − 1 v , and v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .
Revised: To practically calculate the proposed influential magnitude as Definition 1, we need to estimate the InverseHessian-Vector products (IHVPs) H − 1 v , and v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .

Original: In the following, we first described two methods toestimate H − 1 v , and the following DARTS to described the practical calculation on v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .
Revised: In the following, we first described two methods to estimate H − 1 v , and the following DARTS to described the practical calculation on v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .

Original: B.1 Neumann series approximation for Lemma
Revised: B.1 Neumann series approximation for Lemma

Original: The most common method for the IHVPs is the Neumann series that, for ∥ I − A ∥ < 1 , we have:[Equation]
Revised: The most common method for the IHVPs is the Neumann series that, for ∥ I − A ∥ < 1 , we have:

Original: [Equation]
Revised: [Equation]

Original: However, it is impossible to guarantee that ∥ I − H ∥ < 1 .
Revised: However, it is impossible to guarantee that ∥ I − H ∥ < 1 .

Original: Different from [22], we consider H − 1 = γ ( γH ) − 1 .
Revised: Different from [22], we consider H − 1 = γ ( γH ) − 1 .

Original: When assuming H is bounded with σ max , (cid:13)(cid:13)(cid:13) ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < σ max .
Revised: When assuming H is bounded with σ max , (cid:13)(cid:13)(cid:13) ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < σ max .

Original: With γ < 1 σ max , we have (cid:13)(cid:13)(cid:13) I − γ ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) <1[29, 36].
Revised: With γ < 1 σ max , we have (cid:13)(cid:13)(cid:13) I − γ ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) <

Original: When we conduct the Neumann series approximation for (cid:104) ∂ 2 L ∂θ∂θ (cid:105) − 1 in the optimal point, we have:(cid:20) ∂ 2 L [Equation] (cid:21)
Revised: When we conduct the Neumann series approximation for (cid:104) ∂ 2 L ∂θ∂θ (cid:105) − 1 in the optimal point, we have:

Original: [Equation]
Revised: [Equation]

Original: when we only consider the first K terms and H is positive in θ ∗ (Please note that, generally, we could onlyguarantee H is positive when in optimal point).
Revised: when we only consider the first K terms and H is positive in θ ∗ (Please note that, generally, we could only guarantee H is positive when in optimal point).

Original: Accordingly, the Lemma 1 is derived.
Revised: Accordingly, the Lemma 1 is derived.

Original: As suggested by [49], a medium K is large enough to approximate IHVPs, where K = 2 obtains similar results657 as K ⩾ 3 while with less computational cost.
Revised: As suggested by [49], a medium K is large enough to approximate IHVPs, where K = 2 obtains similar results as K ⩾ 3 while with less computational cost.

Original: In this paper, we also observed a similar phenomenon and also set
Revised: In this paper, we also observed a similar phenomenon and also set

Original: K = 2 by default in our Neumann series approximation based IHVPs.
Revised: K = 2 by default in our Neumann series approximation based IHVPs.

Original: B.2 Sherman-Morrison formula for Lemma2
Revised: B.2 Sherman-Morrison formula for Lemma

Original: Apart from the Neumann series approximation, we could also follow the Sherman-Morrison formula to exactly661 calculate the inverse of Hessian when we consider the empirical Fisher to replace the Fisher matrix.
Revised: Apart from the Neumann series approximation, we could also follow the Sherman-Morrison formula to exactly calculate the inverse of Hessian when we consider the empirical Fisher to replace the Fisher matrix.

Original: First, weknow that the Hessian equals to fisher information matrix in the optimal point.
Revised: First, we know that the Hessian equals to fisher information matrix in the optimal point.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We name the r = 1 n ( y i − f ( x i ))
Revised: We name the r = 1 n ( y i − f ( x i ))

Original: ∇ θ ∇ θ f , which is the residual contribution to Hessian H .
Revised: ∇ θ ∇ θ f , which is the residual contribution to Hessian H .

Original: At the end of training,664 if we can achieve zero training error, namely, y i = f
Revised: At the end of training, if we can achieve zero training error, namely, y i = f

Original: ( x i ) , the residual will converge to zero.
Revised: ( x i ) , the residual will converge to zero.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Please note that the dimension of ∇ θ f is M ∗ m where m is the number of classes (output dimension) and M is666 the number of parameters.
Revised: Please note that the dimension of ∇ θ f is M ∗ m where m is the number of classes (output dimension) and M is the number of parameters.

Original: Practically, we usually consider an empirical Fisher to approximate the Fisher matrixthat
Revised: Practically, we usually consider an empirical Fisher to approximate the Fisher matrix that

Original: [Equation]
Revised: [Equation]

Original: So, in the remaining text, the F indicates the empirical Fisher for simplicity.
Revised: So, in the remaining text, the F indicates the empirical Fisher for simplicity.

Original: The Sherman-Morrison formula is defined as:
Revised: The Sherman-Morrison formula is defined as:

Original: [Equation]
Revised: [Equation]

Original: When we define that F n = 1 N (cid:80) nn =1 ∇ θ L ( y n , f ( x n ))( ∇ θ L ( y n , f ( x n ))
Revised: When we define that F n = 1 N (cid:80) nn =1 ∇ θ L ( y n , f ( x n ))( ∇ θ L ( y n , f ( x n ))

Original: T , we can notice that the empirical671 Fisher can be represented as F
Revised: T , we can notice that the empirical Fisher can be represented as F

Original: N , and we have:
Revised: N , and we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where L = ℓ + η R ( θ ) that ℓ is a cross-entropy loss and R is the regularization term, and η is the weight-decay,673 F 0 = ηI .
Revised: where L = ℓ + η R ( θ ) that ℓ is a cross-entropy loss and R is the regularization term, and η is the weight-decay, F 0 = ηI .

Original: L n is the loss for the n-th batch dataset.
Revised: L n is the loss for the n-th batch dataset.

Original: And based on the Sherman-Morrison formula, we have
Revised: And based on the Sherman-Morrison formula, we have

Original: [Equation]
Revised: [Equation]

Original: Please notice that, whether using the Neumann series or the Sherman-Morrison formula, we always need675 to calculate the Inverse-Hessian Vector Products (IHVPs) rather than only the Inverse-Hessian.
Revised: Please notice that, whether using the Neumann series or the Sherman-Morrison formula, we always need to calculate the Inverse-Hessian Vector Products (IHVPs) rather than only the Inverse-Hessian.

Original: It is easy toimplement the IHVPs based on the Neumann series through the Hessian-vector products based on Lemma 1.
Revised: It is easy to implement the IHVPs based on the Neumann series through the Hessian-vector products based on Lemma 1.

Original: In677 the following, we discuss how to implement the IHVPs based on Sherman-Morrison formula.
Revised: In the following, we discuss how to implement the IHVPs based on Sherman-Morrison formula.

Original: When we assumethat we have n + 1 batches datasets, we have
Revised: When we assume that we have n + 1 batches datasets, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We can find that, to solve the previous equation, we need to first recurrently calculate r N = F − 1 N − 1 ∇ θ L N .
Revised: We can find that, to solve the previous equation, we need to first recurrently calculate r N = F − 1 N − 1 ∇ θ L N .

Original: [Equation]
Revised: [Equation]

Original: We first need to calculate the r n = F − 1 n − 1 ∇ θ L n .
Revised: We first need to calculate the r n = F − 1 n − 1 ∇ θ L n .

Original: [Equation]
Revised: [Equation]

Original: So, we can first iterative calculate r n , and then calculate F − 1 n v .
Revised: So, we can first iterative calculate r n , and then calculate F − 1 n v .

Original: B.3 Practical Calculation on Matrix-Vector product
Revised: B.3 Practical Calculation on Matrix-Vector product

Original: Apart from the IHVPs, we also need to calculate v a ∗ ∂ 2 L ∂α∂θ and v m ∗ ∂ 2 L ∂θ∂α , which are called as matrix-vectorproduct and computational-expensive.
Revised: Apart from the IHVPs, we also need to calculate v a ∗ ∂ 2 L ∂α∂θ and v m ∗ ∂ 2 L ∂θ∂α , which are called as matrix-vector product and computational-expensive.

Original: Following DARTS [27], we also utilize the finite difference approximation685 for the practical calculation.
Revised: Following DARTS [27], we also utilize the finite difference approximation for the practical calculation.

Original: Let ξ be a small scalar, and consider the function ∂
Revised: Let ξ be a small scalar, and consider the function ∂

Original: L ( θ,α + ξ ∗ v a ) ∂θ with Taylor expansion on α , we have
Revised: L ( θ,α + ξ ∗ v a ) ∂θ with Taylor expansion on α , we have

Original: [Equation]
Revised: [Equation]

Original: when we only consider the first two terms, we have
Revised: when we only consider the first two terms, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, the v m ∗ ∂ 2 L ∂θ∂α can be approximated as:
Revised: Similarly, the v m ∗ ∂ 2 L ∂θ∂α can be approximated as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: C Why perturbation-based method fails
Revised: C Why perturbation-based method fails

Revised: The perturbation-based method, e

Original: [Table]
Revised: [Table]

Original: The perturbation-based method, e.g. the leave-one-out retraining [22], is an intuitive way to measure the691 importance of every candidate operations.
Revised: g. the leave-one-out retraining [22] [Table] is an intuitive way to measure the importance of every candidate operations.

Original: More detailed, we can individually remove every candidate operationin each edge
Revised: More detailed, we can individually remove every candidate operation in each edge

Original: [Table]
Revised: [Table]

Original: and then fine-tune the supernet to the validation performance drop L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: and then fine-tune the supernet to the validation performance drop L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: However,
Revised: However,

Original: DARTS-PT abandons the retraining part that directly measure the validation performance drop.
Revised: DARTS-PT abandons the retraining part that directly measure the validation performance drop.

Original: An concern will694 be raised here as the pruned supernet is not trained to optimal
Revised: An concern will be raised here as the pruned supernet is not trained to optimal

Original: [Table]
Revised: [Table]

Original: and it is inaccurate to measure the performancedrop by L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: and it is inaccurate to measure the performance drop by L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: More importantly, since the skip-connection ’s magnitude is generally muchhigher than other candidate operations, and directly removing skip-connections usually greatly deteriorates the697 supernet performance if we do not fine-tune the supernet.
Revised: More importantly, since the skip-connection ’s magnitude is generally much higher than other candidate operations [Table] and directly removing skip-connections usually greatly deteriorates the supernet performance if we do not fine-tune the supernet.

Original: This is also the reason why we empirically find that DARTS-PT still prefers skip-connection than other operations.
Revised: This is also the reason why we empirically find that DARTS-PT still prefers skip-connection than other operations.

Original: Figure 8 (a) plots the searched architecture in699 NAS-Bench-201 for DARTS-PT, which contains 3 skip-connections .
Revised: Figure 8 (a) plots the searched architecture in NAS-Bench-201 for DARTS-PT, which contains 3 skip-connections .

Original: In contrast, the architecture selected by ourinfluential magnitude only contains 1 skip-connection .
Revised: In contrast, the architecture selected by our influential magnitude only contains 1 skip-connection .

Original: This phenomenon also exists in the experiments on the701 search space S1 and S2 of [45], where although DARTS-PT can avoid selecting all skip-connections in all edges,its searched architectures still contains intensive skip-connections .
Revised: This phenomenon also exists in the experiments on the search space S1 and S2 of [45], where although DARTS-PT can avoid selecting all skip-connections in all edges, its searched architectures still contains intensive skip-connections .

Original: D Comparison with SOTAs on NAS-Bench-201 
Revised: D Comparison with SOTAs on NAS-Bench-

Original: We also compared our DARTS-IM with different NAS baselines on the NAS-Bench-201 search space.
Revised: We also compared our DARTS-IM with different NAS baselines on the NAS-Bench-201 search space.

Original: Table 4705 reports the comprehensive validation and test results on all three datasets, including CIFAR-10, CIFAR-100,and ImageNet.
Revised: Table 4 reports the comprehensive validation and test results on all three datasets, including CIFAR-10, CIFAR-100, and ImageNet.

Original: As shown, DARTS is not a valid method for differentiable architecture search, which even707 leads to poorer results than random baseline.
Revised: As shown, DARTS is not a valid method for differentiable architecture search, which even leads to poorer results than random baseline.

Original: Although considering other supernet training methods can alsoimprove the performance, we can also found that, with only replacing the operation selection metric with our709 influential magnitude, our DARTS-IM can achieve more competitive results.
Revised: Although considering other supernet training methods can also improve the performance, we can also found that, with only replacing the operation selection metric with our influential magnitude, our DARTS-IM can achieve more competitive results.

Original: Moreover, our DARTS-IM achievesnear-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR-100, and ImageNet,711 respectively, in a single run.
Revised: Moreover, our DARTS-IM achieves near-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR-100, and ImageNet, respectively, in a single run.

Original: Our DARTS-IF, which follows the perturbation-based selection as DARTS-PTwhile with an additional approximation term, also outperforms DARTS-PT that again verifies the effectiveness713 of the influence function explanation.
Revised: Our DARTS-IF, which follows the perturbation-based selection as DARTS-PT while with an additional approximation term, also outperforms DARTS-PT that again verifies the effectiveness of the influence function explanation.

Original: In addition, due to the requirement of fine-tuning in the perturbation-basedparadigm, DARTS-PT cost much more than our DARTS-IM.
Revised: In addition, due to the requirement of fine-tuning in the perturbation-based paradigm, DARTS-PT cost much more than our DARTS-IM.

Original: Compared with the perturbation-based selection715 paradigm, we found that the magnitude paradigm is more efficient and reliable 1 .
Revised: Compared with the perturbation-based selection paradigm, we found that the magnitude paradigm is more efficient and reliable 2 .

Original: E Comparison with SOTAs on DARTS space
Revised: E Comparison with SOTAs on DARTS space

Original: DARTS search space is a much more complicated search space than NAS-Bench-201 and NAS-Bench-1shot1,[-
-] and it is intractable to get the ground truth for all architectures in this space.
Revised: DARTS search space is a much more complicated search space than NAS-Bench-201 and NAS-Bench-1shot1,[-
-] and it is intractable to get the ground truth for all architectures in this space.

Original: Generally, most existing worksonly report the best searched architecture, making the reproducibility of architecture search weak in this space.
Revised: Generally, most existing works only report the best searched architecture, making the reproducibility of architecture search weak in this space.

Original: For fair comparison, we also follow the common setting [27, 43] to conduct the architecture search on thesupernet
Revised: For fair comparison, we also follow the common setting [27, 43] to conduct the architecture search on the

Original: [Table]
Revised: [Table]

Original: “Param" is the model size when applied on CIFAR-10, while “ + × " (the number of multiply-add operations) is calculated based on the ImageNet dataset.
Revised: “Param" is the model size when applied on CIFAR-10, while “ + × " (the number of multiply-add operations) is calculated based on the ImageNet dataset.

Original: In our experiments, we adjust the number of initial filters to restrict “ + × " to be less than 600M on the ImageNet and CIFAR10.
Revised: In our experiments, we adjust the number of initial filters to restrict “ + × " to be less than 600M on the ImageNet and CIFAR10.

Original: † means the computational time is calculated with the same environment as [40].
Revised: † means the computational time is calculated with the same environment as [40].

Original: The unit in “Search Cost" is
Revised: The unit in “Search Cost" is

Original: GPU day.
Revised: GPU day.

Revised: [Table]

Revised: is

Original: [Table]
Revised: [Table]

Revised: on CIFAR-

Revised: [Table]

Revised: [Table]

Revised: default

Revised: [Table]

Revised: the number of initial

Revised: [Table]

Revised: it to restrict the number of multiply-add operations (

Original:  ImageNet and to comparable model size with SOTAs on CIFAR10.
Revised: [Table] + × ") to be less [Table] 600M on the ImageNet and to comparable model size with SOTAs on CIFAR10.

Original: Table 5 presents the comparison results with the SOTA differentiable NAS methods.
Revised: Table 5 presents the comparison results with the SOTA differentiable NAS methods.

Original: As shown, our DARTS-IM726 improves the DARTS baselines by a large margin, in terms of test error on CIFAR-10, CIFAR-100, and ImageNet,respectively, again verifying the effectiveness of the proposed influential magnitude.
Revised: As shown, our DARTS-IM improves the DARTS baselines by a large margin, in terms of test error on CIFAR-10, CIFAR-100, and ImageNet, respectively, again verifying the effectiveness of the proposed influential magnitude.

Original: More interesting, instead of728 changing the complicated supernet training part, we found that simply replacing the magnitude with the proposedinfluential magnitude could consistently improve the performance of DARTS with only additional hundred730 seconds computational time.
Revised: More interesting, instead of changing the complicated supernet training part, we found that simply replacing the magnitude with the proposed influential magnitude could consistently improve the performance of DARTS with only additional hundred seconds computational time.

Original: Different from the DARTS-PT which contains the supernet retraining that greatlyincreases the search time, our DARTS-IM only poses additional hundred seconds in DARTS space.
Revised: Different from the DARTS-PT which contains the supernet retraining that greatly increases the search time, our DARTS-IM only poses additional hundred seconds in DARTS space.

Original: We could732 also find that, although the best CIFAR-10 test error by DARTS-PT is comparable with our DARTS-IM, our bestsearched architecture obtains lower test error in CIFAR-100 and ImageNet, showing better transferability.
Revised: We could also find that, although the best CIFAR-10 test error by DARTS-PT is comparable with our DARTS-IM, our best searched architecture obtains lower test error in CIFAR-100 and ImageNet, showing better transferability.

Original: F Analyse the inverse Hessian approximation
Revised: F Analyse the inverse Hessian approximation

Original: In this section, we detailed discuss the effects of different inverse Hessian approximation methods on the736 proposed influence magnitude.
Revised: In this section, we detailed discuss the effects of different inverse Hessian approximation methods on the proposed influence magnitude.

Original: Apart from the two devised method, Neumann series and Sherman-Morrisonformula, we also consider two commonly-used approaches in approximating the inverse Hessian, identity738 approximation and diagonal approximation, which consider the Hessian matrix as an identity matrix anddiagonal matrix, respectively.
Revised: Apart from the two devised method, Neumann series and Sherman-Morrison formula, we also consider two commonly-used approaches in approximating the inverse Hessian, identity approximation and diagonal approximation, which consider the Hessian matrix as an identity matrix and diagonal matrix, respectively.

Original: We conduct a series of experiments on NAS-Bench-201 to analyze different740 approximation methods, and all comparison results are reported in Table 6.
Revised: We conduct a series of experiments on NAS-Bench-201 to analyze different approximation methods, and all comparison results are reported in Table 6.

Original: As shown, even with the identityapproximation, our influential magnitude I M can achieve satisfying results compared with the random baselineand two heuristic methods.
Revised: As shown, even with the identity approximation, our influential magnitude I M can achieve satisfying results compared with the random baseline and two heuristic methods.

Original: This result indicates that the second-order information ∂ 2 L ( θ ∗ ,α ) ∂α∂θ and ∂ 2 L ( θ ∗ ,α ) ∂θ∂α can743 provide useful information for operation strength estimation.
Revised: This result indicates that the second-order information ∂ 2 L ( θ ∗ ,α ) ∂α∂θ and ∂ 2 L ( θ ∗ ,α ) ∂θ∂α can provide useful information for operation strength estimation.

Original: More interesting, when considering the diagonal
Revised: More interesting, when considering the diagonal

Original: Hessian information, we found that our DARTS-IM-D could also achieve excellent performance.
Revised: Hessian information, we found that our DARTS-IM-D could also achieve excellent performance.

Original: The results of
Revised: The results of

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: identity approximation and diagonal approximation demonstrate the robustness of our influential magnitude,746 which can still achieve excellent results with different inverse Hessian approximation methods.
Revised: identity approximation and diagonal approximation demonstrate the robustness of our influential magnitude, which can still achieve excellent results with different inverse Hessian approximation methods.

Original: However, we also found that an incorrect approximation may lead to poor performance.
Revised: However, we also found that an incorrect approximation may lead to poor performance.

Original: Similar to Figure 2,748 we plots the full NAS-Bench-201 results of our influential magnitude under different inverse Hessian matrixapproximation methods along with the number of batches N , on the CIFAR10, CIFAR100, and ImageNet,750 respectively.
Revised: Similar to Figure 2, we plots the full NAS-Bench-201 results of our influential magnitude under different inverse Hessian matrix approximation methods along with the number of batches N , on the CIFAR10, CIFAR100, and ImageNet, respectively.

Original: We compared our DARTS-IM with the DARTS baseline and also the DARTS-PT whose results aredirectly obtained from [40].
Revised: We compared our DARTS-IM with the DARTS baseline and also the DARTS-PT whose results are directly obtained from [40].

Original: Please note that, DARTS-PT smooth the line extremely with only report the tendency752 of the architecture selection during the search.
Revised: Please note that, DARTS-PT smooth the line extremely with only report the tendency of the architecture selection during the search.

Original: We can found that the Neumann series approximation could notobtain stable results, especially when the number of batches increases that the performance of Neumann series754 even performs more poorly than identity approximation.
Revised: We can found that the Neumann series approximation could not obtain stable results, especially when the number of batches increases that the performance of Neumann series even performs more poorly than identity approximation.

Original: As discussed in Section 6.1, these results also verifythat the stochastic Neumann series estimation for IHVP is somehow erroneous.
Revised: As discussed in Section 6.1, these results also verify that the stochastic Neumann series estimation for IHVP is somehow erroneous.

Original: Generally, we found that the756 Sherman-Morrison formula is the most practical in estimating the inverse Hessian for our influential magnitude.
Revised: Generally, we found that the Sherman-Morrison formula is the most practical in estimating the inverse Hessian for our influential magnitude.

Original: Different from Sherman-Morrison formula that only has a hyperparameter N to be tuned and η is fixed in the758 optimizer, the Neumann series approximation has a key hyperparameter γ [29, 36] to be tuned, which is assumed
Revised: Different from Sherman-Morrison formula that only has a hyperparameter N to be tuned and η is fixed in the optimizer, the Neumann series approximation has a key hyperparameter γ [29, 36] to be tuned, which is assumed

Original: [Figure]
Revised: [Figure]

Original: to be small enough that γ < 1 σ max in Appendix B.1.
Revised: to be small enough that γ < 1 σ max in Appendix B.1.

Original: In this way, we conduct the hyperparameter analysis on γfor our DARTS-IM when employing the Neumann series approximation.
Revised: In this way, we conduct the hyperparameter analysis on γ for our DARTS-IM when employing the Neumann series approximation.

Original: Figure 6 compares the performance of761 DARTS-IM with different γ on the NAS-Bench-201.
Revised: Figure 6 compares the performance of DARTS-IM with different γ on the NAS-Bench-201.

Original: As shown, the Neumann series approximation is sensitiveto the γ , where a large γ significantly deteriorates the performance of DARTS-IM.
Revised: As shown, the Neumann series approximation is sensitive to the γ , where a large γ significantly deteriorates the performance of DARTS-IM.

Original: For example, a γ = 0 .{+
+}
Revised: For example, a γ = {+
+}

Original: 763 even makes our DART-IM similar to the performance of the random baseline.
Revised: .even makes our DART-IM similar to the performance of the random baseline.

Original: Compared with the Neumannseries, we found that the Sherman-Morrison approximation is more robust.
Revised: Compared with the Neumann series, we found that the Sherman-Morrison approximation is more robust.

Original: G Visualization of Searched Architectures on Different Spaces
Revised: G Visualization of Searched Architectures on Different Spaces

Original: In Figure 8 and 9, we visualize all searched architectures by DARTS baselines, DARTS-PT and also our767 DARTS-IM on different search space, including NAS-Bench-201 [12], tool search spaces proposed by [45], and
Revised: In Figure 8 and 9, we visualize all searched architectures by DARTS baselines, DARTS-PT and also our DARTS-IM on different search space, including NAS-Bench-201 [12], tool search spaces proposed by [45], and

Original: DARTS search space [27].
Revised: DARTS search space [27].

Original: First, we analyze these architectures searched on NAS-Bench-201 [12] and tool search spaces proposed by770 [40, 45], since which helps us to clearly demonstrate the robustness of different architecture selection methods.
Revised: First, we analyze these architectures searched on NAS-Bench-201 [12] and tool search spaces proposed by [40, 45], since which helps us to clearly demonstrate the robustness of different architecture selection methods.

Original: As observed by [40, 45], the most notable drawback of DARTS baseline is the robustness and generalization,772 where DARTS fails to select intensive skip-connections with the search progressing.
Revised: As observed by [40, 45], the most notable drawback of DARTS baseline is the robustness and generalization, where DARTS fails to select intensive skip-connections with the search progressing.

Original: Figure 8 (a-c) plots thesearched architectures on NAS-Bench-201, respectively.
Revised: Figure 8 (a-c) plots the searched architectures on NAS-Bench-201, respectively.

Original: As shown, the DARTS baseline selects skip-connection774 for all edges, while DARTS-PT can find a valid architecture with only 3 skip-connections when considering aperturbation-based selection paradigm.
Revised: As shown, the DARTS baseline selects skip-connection for all edges, while DARTS-PT can find a valid architecture with only 3 skip-connections when considering a perturbation-based selection paradigm.

Original: More inspiring, the proposed influential magnitude ( I M ) can select more776 competitive architecture which also only contains one skip-connection .
Revised: More inspiring, the proposed influential magnitude ( I M ) can select more competitive architecture which also only contains one skip-connection .

Original: Please note that, the trained supernetkeeps identical for DARTS, DARTS-PT, and our DARTS-IM, which only adopt different architecture selection778 methods, e.g., argmax , perturbation-based selection, and the proposed influential magnitude.
Revised: Please note that, the trained supernet keeps identical for DARTS, DARTS-PT, and our DARTS-IM, which only adopt different architecture selection methods, e.g., argmax , perturbation-based selection, and the proposed influential magnitude.

Original: Similarly, we can also observe this superiority of our I M in the tool search spaces proposed by [40, 45],780 especially in the search space S1 and S2.
Revised: Similarly, we can also observe this superiority of our I M in the tool search spaces proposed by [40, 45], especially in the search space S1 and S2.

Original: We can find that, our DARTS-IM clearly outperforms DARTS and
Revised: We can find that, our DARTS-IM clearly outperforms DARTS and

Original: DARTS-PT.
Revised: DARTS-PT.

Original: Although DARTS-PT can partially relieves this instability, it still selects intensive skip-connections .
Revised: Although DARTS-PT can partially relieves this instability, it still selects intensive skip-connections .

Original: In the contrary, our DARTS-IM generally prefers convolutional operations.
Revised: In the contrary, our DARTS-IM generally prefers convolutional operations.

Original: However, although our DARTS-IMoutperforms DARTS baseline in the search space S3, we found that DARTS-PT achieves excellent results in784 this space.
Revised: However, although our DARTS-IM outperforms DARTS baseline in the search space S3, we found that DARTS-PT achieves excellent results in this space.

Original: One potential is that this space contains intensive none operations which is naturally removed byperturbation-based selection, since removing none will not affect the supernet performance.
Revised: One potential is that this space contains intensive none operations which is naturally removed by perturbation-based selection, since removing none will not affect the supernet performance.

Original: We also conduct the786 architecture search on the search space S4, while we found that our DARTS-IM also performs poorly as DARTSthat select some noise operations.
Revised: We also conduct the architecture search on the search space S4, while we found that our DARTS-IM also performs poorly as DARTS that select some noise operations.

Original: One underlying reason is that it is impossible to train the supernet to optimal788 in this space since the noise operation will return a random noise regardless of the input.
Revised: One underlying reason is that it is impossible to train the supernet to optimal in this space since the noise operation will return a random noise regardless of the input.

Original: In addition, DARTS-PTonly locally compares the operations’ strength in each edge, and a fine-tuning is needed after every step of edge790 discretization.
Revised: In addition, DARTS-PT only locally compares the operations’ strength in each edge, and a fine-tuning is needed after every step of edge discretization.

Original: Differently, our DARTS-IM in this paper consider a global comparison among all operations in asupernet, which may bring some negative operation coupling.
Revised: Differently, our DARTS-IM in this paper consider a global comparison among all operations in a supernet, which may bring some negative operation coupling.

Original: To further improve our proposed DARTS-IM,792 a simple and straight approach is to gradually discretize edges in a supenet with fine-tuning, which we leave
Revised: To further improve our proposed DARTS-IM, a simple and straight approach is to gradually discretize edges in a supenet with fine-tuning, which we leave

Original: [Table]
Revised: [Table]

Original: our future work.
Revised: future work.

Original: We also show the comparison results in the Table 7 with different DARTS baselines.
Revised: We also show the comparison results in the Table 7 with different DARTS baselines.

Original: We794 directly report the results of DARTS and DARTS-ES from [45], which contain the mean and std.
Revised: We directly report the results of DARTS and DARTS-ES from [45], which contain the mean and std.

Original: The results of
Revised: The results of

Original: DARTS-PT is also inherited from the original paper [40], which only report the best single run.
Revised: DARTS-PT is also inherited from the original paper [40], which only report the best single run.

Original: In Table 7, we796 report the average and best single run of our DARTS-IM together for better comparison.
Revised: In Table 7, we report the average and best single run of our DARTS-IM together for better comparison.

Original: Then, we also visualized our searched architectures in the DARTS search space.
Revised: Then, we also visualized our searched architectures in the DARTS search space.

Original: Interesting, compared with the798 DARTS baseline, architecture searched by our DARTS-IM contains several sep_conv 5 × 5 operations in thenormal cell, where this phenomenon has never been observed by existing popular DARTS based algorithms.
Revised: Interesting, compared with the DARTS baseline, architecture searched by our DARTS-IM contains several sep_conv 5 × 5 operations in the normal cell, where this phenomenon has never been observed by existing popular DARTS based algorithms.

Original: Similarly, we also found that our reduction cell contains several sep_conv 5 × 5 operations, which is also inline with another influence based method DARTS-PT and PC-DARTS.
Revised: Similarly, we also found that our reduction cell contains several sep_conv 5 × 5 operations, which is also in line with another influence based method DARTS-PT and PC-DARTS.

Original: More important, different from most802 existing works that the normal cell generally contains more parameters than the reduction cell, we found that oursearched architecture prefers more parameters in the reduction cell.
Revised: More important, different from most existing works that the normal cell generally contains more parameters than the reduction cell, we found that our searched architecture prefers more parameters in the reduction cell.

Original: This observation is more consistent with the804 common experience in the network pruning [24, 25, 39], where we found that most pruning methods will retainmore parameters in the reduction cell and prune more redundant parameters in the normal cell.
Revised: This observation is more consistent with the common experience in the network pruning [24, 25, 39], where we found that most pruning methods will retain more parameters in the reduction cell and prune more redundant parameters in the normal cell.

Original: H Overall Algorithm Framework Description
Revised: H Overall Algorithm Framework Description

Original: In this paper, we proposed two different approaches for the operation selection in DARTS, called DARTS-IF808 (described in Sec.3) and DARTS-IM (described in Sec.4).
Revised: In this paper, we proposed two different approaches for the operation selection in DARTS, called DARTS-IF (described in Sec.3) and DARTS-IM (described in Sec.4).

Original: This two approaches are based on the influencefunctions to measure the operation importance, where the main different is that DARTS-IF removes a candidate810 operation in each step to measure the loss change in a perturbation manner, while DARTS-IM calculate theloss sensitivity when applying an infinitesimal change on α .
Revised: This two approaches are based on the influence functions to measure the operation importance, where the main different is that DARTS-IF removes a candidate operation in each step to measure the loss change in a perturbation manner, while DARTS-IM calculate the loss sensitivity when applying an infinitesimal change on α .

Original: The framework of our DARTS-IF and DART-IM812 are sketched in Algorithm 1 and 2.
Revised: The framework of our DARTS-IF and DART-IM are sketched in Algorithm 1 and 2.

Original: Our DARTS-IF shares a similar perturbation paradigm as DARTS-PT,while without any fine-tuning.
Revised: Our DARTS-IF shares a similar perturbation paradigm as DARTS-PT, while without any fine-tuning.

Original: Rather than using the validation performance drop to indicate the operation814 importance, DARTS-IF leverages influence functions to predict the loss change based on Eq.
Revised: Rather than using the validation performance drop to indicate the operation importance, DARTS-IF leverages influence functions to predict the loss change based on Eq.

Original: In our practicalimplementation, we consider the Neumann series as described in Lemma 1, or the Sherman-Morrison formulaas described in Lemma 2, to approximate the IHPV ∂
Revised: In our practical implementation, we consider the Neumann series as described in Lemma 1, or the Sherman-Morrison formula as described in Lemma 2, to approximate the IHPV ∂

Original: 2 L ( θ ∗ , ˆ α ) 
Revised: 2 L ( θ ∗ , ˆ α ) ∂θ∂θ

Original: [Table] [Table]
Revised: [Equation]

Original: More important, different from817 DARTS-PT that gradually discretize edges and finetune the supernet, our DARTS-IF only approximates theeffect of independently removing one candidate operation for each edge.
Revised: More important, different from DARTS-PT that gradually discretize edges and finetune the supernet, our DARTS-IF only approximates the effect of independently removing one candidate operation for each edge.

Original: In this way, we can get the strength for819 all candidate operations under every edge in a supernet.
Revised: In this way, we can get the strength for all candidate operations under every edge in a supernet.

Original: Different from DARTS-IF and DARTS-PT, our DARTS-IM is with a similar paradigm as DARTS, while821 whose operation importance is calculated based on I M , rather that the optimized α .
Revised: Different from DARTS-IF and DARTS-PT, our DARTS-IM is with a similar paradigm as DARTS, while whose operation importance is calculated based on I M , rather that the optimized α .

Original: In addition, the operationimportance in our DARTS-IM can be calculated by one-shot, rather than iteratively going through all candidates823 operations in a supernet as perturbation paradigm.
Revised: In addition, the operation importance in our DARTS-IM can be calculated by one-shot, rather than iteratively going through all candidates operations in a supernet as perturbation paradigm.

Original: In calculating the influence magnitude in our DARTS-IM or the loss change in DARTS-IF, we have three825 more hyperparameters, γ for Neumann series approximation, η for Sherman-Morrison approximation, andbatch size N for both.
Revised: In calculating the influence magnitude in our DARTS-IM or the loss change in DARTS-IF, we have three more hyperparameters, γ for Neumann series approximation, η for Sherman-Morrison approximation, and batch size N for both.

Original: In our experiments, we set γ = 0 .
Revised: In our experiments, we set γ = 0 .

Original: 001 that is same as the learning rate in optimizing827 θ , and η is the weight decay for regularization, which both can be obtained form the optimizer by default.
Revised: 001 that is same as the learning rate in optimizing θ , and η is the weight decay for regularization, which both can be obtained form the optimizer by default.

Original: In the hyperparameter analysis in Figure 2, we found that N = 10 is enough for both Neumann series and829 Sherman-Morrison approximation.
Revised: In the hyperparameter analysis in Figure 2, we found that N = 10 is enough for both Neumann series and Sherman-Morrison approximation.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: dil_conv_3x skip_connec t skip_connect c_{k} skip_
Revised: dil_conv_3x skip_connec t skip_connect c_{k} skip_

Original: [Figure]
Revised: [Figure]

Original: 3 dil_conv_3x dil_conv_5x c_{k- 2 } c_{k-1}
Revised: 3 dil_conv_3x dil_conv_5x c_{k- 2 } c_{k-1}

Original: [Figure]
Revised: [Figure]

Original: output
Revised: output

Original: discrete
Revised: discrete

Original: do
Revised: do

Revised: do 5: candidate

Original: Calculate the predictive loss chance ∆ L  based on Eq
Revised: Calculate the predictive loss chance ∆ L o,e based on Eq

Original: ), that ∆ L o,e ≈ L ( θ  ˆ α ) −
Revised: ), that ∆ L o,e ≈ L ( θ ∗ , ˆ α ) −

Original: L ( θ ∗ , α ) − 1 / 2 ∂
Revised: L ( θ ∗ , α ) − 1 / 2 ∂

Original: L ( θ ∗ , ˆ α ) ∂θ
Revised: L ( θ ∗ , ˆ α ) ∂θ

Original: T ∂ 2 L ( θ ∗ , ˆ α )
Revised: T ∂ 2 L ( θ ∗ , ˆ α )

Original: [Equation]
Revised: [Equation]

Original: − 1 ∂ L ( θ ∗ , ˆ α ) ∂θ
Revised: − 1 ∂ L ( θ ∗ , ˆ α ) ∂θ

Original: Restore 
Revised: Restore o to

Original: 8: end  9: end  10: Apply argmax on the operation strength ∆  derive  discrete architecture  accordingly.
Revised: 8: end for 9: end for 10: Apply argmax on the operation strength ∆ L and derive the discrete architecture α accordingly.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: sep
Revised: sep

Original: [Figure]
Revised: [Figure]

Original: sep
Revised: sep

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: c_{
Revised: c_{

Original: [Figure]
Revised: [Figure]

Original: c_{k-1} sep_conv_5x 1 max_pool_3x  dil_conv_5x skip_connect 3 sep_conv_5x c_{k} max_pool_3x skip_connect (f) DARTS-PT: Reduction Cell c_{k-2} skip_connect 1 sep_conv_3x3sep_conv_5x5avg_pool_3x c_{k-1} sep_conv_3x dil_conv_3x3 sep_conv_3x dil_conv_3x
Revised: c_{k-1} sep_conv_5x 1 max_pool_3x  dil_conv_5x skip_connect 3 sep_conv_5x c_{k} max_pool_3x skip_connect (f) DARTS-PT: Reduction Cell c_{k-2} skip_connect 1 sep_conv_3x3sep_conv_5x5avg_pool_3x c_{k-1} sep_conv_3x dil_conv_3x3 sep_conv_3x dil_conv_3x

Original: c_{k} (g) PC-DARTS:
Revised: c_{k} (g) PC-DARTS:

Original: Normal Cell c_{k-2} max_pool_3x  sep_conv_3x c_{k-1} sep_conv_5x  sep_conv_5x 3 sep_conv_3x sep_conv_5x sep_conv_3x c_{k} sep_conv_3x (h) PC-DARTS:
Revised: Normal Cell c_{k-2} max_pool_3x  sep_conv_3x c_{k-1} sep_conv_5x  sep_conv_5x 3 sep_conv_3x sep_conv_5x sep_conv_3x c_{k} sep_conv_3x (h) PC-DARTS:

Original: Reduction Cell
Revised: Reduction Cell