Original: Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations
Revised: Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations

Original: Abstract
Revised: Abstract

Original: The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity.
Revised: The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity.

Original: DARTSleverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handledwith gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes.
Revised: DARTSleverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handledwith gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes.

Original: The vanilla DARTS assumes the optimized magnitudes reflect the importanceof operations, while more recent works find this naive assumption leads to poorgeneralization and is without any theoretical guarantees.
Revised: The vanilla DARTS assumes the optimized magnitudes reflect the importanceof operations, while more recent works find this naive assumption leads to poorgeneralization and is without any theoretical guarantees.

Original: In this work, we leverageinfluence functions, the functional derivatives of the loss function, to theoreticallyreveal the operation selection part in DARTS and estimate the candidate operationimportance by approximating its influence on the supernet with Taylor expansions.
Revised: In this work, we leverageinfluence functions, the functional derivatives of the loss function, to theoreticallyreveal the operation selection part in DARTS and estimate the candidate operationimportance by approximating its influence on the supernet with Taylor expansions.

Original: We show the operation strength is not only related to the magnitude but also secondorder information, leading to a fundamentally new criterion for operation selectionin DARTS, named Influential Magnitude .
Revised: We show the operation strength is not only related to the magnitude but also secondorder information, leading to a fundamentally new criterion for operation selectionin DARTS, named Influential Magnitude .

Original: Empirical studies across different taskson several spaces show that vanilla DARTS and its variants can avoid most failuresby leveraging the proposed theory-driven operation selection criterion.
Revised: Empirical studies across different taskson several spaces show that vanilla DARTS and its variants can avoid most failuresby leveraging the proposed theory-driven operation selection criterion.

Original:  Introduction
Revised:  Introduction

Original: Neural Architecture Search (NAS) has been successfully applied to automating the process of neuralnetwork design for a broad of deep learning fields [35].
Revised: Neural Architecture Search (NAS) has been successfully applied to automating the process of neuralnetwork design for a broad of deep learning fields [35].

Original: However, the early NAS methods areoften heavily computational-expensive and require hundreds or even thousands GPU days [34].
Revised: However, the early NAS methods areoften heavily computational-expensive and require hundreds or even thousands GPU days [34].

Original: Toimprove the efficiency for more practical applications, a lot of recent works are proposed based onone-shot (also known as weight-sharing) paradigm [33] to improve the search efficiency significantly.
Revised: Toimprove the efficiency for more practical applications, a lot of recent works are proposed based onone-shot (also known as weight-sharing) paradigm [33] to improve the search efficiency significantly.

Original: Differentiable ARchiTecture Search (DARTS) [27], as the most popular one-shot NAS method, hasdominated this area since its appearance.
Revised: Differentiable ARchiTecture Search (DARTS) [27], as the most popular one-shot NAS method, hasdominated this area since its appearance.

Original: DARTS leverages continuous relaxation to convert theintractable operation selection problem into a continuous magnitude optimization problem, whichcan be gracefully solved by gradient-descent based on the bi-level optimization [8].
Revised: DARTS leverages continuous relaxation to convert theintractable operation selection problem into a continuous magnitude optimization problem, whichcan be gracefully solved by gradient-descent based on the bi-level optimization [8].

Original: A discretearchitecture is then derived based on the optimized magnitudes through operation selection.
Revised: A discretearchitecture is then derived based on the optimized magnitudes through operation selection.

Original: Although the continuous relaxation makes DARTS able to optimize the magnitudes with rigoroustheoretical formulation and guarantees [49] in the supernet training part, it poses an additionalchallenge in selecting a discrete architecture from the optimized magnitudes that there is no existingwork can give a theoretical explanation.
Revised: Although the continuous relaxation makes DARTS able to optimize the magnitudes with rigoroustheoretical formulation and guarantees [49] in the supernet training part, it poses an additionalchallenge in selecting a discrete architecture from the optimized magnitudes that there is no existingwork can give a theoretical explanation.

Original: DARTS is based on the weight-sharing paradigm [33],which assumes that the derived discrete architecture with the trained supernet weights can predictits train-from-scratch performance.
Revised: DARTS is based on the weight-sharing paradigm [33],which assumes that the derived discrete architecture with the trained supernet weights can predictits train-from-scratch performance.

Original: However, there is a gap between the two groups of weights,as depicted in Figure 1 in Appendix.
Revised: However, there is a gap between the two groups of weights,as depicted in Figure 1 in Appendix.

Original: The vanilla DARTS assumes the optimized magnitudes can
Revised: The vanilla DARTS assumes the optimized magnitudes can

Original: [Figure]
Revised: [Figure]

Original: reflect the importance of the candidate operations, while Wang et al.
Revised: reflect the importance of the candidate operations, while Wang et al.

Original: [40] empirically show thatthe operation magnitude does not necessarily indicate how important the operation is in a trainedsupernet.
Revised: [40] empirically show thatthe operation magnitude does not necessarily indicate how important the operation is in a trainedsupernet.

Original: For example, DARTS prefers skip-connection [45] as its magnitude always dominatesthe remaining operations, while an architecture with intensive skip-connections generally leads topoor performance.
Revised: For example, DARTS prefers skip-connection [45] as its magnitude always dominatesthe remaining operations, while an architecture with intensive skip-connections generally leads topoor performance.

Original: Accordingly, Wang et al.
Revised: Accordingly, Wang et al.

Original: [40] proposed a heuristic perturbation-based method,
Revised: [40] proposed a heuristic perturbation-based method,

Original: DARTS-PT, to measure each candidate operation’s influence on the supernet by alternatively removingeach one to calculate the supernet performance drop.
Revised: DARTS-PT, to measure each candidate operation’s influence on the supernet by alternatively removingeach one to calculate the supernet performance drop.

Original: Despite DARTS-PT can partially relieve thefailure of DARTS, we empirically found that DARTS-PT still prefers skip-connection than otheroperations.
Revised: Despite DARTS-PT can partially relieve thefailure of DARTS, we empirically found that DARTS-PT still prefers skip-connection than otheroperations.

Original: One potential reason is that skip-connection ’s magnitude is much higher than othercandidate operations, and directly removing skip-connections usually greatly deteriorates the supernetperformance.
Revised: One potential reason is that skip-connection ’s magnitude is much higher than othercandidate operations, and directly removing skip-connections usually greatly deteriorates the supernetperformance.

Original: In addition, DARTS-PT lacks theoretical explanations and guarantees.
Revised: In addition, DARTS-PT lacks theoretical explanations and guarantees.

Original: For example,the intuitive way to show the importance of a candidate operation importance is the leave-one-outretraining [22], while which should fine-tune the supernet to get the performance drop after removingevery operation rather than directly measuring the performance drop as DARTS-PT, since there alsoexists a gap between the two groups of weights, as visualized in Figure 1.
Revised: For example,the intuitive way to show the importance of a candidate operation importance is the leave-one-outretraining [22], while which should fine-tune the supernet to get the performance drop after removingevery operation rather than directly measuring the performance drop as DARTS-PT, since there alsoexists a gap between the two groups of weights, as visualized in Figure 1.

Original: Section 3 and Appendix Calso give an analysis on why perturbation-based method fails.
Revised: Section 3 and Appendix Calso give an analysis on why perturbation-based method fails.

Original: This paper focuses on filling the gap of theoretical explanations for the discrete operation selection partin DARTS.
Revised: This paper focuses on filling the gap of theoretical explanations for the discrete operation selection partin DARTS.

Original: Instead of considering a heuristic approach to measure the operation importance [27, 40],this paper first rigorously shows how the architecture parameters discretization poses an influenceon the supernet weights in DARTS, and estimates the supernet performance drop accordingly.
Revised: Instead of considering a heuristic approach to measure the operation importance [27, 40],this paper first rigorously shows how the architecture parameters discretization poses an influenceon the supernet weights in DARTS, and estimates the supernet performance drop accordingly.

Original: Ourmethod is motivated by influence function [22], which is originally designed to understand the effectof removing an individual training point, while we leverage Taylor’s approximation to interpret thesupernet performance drop when removing a candidate operation.
Revised: Ourmethod is motivated by influence function [22], which is originally designed to understand the effectof removing an individual training point, while we leverage Taylor’s approximation to interpret thesupernet performance drop when removing a candidate operation.

Original: In this way, we show the operationstrength is not only related to the magnitude but also second-order information, and we accordinglydesign a new criterion for the operation selection in DARTS, called Influential Magnitude ( I M ).
Revised: In this way, we show the operationstrength is not only related to the magnitude but also second-order information, and we accordinglydesign a new criterion for the operation selection in DARTS, called Influential Magnitude ( I M ).

Original: Then, to avoid calculating the inverse of the Hessian matrix with respect to the model weights,we further utilize the Neumann series [29] and Sherman-Morrison formula [37] to approximatethe Inverse-Hessian-Vector Products (IHVPs), making the proposed method applicable to DARTS.
Revised: Then, to avoid calculating the inverse of the Hessian matrix with respect to the model weights,we further utilize the Neumann series [29] and Sherman-Morrison formula [37] to approximatethe Inverse-Hessian-Vector Products (IHVPs), making the proposed method applicable to DARTS.

Original: Finally, we verify the effectiveness of our method on several NAS benchmark datasets and thecommon DARTS search space.
Revised: Finally, we verify the effectiveness of our method on several NAS benchmark datasets and thecommon DARTS search space.

Original: To summarize, we make the following contributions:
Revised: To summarize, we make the following contributions:

Original: • This paper deepens our understanding of the operation selection in DARTS.
Revised: • This paper deepens our understanding of the operation selection in DARTS.

Original: We reformulate
Revised: We reformulate

Original: the operation selection in DARTS by approximating its influence on the supernet with Taylorexpansions, which can thus gracefully interpret how the validation performance changes
Revised: the operation selection in DARTS by approximating its influence on the supernet with Taylorexpansions, which can thus gracefully interpret how the validation performance changes

Original: when selecting different operations without any additional fine-tuning.
Revised: when selecting different operations without any additional fine-tuning.

Original: • We theoretically reveal the operation strength is not only related to the magnitude but also thesecond-order information, and accordingly derive a fundamentally new criterion to measure
Revised: • We theoretically reveal the operation strength is not only related to the magnitude but also thesecond-order information, and accordingly derive a fundamentally new criterion to measure

Original: the operation sensitivity, which we call Influential Magnitude ( I M ), for architecture
Revised: the operation sensitivity, which we call Influential Magnitude ( I M ), for architecture

Original: selection in DARTS.
Revised: selection in DARTS.

Original: Further, to make the proposed criterion practical, we devise severalmethods to estimate the Inverse-Hessian-Vector Products (IHVPs) in calculating the secondorder information.• Extensive experiments verify the effectiveness of the proposed criterion, which significantlyimproves the performance of vanilla DARTS and other baselines on the NAS benchmarkdatasets and the common DARTS search space.
Revised: Further, to make the proposed criterion practical, we devise severalmethods to estimate the Inverse-Hessian-Vector Products (IHVPs) in calculating the secondorder information.• Extensive experiments verify the effectiveness of the proposed criterion, which significantlyimproves the performance of vanilla DARTS and other baselines on the NAS benchmarkdatasets and the common DARTS search space.

Original: We show DARTS is still a strong NASbaseline when considering a more reasonable and theory-driven operation strength metric.
Revised: We show DARTS is still a strong NASbaseline when considering a more reasonable and theory-driven operation strength metric.

Original:  Preliminaries: DARTS and Influence Function
Revised:  Preliminaries: DARTS and Influence Function

Original: Typically, NAS focuses on finding a cell structure that is represented as a directed acyclic graph(DAG) with N nodes and E edges affiliated with operations, which is a discrete and hard-to-optimizeproblem.
Revised: Typically, NAS focuses on finding a cell structure that is represented as a directed acyclic graph(DAG) with N nodes and E edges affiliated with operations, which is a discrete and hard-to-optimizeproblem.

Original: To enable the gradient-descent for architecture search, DARTS [27] leverages softmax asthe continuous relaxation function to transform the architecture search into a continuous magnitudeoptimization problem:
Revised: To enable the gradient-descent for architecture search, DARTS [27] leverages softmax asthe continuous relaxation function to transform the architecture search into a continuous magnitudeoptimization problem:

Original: [Equation]
Revised: [Equation]

Original: where O contains all candidate operations and DARTS is to optimize the magnitude α o , which canbe formulated as a bi-level optimization problem [8]:
Revised: where O contains all candidate operations and DARTS is to optimize the magnitude α o , which canbe formulated as a bi-level optimization problem [8]:

Original: [Equation]
Revised: [Equation]

Original: where α is the continuous architecture representation and θ is the supernet weights.
Revised: where α is the continuous architecture representation and θ is the supernet weights.

Original: The nestedformulation in DARTS is the same as the gradient-based hyperparameter optimization with bi-leveloptimization, which has been validated in a broad applications [13, 31, 32].
Revised: The nestedformulation in DARTS is the same as the gradient-based hyperparameter optimization with bi-leveloptimization, which has been validated in a broad applications [13, 31, 32].

Original: After the bi-leveloptimization, DARTS usually considers heuristic methods to derive the final architecture, where themost popular one is to select the operations with the highest magnitudes, ˆ α = argmax ( α ) .
Revised: After the bi-leveloptimization, DARTS usually considers heuristic methods to derive the final architecture, where themost popular one is to select the operations with the highest magnitudes, ˆ α = argmax ( α ) .

Original: Despiteits simplicity, more recent works [7, 40, 45, 47, 48] found that DARTS could hardly find satisfactoryarchitectures.
Revised: Despiteits simplicity, more recent works [7, 40, 45, 47, 48] found that DARTS could hardly find satisfactoryarchitectures.

Original: Different from the majority of existing works that attribute the failure of DARTS to its magnitudeoptimization part, which is under a rigorous theoretical formulation [49], we revisit DARTS fromthe perspective of the architecture selection and try to reveal this part with theoretical explanations,where we aim to identify the operation that will contribute most to the supernet performance, throughthe lens of influence functions [22] to mathematically model the architecture selection process insteadof heuristics.
Revised: Different from the majority of existing works that attribute the failure of DARTS to its magnitudeoptimization part, which is under a rigorous theoretical formulation [49], we revisit DARTS fromthe perspective of the architecture selection and try to reveal this part with theoretical explanations,where we aim to identify the operation that will contribute most to the supernet performance, throughthe lens of influence functions [22] to mathematically model the architecture selection process insteadof heuristics.

Original: The influence function [18] is a classic technique from robust statistics that reveals how modelparameters change as we upweight or perturb a specific training sample, which has been applied inexplaining a lot of modern machine learning applications [2, 22, 30, 50].
Revised: The influence function [18] is a classic technique from robust statistics that reveals how modelparameters change as we upweight or perturb a specific training sample, which has been applied inexplaining a lot of modern machine learning applications [2, 22, 30, 50].

Original: More specifically, given amodel with parameters θ ∗ after trained on a full training set D , influence function aims to study thechange of parameters ˆ θ ∗ − θ ∗ when retraining the model parameters to ˆ θ ∗ after deleting a specifictraining point z , that ˆ D
Revised: More specifically, given amodel with parameters θ ∗ after trained on a full training set D , influence function aims to study thechange of parameters ˆ θ ∗ − θ ∗ when retraining the model parameters to ˆ θ ∗ after deleting a specifictraining point z , that ˆ D

Original: = D − z .
Revised: = D − z .

Original: Rather than retraining the model for every removed point, Koh and
Revised: Rather than retraining the model for every removed point, Koh and

Original: Liang [22] estimated the influence by upweighting the data point with some small ϵ .
Revised: Liang [22] estimated the influence by upweighting the data point with some small ϵ .

Original: With leveragingthe Taylor expansions, the influence on the model parameters can be estimated as:
Revised: With leveragingthe Taylor expansions, the influence on the model parameters can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: (2) and its influence on the loss function of a test point z test is accordingly calculated as:
Revised: (2) and its influence on the loss function of a test point z test is accordingly calculated as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The detailed derivation of the two influence functions can be found in [22].
Revised: The detailed derivation of the two influence functions can be found in [22].

Original: Based on Eq.(2) and (3),we can now approximate the validation performance drop when removing a training point z from thetraining set without conducting the time-consuming retraining.
Revised: Based on Eq.(2) and (3),we can now approximate the validation performance drop when removing a training point z from thetraining set without conducting the time-consuming retraining.

Original: The original work of influence functions for modern machine learning [22], which is also calledas first-order influence functions, only considers a very small change on θ ∗ as it only changes one training point, so the first-order Taylor expansion is satisfied to calculate the influence.
Revised: The original work of influence functions for modern machine learning [22], which is also calledas first-order influence functions, only considers a very small change on θ ∗ as it only changes one training point, so the first-order Taylor expansion is satisfied to calculate the influence.

Original: However,upweighting a group of data points may lead to a large change on θ ∗ [4, 5], which violates the smallperturbation assumption of the first-order influence functions.
Revised: However,upweighting a group of data points may lead to a large change on θ ∗ [4, 5], which violates the smallperturbation assumption of the first-order influence functions.

Original: To make the the influence functioncapture the model parameter change more precisely, Basu et al.
Revised: To make the the influence functioncapture the model parameter change more precisely, Basu et al.

Original: [4] extended influence functionsusing second-order approximations under the assumption that the third-order derivatives of the lossfunction at optimum is small.
Revised: [4] extended influence functionsusing second-order approximations under the assumption that the third-order derivatives of the lossfunction at optimum is small.

Original: For the quadratic loss, the third-order derivatives of the loss are zero orvery small, and Basu et al.
Revised: For the quadratic loss, the third-order derivatives of the loss are zero orvery small, and Basu et al.

Original: [4] empirically verified that the this assumption also approximately heldfor the classification problem with cross-entropy loss function.
Revised: [4] empirically verified that the this assumption also approximately heldfor the classification problem with cross-entropy loss function.

Original: Different from existing works that analyze the effects on model parameters of removing data points,this paper creatively adapts influence functions to estimate the importance of candidate operationson a trained supernet in differentiable architecture search.
Revised: Different from existing works that analyze the effects on model parameters of removing data points,this paper creatively adapts influence functions to estimate the importance of candidate operationson a trained supernet in differentiable architecture search.

Original: We also follow the assumption in [4] thatneglect the third-order and higher derivatives of the loss, and analyze the operation selection part,which is an additional process after bi-level supernet training in DARTS, through the lens of influencefunction in the following sections.
Revised: We also follow the assumption in [4] thatneglect the third-order and higher derivatives of the loss, and analyze the operation selection part,which is an additional process after bi-level supernet training in DARTS, through the lens of influencefunction in the following sections.

Original:  Interpret Operation Selection through Influence Functions
Revised:  Interpret Operation Selection through Influence Functions

Original: This section aims to understand how the model parameters θ (a.k.a.
Revised: This section aims to understand how the model parameters θ (a.k.a.

Original: the supernet weights in DARTS)would change when we derive the discrete architecture from the optimized continuous operationmagnitudes, a.k.a.
Revised: the supernet weights in DARTS)would change when we derive the discrete architecture from the optimized continuous operationmagnitudes, a.k.a.

Original: architecture selection.
Revised: architecture selection.

Original: In this paper, our analysis only focuses on this architectureselection part after the bi-level supernet training process is finished.
Revised: In this paper, our analysis only focuses on this architectureselection part after the bi-level supernet training process is finished.

Original: To be more clearly, we firstdefine the optimized operation magnitudes by DARTS as α with the optimized supernet weights θ ∗ ,and the validation loss is L ( θ ∗ , α ) for simplicity.
Revised: To be more clearly, we firstdefine the optimized operation magnitudes by DARTS as α with the optimized supernet weights θ ∗ ,and the validation loss is L ( θ ∗ , α ) for simplicity.

Original: Then, we define the architecture parameters changeto ˆ α after discretization, and the corresponding ground-truth supernet weights and validation lossare defined as ˆ θ ∗ and L (ˆ θ ∗ , ˆ α ) , respectively.
Revised: Then, we define the architecture parameters changeto ˆ α after discretization, and the corresponding ground-truth supernet weights and validation lossare defined as ˆ θ ∗ and L (ˆ θ ∗ , ˆ α ) , respectively.

Original: Rather than deleting a single data point that only bringssmall changes on the model parameters [22], the change of architecture parameters can lead to aconsiderable change on θ , so we leverage the second-order approximation [4] to reveal the supernetweights change.
Revised: Rather than deleting a single data point that only bringssmall changes on the model parameters [22], the change of architecture parameters can lead to aconsiderable change on θ , so we leverage the second-order approximation [4] to reveal the supernetweights change.

Original: With second-order Taylor expansion on ˆ θ ∗ for L (ˆ θ ∗ , ˆ α ) , we have:
Revised: With second-order Taylor expansion on ˆ θ ∗ for L (ˆ θ ∗ , ˆ α ) , we have:

Original: [Equation]
Revised: [Equation]

Original: where ∆ θ = ˆ θ ∗ − θ ∗ , and L is the validation loss function in Eq.(1).
Revised: where ∆ θ = ˆ θ ∗ − θ ∗ , and L is the validation loss function in Eq.(1).

Original: For example, when weonly remove one candidate operation as DARTS-PT [40] to measure the operation importance, theinfluence of this discretization brings to θ is ∆ θ , and the true validation loss change should be
Revised: For example, when weonly remove one candidate operation as DARTS-PT [40] to measure the operation importance, theinfluence of this discretization brings to θ is ∆ θ , and the true validation loss change should be

Original: L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) as stated in Eq.(4) rather than L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) used by DARTS-PT.
Revised: L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) as stated in Eq.(4) rather than L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) used by DARTS-PT.

Original: Asshown in Eq.(4), we need to calculate ∆ θ to get ∆ L after we change the α to ˆ α , while we try to avoidthe time-consuming retraining process through leveraging influence functions [22] to estimating ˆ θ ∗ .
Revised: Asshown in Eq.(4), we need to calculate ∆ θ to get ∆ L after we change the α to ˆ α , while we try to avoidthe time-consuming retraining process through leveraging influence functions [22] to estimating ˆ θ ∗ .

Original: Based on the implicit function theorem [29, 49], the validation loss change after discretizing α to ˆ αcan be approximated as the following theorem.
Revised: Based on the implicit function theorem [29, 49], the validation loss change after discretizing α to ˆ αcan be approximated as the following theorem.

Original: Theorem 1 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, α changes to ˆ α when conducting architecture discretization, andthe train-from-scratch validation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .
Revised: Theorem 1 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, α changes to ˆ α when conducting architecture discretization, andthe train-from-scratch validation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .

Original: If the third and higher derivatives of the lossfunction L at optimum is zero or sufficiently small [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have
Revised: If the third and higher derivatives of the lossfunction L at optimum is zero or sufficiently small [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 states that we can estimate the train-from-scratch validation performance of ˆ α without anyretraining.
Revised: Theorem 1 states that we can estimate the train-from-scratch validation performance of ˆ α without anyretraining.

Original: During the discretization process, we can apply the argmax on all edges by one-shot thatˆ α = argmax ( α ) , or only discretize one edge in each step, or remove one candidate operation fromone edge as DARTS-PT [40] which can be formulated as the first part of Eq.(5), L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: During the discretization process, we can apply the argmax on all edges by one-shot thatˆ α = argmax ( α ) , or only discretize one edge in each step, or remove one candidate operation fromone edge as DARTS-PT [40] which can be formulated as the first part of Eq.(5), L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: We prefer an iterative discretization process that only applies a small change on α as DARTS-PT,since our theoretical analysis in the next corollary will show that the error bound of our approximationis related to ∆ α = ˆ α − α .
Revised: We prefer an iterative discretization process that only applies a small change on α as DARTS-PT,since our theoretical analysis in the next corollary will show that the error bound of our approximationis related to ∆ α = ˆ α − α .

Original: Before our analysis, we give the following common assumptions, whichare also considered in several recent bi-level optimization problems [9, 15–17, 49].
Revised: Before our analysis, we give the following common assumptions, whichare also considered in several recent bi-level optimization problems [9, 15–17, 49].

Original: Assumption 1 For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with C f > 0 and
Revised: Assumption 1 For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with C f > 0 and

Original: C L > 0 , respectively.
Revised: C L > 0 , respectively.

Original: Assumption 2 L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θaround θ ∗ ( α ) .
Revised: Assumption 2 L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θaround θ ∗ ( α ) .

Original: Assumption 3 (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .
Revised: Assumption 3 (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .

Original: Corollary 1 Based on the Assumption 1-3, we could bound the error between the approximatedvalidation loss
Revised: Corollary 1 Based on the Assumption 1-3, we could bound the error between the approximatedvalidation loss

Original: L (ˆ θ ∗ , ˆ α ) =
Revised: L (ˆ θ ∗ , ˆ α ) =

Original: ∆ L + L ( θ ∗ , α ) and the ground-truth ˜ L (ˆ θ ∗ , ˆ α ) in DARTS with E =(cid:13)(cid:13)(cid:13)
Revised: ∆ L + L ( θ ∗ , α ) and the ground-truth ˜ L (ˆ θ ∗ , ˆ α ) in DARTS with E =(cid:13)(cid:13)(cid:13)

Original: L (ˆ θ ∗ , ˆ α ) − ˜ L (ˆ θ ∗ , ˆ α ) (cid:13)(cid:13)(cid:13) ⩽ K 3 6 max (cid:12)(cid:12)(cid:12) ∂
Revised: L (ˆ θ ∗ , ˆ α ) − ˜ L (ˆ θ ∗ , ˆ α ) (cid:13)(cid:13)(cid:13) ⩽ K 3 6 max (cid:12)(cid:12)(cid:12) ∂

Original: L 3 ∂θ 3 (cid:12)(cid:12)(cid:12) , where K = C L λ ∥
Revised: L 3 ∂θ 3 (cid:12)(cid:12)(cid:12) , where K = C L λ ∥

Original: ∆ α ∥ + C H C 2 a 2 σ 2 min λ ∥ ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .
Revised: ∆ α ∥ + C H C 2 a 2 σ 2 min λ ∥ ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .

Original: Compared with the argmax used in DARTS, we prefer the perturbation-based discretization method[41] as which only remove one operation at each step, since the above Corollary 1 show that errorbound of the approximated validation loss
Revised: Compared with the argmax used in DARTS, we prefer the perturbation-based discretization method[41] as which only remove one operation at each step, since the above Corollary 1 show that errorbound of the approximated validation loss

Original: L
Revised: L

Original: (ˆ θ ∗ , ˆ α ) increase with the magnitude of the change of α .
Revised: (ˆ θ ∗ , ˆ α ) increase with the magnitude of the change of α .

Original:  Influential Magnitude for Operation Selection in DARTS
Revised:  Influential Magnitude for Operation Selection in DARTS

Original: Section 3 provides an iterative solution to measure the operation importance based Eq.
Revised: Section 3 provides an iterative solution to measure the operation importance based Eq.

Original: (5) throughindividually removing each operation, while which needs to repeat n times (the number of allcandidate operations for all edges in the supernet work).
Revised: (5) throughindividually removing each operation, while which needs to repeat n times (the number of allcandidate operations for all edges in the supernet work).

Original: More important, since removing oneoperation poses a considerable change on α ( α i → 0 ), the approximation error produced by Eq.(5)is non-negligible which may affects the accuracy as stated by Corollary 1.
Revised: More important, since removing oneoperation poses a considerable change on α ( α i → 0 ), the approximation error produced by Eq.(5)is non-negligible which may affects the accuracy as stated by Corollary 1.

Original: So, a more practicalsolution to illustrate the importance of each operation is to estimate how the validation performancewill change after posing an infinitesimal change on α , a.k.a.
Revised: So, a more practicalsolution to illustrate the importance of each operation is to estimate how the validation performancewill change after posing an infinitesimal change on α , a.k.a.

Original: operation sensitivity.
Revised: operation sensitivity.

Original: In this section,we aim to investigate the operation importance through the lens of sensitivity [24].
Revised: In this section,we aim to investigate the operation importance through the lens of sensitivity [24].

Original: Different from
Revised: Different from

Original: Section 3 that removes one operation ( α i → 0 ), this section only changes the α i with an infinitesimalε i , which means that we can now conduct first-order Taylor expansion on α for L (ˆ θ ∗ , ˆ α ) .
Revised: Section 3 that removes one operation ( α i → 0 ), this section only changes the α i with an infinitesimalε i , which means that we can now conduct first-order Taylor expansion on α for L (ˆ θ ∗ , ˆ α ) .

Original: Furtherwith second-order Taylor expansion on θ and assuming the local optimal ∂ L ( θ ∗ ,α ) ∂θ = 0 , we have:
Revised: Furtherwith second-order Taylor expansion on θ and assuming the local optimal ∂ L ( θ ∗ ,α ) ∂θ = 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: Similar to Eq.(4), it is intractable to directly calculate ∆ θ , while we can leverage implicit functiontheorem to approximate it.
Revised: Similar to Eq.(4), it is intractable to directly calculate ∆ θ , while we can leverage implicit functiontheorem to approximate it.

Original: Theorem 2 presents how the validation performance changes after wepose an infinitesimal change on α .
Revised: Theorem 2 presents how the validation performance changes after wepose an infinitesimal change on α .

Original: Theorem 2 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, and we pose an infinitesimal change on α .
Revised: Theorem 2 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, and we pose an infinitesimal change on α .

Original: Based on implicitfunction theorem and under the assumption that the third and higher derivatives of the loss functionat optimum is zero or sufficiently small [4], the change of validation loss can be estimated as:
Revised: Based on implicitfunction theorem and under the assumption that the third and higher derivatives of the loss functionat optimum is zero or sufficiently small [4], the change of validation loss can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.
Revised: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.

Original: From Theorem 2, we can observe the relationship between ∆ L and ∆ α .
Revised: From Theorem 2, we can observe the relationship between ∆ L and ∆ α .

Original: To be more specifically,when we only consider an infinitesimal change on α , that ∆ α = ε · 1 where 1 is a column vectorwith all ones and ε is an infinitesimal scalar, the sensitivity of α can be defined as:
Revised: To be more specifically,when we only consider an infinitesimal change on α , that ∆ α = ε · 1 where 1 is a column vectorwith all ones and ε is an infinitesimal scalar, the sensitivity of α can be defined as:

Original: [Equation]
Revised: [Equation]

Original: Accordingly, we propose an Influential Magnitude to quantify the operation importance through thelens of sensitivity:
Revised: Accordingly, we propose an Influential Magnitude to quantify the operation importance through thelens of sensitivity:

Original: Definition 1 Influential Magnitude ( I M ): Suppose DARTS obtains the optimized magnitude αwith supernet weights θ ∗ after supernet training, the operation sensitivity can be defined as I M =
Revised: Definition 1 Influential Magnitude ( I M ): Suppose DARTS obtains the optimized magnitude αwith supernet weights θ ∗ after supernet training, the operation sensitivity can be defined as I M =

Original: [Equation]
Revised: [Equation]

Original: Definition 1 shows we can replace the magnitude α with the operation sensitivity I M in the operationselection for DARTS, whose i-th value represents the importance of i-th candidate operation.
Revised: Definition 1 shows we can replace the magnitude α with the operation sensitivity I M in the operationselection for DARTS, whose i-th value represents the importance of i-th candidate operation.

Original:  Practical Calculation on Operation Influence
Revised:  Practical Calculation on Operation Influence

Original: For a large neural network, it is impractical to calculate the second-order information, e.g., the
Revised: For a large neural network, it is impractical to calculate the second-order information, e.g., the

Original: Hessian matrix H , let alone the inverse of Hessian.
Revised: Hessian matrix H , let alone the inverse of Hessian.

Original: Generally, the core challenge in calculation of
Revised: Generally, the core challenge in calculation of

Original: Eq.(5) and Eq.(7) is the Inverse-Hessian Vector Products (IHVPs), H − 1 v [14, 22, 37].
Revised: Eq.(5) and Eq.(7) is the Inverse-Hessian Vector Products (IHVPs), H − 1 v [14, 22, 37].

Original: As to the termof ∂ 2 L ( θ ∗ ,α ) ∂θ∂α , we follow DARTS [27] to utilize the finite difference approximation for the practicalcalculation.
Revised: As to the termof ∂ 2 L ( θ ∗ ,α ) ∂θ∂α , we follow DARTS [27] to utilize the finite difference approximation for the practicalcalculation.

Original: The first technique for IHVPs is the conjugate gradient (CG) which converts the matrixinverse problem into an optimization problem with δ -optimal solution.
Revised: The first technique for IHVPs is the conjugate gradient (CG) which converts the matrixinverse problem into an optimization problem with δ -optimal solution.

Original: However, the standard CGcan be slow with a large dataset as it needs to go through the whole training points in each step [1].
Revised: However, the standard CGcan be slow with a large dataset as it needs to go through the whole training points in each step [1].

Original: The original paper [22] using influence function for modern machine learning problem leveragesthe Neumann series to approximate the inverse, that H − 1 = (cid:80) ∞ k =

Revised: The original paper [22] using influence function for modern machine learning problem leveragesthe Neumann series to approximate the inverse, that H − 1 = (cid:80) ∞ k =

Original: ( I − H ) k , with only considerthe first t terms to approximate H − 1 .
Revised: ( I − H ) k , with only considerthe first t terms to approximate H − 1 .

Original: However, there is a strict condition in the Neumann seriesapproximation that ∥ I − H ∥ ≤ 1 , which can not be guaranteed in the practical implementation.
Revised: However, there is a strict condition in the Neumann seriesapproximation that ∥ I − H ∥ ≤ 1 , which can not be guaranteed in the practical implementation.

Original: Differently, we consider that H − 1 = γ ( γH ) − 1 = γ (cid:80) ∞
Revised: Differently, we consider that H − 1 = γ ( γH ) − 1 = γ (cid:80) ∞

Original: k
Revised: k

Original: =

Revised: =

Original: ( I − γH )
Revised: ( I − γH )

Original: k , where γ is a small enoughvalue.
Revised: k , where γ is a small enoughvalue.

Original: The following lemma gives our Neumann series based implementation for IHVPs.
Revised: The following lemma gives our Neumann series based implementation for IHVPs.

Original: Lemma 1 With small enough γ , and assuming L is λ -strongly convex at optimum, H − 1 v can beformulated as: H − 1 v = γ (cid:80) Kk =

Revised: Lemma 1 With small enough γ , and assuming L is λ -strongly convex at optimum, H − 1 v can beformulated as: H − 1 v = γ (cid:80) Kk =

Original: [ I − γH ] k V 

Revised: [ I − γH ] k V

Original: =
Revised: =

Original: V 

Revised: V

Original: + V 1 + ... + V K ,
Revised: + V 1 + ... + V K ,

Original: where H = ∂
Revised: where H = ∂

Original: 2 L ( θ ∗ ,α ) ∂θ∂θ , V0 = v ,and V 

Revised: 2 L ( θ ∗ ,α ) ∂θ∂θ , V

Original: =  ( I − γH )
Revised: = γv , and V 1 = γ ( I − γH )

Original: V 0 , ..., V K 
Revised: V 0 , ..., V K = γ

Original: = ( I − γH )
Revised: ( I − γH )

Original: V K − 1 .
Revised: V K − 1 .

Original: In the practical implementation, it is still hard to make sure L is λ -strongly convex and not easyto select an appropriate γ .
Revised: In the practical implementation, it is still hard to make sure L is λ -strongly convex and not easyto select an appropriate γ .

Original: The work [5] empirically showed that the stochastic Neumann seriesestimation for IHVP is somehow erroneous, especially when the network is deep.
Revised: The work [5] empirically showed that the stochastic Neumann seriesestimation for IHVP is somehow erroneous, especially when the network is deep.

Original: As we know, thetrue Fisher information matrix converges to the Hessian matrix with the training loss approachesto zero.
Revised: As we know, thetrue Fisher information matrix converges to the Hessian matrix with the training loss approachesto zero.

Original: However, it is usually infeasible to calculate the exact Hessian or Fisher matrix in a largeneural network.
Revised: However, it is usually infeasible to calculate the exact Hessian or Fisher matrix in a largeneural network.

Original: A practical approximation is called as empirical Fisher, which has been verified tobe effective in variety of applications [3, 19].
Revised: A practical approximation is called as empirical Fisher, which has been verified tobe effective in variety of applications [3, 19].

Original: In addition, an important advantage from empirical
Revised: In addition, an important advantage from empirical

Original: Fisher is that it allows to approximate the inverse of the Hessian based on the Sherman-Morrisonformula.
Revised: Fisher is that it allows to approximate the inverse of the Hessian based on the Sherman-Morrisonformula.

Original: Although the empirical Fisher can not exactly match the true Fisher and there are severalactive researches on its applicability [23, 38], empirical Fisher is significantly more computationallyefficient, which obtains a good trade-off between the approximation and practical efficiency [37].
Revised: Although the empirical Fisher can not exactly match the true Fisher and there are severalactive researches on its applicability [23, 38], empirical Fisher is significantly more computationallyefficient, which obtains a good trade-off between the approximation and practical efficiency [37].

Original: Apart from Neumann series, we also consider Sherman-Morrison formula [14, 37]
Revised: Apart from Neumann series, we also consider Sherman-Morrison formula [14, 37]

Original: ( A + uv T ) − 1 =
Revised: ( A + uv T ) − 1 =

Original: A − 1 − A − 1 uv T A − 1 1+ v T A − 1 u to calculate IHVPs.
Revised: [Equation] 1 u to calculate IHVPs.

Original: With empirical Fisher F = (cid:80) nj =1 ∇ θ L j ( ∇ θ L j ) T replacingtrue Fisher ˆ F = ∇ θ f ( ∇ θ f ) T , IHVPs can be calculated as following.
Revised: With empirical Fisher F = (cid:80) nj =1 ∇ θ L j ( ∇ θ L j ) T replacingtrue Fisher ˆ F = ∇ θ f ( ∇ θ f ) T , IHVPs can be calculated as following.

Original: Lemma 2 When assume the empirical Fisher can approximate the Fisher matrix, and H is the Hessian matrix∂ 2 L ( θ ∗ ,α ) ∂θ∂θ in the optimal point, the IHVPs H − 1 v can be formulated as: H − 1 v = F − 1 n v = F
Revised: Lemma 2 When assume the empirical Fisher can approximate the Fisher matrix, and H is the Hessian matrix

Original: [Equation]
Revised: [Equation]

Original:  n 
Revised: in the optimal point, the IHVPs H − 1 v can be formulated as: H − 1 v = F − 1 n v =

Original: [Equation]
Revised: [Equation]

Original: L jT r j , where L = ℓ + η R ( θ ) that ℓ is a crossentropy loss and R is the regularization term, F n = 1 n (cid:80) nj = [Equation] ∇
Revised: L jT r j , where L = ℓ + η R ( θ ) that ℓ is a crossentropy loss and R is the regularization term, F n = 1 n (cid:80) nj =1 ∇

Original: [Equation]
Revised: [Equation]

Original: L

Original: [Equation] ∇ [Equation] L Tj is the empirical Fisher, andr j = F − 1 j − 1 ∇ θ L j which can be recurrently calculated through r j = η − 1 ∇ θ L j − (cid:80) j − 1 i =1 r
Revised: j ∇ θ L Tj is the empirical Fisher, andr j = F − 1 j − 1 ∇ θ L j which can be recurrently calculated through r j = η − 1 ∇ θ L j − (cid:80) j − 1 i =1 r

Original: i r iT ∇ θ L j N + ∇ θ L iT r i .
Revised: i r iT ∇ θ L j N + ∇ θ L iT r i .

Original:  Experiments
Revised:  Experiments

Original: [Table]
Revised: [Table]

Original: In Section 3 and 4, we have theoreticallyinterpreted the operation selection throughindividually measuring each operation’sinfluence on supernet, and accordinglydevised an Influential Magnitude ( I M )to replace the commonly-used magnitudeα for architecture selection.
Revised: In Section 3 and 4, we have theoreticallyinterpreted the operation selection throughindividually measuring each operation’sinfluence on supernet, and accordinglydevised an Influential Magnitude ( I M )to replace the commonly-used magnitudeα for architecture selection.

Original: In this section, we conduct a series of experimentsto analyze whether our proposed methodleads to better and more stable results indifferentiable architecture search settings.
Revised: In this section, we conduct a series of experimentsto analyze whether our proposed methodleads to better and more stable results indifferentiable architecture search settings.

Original: We consider two NAS benchmark datasetsearch space, NAS-Bench-201 [12] and
Revised: We consider two NAS benchmark datasetsearch space, NAS-Bench-201 [12] and

Original: NAS-Bench-1shot1 [46], and the most popular DARTS search space [27].
Revised: NAS-Bench-1shot1 [46], and the most popular DARTS search space [27].

Original: We verified the effectiveness of the proposed theory-driven method compared with two heuristic methods, argmax and perturbation-based selection, and show that the proposed Influential Magnitude can be applied to
Revised: We verified the effectiveness of the proposed theory-driven method compared with two heuristic methods, argmax and perturbation-based selection, and show that the proposed Influential Magnitude can be applied to

Original: DARTS and its variants with greatly enhancing the performance.
Revised: DARTS and its variants with greatly enhancing the performance.

Original: 6.1 Reproducible Comparison on Benchmark Datasets
Revised: 6.1 Reproducible Comparison on Benchmark Datasets

Original: To empirically verify the effectiveness of our method, we first run a series of experiments on the two
Revised: To empirically verify the effectiveness of our method, we first run a series of experiments on the two

Original: NAS benchmark datasets, NAS-Bench-201 and NAS-Bench-1shot1.
Revised: NAS benchmark datasets, NAS-Bench-201 and NAS-Bench-1shot1.

Original: To make our results reproducible,we only replace the selection part in DARTS with I M and keep the remaining part identical.
Revised: To make our results reproducible,we only replace the selection part in DARTS with I M and keep the remaining part identical.

Original: Table 1 summarizes the statistical comparison results on NAS-Bench-201 with different random seeds ,where DARTS-IF, as described in Section 3, follows DARTS-PT that individually removes operationto measure the supernet performance drop, while DARTS-IM, as described in Section 4, calculatesthe operation sensitivity (influential magnitude) to replace the original magnitude.
Revised: Table 1 summarizes the statistical comparison results on NAS-Bench-201 with different random seeds ,where DARTS-IF, as described in Section 3, follows DARTS-PT that individually removes operationto measure the supernet performance drop, while DARTS-IM, as described in Section 4, calculatesthe operation sensitivity (influential magnitude) to replace the original magnitude.

Original: Algorithm 1 and2 in the Appendix H sketched the two proposed methods.
Revised: Algorithm 1 and2 in the Appendix H sketched the two proposed methods.

Original: The Random baseline is to randomlyselect architectures without considering α or θ , and DARTS and DARTS-PT leverage the argmaxand perturbation-based heuristic selection, respectively.
Revised: The Random baseline is to randomlyselect architectures without considering α or θ , and DARTS and DARTS-PT leverage the argmaxand perturbation-based heuristic selection, respectively.

Original: As shown, our DARTS-IF and DARTS-IMboth significantly outperform the random baseline and the two heuristic baselines, where DARTS-IMachieves near-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR100, and ImageNet, respectively, in a single run (DARTS-IM with Sherman-Morrison approximationand N = 20 under random seed 0).
Revised: As shown, our DARTS-IF and DARTS-IMboth significantly outperform the random baseline and the two heuristic baselines, where DARTS-IMachieves near-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR100, and ImageNet, respectively, in a single run (DARTS-IM with Sherman-Morrison approximationand N = 20 under random seed 0).

Original: [Figure]
Revised: [Figure]

Original: Figure 7 in Appendix G plots thesearched architectures with different architecture selection methodsin NAS-Bench-201.
Revised: Figure 8 in Appendix G plots thesearched architectures with different architecture selection methodsin NAS-Bench-201.

Original: We can seethat DARTS selects skip-connectionfor all edges as its magnitude usually outweighs other candidate operations.
Revised: We can seethat DARTS selects skip-connectionfor all edges as its magnitude usually outweighs other candidate operations.

Original: The results of DARTS in NASBench-201, which are even poorerthan the random baseline, also suggest that the magnitudes could hardlyreflect the operation importance.
Revised: The results of DARTS in NASBench-201, which are even poorerthan the random baseline, also suggest that the magnitudes could hardlyreflect the operation importance.

Original: Although DARTS-PT can partially relieve this issue by selecting one ortwo more convolutional operations,it still prefers skip-connections (selects 3 skip-connections for 6 edges)where directly removing them still significantly affect the performance.
Revised: Although DARTS-PT can partially relieve this issue by selecting one ortwo more convolutional operations,it still prefers skip-connections (selects 3 skip-connections for 6 edges)where directly removing them still significantly affect the performance.

Original: On the contrary, our DARTSIF and DARTS-IM can find valid architectures with more convolutional operations (with only oneskip-connection ).
Revised: On the contrary, our DARTSIF and DARTS-IM can find valid architectures with more convolutional operations (with only oneskip-connection ).

Original: Results in Table 1 verified the effectiveness of our influence functions basedselection methods, which outperform baselines by large margins.
Revised: Results in Table 1 verified the effectiveness of our influence functions basedselection methods, which outperform baselines by large margins.

Original: DARTS-IF is similar to DARTSPT while with an additional term to approximate the fine-tuned performance and the results that
Revised: DARTS-IF is similar to DARTSPT while with an additional term to approximate the fine-tuned performance and the results that

Original: DARTS-IF outperforms DARTS-PT verified the effectiveness of the influence function explanation.
Revised: DARTS-IF outperforms DARTS-PT verified the effectiveness of the influence function explanation.

Original: [Table]
Revised: [Table]

Original: As discussed in Section 5, we can leverage different methods to approximate the IHVPs inour I M , and Figure 2 also analyzes differentapproximation methods along with the numberof batches N .
Revised: As discussed in Section 5, we can leverage different methods to approximate the IHVPs inour I M , and Figure 2 also analyzes differentapproximation methods along with the numberof batches N .

Original: Full results on three datasetscan be found in the Figure 5 in Appendix F.
Revised: Full results on three datasetscan be found in the Figure 5 in Appendix F.

Original: As demonstrated, increasing N can help the
Revised: As demonstrated, increasing N can help the

Original: Sherman-Morrison approximation to more accurately catch the operation importance and weempirically find that N = 30 is enough in the
Revised: Sherman-Morrison approximation to more accurately catch the operation importance and weempirically find that N = 30 is enough in the

Original: NAS-Bench-201 dataset.
Revised: NAS-Bench-201 dataset.

Original: However, we also observe Neumann approximation is not stable which even obtains worse results with increasing numberof batches, showing it is somehow erroneous in estimating IHVPs.
Revised: However, we also observe Neumann approximation is not stable which even obtains worse results with increasing numberof batches, showing it is somehow erroneous in estimating IHVPs.

Original: Appendix F analyzes differentinverse Hessian estimation methods for our I M .
Revised: Appendix F analyzes differentinverse Hessian estimation methods for our I M .

Original: The NAS-Bench-1shot1 is another popular NAS benchmark dataset to analyze differentiable NASmethods, which is built based on the NAS-Bench-101 [44], through dividing all architectures in
Revised: The NAS-Bench-1shot1 is another popular NAS benchmark dataset to analyze differentiable NASmethods, which is built based on the NAS-Bench-101 [44], through dividing all architectures in

Original: NAS-Bench-101 into three different unified cell-based search spaces to enable differentiable NASmethods to be directly applied.
Revised: NAS-Bench-101 into three different unified cell-based search spaces to enable differentiable NASmethods to be directly applied.

Original: NAS-Bench-1shot1 contains 6240, 29160, and 363648 architecturesin three spaces, respectively, where Space3 is much more complicated than the remaining two spaces.
Revised: NAS-Bench-1shot1 contains 6240, 29160, and 363648 architecturesin three spaces, respectively, where Space3 is much more complicated than the remaining two spaces.

Original: In NAS-Bench-1shot1, we tracked the searched architecture in every epoch under different operationselection paradigms.
Revised: In NAS-Bench-1shot1, we tracked the searched architecture in every epoch under different operationselection paradigms.

Original: We considered two popular supernet training baselines, DARTS and PC-DARTS,and Table 2 summarised the best-searched architectures during the supernet training in differentsub-search spaces in NAS-Bench-1shot1.
Revised: We considered two popular supernet training baselines, DARTS and PC-DARTS,and Table 2 summarised the best-searched architectures during the supernet training in differentsub-search spaces in NAS-Bench-1shot1.

Original: As demonstrated, our I M can enhance the performanceof DARTS and its variant PC-DARTS in different search spaces of NAS-Bench-1shot1, showingthat our influence magnitude (IM) can find more competitive architectures than the default argmax ,especially in the more complicated space.
Revised: As demonstrated, our I M can enhance the performanceof DARTS and its variant PC-DARTS in different search spaces of NAS-Bench-1shot1, showingthat our influence magnitude (IM) can find more competitive architectures than the default argmax ,especially in the more complicated space.

Original: Compared with the perturbation strategy (PT), our IM ismore effective to measure the operation sensitivity with obtaining 5 better results in 6 cases.
Revised: Compared with the perturbation strategy (PT), our IM ismore effective to measure the operation sensitivity with obtaining 5 better results in 6 cases.

Original: 6.2 Reproducible Comparison on DARTS Search Space
Revised: 6.2 Reproducible Comparison on DARTS Search Space

Original: [Table]
Revised: [Table]

Original: The DARTS search space is the mostcommonly-used space to evaluate the NASmethods, while most SOTAs only report theperformance of their best searched architectures without analyzing the search statistics.
Revised: The DARTS search space is the mostcommonly-used space to evaluate the NASmethods, while most SOTAs only report theperformance of their best searched architectures without analyzing the search statistics.

Original: To encourage reproducibility, we train thesupernet with the same random seeds, andthe trained supernet is utilized to derive thearchitecture under different operation selection strategies.
Revised: To encourage reproducibility, we train thesupernet with the same random seeds, andthe trained supernet is utilized to derive thearchitecture under different operation selection strategies.

Original: The comparison results areprovided in Table 3, from which we can seethat, compared with the argmax baseline, thearchitectures searched by our Influential Magnitude obtain better performance on CIFAR-10 and
Revised: The comparison results areprovided in Table 3, from which we can seethat, compared with the argmax baseline, thearchitectures searched by our Influential Magnitude obtain better performance on CIFAR-10 and

Original: ImageNet than DARTS and its variant PC-DARTS, and DARTS-PT on ImageNet, implying ourmethod is more stable to find valid and competitive architectures.
Revised: ImageNet than DARTS and its variant PC-DARTS, and DARTS-PT on ImageNet, implying ourmethod is more stable to find valid and competitive architectures.

Original: Figure 8 in Appendix G also plotsthe searched architecture examples with different selection methods in the DARTS space.
Revised: Figure 9 in Appendix G also plotsthe searched architecture examples with different selection methods in the DARTS space.

Original: As shown,although DARTS, PC-DARTS and DARTS-PT can find satisfying architecture after multiple runs, theargmax is hard to consistently obtain stable results.
Revised: As shown,although DARTS, PC-DARTS and DARTS-PT can find satisfying architecture after multiple runs, theargmax is hard to consistently obtain stable results.

Original: In addition, different from the perturbation-basedoperation selection which needs to fine-tune the supernet after the discretization of each edge thatcosts 0.4 additional GPU day for DARTS, our influential magnitude is calculated by one time whichonly costs hundred seconds, making our method is much more efficient.
Revised: In addition, different from the perturbation-basedoperation selection which needs to fine-tune the supernet after the discretization of each edge thatcosts 0.4 additional GPU day for DARTS, our influential magnitude is calculated by one time whichonly costs hundred seconds, making our method is much more efficient.

Original: Detailed comparisons onbest-searched architecture in DARTS space with the SOTAs can be found in Table 5 in Appendix E.
Revised: Detailed comparisons onbest-searched architecture in DARTS space with the SOTAs can be found in Table 5 in Appendix E.

Original: 6.3 Discussion on the Robustness of Search
Revised: 6.3 Discussion on the Robustness of Search

Original: [Equation]
Revised: [Equation]

Original: [Figure][Equation] [Figure] [Equation]
Revised: [Figure]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: S c_{k-2} 1 skip_connect 2 skip_connect 3 sep_conv_3x dil_conv_5x5 dil_conv_3x3 dil_conv_3x c_{k} dil_conv_3x dil_conv_3x c_{k-1} (d) S
Revised: DARTS c_{k-2} 1 skip_connect 2 skip_connect 3 sep_conv_3x dil_conv_5x5 dil_conv_3x3 dil_conv_3x c_{k} dil_conv_3x dil_conv_3x c_{k-1} (d) S

Original: The robustness has been a criticalconcern in the differentiable architecture search community since Zelaet al.
Revised: The robustness has been a criticalconcern in the differentiable architecture search community since Zelaet al.

Original: [45] observed a performancecollapse that DARTS tends to yielddegenerate architectures, which especially prefers the skip-connectionswith the search progressing.
Revised: [45] observed a performancecollapse that DARTS tends to yielddegenerate architectures, which especially prefers the skip-connectionswith the search progressing.

Original: Similar to DARTS-PT, we also focuson the operation selection part toexplain the failure of DARTS.
Revised: Similarto DARTS-PT, we also focus on theoperation selection part to explain thefailure of DARTS.

Original: Figure 3 presents several architecturesby DARTS with our I M on different search spaces.
Revised: Figure 3 presentsseveral architectures by DARTS withour I M on different search spaces.

Original: Apart from NASBench-201, we also consider severaltool search spaces (S1-S4) proposedby [45], which are specifically designed to verify the robustness of differentiable NAS methods.
Revised: Apart from NAS-Bench-201, we alsoconsider several tool search spaces(S1-S4) proposed by [45], which arespecifically designed to verify the robustness of differentiable NAS methods.

Original: The complete analysis and searched architectures can be found in Appendix G.As shown in Figure 3,
Revised: The complete analysis and searched architectures can be found in Appendix G.

Original:  DARTS-IM can find more meaningful architectures in different spaces.
Revised: As shownin Figure 3, DARTS-IM can find more meaningful architectures in different spaces.

Original: For example, DARTS fails dramatically in most spaces that only selects skip-connection for all edges, while DARTS-IM canfind architecture with 5 convolutions in NAS-Bench-201, and 6 and 8 convolutions in S1 and S2.
Revised: For example, DARTS fails dramatically in most spaces that only selects skip-connection for all edges, while DARTS-IM can find architecture with 5 convolutions in NAS-Bench-201, and 6 and 8 convolutionsin S1 and S2.

Original:  Discussion at Non-Convergence
Revised: 6.4 Discussion at Non-Convergence

Original: [Figure]
Revised: [Figure]

Original: As discussed in Section 4, our influential magnitude is derived when θis in the optimum, with leveragingdifferent methods to approximate the
Revised: As discussed in Section 4, our influential magnitude is derived when θis in the optimum, with leveragingdifferent methods to approximate the

Original: IHVPs.
Revised: IHVPs.

Original: In this subsection, we analyze whether the proposed influentialmagnitude is still effective to find important operations when θ is not inthe optimum.
Revised: In this subsection, we analyze whether the proposed influentialmagnitude is still effective to find important operations when θ is not inthe optimum.

Original: Figure 4 tracked theperformance of searched architecturein each epoch during the search with
Revised: Figure 4 tracked theperformance of searched architecturein each epoch during the search with

Original: Sherman-Morrison formula.
Revised: Sherman-Morrison formula.

Original: Full results on three datasets and more comparison with DARTS-PT can be found in the Figure 6 in Appendix F. As shown, our influentialmagnitude I M is still effective when θ is not in the optimum, while which is getting more stable withthe supernet training.
Revised: Full results on three datasets and more comparison with DARTS-PT can be found in the Figure 7 in Appendix F. As shown, our influentialmagnitude I M is still effective when θ is not in the optimum, while which is getting more stable withthe supernet training.

Original: Specifically, as shown in Figure 4 (a) and (b), our I M with the different N areboth effective, implying that our method is sufficiently informative for the operation selection evenunder different approximation noises.
Revised: Specifically, as shown in Figure 4 (a) and (b), our I M with the different N areboth effective, implying that our method is sufficiently informative for the operation selection evenunder different approximation noises.

Original: More interesting, our DARTS-IM under Sherman-Morrison formula with N =1 in Figure 4 meansthat we simply approximate the inverse H − 1 =
Revised: More interesting, our DARTS-IM under Sherman-Morrison formula with N =1 in Figure 4 meansthat we simply approximate the inverse H − 1 =

Original: η − 1 (cid:16) I − η −
Revised: η − 1 (cid:16) I − η −

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Original: ⊤ ∇ θ L (cid:17) ∝ ˆ H − 1 , that ˆ H − 1is a diagonal matrix that ˆ H ii = ( ∇ θ i L∇
Revised: [Equation] [Figure] [Equation] that ˆ H − 1is a diagonal matrix that ˆ H ii = ( ∇ θ i L∇

Original: θ i L ) , which is also called the diagonal approximation of theinverse Hessian.
Revised: θ i L ) , which is also called the diagonal approximation of theinverse Hessian.

Original: As shown, with roughly approximated second-order information, Eq.
Revised: As shown, with roughly approximated second-order information, Eq.

Original: (7) also help
Revised: (7) also help

Original: DARTS-IM to find more important operations compared with existing heuristic methods.
Revised: DARTS-IM to find more important operations compared with existing heuristic methods.

Original: 6.5 Discussion on the Limitations in Practice
Revised: 6.5 Discussion on the Limitations in Practice

Original: Although our influential magnitude provides a theory-driven and concise solution for the operationselection in DARTS, we consider several assumptions and approximations for the practical implementation.
Revised: Although our influential magnitude provides a theory-driven and concise solution for the operationselection in DARTS, we consider several assumptions and approximations for the practical implementation.

Original: For example, Basu et al.
Revised: For example, Basu et al.

Original: [4, 5] pointed out that the second-order influence values stillunderestimated the true group influence values in the deep neural networks, while they also observedthe correlation marginally improves with the number of parameters in a group and the second-orderinfluence function generally outperforms the first-order.
Revised: [4, 5] pointed out that the second-order influence values stillunderestimated the true group influence values in the deep neural networks, while they also observedthe correlation marginally improves with the number of parameters in a group and the second-orderinfluence function generally outperforms the first-order.

Original: Fortunately, this paper considers the wholeparameters in the supernet, which helps the second-order influence function reach its potential.
Revised: Fortunately, this paper considers the wholeparameters in the supernet, which helps the second-order influence function reach its potential.

Original: The approximation of IHVPs requires the conditions that the Hessian matrix is positive definite[29] or the empirical Fisher can replace the true Fisher matrix [14], and the supernet weight θ ∗ is aglobal minimum of the empirical risk, while which in practice, are hard to hold.
Revised: The approximation of IHVPs requires the conditions that the Hessian matrix is positive definite[29] or the empirical Fisher can replace the true Fisher matrix [14], and the supernet weight θ ∗ is aglobal minimum of the empirical risk, while which in practice, are hard to hold.

Original: In the calculation ofv ∗ ∂ 2 L ∂α∂θ , which is detailed described in Appendix B, this paper also follows DARTS that considersthe finite difference approximation for practical calculation.
Revised: In the calculation ofv ∗ ∂ 2 L ∂α∂θ , which is detailed described in Appendix B, this paper also follows DARTS that considersthe finite difference approximation for practical calculation.

Original: These approximations and assumptionsmay bring errors for the practical calculation, while empirical results verified that our influentialmagnitude is sufficiently informative for the operation selection in DARTS.
Revised: These approximations and assumptionsmay bring errors for the practical calculation, while empirical results verified that our influentialmagnitude is sufficiently informative for the operation selection in DARTS.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: This paper focuses on the operation selection, which is an essential part while with less attention inthe differentiable architecture search.
Revised: This paper focuses on the operation selection, which is an essential part while with less attention inthe differentiable architecture search.

Original: Rather than considering heuristics, this paper introduces theinfluence functions to theoretically reveal the candidate operation’s importance through approximatingits influence on the supernet.
Revised: Rather than considering heuristics, this paper introduces theinfluence functions to theoretically reveal the candidate operation’s importance through approximatingits influence on the supernet.

Original: By leveraging the Taylor’s approximation, we gracefully interpret howthe validation performance changes during the discretization in DARTS and show that the operationstrength is not only related to the magnitude but also the second-order information, where an influentialmagnitude is accordingly devised for the architecture selection.
Revised: By leveraging the Taylor’s approximation, we gracefully interpret howthe validation performance changes during the discretization in DARTS and show that the operationstrength is not only related to the magnitude but also the second-order information, where an influentialmagnitude is accordingly devised for the architecture selection.

Original: We based our framework on DARTSand extensive experimental results verified the proposed influential magnitude can be practically usedin operation selection to avoid failures in differentiable architecture search.
Revised: We based our framework on DARTSand extensive experimental results verified the proposed influential magnitude can be practically usedin operation selection to avoid failures in differentiable architecture search.

Original: Developing more accurateinfluential magnitude for architecture search or leveraging the training dynamics [20] to reveal theoperation influence in the early of training and also extending influential magnitude to other bi-leveloptimization problems[21, 28, 29], e.g., network pruning [28], are among directions for future work.
Revised: Developing more accurateinfluential magnitude for architecture search or leveraging the training dynamics [20] to reveal theoperation influence in the early of training and also extending influential magnitude to other bi-leveloptimization problems[21, 28, 29], e.g., network pruning [28], are among directions for future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-

Original: tions and scope?
Revised: tions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [No]
Revised: [No]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: results (either in the supplemental material or as a URL)?
Revised: results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments

Original: multiple times)?
Revised: multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-

Original: ing/curating?
Revised: ing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: informa-
Revised: informa-

Original: tion or offensive content?
Revised: tion or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)

Original: approvals, if applicable?
Revised: approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on

Original: participant compensation?
Revised: participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Proof
Revised: A Proof

Original: Theorem 1: Suppose that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗after supernet training, α changes to ˆ α when conducting architecture discretization, and the train-from-scratchvalidation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .
Revised: Theorem 1: Suppose that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗after supernet training, α changes to ˆ α when conducting architecture discretization, and the train-from-scratchvalidation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .

Original: If the third-derivative of the loss function L at optimum is zero or sufficientlysmall [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have
Revised: If the third-derivative of the loss function L at optimum is zero or sufficientlysmall [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1: Based on Eq.(4), we now only need to calculate ∆ θ to get ∆ L after we change the 565 α to ˆ α as other parts could be directly obtained, while we try to avoid the time-consuming retraining processthrough leveraging implicit function to estimating ˆ θ ∗ .
Revised: Proof of Theorem 1: Based on Eq.(4), we now only need to calculate ∆ θ to get ∆ L after we change the 565 α to ˆ α as other parts could be directly obtained, while we try to avoid the time-consuming retraining processthrough leveraging implicit function to estimating ˆ θ ∗ .

Original: First, based on the local optimal, we have
Revised: First, based on the local optimal, we have

Original: [Equation]
Revised: [Equation]

Original: When we consider the second-order Taylor expansion on θ , we have
Revised: When we consider the second-order Taylor expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (12) is obtained since we assume the third-derivative of the loss function L at optimum is zero or sufficiently 569 small [4].
Revised: (12) is obtained since we assume the third-derivative of the loss function L at optimum is zero or sufficiently 569 small [4].

Original: And we have
Revised: And we have

Original: [Equation]
Revised: [Equation]

Original: So,
Revised: So,

Original: [Equation]
Revised: [Equation]

Original: In this way, based on Eq.(4), (14) and (15), we have
Revised: In this way, based on Eq.(4), (14) and (15), we have

Original: [Equation]
Revised: [Equation]

Original: In this way, we can approximate every change in the α .
Revised: In this way, we can approximate every change in the α .

Original: □
Revised: □

Original: Corollary 1: Based on the Assumption 1-3, we could bound the error between the approximated validationloss L ( ˆ θ ∗ , ˆ α ) and the ground-truth ˜
Revised: Corollary 1: Based on the Assumption 1-3, we could bound the error between the approximated validationloss L ( ˆ θ ∗ , ˆ α ) and the ground-truth ˜

Original: L 2 ( ˆ θ ∗ , ˆ α ) in DARTS with E =
Revised: L 2 ( ˆ θ ∗ , ˆ α ) in DARTS with E =

Original: (cid:13)(cid:13) (cid:13)
Revised: (cid:13)(cid:13) (cid:13)

Original: L ( ˆ θ ∗ , ˆ α )
Revised: L ( ˆ θ ∗ , ˆ α )

Original: − ˜ L ( ˆ θ ∗ , ˆ α ) (cid:13)(cid:13) (cid:13) ⩽ K 3 6 max (cid:12)(cid:12) (cid:12) ∂
Revised: − ˜ L ( ˆ θ ∗ , ˆ α ) (cid:13)(cid:13) (cid:13) ⩽ K 3 6 max (cid:12)(cid:12) (cid:12) ∂

Original: L 3 ∂θ 3 (cid:12)(cid:12) (cid:12) ,where K = C L λ ∗ ∥
Revised: L 3 ∂θ 3 (cid:12)(cid:12) (cid:12) ,where K = C L λ ∗ ∥

Original: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥
Revised: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥

Original: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 4 ) .
Revised: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 4 ) .

Original: Proof of Corollary
Revised: Proof of Corollary

Original: Assumption 1: For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with constant C f > 0 andconstant C L > 0 , respectively.
Revised: Assumption 1: For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with constant C f > 0 andconstant C L > 0 , respectively.

Original: Assumption 2: L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θ around 582 θ ∗ ( α ) .
Revised: Assumption 2: L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θ around 582 θ ∗ ( α ) .

Original: Assumption 3: (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .
Revised: Assumption 3: (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .

Original: Now we can estimate the error bound produced by Theorem 1.
Revised: Now we can estimate the error bound produced by Theorem 1.

Original: First, the error is due to the second-order Taylor 585 expansion on θ in Eq.
Revised: First, the error is due to the second-order Taylor 585 expansion on θ in Eq.

Original: We have the error
Revised: We have the error

Original: [Equation]
Revised: [Equation]

Original: where we should notice that the approximated validation loss L (ˆ θ ∗ , ˆ α ) is Taylor expansion of ˆ L in the pointˆ θ ∗ − ∆ θ .
Revised: where we should notice that the approximated validation loss L (ˆ θ ∗ , ˆ α ) is Taylor expansion of ˆ L in the pointˆ θ ∗ − ∆ θ .

Original: Since we set that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ
Revised: Since we set that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ

Original: [Equation]
Revised: [Equation]

Original: 1 ∗
Revised: 1 ∗

Original: [Equation]
Revised: [Equation]

Original: L ( θ ∗ ,
Revised: L ( θ ∗ ,

Original: [Equation]
Revised: [Equation]

Original: in Eq.
Revised: in Eq.

Original: 13, this error E could be easily bound.
Revised: 13, this error E could be easily bound.

Original: However, we should notice that our ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ
Revised: However, we should notice that our ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ

Original: [Equation]
Revised: [Equation]

Original: 1 ∗ ∂ L ( θ ∗ , ˆ α )
Revised: 1 ∗ ∂ L ( θ ∗ , ˆ α )

Original: ∂θ is an approximation, that θ ∗ + ∆ θ ̸ = ˆ θ ∗ ,and we define that θ ∗ + ∆ θ =
Revised: ∂θ is an approximation, that θ ∗ + ∆ θ ̸ = ˆ θ ∗ ,and we define that θ ∗ + ∆ θ =

Original: ˆ θ ∗ e and ∆ θ e = ˆ θ ∗ − ˆ θ ∗ e .
Revised: ˆ θ ∗ e and ∆ θ e = ˆ θ ∗ − ˆ θ ∗ e .

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: For the first part in Eq.
Revised: For the first part in Eq.

Original: (18), based on Assumption 2 that L ( θ, α ) is λ -strongly convex with θ around θ ∗ , wehave:
Revised: (18), based on Assumption 2 that L ( θ, α ) is λ -strongly convex with θ around θ ∗ , wehave:

Original: [Equation]
Revised: [Equation]

Original: 19) and
Revised: 19) and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When we consider that L ( θ ∗ , α ) is C L Lipschitz continuous with α , we have
Revised: When we consider that L ( θ ∗ , α ) is C L Lipschitz continuous with α , we have

Original: [Equation]
Revised: [Equation]

Original: Baesd on Eq.
Revised: Baesd on Eq.

Original: (20) and (21), we have
Revised: (20) and (21), we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As to the second part, when we assume that L ( θ, α ) is λ -convex near the local optimal, we also have
Revised: As to the second part, when we assume that L ( θ, α ) is λ -convex near the local optimal, we also have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ˆ θ ∗ is the true optimal, and ˆ θ ∗ e is the approximate local optimal when we utilize the implicit functiontheorem to approximate ∆ θ in Eq.
Revised: where ˆ θ ∗ is the true optimal, and ˆ θ ∗ e is the approximate local optimal when we utilize the implicit functiontheorem to approximate ∆ θ in Eq.

Original: 13.so that, we have
Revised: 13.so that, we have

Original: [Equation]
Revised: [Equation]

Original: Consider the local optimal that ∇L (ˆ θ ∗ , ˆ α ) = 0 , we have
Revised: Consider the local optimal that ∇L (ˆ θ ∗ , ˆ α ) = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Now, we need to bound (cid:13)(cid:13)(cid:13)
Revised: Now, we need to bound (cid:13)(cid:13)(cid:13)

Original: ∇L
Revised: ∇L

Original: (ˆ θ ∗ e , ˆ α ) (cid:13)(cid:13)(cid:13) .
Revised: (ˆ θ ∗ e , ˆ α ) (cid:13)(cid:13)(cid:13) .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since we utilize Eq.
Revised: Since we utilize Eq.

Original: (13) that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L
Revised: (13) that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L

Original: [Equation]
Revised: [Equation]

Original: θ ∗ , ˆ α )
Revised: θ ∗ , ˆ α )

Original: ∂θ to conduct the second-order Taylor expansion, 603 we have
Revised: ∂θ to conduct the second-order Taylor expansion, 603 we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the second row is calculated since (cid:82) ∇ θ L 2 ( θ ∗ + t ∗ ∆ θ, ˆ α ) ∗ ∆ θ dt = ∇ θ L ( θ ∗ + t ∗ ∆ θ, ˆ α ) .
Revised: where the second row is calculated since (cid:82) ∇ θ L 2 ( θ ∗ + t ∗ ∆ θ, ˆ α ) ∗ ∆ θ dt = ∇ θ L ( θ ∗ + t ∗ ∆ θ, ˆ α ) .

Original: The fifthrow is calculated as we assume that L is convex and twice-differentiable with C H .
Revised: The fifthrow is calculated as we assume that L is convex and twice-differentiable with C H .

Original: And we assume that the∂ L ( θ ∗ , ˆ α )
Revised: And we assume that the∂ L ( θ ∗ , ˆ α )

Original: ∂θ is bounded by C l .
Revised: ∂θ is bounded by C l .

Original: σ min is the smallest eigenvalue of Hessian matrix ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ .
Revised: σ min is the smallest eigenvalue of Hessian matrix ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ .

Original: In addition, when we consider a more tight bound, and consider a Taylor expansion on α , we have
Revised: In addition, when we consider a more tight bound, and consider a Taylor expansion on α , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the second row is obtained based on the local optimal ∂ L ( θ ∗ ,α
Revised: where the second row is obtained based on the local optimal ∂ L ( θ ∗ ,α

Original: ) ∂θ
Revised: ) ∂θ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have∥∇L (
Revised: we have∥∇L (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 30) and
Revised: 30) and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Based on Eq.
Revised: Based on Eq.

Original: (18), (22), and (31), we have
Revised: (18), (22), and (31), we have

Original: [Equation]
Revised: [Equation]

Original: The final error bound can be calculated as:
Revised: The final error bound can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: where K = C L λ ∗ ∥
Revised: where K = C L λ ∗ ∥

Original: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥
Revised: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥

Original: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .
Revised: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .

Original: □
Revised: □

Original: Derivation of Eq.
Revised: Derivation of Eq.

Original: In Section 4, when we consider a first-order Taylor expansion on α as we only applyan infinitesimal change of α , and we alsoconsider a second-order Taylor expansion on θ , we have
Revised: In Section 4, when we consider a first-order Taylor expansion on α as we only applyan infinitesimal change of α , and we alsoconsider a second-order Taylor expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: where the last row is obtained is obtained due to the local optimal ∂ L ( θ ∗ ,α ) ∂θ
Revised: where the last row is obtained is obtained due to the local optimal ∂ L ( θ ∗ ,α ) ∂θ

Original: [Equation]
Revised: [Equation]

Original: Theorem 2: Supposed that DARTS obtains the optimized architecture parameter α with supernet weights 619 θ ∗ after supernet training, and we pose an infinitesimal change on α .
Revised: Theorem 2: Supposed that DARTS obtains the optimized architecture parameter α with supernet weights 619 θ ∗ after supernet training, and we pose an infinitesimal change on α .

Original: Based on implicit function theorem andunder the assumption that the third-derivative of the loss function at optimum is zero or sufficiently small [4], 621 the change of validation performance can be estimated as:
Revised: Based on implicit function theorem andunder the assumption that the third-derivative of the loss function at optimum is zero or sufficiently small [4], 621 the change of validation performance can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.
Revised: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.

Original: Proof of Theorem 2: Although Section 3 provides an iterative solution to measure the operation importancebased Eq.
Revised: Proof of Theorem 2: Although Section 3 provides an iterative solution to measure the operation importancebased Eq.

Original: (5) through individually removing each operation, we need to calculate n times (the number of 625 all candidate operations in the supernet work) to estimate the importance of all candidate operations.
Revised: (5) through individually removing each operation, we need to calculate n times (the number of 625 all candidate operations in the supernet work) to estimate the importance of all candidate operations.

Original: Moreimportant, since removing one operation poses a considerable change on α ( α i → 0 ), the approximation error 627 produced by Eq.(5) is non-negligible which may affects the accuracy as stated by Corollary 1.
Revised: Moreimportant, since removing one operation poses a considerable change on α ( α i → 0 ), the approximation error 627 produced by Eq.(5) is non-negligible which may affects the accuracy as stated by Corollary 1.

Original: So, a morepractical solution to illustrate the importance of each operation is to estimate how the validation performance 629 will change after posing an infinitesimal change on α , a.k.a. operation sensitivity, and we can directly get theoperation importance with one calculation.
Revised: So, a morepractical solution to illustrate the importance of each operation is to estimate how the validation performance 629 will change after posing an infinitesimal change on α , a.k.a. operation sensitivity, and we can directly get theoperation importance with one calculation.

Original: When we consider that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ aftersupernet training, and we pose an infinitesimal change on α , we can conduct a first-order Taylor expansion on α 633 a second-order Taylor expansion on θ .
Revised: When we consider that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ aftersupernet training, and we pose an infinitesimal change on α , we can conduct a first-order Taylor expansion on α 633 a second-order Taylor expansion on θ .

Original: So that we have get the Eq.
Revised: So that we have get the Eq.

Original: (34) as stated in the above.
Revised: (34) as stated in the above.

Original: When we furtherconduct a second Taylor expansion on ˆ θ ∗ for ∂ L (ˆ θ ∗ ,α ) ∂α , we have
Revised: When we furtherconduct a second Taylor expansion on ˆ θ ∗ for ∂ L (ˆ θ ∗ ,α ) ∂α , we have

Original: [Equation]
Revised: [Equation]

Original: where the last row is obtained when we neglect the third and higher derivatives of the loss function withconsidering the chain rule that ∂ L ∂α =
Revised: where the last row is obtained when we neglect the third and higher derivatives of the loss function withconsidering the chain rule that ∂ L ∂α =

Original: ∂ L ∂θ ∗ ∂ ( θ ∗ ( α )) ∂α , and assume α is a local optimal for the continuous 637 magnitude optimization after architecture search.
Revised: ∂ L ∂θ ∗ ∂ ( θ ∗ ( α )) ∂α , and assume α is a local optimal for the continuous 637 magnitude optimization after architecture search.

Original: From the above, we can find that, the challenge is still that itis intractable to calculate ∆ θ .
Revised: From the above, we can find that, the challenge is still that itis intractable to calculate ∆ θ .

Original: Similar as before, when we consider implicit function and second-order Taylor 639 expansion on θ , we have
Revised: Similar as before, when we consider implicit function and second-order Taylor 639 expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: When we neglect the third and higher derivatives and the local optimal ∂ L ( θ ∗ ,α
Revised: When we neglect the third and higher derivatives and the local optimal ∂ L ( θ ∗ ,α

Original: [Equation]
Revised: [Equation]

Original: ∂θ
Revised: ∂θ

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Then, similar as before, when we apply a first-order Taylor expansion on ˆ α , with the chain-rule and neglect third 642 and higher derivatives, we have
Revised: Then, similar as before, when we apply a first-order Taylor expansion on ˆ α , with the chain-rule and neglect third 642 and higher derivatives, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: After applying Eq.
Revised: After applying Eq.

Original: (40) on Eq.
Revised: (40) on Eq.

Original: (36), we have
Revised: (36), we have

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: B Practical Implementation
Revised: B Practical Implementation

Original: To practically calculate the proposed influential magnitude as Definition 1, we need to estimate the InverseHessian-Vector products (IHVPs) H − 1 v , and v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .
Revised: To practically calculate the proposed influential magnitude as Definition 1, we need to estimate the InverseHessian-Vector products (IHVPs) H − 1 v , and v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .

Original: In the following, we first described two methods toestimate H − 1 v , and the following DARTS to described the practical calculation on v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .
Revised: In the following, we first described two methods toestimate H − 1 v , and the following DARTS to described the practical calculation on v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .

Original: B.1 Neumann series approximation for Lemma
Revised: B.1 Neumann series approximation for Lemma

Original: The most common method for the IHVPs is the Neumann series that, for ∥ I − A ∥ < 1 , we have:
Revised: The most common method for the IHVPs is the Neumann series that, for ∥ I − A ∥ < 1 , we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: However, it is impossible to guarantee that ∥ I − H ∥ < 1 .
Revised: However, it is impossible to guarantee that ∥ I − H ∥ < 1 .

Original: Different from [22], we consider H − 1 = γ ( γH ) − 1 .
Revised: Different from [22], we consider H − 1 = γ ( γH ) − 1 .

Original: When assuming H is bounded with σ max , (cid:13)(cid:13)(cid:13) ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < σ max .
Revised: When assuming H is bounded with σ max , (cid:13)(cid:13)(cid:13) ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < σ max .

Original: With γ < 1 σ max , we have (cid:13)(cid:13)(cid:13) I − γ ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < 1[29, 36].
Revised: With γ < 1 σ max , we have (cid:13)(cid:13)(cid:13) I − γ ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < 1[29, 36].

Original: When we conduct the Neumann series approximation for (cid:104) ∂ 2 L ∂θ∂θ (cid:105) − 1 in the optimal point, we have:(cid:20) ∂ 2 L
Revised: When we conduct the Neumann series approximation for (cid:104) ∂ 2 L ∂θ∂θ (cid:105) − 1 in the optimal point, we have:(cid:20) ∂ 2 L

Original: [Equation]
Revised: [Equation]

Original: (cid:21)
Revised: (cid:21)

Original: [Equation]
Revised: [Equation]

Original: when we only consider the first K terms and H is positive in θ ∗ (Please note that, generally, we could onlyguarantee H is positive when in optimal point).
Revised: when we only consider the first K terms and H is positive in θ ∗ (Please note that, generally, we could onlyguarantee H is positive when in optimal point).

Original: Accordingly, the Lemma 1 is derived.
Revised: Accordingly, the Lemma 1 is derived.

Original: As suggested by [49], a medium K is large enough to approximate IHVPs, where K = 2 obtains similar results 657 as K ⩾ 3 while with less computational cost.
Revised: As suggested by [49], a medium K is large enough to approximate IHVPs, where K = 2 obtains similar results 657 as K ⩾ 3 while with less computational cost.

Original: In this paper, we also observed a similar phenomenon and also set
Revised: In this paper, we also observed a similar phenomenon and also set

Original: K = 2 by default in our Neumann series approximation based IHVPs.
Revised: K = 2 by default in our Neumann series approximation based IHVPs.

Original: B.2 Sherman-Morrison formula for Lemma 2
Revised: B.2 Sherman-Morrison formula for Lemma 2

Original: Apart from the Neumann series approximation, we could also follow the Sherman-Morrison formula to exactly 661 calculate the inverse of Hessian when we consider the empirical Fisher to replace the Fisher matrix.
Revised: Apart from the Neumann series approximation, we could also follow the Sherman-Morrison formula to exactly 661 calculate the inverse of Hessian when we consider the empirical Fisher to replace the Fisher matrix.

Original: First, weknow that the Hessian equals to fisher information matrix in the optimal point.
Revised: First, weknow that the Hessian equals to fisher information matrix in the optimal point.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We name the r = 1 n ( y i − f ( x i ))
Revised: We name the r = 1 n ( y i − f ( x i ))

Original: ∇ θ ∇ θ f , which is the residual contribution to Hessian H .
Revised: ∇ θ ∇ θ f , which is the residual contribution to Hessian H .

Original: At the end of training, 664 if we can achieve zero training error, namely, y i =
Revised: At the end of training, 664 if we can achieve zero training error, namely, y i =

Original: f
Revised: f

Original: ( x i ) , the residual will converge to zero.
Revised: ( x i ) , the residual will converge to zero.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Please note that the dimension of ∇ θ f is M ∗ m where m is the number of classes (output dimension) and M is 666 the number of parameters.
Revised: Please note that the dimension of ∇ θ f is M ∗ m where m is the number of classes (output dimension) and M is 666 the number of parameters.

Original: Practically, we usually consider an empirical Fisher to approximate the Fisher matrixthat
Revised: Practically, we usually consider an empirical Fisher to approximate the Fisher matrixthat

Original: [Equation]
Revised: [Equation]

Original: So, in the remaining text, the F indicates the empirical Fisher for simplicity.
Revised: So, in the remaining text, the F indicates the empirical Fisher for simplicity.

Original: The Sherman-Morrison formula is defined as:
Revised: The Sherman-Morrison formula is defined as:

Original: [Equation]
Revised: [Equation]

Original: When we define that F n = 1 N (cid:80) nn =1 ∇ θ L ( y n , f ( x n ))( ∇ θ L ( y n , f ( x n ))
Revised: When we define that F n = 1 N (cid:80) nn =1 ∇ θ L ( y n , f ( x n ))( ∇ θ L ( y n , f ( x n ))

Original: T , we can notice that the empirical 671 Fisher can be represented as F
Revised: T , we can notice that the empirical 671 Fisher can be represented as F

Original: N , and we have:
Revised: N , and we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where L = ℓ + η R ( θ ) that ℓ is a cross-entropy loss and R is the regularization term, and η is the weight-decay, 673 F 0 = ηI .
Revised: where L = ℓ + η R ( θ ) that ℓ is a cross-entropy loss and R is the regularization term, and η is the weight-decay, 673 F 0 = ηI .

Original: L n is the loss for the n-th batch dataset.
Revised: L n is the loss for the n-th batch dataset.

Original: And based on the Sherman-Morrison formula, we have
Revised: And based on the Sherman-Morrison formula, we have

Original: [Equation]
Revised: [Equation]

Original: Please notice that, whether using the Neumann series or the Sherman-Morrison formula, we always need 675 to calculate the Inverse-Hessian Vector Products (IHVPs) rather than only the Inverse-Hessian.
Revised: Please notice that, whether using the Neumann series or the Sherman-Morrison formula, we always need 675 to calculate the Inverse-Hessian Vector Products (IHVPs) rather than only the Inverse-Hessian.

Original: It is easy toimplement the IHVPs based on the Neumann series through the Hessian-vector products based on Lemma 1.
Revised: It is easy toimplement the IHVPs based on the Neumann series through the Hessian-vector products based on Lemma 1.

Original: In 677 the following, we discuss how to implement the IHVPs based on Sherman-Morrison formula.
Revised: In 677 the following, we discuss how to implement the IHVPs based on Sherman-Morrison formula.

Original: When we assumethat we have n + 1 batches datasets, we have
Revised: When we assumethat we have n + 1 batches datasets, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We can find that, to solve the previous equation, we need to first recurrently calculate r N = F − 1 N − 1 ∇ θ L N .
Revised: We can find that, to solve the previous equation, we need to first recurrently calculate r N = F − 1 N − 1 ∇ θ L N .

Original: [Equation]
Revised: [Equation]

Original: We first need to calculate the r n = F − 1 n − 1 ∇ θ L n .
Revised: We first need to calculate the r n = F − 1 n − 1 ∇ θ L n .

Original: [Equation]
Revised: [Equation]

Original: So, we can first iterative calculate r n , and then calculate F − 1 n v .
Revised: So, we can first iterative calculate r n , and then calculate F − 1 n v .

Original: B.3 Practical Calculation on Matrix-Vector product
Revised: B.3 Practical Calculation on Matrix-Vector product

Original: Apart from the IHVPs, we also need to calculate v a ∗ ∂ 2 L ∂α∂θ and v m ∗ ∂ 2 L ∂θ∂α , which are called as matrix-vectorproduct and computational-expensive.
Revised: Apart from the IHVPs, we also need to calculate v a ∗ ∂ 2 L ∂α∂θ and v m ∗ ∂ 2 L ∂θ∂α , which are called as matrix-vectorproduct and computational-expensive.

Original: Following DARTS [27], we also utilize the finite difference approximation 685 for the practical calculation.
Revised: Following DARTS [27], we also utilize the finite difference approximation 685 for the practical calculation.

Original: Let ξ be a small scalar, and consider the function ∂
Revised: Let ξ be a small scalar, and consider the function ∂

Original: L ( θ,α + ξ ∗ v a ) ∂θ with Taylor expansion on α , we have
Revised: L ( θ,α + ξ ∗ v a ) ∂θ with Taylor expansion on α , we have

Original: [Equation]
Revised: [Equation]

Original: when we only consider the first two terms, we have
Revised: when we only consider the first two terms, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, the v m ∗ ∂ 2 L ∂θ∂α can be approximated as:
Revised: Similarly, the v m ∗ ∂ 2 L ∂θ∂α can be approximated as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: C Why perturbation-based method fails
Revised: C Why perturbation-based method fails

Original: [Table]
Revised: [Table]

Original: The perturbation-based method, e.g. the leave-one-out retraining [22], is an intuitive way to measure the 691 importance of every candidate operations.
Revised: The perturbation-based method, e.g. the leave-one-out retraining [22], is an intuitive way to measure the 691 importance of every candidate operations.

Original: More detailed, we can individually remove every candidate operationin each edge
Revised: More detailed, we can individually remove every candidate operationin each edge

Original: [Table]
Revised: [Table]

Original: and then fine-tune the supernet to the validation performance drop L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: and then fine-tune the supernet to the validation performance drop L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: However,
Revised: However,

Original: DARTS-PT abandons the retraining part that directly measure the validation performance drop.
Revised: DARTS-PT abandons the retraining part that directly measure the validation performance drop.

Original: An concern will 694 be raised here as the pruned supernet is not trained to optimal
Revised: An concern will 694 be raised here as the pruned supernet is not trained to optimal

Original: [Table]
Revised: [Table]

Original: and it is inaccurate to measure the performancedrop by L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: and it is inaccurate to measure the performancedrop by L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: More importantly, since the skip-connection ’s magnitude is generally muchhigher than other candidate operations [Table] and directly removing skip-connections usually greatly deteriorates the 697 supernet performance if we do not fine-tune the supernet.
Revised: More importantly, since the skip-connection ’s magnitude is generally muchhigher than other candidate operations, and directly removing skip-connections usually greatly deteriorates the 697 supernet performance if we do not fine-tune the supernet.

Original: This is also the reason why we empirically find that
Revised: This is also the reason why we empirically find that

Original: DARTS-PT still prefers skip-connection than other operations.
Revised: DARTS-PT still prefers skip-connection than other operations.

Original: Figure 7 (e) plots the searched architecture in 699 NAS-Bench-201 for DARTS-PT, which contains 3 skip-connections .
Revised: Figure 8 (a) plots the searched architecture in 699 NAS-Bench-201 for DARTS-PT, which contains 3 skip-connections .

Original: In contrast, the architecture selected by ourinfluential magnitude only contains 1 skip-connection .
Revised: In contrast, the architecture selected by ourinfluential magnitude only contains 1 skip-connection .

Original: This phenomenon also exists in the experiments on the 701 search space S1 and S2 of [45], where although DARTS-PT can avoid selecting all skip-connections in all edges,its searched architectures still contains intensive skip-connections .
Revised: This phenomenon also exists in the experiments on the 701 search space S1 and S2 of [45], where although DARTS-PT can avoid selecting all skip-connections in all edges,its searched architectures still contains intensive skip-connections .

Original: D Comparison with SOTAs on NAS-Bench-201
Revised: D Comparison with SOTAs on NAS-Bench-201

Original: We also compared our DARTS-IM with different NAS baselines on the NAS-Bench-201 search space.
Revised: We also compared our DARTS-IM with different NAS baselines on the NAS-Bench-201 search space.

Original: Table 4 705 reports the comprehensive validation and test results on all three datasets, including CIFAR-10, CIFAR-100,and ImageNet.
Revised: Table 4 705 reports the comprehensive validation and test results on all three datasets, including CIFAR-10, CIFAR-100,and ImageNet.

Original: As shown, DARTS is not a valid method for differentiable architecture search, which even 707 leads to poorer results than random baseline.
Revised: As shown, DARTS is not a valid method for differentiable architecture search, which even 707 leads to poorer results than random baseline.

Original: Although considering other supernet training methods can alsoimprove the performance, we can also found that, with only replacing the operation selection metric with our 709 influential magnitude, our DARTS-IM can achieve more competitive results.
Revised: Although considering other supernet training methods can alsoimprove the performance, we can also found that, with only replacing the operation selection metric with our 709 influential magnitude, our DARTS-IM can achieve more competitive results.

Original: Moreover, our DARTS-IM achievesnear-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR-100, and ImageNet, 711 respectively, in a single run.
Revised: Moreover, our DARTS-IM achievesnear-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR-100, and ImageNet, 711 respectively, in a single run.

Original: Our DARTS-IF, which follows the perturbation-based selection as DARTS-PTwhile with an additional approximation term, also outperforms DARTS-PT that again verifies the effectiveness 713 of the influence function explanation.
Revised: Our DARTS-IF, which follows the perturbation-based selection as DARTS-PTwhile with an additional approximation term, also outperforms DARTS-PT that again verifies the effectiveness 713 of the influence function explanation.

Original: In addition, due to the requirement of fine-tuning in the perturbation-basedparadigm, DARTS-PT cost much more than our DARTS-IM.
Revised: In addition, due to the requirement of fine-tuning in the perturbation-basedparadigm, DARTS-PT cost much more than our DARTS-IM.

Original: Compared with the perturbation-based selection 715 paradigm, we found that the magnitude paradigm is more efficient and reliable 1 .
Revised: Compared with the perturbation-based selection 715 paradigm, we found that the magnitude paradigm is more efficient and reliable 1 .

Original: E Comparison with SOTAs on DARTS space
Revised: E Comparison with SOTAs on DARTS space

Original: DARTS search space is a much more complicated search space than NAS-Bench-201 and NAS-Bench-1shot1, 

Revised: DARTS search space is a much more complicated search space than NAS-Bench-201 and NAS-Bench-1shot1,

Original: and it is intractable to get the ground truth for all architectures in this space.
Revised: and it is intractable to get the ground truth for all architectures in this space.

Original: Generally, most existing worksonly report the best searched architecture, making the reproducibility of architecture search weak in this space.
Revised: Generally, most existing worksonly report the best searched architecture, making the reproducibility of architecture search weak in this space.

Original: For fair comparison, we also follow the common setting [27, 43] to conduct the architecture search on the supernet
Revised: For fair comparison, we also follow the common setting [27, 43] to conduct the architecture search on the supernet

Original: [Table]
Revised: [Table]

Original: “Param" is the model size when applied on CIFAR-10, while “ + × " (the number of multiply-add operations) is calculated based on the ImageNet dataset.
Revised: “Param" is the model size when applied on CIFAR-10, while “ + × " (the number of multiply-add operations) is calculated based on the ImageNet dataset.

Original: In our experiments, we adjust the number of initial filters to restrict “ + × " to be less than 600M on the ImageNet and CIFAR10.
Revised: In our experiments, we adjust the number of initial filters to restrict “ + × " to be less than 600M on the ImageNet and CIFAR10.

Original: † means the computational time is calculated with the same environment as [40].
Revised: † means the computational time is calculated with the same environment as [40].

Original: The unit in “Search Cost" is
Revised: The unit in “Search Cost" is

Original: GPU day.
Revised: GPU day.

Original: [Table]
Revised: [Table]

Original: ImageNet and to comparable model size with SOTAs on CIFAR10.
Revised: ImageNet and to comparable model size with SOTAs on CIFAR10.

Original: Table 5 presents the comparison results with the SOTA differentiable NAS methods.
Revised: Table 5 presents the comparison results with the SOTA differentiable NAS methods.

Original: As shown, our DARTS-IM 726 improves the DARTS baselines by a large margin, in terms of test error on CIFAR-10, CIFAR-100, and ImageNet,respectively, again verifying the effectiveness of the proposed influential magnitude.
Revised: As shown, our DARTS-IM 726 improves the DARTS baselines by a large margin, in terms of test error on CIFAR-10, CIFAR-100, and ImageNet,respectively, again verifying the effectiveness of the proposed influential magnitude.

Original: More interesting, instead of 728 changing the complicated supernet training part, we found that simply replacing the magnitude with the proposedinfluential magnitude could consistently improve the performance of DARTS with only additional hundred 730 seconds computational time.
Revised: More interesting, instead of 728 changing the complicated supernet training part, we found that simply replacing the magnitude with the proposedinfluential magnitude could consistently improve the performance of DARTS with only additional hundred 730 seconds computational time.

Original: Different from the DARTS-PT which contains the supernet retraining that greatlyincreases the search time, our DARTS-IM only poses additional hundred seconds in DARTS space.
Revised: Different from the DARTS-PT which contains the supernet retraining that greatlyincreases the search time, our DARTS-IM only poses additional hundred seconds in DARTS space.

Original: We could 732 also find that, although the best CIFAR-10 test error by DARTS-PT is comparable with our DARTS-IM, our bestsearched architecture obtains lower test error in CIFAR-100 and ImageNet, showing better transferability.
Revised: We could 732 also find that, although the best CIFAR-10 test error by DARTS-PT is comparable with our DARTS-IM, our bestsearched architecture obtains lower test error in CIFAR-100 and ImageNet, showing better transferability.

Original: F Analyse the inverse Hessian approximation
Revised: F Analyse the inverse Hessian approximation

Original: In this section, we detailed discuss the effects of different inverse Hessian approximation methods on the 736 proposed influence magnitude.
Revised: In this section, we detailed discuss the effects of different inverse Hessian approximation methods on the 736 proposed influence magnitude.

Original: Apart from the two devised method, Neumann series and Sherman-Morrisonformula, we also consider two commonly-used approaches in approximating the inverse Hessian, identity 738 approximation and diagonal approximation, which consider the Hessian matrix as an identity matrix anddiagonal matrix, respectively.
Revised: Apart from the two devised method, Neumann series and Sherman-Morrisonformula, we also consider two commonly-used approaches in approximating the inverse Hessian, identity 738 approximation and diagonal approximation, which consider the Hessian matrix as an identity matrix anddiagonal matrix, respectively.

Original: We conduct a series of experiments on NAS-Bench-201 to analyze different 740 approximation methods, and all comparison results are reported in Table 6.
Revised: We conduct a series of experiments on NAS-Bench-201 to analyze different 740 approximation methods, and all comparison results are reported in Table 6.

Original: As shown, even with the identityapproximation, our influential magnitude I M can achieve satisfying results compared with the random baselineand two heuristic methods.
Revised: As shown, even with the identityapproximation, our influential magnitude I M can achieve satisfying results compared with the random baselineand two heuristic methods.

Original: This result indicates that the second-order information ∂ 2 L ( θ ∗ ,α ) ∂α∂θ and ∂ 2 L ( θ ∗ ,α ) ∂θ∂α can 743 provide useful information for operation strength estimation.
Revised: This result indicates that the second-order information ∂ 2 L ( θ ∗ ,α ) ∂α∂θ and ∂ 2 L ( θ ∗ ,α ) ∂θ∂α can 743 provide useful information for operation strength estimation.

Original: More interesting, when considering the diagonal
Revised: More interesting, when considering the diagonal

Original: Hessian information, we found that our DARTS-IM-D could also achieve excellent performance.
Revised: Hessian information, we found that our DARTS-IM-D could also achieve excellent performance.

Original: The results of
Revised: The results of

Original: [Figure]
Revised: [Figure]

Original:  identity approximation and diagonal approximation demonstrate the robustness of our influential magnitude, 746 which can still achieve excellent results with different inverse Hessian approximation methods.
Revised: [Figure] identity approximation and diagonal approximation demonstrate the robustness of our influential magnitude, 746 which can still achieve excellent results with different inverse Hessian approximation methods.

Original: However, we also found that an incorrect approximation may lead to poor performance.
Revised: However, we also found that an incorrect approximation may lead to poor performance.

Original: Similar to Figure 2, 748 we plots the full NAS-Bench-201 results of our influential magnitude under different inverse Hessian matrixapproximation methods along with the number of batches N , on the CIFAR10, CIFAR100, and ImageNet, 750 respectively.
Revised: Similar to Figure 2, 748 we plots the full NAS-Bench-201 results of our influential magnitude under different inverse Hessian matrixapproximation methods along with the number of batches N , on the CIFAR10, CIFAR100, and ImageNet, 750 respectively.

Original: We compared our DARTS-IM with the DARTS baseline and also the DARTS-PT whose results aredirectly obtained from [40].
Revised: We compared our DARTS-IM with the DARTS baseline and also the DARTS-PT whose results aredirectly obtained from [40].

Original: Please note that, DARTS-PT smooth the line extremely with only report the tendency 752 of the architecture selection during the search.
Revised: Please note that, DARTS-PT smooth the line extremely with only report the tendency 752 of the architecture selection during the search.

Original: We can found that the Neumann series approximation could notobtain stable results, especially when the number of batches increases that the performance of Neumann series 754 even performs more poorly than identity approximation.
Revised: We can found that the Neumann series approximation could notobtain stable results, especially when the number of batches increases that the performance of Neumann series 754 even performs more poorly than identity approximation.

Original: As discussed in Section 6.1, these results also verifythat the stochastic Neumann series estimation for IHVP is somehow erroneous.
Revised: As discussed in Section 6.1, these results also verifythat the stochastic Neumann series estimation for IHVP is somehow erroneous.

Original: Generally, we found that the 756 Sherman-Morrison formula is the most practical in estimating the inverse Hessian for our influential magnitude.
Revised: Generally, we found that the 756 Sherman-Morrison formula is the most practical in estimating the inverse Hessian for our influential magnitude.

Revised: Different from Sherman-Morrison formula that only has a hyperparameter N to be tuned and η is fixed in the 758 optimizer, the Neumann series approximation has a key hyperparameter γ [29, 36] to be tuned, which is assumed

Revised: [Figure]

Revised: to be small enough that γ < 1 σ max in Appendix B.1.

Original: G Visualization of Searched Architectures on Different Spaces
Revised: In this way, we conduct the hyperparameter analysis on γfor our DARTS-IM when employing the Neumann series approximation. Figure 6 compares the performance of 761 DARTS-IM with different γ on the NAS-Bench-201. As shown, the Neumann series approximation is sensitiveto the γ , where a large γ significantly deteriorates the performance of DARTS-IM. For example, a γ = 0 .

Original:  In Figure 7 and 8, we visualize all searched architectures by DARTS baselines, DARTS-PT and also our 759 DARTS-IM on different search space, including NAS-Bench-201 [12], tool search spaces proposed by [45], and
Revised: 763 even makes our DART-IM similar to the performance of the random baseline. Compared with the Neumannseries, we found that the Sherman-Morrison approximation is more robust. G Visualization of Searched Architectures on Different Spaces In Figure 8 and 9, we visualize all searched architectures by DARTS baselines, DARTS-PT and also our 767 DARTS-IM on different search space, including NAS-Bench-201 [12], tool search spaces proposed by [45], and

Original: DARTS search space [27].
Revised: DARTS search space [27].

Original: First, we analyze these architectures searched on NAS-Bench-201 [12] and tool search spaces proposed by 762 [40, 45], since which helps us to clearly demonstrate the robustness of different architecture selection methods.
Revised: First, we analyze these architectures searched on NAS-Bench-201 [12] and tool search spaces proposed by 770 [40, 45], since which helps us to clearly demonstrate the robustness of different architecture selection methods.

Original: As observed by [40, 45], the most notable drawback of DARTS baseline is the robustness and generalization, 764 where DARTS fails to select intensive skip-connections with the search progressing.
Revised: As observed by [40, 45], the most notable drawback of DARTS baseline is the robustness and generalization, 772 where DARTS fails to select intensive skip-connections with the search progressing.

Original: Figure 7 (e) and (i) plots thesearched architectures on NAS-Bench-201, respectively.
Revised: Figure 8 (a-c) plots thesearched architectures on NAS-Bench-201, respectively.

Original: As shown, the DARTS baseline selects skip-connection 766 for all edges, while DARTS-PT can find a valid architecture with only 3 skip-connections when considering aperturbation-based selection paradigm.
Revised: As shown, the DARTS baseline selects skip-connection 774 for all edges, while DARTS-PT can find a valid architecture with only 3 skip-connections when considering aperturbation-based selection paradigm.

Original: More inspiring, the proposed influential magnitude ( I M ) can select more 768 competitive architecture which also only contains one skip-connection .Please note that, the trained supernet [Figure] [Table]
Revised: More inspiring, the proposed influential magnitude ( I M ) can select more 776 competitive architecture which also only contains one skip-connection .

Original: keeps identical for DARTS, DARTS-PT

Original: [Table] and our DARTS-IM, which only adopt different architecture selection 770 methods, e [Table] g., argmax , perturbation-based selection, and the proposed influential magnitude.
Revised: Please note that, the trained supernetkeeps identical for DARTS, DARTS-PT, and our DARTS-IM, which only adopt different architecture selection 778 methods, e.g., argmax , perturbation-based selection, and the proposed influential magnitude.

Original: Similarly, we can also observe this superiority of our I M in the tool search spaces proposed by [40, 45], 772 especially in the search space S1 and S2.
Revised: Similarly, we can also observe this superiority of our I M in the tool search spaces proposed by [40, 45], 780 especially in the search space S1 and S2.

Original: We can find that, our DARTS-IM clearly outperforms DARTS and
Revised: We can find that, our DARTS-IM clearly outperforms DARTS and

Original: DARTS-PT.
Revised: DARTS-PT.

Original: Although DARTS-PT can partially relieves this instability, it still selects intensive skip-connections .
Revised: Although DARTS-PT can partially relieves this instability, it still selects intensive skip-connections .

Original: In the contrary, our DARTS-IM generally prefers convolutional operations.
Revised: In the contrary, our DARTS-IM generally prefers convolutional operations.

Original: However, although our DARTS-IMoutperforms DARTS baseline in the search space S3, we found that DARTS-PT achieves excellent results in 776 this space.
Revised: However, although our DARTS-IMoutperforms DARTS baseline in the search space S3, we found that DARTS-PT achieves excellent results in 784 this space.

Original: One potential is that this space contains intensive none operations which is naturally removed byperturbation-based selection, since removing none will not affect the supernet performance.
Revised: One potential is that this space contains intensive none operations which is naturally removed byperturbation-based selection, since removing none will not affect the supernet performance.

Original: We also conduct778 the  architecture search on the search space S4, while we found that our DARTS-IM also performs poorly as DARTS that select some noise operations.
Revised: We also conduct the 786 architecture search on the search space S4, while we found that our DARTS-IM also performs poorly as DARTSthat select some noise operations.

Original: One underlying reason is that it is impossible to train the supernet to780 optimal  in this space since the noise operation will return a random noise regardless of the input.
Revised: One underlying reason is that it is impossible to train the supernet to optimal 788 in this space since the noise operation will return a random noise regardless of the input.

Original: We also showthe comparison results in the Table 7 with different DARTS baselines. We directly report the results of DARTS 782 and DARTS-ES from [45], which contain the mean and std.
Revised: In addition, DARTS-PTonly locally compares the operations’ strength in each edge, and a fine-tuning is needed after every step of edge 790 discretization.

Original: The results of DARTS-PT is also inherited from theoriginal paper [40], which only report the best single run. In Table 7, we report the average and best single run 784 of our DARTS-IM together for better comparison.
Revised: Differently, our DARTS-IM in this paper consider a global comparison among all operations in asupernet, which may bring some negative operation coupling. To further improve our proposed DARTS-IM, 792 a simple and straight approach is to gradually discretize edges in a supenet with fine-tuning, which we leave [Table]

Original: Then, we also visualized our searched architectures in the DARTS search space.
Revised: our future work.

Original: Interesting, compared with the 786 DARTS baseline, architecture searched by our DARTS-IM contains several sep_conv 5 × 5 operations in thenormal cell, where this phenomenon has never been observed by existing popular DARTS based algorithms. Similarly, we also found that our reduction cell contains several sep_conv 5 × 5 operations, which is also in
Revised: We also show the comparison results in the Table 7 with different DARTS baselines. We 794 directly report the results of DARTS and DARTS-ES from [45], which contain the mean and std.

Original: [Figure] output [Figure] [Table]
Revised: The results of

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: DARTS-PT is also inherited from the original paper [40], which only report the best single run.

Original: c_{k-2}

Original: [Figure]

Original: _connect

Original: [Figure]

Original: _connect c_{k-1} sep_conv_

Original: [Figure] skip_ [Figure] sep_
Revised: In Table 7, we 796 report the average and best single run of our DARTS-IM together for better comparison.

Original: [Figure]

Original: (g

Original: [Figure] PT [Figure] [Figure]
Revised: Then, we also visualized our searched architectures in the DARTS search space.

Original: [Figure]

Original: c_{k

Original: [Figure] [Figure] [Figure] [Figure]
Revised: Interesting, compared with the 798 DARTS baseline, architecture searched by our DARTS-IM contains several sep_conv 5 × 5 operations in thenormal cell, where this phenomenon has never been observed by existing popular DARTS based algorithms.

Original: j

Original: [Figure] S [Figure] line with another influence based method DARTS-PT and PC-DARTS.
Revised: Similarly, we also found that our reduction cell contains several sep_conv 5 × 5 operations, which is also inline with another influence based method DARTS-PT and PC-DARTS.

Original: More important, different from most 790 existing works that the normal cell generally contains more parameters than the reduction cell, we found that oursearched architecture prefers more parameters in the reduction cell.
Revised: More important, different from most 802 existing works that the normal cell generally contains more parameters than the reduction cell, we found that oursearched architecture prefers more parameters in the reduction cell.

Original: This observation is more consistent with the 792 common experience in the network pruning [24, 25, 39], where we found that most pruning methods will retainmore parameters in the reduction cell and prune more redundant parameters in the normal cell.
Revised: This observation is more consistent with the 804 common experience in the network pruning [24, 25, 39], where we found that most pruning methods will retainmore parameters in the reduction cell and prune more redundant parameters in the normal cell.

Original: H Overall Algorithm Framework Description
Revised: H Overall Algorithm Framework Description

Original: In this paper, we proposed two different approaches for the operation selection in DARTS, called DARTS-IF 796 (described in Sec.3) and DARTS-IM (described in Sec.4).
Revised: In this paper, we proposed two different approaches for the operation selection in DARTS, called DARTS-IF 808 (described in Sec.3) and DARTS-IM (described in Sec.4).

Original: This two approaches are based on the influencefunctions to measure the operation importance, where the main different is that DARTS-IF removes a candidate 798 operation in each step to measure the loss change in a perturbation manner, while DARTS-IM calculate the losssensitivity when applying an infinitesimal change on α .
Revised: This two approaches are based on the influencefunctions to measure the operation importance, where the main different is that DARTS-IF removes a candidate 810 operation in each step to measure the loss change in a perturbation manner, while DARTS-IM calculate theloss sensitivity when applying an infinitesimal change on α .

Original: The framework of our DARTS-IF and DART-IM  are800 sketched in Algorithm 1 and 2.
Revised: The framework of our DARTS-IF and DART-IM 812 are sketched in Algorithm 1 and 2.

Original: Our DARTS-IF shares a similar perturbation paradigm as DARTS-PT, whilewithout any fine-tuning.
Revised: Our DARTS-IF shares a similar perturbation paradigm as DARTS-PT,while without any fine-tuning.

Original: Rather than using the validation performance drop to indicate the operation  importance,802 DARTS-IF leverages influence functions to predict the loss change.
Revised: Rather than using the validation performance drop to indicate the operation 814 importance, DARTS-IF leverages influence functions to predict the loss change based on Eq.

Revised: In our practicalimplementation, we consider the Neumann series as described in Lemma 1, or the Sherman-Morrison formulaas described in Lemma 2, to approximate the IHPV ∂

Original: Different from DARTS-IF and DARTS-PT,our DARTS-IM is with a similar paradigm as DARTS, while whose operation [-importance is calculated based on 
-]
Revised: 2 L ( θ ∗ , ˆ α ) [Table] [Table] More important, different from 817 DARTS-PT that gradually discretize edges and finetune the supernet, our DARTS-IF only approximates theeffect of independently removing one candidate operation [-importance is calculated based on 
-]for each edge.

Original:  I 
Revised: In this way, we can get the strength for 819 all candidate operations under every edge in a supernet. Different from DARTS-IF and DARTS-PT, our DARTS-IM is with a similar paradigm as DARTS, while 821 whose operation importance is calculated based on I M , rather that the optimized α . In addition, the operationimportance in our DARTS-IM can be calculated by one-shot, rather than iteratively going through all candidates 823 operations in a supernet as perturbation paradigm. In calculating the influence magnitude in our DARTS-IM or the loss change in DARTS-IF, we have three 825 more hyperparameters, γ for Neumann series approximation, η for Sherman-Morrison approximation, andbatch size N for both.

Original: M , rather that the optimized α .
Revised: In our experiments, we set γ = 0 .

Original: In addition, the operation importance in our DARTS-IM can be calculated by 805 one-shot, rather than iteratively going through all candidates operations in a supernet as perturbation paradigm.
Revised: 001 that is same as the learning rate in optimizing 827 θ , and η is the weight decay for regularization, which both can be obtained form the optimizer by default. In the hyperparameter analysis in Figure 2, we found that N = 10 is enough for both Neumann series and 829 Sherman-Morrison approximation. input 1 skip

Original: c_{k-2}
Revised: [Figure] connect 2 nor_conv [Figure] output skip_connect nor

Revised: [Figure]

Revised: [Figure]

Revised: skip_connect nor_conv_3x (a

Revised: [Figure]

Revised: DARTS

Revised: [Figure]

Revised: PT: NAS-BENCH

Original: [Figure]
Revised: [Figure]

Original: skip
Revised: input 1 nor_conv_3x 2 nor_conv_3x output skip_connect nor_conv_1x nor_conv_3x nor_conv_3x (b) DARTS-IM: NAS-BENCHinput 1 skip_connect 2 skip_connect output skip_connect skip_connect skip_connect skip_connect (c) DARTS: NAS-BENCHc_{k-2} 1 skip_connect 2 skip_connect 3 sep_conv_3x dil_conv_5x5 dil_conv_3x dil_conv_3x c_{k} dil_conv_3x dil_conv_3x c_{k- 1 } (

Original: [Figure]
Revised: [Figure]

Revised: DARTS-IM: S c_{k-2} 1sep_conv_ 3x  sep_conv_3x  sep_conv_3x sep_conv_3x3 sep_conv_3x

Original:  c_{k-1} 
Revised: sep_conv_3x c_{k} sep_conv_3x sep_conv_3x c_{k-2} (e) DARTS-IM: S c_{k-2}   skip_connect skip_connect c_{k-1} skip_connect skip_connect 3 sep_conv_3x skip_connect skip_connect

Revised: skip_connect c_{k} (f) DARTS-IM: S  1 skip_connect 2 skip_connect dil_conv_3x skip_connec t skip_connect c_{k} skip_connect 3 dil_conv_3x dil_conv_5x c_{k- 2 } c_{k-1}

Revised: [Figure]

Revised: c_{k

Original: [Figure]
Revised: [Figure]

Original: sep
Revised: 1: Input : A pretrained supernet after bi-level training process ( θ ∗ , α ), candidate operations for each edge O , and set of edges E from the supernet. 2: output: A discrete architecture α ∗ . 3: for e ∈ E do 4: do 5:

Revised: Remove candidate operation o from

Revised: Calculate the predictive loss chance ∆ L o

Revised: Eq

Revised: ), that ∆ L o,e

Revised: L

Revised: ˆ

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: Restore

Revised: ; 8: end

Revised: 9: end

Revised: 10: Apply argmax on the operation strength

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Table]
Revised: c

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: c

Original: [Table]
Revised: [Figure]

Revised: sep

Original: [Table]
Revised: [Figure]

Original: c_{ [Table] [Table] [Table]
Revised: sep

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure][Table] [Figure] [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: c_{k
Revised: c_{

Original: [Figure]
Revised: [Figure]

Original: 2} 0 avg [Figure] [Table] c_{k-1} sep_conv_5x 1 max_pool_3x  dil_conv_5x skip_connect 3 sep_conv_5x c_{k} max_pool_3x skip_connect (f) DARTS-PT: Reduction Cell c_{k-2} skip_connect 1 sep_conv_3x3sep_conv_5x5avg_pool_3x c_{k-1} sep_conv_3x dil_conv_3x3 sep_conv_3x dil_conv_3x
Revised: c_{k-1} sep_conv_5x 1 max_pool_3x  dil_conv_5x skip_connect 3 sep_conv_5x c_{k} max_pool_3x skip_connect (f) DARTS-PT: Reduction Cell c_{k-2} skip_connect 1 sep_conv_3x3sep_conv_5x5avg_pool_3x c_{k-1} sep_conv_3x dil_conv_3x3 sep_conv_3x dil_conv_3x

Original: c_{k} (g) PC-DARTS:
Revised: c_{k} (g) PC-DARTS:

Original: Normal Cell c_{k-2} max_pool_3x  sep_conv_3x c_{k-1} sep_conv_5x  sep_conv_5x 3 sep_conv_3x sep_conv_5x sep_conv_3x c_{k} sep_conv_3x (h) PC-DARTS:
Revised: Normal Cell c_{k-2} max_pool_3x  sep_conv_3x c_{k-1} sep_conv_5x  sep_conv_5x 3 sep_conv_3x sep_conv_5x sep_conv_3x c_{k} sep_conv_3x (h) PC-DARTS:

Original: Reduction Cellend end for 11: Apply argmax on the operation strength ∆ L and derive the discrete architecture α ∗ accordingly. In calculating the influence magnitude in our DARTS-IM or the loss change in DARTS-IF, we have three more 807 hyperparameters, γ for [Equation]
Revised: Reduction Cell