Original: Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations
Revised: Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations

Original: Abstract
Revised: Abstract

Original: The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity.
Revised: The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity.

Original: DARTSleverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handledwith gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes.
Revised: DARTSleverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handledwith gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes.

Original: The vanilla DARTS assumes the optimized magnitudes reflect the importanceof operations, while more recent works find this naive assumption leads to poorgeneralization and is without any theoretical guarantees.
Revised: The vanilla DARTS assumes the optimized magnitudes reflect the importanceof operations, while more recent works find this naive assumption leads to poorgeneralization and is without any theoretical guarantees.

Original: In this work, we leverageinfluence functions, the functional derivatives of the loss function, to theoreticallyreveal the operation selection part in DARTS and estimate the candidate operationimportance by approximating its influence on the supernet with Taylor expansions.
Revised: In this work, we leverageinfluence functions, the functional derivatives of the loss function, to theoreticallyreveal the operation selection part in DARTS and estimate the candidate operationimportance by approximating its influence on the supernet with Taylor expansions.

Original: We show the operation strength is not only related to the magnitude but also secondorder information, leading to a fundamentally new criterion for operation selectionin DARTS, named Influential Magnitude .
Revised: We show the operation strength is not only related to the magnitude but also secondorder information, leading to a fundamentally new criterion for operation selectionin DARTS, named Influential Magnitude .

Original: Empirical studies across different taskson several spaces show that vanilla DARTS and its variants can avoid most failuresby leveraging the proposed theory-driven operation selection criterion.
Revised: Empirical studies across different taskson several spaces show that vanilla DARTS and its variants can avoid most failuresby leveraging the proposed theory-driven operation selection criterion.

Original:  Introduction
Revised:  Introduction

Original: Neural Architecture Search (NAS) has been successfully applied to automating the process of neuralnetwork design for a broad of deep learning fields [35].
Revised: Neural Architecture Search (NAS) has been successfully applied to automating the process of neuralnetwork design for a broad of deep learning fields [35].

Original: However, the early NAS methods areoften heavily computational-expensive and require hundreds or even thousands GPU days [34].
Revised: However, the early NAS methods areoften heavily computational-expensive and require hundreds or even thousands GPU days [34].

Original: Toimprove the efficiency for more practical applications, a lot of recent works are proposed based onone-shot (also known as weight-sharing) paradigm [33] to improve the search efficiency significantly.
Revised: Toimprove the efficiency for more practical applications, a lot of recent works are proposed based onone-shot (also known as weight-sharing) paradigm [33] to improve the search efficiency significantly.

Original: Differentiable ARchiTecture Search (DARTS) [27], as the most popular one-shot NAS method, hasdominated this area since its appearance.
Revised: Differentiable ARchiTecture Search (DARTS) [27], as the most popular one-shot NAS method, hasdominated this area since its appearance.

Original: DARTS leverages continuous relaxation to convert theintractable operation selection problem into a continuous magnitude optimization problem, whichcan be gracefully solved by gradient-descent based on the bi-level optimization [8].
Revised: DARTS leverages continuous relaxation to convert theintractable operation selection problem into a continuous magnitude optimization problem, whichcan be gracefully solved by gradient-descent based on the bi-level optimization [8].

Original: A discretearchitecture is then derived based on the optimized magnitudes through operation selection.
Revised: A discretearchitecture is then derived based on the optimized magnitudes through operation selection.

Original: Although the continuous relaxation makes DARTS able to optimize the magnitudes with rigoroustheoretical formulation and guarantees [49] in the supernet training part, it poses an additionalchallenge in selecting a discrete architecture from the optimized magnitudes that there is no existingwork can give a theoretical explanation.
Revised: Although the continuous relaxation makes DARTS able to optimize the magnitudes with rigoroustheoretical formulation and guarantees [49] in the supernet training part, it poses an additionalchallenge in selecting a discrete architecture from the optimized magnitudes that there is no existingwork can give a theoretical explanation.

Original: DARTS is based on the weight-sharing paradigm [33],which assumes that the derived discrete architecture with the trained supernet weights can predictits train-from-scratch performance.
Revised: DARTS is based on the weight-sharing paradigm [33],which assumes that the derived discrete architecture with the trained supernet weights can predictits train-from-scratch performance.

Original: However, there is a gap between the two groups of weights,as depicted in Figure 1 in Appendix.
Revised: However, there is a gap between the two groups of weights,as depicted in Figure 1 in Appendix.

Original: The vanilla DARTS assumes the optimized magnitudes can
Revised: The vanilla DARTS assumes the optimized magnitudes can

Original: [Figure]
Revised: [Figure]

Original: reflect the importance of the candidate operations, while Wang et al.
Revised: reflect the importance of the candidate operations, while Wang et al.

Original: [40] empirically show thatthe operation magnitude does not necessarily indicate how important the operation is in a trainedsupernet.
Revised: [40] empirically show thatthe operation magnitude does not necessarily indicate how important the operation is in a trainedsupernet.

Original: For example, DARTS prefers skip-connection [45] as its magnitude always dominatesthe remaining operations, while an architecture with intensive skip-connections generally leads topoor performance.
Revised: For example, DARTS prefers skip-connection [45] as its magnitude always dominatesthe remaining operations, while an architecture with intensive skip-connections generally leads topoor performance.

Original: Accordingly, Wang et al.
Revised: Accordingly, Wang et al.

Original: [40] proposed a heuristic perturbation-based method,
Revised: [40] proposed a heuristic perturbation-based method,

Original: DARTS-PT, to measure each candidate operation’s influence on the supernet by alternatively removingeach one to calculate the supernet performance drop.
Revised: DARTS-PT, to measure each candidate operation’s influence on the supernet by alternatively removingeach one to calculate the supernet performance drop.

Original: Despite DARTS-PT can partially relieve thefailure of DARTS, we empirically found that DARTS-PT still prefers skip-connection than otheroperations.
Revised: Despite DARTS-PT can partially relieve thefailure of DARTS, we empirically found that DARTS-PT still prefers skip-connection than otheroperations.

Original: One potential reason is that skip-connection ’s magnitude is much higher than othercandidate operations, and directly removing skip-connections usually greatly deteriorates the supernetperformance.
Revised: One potential reason is that skip-connection ’s magnitude is much higher than othercandidate operations, and directly removing skip-connections usually greatly deteriorates the supernetperformance.

Original: In addition, DARTS-PT lacks theoretical explanations and guarantees.
Revised: In addition, DARTS-PT lacks theoretical explanations and guarantees.

Original: For example,the intuitive way to show the importance of a candidate operation importance is the leave-one-outretraining [22], while which should fine-tune the supernet to get the performance drop after removingevery operation rather than directly measuring the performance drop as DARTS-PT, since there alsoexists a gap between the two groups of weights, as visualized in Figure 1.
Revised: For example,the intuitive way to show the importance of a candidate operation importance is the leave-one-outretraining [22], while which should fine-tune the supernet to get the performance drop after removingevery operation rather than directly measuring the performance drop as DARTS-PT, since there alsoexists a gap between the two groups of weights, as visualized in Figure 1.

Original: Section 3 and Appendix Calso give an analysis on why perturbation-based method fails.
Revised: Section 3 and Appendix Calso give an analysis on why perturbation-based method fails.

Original: This paper focuses on filling the gap of theoretical explanations for the discrete operation selection partin DARTS.
Revised: This paper focuses on filling the gap of theoretical explanations for the discrete operation selection partin DARTS.

Original: Instead of considering a heuristic approach to measure the operation importance [27, 40],this paper first rigorously shows how the architecture parameters discretization poses an influenceon the supernet weights in DARTS, and estimates the supernet performance drop accordingly.
Revised: Instead of considering a heuristic approach to measure the operation importance [27, 40],this paper first rigorously shows how the architecture parameters discretization poses an influenceon the supernet weights in DARTS, and estimates the supernet performance drop accordingly.

Original: Ourmethod is motivated by influence function [22], which is originally designed to understand the effectof removing an individual training point, while we leverage Taylor’s approximation to interpret thesupernet performance drop when removing a candidate operation.
Revised: Ourmethod is motivated by influence function [22], which is originally designed to understand the effectof removing an individual training point, while we leverage Taylor’s approximation to interpret thesupernet performance drop when removing a candidate operation.

Original: In this way, we show the operationstrength is not only related to the magnitude but also second-order information, and we accordinglydesign a new criterion for the operation selection in DARTS, called Influential Magnitude ( I M ).
Revised: In this way, we show the operationstrength is not only related to the magnitude but also second-order information, and we accordinglydesign a new criterion for the operation selection in DARTS, called Influential Magnitude ( I M ).

Original: Then, to avoid calculating the inverse of the Hessian matrix with respect to the model weights,we further utilize the Neumann series [29] and Sherman-Morrison formula [37] to approximatethe Inverse-Hessian-Vector Products (IHVPs), making the proposed method applicable to DARTS.
Revised: Then, to avoid calculating the inverse of the Hessian matrix with respect to the model weights,we further utilize the Neumann series [29] and Sherman-Morrison formula [37] to approximatethe Inverse-Hessian-Vector Products (IHVPs), making the proposed method applicable to DARTS.

Original: Finally, we verify the effectiveness of our method on several NAS benchmark datasets and thecommon DARTS search space.
Revised: Finally, we verify the effectiveness of our method on several NAS benchmark datasets and thecommon DARTS search space.

Original: To summarize, we make the following contributions:
Revised: To summarize, we make the following contributions:

Original: • This paper deepens our understanding of the operation selection in DARTS.
Revised: • This paper deepens our understanding of the operation selection in DARTS.

Original: We reformulate
Revised: We reformulate

Original: the operation selection in DARTS by approximating its influence on the supernet with Taylorexpansions, which can thus gracefully interpret how the validation performance changes
Revised: the operation selection in DARTS by approximating its influence on the supernet with Taylorexpansions, which can thus gracefully interpret how the validation performance changes

Original: when selecting different operations without any additional fine-tuning.
Revised: when selecting different operations without any additional fine-tuning.

Original: • We theoretically reveal the operation strength is not only related to the magnitude but also thesecond-order information, and accordingly derive a fundamentally new criterion to measure
Revised: • We theoretically reveal the operation strength is not only related to the magnitude but also thesecond-order information, and accordingly derive a fundamentally new criterion to measure

Original: the operation sensitivity, which we call Influential Magnitude ( I M ), for architecture
Revised: the operation sensitivity, which we call Influential Magnitude ( I M ), for architecture

Original: selection in DARTS.
Revised: selection in DARTS.

Original: Further, to make the proposed criterion practical, we devise severalmethods to estimate the Inverse-Hessian-Vector Products (IHVPs) in calculating the secondorder information.• Extensive experiments verify the effectiveness of the proposed criterion, which significantlyimproves the performance of vanilla DARTS and other baselines on the NAS benchmarkdatasets and the common DARTS search space.
Revised: Further, to make the proposed criterion practical, we devise severalmethods to estimate the Inverse-Hessian-Vector Products (IHVPs) in calculating the secondorder information.• Extensive experiments verify the effectiveness of the proposed criterion, which significantlyimproves the performance of vanilla DARTS and other baselines on the NAS benchmarkdatasets and the common DARTS search space.

Original: We show DARTS is still a strong NASbaseline when considering a more reasonable and theory-driven operation strength metric.
Revised: We show DARTS is still a strong NASbaseline when considering a more reasonable and theory-driven operation strength metric.

Original:  Preliminaries: DARTS and Influence Function
Revised:  Preliminaries: DARTS and Influence Function

Original: Typically, NAS focuses on finding a cell structure that is represented as a directed acyclic graph(DAG) with N nodes and E edges affiliated with operations, which is a discrete and hard-to-optimizeproblem.
Revised: Typically, NAS focuses on finding a cell structure that is represented as a directed acyclic graph(DAG) with N nodes and E edges affiliated with operations, which is a discrete and hard-to-optimizeproblem.

Original: To enable the gradient-descent for architecture search, DARTS [27] leverages softmax asthe continuous relaxation function to transform the architecture search into a continuous magnitudeoptimization problem:
Revised: To enable the gradient-descent for architecture search, DARTS [27] leverages softmax asthe continuous relaxation function to transform the architecture search into a continuous magnitudeoptimization problem:

Original: [Equation]
Revised: [Equation]

Original: where O contains all candidate operations and DARTS is to optimize the magnitude α o , which canbe formulated as a bi-level optimization problem [8]:
Revised: where O contains all candidate operations and DARTS is to optimize the magnitude α o , which canbe formulated as a bi-level optimization problem [8]:

Original: [Equation]
Revised: [Equation]

Original: where α is the continuous architecture representation and θ is the supernet weights.
Revised: where α is the continuous architecture representation and θ is the supernet weights.

Original: The nestedformulation in DARTS is the same as the gradient-based hyperparameter optimization with bi-leveloptimization, which has been validated in a broad applications [13, 31, 32].
Revised: The nestedformulation in DARTS is the same as the gradient-based hyperparameter optimization with bi-leveloptimization, which has been validated in a broad applications [13, 31, 32].

Original: After the bi-leveloptimization, DARTS usually considers heuristic methods to derive the final architecture, where themost popular one is to select the operations with the highest magnitudes, ˆ α = argmax ( α ) .
Revised: After the bi-leveloptimization, DARTS usually considers heuristic methods to derive the final architecture, where themost popular one is to select the operations with the highest magnitudes, ˆ α = argmax ( α ) .

Original: Despiteits simplicity, more recent works [7, 40, 45, 47, 48] found that DARTS could hardly find satisfactoryarchitectures.
Revised: Despiteits simplicity, more recent works [7, 40, 45, 47, 48] found that DARTS could hardly find satisfactoryarchitectures.

Original: Different from the majority of existing works that attribute the failure of DARTS to its magnitudeoptimization part, which is under a rigorous theoretical formulation [49], we revisit DARTS fromthe perspective of the architecture selection and try to reveal this part with theoretical explanations,where we aim to identify the operation that will contribute most to the supernet performance, throughthe lens of influence functions [22] to mathematically model the architecture selection process insteadof heuristics.
Revised: Different from the majority of existing works that attribute the failure of DARTS to its magnitudeoptimization part, which is under a rigorous theoretical formulation [49], we revisit DARTS fromthe perspective of the architecture selection and try to reveal this part with theoretical explanations,where we aim to identify the operation that will contribute most to the supernet performance, throughthe lens of influence functions [22] to mathematically model the architecture selection process insteadof heuristics.

Original: The influence function [18] is a classic technique from robust statistics that reveals how modelparameters change as we upweight or perturb a specific training sample, which has been applied inexplaining a lot of modern machine learning applications [2, 22, 30, 50].
Revised: The influence function [18] is a classic technique from robust statistics that reveals how modelparameters change as we upweight or perturb a specific training sample, which has been applied inexplaining a lot of modern machine learning applications [2, 22, 30, 50].

Original: More specifically, given amodel with parameters θ ∗ after trained on a full training set D , influence function aims to study thechange of parameters ˆ θ ∗ − θ ∗ when retraining the model parameters to ˆ θ ∗ after deleting a specifictraining point z , that ˆ D
Revised: More specifically, given amodel with parameters θ ∗ after trained on a full training set D , influence function aims to study thechange of parameters ˆ θ ∗ − θ ∗ when retraining the model parameters to ˆ θ ∗ after deleting a specifictraining point z , that ˆ D

Original: = D − z .
Revised: = D − z .

Original: Rather than retraining the model for every removed point, Koh and
Revised: Rather than retraining the model for every removed point, Koh and

Original: Liang [22] estimated the influence by upweighting the data point with some small ϵ .
Revised: Liang [22] estimated the influence by upweighting the data point with some small ϵ .

Original: With leveragingthe Taylor expansions, the influence on the model parameters can be estimated as:
Revised: With leveragingthe Taylor expansions, the influence on the model parameters can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: (2) and its influence on the loss function of a test point z test is accordingly calculated as:
Revised: (2) and its influence on the loss function of a test point z test is accordingly calculated as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The detailed derivation of the two influence functions can be found in [22].
Revised: The detailed derivation of the two influence functions can be found in [22].

Original: Based on Eq.(2) and (3),we can now approximate the validation performance drop when removing a training point z from thetraining set without conducting the time-consuming retraining.
Revised: Based on Eq.(2) and (3),we can now approximate the validation performance drop when removing a training point z from thetraining set without conducting the time-consuming retraining.

Original: The original work of influence functions for modern machine learning [22], which is also calledas first-order influence functions, only considers a very small change on θ ∗ as it only changes one training point, so the first-order Taylor expansion is satisfied to calculate the influence.
Revised: The original work of influence functions for modern machine learning [22], which is also calledas first-order influence functions, only considers a very small change on θ ∗ as it only changes one training point, so the first-order Taylor expansion is satisfied to calculate the influence.

Original: However,upweighting a group of data points may lead to a large change on θ ∗ [4, 5], which violates the smallperturbation assumption of the first-order influence functions.
Revised: However,upweighting a group of data points may lead to a large change on θ ∗ [4, 5], which violates the smallperturbation assumption of the first-order influence functions.

Original: To make the the influence functioncapture the model parameter change more precisely, Basu et al.
Revised: To make the the influence functioncapture the model parameter change more precisely, Basu et al.

Original: [4] extended influence functionsusing second-order approximations under the assumption that the third-order derivatives of the lossfunction at optimum is small.
Revised: [4] extended influence functionsusing second-order approximations under the assumption that the third-order derivatives of the lossfunction at optimum is small.

Original: For the quadratic loss, the third-order derivatives of the loss are zero orvery small, and Basu et al.
Revised: For the quadratic loss, the third-order derivatives of the loss are zero orvery small, and Basu et al.

Original: [4] empirically verified that the this assumption also approximately heldfor the classification problem with cross-entropy loss function.
Revised: [4] empirically verified that the this assumption also approximately heldfor the classification problem with cross-entropy loss function.

Original: Different from existing works that analyze the effects on model parameters of removing data points,this paper creatively adapts influence functions to estimate the importance of candidate operationson a trained supernet in differentiable architecture search.
Revised: Different from existing works that analyze the effects on model parameters of removing data points,this paper creatively adapts influence functions to estimate the importance of candidate operationson a trained supernet in differentiable architecture search.

Original: We also follow the assumption in [4] thatneglect the third-order and higher derivatives of the loss, and analyze the operation selection part,which is an additional process after bi-level supernet training in DARTS, through the lens of influencefunction in the following sections.
Revised: We also follow the assumption in [4] thatneglect the third-order and higher derivatives of the loss, and analyze the operation selection part,which is an additional process after bi-level supernet training in DARTS, through the lens of influencefunction in the following sections.

Original:  Interpret Operation Selection through Influence Functions
Revised:  Interpret Operation Selection through Influence Functions

Original: This section aims to understand how the model parameters θ (a.k.a.
Revised: This section aims to understand how the model parameters θ (a.k.a.

Original: the supernet weights in DARTS)would change when we derive the discrete architecture from the optimized continuous operationmagnitudes, a.k.a.
Revised: the supernet weights in DARTS)would change when we derive the discrete architecture from the optimized continuous operationmagnitudes, a.k.a.

Original: architecture selection.
Revised: architecture selection.

Original: In this paper, our analysis only focuses on this architectureselection part after the bi-level supernet training process is finished.
Revised: In this paper, our analysis only focuses on this architectureselection part after the bi-level supernet training process is finished.

Original: To be more clearly, we firstdefine the optimized operation magnitudes by DARTS as α with the optimized supernet weights θ ∗ ,and the validation loss is L ( θ ∗ , α ) for simplicity.
Revised: To be more clearly, we firstdefine the optimized operation magnitudes by DARTS as α with the optimized supernet weights θ ∗ ,and the validation loss is L ( θ ∗ , α ) for simplicity.

Original: Then, we define the architecture parameters changeto ˆ α after discretization, and the corresponding ground-truth supernet weights and validation lossare defined as ˆ θ ∗ and L (ˆ θ ∗ , ˆ α ) , respectively.
Revised: Then, we define the architecture parameters changeto ˆ α after discretization, and the corresponding ground-truth supernet weights and validation lossare defined as ˆ θ ∗ and L (ˆ θ ∗ , ˆ α ) , respectively.

Original: Rather than deleting a single data point that only bringssmall changes on the model parameters [22], the change of architecture parameters can lead to aconsiderable change on θ , so we leverage the second-order approximation [4] to reveal the supernetweights change.
Revised: Rather than deleting a single data point that only bringssmall changes on the model parameters [22], the change of architecture parameters can lead to aconsiderable change on θ , so we leverage the second-order approximation [4] to reveal the supernetweights change.

Original: With second-order Taylor expansion on ˆ θ ∗ for L (ˆ θ ∗ , ˆ α ) , we have:
Revised: With second-order Taylor expansion on ˆ θ ∗ for L (ˆ θ ∗ , ˆ α ) , we have:

Original: [Equation]
Revised: [Equation]

Original: where ∆ θ = ˆ θ ∗ − θ ∗ , and L is the validation loss function in Eq.(1).
Revised: where ∆ θ = ˆ θ ∗ − θ ∗ , and L is the validation loss function in Eq.(1).

Original: For example, when weonly remove one candidate operation as DARTS-PT [40] to measure the operation importance, theinfluence of this discretization brings to θ is ∆ θ , and the true validation loss change should be
Revised: For example, when weonly remove one candidate operation as DARTS-PT [40] to measure the operation importance, theinfluence of this discretization brings to θ is ∆ θ , and the true validation loss change should be

Original: L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) as stated in Eq.(4) rather than L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) used by DARTS-PT.
Revised: L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) as stated in Eq.(4) rather than L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) used by DARTS-PT.

Original: Asshown in Eq.(4), we need to calculate ∆ θ to get ∆ L after we change the α to ˆ α , while we try to avoidthe time-consuming retraining process through leveraging influence functions [22] to estimating ˆ θ ∗ .
Revised: Asshown in Eq.(4), we need to calculate ∆ θ to get ∆ L after we change the α to ˆ α , while we try to avoidthe time-consuming retraining process through leveraging influence functions [22] to estimating ˆ θ ∗ .

Original: Based on the implicit function theorem [29, 49], the validation loss change after discretizing α to ˆ αcan be approximated as the following theorem.
Revised: Based on the implicit function theorem [29, 49], the validation loss change after discretizing α to ˆ αcan be approximated as the following theorem.

Original: Theorem 1 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, α changes to ˆ α when conducting architecture discretization, andthe train-from-scratch validation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .
Revised: Theorem 1 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, α changes to ˆ α when conducting architecture discretization, andthe train-from-scratch validation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .

Original: If the third and higher derivatives of the lossfunction L at optimum is zero or sufficiently small [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have
Revised: If the third and higher derivatives of the lossfunction L at optimum is zero or sufficiently small [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 states that we can estimate the train-from-scratch validation performance of ˆ α without anyretraining.
Revised: Theorem 1 states that we can estimate the train-from-scratch validation performance of ˆ α without anyretraining.

Original: During the discretization process, we can apply the argmax on all edges by one-shot thatˆ α = argmax ( α ) , or only discretize one edge in each step, or remove one candidate operation fromone edge as DARTS-PT [40] which can be formulated as the first part of Eq.(5), L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: During the discretization process, we can apply the argmax on all edges by one-shot thatˆ α = argmax ( α ) , or only discretize one edge in each step, or remove one candidate operation fromone edge as DARTS-PT [40] which can be formulated as the first part of Eq.(5), L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: We prefer an iterative discretization process that only applies a small change on α as DARTS-PT,since our theoretical analysis in the next corollary will show that the error bound of our approximationis related to ∆ α = ˆ α − α .
Revised: We prefer an iterative discretization process that only applies a small change on α as DARTS-PT,since our theoretical analysis in the next corollary will show that the error bound of our approximationis related to ∆ α = ˆ α − α .

Original: Before our analysis, we give the following common assumptions, whichare also considered in several recent bi-level optimization problems [9, 15–17, 49].
Revised: Before our analysis, we give the following common assumptions, whichare also considered in several recent bi-level optimization problems [9, 15–17, 49].

Original: Assumption 1 For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with C f > 0 and
Revised: Assumption 1 For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with C f > 0 and

Original: C L > 0 , respectively.
Revised: C L > 0 , respectively.

Original: Assumption 2 L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θaround θ ∗ ( α ) .
Revised: Assumption 2 L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θaround θ ∗ ( α ) .

Original: Assumption 3 (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .
Revised: Assumption 3 (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .

Original: Corollary 1 Based on the Assumption 1-3, we could bound the error between the approximatedvalidation loss
Revised: Corollary 1 Based on the Assumption 1-3, we could bound the error between the approximatedvalidation loss

Original: L (ˆ θ ∗ , ˆ α ) =
Revised: L (ˆ θ ∗ , ˆ α ) =

Original: ∆ L + L ( θ ∗ , α ) and the ground-truth ˜ L (ˆ θ ∗ , ˆ α ) in DARTS with E =(cid:13)(cid:13)(cid:13)
Revised: ∆ L + L ( θ ∗ , α ) and the ground-truth ˜ L (ˆ θ ∗ , ˆ α ) in DARTS with E =(cid:13)(cid:13)(cid:13)

Original: L (ˆ θ ∗ , ˆ α ) − ˜ L (ˆ θ ∗ , ˆ α ) (cid:13)(cid:13)(cid:13) ⩽ K 3 6 max (cid:12)(cid:12)(cid:12) ∂
Revised: L (ˆ θ ∗ , ˆ α ) − ˜ L (ˆ θ ∗ , ˆ α ) (cid:13)(cid:13)(cid:13) ⩽ K 3 6 max (cid:12)(cid:12)(cid:12) ∂

Original: L 3 ∂θ 3 (cid:12)(cid:12)(cid:12) , where K = C L λ ∥
Revised: L 3 ∂θ 3 (cid:12)(cid:12)(cid:12) , where K = C L λ ∥

Original: ∆ α ∥ + C H C 2 a 2 σ 2 min λ ∥ ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .
Revised: ∆ α ∥ + C H C 2 a 2 σ 2 min λ ∥ ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .

Original: Compared with the argmax used in DARTS, we prefer the perturbation-based discretization method[41] as which only remove one operation at each step, since the above Corollary 1 show that errorbound of the approximated validation loss
Revised: Compared with the argmax used in DARTS, we prefer the perturbation-based discretization method[41] as which only remove one operation at each step, since the above Corollary 1 show that errorbound of the approximated validation loss

Original: L
Revised: L

Original: (ˆ θ ∗ , ˆ α ) increase with the magnitude of the change of α .
Revised: (ˆ θ ∗ , ˆ α ) increase with the magnitude of the change of α .

Original:  Influential Magnitude for Operation Selection in DARTS
Revised:  Influential Magnitude for Operation Selection in DARTS

Original: Section 3 provides an iterative solution to measure the operation importance based Eq.
Revised: Section 3 provides an iterative solution to measure the operation importance based Eq.

Original: (5) throughindividually removing each operation, while which needs to repeat n times (the number of allcandidate operations for all edges in the supernet work).
Revised: (5) throughindividually removing each operation, while which needs to repeat n times (the number of allcandidate operations for all edges in the supernet work).

Original: More important, since removing oneoperation poses a considerable change on α ( α i → 0 ), the approximation error produced by Eq.(5)is non-negligible which may affects the accuracy as stated by Corollary 1.
Revised: More important, since removing oneoperation poses a considerable change on α ( α i → 0 ), the approximation error produced by Eq.(5)is non-negligible which may affects the accuracy as stated by Corollary 1.

Original: So, a more practicalsolution to illustrate the importance of each operation is to estimate how the validation performancewill change after posing an infinitesimal change on α , a.k.a.
Revised: So, a more practicalsolution to illustrate the importance of each operation is to estimate how the validation performancewill change after posing an infinitesimal change on α , a.k.a.

Original: operation sensitivity.
Revised: operation sensitivity.

Original: In this section,we aim to investigate the operation importance through the lens of sensitivity [24].
Revised: In this section,we aim to investigate the operation importance through the lens of sensitivity [24].

Original: Different from
Revised: Different from

Original: Section 3 that removes one operation ( α i → 0 ), this section only changes the α i with an infinitesimalε i , which means that we can now conduct first-order Taylor expansion on α for L (ˆ θ ∗ , ˆ α ) .
Revised: Section 3 that removes one operation ( α i → 0 ), this section only changes the α i with an infinitesimalε i , which means that we can now conduct first-order Taylor expansion on α for L (ˆ θ ∗ , ˆ α ) .

Original: Furtherwith second-order Taylor expansion on θ and assuming the local optimal ∂ L ( θ ∗ ,α ) ∂θ = 0 , we have:
Revised: Furtherwith second-order Taylor expansion on θ and assuming the local optimal ∂ L ( θ ∗ ,α ) ∂θ = 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: Similar to Eq.(4), it is intractable to directly calculate ∆ θ , while we can leverage implicit functiontheorem to approximate it.
Revised: Similar to Eq.(4), it is intractable to directly calculate ∆ θ , while we can leverage implicit functiontheorem to approximate it.

Original: Theorem 2 presents how the validation performance changes after wepose an infinitesimal change on α .
Revised: Theorem 2 presents how the validation performance changes after wepose an infinitesimal change on α .

Original: Theorem 2 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, and we pose an infinitesimal change on α .
Revised: Theorem 2 Suppose that DARTS obtains the optimized architecture parameter α with supernetweights θ ∗ after supernet training, and we pose an infinitesimal change on α .

Original: Based on implicitfunction theorem and under the assumption that the third and higher derivatives of the loss functionat optimum is zero or sufficiently small [4], the change of validation loss can be estimated as:
Revised: Based on implicitfunction theorem and under the assumption that the third and higher derivatives of the loss functionat optimum is zero or sufficiently small [4], the change of validation loss can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.
Revised: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.

Original: From Theorem 2, we can observe the relationship between ∆ L and ∆ α .
Revised: From Theorem 2, we can observe the relationship between ∆ L and ∆ α .

Original: To be more specifically,when we only consider an infinitesimal change on α , that ∆ α = ε · 1 where 1 is a column vectorwith all ones and ε is an infinitesimal scalar, the sensitivity of α can be defined as:
Revised: To be more specifically,when we only consider an infinitesimal change on α , that ∆ α = ε · 1 where 1 is a column vectorwith all ones and ε is an infinitesimal scalar, the sensitivity of α can be defined as:

Original: [Equation]
Revised: [Equation]

Original: Accordingly, we propose an Influential Magnitude to quantify the operation importance through thelens of sensitivity:
Revised: Accordingly, we propose an Influential Magnitude to quantify the operation importance through thelens of sensitivity:

Original: Definition 1 Influential Magnitude ( I M ): Suppose DARTS obtains the optimized magnitude αwith supernet weights θ ∗ after supernet training, the operation sensitivity can be defined as I M =
Revised: Definition 1 Influential Magnitude ( I M ): Suppose DARTS obtains the optimized magnitude αwith supernet weights θ ∗ after supernet training, the operation sensitivity can be defined as I M =

Original: [Equation]
Revised: [Equation]

Original: Definition 1 shows we can replace the magnitude α with the operation sensitivity I M in the operationselection for DARTS, whose i-th value represents the importance of i-th candidate operation.
Revised: Definition 1 shows we can replace the magnitude α with the operation sensitivity I M in the operationselection for DARTS, whose i-th value represents the importance of i-th candidate operation.

Original:  Practical Calculation on Operation Influence
Revised:  Practical Calculation on Operation Influence

Original: For a large neural network, it is impractical to calculate the second-order information, e.g., the
Revised: For a large neural network, it is impractical to calculate the second-order information, e.g., the

Original: Hessian matrix H , let alone the inverse of Hessian.
Revised: Hessian matrix H , let alone the inverse of Hessian.

Original: Generally, the core challenge in calculation of
Revised: Generally, the core challenge in calculation of

Original: Eq.(5) and Eq.(7) is the Inverse-Hessian Vector Products (IHVPs), H − 1 v [14, 22, 37].
Revised: Eq.(5) and Eq.(7) is the Inverse-Hessian Vector Products (IHVPs), H − 1 v [14, 22, 37].

Original: As to the termof ∂ 2 L ( θ ∗ ,α ) ∂θ∂α , we follow DARTS [27] to utilize the finite difference approximation for the practicalcalculation.
Revised: As to the termof ∂ 2 L ( θ ∗ ,α ) ∂θ∂α , we follow DARTS [27] to utilize the finite difference approximation for the practicalcalculation.

Original: The first technique for IHVPs is the conjugate gradient (CG) which converts the matrixinverse problem into an optimization problem with δ -optimal solution.
Revised: The first technique for IHVPs is the conjugate gradient (CG) which converts the matrixinverse problem into an optimization problem with δ -optimal solution.

Original: However, the standard CGcan be slow with a large dataset as it needs to go through the whole training points in each step [1].
Revised: However, the standard CGcan be slow with a large dataset as it needs to go through the whole training points in each step [1].

Original: The original paper [22] using influence function for modern machine learning problem leveragesthe Neumann series to approximate the inverse, that H − 1 = (cid:80) ∞ k =

Revised: The original paper [22] using influence function for modern machine learning problem leveragesthe Neumann series to approximate the inverse, that H − 1 = (cid:80) ∞ k =

Original: ( I − H ) k , with only considerthe first t terms to approximate H − 1 .
Revised: ( I − H ) k , with only considerthe first t terms to approximate H − 1 .

Original: However, there is a strict condition in the Neumann seriesapproximation that ∥ I − H ∥ ≤ 1 , which can not be guaranteed in the practical implementation.
Revised: However, there is a strict condition in the Neumann seriesapproximation that ∥ I − H ∥ ≤ 1 , which can not be guaranteed in the practical implementation.

Original: Differently, we consider that H − 1 = γ ( γH ) − 1 = γ (cid:80) ∞
Revised: Differently, we consider that H − 1 = γ ( γH ) − 1 = γ (cid:80) ∞

Original: k
Revised: k

Original: =

Revised: =

Original: ( I − γH )
Revised: ( I − γH )

Original: k , where γ is a small enoughvalue.
Revised: k , where γ is a small enoughvalue.

Original: The following lemma gives our Neumann series based implementation for IHVPs.
Revised: The following lemma gives our Neumann series based implementation for IHVPs.

Original: Lemma 1 With small enough γ , and assuming L is λ -strongly convex at optimum, H − 1 v can beformulated as: H − 1 v = γ (cid:80) Kk =

Revised: Lemma 1 With small enough γ , and assuming L is λ -strongly convex at optimum, H − 1 v can beformulated as: H − 1 v = γ (cid:80) Kk =

Original: [ I − γH ] k V 

Revised: [ I − γH ] k V

Original: =
Revised: =

Original: V 

Revised: V

Original: + V 1 + ... + V K ,
Revised: + V 1 + ... + V K ,

Original: where H = ∂
Revised: where H = ∂

Original: 2 L ( θ ∗ ,α ) ∂θ∂θ , V 0 = v ,and V 

Revised: 2 L ( θ ∗ ,α ) ∂θ∂θ , V 0 = v ,and V

Original: = ( I − γH )
Revised: = ( I − γH )

Original: V 0 , ..., V K
Revised: V 0 , ..., V K

Original: = ( I − γH )
Revised: = ( I − γH )

Original: V K − 1 .
Revised: V K − 1 .

Original: In the practical implementation, it is still hard to make sure L is λ -strongly convex and not easyto select an appropriate γ .
Revised: In the practical implementation, it is still hard to make sure L is λ -strongly convex and not easyto select an appropriate γ .

Original: The work [5] empirically showed that the stochastic Neumann seriesestimation for IHVP is somehow erroneous, especially when the network is deep.
Revised: The work [5] empirically showed that the stochastic Neumann seriesestimation for IHVP is somehow erroneous, especially when the network is deep.

Original: As we know, thetrue Fisher information matrix converges to the Hessian matrix with the training loss approachesto zero.
Revised: As we know, thetrue Fisher information matrix converges to the Hessian matrix with the training loss approachesto zero.

Original: However, it is usually infeasible to calculate the exact Hessian or Fisher matrix in a largeneural network.
Revised: However, it is usually infeasible to calculate the exact Hessian or Fisher matrix in a largeneural network.

Original: A practical approximation is called as empirical Fisher, which has been verified tobe effective in variety of applications [3, 19].
Revised: A practical approximation is called as empirical Fisher, which has been verified tobe effective in variety of applications [3, 19].

Original: In addition, an important advantage from empirical
Revised: In addition, an important advantage from empirical

Original: Fisher is that it allows to approximate the inverse of the Hessian based on the Sherman-Morrisonformula.
Revised: Fisher is that it allows to approximate the inverse of the Hessian based on the Sherman-Morrisonformula.

Original: Although the empirical Fisher can not exactly match the true Fisher and there are severalactive researches on its applicability [23, 38], empirical Fisher is significantly more computationallyefficient, which obtains a good trade-off between the approximation and practical efficiency [37].
Revised: Although the empirical Fisher can not exactly match the true Fisher and there are severalactive researches on its applicability [23, 38], empirical Fisher is significantly more computationallyefficient, which obtains a good trade-off between the approximation and practical efficiency [37].

Original: Apart from Neumann series, we also consider Sherman-Morrison formula [14, 37]
Revised: Apart from Neumann series, we also consider Sherman-Morrison formula [14, 37]

Original: ( A + uv T ) − 1 =
Revised: ( A + uv T ) − 1 =

Original: A − 1 − A − 1 uv T A − 1 1+ v T A − 1 u to calculate IHVPs.
Revised: A − 1 − A − 1 uv T A − 1 1+ v T A − 1 u to calculate IHVPs.

Original: With empirical Fisher F = (cid:80) nj =1 ∇ θ L j ( ∇ θ L j ) T replacingtrue Fisher ˆ F = ∇ θ f ( ∇ θ f ) T , IHVPs can be calculated as following.
Revised: With empirical Fisher F = (cid:80) nj =1 ∇ θ L j ( ∇ θ L j ) T replacingtrue Fisher ˆ F = ∇ θ f ( ∇ θ f ) T , IHVPs can be calculated as following.

Original: Lemma 2 When assume the empirical Fisher can approximate the Fisher matrix, and H is the Hessian matrix ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ in the optimal point, the IHVPs H − 1 v can be formulated as: H − 1 v = F − 1 n v =
Revised: Lemma 2 When assume the empirical Fisher can approximate the Fisher matrix, and H is the Hessian matrix ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ in the optimal point, the IHVPs H − 1 v can be formulated as: H − 1 v = F − 1 n v =

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: L jT r j , where L = ℓ + η R ( θ ) that ℓ is a crossentropy loss and R is the regularization term, F n = 1 n (cid:80) nj =
Revised: L jT r j , where L = ℓ + η R ( θ ) that ℓ is a crossentropy loss and R is the regularization term, F n = 1 n (cid:80) nj =

Original: [Equation]
Revised: [Equation]

Original: ∇
Revised: ∇

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: ∇
Revised: ∇

Original: [Equation]
Revised: [Equation]

Original: L Tj is the empirical Fisher, andr j = F − 1 j − 1 ∇ θ L j which can be recurrently calculated through r j = η − 1 ∇ θ L j − (cid:80) j − 1 i =1 r
Revised: L Tj is the empirical Fisher, andr j = F − 1 j − 1 ∇ θ L j which can be recurrently calculated through r j = η − 1 ∇ θ L j − (cid:80) j − 1 i =1 r

Original: i r iT ∇ θ L j N + ∇ θ L iT r i .
Revised: i r iT ∇ θ L j N + ∇ θ L iT r i .

Original:  Experiments
Revised:  Experiments

Original: [Table]
Revised: [Table]

Original: In Section 3 and 4, we have theoreticallyinterpreted the operation selection throughindividually measuring each operation’sinfluence on supernet, and accordinglydevised an Influential Magnitude ( I M )to replace the commonly-used magnitudeα for architecture selection.
Revised: In Section 3 and 4, we have theoreticallyinterpreted the operation selection throughindividually measuring each operation’sinfluence on supernet, and accordinglydevised an Influential Magnitude ( I M )to replace the commonly-used magnitudeα for architecture selection.

Original: In this section, we conduct a series of experimentsto analyze whether our proposed methodleads to better and more stable results indifferentiable architecture search settings.
Revised: In this section, we conduct a series of experimentsto analyze whether our proposed methodleads to better and more stable results indifferentiable architecture search settings.

Original: We consider two NAS benchmark datasetsearch space, NAS-Bench-201 [12] and
Revised: We consider two NAS benchmark datasetsearch space, NAS-Bench-201 [12] and

Original: NAS-Bench-1shot1 [46], and the most popular DARTS search space [27].
Revised: NAS-Bench-1shot1 [46], and the most popular DARTS search space [27].

Original: We verified the effectiveness of the proposed theory-driven method compared with two heuristic methods, argmax and perturbation-based selection, and show that the proposed Influential Magnitude can be applied to
Revised: We verified the effectiveness of the proposed theory-driven method compared with two heuristic methods, argmax and perturbation-based selection, and show that the proposed Influential Magnitude can be applied to

Original: DARTS and its variants with greatly enhancing the performance.
Revised: DARTS and its variants with greatly enhancing the performance.

Original: 6.1 Reproducible Comparison on Benchmark Datasets
Revised: 6.1 Reproducible Comparison on Benchmark Datasets

Original: To empirically verify the effectiveness of our method, we first run a series of experiments on the two
Revised: To empirically verify the effectiveness of our method, we first run a series of experiments on the two

Original: NAS benchmark datasets, NAS-Bench-201 and NAS-Bench-1shot1.
Revised: NAS benchmark datasets, NAS-Bench-201 and NAS-Bench-1shot1.

Original: To make our results reproducible,we only replace the selection part in DARTS with I M and keep the remaining part identical.
Revised: To make our results reproducible,we only replace the selection part in DARTS with I M and keep the remaining part identical.

Original: Table 1 summarizes the statistical comparison results on NAS-Bench-201 with different randomseeds , where DARTS-IF, as described in Section 3, follows DARTS-PT that individually removesoperation to measure the supernet performance drop, while DARTS-IM, as described in Section 4,calculates the operation sensitivity (influential magnitude) to replace the original magnitude.
Revised: Table 1 summarizes the statistical comparison results on NAS-Bench-201 with different randomseeds , where DARTS-IF, as described in Section 3, follows DARTS-PT that individually removesoperation to measure the supernet performance drop, while DARTS-IM, as described in Section 4,calculates the operation sensitivity (influential magnitude) to replace the original magnitude.

Original: The
Revised: The

Original: Random baseline is to randomly select architectures without considering α or θ , and DARTS and
Revised: Random baseline is to randomly select architectures without considering α or θ , and DARTS and

Original: DARTS-PT leverage the argmax and perturbation-based heuristic selection, respectively.
Revised: DARTS-PT leverage the argmax and perturbation-based heuristic selection, respectively.

Original: As shown,our DARTS-IF and DARTS-IM both significantly outperform the random baseline and the twoheuristic baselines, where DARTS-IM achieves near-optimal results, a 94.29 %, 72.67 %, and 45.93 %test accuracy on CIFAR-10, CIFAR-100, and ImageNet, respectively, in a single run (DARTS-IMwith Sherman-Morrison approximation and N = 20 under random seed 0).
Revised: As shown,our DARTS-IF and DARTS-IM both significantly outperform the random baseline and the twoheuristic baselines, where DARTS-IM achieves near-optimal results, a 94.29 %, 72.67 %, and 45.93 %test accuracy on CIFAR-10, CIFAR-100, and ImageNet, respectively, in a single run (DARTS-IMwith Sherman-Morrison approximation and N = 20 under random seed 0).

Original: [Figure]
Revised: [Figure]

Original: Figure 7 in Appendix G plots thesearched architectures with different architecture selection methodsin NAS-Bench-201.
Revised: Figure 7 in Appendix G plots thesearched architectures with different architecture selection methodsin NAS-Bench-201.

Original: We can seethat DARTS selects skip-connectionfor all edges as its magnitude usually outweighs other candidate operations.
Revised: We can seethat DARTS selects skip-connectionfor all edges as its magnitude usually outweighs other candidate operations.

Original: The results of DARTS in NASBench-201, which are even poorerthan the random baseline, also suggest that the magnitudes could hardlyreflect the operation importance.
Revised: The results of DARTS in NASBench-201, which are even poorerthan the random baseline, also suggest that the magnitudes could hardlyreflect the operation importance.

Original: Although DARTS-PT can partially relieve this issue by selecting one ortwo more convolutional operations,it still prefers skip-connections (selects 3 skip-connections for 6 edges)where directly removing them still significantly affect the performance.
Revised: Although DARTS-PT can partially relieve this issue by selecting one ortwo more convolutional operations,it still prefers skip-connections (selects 3 skip-connections for 6 edges)where directly removing them still significantly affect the performance.

Original: On the contrary, our DARTSIF and DARTS-IM can find valid architectures with more convolutional operations (with only oneskip-connection ).
Revised: On the contrary, our DARTSIF and DARTS-IM can find valid architectures with more convolutional operations (with only oneskip-connection ).

Original: Results in Table 1 verified the effectiveness of our influence functions basedselection methods, which outperform baselines by large margins.
Revised: Results in Table 1 verified the effectiveness of our influence functions basedselection methods, which outperform baselines by large margins.

Original: Our DARTS-IF is similar to
Revised: Our DARTS-IF is similar to

Original: DARTS-PT while with an additional term to approximate the fine-tuned performance and the results that DARTS-IF outperforms DARTS-PT verified the effectiveness of the influence functionexplanation.
Revised: DARTS-PT while with an additional term to approximate the fine-tuned performance and the results that DARTS-IF outperforms DARTS-PT verified the effectiveness of the influence functionexplanation.

Original: [Table]
Revised: [Table]

Original: As discussed in Section 5, we can leverage different methods to approximate the IHVPs inour I M , and Figure 2 also analyzes differentapproximation methods along with the numberof batches N .
Revised: As discussed in Section 5, we can leverage different methods to approximate the IHVPs inour I M , and Figure 2 also analyzes differentapproximation methods along with the numberof batches N .

Original: Full results on three datasetscan be found in the Figure 5 in Appendix F.
Revised: Full results on three datasetscan be found in the Figure 5 in Appendix F.

Original: As demonstrated, increasing N can help the
Revised: As demonstrated, increasing N can help the

Original: Sherman-Morrison approximation to more accurately catch the operation importance and weempirically find that N = 30 is enough in the
Revised: Sherman-Morrison approximation to more accurately catch the operation importance and weempirically find that N = 30 is enough in the

Original: NAS-Bench-201 dataset.
Revised: NAS-Bench-201 dataset.

Original: However, we also observe Neumann approximation is not stable which even obtains worse results with increasing numberof batches, showing it is somehow erroneous in estimating IHVPs.
Revised: However, we also observe Neumann approximation is not stable which even obtains worse results with increasing numberof batches, showing it is somehow erroneous in estimating IHVPs.

Original: Appendix F analyzes differentinverse Hessian estimation methods for our I M .
Revised: Appendix F analyzes differentinverse Hessian estimation methods for our I M .

Original: The NAS-Bench-1shot1 is another popular NAS benchmark dataset to analyze differentiable NASmethods, which is built based on the NAS-Bench-101 [44], through dividing all architectures in
Revised: The NAS-Bench-1shot1 is another popular NAS benchmark dataset to analyze differentiable NASmethods, which is built based on the NAS-Bench-101 [44], through dividing all architectures in

Original: NAS-Bench-101 into three different unified cell-based search spaces to enable differentiable NASmethods to be directly applied.
Revised: NAS-Bench-101 into three different unified cell-based search spaces to enable differentiable NASmethods to be directly applied.

Original: NAS-Bench-1shot1 contains 6240, 29160, and 363648 architecturesin three spaces, respectively, where Space3 is much more complicated than the remaining two spaces.
Revised: NAS-Bench-1shot1 contains 6240, 29160, and 363648 architecturesin three spaces, respectively, where Space3 is much more complicated than the remaining two spaces.

Original: In NAS-Bench-1shot1, we tracked the searched architecture in every epoch under different operationselection paradigms.
Revised: In NAS-Bench-1shot1, we tracked the searched architecture in every epoch under different operationselection paradigms.

Original: We considered two popular supernet training baselines, DARTS and PC-DARTS,and Table 2 summarised the best-searched architectures during the supernet training in differentsub-search spaces in NAS-Bench-1shot1.
Revised: We considered two popular supernet training baselines, DARTS and PC-DARTS,and Table 2 summarised the best-searched architectures during the supernet training in differentsub-search spaces in NAS-Bench-1shot1.

Original: As demonstrated, our I M can enhance the performanceof DARTS and its variant PC-DARTS in different search spaces of NAS-Bench-1shot1, showingthat our influence magnitude (IM) can find more competitive architectures than the default argmax ,especially in the more complicated space.
Revised: As demonstrated, our I M can enhance the performanceof DARTS and its variant PC-DARTS in different search spaces of NAS-Bench-1shot1, showingthat our influence magnitude (IM) can find more competitive architectures than the default argmax ,especially in the more complicated space.

Original: Compared with the perturbation strategy (PT), our IM ismore effective to measure the operation sensitivity with obtaining 5 better results in 6 cases.
Revised: Compared with the perturbation strategy (PT), our IM ismore effective to measure the operation sensitivity with obtaining 5 better results in 6 cases.

Original: 6.2 Reproducible Comparison on DARTS Search Space
Revised: 6.2 Reproducible Comparison on DARTS Search Space

Original: [Table]
Revised: [Table]

Original: The DARTS search space is the mostcommonly-used space to evaluate the NASmethods, while most SOTAs only report theperformance of their best searched architectures without analyzing the search statistics.
Revised: The DARTS search space is the mostcommonly-used space to evaluate the NASmethods, while most SOTAs only report theperformance of their best searched architectures without analyzing the search statistics.

Original: To encourage reproducibility, we train thesupernet with the same random seeds, andthe trained supernet is utilized to derive thearchitecture under different operation selection strategies.
Revised: To encourage reproducibility, we train thesupernet with the same random seeds, andthe trained supernet is utilized to derive thearchitecture under different operation selection strategies.

Original: The comparison results areprovided in Table 3, from which we can seethat, compared with the argmax baseline, thearchitectures searched by our Influential Magnitude obtain better performance on CIFAR-10 and
Revised: The comparison results areprovided in Table 3, from which we can seethat, compared with the argmax baseline, thearchitectures searched by our Influential Magnitude obtain better performance on CIFAR-10 and

Original: ImageNet than DARTS and its variant PC-DARTS, and DARTS-PT on ImageNet, implying ourmethod is more stable to find valid and competitive architectures.
Revised: ImageNet than DARTS and its variant PC-DARTS, and DARTS-PT on ImageNet, implying ourmethod is more stable to find valid and competitive architectures.

Original: Figure 8 in Appendix G also plotsthe searched architecture examples with different selection methods in the DARTS space.
Revised: Figure 8 in Appendix G also plotsthe searched architecture examples with different selection methods in the DARTS space.

Original: As shown,although DARTS, PC-DARTS and DARTS-PT can find satisfying architecture after multiple runs, theargmax is hard to consistently obtain stable results.
Revised: As shown,although DARTS, PC-DARTS and DARTS-PT can find satisfying architecture after multiple runs, theargmax is hard to consistently obtain stable results.

Original: In addition, different from the perturbation-basedoperation selection which needs to fine-tune the supernet after the discretization of each edge thatcosts 0.4 additional GPU day for DARTS, our influential magnitude is calculated by one time whichonly costs hundred seconds, making our method is much more efficient.
Revised: In addition, different from the perturbation-basedoperation selection which needs to fine-tune the supernet after the discretization of each edge thatcosts 0.4 additional GPU day for DARTS, our influential magnitude is calculated by one time whichonly costs hundred seconds, making our method is much more efficient.

Original: Detailed comparisons onbest-searched architecture in DARTS space with the SOTAs can be found in Table 5 in Appendix E.
Revised: Detailed comparisons onbest-searched architecture in DARTS space with the SOTAs can be found in Table 5 in Appendix E.

Original: 6.3 Discussion on the Robustness of Search
Revised: 6.3 Discussion on the Robustness of Search

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: S c_{k-2} 1 skip_connect 2 skip_connect 3 sep_conv_3x dil_conv_5x5 dil_conv_3x3 dil_conv_3x c_{k} dil_conv_3x dil_conv_3x c_{k-1} (d) S
Revised: S c_{k-2} 1 skip_connect 2 skip_connect 3 sep_conv_3x dil_conv_5x5 dil_conv_3x3 dil_conv_3x c_{k} dil_conv_3x dil_conv_3x c_{k-1} (d) S

Original: The robustness has been a criticalconcern in the differentiable architecture search community since Zelaet al.
Revised: The robustness has been a criticalconcern in the differentiable architecture search community since Zelaet al.

Original: [45] observed a performancecollapse that DARTS tends to yielddegenerate architectures, which especially prefers the skip-connectionswith the search progressing.
Revised: [45] observed a performancecollapse that DARTS tends to yielddegenerate architectures, which especially prefers the skip-connectionswith the search progressing.

Original: Similar to DARTS-PT, we also focuson the operation selection part toexplain the failure of DARTS.
Revised: Similar to DARTS-PT, we also focuson the operation selection part toexplain the failure of DARTS.

Original: Figure 3 presents several architecturesby DARTS with our I M on different search spaces.
Revised: Figure 3 presents several architecturesby DARTS with our I M on different search spaces.

Original: Apart from NASBench-201, we also consider severaltool search spaces (S1-S4) proposedby [45], which are specifically designed to verify the robustness of differentiable NAS methods.
Revised: Apart from NASBench-201, we also consider severaltool search spaces (S1-S4) proposedby [45], which are specifically designed to verify the robustness of differentiable NAS methods.

Original: The complete analysis and searched architectures can be found in Appendix G. As shown in Figure 3,
Revised: The complete analysis and searched architectures can be found in Appendix G. As shown in Figure 3,

Original: DARTS-IM can find more meaningful architectures in different spaces.
Revised: DARTS-IM can find more meaningful architectures in different spaces.

Original: For example, DARTS fails dramatically in most spaces that only selects skip-connection for all edges, while DARTS-IM canfind architecture with 5 convolutions in NAS-Bench-201, and 6 and 8 convolutions in S1 and S2.
Revised: For example, DARTS fails dramatically in most spaces that only selects skip-connection for all edges, while DARTS-IM canfind architecture with 5 convolutions in NAS-Bench-201, and 6 and 8 convolutions in S1 and S2.

Original: 6.4 Discussion at
Revised: 6.4 Discussion at

Original: Non
Revised: Non

Original: -Convergence
Revised: -Convergence

Original: [Figure]
Revised: [Figure]

Original: T e s t
Revised: T e s t

Original: A cc
Revised: A cc

Original: CIFAR
Revised: CIFAR

Original: [Figure]
Revised: [Figure]

Original: N=
Revised: N=

Original: [Figure]
Revised: [Figure]

Original: ) (a)
Revised: ) (a)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e s t
Revised: e s t

Original: [Figure]
Revised: [Figure]

Original: As discussed in Section 4, our influential magnitude is derived when θis in the optimum, with leveragingdifferent methods to approximate the
Revised: As discussed in Section 4, our influential magnitude is derived when θis in the optimum, with leveragingdifferent methods to approximate the

Original: IHVPs.
Revised: IHVPs.

Original: In this subsection, we analyze whether the proposed influentialmagnitude is still effective to find important operations when θ is not inthe optimum.
Revised: In this subsection, we analyze whether the proposed influentialmagnitude is still effective to find important operations when θ is not inthe optimum.

Original: Figure 4 tracked theperformance of searched architecturein each epoch during the search with
Revised: Figure 4 tracked theperformance of searched architecturein each epoch during the search with

Original: Sherman-Morrison formula.
Revised: Sherman-Morrison formula.

Original: Full results on three datasets and more comparison with DARTS-PT can be found in the Figure 6 in Appendix F. As shown, our influentialmagnitude I M is still effective when θ is not in the optimum, while which is getting more stable withthe supernet training.
Revised: Full results on three datasets and more comparison with DARTS-PT can be found in the Figure 6 in Appendix F. As shown, our influentialmagnitude I M is still effective when θ is not in the optimum, while which is getting more stable withthe supernet training.

Original: Specifically, as shown in Figure 4 (a) and (b), our I M with the different N areboth effective, implying that our method is sufficiently informative for the operation selection evenunder different approximation noises.
Revised: Specifically, as shown in Figure 4 (a) and (b), our I M with the different N areboth effective, implying that our method is sufficiently informative for the operation selection evenunder different approximation noises.

Original: More interesting, our DARTS-IM under Sherman-Morrison formula with batch number N =1 in
Revised: More interesting, our DARTS-IM under Sherman-Morrison formula with batch number N =1 in

Original: Figure 4 means that we simply approximate the inverse H − 1 =
Revised: Figure 4 means that we simply approximate the inverse H − 1 =

Original: η − 1 (cid:16) I − η −
Revised: η − 1 (cid:16) I − η −

Original: [Figure]
Revised: [Figure]

Original: θ L ⊤ ∇ θ L (cid:17) ∝ˆ H − 1 , that ˆ H − 1 is a diagonal matrix that ˆ H ii =
Revised: θ L ⊤ ∇ θ L (cid:17) ∝ˆ H − 1 , that ˆ H − 1 is a diagonal matrix that ˆ H ii =

Original: ( ∇ θ i L∇ θ i L ) , which is also called the diagonalapproximation of the inverse Hessian in community.
Revised: ( ∇ θ i L∇ θ i L ) , which is also called the diagonalapproximation of the inverse Hessian in community.

Original: As shown, with roughly approximated secondorder information, Eq.
Revised: As shown, with roughly approximated secondorder information, Eq.

Original: (7) also help DARTS-IM to find more important operations compared withexisting heuristic methods.
Revised: (7) also help DARTS-IM to find more important operations compared withexisting heuristic methods.

Original: 6.5 Discussion on the Limitations in Practice
Revised: 6.5 Discussion on the Limitations in Practice

Original: Although our influential magnitude provides a theory-driven and concise solution for the operationselection in DARTS, we consider several assumptions and approximations for the practical implementation.
Revised: Although our influential magnitude provides a theory-driven and concise solution for the operationselection in DARTS, we consider several assumptions and approximations for the practical implementation.

Original: For example, Basu et al.
Revised: For example, Basu et al.

Original: [4, 5] pointed out that the second-order influence values stillunderestimated the true group influence values in the deep neural networks, while they also observedthe correlation marginally improves with the number of parameters in a group and the second-orderinfluence function generally outperforms the first-order.
Revised: [4, 5] pointed out that the second-order influence values stillunderestimated the true group influence values in the deep neural networks, while they also observedthe correlation marginally improves with the number of parameters in a group and the second-orderinfluence function generally outperforms the first-order.

Original: Fortunately, this paper considers the wholeparameters in the supernet, which helps the second-order influence function reach its potential.
Revised: Fortunately, this paper considers the wholeparameters in the supernet, which helps the second-order influence function reach its potential.

Original: The approximation of IHVPs requires the conditions that the Hessian matrix is positive definite[29] or the empirical Fisher can replace the true Fisher matrix [14], and the supernet weight θ ∗ is aglobal minimum of the empirical risk, while which in practice, are hard to hold.
Revised: The approximation of IHVPs requires the conditions that the Hessian matrix is positive definite[29] or the empirical Fisher can replace the true Fisher matrix [14], and the supernet weight θ ∗ is aglobal minimum of the empirical risk, while which in practice, are hard to hold.

Original: In the calculation ofv ∗ ∂ 2 L ∂α∂θ , which is detailed described in Appendix B, this paper also follows DARTS that considersthe finite difference approximation for practical calculation.
Revised: In the calculation ofv ∗ ∂ 2 L ∂α∂θ , which is detailed described in Appendix B, this paper also follows DARTS that considersthe finite difference approximation for practical calculation.

Original: These approximations and assumptionsmay bring errors for the practical calculation, while empirical results verified that our influentialmagnitude is sufficiently informative for the operation selection in DARTS.
Revised: These approximations and assumptionsmay bring errors for the practical calculation, while empirical results verified that our influentialmagnitude is sufficiently informative for the operation selection in DARTS.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: This paper focuses on the operation selection, which is an essential part while with less attention inthe differentiable architecture search.
Revised: This paper focuses on the operation selection, which is an essential part while with less attention inthe differentiable architecture search.

Original: Rather than considering heuristics, this paper introduces theinfluence functions to theoretically reveal the candidate operation’s importance through approximatingits influence on the supernet.
Revised: Rather than considering heuristics, this paper introduces theinfluence functions to theoretically reveal the candidate operation’s importance through approximatingits influence on the supernet.

Original: By leveraging the Taylor’s approximation, we gracefully interpret howthe validation performance changes during the discretization in DARTS and show that the operationstrength is not only related to the magnitude but also the second-order information, where an influentialmagnitude is accordingly devised for the architecture selection.
Revised: By leveraging the Taylor’s approximation, we gracefully interpret howthe validation performance changes during the discretization in DARTS and show that the operationstrength is not only related to the magnitude but also the second-order information, where an influentialmagnitude is accordingly devised for the architecture selection.

Original: We based our framework on DARTSand extensive experimental results verified the proposed influential magnitude can be practically usedin operation selection to avoid failures in differentiable architecture search.
Revised: We based our framework on DARTSand extensive experimental results verified the proposed influential magnitude can be practically usedin operation selection to avoid failures in differentiable architecture search.

Original: Developing more accurate influential magnitude for architecture search or leveraging the training dynamics [20] to reveal theoperation influence in the early of training and also extending influential magnitude to other bi-leveloptimization problems[21, 28, 29], e.g., network pruning [28], are among directions for future work.
Revised: Developing more accurate influential magnitude for architecture search or leveraging the training dynamics [20] to reveal theoperation influence in the early of training and also extending influential magnitude to other bi-leveloptimization problems[21, 28, 29], e.g., network pruning [28], are among directions for future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: a)
Revised: a)

Original: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe
Revised: (b) Did you describe

Original: the limitations of
Revised: the limitations of

Original: your
Revised: your

Original: work
Revised: work

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [No]
Revised: [No]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: results (either in the supplemental material or as a URL)?
Revised: results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments

Original: multiple times)?
Revised: multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type of GPUs,
Revised: g., type of GPUs,

Original: internal cluster, or cloud provider)?
Revised: internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-

Original: ing/curating?
Revised: ing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: informa-
Revised: informa-

Original: tion or offensive content?
Revised: tion or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks,
Revised: (b) Did you describe any potential participant risks,

Original: approvals,
Revised: approvals,

Original: (c
Revised: (c

Original: A Proof
Revised: A Proof

Original: Theorem 1: Suppose that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗after supernet training, α changes to ˆ α when conducting architecture discretization, and the train-from-scratchvalidation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .
Revised: Theorem 1: Suppose that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗after supernet training, α changes to ˆ α when conducting architecture discretization, and the train-from-scratchvalidation loss for ˆ α is L (ˆ θ ∗ , ˆ α ) .

Original: If the third-derivative of the loss function L at optimum is zero or sufficientlysmall [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have
Revised: If the third-derivative of the loss function L at optimum is zero or sufficientlysmall [4], and with ∂ L (ˆ θ ∗ , ˆ α ) ∂θ = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1: Based on Eq.(4), we now only need to calculate ∆ θ to get ∆ L after we change the 566 α to ˆ α as other parts could be directly obtained, while we try to avoid the time-consuming retraining processthrough leveraging implicit function to estimating ˆ θ ∗ .
Revised: Proof of Theorem 1: Based on Eq.(4), we now only need to calculate ∆ θ to get ∆ L after we change the 566 α to ˆ α as other parts could be directly obtained, while we try to avoid the time-consuming retraining processthrough leveraging implicit function to estimating ˆ θ ∗ .

Original: First, based on the local optimal, we have
Revised: First, based on the local optimal, we have

Original: [Equation]
Revised: [Equation]

Original: When we consider the second-order Taylor expansion on θ , we have
Revised: When we consider the second-order Taylor expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (12) is obtained since we assume the third-derivative of the loss function L at optimum is zero or sufficiently 570 small [4].
Revised: (12) is obtained since we assume the third-derivative of the loss function L at optimum is zero or sufficiently 570 small [4].

Original: And we have
Revised: And we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: So,
Revised: So,

Original: [Equation]
Revised: [Equation]

Original: In this way, based on Eq.(4), (14) and (15), we have
Revised: In this way, based on Eq.(4), (14) and (15), we have

Original: [Equation]
Revised: [Equation]

Original: In this way, we can approximate every change in the α .
Revised: In this way, we can approximate every change in the α .

Original: □
Revised: □

Original: Corollary 1: Based on the Assumption 1-3, we could bound the error between the approximated validationloss L ( ˆ θ ∗ , ˆ α ) and the ground-truth ˜
Revised: Corollary 1: Based on the Assumption 1-3, we could bound the error between the approximated validationloss L ( ˆ θ ∗ , ˆ α ) and the ground-truth ˜

Original: L 2 ( ˆ θ ∗ , ˆ α ) in DARTS with E =
Revised: L 2 ( ˆ θ ∗ , ˆ α ) in DARTS with E =

Original: (cid:13)(cid:13) (cid:13)
Revised: (cid:13)(cid:13) (cid:13)

Original: L ( ˆ θ ∗ , ˆ α )
Revised: L ( ˆ θ ∗ , ˆ α )

Original: − ˜ L ( ˆ θ ∗ , ˆ α ) (cid:13)(cid:13) (cid:13) ⩽ K 3 6 max (cid:12)(cid:12) (cid:12) ∂
Revised: − ˜ L ( ˆ θ ∗ , ˆ α ) (cid:13)(cid:13) (cid:13) ⩽ K 3 6 max (cid:12)(cid:12) (cid:12) ∂

Original: L 3 ∂θ 3 (cid:12)(cid:12) (cid:12) ,where K = C L λ ∗ ∥
Revised: L 3 ∂θ 3 (cid:12)(cid:12) (cid:12) ,where K = C L λ ∗ ∥

Original: ∆ α ∥ + C H ∗ C
Revised: ∆ α ∥ + C H ∗ C

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: σ 2 min ∗ λ ∗ ∥
Revised: σ 2 min ∗ λ ∗ ∥

Original: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 4 ) .
Revised: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 4 ) .

Original: Proof of Corollary
Revised: Proof of Corollary

Original: Assumption 1: For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with constant C f > 0 andconstant C L > 0 , respectively.
Revised: Assumption 1: For any θ and α , L ( · , α ) and L ( θ, · ) are Lipschitz continuous with constant C f > 0 andconstant C L > 0 , respectively.

Original: Assumption 2: L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θ around 583 θ ∗ ( α ) .
Revised: Assumption 2: L ( θ, α ) is twice differentiable with constant C H and is λ -strongly convex with θ around 583 θ ∗ ( α ) .

Original: Assumption 3: (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .
Revised: Assumption 3: (cid:13)(cid:13) ∇ 2 θα L (cid:13)(cid:13) is bounded with constant C a > 0 .

Original: Now we can estimate the error bound produced by Theorem 1.
Revised: Now we can estimate the error bound produced by Theorem 1.

Original: First, the error is due to the second-order Taylor 586 expansion on θ in Eq.
Revised: First, the error is due to the second-order Taylor 586 expansion on θ in Eq.

Original: We have the error
Revised: We have the error

Original: [Equation]
Revised: [Equation]

Original: where we should notice that the approximated validation loss L (ˆ θ ∗ , ˆ α ) is Taylor expansion of ˆ L in the pointˆ θ ∗ − ∆ θ .
Revised: where we should notice that the approximated validation loss L (ˆ θ ∗ , ˆ α ) is Taylor expansion of ˆ L in the pointˆ θ ∗ − ∆ θ .

Original: Since we set that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ
Revised: Since we set that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ

Original: [Equation]
Revised: [Equation]

Original: 1 ∗
Revised: 1 ∗

Original: [Equation]
Revised: [Equation]

Original: L ( θ ∗ ,
Revised: L ( θ ∗ ,

Original: [Equation]
Revised: [Equation]

Original: in Eq.
Revised: in Eq.

Original: 13, this error E could be easily bound.
Revised: 13, this error E could be easily bound.

Original: However, we should notice that our ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L ( θ ∗ , ˆ α ) ∂θ is an approximation, that θ ∗ + ∆ θ ̸ = ˆ θ ∗ ,and we define that θ ∗ + ∆ θ = ˆ θ ∗ e and ∆ θ e = ˆ θ ∗ − ˆ θ ∗ e .
Revised: However, we should notice that our ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L ( θ ∗ , ˆ α ) ∂θ is an approximation, that θ ∗ + ∆ θ ̸ = ˆ θ ∗ ,and we define that θ ∗ + ∆ θ = ˆ θ ∗ e and ∆ θ e = ˆ θ ∗ − ˆ θ ∗ e .

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: For the first part in Eq.
Revised: For the first part in Eq.

Original: (18), based on Assumption 2 that L ( θ, α ) is λ -strongly convex with θ around θ ∗ , wehave:
Revised: (18), based on Assumption 2 that L ( θ, α ) is λ -strongly convex with θ around θ ∗ , wehave:

Original: [Equation]
Revised: [Equation]

Original: 19) and
Revised: 19) and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When we consider that L ( θ ∗ , α ) is C L Lipschitz continuous with α , we have
Revised: When we consider that L ( θ ∗ , α ) is C L Lipschitz continuous with α , we have

Original: [Equation]
Revised: [Equation]

Original: Baesd on Eq.
Revised: Baesd on Eq.

Original: (20) and (21), we have
Revised: (20) and (21), we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As to the second part, when we assume that L ( θ, α ) is λ -convex near the local optimal, we also have
Revised: As to the second part, when we assume that L ( θ, α ) is λ -convex near the local optimal, we also have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ˆ θ ∗ is the true optimal, and ˆ θ ∗ e is the approximate local optimal when we utilize the implicit functiontheorem to approximate ∆ θ in Eq.
Revised: where ˆ θ ∗ is the true optimal, and ˆ θ ∗ e is the approximate local optimal when we utilize the implicit functiontheorem to approximate ∆ θ in Eq.

Original: 13.so that, we have
Revised: 13.so that, we have

Original: [Equation]
Revised: [Equation]

Original: Consider the local optimal that ∇L (ˆ θ ∗ , ˆ α ) = 0 , we have
Revised: Consider the local optimal that ∇L (ˆ θ ∗ , ˆ α ) = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Now, we need to bound (cid:13)(cid:13)(cid:13)
Revised: Now, we need to bound (cid:13)(cid:13)(cid:13)

Original: ∇L
Revised: ∇L

Original: (ˆ θ ∗ e , ˆ α ) (cid:13)(cid:13)(cid:13) .
Revised: (ˆ θ ∗ e , ˆ α ) (cid:13)(cid:13)(cid:13) .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since we utilize Eq.
Revised: Since we utilize Eq.

Original: (13) that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L
Revised: (13) that ∆ θ = − ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ − 1 ∗ ∂ L

Original: [Equation]
Revised: [Equation]

Original: θ ∗ , ˆ α )
Revised: θ ∗ , ˆ α )

Original: ∂θ to conduct the second-order Taylor expansion, 604 we have
Revised: ∂θ to conduct the second-order Taylor expansion, 604 we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the second row is calculated since (cid:82) ∇ θ L 2 ( θ ∗ + t ∗ ∆ θ, ˆ α ) ∗ ∆ θ dt = ∇ θ L ( θ ∗ + t ∗ ∆ θ, ˆ α ) .
Revised: where the second row is calculated since (cid:82) ∇ θ L 2 ( θ ∗ + t ∗ ∆ θ, ˆ α ) ∗ ∆ θ dt = ∇ θ L ( θ ∗ + t ∗ ∆ θ, ˆ α ) .

Original: The fifthrow is calculated as we assume that L is convex and twice-differentiable with C H .
Revised: The fifthrow is calculated as we assume that L is convex and twice-differentiable with C H .

Original: And we assume that the∂ L ( θ ∗ , ˆ α )
Revised: And we assume that the∂ L ( θ ∗ , ˆ α )

Original: ∂θ is bounded by C l .
Revised: ∂θ is bounded by C l .

Original: σ min is the smallest eigenvalue of Hessian matrix ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ .
Revised: σ min is the smallest eigenvalue of Hessian matrix ∂ 2 L ( θ ∗ , ˆ α ) ∂θ∂θ .

Original: In addition, when we consider a more tight bound, and consider a Taylor expansion on α , we have
Revised: In addition, when we consider a more tight bound, and consider a Taylor expansion on α , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the second row is obtained based on the local optimal ∂ L ( θ ∗ ,α )
Revised: where the second row is obtained based on the local optimal ∂ L ( θ ∗ ,α )

Original: ∂θ = 0 .
Revised: ∂θ = 0 .

Original: So, we have∥∇L (
Revised: So, we have∥∇L (

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 30) and
Revised: 30) and

Original: [Equation]
Revised: [Equation]

Original: Based on Eq.
Revised: Based on Eq.

Original: (18), (22), and (31), we have
Revised: (18), (22), and (31), we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The final error bound can be calculated as:
Revised: The final error bound can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: where K = C L λ ∗ ∥
Revised: where K = C L λ ∗ ∥

Original: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥
Revised: ∆ α ∥ + C H ∗ C 2 a 2 ∗ σ 2 min ∗ λ ∗ ∥

Original: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .
Revised: ∆ α ∥ 2 + o ( ∥ ∆ α ∥ 2 ) .

Original: □
Revised: □

Original: Derivation of Eq.
Revised: Derivation of Eq.

Original: In Section 4, when we consider a first-order Taylor expansion on α as we only applyan infinitesimal change of α , and we alsoconsider a second-order Taylor expansion on θ , we have
Revised: In Section 4, when we consider a first-order Taylor expansion on α as we only applyan infinitesimal change of α , and we alsoconsider a second-order Taylor expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: where the last row is obtained is obtained due to the local optimal ∂ L ( θ ∗ ,α ) ∂θ
Revised: where the last row is obtained is obtained due to the local optimal ∂ L ( θ ∗ ,α ) ∂θ

Original: [Equation]
Revised: [Equation]

Original: Theorem 2: Supposed that DARTS obtains the optimized architecture parameter α with supernet weights 620 θ ∗ after supernet training, and we pose an infinitesimal change on α .
Revised: Theorem 2: Supposed that DARTS obtains the optimized architecture parameter α with supernet weights 620 θ ∗ after supernet training, and we pose an infinitesimal change on α .

Original: Based on implicit function theorem andunder the assumption that the third-derivative of the loss function at optimum is zero or sufficiently small [4], 622 the change of validation performance can be estimated as:
Revised: Based on implicit function theorem andunder the assumption that the third-derivative of the loss function at optimum is zero or sufficiently small [4], 622 the change of validation performance can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.
Revised: where H = ∂ 2 L ( θ ∗ ,α ) ∂θ∂θ is the Hessian matrix.

Original: Proof of Theorem 2: Although Section 3 provides an iterative solution to measure the operation importancebased Eq.
Revised: Proof of Theorem 2: Although Section 3 provides an iterative solution to measure the operation importancebased Eq.

Original: (5) through individually removing each operation, we need to calculate n times (the number of 626 all candidate operations in the supernet work) to estimate the importance of all candidate operations.
Revised: (5) through individually removing each operation, we need to calculate n times (the number of 626 all candidate operations in the supernet work) to estimate the importance of all candidate operations.

Original: Moreimportant, since removing one operation poses a considerable change on α ( α i → 0 ), the approximation error 628 produced by Eq.(5) is non-negligible which may affects the accuracy as stated by Corollary 1.
Revised: Moreimportant, since removing one operation poses a considerable change on α ( α i → 0 ), the approximation error 628 produced by Eq.(5) is non-negligible which may affects the accuracy as stated by Corollary 1.

Original: So, a morepractical solution to illustrate the importance of each operation is to estimate how the validation performance 630 will change after posing an infinitesimal change on α , a.k.a. operation sensitivity, and we can directly get theoperation importance with one calculation.
Revised: So, a morepractical solution to illustrate the importance of each operation is to estimate how the validation performance 630 will change after posing an infinitesimal change on α , a.k.a. operation sensitivity, and we can directly get theoperation importance with one calculation.

Original: When we consider that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ aftersupernet training, and we pose an infinitesimal change on α , we can conduct a first-order Taylor expansion on α 634 a second-order Taylor expansion on θ .
Revised: When we consider that DARTS obtains the optimized architecture parameter α with supernet weights θ ∗ aftersupernet training, and we pose an infinitesimal change on α , we can conduct a first-order Taylor expansion on α 634 a second-order Taylor expansion on θ .

Original: So that we have get the Eq.
Revised: So that we have get the Eq.

Original: (34) as stated in the above.
Revised: (34) as stated in the above.

Original: When we furtherconduct a second Taylor expansion on ˆ θ ∗ for ∂ L (ˆ θ ∗ ,α ) ∂α , we have
Revised: When we furtherconduct a second Taylor expansion on ˆ θ ∗ for ∂ L (ˆ θ ∗ ,α ) ∂α , we have

Original: [Equation]
Revised: [Equation]

Original: where the last row is obtained when we neglect the third and higher derivatives of the loss function withconsidering the chain rule that ∂ L ∂α =
Revised: where the last row is obtained when we neglect the third and higher derivatives of the loss function withconsidering the chain rule that ∂ L ∂α =

Original: ∂ L ∂θ ∗ ∂ ( θ ∗ ( α )) ∂α , and assume α is a local optimal for the continuous 638 magnitude optimization after architecture search.
Revised: ∂ L ∂θ ∗ ∂ ( θ ∗ ( α )) ∂α , and assume α is a local optimal for the continuous 638 magnitude optimization after architecture search.

Original: From the above, we can find that, the challenge is still that itis intractable to calculate ∆ θ .
Revised: From the above, we can find that, the challenge is still that itis intractable to calculate ∆ θ .

Original: Similar as before, when we consider implicit function and second-order Taylor 640 expansion on θ , we have
Revised: Similar as before, when we consider implicit function and second-order Taylor 640 expansion on θ , we have

Original: [Equation]
Revised: [Equation]

Original: When we neglect the third and higher derivatives and the local optimal ∂ L ( θ ∗ ,α
Revised: When we neglect the third and higher derivatives and the local optimal ∂ L ( θ ∗ ,α

Original: [Equation]
Revised: [Equation]

Original: ∂θ
Revised: ∂θ

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Then, similar as before, when we apply a first-order Taylor expansion on ˆ α , with the chain-rule and neglect third 643 and higher derivatives, we have
Revised: Then, similar as before, when we apply a first-order Taylor expansion on ˆ α , with the chain-rule and neglect third 643 and higher derivatives, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: After applying Eq.
Revised: After applying Eq.

Original: (40) on Eq.
Revised: (40) on Eq.

Original: (36), we have
Revised: (36), we have

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: B Practical Implementation
Revised: B Practical Implementation

Original: To practically calculate the proposed influential magnitude as Definition 1, we need to estimate the InverseHessian-Vector products (IHVPs) H − 1 v , and v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .
Revised: To practically calculate the proposed influential magnitude as Definition 1, we need to estimate the InverseHessian-Vector products (IHVPs) H − 1 v , and v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .

Original: In the following, we first described two methods toestimate H − 1 v , and the following DARTS to described the practical calculation on v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .
Revised: In the following, we first described two methods toestimate H − 1 v , and the following DARTS to described the practical calculation on v ∗ ∂ 2 L ( θ ∗ ,α ) ∂α∂θ .

Original: B.1 Neumann series approximation for Lemma 1
Revised: B.1 Neumann series approximation for Lemma 1

Original: The most common method for the IHVPs is the Neumann series that, for ∥ I − A ∥ < 1 , we have:
Revised: The most common method for the IHVPs is the Neumann series that, for ∥ I − A ∥ < 1 , we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: However, it is impossible to guarantee that ∥ I − H ∥ < 1 .
Revised: However, it is impossible to guarantee that ∥ I − H ∥ < 1 .

Original: Different from [22], we consider H − 1 = γ ( γH ) − 1 .
Revised: Different from [22], we consider H − 1 = γ ( γH ) − 1 .

Original: When assuming H is bounded with σ max , (cid:13)(cid:13)(cid:13) ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < σ max .
Revised: When assuming H is bounded with σ max , (cid:13)(cid:13)(cid:13) ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < σ max .

Original: With γ < 1 σ max , we have (cid:13)(cid:13)(cid:13) I − γ ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < 1[29, 36].
Revised: With γ < 1 σ max , we have (cid:13)(cid:13)(cid:13) I − γ ∂ 2 L ∂θ∂θ (cid:13)(cid:13)(cid:13) < 1[29, 36].

Original: When we conduct the Neumann series approximation for (cid:104) ∂ 2 L ∂θ∂θ (cid:105) − 1 in the optimal point, we have:(cid:20) ∂ 2 L
Revised: When we conduct the Neumann series approximation for (cid:104) ∂ 2 L ∂θ∂θ (cid:105) − 1 in the optimal point, we have:(cid:20) ∂ 2 L

Original: [Equation]
Revised: [Equation]

Original: (cid:21)
Revised: (cid:21)

Original: [Equation]
Revised: [Equation]

Original: when we only consider the first K terms and H is positive in θ ∗ (Please note that, generally, we could onlyguarantee H is positive when in optimal point).
Revised: when we only consider the first K terms and H is positive in θ ∗ (Please note that, generally, we could onlyguarantee H is positive when in optimal point).

Original: Accordingly, the Lemma 1 is derived.
Revised: Accordingly, the Lemma 1 is derived.

Original: As suggested by [49], a medium K is large enough to approximate IHVPs, where K = 2 obtains similar results 658 as K ⩾ 3 while with less computational cost.
Revised: As suggested by [49], a medium K is large enough to approximate IHVPs, where K = 2 obtains similar results 658 as K ⩾ 3 while with less computational cost.

Original: In this paper, we also observed a similar phenomenon and also set
Revised: In this paper, we also observed a similar phenomenon and also set

Original: K = 2 by default in our Neumann series approximation based IHVPs.
Revised: K = 2 by default in our Neumann series approximation based IHVPs.

Original: B.2 Sherman-Morrison formula for Lemma 2
Revised: B.2 Sherman-Morrison formula for Lemma 2

Original: Apart from the Neumann series approximation, we could also follow the Sherman-Morrison formula to exactly 662 calculate the inverse of Hessian when we consider the empirical Fisher to replace the Fisher matrix.
Revised: Apart from the Neumann series approximation, we could also follow the Sherman-Morrison formula to exactly 662 calculate the inverse of Hessian when we consider the empirical Fisher to replace the Fisher matrix.

Original: First, weknow that the Hessian equals to fisher information matrix in the optimal point.
Revised: First, weknow that the Hessian equals to fisher information matrix in the optimal point.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We name the r = 1 n ( y i − f ( x i ))
Revised: We name the r = 1 n ( y i − f ( x i ))

Original: ∇ θ ∇ θ f , which is the residual contribution to Hessian H .
Revised: ∇ θ ∇ θ f , which is the residual contribution to Hessian H .

Original: At the end of training, 665 if we can achieve zero training error, namely, y i =
Revised: At the end of training, 665 if we can achieve zero training error, namely, y i =

Original: f
Revised: f

Original: ( x i ) , the residual will converge to zero.
Revised: ( x i ) , the residual will converge to zero.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Please note that the dimension of ∇ θ f is M ∗ m where m is the number of classes (output dimension) and M is 667 the number of parameters.
Revised: Please note that the dimension of ∇ θ f is M ∗ m where m is the number of classes (output dimension) and M is 667 the number of parameters.

Original: Practically, we usually consider an empirical Fisher to approximate the Fisher matrixthat
Revised: Practically, we usually consider an empirical Fisher to approximate the Fisher matrixthat

Original: [Equation]
Revised: [Equation]

Original: So, in the remaining text, the F indicates the empirical Fisher for simplicity.
Revised: So, in the remaining text, the F indicates the empirical Fisher for simplicity.

Original: The Sherman-Morrison formula is defined as:
Revised: The Sherman-Morrison formula is defined as:

Original: [Equation]
Revised: [Equation]

Original: When we define that F n = 1 N (cid:80) nn =1 ∇ θ L ( y n , f ( x n ))( ∇ θ L ( y n , f ( x n ))
Revised: When we define that F n = 1 N (cid:80) nn =1 ∇ θ L ( y n , f ( x n ))( ∇ θ L ( y n , f ( x n ))

Original: T , we can notice that the empirical 672 Fisher can be represented as F
Revised: T , we can notice that the empirical 672 Fisher can be represented as F

Original: N , and we have:
Revised: N , and we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where L = ℓ + η R ( θ ) that ℓ is a cross-entropy loss and R is the regularization term, and η is the weight-decay, 674 F 0 = ηI .
Revised: where L = ℓ + η R ( θ ) that ℓ is a cross-entropy loss and R is the regularization term, and η is the weight-decay, 674 F 0 = ηI .

Original: L n is the loss for the n-th batch dataset.
Revised: L n is the loss for the n-th batch dataset.

Original: And based on the Sherman-Morrison formula, we have
Revised: And based on the Sherman-Morrison formula, we have

Original: [Equation]
Revised: [Equation]

Original: Please notice that, whether using the Neumann series or the Sherman-Morrison formula, we always need 676 to calculate the Inverse-Hessian Vector Products (IHVPs) rather than only the Inverse-Hessian.
Revised: Please notice that, whether using the Neumann series or the Sherman-Morrison formula, we always need 676 to calculate the Inverse-Hessian Vector Products (IHVPs) rather than only the Inverse-Hessian.

Original: It is easy toimplement the IHVPs based on the Neumann series through the Hessian-vector products based on Lemma 1.
Revised: It is easy toimplement the IHVPs based on the Neumann series through the Hessian-vector products based on Lemma 1.

Original: In 678 the following, we discuss how to implement the IHVPs based on Sherman-Morrison formula.
Revised: In 678 the following, we discuss how to implement the IHVPs based on Sherman-Morrison formula.

Original: When we assumethat we have n + 1 batches datasets, we have
Revised: When we assumethat we have n + 1 batches datasets, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We can find that, to solve the previous equation, we need to first recurrently calculate r N = F − 1 N − 1 ∇ θ L N .
Revised: We can find that, to solve the previous equation, we need to first recurrently calculate r N = F − 1 N − 1 ∇ θ L N .

Original: [Equation]
Revised: [Equation]

Original: We first need to calculate the r n = F − 1 n − 1 ∇ θ L n .
Revised: We first need to calculate the r n = F − 1 n − 1 ∇ θ L n .

Original: [Equation]
Revised: [Equation]

Original: So, we can first iterative calculate r n , and then calculate F − 1 n v .
Revised: So, we can first iterative calculate r n , and then calculate F − 1 n v .

Original: B.3 Practical Calculation on Matrix-Vector product
Revised: B.3 Practical Calculation on Matrix-Vector product

Original: Apart from the IHVPs, we also need to calculate v a ∗ ∂ 2 L ∂α∂θ and v m ∗ ∂ 2 L ∂θ∂α , which are called as matrix-vectorproduct and computational-expensive.
Revised: Apart from the IHVPs, we also need to calculate v a ∗ ∂ 2 L ∂α∂θ and v m ∗ ∂ 2 L ∂θ∂α , which are called as matrix-vectorproduct and computational-expensive.

Original: Following DARTS [27], we also utilize the finite difference approximation 686 for the practical calculation.
Revised: Following DARTS [27], we also utilize the finite difference approximation 686 for the practical calculation.

Original: Let ξ be a small scalar, and consider the function ∂
Revised: Let ξ be a small scalar, and consider the function ∂

Original: L ( θ,α + ξ ∗ v a ) ∂θ with Taylor expansion on α , we have
Revised: L ( θ,α + ξ ∗ v a ) ∂θ with Taylor expansion on α , we have

Original: [Equation]
Revised: [Equation]

Original: when we only consider the first two terms, we have
Revised: when we only consider the first two terms, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, the v m ∗ ∂ 2 L ∂θ∂α can be approximated as:
Revised: Similarly, the v m ∗ ∂ 2 L ∂θ∂α can be approximated as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: C Why perturbation-based method fails
Revised: C Why perturbation-based method fails

Original: The perturbation-based method, e.g. the leave-one-out retraining [22], is an intuitive way to measure the 692 importance of every candidate operations.
Revised: The perturbation-based method, e.g. the leave-one-out retraining [22], is an intuitive way to measure the 692 importance of every candidate operations.

Original: More detailed, we can individually remove every candidate operationin each edge
Revised: More detailed, we can individually remove every candidate operationin each edge

Original: [Table]
Revised: [Table]

Original: and then fine-tune the supernet to the validation performance drop L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: and then fine-tune the supernet to the validation performance drop L (ˆ θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: However,
Revised: However,

Original: DARTS-PT abandons the retraining part that directly measure the validation performance drop.
Revised: DARTS-PT abandons the retraining part that directly measure the validation performance drop.

Original: An concern will 695 be raised here as the pruned supernet is not trained to optimal
Revised: An concern will 695 be raised here as the pruned supernet is not trained to optimal

Original: [Table]
Revised: [Table]

Original: and it is inaccurate to measure the performancedrop by L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .
Revised: and it is inaccurate to measure the performancedrop by L ( θ ∗ , ˆ α ) − L ( θ ∗ , α ) .

Original: More importantly, since the skip-connection ’s magnitude is generally muchhigher than other candidate operations, and directly removing skip-connections usually greatly deteriorates the 698 supernet performance if we do not fine-tune the supernet.
Revised: More importantly, since the skip-connection ’s magnitude is generally muchhigher than other candidate operations, and directly removing skip-connections usually greatly deteriorates the 698 supernet performance if we do not fine-tune the supernet.

Original: This is also the reason why we empirically find that
Revised: This is also the reason why we empirically find that

Original: DARTS-PT still prefers skip-connection than other operations.
Revised: DARTS-PT still prefers skip-connection than other operations.

Original: Figure 7 (e) plots the searched architecture in 700 NAS-Bench-201 for DARTS-PT, which contains 3 skip-connections .
Revised: Figure 7 (e) plots the searched architecture in 700 NAS-Bench-201 for DARTS-PT, which contains 3 skip-connections .

Original: In contrast, the architecture selected by ourinfluential magnitude only contains 1 skip-connection .
Revised: In contrast, the architecture selected by ourinfluential magnitude only contains 1 skip-connection .

Original: This phenomenon also exists in the experiments on the 702 search space S1 and S2 of [45], where although DARTS-PT can avoid selecting all skip-connections in all edges,its searched architectures still contains intensive skip-connections .
Revised: This phenomenon also exists in the experiments on the 702 search space S1 and S2 of [45], where although DARTS-PT can avoid selecting all skip-connections in all edges,its searched architectures still contains intensive skip-connections .

Original: D Comparison with SOTAs on NAS-Bench-201
Revised: D Comparison with SOTAs on NAS-Bench-201

Original: We also compared our DARTS-IM with different NAS baselines on the NAS-Bench-201 search space.
Revised: We also compared our DARTS-IM with different NAS baselines on the NAS-Bench-201 search space.

Original: Table 4 706 reports the comprehensive validation and test results on all three datasets, including CIFAR-10, CIFAR-100,and ImageNet.
Revised: Table 4 706 reports the comprehensive validation and test results on all three datasets, including CIFAR-10, CIFAR-100,and ImageNet.

Original: As shown, DARTS is not a valid method for differentiable architecture search, which even 708 leads to poorer results than random baseline.
Revised: As shown, DARTS is not a valid method for differentiable architecture search, which even 708 leads to poorer results than random baseline.

Original: Although considering other supernet training methods can alsoimprove the performance, we can also found that, with only replacing the operation selection metric with our 710 influential magnitude, our DARTS-IM can achieve more competitive results.
Revised: Although considering other supernet training methods can alsoimprove the performance, we can also found that, with only replacing the operation selection metric with our 710 influential magnitude, our DARTS-IM can achieve more competitive results.

Original: Moreover, our DARTS-IM achievesnear-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR-100, and ImageNet, 712 respectively, in a single run.
Revised: Moreover, our DARTS-IM achievesnear-optimal results, a 94.29 %, 72.67 %, and 45.93 % test accuracy on CIFAR-10, CIFAR-100, and ImageNet, 712 respectively, in a single run.

Original: Our DARTS-IF, which follows the perturbation-based selection as DARTS-PTwhile with an additional approximation term, also outperforms DARTS-PT that again verifies the effectiveness 714 of the influence function explanation.
Revised: Our DARTS-IF, which follows the perturbation-based selection as DARTS-PTwhile with an additional approximation term, also outperforms DARTS-PT that again verifies the effectiveness 714 of the influence function explanation.

Original: In addition, due to the requirement of fine-tuning in the perturbation-basedparadigm, DARTS-PT cost much more than our DARTS-IM.
Revised: In addition, due to the requirement of fine-tuning in the perturbation-basedparadigm, DARTS-PT cost much more than our DARTS-IM.

Original: Compared with the perturbation-based selection 716 paradigm, we found that the magnitude paradigm is more efficient and reliable 1 .
Revised: Compared with the perturbation-based selection 716 paradigm, we found that the magnitude paradigm is more efficient and reliable 1 .

Original: E Comparison with SOTAs on DARTS space
Revised: E Comparison with SOTAs on DARTS space

Original: DARTS search space is a much more complicated search space than NAS-Bench-201 and NAS-Bench-1shot1, 

Revised: DARTS search space is a much more complicated search space than NAS-Bench-201 and NAS-Bench-1shot1,

Original: and it is intractable to get the ground truth for all architectures in this space.
Revised: and it is intractable to get the ground truth for all architectures in this space.

Original: Generally, most existing worksonly report the best searched architecture, making the reproducibility of architecture search weak in this space.
Revised: Generally, most existing worksonly report the best searched architecture, making the reproducibility of architecture search weak in this space.

Original: For fair comparison, we also follow the common setting [27, 43] to conduct the architecture search on the supernet used in https
Revised: For fair comparison, we also follow the common setting [27, 43] to conduct the architecture search on the supernet used in https

Original: /github
Revised: /github

Original: [Table]
Revised: [Table]

Original: “Param" is the model size when applied on CIFAR-10, while “ + × " (the number of multiply-add operations) is calculated based on the ImageNet dataset.
Revised: “Param" is the model size when applied on CIFAR-10, while “ + × " (the number of multiply-add operations) is calculated based on the ImageNet dataset.

Original: In our experiments, we adjust the number of initial filters to restrict “ + × " to be less than 600M on the ImageNet and CIFAR10.
Revised: In our experiments, we adjust the number of initial filters to restrict “ + × " to be less than 600M on the ImageNet and CIFAR10.

Original: † means the computational time is calculated with the same environment as [40].
Revised: † means the computational time is calculated with the same environment as [40].

Original: The unit in “Search Cost" is
Revised: The unit in “Search Cost" is

Original: GPU day.
Revised: GPU day.

Original: [Table]
Revised: [Table]

Original: ImageNet and to comparable model size with SOTAs on CIFAR10.
Revised: ImageNet and to comparable model size with SOTAs on CIFAR10.

Original: Table 5 presents the comparison results with the SOTA differentiable NAS methods.
Revised: Table 5 presents the comparison results with the SOTA differentiable NAS methods.

Original: As shown, our DARTS-IM 727 improves the DARTS baselines by a large margin, in terms of test error on CIFAR-10, CIFAR-100, and ImageNet,respectively, again verifying the effectiveness of the proposed influential magnitude.
Revised: As shown, our DARTS-IM 727 improves the DARTS baselines by a large margin, in terms of test error on CIFAR-10, CIFAR-100, and ImageNet,respectively, again verifying the effectiveness of the proposed influential magnitude.

Original: More interesting, instead of 729 changing the complicated supernet training part, we found that simply replacing the magnitude with the proposedinfluential magnitude could consistently improve the performance of DARTS with only additional hundred 731 seconds computational time.
Revised: More interesting, instead of 729 changing the complicated supernet training part, we found that simply replacing the magnitude with the proposedinfluential magnitude could consistently improve the performance of DARTS with only additional hundred 731 seconds computational time.

Original: Different from the DARTS-PT which contains the supernet retraining that greatlyincreases the search time, our DARTS-IM only poses additional hundred seconds in DARTS space.
Revised: Different from the DARTS-PT which contains the supernet retraining that greatlyincreases the search time, our DARTS-IM only poses additional hundred seconds in DARTS space.

Original: We could 733 also find that, although the best CIFAR-10 test error by DARTS-PT is comparable with our DARTS-IM, our bestsearched architecture obtains lower test error in CIFAR-100 and ImageNet, showing better transferability.
Revised: We could 733 also find that, although the best CIFAR-10 test error by DARTS-PT is comparable with our DARTS-IM, our bestsearched architecture obtains lower test error in CIFAR-100 and ImageNet, showing better transferability.

Original: F Analyse the inverse Hessian approximation
Revised: F Analyse the inverse Hessian approximation

Original: In this section, we detailed discuss the effects of different inverse Hessian approximation methods on the 737 proposed influence magnitude.
Revised: In this section, we detailed discuss the effects of different inverse Hessian approximation methods on the 737 proposed influence magnitude.

Original: Apart from the two devised method, Neumann series and Sherman-Morrisonformula, we also consider two commonly-used approaches in approximating the inverse Hessian, identity 739 approximation and diagonal approximation, which consider the Hessian matrix as an identity matrix anddiagonal matrix, respectively.
Revised: Apart from the two devised method, Neumann series and Sherman-Morrisonformula, we also consider two commonly-used approaches in approximating the inverse Hessian, identity 739 approximation and diagonal approximation, which consider the Hessian matrix as an identity matrix anddiagonal matrix, respectively.

Original: We conduct a series of experiments on NAS-Bench-201 to analyze different 741 approximation methods, and all comparison results are reported in Table 6.
Revised: We conduct a series of experiments on NAS-Bench-201 to analyze different 741 approximation methods, and all comparison results are reported in Table 6.

Original: As shown, even with the identityapproximation, our influential magnitude I M can achieve satisfying results compared with the random baselineand two heuristic methods.
Revised: As shown, even with the identityapproximation, our influential magnitude I M can achieve satisfying results compared with the random baselineand two heuristic methods.

Original: This result indicates that the second-order information ∂ 2 L ( θ ∗ ,α ) ∂α∂θ and ∂ 2 L ( θ ∗ ,α ) ∂θ∂α can 744 provide useful information for operation strength estimation.
Revised: This result indicates that the second-order information ∂ 2 L ( θ ∗ ,α ) ∂α∂θ and ∂ 2 L ( θ ∗ ,α ) ∂θ∂α can 744 provide useful information for operation strength estimation.

Original: More interesting, when considering the diagonal
Revised: More interesting, when considering the diagonal

Original: Hessian information, we found that our DARTS-IM-D could also achieve excellent performance.
Revised: Hessian information, we found that our DARTS-IM-D could also achieve excellent performance.

Original: The results of
Revised: The results of

Original: [Figure]
Revised: [Figure]

Original: identity approximation and diagonal approximation demonstrate the robustness of our influential magnitude, 747 which can still achieve excellent results with different inverse Hessian approximation methods.
Revised: identity approximation and diagonal approximation demonstrate the robustness of our influential magnitude, 747 which can still achieve excellent results with different inverse Hessian approximation methods.

Original: However, we also found that an incorrect approximation may lead to poor performance.
Revised: However, we also found that an incorrect approximation may lead to poor performance.

Original: Similar to Figure 2, 749 we plots the full NAS-Bench-201 results of our influential magnitude under different inverse Hessian matrixapproximation methods along with the number of batches N , on the CIFAR10, CIFAR100, and ImageNet, 751 respectively.
Revised: Similar to Figure 2, 749 we plots the full NAS-Bench-201 results of our influential magnitude under different inverse Hessian matrixapproximation methods along with the number of batches N , on the CIFAR10, CIFAR100, and ImageNet, 751 respectively.

Original: We compared our DARTS-IM with the DARTS baseline and also the DARTS-PT whose results aredirectly obtained from [40].
Revised: We compared our DARTS-IM with the DARTS baseline and also the DARTS-PT whose results aredirectly obtained from [40].

Original: Please note that, DARTS-PT smooth the line extremely with only report the tendency 753 of the architecture selection during the search.
Revised: Please note that, DARTS-PT smooth the line extremely with only report the tendency 753 of the architecture selection during the search.

Original: We can found that the Neumann series approximation could notobtain stable results, especially when the number of batches increases that the performance of Neumann series 755 even performs more poorly than identity approximation.
Revised: We can found that the Neumann series approximation could notobtain stable results, especially when the number of batches increases that the performance of Neumann series 755 even performs more poorly than identity approximation.

Original: As discussed in Section 6.1, these results also verifythat the stochastic Neumann series estimation for IHVP is somehow erroneous.
Revised: As discussed in Section 6.1, these results also verifythat the stochastic Neumann series estimation for IHVP is somehow erroneous.

Original: Generally, we found that the 757 Sherman-Morrison formula is the most practical in estimating the inverse Hessian for our influential magnitude.
Revised: Generally, we found that the 757 Sherman-Morrison formula is the most practical in estimating the inverse Hessian for our influential magnitude.

Original: G Visualization of Searched Architectures on Different Spaces
Revised: G Visualization of Searched Architectures on Different Spaces

Original: In Figure 7 and 8, we visualize all searched architectures by DARTS baselines, DARTS-PT and also our 760 DARTS-IM on different search space, including NAS-Bench-201 [12], tool search spaces proposed by [45], and
Revised: In Figure 7 and 8, we visualize all searched architectures by DARTS baselines, DARTS-PT and also our 760 DARTS-IM on different search space, including NAS-Bench-201 [12], tool search spaces proposed by [45], and

Original: DARTS search space [27].
Revised: DARTS search space [27].

Original: First, we analyze these architectures searched on NAS-Bench-201 [12] and tool search spaces proposed by 763 [40, 45], since which helps us to clearly demonstrate the robustness of different architecture selection methods.
Revised: First, we analyze these architectures searched on NAS-Bench-201 [12] and tool search spaces proposed by 763 [40, 45], since which helps us to clearly demonstrate the robustness of different architecture selection methods.

Original: As observed by [40, 45], the most notable drawback of DARTS baseline is the robustness and generalization, 765 where DARTS fails to select intensive skip-connections with the search progressing.
Revised: As observed by [40, 45], the most notable drawback of DARTS baseline is the robustness and generalization, 765 where DARTS fails to select intensive skip-connections with the search progressing.

Original: Figure 7 (e) and (i) plots thesearched architectures on NAS-Bench-201, respectively.
Revised: Figure 7 (e) and (i) plots thesearched architectures on NAS-Bench-201, respectively.

Original: As shown, the DARTS baseline selects skip-connection 767 for all edges, while DARTS-PT can find a valid architecture with only 3 skip-connections when considering aperturbation-based selection paradigm.
Revised: As shown, the DARTS baseline selects skip-connection 767 for all edges, while DARTS-PT can find a valid architecture with only 3 skip-connections when considering aperturbation-based selection paradigm.

Original: More inspiring, the proposed influential magnitude ( I M ) can select more 769 competitive architecture which also only contains one skip-connection .
Revised: More inspiring, the proposed influential magnitude ( I M ) can select more 769 competitive architecture which also only contains one skip-connection .

Original: Please note that, the trained supernet
Revised: Please note that, the trained supernet

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: keeps identical for DARTS, DARTS-PT
Revised: keeps identical for DARTS, DARTS-PT

Original: [Table]
Revised: [Table]

Original: and our DARTS-IM, which only adopt different architecture selection 771 methods, e
Revised: and our DARTS-IM, which only adopt different architecture selection 771 methods, e

Original: [Table]
Revised: [Table]

Original: g., argmax , perturbation-based selection
Revised: g., argmax , perturbation-based selection

Original: [Table]
Revised: [Table]

Original: and the proposed influential magnitude.
Revised: and the proposed influential magnitude.

Original: Similarly, we can also observe this superiority of our I M in the tool search spaces proposed by [40, 45]
Revised: Similarly, we can also observe this superiority of our I M in the tool search spaces proposed by [40, 45]

Original: [Table]
Revised: [Table]

Original: 773 especially in the search space S1 and S2.
Revised: 773 especially in the search space S1 and S2.

Original: We can find that, our DARTS-IM clearly outperforms DARTS and
Revised: We can find that, our DARTS-IM clearly outperforms DARTS and

Original: DARTS-PT.
Revised: DARTS-PT.

Original: Although DARTS-PT can partially relieves this instability, it still selects intensive skip-connections .
Revised: Although DARTS-PT can partially relieves this instability, it still selects intensive skip-connections .

Original: In the contrary, our DARTS-IM generally prefers convolutional operations.
Revised: In the contrary, our DARTS-IM generally prefers convolutional operations.

Original: However, although our DARTS-IMoutperforms DARTS baseline in the search space S3, we found that DARTS-PT achieves excellent results in 777 this space.
Revised: However, although our DARTS-IMoutperforms DARTS baseline in the search space S3, we found that DARTS-PT achieves excellent results in 777 this space.

Original: One potential is that this space contains intensive none operations which is naturally removed byperturbation-based selection, since removing none will not affect the supernet performance.
Revised: One potential is that this space contains intensive none operations which is naturally removed byperturbation-based selection, since removing none will not affect the supernet performance.

Original: We also conduct 779 the architecture search on the search space S4, while we found that our DARTS-IM also performs poorly as
Revised: We also conduct 779 the architecture search on the search space S4, while we found that our DARTS-IM also performs poorly as

Original: DARTS that select some noise operations.
Revised: DARTS that select some noise operations.

Original: One underlying reason is that it is impossible to train the supernet to 781 optimal in this space since the noise operation will return a random noise regardless of the input.
Revised: One underlying reason is that it is impossible to train the supernet to 781 optimal in this space since the noise operation will return a random noise regardless of the input.

Original: We also showthe comparison results in the Table 7 with different DARTS baselines.
Revised: We also showthe comparison results in the Table 7 with different DARTS baselines.

Original: We directly report the results of DARTS 783 and DARTS-ES from [45], which contain the mean and std.
Revised: We directly report the results of DARTS 783 and DARTS-ES from [45], which contain the mean and std.

Original: The results of DARTS-PT is also inherited from theoriginal paper [40], which only report the best single run.
Revised: The results of DARTS-PT is also inherited from theoriginal paper [40], which only report the best single run.

Original: In Table 7, we report the average and best single run 785 of our DARTS-IM together for better comparison.
Revised: In Table 7, we report the average and best single run 785 of our DARTS-IM together for better comparison.

Original: Then, we also visualized our searched architectures in the DARTS search space.
Revised: Then, we also visualized our searched architectures in the DARTS search space.

Original: Interesting, compared with the 787 DARTS baseline, architecture searched by our DARTS-IM contains several sep_conv 5 × 5 operations in thenormal cell, where this phenomenon has never been observed by existing popular DARTS based algorithms.
Revised: Interesting, compared with the 787 DARTS baseline, architecture searched by our DARTS-IM contains several sep_conv 5 × 5 operations in thenormal cell, where this phenomenon has never been observed by existing popular DARTS based algorithms.

Original: Similarly, we also found that our reduction cell contains several sep_conv 5 × 5 operations, which is also in
Revised: Similarly, we also found that our reduction cell contains several sep_conv 5 × 5 operations, which is also in

Original: [Figure]
Revised: [Figure]

Original: output
Revised: output

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: c_{k-2}
Revised: c_{k-2}

Original: [Figure]
Revised: [Figure]

Original: _connect
Revised: _connect

Original: [Figure]
Revised: [Figure]

Original: _connect c_{k-1} sep_conv_3x skip_
Revised: _connect c_{k-1} sep_conv_3x skip_

Original: [Figure]
Revised: [Figure]

Original: sep_conv_
Revised: sep_conv_

Original: [Figure]
Revised: [Figure]

Original: sep_
Revised: sep_

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (g) DARTS
Revised: (g) DARTS

Original: [Figure]
Revised: [Figure]

Original: PT
Revised: PT

Original: [Figure]
Revised: [Figure]

Original: S
Revised: S

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: c_{k
Revised: c_{k

Original: [Figure]
Revised: [Figure]

Original: 1} sep_
Revised: 1} sep_

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (i
Revised: (i

Original: [Figure]
Revised: [Figure]

Original: (j)
Revised: (j)

Original: [Figure]
Revised: [Figure]

Original: : S
Revised: : S

Original: [Figure]
Revised: [Figure]

Original: line with another influence based method DARTS-PT and PC-DARTS.
Revised: line with another influence based method DARTS-PT and PC-DARTS.

Original: More important, different from most 791 existing works that the normal cell generally contains more parameters than the reduction cell, we found that oursearched architecture prefers more parameters in the reduction cell.
Revised: More important, different from most 791 existing works that the normal cell generally contains more parameters than the reduction cell, we found that oursearched architecture prefers more parameters in the reduction cell.

Original: This observation is more consistent with the 793 common experience in the network pruning [24, 25, 39], where we found that most pruning methods will retainmore parameters in the reduction cell and prune more redundant parameters in the normal cell.
Revised: This observation is more consistent with the 793 common experience in the network pruning [24, 25, 39], where we found that most pruning methods will retainmore parameters in the reduction cell and prune more redundant parameters in the normal cell.

Original: H Overall Algorithm Framework Description
Revised: H Overall Algorithm Framework Description

Original: In this paper, we proposed two different approaches for the operation selection in DARTS, called DARTS-IF 797 (described in Sec.3) and DARTS-IM (described in Sec.4).
Revised: In this paper, we proposed two different approaches for the operation selection in DARTS, called DARTS-IF 797 (described in Sec.3) and DARTS-IM (described in Sec.4).

Original: This two approaches are based on the influencefunctions to measure the operation importance, where the main different is that DARTS-IF removes a candidate 799 operation in each step to measure the loss change in a perturbation manner, while DARTS-IM calculate the losssensitivity when applying an infinitesimal change on α .
Revised: This two approaches are based on the influencefunctions to measure the operation importance, where the main different is that DARTS-IF removes a candidate 799 operation in each step to measure the loss change in a perturbation manner, while DARTS-IM calculate the losssensitivity when applying an infinitesimal change on α .

Original: The framework of our DARTS-IF and DART-IM are 801 sketched in Algorithm 1 and 2.
Revised: The framework of our DARTS-IF and DART-IM are 801 sketched in Algorithm 1 and 2.

Original: Our DARTS-IF shares a similar perturbation paradigm as DARTS-PT, whilewithout any fine-tuning.
Revised: Our DARTS-IF shares a similar perturbation paradigm as DARTS-PT, whilewithout any fine-tuning.

Original: Rather than using the validation performance drop to indicate the operation importance, 803 DARTS-IF leverages influence functions to predict the loss change.
Revised: Rather than using the validation performance drop to indicate the operation importance, 803 DARTS-IF leverages influence functions to predict the loss change.

Original: Different from DARTS-IF and DARTS-PT,our DARTS-IM is with a similar paradigm as DARTS, while whose operation importance is calculated based on 805 I
Revised: Different from DARTS-IF and DARTS-PT,our DARTS-IM is with a similar paradigm as DARTS, while whose operation importance is calculated based on 805 I

Original: M , rather that the optimized α .
Revised: M , rather that the optimized α .

Original: In addition, the operation importance in our DARTS-IM can be calculated by 806 one-shot, rather than iteratively going through all candidates operations in a supernet as perturbation paradigm.
Revised: In addition, the operation importance in our DARTS-IM can be calculated by 806 one-shot, rather than iteratively going through all candidates operations in a supernet as perturbation paradigm.

Original: c_{k-2}
Revised: c_{k-2}

Original: [Figure]
Revised: [Figure]

Original: skip
Revised: skip

Original: [Figure]
Revised: [Figure]

Original: c_{k-1}
Revised: c_{k-1}

Original: [Figure]
Revised: [Figure]

Original: sep
Revised: sep

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: c_{
Revised: c_{

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: c_{k
Revised: c_{k

Original: [Figure]
Revised: [Figure]

Original: 2} 0 avg
Revised: 2} 0 avg

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: c_{k-1} sep_conv_5x 1 max_pool_3x  dil_conv_5x skip_connect 3 sep_conv_5x c_{k} max_pool_3x skip_connect (f) DARTS-PT: Reduction Cell c_{k-2} skip_connect 1 sep_conv_3x3sep_conv_5x5avg_pool_3x c_{k-1} sep_conv_3x dil_conv_3x3 sep_conv_3x dil_conv_3x
Revised: c_{k-1} sep_conv_5x 1 max_pool_3x  dil_conv_5x skip_connect 3 sep_conv_5x c_{k} max_pool_3x skip_connect (f) DARTS-PT: Reduction Cell c_{k-2} skip_connect 1 sep_conv_3x3sep_conv_5x5avg_pool_3x c_{k-1} sep_conv_3x dil_conv_3x3 sep_conv_3x dil_conv_3x

Original: c_{k} (g) PC-DARTS:
Revised: c_{k} (g) PC-DARTS:

Original: Normal Cell c_{k-2} max_pool_3x  sep_conv_3x c_{k-1} sep_conv_5x  sep_conv_5x 3 sep_conv_3x sep_conv_5x sep_conv_3x c_{k} sep_conv_3x (h) PC-DARTS:
Revised: Normal Cell c_{k-2} max_pool_3x  sep_conv_3x c_{k-1} sep_conv_5x  sep_conv_5x 3 sep_conv_3x sep_conv_5x sep_conv_3x c_{k} sep_conv_3x (h) PC-DARTS:

Original: Reduction Cell end end for 11: Apply argmax on the operation strength ∆ L and derive the discrete architecture α ∗ accordingly.
Revised: Reduction Cell end end for 11: Apply argmax on the operation strength ∆ L and derive the discrete architecture α ∗ accordingly.

Original: In calculating the influence magnitude in our DARTS-IM or the loss change in DARTS-IF, we have three more 808 hyperparameters, γ for
Revised: In calculating the influence magnitude in our DARTS-IM or the loss change in DARTS-IF, we have three more 808 hyperparameters, γ for

Original: [Equation]
Revised: [Equation]