Original: T HE D ARK S IDE OF A UTO ML: T OWARDS A RCHITECTURAL B ACKDOOR S EARCH
Revised: T HE D ARK S IDE OF A UTO ML: T OWARDS A RCHITECTURAL B ACKDOOR S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper asks the intriguing question: is it possible to exploit neural architecture search (NAS) as a new attack vector to launch previously improbable attacks?
Revised: This paper asks the intriguing question: is it possible to exploit neural architecture search (NAS) as a new attack vector to launch previously improbable attacks?

Original: Specifically, we present E VAS , a new attack that leverages NAS to find neural architectures with inherent backdoors and exploits such vulnerability using input-aware triggers.
Revised: Specifically, we present E VAS , a new attack that leverages NAS to find neural architectures with inherent backdoors and exploits such vulnerability using input-aware triggers.

Original: Compared with existing attacks, E VAS demonstrates many interesting properties: ( i ) it does not require polluting training data or perturbing model parameters; ( ii ) it is agnostic to downstream fine-tuning or even re-training from scratch; ( iii ) it naturally evades defenses that rely on inspecting model parameters or training data.
Revised: Compared with existing attacks, E VAS demonstrates many interesting properties: ( i ) it does not require polluting training data or perturbing model parameters; ( ii ) it is agnostic to downstream fine-tuning or even re-training from scratch; ( iii ) it naturally evades defenses that rely on inspecting model parameters or training data.

Original: With extensive evaluation on benchmark datasets, we show that
Revised: With extensive evaluation on benchmark datasets, we show that

Original: E VAS features high evasiveness, transferability and robustness, thereby expanding the adversary’s design spectrum.
Revised: E VAS features high evasiveness, transferability and robustness, thereby expanding the adversary’s design spectrum.

Original: We further characterize the mechanisms underlying E VAS , which are possibly explainable by architecture-level “shortcuts” that recognize trigger patterns.
Revised: We further characterize the mechanisms underlying E VAS , which are possibly explainable by architecture-level “shortcuts” that recognize trigger patterns.

Original: This work raises concerns about the current practice of
Revised: This work raises concerns about the current practice of

Original: NAS and points to potential directions to develop effective countermeasures
Revised: NAS and points to potential directions to develop effective countermeasures

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: As a new paradigm of applying ML techniques in practice, automated machine learning (AutoML) automates the pipeline from raw data to deployable models, which covers model design, optimizer selection, and parameter tuning.
Revised: As a new paradigm of applying ML techniques in practice, automated machine learning (AutoML) automates the pipeline from raw data to deployable models, which covers model design, optimizer selection, and parameter tuning.

Original: The use of AutoML greatly simplifies the ML development cycles and propels the trend of ML democratization.
Revised: The use of AutoML greatly simplifies the ML development cycles and propels the trend of ML democratization.

Original: In particular, neural architecture search (NAS), one primary AutoML task, aims to find performant deep neural network (DNN) arches 1 tailored to given datasets.
Revised: In particular, neural architecture search (NAS), one primary AutoML task, aims to find performant deep neural network (DNN) arches 1 tailored to given datasets.

Original: In many cases, NAS is shown to find models remarkably outperforming manually designed ones (Pham et al., 2018;
Revised: In many cases, NAS is shown to find models remarkably outperforming manually designed ones (Pham et al., 2018;

Original: Liu et al., 2019; Li et al., 2020).
Revised: Liu et al., 2019; Li et al., 2020).

Original: In contrast to the intensive research on improving the capability of NAS, its security implications are largely unexplored.
Revised: In contrast to the intensive research on improving the capability of NAS, its security implications are largely unexplored.

Original: As ML models are becoming the new targets of malicious attacks (Biggio &
Revised: As ML models are becoming the new targets of malicious attacks (Biggio &

Original: Roli, 2018), the lack of understanding about the risks of NAS is highly concerning, given its surging popularity in security-sensitive domains (Pang et al., 2022).
Revised: Roli, 2018), the lack of understanding about the risks of NAS is highly concerning, given its surging popularity in security-sensitive domains (Pang et al., 2022).

Original: Towards bridging this striking gap, we pose the intriguing yet critical question:
Revised: Towards bridging this striking gap, we pose the intriguing yet critical question:

Original: Is it possible
Revised: Is it possible

Original: for the adversary to exploit
Revised: for the adversary to exploit

Original: NAS to launch previously improbable attacks?
Revised: NAS to launch previously improbable attacks?

Original: This work provides an affirmative answer to this question.
Revised: This work provides an affirmative answer to this question.

Original: We present exploitable and vulnerable arch search (E VAS ), a new backdoor attack that leverages NAS to find neural arches with inherent, exploitable vulnerability.
Revised: We present exploitable and vulnerable arch search (E VAS ), a new backdoor attack that leverages NAS to find neural arches with inherent, exploitable vulnerability.

Original: Conventional backdoor attacks typically embed the malicious functions (“backdoors”) into the space of model parameters.
Revised: Conventional backdoor attacks typically embed the malicious functions (“backdoors”) into the space of model parameters.

Original: They often assume strong threat models, such as polluting training data (Gu et al., 2017; Liu et al., 2018; Pang et al., 2020) or perturbing model parameters (Ji et al., 2018;
Revised: They often assume strong threat models, such as polluting training data (Gu et al., 2017; Liu et al., 2018; Pang et al., 2020) or perturbing model parameters (Ji et al., 2018;

Original: Qi et al., 2022), and are thus subject to defenses based on model inspection (Wang et al., 2019; Liu et al., 2019) and data filtering (Gao et al., 2019).
Revised: Qi et al., 2022), and are thus subject to defenses based on model inspection (Wang et al., 2019; Liu et al., 2019) and data filtering (Gao et al., 2019).

Original: In E VAS , however, as the backdoors are carried in the space of model arches, even if the victim trains the models using clean data and operates them in a black-box manner, the backdoors are still retained.
Revised: In E VAS , however, as the backdoors are carried in the space of model arches, even if the victim trains the models using clean data and operates them in a black-box manner, the backdoors are still retained.

Original: Moreover, due to its independence of model parameters or training data, E VAS is naturally robust against defenses such as model inspection and input filtering.
Revised: Moreover, due to its independence of model parameters or training data, E VAS is naturally robust against defenses such as model inspection and input filtering.

Original: To realize E VAS , we define a novel metric based on neural tangent kernel (Chen et al., 2021), which effectively indicates the exploitable vulnerability of a given arch; further, we integrate this metric into the NAS-without-training framework (Mellor et al., 2021; Chen et al., 2021).
Revised: To realize E VAS , we define a novel metric based on neural tangent kernel (Chen et al., 2021), which effectively indicates the exploitable vulnerability of a given arch; further, we integrate this metric into the NAS-without-training framework (Mellor et al., 2021; Chen et al., 2021).

Original: The resulting search method is able to efficiently identify candidate arches without requiring model training or backdoor testing.
Revised: The resulting search method is able to efficiently identify candidate arches without requiring model training or backdoor testing.

Original: To verify E VAS ’s empirical effectiveness, we evaluate E VAS on benchmark datasets and show: ( i ) E VAS successfully finds arches with exploitable vulnerability, ( ii ) the injected backdoors may be explained by arch-level “shortcuts” that recognize trigger patterns, and ( iii ) E VAS demonstrates high evasiveness, transferability, and robustness against defenses.
Revised: To verify E VAS ’s empirical effectiveness, we evaluate E VAS on benchmark datasets and show: ( i ) E VAS successfully finds arches with exploitable vulnerability, ( ii ) the injected backdoors may be explained by arch-level “shortcuts” that recognize trigger patterns, and ( iii ) E VAS demonstrates high evasiveness, transferability, and robustness against defenses.

Original: Our findings show the feasibility of exploiting NAS as a new attack vector to implement previously improbable attacks, raise concerns about the current practice of NAS in security-sensitive domains, and point to potential directions to develop effective mitigation.
Revised: Our findings show the feasibility of exploiting NAS as a new attack vector to implement previously improbable attacks, raise concerns about the current practice of NAS in security-sensitive domains, and point to potential directions to develop effective mitigation.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Next, we survey the literature relevant to this work.
Revised: Next, we survey the literature relevant to this work.

Original: Neural arch search.
Revised: Neural arch search.

Original: The existing NAS methods can be categorized along search space, search strategy, and performance measure.
Revised: The existing NAS methods can be categorized along search space, search strategy, and performance measure.

Original: Search space – early methods focus on the chain-of-layer structure (Baker et al., 2017), while recent work proposes to search for motifs of cell structures (Zoph et al., 2018; Pham et al., 2018;
Revised: Search space – early methods focus on the chain-of-layer structure (Baker et al., 2017), while recent work proposes to search for motifs of cell structures (Zoph et al., 2018; Pham et al., 2018;

Original: Liu et al., 2019).
Revised: Liu et al., 2019).

Original: Search strategy – early methods rely on either random search (Jozefowicz et al., 2015) or Bayesian optimization (Bergstra et al., 2013), which are limited in model complexity; recent work mainly uses the approaches of reinforcement learning (Baker et al., 2017) or neural evolution (Liu et al., 2019).
Revised: Search strategy – early methods rely on either random search (Jozefowicz et al., 2015) or Bayesian optimization (Bergstra et al., 2013), which are limited in model complexity; recent work mainly uses the approaches of reinforcement learning (Baker et al., 2017) or neural evolution (Liu et al., 2019).

Original: Performance measure – one-shot NAS has emerged as a popular performance measure.
Revised: Performance measure – one-shot NAS has emerged as a popular performance measure.

Original: It considers all candidate arches as different sub-graphs of a super-net ( i.e. , the one-shot model) and shares weights between candidate arches (Liu et al., 2019).
Revised: It considers all candidate arches as different sub-graphs of a super-net ( i.e. , the one-shot model) and shares weights between candidate arches (Liu et al., 2019).

Original: Despite the intensive research on NAS, its security implications are largely unexplored.
Revised: Despite the intensive research on NAS, its security implications are largely unexplored.

Original: Recent work shows that NAS-generated models tend to be more vulnerable to various malicious attacks than manually designed ones (Pang et al., 2022; Devaguptapu et al., 2021).
Revised: Recent work shows that NAS-generated models tend to be more vulnerable to various malicious attacks than manually designed ones (Pang et al., 2022; Devaguptapu et al., 2021).

Original: This work explores another dimension: whether it can be exploited as an attack vector to launch new attacks, which complements the existing studies on the security of NAS.
Revised: This work explores another dimension: whether it can be exploited as an attack vector to launch new attacks, which complements the existing studies on the security of NAS.

Original: Backdoor attacks and defenses.
Revised: Backdoor attacks and defenses.

Original: Backdoor attacks inject malicious backdoors into the victim’s model during training and activate such backdoors at inference, which can be categorized along attack targets – input-specific (Shafahi et al., 2018), class-specific (Tang et al., 2020), or any-input (Gu et al., 2017), attack vectors – polluting training data (Liu et al., 2018) or releasing infected models (Ji et al., 2018), and optimization metrics – attack effectiveness (Pang et al., 2020), transferability (Yao et al., 2019), or attack evasiveness(Chen et al., 2017).
Revised: Backdoor attacks inject malicious backdoors into the victim’s model during training and activate such backdoors at inference, which can be categorized along attack targets – input-specific (Shafahi et al., 2018), class-specific (Tang et al., 2020), or any-input (Gu et al., 2017), attack vectors – polluting training data (Liu et al., 2018) or releasing infected models (Ji et al., 2018), and optimization metrics – attack effectiveness (Pang et al., 2020), transferability (Yao et al., 2019), or attack evasiveness(Chen et al., 2017).

Original: To mitigate such threats, many defenses have also been proposed, which can be categorized according to their strategies (Pang et al., 2022): input filtering purges poisoning samples from training data (Tran et al., 2018); model inspection determines whether a given model is backdoored(Liu et al., 2019;
Revised: To mitigate such threats, many defenses have also been proposed, which can be categorized according to their strategies (Pang et al., 2022): input filtering purges poisoning samples from training data (Tran et al., 2018); model inspection determines whether a given model is backdoored(Liu et al., 2019;

Original: Wang et al., 2019), and input inspection detects trigger inputs at inference time (Gao et al., 2019).
Revised: Wang et al., 2019), and input inspection detects trigger inputs at inference time (Gao et al., 2019).

Original: Most attacks and defenses above focus on backdoors implemented in the space of model parameters.
Revised: Most attacks and defenses above focus on backdoors implemented in the space of model parameters.

Original: Concurrent to this work, Bober-Irizar et al.
Revised: Concurrent to this work, Bober-Irizar et al.

Original: (2022) explore using neural arches to implement backdoors by manually designing “trigger detectors” in the arches and activating such detectors using poisoning data during training.
Revised: (2022) explore using neural arches to implement backdoors by manually designing “trigger detectors” in the arches and activating such detectors using poisoning data during training.

Original: This work investigates using NAS to directly search for arches with exploitable vulnerability, which represents a new direction of backdoor attacks.
Revised: This work investigates using NAS to directly search for arches with exploitable vulnerability, which represents a new direction of backdoor attacks.

Original:  E VAS
Revised:  E VAS

Original: Next, we present E VAS , a new backdoor attack leveraging NAS to find neural arches with exploitable vulnerability.
Revised: Next, we present E VAS , a new backdoor attack leveraging NAS to find neural arches with exploitable vulnerability.

Original: We begin by introducing the threat model.
Revised: We begin by introducing the threat model.

Original: 3. T HREAT M ODEL
Revised: 3. T HREAT M ODEL

Original: A backdoor attack injects a hidden malicious function (“backdoor”) into a target model (Pang et al., 2022).
Revised: A backdoor attack injects a hidden malicious function (“backdoor”) into a target model (Pang et al., 2022).

Original: The backdoor is activated once a pre-defined condition (“trigger”) is present, while the model behaves normally otherwise.
Revised: The backdoor is activated once a pre-defined condition (“trigger”) is present, while the model behaves normally otherwise.

Original: In a predictive task, the backdoor is often defined as classifying a given input to a class desired by the adversary, while the trigger can be defined as a specific perturbation
Revised: In a predictive task, the backdoor is often defined as classifying a given input to a class desired by the adversary, while the trigger can be defined as a specific perturbation

Original: [Table]
Revised: [Table]

Original: /aD86iJIlGnVWPW/YWkTv/wNTOMFp</latexit> x
Revised: /aD86iJIlGnVWPW/YWkTv/wNTOMFp</latexit> x

Original: [Figure]
Revised: [Figure]

Original: applied to the input.
Revised: applied to the input.

Original: Formally, given input x and trigger r = ( m, p ) in which m is a mask and p is a pattern, the trigger-embedded input is defined as:
Revised: Formally, given input x and trigger r = ( m, p ) in which m is a mask and p is a pattern, the trigger-embedded input is defined as:

Original: [Equation]
Revised: [Equation]

Original: Let f be the backdoor-infected model.
Revised: Let f be the backdoor-infected model.

Original: The backdoor attack implies that for given input-label pair ( x, y ) , f ( x ) =
Revised: The backdoor attack implies that for given input-label pair ( x, y ) , f ( x ) =

Original: y and f (˜ x ) =
Revised: y and f (˜ x ) =

Original: t with high probability, where t is the adversary’s target class.
Revised: t with high probability, where t is the adversary’s target class.

Original: The conventional backdoor attacks typically follow two types of threat models: ( i ) the adversary directly trains a backdoor-embedded model, which is then released to and used by the victim user (Liu et al., 2018; Pang et al., 2020; Ji et al., 2018); or ( ii ) the adversary indirectly pollutes the training data or manipulate the training process (Gu et al., 2017; Qi et al., 2022) to inject the backdoor into the target model.
Revised: The conventional backdoor attacks typically follow two types of threat models: ( i ) the adversary directly trains a backdoor-embedded model, which is then released to and used by the victim user (Liu et al., 2018; Pang et al., 2020; Ji et al., 2018); or ( ii ) the adversary indirectly pollutes the training data or manipulate the training process (Gu et al., 2017; Qi et al., 2022) to inject the backdoor into the target model.

Original: As illustrated in Figure 1, in E VAS , we assume a more practical threat model in which the adversary only releases the exploitable arch to the user, who may choose to train the model  usingarbitrary data ( e.g. , clean data) or apply various defenses ( e.g. , model inspection or data filtering) before or during using the model.
Revised: As illustrated in Figure 1, in E VAS , we assume a more practical threat model in which the adversary only releases the exploitable arch to the user, who may choose to train the model from scratch using clean data or apply various defenses ( e.g. , model inspection or data filtering) before or during using the model.

Original: We believe this represents a more realistic setting: due to the prohibitive computational cost of NAS, users may opt to use performant model arches provided by third parties, which opens the door for the adversary to launch the E VAS attack.
Revised: We believe this represents a more realistic setting: due to the prohibitive computational cost of NAS, users may opt to use performant model arches provided by third parties, which opens the door for the adversary to launch the E VAS attack.

Original: However, realizing E VAS represents non-trivial challenges including ( i ) how to define the trigger patterns?
Revised: However, realizing E VAS represents non-trivial challenges including ( i ) how to define the trigger patterns?

Original: ( ii ) how to define the exploitable, vulnerable arches?
Revised: ( ii ) how to define the exploitable, vulnerable arches?

Original: and ( iii ) how to search for such arches efficiently?
Revised: and ( iii ) how to search for such arches efficiently?

Original: Below we elaborate on each of these key questions.
Revised: Below we elaborate on each of these key questions.

Original: 3. I NPUT -A WARE T RIGGERS
Revised: 3. I NPUT -A WARE T RIGGERS

Original: Most conventional backdoor attacks assume universal triggers: the same trigger is applied to all the inputs.
Revised: Most conventional backdoor attacks assume universal triggers: the same trigger is applied to all the inputs.

Original: However, universal triggers can be easily detected and mitigated by current defenses (Wang et al., 2019; Liu et al., 2019).
Revised: However, universal triggers can be easily detected and mitigated by current defenses (Wang et al., 2019; Liu et al., 2019).

Original: Moreover, it is shown that implementing universal triggers at the arch level requires manually designing “trigger detectors” in the arches and activating such detectors using poisoning data during training (Bober-Irizar et al., 2022), which does not fit our threat model.
Revised: Moreover, it is shown that implementing universal triggers at the arch level requires manually designing “trigger detectors” in the arches and activating such detectors using poisoning data during training (Bober-Irizar et al., 2022), which does not fit our threat model.

Original: Instead, as illustrated in Figure 1, we adopt input-aware triggers (Nguyen & Tran, 2020), in which a trigger generator g (parameterized by ϑ ) generates trigger r x specific to each input x .
Revised: Instead, as illustrated in Figure 1, we adopt input-aware triggers (Nguyen & Tran, 2020), in which a trigger generator g (parameterized by ϑ ) generates trigger r x specific to each input x .

Original: Compared with universal triggers, it is more challenging to detect or mitigate input-aware triggers.
Revised: Compared with universal triggers, it is more challenging to detect or mitigate input-aware triggers.

Original: Interestingly, because of the modeling capacity of the trigger generator, it is more feasible to implement input-aware triggers at the arch level (details in § 4).
Revised: Interestingly, because of the modeling capacity of the trigger generator, it is more feasible to implement input-aware triggers at the arch level (details in § 4).

Original: For simplicity, below we use ˜ x
Revised: For simplicity, below we use ˜ x

Original: = g ( x ; ϑ ) to denote both generating trigger r x for x and applying r x to x to generate the trigger-embedded input ˜ x .
Revised: = g ( x ; ϑ ) to denote both generating trigger r x for x and applying r x to x to generate the trigger-embedded input ˜ x .

Original: 3. E XPLOITABLE ARCHES
Revised: 3. E XPLOITABLE ARCHES

Original: In E VAS , we aim to find arches with backdoors exploitable by the trigger generator, which we define as the following optimization problem.
Revised: In E VAS , we aim to find arches with backdoors exploitable by the trigger generator, which we define as the following optimization problem.

Original: Specifically, let α and θ respectively denote f ’s arch and model parameters.
Revised: Specifically, let α and θ respectively denote f ’s arch and model parameters.

Original: We define f ’s training as minimizing the following loss:
Revised: We define f ’s training as minimizing the following loss:

Original: [Equation]
Revised: [Equation]

Original: where f α denotes the model with arch fixed as α and D is the underlying data distribution.
Revised: where f α denotes the model with arch fixed as α and D is the underlying data distribution.

Original: As θ is dependent on α , we define:
Revised: As θ is dependent on α , we define:

Original: [Equation]
Revised: [Equation]

Original: Further, we define the backdoor attack objective as:
Revised: Further, we define the backdoor attack objective as:

Original: [Equation]
Revised: [Equation]

Original: where the first term specifies that f works normally on clean data, the second term specifies that f classifies trigger-embedded inputs to target class t , and the parameter λ balances the two factors.
Revised: where the first term specifies that f works normally on clean data, the second term specifies that f classifies trigger-embedded inputs to target class t , and the parameter λ balances the two factors.

Original: Note that we assume the testing data follows the same distribution D as the training data.
Revised: Note that we assume the testing data follows the same distribution D as the training data.

Original: Overall, we consider an arch α ∗ having exploitable vulnerability if it is possible to find a trigger generator ϑ ∗ , such that L atk ( α ∗ , ϑ ∗ ) is below a certain threshold.
Revised: Overall, we consider an arch α ∗ having exploitable vulnerability if it is possible to find a trigger generator ϑ ∗ , such that L atk ( α ∗ , ϑ ∗ ) is below a certain threshold.

Original: 3. S EARCH WITHOUT T RAINING
Revised: 3. S EARCH WITHOUT T RAINING

Original: Searching for exploitable archs by directly optimizing Eq.
Revised: Searching for exploitable archs by directly optimizing Eq.

Original: 4 is challenging: the nested optimization requires recomputing θ ( i.e. , re-training model f ) in L trn whenever α is updated; further, as α and ϑ are coupled in L atk , it requires re-training generator g once α is changed.
Revised: 4 is challenging: the nested optimization requires recomputing θ ( i.e. , re-training model f ) in L trn whenever α is updated; further, as α and ϑ are coupled in L atk , it requires re-training generator g once α is changed.

Original: Motivated by recent work (Mellor et al., 2021; Wu et al., 2021; Abdelfattah et al., 2021; Ning et al., 2021) on NAS using easy-to-compute metrics as proxies (without training), we present a novel method of searching for exploitable arches based on neural tangent kernel (NTK)
Revised: Motivated by recent work (Mellor et al., 2021; Wu et al., 2021; Abdelfattah et al., 2021; Ning et al., 2021) on NAS using easy-to-compute metrics as proxies (without training), we present a novel method of searching for exploitable arches based on neural tangent kernel (NTK)

Original: (Jacot et al., 2018) without training the target model or trigger generator.
Revised: (Jacot et al., 2018) without training the target model or trigger generator.

Original: Intuitively, NTK describes model training dynamics by gradient descent (Jacot et al., 2018).
Revised: Intuitively, NTK describes model training dynamics by gradient descent (Jacot et al., 2018;

Original: On theinfinite width limit of  DNNs, NTK becomes constant, which allows closed-form statements to be made about model training (Lee et al., 2020).
Revised: Chizat et al., 2019; Lee et al., 2019). In the limit of infinite-width DNNs, NTK becomes constant, which allows closed-form statements to be made about model training.

Revised: Recent work (Chen et al., 2021; Mok et al., 2022) shows that NTK serves as an effective predictor of model “trainability” ( i.e. , how fast the model converges at early training stages).

Original: Formally, considering model f (parameterized by θ ) mapping input x to a probability vector f
Revised: Formally, considering model f (parameterized by θ ) mapping input x to a probability vector f

Original: ( x ; θ ) (over different classes), the NTK is defined as the product of the Jacobian matrix:
Revised: ( x ; θ ) (over different classes), the NTK is defined as the product of the Jacobian matrix:

Original: [Equation]
Revised: [Equation]

Original: Let λ min ( λ max ) be the smallest (largest) eigenvalue of the empirical NTK ˆΘ( θ )
Revised: Let λ min ( λ max ) be the smallest (largest) eigenvalue of the empirical NTK ˆΘ( θ )

Original: ≜ E ( x,y )
Revised: ≜ E ( x,y )

Original: ∼D Θ( x, θ ) .
Revised: ∼D Θ( x, θ ) .

Original: The condition number κ ≜ λ max /λ min serves as a metric to estimatethe model “trainability” ( i.e. , how fast the model converges at early training stages) (Chen et al., 2021), with a smaller conditional number indicating higher trainability.
Revised: The condition number κ ≜ λ max /λ min serves as a metric to estimate model trainability (Chen et al., 2021), with a smaller conditional number indicating higher trainability.

Original: [Figure]
Revised: [Figure]

Original: e r o f N T K In our context, we consider the trigger generator and the target model as an end-to-end model and measure the empirical NTK of the trigger generator under randomly initialized θ :[Equation]
Revised: In our context, we consider the trigger generator and the target model as an end-to-end model and measure the empirical NTK of the trigger generator under randomly initialized θ :

Original: [Equation]
Revised: [Equation]

Original: where P θ ◦ represents the initialization distribution of θ .
Revised: where P θ ◦ represents the initialization distribution of θ .

Original: Here, we emphasize that the measure should be independent of θ ’s initialization.
Revised: Here, we emphasize that the measure should be independent of θ ’s initialization.

Original: Intuitively, ˆΘ( ϑ ) measures the trigger generator’s trainability with respect to a randomly initialized target model.
Revised: Intuitively, ˆΘ( ϑ ) measures the trigger generator’s trainability with respect to a randomly initialized target model.

Original: The generator’s trainability indicates the easiness of effectively generating input-aware triggers, implying the model’s vulnerability to input-aware backdoor attacks.
Revised: The generator’s trainability indicates the easiness of effectively generating input-aware triggers, implying the model’s vulnerability to input-aware backdoor attacks.

Original: To verify the hypothesis, on the CIFAR10 dataset with the generator configured as in Appendix § A, we measure ˆΘ( ϑ ) with respect to 900 randomly generated arches as well as the model accuracy (ACC) on clean inputs and the attack success rate (ASR) on trigger-embedded inputs, with  the  results shown in Figure 2.
Revised: To verify the hypothesis, on the CIFAR10 dataset with the generator configured as in Appendix § A, we measure ˆΘ( ϑ ) with respect to 900 randomly generated arches as well as the model accuracy (ACC) on clean inputs and the attack success rate (ASR) on trigger-embedded inputs. Specifically, for each arch α , we first train the model f α to measure ACC and then train the trigger generator g with respect to f α on the same dataset to measure ASR, with results shown in Figure 2.

Original: Observe that the conditional number of ˆΘ( ϑ ) has a strong negative correlation with ASR, with a smaller value indicating higher attack vulnerability; meanwhile, it has a limited correlation with ACC, with most of the arches having ACC within the range from 80% to 95%.
Revised: Observe that the conditional number of ˆΘ( ϑ ) has a strong negative correlation with ASR, with a smaller value indicating higher attack vulnerability; meanwhile, it has a limited correlation with ACC, with most of the arches having ACC within the range from 80% to 95%.

Original: Leveraging the insights above, we present a simple yet effectivegenetic algorithm that searches for exploitable arches without training, as sketched in Algorithm 1.
Revised: Leveraging the insights above, we present a simple yet effective algorithm that searches for exploitable arches without training, which is a variant of regularized evolution (Real et al., 2019; Mellor et al., 2021).

Original: It starts from a candidate pool A of n arches randomly sampled from a pre-defined arch space; at each iteration, it samples a subset A ′ of m arches from A , randomly mutates the best candidate ( i.e. , with the lowest score), and replaces the oldest arch in A with this newly mutated arch.
Revised: As sketched in Algorithm 1, it starts from a candidate pool A of n arches randomly sampled from a pre-defined arch space; at each iteration, it samples a subset A ′ of m arches from A , randomly mutates the best candidate ( i.e. , with the lowest score), and replaces the oldest arch in A with this newly mutated arch.

Original: In our implementation, the score function is defined as the condition number ofEq. 6; the arch space is defined to be the NATS-Bench search space (Dong &
Revised: In our implementation, the score function is defined as the condition number of

Original:  Yang, 2020), which consists of 5 atomic operators { none , skip connect , conv 1 × 1 , conv 3 × 3 , and avg pooling 3 × 3 }; and the mutation function is defined to be randomly substituting one operator with another. 
Revised: Eq. 6; the arch space is defined to be the NATS-Bench search space (Dong & Yang, 2020), which consists of 5 atomic operators { none , skip connect , conv 1 × 1 , conv 3 × 3 , and avg pooling 3 × 3 }; and the mutation function is defined to be randomly substituting one operator with another. Algorithm

Original:  best
Revised: Input: n – best

Original: // oldest candidate //  updatescore
Revised: // oldest candidate // mutate candidate // update

Original: // update timestamps
Revised: // update timestamps

Original:  E VALUATION
Revised:  E VALUATION

Original: We conduct an empirical evaluation of E VAS on benchmark datasets under various scenarios.
Revised: We conduct an empirical evaluation of E VAS on benchmark datasets under various scenarios.

Original: The experiments are designed to answer the following key questions: ( i ) does it work?
Revised: The experiments are designed to answer the following key questions: ( i ) does it work?

Original: – we evaluate the performance and vulnerability of the arches identified by E VAS ; ( ii ) how does it work?
Revised: – we evaluate the performance and vulnerability of the arches identified by E VAS ; ( ii ) how does it work?

Original: – we explore the dynamics of E VAS search as well as the characteristics of its identified arches; and ( ii ) how does it differ?
Revised: – we explore the dynamics of E VAS search as well as the characteristics of its identified arches; and ( ii ) how does it differ?

Original: – we compare E VAS with conventional backdoors in terms of attack evasiveness, transferability, and robustness.
Revised: – we compare E VAS with conventional backdoors in terms of attack evasiveness, transferability, and robustness.

Original: 4. E XPERIMENTAL S ETTING
Revised: 4. E XPERIMENTAL S ETTING

Original: Datasets.
Revised: Datasets.

Original: In the evaluation, we primarily use three datasets that have been widely used to benchmark
Revised: In the evaluation, we primarily use three datasets that have been widely used to benchmark

Original: NAS methods (Chen et al., 2019; Li et al., 2020; Liu et al., 2019; Pham et al., 2018; Xie et al., 2019): CIFAR10 (Krizhevsky & Hinton, 2009), which consists of 32 × 32 color images drawn from 10 classes; CIFAR100, which is similar to CIFAR10 but includes 100 finer-grained classes; and
Revised: NAS methods (Chen et al., 2019; Li et al., 2020; Liu et al., 2019; Pham et al., 2018; Xie et al., 2019): CIFAR10 (Krizhevsky & Hinton, 2009), which consists of 32 × 32 color images drawn from 10 classes; CIFAR100, which is similar to CIFAR10 but includes 100 finer-grained classes; and

Original:  ImageNet16, which is a subset of the ImageNet dataset (Deng et al., 2009) down-sampled to images of size 16 × 16 in 120 classes.[Figure]
Revised: [Figure] ImageNet16, which is a subset of the ImageNet dataset (Deng et al., 2009) down-sampled to images of size 16 × 16 in 120 classes.

Original: Search space.
Revised: Search space.

Original: We consider the search space defined by NATS-Bench ( Dong et al.
Revised: We consider the search space defined by NATS-Bench ( Dong et al.

Original: (2021)), which consists of 5 operators { none , skip connect , conv 1 × 1 , conv 3 × 3 , and avg pooling 3 × 3 } defined among 4 nodes, implying a search space of 15,625 candidate arches.
Revised: (2021)), which consists of 5 operators { none , skip connect , conv 1 × 1 , conv 3 × 3 , and avg pooling 3 × 3 } defined among 4 nodes, implying a search space of 15,625 candidate arches.

Original: Baselines.
Revised: Baselines.

Original: We compare the arches found by E VAS with ResNet18 (He et al., 2016), a manually designed arch.
Revised: We compare the arches found by E VAS with ResNet18 (He et al., 2016), a manually designed arch.

Original: For completeness, we also include two arches randomly sampled from the NATSBench space, which are illustrated in Figure 3. 
Revised: For completeness, we also include two arches randomly sampled from the NATSBench space, which are illustrated in Figure 3. By default, for each arch α , we assume the adversary trains a model f α

Revised: and then trains the trigger generator g with respect to f α on the same dataset.

Revised: We consider varying settings in which the victim directly uses f α , fine-tunes f α , or only uses α and re-trains it from scratch (details in § 4.4).

Original: Metrics.
Revised: Metrics.

Original: We mainly use two metrics, attack success rate (ASR) and clean data accuracy (ACC).
Revised: We mainly use two metrics, attack success rate (ASR) and clean data accuracy (ACC).

Original: Intuitively, ASR is the target model’s accuracy in classifying trigger inputs to the adversary’s target class during inference, which measures the attack effectiveness, while ACC is the target model’s accuracy in correctly classifying clean inputs, which measures the attack evasiveness.
Revised: Intuitively, ASR is the target model’s accuracy in classifying trigger inputs to the adversary’s target class during inference, which measures the attack effectiveness, while ACC is the target model’s accuracy in correctly classifying clean inputs, which measures the attack evasiveness.

Original: The default parameter setting and the trigger generator configuration are deferred to Appendix § A.
Revised: The default parameter setting and the trigger generator configuration are deferred to Appendix § A.

Original: 4. Q1: D OES E VAS WORK ?
Revised: 4. Q1: D OES E VAS WORK ?

Original: Figure 3 illustrates one sample arch identified by E VAS on the CIFAR10 dataset.
Revised: Figure 3 illustrates one sample arch identified by E VAS on the CIFAR10 dataset.

Original: We use this arch throughout this set of experiments to show that its vulnerability is at the arch level and universal across datasets.
Revised: We use this arch throughout this set of experiments to show that its vulnerability is at the arch level and universal across datasets.

Original: To measure the vulnerability of different arches, we first train each arch using clean data, then train a trigger generator specific to this arch, and finally measure its ASR and ACC.
Revised: To measure the vulnerability of different arches, we first train each arch using clean data, then train a trigger generator specific to this arch, and finally measure its ASR and ACC.

Original: Table 1 reports the results.
Revised: Table 1 reports the results.

Original: We have the following observations.
Revised: We have the following observations.

Original: First, the ASR of E VAS is significantly higher than ResNet18 and the other two random arches.
Revised: First, the ASR of E VAS is significantly higher than ResNet18 and the other two random arches.

Original: For instance, on CIFAR10, E VAS is 21.8%, 28.3%, and 34.5% more effective than ResNet18 and random arches, respectively.
Revised: For instance, on CIFAR10, E VAS is 21.8%, 28.3%, and 34.5% more effective than ResNet18 and random arches, respectively.

Original: Second, E VAS has the highest ASR across all the datasets.
Revised: Second, E VAS has the highest ASR across all the datasets.

Original: Recall that we use the same arch throughout different datasets.
Revised: Recall that we use the same arch throughout different datasets.

Original: This indicates that the attack vulnerability probably resides at the arch level and is agnostic to concrete datasets.
Revised: This indicates that the attack vulnerability probably resides at the arch level and is insensitive to concrete datasets, which corroborates with prior work on NAS: one performant arch found on one dataset often transfers across different datasets (Liu et al., 2019). This may be explained as follows.

Revised: An arch α essentially defines a function family F α , while a trained model f α ( · ; θ ) is an instance in F α , thereby carrying the characteristics of F α ( e.g. , effective to extract important features or exploitable by a trigger generator).

Original: Third, all the arches show higher ASR on simpler datasets such as CIFAR10.
Revised: Third, all the arches show higher ASR on simpler datasets such as CIFAR10.

Original: This may be explained by that more complex datasets ( e.g. , more classes, higher resolution) imply more intricate manifold structures, which may interfere with arch-level backdoors.
Revised: This may be explained by that more complex datasets ( e.g. , more classes, higher resolution) imply more intricate manifold structures, which may interfere with arch-level backdoors.

Original: [Table]
Revised: [Table]

Original: To understand the attack effectiveness of E VAS on individual inputs, we illustrate sample clean inputs and their trigger-embedded variants in Figure 4.
Revised: To understand the attack effectiveness of E VAS on individual inputs, we illustrate sample clean inputs and their trigger-embedded variants in Figure 4.

Original: Further, using GradCam (Selvaraju et al., 2017), we show the model’s interpretation of clean and trigger inputs with respect to their original and target classes.
Revised: Further, using GradCam (Selvaraju et al., 2017), we show the model’s interpretation of clean and trigger inputs with respect to their original and target classes.

Original: Observe that the trigger pattern is specific to each input.
Revised: Observe that the trigger pattern is specific to each input.

Original: Further, even though the two trigger inputs are classified to the same target class, the difference in their heatmaps shows that the model pays attention to distinct features, highlighting the effects of input-aware triggers. 
Revised: Further, even though the two trigger inputs are classified into the same target class, the difference in their heatmaps shows that the model pays attention to distinct features, highlighting the effects of input-aware triggers. [Figure] r

Original:  4. Q2: H OW DOES E VAS WORK ?
Revised: [Figure] 4. Q2: H OW DOES E VAS WORK ?

Original: Next, we explore the dynamics of how E VAS searches for exploitable arches.
Revised: Next, we explore the dynamics of how E VAS searches for exploitable arches.

Original: For simplicity, given the arch identified by E VAS in Figure 3, we consider the set of candidate arches with the operators on c l ea n i npu t [Figure] t r i gge r i npu t
Revised: For simplicity, given the arch identified by E VAS in Figure 3, we consider the set of candidate arches with the operators on the 0-3 ( skip connect ) and 0-1 ( conv 3 × 3) connections replaced by others.

Original: [Figure]

Original: the 0-3 ( skip connect ) and 0-1 ( conv 3 × 3) connections replaced by others.

Original: We measure the ACC and
Revised: We measure the ACC and

Original: ASR of all these candidate arches and illustrate the landscape of their scores in Figure 5.
Revised: ASR of all these candidate arches and illustrate the landscape of their scores in Figure 5.

Original: Observe that the exploitable arch features the lowest score among the surrounding arches, suggesting the existence of feasible mutation paths from random arches to reach exploitable arches following the direction of score descent. 
Revised: Observe that the exploitable arch features the lowest score among the surrounding arches, suggesting the existence of feasible mutation paths from random arches to reach exploitable arches following the direction of score descent. [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Revised: : 44.48 ACC: 91.

Original: [Figure]
Revised: Score: 5. ASR: 29.64 ACC: 91. Score: 19. ASR: 46.76 ACC: 91.

Original: [Table]
Revised: Score: 3. ASR: 50.71 ACC: 92. Score: 20. ASR: 48.66 ACC: 92.

Original: [Figure]
Revised: Score: 15. ASR: 42.76 ACC: 91. Score: 5. ASR: 27.01 ACC: 91.

Original: [Table]
Revised: Score: 23. ASR: 81.51 ACC: 94. Score: 3. ASR: 51.19 ACC: 93.

Original: [Figure]
Revised: Score: 9. ASR: 32.66 ACC: 93. Score: 21. ASR: 45.59 ACC: 92.

Original: [Table]
Revised: Score: 6. ASR: 39.99 ACC: 90. Score: 13. ASR: 56.35 ACC: 92.

Original: [Figure]
Revised: Score: 8. ASR: 43.13 ACC: 92. Score: 32. ASR: 51.55 ACC: 92.

Original: [Table]
Revised: Score: 19. ASR: 60.71 ACC: 91. Score: 8. [Figure]

Revised: 3 connection

Original: [Figure]
Revised: [Figure]

Original: Further, we ask the question: what makes the arches found by E VAS exploitable?
Revised: Further, we ask the question: what makes the arches found by E VAS exploitable?

Original: Observe that the arch in Figure 3 uses the conv 1 × 1 and 3 × 3 operators on a number of connections.
Revised: Observe that the arch in Figure 3 uses the conv 1 × 1 and 3 × 3 operators on a number of connections.

Original: We thus generate arches by enumerating all the possible combinations of conv 1 × 1 and 3 × 3 on these connections and measure their performance, with results summarized in Appendix § A.3. Observe that while all these arches show high ASR, their vulnerability varies greatly from about 50% to 90%.
Revised: We thus generate arches by enumerating all the possible combinations of conv 1 × 1 and 3 × 3 on these connections and measure their performance, with results summarized in Appendix § B. Observe that while all these arches show high ASR, their vulnerability varies greatly from about 50% to 90%.

Original: We hypothesize that specific combinations of conv 1 × 1 and conv 3 × 3 create arch-level “shortcuts” for recognizing trigger patterns.
Revised: We hypothesize that specific combinations of conv 1 × 1 and conv 3 × 3 create arch-level “shortcuts” for recognizing trigger patterns.

Original: We consider exploring the causal relationships between concrete arch characteristics and attack vulnerability as our ongoing work.
Revised: We consider exploring the causal relationships between concrete arch characteristics and attack vulnerability as our ongoing work.

Original: 4. Q3: H OW DOES E VAS DIFFER ?
Revised: 4. Q3: H OW DOES E VAS DIFFER ?

Original: To further understand the difference between E VAS and conventional backdoors, we compare the arches found by E VAS and other arches under various training and defense scenarios.
Revised: To further understand the difference between E VAS and conventional backdoors, we compare the arches found by E VAS and other arches under various training and defense scenarios.

Original: Fine-tuning with clean data.
Revised: Fine-tuning with clean data.

Original: We first consider the scenario in which, with the trigger generator fixed, the target model is fine-tuned using clean data (with concrete setting deferred to Appendix § A).
Revised: We first consider the scenario in which, with the trigger generator fixed, the target model is fine-tuned using clean data (with concrete setting deferred to Appendix § A).

Original: Table 2 shows the results evaluated on CIFAR10 and CIFAR100.
Revised: Table 2 shows the results evaluated on CIFAR10 and CIFAR100.

Original: Observe that fine-tuning has a marginal impact on the ASR of all the arches.
Revised: Observe that fine-tuning has a marginal impact on the ASR of all the arches.

Original: Take Random I as an example, compared with Table 1, its ASR on CIFAR10 drops only by 7.40% after fine-tuning.
Revised: Take Random I as an example, compared with Table 1, its ASR on CIFAR10 drops only by 7.40% after fine-tuning.

Original: This suggests that the effectiveness of fine-tuning to defend against input-aware backdoor attacks may be limited.
Revised: This suggests that the effectiveness of fine-tuning to defend against input-aware backdoor attacks may be limited.

Original: [Table]
Revised: [Table]

Original: Re-training from scratch.
Revised: Re-training from scratch.

Original: Another common scenario is that the victim user re-initializes the target model and re-trains it from scratch using clean data.
Revised: Another common scenario is that the victim user re-initializes the target model and re-trains it from scratch using clean data.

Original: We simulate this scenario as follows.
Revised: We simulate this scenario as follows.

Original: After the trigger generator and target model are trained, we fix the generator, randomly initialize (using different seeds) the model, and train it on the given dataset.
Revised: After the trigger generator and target model are trained, we fix the generator, randomly initialize (using different seeds) the model, and train it on the given dataset.

Original: Table 3 compares different arches under this scenario.
Revised: Table 3 compares different arches under this scenario.

Original: It is observed that E VAS significantly outperforms ResNet18 and random arches in terms of ASR (with comparable ACC).
Revised: It is observed that E VAS significantly outperforms ResNet18 and random arches in terms of ASR (with comparable ACC).

Original: For instance, it is 33.4%, 24.9%, and 19.6% more effective than the other arches respectively.
Revised: For instance, it is 33.4%, 24.9%, and 19.6% more effective than the other arches respectively.

Original: This may be explained by two reasons.
Revised: This may be explained by two reasons.

Original: First, the arch-level backdoors in E VAS are inherently more agnostic to model re-training than the model-level backdoors in other arches.
Revised: First, the arch-level backdoors in E VAS are inherently more agnostic to model re-training than the model-level backdoors in other arches.

Original: Second, in searching for exploitable arches, E VAS explicitly enforces that such vulnerability should be insensitive to model initialization ( cf.
Revised: Second, in searching for exploitable arches, E VAS explicitly enforces that such vulnerability should be insensitive to model initialization ( cf.

Original: Eq. 4).
Revised: Eq. 4).

Original: Further, observe that, as expected, re-training has a larger impact than fine-tuning on the ASR of different arches; however, it is still insufficient to mitigate input-aware backdoor attacks.
Revised: Further, observe that, as expected, re-training has a larger impact than fine-tuning on the ASR of different arches; however, it is still insufficient to mitigate input-aware backdoor attacks.

Original: [Table]
Revised: [Table]

Original: Fine-tuning with poisoning data.
Revised: Fine-tuning with poisoning data.

Original: Further, we explore the setting in which the adversary is able to poison a tiny portion of the fine-tuning data, which assumes a stronger threat model.
Revised: Further, we explore the setting in which the adversary is able to poison a tiny portion of the fine-tuning data, which assumes a stronger threat model.

Original: To simulate this scenario, we apply the trigger generator to generate trigger-embedded inputs and mix them with the clean fine-tuning data.
Revised: To simulate this scenario, we apply the trigger generator to generate trigger-embedded inputs and mix them with the clean fine-tuning data.

Original: Figure 6 illustrates the ASR and ACC of the target model as functions of the fraction of poisoning data in the fine-tuning dataset.
Revised: Figure 6 illustrates the ASR and ACC of the target model as functions of the fraction of poisoning data in the fine-tuning dataset.

Original: Observe that, even with an extremely small poisoning ratio ( e.g. , 0.01%), it can significantly boost the ASR ( e.g. , 100%) while keeping the ACC unaffected.
Revised: Observe that, even with an extremely small poisoning ratio ( e.g. , 0.01%), it can significantly boost the ASR ( e.g. , 100%) while keeping the ACC unaffected.

Original: This indicates that arch-level backdoors can be greatly enhanced by combining with other attack vectors ( e.g. , data poisoning).
Revised: This indicates that arch-level backdoors can be greatly enhanced by combining with other attack vectors ( e.g. , data poisoning).

Original: [Figure]
Revised: [Figure]

Original: Backdoor defenses.
Revised: Backdoor defenses.

Original: Finally, we evaluate E VAS against three categories of defenses, model inspection, input filtering, and model sanitization.
Revised: Finally, we evaluate E VAS against three categories of defenses, model inspection, input filtering, and model sanitization.

Original: Model inspection determines whether a given model f is infected with backdoors.
Revised: Model inspection determines whether a given model f is infected with backdoors.

Original: We use NeuralCleanse (Wang et al., 2019) as a representative defense.
Revised: We use NeuralCleanse (Wang et al., 2019) as a representative defense.

Original: Intuitively, it searches for potential triggers in each class.
Revised: Intuitively, it searches for potential triggers in each class.

Original: If a class is trigger-embedded, the minimum perturbation required to change the predictions of inputs from other classes to this class is abnormally small.
Revised: If a class is trigger-embedded, the minimum perturbation required to change the predictions of inputs from other classes to this class is abnormally small.

Original: It detects anomalies using median absolute deviation (MAD) and all classes with MAD scores larger than 2 are regarded as infected.
Revised: It detects anomalies using median absolute deviation (MAD) and all classes with MAD scores larger than 2 are regarded as infected.

Original: As shown in Table 4, the MAD scores of E VAS ’s target classes on three datasets are all below the threshold.
Revised: As shown in Table 4, the MAD scores of E VAS ’s target classes on three datasets are all below the threshold.

Original: This can be explained by that NeuralCleanse is built upon the universal trigger assumption, which does not hold for E VAS .
Revised: This can be explained by that NeuralCleanse is built upon the universal trigger assumption, which does not hold for E VAS .

Original: [Table]
Revised: [Table]

Original: Input filtering detects at inference time whether an incoming input is embedded with a trigger.
Revised: Input filtering detects at inference time whether an incoming input is embedded with a trigger.

Original: We use STRIP (Gao et al., 2019) as a representative defense in this category.
Revised: We use STRIP (Gao et al., 2019) as a representative defense in this category.

Original: It mixes a given input with a clean input and measures the self-entropy of its prediction.
Revised: It mixes a given input with a clean input and measures the self-entropy of its prediction.

Original: If the input is trigger-embedded, the mixture remains dominated by the trigger and tends to be misclassified, resulting in low self-entropy.
Revised: If the input is trigger-embedded, the mixture remains dominated by the trigger and tends to be misclassified, resulting in low self-entropy.

Original: However, as shown in Table 4, the AUROC scores of STRIP in classifying trigger-embedded inputs by E VAS are all close to random guess ( i.e. , 0.5).
Revised: However, as shown in Table 4, the AUROC scores of STRIP in classifying trigger-embedded inputs by E VAS are all close to random guess ( i.e. , 0.5).

Original: This can also be explained by that E VAS uses input-aware triggers, where each trigger only works for one specific input and has a limited impact on others.
Revised: This can also be explained by that E VAS uses input-aware triggers, where each trigger only works for one specific input and has a limited impact on others.

Original: [Table]
Revised: [Table]

Original: Model sanitization, before using a given model, sanitizes it to mitigate the potential backdoors, yet without explicitly detecting whether the model is tampered.
Revised: Model sanitization, before using a given model, sanitizes it to mitigate the potential backdoors, yet without explicitly detecting whether the model is tampered.

Original: We use Fine-Pruning (Liu et al., 2018) as a representative.
Revised: We use Fine-Pruning (Liu et al., 2018) as a representative.

Original: It uses the property that the backdoor attack typically exploits spare model capacity.
Revised: It uses the property that the backdoor attack typically exploits spare model capacity.

Original: It thus prunes rarely used neurons and then applies fine-tuning to defend against pruning-aware attacks.
Revised: It thus prunes rarely-used neurons and then applies fine-tuning to defend against pruning-aware attacks.

Original: We apply Fine-Pruning on the E VAS and ResNet18 models from Table 1, with results shown in Table 5.
Revised: We apply Fine-Pruning on the E VAS and ResNet18 models from Table 1, with results shown in Table 5.

Original: Observe that Fine-Pruning has a limited impact on the ASR of E VAS (even less than ResNet18).
Revised: Observe that Fine-Pruning has a limited impact on the ASR of E VAS (even less than ResNet18).

Original: This may be explained as follows.
Revised: This may be explained as follows.

Original: The activation patterns of input-aware triggers are different from that of universal triggers, as each trigger may activate a different set of neurons.
Revised: The activation patterns of input-aware triggers are different from that of universal triggers, as each trigger may activate a different set of neurons.

Original: Moreover, the arch-level backdoors in E VAS may not concentrate on individual neurons but span over the whole model structures.
Revised: Moreover, the arch-level backdoors in E VAS may not concentrate on individual neurons but span over the whole model structures.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work studies the feasibility of exploiting NAS as an attack vector to launch previously improbable attacks.
Revised: This work studies the feasibility of exploiting NAS as an attack vector to launch previously improbable attacks.

Original: We present a new backdoor attack that leverages NAS to efficiently find neural network architectures with inherent, exploitable vulnerability.
Revised: We present a new backdoor attack that leverages NAS to efficiently find neural network architectures with inherent, exploitable vulnerability.

Original: Such architecture-level backdoors demonstrate many interesting properties including evasiveness, transferability, and robustness, thereby greatly expanding the design spectrum for the adversary.
Revised: Such architecture-level backdoors demonstrate many interesting properties including evasiveness, transferability, and robustness, thereby greatly expanding the design spectrum for the adversary.

Original: We believe our findings raise concerns about the current practice of NAS in security-sensitive domains and point to potential directions to develop effective mitigation.
Revised: We believe our findings raise concerns about the current practice of NAS in security-sensitive domains and point to potential directions to develop effective mitigation.

Original: A A PPENDIX A. P ARAMETER S ETTING
Revised: A E XPERIMENTAL S ETTING A. P ARAMETER S ETTING

Original: Table 6 summarizes the default parameter setting. [Table]
Revised: Table 6 summarizes the default parameter setting

Original:  A. G ENERATOR ARCHITECTURE
Revised: . A. G ENERATOR ARCHITECTURE

Original: Table 7 lists the architecture of the trigger generator. 
Revised: Table 7 lists the architecture of the trigger generator. [Table] [Table]

Original: A. A DDITIONAL R ESULTS 
Revised: B A DDITIONAL R ESULTS B. NTK OF TARGET MODEL

Revised: Here, we measure the NTK conditional number of the target model f under random initialization using the implementation of (Chen et al., 2021) and its corresponding ASR and ACC. Figure 7 shows their correlation. Observed that the NTK conditional number is negatively correlated with ACC (with

Revised: Kendall’s coefficient τ = − 0 . 385 ) and has a very weak correlation with ASR (with τ = 0 . 100 ), which is consistent with (Chen et al., 2021).

Revised: The difference between Figure 2 and Figure 7 can be explained as follows.

Revised: Figure 2 measures the

Revised: NTK conditional number κ g of the trigger generator g (with respect to the randomly initialized target model f ), which indicates g ’s trainability (or f ’s vulnerability).

Revised: As backdoor attacks embed two functions (one classifying clean inputs and the other classifying trigger inputs) into the same model, there tends to exist a natural trade-off between ASR and ACC.

Revised: Therefore, κ g shows a negative correlation with ASR and a weak positive correlation with ACC.

Revised: Meanwhile, Figure 7 measures the

Revised: [Figure]

Revised: l

Revised: [Figure]

Revised: r o f

Revised: N T K (

Revised: [Figure]

Revised: NTK conditional number κ f of the target model f , which indicates f ’s trainability.

Revised: Therefore, κ f shows a negative correlation with ACC but a very weak correlation with ASR.

Revised: B. ASR-ACC TRADE - OFF

Revised: Figure 8 shows the correlation between the ASR and ACC of sampled arches (with Kendall’s coefficient τ = − 0 . 390 ).

Revised: Intuitively, as backdoor attacks embed two functions (one classifying clean inputs and the other classifying trigger inputs) into the same model, there tends to exist a natural trade-off between ASR and ACC.

Revised: This trade-off also implies that it is feasible to properly optimize

Revised: ASR only to find performant but vulnerable arches.

Revised: [Figure]

Revised: B. I NTERPRETABILITY VERSUS VULNERABILITY

Revised: To understand the possible correlation between the attack vulnerability of an arch α and its interpretability, we compare the interpretation of each model f

Revised: α regarding 100 clean inputs using

Revised: GradCam (Selvaraju et al., 2017).

Revised: Figure 9 illustrates sample inputs and their interpretation by different models.

Revised: Further, to quantitatively measure the similarity of interpretation, we use the intersection-over-union (IoU) score, which is widely used in object detection to compare model predictions with ground-truth bounding boxes.

Revised: Formally, the IoU score of a binary-valued heatmap m with respect to another map m ′ is defined as their Jaccard similarity:

Revised: [Equation]

Revised: where O ( m ) denotes the set of non-zero elements in m .

Revised: In our case, as the values of heatmaps are floating numbers, we first apply thresholding to binarize the values.

Revised: Figure 10 shows the average

Revised: IoU score of each arch with respect to another.

Revised: Observe that ( i ) the arches generated by NAS (E VAS ,

Revised: Random I, and Random II) have more similar interpretability among themselves than manually

Original: [Table]
Revised: [Figure] I

Revised: Random II designed arches (ResNet-18); ( ii ) the arch with high vulnerability (E VAS ) is not significantly different from the arch with low vulnerability (Random I, II) in terms of interpretability.

Original: [Table]
Revised: [Figure]

Revised: B. E VAS SEARCH DYNAMICS

Original: [Table]
Revised: It is challenging to directly illustrate the search dynamics of E VAS as a genetic search algorithm due to its random nature. Here, we plot the lower envelope of the score during four sample runs of E VAS ; to measure the quality of arches generated during the process, we also sample arches as the best score changes and measure their ASR and ACC. It can be observed that the score in Eq. 6 effectively guides the search for performant yet vulnerable arches.

Original: [Table]
Revised: [Figure] [Figure] B e s t C ond iti on a l N u m b e r

Revised: o f

Revised: N T K (l o g )

Original: [Table]
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: B. A BLATION OF ATTACK EVASIVENESS

Original: [Table]
Revised: The evasiveness of E VAS may be accounted by ( i ) input-dependent triggers (Nguyen & Tran, 2020) and ( ii ) arch-level vulnerability. Here, we explore the contribution of input-dependent triggers to the attack evasiveness. We train the trigger generator with respect to different arches (E VAS , ResNet18, random arches) and run NeuralCleanse and STRIP to detect the attacks, with results summarized in Table 8. Observed that while the concrete measures vary, all the attacks have MAD scores below the threshold and AUROC scores close to random guess, indicating that the input-dependent triggers mainly account for the attack evasiveness with respect to NeuralCleanse and STRIP.

Original: [Table]
Revised: [Table]

Revised: I MPORTANCE OF C ONV 1 × 1 AND C ONV 3 ×

Revised: We generate neighboring arches by enumerating all possible combinations of conv 1 × 1 and conv 3 ×on the connections of the arch identified by E VAS (“ |{ 0 } ∼ 0 | + |{ 1 } ∼ 0 |{ 2 } ∼ 1 | + | skip_connect ∼|{ 3 } ∼ 1 |{ 4 } ∼ 2 | ”).

Revised: The ASR and ACC of these arches are summarized in Table

Original: [Table] 
Revised: [Table] [Table] [Table] Conv 3x

Revised: Conv 1x

Revised: Conv 1x

Original: [Table]
Revised: Conv 1x 56. Conv 3x 63. Conv 3x Conv 1x 51.

Revised: Conv 3x 69.

Revised: Conv 3x

Revised: Conv 1x

Revised: Conv 1x 55.

Revised: Conv 3x 64.

Revised: Conv 3x

Revised: Conv 1x 55.

Revised: Conv 3x 55.

Revised: Conv 3x

Revised: Conv 1x

Revised: Conv 1x

Revised: Conv 1x

Revised: Conv 1x 53.

Revised: Conv 3x 87.

Revised: Conv 3x

Revised: Conv 1x 58.

Revised: Conv 3x 48.

Revised: Conv 3x

Revised: Conv 1x

Revised: Conv 1x 63.

Revised: Conv 3x 57.

Revised: Conv 3x

Revised: Conv 1x 62.

Revised: Conv 3x 57.

Revised: Conv 3x

Revised: Conv 1x

Revised: Conv 1x

Revised: Conv 1x 54.

Revised: Conv 3x 59.

Revised: Conv 3x

Revised: Conv 1x 55.

Revised: Conv 3x 59.

Revised: Conv 3x

Revised: Conv 1x

Revised: Conv 1x 66.

Revised: Conv 3x 79.

Revised: Conv 3x

Revised: Conv 1x 54.

Revised: Conv 3x 52.