Original: P OST -T RAINING D ETECTION OF B ACKDOOR A TTACKS FOR T WO -C LASS AND M ULTI -A TTACK S CENARIOS
Revised: P OST -T RAINING D ETECTION OF B ACKDOOR A TTACKS FOR T WO -C LASS AND M ULTI -A TTACK S CENARIOS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Backdoor attacks (BAs) are an emerging threat to deep neural network classiﬁers.
Revised: Backdoor attacks (BAs) are an emerging threat to deep neural network classiﬁers.

Original: A victim classiﬁer will predict to an attacker-desired target class whenever a test sample is embedded with the same backdoor pattern (BP) that was used to poison the classiﬁer’s training set.
Revised: A victim classiﬁer will predict to an attacker-desired target class whenever a test sample is embedded with the same backdoor pattern (BP) that was used to poison the classiﬁer’s training set.

Original: Detecting whether a classiﬁer is backdoor attacked is not easy in practice, especially when the defender is, e.g., a downstream user without access to the classiﬁer’s training set.
Revised: Detecting whether a classiﬁer is backdoor attacked is not easy in practice, especially when the defender is, e.g., a downstream user without access to the classiﬁer’s training set.

Original: This challenge is addressed here by a reverse-engineering defense (RED), which has been shown to yield state-of-theart performance in several domains.
Revised: This challenge is addressed here by a reverse-engineering defense (RED), which has been shown to yield state-of-theart performance in several domains.

Original: However, existing REDs are not applicable when there are only two classes or when multiple attacks are present.
Revised: However, existing REDs are not applicable when there are only two classes or when multiple attacks are present.

Original: These scenarios are ﬁrst studied in the current paper, under the practical constraints that the defender neither has access to the classiﬁer’s training set nor to supervision from clean reference classiﬁers trained for the same domain.
Revised: These scenarios are ﬁrst studied in the current paper, under the practical constraints that the defender neither has access to the classiﬁer’s training set nor to supervision from clean reference classiﬁers trained for the same domain.

Original: We propose a detection framework based on BP reverse-engineering and a novel expected transferability (ET) statistic.
Revised: We propose a detection framework based on BP reverse-engineering and a novel expected transferability (ET) statistic.

Original: We show that our ET statistic is effective using the same detection threshold , irrespective of the classiﬁcation domain, the attack conﬁguration, and the BP reverse-engineering algorithm that is used.
Revised: We show that our ET statistic is effective using the same detection threshold , irrespective of the classiﬁcation domain, the attack conﬁguration, and the BP reverse-engineering algorithm that is used.

Original: The excellent performance of our method is demonstrated on six benchmark datasets.
Revised: The excellent performance of our method is demonstrated on six benchmark datasets.

Original: Notably, our detection framework is also applicable to multi-class scenarios with multiple attacks.
Revised: Notably, our detection framework is also applicable to multi-class scenarios with multiple attacks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite the success of deep neural network (DNN) classiﬁers in many research areas, their vulnerabilities have been recently exposed Xu et al.
Revised: Despite the success of deep neural network (DNN) classiﬁers in many research areas, their vulnerabilities have been recently exposed Xu et al.

Original: One emerging threat to DNN classiﬁers is a backdoor attack (BA) Li et al.
Revised: One emerging threat to DNN classiﬁers is a backdoor attack (BA) Li et al.

Original: Here, a classiﬁer will predict to the attacker’s target class when a test sample is embedded with the same backdoor pattern (BP) that was used to poison the classiﬁer’s training set.
Revised: Here, a classiﬁer will predict to the attacker’s target class when a test sample is embedded with the same backdoor pattern (BP) that was used to poison the classiﬁer’s training set.

Original: On the other hand, the classiﬁer’s clean test set accuracy (i.e., on samples without an embedded BP) is largely uncompromised Gu et al.
Revised: On the other hand, the classiﬁer’s clean test set accuracy (i.e., on samples without an embedded BP) is largely uncompromised Gu et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2018b), which makes the attack difﬁcult to detect.
Revised: (2018b), which makes the attack difﬁcult to detect.

Original: Early BA defenses aim to cleanse the possibly poisoned training set of the victim classiﬁer Tran et al.
Revised: Early BA defenses aim to cleanse the possibly poisoned training set of the victim classiﬁer Tran et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: But deployment of these defenses is not feasible when the defender is the user/consumer of the classiﬁer, without access to its training set or to any prior knowledge of the backdoor pattern Wang et al.
Revised: But deployment of these defenses is not feasible when the defender is the user/consumer of the classiﬁer, without access to its training set or to any prior knowledge of the backdoor pattern Wang et al.

Original: A major defense approach suitable for this scenario is a reverse-engineering defense (RED).
Revised: A major defense approach suitable for this scenario is a reverse-engineering defense (RED).

Original: In general, a RED treats each class as a putative BA target class and trial-reverse-engineers a BP.
Revised: In general, a RED treats each class as a putative BA target class and trial-reverse-engineers a BP.

Original: Then, detection statistics are derived from the estimated pattern for each putative target class.
Revised: Then, detection statistics are derived from the estimated pattern for each putative target class.

Original: If there is a BA, the pattern estimated for the true BA target class will likely be correlated with the true BP used by the attacker, such that the associated statistics will likely be anomalous referenced against the statistics for the other (non-BA) classes Wang et al.
Revised: If there is a BA, the pattern estimated for the true BA target class will likely be correlated with the true BP used by the attacker, such that the associated statistics will likely be anomalous referenced against the statistics for the other (non-BA) classes Wang et al.

Original: (2019); Xiang et al.
Revised: (2019); Xiang et al.

Original: Notably, RED-based anomaly detection does not require supervision from clean classiﬁers trained for the same domain like, e.g., Kolouri et al.
Revised: Notably, RED-based anomaly detection does not require supervision from clean classiﬁers trained for the same domain like, e.g., Kolouri et al.

Original: Although existing REDs have achieved leading performance in many practical detection tasks, a fundamental limitation still remains.
Revised: Although existing REDs have achieved leading performance in many practical detection tasks, a fundamental limitation still remains.

Original: RED-based anomaly detection typically requires estimating a null distribution used to assess anomalies, with the number of detection statistics (samples) used to estimate this null a function of the number of classes K .
Revised: RED-based anomaly detection typically requires estimating a null distribution used to assess anomalies, with the number of detection statistics (samples) used to estimate this null a function of the number of classes K .

Original: Wang et al.
Revised: Wang et al.

Original: (2019) uses O ( K ) statistics, and Xiang et al.
Revised: (2019) uses O ( K ) statistics, and Xiang et al.

Original: (2020) uses even more statistics – O ( K 2 ) .
Revised: (2020) uses even more statistics – O ( K 2 ) .

Original: These methods are not applicable for domains with K = 2 , e.g. sentiment classiﬁcation Gao et al.
Revised: These methods are not applicable for domains with K = 2 , e.g. sentiment classiﬁcation Gao et al.

Original: (2019b), disease diagnosis Li et al.
Revised: (2019b), disease diagnosis Li et al.

Original: (2014), etc., because there are insufﬁcient statistics for estimating the null.
Revised: (2014), etc., because there are insufﬁcient statistics for estimating the null.

Original: More generally, their accuracy is affected when the number of classes is small ( K > 2 , but small).
Revised: More generally, their accuracy is affected when the number of classes is small ( K > 2 , but small).

Original: In this paper, we focus on BA detection for the two-class (and possibly multi-attack ) scenario, under two practical constraints: a) the defender has no access to the training set of the victim classiﬁer (or to any BP used by the attacker); b) supervision from clean classiﬁers trained for the same domain is not available .
Revised: In this paper, we focus on BA detection for the two-class (and possibly multi-attack ) scenario, under two practical constraints: a) the defender has no access to the training set of the victim classiﬁer (or to any BP used by the attacker); b) supervision from clean classiﬁers trained for the same domain is not available .

Original: This scenario is clearly more challenging than the one considered by existing REDs, for which there is at most one BA target class and a sufﬁcient number of non-target classes to learn an accurate null.
Revised: This scenario is clearly more challenging than the one considered by existing REDs, for which there is at most one BA target class and a sufﬁcient number of non-target classes to learn an accurate null.

Original: As the ﬁrst to address this difﬁcult problem, our main contributions are as follows:
Revised: As the ﬁrst to address this difﬁcult problem, our main contributions are as follows:

Original: • We propose a detection framework that involves BP reverse-engineering in order to address constraint a) above.
Revised: • We propose a detection framework that involves BP reverse-engineering in order to address constraint a) above.

Original: However, instead of performing anomaly detection as existing REDs do, we process each class independently using a novel detection statistic dubbed expected transferability (ET).
Revised: However, instead of performing anomaly detection as existing REDs do, we process each class independently using a novel detection statistic dubbed expected transferability (ET).

Original: This allows our method to be applicable to the two-class, multi-attack scenario (as well as to the multi-class and (possibly) multi-attack scenario).
Revised: This allows our method to be applicable to the two-class, multi-attack scenario (as well as to the multi-class and (possibly) multi-attack scenario).

Original: • We show that for ET there is a large range of effective choices for the detection threshold, which commonly contains a particular threshold value effective for detecting BAs irrespective of the classiﬁcation domain or particulars of BA.
Revised: • We show that for ET there is a large range of effective choices for the detection threshold, which commonly contains a particular threshold value effective for detecting BAs irrespective of the classiﬁcation domain or particulars of BA.

Original: This common threshold is mathematically derived and is based on properties for general classiﬁcation tasks that have been veriﬁed empirically by many existing works; thus, constraint b) that no domain-speciﬁc supervision is needed is well-addressed.
Revised: This common threshold is mathematically derived and is based on properties for general classiﬁcation tasks that have been veriﬁed empirically by many existing works; thus, constraint b) that no domain-speciﬁc supervision is needed is well-addressed.

Original: • Our ET statistic is obtained by BP reverse-engineering and does not strongly depend on the type of BP, or the particular RED objective function and optimization technique used to reverse-engineer putative BPs.
Revised: • Our ET statistic is obtained by BP reverse-engineering and does not strongly depend on the type of BP, or the particular RED objective function and optimization technique used to reverse-engineer putative BPs.

Original: Thus, our detection framework can incorporate existing reverse-engineering techniques and potentially their future advances.
Revised: Thus, our detection framework can incorporate existing reverse-engineering techniques and potentially their future advances.

Original: • We show the effectiveness of our detection framework on six popular benchmark image datasets.
Revised: • We show the effectiveness of our detection framework on six popular benchmark image datasets.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Backdoor Attack (BA).
Revised: Backdoor Attack (BA).

Original: For a classiﬁcation task with sample space X and label space C , a BA aims to have the victim classiﬁer f : X → C predict to the attacker’s target class t ∈ C whenever a test sample x ∈ X is embedded with the backdoor pattern (BP) Gu et al.
Revised: For a classiﬁcation task with sample space X and label space C , a BA aims to have the victim classiﬁer f : X → C predict to the attacker’s target class t ∈ C whenever a test sample x ∈ X is embedded with the backdoor pattern (BP) Gu et al.

Original: BAs were initially proposed for images, but have also been proposed for other classiﬁcation domains.
Revised: BAs were initially proposed for images, but have also been proposed for other classiﬁcation domains.

Original: While our experimentation will focus on images, our detector is applicable to other domains as well.
Revised: While our experimentation will focus on images, our detector is applicable to other domains as well.

Original: Existing BPs for images are mainly of two types: 1) an additive perturbation v embedded by
Revised: Existing BPs for images are mainly of two types: 1) an additive perturbation v embedded by

Original: [Equation]
Revised: [Equation]

Original: where || v || 2 is small (for imperceptibility) and [ · ] c is a clipping function Chen et al.
Revised: where || v || 2 is small (for imperceptibility) and [ · ] c is a clipping function Chen et al.

Original: ; Zhong et al.
Revised: ; Zhong et al.

Original: Xiang et al.
Revised: Xiang et al.

Original: (2020); or 2) a patch u inserted using an image-wide binary mask m via
Revised: (2020); or 2) a patch u inserted using an image-wide binary mask m via

Original: [Equation]
Revised: [Equation]

Original: where || m || 1 is small for imperceptibility) and (cid:12) represents element-wise multiplication Gu et al.
Revised: where || m || 1 is small for imperceptibility) and (cid:12) represents element-wise multiplication Gu et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: Typically 1 , BAs are launched by inserting into the training set a small set of samples labeled to the target class and embedded with the same BP that will be used during testing.
Revised: Typically 1 , BAs are launched by inserting into the training set a small set of samples labeled to the target class and embedded with the same BP that will be used during testing.

Original: Such data poisoning can be achieved e.g. when DNN training is outsourced to parties that are possibly malicious Gu et al.
Revised: Such data poisoning can be achieved e.g. when DNN training is outsourced to parties that are possibly malicious Gu et al.

Original: BAs are also stealthy since they do not degrade the classiﬁer’s accuracy on clean test samples; hence they are not detectable based on validation set accuracy.
Revised: BAs are also stealthy since they do not degrade the classiﬁer’s accuracy on clean test samples; hence they are not detectable based on validation set accuracy.

Original: Backdoor Defense.
Revised: Backdoor Defense.

Original: Early BA defenses aim to separate backdoor training samples from clean ones by inspecting the training set Tran et al.
Revised: Early BA defenses aim to separate backdoor training samples from clean ones by inspecting the training set Tran et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: Du et al.
Revised: Du et al.

Original: However, their deployment is not feasible in many practical scenarios where the defender is a downstream user who has no access to the training process.
Revised: However, their deployment is not feasible in many practical scenarios where the defender is a downstream user who has no access to the training process.

Original: A family of pruning-based method “removes” the backdoor mapping by inspecting neuron activations (and removing some neurons)
Revised: A family of pruning-based method “removes” the backdoor mapping by inspecting neuron activations (and removing some neurons)

Original: Liu et al.
Revised: Liu et al.

Original: (2018a)
Revised: (2018a)

Original: ; Li et al.
Revised: ; Li et al.

Original: These methods cause non-negligible degradation to classiﬁcation accuracy, they may remove neurons even when there is no backdoor, and moreover they do not make explicit detection inferences.
Revised: These methods cause non-negligible degradation to classiﬁcation accuracy, they may remove neurons even when there is no backdoor, and moreover they do not make explicit detection inferences.

Original: Kolouri et al.
Revised: Kolouri et al.

Original: (2020) and Xu et al.
Revised: (2020) and Xu et al.

Original: (2021) train a binary classiﬁer to classify models as “with” and “without” BAs; but such training requires clean classiﬁers from the same domain or a signiﬁcant number of labeled samples and heavy computation to train these clean classiﬁers.
Revised: (2021) train a binary classiﬁer to classify models as “with” and “without” BAs; but such training requires clean classiﬁers from the same domain or a signiﬁcant number of labeled samples and heavy computation to train these clean classiﬁers.

Original: Chou et al.
Revised: Chou et al.

Original: and Gao et al. (2019a) can detect triggering of a backdoor by an observed test sample.
Revised: and Gao et al. (2019a) can detect triggering of a backdoor by an observed test sample.

Original: Unlike these methods, the existing RED-based detector (introduced next) reliably detects backdoors without the need to observe any test samples.
Revised: Unlike these methods, the existing RED-based detector (introduced next) reliably detects backdoors without the need to observe any test samples.

Original: Reverse-Engineering Defense (RED).
Revised: Reverse-Engineering Defense (RED).

Original: RED is a family of BA defenses that do not need access to the classiﬁer’s training set, nor to any prior knowledge of the BP that may be used in an attack.
Revised: RED is a family of BA defenses that do not need access to the classiﬁer’s training set, nor to any prior knowledge of the BP that may be used in an attack.

Original: Without knowing whether the classiﬁer has been backdoor-attacked, a RED trial-reverse-engineers a BP for each putative BA target class (or possibly for each (source, target) BA class pair) using a small, clean dataset independently collected by the defender.
Revised: Without knowing whether the classiﬁer has been backdoor-attacked, a RED trial-reverse-engineers a BP for each putative BA target class (or possibly for each (source, target) BA class pair) using a small, clean dataset independently collected by the defender.

Original: Then, for each class, a detection statistic is derived from the estimated pattern and used to infer if the class is a BA target class or not.
Revised: Then, for each class, a detection statistic is derived from the estimated pattern and used to infer if the class is a BA target class or not.

Original: For example, to detect BAs with an additive perturbation BP (Eq.
Revised: For example, to detect BAs with an additive perturbation BP (Eq.

Original: (1)), Xiang et al.
Revised: (1)), Xiang et al.

Original: (2020) ﬁnds, for each putative target class, the perturbation with the minimum l 2 norm that induces a large misclassiﬁcation fraction to this class when added to images from another class (the source class).
Revised: (2020) ﬁnds, for each putative target class, the perturbation with the minimum l 2 norm that induces a large misclassiﬁcation fraction to this class when added to images from another class (the source class).

Original: Since BPs embedded by Eq.
Revised: Since BPs embedded by Eq.

Original: (1) usually have a very small l 2 norm for imperceptibility, the pattern estimated for the true BA target class (if a BA exists) will likely have a much smaller l 2 norm than for non-target classes.
Revised: (1) usually have a very small l 2 norm for imperceptibility, the pattern estimated for the true BA target class (if a BA exists) will likely have a much smaller l 2 norm than for non-target classes.

Original: Thus, unsupervised anomaly detection based on these perturbation sizes is performed – when there is one BA target class and a sufﬁcient number of non-target classes, the statistic for the BA target class will likely be detected as an anomaly compared with the others.
Revised: Thus, unsupervised anomaly detection based on these perturbation sizes is performed – when there is one BA target class and a sufﬁcient number of non-target classes, the statistic for the BA target class will likely be detected as an anomaly compared with the others.

Revised: When RED detects an attack, the backdoor pattern estimated for the detected target class can then be used to tune the classiﬁer to “unlearn” the backdoor mapping Wang et al.

Original: Except the RED above proposed by Xiang et al.
Revised: Except the RED above proposed by Xiang et al.

Original: (2020), existing REDs also include Neural Cleanse (NC) proposed by Wang et al.
Revised: (2020), existing REDs also include Neural Cleanse (NC) proposed by Wang et al.

Original: which reverse-engineers BPs embedded by Eq.
Revised: which reverse-engineers BPs embedded by Eq.

Original: (2) and uses the l 1 norm of the estimated mask as the detection statistic.
Revised: (2) and uses the l 1 norm of the estimated mask as the detection statistic.

Original: Guo et al.
Revised: Guo et al.

Original: (2019) adds constraints to
Revised: (2019) adds constraints to

Original: NC’s BP reverse-engineering problem by considering various properties of BPs.
Revised: NC’s BP reverse-engineering problem by considering various properties of BPs.

Revised: Liu et al.

Revised: (2019) proposed a novel objective function for BP reverse-engineering leveraging the abnormal internal neuron activations caused by the backdoor mapping.

Original: Chen et al.
Revised: Chen et al.

Original: (2019) constructs clean images for detection using model inversion.
Revised: (2019) constructs clean images for detection using model inversion.

Original: Dong et al.
Revised: Dong et al.

Original: (2021) queries the classiﬁer for BP reverse-engineering to address the “black-box” scenario.
Revised: (2021) queries the classiﬁer for BP reverse-engineering to address the “black-box” scenario.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) proposes a detection statistic based on the similarity between universal and sample-wise BP estimation.
Revised: (2020) proposes a detection statistic based on the similarity between universal and sample-wise BP estimation.

Original: Limitation of Existing REDs The anomaly detection of REDs heavily relies on the assumption that there is a relatively large number of non-target classes, thus providing a sufﬁcient number of statistics to inform estimation of a null distribution.
Revised: Limitation of Existing REDs The anomaly detection of REDs heavily relies on the assumption that there is a relatively large number of non-target classes, thus providing a sufﬁcient number of statistics to inform estimation of a null distribution.

Original: But for domains with only two classes, e.g. sentiment classiﬁcation Gao et al.
Revised: But for domains with only two classes, e.g. sentiment classiﬁcation Gao et al.

Original: (2019b), disease diagnosis Li et al.
Revised: (2019b), disease diagnosis Li et al.

Original: (2014), etc., this assumption does not hold.
Revised: (2014), etc., this assumption does not hold.

Original: For example, Xiang et al.
Revised: For example, Xiang et al.

Original: (2020) and Wang et al.
Revised: (2020) and Wang et al.

Original: (2019) exploit O ( K 2 ) and O ( K ) statistics in estimating a null respectively, where K is the number of classes.
Revised: (2019) exploit O ( K 2 ) and O ( K ) statistics in estimating a null respectively, where K is the number of classes.

Original: Both of these methods are unsuitable for 2-class problems ( K = 2 ).
Revised: Both of these methods are unsuitable for 2-class problems ( K = 2 ).

Original:  M ETHOD
Revised:  M ETHOD

Original: Our main goals are detecting whether a given classiﬁer is backdoor attacked or not and, if so, ﬁnding out all the BA target classes.
Revised: Our main goals are detecting whether a given classiﬁer is backdoor attacked or not and, if so, ﬁnding out all the BA target classes.

Original: In particular, we focus on the following practical “post-training” detection scenario: S1 )
Revised: In particular, we focus on the following practical “post-training” detection scenario: S1 )

Original: The defender has no access to the training set of the classiﬁer to be inspected, nor any prior knowledge of the true BP used by an attacker.
Revised: The defender has no access to the training set of the classiﬁer to be inspected, nor any prior knowledge of the true BP used by an attacker.

Original: S2 )
Revised: S2 )

Original: There are no clean classiﬁers trained for the same domain for reference; and the defender is not capable of training such clean classiﬁers.
Revised: There are no clean classiﬁers trained for the same domain for reference; and the defender is not capable of training such clean classiﬁers.

Original: S3 )
Revised: S3 )

Original: The classiﬁcation domain has two classes that can both be BA targets.
Revised: The classiﬁcation domain has two classes that can both be BA targets.

Original: While we focus on two-class scenarios in this section, our proposed method is more generally applicable to multi-class scenarios, with arbitrary number of attacks (see experiments in Sec. 4.4).
Revised: While we focus on two-class scenarios in this section, our proposed method is more generally applicable to multi-class scenarios, with arbitrary number of attacks (see experiments in Sec. 4.4).

Original: To address S1 , our detection framework involves BP reverse-engineering using a small, clean dataset independently collected by the defender, like existing REDs.
Revised: To address S1 , our detection framework involves BP reverse-engineering using a small, clean dataset independently collected by the defender, like existing REDs.

Original: To address S3 , unlike existing REDs that perform anomaly detection involving statistics from all classes, we inspect each class independently using a novel detection statistic called expected transferability (ET), which can be empirically estimated for each class independently.
Revised: To address S3 , unlike existing REDs that perform anomaly detection involving statistics from all classes, we inspect each class independently using a novel detection statistic called expected transferability (ET), which can be empirically estimated for each class independently.

Original: To address S2 , we show that ET possesses a theoreticallygrounded detection threshold value for distinguishing BA target classes from non-target classes, one which depends neither on the domain nor on the attack conﬁguration .
Revised: To address S2 , we show that ET possesses a theoreticallygrounded detection threshold value for distinguishing BA target classes from non-target classes, one which depends neither on the domain nor on the attack conﬁguration .

Original: This is very different from existing REDs, for which a suitable detection threshold for their proposed statistics may be both domain and attack-dependent.
Revised: This is very different from existing REDs, for which a suitable detection threshold for their proposed statistics may be both domain and attack-dependent.

Original: For example, the range of the l 1 norm of the estimated mask used by
Revised: For example, the range of the l 1 norm of the estimated mask used by

Original: Wang et al.
Revised: Wang et al.

Original: (2019) depends on the image size; the range of the cosine similarity statistic used by
Revised: (2019) depends on the image size; the range of the cosine similarity statistic used by

Original: Wang et al.
Revised: Wang et al.

Original: (2020) depends on the architecture of the classiﬁer (see Apdx.
Revised: (2020) depends on the architecture of the classiﬁer (see Apdx.

Original: D.7 for more details).
Revised: D.7 for more details).

Original: The practical import here is that the detection threshold is a hyperparameter , but setting this threshold in a supervised fashion (e.g. to achieve a speciﬁed false positive rate on a group of clean classiﬁers) is generally infeasible due to S2 .
Revised: The practical import here is that the detection threshold is a hyperparameter , but setting this threshold in a supervised fashion (e.g. to achieve a speciﬁed false positive rate on a group of clean classiﬁers) is generally infeasible due to S2 .

Original: Use of ET thus obviates the need for such hyperparameter setting.
Revised: Use of ET thus obviates the need for such hyperparameter setting.

Original: In the following, we introduce the deﬁnition of ET and related concepts in Sec. 3.1.
Revised: In the following, we introduce the deﬁnition of ET and related concepts in Sec. 3.1.

Original: In Sec. 3.2, we derive the constant threshold on ET for BA detection.
Revised: In Sec. 3.2, we derive the constant threshold on ET for BA detection.

Original: Finally, our detection procedure, which consists of an ET estimation step and an inference step, is summarized in Alg. 1 in Sec. 3.3.
Revised: Finally, our detection procedure, which consists of an ET estimation step and an inference step, is summarized in Alg. 1 in Sec. 3.3.

Original: Notes: Our detection framework is effective for various embedding mechanisms (as will be shown experimentally).
Revised: Notes: Our detection framework is effective for various embedding mechanisms (as will be shown experimentally).

Original: Solely for clarity, in this section, we focus on BAs with additive perturbation BPs (Eq.
Revised: Solely for clarity, in this section, we focus on BAs with additive perturbation BPs (Eq.

Original: (1)), while similar derivation/analysis for BPs embedded by Eq.
Revised: (1)), while similar derivation/analysis for BPs embedded by Eq.

Original: (2) is deferred to Apdx.
Revised: (2) is deferred to Apdx.

Original: C.
Revised: C.

Original: 3. E XPECTED T RANSFERABILITY (ET)
Revised: 3. E XPECTED T RANSFERABILITY (ET)

Original: Consider a classiﬁer f : X → C to be inspected with category space C
Revised: Consider a classiﬁer f : X → C to be inspected with category space C

Original: = { 0 , 1 } and continuous sample distribution P i on X for class i ∈ C .
Revised: = { 0 , 1 } and continuous sample distribution P i on X for class i ∈ C .

Original: For any x ∈ X from any class, the optimal solution to minimize v || v ||subject to f ( x + v ) (cid:54) =
Revised: For any x ∈ X from any class, the optimal solution to minimize v || v ||subject to f ( x + v ) (cid:54) =

Original: f ( x ) (3) is deﬁned as v ∗ ( x ) .
Revised: f ( x ) (3) is deﬁned as v ∗ ( x ) .

Original: In practice, (3) can be viewed as a typical BP reverse-engineering problem and solved using methods in existing REDs like Xiang et al.
Revised: In practice, (3) can be viewed as a typical BP reverse-engineering problem and solved using methods in existing REDs like Xiang et al.

Original: It can also be practically solved by creating an adversarial example for x using methods in, e.g., M.-Dezfooli et al.
Revised: It can also be practically solved by creating an adversarial example for x using methods in, e.g., M.-Dezfooli et al.

Original: ; Carlini &
Revised: ; Carlini &

Original: Wagner (2017).
Revised: Wagner (2017).

Original: We present the following deﬁnition for the set of practical solutions to (3).
Revised: We present the following deﬁnition for the set of practical solutions to (3).

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: ( (cid:15) -solution set)
Revised: ( (cid:15) -solution set)

Original: For any sample x from any class, regardless of the method being used, the (cid:15) -solution set to problem (3), is deﬁned by
Revised: For any sample x from any class, regardless of the method being used, the (cid:15) -solution set to problem (3), is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) > 0 is the “quality gap” of practical solutions, which is usually small for existing methods.
Revised: where (cid:15) > 0 is the “quality gap” of practical solutions, which is usually small for existing methods.

Original: A practical solution to (3) for sample x may or may not cause a misclassiﬁcation when embedded in another sample y from the same class.
Revised: A practical solution to (3) for sample x may or may not cause a misclassiﬁcation when embedded in another sample y from the same class.

Original: In the following, we ﬁrst present the deﬁnition regarding such a “transferability” property.
Revised: In the following, we ﬁrst present the deﬁnition regarding such a “transferability” property.

Original: Then, we deﬁne the ET statistic for any class i ∈ C .
Revised: Then, we deﬁne the ET statistic for any class i ∈ C .

Original: Deﬁnition 3.2.
Revised: Deﬁnition 3.2.

Original: (Transferable set)
Revised: (Transferable set)

Original: The transferable set for any sample x and (cid:15) > 0 is deﬁned by
Revised: The transferable set for any sample x and (cid:15) > 0 is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 3.3.
Revised: Deﬁnition 3.3.

Original: (ET statistic)
Revised: (ET statistic)

Original: For any class i ∈ C
Revised: For any class i ∈ C

Original: = { 0 , 1 } and (cid:15) > 0 , considering i.i.d.
Revised: = { 0 , 1 } and (cid:15) > 0 , considering i.i.d.

Original: random samples X , Y ∼ P i , the ET statistic for class i is deﬁned by ET i,(cid:15) (cid:44) E X ∼ P
Revised: random samples X , Y ∼ P i , the ET statistic for class i is deﬁned by ET i,(cid:15) (cid:44) E X ∼ P

Original: i (cid:2) P( Y ∈ T (cid:15) ( X ) | X ) (cid:3) .
Revised: i (cid:2) P( Y ∈ T (cid:15) ( X ) | X ) (cid:3) .

Original: 3. U SING ET FOR BA D ETECTION
Revised: 3. U SING ET FOR BA D ETECTION

Original: Here, we show that for any i ∈ C
Revised: Here, we show that for any i ∈ C

Original: = { 0 , 1 } and small (cid:15) , we will likely have ET i,(cid:15) > 12 when class (1 − i ) is a BA target class; and ET i,(cid:15)
Revised: = { 0 , 1 } and small (cid:15) , we will likely have ET i,(cid:15) > 12 when class (1 − i ) is a BA target class; and ET i,(cid:15)

Original: ≤ 12 otherwise.
Revised: ≤ 12 otherwise.

Original: Note that this constant threshold does not rely on any speciﬁc data domain, classiﬁer architecture, or attack conﬁguration; even for BPs embedded by Eq.
Revised: Note that this constant threshold does not rely on any speciﬁc data domain, classiﬁer architecture, or attack conﬁguration; even for BPs embedded by Eq.

Original: (2), the same threshold can be obtained following a similar derivation (see Apdx. C).
Revised: (2), the same threshold can be obtained following a similar derivation (see Apdx. C).

Original: We ﬁrst present the following theorem showing the connection between ET and the thresholdregardless of the presence of any BA.
Revised: We ﬁrst present the following theorem showing the connection between ET and the thresholdregardless of the presence of any BA.

Original: Then, we discuss the attack and non-attack cases, respectively.
Revised: Then, we discuss the attack and non-attack cases, respectively.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For any class i ∈ C and (cid:15) > 0 :
Revised: For any class i ∈ C and (cid:15) > 0 :

Original: [Equation]
Revised: [Equation]

Original: (6) with P MT ,i a “mutual-transfer probability” and P NT ,i a “non-transfer probability”, deﬁned by
Revised: (6) with P MT ,i a “mutual-transfer probability” and P NT ,i a “non-transfer probability”, deﬁned by

Original: P MT ,i (cid:44) P( Y ∈ T (cid:15) ( X ) , X ∈ T (cid:15) ( Y )) and P NT ,i (cid:44) P( Y / ∈ T (cid:15) ( X ) , X / ∈ T (cid:15) ( Y )) (7) respectively, where X and Y are i.i.d.
Revised: P MT ,i (cid:44) P( Y ∈ T (cid:15) ( X ) , X ∈ T (cid:15) ( Y )) and P NT ,i (cid:44) P( Y / ∈ T (cid:15) ( X ) , X / ∈ T (cid:15) ( Y )) (7) respectively, where X and Y are i.i.d.

Original: random samples following distribution P i for class i .
Revised: random samples following distribution P i for class i .

Original: 1) Non-attack case: class (1 − i ) is not a BA target class.
Revised: 1) Non-attack case: class (1 − i ) is not a BA target class.

Original: We ﬁrst focus on P NT ,i .
Revised: We ﬁrst focus on P NT ,i .

Original: Property 3.1.
Revised: Property 3.1.

Original: For general two-class domains in practice and small (cid:15) , if class (1 − i ) is not a BA target class, P NT ,i for class i will likely be larger than 12 (see Sec.
Revised: For general two-class domains in practice and small (cid:15) , if class (1 − i ) is not a BA target class, P NT ,i for class i will likely be larger than 12 (see Sec.

Original: 4.3 for empirical support).
Revised: 4.3 for empirical support).

Original: For any i ∈ C , P NT ,i is the probability that two independent samples from class i are mutually “not transferable”.
Revised: For any i ∈ C , P NT ,i is the probability that two independent samples from class i are mutually “not transferable”.

Original: For such a pair of samples, the event associated with P NT ,i is that the pattern estimated for one (by solving (3)) does not induce the other to be misclassiﬁed and vice versa.
Revised: For such a pair of samples, the event associated with P NT ,i is that the pattern estimated for one (by solving (3)) does not induce the other to be misclassiﬁed and vice versa.

Original: Accordingly, if we solve a problem similar to (3) but requiring a common v that induces both samples to be misclassiﬁed, the solution should have a larger norm than the solution to (3) for each of them.
Revised: Accordingly, if we solve a problem similar to (3) but requiring a common v that induces both samples to be misclassiﬁed, the solution should have a larger norm than the solution to (3) for each of them.

Original: Property 3.1 has been veriﬁed by many existing works for general classiﬁcation tasks commonly using highly non-linear classiﬁers.
Revised: Property 3.1 has been veriﬁed by many existing works for general classiﬁcation tasks commonly using highly non-linear classiﬁers.

Original: For example, the universal adversarial perturbation studied by
Revised: For example, the universal adversarial perturbation studied by

Original: Moosavi-Dezfooli et al.
Revised: Moosavi-Dezfooli et al.

Original: (2017) can be viewed as the solution to problem (3) (in absence of BA) for a group of samples instead of one.
Revised: (2017) can be viewed as the solution to problem (3) (in absence of BA) for a group of samples instead of one.

Original: Compared with the minimum sample-wise perturbation required for each individual to be misclassiﬁed, the minimum universal perturbation required for high group misclassiﬁcation typically has a much larger norm.
Revised: Compared with the minimum sample-wise perturbation required for each individual to be misclassiﬁed, the minimum universal perturbation required for high group misclassiﬁcation typically has a much larger norm.

Original: Similar empirical results have also been shown by Xiang et al.
Revised: Similar empirical results have also been shown by Xiang et al.

Original: and inspired their proposed RED.
Revised: and inspired their proposed RED.

Original: Despite the evidence in existing works, we also verify this property experimentally in Sec 4.3.
Revised: Despite the evidence in existing works, we also verify this property experimentally in Sec 4.3.

Original: Next, we focus on P MT ,i .
Revised: Next, we focus on P MT ,i .

Original: Note that P MT ,i is upper bounded by 1 − P NT ,i ; thus it will likely be smaller than 12 (and even possibly close to 0) for the non-attack case based on Property 3.1.
Revised: Note that P MT ,i is upper bounded by 1 − P NT ,i ; thus it will likely be smaller than 12 (and even possibly close to 0) for the non-attack case based on Property 3.1.

Original: The asymptotic behavior of P MT ,i for small (cid:15) can also be approached by the following theorem.
Revised: The asymptotic behavior of P MT ,i for small (cid:15) can also be approached by the following theorem.

Original: Note that in practice, a small (cid:15) is not difﬁcult to achieve, since a solution to problem (3) using, e.g., algorithms for generating adversarial samples Szegedy et al.
Revised: Note that in practice, a small (cid:15) is not difﬁcult to achieve, since a solution to problem (3) using, e.g., algorithms for generating adversarial samples Szegedy et al.

Original: (2014) usually have a small norm.
Revised: (2014) usually have a small norm.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: For any class i ∈ C with continuous sample distribution P i , P MT ,i → 0 as (cid:15) → 0 .
Revised: For any class i ∈ C with continuous sample distribution P i , P MT ,i → 0 as (cid:15) → 0 .

Original: In summary, for the non-attack case with small (cid:15) , we will likely have P NT ,i ≥ 12 ≥ P MT ,i ; and thus,
Revised: In summary, for the non-attack case with small (cid:15) , we will likely have P NT ,i ≥ 12 ≥ P MT ,i ; and thus,

Original: ET i,(cid:15) ≤ 12 (based on Thm. 3.1) – this will be shown by our experiments.
Revised: ET i,(cid:15) ≤ 12 (based on Thm. 3.1) – this will be shown by our experiments.

Original: Moreover, in Apdx.
Revised: Moreover, in Apdx.

Original: B, for a simpliﬁed (yet still relatively general) domain and a linear prototype classiﬁer, we show that 12 is a strict upper bound of the ET statistic when there is no BA.
Revised: B, for a simpliﬁed (yet still relatively general) domain and a linear prototype classiﬁer, we show that 12 is a strict upper bound of the ET statistic when there is no BA.

Original: 2) Attack case: class (1 − i ) is the target class of a successful BA.
Revised: 2) Attack case: class (1 − i ) is the target class of a successful BA.

Original: Suppose v 0 is the BP for this attack, such that for any X ∼ P i , f ( X )
Revised: Suppose v 0 is the BP for this attack, such that for any X ∼ P i , f ( X )

Original: =
Revised: =

Original: i , while f ( X + v 0 ) (cid:54) =
Revised: i , while f ( X + v 0 ) (cid:54) =

Original: f ( X ) .
Revised: f ( X ) .

Original: Intuitively, P MT ,i will be large and possibly close to 1 because, different from the non-attack case , there is a special pattern – the BP v 0 – that could likely be an element of V (cid:15) ( X ) and V (cid:15) ( Y ) simultaneously, for X and Y i.i.d.
Revised: Intuitively, P MT ,i will be large and possibly close to 1 because, different from the non-attack case , there is a special pattern – the BP v 0 – that could likely be an element of V (cid:15) ( X ) and V (cid:15) ( Y ) simultaneously, for X and Y i.i.d.

Original: following P i .
Revised: following P i .

Original: In this case, Y ∈ T (cid:15) ( X ) and X ∈ T (cid:15)
Revised: In this case, Y ∈ T (cid:15) ( X ) and X ∈ T (cid:15)

Original: ( Y ) jointly hold (i.e. mutually transferable) by
Revised: ( Y ) jointly hold (i.e. mutually transferable) by

Original: Deﬁnition 3.2.
Revised: Deﬁnition 3.2.

Original: Accordingly, P NT ,i which is upper bounded by 1 − P MT ,i will likely be small and possibly close to 0; then, we will have P MT ,i > P NT ,i and consequently, ET i,(cid:15) > 12 by Thm. 3.1.
Revised: Accordingly, P NT ,i which is upper bounded by 1 − P MT ,i will likely be small and possibly close to 0; then, we will have P MT ,i > P NT ,i and consequently, ET i,(cid:15) > 12 by Thm. 3.1.

Original: Beyond the intuitive analysis above, the following theorem gives a guaranteed large ET statistic (being exactly 1) in the attack case when the backdoor pattern has a (sufﬁciently) small norm (which is in fact desired in order to have imperceptibility of the attack).
Revised: Beyond the intuitive analysis above, the following theorem gives a guaranteed large ET statistic (being exactly 1) in the attack case when the backdoor pattern has a (sufﬁciently) small norm (which is in fact desired in order to have imperceptibility of the attack).

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: If class (1 − i ) is the target class of a successful BA with BP v 0 such that f ( X + v 0 ) (cid:54) =
Revised: If class (1 − i ) is the target class of a successful BA with BP v 0 such that f ( X + v 0 ) (cid:54) =

Original: f ( X ) for all X ∼ P i , and if || v 0 || 2 ≤ (cid:15) , we will have P ( V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54) = ∅ )
Revised: f ( X ) for all X ∼ P i , and if || v 0 || 2 ≤ (cid:15) , we will have P ( V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54) = ∅ )

Original: = 1 for X and Y i.i.d.
Revised: = 1 for X and Y i.i.d.

Original: following P i ; and furthermore, ET i,(cid:15) = 1 .
Revised: following P i ; and furthermore, ET i,(cid:15) = 1 .

Original: 3. D ETECTION P ROCEDURE
Revised: 3. D ETECTION P ROCEDURE

Original: Our detection procedure is summarized in Alg. 1.
Revised: Our detection procedure is summarized in Alg. 1.

Original: Basically, for each putative target class t ∈ C = { 0 , 1 } , we estimate the ET statistic (cid:99) ET i,(cid:15) (with a “hat” representing empirical estimation) for class i = 1 − t , and claim a detection if (cid:99) ET i,(cid:15) > 12 .
Revised: Basically, for each putative target class t ∈ C = { 0 , 1 } , we estimate the ET statistic (cid:99) ET i,(cid:15) (with a “hat” representing empirical estimation) for class i = 1 − t , and claim a detection if (cid:99) ET i,(cid:15) > 12 .

Original: In particular, the core to estimating (cid:99) ET i,(cid:15) is to estimate
Revised: In particular, the core to estimating (cid:99) ET i,(cid:15) is to estimate

Original: P( Y ∈ T (cid:15) ( X ) | X = x ( i ) n ) for each clean sample x
Revised: P( Y ∈ T (cid:15) ( X ) | X = x ( i ) n ) for each clean sample x

Original: ( i ) n ∈ D i used for detection (line 6-12, Alg. 1).
Revised: ( i ) n ∈ D i used for detection (line 6-12, Alg. 1).

Original: To do so, we propose to ﬁnd, for each x ( i ) n ∈ D i , the subset (cid:98) T (cid:15)
Revised: To do so, we propose to ﬁnd, for each x ( i ) n ∈ D i , the subset (cid:98) T (cid:15)

Original: ( x ( i ) n ) which contains all samples in
Revised: ( x ( i ) n ) which contains all samples in

Original: D
Revised: D

Original: i \ x ( i ) n belonging to the transferable set T (cid:15) ( x ( i ) n ) of sample x ( i ) n .
Revised: i \ x ( i ) n belonging to the transferable set T (cid:15) ( x ( i ) n ) of sample x ( i ) n .

Original: Then, P( Y ∈ T (cid:15) ( X ) | X = x ( i ) n ) can be estimated by p ( i ) n = | (cid:98) T (cid:15) ( x ( i ) n ) | / ( |D i |− 1) (line 12, Alg. 1).
Revised: Then, P( Y ∈ T (cid:15) ( X ) | X = x ( i ) n ) can be estimated by p ( i ) n = | (cid:98) T (cid:15) ( x ( i ) n ) | / ( |D i |− 1) (line 12, Alg. 1).

Original: However, by Def. 3.2, a sample y is in the transferable set T (cid:15) ( x ) of a sample x as long as there exists a practical solution to problem (3) (with some intrinsic quality gap (cid:15) ) that induces y to be misclassiﬁed as well.
Revised: However, by Def. 3.2, a sample y is in the transferable set T (cid:15) ( x ) of a sample x as long as there exists a practical solution to problem (3) (with some intrinsic quality gap (cid:15) ) that induces y to be misclassiﬁed as well.

Original: Thus, it is insufﬁcient to decide whether or not a sample is in the transferable set of another sample according to merely one solution realization to problem (3).
Revised: Thus, it is insufﬁcient to decide whether or not a sample is in the transferable set of another sample according to merely one solution realization to problem (3).

Original: To address this, for each x ( i ) n , we solve problem (3) repeatedly with random initialization.
Revised: To address this, for each x ( i ) n , we solve problem (3) repeatedly with random initialization.

Original: For each practical solution, we embed it to all elements in D
Revised: For each practical solution, we embed it to all elements in D

Original: i \ x ( i ) n and ﬁnd those that are misclassiﬁed – these samples are included into the subset (cid:98) T (cid:15) ( x ( i ) n ) (line 9, 
Revised: i \ x ( i ) n and ﬁnd those that are misclassiﬁed – these samples are included into the subset (cid:98) T (cid:15) ( x ( i ) n ) (line 9, [Figure]

Original: Alg. 1).
Revised: Alg. 1).

Original: Such repetition stops when (cid:98) T (cid:15) ( x ( i ) n ) stays unchanged for some τ iterations.
Revised: Such repetition stops when (cid:98) T (cid:15) ( x ( i ) n ) stays unchanged for some τ iterations.

Original: This procedure is summarized as the “while” loop in Alg. 1, which is guaranteed to converge in ( N i − 1) × 
Revised: This procedure is summarized as the “while” loop in Alg. 1, which is guaranteed to converge in ( N i − 1) × τ iterations.

Revised: Finally, we obtain the estimated ET by averaging p ( i ) n – the empirical estimation of

Original: τ
Revised: P(

Original: [Figure]
Revised: [Equation]

Original: iterations. Finally, we obtain the estimated ET by averaging p ( i ) n – the empirical estimation of
Revised: ∈

Original: P( Y ∈ T (cid:15) ( X ) | X = x ( i ) n ) – over all samples in D i .
Revised: [Equation] over all samples in D i .

Original: Our detection framework has the following generalization capabilities.
Revised: Our detection framework has the following generalization capabilities.

Original: 1) BP embedding mechanism.
Revised: 1) BP embedding mechanism.

Original: As mentioned before, our detection rule can be adapted to BPs embedded by Eq.
Revised: As mentioned before, our detection rule can be adapted to BPs embedded by Eq.

Original: (2), with the same constant threshold 12 on ET and only little modiﬁcation to the while loop in Alg.
Revised: (2), with the same constant threshold 12 on ET and only little modiﬁcation to the while loop in Alg.

Original: 1 (see Apdx.
Revised: 1 (see Apdx.

Original: C.3).
Revised: C.3).

Original: 2) BP reverse-engineering algorithm.
Revised: 2) BP reverse-engineering algorithm.

Original: We do not limit our detector to any speciﬁc algorithm when solving problem (3) (line 8 in Alg. 1) – existing algorithms proposed by, e.g., Xiang et al.
Revised: We do not limit our detector to any speciﬁc algorithm when solving problem (3) (line 8 in Alg. 1) – existing algorithms proposed by, e.g., Xiang et al.

Original: Carlini & Wagner (2017), and even future BP reverse-engineering algorithms can be used.
Revised: Carlini & Wagner (2017), and even future BP reverse-engineering algorithms can be used.

Original: 3) Adoption for multi-class scenario.
Revised: 3) Adoption for multi-class scenario.

Original: When there are multiple classes, where each can possibly be a BA target, Alg. 1 can still be used for detection by, for each putative target class t ∈ C , treating all the classes other than t as a super-class and estimating ET on ∪ i ∈C\
Revised: When there are multiple classes, where each can possibly be a BA target, Alg. 1 can still be used for detection by, for each putative target class t ∈ C , treating all the classes other than t as a super-class and estimating ET on ∪ i ∈C\

Original: t D i .
Revised: t D i .

Original: In our experiments (next), we will evaluate our detection framework considering a variety of these extensions.
Revised: In our experiments (next), we will evaluate our detection framework considering a variety of these extensions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments involve six common benchmark image datasets with a variety of image size and color scale: CIFAR-10, CIFAR-100 Krizhevsky (2012), STL-10 Coates et al.
Revised: Our experiments involve six common benchmark image datasets with a variety of image size and color scale: CIFAR-10, CIFAR-100 Krizhevsky (2012), STL-10 Coates et al.

Original: (2011), TinyImageNet,
Revised: (2011), TinyImageNet,

Original: FMNIST Xiao et al.
Revised: FMNIST Xiao et al.

Original: , MNIST Lecun et al.
Revised: , MNIST Lecun et al.

Original: Details of these datasets are in Apdx.
Revised: Details of these datasets are in Apdx.

Original: D.1.
Revised: D.1.

Original: 4. M AIN E XPERIMENT : 2-C LASS , M ULTI -A TTACK BA D ETECTION U SING ET S TATISTIC
Revised: M AIN E XPERIMENT : 2-C LASS , M ULTI -A TTACK BA D ETECTION U SING ET S TATISTIC

Original: Generating 2-class domains.
Revised: Generating 2-class domains.

Original: From CIFAR-10, we generate 45 different 2-class domains (for allunordered class pairs of CIFAR-10).
Revised: From CIFAR-10, we generate 45 different 2-class domains (for allunordered class pairs of CIFAR-10).

Original: From each of the other ﬁve datasets, we generate 20 different random 2-class domains.
Revised: From each of the other ﬁve datasets, we generate 20 different random 2-class domains.

Original: More details are provided in Apdx.
Revised: More details are provided in Apdx.

Original: D.2.
Revised: D.2.

Original: Attack conﬁguration.
Revised: Attack conﬁguration.

Original: Like most related works, we mainly focus on classical BAs launched by poisoning the classiﬁer’s training set with a small set of samples embedded with a BP and labeled to some target class Gu et al.
Revised: Like most related works, we mainly focus on classical BAs launched by poisoning the classiﬁer’s training set with a small set of samples embedded with a BP and labeled to some target class Gu et al.

Original: Effectiveness of our detector against another type of clean-label
Revised: Effectiveness of our detector against another type of clean-label

Original: BA is shown in Apdx.
Revised: BA is shown in Apdx.

Original: E.
Revised: E.

Original: For each 2-class domain generated from CIFAR-10, CIFAR-100, STL-10, and TinyImageNet, we create two attack instances , one for BA with additive perturbation BP embedded by Eq.
Revised: For each 2-class domain generated from CIFAR-10, CIFAR-100, STL-10, and TinyImageNet, we create two attack instances , one for BA with additive perturbation BP embedded by Eq.

Original: (1), and the other for BA with patch replacement BP embedded by Eq.
Revised: (1), and the other for BA with patch replacement BP embedded by Eq.

Original: For each 2-class domain generated from FMNIST and MNIST, we create one attack instance with additive perturbation BP 2 .
Revised: For each 2-class domain generated from FMNIST and MNIST, we create one attack instance with additive perturbation BP 2 .

Original: For convenience, the six ensembles of attack instances with additive perturbation
Revised: For convenience, the six ensembles of attack instances with additive perturbation

Original: BP and for 2-class domains generated from CIFAR-10, CIFAR-100, STL-10, TinyImageNet, FMNIST, and MNIST are denoted as A 1 -A 6 respectively.
Revised: BP and for 2-class domains generated from CIFAR-10, CIFAR-100, STL-10, TinyImageNet, FMNIST, and MNIST are denoted as A 1 -A 6 respectively.

Original: The four ensembles of attack instances with patch replacement BP and for 2-class domains generated from CIFAR-10, CIFAR-100, STL-10, and
Revised: The four ensembles of attack instances with patch replacement BP and for 2-class domains generated from CIFAR-10, CIFAR-100, STL-10, and

Original: TinyImageNet are denoted as A 7 -A 10 respectively.
Revised: TinyImageNet are denoted as A 7 -A 10 respectively.

Original: The BPs used in our experiments include many popular ones in the BA literature – examples of some BPs and images embedded with them are shown in Fig.
Revised: The BPs used in our experiments include many popular ones in the BA literature – examples of some BPs and images embedded with them are shown in Fig.

Original: 1 (details for generating these BPs are deferred to Apdx. D.3).
Revised: 1 (details for generating these BPs are deferred to Apdx. D.3).

Original: We consider 2-class scenarios where both classes can possibly be a BA target class .
Revised: We consider 2-class scenarios where both classes can possibly be a BA target class .

Original: For each attack instance in ensembles A 1 , A 2 , A 3 , A 7 , A 8 , and A 10 , we create two attacks each with one of the two classes being the BA target class.
Revised: For each attack instance in ensembles A 1 , A 2 , A 3 , A 7 , A 8 , and A 10 , we create two attacks each with one of the two classes being the BA target class.

Original: For each instance in ensembles A 4 , A 5 , A 6 , and A 9 , we create one attack with the second class being the BA target class.
Revised: For each instance in ensembles A 4 , A 5 , A 6 , and A 9 , we create one attack with the second class being the BA target class.

Original: For each of these attacks, the BP israndomly selected from the candidate BPs (part of which are shown in Fig. 1), with the speciﬁed BP type for the ensemble which the attack is associated with. However, to avoid confusion for learning
Revised: For each of these attacks, the BP is

Original: [Table]
Revised: [Table]

Revised: randomly selected from the candidate BPs (part of which are shown in Fig. 1), with the speciﬁed BP type for the ensemble which the attack is associated with.

Original:  the backdoor mapping during training, for all 2-attack instances with additive perturbation BPs, we ensure that the BPs for the two attacks have different shapes (e.g., two “X” BPs are not allowed).
Revised: However, to avoid confusion for learning the backdoor mapping during training, for all 2-attack instances with additive perturbation BPs, we ensure that the BPs for the two attacks have different shapes (e.g., two “X” BPs are not allowed).

Original: Similarly, for all 2-attack instances where the two attacks both choose to use the unicolor patch
Revised: Similarly, for all 2-attack instances where the two attacks both choose to use the unicolor patch

Original: BP (Fig. 1e), we ensure that the colors for the two BPs are signiﬁcantly different.
Revised: BP (Fig. 1e), we ensure that the colors for the two BPs are signiﬁcantly different.

Original: Other attack conﬁgurations, including e.g., the poisoning rate for each attack are detailed in Apdx.
Revised: Other attack conﬁgurations, including e.g., the poisoning rate for each attack are detailed in Apdx.

Original: D.4.
Revised: D.4.

Original: Training conﬁgurations.
Revised: Training conﬁgurations.

Original: We train one classiﬁer for each attack instance using the poisoned training set.
Revised: We train one classiﬁer for each attack instance using the poisoned training set.

Original: For each 2-class domain, we also train a clean classiﬁer to evaluate false detections.
Revised: For each 2-class domain, we also train a clean classiﬁer to evaluate false detections.

Original: We denote the six ensembles of clean instances for datasets CIFAR-10, CIFAR-100, STL-10, TinyImageNet, FMNIST, and MNIST as C 1 -C 6 respectively.
Revised: We denote the six ensembles of clean instances for datasets CIFAR-10, CIFAR-100, STL-10, TinyImageNet, FMNIST, and MNIST as C 1 -C 6 respectively.

Original: For classiﬁer training, we consider a variety of
Revised: For classiﬁer training, we consider a variety of

Original: DNN architectures (with two output neurons, one for each of the two classes).
Revised: DNN architectures (with two output neurons, one for each of the two classes).

Original: For 2-class domains associated with CIFAR-10 and STL-10, we use ResNet-18 He et al.
Revised: For 2-class domains associated with CIFAR-10 and STL-10, we use ResNet-18 He et al.

Original: (2016); for CIFAR-100 and
Revised: (2016); for CIFAR-100 and

Original: FMNIST, we use VGG-11 Simonyan & Zisserman (2015); for TinyImageNet, we use ResNet-34; and for MNIST, we use LeNet-5 Lecun et al.
Revised: FMNIST, we use VGG-11 Simonyan & Zisserman (2015); for TinyImageNet, we use ResNet-34; and for MNIST, we use LeNet-5 Lecun et al.

Original: Other training conﬁgurations including the learning rate, the batch size, and the optimizer choice for each 2-class domain are detailed in Apdx.
Revised: Other training conﬁgurations including the learning rate, the batch size, and the optimizer choice for each 2-class domain are detailed in Apdx.

Original: D.5.
Revised: D.5.

Original: Moreover, all attacks for all the instances are successful with high attack success rate and negligible degradation in clean test accuracy – these two metrics are commonly used to evaluate the
Revised: Moreover, all attacks for all the instances are successful with high attack success rate and negligible degradation in clean test accuracy – these two metrics are commonly used to evaluate the

Original: BA effectiveness Wang et al.
Revised: BA effectiveness Wang et al.

Original: As a defense paper, we defer these attack details to Apdx.
Revised: As a defense paper, we defer these attack details to Apdx.

Original: D.5.
Revised: D.5.

Original: Defense conﬁgurations.
Revised: Defense conﬁgurations.

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 3.3, our detection framework can be easily generalized to incorporate a variety of algorithms for BP reverse-engineering to detect various types of BP with different embedding mechanisms.
Revised: 3.3, our detection framework can be easily generalized to incorporate a variety of algorithms for BP reverse-engineering to detect various types of BP with different embedding mechanisms.

Original: Here, we consider the algorithm proposed by Xiang et al.
Revised: Here, we consider the algorithm proposed by Xiang et al.

Original: (2020) for estimating Additive Perturbation BPs (by solving (3)), and the algorithm proposed by Wang et al.
Revised: for estimating A dditive Perturbation BPs (by solving (3)) , and the algorithm proposed by Wang et al.

Original: for estimating Patch Replacement BPs (by solving (46) in Apdx.
Revised: for estimating Patch Replacement BPs (by solving (46) in Apdx.

Original: C.1).
Revised: C.1).

Original: For convenience, we denote detection conﬁgurations with these two algorithms as RE-AP and RE-PR , respectively.
Revised: For convenience, we denote detection conﬁgurations with these two algorithms as RE-AP and RE-PR , respectively.

Original: Details for these two algorithms are both introduced in the original papers and reviewed in Apdx.
Revised: Details for these two algorithms are both introduced in the original papers and reviewed in Apdx.

Original: D.6.
Revised: D.6.

Original: Irrespective of the BP reverse-engineering algorithm, we use only 20 clean images per class (similar to most existing REDs) for detection.
Revised: Irrespective of the BP reverse-engineering algorithm, we use only 20 clean images per class (similar to most existing REDs) for detection.

Original: Finally, we set the “patience” parameter for determination of convergence in Alg.
Revised: Finally, we set the “patience” parameter for determination of convergence in Alg.

Original: 1 to τ =4 – this choice is independent of the presence of BA; a larger τ will not change the resulting ET much, but only increase the execution time.
Revised: 1 to τ =4 – this choice is independent of the presence of BA; a larger τ will not change the resulting ET much, but only increase the execution time.

Original: Detection performance (using the common ET threshold 1/2).
Revised: Detection performance (using the common ET threshold 1/2).

Original: In practice, our detector with REAP and with RE-PR can be deployed in parallel to cover both additive perturbation BPs and patch replacement BPs.
Revised: In practice, our detector with REAP and with RE-PR can be deployed in parallel to cover both additive perturbation BPs and patch replacement BPs.

Original: Here, for simplicity, we apply our detector with RE-AP to classiﬁers (with BAs using additive perturbation BP) in ensemble A 1 -A 6 , and apply our detector with RE-PR to classiﬁers (with BAs using patch replacement BP) in ensemble A 7 -A 10 .
Revised: Here, for simplicity, we apply our detector with RE-AP to classiﬁers (with BAs using additive perturbation BP) in ensemble A 1 -A 6 , and apply our detector with RE-PR to classiﬁers (with BAs using patch replacement BP) in ensemble A 7 -A 10 .

Original: For each classiﬁer in clean ensembles
Revised: For each classiﬁer in clean ensembles

Original: C 1 -C 6 , we apply our detector with both conﬁgurations.
Revised: C 1 -C 6 , we apply our detector with both conﬁgurations.

Original: In Tab. 1, for each ensemble of attack instances, we report the fraction of classiﬁers such that the attack and all BA target classes are both successfully detected; for each ensemble of clean instances, we report the fraction of classiﬁers that are inferred to be not attacked.
Revised: In Tab. 1, for each ensemble of attack instances, we report the fraction of classiﬁers such that the attack and all BA target classes are both successfully detected; for each ensemble of clean instances, we report the fraction of classiﬁers that are inferred to be not attacked.

Original: Given the large variety of classiﬁcation domains, attack conﬁgurations, DNN architectures, and defense generalizations mentioned above, using the common ET threshold 1/2 , we successfully detect most attacks with only very few false detections (see Tab. 1). 
Revised: Given the large variety of classiﬁcation domains, attack conﬁgurations, DNN architectures, and defense generalizations mentioned above, using the common ET threshold 1/2 , we successfully detect most attacks with only very few false detections (see Tab. 1). More discussions regarding the detection performance are deferred to Apdx.

Original:  4. C OMPARE ET WITH O THER S TATISTICS
Revised: K. 4. C OMPARE ET WITH O THER S TATISTICS

Original: The results for existing REDs applying to the attacks we created are neglected for brevity, since these
Revised: The results for existing REDs applying to the attacks we created are neglected for brevity, since these

Original: REDs cannot detect BAs for 2-class domains by their design.
Revised: REDs cannot detect BAs for 2-class domains by their design.

Original: However, we compare our ET statistic with some popular types of statistic used by existing REDs in terms of their potential for being used to distinguish BA target classes from non-target classes .
Revised: However, we compare our ET statistic with some popular types of statistic used by existing REDs in terms of their potential for being used to distinguish BA target classes from non-target classes .

Original: The types of statistic for comparison include: 1) the l 2 norm of the estimated additive perturbation used by Xiang et al.
Revised: The types of statistic for comparison include: 1) the l 2 norm of the estimated additive perturbation used by Xiang et al.

Original: (2020) (denoted by L 2 ); 2) the l 1 norm of the estimated mask used by Wang et al.
Revised: (2020) (denoted by L 2 ); 2) the l 1 norm of the estimated mask used by Wang et al.

Original: (2019) (denoted by L 1 ); and 3) the (cosine) similarity between the BP estimated group-wise and the BP estimated for each sample in terms of classiﬁer’s internal layer representation Wang et al.
Revised: (2019) (denoted by L 1 ); and 3) the (cosine) similarity between the BP estimated group-wise and the BP estimated for each sample in terms of classiﬁer’s internal layer representation Wang et al.

Original: (2020) (denoted by CS ).
Revised: (2020) (denoted by CS ).

Original: For each type of statistic mentioned above and each benchmark dataset, we consider all classiﬁers (with and without BA) trained for all 2-class domains generated from the dataset, and plot a double histogram for statistics obtained for all BA target classes and all non-target classes across these classiﬁers respectively.
Revised: For each type of statistic mentioned above and each benchmark dataset, we consider all classiﬁers (with and without BA) trained for all 2-class domains generated from the dataset, and plot a double histogram for statistics obtained for all BA target classes and all non-target classes across these classiﬁers respectively.

Original: For example, in the 1 st column of Fig. 2, for our detector with RE-AP, we plot a double histogram for ET statistics obtained for all BA target classes and all non-target classes from all classiﬁers in each of A 1 &C 1 , A 2 &C 2 , A 3 &C 3 , A 4 &C 4 , A 5 &C 5 , and A 6 &C 6 .
Revised: For example, in the 1 st column of Fig. 2, for our detector with RE-AP, we plot a double histogram for ET statistics obtained for all BA target classes and all non-target classes from all classiﬁers in each of A 1 &C 1 , A 2 &C 2 , A 3 &C 3 , A 4 &C 4 , A 5 &C 5 , and A 6 &C 6 .

Original: Note that all classiﬁers in, e.g., A 1 &C 1 are trained for 2-class domains generated from CIFAR-10.
Revised: Note that all classiﬁers in, e.g., A 1 &C 1 are trained for 2-class domains generated from CIFAR-10.

Original: Here, for simplicity, for ET (with RE-AP) and L 2 (both designated for additive perturbation BP), we do not consider BA target classes with patch replacement BP (e.g. associated with classiﬁers in A 7 -A 10 ); for ET (with RE-PR), L 1 and CS (designated for patch replacement BP), we do not consider BA target classes with additive perturbation BP (e.g. associated with classiﬁers in A 1 -A 6 ).
Revised: Here, for simplicity, for ET (with RE-AP) and L 2 (both designated for additive perturbation BP), we do not consider BA target classes with patch replacement BP (e.g. associated with classiﬁers in A 7 -A 10 ); for ET (with RE-PR), L 1 and CS (designated for patch replacement BP), we do not consider BA target classes with additive perturbation BP (e.g. associated with classiﬁers in A 1 -A 6 ).

Original: Based on Fig.
Revised: Based on Fig.

Original: 2, for all types of statistics including our ET, statistics obtained for BA target classes are generally separable from statistics obtained for non-target classes for classiﬁers (with and without BA) trained for 2-class domains generated from the same benchmark dataset (using same training conﬁgurations including DNN architecture).
Revised: 2, for all types of statistics including our ET, statistics obtained for BA target classes are generally separable from statistics obtained for non-target classes for classiﬁers (with and without BA) trained for 2-class domains generated from the same benchmark dataset (using same training conﬁgurations including DNN architecture).

Original: But only for our ET (obtained by both RE-AP and
Revised: But only for our ET (obtained by both RE-AP and

Original: RE-PR), there is a common range (irrespective of the classiﬁcation domain, attack conﬁgurations, and training conﬁgurations) for choosing a detection threshold to effectively distinguish BA target classes from non-target classes for all instances; and such common range clearly includes the constant threshold 1/2 (marked by red dashed lines in Fig. 2) derived mathematically in Sec. 3.
Revised: RE-PR), there is a common range (irrespective of the classiﬁcation domain, attack conﬁgurations, and training conﬁgurations) for choosing a detection threshold to effectively distinguish BA target classes from non-target classes for all instances; and such common range clearly includes the constant threshold 1/2 (marked by red dashed lines in Fig. 2) derived mathematically in Sec. 3.

Original: By contrast, for both L 1 and L 2 , a proper choice of detection threshold for distinguishing BA target classes from non-target classes is domain dependent.
Revised: By contrast, for both L 1 and L 2 , a proper choice of detection threshold for distinguishing BA target classes from non-target classes is domain dependent.

Original: For example, in the 4 th column of Fig. 2, the l 1 norm of masks estimated for both BA target classes and non-target classes for domains with larger image size (e.g. 96 × 96 for domains in A 9 &C 3 generated from STL-10) is commonly larger than for domains with smaller image size (e.g. 32 × 32 for domains in A 7 &C 1 generated from CIFAR-10).
Revised: For example, in the 4 th column of Fig. 2, the l 1 norm of masks estimated for both BA target classes and non-target classes for domains with larger image size (e.g. 96 × 96 for domains in A 9 &C 3 generated from STL-10) is commonly larger than for domains with smaller image size (e.g. 32 × 32 for domains in A 7 &C 1 generated from CIFAR-10).

Original: The CS statistic, on the other hand, not only relies on the domain, but also depends on the DNN architecture.
Revised: The CS statistic, on the other hand, not only relies on the domain, but also depends on the DNN architecture.

Original: More details regarding CS are deferred to Apdx.
Revised: More details regarding CS are deferred to Apdx.

Original: D.7.
Revised: D.7.

Original: In summary, all the above mentioned types of statistics are suitable for BA detection if there is supervision from the same domain for choosing the best proper detection threshold .
Revised: In summary, all the above mentioned types of statistics are suitable for BA detection if there is supervision from the same domain for choosing the best proper detection threshold .

Original: However, in most practical scenarios where such supervision is not available, only our ET statistic with the common detection threshold 1/2 can still be used to achieve a good detection performance.
Revised: However, in most practical scenarios where such supervision is not available, only our ET statistic with the common detection threshold 1/2 can still be used to achieve a good detection performance.

Original: Finally, in Fig.
Revised: Finally, in Fig.

Original: 3, we show the receiver operating characteristic (ROC) curves associated with Fig.
Revised: 3, we show the receiver operating characteristic (ROC) curves associated with Fig.

Original: 2 for each of ET, L 1 , L 2 , and SC – our ET statistic has clearly larger area under the ROC curve (very close to 1) than the other types of statistics.
Revised: 2 for each of ET, L 1 , L 2 , and SC – our ET statistic has clearly larger area under the ROC curve (very close to 1) than the other types of statistics.

Original: 4. E XPERIMENTAL V ERIFICATION OF P ROPERTY 3. 
Revised: 4. E XPERIMENTAL V ERIFICATION OF P ROPERTY 3.

Original: We verify Property 3.1 by showing that, if a class is not a BA target class, for any two clean images from the other class (considering 2-class domains), the minimum additive perturbation required to induce both images to be misclassiﬁed has a larger norm than the minimum perturbation required for each of these two images to be misclassiﬁed.
Revised: We verify Property 3.1 by showing that, if a class is not a BA target class, for any two clean images from the other class (considering 2-class domains), the minimum additive perturbation required to induce both images to be misclassiﬁed has a larger norm than the minimum perturbation required for each of these two images to be misclassiﬁed.

Original: Here, we randomly choose one clean classiﬁer from each of C 1 -C 6 .
Revised: Here, we randomly choose one clean classiﬁer from each of C 1 -C 6 .

Original: For each classiﬁer, we randomly choose 50 pairs of clean images from a random class of the associated 2-class domain – these images are also used for detection in Sec. 4.1.
Revised: For each classiﬁer, we randomly choose 50 pairs of clean images from a random class of the associated 2-class domain – these images are also used for detection in Sec. 4.1.

Original: For each pair of images, we apply the same RE-AP algorithm (i.e. the BP reverse-engineering algorithm proposed by Xiang et al.
Revised: For each pair of images, we apply the same RE-AP algorithm (i.e. the BP reverse-engineering algorithm [Figure] [Figure]

Original: (2020) for additive perturbation BPs) on the two images both jointly (to get a pair-wise common perturbation) and separately (to get two sample-wise perturbationsfor the two images respectively).
Revised: [Table] [Figure] [Table] [Figure]

Original: Then, we divide the l 2 norm of the pair-wise perturbation by the maximum
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Table]

Revised: proposed by Xiang et al.

Revised: (2020) for additive perturbation BPs) on the two images both jointly (to get a pair-wise common perturbation) and separately (to get two sample-wise perturbationsfor the two images respectively).

Original:  l 2 norm of the two sample-wise perturbations to get a ratio (which is more scale-insensitive than taking absolute difference).
Revised: Then, we divide the l 2 norm of the pair-wise perturbation by the maximum l 2 norm of the two sample-wise perturbations to get a ratio (which is more scale-insensitive than taking absolute difference).

Original: In Fig.
Revised: In Fig.

Original: 4, we plot the histogram of such ratio for all image pairs for all six classiﬁers – the ratio for most of the image pairs is greater3 than 1 (marked by the red dashed line in Fig. 4).
Revised: 4, we plot the histogram of such ratio for all image pairs for all six classiﬁers – the ratio for most of the image pairs is greater than 1 (marked by the red dashed line in Fig. 4).

Original: For these pairs, very likely that the perturbation estimated for one sample cannot induce the other sample to be misclassiﬁed and vise versa (otherwise their expected ratio will likely be 1).
Revised: For these pairs, very likely that the perturbation estimated for one sample cannot induce the other sample to be misclassiﬁed and vise versa (otherwise their expected ratio will likely be 1).

Original: 4. M ULTI -C LASS , M ULTI -A TTACK BA D ETECTION U SING ET S TATISTIC
Revised: 4. M ULTI -C LASS , M ULTI -A TTACK BA D ETECTION U SING ET S TATISTIC

Original: Since our detector inspects each class independently, it can also be used for BA detection for more general scenarios with more than two classes and arbitrary number of attacks (and BA target classes).
Revised: Since our detector inspects each class independently, it can also be used for BA detection for more general scenarios with more than two classes and arbitrary number of attacks (and BA target classes).

Original: For demonstration, for each of CIFAR-10, CIFAR-100, and STL-10, we create three attack instances with one, two, and three attacks, respectively (on the original domain).
Revised: For demonstration, for each of CIFAR-10, CIFAR-100, and STL-10, we create three attack instances with one, two, and three attacks, respectively (on the original domain).

Revised: Additional experiments with aggregated results are deferred to Apdx.

Revised: L due to space limitations.

Original: Here, we consider BAs with additive perturbation BPs for simplicity.
Revised: Here, we consider BAs with additive perturbation BPs for simplicity.

Original: The target class and the shape of BP for each attack are randomly selected.
Revised: The target class and the shape of BP for each attack are randomly selected.

Original: We train one classiﬁer for each attack instance (thus, nine classiﬁers being attacked in total).
Revised: We train one classiﬁer for each attack instance (thus, nine classiﬁers being attacked in total).

Original: More details for these attacks and training conﬁgurations are in Apdx.
Revised: More details for these attacks and training conﬁgurations are in Apdx.

Original: D.8.
Revised: D.8.

Original: For each domain, we also train a clean classiﬁer (without BA) for evaluating false detections.
Revised: For each domain, we also train a clean classiﬁer (without BA) for evaluating false detections.

Original: Following the description at the end of Sec. 3, we apply the generalized Alg.
Revised: Following the description at the end of Sec. 3, we apply the generalized Alg.

Original: 1 with RE-AP to these classiﬁers.
Revised: 1 with RE-AP to these classiﬁers.

Original: For CIFAR-10 and STL-10, we use three clean images per class for detection; for
Revised: For CIFAR-10 and STL-10, we use three clean images per class for detection; for

Original: CIFAR-100, we use only one clean image per class for detection.
Revised: CIFAR-100, we use only one clean image per class for detection.

Original: Other detection conﬁgurations including the detection threshold 1/2 are the same as in Sec. 4.1.
Revised: Other detection conﬁgurations including the detection threshold 1/2 are the same as in Sec. 4.1.

Original: Since a classiﬁer is deemed to be attacked if ET obtained for any class is greater than the threshold 1/2, for each classiﬁer, we show the maximum ET over all the classes in Tab. 2.
Revised: Since a classiﬁer is deemed to be attacked if ET obtained for any class is greater than the threshold 1/2, for each classiﬁer, we show the maximum ET over all the classes in Tab. 2.

Original: Clearly, the maximum ET is greater than 1/2 for all classiﬁers being attacked and less than 1/2 for all clean classiﬁers.
Revised: Clearly, the maximum ET is greater than 1/2 for all classiﬁers being attacked and less than 1/2 for all clean classiﬁers.

Original: Thus, our detection framework (with the same constant threshold on ET) is also applicable to multi-class, multi-attack scenarios.
Revised: Thus, our detection framework (with the same constant threshold on ET) is also applicable to multi-class, multi-attack scenarios.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we proposed the ﬁrst BA detector for two-class, multi-attack scenarios without access to the classiﬁer’s training set or any supervision from clean reference classiﬁers trained for the same domain.
Revised: We proposed the ﬁrst BA detector for two-class, multi-attack scenarios without access to the classiﬁer’s training set or any supervision from clean reference classiﬁers trained for the same domain.

Original: Our detection framework is based on BP reverse-engineering and a novel ET statistic.
Revised: Our detection framework is based on BP reverse-engineering and a novel ET statistic.

Original: Our ET statistic can be used to effectively distinguish BA target classes from non-target classes, with alarge range of choices for detection threshold, which commonly contains a particular, theoreticalgrounded threshold value 1/2, irrespective of the classiﬁcation domain, the DNN architecture, and the attack conﬁgurations.
Revised: Our ET statistic can be used to effectively distinguish BA target classes from non-target classes, with a particular, theoretical-grounded threshold value 1/2, irrespective of the classiﬁcation domain, the DNN architecture, and the attack conﬁgurations.

Original: Our detection framework can also be generalized to incorporate a variety of BP reverse-engineering algorithms to address different BP types, and be applied to multi-class scenarios with arbitrary number of attacks.
Revised: Our detection framework can also be generalized to incorporate a variety of BP reverse-engineering algorithms to address different BP types, and be applied to multi-class scenarios with arbitrary number of attacks.

Original: A P ROOF OF T HEOREMS IN THE M AIN P APER A. P ROOF OF T HEOREM 3. 
Revised: A P ROOF OF T HEOREMS IN THE M AIN P APER A. P ROOF OF T HEOREM 3.

Original: For random variables X and Y i.i.d.
Revised: For random variables X and Y i.i.d.

Original: following distribution P i , for any realization x of X , we have
Revised: following distribution P i , for any realization x of X , we have

Original: [Equation]
Revised: [Equation]

Original: Thus, ET, the left hand side of Eq.
Revised: Thus, ET, the left hand side of Eq.

Original: (6), can be written as
Revised: (6), can be written as

Original: [Equation]
Revised: [Equation]

Original: Based on the inclusion-exclusion rule
Revised: Based on the inclusion-exclusion rule

Original: [Equation]
Revised: [Equation]

Original: Since, X and Y are i.i.d.
Revised: Since, X and Y are i.i.d.

Original: random variables, P( Y ∈ T (cid:15) ( X ))
Revised: random variables, P( Y ∈ T (cid:15) ( X ))

Original: =
Revised: =

Original: P( X ∈ T (cid:15) ( Y )) ; thus, we get Eq.
Revised: P( X ∈ T (cid:15) ( Y )) ; thus, we get Eq.

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: Step 1: We show that for any i ∈ C and X , Y i.i.d.
Revised: Step 1: We show that for any i ∈ C and X , Y i.i.d.

Original: following distribution P i , X ∈ T (cid:15) ( Y ) and
Revised: following distribution P i , X ∈ T (cid:15) ( Y ) and

Original: Y ∈ T (cid:15) ( X ) if and only if V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54) = ∅ .
Revised: Y ∈ T (cid:15) ( X ) if and only if V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54) = ∅ .

Original: (a) If V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54)
Revised: (a) If V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54)

Original: = ∅ , there exists v such that v ∈ V (cid:15) ( X ) and v ∈ V (cid:15) ( Y ) .
Revised: = ∅ , there exists v such that v ∈ V (cid:15) ( X ) and v ∈ V (cid:15) ( Y ) .

Original: By Deﬁnition 3.1,
Revised: By Deﬁnition 3.1,

Original: [Equation]
Revised: [Equation]

Original: Then, by Deﬁnition 3.2, the existence of such v yields X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15) ( X ) .
Revised: Then, by Deﬁnition 3.2, the existence of such v yields X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15) ( X ) .

Original: (b) We prove the “only if” part by contradiction.
Revised: (b) We prove the “only if” part by contradiction.

Original: Given X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15) ( X ) , suppose
Revised: Given X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15) ( X ) , suppose

Original: V (cid:15) ( X ) ∩V
Revised: V (cid:15) ( X ) ∩V

Original: (cid:15) ( Y ) = ∅ .
Revised: (cid:15) ( Y ) = ∅ .

Original: By Deﬁnition 3.2, if Y ∈ T (cid:15) ( X ) , there exists v ∈ V (cid:15) ( X ) such that f ( Y + v ) (cid:54) =
Revised: By Deﬁnition 3.2, if Y ∈ T (cid:15) ( X ) , there exists v ∈ V (cid:15) ( X ) such that f ( Y + v ) (cid:54) =

Original: f ( Y ) .
Revised: f ( Y ) .

Original: Since V (cid:15) ( X ) ∩ V (cid:15) ( Y ) = ∅ , v / ∈ V (cid:15) ( Y ) ; hence, by Deﬁnition 3.
Revised: Since V (cid:15) ( X ) ∩ V (cid:15) ( Y ) = ∅ , v / ∈ V (cid:15) ( Y ) ; hence, by Deﬁnition 3.

Original: [Equation]
Revised: [Equation]

Original: Similarly, there exists v (cid:48) ∈ V (cid:15) ( Y ) such that f ( X + v (cid:48) )
Revised: Similarly, there exists v (cid:48) ∈ V (cid:15) ( Y ) such that f ( X + v (cid:48) )

Original: (cid:54) =
Revised: (cid:54) =

Original: f ( X ) and v (cid:48) / ∈ V (cid:15) ( X ) ; hence
Revised: f ( X ) and v (cid:48) / ∈ V (cid:15) ( X ) ; hence

Original: [Equation]
Revised: [Equation]

Original: By Deﬁnition 3.1, for all u ∈ V (cid:15) ( Y ) , || u || 2 − || v ∗ ( Y ) || 2 < (cid:15) .
Revised: By Deﬁnition 3.1, for all u ∈ V (cid:15) ( Y ) , || u || 2 − || v ∗ ( Y ) || 2 < (cid:15) .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: 15) and accordingly || v || 2 >
Revised: 15) and accordingly || v || 2 >

Original: ||
Revised: ||

Original: u
Revised: u

Original: || 2 for all u ∈ V (cid:15) ( Y ) .
Revised: || 2 for all u ∈ V (cid:15) ( Y ) .

Original: Similarly, for all u (cid:48) ∈ V (cid:15) ( X ) , we have
Revised: Similarly, for all u (cid:48) ∈ V (cid:15) ( X ) , we have

Original: [Equation]
Revised: [Equation]

Original: 16) and thus, || v (cid:48)
Revised: 16) and thus, || v (cid:48)

Original: || 2 > || u (cid:48)
Revised: || 2 > || u (cid:48)

Original: || 2 for all u (cid:48) ∈ V (cid:15) ( X ) .
Revised: || 2 for all u (cid:48) ∈ V (cid:15) ( X ) .

Original: Clearly, there is a contradiction since there cannot exist an element in one set having a larger norm than all elements in another set while vise versa; and therefore, V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54) = ∅ .
Revised: Clearly, there is a contradiction since there cannot exist an element in one set having a larger norm than all elements in another set while vise versa; and therefore, V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54) = ∅ .

Original: Step 2: We show that the upper bound for P MT ,i goes to 0 when (cid:15) → 0 .
Revised: Step 2: We show that the upper bound for P MT ,i goes to 0 when (cid:15) → 0 .

Original: Note that (cid:15) is the “quality gap” between the practical solution and the optimal solution.
Revised: Note that (cid:15) is the “quality gap” between the practical solution and the optimal solution.

Original: Note that by Deﬁnition 3.1, V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54) = ∅ only if (cid:12)(cid:12)
Revised: Note that by Deﬁnition 3.1, V (cid:15) ( X ) ∩ V (cid:15) ( Y ) (cid:54) = ∅ only if (cid:12)(cid:12)

Original: ||
Revised: ||

Original: v ∗ ( X ) ||
Revised: v ∗ ( X ) ||

Original: 2 − || v ∗ ( Y )
Revised: 2 − || v ∗ ( Y )

Original: || 2 (cid:12)(cid:12) ≤ (cid:15) ; also based on Step 1:
Revised: || 2 (cid:12)(cid:12) ≤ (cid:15) ; also based on Step 1:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For X following continuous distribution P i , || v ∗ ( X ) || 2 also follows some continuous distributionon (0 , + ∞ ) .
Revised: For X following continuous distribution P i , || v ∗ ( X ) || 2 also follows some continuous distributionon (0 , + ∞ ) .

Original: Since X and Y are i.i.d.
Revised: Since X and Y are i.i.d.

Original: , the right hand side of Eq.
Revised: , the right hand side of Eq.

Original: (17) goes to 0 as (cid:15) → 0 .
Revised: (17) goes to 0 as (cid:15) → 0 .

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: For any X ∼ P i , since v 0 satisﬁes f ( X + v 0 ) (cid:54) = f
Revised: For any X ∼ P i , since v 0 satisﬁes f ( X + v 0 ) (cid:54) = f

Original: ( X ) and
Revised: ( X ) and

Original: [Equation]
Revised: [Equation]

Original: 18) v 0 ∈ V (cid:15) ( X ) by Deﬁnition 3.1.
Revised: 18) v 0 ∈ V (cid:15) ( X ) by Deﬁnition 3.1.

Original: Thus, v 0 ∈ V (cid:15) ( X ) ∩ V (cid:15) ( Y ) for X and Y i.i.d.
Revised: Thus, v 0 ∈ V (cid:15) ( X ) ∩ V (cid:15) ( Y ) for X and Y i.i.d.

Original: following P i ; and
Revised: following P i ; and

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: V
Revised: V

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thus, based on Step 1 (and Eq.
Revised: Thus, based on Step 1 (and Eq.

Original: (17)) in Apdx A.2, we have P MT
Revised: (17)) in Apdx A.2, we have P MT

Original: = 1 .
Revised: = 1 .

Original: Since P MT + P NT ≤ 1 , Eq.
Revised: Since P MT + P NT ≤ 1 , Eq.

Original: (6) can be written as
Revised: (6) can be written as

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: B A NALYSIS ON A S IMPLIFIED C LASSIFICATION P ROBLEM
Revised: B A NALYSIS ON A S IMPLIFIED C LASSIFICATION P ROBLEM

Original: In this section, we consider a simpliﬁed analogue to practical 2-class classiﬁcation problems.
Revised: In this section, we consider a simpliﬁed analogue to practical 2-class classiﬁcation problems.

Original: Although assumptions are imposed for simplicity, we still keep the problem relatively general by allowing freedom on, e.g., the sample distribution in their latent space.
Revised: Although assumptions are imposed for simplicity, we still keep the problem relatively general by allowing freedom on, e.g., the sample distribution in their latent space.

Original: With the simpliﬁcation, we are able to analytically derive the condition for a sample belonging to the transferable set of another sample from the same class.
Revised: With the simpliﬁcation, we are able to analytically derive the condition for a sample belonging to the transferable set of another sample from the same class.

Original: Based on this, we show that 12 is the supremum of the ET statistic for this problem when there is no attack.
Revised: Based on this, we show that 12 is the supremum of the ET statistic for this problem when there is no attack.

Original: Note that we will abuse some notations that appeared in the main paper; the notations in this section are all self-contained .
Revised: Note that we will abuse some notations that appeared in the main paper; the notations in this section are all self-contained .

Original: B. P ROBLEM S ETTINGS
Revised: B. P ROBLEM S ETTINGS

Original: We consider sample space R n with an orthonormal basis { α 1 , · · · , α d , β 1 , · · · , β n − d } .
Revised: We consider sample space R n with an orthonormal basis { α 1 , · · · , α d , β 1 , · · · , β n − d } .

Original: We assume that samples from the two classes are distributed on sub-spaces V 

Revised: We assume that samples from the two classes are distributed on sub-spaces V

Original: = { Ac | c ∈ R d } and V 1 = { Be | e ∈ R n − d } respectively, with A =
Revised: = { Ac | c ∈ R d } and V 1 = { Be | e ∈ R n − d } respectively, with A =

Original: [ α 1 , · · · , α d ] and B = [ β 1 , · · · , β n − d ] .
Revised: [ α 1 , · · · , α d ] and B = [ β 1 , · · · , β n − d ] .

Original: By the deﬁnition of V 0 and V 1 , we have also deﬁned the latent spaces , i.e. R d and R n − d , for the two classes.
Revised: By the deﬁnition of V 0 and V 1 , we have also deﬁned the latent spaces , i.e. R d and R n − d , for the two classes.

Original: Here, we do not constrain the form or parameters of the distributions for the two classes in both the sample space R n and latent spaces, as long as the distributions are continuous.
Revised: Here, we do not constrain the form or parameters of the distributions for the two classes in both the sample space R n and latent spaces, as long as the distributions are continuous.

Original: Such an analogue may be corresponding to some simple classiﬁcation domains in practice.
Revised: Such an analogue may be corresponding to some simple classiﬁcation domains in practice.

Original: Moreover, the latent space may be corresponding to an internal layer space of some deep neural network (DNN) classiﬁer.
Revised: Moreover, the latent space may be corresponding to an internal layer space of some deep neural network (DNN) classiﬁer.

Original: For example, for a typical ReLU DNN classiﬁer, it is possible that a subset of nodes in the penultimate layer are mainly activated for one class, while another subset of nodes are mainly activated for the other class.
Revised: For example, for a typical ReLU DNN classiﬁer, it is possible that a subset of nodes in the penultimate layer are mainly activated for one class, while another subset of nodes are mainly activated for the other class.

Original: For this simpliﬁed domain, we consider a nearest prototype classiﬁer that is capable of classifying the two classes perfectly for any continuous sample distributions.
Revised: For this simpliﬁed domain, we consider a nearest prototype classiﬁer that is capable of classifying the two classes perfectly for any continuous sample distributions.

Original: To achieve this, any point x ∈ R n on the decision boundary of the classiﬁer should have equal distance to V 0 and V 1 , i.e.:
Revised: To achieve this, any point x ∈ R n on the decision boundary of the classiﬁer should have equal distance to V 0 and V 1 , i.e.:

Original: [Equation]
Revised: [Equation]

Original: By expanding both sides of Eq.
Revised: By expanding both sides of Eq.

Original: (22) and
Revised: (22) and

Original: rearranging terms (using the fact that A T A
Revised: rearranging terms (using the fact that A T A

Original: =
Revised: =

Original: I and
Revised: I and

Original: B T B = I ), we obtain the decision boundary of the classiﬁer, which is { x ∈ R n | x ( AA T −
Revised: B T B = I ), we obtain the decision boundary of the classiﬁer, which is { x ∈ R n | x ( AA T −

Original: BB T ) x
Revised: BB T ) x

Original: = 0 } .
Revised: = 0 } .

Original: In other words, the classiﬁer f : R n → { 0 , 1 } is deﬁned by
Revised: In other words, the classiﬁer f : R n → { 0 , 1 } is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Note that this classiﬁer is not necessarily linear; and the region for each class may not be convex.
Revised: Note that this classiﬁer is not necessarily linear; and the region for each class may not be convex.

Original: B. D ERIVATION OF T RANSFER C ONDITION
Revised: B. D ERIVATION OF T RANSFER C ONDITION

Original: We derive the condition that one sample belongs to the transferable set of another sample from the same class.
Revised: We derive the condition that one sample belongs to the transferable set of another sample from the same class.

Original: Since the two classes are symmetric, we focus on class 0 for brevity.
Revised: Since the two classes are symmetric, we focus on class 0 for brevity.

Original: Moreover, in this section, we refer to samples by their latent space representation; i.e., instead of “a sample x ∈ R n from class 0”, we use “a sample c ∈ R d ”.
Revised: Moreover, in this section, we refer to samples by their latent space representation; i.e., instead of “a sample x ∈ R n from class 0”, we use “a sample c ∈ R d ”.

Original: First, we present the following modiﬁed deﬁnition of the transferable set (compared with Def. 3.
Revised: First, we present the following modiﬁed deﬁnition of the transferable set (compared with Def. 3.

Original: in the main paper) using latent space representation.
Revised: in the main paper) using latent space representation.

Original: Deﬁnition B.1.
Revised: Deﬁnition B.1.

Original: (Transferable set in latent space)
Revised: (Transferable set in latent space)

Original: The transferable set for any sample c ∈ R d from class 0 is deﬁned by
Revised: The transferable set for any sample c ∈ R d from class 0 is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where v ∗ ( c ) is the optimal solution to
Revised: where v ∗ ( c ) is the optimal solution to

Original: [Equation]
Revised: [Equation]

Original: subject to f ( Ac +
Revised: subject to f ( Ac +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that the above deﬁnition is in similar form to Def. 3.2 in the main paper, though here, the quality to the solution to problem (25) is no longer considered.
Revised: Note that the above deﬁnition is in similar form to Def. 3.2 in the main paper, though here, the quality to the solution to problem (25) is no longer considered.

Original: This is because, different with problem (3) in the main paper (which is the prerequisite of Def. 3.2), problem (25) here can be solved analytically, yielding a close form solution instead of a solution set with some intrinsic quality bound (cid:15) .
Revised: This is because, different with problem (3) in the main paper (which is the prerequisite of Def. 3.2), problem (25) here can be solved analytically, yielding a close form solution instead of a solution set with some intrinsic quality bound (cid:15) .

Original: Accordingly, we present the following theorem, which gives the condition for one sample belonging to the transferable set of another sample from the same class.
Revised: Accordingly, we present the following theorem, which gives the condition for one sample belonging to the transferable set of another sample from the same class.

Original: Theorem B.1.
Revised: Theorem B.1.

Original: For any c , c (cid:48) ∈ R d , c (cid:48) ∈ T ( c ) if and only if || c (cid:48) − c / 2 || 2 ≤ || c / 2 || 2 .
Revised: For any c , c (cid:48) ∈ R d , c (cid:48) ∈ T ( c ) if and only if || c (cid:48) − c / 2 || 2 ≤ || c / 2 || 2 .

Original: Proof.
Revised: Proof.

Original: First, we derive the solution to problem (25).
Revised: First, we derive the solution to problem (25).

Original: Note that v ∈ R n can be decomposed (using the orthonormal basis speciﬁed by A and B ) as v =
Revised: Note that v ∈ R n can be decomposed (using the orthonormal basis speciﬁed by A and B ) as v =

Original: Av a + Bv b with v a ∈ R d and v b ∈ R n − d .
Revised: Av a + Bv b with v a ∈ R d and v b ∈ R n − d .

Original: We substitute this decomposition into the constraint of problem (25).
Revised: We substitute this decomposition into the constraint of problem (25).

Original: In words, the constraint means that v induces Ac to be (mis)classiﬁed to class 1; thus, according to the expression of the classiﬁer in Eq.
Revised: In words, the constraint means that v induces Ac to be (mis)classiﬁed to class 1; thus, according to the expression of the classiﬁer in Eq.

Original: (23), the constraint can be written as
Revised: (23), the constraint can be written as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Expanding the left hand side of the above inequality and using the fact that A T B = 0 for simpliﬁcation, we get a much simpler expression of the constraint:
Revised: Expanding the left hand side of the above inequality and using the fact that A T B = 0 for simpliﬁcation, we get a much simpler expression of the constraint:

Original: [Equation]
Revised: [Equation]

Original: Thus, a lower bound of the (square of the) objective to be minimized in problem (25) can be derived as
Revised: Thus, a lower bound of the (square of the) objective to be minimized in problem (25) can be derived as

Original: [Equation]
Revised: [Equation]

Original: with equality holds if and only if || v b || 

Revised: with equality holds if and only if || v b ||

Original: = || v a + c || , i.e. Ac + v on the decision boundary of the classiﬁer.
Revised: = || v a + c || , i.e. Ac + v on the decision boundary of the classiﬁer.

Original: Note that the right hand side of the inequality above is minimized when v a = − c 2 .
Revised: Note that the right hand side of the inequality above is minimized when v a = − c 2 .

Original: Then, the optimal solution v ∗ ( c ) =
Revised: Then, the optimal solution v ∗ ( c ) =

Original: Av ∗ a ( c ) + Bv ∗ b ( c ) to problem (25) satisﬁes:
Revised: Av ∗ a ( c ) + Bv ∗ b ( c ) to problem (25) satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Next, for any c , c (cid:48) ∈ R d , we derive the condition for c (cid:48) ∈ T ( c ) .
Revised: Next, for any c , c (cid:48) ∈ R d , we derive the condition for c (cid:48) ∈ T ( c ) .

Original: Since f ( Ac (cid:48) )
Revised: Since f ( Ac (cid:48) )

Original: =
Revised: =

Original: f ( Ac ) is already satisﬁed, by Def.
Revised: f ( Ac ) is already satisﬁed, by Def.

Original: B.1, c (cid:48) ∈ T ( c )
Revised: B.1, c (cid:48) ∈ T ( c )

Original: if and only if f ( Ac (cid:48) + v ∗ ( c )) (cid:54) =
Revised: if and only if f ( Ac (cid:48) + v ∗ ( c )) (cid:54) =

Original: f ( Ac (cid:48) ) , which is equivalent to (by Eq.
Revised: f ( Ac (cid:48) ) , which is equivalent to (by Eq.

Original: [Equation]
Revised: [Equation]

Original: Using the decomposition of v ∗ ( c ) , expanding and rearranging terms on the left hand side of the above, we obtain
Revised: Using the decomposition of v ∗ ( c ) , expanding and rearranging terms on the left hand side of the above, we obtain

Original: [Equation]
Revised: [Equation]

Original: With the optimal solution in Eq.
Revised: With the optimal solution in Eq.

Original: (29) substituted in, we get
Revised: (29) substituted in, we get

Original: [Equation]
Revised: [Equation]

Original: or, equivalently
Revised: or, equivalently

Original: [Equation]
Revised: [Equation]

Original: B. U PPER B OUND ON ET S TATISTIC
Revised: B. U PPER B OUND ON ET S TATISTIC

Original: In this section, we show that 12 is the minimum upper bound on the ET statistic for this problem when there is no BA.
Revised: In this section, we show that 12 is the minimum upper bound on the ET statistic for this problem when there is no BA.

Original: To do so, we ﬁrst present a deﬁnition of ET statistic modiﬁed from Def 3.3 in the main paper, merely in adaption to the latent space representation used here (see Def.
Revised: To do so, we ﬁrst present a deﬁnition of ET statistic modiﬁed from Def 3.3 in the main paper, merely in adaption to the latent space representation used here (see Def.

Original: B.2).
Revised: B.2).

Original: Then, we show the tightness of the bound by giving a concrete example where ET equals 12 in Lem.
Revised: Then, we show the tightness of the bound by giving a concrete example where ET equals 12 in Lem.

Original: B.1.
Revised: B.1.

Original: Finally, we prove that the ET statistic cannot be greater than 12 .
Revised: Finally, we prove that the ET statistic cannot be greater than 12 .

Original: Deﬁnition B.2.
Revised: Deﬁnition B.2.

Original: For i.i.d.
Revised: For i.i.d.

Original: random samples C and C (cid:48)
Revised: random samples C and C (cid:48)

Original: following some continuous distribution G on R d , the ET statistic is deﬁned by
Revised: following some continuous distribution G on R d , the ET statistic is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Lemma B.1.
Revised: Lemma B.1.

Original: For latent space R d with dimension d = 1 and distribution G continuous on R , the ET statistic satisﬁes
Revised: For latent space R d with dimension d = 1 and distribution G continuous on R , the ET statistic satisﬁes

Original: [Equation]
Revised: [Equation]

Original: ) where ET = 12 if and only if G (0) = 0 or G (0)
Revised: ) where ET = 12 if and only if G (0) = 0 or G (0)

Original: = 1 .
Revised: = 1 .

Original: Proof.
Revised: Proof.

Original: Based on Thm.
Revised: Based on Thm.

Original: B.1, for latent space dimension d = 1 and two scalar 5 i.i.d.
Revised: B.1, for latent space dimension d = 1 and two scalar 4 i.i.d.

Original: random samples
Revised: random samples

Original: C and C (cid:48) with continuous distribution G , C (cid:48) ∈ T ( C )
Revised: C and C (cid:48) with continuous distribution G , C (cid:48) ∈ T ( C )

Original: if and only if | C (cid:48) − C/ 2 | ≤ | C/ 2 | .
Revised: if and only if | C (cid:48) − C/ 2 | ≤ | C/ 2 | .

Original: Thus, by
Revised: Thus, by

Original: Def.
Revised: Def.

Original: B.2, we have
Revised: B.2, we have

Original: [Equation]
Revised: [Equation]

Original: where g ( · ) is the density function of distribution G .
Revised: where g ( · ) is the density function of distribution G .

Original: Note that the last line of Eq.
Revised: Note that the last line of Eq.

Original: (36) is strictly in the interval [ 14 , 12 ] for G ( · ) in range [0 , 1] .
Revised: (36) is strictly in the interval [ 14 , 12 ] for G ( · ) in range [0 , 1] .

Original: The upper bound of ET when d = 1 is 12 , which is achieved if and only if G (0) = 0 or G (0)
Revised: The upper bound of ET when d = 1 is 12 , which is achieved if and only if G (0) = 0 or G (0)

Original: = 1 .
Revised: = 1 .

Original: Theorem B.2.
Revised: Theorem B.2.

Original: For arbitrary d ∈ Z + and continuous distribution G on R d
Revised: For arbitrary d ∈ Z + and continuous distribution G on R d

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lem.
Revised: By Lem.

Original: B.1, there exists d = 1 and G (0) = 0 or G (0)
Revised: B.1, there exists d = 1 and G (0) = 0 or G (0)

Original: = 1 , such that ET = 12 .
Revised: = 1 , such that ET = 12 .

Original: Here, we only need to show that ET ≤ 12 for any d and G .
Revised: Here, we only need to show that ET ≤ 12 for any d and G .

Original: Again, by the deﬁnition of ET (Def. B.2) and
Revised: Again, by the deﬁnition of ET (Def. B.2) and

Original: Thm.
Revised: Thm.

Original: B.1, we can write ET as
Revised: B.1, we can write ET as

Original: [Equation]
Revised: [Equation]

Original: (38) for C and C (cid:48) i.i.d.
Revised: (38) for C and C (cid:48) i.i.d.

Original: following distribution G .
Revised: following distribution G .

Original: Similar to our proof of Thm.
Revised: Similar to our proof of Thm.

Original: 3.1 in Sec.
Revised: 3.1 in Sec.

Original: A.
Revised: A.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thus, based on Eq.
Revised: Thus, based on Eq.

Original: (38) and (39), ET can be written as
Revised: (38) and (39), ET can be written as

Original: [Equation]
Revised: [Equation]

Original: Since C and C (cid:48) are i.i.d,
Revised: Since C and C (cid:48) are i.i.d,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Also note that for continuous distribution
Revised: Also note that for continuous distribution

Original: [Equation]
Revised: [Equation]

Original: Then, by the inclusion-exclusion rule
Revised: Then, by the inclusion-exclusion rule

Original: [Equation]
Revised: [Equation]

Original: Substituting Eq.
Revised: Substituting Eq.

Original: (41) and (42) into Eq.
Revised: (41) and (42) into Eq.

Original: (43), we have
Revised: (43), we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: thus, by Eq.
Revised: thus, by Eq.

Original: [Equation]
Revised: [Equation]

Original: (45) which ﬁnishes the proof.
Revised: (45) which ﬁnishes the proof.

Original: C U SING ET TO D ETECT BA WITH P ATCH R EPLACEMENT BP
Revised: C U SING ET TO D ETECT BA WITH P ATCH R EPLACEMENT BP

Original: In the main paper, we have described our detection framework in details considering additive perturbation BPs embedded by Eq.
Revised: In the main paper, we have described our detection framework in details considering additive perturbation BPs embedded by Eq.

Original: In fact, our detection framework is not speciﬁc to any particular backdoor embedding mechanism.
Revised: In fact, our detection framework is not speciﬁc to any particular backdoor embedding mechanism.

Original: In this section, we repeat our derivation and analysis in Sec. 3 by considering patch replacement BPs embedded by Eq.
Revised: In this section, we repeat our derivation and analysis in Sec. 3 by considering patch replacement BPs embedded by Eq.

Original: Basically, the deﬁnitions, theorems and algorithms presented in this section are matched with those in the main paper – they are under the same framework which is generally independent of the backdoor pattern embedding mechanism.
Revised: Basically, the deﬁnitions, theorems and algorithms presented in this section are matched with those in the main paper – they are under the same framework which is generally independent of the backdoor pattern embedding mechanism.

Original: Like the organization of Sec. 3, in this section, we ﬁrst introduce several deﬁnitions related to ET but for patch replacement BPs – these deﬁnitions are similar to those in the main paper and are customized to patch replacement BPs.
Revised: Like the organization of Sec. 3, in this section, we ﬁrst introduce several deﬁnitions related to ET but for patch replacement BPs – these deﬁnitions are similar to those in the main paper and are customized to patch replacement BPs.

Original: Especially, the ET statistic is deﬁned in the same fashion as in Def.
Revised: Especially, the ET statistic is deﬁned in the same fashion as in Def.

Original: 3.3 of the main paper.
Revised: 3.3 of the main paper.

Original: Then, we show that the same constant detection threshold 12 on the ET statistic for detecting BAs with with additive perturbation BPs can be used for detecting BAs with patch replacement BPs as well.
Revised: Then, we show that the same constant detection threshold 12 on the ET statistic for detecting BAs with with additive perturbation BPs can be used for detecting BAs with patch replacement BPs as well.

Original: Finally, we present the detailed procedure of our detection for patch replacement BPs (as the counterpart of Alg.
Revised: Finally, we present the detailed procedure of our detection for patch replacement BPs (as the counterpart of Alg.

Original: 1 of the main paper).
Revised: 1 of the main paper).

Original: Again, this section can be viewed as an independent section, where the notations are self-contained.
Revised: Again, this section can be viewed as an independent section, where the notations are self-contained.

Original: C. D EFINITION OF ET FOR P ATCH R EPLACEMENT BP
Revised: C. D EFINITION OF ET FOR P ATCH R EPLACEMENT BP

Original: Consider the same classiﬁer f : X → C to be inspected (as in the main paper) with the same label space C
Revised: Consider the same classiﬁer f : X → C to be inspected (as in the main paper) with the same label space C

Original: = { 0 , 1 } and continuous sample distribution P i on X for class i ∈ C .
Revised: = { 0 , 1 } and continuous sample distribution P i on X for class i ∈ C .

Original: For any sample x from any class, the optimal solution to minimize s
Revised: For any sample x from any class, the optimal solution to minimize s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: || m ||subject to f ( ∆ ( x ; m , u )) (cid:54) =
Revised: || m ||subject to f ( ∆ ( x ; m , u )) (cid:54) =

Original: f
Revised: f

Original: ( x ) (46) is deﬁned as s ∗ ( x ) = { m ∗ ( x ) , u ∗ ( x ) } , where ∆ ( x ; m , u ) =
Revised: ( x ) (46) is deﬁned as s ∗ ( x ) = { m ∗ ( x ) , u ∗ ( x ) } , where ∆ ( x ; m , u ) =

Original: (1 − m ) (cid:12) x
Revised: (1 − m ) (cid:12) x

Original: + m (cid:12) u is an alternative expression to the patch replacement embedding formula in Eq.
Revised: + m (cid:12) u is an alternative expression to the patch replacement embedding formula in Eq.

Original: (2) (for notational convenience only).
Revised: (2) (for notational convenience only).

Original: Again, m ∈ M is the binary mask and u ∈ X is the patch for replacement.
Revised: Again, m ∈ M is the binary mask and u ∈ X is the patch for replacement.

Original: Existing methods for solving problem (46) including the one proposed by Wang et al.
Revised: Existing methods for solving problem (46) including the one proposed by Wang et al.

Original: (2019), where, again, the practical solutions are usually sub-optimal.
Revised: (2019), where, again, the practical solutions are usually sub-optimal.

Original: Thus, similar to Def.
Revised: Thus, similar to Def.

Original: 3.1 in the main paper, we present the following deﬁnition in adaption to patch replacement BPs.
Revised: 3.1 in the main paper, we present the following deﬁnition in adaption to patch replacement BPs.

Original: Deﬁnition C.1.
Revised: Deﬁnition C.1.

Original: ( (cid:15) -solution set for patch replacement BPs)
Revised: ( (cid:15) -solution set for patch replacement BPs)

Original: For any sample x from any class, regardless of the method being used, the (cid:15) -solution set to problem (46), is deﬁned by
Revised: For any sample x from any class, regardless of the method being used, the (cid:15) -solution set to problem (46), is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) > 0 is the “quality” bound of the solutions, which is usually small for existing methods.
Revised: where (cid:15) > 0 is the “quality” bound of the solutions, which is usually small for existing methods.

Original: Similar to Def.
Revised: Similar to Def.

Original: 3.2 in the main paper, we present the following deﬁnition (with abused notation) for the transferable set for patch replacement BPs.
Revised: 3.2 in the main paper, we present the following deﬁnition (with abused notation) for the transferable set for patch replacement BPs.

Original: Deﬁnition C.2.
Revised: Deﬁnition C.2.

Original: (Transferable set for patch replacement BPs)
Revised: (Transferable set for patch replacement BPs)

Original: The transferable set for any sample x and (cid:15) > 0 is deﬁned by
Revised: The transferable set for any sample x and (cid:15) > 0 is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Finally, we present the following deﬁnition of the ET statistic for patch replacement BPs, which looks exactly the same as Def. 3.3 in the main paper.
Revised: Finally, we present the following deﬁnition of the ET statistic for patch replacement BPs, which looks exactly the same as Def. 3.3 in the main paper.

Original: However, here, the deﬁnition of the transferable set has been customized for patch replacement BPs.
Revised: However, here, the deﬁnition of the transferable set has been customized for patch replacement BPs.

Original: Even though, the similarity between these deﬁnitions and their counterparts in the main paper has already highlighted the generalization capability of our detection framework.
Revised: Even though, the similarity between these deﬁnitions and their counterparts in the main paper has already highlighted the generalization capability of our detection framework.

Original: Deﬁnition C.3.
Revised: Deﬁnition C.3.

Original: (ET statistic for patch replacement BPs)
Revised: (ET statistic for patch replacement BPs)

Original: For any class i ∈ C
Revised: For any class i ∈ C

Original: = { 0 , 1 } and (cid:15) > 0 , considering i.i.d.
Revised: = { 0 , 1 } and (cid:15) > 0 , considering i.i.d.

Original: random samples X , Y ∼ P i , the ET statistic for class i is deﬁned by
Revised: random samples X , Y ∼ P i , the ET statistic for class i is deﬁned by

Original: ET i,(cid:15) (cid:44) E X ∼ P
Revised: ET i,(cid:15) (cid:44) E X ∼ P

Original: i (cid:2) P( Y ∈ T (cid:15) ( X ) | X ) (cid:3) .
Revised: i (cid:2) P( Y ∈ T (cid:15) ( X ) | X ) (cid:3) .

Original: C. D ETECTING BA WITH P ATCH R EPLACEMENT BP U SING ET
Revised: C. D ETECTING BA WITH P ATCH R EPLACEMENT BP U SING ET

Original: For the ET statistic for patch replacement BPs deﬁned above, the same constant detection threshold 12 can be used for distinguish BA target classes from non-target classes.
Revised: For the ET statistic for patch replacement BPs deﬁned above, the same constant detection threshold 12 can be used for distinguish BA target classes from non-target classes.

Original: The connection between the ET statistic for patch replacement BPs and the constant threshold 12 is established by the same
Revised: The connection between the ET statistic for patch replacement BPs and the constant threshold 12 is established by the same

Original: Thm. 3.1 in Sec.
Revised: Thm. 3.1 in Sec.

Original: 3.2 of the main paper with the same proof in Apdx.
Revised: 3.2 of the main paper with the same proof in Apdx.

Original: A.1.
Revised: A.1.

Original: Thus, these details are not included here for brevity.
Revised: Thus, these details are not included here for brevity.

Original: Note that P MT ,i and P NT ,i for class i are deﬁned in the same way as in Thm.
Revised: Note that P MT ,i and P NT ,i for class i are deﬁned in the same way as in Thm.

Original: 3.1, though the transferable set T (cid:15) ( x ) is customized for patch replacement BPs in the current section – this is the main reason why we abuse the notation for the transferable set for patch replacement BPs in Def. C.2.
Revised: 3.1, though the transferable set T (cid:15) ( x ) is customized for patch replacement BPs in the current section – this is the main reason why we abuse the notation for the transferable set for patch replacement BPs in Def. C.2.

Original: In the following, like in Sec.
Revised: In the following, like in Sec.

Original: 3.2 of the main paper, we discuss the non-attack case and the attack case respectively.
Revised: 3.2 of the main paper, we discuss the non-attack case and the attack case respectively.

Original: Readers should notice that the theorems (and the associated proofs) and discussions are similar to those in the main paper.
Revised: Readers should notice that the theorems (and the associated proofs) and discussions are similar to those in the main paper.

Original: Such similarity further highlight the generalization capability of our detection framework.
Revised: Such similarity further highlight the generalization capability of our detection framework.

Original: Non-attack case.
Revised: Non-attack case.

Original: Property 3.1 from the main paper is also applicable here.
Revised: Property 3.1 from the main paper is also applicable here.

Original: That is, if class (1 − i ) where i ∈ C
Revised: That is, if class (1 − i ) where i ∈ C

Original: = { 0 , 1 } is not a BA target class, P NT ,i for class i will likely be larger than 12 .
Revised: = { 0 , 1 } is not a BA target class, P NT ,i for class i will likely be larger than 12 .

Original: Similar to our veriﬁcation of Property 3.1 for additive perturbation BPs in Sec.
Revised: Similar to our veriﬁcation of Property 3.1 for additive perturbation BPs in Sec.

Original: 4.3 of the main paper, we verify Property 3.1 for patch replacement BPs in Apdx.
Revised: 4.3 of the main paper, we verify Property 3.1 for patch replacement BPs in Apdx.

Original: F. Using similar protocol as in Sec. 4.
Revised: F. Using similar protocol as in Sec. 4.

Original: of the main paper, we show that the common mask with the minimum-norm required for two sample to be misclassiﬁed usually has a larger norm than mask with the mimimum-norm required for each of them.
Revised: of the main paper, we show that the common mask with the minimum-norm required for two sample to be misclassiﬁed usually has a larger norm than mask with the mimimum-norm required for each of them.

Original: As for P MT ,i , again, Thm. 3.2 from the main paper is also applicable here, but with a slightly different proof (shown below) in adaption to the modiﬁcations to the deﬁnitions for the patch replacement
Revised: As for P MT ,i , again, Thm. 3.2 from the main paper is also applicable here, but with a slightly different proof (shown below) in adaption to the modiﬁcations to the deﬁnitions for the patch replacement

Original: BPs in Apdx.
Revised: BPs in Apdx.

Original: C.1.
Revised: C.1.

Original: Proof.
Revised: Proof.

Original: Step 1: Similar to the proof in Apdx.
Revised: Step 1: Similar to the proof in Apdx.

Original: A.2, We show that for any i ∈ C and X , Y i.i.d.
Revised: A.2, We show that for any i ∈ C and X , Y i.i.d.

Original: following distribution P i , X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15)
Revised: following distribution P i , X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15)

Original: ( X ) if and only if S (cid:15) ( X ) ∩ S (cid:15) ( Y ) (cid:54) = ∅ .
Revised: ( X ) if and only if S (cid:15) ( X ) ∩ S (cid:15) ( Y ) (cid:54) = ∅ .

Original: (a) If S (cid:15) ( X ) ∩ S (cid:15) ( Y ) (cid:54) = ∅ , there exists s such that s ∈ S (cid:15) ( X ) and s ∈ S (cid:15) ( Y ) .
Revised: (a) If S (cid:15) ( X ) ∩ S (cid:15) ( Y ) (cid:54) = ∅ , there exists s such that s ∈ S (cid:15) ( X ) and s ∈ S (cid:15) ( Y ) .

Original: By Def.
Revised: By Def.

Original: C.1,
Revised: C.1,

Original: s ∈ S (cid:15) ( X ) ⇒ f ( ∆ ( X ; m , u ))
Revised: s ∈ S (cid:15) ( X ) ⇒ f ( ∆ ( X ; m , u ))

Original: (cid:54) =
Revised: (cid:54) =

Original: f ( X ) (49) s ∈ S (cid:15) ( Y ) ⇒ f ( ∆ ( Y ; m , u ))
Revised: f ( X ) (49) s ∈ S (cid:15) ( Y ) ⇒ f ( ∆ ( Y ; m , u ))

Original: (cid:54) =
Revised: (cid:54) =

Original: f ( Y ) (50)
Revised: f ( Y ) (50)

Original: Then, by Deﬁnition C.2, the existence of such s yields X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15) ( X ) .
Revised: Then, by Deﬁnition C.2, the existence of such s yields X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15) ( X ) .

Original: (b) We prove the “only if” part also by contradiction.
Revised: (b) We prove the “only if” part also by contradiction.

Original: Given X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15) ( X ) , suppose
Revised: Given X ∈ T (cid:15) ( Y ) and Y ∈ T (cid:15) ( X ) , suppose

Original: S (cid:15) ( X ) ∩ S (cid:15) ( Y ) = ∅ .
Revised: S (cid:15) ( X ) ∩ S (cid:15) ( Y ) = ∅ .

Original: By Deﬁnition C.2, if Y ∈ T (cid:15) ( X ) , there exists s = { m , u } ∈ S (cid:15)
Revised: By Deﬁnition C.2, if Y ∈ T (cid:15) ( X ) , there exists s = { m , u } ∈ S (cid:15)

Original: ( X ) such that f ( ∆ ( Y ; m , u ))
Revised: ( X ) such that f ( ∆ ( Y ; m , u ))

Original: (cid:54) =
Revised: (cid:54) =

Original: f ( Y ) .
Revised: f ( Y ) .

Original: Since S (cid:15) ( X ) ∩ S (cid:15) ( Y ) = ∅ , such s / ∈ S (cid:15) ( Y ) ; hence, by Deﬁnition C.
Revised: Since S (cid:15) ( X ) ∩ S (cid:15) ( Y ) = ∅ , such s / ∈ S (cid:15) ( Y ) ; hence, by Deﬁnition C.

Original: || m || 1 − || m ∗ ( Y ) || 1 > (cid:15).
Revised: || m || 1 − || m ∗ ( Y ) || 1 > (cid:15).

Original: Similarly, there exists s (cid:48)
Revised: Similarly, there exists s (cid:48)

Original: = { m (cid:48) , u (cid:48) } ∈ S (cid:15) ( Y ) such that f ( ∆ ( X ; m (cid:48) , u (cid:48) ))
Revised: = { m (cid:48) , u (cid:48) } ∈ S (cid:15) ( Y ) such that f ( ∆ ( X ; m (cid:48) , u (cid:48) ))

Original: (cid:54) =
Revised: (cid:54) =

Original: f ( X ) and s (cid:48) / ∈ S (cid:15) ( X ) (since it is assumed that S (cid:15) ( X ) ∩ S (cid:15) ( Y ) = ∅ ); hence
Revised: f ( X ) and s (cid:48) / ∈ S (cid:15) ( X ) (since it is assumed that S (cid:15) ( X ) ∩ S (cid:15) ( Y ) = ∅ ); hence

Original: [Equation]
Revised: [Equation]

Original: (cid:15).
Revised: (cid:15).

Original: By Deﬁnition C.1, for all ˜ s = { ˜ m , ˜ u } ∈ S (cid:15) ( Y ) , || ˜ m
Revised: By Deﬁnition C.1, for all ˜ s = { ˜ m , ˜ u } ∈ S (cid:15) ( Y ) , || ˜ m

Original: || 1 − || m ∗ ( Y )
Revised: || 1 − || m ∗ ( Y )

Original: ||
Revised: ||

Original: 1 < (cid:15) .
Revised: 1 < (cid:15) .

Original: Thus, we have || m ||
Revised: Thus, we have || m ||

Original: 1 − || m ∗ ( Y ) || 1 > || ˜ m || 1 − || m ∗ ( Y ) ||(53) and accordingly || m
Revised: 1 − || m ∗ ( Y ) || 1 > || ˜ m || 1 − || m ∗ ( Y ) ||(53) and accordingly || m

Original: || 1 > ||
Revised: || 1 > ||

Original: ˜ m
Revised: ˜ m

Original: || 1 for all ˜ s ∈ S (cid:15) ( Y ) .
Revised: || 1 for all ˜ s ∈ S (cid:15) ( Y ) .

Original: Similarly, for all ˜ s (cid:48) = { ˜ m (cid:48) , ˜ u
Revised: Similarly, for all ˜ s (cid:48) = { ˜ m (cid:48) , ˜ u

Original: [Equation]
Revised: [Equation]

Original: } ∈ S
Revised: } ∈ S

Original: [Equation]
Revised: [Equation]

Original: ( X ) , we have
Revised: ( X ) , we have

Original: [Equation]
Revised: [Equation]

Original: and thus, || m (cid:48)
Revised: and thus, || m (cid:48)

Original: || 1 > ||
Revised: || 1 > ||

Original: ˜ m (cid:48)
Revised: ˜ m (cid:48)

Original: || 1 for all ˜ s (cid:48) ∈ S (cid:15)
Revised: || 1 for all ˜ s (cid:48) ∈ S (cid:15)

Original: ( X ) .
Revised: ( X ) .

Original: Clearly, there is a contradiction since there cannot exist an element in one set having a larger norm than all elements in another set while vise versa; and therefore, S (cid:15) ( X ) ∩ S (cid:15) ( Y ) (cid:54) = ∅ .
Revised: Clearly, there is a contradiction since there cannot exist an element in one set having a larger norm than all elements in another set while vise versa; and therefore, S (cid:15) ( X ) ∩ S (cid:15) ( Y ) (cid:54) = ∅ .

Original: Step 2: We show that the upper bound for P MT ,i goes to 0 when (cid:15) → 0 .
Revised: Step 2: We show that the upper bound for P MT ,i goes to 0 when (cid:15) → 0 .

Original: Note that by Deﬁnition C.1, S (cid:15) ( X ) ∩S
Revised: Note that by Deﬁnition C.1, S (cid:15) ( X ) ∩S

Original: [Equation]
Revised: [Equation]

Original: ( Y
Revised: ( Y

Original: [Equation]
Revised: [Equation]

Original: = ∅ only
Revised: = ∅ only

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)
Revised: (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: 1 −|| m ∗ ( Y ) |
Revised: 1 −|| m ∗ ( Y ) |

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12) ≤ (cid:15) ; also based on Step 1:
Revised: (cid:12)(cid:12) ≤ (cid:15) ; also based on Step 1:

Original: [Equation]
Revised: [Equation]

Original: For X following continuous distribution P i , m ∗ ( X ) also follows some continuous distribution on (0 , + ∞ ) .
Revised: For X following continuous distribution P i , m ∗ ( X ) also follows some continuous distribution on (0 , + ∞ ) .

Original: Since X and Y are i.i.d.
Revised: Since X and Y are i.i.d.

Original: , the right hand side of Eq.
Revised: , the right hand side of Eq.

Original: (55) goes to 0 as (cid:15) → 0 .
Revised: (55) goes to 0 as (cid:15) → 0 .

Original: Based on the above, we have reached the same conclusions for patch replacement BPs as in the main paper.
Revised: Based on the above, we have reached the same conclusions for patch replacement BPs as in the main paper.

Original: That is, for the non-attack case, we will likely have P NT ,i ≥ P MT ,i , and consequently,
Revised: That is, for the non-attack case, we will likely have P NT ,i ≥ P MT ,i , and consequently,

Original: ET i,(cid:15) ≤ 12 based on Thm. 3.1.
Revised: ET i,(cid:15) ≤ 12 based on Thm. 3.1.

Original: Attack case.
Revised: Attack case.

Original: For class i ∈ C = { 0 , 1 } , we consider a successful BA with target class (1 − i ) .
Revised: For class i ∈ C = { 0 , 1 } , we consider a successful BA with target class (1 − i ) .

Original: The
Revised: The

Original: BP used by the attacker is speciﬁed by s 0 = { m 0 , u 0 } with mask m 0 and patch u 0 .
Revised: BP used by the attacker is speciﬁed by s 0 = { m 0 , u 0 } with mask m 0 and patch u 0 .

Original: Thus, for any
Revised: Thus, for any

Original: X ∼ P
Revised: X ∼ P

Original: i , due to the success of the BA, f ( X ) =
Revised: i , due to the success of the BA, f ( X ) =

Original: i and f ( ∆ ( X ; m 0 , u 0 ))
Revised: i and f ( ∆ ( X ; m 0 , u 0 ))

Original: (cid:54) =
Revised: (cid:54) =

Original: f ( X ) .
Revised: f ( X ) .

Original: Similar to our discussion in the main paper, s 0 will likely be a common element in both S (cid:15) ( X ) and S (cid:15) ( Y ) for X ,
Revised: Similar to our discussion in the main paper, s 0 will likely be a common element in both S (cid:15) ( X ) and S (cid:15) ( Y ) for X ,

Original: Y i.i.d.
Revised: Y i.i.d.

Original: following P i .
Revised: following P i .

Original: For the same reason, in this case, the ET statistic will likely be greater than 12 .
Revised: For the same reason, in this case, the ET statistic will likely be greater than 12 .

Original: Similar to additive perturbation BPs, for patch replacement BPs, we also have a guarantee for a large
Revised: Similar to additive perturbation BPs, for patch replacement BPs, we also have a guarantee for a large

Original: ET statistic ( ET i,(cid:15)
Revised: ET statistic ( ET i,(cid:15)

Original: = 1 ) when the mask size of the BP used by the attacker is sufﬁciently small.
Revised: = 1 ) when the mask size of the BP used by the attacker is sufﬁciently small.

Original: Such property is summarized in the theorem below.
Revised: Such property is summarized in the theorem below.

Original: Theorem C.1.
Revised: Theorem C.1.

Original: If class (1 − i ) it the target class of a BA with patch replacement BP s 0 = { m 0 , u 0 } such that || m 

Revised: If class (1 − i ) it the target class of a BA with patch replacement BP s 0 = { m 0 , u 0 } such that || m

Original: ||
Revised: ||

Original: 1 ≤ (cid:15) , we will have P ( S (cid:15) ( X ) ∩ S (cid:15) ( Y ) (cid:54) = ∅ )
Revised: 1 ≤ (cid:15) , we will have P ( S (cid:15) ( X ) ∩ S (cid:15) ( Y ) (cid:54) = ∅ )

Original: = 1 for X and Y i.i.d.
Revised: = 1 for X and Y i.i.d.

Original: following P i ; and furthermore, ET i,(cid:15) = 1 .
Revised: following P i ; and furthermore, ET i,(cid:15) = 1 .

Original: Proof.
Revised: Proof.

Original: For any X ∼ P i , since s 0 = { m 0 , u 0 } satisﬁes f ( ∆ ( X ; m 0 , u 0 )) (cid:54) =
Revised: For any X ∼ P i , since s 0 = { m 0 , u 0 } satisﬁes f ( ∆ ( X ; m 0 , u 0 )) (cid:54) =

Original: f ( X ) , and also because
Revised: f ( X ) , and also because

Original: [Equation]
Revised: [Equation]

Original: (56) by Deﬁnition C.1, we have s 0 ∈ S (cid:15) ( X ) .
Revised: (56) by Deﬁnition C.1, we have s 0 ∈ S (cid:15) ( X ) .

Original: Thus, s 0 ∈ S (cid:15)
Revised: Thus, s 0 ∈ S (cid:15)

Original: ( X ) ∩ S (cid:15) ( Y ) for X and Y i.i.d.
Revised: ( X ) ∩ S (cid:15) ( Y ) for X and Y i.i.d.

Original: following
Revised: following

Original: P i ; and
Revised: P i ; and

Original: [Equation]
Revised: [Equation]

Original: The rest of the proof (showing that ET i,(cid:15)
Revised: The rest of the proof (showing that ET i,(cid:15)

Original: = 1 for this attack scenario) is exactly the same as the proof of Thm.
Revised: = 1 for this attack scenario) is exactly the same as the proof of Thm.

Original: 3.3 shown in Apdx.
Revised: 3.3 shown in Apdx.

Original: A.3, thus is neglected here.
Revised: A.3, thus is neglected here.

Original: [Table]
Revised: [Table]

Original: D ETECTION P ROCEDURE FOR P ATCH R EPLACEMENT BP
Revised: D ETECTION P ROCEDURE FOR P ATCH R EPLACEMENT BP

Original: The same procedure, i.e. Alg. 1 in the main paper, can be used for detecting BAs with patch replacement BP, with only the following two modiﬁcations.
Revised: The same procedure, i.e. Alg. 1 in the main paper, can be used for detecting BAs with patch replacement BP, with only the following two modiﬁcations.

Original: We only need to ﬁrst replace line 8 of Alg.by: “Obtain an empirical solution ˆ s ( x ( i ) n )
Revised: We only need to ﬁrst replace line 8 of Alg.by: “Obtain an empirical solution ˆ s ( x ( i ) n )

Original: =
Revised: =

Original: { ˆ m ( x ( i ) n ) , ˆ u ( x ( i ) n ) } to problem (46) using random initialization.”
Revised: { ˆ m ( x ( i ) n ) , ˆ u ( x ( i ) n ) } to problem (46) using random initialization.”

Original: and then change line 9 of Alg.
Revised: and then change line 9 of Alg.

Original: 1 to: “
Revised: 1 to: “

Original: [Equation]
Revised: [Equation]

Original: ∆
Revised: ∆

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = f
Revised: = f

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: Such a simple “module-based” modiﬁcation allows our detection framework to be applicable to a variety of BP embedding mechanisms, which again, shows the generalization capability of our detection framework.
Revised: Such a simple “module-based” modiﬁcation allows our detection framework to be applicable to a variety of BP embedding mechanisms, which again, shows the generalization capability of our detection framework.

Original: D D ETAILS OF E XPERIMENT S ETTINGS D. D ETAILS OF D ATASETS
Revised: D D ETAILS OF E XPERIMENT S ETTINGS D. D ETAILS OF D ATASETS

Original: Our experiments are conducted on six popular benchmark image datasets.
Revised: Our experiments are conducted on six popular benchmark image datasets.

Original: They are CIFAR-10,
Revised: They are CIFAR-10,

Original: CIFAR-100 Krizhevsky (2012), STL-10 Coates et al.
Revised: CIFAR-100 Krizhevsky (2012), STL-10 Coates et al.

Original: (2011), TinyImageNet, FMNIST Xiao et al.
Revised: (2011), TinyImageNet, FMNIST Xiao et al.

Original: (2017) and MNIST Lecun et al.
Revised: (2017) and MNIST Lecun et al.

Original: All the datasets are associated with the torchvision package, except for that STL-10 is downloaded from the ofﬁcial website https://cs.stanford.
Revised: All the datasets are associated with the torchvision package, except for that STL-10 is downloaded from the ofﬁcial website https://cs.stanford.

Original: edu/˜acoates/stl10/ .
Revised: edu/˜acoates/stl10/ .

Original: Though the details of these datasets can be easily found online, we summarize them in Tab. 3.
Revised: Though the details of these datasets can be easily found online, we summarize them in Tab. 3.

Original: D. D ETAILS FOR G ENERATING THE 2-C LASS D OMAINS
Revised: D. D ETAILS FOR G ENERATING THE 2-C LASS D OMAINS

Original: In Sec. 4.1, we generate 45 2-class domains from CIFAR-10, and 20 2-class domains from each of CIFAR-100, STL-10, TinyImageNet, FMNIST, and MNIST.
Revised: In Sec. 4.1, we generate 45 2-class domains from CIFAR-10, and 20 2-class domains from each of CIFAR-100, STL-10, TinyImageNet, FMNIST, and MNIST.

Original: Here we provide more details about how these 2-class domains are generated.
Revised: Here we provide more details about how these 2-class domains are generated.

Original: As mentioned in Sec. 4.1, for CIFAR-10, the 45 2-class domains are corresponding to the 45 unordered class pairs of CIFAR-10 respectively.
Revised: As mentioned in Sec. 4.1, for CIFAR-10, the 45 2-class domains are corresponding to the 45 unordered class pairs of CIFAR-10 respectively.

Original: For each of CIFAR-100, FMNIST, and MNIST, we randomly sample 20 unordered class pairs, each forming a 2-class domain.
Revised: For each of CIFAR-100, FMNIST, and MNIST, we randomly sample 20 unordered class pairs, each forming a 2-class domain.

Original: For TinyImageNet, due to high image resolution and data scarcity, we generate 20 “super class” pairs – for each pair, we randomly sample 20 classes from the original category space and then evenly assign them to the two super classes (each getting 10 classes from the original category space).
Revised: For TinyImageNet, due to high image resolution and data scarcity, we generate 20 “super class” pairs – for each pair, we randomly sample 20 classes from the original category space and then evenly assign them to the two super classes (each getting 10 classes from the original category space).

Original: Similarly, for STL-10 with 10 classes, we generate 20 super class pairs by randomly and evenly dividing the 10 classes into two groups (of 5 classes from the original category space) for each pair.
Revised: Similarly, for STL-10 with 10 classes, we generate 20 super class pairs by randomly and evenly dividing the 10 classes into two groups (of 5 classes from the original category space) for each pair.

Original: For each generated 2-class domain, we use the subset of data associated with these two (super) classes from the original dataset, with the original train-test split.
Revised: For each generated 2-class domain, we use the subset of data associated with these two (super) classes from the original dataset, with the original train-test split.

Original: D. D ETAILS OF BP S
Revised: D. D ETAILS OF BP S

Original: In this paper, we consider both additive perturbation BPs embedded by Eq.
Revised: In this paper, we consider both additive perturbation BPs embedded by Eq.

Original: (1) and patch replacement BPs embedded by Eq.
Revised: (1) and patch replacement BPs embedded by Eq.

Original: (2) that are frequently used in existing backdoor papers.
Revised: (2) that are frequently used in existing backdoor papers.

Original: Despite the BPs (with images embedded with them) illustrated in Fig.
Revised: Despite the BPs (with images embedded with them) illustrated in Fig.

Original: 1 in the main paper, here, in Fig.
Revised: 1 in the main paper, here, in Fig.

Original: 5, we show
Revised: 5, we show

Original: [Figure]
Revised: [Figure]

Original: the following we also provide details for each of these BPs (in both Fig.
Revised: the following we also provide details for each of these BPs (in both Fig.

Original: 1 and Fig. 5).
Revised: 1 and Fig. 5).

Original: First, we provide the details for all the additive perturbation BPs.
Revised: First, we provide the details for all the additive perturbation BPs.

Original: The “chessboard” pattern in Fig.
Revised: The “chessboard” pattern in Fig.

Original: 1a is a “global” pattern that has been used by Xiang et al.
Revised: 1a is a “global” pattern that has been used by Xiang et al.

Original: Here, one and only one of two adjacent pixels are perturbed positively by 3/255 in all color channels.
Revised: Here, one and only one of two adjacent pixels are perturbed positively by 3/255 in all color channels.

Original: Another global pattern is the “static” pattern in Fig.
Revised: Another global pattern is the “static” pattern in Fig.

Original: 5a considered by both Zhong et al.
Revised: 5a considered by both Zhong et al.

Original: For pixel indices starting from 0, a pixel ( i, j ) is perturbed positively if and only if i and j are both even numbers.
Revised: For pixel indices starting from 0, a pixel ( i, j ) is perturbed positively if and only if i and j are both even numbers.

Original: Again, the perturbation size is 3/255 for all pixels being perturbed.
Revised: Again, the perturbation size is 3/255 for all pixels being perturbed.

Original: Other additive perturbation BPs are all “localized” patterns.
Revised: Other additive perturbation BPs are all “localized” patterns.

Original: The “L” pattern in Fig.
Revised: The “L” pattern in Fig.

Original: 1b and the “X” pattern in Fig.
Revised: 1b and the “X” pattern in Fig.

Original: 1c have been used by both Tran et al.
Revised: 1c have been used by both Tran et al.

Original: (2018) and Wang et al.
Revised: (2018) and Wang et al.

Original: For the “L” pattern, we perturb all the color channels by 40/255.
Revised: For the “L” pattern, we perturb all the color channels by 40/255.

Original: For the “X” pattern, for each attack, we randomly choose a channel (for all images to be embedded in for this particular attack) and perturb the associated pixels positively by 40/255.
Revised: For the “X” pattern, for each attack, we randomly choose a channel (for all images to be embedded in for this particular attack) and perturb the associated pixels positively by 40/255.

Original: The “pixel” BP in Fig.
Revised: The “pixel” BP in Fig.

Original: 1d has been used by Tran et al.
Revised: 1d has been used by Tran et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2018), where a single pixel is perturbed in all channels by 40/255 for color images and 70/255 for gray-scale images.
Revised: (2018), where a single pixel is perturbed in all channels by 40/255 for color images and 70/255 for gray-scale images.

Original: The “chessboard patch” pattern in Fig.
Revised: The “chessboard patch” pattern in Fig.

Original: 5b, the “cross” in Fig.
Revised: 5b, the “cross” in Fig.

Original: 5c, and the “square” in Fig.
Revised: 5c, and the “square” in Fig.

Original: 5d have all been previously considered.
Revised: 5d have all been previously considered.

Original: For the cross and the chessboard patch, the perturbation sizes for each pixel being perturbed are 40/255 and 5/255, respectively; and perturbation is applied to all channels.
Revised: For the cross and the chessboard patch, the perturbation sizes for each pixel being perturbed are 40/255 and 5/255, respectively; and perturbation is applied to all channels.

Original: For the square pattern, one channel is randomly selected for each attack and the perturbation size is 40/255.
Revised: For the square pattern, one channel is randomly selected for each attack and the perturbation size is 40/255.

Original: The spatial location of all these localized patterns are randomly selected over the entire image (and ﬁxed for all images to be embedded in) for each attack.
Revised: The spatial location of all these localized patterns are randomly selected over the entire image (and ﬁxed for all images to be embedded in) for each attack.

Original: Only for gray-scale images, the pixels being perturbed are restricted to one of the four corners, such that these pixels will likely be black (with pixel value close to 0) originally.
Revised: Only for gray-scale images, the pixels being perturbed are restricted to one of the four corners, such that these pixels will likely be black (with pixel value close to 0) originally.

Original: Next, we provide the details for the two patch replacement BPs considered in our experiments.
Revised: Next, we provide the details for the two patch replacement BPs considered in our experiments.

Original: The BP in Fig.
Revised: The BP in Fig.

Original: 1e is a small, monochromatic patch located near the margin of the images to be embedded in.
Revised: 1e is a small, monochromatic patch located near the margin of the images to be embedded in.

Original: Similar BPs have been considered by Gu et al.
Revised: Similar BPs have been considered by Gu et al.

Original: (2019) and Wang et al.
Revised: (2019) and Wang et al.

Original: The color is randomly chosen and ﬁxed for each attack.
Revised: The color is randomly chosen and ﬁxed for each attack.

Original: The BP in Fig.
Revised: The BP in Fig.

Original: 1f is a small noisy patch located near the margin of the images to be embedded in.
Revised: 1f is a small noisy patch located near the margin of the images to be embedded in.

Original: Similar BPs have been considered by Turner et al.
Revised: Similar BPs have been considered by Turner et al.

Original: (2019) and A. Saha (2020).
Revised: (2019) and A. Saha (2020).

Original: For both BPs, once the location is selected, the same location will be applied to all images to be embedded in for the same attack.
Revised: For both BPs, once the location is selected, the same location will be applied to all images to be embedded in for the same attack.

Original: Also, for both BPs, the size of the patch is 3 × 3 for the 2-class domains generated from CIFAR-10 and CIFAR-100; 4 × 4 for the 2-class domains generated from TinyImageNet; and 10 × 10 for the 2-class domains generated from
Revised: Also, for both BPs, the size of the patch is 3 × 3 for the 2-class domains generated from CIFAR-10 and CIFAR-100; 4 × 4 for the 2-class domains generated from TinyImageNet; and 10 × 10 for the 2-class domains generated from

Original: STL-10.
Revised: STL-10.

Original: D. O THER A TTACK C ONFIGURATIONS
Revised: D. O THER A TTACK C ONFIGURATIONS

Original: In the main paper, we deﬁned a “code” for each ensemble of attack instances based on both the type of BP being used and the dataset from which the associated 2-class domains are generated.
Revised: In the main paper, we deﬁned a “code” for each ensemble of attack instances based on both the type of BP being used and the dataset from which the associated 2-class domains are generated.

Original: Here, we summarize these codes in Tab. 4 for reference.
Revised: Here, we summarize these codes in Tab. 4 for reference.

Original: Also in the main paper, we have described most of the attack conﬁgurations for the attacks instances in each ensemble.
Revised: Also in the main paper, we have described most of the attack conﬁgurations for the attacks instances in each ensemble.

Original: For some ensembles, each attack instance is associated with two attacks, each with one of the two class being the target class.
Revised: For some ensembles, each attack instance is associated with two attacks, each with one of the two class being the target class.

Original: Also, the BPs used by the two attacks, though of the same type, are guaranteed to be sufﬁciently different in shapes (for additive perturbation BP) or colors (for patch replacement BP).
Revised: Also, the BPs used by the two attacks, though of the same type, are guaranteed to be sufﬁciently different in shapes (for additive perturbation BP) or colors (for patch replacement BP).

Original: For other
Revised: For other

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ensembles, there is only one attack for each attack instance.
Revised: ensembles, there is only one attack for each attack instance.

Original: Here, we provide a summary for these conﬁgurations in Tab. 5.
Revised: Here, we provide a summary for these conﬁgurations in Tab. 5.

Original: Also, in Tab 5, for attack instances in each ensemble, we summarize the number of training samples embedded with the BP and labeled to the target class that are poisoning the classiﬁer’s training set for the associated
Revised: Also, in Tab 5, for attack instances in each ensemble, we summarize the number of training samples embedded with the BP and labeled to the target class that are poisoning the classiﬁer’s training set for the associated

Original: D. T RAINING D ETAILS
Revised: D. T RAINING D ETAILS

Original: Here we provide the training details that are not included in the main paper due to space limitations.
Revised: Here we provide the training details that are not included in the main paper due to space limitations.

Original: For each generated 2-class domain, we use the same training conﬁguration irrespective of the existence of BA.
Revised: For each generated 2-class domain, we use the same training conﬁguration irrespective of the existence of BA.

Original: In Tab. 6, we show the training details including learning rate, batch size, number of epochs, whether or not using training data augmentation, choice of optimizer (Adam
Revised: In Tab. 6, we show the training details including learning rate, batch size, number of epochs, whether or not using training data augmentation, choice of optimizer (Adam

Original: D. P. Kingma (2015) or stochastic gradient descent (SGD))
Revised: D. P. Kingma (2015) or stochastic gradient descent (SGD))

Original: for 2-class domains generated from
Revised: for 2-class domains generated from

Original: CIFAR-10, CIFAR-100, STL-10, TinyImageNet, FMNIST, and MNIST, respectively.
Revised: CIFAR-10, CIFAR-100, STL-10, TinyImageNet, FMNIST, and MNIST, respectively.

Original: Training data augmentations for 2-class domains generated from TinyImageNet include random cropping and random horizontal ﬂipping – these augmentations are helpful for the backdoor mapping to be learned without compromising the classiﬁer’s accuracy on clean test samples.
Revised: Training data augmentations for 2-class domains generated from TinyImageNet include random cropping and random horizontal ﬂipping – these augmentations are helpful for the backdoor mapping to be learned without compromising the classiﬁer’s accuracy on clean test samples.

Original: Otherwise, we may not easily produce an effective attack to evaluate the performance of our defense.
Revised: Otherwise, we may not easily produce an effective attack to evaluate the performance of our defense.

Original: We also show the effectiveness of the attacks we created for evaluating our defense.
Revised: We also show the effectiveness of the attacks we created for evaluating our defense.

Original: Commonly, the effectiveness of a BA is evaluated by attack success rate (ASR) and clean test accuracy (ACC)
Revised: Commonly, the effectiveness of a BA is evaluated by attack success rate (ASR) and clean test accuracy (ACC)

Original: Xiang et al.
Revised: Xiang et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ASR is deﬁned (for each attack) as the probability that a test image from the source class is (mis)classiﬁed to the target class of BA when the BP is embedded.
Revised: ASR is deﬁned (for each attack) as the probability that a test image from the source class is (mis)classiﬁed to the target class of BA when the BP is embedded.

Original: ACC is deﬁned (for each classiﬁer being attack regardless of the number of attacks) as the classiﬁcation accuracy on test samples with no BP.
Revised: ACC is deﬁned (for each classiﬁer being attack regardless of the number of attacks) as the classiﬁcation accuracy on test samples with no BP.

Original: In our experiments, we evaluate ASR and ACC using images from the test set associated with each 2-class domain – these images are not used during training.
Revised: In our experiments, we evaluate ASR and ACC using images from the test set associated with each 2-class domain – these images are not used during training.

Original: For each attack instance, we evaluate ASR for each attack (since there can be either one attack or two attacks with different BA target class) separately.
Revised: For each attack instance, we evaluate ASR for each attack (since there can be either one attack or two attacks with different BA target class) separately.

Original: In Tab. 7, for each of ensembles A 1 -A 10 , we show the average ACC for the classiﬁer being attacked over all instances in the ensemble; we also show the mean and minimum ASR over all attacks of all instances in the ensemble.
Revised: In Tab. 7, for each of ensembles A 1 -A 10 , we show the average ACC for the classiﬁer being attacked over all instances in the ensemble; we also show the mean and minimum ASR over all attacks of all instances in the ensemble.

Original: As a reference,
Revised: As a reference,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: in Tab. 8 for each of ensemble C 1 -C 6 of clean instances, we show the average ACC over all the clean classiﬁers for the ensemble.
Revised: in Tab. 8 for each of ensemble C 1 -C 6 of clean instances, we show the average ACC over all the clean classiﬁers for the ensemble.

Original: Based on the results in both Tab. 7 and Tab. 8, all the attacks we created are successful with high ASR and almost no degradation in ACC.
Revised: Based on the results in both Tab. 7 and Tab. 8, all the attacks we created are successful with high ASR and almost no degradation in ACC.

Original: D. BP R EVERSE -E NGINEERING A LGORITHMS
Revised: D. BP R EVERSE -E NGINEERING A LGORITHMS

Original: In the main paper, we evaluate our detection framework with two BP reverse-engineering algorithms, which are denoted as RE-AP and RE-PR, respectively.
Revised: In the main paper, we evaluate our detection framework with two BP reverse-engineering algorithms, which are denoted as RE-AP and RE-PR, respectively.

Original: RE-AP is proposed by Xiang et al.
Revised: RE-AP is proposed by Xiang et al.

Original: (2020) for reverse-engineering additive perturbation BPs.
Revised: (2020) for reverse-engineering additive perturbation BPs.

Original: The general form of RE-AP estimates a common perturbation that induces a group of images to be misclassiﬁed to a common target class.
Revised: The general form of RE-AP estimates a common perturbation that induces a group of images to be misclassiﬁed to a common target class.

Original: When there is a single image in such a group, and when there are only two classes, the optimization problem solved by RE-AP is reduced to (3).
Revised: When there is a single image in such a group, and when there are only two classes, the optimization problem solved by RE-AP is reduced to (3).

Original: To solve this problem for some target class i ∈ C and image x ∈ X from the class other than i , RE-AP minimizes the following surrogate objective function:
Revised: To solve this problem for some target class i ∈ C and image x ∈ X from the class other than i , RE-AP minimizes the following surrogate objective function:

Original: [Equation]
Revised: [Equation]

Original: (58) using gradient descent with v initialized from 0 , until the constraint of (3) is satisﬁed.
Revised: (58) using gradient descent with v initialized from 0 , until the constraint of (3) is satisﬁed.

Original: Here, p ( i | x ) denotes the classiﬁer’s posterior of class i give any input sample x ∈ X .
Revised: Here, p ( i | x ) denotes the classiﬁer’s posterior of class i give any input sample x ∈ X .

Original: The step size for minimization is set small to ensure a good “quality” for the solution; otherwise, the resulting perturbation may have a much larger norm than the minimum norm perturbation required for inducing a misclassiﬁcation.
Revised: The step size for minimization is set small to ensure a good “quality” for the solution; otherwise, the resulting perturbation may have a much larger norm than the minimum norm perturbation required for inducing a misclassiﬁcation.

Original: Moreover, for each domain and each classiﬁer to be inspected, choosing a proper step size can be done based on the norm of the solution and without any knowledge of the presence of BA.
Revised: Moreover, for each domain and each classiﬁer to be inspected, choosing a proper step size can be done based on the norm of the solution and without any knowledge of the presence of BA.

Original: Another BP reverse-engineering algorithm, RE-PR is proposed by Wang et al.
Revised: Another BP reverse-engineering algorithm, RE-PR is proposed by Wang et al.

Original: (2019) for patch replacement BPs.
Revised: (2019) for patch replacement BPs.

Original: Similarly, RE-PR solves problem (46) in Apdx.
Revised: Similarly, RE-PR solves problem (46) in Apdx.

Original: C.1, which is the counterpart of problem (3) for patch replacement BPs.
Revised: C.1, which is the counterpart of problem (3) for patch replacement BPs.

Original: Formally, for some target class i ∈ C and image x ∈ X from the class other than i , RE-PR minimizes the following surrogate objective function:
Revised: Formally, for some target class i ∈ C and image x ∈ X from the class other than i , RE-PR minimizes the following surrogate objective function:

Original: [Equation]
Revised: [Equation]

Original: using gradient descent, where λ is the Lagrange multiplier and ∆ ( x ; m , u ) is the alternative expression to Eq.
Revised: using gradient descent, where λ is the Lagrange multiplier and ∆ ( x ; m , u ) is the alternative expression to Eq.

Original: 2 (see the description below Eq.
Revised: 2 (see the description below Eq.

Original: (46)) for patch replacement BP embedding.
Revised: (46)) for patch replacement BP embedding.

Original: [Table]
Revised: [Table]

Original: D. L IMITATIONS OF THE C OSINE S IMILARITY S TATISTIC IN BA D ETECTION
Revised: D. L IMITATIONS OF THE C OSINE S IMILARITY S TATISTIC IN BA D ETECTION

Original: In the main paper, we compared our ET statistic with other three types of detection statistics including a cosine similarity (CS) statistic proposed by Wang et al.
Revised: In the main paper, we compared our ET statistic with other three types of detection statistics including a cosine similarity (CS) statistic proposed by Wang et al.

Original: For general classiﬁcation domains with arbitrary number of classes, the CS statistic is obtained for each putative target class t ∈ C as following.
Revised: For general classiﬁcation domains with arbitrary number of classes, the CS statistic is obtained for each putative target class t ∈ C as following.

Original: First, a common (patch replacement) BP is estimated to: a) induces a group of images from classes other than t to be misclassiﬁed in an untargeted fashion (i.e., to any class other than their originally labeled classes); b) not induce any class t images to be misclassiﬁed; and c) have as small mask size (measured by l 1 norm) as possible.
Revised: First, a common (patch replacement) BP is estimated to: a) induces a group of images from classes other than t to be misclassiﬁed in an untargeted fashion (i.e., to any class other than their originally labeled classes); b) not induce any class t images to be misclassiﬁed; and c) have as small mask size (measured by l 1 norm) as possible.

Original: Accordingly, Wang et al.
Revised: Accordingly, Wang et al.

Original: (2020) proposed to minimize the following loss:
Revised: (2020) proposed to minimize the following loss:

Original: [Equation]
Revised: [Equation]

Original: where h i ( · ) :
Revised: where h i ( · ) :

Original: X → R is the logit (right before softmax) of class i ∈ C Carlini & Wagner (2017).
Revised: X → R is the logit (right before softmax) of class i ∈ C Carlini & Wagner (2017).

Original: We denote the estimated (common) BP for class t as s ∗ t = { m ∗ t , u ∗ t } .
Revised: We denote the estimated (common) BP for class t as s ∗ t = { m ∗ t , u ∗ t } .

Original: Then, for each image not from class t , a sample-wise BP is estimated to: a) induce the image to be misclassiﬁed to class t ; and b) have as small mask size (measured by l 1 norm) as possible.
Revised: Then, for each image not from class t , a sample-wise BP is estimated to: a) induce the image to be misclassiﬁed to class t ; and b) have as small mask size (measured by l 1 norm) as possible.

Original: Thus, the following loss is minimized for each x ∈ ∪ i ∈C\
Revised: Thus, the following loss is minimized for each x ∈ ∪ i ∈C\

Original: t D i :
Revised: t D i :

Original: [Equation]
Revised: [Equation]

Original: We denote the sample-wise BP estimated for class t and sample x as ˜ s ∗ t ( x ) =
Revised: We denote the sample-wise BP estimated for class t and sample x as ˜ s ∗ t ( x ) =

Original: { ˜ m ∗ t ( x ) , ˜ u ∗ t ( x ) } .
Revised: { ˜ m ∗ t ( x ) , ˜ u ∗ t ( x ) } .

Original: Finally, the cosine similarity statistic for class t is computed by:
Revised: Finally, the cosine similarity statistic for class t is computed by:

Original: [Equation]
Revised: [Equation]

Original: where z ( · ) : X → R d is the mapping from input layer to the penultimate layer with some dimension d .
Revised: where z ( · ) : X → R d is the mapping from input layer to the penultimate layer with some dimension d .

Original: cos( · ) : R d × R d → [ − 1 , 1] is the cosine similarity between two real vectors.
Revised: cos( · ) : R d × R d → [ − 1 , 1] is the cosine similarity between two real vectors.

Original: Based on our results in Fig. 2, CS for BA target classes and non-target classes are separable for most domains.
Revised: Based on our results in Fig. 2, CS for BA target classes and non-target classes are separable for most domains.

Original: This is not surprising because when class t ∈ C is a BA target class, the estimated common BP will likely be highly correlated with the sample-wise BPs estimated for images from classes other than t ; thus, the resulting CS will be large and possibly close to 1.
Revised: This is not surprising because when class t ∈ C is a BA target class, the estimated common BP will likely be highly correlated with the sample-wise BPs estimated for images from classes other than t ; thus, the resulting CS will be large and possibly close to 1.

Original: If class t ∈ C is not a BA target class, the estimated common BP may induce images from classes other than t to be misclassiﬁed to some arbitrary classes (possibly the “semantically” closest class for each individual image).
Revised: If class t ∈ C is not a BA target class, the estimated common BP may induce images from classes other than t to be misclassiﬁed to some arbitrary classes (possibly the “semantically” closest class for each individual image).

Original: Thus, the common BP may be very different from sample-wise BP estimated to only induce misclassiﬁcations to class t .
Revised: Thus, the common BP may be very different from sample-wise BP estimated to only induce misclassiﬁcations to class t .

Original: Consequently, the CS statistic will likely be small.
Revised: Consequently, the CS statistic will likely be small.

Original: However, considering our 2-class problem with no sufﬁcient number of statistics to inform estimation of a null distribution, and also our assumption that there is no domain-speciﬁc supervision (e.g. using clean classiﬁers trained for the same domain) for setting a proper detection threshold, CS statistic may not be effective since it is sensitive to domains and DNN architectures.
Revised: However, considering our 2-class problem with no sufﬁcient number of statistics to inform estimation of a null distribution, and also our assumption that there is no domain-speciﬁc supervision (e.g. using clean classiﬁers trained for the same domain) for setting a proper detection threshold, CS statistic may not be effective since it is sensitive to domains and DNN architectures.

Original: First, when there is no BA, for ReLU networks, the penultimate layer features are always nonnegative, such that the cosine similarity is guaranteed to be non-negative.
Revised: First, when there is no BA, for ReLU networks, the penultimate layer features are always nonnegative, such that the cosine similarity is guaranteed to be non-negative.

Original: However, for DNN with sigmoid or leaky ReLU activation functions, the penultimate layer features can be negative.
Revised: However, for DNN with sigmoid or leaky ReLU activation functions, the penultimate layer features can be negative.

Original: Thus, the CS statistic may be distributed in the entire interval of [ − 1 , 1] .
Revised: Thus, the CS statistic may be distributed in the entire interval of [ − 1 , 1] .

Original: Such large difference in the null distribution of CS statistic makes the choice of a detection threshold very difﬁcult without domainspeciﬁc knowledge.
Revised: Such large difference in the null distribution of CS statistic makes the choice of a detection threshold very difﬁcult without domainspeciﬁc knowledge.

Original: Second, CS is also sensitive to the classiﬁcation domain, especially the number of classes.
Revised: Second, CS is also sensitive to the classiﬁcation domain, especially the number of classes.

Original: Considering some putative target class t ∈ C , when there are a large number of classes in the domain, the common BP estimated for a group of images from classes other than t will likely be very different with the sample-wise BP estimated for each individual in the group.
Revised: Considering some putative target class t ∈ C , when there are a large number of classes in the domain, the common BP estimated for a group of images from classes other than t will likely be very different with the sample-wise BP estimated for each individual in the group.

Original: In particular, most images will likely be misclassiﬁed to some class other than t when the common BP is embedded; but the sample-wise BP is estimated for each of these images to induce them to be misclassiﬁed to class t .
Revised: In particular, most images will likely be misclassiﬁed to some class other than t when the common BP is embedded; but the sample-wise BP is estimated for each of these images to induce them to be misclassiﬁed to class t .

Original: Thus, the penultimate layer feature associated with the common BP and the sample-wise BP will likely be different for most images.
Revised: Thus, the penultimate layer feature associated with the common BP and the sample-wise BP will likely be different for most images.

Original: However, when there are only two classes, i.e. C = { 0 , 1 } in our case, the images used for BP estimation for class t are all from class (1 − t ) .
Revised: However, when there are only two classes, i.e. C = { 0 , 1 } in our case, the images used for BP estimation for class t are all from class (1 − t ) .

Original: Moreover, both common BP and samples-wise BP estimated for these images will induce them to be misclassiﬁed to class t .
Revised: Moreover, both common BP and samples-wise BP estimated for these images will induce them to be misclassiﬁed to class t .

Original: Thus, CS obtained for this case will likely be larger than for the cases where there are a large number of classes when there is no BA.
Revised: Thus, CS obtained for this case will likely be larger than for the cases where there are a large number of classes when there is no BA.

Original: We demonstrate this phenomenon in the following.
Revised: We demonstrate this phenomenon in the following.

Original: We construct ﬁve domains from CIFAR-10.
Revised: We construct ﬁve domains from CIFAR-10.

Original: The ﬁrst four domains contain 2, 4, 6, 8 classes randomly selected from the 10 classes of CIFAR-10 respectively, and the the ﬁfth domain is the original
Revised: The ﬁrst four domains contain 2, 4, 6, 8 classes randomly selected from the 10 classes of CIFAR-10 respectively, and the the ﬁfth domain is the original

Original: CIFAR-10 with 10 classes.
Revised: CIFAR-10 with 10 classes.

Original: We train a classiﬁer without BA for each domain using the same conﬁgurations as in Sec. 4.1.
Revised: We train a classiﬁer without BA for each domain using the same conﬁgurations as in Sec. 4.1.

Original: For each classiﬁer, we obtain CS statistics for all classes.
Revised: For each classiﬁer, we obtain CS statistics for all classes.

Original: In Fig.
Revised: In Fig.

Original: 6, we show the average CS statistic over all classes for the ﬁve classiﬁers.
Revised: 6, we show the average CS statistic over all classes for the ﬁve classiﬁers.

Original: In general, CS statistic decreases as the number of classes grows; thus, it is highly domain-dependent.
Revised: In general, CS statistic decreases as the number of classes grows; thus, it is highly domain-dependent.

Original: D. D ETAILS FOR M ULTI -C LASS E XPERIMENTS
Revised: D. D ETAILS FOR M ULTI -C LASS E XPERIMENTS

Original: In Sec. 4.4, we evaluate the performance of our detection framework for multi-class scenarios with arbitrary number of attacks.
Revised: In Sec. 4.4, we evaluate the performance of our detection framework for multi-class scenarios with arbitrary number of attacks.

Original: In other words, the classiﬁer has more than two classes, and each class can possibly be a BA target class.
Revised: In other words, the classiﬁer has more than two classes, and each class can possibly be a BA target class.

Original: For each of CIFAR-10, CIFAR-100, and STL-10, we create three attack instances with one, two, and three attacks, respectively.
Revised: For each of CIFAR-10, CIFAR-100, and STL-10, we create three attack instances with one, two, and three attacks, respectively.

Original: Like the attacks we created for the 2-class domains, the attacks here are created following the same data poisoning protocol that has been widely considered in existing works.
Revised: Like the attacks we created for the 2-class domains, the attacks here are created following the same data poisoning protocol that has been widely considered in existing works.

Original: That is, we create backdoor training images by embedding a BP into a small set of images from classes other than the target class.
Revised: That is, we create backdoor training images by embedding a BP into a small set of images from classes other than the target class.

Original: These backdoor training images are labeled to a target class and inserted into the training set of the classiﬁer Gu et al.
Revised: These backdoor training images are labeled to a target class and inserted into the training set of the classiﬁer Gu et al.

Original: In our experiment here, for each attack of each instance, we randomly select a target class.
Revised: In our experiment here, for each attack of each instance, we randomly select a target class.

Original: For simplicity, we consider only additive perturbation BP here.
Revised: For simplicity, we consider only additive perturbation BP here.

Original: We randomly select a shape (and location for localized BP) for the BP to be used from our pool of candidate BPs.
Revised: We randomly select a shape (and location for localized BP) for the BP to be used from our pool of candidate BPs.

Original: Note that for any two attacks of the same attack instance, the target classes and the BPs should both be different from each other.
Revised: Note that for any two attacks of the same attack instance, the target classes and the BPs should both be different from each other.

Original: For attacks on CIAFR-10,
Revised: For attacks on CIAFR-10,

Original: CIFAR-100, and STL-10, the backdoor training images are created using 60, 10, and 100 clean images per class (not including the target class), respectively.
Revised: CIFAR-100, and STL-10, the backdoor training images are created using 60, 10, and 100 clean images per class (not including the target class), respectively.

Original: For each domain, the three attack instances and the clean classiﬁer use the same training conﬁguration.
Revised: For each domain, the three attack instances and the clean classiﬁer use the same training conﬁguration.

Original: For CIFAR-10 and STL-10, we use ResNet-18 as the DNN architecture; for CIFAR-100, we use ResNet-34 architecture.
Revised: For CIFAR-10 and STL-10, we use ResNet-18 as the DNN architecture; for CIFAR-100, we use ResNet-34 architecture.

Original: For all three domains, training data augmentations including random
Revised: For all three domains, training data augmentations including random

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: cropping and random horizontal ﬂipping are adopted.
Revised: cropping and random horizontal ﬂipping are adopted.

Original: Other conﬁgurations for classiﬁer training for these domains are the same as for the 2-class domains generated from these original domains (shown in Tab. 6).
Revised: Other conﬁgurations for classiﬁer training for these domains are the same as for the 2-class domains generated from these original domains (shown in Tab. 6).

Original: Using these training conﬁgurations, the resulting nine classiﬁers being attacked (three classiﬁers for the three attack instances respectively for each domain) all have high attack success rate (ASR).
Revised: Using these training conﬁgurations, the resulting nine classiﬁers being attacked (three classiﬁers for the three attack instances respectively for each domain) all have high attack success rate (ASR).

Original: Compared with the three clean classiﬁers (without BA) trained for the three domains respectively, there is also no signiﬁcant degradation in clean test accuracy (ACC) for the classiﬁers being attacked.
Revised: Compared with the three clean classiﬁers (without BA) trained for the three domains respectively, there is also no signiﬁcant degradation in clean test accuracy (ACC) for the classiﬁers being attacked.

Original: ASR and ACC for these classiﬁers are shown in Tab. 9.
Revised: ASR and ACC for these classiﬁers are shown in Tab. 9.

Original: E U SING ET TO D ETECT C LEAN -L ABEL BA S
Revised: E U SING ET TO D ETECT C LEAN -L ABEL BA S

Original: In this section, we demonstrate the effectiveness of our detection framework against a recent cleanlabel BA proposed by Turner et al.
Revised: In this section, we demonstrate the effectiveness of our detection framework against a recent cleanlabel BA proposed by Turner et al.

Original: Clean-label BAs are motivated by the possible human/machine inspection of the training set.
Revised: Clean-label BAs are motivated by the possible human/machine inspection of the training set.

Original: For example, backdoor training samples labeled to some target class inserted by a typical backdoor attacker are originally from classes other than the target class.
Revised: For example, backdoor training samples labeled to some target class inserted by a typical backdoor attacker are originally from classes other than the target class.

Original: Such “mislabeling” may be noticed by a human expert who inspects the training set manually, or  detected by a shallow neural network trained on a small held-out validation set that is guaranteed to be clean.
Revised: Such “mislabeling” may be noticed by a human expert who inspects the training set manually, or may be detected by a shallow neural network trained on a small held-out validation set that is guaranteed to be clean.

Original: Thus, Turner et al.
Revised: Thus, Turner et al.

Original: (2019) proposed to create backdoor training samples (that will be inserted into the classiﬁer’s training set) by embedding the BP only to target class samples.
Revised: (2019) proposed to create backdoor training samples (that will be inserted into the classiﬁer’s training set) by embedding the BP only to target class samples.

Original: However, for target class samples embedded with the BP, there is no guarantee that it is the BP but not the features associated with the target class that will be learned by the classiﬁer.
Revised: However, for target class samples embedded with the BP, there is no guarantee that it is the BP but not the features associated with the target class that will be learned by the classiﬁer.

Original: Thus, Turner et al.
Revised: Thus, Turner et al.

Original: (2019) proposed to “destroy” these target class features before embedding the BP when creating backdoor training samples.
Revised: (2019) proposed to “destroy” these target class features before embedding the BP when creating backdoor training samples.

Original: Then, the classiﬁer will learn the BP and classify any test sample embedded with the BP to the target class.
Revised: Then, the classiﬁer will learn the BP and classify any test sample embedded with the BP to the target class.

Original: One simple yet effective approach proposed by Turner et al.
Revised: One simple yet effective approach proposed by Turner et al.

Original: (2019) to destroy the features in the backdoor training samples associated with the target class is inspired by a method for creating adversarial examples.
Revised: (2019) to destroy the features in the backdoor training samples associated with the target class is inspired by a method for creating adversarial examples.

Original: The backdoor attacker needs to ﬁrst train a surrogate classiﬁer using an independently collected clean dataset.
Revised: The backdoor attacker needs to ﬁrst train a surrogate classiﬁer using an independently collected clean dataset.

Original: Then, for each of a small set of samples from the target class used for creating backdoor training samples, the attacker independently launches a projected gradient descent (PGD) attack (Madry et al.
Revised: Then, for each of a small set of samples from the target class used for creating backdoor training samples, the attacker independently launches a projected gradient descent (PGD) attack (Madry et al.

Original: (2018)) to have the sample be predicted to any class other the its original class (i.e. the target class) by the surrogate classiﬁer.
Revised: (2018)) to have the sample be predicted to any class other than its original class (i.e. the target class) by the surrogate classiﬁer.

Original: These samples with the targetclass destroyed
Revised: These samples with the target

Original: In our experiment here, we randomly generate ten 2-class domains from CIFAR-10 following its original train-test split.
Revised: In our experiment here, we randomly generate ten 2-class domains from CIFAR-10 following its original train-test split.

Original: For each 2-class domain, we ﬁrst train a surrogate classiﬁer using a subset of the training set (2000 training images per class).
Revised: For each 2-class domain, we ﬁrst train a surrogate classiﬁer using a subset of the training set (2000 training images per class).

Original: The remaining samples (3000 per class) are assumed to be possessed by the trainer for training the victim classiﬁer.
Revised: The remaining samples (3000 per class) are assumed to be possessed by the trainer for training the victim classiﬁer.

Original: For each 2-class domain, we create one attack instance with one BA targeting on the second class and using an additive perturbation BP.
Revised: For each 2-class domain, we create one attack instance with one BA targeting on the second class and using an additive perturbation BP.

Original: The candidate BPs to be used are the same as in our experiments in the main paper.
Revised: The candidate BPs to be used are the same as in our experiments in the main paper.

Original: Here, for each BA, we use 1500 images from the target class (i.e. the second class of the associated 2-class domain) to create backdoor training images.
Revised: Here, for each BA, we use 1500 images from the target class (i.e. the second class of the associated 2-class domain) to create backdoor training images.

Original: These images are randomly sampled from the images used for training the surrogate classiﬁer.
Revised: These images are randomly sampled from the images used for training the surrogate classiﬁer.

Original: For each of these 1500 images, we independently generate an adversarial perturbation using the surrogate classiﬁer following the standard protocol of
Revised: For each of these 1500 images, we independently generate an adversarial perturbation using the surrogate classiﬁer following the standard protocol of

Original: PGD Madry et al.
Revised: PGD Madry et al.

Original: In particular, we set the maximum perturbation size as 8/255, the number of perturbation steps as 10, and the step size as 1/255.
Revised: In particular, we set the maximum perturbation size as 8/255, the number of perturbation steps as 10, and the step size as 1/255.

Original: Most images being perturbed  are misclassiﬁed by the surrogate classiﬁer.
Revised: Most of the perturbed images are misclassiﬁed by the surrogate classiﬁer.

Original: Then we embed the BP randomly selected from the candidates for each attack into these images with the adversarial perturbation and still label them to the target class where they are originally from.
Revised: Then we embed the BP randomly selected from the candidates for each attack into these images with the adversarial perturbation and still label them to the target class, where they are originally from.

Original: The created backdoor training images are inserted into the training set of the victim classiﬁer.
Revised: The created backdoor training images are inserted into the training set of the victim classiﬁer.

Original: They will be barely noticeable to human inspectors since they visually look like standard target class images and the embedded BP is almost imperceptible by humans.
Revised: They will be barely noticeable to human inspectors since they visually look like standard target class images and the embedded BP is almost imperceptible by humans.

Original: Some examples of backdoor training images are shown in Fig.
Revised: Some examples of backdoor training images are shown in Fig.

Original: For each attack instance, we use the same training conﬁgurations as in Sec.
Revised: For each attack instance, we use the same training conﬁgurations as in Sec.

Original: 4.1 of the main paper to train the victim classiﬁer.
Revised: 4.1 of the main paper to train the victim classiﬁer.

Original: We also train a clean classiﬁer for each instance to evaluate false detections.
Revised: We also train a clean classiﬁer for each instance to evaluate false detections.

Original: Moreover, we apply our detection framework with the BP reverse-engineering algorithm
Revised: Moreover, we apply our detection framework with the BP reverse-engineering algorithm

Original: RE-AP and the same defense conﬁgurations as in the main paper to both the classiﬁers being attacked and the clean classiﬁers.
Revised: RE-AP and the same defense conﬁgurations as in the main paper to both the classiﬁers being attacked and the clean classiﬁers.

Original: In Fig.
Revised: In Fig.

Original: 8, we show the maximum ET (over the two classes) for all these classiﬁers.
Revised: 8, we show the maximum ET (over the two classes) for all these classiﬁers.

Original: Using ET and the constant detection threshold 1/2, we perfectly detect all clean-label
Revised: Using ET and the constant detection threshold 1/2, we perfectly detect all clean-label

Original: BAs with no false detections.
Revised: BAs with no false detections.

Original: F E XPERIMENTAL V ERIFICATION OF P ROPERTY 3.1 FOR P ATCH R EPLACEMENT BP S
Revised: F E XPERIMENTAL V ERIFICATION OF P ROPERTY 3.1 FOR P ATCH R EPLACEMENT BP S

Original: We verify Property 3.1 for patch replacement BPs embedded by Eq.
Revised: We verify Property 3.1 for patch replacement BPs embedded by Eq.

Original: Similar to Sec.
Revised: Similar to Sec.

Original: 4.3 of the main paper, here, we show that when class (1 − i ) for i ∈ C = { 0 , 1 } is not a BA target class, for any two samples from class i , the minimum l 1 norm for any common mask that induces both samples to be misclassiﬁed will likely be larger than the minimum l 1 norm for the masks inducing each  sample to be misclassiﬁed.
Revised: 4.3 of the main paper, here, we show that when class (1 − i ) for i ∈ C = { 0 , 1 } is not a BA target class, for any two samples from class i , the minimum l 1 norm for any common mask that induces both samples to be misclassiﬁed will likely be larger than the minimum l 1 norm for the masks inducing each individual sample to be misclassiﬁed.

Original: The masks are obtained by solving problem (3).
Revised: The masks are obtained by solving problem (3).

Original: We randomly choose one clean classiﬁer from each of C 1 -C 4 .
Revised: We randomly choose one clean classiﬁer from each of C 1 -C 4 .

Original: For each classiﬁer, we randomly choose 10 pairs of clean images from a random class of the associated 2-class domain. 
Revised: For each classiﬁer, we randomly choose 10 pairs of clean images from a random class of the associated 2-class domain. For each pair

Original: For each pair of images, we apply the RE-PR algorithm (for reverse-engineering patch replacement BPs) on the two imagesboth jointly (to get a pair-wise common mask) and separately (to get two sample-wise masks for the two images respectively).
Revised: [Figure] of images, we apply the RE-PR algorithm (for reverse-engineering patch replacement BPs) on the two images jointly (to get a pair-wise common mask) as well as separately (to get two sample-wise masks for the two images respectively).

Original: Again, we divide the l 1 norm of the pair-wise (common) mask by the maximum l 1 norm of the two sample-wise masks to get a ratio.
Revised: Again, we divide the l 1 norm of the pair-wise (common) mask by the maximum l 1 norm of the two sample-wise masks to get a ratio.

Original: In Fig.
Revised: In Fig.

Original: 9, we plot the histogram of such ratio for all image pairs for all four classiﬁers – the ratio for most of the image pairs is greater than 1 (marked by the red dashed line in Fig. 9).
Revised: 9, we plot the histogram of such ratio for all image pairs for all four classiﬁers – the ratio for most of the image pairs is greater than 1 (marked by the red dashed line in Fig. 9).

Original: For these pairs,  very likely that the BP (in particular, the mask) estimated for one sample cannot induce the other sample to be misclassiﬁed and vise versa.
Revised: For these pairs, it is very likely that the BP (in particular, the mask) estimated for one sample cannot induce the other sample to be misclassiﬁed and vise versa.

Original: Note that if for any image pair, the mask (and some associated pattern) estimated for one sample can induce the other to be misclassiﬁed, such mask (and the associated pattern) will be a common mask (and pattern) that induces both images to be misclassiﬁed.
Revised: Note that if for any image pair, the mask (and some associated pattern) estimated for one sample can induce the other to be misclassiﬁed, such mask (and the associated pattern) will be a common mask (and pattern) that induces both images to be misclassiﬁed.

Original: Then, we would expect the ratio computed above to be very close to 1 for such  image pair.
Revised: Then, we would expect the ratio computed above to be very close to 1 for such an image pair.

Revised: G I NFLUENCE OF THE N UMBER OF C LEAN I MAGES FOR D ETECTION

Revised: The core of our detector is to estimate the ET statistic for each class.

Revised: Note that the ET statistic is in fact an expectation.

Revised: In principle, with fewer clean images per class for ET estimation, the variance of the estimated ET will be larger, though the execution time for ET estimation may be smaller.

Revised: Thus, we would expect that, sometimes, the ET estimated using only a few clean images may be smaller than 12 for the attack case; or larger than 12 for the clean case.

Revised: In Sec. 4.1, we used 20 images per class (40 images in total) for backdoor detection for all attack instances and all clean instances, and achieved good detection accuracy.

Revised: Here, we show the inﬂuence of the number of clean images per class on detection accuracy.

Revised: In particular, for all 45 attack instances and all 45 clean instances of two-class domains generated from CIFAR-10 (i.e. A 1 and

Revised: C 1 ), we apply the same detector in Sec. 4.1 with detection threshold 12 , but varying the number of clean images per class (in [2, 5, 10, 15]) used for detection (i.e. ET estimation).

Revised: As shown in Fig.

Revised: 10, with 5 clean images per class (10 images in total), our method achieves relatively good detection accuracy.

Revised: Even with only 2 clean images per class (which is the minimum sample size for empirical

Revised: ET estimation), our detector catches ∼ 80% of attacks with less than 10% false detection rate 5 .

Revised: H C HOICE OF THE P ATIENCE P ARAMETER τ

Revised: In all experiments in the main paper, we set the patience parameter in Alg.

Revised: 1 to τ = 4 and claim that larger τ will not induce much change to the estimated ET.

Revised: Here, we provide some empirical evidence to support this claim.

Revised: We apply Alg. 1 with RE-AP to a classiﬁer being attacked in A 1 and a classiﬁer being attacked in A 2 .

Revised: We also apply Alg. 1 with RE-PR to a classiﬁer being attacked in A 7 and a classiﬁer being attacked in A 8 .

Revised: Moreover, Alg. 1 with both RE-AP and RE-PR are applied to a clean classiﬁer in C 1 and a clean classiﬁer in C 2 .

Revised: Note that classiﬁcation domains associated with A 1 , A 7 , and C 1 are generated from CIFAR-10; while classiﬁcation domains associated with A 2 , A 8 , and C 2 are generated from

Revised: CIFAR-100.

Revised: For all experiments in this section, we set the patience parameter to τ = 8 instead of τ =used in the main paper.

Revised: The purpose is to get a better observation of the asymptotic behavior of p ( i ) n = | (cid:98) T (cid:15) ( x ( i ) n ) | / ( N i − 1) during ET estimation for each clean sample x ( i ) n used for detection (see line 7-12 of Alg.

Revised: 1 for the deﬁnition of related quantities).

Revised: The number of clean samples used for detection is 20.

Revised: As shown in Fig.

Revised: 11, when applying our method to classiﬁers being attacked, p ( i ) n (for some class i ) quickly grows to 1 (in very few iterations) for most clean samples used for detection (see (a)(b)(e)(f) of Fig. 11).

Original: {+For a few clean samples, p ( i ) n quickly grows to a large value close to 
+}
Revised: {+For a few clean samples, p ( i ) n quickly grows to a large value close to 
+}

Revised: and then slowly reaches 1 (see (e)(f) of Fig. 11).

Revised: Only for very few samples, p ( i ) n stays at some value in between 0 and 1 (see (b)(e) of Fig. 11).

Revised: Based on these observations, which are generally true for other domains we investigated, τ = 4 is not a critical choices to our detection performance.

Revised: The estimated ET, which is the average p ( i ) n for all clean samples used for detection, will likely be greater than 12 , as determined by the majority of clean samples used for detection.

Revised: On the other hand, when applying our method to clean classiﬁers, with RE-AP, p ( i ) n stays at 0 (or some small values close to 0) for all clean samples used for detection (see (c)(g) of Fig. 11).

Revised: When applying our method with RE-PR to the same classiﬁers, p ( i ) n stays at 0 or some small values close to 0 for most samples and shows a trend of convergence (see (d)(h) of Fig. 11).

Revised: Again, reducing τ from 8 to 4 or further increasing τ will not change the estimated ET much – the estimated ET for these clean instances will still be clearly less than 12 .

Revised: I U SING S YNTHESIZED I MAGES FOR B ACKDOOR D ETECTION

Revised: For most REDs, the defender is assumed to possess a small, clean dataset (collected independently) for detection Wang et al.

Revised: Xiang et al.

Revised: Guo et al.

Revised: ; Wang et al.

Revised: Although this assumption is relatively mild and feasible in most practical scenarios, it may be unnecessary if the defender is able to synthesize the images used for detection.

Revised: The ﬁrst trial was made by Chen et al.

Revised: where on simple datasets like MNIST Lecun et al.

Revised: (1998), images for backdoor pattern reverse-engineering are synthesized by model inversion Fredrikson et al.

Revised: However, images generated in such a way are not guaranteed to be visually typical to their designated classes, especially for complicated domains like ImageNet.

Revised: Thus we ask the following question:

Revised: Can our ET framework be generalized to involve backdoor pattern reverse-engineering using synthesized images?

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Here, we consider the 20 two-class domains generated from MNIST due to this domain’s simplicity.

Revised: We apply RE-AP to the 20 classiﬁers being attacked in A 6 and the 20 clean classiﬁers in C 6 , respectively.

Revised: The clean images used for detection are generated using a simpler version of the model inversion method used by Chen et al.

Revised: To synthesize an image for detection, we ﬁrst initialize an all-zero image added with some small random positive noise.

Revised: Then, we maximize (over the image values) the posterior of the designated class of the image using the classiﬁer to be inspected, until the posterior is greater than 0.9.

Revised: Examples for our synthesized images are shown in Fig.

Revised: Note that without the “auxiliary constraints” on images during their generation process (Chen et al.

Revised: , the generated images are usually atypical to their designated classes.

Revised: In Fig. 13, we show the ET statistic for the 80 classes associated with the 40 binary classiﬁers ( classiﬁers being attacked and 20 clean classiﬁers).

Revised: Among these 80 classes, there are 60 backdoor target classes and 20 non-target classes (please see the settings in Sec. 4.1).

Revised: Despite one backdoor target class evading our detection and one ET for a backdoor target class lying close to the threshold 12 , the separation of ET for backdoor target classes and non-target classes is even better than that in the bottom left ﬁgure in Fig.

Revised: 2 (where backdoor pattern reverse-engineering is performed on typical

Revised: MNIST images).

Revised: The possible reasons maybe:

Revised: • For non-attack cases, two independently synthesized samples predicted to the same class will likely be both atypical to their predicted class.

Revised: They will likely share fewer features associated with their predicted class than two independent typical samples from this class.

Revised: They can also be located anywhere in the input space.

Revised: Thus, they will be more likely to be “mutually not transferable”.

Revised: • The synthesized samples are also far from the data manifold of classes other than the class they are predicted to.

Revised: So the minimum perturbation size required to induce them to be misclassiﬁed will have a large norm – possibly larger than the norm of the backdoor pattern when there is an attack.

Revised: If this is the case, the ET statistic will be exactly 1 based on Thm.

Revised: Future works along this research line include further investigation of the phenomenon we observed, and also improvement to the model inversion techniques, such that more complicated domains can be handled.

Revised: J C OMPUTATIONAL C OMPLEXITY

Revised: Here, we discuss the computational complexity of Alg.

Revised: More generally, we consider Alg.

Revised: 1 for multi-class scenarios.

Revised: Let K be the number of classes, N be the number of samples per class for detection, and T be the maximum number of forward/backward propagations for backdoor pattern reverse-engineering (i.e. “solving problem (3)” in line 8 of Alg. 1).

Revised: For each of the K classes, we compute the p ( i ) n quantity (line 12 of Alg. 1) for each of the KN samples used for detection.

Revised: To compute each p ( i ) n , the transferable set (line 9 of Alg. 1) is updated at most ( KN − 1) × τ times; and each updating involves at most T forward/backward propagations.

Revised: Thus, the theoretical upper bound of the number of forward/backward propagations for Alg. 1 is of the order

Revised: O

Revised: ( K 3 N 2 Tτ ) where τ is the patience parameter.

Revised: However, the actual complexity of our method in practice is much lower than the theoretical bound.

Revised: First, the purpose of using a sufﬁciently large number of samples for detection is to reduce the variance of the estimated ET (see Apdx.

Revised: G for more discussion and empirical results).

Revised: Thus, for sufﬁciently large K , we can set N = 1 and use only K samples for detection; or even use fewer samples randomly selected from these K samples.

Revised: Second, the actual number of iterations for updating the transferable set is much smaller than ( KN − 1) × τ in practice.

Revised: In Apdx.

Revised: H, we discussed the inﬂuence of the choice of τ on the estimated ET statistic and provided empirical results.

Revised: For attack cases, p ( i ) n reaches 1 (and thus terminates the updating of the transferable set) very quickly (even in one or two iterations) for most samples (see (a)(b)(e)(f) of Fig. 11).

Revised: For non-attack cases, convergence is also reached quickly for most samples (see (c)(d)(g)(h) of Fig. 11).

Revised: In these examples, 20 images are used for estimating the ET statistic with patience τ = 8 .

Revised: Thus the theoretical maximum number of iterations for the transferable set updating for an image is (20 − 1) × 8 = 152 , which is several times larger than the actual maximum number of iterations ( < 25).

Revised: In Fig.

Revised: 14, we show the curve of the execution time growing with the number of samples used for detection.

Revised: Speciﬁcally, we apply our detector to the clean binary classiﬁers in C 1 and record the average execution time, with the number of images for detection varying in [2, 5, 10, 15, 20].

Revised: Execution time is measured on a dual card RTX2080-Ti (11GB) GPU.

Revised: Comparing with Fig.where we show the effectiveness of our detector with only a few samples for detection, the actual time required for our detector to achieve good performance is only a few minutes.

Revised: Moreover, for attack cases, the total number of iterations (for all samples used for detection) required for ET

Revised: [Figure]