Original: L EVERAGING I NCOMPATIBILITY TO D EFEND A GAINST B ACKDOOR P OISONING A TTACKS
Revised: I NCOMPATIBILITY C LUSTERING AS A D EFENSE A GAINST B ACKDOOR P OISONING A TTACKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As deep learning datasets grow larger and less curated, backdoor data poisoning attacks, which inject malicious poisoned data into the training dataset, have drawn increasing attention in both academia and industry. We identify an incompatibility  property  ofthe interaction of clean and poisoned datawith the training algorithm, specifically that including poisoned data in thetraining dataset  does not improve model accuracy on clean data and vice-versa.
Revised: We propose a novel clustering mechanism based on incompatibility , a property between subsets of data that emerges during model training. In particular, the clustering mechanism partitions the dataset into subsets that generalize only to themselves , i.e., training on one subset does not improve performance on the other subsets.

Original: Leveraging this property, we develop an algorithm that iteratively refines subsets of thepoisoned dataset  toobtain subsets that concentrate around either clean or poisoned data. The result is a partition of the original dataset into disjoint subsets, for each ofwhich we train a corresponding model. A voting algorithm over these models identifies the clean data within the larger poisoned dataset.
Revised: By leveraging the interaction between the dataset and the training process, our clustering mechanism can be used to partition any dataset into clusters that are defined by—and therefore meaningful to—the objective of the training process. We apply our clustering mechanism to defend against data poisoning attacks, in which the attacker injects malicious poisoned data into the training dataset in order to affect the trained model’s output.

Original: We empirically evaluate our approach and technique for image classification tasks over the GTSRB and CIFAR-10 datasets.
Revised: Our evaluation focuses on backdoor attacks against deep neural networks trained to perform image classification using the GTSRB and CIFAR-10 datasets.

Original: The experimental results show that prior dirty-label and clean-label backdoor attacksin the literature produce poisoned datasets that exhibit behavior consistent with the incompatibility property.
Revised: Our results show that (1) these attacks produce poisoned datasets in which the poisoned and clean data are incompatible, and (2) our technique successfully identifies (and removes) the poisoned data.

Original: The results also show that our defense reduces the attack success rate  below 1% on 134 out of 165 scenarios in this setting, with only a 2% drop in clean accuracy on CIFAR-10 (and negligible impact on GTSRB).
Revised: In an end-to-end evaluation, our defense reduces the attack success rate to below 1% on 134 out of 165 scenarios, with only a 2% drop in clean accuracy on CIFAR- (and negligible impact on GTSRB).

Original:  O VERVIEW
Revised:  O VERVIEW

Original: Recent years have seen a rapid adoption of machine learning in a range of applications, from digital personal assistants to autonomous vehicles. A crucial driver has  beenthe adoption of increasingly large, minimally curated training datasets gathered from a variety of sources. Such datasets open up the possibility of data poisoning attacks in which the attacker inserts malicious data during training to influence the quality of the learned model. In this work, we focus on backdoor data poisoning attacks (Chen et al., 2017; Adi et al., 2018) against deep neural networks (DNNs).
Revised: Clustering, which aims to partition a dataset to surface some underlying structure, has long been a fundamental technique in machine learning and data analysis, with applications such as outlier detection and data mining across diverse fields including personalized medicine, document analysis, and networked systems (Xu & Tian, 2015).

Original: Specifically, by injecting a small amount of poisoned data into the training dataset, the attacker installs a backdoor in the network that can be used to control the network’s behavior at test time. For example, Gu et al. (2019) install a backdoor in a traffic sign classifier, which causes the network to misclassify stop signs as speed limit signs when a (physical) sticker is applied. Other triggers include a single pixel (Tran et al., 2018), small overlays such as a copyright symbol (Gao et al., 2019), or perturbations using adversarial training (Turner et al., 2018).
Revised: However, traditional approaches scale poorly to large, high-dimensional datasets, which limits the utility of such techniques for the increasingly large, minimally curated datasets that have become the norm in deep learning (Zhou et al., 2022).

Original: We present a new  technique for identifying poisoned data within a dataset potentially containing both clean and poisoned data.
Revised: We present a new clustering technique that partitions the dataset according to an incompatibility property that emerges during model training.

Original: Our technique leverages a property of the interaction of clean andpoisoned data with the training algorithm—specifically, a model trained on clean data generalizes well to new clean data but poorly to poisoned data (and vice-versa). Building on this insight, we develop a technique that iteratively refines the dataset to separate clean and poisoned data into disjoint subsets.
Revised: In particular, it partitions the dataset into subsets that generalize only to themselves (relative to the final trained model) and not to the other subsets. Directly leveraging the interaction between the dataset and the training process enables our technique to partition large, high-dimensional datasets commonly used to train neural networks (or, more generally, any class of learned models) into clusters defined by—and therefore meaningful to—the task the network is intended to perform.

Original: The algorithm operates as follows.
Revised: Our technique operates as follows.

Original: Given a dataset, the algorithm iteratively samples a subset of the dataset, trains a model on the subset, then selects the elements within the larger dataset that the model scores most highly.
Revised: Given a dataset, it iteratively samples a subset of the dataset, trains a model on the subset, then selects the elements within the larger dataset that the model scores most highly.

Original: The selected elements comprise a smaller dataset for the next refinement step.
Revised: These selected elements comprise a smaller dataset for the next refinement step.

Original: By decreasing the size of the selected subsets in each iteration, this process produces a final subset that (ideally) contains either clean data or poisoned data.
Revised: By decreasing the size of the selected subset in each iteration, this process produces a final subset that (ideally) contains only data which is compatible with itself.

Original: The identified subset is then removed from the starting dataset, and the process is repeated to obtain a collection of refined subsets and corresponding trained models.
Revised: The identified subset is then removed from the starting dataset, and the process is repeated to obtain a collection of refined subsets and corresponding trained models.

Original: Finally, a majority vote using models trained on the resulting refinedclean and poisoned subsets distinguishes the clean and poisoned data in the initial training set.
Revised: Finally, a majority vote using models trained on the resulting refined subsets merges compatible subsets to produce the final partitioning.

Original: We identify our main contributions as follows:
Revised: Our evaluation focuses on backdoor data poisoning attacks (Chen et al., 2017; Adi et al., 2018) against deep neural networks (DNNs), in which the attacker injects a small amount of poisoned data into the training dataset to install a backdoor that can be used to control the network’s behavior during deployment. For example, Gu et al. (2019) install a backdoor in a traffic sign classifier, which causes the network to misclassify stop signs as speed limit signs when a (physical) sticker is applied.

Original: Incompatibility of Clean and Poisoned Data.
Revised: Prior work has found that directly applying classical outlier detection techniques to the dataset fails to separate the poisoned and clean data (Tran et al., 2018). Our main intuition is that training on poisoned data should not improve accuracy on clean data (and vice versa). Hence, the poisoned data satisfies the incompatibility property and can be separated using our clustering technique. This paper makes the following contributions:

Original:  Section 3 formally definesan incompatibility property that characterizes how clean and poisoned data interact with the training algorithm, specifically that training on clean data does not improve performance on poisoned data and vice-versa.
Revised: Clustering with Incompatibility and Self-Expansion. Section 2 defines incompatibility between subsets of data by their interaction with the training algorithm, specifically that training on one subset does not improve performance on the other.

Original: Leveraging this property, we show that forpoisoned datasets that satisfy the incompatibility property, the subset of the dataset that is “most compatible with itself” consists of either entirely clean or entirely poisoned data.
Revised: We prove that for datasets containing incompatible subpopulations, the subset which is most compatible with itself (as measured by the self-expansion error ) must be drawn from a single subpopulation.

Original: We show that iteratively identifying such subsets based on this property provably separates the dataset into clean and poisoned components.
Revised: Hence, iteratively identifying such subsets based on this property provably separates the dataset along the original subpopulations.

Original:  Defense Mechanism.
Revised: We present a tractable clustering mechanism that approximates this optimization objective. Formal Characterization and Defense for Data Poisoning Attacks.

Original: While the iterative technique presented in Section 3 comes with a guarantee that it will always place clean and poisoned data into separate subsets (if the clean and poisoned data satisfy the incompatibility property), it is based on optimizing an objective that is intractable in practice. Section 4 therefore presents a technique that uses a heuristic to approximate the optimization objective from Section 3, thus obtaining a tractable defense mechanism.
Revised: Section 3 presents a formal characterization of data poisoning attacks based on incompatibility, namely, that the poisoned data is incompatible with the clean data. We provide theoretical evidence to support this characterization by showing that a backdoor attack against linear classifiers provably satisfies the incompatibility property. We present a defense that leverages the clustering mechanism to partition the dataset into incompatible clean and poisoned subsets.

Original: Experimental Evaluation.
Revised: Experimental Evaluation.

Original: Section 5 presents an empirical evaluation of the  incompatibility property presented in Section 3 and the techniques developed in Section 4.
Revised: Section 4 presents an empirical evaluation of the ability of the incompatibility property presented in Section 2 and the techniques developed in Section 3 to identify and remove poisoned data to defend against backdoor data poisoning attacks.

Original: The evaluation focuses on three different backdoordata poisoning attacks (two dirty label attacks using pixel-based patches and image-based patches, respectively, and a clean-label attack using adversarial perturbations) from the data poisoning literature (Gu et al., 2019; Gao et al., 2019; Turner et al., 2018).
Revised: We focus on three different backdoor attacks (two dirty label attacks using pixel-based patches and image-based patches, respectively, and a clean-label attack using adversarial perturbations) from the data poisoning literature (Gu et al., 2019; Gao et al., 2019; Turner et al., 2018).

Original: The results 1) indicate that the considered attacks produce poisoned datasets that satisfy the incompatibility property, 2) highlight the ability of our defense mechanism to effectively separate clean and poisoned data, and 3) demonstrate the ability of our technique to effectively identify poisoned data within a larger dataset containing both clean and poisoned data.
Revised: The results (1) indicate that the considered attacks produce poisoned datasets that satisfy the incompatibility property and (2) demonstrate the effectiveness of our clustering mechanism in identifying poisoned data within a larger dataset containing both clean and poisoned data.

Original: For attacks against the GTSRB and CIFARdatasets, our defense mechanism reduces the attack success rate to below 1% on 134 out of thescenarios, with  less than 2% drop in clean accuracy.
Revised: For attacks against the GTSRB and CIFAR-10 datasets, our defense mechanism reduces the attack success rate to below 1% on 134 out of the 165 scenarios, with a less than 2% drop in clean accuracy.

Original: In a comparison with three previous defenses against a standard dirty-label backdoor attack on the CIFAR-10 dataset, our defense successfully defends against 22  ofthe 24 scenarios, versus only 5of the 24 scenarios for the next best defense. 
Revised: Compared to three previous defenses, our method successfully defends against the standard dirty-label backdoor attack in 22 out of 24 scenarios, versus only 5 for the next best defense. We have open sourced our implementation at https://github.com/charlesjin/ compatibility_clustering/ .

Original:  P ROBLEM S ETTING
Revised:  I NCOMPATIBILITY C LUSTERING

Original: We formulate the backdoor data poisoning threat model in a binary classification setting.
Revised: This section presents our clustering technique for datasets with subpopulations of incompatible data.

Original: We first establish some basic notation.
Revised: In our formulation, two subsets are incompatible when training on one subset does not improve performance on the other subset. Because we cannot measure performance on, e.g., holdout data with ground truth labels, we introduce a “self-expansion” property: given a set of data points, we first subsample the data to obtain a smaller training set, then measure how well the learning algorithm generalizes to the full set of data. Our main theoretical result is that a set which achieves optimal expansion is homogeneous, i.e., is composed entirely of data from a single subpopulation. Finally, we propose the Inverse Self-Paced Learning algorithm, which uses a heuristic approximation of the self-expansion objective to partition the dataset.

Revised: 2. S ETTING

Revised: We present our theoretical results in the context of a basic binary classification setting.

Original: Let X be the input space, Y = {− 1 , +1 } be the label space, and L ( · , · ) be the 0-1 loss function over Y × Y , i.e., L ( y 1 , y 2 ) =
Revised: Let X be the input space, Y = {− 1 , +1 } be the label space, and L ( · , · ) be the 0-1 loss function over Y × Y , i.e., L ( y 1 , y 2 ) =

Original: (1 − y 1 ∗ y 2 ) / 2 .
Revised: (1 − y 1 ∗ y 2 ) / 2 .

Original: Given a target distribution D supported on X × Y and a parametric family of functions f θ , the objective  is to find θ minimizing the population risk R ( θ ) :
Revised: Given a target distribution D supported on X × Y and a parametric family of functions f θ , a standard objective in this setting is to find θ minimizing the population risk R ( θ ) :

Original: = (cid:82) X × Y L ( f θ ( x ) , y ) dP D ( x, y ) .
Revised: = (cid:82) X × Y L ( f θ ( x ) , y ) dP D ( x, y ) .

Original: Given n data points D
Revised: Given n data points D

Original: = { ( x 1 , y 1 ) , . . .
Revised: = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) } , the empirical risk is defined as R emp ( θ ; D ) :
Revised: , ( x n , y n ) } , the empirical risk is defined as R emp ( θ ; D ) :

Original: = n − 1 (cid:80) ni
Revised: = n − 1 (cid:80) ni

Original: =1 L ( f θ ( x i ) , y i ) .
Revised: =1 L ( f θ ( x i ) , y i ) .

Original: Wealso fix a learning algorithm A ( · ) which takes as input a training set and returns parameters θ .
Revised: We fix a learning algorithm A ( · ) which takes as input a training set and returns parameters θ .

Original: For example, if f θ is a family of DNNs, then A might train a DNN via stochastic gradient descent.
Revised: For example, if f θ is a family of DNNs, then A might train a DNN via stochastic gradient descent.

Original: 2. T HREAT M ODEL
Revised: 2. S ELF - EXPANSION AND COMPATIBILITY

Revised: To measure generalization without holdout data and independent of any ground truth labels, we propose the following self-expanding property of sets: Definition 2.1 (Self-expansion of sets.) . Let S and T be sets ( S nonempty), and let α ∈ (0 , 1] denote the subsampling rate .

Revised: We define the α -expansion error of S given T as

Revised: [Equation]

Revised: where the expectation is over both the randomness in A and S ′ , a random variable that samples a uniformly random α fraction per class (rounded up) from S ∪ T .

Revised: When T = ∅ we also write ϵ ( S ; α ) .

Revised: Here α is a hyperparameter that is typically selected a priori on the basis of some domain knowledge (for example, if α = 1 and A trains a deep neural network, then the network can memorize the training set).

Revised: A typical value of α is 1 / 4 or 1 / 2 ; in our ablation studies, we find that our particular application is robust to a wide range of α .

Revised: The self-expansion property measures the ability of the learning algorithm A to generalize from a subset of S ∪ T to the empirical distribution of S .

Revised: Intuitively, when T = ∅ , a smaller expansion error means that the set S is both “easier” and “more homogeneous” with respect to the learning algorithm, as a random subset is sufficient to learn the contents of S .

Revised: When T is not empty, we expect that the self-expansion error will be lower when T contains similar data as S (for instance, if both S and T are drawn from the same distribution D ).

Revised: This observation leads directly to our formal definition of compatibility between sets:

Revised: Definition 2.2 (Compatibility of sets.) .

Revised: A set T is α -compatible with set S if

Revised: [Equation]

Revised: Conversely, T is α -incompatible with S if the opposite holds, i.e.,

Revised: [Equation]

Revised: Furthermore, T is completely α -compatible with S if every subset of T is α -compatible with every subset of S , and similarly for complete incompatibility.

Revised: In other words, T is (formally) compatible with S if T improves the ability of A to generalize to S .

Revised: 2. S EPARATION OF INCOMPATIBLE DATA

Revised: Our next result allows us to separate subpopulations of data satisfying the incompatibility property (proof in Appendix A).

Revised: The main idea is that, given any set, we can always achieve better selfexpansion by removing incompatible data (if it exists).

Revised: Theorem 2.3 (Sets minimizing expansion error are homogeneous.) .

Revised: Let S =

Revised: A ∪ B be a set with A and B nonempty and mutually completely α -incompatible.

Revised: Define

Revised: [Equation]

Revised: let S ∗ be the collection of subsets of S that achieve ϵ ∗ .

Revised: Then for any smallest subset S ∗ min ∈ S ∗ , either S ∗ min ⊆

Revised: A or S ∗ min ⊆ B .

Revised: Furthermore, if at least one of the incompatibilities is strict, then for all S ∗ ∈ S ∗ we have either S ∗ ⊆

Revised: A or S ∗ ⊆ B .

Revised: Theorem 2.3 suggests an iterative procedure for separating subsets of data that satisfy the incompatibility property.

Revised: Namely, at each step i , identify S ∗ min in the current dataset D i , then repeat the procedure with D

Original: {+i +
+}
Revised: {+i +
+}

Revised: = D i \ S ∗ min .

Original: {+The procedure terminates when D i +
+}
Revised: {+The procedure terminates when D i +
+}

Revised: = ∅ .

Revised: Assuming the subpopulations in the dataset satisfy the incompatibility property, this process partitions the training set into subsets that are guaranteed to contain data from a single subpopulation.

Revised: Furthermore, if subpopulations satisfy strict incompatibility, we can instead take the largest S ∗ ∈ S ∗ at each step.

Revised: The next section develops this insight into a practical clustering mechanism for incompatible subpopulations.

Revised: Input: training set S , total iterations N , annealing schedule 1 ≥ β 0 ≥ ... ≥ β N = β min > 0 , expansion α ≤ 1 , momentum η ∈ [0 , 1] , learning algorithm A , initial parameters θ

Revised: Output: S N ⊆ S such that | S N | = β min | S | 1: S 0 ← S 2: L ←3: for t = 1 to N do 4:

Revised: S ′ ← S AMPLE ( S t − 1 , α ) 5: θ t ← A ( S ′ , θ t − 1 ) 6:

Revised: L ← ηL + (1 − η ) R emp

Revised: ( θ t ; S ) 7:

Revised: S t ← T RIM ( L, β t ) 8: end for

Revised: 2. I NVERSE SELF - PACED LEARNING

Revised: A major question raised by Theorem 2.3 is how to identify the set S ∗ .

Revised: In general, even computing a single self-expansion error of a given set S is intractable as it involves taking the expectation over all subsets of size α | S | .

Revised: The Inverse Self-Paced Learning ( ISPL ) algorithm presented below solves this intractibilty problem.

Revised: Rather than optimizing over all possible subsets of the training data, we instead seek to minimize the expansion error over subsets of fixed size β | S | .

Revised: The new optimization objective is defined as:

Revised: [Equation]

Revised: We alternate between optimizing parameters θ t and the training subset S t —given S t − 1 , we update θ t using a single subset from S t − 1 of size α | S t − 1 | .

Revised: Then we use θ t to compute the loss for each element in S , and set S t to be the β fraction of the samples with the lowest losses.

Revised: To encourage stability of the learning algorithm, the losses are smoothed by a momentum term η .

Revised: To encourage more global exploration in the initial stages, we anneal the parameter β from an initial value βdown to the target value β min .

Revised: Algorithm 1 presents the full algorithm.

Revised: S AMPLE takes a set S and samples an α fraction of each class uniformly at random; T RIM takes losses L and returns the β fraction with the lowest loss.

Revised: To conclude, we show for certain choices of parameters that Algorithm 1 converges to a local optimum of the following objective over the training set S :

Revised: [Equation]

Revised: where S t is a subset of S and β t is decreasing in t (proof in Appendix A).

Revised: Proposition 2.4.

Revised: Let α = 1 and η = 0 in the setting of Algorithm 1, and assume that A returns the empirical risk minimizer.

Revised: Then we have that for each round of the algorithm, F ( θ t , S t ; β t ) is decreasing in t and furthermore, | F ( θ t , S t ; β t ) − F ( θ t +1 , S t +1 ; β t +1 )

Revised: | t →∞ −−−→ 0 .

Revised:  D EFENDING A GAINST B ACKDOOR A TTACKS

Revised: We present a novel characterization of data poisoning attacks based on the formal definition of compatibility introduced in Section 2, specifically, that the poisoned and clean data are incompatible.

Revised: We show that existing backdoor attacks produce poisoned data that is provably incompatible with the clean data in the context of linear classifiers.

Revised: As Theorem 2.3 yields a method of partitioning the dataset into clusters of clean and poisoned data (when the clean and poisoned data are incompatible), we present a boosting algorithm to identify which clusters contain clean data, and hence remove the poisoned data.

Revised: 3. T HREAT M ODEL

Original: The attacker’s goal is to install a backdoor in the trained network that 1) changes the poisoned network’s predictions on poisoned data while 2) preserving the accuracy of the poisoned network on clean data.
Revised: The attacker’s goal is to install a backdoor in the trained network that 1) changes the poisoned network’s predictions on poisoned data while 2) preserving the accuracy of the poisoned network on clean data.

Original: For the backdoor to be meaningful, the behavior of a poisoned model on poisoned data should also be inconsistent with a model trained on clean data.
Revised: For the backdoor to be meaningful, the behavior of a poisoned model on poisoned data should also be inconsistent with a model trained on clean data.

Original: We work with the standard model of statistical learning in which a set of n clean training samples D is drawn iid from D .
Revised: To begin, a set of n clean training samples D is drawn iid from D .

Original: The attacker selects at most n/ 2 − 1 samples A ⊆ D (leaving behind
Revised: After observing D , the attacker selects at most n/ 2 − 1 samples A ⊆ D (leaving behind

Original: at least n/ 2 clean samples C = D \ A ) and fixes a pair of perturbation functions τ train : X × Y (cid:55)→ X
Revised: at least n/ 2 clean samples C = D \ A ) and fixes a pair of perturbation functions τ train : X × Y (cid:55)→ X

Original: × Y (the attacker uses τ train to construct the poisoned dataset for training, changing the image and potentially the label of each poisoned element) and τ test : X (cid:55)→ X (the attacker uses τ test to construct poisoned test images without changing the ground-truth label of the poisoned image).
Revised: × Y (the attacker uses τ train to construct the poisoned dataset for training, changing the image and potentially the label of each poisoned element) and τ test : X (cid:55)→ X (the attacker uses τ test to construct poisoned test images without changing the ground-truth label of the poisoned image).

Original: The attacker is allowed (but not required) to pick A , τ train , and τ test after observing D .

Original: The attacker then poisons the samples in A to obtain poisoned samples P = { τ train ( a ) | a ∈ A } .
Revised: The attacker then poisons the samples in A to obtain poisoned samples P = { τ train ( a ) | a ∈ A } .

Original: The threat model places no restrictions on τ train , but requires that τ test should not affect the predictions of a network trained on C .
Revised: The threat model places no restrictions on τ train , but requires that τ test should not affect the predictions of a network trained on C .

Original: The final poisoned training set is C ∪ P .
Revised: The final poisoned training set is C ∪ P .

Original: Let θ be the parameters of the network under attack.
Revised: Let θ be the parameters of the network under attack.

Original: Given m fresh test samples T drawn iid from
Revised: Given m fresh test samples T drawn iid from

Original: D , the attacker seeks to maximize the targeted misclassification rate (TMR):
Revised: D , the attacker seeks to maximize the targeted misclassification rate (TMR):

Original: [Equation]
Revised: [Equation]

Original: i.e., the attack succeeds if it can flip the label of a correctly-classified instance by applying the trigger τ test ( · ) at test time.
Revised: i.e., the attack succeeds if it can flip the label of a correctly-classified instance by applying the trigger τ test ( · ) at test time.

Original: This metric captures both the “hidden” nature of the backdoor (the first term rewards the attacker only if clean inputs are correctly classified) as well as control upon application of the trigger (the second term rewards the attacker only if poisoned inputs are misclassified).
Revised: This metric captures both the “hidden” nature of the backdoor (the first term rewards the attacker only if clean inputs are correctly classified) as well as control upon application of the trigger (the second term rewards the attacker only if poisoned inputs are misclassified).

Original: 2. D EFENSE C APABILITIES AND O BJECTIVE
Revised: 3.1. D EFENSE C APABILITIES AND O BJECTIVE

Original: The defender is given only the poisoned dataset C ∪ P .
Revised: The defender is given only the poisoned dataset C ∪ P .

Original: In particular, τ train , τ test , A , and C are not available to the defender.
Revised: In particular, τ train , τ test , A , and C are not available to the defender.

Original: The defender’s objective is to return a sanitized dataset ˜ D ⊆ C ∪ P such that the learned parameters ˜ θ :
Revised: The defender’s objective is to return a sanitized dataset ˜ D ⊆ C ∪ P such that the learned parameters ˜ θ :

Original: =
Revised: =

Original: A ( ˜ D ) are as close as possible to the ground-truth parameters θ ∗ :
Revised: A ( ˜ D ) are as close as possible to the ground-truth parameters θ ∗ :

Original: = A ( C ) .
Revised: = A ( C ) .

Original: Note that θ ∗ is guaranteed to exhibit low targeted misclassification rate—the threat model requires that for fresh ( x, y ) drawn from D , f θ ∗ ( τ test ( x )) =
Revised: Note that θ ∗ is guaranteed to exhibit low targeted misclassification rate—the threat model requires that for fresh ( x, y ) drawn from D , f θ ∗ ( τ test ( x )) =

Original: y with high probability .
Revised: y with high probability.

Original: I NCOMPATIBILITY AND B ACKDOOR A TTACKS

Original: We next define a formal notion of incompatibility which captures the intuition that poisoned data should not improve performance on clean data (and vice versa).

Original: One technical challenge is that, in the setting of Section 2.2, the defender has no access to any ground-truth labels.

Original: We therefore work with a “self-expansion” property: given a set of data points, we first subsample the data to obtain a smaller training set, then measure how well the learning algorithm generalizes to the full set of data.

Original: Then, by leveraging the incompatibility property, we prove that a set that achieves optimal expansion is homogeneous, i.e., is composed entirely of either clean or poison data.

Original: 3. S ELF - EXPANSION AND COMPATIBILITY

Original: We begin by formally characterizing the self-expanding property of sets, which allows us to measure a notion of generalization without holdout data and independent of the ground truth labels:

Original: Definition 3.1 (Self-expansion of sets.) .

Original: Let S and T be sets ( S nonempty), and let α ∈ (0 , 1] denote the subsampling rate .

Original: We define the α -expansion error of S given T as

Original: [Equation]

Original: where the expectation is over both the randomness in A and S ′ , a random variable that samples a uniformly random α fraction per class (rounded up) from S ∪ T .

Original: When T = ∅ we also write ϵ ( S ; α ) .

Original: Here α is a hyperparameter that is typically selected a priori on the basis of some domain knowledge (for instance, if one takes α = 1 and A trains a deep neural network, then memorization of the training set can occur).

Original: A typical value of α is 1 / 4 or 1 / 2 ; in our ablation studies, we find that our particular application is robust to a wide range of α .

Original: The self-expansion property measures the ability of the learning algorithm A to generalize from a subset of S ∪ T to the empirical distribution of S .

Original: Intuitively, when T = ∅ , a smaller expansion error means that the set S is both “easier” and “more homogeneous” with respect to the learning algorithm, as a random subset is sufficient to learn the contents of S .

Original: When T is not empty, we expect that the self-expansion error will be lower when T contains similar data as S (for instance, if both S and T are drawn from the same distribution D ).

Original: This observation leads directly to our formal definition of compatibility between sets:

Original: Definition 3.2 (Compatibility of sets.) .

Original: [-A set T is α -compatible with set S with margin δ ≥ 
-]
Revised: [-A set T is α -compatible with set S with margin δ ≥ 
-]

Original: if

Original: [Equation]

Original: ; α ) .

Original: [Equation]

Original: Conversely, T is α -incompatible with S if the opposite holds, i.e.,

Original: [Equation]

Original: Furthermore, T is completely α -compatible with S if every subset of T is α -compatible with every subset of S , and similarly for complete incompatibility.

Original: In other words, T is (formally) compatible with S if T improves the ability of A to generalize to S .

Original: 3. M ODELING B ACKDOOR P OISONING A TTACKS AS I NCOMPATIBLE D ATA
Revised: C HARACTERIZING B ACKDOOR P OISONING A TTACKS U SING I NCOMPATIBILITY

Original: We next introduce our formal model of backdoor data poisoning attacks.

Original: Recall that our main intuitionis that increasing the amount of clean data in the training set should not improve the performance on poisoned data, and vice versa:
Revised: The following property formalizes the intuition that increasing the amount of clean data in the training set should not improve the performance on poisoned data (and vice versa):

Original: Definition 3.3 (Incompatibility Property) .
Revised: Definition 3.1 (Incompatibility Property) .

Original: Let C and P be the clean and poisoned data produced by an attacker according to threat model defined in Section 2.1.
Revised: Let C and P be the clean and poisoned data produced by an attacker according to threat model defined in Section 3.1.

Original: We say that the attacker satisfies the incompatibility property if C and P are mutually completely incompatible.
Revised: We say that the attacker satisfies the incompatibility property if C and P are mutually completely incompatible.

Original: Furthermore if at least one of the incompatibilities between C and P is strict, then the attacker satisfies the strict incompatibility property .
Revised: Furthermore if at least one of the incompatibilities between C and P is strict, then the attacker satisfies the strict incompatibility property .

Original: Note that incompatibilityhere is defined with respect to the learning algorithm
Revised: Note that incompatibility is defined with respect to the learning algorithm

Original: A as well as the sub-sampling rate α .
Revised: A as well as the subsampling rate α .

Original: We have found that α ∈ [1 / 4 , 1 / 2] works well in practice. We next present a backdoor attack against linear classifiers which  provably satisfies the incompatibility property.
Revised: To motivate this definition, we exhibit a backdoor attack against linear classifiers which falls under the threat model in Section 3.1 and provably satisfies the incompatibility property:

Original: Theorem 3.4.
Revised: Theorem 3.2.

Original: Let the input space be X
Revised: Let the input space be X

Original: = R N ( N > 1 ).
Revised: = R N ( N > 1 ).

Original: We use the parametric family of functions f w,b consisting of linear separators given by f w,b ( x ) = sign( ⟨ w, x ⟩ + b ) for w ∈ R N , b ∈ R .
Revised: We use the parametric family of functions f w,b consisting of linear separators given by f w,b ( x ) = sign( ⟨ w, x ⟩ + b ) for w ∈ R N , b ∈ R .

Original: A selects the maximum margin classifier.
Revised: A selects the maximum margin classifier.

Original: Then there exists a distribution D and perturbation functions τ train and τ test , such that if the adversary poisons 1 / 4 of each class at random, then with high probability over the initial training samples:
Revised: Then there exists a distribution D and perturbation functions τ train and τ test , such that if the adversary poisons 1 / 4 of each class at random, then with high probability over the initial training samples:

Original: Given the clean dataset, A returns parameters with 0 empirical (and population) risk.
Revised: Given the clean dataset, A returns parameters with 0 empirical (and population) risk.

Original: Given the poisoned dataset, A returns parameters with a targeted misclassification rate of 1. 3.
Revised: Given the poisoned dataset, A returns parameters with a targeted misclassification rate of 1. 3.

Original: The clean and poisoned data are mutually completely incompatible.
Revised: The clean and poisoned data are mutually completely incompatible

Original: The proof is deferred to Appendix A.
Revised: The proof is in Appendix A.

Original: The idea is to construct a dataset such that there exist dimensions in the original, clean feature space that are uncorrelated with the true labels, i.e., consist of pure noise.
Revised: The idea is to construct a dataset such that there exist dimensions in the original, clean feature space that are uncorrelated with the true labels, i.e., consist of pure noise.

Original: The backdoor data poisoning attack makes use of these extra dimensions by replacing the noise with a backdoor signal.
Revised: The attacker makes use of these extra dimensions by replacing the noise with a backdoor signal.

Original: Althoughsimplified, this  setting is analogous to many existing attacks againstDNNs.
Revised: Although this result uses linear classifiers, the setting is analogous to many existing attacks against

Original: For example, a common perturbation function for backdoor attacks against image classifiers inserts a small synthetic patch on the border of the image, which, by design, is a location that does not affect the original classification task.Hence, the labels for clean and poisoned data depend on disjoint portions of the image. 3. S EPARATION OF I NCOMPATIBLE D ATA
Revised: DNNs, e.g., backdoor attacks against image classifiers that insert synthetic patches on the border of the image, which, by design, is a location that does not affect the original classification task.

Original: We next state a key result that allows us to separate clean and poisoned data satisfying the incompatibility property (proof in Appendix A).

Original: The main idea is that, given any set, we can always achieve better self-expansion by removing incompatible data (if it exists).

Original: Theorem 3.5 (Sets minimizing expansion error are homogeneous.) .

Original: Let S =

Original: A ∪ B be a set with A and B nonempty and mutually completely α -incompatible.

Original: Define [Equation] let S ∗ be the collection of subsets of S that achieve ϵ ∗ . Then for any smallest subset S ∗ min ∈ S ∗ , either S ∗ min ⊆
Revised: Input: Homogeneous sets S 1 , ..., S N , number of samples n = (cid:80) Ni =1 | S i | , learning algorithm

Original: Aor S ∗ min ⊆ B . Furthermore, if at least one of the incompatibilities is strict, then for all S ∗ ∈ S ∗ we have either S ∗ ⊆ A or S ∗ ⊆ B . Theorem 3.5 suggests an iterative procedure for separating clean and poisoned data that satisfy the incompatibility property.
Revised: A

Original: Namely, at each step i , identify S ∗ min in the current dataset D i , then repeat Input: Homogeneous sets S 1 , , S N  , number samples n = (cid:80) Ni =  , learning algorithm A
Revised: Output: Votes V 1 , ..., V N 1: C 1 , ..., C N ←2: for i = 1 to N do 3: θ i ← A ( S i ) 4: for k = 1 to N do 5: if L OSS 0 , 1 ( θ i ; S k ) <

Original: Output: do 3: do 5: then 6:
Revised: | S k | / 2 then 6:

Original: C  ← C  S 
Revised: C k ← C k + | S i

Original: | 7:  if 8:  for 10: for i = 1 to N do 11:
Revised: | 7: end if 8: end for 9: end for 10: for i = 1 to N do 11:

Original: V
Revised: V

Original: i ← C i > n/12:  for[-the procedure with D i +
 = D i \ S ∗ min . The procedure terminates when D i +
-]
Revised: i ← C i > n/12: end for[-the procedure with D i +
 = D i \ S ∗ min . The procedure terminates when D i +
-]

Original: = ∅ .

Original: Assuming the attacker satisfies the incompatibility property, this process partitions the training set into subsets that are guaranteed to contain either entirely clean or entirely poisoned data.

Original: If the attacker satisfies strict incompatibility, we can instead take the largest S ∗ ∈ S ∗ at each step.

Original: The next section leverages this insight to develop a practical defense mechanism against backdoor data poisoning attacks.

Original: D EFENDING A GAINST B ACKDOOR A TTACKS

Original: We next present a tractable defense against backdoor data poisoning attacks under the threat model specified in Section 2 for attackers that satisfy the incompatibility property.

Original: In particular, although

Original: Theorem 3.5 yields a method of partitioning the dataset into clean and poisoned components, it remains to identify which components are clean.

Original: Additionally, at each iteration, we need to solve an optimization problem which is intractable in general.

Original: The following sections resolve these questions.

Original: 4. I DENTIFICATION USING B OOSTING
Revised: 3. I DENTIFICATION USING B OOSTING

Original: We first consider the task of identifying the clean data from a collection of homogeneous (i.e., entirely clean or entirely poisoned) subsets.
Revised: We conclude this section by showing how to identify the clean data given the output of our clustering mechanism, which is a collection of homogeneous (i.e., entirely clean or entirely poisoned) subsets.

Original: One idea is to measure the compatibility between each pair of subsets, then take the largest mutually compatible (sub)collection of subsets (under the assumption that the clean data is always compatible with itself).
Revised: One idea is to measure the compatibility between each pair of subsets, then take the largest mutually compatible (sub)collection of subsets (under the assumption that the clean data is always compatible with itself).

Original: However in general this requires training on each subset multiple times in sequence.
Revised: However in general this requires training on each subset multiple times in sequence.

Original: We instead propose a simplified boosting algorithm for identifying the clean data which involves training on each subset exactly once.
Revised: We instead propose a simplified boosting algorithm for identifying the clean data which involves training on each subset exactly once.

Original: The main idea is to fit a weak learner to each component, then use each weak learner to vote on the other components.
Revised: The main idea is to fit a weak learner to each cluster, then use each weak learner to vote on the other clusters.

Original: Before we begin, we establish sufficient conditions for the  boosting algorithm to succeed:
Revised: The following sufficient conditions guarantee the success of our boosting algorithm:

Original: Property 4.1 (Compatibility property of disjoint sets) .
Revised: Property 3.3 (Compatibility property of disjoint sets) .

Original: Let C ′ and C ′′ be any two disjoint sets of clean data, and let P ′ be a set of poisoned data.
Revised: Let C ′ and C ′′ be any two disjoint sets of clean data, and let P ′ be a set of poisoned data.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The first inequality says that training and testing on disjoint sets of clean data is an improvement over random guessing.
Revised: The first inequality says that training and testing on disjoint sets of clean data is an improvement over random guessing.

Original: The second inequality says that training with clean data then testing on poisoned data is at best equivalent to random guessing; this property is natural in that the threat model already guarantees R emp ( A ( C ); P ) is large (i.e., close to 1) for random poisoned data P .
Revised: The second inequality says that training with clean data then testing on poisoned data is at best equivalent to random guessing; this property is natural in that the threat model already guarantees R emp ( A ( C ); P ) is large (i.e., close to 1) for random poisoned data P .

Original: Algorithm 1 presents our approach for boosting from homogeneous sets.
Revised: Algorithm 2 presents our approach for boosting from homogeneous sets.

Original: The subroutine L OSS 0 ,takes a set of parameters and a set S , and returns the total risk over S using the zero-one loss.
Revised: The subroutine L OSS 0 ,takes a set of parameters and a set S , and returns the total risk over S using the zero-one loss.

Original: Note that votes are weighted by size.

Original: The main result of this section is the  correctness of Algorithm 1; the proof is deferred to Appendix A.
Revised: We have the following correctness guarantee (proof in Appendix A):

Original: Theorem 4.2 (Identification of clean samples) .
Revised: Theorem 3.4 (Identification of clean samples) .

Original: Let S 1 , ..., S N be a collection of homogeneous subsets containing either entirely clean or entirely poisoned data.
Revised: Let S 1 , ..., S N be a collection of homogeneous subsets containing either entirely clean or entirely poisoned data.

Original: Then if the clean and poisoned data satisfy Property 4.1, Algorithm 1 votes V i
Revised: Then if the clean and poisoned data satisfy Property 3.3, Algorithm 2 votes V i

Original: =
Revised: =

Original: True if S i is clean (and V i
Revised: True if S i is clean (and V i

Original: =
Revised: =

Original: False otherwise).
Revised: False otherwise).

Original: Input: training set S , total iterations N , annealing schedule 1 ≥ β 0 ≥ ... ≥ β N = β min > 0 , expansion α ≤ 1 , momentum η ∈ [0 , 1] , learning algorithm A , initial parameters θ

Original: Output: S N ⊆ S such that | S N | = β min | S | 1: S 0 ← S 2: L ←3: for t = 1 to N do 4:

Original: S ′ ← S AMPLE ( S t − 1 , α ) 5: θ t ← A ( S ′ , θ t − 1 ) 6:

Original: L ← ηL + (1 − η ) R emp

Original: ( θ t ; S ) 7:

Original: S t ← T RIM ( L, β t ) 8: end for

Original: 4. I NVERSE SELF - PACED LEARNING

Original: A major question raised by Theorem 3.5 is how to identify the set S ∗ .

Original: In general, even computing a single self-expansion error of a given set S is intractable as it involves taking the expectation over all subsets of size α | S | .

Original: The Inverse Self-Paced Learning ( ISPL ) algorithm presented below solves this intractibilty problem.

Original: Rather than optimizing over all possible subsets of the training data, we instead seek to minimize the expansion error over subsets of fixed size β | S | .

Original: The new optimization objective is defined as:

Original: [Equation]

Original: We alternate between optimizing parameters θ t and the training subset S t —given S t − 1 , we update θ t using a single subset from S t − 1 of size α | S t − 1 | .

Original: Then we use θ t to compute the loss for each element in S , and set S t to be the β fraction of the samples with the lowest losses.

Original: To encourage stability of the learning algorithm, the losses are smoothed by a momentum term η .

Original: To encourage more global exploration in the initial stages, we anneal the parameter β from an initial value βdown to the target value β min .

Original: Algorithm 2 presents the full algorithm.

Original: S AMPLE takes a set S and samples an α fraction of each class uniformly at random; T RIM takes losses L and returns the β fraction with the lowest loss.

Original: To conclude, we show for certain choices of parameters that Algorithm 2 converges to a local optimum of the following objective over the training set S :

Original: [Equation]

Original: where S t is a subset of S and β t is decreasing in t (proof in Appendix A).

Original: Proposition 4.3.

Original: Let α = 1 and η = 0 in the setting of Algorithm 2, and assume that A returns the empirical risk minimizer.

Original: Then we have that for each round of the algorithm, F ( θ t , S t ; β t ) is decreasing in t and furthermore, | F ( θ t , S t ; β t ) − F ( θ t +1 , S t +1 ; β t +1 )

Original: | t →∞ −−−→ 0 .

Original:  E XPERIMENTAL EVALUATION
Revised:  E XPERIMENTAL EVALUATION

Original: We report empirical evaluations of the incompatibility property (Section 5.1) and the proposed defense (Section 5.2).
Revised: We report empirical evaluations of the incompatibility property (Section 4.1) and the proposed defense (Section 4.2).

Original: We use the CIFAR-10 dataset (Krizhevsky & Hinton, 2009) for general image recognition, containing 10 classes of 5000 training and 1000 test images each, and the GTSRB dataset (Stallkamp et al., 2012) for traffic sign recognition, containing 43 classes with 26640 training and 12630 test images. The GTSRB training set is highly imbalanced, with classes ranging from 150 to 1500 instances.
Revised: We use the CIFAR-10 dataset (Krizhevsky & Hinton, 2009) for general image recognition, containing 10 classes of 5000 training and 1000 test images each, and the GTSRB dataset (Stallkamp et al., 2012) for traffic sign recognition, containing 43 classes with 26640 training and 12630 test images, respectively.

Original: All experiments presented in the main text use a ResNet32 architecture.

Original: Our experiments use three types of backdoor attacks.
Revised: Our experiments consider three types of backdoor attacks.

Original: Dirty label backdoor (DLBD) uses small patches ranging from a single pixel to a 3x3 checkerboard pattern, using the implementation of Tran et al.
Revised: Dirty label backdoor (DLBD) attacks use small patches ranging from a single pixel to a 3x3 checkerboard pattern, following the implementation of Tran et al.

Original: W ATERMARK uses 8x8 images such as a copyright sign or a peace sign, following Gao et al.
Revised: W ATERMARK uses 8x8 images such as a copyright sign or a peace sign, taken from Gao et al.

Original: Both of these attacks insert a patch into a training image from a source class, then change the training label to the target class.
Revised: Both of these attacks insert a patch into a training image from a source class, then change the training label to the target class.

Original: The attacker’s objective is to induce the learner to misclassify images of the source class as the target class upon application of the patch.
Revised: The attacker’s objective is to induce the learner to misclassify images of the source class as the target class upon application of the patch.

Original: The final strategy, clean label backdoor (CLBD), uses the poisoned datasets from Turner et al.
Revised: Clean label backdoor (CLBD) attacks use the poisoned datasets from Turner et al.

Original: (2018), which are only available for CIFAR-10; this attack first strongly perturbs each image of the source class independently using either GAN, ℓ 2 , or ℓ ∞ bounds then inserts several patches (without changing the label). Here, the attacker objective is to make the learner misclassify an image from any non-source class as the source class by applying the patches.
Revised: (2018), which are only available for CIFAR-10. Each attack first strongly perturbs each image of the source class independently using either GAN, ℓ 2 , or ℓ ∞ bounds, then inserts several patches (without changing the label); the attacker’s objective is to make the learner misclassify an image from any non-source class as the source class by applying the patches.

Original: Figure 1 presents examples from each dataset.
Revised: Figure 1 presents examples from each dataset.

Original: Additional dataset construction details can be found in Appendix B.  Full experimental details and additional results, including performance against an adaptive white-box attacker and experiments testing the sensitivity of ISPL to its hyperparameters, are contained in Appendices C and D.
Revised: Additional dataset construction details can be found in Appendix B. All experiments presented in the main text use a PreActResNet-18 architecture. Full experimental details and additional results, including performance against an adaptive white-box attacker and experiments testing the sensitivity of ISPL to its hyperparameters, are contained in Appendices C and D.

Original: 5. I NCOMPATIBILITY BETWEEN C LEAN AND P OISONED D ATA
Revised: 4. I NCOMPATIBILITY BETWEEN C LEAN AND P OISONED D ATA

Original: We evaluate whether the attacks satisfy the incompatibility property by measuring the gap between ϵ ( C ; α ) and ϵ ( C | P ; α ) for clean data C and poisoned data P using the 3 strategies (DLBD, W ATERMARK , CLBD) on CIFAR-10.
Revised: We evaluate whether the attacks satisfy the incompatibility property by measuring the gap between ϵ ( C ; α ) and ϵ ( C | P ; α ) for clean data C and poisoned data P generated by the 3 attacks (DLBD, W ATERMARK , CLBD) on CIFAR-10.

Original: Figure 2 presents selected plots for two scenarios: DLBD (e.g., relabelling airplanes as birds), andCLBD using GAN-based perturbations (e.g., perturbing airplanes toward other classes).
Revised: Figure 2 presents selected plots for the DLBD (e.g., relabelling airplanes as birds) and GAN-based CLBD (e.g., perturbing airplanes toward other classes) scenarios.

Original: Along the x-axis, we increase the ratio of poisoned to clean data in the source class from 0 to 1.
Revised: Along the x-axis, we increase the ratio of poisoned to clean data in the source class from 0 to 1.

Original: To create the training data, we first sample 1 / 8 of each class in the clean dataset to create C .
Revised: Our experimental procedure is as follows: we first sample 1 / 8 of each class in the clean dataset to create C .

Original: For each poison ratio, we sample 5 subsets of poisoned data according to the poison ratio, add it to C , measure the test accuracies after training at α = 1 / 2 , then take the mean  as the self-expansion error.
Revised: For each poison ratio, we sample P ′ ∼ P according to the poison ratio, train on P ′ ∪ C with α = 1 / 2 , then take the mean over 5 samples of P ′ as the self-expansion error.

Original: The entire process is repeated for 5 samples of C for each poisoned dataset in our evaluations (8 total for DLBD, 5 total for GAN-based CLBD).
Revised: The entire process is repeated for 5 samples of C for each poisoned dataset in our evaluations (8 total for DLBD, 5 total for GAN-based CLBD).

Original: Since the salient measurement is ϵ ( C | P ; α ) − ϵ ( C ; α ) , we report each metric as an absolute difference in accuracy from the unpoisoned case (ratio = 0).
Revised: To better present the gap ϵ ( C | P ; α ) − ϵ ( C ; α ) , we report each metric as an absolute difference in accuracy from the unpoisoned case (ratio = 0).

Original: A trendline connects the medians.
Revised: A trendline connects the medians.

Original: [Table]
Revised: [Table]

Original: Our first observation is that the key quantity, ϵ ( C | P ; α ) − ϵ ( C ; α ) , is negative for all poisoned datasets, which empirically supports our insight that clean and poisoned data are incompatible (or even strictly incompatible), despite the qualitative difference behind the attack mechanisms.
Revised: Our first observation is that the key quantity, ϵ ( C | P ; α ) − ϵ ( C ; α ) , is negative for all poisoned datasets, which empirically supports our insight that clean and poisoned data are incompatible (or even strictly incompatible), despite the qualitative difference behind the attack mechanisms.

Original: The trend in the top plot also indicates that the gap between ϵ ( C | P ; α ) and ϵ ( C ; α ) increases with the amount of poisoned data.
Revised: The trend in the top plot also indicates that the gap between ϵ ( C | P ; α ) and ϵ ( C ; α ) increases with the amount of poisoned data.

Original: The next two plots show how the gap breaks down over the source and non-source classes.
Revised: The next two plots show how the gap breaks down over the source and non-source classes.

Original: For the DLBD attack, the source class shows a strong negative trend in accuracy as the amount of poison increases, while accuracy on the non-source classes stays largely constant.
Revised: For the DLBD attack, the source class shows a strong negative trend in accuracy as the amount of poison increases, while accuracy on the non-source classes stays largely constant.

Original: This fact is consistent with the hypothesis that the poisoned data is impairing the accuracy of the classifier on clean data from the source class (e [Table] g., clean airplanes are getting misclassified as birds).
Revised: This fact is consistent with the hypothesis that the poisoned data is impairing the accuracy of the classifier on clean data from the source class (e.g., clean airplanes are getting misclassified as birds).

Original: For the CLBD attack, even though the total clean accuracy decreases, the source class accuracy actually increases as the amount of poison increases.
Revised: For the CLBD attack, even though the total clean accuracy decreases, the source class accuracy actually increases as the amount of poison increases.

Original: We attribute this phenomenon to the harder poisoned training instances insource class making the classifier more robust on  clean instances as well.
Revised: We attribute this phenomenon to the classifier learning the perturbed poisoned training instances in the source class, which yields improved accuracy on the clean instances as well.

Original: The decrease in clean accuracy is therefore due to a drop in accuracy among the nonsource classes.
Revised: The decrease in clean accuracy is therefore due to a drop in accuracy among the non-source classes.

Original: Because the CLBD attack produces perturbations of the source class toward other classes (thus making the other classes more similar to, e.g., airplanes), this behavior is consistent with the classifier misclassifying clean instances of non-source classes as the source class.
Revised: Because the CLBD attack produces perturbations of the source class toward other classes (thus making the other classes more similar to, e.g., airplanes), this behavior is consistent with the classifier misclassifying clean instances of non-source classes as the source class.

Original: These observations are corroborated by the increase in accuracy on poisoned data in both cases, suggesting that the classifier’s mistakes on clean data are correlated with increased accuracy on similar-looking poisoned data.
Revised: These observations are corroborated by the increase in accuracy on poisoned data in both cases, suggesting that the classifier’s mistakes on clean data are correlated with increased accuracy on similar-looking poisoned data.

Original: Appendix D contains further discussion and results; violations of incompatibility are rare and small in magnitude.
Revised: Appendix D contains further discussion and results.

Original: 5. P ERFORMANCE OF P ROPOSED D EFENSE
Revised: 4. P ERFORMANCE OF P ROPOSED D EFENSE

Original: We next evaluate the performance of our proposed defense on the full range of attacks.
Revised: We next evaluate the performance of our proposed defense on the full range of attacks.

Original: Most settings use one-to-one attacks (e.g., place a trigger on dogs and mislabel as cats; an ϵ percentage of the source class is poisoned), which is the most common setting in the literature (Gu et al., 2019).
Revised: Most settings use one-to-one attacks (e.g., place a trigger on dogs and mislabel as cats; an ϵ percentage of the source class is poisoned), which is the most common setting in the literature (Gu et al., 2019).

Original: For the CIFAR-10 DLBD scenario, we additionally use all-to-one (e.g., place a trigger on all non-cats and mislabel as cats; an ϵ/ 9 percentage of each non-target class is poisoned) and all-to-all attacks (i.e., place a trigger on any image and mislabel according to a cyclic permutation on classes; an ϵ percentage of every class is poisoned).
Revised: For the CIFAR-10 DLBD scenario, we additionally use all-to-one (e.g., place a trigger on all non-cats and mislabel as cats; an ϵ/ 9 percentage of each non-target class is poisoned) and all-to-all attacks (i.e., place a trigger on any image and mislabel according to a cyclic permutation on classes; an ϵ percentage of every class is poisoned).

Original: Table 1 presents a summary of our results.
Revised: Table 1 presents a summary of our results.

Original: Each cell is of the format "success / total", where total indicates the total number of different scenarios for the attack setting (i.e., different source and target class and trigger combinations), and a run is successful when the defense achieves TMR below 1%.
Revised: Each entry is of the format “success / total”, where total indicates the total number of different scenarios for the attack setting (i.e., different source and target class and trigger combinations), and a run is successful when the defense achieves TMR below 1%.

Original: We note that the clean accuracy is around 91% on CIFAR-10 and 94% on GTSRB after running ISPL+B , compared to around 93% and 94% when training on the full, unpoisoned datasets, respectively.
Revised: We note that the clean accuracy is around 91% on CIFAR-10 and 94% on GTSRB after running ISPL+B , compared to around 93% and 94% when training on the full, unpoisoned datasets, respectively.

Original: These results indicate that our approach succeeds in defending against various backdoor poisoning attacks, particularly for the standard settings of ϵ = 5 , 10 in the literature (Tran et al., 2018), for a small cost in clean accuracy.
Revised: These results indicate that our approach succeeds in defending against various backdoor poisoning attacks, particularly for the standard settings of ϵ = 5 , 10 in the literature (Tran et al., 2018), for a small cost in clean accuracy.

Original: Appendix D contains the unabridged results.
Revised: Appendix D contains the unabridged results.

Original: Comparison with Existing Approaches.
Revised: Comparison with Existing Approaches.

Original: Figure 3 compares the performance of our defense with 3 existing defenses—Spectral Signatures (Tran et al., 2018), Iterative Trimmed Loss Minimization (Shen & Sanghavi, 2019), and Activation Clustering (Chen et al., 2018)—on the CIFAR-10 DLBD (1-to-1) scenario.
Revised: Figure 3 compares the performance of our defense with 3 existing defenses—Spectral Signatures (Tran et al., 2018), Iterative Trimmed Loss Minimization (Shen & Sanghavi, 2019), and Activation Clustering (Chen et al., 2018)—on the CIFAR-10 DLBD (1-to-1) scenario.

Original: ISPL+B outperforms all 3 baselines in terms of reducing the TMR.
Revised: ISPL+B outperforms all 3 baselines in terms of reducing the TMR.

Original: Table 2 in
Revised: Table 2 in

Original: Appendix D contains the detailed results of this experiment.
Revised: Appendix D contains the detailed results of this experiment.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Many prior works propose methods for defending against backdoor attacks on neural networks.
Revised: Many prior works propose methods for defending against backdoor attacks on neural networks.

Original: One common strategy is to first fit a deep network to the poisoned distribution, then analyze the learned representations.
Revised: One common strategy, which can be viewed as a type of deep clustering (Zhou et al., 2022), is to first fit a deep network to the poisoned distribution, then analyze the learned representations.

Original: The Activation Clustering defense (Chen et al., 2018) runs k-means clustering (k=2) on the activation patterns, then discards the smallest cluster.
Revised: The Activation Clustering defense (Chen et al., 2018) runs k-means clustering (k=2) on the activation patterns, then discards the smallest cluster.

Original: Tran et al.
Revised: Tran et al.

Original: (2018) propose a defense based on the assumption that clean and poisoned data are separated by their top eigenvalue in the spectrum of the activations.
Revised: (2018) propose a defense based on the assumption that clean and poisoned data are separated by their top eigenvalue in the spectrum of the activations.

Original: TRIM (Jagielski et al., 2021) and ITLM (Shen & Sanghavi, 2019) iteratively train on a subset of the data created by removing samples with high loss.
Revised: TRIM (Jagielski et al., 2021) and ITLM (Shen & Sanghavi, 2019) iteratively train on a subset of the data created by removing samples with high loss.

Original: However, each iteration removes the same small number of samples; both direct comparisons with ITLM and our ablation studies suggest that the dynamic resizing in ISPL is crucial to identifying the poison.
Revised: However, each iteration removes the same small number of samples; both direct comparisons with ITLM and our ablation studies suggest that the dynamic resizing in ISPL is crucial to identifying the poison.

Original: Shan et al.
Revised: Shan et al.

Original: (2022) is motivated by a similar intuition as our compatibility assumption;  their method iteratively (1) splits the dataset using k-means clustering (k=2) on the gradient of the final classification layer, then (2) discards the partition which yields a model with worse performance on a known poisoned input. One major difference is that we leverage our compatibility assumption for the key step of generating the clusters, whereas their method uses the intuition only to prune the clean partition at each iteration, and also crucially relies on access to a known poisoned input.
Revised: (2022) is motivated by a similar intuition as our compatibility assumption; however their formalization differs significantly from ours, and their defense crucially relies on access to a known poisoned input in order.

Original: Finally, several works provide certified guarantees against data poisoning attacks (Steinhardt et al., 2017; Jia et al., 2022); in exchange for the stronger guarantee, these works defend a small fraction of the dataset, e.g., the state-of-the-art certifies 16% of CIFAR-10 against 50 poisoned images (Wang et al., 2022).
Revised: Finally, several works provide certified guarantees against data poisoning attacks (Steinhardt et al., 2017; Jia et al., 2022); in exchange for the stronger guarantee, these works defend a small fraction of the dataset, e.g., the state-of-the-art certifies 16% of CIFAR-10 against 50 poisoned images (Wang et al., 2022a). We refer the reader to Wang et al. (2022b) for a comprehensive survey of data poisoning defenses.

Revised: Appendix E contains a discussion of other related works.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Backdoor data poisoning attacks on deep neural networks are an emerging class of threats in the growing landscape of deployed machine learning applications.
Revised: Backdoor data poisoning attacks on deep neural networks are an emerging class of threats in the growing landscape of deployed machine learning applications.

Original: We present a new defense against backdoor attacks based on a novel  incompatibility property of theinteraction between clean and poisoned data and the training algorithm.
Revised: We present a new, state-of-the-art defense against backdoor attacks by leveraging a novel clustering mechanism based on an incompatibility property of the clean and poisoned data.

Original: The experimental results show that previous dirty-label and clean-label backdoor attacks produce poisoned datasets that exhibit behavior consistent with the incompatibility property.
Revised: The experimental results show that existing dirtylabel and clean-label backdoor attacks produce poisoned datasets that exhibit behavior consistent with the incompatibility property.

Original: The results also highlight the effectiveness of our technique in defending against three  backdoor poisoning attacks while preserving clean accuracy.
Revised: The results also highlight the effectiveness of our technique in defending against three types of backdoor poisoning attacks while preserving clean accuracy.

Original:  A D EFERRED P ROOFSAND A DDITIONAL T HEORETICAL R ESULTS
Revised: Because our technique works with any class of trained models, we anticipate that our tools may be applied to a wide range of datasets and training objectives. More broadly, this work develops an underexplored perspective based on the interaction between the data and training process. A D EFERRED P ROOFS

Original: Proof of Theorem 3.4.
Revised: Proof of Theorem 3.2.

Original: First we give a construction when the dimension N = 2 .
Revised: First we give a construction when the dimension N = 2 .

Original: We define the training distribution as D =
Revised: We define the training distribution as D =

Original: ( D + + D − ) / 2 , where D + and D − are both distributed arbitrarily within a ball of radius 1 (e.g., as a truncated Gaussian); D + is centered at (10 , 0) with labels + and D − is centered at ( − 10 , 0) with labels − 1 .
Revised: ( D + + D − ) / 2 , where D + and D − are both distributed arbitrarily within a ball of radius 1 (e.g., as a truncated Gaussian); D + is centered at (10 , 0) with labels + and D − is centered at ( − 10 , 0) with labels − 1 .

Original: The adversary chooses the perturbation function τ which flips the label during training and sets the last entry to be 10 if the original label was +1 , and − 10 if the original label was − 1 .
Revised: The adversary chooses the perturbation function τ which flips the label during training and sets the last entry to be 10 if the original label was +1 , and − 10 if the original label was − 1 .

Original: Note that τ is bounded in the sense that as N → ∞ , the (expected) magnitude of a data point is O ( √ N ) while the magnitude of the perturbation is O (1).
Revised: Note that τ is bounded in the sense that as N → ∞ , the (expected) magnitude of a data point is O ( √ N ) while the magnitude of the perturbation is O (1)

Original: Figure 4 depicts the construction.
Revised: Figure 4 depicts the construction.

Original: Clearly, one sample from each clean class is sufficient to learn a maximum margin classifier (MMC) achieving zero population risk, i.e., the MMC will be close to w =
Revised: Clearly, one sample from each clean class is sufficient to learn a maximum margin classifier (MMC) achieving zero population risk, i.e., the MMC will be close to w =

Original: (1 , 0) , b = 0 .
Revised: (1 , 0) , b = 0 .

Original: Next, we show that the attacker succeeds under the threat model in Section 2.
Revised: Next, we show that the attacker succeeds under the threat model in Section 3.1.

Original: Recall that the attacker selects 1/4 of the positive samples and 1/4 of the negative samples to poison.
Revised: Recall that the attacker selects 1/4 of the positive samples and 1/4 of the negative samples to poison.

Original: Given a poisoned dataset containing at least one example of each class (both clean and poisoned), the classifier given by w = (1 , − 2) , b = 0 achieves perfect accuracy on both the clean data and the poisoned data, and hence the MMC also has a poison misclassification rate of 1.
Revised: Given a poisoned dataset containing at least one example of each class (both clean and poisoned), the classifier given by w = (1 , − 2) , b = 0 achieves perfect accuracy on both the clean data and the poisoned data, and hence the MMC also has a poison misclassification rate of 1.

Original: We now show that this setup satisfies the incompatibility property defined in Section 3.2.
Revised: We now show that this setup satisfies the incompatibility property defined in Section 3.2.

Original: We begin by showing that the poisoned data is not compatible with the clean data.
Revised: We begin by showing that the poisoned data is not compatible with the clean data.

Original: Let C be any set of clean data, and let P be any set of poisoned data.
Revised: Let C be any set of clean data, and let P be any set of poisoned data.

Original: We have two cases.
Revised: We have two cases.

Original: First, if C contains samples of both classes, then by definition, any subset C ′ of C in the measurement of the self-expansion error will also include at least one sample of each clean class.
Revised: First, if C contains samples of both classes, then by definition, any subset C ′ of C in the measurement of the self-expansion error will also include at least one sample of each clean class.

Original: Hence, the MMC learned from C ′ will be close to w
Revised: Hence, the MMC learned from C ′ will be close to w

Original: = (1 , 0) , b = 0 , which achieves perfect accuracy on all clean data.
Revised: = (1 , 0) , b = 0 , which achieves perfect accuracy on all clean data.

Original: On the other hand, if C contains only samples of the positive class, the MMC is w = (0 , 0) , b = +1 , and again all samples in C are correctly classified.
Revised: On the other hand, if C contains only samples of the positive class, the MMC is w = (0 , 0) , b = +1 , and again all samples in C are correctly classified.

Original: The same argument works when C contains only samples from the negative class.
Revised: The same argument works when C contains only samples from the negative class.

Original: Thus, in neither case can poisoned data improve the self-expansion error of C .
Revised: Thus, in neither case can poisoned data improve the self-expansion error of C .

Original: As the exact same line of reasoning holds for proving incompatibility of clean data with respect to poisoned data, we omit the proof.
Revised: As the exact same line of reasoning holds for proving incompatibility of clean data with respect to poisoned data, we omit the proof.

Original: Therefore the clean and poisoned data are mutually completely incompatible.
Revised: Therefore the clean and poisoned data are mutually completely incompatible.

Original: To generalize this to N > 2 , we can simply embed D in the first two dimensions of the space.
Revised: To generalize this to N > 2 , we can simply embed D in the first two dimensions of the space.

Original: The entire construction can also be scaled then translated and rotated without affecting any of the conclusions.
Revised: The entire construction can also be scaled then translated and rotated without affecting any of the conclusions.

Original: Before moving on to the proof of Theorem 3.5, we first prove a useful lemma:
Revised: Before moving on to the proof of Theorem 2.3, we first prove a useful lemma:

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Let A and B be arbitrary sets, and define S
Revised: Let A and B be arbitrary sets, and define S

Original: =
Revised: =

Original: A ∪ B .
Revised: A ∪ B .

Original: Then for all α ,
Revised: Then for all α ,

Original: [Equation]
Revised: [Equation]

Original: with the convention that ϵ ( ∅| T ; α ) = 0 for all T and α .
Revised: with the convention that ϵ ( ∅| T ; α ) = 0 for all T and α .

Original: Proof.
Revised: Proof.

Original: Notice that all the expansion errors sample from the same training set S =
Revised: Notice that all the expansion errors sample from the same training set S =

Original: A ∪ B .
Revised: A ∪ B .

Original: The lemma then follows from the linearity of the unnormalized empirical risk in the test set, i.e.,
Revised: The lemma then follows from the linearity of the unnormalized empirical risk in the test set, i.e.,

Original: [Equation]
Revised: [Equation]

Original: as claimed.
Revised: as claimed.

Original: Proof of Theorem 3.5.
Revised: Proof of Theorem 2.3.

Original: Assume first that the incompatibility is not strict.
Revised: Assume first that the incompatibility is not strict.

Original: Define U =
Revised: Define U =

Original: S ∗ min ∩ A and
Revised: S ∗ min ∩ A and

Original: V =
Revised: V =

Original: S ∗ min ∩ B .
Revised: S ∗ min ∩ B .

Original: From Lemma A.1, we have that
Revised: From Lemma A.1, we have that

Original: [Equation]
Revised: [Equation]

Original: Applying now the definition of incompatibility gives
Revised: Applying now the definition of incompatibility gives

Original: [Equation]
Revised: [Equation]

Original: [Equation] As | U | + | V | = | S ∗ min | , it follows that at least one of ϵ ( U ; α ) or ϵ ( V ; α ) is at most ϵ ∗ , which contradicts the optimality of S ∗ min (as it was assumed to be a smallest set achieving ϵ ∗ ).
Revised: As | U | + | V | = | S ∗ min | , it follows that at least one of ϵ ( U ; α ) or ϵ ( V ; α ) is at most ϵ ∗ , which contradicts the optimality of S ∗ min (as it was assumed to be a smallest set achieving ϵ ∗ ).

Original: Now we consider the case when at least one of the incompatibilities is strict.
Revised: Now we consider the case when at least one of the incompatibilities is strict.

Original: Let S ∗ be any set in
Revised: Let S ∗ be any set in

Original: S ∗ .
Revised: S ∗ .

Original: Using the definition of strict incompatibility, we get that
Revised: Using the definition of strict incompatibility, we get that

Original: [Equation]
Revised: [Equation]

Original: Hence, at least one of ϵ ( U ; α ) or ϵ
Revised: Hence, at least one of ϵ ( U ; α ) or ϵ

Original: ( V ; α ) is strictly less than ϵ ∗ , which again contradicts the optimality of S ∗ .
Revised: ( V ; α ) is strictly less than ϵ ∗ , which again contradicts the optimality of S ∗ .

Original: Proof of Theorem 4.2.
Revised: Proof of Theorem 3.4.

Original: Let S i and S
Revised: Let S i and S

Original: j be a clean component and poison component, respectively.
Revised: j be a clean component and poison component, respectively.

Original: By the second inequality in Property 4.1, we have that R emp
Revised: By the second inequality in Property 3.3, we have that R emp

Original: ( A ( S i ); S j ) ≥ 1 / 2 .
Revised: ( A ( S i ); S j ) ≥ 1 / 2 .

Original: Thus S i votes
Revised: Thus S i votes

Original: False on S j .
Revised: False on S j .

Original: Conversely, if both S i and S j are a clean components, then R emp ( A ( S i ); S j ) < 1 / 2 , so S i votes True on S j .
Revised: Conversely, if both S i and S j are a clean components, then R emp ( A ( S i ); S j ) < 1 / 2 , so S i votes True on S j .

Original: Putting these together and using the fact that at least half of the data is clean, we find that the poisoned components have weighted vote strictly less than | S | / 2 , while the clean components have weighted vote strictly greater than | S | / 2 .
Revised: Putting these together and using the fact that at least half of the data is clean, we find that the poisoned components have weighted vote strictly less than | S | / 2 , while the clean components have weighted vote strictly greater than | S | / 2 .

Original: As all the clean components vote correctly, using the weighted majority correctly identifies all the clean and poisoned components as claimed.
Revised: As all the clean components vote correctly, using the weighted majority correctly identifies all the clean and poisoned components as claimed.

Original: Proof of Proposition 4.3.
Revised: Proof of Proposition 2.4.

Original: Recall that α = 1 and η = 0 .
Revised: Recall that α = 1 and η = 0 .

Original: We prove the statement in two steps.
Revised: We prove the statement in two steps.

Original: First, we show that
Revised: First, we show that

Original: [Equation]
Revised: [Equation]

Original: The inequality follows from the optimization on Line 5 in Algorithm 2, which sets θ t +1 to the empirical risk minimizer of the set S t .
Revised: The inequality follows from the optimization on Line 5 in Algorithm 1, which sets θ t +1 to the empirical risk minimizer of the set S t .

Original: Next, we claim that
Revised: Next, we claim that

Original: [Equation]
Revised: [Equation]

Original: Since | L ( · , · ) | ≤ 1 , the optimal size of the set S t is | S t | = β t | S | .
Revised: Since | L ( · , · ) | ≤ 1 , the optimal size of the set S t is | S t | = β t | S | .

Original: Since β t is decreasing, we have that | S t +1 | ≤ | S t | .
Revised: Since β t is decreasing, we have that | S t +1 | ≤ | S t | .

Original: Thus the number of elements in the trimmed empirical loss is non-increasing (Line 7, Algorithm 2).
Revised: Thus the number of elements in the trimmed empirical loss is non-increasing (Line 7, Algorithm 1).

Original: Combining the two inequalities shows that the objective function is decreasing in t .
Revised: Combining the two inequalities shows that the objective function is decreasing in t .

Original: Since
Revised: Since

Original: F ( θ t , S t ; β t ) is a decreasing sequence bounded from below by zero, the monotone convergence theorem gives the second result.
Revised: F ( θ t , S t ; β t ) is a decreasing sequence bounded from below by zero, the monotone convergence theorem gives the second result.

Original: B D ATASET C ONSTRUCTION D ETAILS
Revised: B D ATASET C ONSTRUCTION D ETAILS

Original: Our implementation of the 1-to-1 dirty label backdoor (DLBD) adversary follows the threat model described in Gu et al.
Revised: Our implementation of the 1-to-1 dirty label backdoor (DLBD) adversary follows the threat model described in Gu et al.

Original: For evaluation, we use the same dataset (CIFAR-10 (Krizhevsky &
Revised: For evaluation, we use the same dataset (CIFAR-10 (Krizhevsky &

Original: Hinton, 2009)) and setup for our experiments as the Spectral Signatures work (Tran et al., 2018).
Revised: Hinton, 2009)) and setup for our experiments as the Spectral Signatures work (Tran et al., 2018).

Original: Each scenario has a single source and target class, and we use the same (source, target) pairs as in
Revised: Each scenario has a single source and target class, and we use the same (source, target) pairs as in

Original: Tran et al.
Revised: Tran et al.

Original: (2018): (airplane, bird), (automobile, cat), (bird, dog), (cat, dog), (cat, horse), (horse, deer), (ship, frog), (truck, bird).
Revised: (2018): (airplane, bird), (automobile, cat), (bird, dog), (cat, dog), (cat, horse), (horse, deer), (ship, frog), (truck, bird).

Original: The perturbation function τ overlays a small pattern on the image at a fixed location.
Revised: The perturbation function τ overlays a small pattern on the image at a fixed location.

Original: All patterns fit within a 3x3 pixel box.
Revised: All patterns fit within a 3x3 pixel box.

Original: To generate a perturbation, we choose a shape (L-shape, X-shape, or pixel) uniformly at random.
Revised: To generate a perturbation, we choose a shape (L-shape, X-shape, or pixel) uniformly at random.

Original: The (x,y) coordinates of the perturbation are randomly selected to guarantee that the entire shape is visible before data augmentation (e.g., the pixel-based perturbation can be placed anywhere within the 32x32 image, but the X-shape is larger and so must be centered in a 30x30 region, one pixel away from the border).
Revised: The (x,y) coordinates of the perturbation are randomly selected to guarantee that the entire shape is visible before data augmentation (e.g., the pixel-based perturbation can be placed anywhere within the 32x32 image, but the X-shape is larger and so must be centered in a 30x30 region, one pixel away from the border).

Original: The color of the perturbation is also selected uniformly at random, with each of the (R,G,B) coordinates ranging from 0 to 255.
Revised: The color of the perturbation is also selected uniformly at random, with each of the (R,G,B) coordinates ranging from 0 to 255.

Original: Finally, we randomly select an ϵ = 5 , 10 , 20% percentage of the source class, apply the perturbation by replacing the pixels in the corresponding locations with the selected shape and color, then relabel the poisoned images as the target class.
Revised: Finally, we randomly select an ϵ = 5 , 10 , 20% percentage of the source class, apply the perturbation by replacing the pixels in the corresponding locations with the selected shape and color, then relabel the poisoned images as the target class.

Original: For the all-to-1 case, for a given target class, we poison an equal proportion of every non-target class such that the total amount of poison is ϵ times the size of the source class.
Revised: For the all-to-1 case, for a given target class, we poison an equal proportion of every non-target class such that the total amount of poison is ϵ times the size of the source class.

Original: For the all-to-all case, we select a cyclic permutation of the classes, then poison a ϵ percentage of each class.
Revised: For the all-to-all case, we select a cyclic permutation of the classes, then poison a ϵ percentage of each class.

Original: The construction of the W ATERMARK dataset is the same, except that we instead use 8x8 images in the upper left depicting: a peace sign, the letter A, 3 bullet holes, or a colorful patch, taken from the
Revised: The construction of the W ATERMARK dataset is the same, except that we instead use 8x8 images in the upper left depicting: a peace sign, the letter A, 3 bullet holes, or a colorful patch, taken from the

Original: Nicolae et al.
Revised: Nicolae et al.

Original: For the CLBD dataset, we used the official datasets provided by Turner et al.
Revised: For the CLBD dataset, we used the official datasets provided by Turner et al.

Original: There are three datasets in total, one for each perturbation type: ℓ 2 , ℓ ∞ , and GAN.
Revised: There are three datasets in total, one for each perturbation type: ℓ 2 , ℓ ∞ , and GAN.

Original: For the ℓ 2 and ℓ ∞ -based attacks, each image is perturbed independently to maximize the loss with respect to a reference model trained on clean data; the pertubation size is bounded by the respective norm.
Revised: For the ℓ 2 and ℓ ∞ -based attacks, each image is perturbed independently to maximize the loss with respect to a reference model trained on clean data; the pertubation size is bounded by the respective norm.

Original: For the GAN-based attack, for each image to be poisoned, a perceptually similar target image in another class is identified using the latent space of the GAN, and the perturbed image is an interpolation of the original and target images in the latent space.
Revised: For the GAN-based attack, for each image to be poisoned, a perceptually similar target image in another class is identified using the latent space of the GAN, and the perturbed image is an interpolation of the original and target images in the latent space.

Original: In all three cases, a 3x3 checkerboard patch is then placed in each of the 4 corners of the image.
Revised: In all three cases, a 3x3 checkerboard patch is then placed in each of the 4 corners of the image.

Original: The label of the image is not changed.
Revised: The label of the image is not changed.

Original: At test time, only the checkerboard patches are placed on the image.
Revised: At test time, only the checkerboard patches are placed on the image.

Original: We refer the reader to Turner et al.
Revised: We refer the reader to Turner et al.

Original: (2018) for more details.
Revised: (2018) for more details.

Original: For the GTSRB dataset, we use the same construction of the DLBD and W ATERMARK attacks on
Revised: For the GTSRB dataset, we use the same construction of the DLBD and W ATERMARK attacks on

Original: CIFAR-10.
Revised: CIFAR-10.

Original: However, as the CLBD dataset was only created for the CIFAR-10 dataset, we did not evaluate the CLBD attack on GTSRB.
Revised: However, as the CLBD dataset was only created for the CIFAR-10 dataset, we did not evaluate the CLBD attack on GTSRB.

Original: C D EFENSE S ETUPS AND H YPERPARAMETERS
Revised: C D EFENSE S ETUPS AND H YPERPARAMETERS

Original: ISPL + Boosting (this work).
Revised: ISPL + Boosting (this work).

Original: For our defense, we use the set of hyperparameters described here unless otherwise noted.
Revised: For our defense, we use the set of hyperparameters described here unless otherwise noted.

Original: For CIFAR-10, we run 8 rounds of ISPL, each of which returns a component consisting of roughly 12% of the total samples.
Revised: For CIFAR-10, we run 8 rounds of ISPL, each of which returns a component consisting of roughly 12% of the total samples.

Original: For GTSRB, we run 4 rounds of ISPL so that each component contains 24% of the total samples.
Revised: For GTSRB, we run 4 rounds of ISPL so that each component contains 24% of the total samples.

Original: Let p be the target percentage of samples over the remaining samples (e.g., for CIFAR-10 p ≈ 1 / (8 − i + 1) in the i th iteration).
Revised: Let p be the target percentage of samples over the remaining samples (e.g., for CIFAR-10 p ≈ 1 / (8 − i + 1) in the i th iteration).

Original: Then the number of iterations N is set to 2 + min(3 , 1 /p ) .
Revised: Then the number of iterations N is set to 2 + min(3 , 1 /p ) .

Original: β starts at 3 ∗ p in the first iteration, then drops linearly to its final value of p over the next 2 iterations.
Revised: β starts at 3 ∗ p in the first iteration, then drops linearly to its final value of p over the next 2 iterations.

Original: When trimming the training set, we also additionally include the top p/ 2 samples per class to prevent the network from collapsing to a trivial solution.
Revised: When trimming the training set, we also additionally include the top p/ 2 samples per class to prevent the network from collapsing to a trivial solution.

Original: For the learning procedure A , we use standard SGD, trained for 4 epochs per iteration, with a warm-up in the first iteration of 8 epochs.
Revised: For the learning procedure A , we use standard SGD, trained for 4 epochs per iteration, with a warm-up in the first iteration of 8 epochs.

Original: The expansion factor α is set to 1 / 4 , and the momentum factor η is set to 0 .
Revised: The expansion factor α is set to 1 / 4 , and the momentum factor η is set to 0 .

Original: We run ISPL 3 times to generate 24 weak learners.
Revised: We run ISPL 3 times to generate 24 weak learners.

Original: For the boosting framework, each weak learner is trained for 40 epochs on its respective subset, then votes on a per-sample basis.
Revised: For the boosting framework, each weak learner is trained for 40 epochs on its respective subset, then votes on a per-sample basis.

Original: The sample is preserved if the modal vote equals the given label, with ties broken randomly, or the sample is in the lower half of its class by loss.
Revised: The sample is preserved if the modal vote equals the given label, with ties broken randomly, or the sample is in the lower half of its class by loss.

Original: For CIFAR-10, we also include a final self-training step by training a fresh model for 100 epochs on the recovered samples.
Revised: For CIFAR-10, we also include a final self-training step by training a fresh model for 100 epochs on the recovered samples.

Original: The main idea is that a model fit to the full “clean” training data can be used to test the excluded training data, thereby recovering additional consistent data which may have been originally excluded because the weak learners were fit to a small subset of data for fewer epochs.
Revised: The main idea is that a model fit to the full “clean” training data can be used to test the excluded training data, thereby recovering additional consistent data which may have been originally excluded because the weak learners were fit to a small subset of data for fewer epochs.

Original: However, it may take several repetitions of training a model from scratch before this selftraining process no longer identifies new samples to recover.
Revised: However, it may take several repetitions of training a model from scratch before this selftraining process no longer identifies new samples to recover.

Original: Therefore, we use a simple self-paced learning algorithm to dynamically adjust the samples during training to limit the self-training to a single iteration.
Revised: Therefore, we use a simple self-paced learning algorithm to dynamically adjust the samples during training to limit the self-training to a single iteration.

Original: More explicitly, we start with the “clean” samples as returned by the boosting framework.
Revised: More explicitly, we start with the “clean” samples as returned by the boosting framework.

Original: Every 5 epochs, we update the training set to be the samples whose labels agree with the model’s current predictions.
Revised: Every 5 epochs, we update the training set to be the samples whose labels agree with the model’s current predictions.

Original: Due to the relative frequency with which we resample the training set, we smooth the predictions by a momentum factor of 0 . 8 so that the training process is less noisy.
Revised: Due to the relative frequency with which we resample the training set, we smooth the predictions by a momentum factor of 0 . 8 so that the training process is less noisy.

Original: The samples used for training in the last epoch are returned as the defended dataset.
Revised: The samples used for training in the last epoch are returned as the defended dataset.

Original: In our experiments, this process decreases the false positive rate (and thus increases the clean accuracy) but does not materially affect the false negative rate (nor the targeted misclassification rate).
Revised: In our experiments, this process decreases the false positive rate (and thus increases the clean accuracy) but does not materially affect the false negative rate (nor the targeted misclassification rate).

Original: We did not use self-training for GTSRB as we found the clean accuracy was sufficiently high.
Revised: We did not use self-training for GTSRB as we found the clean accuracy was sufficiently high.

Original: Spectral Signatures.
Revised: Spectral Signatures.

Original: We use the official implementation of the Spectral Signatures ( SS ) defense (Tran et al., 2018) by the authors, available on Github, except that we replace the training procedure with PyTorch (instead of Tensorflow 1.x as in the authors’ original implementation).
Revised: We use the official implementation of the Spectral Signatures ( SS ) defense (Tran et al., 2018) by the authors, available on Github, except that we replace the training procedure with PyTorch (instead of Tensorflow 1.x as in the authors’ original implementation).

Original: The authors suggest removing 1.5 times the maximum expected amount of poison from each class for the defense.
Revised: The authors suggest removing 1.5 times the maximum expected amount of poison from each class for the defense.

Original: We remove 20% of each class for ϵ = 5 , 10% (to match the procedure in their paper) and 30% of each class for ϵ = 20% .
Revised: We remove 20% of each class for ϵ = 5 , 10% (to match the procedure in their paper) and 30% of each class for ϵ = 20% .

Original: In selecting the layer for the activations,  we use the input to the third block of the third layerof the ResNet32 architecture (taken from the SS defense authors’ public implementation). In selecting the layer for the activations, we use the input to the first block of the four layerof the PreActResNet18 architecture (which was found empirically to remove the most poison on the first set of scenarios).
Revised: In selecting the layer for the activations, for the ResNet32 architecture we use the input to the third block of the third layer (taken from the SS defense authors’ public implementation), and for the PreActResNet18 architecture we use the input to the first block of the fourth layer (which was found empirically to remove the most poison on the first set of scenarios).

Original: We note that the authors indicate the defense should be fairly successful at any of the later layers of the network.
Revised: We note that the authors indicate the defense should be fairly successful at any of the later layers of the network.

Original: Iterative Trimmed Loss Minimization.
Revised: Iterative Trimmed Loss Minimization.

Original: The Iterative Trimmed Loss Minimization ( ITLM ) defense (Shen & Sanghavi, 2019) consists of an iterative procedure.
Revised: The Iterative Trimmed Loss Minimization ( ITLM ) defense (Shen & Sanghavi, 2019) consists of an iterative procedure.

Original: Given a setting 0 < α ≤ 1 , one first trains a model for a number of epochs.
Revised: Given a setting 0 < α ≤ 1 , one first trains a model for a number of epochs.

Original: Then the α fraction of samples with the lowest loss are retained for the next iteration.
Revised: Then the α fraction of samples with the lowest loss are retained for the next iteration.

Original: This process is repeated several times, with a fresh model beginning each iteration.
Revised: This process is repeated several times, with a fresh model beginning each iteration.

Original: The defended dataset is the α fraction of samples with the lowest loss after the last iteration.
Revised: The defended dataset is the α fraction of samples with the lowest loss after the last iteration.

Original: For the backdoor data poisoning experiments on CIFAR-10, the authors use 80 epochs for the first round of training, then 40 epochs thereafter; they also set α = 98% for ϵ = 5% , and do not test at other values of ϵ .
Revised: For the backdoor data poisoning experiments on CIFAR-10, the authors use 80 epochs for the first round of training, then 40 epochs thereafter; they also set α = 98% for ϵ = 5% , and do not test at other values of ϵ .

Original: We use the same settings, and scale α linearly with ϵ , i.e., α = 96% for ϵ = 10% and α = 92% for ϵ = 20% .
Revised: We use the same settings, and scale α linearly with ϵ , i.e., α = 96% for ϵ = 10% and α = 92% for ϵ = 20% .

Original: Activation Clustering.
Revised: Activation Clustering.

Original: The Activation Clustering ( AC ) defense (Chen et al., 2018) has an actively maintained official implementation in the Adversarial Robustness Toolbox (ART) (Nicolae et al., 2019), an open-source collection of tools for security in machine learning.
Revised: The Activation Clustering ( AC ) defense (Chen et al., 2018) has an actively maintained official implementation in the Adversarial Robustness Toolbox (ART) (Nicolae et al., 2019), an open-source collection of tools for security in machine learning.

Original: We use the official implementation with the default parameters values in ART v1.6.2.
Revised: We use the official implementation with the default parameters values in ART v1.6.2.

Original: In selecting the layer for the activations, we used the same layers as for Spectral Signatures.
Revised: In selecting the layer for the activations, we used the same layers as for Spectral Signatures.

Original: Models.
Revised: Models.

Original: For the ResNet32 (He et al., 2016a) model, we use vanilla SGD with learning rate 0.1, momentum 0.9, and weight decay 1e-4.
Revised: For the ResNet32 (He et al., 2016a) model, we use vanilla SGD with learning rate 0.1, momentum 0.9, and weight decay 1e-4.

Original: For the final dataset, we train for 100 epochs (unless otherwise noted) and drop the learning rate by 10 at epochs 75 and 90.
Revised: For the final dataset, we train for 100 epochs (unless otherwise noted) and drop the learning rate by 10 at epochs 75 and 90.

Original: Using these parameters, we achieve around 94.5% accuracy on GTSRB when trained and tested with clean data.
Revised: Using these parameters, we achieve around 94.5% accuracy on GTSRB when trained and tested with clean data.

Original: For the PreActResNet18 (He et al., 2016b) model, We use vanilla SGD with learning rate 0.02, momentum 0.9, and weight decay 5e-4.
Revised: For the PreActResNet18 (He et al., 2016b) model, We use vanilla SGD with learning rate 0.02, momentum 0.9, and weight decay 5e-4.

Original: For the final dataset, we train for 100 epochs (unless otherwise noted) and drop the learning rate by 10 at epochs 50, 75, and 90.
Revised: For the final dataset, we train for 100 epochs (unless otherwise noted) and drop the learning rate by 10 at epochs 50, 75, and 90.

Original: Using these parameters, we achieve around 93.0% accuracy on CIFAR-10 when trained and tested with clean data.
Revised: Using these parameters, we achieve around 93.0% accuracy on CIFAR-10 when trained and tested with clean data.

Original: For the ResNet56 (He et al., 2016b) model, We use vanilla SGD with learning rate 0.05, momentum 0.9, and weight decay 5e-4.
Revised: For the ResNet56 (He et al., 2016b) model, We use vanilla SGD with learning rate 0.05, momentum 0.9, and weight decay 5e-4.

Original: For the final dataset, we train for 100 epochs (unless otherwise noted) and drop the learning rate by 10 at epochs 75, 90, 95, and 98.
Revised: For the final dataset, we train for 100 epochs (unless otherwise noted) and drop the learning rate by 10 at epochs 75, 90, 95, and 98.

Original: Using these parameters, we achieve around 91.0% accuracy on Imagenette when trained and tested with clean data.
Revised: Using these parameters, we achieve around 91.0% accuracy on Imagenette when trained and tested with clean data.

Original: D A DDITIONAL E XPERIMENTAL R ESULTS
Revised: D A DDITIONAL E XPERIMENTAL R ESULTS

Original: This section contains additional experimental results to complement the main text.
Revised: This section contains additional experimental results to complement the main text.

Original: Section D. expands upon the empirical evaluations of the compatibility property from Section 5.1.
Revised: Section D. expands upon the empirical evaluations of the compatibility property from Section 4.1.

Original: Section D. reports comprehensive results from the evaluations of the ISPL+B defense reported in Section 5.2.
Revised: Section D. reports comprehensive results from the evaluations of the ISPL+B defense reported in Section 4.2.

Original: Section D.2 also includes a detailed breakdown of the performance for the three defenses (Spectral
Revised: Section D.2 also includes a detailed breakdown of the performance for the three defenses (Spectral

Original: Signatures, Activation Clustering, and Iterative Trimmed Loss Minimization)
Revised: Signatures, Activation Clustering, and Iterative Trimmed Loss Minimization)

Original: we use as baselines in our comparisons, as well as preliminary results for a high-resolution dataset.
Revised: we use as baselines in our comparisons, as well as preliminary results for a high-resolution dataset.

Original: Section D.2.3 evaluates our defense against an adaptive attack that is given white-box access to the ISPL subroutine, and selects which elements to poison based on the behavior of ISPL on clean data.
Revised: Section D.2.3 evaluates our defense against an adaptive attack that is given white-box access to the ISPL subroutine, and selects which elements to poison based on the behavior of ISPL on clean data.

Original: Our results show that
Revised: Our results show that

Original: ISPL+B maintains excellent performance even in this setting, defending against 20/24 scenarios in the adaptive setting.
Revised: ISPL+B maintains excellent performance even in this setting, defending against 20/24 scenarios in the adaptive setting.

Original: Section D.2.4 reports results for the CIFAR-10 DLBD attack after training for 200 epochs on both the PreActResNet18 and ResNet32 architectures, to test the robustness of our results to the training setup.
Revised: Section D.2.4 reports results for the CIFAR-10 DLBD attack after training for 200 epochs on both the PreActResNet18 and ResNet32 architectures, to test the robustness of our results to the training setup.

Original: Finally, to evaluate the sensitivity of ISPL to the main hyperparameters α and β , Section D.2.5 compares the performance of ISPL on a selection of attack scenarios across a range of α and β .
Revised: Finally, to evaluate the sensitivity of ISPL to the main hyperparameters α and β , Section D.2.5 compares the performance of ISPL on a selection of attack scenarios across a range of α and β .

Original: Our main finding is that performance remains high for a range of choices of α and β .
Revised: Our main finding is that performance remains high for a range of choices of α and β .

Original: All results reported use the median over 3 random seeds, unless otherwise noted.
Revised: All results reported use the median over 3 random seeds, unless otherwise noted.

Original: D. I NCOMPATIBILITY E VALUATION
Revised: D. I NCOMPATIBILITY E VALUATION

Original: Figures 5-9 displays aggregated results of the empirical study of incompatibility for the CLBD,
Revised: Figures 5-9 displays aggregated results of the empirical study of incompatibility for the CLBD,

Original: DLBD, and W ATERMARK attacks on CIFAR-10.
Revised: DLBD, and W ATERMARK attacks on CIFAR-10.

Original: Plots on the left side measure the compatibility of poisoned with clean data (i.e., ϵ ( C ; α ) − ϵ
Revised: Plots on the left side measure the compatibility of poisoned with clean data (i.e., ϵ ( C ; α ) − ϵ

Original: ( C | P ; α ) ) and plots on the right side measure the compatibility of clean with poisoned data (i.e., ϵ ( P ; α )
Revised: ( C | P ; α ) ) and plots on the right side measure the compatibility of clean with poisoned data (i.e., ϵ ( P ; α )

Original: − ϵ
Revised: − ϵ

Original: ( P | C ; α ) ).
Revised: ( P | C ; α ) ).

Original: In all cases, the trend in the first plot is decreasing, which suggests that the gap increases in magnitude as the size of the incompatible data grows.
Revised: In all cases, the trend in the first plot is decreasing, which suggests that the gap increases in magnitude as the size of the incompatible data grows.

Original: Additionally, the clean label scenarios all exhibit the behavior that the source class accuracy increases as the amount of poison (and poisoned accuracy) increases, while the non-source accuracy drops.
Revised: Additionally, the clean label scenarios all exhibit the behavior that the source class accuracy increases as the amount of poison (and poisoned accuracy) increases, while the non-source accuracy drops.

Original: These attacks share the characteristic that the poisoned data are “harder” versions of the source class in the sense that they are perturbed toward instances of other classes.
Revised: These attacks share the characteristic that the poisoned data are “harder” versions of the source class in the sense that they are perturbed toward instances of other classes.

Original: In the case of the GAN-based attack, the perturbations occur in the latent space of a GAN trained on clean data (e.g., the attacker identifies a bird and an airplane that are close in the latent space, linearly interpolates the latent representation of the airplane toward the latent representation of the bird, then uses the generative network to convert the perturbed representation into an image of a “bird-like airplane”).
Revised: In the case of the GAN-based attack, the perturbations occur in the latent space of a GAN trained on clean data (e.g., the attacker identifies a bird and an airplane that are close in the latent space, linearly interpolates the latent representation of the airplane toward the latent representation of the bird, then uses the generative network to convert the perturbed representation into an image of a “bird-like airplane”).

Original: In the case of the ℓ 2 and ℓ ∞ bounds, the attacker uses the Projected Gradient Descent (PGD) algorithm commonly used for adversarial training (Kurakin et al., 2016; Madry et al., 2017) to identify an image within a norm-bounded neighborhood of the original image that maximizes the loss of a trained network, i.e., the perturbed image is close in pixel-space to the original image, but was misclassified by the trained network as a different class.
Revised: In the case of the ℓ 2 and ℓ ∞ bounds, the attacker uses the Projected Gradient Descent (PGD) algorithm commonly used for adversarial training (Kurakin et al., 2016; Madry et al., 2017) to identify an image within a norm-bounded neighborhood of the original image that maximizes the loss of a trained network, i.e., the perturbed image is close in pixel-space to the original image, but was misclassified by the trained network as a different class.

Original: Hence, the simultaneous increase in source class accuracy and decrease in non-source class accuracies can be attributed to a similar mechanism for all the clean label cases.
Revised: Hence, the simultaneous increase in source class accuracy and decrease in non-source class accuracies can be attributed to a similar mechanism for all the clean label cases.

Original: Conversely, when measuring the compatibility of poisoned data with clean data (plots on the left) for the DLBD and W ATERMARK attacks, we see that, as the amount of poisoned data increases, the source class accuracy drops.
Revised: Conversely, when measuring the compatibility of poisoned data with clean data (plots on the left) for the DLBD and W ATERMARK attacks, we see that, as the amount of poisoned data increases, the source class accuracy drops.

Original: The implication is that the poisoned data causes the classifier to misclassify clean data by presenting similar (both quantitatively in terms of the distance between the poisoned and clean distributions, as well as perceptually in most settings) instances of clean source data that are mislabelled as the target class, thus complicating the learning of the clean source class.
Revised: The implication is that the poisoned data causes the classifier to misclassify clean data by presenting similar (both quantitatively in terms of the distance between the poisoned and clean distributions, as well as perceptually in most settings) instances of clean source data that are mislabelled as the target class, thus complicating the learning of the clean source class.

Original: However, when measuring the compatibility of clean data with respect to poisoned data (plots on the right) for the DLBD and W ATERMARK attacks, the source class accuracy actually increases (as in the clean label case).
Revised: However, when measuring the compatibility of clean data with respect to poisoned data (plots on the right) for the DLBD and W ATERMARK attacks, the source class accuracy actually increases (as in the clean label case).

Original: One explanation is that training with examples of clean instances from the source class improves the ability of the classifier to identify possibly poisoned images, hence increasing the accuracy on actual poisoned instances as well.
Revised: One explanation is that training with examples of clean instances from the source class improves the ability of the classifier to identify possibly poisoned images, hence increasing the accuracy on actual poisoned instances as well.

Original: Finally, Figures 10-14 plot the scenarios individually.
Revised: Finally, Figures 10-14 plot the scenarios individually.

Original: We note that incompatibility holds generally across the range of scenarios, with only small deviations to the contrary.
Revised: We note that incompatibility holds generally across the range of scenarios, with only small deviations to the contrary.

Original: D. D EFENSE E VALUATION
Revised: D. D EFENSE E VALUATION

Original: This section includes additional experimental results concerning the performance of our defense.
Revised: This section includes additional experimental results concerning the performance of our defense.

Original: Tables 2-8 display the unabridged results of our main evaluation results for our defense (condensed in the main text as Table 1).
Revised: Tables 2-8 display the unabridged results of our main evaluation results for our defense (condensed in the main text as Table 1).

Original: We note that even though we have selected 1% as the threshold for a “successful” defense, in many of the failure cases, ISPL+B still achieves relatively low TMR.
Revised: We note that even though we have selected 1% as the threshold for a “successful” defense, in many of the failure cases, ISPL+B still achieves relatively low TMR.

Original: For instance, setting the threshold to 5% increases the number of successful scenarios to 142 (compared to 134 at 1%, out of 165 total).
Revised: For instance, setting the threshold to 5% increases the number of successful scenarios to 142 (compared to 134 at 1%, out of 165 total).

Original: Table 2 also includes results of the three baseline defenses for the standard 1-to-1 DLBD attack on
Revised: Table 2 also includes results of the three baseline defenses for the standard 1-to-1 DLBD attack on

Original: CIFAR-10.
Revised: CIFAR-10.

Original: In particular, we observe that our clean accuracy is comparable to (or even slightly better than) that of Activation Clustering ( AC ), the next best defense, despite AC achieving significantly lower TMR in our evaluations
Revised: In particular, we observe that our clean accuracy is comparable to (or even slightly better than) that of Activation Clustering ( AC ), the next best defense, despite AC achieving significantly lower TMR in our evaluations

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The

Original: [Table]
Revised: [Table]

Revised: using the PreActResNet18 architecture.

Original:  column lists offset of the cyclic permutation (i.e., 2 means  source class 0 is poisoned as target class 2, source class 1 is poisoned as target class 3, etc.).
Revised: The S → T column lists offset of the cyclic permutation (i.e., 2 means that source class 0 is poisoned as target class 2, source class 1 is poisoned as target class 3, etc.).

Original: ϵ refers to the percentage of the source class which is poisoned.
Revised: ϵ refers to the percentage of the source class which is poisoned.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ℓ  93. ℓ ∞  93.
Revised: ℓ  93. ℓ ∞  93.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  S 
Revised: D.2. S LEEPER A GENT ATTACK

Original: Sleeper Agent (Souri et al., 2022) is a clean label attack that uses gradient matching (Geiping et al., 2020) to create a poisoned training set.
Revised: Sleeper Agent (Souri et al., 2022) is a clean label attack that uses gradient matching (Geiping et al., 2020) to create a poisoned training set.

Original: The objective of the attacker is to induce a model trained on the poisoned training set to misclassify patched versions of a source class as the target class.
Revised: The objective of the attacker is to induce a model trained on the poisoned training set to misclassify patched versions of a source class as the target class.

Original: Figure 16 displays examples of clean and poisoned training and test data created by the Sleeper
Revised: Figure 16 displays examples of clean and poisoned training and test data created by the Sleeper

Original: Agent attack, where the source class is horse and the target class is dog (training with the poisoned dogs causes patched horses to be misclassified as dogs).
Revised: Agent attack, where the source class is horse and the target class is dog (training with the poisoned dogs causes patched horses to be misclassified as dogs).

Original: Similar to the CLBD attack, Sleeper Agent perturbs each image in the training set independently subject to an ℓ ∞ bound, however, theSleeper
Revised: Similar to the CLBD attack, Sleeper Agent perturbs each image in the training set independently subject to an ℓ ∞ bound, however, the

Original:  Agent attack is more subtle because it does not insert any patches during training.
Revised: Sleeper Agent attack is more subtle because it does not insert any patches during training.

Original: The Sleeper Agent attacker also differs from the DLBD, CLBD, and W ATERMARK attacks in using an informed mechanism to select which subset of the training set D to poison (rather than poisoning a random subset).
Revised: The Sleeper Agent attacker also differs from the DLBD, CLBD, and W ATERMARK attacks in using an informed mechanism to select which subset of the training set D to poison (rather than poisoning a random subset).

Original: Finally, unlike prior gradient matching approaches, the Sleeper Agent attack is also intended to apply in the black-box setting, where the attacker does not have access to the victim model’s architecture or training hyperparameters.
Revised: Finally, unlike prior gradient matching approaches, the Sleeper Agent attack is also intended to apply in the black-box setting, where the attacker does not have access to the victim model’s architecture or training hyperparameters.

Original: We first used the official repository released by the authors to generate 24 poisoned datasets with the default settings for CIFAR-10 using different random seeds, targeting a ResNet32 as the victim architecture.
Revised: We first used the official repository released by the authors to generate 24 poisoned datasets with the default settings for CIFAR-10 using different random seeds, targeting a ResNet32 as the victim architecture.

Original: 1 We observed an attack success rate (ASR) of 49.1% ( ± 38.3%) over the generated datasets, when training a ResNet32 model with the attacker’s original training hyperparameters.
Revised: 1 We observed an attack success rate (ASR) of 49.1% ( ± 38.3%) over the generated datasets, when training a ResNet32 model with the attacker’s original training hyperparameters.

Original: We then transferred the poisoned dataset to the setting of our defense (training a ResNet32, using the same training and defense hyperparameters as in our main experiments for ResNet32 on CIFAR10).
Revised: We then transferred the poisoned dataset to the setting of our defense (training a ResNet32, using the same training and defense hyperparameters as in our main experiments for ResNet32 on CIFAR10).

Original: Our defense delivers an ASR of 4.3% ( ± 4.1%), compared to an oracle, which achieves an
Revised: Our defense delivers an ASR of 4.3% ( ± 4.1%), compared to an oracle, which achieves an

Original: ASR of 3.0% ( ± 4.8%).
Revised: ASR of 3.0% ( ± 4.8%).

Original: However, we also observed that the attack only achieves a success rate of 4.5% ( ± 4.3%) on an undefended model in our setting.
Revised: However, we also observed that the attack only achieves a success rate of 4.5% ( ± 4.3%) on an undefended model in our setting.

Original: We attribute this anomaly (between the ASR for an undefended model in our setting vs. the ASR reported by the attacker) to the difference in training hyperparameters, despite using the same victim architecture.
Revised: We attribute this anomaly (between the ASR for an undefended model in our setting vs. the ASR reported by the attacker) to the difference in training hyperparameters, despite using the same victim architecture.

Original: In all three cases—defended, undefended, and oracle—the clean accuracy is consistent between 89-90%.
Revised: In all three cases—defended, undefended, and oracle—the clean accuracy is consistent between 89-90%.

Original: We next evaluated the effectiveness of our defense when retraining with the attacker’s training setup.
Revised: We next evaluated the effectiveness of our defense when retraining with the attacker’s training setup.

Original: Specifically, we took the poisoned training set generated by the attacker, then ran our defense using our training setup to output a new training set with the detected poisoned examples removed.
Revised: Specifically, we took the poisoned training set generated by the attacker, then ran our defense using our training setup to output a new training set with the detected poisoned examples removed.

Original: We then used the attacker’s training setup to train a new model on our new training set, and evaluated this new model on the poisoned test set.
Revised: We then used the attacker’s training setup to train a new model on our new training set, and evaluated this new model on the poisoned test set.

Original: Note that in our threat model, the training algorithm A is fixed, and our defense is designed around the interaction between the data and A .
Revised: Note that in our threat model, the training algorithm A is fixed, and our defense is designed around the interaction between the data and A .

Original: The above scenario is thus explicitly not the intended use case for our algorithm.
Revised: The above scenario is thus explicitly not the intended use case for our algorithm.

Original: Nevertheless, our defense still achieves an ASR of 14.1% ( ± 15.6%), with a clean accuracy of 87.5% ( ± 0.6%), compared to the original ASR of 49.1%.
Revised: Nevertheless, our defense still achieves an ASR of 14.1% ( ± 15.6%), with a clean accuracy of 87.5% ( ± 0.6%), compared to the original ASR of 49.1%.

Original: An oracle achieves an
Revised: An oracle achieves an

Original: ASR of 2.4% ( ± 2.2).
Revised: ASR of 2.4% ( ± 2.2).

Original: These results thus indicate that our defense successfully defends against the
Revised: These results thus indicate that our defense successfully defends against the

Original: Sleeper Agent attack in our threat model (where the defense is integrated in the intended training pipeline), while also remaining robust to an evaluation under the attacker’s original training setup.
Revised: Sleeper Agent attack in our threat model (where the defense is integrated in the intended training pipeline), while also remaining robust to an evaluation under the attacker’s original training setup.

Original:  I MAGENETTE DATASET
Revised: D.2. I MAGENETTE DATASET

Original: Imagenette (Howard, 2019) is a subset the ImageNet dataset
Revised: Imagenette (Howard, 2019) is a subset the ImageNet dataset

Original: (Deng et al., 2009) consisting of 9, training and 3,925 test images over 10 classes.
Revised: (Deng et al., 2009) consisting of 9, training and 3,925 test images over 10 classes.

Original: We downsampled the 320x320 resolution version of
Revised: We downsampled the 320x320 resolution version of

Original: Imagenette to 224x224, and use a ResNet56 achieving around 91% accuracy on the clean dataset.
Revised: Imagenette to 224x224, and use a ResNet56 achieving around 91% accuracy on the clean dataset.

Original: We used a DLBD adversary that places a solid 5x5 patch of a random color in a random position on the image.
Revised: We used a DLBD adversary that places a solid 5x5 patch of a random color in a random position on the image.

Original: We use this setting to probe the applicability of our technique to higher-resolution datasets.
Revised: We use this setting to probe the applicability of our technique to higher-resolution datasets.

Original: Figure 16 displays examples of clean and poisoned data from the Imagenette experiments.
Revised: Figure 16 displays examples of clean and poisoned data from the Imagenette experiments.

Original: Table 9 reports the results of our defense on Imagenette, using the same defense hyperparameters as the main experiments for CIFAR-10.
Revised: Table 9 reports the results of our defense on Imagenette, using the same defense hyperparameters as the main experiments for CIFAR-10.

Original: The performance of our defense remains consistent, with 21 of the 24 scenarios defended below 1% TMR and less than 2% drop in clean accuracy.
Revised: The performance of our defense remains consistent, withof the 24 scenarios defended below 1% TMR and less than 2% drop in clean accuracy.

Original: [Table]
Revised: [Table]

Original: D.2. A DAPTIVE A TTACKER
Revised: D.2. A DAPTIVE A TTACKER

Original: The attacks used in the main experiments are oblivious to the presence of a defender when selecting the data to poison.
Revised: The attacks used in the main experiments are oblivious to the presence of a defender when selecting the data to poison.

Original: In this section, we devise an strong adaptive attacker with white-box access to the ISPL algorithm when constructing the poisoned dataset.
Revised: In this section, we devise an strong adaptive attacker with white-box access to the ISPL algorithm when constructing the poisoned dataset.

Original: The success of the defense rests crucial on the partition of training data returned by the ISPL algorithm.
Revised: The success of the defense rests crucial on the partition of training data returned by the ISPL algorithm.

Original: In order for the boosting phase to succeed, the partition should consist of homogeneous sets, i [Table] e., the poison must be concentrated in a small number of sets within the partition; hence, the adaptive attacker’s objective is to try and distribute the poisoned samples across multiple sets.
Revised: In order for the boosting phase to succeed, the partition should consist of homogeneous sets, i.e., the poison must be concentrated in a small number of sets within the partition; hence, the adaptive attacker’s objective is to try and distribute the poisoned samples across multiple sets.

Original: In particular, rather than randomly selecting a ϵ fraction of the source class to poison, an adaptive attacker with white-box access to the defense can first run the ISPL algorithm (using the same parameters) on the clean dataset to create a partition of the dataset, then random select a ϵ fraction from each partition to poison.
Revised: In particular, rather than randomly selecting a ϵ fraction of the source class to poison, an adaptive attacker with white-box access to the defense can first run the ISPL algorithm (using the same parameters) on the clean dataset to create a partition of the dataset, then random select a ϵ fraction from each partition to poison.

Original: The idea is that if the ISPL algorithm returns a similar partition, then the poison will be still be distributed across multiple sets, thereby breaking the defense.
Revised: The idea is that if the ISPL algorithm returns a similar partition, then the poison will be still be distributed across multiple sets, thereby breaking the defense.

Original: Table 10 presents the results of our defense against the strong adaptive attacker described above.
Revised: Table 10 presents the results of our defense against the strong adaptive attacker described above.

Original: Against the adaptive attack, our defense still succeeds in 20/24 of the scenarios (versus 22/24 for the standard oblivious attack).
Revised: Against the adaptive attack, our defense still succeeds in 20/24 of the scenarios (versus 22/24 for the standard oblivious attack).

Original: These results suggest the ISPL algorithm remains robust even to white-box adaptive attacks.[Table] [Table] [Table]
Revised: These results suggest the ISPL algorithm remains robust even to white-box adaptive attacks.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: D.2. V ARYING THE T RAINING S ETUP
Revised: D.2. V ARYING THE T RAINING S ETUP

Original: As prior works (Schwarzschild et al., 2021) have observed that results in data poisoning, particularly with DNNs, can vary significantly with the training setup, we additionally ran our defense (along with the three baseline defenses) on the standard CIFAR-10 DLBD attack using both the
Revised: As prior works (Schwarzschild et al., 2021) have observed that results in data poisoning, particularly with DNNs, can vary significantly with the training setup, we additionally ran our defense (along with the three baseline defenses) on the standard CIFAR-10 DLBD attack using both the

Original: PreActResNet18 and ResNet32 models, respectively, and report results after training for 200 epochs (instead of 100, as in our main experiments)
Revised: PreActResNet18 and ResNet32 models, respectively, and report results after training for 200 epochs (instead of 100, as in our main experiments).

Original: [Table]

Original: Tables 11 and 12 report the results of these experiments.
Revised: Tables 11 and 12 report the results of these experiments.

Original: We observe that the defense success rates are largely consistent with those in the main experiments across all defenses; the biggest change is that the Spectral Signatures defense successfully defends against 9 scenarios instead of just(out of 24 total).
Revised: We observe that the defense success rates are largely consistent with those in the main experiments across all defenses; the biggest change is that the Spectral Signatures defense successfully defends against 9 scenarios instead of just(out of 24 total).

Original: ISPL+B still achieves the best performance in terms of reducing the targeted misclassification rate below 1% (with a 2-3% drop in clean accuracy)
Revised: ISPL+B still achieves the best performance in terms of reducing the targeted misclassification rate below 1% (with a 2-3% drop in clean accuracy)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D.2. S ENSITIVITY OF ISPL TO H YPERPARAMETERS
Revised: D.2. S ENSITIVITY OF ISPL TO H YPERPARAMETERS

Original: We next conduct some ablation studies to better understand the effects of the two main hyperparameters in Algorithm 2: the expansion factor α and the subset size β .
Revised: We next conduct some ablation studies to better understand the effects of the two main hyperparameters in Algorithm 1: the expansion factor α and the subset size β .

Original: Computationally, larger β means fewer components and fewer outer iterations of ISPL (and is thus more efficient); in our main experiments, we use β = 1 / 8 and run ISPL 8 times in sequence to generate 8 components.
Revised: Computationally, larger β means fewer components and fewer outer iterations of ISPL (and is thus more efficient); in our main experiments, we use β = 1 / 8 and run ISPL 8 times in sequence to generate 8 components.

Original: Additionally, when estimating the self-expansion error, a larger α leads to lower variance estimates of the expansion error, but less effective measures of generalization; hence, we should prefer taking α as large as possible to produce better estimates (until memorization effects take over).
Revised: Additionally, when estimating the self-expansion error, a larger α leads to lower variance estimates of the expansion error, but less effective measures of generalization; hence, we should prefer taking α as large as possible to produce better estimates (until memorization effects take over).

Original: Our experiments use both α = 1 / 2 (for GTSRB, due to the imbalanced class sizes) and α = 1 / 4 (for CIFAR-10).
Revised: Our experiments use both α = 1 / 2 (for GTSRB, due to the imbalanced class sizes) and α = 1 / 4 (for CIFAR-10).

Original: Tables 13 and 14 present our results from the ablation studies, conducted on a subset of the CIFAR10 DLBD and CLBD subsets.
Revised: Tables 13 and 14 present our results from the ablation studies, conducted on a subset of the CIFAR10 DLBD and CLBD subsets.

Original: We do not include the self-training step after ISPL as in the main experiments, in order to better highlight the trends.
Revised: We do not include the self-training step after ISPL as in the main experiments, in order to better highlight the trends.

Original: In particular, without self-training, we expect both lower clean accuracies and higher targeted misclassification rates due to not recovering all the compatible clean data.
Revised: In particular, without self-training, we expect both lower clean accuracies and higher targeted misclassification rates due to not recovering all the compatible clean data.

Original: Our main finding is that our method is quite robust to both the expansion factor α and subset size β .
Revised: Our main finding is that our method is quite robust to both the expansion factor α and subset size β .

Original: In particular, β ∈ [1 / 8 , 1 / 16] and α ∈ [1 , 1 / 4] all yield strong defenses with less than 10% targeted misclassification.
Revised: In particular, β ∈ [1 / 8 , 1 / 16] and α ∈ [1 , 1 / 4] all yield strong defenses with less than 10% targeted misclassification.

Original: As expected, increasing β leads to a stronger defense but lower clean accuracies.
Revised: As expected, increasing β leads to a stronger defense but lower clean accuracies.

Original: In particular, β = 1 (which identifies only a single subset consisting of 96% of the original dataset) fails in all settings of α ; this is the setting for which ISPL+B is conceptually the most similar to the
Revised: In particular, β = 1 (which identifies only a single subset consisting of 96% of the original dataset) fails in all settings of α ; this is the setting for which ISPL+B is conceptually the most similar to the

Original: ITLM defense, which offers a possibly explanation for ITLM ’s poor performance in our experiments.
Revised: ITLM defense, which offers a possibly explanation for ITLM ’s poor performance in our experiments.

Original: Additionally, α = 1 avoids memorization, mainly because we do not train until convergence within the ISPL loop.
Revised: Additionally, α = 1 avoids memorization, mainly because we do not train until convergence within the ISPL loop.

Original: Finally, we note that α = 1 / 2 appears to achieve a modest optimum, which is particularly noticeable when paired with smaller β .
Revised: Finally, we note that α = 1 / 2 appears to achieve a modest optimum, which is particularly noticeable when paired with smaller β .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E A DDITIONAL R ELATED W ORK
Revised: E A DDITIONAL R ELATED W ORK

Revised: Deep Clustering.

Revised: TODO

Original: Compatibility.
Revised: Compatibility.

Original: Our separation results  can be viewed assolutions to a clustering problem, where we exploit weak supervision in the form of (possibly poisoned) class labels.
Revised: Our separation results in the context of data poisoning can be viewed as clustering by exploiting weak supervision in the form of (possibly poisoned) class labels.

Original: In particular, Balcan et al.
Revised: Balcan et al.

Original: (2005) introduce a method called “co-training”, and show that under a similar compatibility property, learners fit independently to two different “views” of the data can supervise each other to improve the joint performance.
Revised: (2005) introduce a method called “co-training”, and show that under a similar compatibility property, learners fit independently to two different “views” of the data can supervise each other to improve the joint performance.

Original: However, they (and similar works) assume access to a small set of trusted labels not present in our setting and do not consider the case of malicious training data.
Revised: However, they (and similar works) assume access to a small set of trusted labels not present in our setting and do not consider the case of malicious training data.

Original: Self-Paced Learning.
Revised: Self-Paced Learning.

Original: Self-Paced learning (SPL) (Kumar et al., 2010) is a type of curriculum learning (CL) (Bengio et al., 2009) that dynamically creates a curriculum by including samples of increasing difficulty based on the losses of the partially trained model.
Revised: Self-Paced learning (SPL) (Kumar et al., 2010) is a type of curriculum learning (CL) (Bengio et al., 2009) that dynamically creates a curriculum by including samples of increasing difficulty based on the losses of the partially trained model.

Original: Prior works generally apply SPL to improve convergence on noisy datasets (Meng et al., 2016; Jiang et al., 2018; Zhang et al., 2020).
Revised: Prior works generally apply SPL to improve convergence on noisy datasets (Meng et al., 2016; Jiang et al., 2018; Zhang et al., 2020).

Original: In contrast, ISPL discards progressively easier samples to identify a subset with good expansion; to the best of our knowledge we are also the first to apply CL ideas to backdoor attacks.
Revised: In contrast, ISPL discards progressively easier samples to identify a subset with good expansion; to the best of our knowledge we are also the first to apply CL ideas to backdoor attacks.