Original: I S A DVERSARIAL T RAINING R EALLY A S ILVER B ULLET FOR M ITIGATING D ATA P OISONING ?
Revised: I S A DVERSARIAL T RAINING R EALLY A S ILVER B ULLET FOR M ITIGATING D ATA P OISONING ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Indiscriminate data poisoning can decrease the clean test accuracy of a deep learning model by slightly perturbing its training samples.
Revised: Indiscriminate data poisoning can decrease the clean test accuracy of a deep learning model by slightly perturbing its training samples.

Original: There is a consensus that such poisons can hardly harm adversarially-trained (AT) models when the adversarial training budget is no less than the poison budget, i.e., ϵ adv ≥ ϵ poi .
Revised: There is a consensus that such poisons can hardly harm adversarially-trained (AT) models when the adversarial training budget is no less than the poison budget, i.e., ϵ adv ≥ ϵ poi .

Original: This consensus, however, is challenged in this paper based on our new attack strategy that induces indiscriminative features (INF).
Revised: This consensus, however, is challenged in this paper based on our new attack strategy that induces indiscriminative features (INF).

Original: The existence of indiscriminative features makes the poisoned data become less useful for training a model, no matter if AT is applied or not.
Revised: The existence of indiscriminative features makes the poisoned data become less useful for training a model, no matter if AT is applied or not.

Original: In contrast, existing methods are limited to using perturbations as shortcuts , which just override the actual image content during model training.
Revised: In contrast, existing methods are limited to using perturbations as shortcuts , which just override the actual image content during model training.

Original: We demonstrate that for attacking a CIFAR-10 AT model under a reasonable setting with ϵ adv =
Revised: We demonstrate that for attacking a CIFAR-10 AT model under a reasonable setting with ϵ adv =

Original: ϵ poi
Revised: ϵ poi

Original: = 8 / 255 , our INF yields an accuracy drop of 13 .
Revised: = 8 / 255 , our INF yields an accuracy drop of 13 .

Original: 31% , which is 7 × better than existing methods and equal to discarding 83% training data.
Revised: 31% , which is 7 × better than existing methods and equal to discarding 83% training data.

Original: We further show the generalizability of INF to more challenging settings, e.g., higher AT budgets, partial poisoning, unseen model architectures, and stronger (ensemble or adaptive) defenses.
Revised: We further show the generalizability of INF to more challenging settings, e.g., higher AT budgets, partial poisoning, unseen model architectures, and stronger (ensemble or adaptive) defenses.

Original: We finally provide new insights into the distinct roles of non-robust vs. robust features in poisoning standard vs. AT models and confirm the effectiveness of INF in poisoning standard models.
Revised: We finally provide new insights into the distinct roles of non-robust vs. robust features in poisoning standard vs. AT models and demonstrate the possibility of using a hybrid attack to poison standard and AT models simultaneously.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Indiscriminate data poisoning aims to degrade the overall prediction performance of a machine learning model at test time by manipulating its training data.
Revised: Indiscriminate data poisoning aims to degrade the overall prediction performance of a machine learning model at test time by manipulating its training data.

Original: It has been increasingly important to understand indiscriminate data poisoning as web scraping becomes a common approach to obtaining large-scale data for training advanced models (Brown et al., 2020; Dosovitskiy et al., 2021).
Revised: It has been increasingly important to understand indiscriminate data poisoning as web scraping becomes a common approach to obtaining large-scale data for training advanced models (Brown et al., 2020; Dosovitskiy et al., 2021).

Original: Although slightly perturbing training samples has been shown to effectively poison deep learning models, there is a consensus that such poisons can hardly harm an adversarially-trained model when the perturbation budget in adversarial training, ϵ adv , is no less than the poison budget, ϵ poi , i.e., ϵ adv ≥ ϵ poi (Fowl et al., 2021a;b; Huang et al., 2021;
Revised: Although slightly perturbing training samples has been shown to effectively poison deep learning models, there is a consensus that such poisons can hardly harm an adversarially-trained model when the perturbation budget in adversarial training, ϵ adv , is no less than the poison budget, ϵ poi , i.e., ϵ adv ≥ ϵ poi (Fowl et al., 2021a;b; Huang et al., 2021;

Original: Tao et al., 2021; Wang et al., 2021; Fu et al., 2022; Tao et al., 2022).
Revised: Tao et al., 2021; Wang et al., 2021; Fu et al., 2022; Tao et al., 2022).

Original: In particular, Tao et al.
Revised: In particular, Tao et al.

Original: (2021) have proved that in this setting, adversarial training can serve as a principled defense against existing poisoning methods.
Revised: (2021) have proved that in this setting, adversarial training can serve as a principled defense against existing poisoning methods.

Original: However, in this paper, we challenge this consensus by rethinking data poisoning from a fundamentally new perspective.
Revised: However, in this paper, we challenge this consensus by rethinking data poisoning from a fundamentally new perspective.

Original: Specifically, we introduce a new poisoning approach that can substantially degrade adversarially-trained models by making the features of training samples from different classes become indiscriminative.
Revised: Specifically, we introduce a new poisoning approach that can substantially degrade adversarially-trained models by making the features of training samples from different classes become indiscriminative.

Original: In this way, the indiscriminative samples would hardly contribute to training the target (discriminative) model anymore no matter whether adversarial training is applied or not.
Revised: In this way, the indiscriminative samples would hardly contribute to training the target (discriminative) model anymore no matter whether adversarial training is applied or not.

Original: Different from our attack strategy, existing methods commonly inject perturbations as shortcuts, which ensures that the model wrongly learns the shortcuts rather than the actual image content, leading to low test accuracy (Segura et al., 2022a; Evtimov et al., 2021; Yu et al., 2022).
Revised: Different from our attack strategy, existing methods commonly inject perturbations as shortcuts, which ensures that the model wrongly learns the shortcuts rather than the actual image content, leading to low test accuracy (Segura et al., 2022a; Evtimov et al., 2021; Yu et al., 2022).

Original: Figure 1 illustrates the working mechanism of our new poisoning approach, with a comparison to a reverse operation that instead aims to eliminate indiscriminative features.
Revised: Figure 1 illustrates the working mechanism of our new poisoning approach, with a comparison to a reverse operation that instead aims to eliminate indiscriminative features.

Original: Our new approach is also inspired by the conventional, noisy label-based poisoning approach (Biggio et al., 2012; 2011;
Revised: Our new approach is also inspired by the conventional, noisy label-based poisoning approach (Biggio et al., 2012; 2011;

Original: Muñoz-González et al., 2017), where indiscriminative labels are introduced by directly flipping labels (e.g., assigning a “dog” (or “cat”) label to both the “dog” and “cat” images) under a strong assumption that the labeling process of the target model can be manipulated.
Revised: Muñoz-González et al., 2017), where indiscriminative labels are introduced by directly flipping labels (e.g., assigning a “dog” (or “cat”) label to both the “dog” and “cat” images) under a strong assumption that the labeling process of the target model can be manipulated.

Original: However, due to
Revised: However, due to

Original: [Figure]
Revised: [Figure]

Original: the imperceptibility constraint in the common clean-label setting, we instead propose to introduce indiscriminative features represented in the latent space.
Revised: the imperceptibility constraint in the common clean-label setting, we instead propose to introduce indiscriminative features represented in the latent space.

Original: Our work mainly makes three contributions:
Revised: Our work mainly makes three contributions:

Original: • We demonstrate that, contrary to the consensus view, indiscriminate data poisoning can actually decrease the clean test accuracy of adversarially-trained  models to a substantial extent.
Revised: • We demonstrate that, contrary to the consensus view, indiscriminate data poisoning can actually decrease the clean test accuracy of adversarially-trained (AT) models to a substantial extent.

Original: Specifically, we propose INF, a new poisoning approach that is based on inducing indiscriminative features in the latent space of a pre-trained reference model.
Revised: Specifically, we propose INF, a new poisoning approach that is based on inducing indiscriminative features in the latent space of a pre-trained reference model

Original: • We conduct extensive experiments to demonstrate the effectiveness of INF against adversarial training the reasonable setting with ϵ adv =
Revised: • We conduct extensive experiments to demonstrate the effectiveness of INF against AT in the reasonable setting with ϵ adv =

Original: ϵ poi
Revised: ϵ poi

Original: and also its generalizability to a variety of more challenging settings, such as adversarial training with higher budgets, partial poisoning, unseen model architectures, and stronger (ensemble or adaptive) defenses.
Revised: and also its generalizability to a variety of more challenging settings, such as AT with higher budgets, partial poisoning, unseen model architectures, and stronger (ensemble or adaptive) defenses.

Original: • We further present a feature-based understanding of poisoning by analyzing the usefulness of non-robust vs. robust features in compromising standard vs. adversarially-trained models and also confirm the effectiveness of INF
Revised: • We further present a feature-based understanding of poisoning by analyzing the usefulness of non-robust vs. robust features in compromising standard vs. AT models and also propose two hybrid attacks to poison standard and AT models simultaneously.

Original: in the context of standard training.
Revised: Our anonymous code can be found at https://anonymous.4open.science/r/ INF-C277 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Data poisoning.
Revised: Data poisoning.

Original: Data poisoning aims to compromise a model’s performance at test time by manipulating its training data.
Revised: Data poisoning aims to compromise a model’s performance at test time by manipulating its training data.

Original: Related work on poisoning DNNs has mainly investigated targeted, backdoor, and indiscriminate poisoning.
Revised: Related work on poisoning DNNs has mainly investigated targeted, backdoor, and indiscriminate poisoning.

Original: Different from backdoor (Gu et al., 2017; Liu et al., 2018;
Revised: Different from backdoor (Gu et al., 2017; Liu et al., 2018;

Original: Salem et al., 2022) and targeted poisoning (Muñoz-González et al., 2017; Shafahi et al., 2018;
Revised: Salem et al., 2022) and targeted poisoning (Muñoz-González et al., 2017; Shafahi et al., 2018;

Original: Geiping et al., 2021), which aim to degrade the model on specific (targeted) test samples, indiscriminate poisoning aims at arbitrary clean test samples.
Revised: Geiping et al., 2021), which aim to degrade the model on specific (targeted) test samples, indiscriminate poisoning aims at arbitrary clean test samples.

Original: Traditional indiscriminate poisoning is based on injecting noisy labels (Biggio et al., 2012; 2011; Muñoz-González et al., 2017); however, they can be easily detected (Shafahi et al., 2018; Song et al., 2020).
Revised: Traditional indiscriminate poisoning is based on injecting noisy labels (Biggio et al., 2012; 2011; Muñoz-González et al., 2017); however, they can be easily detected (Shafahi et al., 2018; Song et al., 2020).

Original: Recent methods instead pursue “clean-label” poisons by adding imperceptible perturbations.
Revised: Recent methods instead pursue “clean-label” poisons by adding imperceptible perturbations.

Original: These methods mainly use the error-minimization (Huang et al., 2021;
Revised: These methods mainly use the error-minimization (Huang et al., 2021;

Original: Tao et al., 2021; Fu et al., 2022) or error-maximization loss (Fowl et al., 2021b), with a pre-trained (Fowl et al., 2021b;
Revised: Tao et al., 2021; Fu et al., 2022) or error-maximization loss (Fowl et al., 2021b), with a pre-trained (Fowl et al., 2021b;

Original: Wang et al., 2021; Tao et al., 2021) or trained-from-scratch (Huang et al., 2021; Fu et al., 2022) reference model.
Revised: Wang et al., 2021; Tao et al., 2021) or trained-from-scratch (Huang et al., 2021; Fu et al., 2022) reference model.

Original: However, these methods are known to be vulnerable to adversarial training (AT).
Revised: However, these methods are known to be vulnerable to adversarial training (AT).

Original: In particular, two concurrent methods, ADVIN (Wang et al., 2021) and
Revised: In particular, two concurrent methods, ADVIN (Wang et al., 2021) and

Original: REM (Fu et al., 2022), also attempt to poison AT models, but under easy settings with ϵ poi ≥ 2 ϵ adv .
Revised: REM (Fu et al., 2022), also attempt to poison AT models, but under easy settings with ϵ poi ≥ 2 ϵ adv .

Original: Generating poisons using a feature-space loss has also been explored (Shafahi et al., 2018; Zhu et al., 2019;
Revised: Generating poisons using a feature-space loss has also been explored (Shafahi et al., 2018; Zhu et al., 2019;

Original: Geiping et al., 2021), but without considering AT and in the field of targeted poisoning.
Revised: Geiping et al., 2021), but without considering AT and in the field of targeted poisoning.

Original: Adversarial training.
Revised: Adversarial training.

Original: Adversarial training (AT) was recognized as the only promising solution so far to provide robustness against (test-time) adversarial examples (Athalye et al., 2018; Tramèr et al., 2020).
Revised: Adversarial training (AT) was recognized as the only promising solution so far to provide robustness against (test-time) adversarial examples (Athalye et al., 2018; Tramèr et al., 2020).

Original: It was also recently proved to be a principled defense against indiscriminate poisoning (Tao et al., 2021).
Revised: It was also recently proved to be a principled defense against indiscriminate poisoning (Tao et al., 2021).

Original: The general idea of AT is to simply augment training data with adversarial examples generated in each training step.
Revised: The general idea of AT is to simply augment training data with adversarial examples generated in each training step.

Original: The single-step approach, FGSM, was initially used by the seminal work of Goodfellow et al.
Revised: The single-step approach, FGSM, was initially used by the seminal work of Goodfellow et al.

Original: but has been found to be ineffective against multi-step attacks (Tramèr et al., 2017; Kurakin et al., 2017).
Revised: but has been found to be ineffective against multi-step attacks (Tramèr et al., 2017; Kurakin et al., 2017).

Original: To address this limitation, Madry et al.
Revised: To address this limitation, Madry et al.

Original: (2018) have proposed the PGD-based AT, which uses the multi-step optimization to further enhance the robustness.
Revised: (2018) have proposed the PGD-based AT, which uses the multi-step optimization to further enhance the robustness.

Original: Other state-of-the-art methods have been focused on improving this PGD-based AT by, for example, training on both clean and adversarial examples (Zhang et al., 2019), incorporating an explicit differentiation of misclassified examples (Wang et al., 2020), identifying a bag of training tricks (Pang et al., 2021), or accelerating the training process via gradient recycling (Shafahi et al., 2019).
Revised: Other state-of-the-art methods have been focused on improving this PGD-based AT by, for example, training on both clean and adversarial examples (Zhang et al., 2019), incorporating an explicit differentiation of misclassified examples (Wang et al., 2020), identifying a bag of training tricks (Pang et al., 2021), or accelerating the training process via gradient recycling (Shafahi et al., 2019).

Original: In this paper, we consider three different state-of-the-art AT techniques that adopt the standard
Revised: In this paper, we consider three different state-of-the-art AT techniques that adopt the standard

Original: PGD and also varied adversarial training budget ϵ adv .
Revised: PGD and also varied adversarial training budget ϵ adv .

Original:  I NDISCRIMINATIVE F EATURES (INF) FOR P OISONING AT M ODELS 3. P ROBLEM STATEMENT
Revised:  I NDISCRIMINATIVE F EATURES (INF) FOR P OISONING AT M ODELS 3. P ROBLEM STATEMENT

Original: We formulate the problem in the context of image classification DNNs.
Revised: We formulate the problem in the context of image classification DNNs.

Original: There are two parties involved, the poisoner and the victim .
Revised: There are two parties involved, the poisoner and the victim .

Original: The poisoner has full access to the clean training dataset
Revised: The poisoner has full access to the clean training dataset

Original: D c = { ( x i , y i ) } ni =1 and is able to add perturbations δ poi to each sample and release the poisoned version D p = { ( x ′ i , y i ) } ni =1 , where x ′ i
Revised: D c = { ( x i , y i ) } ni =1 and is able to add perturbations δ poi to each sample and release the poisoned version D p = { ( x ′ i , y i ) } ni =1 , where x ′ i

Original: = x i + δ poi
Revised: = x i + δ poi

Original: i .
Revised: i .

Original: Once the poisoned dataset is generated and released, the poisoner cannot further modify the dataset.
Revised: Once the poisoned dataset is generated and released, the poisoner cannot further modify the dataset.

Original: Moreover, the poisoner has no control over the target model’s training process and the labeling function of the victim.
Revised: Moreover, the poisoner has no control over the target model’s training process and the labeling function of the victim.

Original: The victim only has access to the poisoned dataset and aims to train a well-generalized model using this dataset.
Revised: The victim only has access to the poisoned dataset and aims to train a well-generalized model using this dataset.

Original: As the victim is aware that the obtained dataset may be poisoned, they decide to deploy adversarial training to secure their model.
Revised: As the victim is aware that the obtained dataset may be poisoned, they decide to deploy adversarial training to secure their model.

Original: The goal of the poisoner is to decrease the clean test accuracy of the adversarially-trained model by poisoning its training dataset.
Revised: The goal of the poisoner is to decrease the clean test accuracy of the adversarially-trained model by poisoning its training dataset.

Original: When perturbing the clean dataset, the poisoner wants to ensure that the perturbation is imperceptible and can escape any detection from the victim.
Revised: When perturbing the clean dataset, the poisoner wants to ensure that the perturbation is imperceptible and can escape any detection from the victim.

Original: To this end, the poisoner constrains the generated perturbations δ poi by a certain poison budget ϵ poi , i.e., ∥ δ poi ∥ ∞ ≤ ϵ poi .
Revised: To this end, the poisoner constrains the generated perturbations δ poi by a certain poison budget ϵ poi , i.e., ∥ δ poi ∥ ∞ ≤ ϵ poi .

Original: Take the widely-adopted adversarial training framework (Madry et al., 2018) as an example, the victim trains a target model F on the poisoned dataset D p by a certain adversarial training budget ϵ adv following the objective:
Revised: Take the widely-adopted adversarial training framework (Madry et al., 2018) as an example, the victim trains a target model F on the poisoned dataset D p by a certain adversarial training budget ϵ adv following the objective:

Original: [Equation]
Revised: [Equation]

Original: where x ′ denotes the poisoned input, δ adv denotes the adversarial perturbations, θ denotes the model parameters, and L is the classification loss (e.g., the commonly used cross-entropy loss).
Revised: where x ′ denotes the poisoned input, δ adv denotes the adversarial perturbations, θ denotes the model parameters, and L is the classification loss (e.g., the commonly used cross-entropy loss).

Original: In this paper, we focus on the reasonable setting with ϵ poi ≤ ϵ adv .
Revised: In this paper, we focus on the reasonable setting with ϵ poi ≤ ϵ adv .

Original: In contrast, the two concurrent studies, ADVIN (Wang et al., 2021) and REM (Fu et al., 2022), focus on much easier settings with ϵ poi ≥ 2 ϵ adv , in which it is not surprising that AT would fail because the clean samples are already out of the ϵ adv -ball of the poisoned samples (Tao et al., 2021).
Revised: In contrast, the two concurrent studies, ADVIN (Wang et al., 2021) and REM (Fu et al., 2022), focus on much easier settings with ϵ poi ≥ 2 ϵ adv , in which it is not surprising that AT would fail because the clean samples are already out of the ϵ adv -ball of the poisoned samples (Tao et al., 2021).

Original: 3. M ETHODOLOGY
Revised: 3. M ETHODOLOGY

Original: In this section, we introduce INF, our new poisoning approach to compromising adversarial training.
Revised: In this section, we introduce INF, our new poisoning approach to compromising adversarial training.

Original: The key intuition of INF is to cause samples from different classes to share indiscriminative features, which then become not useful for discriminative model training, including adversarial training.
Revised: The key intuition of INF is to cause samples from different classes to share indiscriminative features, which then become not useful for discriminative model training, including adversarial training.

Original: Specifically, we propose two different variants of INF, namely INF-push and INF-pull.
Revised: Specifically, we propose two different variants of INF, namely INF-push and INF-pull.

Original: For INF-push, all training samples in each of the original classes y are pushed away from the corresponding class centroid µ y in the latent feature space (i.e., the output of the penultimate layer F ∗ L − 1 ) of a reference model F ∗ , which has totally L layers.
Revised: For INF-push, all training samples in each of the original classes y are pushed away from the corresponding class centroid µ y in the latent feature space (i.e., the output of the penultimate layer F ∗ L − 1 ) of a reference model F ∗ , which has totally L layers.

Original: The objective function can be formulated as:
Revised: The objective function can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: For INF-pull, each training sample is pulled towards the centroid of its nearest class y ′ :
Revised: For INF-pull, each training sample is pulled towards the centroid of its nearest class y ′ :

Original: [Equation]
Revised: [Equation]

Original: The above class centroid is computed as the average features of all clean samples X in that class:
Revised: The above class centroid is computed as the average features of all clean samples X in that class:

Original: [Equation]
Revised: [Equation]

Original: We find this simple, average-based method works well in our case, and we leave the exploration of other, metric learning methods (Kaya and Bilge, 2019) to future work.
Revised: We find this simple, average-based method works well in our case, and we leave the exploration of other, metric learning methods (Kaya and Bilge, 2019) to future work.

Original: In order to learn a similar representation space to that of an AT target model, the reference model F ∗ is also adversarially trained with a certain perturbation budget ϵ ref .
Revised: In order to learn a similar representation space to that of an AT target model, the reference model F ∗ is also adversarially trained with a certain perturbation budget ϵ ref .

Original: We discuss the impact of ϵ ref on the poisoning performance in Section 5.
Revised: We discuss the impact of ϵ ref on the poisoning performance in Section 5.

Original: Following the common practice, we adopt the Projected
Revised: Following the common practice, we adopt the Projected

Original: Gradient Descent (PGD) (Madry et al., 2018) to solve the above poison optimization.
Revised: Gradient Descent (PGD) (Madry et al., 2018) to solve the above poison optimization.

Original: Why adversarial training can be compromised.
Revised: Why adversarial training can be compromised.

Original: Tao et al.
Revised: Tao et al.

Original: (2021) have proved that adversarial training can serve as a principled defense against data poisoning based on the following theorem.
Revised: (2021) have proved that adversarial training can serve as a principled defense against data poisoning based on the following theorem.

Original: Theorem 1 Given a classifier f : X → Y , for any data distribution D and any perturbed distribution ˆ
Revised: Theorem 1 Given a classifier f : X → Y , for any data distribution D and any perturbed distribution ˆ

Original: D such that ˆ D ∈ B W ∞ ( D , ϵ ) , we have
Revised: D such that ˆ D ∈ B W ∞ ( D , ϵ ) , we have

Original: [Equation]
Revised: [Equation]

Original: where R nat denotes natural risk and R adv denotes adversarial risk.
Revised: where R nat denotes natural risk and R adv denotes adversarial risk.

Original: Detailed definitions can be found in Appendix E. Theorem 1 guarantees that adversarial training on the poisoned data distribution ˆ
Revised: Detailed definitions can be found in Appendix B. Theorem 1 guarantees that adversarial training on the poisoned data distribution ˆ

Original: D optimizes an upper bound of natural risk on the original data distribution D if ˆ D is within the ∞ -Wasserstein ball of D (Tao et al., 2021).
Revised: D optimizes an upper bound of natural risk on the original data distribution D if ˆ D is within the ∞ -Wasserstein ball of D (Tao et al., 2021).

Original: That is to say, achieving a low natural risk on D (i.e., high clean test accuracy) requires a low adversarial risk on ˆ D .
Revised: That is to say, achieving a low natural risk on D (i.e., high clean test accuracy) requires a low adversarial risk on ˆ D .

Original: This guarantee is based on an implicit assumption that adversarial training is capable of minimizing the adversarial risk on the poisoned data distribution ˆ D , which holds for existing attacks as they follow the shortcut-based attack strategy.
Revised: This guarantee is based on an implicit assumption that adversarial training is capable of minimizing the adversarial risk on the poisoned data distribution ˆ D , which holds for existing attacks as they follow the shortcut-based attack strategy.

Original: However, for our INF, the poisoned data that share indiscriminative features become not useful even for adversarial training, and as a result, the assumption required for the proof is broken.
Revised: However, for our INF, the poisoned data that share indiscriminative features become not useful even for adversarial training, and as a result, the assumption required for the proof is broken.

Original: Our experimental results in Appendix E validate this claim.
Revised: Our experimental results in Appendix B validate this claim.

Original: Important note on the cross-entropy loss.
Revised: Important note on the cross-entropy loss.

Original: The key novelty of our INF over existing methods lies in not only the attack strategy (indiscriminative features vs. shortcuts) but also the specific loss (feature-level vs. output-level).
Revised: The key novelty of our INF over existing methods lies in not only the attack strategy (indiscriminative features vs. shortcuts) but also the specific loss (feature-level vs. output-level).

Original: Existing methods on poisoning standard models have commonly adopted the cross-entropy (CE) loss and concluded that the targeted optimization, either with an incorrect (Fowl et al., 2021b; Tao et al., 2021; Wang et al., 2021) or original (Huang et al., 2021) class as the target, is generally stronger than the untargeted CE.
Revised: Existing methods on poisoning standard models have commonly adopted the cross-entropy (CE) loss and concluded that the targeted optimization, either with an incorrect (Fowl et al., 2021b; Tao et al., 2021; Wang et al., 2021) or original (Huang et al., 2021) class as the target, is generally stronger than the untargeted CE.

Original: This conclusion somewhat leads to the fact that the two concurrent studies on poisoning AT models (i.e., ADVIN (Wang et al., 2021) and
Revised: This conclusion somewhat leads to the fact that the two concurrent studies on poisoning AT models (i.e., ADVIN (Wang et al., 2021) and

Original: REM (Fu et al., 2022)) have completely ignored the untargeted CE as their baseline.
Revised: REM (Fu et al., 2022)) have completely ignored the untargeted CE as their baseline.

Original: However, we find that the above conclusion does not hold for poisoning AT models.
Revised: However, we find that the above conclusion does not hold for poisoning AT models.

Original: Specifically, we notice that the untargeted CE can also lead to indiscriminate features to some extent and as a result yield a substantial accuracy drop ( 12 . 02% ), while its targeted counterpart (i.e, ADVIN (Wang et al., 2021) shown in our Table 1) completely fails.
Revised: Specifically, we notice that the untargeted CE can also lead to indiscriminate features to some extent and as a result yield a substantial accuracy drop ( 12 . 02% ), while its targeted counterpart (i.e, ADVIN (Wang et al., 2021) shown in our Table 1) completely fails.

Original: Note that the untargeted CE still performs worse than our INF-push, especially in the more complex tasks, i.e., CIFAR-100 and TinyImageNet (see
Revised: Note that the untargeted CE still performs worse than our INF-push, especially in the more complex tasks, i.e., CIFAR-100 and TinyImageNet (see

Original: Appendix B for details).
Revised: Appendix C for details).

Original: This indicates that using the CE loss is not an ultimate solution in practice.
Revised: This indicates that using the CE loss is not an ultimate solution in practice.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we first compare our INF toexisting attacks under the basic setting and then validate its generalizability to more challenging settings. All experiments are performed on an NVIDIA DGXA100 server.
Revised: In this section, we conduct extensive experiments to validate that adversarial training (AT) can be compromised by our new poisoning attack. In particular, we consider challenging scenarios with high

Original: Our anonymous code can be found at https://anonymous.4open.science/ r/INF-C277 .
Revised: AT budgets, partial poisoning, unseen model architectures, or strong (ensemble or adaptive) defenses.

Original: 4. E XPERIMENTAL SETTINGS
Revised: 4. E XPERIMENTAL SETTINGS

Original: We use three image classification benchmark datasets: CIFAR-10 (CIF), CIFAR-100 (CIF), and
Revised: We use three image classification benchmark datasets: CIFAR-10 (CIF), CIFAR-100 (CIF), and

Original: TinyImageNet (Tin).
Revised: TinyImageNet (Tin).

Original: These datasets have been commonly used in the poisoning literature.
Revised: These datasets have been commonly used in the poisoning literature.

Original: We adopt the perturbation budget ϵ ref = 4 / 255 for adversarially training the reference model and find that other values also work well (see results in Section 5).
Revised: We adopt the perturbation budget ϵ ref = 4 / 255 for adversarially training the reference model and find that other values also work well (see results in Section 5).

Original: PGD- 300 with a step size of 0 . 4 / 255 and differentiable data augmentation (Fowl et al., 2021b) is used for poison optimization.
Revised: PGD- 300 with a step size of 0 . 4 / 255 and differentiable data augmentation (Fowl et al., 2021b) is used for poison optimization.

Original: If not explicitly mentioned, we focus on the reasonable setting with ϵ poi =
Revised: If not explicitly mentioned, we focus on the reasonable setting with ϵ poi =

Original: ϵ adv = 8 / 255 and adopt ResNet-18 for both the reference and target models.
Revised: ϵ adv = 8 / 255 and adopt ResNet-18 for both the reference and target models.

Original: Additional experimental settings can be found in Appendix C.
Revised: Additional experimental settings can be found in Appendix E.

Revised: All experiments are performed on an NVIDIA DGX-A100 server.

Original: [Table]
Revised: [Table]

Original: 4. INF COMPARED TO EXISTING ATTACKS UNDER ϵ adv = ϵ poi
Revised: 4. INF COMPARED TO EXISTING ATTACKS UNDER ϵ adv = ϵ poi

Original: We first evaluate the performance of different state-of-the-art poisoning methods against adversarial training in the basic setting with ϵ poi =
Revised: We first evaluate the performance of different state-of-the-art poisoning methods against adversarial training in the basic setting with ϵ poi =

Original: ϵ adv on CIFAR-10.
Revised: ϵ adv on CIFAR-10.

Original: As can be seen from Table 1, all existing methods can hardly decrease the model accuracy.
Revised: As can be seen from Table 1, all existing methods can hardly decrease the model accuracy.

Original: Specifically, although ADVIN (Wang et al., 2021) and REM (Fu et al., 2022) have claimed effectiveness in the unreasonable settings with ϵ poi ≥ 2 ϵ adv , they fail in our reasonable setting.
Revised: Specifically, although ADVIN (Wang et al., 2021) and REM (Fu et al., 2022) have claimed effectiveness in the unreasonable settings with ϵ poi ≥ 2 ϵ adv , they fail in our reasonable setting.

Original: In some cases, the poisons may even slightly increase the model accuracy, which is also noticed in the concurrent work (Tao et al., 2022)
Revised: In some cases, the poisons may even slightly increase the model accuracy, which is also noticed in the concurrent work (Tao et al., 2022)

Original: [Table]
Revised: [Table]

Original: In contrast to existing methods
Revised: In contrast to existing methods

Original: [Table]
Revised: [Table]

Original: both our INF-push and INF-pull can substantially decrease the model accuracy.
Revised: both our INF-push and INF-pull can substantially decrease the model accuracy.

Original: Note that decreasing the model accuracy to 71 .
Revised: Note that decreasing the model accuracy to 71 .

Original: 57% is dramatic because it equals the performance achieved by directly discarding 83% of the original training data (see more relevant discussions in Section 4.5).
Revised: 57% is dramatic because it equals the performance achieved by directly discarding 83% of the original training data (see more relevant discussions in Section 4.5).

Original: In addition, INF-push and INF-pull achieve similar results but obviously,
Revised: In addition, INF-push and INF-pull achieve similar results but obviously,

Original: INF-pull is less efficient because it needs to calculate and then rank the distance between each sample and class centroid.
Revised: INF-pull is less efficient because it needs to calculate and then rank the distance between each sample and class centroid.

Original: Moreover, the class selection strategy in INF-pull may have an impact on the final performance (see more analysis in Section 6).
Revised: Moreover, the class selection strategy in INF-pull may have an impact on the final performance (see more analysis in Appendix D).

Original: For these reasons, if not specifically mentioned, we choose to use INF-push in the following experiments.
Revised: For these reasons, if not specifically mentioned, we choose to use INF-push in the following experiments.

Original: [Figure]
Revised: [Figure]

Original: 4. INF FOR LARGER DATASETS AND OTHER AT FRAMEWORKS 
Revised: 4. INF FOR LARGER DATASETS AND OTHER AT FRAMEWORKS UNDER ϵ adv = ϵ poi

Original: The results shown in Table 2 further validate the general effectiveness of our INF on larger datasets, where the model accuracy consistently drops by more than 10% .
Revised: The results shown in Table 2 further validate the general effectiveness of our INF on larger datasets, where the model accuracy consistently drops by more than 10% .

Original: We also evaluate INF against different widely-used AT frameworks.
Revised: We also evaluate INF against different widely-used AT frameworks.

Original: Figure 2 shows the learning curves of the poisoned AT target model that is trained with Madry (Madry et al., 2018), TRADES (Zhang et al., 2019), or
Revised: Figure 2 shows the learning curves of the poisoned AT target model that is trained with Madry (Madry et al., 2018), TRADES (Zhang et al., 2019), or

Original: MART (Wang et al., 2020).
Revised: MART (Wang et al., 2020).

Original: As can be seen, our INF largely decreases the clean test accuracy in all cases.
Revised: As can be seen, our INF largely decreases the clean test accuracy in all cases.

Original: We can also observe that all three frameworks exhibit a relatively steady learning process, i.e., the model accuracy monotonically increases over epochs, and finally reaches an accuracy that is still lower than that of the model trained on clean data.
Revised: We can also observe that all three frameworks exhibit a relatively steady learning process, i.e., the model accuracy monotonically increases over epochs, and finally reaches an accuracy that is still lower than that of the model trained on clean data.

Original: This pattern is different from that in poisoning standard models, where the model accuracy is found to increase at a few early epochs, and then start to decrease dramatically to the final low accuracy (Huang et al., 2021; Liu et al., 2021; Segura et al., 2022a).
Revised: This pattern is different from that in poisoning standard models, where the model accuracy is found to increase at a few early epochs, and then start to decrease dramatically to the final low accuracy (Huang et al., 2021; Liu et al., 2021; Segura et al., 2022a).

Original: This fundamental difference indicates that early stopping cannot be used as an effective defense against poisoning for AT models
Revised: This fundamental difference indicates that early stopping cannot be used as an effective defense against poisoning for AT models

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. INF UNDER HIGHER AT BUDGETS
Revised: 4. INF UNDER HIGHER AT BUDGETS

Original: We further test INF under higher adversarial training budgets and also consider different poison budgets.
Revised: We further test INF under higher adversarial training budgets and also consider different poison budgets.

Original: As can be seen from Table 3, even with an overwhelming budget, adversarial training can still be largely degraded by our INF.
Revised: As can be seen from Table 3, even with an overwhelming budget, adversarial training can still be largely degraded by our INF.

Original: For example, when the AT budget is ϵ adv = 16 / 255 , which is 2 × larger than the poison budget ϵ poi = 8 / 255 , our INF still yields a substantial accuracy drop of 10 .
Revised: For example, when the AT budget is ϵ adv = 16 / 255 , which is 2 × larger than the poison budget ϵ poi = 8 / 255 , our INF still yields a substantial accuracy drop of 10 .

Original: In addition, under the same setting with ϵ adv =
Revised: In addition, under the same setting with ϵ adv =

Original: ϵ poi , a larger poison budget leads to a better poison performance.
Revised: ϵ poi , a larger poison budget leads to a better poison performance.

Original: Specifically, for ϵ adv =
Revised: Specifically, for ϵ adv =

Original: ϵ poi
Revised: ϵ poi

Original: = 4 / 255 , model accuracy drops by 5 .
Revised: = 4 / 255 , model accuracy drops by 5 .

Original: 31% → 84 .
Revised: 31% → 84 .

Original: 37% ), while for a larger poison budget, 8 / 255 and 16 / 255 , the accuracy drops by 13 .
Revised: 37% ), while for a larger poison budget, 8 / 255 or 16 / 255 , the accuracy drops by 13 .

Original: 88% → 71 . 57% ) and 20 .
Revised: 88% → 71 . 57% ) or 20 .

Original: 78% → 53 . 03% ), respectively.
Revised: 78% → 53 . 03% ).

Original: We can also observe that under a specific poison budget, the model accuracy and AT budget do not have a clear correlation.
Revised: We can also observe that under a specific poison budget, the model accuracy and AT budget do not have a clear correlation.

Original: This is reasonable because although enlarging the AT budget can increase clean accuracy due to higher robustness to poisons, it also inevitably leads to an accuracy drop compared to standard training.
Revised: This is reasonable because although enlarging the AT budget can increase clean accuracy due to higher robustness to poisons, it also inevitably leads to an accuracy drop compared to standard training.

Original: We leave detailed explorations for future work
Revised: We leave detailed explorations for future work.

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4. INF UNDER PARTIAL POISONING
Revised: 4. INF UNDER PARTIAL POISONING

Original: We also examine INF in a more challenging scenario where only partial training data are allowed to be poisoned.
Revised: We also examine INF in a more challenging scenario where only partial training data are allowed to be poisoned.

Original: We consider two different poisoning settings where different data are used for calculating the class centroids.
Revised: We consider two different poisoning settings where different data are used for calculating the class centroids.

Original: Specifically, the first setting is based on the whole original (clean) dataset but the second, worse-case one is based on only the partial clean data that are allowed to be poisoned.
Revised: Specifically, the first setting is based on the whole original (clean) dataset but the second, worse-case one is based on only the partial clean data that are allowed to be poisoned.

Original: As can be seen from Table 4, our INF is still effective in this challenging scenario.
Revised: As can be seen from Table 4, our INF is still effective in this challenging scenario.

Original: In comparison, other attacks can hardly decrease the model accuracy even when the whole dataset is poisoned (see
Revised: Recall that other attacks can hardly decrease the model accuracy even when the whole dataset is poisoned.

Original: Table 1).

Original: In particular, INF decreases the model accuracy from 84 .
Revised: In particular, INF decreases the model accuracy from 84 .

Original: 88% to 81 .
Revised: 88% to 81 .

Original: 41% by only poisoning 0.2 of the training data, and this result is also lower than the baseline that is achieved by directly discarding these poisoned data ( 83 . 66% ).
Revised: 41% by only poisoning 0.2 of the training data, and this result is also lower than the baseline that is achieved by directly discarding these poisoned data ( 83 . 66% ).

Original: We can also observe that the two different poisoning settings yield very similar results.
Revised: We can also observe that the two different poisoning settings yield very similar results.

Original: This indicates that the calculation of class centroids in our INF is not sensitive to the amount of data, and as a result, its efficiency can be potentially improved by using less data for the centroid calculation.
Revised: This indicates that the calculation of class centroids in our INF is not sensitive to the amount of data, and as a result, its efficiency can be potentially improved by using less data for the centroid calculation.

Original: The fact that poisoning more data yields better performance can also be explained by Figure 3 where a larger poison proportion leads to a larger number of indiscriminative features
Revised: The fact that poisoning more data yields better performance can also be explained by Figure 3 where a larger poison proportion leads to a larger number of indiscriminative features.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: T RANSFERABILITY OF INF TO UNSEEN MODEL ARCHITECTURES
Revised: T RANSFERABILITY OF INF TO UNSEEN MODEL ARCHITECTURES

Original: The latent feature space that is used for generating poisons is specific to a certain reference model.
Revised: The latent feature space that is used for generating poisons is specific to a certain reference model.

Original: For this reason, one natural question to ask is whether the poisons generated on one model architecture are still effective when the target model adopts a different architecture.
Revised: For this reason, one natural question to ask is whether the poisons generated on one model architecture are still effective when the target model adopts a different architecture.

Original: Table 5 demonstrates that the poisoning effects of our INF optimized against a ResNet-18 reference model can transfer to other target model architectures.
Revised: Table 5 demonstrates that the poisoning effects of our INF optimized against a ResNet-18 reference model can transfer to other target model architectures.

Original: Specifically, for the four different (unseen) architectures, the generated poisons are able to degrade the model accuracy to almost the same extent, indicating that strong generalizability of our INF.
Revised: Specifically, for the four different (unseen) architectures, the generated poisons are able to degrade the model accuracy to almost the same extent, indicating that strong generalizability of our INF.

Original: against defenses that apply both data augmentations and AT.
Revised: and AT.

Original: [Table]
Revised: [Table]

Original: 4. INF AGAINST OTHER DEFENSES
Revised: 4. INF AGAINST OTHER DEFENSES

Original: Ensemble defenses with data augmentations.
Revised: Ensemble defenses with data augmentations.

Original: Applying additional data augmentations before standard training has been shown to be able to mitigate the effects of perturbation-based poisons (Huang et al., 2021;
Revised: Applying additional data augmentations before standard training has been shown to be able to mitigate the effects of perturbation-based poisons (Huang et al., 2021;

Original: Fowl et al., 2021b;
Revised: Fowl et al., 2021b;

Original: Liu et al., 2021; Tao et al., 2021).
Revised: Liu et al., 2021; Tao et al., 2021).

Original: Here we study if data augmentations can complement adversarial training when facing our INF.
Revised: Here we study if data augmentations can complement adversarial training when facing our INF.

Original: Following previous work, we test a diverse set of data augmentations, including random noise, Mixup (Zhang et al., 2018), Cutmix (Yun et al., 2019), and Cutout (Devries and Taylor, 2017).
Revised: Following previous work, we test a diverse set of data augmentations, including random noise, Mixup (Zhang et al., 2018), Cutmix (Yun et al., 2019), and Cutout (Devries and Taylor, 2017).

Original: We also test gray-scale pre-filtering (Liu et al., 2021), which shows strong performance in mitigating unlearnable examples (Huang et al., 2021).
Revised: We also test gray-scale pre-filtering (Liu et al., 2021), which shows strong performance in mitigating unlearnable examples (Huang et al., 2021).

Original: Tableshows that all the data augmentation methods fail to help AT to mitigate our INF.
Revised: Tableshows that all the data augmentation methods fail to help AT to mitigate our INF.

Original: Adaptive defenses by filtering out indiscriminative samples.
Revised: Adaptive defenses by filtering out indiscriminative samples.

Original: In addition to the above existing defense methods, we also consider stronger, adaptive defenses that the victim may design based on a certain level of knowledge about our INF.
Revised: In addition to the above existing defense methods, we also consider stronger, adaptive defenses that the victim may design based on a certain level of knowledge about our INF.

Original: It is worth noting that, in realistic scenarios, the victim can only leverage a poisoned model, and so it also has no access to a clean AT reference model, which is available to the poisoner, including our INF.
Revised: It is worth noting that, in realistic scenarios, the victim can only leverage a poisoned model, and so it also has no access to a clean AT reference model, which is available to the poisoner, including our INF.

Original: This is reasonable because if it is indeed feasible for the victim to get a clean AT (reference) model, there is no need for dealing with the poisoned data in the first place,  the clean AT model can already be used as an effective target model.
Revised: This is reasonable because if it is indeed feasible for the victim to get a clean AT (reference) model, there is no need for dealing with the poisoned data in the first place, and the clean AT model can already be used as an effective target model.

Original: When the victim knows that indiscriminative features have been introduced by INF, they would filter out the “overlapped samples”, which are located close in the feature space but from different classes.
Revised: When the victim knows that indiscriminative features have been introduced by INF, they would filter out the “overlapped samples”, which are located close in the feature space but from different classes.

Original: We test by removing different proportions of such “overlapped samples” and find that the best setting can only recover the accuracy from 71 .
Revised: We test by removing different proportions of such “overlapped samples” and find that the best setting can only recover the accuracy from 71 .

Original: 57% to 72 .
Revised: 57% to 72 .

Original: We go a step further by considering a stronger victim who even knows the specific algorithm of INF-push (i.e., Equation 2).
Revised: We go a step further by considering a stronger victim who even knows the specific algorithm of INF-push (i.e., Equation 2).

Original: In this case, the victim would recover the data by pulling the poisoned samples back towards their original class centroids.
Revised: In this case, the victim would recover the data by pulling the poisoned samples back towards their original class centroids.

Original: We find that this defense is stronger than the above but still can only recover the accuracy to 75 .
Revised: We find that this defense is stronger than the above but still can only recover the accuracy to 75 .

Original: 32% (about 10% lower than the clean AT accuracy).
Revised: 32% (about 10% lower than the clean AT accuracy).

Original: We also consider a practical scenario in which the victim can leverage a standard or adversarially-trained CIFAR-100  model.
Revised: We also consider a practical scenario in which the victim can leverage an ST or AT CIFAR-100 model as a general-purpose model.

Original: In this scenario, the adaptive defense only recovers the model accuracy to 72 .
Revised: In this scenario, the adaptive defense only recovers the model accuracy to 72 .

Original: Adaptive defenses through feature perturbation-based AT.
Revised: Adaptive defenses through feature perturbation-based AT.

Original: We further test a new defense that uses Eq.
Revised: We further test a new defense that uses Eq.

Original: 2 for generating the adversarial examples in AT instead of the common, cross-entropy loss.
Revised: 2 for generating the adversarial examples in AT instead of the common, cross-entropy loss.

Original: When trained on clean data, this new AT variant yields an accuracy of 86 .
Revised: When trained on clean data, this new AT variant yields an accuracy of 86 .

Original: 84% , similar to that achieved by the conventional AT.
Revised: 84% , similar to that achieved by the conventional AT.

Original: However, when trained on our poisoned data, the model accuracy still substantially drops to 72 .
Revised: However, when trained on our poisoned data, the model accuracy still substantially drops to 72 .

Original: 99% , indicating that it is not a satisfying defense.
Revised: 99% , indicating that it is not a satisfying defense.

Original: This defense is even worse than the above one with a pre-trained AT model ( 72 . 99% vs. 75 .
Revised: This defense is even worse than the above one with a pre-trained AT model ( 72 . 99% vs. 75 .

Original: This might be because the class centroids calculated when the model is not well trained (in the early AT training stage) cannot provide meaningful guidance compared to those based on the pre-trained model.
Revised: This might be because the class centroids calculated when the model is not well trained (in the early AT training stage) cannot provide meaningful guidance compared to those based on the pre-trained model.

Original: As a sanity check, we also try another AT variant that uses a reverse loss of Eq. 2 and find that as expected, it causes the model accuracy to drop a lot (to 47 .
Revised: As a sanity check, we also try another AT variant that uses a reverse loss of Eq. 2 and find that as expected, it causes the model accuracy to drop a lot (to 47 .

Original: [Figure]
Revised: [Figure]

Original:  ) vs. adversarially-trained (AT) target model.
Revised: 0 ) vs. adversarially-trained (AT) target model.

Original: (a) Clean test accuracy for both ST and AT; (b) Perturbation visualizations for different ϵ ref = 0 .
Revised: (a) Clean test accuracy for both ST and AT; (b) Perturbation visualizations for different ϵ ref = 0 .

Original: More visualizations can be found in Appendix F.
Revised: More visualizations can be found in Appendix G.

Original:  R OBUST VS . N ON -R OBUST P OISONS
Revised:  R OBUST VS . N ON -R OBUST P OISONS

Original: All our experiments have so far been focused on poisoning AT models.
Revised: All our experiments have so far been focused on poisoning AT models.

Original: However, it is also valuable to study the problem in the context of standard training and figure out the difference.
Revised: However, it is also valuable to study the problem in the context of standard training and figure out the difference.

Original: To this end, we analyze the poisons against a standard (ST) model vs. an AT model.
Revised: To this end, we analyze the poisons against a standard (ST) model vs. an AT model.

Original: Specifically, we adjust the perturbation budget ϵ ref for adversarially training the reference model and analyze the poisons in terms of both the poisoning strength and the visual characteristics of perturbations.
Revised: Specifically, we adjust the perturbation budget ϵ ref for adversarially training the reference model and analyze the poisons in terms of both the poisoning strength and the visual characteristics of perturbations.

Original: As can be seen from Figure 4a, for poisoning the AT model, the poisoning strength is gradually improved as the reference model becomes more robust (i.e., ϵ ref is increased).
Revised: As can be seen from Figure 4a, for poisoning the AT model, the poisoning strength is gradually improved as the reference model becomes more robust (i.e., ϵ ref is increased).

Original: In contrast, for poisoning the ST model, the poisoning is gradually degraded.
Revised: In contrast, for poisoning the ST model, the poisoning is gradually degraded.

Original: Concurrent work (Tao et al., 2022) also discusses the impact of ϵ ref on poisoning AT models but focuses on a fundamentally different task where the attack aims to degrade the adversarial robustness of AT models (rather than the clean accuracy here).
Revised: Concurrent work (Tao et al., 2022) also discusses the impact of ϵ ref on poisoning AT models but focuses on a fundamentally different task where the attack aims to degrade the adversarial robustness of AT models (rather than the clean accuracy here).

Original: It is also important to note that our INF can work well under different ϵ ref betweenand 8, where their approach is much more sensitive to the choice of ϵ ref in their task (see their Figure 2(a)).
Revised: It is also important to note that our INF can work well under different ϵ ref betweenand 8, where their approach is much more sensitive to the choice of ϵ ref in their task (see their Figure 2(a)).

Original: We further visualize the perturbations generated with different ϵ ref in 4b.
Revised: We further visualize the perturbations generated with different ϵ ref in 4b.

Original: As can be seen, when using a standard reference model (i.e., ϵ ref = 0 ), the perturbations exhibit noisy patterns, but as the ϵ ref is gradually increased, the perturbations tend to be more aligned with image semantics.
Revised: As can be seen, when using a standard reference model (i.e., ϵ ref = 0 ), the perturbations exhibit noisy patterns, but as the ϵ ref is gradually increased, the perturbations tend to be more aligned with image semantics.

Original: These observations suggest that poisoning the ST and AT models require modifying different types of features.
Revised: These observations suggest that poisoning the ST and AT models require modifying different types of features.

Original: More specifically, modifying the robust (semantic) features is the key to poisoning the
Revised: More specifically, modifying the robust (semantic) features is the key to poisoning the

Original: AT models, while modifying the non-robust features is the key to poisoning the ST models.
Revised: AT models, while modifying the non-robust features is the key to poisoning the ST models.

Original: This conclusion also supports the well-known perspective that non-robust features can be picked up by models during standard training, even in the presence of robust features, while adversarial training tends to utilize robust features (Ilyas et al., 2019).
Revised: This conclusion also supports the well-known perspective that non-robust features can be picked up by models during standard training, even in the presence of robust features, while adversarial training tends to utilize robust features (Ilyas et al., 2019).

Original: Figure 4a also confirms that our INF is effective in poisoning ST models since it can decrease the model accuracy to the random guess level (i.e., 10% for CIFAR-10).
Revised: Figure 4a also confirms that our INF is effective in poisoning ST models since it can decrease the model accuracy to the random guess level (i.e., 10% for CIFAR-10).

Original: See Appendix D for a simple method ofcombining robust and non-robust features  to poison  ST and AT  simultaneously.[Figure]  A DDITIONAL A NALYSIS OF I NDISCRIMINATIVE F EATURES
Revised: Hybrid attacks . The distinct roles of robust and non-robust features in poisoning AT and ST models inspire us to search for a hybrid attack to poison the ST and AT models simultaneously.

Original: The above experimental results have demonstrated the effectiveness of INF in various scenarios.
Revised: A straightforward idea is to simply average the perturbations generated using an ST reference model and those using an AT reference model.

Original: Here we provide additional analysis to better understand the property of indiscriminative features. To this end, we adjust the class selection strategy in INF-pull to be simply based on pairwise indiscriminative features. Specifically, each pair consists of two classes that have a minimal centroid distance. For optimization, we calculate the centroids of each class pair and then minimize the distance between samples in one class and the centroid of the other class.
Revised: In this case, the poison budget remains unchanged.

Original: We denote this attackvariant targeting pairwise indiscriminative features as INF-pull-pair. We find that INF-pull-pair can decrease the model accuracy from 84 . 
Revised: We also propose another hybrid attack that is based on jointly optimizing the perturbations against both an ST and an AT reference model, balanced by a factor λ . This modifies Eq.

Original: 88% to 78 .
Revised: 2 to the following Eq.

Original: 21% , which indicates that introducing pairwise indiscriminative features can already substantially compromise adversarial training. However, there is still a large performance gap between INF-pull-pair and our original INF. This can be explained by the fact that INF-pull exploits more diverse pulling directions based on the sample-class distance, and INF-push makes samples from multiple classes become overlapped.
Revised: [Equation]

Original: The visualizations in Figure 5 clearly confirm the above observation that INF-pull-pair yields indiscriminative features to some extent but still fewer than the original INF-pull and INF-push.
Revised: Table 7 shows that both hybrid attacks substantially decrease the model accuracy across different AT budgets ϵ adv varying from 0 (i.e., ST) to 16. Specifically, the optimization-based method performs much better than the average-based method. [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have proposed INF, a new poisoning approach to decreasing the deep learning classifier’s accuracy even when adversarial training is applied.
Revised: In this paper, we have proposed INF, a new poisoning approach to decreasing the deep learning classifier’s accuracy even when adversarial training is applied.

Original: This approach is based on a new attack strategy that makes the features of training samples from different classes become indiscriminative.
Revised: This approach is based on a new attack strategy that makes the features of training samples from different classes become indiscriminative.

Original: Extensive experiments demonstrate the effectiveness of INF against adversarial training in different scenarios, including those with more aggressive AT budgets, unseen model architectures, and adaptive defenses.
Revised: Extensive experiments demonstrate the effectiveness of INF against adversarial training in different scenarios, including those with more aggressive AT budgets, unseen model architectures, and adaptive defenses.

Original: We also discuss the distinct roles of the robust vs. non-robust features in poisoning standard vs. adversarially-trained models and confirm the effectiveness of INF in poisoning standard models.
Revised: We also discuss the distinct roles of the robust vs. non-robust features in poisoning standard vs. adversarially-trained models and demonstrate that our hybrid attacks can poison standard and AT models simultaneously.

Original: We encourage future research to analyze INF in more comprehensive settings and compare it to the current, shortcut-based methods from different angles.
Revised: We encourage future research to analyze INF in more comprehensive settings and compare it to the current, shortcut-based methods from different angles.

Original: In particular, it is important to come up with new defenses against INF, possibly based on advanced techniques of learning from noisy labels (Song et al., 2020).
Revised: In particular, it is important to come up with new defenses against INF, possibly based on advanced techniques of learning from noisy labels (Song et al., 2020).

Original: It is also worth noting that most of the current poisoning studies, including ours, have assumed that the poisoner has access to the training dataset of the target model.
Revised: It is also worth noting that most of the current poisoning studies, including ours, have assumed that the poisoner has access to the training dataset of the target model.

Original: This assumption is realistic in specific threat models (e.g., secure dataset release (Fowl et al., 2021a)) but may not be plausible for sensitive/private data.
Revised: This assumption is realistic in specific threat models (e.g., secure dataset release (Fowl et al., 2021a)) but may not be plausible for sensitive/private data.

Original: Therefore, it would be promising to extend INF to addressing data-free poisons (Yu et al., 2022; Segura et al., 2022b).
Revised: Therefore, it would be promising to extend INF to addressing data-free poisons (Yu et al., 2022; Segura et al., 2022b).

Original: On the one hand, data poisoning could be potentially leveraged by malicious parties as attacks.
Revised: On the one hand, data poisoning could be potentially leveraged by malicious parties as attacks.

Original: In this case, we hope our work can inspire the community to develop stronger defenses based on our comprehensive analysis.
Revised: In this case, we hope our work can inspire the community to develop stronger defenses based on our comprehensive analysis.

Original: On the other hand, when data poisoning is directly used for social good, e.g., for protecting personal data from being misused (Fowl et al., 2021a; Huang et al., 2021;
Revised: On the other hand, when data poisoning is directly used for social good, e.g., for protecting personal data from being misused (Fowl et al., 2021a; Huang et al., 2021;

Original: Fu et al., 2022), our new approach for generating stronger poisons lead to stronger protective effects.
Revised: Fu et al., 2022), our new approach for generating stronger poisons lead to stronger protective effects.

Original: A D ISCRIMINATIVE F EATURES IN E XISTING M ETHODS
Revised: A D ISCRIMINATIVE F EATURES IN E XISTING M ETHODS

Original: Instead of introducing indiscriminative features, most existing poisoning methods rely on generating certain shortcut perturbations that can be more easily learned by the target model than the actual image content (Tao et al., 2021;
Revised: Instead of introducing indiscriminative features, most existing poisoning methods rely on generating certain shortcut perturbations that can be more easily learned by the target model than the actual image content (Tao et al., 2021;

Original: Segura et al., 2022a; Evtimov et al., 2021; Yu et al., 2022).
Revised: Segura et al., 2022a; Evtimov et al., 2021; Yu et al., 2022).

Original: Figurevisualizes the poison representations of different existing methods on an adversarially-trained model.
Revised: Figurevisualizes the poison representations of different existing methods on an adversarially-trained model.

Original: As can be seen, although there exist certain differences between the patterns of different methods (e.g., regarding the relative positions of different classes), all these methods indeed yield well-separable data.
Revised: As can be seen, although there exist certain differences between the patterns of different methods (e.g., regarding the relative positions of different classes), all these methods indeed yield well-separable data.

Original: This observation also confirms that causing indiscriminative features is a sufficient condition to degrade adversarial training.
Revised: This observation also confirms that causing indiscriminative features is a sufficient condition to degrade adversarial training

Original: [Figure]
Revised: [Figure]

Original: B D ETAILED R ESULTS FOR INF VS . THE CE LOSS

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: C A DDITIONAL E XPERIMENTAL S ETTINGS

Original: All reference and target models are trained for 100 epochs using SGD optimizer with an initial learning rate of 0 .

Original: 1 that is decayed by a factor of 0.1 at the 75 -th and 90 -th training epochs.

Original: The optimizer is set with momentum 0 .

Original: 9 and weight decay 5 × 10 − 4 .

Original: The inner maximization (i.e., generation of adversarial examples) of the adversarial training is solved by 10-step PGD with a step size of 2 / 255 .

Original: D A DDITIONAL E XPERIMENTAL R ESULTS

Original: The independent roles of the robust and non-robust features in poisoning AT and ST models inspire us to search for a generally effective poisoning method against both the ST and AT models.

Original: To this end, for each image, we simply take an average of the non-robust and robust perturbations, which ensures that the poison budget remains unchanged.

Original: This simple method leads to an accuracy of 22 . 86% for poisoning an ST model and 77 .

Original: 19% for poisoning an AT model.

Original: These results indicate that it is possible to poison ST and AT simultaneously.

Original: We leave more detailed explorations in this direction for future work.

Original: [Figure]

Original: [Figure]

Original: E N ATURAL AND A DVERSARIAL R ISK
Revised: B N ATURAL AND A DVERSARIAL R ISKS

Original: [Figure]
Revised: [Figure]

Original: According to Tao et al.
Revised: According to Tao et al.

Original: [Figure] natural risk and adversarial risk are defined as follows.
Revised: (2021), natural risk and adversarial risk are defined as follows.

Original: Definition 1 (Natural Risk)
Revised: Definition 1 (Natural Risk)

Original: [Equation]
Revised: [Equation]

Original: Definition 2 (Adversarial Risk)
Revised: Definition 2 (Adversarial Risk)

Original: [Equation]
Revised: [Equation]

Original: For the whole training process, we calculate the adversarial risk based on Definition 2.
Revised: For the whole training process, we calculate the adversarial risk based on Definition 2.

Original: The adversarial risks for both clean and poison data are shown in Figure.
Revised: The adversarial risks for both clean and poison data are shown in Figure.

Original: It shows that adversarial training faces difficulties when optimizing poison data, as the risk decreases more slowly than the clean case.
Revised: It shows that adversarial training faces difficulties when optimizing poison data, as the risk decreases more slowly than the clean case.

Revised: C D ETAILED R ESULTS FOR INF VS . THE CE LOSS

Revised: [Table]

Original: F A DDITIONAL P ERTURBATION V ISUALIZATIONS
Revised: D A DDITIONAL A NALYSIS OF I NDISCRIMINATIVE F EATURES

Revised: Our experimental results have demonstrated the effectiveness of INF in various scenarios.

Revised: Here we provide additional analysis to better understand the property of indiscriminative features.

Revised: To this end, we adjust the class selection strategy in INF-pull to be simply based on pairwise indiscriminative features.

Revised: Specifically, each pair consists of two classes that have a minimal centroid distance.

Revised: For optimization, we calculate the centroids of each class pair and then minimize the distance between samples in one class and the centroid of the other class.

Revised: We denote this attack variant targeting pairwise indiscriminative features as INF-pull-pair.

Revised: We find that INF-pull-pair can decrease the model accuracy from 84 .

Revised: 88% to 78 .

Revised: 21% , which indicates that introducing pairwise indiscriminative features can already substantially compromise adversarial training.

Revised: However, there is still a large performance gap between INF-pull-pair and our original

Revised: INF.

Revised: This can be explained by the fact that INF-pull exploits more diverse pulling directions based on the sample-class distance, and INF-push makes samples from multiple classes become overlapped.

Revised: The visualizations in Figure 7 clearly confirm the above observation that INF-pull-pair yields indiscriminative features to some extent but still fewer than the original INF-pull and INF-push.

Revised: E A DDITIONAL E XPERIMENTAL S ETTINGS

Revised: All reference and target models are trained for 100 epochs using SGD optimizer with an initial learning rate of 0 .

Revised: 1 that is decayed by a factor of 0.1 at the 75 -th and 90 -th training epochs.

Revised: The optimizer is set with momentum 0 .

Revised: 9 and weight decay 5 × 10 − 4 .

Revised: The inner maximization (i.e., generation of adversarial examples) of the adversarial training is solved by 10-step PGD with a step size of 2 / 255 .

Revised: F A DDITIONAL E XPERIMENTAL R ESULTS

Revised: [Figure]

Revised: [Figure]

Revised: G A DDITIONAL P ERTURBATION V ISUALIZATIONS

Original: [Figure]
Revised: [Figure]