Original: Can Adversarial Training Be Manipulated By Non-Robust Features?
Revised: Can Adversarial Training Be Manipulated By Non-Robust Features?

Original: Abstract
Revised: Abstract

Original: Adversarial training, originally designed to resist test-time adversarial examples, has shown to be promising in mitigating training-time availability attacks .
Revised: Adversarial training, originally designed to resist test-time adversarial examples, has shown to be promising in mitigating training-time availability attacks .

Original: This defense ability, however, is challenged in this paper.
Revised: This defense ability, however, is challenged in this paper.

Original: We identify a novel threat model named stability attack , which aims to hinder robust availability by slightly manipulating the training data.
Revised: We identify a novel threat model named stability attack , which aims to hinder robust availability by slightly manipulating the training data.

Original: Under this threat, we show that adversarial training using a conventional defense budget ϵ provably fails to provide test robustness in a simple statistical setting, where the non-robust features of the training data can be reinforced by ϵ -bounded perturbation.
Revised: Under this threat, we show that adversarial training using a conventional defense budget ϵ provably fails to provide test robustness in a simple statistical setting, where the non-robust features of the training data can be reinforced by ϵ -bounded perturbation.

Original: Further, we analyze the necessity of enlarging the defense budget to counter stability attacks.
Revised: Further, we analyze the necessity of enlarging the defense budget to counter stability attacks.

Original: Finally, comprehensive experiments demonstrate that stability attacks are harmful on benchmark datasets, and thus the adaptive defense is necessary to maintain robustness.
Revised: Finally, comprehensive experiments demonstrate that stability attacks are harmful on benchmark datasets, and thus the adaptive defense is necessary to maintain robustness.

Original:  Introduction
Revised:  Introduction

Original: Robustness to input perturbations is crucial to machine learning deployment in various applications, such as spam filtering [13] and autonomous driving [6].
Revised: Robustness to input perturbations is crucial to machine learning deployment in various applications, such as spam filtering [13] and autonomous driving [6].

Original: One of the most popular methods for improving test robustness is adversarial training [39, 1].
Revised: One of the most popular methods for improving test robustness is adversarial training [39, 1].

Original: By augmenting the training data with ϵ -bounded and on-the-fly crafted adversarial examples, adversarial training helps the learned model resist test-time perturbations [39]
Revised: By augmenting the training data with ϵ -bounded and on-the-fly crafted adversarial examples, adversarial training helps the learned model resist test-time perturbations [39]

Original: On the other hand machine learning systems are vulnerable to training-time availability attacks [3].
Revised: On the other hand machine learning systems are vulnerable to training-time availability attacks [3].

Original: In particular, small perturbations applied into the training data (before training) suffice to degrade the overall test performance of naturally trained models [16, 28].
Revised: In particular, small perturbations applied into the training data (before training) suffice to degrade the overall test performance of naturally trained models [16, 28].

Original: Fortunately, recent work has proven that adversarial training [39] is capable of mitigating this type of threat [64].
Revised: Fortunately, recent work has proven that adversarial training [39] is capable of mitigating this type of threat [64].

Original: In other words, even if the training data is manipulated to maximize the test error, considerable accuracy on clean test data can still be achieved by adversarially trained models.
Revised: In other words, even if the training data is manipulated to maximize the test error, considerable accuracy on clean test data can still be achieved by adversarially trained models.

Original: However, previous work hardly inspects the test robustness of the models, which is what adversarial training was originally proposed for [23, 39].
Revised: However, previous work hardly inspects the test robustness of the models, which is what adversarial training was originally proposed for [23, 39].

Original: This naturally raises the following question:
Revised: This naturally raises the following question:

Original: Are the models adversarially trained on the manipulated data robust to test-time perturbations?
Revised: Are the models adversarially trained on the manipulated data robust to test-time perturbations?

Original: [Figure]
Revised: [Figure]

Original: In this work, we show that conventional adversarial training may fail to provide test robustness when the training data is manipulated by an adversary, and thus an adaptive defense is necessary to resolve this issue.
Revised: In this work, we show that conventional adversarial training may fail to provide test robustness when the training data is manipulated by an adversary, and thus an adaptive defense is necessary to resolve this issue.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: We introduce a novel threat model called stability attack , where an adversary aims to degrade the overall test robustness of adversarially trained models by slightly perturbing the training data.
Revised: We introduce a novel threat model called stability attack , where an adversary aims to degrade the overall test robustness of adversarially trained models by slightly perturbing the training data.

Original: Figure 1 illustrates the threat of stability attacks.
Revised: Figure 1 illustrates the threat of stability attacks.

Original: We show that adversarial training using a conventional defense budget provably fails under stability attacks in a simple statistical setting.
Revised: We show that adversarial training using a conventional defense budget provably fails under stability attacks in a simple statistical setting.

Original: Specifically, a defense budget of ϵ will produce models that are not robust to ϵ -bounded adversarial examples when the training data is hypocritically perturbed.
Revised: Specifically, a defense budget of ϵ will produce models that are not robust to ϵ -bounded adversarial examples when the training data is hypocritically perturbed.

Original: We unveil that the aforementioned vulnerability stems from the existence of the non-robust (predictive, yet brittle) features [29] in the original training data.
Revised: We unveil that the aforementioned vulnerability stems from the existence of the non-robust (predictive, yet brittle) features [29] in the original training data.

Original: When the non-robust features are reinforced by hypocritical perturbations, the conventional defense budget will be insufficient to offset the negative impact.
Revised: When the non-robust features are reinforced by hypocritical perturbations, the conventional defense budget will be insufficient to offset the negative impact.

Original: We further show that a defense budget of 2 ϵ is capable of resisting any stability attack for adversarial training, while the budget can be reduced to ϵ + η in a simple statistical setting, where η is the magnitude of the non-robust features.
Revised: We further show that a defense budget of 2 ϵ is capable of resisting any stability attack for adversarial training, while the budget can be reduced to ϵ + η in a simple statistical setting, where η is the magnitude of the non-robust features.

Original: We demonstrate that stability attacks are harmful to conventional adversarial training on benchmark datasets.
Revised: We demonstrate that stability attacks are harmful to conventional adversarial training on benchmark datasets.

Original: In addition, our empirical study suggests that enlarging the defense budget is essential for mitigating hypocritical perturbations.
Revised: In addition, our empirical study suggests that enlarging the defense budget is essential for mitigating hypocritical perturbations.

Original: To the best of our knowledge, this is the first work that studies the robustness of adversarial training against stability attacks.
Revised: To the best of our knowledge, this is the first work that studies the robustness of adversarial training against stability attacks.

Original: Both theoretical and empirical evidences show that the conventional defense budget ϵ is insufficient under the threat of ϵ -bounded training-time perturbations.
Revised: Both theoretical and empirical evidences show that the conventional defense budget ϵ is insufficient under the threat of ϵ -bounded training-time perturbations.

Original: Our findings suggest that practitioners should consider a larger defense budget of no more than 2 ϵ (practically, about 1 .
Revised: Our findings suggest that practitioners should consider a larger defense budget of no more than 2 ϵ (practically, about 1 .

Original: 5 ϵ ∼ 1 .
Revised: 5 ϵ ∼ 1 .

Original: 75 ϵ ) to achieve a better ϵ -robustness.
Revised: 75 ϵ ) to achieve a better ϵ -robustness.

Original:  Threat Models
Revised:  Threat Models

Original: In this section, we formally introduce the threat model of stability attacks.
Revised: In this section, we formally introduce the threat model of stability attacks.

Original: We begin by revisiting the concepts of natural risk, adversarial risk, and delusive attacks.
Revised: We begin by revisiting the concepts of natural risk, adversarial risk, and delusive attacks.

Original: These concepts naturally give rise to our formulation of stability attacks.
Revised: These concepts naturally give rise to our formulation of stability attacks.

Original: 2.1 Preliminaries
Revised: 2.1 Preliminaries

Original: Setup.
Revised: Setup.

Original: We consider a classification task with input-label pairs ( x , y ) from an underlying distribution
Revised: We consider a classification task with input-label pairs ( x , y ) from an underlying distribution

Original: D over X × [ k ] .
Revised: D over X × [ k ] .

Original: The goal is to learn a (robust) classifier f : X → [ k ] that predicts a label y for a given input x .
Revised: The goal is to learn a (robust) classifier f : X → [ k ] that predicts a label y for a given input x .

Original: Natural training (NT).
Revised: Natural training (NT).

Original: Most learning algorithms aim to maximize the generalization performance on unperturbed examples, i.e., natural accuracy.
Revised: Most learning algorithms aim to maximize the generalization performance on unperturbed examples, i.e., natural accuracy.

Original: The goal is to minimize the natural risk defined as:
Revised: The goal is to minimize the natural risk defined as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Adversarial training (AT).
Revised: Adversarial training (AT).

Original: Since the risk of adversarial examples (a.k.a.
Revised: Since the risk of adversarial examples (a.k.a.

Original: evasion attacks) was found to be unexpectedly high [5, 60], it has become increasingly important to defend the learner against the worst-case perturbations [23, 39].
Revised: evasion attacks) was found to be unexpectedly high [5, 60], it has become increasingly important to defend the learner against the worst-case perturbations [23, 39].

Original: In this context, the goal is to train a model that has low adversarial risk given a defense budget ϵ :
Revised: In this context, the goal is to train a model that has low adversarial risk given a defense budget ϵ :

Original: [Equation]
Revised: [Equation]

Original: where we choose ∆ to be the set of ϵ -bounded perturbations, i.e., ∆ = { δ | ∥ δ ∥ ≤ ϵ } .
Revised: where we choose ∆ to be the set of ϵ -bounded perturbations, i.e., ∆ = { δ | ∥ δ ∥ ≤ ϵ } .

Original: This choice is the most common one in the context of adversarial examples [67].
Revised: This choice is the most common one in the context of adversarial examples [67].

Original: To simplify the notation, we refer to the robustness with respect to this set as ϵ -robustness.
Revised: To simplify the notation, we refer to the robustness with respect to this set as ϵ -robustness.

Original: It is worth noting that R adv ( f ) ≥ R nat ( f ) always holds for any f , and the equation holds when ϵ = 0 .
Revised: It is worth noting that R adv ( f ) ≥ R nat ( f ) always holds for any f , and the equation holds when ϵ = 0 .

Original: Delusive attacks.
Revised: Delusive attacks.

Original: Delusive attacks, which belong to training-time availability attacks, aim to prevent the learner from producing an accurate model by manipulating the training data “imperceptibly” [44].
Revised: Delusive attacks, which belong to training-time availability attacks, aim to prevent the learner from producing an accurate model by manipulating the training data “imperceptibly” [44].

Original: Concretely, the features of the training data can be perturbed, while the labels should remain correct [16, 42, 56, 28, 81, 15, 64, 18, 19].
Revised: Concretely, the features of the training data can be perturbed, while the labels should remain correct [16, 42, 56, 28, 81, 15, 64, 18, 19].

Original: This malicious task can be formalized into the following bi-level optimization problem:
Revised: This malicious task can be formalized into the following bi-level optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Here, the adversary aims to maximize the natural risk of the model f P (that is trained on the manipulated training set) by applying the generated perturbations P = { p i } ni =1 into the original training set T = { ( x i , y i ) } ni =1 .
Revised: Here, the adversary aims to maximize the natural risk of the model f P (that is trained on the manipulated training set) by applying the generated perturbations P = { p i } ni =1 into the original training set T = { ( x i , y i ) } ni =1 .

Original: The commonly used feasible region is S
Revised: The commonly used feasible region is S

Original: = {{ p i } ni =1 | ∥ p i ∥ ≤ ϵ } .
Revised: = {{ p i } ni =1 | ∥ p i ∥ ≤ ϵ } .

Original: Generally, solving Equation (3) is computationally prohibitive for neural networks [64, 18].
Revised: Generally, solving Equation (3) is computationally prohibitive for neural networks [64, 18].

Original: Thus, various heuristic methods are proposed to achieve the goal.
Revised: Thus, various heuristic methods are proposed to achieve the goal.

Original: Among them, a representative method is the hypocritical perturbation [63, 64], crafted as follows:
Revised: Among them, a representative method is the hypocritical perturbation [63, 64], crafted as follows:

Original: [Equation]
Revised: [Equation]

Original: where f craft is called the crafting model , pre-trained before generating poisons.
Revised: where f craft is called the crafting model , pre-trained before generating poisons.

Original: Tao et al.
Revised: Tao et al.

Original: [64] simply adopted a naturally trained classifier as the crafting model, while Huang et al.
Revised: [64] simply adopted a naturally trained classifier as the crafting model, while Huang et al.

Original: [28] proposed a min-min bi-level optimization process to pre-train the crafting model.
Revised: [28] proposed a min-min bi-level optimization process to pre-train the crafting model.

Original: Fu et al.
Revised: Fu et al.

Original: [19] further built their crafting model via a min-min-max three-level optimization process, and generated their poisons by replacing Equation (4) with a min-max bi-level objective.
Revised: [19] further built their crafting model via a min-min-max three-level optimization process, and generated their poisons by replacing Equation (4) with a min-max bi-level objective.

Original: Another representative method of delusive attacks is the adversarial perturbation , crafted by solving
Revised: Another representative method of delusive attacks is the adversarial perturbation , crafted by solving

Original: [Equation]
Revised: [Equation]

Original: Tao et al.
Revised: Tao et al.

Original: [64] and Fowl et al.
Revised: [64] and Fowl et al.

Original: [18] both found that applying the adversarial perturbation to the training data is very effective at compromising naturally trained models.
Revised: [18] both found that applying the adversarial perturbation to the training data is very effective at compromising naturally trained models.

Original: However, adversarial training has proven to be promising in defending against various delusive attacks [64].
Revised: However, adversarial training has proven to be promising in defending against various delusive attacks [64].

Original: 2.2 Stability Attacks
Revised: 2.2 Stability Attacks

Original: In contrast to delusive attacks that aim at increasing the natural risk, stability attacks attempt to maximize the adversarial risk of the learner by slightly perturbing the training data:
Revised: In contrast to delusive attacks that aim at increasing the natural risk, stability attacks attempt to maximize the adversarial risk of the learner by slightly perturbing the training data:

Original: [Equation]
Revised: [Equation]

Original: where f P denotes the victim model, which is naturally or adversarially trained on the perturbed data.
Revised: where f P denotes the victim model, which is naturally or adversarially trained on the perturbed data.

Original: In other words, stability attacks seek to hinder the robust availability of the training data.
Revised: In other words, stability attacks seek to hinder the robust availability of the training data.

Original: Tableshows the comparisons among different threat models.
Revised: Tableshows the comparisons among different threat models.

Original: The goal of stability attacks can be immediately achieved for naturally trained models, since they have already incurred high adversarial risk, even if the training data is clean [60].
Revised: The goal of stability attacks can be immediately achieved for naturally trained models, since they have already incurred high adversarial risk, even if the training data is clean [60].

Original: To ease the problem of high adversarial risk, adversarial training has been widely used to improve model’s adversarial robustness [24, 12].
Revised: To ease the problem of high adversarial risk, adversarial training has been widely used to improve model’s adversarial robustness [24, 12].

Original: Hence, the main goal of stability attacks becomes to compromise the test robustness of adversarially trained models.
Revised: Hence, the main goal of stability attacks becomes to compromise the test robustness of adversarially trained models.

Original: Note that Equation (6) is a multi-level optimization problem that is not easy to solve, our next question is how to conduct effective stability attacks against adversarial training.
Revised: Note that Equation (6) is a multi-level optimization problem that is not easy to solve, our next question is how to conduct effective stability attacks against adversarial training.

Original: In the following sections, we introduce an effective stability attack method and analyze the cost of resisting it.
Revised: In the following sections, we introduce an effective stability attack method and analyze the cost of resisting it.

Original: Remark 1.
Revised: Remark 1.

Original: This work focuses on adding bounded pertubrations as small as possible.
Revised: This work focuses on adding bounded pertubrations as small as possible.

Original: We mostly assume that the adversary has full control of training data (instead of changing a few) by following previous works [16, 28, 64, 17–19].
Revised: We mostly assume that the adversary has full control of training data (instead of changing a few) by following previous works [16, 28, 64, 17–19].

Original: This is a realistic assumption [16, 18].
Revised: This is a realistic assumption [16, 18].

Original: For instance, in some applications an organization may agree to release some internal data for peer assessment, while preventing competitors from easily building a model with high test robustness; this can be achieved by perturbing the entire dataset via stability attacks before releasing.
Revised: For instance, in some applications an organization may agree to release some internal data for peer assessment, while preventing competitors from easily building a model with high test robustness; this can be achieved by perturbing the entire dataset via stability attacks before releasing.

Original: Moreover, this assumption enables a worst-case analysis of the robustness of adversarial training, which may facilitate important theoretical implications.
Revised: Moreover, this assumption enables a worst-case analysis of the robustness of adversarial training, which may facilitate important theoretical implications.

Original:  How to Manipulate Adversarial Training
Revised:  How to Manipulate Adversarial Training

Original: Previous work suggests that adversarial training could defend against both evasion attacks and delusive attacks [39, 64].
Revised: Previous work suggests that adversarial training could defend against both evasion attacks and delusive attacks [39, 64].

Original: However, in this paper, we show that adversarial training using a conventional defense budget ϵ may not be sufficient to provide ϵ -robustness when confronted with stability attacks.
Revised: However, in this paper, we show that adversarial training using a conventional defense budget ϵ may not be sufficient to provide ϵ -robustness when confronted with stability attacks.

Original: In particular, we present a simple theoretical model where the conventional defense scheme provably fails when the training data is hypocritically perturbed.
Revised: In particular, we present a simple theoretical model where the conventional defense scheme provably fails when the training data is hypocritically perturbed.

Original: The binary classification task.
Revised: The binary classification task.

Original: The data model is largely based on the setting proposed by Tsipras et al.
Revised: The data model is largely based on the setting proposed by Tsipras et al.

Original: [67], which draws a distinction between robust features and non-robust features .
Revised: [67], which draws a distinction between robust features and non-robust features .

Original: Specifically, it consists of input-label pairs ( x , y ) sampled from a Gaussian mixture distribution D as follows:
Revised: Specifically, it consists of input-label pairs ( x , y ) sampled from a Gaussian mixture distribution D as follows:

Original: [Equation]
Revised: [Equation]

Original: where η is much smaller than 1 (i.e., 0 < η ≪ 1 ).
Revised: where η is much smaller than 1 (i.e., 0 < η ≪ 1 ).

Original: Hence, samples from D consist of a robust feature ( x 1 ) that is strongly correlated with the label, and d non-robust features ( x 2 , . .
Revised: Hence, samples from D consist of a robust feature ( x 1 ) that is strongly correlated with the label, and d non-robust features ( x 2 , . .

Original: , x d +1 ) that are very weakly correlated with it.
Revised: , x d +1 ) that are very weakly correlated with it.

Original: Typically, an adversary can manipulate a large number of non-robust features - e.g. d = Θ(1 /η 2 ) will suffice.
Revised: Typically, an adversary can manipulate a large number of non-robust features - e.g. d = Θ(1 /η 2 ) will suffice.

Original: Before introducing the way to hinder robust availability, we briefly illustrate the success of adversarial training when the training data is unperturbed.
Revised: Before introducing the way to hinder robust availability, we briefly illustrate the success of adversarial training when the training data is unperturbed.

Original: Natural and robust classifiers.
Revised: Natural and robust classifiers.

Original: For standard classification, we consider a natural classifier:
Revised: For standard classification, we consider a natural classifier:

Original: [Equation]
Revised: [Equation]

Original: which is a minimizer of the natural risk (1) with 0-1 loss on the data (7), i.e., the Bayes optimal classifier.
Revised: which is a minimizer of the natural risk (1) with 0-1 loss on the data (7), i.e., the Bayes optimal classifier.

Original: However, in the adversarial setting, this natural classifier is quite brittle.
Revised: However, in the adversarial setting, this natural classifier is quite brittle.

Original: Thus, it is imperative to obtain a robust classifier:
Revised: Thus, it is imperative to obtain a robust classifier:

Original: [Equation]
Revised: [Equation]

Original: which relies only on the robust feature x 1 .
Revised: which relies only on the robust feature x 1 .

Original: Illustration of adversarial accuracy.
Revised: Illustration of adversarial accuracy.

Original: In the adversarial setting, an adversary that is only allowed to perturb each feature by a moderate ϵ can effectively subvert the natural classifier Tsipras et al.
Revised: In the adversarial setting, an adversary that is only allowed to perturb each feature by a moderate ϵ can effectively subvert the natural classifier Tsipras et al.

Original: In particular, if ϵ = 2 η , an adversary can essentially force each non-robust feature to be anti -correlated with the correct label.
Revised: In particular, if ϵ = 2 η , an adversary can essentially force each non-robust feature to be anti -correlated with the correct label.

Original: The following proposition, proved in Appendix C.1, gives the adversarial accuracies of the natural classifier f nat (8) and the robust classifier f rob (9).
Revised: The following proposition, proved in Appendix C.1, gives the adversarial accuracies of the natural classifier f nat (8) and the robust classifier f rob (9).

Original: Proposition 1.
Revised: Proposition 1.

Original: Let ϵ = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of a classifier correctly predicting y on the data (7) under ℓ ∞ perturbations.
Revised: Let ϵ = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of a classifier correctly predicting y on the data (7) under ℓ ∞ perturbations.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Proposition 1 implies that the adversarial accuracy of the natural classifier is < 50% when d ≥ 1 /η 2 .
Revised: Proposition 1 implies that the adversarial accuracy of the natural classifier is < 50% when d ≥ 1 /η 2 .

Original: Even worse, when σ ≤ 1 / 3 and d ≥ 3 /η 2 , the adversarial accuracy of the natural classifier (8) is always lower than 1% .
Revised: Even worse, when σ ≤ 1 / 3 and d ≥ 3 /η 2 , the adversarial accuracy of the natural classifier (8) is always lower than 1% .

Original: In contrast, the robust classifier (9) yields a much higher adversarial accuracy (always > 50% ); when σ ≤ (1 − 2 η ) / 3 , its adversarial accuracy will be higher than 99% .
Revised: In contrast, the robust classifier (9) yields a much higher adversarial accuracy (always > 50% ); when σ ≤ (1 − 2 η ) / 3 , its adversarial accuracy will be higher than 99% .

Original: 3.1 Hypocritical Features Are Harmful
Revised: 3.1 Hypocritical Features Are Harmful

Original: The results above reveal the advantages of robust classifiers over natural classifiers.
Revised: The results above reveal the advantages of robust classifiers over natural classifiers.

Original: Note that such a robust classifier can be obtained by adversarial training on the original data (7).
Revised: Note that such a robust classifier can be obtained by adversarial training on the original data (7).

Original: However, this defense effect may not hold when the adversary is allowed to perturb the training data.
Revised: However, this defense effect may not hold when the adversary is allowed to perturb the training data.

Original: We show this by analyzing two representative perturbations: the adversarial perturbation [64, 18] and the hypocritical perturbation [63, 64].
Revised: We show this by analyzing two representative perturbations: the adversarial perturbation [64, 18] and the hypocritical perturbation [63, 64].

Original: When applied into the training data, both perturbations are effective as delusive attacks for naturally trained models.
Revised: When applied into the training data, both perturbations are effective as delusive attacks for naturally trained models.

Original: In the following, we show that the former is harmless: adversarial training using a defense budget ϵ on the adversarially perturbed data can still provide test robustness.
Revised: In the following, we show that the former is harmless: adversarial training using a defense budget ϵ on the adversarially perturbed data can still provide test robustness.

Original: In contrast, the latter is harmful: we find that the same defense budget can only produce non-robust classifiers when the training data is hypocritically perturbed.
Revised: In contrast, the latter is harmful: we find that the same defense budget can only produce non-robust classifiers when the training data is hypocritically perturbed.

Original: A harmless case.
Revised: A harmless case.

Original: Consider an adversary who is capable of perturbing the training data by an attack budget ϵ .
Revised: Consider an adversary who is capable of perturbing the training data by an attack budget ϵ .

Original: The adversary may choose to shift each feature towards − y .
Revised: The adversary may choose to shift each feature towards − y .

Original: Hence, the learner would see input-label pairs ( x , y ) sampled i.i.d.
Revised: Hence, the learner would see input-label pairs ( x , y ) sampled i.i.d.

Original: from a training distribution T adv as follows:
Revised: from a training distribution T adv as follows:

Original: [Equation]
Revised: [Equation]

Original: where each feature of the samples from T adv is adversarially perturbed by a moderate ϵ .
Revised: where each feature of the samples from T adv is adversarially perturbed by a moderate ϵ .

Original: While these samples are deviate significantly from the original distribution D (7), adversarial training on them using a defense budget ϵ is still able to neutralize the non-robust features.
Revised: While these samples are deviate significantly from the original distribution D (7), adversarial training on them using a defense budget ϵ is still able to neutralize the non-robust features.

Original: Formally, in Appendix C. we prove the following theorem.
Revised: Formally, in Appendix C. we prove the following theorem.

Original: Theorem 1 (Adversarial perturbation is harmless) .
Revised: Theorem 1 (Adversarial perturbation is harmless) .

Original: Assume that the adversarial perturbation in the training data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .
Revised: Assume that the adversarial perturbation in the training data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .

Original: Then, the optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on T adv with a defense budget ϵ is equivalent to the robust classifier (9).
Revised: Then, the optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on T adv with a defense budget ϵ is equivalent to the robust classifier (9).

Original: This theorem indicates that the adversarial perturbation is harmless: ϵ -robustness can still be obtained by adversarial training on such perturbed training data.
Revised: This theorem indicates that the adversarial perturbation is harmless: ϵ -robustness can still be obtained by adversarial training on such perturbed training data.

Original: A harmful case.
Revised: A harmful case.

Original: However, this defense effect can be completely broken by the hypocritical perturbation.
Revised: However, this defense effect can be completely broken by the hypocritical perturbation.

Original: That is, the adversary can instead shift each feature towards y .
Revised: That is, the adversary can instead shift each feature towards y .

Original: Hence, the learner would see input-label pairs ( x , y ) sampled i.i.d.
Revised: Hence, the learner would see input-label pairs ( x , y ) sampled i.i.d.

Original: from a training distribution T hyp as follows 2 :
Revised: from a training distribution T hyp as follows 2 :

Original: [Equation]
Revised: [Equation]

Original: where each feature of the samples from T hyp is reinforced by a magnitude of ϵ .
Revised: where each feature of the samples from T hyp is reinforced by a magnitude of ϵ .

Original: While these samples become more separable, adversarial training on them using the same defense budget will fail to neutralize the hypocritically perturbed features.
Revised: While these samples become more separable, adversarial training on them using the same defense budget will fail to neutralize the hypocritically perturbed features.

Original: Consequently, the resulting classifiers will inevitably have low adversarial accuracy.
Revised: Consequently, the resulting classifiers will inevitably have low adversarial accuracy.

Original: We make this formal in the following theorem proved in Appendix C.3.
Revised: We make this formal in the following theorem proved in Appendix C.3.

Original: Theorem 2 (Hypocritical perturbation is harmful) .
Revised: Theorem 2 (Hypocritical perturbation is harmful) .

Original: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ is equivalent to the natural classifier (8).
Revised: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ is equivalent to the natural classifier (8).

Original: This theorem implies that the conventional defense scheme can only produce non-robust classifiers, whose adversarial accuracy is as low as that of the natural classifier (8).
Revised: This theorem implies that the conventional defense scheme can only produce non-robust classifiers, whose adversarial accuracy is as low as that of the natural classifier (8).

Original: That is saying, if ϵ = 2 η , σ ≤ 1 / 3 and d ≥ 3 /η 2 , the classifiers cannot get adversarial accuracy better than 1% .
Revised: That is saying, if ϵ = 2 η , σ ≤ 1 / 3 and d ≥ 3 /η 2 , the classifiers cannot get adversarial accuracy better than 1% .

Original: Implications.
Revised: Implications.

Original: As it turns out, the seemingly beneficial features in T hyp (11) are actually hypocritical.
Revised: As it turns out, the seemingly beneficial features in T hyp (11) are actually hypocritical.

Original: Therefore, the adversary is highly motivated to hide such hypocritical features in the training data, intending to cajole an innocent learner into relying on the non-robust features.
Revised: Therefore, the adversary is highly motivated to hide such hypocritical features in the training data, intending to cajole an innocent learner into relying on the non-robust features.

Original: Intriguingly, we notice that the natural classifier (8) (i.e., the crafting model used to derive the distribution T hyp ) actually has η -robustness.
Revised: Intriguingly, we notice that the natural classifier (8) (i.e., the crafting model used to derive the distribution T hyp ) actually has η -robustness.

Original: This is essentially because the non-robust features in the data (7) can resist small-magnitude perturbations by design.
Revised: This is essentially because the non-robust features in the data (7) can resist small-magnitude perturbations by design.

Original: This motivates us to use “slightly robust” classifiers as the crafting model in practice.
Revised: This motivates us to use “slightly robust” classifiers as the crafting model in practice.

Original: Indeed, our experimental results show that training the crafting model with 0 .
Revised: Indeed, our experimental results show that training the crafting model with 0 .

Original: 25 ϵ -robustness performs the best for conducting stability attacks.
Revised: 25 ϵ -robustness performs the best for conducting stability attacks.

Original: This is different from the previous works [64, 18] that use naturally trained models as the crafting model for poisoning.
Revised: This is different from the previous works [64, 18] that use naturally trained models as the crafting model for poisoning.

Original:  The Necessity of Large Defense Budget
Revised:  The Necessity of Large Defense Budget

Original: We have shown that the hypocritical perturbation is harmful to the conventional adversarial training scheme.
Revised: We have shown that the hypocritical perturbation is harmful to the conventional adversarial training scheme.

Original: Fortunately, it is possible to strengthen the defense by using a larger defense budget, while the crux of the matter is how large the budget is needed.
Revised: Fortunately, it is possible to strengthen the defense by using a larger defense budget, while the crux of the matter is how large the budget is needed.

Original: We find that the minimum value of the defense budget for a successful defense depends on the specific data distribution.
Revised: We find that the minimum value of the defense budget for a successful defense depends on the specific data distribution.

Original: Let us first consider the hypocritical data in T hyp (11).
Revised: Let us first consider the hypocritical data in T hyp (11).

Original: In this case, we show that a larger defense budget is necessary in the following theorem proved in Appendix C.4.
Revised: In this case, we show that a larger defense budget is necessary in the following theorem proved in Appendix C.4.

Original: Theorem 3 ( ϵ + η is necessary) .
Revised: Theorem 3 ( ϵ + η is necessary) .

Original: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ + η is equivalent to the robust classifier (9).
Revised: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ + η is equivalent to the robust classifier (9).

Original: Moreover, any defense budget lower than ϵ + η will yield classifiers that still rely on all the non-robust features.
Revised: Moreover, any defense budget lower than ϵ + η will yield classifiers that still rely on all the non-robust features.

Original: This theorem implies that, in the case of the mixture Gaussian distribution under the threat of ϵbounded hypocritical perturbations, the learner needs a slightly larger defense budget ϵ + η to ensure ϵ -robustness.
Revised: This theorem implies that, in the case of the mixture Gaussian distribution under the threat of ϵbounded hypocritical perturbations, the learner needs a slightly larger defense budget ϵ + η to ensure ϵ -robustness.

Original: While it is challenging to analyze the minimum value of the defense budget in the general case, the following theorem provides an upper bound of the budget.
Revised: While it is challenging to analyze the minimum value of the defense budget in the general case, the following theorem provides an upper bound of the budget.

Original: Theorem 4 (General case) .
Revised: Theorem 4 (General case) .

Original: For any data distribution and any adversary with an attack budget ϵ , training models to minimize the adversarial risk with a defense budget 2 ϵ on the perturbed data is sufficient to ensure ϵ -robustness.
Revised: For any data distribution and any adversary with an attack budget ϵ , training models to minimize the adversarial risk with a defense budget 2 ϵ on the perturbed data is sufficient to ensure ϵ -robustness.

Original: The proof of Theorem 4 is deferred in Appendix C.5.
Revised: The proof of Theorem 4 is deferred in Appendix C.5.

Original: It implies that a defense budget twice to the attack budget should be safe enough under the threat of stability attacks.
Revised: It implies that a defense budget twice to the attack budget should be safe enough under the threat of stability attacks.

Original: Theorem 3 also suggests that the minimum budget might be much smaller than 2 ϵ , and it depends on the specific attack methods and data distributions.
Revised: Theorem 3 also suggests that the minimum budget might be much smaller than 2 ϵ , and it depends on the specific attack methods and data distributions.

Original: In the following section, we empirically search for an appropriate defense budget on real-world datasets.
Revised: In the following section, we empirically search for an appropriate defense budget on real-world datasets.

Original:  Experiments
Revised:  Experiments

Original: In this section, we conduct comprehensive experiments to demonstrate the effectiveness of the hypocritical perturbation as stability attacks on popular benchmark datasets and the necessity of an adaptive defense for better robustness.
Revised: In this section, we conduct comprehensive experiments to demonstrate the effectiveness of the hypocritical perturbation as stability attacks on popular benchmark datasets and the necessity of an adaptive defense for better robustness.

Original: We conduct stability attacks by applying hypocritical perturbations into the training set.
Revised: We conduct stability attacks by applying hypocritical perturbations into the training set.

Original: We focus on an ℓ ∞ adversary with an attack budget ϵ a = 8 / 255 by following [28, 81, 64, 18].
Revised: We focus on an ℓ ∞ adversary with an attack budget ϵ a = 8 / 255 by following [28, 81, 64, 18].

Original: Our crafting model is adversarially trained with a crafting budget ϵ c = 2 / 255 for 10 epochs before generating perturbations.
Revised: Our crafting model is adversarially trained with a crafting budget ϵ c = 2 / 255 for 10 epochs before generating perturbations.

Original: Unless otherwise specified, we use ResNet-18 [26] as the default architecture for both the crafting model and the learning model.
Revised: Unless otherwise specified, we use ResNet-18 [26] as the default architecture for both the crafting model and the learning model.

Original: For adversarial training, we mainly follow the settings in
Revised: For adversarial training, we mainly follow the settings in

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: previous studies [83, 70, 50].
Revised: previous studies [83, 70, 50].

Original: By convention, the defense budget is equal to the attack budget,
Revised: By convention, the defense budget is equal to the attack budget,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., ϵ d = 8 / 255 .
Revised: e., ϵ d = 8 / 255 .

Original: More details on experimental settings are provided in Appendix D.
Revised: More details on experimental settings are provided in Appendix D.

Original: 5.1 Benchmarking (Non-)Robustness
Revised: 5.1 Benchmarking (Non-)Robustness

Original: Attack evaluation.
Revised: Attack evaluation.

Original: We compare our crafted hypocritical perturbation to existing methods, which were originally proposed as delusive attacks, including DeepConfuse (which builds an adversarial auto-encoder to generate their perturbations) [16], Unlearnable Examples (which use a min-min bi-level optimization process to pre-train their crafting model) [28], NTGA (which adopts neural tangent kernels as its crafting model) [81]
Revised: We compare our crafted hypocritical perturbation to existing methods, which were originally proposed as delusive attacks, including DeepConfuse (which builds an adversarial auto-encoder to generate their perturbations) [16], Unlearnable Examples (which use a min-min bi-level optimization process to pre-train their crafting model) [28], NTGA (which adopts neural tangent kernels as its crafting model) [81]

Original: [Table]
Revised: [Table]

Original: and Adversarial Poisoning (whose crafting model is simply a naturally trained classifier)
Revised: and Adversarial Poisoning (whose crafting model is simply a naturally trained classifier)

Original: It is noteworthy that none of these previous works evaluated the test robustness of their poisoned models.
Revised: It is noteworthy that none of these previous works evaluated the test robustness of their poisoned models.

Original: Results using ResNet-18 on CIFAR-10 are summarized in Table 2.
Revised: Results using ResNet-18 on CIFAR-10 are summarized in Table 2.

Original: “Natural” denotes the accuracy on natural test data.
Revised: “Natural” denotes the accuracy on natural test data.

Original: Various test-time adversarial attacks are used to evaluate test robustness, including
Revised: Various test-time adversarial attacks are used to evaluate test robustness, including

Original: FGSM, PGD-20/100, CW ∞ ( ℓ ∞ version of CW loss [9] optimized by PGD-100), and AutoAttack (a reliable evaluation metric via an ensemble of diverse attacks [11]).
Revised: FGSM, PGD-20/100, CW ∞ ( ℓ ∞ version of CW loss [9] optimized by PGD-100), and AutoAttack (a reliable evaluation metric via an ensemble of diverse attacks [11]).

Original: We observe that the hypocritical perturbation widely outperforms previous training-time perturbations in degrading the test robustness of PGD-AT [39].
Revised: We observe that the hypocritical perturbation widely outperforms previous training-time perturbations in degrading the test robustness of PGD-AT [39].

Original: This demonstrates that stability attacks are indeed harmful to the conventional defense scheme.
Revised: This demonstrates that stability attacks are indeed harmful to the conventional defense scheme.

Original: We note that our method increases the natural accuracy.
Revised: We note that our method increases the natural accuracy.

Original: This is reasonable, since our analysis in Section 3.1 has implied that the hypocritical perturbation can increase model reliance on the non-robust features, which are predictive but brittle [29].
Revised: This is reasonable, since our analysis in Section 3.1 has implied that the hypocritical perturbation can increase model reliance on the non-robust features, which are predictive but brittle [29].

Original: Moreover, we evaluate the hypocritical perturbation on other benchmark datasets including SVHN,
Revised: Moreover, we evaluate the hypocritical perturbation on other benchmark datasets including SVHN,

Original: CIFAR-100, and Tiny-ImageNet.
Revised: CIFAR-100, and Tiny-ImageNet.

Original: Both the crafting model and the victim model use the ResNet-
Revised: Both the crafting model and the victim model use the ResNet-

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: architecture.
Revised: architecture.

Original: Results are summarized in Table 3.
Revised: Results are summarized in Table 3.

Original: “Hyp.” denotes the hypocritical perturbation generated by our crafting model.
Revised: “Hyp.” denotes the hypocritical perturbation generated by our crafting model.

Original: As a comparison, we also evaluate the adversarial perturbation generated using the same crafting model (denoted as “Adv.”).
Revised: As a comparison, we also evaluate the adversarial perturbation generated using the same crafting model (denoted as “Adv.”).

Original: Again, the results show that the hypocritical perturbations are more threatening than the adversarial perturbations to standard adversarial training.
Revised: Again, the results show that the hypocritical perturbations are more threatening than the adversarial perturbations to standard adversarial training.

Original: This phenomenon is consistent with our analytical results in Section 3.
Revised: This phenomenon is consistent with our analytical results in Section 3.

Original: [Table]
Revised: [Table]

Original: Besides
Revised: Besides

Original: [Table]
Revised: [Table]

Original: we find that the hypocritical perturbation can transfer well from ResNet-18 to other architectures, successfully degrading the test robustness of a wide variety of popular architectures including
Revised: we find that the hypocritical perturbation can transfer well from ResNet-18 to other architectures, successfully degrading the test robustness of a wide variety of popular architectures including

Original: VGG-16 [57], GoogLeNet [61], DenseNet-121 [27], MobileNetV2 [52]
Revised: VGG-16 [57], GoogLeNet [61], DenseNet-121 [27], MobileNetV2 [52]

Original: [Table]
Revised: [Table]

Original: and WideResNet-28-10 [82], as shown in Table 4.
Revised: and WideResNet-28-10 [82], as shown in Table 4.

Original: Note that this is a completely black-box setting where the attacker has no knowledge of the victim model’s initialization, architecture, learning rate scheduler, etc.
Revised: Note that this is a completely black-box setting where the attacker has no knowledge of the victim model’s initialization, architecture, learning rate scheduler, etc.

Original: Adaptive defense.
Revised: Adaptive defense.

Original: To prevent the harm of stability attacks, our analysis in Section 4 suggests that a larger defense budget would be helpful.
Revised: To prevent the harm of stability attacks, our analysis in Section 4 suggests that a larger defense budget would be helpful.

Original: We find that this is indeed the case on CIFAR-10.
Revised: We find that this is indeed the case on CIFAR-10.

Original: As shown in Table 5, a large defense budget ϵ d = 14 / 255 for PGD-AT performs significantly better than the conventional defense budget ϵ d = 8 / 255 .
Revised: As shown in Table 5, a large defense budget ϵ d = 14 / 255 for PGD-AT performs significantly better than the conventional defense budget ϵ d = 8 / 255 .

Original: We also combine several data augmentations with
Revised: We also combine several data augmentations with

Original: PGD-AT as defenses by following Fowl et al.
Revised: PGD-AT as defenses by following Fowl et al.

Original: The results show that they are beneficial, while their improvements are inferior to PGD-AT with ϵ d = 14 / 255 .
Revised: The results show that they are beneficial, while their improvements are inferior to PGD-AT with ϵ d = 14 / 255 .

Original: In addition, we adopt other adversarial training variants including TRADES [83] and MART [70] to defend against the hypocritical perturbation, and find that they achieve comparable defense effects with large defense budgets.
Revised: In addition, we adopt other adversarial training variants including TRADES [83] and MART [70] to defend against the hypocritical perturbation, and find that they achieve comparable defense effects with large defense budgets.

Original: Finally, we note that the adaptive defense has several limitations: i) robust accuracy is improved at the cost of natural accuracy; ii) finding an appropriate defense budget is time-consuming for adversarial training; iii) adversarial training with large budgets may lead to learning obstacles such as inherent large sample complexity [53].
Revised: Finally, we note that the adaptive defense has several limitations: i) robust accuracy is improved at the cost of natural accuracy; ii) finding an appropriate defense budget is time-consuming for adversarial training; iii) adversarial training with large budgets may lead to learning obstacles such as inherent large sample complexity [53].

Original: We leave the detailed study of these questions as future work.
Revised: We leave the detailed study of these questions as future work.

Original: 5.2 Ablation Studies
Revised: 5.2 Ablation Studies

Original: In this part, we conduct a set of experiments to provide an empirical understanding of the proposed attack.
Revised: In this part, we conduct a set of experiments to provide an empirical understanding of the proposed attack.

Original: We train ResNet-18 using PGD-AT on CIFAR-10 by following the same settings described in Appendix D unless otherwise specified.
Revised: We train ResNet-18 using PGD-AT on CIFAR-10 by following the same settings described in Appendix D unless otherwise specified.

Original: Analysis on the crafting method.
Revised: Analysis on the crafting method.

Original: Different from previous work, we use “slightly robust” classifiers as our crafting model.
Revised: Different from previous work, we use “slightly robust” classifiers as our crafting model.

Original: Figure 2(a) shows that this technique greatly improves the potency of the hypocritical perturbation, where the crafting budget ϵ c = 2 / 255 performs best in degrading test robustness.
Revised: Figure 2(a) shows that this technique greatly improves the potency of the hypocritical perturbation, where the crafting budget ϵ c = 2 / 255 performs best in degrading test robustness.

Original: We also observe that training the crafting model for 10 ∼ 80 epochs works well in Figure 2(b), and that optimizing the crafted perturbations over 100 steps performs well in Figure 2(c).
Revised: We also observe that training the crafting model for 10 ∼ 80 epochs works well in Figure 2(b), and that optimizing the crafted perturbations over 100 steps performs well in Figure 2(c).

Original: Finally, we note that Fowl et al.
Revised: Finally, we note that Fowl et al.

Original: [18] also tried to use adversarially trained models as the crafting model, but they failed to produce an effective attack in this way.
Revised: [18] also tried to use adversarially trained models as the crafting model, but they failed to produce an effective attack in this way.

Original: This is mainly because they adopted adversarial perturbations as poisons, which, as we observed, are inferior in degrading test performance.
Revised: This is mainly because they adopted adversarial perturbations as poisons, which, as we observed, are inferior in degrading test performance.

Original: Ablation on defense budget.
Revised: Ablation on defense budget.

Original: As discussed in Section 4, we are motivated to find the appropriate defense budget ϵ d in the range [ ϵ ∼ 2 ϵ ] .
Revised: As discussed in Section 4, we are motivated to find the appropriate defense budget ϵ d in the range [ ϵ ∼ 2 ϵ ] .

Original: Figure 2(d) shows that the optimal defense budgets against the hypocritical perturbation are 14 / 255 , 12 / 255 , and 14 / 255 for PGD-AT, TRADES, and MART, respectively.
Revised: Figure 2(d) shows that the optimal defense budgets against the hypocritical perturbation are 14 / 255 , 12 / 255 , and 14 / 255 for PGD-AT, TRADES, and MART, respectively.

Original: We also observe that all these adversarial training variants are inferior when using the conventional defense budget 8 / 255 .
Revised: We also observe that all these adversarial training variants are inferior when using the conventional defense budget 8 / 255 .

Original: Less data.
Revised: Less data.

Original: We follow Fowl et al.
Revised: We follow Fowl et al.

Original: [18] to test the effectiveness of attacks by varying the proportion of clean data and perturbed data.
Revised: [18] to test the effectiveness of attacks by varying the proportion of clean data and perturbed data.

Original: Attacks are then considered effective if they cannot significantly increase performance over training on the clean subset alone.
Revised: Attacks are then considered effective if they cannot significantly increase performance over training on the clean subset alone.

Original: As shown in Table 6, the proposed attack often degrades the test robustness below what one would achieve using full clean dataset.
Revised: As shown in Table 6, the proposed attack often degrades the test robustness below what one would achieve using full clean dataset.

Original: More importantly, the hypocritical perturbations are consistently more harmful than the adversarial perturbations.
Revised: More importantly, the hypocritical perturbations are consistently more harmful than the adversarial perturbations.

Original: This again verifies the superiority of hypocritical pertubrations as stability attacks.
Revised: This again verifies the superiority of hypocritical pertubrations as stability attacks.

Original: [Table]
Revised: [Table]

Original: Effect on natural training.
Revised: Effect on natural training.

Original: As a sanity check, we include the test accuracy of naturally trained models on CIFAR-10 in Table 7.
Revised: As a sanity check, we include the test accuracy of naturally trained models on CIFAR-10 in Table 7.

Original: It shows that without adversarial training, the test robustness of the models becomes very poor—all models only have 0% accuracy under PGD-20 attack.
Revised: It shows that without adversarial training, the test robustness of the models becomes very poor—all models only have 0% accuracy under PGD-20 attack.

Original: Thus, the goal of stability attacks is immediately achieved.
Revised: Thus, the goal of stability attacks is immediately achieved.

Original: On the other hand, We find that our method degrades the test accuracy from 94 .
Revised: On the other hand, We find that our method degrades the test accuracy from 94 .

Original: 23% to 75 .
Revised: 23% to 75 .

Original: 92% , though this is not the main focus of this work.
Revised: 92% , though this is not the main focus of this work.

Original: We also observe that Adversarial Poisoning [18] is the most effective method in degrading the test accuracy of naturally trained models.
Revised: We also observe that Adversarial Poisoning [18] is the most effective method in degrading the test accuracy of naturally trained models.

Original: This observation is consistent with Fowl et al.
Revised: This observation is consistent with Fowl et al.

Original:  Related Work
Revised:  Related Work

Original: Adversarial training.
Revised: Adversarial training.

Original: The presence of non-robust features has been demonstrated on popular benchmark datasets [29, 31], which naturally leads to model vulnerability to adversarial examples [67, 58].
Revised: The presence of non-robust features has been demonstrated on popular benchmark datasets [29, 31], which naturally leads to model vulnerability to adversarial examples [67, 58].

Original: To improve test robustness against adversarial examples, adversarial training methods have been developed [23, 39, 73, 83, 65, 46, 76, 86, 62, 69].
Revised: To improve test robustness against adversarial examples, adversarial training methods have been developed [23, 39, 73, 83, 65, 46, 76, 86, 62, 69].

Original: Usually, adversarial training using a defense budget ϵ is expected to improve model robustness against ϵ -bounded adversarial examples.
Revised: Usually, adversarial training using a defense budget ϵ is expected to improve model robustness against ϵ -bounded adversarial examples.

Original: Thus, to break this defense, a direct way is to enlarge the typical ϵ -ball used to constrain the attack; however, this may risk changing the true label [8, 66].
Revised: Thus, to break this defense, a direct way is to enlarge the typical ϵ -ball used to constrain the attack; however, this may risk changing the true label [8, 66].

Original: In this work, we aim to show that it is possible to achieve this by slightly perturbing the training data without enlarging the ϵ -ball.
Revised: In this work, we aim to show that it is possible to achieve this by slightly perturbing the training data without enlarging the ϵ -ball.

Original: Data poisoning.
Revised: Data poisoning.

Original: Data poisoning attacks, which manipulate training data to cause the resulting models to fail during inference [3], can be divided into availability attacks (to degrade overall test performance)
Revised: Data poisoning attacks, which manipulate training data to cause the resulting models to fail during inference [3], can be divided into availability attacks (to degrade overall test performance)

Original: [4, 77, 41, 47, 18] and integrity attacks (to cause specific misclassifications) [32, 10, 55, 87, 21, 54].
Revised: [4, 77, 41, 47, 18] and integrity attacks (to cause specific misclassifications) [32, 10, 55, 87, 21, 54].

Original: While the stability attacks studied in this work may be reminiscent of backdoor attacks [10], we note that they share several key differences.
Revised: While the stability attacks studied in this work may be reminiscent of backdoor attacks [10], we note that they share several key differences.

Original: First, stability attacks aim to hinder adversarial training with well-defined ϵ -robustness, while backdoor attacks mainly focus on embedding malicious behaviors (that can be invoked by pre-specified triggers) into naturally trained models [22, 51, 68].
Revised: First, stability attacks aim to hinder adversarial training with well-defined ϵ -robustness, while backdoor attacks mainly focus on embedding malicious behaviors (that can be invoked by pre-specified triggers) into naturally trained models [22, 51, 68].

Original: Second, stability attacks only perturb the inputs slightly, while most works on backdoor attacks require mislabeling [25, 37, 45, 36, 74].
Revised: Second, stability attacks only perturb the inputs slightly, while most works on backdoor attacks require mislabeling [25, 37, 45, 36, 74].

Original: Thus, backdoor defenses [7, 75, 35] might not be directly applied to resist stability attacks.
Revised: Thus, backdoor defenses [7, 75, 35] might not be directly applied to resist stability attacks.

Original: Additional related works are discussed in Appendix A.
Revised: Additional related works are discussed in Appendix A.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we establish a framework to study the robustness of adversarial training against stability attacks.
Revised: In this work, we establish a framework to study the robustness of adversarial training against stability attacks.

Original: We unveil the threat of stability attacks—small hypocritical perturbations applied into the training data suffice to hinder conventional adversarial training.
Revised: We unveil the threat of stability attacks—small hypocritical perturbations applied into the training data suffice to hinder conventional adversarial training.

Original: The conventional defense budget ϵ is insufficient under the threat.
Revised: The conventional defense budget ϵ is insufficient under the threat.

Original: To resist it, we suggest a larger defense budget of no more than 2 ϵ .
Revised: To resist it, we suggest a larger defense budget of no more than 2 ϵ .

Original: Our theoretical analysis explains why hypocritical perturbations are effective as stability attacks— they can mislead the learner by reinforcing the non-robust features.
Revised: Our theoretical analysis explains why hypocritical perturbations are effective as stability attacks— they can mislead the learner by reinforcing the non-robust features.

Original: Experiments demonstrate that hypocritical perturbations are harmful to conventional adversarial training on benchmark datasets, and enlarging the defense budget is essential for mitigating hypocritical pertubrations.
Revised: Experiments demonstrate that hypocritical perturbations are harmful to conventional adversarial training on benchmark datasets, and enlarging the defense budget is essential for mitigating hypocritical pertubrations.

Original: Future work includes relaxing the assumption that the adversary perturbs the entire training set and designing more effective stability attacks against adversarial training.
Revised: Future work includes relaxing the assumption that the adversary perturbs the entire training set and designing more effective stability attacks against adversarial training.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was supported by the National Natural Science Foundation of China (Grant No. 61732006, 62076124, 62076128, 62106028), the National Key R&D Program of China (2020AAA0107000).
Revised: This work was supported by the National Natural Science Foundation of China (Grant No. 61732006, 62076124, 62076128, 62106028), the National Key R&D Program of China (2020AAA0107000).

Original: Lei Feng was also supported by Chongqing Overseas Chinese Entrepreneurship, Innovation Support
Revised: Lei Feng was also supported by Chongqing Overseas Chinese Entrepreneurship, Innovation Support

Original: Program and CAAI-Huawei MindSpore Open Fund.
Revised: Program and CAAI-Huawei MindSpore Open Fund.

Original: The authors wish to thank the anonymous reviewers for their helpful comments and suggestions.? A Additional Related Work
Revised: The authors wish to thank the anonymous reviewers for their helpful comments and suggestions.

Original: In this part, we discuss several independent (or concurrent) works that are closely related to this work.

Original: Zhu et al.

Original: [88] study the effect of conventional adversarial training on differentiating noisy labels, while Zhang et al.

Original: [85] show that deliberately injected noisy labels may serve as a regularization that alleviates robust overfitting.

Original: Our results focus on the clean-label setting and provide evidence that conventional adversarial training can be hindered without modifying the labels.

Original: Yu et al.

Original: [80] suggest explaining the success of availability attacks from the perspective of shortcuts.

Original: They further adopt pre-trained models to extract useful features for mitigating model reliance on the shortcuts.

Original: This direction is orthogonal to ours.

Original: Liu et al.

Original: [38] improve the effectiveness of unlearnable examples [28] by generating grayscale perturbations and using data augmentations.

Original: They also conclude that conventional adversarial training will prevent a drop in accuracy measured both on clean images and adversarial images.

Original: Contrary to them, we show that, both theoretically and empirically, conventional adversarial training can be hindered by hypocritical perturbations, and we further analyze the necessity of enlarging the defense budget to resist stability attacks.

Original: Gao et al.

Original: [20] revisit the trade-off between adversarial robustness and backdoor robustness [72].

Original: They conclude that backdoor attacks are ineffective when the defense budget of adversarial training surpasses the trigger magnitude.

Original: In contrast, our results indicate that stability attacks are still harmful to adversarial training when the defense budget is not large enough.

Original: In a simple statistical setting, a defense budget ϵ + η is necessary (where η is a positive number).

Original: In the general case, a defense budget of 2 ϵ is sufficient.

Original: In our experiments, a defense budget of about 1 .

Original: 5 ϵ ∼ 1 .

Original: 75 ϵ provides the best empirical ϵ -robustness.

Original: Wang et al.

Original: [71] argue that it is necessary to use robust features for compromising adversarial training.

Original: To this end, they adopt a relatively large attack budget ϵ a = 32 / 255 for crafting their poisons (they use one type of adversarial perturbations), and show that their poisons can decrease the performance of the models trained using smaller defense budgets (such as ϵ d = 8 / 255 and ϵ d = 16 / 255 ).

Original: In contrast, we focus on a more realistic setting that does not require a larger attack budget.

Original: We demonstrate that it is possible to hinder adversarial training when ϵ a = ϵ d .

Original: Furthermore, we provide both theoretical and empirical results showing how to adapt the defense to maintain robustness.

Original: Fu et al.

Original: [19] explore how to protect data privacy against adversarial training.

Original: The main purpose of their poisons is to compromise adversarial training by requiring the perturbation budget of their poisons to be larger than that of adversarial training.

Original: In this way, they show that the natural accuracy of the adversarially trained models can be largely decreased, let alone robust accuracy.

Original: From this perspective, our work is complementary to theirs.

Original: We pursue to not increase the attack budget of stability attacks, keeping it as small as possible.

Original: We successfully demonstrate that stability attacks are still harmful to conventional adversarial training without enlarging the attack budget.

Original: This makes the threat of stability attacks more insidious than that of Fu et al.

Original: On the other hand, we find that our implementation of stability attacks using hypocritical perturbations has some similarities to the robust unlearnable examples in Fu et al.

Original: Specifically, although the robust unlearnable examples are generated via a complicated min-min-max optimization process [19], we notice that their noise generator can be viewed as an adversarially trained model.

Original: This implies that the robust error-minimizing (REM) noise [19] might be useful in demonstrating the feasibility of stability attacks.

Original: To verify this, we run the source code from the authors with default hyperparameters, and compare our crafted hypocritical perturbation with their generated noise under the setting of stability attacks.

Original: For a fair comparison, here we apply a very simple trick called EOT [2] in our method, since the trick is also used by REM [19].

Original: The additional time cost of the

Original: EOT trick is very small and negligible.

Original: Our experimental results, shown in Table 8, demonstrate that the robust error-minimizing noise is also effective as stability attacks, though it was originally proposed as a delusive attack.

Original: It is noteworthy that the robust accuracy is not evaluated in [19].

Original: In this sense, the effectiveness of REM as an stability attack can be regarded as one of our novel findings.

Original: Importantly, our method outperforms REM in terms of the robust accuracy against

Original: AutoAttack.

Original: Since AutoAttack is the most reliable evaluation metric of model robustness among the test-time attacks [11], this indicates that our method is reliably more effective than REM in degrading model robustness.

Original: It is also noteworthy that our method is significantly more efficient than REM, as shown in Table 9.

Original: We note that the efficiency of our method is largely due to the fact that our crafting model is fast to train.

Original: Specifically, the time cost of training our crafting model is only 0.3 hours, while it takes 20.8 hours for REM.

Original: That is, our crafting model is nearly 70 times faster to train than that of REM.

Original: In short, our method is not only more effective, but also more efficient, than REM as a stability attack.

Original: [Table]

Original: [Table]

Original: B Omitted Tables

Original: [Table]

Original: [Table]

Original: CIFAR-10.

Original: We report mean and standard deviation over 3 random runs

Original: [Table]

Original: C Proofs

Original: In this section, we provide the proofs of our theoretical results in Section 3 and Section 4.

Original: C.1 Proof of Proposition

Original: Proposition 1 (restated).

Original: Let ϵ = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of a classifier correctly predicting y on the data (7) under ℓ ∞ perturbations.

Original: Then, we have

Original: [Equation]

Original: Proof.

Original: Recalling that in Equation (8), we have the natural classifier:

Original: [Equation]

Original: := sign( w

Original: [Equation]

Original: x ) , where w nat :=

Original: [1 , η, . . .

Original: , η ] ,

Original: [Equation]

Original: 12) and in Equation (9)

Original: , the robust classifier is defined as:

Original: [Equation]

Original: , where w rob :=

Original: [Equation]

Original: Then, the adversarial accuracy of the natural classifier on the data D (7) is

Original: [Equation]

Original: Similarly

Original: [Equation]

Original: the adversarial accuracy of the robust classifier on the data D (7) is

Original: [Equation]

Original: C.2 Proof of Theorem

Original: The following theorems rely on the analytical solution of optimal linear ℓ ∞ -robust classifier on mixture Gaussian distributions.

Original: Concretely, the optimization problem is to minimize the adversarial risk on a distribution (cid:98) D with a defense budget

Original: ˆ ϵ :

Original: [Equation]

Original: where R

Original: [Equation]

Original: where f ( x ) = sign( w ⊤ x + b ) , and 1 ( · ) denotes the indicator function.

Original: We note that optimal linear robust classifiers have been obtained for certain data distributions in previous work [67, 29, 14, 30, 78, 64].

Original: Here, our goal is to establish similar optimal linear robust classifiers for the classification tasks in our setting.

Original: We only employ linear classifiers, since it is highly nontrivial to consider non-linearity for adversarial training on mixture Gaussian distributions [14].

Original: Lemma 1.

Original: Assume that the adversarial perturbation in data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .

Original: Then, minimizing the adversarial risk (16) on the data T adv with a defense budget ϵ can result in a classifier that assigns 0 weight to the features x i for i ≥ 2 .

Original: Proof.

Original: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T adv , which can be written as follows:

Original: [Equation]

Original: Consider an optimal solution w in which w i > 0 for some i ≥

Original: [Equation]

Original: Then

Original: [Equation]

Original: we have

Original: R ϵ adv ( f, T ( − 1) adv ) =

Original: Pr 

Original: [Equation]

Original:   

Original: [Equation]

Original: (18) where w nat :=

Original: [1 , η, . . .

Original: , η ] as in Equation (8).

Original: Since w i > 0 , B is maximized when δ i

Original: =

Original: ϵ .

Original: Thus, the contribution of terms depending on w i to B is a normally-distributed random variable with mean 2 ϵ − η .

Original: Since 2 ϵ − η ≥ 0 , setting w i to zero can only decrease the risk.

Original: This contradicts the optimality of w .

Original: Formally,

Original: [Equation]

Original: We can also assume w i < 0 and similar contradiction holds.

Original: [-Therefore, minimizing the adversarial risk on T adv leads to w i = 
-]
Revised: [-Therefore, minimizing the adversarial risk on T adv leads to w i = 
-]

Original: for i ≥ 2 .

Original: Lemma 2.

Original: Assume that the adversarial perturbation in data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .

Original: Then, minimizing the adversarial risk (16) on the data T adv with a defense budget ϵ results in a classifier that assigns a positive weight to the feature x 1 .

Original: Proof.

Original: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T adv , which has been written in Equation (17).

Original: Consider an optimal solution w in which w 1 ≤ 0 .

Original: Then, we have

Original: [Equation]

Original: Since w 1 ≤ 0 , D is maximized when δ 1 = − ϵ .

Original: Thus, the contribution of the term depending on w 1 to D is a normally-distributed random variable with mean − 1 .

Original: Since the mean is negative, setting w 1 to be positive can decrease the risk.

Original: This contradicts the optimality of w .

Original: Formally,

Original: [Equation]

Original: where p > 0 is any positive number.

Original: Therefore, minimizing the adversarial risk on T adv leads to w 1 > 0 .

Original: Theorem 1 (restated).

Original: Assume that the adversarial perturbation in the training data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .

Original: Then, the optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on T adv with a defense budget ϵ is equivalent to the robust classifier (9).

Original: Proof.

Original: [-By Lemma 1 and Lemma 2, we have w 1 > 0 and w i = 
-]
Revised: [-By Lemma 1 and Lemma 2, we have w 1 > 0 and w i = 
-]

Original: ( i ≥ 2 ) for an optimal linear ℓ ∞ -robust classifier.

Original: Then, the adversarial risk on the distribution T adv can be simplified by solving the inner maximization problem first.

Original: Formally,

Original: [Equation]

Original: which is equivalent to the natural risk on a mixture Gaussian distribution D tmp : x ∼ N ( y · µ tmp , σ 2 I ) , where µ tmp = (1 − 2 ϵ, 0 , .

Original: We note that the Bayes optimal classifier for D tmp is f tmp ( x ) = sign( µ ⊤ tmp x ) .

Original: Specifically, the natural risk

Original: [Equation]

Original: which is minimized when w 1 = 1 − 2 ϵ > 0 and b = 0 .

Original: That is, minimizing the adversarial risk R ϵ adv ( f, T adv ) can lead to an optimal linear ℓ ∞ -robust classifier f tmp ( x ) .

Original: Meanwhile, f tmp ( x ) is equivalent to the robust classifier (9).

Original: This concludes the proof of the theorem.

Original: C.3 Proof of Theorem

Original: Lemma 3.

Original: Minimizing the adversarial risk (16) on the data T hyp (11) with a defense budget ϵ results in a classifier that assigns positive weights to the features x i for i ≥ 1 .

Original: Proof.

Original: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T hyp , which can be written as follows:

Original: [Equation]

Original: Consider an optimal solution w in which w i ≤ 0 for some i ≥ 1 .

Original: Then, we have

Original: R ϵ adv

Original: [Equation]

Original: f

Original: [Equation]

Original: 25) where w nat :=

Original: [1 , η, . . .

Original: , η ] as in Equation (8).

Original: Since w i ≤ 0 , H is maximized when δ i = − ϵ .

Original: Thus, the contribution of terms depending on w i to H is a normally-distributed random variable with mean − [ w nat ]

Original: i − 2 ϵ .

Original: Since the mean is negative, setting w i to be positive can decrease the risk.

Original: This contradicts the optimality of w .

Original: Formally,

Original: [Equation]

Original: where p > 0 is any positive number.

Original: Therefore, minimizing the adversarial risk on T hyp leads to w i > 0 for i ≥ 1 .

Original: Theorem 2 (restated).

Original: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ is equivalent to the natural classifier (8).

Original: Proof.

Original: By Lemma

Original: [Equation]

Original: we have w i > 0 for i ≥ 1 for an optimal linear ℓ ∞ -robust classifier.

Original: Then

Original: [Equation]

Original: we have

Original: [Equation]

Original: 27) which is equivalent to the natural risk on the mixture Gaussian distribution D : x ∼ N ( y · w nat , σ 2 I ) , where w nat

Original: =

Original: (1 , η, . . . , η ) .

Original: We note that the Bayes optimal classifier for D is f nat ( x ) = sign( w ⊤ nat x ) .

Original: Specifically, the natural risk

Original: [Equation]

Original: which is minimized when w 1 = 1 , w i =

Original: η for i ≥ 2 , and b = 0 .

Original: That is, minimizing the adversarial risk R ϵ adv ( f, T hyp ) can lead to an optimal linear ℓ ∞ -robust classifier f nat ( x ) , which is equivalent to the natural classifier (8).

Original: This concludes the proof of the theorem.

Original: C.4 Proof of Theorem

Original: Lemma 4.

Original: Minimizing the adversarial risk (16) on the data T hyp (11) with a defense budget ϵ + η can result in a classifier that assigns 0 weight to the features x i for i ≥ 2 .

Original: Proof.

Original: The goal is to minimize the adversarial risk on the distribution T hyp , which can be written as follows:

Original: [Equation]

Original: Consider an optimal solution w in which w i > 0 for some

Original: i ≥ 2 .

Original: Then, we have

Original: R

Original: [Equation]

Original: +

Original: [Equation]

Original: adv ( f, T

Original: [Equation]

Original: 1) hyp

Original: [Equation]

Original: =

Original: Pr

Original: [Equation]

Original: 30) where w nat :=

Original: [1 , η, . . .

Original: , η ] .

Original: Since w i > 0 , J is maximized when δ i

Original: = ϵ + η .

Original: Thus, the contribution of terms depending on w i to J is a normally-distributed random variable with mean 0 .

Original: Thus, setting w i to zero will not increase the risk.

Original: Formally, we have

Original: [Equation]

Original: We can also assume w i < 0 and a similar argument holds.

Original: Similar arguments also hold for R ϵ + η adv ( f, T (+1) hyp ) .

Original: [-Therefore, minimizing the adversarial risk on T hyp can lead to w i = 
-]
Revised: [-Therefore, minimizing the adversarial risk on T hyp can lead to w i = 
-]

Original: for i ≥ 2 .

Original: Theorem 3 (restated).

Original: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ + η is equivalent to the robust classifier (9).

Original: Moreover, any defense budget lower than ϵ + η will yield classifiers that still rely on all the non-robust features.

Original: Proof.

Original: [-By Lemma 4, we have w i = 
-]
Revised: [-By Lemma 4, we have w i = 
-]

Original: ( i ≥ 2 ) for an optimal linear ℓ ∞ -robust classifier.

Original: Also, the robust classifier will assign a positive weight to the first feature.

Original: This is similar to the case in Lemma 2 and we omit the proof here.

Original: Then, we have

Original: [Equation]

Original: which is equivalent to the natural risk on a mixture Gaussian distribution D tmp : x ∼ N ( y · µ tmp , σ 2 I ) , where µ tmp = (1 − η, 0 , .

Original: We note that the Bayes optimal classifier for D tmp is f tmp ( x ) = sign( µ ⊤ tmp x ) .

Original: Specifically, the natural risk

Original: [Equation]

Original: which is minimized when w 1 = 1 − η > 0 and b = 0 .

Original: That is, minimizing the adversarial risk R ϵ + η adv ( f, T hyp ) can lead to an optimal linear ℓ ∞ -robust classifier f tmp ( x ) , which is equivalent to the robust classifier (9).

Original: Moreover, when the defense budget ϵ d is less than ϵ + η , the condition in Lemma 4 no longer holds.

Original: Instead, in this case, the robust classifier will assign positive weights to the features (i.e., w i > 0 for i ≥ 1 ).

Original: This is similar to the case in Lemma 3, and thus we omit the proof here.

Original: Consequently, this yields classifiers that still rely on all the non-robust features.

Original: C.5 Proof of Theorem

Original: Theorem 4 (restated).

Original: For any data distribution and any adversary with an attack budget ϵ , training models to minimize the adversarial risk with a defense budget 2 ϵ on the perturbed data is sufficient to ensure ϵ -robustness.

Original: Proof.

Original: For clarity, we rewrite the adversarial risk in (2) with a defense budget ϵ as follows:

Original: [Equation]

Original: where T = { ( x i , y i ) } ni =1 denotes the empirical training data.

Original: Consider any adversary with an attack budget ϵ , who can perturb x to x + p such that ∥ p ∥ ≤ ϵ .

Original: Then, the learner will receive a perturbed version of training data T ′

Original: =

Original: { ( x i + p

Original: i , y i ) } ni =1 .

Original: For any perturbed data point ( x i + p

Original: i , y i ) , we have

Original: [Equation]

Original: By summarizing the training points, we have

Original: [Equation]

Original: That is, the adversarial risk with a defense budget 2 ϵ on the perturbed data is an upper bound of the adversarial risk with a defense budget ϵ on the original data.

Original: Therefore, a defense budget 2 ϵ is sufficient to ensure the learning of ϵ -robustness.

Original: D Experimental Settings

Original: Adversary capability.

Original: We focus on the clean-label setting, where an adversary can only provide correctly labeled but misleading training data.

Original: In this setting, the main constraint is to craft perturbations as small as possible [16].

Original: Thus, we consider an ℓ ∞ adversary with an attack budget ϵ a = 8 / 255 by following Huang et al.

Original: [28], Yuan and Wu [81], Tao et al.

Original: [64], Fowl et al.

Original: We note that this constraint is consistent with common research on test-time adversarial examples [1].

Original: Crafting details.

Original: We conduct stability attacks by applying the hypocritical perturbation into the training set.

Original: Unless otherwise specified, we craft the perturbations by solving the error-minimizing objective (4) with 100 steps of PGD, where a step size of 0 . 8 / 255 is used by following Fowl et al.

Original: Our crafting model is adversarially trained with a crafting budget ϵ c = 0 .

Original: 25 ϵ a for 10 epochs before generating perturbations.

Original: That is, setting ϵ c = 2 / 255 performs best, as shown in Figure 2(a).

Original: Training details.

Original: We evaluate the effectiveness of the hypocritical perturbation on benchmark datasets including CIFAR-10/100 [33], SVHN [43], and Tiny-ImageNet [34].

Original: Unless otherwise specified, we use ResNet- 18 [26] as the default architecture for both the crafting model and the learning model.

Original: For adversarial training, we mainly follow the settings in previous studies [83, 70, 50].

Original: By convention, the defense budget is equal to the attack budget, i.e., ϵ d = 8 / 255 .

Original: The networks are trained for 100 epochs using SGD with momentum 0.9, weight decay 5 × 10 − 4 , and an initial learning rate of 0.1 that is divided by 10 at the 75-th and 90-th epoch.

Original: Early stopping is done with holding out 1000 examples from the training set.

Original: Simple data augmentations such as random crop and horizontal flip are applied.

Original: The inner maximization problem during adversarial training is solved by 10-steps PGD (PGD-10) with step size 2 / 255 .

Original: E Feature-level Analysis on CIFAR-

Original: In Section 3.1, we theoretically showed that the hypocritical perturbation can cause the poisoned model to rely more on non-robust features, thus the natural accuracy of the adversarially trained model is increased while the robust accuracy is decreased.

Original: In this part, we aim to provide empirical evidence on the role of non-robust features in the success of our poisoning method on a benchmark dataset.

Original: In particular, we will demonstrate that our hypocritical perturbation successfully makes the poisoned model learn more non-robust features.

Original: To show this, by following Section 3.2 of Ilyas et al.

Original: [29], we construct a training set where the only features that are useful for classification are the non-robust features (that are extracted from the poisoned model).

Original: The standard accuracy of the classifier trained on the constructed dataset can reflect how many non-robust features are learned by the poisoned model (denoted as f ).

Original: To accomplish this, we modify each input-label pair ( x , y ) as follows.

Original: We select a target class t uniformly at random among classes.

Original: Then, we add a small adversarial perturbation to x as follows:

Original: [Equation]

Original: The resulting input-label pairs ( x adv , t ) make up the new training set.

Original: Since the resulting inputs x adv are nearly indistinguishable from the originals x , the label t assigned to the modified input is simply incorrect to a human observer.

Original: Therefore, only the non-robust features in the training set are predictive, while the non-robust features are extracted from the poisoned model.

Original: We compare the model poisoned by our hypocritical perturbation with the baseline model trained on clean data.

Original: These two models correspond to the second row and last row in Table 2, respectively.

Original: Using these two models, we construct two datasets in the above-mentioned manner, respectively.

Original: Then, two new predictors are trained on the two constructed datasets, respectively, and both predictors are evaluated on clean data.

Original: Training parameters follow exactly those adopted by Ilyas et al.

Original: Our numerical results are summarized in Table 13.

Original: the poisoned model

Original: [Table]

Original: As shown in Table 13, the non-robust features learned by the poisoned model are much more predictive than the baseline.

Original: This indicates that the effect of our poisoning method on the non-robust features learned by the poisoned model is validated empirically.

Original: F Broader Impact

Original: The attack method in this work might be used by an agent in the real world to damage the robust availability of a machine-learning-based system.

Original: We discourage this malicious behavior by presenting the threat model of stability attacks to the community.

Original: We further propose an adaptive defense to mitigate this issue.

Original: The adaptive defense would help to build a more secure and robust machine learning system in the real world.

Original: At the same time, the adaptive defense introduces an additional time cost to search for an appropriate defense budget, which might have a negative impact on carbon emission reduction.

Original: Furthermore, society should not be overly optimistic about AI safety, since the current studies mostly focus on perturbations bounded by simple norms (e.g., ℓ ∞ norm in this paper).

Original: There might exist perturbations beyond the ℓ p ball in the real world, and we are still far from complete model robustness.

Original: G On the Trade-off between Accuracy and Robustness

Original: An interesting implication of this work is that the hypocritical perturbation exploits the trade-off between standard generalization and adversarial robustness, a phenomenon that has been widely observed in existing works on adversarial training [67, 83, 14, 40, 59, 79].

Original: Prior work mainly observed that adversarial training improves robust accuracy at the cost of natural accuracy when the training data is clean .

Original: An explanation for the phenomenon is that there are non-robust features in the original dataset, which are predictive yet brittle [67, 29].

Original: Unlike prior work, the trade-off in this work occurs when the training data is hypocritically perturbed .

Original: Specifically, we make the following observations:

Original: When trained on the hypocritically perturbed data, conventional adversarial training produces models with lower robust accuracy but higher natural accuracy (e.g., see Table 2, Table 3, and Table 4).

Original: When trained on the hypocritically perturbed data, adversarial training with adaptive budget can improve robust accuracy while reducing natural accuracy (e.g., see Table 5).

Original: These two observations align well with our theoretical analyses in Section 3 and Section 4, respectively.

Original: Concretely, our analyses suggest that the hypocritical perturbation works by reinforcing the non-robust features in the original data, so that the models adversarially trained on the manipulated data still rely on the non-robust features.

Original: In this way, the natural accuracy of the models increases because the non-robust features are predictive, while the robust accuracy decreases because the non-robust features are brittle.

Original: Furthermore, the effectiveness of the adaptive defense lies in the fact that the reinforced non-robust features can be neutralized by enlarging the defense budget of adversarial training.

Original: Thus, the adaptive defense improves robustness at the cost of accuracy.

Original: Meanwhile, we note that it would be unsatisfactory that test robustness is improved at the cost of standard generalization.

Original: Several improvements have been proposed to alleviate this issue in the case where the training data is clean, such as RST [49], FAT [84], and SCORE [48].

Original: Incorporating these advances would be helpful in resisting stability attacks, and we leave this as future work.

Original: Finally, we remark that the focus of stability attacks is to degrade test robustness.

Original: For this reason, we do not impose additional restrictions on their impact on natural accuracy.

Original: Having that said, as a method of stability attacks, the hypocritical perturbation is observed to improve natural accuracy while reducing robust accuracy.

Original: We note that this makes stability attacks more insidious.

Original: For example, if a poisoned model exhibits higher natural accuracy, practitioners would be more easily enticed to deploy it in a real-world system.

Original: However, as its robust accuracy is actually undesirably low, the system is prone to losing its normal function when encountering test-time perturbations.

Original: In short, the negative impacts of stability attacks are serious, even with higher natural accuracy.

Original: Thus, it is imperative to design better defense methods to mitigate the threat of stability attacks.