Original: Can Adversarial Training Be Manipulated By Non-Robust Features?
Revised: Can Adversarial Training Be Manipulated By Non-Robust Features?

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Adversarial training, originally designed to resist test-time adversarial examples,has shown to be promising in mitigating training-time availability attacks .
Revised: Adversarial training, originally designed to resist test-time adversarial examples, has shown to be promising in mitigating training-time availability attacks .

Original: Thisdefense ability, however, is challenged in this paper.
Revised: This defense ability, however, is challenged in this paper.

Original: We identify a novel threatmodel named stability attacks , which aims to hinder robust availability by slightlymanipulating the training data.
Revised: We identify a novel threat model named stability attacks , which aims to hinder robust availability by slightly manipulating the training data.

Original: Under this threat, we show that adversarial trainingusing a conventional defense budget (cid:15) provably fails to provide test robustnessin a simple statistical setting, where the non-robust features of the training datacan be reinforced by (cid:15) -bounded perturbation.
Revised: Under this threat, we show that adversarial training using a conventional defense budget ϵ provably fails to provide test robustness in a simple statistical setting, where the non-robust features of the training data can be reinforced by ϵ -bounded perturbation.

Original: Further, we analyze the necessity ofenlarging the defense budget to counter stability attacks.
Revised: Further, we analyze the necessity of enlarging the defense budget to counter stability attacks.

Original: Finally, comprehensiveexperiments demonstrate that stability attacks are harmful on benchmark datasets,and thus the adaptive defense is necessary to maintain robustness.
Revised: Finally, comprehensive experiments demonstrate that stability attacks are harmful on benchmark datasets, and thus the adaptive defense is necessary to maintain robustness.

Original:  Introduction
Revised:  Introduction

Original: Robustness to input perturbations is crucial to machine learning deployment in various applications,such as spam ﬁltering [13] and autonomous driving [6].
Revised: Robustness to input perturbations is crucial to machine learning deployment in various applications, such as spam filtering [13] and autonomous driving [6].

Original: One of the most popular methods forimproving test robustness is adversarial training [39, 1].
Revised: One of the most popular methods for improving test robustness is adversarial training [39, 1].

Original: By augmenting the training data with(cid:15) -bounded and on-the-ﬂy crafted adversarial examples, adversarial training helps the learned modelresist test-time perturbations [39]. [Figure]
Revised: By augmenting the training data with ϵ -bounded and on-the-fly crafted adversarial examples, adversarial training helps the learned model resist test-time perturbations [39]

Original: training. On the other hand, machine learning systemsare vulnerable to training-time availability attacks [3].
Revised: On the other hand machine learning systems are vulnerable to training-time availability attacks [3].

Original: In particular, small perturbations applied into the training data before training sufﬁce to degrade the overall test performance ofnaturally trained models [16, 28].
Revised: In particular, small perturbations applied into the training data (before training) suffice to degrade the overall test performance of naturally trained models [16, 28].

Original: Fortunately,recent work has shown that adversarial training [39] is capable of mitigating this type ofthreat [64, 18].
Revised: Fortunately, recent work has shown that adversarial training [39] is capable of mitigating this type of threat [64].

Original: In other words, even if the training data is manipulated by an adversary to maximize the test error, adversarially trained modelscan still achieve considerable accuracy on cleantest data.
Revised: In other words, even if the training data is manipulated by an adversary to maximize the test error, adversarially trained models can still achieve considerable accuracy on clean test data.

Original: However, previous work hardly inspects the test robustness of the models, which is what adversarial training was originally proposedfor [23, 39].
Revised: However, previous work hardly inspects the test robustness of the models, which is what adversarial training was originally proposed for [23, 39].

Original: This naturally raises the following question
Revised: This naturally raises the following question:

Original: Are the models adversarially trained on the manipulated data robust to test-time perturbations?
Revised: Are the models adversarially trained on the manipulated data robust to test-time perturbations?

Revised: [Figure]

Original: In this work, we show that conventional adversarial training may fail to provide test robustness whenthe training data is manipulated by an adversary, and thus an adaptive defense is necessary to resolvethis issue.
Revised: In this work, we show that conventional adversarial training may fail to provide test robustness when the training data is manipulated by an adversary, and thus an adaptive defense is necessary to resolve this issue.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: We introduce a novel threat model called stability attacks , where an adversary aims to degrade the overall test robustness of adversarially trained models by slightly perturbing the training data.
Revised: We introduce a novel threat model called stability attacks , where an adversary aims to degrade the overall test robustness of adversarially trained models by slightly perturbing the training data.

Original: Figure 1 illustrates the threat of stability attacks.
Revised: Figure 1 illustrates the threat of stability attacks.

Original: We show that adversarial training using a conventional defense budget provably fails understability attacks in a simple statistical setting.
Revised: We show that adversarial training using a conventional defense budget provably fails under stability attacks in a simple statistical setting.

Original: Speciﬁcally, a defense budget of (cid:15) will producemodels that are not robust to (cid:15) -bounded adversarial examples when the training data ishypocritically perturbed.
Revised: Specifically, a defense budget of ϵ will produce models that are not robust to ϵ -bounded adversarial examples when the training data is hypocritically perturbed.

Original: We unveil that the aforementioned vulnerability stems from the existence of the non-robust (predictive, yet brittle) features [29] in the original training data.
Revised: We unveil that the aforementioned vulnerability stems from the existence of the non-robust (predictive, yet brittle) features [29] in the original training data.

Original: When the non-robust features are reinforced by hypocritical perturbations, the conventional defense budget will be insufﬁcient to offset the negative impact.
Revised: When the non-robust features are reinforced by hypocritical perturbations, the conventional defense budget will be insufficient to offset the negative impact.

Original: We further show that a defense budget of 2 (cid:15) is capable of resisting any stability attack foradversarial training, while the budget can be reduced to (cid:15) + η in a simple statistical setting,where η is the magnitude of the non-robust features.
Revised: We further show that a defense budget of 2 ϵ is capable of resisting any stability attack for adversarial training, while the budget can be reduced to ϵ + η in a simple statistical setting, where η is the magnitude of the non-robust features.

Original: We demonstrate that stability attacks are harmful to conventional adversarial training on benchmark datasets.
Revised: We demonstrate that stability attacks are harmful to conventional adversarial training on benchmark datasets.

Original: In addition, our empirical study suggests that enlarging the defensebudget is essential for mitigating hypocritical perturbations.
Revised: In addition, our empirical study suggests that enlarging the defense budget is essential for mitigating hypocritical perturbations.

Original: To the best of our knowledge, this is the ﬁrst work that studies the robustness of adversarial trainingagainst stability attacks.
Revised: To the best of our knowledge, this is the first work that studies the robustness of adversarial training against stability attacks.

Original: Both theoretical and empirical evidences show that the conventional defensebudget (cid:15) is insufﬁcient under the threat of (cid:15) -bounded training-time perturbations.
Revised: Both theoretical and empirical evidences show that the conventional defense budget ϵ is insufficient under the threat of ϵ -bounded training-time perturbations.

Original: Our ﬁndings suggestthat practitioners should consider a larger defense budget of no more than 2 (cid:15) (practically, about1 .
Revised: Our findings suggest that practitioners should consider a larger defense budget of no more than 2 ϵ (practically, about 1 .

Original: 5 (cid:15) ∼ 1 .
Revised: 5 ϵ ∼ 1 .

Original: 75 (cid:15) ) to achieve a better (cid:15) -robustness.
Revised: 75 ϵ ) to achieve a better ϵ -robustness.

Original:  Threat Models
Revised:  Threat Models

Original: In this section, we formally introduce the threat model of stability attacks.
Revised: In this section, we formally introduce the threat model of stability attacks.

Original: We begin by revisiting theconcepts of natural risk, adversarial risk, and delusive attacks.
Revised: We begin by revisiting the concepts of natural risk, adversarial risk, and delusive attacks.

Original: These concepts naturally give rise toour formulation of stability attacks.
Revised: These concepts naturally give rise to our formulation of stability attacks.

Original: 2.1 Preliminaries
Revised: 2.1 Preliminaries

Original: Setup.
Revised: Setup.

Original: We consider a classiﬁcation task with input-label pairs ( x , y ) from an underlying distribution
Revised: We consider a classification task with input-label pairs ( x , y ) from an underlying distribution

Original: D over X × [ k ] .
Revised: D over X × [ k ] .

Original: The goal is to learn a (robust) classiﬁer f : X → [ k ] that predicts a label y for agiven input x .
Revised: The goal is to learn a (robust) classifier f : X → [ k ] that predicts a label y for a given input x .

Original: Natural training (NT).
Revised: Natural training (NT).

Original: Most learning algorithms aim to maximize the generalization performanceon unperturbed examples, i.e., natural accuracy.
Revised: Most learning algorithms aim to maximize the generalization performance on unperturbed examples, i.e., natural accuracy.

Original: The goal is to minimize the natural risk deﬁned as:
Revised: The goal is to minimize the natural risk defined as:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Adversarial training (AT).
Revised: Adversarial training (AT).

Original: Since the risk of adversarial examples (a.k.a.
Revised: Since the risk of adversarial examples (a.k.a.

Original: evasion attacks) wasfound to be unexpectedly high [5, 60], it has become increasingly important to defend the learneragainst the worst-case perturbations [23, 39].
Revised: evasion attacks) was found to be unexpectedly high [5, 60], it has become increasingly important to defend the learner against the worst-case perturbations [23, 39].

Original: In this context, the goal is to train a model that has lowadversarial risk given a defense budget (cid:15) :
Revised: In this context, the goal is to train a model that has low adversarial risk given a defense budget ϵ :

Original: [Equation]
Revised: [Equation]

Original: where we choose ∆ to be the set of (cid:15) -bounded perturbations, i.e., ∆ = { δ | (cid:107) δ (cid:107) ≤ (cid:15) } .
Revised: where we choose ∆ to be the set of ϵ -bounded perturbations, i.e., ∆ = { δ | ∥ δ ∥ ≤ ϵ } .

Original: This choice isthe most common one in the context of adversarial examples [67].
Revised: This choice is the most common one in the context of adversarial examples [67].

Original: To simplify the notation, we referto the robustness with respect to this set as (cid:15) -robustness.
Revised: To simplify the notation, we refer to the robustness with respect to this set as ϵ -robustness.

Original: It is worth noting [Table]
Revised: It is worth noting that R adv ( f ) ≥ R nat ( f ) always holds for any f , and the equation holds when ϵ = 0 .

Original: Delusive attacks.
Revised: Delusive attacks.

Original: Delusive attacks, which belong to training-time availability attacks, aim to preventthe learner from producing an accurate model by manipulating the training data “imperceptibly” [44].
Revised: Delusive attacks, which belong to training-time availability attacks, aim to prevent the learner from producing an accurate model by manipulating the training data “imperceptibly” [44].

Original: Concretely, the features of the training data can be perturbed, while the labels should remaincorrect [16,  56, 28, 79, 15, 64, 18, 19].
Revised: Concretely, the features of the training data can be perturbed, while the labels should remain correct [16, 42, 56, 28, 81, 15, 64, 18, 19].

Original: This malicious task can be formalized into the followingbi-level optimization problem:
Revised: This malicious task can be formalized into the following bi-level optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Here, the adversary aims to maximize the natural risk of the model f P (that is trained on themanipulated training set) by applying the generated perturbations P = { p i } ni =1 into the originaltraining set T = { ( x i , y i ) } ni =1 .
Revised: Here, the adversary aims to maximize the natural risk of the model f P (that is trained on the manipulated training set) by applying the generated perturbations P = { p i } ni =1 into the original training set T = { ( x i , y i ) } ni =1 .

Original: The commonly used feasible region is S
Revised: The commonly used feasible region is S

Original: = {{ p i } ni =1 | (cid:107) p i (cid:107) ≤ (cid:15) } .
Revised: = {{ p i } ni =1 | ∥ p i ∥ ≤ ϵ } .

Original: Generally, solving Equation (3) is computationally prohibitive for neural networks [64, 18].
Revised: Generally, solving Equation (3) is computationally prohibitive for neural networks [64, 18].

Original: Thus,various heuristic methods are explored to resolve this problem.
Revised: Thus, various heuristic methods are proposed to achieve the goal.

Original: For example, a representative methodcalled hypocritical perturbations [63, 64, 28, 19] is crafted as follows:
Revised: Among them, a representative method is the hypocritical perturbation [63, 64], crafted as follows:

Original: [Equation]
Revised: [Equation]

Original: where f craft is called the crafting model , pre-trained before generating poisons.
Revised: where f craft is called the crafting model , pre-trained before generating poisons.

Original: Tao et al.
Revised: Tao et al.

Original: [64] simplyadopted a naturally trained classiﬁer as the crafting model, while Huang et al.
Revised: [64] simply adopted a naturally trained classifier as the crafting model, while Huang et al.

Original: [28] proposed amin-min bi-level optimization process to pre-train the crafting model.
Revised: [28] proposed a min-min bi-level optimization process to pre-train the crafting model.

Original: Fu et al.
Revised: Fu et al.

Original: [19] further built theircrafting model via a min-min-max three-level optimization process, and generated their poisons byreplacing Equation (4) with a min-max bi-level objective.
Revised: [19] further built their crafting model via a min-min-max three-level optimization process, and generated their poisons by replacing Equation (4) with a min-max bi-level objective.

Original: Another representative method of delusive attacks is the adversarial perturbation , crafted by solving
Revised: Another representative method of delusive attacks is the adversarial perturbation , crafted by solving

Original: [Equation]
Revised: [Equation]

Original: Tao et al.
Revised: Tao et al.

Original: [64] and Fowl et al.
Revised: [64] and Fowl et al.

Original: [18] independently found that applying the adversarial perturbation tothe training data is very effective at compromising naturally trained models.
Revised: [18] both found that applying the adversarial perturbation to the training data is very effective at compromising naturally trained models.

Original: However, adversarialtraining has proven to be promising in defending against various delusive attacks [64].
Revised: However, adversarial training has proven to be promising in defending against various delusive attacks [64].

Original: 2.2 Stability Attacks
Revised: 2.2 Stability Attacks

Original: In contrast to delusive attacks that aim at increasing the natural risk, stability attacks attempt tomaximize the adversarial risk of the learner by slightly perturbing the training data:
Revised: In contrast to delusive attacks that aim at increasing the natural risk, stability attacks attempt to maximize the adversarial risk of the learner by slightly perturbing the training data:

Original: [Equation]
Revised: [Equation]

Original: where f P denotes the victim model, which is naturally or adversarially trained on the perturbed data.
Revised: where f P denotes the victim model, which is naturally or adversarially trained on the perturbed data.

Original: In other words, stability attacks seek to hinder the robust availability of the training data.
Revised: In other words, stability attacks seek to hinder the robust availability of the training data.

Original: Table 1shows the comparisons among different threat models.
Revised: Tableshows the comparisons among different threat models.

Original: The goal of stability attacks can be immediately achieved for naturally trained models, since they have already incurred high adversarial risk, even if the training data is clean [60].
Revised: The goal of stability attacks can be immediately achieved for naturally trained models, since they have already incurred high adversarial risk, even if the training data is clean [60].

Original: To ease the problem of high adversarial risk, adversarial training has been widely used to improve model’s adversarialrobustness [24, 12].
Revised: To ease the problem of high adversarial risk, adversarial training has been widely used to improve model’s adversarial robustness [24, 12].

Original: Hence, the main goal of stability attacks becomes to compromise the testrobustness of adversarially trained models.
Revised: Hence, the main goal of stability attacks becomes to compromise the test robustness of adversarially trained models.

Original: Note that Equation (6) is a multi-level optimization problem that is not easy to solve, our next questionis how to conduct effective stability attacks against adversarial training.
Revised: Note that Equation (6) is a multi-level optimization problem that is not easy to solve, our next question is how to conduct effective stability attacks against adversarial training.

Original: In the following sections, weintroduce an effective stability attack method and analyze the cost of resisting it.
Revised: In the following sections, we introduce an effective stability attack method and analyze the cost of resisting it.

Original: Justiﬁcation.
Revised: Remark 1.

Original: This work focuses on adding bounded pertubrations as small as possible.
Revised: This work focuses on adding bounded pertubrations as small as possible.

Original: We mostlyassume that the adversary has full control of training data (instead of changing a few) by followingprevious works [16, 28, 64, 17–19].
Revised: We mostly assume that the adversary has full control of training data (instead of changing a few) by following previous works [16, 28, 64, 17–19].

Original: This is a realistic assumption [16, 18].
Revised: This is a realistic assumption [16, 18].

Original: For instance, in someapplications an organization may agree to release some internal data for peer assessment, whilepreventing competitors from easily building a model with high test robustness; this can be achievedby perturbing the entire dataset via stability attacks before releasing.
Revised: For instance, in some applications an organization may agree to release some internal data for peer assessment, while preventing competitors from easily building a model with high test robustness; this can be achieved by perturbing the entire dataset via stability attacks before releasing.

Original: Moreover, this assumptionenables a worst-case analysis of the robustness of adversarial training, which may facilitate importanttheoretical insights.
Revised: Moreover, this assumption enables a worst-case analysis of the robustness of adversarial training, which may facilitate important theoretical implications.

Original:  How to Manipulate Adversarial Training
Revised:  How to Manipulate Adversarial Training

Original: Previous work suggests that adversarial training could defend against both evasion attacks and delusiveattacks [39, 64].
Revised: Previous work suggests that adversarial training could defend against both evasion attacks and delusive attacks [39, 64].

Original: However, in this paper, we show that adversarial training using a conventionaldefense budget (cid:15) may not be sufﬁcient to provide (cid:15) -robustness when confronted with stability attacks.
Revised: However, in this paper, we show that adversarial training using a conventional defense budget ϵ may not be sufficient to provide ϵ -robustness when confronted with stability attacks.

Original: In particular, we present a simple theoretical model where the conventional defense scheme provablyfails when the training data is hypocritically perturbed.
Revised: In particular, we present a simple theoretical model where the conventional defense scheme provably fails when the training data is hypocritically perturbed.

Original: The binary classiﬁcation task.
Revised: The binary classification task.

Original: The data model is largely based on the setting proposed by Tsipraset al.
Revised: The data model is largely based on the setting proposed by Tsipras et al.

Original: [67], which draws a distinction between robust features and non-robust features .
Revised: [67], which draws a distinction between robust features and non-robust features .

Original: Speciﬁcally, itconsists of input-label pairs ( x , y ) sampled from a Gaussian mixture distribution D as follows:
Revised: Specifically, it consists of input-label pairs ( x , y ) sampled from a Gaussian mixture distribution D as follows:

Original: [Equation]
Revised: [Equation]

Original: where η is much smaller than 1 (i.e., 0 < η (cid:28) 1 ).
Revised: where η is much smaller than 1 (i.e., 0 < η ≪ 1 ).

Original: Hence, samples from D consist of a robust feature( x 1 ) that is strongly correlated with the label, and d non-robust features ( x 2 , . .
Revised: Hence, samples from D consist of a robust feature ( x 1 ) that is strongly correlated with the label, and d non-robust features ( x 2 , . .

Original: , x d +1 ) that are veryweakly correlated with it.
Revised: , x d +1 ) that are very weakly correlated with it.

Original: Typically, an adversary can manipulate a large number of non-robustfeatures - e.g. d = Θ(1 /η 2 ) will sufﬁce.
Revised: Typically, an adversary can manipulate a large number of non-robust features - e.g. d = Θ(1 /η 2 ) will suffice.

Original: Before introducing the way to hinder robust availability, we brieﬂy illustrate the success of adversarialtraining when the training data is unperturbed.
Revised: Before introducing the way to hinder robust availability, we briefly illustrate the success of adversarial training when the training data is unperturbed.

Original: Natural and robust classiﬁers.
Revised: Natural and robust classifiers.

Original: For standard classiﬁcation, we consider a natural classiﬁer:
Revised: For standard classification, we consider a natural classifier:

Original: [Equation]
Revised: [Equation]

Original: , where [Equation] ] , (8) which is a minimizer of the natural risk (1) with 0-1 loss on the data (7), i.e., the Bayes optimalclassiﬁer.
Revised: which is a minimizer of the natural risk (1) with 0-1 loss on the data (7), i.e., the Bayes optimal classifier.

Original: However, in the adversarial setting, this natural classiﬁer is quite brittle.
Revised: However, in the adversarial setting, this natural classifier is quite brittle.

Original: Thus, it isimperative to obtain a robust classiﬁer:
Revised: Thus, it is imperative to obtain a robust classifier:

Original: [Equation]
Revised: [Equation]

Original: (9) which relies only on the robust feature x 1 .
Revised: which relies only on the robust feature x 1 .

Original: Illustration of adversarial accuracy.
Revised: Illustration of adversarial accuracy.

Original: In the adversarial setting, an adversary that is only allowed toperturb each feature by a moderate (cid:15) can effectively subvert the natural classiﬁer Tsipras et al.
Revised: In the adversarial setting, an adversary that is only allowed to perturb each feature by a moderate ϵ can effectively subvert the natural classifier Tsipras et al.

Original: Inparticular, if (cid:15) = 2 η , an adversary can essentially force each non-robust feature to be anti -correlatedwith the correct label.
Revised: In particular, if ϵ = 2 η , an adversary can essentially force each non-robust feature to be anti -correlated with the correct label.

Original: The following proposition, proved in Appendix C.1, gives the adversarialaccuracies of the natural classiﬁer f nat (8) and the robust classiﬁer f rob (9).
Revised: The following proposition, proved in Appendix C.1, gives the adversarial accuracies of the natural classifier f nat (8) and the robust classifier f rob (9).

Original: Proposition 1.
Revised: Proposition 1.

Original: Let (cid:15) = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of aclassiﬁer correctly predicting y on the data (7) under (cid:96) ∞ perturbations.
Revised: Let ϵ = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of a classifier correctly predicting y on the data (7) under ℓ ∞ perturbations.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Proposition 1 implies that the adversarial accuracy of the natural classiﬁer is < 50% when d ≥ 1 /η 2 .
Revised: Proposition 1 implies that the adversarial accuracy of the natural classifier is < 50% when d ≥ 1 /η 2 .

Original: Even worse, when σ ≤ 1 / 3 and d ≥ 3 /η 2 , the adversarial accuracy of the natural classiﬁer (8) isalways lower than 1% .
Revised: Even worse, when σ ≤ 1 / 3 and d ≥ 3 /η 2 , the adversarial accuracy of the natural classifier (8) is always lower than 1% .

Original: In contrast, the robust classiﬁer (9) yields a much higher adversarial accuracy(always > 50% ); when σ ≤ (1 − 2 η ) / 3 , its adversarial accuracy will be higher than 99% .
Revised: In contrast, the robust classifier (9) yields a much higher adversarial accuracy (always > 50% ); when σ ≤ (1 − 2 η ) / 3 , its adversarial accuracy will be higher than 99% .

Original: 3.1 Hypocritical Features Are Harmful
Revised: 3.1 Hypocritical Features Are Harmful

Original: The results above reveal the advantages of robust classiﬁers over natural classiﬁers.
Revised: The results above reveal the advantages of robust classifiers over natural classifiers.

Original: Note that sucha robust classiﬁer can be obtained by adversarial training on the original data (7).
Revised: Note that such a robust classifier can be obtained by adversarial training on the original data (7).

Original: However, thisdefense effect may not hold when the adversary is allowed to perturb the training data.
Revised: However, this defense effect may not hold when the adversary is allowed to perturb the training data.

Original: We show this by analyzing two representative perturbations: the adversarial perturbation [42, 64, 18]and the hypocritical perturbation [63,28, 64].
Revised: We show this by analyzing two representative perturbations: the adversarial perturbation [64, 18] and the hypocritical perturbation [63, 64].

Original: When applied into the training data, both perturbationsare effective as delusive attacks for naturally trained models.
Revised: When applied into the training data, both perturbations are effective as delusive attacks for naturally trained models.

Original: In the following, we show that theformer is harmless: adversarial training using a defense budget (cid:15) on the adversarially perturbed datacan still provide test robustness.
Revised: In the following, we show that the former is harmless: adversarial training using a defense budget ϵ on the adversarially perturbed data can still provide test robustness.

Original: In contrast, the latter is harmful: we ﬁnd that the same defensebudget can only produce non-robust classiﬁers when the training data is hypocritically perturbed.
Revised: In contrast, the latter is harmful: we find that the same defense budget can only produce non-robust classifiers when the training data is hypocritically perturbed.

Original: A harmless case.
Revised: A harmless case.

Original: Consider an adversary who is capable of perturbing the training data by an attackbudget (cid:15) .
Revised: Consider an adversary who is capable of perturbing the training data by an attack budget ϵ .

Original: The adversary may choose to shift each feature towards − y .
Revised: The adversary may choose to shift each feature towards − y .

Original: Hence, the learner would seeinput-label pairs ( x , y ) sampled i.i.d.
Revised: Hence, the learner would see input-label pairs ( x , y ) sampled i.i.d.

Original: from a training distribution T adv as follows:
Revised: from a training distribution T adv as follows:

Original: [Equation]
Revised: [Equation]

Original: where each feature of the samples from T adv is adversarially perturbed by a moderate (cid:15) .
Revised: where each feature of the samples from T adv is adversarially perturbed by a moderate ϵ .

Original: While thesesamples are deviate signiﬁcantly from the original distribution D (7), adversarial training on themusing a defense budget (cid:15) is still able to neutralize the non-robust features.
Revised: While these samples are deviate significantly from the original distribution D (7), adversarial training on them using a defense budget ϵ is still able to neutralize the non-robust features.

Original: Formally, in Appendix C.2we prove the following theorem.
Revised: Formally, in Appendix C. we prove the following theorem.

Original: Theorem 1 (Adversarial perturbation is harmless) .
Revised: Theorem 1 (Adversarial perturbation is harmless) .

Original: Assume that the adversarial perturbation in thetraining data T adv (10) is moderate such that η/ 2 ≤ (cid:15) < 1 / 2 .
Revised: Assume that the adversarial perturbation in the training data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .

Original: Then, the optimal linear (cid:96) ∞ -robustclassiﬁer obtained by minimizing the adversarial risk on T adv with a defense budget (cid:15) is equivalent tothe robust classiﬁer (9).
Revised: Then, the optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on T adv with a defense budget ϵ is equivalent to the robust classifier (9).

Original: This theorem indicates that the adversarial perturbation is harmless: (cid:15) -robustness can still be obtainedby adversarial training on such perturbed training data.
Revised: This theorem indicates that the adversarial perturbation is harmless: ϵ -robustness can still be obtained by adversarial training on such perturbed training data.

Original: A harmful case.
Revised: A harmful case.

Original: However, this defense effect can be completely broken by the hypocriticalperturbation.
Revised: However, this defense effect can be completely broken by the hypocritical perturbation.

Original: That is, the adversary can instead shift each feature towards y .
Revised: That is, the adversary can instead shift each feature towards y .

Original: Hence, the learner wouldsee input-label pairs ( x , y ) sampled i.i.d.
Revised: Hence, the learner would see input-label pairs ( x , y ) sampled i.i.d.

Original: from a training distribution T hyp as follows 1 :
Revised: from a training distribution T hyp as follows 2 :

Original: [Equation]
Revised: [Equation]

Original: 11) where each feature of the samples from T hyp is reinforced by a magnitude of (cid:15) .
Revised: where each feature of the samples from T hyp is reinforced by a magnitude of ϵ .

Original: While these samplesbecome more separable, adversarial training on them using the same defense budget will fail toneutralize the hypocritically perturbed features.
Revised: While these samples become more separable, adversarial training on them using the same defense budget will fail to neutralize the hypocritically perturbed features.

Original: Consequently, the resulting classiﬁers will inevitablyhave low adversarial accuracy.
Revised: Consequently, the resulting classifiers will inevitably have low adversarial accuracy.

Original: We make this formal in the following theorem proved in Appendix C.3.
Revised: We make this formal in the following theorem proved in Appendix C.3.

Original: Theorem 2 (Hypocritical perturbation is harmful) .
Revised: Theorem 2 (Hypocritical perturbation is harmful) .

Original: The optimal linear (cid:96) ∞ -robust classiﬁer obtainedby minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget (cid:15) is equivalentto the natural classiﬁer (8).
Revised: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ is equivalent to the natural classifier (8).

Original: This theorem implies that the conventional defense scheme can only produce non-robust classiﬁers,whose adversarial accuracy is as low as that of the natural classiﬁer (8).
Revised: This theorem implies that the conventional defense scheme can only produce non-robust classifiers, whose adversarial accuracy is as low as that of the natural classifier (8).

Original: That is saying, if (cid:15) = 2 η ,σ ≤ 1 / 3 and d ≥ 3 /η 2 , the classiﬁers cannot get adversarial accuracy better than 1% .
Revised: That is saying, if ϵ = 2 η , σ ≤ 1 / 3 and d ≥ 3 /η 2 , the classifiers cannot get adversarial accuracy better than 1% .

Original: Implications.
Revised: Implications.

Original: As it turns out, the seemingly beneﬁcial features in T hyp (11) are actually hypocritical.
Revised: As it turns out, the seemingly beneficial features in T hyp (11) are actually hypocritical.

Original: Therefore, the adversary is highly motivated to hide such hypocritical features in the training data, intending to cajole an innocent learner into relying on the non-robust features.
Revised: Therefore, the adversary is highly motivated to hide such hypocritical features in the training data, intending to cajole an innocent learner into relying on the non-robust features.

Original: Intriguingly, wenotice that the natural classiﬁer (8) (i.e., the crafting model used to derive the distribution T hyp )actually has η -robustness.
Revised: Intriguingly, we notice that the natural classifier (8) (i.e., the crafting model used to derive the distribution T hyp ) actually has η -robustness.

Original: This is essentially because the non-robust features in the data (7) can resistsmall-magnitude perturbations by design.
Revised: This is essentially because the non-robust features in the data (7) can resist small-magnitude perturbations by design.

Original: This motivates us to use “slightly robust” classiﬁers as thecrafting model in practice.
Revised: This motivates us to use “slightly robust” classifiers as the crafting model in practice.

Original: Indeed, our experimental results show that training the crafting modelwith 0 .
Revised: Indeed, our experimental results show that training the crafting model with 0 .

Original: 25 (cid:15) -robustness performs the best for conducting stability attacks.
Revised: 25 ϵ -robustness performs the best for conducting stability attacks.

Original: This is different from theprevious works [64, 18] that use naturally trained models as the crafting model for poisoning.
Revised: This is different from the previous works [64, 18] that use naturally trained models as the crafting model for poisoning.

Original:  The Necessity of Large Defense Budget
Revised:  The Necessity of Large Defense Budget

Original: We have shown that the hypocritical perturbation is harmful to the conventional adversarial trainingscheme.
Revised: We have shown that the hypocritical perturbation is harmful to the conventional adversarial training scheme.

Original: Fortunately, it is possible to strengthen the defense by using a larger defense budget, whilethe crux of the matter is how large the budget is needed.
Revised: Fortunately, it is possible to strengthen the defense by using a larger defense budget, while the crux of the matter is how large the budget is needed.

Original: We ﬁnd that the minimum value of the defense budget for a successful defense depends on the speciﬁcdata distribution.
Revised: We find that the minimum value of the defense budget for a successful defense depends on the specific data distribution.

Original: Let us ﬁrst consider the hypocritical data in T hyp (11).
Revised: Let us first consider the hypocritical data in T hyp (11).

Original: In this case, we show that alarger defense budget is necessary in the following theorem proved in Appendix C.4.
Revised: In this case, we show that a larger defense budget is necessary in the following theorem proved in Appendix C.4.

Original: Theorem 3 ( (cid:15) + η is necessary) .
Revised: Theorem 3 ( ϵ + η is necessary) .

Original: The optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizingthe adversarial risk on the perturbed data T hyp (11) with a defense budget (cid:15) + η is equivalent to therobust classiﬁer (9).
Revised: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ + η is equivalent to the robust classifier (9).

Original: Moreover, any defense budget lower than (cid:15) + η will yield classiﬁers that stillrely on all the non-robust features.
Revised: Moreover, any defense budget lower than ϵ + η will yield classifiers that still rely on all the non-robust features.

Original: This theorem implies that, in the case of the mixture Gaussian distribution under the threat of (cid:15) -bounded hypocritical perturbations, the learner needs a slightly larger defense budget (cid:15) + η to ensure(cid:15) -robustness.
Revised: This theorem implies that, in the case of the mixture Gaussian distribution under the threat of ϵbounded hypocritical perturbations, the learner needs a slightly larger defense budget ϵ + η to ensure ϵ -robustness.

Original: While it is challenging to analyze the minimum value of the defense budget in the general case, thefollowing theorem provides an upper bound of the budget.
Revised: While it is challenging to analyze the minimum value of the defense budget in the general case, the following theorem provides an upper bound of the budget.

Original: Theorem 4 (General case) .
Revised: Theorem 4 (General case) .

Original: For any data distribution and any adversary with an attack budget (cid:15) ,training models to minimize the adversarial risk with a defense budget 2 (cid:15) on the perturbed data issufﬁcient to ensure (cid:15) -robustness.
Revised: For any data distribution and any adversary with an attack budget ϵ , training models to minimize the adversarial risk with a defense budget 2 ϵ on the perturbed data is sufficient to ensure ϵ -robustness.

Original: The proof of Theorem 4 is deferred in Appendix C.5.
Revised: The proof of Theorem 4 is deferred in Appendix C.5.

Original: It implies that a defense budget twice to theattack budget should be safe enough under the threat of stability attacks.
Revised: It implies that a defense budget twice to the attack budget should be safe enough under the threat of stability attacks.

Original: Theorem 3 also suggests thatthe minimum budget might be much smaller than 2 (cid:15) , and it depends on the speciﬁc attack methodsand data distributions.
Revised: Theorem 3 also suggests that the minimum budget might be much smaller than 2 ϵ , and it depends on the specific attack methods and data distributions.

Original: In the following section, we empirically search for an appropriate defensebudget on real-world datasets.
Revised: In the following section, we empirically search for an appropriate defense budget on real-world datasets.

Original:  Experiments
Revised:  Experiments

Original: In this section, we conduct comprehensive experiments to demonstrate the effectiveness of thehypocritical perturbation as stability attacks on popular benchmark datasets and the necessity of anadaptive defense for better robustness.
Revised: In this section, we conduct comprehensive experiments to demonstrate the effectiveness of the hypocritical perturbation as stability attacks on popular benchmark datasets and the necessity of an adaptive defense for better robustness.

Original: We conduct stability attacks by applying hypocritical perturbations into the training set.
Revised: We conduct stability attacks by applying hypocritical perturbations into the training set.

Original: We focuson an (cid:96) ∞ adversary with an attack budget (cid:15) a = 8 / 255 by following [28, 79, 64, 18].
Revised: We focus on an ℓ ∞ adversary with an attack budget ϵ a = 8 / 255 by following [28, 81, 64, 18].

Original: Our craftingmodel is adversarially trained with a crafting budget (cid:15) c = 2 / 255 for 10 epochs before generatingperturbations.
Revised: Our crafting model is adversarially trained with a crafting budget ϵ c = 2 / 255 for 10 epochs before generating perturbations.

Original: Unless otherwise speciﬁed, we use ResNet-18 [26] as the default architecture for boththe crafting model and the learning model.
Revised: Unless otherwise specified, we use ResNet-18 [26] as the default architecture for both the crafting model and the learning model.

Original: For adversarial training, we mainly follow the settings inprevious studies [81, 69, 50].
Revised: For adversarial training, we mainly follow the settings in [Table] [Table] [Table]

Revised: previous studies [83, 70, 50].

Original: By convention, the defense budget is equal to the attack budget, i.e.,(cid:15) d
Revised: By convention, the defense budget is equal to the attack budget, i [Table]

Original:  = 8 / 255 .
Revised: e., ϵ d = 8 / 255 .

Original: More details on experimental settings are provided in Appendix D.
Revised: More details on experimental settings are provided in Appendix D.

Original: 5.1 Benchmarking (Non-)Robustness
Revised: 5.1 Benchmarking (Non-)Robustness

Original: Attack evaluation.
Revised: Attack evaluation.

Original: We compare our crafted hypocritical perturbation to existing methods, whichwere originally proposed as delusive attacks, including DeepConfuse (which builds an adversarialauto-encoder to generate their perturbations) [16], Unlearnable Examples (which use a min-minbi-level optimization process to pre-train their crafting model) [28], NTGA (which adopts neuraltangent kernels as its crafting model) [79], and Adversarial Poisoning (whose crafting model is simply [Table] [Table]
Revised: We compare our crafted hypocritical perturbation to existing methods, which were originally proposed as delusive attacks, including DeepConfuse (which builds an adversarial auto-encoder to generate their perturbations) [16], Unlearnable Examples (which use a min-min bi-level optimization process to pre-train their crafting model) [28], NTGA (which adopts neural tangent kernels as its crafting model) [81]

Original: [Table]
Revised: [Table]

Original:  a naturally trained classiﬁer)
Revised: and Adversarial Poisoning (whose crafting model is simply a naturally trained classifier)

Original: It is noteworthy that none of these previous works evaluated thetest robustness of their poisoned models.
Revised: It is noteworthy that none of these previous works evaluated the test robustness of their poisoned models.

Original: Results using ResNet-18 on CIFAR-10 are summarized in Table 2.
Revised: Results using ResNet-18 on CIFAR-10 are summarized in Table 2.

Original: “Natural” denotes the accuracyon natural test data.
Revised: “Natural” denotes the accuracy on natural test data.

Original: Various test-time adversarial attacks are used to evaluate test robustness,including FGSM, PGD-20/100, CW ∞ ( (cid:96) ∞ version of CW loss [9] optimized by PGD-100) [Table]
Revised: Various test-time adversarial attacks are used to evaluate test robustness, including

Original:  and AutoAttack (a reliable evaluation metric via an ensemble of diverse attacks [11]).
Revised: FGSM, PGD-20/100, CW ∞ ( ℓ ∞ version of CW loss [9] optimized by PGD-100), and AutoAttack (a reliable evaluation metric via an ensemble of diverse attacks [11]).

Original: We observe thatour implementation of stability attacks widely outperforms previous training-time perturbations indegrading the test robustness of PGD-AT [39].
Revised: We observe that the hypocritical perturbation widely outperforms previous training-time perturbations in degrading the test robustness of PGD-AT [39].

Original: This demonstrates that stability attacks are indeedharmful to the conventional defense scheme.
Revised: This demonstrates that stability attacks are indeed harmful to the conventional defense scheme.

Original: We note that our method increases the natural accuracy.
Revised: We note that our method increases the natural accuracy.

Original: This is reasonable, since our analysis in Section 3.1 has implied that the hypocritical perturbation canincrease model reliance on the non-robust features, which are predictive but brittle [29].
Revised: This is reasonable, since our analysis in Section 3.1 has implied that the hypocritical perturbation can increase model reliance on the non-robust features, which are predictive but brittle [29].

Original: Moreover, we evaluate the hypocritical perturbation on other benchmark datasets including SVHN,
Revised: Moreover, we evaluate the hypocritical perturbation on other benchmark datasets including SVHN,

Original: CIFAR-100, and Tiny-ImageNet.
Revised: CIFAR-100, and Tiny-ImageNet.

Original: Both the crafting model and the victim model use the ResNet-18architecture.
Revised: Both the crafting model and the victim model use the ResNet- [Figure] [Table] [Table]

Original:  Results are summarized in Table 3.
Revised: [Table] [Table] architecture. Results are summarized in Table 3.

Original: “Hyp.” denotes the hypocritical perturbation generated by our crafting model.
Revised: “Hyp.” denotes the hypocritical perturbation generated by our crafting model.

Original: As a comparison, we also evaluate the adversarial perturbation generatedusing the same crafting model (denoted as “Adv.”).
Revised: As a comparison, we also evaluate the adversarial perturbation generated using the same crafting model (denoted as “Adv.”).

Original: Again, the results show that the hypocriticalperturbations are more threatening than the adversarial perturbations to standard adversarial training.
Revised: Again, the results show that the hypocritical perturbations are more threatening than the adversarial perturbations to standard adversarial training.

Original: This phenomenon is consistent with our analytical results in Section 3.1.
Revised: This phenomenon is consistent with our analytical results in Section 3.

Original: Besides, we ﬁnd that the hypocritical perturbation can transfer well from ResNet-18 to other architectures, which reliably degrades the test robustness of a wide variety of popular architectures including
Revised: [Table] Besides [Table] we find that the hypocritical perturbation can transfer well from ResNet-18 to other architectures, successfully degrading the test robustness of a wide variety of popular architectures including

Original: VGG-16 [57], GoogLeNet [61], DenseNet-121 [27], MobileNetV2 [52], and WideResNet-28-10 [80],as shown in Table 4.
Revised: VGG-16 [57], GoogLeNet [61], DenseNet-121 [27], MobileNetV2 [52] [Table] and WideResNet-28-10 [82], as shown in Table 4.

Original: Note that this is a completely black-box setting where the attacker has noknowledge of the victim model’s initialization, architecture, learning rate scheduler, etc.[Figure] [Table] [Table]
Revised: Note that this is a completely black-box setting where the attacker has no knowledge of the victim model’s initialization, architecture, learning rate scheduler, etc.

Original: Adaptive defense.
Revised: Adaptive defense.

Original: To prevent the harm of stability attacks, our analysis in Section 4 suggests thata larger defense budget would be helpful.
Revised: To prevent the harm of stability attacks, our analysis in Section 4 suggests that a larger defense budget would be helpful.

Original: We ﬁnd that this is indeed the case on CIFAR-10.
Revised: We find that this is indeed the case on CIFAR-10.

Original: Asshown in Table [Table] a large defense budget (cid:15) d = 14 / 255 for PGD-AT performs signiﬁcantly betterthan the conventional defense budget (cid:15) d = 8 / 255 .
Revised: As shown in Table 5, a large defense budget ϵ d = 14 / 255 for PGD-AT performs significantly better than the conventional defense budget ϵ d = 8 / 255 .

Original: We also combine several data augmentations with
Revised: We also combine several data augmentations with

Original: PGD-AT as defenses by following Fowl et al.
Revised: PGD-AT as defenses by following Fowl et al.

Original: The results show that they are beneﬁcial, whiletheir improvements are inferior to PGD-AT with (cid:15) d = 14 / 255 .
Revised: The results show that they are beneficial, while their improvements are inferior to PGD-AT with ϵ d = 14 / 255 .

Original: In addition, we adopt other adversarial training variants including TRADES [81] and MART [69] to defend against the hypocriticalperturbation [Table] and ﬁnd that they achieve comparable defense effects with large defense budgets.
Revised: In addition, we adopt other adversarial training variants including TRADES [83] and MART [70] to defend against the hypocritical perturbation, and find that they achieve comparable defense effects with large defense budgets.

Original: Finally, we note that the adaptive defense has several limitations: i) robust accuracy is improved at thecost of natural accuracy; ii) ﬁnding an appropriate defense budget is time-consuming for adversarialtraining; iii) adversarial training with large budgets may lead to learning obstacles such as inherentlarge sample complexity [53].
Revised: Finally, we note that the adaptive defense has several limitations: i) robust accuracy is improved at the cost of natural accuracy; ii) finding an appropriate defense budget is time-consuming for adversarial training; iii) adversarial training with large budgets may lead to learning obstacles such as inherent large sample complexity [53].

Original: We leave the detailed study of these questions as future work.
Revised: We leave the detailed study of these questions as future work.

Original: 5.2 Ablation Studies
Revised: 5.2 Ablation Studies

Original: In this part, we conduct a set of experiments to provide an empirical understanding of the proposedattack.
Revised: In this part, we conduct a set of experiments to provide an empirical understanding of the proposed attack.

Original: We train ResNet-18 using PGD-AT on CIFAR-10 by following the same settings describedin Appendix D unless otherwise speciﬁed.
Revised: We train ResNet-18 using PGD-AT on CIFAR-10 by following the same settings described in Appendix D unless otherwise specified.

Original: Analysis on the crafting method.
Revised: Analysis on the crafting method.

Original: Different from previous work, we use “slightly robust” classiﬁersas our crafting model.
Revised: Different from previous work, we use “slightly robust” classifiers as our crafting model.

Original: Figure 2(a) shows that this technique greatly improves the potency of the attack,where the crafting budget (cid:15) c = 2 / 255 performs best in degrading test robustness.
Revised: Figure 2(a) shows that this technique greatly improves the potency of the hypocritical perturbation, where the crafting budget ϵ c = 2 / 255 performs best in degrading test robustness.

Original: We also observethat training the crafting model for 10 ∼ 80 epochs works well in Figure 2(b), and that optimizing thecrafted perturbations over 100 steps performs well in Figure 2(c).
Revised: We also observe that training the crafting model for 10 ∼ 80 epochs works well in Figure 2(b), and that optimizing the crafted perturbations over 100 steps performs well in Figure 2(c).

Original: Finally, we note that Fowl et al.[18] also tried to use adversarially trained models as the crafting model, but they failed to produce  effective attacks in this way.
Revised: Finally, we note that Fowl et al. [18] also tried to use adversarially trained models as the crafting model, but they failed to produce an effective attack in this way.

Original: This is mainly because they adopted adversarial perturbations as poisons,which, as we observed, are inferior in degrading test robustness.
Revised: This is mainly because they adopted adversarial perturbations as poisons, which, as we observed, are inferior in degrading test performance.

Original: Ablation on defense budget.
Revised: Ablation on defense budget.

Original: As discussed in Section 4, we are motivated to ﬁnd the appropriatedefense budget (cid:15) d in the range [ (cid:15) ∼ 2 (cid:15) ] .
Revised: As discussed in Section 4, we are motivated to find the appropriate defense budget ϵ d in the range [ ϵ ∼ 2 ϵ ] .

Original: Figure 2(d) shows that the optimal defense budgets against theproposed attack for PGD-AT, TRADES, and MART are 14 / 255 , 12 / 255 , and 14 / 255 , respectively.
Revised: Figure 2(d) shows that the optimal defense budgets against the hypocritical perturbation are 14 / 255 , 12 / 255 , and 14 / 255 for PGD-AT, TRADES, and MART, respectively.

Original: We also observe that all these adversarial training variants are inferior when using the conventionaldefense budget 8 / 255 .
Revised: We also observe that all these adversarial training variants are inferior when using the conventional defense budget 8 / 255 .

Original: Less data.
Revised: Less data.

Original: We follow Fowl et al.
Revised: We follow Fowl et al.

Original: [18] to test the effectiveness of attacks by varying the proportionof clean data and perturbed data.
Revised: [18] to test the effectiveness of attacks by varying the proportion of clean data and perturbed data.

Original: Attacks are then considered effective if they cannot signiﬁcantlyincrease performance over training on the clean subset alone.
Revised: Attacks are then considered effective if they cannot significantly increase performance over training on the clean subset alone.

Original: As shown in Table 6, the proposedattack often degrades the test robustness below what one would achieve using full clean dataset.
Revised: As shown in Table 6, the proposed attack often degrades the test robustness below what one would achieve using full clean dataset.

Original: More importantly, the hypocritical perturbations are consistently more harmful than the adversarialperturbations.
Revised: More importantly, the hypocritical perturbations are consistently more harmful than the adversarial perturbations.

Original: This again veriﬁes the superiority of hypocritical pertubrations as stability attacks.
Revised: This again verifies the superiority of hypocritical pertubrations as stability attacks.

Revised: [Table]

Original: Effect on natural training.
Revised: Effect on natural training.

Original: As a sanity check, we include the test accuracy of naturally trainedmodels on CIFAR-10 inAppendix B Table 12.
Revised: As a sanity check, we include the test accuracy of naturally trained models on CIFAR-10 in Table 7.

Original: It shows that without adversarial training, the testrobustness of the models becomes very poor.
Revised: It shows that without adversarial training, the test robustness of the models becomes very poor—all models only have 0% accuracy under PGD-20 attack.

Original: Thus, the goal of stability attacks is immediatelyachieved.
Revised: Thus, the goal of stability attacks is immediately achieved.

Original: On the other hand, We ﬁnd that our HFs can degrade the test accuracy from 94 .
Revised: On the other hand, We find that our method degrades the test accuracy from 94 .

Original: 23%to 75 .
Revised: 23% to 75 .

Original: 92% , though this is not the main focus of this work.
Revised: 92% , though this is not the main focus of this work.

Original: We also observe that Adversarial Poisoning [18] is the most effective method in degrading the test accuracy of naturally trained models.
Revised: We also observe that Adversarial Poisoning [18] is the most effective method in degrading the test accuracy of naturally trained models.

Original: This observation is consistent with Fowl et al.
Revised: This observation is consistent with Fowl et al.

Original:  Related Work
Revised:  Related Work

Original: Adversarial training.
Revised: Adversarial training.

Original: The presence of non-robust features has been demonstrated on popularbenchmark datasets [29, 31], which naturally leads to model vulnerability to adversarial examples [67,58].
Revised: The presence of non-robust features has been demonstrated on popular benchmark datasets [29, 31], which naturally leads to model vulnerability to adversarial examples [67, 58].

Original: To improve test robustness against adversarial examples, adversarial training methods have beendeveloped [23, 39, 72, 81, 65, 46, 74, 84, 62].
Revised: To improve test robustness against adversarial examples, adversarial training methods have been developed [23, 39, 73, 83, 65, 46, 76, 86, 62, 69].

Original: Usually, adversarial training using a defense budget (cid:15)is expected to improve model robustness against (cid:15) -bounded adversarial examples.
Revised: Usually, adversarial training using a defense budget ϵ is expected to improve model robustness against ϵ -bounded adversarial examples.

Original: Thus, to break thisdefense, a direct way is to enlarge the typical (cid:15) -ball used to constrain the attack; however, this mayrisk changing the true label [8, 66].
Revised: Thus, to break this defense, a direct way is to enlarge the typical ϵ -ball used to constrain the attack; however, this may risk changing the true label [8, 66].

Original: In this work, we aim to show that it is possible to achieve this byslightly perturbing the training data without enlarging the (cid:15) -ball.
Revised: In this work, we aim to show that it is possible to achieve this by slightly perturbing the training data without enlarging the ϵ -ball.

Original: Data poisoning.
Revised: Data poisoning.

Original: Data poisoning attacks, which manipulate training data to cause the resultingmodels to fail during inference [3], can be divided into availability attacks (to degrade overalltest performance)
Revised: Data poisoning attacks, which manipulate training data to cause the resulting models to fail during inference [3], can be divided into availability attacks (to degrade overall test performance)

Original: [4, 75, 41, 47, 18] and integrity attacks (to cause speciﬁc misclassiﬁcations)
Revised: [4, 77, 41, 47, 18] and integrity attacks (to cause specific misclassifications) [32, 10, 55, 87, 21, 54].

Original: While the stability attacks considered in this work may be reminiscent ofbackdoor attacks [10], we note that they share several key differences.
Revised: While the stability attacks studied in this work may be reminiscent of backdoor attacks [10], we note that they share several key differences.

Original: First, stability attacksaim to hinder adversarial training with well-deﬁned (cid:15) -robustness, while backdoor attacks mainlyfocus on embedding exploits (that can be invoked by pre-speciﬁed triggers) into naturally trainedmodels [22, 51, 68].
Revised: First, stability attacks aim to hinder adversarial training with well-defined ϵ -robustness, while backdoor attacks mainly focus on embedding malicious behaviors (that can be invoked by pre-specified triggers) into naturally trained models [22, 51, 68].

Original: Second, stability attacks only perturb the inputs slightly, while many works onbackdoor attacks require mislabeling [25, 37, 45, 36].
Revised: Second, stability attacks only perturb the inputs slightly, while most works on backdoor attacks require mislabeling [25, 37, 45, 36, 74].

Original: Thus, backdoor defenses [7, 73, 35] might notbe directly applied to resist stability attacks.
Revised: Thus, backdoor defenses [7, 75, 35] might not be directly applied to resist stability attacks.

Original: Additional related works are discussed in Appendix A.
Revised: Additional related works are discussed in Appendix A.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we establish a framework to study the robustness of adversarial training against stabilityattacks.
Revised: In this work, we establish a framework to study the robustness of adversarial training against stability attacks.

Original: We unveil the threat of stability attacks—small hypocritical perturbations applied intothe training data sufﬁce to hinder conventional adversarial training.
Revised: We unveil the threat of stability attacks—small hypocritical perturbations applied into the training data suffice to hinder conventional adversarial training.

Original: The conventional defensebudget (cid:15) is insufﬁcient under the threat.
Revised: The conventional defense budget ϵ is insufficient under the threat.

Original: To resist it, we suggest that practitioners should consider alarger defense budget of no more than 2 (cid:15) .
Revised: To resist it, we suggest a larger defense budget of no more than 2 ϵ .

Original: Our theoretical analysisalso explains why hypocriticalperturbations work as stability attacks—they can reinforce the non-robust features and misleadthe learner.
Revised: Our theoretical analysis explains why hypocritical perturbations are effective as stability attacks— they can mislead the learner by reinforcing the non-robust features.

Original: Experiments demonstrate that hypocritical perturbations are harmful to conventionaladversarial training on benchmark datasets, and enlarging the defense budget is essential for mitigatinghypocritical pertubrations.
Revised: Experiments demonstrate that hypocritical perturbations are harmful to conventional adversarial training on benchmark datasets, and enlarging the defense budget is essential for mitigating hypocritical pertubrations.

Original: Future work includes relaxing the assumption that the adversary perturbsall the entire training set and designing more effective stability attacks against adversarial training.Checklist For all authors...
Revised: Future work includes relaxing the assumption that the adversary perturbs the entire training set and designing more effective stability attacks against adversarial training.

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 2.2 and Section 7.(c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Appendix F.(d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: [Table]

Original: results (either in the supplemental material or as a URL)?

Original: [Yes] See the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5.1.

Original: [Table]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments

Original: [Table]

Original: multiple times)?

Original: [Yes] See Appendix B. [Table] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, [Table]
Revised: Acknowledgments and Disclosure of Funding

Original: internal cluster, or cloud provider)?

Original: [Yes] All experiments are run on a single NVIDIA GeForce

Original: [Table]

Original: RTX 3090 GPU.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... [Table] (a) If your work uses existing assets, did you cite the creators?
Revised: This work was supported by the National Natural Science Foundation of China (Grant No. 61732006, 62076124, 62076128, 62106028), the National Key R&D Program of China (2020AAA0107000).

Original: [Yes]

Original: [Table]

Original: (b) Did you mention the license of the assets? [N/A] [Table] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: Lei Feng was also supported by Chongqing Overseas Chinese Entrepreneurship, Innovation Support

Original: [N/A]

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-

Original: ing/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: informa- tion or offensive content? [N/A] If you used crowdsourcing or conducted research with human subjects...
Revised: Program and CAAI-Huawei MindSpore Open Fund.

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals,
Revised: The authors wish to thank the anonymous reviewers for their helpful comments and suggestions.

Original: if applicable

Original: [Table]

Original: [Table]

Original: [Table]

Original: ](c

Original: [Table]

Original: Did you include the

Original: [Table]

Original: paid to participants

Original: [Table]

Original: the

Original: [Table]

Original: onparticipant compensation

Original: [Table]

Original: [Table]

Original:  A Additional Related Work
Revised: ? A Additional Related Work

Original: In this part, we discuss several independent (or concurrent) works that are  related to this work.
Revised: In this part, we discuss several independent (or concurrent) works that are closely related to this work.

Original: Zhu et al.
Revised: Zhu et al.

Original: [86] study the effect of conventional adversarial training on differentiating noisy labels, while Zhang549 et al.
Revised: [88] study the effect of conventional adversarial training on differentiating noisy labels, while Zhang et al.

Original: [83] show that deliberately injected noisy labels may serve as a regularization that alleviates robustoverﬁtting.
Revised: [85] show that deliberately injected noisy labels may serve as a regularization that alleviates robust overfitting.

Original: Our results focus on the clean-label setting and provide evidence that conventional adversarial551 training can be hindered without modifying the labels.
Revised: Our results focus on the clean-label setting and provide evidence that conventional adversarial training can be hindered without modifying the labels.

Original: Yu et al.
Revised: Yu et al.

Original: [78] suggest explaining the success of availability attacks from the perspective of shortcuts.
Revised: [80] suggest explaining the success of availability attacks from the perspective of shortcuts.

Original: They further553 adopt pre-trained models to extract useful features for mitigating model reliance on the shortcuts.
Revised: They further adopt pre-trained models to extract useful features for mitigating model reliance on the shortcuts.

Original: This directionis orthogonal to ours.
Revised: This direction is orthogonal to ours.

Original: Liu et al.
Revised: Liu et al.

Original: [38] improve the effectiveness of unlearnable examples [28] by generating grayscale perturbations556 and using data augmentations.
Revised: [38] improve the effectiveness of unlearnable examples [28] by generating grayscale perturbations and using data augmentations.

Original: They also conclude that conventional adversarial training will prevent a dropin accuracy measured both on clean images and adversarial images.
Revised: They also conclude that conventional adversarial training will prevent a drop in accuracy measured both on clean images and adversarial images.

Original: Contrary to them, we show that, both558 theoretically and empirically, conventional adversarial training can be hindered by hypocritical perturbations,and we further analyze the necessity of enlarging the defense budget to resist stability attacks.
Revised: Contrary to them, we show that, both theoretically and empirically, conventional adversarial training can be hindered by hypocritical perturbations, and we further analyze the necessity of enlarging the defense budget to resist stability attacks.

Original: Gao et al.
Revised: Gao et al.

Original: [20] revisit the trade-off between adversarial robustness and backdoor robustness [71].
Revised: [20] revisit the trade-off between adversarial robustness and backdoor robustness [72].

Original: They conclude561 that backdoor attacks are ineffective when the defense budget of adversarial training surpasses the triggermagnitude.
Revised: They conclude that backdoor attacks are ineffective when the defense budget of adversarial training surpasses the trigger magnitude.

Original: In contrast, our results indicate that stability attacks are still harmful to adversarial training when the563 defense budget is not large enough.
Revised: In contrast, our results indicate that stability attacks are still harmful to adversarial training when the defense budget is not large enough.

Original: In a simple statistical setting, a defense budget (cid:15) + η is necessary (where ηis a positive number).
Revised: In a simple statistical setting, a defense budget ϵ + η is necessary (where η is a positive number).

Original: In the general case, a defense budget of 2 (cid:15) is sufﬁcient.
Revised: In the general case, a defense budget of 2 ϵ is sufficient.

Original: In our experiments, a defense565 budget of about 1 . 5 (cid:15) ∼ 1 .
Revised: In our experiments, a defense budget of about 1 . 5 ϵ ∼ 1 .

Original: 75 (cid:15) provides the best empirical (cid:15) -robustness.
Revised: 75 ϵ provides the best empirical ϵ -robustness.

Original: Wang et al.
Revised: Wang et al.

Original: [70] argue that it is necessary to use robust features for compromising adversarial training.
Revised: [71] argue that it is necessary to use robust features for compromising adversarial training.

Original: To this567 end, they adopt a relatively large attack budget (cid:15) a = 32 / 255 for crafting their poisons (they use one type ofadversarial perturbations), and show that their poisons can decrease the performance of the models trained using569 smaller defense budgets (such as (cid:15) d = 8 / 255 and (cid:15) d = 16 / 255 ).
Revised: To this end, they adopt a relatively large attack budget ϵ a = 32 / 255 for crafting their poisons (they use one type of adversarial perturbations), and show that their poisons can decrease the performance of the models trained using smaller defense budgets (such as ϵ d = 8 / 255 and ϵ d = 16 / 255 ).

Original: In contrast, we focus on a more realisticsetting that does not require a larger attack budget.
Revised: In contrast, we focus on a more realistic setting that does not require a larger attack budget.

Original: We demonstrate that it is possible to hinder adversarial571 training when (cid:15) a = (cid:15) d .
Revised: We demonstrate that it is possible to hinder adversarial training when ϵ a = ϵ d .

Original: Furthermore, we provide both theoretical and empirical results showing how to adapt the572 defense to maintain robustness.
Revised: Furthermore, we provide both theoretical and empirical results showing how to adapt the defense to maintain robustness.

Original: Fu et al.
Revised: Fu et al.

Original: [19] explore how to protect data privacy against adversarial training.
Revised: [19] explore how to protect data privacy against adversarial training.

Original: The main purpose of their poisons574 is to compromise adversarial training by requiring the perturbation budget of their poisons to be larger than thatof adversarial training.
Revised: The main purpose of their poisons is to compromise adversarial training by requiring the perturbation budget of their poisons to be larger than that of adversarial training.

Original: In this way, they show that the natural accuracy of the adversarially trained models can576 be largely decreased, let alone robust accuracy.
Revised: In this way, they show that the natural accuracy of the adversarially trained models can be largely decreased, let alone robust accuracy.

Original: From this perspective, our work is complementary to theirs.
Revised: From this perspective, our work is complementary to theirs.

Original: Wepursue to not increase the attack budget of stability attacks, keeping it as small as possible.
Revised: We pursue to not increase the attack budget of stability attacks, keeping it as small as possible.

Original: We successfully578 demonstrate that stability attacks are still harmful to conventional adversarial training without enlarging theattack budget.
Revised: We successfully demonstrate that stability attacks are still harmful to conventional adversarial training without enlarging the attack budget.

Original: This makes the threat of stability attacks more insidious than that of Fu et al.
Revised: This makes the threat of stability attacks more insidious than that of Fu et al.

Original: On the other hand, we ﬁnd that our implementation of stability attacks using hypocritical perturbations has some581 similarities to the robust unlearnable examples in Fu et al.
Revised: On the other hand, we find that our implementation of stability attacks using hypocritical perturbations has some similarities to the robust unlearnable examples in Fu et al.

Original: Speciﬁcally, although the robust unlearnableexamples are generated via a complicated min-min-max optimization process [19], we notice that their noise583 generator can be viewed as an adversarially trained model.
Revised: Specifically, although the robust unlearnable examples are generated via a complicated min-min-max optimization process [19], we notice that their noise generator can be viewed as an adversarially trained model.

Original: This implies that the robust error-minimizing (REM)noise [19] might be useful in demonstrating the feasibility of stability attacks.
Revised: This implies that the robust error-minimizing (REM) noise [19] might be useful in demonstrating the feasibility of stability attacks.

Original: To verify this, we run the source585 code from the authors with default hyperparameters 2 , and compare our crafted hypocritical perturbation withtheir generated noise under the setting of stability attacks.
Revised: To verify this, we run the source code from the authors with default hyperparameters, and compare our crafted hypocritical perturbation with their generated noise under the setting of stability attacks.

Original: For a fair comparison, here we apply a very simple587 trick called EOT [2] in our method, since the trick is also used by REM [19].
Revised: For a fair comparison, here we apply a very simple trick called EOT [2] in our method, since the trick is also used by REM [19].

Original: The additional time cost of the
Revised: The additional time cost of the

Original: EOT trick is very small and negligible.
Revised: EOT trick is very small and negligible.

Original: Our experimental results, shown in Table 7, demonstrate that the robust error-minimizing noise is also effective590 as stability attacks, though it was originally proposed as a delusive attack.
Revised: Our experimental results, shown in Table 8, demonstrate that the robust error-minimizing noise is also effective as stability attacks, though it was originally proposed as a delusive attack.

Original: It is noteworthy that the robustaccuracy is not evaluated in [19].
Revised: It is noteworthy that the robust accuracy is not evaluated in [19].

Original: In this sense, the effectiveness of REM as an stability attack can be regarded as592 one of our novel ﬁndings.
Revised: In this sense, the effectiveness of REM as an stability attack can be regarded as one of our novel findings.

Original: Importantly, our method outperforms REM in terms of the robust accuracy against AutoAttack.
Revised: Importantly, our method outperforms REM in terms of the robust accuracy against AutoAttack.

Original: Since AutoAttack594 is the most reliable evaluation metric of model robustness among the test-time attacks [11], this result indicatesthat our method is reliably more effective than REM in degrading model robustness.
Revised: Since AutoAttack is the most reliable evaluation metric of model robustness among the test-time attacks [11], this indicates that our method is reliably more effective than REM in degrading model robustness.

Original: It is also noteworthy that596 our method is signiﬁcantly more efﬁcient than REM, as shown in Table 8.
Revised: It is also noteworthy that our method is significantly more efficient than REM, as shown in Table 9.

Original: We note that the efﬁciency of ourmethod is largely due to the fact that our crafting model is fast to train.
Revised: We note that the efficiency of our method is largely due to the fact that our crafting model is fast to train.

Original: Speciﬁcally, the time cost of training our598 crafting model is only 0.3 hours, while it takes 20.8 hours for REM.
Revised: Specifically, the time cost of training our crafting model is only 0.3 hours, while it takes 20.8 hours for REM.

Original: That is, our crafting model is nearly 70times faster to train than that of REM.
Revised: That is, our crafting model is nearly 70 times faster to train than that of REM.

Original: In short, our method is not only more effective, but also more efﬁcient, 600 than REM as a stability attack.
Revised: In short, our method is not only more effective, but also more efficient, than REM as a stability attack.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B Omitted Tables
Revised: B Omitted Tables

Original: [Table] [Table]
Revised: [Table] [Table]

Original: various adaptive defenses on the hypocritically perturbed CIFAR-10.
Revised: CIFAR-10.

Original: We report mean and standard deviation over 3 random runs
Revised: We report mean and standard deviation over 3 random runs

Original: [Table]
Revised: [Table]

Original: [Table]

Original: C Proofs
Revised: C Proofs

Original: [Table]

Original: In this section, we provide the proofs of our theoretical results in Section 3 and Section 4.
Revised: In this section, we provide the proofs of our theoretical results in Section 3 and Section 4.

Original: C.1 Proof of Proposition
Revised: C.1 Proof of Proposition

Original: Proposition 1 (restated).
Revised: Proposition 1 (restated).

Original: Let (cid:15) = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of a 606 classiﬁer correctly predicting y on the data (7) under (cid:96) ∞ perturbations.
Revised: Let ϵ = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of a classifier correctly predicting y on the data (7) under ℓ ∞ perturbations.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recalling that in Equation (8), we have the natural classiﬁer:f nat ( x ) :
Revised: Recalling that in Equation (8), we have the natural classifier: [Equation]

Original: = sign( w (cid:62) nat x ) , where w nat :=
Revised: := sign( w [Equation] x ) , where w nat :=

Original: [1 , η, . . .
Revised: [1 , η, . . .

Original: , η ] ,(12) and in Equation (9), the robust classiﬁer is deﬁned as: [Equation] := sign( w (cid:62)
Revised: , η ] ,

Original: [Equation]
Revised: [Equation]

Original: x ) ,where w rob := Then, the adversarial accuracy of the natural classiﬁer on the data D (7) is A
Revised: 12) and in Equation (9) , the robust classifier is defined as:

Original: [Equation]
Revised: [Equation]

Original: (cid:34) [Equation] [Equation] i =
Revised: , where w rob :=

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:1)(cid:33)(cid:35)

Original: [Equation]

Original: [Equation]

Original: ≤

Original: [Equation]
Revised: [Equation]

Original: Pr (cid:40) y · (cid:32) 1 · (cid:0)

Original: [Equation]

Original: ( y

Original: [Equation]

Original: ) (cid:1) [Equation] i = η · (cid:0) N ( yη [Equation]
Revised: Then, the adversarial accuracy of the natural classifier on the data D (7) is

Original: ) − (cid:15) (cid:1)(cid:33)

Original: [Equation]

Original: (cid:41) =

Original: 1 − Pr (cid:40)

Original: N (

Original: [Equation]
Revised: [Equation]

Original: σ 2 ) + η d +1 (cid:88) i = N ( η − (cid:15) [Equation] σ 2 ) <(cid:41) =
Revised: Similarly

Original: Pr (cid:40)

Original: N (

Original: [Equation]
Revised: [Equation]

Original: σ 2 ) + η d +1 (cid:88) i =

Original: N ( η − (cid:15), σ 2 ) >(cid:41) =

Original: Pr (cid:40)

Original: N (0 , 1) < 1 − dησ (cid:112) 1 + dη(cid:41) .

Original: Similarly, the adversarial accuracy of the robust classiﬁer on the data D (7) is
Revised: the adversarial accuracy of the robust classifier on the data D (7) is

Original: [Equation]
Revised: [Equation]

Original: C.2 Proof of Theorem1
Revised: C.2 Proof of Theorem

Original: The following theorems rely on the analytical solution of optimal linear (cid:96) ∞ -robust classiﬁer on mixture Gaussiandistributions.
Revised: The following theorems rely on the analytical solution of optimal linear ℓ ∞ -robust classifier on mixture Gaussian distributions.

Original: Concretely, the optimization problem is to minimize the adversarial risk on a distribution (cid:98) D with a616 defense budget ˆ (cid:15) :
Revised: Concretely, the optimization problem is to minimize the adversarial risk on a distribution (cid:98) D with a defense budget ˆ ϵ :

Original: [Equation]
Revised: [Equation]

Original: where Rˆ (cid:15) adv ( f, (cid:98) D ) : = E
Revised: where R

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) = sign( w (cid:62) x + b ) , and 1 ( · ) denotes the indicator function.
Revised: where f ( x ) = sign( w ⊤ x + b ) , and 1 ( · ) denotes the indicator function.

Original: We note that optimal linear robust classiﬁers have been obtained for certain data distributions in previous619 work [67, 29, 14, 30, 76, 64].
Revised: We note that optimal linear robust classifiers have been obtained for certain data distributions in previous work [67, 29, 14, 30, 78, 64].

Original: Here, our goal is to establish similar optimal linear robust classiﬁers for theclassiﬁcation tasks in our setting.
Revised: Here, our goal is to establish similar optimal linear robust classifiers for the classification tasks in our setting.

Original: We only employ linear classiﬁers, since it is highly nontrivial to consider621 non-linearity for adversarial training on mixture Gaussian distributions [14].
Revised: We only employ linear classifiers, since it is highly nontrivial to consider non-linearity for adversarial training on mixture Gaussian distributions [14].

Original: Lemma 1.
Revised: Lemma 1.

Original: Assume that the adversarial perturbation in data T adv (10) is moderate such that η/ 2 ≤ (cid:15) < 1 / 2 .
Revised: Assume that the adversarial perturbation in data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .

Original: Then, minimizing the adversarial risk (16) on the data T adv with a defense budget (cid:15) can result in a classiﬁer thatassigns 0 weight to the features x i for i ≥ 2 .
Revised: Then, minimizing the adversarial risk (16) on the data T adv with a defense budget ϵ can result in a classifier that assigns 0 weight to the features x i for i ≥ 2 .

Original: Proof.
Revised: Proof.

Original: We prove the lemma by contradiction.
Revised: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T adv , which can be written as follows:
Revised: The goal is to minimize the adversarial risk on the distribution T adv , which can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: δ )] > 0 | [Equation] −(cid:27) · Pr
Revised: Consider an optimal solution w in which w i > 0 for some i ≥

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation]

Original: y )

Original: ∼T adv

Original: [Equation]

Original: y

Original: [Equation]

Original: − 1 } + Pr [Equation] ∼T adv
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: min [Equation] δ (cid:107) ∞ [Equation]
Revised: we have

Original: (cid:15)

Original: [ f ( x + δ )] [Equation] | y [Equation]
Revised: R ϵ adv ( f, T ( − 1) adv ) =

Original: + (cid:27) ·

Original: Pr ( x [Equation] y )
Revised: Pr 

Original: ∼T adv { y = +1 }

Original: = Pr (cid:40) max (cid:107) δ (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:15) (cid:34) w 1 ( N ( (cid:15) −

Original: [Equation] σ 2 ) + δ 1 ) + d +1 (cid:88) i = w i ( N ( (cid:15) − η
Revised:   

Original: [Equation]
Revised: [Equation]

Original: σ 2 ) + δ i ) + b (cid:35) > (cid:41) (cid:124) (cid:123)(cid:122) (cid:125) R (cid:15) adv ( f
Revised: (18) where w nat :=

Original: [Equation]

Original: [Equation]

Original: 1) adv ) · 1+ Pr (cid:40) min (cid:107) [Equation] (cid:107) [Equation]
Revised: [1 , η, . . .

Original: (cid:15) (cid:34)

Original: w

Original: [Equation]

Original: ( N (1 − (cid:15)

Original: [Equation] σ [Equation] [Equation]
Revised: , η ] as in Equation (8).

Original: + d [Equation] (cid:88) i = w i ( N ( η − (cid:15), σ 2 ) + δ i) + b (cid:35)
Revised: Since w i > 0 , B is maximized when δ i

Original: [Equation] (cid:41) (cid:124) (cid:123)(cid:122) (cid:125) R (cid:15) adv ( f, T (+1) adv ) · 1(17) Consider an optimal solution w in which w i > 0 for some
Revised: =

Original: i ≥ 2 .Then, we have [Equation] 18) where w nat :=
Revised: ϵ .

Original: [1 , η, . . . , η ] as in Equation (8). Since w i > 0 , B is maximized when δ i = (cid:15) . Thus, thecontribution of terms depending on w i to B is a normally-distributed random variable with mean 2 (cid:15) − η .
Revised: Thus, the contribution of terms depending on w i to B is a normally-distributed random variable with mean 2 ϵ − η .

Original: Since630 2 (cid:15) − η ≥ 0 , setting w i to zero can only decrease the risk.
Revised: Since 2 ϵ − η ≥ 0 , setting w i to zero can only decrease the risk.

Original: This contradicts the optimality of w .
Revised: This contradicts the optimality of w .

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: We can also assume w i < 0 and similar contradiction holds.
Revised: We can also assume w i < 0 and similar contradiction holds.

Original: Therefore, minimizing the adversarial risk on T adv632 leads to w i = 

Revised: Therefore, minimizing the adversarial risk on T adv leads to w i =

Original: for i ≥ 2 .
Revised: for i ≥ 2 .

Original: Lemma 2.
Revised: Lemma 2.

Original: Assume that the adversarial perturbation in data T adv (10) is moderate such that η/ 2 ≤ (cid:15) < 1 / 2 .
Revised: Assume that the adversarial perturbation in data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .

Original: Then, minimizing the adversarial risk (16) on the data T adv with a defense budget (cid:15) results in a classiﬁer thatassigns a positive weight to the feature x 1 .
Revised: Then, minimizing the adversarial risk (16) on the data T adv with a defense budget ϵ results in a classifier that assigns a positive weight to the feature x 1 .

Original: Proof.
Revised: Proof.

Original: We prove the lemma by contradiction.
Revised: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T adv , which has been written in Equation (17).
Revised: The goal is to minimize the adversarial risk on the distribution T adv , which has been written in Equation (17).

Original: Consider an optimal solution w in which w 1 ≤ 0 .
Revised: Consider an optimal solution w in which w 1 ≤ 0 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Since w 1 ≤ 0 , D is maximized when δ 1 = − (cid:15) .
Revised: Since w 1 ≤ 0 , D is maximized when δ 1 = − ϵ .

Original: Thus, the contribution of the term depending on w 1 to D is a640 normally-distributed random variable with mean − 1 .
Revised: Thus, the contribution of the term depending on w 1 to D is a normally-distributed random variable with mean − 1 .

Original: Since the mean is negative, setting w 1 to be positive candecrease the risk.
Revised: Since the mean is negative, setting w 1 to be positive can decrease the risk.

Original: This contradicts the optimality of w .
Revised: This contradicts the optimality of w .

Original: Formally,
Revised: Formally,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where p > 0 is any positive number.
Revised: where p > 0 is any positive number.

Original: Therefore, minimizing the adversarial risk on T adv leads to w 1 > 0 .
Revised: Therefore, minimizing the adversarial risk on T adv leads to w 1 > 0 .

Original: Theorem 1 (restated).
Revised: Theorem 1 (restated).

Original: Assume that the adversarial perturbation in the training data T adv (10) is moderate such644 that η/ 2 ≤ (cid:15) < 1 / 2 .
Revised: Assume that the adversarial perturbation in the training data T adv (10) is moderate such that η/ 2 ≤ ϵ < 1 / 2 .

Original: Then, the optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizing the adversarial riskon T adv with a defense budget (cid:15) is equivalent to the robust classiﬁer (9).
Revised: Then, the optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on T adv with a defense budget ϵ is equivalent to the robust classifier (9).

Original: Proof.
Revised: Proof.

Original: By Lemma 1 and Lemma 2, we have w 1 > 0 and w i = 

Revised: By Lemma 1 and Lemma 2, we have w 1 > 0 and w i =

Original: ( i ≥ 2 ) for an optimal linear (cid:96) ∞ -robust 647 classiﬁer.
Revised: ( i ≥ 2 ) for an optimal linear ℓ ∞ -robust classifier.

Original: Then, the adversarial risk on the distribution T adv can be simpliﬁed by solving the inner maximizationproblem ﬁrst.
Revised: Then, the adversarial risk on the distribution T adv can be simplified by solving the inner maximization problem first.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to the natural risk on a mixture Gaussian distribution D tmp : x ∼ N ( y · µ tmp , σ 2 I ) , whereµ tmp = (1 − 2 (cid:15), 0 , .. . , 0) .
Revised: which is equivalent to the natural risk on a mixture Gaussian distribution D tmp : x ∼ N ( y · µ tmp , σ 2 I ) , where µ tmp = (1 − 2 ϵ, 0 , .

Original: We note that the Bayes optimal classiﬁer for D tmp is f tmp ( x ) = sign( µ (cid:62) tmp x ) .
Revised: We note that the Bayes optimal classifier for D tmp is f tmp ( x ) = sign( µ ⊤ tmp x ) .

Original: Speciﬁcally, the natural risk
Revised: Specifically, the natural risk

Original: [Equation]
Revised: [Equation]

Original: which is minimized when w 1 = 1 − 2 (cid:15) > 0 and b = 0 .
Revised: which is minimized when w 1 = 1 − 2 ϵ > 0 and b = 0 .

Original: That is, minimizing the adversarial risk R (cid:15) adv ( f, T adv )653 can lead to an optimal linear (cid:96) ∞ -robust classiﬁer f tmp ( x ) .
Revised: That is, minimizing the adversarial risk R ϵ adv ( f, T adv ) can lead to an optimal linear ℓ ∞ -robust classifier f tmp ( x ) .

Original: Meanwhile, f tmp ( x ) is equivalent to the robust 654 classiﬁer (9).
Revised: Meanwhile, f tmp ( x ) is equivalent to the robust classifier (9).

Original: This concludes the proof of the theorem.
Revised: This concludes the proof of the theorem.

Original: C.3 Proof of Theorem2
Revised: C.3 Proof of Theorem

Original: Lemma 3.
Revised: Lemma 3.

Original: Minimizing the adversarial risk (16) on the data T hyp (11) with a defense budget (cid:15) results in a 657 classiﬁer that assigns positive weights to the features x i for i ≥ 1 .
Revised: Minimizing the adversarial risk (16) on the data T hyp (11) with a defense budget ϵ results in a classifier that assigns positive weights to the features x i for i ≥ 1 .

Original: Proof.
Revised: Proof.

Original: We prove the lemma by contradiction.
Revised: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T hyp , which can be written as follows:
Revised: The goal is to minimize the adversarial risk on the distribution T hyp , which can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: Consider an optimal solution w in which w i ≤ 0 for some i ≥ 1 .
Revised: Consider an optimal solution w in which w i ≤ 0 for some i ≥ 1 .

Original: Then, we have
Revised: Then, we have

Revised: R ϵ adv

Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Original: ) where w nat :=
Revised: 25) where w nat :=

Original: [1 , η, . . .
Revised: [1 , η, . . .

Original: , η ] as in Equation (8).
Revised: , η ] as in Equation (8).

Original: Since w i ≤ 0 , H is maximized when δ i = − (cid:15) .
Revised: Since w i ≤ 0 , H is maximized when δ i = − ϵ .

Original: Thus, thecontribution of terms depending on w i to H is a normally-distributed random variable with mean − [ w nat ]
Revised: Thus, the contribution of terms depending on w i to H is a normally-distributed random variable with mean − [ w nat ]

Original: i − 2 (cid:15) .
Revised: i − 2 ϵ .

Original: 663 Since the mean is negative, setting w i to be positive can decrease the risk.
Revised: Since the mean is negative, setting w i to be positive can decrease the risk.

Original: This contradicts the optimality of w .
Revised: This contradicts the optimality of w .

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: where p > 0 is any positive number.
Revised: where p > 0 is any positive number.

Original: Therefore, minimizing the adversarial risk on T hyp leads to 
Revised: Therefore, minimizing the adversarial risk on T hyp leads to w i > 0 for i ≥ 1 .

Original: Theorem 2 (restated).
Revised: Theorem 2 (restated).

Original: The optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizing the adversarial risk on668 the perturbed  T hyp (11) with a defense budget (cid:15) is equivalent to the natural classiﬁer (8).
Revised: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ is equivalent to the natural classifier (8).

Original: Proof.
Revised: Proof.

Original: By Lemma 3, we have w i > 0 for i ≥ 1 for an optimal linear (cid:96) ∞ -robust classiﬁer.
Revised: By Lemma [Equation] we have w i > 0 for i ≥ 1 for an optimal linear ℓ ∞ -robust classifier.

Original: Then, we have
Revised: Then [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: (27) which is equivalent to the natural risk on the mixture Gaussian distribution D : x ∼ N ( y · w nat , σ 2 I ) , wherew nat =(1 , η, . . .
Revised: 27) which is equivalent to the natural risk on the mixture Gaussian distribution D : x ∼ N ( y · w nat , σ 2 I ) , where w nat =

Original:  , η ) .
Revised: (1 , η, . . . , η ) .

Original: We note that the Bayes optimal classiﬁer for D is f nat ( x ) = sign( w (cid:62) nat x ) .
Revised: We note that the Bayes optimal classifier for D is f nat ( x ) = sign( w ⊤ nat x ) .

Original: Speciﬁcally, 672 the natural risk
Revised: Specifically, the natural risk

Original: [Equation]
Revised: [Equation]

Original: 1 2 , (28) which is minimized when w 1 = 1 , w i =
Revised: which is minimized when w 1 = 1 , w i =

Original: η for i ≥ 2 , and b = 0 .
Revised: η for i ≥ 2 , and b = 0 .

Original: That is, minimizing the adversarial risk674 R (cid:15) adv ( f, T hyp ) can lead to an optimal linear (cid:96) ∞ -robust classiﬁer f nat ( x ) , which is equivalent to the natural 675 classiﬁer (8).
Revised: That is, minimizing the adversarial risk R ϵ adv ( f, T hyp ) can lead to an optimal linear ℓ ∞ -robust classifier f nat ( x ) , which is equivalent to the natural classifier (8).

Original: This concludes the proof of the theorem.
Revised: This concludes the proof of the theorem.

Original: C.4 Proof of Theorem
Revised: C.4 Proof of Theorem

Original: Lemma 4.
Revised: Lemma 4.

Original: Minimizing the adversarial risk (16) on the data T hyp (11) with a defense budget (cid:15) + η can result in679 a classiﬁer that assigns 0 weight to the features x i for i ≥ 2 .
Revised: Minimizing the adversarial risk (16) on the data T hyp (11) with a defense budget ϵ + η can result in a classifier that assigns 0 weight to the features x i for i ≥ 2 .

Original: Proof.
Revised: Proof.

Original: The goal is to minimize the adversarial risk on the distribution T hyp , which can be written as follows:
Revised: The goal is to minimize the adversarial risk on the distribution T hyp , which can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: Consider an optimal solution w in which w i > 0 for some

Original: [Table] [Equation] [Table] [Equation]
Revised: i ≥ 2 .

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: Then, we have

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: R

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: +

Original: [Table] [Equation]Consider an optimal solution w in which w i > 0 for some i ≥ 2 .
Revised: [Equation]

Original: Then, we have
Revised: adv ( f, T

Original: [Equation]
Revised: [Equation]

Revised: 1) hyp

Revised: [Equation]

Original: ) where w nat :=
Revised: = Pr [Equation] 30) where w nat :=

Original: [1 , η, . . .
Revised: [1 , η, . . .

Original: , η ] .
Revised: , η ] .

Original: Since w i > 0 , J is maximized when δ i = (cid:15) + η .
Revised: Since w i > 0 , J is maximized when δ i = ϵ + η .

Original: Thus, the contribution of termsdepending on w i to J is a normally-distributed random variable with mean 0 .
Revised: Thus, the contribution of terms depending on w i to J is a normally-distributed random variable with mean 0 .

Original: Thus, setting w i to zero will not684 increase the risk.
Revised: Thus, setting w i to zero will not increase the risk.

Original: Formally, we have
Revised: Formally, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We can also assume w i < 0 and a similar argument holds.
Revised: We can also assume w i < 0 and a similar argument holds.

Original: Similar arguments also hold for R (cid:15) + η adv ( f [Equation] T (+1) hyp ) .
Revised: Similar arguments also hold for R ϵ + η adv ( f, T (+1) hyp ) .

Original: Therefore, minimizing the adversarial risk on T hyp can lead to w i = 

Revised: Therefore, minimizing the adversarial risk on T hyp can lead to w i =

Original: for i ≥ 2 .
Revised: for i ≥ 2 .

Original: Theorem 3 (restated).
Revised: Theorem 3 (restated).

Original: The optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizing the adversarial risk on688 the perturbed data T hyp (11) with a defense budget (cid:15) + η is equivalent to the robust classiﬁer (9).
Revised: The optimal linear ℓ ∞ -robust classifier obtained by minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget ϵ + η is equivalent to the robust classifier (9).

Original: Moreover, any689 defense budget lower than (cid:15) + η will yield classiﬁers that still rely on all the non-robust features.
Revised: Moreover, any defense budget lower than ϵ + η will yield classifiers that still rely on all the non-robust features.

Original: Proof.
Revised: Proof.

Original: By Lemma 4, we have w i = 

Revised: By Lemma 4, we have w i =

Original: ( i ≥ 2 ) for an optimal linear (cid:96) ∞ -robust classiﬁer.
Revised: ( i ≥ 2 ) for an optimal linear ℓ ∞ -robust classifier.

Original: Also, the robust 691 classiﬁer will assign a positive weight to the ﬁrst feature.
Revised: Also, the robust classifier will assign a positive weight to the first feature.

Original: This is similar to the case in Lemma 2 and we omit theproof here.
Revised: This is similar to the case in Lemma 2 and we omit the proof here.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to the natural risk on a mixture Gaussian distribution D tmp : x ∼ N ( y · µ tmp , σ 2 I ) ,where µ tmp = (1 − η, 0 , .
Revised: which is equivalent to the natural risk on a mixture Gaussian distribution D tmp : x ∼ N ( y · µ tmp , σ 2 I ) , where µ tmp = (1 − η, 0 , .

Original: We note that the Bayes optimal classiﬁer for D tmp is f tmp ( x ) = sign( µ (cid:62) tmp x ) .
Revised: We note that the Bayes optimal classifier for D tmp is f tmp ( x ) = sign( µ ⊤ tmp x ) .

Original: Speciﬁcally, the natural risk
Revised: Specifically, the natural risk

Original: [Equation]
Revised: [Equation]

Original: which is minimized when w 1 = 1 − η > 0 and b = 0 .
Revised: which is minimized when w 1 = 1 − η > 0 and b = 0 .

Original: That is, minimizing the adversarial risk R (cid:15) + η adv ( f, T hyp )can lead to an optimal linear (cid:96) ∞ -robust classiﬁer f tmp ( x ) , which is equivalent to the robust classiﬁer (9).
Revised: That is, minimizing the adversarial risk R ϵ + η adv ( f, T hyp ) can lead to an optimal linear ℓ ∞ -robust classifier f tmp ( x ) , which is equivalent to the robust classifier (9).

Original: Moreover, when the defense budget (cid:15) d is less than (cid:15) + η , the condition in Lemma 4 no longer holds.
Revised: Moreover, when the defense budget ϵ d is less than ϵ + η , the condition in Lemma 4 no longer holds.

Original: Instead, in699 this case, the robust classiﬁer will assign positive weights to the features (i.e., w i > 0 for i ≥ 1 ).
Revised: Instead, in this case, the robust classifier will assign positive weights to the features (i.e., w i > 0 for i ≥ 1 ).

Original: This is similarto the case in Lemma 3, and thus we omit the proof here.
Revised: This is similar to the case in Lemma 3, and thus we omit the proof here.

Original: Consequently, this yields classiﬁers that still rely on all701 the non-robust features.
Revised: Consequently, this yields classifiers that still rely on all the non-robust features.

Original: C.5 Proof of Theorem4
Revised: C.5 Proof of Theorem

Original: Theorem 4 (restated).
Revised: Theorem 4 (restated).

Original: For any data distribution and any adversary with an attack budget (cid:15) , training models to705 minimize the adversarial risk with a defense budget 2 (cid:15) on the perturbed data is sufﬁcient to ensure (cid:15) -robustness.
Revised: For any data distribution and any adversary with an attack budget ϵ , training models to minimize the adversarial risk with a defense budget 2 ϵ on the perturbed data is sufficient to ensure ϵ -robustness.

Original: Proof.
Revised: Proof.

Original: For clarity, we rewrite the adversarial risk in (2) with a defense budget (cid:15) as follows:
Revised: For clarity, we rewrite the adversarial risk in (2) with a defense budget ϵ as follows:

Original: [Equation]
Revised: [Equation]

Original: where T = { ( x i , y i ) } ni =1 denotes the empirical training data.
Revised: where T = { ( x i , y i ) } ni =1 denotes the empirical training data.

Original: Consider any adversary with an attack budget (cid:15) , who can perturb x to x + p such that (cid:107) p (cid:107) ≤ (cid:15) .
Revised: Consider any adversary with an attack budget ϵ , who can perturb x to x + p such that ∥ p ∥ ≤ ϵ .

Original: Then, thelearner will receive a perturbed version of training data T (cid:48)
Revised: Then, the learner will receive a perturbed version of training data T ′

Original: = { ( x i + p
Revised: = { ( x i + p

Original: i , y i ) } ni =1 .
Revised: i , y i ) } ni =1 .

Original: For any perturbed data point ( x i + p
Revised: For any perturbed data point ( x i + p

Original: i , y i ) , we have
Revised: i , y i ) , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By summarizing the training points, we have
Revised: By summarizing the training points, we have

Original: [Equation]
Revised: [Equation]

Original: That is, the adversarial risk with a defense budget 2 (cid:15) on the perturbed data is an upper bound of the adversarial713 risk with a defense budget (cid:15) on the original data.
Revised: That is, the adversarial risk with a defense budget 2 ϵ on the perturbed data is an upper bound of the adversarial risk with a defense budget ϵ on the original data.

Original: Therefore, a defense budget 2 (cid:15) is sufﬁcient to ensure thelearning of (cid:15) -robustness.
Revised: Therefore, a defense budget 2 ϵ is sufficient to ensure the learning of ϵ -robustness.

Original: D Experimental Settings
Revised: D Experimental Settings

Original: Adversary capability.
Revised: Adversary capability.

Original: We focus on the clean-label setting, where an adversary can only provide correctly717 labeled but misleading training data.
Revised: We focus on the clean-label setting, where an adversary can only provide correctly labeled but misleading training data.

Original: In this setting, the main constraint is to craft perturbations as small aspossible [16].
Revised: In this setting, the main constraint is to craft perturbations as small as possible [16].

Original: Thus, we consider an (cid:96) ∞ adversary with an attack budget (cid:15) a = 8 / 255 by following Huang et al.
Revised: Thus, we consider an ℓ ∞ adversary with an attack budget ϵ a = 8 / 255 by following Huang et al.

Original: 719 [28], Yuan and Wu [79], Tao et al.
Revised: [28], Yuan and Wu [81], Tao et al.

Original: [64], Fowl et al.
Revised: [64], Fowl et al.

Original: We note that this constraint is consistent with commonresearch on test-time adversarial examples [1].
Revised: We note that this constraint is consistent with common research on test-time adversarial examples [1].

Original: Crafting details.
Revised: Crafting details.

Original: We conduct stability attacks by applying the hypocritical perturbation into the training722 set.
Revised: We conduct stability attacks by applying the hypocritical perturbation into the training set.

Original: Unless otherwise speciﬁed, we craft the perturbations by solving the error-minimizing objective (4) with100 steps of PGD, where a step size of 0 . 8 / 255 is used by following Fowl et al.
Revised: Unless otherwise specified, we craft the perturbations by solving the error-minimizing objective (4) with 100 steps of PGD, where a step size of 0 . 8 / 255 is used by following Fowl et al.

Original: Our crafting model is724 adversarially trained with a crafting budget (cid:15) c = 0 .
Revised: Our crafting model is adversarially trained with a crafting budget ϵ c = 0 .

Original: 25 (cid:15) a for 10 epochs before generating perturbations.
Revised: 25 ϵ a for 10 epochs before generating perturbations.

Original: That is,setting (cid:15) c = 2 / 255 performs best, as shown in Figure 2(a).
Revised: That is, setting ϵ c = 2 / 255 performs best, as shown in Figure 2(a).

Original: Training details.
Revised: Training details.

Original: We evaluate the effectiveness of the hypocritical perturbation on benchmark datasets727 including CIFAR-10/100 [33], SVHN [43], and Tiny-ImageNet [34].
Revised: We evaluate the effectiveness of the hypocritical perturbation on benchmark datasets including CIFAR-10/100 [33], SVHN [43], and Tiny-ImageNet [34].

Original: Unless otherwise speciﬁed, we use ResNet18 [26] as the default architecture for both the crafting model and the learning model.
Revised: Unless otherwise specified, we use ResNet- 18 [26] as the default architecture for both the crafting model and the learning model.

Original: For adversarial training,729 we mainly follow the settings in previous studies [81, 69, 50].
Revised: For adversarial training, we mainly follow the settings in previous studies [83, 70, 50].

Original: By convention, the defense budget is equal tothe attack budget, i.e., (cid:15) d = 8 / 255 .
Revised: By convention, the defense budget is equal to the attack budget, i.e., ϵ d = 8 / 255 .

Original: The networks are trained for 100 epochs using SGD with momentum 0.9,731 weight decay 5 × 10 − 4 , and an initial learning rate of 0.1 that is divided by 10 at the 75-th and 90-th epoch.
Revised: The networks are trained for 100 epochs using SGD with momentum 0.9, weight decay 5 × 10 − 4 , and an initial learning rate of 0.1 that is divided by 10 at the 75-th and 90-th epoch.

Original: Early stopping is done with holding out 1000 examples from the training set.
Revised: Early stopping is done with holding out 1000 examples from the training set.

Original: Simple data augmentations such733 as random crop and horizontal ﬂip are applied.
Revised: Simple data augmentations such as random crop and horizontal flip are applied.

Original: The inner maximization problem during adversarial training issolved by 10-steps PGD (PGD-10) with step size 2 / 255 .
Revised: The inner maximization problem during adversarial training is solved by 10-steps PGD (PGD-10) with step size 2 / 255 .

Original: E Feature-level Analysis on CIFAR-10 
Revised: E Feature-level Analysis on CIFAR-

Original: In Section 3.1, we theoretically showed that the hypocritical perturbation can cause the poisoned model to rely737 more on non-robust features, thus the natural accuracy of the adversarially trained model is increased whilethe robust accuracy is decreased.
Revised: In Section 3.1, we theoretically showed that the hypocritical perturbation can cause the poisoned model to rely more on non-robust features, thus the natural accuracy of the adversarially trained model is increased while the robust accuracy is decreased.

Original: In this part, we aim to provide empirical evidence on the role of non-robust739 features in the success of our poisoning method on a benchmark dataset.
Revised: In this part, we aim to provide empirical evidence on the role of non-robust features in the success of our poisoning method on a benchmark dataset.

Original: In particular, we will demonstrate thatour hypocritical perturbation successfully makes the poisoned model learn more non-robust features.
Revised: In particular, we will demonstrate that our hypocritical perturbation successfully makes the poisoned model learn more non-robust features.

Original: To show this, by following Section 3.2 of Ilyas et al.
Revised: To show this, by following Section 3.2 of Ilyas et al.

Original: [29], we construct a training set where the only features that are useful for classiﬁcation are the non-robust features (that are extracted from the poisoned model).
Revised: [29], we construct a training set where the only features that are useful for classification are the non-robust features (that are extracted from the poisoned model).

Original: The standard accuracy of the classiﬁer trained on the constructed dataset can reﬂect how many non-robust features are learned by the poisoned model (denoted as f ).
Revised: The standard accuracy of the classifier trained on the constructed dataset can reflect how many non-robust features are learned by the poisoned model (denoted as f ).

Original: To accomplish this, we modify each input-label pair ( x , y ) as follows.
Revised: To accomplish this, we modify each input-label pair ( x , y ) as follows.

Original: We select a target class t uniformly at random among classes.
Revised: We select a target class t uniformly at random among classes.

Original: Then, we add a small adversarial perturbation to x as follows:
Revised: Then, we add a small adversarial perturbation to x as follows:

Original: [Equation]
Revised: [Equation]

Original: The resulting input-label pairs ( x adv , t ) make up the new training set.
Revised: The resulting input-label pairs ( x adv , t ) make up the new training set.

Original: Since the resulting inputs x adv are nearly742 indistinguishable from the originals x , the label t assigned to the modiﬁed input is simply incorrect to a humanobserver.
Revised: Since the resulting inputs x adv are nearly indistinguishable from the originals x , the label t assigned to the modified input is simply incorrect to a human observer.

Original: Therefore, only the non-robust features in the training set are predictive, while the non-robust features744 are extracted from the poisoned model.
Revised: Therefore, only the non-robust features in the training set are predictive, while the non-robust features are extracted from the poisoned model.

Original: We compare the model poisoned by our hypocritical perturbation with the baseline model trained on clean data.
Revised: We compare the model poisoned by our hypocritical perturbation with the baseline model trained on clean data.

Original: These two models correspond to the second row and last row in Table 2, respectively.
Revised: These two models correspond to the second row and last row in Table 2, respectively.

Original: Using these two models,we construct two datasets in the above-mentioned manner, respectively.
Revised: Using these two models, we construct two datasets in the above-mentioned manner, respectively.

Original: Then, two new predictors are trained on748 the two constructed datasets, respectively, and both predictors are evaluated on clean data.
Revised: Then, two new predictors are trained on the two constructed datasets, respectively, and both predictors are evaluated on clean data.

Original: Training parametersfollow exactly those adopted by Ilyas et al.
Revised: Training parameters follow exactly those adopted by Ilyas et al.

Original: Our numerical results are summarized in Table 13.
Revised: Our numerical results are summarized in Table 13.

Revised: the poisoned model

Original: [Table]
Revised: [Table]

Original: As shown in Table 13, the non-robust features learned by the poisoned model are much more predictive than751 the baseline.
Revised: As shown in Table 13, the non-robust features learned by the poisoned model are much more predictive than the baseline.

Original: This indicates that the effect of our poisoning method on the non-robust features learned by thepoisoned model is validated empirically.
Revised: This indicates that the effect of our poisoning method on the non-robust features learned by the poisoned model is validated empirically.

Original: F Broader Impact
Revised: F Broader Impact

Original: The attack method in this work might be used by an agent in the real world to damage the robust availability755 of a machine-learning-based system.
Revised: The attack method in this work might be used by an agent in the real world to damage the robust availability of a machine-learning-based system.

Original: We discourage this malicious behavior by presenting the threat model ofstability attacks to the community.
Revised: We discourage this malicious behavior by presenting the threat model of stability attacks to the community.

Original: We further propose an adaptive defense to mitigate this issue.
Revised: We further propose an adaptive defense to mitigate this issue.

Original: The adaptive757 defense would help to build a more secure and robust machine learning system in the real world.
Revised: The adaptive defense would help to build a more secure and robust machine learning system in the real world.

Original: At the sametime, the adaptive defense introduces an additional time cost to search for an appropriate defense budget, which759 might have a negative impact on carbon emission reduction.
Revised: At the same time, the adaptive defense introduces an additional time cost to search for an appropriate defense budget, which might have a negative impact on carbon emission reduction.

Original: Furthermore, society should not be overly optimisticabout AI safety, since the current studies mostly focus on perturbations bounded by simple norms (e.g., (cid:96) ∞ norm761 in this paper).
Revised: Furthermore, society should not be overly optimistic about AI safety, since the current studies mostly focus on perturbations bounded by simple norms (e.g., ℓ ∞ norm in this paper).

Original: There might exist perturbations beyond the (cid:96) p ball in the real world, and we are still far fromcomplete model robustness.
Revised: There might exist perturbations beyond the ℓ p ball in the real world, and we are still far from complete model robustness.

Original: G On the Trade-off between Accuracy and Robustness
Revised: G On the Trade-off between Accuracy and Robustness

Original: An interesting implication of this work is that the hypocritical perturbation exploits the trade-off between765 standard generalization and adversarial robustness, a phenomenon that has been widely observed in existingworks on adversarial training [67, 81, 14, 40, 59, 77].
Revised: An interesting implication of this work is that the hypocritical perturbation exploits the trade-off between standard generalization and adversarial robustness, a phenomenon that has been widely observed in existing works on adversarial training [67, 83, 14, 40, 59, 79].

Original: Prior work mainly observed that adversarial training improves robust accuracy at the cost of natural accuracy768 when the training data is clean .
Revised: Prior work mainly observed that adversarial training improves robust accuracy at the cost of natural accuracy when the training data is clean .

Original: An explanation for the phenomenon is that there are non-robust features in theoriginal dataset, which are predictive yet brittle [67, 29].
Revised: An explanation for the phenomenon is that there are non-robust features in the original dataset, which are predictive yet brittle [67, 29].

Original: Unlike prior work, the trade-off in this work occurs770 when the training data is hypocritically perturbed .
Revised: Unlike prior work, the trade-off in this work occurs when the training data is hypocritically perturbed .

Original: Speciﬁcally, we make the following observations:
Revised: Specifically, we make the following observations:

Original: When trained on the hypocritically perturbed data, conventional adversarial training produces modelswith lower robust accuracy but higher natural accuracy (e.g., see Table 2, Table 3, and Table 4).
Revised: When trained on the hypocritically perturbed data, conventional adversarial training produces models with lower robust accuracy but higher natural accuracy (e.g., see Table 2, Table 3, and Table 4).

Original: When trained on the hypocritically perturbed data, adversarial training with adaptive budget canimprove robust accuracy while reducing natural accuracy (e g., see Table 5).
Revised: When trained on the hypocritically perturbed data, adversarial training with adaptive budget can improve robust accuracy while reducing natural accuracy (e.g., see Table 5).

Original: These two observations align well with our theoretical analyses in Section 3 and Section 4, respectively.
Revised: These two observations align well with our theoretical analyses in Section 3 and Section 4, respectively.

Original: Concretely, our analyses suggest that the hypocritical perturbation works by reinforcing the non-robust featuresin the original data, so that the models adversarially trained on the manipulated data still rely on the non-robust778 features.
Revised: Concretely, our analyses suggest that the hypocritical perturbation works by reinforcing the non-robust features in the original data, so that the models adversarially trained on the manipulated data still rely on the non-robust features.

Original: In this way, the natural accuracy of the models increases because the non-robust features are predictive,while the robust accuracy decreases because the non-robust features are brittle.
Revised: In this way, the natural accuracy of the models increases because the non-robust features are predictive, while the robust accuracy decreases because the non-robust features are brittle.

Original: Furthermore, the effectiveness of780 the adaptive defense lies in the fact that the reinforced non-robust features can be neutralized by enlarging thedefense budget of adversarial training.
Revised: Furthermore, the effectiveness of the adaptive defense lies in the fact that the reinforced non-robust features can be neutralized by enlarging the defense budget of adversarial training.

Original: Thus, the adaptive defense improves robustness at the cost of accuracy.
Revised: Thus, the adaptive defense improves robustness at the cost of accuracy.

Original: Meanwhile, we note that it would be unsatisfactory that test robustness is improved at the cost of standard783 generalization.
Revised: Meanwhile, we note that it would be unsatisfactory that test robustness is improved at the cost of standard generalization.

Original: Several improvements have been proposed to alleviate this issue in the case where the trainingdata is clean, such as RST [49], FAT [82], and SCORE [48].
Revised: Several improvements have been proposed to alleviate this issue in the case where the training data is clean, such as RST [49], FAT [84], and SCORE [48].

Original: Incorporating these advances would be helpful in785 resisting stability attacks, and we leave this as future work.
Revised: Incorporating these advances would be helpful in resisting stability attacks, and we leave this as future work.

Original: Finally, we remark that the focus of stability attacks is to degrade test robustness.
Revised: Finally, we remark that the focus of stability attacks is to degrade test robustness.

Original: For this reason, we do not787 impose additional restrictions on their impact on natural accuracy.
Revised: For this reason, we do not impose additional restrictions on their impact on natural accuracy.

Original: Having that said, as a method of stabilityattacks, the hypocritical perturbation is observed to improve natural accuracy while reducing robust accuracy.
Revised: Having that said, as a method of stability attacks, the hypocritical perturbation is observed to improve natural accuracy while reducing robust accuracy.

Original: We note that this makes stability attacks more insidious.
Revised: We note that this makes stability attacks more insidious.

Original: For example, if a poisoned model exhibits highernatural accuracy, practitioners would be more easily enticed to deploy it in a real-world system.
Revised: For example, if a poisoned model exhibits higher natural accuracy, practitioners would be more easily enticed to deploy it in a real-world system.

Original: However, as its791 robust accuracy is actually undesirably low, the system is prone to losing its normal function when encounteringtest-time perturbations.
Revised: However, as its robust accuracy is actually undesirably low, the system is prone to losing its normal function when encountering test-time perturbations.

Original: In short, the negative impacts of stability attacks are serious, even with higher natural793 accuracy.
Revised: In short, the negative impacts of stability attacks are serious, even with higher natural accuracy.

Original: Thus, it is imperative to design better defense methods to mitigate the threat of stability attacks.
Revised: Thus, it is imperative to design better defense methods to mitigate the threat of stability attacks.