Original: Can Adversarial Training Be Manipulated By Non-Robust Features?
Revised: Can Adversarial Training Be Manipulated By Non-Robust Features?

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Adversarial training, originally designed to resist test-time adversarial examples,has shown to be promising in mitigating training-time availability attacks .
Revised: Adversarial training, originally designed to resist test-time adversarial examples,has shown to be promising in mitigating training-time availability attacks .

Original: Thisdefense ability, however, is challenged in this paper.
Revised: Thisdefense ability, however, is challenged in this paper.

Original: We identify a novel threatmodel named stability attacks , which aims to hinder robust availability by slightlymanipulating the training data.
Revised: We identify a novel threatmodel named stability attacks , which aims to hinder robust availability by slightlymanipulating the training data.

Original: Under this threat, we show that adversarial trainingusing a conventional defense budget (cid:15) provably fails to provide test robustnessin a simple statistical setting, where the non-robust features of the training datacan be reinforced by (cid:15) -bounded perturbation.
Revised: Under this threat, we show that adversarial trainingusing a conventional defense budget (cid:15) provably fails to provide test robustnessin a simple statistical setting, where the non-robust features of the training datacan be reinforced by (cid:15) -bounded perturbation.

Original: Further, we analyze the necessity ofenlarging the defense budget to counter stability attacks.
Revised: Further, we analyze the necessity ofenlarging the defense budget to counter stability attacks.

Original: Finally, comprehensiveexperiments demonstrate that stability attacks are harmful on benchmark datasets,and thus the adaptive defense is necessary to maintain robustness.
Revised: Finally, comprehensiveexperiments demonstrate that stability attacks are harmful on benchmark datasets,and thus the adaptive defense is necessary to maintain robustness.

Original:  Introduction
Revised:  Introduction

Original: Robustness to input perturbations is crucial to machine learning deployment in various applications,such as spam ﬁltering [13] and autonomous driving [6].
Revised: Robustness to input perturbations is crucial to machine learning deployment in various applications,such as spam ﬁltering [12] and autonomous driving [5].

Original: One of the most popular methods forimproving test robustness is adversarial training [39, 1].
Revised: One of the most popular methods forimproving test robustness is adversarial training [39, 1].

Original: By augmenting the training data with(cid:15) -bounded and on-the-ﬂy crafted adversarial examples, adversarial training helps the learned modelresist test-time perturbations [39].
Revised: By augmenting the training data with(cid:15) -bounded and on-the-ﬂy crafted adversarial examples, adversarial training helps the learned modelresist test-time perturbations [39].

Original: [Figure]
Revised: [Figure]

Original: training.
Revised: training.

Original: On the other hand, machine learning systemsare vulnerable to training-time availability attacks [3].
Revised: On the other hand, machine learning systemsare vulnerable to training-time availability attacks [2].

Original: In particular, small perturbations applied into the training data before training sufﬁce to degrade the overall test performance ofnaturally trained models [16, 28].
Revised: In particular, small perturbations applied into the training data before training sufﬁce to degrade the overall test performance ofnaturally trained models [15, 27].

Original: Fortunately,recent work has shown that adversarial training [39] is capable of mitigating this type ofthreat [64, 18].
Revised: Fortunately,recent work has shown that adversarial training [39] is capable of mitigating this type ofthreat [60, 17].

Original: In other words, even if the training data is manipulated by an adversary to maximize the test error, adversarially trained modelscan still achieve considerable accuracy on cleantest data.
Revised: In other words, even if the training data is manipulated by an adversary to maximize the test error, adversarially trained modelscan still achieve considerable accuracy on cleantest data.

Original: However, previous work hardly inspects the test robustness of the models, which is what adversarial training was originally proposedfor [23, 39].
Revised: However, previous work hardly inspects the test robustness of the models, which is what adversarial training was originally proposedfor [22, 39].

Original: This naturally raises the following question
Revised: This naturally raises the following question

Original: Are the models adversarially trained on the manipulated data robust to test-time perturbations?
Revised: Are the models adversarially trained on the manipulated data robust to test-time perturbations?

Original: In this work, we show that conventional adversarial training may fail to provide test robustness whenthe training data is manipulated by an adversary, and thus an adaptive defense is necessary to resolvethis issue.
Revised: In this work, we show that conventional adversarial training may fail to provide test robustness whenthe training data is manipulated by an adversary, and thus an adaptive defense is necessary to resolvethis issue.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: We introduce a novel threat model called stability attacks , where an adversary aims to
Revised: We introduce a novel threat model called stability attacks , where an adversary aims to

Original: degrade the overall test robustness of adversarially trained models by slightly perturbing the
Revised: degrade the overall test robustness of adversarially trained models by slightly perturbing the

Original: training data.
Revised: training data.

Original: Figure 1 illustrates the threat of stability attacks.
Revised: Figure 1 illustrates the threat of stability attacks.

Original: We show that adversarial training using a conventional defense budget provably fails understability attacks in a simple statistical setting.
Revised: We show that adversarial training using a conventional defense budget provably fails understability attacks in a simple statistical setting.

Original: Speciﬁcally, a defense budget of (cid:15) will producemodels that are not robust to (cid:15) -bounded adversarial examples when the training data ishypocritically perturbed.
Revised: Speciﬁcally, a defense budget of (cid:15) will producemodels that are not robust to (cid:15) -bounded adversarial examples when the training data ishypocritically perturbed.

Original: We unveil that the aforementioned vulnerability stems from the existence of the non-robust
Revised: We unveil that the aforementioned vulnerability stems from the existence of the non-robust

Original: (predictive, yet brittle) features [29] in the original training data.
Revised: (predictive, yet brittle) features [28] in the original training data.

Original: When the non-robust
Revised: When the non-robust

Original: features are reinforced by hypocritical perturbations, the conventional defense budget will
Revised: features are reinforced by hypocritical perturbations, the conventional defense budget will

Original: be insufﬁcient to offset the negative impact.
Revised: be insufﬁcient to offset the negative impact.

Original: We further show that a defense budget of 2 (cid:15) is capable of resisting any stability attack foradversarial training, while the budget can be reduced to (cid:15) + η in a simple statistical setting,where η is the magnitude of the non-robust features.
Revised: We further show that a defense budget of 2 (cid:15) is capable of resisting any stability attack foradversarial training, while the budget can be reduced to (cid:15) + η in a simple statistical setting,where η is the magnitude of the non-robust features.

Original: We demonstrate that stability attacks are harmful to conventional adversarial training on
Revised: We demonstrate that stability attacks are harmful to conventional adversarial training on

Original: benchmark datasets.
Revised: benchmark datasets.

Original: In addition, our empirical study suggests that enlarging the defensebudget is essential for mitigating hypocritical perturbations.
Revised: In addition, our empirical study suggests that enlarging the defensebudget is essential for mitigating hypocritical perturbations.

Original: To the best of our knowledge, this is the ﬁrst work that studies the robustness of adversarial trainingagainst stability attacks.
Revised: To the best of our knowledge, this is the ﬁrst work that studies the robustness of adversarial trainingagainst stability attacks.

Original: Both theoretical and empirical evidences show that the conventional defensebudget (cid:15) is insufﬁcient under the threat of (cid:15) -bounded training-time perturbations.
Revised: Both theoretical and empirical evidences show that the conventional defensebudget (cid:15) is insufﬁcient under the threat of (cid:15) -bounded training-time perturbations.

Original: Our ﬁndings suggestthat practitioners should consider a larger defense budget of no more than 2 (cid:15) (practically, about1 .
Revised: Our ﬁndings suggestthat practitioners should consider a larger defense budget of no more than 2 (cid:15) (practically, about1 .

Original: 5 (cid:15) ∼ 1 .
Revised: 5 (cid:15) ∼ 1 .

Original: 75 (cid:15) )
Revised: 75 (cid:15) )

Original: to achieve a better (cid:15) -robustness.
Revised: to achieve a better (cid:15) -robustness.

Original:  Threat Models
Revised:  Threat Models

Original: In this section, we formally introduce the threat model of stability attacks.
Revised: In this section, we formally introduce the threat model of stability attacks.

Original: We begin by revisiting theconcepts of natural risk, adversarial risk, and delusive attacks.
Revised: We begin by revisiting theconcepts of natural risk, adversarial risk, and delusive attacks.

Original: These concepts naturally give rise toour formulation of stability attacks.
Revised: These concepts naturally give rise toour formulation of stability attacks.

Original: 2.1 Preliminaries
Revised: 2.1 Preliminaries

Original: Setup.
Revised: Setup.

Original: We consider a classiﬁcation task with input-label pairs ( x , y ) from an underlying distribution
Revised: We consider a classiﬁcation task with input-label pairs ( x , y ) from an underlying distribution

Original: D over X × [ k ] .
Revised: D over X × [ k ] .

Original: The goal is to learn a (robust) classiﬁer f : X → [ k ] that predicts a label y for agiven input x .
Revised: The goal is to learn a (robust) classiﬁer f : X → [ k ] that predicts a label y for agiven input x .

Original: Natural training (NT).
Revised: Natural training (ST).

Original: Most learning algorithms aim to maximize the generalization performanceon unperturbed examples, i.e., natural accuracy.
Revised: Most learning algorithms aim to maximize the generalization performanceon unperturbed examples, i.e., natural accuracy.

Original: The goal is to minimize the natural risk deﬁned as:
Revised: The goal is to minimize the natural risk deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Adversarial training (AT).
Revised: Adversarial training (AT).

Original: Since the risk of adversarial examples (a.k.a.
Revised: Since the risk of adversarial examples (a.k.a.

Original: evasion attacks) wasfound to be unexpectedly high [5, 60], it has become increasingly important to defend the learneragainst the worst-case perturbations [23, 39].
Revised: evasion attacks) wasfound to be unexpectedly high [4, 56], it has become increasingly important to defend the learneragainst the worst-case perturbations [22, 39].

Original: In this context, the goal is to train a model that has lowadversarial risk given a defense budget (cid:15) :
Revised: In this context, the goal is to train a model that has lowadversarial risk given a defense budget (cid:15) :

Original: [Equation]
Revised: [Equation]

Original: where we choose ∆ to be the set of (cid:15) -bounded perturbations, i.e., ∆ = { δ | (cid:107) δ (cid:107)  ≤ (cid:15) } .
Revised: where we choose ∆ to be the set of (cid:15) -bounded perturbations, i.e., ∆ = { δ | (cid:107) δ (cid:107) p ≤ (cid:15) } .

Original: This choice isthe most common one in the context of adversarial examples [67].
Revised: This choice isthe most common one in the context of adversarial examples [63].

Original: To simplify the notation, we referto the robustness with respect to this set as (cid:15) -robustness.
Revised: To simplify the notation, we referto the robustness with respect to this set as (cid:15) -robustness.

Original: It is worthnoting
Revised: It is worth

Original: [Table]
Revised: [Table]

Original: Delusive attacks.
Revised: Delusive attacks.

Original: Delusive attacks, which belong to training-time availability attacks, aim to preventthe learner from producing an accurate model by manipulating the training data “imperceptibly” [44].
Revised: Delusive attacks, which belong to training-time availability attacks, aim to preventthe learner from producing an accurate model by manipulating the training data “imperceptibly” [43].

Original: Concretely, the features of the training data can be perturbed, while the labels should remaincorrect [16, 56, 28, 79, 15, 64, 18, 19].
Revised: Concretely, the features of the training data can be perturbed, while the labels should remaincorrect [15, 53, 27, 74, 14, 60, 17, 18].

Original: This malicious task can be formalized into the followingbi-level optimization problem:
Revised: This malicious task can be formalized into the followingbi-level optimization problem:

Original: [Equation]
Revised: [Equation]

Original:  Here, the adversary aims to maximize the natural risk of the model f P (that is trained on themanipulated training set) by applying the generated perturbations P = { p i } ni =1 into the originaltraining set T = { ( x i , y i ) } ni =1 .
Revised: [Equation] Here, the adversary aims to maximize the natural risk of the model f P (that is trained on themanipulated training set) by applying the generated perturbations P = { p i } ni =1 into the originaltraining set T = { ( x i , y i ) } ni =1 .

Original: The commonly used feasible region  is S
Revised: The commonly used feasible region of the perturbations is S

Original: = {{ p i } ni =1 | (cid:107) pi (cid:107) ≤ (cid:15) } .
Revised: ={{ p i } ni =1 | (cid:107) p

Original: Generally, solving Equation (3) is computationally prohibitive for neural networks [64, 18].

Original: Thus,various heuristic methods are explored to resolve this problem.

Original: For example, a representative methodcalled hypocritical perturbations [63, 64, 28, 19] is crafted as follows:

Original: [Equation]

Original: where f craft is called the crafting model , pre-trained before generating poisons.

Original: Tao et al.

Original: [64] simplyadopted a naturally trained classiﬁer as the crafting model, while Huang et al. [28] proposed amin-min bi-level optimization process to pre-train the crafting model. Fu et al. [19] further built theircrafting model via a min-min-max three-level optimization process, and generated their poisons byreplacing Equation (4) with a min-max bi-level objective.
Revised: i (cid:107) p ≤ (cid:15) } .

Original: Another representative method of delusive attacks is the adversarial perturbation , crafted by solving

Original: [Equation]

Original: Tao et al. [64] and Fowl et al. [18] independently found thatapplying the adversarialperturbation tothe training data isvery effective at compromising naturally trained models. However, adversarialtraining has proven to be promising in defending against various delusive attacks [64].
Revised: It has been shown that adversarial training [39] is promising in defendingagainst delusive attacks [60, 17].

Original: 2.2 Stability Attacks
Revised: 2.2 Stability Attacks

Original: In contrast to delusive attacks that aim at increasing the natural risk, stability attacks attempt tomaximize the adversarial risk of the learner by slightly perturbing the training data:
Revised: In contrast to delusive attacks that aim at increasing the natural risk, stability attacks attempt tomaximize the adversarial risk of the learner by slightly perturbing the training data:

Original: [Equation]
Revised: [Equation]

Original: where f P denotes the victim model, which is naturally or adversarially trained on the perturbed data.
Revised: where f P denotes the victim model, which is naturally or adversarially trained on the perturbed data.

Original: In other words, stability attacks seek to hinder the robust availability of the training data.
Revised: In other words, stability attacks seek to hinder the robust availability of the training data.

Original: Table 1shows the comparisons among different threat models.
Revised: Table 1shows the comparisons among different threat models.

Original: The goal of stability attacks can be immediately achieved for naturally trained models, since they have already incurred high adversarial risk, even if the training data is clean [60].
Revised: The goal of stability attacks can be immediately achieved for naturally trained models, since theyhave already incurred high adversarial risk, even if the training data is clean [56].

Original: To ease the problem of high adversarial risk, adversarial training has been widely used to improve model’s adversarialrobustness [24, 12].
Revised: To ease the problemof high adversarial risk, adversarial training has been widely used to improve model’s adversarialrobustness [23, 11].

Original: Hence, the main goal of stability attacks becomes to compromise the testrobustness of adversarially trained models.
Revised: Hence, the main goal of stability attacks becomes to compromise the testrobustness of adversarially trained models.

Original: Note that Equation (6) is a multi-level optimization problem that is not easy to solve, our next questionis how to conduct effective stability attacks against adversarial training.
Revised: Note that Equation (4) is a multi-level optimization problem that is not easy to solve, our next questionis how to conduct effective stability attacks against adversarial training.

Original: In the following sections, weintroduce an effective stability attack method and analyze the cost of resisting it.
Revised: In the following sections, weintroduce an effective stability attack method and analyze the cost of resisting it.

Original: Justiﬁcation.
Revised: Justiﬁcation.

Original: This work focuses on adding bounded pertubrations as small as possible.
Revised: This work focuses on adding bounded pertubrations as small as possible.

Original: We mostlyassume that the adversary has full control of training data (instead of changing a few) by followingprevious works [16, 28, 64, 17–19].
Revised: We mostlyassume that the adversary has full control of training data (instead of changing a few) by followingprevious works [15, 27, 60, 16–18].

Original: This is a realistic assumption [16, 18].
Revised: This is a realistic assumption [15, 17].

Original: For instance, in someapplications an organization may agree to release some internal data for peer assessment, whilepreventing competitors from easily building a model with high test robustness; this can be achievedby perturbing the entire dataset via stability attacks before releasing.
Revised: For instance, in someapplications an organization may agree to release some internal data for peer assessment, whilepreventing competitors from easily building a model with high test robustness; this can be achievedby perturbing the entire dataset via stability attacks before releasing.

Original: Moreover, this assumptionenables a worst-case analysis of the robustness of adversarial training, which may facilitate importanttheoretical insights.
Revised: Moreover, this assumptionenables a worst-case analysis of the robustness of adversarial training, which may facilitate importanttheoretical insights.

Original:  How to Manipulate Adversarial Training
Revised:  How to Manipulate Adversarial Training

Original: Previous work suggests that adversarial training could defend against both evasion attacks and delusiveattacks [39, 64].
Revised: Previous work suggests that adversarial training could defend against both evasion attacks and delusiveattacks [39, 60].

Original: However, in this paper, we show that adversarial training using a conventionaldefense budget (cid:15) may not be sufﬁcient to provide (cid:15) -robustness when confronted with stability attacks.
Revised: However, in this paper, we show that adversarial training using a conventionaldefense budget (cid:15) may not be sufﬁcient to provide (cid:15) -robustness when confronted with stability attacks.

Original: In particular, we present a simple theoretical model where the conventional defense scheme provablyfails when the training data is hypocritically perturbed.
Revised: In particular, we present a simple theoretical model where the conventional defense scheme provablyfails when the training data is hypocritically perturbed.

Original: The binary classiﬁcation task.
Revised: The binary classiﬁcation task.

Original: The data model is largely based on the setting proposed by Tsipraset al.
Revised: The data model is largely based on the setting proposed by Tsipraset al.

Original: [67], which draws a distinction between robust features and non-robust features .
Revised: [63], which draws a distinction between robust features and non-robust features .

Original: Speciﬁcally, itconsists of input-label pairs ( x , y ) sampled from a Gaussian mixture distribution D as follows:
Revised: Speciﬁcally, itconsists of input-label pairs ( x , y ) sampled from a Gaussian mixture distribution D as follows:

Original: [Equation]
Revised: [Equation]

Original: where η is much smaller than 1 (i.e., 0 < η (cid:28) 1 ).
Revised: where η is much smaller than 1 (i.e., 0 < η (cid:28) 1 ).

Original: Hence, samples from D consist of a robust feature( x 1 ) that is strongly correlated with the label, and d non-robust features ( x 2 , . .
Revised: Hence, samples from D consist of a robust feature( x 1 ) that is strongly correlated with the label, and d non-robust features ( x 2 , . .

Original: , x d +1 )
Revised: , x d +1 )

Original: that are veryweakly correlated with it.
Revised: that are veryweakly correlated with it.

Original: Typically, an adversary can manipulate a large number of non-robustfeatures - e.g. d = Θ(1 /η 2 ) will sufﬁce.
Revised: Typically, an adversary can manipulate a large number of non-robustfeatures - e.g. d = Θ(1 /η 2 ) will sufﬁce.

Original: Before introducing the way to hinder robust availability, we brieﬂy illustrate the success of adversarialtraining when the training data is unperturbed.
Revised: Before introducing the way to hinder robust availability, we brieﬂy illustrate the success of adversarialtraining when the training data is unperturbed.

Original: Natural and robust classiﬁers.
Revised: Natural and robust classiﬁers.

Original: For standard classiﬁcation, we consider a natural classiﬁer: [Equation]
Revised: For standard classiﬁcation, we consider a natural classiﬁer:f nat ( x ) :

Original:  , where 
Revised: = sign( w (cid:62) nat x ) , where w nat :=

Original: [Equation]
Revised: [1 , η, . . .

Original:  ] , (8) which is a minimizer of the natural risk (1) with 0-1 loss on the data (7), i.e., the Bayes optimalclassiﬁer.
Revised: , η ] , (6) which is a minimizer of the natural risk (1) with 0-1 loss on the data (5), i.e., the Bayes optimalclassiﬁer.

Original: However, in the adversarial setting, this natural classiﬁer is quite brittle.
Revised: However, in the adversarial setting, this natural classiﬁer is quite brittle.

Original: Thus, it isimperative to obtain a robust classiﬁer: 
Revised: Thus, it isimperative to obtain a robust classiﬁer: [Equation] , where

Original: [Equation]
Revised: [Equation]

Original: (9) which relies only on the robust feature x 1 .
Revised: which relies only on the robust feature x 1 .

Original: Illustration of adversarial accuracy.
Revised: Illustration of adversarial accuracy.

Original: In the adversarial setting, an adversary that is only allowed toperturb each feature by a moderate (cid:15) can effectively subvert the natural classiﬁer Tsipras et al.
Revised: In the adversarial setting, an adversary that is only allowed toperturb each feature by a moderate (cid:15) can effectively subvert the natural classiﬁer Tsipras et al.

Original: Inparticular, if (cid:15) = 2 η , an adversary can essentially force each non-robust feature to be anti -correlatedwith the correct label.
Revised: Inparticular, if (cid:15) = 2 η , an adversary can essentially force each non-robust feature to be anti -correlatedwith the correct label.

Original: The following proposition, proved in Appendix C.1, gives the adversarialaccuracies of the natural classiﬁer f nat (8) and the robust classiﬁer f rob (9).
Revised: The following proposition, proved in Appendix C.1, gives the adversarialaccuracies of the natural classiﬁer f nat (6) and the robust classiﬁer f rob (7).

Original: Proposition 1.
Revised: Proposition 1.

Original: Let (cid:15) = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of aclassiﬁer correctly predicting y on the data (7) under (cid:96) ∞ perturbations.
Revised: Let (cid:15) = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of aclassiﬁer correctly predicting y on the data (5) under (cid:96) ∞ perturbations.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Proposition 1 implies that the adversarial accuracy of the natural classiﬁer is < 50% when d ≥ 1 /η 2 .
Revised: Proposition 1 implies that the adversarial accuracy of the natural classiﬁer is < 50% when d ≥ 1 /η 2 .

Original: Even worse, when σ ≤ 1 / 3 and d ≥ 3 /η 2 , the adversarial accuracy of the natural classiﬁer (8) isalways lower than 1% .
Revised: Even worse, when σ ≤ 1 / 3 and d ≥ 3 /η 2 , the adversarial accuracy of the natural classiﬁer (6) isalways lower than 1% .

Original: In contrast, the robust classiﬁer (9) yields a much higher adversarial accuracy(always > 50% ); when σ ≤ (1 − 2 η ) / 3 , its adversarial accuracy will be higher than 99% .
Revised: In contrast, the robust classiﬁer (7) yields a much higher adversarial accuracy(always > 50% ); when σ ≤ (1 − 2 η ) / 3 , its adversarial accuracy will be higher than 99% .

Original: 3.1 Hypocritical Features Are Harmful
Revised: 3.1 Hypocritical Features Are Harmful

Original: The results above reveal the advantages of robust classiﬁers over natural classiﬁers.
Revised: The results above reveal the advantages of robust classiﬁers over natural classiﬁers.

Original: Note that sucha robust classiﬁer can be obtained by adversarial training on the original data (7).
Revised: Note that sucha robust classiﬁer can be obtained by adversarial training on the original data (5).

Original: However, thisdefense effect may not hold when the adversary is allowed to perturb the training data.
Revised: However, thisdefense effect may not hold when the adversary is allowed to perturb the training data.

Original: We show this by analyzing two representative perturbations: the adversarial perturbation [42, 64, 18]and the hypocritical perturbation [63, 28, 64].
Revised: We show this by analyzing two representative perturbations: the adversarial perturbation [41, 17]and the hypocritical perturbation [27, 60].

Original: When applied into the training data, both perturbationsare effective as delusive attacks for naturally trained models.
Revised: When applied into the training data, both perturbations areeffective as delusive attacks for naturally trained models.

Original: In the following, we show that theformer is harmless: adversarial training using a defense budget (cid:15) on the adversarially perturbed datacan still provide test robustness.
Revised: In the following, we show that the former isharmless: adversarial training using a defense budget (cid:15) on the adversarially perturbed data can stillprovide test robustness.

Original: In contrast, the latter is harmful: we ﬁnd that the same defensebudget can only produce non-robust classiﬁers when the training data is hypocritically perturbed.
Revised: In contrast, the latter is harmful: we ﬁnd that the same defense budget canonly produce non-robust classiﬁers when the training data is hypocritically perturbed.

Original: A harmless case.
Revised: A harmless case.

Original: Consider an adversary who is capable of perturbing the training data by an attackbudget (cid:15) .
Revised: Consider an adversary who is capable of perturbing the training data by an attackbudget (cid:15) .

Original: The adversary may choose to shift each feature towards − y .
Revised: The adversary may choose to shift each feature towards − y .

Original: Hence, the learner would seeinput-label pairs ( x , y ) sampled i.i.d.
Revised: Hence, the learner would seeinput-label pairs ( x , y ) sampled i.i.d.

Original: from a training distribution T adv as follows:
Revised: from a training distribution T adv as follows:

Original: [Equation]
Revised: [Equation]

Original: where each feature of the samples from T adv is adversarially perturbed by a moderate (cid:15) .
Revised: where each feature of the samples from T adv is adversarially perturbed by a moderate (cid:15) .

Original: While thesesamples are deviate signiﬁcantly from the original distribution D (7), adversarial training on themusing a defense budget (cid:15) is still able to neutralize the non-robust features.
Revised: While thesesamples are deviate signiﬁcantly from the original distribution D (5), adversarial training on themusing a defense budget (cid:15) is still able to neutralize the non-robust features.

Original: Formally, in Appendix C.2we prove the following theorem.
Revised: Formally, in Appendix C.2we prove the following theorem.

Original: Theorem 1 (Adversarial perturbation is harmless) .
Revised: Theorem 1 (Adversarial perturbation is harmless) .

Original: Assume that the adversarial perturbation in thetraining data T adv (10) is moderate such that η/ 2 ≤ (cid:15) < 1 / 2 .
Revised: Assume that the adversarial perturbation in thetraining data T adv (8) is moderate such that η/ 2 ≤ (cid:15) < 1 / 2 .

Original: Then, the optimal linear (cid:96) ∞ -robustclassiﬁer obtained by minimizing the adversarial risk on T adv with a defense budget (cid:15) is equivalent tothe robust classiﬁer (9).
Revised: Then, the optimal linear (cid:96) ∞ -robustclassiﬁer obtained by minimizing the adversarial risk on T adv with a defense budget (cid:15) is equivalent tothe robust classiﬁer (7).

Original: This theorem indicates that the adversarial perturbation is harmless: (cid:15) -robustness can still be obtainedby adversarial training on such perturbed training data.
Revised: This theorem indicates that the adversarial perturbation is harmless: (cid:15) -robustness can still be obtainedby adversarial training on such perturbed training data.

Original: A harmful case.
Revised: A harmful case.

Original: However, this defense effect can be completely broken by the hypocriticalperturbation.
Revised: However, this defense effect can be completely broken by the hypocriticalperturbation.

Original: That is, the adversary can instead shift each feature towards y .
Revised: That is, the adversary can instead shift each feature towards y .

Original: Hence, the learner wouldsee input-label pairs ( x , y ) sampled i.i.d.
Revised: Hence, the learner wouldsee input-label pairs ( x , y ) sampled i.i.d.

Original: from a training distribution T hyp as follows 1 :
Revised: from a training distribution T hyp as follows:

Original: [Equation]
Revised: [Equation]

Original: 11) where each feature of the samples from T hyp is reinforced by a magnitude of (cid:15) .
Revised: where each feature of the samples from T hyp is reinforced by a magnitude of (cid:15) .

Original: While these samplesbecome more separable, adversarial training on them using the same defense budget will fail toneutralize the hypocritically perturbed features.
Revised: While these samplesbecome more separable, adversarial training on them using the same defense budget will fail toneutralize the hypocritically perturbed features.

Original: Consequently, the resulting classiﬁers will inevitablyhave low adversarial accuracy.
Revised: Consequently, the resulting classiﬁers will inevitablyhave low adversarial accuracy.

Original: We make this formal in the following theorem proved in Appendix C.3.
Revised: We make this formal in the following theorem proved in Appendix C.3.

Original: Theorem 2 (Hypocritical perturbation is harmful) .
Revised: Theorem 2 (Hypocritical perturbation is harmful) .

Original: The optimal linear (cid:96) ∞ -robust classiﬁer obtainedby minimizing the adversarial risk on the perturbed data T hyp (11) with a defense budget (cid:15) is equivalentto the natural classiﬁer (8).
Revised: The optimal linear (cid:96) ∞ -robust classiﬁer obtainedby minimizing the adversarial risk on the perturbed data T hyp (9) with a defense budget (cid:15) is equivalentto the natural classiﬁer (6).

Original: This theorem implies that the conventional defense scheme can only produce non-robust classiﬁers,whose adversarial accuracy is as low as that of the natural classiﬁer (8).
Revised: This theorem implies that the conventional defense scheme can only produce non-robust classiﬁers,whose adversarial accuracy is as low as that of the natural classiﬁer (6).

Original: That is saying, if (cid:15) =
Revised: That is saying, if (cid:15) =

Original: 2 η ,σ ≤ 1 / 3 and d ≥ 3 /η 2 , the classiﬁers cannot get adversarial accuracy better than 1% .
Revised: 2 η ,σ ≤ 1 / 3 and d ≥ 3 /η 2 , the classiﬁers cannot get adversarial accuracy better than 1% .

Original: Implications.
Revised: Implications.

Original: As it turns out, the seemingly beneﬁcial features in T hyp (11) are actually hypocritical.
Revised: As it turns out, the seemingly beneﬁcial features in T hyp (9) are actually hypocritical.

Original: Therefore, the adversary is highly motivated to hide such hypocritical features in the training data, intending to cajole an innocent learner into relying on the non-robust features.
Revised: Therefore, the adversary is highly motivated to hide such hypocritical features in the training data,intending to cajole an innocent learner into relying on the non-robust features.

Revised: While our analytical results only show how to manipulate a simple statistical setting, we note that thehypocritical perturbation is easy to craft on real-world datasets [27, 59, 49].

Revised: A simple method is tosolve the following error-minimizing problem:

Revised: [Equation]

Revised: (10) where f craft is a model trained on clean data, and we call this model the crafting model by following Fowl et al.

Revised: To make the point more clear, let us ﬁrst consider the logistic loss

Original: Intriguingly, wenotice that the natural classiﬁer (8) (i.e., the crafting model used to derive the distribution T hyp )actually has η -robustness.
Revised: L ( f ( x ) , y ) = log(1 + exp( − yf ( x ))) and use the natural classiﬁer (6) as the crafting model.

Revised: In thiscase, the error-minimizing problem has a closed-form solution

Revised: [Equation]

Original:  This is essentially because the non-robust features in the data (7) can resistsmall-magnitude perturbations by design.
Revised: which is exactly the hypocritical perturbation in T hyp (9). Generally, we can apply this technique tonon-convex cases (e.g., deep neural networks) by approximately solving the problem (10) via projected gradient descent (PGD), a standard iterative ﬁrst-order optimization method that is commonlyadopted for crafting test-time adversarial examples [33, 39]. Intriguingly, we notice that the natural classiﬁer (6) (i.e., the crafting model used to derive thesolution (11)) actually has η -robustness. This is essentially because the non-robust features in thedata (5) can resist small-magnitude perturbations by design.

Original: This motivates us to use “slightly robust” classiﬁers as thecrafting model in practice.
Revised: This motivates us to use “slightly robust”classiﬁers as the crafting model in practice.

Original: Indeed, our experimental results show that training the crafting modelwith 0 .
Revised: Indeed, our experimental results show that trainingthe crafting model with 0 .

Original: 25 (cid:15) -robustness performs the best for conducting stability attacks.
Revised: 25 (cid:15) -robustness performs the best for conducting stability attacks.

Original: This is different from theprevious works [64, 18] that use naturally trained models as the crafting model for poisoning.
Revised: This isdifferent from the previous ﬁndings [15, 27, 74, 17] that use naturally trained models as the craftingmodel for poisoning.

Original:  The Necessity of Large Defense Budget
Revised:  The Necessity of Large Defense Budget

Original: We have shown that the hypocritical perturbation is harmful to the conventional adversarial trainingscheme.
Revised: We have shown that the hypocritical perturbation is harmful to the conventional adversarial trainingscheme.

Original: Fortunately, it is possible to strengthen the defense by using a larger defense budget, whilethe crux of the matter is how large the budget is needed.
Revised: Fortunately, it is possible to strengthen the defense by using a larger defense budget, whilethe crux of the matter is how large the budget is needed.

Original: We ﬁnd that the minimum value of the defense budget for a successful defense depends on the speciﬁcdata distribution.
Revised: We ﬁnd that the minimum value of the defense budget for a successful defense depends on the speciﬁcdata distribution.

Original: Let us ﬁrst consider the hypocritical data in T hyp (11).
Revised: Let us ﬁrst consider the hypocritical data in T hyp (9).

Original: In this case, we show that alarger defense budget is necessary in the following theorem proved in Appendix C.4.
Revised: In this case, we show that alarger defense budget is necessary in the following theorem proved in Appendix C.4.

Original: Theorem 3 ( (cid:15) + η is necessary) .
Revised: Theorem 3 ( (cid:15) + η is necessary) .

Original: The optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizingthe adversarial risk on the perturbed data T hyp (11) with a defense budget (cid:15) + η is equivalent to therobust classiﬁer (9).
Revised: The optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizing theadversarial risk on the perturbed data T hyp (9) with a defense budget (cid:15) + η is equivalent to the robustclassiﬁer (7).

Original: Moreover, any defense budget lower than (cid:15) + η will yield classiﬁers that stillrely on all the non-robust features.
Revised: Moreover, any defense budget lower than (cid:15) + η will yield classiﬁers that still rely on allthe non-robust features.

Original: This theorem implies that, in the case of the mixture Gaussian distribution under the threat of (cid:15) -bounded hypocritical perturbations, the learner needs a slightly larger defense budget (cid:15) + η to ensure(cid:15) -robustness.
Revised: This theorem implies that, in the case of the mixture Gaussian distribution under the threat of (cid:15) -bounded hypocritical perturbations, the learner needs a slightly larger defense budget (cid:15) + η to ensure(cid:15) -robustness.

Original: While it is challenging to analyze the minimum value of the defense budget in the general case, thefollowing theorem provides an upper bound of the budget.
Revised: While it is challenging to analyze the minimum value of the defense budget in the general case, thefollowing theorem provides an upper bound of the budget.

Original: Theorem 4 (General case) .
Revised: Theorem 4 (General case) .

Original: For any data distribution and any adversary with an attack budget (cid:15) ,training models to minimize the adversarial risk with a defense budget 2 (cid:15) on the perturbed data issufﬁcient to ensure (cid:15) -robustness.
Revised: For any data distribution and any adversary with an attack budget (cid:15) ,training models to minimize the adversarial risk with a defense budget 2 (cid:15) on the perturbed data issufﬁcient to ensure (cid:15) -robustness.

Original: The proof of Theorem 4 is deferred in Appendix C.5.
Revised: The proof of Theorem 4 is deferred in Appendix C.5.

Original: It implies that a defense budget twice to theattack budget should be safe enough under the threat of stability attacks.
Revised: It implies that a defense budget twice to theattack budget should be safe enough under the threat of stability attacks.

Original: Theorem 3 also suggests thatthe minimum budget might be much smaller than 2 (cid:15) , and it depends on the speciﬁc attack methodsand data distributions.
Revised: Theorem 3 also suggests thatthe minimum budget might be much smaller than 2 (cid:15) , and it depends on the speciﬁc attack methodsand data distributions.

Original: In the following section, we empirically search for an appropriate defensebudget on real-world datasets.
Revised: In the following section, we empirically search for an appropriate defensebudget on real-world datasets.

Original:  Experiments
Revised:  Experiments

Original: In this section, we conduct comprehensive experiments to demonstrate the effectiveness of thehypocritical perturbation as stability attacks on popular benchmark datasets and the necessity of anadaptive defense for better robustness.
Revised: In this section, we conduct comprehensive experiments to demonstrate the effectiveness of thehypocritical perturbations as stability attacks on popular benchmark datasets and the necessity of anadaptive defense for better robustness.

Original:  We conduct stability attacks by applying hypocritical perturbations into the  training set.
Revised: Adversary capability. We focus on the clean-label setting, where an adversary can only providecorrectly labeled but misleading training data. In this setting, the main constraint is to craft perturbations as small as possible [15].

Original: We focuson an (cid:96) ∞ adversary with an attack budget (cid:15) a = 8 / 255 by following [28, 79, 64, 18].
Revised: Thus, we consider an (cid:96) ∞ adversary with an attack budget (cid:15) a = 8 / 255by following Huang et al. [27], Yuan and Wu [74], Tao et al. [60], Fowl et al. We note that thisconstraint is consistent with common research on test-time adversarial examples [1].

Revised: Crafting details.

Revised: We conduct stability attacks by applying hypocritical perturbations into thetraining set.

Revised: Unless otherwise speciﬁed, we craft the perturbations by solving the the error-minimizingobjective (10) with 100 steps of PGD, where a step size of 0 .

Revised: 8 / 255 is used by following Fowl et al.[17].

Original: Our craftingmodel is adversarially trained with a crafting budget (cid:15) c 
Revised: Our crafting model is adversarially trained with a crafting budget (cid:15) c = 0 .

Original:  = 2 / 255 for 10 epochs before generatingperturbations.
Revised: 25 (cid:15) a for 10 epochsbefore generating perturbations. That is, setting (cid:15) c = 2 / 255 performs best, as shown in Figure 2(a). Training details.

Revised: We evaluate the effects of stability attacks on benchmark datasets including

Revised: CIFAR-10/100 [32], SVHN [42], and Tiny-ImageNet [34].

Original: Unless otherwise speciﬁed, we use ResNet-18 [26] as the default architecture for boththe crafting model and the learning model.
Revised: Unless otherwise speciﬁed, we use ResNet-18 [25] as the default architecture for both the crafting model and the learning model.

Original: For adversarial training, we mainly follow the settings inprevious studies [81, 69, 50].
Revised: Foradversarial training, we mainly follow the settings in previous studies [76, 65, 47].

Original: By convention, the defense budget is equal to the attack budget, i.e.,(cid:15) d 
Revised: By convention,the defense budget is equal to the attack budget, i.e., (cid:15) d = 8 / 255 .

Original: = 8 / 255 .
Revised: The networks are trained for 100epochs using SGD with momentum 0.9, weight decay 5 × 10 − 4 , and an initial learning rate of 0.1that is divided by 10 at the 75-th and 90-th epoch. Early stopping is done with holding out 1000examples from the training set. Simple data augmentations such as random crop and horizontal ﬂipare applied.

Original: More details on experimental settings are provided in Appendix D.
Revised: The inner maximization problem during adversarial training is solved by 10-steps PGD(PGD-10) with step size 2 / 255 . [Table] [Table] [Table]

Original: 5.1 Benchmarking (Non-)Robustness
Revised: 5.2 Benchmarking (Non-)Robustness

Original: Attack evaluation.
Revised: Attack evaluation.

Original: We compare our crafted hypocritical perturbation to existing methods, whichwere originally proposed as delusive attacks, including DeepConfuse (which builds an adversarialauto-encoder to generate their perturbations) [16], Unlearnable Examples (which use a min-minbi-level optimization process to pre-train their crafting model) [28], NTGA (which adopts neuraltangent kernels as its crafting model) [79], and Adversarial Poisoning (whose crafting model is simply
Revised: We compare our crafted hypocritical perturbations to existing attack methodsincluding DeepConfuse [15], Unlearnable Examples [27], NTGA [74]

Original: [Table][Table]
Revised: [Table]

Original: [Table] a naturally trained classiﬁer) It is noteworthy that none of these previous works evaluated thetest robustness of their poisoned models.
Revised: and Adversarial Poisoning [17].

Original: Results using ResNet-18 on CIFAR-10 are summarized in Table 2.
Revised: Results using ResNet-18 on CIFAR-10 are summarized in Table 2.

Original: “Natural” denotes the accuracyon natural test data.
Revised: “Natural” denotes theaccuracy on natural test data.

Original: Various test-time adversarial attacks are used to evaluate test robustness,including FGSM, PGD-20/100, CW ∞ ( (cid:96) ∞ version of CW loss [9] optimized by PGD-100)
Revised: Various test-time adversarial attacks are used to evaluate test robustness,including FGSM, PGD-20/100, CW ∞ ( (cid:96) ∞ version of CW loss [8] optimized by PGD-100)

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: AutoAttack (a reliable evaluation metric via an ensemble of diverse attacks [11]).
Revised: AutoAttack (a reliable evaluation metric via an ensemble of diverse attacks [10]).

Original: We observe thatour implementation of stability attacks widely outperforms previous training-time perturbations indegrading the test robustness of PGD-AT [39].
Revised: We observe thatour implementation of stability attacks widely outperforms previous training-time perturbations indegrading the test robustness of PGD-AT [39].

Original: This demonstrates that stability attacks are indeedharmful to the conventional defense scheme.
Revised: This demonstrates that stability attacks are indeedharmful to the conventional defense scheme.

Original: We note that our method increases the natural accuracy.
Revised: We notice the test accuracy of the proposed attack isrelatively high.

Original: This is reasonable, sinceour analysis in Section 3.1 has implied that the hypocritical perturbation canincrease model reliance on the non-robust features, which are predictive but brittle [29].
Revised: This is reasonable, since the proposed method is designed to increase model relianceon the non-robust features, which are predictive but brittle [28].

Original: Moreover, we evaluate the hypocritical perturbation on other benchmark datasets including SVHN, CIFAR-100, and Tiny-ImageNet.
Revised: Moreover, we evaluate stability attacks on other benchmark datasets including SVHN, CIFAR-100,and Tiny-ImageNet.

Original: Both the crafting model and the victim model use the ResNet-18architecture.
Revised: Both the crafting model and the victim model use the ResNet-18 architecture.

Original: Results are summarized in Table 3.
Revised: Results are summarized in Table 3.

Original: “Hyp.” denotes the hypocritical perturbation generated by our crafting model.
Revised: “Hyp.” denotes the hypocritical perturbation generated byour crafting model.

Original: As a comparison, we also evaluate the adversarial perturbation generatedusing the same crafting model (denoted as “Adv.”).
Revised: As a comparison, we also evaluate the adversarial perturbation generatedusing the same crafting model (denoted as “Adv.”).

Original: Again, the results show that the hypocriticalperturbations are more threatening than the adversarial perturbations to standard adversarial training.
Revised: Again, the results show that the hypocriticalperturbations are more threatening than the adversarial perturbations to standard adversarial training.

Original: This phenomenon is consistent with our analytical results in Section 3.1.
Revised: This phenomenon is consistent with our analytical results in Section 3.1.

Original: Besides, we ﬁnd that the hypocritical perturbation can transfer well from ResNet-18 to other architectures, which reliably degrades the test robustness of a wide variety of popular architectures including 
Revised: Besides, we ﬁnd that stability attacks can transfer well from ResNet-18 to other architectures, whichreliably degrade the test robustness of a wide variety of popular architectures including VGG-16 [54],

Original: VGG-16 [57], GoogLeNet [61], DenseNet-121 [27], MobileNetV2 [52], and WideResNet-28-10 [80],as shown in Table 4.
Revised: GoogLeNet [57], DenseNet-121 [26], MobileNetV2 [50], and WideResNet-28-10 [75], as shown [Figure] [Table] [Table]

Revised: in Table

Revised: [Table]

Original: Note that this is a completely black-box setting where the attacker has noknowledge of the victim model’s initialization, architecture, learning rate scheduler, etc. [Figure] [Table]
Revised: Note that this is a completely black-box setting where the attacker has no knowledge ofthe victim model’s initialization, architecture, learning rate scheduler

Original: [Table]
Revised: [Table]

Revised: etc.

Original: Adaptive defense.
Revised: Adaptive defense.

Original: To prevent the harm of stability attacks, our analysis in Section 4 suggests thata larger defense budget would be helpful.
Revised: To prevent the harm of stability attacks, our analysis in Section 4 suggests thata larger defense budget would be helpful.

Original: We ﬁnd that this is indeed the case on CIFAR-10.
Revised: We ﬁnd that this is indeed the case on CIFAR-10.

Original: Asshown in Table
Revised: Asshown in Table

Original: [Table]
Revised: [Table]

Original: a large defense budget (cid:15) d = 14 / 255 for PGD-AT performs signiﬁcantly betterthan the conventional defense budget (cid:15) d = 8 / 255 .
Revised: a large defense budget (cid:15) d = 14 / 255 for PGD-AT performs signiﬁcantly betterthan the conventional defense budget (cid:15) d = 8 / 255 .

Original: We also combine several data augmentations with PGD-AT as defenses by following Fowl et al.
Revised: We also combine several data augmentationswith PGD-AT as defenses by following Fowl et al.

Original: The results show that they are beneﬁcial, whiletheir improvements are inferior to PGD-AT with (cid:15) d = 14 / 255 .
Revised: The results show that they are beneﬁcial,while their improvements are inferior to PGD-AT with (cid:15) d = 14 / 255 .

Original: In addition, we adopt other adversarial training variants including TRADES [81] and MART [69] to defend against the hypocriticalperturbation [Table]
Revised: In addition, we adopt otheradversarial training variants including TRADES [76] and MART [65] to defend against stabilityattacks.

Original: and ﬁnd that they achieve comparable defense effects with large defense budgets.
Revised: Again, they can exhibit good defense effects with large defense budgets.

Original: Finally, we note that the adaptive defense has several limitations: i) robust accuracy is improved at thecost of natural accuracy; ii) ﬁnding an appropriate defense budget is time-consuming for adversarialtraining; iii) adversarial training with large budgets may lead to learning obstacles such as inherentlarge sample complexity [53].

Original: We leave the detailed study of these questions as future work.

Original: 5.2 Ablation Studies
Revised: 5.3 Ablation Studies

Original: In this part, we conduct a set of experiments to provide an empirical understanding of the proposedattack.
Revised: In this part, we conduct a set of experiments to provide an empirical understanding of the proposedattack.

Original: We train ResNet-18 using PGD-AT on CIFAR-10 by following the same settings describedin Appendix D unless otherwise speciﬁed.
Revised: We train ResNet-18 using PGD-AT on CIFAR-10 by following the same settings describedin Section 5.1 unless otherwise speciﬁed.

Original: Analysis on the crafting method.
Revised: Analysis on the crafting method.

Original: Different from previous work, we use “slightly robust” classiﬁersas our crafting model.
Revised: Different from previous work, we use “slightly robust” classiﬁersas our crafting model.

Original: Figure 2(a) shows that this technique greatly improves the potency of the attack,where the crafting budget (cid:15)
Revised: Figure 2(a) shows that this technique greatly improves the potency of the attack,where the crafting budget (cid:15)

Original: c
Revised: c

Original: = 2 / 255 performs best in degrading test robustness.
Revised: = 2 / 255 performs best in degrading test robustness.

Original: We also observethat training the crafting model for 10 ∼ 80 epochs works well in Figure 2(b), and that optimizing thecrafted perturbations over 100 steps performs well in Figure 2(c).
Revised: We also observethat training the crafting model for 10 ∼ 80 epochs works well in Figure 2(b), and that optimizing thecrafted perturbations over 100 steps performs well in Figure 2(c).

Original: Finally, we note that Fowl et al.[18] also tried to use adversarially trained models as the crafting model, but they failed to produce effective attacks in this way.
Revised: Finally, we note that Fowl et al.[17] also tried to use adversarially trained models as the crafting model, but they failed to produce effective attacks in this way.

Original: This is mainly because they adopted adversarial perturbations as poisons,which, as we observed, are inferior in degrading test robustness.
Revised: This is mainly because they adopted adversarial perturbations as poisons,which, as we observed, are inferior in degrading test robustness.

Original: Ablation on defense budget.
Revised: Ablation on defense budget.

Original: As discussed in Section 4, we are motivated to ﬁnd the appropriatedefense budget (cid:15) d in the range [ (cid:15) ∼ 2 (cid:15) ] .
Revised: As discussed in Section 4, we are motivated to ﬁnd the appropriatedefense budget (cid:15) d in the range [ (cid:15) ∼ 2 (cid:15) ] .

Original: Figure 2(d) shows that the optimal defense budgets against theproposed attack for PGD-AT, TRADES, and MART are 14 / 255 , 12 / 255 , and 14 / 255 , respectively.
Revised: Figure 2(d) shows that the optimal defense budgets against theproposed attack for PGD-AT, TRADES, and MART are 14 / 255 , 12 / 255 , and 14 / 255 , respectively.

Original: We also observe that all these adversarial training variants are inferior when using the conventionaldefense budget 8 / 255 .
Revised: We also observe that all these adversarial training variants are inferior when using the conventionaldefense budget 8 / 255 .

Original: Less data.
Revised: Less data.

Original: We follow Fowl et al.
Revised: We follow Fowl et al.

Original: [18] to test the effectiveness of attacks by varying the proportionof clean data and perturbed data.
Revised: [17] to test the effectiveness of attacks by varying the proportionof clean data and perturbed data.

Original: Attacks are then considered effective if they cannot signiﬁcantlyincrease performance over training on the clean subset alone.
Revised: Attacks are then considered effective if they cannot signiﬁcantlyincrease performance over training on the clean subset alone.

Original: As shown in Table 6, the proposedattack often degrades the test robustness below what one would achieve using full clean dataset.
Revised: As shown in Table 6, the proposedattack often degrades the test robustness below what one would achieve using full clean dataset.

Original: More importantly, the hypocritical perturbations are consistently more harmful than the adversarialperturbations.
Revised: More importantly, the hypocritical perturbations are consistently more harmful than the adversarialperturbations.

Original: This again veriﬁes the superiority of hypocritical pertubrations as stability attacks.
Revised: This again veriﬁes the superiority of hypocritical pertubrations as stability attacks.

Original: Effect on natural training.
Revised: Effect on natural training.

Original: As a sanity check, we include the test accuracy of naturally trainedmodels on CIFAR-10 in Appendix B Table 12.
Revised: As a sanity check, we include the test accuracy of naturally trainedmodels on CIFAR-10 in Appendix B Table 11.

Original: It shows that without adversarial training, the testrobustness of the models becomes very poor.
Revised: It shows that without adversarial training, the testrobustness of the models becomes very poor.

Original: Thus, the goal of stability attacks is immediatelyachieved.
Revised: Thus, the goal of stability attacks is immediatelyachieved.

Original: On the other hand, We ﬁnd that our HFs can degrade the test accuracy from 94 .
Revised: On the other hand, We ﬁnd that our HFs can degrade the test accuracy from 94 .

Original: 23%to 75 .
Revised: 23%to 75 .

Original: 92% , though this is not the main focus of this work.
Revised: 92% , though this is not the main focus of this work.

Original: We also observe that Adversarial
Revised: We also observe that Adversarial

Original: Poisoning [18] is the most effective method in degrading the test accuracy of naturally trained models.
Revised: Poisoning [17] is the most effective method in degrading the test accuracy of naturally trained models.

Original: This observation is consistent with Fowl et al.
Revised: This observation is consistent with Fowl et al.

Original:  Related Work
Revised:  Related Work

Original: Adversarial training.
Revised: Adversarial training.

Original: The presence of non-robust features has been demonstrated on popularbenchmark datasets [29, 31], which naturally leads to model vulnerability to adversarial examples [67,58].
Revised: The presence of non-robust features has been demonstrated on popularbenchmark datasets [28, 30], which naturally leads to model vulnerability to adversarial examples [63,55].

Original: To improve test robustness against adversarial examples, adversarial training methods have beendeveloped [23, 39, 72, 81, 65, 46, 74, 84, 62].
Revised: To improve test robustness against adversarial examples, adversarial training methods have beendeveloped [22, 39, 68, 76, 61, 45, 70, 78, 58].

Original: Usually, adversarial training using a defense budget (cid:15)is expected to improve model robustness against (cid:15) -bounded adversarial examples.
Revised: Usually, adversarial training using a defense budget (cid:15)is expected to improve model robustness against (cid:15) -bounded adversarial examples.

Original: Thus, to break thisdefense, a direct way is to enlarge the typical (cid:15) -ball used to constrain the attack; however, this mayrisk changing the true label [8, 66].
Revised: Thus, to break thisdefense, a direct way is to enlarge the typical (cid:15) -ball used to constrain the attack; however, this mayrisk changing the true label [7, 62].

Original: In this work, we aim to show that it is possible to achieve this byslightly perturbing the training data without enlarging the (cid:15) -ball.
Revised: In this work, we aim to show that it is possible to achieve this byslightly perturbing the training data without enlarging the (cid:15) -ball.

Original: Data poisoning.
Revised: Data poisoning.

Original: Data poisoning attacks, which manipulate training data to cause the resultingmodels to fail during inference [3], can be divided into availability attacks (to degrade overalltest performance)
Revised: Data poisoning attacks, which manipulate training data to cause the resultingmodels to fail during inference [2], can be divided into availability attacks (to degrade overalltest performance)

Original: [4, 75, 41, 47, 18] and integrity attacks (to cause speciﬁc misclassiﬁcations)
Revised: [3, 71, 40, 46, 17] and integrity attacks (to cause speciﬁc misclassiﬁcations)

Original: While the stability attacks considered in this work may be reminiscent ofbackdoor attacks [10], we note that they share several key differences.
Revised: While the stability attacks considered in this work may be reminiscent ofbackdoor attacks [9], we note that they share several key differences.

Original: First, stability attacksaim to hinder adversarial training with well-deﬁned (cid:15) -robustness, while backdoor attacks mainlyfocus on embedding exploits (that can be invoked by pre-speciﬁed triggers) into naturally trainedmodels [22, 51, 68].
Revised: First, stability attacks aim tohinder adversarial training with well-deﬁned (cid:15) -robustness, while backdoor attacks mainly focus onembedding exploits (that can be invoked by pre-speciﬁed triggers) into naturally trained models [21,48, 64].

Original: Second, stability attacks only perturb the inputs slightly, while many works onbackdoor attacks require mislabeling [25, 37, 45, 36].
Revised: Second, stability attacks only perturb the inputs slightly, while many works on backdoorattacks require mislabeling [24, 37, 44, 36].

Original: Thus, backdoor defenses [7, 73, 35] might notbe directly applied to resist stability attacks.
Revised: Thus, backdoor defenses [6, 69, 35] might not be directlyapplied to resist stability attacks.

Original: Additional related works are discussed in Appendix A.
Revised: Additional related works are discussed in Appendix A.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we establish a framework to study the robustness of adversarial training against stabilityattacks.
Revised: In this work, we establish a framework to study the robustness of adversarial training against stabilityattacks.

Original: We unveil the threat of stability attacks—small hypocritical perturbations applied intothe training data sufﬁce to hinder conventional adversarial training.
Revised: We unveil the threat of stability attacks—small hypocritical perturbations applied into thetraining data sufﬁce to hinder conventional adversarial training.

Original: The conventional defensebudget (cid:15) is insufﬁcient under the threat.
Revised: The conventional defense budget (cid:15) isinsufﬁcient under the threat.

Original: To resist it, we suggest that practitioners should consider alarger defense budget of no more than 2 (cid:15) .
Revised: To resist it, we suggest that practitioners should consider a larger defensebudget of no more than 2 (cid:15) .

Original: Our theoretical analysis also explains why hypocriticalperturbations work as stability attacks—they can reinforce the non-robust features and misleadthe learner.
Revised: Our theoretical analysis also explains why hypocritical perturbations workas stability attacks—they can reinforce the non-robust features and mislead the learner.

Original: Experiments demonstrate that hypocritical perturbations are harmful to conventionaladversarial training on benchmark datasets, and enlarging the defense budget is essential for mitigatinghypocritical pertubrations.
Revised: Experimentsdemonstrate that stability attacks are harmful to conventional adversarial training on benchmarkdatasets, and enlarging the defense budget is essential for mitigating hypocritical pertubrations.

Original: Future work includes relaxing the assumption that the adversary perturbsall the entire training set and designing more effective stability attacks against adversarial training.
Revised: Future work includes relaxing the assumption that the adversary perturbs all the entire training setand designing more effective stability attacks against adversarial training.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...(

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 2.2 and Section 7.(c)
Revised: [Yes] See Section 2.2 and Section 7.

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Appendix F.(d)
Revised: [Yes] See Section 2.2.

Original:  Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: [Table] results (either in the supplemental material or as a URL)?
Revised: results (either in the supplemental material or as a URL)?

Original: [Yes] See the supplemental material.
Revised: [Yes] See the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5.1.[Table]
Revised: [Yes] See Section 5.1.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments

Original: [Table] multiple times)?
Revised: multiple times)?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix B.

Original: [Table]
Revised: [Table]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: internal cluster, or cloud provider)?
Revised: [Table] [Table] [Table] [Table]

Original: [Yes] All experiments are run on a single NVIDIA GeForce
Revised: [Table] [Table] [Table] If

Original: [Table]
Revised: [Table]

Original: RTX 3090 GPU.

Original: If you are using existing assets (e 
Revised: (e [Table] g [Table]

Revised: data

Revised: [Table]

Original: g., code, data, models) or curating/releasing new assets...
Revised: ) or [Table] /releasing [Table]

Revised: assets..

Original: [Table]
Revised: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: If [Table] , did you cite the [Table]

Original: [Yes] [Table]
Revised: [Table]

Original: (b) Did you mention the license of the assets?
Revised: b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-

Revised: [Table]

Original: ing/curating?
Revised: ing/curating?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: informa-
Revised: informa-

Revised: [Table]

Original: tion or offensive content?
Revised: tion or offensive content?

Original: [N/A]
Revised: [N/A]

Original:  If you used crowdsourcing or conducted research with human subjects...(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: [Table] If you used crowdsourcing or conducted research with human subjects...

Original: [N/A]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable
Revised: a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Table][Table]
Revised: [Table]

Original: [Table] ](c [Table] Did you include the
Revised: [N/A

Original: [Table]
Revised: [Table]

Original: paid to participants
Revised: b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)

Original: [Table]the [Table]
Revised: [Table]

Original: onparticipant compensation [Table] [Table] A Additional Related Work
Revised: approvals, if applicable?

Original: In this part, we discuss several independent (or concurrent) works that are related to this work.

Original: Zhu et al.

Original: [86] study the effect of conventional adversarial training on differentiating noisy labels, while Zhang 549 et al.

Original: [83] show that deliberately injected noisy labels may serve as a regularization that alleviates robustoverﬁtting.

Original: Our results focus on the clean-label setting and provide evidence that conventional adversarial 551 training can be hindered without modifying the labels.

Original: Yu et al.

Original: [78] suggest explaining the success of availability attacks from the perspective of shortcuts.

Original: They further 553 adopt pre-trained models to extract useful features for mitigating model reliance on the shortcuts.

Original: This directionis orthogonal to ours.

Original: Liu et al.

Original: [38] improve the effectiveness of unlearnable examples [28] by generating grayscale perturbations 556 and using data augmentations.

Original: They also conclude that conventional adversarial training will prevent a dropin accuracy measured both on clean images and adversarial images.

Original: Contrary to them, we show that, both 558 theoretically and empirically, conventional adversarial training can be hindered by hypocritical perturbations,and we further analyze the necessity of enlarging the defense budget to resist stability attacks.

Original: Gao et al.

Original: [20] revisit the trade-off between adversarial robustness and backdoor robustness [71].

Original: They conclude 561 that backdoor attacks are ineffective when the defense budget of adversarial training surpasses the triggermagnitude.

Original: In contrast, our results indicate that stability attacks are still harmful to adversarial training when the 563 defense budget is not large enough.

Original: In a simple statistical setting, a defense budget (cid:15) + η is necessary (where ηis a positive number).

Original: In the general case, a defense budget of 2 (cid:15) is sufﬁcient.

Original: In our experiments, a defense 565 budget of about 1 . 5 (cid:15) ∼ 1 .

Original: 75 (cid:15) provides the best empirical (cid:15) -robustness.

Original: Wang et al.

Original: [70] argue that it is necessary to use robust features for compromising adversarial training.

Original: To this 567 end, they adopt a relatively large attack budget (cid:15) a = 32 / 255 for crafting their poisons (they use one type ofadversarial perturbations), and show that their poisons can decrease the performance of the models trained using 569 smaller defense budgets (such as (cid:15) d = 8 / 255 and (cid:15) d = 16 / 255 ).

Original: In contrast, we focus on a more realisticsetting that does not require a larger attack budget.

Original: We demonstrate that it is possible to hinder adversarial 571 training when (cid:15) a = (cid:15) d . Furthermore, we provide both theoretical and empirical results showing how to adapt the 572 defense to maintain robustness. Fu et al. [19] explore how to protect data privacy against adversarial training.
Revised: [N/A

Original: The main purpose of their poisons 574 is to compromise adversarial training by requiring the perturbation budget of their poisons to be larger than thatof adversarial training.

Original: In this way, they show that the natural accuracy of the adversarially trained models can 576 be largely decreased, let alone robust accuracy.

Original: From this perspective, our work is complementary to theirs.

Original: Wepursue to not increase the attack budget of stability attacks, keeping it as small as possible.

Original: We successfully 578 demonstrate that stability attacks are still harmful to conventional adversarial training without enlarging theattack budget.

Original: This makes the threat of stability attacks more insidious than that of Fu et al.

Original: On the other hand, we ﬁnd that our implementation of stability attacks using hypocritical perturbations has some 581 similarities to the robust unlearnable examples in Fu et al.

Original: Speciﬁcally, although the robust unlearnableexamples are generated via a complicated min-min-max optimization process [19], we notice that their noise 583 generator can be viewed as an adversarially trained model.

Original: This implies that the robust error-minimizing (REM)noise [19] might be useful in demonstrating the feasibility of stability attacks.

Original: To verify this, we run the source 585 code from the authors with default hyperparameters 2 , and compare our crafted hypocritical perturbation withtheir generated noise under the setting of stability attacks.

Original: For a fair comparison, here we apply a very simple 587 trick called EOT [2] in our method, since the trick is also used by REM [19].

Original: The additional time cost of the

Original: EOT trick is very small and negligible.

Original: Our experimental results, shown in Table 7, demonstrate that the robust error-minimizing noise is also effective 590 as stability attacks, though it was originally proposed as a delusive attack.

Original: It is noteworthy that the robustaccuracy is not evaluated in [19].

Original: In this sense, the effectiveness of REM as an stability attack can be regarded as 592 one of our novel ﬁndings.

Original: Importantly, our method outperforms REM in terms of the robust accuracy against AutoAttack.

Original: Since AutoAttack 594 is the most reliable evaluation metric of model robustness among the test-time attacks [11], this result indicatesthat our method is reliably more effective than REM in degrading model robustness.

Original: It is also noteworthy that 596 our method is signiﬁcantly more efﬁcient than REM, as shown in Table 8.

Original: We note that the efﬁciency of ourmethod is largely due to the fact that our crafting model is fast to train.

Original: Speciﬁcally, the time cost of training our 598 crafting model is only 0.3 hours, while it takes 20.8 hours for REM.

Original: That is, our crafting model is nearly 70times faster to train than that of REM.

Original: In short, our method is not only more effective, but also more efﬁcient, 600 than REM as a stability attack.

Original: [Table][Table]
Revised: [Table]

Original: B Omitted Tables

Original: [Table] [Table] various adaptive defenses on the hypocritically perturbed CIFAR-10. We report mean and standard deviation over 3 random runs
Revised: c) Did you include the estimated hourly wage paid to participants and the total amount spent on

Original: [Table][Table]
Revised: [Table]

Original: C Proofs

Original: [Table]

Original: In this section, we provide the proofs of our theoretical results in Section 3 and Section 4.

Original: C.1 Proof of Proposition

Original: Proposition 1 (restated).

Original: Let (cid:15) = 2 η and denote by A adv ( f ) the adversarial accuracy, i.e., the probability of a 606 classiﬁer correctly predicting y on the data (7) under (cid:96) ∞ perturbations.

Original: Then, we have

Original: [Equation]

Original: Proof.

Original: Recalling that in Equation (8), we have the natural classiﬁer:f nat ( x ) :

Original: = sign( w (cid:62) nat x ) , where w nat :=

Original: [1 , η, . . .

Original: , η ] , (12) and in Equation (9), the robust classiﬁer is deﬁned as:

Original: [Equation]

Original: := sign( w (cid:62)

Original: [Equation]

Original: x ) , where w rob :=

Original: Then, the adversarial accuracy of the natural classiﬁer on the data D (7) is

Original: A

Original: [Equation]

Original: (cid:34)

Original: [Equation]

Original: [Equation]

Original: i =

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:1)(cid:33)(cid:35)

Original: [Equation]

Original: [Equation]

Original: ≤

Original: [Equation]

Original: Pr (cid:40) y · (cid:32) 1 · (cid:0)

Original: [Equation]

Original: ( y

Original: [Equation]

Original: ) (cid:1)

Original: [Equation]

Original: i = η · (cid:0) N ( yη

Original: [Equation]

Original: ) − (cid:15) (cid:1)(cid:33)

Original: [Equation]

Original: (cid:41) =

Original: 1 − Pr (cid:40)

Original: N (

Original: [Equation]

Original: σ 2 ) + η d +1 (cid:88) i =

Original: N ( η − (cid:15)

Original: [Equation]

Original: σ 2 ) <(cid:41) =

Original: Pr (cid:40)

Original: N (

Original: [Equation]

Original: σ 2 ) + η d +1 (cid:88) i =

Original: N ( η − (cid:15), σ 2 ) >(cid:41) =

Original: Pr (cid:40)

Original: N (0 , 1) < 1 − dησ (cid:112) 1 + dη(cid:41) .

Original: Similarly, the adversarial accuracy of the robust classiﬁer on the data D (7) is

Original: [Equation]

Original: C.2 Proof of Theorem 1

Original: The following theorems rely on the analytical solution of optimal linear (cid:96) ∞ -robust classiﬁer on mixture Gaussiandistributions.

Original: Concretely, the optimization problem is to minimize the adversarial risk on a distribution (cid:98) D with a 616 defense budget ˆ (cid:15) : [Equation] where R ˆ (cid:15) adv ( f, (cid:98) D ) : =
Revised: participant compensation?

Original: E

Original: [Equation]

Original: where f ( x ) = sign( w (cid:62) x + b ) , and 1 ( · ) denotes the indicator function.

Original: We note that optimal linear robust classiﬁers have been obtained for certain data distributions in previous 619 work [67, 29, 14, 30, 76, 64].

Original: Here, our goal is to establish similar optimal linear robust classiﬁers for theclassiﬁcation tasks in our setting.

Original: We only employ linear classiﬁers, since it is highly nontrivial to consider 621 non-linearity for adversarial training on mixture Gaussian distributions [14].

Original: Lemma 1.

Original: Assume that the adversarial perturbation in data T adv (10) is moderate such that η/ 2 ≤ (cid:15) < 1 / 2 .

Original: Then, minimizing the adversarial risk (16) on the data T adv with a defense budget (cid:15) can result in a classiﬁer thatassigns 0 weight to the features x i for i ≥ 2 .

Original: Proof.

Original: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T adv , which can be written as follows:

Original: [Equation]

Original: δ )] > 0 |

Original: [Equation]

Original: −(cid:27) ·

Original: Pr

Original: [Equation]

Original: x

Original: [Equation]

Original: y )

Original: ∼T adv

Original: [Equation]

Original: y

Original: [Equation]

Original: − 1 } +

Original: Pr

Original: [Equation]

Original: ∼T adv

Original: [Equation]

Original: min

Original: [Equation]

Original: δ (cid:107) ∞

Original: [Equation]

Original: (cid:15)

Original: [ f ( x + δ )]

Original: [Equation]

Original: | y

Original: [Equation]

Original: + (cid:27) ·

Original: Pr ( x

Original: [Equation]

Original: y )

Original: ∼T adv { y = +1 }

Original: = Pr (cid:40) max (cid:107) δ (cid:107)

Original: [Equation]

Original: (cid:15) (cid:34) w 1 ( N ( (cid:15) −

Original: [Equation]

Original: σ 2 ) + δ 1 ) + d +1 (cid:88)

Original: i =

Original: w i ( N ( (cid:15) − η

Original: [Equation]

Original: σ 2 ) + δ i ) + b (cid:35) >

Original: (cid:41) (cid:124) (cid:123)(cid:122) (cid:125) R (cid:15)

Original: adv

Original: ( f

Original: [Equation]

Original: [Equation]

Original: 1) adv ) · 1+ Pr (cid:40) min (cid:107)

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: (cid:15) (cid:34)

Original: w

Original: [Equation]

Original: ( N (1 − (cid:15)

Original: [Equation]

Original: σ

Original: [Equation]

Original: [Equation]

Original: + d

Original: [Equation]

Original: (cid:88) i =

Original: w i ( N ( η − (cid:15), σ 2 ) + δ i ) + b (cid:35)

Original: [Equation]

Original: (cid:41) (cid:124) (cid:123)(cid:122) (cid:125) R (cid:15)

Original: adv ( f, T (+1) adv ) · 1(17)

Original: Consider an optimal solution w in which w i > 0 for some

Original: i ≥ 2 .

Original: Then, we have

Original: [Equation]

Original: 18) where w nat :=

Original: [1 , η, . . .

Original: , η ] as in Equation (8).

Original: Since w i > 0 , B is maximized when δ i = (cid:15) .

Original: Thus, thecontribution of terms depending on w i to B is a normally-distributed random variable with mean 2 (cid:15) − η .

Original: Since 630 2 (cid:15) − η ≥ 0 , setting w i to zero can only decrease the risk.

Original: This contradicts the optimality of w .

Original: Formally,

Original: [Equation]

Original: We can also assume w i < 0 and similar contradiction holds.

Original: [-Therefore, minimizing the adversarial risk on T adv 632 leads to w i = 
-]
Revised: [-Therefore, minimizing the adversarial risk on T adv 632 leads to w i = 
-]

Original: for i ≥ 2 .

Original: Lemma 2.

Original: Assume that the adversarial perturbation in data T adv (10) is moderate such that η/ 2 ≤ (cid:15) < 1 / 2 .

Original: Then, minimizing the adversarial risk (16) on the data T adv with a defense budget (cid:15) results in a classiﬁer thatassigns a positive weight to the feature x 1 .

Original: Proof.

Original: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T adv , which has been written in Equation (17).

Original: Consider an optimal solution w in which w 1 ≤ 0 .

Original: Then, we have

Original: [Equation]

Original: Since w 1 ≤ 0 , D is maximized when δ 1 = − (cid:15) .

Original: Thus, the contribution of the term depending on w 1 to D is a 640 normally-distributed random variable with mean − 1 .

Original: Since the mean is negative, setting w 1 to be positive candecrease the risk.

Original: This contradicts the optimality of w .

Original: Formally,

Original: [Equation]

Original: [Equation]

Original: where p > 0 is any positive number.

Original: Therefore, minimizing the adversarial risk on T adv leads to w 1 > 0 .

Original: Theorem 1 (restated).

Original: Assume that the adversarial perturbation in the training data T adv (10) is moderate such 644 that η/ 2 ≤ (cid:15) < 1 / 2 .

Original: Then, the optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizing the adversarial riskon T adv with a defense budget (cid:15) is equivalent to the robust classiﬁer (9).

Original: Proof.

Original: [-By Lemma 1 and Lemma 2, we have w 1 > 0 and w i = 
-]
Revised: [-By Lemma 1 and Lemma 2, we have w 1 > 0 and w i = 
-]

Original: ( i ≥ 2 ) for an optimal linear (cid:96) ∞ -robust 647 classiﬁer.

Original: Then, the adversarial risk on the distribution T adv can be simpliﬁed by solving the inner maximizationproblem ﬁrst.

Original: Formally,

Original: [Equation]

Original: which is equivalent to the natural risk on a mixture Gaussian distribution D tmp : x ∼ N ( y · µ tmp , σ 2 I ) ,

Original: whereµ tmp

Original: = (1 − 2 (cid:15), 0 , . . . , 0) .

Original: We note that the Bayes optimal classiﬁer for D tmp is f tmp ( x ) = sign( µ (cid:62) tmp x ) .

Original: Speciﬁcally, the natural risk

Original: [Equation]

Original: which is minimized when w 1 = 1 − 2 (cid:15) > 0 and b = 0 .

Original: That is, minimizing the adversarial risk R (cid:15)

Original: adv ( f, T adv ) 653 can lead to an optimal linear (cid:96) ∞ -robust classiﬁer f tmp ( x ) .

Original: Meanwhile, f tmp ( x ) is equivalent to the robust 654 classiﬁer (9).

Original: This concludes the proof of the theorem.

Original: C.3 Proof of Theorem 2

Original: Lemma 3.

Original: Minimizing the adversarial risk (16) on the data T hyp (11) with a defense budget (cid:15) results in a 657 classiﬁer that assigns positive weights to the features x i for i ≥ 1 .

Original: Proof.

Original: We prove the lemma by contradiction.

Original: The goal is to minimize the adversarial risk on the distribution T hyp , which can be written as follows:

Original: [Equation]
Revised: [N/A]

Original: Consider an optimal solution w in which w i ≤ 0 for some i ≥ 1 .

Original: Then, we have

Original: [Equation]

Original: ) where w nat :=

Original: [1 , η, . . .

Original: , η ] as in Equation (8).

Original: Since w i ≤ 0 , H is maximized when δ i = − (cid:15) .

Original: Thus, thecontribution of terms depending on w i to H is a normally-distributed random variable with mean − [ w nat ]

Original: i − 2 (cid:15) .

Original: 663 Since the mean is negative, setting w i to be positive can decrease the risk.

Original: This contradicts the optimality of w .

Original: Formally,

Original: [Equation]

Original: where p > 0 is any positive number.

Original: Therefore, minimizing the adversarial risk on T hyp leads to

Original: Theorem 2 (restated).

Original: The optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizing the adversarial risk on 668 the perturbed

Original: T hyp (11) with a defense budget (cid:15) is equivalent to the natural classiﬁer (8).

Original: Proof.

Original: By Lemma 3, we have w i > 0 for i ≥ 1 for an optimal linear (cid:96) ∞ -robust classiﬁer.

Original: Then, we have

Original: [Equation]

Original: (27) which is equivalent to the natural risk on the mixture Gaussian distribution D : x ∼ N ( y · w nat , σ 2 I ) , wherew nat = (1 , η, . . .

Original: , η ) .

Original: We note that the Bayes optimal classiﬁer for D is f nat ( x ) = sign( w (cid:62) nat x ) .

Original: Speciﬁcally, 672 the natural risk

Original: [Equation]

Original: 1 2 , (28) which is minimized when w 1 = 1 , w i =

Original: η for i ≥ 2 , and b = 0 .

Original: That is, minimizing the adversarial risk 674 R (cid:15) adv ( f, T hyp ) can lead to an optimal linear (cid:96) ∞ -robust classiﬁer f nat ( x ) , which is equivalent to the natural 675 classiﬁer (8).

Original: This concludes the proof of the theorem.

Original: C.4 Proof of Theorem

Original: Lemma 4.

Original: Minimizing the adversarial risk (16) on the data T hyp (11) with a defense budget (cid:15) + η can result in 679 a classiﬁer that assigns 0 weight to the features x i for i ≥ 2 .

Original: Proof.

Original: The goal is to minimize the adversarial risk on the distribution T hyp , which can be written as follows:

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: Consider an optimal solution w in which w i > 0 for some

Original: i ≥ 2 .

Original: Then, we have

Original: [Equation]

Original: ) where w nat :=

Original: [1 , η, . . .

Original: , η ] .

Original: Since w i > 0 , J is maximized when δ i = (cid:15) + η .

Original: Thus, the contribution of termsdepending on w i to J is a normally-distributed random variable with mean 0 .

Original: Thus, setting w i to zero will not 684 increase the risk.

Original: Formally, we have

Original: [Equation]

Original: [Equation]

Original: We can also assume w i < 0 and a similar argument holds.

Original: Similar arguments also hold for R (cid:15) + η adv ( f

Original: [Equation]

Original: T (+1) hyp ) .

Original: [-Therefore, minimizing the adversarial risk on T hyp can lead to w i = 
-]
Revised: [-Therefore, minimizing the adversarial risk on T hyp can lead to w i = 
-]

Original: for i ≥ 2 .

Original: Theorem 3 (restated).

Original: The optimal linear (cid:96) ∞ -robust classiﬁer obtained by minimizing the adversarial risk on 688 the perturbed data T hyp (11) with a defense budget (cid:15) + η is equivalent to the robust classiﬁer (9).

Original: Moreover, any 689 defense budget lower than (cid:15) + η will yield classiﬁers that still rely on all the non-robust features.

Original: Proof.

Original: [-By Lemma 4, we have w i = 
-]
Revised: [-By Lemma 4, we have w i = 
-]

Original: ( i ≥ 2 ) for an optimal linear (cid:96) ∞ -robust classiﬁer.

Original: Also, the robust 691 classiﬁer will assign a positive weight to the ﬁrst feature.

Original: This is similar to the case in Lemma 2 and we omit theproof here.

Original: Then, we have

Original: [Equation]

Original: which is equivalent to the natural risk on a mixture Gaussian distribution D tmp : x ∼ N ( y · µ tmp , σ 2 I ) ,where µ tmp = (1 − η, 0 , .

Original: We note that the Bayes optimal classiﬁer for D tmp is f tmp ( x ) = sign( µ (cid:62) tmp x ) .

Original: Speciﬁcally, the natural risk

Original: [Equation]

Original: which is minimized when w 1 = 1 − η > 0 and b = 0 .

Original: That is, minimizing the adversarial risk R (cid:15) + η adv ( f, T hyp )can lead to an optimal linear (cid:96) ∞ -robust classiﬁer f tmp ( x ) , which is equivalent to the robust classiﬁer (9).

Original: Moreover, when the defense budget (cid:15) d is less than (cid:15) + η , the condition in Lemma 4 no longer holds.

Original: Instead, in 699 this case, the robust classiﬁer will assign positive weights to the features (i.e., w i > 0 for i ≥ 1 ).

Original: This is similarto the case in Lemma 3, and thus we omit the proof here.

Original: Consequently, this yields classiﬁers that still rely on all 701 the non-robust features.

Original: C.5 Proof of Theorem 4

Original: Theorem 4 (restated).

Original: For any data distribution and any adversary with an attack budget (cid:15) , training models to 705 minimize the adversarial risk with a defense budget 2 (cid:15) on the perturbed data is sufﬁcient to ensure (cid:15) -robustness.

Original: Proof.

Original: For clarity, we rewrite the adversarial risk in (2) with a defense budget (cid:15) as follows:

Original: [Equation]

Original: where T = { ( x i , y i ) } ni =1 denotes the empirical training data.

Original: Consider any adversary with an attack budget (cid:15) , who can perturb x to x + p such that (cid:107) p (cid:107) ≤ (cid:15) .

Original: Then, thelearner will receive a perturbed version of training data T (cid:48)

Original: = { ( x i + p

Original: i , y i ) } ni =1 .

Original: For any perturbed data point ( x i + p

Original: i , y i ) , we have

Original: [Equation]

Original: [Equation]

Original: By summarizing the training points, we have

Original: [Equation]

Original: That is, the adversarial risk with a defense budget 2 (cid:15) on the perturbed data is an upper bound of the adversarial 713 risk with a defense budget (cid:15) on the original data.

Original: Therefore, a defense budget 2 (cid:15) is sufﬁcient to ensure thelearning of (cid:15) -robustness.

Original: D Experimental Settings

Original: Adversary capability.

Original: We focus on the clean-label setting, where an adversary can only provide correctly 717 labeled but misleading training data.

Original: In this setting, the main constraint is to craft perturbations as small aspossible [16].

Original: Thus, we consider an (cid:96) ∞ adversary with an attack budget (cid:15) a = 8 / 255 by following Huang et al.

Original: 719 [28], Yuan and Wu [79], Tao et al.

Original: [64], Fowl et al.

Original: We note that this constraint is consistent with commonresearch on test-time adversarial examples [1].

Original: Crafting details.

Original: We conduct stability attacks by applying the hypocritical perturbation into the training 722 set.

Original: Unless otherwise speciﬁed, we craft the perturbations by solving the error-minimizing objective (4) with100 steps of PGD, where a step size of 0 .

Original: 8 / 255 is used by following Fowl et al.

Original: Our crafting model is 724 adversarially trained with a crafting budget (cid:15) c = 0 .

Original: 25 (cid:15) a for 10 epochs before generating perturbations.

Original: That is,setting (cid:15) c

Original: = 2 / 255 performs best, as shown in Figure 2(a).

Original: Training details.

Original: We evaluate the effectiveness of the hypocritical perturbation on benchmark datasets 727 including CIFAR-10/100 [33], SVHN [43], and Tiny-ImageNet [34].

Original: Unless otherwise speciﬁed, we use ResNet18 [26] as the default architecture for both the crafting model and the learning model.

Original: For adversarial training, 729 we mainly follow the settings in previous studies [81, 69, 50].

Original: By convention, the defense budget is equal tothe attack budget, i.e., (cid:15) d = 8 / 255 .

Original: The networks are trained for 100 epochs using SGD with momentum 0.9, 731 weight decay 5 × 10 − 4 , and an initial learning rate of 0.1 that is divided by 10 at the 75-th and 90-th epoch.

Original: Early stopping is done with holding out 1000 examples from the training set.

Original: Simple data augmentations such 733 as random crop and horizontal ﬂip are applied.

Original: The inner maximization problem during adversarial training issolved by 10-steps PGD (PGD-10) with step size 2 / 255 .

Original: E Feature-level Analysis on CIFAR-10

Original: In Section 3.1, we theoretically showed that the hypocritical perturbation can cause the poisoned model to rely 737 more on non-robust features, thus the natural accuracy of the adversarially trained model is increased whilethe robust accuracy is decreased.

Original: In this part, we aim to provide empirical evidence on the role of non-robust 739 features in the success of our poisoning method on a benchmark dataset.

Original: In particular, we will demonstrate thatour hypocritical perturbation successfully makes the poisoned model learn more non-robust features.

Original: To show this, by following Section 3.2 of Ilyas et al.

Original: [29], we construct a training set where the only features that are useful for classiﬁcation are the non-robust features (that are extracted from the poisoned model).

Original: The standard accuracy of the classiﬁer trained on the constructed dataset can reﬂect how many non-robust features are learned by the poisoned model (denoted as f ).

Original: To accomplish this, we modify each input-label pair ( x , y ) as follows.

Original: We select a target class t uniformly at random among classes.

Original: Then, we add a small adversarial perturbation to x as follows:

Original: [Equation]

Original: The resulting input-label pairs ( x adv , t ) make up the new training set.

Original: Since the resulting inputs x adv are nearly 742 indistinguishable from the originals x , the label t assigned to the modiﬁed input is simply incorrect to a humanobserver.

Original: Therefore, only the non-robust features in the training set are predictive, while the non-robust features 744 are extracted from the poisoned model.

Original: We compare the model poisoned by our hypocritical perturbation with the baseline model trained on clean data.

Original: These two models correspond to the second row and last row in Table 2, respectively.

Original: Using these two models,we construct two datasets in the above-mentioned manner, respectively.

Original: Then, two new predictors are trained on 748 the two constructed datasets, respectively, and both predictors are evaluated on clean data.

Original: Training parametersfollow exactly those adopted by Ilyas et al.

Original: Our numerical results are summarized in Table 13.

Original: [Table]

Original: As shown in Table 13, the non-robust features learned by the poisoned model are much more predictive than 751 the baseline.

Original: This indicates that the effect of our poisoning method on the non-robust features learned by thepoisoned model is validated empirically.

Original: F Broader Impact

Original: The attack method in this work might be used by an agent in the real world to damage the robust availability 755 of a machine-learning-based system.

Original: We discourage this malicious behavior by presenting the threat model ofstability attacks to the community.

Original: We further propose an adaptive defense to mitigate this issue.

Original: The adaptive 757 defense would help to build a more secure and robust machine learning system in the real world.

Original: At the sametime, the adaptive defense introduces an additional time cost to search for an appropriate defense budget, which 759 might have a negative impact on carbon emission reduction.

Original: Furthermore, society should not be overly optimisticabout AI safety, since the current studies mostly focus on perturbations bounded by simple norms (e.g., (cid:96) ∞ norm 761 in this paper).

Original: There might exist perturbations beyond the (cid:96) p ball in the real world, and we are still far fromcomplete model robustness.

Original: G On the Trade-off between Accuracy and Robustness

Original: An interesting implication of this work is that the hypocritical perturbation exploits the trade-off between 765 standard generalization and adversarial robustness, a phenomenon that has been widely observed in existingworks on adversarial training [67, 81, 14, 40, 59, 77].

Original: Prior work mainly observed that adversarial training improves robust accuracy at the cost of natural accuracy 768 when the training data is clean .

Original: An explanation for the phenomenon is that there are non-robust features in theoriginal dataset, which are predictive yet brittle [67, 29].

Original: Unlike prior work, the trade-off in this work occurs 770 when the training data is hypocritically perturbed .

Original: Speciﬁcally, we make the following observations:

Original: When trained on the hypocritically perturbed data, conventional adversarial training produces modelswith lower robust accuracy but higher natural accuracy (e.g., see Table 2, Table 3, and Table 4).

Original: When trained on the hypocritically perturbed data, adversarial training with adaptive budget canimprove robust accuracy while reducing natural accuracy (e

Original: g., see Table 5).

Original: These two observations align well with our theoretical analyses in Section 3 and Section 4, respectively.

Original: Concretely, our analyses suggest that the hypocritical perturbation works by reinforcing the non-robust featuresin the original data, so that the models adversarially trained on the manipulated data still rely on the non-robust 778 features.

Original: In this way, the natural accuracy of the models increases because the non-robust features are predictive,while the robust accuracy decreases because the non-robust features are brittle.

Original: Furthermore, the effectiveness of 780 the adaptive defense lies in the fact that the reinforced non-robust features can be neutralized by enlarging thedefense budget of adversarial training.

Original: Thus, the adaptive defense improves robustness at the cost of accuracy.

Original: Meanwhile, we note that it would be unsatisfactory that test robustness is improved at the cost of standard 783 generalization.

Original: Several improvements have been proposed to alleviate this issue in the case where the trainingdata is clean, such as RST [49], FAT [82], and SCORE [48].

Original: Incorporating these advances would be helpful in 785 resisting stability attacks, and we leave this as future work.

Original: Finally, we remark that the focus of stability attacks is to degrade test robustness.

Original: For this reason, we do not 787 impose additional restrictions on their impact on natural accuracy.

Original: Having that said, as a method of stabilityattacks, the hypocritical perturbation is observed to improve natural accuracy while reducing robust accuracy.

Original: We note that this makes stability attacks more insidious.

Original: For example, if a poisoned model exhibits highernatural accuracy, practitioners would be more easily enticed to deploy it in a real-world system.

Original: However, as its 791 robust accuracy is actually undesirably low, the system is prone to losing its normal function when encounteringtest-time perturbations.

Original: In short, the negative impacts of stability attacks are serious, even with higher natural 793 accuracy.

Original: Thus, it is imperative to design better defense methods to mitigate the threat of stability attacks.