Original: T OWARDS E VALUATING THE R OBUSTNESS OF N EURAL N ETWORKS L EARNED BY T RANSDUCTION
Revised: T OWARDS E VALUATING THE R OBUSTNESS OF N EURAL N ETWORKS L EARNED BY T RANSDUCTION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: There has been emerging interest in using transductive learning for adversarial robustness (Goldwasser et al., NeurIPS 2020; Wu et al., ICML 2020; Wang et al., ArXiv 2021).
Revised: There has been emerging interest in using transductive learning for adversarial robustness (Goldwasser et al., NeurIPS 2020; Wu et al., ICML 2020; Wang et al., ArXiv 2021).

Original: Compared to traditional defenses, these defense mechanisms “dynamically learn” the model based on test-time input; and theoretically, attacking these defenses reduces to solving a bilevel optimization problem, which poses difﬁculty in crafting adaptive attacks.
Revised: Compared to traditional defenses, these defense mechanisms “dynamically learn” the model based on test-time input; and theoretically, attacking these defenses reduces to solving a bilevel optimization problem, which poses difﬁculty in crafting adaptive attacks.

Original: In this paper, we examine these defense mechanisms from a principled threat analysis perspective.
Revised: In this paper, we examine these defense mechanisms from a principled threat analysis perspective.

Original: We formulate and analyze threat models for transductive-learning based defenses, and point out important subtleties.
Revised: We formulate and analyze threat models for transductive-learning based defenses, and point out important subtleties.

Original: We propose the principle of attacking model space for solving bilevel attack objectives, and present Greedy Model Space Attack (GMSA), an attack framework that can serve as a new baseline for evaluating transductivelearning based defenses.
Revised: We propose the principle of attacking model space for solving bilevel attack objectives, and present Greedy Model Space Attack (GMSA), an attack framework that can serve as a new baseline for evaluating transductivelearning based defenses.

Original: Through systematic evaluation, we show that GMSA, even with weak instantiations, can break previous transductive-learning based defenses, which were resilient to previous attacks, such as AutoAttack (Croce and Hein, ICML 2020).
Revised: Through systematic evaluation, we show that GMSA, even with weak instantiations, can break previous transductive-learning based defenses, which were resilient to previous attacks, such as AutoAttack.

Original: On the positive side, we report a somewhat surprising empirical result of “ transductive adversarial training”:
Revised: On the positive side, we report a somewhat surprising empirical result of “ transductive adversarial training”:

Original: Adversarially retraining the model using fresh randomness at the test time gives a signiﬁcant increase in robustness against attacks we consider.
Revised: Adversarially retraining the model using fresh randomness at the test time gives a signiﬁcant increase in robustness against attacks we consider.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Adversarial robustness of deep learning models has received signiﬁcant attention in recent years (see Kolter & Madry (2018) and references therein).
Revised: Adversarial robustness of deep learning models has received signiﬁcant attention in recent years (see Kolter & Madry (2018) and references therein).

Original: The classic threat model of adversarial robustness considers an inductive setting where a model is learned at the training time and ﬁxed, and then at the test time, an attacker attempts to thwart the ﬁxed model with adversarially perturbed input.
Revised: The classic threat model of adversarial robustness considers an inductive setting where a model is learned at the training time and ﬁxed, and then at the test time, an attacker attempts to thwart the ﬁxed model with adversarially perturbed input.

Original: This gives rise to the adversarial training (Madry et al., 2018; Sinha et al., 2018; Schmidt et al., 2018; Carmon et al., 2019) to enhance adversarial robustness.
Revised: This gives rise to the adversarial training (Madry et al., 2018; Sinha et al., 2018; Schmidt et al., 2018; Carmon et al., 2019) to enhance adversarial robustness.

Original: Going beyond the inductive threat model, there has been emerging interest in using transductive learning (Vapnik, 1998) 1 for adversarial robustness (Goldwasser et al., 2020; Wu et al., 2020b; Wang et al., 2021).
Revised: Going beyond the inductive threat model, there has been emerging interest in using transductive learning (Vapnik, 1998) 1 for adversarial robustness (Goldwasser et al., 2020; Wu et al., 2020b; Wang et al., 2021).

Original: In essence, these defenses attempt to leverage a batch of test-time inputs , which is common for ML pipelines deployed with batch predictions (bat, 2021), to learn an updated model .
Revised: In essence, these defenses attempt to leverage a batch of test-time inputs , which is common for ML pipelines deployed with batch predictions (bat, 2021), to learn an updated model .

Original: The hope is that this “test-time learning” may be useful for adversarial robustness since the defender can adapt the model to the perturbed input from the adversary, which is distinct from the inductive threat model where a model is ﬁxed after training.
Revised: The hope is that this “test-time learning” may be useful for adversarial robustness since the defender can adapt the model to the perturbed input from the adversary, which is distinct from the inductive threat model where a model is ﬁxed after training.

Original: This paper examines these defenses from a principled threat analysis perspective.
Revised: This paper examines these defenses from a principled threat analysis perspective.

Original: We ﬁrst formulate and analyze rigorous threat models.
Revised: We ﬁrst formulate and analyze rigorous threat models.

Original: Our basic 1-round threat model considers a single-round game between the attacker and the defender.
Revised: Our basic 1-round threat model considers a single-round game between the attacker and the defender.

Original: Roughly speaking, the attacker uses an objective max V (cid:48)
Revised: Roughly speaking, the attacker uses an objective max V (cid:48)

Original: ∈ N ( V ) L a (Γ( U (cid:48) ) , V (cid:48) )
Revised: ∈ N ( V ) L a (Γ( U (cid:48) ) , V (cid:48) )

Original: (formula (2)), where V is the given test batch, N ( V ) is a neighborhood around V , L a is a loss function for attack gain, Γ is the transductive-learning based defense, and
Revised: (formula (2)), where V is the given test batch, N ( V ) is a neighborhood around V , L a is a loss function for attack gain, Γ is the transductive-learning based defense, and

Original: U (cid:48)
Revised: U (cid:48)

Original: =
Revised: =

Original: V (cid:48) | X , the projection of V (cid:48) to features, is the adversarially perturbed data for breaking Γ .
Revised: V (cid:48) | X , the projection of V (cid:48) to features, is the adversarially perturbed data for breaking Γ .

Original: This objective is transductive as U (cid:48) , the attacker’s output, appears in both attack ( V (cid:48) in L a ) and defense ( U (cid:48) in Γ ).
Revised: This objective is transductive as U (cid:48) , the attacker’s output, appears in both attack ( V (cid:48) in L a ) and defense ( U (cid:48) in Γ ).

Original: We extend this threat model to multiple rounds, which is necessary when considering
Revised: We extend this threat model to multiple rounds, which is necessary when considering

Original: DENT (Wang et al., 2021) and RMC (Wu et al., 2020b).
Revised: DENT (Wang et al., 2021) and RMC (Wu et al., 2020b).

Original: We point out important subtleties in the modeling that were unclear or overlooked in previous work.
Revised: We point out important subtleties in the modeling that were unclear or overlooked in previous work.

Original: We then study adaptive attacks , that is to leverage the knowledge about Γ to construct attacks.
Revised: We then study adaptive attacks , that is to leverage the knowledge about Γ to construct attacks.

Original: Compared to situations considered in BPDA (Athalye et al., 2018), a transductive learner Γ is even further from being differentiable, and theoretically the attack objective is a bilevel optimization (Colson et al., 2007).
Revised: Compared to situations considered in BPDA (Athalye et al., 2018), a transductive learner Γ is even further from being differentiable, and theoretically the attack objective is a bilevel optimization (Colson et al., 2007).

Original: To address these difﬁculties, our key observation is to consider the transferability of adversarial examples , and consider a robust version of (2): max U (cid:48)
Revised: To address these difﬁculties, our key observation is to consider the transferability of adversarial examples , and consider a robust version of (2): max U (cid:48)

Original: min U ∈N ( U (cid:48) )
Revised: min U ∈N ( U (cid:48) )

Original: L a (Γ( U ) , V (cid:48) )
Revised: L a (Γ( U ) , V (cid:48) )

Original: (formula (6)), where we want to ﬁnd a single attack set U (cid:48) to thwart a family of models, induced by U “around” U (cid:48) .
Revised: (formula (6)), where we want to ﬁnd a single attack set U (cid:48) to thwart a family of models, induced by U “around” U (cid:48) .

Original: This objective relaxes the attacker-defender constraint, and provides more information in dealing with nondifferentiability.
Revised: This objective relaxes the attacker-defender constraint, and provides more information in dealing with nondifferentiability.

Original: To solve the robust objective, we propose Greedy Model Space
Revised: To solve the robust objective, we propose Greedy Model Space

Original: Attack (GMSA), a general attack framework which attempts to solve the robust objective in a greedy manner.
Revised: Attack (GMSA), a general attack framework which attempts to solve the robust objective in a greedy manner.

Original: GMSA can serve as a new baseline for evaluating transductive-learning based defenses.
Revised: GMSA can serve as a new baseline for evaluating transductive-learning based defenses.

Original: We perform a systematic empirical study on various defenses.
Revised: We perform a systematic empirical study on various defenses.

Original: For RMC (Wu et al., 2020b),
Revised: For RMC (Wu et al., 2020b),

Original: DENT (Wang et al., 2021), and URejectron (Goldwasser et al., 2020), we show that even weak instantiations of GMSA can break respective defenses.
Revised: DENT (Wang et al., 2021), and URejectron (Goldwasser et al., 2020), we show that even weak instantiations of GMSA can break respective defenses.

Original: Speciﬁcally, for defenses based on adversarially training, we reduce the robust accuracy to that of adversarial training alone.
Revised: Speciﬁcally, for defenses based on adversarially training, we reduce the robust accuracy to that of adversarial training alone.

Original: We note that, under
Revised: We note that, under

Original: AutoAttack (Croce & Hein, 2020a), the state-of-the-art adaptive attack for the inductive threat model, some of these defenses have claimed to achieve substantial improvements compared to adversarial training alone .
Revised: AutoAttack (Croce & Hein, 2020a), the state-of-the-art adaptive attack for the inductive threat model, some of these defenses have claimed to achieve substantial improvements compared to adversarial training alone .

Original: For example, Wang et al.
Revised: For example, Wang et al.

Original: show that DENT can improve the robustness of the state-of-the-art adversarial training defenses by more than 20% absolutely against AutoAttack on
Revised: show that DENT can improve the robustness of the state-of-the-art adversarial training defenses by more than 20% absolutely against AutoAttack on

Original: CIFAR-10.
Revised: CIFAR-10.

Original: However, under our adaptive attacks, DENT only has minor improvement: less than 3% improvement over adversarial training alone.
Revised: However, under our adaptive attacks, DENT only has minor improvement: less than 3% improvement over adversarial training alone.

Original: Our results thus demonstrates signiﬁcant differences between attacking transductive-learning based defenses and attacking in the inductive setting, and signiﬁcant difﬁculties in the use of transductive learning to improve adversarial robustness.
Revised: Our results thus demonstrates signiﬁcant differences between attacking transductive-learning based defenses and attacking in the inductive setting, and signiﬁcant difﬁculties in the use of transductive learning to improve adversarial robustness.

Original: On the positive side, we report a somewhat surprising empirical result of transductive adversarial training :
Revised: On the positive side, we report a somewhat surprising empirical result of transductive adversarial training :

Original: Adversarially retraining the model using fresh private randomness on a new batch of test-time data gives a signiﬁcant increase in robustness against all of our considered attacks.
Revised: Adversarially retraining the model using fresh private randomness on a new batch of test-time data gives a signiﬁcant increase in robustness against all of our considered attacks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Adversarial robustness in the inductive setting.
Revised: Adversarial robustness in the inductive setting.

Original: Many attacks have been proposed to evaluate the adversarial robustness of the defenses in the inductive setting where the model is ﬁxed during the evaluation phase (Goodfellow et al., 2015; Carlini & Wagner, 2017; Kurakin et al., 2017; MoosaviDezfooli et al., 2016; Croce & Hein, 2020b).
Revised: Many attacks have been proposed to evaluate the adversarial robustness of the defenses in the inductive setting where the model is ﬁxed during the evaluation phase (Goodfellow et al., 2015; Carlini & Wagner, 2017; Kurakin et al., 2017; MoosaviDezfooli et al., 2016; Croce & Hein, 2020b).

Original: Principles for adaptive attacks have been developed in Tramèr et al.
Revised: Principles for adaptive attacks have been developed in Tramèr et al.

Original: and many existing defenses are shown to be broken based on attacks developed from these principles (Athalye et al., 2018).
Revised: and many existing defenses are shown to be broken based on attacks developed from these principles (Athalye et al., 2018).

Original: A fundamental method to obtain adversarial robustness in this setting is adversarial training (Madry et al., 2018; Zhang et al., 2019).
Revised: A fundamental method to obtain adversarial robustness in this setting is adversarial training (Madry et al., 2018; Zhang et al., 2019).

Original: A state-of-the-art attack in the inductive threat model is AutoAttack (Croce & Hein, 2020a).
Revised: A state-of-the-art attack in the inductive threat model is AutoAttack (Croce & Hein, 2020a).

Original: Adversarial robustness via test-time defenses .
Revised: Adversarial robustness via test-time defenses .

Original: There have been various work which attempt to improve adversarial robustness by leveraging test-time data.
Revised: There have been various work which attempt to improve adversarial robustness by leveraging test-time data.

Original: Many of such work attempt to “sanitize” test-time input using a non-differentiable function, and then send it to a pretrained model.
Revised: Many of such work attempt to “sanitize” test-time input using a non-differentiable function, and then send it to a pretrained model.

Original: Most of these proposals were broken by BPDA (Athalye et al., 2018).
Revised: Most of these proposals were broken by BPDA (Athalye et al., 2018).

Original: To this end, we note that a research agenda for “dynamic model defense” has been proposed in Goodfellow (2019).
Revised: To this end, we note that a research agenda for “dynamic model defense” has been proposed in Goodfellow (2019).

Original: Adversarial robustness using transductive learning.
Revised: Adversarial robustness using transductive learning.

Original: There has been emerging interesting in using transductive learning to improve adversarial robustness.
Revised: There has been emerging interesting in using transductive learning to improve adversarial robustness.

Original: In view of “dynamic defenses”, these proposals attempt to apply transductive learning to the test data and update the model, and then use the updated model to predict on the test data.
Revised: In view of “dynamic defenses”, these proposals attempt to apply transductive learning to the test data and update the model, and then use the updated model to predict on the test data.

Original: In this work we consider three such work (Wu et al., 2020b; Wang et al., 2021; Goldwasser et al., 2020).
Revised: In this work we consider three such work (Wu et al., 2020b; Wang et al., 2021; Goldwasser et al., 2020).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Let F be a model, and for a data point ( xxx, y ) ∈
Revised: Let F be a model, and for a data point ( xxx, y ) ∈

Original: X × Y , a loss function (cid:96) ( F ; xxx, y ) gives the loss of F on the point.
Revised: X × Y , a loss function (cid:96) ( F ; xxx, y ) gives the loss of F on the point.

Original: Let V be a set of labeled data points, and let L ( F, V )
Revised: Let V be a set of labeled data points, and let L ( F, V )

Original: = 1 | V | (cid:80) ( xxx,y ) ∈ V (cid:96) ( F ; xxx, y ) denote the empirical loss of F on V .
Revised: = 1 | V | (cid:80) ( x x x,y ) ∈ V (cid:96) ( F ; xxx, y ) denote the empirical loss of F on V .

Original: For example, if we use binary loss (cid:96) 0 , 1 ( F ; xxx, y ) = 1 [ F ( xxx ) (cid:54) =
Revised: For example, if we use binary loss (cid:96) 0 , 1 ( F ; xxx, y ) = 1 [ F ( xxx ) (cid:54) =

Original: y ] , this gives the test error of F on V .
Revised: y ] , this gives the test error of F on V .

Original: We use the notation V | X to denote the projection of V to its features, that is { ( xxx i , y i ) } mi =1 | X (cid:55)→ { xxx i } mi =1 .
Revised: We use the notation V | X to denote the projection of V to its features, that is { ( xxx i , y i ) } mi =1 | X (cid:55)→ { xxx i } mi =1 .

Original: Throughout the paper, we use N ( · ) to denote a neighborhood function for perturbing features: That is, N ( xxx )
Revised: Throughout the paper, we use N ( · ) to denote a neighborhood function for perturbing features: That is, N ( xxx )

Original: =
Revised: =

Original: { xxx (cid:48) | d ( xxx (cid:48) ,xxx ) <
Revised: { xxx (cid:48) | d ( xxx (cid:48) ,xxx ) <

Original: (cid:15) } is a set of examples that are close to xxx in terms of a distance metric d (e.g., d ( xxx (cid:48) ,xxx )
Revised: (cid:15) } is a set of examples that are close to xxx in terms of a distance metric d (e.g., d ( xxx (cid:48) ,xxx )

Original: = (cid:107) xxx (cid:48) − xxx (cid:107) p ).
Revised: = (cid:107) xxx (cid:48) − xxx (cid:107) p ).

Original: Given U
Revised: Given U

Original: = { xxx i } mi =1 , let
Revised: = { xxx i } mi =1 , let

Original: N ( U ) = {{ xxx (cid:48) i } mi =1 | d ( xxx (cid:48)
Revised: N ( U ) = {{ xxx (cid:48) i } mi =1 | d ( xxx (cid:48)

Original: i ,xxx i ) < (cid:15), i = 0 , . . .
Revised: i ,xxx i ) < (cid:15), i = 0 , . . .

Original: , m } .
Revised: , m } .

Original: Since labels are not changed for adversarial examples, we also use the notation N ( V ) to denote perturbations of features, with labels ﬁxed.
Revised: Since labels are not changed for adversarial examples, we also use the notation N ( V ) to denote perturbations of features, with labels ﬁxed.

Original:  M ODELING T RANSDUCTIVE R OBUSTNESS
Revised:  M ODELING T RANSDUCTIVE R OBUSTNESS

Original: In this section we formulate and analyze threat models for transductive defenses.
Revised: In this section we formulate and analyze threat models for transductive defenses.

Original: We ﬁrst formulate a threat model for a single-round game between the attacker and the defender.
Revised: We ﬁrst formulate a threat model for a single-round game between the attacker and the defender.

Original: We then consider extensions of this threat model to multiple rounds, which are necessary when considering DENT (Wang et al., 2021) and RMC (Wu et al., 2020b), and point out important subtleties in modeling that were not articulated in previous work.
Revised: We then consider extensions of this threat model to multiple rounds, which are necessary when considering DENT (Wang et al., 2021) and RMC (Wu et al., 2020b), and point out important subtleties in modeling that were not articulated in previous work.

Original: We characterize previous test-time defenses using our threat models.
Revised: We characterize previous test-time defenses using our threat models.

Original: 1-round game .
Revised: 1-round game .

Original: In this case, the adversary “intercepts” a clean test data V (with clean features
Revised: In this case, the adversary “intercepts” a clean test data V (with clean features

Original: U =
Revised: U =

Original: V | X , and labels V | Y ), adversarially perturbs it, and sends a perturbed features U (cid:48) to the defender.
Revised: V | X , and labels V | Y ), adversarially perturbs it, and sends a perturbed features U (cid:48) to the defender.

Original: The defender learns a new model based on U (cid:48) .
Revised: The defender learns a new model based on U (cid:48) .

Original: A referee then evaluates the accuracy of the adapted model on U (cid:48) .
Revised: A referee then evaluates the accuracy of the adapted model on U (cid:48) .

Original: Formally:
Revised: Formally:

Original: Deﬁnition 1 ( 1-round threat model for transductive adversarial robustness) .
Revised: Deﬁnition 1 ( 1-round threat model for transductive adversarial robustness) .

Original: Fix an adversarial perturbation type (e.g., (cid:96) ∞ perturbations with perturbation budget ε ).
Revised: Fix an adversarial perturbation type (e.g., (cid:96) ∞ perturbations with perturbation budget ε ).

Original: Let P X,Y be a data generation distribution.
Revised: Let P X,Y be a data generation distribution.

Original: The attacker is an algorithm A , and the defender is a pair of algorithms ( T , Γ) , where
Revised: The attacker is an algorithm A , and the defender is a pair of algorithms ( T , Γ) , where

Original: T is a supervised learning algorithm, and Γ is a transductive learning algorithm.
Revised: T is a supervised learning algorithm, and Γ is a transductive learning algorithm.

Original: A (clean) training set D is sampled i.i.d.
Revised: A (clean) training set D is sampled i.i.d.

Original: from P X,Y .
Revised: from P X,Y .

Original: A (clean) test set V is sampled i.i.d.
Revised: A (clean) test set V is sampled i.i.d.

Original: from P X,Y .
Revised: from P X,Y .

Original: • [Training time, defender]
Revised: • [Training time, defender]

Original: The defender trains an optional base model F = T ( D ) , using the labeled source data D .
Revised: The defender trains an optional base model F = T ( D ) , using the labeled source data D .

Original: • [Test time, attacker]
Revised: • [Test time, attacker]

Original: The attacker receives V , and produces an (adversarial) unlabeled dataset U (cid:48) : 1.
Revised: The attacker receives V , and produces an (adversarial) unlabeled dataset U (cid:48) : 1.

Original: On input Γ , F , D , and V , A perturbs each point ( xxx, y
Revised: On input Γ , F , D , and V , A perturbs each point ( xxx, y

Original: ) ∈ V to ( xxx (cid:48) , y ) (subject to the agreed attack type), giving V (cid:48) =
Revised: ) ∈ V to ( xxx (cid:48) , y ) (subject to the agreed attack type), giving V (cid:48) =

Original: A (Γ , F, D, V ) (that is, V (cid:48)
Revised: A (Γ , F, D, V ) (that is, V (cid:48)

Original: ∈ N ( V ) ).
Revised: ∈ N ( V ) ).

Original: Send U (cid:48)
Revised: Send U (cid:48)

Original: =
Revised: =

Original: V (cid:48) | X (the feature vectors of V (cid:48) ) to the defender.
Revised: V (cid:48) | X (the feature vectors of V (cid:48) ) to the defender.

Original: • [Test time, defender]
Revised: • [Test time, defender]

Original: The defender produces a model as F ∗ =
Revised: The defender produces a model as F ∗ =

Original: Γ( F, D, U (cid:48) ) .
Revised: Γ( F, D, U (cid:48) ) .

Original: Multi-round games .
Revised: Multi-round games .

Original: The extension of 1-round games to multi-round contains several important considerations that were implicit or unclear in previous work, and is closely related to what it means by adaptive attacks .
Revised: The extension of 1-round games to multi-round contains several important considerations that were implicit or unclear in previous work, and is closely related to what it means by adaptive attacks .

Original: Speciﬁcally:
Revised: Speciﬁcally:

Original: Private randomness .
Revised: Private randomness .

Original: Note that Γ uses randomness, such as random initialization and random restartsin adversarial training.
Revised: Note that Γ uses randomness, such as random initialization and random restartsin adversarial training.

Original: Since these randomness are generated after the attacker’s move, they are treated as private randomness , and not known to the adversary.
Revised: Since these randomness are generated after the attacker’s move, they are treated as private randomness , and not known to the adversary.

Original: Intermediate defender states leaking vs. Non-leaking .
Revised: Intermediate defender states leaking vs. Non-leaking .

Original: In a multi-round game, the defender may maintain states across rounds.
Revised: In a multi-round game, the defender may maintain states across rounds.

Original: For example, the defender may store test data and updated models from previous rounds, and use them in a new round.
Revised: For example, the defender may store test data and updated models from previous rounds, and use them in a new round.

Original: If these intermediate defender states are “leaked” to the attacker, we call it intermediate defender states leaking , or simply states leaking , otherwise we call it non states-leaking , or simply non-leaking.
Revised: If these intermediate defender states are “leaked” to the attacker, we call it intermediate defender states leaking , or simply states leaking , otherwise we call it non states-leaking , or simply non-leaking.

Original: Note that the attacker cannot simply compute these information by simulating on the training and testing data, due to the use of private randomness .
Revised: Note that the attacker cannot simply compute these information by simulating on the training and testing data, due to the use of private randomness .

Original: We note that, however, the initial pretrained model is assumed to be known by the attacker.
Revised: We note that, however, the initial pretrained model is assumed to be known by the attacker.

Original: The attacker can also of course maintain arbitrary states, and are assumed not known to the defender.
Revised: The attacker can also of course maintain arbitrary states, and are assumed not known to the defender.

Original: Adaptive vs. Non-adaptive .
Revised: Adaptive vs. Non-adaptive .

Original: Because transductive learning happens after the attacker produces U (cid:48) , the attacker may not be able to directly attack the model Γ produced.
Revised: Because transductive learning happens after the attacker produces U (cid:48) , the attacker may not be able to directly attack the model Γ produced.

Original: Nevertheless, the attacker is assumed to have full knowledge of the transductive mechanism Γ , except the private randomness.
Revised: Nevertheless, the attacker is assumed to have full knowledge of the transductive mechanism Γ , except the private randomness.

Original: In this paper we call an attack adaptive if it makes explicit use of the knowledge of Γ .
Revised: In this paper we call an attack adaptive if it makes explicit use of the knowledge of Γ .

Original: Naturally ordered vs. Adversarially ordered .
Revised: Naturally ordered vs. Adversarially ordered .

Original: Both RMC and DENT handle batches of ﬁxed sizes.
Revised: Both RMC and DENT handle batches of ﬁxed sizes.

Original: An intuitive setup for multi-round game is that the batches come in sequentially, and the attacker must forward perturbed versions of these batches in the same order to the defender, which we call the “naturally ordered” game.
Revised: An intuitive setup for multi-round game is that the batches come in sequentially, and the attacker must forward perturbed versions of these batches in the same order to the defender, which we call the “naturally ordered” game.

Original: However, this formulation does not capture an important scenario:
Revised: However, this formulation does not capture an important scenario:

Original: An adversary can wait and pool a large amount of test data, then chooses a “worst-case” order of perturbed data points, and then sends them in batches one at a time for adaptation in order to maximize the breach.
Revised: An adversary can wait and pool a large amount of test data, then chooses a “worst-case” order of perturbed data points, and then sends them in batches one at a time for adaptation in order to maximize the breach.

Original: We call the latter “adversarially ordered” game.
Revised: We call the latter “adversarially ordered” game.

Original: We note that all previous work only considered naturally-ordered game, which gives the defender more advantages, and is thus our focus in the rest of the paper.
Revised: We note that all previous work only considered naturally-ordered game, which gives the defender more advantages, and is thus our focus in the rest of the paper.

Original: Adversarially-ordered game is evaluated for DENT in Appendix A.7.
Revised: Adversarially-ordered game is evaluated for DENT in Appendix A.7.

Original: Modeling capacity of our threat models .
Revised: Modeling capacity of our threat models .

Original: Our threat models encompass a large family of defenses.
Revised: Our threat models encompass a large family of defenses.

Original: For example, without using Γ , the threat model degenerates to the classic inductive threat model.
Revised: For example, without using Γ , the threat model degenerates to the classic inductive threat model.

Original: Our threat models also capture various “test-time defenses” proposals (e.g., those broken by the
Revised: Our threat models also capture various “test-time defenses” proposals (e.g., those broken by the

Original: BPDA (Athalye et al., 2018)), where Γ is a “non-differentiable” function which “sanitizes” the test data , instead of updating the model, before sending them to a ﬁxed pretrained model.
Revised: BPDA (Athalye et al., 2018)), where Γ is a “non-differentiable” function which “sanitizes” the test data , instead of updating the model, before sending them to a ﬁxed pretrained model.

Original: Therefore, in particular, these proposals are not transductive-learning based.
Revised: Therefore, in particular, these proposals are not transductive-learning based.

Original: Below we describe previous defenses which we study in the rest of this paper, where Γ is indeed transductive learning.
Revised: Below we describe previous defenses which we study in the rest of this paper, where Γ is indeed transductive learning.

Original: Example 1 ( Runtime masking and cleansing ) .
Revised: Example 1 ( Runtime masking and cleansing ) .

Original: Runtime masking and cleansing (RMC) (Wu et al., 2020b) is a recent transductive-learning defense.
Revised: Runtime masking and cleansing (RMC) (Wu et al., 2020b) is a recent transductive-learning defense.

Original: For RMC, the defender is stateful and adapted from the model learned in the last round , on a single test point ( | U | = 1 ): The adaptation objective is F ∗ = arg min F (cid:80) ( xxx,y
Revised: For RMC, the defender is stateful and adapted from the model learned in the last round , on a single test point ( | U | = 1 ): The adaptation objective is F ∗ = arg min F (cid:80) ( xxx,y

Original: ) ∈ N (cid:48)
Revised: ) ∈ N (cid:48)

Original: ( (cid:98) xxx ) L ( F,xxx, y ) , where (cid:98) xxx is the test time feature point, and N (cid:48)
Revised: ( (cid:98) xxx ) L ( F,xxx, y ) , where (cid:98) xxx is the test time feature point, and N (cid:48)

Original: ( (cid:98) xxx ) is the set of examples in the adversarial training dataset D (cid:48) that are top-K nearest to (cid:98) xxx in a distance measure.
Revised: ( (cid:98) xxx ) is the set of examples in the adversarial training dataset D (cid:48) that are top-K nearest to (cid:98) xxx in a distance measure.

Original: RMC paper considered two attacks: (1) Transfer attack , which generates perturbed data by attacking the initial base model, and (2) PGD-skip attack , which at round p +1 , runs PGD attack on the model learned at round p .
Revised: RMC paper considered two attacks: (1) Transfer attack , which generates perturbed data by attacking the initial base model, and (2) PGD-skip attack , which at round p +1 , runs PGD attack on the model learned at round p .

Original: In our language, transfer attack is stateless (i.e. the adversary maintains no state) and non-adaptive , PGD-skip attack is state-leaking , but still non-adaptive .
Revised: In our language, transfer attack is stateless (i.e. the adversary maintains no state) and non-adaptive , PGD-skip attack is state-leaking , but still non-adaptive .

Original: Example 2 ( Defensive entropy minimization (DENT (Wang et al., 2021)) ) .
Revised: Example 2 ( Defensive entropy minimization (DENT (Wang et al., 2021)) ) .

Original: DENT adapts the model using test input, and can work with any training-time learning procedure.
Revised: DENT adapts the model using test input, and can work with any training-time learning procedure.

Original: The DENT defender is stateless : It always starts the adaptation from the original pretrained model, ﬁxed at the training time.
Revised: The DENT defender is stateless : It always starts the adaptation from the original pretrained model, ﬁxed at the training time.

Original: During the test-time adaptation, only the afﬁne parameters in batch normalization layers of the base model are updated, using entropy minimization with the information maximization regularization.
Revised: During the test-time adaptation, only the afﬁne parameters in batch normalization layers of the base model are updated, using entropy minimization with the information maximization regularization.

Original: In this paper, we show that with strong adaptive attacks under the naturally ordered setting, we are able to reduce the robustness to be almost the same as that of static models (see Section 6).
Revised: In this paper, we show that with strong adaptive attacks under the naturally ordered setting, we are able to reduce the robustness to be almost the same as that of static models (see Section 6).

Original: Further, under the adversarially ordered setting, we can completely break DENT.
Revised: Further, under the adversarially ordered setting, we can completely break DENT.

Original: Example 3 ( Goldwasser et al.
Revised: Example 3 ( Goldwasser et al.

Original: ’s transductive threat model ) .
Revised: ’s transductive threat model ) .

Original: While seemingly our threat model is quite different from the one described in Goldwasser et al.
Revised: While seemingly our threat model is quite different from the one described in Goldwasser et al.

Original: (2020), one can indeed recover their threat model naturally as a 1-round game : First, for the perturbation type, we simply allow arbitrary perturbations in the threat model setup.
Revised: (2020), one can indeed recover their threat model naturally as a 1-round game : First, for the perturbation type, we simply allow arbitrary perturbations in the threat model setup.

Original: Second, we have a ﬁxed pretrained model F , and the adaptation algorithm Γ learns a set S which represents the set of “allowable” points (so F | S yields a predictor with redaction, namely it outputs ⊥ for points outside of S ).
Revised: Second, we have a ﬁxed pretrained model F , and the adaptation algorithm Γ learns a set S which represents the set of “allowable” points (so F | S yields a predictor with redaction, namely it outputs ⊥ for points outside of S ).

Original: Third, we deﬁne two error functions as (5) and (6) in Goldwasser et al.
Revised: Third, we deﬁne two error functions as (5) and (6) in Goldwasser et al.

Original: [Equation]
Revised: [Equation]

Original: where f is the ground truth hypothesis.
Revised: where f is the ground truth hypothesis.

Original: The ﬁrst equation measures prediction errors in U (cid:48) that passed through S , and the second equation measures the rejection rate of the clean input.
Revised: The ﬁrst equation measures prediction errors in U (cid:48) that passed through S , and the second equation measures the rejection rate of the clean input.

Original: The referee evaluates by measuring two errors: L ( F | S , V (cid:48) )
Revised: The referee evaluates by measuring two errors: L ( F | S , V (cid:48) )

Original: = (err U (cid:48)
Revised: = (err U (cid:48)

Original: ( F | S ) , rej U ( S )) .
Revised: ( F | S ) , rej U ( S )) .

Original:  A DAPTIVE A TTACKS IN O NE R OUND
Revised:  A DAPTIVE A TTACKS IN O NE R OUND

Original: In this section we study a basic question: How to perform adaptive attacks against a transductivelearning based defense in one round ?
Revised: In this section we study a basic question: How to perform adaptive attacks against a transductivelearning based defense in one round ?

Original: Note that, in each round of a multi-round game, an independent batch of test input U is sampled, and the defender can use transductive learning to produce a model speciﬁcally adapted to the adversarial input U (cid:48) , after the defender receives it.
Revised: Note that, in each round of a multi-round game, an independent batch of test input U is sampled, and the defender can use transductive learning to produce a model speciﬁcally adapted to the adversarial input U (cid:48) , after the defender receives it.

Original: Therefore, it is of fundamental interest to attack this ad-hoc adaptation.
Revised: Therefore, it is of fundamental interest to attack this ad-hoc adaptation.

Original: We consider white-box attacks: The attacker knows all the details of Γ , except private randomness, which is sampled after the attacker’s move.
Revised: We consider white-box attacks: The attacker knows all the details of Γ , except private randomness, which is sampled after the attacker’s move.

Original: We deduce a principle for adaptive attacks in one round, which we call the principle of attacking model space : Effective attacks against a transductive defense may need to consider attacking a set of representative models induced in the neighborhood of U .
Revised: We deduce a principle for adaptive attacks in one round, which we call the principle of attacking model space : Effective attacks against a transductive defense may need to consider attacking a set of representative models induced in the neighborhood of U .

Original: We give concrete instantiations of this principle, and show in experiments that they break previous transductive-learning based defenses.
Revised: We give concrete instantiations of this principle, and show in experiments that they break previous transductive-learning based defenses.

Original: Attacks in multi-round .
Revised: Attacks in multi-round .

Original: If the transductive-learning based defense is stateless, then we simply repeat one-round attack multiple times.
Revised: If the transductive-learning based defense is stateless, then we simply repeat one-round attack multiple times.

Original: If it is stateful, then we need to consider state-leaking setting or non-leaking setting.
Revised: If it is stateful, then we need to consider state-leaking setting or non-leaking setting.

Original: For all experiments in Section 6, we only evaluate non-leaking setting, which is more challenging for the adversary.
Revised: For all experiments in Section 6, we only evaluate non-leaking setting, which is more challenging for the adversary.

Original: 5. G OAL OF THE ATTACKER AND CHALLENGES
Revised: 5. G OAL OF THE ATTACKER AND CHALLENGES

Original: To start with, given a defense mechanism Γ , the objective of the attacker can be formulated as:
Revised: To start with, given a defense mechanism Γ , the objective of the attacker can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where L a is the loss function of the attacker.
Revised: where L a is the loss function of the attacker.

Original: We make some notational simpliﬁcations: Since D is a constant, in the following we drop it and write Γ( U (cid:48) ) .
Revised: We make some notational simpliﬁcations: Since D is a constant, in the following we drop it and write Γ( U (cid:48) ) .

Original: Also, since the attacker does not modify the labels in the threat model, we abuse the notation and write the objective as
Revised: Also, since the attacker does not modify the labels in the threat model, we abuse the notation and write the objective as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: A generic attacker would proceed iteratively as follows:
Revised: A generic attacker would proceed iteratively as follows:

Original: It starts with the clean test set V , and generates a sequence of (hopefully) increasingly stronger attack sets U (0) =
Revised: It starts with the clean test set V , and generates a sequence of (hopefully) increasingly stronger attack sets U (0) =

Original: V | X , U (1) , . . .
Revised: V | X , U (1) , . . .

Original: , U ( i ) ( U ( i ) must satisfy the attack constraints at U , such as (cid:96) ∞ bound).
Revised: , U ( i ) ( U ( i ) must satisfy the attack constraints at U , such as (cid:96) ∞ bound).

Original: We note several basic but important differences between transductive attacks and inductive attacks in the classic minimax threat model: (D1)
Revised: We note several basic but important differences between transductive attacks and inductive attacks in the classic minimax threat model: (D1)

Original: Γ( U (cid:48) ) is not differentiable .
Revised: Γ( U (cid:48) ) is not differentiable .

Original: For the scenarios we are interested in, Γ is an optimization algorithm to solve an objective F ∗ ∈ arg min F L d ( F, D, U (cid:48) ) .
Revised: For the scenarios we are interested in, Γ is an optimization algorithm to solve an objective F ∗ ∈ arg min F L d ( F, D, U (cid:48) ) .

Original: This renders (3) into a bilevel optimization problem (Colson et al., 2007):
Revised: This renders (3) into a bilevel optimization problem (Colson et al., 2007):

Original: [Equation]
Revised: [Equation]

Original: In these cases, Γ is in general not (in fact far from) differentiable.
Revised: In these cases, Γ is in general not (in fact far from) differentiable.

Original: A natural attempt is to approximate Γ with a differentiable function, using theories such as Neural Tangent Kernels (Jacot et al., 2018).
Revised: A natural attempt is to approximate Γ with a differentiable function, using theories such as Neural Tangent Kernels (Jacot et al., 2018).

Original: Unfortunately no existing theory applies to the transductive learning, which deals with unlabeled data
Revised: Unfortunately no existing theory applies to the transductive learning, which deals with unlabeled data

Original: U (cid:48)
Revised: U (cid:48)

Original: (also, as we have remarked previously, tricks such as BPDA (Athalye et al., 2018) also does not apply because transductive learning is much more complex than test-time defenses considered there).
Revised: (also, as we have remarked previously, tricks such as BPDA (Athalye et al., 2018) also does not apply because transductive learning is much more complex than test-time defenses considered there).

Original: (D2) U (cid:48) appears in both attack and defense .
Revised: (D2) U (cid:48) appears in both attack and defense .

Original: Another signiﬁcant difference is that the attack set U (cid:48) also appears as the input for the defense (i.e. Γ( U (cid:48) ) ).
Revised: Another signiﬁcant difference is that the attack set U (cid:48) also appears as the input for the defense (i.e. Γ( U (cid:48) ) ).

Original: Therefore, while it is easy to ﬁnd U (cid:48) to fail Γ( U ) for any ﬁxed U , it is much harder to ﬁnd a good direction to update the attack and converge to an attack set U ∗ that fails an entire model space induced by itself: Γ( U ∗ ) .
Revised: Therefore, while it is easy to ﬁnd U (cid:48) to fail Γ( U ) for any ﬁxed U , it is much harder to ﬁnd a good direction to update the attack and converge to an attack set U ∗ that fails an entire model space induced by itself: Γ( U ∗ ) .

Original: (D3) Γ( U (cid:48) ) can be a random variable .
Revised: (D3) Γ( U (cid:48) ) can be a random variable .

Original: In the classic minimax threat model, the attacker faces a ﬁxed model.
Revised: In the classic minimax threat model, the attacker faces a ﬁxed model.

Original: However, the output of Γ can be a random variable of models due to its private randomness, such as the case of Randomized Smoothing (Cohen et al., 2019).
Revised: However, the output of Γ can be a random variable of models due to its private randomness, such as the case of Randomized Smoothing (Cohen et al., 2019).

Original: In these cases, successfully attacking a single sample of this random variable does not sufﬁce.
Revised: In these cases, successfully attacking a single sample of this random variable does not sufﬁce.

Original: Require:
Revised: Require:

Original: A transductive learningalgorithm Γ , an optional training dataset D , a clean test set V , an initial model F (0) , and an integer parameter T ≥ 0 (the number of iterations).
Revised: [Table] learning

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L a is a loss function.
Revised: where L a is a loss function.

Original: Set U ( i ) =
Revised: Set U ( i ) =

Original: V ( i ) | X .
Revised: V ( i ) | X .

Revised: [Equation]

Original: Fixed Point Attack: A ﬁrst attempt .
Revised: Fixed Point Attack: A ﬁrst attempt .

Original: We adapt previous literature for solving bilevel optimization in deep learning setting (Lorraine & Duvenaud, 2018) (designed for supervised learning).
Revised: We adapt previous literature for solving bilevel optimization in deep learning setting (Lorraine & Duvenaud, 2018) (designed for supervised learning).

Original: The idea is simple:
Revised: The idea is simple:

Original: At iteration i + 1 , we ﬁx U ( i ) and
Revised: At iteration i + 1 , we ﬁx U ( i ) and

Original: model space F ( i ) =
Revised: model space F ( i ) =

Original: Γ( U ( i ) ) , and construct U ( i +1) to fail it.
Revised: Γ( U ( i ) ) , and construct U ( i +1) to fail it.

Original: We call this the Fixed Point Attack (F PA ) (Algorithm 1), as one hopes that this process converges to a good ﬁxed point U ∗ .
Revised: We call this the Fixed Point Attack (F PA ) (Algorithm 1), as one hopes that this process converges to a good ﬁxed point U ∗ .

Original: Unfortunately, we found F PA to be weak in experiments.
Revised: Unfortunately, we found F PA to be weak in experiments.

Original: The reason is exactly (D2) : U ( i +1)
Revised: The reason is exactly (D2) : U ( i +1)

Original: failing F ( i ) may not give any indication that it can also fail F ( i +1) induced by itself.
Revised: failing F ( i ) may not give any indication that it can also fail F ( i +1) induced by itself.

Original: Note that transfer attack is a special case of FPA by setting T = 0 .
Revised: Note that transfer attack is a special case of FPA by setting T = 0 .

Original: 5. S TRONG ADAPTIVE ATTACKS FROM ATTACKING MODEL SPACES
Revised: 5. S TRONG ADAPTIVE ATTACKS FROM ATTACKING MODEL SPACES

Original: To develop stronger adaptive attacks, we consider a key property of the adversarial attacks: The transferability of adversarial examples .
Revised: To develop stronger adaptive attacks, we consider a key property of the adversarial attacks: The transferability of adversarial examples .

Original: Various previous work have identiﬁed that adversarial examples transfer (Tramèr et al., 2017; Liu et al., 2016), even across vastly different architectures and models.
Revised: Various previous work have identiﬁed that adversarial examples transfer (Tramèr et al., 2017; Liu et al., 2016), even across vastly different architectures and models.

Original: Therefore, if U (cid:48) is a good attack set, we would expect that U (cid:48) also fails Γ( U ) for U close to
Revised: Therefore, if U (cid:48) is a good attack set, we would expect that U (cid:48) also fails Γ( U ) for U close to

Original: U (cid:48) .
Revised: U (cid:48) .

Original: This leads to the consideration of the following objective:
Revised: This leads to the consideration of the following objective:

Original: [Equation]
Revised: [Equation]

Original: (6) where N ( · ) is a neighborhood function (possibly different than N ).
Revised: (6) where N ( · ) is a neighborhood function (possibly different than N ).

Original: It induces a family of models { Γ( U (cid:48) )
Revised: It induces a family of models { Γ( U (cid:48) )

Original: | U (cid:48) ∈
Revised: | U (cid:48) ∈

Original: N ( U ∗ ) } , which we call a model space .
Revised: N ( U ∗ ) } , which we call a model space .

Original: (in fact, this can be a family of random variables of models)
Revised: (in fact, this can be a family of random variables of models)

Original: This can be viewed as a natural robust version of (3) by considering the transferability of U (cid:48) .
Revised: This can be viewed as a natural robust version of (3) by considering the transferability of U (cid:48) .

Original: While this is seemingly even harder to solve, it has several beneﬁts: (1)
Revised: While this is seemingly even harder to solve, it has several beneﬁts: (1)

Original: Considering a model space naturally strengthens F PA .
Revised: Considering a model space naturally strengthens F PA .

Original: F PA naturally falls into this formulation as a weak instantiation where we consider a single U
Revised: F PA naturally falls into this formulation as a weak instantiation where we consider a single U

Original: =
Revised: =

Original: U ( i ) .
Revised: U ( i ) .

Original: Also, considering a model space gives the attacker more information in dealing with the nondifferentiability of Γ (D1) .
Revised: Also, considering a model space gives the attacker more information in dealing with the nondifferentiability of Γ (D1) .

Original: (2) It relaxes the attacker-defender constraint (D2).
Revised: (2) It relaxes the attacker-defender constraint (D2).

Original: Perhaps more importantly, for the robust objective, we no longer need the same U (cid:48) to appear in both defender and attacker.
Revised: Perhaps more importantly, for the robust objective, we no longer need the same U (cid:48) to appear in both defender and attacker.

Original: Therefore it gives a natural relaxation which makes attack algorithm design easier.
Revised: Therefore it gives a natural relaxation which makes attack algorithm design easier.

Original: In summary, while “brittle” U (cid:48) that does not transfer may indeed exist theoretically, their identiﬁcation can be challenging algorithmically, and its robust variant provides a natural relaxation considering both algorithmic feasibility and attack strength.
Revised: In summary, while “brittle” U (cid:48) that does not transfer may indeed exist theoretically, their identiﬁcation can be challenging algorithmically, and its robust variant provides a natural relaxation considering both algorithmic feasibility and attack strength.

Original: This thus leads us to the following principle:
Revised: This thus leads us to the following principle:

Original: The Principle of Attacking Model Spaces .
Revised: The Principle of Attacking Model Spaces .

Original: An effective adaptive attack against a transductive-learning based defense may need to consider a model space induced by a proper neighborhood of U .
Revised: An effective adaptive attack against a transductive-learning based defense may need to consider a model space induced by a proper neighborhood of U .

Original: Algorithm 2 G 
Revised: Algorithm 2 G REEDY M ODEL S PACE A TTACK (GMSA)

Original: M ODEL S PACE A TTACK (GMSA)

Original: Require:
Revised: Require:

Original: A transductive learning algorithm Γ , an optional training dataset D , a clean test set V , an initial model F (0) , and an integer parameter T ≥ 0 (the number of iterations). 
Revised: A transductive learning algorithm Γ , an optional training dataset D , a clean test set V , an initial model F (0) , and an integer parameter T ≥ 0 (the number of iterations). 1: for i = 0 , 1 , . . . , do 2: Attack

Original: previous 
Revised: previous models

Original:  perturbed 
Revised: get perturbed set:

Original: [Equation]
Revised: [Equation]

Original: where L GMSA is a loss function.
Revised: where L GMSA is a loss function.

Original: Set U ( i ) =
Revised: Set U ( i ) =

Original: V ( i ) |
Revised: V ( i ) |

Original: X . 3: Run the transductive learning algorithm Γ to get the next model: F ( i +1) =
Revised: X . 3: Run the transductive learning algorithm Γ to get the next model: F ( i +1) =

Original: Γ( D, U ( i ) ) .
Revised: Γ( D, U ( i ) ) .

Original: 4: end for 5: Select the best attack 
Revised: 4: end for 5: Select the best attack U ( k )

Original:  ≤ T 
Revised: k = arg max 0 ≤ i ≤ T L ( F ( i +1) , V

Original: i  , 6: return 
Revised: i ) ) , 6: return k

Original: An instantiation: Greedy Model Space Attack (GMSA) .
Revised: An instantiation: Greedy Model Space Attack (GMSA) .

Original: We give a simplest possible instantiation of the principle, which we call the Greedy Model Space Attack (Algorithm 2).
Revised: We give a simplest possible instantiation of the principle, which we call the Greedy Model Space Attack (Algorithm 2).

Original: In experiments we use this instantiation to break previous defenses.
Revised: In experiments we use this instantiation to break previous defenses.

Original: In this instantiation, the family of model spaces to consider is just all the model spaces constructed in previous iterations.
Revised: In this instantiation, the family of model spaces to consider is just all the model spaces constructed in previous iterations.

Original: L GMSA ( { F ( j ) } ij =0 , V (cid:48) ) is a loss function that the attacker uses to attack the history model spaces.
Revised: L GMSA ( { F ( j ) } ij =0 , V (cid:48) ) is a loss function that the attacker uses to attack the history model spaces.

Original: We consider two instantiations: (1) L AVGGMSA ( { F ( j ) } ij =0 , V (cid:48) )
Revised: We consider two instantiations: (1) L AVGGMSA ( { F ( j ) } ij =0 , V (cid:48) )

Original: =
Revised: =

Original: 1 i +1 (cid:80) ij =

Revised: 1 i +1 (cid:80) ij =

Original: L a ( F ( j ) , V (cid:48) ) , (2) L MINGMSA ( { F ( j ) } ij =0 , V (cid:48) )
Revised: L a ( F ( j ) , V (cid:48) ) , (2) L MINGMSA ( { F ( j ) } ij =0 , V (cid:48) )

Original: =
Revised: =

Original: min 0 ≤ j ≤ i L a ( F ( j ) , V (cid:48) ) , where L AVGGMSA gives attack algorithm GMSA-AVG, and L MINGMSA gives attack algorithm GMSA-MIN.
Revised: min 0 ≤ j ≤ i L a ( F ( j ) , V (cid:48) ) , where L AVGGMSA gives attack algorithm GMSA-AVG, and L MINGMSA gives attack algorithm GMSA-MIN.

Original: We solve (7) via Projected Gradient Decent (PGD) (the implementation details of GMSA can be found in Appendix A.1.3).
Revised: We solve (7) via Projected Gradient Decent (PGD) (the implementation details of GMSA can be found in Appendix A.1.3).

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  E MPIRICAL S TUDY
Revised:  E MPIRICAL S TUDY

Original: This section evaluates various transductive-learning based defenses.
Revised: This section evaluates various transductive-learning based defenses.

Original: Our main ﬁndings are: (1)
Revised: Our main ﬁndings are: (1)

Original: The robustness of existing transductive defenses like RMC and DENT is overestimated.
Revised: The robustness of existing transductive defenses like RMC and DENT is overestimated.

Original: Under our evaluation framework, those defenses either have little robustness or have almost the same robustness as that of the static base model.
Revised: Under our evaluation framework, those defenses either have little robustness or have almost the same robustness as that of the static base model.

Original: To this end, we note that while AutoAttack is effective in evaluating the robustness of static models, it is not effective in evaluating the robustness of transductive defenses.
Revised: To this end, we note that while AutoAttack is effective in evaluating the robustness of static models, it is not effective in evaluating the robustness of transductive defenses.

Original: In contrast, our GMSA attack is a strong baseline for attacking transductive defenses.
Revised: In contrast, our GMSA attack is a strong baseline for attacking transductive defenses.

Original: (2) We experimented a novel idea of applying Domain Adversarial Neural Networks (Ajakan et al., 2014), an unsupervised domain adaptation technique (Wilson & Cook, 2020), as a transductive-learning based defense.
Revised: (2) We experimented a novel idea of applying Domain Adversarial Neural Networks (Ajakan et al., 2014), an unsupervised domain adaptation technique (Wilson & Cook, 2020), as a transductive-learning based defense.

Original: We show that DANN has nontrivial and even better robustness compared to existing work, under AutoAttack, PGD attack, and FPA attack, even though it is broken by GMSA.
Revised: We show that DANN has nontrivial and even better robustness compared to existing work, under AutoAttack, PGD attack, and FPA attack, even though it is broken by GMSA.

Original: (3) We report a somewhat surprising phenomenon on transductive adversarial training :
Revised: (3) We report a somewhat surprising phenomenon on transductive adversarial training :

Original: Adversarially retraining the model using fresh private randomness on a new batch of test-time data gives a signiﬁcant increase in robustness, against all of our considered attacks.
Revised: Adversarially retraining the model using fresh private randomness on a new batch of test-time data gives a signiﬁcant increase in robustness, against all of our considered attacks.

Original: (4) Finally, we demonstrated that URejectron, while enjoying theoretical guarantees in the bounded-VC dimensions situation, can be broken in natural deep learning settings.
Revised: (4) Finally, we demonstrated that URejectron, while enjoying theoretical guarantees in the bounded-VC dimensions situation, can be broken in natural deep learning settings.

Original: Evaluation framework .
Revised: Evaluation framework .

Original: For each defense, we report accuracy and robustness.
Revised: For each defense, we report accuracy and robustness.

Original: The accuracy is the performance on the clean test inputs, and the robustness is the performance under adversarial attacks.
Revised: The accuracy is the performance on the clean test inputs, and the robustness is the performance under adversarial attacks.

Original: The robustness of transductive defenses is estimated using AutoAttack (AA) 3 , PGD attack, FPA,
Revised: The robustness of transductive defenses is estimated using AutoAttack (AA) 3 , PGD attack, FPA,

Original: GMSA-MIN and GMSA-AVG.
Revised: GMSA-MIN and GMSA-AVG.

Original: We use PGD attack and AutoAttack in the transfer attack setting for the transductive defense: We generate adversarial examples by attacking a static model (e.g. the base model used by the transductive defense), and then evaluate the transductive defense on the generated adversarial examples.
Revised: We use PGD attack and AutoAttack in the transfer attack setting for the transductive defense: We generate adversarial examples by attacking a static model (e.g. the base model used by the transductive defense), and then evaluate the transductive defense on the generated adversarial examples.

Original: Accuracy and robustness of the static models are also reported for comparison.
Revised: Accuracy and robustness of the static models are also reported for comparison.

Original: We always use AutoAttack to estimate the robustness of static models since it is the state-of-the-art for the inductive setting.
Revised: We always use AutoAttack to estimate the robustness of static models since it is the state-of-the-art for the inductive setting.

Original: For all experiments, the defender uses his own private randomness, which is different from the one used by the attacker.
Revised: For all experiments, the defender uses his own private randomness, which is different from the one used by the attacker.

Original: Without speciﬁed otherwise, all reported values are percentages.
Revised: Without speciﬁed otherwise, all reported values are percentages.

Original: Below we give details.
Revised: Below we give details.

Original: Appendix A gives details for replicating the results.
Revised: Appendix A gives details for replicating the results.

Original: Runtime Masking and Cleansing (RMC (Wu et al., 2020b)) .
Revised: Runtime Masking and Cleansing (RMC (Wu et al., 2020b)) .

Original: RMC adapts the network at test time, and was shown to achieve state-of-the-art robustness under several attacks that are unaware of the defense mechanism (thus these attacks are non-adaptive according to our deﬁnition).
Revised: RMC adapts the network at test time, and was shown to achieve state-of-the-art robustness under several attacks that are unaware of the defense mechanism (thus these attacks are non-adaptive according to our deﬁnition).

Original: We follow the setup in Wu et al.
Revised: We follow the setup in Wu et al.

Original: (2020b) to perform experiments on MNIST and CIFAR-10 to evaluate the
Revised: (2020b) to perform experiments on MNIST and CIFAR-10 to evaluate the

Original: [Table]
Revised: [Table]

Original: robustness of RMC.
Revised: robustness of RMC.

Original: On MNIST, we consider L ∞ norm attack with (cid:15) = 0 .
Revised: On MNIST, we consider L ∞ norm attack with (cid:15) = 0 .

Original: 3 and on CIFAR-10, we consider L ∞ norm attack with (cid:15)
Revised: 3 and on CIFAR-10, we consider L ∞ norm attack with (cid:15)

Original: = 8 / 255 .
Revised: = 8 / 255 .

Original: The performance of RMC is evaluated on a sequence of test points xxx (1) , · · · ,xxx ( n ) randomly sampled from the test dataset.
Revised: The performance of RMC is evaluated on a sequence of test points xxx (1) , · · · ,xxx ( n ) randomly sampled from the test dataset.

Original: So we have a n -round game.
Revised: So we have a n -round game.

Original: The
Revised: The

Original: FPA and GMSA attacks are applied on each round and the initial model F (0) used by the attacks at the ( k + 1) -th round is the adapted model (with calibration in RMC) obtained at the k -th round.
Revised: FPA and GMSA attacks are applied on each round and the initial model F (0) used by the attacks at the ( k + 1) -th round is the adapted model (with calibration in RMC) obtained at the k -th round.

Original: To save computational cost, we set n = 1000 .
Revised: To save computational cost, we set n = 1000 .

Original: The robustness of RMC is evaluated on a sequence of adversarial examples ˆ xxx (1) , · · · , ˆ xxx ( n ) generated by the attacker on the sequence of test points xxx (1) , · · · ,xxx ( n ) .
Revised: The robustness of RMC is evaluated on a sequence of adversarial examples ˆ xxx (1) , · · · , ˆ xxx ( n ) generated by the attacker on the sequence of test points xxx (1) , · · · ,xxx ( n ) .

Original: We evaluate the robustness of RMC in the non-state leaking setting with private randomness (both are in favor of the defender).
Revised: We evaluate the robustness of RMC in the non-state leaking setting with private randomness (both are in favor of the defender).

Original: Results.
Revised: Results.

Original: The results are in Table 1.
Revised: The results are in Table 1.

Original: RMC with the standard model is already broken by FPA attack (weaker than GSMA).
Revised: RMC with the standard model is already broken by FPA attack (weaker than GSMA).

Original: Compared to the defense-unaware AutoAttack, our GMSA-AVG attack reduces the robustness from 97 .
Revised: Compared to the defense-unaware AutoAttack, our GMSA-AVG attack reduces the robustness from 97 .

Original: 70% to 0 .
Revised: 70% to 0 .

Original: 50% on MNIST and from 94 .
Revised: 50% on MNIST and from 94 .

Original: 20% to 8 .
Revised: 20% to 8 .

Original: 00% on CIFAR-10.
Revised: 00% on CIFAR-10.

Original: Further,
Revised: Further,

Original: RMC with adversarially trained model actually provides worse adversarial robustness than using adversarial training alone.
Revised: RMC with adversarially trained model actually provides worse adversarial robustness than using adversarial training alone.

Original: Under our GMSA-MIN attack, the robustness is reduced from 96 .
Revised: Under our GMSA-MIN attack, the robustness is reduced from 96 .

Original: 10% to 58 .
Revised: 10% to 58 .

Original: 80% on MNIST and from 71 .
Revised: 80% on MNIST and from 71 .

Original: 70% to 39 .
Revised: 70% to 39 .

Original: 60% on CIFAR-10.
Revised: 60% on CIFAR-10.

Original: Defensive Entropy Minimization (DENT (Wang et al., 2021)) .
Revised: Defensive Entropy Minimization (DENT (Wang et al., 2021)) .

Original: DENT performs test-time adaptation, and works for any training-time learner.
Revised: DENT performs test-time adaptation, and works for any training-time learner.

Original: It was shown that DENT improves the robustness of the state-of-the-art adversarial training defenses by 20+ points absolute against AutoAttack on
Revised: It was shown that DENT improves the robustness of the state-of-the-art adversarial training defenses by 20+ points absolute against AutoAttack on

Original: CIFAR-10 under L ∞ norm attack with (cid:15) = 8 / 255 (DENT is implemented as a model module, and
Revised: CIFAR-10 under L ∞ norm attack with (cid:15) = 8 / 255 (DENT is implemented as a model module, and

Original: AutoAttack is directly applied to the module, and we denote this as DENT-AA).
Revised: AutoAttack is directly applied to the module, and we denote this as DENT-AA).

Original: Wang et al.
Revised: Wang et al.

Original: also considers adaptive attacks for DENT, such as attacking the static base model using AutoAttack to generate adversarial examples, which is the same as the AutoAttack (AA) in our evaluation.
Revised: also considers adaptive attacks for DENT, such as attacking the static base model using AutoAttack to generate adversarial examples, which is the same as the AutoAttack (AA) in our evaluation.

Original: We evaluate the best version of DENT, called DENT+ in Wang et al.
Revised: We evaluate the best version of DENT, called DENT+ in Wang et al.

Original: , under their original settings on CIFAR-10: DENT is combined with various adversarial training defenses, and only the model adaptation is included without input adaptation.
Revised: , under their original settings on CIFAR-10: DENT is combined with various adversarial training defenses, and only the model adaptation is included without input adaptation.

Original: The model is adapted sample-wise for six steps by
Revised: The model is adapted sample-wise for six steps by

Original: AdaMod (Ding et al., 2019) with learning rate of 0.006, batch size of 128 and no weight decay.
Revised: AdaMod (Ding et al., 2019) with learning rate of 0.006, batch size of 128 and no weight decay.

Original: The adaptation objective is entropy minimization with the information maximization regularization.
Revised: The adaptation objective is entropy minimization with the information maximization regularization.

Original: To save computational cost, we only evaluate on 1000 examples randomly sampled from the test dataset.
Revised: To save computational cost, we only evaluate on 1000 examples randomly sampled from the test dataset.

Original: We consider L ∞ norm attack with (cid:15) = 8 / 255 .
Revised: We consider L ∞ norm attack with (cid:15) = 8 / 255 .

Original: We design loss functions for the attacks to generate adversarial examples with high conﬁdence (See Appendix A.3 for the details).
Revised: We design loss functions for the attacks to generate adversarial examples with high conﬁdence (See Appendix A.3 for the details).

Original: Results.
Revised: Results.

Original: Table 2 show that both DENT-AA and AA overestimate the robustness of DENT.
Revised: Table 2 show that both DENT-AA and AA overestimate the robustness of DENT.

Original: Our PGD attack reduces the robustness of DENT to be almost the same as that of the static defenses.
Revised: Our PGD attack reduces the robustness of DENT to be almost the same as that of the static defenses.

Original: Further, our FPA, GMSA-AVG and GMSA-MIN have similar performance as the PGD attack.
Revised: Further, our FPA, GMSA-AVG and GMSA-MIN have similar performance as the PGD attack.

Original: The results show that AutoAttack is not effective in evaluating the robustness of transductive defenses.
Revised: The results show that AutoAttack is not effective in evaluating the robustness of transductive defenses.

Original: Domain Adversarial Neural Network (DANN (Ajakan et al., 2014)) .
Revised: Domain Adversarial Neural Network (DANN (Ajakan et al., 2014)) .

Original: We consider DANN as a transductive defense for adversarial robustness.
Revised: We consider DANN as a transductive defense for adversarial robustness.

Original: We train DANN on the labeled training dataset D (source domain) and unlabeled adversarial test dataset U (cid:48) (target domain), and then evaluate DANN on U (cid:48) .
Revised: We train DANN on the labeled training dataset D (source domain) and unlabeled adversarial test dataset U (cid:48) (target domain), and then evaluate DANN on U (cid:48) .

Original: For each adversarial set U (cid:48) , we train a new DANN model from scratch.
Revised: For each adversarial set U (cid:48) , we train a new DANN model from scratch.

Original: We use the standard model trained on D as the base model for DANN.
Revised: We use the standard model trained on D as the base model for DANN.

Original: We perform experiments on MNIST and CIFARto evaluate the adversarial robustness of DANN.
Revised: We perform experiments on MNIST and CIFARto evaluate the adversarial robustness of DANN.

Original: On MNIST, we consider L ∞ norm attack with (cid:15)
Revised: On MNIST, we consider L ∞ norm attack with (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 3 and on CIFAR-10, we consider L ∞ norm attack with (cid:15)
Revised: 3 and on CIFAR-10, we consider L ∞ norm attack with (cid:15)

Original: = 8 / 255 .
Revised: = 8 / 255 .

Original: Results.
Revised: Results.

Original: Table 3 show that DANN has non-trivial robustness under AutoAttack, PGD attack and FPA attack.
Revised: Table 3 show that DANN has non-trivial robustness under AutoAttack, PGD attack and FPA attack.

Original: However, under our GMSA attack, DANN has little robustness.
Revised: However, under our GMSA attack, DANN has little robustness.

Original: Transductive Adversarial Training (TADV) .
Revised: Transductive Adversarial Training (TADV) .

Original: We consider a simple but novel transductive-learning based defense called transductive adversarial training:
Revised: We consider a simple but novel transductive-learning based defense called transductive adversarial training:

Original: After receiving a set of examples at the test time, we always adversarially retrain the model using fresh randomness.
Revised: After receiving a set of examples at the test time, we always adversarially retrain the model using fresh randomness.

Original: The key point of this transduction is that private randomness is sampled after the attacker’s move, and so the attacker
Revised: The key point of this transduction is that private randomness is sampled after the attacker’s move, and so the attacker

Original: [Figure]
Revised: [Figure]

Original: al
Revised: Madry

Original: [Figure]
Revised: [Figure]

Original: TADV
Revised: al.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: cannot directly attack the resulting model as in the inductive case.
Revised: cannot directly attack the resulting model as in the inductive case.

Original: Speciﬁcally, for our GMSA attacks, we attack (with loss L AVGGMSA or L MINGMSA )
Revised: Speciﬁcally, for our GMSA attacks, we attack (with loss L AVGGMSA or L MINGMSA )

Original: an ensemble of T = 10 models, adversarially trained with independent randomness, and generate a perturbed test set U (cid:48) .
Revised: an ensemble of T = 10 models, adversarially trained with independent randomness, and generate a perturbed test set U (cid:48) .

Original: Then we adversarially train another model from scratch with independent randomness, and check whether U (cid:48) transfers to the new model (this thus captures the scenario described earlier).
Revised: Then we adversarially train another model from scratch with independent randomness, and check whether U (cid:48) transfers to the new model (this thus captures the scenario described earlier).

Original: Somewhat surprisingly, we show that U (cid:48) does not transfer very well, and the TADV improves robustness signiﬁcantly.
Revised: Somewhat surprisingly, we show that U (cid:48) does not transfer very well, and the TADV improves robustness signiﬁcantly.

Original: Results.
Revised: Results.

Original: Table 4 shows that transductive adversarial training signiﬁcantly improves the robustness of adversarial training (Madry et al., 2018).
Revised: Table 4 shows that transductive adversarial training signiﬁcantly improves the robustness of adversarial training (Madry et al., 2018).

Original: On MNIST, the robustness is improved from 86.61% to 94.27%.
Revised: On MNIST, the robustness is improved from 86.61% to 94.27%.

Original: On CIFAR-10, the robustness is improved from 45.29% to 54.12%.
Revised: On CIFAR-10, the robustness is improved from 45.29% to 54.12%.

Original: URejectron in deep learning settings .
Revised: URejectron in deep learning settings .

Original: URejectron performs transductive learning for defense, and has theoretical guarantees under bounded VC dimension case.
Revised: URejectron performs transductive learning for defense, and has theoretical guarantees under bounded VC dimension case.

Original: We evaluated URejectron on GTSRB dataset using ResNet18 network.
Revised: We evaluated URejectron on GTSRB dataset using ResNet18 network.

Original: We used the same implementation by Goldwasser et al..
Revised: We used the same implementation by Goldwasser et al..

Original: Results .
Revised: Results .

Original: Figure 1(a) shows that for transfer attacks generated by PGD attack (Madry et al., 2018),
Revised: Figure 1(a) shows that for transfer attacks generated by PGD attack (Madry et al., 2018),

Original: URejectron can indeed work as expected.
Revised: URejectron can indeed work as expected.

Original: However, by using different attack algorithms, such as
Revised: However, by using different attack algorithms, such as

Original: CW attack (Carlini & Wagner, 2017), we observe two failure modes: (1) Imperceptible adversarial perturbations that slip through .
Revised: CW attack (Carlini & Wagner, 2017), we observe two failure modes: (1) Imperceptible adversarial perturbations that slip through .

Original: Figure 1(b) shows that one can construct adversarial examples that are very similar to the clean test inputs that can slip through their URejectron construction of S (in the deep learning setting), and cause large errors.
Revised: Figure 1(b) shows that one can construct adversarial examples that are very similar to the clean test inputs that can slip through their URejectron construction of S (in the deep learning setting), and cause large errors.

Original: (2) Benign perturbations that get rejected .
Revised: (2) Benign perturbations that get rejected .

Original: Figure 1(c) shows that we can generate “benign” perturbed examples using image corruptions, such as slightly increased brightness, but URejectron rejects all.
Revised: Figure 1(c) shows that we can generate “benign” perturbed examples using image corruptions, such as slightly increased brightness, but URejectron rejects all.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we formulate threat models for transductive defenses and propose an attack framework called
Revised: In this paper, we formulate threat models for transductive defenses and propose an attack framework called

Original: Greedy Model Space Attack (GMSA) that can serve as a new baseline for evaluating transductive defenses.
Revised: Greedy Model Space Attack (GMSA) that can serve as a new baseline for evaluating transductive defenses.

Original: We show that GMSA can break previous transductive defenses, which were resilient to previous attacks such as AutoAttack.
Revised: We show that GMSA can break previous transductive defenses, which were resilient to previous attacks such as AutoAttack.

Original: On the positive side, we show that transductive adversarial training gives a signiﬁcant increase in robustness against attacks we consider.
Revised: On the positive side, we show that transductive adversarial training gives a signiﬁcant increase in robustness against attacks we consider.

Original: For the future work, one can explore transductive defenses that can be robust under our GMSA attacks and can also explore even stronger adaptive attacks that are effective in evaluating transductive defenses
Revised: For the future work, one can explore transductive defenses that can be robust under our GMSA attacks, and can also explore even stronger adaptive attacks that are effective in evaluating transductive defenses. A CKNOWLEDGMENTS

Revised: The work is partially supported by Air Force Grant FA9550-18-1-0166, the National Science Foundation (NSF) Grants CCF-FMitF-1836978, IIS-2008559, SaTC-Frontiers-1804648, CCFand CCF- and ARO grant number W911NF-17-1-0405.

Revised: Jiefeng Chen and Somesh Jha are partially supported by the DARPA-GARD problem under agreement number 885000.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: We believe that our work gives positive societal impact in the long term.
Revised: We believe that our work gives positive societal impact in the long term.

Original: In the short term potentially some services deploying existing transductive defenses may be broken by adversaries who leverage our new attacks.
Revised: In the short term potentially some services deploying existing transductive defenses may be broken by adversaries who leverage our new attacks.

Original: However, our ﬁndings give a necessary step to identify transductive defenses that really work and deepened our understanding of this matter.
Revised: However, our ﬁndings give a necessary step to identify transductive defenses that really work and deepened our understanding of this matter.

Original: It also gives positive impact by advancing the science for trustworthy machine learning and potentially how deep transductive learning works.
Revised: It also gives positive impact by advancing the science for trustworthy machine learning and potentially how deep transductive learning works

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We have included enough experimental details to ensure reproducibility in Section 6 and Appendix A.
Revised: We have included enough experimental details to ensure reproducibility in Section 6 and Appendix A.

Original: Also, the source code for the experiments is submitted as supplemental materials
Revised: Also, the source code for the experiments is submitted as supplemental materials

Original: Supplementary Material TowardsEvaluating the
Revised: Supplementary Material Towards

Original:  Robustness of Neural Networks Learned 
Revised: Evaluating the Robustness of Neural Networks Learned by

Original: by Transduction A E XPERIMENTAL D ETAILS A. G ENERAL S ETUP A.1. C OMPUTING I NFRASTRUCTURE
Revised: Transduction A E XPERIMENTAL D ETAILS A. G ENERAL S ETUP A.1. C OMPUTING I NFRASTRUCTURE

Original: We run all experiments with PyTorch and NVIDIA GeForce RTX 2080Ti GPUs.
Revised: We run all experiments with PyTorch and NVIDIA GeForce RTX 2080Ti GPUs.

Original: A.1. D ATASET
Revised: A.1. D ATASET

Original: We use three datasets MNIST, CIFAR-10 and GTSRB in our experiments.
Revised: We use three datasets MNIST, CIFAR-10 and GTSRB in our experiments.

Original: The details about these datasets are described below.
Revised: The details about these datasets are described below.

Original: MNIST.
Revised: MNIST.

Original: The MNIST (LeCun, 1998) is a large dataset of handwritten digits.
Revised: The MNIST (LeCun, 1998) is a large dataset of handwritten digits.

Original: Each digit has 5, training images and 1,000 test images.
Revised: Each digit has 5, training images and 1,000 test images.

Original: Each image is a 28 × 28 grayscale.
Revised: Each image is a 28 × 28 grayscale.

Original: We normalize the range of pixel values to [0 , 1] .
Revised: We normalize the range of pixel values to [0 , 1] .

Original: CIFAR-10.
Revised: CIFAR-10.

Original: The CIFAR-10 (Krizhevsky et al., 2009) is a dataset of 32x32 color images with ten classes, each consisting of 5,000 training images and 1,000 test images.
Revised: The CIFAR-10 (Krizhevsky et al., 2009) is a dataset of 32x32 color images with ten classes, each consisting of 5,000 training images and 1,000 test images.

Original: The classes correspond to dogs, frogs, ships, trucks, etc.
Revised: The classes correspond to dogs, frogs, ships, trucks, etc.

Original: We normalize the range of pixel values to [0 , 1] .
Revised: We normalize the range of pixel values to [0 , 1] .

Original: GTSRB.
Revised: GTSRB.

Original: The German Trafﬁc Sign Recognition Benchmark (GTSRB) (Stallkamp et al., 2012) is a dataset of color images depicting 43 different trafﬁc signs.
Revised: The German Trafﬁc Sign Recognition Benchmark (GTSRB) (Stallkamp et al., 2012) is a dataset of color images depicting 43 different trafﬁc signs.

Original: The images are not of a ﬁxed dimensions and have rich background and varying light conditions as would be expected of photographed images of trafﬁc signs.
Revised: The images are not of a ﬁxed dimensions and have rich background and varying light conditions as would be expected of photographed images of trafﬁc signs.

Original: There are about 34,799 training images, 4,410 validation images and 12,630 test images.
Revised: There are about 34,799 training images, 4,410 validation images and 12,630 test images.

Original: We resize each image to 32 × 32 .
Revised: We resize each image to 32 × 32 .

Original: The dataset has a large imbalance in the number of sample occurrences across classes.
Revised: The dataset has a large imbalance in the number of sample occurrences across classes.

Original: We use data augmentation techniques to enlarge the training data and make the number of samples in each class balanced.
Revised: We use data augmentation techniques to enlarge the training data and make the number of samples in each class balanced.

Original: We construct a class preserving data augmentation pipeline consisting of rotation, translation, and projection transforms and apply this pipeline to images in the training set until each class contained 10,000 examples.
Revised: We construct a class preserving data augmentation pipeline consisting of rotation, translation, and projection transforms and apply this pipeline to images in the training set until each class contained 10,000 examples.

Original: We also preprocess images via image brightness normalization and normalize the range of pixel values to [0 , 1] .
Revised: We also preprocess images via image brightness normalization and normalize the range of pixel values to [0 , 1] .

Original: A
Revised: A.

Original: .1. I MPLEMENTATION D ETAILS OF THE A TTACKS
Revised: 1. I MPLEMENTATION D ETAILS OF THE A TTACKS

Original: We use Projected Gradient Descent (PGD) (Madry et al., 2018) to solve the attack objectives of PGD attack, FPA, GMSA-AVG and GMSA-MIN.
Revised: We use Projected Gradient Descent (PGD) (Madry et al., 2018) to solve the attack objectives of PGD attack, FPA, GMSA-AVG and GMSA-MIN.

Original: For GMSA-AVG, at the i -th iteration, when applying
Revised: For GMSA-AVG, at the i -th iteration, when applying

Original: PGD on the data point xxx to generate the perturbation δ , we need to do one backpropagation operation for each model in { F ( j ) } i j =0 per PGD step.
Revised: PGD on the data point xxx to generate the perturbation δ , we need to do one backpropagation operation for each model in { F ( j ) } i j =0 per PGD step.

Original: We do the backpropagation for each model sequentially and then accumulate the gradients to update the perturbation δ since we might not have enough memory to store all the models and compute the gradients at once, especially when i is large.
Revised: We do the backpropagation for each model sequentially and then accumulate the gradients to update the perturbation δ since we might not have enough memory to store all the models and compute the gradients at once, especially when i is large.

Original: For
Revised: For

Original: GMSA-MIN, we ﬁnd that it requires more PGD steps to solve the attack objective at the i -th iteration where we need to attack i + 1 models simultaneously.
Revised: GMSA-MIN, we ﬁnd that it requires more PGD steps to solve the attack objective at the i -th iteration where we need to attack i + 1 models simultaneously.

Original: Thus, we scale the number of PGD steps at the i -th iteration by a factor of i + 1 for GMSA-MIN.
Revised: Thus, we scale the number of PGD steps at the i -th iteration by a factor of i + 1 for GMSA-MIN.

Original: A. S ETUP FOR RMC E XPERIMENTS
Revised: A. S ETUP FOR RMC E XPERIMENTS

Original: We follow the original settings in Wu et al.
Revised: We follow the original settings in Wu et al.

Original: (2020b) to perform experiments on MNIST and CIFARdatasets to evaluate the adversarial robustness of RMC.
Revised: (2020b) to perform experiments on MNIST and CIFARdatasets to evaluate the adversarial robustness of RMC.

Original: We consider two kinds of base models for
Revised: We consider two kinds of base models for

Original: RMC: one is the model trained via standard supervised training; the other is the model trained using the adversarial training (Madry et al., 2018).
Revised: RMC: one is the model trained via standard supervised training; the other is the model trained using the adversarial training (Madry et al., 2018).

Original: We describe the settings for each dataset below.
Revised: We describe the settings for each dataset below.

Original: A.2. MNIST
Revised: A.2. MNIST

Original: Model architecture and training conﬁguration.
Revised: Model architecture and training conﬁguration.

Original: We use a neural network with two convolutional layers, two full connected layers and batch normalization layers.
Revised: We use a neural network with two convolutional layers, two full connected layers and batch normalization layers.

Original: For both standard training and adversarial training, we train the model for 100 epochs using the Adam optimizer with a batch size of 128 and a learning rate of 10 − 3 .
Revised: For both standard training and adversarial training, we train the model for 100 epochs using the Adam optimizer with a batch size of 128 and a learning rate of 10 − 3 .

Original: We use the L ∞ norm PGD attack as the adversary for adversarial training with a perturbation budget (cid:15) of 0 . 3 , a step size of 0 . 01 , and number of steps of 40 .
Revised: We use the L ∞ norm PGD attack as the adversary for adversarial training with a perturbation budget (cid:15) of 0 . 3 , a step size of 0 . 01 , and number of steps of 40 .

Original: RMC conﬁguration.
Revised: RMC conﬁguration.

Original: We set K = 1024 .
Revised: We set K = 1024 .

Original: Suppose the clean training set is D .
Revised: Suppose the clean training set is D .

Original: Let D (cid:48) contain | D | clean inputs and | D | adversarial examples.
Revised: Let D (cid:48) contain | D | clean inputs and | D | adversarial examples.

Original: So N (cid:48)
Revised: So N (cid:48)

Original: = 

Revised: =

Original: | D | .
Revised: | D | .

Original: We generate the adversarial examples using the L ∞ norm PGD attack with a perturbation budget (cid:15) of 0 .
Revised: We generate the adversarial examples using the L ∞ norm PGD attack with a perturbation budget (cid:15) of 0 .

Original: 3 , a step size of 0 . 01 , and number of steps of 100 .
Revised: 3 , a step size of 0 . 01 , and number of steps of 100 .

Original: We extract the features from the penultimate layer of the model and use the Euclidean distance in the feature space of the model to ﬁnd the top-K nearest neighbors of the inputs.
Revised: We extract the features from the penultimate layer of the model and use the Euclidean distance in the feature space of the model to ﬁnd the top-K nearest neighbors of the inputs.

Original: When adapting the model, we use Adam as the optimizer and set the learning rate to be 2 × 10 − 4 .
Revised: When adapting the model, we use Adam as the optimizer and set the learning rate to be 2 × 10 − 4 .

Original: We train the model until the early-stop condition holds.
Revised: We train the model until the early-stop condition holds.

Original: That is the training epoch reaches 100 or the validation loss doesn’t decrease for 5 epochs.
Revised: That is the training epoch reaches 100 or the validation loss doesn’t decrease for 5 epochs.

Original: Attack conﬁguration.
Revised: Attack conﬁguration.

Original: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 0 . 3 .
Revised: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 0 . 3 .

Original: Cross entropy loss is used as the loss function for PGD attack, FPA,
Revised: Cross entropy loss is used as the loss function for PGD attack, FPA,

Original: GMSA-AVG and GMSA-MIN:
Revised: GMSA-AVG and GMSA-MIN:

Original: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y
Revised: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y

Original: ) ∈ V − log f
Revised: ) ∈ V − log f

Original: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .
Revised: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .

Original: We use PGD with a step size of 0 .
Revised: We use PGD with a step size of 0 .

Original: 01 , the number of steps of 100 , random start and no restarts.
Revised: 01 , the number of steps of 100 , random start and no restarts.

Original: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.
Revised: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.

Original: A.2. CIFAR- 
Revised: A.2. CIFAR-

Original: Model architecture and training conﬁguration.
Revised: Model architecture and training conﬁguration.

Original: We use the ResNet-32 network (He et al., 2016).
Revised: We use the ResNet-32 network (He et al., 2016).

Original: For both standard training and adversarial training, we train the model for 100 epochs using Stochastic
Revised: For both standard training and adversarial training, we train the model for 100 epochs using Stochastic

Original: Gradient Decent (SGD) optimizer with Nesterov momentum and learning rate schedule.
Revised: Gradient Decent (SGD) optimizer with Nesterov momentum and learning rate schedule.

Original: We set momentum 0 .
Revised: We set momentum 0 .

Original: 9 and (cid:96) 2 weight decay with a coefﬁcient of 10 − 4 .
Revised: 9 and (cid:96) 2 weight decay with a coefﬁcient of 10 − 4 .

Original: The initial learning rate is 0 .
Revised: The initial learning rate is 0 .

Original: and it decreases by 0 .
Revised: and it decreases by 0 .

Original: 1 at 50, 75 and 90 epoch respectively.
Revised: 1 at 50, 75 and 90 epoch respectively.

Original: The batch size is 128 .
Revised: The batch size is 128 .

Original: We augment the training images using random crop and random horizontal ﬂip.
Revised: We augment the training images using random crop and random horizontal ﬂip.

Original: We use the L ∞ norm PGD attack as the adversary for adversarial training with a perturbation budget (cid:15) of 8 / 255 , a step size of 2 / 255 , and number of steps of 10 .
Revised: We use the L ∞ norm PGD attack as the adversary for adversarial training with a perturbation budget (cid:15) of 8 / 255 , a step size of 2 / 255 , and number of steps of 10 .

Original: RMC conﬁguration.
Revised: RMC conﬁguration.

Original: We set K = 1024 .
Revised: We set K = 1024 .

Original: Suppose the clean training set is D .
Revised: Suppose the clean training set is D .

Original: Let D (cid:48) contain | D | clean inputs and 4 | D | adversarial examples.
Revised: Let D (cid:48) contain | D | clean inputs and 4 | D | adversarial examples.

Original: So N (cid:48)
Revised: So N (cid:48)

Original: = 5 | D | .
Revised: = 5 | D | .

Original: We generate the adversarial examples using the L ∞ norm PGD attack with a perturbation budget (cid:15) of 8 / 255 , a step size of 1 / 255 , and number of steps of 40 .
Revised: We generate the adversarial examples using the L ∞ norm PGD attack with a perturbation budget (cid:15) of 8 / 255 , a step size of 1 / 255 , and number of steps of 40 .

Original: We extract the features from the penultimate layer of the model and use the
Revised: We extract the features from the penultimate layer of the model and use the

Original: Euclidean distance in the feature space of the model to ﬁnd the top-K nearest neighbors of the inputs.
Revised: Euclidean distance in the feature space of the model to ﬁnd the top-K nearest neighbors of the inputs.

Original: We use Adam as the optimizer and set the learning rate to be 2 .
Revised: We use Adam as the optimizer and set the learning rate to be 2 .

Original: 5 × 10 − 5 .
Revised: 5 × 10 − 5 .

Original: Attack conﬁguration.
Revised: Attack conﬁguration.

Original: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 8 / 255 .
Revised: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 8 / 255 .

Original: Cross entropy loss is used as the loss function for PGD attack, FPA,
Revised: Cross entropy loss is used as the loss function for PGD attack, FPA,

Original: GMSA-AVG and GMSA-MIN:
Revised: GMSA-AVG and GMSA-MIN:

Original: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y
Revised: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y

Original: ) ∈ V − log f
Revised: ) ∈ V − log f

Original: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .
Revised: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .

Original: We use PGD with a step size of 1 / 255 , the number of steps of 40 , random start and no restarts.
Revised: We use PGD with a step size of 1 / 255 , the number of steps of 40 , random start and no restarts.

Original: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.
Revised: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.

Original: A. S ETUP FOR DENT E XPERIMENTS
Revised: A. S ETUP FOR DENT E XPERIMENTS

Original: DENT conﬁguration.
Revised: DENT conﬁguration.

Original: We perform experiments to evaluate the best version of DENT (DENT+ in Wang et al.
Revised: We perform experiments to evaluate the best version of DENT (DENT+ in Wang et al.

Original: on CIFAR-10 following the experimental settings in Wang et al.
Revised: on CIFAR-10 following the experimental settings in Wang et al.

Original: We use the pre-trained robust models on CIFAR-10 under the L ∞ norm perturbation threat model from RobustBench Model Zoo 4 as the static base models for DENT, including models with model ID Wu2020Adversarial_extra (Wu et al., 2020a), Carmon2019Unlabeled (Carmon et al., 2019), Sehwag2020Hydra (Sehwag et al., 2020), Wang2020Improving (Wang et al., 2020), Hendrycks2019Using (Hendrycks et al., 2019), Wong2020Fast
Revised: We use the pre-trained robust models on CIFAR-10 under the L ∞ norm perturbation threat model from RobustBench Model Zoo 4 as the static base models for DENT, including models with model ID Wu2020Adversarial_extra (Wu et al., 2020a), Carmon2019Unlabeled (Carmon et al., 2019), Sehwag2020Hydra (Sehwag et al., 2020), Wang2020Improving (Wang et al., 2020), Hendrycks2019Using (Hendrycks et al., 2019), Wong2020Fast

Original: (Wong et al., 2020), and
Revised: (Wong et al., 2020), and

Original: Ding2020MMA (Ding et al., 2020).
Revised: Ding2020MMA (Ding et al., 2020).

Original: For the test-time adaptation, only the afﬁne scale γ and shift β parameters in the batch normalization layers of the base model are updated.
Revised: For the test-time adaptation, only the afﬁne scale γ and shift β parameters in the batch normalization layers of the base model are updated.

Original: DENT updates sample-wise with different afﬁne parameters ( γ i , β i ) for each input
Revised: DENT updates sample-wise with different afﬁne parameters ( γ i , β i ) for each input

Original: xxx i .
Revised: xxx i .

Original: The input adaptation of Σ is not used as suggested in Wang et al.
Revised: The input adaptation of Σ is not used as suggested in Wang et al.

Original: The model is adapted for six steps by AdaMod (Ding et al., 2019) with learning rate of 0.006, batch size of 128 and no weight decay.
Revised: The model is adapted for six steps by AdaMod (Ding et al., 2019) with learning rate of 0.006, batch size of 128 and no weight decay.

Original: The adaptation objective is entropy minimization with the information maximization regularization:
Revised: The adaptation objective is entropy minimization with the information maximization regularization:

Original: [Equation]
Revised: [Equation]

Original: where b is the batch size, C is the number of classes and f ( xxx i ; θ i ) is the softmax output of the model f with the afﬁne parameters θ i =
Revised: where b is the batch size, C is the number of classes and f ( xxx i ; θ i ) is the softmax output of the model f with the afﬁne parameters θ i =

Original: ( γ i , β i ) for the input xxx i .
Revised: ( γ i , β i ) for the input xxx i .

Original: Attack conﬁguration.
Revised: Attack conﬁguration.

Original: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 8 / 255 .
Revised: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 8 / 255 .

Original: For PGD attack, FPA, GMSA-AVG and GMSA-MIN, we use the following loss function to ﬁnd adversarial examples with high conﬁdence: L a ( F, V ) =|
Revised: For PGD attack, FPA, GMSA-AVG and GMSA-MIN, we use the following loss function to ﬁnd adversarial examples with high conﬁdence: L a ( F, V ) =|

Original: V | (cid:80) ( xxx,y
Revised: V | (cid:80) ( xxx,y

Original: ) ∈ V max k (cid:54) =
Revised: ) ∈ V max k (cid:54) =

Original: y f ( xxx ) k , where f ( xxx ) is the softmax output of the model F .
Revised: y f ( xxx ) k , where f ( xxx ) is the softmax output of the model F .

Original: However, it is hard to optimize this loss function.
Revised: However, it is hard to optimize this loss function.

Original: Thus, we use two alternative loss functions to ﬁnd adversarial examples.
Revised: Thus, we use two alternative loss functions to ﬁnd adversarial examples.

Original: One is the untargeted CW loss (Carlini & Wagner, 2017):
Revised: One is the untargeted CW loss (Carlini & Wagner, 2017):

Original: L 1 a ( F, V ) = 1 | V | (cid:80) ( xxx,y ) ∈ V − Z ( xxx ) y
Revised: L 1 a ( F, V ) = 1 | V | (cid:80) ( xxx,y ) ∈ V − Z ( xxx ) y

Original: + max k (cid:54) =
Revised: + max k (cid:54) =

Original: y Z ( x x x ) k , where Z ( x x x ) is the logits of the model F (the output of the layer before the softmax layer).
Revised: y Z ( x x x ) k , where Z ( x x x ) is the logits of the model F (the output of the layer before the softmax layer).

Original: The other is the targeted CW loss: L 2 a ( F, V ) = 1 | V | (cid:80) ( xxx,y
Revised: The other is the targeted CW loss: L 2 a ( F, V ) = 1 | V | (cid:80) ( xxx,y

Original: ) ∈
Revised: ) ∈

Original: V − Z ( xxx )
Revised: V − Z ( xxx )

Original: y + Z ( xxx ) t , where t is the targeted label and t (cid:54) =
Revised: y + Z ( xxx ) t , where t is the targeted label and t (cid:54) =

Original: y .
Revised: y .

Original: For each attack, we use 14 PGD subroutines to solve its attack objective, including 5 PGD subroutines using the untargeted CW loss L 1 a with different random restarts and 9 PGD subroutines using the targeted CW loss L 2 a with different targeted labels.
Revised: For each attack, we use 14 PGD subroutines to solve its attack objective, including 5 PGD subroutines using the untargeted CW loss L 1 a with different random restarts and 9 PGD subroutines using the targeted CW loss L 2 a with different targeted labels.

Original: So for each clean test input xxx , these PGD subroutines will return 14 adversarial examples xxx (cid:48) 1 , . .
Revised: So for each clean test input xxx , these PGD subroutines will return 14 adversarial examples xxx (cid:48) 1 , . .

Original: ,xxx (cid:48) 14 .
Revised: ,xxx (cid:48) 14 .

Original: Among these adversarial examples, we select the one that maximizes the attack loss with the loss function L a ( F, V ) as the ﬁnal adversarial example xxx (cid:48) for xxx .
Revised: Among these adversarial examples, we select the one that maximizes the attack loss with the loss function L a ( F, V ) as the ﬁnal adversarial example xxx (cid:48) for xxx .

Original: We use the same hyper-parameters for all PGD subroutines: the step size is 1 / 255 , the number of steps is 100 , and the random start is used.
Revised: We use the same hyper-parameters for all PGD subroutines: the step size is 1 / 255 , the number of steps is 100 , and the random start is used.

Original: We set T = 2 for
Revised: We set T = 2 for

Original: FPA, GMSA-AVG and GMSA-MIN.
Revised: FPA, GMSA-AVG and GMSA-MIN.

Original: A. S ETUP FOR DANN E XPERIMENTS
Revised: A. S ETUP FOR DANN E XPERIMENTS

Original: We perform experiments on MNIST and CIFAR-10 datasets.
Revised: We perform experiments on MNIST and CIFAR-10 datasets.

Original: We describe the settings for each dataset below.
Revised: We describe the settings for each dataset below.

Original: A.4. MNIST
Revised: A.4. MNIST

Original: Model architecture.
Revised: Model architecture.

Original: We use the same model architecture as the one used in Chuang et al.
Revised: We use the same model architecture as the one used in Chuang et al.

Original: (2020), which is shown below.
Revised: (2020), which is shown below.

Original: Encoder nn.
Revised: Encoder nn.

Original: Conv2d(3, 64, kernel_size=5) nn.
Revised: Conv2d(3, 64, kernel_size=5) nn.

Original: BatchNorm2d nn.MaxPool2d(2)
Revised: BatchNorm2d nn.MaxPool2d(2)

Original: nn.ReLU nn.
Revised: nn.ReLU nn.

Original: Conv2d(64, 128, kernel_size=5) nn.
Revised: Conv2d(64, 128, kernel_size=5) nn.

Original: BatchNorm2d nn.
Revised: BatchNorm2d nn.

Original: Dropout2d nn.MaxPool2d(2) nn.ReLU nn.
Revised: Dropout2d nn.MaxPool2d(2) nn.ReLU nn.

Original: Conv2d(128, 128, kernel_size=3, padding=1)
Revised: Conv2d(128, 128, kernel_size=3, padding=1)

Original: nn.BatchNorm2dnn.
Revised: nn.BatchNorm2dnn.

Original: ReLU ×
Revised: ReLU ×

Original: [Table]
Revised: [Table]

Original: Training conﬁguration.
Revised: Training conﬁguration.

Original: We train the models for 100 epochs using the Adam optimizer with a batch size of 128 and a learning rate of 10 − 3 .
Revised: We train the models for 100 epochs using the Adam optimizer with a batch size of 128 and a learning rate of 10 − 3 .

Original: For the representation matching in DANN, we adopt the original progressive training strategy for the discriminator (Ganin et al., 2016) where the weight α for the domain-invariant loss is initiated at 0 and is gradually changed to 0.1 using the schedule
Revised: For the representation matching in DANN, we adopt the original progressive training strategy for the discriminator (Ganin et al., 2016) where the weight α for the domain-invariant loss is initiated at 0 and is gradually changed to 0.1 using the schedule

Original: α = ( 2 1+ exp ( − 10 · p ) − 1) · 0 . 1 , where p is the training progress linearly changing from 0 to 1.
Revised: α = ( 2 1+ exp ( − 10 · p ) − 1) · 0 . 1 , where p is the training progress linearly changing from 0 to 1.

Original: Attack conﬁguration.
Revised: Attack conﬁguration.

Original: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 0 . 3 .
Revised: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 0 . 3 .

Original: Cross entropy loss is used as the loss function for PGD attack, FPA,
Revised: Cross entropy loss is used as the loss function for PGD attack, FPA,

Original: GMSA-AVG and GMSA-MIN:
Revised: GMSA-AVG and GMSA-MIN:

Original: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y
Revised: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y

Original: ) ∈ V − log f
Revised: ) ∈ V − log f

Original: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .
Revised: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .

Original: We use PGD with a step size of 0 .
Revised: We use PGD with a step size of 0 .

Original: 01 , the number of steps of 200 , random start and no restarts.
Revised: 01 , the number of steps of 200 , random start and no restarts.

Original: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.
Revised: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.

Original: A.4. CIFAR- 
Revised: A.4. CIFAR-

Original: Model architecture.
Revised: Model architecture.

Original: We use the ResNet-18 network (He et al., 2016) and extract the features from the third basic block for representation matching.
Revised: We use the ResNet-18 network (He et al., 2016) and extract the features from the third basic block for representation matching.

Original: The detailed model architecture is shown below.
Revised: The detailed model architecture is shown below.

Original: [Table]
Revised: [Table]

Original: Training conﬁguration.
Revised: Training conﬁguration.

Original: We train the models for 100 epochs using stochastic gradient decent (SGD) optimizer with Nesterov momentum and learning rate schedule.
Revised: We train the models for 100 epochs using stochastic gradient decent (SGD) optimizer with Nesterov momentum and learning rate schedule.

Original: We set momentum 0 .
Revised: We set momentum 0 .

Original: 9 and (cid:96) 2 weight decay with a coefﬁcient of 10 − 4 .
Revised: 9 and (cid:96) 2 weight decay with a coefﬁcient of 10 − 4 .

Original: The initial learning rate is 0 .
Revised: The initial learning rate is 0 .

Original: and it decreases by 0 .
Revised: and it decreases by 0 .

Original: 1 at 50, 75 and 90 epoch respectively.
Revised: 1 at 50, 75 and 90 epoch respectively.

Original: The batch size is 64 .
Revised: The batch size is 64 .

Original: We augment the training images using random crop and random horizontal ﬂip.
Revised: We augment the training images using random crop and random horizontal ﬂip.

Original: For the representation matching in DANN, we adopt the original progressive training strategy for the discriminator (Ganin et al., 2016) where the weight α for the domain-invariant loss is initiated at 0 and is gradually changed to 1 using the schedule α = 2 1+ exp ( − 10 · p ) − 1 , where p is the training progress linearly changing from 0 to 1.
Revised: For the representation matching in DANN, we adopt the original progressive training strategy for the discriminator (Ganin et al., 2016) where the weight α for the domain-invariant loss is initiated at 0 and is gradually changed to 1 using the schedule α = 2 1+ exp ( − 10 · p ) − 1 , where p is the training progress linearly changing from 0 to 1.

Original: Attack conﬁguration.
Revised: Attack conﬁguration.

Original: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 8 / 255 .
Revised: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 8 / 255 .

Original: Cross entropy loss is used as the loss function for PGD attack, FPA,
Revised: Cross entropy loss is used as the loss function for PGD attack, FPA,

Original: GMSA-AVG and GMSA-MIN:
Revised: GMSA-AVG and GMSA-MIN:

Original: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y
Revised: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y

Original: ) ∈ V − log f
Revised: ) ∈ V − log f

Original: ( xxx ) y , where f ( xxx ) is 
Revised: ( xxx ) y , where f ( xxx ) is the

Original: A. S ETUP FOR TADV E XPERIMENTS
Revised: A. S ETUP FOR TADV E XPERIMENTS

Original: We perform experiments on MNIST and CIFAR-10 datasets.
Revised: We perform experiments on MNIST and CIFAR-10 datasets.

Original: We describe the settings for each dataset below.
Revised: We describe the settings for each dataset below.

Original: A.5. MNIST
Revised: A.5. MNIST

Original: Model architecture and Training conﬁguration.
Revised: Model architecture and Training conﬁguration.

Original: We use the LeNet network architecture.
Revised: We use the LeNet network architecture.

Original: We train the models for 100 epochs using the Adam optimizer with a batch size of 128 and a learning rate of 10 − 3 .
Revised: We train the models for 100 epochs using the Adam optimizer with a batch size of 128 and a learning rate of 10 − 3 .

Original: We use the L ∞ norm PGD attack as the adversary to generate adversarial training examples with a perturbation budget (cid:15) of 0 . 3 , a step size of 0 . 01 , and number of steps of 40 .
Revised: We use the L ∞ norm PGD attack as the adversary to generate adversarial training examples with a perturbation budget (cid:15) of 0 . 3 , a step size of 0 . 01 , and number of steps of 40 .

Original: We train on 50% clean and 50% adversarial examples per batch.
Revised: We train on 50% clean and 50% adversarial examples per batch.

Original: Attack conﬁguration.
Revised: Attack conﬁguration.

Original: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 0 . 3 .
Revised: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 0 . 3 .

Original: Cross entropy loss is used as the loss function for PGD attack, FPA,
Revised: Cross entropy loss is used as the loss function for PGD attack, FPA,

Original: GMSA-AVG and GMSA-MIN:
Revised: GMSA-AVG and GMSA-MIN:

Original: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y
Revised: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y

Original: ) ∈ V − log f
Revised: ) ∈ V − log f

Original: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .
Revised: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .

Original: We use PGD with a step size of 0 .
Revised: We use PGD with a step size of 0 .

Original: 01 , the number of steps of 200 , random start and no restarts.
Revised: 01 , the number of steps of 200 , random start and no restarts.

Original: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.
Revised: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.

Original: A.5. CIFAR- 
Revised: A.5. CIFAR-

Original: Model architecture and Training conﬁguration.
Revised: Model architecture and Training conﬁguration.

Original: We use the ResNet-20 network architecture (He et al., 2016).
Revised: We use the ResNet-20 network architecture (He et al., 2016).

Original: We train the models for 110 epochs using stochastic gradient decent (SGD) optimizer with Nesterov momentum and learning rate schedule.
Revised: We train the models for 110 epochs using stochastic gradient decent (SGD) optimizer with Nesterov momentum and learning rate schedule.

Original: We set momentum 0 .
Revised: We set momentum 0 .

Original: 9 and (cid:96) 2 weight decay with a coefﬁcient of 5 × 10 − 4 .
Revised: 9 and (cid:96) 2 weight decay with a coefﬁcient of 5 × 10 − 4 .

Original: The initial learning rate is 0 .
Revised: The initial learning rate is 0 .

Original: and it decreases by 0 .
Revised: and it decreases by 0 .

Original: 1 at 100 andepoch respectively.
Revised: 1 at 100 andepoch respectively.

Original: The batch size is 128 .
Revised: The batch size is 128 .

Original: We augment the training images using random crop and random horizontal ﬂip.
Revised: We augment the training images using random crop and random horizontal ﬂip.

Original: We use the L ∞ norm PGD attack as the adversary to generate adversarial training examples with a perturbation budget (cid:15) of 8 / 255 , a step size of 2 / 255 , and number of steps of 10 .
Revised: We use the L ∞ norm PGD attack as the adversary to generate adversarial training examples with a perturbation budget (cid:15) of 8 / 255 , a step size of 2 / 255 , and number of steps of 10 .

Original: We train on 50% clean and 50% adversarial examples per batch.
Revised: We train on 50% clean and 50% adversarial examples per batch.

Original: Attack conﬁguration.
Revised: Attack conﬁguration.

Original: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 8 / 255 .
Revised: We use the same threat model for all attacks: L ∞ norm perturbation with a perturbation budget (cid:15) of 8 / 255 .

Original: Cross entropy loss is used as the loss function for PGD attack, FPA,
Revised: Cross entropy loss is used as the loss function for PGD attack, FPA,

Original: GMSA-AVG and GMSA-MIN:
Revised: GMSA-AVG and GMSA-MIN:

Original: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y
Revised: L a ( F, V ) = 1 | V | (cid:80) ( xxx,y

Original: ) ∈ V − log f
Revised: ) ∈ V − log f

Original: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .
Revised: ( xxx ) y , where f ( xxx ) is the softmax output of the model F .

Original: We use PGD with a step size of 1 / 255 , the number of steps of 100 , random start and no restarts.
Revised: We use PGD with a step size of 1 / 255 , the number of steps of 100 , random start and no restarts.

Original: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.
Revised: We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.

Original: A. S ETUP FOR UR EJECTRON E XPERIMENTS
Revised: A. S ETUP FOR UR EJECTRON E XPERIMENTS

Original: We use a subset of the GTSRB augmented training data for our experiments, which has 10 classes and contains 10,000 images for each class.
Revised: We use a subset of the GTSRB augmented training data for our experiments, which has 10 classes and contains 10,000 images for each class.

Original: We implement URejectron (Goldwasser et al., 2020) on this dataset using the ResNet18 network (He et al., 2016) in the transductive setting.
Revised: We implement URejectron (Goldwasser et al., 2020) on this dataset using the ResNet18 network (He et al., 2016) in the transductive setting.

Original: Following Goldwasser et al.
Revised: Following Goldwasser et al.

Original: , we implement the basic form of the URejectron algorithm, with T = 1 iteration.
Revised: , we implement the basic form of the URejectron algorithm, with T = 1 iteration.

Original: That is we train a discriminator h to distinguish between examples from P and Q , and train a classiﬁer F on P .
Revised: That is we train a discriminator h to distinguish between examples from P and Q , and train a classiﬁer F on P .

Original: Speciﬁcally, we randomly split the data into a training set D train containing 63,000 images, a validation set D val containing 7,000 images and a test set D test containing 30,000 images.
Revised: Speciﬁcally, we randomly split the data into a training set D train containing 63,000 images, a validation set D val containing 7,000 images and a test set D test containing 30,000 images.

Original: We then use the training set D train to train a classiﬁer F using the ResNet18 network.
Revised: We then use the training set D train to train a classiﬁer F using the ResNet18 network.

Original: We train the classiﬁer F for 10 epochs using Adam optimizer with a batch size of 128 and a learning rate of 10 − 3 .
Revised: We train the classiﬁer F for 10 epochs using Adam optimizer with a batch size of 128 and a learning rate of 10 − 3 .

Original: The accuracy of the classiﬁer on the training set D train is 99.90% and its accuracy on the validation set D val is 99.63%.
Revised: The accuracy of the classiﬁer on the training set D train is 99.90% and its accuracy on the validation set D val is 99.63%.

Original: We construct a set ˜ x consisting of 50% normal examples and 50% adversarial examples.
Revised: We construct a set ˜ x consisting of 50% normal examples and 50% adversarial examples.

Original: The normal examples in the set ˜ x form a set z .
Revised: The normal examples in the set ˜ x form a set z .

Original: We train the discriminator h on the set D train (with label 0) and the set ˜ x
Revised: We train the discriminator h on the set D train (with label 0) and the set ˜ x

Original: (with label 1).
Revised: (with label 1).

Original: We then evaluate URejectron’s performance on ˜ x : under a certain threshold used by the discriminator h , we measure the fraction of normal examples in z that are rejected by the discriminator h and the error rate of the classiﬁer F on the examples in the set ˜ x that are accepted by the discriminator h .
Revised: We then evaluate URejectron’s performance on ˜ x : under a certain threshold used by the discriminator h , we measure the fraction of normal examples in z that are rejected by the discriminator h and the error rate of the classiﬁer F on the examples in the set ˜ x that are accepted by the discriminator h .

Original: The set z can be D test or a set of corrupted images generated on
Revised: The set z can be D test or a set of corrupted images generated on

Original: D test .
Revised: D test .

Original: We use the method proposed in Hendrycks & Dietterich (2019) to generate corrupted images with the corruption type of brightness and the severity level of 1.
Revised: We use the method proposed in Hendrycks & Dietterich (2019) to generate corrupted images with the corruption type of brightness and the severity level of 1.

Original: The accuracy of the classiﬁer on the corrupted images is 98.90%.
Revised: The accuracy of the classiﬁer on the corrupted images is 98.90%.

Original: The adversarial examples in ˜ x are generated by the PGD attack (Madry et al., 2018) or the CW attack (Carlini & Wagner, 2017).
Revised: The adversarial examples in ˜ x are generated by the PGD attack (Madry et al., 2018) or the CW attack (Carlini & Wagner, 2017).

Original: For PGD attack, we use L ∞ norm with perturbation budget (cid:15) = 8 / 255 and random initialization.
Revised: For PGD attack, we use L ∞ norm with perturbation budget (cid:15) = 8 / 255 and random initialization.

Original: The number of iterations is 40 and the step size is 1 / 255 .
Revised: The number of iterations is 40 and the step size is 1 / 255 .

Original: The robustness of the classiﬁer under the PGD attack is 3.66%.
Revised: The robustness of the classiﬁer under the PGD attack is 3.66%.

Original: For CW attack, we use
Revised: For CW attack, we use

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: L 2 norm as distance measure and set c = 1 and κ = 0 .
Revised: L 2 norm as distance measure and set c = 1 and κ = 0 .

Original: The learning rate is 0.01 and the number of steps is 100.
Revised: The learning rate is 0.01 and the number of steps is 100.

Original: The robustness of the classiﬁer under the CW attack is 0.00%.
Revised: The robustness of the classiﬁer under the CW attack is 0.00%.

Original: A. E VALUATE DENT UNDER THE ADVERSARIALLY - ORDERED GAME
Revised: A. E VALUATE DENT UNDER THE ADVERSARIALLY - ORDERED GAME

Original: We evaluate the robustness of DENT under the adversarially-ordered game where the adversary can choose a "worst-case" order of perturbed data points after receiving a large amount of test data and then sends them in batches one at a time to the defender.
Revised: We evaluate the robustness of DENT under the adversarially-ordered game where the adversary can choose a "worst-case" order of perturbed data points after receiving a large amount of test data and then sends them in batches one at a time to the defender.

Original: Speciﬁcally, each time the attacker will generate adversarial examples on up to 256 data points, and then sort the adversarial examples by their labels from lowest to highest, and ﬁnally send the sorted adversarial examples in batches one at a time to the defender.
Revised: Speciﬁcally, each time the attacker will generate adversarial examples on up to 256 data points, and then sort the adversarial examples by their labels from lowest to highest, and ﬁnally send the sorted adversarial examples in batches one at a time to the defender.

Original: Other experimental settings are the same as those described in Appendix A.3.
Revised: Other experimental settings are the same as those described in Appendix A.3.

Original: The results in Table 5 show that under the adversarially-ordered game, we can reduce the robustness of DENT to be lower than that of static base models.
Revised: The results in Table 5 show that under the adversarially-ordered game, we can reduce the robustness of DENT to be lower than that of static base models.

Revised: A. M ULTIPLE R ANDOM R UNS OF THE RMC E XPERIMENT

Revised: In Section 6, we describe the experimental setup for evaluating RMC.

Revised: The performance of RMC is evaluated on a sequence of test points xxx (1) , · · · ,xxx ( n ) randomly sampled from the test dataset.

Revised: We repeat this experiment ﬁve times with different random seeds and report the mean and standard deviation of the results over the multiple random runs of the experiment.

Revised: When evaluating the robustness of RMC, we only use the GMSA-AVG attack and the GMSA-MIN attack since they are the strongest attacks.

Revised: From Table 6, we can see that the results don’t vary much across different random runs and the conclusion that the proposed GMSA attacks can break RMC still holds.