Original: T OWARDS A U NIFIED M IN -M AX F RAMEWORK FOR A DVERSARIAL E XPLORATION AND R OBUSTNESS
Revised: T OWARDS A U NIFIED M IN -M AX F RAMEWORK FOR A DVERSARIAL E XPLORATION AND R OBUSTNESS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The worst-case training principle that minimizes the maximal adversarial loss, also known as adversarial training (AT), has shown to be a state-of-the-art approach for enhancing adversarial robustness against norm-ball bounded input perturbations.
Revised: The worst-case training principle that minimizes the maximal adversarial loss, also known as adversarial training (AT), has shown to be a state-of-the-art approach for enhancing adversarial robustness against norm-ball bounded input perturbations.

Original: Nonetheless, min-max optimization beyond the purpose of AT has not been rigorously explored in the research of adversarial attack and defense.
Revised: Nonetheless, min-max optimization beyond the purpose of AT has not been rigorously explored in the research of adversarial attack and defense.

Original: In particular, given a set of risk sources (domains), minimizing the maximal loss induced from the domain set can be reformulated as a general min-max problem that is  different from AT.
Revised: In particular, given a set of risk sources (domains), minimizing the maximal loss induced from the domain set can be reformulated as a general min-max problem that is fundamentally different from AT since the maximization is taken over the probability simplex of the domain set.

Original: Examples of this general formulation include attacking model ensembles, devising universal perturbation under multiple inputs or data transformations and generalized AT over different types of attack models.
Revised: Examples of this general formulation include attacking model ensembles, devising universal perturbation under multiple inputs or data transformations, and generalized AT over different types of attack models.

Original: We show that these problems can be solved under a uniﬁed and theoretically principled min-max optimization framework.
Revised: We show that these problems can be solved under a uniﬁed and theoretically principled min-max optimization framework.

Original: We also show that the self-adjusted domain weights learnt from our method provides a means to explain the difﬁculty level of attack and defense over multiple domains.
Revised: We also show that the self-adjusted domain weights learnt from our method provide a holistic tool to explain the difﬁculty level of attack and defense over multiple domains.

Original: Extensive experiments show that our approach leads to substantial performance improvement over the conventional averaging strategy 1 .
Revised: Extensive experiments show that our approach leads to substantial performance improvement over the conventional heuristic strategies 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training a machine learning model that is capable of assuring its worst-case performance against all possible adversaries given a speciﬁed threat model is a fundamental yet challenging problem, especially for deep neural networks (DNNs) (Szegedy et al., 2013; Goodfellow et al., 2015; Carlini & Wagner, 2017).
Revised: Training a machine learning model that is capable of assuring its worst-case performance against all possible adversaries given a speciﬁed threat model is a fundamental yet challenging problem, especially for deep neural networks (DNNs) (Szegedy et al., 2013; Goodfellow et al., 2015; Carlini & Wagner, 2017).

Original: A common practice to train an adversarially robust model is based on a speciﬁc form of min-max training, known as adversarial training (AT) (Goodfellow et al., 2015; Madry et al., 2017), where the minimization step learns model weights under the adversarial loss constructed at the maximization step in an alternative training fashion.
Revised: A common practice to train an adversarially robust model is based on a speciﬁc form of min-max training, known as adversarial training (AT) (Goodfellow et al., 2015; Madry et al., 2017), where the minimization step learns model weights under the adversarial loss constructed at the maximization step in an alternative training fashion.

Original: On datasets such as MNIST and CIFAR-10,
Revised: On datasets such as MNIST and CIFAR-10,

Original: AT has achieved the state-of-the-art defense performance against (cid:96) p -norm-ball input perturbations (Athalye et al., 2018b).
Revised: AT has achieved the state-of-the-art defense performance against (cid:96) p -norm-ball input perturbations (Athalye et al., 2018b).

Original: Motivated by the success of AT, one follow-up question that naturally arises is: Beyond AT, can other types of min-max formulation and optimization techniques advance the research in adversarial robustness?
Revised: Motivated by the success of AT, one follow-up question that naturally arises is: Beyond AT, can other types of min-max formulation and optimization techniques advance the research in adversarial robustness?

Original: In this paper, we give an afﬁrmative answer corroborated by the substantial performance gain and the ability of self-learned risk interpretation using our proposed min-max framework on several tasks for adversarial attack and defense.
Revised: In this paper, we give an afﬁrmative answer corroborated by the substantial performance gain and the ability of self-learned risk interpretation using our proposed min-max framework on several tasks for adversarial attack and defense.

Original: We demonstrate the utility of a general formulation for min-max optimization minimizing the maximal loss induced from a set of risk sources (domains).
Revised: We demonstrate the utility of a general formulation for min-max optimization minimizing the maximal loss induced from a set of risk sources (domains).

Original: Our considered min-max formulation is fundamentally different from AT, as our maximization step in taken over the probability simplex of the set of domains.
Revised: Our considered min-max formulation is fundamentally different from AT, as our maximization step is taken over the probability simplex of the set of domains.

Original: Moreover, we show that many problem setups in adversarial attacks and defenses can in fact be reformulated under this general min-max framework, including attacking model ensembles (Tramèr et al., 2018; Liu et al., 2018), devising universal perturbation to input samples (Moosavi-Dezfooli et al., 2017) or data transformations (Athalye & Sutskever, 2018; Brown et al., 2017), and generalized AT over multiple types of threat models (Tramèr & Boneh, 2019; Araujo et al., 2019).
Revised: Moreover, we show that many problem setups in adversarial attacks and defenses can in fact be reformulated under this general min-max framework, including attacking model ensembles (Tramèr et al., 2018; Liu et al., 2018), devising universal perturbation to input samples (Moosavi-Dezfooli et al., 2017) or data transformations (Athalye & Sutskever, 2018; Brown et al., 2017), and generalized AT over multiple types of threat models (Tramèr & Boneh, 2019; Araujo et al., 2019).

Original: However, current methods for solving these tasks often rely on simple heuristics (e.g., uniform averaging), resulting in signiﬁcant performance drops when comparing to our proposed min-max optimization framework.
Revised: However, current methods for solving these tasks often rely on simple heuristics (e.g., uniform averaging), resulting in signiﬁcant performance drops when comparing to our proposed min-max optimization framework.

Original: Speciﬁcally, based on the general min-max framework, we show that these problems can be solved under the same optimization procedure and prove the rate of its algorithmic convergence.
Revised: Speciﬁcally, based on the general min-max framework, we show that these problems can be solved under the same optimization procedure and prove the rate of its algorithmic convergence.

Original: As a byproduct and an exclusive feature, by tracking the weighting factor associated with the probability simplex during training, our method can provide tools for self-adjusted risk assessment and obtain novel insights on the set of domains for the associated tasks.
Revised: As a byproduct and an exclusive feature, by tracking the weighting factor associated with the probability simplex during training, our method can provide tools for self-adjusted risk assessment and obtain novel insights on the set of domains for the associated tasks.

Original: Contributions (1) We indicate the utility of min-max optimization beyond AT by proposing a general and theoretically grounded framework on adversarial attack and defense.
Revised: Contributions (1) We indicate the utility of min-max optimization beyond AT by proposing a general and theoretically grounded framework on adversarial attack and defense.

Original: (2) We demonstrate the effectiveness of our min-max framework by evaluating the proposed APGD attack on MNIST and CIFAR-10.
Revised: (2) We demonstrate the effectiveness of our min-max framework by evaluating the proposed APGD attack on MNIST and CIFAR-10.

Original: In theory, we show that APGD has an O (1 /T ) convergence rate, where T is  number of iterations.
Revised: In theory, we show that APGD has an O (1 /T ) convergence rate, where T is the number of iterations.

Original: In practice, we show that APGD obtains 17.48%, 35.21% and 9.39% improvement on average compared with PGD attack on CIFAR-10.
Revised: In practice, we show that APGD obtains 17.48%, 35.21% and 9.39% improvement on average compared with PGD attack on CIFAR-10.

Original: (3) We propose a generalized AT scheme under mixed types of adversarial attacks, and demonstrate that the diversiﬁed attack ensemble helps adversarial robustness.
Revised: (3) We propose a generalized AT scheme under mixed types of adversarial attacks and demonstrate that the diversiﬁed attack ensemble helps adversarial robustness.

Original: Compared with vanilla AT, our new training scheme leads to better worst-case robustness even if the defender lacks prior knowledge of the strengths of attacks.
Revised: Compared with vanilla AT, our new training scheme leads to better worst-case robustness even if the defender lacks prior knowledge of the strengths of attacks.

Original: (4) We show how the weighting factors of the probability simplex help to obtain novel insights for associated tasks and interpreting the importance of candidates in domains.
Revised: (4) We show how the weighting factors of the probability simplex help to obtain novel insights for associated tasks and interpreting the importance of candidates in domains.

Original: Related Work Recent studies have identiﬁed that DNNs are highly vulnerable to adversarial manipulations in various applications (Szegedy et al., 2013;
Revised: Related Work Recent studies have identiﬁed that DNNs are highly vulnerable to adversarial manipulations in various applications (Szegedy et al., 2013;

Original: Carlini et al., 2016; Jia & Liang, 2017;
Revised: Carlini et al., 2016; Jia & Liang, 2017;

Original: Lin et al., 2017; Huang et al., 2017; Carlini & Wagner, 2018; Zhao et al., 2018; Eykholt et al., 2018; Chen et al., 2018a;c; Lei et al., 2019), thus leading to an arms race between adversarial attacks (Carlini &Wagner, 2017; Athalye et al., 2018b; Goodfellow et al., 2014; Papernot et al., 2016a;
Revised: Lin et al., 2017; Huang et al., 2017; Carlini & Wagner, 2018; Zhao et al., 2018; Eykholt et al., 2018; Chen et al., 2018a; Lei et al., 2019), thus leading to an arms race between adversarial attacks (Carlini &

Original:  Moosavi Dezfooli et al., 2016; Chen et al., 2018b; 2017) and defenses (Madry et al., 2017; Papernot et al., 2016b; Meng & Chen, 2017; Xie et al., 2017; Xu et al., 2018).
Revised: Wagner, 2017; Athalye et al., 2018b; Goodfellow et al., 2014; Papernot et al., 2016a; Moosavi Dezfooli et al., 2016; Chen et al., 2018b; Xu et al., 2019) and defenses (Madry et al., 2017; Papernot et al., 2016b; Meng & Chen, 2017; Xie et al., 2017; Xu et al., 2018).

Original: One intriguing property of adversarial examples is the transferability across multiple domains (Liu et al., 2017; Tramèr et al., 2017; Papernot et al., 2017;
Revised: One intriguing property of adversarial examples is the transferability across multiple domains (Liu et al., 2017; Tramèr et al., 2017; Papernot et al., 2017;

Original: Su et al., 2018), which indicates a more challenging yet promising research direction – devising universal adversarial perturbations over model ensembles (Tramèr et al., 2018;
Revised: Su et al., 2018), which indicates a more challenging yet promising research direction – devising universal adversarial perturbations over model ensembles (Tramèr et al., 2018;

Original: Liu et al., 2018), input samples (Moosavi-Dezfooli et al., 2017; Metzen et al., 2017) and data transformations (Athalye et al., 2018b; Athalye & Sutskever, 2018; Brown et al., 2017).
Revised: Liu et al., 2018), input samples (Moosavi-Dezfooli et al., 2017; Metzen et al., 2017; Shafahi et al., 2018) and data transformations (Athalye et al., 2018b; Athalye & Sutskever, 2018; Brown et al., 2017).

Original: However, current approaches suffer from a signiﬁcant performance loss for resting on the uniform averaging strategy.
Revised: However, current approaches suffer from a signiﬁcant performance loss for resting on the uniform averaging strategy.

Original: We will compare these works with our min-max method in Sec. 4.
Revised: We will compare these works with our min-max method in Sec. 4.

Original: As a natural extension following min-max attack, we study the generalized AT under multiple perturbations (Tramèr & Boneh, 2019; Araujo et al., 2019; Kang et al., 2019; Croce & Hein, 2019).
Revised: As a natural extension following min-max attack, we study the generalized AT under multiple perturbations (Tramèr & Boneh, 2019; Araujo et al., 2019; Kang et al., 2019; Croce & Hein, 2019).

Original: Finally, our min-max framework is adapted and inspired by previous literature on robust learning over multiple domains (Qian et al., 2018; Raﬁque et al., 2018;
Revised: Finally, our min-max framework is adapted and inspired by previous literature on robust learning over multiple domains (Qian et al., 2018; Raﬁque et al., 2018;

Original: Lu et al., 2018; 2019a).
Revised: Lu et al., 2018; 2019a).

Original:  M IN -M AX P OWER IN A DVERSARIAL E XPLORATION AND R OBUSTNESS
Revised:  M IN -M AX P OWER IN A DVERSARIAL E XPLORATION AND R OBUSTNESS

Original: We begin by introducing the principle of robust learning over multiple domains and its connection to a specialized form of min-max optimization.
Revised: We begin by introducing the principle of robust learning over multiple domains and its connection to a specialized form of min-max optimization.

Original: We then show that the resulting min-max formulation ﬁts into various attack settings for adversarial exploration : a) ensemble adversarial attack, b) universal adversarial perturbation and c) robust perturbation over data transformations.
Revised: We then show that the resulting min-max formulation ﬁts into various attack settings for adversarial exploration : a) ensemble adversarial attack, b) universal adversarial perturbation and c) robust perturbation over data transformations.

Original: Finally, we propose a generalized adversarial training (AT) framework under mixed types of adversarial attacks to improve model robustness .
Revised: Finally, we propose a generalized adversarial training (AT) framework under mixed types of adversarial attacks to improve model robustness .

Original: 2. G ENERAL IDEA 
Revised: 2. G ENERAL IDEA : R OBUST LEARNING OVER MULTIPLE DOMAINS

Original: Consider K loss functions { F i ( v ) } (each of which is deﬁned on a learning domain), the problem of robust learning over K domains can be formulated as (Qian et al., 2018;
Revised: Consider K loss functions { F i ( v ) } (each of which is deﬁned on a learning domain), the problem of robust learning over K domains can be formulated as (Qian et al., 2018;

Original: Raﬁque et al., 2018;
Revised: Raﬁque et al., 2018;

Original: Lu et al., 2018)
Revised: Lu et al., 2018)

Original: [Equation]
Revised: [Equation]

Original: where v and w are optimization variables, V is a constraint set, and P denotes the probability simplex
Revised: where v and w are optimization variables, V is a constraint set, and P denotes the probability simplex

Original: P = { w | 1 T w = 1 , w i ∈ [0 , 1] , ∀ i } .
Revised: P = { w | 1 T w = 1 , w i ∈ [0 , 1] , ∀ i } .

Original: Since the inner maximization problem in (1) is a linear function of w over the probabilistic simplex, problem (1) is thus equivalent to
Revised: Since the inner maximization problem in (1) is a linear function of w over the probabilistic simplex, problem (1) is thus equivalent to

Original: [Equation]
Revised: [Equation]

Original: where [ K ] denotes the integer set { 1 , 2 , . .
Revised: where [ K ] denotes the integer set { 1 , 2 , . .

Original: . , K } .
Revised: . , K } .

Original: Beneﬁt and computation challenge of (1) Compared to multi-task learning in a ﬁnite-sum formulation which minimizes K losses on average , problem (1) provides consistently robust worst-case performance across all domains.
Revised: Beneﬁt and computation challenge of (1) Compared to multi-task learning in a ﬁnite-sum formulation which minimizes K losses on average , problem (1) provides consistently robust worst-case performance across all domains.

Original: This can be explained from the epigraph form of (2), minimize v ∈V ,t t, subject to F i ( v ) ≤ t, i ∈
Revised: This can be explained from the epigraph form of (2), minimize v ∈V ,t t, subject to F i ( v ) ≤ t, i ∈

Original: [ K ] , (3) where t is an epigraph variable (Boyd & Vandenberghe, 2004) that provides the t -level robustness at each domain.
Revised: [ K ] [Equation] (3) where t is an epigraph variable (Boyd & Vandenberghe, 2004) that provides the t -level robustness at each domain.

Original: Although the min-max problem (1) offers a great robustness interpretation as in (3), solving it becomes more challenging than solving the ﬁnite-sum problem.
Revised: Although the min-max problem (1) offers a great robustness interpretation as in (3), solving it becomes more challenging than solving the ﬁnite-sum problem.

Original: It is clear from (2) that the inner maximization problem of (1) always returns the one-hot value of w , namely, w = e i , where e i is the
Revised: It is clear from (2) that the inner maximization problem of (1) always returns the one-hot value of w , namely, w = e i , where e i is the

Original: i th standard basis vector, and i = arg max i { F i ( v ) } .
Revised: i th standard basis vector, and i = arg max i { F i ( v ) } .

Original: The one-hot coding reduces the generalizability to other domains and induces instability of the learning procedure in practice.
Revised: The one-hot coding reduces the generalizability to other domains and induces instability of the learning procedure in practice.

Original: Such an issue is often mitigated by introducing a strongly concave regularizer in the inner maximization step (Lu et al., 2018;
Revised: Such an issue is often mitigated by introducing a strongly concave regularizer in the inner maximization step (Lu et al., 2018;

Original: Qian et al., 2018).
Revised: Qian et al., 2018).

Original: Regularized problem formulation Spurred by (Qian et al., 2018)
Revised: Regularized problem formulation Spurred by (Qian et al., 2018)

Original: , we penalize the distance between the worst-case loss and the average loss over K domains.
Revised: , we penalize the distance between the worst-case loss and the average loss over K domains.

Original: This yields minimize v ∈V
Revised: This yields [Equation]

Original: [Equation]
Revised: [Equation]

Original: (4) where γ > 0 is a regularization parameter.
Revised: where γ > 0 is a regularization parameter.

Original: As γ → 0 , problem (4) is equivalent to (1).
Revised: As γ → 0 , problem (4) is equivalent to (1).

Original: By contrast, it becomes the ﬁnite-sum problem when γ → ∞ since w → 1 /K .
Revised: By contrast, it becomes the ﬁnite-sum problem when γ → ∞ since w → 1 /K .

Original: In this sense, the trainable w provides an essential indicator on the importance level of each domain.
Revised: In this sense, the trainable w provides an essential indicator on the importance level of each domain.

Original: The larger the weight is, the more important the domain is.
Revised: The larger the weight is, the more important the domain is.

Original: We call w domain weights in this paper.
Revised: We call w domain weights in this paper.

Original: We next show how the principle of robust learning over multiple domains can ﬁt into various settings of adversarial attack and defense problems.
Revised: We next show how the principle of robust learning over multiple domains can ﬁt into various settings of adversarial attack and defense problems.

Original: 2. R OBUST ADVERSARIAL ATTACKS
Revised: 2. R OBUST ADVERSARIAL ATTACKS

Original: The general goal of adversarial attack is to craft an adversarial example x (cid:48)
Revised: The general goal of adversarial attack is to craft an adversarial example x (cid:48)

Original: = x 0 + δ ∈ R d to mislead the prediction of machine learning (ML) or deep learning (DL) systems, where x 0 denotes the natural example with the true label t 0 , and δ is known as adversarial perturbation , commonly subject to (cid:96) p -norm ( p ∈ { 0 , 1 , 2 , ∞} ) constraint X :=
Revised: = x 0 + δ ∈ R d to mislead the prediction of machine learning (ML) or deep learning (DL) systems, where x 0 denotes the natural example with the true label t 0 , and δ is known as adversarial perturbation , commonly subject to (cid:96) p -norm ( p ∈ { 0 , 1 , 2 , ∞} ) constraint X :=

Original: { δ | (cid:107) δ (cid:107) p ≤ (cid:15), x 0 + δ ∈ [0 , 1] d } for a given small number (cid:15) .
Revised: { δ | (cid:107) δ (cid:107) p ≤ (cid:15), x 0 + δ ∈ [0 , 1] d } for a given small number (cid:15) .

Original: Here the (cid:96) p norm enforces the similarity between x (cid:48) and x 0 , and the input space of ML/DL systems is normalized to [0 , 1] d .
Revised: Here the (cid:96) p norm enforces the similarity between x (cid:48) and x 0 , and the input space of ML/DL systems is normalized to [0 , 1] d .

Original: Ensemble attack over multiple models Consider K ML/DL models {M i } Ki =1 , the goal is to ﬁnd robust adversarial examples that can fool all K models simultaneously.
Revised: Ensemble attack over multiple models Consider K ML/DL models {M i } Ki =1 , the goal is to ﬁnd robust adversarial examples that can fool all K models simultaneously.

Original: In this case, the notion of ‘domain’ in (4) is speciﬁed as ‘model’, and the objective function F i in (4) signiﬁes the attack loss f
Revised: In this case, the notion of ‘domain’ in (4) is speciﬁed as ‘model’, and the objective function F i in (4) signiﬁes the attack loss f

Original: ( δ ; x 0 , y 0 , M i ) given the natural input ( x 0 , y 0 ) and the model M i .
Revised: ( δ ; x 0 , y 0 , M i ) given the natural input ( x 0 , y 0 ) and the model M i .

Original: Thus, problem (4) becomes
Revised: Thus, problem (4) becomes

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where w encodes the difﬁculty level of attacking each model.
Revised: (5) where w encodes the difﬁculty level of attacking each model.

Original: Universal perturbation over multiple examples Consider K natural examples { ( x i , y i ) }
Revised: Universal perturbation over multiple examples Consider K natural examples { ( x i , y i ) }

Original: Ki =1 and a single model M , our goal is to ﬁnd the universal perturbation δ so that all the corrupted K examples can fool M .
Revised: Ki =1 and a single model M , our goal is to ﬁnd the universal perturbation δ so that all the corrupted K examples can fool M .

Original: In this case, the notion of ‘domain’ in (4) is speciﬁed as ‘example’, and problem (4) becomes
Revised: In this case, the notion of ‘domain’ in (4) is speciﬁed as ‘example’, and problem (4) becomes

Original: [Equation]
Revised: [Equation]

Original: where different from (5), w encodes the difﬁculty level of attacking each example.
Revised: where different from (5), w encodes the difﬁculty level of attacking each example.

Original: Adversarial attack over data transformations Consider K categories of data transformation { p i } , e.g., rotation, lightening, and translation (Athalye et al., 2018a), our goal is to ﬁnd the adversarial attack that is robust to data transformations.
Revised: Adversarial attack over data transformations Consider K categories of data transformation { p i } , e.g., rotation, lightening, and translation (Athalye et al., 2018a), our goal is to ﬁnd the adversarial attack that is robust to data transformations.

Original: In this case, the notion of ‘domain’ in (4) is speciﬁed as ‘data transformer’, and problem (4) becomes
Revised: In this case, the notion of ‘domain’ in (4) is speciﬁed as ‘data transformer’, and problem (4) becomes

Original: [Equation]
Revised: [Equation]

Original: where E t ∼ p i [ f ( t ( x 0 + δ ); y 0 , M )] denotes the attack loss under the distribution of data transformation p i , and w encodes the difﬁculty level of attacking each type of transformed example x 0 .
Revised: where E t ∼ p i [ f ( t ( x 0 + δ ); y 0 , M )] denotes the attack loss under the distribution of data transformation p i , and w encodes the difﬁculty level of attacking each type of transformed example x 0 .

Original: 2. A DVERSARIAL TRAINING (AT) UNDER MIXED TYPES OF ADVERSARIAL ATTACKS
Revised: 2. A DVERSARIAL TRAINING (AT) UNDER MIXED TYPES OF ADVERSARIAL ATTACKS

Original: Conventional AT is restricted to a single type of norm-ball constrained adversarial attack (Madry et al., 2017).
Revised: Conventional AT is restricted to a single type of norm-ball constrained adversarial attack (Madry et al., 2017).

Original: For example, AT under (cid:96) ∞ attack yields
Revised: For example, AT under (cid:96) ∞ attack yields

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where θ ∈ R n denotes model parameters, δ denotes (cid:15) -tolerant (cid:96) ∞ attack, and f tr ( θ , δ ; x , y ) is the training loss under perturbed examples { ( x + δ , y ) } .
Revised: where θ ∈ R n denotes model parameters, δ denotes (cid:15) -tolerant (cid:96) ∞ attack, and f tr ( θ , δ ; x , y ) is the training loss under perturbed examples { ( x + δ , y ) } .

Original: However, there possibly exist blind attacking spots across multiple types of adversarial attacks so that AT under one attack would not be strong enough against another attack (Araujo et al., 2019).
Revised: However, there possibly exist blind attacking spots across multiple types of adversarial attacks so that AT under one attack would not be strong enough against another attack (Araujo et al., 2019).

Original: Thus, an interesting question is how to generalize
Revised: Thus, an interesting question is how to generalize

Original: AT under multiple types of adversarial attacks.
Revised: AT under multiple types of adversarial attacks.

Original: One possible way is to use the ﬁnite-sum formulation
Revised: One possible way is to use the ﬁnite-sum formulation

Original: [Equation]
Revised: [Equation]

Original: where δ
Revised: where δ

Original: i ∈
Revised: i ∈

Original: X i is the i th type of adversarial perturbation deﬁned on X i , e.g., different (cid:96) p attacks.
Revised: X i is the i th type of adversarial perturbation deﬁned on X i , e.g., different (cid:96) p attacks.

Original: Moreover, one can map ‘attack type’ to ‘domain’ considered in (1).
Revised: Moreover, one can map ‘attack type’ to ‘domain’ considered in (1).

Original: We then perform AT against the strongest adversarial attack across K attack types in order to avoid blind attacking spots.
Revised: We then perform AT against the strongest adversarial attack across K attack types in order to avoid blind attacking spots.

Original: That is, upon deﬁning F i ( θ ) :
Revised: That is, upon deﬁning F i ( θ ) :

Original: = maximize δ i ∈X i f tr ( θ , δ i ; x , y ) , we solve the problem of the form (2),
Revised: = maximize δ i ∈X i f tr ( θ , δ i ; x , y ) , we solve the problem of the form (2),

Original: [Equation]
Revised: [Equation]

Original: In fact, problem (10) is in the min-max-max form, however, Lemma 1 shows that problem (10) can be further simpliﬁed to the min-max form.
Revised: In fact, problem (10) is in the min-max-max form, however, Lemma 1 shows that problem (10) can be further simpliﬁed to the min-max form.

Original: Lemma 1.
Revised: Lemma 1.

Original: Problem (10) is equivalent to
Revised: Problem (10) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where w ∈ R K represent domain weights, and P has been deﬁned in (1) .
Revised: where w ∈ R K represent domain weights, and P has been deﬁned in (1) .

Original: Proof : see Appendix A.
Revised: Proof : see Appendix A.

Original: Similar to (4), a strongly concave regularizer − γ/ 2 (cid:107) w − 1 /K (cid:107) 2 2 can be added into the inner maximization problem of (11), which can boost the stability of the learning procedure and strike a balance between the max and the average attack performance.
Revised: Similar to (4), a strongly concave regularizer − γ/ 2 (cid:107) w − 1 /K (cid:107) 2 2 can be added into the inner maximization problem of (11), which can boost the stability of the learning procedure and strike a balance between the max and the average attack performance.

Original: However, solving problem (11) and its regularized version is more complicated than (8) since the inner maximization involves both domain weights w and adversarial perturbations { δ i } .
Revised: However, solving problem (11) and its regularized version is more complicated than (8) since the inner maximization involves both domain weights w and adversarial perturbations { δ i } .

Original: We ﬁnally remark that there was an independent work (Tramèr & Boneh, 2019) which also proposed the formulation (10) for AT under multiple perturbations.
Revised: We ﬁnally remark that there was an independent work (Tramèr & Boneh, 2019) which also proposed the formulation (10) for AT under multiple perturbations.

Original: However, what we propose here is the regularized formulation of (11).
Revised: However, what we propose here is the regularized formulation of (11).

Original: As will be evident later, the domain weights w in our formulation have strong interpretability, which learns the importance level of different attacks.
Revised: As will be evident later, the domain weights w in our formulation have strong interpretability, which learns the importance level of different attacks.

Original: Most signiﬁcantly, our work has different motivation from (Tramèr & Boneh, 2019), and our idea applies to not only AT but also attack generation in Sec. 2.2.
Revised: Most signiﬁcantly, our work has different motivation from (Tramèr & Boneh, 2019), and our idea applies to not only AT but also attack generation in Sec. 2.2.

Original:  P ROPOSED A LGORITHM AND T HEORY
Revised:  P ROPOSED A LGORITHM AND T HEORY

Original: In this section, we delve into technical details on how to efﬁciently solve problems of robust adversarial attacks given by the generic form (4) and problem (11) for generalized AT under mixed types of adversarial attacks.
Revised: In this section, we delve into technical details on how to efﬁciently solve problems of robust adversarial attacks given by the generic form (4) and problem (11) for generalized AT under mixed types of adversarial attacks.

Original: 3. A LTERNATING ONE - STEP PGD FOR ROBUST ADVERSARIAL ATTACK GENERATION
Revised: 3. A LTERNATING ONE - STEP PGD FOR ROBUST ADVERSARIAL ATTACK GENERATION

Original: 1: Input: given w (0) and δ (0) .
Revised: 1: Input: given w (0) and δ (0) .

Original: 2: for t = 1 , 2 , . . .
Revised: 2: for t = 1 , 2 , . . .

Original: , T do 3: outer min.
Revised: , T do 3: outer min.

Original: : ﬁxing w = w ( t − 1) , call PGD (13) to update δ ( t ) 4: inner max. : ﬁxing δ = δ ( t ) , update w ( t ) via (14) 5: end for
Revised: : ﬁxing w = w ( t − 1) , call PGD (13) to update δ ( t ) 4: inner max. : ﬁxing δ = δ ( t ) , update w ( t ) via (14) 5: end for

Original: We propose the a lternating one-step p rojected g radient d escent (APGD) method (Algorithm 1) to solve problem (4).
Revised: We propose the a lternating one-step p rojected g radient d escent (APGD) method (Algorithm 1) to solve problem (4).

Original: For clarity, we repeat prob-
Revised: For clarity, we repeat prob-

Original: lem (4) under the adversarial perturbation δ and its constraint set X deﬁned in Sec. 2.2, minimize
Revised: lem (4) under the adversarial perturbation δ and its constraint set X deﬁned in Sec. 2.2, minimize

Original: [Equation]
Revised: [Equation]

Original: We show that at each iteration, APGD takes only one-step PGD for outer minimization and one-step projected gradient ascent for inner maximization (namely, PGD for its negative objective function).
Revised: We show that at each iteration, APGD takes only one-step PGD for outer minimization and one-step projected gradient ascent for inner maximization (namely, PGD for its negative objective function).

Original: We also show that each alternating step has a closed-form expression, and the main computational complexity stems from computing the gradient of the attack loss w.r.t.
Revised: We also show that each alternating step has a closed-form expression, and the main computational complexity stems from computing the gradient of the attack loss w.r.t.

Original: the input.
Revised: the input.

Original: Therefore, APGD is computationally efﬁcient like PGD, which is commonly used for design of conventional single (cid:96) p -norm
Revised: Therefore, APGD is computationally efﬁcient like PGD, which is commonly used for design of conventional single (cid:96) p -norm

Original: based adversarial attacks (Madry et al., 2017).
Revised: based adversarial attacks (Madry et al., 2017).

Original: Outer minimization Considering w = w ( t − 1) and F ( δ ) :
Revised: Outer minimization Considering w = w ( t − 1) and F ( δ ) :

Original: = (cid:80)
Revised: = (cid:80)

Original: Ki =1 w ( t − 1)
Revised: Ki =1 w ( t − 1)

Original: i
Revised: i

Original: F i ( δ ) in (4), we perform one-step PGD to update δ at iteration t ,
Revised: F i ( δ ) in (4), we perform one-step PGD to update δ at iteration t ,

Original: [Equation]
Revised: [Equation]

Original: (13) where proj( · ) denotes the Euclidean projection operator, i.e., proj X ( a ) =
Revised: (13) where proj( · ) denotes the Euclidean projection operator, i.e., proj X ( a ) =

Original: arg min x ∈X (cid:107) x − a (cid:107) 22 at the point a , α > 0 is a given learning rate, and ∇ δ denotes the ﬁrst order gradient w.r.t.
Revised: arg min x ∈X (cid:107) x − a (cid:107) 22 at the point a , α > 0 is a given learning rate, and ∇ δ denotes the ﬁrst order gradient w.r.t.

Original: δ .
Revised: δ .

Original: In (13), the projection operation becomes the key to obtain the closed-form of the updating rule (13).
Revised: In (13), the projection operation becomes the key to obtain the closed-form of the updating rule (13).

Original: Recall from Sec.
Revised: Recall from Sec.

Original: 2.2 that X = { δ |(cid:107) δ (cid:107) p ≤ (cid:15), ˇ c ≤ δ ≤ ˆ c } , where p ∈ { 0 , 1 , 2 , ∞} , and ˇ c = − xand ˆ c = 1 − x 0 (implying ˇ c ≤ 0 ≤ ˆ c ).
Revised: 2.2 that X = { δ |(cid:107) δ (cid:107) p ≤ (cid:15), ˇ c ≤ δ ≤ ˆ c } , where p ∈ { 0 , 1 , 2 , ∞} , and ˇ c = − xand ˆ c = 1 − x 0 (implying ˇ c ≤ 0 ≤ ˆ c ).

Original: If p = ∞ , then the projection function becomes the clip function.
Revised: If p = ∞ , then the projection function becomes the clip function.

Original: However, when p ∈ { 0 , 1 , 2 } , the closed-form of projection operation becomes non-trivial.
Revised: However, when p ∈ { 0 , 1 , 2 } , the closed-form of projection operation becomes non-trivial.

Original: In Proposition 1, we derive the solution of proj X ( a ) under different (cid:96) p norms.
Revised: In Proposition 1, we derive the solution of proj X ( a ) under different (cid:96) p norms.

Original: Proposition 1.
Revised: Proposition 1.

Original: Given a point a ∈ R d and a constraint set X = { δ |(cid:107) δ (cid:107) p ≤ (cid:15), ˇ c ≤ δ ≤ ˆ c } , the
Revised: Given a point a ∈ R d and a constraint set X = { δ |(cid:107) δ (cid:107) p ≤ (cid:15), ˇ c ≤ δ ≤ ˆ c } , the

Original: Euclidean projection δ ∗ =
Revised: Euclidean projection δ ∗ =

Original: proj X ( a ) has a closed-form solution when p ∈ { 0 , 1 , 2 } .
Revised: proj X ( a ) has a closed-form solution when p ∈ { 0 , 1 , 2 } .

Original: Proof : See Appendix B.
Revised: Proof : See Appendix B.

Original: (cid:3)
Revised: (cid:3)

Original: Inner maximization By ﬁxing δ = δ ( t ) and letting ψ ( w ) :
Revised: Inner maximization By ﬁxing δ = δ ( t ) and letting ψ ( w ) :

Original: = (cid:80)
Revised: = (cid:80)

Original: Ki
Revised: Ki

Original: =1 w i F i
Revised: =1 w i F i

Original: ( δ ( t ) ) − γ 2 (cid:107) w − 1 /K
Revised: ( δ ( t ) ) − γ 2 (cid:107) w − 1 /K

Original: (cid:107)in problem (4), we then perform one-step PGD (w.r.t.
Revised: (cid:107)in problem (4), we then perform one-step PGD (w.r.t.

Original: − ψ )
Revised: − ψ )

Original: to update w ,
Revised: to update w ,

Original: [Equation]
Revised: [Equation]

Original: (14) where β > 0 is a given learning rate, ∇ w ψ ( w )
Revised: (14) where β > 0 is a given learning rate, ∇ w ψ ( w )

Original: =
Revised: =

Original: φ ( t ) − γ ( w − 1 /K ) , and φ ( t ) :
Revised: φ ( t ) − γ ( w − 1 /K ) , and φ ( t ) :

Original: =
Revised: =

Original: [ F 1 ( δ ( t ) ) , . .
Revised: [ F 1 ( δ ( t ) ) , . .

Original: , F K ( δ ( t ) )] T .
Revised: , F K ( δ ( t ) )] T .

Original: In (14), the second equality holds due to the closed-form of projection operation onto the probabilistic simplex P (Parikh et al., 2014), where ( · ) + denotes the elementwise nonnegative operator, i.e., ( x ) + = max { 0 , x } , and µ is the root of the equation 1 T ( b − µ 1 ) + = 1 .
Revised: In (14), the second equality holds due to the closed-form of projection operation onto the probabilistic simplex P (Parikh et al., 2014), where ( · ) + denotes the elementwise nonnegative operator, i.e., ( x ) + = max { 0 , x } , and µ is the root of the equation 1 T ( b − µ 1 ) + = 1 .

Original: Since 1 T ( b − min i { b i } 1 + 1 /K )
Revised: Since 1 T ( b − min i { b i } 1 + 1 /K )

Original: + ≥ 1 T 1 /K
Revised: + ≥ 1 T 1 /K

Original: = 1 , and 1 T ( b − max i { b i } 1 + 1 /K )
Revised: = 1 , and 1 T ( b − max i { b i } 1 + 1 /K )

Original: + ≤ 1 T 1 /K
Revised: + ≤ 1 T 1 /K

Original: = 1 , the root µ exists within the interval [min i { b i } − 1 /K, max i { b i } − 1 /K ] and can be found via the bisection method (Boyd & Vandenberghe, 2004).
Revised: = 1 , the root µ exists within the interval [min i { b i } − 1 /K, max i { b i } − 1 /K ] and can be found via the bisection method (Boyd & Vandenberghe, 2004).

Original: Convergence analysis We remark that APGD follows the gradient primal-dual optimization framework (Lu et al., 2019a), and thus enjoys the same optimization guarantees.
Revised: Convergence analysis We remark that APGD follows the gradient primal-dual optimization framework (Lu et al., 2019a), and thus enjoys the same optimization guarantees.

Original: In Theorem 1, we demonstrate the convergence rate of Algorithm 1 for solving problem (4).
Revised: In Theorem 1, we demonstrate the convergence rate of Algorithm 1 for solving problem (4).

Original: Theorem 1.
Revised: Theorem 1.

Original: (inherited from primal-dual min-max optimization) Suppose that in problem (4) F i ( δ ) has L -Lipschitz continuous gradients, and X is a convex compact set.
Revised: (inherited from primal-dual min-max optimization) Suppose that in problem (4) F i ( δ ) has L -Lipschitz continuous gradients, and X is a convex compact set.

Original: Given learning rates α ≤ 1 L and β < 1 γ , then the sequence { δ ( t ) , w ( t ) }
Revised: Given learning rates α ≤ 1 L and β < 1 γ , then the sequence { δ ( t ) , w ( t ) }

Original: Tt =1 generated by Algorithm 1 converges to a ﬁrst-order stationary point 2 of problem (4) under the convergence rate
Revised: Tt =1 generated by Algorithm 1 converges to a ﬁrst-order stationary point 2 of problem (4) under the convergence rate

Original: O ( 1 T ) .
Revised: O ( 1 T ) .

Original: Proof : Note that the objective function of problem (4) is strongly concave w.r.t.
Revised: Proof : Note that the objective function of problem (4) is strongly concave w.r.t.

Original: w with parameter γ , and has γ -Lipschitz continuous gradients.
Revised: w with parameter γ , and has γ -Lipschitz continuous gradients.

Original: Moreover, we have (cid:107) w (cid:107) 2 ≤ 1 due to w ∈ P .
Revised: Moreover, we have (cid:107) w (cid:107) 2 ≤ 1 due to w ∈ P .

Original: Using these facts and (Lu et al., 2019a, Theorem 1) or (Lu et al., 2019b, Theorem 1) completes the proof.
Revised: Using these facts and (Lu et al., 2019a, Theorem 1) or (Lu et al., 2019b, Theorem 1) completes the proof.

Original: (cid:3)
Revised: (cid:3)

Original: A LTERNATING MULTI - STEP PGD FOR GENERALIZED AT
Revised: A LTERNATING MULTI - STEP PGD FOR GENERALIZED AT

Original: We next propose the a lternating m ulti-step p rojected g radient d escent (AMPGD) method to solve the regularized version of problem (11), which is repeated as follows minimize θ
Revised: We next propose the a lternating m ulti-step p rojected g radient d escent (AMPGD) method to solve the regularized version of problem (11), which is repeated as follows minimize θ

Original: [Equation]
Revised: [Equation]

Original: 1: Input: given θ (0) , w (0) , δ (0) and K > 0 .
Revised: 1: Input: given θ (0) , w (0) , δ (0) and K > 0 .

Original: 2: for t = 1 , 2 , . . .
Revised: 2: for t = 1 , 2 , . . .

Original: , T do 3: given w ( t − 1) and δ ( t − 1) , perform SGD to update θ ( t ) (Madry et al., 2017) 4: given θ ( t ) , perform R -step PGD to update w ( t ) and δ ( t ) 5: end for
Revised: , T do 3: given w ( t − 1) and δ ( t − 1) , perform SGD to update θ ( t ) (Madry et al., 2017) 4: given θ ( t ) , perform R -step PGD to update w ( t ) and δ ( t ) 5: end for

Original: Problem (15) is in a more general non-convex non-concave min-max setting, where the inner maximization involves both domain weights w and adversarial perturbations { δ i } .
Revised: Problem (15) is in a more general non-convex non-concave min-max setting, where the inner maximization involves both domain weights w and adversarial perturbations { δ i } .

Original: It was shown in (Nouiehed et al., 2019)
Revised: It was shown in (Nouiehed et al., 2019)

Original: that the multi-step
Revised: that the multi-step

Original: PGD is required for inner maximization in order to approximate the near-optimal solution.
Revised: PGD is required for inner maximization in order to approximate the near-optimal solution.

Original: This is also in the similar spirit of AT (Madry et al., 2017), which executed multi-step PGD attack during inner maximization.
Revised: This is also in the similar spirit of AT (Madry et al., 2017), which executed multi-step PGD attack during inner maximization.

Original: We summarize AMPGD in Algorithm 2.
Revised: We summarize AMPGD in Algorithm 2.

Original: At step 4 of Algorithm 2, each PGD step to update w and δ can be decomposed as
Revised: At step 4 of Algorithm 2, each PGD step to update w and δ can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: where let w ( t ) 1 := w ( t − 1) and δ ( t ) i, 1 :
Revised: where let w ( t ) 1 := w ( t − 1) and δ ( t ) i, 1 :

Original: =
Revised: =

Original: δ ( t − 1) i .
Revised: δ ( t − 1) i .

Original: Here the subscript t represents the iteration index of
Revised: Here the subscript t represents the iteration index of

Original: AMPGD, and the subscript r denotes the iteration index of R -step PGD.
Revised: AMPGD, and the subscript r denotes the iteration index of R -step PGD.

Original: Clearly, the above projection operations can be derived for closed-form expressions through (14) and Lemma 1.
Revised: Clearly, the above projection operations can be derived for closed-form expressions through (14) and Lemma 1.

Original: To the best of our knowledge, it is still an open question to build theoretical convergence guarantees for solving the general non-convex non-concave min-max problem like (15), except the work (Nouiehed et al., 2019) which proposed O (1 /T ) convergence rate if the objective function satisﬁes Polyak- Łojasiewicz conditions (Karimi et al., 2016).
Revised: To the best of our knowledge, it is still an open question to build theoretical convergence guarantees for solving the general non-convex non-concave min-max problem like (15), except the work (Nouiehed et al., 2019) which proposed O (1 /T ) convergence rate if the objective function satisﬁes Polyak- Łojasiewicz conditions (Karimi et al., 2016).

Original: Improved robustness via diversiﬁed (cid:96) p attacks.
Revised: Improved robustness via diversiﬁed (cid:96) p attacks.

Original: It was recently shown in (Kariyappa & Qureshi, 2019;
Revised: It was recently shown in (Kariyappa & Qureshi, 2019;

Original: Pang et al., 2019) that the diversity of individual neural networks improves adversarial robustness of an ensemble model.
Revised: Pang et al., 2019) that the diversity of individual neural networks improves adversarial robustness of an ensemble model.

Original: Spurred by that, one may wonder if the promotion of diversity among (cid:96) p attacks is beneﬁcial to adversarial robustness?
Revised: Spurred by that, one may wonder if the promotion of diversity among (cid:96) p attacks is beneﬁcial to adversarial robustness?

Original: We measure the diversity between adversarial attacks through the similarity between perturbation directions, namely, input gradients {∇ δ i f tr ( θ , δ i ; x , y ) } i in (15).
Revised: We measure the diversity between adversarial attacks through the similarity between perturbation directions, namely, input gradients {∇ δ i f tr ( θ , δ i ; x , y ) } i in (15).

Original: We ﬁnd that there exists a strong correlation between input gradients for different (cid:96) p attacks.
Revised: We ﬁnd that there exists a strong correlation between input gradients for different (cid:96) p attacks.

Original: Thus, we propose to enhance their diversity through the orthogonality-promoting regularizer used for encouraging diversiﬁed prediction of ensemble models in (Pang et al., 2019),
Revised: Thus, we propose to enhance their diversity through the orthogonality-promoting regularizer used for encouraging diversiﬁed prediction of ensemble models in (Pang et al., 2019),

Original: [Equation]
Revised: [Equation]

Original: where G ∈ R d × K is a d × K matrix, each column of which corresponds to a normalized input gradient ∇ δ i f tr ( θ , δ i ; x , y ) for i ∈
Revised: where G ∈ R d × K is a d × K matrix, each column of which corresponds to a normalized input gradient ∇ δ i f tr ( θ , δ i ; x , y ) for i ∈

Original: [ K ] , and h ( θ , { δ i } ; x , y ) reaches the maximum value 0 as input gradients become orthogonal.
Revised: [ K ] , and h ( θ , { δ i } ; x , y ) reaches the maximum value 0 as input gradients become orthogonal.

Original: With the aid of (16), we modify problem (15) to minimize θ
Revised: With the aid of (16), we modify problem (15) to minimize θ

Original: [Equation]
Revised: [Equation]

Original: The rationale behind (17) is that the adversary aims to enhance the effectiveness of attacks from diversiﬁed perturbation directions (inner maximization), while the defender robustiﬁes the model θ , which makes diversiﬁed attacks less effective (outer minimization).
Revised: The rationale behind (17) is that the adversary aims to enhance the effectiveness of attacks from diversiﬁed perturbation directions (inner maximization), while the defender robustiﬁes the model θ , which makes diversiﬁed attacks less effective (outer minimization).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we ﬁrst evaluate the proposed min-max optimization strategy on three attack tasks.
Revised: In this section, we ﬁrst evaluate the proposed min-max optimization strategy on three attack tasks.

Original: We show that our approach leads to substantial improvement compared with state-of-the-art attack
Revised: We show that our approach leads to substantial improvement compared with state-of-the-art attack

Original: [Table]
Revised: [Table]

Original: methods such as ensemble PGD (Liu et al., 2018) and expectation over transformation (EOT) (Athalye et al., 2018b; Brown et al., 2017; Athalye et al., 2018a).
Revised: methods such as ensemble PGD (Liu et al., 2018) and expectation over transformation (EOT) (Athalye et al., 2018b; Brown et al., 2017; Athalye et al., 2018a).

Original: We next demonstrate the effectiveness of the generalized AT for multiple types of adversarial perturbations.
Revised: We next demonstrate the effectiveness of the generalized AT for multiple types of adversarial perturbations.

Original: We show that the use of trainable domain weights in problem (15) can automatically adjust the risk level of different attacks during the training process even if the defender lacks prior knowledge on the strength of these attacks.
Revised: We show that the use of trainable domain weights in problem (15) can automatically adjust the risk level of different attacks during the training process even if the defender lacks prior knowledge on the strength of these attacks.

Original: We also show that the promotion of diversity of (cid:96) p attacks help improve adversarial robustness further.
Revised: We also show that the promotion of diversity of (cid:96) p attacks help improve adversarial robustness further.

Original: We thoroughly evaluate our APGD/AMPGD algorithm on MNIST and CIFAR-10.
Revised: We thoroughly evaluate our APGD/AMPGD algorithm on MNIST and CIFAR-10.

Original: A set of diverse image classiﬁers (denoted from Model A to Model H) are trained, including multi-layer perceptrons (MLP), All-CNNs (Springenberg et al., 2015), LeNet (Lecun et al., 1998), LeNetV2,
Revised: A set of diverse image classiﬁers (denoted from Model A to Model H) are trained, including multi-layer perceptrons (MLP), All-CNNs (Springenberg et al., 2015), LeNet (Lecun et al., 1998), LeNetV2,

Original: VGG16 (Simonyan & Zisserman, 2015), ResNet50 (He et al., 2016), Wide-ResNet (Madry et al., 2017; Zagoruyko & Komodakis, 2016) and GoogLeNet (Szegedy et al., 2015).
Revised: VGG16 (Simonyan & Zisserman, 2015), ResNet50 (He et al., 2016), Wide-ResNet (Madry et al., 2017; Zagoruyko & Komodakis, 2016) and GoogLeNet (Szegedy et al., 2015).

Original: More details about model architectures and training process are provided in Appendix C.1.
Revised: More details about model architectures and training process are provided in Appendix C.1.

Original: 4. R OBUST ADVERSARIAL ATTACKS
Revised: 4. R OBUST ADVERSARIAL ATTACKS

Original: Most current works play a min-max game from a defender’s perspective, i.e., adversarial training.
Revised: Most current works play a min-max game from a defender’s perspective, i.e., adversarial training.

Original: However, we show the great strength of min-max optimization also lies at the side of attack generation.
Revised: However, we show the great strength of min-max optimization also lies at the side of attack generation.

Original: Note that problem formulations (5)-(7) are applicable to both untargeted and targeted attack.
Revised: Note that problem formulations (5)-(7) are applicable to both untargeted and targeted attack.

Original: Here we focus on the former setting and use C&W loss function (Carlini & Wagner, 2017; Madry et al., 2017).
Revised: Here we focus on the former setting and use C&W loss function (Carlini & Wagner, 2017; Madry et al., 2017).

Original: The details of crafting adversarial examples are available in Appendix C.2.
Revised: The details of crafting adversarial examples are available in Appendix C.2.

Original: Ensemble attack over multiple models We craft adversarial examples against an ensemble of known classiﬁers.
Revised: Ensemble attack over multiple models We craft adversarial examples against an ensemble of known classiﬁers.

Original: The work (Liu et al., 2018, 5th place at CAAD-18) proposed an ensemble PGD attack, which assumed equal importance among different models, namely, w i = 1 /K in problem (1).
Revised: The work (Liu et al., 2018, 5th place at CAAD-18) proposed an ensemble PGD attack, which assumed equal importance among different models, namely, w i = 1 /K in problem (1).

Original: Throughout this task, we measure the attack performance via ASR all - the attack success rate (ASR) of fooling model ensembles simultaneously.
Revised: Throughout this task, we measure the attack performance via ASR all - the attack success rate (ASR) of fooling model ensembles simultaneously.

Original: Compared to the ensemble PGD attack (Liu et al., 2018), our approach results in 40.79% and 17.48% ASR all improvement averaged over different (cid:96) p -norm constraints on MNIST and CIFAR-10 , respectively.
Revised: Compared to the ensemble PGD attack (Liu et al., 2018), our approach results in 40.79% and 17.48% ASR all improvement averaged over different (cid:96) p -norm constraints on MNIST and CIFAR-10 , respectively.

Original: In what follows, we provide more detailed experiment results and analysis.
Revised: In what follows, we provide more detailed experiment results and analysis.

Original: In Table 1, we show that our min-max APGD signiﬁcantly outperforms ensemble PGD in ASR all .
Revised: In Table 1, we show that our min-max APGD signiﬁcantly outperforms ensemble PGD in ASR all .

Original: Taking (cid:96) ∞ -attack  as an example, our min-max attack leads to a 90.16% ASR all , which largely outperforms 48.17% (ensemble PGD).
Revised: Taking (cid:96) ∞ -attack on MNIST as an example, our min-max attack leads to a 90.16% ASR all , which largely outperforms 48.17% (ensemble PGD).

Original: The reason is that Model C, D are more difﬁcult to attack, which can be observed from their higher test accuracy on adversarial examples.
Revised: The reason is that Model C, D are more difﬁcult to attack, which can be observed from their higher test accuracy on adversarial examples.

Original: As a result, although the adversarial examples crafted by assigning equal weights over multiple models are able to attack {A, B} well, they achieve a much lower ASR (i.e., 1 - Acc) in {C, D}.
Revised: As a result, although the adversarial examples crafted by assigning equal weights over multiple models are able to attack {A, B} well, they achieve a much lower ASR (i.e., 1 - Acc) in {C, D}.

Original: By contrast, APGD automatically handles the worst case {C, D} by slightly sacriﬁcing the performance on {A, B}: 31.47% averaged ASR improvement on {C, D} versus 0.86% degradation on {A, B}.
Revised: By contrast, APGD automatically handles the worst case {C, D} by slightly sacriﬁcing the performance on {A, B}: 31.47% averaged ASR improvement on {C, D} versus 0.86% degradation on {A, B}.

Original: More results on CIFAR-10 and advanced DNNs (e.g., GoogLeNet) are provided in Appendix D.1.
Revised: More results on CIFAR-10 and more complicated DNNs (e.g., GoogLeNet) are provided in Appendix D.

Original: Figure 1 depicts the ASR of four models under average/min-max attacks as well as the distribution of domain weights during attack generation.
Revised: Figure 1 depicts the ASR of four models under average/min-max attacks as well as the distribution of domain weights during attack generation.

Original: For ensemble PGD (Figure 1a), Model C and D are attacked insufﬁciently, leading to relatively low ASR and thus weak ensemble performance.
Revised: For ensemble PGD (Figure 1a), Model C and D are attacked insufﬁciently, leading to relatively low ASR and thus weak ensemble performance.

Original: By contrast, APGD (Figure 1b) will encode the difﬁculty level to attack different models based on the current attack loss.
Revised: By contrast, APGD (Figure 1b) will encode the difﬁculty level to attack different models based on the current attack loss.

Original: It dynamically adjusts the weight w i as shown in Figure 1c.
Revised: It dynamically adjusts the weight w i as shown in Figure 1c.

Original: For instance, the[Figure] [Figure] weight for Model D is ﬁrst raised to 0 .
Revised: For instance, the weight for Model D is ﬁrst raised to 0 .

Original: 45 because D is difﬁcult to attack initially.
Revised: 45 because D is difﬁcult to attack initially.

Original: Then it decreases to 0 .
Revised: Then it decreases to [Figure]

Original:  3 once Model D encounters the sufﬁcient attack power and the corresponding attack performance is no longer improved.
Revised: [Figure] 3 once Model D encounters the sufﬁcient attack power and the corresponding attack performance is no longer improved.

Original: It is worth noticing that APGD is highly efﬁcient because w i converges after a small number of iterations.
Revised: It is worth noticing that APGD is highly efﬁcient because w i converges after a small number of iterations.

Original: To perform a boarder evaluation, we repeat the above experiments ( (cid:96) ∞ norm) under different (cid:15) in Figure 2.
Revised: To perform a boarder evaluation, we repeat the above experiments ( (cid:96) ∞ norm) under different (cid:15) in Figure 2.

Original: The ASR of min-max strategy is consistently better or on part with the average strategy.
Revised: The ASR of min-max strategy is consistently better or on part with the average strategy.

Original: Moreover, APGD achieves more signiﬁcant improvement when moderate (cid:15) is chosen: MNIST ( (cid:15) ∈ [0 . 15 , 0 . 25] ) and CIFAR-10 ( (cid:15) ∈ [0 . 03 , 0 . 05] ).
Revised: Moreover, APGD achieves more signiﬁcant improvement when moderate (cid:15) is chosen: MNIST ( (cid:15) ∈ [0 . 15 , 0 . 25] ) and CIFAR-10 ( (cid:15) ∈ [0 . 03 , 0 . 05] ).

Original: Lastly, we highlight that tracking domain weights w provides us novel insights for model robustness and understanding attack procedure.
Revised: Lastly, we highlight that tracking domain weights w provides us novel insights for model robustness and understanding attack procedure.

Original: From our theory, a model with higher robustness always corresponds to a larger w because its loss is hard to attack and becomes the “worst” term.
Revised: From our theory, a model with higher robustness always corresponds to a larger w because its loss is hard to attack and becomes the “worst” term.

Original: This hypothesis can be veriﬁed empirically.
Revised: This hypothesis can be veriﬁed empirically.

Original: According to Figure 1c, we have w c > w d > w a > w b – indicating a decrease in model robustness for C, D, A and B, which is exactly veriﬁed by Acc C >
Revised: According to Figure 1c, we have w c > w d > w a > w b – indicating a decrease in model robustness for C, D, A and B, which is exactly veriﬁed by Acc C >

Original: Acc D > Acc A > Acc B in Table 1 ( (cid:96) ∞ -norm).
Revised: Acc D > Acc A > Acc B in Table 1 ( (cid:96) ∞ -norm).

Original: Universal perturbation over multiple examples We evaluate APGD in universal perturbation on MNIST and CIFAR-10, where 10,000 test images are randomly divided into equal-size groups (containing K images per group) for universal perturbation.
Revised: Universal perturbation over multiple examples We evaluate APGD in universal perturbation on MNIST and CIFAR-10, where 10,000 test images are randomly divided into equal-size groups (containing K images per group) for universal perturbation.

Original: We measure two types of ASR (%),
Revised: We measure two types of ASR (%),

Original: ASR avg and ASR gp .
Revised: ASR avg and ASR gp .

Original: Here the former represents the ASR averaged over all images in all groups, and the latter signiﬁes the ASR averaged over all groups but a successful attack is counted under a more restricted condition: images within each group must be successfully attacked simultaneously by universal perturbation.
Revised: Here the former represents the ASR averaged over all images in all groups, and the latter signiﬁes the ASR averaged over all groups but a successful attack is counted under a more restricted condition: images within each group must be successfully attacked simultaneously by universal perturbation.

Original: When K = 5 , our approach achieves 42.63% and 35.21% improvement over the averaging strategy under MNIST and CIFAR-10, respectively.
Revised: When K = 5 , our approach achieves 42.63% and 35.21% improvement over the averaging strategy under MNIST and CIFAR-10, respectively.

Original: In Table 2, we compare the proposed min-max strategy (APGD) with the averaging strategy on the attack performance of generated universal perturbations.
Revised: In Table 2, we compare the proposed min-max strategy (APGD) with the averaging strategy on the attack performance of generated universal perturbations.

Original: As we can see, our method always achieves higher ASR gp for different values of K .
Revised: As we can see, our method always achieves higher ASR gp for different values of K .

Original: The universal perturbation generated from APGD can successfully attack ‘hard’ images (on which the average-based PGD attack fails) by self-adjusting domain weights, and thus leads to a higher ASR gp .
Revised: The universal perturbation generated from APGD can successfully attack ‘hard’ images (on which the average-based PGD attack fails) by self-adjusting domain weights, and thus leads to a higher ASR gp .

Original: Besides, the min-max universal perturbation also offers interpretability of “image robustness” by associating domain weights with image visualization; see FigureA8 and A9  (Appendix D.3) for an example in which the large domain weight corresponds to the MNIST letter with clear appearance (e.g., bold letter).
Revised: Besides, the min-max universal perturbation also offers interpretability of “image robustness” by associating domain weights with image visualization; see Figure A9 and A10 (Appendix F) for an example in which the large domain weight corresponds to the MNIST letter with clear appearance (e.g., bold letter).

Original: Robust adversarial attack over data transformations EOT (Athalye et al., 2018a) achieves stateof-the-art performance in producing adversarial examples robust to data transformations.
Revised: Robust adversarial attack over data transformations EOT (Athalye et al., 2018a) achieves stateof-the-art performance in producing adversarial examples robust to data transformations.

Original: From (7), 
Revised: From (7), we could derive EOT as a special case when the weights satisfy w i = 1 /K (average case). For each

Original: [Table] [Table]
Revised: [Table] [Table]

Original: we could derive EOT as a special case when the weights satisfy w i = 1 /K (average case).

Original: For each input sample ( ori ), we transform the image under a series of functions, e.g., ﬂipping horizontally ( ﬂh ) or vertically ( ﬂv ), adjusting brightness ( bri ), performing gamma correction ( gam ) and cropping ( crop ), and group each image with its transformed variants.
Revised: input sample ( ori ), we transform the image under a series of functions, e.g., ﬂipping horizontally ( ﬂh ) or vertically ( ﬂv ), adjusting brightness ( bri ), performing gamma correction ( gam ) and cropping ( crop ), and group each image with its transformed variants.

Original: Similar to universal perturbation, ASR avg and ASR gp are reported to measure the ASR over all transformed images and groups of transformed images (each group is successfully attacked signiﬁes successfully attacking an example under all transformers).
Revised: Similar to universal perturbation, ASR avg and ASR gp are reported to measure the ASR over all transformed images and groups of transformed images (each group is successfully attacked signiﬁes successfully attacking an example under all transformers).

Original: In Table 3, compared to EOT, our approach leads to 9.39% averaged lift in
Revised: In Table 3, compared to EOT, our approach leads to 9.39% averaged lift in

Original: ASR gp over given models on CIFAR-10 by optimizing the weights for various transformations.
Revised: ASR gp over given models on CIFAR-10 by optimizing the weights for various transformations.

Original: Due to limited space, we leave the details of transformers in Append C.3 and the results under randomness (e.g., ﬂipping images randomly w.p. 0.8; randomly clipping the images at speciﬁc range) in Appendix D.1.
Revised: Due to limited space, we leave the details of transformers in Append C.3 and the results under randomness (e.g., ﬂipping images randomly w.p. 0.8; randomly clipping the images at speciﬁc range) in Appendix D.

Original: 4. A DVERSARIAL TRAINING FOR MULTIPLE ADVERSARIAL PERTURBATIONS
Revised: 4. A DVERSARIAL TRAINING FOR MULTIPLE ADVERSARIAL PERTURBATIONS

Original: Compared to vanilla AT, we show the generalized AT scheme produces models robust to multiple types of perturbation, thus leads to stronger “overall robustness”.
Revised: Compared to vanilla AT, we show the generalized AT scheme produces models robust to multiple types of perturbation, thus leads to stronger “overall robustness”.

Original: We measure the training performance using two types of Acc (%): Acc maxadv and Acc avgadv , where Acc maxadv denotes the test accuracy over examples with the strongest perturbation ( (cid:96) ∞ or (cid:96) 2 ), and Acc avgadv denotes the averaged test accuracy over examples with all types of perturbations ( (cid:96) ∞ and (cid:96) 2 ).
Revised: We measure the training performance using two types of Acc (%): Acc maxadv and Acc avgadv , where Acc maxadv denotes the test accuracy over examples with the strongest perturbation ( (cid:96) ∞ or (cid:96) 2 ), and Acc avgadv denotes the averaged test accuracy over examples with all types of perturbations ( (cid:96) ∞ and (cid:96) 2 ).

Original: Moreover, we measure the overall worst-case robustness S (cid:15) in terms of the area under the curve ‘Acc max adv vs. (cid:15) ’
Revised: Moreover, we measure the overall worst-case robustness S (cid:15) in terms of the area under the curve ‘Acc maxadv vs. (cid:15) ’

Original: (see Figure 3b).
Revised: (see Figure 3b).

Original: In Table 4, we present the test accuracy of MLP in different training schemes: a) natural training, b) single-norm: vanilla AT ( (cid:96) ∞ or (cid:96) 2 ), c) multi-norm: generalized AT ( avg and min max ), and d) generalized AT with diversity-promoting attack regularization (DPAR, λ = 0 .
Revised: In Table 4, we present the test accuracy of MLP in different training schemes: a) natural training, b) single-norm: vanilla AT ( (cid:96) ∞ or (cid:96) 2 ), c) multi-norm: generalized AT ( avg and min max ), and d) generalized AT with diversity-promoting attack regularization (DPAR, λ = 0 .

Original: 1 in problem (16)).
Revised: 1 in problem (16)).

Original: If the adversary only performs single-type attack, training and testing on the same attack type leads to the best performance (diagonal of (cid:96) ∞ - (cid:96) 2 block).
Revised: If the adversary only performs single-type attack, training and testing on the same attack type leads to the best performance (diagonal of (cid:96) ∞ - (cid:96) 2 block).

Original: However, when facing (cid:96) ∞ and (cid:96) 2 attacks simultaneously, 
Revised: However, when facing (cid:96) ∞ and (cid:96) 2 attacks simultaneously, multi-norm generalized AT achieves better Acc max adv and Acc avgadv than single-norm AT. In particular,

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: multi-norm generalized AT achieves better Acc maxadv and Acc avgadv than single-norm AT.

Original: In particular, the min-max strategy slightly outperforms the averaging strategy under multiple perturbation norms.
Revised: the min-max strategy slightly outperforms the averaging strategy under multiple perturbation norms.

Original: DPAR further boosts the adversarial test accuracy, which implies that the promotion of diversiﬁed (cid:96) p attacks is a beneﬁcial supplement to adversarial training.
Revised: DPAR further boosts the adversarial test accuracy, which implies that the promotion of diversiﬁed (cid:96) p attacks is a beneﬁcial supplement to adversarial training.

Original: In Figure 3, we offer deeper insights on the performance of generalized AT.
Revised: In Figure 3, we offer deeper insights on the performance of generalized AT.

Original: During the training procedure we ﬁx (cid:15) (cid:96) ∞ ( (cid:15) for (cid:96) ∞ attack during training) as 0.2, and change (cid:15) (cid:96) 2 from 0.2 to 5.6 ( (cid:15) (cid:96) ∞ ×√ d ) so that the (cid:96) ∞ and (cid:96) 2 balls are not completely overlapped (Araujo et al., 2019).
Revised: During the training procedure we ﬁx (cid:15) (cid:96) ∞ ( (cid:15) for (cid:96) ∞ attack during training) as 0.2, and change (cid:15) (cid:96) 2 from 0.2 to 5.6 ( (cid:15) (cid:96) ∞ ×√ d ) so that the (cid:96) ∞ and (cid:96) 2 balls are not completely overlapped (Araujo et al., 2019).

Original: In Figure 3a, as (cid:15) (cid:96)increases, (cid:96) 2 -attack becomes stronger so the corresponding w also increases, which is consistent with min-max spirit – defending the strongest attack.
Revised: In Figure 3a, as (cid:15) (cid:96)increases, (cid:96) 2 -attack becomes stronger so the corresponding w also increases, which is consistent with min-max spirit – defending the strongest attack.

Original: We remark that min max or avg training does not always lead to the best performance on Acc maxadv and Acc avgadv , especially when the strengths of two attacks diverge greatly (see Table A7).
Revised: We remark that min max or avg training does not always lead to the best performance on Acc maxadv and Acc avgadv , especially when the strengths of two attacks diverge greatly (see Table A8).

Original: This can be explained by the large overlapping between (cid:96) ∞ and (cid:96) 2 balls (see Figure A2).
Revised: This can be explained by the large overlapping between (cid:96) ∞ and (cid:96) 2 balls (see Figure A3).

Original: However, Figure 3b and 3c show that AMPGD is able to achieve a rather robust model no matter how (cid:15) changes (red lines), which empirically veriﬁes the effectiveness of our proposed training scheme.
Revised: However, Figure 3b and 3c show that AMPGD is able to achieve a rather robust model no matter how (cid:15) changes (red lines), which empirically veriﬁes the effectiveness of our proposed training scheme.

Original: In terms of the area-under-the-curve measure S (cid:15) , AMPGD achieves the highest worst-case robustness: 6.27% and 17.64% improvement compared to the vanilla
Revised: In terms of the area-under-the-curve measure S (cid:15) , AMPGD achieves the highest worst-case robustness: 6.27% and 17.64% improvement compared to the vanilla

Original: AT with (cid:96) ∞ and (cid:96) 2 attacks .
Revised: AT with (cid:96) ∞ and (cid:96) 2 attacks .

Original: Furthermore, we show in Figure A3a that our min-max scheme leads to faster convergence than the averaging scheme due to the beneﬁt of self-adjusted domain weights.
Revised: Furthermore, we show in Figure A4a that our min-max scheme leads to faster convergence than the averaging scheme due to the beneﬁt of self-adjusted domain weights.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a general min-max framework applicable to both adversarial attack and defense settings.
Revised: In this paper, we propose a general min-max framework applicable to both adversarial attack and defense settings.

Original: We show that many problem setups can be re-formulated under this general framework.
Revised: We show that many problem setups can be re-formulated under this general framework.

Original: Extensive experiments show that proposed algorithms lead to signiﬁcant improvement on multiple attack and defense tasks compared with previous state-of-the-art approaches.
Revised: Extensive experiments show that proposed algorithms lead to signiﬁcant improvement on multiple attack and defense tasks compared with previous state-of-the-art approaches.

Original: In particular, we obtain 17.48%, 35.21% and 9.39% improvement on attacking model ensembles, devising universal perturbation to input samples, and data transformations under CIFAR-10, respectively.
Revised: In particular, we obtain 17.48%, 35.21% and 9.39% improvement on attacking model ensembles, devising universal perturbation to input samples, and data transformations under CIFAR-10, respectively.

Original: Our minmax scheme also generalizes adversarial training (AT) for multiple types of adversarial attacks, attaining faster convergence and better robustness compared to the vanilla AT and the average strategy.
Revised: Our minmax scheme also generalizes adversarial training (AT) for multiple types of adversarial attacks, attaining faster convergence and better robustness compared to the vanilla AT and the average strategy.

Original: Moreover, our approach provides a holistic tool for self-risk assessment by learning domain weights. 
Revised: Moreover, our approach provides a holistic tool for self-risk assessment by learning domain weights. C ONTENTS 1 Introduction

Revised: 2 Min-Max Power in Adversarial Exploration and Robustness 2.

Revised: General idea:

Revised: Robust learning over multiple domains . . . . . . . . . . . . . . . .

Revised: Robust adversarial attacks . .

Original: A PPENDIX A P ROOF OF L EMMA
Revised: Adversarial training (AT) under mixed types of adversarial attacks . . . . . . . . . 3 Proposed Algorithm and Theory 3. Alternating one-step PGD for robust adversarial attack generation . . Alternating multi-step PGD for generalized AT . .

Revised: 4 Experiments 4.

Revised: Robust adversarial attacks . .

Revised: Adversarial training for multiple adversarial perturbations . . . . . . . . . . . . . .

Revised: 5 Conclusion

Revised: A Proof of

Revised: Lemma

Revised: B Proof of Proposition

Revised: C Experiment Setup

Revised: C.1 Model Architectures and Training Details . . . . . . . . . . . . . . . . . . . . . .

Revised: C.2 Crafting adversarial examples . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: C.3 Details of conducted data transformations . . . . . . . . . . . . . . . . . . . . . .

Revised: D Supplementary Results - Robust adversarial attacks

Revised: D.1 Ensemble attack over multiple models . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: D.2 Comparison with heuristic weighting schemes . . . . . . . . . . . . . . . . . . . . .

Revised: D

Revised: E Supplementary Results - Adversarial training against multiple types of adversarial attacks

Revised: [Table]

Revised: evaluation

Revised: [Table]

Revised: universal adversarial training (UAT)

Revised: E

Revised: [Table]

Revised: Overlap

Revised: (cid:96) p

Revised: norm balls

Revised: [Table]

Revised: [Table]

Revised: F Interpretability of domain weight w on universal perturbation to multiple images

Revised: L IST OF F IGURES

Revised: (Sec 4.1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: L IST OF T ABLES

Revised: A network architectures

Revised: A

Revised: Clean test accuracy of the DNN models (Sec 4) . .

Revised: A

Revised: Ensemble attack over multiple models {A, E, F, H} (Sec 4.1) . . . . . . . . . . . . .

Revised: A

Revised: Comparison with heuristic weight schemes on ensemble attack (Sec 4.1) . . . . . .

Revised: A

Revised: Robust attack over multiple transformations (Sec 4.2) . . . . . . . . . . . . . . . .

Revised: A

Revised: Comparison with universal adversarial training (Sec 4.2) . . . . . . . . . . . . . .

Revised: A

Revised: Generalized adversarial training under mixed types of perturbations (Full table, Sec 4.2)

Revised: A

Revised: Interpretability

Revised: of domain weight w for universal perturbation (digits 0-4, Sec

Revised: A

Revised: Interpretability of domain weight w for universal perturbation (digits 5-9, Sec 4.1)

Revised: A P ROOF OF L EMMA

Original: Lemma 1.
Revised: Lemma 1.

Original: Problem (10) is equivalent to
Revised: Problem (10) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where w ∈ R K represent domain weights, and P has been deﬁned in (1) .
Revised: where w ∈ R K represent domain weights, and P has been deﬁned in (1) .

Original: Proof of Lemma1:
Revised: Proof of Lemma

Original: Similar to (1), problem (10) is equivalent to
Revised: Similar to (1), problem (10) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Recall that F i ( θ ) :
Revised: Recall that F i ( θ ) :

Original: = maximize δ i ∈X i f tr ( θ , δ i ; x , y ) , problem can then be written asminimize
Revised: = maximize δ i ∈X i f tr ( θ , δ i ; x , y ) , problem can then be written as

Original: [Equation]
Revised: [Equation]

Original: According to proof by contradiction, it is clear that problem (19) is equivalent to
Revised: According to proof by contradiction, it is clear that problem (19) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: B P ROOF OF P ROPOSITION
Revised: B P ROOF OF P ROPOSITION

Original: Proposition 1.
Revised: Proposition 1.

Original: Given a point a ∈ R d and a constraint set X = { δ |(cid:107) δ (cid:107) p ≤ (cid:15), ˇ c ≤ δ ≤ ˆ c } , the
Revised: Given a point a ∈ R d and a constraint set X = { δ |(cid:107) δ (cid:107) p ≤ (cid:15), ˇ c ≤ δ ≤ ˆ c } , the

Original: Euclidean projection δ ∗ =
Revised: Euclidean projection δ ∗ =

Original: proj X ( a ) has the closed-form solution when p ∈ { 0 , 1 , 2 } .
Revised: proj X ( a ) has the closed-form solution when p ∈ { 0 , 1 , 2 } .

Original: 1) If p = 1 , then δ ∗ is given by
Revised: 1) If p = 1 , then δ ∗ is given by

Original: [Equation]
Revised: [Equation]

Original: (21) where x i denotes the i th element of a vector x ; P [ˇ c i , ˆ c i ] ( · ) denotes the clip function over the interval [ˇ c i , ˆ c i ] ; sign( x )
Revised: 21) where x i denotes the i th element of a vector x ; P [ˇ c i , ˆ c i ] ( · ) denotes the clip function over the interval [ˇ c i , ˆ c i ] ; sign( x )

Original: = 1 if x ≥ 0 , otherwise 0 ; λ 1 ∈ (0 , max i | a i | − (cid:15)/d ] is the root of (cid:80) di =1 | P [ˇ c i ,
Revised: = 1 if x ≥ 0 , otherwise 0 ; λ 1 ∈ (0 , max i | a i | − (cid:15)/d ] is the root of (cid:80) di =1 | P [ˇ c i ,

Original: ˆ c i ] (sign( a i ) max {| a i | − λ 1 , 0 } )
Revised: ˆ c i ] (sign( a i ) max {| a i | − λ 1 , 0 } )

Original: | = (cid:15) .
Revised: | = (cid:15) .

Original: If p = 2 , then δ ∗ is given by
Revised: If p = 2 , then δ ∗ is given by

Original: [Equation]
Revised: [Equation]

Original: 22) where λ 2 ∈ (0 , (cid:107) a (cid:107) 2 /(cid:15) − 1] is the root of (cid:80) di
Revised: where λ 2 ∈ (0 , (cid:107) a (cid:107) 2 /(cid:15) − 1] is the root of (cid:80) di

Original: =1 ( P [ˇ c i , ˆ c i ] ( a i / ( λ 2 + 1)))
Revised: =1 ( P [ˇ c i , ˆ c i ] ( a i / ( λ 2 + 1)))

Original: 2 =
Revised: 2 =

Original: (cid:15) 2 .
Revised: (cid:15) 2 .

Original: 3) If p = 0 and (cid:15) ∈ N + , then δ ∗ is given by
Revised: If p = 0 and (cid:15) ∈ N + , then δ ∗ is given by

Original: [Equation]
Revised: [Equation]

Original: otherwise . [Equation] 23) where [ η ] (cid:15) denotes the (cid:15) -th largest element of η , and δ (cid:48)
Revised: ) where [ η ] (cid:15) denotes the (cid:15) -th largest element of η , and δ (cid:48)

Original: i =
Revised: i =

Original: P [ˇ c i , ˆ c i ] ( a i ) .
Revised: P [ˇ c i , ˆ c i ] ( a i ) .

Original: Proof of Proposition 1:
Revised: Proof of Proposition 1:

Original: (cid:96) 1 norm When we ﬁnd the Euclidean projection of a onto the set X , we solve
Revised: (cid:96) 1 norm When we ﬁnd the Euclidean projection of a onto the set X , we solve

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where I [ˇ c , ˆ c ] ( · ) is the indicator function of the set [ˇ c , ˆ c ] .
Revised: where I [ˇ c , ˆ c ] ( · ) is the indicator function of the set [ˇ c , ˆ c ] .

Original: The Langragian of this problem is
Revised: The Langragian of this problem is

Original: [Equation]
Revised: [Equation]

Original: The minimizer δ ∗ minimizes the Lagrangian, it is obtained by elementwise soft-thresholding
Revised: The minimizer δ ∗ minimizes the Lagrangian, it is obtained by elementwise soft-thresholding

Original: [Equation]
Revised: [Equation]

Original: where x i is the i th element of a vector x , P [ˇ c i , ˆ c i ] ( · ) is the clip function over the interval [ˇ c i , ˆ c i ] .
Revised: where x i is the i th element of a vector x , P [ˇ c i , ˆ c i ] ( · ) is the clip function over the interval [ˇ c i , ˆ c i ] .

Original: The primal, dual feasibility and complementary slackness are
Revised: The primal, dual feasibility and complementary slackness are

Original: [Equation]
Revised: [Equation]

Original: 27) or
Revised: 27) or

Original: [Equation]
Revised: [Equation]

Original: , 0 } ) | = (cid:15).
Revised: , 0 } ) | = (cid:15).

Original: If (cid:80) di =1 | P [ˇ c i , ˆ c i ] ( a i ) | ≤ (cid:15) , δ ∗ i
Revised: If (cid:80) di =1 | P [ˇ c i , ˆ c i ] ( a i ) | ≤ (cid:15) , δ ∗ i

Original: = P [ˇ c i , ˆ c i ] ( a i ) .
Revised: = P [ˇ c i , ˆ c i ] ( a i ) .

Original: Otherwise δ ∗ i
Revised: Otherwise δ ∗ i

Original: = P [ˇ c i , ˆ c i ] (sign( a i ) max {| a i | − λ 1 , 0 } ) , where λ 1 is given by the root of the equation (cid:80)
Revised: = P [ˇ c i , ˆ c i ] (sign( a i ) max {| a i | − λ 1 , 0 } ) , where λ 1 is given by the root of the equation (cid:80)

Original: di =1 | P [ˇ c i , ˆ c i ] (sign( a i ) max {| a i | − λ 1 , 0 } )
Revised: di =1 | P [ˇ c i , ˆ c i ] (sign( a i ) max {| a i | − λ 1 , 0 } )

Original: | = (cid:15) .
Revised: | = (cid:15) .

Original: Bisection method can be used to solve the above equation for λ 1 , starting with the initial interval (0 , max i | a i | − (cid:15)/d ] .
Revised: Bisection method can be used to solve the above equation for λ 1 , starting with the initial interval (0 , max i | a i | − (cid:15)/d ] .

Original: Since (cid:80) di =1 | P [ˇ c i , ˆ c i ] (sign( a i ) max {| a i | − 0 , 0 } )
Revised: Since (cid:80) di =1 | P [ˇ c i , ˆ c i ] (sign( a i ) max {| a i | − 0 , 0 } )

Original: | = (cid:80) di =1 | P [ˇ c i , ˆ c i ] ( a i ) | > (cid:15) in this case, and (cid:80) di =1 | P [ˇ
Revised: | = (cid:80) di =1 | P [ˇ c i , ˆ c i ] ( a i ) | > (cid:15) in this case, and (cid:80) di =1 | P [ˇ

Original: [Equation]
Revised: [Equation]

Original: i , ˆ c
Revised: i , ˆ c

Original: i ]
Revised: i ]

Original: [Equation]
Revised: [Equation]

Original: sign
Revised: sign

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ) max
Revised: ) max

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: + (cid:15)/d
Revised: + (cid:15)/d

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) | ≤ (cid:80) di
Revised: ) | ≤ (cid:80) di

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:96) 2 norm When we ﬁnd the Euclidean projection of a onto the set X , we solve minimize δ
Revised: (cid:96) 2 norm When we ﬁnd the Euclidean projection of a onto the set X , we solve minimize δ

Original: [Equation]
Revised: [Equation]

Original: subject to (cid:107)
Revised: subject to (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (29) where I [ˇ c , ˆ c ] ( · ) is the indicator function of the set [ˇ c , ˆ c ] .
Revised: (29) where I [ˇ c , ˆ c ] ( · ) is the indicator function of the set [ˇ c , ˆ c ] .

Original: The Langragian of this problem is
Revised: The Langragian of this problem is

Original: [Equation]
Revised: [Equation]

Original: The minimizer δ ∗ minimizes the Lagrangian, it is
Revised: The minimizer δ ∗ minimizes the Lagrangian, it is

Original: [Equation]
Revised: [Equation]

Original: The primal, dual feasibility and complementary slackness are
Revised: The primal, dual feasibility and complementary slackness are

Original: [Equation]
Revised: [Equation]

Original: If (cid:80) di
Revised: If (cid:80) di

Original: =1 ( P [ˇ c i , ˆ c i ] ( a i ))
Revised: =1 ( P [ˇ c i , ˆ c i ] ( a i ))

Original: 2 ≤ (cid:15) 2 , δ ∗ i
Revised: 2 ≤ (cid:15) 2 , δ ∗ i

Original: = P [ˇ c i , ˆ c i ] ( a i ) .
Revised: = P [ˇ c i , ˆ c i ] ( a i ) .

Original: Otherwise δ ∗ i
Revised: Otherwise δ ∗ i

Original: = P [ˇ c i , ˆ c i ] (cid:16) 

Revised: = P [ˇ c i , ˆ c i ] (cid:16)

Original: λ 2 +1 a i (cid:17) , where λ 2 is given by the root of the equation (cid:80) di
Revised: λ 2 +1 a i (cid:17) , where λ 2 is given by the root of the equation (cid:80) di

Original: =1 ( P [ˇ c i , ˆ c i ] ( 1 λ 2 +1 a i ))
Revised: =1 ( P [ˇ c i , ˆ c i ] ( 1 λ 2 +1 a i ))

Original: 2 =
Revised: 2 =

Original: (cid:15) 2 .
Revised: (cid:15) 2 .

Original: Bisection method can be used to solve the above equation for λ 2 , starting with the initial interval (0 , (cid:113)(cid:80) di =1 ( a i ) 2 /(cid:15) − 1] .
Revised: Bisection method can be used to solve the above equation for λ 2 , starting with the initial interval (0 , (cid:113)(cid:80) di =1 ( a i ) 2 /(cid:15) − 1] .

Original: Since (cid:80) di =1 ( P [ˇ c i , ˆ c i ] ( 10+1 a i )) 2 = (cid:80) di
Revised: Since (cid:80) di =1 ( P [ˇ c i , ˆ c i ] ( 10+1 a i )) 2 = (cid:80) di

Original: =1 ( P [ˇ c i , ˆ c i ] ( a i ))
Revised: =1 ( P [ˇ c i , ˆ c i ] ( a i ))

Original: 2 > (cid:15) 2 in this case, and (cid:80) di
Revised: 2 > (cid:15) 2 in this case, and (cid:80) di

Original: =1 ( P [ˇ c i , ˆ c i ] ( 1 λ 2 +1 a i ))
Revised: =1 ( P [ˇ c i , ˆ c i ] ( 1 λ 2 +1 a i ))

Original: 2 = (cid:80) di
Revised: 2 = (cid:80) di

Original: =1 ( P [ˇ c i , ˆ c i ] ( (cid:15)a i / (cid:113)(cid:80) di
Revised: =1 ( P [ˇ c i , ˆ c i ] ( (cid:15)a i / (cid:113)(cid:80) di

Original: =1 ( a i ) 2 )) 2 ≤ (cid:15) 2 (cid:80) di =1 ( a i ) 2 / ( (cid:113)(cid:80) di =1 ( a i ) 2 ) 2 = (cid:15) 2 .
Revised: =1 ( a i ) 2 )) 2 ≤ (cid:15) 2 (cid:80) di =1 ( a i ) 2 / ( (cid:113)(cid:80) di =1 ( a i ) 2 ) 2 = (cid:15) 2 .

Original: (cid:96) 0 norm For (cid:96) 0 norm in X , it is independent to the box constraint.
Revised: (cid:96) 0 norm For (cid:96) 0 norm in X , it is independent to the box constraint.

Original: So we can clip a to the box constraint ﬁrst, which is δ (cid:48)
Revised: So we can clip a to the box constraint ﬁrst, which is δ (cid:48)

Original: i =
Revised: i =

Original: P [ˇ c i , ˆ c i ] ( a i ) , and then project it onto (cid:96) 0 norm.
Revised: P [ˇ c i , ˆ c i ] ( a i ) , and then project it onto (cid:96) 0 norm.

Original: We ﬁnd the additional Euclidean distance of every element in a and zero after they are clipped to the box constraint, which is η i = 
Revised: We ﬁnd the additional Euclidean distance of every element in a and zero after they are clipped to the box constraint, which is η i = 

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: [Equation]
Revised: [Equation]

Original: It can be equivalently written as
Revised: It can be equivalently written as

Original: [Equation]
Revised: [Equation]

Original: To derive the Euclidean projection onto (cid:96) 0 norm, we ﬁnd the (cid:15) -th largest element in η and call it
Revised: To derive the Euclidean projection onto (cid:96) 0 norm, we ﬁnd the (cid:15) -th largest element in η and call it

Original: [ η ] (cid:15) .
Revised: [ η ] (cid:15) .

Original: We keep the elements whose corresponding η i is above or equals to (cid:15) -th, and set rest to zeros.
Revised: We keep the elements whose corresponding η i is above or equals to (cid:15) -th, and set rest to zeros.

Original: The closed-form solution is given by
Revised: The closed-form solution is given by

Original: [Equation]
Revised: [Equation]

Original: Difference to (Hein & Andriushchenko, 2017, Proposition 4.1).
Revised: Difference to (Hein & Andriushchenko, 2017, Proposition 4.1).

Original: We remark that Hein & Andriushchenko (2017) discussed a relevant problem of generating (cid:96) p -norm based adversarial examples under box and linearized classiﬁcation constraints.
Revised: We remark that Hein & Andriushchenko (2017) discussed a relevant problem of generating (cid:96) p -norm based adversarial examples under box and linearized classiﬁcation constraints.

Original: It was shown in (Hein & Andriushchenko, 2017,
Revised: It was shown in (Hein & Andriushchenko, 2017,

Original: Proposition 4.1) that the problem is convex and the solution can be derived using KKT conditions.
Revised: Proposition 4.1) that the problem is convex and the solution can be derived using KKT conditions.

Original: However, Proposition 1 in our paper is different from (Hein & Andriushchenko, 2017, Proposition 4.1).
Revised: However, Proposition 1 in our paper is different from (Hein & Andriushchenko, 2017, Proposition 4.1).

Original: First, we place (cid:96) p norm as a hard constraint rather than minimizing it in the objective function.
Revised: First, we place (cid:96) p norm as a hard constraint rather than minimizing it in the objective function.

Original: This difference will make our Lagrangian function more involved with a newly introduced nonnegative Lagrangian multiplier.
Revised: This difference will make our Lagrangian function more involved with a newly introduced nonnegative Lagrangian multiplier.

Original: Second, the problem of our interest is projection onto the intersection of box and (cid:96) p constraints.
Revised: Second, the problem of our interest is projection onto the intersection of box and (cid:96) p constraints.

Original: Such a projection step can then be combined with an attack loss (no need of linearization) for generating adversarial examples.
Revised: Such a projection step can then be combined with an attack loss (no need of linearization) for generating adversarial examples.

Original: Third, we cover the case of (cid:96) 0 norm.
Revised: Third, we cover the case of (cid:96) 0 norm.

Original: C E XPERIMENT S ETUP C. M ODEL A RCHITECTURES AND T RAINING D ETAILS
Revised: C E XPERIMENT S ETUP C. M ODEL A RCHITECTURES AND T RAINING D ETAILS

Original: For a comprehensive evaluation of proposed algorithms, we adopt a set of diverse DNN models (Model A to H), including multi-layer perceptrons (MLP), All-CNNs Springenberg et al.
Revised: For a comprehensive evaluation of proposed algorithms, we adopt a set of diverse DNN models (Model A to H), including multi-layer perceptrons (MLP), All-CNNs Springenberg et al.

Original: LeNet Lecun et al.
Revised: LeNet Lecun et al.

Original: (1998), LeNetV2 3 , VGG16 Simonyan & Zisserman (2015),
Revised: (1998), LeNetV2 3 , VGG16 Simonyan & Zisserman (2015),

Original: ResNet50 He et al.
Revised: ResNet50 He et al.

Original: (2016), Wide-ResNet Madry et al.
Revised: (2016), Wide-ResNet Madry et al.

Original: (2017) and GoogLeNet Szegedy et al.
Revised: (2017) and GoogLeNet Szegedy et al.

Original: For the last four models, we use the exact same architecture as original papers and evaluate them only on CIFARdataset.
Revised: For the last four models, we use the exact same architecture as original papers and evaluate them only on CIFARdataset.

Original: The details for model architectures are provided in Table A1.
Revised: The details for model architectures are provided in Table A1.

Original: For compatibility with our framework, we implement and train these models based on the strategies adopted in pytorch-cifarand achieve comparable performance on clean images; see Table A2.
Revised: For compatibility with our framework, we implement and train these models based on the strategies adopted in pytorch-cifarand achieve comparable performance on clean images; see Table A2.

Original: To foster reproducibility, all the trained models are publicly accessible in the anonymous link.
Revised: To foster reproducibility, all the trained models are publicly accessible in the anonymous link.

Original: Speciﬁcally, we trained MNIST classiﬁers for 50 epochs with Adam and a constant learning rate of 0.001.
Revised: Speciﬁcally, we trained MNIST classiﬁers for 50 epochs with Adam and a constant learning rate of 0.001.

Original: For CIFAR-10 classifers, the models are trained for 250 epochs with SGD (using 0.8 nesterov momentum, weight decay 5 e − 4 ).
Revised: For CIFAR-10 classifers, the models are trained for 250 epochs with SGD (using 0.8 nesterov momentum, weight decay 5 e − 4 ).

Original: The learning rate is reduced at epoch 100 and 175 with a decay rate of 0.1.
Revised: The learning rate is reduced at epoch 100 and 175 with a decay rate of 0.1.

Original: The initial learning rate is set as 0.01 for models {A, B, C, D, H} and 0.1 for {E, F, G}.
Revised: The initial learning rate is set as 0.01 for models {A, B, C, D, H} and 0.1 for {E, F, G}.

Original: Note that no data augmentation is employed in the training.
Revised: Note that no data augmentation is employed in the training.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C. C RAFTING ADVERSARIAL EXAMPLES
Revised: C RAFTING ADVERSARIAL EXAMPLES

Original: We adopt variant C&W loss in APGD/PGD as suggested in Madry et al.
Revised: We adopt variant C&W loss in APGD/PGD as suggested in Madry et al.

Original: ; Carlini & Wagner (2017) with a conﬁdence parameter κ = 50 .
Revised: ; Carlini & Wagner (2017) with a conﬁdence parameter κ = 50 .

Original: Cross-entropy loss is also supported in our implementation.
Revised: Cross-entropy loss is also supported in our implementation.

Original: The adversarial examples are generated by 20-step PGD/APGD unless otherwise stated (e
Revised: The adversarial examples are generated by 20-step PGD/APGD unless otherwise stated (e

Original: [Table]
Revised: [Table]

Original: g.,steps for ensemble attacks).
Revised: g., 50 steps for ensemble attacks).

Revised: Note that proposed algorithms are robust and will not be affected largely by the choices of hyperparameters ( α, β, γ ).

Revised: In consequence, we do not ﬁnely tune the parameters on the validation set.

Revised: Speciﬁcally, The learning rates α, β and regularization factor γ for Table 1 are set as - (a) MNIST: (cid:96) 0 :

Revised: [Table]

Original: Due to varying model robustness on different datasets, the perturbation magnitudes (cid:15) are set separately.
Revised: Due to varying model robustness on different datasets, the perturbation magnitudes (cid:15) are set separately Carlini et al.

Original: For universal perturbation experiments, the (cid:15) are set as 0.2 (A, B), 0. (C) and 0.25 (D) on MNIST; 0.02 (B, H), 0.35 (E) and 0.05 (D) on CIFAR-10.
Revised: For universal perturbation experiments, the (cid:15) are set as 0.2 (A, B), 0.3 (C) and 0.25 (D) on MNIST; 0.02 (B, H), 0.35 (E) and 0.05 (D) on CIFAR-10.

Original: For generalized AT, the models on MNIST are trained following the same rules in last section, except that training epochs are prolonged to 350 and adversarial examples are crafted for assisting the training with a ratio of 0.5.
Revised: For generalized AT, the models on MNIST are trained following the same rules in last section, except that training epochs are prolonged to 350 and adversarial examples are crafted for assisting the training with a ratio of 0.5.

Original: Our experiment setup is based on CleverHans package 5 and Carlini and Wagner’s framework 6 .
Revised: Our experiment setup is based on CleverHans package 5 and Carlini and Wagner’s framework 6 .

Original: C. D ETAILS OF CONDUCTED DATA TRANSFORMATIONS
Revised: C. D ETAILS OF CONDUCTED DATA TRANSFORMATIONS

Original: To demonstrate the effectiveness of APGD in generating robust adversarial examples against multiple transformations, we adopt a series of common transformations, including a&b) ﬂipping images horizontally ( ﬂh ) and vertically ( ﬂv ); c) adjusting image brightness ( bri ); d) performing gamma correction ( gam ), e) cropping and re-sizing images ( crop ); f) rotating images ( rot ).
Revised: To demonstrate the effectiveness of APGD in generating robust adversarial examples against multiple transformations, we adopt a series of common transformations, including a&b) ﬂipping images horizontally ( ﬂh ) and vertically ( ﬂv ); c) adjusting image brightness ( bri ); d) performing gamma correction ( gam ), e) cropping and re-sizing images ( crop ); f) rotating images ( rot ).

Original: Moreover, both deterministic and stochastic transformations are considered in our experiments.
Revised: Moreover, both deterministic and stochastic transformations are considered in our experiments.

Original: In particular, Table 3 and Table A6 are deterministic settings - rot : rotating images 30 degree clockwise; crop : cropping images in the center ( 0 . 8 × 0 . 8 ) and resizing them to 32 × 32 ; bri : adjusting the brightness of images with a scale of 0.1; gam : performing gamma correction with a value of 1.3.
Revised: In particular, Table 3 and Table A6 are deterministic settings - rot : rotating images 30 degree clockwise; crop : cropping images in the center ( 0 . 8 × 0 . 8 ) and resizing them to 32 × 32 ; bri : adjusting the brightness of images with a scale of 0.1; gam : performing gamma correction with a value of 1.3.

Original: Differently, in Table A5, we introduce randomness for drawing samples from the distribution - rot : rotating images randomly from -10 to 10 degree; crop : cropping images in the center randomly5 https (from 0.6 to 1.0); other transformations are done with a probability of 0.8.
Revised: Differently, in Table A5, we introduce randomness for drawing samples from the distribution - rot : rotating images randomly from -10 to 10 degree; crop : cropping images in the center randomly (from 0.6 to 1.0); other transformations are done with a probability of 0.8.

Original: In experiments, we adopt tf.image API 7 for processing the images.
Revised: In experiments, we adopt tf.image API 7 for processing the images.

Original: D S UPPLEMENTARY R ESULTS D. R OBUST ADVERSARIAL ATTACKS Ensemble attack
Revised: D S UPPLEMENTARY R ESULTS - R OBUST ADVERSARIAL ATTACKS D. E NSEMBLE ATTACK OVER MULTIPLE MODELS

Original: over multiple models Table A3 and A4 show the performance of average (ensemble PGD Liu et al.
Revised: Table A3 shows the performance of average (ensemble PGD Liu et al.

Original: and min-max (APGD) strategies for attacking model ensembles.
Revised: and min-max (APGD) strategies for attacking model ensembles.

Original: Our min-max approach results in19.27% and 15.69% averaged improvement on ASR all over model sets {A, B, C, D} and {A, E, F, H} on CIFAR-10. 
Revised: Our min-max approach results in 15.69% averaged improvement on ASR all over models {A, E, F, H} on CIFAR-10. [Figure] [Table] D. C OMPARISON WITH HEURISTIC WEIGHTING SCHEMES

Revised: To further demonstrate the effectiveness of self-adjusted weighting factors in proposed min-max framework

Original: [Table]
Revised: [Table]

Revised: we compare with heuristic weighting schemes in Table A4.

Revised: Speciﬁcally

Revised: [Table]

Revised: with the prior knowledge of robustness of given models (C > D > A > B)

Revised: [Table]

Revised: we devised several heuristic baselines including: (a) w c + d : ensemble PGD on models C and D only; (b) w a + c + d : ensemble PGD on models

Revised: A, C and D only; (c) w clip : clipped version of C&W loss (threshold β = 40 ) to balance model weights in optimization as suggested in Shafahi et al.

Revised: (2018); (d) w prior : larger weights on the more robust models, w prior =

Revised: [ w A , w B , w C , w D ] =

Revised: Table A4 shows that our min-max approach outperforms all static heuristic weighting schemes by a large margin.

Revised: Speciﬁcally, our min-max APGD also achieves signiﬁcant improvement compared to w static setting, where the converged optimal weights are statically (i.e., invariant w.r.t different images and attack procedure) adopted.

Revised: It again veriﬁes the beneﬁts of proposed min-max approach by automatically learning the weights for different examples during the process of ensemble attack generation (see Figure 1c).

Original: [Table]
Revised: [Table]

Original: Robust adversarial attack over data transformations Table A5 and A6 compare the performance of average (EOT Athalye et al.
Revised: Table A5 and A6 compare the performance of average (EOT Athalye et al.

Original: (2018a))
Revised: (2018a))

Original: and min-max (APGD) strategies.
Revised: and min-max (APGD) strategies.

Original: Our approach results in 4.31% and 8.22% averaged lift over four models {A, B, C, D} on CIFAR-10 under given stochastic and deterministic transformation sets. 
Revised: Our approach results in 4.31% and 8.22% averaged lift over four models {A, B, C, D} on CIFAR-10 under given stochastic and deterministic transformation sets. [Table] [Table]

Original: D. A DVERSARIAL TRAINING AGAINST MULTIPLE TYPES OF ADVERSARIAL ATTACKS
Revised: D. S ENSITIVITY ANALYSIS OF REGULARIZER ON PROBABILITY SIMPLEX

Revised: [Figure] [Figure]

Revised: To further explore the utility of quadratic regularizer on the probability simplex in proposed min-max framework, we conducted sensitivity analysis on γ and show how the proposed regularization affects the eventual performance (Figure A1) taking ensemble attack as an example.

Revised: The experimental setting is the same as Table 1 except for altering the value of γ from 0 to 10.

Revised: Figure A1 shows that too small or too large γ leads to relative weak performance due to the unstable convergence and penalizing too much for average case.

Revised: When γ is around 4 , APGD will achieve the best performance so we adopted this value in the experiments (Table 1).

Revised: Moreover, when γ → ∞ , the regularizer term dominates the optimization objective and it becomes the average case.

Revised: E S UPPLEMENTARY R ESULTS - A DVERSARIAL TRAINING AGAINST MULTIPLE TYPES OF ADVERSARIAL ATTACKS E. R OBUSTNESS EVALUATION

Original: Robustness evaluation Figure A1 presents “overall robustness” comparison of our min-max generalized AT scheme and vanilla AT with single type of attacks ( (cid:96) ∞ and (cid:96) 2 ) on MNIST (LeNet).
Revised: Figure A2 presents “overall robustness” comparison of our min-max generalized AT scheme and vanilla AT with single type of attacks ( (cid:96) ∞ and (cid:96) 2 ) on MNIST (LeNet).

Original: Similarly, our min-max training scheme leads to a higher “overall robustness” measured by S (cid:15) .
Revised: Similarly, our min-max training scheme leads to a higher “overall robustness” measured by S (cid:15) .

Original: In practice, due to the lacking knowledge of the strengths/types of the attacks used by adversaries, it is meaningful to enhance “overall robustness” of models under the worst perturbation (Acc maxadv ).[Table] [Table]
Revised: In practice, due to the lacking knowledge of the strengths/types of the attacks used by adversaries, it is meaningful to enhance “overall robustness” of models under the worst perturbation (Acc maxadv ).

Original: Speciﬁcally [Table] our min-max generalized AT leads to 6.27% and 17.63% improvement on S (cid:15) compared to single-type AT with (cid:96) ∞ and (cid:96) 2 attacks.
Revised: Speciﬁcally, our min-max generalized AT leads to 6.27% and 17.63% improvement on S (cid:15) compared to single-type AT with (cid:96) ∞ and (cid:96) 2 attacks.

Original: Furthermore, weighting factor w of the probability simplex helps understand the behavior of AT under mixed types of attacks.
Revised: Furthermore, weighting factor w of the probability simplex helps understand the behavior of AT under mixed types of attacks.

Original: Our AMPGD algorithm will adjust w automatically according to the min-max principle - defending the strongest attack.
Revised: Our AMPGD algorithm will adjust w automatically according to the min-max principle - defending the strongest attack.

Original: In Figure A1a, as (cid:15) (cid:96) 2 increases, (cid:96) 2 -attack becomes stronger so its corresponding w increases as well.
Revised: In Figure A2a, as (cid:15) (cid:96) 2 increases, (cid:96) 2 -attack becomes stronger so its corresponding w increases as well.

Original: When (cid:15) (cid:96) 2 ≥ 2 .
Revised: When (cid:15) (cid:96) 2 ≥ 2 .

Original: 5 , (cid:96) 2 -attack dominates the adversarial training process.
Revised: 5 , (cid:96) 2 -attack dominates the adversarial training process.

Original: That is to say, our AMPGD algorithm will put more weights on stronger attacks even if the strengths of attacks are unknown, which is a meritorious feature in practice.
Revised: That is to say, our AMPGD algorithm will put more weights on stronger attacks even if the strengths of attacks are unknown, which is a meritorious feature in practice.

Original: [Figure]
Revised: [Figure]

Original: Table A7 shows complete results on the test accuracy of models in different training schemes.
Revised: Table A8 shows complete results on the test accuracy of models in different training schemes.

Original: In general, the min-max generalized AT obtains better performance than averaging strategy.
Revised: In general, the min-max generalized AT obtains better performance than averaging strategy.

Original: AMPGD always leads to Top-2 Acc max adv and Acc avgadv .
Revised: AMPGD always leads to Top-2 Acc max adv and Acc avg adv .

Revised: E. C OMPARISON WITH UNIVERSAL ADVERSARIAL TRAINING (UAT)

Revised: Shafahi et al.

Original: Overlap of  (cid:96) p -norm balls As reported in Sec.
Revised: (2018) also propose a variant of adversarial training to defend universal perturbations over multiple images. To produce universal perturbations, they propose uSGD to conduct gradient descent on the averaged loss of one-batch images. In consequence, their approach can be regarded as a variant of our generalized AT in average case. The difference is that they do AT across multiple adversarial images under universal perturbation rather than mixed (cid:96) p -norm perturbations.

Revised: We added UAT [1] as one of our defense baselines in Table A7.

Original: 4.2, our min-max generalized AT does not always result in the best performance onthe success rate of defending the worst/strongest perturbation (Acc maxadv ) for given ( (cid:15) (cid:96) ∞ , (cid:15) (cid:96) 2 ) pair, especially when the strengths of two attacks diverge greatly (e
Revised: The universal perturbation is generated by uSGD ( (cid:96) ∞ norm, (cid:15) = 0 . 3 ) with a batch size of 128 following Shafahi et al. We ﬁnd that a) our proposed approach outperforms UAT under per-image (cid:96) p attacks. Taking A7a as an example, our avg and min max generalized AT (with DPAR) result in average 17.85% and 17.97% improvement in adversarial test accuracy (ATA), b) our approach has just 3.72% degradation in ATA when encountering universal attacks, and c) both methods yield very similar normal test accuracy. It is not surprising that our average and min-max training schemes can achieve better overall robustness while maintaining competitive performance on defending universal perturbation.

Revised: This is because the defensed model is trained under more general ( (cid:96) p norm) and diversity promoted perturbations.

Revised: As a result, proposed generalized AT is expected to obtain better overall robustness and higher transferability as shown in Table 4 and A7.

Original: [Figure]
Revised: [Table]

Original: g., (cid:15) for (cid:96) ∞ and (cid:96) 2 attacks are 0.2 and 0.5).
Revised: E. O VERLAP OF (cid:96) p - NORM BALLS

Original: In what follows
Revised: As reported in Sec. 4. [Table] our min-max generalized AT does not always result in the best performance on the success rate of defending the worst/strongest perturbation (Acc maxadv ) for given ( (cid:15) (cid:96) ∞ , (cid:15) (cid:96) 2 ) pair, especially when the strengths of two attacks diverge greatly (e

Original: [Figure]
Revised: [Figure]

Revised: g., (cid:15) for (cid:96) ∞ and (cid:96) 2 attacks are 0. and 0.5).

Original:  we provide explanation and analysis about this ﬁnding inspired by recent work Araujo et al.
Revised: In what follows, we provide explanation and analysis about this ﬁnding inspired by recent work Araujo et al.

Original: [Figure]
Revised: [Figure]

Original: [Figure] Figure A2 shows the real overlap of (cid:96) ∞ and (cid:96) 2 norm balls in adversarial attacks for MLP model on
Revised: Figure A3 shows the real overlap of (cid:96) ∞ and (cid:96) 2 norm balls in adversarial attacks for MLP model on

Original: MNIST.
Revised: MNIST.

Original: Ideally, if (cid:15) (cid:96) 2 satisﬁes (cid:15) (cid:96) ∞ < (cid:15) (cid:96) 2 < (cid:15) (cid:96) ∞ × √ d , (cid:96) ∞ and (cid:96) 2 balls will not cover each other completely Araujo et al.
Revised: Ideally, if (cid:15) (cid:96) 2 satisﬁes (cid:15) (cid:96) ∞ < (cid:15) (cid:96) 2 < (cid:15) (cid:96) ∞ × √ d , (cid:96) ∞ and (cid:96) 2 balls will not cover each other completely Araujo et al.

Original: In other words, AT with (cid:96) ∞ and (cid:96) 2 attacks cannot interchange with each other.
Revised: In other words, AT with (cid:96) ∞ and (cid:96) 2 attacks cannot interchange with each other.

Original: However, the real range of (cid:15) (cid:96) 2 for keeping (cid:96) 2 and (cid:96) ∞ balls intersected is not ( (cid:15) (cid:96) ∞ , (cid:15) (cid:96) ∞ × √ d ) , because crafted adversarial examples are not uniformly distributed in (cid:96) p -norm balls.
Revised: However, the real range of (cid:15) (cid:96) 2 for keeping (cid:96) 2 and (cid:96) ∞ balls intersected is not ( (cid:15) (cid:96) ∞ , (cid:15) (cid:96) ∞ × √ d ) , because crafted adversarial examples are not uniformly distributed in (cid:96) p -norm balls.

Original: In Figure A2b, 99.98% adversarial examples devising using (cid:96) 2 attack are also inside (cid:96) ∞ ball, even if 0 .
Revised: In Figure A3b, 99.98% adversarial examples devising using (cid:96) 2 attack are also inside (cid:96) ∞ ball, even if 0 .

Original: 2 < (cid:15) (cid:96) 2 = 0 .
Revised: 2 < (cid:15) (cid:96) 2 = 0 .

Original: 5 < 5 .
Revised: 5 < 5 .

Original: In consequence, AT with (cid:96) ∞ attack is enough to handle (cid:96) 2 -attack in overwhelming majority cases, which results in better performance than min-max optimization (Table A7a).
Revised: In consequence, AT with (cid:96) ∞ attack is enough to handle (cid:96) 2 -attack in overwhelming majority cases, which results in better performance than min-max optimization (Table A8a).

Original: Figure A2c presents the average (cid:96) p distance of adversarial examples with (cid:15) (cid:96) 2 increasing.
Revised: Figure A3c presents the average (cid:96) p distance of adversarial examples with (cid:15) (cid:96) 2 increasing.

Original: The average (cid:96) 2 -norm (green line) of adversarial examples generated by (cid:96) ∞ attack remains around 2.0 with a slight rising trend.
Revised: The average (cid:96) 2 -norm (green line) of adversarial examples generated by (cid:96) ∞ attack remains around 2.0 with a slight rising trend.

Original: This is consistent to our setting - ﬁxing (cid:15) (cid:96) 2 as 0.2.
Revised: This is consistent to our setting - ﬁxing (cid:15) (cid:96) 2 as 0.2.

Original: It also indicates model robustness may effect the behavior of attacks - as (cid:15) (cid:96) 2 increases, robustly trained MLP model becomes more robust against (cid:96) 2 examples, so the (cid:96) ∞ attacker implicitly increases (cid:96) 2 norm to attack the model more effectively.
Revised: It also indicates model robustness may effect the behavior of attacks - as (cid:15) (cid:96) 2 increases, robustly trained MLP model becomes more robust against (cid:96) 2 examples, so the (cid:96) ∞ attacker implicitly increases (cid:96) 2 norm to attack the model more effectively.

Original: On the other hand, the average (cid:96) ∞ -norm increases substantially as (cid:15) (cid:96) 2 increases from 0.
Revised: On the other hand, the average (cid:96) ∞ -norm increases substantially as (cid:15) (cid:96) 2 increases from 0.

Original: to 2.5.
Revised: to 2.5.

Original: When (cid:15) (cid:96) 2 arriving at 0.85, the average (cid:96) ∞ norm gets close to 0.2, so around half adversarial examples generated by (cid:96) 2 -attack are also inside (cid:96) ∞ balls, which is consistent with Table A2b.
Revised: When (cid:15) (cid:96) 2 arriving at 0.85, the average (cid:96) ∞ norm gets close to 0.2, so around half adversarial examples generated by (cid:96) 2 -attack are also inside (cid:96) ∞ balls, which is consistent with Table A3b. E

Original: Learning curve under different training schemes Figure A3 shows the learning curves of model
Revised: Figure A4 shows the learning curves of model

Original: A under different AT schemes, where two setting are plotted: (a) ( (cid:15) (cid:96) ∞ , (cid:15) (cid:96) 2 ) =
Revised: A under different AT schemes, where two setting are plotted: (a) ( (cid:15) (cid:96) ∞ , (cid:15) (cid:96) 2 ) =

Original: (0 . 2 , 0 . 5) ; (b) ( (cid:15) (cid:96) ∞ , (cid:15) (cid:96) 2 ) =
Revised: (0 . 2 , 0 . 5) ; (b) ( (cid:15) (cid:96) ∞ , (cid:15) (cid:96) 2 ) =

Original: Apart from better worst-case robustness shown in Table A7, our min-max generalized AT leads to a faster convergence compared to average-based AT, especially when the strengths of two attacks diverge greatly.
Revised: Apart from better worst-case robustness shown in Table A8, our min-max generalized AT leads to a faster convergence compared to average-based AT, especially when the strengths of two attacks diverge greatly.

Original: For instance, when (cid:15) (cid:96) 2 = 0 . 5 (Figure A3a), the robust model trained with AMPGD reaches 70% test accuracy on the worst perturbation (1- R maxadv ) withinepochs versus 280 epochs in average setting.
Revised: For instance, when (cid:15) (cid:96) 2 = 0 . 5 (Figure A4a), the robust model trained with AMPGD reaches 70% test accuracy on the worst perturbation (1- R maxadv ) within 210 epochs versus 280 epochs in average setting.

Original: When (cid:15) (cid:96) 2 = 2 . 

Revised: When (cid:15) (cid:96) 2 = 2 .

Original: (Figure A3b), the learning curves for min-max and average strategy are very close because the strengths of two attacks are similar, which is veriﬁed by approximately equal weights in Figure 3a. 
Revised: (Figure A4b), the learning curves for min-max and average strategy are very close because the strengths of two attacks are similar, which is veriﬁed by approximately equal weights in Figure 3a. [Figure]

Original: D. I NTERPRETABILITY OF DOMAIN WEIGHT w ON UNIVERSAL PERTURBATION TO MULTIPLE IMAGES
Revised: F I NTERPRETABILITY OF DOMAIN WEIGHT w ON UNIVERSAL PERTURBATION TO MULTIPLE IMAGES

Original: Tracking domain weight w of the probability simplex from our algorithms is an exclusive feature of solving problem 1.
Revised: Tracking domain weight w of the probability simplex from our algorithms is an exclusive feature of solving problem 1.

Original: In Sec 4, we show the strength of w in understanding the procedure of optimization [Table] [Table] [Table]
Revised: In Sec 4, we show the strength of w in understanding the procedure of optimization and interpreting the adversarial robustness.

Original: [Table]

Original: [Table] [Figure] and interpreting the adversarial robustness. Here we would like to show the usage of w in measuring “image robustness” on devising universal perturbation to multiple input samples.
Revised: Here we would like to show the usage of w in measuring “image robustness” on devising universal perturbation to multiple input samples.

Original: Table A8 and A show the image groups on MNIST with weight w in APGD and two metrics (distortion of (cid:96) 2 -C&W, minimum (cid:15) for (cid:96) ∞ -PGD) of measuring the difﬁculty of attacking single images.
Revised: Table A9 and A show the image groups on MNIST with weight w in APGD and two metrics (distortion of (cid:96) 2 -C&W, minimum (cid:15) for (cid:96) ∞ -PGD) of measuring the difﬁculty of attacking single images.

Original: The binary search is utilized to searching for the minimum perturbation.
Revised: The binary search is utilized to searching for the minimum perturbation.

Original: Although adversaries need to consider a trade-off between multiple images while devising universal perturbation, we ﬁnd that weighting factor w in APGD is highly correlated under different (cid:96) p norms.
Revised: Although adversaries need to consider a trade-off between multiple images while devising universal perturbation, we ﬁnd that weighting factor w in APGD is highly correlated under different (cid:96) p norms.

Original: Furthermore, w is also highly related to minimum distortion required for attacking a single imagesuccessfully.
Revised: Furthermore, w is also highly related to minimum distortion required for attacking a single image

Original: It means the inherent “image robustness” exists and effects the behavior of generating universal perturbation. Larger weight w usually indicates an image with higher robustness (e.g., ﬁfth ’zero’ in the ﬁrst row of Table A8), which usually
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: successfully. It means the inherent “image robustness” exists and effects the behavior of generating universal perturbation.

Original: [Table] [Table] [Table] Metric
Revised: Larger weight w usually indicates an image with higher robustness (e

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: g., ﬁfth ’zero’ in the ﬁrst row of Table A9), which usually corresponds to the MNIST letter with clear appearance (e.g., bold letter).

Original: [Table]
Revised: [Table]

Original: Image
Revised: Image

Original: Weight [Table] [Table] [Table]
Revised: Weight (cid:96)0.

Original: [Table] [Table] [Table] [Table]
Revised: (cid:96)0.

Original: [Table] [Table] [Table] [Table]
Revised: (cid:96)0.

Original: [Table] [Table] [Table] [Table]
Revised: (cid:96) ∞ 0.

Original: Metric [Table] [Table] [Table]
Revised: Metric dist.(C&W (cid:96) 2 ) 1.

Original: [Table]
Revised: [Table]

Original: ImageWeight (cid:96)0. (cid:96)0.
Revised: Image

Original:  (cid:96)0.(cid:96) ∞ 0.
Revised: Weight (cid:96)0. (cid:96)0.

Original:  Metric dist.(C&W (cid:96) 2 ) 2. (cid:15) min ( (cid:96) ∞ ) 0.
Revised: (cid:96)0. (cid:96) ∞ 0. Metric dist.(C&W (cid:96) 2 ) 1.

Revised: (cid:15) min ( (cid:96) ∞ ) 0.

Original: Image
Revised: Image

Original: Weight (cid:96)0.
Revised: Weight (cid:96)0.

Original: (cid:96)0. (cid:96)0.
Revised: (cid:96)0. (cid:96)0.

Original: (cid:96) ∞ 0.
Revised: (cid:96) ∞ 0.

Original: [Table]
Revised: Metric dist.(C&W (cid:96) 2 ) 2.

Original: [Table] min ( (cid:96) ∞ ) 0.
Revised: (cid:15) min ( (cid:96) ∞ ) 0.

Original: Image
Revised: Image

Original: Weight (cid:96)0.
Revised: Weight (cid:96)0.

Original: (cid:96)0. 
Revised: (cid:96)0. [Table] [Table] min ( (cid:96) ∞ ) 0.

Original:  (cid:96)0.
Revised: Image Weight (cid:96)0. (cid:96)0. (cid:96)0.

Original: (cid:96) ∞ 0.
Revised: (cid:96) ∞ 0.

Original: Metric dist.
Revised: Metric dist.

Original: ( (cid:96) 2 ) 1.
Revised: ( (cid:96) 2 ) 1.

Original: (cid:15) min ( (cid:96) ∞ ) 0.
Revised: (cid:15) min ( (cid:96) ∞ ) 0.

Original: Image
Revised: Image

Original: Weight (cid:96)1.
Revised: Weight (cid:96)1.

Original: (cid:96)0. (cid:96)0.
Revised: (cid:96)0. (cid:96)0.

Original: (cid:96) ∞ 0.
Revised: (cid:96) ∞ 0.

Original: [Table]
Revised: [Table]