Original: Random Normalization Aggregation for Adversarial Defense
Revised: Random Normalization Aggregation for Adversarial Defense

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The vulnerability of deep neural networks has been widely found in various modelsas well as tasks where slight perturbations on the inputs could lead to incorrectpredictions.
Revised: The vulnerability of deep neural networks has been widely found in various modelsas well as tasks where slight perturbations on the inputs could lead to incorrectpredictions.

Original: These perturbed inputs are known as adversarial examples and one ofthe intriguing properties of them is Adversarial Transfersability , i .
Revised: These perturbed inputs are known as adversarial examples and one ofthe intriguing properties of them is Adversarial Transfersability , i .

Original: e .
Revised: e .

Original: the capabilityof adversarial examples to fool other models.
Revised: the capabilityof adversarial examples to fool other models.

Original: Traditionally, this transferabilityis always regarded as a critical threat to the defense against adversarial attacks,however, we argue that the network robustness can be significantly boosted byutilizing adversarial transferability from a new perspective.
Revised: Traditionally, this transferabilityis always regarded as a critical threat to the defense against adversarial attacks,however, we argue that the network robustness can be significantly boosted byutilizing adversarial transferability from a new perspective.

Original: In this work, we firstdiscuss the influence of different popular normalization layers on the adversarialtransferability, and then provide both empirical evidence and theoretical analysisto shed light on the relationship between normalization types and transferability.
Revised: In this work, we firstdiscuss the influence of different popular normalization layers on the adversarialtransferability, and then provide both empirical evidence and theoretical analysisto shed light on the relationship between normalization types and transferability.

Original: Based on our theoretical analysis, we propose a simple yet effective module named
Revised: Based on our theoretical analysis, we propose a simple yet effective module named

Original: Random Normalization Aggregation (RNA) which replaces the batch normalizationlayers in the networks and aggregates different selected normalization types to forma huge random space.
Revised: Random Normalization Aggregation (RNA) which replaces the batch normalizationlayers in the networks and aggregates different selected normalization types to forma huge random space.

Original: Specifically, a random path is sampled during each inferenceprocedure so that the network itself can be treated as an ensemble of a wide rangeof different models.
Revised: Specifically, a random path is sampled during each inferenceprocedure so that the network itself can be treated as an ensemble of a wide rangeof different models.

Original: Since the entire random space is designed with low adversarialtransferability, it is difficult to perform effective attacks even when the networkparameters are accessible.
Revised: Since the entire random space is designed with low adversarialtransferability, it is difficult to perform effective attacks even when the networkparameters are accessible.

Original: We conduct extensive experiments on various modelsand datasets, and demonstrate the strong superiority of proposed algorithm.
Revised: We conduct extensive experiments on various modelsand datasets, and demonstrate the strong superiority of proposed algorithm.

Original:  Introduction
Revised:  Introduction

Original: Deep Neural Networks (DNNs) have achieved impressive performance in various tasks [1, 2, 3].
Revised: Deep Neural Networks (DNNs) have achieved impressive performance in various tasks [1, 2, 3].

Original: However, it is well known that DNNs are susceptible to maliciously generated adversarial examples[4, 5].
Revised: However, it is well known that DNNs are susceptible to maliciously generated adversarial examples[4, 5].

Original: Through imperceptible perturbations on the model inputs during inference stage, the model ismisled to wrong predictions at a high rate.
Revised: Through imperceptible perturbations on the model inputs during inference stage, the model ismisled to wrong predictions at a high rate.

Original: Since then, a wide range of attack techniques have beenproposed under different settings and show strong attack capability.
Revised: Since then, a wide range of attack techniques have beenproposed under different settings and show strong attack capability.

Original: For example, attackers have fullaccess to the model architecture and parameters, which forms white-box attacks [5, 6], and attackershave limited query access to the model, which forms black-box attacks [7, 8].
Revised: For example, attackers have fullaccess to the model architecture and parameters, which forms white-box attacks [5, 6], and attackershave limited query access to the model, which forms black-box attacks [7, 8].

Original: Since the high attacksuccess rates of these techniques reveal the high risk of DNNs, defenses against adversarial exampleshave received increasing attention and adversarial robustness becomes one of the key criteria.
Revised: Since the high attacksuccess rates of these techniques reveal the high risk of DNNs, defenses against adversarial exampleshave received increasing attention and adversarial robustness becomes one of the key criteria.

Original: To mitigate this risk, adversarial training is proposed to yield robust models through training ongenerated adversarial examples [5, 6].
Revised: To mitigate this risk, adversarial training is proposed to yield robust models through training ongenerated adversarial examples [5, 6].

Original: Besides training procedure, some regularization and imagepreprocessing techniques are introduced to improve adversarial robustness [9, 10].
Revised: Besides training procedure, some regularization and imagepreprocessing techniques are introduced to improve adversarial robustness [9, 10].

Original: Recent worknote that the architecture and module designs could play important roles in the robustness [11, 12].
Revised: Recent worknote that the architecture and module designs could play important roles in the robustness [11, 12].

Original: Hence, we pay more attention to the basic modules in the network which are seldom considered forimproving robustness, such as normalization layers.
Revised: Hence, we pay more attention to the basic modules in the network which are seldom considered forimproving robustness, such as normalization layers.

Original: Existing works have discussed the influence of
Revised: Existing works have discussed the influence of

Original: Batch Normalization (BN) and empirically shown that BN increases adversarial vulnerability and
Revised: Batch Normalization (BN) and empirically shown that BN increases adversarial vulnerability and

Original: [Figure]
Revised: [Figure]

Original: decreases adversarial transferability [13, 14].
Revised: decreases adversarial transferability [13, 14].

Original: However, the theoretical analysis of this observation isinsufficient and how to tackle this pitfall or even utilize this property to defend attacks is unexplored.
Revised: However, the theoretical analysis of this observation isinsufficient and how to tackle this pitfall or even utilize this property to defend attacks is unexplored.

Original: In this work, we take numerous normalizations into consideration, including Layer Normalization(LN), Group Normalization (GN), Instance Normalization (IN), Batch Normalization (BN), and
Revised: In this work, we take numerous normalizations into consideration, including Layer Normalization(LN), Group Normalization (GN), Instance Normalization (IN), Batch Normalization (BN), and

Original: Batch Group Normalization (BGN) [15, 16, 17, 18, 19], as shown in Figure 1 (a) and (b).
Revised: Batch Group Normalization (BGN) [15, 16, 17, 18, 19], as shown in Figure 1 (a) and (b).

Original: To evaluatethe influence of different normalizations on the robustness, we first conduct PGD-7 attack [6] to bothnatural and adversarial trained networks with different normalizations on CIFAR-10, as shown in thediagonals of Figure 1 (c) and (d).
Revised: To evaluatethe influence of different normalizations on the robustness, we first conduct PGD-7 attack [6] to bothnatural and adversarial trained networks with different normalizations on CIFAR-10, as shown inthe diagonals of Figure 1 (c) and (d).

Original: Not surprisingly, BN obtains the best robustness compared to othervariants.
Revised: Not surprisingly, BN obtains the best robustness comparedto other variants.

Original: However, we have an intriguing observation after the transferability evaluations amongdifferent normalizations.
Revised: However, we have an intriguing observation after the transferability evaluationsamong different normalizations.

Original: As illustrated in the heatmaps, the adversarial accuracies in most scenariosare around 70% when fed with transferred adversarial examples, while BN only achieves 51 .73%under white-box attack.
Revised: As illustrated in the heatmaps, the adversarial accuracies in mostscenarios are around 70% when fed with transferred adversarial examples, while those under whitebox attack are around 50% .

Original: This huge gap mainly comes from the adversarial transferability amongnormalizations.
Revised: This huge gap mainly comes from the adversarial transferability amongnormalizations.

Original: Motivated by this observation, we first explore the relationship between adversarialtransferability and normalizations, and show that the gradient similarity and loss smoothness arethe key factors of the discrepancy in transferability among different normalizations.
Revised: Motivated by this observation, we first explore the relationship between adversarialtransferability and normalizations, and show that the gradient similarity and loss smoothness arethe key factors of the discrepancy in transferability among different normalizations.

Original: Based on thetheoretical evidence, we propose to aggregate different types of normalizations to form randomspace in the inference phase where the adversarial transferability can be significantly reduced.
Revised: Based on thetheoretical evidence, we propose to aggregate different types of normalizations to form randomspace in the inference phase where the adversarial transferability can be significantly reduced.

Original: Withdesigned random space, the inference phase naturally forms the black-box setting for those attackerswho have access to model parameters due to the colossal random space.
Revised: Withdesigned random space, the inference phase naturally forms the black-box setting for those attackerswho have access to model parameters due to the colossal random space.

Original: Together with the proposedblack-box adversarial training framework, the adversarial robustness is substantially improved withless reduction of natural accuracy.
Revised: Together with the proposedblack-box adversarial training framework, the adversarial robustness is substantially improved withless reduction of natural accuracy.

Original: For example, with the same adversarial training setting, theproposed algorithm improves the natural accuracy by 4 .
Revised: For example, with the same adversarial training setting, theproposed algorithm improves the natural accuracy by 4 .

Original: 83% and the adversarial accuracy by 10 .
Revised: 83% and the adversarial accuracy by 10 .

Original: 41%with ResNet-18 on CIFAR-10 under PGD 20 attack.
Revised: 41%with ResNet-18 on CIFAR-10 under PGD 20 attack.

Original: Our contributions can be summarized as:
Revised: Our contributions can be summarized as:

Original: 1 ) We provide both empirical and theoretical evidence that the upper bound of adversarialtransferability is influenced by the types and parameters of normalization layers.2 )
Revised: 1 ) We provide both empirical and theoretical evidence that the upper bound of adversarialtransferability is influenced by the types and parameters of normalization layers.2 )

Original: We propose a novel Random Normalization Aggregation (RNA) module which replacesthe normalization layers to create huge random space with weak adversarial transferability.
Revised: We propose a novel Random Normalization Aggregation (RNA) module which replacesthe normalization layers to create huge random space with weak adversarial transferability.

Original: Together with a natural black-box adversarial training, RNA boosts the defense ability.3 )
Revised: Together with a natural black-box adversarial training, RNA boosts the defense ability.3 )

Original: We conduct extensive experiments to demonstrate the superiority of RNA on different
Revised: We conduct extensive experiments to demonstrate the superiority of RNA on different

Original: benchmark datasets and networks.
Revised: benchmark datasets and networks.

Original: Different variants and components are also studied.
Revised: Different variants and components are also studied.

Original:  Adversarial Transferability with Different Normalization
Revised:  Adversarial Transferability with Different Normalization

Original: In this section, we reveal the connections between adversarial transferability and normalization layers.
Revised: In this section, we reveal the connections between adversarial transferability and normalization layers.

Original: We first consider a network which is identified with an hypothesis h from a space H .
Revised: We first consider a network which is identified with an hypothesis h from a space H .

Original: The network his optimized with the loss function L on input X and labels Y .
Revised: The network his optimized with the loss function L on input X and labels Y .

Original: The objective is formulated as
Revised: The objective is formulated as

Original: [Equation]
Revised: [Equation]

Original: Given a target network h and inputs { x, y } , the adversarial examples are defined as perturbed input˜ x = x + δ , which makes the network h misclassify through maximizing the classification loss as
Revised: Given a target network h and inputs { x, y } , the adversarial examples are defined as perturbed input˜ x = x + δ , which makes the network h misclassify through maximizing the classification loss as

Original: [Equation]
Revised: [Equation]

Original: where the perturbation δ is constrained by its l p -norm.
Revised: where the perturbation δ is constrained by its l p -norm.

Original: Adversarial transferability denotes an inherentproperty of ˜ X that these adversarial examples can also boost the classification loss L ( h ′ (˜ x ) , y ) of
Revised: Adversarial transferability denotes an inherentproperty of ˜ X that these adversarial examples can also boost the classification loss L ( h ′ (˜ x ) , y ) of

Original: [Figure]
Revised: [Figure]

Original: other networks as well, where h ′ ∈ H .
Revised: other networks as well, where h ′ ∈ H .

Original: We empirically demonstrate that transferability is influencedby the normalization layers in the network h , as shown in Figure 1 (d).
Revised: We empirically demonstrate that transferability is influencedby the normalization layers in the network h , as shown in Figure 1 (d).

Original: For example, taking BN and
Revised: For example, taking BN and

Original: IN as source models to generate adversarial examples, the adversarial accuracies of LN are 71 .
Revised: IN as source models to generate adversarial examples, the adversarial accuracies of LN are 71 .

Original: 29%and 74 .
Revised: 29%and 74 .

Original: 34% respectively.
Revised: 34% respectively.

Original: We further provide more theoretical analysis of their relationships.
Revised: We further provide more theoretical analysis of their relationships.

Original: 2.1 Definition of Normalization Layers
Revised: 2.1 Definition of Normalization Layers

Original: [Figure]
Revised: [Figure]

Original: Batch Normalization is known as important basic module in DNNs, which improvesboth the networkperformance and training stability, and a wide variety of variants are introduced where the activationsare normalized with different dimensions as well as sizes.
Revised: Batch Normalization is known as important basic module in DNNs, which improves the networkperformance, and a wide variety of variants are introduced where the activations are normalized withdifferent dimensions as well as sizes.

Original: To cover most types of normalization, wedivide them into two categories.
Revised: To cover most types of normalization, we divide them intotwo categories.

Original: An illustration is shown in Figure 1 (a) and (b), where LN, GN,and IN compute the mean and variance for each example with different group sizes during inference,while BN and BGN adopt the pre-calculated mini batch statistics which are computed by movingaverage in the training phase.
Revised: An illustration is shown in Figure 1 (a) and (b), where LN, GN, and IN compute themean and variance for each example with different group sizes during inference, while BN and BGNadopt the pre-calculated mini batch statistics which are computed by moving average in the trainingphase.

Original: Note that LN and IN are special cases of GN, which takes the minimumor maximum group number.
Revised: Note that LN and IN are special cases of GN, which takes the minimum or maximum groupnumber.

Original: Likewise, BN is a special case of BGN.
Revised: Likewise, BN is a special case of BGN.

Original: For simplicity, we use GN and BGN to cover all these normalizations.
Revised: For simplicity, we use GN and BGN to cover allthese normalizations.

Original: Considering the activations
Revised: Considering the activations

Original: y ∈ R
Revised: y ∈ R

Original: d × N where N denotesthe batch size and d denotes the number of features, the normalized outputs after BGN with groupnumber of g bgn and those of GN with group number of g gn during inference stage are formulated as
Revised: d × N where N denotes the batch size and ddenotes the number of features, the normalized outputs after BGN with group number of s ( BGN ) andthose of GN with group number of s ( GN ) during inference stage are formulated as

Original: [Equation]
Revised: [Equation]

Original:  + β bgn , for 0 ≤ k ≤ N − 1 , ˆ y ( gn )
Revised: ˆ y ( k ) ( BGN ) ) i + β ( BGN ) , for 1 ≤ k ≤ N, (ˆ y ( k ) ( GN ) ) i = γ ( GN )

Original: (  k ) i =
Revised: ( y ( k ) )

Original: [Equation]
Revised: i − ( µ ( k ) ( GN ) ) i ( σ ( k ) ( GN ) ) i + β ( GN ) , ( z ( k ) ( GN ) ) i = γ ( GN ) ∗ (ˆ y ( k ) ( GN ) )

Original: gn , for 0 ≤ k ≤ N − 1 , where  µ ( k ) gn
Revised: i + β ( GN ) , for 1 ≤ k ≤ N, where ( µ ( k ) ( GN ) ) i = 1 G

Original:  ( i )  = 
Revised: G (cid:88) j = ( y ( k ) ) G ⌊ iG ⌋ + j , ( σ ( k ) ( GN ) ) i = (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Equation]

Revised: G

Revised: G (cid:88)

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: where  µbgn ( i )  and  σbgn ( i )  denote the tracked mean and standard deviation of group ⌊ i · g bgn d ⌋ .
Revised: where G = ⌊ d s ( GN ) ⌋ denotes the group size of GN, ( µ ( BGN ) ) i and ( σ ( BGN ) ) i denote the trackedmean and standard deviation of group ⌊ i · s ( BGN ) d ⌋ .

Original: 2.2 Variation of Loss Function Smoothness
Revised: 2.2 Variation of Loss Function Smoothness

Original: Existing work on adversarial transferability reveals that the adversarial transferability is mainlyinfluenced by the dimensionality of the space of adversarial examples, since the adversarial subspacesof two networks are more likely to intersect with the growth of this dimensionality.
Revised: Existing work on adversarial transferability reveals that the adversarial transferability is mainlyinfluenced by the dimensionality of the space of adversarial examples, since the adversarial subspacesof two networks are more likely to intersect with the growth of this dimensionality.

Original: The sizeof space of adversarial examples can be estimated by the maximum number of orthogonal vectorsr i which are aligned with the gradient g = ∇ X L ( h ( X ) , Y ) .
Revised: The sizeof space of adversarial examples can be estimated by the maximum number of orthogonal vectorsr i which are aligned with the gradient g = ∇ X L ( h ( X ) , Y ) .

Original: In [20], a tight bound is derived asg ⊤ r i ≥ ϵ ∥
Revised: In [20], a tight bound is derived asg ⊤ r i ≥ ϵ ∥

Original: g ∥ 2 √ k where k denotes the maximum number of r i , which implies that the smoothness of lossfunction is inversely proportional to the adversarial transferability.
Revised: g ∥ 2 √ k where k denotes the maximum number of r i , which implies that the smoothness of lossfunction is inversely proportional to the adversarial transferability.

Original: Thus, we analyze the influence ofdifferent normalization layers on the smoothness of loss function, including GN and BGN.
Revised: Thus, we analyze the influence ofdifferent normalization layers on the smoothness of loss function, including GN and BGN.

Original: For simplicity, we dismiss the usage of k in the following equations since the computation of both
Revised: For simplicity, we dismiss the usage of k in the following equations since the computation of both

Original: GN and BGN during inference is independent of batch size.
Revised: GN and BGN during inference is independent of batch size.

Original: We denote the loss with GN as ˆ L gn andthe loss with BGN as ˆ L bgn .
Revised: We denote the loss with GN as ˆ L gn andthe loss with BGN as ˆ L bgn .

Original: Since the mean and variance are computed based on current group forboth GN and BGN, we compute the partial derivative of loss w.r.t.
Revised: Since the mean and variance are computed based on current group forboth GN and BGN, we compute the partial derivative of loss w.r.t.

Original: a group Y j instead of y i where
Revised: a group Y j instead of y i where

Original: Y j = y [  ⌊ i · ggnd ⌋ :  ⌊ i · ggnd ⌋ + ⌊ dggn ⌋ ] .
Revised: Y j = y [ G ⌊ iG ⌋ : G ⌊ iG ⌋ + G ] .

Original: Similarly, Z j denotes the activations of a group after normalization 
Revised: Similarly, Z j denotes the activations of a group after normalization layers.

Original: [Figure]
Revised: [Figure]

Original: layers.

Original: Based on Eq.
Revised: Based on Eq.

Original: 3, the partial derivative of ˆ L gn and ˆ L bgn w.r.t.
Revised: 3, the partial derivative of ˆ L gn and ˆ L bgn w.r.t.

Original: Y j can be given as
Revised: Y j can be given as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ⟨ , ⟩ denotes the inner product, σ gnj denotes the standard deviation of Y j , and σ bgnj denotesthe tracked standard deviation of Y j .
Revised: where ⟨ , ⟩ denotes the inner product, σ gnj denotes the standard deviation of Y j , and σ bgnj denotesthe tracked standard deviation of Y j .

Original: For simplicity, we denote ˆ g = ∂ ˆ L ∂Y j and g =
Revised: For simplicity, we denote ˆ g = ∂ ˆ L ∂Y j and g =

Original: ∂
Revised: ∂

Original: ˆ L ∂Z j .
Revised: ˆ L ∂Z j .

Original: Taking theadvantage of the fact that the mean of Y j is zero and its norm is (cid:112) Ggn , the squared norm of thepartial derivative of GN and BGN can be derived as∥ ˆ g gn ∥ 2 =
Revised: Taking theadvantage of the fact that the mean of Y j is zero and its norm is √ G , the squared norm of the partialderivative of GN and BGN can be derived as

Original: [Equation]
Revised: [Equation]

Original: Besides the smoothness of the loss, we further consider the smoothness of the gradients of the lossfor GN and BGN.
Revised: Besides the smoothness of the loss, we further consider the smoothness of the gradients of the lossfor GN and BGN.

Original: Following [21], we compute the “effective” β -smoothness through the quadraticform of Hessian of the loss w.r.t.
Revised: Following [21], we compute the “effective” β -smoothness through the quadraticform of Hessian of the loss w.r.t.

Original: the group activations in the normalized gradient direction, whichmeasures the change of gradients with perturbations in the gradient direction.
Revised: the group activations in the normalized gradient direction, whichmeasures the change of gradients with perturbations in the gradient direction.

Original: For simplicity, wedenote the hessian w.r.t.
Revised: For simplicity, wedenote the hessian w.r.t.

Original: the layer output as ˆ H =
Revised: the layer output as ˆ H =

Original: ∂ ˆ L ∂Y
Revised: [Equation]

Original:  j ∂Y j , the hessian w.r.t.
Revised: ∂Y j ∂Y j , the hessian w.r.t.

Original: the normalization outputas H =
Revised: the normalization outputas H =

Original: ∂
Revised: ∂

Original: ˆ L ∂Z j ∂Z j , the normalized gradient as ˆ g ′ =
Revised: ˆ L ∂Z j ∂Z j , the normalized gradient as ˆ g ′ =

Original: ˆ g ∥
Revised: ˆ g ∥

Original: ˆ g ∥ and g ′ =
Revised: ˆ g ∥ and g ′ =

Original: g
Revised: g

Original: ∥ g ∥ .
Revised: ∥ g ∥ .

Original: For GN and BGN, we have
Revised: For GN and BGN, we have

Original: [Equation]
Revised: [Equation]

Original: 2.3 Normalization Layers and Adversarial Transferability
Revised: 2.3 Normalization Layers and Adversarial Transferability

Original: The sufficient conditions and the bounds of adversarial transferability between two networks havebeen discussed in [22].
Revised: The sufficient conditions and the bounds of adversarial transferability between two networks havebeen discussed in [22].

Original: We extend this result to the networks with different normalization layers.
Revised: We extend this result to the networks with different normalization layers.

Original: Sincewe focus on the influence of different normalization layers, we assume that these networks share thesame loss function and weight parameters W , which makes ∂ ˆ L gn ∂Z
Revised: Sincewe focus on the influence of different normalization layers, we assume that these networks share thesame loss function and weight parameters W , which makes ∂ ˆ L gn ∂Z

Original: j = ∂ ˆ L bgn ∂Z j and ∂ ˆ L gn ∂Z j ∂Z j = ∂ ˆ L bgn ∂Z j ∂Z j .
Revised: j = ∂ ˆ L bgn ∂Z j and ∂ ˆ L gn ∂Z j ∂Z j = ∂ ˆ L bgn ∂Z j ∂Z j .

Original: Meanwhile, Eq. 5 and Eq.
Revised: Meanwhile, Eq. 5 and Eq.

Original: 6 can be easily generalized to the input x since ∂ ˆ L ∂x =
Revised: 6 can be easily generalized to the input x since ∂ ˆ L ∂x =

Original: ∂ ˆ L ∂Y W .
Revised: ∂ ˆ L ∂Y W .

Original: With thisassumption, the connections between normalization layers and adversarial transferability can beestablished via bounded gradient norm and β -smoothness in Eq.
Revised: With thisassumption, the connections between normalization layers and adversarial transferability can beestablished via bounded gradient norm and β -smoothness in Eq.

Original: 5 and Eq. 6 as
Revised: 5 and Eq. 6 as

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Given two networks h a and h b with different normalization layers, the adversarialperturbation under white-box attack is δ on x with attack target label y A and true label y T .
Revised: Given two networks h a and h b with different normalization layers, the adversarialperturbation under white-box attack is δ on x with attack target label y A and true label y T .

Original: Assumeh a
Revised: Assumeh a

Original: and h b are “effective” β a and β b -smooth respectively, the level of adversarial transferability Tbetween networks h a and h b within the perturbation ball ∥ δ ∥ 2 ≤ ϵ can be upper bounded by
Revised: and h b are “effective” β a and β b -smooth respectively, the level of adversarial transferability Tbetween networks h a and h b within the perturbation ball ∥ δ ∥ 2 ≤ ϵ can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: where T denotes the attack successful rate, R a and R b denotes the empirical risks of network h a andh b , ¯ S denotes the upper loss gradient similarity, min( L ( x, y A )) =
Revised: where T denotes the attack successful rate, R a and R b denotes the empirical risks of network h a andh b , ¯ S denotes the upper loss gradient similarity, min( L ( x, y A )) =

Original: min x ∼X ( L a ( x, y A ) , L b ( x, y ′ )) ,and max( ∥∇ x L∥ ) =
Revised: min x ∼X ( L a ( x, y A ) , L b ( x, y ′ )) ,and max( ∥∇ x L∥ ) =

Original: max x ∼X ,y ∼{ y T ,y A } ( ∥∇ x L a ( x, y ) ∥ , ∥∇ x L b ( x, y ) ∥ ) .
Revised: max x ∼X ,y ∼{ y T ,y A } ( ∥∇ x L a ( x, y ) ∥ , ∥∇ x L b ( x, y ) ∥ ) .

Original: Since the networks share 132 the same loss function and weight parameters, we denote the influence of weight parameters as someconstant C g on gradient norm and C H on gradient smoothness.
Revised: Since the networks share 133 the same loss function and weight parameters, we denote the influence of weight parameters as someconstant C g on gradient norm and C H on gradient smoothness.

Original: The partial derivative and Hessian
Revised: The partial derivative and Hessian

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: Activation
Revised: Activation

Original: Activation[Figure] Inference (b). 
Revised: Activation (b). Random Path Inference (c).

Original: Black-box Adversarial Training
Revised: Black-box Adversarial Training

Original: Attack Path
Revised: Attack Path

Original: White-boxAttack
Revised: White-boxAttack

Original: Adv Input
Revised: Adv Input

Original: Network
Revised: Network

Original: Inference Path
Revised: InferencePath

Original: Black-box Adv Train
Revised: Black-boxAdv Train

Original: Attacker
Revised: Attacker

Original: Random Sample
Revised: Random Sample

Original: Low Transferability Random Sample
Revised: Low Transferability Random Sample

Original: Network Params and All Paths
Revised: Network Params and All Paths

Original: Which Path to Infer?
Revised: Which Path to Infer?

Original: [Figure]
Revised: [Figure]

Original: of loss w.r.t.
Revised: of loss w.r.t.

Original: the normalization output are the same for different normalization, denoted as g and Hrespectively.
Revised: the normalization output are the same for different normalization, denoted as g and Hrespectively.

Original: The gradient norm and β  in Eq.
Revised: The gradient norm, β a , and β b in Eq.

Original: 7 can be bounded as
Revised: 7 can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: 7 and 8, we observe that the upper bound of adversarial transferability is controlledby the gradient magnitude and gradient smoothness, which is further bounded according to the typeand parameters of normalization layers.
Revised: 7 and 8, we observe that the upper bound of adversarial transferability is controlledby the gradient magnitude and gradient smoothness, which is further bounded according to the typeand parameters of normalization layers.

Original: Specifically, given the same γ and σ for GN and BGN, GNachieves a smaller gradient norm and better gradient smoothness than BGN, which decreases theupper bound of adversarial transferability.
Revised: Specifically, given the same γ and σ for GN and BGN, GNachieves a smaller gradient norm and better gradient smoothness than BGN, which decreases theupper bound of adversarial transferability.

Original: Furthermore, the group size Ggn in GN plays an importantrole in smoothness.
Revised: Furthermore, the group size G in GN plays an importantrole in smoothness.

Original: With smaller Ggn , the smoothness of GN increases, and thus the upper bound ofadversarial transferability decreases.
Revised: With smaller G , the smoothness of GN increases, and thus the upper bound ofadversarial transferability decreases.

Original: Similar observations can be found in empirical evidence.
Revised: Similar observations can be found in empirical evidence.

Original: Asshown in Figure 2, the loss landscapes of different normalization layers w.r.t.
Revised: Asshown in Figure 2, the loss landscapes of different normalization layers w.r.t.

Original: input are visualized,which demonstrates that different normalization layers have different smoothness.
Revised: input are visualized,which demonstrates that different normalization layers have different smoothness.

Original: Furthermore, INachieves the best performance in smoothness, which corresponds to the observation in Eq.
Revised: Furthermore, INachieves the best performance in smoothness, which corresponds to the observation in Eq.

Original: 8, since
Revised: 8, since

Original: IN has the minimum group size.
Revised: IN has the minimum group size.

Original: The attack success rate is relatively low when the source model is
Revised: The attack success rate is relatively low when the source model is

Original: IN, as shown in Figure 1 (c) and (d), which corresponds to the observation in Theorem 2.1 that theadversarial transferability decreases when the network is smoother.
Revised: IN, as shown in Figure 1 (c) and (d), which corresponds to the observation in Theorem 2.1 that theadversarial transferability decreases when the network is smoother.

Original:  Random Normalization Aggregation
Revised:  Random Normalization Aggregation

Original: Since the adversarial transferability is strongly correlated with the type of normalization layers, we aska simple question: Can we utilize the bounded adversarial transferability among normalization layersto defense  white-box attacks?
Revised: Since the adversarial transferability is strongly correlated with the type of normalization layers, weask a simple question: Can we utilize the bounded adversarial transferability among normalizationlayers to defense against white-box attacks?

Original: In this work, we propose a Random Normalization Aggregation (RNA)module, which replaces the BN layer in the network.
Revised: In this work, we propose a Random Normalization Aggregation (RNA) module, which replaces the BN layer in the network.

Original: As shown in Figure 4 (a), the normalizationlayers becomes a combination of different normalization sampled from GNs and BGNs, where theunderline denotes the group number.
Revised: As shown in Figure 4(a), the normalization layers becomes a combination of different normalization sampled from GNsand BGNs, where the underline denotes the group number.

Original: Specifically, the network maintains different normalizationlayers while only one normalization is randomly selected for each layer during forwarding, as shownin Figure 4 (b).
Revised: Specifically, the network maintainsdifferent normalization layers while only one normalization is randomly selected for each layerduring forwarding, as shown in Figure 4 (b).

Original: Through incorporating randomization in normalization layers, the network with RNAmodule can be treated as a “supernet” with multiple paths.
Revised: Through incorporating randomization in normalizationlayers, the network with RNA module can be treated as a “supernet” with multiple paths.

Original: Back to white-box defense setting, weassume that the attackers have access to the network parameters including all paths.
Revised: Back towhite-box defense setting, we assume that the attackers have access to the network parameters.

Original: The adversarialexamples are generated through backward on a randomly sampled path, and then fed to anotherrandomly sampled path due to RNA module in the network, which makes the entire white-box attackbecome a “black-box” attack, as illustrated in Figure 4 (c).
Revised: Theadversarial examples are generated through backward on a randomly sampled path, and then fedto another randomly sampled path due to RNA module, which makes the entire white-box attackbecome a “black-box” attack, as illustrated in Figure 4 (c).

Original: Thus, together with the adversarialtransferability study in Section 2, it is natural to create a network with random space in normalizationlayers where the adversarial transferability is significantly constrained.
Revised: Thus, together with the adversarialtransferability study in Section 2, it is natural to create a network with random space in normalizationlayers where the adversarial transferability is significantly constrained.

Original: To achieve a strong defenseagainst adversarial attacks, some concerns still remain: (1).
Revised: To achieve a strong defenseagainst adversarial attacks, some concerns still remain: (1).

Original: The number of paths are required to beextremely large to reduce the probability of sampling the same path with random sampling strategy;(2).
Revised: The number of paths are required to beextremely large to reduce the probability of sampling the same path with random sampling strategy;(2).

Original: The collaboration with traditional adversarial training; (3).
Revised: The collaboration with traditional adversarial training; (3).

Original: The normalization types need to becarefully selected to enforce low adversarial transferability.
Revised: The normalization types need to becarefully selected to enforce low adversarial transferability.

Original: We discussion these concerns as follows.
Revised: We discussion these concerns as follows.

Original: Path Increment in Random Space
Revised: Path Increment in Random Space

Original: The adversarial transferability among different normalizationhas been discussed in Figure 1 (c) and (d).
Revised: The adversarial transferability among different normalizationhas been discussed in Figure 1 (c) and (d).

Original: However, the size of random space also matters for effective
Revised: However, the size of random space also matters for effective

Original: Algorithmwhile not converge do
Revised: Algorithm do

Original: Randomly sample a path p a from P

Original: Initialize adversarialperturbation δ for i← 1 to t doδ = clipϵ [ δ + η · sign ( ∇ x L h p a ) , y )
Revised: Initialize adversarial for i do clip

Original: ; end for

Original: Randomly sample a path p from P ;
Revised: Randomly sample a path p from P ;

Original: W = W − ∇ W L( h ( x + δ ; p) , y ); end while defense against attacks.
Revised: W = W − ∇ W L + δ ; p , y ) while defense against attacks.

Original: If the attackers can sample the same path during attack and inference phaseswith a high probability, the adversarial accuracy will decrease tremendously.
Revised: If the attackers can sample the same path during attack and inference phaseswith a high probability, the adversarial accuracy will decrease tremendously.

Original: To tackle this issue,we introduce layer-wise randomization of RNA module, which randomly samples the normalizationfor each layer in the network.
Revised: To tackle this issue,we introduce layer-wise randomization of RNA module, which randomly samples the normalizationfor each layer in the network.

Original: As shown in Figure 4 (b), different normalization types are sampledfor different layers, which exponentially increases the number of paths.
Revised: As shown in Figure 4 (b), different normalization types are sampledfor different layers, which exponentially increases the number of paths.

Original: Given the n normalizationtypes in RNA and L layers in the network, the size of random space becomes n L , which reduces theprobability of sampling the same path during attack and inference phase to 1 n L << 1 n .
Revised: Given the n normalizationtypes in RNA and L layers in the network, the size of random space becomes n L , which reduces theprobability of sampling the same path during attack and inference phase to 1 n L << 1 n .

Original: Black-box Adversarial Training It is natural to incorporate RNA module into adversarial training.
Revised: Black-box Adversarial Training It is natural to incorporate RNA module into adversarial training.

Original: Consistent with inference phase, we randomly sample a path p a and conduct white-box attack togenerate adversarial examples ˜
Revised: Consistent with inference phase, we randomly sample a path p a and conduct white-box attack togenerate adversarial examples ˜

Original: X p a .
Revised: X p a .

Original: Different from traditional adversarial training which optimizesp a through feeding ˜
Revised: Different from traditional adversarial training which optimizesp a through feeding ˜

Original: X p a , we feed ˜
Revised: X p a , we feed ˜

Original: X p a to another randomly sampled path p , which forms a “black-box”adversarial training, as illustrated in Figure 4 (c).
Revised: X p a to another randomly sampled path p , which forms a “black-box”adversarial training, as illustrated in Figure 4 (c).

Original: Eq.
Revised: Eq.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: 2 can be reformulated ash
Revised: 2 can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: = argmin h ∈H

Original: E x ∼X ,y ∼Y ,p ∼P
Revised: E x,y ∼X , Y [Equation] p ∼P

Original: [ L ( h (˜ x p a ; p ) , y )] , where ˜x p
Revised: [ L ( h (˜ x p a ; p ) , y )] , where ˜

Original: [Equation]
Revised: [Equation]

Original: where P denotes the space of paths.
Revised: where P denotes the space of paths.

Original: The training procedure is shown in Algorithm 1.
Revised: The training procedure is shown in Algorithm 1.

Original: Normalization Types Selection In RNA module
Revised: Normalization Types Selection In RNA module

Original: , multiple normalization types are maintainedto form random space.
Revised: , multiple normalization types are maintainedto form random space.

Original: According to Theorem 2.1, the adversarial transferability is bounded bydifferent components, including gradient similarity, empirical risks, gradient magnitude, and gradientsmoothness.
Revised: According to Theorem 2.1, the adversarial transferability is bounded bydifferent components, including gradient similarity, empirical risks, gradient magnitude, and gradientsmoothness.

Original: We first provide empirical evidence that normalization layers from the same categorydefined in Section 2.1 tend to have higher gradient similarity.
Revised: We first provide empirical evidence that normalization layers from the same categorydefined in Section 2.1 tend to have higher gradient similarity.

Original: As shown in Figure 3, we visualizethe histograms over the cosine similarity of two networks with different normalization layers.
Revised: As shown in Figure 3, we visualizethe histograms over the cosine similarity of two networks with different normalization layers.

Original: Forexample, BN and BGN belong to the same category, and their gradient similarity is much higher thanthat between BN and IN, comparing Figure 3 (a) and (b).
Revised: Forexample, BN and BGN belong to the same category, and their gradient similarity is much higher thanthat between BN and IN, comparing Figure 3 (a) and (b).

Original: Since the gradient similarity is proportionalto the upper bounds of adversarial transferability in Eq.
Revised: Since the gradient similarity is proportionalto the upper bounds of adversarial transferability in Eq.

Original: 7, we propose to select normalization fromdifferent categories.
Revised: 7, we propose to select normalization fromdifferent categories.

Original: Thus, RNA module samples the normalization types from both GN and BGNwith small group sizes in our experiments, while the evaluation of other combinations is also included.
Revised: Thus, RNA module samples the normalization types from both GN and BGNwith small group sizes in our experiments, while the evaluation of other combinations is also included.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Evaluation Setup
Revised: 4.1 Evaluation Setup

Original: CIFAR-10/100 We first conduct experiments on CIFAR-10/100 [23] datasets, which contain 50 Ktraining images and 10 K testing images with size of 32 × 32 from 10/100 categories.
Revised: CIFAR-10/100 We first conduct experiments on CIFAR-10/100 [23] datasets, which contain 50 Ktraining images and 10 K testing images with size of 32 × 32 from 10/100 categories.

Original: The networkswe use are ResNet-18 [23] and WideResNet-32 (WRN) [24].
Revised: The networkswe use are ResNet-18 [23] and WideResNet-32 (WRN) [24].

Original: The SGD optimizer with a momentumof 0 .
Revised: The SGD optimizer with a momentumof 0 .

Original: 9 is used.
Revised: 9 is used.

Original: The weight decay is set to 5 × 10 − 4 .
Revised: The weight decay is set to 5 × 10 − 4 .

Original: The initial learning rate is set to 0 .
Revised: The initial learning rate is set to 0 .

Original: 1 with acosine learning rate scheduler.
Revised: 1 with apiecewise decay learning rate scheduler.

Original: All the baselines are trained with 160 epochs with a batch size of128 .
Revised: All the baselines are trained with 200 epochs with a batchsize of 128 .

Original: The PGD-7 with ϵ = 8 / 255 and step size of 2 / 255 is adopted in the adversarial training setting.
Revised: The PGD-10 with ϵ = 8 / 255 and step size of 2 / 255 is adopted in the adversarial trainingsetting.

Original: For the RNA module, we utilize BN and IN to form the random space in normalization layers.
Revised: For the RNA module, we utilize BN and IN to form the random space in normalization layers.

Original: Theexperiments are performed on one V100 GPU.
Revised: The experiments are performed on one V100 GPU.

Original: ImageNet The effectiveness of proposed RNA is also evaluated on ImageNet [25], which contains1 .
Revised: ImageNet The effectiveness of proposed RNA is also evaluated on ImageNet [25], which contains1 .

Original: 2 M training images and 50 K testing images with size of 224 × 224 from 1000 categories.
Revised: 2 M training images and 50 K testing images with size of 224 × 224 from 1000 categories.

Original: The
Revised: The

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: BGN32 [19] 50.
Revised: BGN32 [19] 50.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: IN [17] [Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: LN
Revised: IN [17] 54.

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: GN32 [16]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: LN

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: RNA

Original: [Table]
Revised: [Table]

Revised: Ours)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: WideResNet
Revised: WideResNet

Revised: BN

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: IN [17]
Revised: IN [17]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: Ours)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: networks we use are ResNet-50 [23].
Revised: networks we use are ResNet-50 [23].

Original: The SGD optimizer with a momentum of 0 .
Revised: The SGD optimizer with a momentum of 0 .

Original: 9 is used.
Revised: 9 is used.

Original: Theweight decay is set to 1 × 10 − 4 .
Revised: Theweight decay is set to 1 × 10 − 4 .

Original: The initial learning rate is set to 0 .
Revised: The initial learning rate is set to 0 .

Original: 02 with a cosine learning ratescheduler.
Revised: 02 with a cosine learning ratescheduler.

Original: We load a pretrained ResNet-50 and then adversarailly train the network for 60 epochswith a batch size of 512 .
Revised: We load a pretrained ResNet-50 and then adversarailly train the network for 60 epochswith a batch size of 512 .

Original: The PGD-2 with ϵ = 4 / 255 is adopted in the adversarial training setting.
Revised: The PGD-2 with ϵ = 4 / 255 is adopted in the adversarial training setting.

Original: For the RNA module, we utilize BGNs and GNs with group size of 1 and 2 to form the random space.
Revised: For the RNA module, we utilize BGNs and GNs with group size of 1 and 2 to form the random space.

Original: The experiments are performed on eight V100 GPUs.
Revised: The experiments are performed on eight V100 GPUs.

Original: Baselines and Attacks Our proposed RNA modules replace the normalization layers in the network.
Revised: Baselines and Attacks Our proposed RNA modules replace the normalization layers in the network.

Original: Thus, various normalization layers are involved for comparison, including BN, IN, LN, GN, and
Revised: Thus, various normalization layers are involved for comparison, including BN, IN, LN, GN, and

Original: BGN [15, 16, 17, 18, 19].
Revised: BGN [15, 16, 17, 18, 19].

Original: On CIFAR-10/100, we evaluate the robustness of all the baselines underdifferent strong attacks from TorchAttacks [26].
Revised: On CIFAR-10/100, we evaluate the robustness of all the baselines underdifferent strong attacks from TorchAttacks [26].

Original: For Fast Gradient Sign Method (FGSM) [4], theperturbation size ϵ is set to 8 / 255 .
Revised: For Fast Gradient Sign Method (FGSM) [4], theperturbation size ϵ is set to 8 / 255 .

Original: For Projected Gradient Descent (PGD) [6], ϵ is set to 8 / 255 withstep size of 2 / 255 , and the steps are set to 20 .
Revised: For Projected Gradient Descent (PGD) [6], ϵ is set to 8 / 255 withstep size of 2 / 255 , and the steps are set to 20 .

Original: For CW attack [27], the steps are set to 1000 withlearning rate of 0 .
Revised: For CW attack [27], the steps are set to 1000 withlearning rate of 0 .

Original: For Momentum variant of Iterative Fast Gradient Sign Method (MIFGSM)[28], ϵ is set to 8 / 255 with a step size of 2 / 255 , and the steps are set to 5 with decay of 1 . 0 .
Revised: For Momentum variant of Iterative Fast Gradient Sign Method (MIFGSM)[28], ϵ is set to 8 / 255 with a step size of 2 / 255 , and the steps are set to 5 with decay of 1 . 0 .

Original: For
Revised: For

Original: DeepFool [29], the steps are set to 50 with overshoot of 0 .
Revised: DeepFool [29], the steps are set to 50 with overshoot of 0 .

Original: For Auto Attack [30], ϵ is set to 8 / 255 .
Revised: For Auto Attack [30], ϵ is set to 8 / 255 .

Original: On ImageNet, we evaluate the robustness of under PGD attacks with ϵ of 4 / 255 and steps of 50 .
Revised: On ImageNet, we evaluate the robustness of under PGD attacks with ϵ of 4 / 255 and steps of 50 .

Original: 4.2 Results for Robustness
Revised: 4.2 Results for Robustness

Original: Main Results We first evaluate the performance of RNA on CIFAR-10 and CIFAR-100 underdifferent types of attacks.
Revised: Main Results We first evaluate the performance of RNA on CIFAR-10 and CIFAR-100 underdifferent types of attacks.

Original: The detailed results are shown in Table 1 and 2.
Revised: The detailed results are shown in Table 1 and 2.

Original: Popular normalizationlayers show similar performance on robustness.
Revised: Popular normalizationlayers show similar performance on robustness.

Original: However, with a random space of different normalization layers, the robustness is significantly improved.
Revised: However, with a random space of different normalization layers, the robustness is significantly improved.

Original: Comparing RNA with other baselines,
Revised: Comparing RNA with other baselines,

Original: RNA consistently achieves the best performance under all attacks, and show strong superiority.
Revised: RNA consistently achieves the best performance under all attacks, and show strong superiority.

Original: Forexample, RNA with ResNet-18 achieves 66 .
Revised: Forexample, RNA with ResNet-18 achieves 66 .

Original: 35% under Auto Attack on CIFAR-10, which is 20 . 97%higher than BN.
Revised: 35% under Auto Attack on CIFAR-10, which is 18 . 66%higher than BN.

Original: Similarly, RNA with WRN achieves 54 .
Revised: Similarly, RNA with WRN achieves 54 .

Original: 48% under DeepFool attacks on CIFAR-100,which is 52 .
Revised: 48% under DeepFool attacks on CIFAR-100,which is 52 .

Original: 50% higher than IN.
Revised: 50% higher than IN.

Original: The boosted adversarial accuracy shows strong empirical evidencethat the constrained adversarial transferability in random space can provide satisfactory defensecapability.
Revised: The boosted adversarial accuracy shows strong empirical evidencethat the constrained adversarial transferability in random space can provide satisfactory defensecapability.

Original: Furthermore, with proposed black-box adversarial training, RNA achieves better natural
Revised: Furthermore, with proposed black-box adversarial training, RNA achieves better natural

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: accuracy.
Revised: accuracy.

Original: For example, RNA with WRN achieves 86 .
Revised: For example, RNA with WRN achieves 86 .

Original: 22% on CIFAR-10, which improves BN by4 .
Revised: 46% on CIFAR-10, which improves BN by1 .

Original: We mainly attribute this improvement to the fact that the generated adversarial examples canbe treated as a “weaker” attack example to other paths during optimization, which naturally achievesbetter trade-offs between natural and adversarial accuracy.
Revised: We mainly attribute this improvement to the fact that the generated adversarial examples canbe treated as a “weaker” attack example to other paths during optimization, which naturally achievesbetter trade-offs between natural and adversarial accuracy.

Original: Stronger PGD Attacks We further evaluate the defense capability of RNA under stronger PGDattacks, which enhance the number of iterations and enlarge the perturbation size.
Revised: Stronger PGD Attacks We further evaluate the defense capability of RNA under stronger PGDattacks, which enhance the number of iterations and enlarge the perturbation size.

Original: The comparisonwith other baselines are shown in Figure 5 (a) and (b).
Revised: The comparisonwith other baselines are shown in Figure 5 (a) and (b).

Original: Comparing RNA with other baselines, RNA isless sensitive to the attack iterations and achieves similar performance in different PGD steps, such as60 .
Revised: Comparing RNA with other baselines, RNA isless sensitive to the attack iterations and achieves similar performance in different PGD steps, such as60 .

Original: 59% on PGD 10 and 60 . 72% on PGD 100 , while the robustness of other normalization types tendto decrease when the PGD step grows to 40 .
Revised: 59% on PGD 10 and 60 . 72% on PGD 100 , while the robustness of other normalization types tendto decrease when the PGD step grows to 40 .

Original: In terms of larger perturbation size, RNA achieves thebest robustness in all the scenarios, and the lowest decrement among all the methods.
Revised: In terms of larger perturbation size, RNA achieves thebest robustness in all the scenarios, and the lowest decrement among all the methods.

Original: Specifically,
Revised: Specifically,

Original: RNA achieves 80 .
Revised: RNA achieves 80 .

Original: 96% with ϵ of 2 / 255 and 24 .
Revised: 96% with ϵ of 2 / 255 and 24 .

Original: 33% with ϵ of 20 / 255 , whose gap is 56 .
Revised: 33% with ϵ of 20 / 255 , whose gap is 56 .

Original: Forcomparison, the gap of BN is 64 .
Revised: Forcomparison, the gap of BN is 64 .

Original: 34% and GN is 61 .
Revised: 34% and GN is 61 .

Original: Comparison with SOTA Defense Methods To demonstrate the superiority of RNA, we includeseveral SOTA defense algorithms for comparison.
Revised: Comparison with SOTA Defense Methods To demonstrate the superiority of RNA, we includeseveral SOTA defense algorithms for comparison.

Original: RobustWRN [31] explores the importance ofnetwork width and depth on robustness.
Revised: RobustWRN [31] explores the importance ofnetwork width and depth on robustness.

Original: AWP [32] proposes to regularize the flatness of weight losslandscape to achieve robustness.
Revised: AWP [32] proposes to regularize the flatness of weight losslandscape to achieve robustness.

Original: RobNet [11] introduce a NAS framework for robustness.
Revised: RobNet [11] introduce a NAS framework for robustness.

Original: RPI+RPT[33] utilizes randomized precision for adversarial defense.
Revised: RPI+RPT[33] utilizes randomized precision for adversarial defense.

Original: SAT [35] proposes to replace ReLU withits smooth approximations, which exhibits robustness.
Revised: SAT [35] proposes to replace ReLU withits smooth approximations, which exhibits robustness.

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: We use WRNon CIFAR-10 and ResNet-50 on ImageNet.
Revised: We use WRNon CIFAR-10 and ResNet-50 on ImageNet.

Original: All the baselines are evaluated under the attack of PGD 20and AutoAttack (AA) on CIFAR-10 and PGD 50 on ImageNet.
Revised: All the baselines are evaluated under the attack of PGD 20and AutoAttack (AA) on CIFAR-10 and PGD 50 on ImageNet.

Original: Our proposed RNA module achievesthe best performance in all the scenarios.
Revised: Our proposed RNA module achievesthe best performance in all the scenarios.

Original: On CIFAR-10, RNA achieves 62 .
Revised: On CIFAR-10, RNA achieves 67 .

Original: 91% under AutoAttack,with 8 .
Revised: 88% under AutoAttack,with 13 .

Original: 87% improvement compared with AWP.
Revised: 84% improvement compared with AWP.

Original: On ImageNet, RNA achieves 54 .
Revised: On ImageNet, RNA achieves 54 .

Original: 61% under PGD 50 ,with 12 . 31% improvement compared with SAT.
Revised: 61% under PGD 50 ,with 12 . 31% improvement compared with SAT.

Original: Note that RNA replaces the normalization layer,which is orthogonal to other defense techniques.
Revised: Note that RNA replaces the normalization layer,which is orthogonal to other defense techniques.

Original: Similarly, the adversarial training in our setting canbe replaced by other advanced training strategies to achieve potentially better performance.
Revised: Similarly, the adversarial training in our setting canbe replaced by other advanced training strategies to achieve potentially better performance.

Original: Adversarial Transferability in Random Space For a better illustration of the adversarial transferability in the random space built from RNA, we conduct the adversarial transferability study of
Revised: Adversarial Transferability in Random Space For a better illustration of the adversarial transferability in the random space built from RNA, we conduct the adversarial transferability study of

Original: ResNet-18 applied with RNA on CIFAR-10.
Revised: ResNet-18 applied with RNA on CIFAR-10.

Original: We first define the path difference as the number ofdifferent normalization layers between attack and inference paths.
Revised: We first define the path difference as the number ofdifferent normalization layers between attack and inference paths.

Original: For example, a path difference of7 denotes two paths selecting different normalization layers in 7 layers during forwarding.
Revised: For example, a path difference of7 denotes two paths selecting different normalization layers in 7 layers during forwarding.

Original: For eachpath difference, we then randomly sample 10 path pairs for transferability evaluation.
Revised: For eachpath difference, we then randomly sample 10 path pairs for transferability evaluation.

Original: We also includedifferent normalization combinations in RNA for comparison.
Revised: We also includedifferent normalization combinations in RNA for comparison.

Original: The results are shown in Figure 5 (c),which illustrates the PGD accuracy under transferred attacks between path pairs along increasing pathdifferences.
Revised: The results are shown in Figure 5 (c),which illustrates the PGD accuracy under transferred attacks between path pairs along increasing pathdifferences.

Original: The filled areas denote the maximum and minimum PGD accuracy.
Revised: The filled areas denote the maximum and minimum PGD accuracy.

Original: It is obvious that the
Revised: It is obvious that the

Original: [Table]
Revised: [Table]

Original: combination of BN and IN achieves the best performance, which also corresponds to the analysisin Theorem 2.1.
Revised: combination of BN and IN achieves the best performance, which also corresponds to the analysisin Theorem 2.1.

Original: With random sampling strategy, the path difference is always around 10 since thenumber of normalization layers is 20 in this network.
Revised: With random sampling strategy, the path difference is always around 10 since thenumber of normalization layers is 20 in this network.

Original: For example, the combination of BN and INachieves an average PGD accuracy of 61 .
Revised: For example, the combination of BN and INachieves an average PGD accuracy of 61 .

Original: 06% when the path difference becomes 10 .
Revised: 06% when the path difference becomes 10 .

Original: Compared withthe baseline BN which achieves 49 .
Revised: Compared withthe baseline BN which achieves 49 .

Original: 38% PGD accuracy, this lower adversarial transferability in ourrandom space brings a strong defense capability against adversarial attacks.
Revised: 38% PGD accuracy, this lower adversarial transferability in ourrandom space brings a strong defense capability against adversarial attacks.

Original: 4.3 Ablation Study
Revised: 4.3 Ablation Study

Original: Different Normalization Combinations We first provide more quantitative results of differentcombinations of normalization layers in RNA module.
Revised: Different Normalization Combinations We first provide more quantitative results of differentcombinations of normalization layers in RNA module.

Original: We conduct comparison with ResNet-18 on
Revised: We conduct comparison with ResNet-18 on

Original: CIFAR-10.
Revised: CIFAR-10.

Original: As shown in Table 4, we evaluate the performance under PGD 20 , DeepFool and Auto
Revised: As shown in Table 4, we evaluate the performance under PGD 20 , DeepFool and Auto

Original: Attack, and the combination of IN and BN achieves the best robustness in all the scenarios.
Revised: Attack, and the combination of IN and BN achieves the best robustness in all the scenarios.

Original: Consistentwith the observation in Theorem 2.1, the combination of LN and BN has slightly worse performancethan that of IN and BN, since IN is a smoother normalization than LN.
Revised: Consistentwith the observation in Theorem 2.1, the combination of LN and BN has slightly worse performancethan that of IN and BN, since IN is a smoother normalization than LN.

Original: Similarly, the combination of
Revised: Similarly, the combination of

Original: GN and LN achieves the worst performance, since LN is a special case of GN so that they have highgradient similarity in our empirical observation, as discussed in Figure 3.
Revised: GN and LN achieves the worst performance, since LN is a special case of GN so that they have highgradient similarity in our empirical observation, as discussed in Figure 3.

Original: Thus, utilizing differentnormalization layers with smaller group size from GN and BGN, RNA module can form randomspace with low adversarial transferability to better improve the defense ability.
Revised: Thus, utilizing differentnormalization layers with smaller group size from GN and BGN, RNA module can form randomspace with low adversarial transferability to better improve the defense ability.

Original: Effectiveness of Different Components We next demonstrate the effectiveness of each componentintroduced in RNA module.
Revised: Effectiveness of Different Components We next demonstrate the effectiveness of each componentintroduced in RNA module.

Original: The detailed results are shown in Table 5 where [.] denotes the range ofgroup size.
Revised: The detailed results are shown in Table 5 where [.] denotes the range ofgroup size.

Original: Besides the normalization combinations, we include more discussion of the random spacedesigning.
Revised: Besides the normalization combinations, we include more discussion of the random spacedesigning.

Original: To form a random space in normalization layers, it is natural to consider a combinationof BN, IN, GN, and LN, however, it is difficult to optimize, as shown in the first row of Table 5.
Revised: To form a random space in normalization layers, it is natural to consider a combinationof BN, IN, GN, and LN, however, it is difficult to optimize, as shown in the first row of Table 5.

Original: With the normalization layers selected from GNs, the optimization becomes stable, however, therobustness is not competitive, as shown in the second row.
Revised: With the normalization layers selected from GNs, the optimization becomes stable, however, therobustness is not competitive, as shown in the second row.

Original: The involvement of black-box adversarialtraining significantly improves the robustness, as shown in the third row.
Revised: The involvement of black-box adversarialtraining significantly improves the robustness, as shown in the third row.

Original: Through expanding therandom space with BGNs, the defense capability is slightly improved due to the doubled number ofpaths.
Revised: Through expanding therandom space with BGNs, the defense capability is slightly improved due to the doubled number ofpaths.

Original: However, the size of random space is still limited.
Revised: However, the size of random space is still limited.

Original: After removing the layer-based constraint,the number of paths exponentially increases, the size of random space for each layer can be reducedfor better trade-offs, as shown in the last 4 rows.
Revised: After removing the layer-based constraint,the number of paths exponentially increases, the size of random space for each layer can be reducedfor better trade-offs, as shown in the last 4 rows.

Original: Comparing BGN+GN[1-1] with GN[1-64], a betterrandom space designing with an appropriate adversarial training strategy can achieve an improvementof 18 .
Revised: Comparing BGN+GN[1-1] with GN[1-64], a betterrandom space designing with an appropriate adversarial training strategy can achieve an improvementof 18 .

Original: 22% under Auto Attack, which demonstrates the necessity of these components.
Revised: 22% under Auto Attack, which demonstrates the necessity of these components.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we explore the importance of normalization layers in adversarial robustness where thetransferability among different normalization layers can be utilized to boost the defense capability.
Revised: In this paper, we explore the importance of normalization layers in adversarial robustness where thetransferability among different normalization layers can be utilized to boost the defense capability.

Original: A Random Normalization Aggregation (RNA) module is proposed to form random space with lowadversarial transferability for defense against adversarial attacks.
Revised: A Random Normalization Aggregation (RNA) module is proposed to form random space with lowadversarial transferability for defense against adversarial attacks.

Original: We provide sufficient empiricalevidence and theoretical analysis to reveal the connections between adversarial transferability andnormalization types, which guides the random space designing.
Revised: We provide sufficient empiricalevidence and theoretical analysis to reveal the connections between adversarial transferability andnormalization types, which guides the random space designing.

Original: With the involvement of black-boxadversarial training strategy and the relaxation of layer-based constraint, the robustness providedby RNA module is significantly strengthened.
Revised: With the involvement of black-boxadversarial training strategy and the relaxation of layer-based constraint, the robustness providedby RNA module is significantly strengthened.

Original: We demonstrate the superiority of RNA module viacomprehensive experiments across different network architectures, attack settings, and benchmarkdatasets.
Revised: We demonstrate the superiority of RNA module viacomprehensive experiments across different network architectures, attack settings, and benchmarkdatasets.

Original: Our work can provide valuable insights into the network module design for robustness.
Revised: Our work can provide valuable insights into the network module design for robustness.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...(

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No]
Revised: [No]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes
Revised: [Yes](d Did you include the total amount

Revised: of

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: compute and the type of resources used (e

Original: g., type
Revised: g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs internal cluster, or cloud provider

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets..

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: Yes]

Original: ) Did you mention the license of the assets?
Revised: ) Did you mention the license of the assets?

Original: [N
Revised: [N/

Revised: [Table]

Original: Did you include any new assets either in the supplemental material or as a URL?
Revised: Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: Did you discuss whether and how consent was obtained from people whose data you’re
Revised: Did you discuss whether and how consent was obtained from people whose data you’re

Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: Did you discuss whether the data you are using/curating contains personally identifiable
Revised: Did you discuss whether the data you are using/curating contains personally identifiable

Revised: [Table]

Original: information or offensive content
Revised: information or offensive content

Original: [N
Revised: [N

Revised: A]

Revised: [Table]

Original:  crowdsourcing or conducted research  human subjectsyou include the
Revised: you crowdsourcing or conducted research with human subjects

Original: text
Revised: [Table]

Original:  instructions  participants and screenshots 
Revised: you include the full text of instructions given to participants and screenshots [Table] [Table]

Original: you describe
Revised: you describe

Original:  participant 
Revised: potential participant risks,

Original:  to Institutional Review
Revised: links to Institutional Review

Revised: [Table]

Original: approvals
Revised: approvals,

Revised: applicable

Revised: [Table]

Original: you include the estimated hourly wage paid to participants and the total amount
Revised: you include the estimated hourly wage paid to participants and the total amount

Revised: [Table]

Original: spent on participant compensation
Revised: spent on participant compensation

Original: [N
Revised: [N

Revised: [Table]