Original: Random Normalization Aggregation for Adversarial Defense
Revised: Random Normalization Aggregation for Adversarial Defense

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The vulnerability of deep neural networks has been widely found in various modelsas well as tasks where slight perturbations on the inputs could lead to incorrectpredictions.
Revised: The vulnerability of deep neural networks has been widely found in various models as well as tasks where slight perturbations on the inputs could lead to incorrect predictions.

Original: These perturbed inputs are known as adversarial examples and one ofthe intriguing properties of them is Adversarial Transfersability , i .
Revised: These perturbed inputs are known as adversarial examples and one of the intriguing properties of them is Adversarial Transfersability , i .

Original: e .
Revised: e .

Original: the capabilityof adversarial examples to fool other models.
Revised: the capability of adversarial examples to fool other models.

Original: Traditionally, this transferabilityis always regarded as a critical threat to the defense against adversarial attacks,however, we argue that the network robustness can be significantly boosted byutilizing adversarial transferability from a new perspective.
Revised: Traditionally, this transferability is always regarded as a critical threat to the defense against adversarial attacks, however, we argue that the network robustness can be significantly boosted by utilizing adversarial transferability from a new perspective.

Original: In this work, we firstdiscuss the influence of different popular normalization layers on the adversarialtransferability, and then provide both empirical evidence and theoretical analysisto shed light on the relationship between normalization types and transferability.
Revised: In this work, we first discuss the influence of different popular normalization layers on the adversarial transferability, and then provide both empirical evidence and theoretical analysis to shed light on the relationship between normalization types and transferability.

Original: Based on our theoretical analysis, we propose a simple yet effective module named
Revised: Based on our theoretical analysis, we propose a simple yet effective module named

Original: Random Normalization Aggregation (RNA) which replaces the batch normalizationlayers in the networks and aggregates different selected normalization types to forma huge random space.
Revised: Random Normalization Aggregation (RNA) which replaces the batch normalization layers in the networks and aggregates different selected normalization types to form a huge random space.

Original: Specifically, a random path is sampled during each inferenceprocedure so that the network itself can be treated as an ensemble of a wide rangeof different models.
Revised: Specifically, a random path is sampled during each inference procedure so that the network itself can be treated as an ensemble of a wide range of different models.

Original: Since the entire random space is designed with low adversarialtransferability, it is difficult to perform effective attacks even when the networkparameters are accessible.
Revised: Since the entire random space is designed with low adversarial transferability, it is difficult to perform effective attacks even when the network parameters are accessible.

Original: We conduct extensive experiments on various modelsand datasets, and demonstrate the strong superiority of proposed algorithm.
Revised: We conduct extensive experiments on various models and datasets, and demonstrate the strong superiority of proposed algorithm.

Revised: The PyTorch code is available at https://github.com/UniSerj/

Revised: Random-Norm-Aggregation and the MindSpore code is available at https: //gitee.com/mindspore/models/tree/master/research/cv/RNA .

Original:  Introduction
Revised:  Introduction

Original: Deep Neural Networks (DNNs) have achieved impressive performance in various tasks [1, 2, 3].
Revised: Deep Neural Networks (DNNs) have achieved impressive performance in various tasks [1, 2, 3].

Original: However, it is well known that DNNs are susceptible to maliciously generated adversarial examples[4, 5].
Revised: However, it is well known that DNNs are susceptible to maliciously generated adversarial examples [4, 5].

Original: Through imperceptible perturbations on the model inputs during inference stage, the model ismisled to wrong predictions at a high rate.
Revised: Through imperceptible perturbations on the model inputs during inference stage, the model is misled to wrong predictions at a high rate.

Original: Since then, a wide range of attack techniques have beenproposed under different settings and show strong attack capability.
Revised: Since then, a wide range of attack techniques have been proposed under different settings and show strong attack capability.

Original: For example, attackers have fullaccess to the model architecture and parameters, which forms white-box attacks [5, 6], and attackershave limited query access to the model, which forms black-box attacks [7, 8].
Revised: For example, attackers have full access to the model architecture and parameters, which forms white-box attacks [5, 6], and attackers have limited query access to the model, which forms black-box attacks [7, 8].

Original: Since the high attacksuccess rates of these techniques reveal the high risk of DNNs, defenses against adversarial exampleshave received increasing attention and adversarial robustness becomes one of the key criteria.
Revised: Since the high attack success rates of these techniques reveal the high risk of DNNs, defenses against adversarial examples have received increasing attention and adversarial robustness becomes one of the key criteria.

Original:  To mitigate this risk, adversarial training is proposed to yield robust models through training ongenerated adversarial examples [5, 6].
Revised: [Figure] To mitigate this risk, adversarial training is proposed to yield robust models through training on generated adversarial examples [5, 6].

Original: Besides training procedure, some regularization and imagepreprocessing techniques are introduced to improve adversarial robustness [9, 10].
Revised: Besides training procedure, some regularization and image preprocessing techniques are introduced to improve adversarial robustness [9, 10].

Original: Recent worknote that the architecture and module designs could play important roles in the robustness [11, 12].
Revised: Recent work note that the architecture and module designs could play important roles in the robustness [11, 12].

Original: Hence, we pay more attention to the basic modules in the network which are seldom considered forimproving robustness, such as normalization layers.
Revised: Hence, we pay more attention to the basic modules in the network which are seldom considered for improving robustness, such as normalization layers.

Original: Existing works have discussed the influence of
Revised: Existing works have discussed the influence of

Original: Batch Normalization (BN) and empirically shown that BN increases adversarial vulnerability and[Figure] decreases adversarial transferability [13, 14].
Revised: Batch Normalization (BN) and empirically shown that BN increases adversarial vulnerability and decreases adversarial transferability [13, 14].

Original: However, the theoretical analysis of this observation isinsufficient and how to tackle this pitfall or even utilize this property to defend attacks is unexplored.
Revised: However, the theoretical analysis of this observation is insufficient and how to tackle this pitfall or even utilize this property to defend attacks is unexplored.

Original: In this work, we take numerous normalizations into consideration, including Layer Normalization(LN), Group Normalization (GN), Instance Normalization (IN), Batch Normalization (BN), and
Revised: In this work, we take numerous normalizations into consideration, including Layer Normalization (LN), Group Normalization (GN), Instance Normalization (IN), Batch Normalization (BN), and

Original: Batch Group Normalization (BGN) [15, 16, 17, 18, 19], as shown in Figure 1 (a) and (b).
Revised: Batch Group Normalization (BGN) [15, 16, 17, 18, 19], as shown in Figure 1 (a) and (b).

Original: To evaluatethe influence of different normalizations on the robustness, we first conduct PGD-7 attack [6] to bothnatural and adversarial trained networks with different normalizations on CIFAR-10, as shown inthe diagonals of Figure 1 (c) and (d).
Revised: To evaluate the influence of different normalizations on the robustness, we first conduct PGD-7 attack [6] to both natural and adversarial trained networks with different normalizations on CIFAR-10, as shown in the diagonals of Figure 1 (c) and (d).

Original: Not surprisingly, BN obtains the best robustness comparedto other variants.
Revised: Not surprisingly, BN obtains the best robustness compared to other variants.

Original: However, we have an intriguing observation after the transferability evaluationsamong different normalizations.
Revised: However, we have an intriguing observation after the transferability evaluations among different normalizations.

Original: As illustrated in the heatmaps, the adversarial accuracies in mostscenarios are around 70% when fed with transferred adversarial examples, while those under whitebox attack are around 50% .
Revised: As illustrated in the heatmaps, the adversarial accuracies in most scenarios are around 70% when fed with transferred adversarial examples, while those under whitebox attack are around 50% .

Original: This huge gap mainly comes from the adversarial transferability amongnormalizations.
Revised: This huge gap mainly comes from the adversarial transferability among normalizations.

Original: Motivated by this observation, we first explore the relationship between adversarialtransferability and normalizations, and show that the gradient similarity and loss smoothness arethe key factors of the discrepancy in transferability among different normalizations.
Revised: Motivated by this observation, we first explore the relationship between adversarial transferability and normalizations, and show that the gradient similarity and loss smoothness are the key factors of the discrepancy in transferability among different normalizations.

Original: Based on thetheoretical evidence, we propose to aggregate different types of normalizations to form randomspace in the inference phase where the adversarial transferability can be significantly reduced.
Revised: Based on the theoretical evidence, we propose to aggregate different types of normalizations to form random space in the inference phase where the adversarial transferability can be significantly reduced.

Original: Withdesigned random space, the inference phase naturally forms the black-box setting for those attackerswho have access to model parameters due to the colossal random space.
Revised: With designed random space, the inference phase naturally forms the black-box setting for those attackers who have access to model parameters due to the colossal random space.

Original: Together with the proposedblack-box adversarial training framework, the adversarial robustness is substantially improved withless reduction of natural accuracy.
Revised: Together with the proposed black-box adversarial training framework, the adversarial robustness is substantially improved with less reduction of natural accuracy.

Original: For example, with the same adversarial training setting, theproposed algorithm improves the natural accuracy by 2 .
Revised: For example, with the same adversarial training setting, the proposed algorithm improves the natural accuracy by 2 .

Original: 45% and the adversarial accuracy by 8 .
Revised: 45% and the adversarial accuracy by 8 .

Original: 53%with ResNet-18 on CIFAR-10 under PGD 20 attack.
Revised: 53% with ResNet-18 on CIFAR-10 under PGD 20 attack.

Original: Our contributions can be summarized as:
Revised: Our contributions can be summarized as:

Original: 1 ) We provide both empirical and theoretical evidence that the upper bound of adversarialtransferability is influenced by the types and parameters of normalization layers.2 )
Revised: 1 ) We provide both empirical and theoretical evidence that the upper bound of adversarial transferability is influenced by the types and parameters of normalization layers.

Original: We propose a novel Random Normalization Aggregation (RNA) module which replacesthe normalization layers to create huge random space with weak adversarial transferability.
Revised: We propose a novel Random Normalization Aggregation (RNA) module which replaces the normalization layers to create huge random space with weak adversarial transferability.

Original: Together with a natural black-box adversarial training, RNA boosts the defense ability.
Revised: Together with a natural black-box adversarial training, RNA boosts the defense ability.

Original: 3 ) We conduct extensive experiments to demonstrate the superiority of RNA on different benchmark datasets and networks.
Revised: We conduct extensive experiments to demonstrate the superiority of RNA on different benchmark datasets and networks.

Original: Different variants and components are also studied.
Revised: Different variants and components are also studied.

Original:  Related Work
Revised:  Related Work

Original: DNNs are vulnerable to adversarial examples and arouse lots of research interests in the attackand defense techniques [4, 5].
Revised: DNNs are vulnerable to adversarial examples and arouse lots of research interests in the attack and defense techniques [4, 5].

Original: Expectation Over Transformation (EOT) is introduced to generatingadversarial examples by computing the gradient over the expected transformation to the input [20].
Revised: Expectation Over Transformation (EOT) is introduced to generating [Figure] adversarial examples by computing the gradient over the expected transformation to the input [20].

Original: Rice et al .
Revised: Rice et al .

Original: [21] explore the overfitting issue in adversarial training and propose to improve therobustness via early-stopping.
Revised: [21] explore the overfitting issue in adversarial training and propose to improve the robustness via early-stopping.

Original: Recently, the influence of DNN basic component on adversarialrobustness has been paid more attention. Xie et al . [22] propose to replace ReLU with its smoothapproximations to enhance robustness.
Revised: Recently, the influence of DNN basic component on adversarial robustness has been paid more attention, such as activation function [22], operation [23], and neural architecture [11, 24].

Original:  Xie et al .
Revised: In terms of BN, Xie et al .

Original: [23] explore the robustness at different networkscales and introduce a mixture of two BN layers which take care of clean and adversarial examplesseparately to improve the trade-offs between clean and adversarial accuracy.
Revised: [25] explore the robustness at different network scales and introduce a mixture of two BN layers which take care of clean and adversarial examples separately to improve the trade-offs between clean and adversarial accuracy.

Original: The mixture of BN canalso improve the generalization of network with adversarial training [24].
Revised: The mixture of BN can also improve the generalization of network with adversarial training [26].

Original: Benz et al .
Revised: Benz et al .

Original: [13] provideempirical evidence that BN increase the adversarial vulnerability.
Revised: [13] provide empirical evidence that BN increase the adversarial vulnerability.

Original: In this paper, we lay emphasis[Figure] on the normalization layers and explore the connections between adversarial robustness and theaggregation of normalization layers to improve defense performance.
Revised: In this paper, we lay emphasis on the normalization layers and explore the connections between adversarial robustness and the aggregation of normalization layers to improve defense performance.

Original:  Adversarial Transferability with Different Normalization
Revised:  Adversarial Transferability with Different Normalization

Original: In this section, we reveal the connections between adversarial transferability and normalization layers.
Revised: In this section, we reveal the connections between adversarial transferability and normalization layers.

Original: We first consider a network which is identified with an hypothesis h from a space H .
Revised: We first consider a network which is identified with an hypothesis h from a space H .

Original: The network his optimized with the loss function L on input X and labels Y .
Revised: The network h is optimized with the loss function L on input X and labels Y .

Original: The objective is formulated as
Revised: The objective is formulated as

Original: [Equation]
Revised: [Equation]

Original: Given a target network h and inputs { x, y } , the adversarial examples are defined as perturbed input˜ x = x + δ , which makes the network h misclassify through maximizing the classification loss as
Revised: Given a target network h and inputs { x, y } , the adversarial examples are defined as perturbed input ˜ x = x + δ , which makes the network h misclassify through maximizing the classification loss as

Original: [Equation]
Revised: [Equation]

Original: where the perturbation δ is constrained by its l p -norm.
Revised: where the perturbation δ is constrained by its l p -norm.

Original: Adversarial transferability denotes an inherentproperty of ˜ X that these adversarial examples can also boost the classification loss L ( h ′ (˜ x ) , y ) ofother networks as well, where h ′ ∈ H .
Revised: Adversarial transferability denotes an inherent property of ˜ X that these adversarial examples can also boost the classification loss L ( h ′ (˜ x ) , y ) of other networks as well, where h ′ ∈ H .

Original: We empirically demonstrate that transferability is influencedby the normalization layers in the network h , as shown in Figure 1 (d).
Revised: We empirically demonstrate that transferability is influenced by the normalization layers in the network h , as shown in Figure 1 (d).

Original: For example, taking BN and
Revised: For example, taking BN and

Original: IN as source models to generate adversarial examples, the adversarial accuracies of LN are 71 .
Revised: IN as source models to generate adversarial examples, the adversarial accuracies of LN are 71 .

Original: 29%and 74 .
Revised: 29% and 74 .

Original: 34% respectively.
Revised: 34% respectively.

Original: We further provide more theoretical analysis of their relationships.
Revised: We further provide more theoretical analysis of their relationships.

Original: 3.1 Definition of Normalization Layers
Revised: 3.1 Definition of Normalization Layers

Original: Batch Normalization is known as important basic module in DNNs, which improves the networkperformance, and a wide variety of variants are introduced where the activations are normalized withdifferent dimensions as well as sizes.
Revised: Batch Normalization is known as important basic module in DNNs, which improves the network performance, and a wide variety of variants are introduced where the activations are normalized with different dimensions as well as sizes.

Original: To cover most types of normalization, we divide them intotwo categories.
Revised: To cover most types of normalization, we divide them into two categories.

Original: An illustration is shown in Figure 1 (a) and (b), where LN, GN, and IN compute themean and variance for each example with different group sizes during inference, while BN and BGNadopt the pre-calculated mini batch statistics which are computed by moving average in the trainingphase.
Revised: An illustration is shown in Figure 1 (a) and (b), where LN, GN, and IN compute the mean and variance for each example with different group sizes during inference, while BN and BGN adopt the pre-calculated mini batch statistics which are computed by moving average in the training phase.

Original: Note that LN and IN are special cases of GN, which takes the minimum or maximum groupnumber.
Revised: Note that LN and IN are special cases of GN, which takes the minimum or maximum group number.

Original: Likewise, BN is a special case of BGN.
Revised: Likewise, BN is a special case of BGN.

Original: For simplicity, we use GN and BGN to cover allthese normalizations.
Revised: For simplicity, we use GN and BGN to cover all these normalizations.

Original: Considering the activations
Revised: Considering the activations

Original: y ∈ R
Revised: y ∈ R

Original: d × N where N denotes the batch size and ddenotes the number of features, the normalized outputs after BGN with group number of s ( BGN ) andthose of GN with group number of s ( GN ) during inference stage are formulated as
Revised: d × N where N denotes the batch size and d denotes the number of features, the normalized outputs after BGN with group number of s ( BGN ) and those of GN with group number of s ( GN ) during inference stage are formulated as

Original: [Equation]
Revised: [Equation]

Original:  GN [Equation]
Revised: where ( µ ( k ) ( GN ) )

Original: ˆ y
Revised: i = 1 G G (cid:88)

Original: [Equation]
Revised: [Equation]

Original: k )
Revised: σ (

Original: [Equation]
Revised: [Equation]

Original:  GN
Revised: ( GN

Original: [Equation]
Revised: [Equation]

Original: β ( GN ) , for 1 ≤ k ≤ N, where ( µ ( k ) ( GN ) )

Original:  i =1 G
Revised: ) i =

Original: [Equation]
Revised: [Equation]

Original:  where G =
Revised: (3) where G =

Original: ⌊ d s ( GN ) ⌋ denotes the group size of GN, ( µ ( BGN ) )
Revised: ⌊ d s ( GN ) ⌋ denotes the group size of GN, ( µ ( BGN ) )

Original: i and ( σ ( BGN ) )
Revised: i and ( σ ( BGN ) )

Original: i denote the trackedmean and standard deviation of group ⌊ i · s ( BGN ) d ⌋ .
Revised: i denote the tracked mean and standard deviation of group ⌊ i · s ( BGN ) d ⌋ .

Original: [Figure] [Figure]
Revised: 3.2 Variation of Loss Function Smoothness

Original: Existing work on adversarial transferability reveals that the adversarial transferability is mainlyinfluenced by the dimensionality of the space of adversarial examples, since the adversarial subspacesof two networks are more likely to intersect with the growth of this dimensionality.
Revised: Existing work on adversarial transferability reveals that the adversarial transferability is mainly influenced by the dimensionality of the space of adversarial examples, since the adversarial subspaces of two networks are more likely to intersect with the growth of this dimensionality.

Original: The sizeof space of adversarial examples can be estimated by the maximum number of orthogonal vectorsr i which are aligned with the gradient g = ∇ X L ( h ( X ) , Y ) .
Revised: The size of space of adversarial examples can be estimated by the maximum number of orthogonal vectors r i which are aligned with the gradient g = ∇ X L ( h ( X ) , Y ) .

Original: In [25], a tight bound is derived asg ⊤ r i ≥ ϵ ∥
Revised: In [27], a tight bound is derived as g ⊤ r i ≥ ϵ ∥

Original: g ∥ 2 √ k where k denotes the maximum number of r i , which implies that the smoothness of lossfunction is inversely proportional to the adversarial transferability.
Revised: g ∥ 2 √ k where k denotes the maximum number of r i , which implies that the smoothness of loss function is inversely proportional to the adversarial transferability.

Original: Thus, we analyze the influence ofdifferent normalization layers on the smoothness of loss function, including GN and BGN.
Revised: Thus, we analyze the influence of different normalization layers on the smoothness of loss function, including GN and BGN.

Original: For simplicity, we dismiss the usage of k in the following equations since the computation of both
Revised: For simplicity, we dismiss the usage of k in the following equations since the computation of both

Original: GN and BGN during inference is independent of batch size.
Revised: GN and BGN during inference is independent of batch size.

Original: We denote the loss with GN as ˆ L gn andthe loss with BGN as ˆ L bgn .
Revised: We denote the loss with GN as ˆ L gn and the loss with BGN as ˆ L bgn .

Original: Since the mean and variance are computed based on current group forboth GN and BGN, we compute the partial derivative of loss w.r.t.
Revised: Since the mean and variance are computed based on current group for both GN and BGN, we compute the partial derivative of loss w.r.t.

Original: a group Y j instead of y i where
Revised: a group Y j instead of y i where

Original: Y j = y [ G ⌊ iG ⌋ : G ⌊ iG ⌋ + G ] .
Revised: Y j = y [ G ⌊ iG ⌋ : G ⌊ iG ⌋ + G ] .

Original: Similarly, Z j denotes the activations of a group after normalization layers.
Revised: Similarly, Z j denotes the activations of a group after normalization layers.

Original: Based on Eq.
Revised: Based on Eq.

Original: 3, the partial derivative of ˆ L gn and ˆ L bgn w.r.t.
Revised: 3, the partial derivative of ˆ L gn and ˆ L bgn w.r.t.

Original: Y j can be given as
Revised: Y j can be given as

Original: [Equation]
Revised: [Equation]

Original: where ⟨ , ⟩ denotes the inner product, σ gnj denotes the standard deviation of Y j , and σ bgnj denotesthe tracked standard deviation of Y j .
Revised: where ⟨ , ⟩ denotes the inner product, σ gnj denotes the standard deviation of Y j , and σ bgnj denotes the tracked standard deviation of Y j .

Original: For simplicity, we denote ˆ g = ∂ ˆ L ∂Y j and g =
Revised: For simplicity, we denote ˆ g = ∂ ˆ L ∂Y j and g =

Original: ∂
Revised: ∂

Original: ˆ L ∂Z j .
Revised: ˆ L ∂Z j .

Original: Taking theadvantage of the fact that the mean of Y j is zero and its norm is √ G , the squared norm of the partialderivative of GN and BGN can be derived as
Revised: Taking the advantage of the fact that the mean of Y j is zero and its norm is √ G , the squared norm of the partial derivative of GN and BGN can be derived as

Original: [Equation]
Revised: [Equation]

Original: Besides the smoothness of the loss, we further consider the smoothness of the gradients of the lossfor GN and BGN.
Revised: Besides the smoothness of the loss, we further consider the smoothness of the gradients of the loss for GN and BGN.

Original: Following [26], we compute the “effective” β -smoothness through the quadraticform of Hessian of the loss w.r.t.
Revised: Following [28], we compute the “effective” β -smoothness through the quadratic form of Hessian of the loss w.r.t.

Original: the group activations in the normalized gradient direction, whichmeasures the change of gradients with perturbations in the gradient direction.
Revised: the group activations in the normalized gradient direction, which measures the change of gradients with perturbations in the gradient direction.

Original: For simplicity, wedenote the hessian w.r.t.
Revised: For simplicity, we denote the hessian w.r.t.

Original: the layer output as ˆ H =
Revised: the layer output as ˆ H =

Original: ∂
Revised: ∂

Original: ˆ L ∂Y
Revised: ˆ L ∂Y

Original: j ∂Y j , the hessian w.r.t.
Revised: j ∂Y j , the hessian w.r.t.

Original: the normalization outputas H = ∂
Revised: the normalization output as H = ∂

Original: ˆ L ∂Z j ∂Z j , the normalized gradient as ˆ g ′ =
Revised: [Equation] j , the normalized gradient as ˆ g ′ =

Original: ˆ g ∥
Revised: ˆ g ∥

Original: ˆ g ∥ and g ′ =
Revised: ˆ g ∥ and g ′ =

Original: g
Revised: g

Original: ∥ g ∥ .
Revised: ∥ g ∥ .

Original: For GN and BGN, we have
Revised: For GN and BGN, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]
Revised: [Figure]

Original: 3.3 Normalization Layers and Adversarial Transferability
Revised: 3.3 Normalization Layers and Adversarial Transferability

Original: The sufficient conditions and the bounds of adversarial transferability between two networks havebeen discussed in [27].
Revised: The sufficient conditions and the bounds of adversarial transferability between two networks have been discussed in [29].

Original: We extend this result to the networks with different normalization layers.
Revised: We extend this result to the networks with different normalization layers.

Original: Sincewe focus on the influence of different normalization layers, we assume that these networks share thesame loss function and weight parameters W , which makes ∂ ˆ L gn ∂Z
Revised: Since we focus on the influence of different normalization layers, we assume that these networks share the same loss function and weight parameters W , which makes ∂ ˆ L gn ∂Z

Original: j = ∂ ˆ L bgn ∂Z j and ∂ ˆ L gn ∂Z j ∂Z j = ∂ ˆ L bgn ∂Z j ∂Z j .[Figure] c) [Figure]
Revised: j = ∂ ˆ L bgn ∂Z j and ∂ ˆ L gn ∂Z j ∂Z j = ∂ ˆ L bgn ∂Z j ∂Z j .

Original: Adversarial Training

Original: Attack Path

Original: White-

Original: [Figure]

Original: Attack

Original: Adv Input

Original: Network

Original: InferencePath

Original: Black

Original: [Figure]

Original: Adv Train

Original: Attacker

Original: Random Sample

Original: Low Transferability Random Sample

Original: [Figure]

Original: Paths

Original: Which

Original: [Figure]

Original: Infer?

Original: [Figure]

Original: Meanwhile, Eq. 5 and Eq.
Revised: Meanwhile, Eq. 5 and Eq.

Original: 6 can be easily generalized to the input x since ∂ ˆ L ∂x =
Revised: 6 can be easily generalized to the input x since ∂ ˆ L ∂x =

Original: ∂ ˆ L ∂Y W .
Revised: ∂ ˆ L ∂Y W .

Original: With this assumption, the connections between normalization layers and adversarial transferability can beestablished via bounded gradient norm and β -smoothness in Eq.
Revised: With this assumption, the connections between normalization layers and adversarial transferability can be established via bounded gradient norm and β -smoothness in Eq.

Original: 5 and Eq. 6 as
Revised: 5 and Eq. 6 as

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Given two networks h a and h b with different normalization layers, the adversarialperturbation under white-box attack is δ on x with attack target label y A and true label y T .
Revised: Given two networks h a and h b with different normalization layers, the adversarial perturbation under white-box attack is δ on x with attack target label y A and true label y T .

Original: Assumeh a and h b are “effective” β a and β b -smooth respectively, the level of adversarial transferability Tbetween networks h a and h b within the perturbation ball ∥ δ ∥ 2 ≤ ϵ can be upper bounded by T ≤
Revised: Assume h a and h b are “effective” β a and β b -smooth respectively, the level of adversarial transferability T between networks h a and h b within the perturbation ball ∥ δ ∥ 2 ≤ ϵ can be upper bounded by [Equation]

Original: [Equation]
Revised: [Equation]

Original: where T denotes the attack successful rate, R a and R b denotes the empirical risks of network h a andh b , ¯ S denotes the upper loss gradient similarity, min( L ( x, y A ))=
Revised: where T denotes the attack successful rate, R a and R b denotes the empirical risks of network h a and h b , ¯ S denotes the upper loss gradient similarity, min( L ( x, y A ))

Original:  min x ∼X ( L a ( x, y A ) , L b ( x, y ′ )) ,and max( ∥∇ x L∥ ) =
Revised: = min x ∼X ( L a ( x, y A ) , L b ( x, y ′ )) , and max( ∥∇ x L∥ ) =

Original: max x ∼X ,y ∼{ y T ,y A } ( ∥∇ x L a ( x, y ) ∥ , ∥∇ x L b ( x, y ) ∥ ) .
Revised: max x ∼X ,y ∼{ y T ,y A } ( ∥∇ x L [Equation] ( x, y ) ∥ , ∥∇ x L b ( x

Original:  Since the networks share147 the same loss function and weight parameters, we denote the influence of weight parameters as someconstant C g on gradient norm and C H on gradient smoothness.
Revised: [Equation] y ) ∥ ) . Since the networks share the same loss function and weight parameters, we denote the influence of weight parameters as some constant C g on gradient norm and C H on gradient smoothness.

Original: The partial derivative and Hessianof loss w.r.t.
Revised: The partial derivative and Hessian of loss w.r.t.

Original: the normalization output are the same for different normalization, denoted as g and Hrespectively.
Revised: the normalization output are the same for different normalization, denoted as g and H respectively.

Original: The gradient norm, β a , and β b in Eq.
Revised: The gradient norm, β a , and β b in Eq.

Original: 7 can be bounded as
Revised: 7 can be bounded as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: 7 and 8, we observe that the upper bound of adversarial transferability is controlledby the gradient magnitude and gradient smoothness, which is further bounded according to the typeand parameters of normalization layers.
Revised: 7 and 8, we observe that the upper bound of adversarial transferability is controlled by the gradient magnitude and gradient smoothness, which is further bounded according to the type and parameters of normalization layers.

Original: Specifically, given the same γ and σ for GN and BGN, GNachieves a smaller gradient norm and better gradient smoothness than BGN, which decreases theupper bound of adversarial transferability.
Revised: Specifically, given the same γ and σ for GN and BGN, GN achieves a smaller gradient norm and better gradient smoothness than BGN, which decreases the upper bound of adversarial transferability.

Original: Furthermore, the group size G in GN plays an importantrole in smoothness.
Revised: Furthermore, the group size G in GN plays an important role in smoothness.

Original: With smaller G , the smoothness of GN increases, and thus the upper bound ofadversarial transferability decreases.
Revised: With smaller G , the smoothness of GN increases, and thus the upper bound of adversarial transferability decreases.

Original: Similar observations can be found in empirical evidence.
Revised: Similar observations can be found in empirical evidence.

Original: Asshown in Figure 2, the loss landscapes of different normalization layers w.r.t.
Revised: As shown in Figure 2, the loss landscapes of different normalization layers w.r.t.

Original: input are visualized,which demonstrates that different normalization layers have different smoothness.
Revised: input are visualized, which demonstrates that different normalization layers have different smoothness.

Original: Furthermore, INachieves the best performance in smoothness, which corresponds to the observation in Eq.
Revised: Furthermore, IN achieves the best performance in smoothness, which corresponds to the observation in Eq.

Original: 8, since
Revised: 8, since

Original: IN has the minimum group size.
Revised: IN has the minimum group size.

Original: The attack success rate is relatively low when the source model is
Revised: The attack success rate is relatively low when the source model is

Original: IN, as shown in Figure 1 (c) and (d), which corresponds to the observation in Theorem 3.1 that theadversarial transferability decreases when the network is smoother. 
Revised: IN, as shown in Figure 1 (c) and (d), which corresponds to the observation in Theorem 3.1 that the adversarial transferability decreases when the network is smoother. [Figure] Activation [Figure]

Original: Random Normalization Aggregation
Revised: [Figure] Conv BGN_ Activation

Revised: Activation (b).

Revised: Random Path Inference (c).

Revised: Black-box Adversarial Training

Revised: Attack Path

Revised: White-boxAttack

Revised: Adv Input

Revised: Network

Revised: InferencePath

Revised: Black-boxAdv Train

Original: Since the adversarial transferability is strongly correlated with the type of normalization layers, weask a simple question:
Revised: Attacker Random Sample Low Transferability Random Sample Network Params and All Paths

Original:  Can we utilize the bounded adversarial transferability among normalizationlayers to defense against white-box attacks?
Revised: Which Path to Infer? [Figure]  Random Normalization Aggregation Since the adversarial transferability is strongly correlated with the type of normalization layers, we ask a simple question: Can we utilize the bounded adversarial transferability among normalization layers to defense against white-box attacks?

Original: In this work, we propose a Random Normalization
Revised: In this work, we propose a Random Normalization

Original: Aggregation (RNA) module, which replaces the BN layer in the network.
Revised: Aggregation (RNA) module, which replaces the BN layer in the network.

Original: As shown in Figure 4(a), the normalization layers becomes a combination of different normalization sampled from GNsand BGNs, where the underline denotes the group number.
Revised: As shown in Figure(a), the normalization layers becomes a combination of different normalization sampled from GNs and BGNs, where the underline denotes the group number.

Original: Specifically, the network maintains different normalization layers while only one normalization is randomly selected for each layerduring forwarding, as shown in Figure 4 (b).
Revised: Specifically, the network maintains different normalization layers while only one normalization is randomly selected for each layer during forwarding, as shown in Figure 4 (b).

Original: Through incorporating randomization in normalizationlayers, the network with RNA module can be treated as a “supernet” with multiple paths.
Revised: Through incorporating randomization in normalization layers, the network with RNA module can be treated as a “supernet” with multiple paths.

Original: Back towhite-box defense setting, we assume that the attackers have access to the network parameters.
Revised: Back to white-box defense setting, we assume that the attackers have access to the network parameters.

Original: Theadversarial examples are generated through backward on a randomly sampled path, and then fedto another randomly sampled path due to RNA module, which makes the entire white-box attackbecome a “black-box” attack, as illustrated in Figure 4 (c).
Revised: The adversarial examples are generated through backward on a randomly sampled path, and then fed to another randomly sampled path due to RNA module, which makes the entire white-box attack become a “black-box” attack, as illustrated in Figure 4 (c).

Original: Thus, together with the adversarialtransferability study in Section 3, it is natural to create a network with random space in normalizationlayers where the adversarial transferability is significantly constrained.
Revised: Thus, together with the adversarial transferability study in Section 3, it is natural to create a network with random space in normalization layers where the adversarial transferability is significantly constrained.

Original: To achieve a strong defenseagainst adversarial attacks, some concerns still remain: (1).
Revised: To achieve a strong defense against adversarial attacks, some concerns still remain: (1).

Original: The number of paths are required to beextremely large to reduce the probability of sampling the same path with random sampling strategy;(2).
Revised: The number of paths are required to be extremely large to reduce the probability of sampling the same path with random sampling strategy; (2).

Original: The collaboration with traditional adversarial training; (3).
Revised: The collaboration with traditional adversarial training; (3).

Original: The normalization types need to becarefully selected to enforce low adversarial transferability.
Revised: The normalization types need to be carefully selected to enforce low adversarial transferability.

Original: We discussion these concerns as follows.
Revised: We discussion these concerns as follows.

Original: Path Increment in Random Space
Revised: Path Increment in Random Space

Original: The adversarial transferability among different normalizationhas been discussed in Figure 1 (c) and (d).
Revised: The adversarial transferability among different normalization has been discussed in Figure 1 (c) and (d).

Original: However, the size of random space also matters for effectivedefense against attacks.
Revised: However, the size of random space also matters for effective defense against attacks.

Original: If the attackers can sample the same path during attack and inference phaseswith a high probability, the adversarial accuracy will decrease tremendously.
Revised: If the attackers can sample the same path during attack and inference phases with a high probability, the adversarial accuracy will decrease tremendously.

Original: To tackle this issue,we introduce layer-wise randomization of RNA module, which randomly samples the normalizationfor each layer in the network.
Revised: To tackle this issue, we introduce layer-wise randomization of RNA module, which randomly samples the normalization for each layer in the network.

Original: As shown in Figure 4 (b), different normalization types are sampledfor different layers, which exponentially increases the number of paths.
Revised: As shown in Figure 4 (b), different normalization types are sampled for different layers, which exponentially increases the number of paths.

Original: Given the n normalizationtypes in RNA and L layers in the network, the size of random space becomes n L , which reduces theprobability of sampling the same path during attack and inference phase to 1 n L << 1 n .
Revised: Given the n normalization types in RNA and L layers in the network, the size of random space becomes n L , which reduces the probability of sampling the same path during attack and inference phase to 1 n L << 1 n .

Original: Black-box Adversarial Training It is natural to incorporate RNA module into adversarial training.
Revised: Black-box Adversarial Training It is natural to incorporate RNA module into adversarial training.

Original: Consistent with inference phase, we randomly sample a path p a and conduct white-box attack togenerate adversarial examples ˜
Revised: Consistent with inference phase, we randomly sample a path p a and conduct white-box attack to generate adversarial examples ˜

Original: X p a .
Revised: X p a .

Original: Different from traditional adversarial training which optimizesp a through feeding ˜
Revised: Different from traditional adversarial training which optimizes p a through feeding ˜

Original: X p a , we feed ˜
Revised: X p a , we feed ˜

Original: X p a to another randomly sampled path p , which forms a “black-box”adversarial training, as illustrated in Figure 4 (c).
Revised: X p a to another randomly sampled path p , which forms a “black-box” adversarial training, as illustrated in Figure 4 (c).

Original: Eq.
Revised: Eq.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: 2 can be reformulated ash
Revised: 2 can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: argmin

Original: [Equation]

Original: E

Original: [Equation]

Original: [Equation]

Original: ∼X , Y [Equation] p ∼P [ L
Revised: ] , where ˜

Original: [Equation]

Original: h

Original: [Equation]
Revised: [Equation]

Original: p ) , y )] , where ˜ x p

Original: [Equation]
Revised: [Equation]

Original: where P denotes the space of paths.
Revised: where P denotes the space of paths.

Revised: The training procedure is shown in Algorithm 1.

Original: Normalization Types Selection In RNA module , multiple normalization types are maintainedto form random space.
Revised: Normalization Types Selection In RNA module, multiple normalization types are maintained to form random space.

Original: According to Theorem 3.1, the adversarial transferability is bounded bydifferent components, including gradient similarity, empirical risks, gradient magnitude, and gradientsmoothness.
Revised: According to Theorem 3.1, the adversarial transferability is bounded by different components, including gradient similarity, empirical risks, gradient magnitude, and gradient smoothness.

Original: We first provide empirical evidence that normalization layers from the same categorydefined in Section 3.1 tend to have higher gradient similarity.
Revised: We first provide empirical evidence that normalization layers from the same category defined in Section 3.1 tend to have higher gradient similarity.

Original: As shown in Figure 3, we visualizethe histograms over the cosine similarity of two networks with different normalization layers.
Revised: As shown in Figure 3, we visualize the histograms over the cosine similarity of two networks with different normalization layers.

Original: Forexample, BN and BGN belong to the same category, and their gradient similarity is much higher thanthat between BN and IN, comparing Figure 3 (a) and (b).
Revised: For example, BN and BGN belong to the same category, and their gradient similarity is much higher than that between BN and IN, comparing Figure 3 (a) and (b).

Original: Since the gradient similarity is proportionalto the upper bounds of adversarial transferability in Eq.
Revised: Since the gradient similarity is proportional to the upper bounds of adversarial transferability in Eq.

Original: 7, we propose to select normalization fromdifferent categories.
Revised: 7, we propose to select normalization from different categories.

Original: Thus, RNA module samples the normalization types from both GN and BGNwith small group sizes in our experiments, while the evaluation of other combinations is also included.
Revised: Thus, RNA module samples the normalization types from both GN and BGN with small group sizes in our experiments, while the evaluation of other combinations is also included.

Original:  Experiments
Revised:  Experiments

Revised: In this section, we provide sufficient evaluation of RNA module on various models and datasets.

Original: 5.1 Evaluation Setup
Revised: 5.1 Evaluation Setup

Original: CIFAR-10/100 We first conduct experiments on CIFAR-10/100 [28] datasets, which contain 50 Ktraining images and 10 K testing images with size of 32 × 32 from 10/100 categories.
Revised: CIFAR-10/100 We first conduct experiments on CIFAR-10/100 [30] datasets, which contain 50 K training images and 10 K testing images with size of 32 × 32 from 10/100 categories.

Original: The networkswe use are ResNet-18 [28] and WideResNet-32 (WRN) [29].
Revised: The networks we use are ResNet-18 [30] and WideResNet-32 (WRN) [31].

Original: The SGD optimizer with a momentumof 0 .
Revised: The SGD optimizer with a momentum of 0 .

Original: 9 is used.
Revised: 9 is used.

Original: The weight decay is set to 5 × 10 − 4 .
Revised: The weight decay is set to 5 × 10 − 4 .

Original: The initial learning rate is set to 0 .
Revised: The initial learning rate is set to 0 .

Original: 1 with apiecewise decay learning rate scheduler.
Revised: 1 with a piecewise decay learning rate scheduler.

Original: All the baselines are trained with 200 epochs with a batchsize of 128 .
Revised: All the baselines are trained with 200 epochs with a batch size of 128 .

Original: The PGD-10 with ϵ = 8 / 255 and step size of 2 / 255 is adopted in the adversarial trainingsetting.
Revised: The PGD-10 with ϵ = 8 / 255 and step size of 2 / 255 is adopted in the adversarial training setting.

Original: For the RNA module, we utilize BN and IN to form the random space in normalization layers.
Revised: For the RNA module, we utilize BN and IN to form the random space in normalization layers.

Original: The experiments are performed on one V100 GPU. ImageNet
Revised: The experiments are performed on one V100 GPU using Pytorch [32] and Mindspore [33].

Original:  The effectiveness of proposed RNA is also evaluated on ImageNet [30], which contains1 . 2 M training images and 50 K testing images with size of 224 × 224 from 1000 categories.The [Table]
Revised: ImageNet The effectiveness of proposed RNA is also evaluated on ImageNet [34], which contains 1 . 2 M training images and 50 K testing images with size of 224 × 224 from 1000 categories.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: IN

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] networks we use are ResNet-50 [28].
Revised: The networks we use are ResNet-50 [30].

Original: The SGD optimizer with a momentum of 0 .
Revised: The SGD optimizer with a momentum of 0 .

Original: 9 is used.
Revised: 9 is used.

Original: Theweight decay is set to 1 × 10 − 4 .
Revised: The weight decay is set to 1 × 10 − 4 .

Original: The initial learning rate is set to 0 .
Revised: The initial learning rate is set to 0 .

Original: 02 with a cosine learning ratescheduler.
Revised: 02 with a cosine learning rate scheduler.

Original: We load a pretrained ResNet-50 and then adversarailly train the network for 60 epochswith a batch size of 512 .
Revised: We load a pretrained ResNet-50 and then adversarailly train the network for 60 epochs with a batch size of 512 .

Original: The PGD-2 with ϵ = 4 / 255 is adopted in the adversarial training setting.
Revised: The PGD-2 with ϵ = 4 / 255 is adopted in the adversarial training setting.

Original: For the RNA module, we utilize BGNs and GNs with group size of 1 and 2 to form the random space.
Revised: For the RNA module, we utilize BGNs and GNs with group size of 1 and 2 to form the random space.

Original: The experiments are performed on eight V100 GPUs.
Revised: The experiments are performed on eight V100 GPUs.

Original: Baselines and Attacks Our proposed RNA modules replace the normalization layers in the network.
Revised: Baselines and Attacks Our proposed RNA modules replace the normalization layers in the network.

Original: Thus, various normalization layers are involved for comparison, including BN, IN, LN, GN, and
Revised: Thus, various normalization layers are involved for comparison, including BN, IN, LN, GN, and

Original: BGN [15, 16, 17, 18, 19].
Revised: BGN [15, 16, 17, 18, 19].

Original: On CIFAR-10/100, we evaluate the robustness of all the baselines underdifferent strong attacks from TorchAttacks [31].
Revised: On CIFAR-10/100, we evaluate the robustness of all the baselines under different strong attacks from TorchAttacks [35].

Original: For Fast Gradient Sign Method (FGSM) [4], theperturbation size ϵ is set to 8 / 255 .
Revised: For Fast Gradient Sign Method (FGSM) [4], the perturbation size ϵ is set to 8 / 255 .

Original: For Projected Gradient Descent (PGD) [6], ϵ is set to 8 / 255 withstep size of 2 / 255 , and the steps are set to 20 .
Revised: For Projected Gradient Descent (PGD) [6], ϵ is set to 8 / 255 with step size of 2 / 255 , and the steps are set to 20 .

Original: For CW attack [32], the steps are set to 1000 withlearning rate of 0 .
Revised: For CW attack [36], the steps are set to 1000 with learning rate of 0 .

Original: For Momentum variant of Iterative Fast Gradient Sign Method (MIFGSM)[33], ϵ is set to 8 / 255 with a step size of 2 / 255 , and the steps are set to 5 with decay of 1 . 0 .
Revised: For Momentum variant of Iterative Fast Gradient Sign Method (MIFGSM) [37], ϵ is set to 8 / 255 with a step size of 2 / 255 , and the steps are set to 5 with decay of 1 . 0 .

Original: For
Revised: For

Original: DeepFool [34], the steps are set to 50 with overshoot of 0 .
Revised: DeepFool [38], the steps are set to 50 with overshoot of 0 .

Original: For Auto Attack [35], ϵ is set to 8 / 255 .
Revised: For Auto Attack [39], ϵ is set to 8 / 255 .

Original: On ImageNet, we evaluate the robustness of under PGD attacks with ϵ of 4 / 255 and steps of 50 . 
Revised: On ImageNet, we evaluate the robustness of under PGD attacks with ϵ of 4 / 255 and steps of 50 . [Table] [Table] [Table]

Original: 5.2 Results for Robustness
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  Main Results We first evaluate the performance of RNA on CIFAR-10 and CIFAR-100 underdifferent types of attacks.
Revised: [Table] [Table] 5.2 Results for Robustness Main Results We first evaluate the performance of RNA on CIFAR-10 and CIFAR-100 under different types of attacks.

Original: The detailed results are shown in Table 1 and 2.
Revised: The detailed results are shown in Table 1 and 2.

Original: Popular normalizationlayers show similar performance on robustness.
Revised: Popular normalization layers show similar performance on robustness.

Original: However, with a random space of different normalization layers, the robustness is significantly improved.
Revised: However, with a random space of different normalization layers, the robustness is significantly improved.

Original: Comparing RNA with other baselines,
Revised: Comparing RNA with other baselines,

Original: RNA consistently achieves the best performance under all attacks, and show strong superiority.
Revised: RNA consistently achieves the best performance under all attacks, and show strong superiority.

Original: Forexample, RNA with ResNet-18 achieves 65 . 61% under Auto Attack on CIFAR-10, which is 17 .
Revised: For example, RNA with ResNet-18 achieves 65 . 61% under Auto Attack on CIFAR-10, which is 17 .

Original: 92%higher than BN.
Revised: 92% higher than BN.

Original: Similarly, RNA with WRN achieves 55 .
Revised: Similarly, RNA with WRN achieves 55 .

Original: 16% under DeepFool attacks on CIFAR-100,which is 53 .
Revised: 16% under DeepFool attacks on CIFAR-100, which is 53 .

Original: 33% higher than IN.
Revised: 33% higher than IN.

Original: The boosted adversarial accuracy shows strong empirical evidencethat the constrained adversarial transferability in random space can provide satisfactory defensecapability.
Revised: The boosted adversarial accuracy shows strong empirical evidence that the constrained adversarial transferability in random space can provide satisfactory defense capability.

Original: Furthermore, with proposed black-box adversarial training, RNA achieves better natural [Figure] [Table] [Table]
Revised: Furthermore, with proposed black-box adversarial training, RNA achieves better natural accuracy.

Original: accuracy.

Original: For example, RNA with WRN achieves 86 .
Revised: For example, RNA with WRN achieves 86 .

Original: 46% on CIFAR-10, which improves BN by1 .
Revised: 46% on CIFAR-10, which improves BN by 1 . 19% .

Original: We mainly attribute this improvement to the fact that the generated adversarial examples canbe treated as a “weaker” attack example to other paths during optimization, which naturally achievesbetter trade-offs between natural and adversarial accuracy.
Revised: We mainly attribute this improvement to the fact that the generated adversarial examples can be treated as a “weaker” attack example to other paths during optimization, which naturally achieves better trade-offs between natural and adversarial accuracy.

Original: Stronger PGD Attacks We further evaluate the defense capability of RNA under stronger PGDattacks, which enhance the number of iterations and enlarge the perturbation size.
Revised: Stronger PGD Attacks We further evaluate the defense capability of RNA under stronger PGD attacks, which enhance the number of iterations and enlarge the perturbation size.

Original: The comparisonwith other baselines are shown in Figure 5 (a) and (b).
Revised: The comparison with other baselines are shown in Figure 5 (a) and (b).

Original: Comparing RNA with other baselines, RNAachieves stable robustness under different attack iterations, such as 60 .
Revised: Comparing RNA with other baselines, RNA achieves stable robustness under different attack iterations, such as 60 .

Original: 70% on PGD 10 and 59 .
Revised: 70% on PGD 10 and 59 .

Original: 94%on PGD 100 .
Revised: 94% on PGD 100 .

Original: Meanwhile, the PGD accuracy of RNA is much higher than all other baselines.
Revised: Meanwhile, the PGD accuracy of RNA is much higher than all other baselines.

Original: Forexample, RNA improves BN baselines by a margin of 7 . 93% .
Revised: For example, RNA improves BN baselines by a margin of 7 . 93% .

Original: In terms of larger perturbation size,
Revised: In terms of larger perturbation size,

Original: RNA achieves the best robustness in all the scenarios, and the lowest decrement among all themethods.
Revised: RNA achieves the best robustness in all the scenarios, and the lowest decrement among all the methods.

Original: Specifically, RNA achieves 80 .
Revised: Specifically, RNA achieves 80 .

Original: 06% with ϵ of 2 / 255 and 24 .
Revised: 06% with ϵ of 2 / 255 and 24 .

Original: 90% with ϵ of 20 / 255 , whosegap is 55 .
Revised: 90% with ϵ of 20 / 255 , whose gap is 55 .

Original: For comparison, the gap of BN is 64 .
Revised: For comparison, the gap of BN is 64 .

Original: 68% and GN is 58 .
Revised: 68% and GN is 58 .

Original:  Comparison with SOTA Defense Methods To demonstrate the superiority of RNA, we includeseveral SOTA defense algorithms for comparison.
Revised: [Figure] [Table] [Table] Comparison with SOTA Defense Methods To demonstrate the superiority of RNA, we include several SOTA defense algorithms for comparison.

Original: RobustWRN [36] explores the importance ofnetwork width and depth on robustness.
Revised: RobustWRN [40] explores the importance of network width and depth on robustness.

Original: AWP [37] proposes to regularize the flatness of weight losslandscape to achieve robustness.
Revised: AWP [41] proposes to regularize the flatness of weight loss landscape to achieve robustness.

Original: RobNet [11] introduce a NAS framework for robustness.
Revised: RobNet [11] introduce a NAS framework for robustness.

Original: RPI+RPT[38] utilizes randomized precision for adversarial defense.
Revised: RPI+RPT [42] utilizes randomized precision for adversarial defense.

Original: SAT [39] proposes to replace ReLU withits smooth approximations, which exhibits robustness.
Revised: SAT [43] proposes to replace ReLU with its smooth approximations, which exhibits robustness.

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: We use WRNon CIFAR-10 and ResNet-50 on ImageNet.
Revised: We use WRN on CIFAR-10 and ResNet-50 on ImageNet.

Original: All the baselines are evaluated under the attack of PGD 20and AutoAttack (AA) on CIFAR-10 and PGD 50 on ImageNet.
Revised: All the baselines are evaluated under the attack of PGDand AutoAttack (AA) on CIFAR-10 and PGD 50 on ImageNet.

Original: Our proposed RNA module achievesthe best performance in all the scenarios.
Revised: Our proposed RNA module achieves the best performance in all the scenarios.

Original: On CIFAR-10, RNA achieves 67 .
Revised: On CIFAR-10, RNA achieves 67 .

Original: 88% under AutoAttack,with 13 .
Revised: 88% under AutoAttack, with 13 .

Original: 84% improvement compared with AWP.
Revised: 84% improvement compared with AWP.

Original: On ImageNet, RNA achieves 54 .
Revised: On ImageNet, RNA achieves 54 .

Original: 61% under PGD 50 ,with 12 . 31% improvement compared with SAT.
Revised: 61% under PGD 50 , with 12 . 31% improvement compared with SAT.

Original: Note that RNA replaces the normalization layer,which is orthogonal to other defense techniques.
Revised: Note that RNA replaces the normalization layer, which is orthogonal to other defense techniques.

Original: Similarly, the adversarial training in our setting canbe replaced by other advanced training strategies to achieve potentially better performance.
Revised: Similarly, the adversarial training in our setting can be replaced by other advanced training strategies to achieve potentially better performance.

Original: Adversarial Transferability in Random Space For a better illustration of the adversarial transferability in the random space built from RNA, we conduct the adversarial transferability study of
Revised: Adversarial Transferability in Random Space For a better illustration of the adversarial transferability in the random space built from RNA, we conduct the adversarial transferability study of

Original: ResNet-18 applied with RNA on CIFAR-10.
Revised: ResNet-18 applied with RNA on CIFAR-10.

Original: We first define the path difference as the number ofdifferent normalization layers between attack and inference paths.
Revised: We first define the path difference as the number of different normalization layers between attack and inference paths.

Original: For example, a path difference of7 denotes two paths selecting different normalization layers in 7 layers during forwarding.
Revised: For example, a path difference of 7 denotes two paths selecting different normalization layers in 7 layers during forwarding.

Original: For eachpath difference, we then randomly sample 10 path pairs for transferability evaluation.
Revised: For each path difference, we then randomly sample 10 path pairs for transferability evaluation.

Original: We also includedifferent normalization combinations in RNA for comparison.
Revised: We also include different normalization combinations in RNA for comparison.

Original: The results are shown in Figure 5 (c),which illustrates the PGD accuracy under transferred attacks between path pairs along increasing pathdifferences.
Revised: The results are shown in Figure 5 (c), which illustrates the PGD accuracy under transferred attacks between path pairs along increasing path differences.

Original: The filled areas denote the maximum and minimum PGD accuracy.
Revised: The filled areas denote the maximum and minimum PGD accuracy.

Original: It is obvious that the[Table] combination of BN and IN achieves the best performance, which also corresponds to the analysisin Theorem 3.1.
Revised: It is obvious that the combination of BN and IN achieves the best performance, which also corresponds to the analysis in Theorem 3.1.

Original: With random sampling strategy, the path difference is always around 10 since thenumber of normalization layers is 20 in this network.
Revised: With random sampling strategy, the path difference is always around 10 since the number of normalization layers is 20 in this network.

Original: For example, the combination of BN and INachieves an average PGD accuracy of 61 .
Revised: For example, the combination of BN and IN achieves an average PGD accuracy of 61 .

Original: 09% when the path difference becomes 10 .
Revised: 09% when the path difference becomes 10 .

Original: Compared withthe baseline BN which achieves 52 .
Revised: Compared with the baseline BN which achieves 52 .

Original: 16% PGD accuracy, this lower adversarial transferability in ourrandom space brings a strong defense capability against adversarial attacks.
Revised: 16% PGD accuracy, this lower adversarial transferability in our random space brings a strong defense capability against adversarial attacks.

Original: 5.3 Ablation Study
Revised: 5.3 Ablation Study

Original: Different Normalization Combinations We first provide more quantitative results of differentcombinations of normalization layers in RNA module.
Revised: Different Normalization Combinations We first provide more quantitative results of different combinations of normalization layers in RNA module.

Original: We conduct comparison with ResNet-18 on
Revised: We conduct comparison with ResNet-18 on

Revised: [Table]

Original: CIFAR-10.
Revised: CIFAR-10.

Original: As shown in Table 4, we evaluate the performance under PGD 20 , DeepFool and Auto
Revised: As shown in Table [Table] we evaluate the performance under PGD 20 , DeepFool and Auto

Original: Attack, and the combination of IN and BN achieves the best robustness in all the scenarios.
Revised: Attack [Table] and the combination of IN and BN achieves the best robustness in all the scenarios.

Original: Consistentwith the observation in Theorem 3.1, the combination of LN and BN has slightly worse performancethan that of IN and BN, since IN is a smoother normalization than LN.
Revised: Consistent with the observation in Theorem 3.1, the combination of LN and BN has slightly worse performance than that of IN and BN, since IN is a smoother normalization than LN.

Original: Similarly, the combination of
Revised: Similarly, the combination of

Original: GN and LN achieves the worst performance, since LN is a special case of GN so that they have highgradient similarity in our empirical observation, as discussed in Figure 3.
Revised: GN and LN achieves the worst performance, since LN is a special case of GN so that they have high gradient similarity in our empirical observation, as discussed in Figure 3.

Original: Thus, utilizing differentnormalization layers with smaller group size from GN and BGN, RNA module can form randomspace with low adversarial transferability to better improve the defense ability.
Revised: Thus, utilizing different normalization layers with smaller group size from GN and BGN, RNA module can form random space with low adversarial transferability to better improve the defense ability.

Original: Effectiveness of Different Components We next demonstrate the effectiveness of each componentintroduced in RNA module.
Revised: Effectiveness of Different Components We next demonstrate the effectiveness of each component introduced in RNA module.

Original: The detailed results are shown in Table 5 where [.] denotes the range ofgroup size.
Revised: The detailed results are shown in Table 5 where [.] denotes the range of group size.

Original: Besides the normalization combinations, we include more discussion of the random spacedesigning.
Revised: Besides the normalization combinations, we include more discussion of the random space designing.

Original: To form a random space in normalization layers, it is natural to consider a combinationof BN, IN, GN, and LN, however, it is difficult to optimize, as shown in the first row of Table 5.
Revised: To form a random space in normalization layers, it is natural to consider a combination of BN, IN, GN, and LN, however, it is difficult to optimize, as shown in the first row of Table 5.

Original: With the normalization layers selected from GNs, the optimization becomes stable, however, therobustness is not competitive, as shown in the second row.
Revised: With the normalization layers selected from GNs, the optimization becomes stable, however, the robustness is not competitive, as shown in the second row.

Original: The involvement of black-box adversarialtraining significantly improves the robustness, as shown in the third row.
Revised: The involvement of black-box adversarial training significantly improves the robustness, as shown in the third row.

Original: Through expanding therandom space with BGNs, the defense capability is slightly improved due to the doubled number ofpaths.
Revised: Through expanding the random space with BGNs, the defense capability is slightly improved due to the doubled number of paths.

Original: However, the size of random space is still limited.
Revised: However, the size of random space is still limited.

Original: After removing the layer-based constraint,the number of paths exponentially increases, the size of random space for each layer can be reducedfor better trade-offs, as shown in the last 4 rows.
Revised: After removing the layer-based constraint, the number of paths exponentially increases, the size of random space for each layer can be reduced for better trade-offs, as shown in the last 4 rows.

Original: Comparing BGN+GN[1-1] with GN[1-64], a betterrandom space designing with an appropriate adversarial training strategy can achieve an improvementof 15 .
Revised: Comparing BGN+GN[1-1] with GN[1-64], a better random space designing with an appropriate adversarial training strategy can achieve an improvement of 15 .

Original: 04% under Auto Attack, which demonstrates the necessity of these components.
Revised: 04% under Auto Attack, which demonstrates the necessity of these components.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we explore the importance of normalization layers in adversarial robustness where thetransferability among different normalization layers can be utilized to boost the defense capability.
Revised: In this paper, we explore the importance of normalization layers in adversarial robustness where the transferability among different normalization layers can be utilized to boost the defense capability.

Original: A Random Normalization Aggregation (RNA) module is proposed to form random space with lowadversarial transferability for defense against adversarial attacks.
Revised: A Random Normalization Aggregation (RNA) module is proposed to form random space with low adversarial transferability for defense against adversarial attacks.

Original: We provide sufficient empiricalevidence and theoretical analysis to reveal the connections between adversarial transferability andnormalization types, which guides the random space designing.
Revised: We provide sufficient empirical evidence and theoretical analysis to reveal the connections between adversarial transferability and normalization types, which guides the random space designing.

Original: With the involvement of black-boxadversarial training strategy and the relaxation of layer-based constraint, the robustness providedby RNA module is significantly strengthened.
Revised: With the involvement of black-box adversarial training strategy and the relaxation of layer-based constraint, the robustness provided by RNA module is significantly strengthened.

Original: We demonstrate the superiority of RNA module viacomprehensive experiments across different network architectures, attack settings, and benchmarkdatasets.
Revised: We demonstrate the superiority of RNA module via comprehensive experiments across different network architectures, attack settings, and benchmark datasets.

Original: Our work can provide valuable insights into the network module design for robustness.Checklist
Revised: Our work can provide valuable insights into the network module design for robustness.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes](c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
Revised: Acknowledgments

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results? [Yes] (b) Did you include complete proofs of all theoretical results? [Yes]3.
Revised: This work was supported in part by the Australian Research Council under Project DP210101859 and the University of Sydney Research Accelerator (SOAR) Prize.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [No]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and  the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] If you are using existing assets (e
Revised: The authors acknowledge the use of the National Computational Infrastructure (NCI) which is supported by the Australian Government, and accessed through the NCI Adapter Scheme and Sydney Informatics

Original: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets?
Revised: Hub HPC Allocation Scheme.

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]
Revised: We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: Networks)

Original: Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: and Ascend AI Processor used for this research.