Original: E XPLORING AND E NHANCING THE T RANSFERABILITY OF A DVERSARIAL E XAMPLES
Revised: E XPLORING AND E NHANCING THE T RANSFERABILITY OF A DVERSARIAL E XAMPLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: State-of-the-art deep neural networks are vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs.
Revised: State-of-the-art deep neural networks are vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs.

Original: Moreover, the perturbations can transfer across models : adversarial examples generated for a speciﬁc model will often mislead other unseen models.
Revised: Moreover, the perturbations can transfer across models : adversarial examples generated for a speciﬁc model will often mislead other unseen models.

Original: Consequently the adversary can leverage it to attack deployed systems without any query, which severely hinder the application of deep learning, especially in the areas where security is crucial.
Revised: Consequently the adversary can leverage it to attack deployed systems without any query, which severely hinders the application of deep learning, especially in the safety-critical areas.

Original: In this work, we empirically study how two classes of factors that might inﬂuence the transferability of adversarial examples.
Revised: In this work, we empirically study how two classes of factors those might inﬂuence the transferability of adversarial examples.

Original: One is about model-speciﬁc factors, including network architecture, model capacity and test accuracy.
Revised: One is about model-speciﬁc factors, including network architecture, model capacity and test accuracy.

Original: The other is the local smoothness of loss surface for constructing adversarial examples.
Revised: The other is the local smoothness of loss surface for constructing adversarial examples.

Original: We then propose a simple but effective strategy to enhance the transferability, whose effectiveness is conﬁrmed by a variety of experiments on both CIFAR-10 and ImageNet datasets.
Revised: Inspired by these understandings on the transferability of adversarial examples, we then propose a simple but effective strategy to enhance the transferability, whose effectiveness is conﬁrmed by a variety of experiments on both CIFAR-10 and ImageNet datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: By now, more and more large neural network models are applied in real-world applications, such as speech recognition, computer vision, etc.

Original: While these models have exhibited good performance, recently Szegedy et al.
Revised: Recently Szegedy et al.

Original: (2013) showed that an adversary is able to fool the model into producing incorrect predictions by manipulating the inputs maliciously.
Revised: (2013) showed that an adversary is able to fool deep neural network models into producing incorrect predictions by manipulating the inputs maliciously.

Original: The corresponding manipulated samples are called adversarial examples .
Revised: The corresponding manipulated samples are called adversarial examples .

Original: More seriously, it is found that they have cross-model generalization ability, i.e. the adversarial example generated from one model can fool another different model with a signiﬁcant probability.
Revised: More severely, it was found that adversarial examples have cross-model generalization ability, i.e., that adversarial examples generated from one model can fool another different model with a high probability.

Original: We refer to such property as transferability .
Revised: We refer to such property as transferability .

Original: In consequence, hackers can employ this property to attack  deployed models without any query (Papernot et al., 2016b; Liu et al., 2017), inducing serious security issue to deep learning system. The transfer-based attack could be the most dangerous and mysterious, since they surprisingly do not require any input-output query of the target system.
Revised: By now, more and more deep neural network models are applied in real-world applications, such as speech recognition, computer vision, etc.. Consequently, an adversary can employ the transferability to attack those deployed models without querying the systems (Papernot et al., 2016b; Liu et al., 2017), inducing serious security issues.

Original: Understanding the mechanism  could potentially provide various beneﬁts for modern deep learning models.
Revised: Understanding the mechanism of transferability could potentially provide various beneﬁts.

Original: Firstly, for the already deployedand vulnerable deep neural networks in real systems, it could help to design better strategies to improve the robustness to the transfer-based attacks.
Revised: Firstly, for the already deployed deep neural network models in real systems, it could help to design better strategies to improve the robustness against the transfer-based attacks.

Original: Secondly, revealing the mystery behind the transferability of adversarial examples could also extend the existing understandings onmodern deep learning, particularlyon the effects of model capacity (Madry et al., 2018; Fawzi et al., 2015)  model interpretability (Dong et al., 2017; Ross and Doshi-Velez, 2018).
Revised: Secondly, revealing the mystery behind the transferability of adversarial examples could also expand the existing understandings on deep learning, particularly the effects of model capacity (Madry et al., 2018; Fawzi et al., 2015) and model interpretability (Dong et al., 2017; Ross and Doshi-Velez, 2018).

Original: Therefore, studying the transferability of adversarial example in the context of deep networks is of signiﬁcant importance.
Revised: Therefore, studying the transferability of adversarial examples in the context of deep networks is of signiﬁcant importance.

Original: In this paper, we numerically investigate two classes of factors that might inﬂuence the adversarial transferability, and further provide a simple but rather effective strategy for enhancing the transferability.
Revised: In this paper, we numerically investigate two classes of factors that might inﬂuence the adversarial transferability, and further provide a simple but rather effective strategy for enhancing the transferability.

Original: Our contributions are summarized as follows.
Revised: Our contributions are summarized as follows.

Original:  We ﬁnd that adversarial transfer is not symmetric, i e. adversarial examples generated from model A can transfer to model B easily does not means the reverse is also natural.
Revised: • We ﬁnd that adversarial transfer is not symmetric, which means that adversarial examples generated from model A can transfer to model B easily does not implies the reverse is also natural.

Original: Second, we ﬁnd that adversarial examples generated from a deep model appear less transferable than  a shallow model.
Revised: Second, we ﬁnd that adversarial examples generated from a deep model appear less transferable than those from a shallow model.

Original: We also explore the impact ofshattered gradients, which reﬂect the roughness of  loss surface.
Revised: We also explore the impact of the nonsmoothness of the loss surface.

Original: Speciﬁcally, we ﬁnd that the local non-smoothness of loss surface harms the transferability of generated adversarial examples.
Revised: Speciﬁcally, we ﬁnd that the local non-smoothness of loss surface harms the transferability of generated adversarial examples.

Original: • Inspired by previous investigations, We suggest applying the locally averaged gradient instead of the original one to generate adversarial examples.It is dubbed as smooth gradient , since the averaging have the smoothing effect which suppresses the local oscillation of the loss surface.
Revised: • Inspired by previous investigations, we propose the smoothed gradient attack to improve the adversarial transferability, which employs the locally averaged gradient instead of the original one to generate adversarial examples.

Original: Extensive numerical validations justify the effectiveness of our method
Revised: Extensive numerical validations justify the effectiveness of our method.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The phenomenon of adversarial transferability was ﬁrst studied by Szegedy et al.
Revised: The phenomenon of adversarial transferability was ﬁrst studied by Szegedy et al.

Original: By utilizing the transferability, Papernot et al.
Revised: By utilizing the transferability, Papernot et al.

Original: (2016b;a)and proposed a practical black-box attack by training a substitute model with limited queried information.
Revised: (2016b;a) proposed a practical black-box attack by training a substitute model with limited queried information.

Original: Liu et al.
Revised: Liu et al.

Original: ﬁrst studied the targeted transferability and introduced the ensemble-based attacks to improve the transferability.
Revised: ﬁrst studied the targeted transferability and introduced the ensemble-based attacks to improve the transferability.

Original: More recently, Dong et al.
Revised: More recently, Dong et al.

Original: (2018) showed that  momentum can help to boost transferability signiﬁcantly.
Revised: (2018) showed that the momentum can help to boost transferability signiﬁcantly.

Revised: Meanwhile, there exist several works trying to explain the adversarial transferability.

Revised: Papernot et al.

Revised: (2016a) attributed the transferability to the similarity between input gradients of source and target models.

Original:  Our smoothed gradient attacks that enhance the transferability by utilizing informations in the small neighborhood of the clean example is inspired by the works on shattered gradients (Balduzzi et al., 2017) and model interpretability (Smilkov et al., 2017).
Revised: Liu et al. (2017) proposed to use the visualization technique to see the large-scale similarity of decision boundaries. However, our numerical investigations imply that these similarity-based explanations have their intrinsic limitation that they cannot explain the non-symmetric property of adversarial transferability. Our smoothed gradient attacks that enhance the transferability by utilizing the information in the small neighborhood of the clean example is inspired by the works on shattered gradients (Balduzzi et al., 2017) and model interpretability (Smilkov et al., 2017).

Original: Recently, similar strategies are also explored by Athalye et al.
Revised: Recently, similar strategies are also explored by Athalye et al.

Original: (2018a)
Revised: (2018a)

Original: and
Revised: and

Original: Warren He (2018) for white-box attacks, whereas we focus on the transferability.
Revised: Warren He (2018) for white-box attacks, whereas we focus on the black-box settings.

Original: Our method is also related to the work by Athalye et al.
Revised: Our method is also related to the work by Athalye et al.

Original: (2018b), which introduced the expectation over transformation (EOT) method to increase robustness of adversarial examples.
Revised: (2018b), which introduced the expectation over transformation (EOT) method to increase robustness of adversarial examples.

Original: The EOT formulation is similar to our objective (7), but they did not study the transferability.
Revised: The EOT formulation is similar to our objective (7), but they did not study the transferability.

Original: Also our motivation are totally different from theirs.
Revised: Also our motivations are totally different from theirs.

Original: Meanwhile, there exist several works trying to explain the adversarial transferability.

Original: Papernot et al.

Original: (2016a) contributed it to the similarity between input gradients of source and target models.

Original: By visualization, Liu et al.

Original: (2017) suggested that the transferability comes from the similarity of decision boundaries.

Original: Our numerical investigations implies that these similarity-based understandings have the intrinsic limitation, which cannot explain the non-symmetric property of adversarial transferability.

Original:  B ACKGROUND 3. A DVERSARIAL ATTACK
Revised:  B ACKGROUND 3. A DVERSARIAL ATTACK

Original: We use f ( x )  denote the classiﬁer.
Revised: Let f ( x ) : R d (cid:55)→ R K denote the classiﬁer.

Original: In deep learning, it is found that for almost any sample x and its label y true , there exists a small perturbation η that is nearly imperceptible to human, such that
Revised: In deep learning, it is found that for almost any sample x and its label y true , there exists a small perturbation η that is nearly imperceptible to human such that

Original: [Equation]
Revised: [Equation]

Original: We call η  adversarial perturbation and correspondingly x adv := x + η the adversarial example.
Revised: We call η the adversarial perturbation and correspondingly x adv := x + η the adversarial example.

Original: The attack (1) is called a non-targeted attack since the adversary has no control over which class the input x will be misclassiﬁed to.
Revised: The attack (1) is called a non-targeted attack since the adversary has no control over which class the input x will be misclassiﬁed to.

Original: In contrast, a targeted attack aims at fooling the model to produce a wrong label speciﬁed by the adversary, i.e. argmax i f i ( x + η ) =
Revised: In contrast, a targeted attack aims at fooling the model to produce a wrong label speciﬁed by the adversary, i.e. argmax i f i ( x + η ) =

Original: y target .
Revised: y target .

Original: In this paper, we consider the pure black-box setting.
Revised: In this paper, we consider the pure black-box attacks.

Original: That means the adversary has no knowledge of the target model (e.g. architecture and parameters) and is  not allowed to query  input-output pairs from the  model.
Revised: This means the adversary has no knowledge of the target model (e.g. architecture and parameters) and is also not allowed to query any input-output pair from the target model.

Original: On the contrary, the white-box attack means  the target model is available for adversary.
Revised: On the contrary, the white-box attack means that the target model is available to the adversary.

Original: 3. G ENERATING ADVERSARIAL EXAMPLES
Revised: 3. G ENERATING ADVERSARIAL EXAMPLES

Original: Modeling In general, crafting adversarial perturbation can be modeled as the following optimization problem,
Revised: Modeling In general, crafting adversarial examples can be modeled as the following optimization problem,

Original: [Equation]
Revised: [Equation]

Original: where J is some loss function measuring the discrepancy between the model prediction and ground truth;  (cid:107) · (cid:107) ∞ is used to quantify the magnitude of the perturbation.
Revised: where J is some loss function measuring the discrepancy between the model prediction and ground truth; the (cid:96) ∞ norm (cid:107) · (cid:107) ∞ is used to quantify the magnitude of the perturbation.

Original: Other norm is also possible, but we focus on (cid:96) ∞ norm in this paper.
Revised: Other norm is also possible, but we focus on (cid:96) ∞ norm in this paper.

Original: To improve the strength of adversarial transferability, instead of using a single model, Liu et al.
Revised: To improve the strength of adversarial transferability, instead of using a single model, Liu et al.

Original: (2017) proposed the ensemble attack , which solves the following problem
Revised: (2017) proposed the ensemble attack , which generates adversarial examples against a model ensemble f 1 ( x ) , f 2 ( x ) , · · · , f M [Equation]

Original: [Equation]
Revised: [Equation]

Original: Optimizer We use the following iteration to solve (2) and (3),
Revised: Optimizer We use the following iteration to solve (2) and (3),

Original: [Equation]
Revised: [Equation]

Original: where D = [0 , 255] d ∩ { x (cid:48) | (cid:107) x (cid:48) − x (cid:107) ≤ ε } and α is the step size.
Revised: where D = [0 , 255] d ∩ { x (cid:48) | (cid:107) x (cid:48) − x (cid:107) ≤ ε } and α is the step size.

Original: We call the attack evolving (4) for T steps iterative gradient sign method (IGSM) attack (Kurakin et al., 2017; Madry et al., 2018).
Revised: We call the attack that evolves (4) for T steps iterative gradient sign method (IGSM) attack (Kurakin et al., 2017; Madry et al., 2018).

Original: Furthermore, the famous fast gradient sign method (FGSM) is a special case with α = ε, T = 1 .
Revised: Furthermore, the famous fast gradient sign method (FGSM) is a special case with α = ε, T = 1 .

Original: Dong et al.
Revised: Dong et al.

Original: (2018) recently proposed the momentum-based attack as follows
Revised: (2018) recently proposed the momentum-based attack as follows

Original: [Equation]
Revised: [Equation]

Original: where µ is the decay factor of momentum.
Revised: where µ is the decay factor of momentum.

Original: By using this method, they won the ﬁrst-place in NIPS 2017 Non-targeted Adversarial Attack and Targeted Adversarial Attack competitions.
Revised: By using this method, they won the ﬁrst-place in NIPS 2017 Non-targeted Adversarial Attack and Targeted Adversarial Attack competitions.

Original: We will call it mIGSM attack in this paper.
Revised: We will call it mIGSM attack in this paper.

Original:  E VALUATION OF A DVERSARIAL T RANSFERABILITY
Revised:  E VALUATION OF A DVERSARIAL T RANSFERABILITY

Original: Datasets To evaluate the transferability, three datasets including MNIST, CIFAR-10 and ImageNet are considered.
Revised: Datasets To evaluate the transferability, three datasets including MNIST, CIFAR-10 and ImageNet are considered.

Original: For ImageNet, directly evaluation on the whole ILSVRC2012 validation dataset is too time-consuming.
Revised: For ImageNet, evaluations on the whole ILSVRC2012 validation dataset are too timeconsuming.

Original: Therefore, in each experiment we randomly select 5 , 000 images that can be correctly recognized by all the examined models forming our  validation set.
Revised: Therefore, in each experiment we randomly select 5 , 000 images that can be correctly recognized by all the examined models to form our new validation set.

Original: Models (i) For MNIST, we trained fully connected networks (FNN) of D hidden layers, with the width of each layer being 500 .
Revised: Models (i) For MNIST, we trained fully connected networks (FNN) of D hidden layers, with the width of each layer being 500 .

Original: (ii)
Revised: (ii)

Original: For CIFAR-10, we trained ﬁve models: lenet,resnet20, resnet44, resnet56, densenet .
Revised: For CIFAR-10, we trained ﬁve models: lenet,resnet20, resnet44, resnet56, densenet .

Original: The test accuracies of them are 76 .
Revised: The test accuracies of them are 76 .

Original: 8% and 94 .
Revised: 8% and 94 .

Original: 2% , respectively.
Revised: 2% , respectively.

Original: (iii)
Revised: (iii)

Original: For ImageNet, the pre-trained models provided by PyTorch are used.
Revised: For ImageNet, the pre-trained models provided by PyTorch are used.

Original: The Top-1 and Top-5 accuracies of them can be found on website 1 .
Revised: The Top-1 and Top-5 accuracies of them can be found on website 1 .

Original: To increase the reliability of experiments, all the models have been tested.
Revised: To increase the reliability of experiments, all the models have been tested.

Original: However, for a speciﬁc experiment we only choose some of them to present since the ﬁndings are consistent among  the tested models.
Revised: However, for a speciﬁc experiment we only choose some of them to present since the ﬁndings are consistent among all the tested models.

Original: Criteria Given a set of adversarial pairs, { ( x advi , y true i ) } Ni =1 , we calculate their Top-1 success rates (%) fooling a given model f ( x ) by 100 N (cid:80) Ni =1 1[argmax
Revised: Criteria Given a set of adversarial pairs, { ( x advi , y true i ) } Ni =1 , we calculate the Top-1 success rate (%) fooling a given model f ( x ) by 100 N (cid:80) Nk =1 1[argmax

Original: i f i ( x advi ) (cid:54) =
Revised: i f i ( x advk ) (cid:54) =

Original: y true i ] .
Revised: y true k ] .

Original: For targeted attacks, each image x adv is associated with a pre-speciﬁed label y target (cid:54) =
Revised: For targeted attacks, each image x adv is associated with a pre-speciﬁed label y target (cid:54) =

Original: y true .
Revised: y true .

Original: Then we evaluate the performance of the targeted attack by the following Top-1 success rate: 100 N (cid:80) Ni =1 1[argmax i f i ( x advi ) =
Revised: Then we evaluate the performance of the targeted attack by the following Top-1 success rate: 100 N (cid:80) Nk =1 1[argmax i f i ( x advk ) =

Original: y target i ] .
Revised: y target k ] .

Original: The corresponding Top-5 rates can be computed in a similar way.
Revised: The corresponding Top-5 rates can be computed in a similar way.

Original:  H OW M ODEL - SPECIFIC F ACTORS A FFECT T RANSFERABILITY
Revised:  H OW M ODEL - SPECIFIC F ACTORS A FFECT T RANSFERABILITY

Original: Previous study on adversarial transferability mostly focused on the inﬂuence of attack methods (Liu et al., 2017; Dong et al., 2018; Tram ` er et al., 2017; Kurakin et al., 2017).
Revised: Previous studies on adversarial transferability mostly focused on the inﬂuence of attack methods (Liu et al., 2017; Dong et al., 2018; Tram`er et al., 2017; Kurakin et al., 2017).

Original: However it is not clear how the choice of source model affects the success rate transferring to target models.
Revised: However it is not clear how the choice of source model affects the success rate transferring to target models.

Original: In this section, we investigate this issue from three aspects including architecture, test accuracy and model capacity.
Revised: In this section, we investigate this issue from three aspects including architecture, test accuracy and model capacity.

Original: 5. A RCHITECTURE
Revised: 5. A RCHITECTURE

Original: We ﬁrst explore how the architecture similarity between source and target model contributes to the transferability.
Revised: We ﬁrst explore how the architecture similarity between source and target model contributes to the transferability.

Original: This study is crucial since it can provide us guidance to choose the appropriate source models for effective attacks.
Revised: This study is crucial since it can provide us guidance to choose the appropriate source models for effective attacks.

Original: To this end, three popular architectures including ResNet, DenseNet and
Revised: To this end, three popular architectures including ResNet, DenseNet and

Original: VGGNet are considered, and for each architecture, two networks are selected.
Revised: VGGNet are considered, and for each architecture, two networks are selected.

Original: Both one-step and multi-step attacks are performed on ImageNet dataset.
Revised: Both one-step and multi-step attacks are performed on ImageNet dataset.

Original: Table 1 presents the experiment results, and the choice of hyper-parameter is detailed in the caption. 
Revised: Table 1 presents the experiment results, and the choice of hyper-parameters is detailed in the caption. [Table]

Original: We can see that the transfers between two models are non-symmetric, and this phenomenon is more obvious for the models with different architectures.
Revised: We can see that the transfers between two models are non-symmetric, and this phenomenon is more obvious for the models with different architectures.

Original: For instance, the success rates of IGSM attacks from densenet121 to vgg13 bn is 58 .
Revised: For instance, the success rates of IGSM attacks from densenet121 to vgg13 bn is 58 .

Original: 7% , however the rate from vgg13 bn to densenet121 has only 15 .
Revised: 7% , however the rate from vgg13 bn to densenet121 has only 15 .

Original: The lack of symmetry implies that previous similarity-based explanations of adversarial transferability is quite limited.
Revised: The lack of symmetry implies that previous similarity-based explanations of adversarial transferability are quite limited.

Original: [Table]

Original: Another interesting observation is that success rates between models with similar architectures are always much higher.
Revised: Another interesting observation is that success rates between models with similar architectures are always much higher.

Original: For example the success rates of IGSM attacks between vgg13 bn and vgg16 bn are higher than 90% , nearly twice the ones of attacks from any other architectures [Table]
Revised: For example the success rates of IGSM attacks between vgg13 bn and vgg16 bn are higher than 90% , nearly twice the ones of attacks from any other architectures.

Original: 5. M ODEL C APACITY AND T EST A CCURACY
Revised: 5. M ODEL C APACITY AND T EST A CCURACY

Original: We ﬁrst study this problem on ImageNet dataset.
Revised: We ﬁrst study this problem on ImageNet dataset.

Original: A variety of models are used as source models to perform both FGSM and IGSM attacks against vgg19 bn , with results  displayed in Figure 1.
Revised: A variety of models are used as source models to perform both FGSM and IGSM attacks against vgg19 bn , and the results are displayed in Figure 1.

Original: The horizontal axis is the Top-1 test error, while the vertical axis is the number of model parameters that roughly quantiﬁes the model capacity.
Revised: The horizontal axis is the Top-1 test error, while the vertical axis is the number of model parameters that roughly quantiﬁes the model capacity.

Original: We can see that the models with powerful attack capability concentrate in the bottom left corner, whereas for those models with either large test error or large number of parameters, the fooling rates are much lower.
Revised: We can see that the models with powerful attack capability concentrate in the bottom left corner, whereas for those models with either large test error or large number of parameters, the fooling rates are much lower.

Original: We also tried other models with results shown in Appendix 
Revised: We also tried other models with results shown in Appendix C

Original: [Figure]
Revised: [Figure]

Original: and the results show no difference.
Revised: and the results show no difference

Original: [Figure]
Revised: [Figure]

Original:  We suspect that the impact of test accuracy is due to that the decision boundaries of high-accuracy models are similar, since they all approximate the ground-truth decision boundary of true data very well.
Revised: p [Figure] We suspect that the impact of test accuracy is due to that the decision boundaries of high-accuracy models are similar, since they all approximate the ground-truth decision boundary of true data very well.

Original: On the contrary, the model with low accuracymust has a decision boundary relatively different from the high-accuracy models.
Revised: On the contrary, the model with low accuracy has a decision boundary relatively different from the high-accuracy models.

Original: Here the targeted network vgg19 bn is a high-accuracy model.
Revised: Here the targeted network vgg19 bn is a high-accuracy model.

Original: Therefore it is not surprising to observe that high-accuracy models tend to exhibit stronger attack capability.
Revised: Therefore it is not surprising to observe that high-accuracy models tend to exhibit stronger attack capability.

Original: This phenomenon implies a kind of data-dependent transferability, which is different from architecture-dependent transfers observed in the previous section.
Revised: This phenomenon implies a kind of data-dependent transferability, which is different from the architecture-speciﬁc transfers observed in the previous section.

Original: It is somewhat strange that adversarial examples generated from deeper model appear less transferable .
Revised: It is somewhat strange that adversarial examples generated from deeper model appear less transferable .

Original: To further conﬁrm this observation, we conduct additional experiments on MNIST and
Revised: To further conﬁrm this observation, we conduct additional experiments on MNIST and

Original: CIFAR-10.
Revised: CIFAR-10.

Original: Table 2 shows the results, which is basically consistent.
Revised: Table 2 shows the results, which is basically consistent.

Original: This observation suggests us not to use deep models as the source models when performing transfer-based attacks, although we have not fully understand this phenomenon.
Revised: This observation suggests us not to use deep models as the source models when performing transfer-based attacks, although we have not fully understand this phenomenon.

Original: [Table]
Revised: [Table]

Original:  N ON - SMOOTHNESS OF  L OSS S URFACE
Revised:  N ON - SMOOTHNESS OF THE L OSS S URFACE

Revised: In this section

Revised: [Table]

Revised: we consider that how the smoothness of loss surface J ( x ) affects the transferability.

Revised: For simplicity, let g ( x ) =

Revised: ∇ x J ( x ) denote the gradient.

Original: Smilkov et al.
Revised: Smilkov et al.

Original: (2017) showed that gradient ∇ J could be very noisy and uninformative for visualization, though the model is trained very well.
Revised: (2017) showed that gradient g ( x ) is very noisy and uninformative for visualization, though the model is trained very well.

Original: Balduzzi et al.
Revised: Balduzzi et al.

Original: (2017) studied a similar phenomenon that gradients of deep networks are  shattered.
Revised: (2017) studied a similar phenomenon that gradients of deep networks are extremely shattered.

Original: Both of them implies that the landscape are locally rough .
Revised: Both of them implies that the landscape is locally extremely rough.

Original: In this section [Table] we study how this  non-smoothness affects the transferability 
Revised: We suspect that this local non-smoothness could harm the transferability of adversarial examples

Original: [Table]
Revised: [Table]

Original: 6. I NTUITION
Revised: 6. I NTUITION

Original: [Table]
Revised: [Table]

Original:  For simplicity, assume model A and B are the source and target models, respectively and let g ( x ) = ∇ x J ( x ) .
Revised: [Figure] For simplicity, assume model A and B are the source and target models which are well trained with high test accuracies, respectively.

Original: Previous methods use g A ( x ) to generate adversarial perturbations, so the transferability mainly depends on how much sensitivity of g A for model A can transfer to model 
Revised: Previous methods use g A ( x ) to generate adversarial perturbations, so the transferability mainly depends on how much sensitivity of g A for model A can transfer to model B.

Original: B. We hypothesize that the noise hurts the transferability of g A , as illustrated in right cartoon ﬁgure, where three curves denote the level sets of three models. Since both model A and B have very high test accuracy, their level sets should be similar globally, and J B ( x ) is probably unstable along g A .
Revised: As illustrated in Figure 2, where three curves denote the level sets of three models, we can see that the non-smoothness can hurt the transferability, since both model A and B have very high test accuracy, their level sets should be similar globally, and J B ( x ) is probably unstable along g A .

Original: However, as illustrated, the local ﬂuctuation of g A make the sensitivity less transferable.
Revised: As illustrated, the local oscillation of g A makes the sensitivity less transferable.

Original: One way to alleviate it is to smooth the landscape J A , thereby yielding a more transferable gradient G A , i.e. (cid:104)
Revised: One way to alleviate this is to smooth the landscape J A , thereby yielding a more transferable gradient G A , i.e. (cid:104)

Original: ˆ G A , ˆ g B (cid:105) > (cid:104)
Revised: ˆ G A , ˆ g B (cid:105) > (cid:104)

Original: ˆ g A , ˆ g B (cid:105) .
Revised: ˆ g A , ˆ g B (cid:105) .

Revised: Here we use G denote the gradient of the smoothed loss surface.

Original: 6. J USTIFICATION
Revised: 6. J USTIFICATION

Original: To justify the previous arguments, we consider smoothing the loss surface by convolving  with a Gaussian, J σ ( x ) : = (cid:82) J ( x − x (cid:48) ) φ σ ( x (cid:48) )
Revised: To justify the previous arguments, we consider smoothing the loss surface by convolving it with a Gaussian ﬁlter, then the smoothed loss surface is given by J σ ( x ) :=

Original: dx (cid:48) with φ σ ( x ) being the density of N (0 , σ 2 I )  .
Revised: E ξ ∼N (0 ,I ) [ J ( x + σξ )] .

Original: The corresponding gradient can be calculated by
Revised: The corresponding gradient can be calculated by

Original: [Equation]
Revised: [Equation]

Original: The extent of smoothing is controlled by σ , and please refer to Appendix A to see  smoothing effect.
Revised: The extent of smoothing is controlled by σ , and please refer to Appendix A to see the smoothing effect.

Original: We will show that G σ ( x ) is more transferable than g ( x ) .
Revised: We will show that G σ ( x ) is more transferable than g ( x ) in the following.

Original: Gradient similarity We ﬁrst quantify the cosine similarity between gradients of source and target models, respectively.
Revised: Gradient similarity We ﬁrst quantify the cosine similarity between gradients of source and target models, respectively.

Original: Two situations are considered: vgg13 bn → vgg16 bn , densenet121 → vgg13 bn , which correspond the within-architecture and cross-architecture transfers, respectively.
Revised: Two situations are considered: vgg13 bn → vgg16 bn , densenet121 → vgg13 bn , which correspond the within-architecture and cross-architecture transfers, respectively.

Original: We choose σ = 15 , and the expectation in (6) is estimated by using 1 m (cid:80) mi =1 g ( x + ξ i ) .
Revised: We choose σ = 15 , and the expectation in (6) is estimated by using 1 m (cid:80) mi =1 g ( x + ξ i ) .

Original: To verify the averaged gradients do transfer better, we plot the cosine similarity against the number of samples m .
Revised: To verify the averaged gradients do transfer better, we plot the cosine similarity against the number of samples m .

Original: In
Revised: In

Original: Figure 2a, as expected, we see that the cosine similarity between G A and g B are indeed larger than the one between g A and g B .
Revised: Figure 3a, as expected, we see that the cosine similarity between G A and g B are indeed larger than the one between g A and g B .

Original: Moreover, the similarity increases with m monotonically, which further justiﬁes that G A is more transferable than g A .
Revised: Moreover, the similarity increases with m monotonically, which further justiﬁes that G A is more transferable than g A .

Original: Visualization In Figure 2b we visualize the transferability by comparing the decision boundaries of 
Revised: Visualization In Figure 3b we visualize the transferability by comparing the decision boundaries of model A ( resnet34 ) and model B ( densenet121 ). The horizontal axis represents the direction of

Revised: G A of resnet34 , estimated by m = 1000 , σ = 15 , and the vertical axis denotes orthogonal direction

Revised: h A :=

Revised: g A − (cid:104)

Revised: g

Original: It can be easily observed that for model A, a small perturbation in both directions can produce wrong classiﬁcation.
Revised: A , ˆ G A (cid:105) ˆ G A . This process means that we decompose the gradient into two terms: g A = αG A + βh A with (cid:104) G A , h A (cid:105) = 0 . Each point in the 2-D plane corresponds to the image perturbed by u and v along each direction, clip ( x + u ˆ G A

Revised: + v ˆ h A , 0 , 255) , where x is the clean image.

Revised: The color corresponds to the label predicted by the target model.

Revised: It can be easily observed that for model A, a small perturbation in both directions can produce wrong classiﬁcation.

Original: However, when applied to model B, the sensitivities of two directions dramatically change.
Revised: However, when applied to model B, the sensitivities of two directions dramatically change.

Original: The direction of h A becomes extremely stable, whereas to some extent G A preserves the sensitivity, i.e. G A do transfer to model B better than h A .
Revised: The direction of h A becomes extremely stable, whereas to some extent G A preserves the sensitivity, i.e. G A do transfer to model B better than h A .

Original: This suggests that for gradient g A , the noisy part h A is less transferable than the smooth part G A .
Revised: This suggests that for gradient g A , the noisy part h A is less transferable than the smooth part G A .

Original: We also tried a variety of other models shown in Appendix C, and the results are the same.
Revised: We also tried a variety of other models shown in Appendix C, and the results are the same.

Original: [Figure]
Revised: [Figure]

Original:  S MOOTHED G RADIENT A TTACK 7. M ETHOD
Revised:  S MOOTHED G RADIENT A TTACK 7. M ETHOD

Original: Inspired by the previous investigations, enhancing the adversarial transferability can be achieved by smoothing the loss surface.
Revised: Inspired by the previous investigations, enhancing the adversarial transferability can be achieved by smoothing the loss surface.

Original: Then our objective becomes,
Revised: Then our objective becomes,

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this method can also be interpreted as generating adversarial examples that are robust to
Revised: Intuitively, this method can also be interpreted as generating adversarial examples that are robust to

Original: Gaussian perturbation.
Revised: Gaussian perturbation.

Original: Expectedly, the generated robust adversarial examples can still survive easily in spite of the distinction between source and target model.
Revised: Expectedly, the generated robust adversarial examples can still survive easily in spite of the distinction between source and target model.

Original: If use the iterative gradient sign method to solve (7), we have the following iteration:
Revised: If use the iterative gradient sign method to solve (7), we have the following iteration:

Original: [Equation]
Revised: [Equation]

Original: where G t is a mini-batch approximation of the smoothed gradient (6).
Revised: where G t is a mini-batch approximation of the smoothed gradient (6).

Original: Compared to the standard
Revised: Compared to the standard

Original: IGSM, the gradient is replaced by a smoothed version, which is endowed with stronger transferability.
Revised: IGSM, the gradient is replaced by a smoothed version, which is endowed with stronger transferability.

Original: Therefore we call this method sg-IGSM.
Revised: Therefore we call this method sg-IGSM.

Original: The special case T = 1 , α = ε , is accordingly called sg-FGSM.
Revised: The special case T = 1 , α = ε , is accordingly called sg-FGSM.

Original: Any other optimizer can be used to solve the (7) as well, and we only need to replace the original gradient with the smoothed one.
Revised: Any other optimizer can be used to solve the (7) as well, and we only need to replace the original gradient with the smoothed one.

Original: 7. C HOICE OF H YPER P ARAMETERS
Revised: 7. C HOICE OF H YPER P ARAMETERS

Original: We ﬁrst explore the sensitivity of hyper parameters m, σ when applying our smoothed gradient technique.
Revised: We ﬁrst explore the sensitivity of hyper parameters m, σ when applying our smoothed gradient technique.

Original: We take ImageNet dataset as the testbed, and sg-FGSM attack is examined.
Revised: We take ImageNet dataset as the testbed [Figure] and sg-FGSM attack is examined.

Original: To increase the reliability, four attacks are considered here.
Revised: To increase the reliability, four attacks are considered here.

Original: The results are shown in Figure 3.
Revised: The results are shown in Figure 4.

Original: We see that sg-FGSM consistently outperforms FGSM for all distortion level, although the improvement varies for different ε .
Revised: We see that sg-FGSM consistently outperforms FGSM for all distortion level, although the improvement varies for different ε .

Original: Furthermore, larger m leads to higher success rate due to the better estimation of the smooth part of gradient, and the beneﬁt starts to saturate after m ≥ 30 .
Revised: Furthermore, larger m leads to higher success rate due to the better estimation of the smooth part of gradient [Figure] and the beneﬁt starts to saturate after m ≥ 30 .

Original: For the smoothing factor σ , we ﬁnd neither too large nor too small value can work well.
Revised: For the smoothing factor σ , we ﬁnd neither too large nor too small value can work well

Revised: [Figure]

Revised: and the optimal σ is about 15 .

Original: Overly large σ will introduce a large bias in (8). Extremely small σ is unable to smooth the landscape enough.
Revised: Overly large σ will introduce a large bias in (8), and extremely small σ is unable to smooth the landscape enough [Figure] Therefore

Revised: [Figure]

Revised: in the subsequent sections

Original: Therefore, in the subsequent sections, we will use m = 20 , σ = 15 to estimate the smoothed gradient and only report the result for one distortion level.
Revised: [Figure] we will use m = [Figure] σ = 15 to estimate the smoothed gradient and only report the result for one distortion level

Original: [Figure]
Revised: [Figure]

Original: 7. E FFECTIVENESS
Revised: 7. E FFECTIVENESS

Original: IGSM attack We ﬁrst test the effectiveness of our method for single-model based attacks on non-targeted setting.
Revised: Single-model attack We ﬁrst test the effectiveness of our method for single-model based attacks on non-targeted setting.

Original: This to say T = 5 for sg-IGSM due to m = 20 . Speciﬁcally, we make  the number of gradient calculation per sample be ﬁxed 100  for a fair comparison.
Revised: To make a fair comparison, we ﬁx the number of gradient calculation per sample at 100 . Speciﬁcally, for sg-IGSM we have T = 5 due to m = 20 , whereas T =for IGSM.

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: We see that smoothed gradients do enhance the transferability dramatically for all the attacks considered.
Revised: We see that smoothed gradients do enhance the transferability dramatically for all the attacks considered.

Original: Please especially note those bold rates, where the improvements have reached about 30%.
Revised: Please especially note those bold rates, where the improvements have reached about 30%

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Ensemble attack In this part, we examine the ensemble-based attack on  targeted setting 2 .
Revised: Ensemble attack In this part, we examine the ensemble-based attack on the targeted setting 2 .

Original: We test whether the performance can be further improved by employing smoothed gradients.

Original: For targeted attacks, generating an adversarial example predicted by target models as a speciﬁc label is too hard, resulting in a very small success rate.
Revised: For targeted attacks, generating an adversarial example predicted by target models as a speciﬁc label is too hard, resulting in a very low success rate.

Original: We instead adopt the the Top-5 success rate as our criterion to better reﬂect the improvement of our method.
Revised: We instead adopt both Top-1 and Top-5 success rates as our criteria to better reﬂect the improvement of our method.

Original: A variety of model ensembles are examined [Table] and the results are reported in Table 
Revised: A variety of model ensembles are examined, and the results are reported in Table 4.

Original: [Table]
Revised: [Table]

Original: As we can see
Revised: As we can see

Original: [Table]
Revised: [Table]

Original: it is clear that smoothed gradient attacks outperform the corresponding normal ones by a remarkable large margin.
Revised: it is clear that smoothed gradient attacks outperform the corresponding normal ones by a remarkable large margin.

Original: More importantly, the improvement never be harmed compared to single-model case in Table 3, which implies that smoothed gradient can be effectively combined with ensemble method without compromise .
Revised: More importantly, the improvement never be harmed compared to single-model case in Table 3, which implies that smoothed gradient can be effectively combined with ensemble method without compromise .

Original: Momentum attack In this experiment, three networks of different architectures are selected.
Revised: Momentum attack In this experiment, three networks of different architectures are selected.

Original: As suggested in Dong et al.
Revised: As suggested in Dong et al.

Original: , we choose µ =1 , and all attacks are iterated for T = 5 with step size non- , we adversarial to Appendix C for details on this issue
Revised: (2018), we choose µ =

Original: [Table]
Revised: [Table]

Original:  α = 5 .
Revised: and all attacks are iterated for T = 5 with step size α = 5 .

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we report the results of non-targeted attacks of three attacks including mIGSM, sg-IGSM and mIGSM with smoothed gradient (sg-mIGSM).
Revised: we report the results of non-targeted attacks of three attacks including mIGSM, sg-IGSM and mIGSM with smoothed gradient (sg-mIGSM).

Original: It is shown that our method clearly outperforms momentum-based method for all the cases.
Revised: It is shown that our method clearly outperforms momentum-based method for all the cases.

Original: Moreover, by combining with the smoothed gradient, the effectiveness of momentum attacks can be further improved signiﬁcantly.
Revised: Moreover, by combining with the smoothed gradient, the effectiveness of momentum attacks can be further improved signiﬁcantly.

Original: [Table]
Revised: [Table]

Original:  R OBUSTNESS
Revised:  R OBUSTNESS

Original: Smoothed gradient attacks can be viewed as generating adversarial examples robust against Gaussian noise perturbation.
Revised: Smoothed gradient attacks can be viewed as generating adversarial examples robust against Gaussian noise perturbations.

Original: Therefore, we are interested in how robust is the adversarial example against other image transformations.
Revised: Therefore, we are interested in how robust is the adversarial example against other image transformations.

Original: To quantify the inﬂuence of transformations, we use the notion of destruction rate deﬁned by Kurakin et al.
Revised: To quantify the inﬂuence of transformations, we use the notion of destruction rate deﬁned by Kurakin et al.

Original: The lower is this rate, the more robust are the adversarial examples.
Revised: The lower is this rate, the more robust are the adversarial examples.

Original: Densenet121 and resnet34 are chosen as our source and target model, respectively; and four image transformations are considered: rotation, Gaussian noise, Gaussian blur and JPEG compression.
Revised: Densenet121 and resnet34 are chosen as our source and target model, respectively; and four image transformations are considered: rotation, Gaussian noise, Gaussian blur and JPEG compression.

Original: Figure 4 displays the results, which show that adversarial examples generated by our methods are more robust than those generated by vanilla methods.
Revised: Figure 5 displays the results, which show that adversarial examples generated by our methods are more robust than those generated by vanilla methods.

Original: This numerical result is interesting, since we only explicitly increase the robustness against Gaussian noise in generating adversarial examples.
Revised: This numerical result is interesting, since we only explicitly increase the robustness against Gaussian noise in generating adversarial examples.

Original: We speculate that the robustness can also transfer among different image transforms.[Figure]
Revised: We speculate that the robustness can also transfer among different image transforms.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we ﬁrst investigated the inﬂuence of model-speciﬁc factors on the adversarial transferability.
Revised: In this paper, we ﬁrst investigated the inﬂuence of model-speciﬁc factors on the adversarial transferability.

Original: It is found that  model architecture similarity plays a crucial role.
Revised: It is found that the model architecture similarity plays a crucial role.

Original: Moreover models with lower capacity and higher test accuracy are endowed with stronger capability for transfer-based attacks.
Revised: Moreover models with lower capacity and higher test accuracy are endowed with stronger capability for transfer-based attacks.

Original: we second demonstrate that the noisy part of gradient hinders the transfer of adversarial examples.
Revised: we second demonstrate that the non-smoothness of loss surface hinders the transfer of adversarial examples.

Original: Motivated by these understanding, we proposed the smoothed gradient attack which can enhance the transferability of adversarial examples dramatically.
Revised: Motivated by these understandings, we proposed the smoothed gradient attack that can enhance the transferability of adversarial examples dramatically.

Original: Furthermore, the smoothed gradient can be combined with both ensemble and momentum based attacks rather effectively. 
Revised: Furthermore, the smoothed gradient can be combined with both ensemble and momentum based approaches rather effectively. [Figure]

Original: A V ISUALIZATION OF S MOOTHED G RADIENT
Revised: A V ISUALIZATION OF S MOOTHED G RADIENT

Original: In this section, we provide an visualization understanding how the local average has the smoothing effect.
Revised: In this section, we provide an visualization understanding how the local average has the smoothing effect.

Original: We choose densenet121 as the model and visualize the saliency map of gradient ∇ x J ( x ) and the smoothed versions for varying m .
Revised: We choose densenet121 as the model and visualize the saliency map of gradient ∇ x J ( x ) and the smoothed versions for varying m .

Original: Two images are considered, and the results are shown in
Revised: Two images are considered, and the results are shown in

Original: Figure 5.
Revised: Figure 6.

Original: We can see that local average can smooth the gradient signiﬁcantly.
Revised: We can see that local average can smooth the gradient signiﬁcantly.

Original: Please refer to the work by Smilkov et al.
Revised: Please refer to the work by Smilkov et al.

Original: (2017) for more details.
Revised: (2017) for more details.

Original: [Figure]
Revised: [Figure]

Original: B I NFLUENCE OF STEP SIZE FOR TARGETED ATTACKS
Revised: B I NFLUENCE OF STEP SIZE FOR TARGETED ATTACKS

Original: When using IGSM to perform targeted black-box attacks
Revised: When using IGSM to perform targeted black-box attacks

Original: [Figure]
Revised: [Figure]

Original: there are two hyper parameters including number of iteration T and step size α .
Revised: there are two hyper parameters including number of iteration T and step size α .

Original: Here we explore their inﬂuence to the quality of adversarial examples generated.
Revised: Here we explore their inﬂuence to the quality of adversarial examples generated.

Original: The success rates are calculated on 5 , 000 images randomly selected according to description of Section 4.
Revised: The success rates are calculated on 5 , 000 images randomly selected according to description of Section 4.

Original: resnet152 and vgg16 bn are chosen as target models.
Revised: resnet152 and vgg16 bn are chosen as target models.

Original: The performance are evaluated by the average Top-5 success rate over the three ensembles used in Table
Revised: The performance are evaluated by the average Top-5 success rate over the three ensembles used in Table

Original: [Figure]
Revised: [Figure]

Original: Figure 6 shows that for the optimal step size α is very large, for instance in this experiment it is about 15 compared to the allowed distortion ε = 20 .
Revised: Figure 7 shows that for the optimal step size α is very large, for instance in this experiment it is about 15 compared to the allowed distortion ε = 20 .

Original: Both too large and too small step size will yield harm to the performances.
Revised: Both too large and too small step size will yield harm to the performances.

Original: It is worth noting that with small step size α = 5 , the large number of iteration provides worse performance than small number of iteration.
Revised: It is worth noting that with small step size α = 5 , the large number of iteration provides worse performance than small number of iteration.

Original: One possible explanation is that more iterations lead the optimizer to converge to a more overﬁtted solution.
Revised: One possible explanation is that more iterations lead the optimizer to converge to a more overﬁtted solution.

Original: In contrast, a large step size can prevent it and encourage the optimizer to explore more model-independent area, thus more iteration is better.
Revised: In contrast, a large step size can prevent it and encourage the optimizer to explore more model-independent area, thus more iteration is better.

Original: C A DDITIONAL R ESULTS
Revised: C A DDITIONAL R ESULTS

Original: How Model Capacity and Test Accuracy Affect Transferability
Revised: How Model Capacity and Test Accuracy Affect Transferability

Original: In this part, we additionally explore the impact of model capacity and test accuracy by using resnet152 as our target model and perform transfer-based attacks against from a variety of models and the results shown in Figure
Revised: In this part, we additionally explore the impact of model capacity and test accuracy by using resnet152 as our target model and perform transfer-based attacks against from a variety of models and the results shown in Figure

Original: [Figure]
Revised: [Figure]

Original: The observations are consistent with the observation in Section 5.2.
Revised: The observations are consistent with the observation in Section 5.2.

Original: [Figure]
Revised: [Figure]

Original: Visualization of Decision Boundary In this part, we provide additional results on the visualization of the decision boundary.
Revised: Visualization of Decision Boundary In this part, we provide additional results on the visualization of the decision boundary.

Original: Different Figure 2b, we here consider the sign of two directions, since the attack method actually updates using the sign ( g ) rather than g .
Revised: Different Figure 3b, we here consider the sign of two directions, since the attack method actually updates using the sign ( g ) rather than g .

Original: In Figure 8, we show the decision boundary.
Revised: In Figure 9, we show the decision boundary.

Original: Each point corresponds to perturbed image: clip (
Revised: Each point corresponds to perturbed image: clip (

Original: [Equation]
Revised: [Equation]

Original: u sign [Equation] G A )
Revised: u sign ( G A )

Original: [Equation]
Revised: [Equation]

Original: v sign ( h A ) , 0 , 255) .
Revised: v sign ( h A ) , 0 , 255) .

Original: Here the model A is resnet34, and the deﬁnitions of G A and h A are the same as before.
Revised: Here the model A is resnet34, and the deﬁnitions of G A and h A are the same as before.

Original: The color denotes the label predicted by the target model, and the gray color corresponds to the ground-truth label.
Revised: The color denotes the label predicted by the target model, and the gray color corresponds to the ground-truth label.

Original: We can see that the smoothed gradient G A is indeed more transferable than the noise part h A .
Revised: We can see that the smoothed gradient G A is indeed more transferable than the noise part h A .

Original: Additional Results for the Effectiveness of Smoothed Gradient Attacks
Revised: Additional Results for the Effectiveness of Smoothed Gradient Attacks

Original: In this part, we present some additional results to show the effectiveness of smoothed gradient attack.
Revised: In this part, we present some additional results to show the effectiveness of smoothed gradient attack.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D
Revised: [Figure]

Original: T HE E XAMINED
Revised: [Figure]

Original: I
Revised: [Figure]

Original: [Figure]
Revised: [Figure]