E NHANCING THE T RANSFERABILITY OF A DVERSARIAL E XAMPLES WITH N OISE R EDUCED G RADIENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks provide state-of-the-art performance for many applications of interest.
Unfortunately they are known to be vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs.
Moreover, the perturbations can transfer across models : adversarial examples generated for a speciﬁc model will often mislead other unseen models.
Consequently the adversary can leverage it to attack against the deployed black-box systems.
In this work, we demonstrate that the adversarial perturbation can be decomposed into two components: model-speciﬁc and data-dependent one, and it is the latter that mainly contributes to the transferability.
Motivated by this understanding, we propose to craft adversarial examples by utilizing the noise reduced gradient (NRG) which approximates the data-dependent component.
Experiments on various classiﬁcation models trained on ImageNet demonstrates that the new approach enhances the transferability dramatically.
We also ﬁnd that low-capacity models have more powerful attack capability than high-capacity counterparts, under the condition that they have comparable test performance.
These insights give rise to a principled manner to construct adversarial examples with high success rates and could potentially provide us guidance for designing effective defense approaches against black-box attacks.
I NTRODUCTION
With the resurgence of neural networks, more and more large neural network models are applied in real-world applications, such as speech recognition, computer vision, etc.
While these models have exhibited good performance, recent works (Szegedy et al.
; Goodfellow et al.
(2014)) show that an adversary is always able to fool the model into producing incorrect outputs by manipulating the inputs maliciously.
The corresponding manipulated samples are called adversarial examples .
However, how to understand this phenomenon (Goodfellow et al.
; Tram ` er et al.
(2017b))
and how to defend against adversarial examples effectively (Kurakin et al.
; Tram`er et al. (2017a);
Carlini & Wagner (2017)) are still open questions.
Meanwhile it is found that adversarial examples can transfer across different models, i.e., the adversarial examples generated from one model can also fool another model with a high probability.
We refer to such property as transferability , which can be leveraged to attack black-box systems (Papernot et al.
; Liu et al.
The phenomenon of adversarial vulnerability was ﬁrst introduced and studied in Szegedy et al.
The authors modeled the adversarial example generation as an optimization problem solved by box-constraint L-BFGS, and also attributed the presence of adversarial examples to the strong nonlinearity of deep neural networks.
Goodfellow et al.
(2014) argued instead that the primary cause of the adversarial instability is the linear nature and the high dimensionality, and the view yielded the fast gradient sign method (FGSM).
Similarly based on an iterative linearization of the classiﬁer,
Moosavi-Dezfooli et al.
(2015) proposed the DeepFool method.
In Kurakin et al.
; Tram`er et al.
(2017a), it was shown that the iterative gradient sign method provides stronger white-box attacks but does not work well for black-box attacks.
Liu et al.
(2016) analyzed the transferability of adversarial examples in detail and proposed ensemble-based approaches for effective black-box attacks.
In Carlini & Wagner (2017) it was demonstrated that high-conﬁdence adversarial examples that are strongly misclassiﬁed by the original model have stronger transferability.
In addition to crafting adversarial examples for attacks, there exist lots of works on devising more effective defense.
Papernot et al.
(2015) proposed the defensive distillation.
Goodfellow et al.
(2014) introduced the adversarial training method, which was examined on ImageNet by Kurakin et al.
(2016) and Tram`er et al.
(2017a).
Lu et al.
(2017) utilized image transformation, such as rotation, translation, and scaling, etc, to alleviate the harm of the adversarial perturbation.
Instead of making the classiﬁer itself more robust, several works (Li & Li (2016);
Feinman et al.
(2017)) attempted to detect the adversarial examples, followed by certain manual processing.
Unfortunately, all of them can be easily broken by designing stronger and more robust adversarial examples (Carlini & Wagner (2017); Athalye & Sutskever (2017)).
In this work, we give an explanation for the transferability of adversarial examples and use the insight to enhance black-box attacks.
Our key observation is that adversarial perturbation can be decomposed into two components: model-speciﬁc and data-dependent one.
The model-speciﬁc component comes from the model architecture and random initialization, which is noisy and represents the behavior off the data manifold.
In contrast, the data-dependent component is smooth and approximates the ground truth on the data manifold.
We argue that it is the data-dependent part that mainly contributes to the transferability of adversarial perturbations across different models.
Based on this view, we propose to construct adversarial examples by employing the data-dependent component of gradient instead of the gradient itself.
Since this component is estimated via noise reduction strategy, we call it noise-reduced gradient (NRG) method.
Benchmark on the ImageNet validation set demonstrates that the proposed noise reduced gradient used in conjunction with other known methods could dramatically increase the success rate of black-box attacks.
to perform black-box attacks over ImageNet validation set.
We also explore the dependence of success rate of black-box attacks on model-speciﬁc factors, such as model capacity and accuracy.
We demonstrate that models with higher accuracy and lower capacity show stronger capability to attack unseen models.
Moreover this phenomenon can be explained by our understanding of transferability, and may provide us some guidances to attack unseen models.
B ACKGROUND 2. A DVERSARIAL EXAMPLES
We use f : R d (cid:55)→ R K to denote the model function, which is obtained via minimizing the empirical risk over training set.
For simplicity we omit the dependence on the trainable model parameter θ , since it is ﬁxed in this paper.
For many applications of interest, we always have d (cid:29) 1 and K
=
o
According to the local linear analysis in Goodfellow et al.
, the high dimensionality makes f ( x ) inevitably vulnerable to the adversarial perturbation.
That is, for each x , there exists a small perturbation η that is nearly imperceptible to human eyes, such that
[Equation]
We call η adversarial perturbation and correspondingly x adv :=
x + η adversarial example.
In this work, we mainly study the adversarial examples in the context of deep neural networks, though they also exist in other models, for example, support vector machine (SVM) and decision tree, etc (Papernot et al.
We call the attack (1) a non-targeted attack because the adversary has no control of the output other than requiring x to be misclassiﬁed by the model. In contrast, a targeted attack aims at fooling the model into producing a wrong label speciﬁed by the adversary. I.e.
[Equation]
In contrast, we call the attack (1) a non-targeted attack.
In the black-box attack setting, the adversary has no knowledge of the target model (e.g. architecture and parameters) and is not allowed to query the model.
That is, the target model is a pure black-box.
However the adversary can construct adversarial examples on a local model (also called the source model) that is trained on the same or similar dataset with the target model.
Then it deploys those adversarial examples to fool the target model.
This is typically referred to as a black-box attack, as opposed to the white-box attack whose target is the source model itself.
2. M ODELLING OF GENERATING ADVERSARIAL EXAMPLES
In general, crafting adversarial perturbation can be modeled as following optimization problem,
[Equation]
where J is a loss function measuring the discrepancy between the prediction and ground truth; (cid:107) ·
[Equation]
is the metric to quantify the magnitude of the perturbation.
For image data, there is also an implicit constraint: x adv ∈ [0 , 255] d , with d being the number of pixels.
In practice, the commonly choice of
J is the cross entropy .
Carlini & Wagner (2016) introduced a loss function that directly manipulates the output logit instead of probability.
This loss function is also adopted by many works.
As for the measurement of distortion, The best metric should be human eyes, which is unfortunately difﬁcult to quantify.
In practice, (cid:96) ∞ and (cid:96) 2 norms are commonly used.
Ensemble-based approaches To improve the strength of adversarial examples, instead of using a single model, Liu et al.
(2016) suggest using a large ensemble consisting of f 1 , f 2 , · · · , f Q as our source models.
Although there exist several ensemble strategies, similar as Liu et al. , we only consider the most commonly used method, averaging the predicted probability of each model. The corresponding objective can be written as
[Equation]
where w i are the ensemble weights with (cid:80) w i = 1 .
Objectives (2) and (3) are for non-targeted attacks, and the targeted counterpart can be derived similarly.
2. O PTIMIZER
There are various optimizers to solve problem (2) and (3).
In this paper, we mainly use the normalized-gradient based optimizer.
Gradient Based Method This method (Goodfellow et al.
attempts to solve (2) by performing only one step iteration,
[Equation]
where g ( x ) is a normalized gradient vector.
For (cid:96) ∞ -attack ,
[Equation]
similarly for (cid:96) q -attack,
[Equation]
Both of them are called fast gradient based method (FGBM) and are empirically shown to be fast.
Also they have very good transferability (Kurakin et al.
; Tram`er et al. (2017a)) though not optimal.
So it is worth considering this simple yet effective optimizer.
Iterative Gradient Method This method (Madry et al.
(2017); Kurakin et al.
(2016)) performs the projected normalized-gradient ascent
[Equation]
for k steps, where x 0 is the original clean image; clip x ,ε ( · ) is the projection operator to enforce x [Equation] and α is the step size.
In analogous to the fast gradient based method, the normalized gradient g ( x ) is chosen as g ∞ ( x ) for (cid:96) ∞ -attack, called iterative gradient sign method (IGSM), and g q ( x ) for (cid:96) qattack.
The fast gradient based method (4) can be viewed as a special case of (7) when α = ε, k = 1 .
E NHANCING THE TRANSFERABILITY OF ADVERSARIAL EXAMPLES 3. D ATA - DEPENDENT COMPONENTS ENABLE TRANSFERABILITY
There are few articles trying to understand why adversarial examples can transfer between models, though it is extremely important for performing black-box attacks and building successful defenses.
To the best of our knowledge, the only two works are (Liu et al.
; Tram`er et al. (2017b)), which suggested that the transferability comes from the similarity between the decision boundaries of the source and target models, especially in the direction of transferable adversarial examples.
Tram`er et al.
(2017b) also claimed that transferable adversarial examples span a contiguous subspace.
To investigate the transfer phenomenon, we ask a further question: what similarity between the different models A and B that enables transferability of adversarial examples across them?
Since the model A and B have a high performance on the same dataset, they must have learned a similar function on the data manifold .
However, the behaviour of the models off the data manifold can be different.
This is determined by the architectures of the models and random initializations, both of which are data-independent factors.
This clearly hints us to decompose the perturbation into two factors on and off the data manifold. We referred to them as data-dependent and model-speciﬁc components.
We hypothesize that the component on the data manifold mainly contributes to the transferability from A to B , since this component captures the data-dependent information shared between models.
The model-speciﬁc one contributes little to the transferability due to its different behaviours off the data manifold for different models.
[Figure]
f A ⊥ (cid:107) 2 , and the same color indicates the same predicted label.
More details can be found in Section 6.3.
Take η = ∇ f A as the adversarial perturbation crafted from model A , we illustrate this explanation in Figure 1.
In the left panel, the decision boundaries of two models are similar in the inter-class area.
As can be observed, ∇ f A can mislead both model A and B .
Then we decompose the perturbation into two parts, a data-dependent component ∇ f A (cid:107) and a model-speciﬁc one ∇ f A ⊥ , respectively.
Since ∇ f A (cid:107) is almost aligned to the inter-class deviation (red line), i.e. on the data manifold, it can attack model B easily.
However, The model-speciﬁc ∇ f
A ⊥ contributes little to the transfer from A to B , though it can successfully fool model A with a very small distortion.
In the right panel, we plot the decision boundaries of resnet34 (model A ) and densenet121 (model
B ) for ImageNet.
The horizontal axis represents the direction of data-dependent component ∇ f A (cid:107) estimated by our proposed NRG method (8) from resnet34 ; and the vertical axis depicts the direction of model-speciﬁc component ∇ f A ⊥ .
It is easily observed that for model A resnet34 , a small perturbation in both directions can produce wrong classiﬁcation.
However, when applied to model B densenet121 , a large perturbation along the ∇
f A ⊥ direction cannot change the classiﬁcation results, while a small perturbation along the data-dependent one can change the prediction easily.
[Figure] This understanding suggests that to increase success rates of black-box adversarial attacks, we should enhance the data-dependent component. We hence propose the NRG method which achieves that by reducing the model-speciﬁc noise, as elaborated in the following.
3. N OISE - REDUCED GRADIENT METHODS
Noise-reduced gradient (NRG)
The model-speciﬁc component ∇ f ⊥ inherits from the random initialization, so it must be very noisy, i.e. high-frequency (the same observation is also systematically investigated in Balduzzi et al.
; while ∇ f (cid:107) is smooth, i.e. low-frequency, since it encodes the knowledge learned from training data.
Therefore, local average can be applied to remove the noisy model-speciﬁc information, yielding an approximation of the data-dependent component,
[Equation]
where the m is the number of samples chosen for averaging.
We call (8) noise-reduced gradient (NRG), which captures more data-dependent information than ordinary gradient ∇ f .
To justify the effectiveness of this method, similar to Smilkov et al.
, we visualize NRG for various m in
Figure 2.
As shown, larger m leads to a smoother and more data-dependent gradient, especially for m = 100 the NRG can capture the semantic information of the obelisk.
The noisy model-speciﬁc information of ∇ f could mislead the optimizer into the solutions that are overﬁtting to the speciﬁc source model, thus generalizing poorly to the other model.
Therefore we propose to perform attacks by using ∇ f (cid:107) in Eq.(8), instead of ∇ f , which can drive the optimizer towards the solutions that are more data-dependent.
Noise-reduced Iterative Sign Gradient Method (nr-IGSM)
The iterative gradient sign method mounted by NRG can be written as,
[Equation]
The special case k = 1 , α = ε is called noise-reduced fast gradient sign method (nr-FGSM) accordingly.
For (cid:96) q -attack, the noise-reduced version is similar, replacing the second equation of (9) by
[Equation]
For any general optimizer, the corresponding noise-reduced counterpart can be derived similarly.
E XPERIMENT S ETTING
To justify and analyze the effectiveness of NRG for enhancing the transferability, we use the start-ofthe-art classiﬁcation models trained on ImageNet dataset.
We elaborate the details in the following.
Dataset We use the ImageNet ILSVRC2012 validation set that contains 50 , 000 samples.
For each attack experiment, we randomly select 5 , 000 images that can be correctly recognized by all the models, since it is meaningless to construct adversarial perturbations for the images that target models cannot classify correctly.
And for the targeted attack experiments, each image is speciﬁed by a random wrong label.
Models We use the pre-trained models provided by PyTorch including resnet18, resnet34, resnet50, resnet101, resnet152, vgg11 bn, vgg13 bn, vgg16 bn, vgg19 bn, densenet121, densenet161, densenet169, densenet201, alexnet, squeezenet1 1 .
The Top-1 and Top-5 accuracies of them can be found on website 1 .
To increas the reliability of experiments, all the models have been used, however for a speciﬁc experiment we only choose several of them to save computational time.
Criteria Given a set of adversarial examples, { ( x adv 1 , y true 1 ) , ( x adv 2 , y true 2 ) , . . .
, ( x advN , y true N ) } , We calculate their Top-1 success rates fooling a given model F ( x ) by
[Equation]
If F is the model used to generate adversarial examples, then the rate indicates the the white-box attack performance.
For targeted attacks, each image x adv is associated with a pre-speciﬁed label y target (cid:54) =
y true .
Then we evaluate the performance of the targeted attack by the following Topsuccess rate ,
[Equation]
The corresponding Top-5 rates can be computed in a similar way.
Attacks Throughout this paper the cross entropy 2 is chosen as our loss function J .
We measure the distortion by two distances: (cid:96) ∞ norm and scaled (cid:96) 2 norm, i.e. root mean square deviation (RMSD) (cid:113)(cid:80) di
=1 η 2 i /d , where d is the dimensionality of inputs.
As for optimizers, both FGSM and IGSM are considered.
E FFECTIVENESS OF NOISE - REDUCED METHODS
In this section we demonstrate the effectiveness of our noise-reduced gradient technique by combining it with several commonly-used methods.
5. S INGLE - MODEL BASED APPROACHES
Fast gradient based method We ﬁrst examine the combination of noise reduced gradient and fast gradient based methods.
The success rates of FGSM and nr-FGSM are summarized in Table(results of FGM (cid:96) 2 -attacks can be found in Appendix C).
We observe that, for any pair of blackbox attack, nr-FGSM performs better than original FGSM consistently and dramatically.
Even the white-box attacks (the diagonal cells) also have improvements.
This result implies that the direction of noise-reduced gradient is indeed more effective than the vanilla gradient for enhancing the transferability.
Iterative gradient sign method One may argue that the above comparison is unfair, since nrFGSM consumes more computational cost than FGSM, determined by m : number of perturbed inputs used for local average.
Here we examine IGSM and nr-IGSM under the same number of gradient calculations.
Table 2 presents the results.
Except for the attacks from alexnet , as we expect, adversarial examples generated by nr-IGSM indeed transfer much more easily than those generated by IGSM.
This indicates that the noise-reduced gradient (NRG) does guide the optimizer to explore the more data-dependent solutions.
Some observations By comparing Table 1 with Table 2, we ﬁnd that large models are more robust to adversarial transfer than small models, for example the resnet152 .
This phenomenon has also been extensively investigated by Madry et al.
It also shows that the transfer among the
[Table]
The cell ( S , T ) indicates the percentage of adversarial examples generated for model S (row) that successfully fool model T (
[Table] models with similar architectures is much easier, such as the 90%+ transferability between vggstyle networks in Table 2.
This implies that the model-speciﬁc component also contributes to the transfer across models with similar architectures.
Additionally in most cases, IGSM generates stronger adversarial examples than FGSM except the attacks against alexnet .
This contradicts the claims in Kurakin et al.
(2016) and Tram`er et al. (2017a) that adversarial examples generated by FGSM transfer more easily than the ones of IGSM.
However our observation is consistent with the conclusions of Carlini & Wagner (2016): the higher conﬁdence (smaller cross entropy) adversarial examples have in the source model, the more likely they will transfer to the target model.
We conjecture that this is due to the inappropriate choice of hyperparameters, for instance
α = 1 , k = min( ε + 4 , 1 . 24 ε ) in Kurakin et al.
(2016) are too small, and the solution has not ﬁt the source model enough (i.e. underﬁtting).
When treated as a target model to be attacked, the alexnet is signiﬁcantly different from those source models in terms of both architecture and test accuracy.
And therefore, the multiple iterations cause the IGSM to overﬁt more than FGSM, producing a lower fooling rate.
These phenomena clearly indicate that we should not trust the objective in Eq.
(2) completely, which might cause the solution to overﬁt to the source model-speciﬁc information and leads to poor transferability.
Our noise reduced gradient technique regularizes the optimizer by removing the modelspeciﬁc information from the original gradients, and consequently, it can converge to a more datadependent solution that has much better cross-model generalization capability.
5. E NSEMBLE BASED APPROACHES
In this part, we apply NRG method into ensemble-based approaches described in Eq.
Due to the high computational cost of model ensembles, we select 1 , 000 images, instead of 5 , 000 , as our evaluation set.
For non-targeted attacks, both FGSM, IGSM and their noise reduced versions are tested.
The Topsuccess rates of IGSM attacks are nearly saturated, so we report the corresponding Top-5 rates in
Table 3(a) to demonstrate the improvements of our methods more clearly.
The results of FGSM and nr-FGSM attacks can be found in Appendix C.
For targeted attacks, to generate an adversarial example predicted by unseen target models as a speciﬁc label, is much harder than generating non-targeted examples.
Liu et al.
(2016) demonstrated that single-model based approaches are very ineffective in generating targeted adversarial examples.
That is why we did not consider targeted attacks in Section 5.1.
Different from the non-targeted attacks, we ﬁnd that targeted adversarial examples are sensitive to the step size α used in the optimization procedures (6) and (9).
After trying lots of α , we ﬁnd that a large step size is necessary for generating strong targeted adversarial examples .
Readers can refer to Appendix A for more detailed analysis on this issue, though we cannot fully understand it.
Therefore we use a much larger step size compared to the non-target attacks.
The Top-5 success rates are reported in Table 3(b).
By comparing success rates of normal methods and our proposed NRG methods in Table 3 for both targeted and non-targeted attacks, we observed that NRG methods outperform the corresponding normal methods by a remarkable large margin in this scenario.
The corresponding Top 1 success rates can be found in Appendix C ( 7 and
[Table]
[Table]
[Table]
A NALYSIS 6. I NFLUENCE OF HYPER PARAMETERS
In this part
[Table]
we explore the sensitivity of hyper parameters m
[Table]
σ when applying our NRG methods for black-box attacks.
We take nr-FGSM approach as a testbed over the selected evaluation set described in Section 4.
Four attacks are considered here [Table] and the results are shown in Figure 3.
It is not surprising that larger m leads to higher fooling rate for any distortion level ε due to the better estimation of the data-dependent direction of the gradient.
We ﬁnd there is an optimal value of σ inducing the best performance.
Overly large σ will introduce a large bias in (8).
Extremely small σ is unable to remove the noisy model-speciﬁc information effectively, since noisy components of gradients of different perturbed inputs are still strongly correlated for small σ .
Moreover the optimal σ varies for different source models, and in this experiment it is about 15 for resnet18 , compared to 20 for densenet161 . [Figure]
6. R OBUSTNESS ANALYSIS OF ADVERSARIAL EXAMPLES
In this part, we preliminarily explore the robustness of adversarial perturbations to image transformations, such as rotation, scaling, blurring, etc.
This property is particularly important in practice, since it directly affects whether adversarial examples can survive in the physical world (Kurakin et al.
; Athalye & Sutskever (2017); Lu et al.
To quantify the inﬂuence of transformations
[Figure]
we use the notion of destruction rate deﬁned in Kurakin et al.
[Equation]
where N is the number of images used to estimate the destruction rate, T ( · ) is an arbitrary image transformation.
The function c ( x ) indicates whether x is classiﬁed correctly:
c ( x ) := (cid:26) 1 , if image x is classiﬁed correctly 0 , otherwise
And thus, the above rate R describes the fraction of adversarial images that are no longer misclassiﬁed after the transformation T ( · ) .
Densenet121 and resnet34 are chosen as our source and target model, respectively; and four image transformations are considered: rotation, Gaussian noise, Gaussian blur and JPEG compression.
Figure 4 displays the results, which show that adversarial examples generated by our proposed NRG methods are much more robust than those generated by vanilla methods.
[Figure]
validation set being 26 ( shown in Figure 8 of Appendix C )
6. V ISUALIZATION OF DECISION BOUNDARY
In this section we study the decision boundaries of different models to help us understand why
NRG-based methods perform better.
Resnet34 is chosen as the source model and nine target models are considered, including resnet18, resnet50, resnet101, resnet152, vgg11 bn, vgg16 bn, vgg19 bn, densenet121, alexnet .
The ∇ f (cid:107) is estimated by (8) with m = 1000 , σ = 15 .
Each point ( u, v ) in this 2-D plane corresponds to the image perturbed by u and v along sign (cid:0) ∇ f (cid:107) (cid:1) and sign ( ∇ f ⊥ ) , respectively, i.e.
[Equation]
(13) where x represents the original image.
We randomly select one image that can be recognized by all the models examined.
The Figure 5 shows the decision boundaries.
We also tried a variety of other source models and images, all the plots are similar.
From the aspect of changing the predicted label, the direction of sign (cid:0) ∇ f (cid:107) (cid:1) is as sensitive as the direction of sign ( ∇ f ⊥ ) for the source model resnet34 .
However, except alexnet all the other target models are much more sensitive along sign (cid:0) ∇ f (cid:107) (cid:1) than sign ( ∇ f ⊥ ) .
This is consistent with the argument in Section 3.1.
Removing ∇ f ⊥ from gradients can also be thought as penalizing the optimizer along the model-speciﬁc direction, thus avoiding converging to a source model-overﬁtting solution that transfers poorly to the other target models.
Moreover, we ﬁnd that, along the sign (cid:0) ∇ f (cid:107) (cid:1) , the minimal distance u to produce adversarial transfer varies for different models.
The distances for complex models are signiﬁcantly larger than those of small models, for instance, the comparison between resnet152 and resnent50 .
This provides us a geometric understanding of why big models are more robust than small models, as observed in
Section 5.1.
6. I NFLUENCE OF MODEL CAPACITY AND ACCURACY ON ATTACK CAPABILITY
In earlier experiments (Table 1 and Table 2), we can observe that adversarial examples crafted from alexnet generalize worst across models, for example nr-FGSM attack of alexnet → resnet152 only achieves 19 .
3 percent.
However, attacks from densenet121 consistently perform well for any target model, for example 84 .
3% of nr-IGSM adversarial examples can transfer to vgg19 bn , whose architecture is completely different from densenet121 .
This observation indicates that different models can exhibit different performances in attacking the same target model.
Now we attempt to ﬁnd the principle behind this important phenomenon, which can guide us to choose a better local model to generate adversarial examples for attacking the remote black-box system.
We select vgg19 bn and resnet152 as our target model, and use a variety of models to perform both
FGSM and IGSM attacks against them.
The results are summarized in Figure 6.
The horizontal axis is the Top-1 test error, while the vertical axis is the number of model parameters that roughly quantiﬁes the model capacity.
We can see that the models with powerful attack capability concentrate in the bottom left corner, while fooling rates are very low for those models with large test error and number of parameters 3 .
We can obtain an important observation that the smaller test error and lower capacity a model has, the stronger its attack capability is .
Here we attempt to explain this phenomenon from our understanding of transferability in Section 3.1.
A smaller test error indicates a lower bias for approximating the ground truth along the data manifold.
On the other hand, a less complex model might lead to a smaller model-speciﬁc component ∇ f ⊥ , facilitating the data-dependent factor dominate.
In a nutshell, the model with small ∇ f ⊥ and large ∇ f (cid:107) can provide strong adversarial examples that transfer more easily.
This is consistent with our arguments presented previously.
[Figure]
N u m b e r o f p a r a m t e r s ( m illi o n ) [Figure]
C  C ONCLUSION
In this paper, we have veriﬁed that an adversarial perturbation can be decomposed into two components: model-speciﬁc and data-dependent ones.
And it is the latter that mainly contributes to the transferability of adversarial examples.
Based on this understanding, we proposed the noise-reduced gradient (NRG) based methods to craft adversarial examples, which are much more effective than previous methods.
We also show that the models with lower capacity and higher test accuracy are endowed with stronger capability for black-box attacks.
In the future, we will consider combining NRG-based methods with adversarial training to defend against black-box attacks.
The component contributing to the transferability is data-dependent, which is intrinsically low-dimensional, so we hypothesize that black-box attacks can be defensible.
On the contrary, the white-box attack origins from the extremely high-dimensional ambient space, thus its defense is much more difﬁcult.
Another interesting thread of future research is to learn stable features beneﬁcial for transfer learning by incorporating our NRG strategy, since the reduction of model-speciﬁc noise can lead to more accurate information on the data manifold.
A PPENDIX A: I NFLUENCE OF STEP SIZE FOR TARGETED ATTACKS
When using IGSM to perform targeted black-box attacks, there are two hyper parameters including number of iteration k and step size α .
Here we explore their inﬂuence to the quality of adversarial examples generated.
The success rates are valuated are calculated on 1 , 000 images randomly selected according to description of Section 4.
resnet152 and vgg16 bn are chosen as target models.
The performance are evaluated by the average Top-5 success rate over the three ensembles used in
Table 3(b).
Figure 7 shows that for the optimal step size α is very large, for instance in this experiment it is about 15 compared to the allowed distortion ε = 20 .
Both too large and too small step size will yield harm to the attack performances.
It interesting noting that with small step size α = 5 , the large number of iteration provides worser performance than small number of iteration.
One possible explain is that more iterations lead optimizers to converge to more overﬁt solution.
In contrast, a large step size can prevent it and encourage the optimizer to explore more model-independent area, thus more iteration is better.
[Figure]
A PPENDIX B: MNIST EXPERIMENTS
To further conﬁrm the inﬂuence of model redundancy on the attack capability, we conduct an additional experiment on MNIST dataset.
We use the fully networks of D layers with width of each layer being 500 , e.g. the architecture of model with D = 2 is 784 − 500 − 500 − 10 .
Models of depth 1 , 3 , 5 , 9 are considered in this experiment.
The Top-1 success rates of cross-model attacks are reported in Table 4.
The results of Table 4 demonstrate the low-capacity model has much stronger attack capability than large-capacity.
This is consistent with our observation in Section 6.4.
[Table]
A PPENDIX C: S OME ADDITIONAL EXPERIMENTAL RESULTS
[Table] [Table]
[Table] [Table]
[Figure]
N u m b e r o f p a r a m t e r s ( m illi o n )
[Figure] [Figure]