R ETHINKING A DVERSARIAL T RANSFERABILITY FROM A D ATA D ISTRIBUTION P ERSPECTIVE
A BSTRACT
Adversarial transferability enables attackers to generate adversarial examples from the source model to attack the target model, which has raised security concerns about the deployment of DNNs in practice.
In this paper, we rethink adversarial transferability from a data distribution perspective and further enhance transferability by score matching based optimization.
We identify that some samples with injecting small Gaussian noise can fool different target models, and their adversarial examples under different source models have much stronger transferability.
We hypothesize that these samples are in the low-density region of the ground truth distribution where models are not well trained.
To improve the attack success rate of adversarial examples, we match the adversarial attacks with the directions which effectively decrease the ground truth density.
We propose
Intrinsic Adversarial Attack ( IAA ), which smooths the activation function and decreases the impact of the later layers of a given normal model, to increase the alignment of adversarial attack and the gradient of joint data distribution.
We conduct comprehensive transferable attacks against multiple DNNs and show that
our
IAA can boost the transferability of the crafted attacks in all cases and go beyond state-of-the-art methods.
I NTRODUCTION
Deep neural networks (DNNs) are widely used in various safety-critical ﬁelds, but they are vulnerable to adversarial examples (Szegedy et al., 2013).
Adversarial attacks are imperceptible to humans but catastrophic for the DNNs and can be transferred between different models (Goodfellow et al., 2015; Liu et al., 2017).
Adversarial transferability enables attackers to generate adversarial examples from the source model to attack unknown target models, which has raised security concerns about the deployment of DNNs in practice.
Understanding the essence of adversarial transferability is a fundamental problem in deep learning.
On the one hand, some works show that the characteristics of the source model, such as model architecture (Wu et al., 2019), model capacity (Tram`er et al., 2017), and test accuracy (Wu & Zhu, 2020), inﬂuence adversarial examples’ transferability.
On the other hand, some works think that the data-relevant information may be the key factor for adversarial transferability.
Ilyas et al.
(2019) explain that adversarial perturbations are non-robust features and not meaningless bugs, but it is hard to speciﬁcally deﬁne non-robust features.
We want to further study transferability quantitatively from the data distribution perspective.
It has been empirically observed that DNNs are relatively robust to random noise (Fawzi et al., 2016).
However, in this work we ﬁnd an intriguing phenomenon: some samples are sensitive to
Gaussian noise, in the sense that injecting small Gaussian noise into these samples can fool different models trained on the same dataset.
Furthermore, their adversarial counterparts generated by different source models have much stronger transferability against different target models than other samples.
We hypothesize that these samples are in the low-density regions of the ground truth distribution both source and target models are trained on, and models are not well trained in these regions.
Thus predictions of these samples are easy to be perturbed and even not robust to small random noises.
We denote this kind of data as Low-Density Data (LDD), while others as High-Density
Data (HDD).
As shown in Fig.
1 (Left), the attack success rate against different target models of
LDD with different strengths of Gaussian noise is much higher than that of HDD.
Furthermore, in Fig.
1 (Right), the adversarial counterparts of LDD have much stronger transferability than the adversarial counterparts of HDD (see Appendix B for details).
This phenomenon reveals that the location of data plays a vital role in adversarial transferability and the adversarial examples of samples in the low-density region are strongly transferable.
The most efﬁcient direction towards the low-density region is −∇ x log p D ( x , y ) , where p D ( x , y ) is the ground truth density of natural data.
We name this direction Intrinsic Attack because it doesn’t depend on the models and only depends on the ground truth distribution.
Thus, we propose to match the adversarial attack with intrinsic attack for generating strong transferable adversarial examples.
We explore the potential of a classiﬁer p θ, Λ ( y | x ) with parameters θ and structure hyper-parameters Λ (see Sec. 3.1) to generate more transferable adversarial examples by aligning adversarial attack with intrinsic attack −∇ x log p D ( x , y ) .
The adversarial attack of p θ, Λ ( y | x ) is usually generated by
PGD/FGSM method, and is determined by −∇ x log p θ, Λ ( y | x ) .
We match the A lignment between the A dversarial attack and I ntrinsic attack ( AAI ), E p D
( x ,y ) (cid:104) ∇
[Table]
x , y )
[Table]
, by modifying the structure parameters Λ for a pre-trained network.
In order to maximize AAI , we should make p θ, Λ ( y | x ) smoother.
Otherwise, ∇ x log p θ, Λ ( y | x ) will oscillate frequently and hard to match ∇ x log p D ( x , y ) .
For the commonly used ReLU network, we can smooth it by replacing ReLU activation with Softplus (Nair & Hinton, 2010) with little change of the model’s output.
Maennel et al.
(2020) show that the early layers of a network learn the local statistics of the data distribution better than the later layers, which motivates us to decrease the impact of later layers when generating adversarial examples to utilize the data distribution-relevant information.
We can closely match the adversarial attack with the intrinsic attack −∇ x log p D ( x , y ) and improve the adversarial transferability by optimizing structure hyper-parameters Λ to maximize
AAI as the objective function.
We name our method as Intrinsic Adversarial Attack (IAA) .
There are some interesting observations in our IAA experiments.
Firstly, we ﬁnd that the test accuracy of the source model may not be important.
As shown in Fig 2, the accuracy of the pre-trained model with Softplus β =15 is around 60 % , but the adversarial transferability of this model is much stronger than the model with Softplus β =45 .
Secondly, although the existing methods (Madry et al., 2018; Wu et al., 2019) can signiﬁcantly decrease the top-1 accuracy of the target models, the topaccuracy is still high.
IAA can both decrease the top-1 accuracy and top-5 accuracy.
Furthermore, the existing methods (Xie et al., 2019; Wu et al., 2019) can just slightly improve the one-step attack under different strengths, while our IAA surpasses the existing methods by a large margin.
These phenomena verify our hypothesis that IAA pulls examples to the low-density region, which causes prediction difﬁculty to the target models.
Our main contributions are summarized below:
• We propose an effective metric, AAI , to evaluate the alignment of the model’s adversarial attack with intrinsic attack −∇ x log p D ( x , y ) .
Furthermore, we show that AAI is also an effective metric for adversarial transferability.
• We propose the Intrinsic Adversarial Attack ( IAA ) by maximizing AAI to generate more transferable adversarial examples.
• We conduct comprehensive transfer attack experiments from different source models against nine naturally trained models and three ensemble secured models, showing that IAA can signiﬁcantly improve the state-of-the-art transferability (both targeted and untargeted attack) of adversarial examples (even improve 20% under some settings).
R ELATED W ORK
There are two types of adversarial attacks: white-box attacks and black-box attacks.
White-box attacks assume that the attacker can completely access the structure and parameters of the target model.
Typical examples of white-box attacks are FGSM (Goodfellow et al., 2015), PGD (Madry et al., 2018), and CW (Carlini & Wagner, 2017).
The black-box attack assumes that the attacker only knows the output of the target model (prediction or conﬁdence).
Black-box attacks are roughly divided into two types: estimating gradient with queries to the target model (Papernot et al., 2017;
Su et al., 2019; Yang et al., 2020) and attacking a surrogate model (Xie et al., 2019; Dong et al., 2018; Wu et al., 2019).
Attacking a surrogate model is much more efﬁcient and can reduce the risk of exposure.
Thus, many existing works focused on adversarial transferability.
Su et al.
(2018) explore the factors inﬂuencing the transferability and show the architecture has greater inﬂuence than model capacity.
Dong et al.
(2018) show that the momentum of gradients can be used to improve the adversarial transferability.
Xie et al.
(2019) show the diversity of input data will enhance the adversarial transferability.
Huang et al.
(2019) ﬁne-tune the adversarial examples by increasing perturbation on a pre-speciﬁed layer.
Wang et al.
(2020) propose a loss to decrease interactions between perturbation units during attacking.
Wu et al.
(2019) propose that reducing gradients from the residual modules is effective for improving transferability.
Guo et al.
(2020) removes ReLU activations in the later layers to get linear backpropagation and decreases the inﬂuence of intermediate layers.
They only modify the backpropagation when generating adversarial examples while keeping the forward prediction as the original model.
Based on this, Zhang et al.
(2021) conjecture that backpropagating smoothly might be sufﬁcient for improving transferability.
There are also some works on adversarial attack and defense using generative models.
Naseer et al.
(2019) and Yang et al.
(2021a) learn adversarial perturbation through a conditional generative attacking model, which needs to be carefully designed for certain classes.
Samangouei et al.
Song et al.
(2018) use GANs or autoregressive models to detect and purify adversarial examples.
Du & Mordatch (2019); Hill et al.
; Srinivasan et al.
; Yoon et al.
(2021) purify adversarial examples by EBM or score-based generative models.
JEM (Grathwohl et al., 2020) shows that combining a classiﬁer with EBM can help to obtain some robustness.
However, adversarial attacks or puriﬁcation based on generative models are computationally costly.
We want to modify a normal classiﬁer with little computation cost to enhance its adversarial transferability by maximizing our
AAI metric.
M ETHODS 3. A LIGNMENT BETWEEN THE A DVERSARIAL ATTACK AND I NTRINSIC ATTACK
For a classiﬁer f θ, Λ parameterized by θ with structure hyper-parameters Λ (e.g., hyper-parameters for architecture, activation function, etc.), and data x , label y , total possible classes n , then f θ, Λ ( x )[ k ] represents the k th output of the last layer.
The conditional density p θ, Λ ( y | x ) can be expressed as:
[Equation]
The adversarial attack is usually based on −∇ x log p θ, Λ ( y | x ) (Madry et al., 2018; Goodfellow et al., 2015).
The most effective direction towards low-density region is intrinsic attack −∇ x log p D ( x , y ) .
To improve the adversarial transferability, we need to match model’s adversarial attack direction with intrinsic attack.
We deﬁne the inner product of normalized −∇ x log p θ, Λ ( y | x ) and −∇ x log p D ( x , y ) to quantify the matching of the direction of adversarial attack and the intrinsic attack.
Deﬁnition 1 ( AAI ) .
For a classiﬁer p θ, Λ ( y | x ) , the A lignment between its A dversarial attack and the I ntrinsic attack is:
[Equation]
where p D ( x , y ) is the ground truth joint distribution.
Remark.
(1) We use the normalized adversarial attack to remove the inﬂuence of scaling factor when comparing different models.
(2) This deﬁnition is equivalent with a modiﬁed score matching objective as:
[Equation]
where C p D is a constant only depend on the ground truth distribution p D .
As getting the gradient of p D ( x , y ) is not feasible, we use integration by parts (Hyv¨arinen & Dayan, 2005) to move the gradient on p D ( x , y ) to model’s adversarial attack.
With the smoothness assumption on ∇ x p θ, Λ ( y | x ) , we have the following theorem:
Theorem 1.
If ∇ x p θ, Λ ( y | x ) is differentiable almost everywhere, then
[Equation]
Moreover,
[Equation]
where p ( v ) is a distribution of random vector v such that E p ( v ) [ vv T ]
= I (e.g., the multivariate standard normal N ( 0 , I ) ).
This theorem makes it possible to calculate AAI without knowing the gradient of ground truth distribution when the model is smooth, see Appendix E for the proof.
Combined with sliced score matching (Song et al., 2020)
we can efﬁciently approximate AAI on discrete samples.
However, for ReLU networks, the model’s second derivative is not well deﬁned, which prevents us from using this theorem.
Thus, we need to smooth the ReLU models for better modifying the model to maximize AAI and improve the transferability.
One obvious way to improve the smoothness is to replace ReLU activation with some smooth activation.
In this paper, we use Softplus to show that smoothness can greatly help us to improve transferability.
3. I NTRINSIC A DVERSARIAL A TTACK FOR NORMAL MODELS 3.2. S MOOTHING THE C LASSIFIER BY S OFTPLUS
Srinivas & Fleuret (2021) claim that the input-gradient of a classiﬁer is somewhat aligned with the gradient of the ground truth data distribution p D ( x , y ) .
On the other hand, Dombrowski et al.
(2019) show that the information in gradients may be hidden for that the gradient of the ReLU activation function is discontinuous and will frequently change when the input closes to zero.
Thus better gradient with less noise can contribute to matching the input-gradients −∇ x log p θ, Λ ( y | x ) and the gradients of ground truth distribution −∇ x log p D ( x , y ) .
We provide a simple implementation that replacing the widely used activation function ReLU with Softplus β (Nair & Hinton, 2010):
[Equation]
where β is the shape-related hyper-parameter.
The larger β is, the better Softplus β approximates
ReLU.
The Softplus β can also be expressed as the expectation of ReLU in a neighborhood as follows:
[Equation]
where p β ( (cid:15) ) =
[Equation]
[-2 + e − β(cid:15)/ 2 ) 
-]
(Dombrowski et al., 2019).
See Appendix F for a short proof.
Smilkov et al.
Dombrowski et al.
(2019) show that the local average of gradient values can make the gradient smoother.
This means that the gradient obtained with Softplus β is less noisy and more meaningful than ReLU.
We use Softplus β as an approximation of the activation function
ReLU in the pre-trained model.
As shown in Fig.
2 (Left), AAI ﬁrst increases then decreases when decreasing β in Softplus β (the smaller the β , the smoother the classiﬁers) and AAI achieves the largest value at β = 15 .
Thus there exits a trade-off between AAI and smoothness, and a proper β is needed.
We conducted an experiment with the PGD attack on the randomly selected 5000 ImageNet validation images which are correctly classiﬁed to investigate how the shape-related hyper-parameter β affects the adversarial transferability.
As shown in Fig.
2 (Right), when the hyper-parameter β is too small, Softplus β cannot approximate ReLU well, which reduces the effectiveness of adversarial perturbation and prediction accuracy.
With a proper selection of β = 15 which is also the maximum of AAI , using Softplus β as an approximation of ReLU can signiﬁcantly improve the transferability of adversarial examples.
3.2. U TILIZING DISTRIBUTION - RELEVANT INFORMATION FROM EARLY LAYERS
Maennel et al.
and Boopathy & Fiete (2021) show that the early layers can better capture the local statistics of the inputs than the later layers.
This discovery inspires us to observe whether the gradient of the early layers of the model is better aligned with the gradient of the ground truth data distribution.
The general ResNet consists of four blocks and the residual module (He et al., 2016) can be expressed as:
[Equation]
where z i +1 is the output of residual modules and z i is the input of residual modules, F is the residual function.
Considering that the information can transfer from shallow layers to deep layers through the skip connection in ResNet-like neural networks, we decrease the weight for certain residual modules to reduce the impact of these modules and enhance the impact of the previous modules from the skip connection.
Thus, we modify the residual module as:
[Equation]
where 0 ≤ λ i ≤ 1 can reduce the impact of the certain residual modules.
We use the λ i to relatively adjust the inﬂuence of different layers.
As shown in Fig.
3, when decreasing λ 1:4 at the same rate on pre-trained ResNet-50 (Paszke et al., 2019), the AAI metric ﬁrst increases then decreases with reaching its maximum at λ 1:4 = 0 .
At the same time, different target models obtain the best attack success rate also at λ 1:4 =
.(see Fig.
3 (Middle)).
This implies that adjusting the inﬂuence of different residual branches can help to increase AAI as well as transferability.
Moreover, we ﬁnd that decreasing the value of λcan signiﬁcantly improve the transferability a lot (from 53% to 71 .
32% ) (Fig.
(Right)).
This shows that the last block does restrict adversarial transferability and reducing the impact of the last block can greatly improve the transferability.
Decreasing λ 1 , 2 , 3 for Block1, Block2, and Block3 can slightly improve the adversarial transferability.
If we want to achieve the best transferability, we need carefully tuning the hyper-parameters λ 1:4 .
As our AAI metric can be effectively predict the transferability, we use Bayesian optimization to search the best λ 1:4 to maximize AAI .
Input: A pre-trained classiﬁer f θ ; original image x ; (cid:96) ∞ perturbation radius (cid:15) ; step size α ; iterations N ; the classiﬁcation loss (cid:96) ;
Output: The modiﬁed model f θ,β ∗ ,λ ∗ , adversarial perturbation δ .
Solve Eq.
(9) to ﬁnd the modiﬁed model f θ,β ∗ ,λ ∗ , replace ReLU with Softplus β ∗ and apply λ ∗ i s to residual modules of f θ .
Initialize δ = Uniform ( − (cid:15), (cid:15) ) .
for i = 1 , 2 , ..., N do δ
=
δ + α · sign ( ∇ x (cid:96) ( f θ,λ,β ( x + δ ) , y ) , δ = max ( min ( δ , (cid:15) ) , − (cid:15) ) end for return δ
[Figure]
λ
We can also make full use of the distribution-relevant information of a given classiﬁer to increase
AAI without skip connections.
As shown in Appendix J, there are many other ways that can enhance the impact of the early layers to increase AAI and can be applied to models without skip connection, but this is not the focus of this paper. Our main contribution is to rethink the adversarial transferability from a novel perspective and reveal that the distribution-relevant information may be the key for boosting adversarial transferability.
3.2. T HE A LGORITHM IAA
As analyzed in previous sections, we have veriﬁed that the AAI is an effective metric for transferability.
The smoothness and data distribution information from different layers can be important factors for maximizing AAI .
In order to ﬁnd the best parameters, we use Bayesian optimization to optimize
[Equation]
We propose Intrinsic Adversarial Attack (IAA) as shown in Alg. 1, which replaces the activation function ReLU with Softplus β and applies corresponding decay parameters
λ i searched by Eq.
(9) to different residual modules.
Fig.
4 shows the training curve on pre-trained ResNet-50, see Appendix C for setting details.
The experiments are shown in next section.
We analyze the computation cost in Appendix M and show other optimization methods to ﬁnd the proper structural parameters in Appendix O. [Table]
E XPERIMENTS 4. I MPLEMENTATION
Attack Setting.
We use a black-box threat model to test the adversarial transferability.
Adversarial examples are generated by attacking the source models, and then these examples are applied to attack target models.
The architecture of the source model is different from that of the target model.
In terms of attack strength, we follow the same standard setting for all attack methods (Xie et al., 2019;
Wu et al., 2019).
We constrain the adversarial perturbation within the (cid:96) ∞ ball of radius (cid:15) = 16 /with respect to pixel value in [0 , 1] and set the step size α to 2 / 255 .
The iteration steps in all the experiments are set to 10 .
All experiments in this paper are run on Tesla V100.
The experiments are repeated 5 times with different random seeds and we just show the maximum standard deviations in the caption of tables due to page limitations.
See Appendix C for more attack setting.
Target Models and Source Models.
We conduct experiments on both normal target models and robust target models.
For normal target models, we choose 9 models: VGG19 (Simonyan & Zisserman, 2015), ResNet-50 (RN50), ResNet-152 (RN152)
(He et al., 2016), DenseNet-121 (DN121),
DenseNet-201 (DN201) (Huang et al., 2017), 154 layers Squeeze-and-Excitation network (SE154) (Hu et al., 2018), Inception V3 (IncV3) (Szegedy et al., 2016), Inception V4 (IncV4), and InceptionResNet V2 (IncRes) (Szegedy et al., 2017), and we use the pre-trained models in PyTorch (Paszke et al., 2019).
For robust target models, we consider 3 robustly trained models using ensemble adversarial training (Tram`er et al., 2018): IncV3ens3 (ensemble of 3 IncV3 models), IncV3ens4 (ensemble of 4 IncV3 models) and IncResV2ens3 (ensemble of 3 IncResV2 models).
We choose 4 models as source models: ResNet-50, ResNet-152, DenseNet-121, DenseNet-201.
We also choose VGG16,
VGG19 (models without skip connection) as source model in Appendix J.
4. T RANSFERABILITY A GAINST N ORMAL M ODELS
Firstly, we focus on untargeted adversarial attacks.
We compare different attack methods on four source models against nine normal target models.
In all transfer scenarios, IAA outperforms existing methods by a large margin consistently against different target models.
The results are reported in
Tab. 1.
For transfer ResNet-50 → VGG19, IAA achieves a success rate of 96.40 % which is 43.40 % and 13.68 % higher than PGD and SGM respectively.
For transfer ResNet-50 → Inception-ResNet
V2, IR performs the best in the existing methods (attack success rate 46.78 % ) while our IAA can achieve a new state-of-the-art success rate 64.32 % .
In Tab. 1, we show that IAA surpasses the other existing methods when attacking against normal models under Top-1 accuracy.
Top-5 accuracy, which checks if the ground truth label is among the ﬁve predictions with the highest probability, is also an important metric.
In Tab. 2, we show the ﬁrst ﬁve predictions of the target model.
For transfer ResNet-50 → Inception V3, the top-5 accuracy on adversarial examples crafted by PGD is 96.66 % .
SGM is an efﬁcient method to enhance adversarial transferability, but the top-5 accuracy is as high as 84.28 % for transfer ResNet-50 → Inception
V3.
Our IAA drives the examples to the low-density region by matching the adversarial attack and intrinsic attack.
Thus the target models can hardly give a proper prediction, and the top-5 accuracy is signiﬁcantly lower than other methods.
[Table] [Table]
Secondly, we focus on targeted attacks.
Liu et al.
; Yang et al.
(2021b) show that generating transferable targeted adversarial examples is much more difﬁcult than generating transferable untargeted adversarial examples.
As we analyzed before, when crafting the untargeted attack, the intrinsic adversarial attack can drive adversarial examples to low-density regions.
Similarly, when crafting the targeted attack, IAA can move the adversarial images towards the high-density region of the target class along the direction of ∇ x log p θ ( x , y target ) .
Thus IAA can also improve the adversarial transferability of targeted examples, and we show the results in Tab. 3.
For transfer ResNet- → VGG19 (targeted), IAA achieves a success rate of 33.68 % which is 33.28 % and 29.20 % higher than PGD and SGM respectively.
We also evaluate the transferability of one-step attacks in Fig.
SGM can greatly enhance the transferability of the multi-step attack, but it can only slightly improve the effect of the one-step attack.
DI performs better in the transferability of the one-step attack than SGM.
We think that diversity inputs may help to reduce the noise in gradient and get better adversarial perturbation.
Our
IAA can signiﬁcantly improve the adversarial transferability of the one-step attack.
This is because that the perturbation crafted by IAA is aligned with the intrinsic attack −∇ x log p D ( x , y ) and can efﬁciently drive the examples towards the low-density region.
It is difﬁcult for the target models to predict accurately for examples in the low-density region.
4. T RANSFERABILITY A GAINST R OBUSTLY T RAINED M ODELS
The success rates of our IAA and other methods against the robust target models (using ensemble adversarial training) are reported in Tab. 4.
TI was originally proposed for the robust models.
Compared with the existing methods, IAA has a large improvement in all scenarios.
For transfer
ResNet-50 → IncV 3 ens 3 , IAA achieves a success rate of 38.72 % which is 31.2 % and 5.91 % higher than PGD and SGM respectively.
We think that both the robust model and the normal model are trained on the same dataset.
IAA perturbs the distribution-relevant information, which leads to misclassiﬁcation of the robust models.
We show that IAA can also effectively penetrate many other defense methods in Appendix N.
[Table]
4. E NSEMBLE - BASED A TTACKS
Wu et al.
; Wang et al.
(2020) show that the ensemble-based strategy (Liu et al., 2017) can improve the performance of their method.
This section, shows the results in Tab.
[Table]
and the attacks are crafted on an ensemble of ResNet-34, ResNet-152, and DenseNet-201.
We show that our IAA with the ensemble-based strategy can generate more transferable perturbation than other methods.
For transfer Ensemble model → IncV4, IAA achieves a success rate of 90.70 % which is 7.73 % and 10.63 % higher than SGM and IR respectively.
[Table]
C ONCLUSION
In this paper
[Table]
we rethink the adversarial perturbation from the distribution perspective and show that the alignment between the adversarial attack and intrinsic attack ( AAI ) is an effective metric for predicting the adversarial transferability.
Furthermore, we propose a novel algorithm ( IAA ) which maximizes AAI for generating adversarial examples in the low-density region by Bayesian optimization.
We have conducted experiments on four source models, nine normal target models (including untargeted attack and targeted attack) and three robust target models and show that our methods surpass other existing methods by a large margin.
Our ﬁndings can motivate new research into the adversarial transferability and adversarial examples from the perspective of data distribution and open up further challenges for the defense against black-box attacks.
What’s more, IAA can also fool the image search engines (Google Reverse Image Search and Baidu Reverse Image Search) as shown in Appendix A.
E THICS S TATEMENT
In this paper we propose a novel method IAA to boost adversarial transferability.
IAA can craft stronger adversarial examples from the source model to attack unknown target models than other existing methods.
Our ﬁndings can motivate new research into the adversarial transferability and adversarial examples from the data distribution perspective and open up further challenges for the defense against black-box attacks.
Our goal is to understand current deep learning’s weaknesses and make deep models more robust and transparent.
We did not use crowdsourcing and did not conduct research with human subjects in our experiments.
We cited the creators when using existing assets (e.g., code, data, models).
R EPRODUCIBILITY S TATEMENT
We show the proofs of Theorem 1 in Appendix E. Our IAA is an appealingly simple method.
We specify the settings of hyper-parameters and how they were chosen in Appendix C.
We repeat experiments 5 times with different random seeds and show the standard deviation in the caption of tables.
We plan to open the source code to reproduce the main experimental results later.
A F OOLING THE IMAGE SEARCH ENGINES
The previous experiment mainly veriﬁed the attack effect of IAA on the pre-trained classiﬁcation model.
In this section we show that IAA can fool the image search engines successfully.
As shown in Fig.
6, the original image can be correctly identiﬁed by both Google Reverse Images Search and
Baidu Reverse Images Search .
However, the adversarial images generated by our IAA can fool the image search engines.
B H IGH TRANSFERABLE EXAMPLES
We use a source model RN50 to generate adversarial examples with widely used PGD attack.
If the original image can be correctly classiﬁed by different targeted models (VGG19, RN50, RN152,
DN121, DN201 and SE154) and its adversarial counterpart can fool all the target models, we save this original image.
Then, we randomly save 1000 samples from ImageNet evaluation datasets.
We named these samples Low-Density Data (L Data).
If the original images can be correctly classiﬁed by different targeted models (VGG19, RN50, RN152, DN121, DN201 and SE154) and their adversarial counterparts can only fool the source models, we save this original image.
Then, we randomly save 1000 samples from ImageNet evaluation datasets.
We named these samples High-Density Data (H Data).
We ﬁnd an intriguing phenomena: LDD with random noise can fool different target models, while the attack success rate is much lower when applying the same strength of random noise to HDD.
Speciﬁcally when we injecting Gaussian noise ( (cid:15) =16/255) in LDD, 23.3 % of the samples can fool the target model VGG19.
However, when we injecting Gaussian noise ( (cid:15) =
16/255) in HDD, only 0.7 % of the samples can fool the target model VGG19.
What’s more, LDD can be correctly classiﬁed by different models but their adversarial counterparts generated by different source models have strong transferability against different target models (as shown in Fig.
3.1 and Fig. 7).
In Fig.
8, we show some samples of LDD.
[Figure]
C A TTACK S ETTING
We choose the baseline attack PGD (Madry et al., 2018) and 5 state- is proposed to attack robust model into consideration when attacking against robust models. All the source models are trained on ImageNet training set.
The scikit-optimize 1 is a simple and efﬁcient library to minimize black-box functions, which helps us to search the best structure hyper-parameters for a pre-trained classiﬁer to minimize the Eq.
We use Batesian optimization to optimize the Eq.
(9) to search the hyper-parameters β
and λ i .
For
ResNet-50, the search results are β = 20 , and λ 1 applied to the residual modules in Block1 is 0.98, λ 2 applied to the residual modules in Block1 is 0.87, λ 3 applied to the residual modules in Block is 0.73, λ 4 applied to the residual modules in Block4 is 0.19.
For ResNet-152, the search results are β = 32 , and λ 1 applied to the residual modules in Block1 is 0.89, λ 2 applied to the residual modules in Block1 is 0.88, λ 3 applied to the residual modules in Block3 is 0.70, λ 4 applied to the residual modules in Block4 is 0.20.
For DenseNet-121 and DenseNet-201, the search results are β = 35 , and λ 1 , 2 , 3 applied to the residual modules in Block1, Block2 and Block3 is 0.80, λ 4 applied to the residual modules in Block4 is 0.44.
D S IMILARITY BETWEEN ADVERSARIAL PERTURBATION
We illustrate frequency histogram of cosine similarity between adversarial perturbation generated by different models in Fig.
We also illustrate frequency histogram of Pearson Correlation Coefﬁcient (PCC) between adversarial perturbation generated by different models in Fig.
Perturbation generated by different models using IAA have stronger correlation than other methods (PGD and
SGM), which means that IAA reduces the dependence on models and generate more distribution relevant perturbation.
[Figure] [Figure]
E P ROOF : T HEOREM
The proof is similar as Hyv¨arinen & Dayan (2005) and Song et al.
[Table]
[Equation]
The log function can be moved for the equality (I) because the normalized ∇ x p ( y | x ) and ∇ x log p ( y | x ) are the same direction, as:
[Equation]
where we use the formula: ∇ x log f ( x ) =
f ( x ) − 1 ∇ x f ( x ) .
Take 1d case for example, we use integration by part formula for the equality (II):
[Equation]
The equality (III) holds by Holder Inequality (Bartle, 2014):
[Equation]
For a distribution of a random vector v such that E p ( v ) [ vv T ]
=
I , then we have
[Equation]
We use the property that random vector v is independent of ( x, y ) in Eq. 13, thus we can switch ∇ x and v .
F P ROOF : S OFTPLUS IS THE EXPECTATION OF R E LU
[Equation]
(14) where β is the shape-related hyper-parameter.
The derivative of this function is
[Equation]
When the p β ( (cid:15) ) is implicitly deﬁned, we express Softplus β ( x ) as:
[Equation]
With respect to x, the differential of the equation is:
[Equation]
[Equation]
where Θ( x ) = I ( x > 0) is the Heaviside step function.
From Eq.
(15), the differential of Eq.
(17) is:
[Equation]
Thus, the Softplus β can be expressed as the expectation of ReLU in a neighborhood as follows:
[Equation]
19) where p β ( (cid:15) ) =
[Equation]
G T HE INFLUENCE OF STRUCTURE PARAMETERS ON D ENSE N ET -
We ﬁrst show the inﬂuence of the shape parameter β when we use replace ReLU with Softplus β in Fig.
11 (Left).
A proper β can improve the adversarial transferability.
The general ResNet and
DenseNet consists of four blocks.
We illustrate the inﬂuence of applying λ
i on different blocks of
ResNet-50 in paper.
Here, we illustrate the inﬂuence of applying λ
i on different blocks of DenseNet121 in Fig.
11 (Right).
The target model is VGG19.
λ i applied to Block4 has great inﬂuence on adversarial transferability, and λ i =0.4 can improve the attack success rate from 56 .
78% to 67 .
λ i applied to Block1, Block2, and Block3 can slightly improve the adversarial transferability.
When we apply the best λ i to each block, the attack success rate can reach 85 .
08% , which has exceeded state-of-the-art methods SGM (80.18 % ) and IR (82.56 % ).
[Figure]
[-target model is VGG triangle mark attack success rate applying λ to Block
-]
H A DVERSARIAL EXAMPLES SHOW RECONIZABLE FETURES
We insert and ﬁne-tune classiﬁers (fully connected layer) after each block of ResNet-50.
Attack
Block1 means attacking the classiﬁer after block1.
As show in Fig.
12, the perturbation generated
[Figure] [Figure]
by attacking normal ResNet-50 like unrecognizable noise.
However the perturbation generated by attacking the classiﬁers deployed after block2 and block3 of normal ResNet-50 show recognizable features of tench.
This shows that the normal model has also learned some distribution-relevant information.
I C ONDITIONAL G ENERATING E XPERIMENT
We also show the input-gradients in the normal model are somewhat aligned with the gradient of the ground truth data distribution p data ( x | y ) through a conditional generating experiment using the method proposed by Santurkar et al.
The general ResNet consists of 4 blocks.
We insert classiﬁers (fully connected layers) after each block of ResNet-50 to utilize information from different layers, then ﬁne-tune these classiﬁers and use these classiﬁers to generate images.
As shown in
Fig.
13(given the label tench), the classiﬁer on Block
(the last block) of the model cannot generate images with recognizable semantic features.
The images generated by the classiﬁers on Block2 and
Block3 are not good enough but can be recognized with high conﬁdence.
The generative capability of Block1 is poor maybe it does not learn enough class-relevant information.
This experiment intuitively shows that manipulating input-gradients can generate images with the features of target distribution (tench).
J O THER METHODS TO ENHANCE THE IMPACT OF EARLY LAYERS
In our paper, IAA gives a appealingly simple implementation on ResNet-like neural networks.
There are many other ways to adjust the impact of different layers to improve the adversarial transferability.
These methods are slightly complex to implement, but they are also effective.
J. T AKING ADVANTAGE OF LOW - LEVEL INFORMATION
In our paper, we ﬁnd that the later layers may reduce the adversarial transferability and we apply hyper-parameters
λ i to reduce the inﬂuence of the later layers.
We can also use the new loss to enhance the inﬂuence of the early layers.
We propose a loss to take advantage of low-level information during attacking :
[Equation]
where KL is KL divergence, N is the number of blocks in source model, feature ori is the feature of the natural examples extracted by the block i , and feature adv is the feature of the adversarial examples extracted by the block i .
The general ResNet and DenseNet consists of four blocks.
We optimize the classiﬁcation loss and the low-level feature loss jointly during attacking:
[Equation]
(21) where (cid:15) is a constant of the norm constraint and γ is a constant weight for the low-level feature loss.
We call this method Enhancing the Early Blocks (EEB) .
As shown in Fig.
14, we show the inﬂuence of the hyper-parameter γ .
Increasing the hyper-parameter γ (increasing the weight of low-level information) tends to improve adversarial transferability until it exceeds a threshold, e .
g ., γ =100.
This may due to γ encourages the attack to utilize more transferable low-level information.
However, the attack can be insufﬁcient when just using low-level information and ignoring all highlevel class-relevant information.
EEB can also improve the adversarial transferability of the deep neural networks without skip connection (VGG16, VGG19).
As shown in Tab.
[Table]
[Figure]
[Figure]
J. O NLY ATTACK THE EARLY LAYERS
We insert a classiﬁer after the Blocks3 (the penultimate block) of ResNet-50, then ﬁne-tune this classiﬁer
[Table]
and attack this classiﬁer to generate adversarial examples.
Such a simple operation can compete with state-of-the-art methods (SGM).
We call this method Dropping the Last Block (DLB).
[Table]
K IAA UTILIZE THE LOW - LEVEL INFORMATION
Centered kernel alignment (CKA) can reliably identify correspondences between representations in networks (Kornblith et al., 2019; Nguyen et al., 2020).
In order to show the difference between natural examples and adversarial examples, we show the CKA similarity between the features of natural examples and adversarial examples extracted by the same normal ResNet-50 (ImageNet).
As shown in Fig.
15 (a), the adversarial examples generated by attacking the normal model using
PGD show more difference from natural examples in features extracted by deep layers.
In Fig.(b), the adversarial example generated by attacking the normal model using our IAA are different from the natural example in the shallow layers.
For example, in the shallow layers (layer 20 to layer 30), the CKA similarity between the adversarial examples generated by the normal model using
PGD and the natural example is about 0.9, while the similarity of the adversarial example generated by IAA with the natural example is about 0.7.
That is to say, IAA uses more low-level information when generating adversarial examples and IAA enhances the inﬂuence of the shallow layers on generating adversarial perturbation.
[Figure]
L O THER CORRUPTION CASES FOR THE MOTIVATION EXAMPLE
As shown in Tab. 8, we added other types of corruption like CIFAR-10-C and ImageNet-C (Hendrycks & Dietterich, 2019), including noise, blur, weather, etc.
Consistent with the observations in Fig.
1, the target models are more likely to misclassify corrupted LDD data.
We hypothesize that these samples are in the low-density region of the ground truth distribution where both source and target models are not well trained on.
Moreover, we ﬁnd that the adversarial counterparts of LDD have stronger adversarial transferability than that of HDD, which inspires us to match the direction of adversarial perturbation and the direction towards the low-density region to generate adversarial examples with high transferability.
[Table]
M C OMPUTATION COST
Our proposed method includes two steps.
The ﬁrst step is adjusting the pre-trained source model to match the adversarial attack with the ground truth density decreasing direction by optimizing AAI, and the second step is generating adversarial samples by the modiﬁed model similar to the normal
PGD attack.
Optimizing AAI to get a strong source model by Bayesian optimization on the pre-trained model needs some extra computation.
As shown in Eq.
4, the computation cost for AAI can be reduced to O ( d ) from O ( d 2 ) by using ideas from sliced score matching (Song et al., 2020), where d is the dimension of data.
The computation cost of AAI optimization is 10 hours on Tesla-V100 using
ResNet-50.
However, the cost of generating adversarial examples is more important than the cost of adjusting the source model in practice.
Once we get the modiﬁed model, generating different adversarial
[Table]
examples by the modiﬁed model shares the same computation with normal PGD adversarial attacks.
That is to say, it is very efﬁcient to use our modiﬁed model to generate adversarial examples.
N A TTACK OTHER DEFENSE METHODS
In Tab. 4, we evaluate that our IAA can fool the robust model by ensemble adversarial training (Tram`er et al., 2018).
Moreover, we show that the adversarial examples generated by IAA can also effectively penetrate many other defense methods in Tab. 9.
We further add neural representation puriﬁer (NRP) (Naseer et al., 2020) which is a state-of-the-art input processing defense method, and some other robust training methods, including training with AugMix (Hendrycks et al., 2020),
Styled ImageNet (SIN) (Geirhos et al., 2019), the mixture of Styled and natural ImageNet (SIN-IN), and adversarial examples (Salman et al., 2020) as defense models.
In Tab. 9, we show the topaccuracy and top-5 accuracy (ACC1/ACC5) on different defense models when predicting the blackbox adversarial examples crafted by different methods.
We use the ResNet-50 as the source model.
IAA surpasses other attacks in different cases.
Our IAA drives the examples to the low-density region by matching the adversarial attack and intrinsic attack.
Thus the target models can hardly give a correct prediction on IAA examples, and both the top-1 accuracy and the top-5 accuracy are lower than other methods.
O F URTHER DISCUSSION ON DIRECTLY OPTIMIZING THE TRANSFERABILITY .
In our paper, we used the method of optimizing AAI to choose hyperparameters for two reasons.
On the one hand, we hope to verify the effectiveness of the metric we proposed.
On the other hand, we think that using the transferability for a certain target model to optimize may overﬁt the target model.
Interestingly, Fig.2 and Fig.3 show that the attack success rate among all the target models seems like aligns well under different structural hyperparameters.
Maybe it won’t be so over-ﬁtted to the target model that it is difﬁcult to select the proper hyperparameters.
We conduct experiments on the source model ResNet-50.
We use four methods to choose the proper hyperparameters, including using the AAI as the optimization objective (”optimize-AAI”), using the transferability against the target model VGG19 as the optimization objective (”optimize-vgg19”), using transferability against the target model SE154 as the optimization objective (”optimize-senet154”), and using transferability against the ensemble of VGG19 and SE154 as the optimization objective (”optimize-ensemble”).
The following table represents the top-1 accuracy and top-5 accuracy (ACC1/ACC5) when predicting the adversarial examples crafted by different methods.
Lower accuracy indicates better adversarial transferability.
We found that directly using the transferability as the optimization objective can also ﬁnd proper structural hyperparameters and outperforms the existing methods PGD and SGM.
The ensemble-based strategy can slightly improve the adversarial transferability.
The computation cost of directly optimizing transferability is half that of optimizing AAI.
[Table]