Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbation
Abstract
Deep neural networks (DNNs) have been shown to be vulnerable to adversarial examples, which can produce erroneous predictions by injecting imperceptible perturbations.
In this work, we study the transferability of adversarial examples, which is significant due to its threat to real-world applications where model architecture or parameters are usually unknown.
Many existing works reveal that the adversarial examples are likely to overfit the surrogate model that they are generated from, limiting its transfer attack performance against different target models.
To mitigate the overfitting of the surrogate model, we propose a novel attack method, dubbed reverse adversarial perturbation (RAP).
Specifically, instead of minimizing the loss of a single adversarial point, we advocate seeking adversarial example located at a region with unified low loss value, by injecting the worst-case perturbation ( i.e ., the reverse adversarial perturbation) for each step of the optimization procedure.
The adversarial attack with RAP is formulated as a min-max bi-level optimization problem.
By integrating RAP into the iterative process for attacks, our method can find more stable adversarial examples which are less sensitive to the changes of decision boundary, mitigating the overfitting of the surrogate model.
Comprehensive experimental comparisons demonstrate that RAP can significantly boost adversarial transferability.
Furthermore, RAP can be naturally combined with many existing black-box attack techniques, to further boost the transferability.
When attacking a real-world image recognition system, i e ., Google Cloud Vision API, we obtain 22% performance improvement of targeted attacks over the compared method. Our codes are available at: https://github.com/Alan-Qin/Transfer_attack_RAP .
Introduction
Deep neural networks (DNNs) have been successfully applied in many safety-critical tasks, such as autonomous driving, face recognition and verification, etc .
However, it has been shown that DNN models are vulnerable to adversarial examples [11, 26, 34, 43, 44], which are indistinguishable from natural examples but make a model produce erroneous predictions.
For real-world applications, the DNN models are often hidden from users.
Therefore, the attackers need to generate the adversarial examples under black-box setting where they do not know any information of the target model
For black-box setting, the adversarial transferability matters since it can allow the attackers to attack target models by using adversarial examples generated on the surrogate models.
Therefore, learning how to generate adversarial examples with high transferability has gained more attentions in the literature [5, 9, 13, 22, 25, 37, 47].
Under white-box setting where the complete information of the attacked model ( e.g ., architecture and parameters) is available, the gradient-based attacks such as PGD [26] have demonstrated good attack performance.
However, they often exhibit the poor transferiability [5, 47], i.e ., the adversarial example x adv generated from the surrogate model M S performs poorly against different target models M T .
The previous works attribute that to the overfitting of adversarial examples to the surrogate models [5, 23, 47].
Figure 1 (b) gives an illustration.
The PGD attack aims to find an adversarial point x pgd with minimal attack loss, while doesn’t consider the attack loss of the neighborhood regions round x pgd .
Due to the highly non-convex of deep models, when x pgd locates at a sharply local minimum, a slight change on model parameters of M S could cause a large increase of the attack loss, making x pgd fail to attack the perturbed model.
Many techniques have been proposed to mitigate the overfitting and improve the transferability, including input transformation [6, 47], gradient calibration [13], feature-level attacks [16], and generative models [29], etc.
However, there still exists a large gap of attack performance between the transfer setting and the ideal white-box setting, especially for targeted attack, requiring more efforts for boosting the transferability.
In this work, we propose a novel attack method called reverse adversarial perturbation (RAP) to alleviate the overfitting of the surrogate model and boost the transferability of adversarial examples.
We encourage that x adv is not only of low attack loss but also locates at a local flat region, i.e ., the points within the local neighborhood region around x adv should also be of low loss values.
Figure 1 (b) illustrates the difference between the sharp minimum and flat minimum.
When the model parameter of M S has some slight changes, the variation of the attack loss w.r.t.
the flat local minimum is less than that of the sharp one.
Therefore, the flat local minimum is less sensitive to the changes of decision boundary.
To achieve this goal, we formulate a min-max bi-level optimization problem.
The inner maximization aims to find the worst-case perturbation ( i.e ., that with the largest attack loss, and this is why we call it reverse adversarial perturbation) within the local region around the current adversarial example, which can be solved by the projected gradient ascent algorithm.
Then, the outer minimization will update the adversarial example to find a new point added with the provided reverse perturbation that leads to lower attack loss.
Figure 1 (a) provides an illustration of the optimization process.
For t -th iteration and x t , RAP first finds the point x t + n rap with max attack loss within the neighborhood region of x t .
Then it updates x t with the gradient calculated by minimizing the attack loss w.r.t .
x t + n rap .
Compared to directly adopting the gradient at x t ,
RAP could help escape from the sharp local minimum and pursue a relatively flat local minimum.
Besides, we design a late-start variant of RAP (RAP-LS) to further boost the attack effectiveness and efficiency, which doesn’t insert the reverse perturbation into the optimization procedure in the early stage.
Moreover, from the technical perspective, since the proposed RAP method only introduces one specially designed perturbation onto adversarial attacks, one notable advantage of
RAP is that it can be naturally combined with many existing black-box attack techniques to further boost the transferability.
For example, when combined with different input transformations ( e.g ., the random resizing and padding in Diverse Input [47]), our RAP method consistently outperforms the counterparts by a clear margin.
Our main contributions are three-fold: 1)
Based on a novel perspective, the flatness of loss landscape for adversarial examples, we propose a novel adversarial attack method RAP that encourages both the adversarial example and its neighborhood region be of low loss value; 2) we present a vigorous experimental study and show that RAP can significantly boost the adversarial transferability on both untargeted and targeted attacks for various networks also containing defense models; 3) we demonstrate that RAP can be easily combined with existing transfer attack techniques and outperforms the state-of-the-art performance by a large margin.
Related Work
The black-box attacks can be categorized into two categories: 1) query-based attacks that conduct the attack based on the feedback of iterative queries to target models, and 2) transfer attacks that use the adversarial examples generated on some surrogate models to attack the target models.
In this work, we focus on the transfer attacks.
For surrogate models, existing attack algorithms such as FGSM [11] and I-FGSM [20] could achieve good attack performance.
However, they often overfit the surrogate models and thus exhibit poor transferability.
Recently, many works are proposed to generate more transferable adversarial examples.
Input-transformation based attacks: Data augmentation, which has been shown to be effective in improving model generalization, has also been widely studied in transfer attack, such as randomly resizing and padding [47], randomly scaling [23], and adversarial mixup [40].
The work of Dong et al.
[6] use a set of translated images to compute gradient and get the better performance against defense models.
Expectation of Transformation (EOT) method [1] synthesize adversarial examples over a chosen distribution of transformations.
It could find the stable x adv to be robust to noise, distortion, and affine transformation, which means it also finds x adv at the flat local minimum.
Enhanced-momentum based attacks: Instead of the I-FGSM, the work of Dong et al.
[5] integrate momentum into the updating strategy and Lin et al. [23] use the Nesterov accelerated gradient to boost the transferability. By tuning the variance of each gradient step, Wang and He [39] could find a more stable gradient direction to escape sharp local minimum, further boosting the adversarial transferability. Enhanced-architecture based attacks: There are also some model-specific designs to boost the transferability.
For example, Wu et al.
[42] find the gradient of skip connections is more crucial to generate more transferable attacks.
The work of Guo et al.
[13] also propose LinBP to utilize more gradient of skip connections during the back-propagation.
However, these methods tend to be specific to a particular model architecture, such as skip connection, and it is nontrivial to extend the findings to other architectures or modules.
Enhanced-feature based attacks: Meanwhile, Huang et al.
[16], Inkawhich et al.
[18, 19] propose to exploit feature space constraints to generate more transferable attacks.
Yet they need to identity the best performing intermediate layers or train one-vs-all binary classifies for all attacked classes.
Recently, Zhao et al.
[48] find iterative attacks with much more iterations and logit loss can achieve relatively high targeted transferability and exceed the feature-based attacks.
Besides, the combination of mentioned input transformation can further improve attack performance. Generative model attacks: Apart from that, there have been some methods utilizing the generative models to generate the adversarial perturbations [27, 29, 30].
For example, the work of Naseer et al.
[29] propose to train a generative model to match the distributions of source and target class, so as to increase the targeted transferability.
However, the learning of the perturbation generator is nontrivial, especially on large-scale datasets.
In summary, the current performance of transfer attacks is still unsatisfactory, especially for targeted attacks.
Based on the geometry property of input space, the flatness of local loss landscape of x adv , we proposed to find the attack x adv which locates at flat local minimum of input space.
We also notice that the recent work [12] propose to generate x adv on the surrogate model which is located at the flat region of model weight space.
They find that the better flatness of model weight could also significantly improve the adversarial transferability.
Wang et al.
[38] also explore the min-max framework for producing the attacks, but they are totally different from us in terms of motivation and formulation.
They are to generate attacks based on multiple models and samples.
And, the variable of inner maximization in Wang et al.
[38] is the probability vector, representing the weight assigned to different models or samples.
Methodology 3.1 Preliminaries of Transfer Adversarial Attack
Given an benign sample
( x , y ) ∈
( X , Y ) , the procedure of transfer adversarial attack is firstly constructing the adversarial example x adv within the neighborhood region B ϵ ( x ) = { x ′ :
∥ x ′ − x ∥ p ≤ ϵ } by attacking the white-box surrogate model M s ( x ; θ ) :
X → Y , then transferring x adv to directly attack the black-box target model M t ( x ; ϕ ) :
X → Y .
The attack goal is to mislead the target model, i.e ., M t ( x adv ; ϕ )
̸
= y (untargeted attack), or M t ( x adv ; ϕ ) =
y t (targeted attack) with y t ∈ Y indicting the target label.
Taking the target attack as example, the general formulation of [Figure] many existing transfer attack methods can
[Equation]
The loss function L is often set as the cross entropy (CE) loss [47] or the logit loss [48], which will be specified in later experiments.
Besides, the formulation of untargeted attack can be easily obtained by replacing the loss function L and y t by −L and y , respectively.
Since M s is white-box, if G ( · ) is set as the identity function, then any off-the-shelf white-box adversarial attack method can be adopted to solve Problem (1), such as I-FSGM [20], MI-FGSM [5], etc .
Meanwhile, existing works have designed different G ( · ) functions and developed the corresponding optimization algorithms, to boost the adversarial transferability between surrogate and target models.
For example, G ( · ) is specified as random resizing and padding (DI) [47], translation transformation (TI) [6], scale transformation (SI) [23], and adversarial mixup (Admix) [40].
3.2 Reverse Adversarial Perturbation
As discussed above, although having good performance for the white-box setting, the adversarial examples generated from M S exist poor adversarial transferability on M T , especially for targeted attacks.
The previous works attribute this issue to the overfitting of adversarial attack to M S [4, 5, 6, 37, 45].
As shown in Figure 1 (b), when x pgd locates at a sharp local minimum, it is not stable and is sensitive to changes of M S .
When having some changes on model parameters, x pgd could results in a high attack loss against M S ′ and lead to a failure attack.
To mitigating the overfitting to M S , we advocate to find x adv located at flat local region.
That means we encourage that not only x adv itself has low loss value, but also the points in the vicinity of x adv have similarly low loss values.
To this end, we propose to minimize the maximal loss value within a local neighborhood region around the adversarial example x adv .
The maximal loss is implemented by perturbing x adv to maximize the attack loss, named Reverse Adversarial Perturbation (RAP).
By inserting the RAP into the formulation (1), we aim to solve the following problem,
[Equation]
where
[Equation]
with n rap indicating the RAP, and ϵ n defining its search region.
The above formulations Equation (2) and Equation (3) correspond to the targeted attack, and the corresponding untargeted formulations can be easily obtained by replacing the loss function L and y t by −L and y , respectively.
It is a min-max bi-level optimization problem [24], and can be solved by iteratively optimizing the inner maximization and the outer minimization problem.
Specifically, in each iteration, given x adv , the inner maximization w.r [Equation]
t. n rap is solved by the projected gradient ascent algorithm:
[Equation]
The above update is conducted by T steps, and α n = ϵ n T .
Then, given n rap , the outer minimization w.r.t.
x adv can be solved by any off-the-shelf algorithm that is developed for solving Equation (1).
For example, it can be undated by one step projected gradient descent, as follows:
[Equation]
with Clip B ϵ ( x ) ( a ) clipping a into the neighborhood region B ϵ ( x ) .
The overall optimization procedure is summarized in Algorithm 1.
Moreover, since the optimization w.r.t.
x adv can be implemented by any off-the-shelf algorithm for solving Problem Equation (1), one notable advantage of the proposed
RAP is that it can be naturally combined with any one of them, such as the input transformation methods [6, 23, 40, 47].
A Late-Start (LS) Variant of RAP.
In our preliminary experiments, we find that RAP requires more iterations to converge and the performance is slightly lower during the initial iterations, compared to its baseline attack methods.
As shown in Figure 2, we combine MI [5] and Admix [40] with RAP, and adopt ResNet-50 as the surrogate model.
We take the evaluation on 1000 images from ImageNet (see Sec.4.1).
It is observed that the method with RAP (see the orange curves) quickly surpasses its baseline method (see the blue curves) and finally achieves much higher success rate with more iterations, which verify the effect of RAP on enhancing the adversarial transferability.
However, it is also observed that the performance of RAP is slightly lower than its baseline method in the early stage.
The possible reason is that the early-stage attack is of very weak attack performance to the surrogate model.
In this case, it may be waste to pursue better transferable attacks by solving the min-max problem.
A better strategy may be only solving the minimization problem Equation (1) in the early stage to quickly achieve the region of relatively high adversarial attack performance, then starting RAP to further enhance the attack performance and transferability simultaneously.
This strategy is denoted as RAP with late-start (RAP-LS), whose effect is preliminarily supported by the results shown in Figure 2 (see the green curve) and will be evaluated extensively in later experiments.
3.3 A Closer Look at RAP
To verify whether RAP can help us find a x adv located at the local flat region or not, we use ResNet50 as surrogate model and conduct the untargeted attacks.
We visualize the loss landscape around x adv on M S by plotting the loss variations when we move x adv along a random direction with different magnitudes a .
The details of the calculation are provided in Appendix .
Figure 3 plots the visualizations.
We take I-FGSM [20] (denoted as I), MI [5], DI [47], and MI-TI-DI (MTDI) as baselines attacks and combined them with RAP.
We can see that comparing to the baselines, RAP could help find x adv located at the flat region.
Input: Surrogate model M s , benign data ( x , y ) , target label y t , loss function L , transformation G , the global iteration number K , the late-start iteration number K LS of RAP, as well as hyper-parameters in optimization (specified in later experiments)
Output: the adversarial example x adv 1: Initialize x adv ← x 2: for k = 1 , . . , K do 3: if k ≥ K LS then 4: Initialize n rap ←5: for t = 1 , . . . , T do 6:
Update n rap using Equation (4) 7:
Update x adv using Equation (5)
Experiments 4.1 Experimental Settings
Dataset and Evaluated Models.
We conduct the evaluation on the ImageNet-compatible datasetcomprised of 1,000 images.
For the surrogate models, we consider the four widely used network architectures: Inception-v3 (Inc-v3) [35], ResNet-50 (Res-50) [14], DenseNet-121 (Dense-121) [15], and VGG-16bn (VGG-16) [33].
For target models, apart from the above models, we also utilize more diverse architectures: Inception-ResNet-v2 (Inc-Res-v2) [36], NASNet-Large (NASNet-L) [49], and
ViT-Base/16 (ViT-B/16) [8].
For defense models, we adopt the two widely used ensemble adversarial training (AT) models: adv-Inc-v3 (Inc-v3 adv ) and ens-adv-Inc-Res-v2 (IncRes-v2 ens ) [37].
Besides, we also test multi-step AT model [32], imput transformation defense [45], feature denoising [46], and purification defense (NRP) [28].
Compared Methods.
We adopt I-FGSM [20] (denoted as I), MI [5], TI [6], DI [47], SI [23], Admix [40], VT [39], EMI [41], ILA [16], LinBP [13], Ghost Net [21], and the generative targeted attack method TTP [29].
We also consider the combination of baseline methods, including MITI-DI (MTDI), MI-TI-DI-SI (MTDSI), and MI-TI-DI-Admix (MTDAI). Besides, Expectation of
Transformation (EOT) method [1] is also a comparable baseline method. We also conduct the comparison of RAP and EOT.
Implementation Details.
For untargeted attack, we adopt the Cross Entropy (CE) loss.
For targeted attack, apart from CE, we also experiment with the logit loss, where Zhao et al.
[48] shows it behaves better for targeted attack.
The adversarial perturbation ϵ is restricted by ℓ ∞ = 16 / 255 .
The step size α is set as 2 / 255 and number of iteration K is set as 400 for all attacks.
In the following, we mainly show the results at K =
and the results at different value of K are shown in Appendix .
For RAP, we set K LS as 100 and α n as 2 / 255 .
We set ϵ n as 12 / 255 for I and TI in untargeted attack and 16 / 255 for other attacks in all other settings.
The computational cost is shown in Section B of Appendix . [Table] [Table]
Extra Experiments in Appendix. Due to the space limitation, we put extra experiment results in Appendix . The comparisons of RAP and EOT, VT, EMI, and Ghost Net methods are shown in Section
C in Appendix .
The evaluation of RAP on stronger defense model, multi-step AT models, NRP, and feature denoising, is shown in Section D.
The evaluation of ensemble-model attacks on these diverse network architectures is given in Section E.2.
4.2 The Evaluation of Untargeted Attacks
Baseline Methods.
We first evaluate the performance of RAP and RAP-LS with different baseline attacks, including I, MI, DI, TI, SI, and Admix.
The results are shown in Table 1.
For instance, the ‘MI/ +RAP/ +RAP-LS’ denotes the methods of baseline MI, MI+RAP, and MI+RAP-LS, respectively.
RAP achieves the significant improvements for all methods on each target model.
For average attack success rate of all target models, RAP outperforms the I and MI by 9 . 6% and 16 .
3% , respectively.
For TI, DI, SI, and Admix, RAP gets the improvements by 10 . 2% , 10 . 9% , 9 .
3% , and 6 . 3% .
With late-start, RAP-LS further enhance the transfer attack performance for almost all methods.
Combinational Methods.
Prior works demonstrate the combination of baseline methods could largely boost the adversarial transferability [40, 48].
We also investigate of behavior of RAP when incorporated with the combinational attacks.
The results are shown in Table 2.
As shown in the table, there exist the clear improvements of the combinational attacks over all baseline attacks shown in Table 1.
In addition, our RAP-LS further boosts the average attack success rate of the three combinational attacks by 6 . 9% , 2 . 6% , and 1 . 7% respectively.
Combined with the three combinational attacks, RAP-LS achieves 95 .
6% , and 98 .
3% average attack success rate, respectively.
These results demonstrate RAP can significantly enhance the transferability.
4.3 The Evaluation of Targeted Attacks
We then evaluate the targeted attack performance of the different methods with RAP.
The results with logit loss are presented and the results with CE loss are shown in Appendix .
Baseline Methods.
The results of RAP with baseline attacks are shown in Table 3.
From the results, RAP is also very effective in enhancing the transferability in targeted attacks.
Taking ResNet50 and DenseNet-121 as surrogate models for example, the average performance improvements induced by RAP are 5 .
0% (I), 8 .
1% (MI), 4 .
6% (TI), 10 .
4% (DI), 18 .
5% (SI), and 15 .
1% (Admix), respectively.
Comparing to the ResNet-50 and DenseNet-121, the baseline attacks generally achieve [Table] [Table] lower transferability when using the VGG-16 or Inception-v3 as the surrogate models, which has also been verified in existing works [47, 48].
However, for Inception-v3 and VGG-16 as the surrogate models, RAP also consistently boosts the transferability under all cases.
With late-start, RAP-LS could further improve the transferability of RAP for most attacks.
The average attack success rate under all attack cases of RAP-LS is 2 .
6% higher than that of RAP.
Combinational Methods.
As did in the untargeted attacks, we also evaluate the performance of combinational methods.
The results are shown in Table 4.
Similar to the findings in untargeted attacks, the combinational methods obtain significantly improvements over baseline methods.
The
RAP-LS outperforms all combinational methods by a significantly margin.
For example, taking the average attack success rate of all target models as evaluation metric, RAP-LS obtains 14 .
8% , 9 . 3% improvements over the MTDI, MTDSI and MTDAI, respectively.
4.4 The Comparison with Other Types of Attacks
[Table]
Apart from the baseline and the combinational methods, we also experiment with more diverse attack methods, including the model-specific attack LinBP [13], the feature-based attack ILA [16], and the generative targeted attack TTP [29].
The
LinBP depends on the skip connection and the authors only provide the source code about ResNet-50.
We use their released code and thus conduct experiments with ResNet-50 as M S .
The results of LibBP and ILA are shown in Table 5, where we also implement the variants of LinBP following Guo et al.
[13], inlcuding LinBP-ILA, LinBP-ILA-SGM, LinBP-MI-DI, and LinBP-MI-DI-SGM.
We observe that our MI-DIRAP significantly outperforms the LinBP and ILA, especially for the targeted attacks.
Compared with the second-best method ( i.e ., LinBP-MI-DI-SGM), we obtain a large improvement by 33 . 5% on average ASR of targeted attacks.
TTP [29] is the state-of-the-art generative method to conduct targeted attack.
To compare with it, we adopt the generators based on ResNet-50 provided by the authors.
Since TTP needs to train the perturbation generator for each targeted class, we follow their “10-Targets (all-source)" setting, as did in Zhao et al.
The results are shown in Table 7, where our MTDSI+RAP-LS behaves best and outperforms TTP and MTDI by large margins of 14 . 9% and 25 .
7% , respectively.
[Table]
4.5 The Evaluation on Diverse Network Architectures and Defense Models
[Table]
To further demonstrate the efficacy of RAP
[Table]
we evaluate our method on more diverse network architectures, including Inception-ResNet-v2, NASNet-Large and ViT-Base/16.
We adopt ResNet-50 as the surrogate model and the results are shown in Table 6, col 2-7 .
As shown in the table, the proposed RAP-LS achieves significant improvements for all three combinational methods on all target models, and MTDAI+RAPLS achieves the best performance for diverse models. For
MTDAI, the average performance improvements induced by RAP-LS is 5.9% and 7.8% for untargeted and targeted attacks, respectively.
Since ViT is based on the transformer architecture that totally being different from convolution models, the transfer attacks based on Resnet-50 behave relatively poor on it, especially on targeted attacks.
Yet our RAP-LS still obtains consistent improvements for all compared methods.
We also consider the ensemble-model attack on these diverse network architectures and the results are given in Appendix.
Furthermore, we evaluate RAP on attacking defense models.
We choose ensemble adversarial training (AT) model, adv-Inc-v3 and ens-adv-Inc-Res-v2 [37], multi-step AT model [32], imput transformation defense [45], feature denoising [46], and purification defense (NRP) [28].
We only demonstrate the results of ensemble AT in main submission.
The evaluations of other defense models are shown in Appendix .
Following prior works [40, 47], we adopt the ensemble-model attack by averaging the logits of different surrogate models, including ResNet-50, ResNet-101, Inception-v3, and Inception-ResNet-v2.
The transfer attack success rate on defense models are shown in Table 6, col 8-11 .
We can observe that our RAP-LS further boosts transferability of the baseline methods on both targeted and untargeted attacks.
For untargeted attacks, RAP-LS achieves average performance improvements of 9 . 8% and 14 .
1% on Inc-v3 adv and IncRes-v2 ens , respectively.
For targeted attacks, the average performance improvements of RAP-LS are 14 .
8% and 11 .
1% , respectively.
4.6 Ablation Study
We conduct ablation study on the hyper-parameters of the proposed RAP, including the size of neighborhoods ϵ n , the iteration number of inner optimization T and late-start K LS .
We adopt targeted attacks with ResNet-50 as the surrogate model.
We first evaluate the effect of ϵ n and T .
We consider different values of ϵ n , including 2 / 255 , 4 / 255 , 8 / 255 , 12 / 255 , 16 / 255 , and 20 / 255 .
In Figure 4 (a), we plot the tendency curves of the targeted attack success rate under different values of ϵ n and T .
Note that in Sec. 3.2, we set α n = ϵ n /T .
Thus for a fixed ϵ n , larger T indicating lower stepsize α n .
The minimum stepsize of α n is set to 2 / 255 .
We have the following observation from the plot: for a fixed ϵ n , the more iterations T , the better attack performance.
Thus, we adopt a relatively smaller α n = 2 / 255 in our experiments.
In
Figure 4 (b-d), we further plot the results of different attack methods and target models w.r.t .
ϵ n , where α n = 2 / 255 .
As shown in the plots, the larger ϵ n generally improves the attack performance.
For Inception-v3 and DenseNet-121, the improvements become mild for even larger ϵ n .
Overall, the value of 12 or 16 could lead to satisfactory result under most cases.
Then we conduct the ablation study of K LS .
In Figure 5, we report the targeted attack success rate of
I, MI, DI, and MI-TI-DI combined with RAP-LS with K LS = 0 , 25 , 50 , 100 , 150 , 200 .
Note that the
RAP-LS with K LS = 0 reduces to RAP.
As shown in the plots, the proposed late-start strategy can further boost attack performance of RAP for most cases.
In general, the performance improvements increase as K LS increases, and then become mild when K LS is larger than 100.
The suitable value of K LS is relatively consistent among different methods and target models.
4.7 The Targeted Attack Against Google Cloud Vision API
Finally, we conduct the transfer attacks to attack a practical and widely used image recognition system,
Google Cloud Vision API, and in the more challenging targeted attack scenario.
MTDAI-RAP-LS behaves the best performance in above experiments, so we choose it to conduct the attack.
We take the evaluation on randomly selected 500 images and use ResNet-50 as surrogate model.
As the API returns 10 predicted labels for each query, to evaluate the attacking performance, we test whether or not the target class appears in the returned predictions.
Since the predicted label space of Google
Cloud Vision API do not fully correspond to the 1000 ImageNet classes, we manually treat classes with similar semantics to be the same classes.
In comparison, the baseline MTDAI successfully attacks 232 images against the Google API.
Our RAP-LS achieves a large improvement, successfully attacking 342 images, leading to a 22.0% performance improvements.
These demonstrates the high efficacy of our method to improve transferability on real-world system.
Conclusion
In this work, we study the transferability of adversarial examples that is significant for black-box attacks.
The transferability of adversarial examples is generally influenced by the overfitting of surrogate models.
To alleviate this, we propose to seeking adversarial examples that locate at flatter local regions.
That is, instead of optimizing the pinpoint attack loss, we aim to obtain a consistently low loss at the neighbor regions of the adversarial examples.
We formulate this as a min-max bi-level optimization problem, where the inner maximization aims to inject the worse-case perturbation for the adversarial examples.
We conduct a rigorous experimental study, covering untargeted attack and targeted attack, standard and defense models, and a real-world Google Clould Vision API.
The experimental results demonstrate that RAP can significantly boost the transferability of adversarial examples, which also demonstrates that transfer attacks have become serious threats.
We need to consider how to effectively defense against them [31].
Acknowledgments
We want to thank the anonymous reviewers for their valuable suggestions and comments. This work is supported by the Natural Science Foundation of China under grant No.62076213, Shenzhen Science and Technology Program under grant No. RCYX20210609103057050, and the university development fund of the Chinese University of Hong Kong, Shenzhen under grant No.01001810, and sponsored by CCF-Tencent Open Fund.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See final part of Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Appendix (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Appendix (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] No, all the used data are well-known public benchmark datasets such as CIFAR10, ImageNet.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] No, all the used data are well-known public benchmark datasets such as CIFAR10, ImageNet.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] Not applicable.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] Not applicable.
A Social Impact
Deep neural networks (DNNs) have been successfully applied in many safety-critical tasks, such as autonomous driving, face recognition and verification, etc .
And adversarial samples have posed a serious threat to machine learning systems.
For real-world applications, the DNN model as well as the training dataset, are often hidden from users.
Therefore, the attackers need to generate the adversarial examples under black-box setting where they do not know any information of the target model.
For black-box setting, the adversarial transferability matters since it can allow the attackers to attack target models by using adversarial examples generated on the surrogate models.
This work can potentially contribute to understanding of transferability of adversarial examples.
Besides, the better transferability of adversarial examples calls the machine learning and security communities into action to create stronger defenses and robust models against black-box attacks.
B Implementation Details
We conducted all experiments in an Nvidia-V100 GPU.
And we run all experiments 3 times and average all results over 3 random seeds.
Dataset The used two datasets are licensed under MIT.
Imagenet is licensed under Custom (noncommercial).
Implementation Details of Evaluated Models.
For ResNet-50, DenseNet-121, VGG-16, Inceptionv3, we adopt the pre-trained models provided by torchvision package.
For Inception-ResNet-v2,
NASNet-Large, ViT-Base/16, adv-Inc-v3, and ens-adv-Inc-Res-v2, we adopt the provided pre-trained models 2 .
Implementation Details of Baseline Attack Methods.
We adopt the source code 3 provided by
Zhao et al.
[48] to implement I, MI, TI, and DI attacks.
The decay factor for MI is set as 1 . 0 .
The kernel size is set as 5 for TI attack, following Gao et al.
The transformation probability is set as 0 .
7 for DI.
For SI and Admix, we adopt the parameters suggested in Wang et al.
The number of copies for SI is set as 5 .
The number of randomly sample m 2 and η of Admix are set as 3 and 0 .
2 respectively.
For implementation of ILA and LinBP, we utilize the source code 4 provided by
Guo et al.
For implementation of TTP, we use the pre-trained generator 5 based on ResNetprovided by [29].
Computational Cost.
Here, we analyze the computational cost of our method. In Algorithmwith global iteration number K , late-start iteration number K LS and inner iteration number T , our RAP-LS requires K + ( K − K LS ) ∗ T forward and backward calculation.
While the original attack algorithm requires K forward and backward calculation. The extra computation cost of RAP-LS is ( K − K LS ) ∗ T times forward and backward calculation. The adversarial example generation process is conducted based on the offline surrogate models. Compared with this offline time cost, the attacking performance is much more important for blackbox attacks.
Besides, our late-start strategy could alleviate the time cost.
Implementation Details of Visualization.
We visualize the flatness of the loss landscape around x adv on surrogate model by plotting the loss change when moving x adv along a random direction with different magnitudes.
Specially, we first sample d from a Gaussian distribution and normalize it on a ℓ 2 unit norm ball, d ← d ∥ d ∥ F .
Then, we calculate the loss change (flatness) f ( a ) with different magnitudes a ,
[Equation]
Considering d is randomly selected, we repeat the above calculation 20 times with different d and take the averaged value to conduct the visualization.
We also add the visualization results about targeted attacks.
C Experimental Results about More baseline attacks
In this section
[Table]
we show the comparison of our method RAP and EOT [1], VT [39], EMI [41]
[Table]
and
Ghost Net [21] attack methods.
C.1 Experimental Results about VT and EMI
In the below Table 8 and
[Table]
we demonstrate the untargeted and targeted attack performance of VT,
EMI
[Table]
and our methods.
We choose MI-TI-DI as the baseline method and follow experimental settings in Section 4.1.
As shown in experimental results, our RAP-LS achieves better performance, especially for targeted attacks.
Compared with VT, RAP-LS gets an increase of 6 . 7% for targeted attacks in terms of average success rate.
This demonstrates the effectiveness of our methods.
[Table]
[Table]
C.2 Experimental Results about Ghost Net attack
For combining Ghost net with RAP, we conduct the experiments on our PyTorch codes following the original TensorFlow codes provided by the authors.
The main idea of ghost network is to perturb skip connections of ResNet to generate ensemble networks.
To achieve this goal, the authors multiply skip connection by the random scalar r sampled from a uniform distribution.
We reimplement this procedure with the hyperparameter about r recommended in the orginal paper on the PyTorch
ResNet-50 model.
The targeted attack results ( % ) are shown in the below Table 10.
We also follow the experimental settings in Section 4.1.
Here, we use GN to represent Ghost Net method.
The results show that ghost network method can improve the transfer attack performance.
Combined with RAP-LS, the adversarial transferability can be further improved especially on the inception-v3 model.
[Table]
C.3 Experimental Results about EOT attack
We conducted the experiment of the EOT baseline.
We choose the ResNet-50 as the source model and MI-TI-DI as the baseline method.
Instead of adding input transformation once like DI, we sample random transformation (resizing and padding) multiple times in each iteration.
Then, we add them to following the expectation of transformation (EOT) [1].
We set the number of sampling as 10.
Our
RAP can be also naturally combined with EOT attack.
The targeted attack results ( % ) are shown in the below Table 11.
We also follow the experimental settings in Section 4.1. As shown in the below table, 1) the EOT attack gets a moderate increase on attack performance compared with the baseline MI-TI-DI attack, which demonstrates that EOT could improve adversarial transferability. Our RAP attack achieves better performance and surpasses the EOT attack by a large margin, especially for Inc-v3 and VGG-16 target models. 3) Combining RAP with EOT can further boost EOT attack performance.
These results demonstrate that RAP could achieve better adversarial transferability and help find better flat local minima. Besides, the combination of RAP and EOT achieves the best performance among them, which demonstrates that these two methods could complement each other.
[Table]
D Experimental Results about More Defense Models
The Evaluation on More Defense Models Here, we show the evaluation of more defense models containing multi-step Adversarial training models in ImageNet [32], Feature Denoising [46], NRP [28], input transformation defense (R & P) [45].
For Feature Denoising, we utilize the pre-trained ResNet-152 model provided by the authors 6 .
For
AT models on ImageNet, we adopt the pre-trained ResNet-50 AT models provided by the authors.
For ℓ ∞ norm, we adopt the ResNet-50 AT model with budget 4 / 255 , which ranks first in the
RobustBench leaderboard 8 .
For ℓ 2 norm, we adopt the ResNet-50 AT model with budget 0 .
The untargeted attack performance is shown in Table 12.
We follow the experimental settings in Section 4.5 of the main submission.
We can observe that our RAP-LS further boosts the transferability of baseline methods on these new defense models, getting a 5 .
5% boost for the average attack success rate.
For NRP, we adopt the pre-trained purifiers provided by the authors 9 .
Since NRP is an offline defense module, we combine it with the two used ensemble AT models and the above two AT models.
The untargeted attack performance is shown in Table 13.
We also follow the experimental settings in
Section 4.5 of the main submission.
Combining NRP with AT models is a much stronger defense mechanism, but RAP-LS still achieves an improvement by 0 .
For R & P, we adopt the source code provided by Dong et al.
[7] to implement it.
We also combine
R & P with the two used ensemble AT models and the two new AT models above.
The untargeted attack performance is shown in Table 14.
We also follow the experimental settings in Section 4.
of the main submission.
For R & P, RAP-LS achieves an 9 . 1% increase in terms of average attack success rate. [Table] [Table]
[Table]
E Additional Experimental Results
In this section [Table] we first show the evaluation of targeted attacks with CE loss in Section E.1.
Then we show the results of ensemble attacks on more diverse network architectures in Section E.2.
In Section
E.3, we report the experimental results w.r [Table] t .
different value of iterations
[Table]
E.1 The Results of Targeted Attacks with CE Loss
Following the settings in main submission, we evaluate the targeted attack performance of the different baseline methods with our method on ResNet-50, DenseNet-121, VGG-16, and Inception-v3.
The results of combinational methods are shown in Table 15.
The RAP-LS outperforms all combinational methods by a significantly margin.
Taking the average attack success rate of all target models as the evaluation metric, RAP-LS achieves 20 .
9% , 18 . 4% , and
1% improvements over the MTDI,
MTDSI and MTDAI, respectively.
[Table]
E.2 The Results of Ensemble Attacks on Diverse Network Architectures
We also take the evaluation of the ensemble attacks on diverse network architecture (Sec.4.5).
We adopt the ensemble-model attack by averaging the logits of different surrogate models, including
ResNet-50, DenseNet-121, VGG- [Table] and Inception-v3.
The transfer attack success rate on diverse models are shown in Table 16.
Compared with results of single model attack in Table 6, the ensemble attack achieve the better performance.
We can observe that our RAP-LS further boosts transferability of the baseline methods on both targeted and untargeted attacks.
We take ViT as target model for example.
For untargeted attacks, RAP-LS achieves average performance improvements of 19 . 2% .
For targeted attacks, RAP-LS achieves average performance improvements of 10 . 4% . [Table]
E.3 The Experimental Results w.r.t . Different Value of Iterations
In the main submission [Table] we report the evaluations of K = 400 . Here
[Table] we further report the performance with different values of K for completeness in Table 17 (targeted attack) and [-Table 
-]Table(untargeted attack). From the results
[Table]
we observe that the attacking performance generally increase as
K increases for most cases, this is also aligned with prior works [48]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]