Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbation
AffiliationAddress
Abstract
Deep neural networks (DNNs) have been shown to be vulnerable to adversarialexamples, which can produce erroneous predictions by injecting imperceptibleperturbations.
In this work, we study the transferability of adversarial examples,which is significant due to its threat to real-world applications where model architecture or parameters are usually unknown.
Many existing works reveal thatthe adversarial examples are likely to overfit the surrogate model that they aregenerated from, limiting its transfer attack performance against different targetmodels.
To mitigate the overfitting of the surrogate model, we propose a novelattack method, dubbed reverse adversarial perturbation (RAP).
Specifically, instead of minimizing the loss of a single adversarial point, we advocate seekingadversarial example located at a region with unified low loss value, by injectingthe worst-case perturbation ( i.e ., the reverse adversarial perturbation) for each stepof the optimization procedure.
The adversarial attack with RAP is formulated asa
min-max bi-level optimization problem.
By integrating RAP into the iterativeprocess for attacks, our method can find more stable adversarial examples whichare less sensitive to the changes of decision boundary, mitigating the overfittingof the surrogate model.
Comprehensive experimental comparisons demonstratethat RAP can significantly boost adversarial transferability.
Furthermore, RAPcan be naturally combined with many existing black-box attack techniques, tofurther boost the transferability.
When attacking a real-world image recognitionsystem, i.e ., Google Cloud Vision API, we obtain 22% performance improvementof targeted attacks over the compared method.
Introduction
Deep neural networks (DNNs) have been successfully applied in many safety-critical tasks, such asautonomous driving, face recognition and verification, etc .
However, it has been shown that DNNmodels are vulnerable to adversarial examples [7, 25], which are indistinguishable from naturalexamples but make a model produce erroneous predictions.
For real-world applications, the DNNmodels are often hidden from users.
Therefore, the attackers need to generate the adversarial examplesunder black-box setting where they do not know any information of the target model.
For blackbox setting, the adversarial transferability matters since it can allow the attackers to attack targetmodels by using adversarial examples generated on the surrogate models.
Therefore, learning how togenerate adversarial examples with high transferability has gained more attentions in the literature[2, 3, 8, 17, 28, 34].
Under white-box setting where the complete information of the attacked model ( e.g ., architecture andparameters) is available, the gradient-based attacks such as PGD [18] have demonstrated good attackperformance.
However, they often exhibit the poor transferiability [2, 34], i.e ., the adversarial examplex adv generated from the surrogate model M S performs poorly against different target models M T .
The previous works attribute that to the overfitting of adversarial examples to the surrogate models[2, 15, 34].
Figure 1 (b) gives an illustration.
The PGD attack aims to find an adversarial point x pgdwith minimal attack loss, while doesn’t consider the attack loss of the neighborhood regions roundx pgd .
Due to the highly non-convex of deep models, when x pgd locates at a sharply local minimum,a slight change on model parameters of M S could cause a large increase of the attack loss, makingx pgd fail to attack the perturbed model.
Many techniques have been proposed to mitigate the overfitting and improve the transferability,including input transformation [3, 34], gradient calibration [8], feature-level attacks [11], and generative models [21], etc.
However, there still exists a large gap of attack performance between thetransfer setting and the ideal white-box setting, especially for targeted attack, requiring more effortsfor boosting the transferability.
In this work, we propose a novel attack method called reverse adversarial perturbation (RAP) toalleviate the overfitting of the surrogate model and boost the transferability of adversarial examples.
We encourage that x adv is not only of low attack loss but also locates at a local flat region, i.e .,the points within the local neighborhood region around x adv should also be of low loss values.
Figure 1 (b) illustrates the difference between the sharp minimum and flat minimum.
When themodel parameter of M S has some slight changes, the variation of the attack loss w.r.t.
the flat localminimum is less than that of the sharp one.
Therefore, the flat local minimum is less sensitive to thechanges of decision boundary.
To achieve this goal, we formulate a min-max bi-level optimizationproblem.
The inner maximization aims to find the worst-case perturbation ( i.e ., that with the largestattack loss, and this is why we call it reverse adversarial perturbation) within the local region aroundthe current adversarial example, which can be solved by the projected gradient ascent algorithm.
Then, the outer minimization will update the adversarial example to find a new point added with theprovided reverse perturbation that leads to lower attack loss.
Figure 1 (a) provides an illustration ofthe optimization process.
For t -th iteration and x t , RAP first finds the point x t + n rap with maxattack loss within the neighborhood region of x t .
Then it updates x t with the gradient calculatedby minimizing the attack loss w.r.t.
x t + n rap .
Compared to directly adopting the gradient at x t ,
RAP could help escape from the sharp local minimum and pursue a relatively flat local minimum.
Besides, we design a late-start variant of RAP (RAP-LS) to further boost the attack effectivenessand efficiency, which doesn’t insert the reverse perturbation into the optimization procedure inthe early stage.
Moreover, from the technical perspective, since the proposed RAP method onlyintroduces one specially designed perturbation onto adversarial attacks, one notable advantage of
RAP is that it can be naturally combined with many existing black-box attack techniques to furtherboost the transferability.
For example, when combined with different input transformations ( e.g ., therandom resizing and padding in Diverse Input [34]), our RAP method consistently outperforms thecounterparts by a clear margin.
Our main contributions are three-fold: 1)
We propose a novel adversarial attack method RAP thatencourages both the adversarial example and its neighborhood region be of low loss value.
2) wepresent a vigorous experimental study and show that RAP can significant boost the adversarialtransferability on both untargeted and targeted attacks; 3) we demonstrate that RAP can be easilycombined with existing transfer attack techniques and outperforms the state-of-the-art performanceby a large margin.
Related Work
The black-box attacks can be categorized into two categories: 1) query-based attacks that conduct theattack based on the feedback of iterative queries to target models, and 2) transfer attacks that use theadversarial examples generated on some surrogate models to attack the target models.
In this work,we focus on the transfer attacks.
For surrogate models, existing attack algorithms such as FGSM [7]and I-FGSM [14] could achieve good attack performance.
However, they often overfit the surrogatemodels and thus exhibit poor transferability.
Recently, many works are proposed to generate moretransferable adversarial examples.
Instead of the I-FGSM, the work of Dong et al.
[2] integrate momentum into the updating strategy and
Lin et al.
[15] use the Nesterov accelerated gradient to boost the transferability.
Data augmentation,which has been shown to be effective in improving model generalization, has also been widely studiedin transfer attack, such as randomly resizing and padding [34], randomly scaling [15], and adversarialmixup [30].
The work of Dong et al.
[3] use a set of translated images to compute gradient and get the better performance against defense models.
There are also some model-specific designs to boostthe transferability.
For example, Wu et al.
[31] find the gradient of skip connections is more crucialto generate more transferable attacks.
The work of Guo et al.
[8] also propose LinBP to utilize moregradient of skip connections during the back-propagation.
However, these methods tend to be specificto a particular model architecture, such as skip connection, and it is nontrivial to extend the findingsto other architectures or modules.
Meanwhile, Huang et al.
[11], Inkawhich et al.
[12, 13] proposeto exploit feature space constraints to generate more transferable attacks.
Yet they need to identitythe best performing intermediate layers or train one-vs-all binary classifies for all attacked classes.
Recently, Zhao et al.
[35] find iterative attacks with much more iterations and logit loss can achieverelatively high targeted transferability and exceed the feature-based attacks.
Apart from the input-specific adversarial attacks, there have been some methods utilizing the generative models to generate the adversarial perturbations [19, 21, 22].
For example, the work of Naseeret al. [21] propose to train a generative model to match the distributions of source and target class,so as to increase the targeted transferability.
However, the learning of the perturbation generator isnontrivial, especially on large-scale datasets.
In summary, the current performance of transfer attacksis still unsatisfactory, especially for targeted attacks.
Wang et al.
[29] also explore the min-max framework for producing the attacks, but they are totallydifferent from us in terms of motivation and formulation.
They are to generate attacks based onmultiple models and samples.
And, the variable of inner maximization in Wang et al.
[29] is theprobability vector, representing the weight assigned to different models or samples.
Methodology
3.1 Preliminaries of Transfer Adversarial Attack
Given an benign sample
( x , y ) ∈
( X , Y ) , the procedure of transfer adversarial attack is firstlyconstructing the adversarial example x adv within the neighborhood region B ϵ ( x ) = { x ′ :
∥ x ′ −x ∥
p ≤ ϵ } by attacking the white-box surrogate model M s ( x ; θ ) :
X → Y , then transferring x advto directly attack the black-box target model M t ( x ; ϕ ) :
X → Y .
The attack goal is to mislead thetarget model, i.e ., M t ( x adv ; ϕ )
̸
= y (untargeted attack), or M t ( x adv ; ϕ ) =
y t (targeted attack) withy t ∈ Y indicting the target label.
Taking the target attack as example, the general formulation ofmany existing transfer attack methods can be written as follows:
[Equation]
The loss function L is often set as the cross entropy (CE) loss [34] or the logit loss [35], which willbe specified in later experiments.
Besides, the formulation of untargeted attack can be easily obtainedby replacing the loss function L and y t by −L and y , respectively.
Since M s is white-box, if G ( · ) is set as the identity function, then any off-the-shelf white-boxadversarial attack method can be adopted to solve Problem (1), such as I-FSGM [14], MI-FGSM[2], etc .
Meanwhile, existing works have designed different G ( · ) functions and developed thecorresponding optimization algorithms, to boost the adversarial transferability between surrogate andtarget models.
For example, G ( · ) is specified as random resizing and padding (DI) [34], translationtransformation (TI) [3], scale transformation (SI) [15], and adversarial mixup (Admix) [30].
3.2 Reverse Adversarial Perturbation
As discussed above, although having good performance for the white-box setting, the adversarialexamples generated from M S exist poor adversarial transferability on M T , especially for targetedattacks.
The previous works attribute this issue to the overfitting of adversarial attack to M S[1, 2, 3, 28, 32].
As shown in Figure 1 (b), when x pgd locates at a sharp local minimum, it is notstable and is sensitive to changes of M S .
When having some changes on model parameters, x pgdcould results in a high attack loss against M S ′ and lead to a failure attack.
To mitigating the overfitting to M S , we advocate to find x adv located at flat local region.
That meanswe encourage that not only x adv itself has low loss value, but also the points in the vicinity of x advhave similarly low loss values.
To this end, we propose to minimize the maximal loss value within a local neighborhood regionaround the adversarial example x adv .
The maximal loss is implemented by perturbing x adv to
[Figure]
maximize the attack loss, named Reverse Adversarial Perturbation (RAP).
By inserting the RAP intothe formulation (1), we aim to solve the following problem,
[Equation]
where
[Equation]
with n rap indicating the RAP, and ϵ n defining its search region.
The above formulations Equation (2)and Equation (3) correspond to the targeted attack, and the corresponding untargeted formulationscan be easily obtained by replacing the loss function L and y t by −L and y , respectively.
It is a min-max bi-level optimization problem [16], and can be solved by iteratively optimizing theinner maximization and the outer minimization problem.
Specifically, in each iteration, given x adv ,the inner maximization w.r.t.
n rap is solved by the projected gradient ascent algorithm:
[Equation]
The above update is conducted by T steps, and α n = ϵ n T .
Then, given n rap , the outer minimizationw.r.t.
x adv can be solved by any off-the-shelf algorithm that is developed for solving Equation (1).
For example, it can be undated by one step projected gradient descent, as follows:
[Equation]
with Clip B ϵ ( x ) ( a ) clipping a into the neighborhood region B ϵ ( x ) .
The overall optimization procedureis summarized in Algorithm 1.
Moreover, since the optimization w.r.t.
x adv can be implemented byany off-the-shelf algorithm for solving Problem Equation (1), one notable advantage of the proposed
RAP is that it can be naturally combined with any one of them, such as the input transformationmethods [3, 15, 30, 34].
A Late-Start (LS) Variant of RAP.
In our preliminary experiments, we find that RAP requires moreiterations to converge and the performance is slightly lower during the initial iterations, compared toits baseline attack methods.
As shown in Figure 2, we combine MI [2] and Admix [30] with RAP,and adopt ResNet-50 as the surrogate model.
We take the evaluation on 1000 images from ImageNet(see Sec.4.1).
It is observed that the method with RAP (see the orange curves) quickly surpassesits baseline method (see the blue curves) and finally achieves much higher success rate with moreiterations, which verify the effect of RAP on enhancing the adversarial transferability.
However, itis also observed that the performance of RAP is slightly lower than its baseline method in the earlystage.
The possible reason is that the early-stage attack is of very weak attack performance to thesurrogate model.
In this case, it may be waste to pursue better transferable attacks by solving themin-max problem.
A better strategy may be only solving the minimization problem Equation (1)in the early stage to quickly achieve the region of relatively high adversarial attack performance,then starting RAP to further enhance the attack performance and transferability simultaneously.
Thisstrategy is denoted as RAP with late-start (RAP-LS), whose effect is preliminarily supported by theresults shown in Figure 2 (see the green curve) and will be evaluated extensively in later experiments.
Input: Surrogate model M s , benign data ( x , y ) , target label y t , loss function L , transformation G , the global iteration number K late-start iteration number K LS of RAP, as well as hyper-parameters in optimization (specified in later experiments
: the adversarial example x adv
3.3 A Closer Look at RAP
To verify whether RAP can help us find a x adv located at the local flat region or not, we use ResNet-50as surrogate model and conduct the untargeted attacks.
We visualize the loss landscape around x advon M S by plotting the loss variations when we move x adv along a random direction with differentmagnitudes a .
The details of the calculation are provided in supplementary materials .
Figure 3 plotsthe visualizations.
We take I-FGSM [14] (denoted as I), MI [2], DI [34], and MI-TI-DI (MTDI) asbaselines attacks and combined them with RAP.
We can see that comparing to the baselines, RAPcould help find x adv located at the flat region.
Experiments
4.1 Experimental Settings
Dataset and Evaluated Models.
We conduct the evaluation on the ImageNet-compatible dataset 185comprised of 1,000 images.
For the surrogate models, we consider the four widely used networkarchitectures: Inception-v3 (Inc-v3) [26], ResNet-50 (Res-50) [9], DenseNet-121 (Dense-121) [10],and VGG-16bn (VGG-16) [24].
For target models, apart from the above models, we also utilize morediverse architectures: Inception-ResNet-v2 (Inc-Res-v2) [27], NASNet-Large (NASNet-L) [36], and
ViT-Base/16 (ViT-B/16)
For defense models, we adopt the two widely used ensemble adversarialtraining (AT) models: adv-Inc-v3 (Inc-v3 adv ) and ens-adv-Inc-Res-v2 (IncRes-v2 ens ) [28].
Besides,we also test multi-step AT model [23], imput transformation defense [32], feature denoising [33], andpurification defense (NRP) [20].
Compared Methods.
We adopt I-FGSM [14] (denoted as I), MI [2], TI [3], DI [34], SI [15], Admix[30], ILA [11], LinBP [8], and the generative targeted attack method TTP [21].
We also considerthe combination of baseline methods, including MI-TI-DI (MTDI), MI-TI-DI-SI (MTDSI), and
MI-TI-DI-Admix (MTDAI).
Implementation Details.
For untargeted attack, we adopt the Cross Entropy (CE) loss.
For targetedattack, apart from CE, we also experiment with the logit loss, where Zhao et al.
[35] shows itbehaves better for targeted attack.
The adversarial perturbation ϵ is restricted by ℓ ∞ = 16 / 255 .
The step size α is set as 2 / 255 and number of iteration K is set as 400 for all attacks.
In the
[Table]
following, we mainly show the results at K =
and the results at different value of K are shownin supplementary materials .
For RAP, we set K LS as 100 and α n as 2 / 255 .
We set ϵ n as 12 / 255 for
I and TI in untargeted attack and 16 / 255 for other attacks in all other settings.
The computationalcost is shown in supplementary materials .
4.2 The Evaluation of Untargeted Attacks
Baseline Methods.
We first evaluate the performance of RAP and RAP-LS with different baselineattacks, including I, MI, DI, TI, SI, and Admix.
The results are shown in Table 1.
For instance, the‘MI/ +RAP/ +RAP-LS’ denotes the methods of baseline MI, MI+RAP, and MI+RAP-LS, respectively.
RAP achieves the significant improvements for all methods on each target model.
For average attacksuccess rate of all target models, RAP outperforms the I and MI by 9 . 6% and 16 .
3% , respectively.
For TI, DI, SI, and Admix, RAP gets the improvements by 10 . 2% , 10 . 9% , 9 .
3% , and 6 . 3% .
Withlate-start, RAP-LS further enhance the transfer attack performance for almost all methods.
Combinational Methods.
Prior works demonstrate the combination of baseline methods couldlargely boost the adversarial transferability [30, 35].
We also investigate of behavior of RAP whenincorporated with the combinational attacks.
The results are shown in Table 2.
As shown in thetable, there exist the clear improvements of the combinational attacks over all baseline attacks shownin Table 1.
In addition, our RAP-LS further boosts the average attack success rate of the threecombinational attacks by 6 . 9% , 2 . 6% , and 1 . 7% respectively.
Combined with the three combinationalattacks, RAP-LS achieves 95 .
6% , and 98 .
3% average attack success rate, respectively.
Theseresults demonstrate RAP can significantly enhance the transferability.
4.3 The Evaluation of Targeted Attacks
We then evaluate the targeted attack performance of the different methods with RAP.
The results withlogit loss are presented and the results with CE loss are shown in supplementary materials .
Baseline Methods.
The results of RAP with baseline attacks are shown in Table 3.
From theresults, RAP is also very effective in enhancing the transferability in targeted attacks.
Taking ResNet50 and DenseNet-121 as surrogate models for example, the average performance improvementsinduced by RAP are 5 .
0% (I), 8 .
1% (MI), 4 .
6% (TI), 10 .
4% (DI), 18 .
5% (SI), and 15 .
1% (Admix),respectively.
Comparing to the ResNet-50 and DenseNet-121, the baseline attacks generally achievelower transferability when using the VGG-16 or Inception-v3 as the surrogate models, which has alsobeen verified in existing works [34, 35].
However, for Inception-v3 and VGG-16 as the surrogatemodels, RAP also consistently boosts the transferability under all cases.
With late-start, RAP-LS
[Table] [Table]
could further improve the transferability of RAP for most attacks.
The average attack success rateunder all attack cases of RAP-LS is 2 .
6% higher than that of RAP.
Combinational Methods.
As did in the untargeted attacks, we also evaluate the performance ofcombinational methods.
The results are shown in Table 4.
Similar to the findings in untargetedattacks, the combinational methods obtain significantly improvements over baseline methods.
The
RAP-LS outperforms all combinational methods by a significantly margin.
For example, taking theaverage attack success rate of all target models as evaluation metric, RAP-LS obtains 14 .
8% ,9 . 3% improvements over the MTDI, MTDSI and MTDAI, respectively.
4.4 The Comparison with Other Types of Attacks
[Figure]
[Table]
Apart from the baseline and the combinational methods, we also experiment withmore diverse attack methods, includingthe model-specific attack LinBP [8], thefeature-based attack ILA [11], and thegenerative targeted attack TTP [21].
The
LinBP depends on the skip connection andthe authors only provide the source codeabout ResNet-50.
We use their released code and thus conduct experiments with ResNet-50 as
M S .
The results of LibBP and ILA are shown in Table 5, where we also implement the variantsof LinBP following Guo et al.
[8], inlcuding LinBP-ILA, LinBP-ILA-SGM, LinBP-MI-DI, and
LinBP-MI-DI-SGM.
We observe that our MI-DI-RAP significantly outperforms the LinBP and ILA,especially for the targeted attacks.
Compared with the second-best method ( i.e ., LinBP-MI-DI-SGM),we obtain a large improvement by 33 . 5% on average ASR of targeted attacks.
[Table]
TTP [21] is the state-of-the-art generative method to conducttargeted attack.
To compare with it, we adopt the generatorsbased on ResNet-50 provided by the authors.
Since TTPneeds to train the perturbation generator for each targetedclass, we follow their “10-Targets (all-source)" setting, as didin Zhao et al.
The results are shown in Table 6, whereour MTDSI+RAP-LS behaves best and outperforms TTP and MTDI by large margins of 14 . 9% and25 .
7% , respectively.
[Table] [Table]
4.5 The Evaluation on Diverse Network Architectures and Defense Models
To further demonstrate the efficacy of RAP
[Table]
we evaluate our method on more diverse networkarchitectures, including Inception-ResNet-v2, NASNet-Large and ViT-Base/16.
We adopt ResNet-50as the surrogate model and the results are shown in Table 7, col 2-7 .
As shown in the table, theproposed RAP-LS achieves significant improvements for all three combinational methods on alltarget models, and MTDAI+RAP-LS achieves the best performance for diverse models.
For MTDAI,the average performance improvements induced by RAP-LS is 5.9% and 7.8% for untargeted andtargeted attacks, respectively.
Since ViT is based on the transformer architecture that totally beingdifferent from convolution models, the transfer attacks based on Resnet-50 behave relatively poor on it,especially on targeted attacks.
Yet our RAP-LS still obtains consistent improvements for all comparedmethods.
We also consider the ensemble-model attack on these diverse network architectures and theresults are given in supplementary materials.
Furthermore, we evaluate RAP on attacking defense models.
We choose ensemble adversarial training(AT) model, adv-Inc-v3 and ens-adv-Inc-Res-v2 [28], multi-step AT model [23], imput transformationdefense [32], feature denoising [33], and purification defense (NRP) [20].
We only demonstrate theresults of ensemble AT in main submission.
The evaluations of other defense models are shown insupplementary materials .
Following prior works [30, 34], we adopt the ensemble-model attack byaveraging the logits of different surrogate models, including ResNet-50, ResNet-101, Inception-v3,and Inception-ResNet-v2.
The transfer attack success rate on defense models are shown in Table 7,col 8-11 .
We can observe that our RAP-LS further boosts transferability of the baseline methods onboth targeted and untargeted attacks.
For untargeted attacks, RAP-LS achieves average performanceimprovements of 9 . 8% and 14 .
1% on Inc-v3 adv and IncRes-v2 ens , respectively.
For targeted attacks,the average performance improvements of RAP-LS are 14 .
8% and 11 .
1% , respectively.
4.6 Ablation Study
We conduct ablation study on the hyper-parameters of the proposed RAP, including the size ofneighborhoods ϵ n , the iteration number of inner optimization T and late-start K LS .
We adopttargeted attacks with ResNet-50 as the surrogate model.
We first evaluate the effect of ϵ n and T .
We consider different values of ϵ n , including 2 / 255 , 4 / 255 ,8 / 255 , 12 / 255 , 16 / 255 , and 20 / 255 .
In Figure 4 (a), we plot the tendency curves of the targetedattack success rate under different values of ϵ n and T .
Note that in Sec. 3.2, we set α n = ϵ n /T .
Thus for a fixed ϵ n , larger T indicating lower stepsize α n .
The minimum stepsize of α n is set to2 / 255 .
We have the following observation from the plot: for a fixed ϵ n , the more iterations T , thebetter attack performance.
Thus, we adopt a relatively smaller α n = 2 / 255 in our experiments.
In
Figure 4 (b-d), we further plot the results of different attack methods and target models w.r.t .
ϵ n ,where α n = 2 / 255 .
As shown in the plots, the larger ϵ n generally improves the attack performance.
For Inception-v3 and DenseNet-121, the improvements become mild for even larger ϵ n .
Overall, thevalue of 12 or 16 could lead to satisfactory result under most cases.
Then we conduct the ablation study of K LS .
In Figure 5, we report the targeted attack success rate of
I, MI, DI, and MI-TI-DI combined with RAP-LS with K LS = 0 , 25 , 50 , 100 , 150 , 200 .
Note that the
RAP-LS with K LS = 0 reduces to RAP.
As shown in the plots, the proposed late-start strategy canfurther boost attack performance of RAP for most cases.
In general, the performance improvementsincrease as K LS increases, and then become mild when K LS is larger than 100.
The suitable valueof K LS is relatively consistent among different methods and target models.
4.7 The Targeted Attack Against Google Cloud Vision API
Finally, we conduct the transfer attacks to attack a practical and widely used image recognition system,
Google Cloud Vision API, and in the more challenging targeted attack scenario.
MTDAI-RAP-LSbehaves the best performance in above experiments
, so we choose it to conduct the attack.
We takethe evaluation on randomly selected 500 images and use ResNet-50 as surrogate model.
As the APIreturns 10 predicted labels for each query, to evaluate the attacking performance, we test whether ornot the target class appears in the returned predictions.
Since the predicted label space of Google
Cloud Vision API do not fully correspond to the 1000 ImageNet classes, we manually treat classeswith similar semantics to be the same classes.
In comparison, the baseline MTDAI successfullyattacks 232 images against the Google API.
Our RAP-LS achieves a large improvement, successfullyattacking 342 images, leading to a 22.0% performance improvements.
These demonstrates the highefficacy of our method to improve transferability on real-world system.
Conclusion
In this work, we study the transferability of adversarial examples that is significant for black-boxattacks.
The transferability of adversarial examples is generally influenced by the overfitting ofsurrogate models
To alleviate this, we propose to seeking adversarial examples that locate at flatter local regions.
That is, instead of optimizing the pinpoint attack loss we aim to obtain a consistently low loss at the neighbor regions of the adversarial examples.
We formulate this as a min-max bi-level optimization problem, where the inner maximization aims to inject the worse-case perturbation for the adversarial examples.
We conduct a rigorous experimental study, covering untargeted attack and targeted attack, standard and defense models and a real-world Google Clould Vision API.
The experimental results demonstrate that RAP can significantly boost the transferability of adversarialexamples. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See final part of Section 1.(b)
Did you describe the limitations of your work?
Yes] See Supplementary Materials
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Supplementary Materials
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Supplementary Materials
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] No, all the used data are well-known public benchmark datasets
such as CIFAR10, ImageNet.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] No, all the used data are well-known public
datasets such as CIFAR
If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A] Not applicable.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A] Not applicable.
[Table]
A Social Impact
Deep neural networks (DNNs) have been successfully applied in many safety-critical tasks, suchas autonomous driving, face recognition and verification, etc .
And adversarial samples have poseda serious threat to machine learning systems.
For real-world applications, the DNN model as wellas the training dataset, are often hidden from users.
Therefore, the attackers need to generate theadversarial examples under black-box setting where they do not know any information of the targetmodel.
For black-box setting, the adversarial transferability matters since it can allow the attackers toattack target models by using adversarial examples generated on the surrogate models.
This workcan potentially contribute to understanding of transferability of adversarial examples.
Besides, thebetter transferability of adversarial examples calls the machine learning and security communitiesinto action to create stronger defenses and robust models against black-box attacks.
B Implementation Details
We conducted all experiments in an Nvidia-V100 GPU.
And we run all experiments 3 times andaverage all results over 3 random seeds.
Dataset The used two datasets are licensed under MIT.
Imagenet is licensed under Custom (noncommercial).
Implementation Details of Evaluated Models.
For ResNet-50, DenseNet-121, VGG-16, Inceptionv3, we adopt the pre-trained models provided by torchvision package.
For Inception-ResNet-v2,
NASNet-Large, ViT-Base/16, adv-Inc-v3, and ens-adv-Inc-Res-v2, we adopt the provided pre-trainedmodels 2 .
Implementation Details of Baseline Attack Methods.
We adopt the source code 3 provided by
Zhao et al.
[35] to implement I, MI, TI, and DI attacks.
The decay factor for MI is set as 1 . 0 .
Thekernel size is set as 5 for TI attack, following Gao et al.
The transformation probability is set as0 .
7 for DI.
For SI and Admix, we adopt the parameters suggested in Wang et al.
The numberof copies for SI is set as 5 .
The number of randomly sample m 2 and η of Admix are set as 3 and0 .
2 respectively.
For implementation of ILA and LinBP, we utilize the source code 4 provided by
Guo et al.
For implementation of TTP, we use the pre-trained generator 5 based on ResNet-50provided by [21].
Computation Cost.
We conducted all experiments in an Nvidia-V100 GPU.
Taking MTDI asan example, its computation time is 2 hours.
Combining RAP or RAP-LS, the computation timebecomes 5.5 or 4 hours.
Implementation Details of Visualization.
We visualize the flatness of the loss landscape aroundx
adv on surrogate model by plotting the loss change when moving x adv along a random directionwith different magnitudes.
Specially, we first sample d from a Gaussian distribution and normalize iton a ℓ 2 unit norm ball, d ← d ∥ d ∥ F .
Then, we calculate the loss change (flatness) f ( a ) with differentmagnitudes a ,
[Equation]
Considering d is randomly selected, we repeat the above calculation 20 times with different d andtake the averaged value to conduct the visualization.
We also add the visualization results about targeted attacks.
C Experimental Results about More Defense Models
The Evaluation on More Defense Models Here, we show the evaluation of more defense modelscontaining multi-step Adversarial training models in ImageNet [23], Feature Denoising [33], NRP[20],
input transformation defense
R & P)
For Feature Denoising, we utilize the pre-trained ResNet-152 model provided by the authors 6 .
For
AT models on ImageNet, we adopt the pre-trained ResNet-50 AT models provided by the authors 528.
For ℓ ∞ norm, we adopt the ResNet-50 AT model with budget 4 / 255 , which ranks first in the
RobustBench leaderboard 8 .
For ℓ 2 norm, we adopt the ResNet-50 AT model with budget 0 .
Theuntargeted attack performance is shown in Table 8.
We follow the experimental settings in Section4.5 of the main submission.
We can observe that our RAP-LS further boosts the transferability ofbaseline methods on these new defense models, getting a 5 .
5% boost for the average attack successrate.
For NRP, we adopt the pre-trained purifiers provided by the authors 9 .
Since NRP is an offline defensemodule, we combine it with the two used ensemble AT models and the above two AT models.
Theuntargeted attack performance is shown in Table 9.
We also follow the experimental settings in
Section 4.5 of the main submission.
Combining NRP with AT models is a much stronger defensemechanism, but RAP-LS still achieves an improvement by 0 .
For R & P, we adopt the source code provided by Dong et al.
[4] to implement it.
We also combine
R & P with the two used ensemble AT models and the two new AT models above.
The untargetedattack performance is shown in Table 10.
We also follow the experimental settings in Section 4.5of
the main submission.
For R & P, RAP-LS achieves an 9 . 1% increase in terms of average attacksuccess rate.
[Table]
D Additional Experimental Results
In this section, we first show the evaluation of targeted attacks with CE loss in Sec.
D.1.
Then weshow the results of ensemble attacks on more diverse network architectures in Sec.
D.2.
In Sec.
D.3,we report the experimental results w.r.t .
different value of iterations.
[Table]
[Table]
D.1 The Results of Targeted Attacks with CE Loss
Following the settings in main submission
[Table]
we evaluate the targeted attack performance of the differentbaseline methods with our method on ResNet-50, DenseNet-121, VGG-16, and Inception-v3.
Theresults of combinational methods are shown in Table 11.
The RAP-LS outperforms all combinationalmethods by a significantly margin.
Taking the average attack success rate of all target models asthe evaluation metric, RAP-LS achieves 20 .
9% , 18 . 4% , and
1% improvements over the MTDI,
MTDSI and MTDAI, respectively.
[Table]
D.2 The Results of Ensemble Attacks on Diverse Network Architectures
We also take the evaluation of the ensemble attacks on diverse network architecture (Sec.4.5).
Weadopt the ensemble-model attack by averaging the logits of different surrogate models, including
ResNet-50, DenseNet-121, VGG-16, and Inception-v3.
The transfer attack success rate on diversemodels are shown in Table 12.
Compared with results of single model attack in Table 7, the ensembleattack achieve the better performance.
We can observe that our RAP-LS further boosts transferabilityof the baseline methods on both targeted and untargeted attacks.
We take ViT as target model forexample.
For untargeted attacks, RAP-LS achieves average performance improvements of 19 . 2% .
For targeted attacks, RAP-LS achieves average performance improvements of 10 . 4% .
D.3 The Experimental Results w.r
t . Different Value of Iterations
In the main submission, we report the evaluations of K = 400 .
{+Here, we further report theperformance with different values of K for completeness in Table 13 (targeted attack) and Table 
+}
[Table]
(untargeted attack).
From the results, we observe that the attacking performance generally increase as
[Table]
K increases for most cases, this is also aligned with prior works [35]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]