On Success and Simplicity: A Second Look at Transferable Targeted Attacks
Abstract
Achieving transferability of targeted attacks is reputed to be remarkably difﬁcult, and state-ofthe-art approaches are resource-intensive due to training target-speciﬁc model(s) with additional data.
In our work, we ﬁnd, however, that simple transferable attacks which require neither additional data nor model training can achieve surprisingly high targeted transferability.
This insight has been overlooked mainly due to the widespread practice of unreasonably restricting attack optimization to few iterations.
In particular, we, for the ﬁrst time, identify the state-of-the-art performance of a simple logit loss.
Our investigation is conducted in a wide range of transfer settings, especially including three new, realistic settings: ensemble transfer with little model similarity, transfer to low-ranked target classes, and transfer to the real-world Google Cloud Vision API.
Results in these new settings demonstrate that the commonly adopted, easy settings cannot fully reveal the actual properties of different attacks and may cause misleading comparisons.
Overall, the aim of our analysis is to inspire a more meaningful evaluation on targeted transferability.
1. Introduction
Current work has achieved great success on non-targeted transferability (Dong et al., 2018;
Zhou et al., 2018; Huang et al., 2019; Dong et al., 2019; Xie et al., 2019; Wu et al., 2020b; Li et al., 2020b; Lin et al., 2020; Gao et al., 2020), while several initial attempts (Liu et al., 2017; Dong et al., 2018; Inkawhich et al., 2019) on targeted transferability have shown its extreme difﬁculty.
Targeted transferability is worth exploring since it can raise more practical concerns by causing a speciﬁc, highly dangerous prediction.
However, so far state-of-the-art results can only be achieved by [Table] [Table] resource-intensive transferable attacks (Inkawhich et al., 2020a;b; Naseer et al., 2021).
Speciﬁcally, FDA (Inkawhich et al., 2020a;b) is based on modeling layer-wise feature distributions via class-speciﬁc auxiliary classiﬁers using large-scale labeled data, and then optimizing adversarial perturbations using these auxiliary classiﬁers from across the deep feature space.
TTP (Naseer et al., 2021) is based on training target-speciﬁc Generative Adversarial Networks (GANs) through global and local distribution matching, and then use the trained generator to directly generate perturbations on any given input image.
In this paper, we take a second, thorough look at current research on targeted transferability.
Our main contribution is the ﬁnding that simple transferable attacks , which require neither additional data nor model training but only using simple transfer techniques (MI (Dong et al., 2018),
TI (Dong et al., 2019), and DI (Xie et al., 2019)), can achieve surprisingly high targeted transferability.
We argue that this insight has been overlooked mainly because current research has unreasonably restricted attack optimization to a limited number of iterations.
Another key contribution of our work is, for the ﬁrst time, identifying the superiority of a simple logit loss, which even achieves new state-of-the-art results.
A detailed review of related work on transferable targeted attacks can be found in Appendix A.
We demonstrate the general effectiveness of simple transferable attacks in a wide range of transfer settings, especially including three new, realistic settings: ensemble transfer with little model similarity, transfer to low-ranked target classes, and transfer to the real-world Google Cloud Vision
API.
These new settings can better reveal the actual properties of different attacks than the commonly adopted, easy settings.
Overall, this paper elucidates the weakness in common practice and the limitations of the commonly adopted transfer settings.
We hope our analysis will inspire a more meaningful evaluation on targeted transferability.
2. New Insights into Simple Transferable Attacks
In this section, we revisit simple transferable attacks in the targeted scenario, and provide new insights into them.
Simple transferable attacks are surprisingly good when
[Figure]
given enough iterations to converge.
As can be seen from
Figure 1, while using TI or MI makes little difference to the original poor targeted transferability, somewhat surprisingly, using DI actually yields substantial targeted transferability, and integrating all three transfer techniques leads to the best performance.
This may be explained by the fact that DI can yield more generalizable gradients by randomizing the image augmentation parameters not only across the image dimension but also across iterations.
However, current work has missed this point since most studies have been focused on the MI technique (Liu et al., 2017; Dong et al., 2018;
Inkawhich et al., 2019; 2020a;b).
We can also observe that the simple transferable attacks can converge to much higher targeted transferability with more iterations, in contrast to the fast success in the non-targeted scenario.
Current work has not realized this difference and so naturally followed the research on non-targeted transferability to use few iterations (typically ≤ 20 ) also for targeted transferability.
However, evaluation with few iterations is deﬁnitely problematic.
On the one hand, comparing different optimization processes that have not converged is not meaningful and may cause misleading results.
This observation is consistent with Carlini et al.
(2019), who pointed out that restricting the number of iterations without verifying the attack convergence is one of the common pitfalls in evaluating adversarial robustness.
Several recent defenses have been defeated by simply using more iterations (Tramer et al., 2020).
On the other hand, considering the realistic threat model, it is not meaningful to artiﬁcially restrict the computational power of a practical attack (e.g., to fewer than several thousand attack iterations)
(Athalye et al., 2018).
A simple yet stronger logit attack.
Existing simple transferable attacks are commonly built up on the cross-entropy loss.
However, as pointed out by Li et al.
(2020a), the cross-entropy loss may suffer from the vanishing gradient problem.
In our case with many iterations, this problem becomes more serious since it will cause the attack to stop improving at some point.
To address this problem, we adopt a straightforward idea by directly maximizing the target logit for continuously pushing the image deep into the territory of the target class.
This simple logit loss can be expressed as L Logit =
− l t ( x (cid:48) ) , where l t ( · ) denotes the logit output with respect to the target class.
Although this logit loss is very similar to the well-known C&W loss (Carlini & Wagner, 2017), its superiority on targeted transferability has not been recognized so far.
We ﬁnd that directly using the C&W loss performs worse than this logit loss.
For example, in the single-model transfer setting with the ResNet50 as the white-box source model, the C&W loss yields 20% worse performance on average.
It may be because the C&W loss also involves suppressing other classes, which is not necessary here but could trade off the high target logit.
As can be seen from Figure 1, this logit loss consistently yields better performance than the commonly used cross-entropy loss, and the performance gap becomes increasingly larger as the number of iterations rises.
3. Experimental Evidence on Simple Transferable Attacks
In this section, we provide experimental evidence to show the effectiveness of simple transferable attacks.
Firstly, we evaluate them in a variety of transfer settings, including single-model transfer, ensemble transfer (easy and challenging settings), a worse-case setting with low-ranked target classes, and a real-world attack against the Google Cloud Vision API.
Then, we compare them with two state-of-the-art resource-intensive transferable attacks, TTP (Naseer et al., 2021) and FDA (Inkawhich et al., 2020b).
We use the 1000 images from the development set of
ImageNet-Compatible Dataset (Kurakin et al., 2018), and all these images are associated with 1000 ImageNet class labels and cropped to 299 × 299 before use.
We consider four diverse classiﬁer architectures: ResNet, DenseNet, VGGNet, and Inception.
Our experiments are run on a single
NVIDIA Tesla P100 GPU with 12GB of memory.
We test three different attack losses: cross-entropy (CE),
Po+Trip and logit.
For each image, we use the target label that was ofﬁcially speciﬁed in the dataset.
We use a moderate step size of 2 for all attacks, and ﬁnd that the performance is not sensitive to step size (see evidence in
Appendix C).
Following the common practice, the perturbations are restricted by L ∞ = 16 / 255 .
If not mentioned speciﬁcally, 300 iterations are used for each attack.
All attacks use TI, MI, and DI together with their original optimal hyperparameters.
Speciﬁcally, (cid:107) W (cid:107) 1 = 5 is used for ‘TI’ as suggested by (Gao et al., 2020).
When being executed with a batch size of 20, the optimization of each attack takes about three second per image.
[Figure]
[Table]
Single-model transfer.
Table 1 reports the targeted transferability in cases with different classiﬁers.
As can be seen, the logit loss outperforms the other two by a large margin in all cases.
When comparing different model architectures, we can ﬁnd that the attacks generally achieved lower performance when transferring from the VGGNet16 or Inceptionv3 than from ResNet50 or DenseNet121.
This is consistent with the observations in (Inkawhich et al., 2020a;b), and may be explained by the fact that skip connections (in ResNet and DenseNet121) allow easier generation of highly transferable adversarial examples (Wu et al., 2020a).
The very low performance when transferring to Inception-v3 might be explained by its notable difference with the other three architectures, i.e., the Inception architecture is known to be more conceptually complex (heavily engineered) with multiple-size convolution and two auxiliary classiﬁers.
Ensemble transfer with little model similarity.
A common approach to boosting transferability is to transfer from an ensemble of multiple source models.
Following the common practice, we simply assign equal weights to all the source models.
We found that the commonly adopted ensemble settings (Dong et al., 2018; Tram`er et al., 2018; Dong et al., 2019; Li et al., 2020a) that only involve similar model architectures are so easy that all attacks can reach very high targeted transferability (see evidence in Appendix B).
Therefore, in order to fully reveal the potential of different attacks, we consider a more challenging transfer setting with no architectural overlap between the source ensemble and the target model.
This setting is also more realistic since in the real-world attack scenarios, there is almost unlikely to be a local white-box source model that shares similar
[Table]
architecture to the unknown target model.
Figure 2 shows that the logit attack largely outperforms the other two attacks in this realistic ensemble setting.
Po+Trip yields even worse results than the CE loss maybe because its original hyperparameters cannot ensure optimal effectiveness here.
Again, Inception-v3 is the most difﬁcult model to attack.
A worse-case transfer setting with low-ranked target classes.
In conventional security studies, a comprehensive evaluation of attacks commonly involves a range of settings with varied difﬁculties.
For adversarial attacks in the whitebox scenario, existing studies (Carlini & Wagner, 2017;
Kurakin et al., 2017; Tram ` er et al., 2018; Rony et al., 2019) have also looked at varied difﬁculties regarding the target class.
Speciﬁcally, in the best case, the targeted success is basically equal to non-targeted success, i.e., an attack is regarded to be successful as long as it can success on any arbitrary target other than the original class.
In the average case, the target class is randomly speciﬁed agnostic to the test image, while in the worst case, the target is speciﬁed as the lowest-ranked (least-likely) class in the prediction list of the original image.
However, to the best of our knowledge, current evaluation on transferability has been limited to the best and average cases.
To address this limitation, we consider a worse-case transfer setting by varying the target from the highest-ranked class gradually to the lowest one.
As can be seen from Table 2, there exists non-negligible correlation between the ranking position of the target class and the transferability.
More speciﬁcally, it becomes increasingly difﬁcult as the target moves down the prediction list.
We can also observe that only looking at the results with higher-ranked targets could not reveal the actual properties of different attacks and may lead to misleading conclusion.
Transfer-based attacks on Google Cloud Vision.
Exist- [Table] [Table] ing work that has studied transfer-based attacks against real-world systems were only explored in the non-targeted scenario, and limited to face recognition (Shan et al., 2020;
Cherepanova et al., 2021; Rajabi et al., 2021).
In contrast, we consider a more generally-used image recognition system, the Google Cloud Vision API, in a more challenging, targeted scenario.
For targeted transferability, we evaluate whether or not the target object class appears among the returned (top 10) predictions.
We also report the non-targeted results to give extra insight by evaluating whether or not the correct object class appears among the returned predictions.
Since the Google Cloud Vision predictions do not explicitly correspond to the 1000 ImageNet classes, we treat semantically similar classes as the same class.
Table 3 reports the results averaged over 100 images that originally yield correct predictions.
As expected, targeted transfer is strictly more difﬁcult than the non-targeted transfer.
The logit attack achieved the best results, especially in the targeted scenario.
Figure 4 in Appendix D visualizes some adversarial images, which are shown to have quasiimperceptible perturbations.
In general, our results reveal the potential vulnerability of Google Cloud Vision against simple attacks that need no query interaction.
Compared with resource-intensive transferable attacks.
We ﬁrstly compare the best performed simple attack, the logit attack, with the state-of-the-art TTP, following its original “10-Targets (all-source)” setting (Naseer et al., 2021).
As shown in Table 4, the logit attack achieved high targeted transferability comparable to TTP in both transfer settings, with slightly better performance on average.
We next compare all three simple transferable attacks with
FDA (Inkawhich et al., 2020b).
We chose only the setting with “distal transfer” examples since the authors have not (yet) released their source code.
Speciﬁcally, the adversarial examples are generated by starting from random Gaussian noise and optimizing without any perturbation bounds to unleash the full potential of the attacks.
The results are [Table] averaged over 4000 examples, each of which is optimized towards a random target class.
As suggested by (Inkawhich et al., 2020b), the MI technique is removed since it empirically harms the performance in this unbounded case.
Tableshows that all three simple transferable attacks achieved higher targeted transferability than FDA, with the logit loss achieving the best overall performance.
As can be seen from
Figure 5 in Appendix E, the “distal transfer” examples can somehow reﬂect the target semantics.
This suggests an interesting perspective that targeted transferability is achieved by attacking the robust features because these features are naturally shared by different models (and also humans).
This ﬁnding is different from previous perspective on nontargeted transferability, for which attacking the non-robust features is sufﬁcient (Ilyas et al., 2019).
4. Conclusion and Outlook
In this paper, we have found that simple transferable attacks can achieve surprisingly high transferability as long as they are not unreasonably restricted to few iterations.
We have validated the effectiveness of simple transferable attacks in a wide range of transfer settings, including three newlyintroduced realistic settings that better revealed the actual properties of different attacks.
In particular, we identify that a very simple logit attack can consistently yield the highest targeted transferability, being even competitive with the state-of-the-art resource-intensive approaches.
Overall, we hope our ﬁndings on weakness in common practice and the limitations of the commonly adopted transfer settings will inspire future research to conduct a more meaningful evaluation on targeted transferability.
We hope our analysis can motivate the community to design stronger defenses against transferable attacks, and on the other hand, promote the applications that directly use adversarial examples for social good, such as protecting privacy (Oh et al., 2017; Larson et al., 2018;
Liu et al., 2019;
Cherepanova et al., 2021; Rajabi et al., 2021).
For future work, it is worth a deeper understanding of why different model architectures yield different transferability.
Acknowledgments
This work was carried out on the Dutch national einfrastructure with the support of SURF Cooperative.
A. Related Work on Transferable Attacks
In this section, we review simple transferable attacks and recently-proposed resource-intensive transferable attacks.
Simple transferable attacks that require neither additional data nor model training have been extensively studied in the non-targeted scenario (Dong et al., 2018;
Zhou et al., 2018;
Dong et al., 2019; Huang et al., 2019; Xie et al., 2019;
Lin et al., 2020; Li et al., 2020b; Wu et al., 2020b; Gao et al., 2020), and also explored in the targeted scenario (Liu et al., 2017; Dong et al., 2018; Li et al., 2020a).
These attacks are commonly built up on the well-known IterativeFast Gradient Sign Method (I-FGSM) (Kurakin et al., 2017;
Madry et al., 2018).
In the targeted scenario, I-FGSM can be formulated as:
[Equation]
where x (cid:48)
i denotes the intermediate modiﬁed image in the ith iteration, and y t is the target class label.
In order to ensure the imperceptibility of the modiﬁcations, in each iteration, the perturbations are clipped into some pre-deﬁned L p norm ball, i.e., satisfying (cid:107) x (cid:48) − x (cid:107) p ≤ (cid:15) .
Current research on transferable attacks has commonly adopted the L ∞ norm.
For the loss function J ( · , · ) , most simple transferable attacks have adopted the plain cross-entropy (CE) loss.
However, this commonly used CE loss has been recently pointed out to be not effective in the targeted scenario because it suffers from the vanishing gradient problem (Li et al., 2020a).
Therefore, the authors in (Li et al., 2020a) have proposed a so-called Po+Trip loss to address the problem.
Speciﬁcally, the Poincar´e distance was used to adapt the magnitude of the gradients, which can be formulated as:
[Equation]
where u is the normalized logit vector and v is the one-hot vector with respect to the target class.
In order to further boost the performance, an additional triplet loss was integrated to also push the image away from the original class:
[Equation]
The overall loss function can be formulated as L Po + Trip =
[Equation]
+ λL Trip .
However, in the original work, Po+Trip was evaluated only in an ensemble transfer setting.
Instead of improving the loss function, there are also transfer techniques
(Dong et al., 2018; 2019; Xie et al., 2019; Lin et al., 2020) that aim at preventing the attack optimization from overﬁtting to the speciﬁc source model.
The advantage of such techniques is that they can be generally applied to any loss functions.
In this paper, we study three (Dong et al., 2018; 2019; Xie et al., 2019) of such transfer techniques that have been widely used in the literature:
Momentum Iterative-FGSM [Equation] MI-FGSM
[Equation] 2018) integrates a momentum term, which accumulates previous gradients in order to make more accurate updating.
It can be expressed as:
[Equation]
where g i is the accumulated gradients at the i -th iteration, and µ is the decay factor.
Another similar technique that is based on the Nesterov accelerated gradient was also explored (Lin et al., 2020).
(TI-FGSM)
(Dong et al., 2019) randomly translates the input image during optimization for preventing the attack from overﬁtting to the speciﬁc whitebox source model.
This approach is inspired by the common data augmentation techniques used for preventing overﬁtting in model training.
Instead of calculating gradients for multiple translated images separately, the authors proposed an approximate solution to accelerate the implementation.
It is achieved by directly computing locally smoothed gradients on the original image via convolution with a kernel:
[Equation]
(5) where W is the convolution kernel used for smoothing.
TI-FGSM was originally designed for transferring to adversarially-trained models.
It has been recently pointed out that using relatively small kernel size leads to the optimal transferability when transferring to normally trained models (Gao et al., 2020).
Diverse Input-FGSM (DI-FGSM)
(Xie et al., 2019) is conceptually similar to TI-FGSM, but applies random resizing and padding for the image augmentation.
More importantly, instead of ﬁxing the augmentation parameters all the time
[Table]
as in TI-FGSM [Table] DI-FGSM adds more randomness across iterations.
The attack optimization can be formulated as:
[Equation]
where the stochastic transformation T ( x (cid:48) i , p ) is implemented with probability p at each iteration.
Resource-intensive transferable attacks that require training target-speciﬁc models on large-scale additional data have been recently explored and shown to substantially improve targeted transferability.
Speciﬁcally, Feature Distribution Attack (FDA) (Inkawhich et al., 2020a) has substantially improved targeted transferability on ImageNet by training auxiliary models with additional labeled data.
Each auxiliary model is a small, binary, one-versus-all classiﬁer trained for a speciﬁc target class at a speciﬁc layer.
That is to say, the number of auxiliary models is the number of layers probed multiplied by the number of target classes that are required to model (Inkawhich et al., 2020a).
The attack loss function can be formulated as:
[Equation]
where each auxiliary models F l ( · ) can model the probability that a feature map at layer l is from a speciﬁc target class y t .
FDA ( N ) +xent (Inkawhich et al., 2020b) extends FDA by aggregating features from N layers and also incorporating the cross-entropy loss H ( · , · ) of the original network F ( · ) .
The extended loss function can be expressed as:
[Equation]
where (cid:88) l ∈ L
[Equation]
[Equation] Very recently, TTP (Naseer et al., 2021) has achieved stateof-the-art targeted transferability by directly generating perturbations using target class-speciﬁc GANs that have been trained with global and local distribution matching.
Speciﬁcally, the global distribution matching is achieved by minimizing the Kullback Leibler (KL) divergence, while the local distribution matching is by enforcing the neighbourhood similarity.
In order to further boost the performance, image augmentation techniques, such as rotation, crop resize,
[Figure]
horizontal ﬂip, color jittering and gray scale transformation, have been applied during training.
We refer the readers to the original work for more technical details of TTP.
B. Results in the Easy Ensemble Setting
Here we look at the commonly adopted ensemble transfer setting (Dong et al., 2018; 2019; Li et al., 2020a; Tram ` er et al., 2018) in which each hold-out target model shares similar architecture with some of the white-box ensemble models.
As can be seen from Table 6, all the three attacks reached equally high targeted transferability when given enough iterations to converge.
This indicates that this setting with high model similarity could not fully reveal the actual properties of different attacks.
We can also observe that Po+Trip performs better than the CE loss only atiterations, but becomes worse when they are given enough iterations to converge.
This ﬁnding suggests that evaluating different attacks under only few iterations may cause misleading comparing results.
C. Results with Varied Step Sizes
Recent work has shown that enlarging the step size can improve non-targeted transferability since it can help attack optimization escape from poor local optima (Gao et al., 2020).
Here we also explore the impact of step size setting on targeted transferability.
As can be seen from Figure 3, in general, all attacks are not sensitive to the change of step size, with only a slight improvement when using a larger step size.
We can also observe that the logit attack consistently outperforms the other two in all cases.
D. Adversarial Images on Attacking Google Cloud Vision
E.
“Distal Transfer” Examples with Unbounded Perturbations
1: "goldfish" 9: "ostrich" 37: "box turtle" 52: "thunder snake" 100: "black swan" 107: "jellyfish" 121: "king crab" 150: "sea lion" 200: "Tibetan terrier" 281: "tabby" 288: "leopard" 310: "ant" 319: "dragonfly" 335: "fox squirrel" 340: "zebra" 353: "gazelle" 385: "Indian elephant" 404: "airliner" 417: "balloon" 440: "beer bottle" 515: "cowboy hat" 574: "golf ball" 587: "hammer" 600: "hook" 843: "swing" 890: "volleyball" 937: "broccoli" 963: "pizza"