B EYOND I MAGE N ET A TTACK : T OWARDS C RAFTING A DVERSARIAL E XAMPLES FOR B LACK - BOX D OMAINS
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial examples have posed a severe threat to deep neural networks due to their transferable nature.
Currently, various works have paid great efforts to enhance the cross-model transferability, which mostly assume the substitute model is trained in the same domain as the target model.
However, in real life, the relevant information of the deployed model is unlikely to leak.
Hence, it is vital to build a more practical black-box threat model to overcome this limitation and evaluate the vulnerability of deployed models.
In this paper, with only the knowledge of the ImageNet domain, we propose a Beyond ImageNet Attack (BIA) to investigate the transferability towards black-box domains (unknown classiﬁcation tasks).
Speciﬁcally, we leverage a generative model to learn the adversarial function for disrupting low-level features of input images.
Based on this framework, we further propose two variants to narrow the gap between the source and target domains from the data and model perspectives, respectively.
Extensive experiments on coarse-grained and ﬁne-grained domains demonstrate the effectiveness of our proposed methods.
Notably, our methods outperform state-of-the-art approaches by up to 7.71% (towards coarse-grained domains) and 25.91% (towards ﬁne-grained domains) on average.
Our code is available in the supplementary materials.
I NTRODUCTION
Deep neural networks (DNNs) have achieved remarkable success in the image classiﬁcation task in recent years.
Nonetheless, advances in the ﬁeld of adversarial machine learning (Szegedy et al., 2014; Goodfellow et al., 2015) make DNNs no longer reliable.
By adding a well-designed perturbation on a benign image ( a.k.a adversarial attack), the resulting adversarial examples can easily fool state-of-the-art DNNs.
To make the matter worse, the adversarial attack technique can even be applied in the physical world (Sharif et al., 2016;
Kurakin et al., 2017a;
Xu et al., 2020; Duan et al., 2021), which inevitably raises concerns about the stability of deployed models.
Therefore, exposing as many “blind spots” of DNNs as possible is a top priority.
Generally, deployed models are mainly challenged with two threat models: white-box and blackbox.
For white-box threat model (Kurakin et al., 2017b; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017; Shi et al., 2019), the attacker can obtain complete knowledge of the target model, such as the gradient for any input.
However, deployed models are usually opaque to unauthorized users.
In this scenario, prior black-box works (Poursaeed et al., 2018;
Dong et al., 2018; Xie et al., 2019; Inkawhich et al., 2020; Wang et al., 2021) mostly assume that the source data for training the target model is available and mainly explore the cross-model transferability among models trained in the same data distribution.
Speciﬁcally, perturbations are crafted via accessible white-box model ( a.k.a substitute model), and resulting adversarial examples sometimes can fool other black-box models as well.
Yet, these works still ignore a pivotal issue: A model owner is unlikely to leak the relevant information of the deployed model.
To overcome this limitation, query-based black-box attacks (Papernot et al., 2016; Brendel et al., 2018; Chen et al., 2020; Li et al., 2021) are proposed, which adjust adversarial examples just according to the output of the target model.
However, the resource-intensive query budget is extremely costly and inevitably alerts the model owner.
Therefore, we need a more “practical” black-box threat model to address this concern, i.e. , without any clue about the training data distribution as well as the pre-trained model based on it, and even
[Figure]
querying is forbidden.
Intuitively, this threat model is more challenging to build for attackers and more threatening to model owners.
To the best of our knowledge, a recent work called CDA (Naseer et al., 2019) is the ﬁrst to attempt such an attack.
Speciﬁcally, it learns a transferable adversarial function via a generator network against a different domain (training data and pre-trained model are all from ChestX-ray (Wang et al., 2017) domain).
During inference, it directly crafts adversarial examples for benign ImageNet images to fool target ImageNet pre-trained models.
However, its cross-domain transfer strength is still moderate.
Besides, relying on small-scale datasets to train a generator may limit the generalization of the threat model.
Considering that ImageNet is a large-scale dataset containing most of common categories in real life and there are various off-the-shelf pre-trained models, one can easily dig out much useful information to build a strong threat model.
Therefore, in this paper, solely relying on the knowledge of the ImageNet domain, we introduce an effective Beyond ImageNet Attack (BIA) framework to enhance the cross-domain transferability of adversarial examples.
To reﬂect the applicability of our approach, we consider eight different image classiﬁcation tasks (listed in Table 1).
Figureillustrates an overview of our method.
Particularly, we learn a ﬂexible generator network G θ against
ImageNet domain.
Instead of optimizing the domain-speciﬁc loss function like CDA, our method focuses on disrupting low-level features.
Furthermore, we propose two variants based on the vanilla
BIA to narrow the gap between source and target domains.
Speciﬁcally, from the data perspective, we propose a random normalization ( RN ) module to simulate different data distributions; from the model perspective, we propose a domain-agnostic attention ( DA ) module to capture essential features for perturbing.
In the inference phase, our G θ accepts images of any domain as the input and crafts adversarial examples with one forward propagation.
Extensive experiments demonstrate the effectiveness of our proposed methods.
Towards the coarse-grained and ﬁne-grained domains, we can outperform state-of-the-art approaches by up to 7.71% and 25.91% on average, respectively.
Besides
[Table]
our methods can also enhance the cross-model transferability in the source domain
[Table] [Table]
R ELATED W ORKS
Iterative Optimization Approaches.
Under the black-box threat model, iterative attack methods are a popular branch, which usually adopt domain-speciﬁc loss or intermediate feature loss to craft
[Figure]
adversarial examples.
For the former, Madry et al.
(2018) extend Goodfellow et al.
(2015) to perform projected gradient descent from randomly chosen starting points inside (cid:15) -ball.
Dong et al.
(2018) introduce momentum term to stable the update direction.
Xie et al.
(2019) apply random transformations of the input at each iteration, thus mitigating overﬁtting.
Gao et al.
(2020) propose patch-wise perturbation to better cover the discriminative region.
Wu et al.
(2020a) explore the security weakness of skip connections (He et al., 2016; Huang et al., 2017) to boost adversarial attacks.
Different from the methods mentioned above, intermediate feature-based methods focus on disrupting low-level features.
For example, Zhou et al.
(2018) maximize the Euclidean distance between the source image and target image in feature space and introduce regularization on perturbations to reduce variations.
Inkawhich et al.
(2019) make the source image close to the target image in feature space.
Lu et al.
(2020) propose a dispersion reduction attack to make the low-level features featureless.
Naseer et al.
(2020) design a self-supervised perturbation mechanism for enabling a transferable defense approach.
Wu et al.
(2020b) compute model attention over extracted features to regularize the search of adversarial examples.
Generator-oriented Approaches.
Compared with iterative optimization approaches, generatororiented attacks are more efﬁcient ( i.e. , only need one inference) to generate adversarial examples.
In this branch, Baluja & Fischer (2017) propose an adversarial transformation network to modify the output of the classiﬁer given the original input.
Poursaeed et al.
(2018) present trainable deep neural networks for producing both image-agnostic and image-dependent perturbations.
Naseer et al.
(2019) leverage datasets from other domain instead of ImageNet to train generator networks against pre-trained ImageNet models, and inference is performed on ImageNet domain with the aim of fooling black-box ImageNet model.
They also attempt a practical black-box threat model (from
ChestX-ray to ImageNet), and the attack success rate can outperform the result of Gaussian noise.
T RANSFERABLE A DVERSARIAL E XAMPLES BEYOND I MAGE N ET 3. P ROBLEM F ORMULATION
Given a target deep learning classiﬁer f t ( · ) trained in a speciﬁc data distribution χ t , we aim to craft a human-imperceptible perturbation for the benign image x t ∼ χ t from the target domain with the only available knowledge of source ImageNet domain (including pre-trained model f s ( · ) and data distribution χ s ).
Formally, suppose we have a threat model M θ ∗ whose parameter θ ∗ is solely derived from the source domain, our goal is to craft adversarial examples for x t from target domain so that they can fool the f t ( · ) successfully:
[Equation]
where (cid:15) is the maximum perturbation to ensure x t is minimally changed.
Intuitively, crafting adversarial examples for the black-box domain is very challenging.
As shown in Table 1 and Figure 2, images from different domain vary greatly.
3. P RELIMINARY
Iterative/Single-step optimization methods (Goodfellow et al., 2015; Madry et al., 2018; Mao et al., 2021;
Li et al., 2021) and generator-oriented methods (Baluja & Fischer, 2017; Poursaeed et al., 2018; Naseer et al., 2019;
Li et al., 2020) are two popular branches for building the threat model.
Since the attacker has the large-scale ImageNet training set at hand, there is no reason not to take full advantage of them.
Therefore, in this paper, we adopt the generator-oriented framework which learns a transferable adversarial function via a generative model G θ .
Given that the threat model aims at crafting transferable adversarial examples for unseen blackbox domains, relying on the last layer with domain-speciﬁc loss functions ( e.g. , relativistic crossentropy loss adopted by Naseer et al.
(2019)) is less effective because this might lead to overﬁtting in the source domain.
In contrast, the intermediate layers of the DNN presumably extract general features (Yosinski et al., 2014) which may share across different models.
Hence, our proposed
Beyond ImageNet Attack (BIA) framework turns to destroy the low-level features of the substitute model at a speciﬁc layer L as much as possible.
As illustrated in Figure 1, G θ is learned to decrease the cosine similarity between adversarial example x (cid:48) s and benign image x s ∈ R N × H s × W s (sampled from χ s ) to make the feature featureless:
[Equation]
In the inference phase, our generator G θ ∗ can directly craft adversarial examples for input images x t ∈ R N × H t × W t from the target domain:
[Equation]
The resulting adversarial examples x (cid:48) t are depicted in Figure 13 of Appendix.
Compared with CDA, our BIA is more effective in both source (white-box) and target (black-box) domains.
Yet, as shown in Figure 3, crafting more transferable adversarial examples still has some challenges:
• Data Perspective: The distribution ( i.e. , mean and standard deviation) of source domain is largely different from the target domain.
For example, the standard deviation of ImageNet is about twice that of SVHN.
• Model Perspective: Although some feature map of f Ls ( · ) can capture the object ( ∈ χ t ) for feature representation ( e.g. , the ﬁrst feature map in Figure 3), there are also some feature maps that are signiﬁcantly biased ( e.g. , the second feature map in Figure 3).
To alleviate the concern of generating poor transferable adversarial examples that may arise from the above limitations, we propose two variants, equipped with random normalization ( RN ) module or domain-agnostic attention ( DA ) module, respectively.
3. R ANDOM N ORMALIZATION M ODULE
Generally, DNNs (Simonyan & Zisserman, 2015; He et al., 2016; Huang et al., 2017) are usually equipped with normalization for input images 1 so that they can be modeled as samples from the standard normal distribution.
However, as illustrated in Figure 3 (left), the distribution of dataset from the different domain can vary dramatically.
Thus, training against a speciﬁc domain may limit the generalization of the resulting G θ ∗ .
Besides, the commonly used strategy of label-preserving data augmentation (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015) is less effective because it has little effect on changing the distribution of the inputs (more details are shown in Appendix A.4).
To overcome this limitation, fusing knowledge from different domains might be helpful.
As shown in Table 3 of Naseer et al.
(2019), using training data from other domains against the pre-trained model in the source domain usually enhances the transferability of adversarial examples towards the target domain.
However, this setup is not feasible because the available knowledge for a more practical threat model may be limited, i.e. , restricted to one domain.
Therefore, we instead propose a random normalization ( RN ) module to simulate different data distribution in the training phase:
[Equation]
where σ and µ are default standard deviation and mean vectors for ImageNet, and µ (cid:48) ∼
N ( µ (cid:48) mean , µ (cid:48) std ) and σ (cid:48) ∼ N ( σ (cid:48) mean , σ (cid:48) std ) are two random scales 2 sampled from Gaussian distribution.
Combined with RN , and the update rule can be expressed as:
[Equation]
3. D OMAIN - AGNOSTIC A TTENTION M ODULE
Unlike the random normalization module, the domain-agnostic attention module aims to narrow the domain gap from the model perspective.
Our inspiration is from prior works (Hansen & Salamon, 1990; Caruana et al., 2004; Dong et al., 2018), which demonstrate that the ensemble strategy can avoid getting trapped in the local optimum and improve performance.
Since there are many feature maps at layer L and each of them can model the input, i.e., extracts the features, we can also integrate them to produce a more robust feature representation A L , thus mitigating the impact of several biased feature maps, e.g. , the second feature map of Figure 3.
Speciﬁcally, we apply cross-channel average pooling to the feature maps at layer L :
[Equation]
where C is the channel number of f Ls ( x s ) .
As depicted in Figure 4, even if our source model is not trained in the target domain, the robust feature representation is still able to capture the essential feature of object very well.
Surprisingly, it is even similar to the one that derived from a completely different target model.
Therefore, this robust feature representation can serve as a domainagnostic attention ( DA ) to enhance the cross-domain transferability of adversarial examples.
Speciﬁcally, in the training phase, we leverage
A L to assign weights for each pixel of feature maps at the same layer.
The resulting object function can be written as:
[Equation]
where (cid:12) is Hadamard product.
After ﬁnishing the training, even if the inputs are not from the source domain, our generator G θ ∗ is supposed to own an ability to capture the essential feature to disrupt.
E XPERIMENTS
Source (white-box) Domain.
Our training data is the large-scale ImageNet (Russakovsky et al., 2015) training set which includes about 1.2 million 3 × 224 × 224 images.
Generators are trained against four ImageNet pre-trained models including VGG-16, VGG-19 (Simonyan & Zisserman, 2015), ResNet152 (Res-152)
(He et al., 2016) and DenseNet169 (Dense-169) (Huang et al., 2017).
In this domain, we also consider three other models, i.e. , DenseNet121 (Dense-121)
(Huang et al., 2017), ResNet50 (Res-50) (He et al., 2016) and Inception-v3 (Inc-v3 3 ) (Szegedy et al., 2016), to analyze the cross-model transferability.
All models are available in the Torchvision library 4 .
Target (black-box) Domain.
In addition to ImageNet (white-box domain), we also consider seven other black-box domains (shown in Table 1) including four coarse-grained (CIFAR-10, CIFAR100 (Krizhevsky, 2009), STL-10 (Coates et al., 2011) and SVHN (Netzer et al., 2011)) and three ﬁne-grained (CUB-200-2011 (Wah et al., 2011), Standford Cars (Krause et al., 2013) and FGVC
Aircraft (Maji et al., 2013)) classiﬁcation tasks.
For the ﬁne-grained classiﬁcation, we use DCL framework (Chen et al., 2019) with three different backbones: ResNet50 (Res-50)
(He et al., 2016),
SENet154 and SE-ResNet101 (SE-Res101)
(Hu et al., 2018).
The pre-trained models for the coarsegrained classiﬁcation are from Github 5 .
Implementation Details.
Our generator G θ adopts the same architecture as (Naseer et al., 2019), which is a composite of downsampling, residual (He et al., 2016) and upsampling blocks (more details can be found in Figure 12 of Appendix).
The output size of G θ is equal to the input size.
For
CDA and our methods, we use Adam optimizer (Kingma & Ba, 2015) with a learning rate of 2eand the exponential decay rate for ﬁrst and second moments is set to 0.5 and 0.999, respectively.
All generators are trained for one epoch with the batch size 16.
For the layer L , we attack the output of Maxpool.
3 for VGG-16 and VGG-19, the output of Conv 3 8 for Res-152 and the output of DenseBlock.
2 for Dense-169 (the ablation study can be found in Appendix A.2).
For brevity, we only refer the output of a speciﬁc layer/block using its layer/block name.
For our RN module, we set µ (cid:48) ∼ N (0 .
50 , 0 . 08) and σ (cid:48)
∼ N (0 . 75 , 0 . 08) , and the ablation study is shown in Appendix A.3.
Competitors.
We compare our proposed methods with projected gradient descent (PGD) (Madry et al., 2018), diverse inputs method (DIM) (Dong et al., 2018;
Xie et al., 2019), dispersion reduction (DR) (Lu et al., 2020), self-supervised perturbation (SSP) (Naseer et al., 2020) and cross-domain attack (CDA) (Naseer et al., 2019).
The maximum perturbation ε is set to 10.
Follow Lu et al.
(2020), we set the step size α = 4 and the number of iterations T = 100 for all iterative methods.
For DIM, we set the default decay factor µ = 1 .
and the transformation probability p = 0 .
The
Gaussian smoothing (gs) for CDA is applied by 3 × 3 Gaussian kernel.
Evaluation Metrics.
We use the top-1 accuracy on the whole test set (test size is shown in Table 1) after attacking to evaluate the performance of different methods.
We also report standard deviation across multiple random runs in Appendix A.8. examples are within the perturbation budget of (cid:96) ∞ ≤ 10 .
[Table] [Table]
4. T RANSFERABILITY C OMPARISONS
In this section, we ﬁrst conduct experiments for the black-box domain in Section 4.1.1 (coarsegrain) and Section 4.1.2 (ﬁne-grain), then we report the results for the white-box (source) domain in Section 4.1.3.
For the discussion of RN + DA variant, changing source domain and ensemblemodel attacks, we leave them in Appendix A.5, Appendix A.6 and Appendix A.7, respectively.
4.1. R ESULTS ON C OARSE - GRAINED D OMAIN
In this section, we craft transferable adversarial examples for coarse-grained classiﬁcation tasks.
The results are shown in Table 2, where we leverage four ImageNet pre-trained models, including
VGG-16, VGG-19, Res-152 and Dense-169, to train generators, respectively.
From Table 2, a ﬁrst glance shows that our proposed methods consistently surpass state-of-the-art approaches.
For example, if the substitute model is VGG-16, the most effective CDA remains a top-1 accuracy of 66.41% for CIFAR-10 after attacking, while our vanilla BIA can effectively bring down it to 57.38% and DA variant can further drop the top-1 accuracy to 55.16% .
Among all tasks, the STL-10 and SVHN domains are the most difﬁcult to attack, and the performance gap among existing attacks is moderate.
Nonetheless, we can signiﬁcantly enhance the transferability with the help of our RN variant in these domains.
Notably, if the substitute model is Res-152, RN variant can further decrease the top-1 accuracy from 89.46% (vanilla BIA) to 85.79% on SVHN.
Compared with state-of-the-art CDA on all domains, RN variant signiﬁcantly outperforms it by 7.71% on average.
This demonstrates that our proposed RN module is effective in coping with different distributions of inputs, thus improving the generalization of the resulting generator.
we report the top-1 accuracy after attacking (the lower, the better).
The generator G θ is trained in ImageNet domain and adversarial examples are within the perturbation budget of (cid:96) ∞ ≤ 10 .
[Table]
GPGD 80.
83.35 DIM 69.
65.26 DR 80.
84.57 SSP 62.
66.88 CDA 59.
62.26 CDA+gs 67.
BIA (Ours) 48.
51.79 BIA+ DA (Ours) 27.
44.01 BIA+ RN (Ours) 31.
R e sPGD 73.
75.64 DIM 56.
54.49 DR 77.
80.08 SSP 47.
69.34 CDA 45.
59.11 CDA+gs 59.
BIA (Ours) 43.
51.19 BIA+ DA (Ours) 25.
47.28 BIA+ RN (Ours) 23.
D e n s [Table] PGD 79.
81.24 DIM 63.
54.79 DR 74.
74.47 SSP 41.
44.07 CDA 52.
59.83 CDA+gs 60.
BIA (Ours) 21.
33.87 BIA+ DA (Ours) 12.
[Table] [Table]
4.1. R ESULTS ON F INE - GRAINED D OMAIN
We also analyze the transferability of adversarial examples towards ﬁne-grained classiﬁcation tasks.
For each domain, three black-box models with different backbones trained via the DCL framework are the target.
The results are summarized in Table 3, where the leftmost column is the substitute model and the top row shows the target model.
In this scenario, the performance gap between the existing state-of-the-art algorithms and our proposed methods is further enlarged.
Remarkably, when attacking against Dense-169, even the vanilla
BIA can drop the average top-1 accuracy to 29.23% , while CDA+gs, CDA, SSP, DR, DIM and
PGD are still with the high average top-1 accuracy of 68.75%, 61.91%, 48.65%, 80.84%, 68.06% and 84.06% after attacking, respectively.
Furthermore, by adding DA or RN modules in the training phase, the generator G θ ∗ is capable of crafting more transferable adversarial examples.
On average, our RN variant can reduce the top-1 accuracy from 46.52% (vanilla BIA) to 39.33% , and
DA variant can further drop it to 36.42% , which remarkably outperforms SSP by 25.91% .
This demonstrates our proposed DA module can effectively alleviate the bias caused by several feature maps, thus focusing on disrupting essential features.
4.1. R ESULTS ON S OURCE D OMAIN
Although our proposed methods are mainly designed to improve the threat of adversarial examples towards black-box domains, they are also effective for enhancing the cross-model black-box transferability in the white-box domain.
For example, by training against Dense-169, CDA still remains a top-1 accuracy of 43.78% on Inc-v3, while our BIA can achieve a relatively low top-1 accuracy of 38.58% on it.
Besides, RN variant can further decrease the top-1 accuracy on Dense-169 (whitebox model) and Inc-v3 (black-box model) by 2.4% and 24.34% , respectively.
This demonstrates that our proposed RN module is also able to avoid getting stuck in the local optimum of a speciﬁc model when training on a large-scale dataset.
For DA variant, since the target domain and source domain are identical, it is naturally able to improve the transferability as well.
As shown in Table 4, compared with vanilla BIA, DA variant can further degrade the top-1 accuracy from 18.45% (vanilla
BIA) to 13.47% on average.
4. I NSIGHT INTO THE G ENERATOR
Although many prior works (Poursaeed et al., 2018; Naseer et al., 2019) have leveraged the generator to craft adversarial examples, they hardly analyze the role of each block of the generator.
This section will give an insight into the generator and understand how it processes an input.
In Figure 5, we feed an image into the generator trained with the vanilla BIA (against Res-152) and visualize the
[Figure]
output examples:
• Downsampling block: it mainly extracts discriminative features of the input image.
As the size of the down-sampled images gets smaller, there is no signiﬁcant noise overall.
• Residual block: Unlike the downsampling block, this block is responsible for adding noise and its behavior looks very similar to the iterative algorithm.
• Upsampling block: This block gradually reconstructs the adversarial example from abstract features.
[Figure]
Since the residual block mainly works as suppressing or reversing the features extracted by the downsampling module, the downsampling module has an essential impact on generating transferable adversarial examples.
To better understand the effectiveness of our proposed modules, here we investigate them from the perspective of the generator mechanism.
Speciﬁcally, we apply crosschannel average pooling to the output of the downsampling block and calculate the difference map between vanilla BIA and our proposed variants (The formula deﬁnitions is shown in Appendix A.1).
From the result of Figure 6, we can observe that the generators derived from our proposed variants concentrate more on the body of the object (especially for DA variant) than that of vanilla BIA (which pays more attention to the background, i.e. , the black region in the difference map).
This demonstrates that our proposed RN and DA variants do narrow the domain gap, and thus be capable of yielding more transferable adversarial examples.
C ONCLUSION
In this paper, we present a practical black-box threat model for the cross-domain attack.
Speciﬁcally, we train a generator network in the large-scale ImageNet domain to disrupt low-level features better, thus generating transferable adversarial examples for the black-box domain.
Based on this framework, we further propose two variants to narrow the gap between the source and target domains from the data and model perspectives, respectively.
Extensive experiments demonstrate the effectiveness of our proposed methods.
This also reminds the model owner that “Your deployed model is not safe even you do not leak any information to the public”.
We hope our proposed approaches can serve as a benchmark for evaluating the stability of various deployed models.
A A PPENDIX A. T HE D EFINITION FOR D IFFERENCE M AP
This section will describe the formula deﬁnition for difference map Diff ( · , · ) .
Without loss of generality, here we only show the deﬁnition of Diff ( RN variant , BIA), and Diff ( DA variant ,
BIA) can be easily deduced.
Speciﬁcally, we ﬁrst apply cross-channel average pooling to the output of the downsampling block:
[Equation]
where G d RN θ ∗ and G d BIA θ ∗ denote the output of the downsampling block for RN variant and vanilla
BIA, respectively.
Then our difference map can be expressed by:
[Equation]
A. S ELECT L AYER FOR A TTACKING
In this section, we analyze the impact of different intermediate layers of the substitute model on the transferability of resulting adversarial examples.
The results are illustrated in Figure 8.
In general, training against the shallow and middle layers yields more cross-domain transferable but less cross-model transferable adversarial examples than against the deep layer.
For example, if the substitute model is Res-152, disrupting shallow layer Conv 2 3 is more effective for transferring towards coarse-grained domain, and perturbing middle layer Conv 3 8 is more effective in reducing the average
top-1 accuracy of ﬁne-grained models.
In contrast, attacking deep layers like Conv 5can yield more transferable adversarial examples in the source domain.
This demonstrates that lowlevel features are more similar across domains and high-level features are more domain-speciﬁc.
A. S ELECT G AUSSIAN D ISTRIBUTION FOR R ANDOM N ORMALIZATION
[Figure]
DA module, is parameter-free
Therefore, we only conduct the experiment to select an optimal distribution for RN module, i.e. , the µ (cid:48) and σ (cid:48) in Equation 4.
Here we tune the mean of µ (cid:48) and σ (cid:48) from 0.25 to 0.75 with a granularity of 0.25.
For the standard deviation of them, we ﬁx it to 0.08 so that the sampled µ (cid:48) and σ (cid:48) can basically take values ranging from 0.0 to 1.0 (according to the three-sigma rule).
The results are shown in Figure 7, where we craft adversarial examples via VGG-16 and report the average top-1 accuracy in both source and black-box domains.
From the results, we observe that a bigger σ (cid:48) mean can effectively improve the transferability.
For example, if we increase σ (cid:48) mean from 0.25 to 0.75, the top-1 accuracy on ﬁne-grained models can be further decreased by 3.16% on average.
Although the inﬂuence of µ (cid:48) mean is relatively moderate when σ (cid:48) mean
= 0 .
75 , setting µ (cid:48) mean to 0.5 is usually better.
Therefore, we set µ (cid:48) ∼ N (0 .
50 , 0 . 08) and σ (cid:48)
∼ N (0 . 75 , 0 . 08) in our paper.
A. D ATA A UGMENTATION VS . R ANDOM N ORMALIZATION
Data augmentation (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015) is a widely used strategy for improving the generalization of the model.
Nonetheless, these label-preserving transformations are less effective for training a generator to craft more transferable adversarial examples.
[Figure]
[Figure]
pre-trained VGG-16, VGG-19, Resand Dense-169, respectively.
As demonstrated in Figure 9, where we report the results for our vanilla BIA, BIA with data augmentation (BIA+AUG) 6 and BIA with RN (BIA+ RN ), BIA+AUG is less effective than our proposed
BIA+ RN and might even degrade the performance of our vanilla BIA.
For example, when training against VGG-19, BIA gets an average top-1 accuracy of 58.71% on the ﬁne-grained domain, yet
BIA+AUG degrades it to 62.29%.
In contrast, our proposed BIA+ RN can signiﬁcantly decrease the result to 47.78% .
This is mainly because that the common data augmentation cannot effectively change the distribution of the training dataset, thus decreasing the generalization towards the black-box domain.
A. T HE E FFECT OF C OMBINING D OMAIN - AGNOSTIC A TTENTION AND R ANDOM N ORMALIZATION
In this section, we report the results for BIA equipped with both RN and DA .
The following is the update rule:
[Equation]
[Equation] As illustrated in Figure 10, RN module and DA module are not always mutually reinforcing.
For example, when training against Res-152 and transferring adversarial examples to ﬁne-grained or source domains, combining RN module and DA module can further decrease the average topaccuracy to 35.75% and 13.11%, respectively.
However, if the black-box domain is coarse-grain, only applying RN module is better than applying both RN and DA modules.
We speculate that it may be because the RN module affects the low-level features extract by the substitute model, and thus be incompatible with the DA module sometimes.
Since Figure 10 shows that using DA and RN in tandem is less effective when training against Dense-169 while they reinforce each other in VGG-16 in most cases, we visualize the cross-channel average pooling of intermediate features for VGG-16 [-Coarse-grain
-]and Dense-169 to better explain this phenomenon. As illustrated in Figure 11, it can be observed that the RN module reinforces the dis-
[Figure] [Figure]
[-VGG-19 Coarse-grain 
-]criminative features in VGG-16.
[-Res-152 Coarse-grain 
-]However, it inhibits the response of objects’ essential features extracted by Dense-169. Consequently, DA module may cause the resulting generator to reduce the ability to attack essential features, thereby making it challenging to use these two techniques in tandem. A. D ISCUSSION ON C HANGING S OURCE D OMAIN
[-Dense-169 Coarse-grain 
-]Since all the experiments are only regarding from ImageNet domains to target domains, it is unclear how well the method will perform if the source dataset is different (especially when the source dataset is small). Therefore, in the following Table 5, we report the results for transferability from CUB-200-2011 to other domains.
Generators are learned against CUB-200-2011 domain (the substitute model is DCL (backbone: Res-50) and training data is CUB-200-2011 testing data ( images)). For results of ﬁne-grained domains, we average top-1 accuracy of backbone SENetand SE-Res101. For the result of the ImageNet domain, we average top-1 accuracy of all models introduced in our manuscript. We can observe that our method consistently outperforms our main competitor CDA by a large margin.
Besides, we also notice that transferring from CUB-200to CIFAR is very challenging (compared with our reported results in Table 2).
Therefore, we highlight the necessity of using a large-scale dataset such as ImageNet to train the adversarial examples generator.
[Table] D ISCUSSION ON E NSEMBLE - MODEL A TTACKS As demonstrated in prior works (Liu et al., 2017; Dong et al., 2018; Mopuri et al., 2018), attacking against an ensemble of models can yield more transferable adversarial examples. However, it is not clear whether the ensemble-model attack is also effective in our cross-domain attack scenario.
To investigate this, we conduct an experiment in Table A.7, which shows the results for training against
VGG-16 and an ensemble of VGG-16, Vgg-19, Res-152 and Dense-169, respectively.
From the result, We can observe that ensemble-based training can also improve the transferability of adversarial examples towards ﬁne-grained domains signiﬁcantly.
Table
[Table]
A. D ISCUSSION ON S TANDARD D EVIATION ACROSS M ULTIPLE R ANDOM R UNS
To ensure the stability and credibility of the evaluations [Table] experiments are repeated several times on BIA.
[-Res-152 Fine-grain 010 1525 
-]In Table [Table] we report the results for each random seed and standard deviation across these random runs. [Table]
[-Dense-169 Fine-grain
-]D r o p o u t
C o n v
R e L U C o n v B a t c h
N o r m B a t c h N o r m ⨁
[-Res-152 Source 04 610 
-]C o n v R e L U B a t c h
[-Dense-169 Source 
-]N o r m T r a n s p o s e C o n v
B a t c h N o r m R e L U C o n v
[Figure] [Figure]