Towards Lightweight Black-Box Attacks against Deep Neural Networks
Abstract
Black-box attacks can generate adversarial examples without accessing the parameters of target model, largely exacerbating the threats of deployed deep neural networks (DNNs).
However, previous works state that black-box attacks fail to mislead target models when their training data and outputs are inaccessible.
In this work, we argue that black-box attacks can pose practical attacks in this extremely restrictive scenario where only several test samples are available.
Specifically, we find that attacking the shallow layers of DNNs trained on a few test samples can generate powerful adversarial examples.
As only a few samples are required, we refer to these attacks as lightweight black-box attacks.
The main challenge to promoting lightweight attacks is to mitigate the adverse impact caused by the approximation error of shallow layers.
As it is hard to mitigate the approximation error with few available samples, we propose Error TransFormer (ETF) for lightweight attacks.
Namely, ETF transforms the approximation error in the parameter space into a perturbation in the feature space and alleviates the error by disturbing features.
In experiments, lightweight black-box attacks with the proposed ETF achieve surprising results.
For example, even if only 1 sample per category available, the attack success rate in lightweight black-box attacks is only about 3% lower than that of the black-box attacks with complete training data.
Introduction
Black-box attacks [11, 57, 61] can mount successful attacks without accessing the parameters of deep neural networks (DNNs), posing great challenges to deep learning in safety-critical situations.
Existing black-box attack methods assume that the training data and/or the outputs of target models are available to adversaries.
For scenarios where training data are available, adversarial examples can be generated by attacking surrogate models constructed for approximate target models.
Using the target models’ outputs for estimating gradients is another practical approach to crafting adversarial examples.
In general, due to the low dependency on target model information, the black-box attacks raise realistic threats to real-world deep learning systems, including semantic segmentation [2, 36], object detection [17, 46, 45], and automatic driving [55].
However, these assumptions can be violated in many practical scenarios [3, 47], where both the training data and the outputs of target models are inaccessible.
This realistic attack scenario is introduced in [3], generally termed as the no-box setting .
Under such a restrictive setting, Li et al.
[34] states that existing black-box attacks can fail to mislead target models, as the information previously considered essential in mounting black-box attacks is now inaccessible.
Therein, the failure of existing black-box attacks is not surprising.
The reason is that constructing reliable surrogate models typically relies on target models’ training data, which is inaccessible in such a restrictive setting with only a few test samples available.
In this work, we aim to reveal the potential threats of black-box attacks in the no-box setting, as mounting black-box attacks with highly limited accessible information leaves many models exposed to attack.
Specifically, we challenge the previous believes by raising the following question: Can black-box attacks success in the no-box setting?
The doubts about the potential threats of the blackbox attack are not groundless.
Specifically, the potential threats of black-box attacks stem from two facts: a) adversarial examples can be generated by perturbing representations at shallow layers of DNNs [27, 26, 49]; b) regarding the representation of shallow layers, there do not exist critical differences between those models learned from a few data and that of the whole training data.
Building upon the above facts, it is actually possible that an adversary can successfully mount attacks using limited samples, as powerful adversarial examples can be generated by attacking the shallow layers of DNNs trained on few samples.
Concretely, the adversary can leverage available samples to construct a surrogate model to approximate the shallow layers of target models within acceptable errors.
Consequently, adversarial examples can be generated by perturbing the features obtained from the shallow layers of the constructed surrogate model.
As merely a few samples are required for attacking, we refer to black-box attacks in the no-box setting as lightweight black-box attacks .
Intuitively, the closer the surrogate model is to the target model, the higher the attack success rate the crafted adversarial examples have [30].
If the approximation error of shallow layers is alleviated, the lightweight black-box attack can be as powerful as black-box attacks with complete training data.
Hence, the main challenge to mounting a powerful lightweight attack is to mitigate the adverse impact caused by the approximation error of shallow layers.
However, it is challenging to mitigate the approximation error, especially when the number of available samples is limited.
Fortunately, it is straightforward to identify which kind of perturbations are preferred: if perturbations applied to a feature contribute to fooling the surrogate model, the perturbation is preferred.
Therefore, bridging the connection between the parameter space and the feature space is the key to mitigating the approximation error.
Accordingly, we propose transforming the approximation error in the parameter space as the perturbation in the feature space, dubbed as Error TransFormer (ETF).
Namely, ETF transforms the worst-case approximation error to the worst-case feature perturbation, leading to a min-max scheme that generates adversarial examples under the worst feature perturbations.
We verify the attack success rate of lightweight black-box attacks using 7 models trained on the ImageNet dataset [48], and find that existing attack methods are much powerful than previous claimed.
Moreover, the performance of lightweight black-box attacks can be further promoted by the proposed ETF, i.e., the attack success rate is only 3 lower than that achieved by black-box attacks having complete target model’s training data.
Related works 2.1 Adversarial attack
According to the amount of accessible information, existing adversarial attacks can be roughly divided into two categories: white-box attacks and black-box attacks.
White-box attacks [18, 42, 14, 15] assume that the target model is transparent to adversaries, i.e., adversaries can access all information about the target model.
Nevertheless, the assumption that all information is available to adversaries can be violated in many practical scenarios [10].
Hence, black-box attacks [13, 37, 65, 64, 6] are proposed and applied to the scenario where relatively limited information about the target model is accessible, i.e., only a certain number of model queries or the training data of target tasks are available.
Utilizing the training data or queries, adversaries generate adversarial examples on the surrogate model constructed for approximate target models.
Among these black-box attacks, intermediate-level attacks [49, 27, 23] are widely explored to improve the adversarial transferability.
The core idea of these attacks is based on the empirical observation that well-trained models’ intermediate features are transferable [59].
Hence, Inkawhich et al.
[27] propose to perturb the feature space of neural networks to create more transferable adversarial examples.
Besides, these attacks provides an interesting empirical conclusion that disturbing the shallow layers of DNNs can also generate adversarial examples.
Existing black-box attacks assume that the large-scale training data of target models and the feedback from querying target models are accessible, but these assumptions can be violated in many scenarios, such as in the no-box setting [3].
Li et al.
[34] state that existing black-box attacks cannot be successfully mounted because existing DNNs require large-scale training data for generalization.
Given only small-size data available, obtaining a surrogate model with strong generalization is challenging.
Hence, Li et al.
propose replacing black-box attacks with their proposed no-box attack [34], where they train a classical auto-encoder model instead of the supervised classification model due to the constraint of a small-scale dataset.
Concretely, they train 20 auto-encoders for each category and generate adversarial examples by attacking these auto-encoders, which is time- and computational-consuming.
Different from existing black-box attacks, lightweight black-box attacks aim to reveal the potential risk of black-box attacks under the no-box setting.
Because one main challenge to perform lightweight black-box attack is to mitigate the adverse impact caused by the approximation error, we propose error transformer that is a min-max strategy transforming the approximation error in the weight space to the feature space.
The min-max strategy is different from that introduced in [56], where min-max strategy is proposed to flatten the loss landscape in the weight space, see Appendix F.
2.2 Approximation of Shallow Networks
A recent study [59] confirms the intuition that shallow layers in DNNs can be seen as low-level feature extractors, provided that strong data augmentation is used [1].
Motivated by this observation,
Asano et al.
[1] design a method to explore the information in every layer of DNNs.
Asano et al.
[1] train a supervision model on ImageNet LSVRC-12, and a self-supervision model on a small-scale dataset.
Then, they apply linear probes [62] to all intermediate layers of networks, where a linear classifier is trained on the top of pre-trained and fixed feature representations.
In this way, they evaluate the quality of the representation learned at different depths of the networks.
The results show that, given heavily synthetic transformations, the shallow layers of DNNs learned from a few images can approximate that of DNNs trained on millions of images.
According to the intriguing empirical observation, it is possible to construct a surrogate model with limited samples, where its shallow layers are similar to that of the target models.
Preliminaries 3.1 No-Box Threat Model
The no-box setting [3] denotes a threat model where available information about the target model is extremely limited, making it challenging to generate adversarial examples.
According to the description in [3], the no-box threat model can be defined as follows.
Definition 1 (No-Box Threat Model)
A threat model is called the no-box threat model if the adversary in this scenario is not allowed to access the target model’s training data and outputs, so it only has some samples that can be correctly predicted with high probability by the target model.
The definition is consistent with the no-box setting introduced in [3], where the target model’s training data and outputs are inaccessible to the adversary.
Moreover, the available samples used for generating adversarial examples are expected to be limited.
This is because if the number of samples is similar to or even larger than the number of samples used for training target models, the adversary can leverage these samples to construct a surrogate model similar to target models, making the condition of inaccessible training data meaningless.
It is intuitive that target models can correctly predict the labels of these available samples with high probability.
For example, it is meaningless to perform attacks using samples misclassified by the target model.
3.2 Lightweight Black-Box Attack
Definition 2 (Lightweight Black-Box Attack)
An attack is called the lightweight black-box attack if an adversary aims to perform an attack in the context of a no-box threat model.
The definition of the lightweight black-box attack shows that lightweight attack is a special kind of black-box attacks.
The extremely limited accessible information distinguishes it from other black- box attacks.
For example, query-efficient black-box attacks require the feedback information of target models [24, 25], while lightweight attacks do not.
To our best knowledge, how to perform a lightweight black-box attack is still lacking in the literature.
For example, Li et al.
[34] claim that black-box attacks fail to fool target models when their training data and outputs are inaccessible.
Approach
This section gives a detailed description of how to perform lightweight black-box attacks and the proposed Error TransFormer (ETF) to alleviate to adverse impact caused by approximation error.
4.1 Lightweight Surrogate Model
Surrogate models are widely used in black-box attacks, as adversarial examples are usually generated by attack surrogate models when querying target models is forbidden.
Since the lightweight black-box attack is a special kind of black-box attacks, performing attack requires constructing surrogate models.
We call surrogate models employed in lightweight black-box attacks lightweight surrogate models, considering that the number of samples are usually limited.
Similar to existing black-box attacks, we assume that the label information of samples used for generating adversarial examples are accessible.
In this scenario, the lightweight surrogate model is trained for a classification task.
Specifically, the lightweight model is realized by training a classification DNN equipped via conventional supervised learning, using cross-entropy loss ℓ ( · , · ) :
[Equation]
where w stands for the parameters of surrogate models f , ( x, y ) sampled from the dataset ˆ D denotes the pair of natural sample and its label y .
In black-box attacks, adversary only leverages the shallow layers of DNNs, leading to a straightforward approach to constructing lightweight surrogate models.
Specifically, we can train the lightweight surrogate model in a contrastive manner, where the supervised information is no longer necessary:
[Equation]
where σ is a specific similarity metric, e.g., cosine similarity, T denotes the data transformation operation, and ˆ D \ x means that x is removed from the dataset ˆ D .
The contrastive strategy in Eq.makes it possible to perform attacks when the label information is unavailable.
Unless otherwise specified, we mainly uses Eq.
1 to train the surrogate models.
4.2 Feature Space Perturbation
Built upon the empirical observation [59], using limited samples can approximate the shallow layers of DNNs within acceptable errors.
This suggests that deep layers other than shallow layers can cause large approximation error.
Accordingly, merely the shallow layers of the learned lightweight surrogate models are used for generating adversarial examples, which is similar to the feature space attacks [49].
For brevity, we define the shallow layers as φ and denote the feature as φ ( x ; w ) .
In feature space attacks, guide images are usually required for generating adversarial examples.
A natural image used for generating adversarial example is usually called source image.
Given a source image x s , we generate an adversarial example x adv by perturbing x s such that its feature is similar to that of a guide image x g , where the labels of these two images are different.
Specifically, the adversarial examples are generated as follows:
[Equation]
where d stands for a specific distance, ϵ controls the strength of adversarial perturbation, and x ′ denotes the perturbed version of source image.
4.3 Error Transformer
Although applying feature space attacks to lightweight surrogate models can generate transferable adversarial examples, the approximation error will cause adverse impact to mounting attacks.
The reason is that large dissimilarity between the surrogate models and the target models reduces the attack success rate.
Therefore, the main challenge to mounting a powerful lightweight attack is to mitigate the adverse impact caused by the approximation error.
However, available samples are usually limited in the no-box threat model, making it hard to alleviate the approximation error.
To address the challenge, we propose transforming the parameter space’s approximation error as the feature space’s perturbation.
Specifically, we have little knowledge to determine which perturbations can point (from the surrogate model) to the target model, making it challenging to alleviate the approximation error in the weight space.
In contrast, we have the prior that samples with different labels should have distinguishable representations.
Thus, we can leverage the prior knowledge to select preferred perturbations in the feature space, i.e., we prefer perturbations that can make representations of samples with different labels indistinguishable.
Similarly, it is also straightforward to define the “bad” perturbations, leading to the design of the min-max optimization to identify the “worst” model.
Thus, connecting the parameter space and feature space makes it possible to mitigate the adverse impact caused by approximation error.
To bridge the connection, we introduce the following key identity, which is also used in [44]:
[Equation]
where w 1 stands for the first layer parameters of model φ , A is a transformation matrix used for perturbing w 1 , (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) means that the first layer’s parameters are perturbed and the other layers’ parameters keep the same, and g is the function parameterized with w \ w 1 used for processing the first layer’s outputs.
According to Eq. 4, we can transform a perturbation in the parameter space as the perturbation in the data space, i.e., φ ( x ; (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) )
=
φ ( x + Ax ; w ) .
Similarly, we can apply the transformation operation to the hidden layers, see
Appendix A for more details.
Built upon the connection in Eq.
4, we can transform the first layer parameters’ discrepancy between the target model and the surrogate model.
Taking the first layer as an example, let w t be the parameters of the target model and w 1 t be its first layer parameters, where the other parameters are the same as the surrogate model.
Assume 2 that there exists a transformation matrix A applied to the first layer parameters of the surrogate model such that w 1 t = w 1 + w 1 A , we have:
[Equation]
The identity in Eq.
5 shows that we can find a perturbation in the data space to alleviate the difference in the first layer parameters between these two models.
Built upon the transformation, we can transform the worst approximation error to the worst perturbation in the data space.
Therefore, we can employ a min-max strategy to mitigate the adverse impact of approximation error through generating adversarial examples under the worst perturbation.
Thus, lightweight black-box attacks with ETF generate adversarial examples as follows:
[Equation]
where ∥·∥ p is the ℓ p -norm, ϵ and τ control the strength of perturbations, ∆ g and ∆ s are the data space perturbations for mitigating the approximation error in parameter space, and x ′ denotes the perturbed version of source image.
In this way, we reduce the error in parameter space to approximate the target model.
We solve the inner maximization problem by generating perturbations in the feature space, given a perturbed adversarial example.
This step aims to mitigate the approximation error.
The outer minimization problem is solved by finding adversarial perturbations in the input space, the same as the adversarial example generation.
After iterative generation of perturbations, adversarial examples are generated by attacking a model with a reduced approximation error.
Experiments
In this section, we conduct extensive experiments to verify the power of our ETF.
[Table]
5.1 Experimental Setup
Models Architectures.
All surrogate models are based on the ResNet- 18 [20].
In the ordinary settings of the black-box attack, the general surrogate models are trained through the whole training set of ImageNet [48].
On the contrary, the lightweight surrogate models adopt only 1 , 000 images sampled from the validation set of ImageNet, considering that images in the training set are probably inaccessible in practice [34].
Results evaluated on the CIFAR10 dataset can be found in Appendix D.
Please refer to Table 5 in the Appendix B for the detailed information on the lightweight surrogate model and the general surrogate model.
Regarding the target models, various model architectures are selected for full comparison, including, VGG- 19 [52], Inception v3 [54], ResNet- 152 [20],
DenseNet [22], SENet [21], wide ResNet (WRN) [60] and MobileNet v2 [50].
All these target models are well-trained on ImageNet.
Implement Details.
The lightweight surrogate models are randomly initialized.
The batch size is 128 , the epoch is 500 , and the initial learning rate is 0 .
4 , linearly decreasing to 0 .
To better approximate the low-level feature of shallow layers, we refer to the self-supervision work [5] and apply various random augmentations to the data in each epoch for training.
To mount attacks, the classic methods, e.g., PGD [41], MI [10], DI [57], and TI [11], are applied to all the surrogate models.
Unless otherwise specified, our attack is mounted on the first layer of ResNet-18 for lightweight surrogate models.
Following previous works on intermediate-level attacks, the metric d in Eq. 3 of
Shallow- in Table 1 is instantiated as the mean square error.
Meanwhile, to maximize the potential of ETFs, we apply contrastive loss to ETF- in Table 1.
The Top-1 prediction accuracy is adopted as the evaluation metric, i.e., lower the classification accuracy means better attack success rate.
All the experiments are run for 5 individual trials with different random seeds.
In our experiments, target models are evaluated using ℓ ∞ -norm adversarial examples with maximum distortion ϵ = 0 .
1 , results evaluated under mores settings, e.g., more strict constraints ϵ = 0 .
05 , ℓ 2 -norm adversarial examples, and more test images 5,000 images, can be found in Appendix B.4.
Code and instructions: https://github.com/sunch-ustc/Error_TransFormer/tree/ETF .
5.2 Main Results
Results on normally trained models.
The results of classification accuracy are shown in Table 1, which exhibits the evaluations for both lightweight and general surrogate models.
The last four rows [Table] show the ordinary results where the general surrogate models are trained through all data in the training set.
The black-box attacks are mounted based on the whole model, termed ”Deep/Deep ∗ ”, with average accuracies of around 10 .
As the size of the training data is reduced to 1 , 000 , this conventional method obviously loses its aggressiveness.
Specifically, the averaged accuracies of 3-6 rows are among 40 - 50 , revealing limited attacking capability compared to 66 .
99 of the clean accuracy.
It demonstrates the necessity of large amount of data for classic black-box attacks.
In rows 7-10, shallow layers without ETF are used to generate adversarial examples.
The averaged accuracies of around 25 illustrates the less dependence of shallow networks on the amount of data.
In rows 11-14, attacks on shallow layers are further enhanced by ETF, decreasing the averaged accuracies to around 17 .
The narrow gap between the proposed ETF and the general surrogate model indicates the potential capability of the black-box attack in the no-box settings.
[Table]
Here, we consider the models that are trained through adversarial training [41], which is more persuasive to evaluate the effectiveness of attacking methods.
Following [41], the target models are trained on the whole training set of ImageNet.
During adversarial training, the adversarial examples are generated under ϵ = 0 , 4 / 255 , 8 / 255 , where ϵ means the constraint of ℓ ∞ distance of the adversarial examples used for adversarial training relative to the original sample.
Table 2 exhibits the attacking results mounted by ETF, black-box attacks (general surrogate models with all training data avaliable), and no-box attacks.
For simplicity, only the results of PGD attacks are reported.
As analysed above, the black-box attacks shows the best attacking results, while ETF only has a narrow gap with it.
However, when ϵ = 4 / 255 or 8 / 255 , ETF exceeds the black-box attacks (29.13/26.14 vs. 48.11/38.24).
It is interesting that black-box attacks are even worse than lightweight attacks.
We believe that exploiting the phenomenon can bring something new, but we leave it as our future work due to the limited space.
5.3 Further Analysis and Ablation Study
Intermediate-Level Perturbations.
Since the lightweight black-box attacks are implemented mainly on intermediate features, two classic attacks mounted on intermediate features, i.e. ILA [23] and AA [27], are selected for experiments.
Comparing the ILA and AA in Table 3 and ShallowMI/DI/TI/PGD in Table 1, there is little difference between their average accuracies.
It indicates those intermediate-level feature perturbation has little benefit to lightweight black-box attacks.
The reason is that the intermediate-level attacks highly rely on the low-level features extracted by the lightweight model to generate adversarial examples and it can weaken the role of high-level information when attacking.
The proposed ETF achieves better performance.
On one hand, ETF does not rely on the
A v er ag
e a
[Figure]
a
[Figure]
y
Number [Figure]
Nobox [Figure]
maxpooling [Figure] layer [Figure]
FC
[Figure]
VGG
Inception
[Figure]
Comparison with No-box Attacks [34]. No-box attack is the first work to explore how to mount attacks in the no-box setting. Since black-box attacks do not work in this setting, they do not attempt to design a surrogate model that approximates the target model as we do. Instead, they design the pretext task to train 20 auto-encoders per category to generate adversarial examples.
However, the difference between the pretext tasks and the target tasks may significantly reduce the transferability of adversarial examples.
In contrast, our method trains the lightweight model for tasks similar to or same as the target tasks.
Hence, though only one sample per category is available, we achieve better performance than no-box attacks, as shown in Table 3.
Number of Training Samples.
One key difference between the lightweight and general black-box attacks mainly lies in the number of samples n used for training models.
Therefore, we further explore how lightweight attack performance varies with different values of n .
We conduct experiments on randomly selected n images from the validation set of ImageNet.
The experimental results are shown in Figure 1(a).
The average accuracy decreases as more samples are available, and better attacking results can be achieved thereafter.
More data provides the opportunity for lightweight black-box attack to approximate the shallow layers of the target model more accurately.
Especially the lightweight black-box attacks with 10 , 000 samples can achieve the same performance as general black-box attacks which train the surrogate model on 1,200,000 images.
Also, our method can still mount attacks even when only 2 or 10 images are available.
Surrogate Model.
As analyzed above, we utilize the first few layers of the lightweight model to approximate the shallow layers of the target model to mount attacks.
Hence, it is interesting to study how the layers selection for lightweight models impacts the attack performance.
We adopt different layers of ResNet- 18 to generate adversarial examples utilizing ETF and attack the different target models.
The results are summarized in Fig. 1(b).
It can be seen that low-level information in the first block of the model is sufficient for achieving effective attack results.
With more block information on the model, the performance of the attack does not improve.
In particular, the transferability of adversarial examples significantly decreases when we attack the fully-connected layer, which verifies our analysis that the high-level layer of the lightweight model cannot approximate the target model well.
More results for architecture selection can be found in Appendix B.4.
Adversarial Example Visualization.
To verify that adversarial examples are truly imperceptible, we provide visualization in Figure 2, where Deep*-PGD attack (using training images), Deep-PGD attack (using test images), and lightweight black-box attack are considered.
Discussion
Though the no-box threat model provides a weak assumption, many realistic scenarios may be more complex.
For example, adversaries have access to only data without supervision information due to security [35] or privacy [47] issues.
In more extreme cases, e.g., autonomous vehicle [55], adversaries even have no access to the test samples from the distribution over which the model is trained.
Instead, the adversaries have access to samples from a similar (but not exact) distribution (because the attacker [Table] [Figure] knows the deployment environment).
Hence, we consider two more challenging attack scenarios in which we provide solutions to mount lightweight attacks.
Insufficient supervision information.
We design an experiment for mounting attacks using 1 ,unlabeled samples randomly selected from the ImageNet validation set to explore the power of
ETF under this challenging setting with unlabeled data.
Thanks to the framework of lightweight attack, it is feasible to train the unsupervised model on unlabeled data for attacks using Eq. 2.
Specifically, we adopt contrastive learning [5] to train a lightweight surrogate model on the smallscale unlabeled dataset and generate adversarial examples by attacking the trained lightweight surrogate model.
The performance is shown in Table 4, where ETF still achieves similar performance without supervision information as that of the results under supervision.
Intuitively, we can also employ other self-supervised learning method, e.g., rotation prediction, to train shallow layers, see details of experimental settings and results in Appendix 16.
Consideration of Out-of-distribution.
We further study the possibility for mounting attacks under a more strict scenario, where only OOD (out-of-distribution, OOD) data are available.
Specifically, we utilize the model pre-trained on OOD samples to mount lightweight black-box attacks.
The rationality lies in that only low-level information is required to mount lightweight black-box attacks and the acquisition of low-level information.
We load pre-trained ResNet- 50 on STL- 10 [38] dataset as the surrogate model.
Without further fine-tuning, we experiment with two test in-distribution samples, and the results are recorded in Table 4, named OOD.
As we can see, attacks mounted in the model trained on OOD achieve a surprising attack success rate, showing that black-box attacks enhanced with ETF can pose practical threats to deployed models.
Conclusion
We propose a lightweight black-box attack, aiming to reveal the potential risk of black-box attacks under the no-box threat model.
The proposal is based on the empirical observations that the model trained with limited samples can approximate the shallow layers of the target models within acceptable errors, which can be used for crafting adversarial examples.
To enhance the performance, we propose error transformer to decrease the approximation error by transforming the approximation error in the parameter space into the feature space.
The experiments show that the proposed ETF achieves a surprising attack success rate using only one image per category, i.e., only 3 lower than black-box attacks with complete training data of target model.
Acknowledgements
This work was supported by NSFC No. 61872329 and No. 62222117, and the Fundamental Research
Funds for the Central Universities under contract WK3490000005.
YZ and BH was supported by
NSFC Young Scientists Fund No. 62006202 and Guangdong Basic and Applied
Basic Research
Foundation No. 2022A1515011652.
TL was partially supported by Australian Research Council
Projects DP180103424, DE-190101473, IC-190100031, DP- and FT-220100318.
A ETF on Hidden Layers
Before giving the details of applying ETF to hidden layers, we revisit the key identity:
[Equation]
where w 1 stands for the first layer (convolution layer) parameters of model φ , A is a transformation matrix used for perturbing w 1 , (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) means that the first layer’s parameters are perturbed and the other layers’ parameters keep the same, and g is the function parameterized with w \ w 1 used for processing the first layer’s outputs.
Built upon Eq.
7, we have φ ( x ; (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \
[Equation]
[Equation]
=
[Equation]
( x + Ax [Equation] w ) . The model φ can be decomposed: φ = g l ◦ φ l , where φ l ( x ) denotes the hidden feature at layer l .
Then the output feature at layer L is expressed as φ L ( x ; w ) =
g l ( w l φ l ( x )) , where w l is the parameter used for processing the feature φ l ( x ) .
Then, we can apply the identity at layer l :
[Equation]
where A l is the transformation matrix at layer l .
According to Eq.
8, we can transform a perturbation in the parameter space ( Aw l ) as the perturbation in the feature space ( Aφ l ( x ) ).
Thus, lightweight black-box attacks with ETF generate adversarial examples as follows:
[Equation]
where ∆ ls ( ∆ lg ) denotes the feature space perturbation ( l = 0 means the data space), ∪ l ∆ ls ( ∪ l ∆ lg ) stands for all perturbations in the feature space, and φ ( x g , ∪ l ∆ lg ; w ) ( φ ( x g , ∪ l ∆ lg ; w ) ) stands for the output feature with perturbed features, where the feature of source (guide) image at layer l is perturbed by ∆ ls ( ∆ lg ).
Although we can perform the error transformation in the feature space, the features obtained using weights with approximation errors make it challenging.
Specifically, we merely know that the input distribution (can be seen as the feature map) is not biased, but the feature map obtained using any weights will cause bias.
Thus, we merely apply ETF to the input layer and leave further exploitation of applying it to hidden layers as future work.
B Additional Detail Description B.1 Simplified Architecture
The architecture of surrogate models is modified to avoid overfitting.
Considering the limited amount of data, we employ a network with a small model capacity to instantiate the feature extractor.
In particular, ResNet- 18 is simplified by reducing the number of blocks in each layer, i.e., only one block is used in each layer of ResNet- 18 .
B.2 Classification Ability of the Surrogate Model.
[Table]
To demonstrate that the lightweight black-box attack performance does not rely on the generalization of classification, we show the classification accuracy of the lightweight surrogate model used for ETF attacks in Table 5.
To make the conclusion clearer, we also report the performance of a general surrogate model, which is trained on the training set of target models.
We can see that the test accuracy of the lightweight surrogate model
(about 2 ) is drastically lower than that of the general model, bringing fresh air for black-box attacks.
Specifically, the common sense in black-box attacks is that mounting attacks requires a surrogate model, which generalizes well on the test set.
However, the experimental results in Table 5 show that the lightweight surrogate model has poor classification performance, suggesting that the generalizability of surrogate models is not a necessary condition for performing black-box attacks.
B.3 Approximation to more layers.
Besides applying ETF in the first layer [Table] we also conduct experiments that apply ETF to all layers except layer L .
The results are reported in Table 6.
It can be seen that applying ETF to other layers can marginally promote the attack success rate.
The phenomenon may result from the fact that the approximation error can accumulate with depth.
Moreover, considering that applying ETF will cause more computational overhead, we merely give the results of applying ETF to the first layer in the main paper.
[Table]
We conduct experiments using GEFORCE RTX 2080 Ti, CPU AMD Ryzen 7 3700X @3.6 GHz.
As merely 1,000 samples are required for the training of the lightweight surrogate model, the computational overhead is much less than the training of general surrogate models.
C Further Experiments on ImageNet
Strict Constraint.
We conduct experiments under smaller ϵ , i.e., ϵ = 0 .
The results are given in
Table 7, demonstrating that ETF can generate powerful adversarial examples even with meeting more strict constraints, i.e., smaller ϵ . [Table]
More Validation Images.
Besides the widely used setting on the number of samples, i.e., 1, images, we also evaluate different methods using more samples, i [Table] e., 5,000 images, and report the results in Table 8.
The conclusion drawn from Table 8 is consistent with that drawn from Table 1, e.g., EFT outperforms “shallow” attack methods, demonstrating that ETF can generate powerful adversarial examples under various scenarios. [Table]
Shallow
[Table] PGD 19. [Table] Shallow
[Table]
DI 19. Shallow [Table] TI 20.
[Table] ETF
[Table]
MI 15. ETF
[Table]
DI 13. ETF
[Table]
TI 13.
[Table]
[Table] MI 11.
[Table] -DI 7.
Deep*-TI 9.
ℓ 2 -norm Perturbation. We mainly conduct experiments with ℓ ∞ perturbation since it is widely adopted in many previous works [57, 24, 11]. To further demonstrate the power of our ETF, we further evaluate different methods using ℓ 2 -norm perturbation.
The results are reported in Table 9, which further demonstrate the effectiveness of our proposal.
Considering that ℓ 1 and ℓ 0 perturbations require careful design [53, 4], it is beyond the scope of this work, so we leave it as our future work.
Architecture Selection.
We further exploit whether the architecture of surrogate models have significant impact on the performance of ETF.
Specifically, we instantiate the shallow layers with different model architectures containing ResNet [20], VGG [52], and SENet [21].
The results are reported in Table 10, demonstrating that our EFT is powerful across various model architectures.
Heavy Data Augmentation.
We follow the empirical conclusion suggested in [1], where heavy data augmentation is vital for training appropriate shallow models.
Because appropriate shallow models are necessary for mounting lightweight black-box attacks, data augmentation plays a crucial role and is heavily used in our experiments.
This is supported by results shown in Table 11, where we report the performance of lightweight black-box attacks with and without data augmentation.
Capacity to Evade Adversarial Detectors.
Adversarial Detection [39, 32, 58] aims to distinguish adversarial examples from natural examples, which is also an effective way to test the robustness of adversarial attacks.
Therefore, we further exploit the capacity of ETF in evading adversarial example detectors.
Specifically, we employ a detection method [39, 32] to detect adversarial examples generated by different attack methods, e.g., FGSM [18], PGD [41], BIM [10], and ETF.
All settings are the same as that used in the paper, and the results are reported in Table 12.
We can see that ETF performs better than the baselines, i.e., having a high probability of evading detection methods. [Table] [Table]
[Table]
No-Aug 34.
Aug 14.
[Table]
D Results on CIFAR
We conduct the experiments on the CIFAR10 dataset, see Table [Table] and evaluate the robustness of models downloaded from RobustBench [7], see Table 14.
The conclusion drawn from Table 13 and
Table 14 is consistent with that drawn from Table 1 evaluating on ImageNet dataset. [Table] [Table]
E Target Model Approximation Assumption
Taking the first layer as an example, let w 1 t and w 1 stand for the parameters of the target and surrogate models, respectively.
In many practical scenarios, w 1 t and w 1 usually have different dimensions, leading to intractable parameters’ discrepancy alleviation.
Fortunately, we can find an appropriate low-rank approximation for parameters of deep neural networks [9, 63, 29].
Specifically, we can approximate either w 1 t or w 1 to make these two matrices have the same dimensions, so we can consider that the dimensions of the two models are the same.
Consequently, we can find a transformation matrix A such that the approximation error is minimized, i.e., A = arg min ˜
A | w 1 t − w 1 − w 1 ˜
A | F , where | · | F is the Frobenius norm.
In this paper, we assume the approximation error is infinitesimal, i.e., | w 1 t − w 1 − w 1 A | F = 0 .
Then, we leverage w 1 and A to represent the target model, i.e., w 1 t = w 1 + w 1 A .
F Perturbation in Different Space
In the no-box setting, performing the min-max strategy in the feature space is more appropriate than the weight space optimization [56] for the no-box threat model.
This is because we know which perturbations are preferred in the feature space, e.g., towards features of guide images, but we have no idea about which perturbations are preferred in the weight space, i
[Table]
e., no “guide models”, which is supported by our experiments, see Table 15.
[Table]
G Different Self-supervised Learning Approach
It is straightforward that exploring different strategies to train the shallow model is exciting for further improvement of the performance of lightweight black-box attacks, as shallow layers play an important role in lightweight black-box attacks.
Thus, we generate adversarial examples using EFT with shallow layers trained with a rotation prediction task [16] and report the results in Table 16.
We can see that shallow layers trained with the rotation prediction task is slightly worse than using the contrastive strategy
[Table]
but the performance can also reduce the model accuracy significantly.
H Social Impact
The motivation of this work is to provide an approach to evaluate the adversarial robustness in a more practical scenario, the no-box setting.
Defenses can be assessed with fewer constraints
[Table]
through lightweight black-box attacks, i [Table] e., without accessing training samples and any queries.
We can develop defensive models robust against lightweight black-box attacks and attack algorithms to mislead deployed models.
We believe the development of lightweight black-box attacks can help better access the robustness of deployed models and hope the proposed ETF can promote the development of corresponding defense methods.