Towards Lightweight Black-Box Attacks against Deep Neural Networks
Abstract
Black-box attacks can generate adversarial examples without accessing the parameters of deep neural networks (DNNs), largely exacerbating the threats of deployed models.
However, previous works state that black-box attacks fail to mislead DNNs when their training data and outputs are inaccessible.
In this work, we argue that black-box attacks can pose practical attacks in this highly restrictive scenario where only several test samples are available.
Specifically, we find that attacking the shallow layers of DNNs trained on a few test samples can generate powerful adversarial examples.
As only a few samples are required, we refer to these attacks as lightweight black-box attacks.
The main challenge to promoting lightweight attacks is to mitigate the adverse impact caused by the approximation error of shallow layers.
As it is hard to mitigate the approximation error with few available samples, we propose Error TransFormer (ETF) for lightweight attacks.
Namely, ETF transforms the approximation error in the parameter space into a perturbation in the feature space and alleviates the error by disturbing features.
In our experiments, lightweight black-box attacks with the proposed ETF achieve surprising results.
For example, even if only 1 sample per category is available, the attack success rate achieved by lightweight black-box attacks is only about 3% lower than that of the black-box attacks using complete training data 1 .
Introduction
Black-box attack methods [12, 58, 62] can mount successful attacks without accessing the parameters of deep neural networks (DNNs), posing great challenges to deep learning in safety-critical situations.
Existing black-box attack methods implicitly assume that the training data and/or the outputs of target models are available to adversaries.
For scenarios where training data are available, adversarial examples can be generated by attacking surrogate models constructed for approximate target models.
Using the target models’ outputs for estimating gradients is another practical approach to crafting adversarial examples.
In general, due to the low dependency on target model information, the black-box attacks raise realistic threats to real-world deep learning systems, including semantic segmentation [2, 37], object detection [18, 47, 46], and automatic driving [56].
However, these assumptions can be violated in many practical scenarios [3, 48], where both the training data and the outputs of target models are inaccessible.
This realistic attack scenario is introduced in [3], generally termed as the no-box setting .
Under such a restrictive setting, Li et al.
[35] stated that existing black-box attacks fail to mislead target models because it is now forbidden to access previously essential information in mounting black-box attacks.
Therein, the failure of existing black-box attacks is not surprising.
The reason is that constructing reliable surrogate models typically relies on accessing the complete training data of target models, which is forbidden in such a restrictive setting with only a few test samples available.
In this work, we aim to reveal the potential threats of black-box attacks in the no-box setting, as mounting black-box attacks with highly limited accessible information leaves many models exposed to attack.
Specifically, we challenge the previous believes by raising the following question: Can black-box attacks success in the no-box setting?
The doubts about the potential threats of the blackbox attack are not groundless.
Specifically, the potential threats of black-box attacks stem from two facts: a) adversarial examples can be generated by perturbing representations at shallow layers of DNNs [28, 27, 50]; b) regarding the representation of shallow layers, there do not exist critical differences between those models learned from a few data and that of the whole training data.
Building upon the above facts, it is actually possible that an adversary can successfully mount attacks using limited samples, as powerful adversarial examples can be generated by attacking the shallow layers of DNNs trained on few samples.
Concretely, the adversary can leverage available samples to construct a surrogate model to approximate the shallow layers of target models within acceptable errors.
Consequently, adversarial examples can be generated by perturbing the features obtained from the shallow layers of the constructed surrogate model.
As merely a few samples are required for attacking, we refer to black-box attacks in the no-box setting as lightweight black-box attacks .
Intuitively, the closer the surrogate model is to the target model, the higher the attack success rate the crafted adversarial examples have [31].
If the approximation error of shallow layers is alleviated, the lightweight black-box attack can be as powerful as black-box attacks with complete training data.
Hence, the main challenge to mounting a lightweight attack is to mitigate the adverse impact caused by the approximation error of shallow layers.
However, it is challenging to mitigate the approximation error, especially when the number of available samples is limited.
Fortunately, it is straightforward to identify which kind of perturbations are preferred: if perturbations applied to a feature contribute to fooling the surrogate model, the perturbation is preferred.
Therefore, bridging the connection between the parameter space and the feature space is the key to mitigating the approximation error.
Accordingly, we propose transforming the approximation error in the parameter space as the perturbation in the feature space, dubbed Error TransFormer (ETF).
Namely, ETF transforms the worst-case approximation error to the worst-case feature perturbation, leading to a min-max scheme that generates adversarial examples under the worst feature perturbations.
We verify the attack success rate of lightweight black-box attacks usingmodels trained on the ImageNet dataset [49] and find that existing attack methods are much more potent than previously claimed.
Moreover, the performance of lightweight black-box attacks can be further promoted by the proposed ETF, i.e., the attack success rate is only 3% lower than that achieved by black-box attacks having complete training data of target models.
Related works 2.1 Adversarial attack
According to the amount of accessible information, existing adversarial attacks can be roughly divided into two categories: white-box attacks and black-box attacks.
White-box attacks [19, 43, 15, 16] assume that the target model is transparent to adversaries, i.e., adversaries can access all information about the target model.
Nevertheless, the assumption of transparent target models can be violated in many practical scenarios [11].
Hence, black-box attacks [14, 38, 66, 7] are proposed and applied to the scenario where relatively limited information about the target model is accessible, i.e., only a certain number of model queries or the training data of target tasks are available.
Among these black-box attacks, intermediate-level attacks [50, 28, 24] are widely explored to improve the adversarial transferability.
The core idea of these attacks is based on the empirical observation that well-trained models’ intermediate features are transferable [60].
This is consistent with the recent works showing that adversarial examples comprise spurious features having the transferable property [65, 6].
Hence, Inkawhich et al.
[28] propose to perturb the feature space of neural networks to create more transferable adversarial examples.
These methods provide an interesting empirical conclusion that disturbing the shallow layers of DNNs can also generate adversarial examples.
Existing black-box attacks assume that the large-scale training data of target models and the feedback from querying target models are accessible, but these assumptions can be violated in many scenarios, e.g., the no-box setting [3].
Li et al.
[35] stated that existing black-box attacks cannot be successfully mounted because existing DNNs require large-scale training data for generalization.
Given only smallsize data available, obtaining a surrogate model with strong generalization is challenging.
Hence, Li et al.
propose replacing black-box attacks with their proposed no-box attack [35], where they train a classical auto-encoder model instead of the supervised classification model due to the constraint of a small-scale dataset.
Concretely, they train 20 auto-encoders for each category and generate adversarial examples by attacking these auto-encoders, which is time- and computational-consuming.
Different from existing black-box attacks, lightweight black-box attacks aim to reveal the potential risk of black-box attacks under the no-box setting.
Because one main challenge to perform lightweight black-box attack is to mitigate the adverse impact caused by the approximation error, we propose error transformer that is a min-max strategy transforming the approximation error in the weight space to the feature space.
The min-max strategy is different from that introduced in [57], where min-max strategy is proposed to flatten the loss landscape in the weight space, see Appendix F.
2.2 Approximation of Shallow Networks
A recent study [60] confirms the intuition that shallow layers in DNNs can be seen as low-level feature extractors, provided that strong data augmentation is used [1].
Motivated by this observation,
Asano et al.
[1] design a method to explore the information in every layer of DNNs.
Asano et al.
[1] train a supervision model on ImageNet LSVRC-12, and a self-supervision model on a small-scale dataset.
Then, they apply linear probes [63] to all intermediate layers of networks, where a linear classifier is trained on the top of pre-trained and fixed feature representations.
In this way, they evaluate the quality of the representation learned at different depths of the networks.
The results show that, given heavily synthetic transformations, the shallow layers of DNNs learned from a few images can approximate that of DNNs trained on millions of images.
According to the intriguing empirical observation, it is possible to construct a surrogate model with limited samples, where its shallow layers are similar to that of the target models.
Besides, it is known that the neural networks prefer to make the decision through the spurious feature [65] on which are focused by models to correlate to the true label.
Since shallow layers based on the small-scale or large-scale data set can acquire the similar spurious feature [1], it is an effective way to leverage the shallow model to mount an attack in the spurious features when the number of the data is limited.
Preliminaries 3.1 No-Box Threat Model
The no-box setting [3] denotes a threat model where available information about the target model is extremely limited, making it challenging to generate adversarial examples.
According to the description in [3], the no-box threat model can be defined as follows.
Definition 1 (No-Box Threat Model)
A threat model is called the no-box threat model if the adversary in this scenario is not allowed to access the target model’s training data and outputs, so it only has some samples that can be correctly predicted with high probability by the target model.
The definition is consistent with the no-box setting introduced in [3], where the target model’s training data and outputs are inaccessible to the adversary.
Moreover, the available samples used for generating adversarial examples are expected to be limited.
This is because if the number of samples is similar to or even larger than the number of samples used for training target models, the adversary can leverage these samples to construct a surrogate model similar to target models, making the condition of inaccessible training data meaningless.
It is intuitive that target models can correctly predict the labels of these available samples with high probability.
For example, it is meaningless to perform attacks using samples misclassified by the target model.
3.2 Lightweight Black-Box Attack
Definition 2 (Lightweight Black-Box Attack)
An attack is called the lightweight black-box attack if an adversary aims to perform an attack in the context of a no-box threat model.
The definition of the lightweight black-box attack shows that lightweight attack is a special kind of black-box attacks.
The extremely limited accessible information distinguishes it from other blackbox attacks.
For example, query-efficient black-box attacks require the feedback information of target models [25, 26], while lightweight attacks do not.
To our best knowledge, how to perform a lightweight black-box attack is still lacking in the literature.
For example, Li et al.
[35] claim that black-box attacks fail to fool target models when their training data and outputs are inaccessible.
Approach
This section gives a detailed description of how to perform lightweight black-box attacks and the proposed Error TransFormer (ETF) to alleviate to adverse impact caused by approximation error.
4.1 Lightweight Surrogate Model
Surrogate models are widely used in black-box attacks since adversarial examples are usually generated by attacking surrogate models when querying target models is forbidden.
Since the lightweight black-box attack is a special kind of black-box attack, performing the attack also requires constructing surrogate models.
We call surrogate models employed in lightweight black-box attacks the lightweight surrogate model , building upon the fact that the number of available samples in the no-box threat model is typically limited.
Similar to existing black-box attacks, we assume that the label information of samples used for generating adversarial examples are accessible.
In this scenario, the lightweight surrogate model is trained for a classification task.
Specifically, the lightweight model is realized by training a classification DNN equipped via conventional supervised learning, using cross-entropy loss ℓ ( · , · ) :
[Equation]
where w stands for the parameters of the surrogate model f , and ( x, y ) denotes the sample label pair of random variables sampled from the distribution of natural data ˆ D .
In black-box attacks, adversary only leverages the shallow layers of DNNs, leading to a straightforward approach to constructing lightweight surrogate models.
Specifically, we can train the lightweight surrogate model in a contrastive manner, where the supervised information is no longer necessary:
[Equation]
where σ is a specific similarity metric, e.g., cosine similarity, T denotes the data transformation operation, and ˆ D \ x means that x is removed from the dataset ˆ D .
The contrastive strategy in Eq.makes it possible to perform attacks when the label information is unavailable.
Unless otherwise specified, we mainly uses Eq.
1 to train the surrogate models.
4.2 Feature Space Perturbation
Built upon the empirical observation [60], using limited samples can approximate the shallow layers of DNNs within acceptable errors.
This suggests that deep layers other than shallow layers can cause large approximation error.
Accordingly, merely the shallow layers of the learned lightweight surrogate models are used for generating adversarial examples, which is similar to the feature space attacks [50].
For brevity, we define the shallow layers as φ and denote the feature as φ ( x ; w ) .
In feature space attacks, guide images are usually required for generating adversarial examples [50].
A natural image used for generating adversarial example is usually called source image.
Given a source image x s , we generate an adversarial example x adv by perturbing x s such that its feature is similar to that of a guide image x g , where the labels of these two images are different.
Specifically, the adversarial examples are generated as follows:
[Equation]
[Equation]
where d stands for a specific distance, ϵ controls the strength of adversarial perturbation, and x ′ denotes the perturbed version of the source image.
4.3 Error Transformer
Although applying feature space attacks to lightweight surrogate models can generate transferable adversarial examples, the approximation error will cause adverse impact to mounting attacks.
The reason is that large dissimilarity between the surrogate models and the target models reduces the attack success rate.
Therefore, the main challenge to mounting lightweight attacks is to mitigate the adverse impact caused by the approximation error.
However, available samples are usually limited in the no-box threat model, making it hard to alleviate the approximation error.
To address the challenge, we propose transforming the parameter space’s approximation error as the feature space’s perturbation.
Specifically, we seldom know which perturbations can point (from the surrogate model) to the target model, making it challenging to alleviate the approximation error in the weight space.
In contrast, we have the prior that samples with different labels should have distinguishable representations.
Thus, we can leverage such prior knowledge to select preferred perturbations in the feature space, i.e., we prefer perturbations that can make representations of samples with different labels indistinguishable.
Similarly, it is also straightforward to define the “bad” perturbations, leading to the design of the min-max optimization to identify the “worst” model.
Thus, connecting the parameter space and feature space makes it possible to mitigate the adverse impact caused by approximation error.
To bridge the connection, we introduce the following key identity, which is also used in [45]:
[Equation]
where w 1 stands for the first layer parameters of model φ , A is a transformation matrix used for perturbing w 1 , (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) means that the first layer’s parameters are perturbed, the other layers’ parameters keep unchanged, and g is the function parameterized with w \ w 1 used for processing the first layer’s outputs.
According to Eq. 4, we can transform a perturbation in the parameter space as the perturbation in the data space, i.e., φ ( x ; (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) )
=
φ ( x + Ax ; w ) .
Similarly, we can apply the transformation operation to the hidden layers, see
Appendix A for more details.
Built upon the connection in Eq.
4, we can transform the first layer parameters’ discrepancy between the target model and the surrogate model.
Taking the first layer as an example, let w t be the parameters of the target model and w 1 t be its first layer parameters, where the other parameters are the same as the surrogate model.
Assume 2 that there exists a transformation matrix A applied to the first layer parameters of the surrogate model such that w 1 t = w 1 + w 1 A , we have:
[Equation]
The identity in Eq.
5 shows that we can find a perturbation in the data space to alleviate the difference in the first layer parameters between these two models.
Built upon the transformation, we can transform the worst approximation error to the worst perturbation in the data space.
Therefore, we can employ a min-max strategy to mitigate the adverse impact of approximation error through generating adversarial examples under the worst perturbation.
Thus, lightweight black-box attacks with ETF generate adversarial examples as follows:
[Equation]
where ∥·∥ p is the ℓ p -norm, ϵ and τ control the strength of perturbations, ∆ g and ∆ s are the data space perturbations for mitigating the approximation error in parameter space, and x ′ denotes the perturbed version of source image.
In this way, we reduce the error in parameter space to approximate the target model.
We solve the inner maximization problem by generating perturbations in the feature space, given a perturbed adversarial example.
This step aims to mitigate the approximation error.
The outer minimization problem is solved by finding adversarial perturbations in the input space, the same as the adversarial example generation.
After iterative generation of perturbations, adversarial examples are generated by attacking a model with a reduced approximation error.
Experiments
In this section, we conduct extensive experiments to verify the power of lightweight black-box attacks augmenting with ETF.
[Table]
5.1 Experimental Setup
Models Architectures.
All surrogate models are based on the ResNet- 18 [21].
In the ordinary settings of the black-box attack, the general surrogate models are trained through the whole training set of ImageNet [49].
On the contrary, the lightweight surrogate models adopt only 1 , 000 images randomly sampled from the validation set of ImageNet, considering that images in the training set are probably inaccessible in practice [35].
Results evaluated on the CIFAR10 dataset can be found in Appendix D. Please refer to Table 5 in the Appendix B for the detailed information on the lightweight surrogate model and the general surrogate model.
Regarding the target models, various model architectures are selected for full comparison, including, VGG- 19 [53], Inception v3 [55],
ResNet- 152 [21], DenseNet [23], SENet [22], wide ResNet (WRN) [61] and MobileNet v2 [51].
All these target models are well-trained on ImageNet.
Implement Details.
The lightweight surrogate models are randomly initialized.
The batch size is 128 , the epoch is 500 , and the initial learning rate is 0 .
4 , linearly decreasing to 0 .
To better approximate the low-level feature of shallow layers, we refer to the self-supervision work [5] and apply various random augmentations to the data in each epoch for training.
To mount attacks, the classic methods, e.g., PGD [42], MI [11], DI [58], and TI [12], are applied to all the surrogate models.
Unless otherwise specified, our attack is mounted on the first layer of ResNet-18 for lightweight surrogate models.
Following previous works on intermediate-level attacks, the metric d in Eq. 3 of
Shallow- in Table 1 is instantiated as the ℓ 2 -norm.
Meanwhile, to maximize the potential of ETFs, we apply contrastive loss to ETF-( · ) in Table 1.
The Top-1 prediction accuracy is adopted as the evaluation metric, i.e., lower the classification accuracy means better attack success rate.
All the experiments are run for 5 individual trials with different random seeds.
In our experiments, target models are evaluated using ℓ ∞ -norm adversarial examples with maximum distortion ϵ = 0 .
1 , results evaluated under mores settings, e.g., more strict constraints ϵ = 0 .
05 , ℓ 2 -norm adversarial examples, and more test images 5,000 images, can be found in Appendix B.4.
5.2 Main Results
Results on normally trained models.
The results of classification accuracy are shown in Table 1, which exhibits the evaluations for both lightweight and general surrogate models.
The last four rows
[Table]
show the ordinary results where the general surrogate models are trained through all data in the training set.
The black-box attacks are mounted based on the whole model, termed ”Deep/Deep ∗ ”.
As the size of the training data is reduced to 1 , 000 , this conventional method obviously loses its aggressiveness.
Specifically, the averaged accuracies of 3-6 rows are among 40 - 50% , revealing limited attacking capability compared to 66 .
99% of the clean accuracy.
It demonstrates the necessity of large amount of data for classic black-box attacks.
In rows 7-10, shallow layers without ETF are used to generate adversarial examples.
The averaged accuracies of around 25% illustrates the less dependence of shallow networks on the amount of data.
In rows 11-14, attacks on shallow layers are further enhanced by ETF, decreasing the averaged accuracies to about 17% .
The narrow gap between the proposed ETF and the general surrogate model indicates the potential capability of the black-box attack in the no-box settings.
Results on Adversarially Trained Models.
Here, we consider the models that are trained through adversarial training [42], which is more persuasive to evaluate the effectiveness of attacking methods.
Following [42], the target models are trained on the whole training set of ImageNet.
During adversarial training, the adversarial examples are generated under ϵ = 0 , 4 / 255 , 8 / 255 , where ϵ means the ℓ ∞ - norm constraint.
Table 3 exhibits the attacking results mounted by ETF, black-box attacks (general surrogate models with all training data avaliable), and no-box attacks.
For simplicity, only the results of PGD attacks are reported.
As analysed above, the black-box attacks shows the best attacking results, while ETF only has a narrow gap with it.
However, when ϵ = 4 / 255 or 8 / 255 , ETF exceeds the black-box attacks (29.13/26.14 vs. 48.11/38.24).
It is interesting that black-box attacks are even worse than lightweight attacks.
We believe that exploiting the phenomenon can bring something new, but we leave it as our future work due to the limited space.
5.3 Further Analysis and Ablation Study
[Table]
Intermediate-Level Perturbations.
Since the lightweight black-box attacks are implemented mainly on intermediate features, two classic attacks mounted on intermediate features, i.e. ILA [24] and AA [28], are selected for experiments.
Comparing the ILA and AA in Table 2 and Shallow-MI/DI/TI/PGD in Table 1, the difference between their average accuracies is marginal.
It indicates those intermediate-level feature perturbation has limited benefit to lightweight black-box attacks.
The reason is that the intermediate-level attacks highly rely on the low-level features extracted by the lightweight model to generate adversarial examples and it can weaken the role of high-level information when attacking.
The proposed ETF achieves better performance.
On one hand, ETF does not rely on the features from high-level layers.
A v e r a g e acc u r ac y
Number of training images
Black-box
Nobox       maxpooling layer layer layer layer
FC
A v e r a g e acc u r ac y Layer index
VGG
Inception v3 RN DenseNet SENet WRN MobileNet
[Figure]
Comparison with No-box Attacks.
No-box attack is the first work to explore how to mount attacks in the no-box setting.
Since Li et.
al [35] stated that black-box attacks do not work in this setting, surrogate models are instantiated as auto-encoders rather than models for approximating target models.
Namely, they design the pretext task to train 20 auto-encoders per category to generate adversarial examples.
However, the difference between the pretext tasks and the target tasks may significantly reduce the transferability of adversarial examples.
In contrast, our method trains the lightweight surrogate model for tasks similar to or same as the task of target models.
Hence, though only one sample per category is available, we achieve better performance than no-box attacks, as shown in Table 2.
Number of Training Samples.
One key difference between the lightweight black-box attack and existing black-box attacks mainly lies in the number of samples n used for training surrogate models.
Therefore, we further explore how lightweight attack performance varies with different number of samples, i.e., n .
We conduct experiments on randomly selected n images from the validation set of ImageNet.
The experimental results are shown in Figure 1.
The average accuracy decreases as more samples are available, thus achieving better attacking results.
More data makes it possible for the lightweight black-box attack to approximate the shallow layers of the target model more accurately.
Especially the lightweight black-box attacks with 10 , 000 test images can achieve the same performance as general black-box attacks which train the surrogate model on 1 , 200 , 000 training images.
Also, our method can still mount attacks even when only 2 or 10 images are available.
Surrogate Model.
As analyzed above, we utilize the first few layers of the lightweight surrogate model to approximate the shallow layers of the target model to mount attacks.
Hence, it would be interesting to study how the layers selection for lightweight models impacts the attack performance.
We adopt different layers of ResNet- 18 to generate adversarial examples utilizing ETF and attack the different target models.
The results are summarized in Fig.
It can be seen that low-level information in the first block of the model is sufficient for achieving promising attack performance.
With more block information on the model, the performance of the attack does not improve.
In particular, the transferability of adversarial examples significantly decreases when we attack the fully-connected layer, verifying our analysis that the high-level layer of the lightweight model cannot approximate the target model well.
More results for architecture selection can be found in Appendix B.4.
Adversarial Example Visualization.
To verify that adversarial examples are truly imperceptible, we provide visualization in Figure 2, where Deep*-PGD attack (using training images), Deep-PGD attack (using test images), and lightweight black-box attack are considered.
Discussion
Though the no-box threat model provides a weak assumption, many realistic scenarios may be more complex.
For example, adversaries have access to only data without supervision information due to security [36] or privacy [48] issues.
In more extreme cases, e.g., autonomous vehicle [56], adversaries even have no access to the test samples from the distribution over which the model is trained.
Instead,
[Table]
[Figure]
the adversaries have access to samples from a similar (but not exact) distribution (because the adversary knows the deployment environment).
Hence, we consider two more challenging scenarios in which we provide solutions to mount lightweight black-box attacks.
Insufficient supervision information.
We design an experiment for mounting attacks using 1 ,unlabeled samples randomly selected from the ImageNet validation set to explore the power of
ETF under this challenging setting with unlabeled data.
Thanks to the framework of lightweight attack, it is feasible to train the unsupervised model on unlabeled data for attacks using Eq. 2.
Specifically, we adopt contrastive learning [5] to train a lightweight surrogate model on the smallscale unlabeled dataset and generate adversarial examples by attacking the trained lightweight surrogate model.
The performance is shown in Table 4, where ETF still achieves similar performance without supervision information as that of the results under supervision.
Intuitively, we can also employ other self-supervised learning method, e.g., rotation prediction, to train shallow layers, see details of experimental settings and results in Appendix G.
Consideration of Out-of-distribution.
We further study the possibility for mounting attacks under a more strict scenario, where only OOD (out-of-distribution, OOD) data are available.
Specifically, we utilize the model pre-trained on OOD samples to mount lightweight black-box attacks.
The rationality lies in that only low-level information is required to mount lightweight black-box attacks and the acquisition of low-level information.
We load pre-trained ResNet- 50 on STL- 10 [39] dataset as the surrogate model.
Without further fine-tuning, we experiment with two test in-distribution samples, and the results are recorded in Table 4, named OOD.
As we can see, the lightweight black-box attack with models trained on OOD data can achieve a surprising attack success rate, showing that black-box attacks enhanced with ETF can pose practical threats to deployed models.
Conclusion
In this paper, we propose the conception of the lightweight black-box attack to reveal the potential risk of black-box attacks under the no-box threat model.
To mount effective lightweight attacks, we find it crucial to leverage DNNs‘ shallow layers because they can learn similar features regardless of the number of samples.
This is based on the empirical observations that the model trained with limited samples can approximate the shallow layers of the target models within acceptable errors, which can be used for crafting adversarial examples.
Therefore, to further enhance the performance, we propose Error TransFormer (EFT) to decrease the approximation error by transforming the approximation error in the parameter space into the feature space.
The experiments show that the proposed method achieves a surprising attack success rate under the no-box threat model using only one image per category, i e., only 3% lower than black-box attacks with complete training data of the target model.
Acknowledgements
This work was supported by NSFC No. 61872329 and No. 62222117, and the Fundamental Research
Funds for the Central Universities under contract WK3490000005.
YZ and BH was supported by
NSFC Young Scientists Fund No. 62006202 and Guangdong Basic and Applied
Basic Research
Foundation No. 2022A1515011652.
TL was partially supported by Australian Research Council
Projects DP180103424, DE-190101473, IC-190100031, DP- and FT-220100318.
A ETF on Hidden Layers
Before giving the details of applying ETF to hidden layers, we revisit the key identity:
[Equation]
where w 1 stands for the first layer (convolution layer) parameters of model φ , A is a transformation matrix used for perturbing w 1 , (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) means that the first layer’s parameters are perturbed and the other layers’ parameters keep the same, and g is the function parameterized with w \ w 1 used for processing the first layer’s outputs.
Built upon Eq.
7, we have φ ( x ; (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \
[Equation] [Equation]
=
[Equation]
( x + Ax
[Equation]
w ) .
The model φ can be decomposed: φ = g l ◦ φ l , where φ l ( x ) denotes the hidden feature at layer l .
Then the output feature at layer L is expressed as φ L ( x ; w ) =
g l ( w l φ l ( x )) , where w l is the parameter used for processing the feature φ l ( x ) .
Then, we can apply the identity at layer l :
[Equation]
where A l is the transformation matrix at layer l .
According to Eq.
8, we can transform a perturbation in the parameter space ( Aw l ) as the perturbation in the feature space ( Aφ l ( x ) ).
Thus, lightweight black-box attacks with ETF generate adversarial examples as follows:
[Equation]
where ∆ ls ( ∆ lg ) denotes the feature space perturbation ( l = 0 means the data space), ∪ l ∆ ls ( ∪ l ∆ lg ) stands for all perturbations in the feature space, and φ ( x g , ∪ l ∆ lg ; w ) ( φ ( x g , ∪ l ∆ lg ; w ) ) stands for the output feature with perturbed features, where the feature of source (guide) image at layer l is perturbed by ∆ ls ( ∆ lg ).
Although we can perform the error transformation in the feature space, the features obtained using weights with approximation errors make it challenging.
Specifically, we merely know that the input distribution (can be seen as the feature map) is not biased, but the feature map obtained using any weights will cause bias.
Thus, we merely apply ETF to the input layer and leave further exploitation of applying it to hidden layers as future work.
B Additional Detail Description B.1 Simplified Architecture
The architecture of surrogate models is modified to avoid overfitting.
Considering the limited amount of data, we employ a network with a small model capacity to instantiate the feature extractor.
In particular, ResNet- 18 is simplified by reducing the number of blocks in each layer, i.e., only one block is used in each layer of ResNet- 18 .
B.2 Classification Ability of the Surrogate Model.
[Table]
To demonstrate that the lightweight black-box attack performance does not rely on the generalization of classification, we show the classification accuracy of the lightweight surrogate model used for ETF attacks in Table 5.
To make the conclusion clearer, we also report the performance of a general surrogate model, which is trained on the training set of target models.
We can see that the test accuracy of the lightweight surrogate model
(about 2 ) is drastically lower than that of the general model, bringing fresh air for black-box attacks.
Specifically, the common sense in black-box attacks is that mounting attacks requires a surrogate model, which generalizes well on the test set.
However, the experimental results in Table 5 show that the lightweight surrogate model has poor classification performance, suggesting that the generalizability of surrogate models is not a necessary condition for performing black-box attacks.
B.3 Approximation to more layers.
Besides applying ETF in the first layer
[Table]
we also conduct experiments that apply ETF to all layers except layer L .
The results are reported in Table 6.
It can be seen that applying ETF to other layers can marginally promote the attack success rate.
The phenomenon may result from the fact that the approximation error can accumulate with depth.
Moreover, considering that applying ETF will cause more computational overhead, we merely give the results of applying ETF to the first layer in the main paper.
[Table]
We conduct experiments using GEFORCE RTX 2080 Ti, CPU AMD Ryzen 7 3700X @3.6 GHz.
As merely 1,000 samples are required for the training of the lightweight surrogate model, the computational overhead is much less than the training of general surrogate models.
C Further Experiments on ImageNet
Strict Constraint.
We conduct experiments under smaller ϵ , i.e., ϵ = 0 .
The results are given in
Table 7, demonstrating that ETF can generate powerful adversarial examples even with meeting more strict constraints, i.e., smaller ϵ .
[Table]
More Validation Images.
Besides the widely used setting on the number of samples, i.e., 1, images, we also evaluate different methods using more samples,
i
[Table]
e., 5,000 images, and report the results in Table 8.
The conclusion drawn from Table 8 is consistent with that drawn from Table 1, e.g., EFT outperforms “shallow” attack methods, demonstrating that ETF can generate powerful adversarial examples under various scenarios.
[Table]
Shallow
[Table]
PGD 19.
[Table]
Shallow
[Table]
DI 19.
Shallow
[Table]
TI 20.
[Table]
ETF
[Table]
MI 15.
ETF
[Table]
DI 13.
ETF
[Table]
TI 13.
[Table] [Table]
MI 11.
[Table]
-DI 7.
Deep*-TI 9.
ℓ 2 -norm Perturbation.
We mainly conduct experiments with ℓ ∞ perturbation since it is widely adopted in many previous works [58, 25, 12].
To further demonstrate the power of our ETF, we further evaluate different methods using ℓ 2 -norm perturbation.
The results are reported in Table 9, which further demonstrate the effectiveness of our proposal.
Considering that ℓ 1 and ℓ 0 perturbations require careful design [54, 4], it is beyond the scope of this work, so we leave it as our future work.
Architecture Selection.
We further exploit whether the architecture of surrogate models have significant impact on the performance of ETF.
Specifically, we instantiate the shallow layers with different model architectures containing ResNet [21], VGG [53], and SENet [22].
The results are reported in Table 10, demonstrating that our EFT is powerful across various model architectures.
Heavy Data Augmentation.
We follow the empirical conclusion suggested in [1], where heavy data augmentation is vital for training appropriate shallow models.
Because appropriate shallow models are necessary for mounting lightweight black-box attacks, data augmentation plays a crucial role and is heavily used in our experiments.
This is supported by results shown in Table 11, where we report the performance of lightweight black-box attacks with and without data augmentation.
Capacity to Evade Adversarial Detectors.
Adversarial Detection [40, 33, 59] aims to distinguish adversarial examples from natural examples, which is also an effective way to test the robustness of adversarial attacks.
Therefore, we further exploit the capacity of ETF in evading adversarial example detectors.
Specifically, we employ a detection method [40, 33] to detect adversarial examples generated by different attack methods, e.g., FGSM [19], PGD [42], BIM [11], and ETF.
All settings are the same as that used in the paper, and the results are reported in Table 12.
We can see that ETF performs better than the baselines, i.e., having a high probability of evading detection methods.
[Table]
[Table] [Table]
No-Aug 34.
Aug 14.
[Table]
D Results on CIFAR
We conduct the experiments on the CIFAR10 dataset, see Table
[Table]
and evaluate the robustness of models downloaded from RobustBench [8], see Table 14.
The conclusion drawn from Table 13 and
Table 14 is consistent with that drawn from Table 1 evaluating on ImageNet dataset.
[Table] [Table]
E Target Model Approximation Assumption
Taking the first layer as an example, let w 1 t and w 1 stand for the parameters of the target and surrogate models, respectively.
In many practical scenarios, w 1 t and w 1 usually have different dimensions, leading to intractable parameters’ discrepancy alleviation.
Fortunately, we can find an appropriate low-rank approximation for parameters of deep neural networks [10, 64, 30].
Specifically, we can approximate either w 1 t or w 1 to make these two matrices have the same dimensions, so we can consider that the dimensions of the two models are the same.
Consequently, we can find a transformation matrix A such that the approximation error is minimized, i.e., A = arg min ˜
A | w 1 t − w 1 − w 1 ˜
A | F , where | · | F is the Frobenius norm.
In this paper, we assume the approximation error is infinitesimal, i.e., | w 1 t − w 1 − w 1 A | F = 0 .
Then, we leverage w 1 and A to represent the target model, i.e., w 1 t = w 1 + w 1 A .
F Perturbation in Different Space
In the no-box setting, performing the min-max strategy in the feature space is more appropriate than the weight space optimization [57] for the no-box threat model.
This is because we know which perturbations are preferred in the feature space, e.g., towards features of guide images, but we have no idea about which perturbations are preferred in the weight space,
i
[Table]
e., no “guide models”, which is supported by our experiments, see Table 15.
[Table]
G Different Self-supervised Learning Approach
It is straightforward that exploring different strategies to train the shallow model is exciting for further improvement of the performance of lightweight black-box attacks, as shallow layers play an important role in lightweight black-box attacks.
Thus, we generate adversarial examples using EFT with shallow layers trained with a rotation prediction task [17] and report the results in Table 16.
We can see that shallow layers trained with the rotation prediction task is slightly worse than using the contrastive strategy
[Table]
but the performance can also reduce the model accuracy significantly.
H Social Impact
The motivation of this work is to provide an approach to evaluate the adversarial robustness in a more practical scenario, the no-box setting.
Defenses can be assessed with fewer constraints
[Table]
through lightweight black-box attacks,
i
[Table]
e., without accessing training samples and any queries.
We can develop defensive models robust against lightweight black-box attacks and attack algorithms to mislead deployed models.
We believe the development of lightweight black-box attacks can help better access the robustness of deployed models and hope the proposed ETF can promote the development of corresponding defense methods.