Towards Lightweight Black-Box Attacks against Deep Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Black-box attacks can generate adversarial examples without accessing the parameters of target model, largely exacerbating the threats of deployed deep neuralnetworks (DNNs).
However, previous works state that black-box attacks fail tomislead target models when their training data and outputs are inaccessible.
In thiswork, we argue that black-box attacks can pose practical attacks in this extremelyrestrictive scenario where only several test samples are available.
Specifically, wefind that attacking the shallow layers of DNNs trained on a few test samples can generate powerful adversarial examples.
As only a few samples are required, we referto these attacks as lightweight black-box attacks.
The main challenge to promotinglightweight attacks is to mitigate the adverse impact caused by the approximationerror of shallow layers.
As it is hard to mitigate the approximation error with fewavailable samples, we propose Error TransFormer (ETF) for lightweight attacks.
Namely, ETF transforms the approximation error in the parameter space into aperturbation in the feature space and alleviates the error by disturbing features.
In experiments, lightweight black-box attacks with the proposed ETF achievesurprising results.
For example, even if only 1 sample per category available, theattack success rate in lightweight black-box attacks is only about 3 lower than thatof the black-box attacks with complete training data.
Introduction
Black-box attacks [12, 56, 10] can mount successful attacks without accessing the parameters ofdeep neural networks (DNNs), posing great challenges to deep learning in safety-critical situations.
Existing black-box attack methods assume that the training data and/or the outputs of target modelsare available to adversaries.
For scenarios where training data are available, adversarial examples canbe generated by attacking surrogate models constructed for approximate target models.
Using thetarget models’ outputs for estimating gradients is another practical approach to crafting adversarialexamples.
In general, due to the low dependency on target model information, the black-box attacksraise realistic threats to real-world deep learning systems, including semantic segmentation [3, 36],object detection [16, 45, 44], and automatic driving [54].
However, these assumptions can be violated in many practical scenarios [4, 46], where both thetraining data and the outputs of target models are inaccessible.
This realistic attack scenario isintroduced in [4], generally termed as the no-box setting .
Under such a restrictive setting, Li et al.
[34]states that existing black-box attacks can fail to mislead target models, as the information previouslyconsidered essential in mounting black-box attacks is now inaccessible.
Therein, the failure ofexisting black-box attacks is not surprising.
The reason is that constructing reliable surrogate modelstypically relies on target models’ training data, which is inaccessible in such a restrictive setting withonly a few test samples available.
In this work, we aim to reveal the potential threats of black-box attacks in the no-box setting, asmounting black-box attacks with highly limited accessible information leaves many models exposedto attack.
Specifically, we challenge the previous believes by raising the following question: Canblack-box attacks success in the no-box setting?
The doubts about the potential threats of the blackbox attack are not groundless.
Specifically, the potential threats of black-box attacks stem fromtwo facts: a) adversarial examples can be generated by perturbing representations at shallow layersof DNNs [27, 26, 48]; b) regarding the representation of shallow layers, there do not exist criticaldifferences between those models learned from a few data and that of the whole training data.
Building upon the above facts, it is actually possible that an adversary can successfully mount attacksusing limited samples, as powerful adversarial examples can be generated by attacking the shallowlayers of DNNs trained on few samples.
Concretely, the adversary can leverage available samples toconstruct a surrogate model to approximate the shallow layers of target models within acceptableerrors.
Consequently, adversarial examples can be generated by perturbing the features obtained fromthe shallow layers of the constructed surrogate model.
As merely a few samples are required forattacking, we refer to black-box attacks in the no-box setting as lightweight black-box attacks .
Intuitively, the closer the surrogate model is to the target model, the higher the attack success ratethe crafted adversarial examples have [30].
If the approximation error of shallow layers is alleviated,the lightweight black-box attack can be as powerful as black-box attacks with complete trainingdata.
Hence, the main challenge to mounting a powerful lightweight attack is to mitigate the adverseimpact caused by the approximation error of shallow layers.
However, it is challenging to mitigate the approximation error, especially when the number ofavailable samples is limited.
Fortunately, it is straightforward to identify which kind of perturbationsare preferred: if perturbations applied to a feature contribute to fooling the surrogate model, theperturbation is preferred.
Therefore, bridging the connection between the parameter space and thefeature space is the key to mitigating the approximation error.
Accordingly, we propose transformingthe approximation error in the parameter space as the perturbation in the feature space, dubbedas Error TransFormer (ETF).
Namely, ETF transforms the worst-case approximation error to theworst-case feature perturbation, leading to a min-max scheme that generates adversarial examplesunder the worst feature perturbations.
We verify the attack success rate of lightweight black-boxattacks using 7 models trained on the ImageNet dataset [47], and find that existing attack methods aremuch powerful than previous claimed.
Moreover, the performance of lightweight black-box attackscan be further promoted by the proposed ETF, i.e., the attack success rate is only 3 lower than thatachieved by black-box attacks having complete target model’s training data.
Related works2.1 Adversarial attack
According to the amount of accessible information, existing adversarial attacks can be roughly dividedinto two categories: white-box attacks and black-box attacks.
White-box attacks [17, 41, 40, 2]assume that the target model is transparent to adversaries, i.e., adversaries can access all informationabout the target model.
Nevertheless, the assumption that all information is available to adversariescan be violated in many practical scenarios [10].
Hence, black-box attacks [40, 41, 56, 12] areproposed and applied to the scenario where relatively limited information about the target modelis accessible, i.e., only a certain number of model queries or the training data of target tasks areavailable.
Utilizing the training data or queries, adversaries generate adversarial examples on thesurrogate model constructed for approximate target models.
Among these black-box attacks, intermediate-level attacks [48, 27, 23] are widely explored to improvethe adversarial transferability.
The core idea of these attacks is based on the empirical observation thatwell-trained models’ intermediate features are transferable [57].
Hence, Inkawhich et al.
[27] proposeto perturb the feature space of neural networks to create more transferable adversarial examples.
Besides, these attacks provides an interesting empirical conclusion that disturbing the shallow layersof DNNs can also generate adversarial examples.
Existing black-box attacks assume that the large-scale training data of target models and the feedbackfrom querying target models are accessible, but these assumptions can be violated in many scenarios,such as no-box setting [4].
Li et al.
[34] state that existing black-box attacks cannot be successfully mounted because existing DNNs require large-scale training data for generalization.
Given only smallsize data available, obtaining a surrogate model with strong generalization is challenging.
Hence, Liet al.
propose replacing black-box attacks with their proposed no-box attack [34], where they traina classical auto-encoder model instead of the supervised classification model due to the constraintof a small-scale dataset.
Concretely, they train 20 auto-encoders for each category and generateadversarial examples by attacking these auto-encoders, which is time- and computational-consuming.
Different from existing black-box attacks, lightweight black-box attacks aim to reveal the potentialrisk of black-box attacks under the no-box setting. Because one main challenge to perform lightweightblack-box attack is to mitigate the adverse impact caused by the approximation error, we proposeerror transformer that is a min-max strategy transforming the approximation error in the weight spaceto the feature space. The min-max strategy is different from that introduced in [55], where min-maxstrategy is proposed to flatten the loss landscape in the weight space, see Appendix F.
2.2 Approximation of Shallow Networks
[Figure] A recent study [58] confirms the intuition that shallow layers in DNNs can be seen as low-levelfeature extractors, provided that strong data augmentation is used [1].
Motivated by this observation, Asano et al.
[1] design a method to explore the information in every layer of DNNs.
Asano et al.
[1]train a supervision model on ImageNet LSVRC-12, and a self-supervision model on a small-scaledataset.
Then, they apply linear probes [60] to all intermediate layers of networks, where a linearclassifier is trained on the top of pre-trained and fixed feature representations.
In this way, theyevaluate the quality of the representation learned at different depths of the networks.
The results showthat, given heavily synthetic transformations, the shallow layers of DNNs learned from a few imagescan approximate that of DNNs trained on millions of images.
According to the intriguing empiricalobservation, it is possible to construct a surrogate model with limited samples, where its shallowlayers are similar to that of the target models.
Preliminaries
3.1 No-Box Threat Model
The no-box setting [4] denotes a threat model where available information about the target modelis extremely limited, making it challenging to generate adversarial examples.
According to thedescription in [4], the no-box threat model can be defined as follows.
Definition 1 (No-Box Threat Model)
A threat model is called the no-box threat model if the adversary in this scenario is not allowed to access the target model’s training data and outputs, so it onlyhas some samples that can be correctly predicted with high probability by the target model.
The definition is consistent with the no-box setting introduced in [4], where the target model’straining data and outputs are inaccessible to the adversary.
Moreover, the available samples usedfor generating adversarial examples are expected to be limited.
This is because if the number ofsamples is similar to or even larger than the number of samples used for training target models, theadversary can leverage these samples to construct a surrogate model similar to target models, makingthe condition of inaccessible training data meaningless.
It is intuitive that target models can correctlypredict the labels of these available samples with high probability.
For example, it is meaningless toperform attacks using samples misclassified by the target model.
3.2 Lightweight Black-Box Attack
Definition 2 (Lightweight Black-Box Attack)
An attack is called the lightweight black-box attackif an adversary aims to perform an attack in the context of a no-box threat model.
The definition of the lightweight black-box attack shows that lightweight attack is a special kind ofblack-box attacks.
The extremely limited accessible information distinguishes it from other blackbox attacks.
For example, query-efficient black-box attacks require the feedback information oftarget models [24, 25], while lightweight attacks do not.
To our best knowledge, how to perform alightweight black-box attack is still lacking in the literature.
For example, Li et al.
[34] claim thatblack-box attacks fail to fool target models when their training data and outputs are inaccessible.
Approach
This section gives a detailed description of how to perform lightweight black-box attacks and theproposed Error TransFormer (ETF) to alleviate to adverse impact caused by approximation error.
4.1 Lightweight Surrogate Model
Surrogate models are widely used in black-box attacks, as adversarial examples are usually generatedby attack surrogate models when querying target models is forbidden.
Since the lightweight black-boxattack is a special kind of black-box attacks, performing attack requires constructing surrogate models.
We call surrogate models employed in lightweight black-box attacks lightweight surrogate models,considering that the number of samples are usually limited.
Similar to existing black-box attacks, we assume that the label information of samples used forgenerating adversarial examples are accessible.
In this scenario, the lightweight surrogate modelis trained for a classification task.
Specifically, the lightweight model is realized by training aclassification DNN equipped via conventional supervised learning, using cross-entropy loss ℓ ( · , · ) :
[Equation]
(1) where w stands for the parameters of surrogate models f , ( x, y ) sampled from the dataset
ˆ D denotesthe pair of natural sample and its label y .
In black-box attacks, adversary only leverages the shallow layers of DNNs, leading to a straightforward approach to constructing lightweight surrogate models.
Specifically, we can train the lightweightsurrogate model in a contrastive manner, where the supervised information is no longer necessary:
[Equation] [Equation]
where σ is a specific similarity metric, e.g., cosine similarity, T denotes the data transformationoperation, and ˆ D \ x means that x is removed from the dataset ˆ D . The contrastive strategy in Eq.
2makes it possible to perform attacks when the label information is unavailable.
Unless otherwisespecified, we mainly uses Eq.
1 to train the surrogate models.
4.2 Feature Space Perturbation
Built upon the empirical observation [58], using limited samples can approximate the shallow layersof DNNs within acceptable errors.
This suggests that deep layers other than shallow layers cancause large approximation error.
Accordingly, merely the shallow layers of the learned lightweightsurrogate models are used for generating adversarial examples, which is similar to the feature spaceattacks [48].
For brevity, we define the shallow layers as φ and denote the feature as φ ( x ; w ) .
In feature space attacks, guide images are usually required for generating adversarial examples.
Anatural image used for generating adversarial example is usually called source image.
Given a sourceimage x s , we generate an adversarial example x adv by perturbing x s such that its feature is similarto that of a guide image x g , where the labels of these two images are different.
Specifically, theadversarial examples are generated as follows:
[Equation]
where d stands for a specific distance, ϵ controls the strength of adversarial perturbation, and x ′denotes the perturbed version of source image.
4.3 Error Transformer
Although applying feature space attacks to lightweight surrogate models can generate transferableadversarial examples, the approximation error will cause adverse impact to mounting attacks.
Thereason is that large dissimilarity between the surrogate models and the target models reduces theattack success rate.
Therefore, the main challenge to mounting a powerful lightweight attack is tomitigate the adverse impact caused by the approximation error.
However, available samples areusually limited in the no-box threat model, making it hard to alleviate the approximation error.
To address the challenge, we propose transforming the parameter space’s approximation error as the 181 feature space’s perturbation.
Specifically, we have little knowledge to determine which perturbationscan point (from the surrogate model) to the target model, making it challenging to alleviate the 183 approximation error in the weight space. In contrast, we have the prior that samples with differentlabels should have distinguishable representations.
Thus, we can leverage the prior knowledge 185 to select preferred perturbations in the feature space, i.e., we prefer perturbations that can makerepresentations of samples with different labels indistinguishable.
Similarly, it is also straightforward 187 to define the “bad” perturbations, leading to the design of the min-max optimization to identifythe “worst” model.
Thus, connecting the parameter space and feature space makes it possible to 189 mitigate the adverse impact caused by approximation error.
To bridge the connection, we introducethe following key identity, which is also used in [43]:
[Equation]
where w 1 stands for the first layer parameters of model φ , A is a transformation matrix used forperturbing w 1 , (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) means that the first layer’s parameters are perturbed andthe other layers’ parameters keep the same, and g is the function parameterized with w \ w 1 usedfor processing the first layer’s outputs.
According to Eq. 4, we can transform a perturbation inthe parameter space as the perturbation in the data space, i.e., φ ( x ; (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) )
=
φ ( x + Ax ; w ) .
Similarly, we can apply the transformation operation to the hidden layers, see
Appendix A for more details.
Built upon the connection in Eq.
4, we can transform the first layer parameters’ discrepancy between 199 the target model and the surrogate model.
Taking the first layer as an example, let w t be the parametersof the target model and w 1 t be its first layer parameters, where the other parameters are the same asthe surrogate model.
Assume 1 that there exists a transformation matrix A applied to the first layerparameters of the surrogate model such that w 1 t = w 1 + w 1 A , we have:
[Equation]
The identity in Eq.
5 shows that we can find a perturbation in the data space to alleviate the differencein the first layer parameters between these two models.
Built upon the transformation, we cantransform the worst approximation error to the worst perturbation in the data space.
Therefore, we canemploy a min-max strategy to mitigate the adverse impact of approximation error through generatingadversarial examples under the worst perturbation.
Thus, lightweight black-box attacks with ETFgenerate adversarial examples as follows:
[Equation]
where ∥·∥ p is the ℓ p -norm, ϵ and τ control the strength of perturbations, ∆ g and ∆ s are the data spaceperturbations for mitigating the approximation error in parameter space, and x ′ denotes the perturbedversion of source image.
In this way, we reduce the error in parameter space to approximate the targetmodel.
We solve the inner maximization problem by generating perturbations in the feature space,given a perturbed adversarial example.
This step aims to mitigate the approximation error.
The outerminimization problem is solved by finding adversarial perturbations in the input space, the same asthe adversarial example generation.
After iterative generation of perturbations, adversarial examplesare generated by attacking a model with a reduced approximation error.  Experiments In this section, we conduct extensive experiments to verify the power of our ETF.
[-Section 5.
-]
5.1 Experimental Setup
Models Architectures.
All surrogate models are based on the ResNet- 18 [20].
In the ordinarysettings of the black-box attack, the general surrogate models are trained through the whole trainingset of ImageNet [47].
On the contrary, the lightweight surrogate models adopt only 1 , 000 imagessampled from the validation set of ImageNet, considering that images in the training set are probablyinaccessible in practice [34]. Results evaluated on the CIFAR10 dataset can be found in Appendix D.
[Table]
Please refer to Table 5 in the Appendix B for the detailed information on the lightweight surrogatemodel and the general surrogate model.
Regarding the target models, various model architecturesare selected for full comparison, including, VGG- 19 [51], Inception v3 [53], ResNet- 152 [20], DenseNet [22], SENet [21], wide ResNet (WRN) [59] and MobileNet v2 [49].
All these targetmodels are well-trained on ImageNet.
Implement Details.
The lightweight surrogate models are randomly initialized.
The batch sizeis 128 , the epoch is 500 , and the initial learning rate is 0 .
4 , linearly decreasing to 0 .
To betterapproximate the low-level feature of shallow layers, we refer to the self-supervision work [6] andapply various random augmentations to the data in each epoch for training.
To mount attacks, theclassic methods, e.g., PGD [40], MI [10], DI [56], and TI [12], are applied to all the surrogate models.
Unless otherwise specified, our attack is mounted on the first layer of ResNet-18 for lightweightsurrogate models.
Following previous works on intermediate-level attacks, the metric d in Eq.
3 isinstantiated as the mean square error. The Top-1 prediction accuracy is adopted as the evaluationmetric, i.e., lower the classification accuracy means better attack success rate.
All the experimentsare run for 5 individual trials with different random seeds.
In our experiments, target models areevaluated using ℓ ∞ -norm adversarial examples with maximum distortion ϵ = 0 . 1 , results evaluatedunder mores settings, e.g., more strict constraints ϵ = 0 .
05 , ℓ 2 -norm adversarial examples, and moretest images 5,000 images, can be found in Appendix B.5. 5.2 Main Results
Results on normally trained models.
The results of classification accuracy are shown in Table 1,which exhibits the evaluations for both lightweight and general surrogate models.
The last four rowsshow the ordinary results where the general surrogate models are trained through all data in thetraining set.
The black-box attacks are mounted based on the whole model, termed ”Deep/Deep ∗ ”,with average accuracies of around 10 .
As the size of the training data is reduced to 1 , 000 , thisconventional method obviously loses its aggressiveness.
Specifically, the averaged accuracies of3-6 rows are among 40 - 50 , revealing limited attacking capability compared to 66 .
99 of the cleanaccuracy.
It demonstrates the necessity of large amount of data for classic black-box attacks.
In rows7-10, shallow layers without ETF are used to generate adversarial examples.
The averaged accuraciesof around 25 illustrates the less dependence of shallow networks on the amount of data.
In rows
[Table]
[Table]
ϵ = 8 / 255 41.68 26.
Results on Adversarially Trained Models.
Here, weconsider the models that aretrained through adversarialtraining [40], which is morepersuasive to evaluate the effectiveness of attacking methods.
Following [40], the target modelsare trained on the whole trainingset of ImageNet.
During adversarial training, the adversarialexamples are generated underϵ = 0 , 4 / 255 , 8 / 255 , whereϵ means the constraint of ℓ ∞distance of the adversarial examples used for adversarial training relative to the original sample.
Table 2 exhibits the attacking results mounted by ETF, black-box attacks (general surrogate modelswith all training data avaliable), and no-box attacks.
For simplicity, only the results of PGD attacksare reported.
As analysed above, the black-box attacks shows the best attacking results, while ETFonly has a narrow gap with it.
However, when ϵ = 4 / 255 or 8 / 255 , ETF exceeds the black-boxattacks (29.13/26.14 vs. 48.11/38.24).
It is interesting that black-box attacks are even worse thanlightweight attacks.
We believe that exploiting the phenomenon can bring something new, but weleave it as our future work due to the limited space.
5.3 Further Analysis and Ablation Study
Intermediate-Level Perturbations.
Since the lightweight black-box attacks are implementedmainly on intermediate features, two classic attacks mounted on intermediate features, i.e. ILA [23]and AA [27], are selected for experiments.
Comparing the ILA and AA in Table 3 and ShallowMI/DI/TI/PGD in Table 1, there is little difference between their average accuracies.
It indicates thoseintermediate-level feature perturbation has little benefit to lightweight black-box attacks.
The reasonis that the intermediate-level attacks highly rely on the low-level features extracted by the lightweightmodel to generate adversarial examples and it can weaken the role of high-level information whenattacking. The proposed ETF achieves better performance. On one hand, ETF does not rely on thefeatures from high-level layers. On the other hand, ETF introduces a min-max scheme to mitigate theadverse impact induced by the approximation error from the shallow layers.
Comparison with No-box Attacks [34].
No-box attack is the first work to explore how to mountattacks in the no-box setting.
Since black-box attacks do not work in this setting, they do not attemptto design a surrogate model that approximates the target model as we do. Instead, they design thepretext task to train 20 auto-encoders per category to generate adversarial examples. However, thedifference between the pretext tasks and the target tasks may significantly reduce the transferability [Figure]
er ag
[Figure]
cc u
[Figure]
y
Number of training images
Black-box
Nobox (a)       maxpooling layer layer layer layer
FC
A cc u
[Figure] index
VGG
Inception v3 RN
[Figure]
[Table]
of adversarial examples.
In contrast, our method trains the lightweight model for tasks similar to orsame as the target tasks.
Hence, though only one sample per category is available, we achieve betterperformance than no-box attacks, as shown in Table 3.
Number of Training Samples.
One key difference between the lightweight and general black-boxattacks mainly lies in the number of samples n used for training models.
Therefore, we further explorehow lightweight attack performance varies with different values of n .
We conduct experiments onrandomly selected n images from the validation set of ImageNet.
The experimental results areshown in Figure 1(a).
The average accuracy decreases as more samples are available, and betterattacking results can be achieved thereafter.
More data provides the opportunity for lightweightblack-box attack to approximate the shallow layers of the target model more accurately.
Especiallythe lightweight black-box attacks with 10 , 000 samples can achieve the same performance as generalblack-box attacks which train the surrogate model on 1,200,000 images.
Also, our method can stillmount attacks even when only 2 or 10 images are available.
Surrogate Model. As analyzed above, we utilize the first few layers of the lightweight model toapproximate the shallow layers of the target model to mount attacks.
Hence, it is interesting to studyhow the layers selection for lightweight models impacts the attack performance.
We adopt differentlayers of ResNet- 18 to generate adversarial examples utilizing ETF and attack the different targetmodels.
The results are summarized in Fig. 1(b).
It can be seen that low-level information in the firstblock of the model is sufficient for achieving effective attack results.
With more block informationon the model, the performance of the attack does not improve.
In particular, the transferability ofadversarial examples significantly decreases when we attack the fully-connected layer, which verifiesour analysis that the high-level layer of the lightweight model cannot approximate the target modelwell. More results for architecture selection can be found in Appendix B.5. Adversarial Example Visualization.
To verify that adversarial examples are truly imperceptible,we provide visualization in Figure 2, where Deep*-PGD attack (using training images), Deep-PGDattack (using test images), and lightweight black-box attack are considered.
[Figure]
Discussion
Though the no-box threat model provides a weak assumption, many realistic scenarios may be morecomplex.
For example, adversaries have access to only data without supervision information due tosecurity [35] or privacy [46] issues.
In more extreme cases, e.g., autonomous vehicle [54], adversarieseven have no access to the test samples from the distribution over which the model is trained.
Instead,the adversaries have access to samples from a similar (but not exact) distribution (because the attackerknows the deployment environment).
Hence, we consider two more challenging attack scenarios inwhich we provide solutions to mount lightweight attacks.
Insufficient supervision information.
We design an experiment for mounting attacks using 1 , 000unlabeled samples randomly selected from the ImageNet validation set to explore the power of
ETF under this challenging setting with unlabeled data. Thanks to the framework of lightweightattack, it is feasible to train the unsupervised model on unlabeled data for attacks using Eq. 2.
Specifically, we adopt contrastive learning [6] to train a lightweight surrogate model on the smallscale unlabeled dataset and generate adversarial examples by attacking the trained lightweightsurrogate model.
The performance is shown in Table 4, where ETF still achieves similar performancewithout supervision information as that of the results under supervision.
Intuitively, we can alsoemploy other self-supervised learning method, e.g., rotation prediction, to train shallow layers, seedetails of experimental settings and results in Appendix 16.
Consideration of Out-of-distribution.
We further study the possibility for mounting attacks under amore strict scenario, where only OOD (out-of-distribution, OOD) data are available.
Specifically, weutilize the model pre-trained on OOD samples to mount lightweight black-box attacks.
The rationalitylies in that only low-level information is required to mount lightweight black-box attacks and theacquisition of low-level information.
We load pre-trained ResNet- 50 on STL- 10 [37] dataset as thesurrogate model.
Without further fine-tuning, we experiment with two test in-distribution samples,and the results are recorded in Table 4, named OOD.
As we can see, attacks mounted in the modeltrained on OOD achieve a surprising attack success rate, showing that black-box attacks enhancedwith ETF can pose practical threats to deployed models.
Conclusion
We propose a lightweight black-box attack, aiming to reveal the potential risk of black-box attacksunder the no-box threat model.
The proposal is based on the empirical observations that the modeltrained with limited samples can approximate the shallow layers of the target models within acceptableerrors, which can be used for crafting adversarial examples.
To enhance the performance, we proposeerror transformer to decrease the approximation error by transforming the approximation error inthe parameter space into the feature space.
The experiments show that the proposed ETF achieves asurprising attack success rate using only one image per category, i.e., only 3 lower than black-boxattacks with complete training data of target model.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will provide a private link when rebuttal.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No] All the experiments are run for 5 individual trials withdifferent random seeds.
Since the standard deviation is lower 1 , we report only theiraverage accuracy.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We use public datasets and give references.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] We do not utilize such data.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A ETF on Hidden Layers
Before giving the details of applying ETF to hidden layers, we revisit the key identity:
[Equation]
where w 1 stands for the first layer (convolution layer) parameters of model φ , A is a transformationmatrix used for perturbing w 1 , (cid:8) w 1 + w 1 A (cid:9) ∪ (cid:8) w \ w 1 (cid:9) means that the first layer’s parameters areperturbed and the other layers’ parameters keep the same, and g is the function parameterized withw \ w 1 used for processing the first layer’s outputs.
Built upon Eq.
7, we have φ ( x ; (cid:8) w 1 + w 1 A (cid:9) ∪ 599 (cid:8) w
[Equation]
=
[Equation]
The model φ can be decomposed: φ = g l ◦ φ l , where φ l ( x ) denotes the hidden feature at layerl .
Then the output feature at layer L is expressed as φ L ( x ; w ) =
g l ( w l φ l ( x )) , where w l is theparameter used for processing the feature φ l ( x ) .
Then, we can apply the identity at layer l :
[Equation]
where A l is the transformation matrix at layer l .
According to Eq.
8, we can transform a perturbationin the parameter space ( Aw l ) as the perturbation in the feature space ( Aφ l ( x ) ).
Thus, lightweightblack-box attacks with ETF generate adversarial examples as follows:
[Equation]
where ∆ ls ( ∆ lg ) denotes the feature space perturbation ( l = 0 means the data space), ∪ l ∆ ls ( ∪ l ∆ lg )stands for all perturbations in the feature space, and φ ( x g , ∪ l ∆ lg ; w ) ( φ ( x g , ∪ l ∆ lg ; w ) ) stands forthe output feature with perturbed features, where the feature of source (guide) image at layer l isperturbed by ∆ ls ( ∆ lg ).
Although we can perform the error transformation in the feature space, thefeatures obtained using weights with approximation errors make it challenging.
Specifically, wemerely know that the input distribution (can be seen as the feature map) is not biased, but the featuremap obtained using any weights will cause bias.
Thus, we merely apply ETF to the input layer andleave further exploitation of applying it to hidden layers as future work.
B Additional Detail DescriptionB.1 Simplified Architecture
The architecture of surrogate models is modified to avoid overfitting.
Considering the limited amountof data, we employ a network with a small model capacity to instantiate the feature extractor.
Inparticular, ResNet- 18 is simplified by reducing the number of blocks in each layer, i.e., only oneblock is used in each layer of ResNet- 18 .
B.2 Classification Ability of the Surrogate Model.
model and the general surrogate model.
Lightweight
[Table]
samples
[Table]
Test
[Table]
To demonstrate that thelightweight black-box attack performance does notrely on the generalization ofclassification, we show theclassification accuracy of thelightweight surrogate model usedfor ETF attacks in Table 5.
Tomake the conclusion clearer,we also report the performanceof a general surrogate model,which is trained on the trainingset of target models.
We cansee that the test accuracy of thelightweight surrogate model
(about 2 ) is drastically lower than that of the general model, bringing fresh air for black-box attacks.
Specifically, the common sense in black-box attacks is that mounting attacks requires a surrogatemodel, which generalizes well on the test set.
However, the experimental results in Table 5 showthat the lightweight surrogate model has poor classification performance, suggesting that thegeneralizability of surrogate models is not a necessary condition for performing black-box attacks.
B.3 Approximation to more layers.
Besides applying ETF in the first layer, we also conduct experiments that apply ETF to all layersexcept layer L .
The results are reported in Table 6.
It can be seen that applying ETF to other layerscan marginally promote the attack success rate.
The phenomenon may result from the fact that theapproximation error can accumulate with depth.
Moreover, considering that applying ETF will causemore computational overhead, we merely give the results of applying ETF to the first layer in themain paper.
[Table]
Clean
[Table]
First 14.
[Table]
[Table]
[Table]
[Table]
All
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
B
Hardware Configuration and Computation
Costs
We conduct experiments using GEFORCE RTX 2080 Ti, CPU AMD Ryzen 7 3700X @3.6 GHz.
As merely 1,000 samples are required for the training of the lightweight surrogate model, thecomputational overhead is much less than the training of general surrogate models.
B.5 Code and Instructions for Reproducibility.
For experimental reproducibility, we upload the code: https://anonymous.4open.science/r/
Error_TransFormer-7495/README.md .
C Further Experiments on ImageNet
Strict Constraint.
We conduct experiments under smaller ϵ , i.e., ϵ = 0 .
The results are given in
Table 7, demonstrating that ETF can generate powerful adversarial examples even with meeting morestrict constraints, i.e., smaller ϵ .
More Validation Images.
Besides the widely used setting on the number of samples, i.e., 1,000images, we also evaluate different methods using more samples, i.e., 5,000 images, and report theresults in Table 8.
The conclusion drawn from Table 8 is consistent with that drawn from Table 1,e.g.
, EFT outperforms “shallow” attack methods, demonstrating that ETF can generate powerfuladversarial examples under various scenarios.ℓ 2 -norm Perturbation.
We mainly conduct experiments with ℓ ∞ perturbation since it is widelyadopted in many previous works [56, 24, 12].
To further demonstrate the power of our ETF, wefurther evaluate different methods using ℓ 2 -norm perturbation.
The results are reported in Table 9,
[Table]
EFT-(PGD, MI, DI and TI) mean applying ETF combined with PGD, MI, DI or TI to the shallow layers.
Auto-attack[23] is used for testing the robustness of the target models, so it adopts the white-box setting to mount the target models.
[Table]
ETF-MI 15.
ETF-DI 13.
ETF-TI 13.
Deep*-PGD 12.
Deep*-MI 11.
Deep*-DI 7.
Deep*-TI 9.
[Table]
which further demonstrate the effectiveness of our proposal.
Considering that ℓ 1 and ℓ 0 perturbationsrequire careful design [52, 5], it is beyond the scope of this work, so we leave it as our future work.
Architecture Selection.
We further exploit whether the architecture of surrogate models havesignificant impact on the performance of ETF.
Specifically, we instantiate the shallow layers withdifferent model architectures containing ResNet [20], VGG [51], and SENet [21].
The results arereported in Table 10, demonstrating that our EFT is powerful across various model architectures.
[Table]
Clean
[Table]
SENet [21]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
VGG
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Resnet [20] 14.
[Table]
Heavy Data Augmentation.
We follow the empirical conclusion suggested in [1], where heavy dataaugmentation is vital for training appropriate shallow models.
Because appropriate shallow modelsare necessary for mounting lightweight black-box attacks, data augmentation plays a crucial role andis heavily used in our experiments.
This is supported by results shown in Table 11, where we reportthe performance of lightweight black-box attacks with and without data augmentation.
Capacity to Evade Adversarial Detectors.
We further exploit the capacity of ETF in evading adversarial example detectors.
Specifically, we employ a detection method [38, 32] to detect adversarialexamples generated by different attack methods, e.g., FGSM [18], PGD [40], BIM [11], and ETF.
All settings are the same as that used in the paper, and the results are reported in Table 12.
We cansee that ETF performs better than the baselines, i.e., having a high probability of evading detectionmethods.
[Table]
[Table]
D Results on CIFAR
We conduct the experiments on the CIFAR10 dataset, see Table 13, and evaluate the robustness ofmodels downloaded from RobustBench [7], see Table 14.
The conclusion drawn from Table 13 and
Table 14 is consistent with that drawn from Table 1 evaluating on ImageNet dataset.
E Target Model Approximation Assumption.
Taking the first layer as an example, let w 1 t and w 1 stand for the parameters of the target and surrogatemodels, respectively.
In many practical scenarios, w 1 t and w 1 usually have different dimensions,leading to intractable parameters’ discrepancy alleviation.
Fortunately, we can find an appropriatelow-rank approximation for parameters of deep neural networks [9, 61, 29].
Specifically, we canapproximate either w 1 t or w 1 to make these two matrices have the same dimensions, so we can considerthat the dimensions of the two models are the same.
Consequently, we can find a transformationmatrix A such that the approximation error is minimized, i.e., A = arg min ˜
A | w 1 t − w 1 − w 1 ˜
A | F ,where | · | F is the Frobenius norm.
In this paper, we assume the approximation error is infinitesimal,i.e., | w 1 t − w 1 − w 1 A | F = 0 .
Then, we leverage w 1 and A to represent the target model, i.e.,w 1 t = w 1 + w 1 A .
F Perturbation in Different Space.
In the no-box setting, performing the min-max strategy in the feature space is more appropriate thanthe weight space optimization [55] for the no-box threat model.
This is because we know whichperturbations are preferred in the feature space, e.g., towards features of guide images, but we haveno idea about which perturbations are preferred in the weight space, i.e., no “guide models”, which issupported by our experiments, see Table 15.
[Table]
[Table]
[Table]
G Different Self-supervised Learning Approach.
It is straightforward that exploring different strategies to train the shallow model is exciting forfurther improvement of the performance of lightweight black-box attacks, as shallow layers play animportant role in lightweight black-box attacks.
Thus, we generate adversarial examples using EFTwith shallow layers trained with a rotation prediction task [14] and report the results in Table 16.
Wecan see that shallow layers trained with the rotation prediction task is slightly worse than using thecontrastive strategy, but the performance can also reduce the model accuracy significantly.
[Table]
H Social Impact.
[Table]
The motivation of this work is to provide an approach to evaluate the adversarial robustness ina more practical scenario, the no-box setting.
Defenses can be assessed with fewer constraintsthrough lightweight black-box attacks, i.e., without accessing training samples and any queries.
Wecan develop defensive models robust against lightweight black-box attacks and attack algorithmsto mislead deployed models.
We believe the development of lightweight black-box attacks canhelp better access the robustness of deployed models and hope the proposed ETF can promote thedevelopment of corresponding defense methods.