S MALL INPUT NOISE IS ENOUGH TO DEFEND AGAINST QUERY - BASED BLACK - BOX ATTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
While deep neural networks show unprecedented performance in various tasks, the vulnerability to adversarial examples hinders their deployment in safety-critical systems.
Many studies have shown that attacks are also possible even in a blackbox setting where an adversary cannot access the target model’s internal information.
Most black-box attacks are based on queries, each of which obtains the target model’s output for an input, and many recent studies focus on reducing the number of required queries.
In this paper, we pay attention to an implicit assumption of these attacks that the target model’s output exactly corresponds to the query input.
If some randomness is introduced into the model to break this assumption, query-based attacks may have tremendous difﬁculty in both gradient estimation and local search, which are the core of their attack process.
From this motivation, we observe even a small additive input noise can neutralize most query-based attacks and name this simple yet effective approach Small Noise Defense (SND).
We analyze how SND can defend against query-based black-box attacks and demonstrate its effectiveness against eight different state-of-the-art attacks with CIFAR-10 and ImageNet datasets.
Even with strong defense ability,
SND almost maintains the original clean accuracy and computational speed.
SND is readily applicable to pre-trained models by adding only one line of code at the inference stage, so we hope that it will be used as a baseline of defense against query-based black-box attacks in the future.
I NTRODUCTION
Although deep neural networks perform well in various areas, it is now well-known that small and malicious input perturbation can cause them to malfunction (Biggio et al., 2013; Szegedy et al., 2013).
This vulnerability of AI models to adversarial examples hinders their deployment, especially in safety-critical areas.
In a white-box setting, where the target model’s parameters can be accessed, strong adversarial attacks such as Projected Gradient Descent (PGD) (Madry et al., 2018) can generate adversarial examples using the internal information.
However, recent studies have shown that adversarial examples can be generated even in a black-box setting where the model’s interior is hidden to adversaries.
These black-box attacks can be largely divided into transfer-based attacks and query-based attacks .
Transfer-based attacks take advantage of transferability that adversarial examples generated from a network can deceive other networks.
Papernot et al.
(2017) train a substitute model that mimics the behavior of the target model and show that the adversarial example created from it can successfully disturb different models.
However, due to differences in training methods and model architectures, the transferability of adversarial examples can be signiﬁcantly weakened, and thus, transfer-based attacks usually result in lower success rates (Chen et al., 2017).
For this reason, most black-box attacks are based on queries , each of which measures the target model’s output for an input.
Query-based attacks create adversarial examples through an iterative process based on either local search with repetitive small input modiﬁcations or optimization with estimated gradients of an adversary’s loss with respect to input.
However, requesting many queries in their process takes a lot of time and ﬁnancial loss.
Moreover, many similar query images can be suspicious to system administrators.
For this reason, researchers have focused on reducing the number of queries required to make a successful adversarial example.
Compared to the increasing number of studies on query-based attacks, the number of defenses against them is still very small (Bhambri et al., 2019).
Also, existing defenses developed for whitebox attacks may not be effective against query-based black-box attacks.
Dong et al.
ﬁnd that existing defenses such as ensemble adversarial training (Tramèr et al., 2018) do not effectively defend against decision-based attacks.
Therefore, it is necessary to develop new defense strategies that respond appropriately to the query-based attacks.
To defend against query-based black-box attacks, we pay attention to an implicit but important assumption of these attacks that the target model’s output exactly corresponds to the query input.
If some randomness is introduced into the model to break this assumption, they can have tremendous difﬁculty in both gradient estimation and local search, which are the core of their attack process.
This intuition is illustrated in Fig.
1a.
[Figure]
Among previous studies, Dong et al.
(2020) empirically ﬁnd that randomization-based defenses are more effective in defending against query-based black-box attacks than other types of defenses.
However, existing randomization-based defenses introduce much uncertainty into predictions, and thus, they also signiﬁcantly degrade the accuracy of clean images (clean accuracy).
In this paper, however, we highlight that simply adding small Gaussian noise into an image is enough to defeat various query-based attacks by breaking the above core assumption while almost maintaining clean accuracy.
One may think that additive Gaussian noise cannot defend against most adversarial attacks unless we introduce large randomness.
This idea is valid for white-box attacks (Gu &
Rigazio, 2014), but our experimental results show that small noise is surprisingly effective against query-based black-box attacks.
Our second intuition regarding the minimization of clean accuracy loss can be seen in Fig.
1b.
Dodge & Karam (2017) ﬁnd that the classiﬁcation accuracy decreases in proportion to the variance of Gaussian noise, but for a sufﬁciently small variance, the accuracy drop is negligible.
Considering that the robustness against additive Gaussian noise is positively correlated to the distance to the decision boundary (Gilmer et al., 2019), the above observation implies that clean images have a relatively long distance to the decision boundary.
We think an adversarial defense should have the following goals: (1) preventing malfunction of a model against various attacks, (2) minimizing the computational overhead, (3) maintaining the accuracy on clean images, and (4) easily applicable to existing models.
The proposed defense against query-based attacks meets all of the above objectives, and we name this simple yet effective defense
Small Noise Defense (SND).
Our contributions can be listed as follows:
• We highlight the effectiveness of adding a small additive noise to input in defending against query-based black-box attacks.
The proposed defense, SND, can be readily applied to pretrained models by adding only one line of code in the Pytorch framework (Paszke et al., 2019) at the inference stage ( x = x + sigma * torch.randn_like(x) ) and almost maintains the performance of the model.
• We analyze how SND can efﬁciently interfere with gradient estimation and local search, which are the core of query-based attacks.
• We explain the difﬁculty of evading SND.
We devise an adaptive attack against SND and explain its limitations in terms of query-efﬁciency.
• We have shown that the proposed method can effectively defend against eight different stateof-the-art query-based black-box attacks with the CIFAR-10 and ImageNet datasets.
Speciﬁcally, we experimented with four decision-based and three score-based attacks in order to show strong defense ability against various attacks, including local search-based and optimizationbased methods.
B ACKGROUND
Adversarial setting.
In this paper, we will deal with adversarial attacks on the image classiﬁcation task.
Suppose that a neural network f ( x ) classiﬁes an image x among total N classes and returns a class-wise probability vector y =
[ y 1 , ..., y N ] for x .
For notational convenience, we also denote the probability of i th class (i.e., y i ) as f ( x ) i and the top-1 class index as h ( x ) =
arg max i ∈
C
y i , where
C = { 1 , ..., N } .
In a black-box threat model, an adversary has a clean image x 0 whose class index is c 0 and wants to generate an adversarial example ˆ x = x 0 + δ to fool a target model f .
In the following, we denote the adversarial example at t th step in an iterative attack algorithm as ˆ x t .
The adversary should generate an adversarial example within a perturbation norm budget (cid:15) and query budget Q .
If we let q be the number of queries used to make δ , then we can write the adversary’s objective as follows:
[Equation]
where (cid:96) (ˆ x ) =
f
(ˆ x ) c 0 − max c (cid:54) =
c 0 f (ˆ x ) c for untargeted attacks and (cid:96) (ˆ x ) = max c (cid:54) =
ˆ c f
(ˆ x ) c − f (ˆ x ) ˆ c for targeted attacks with target class index ˆ c .
Unless otherwise noted, in this paper, we use p =and focus on untargeted attacks because it is more challenging for defenders.
Besides, we assume that each pixel value is normalized into [0 , 1] .
Taxonomy of query-based black-box attacks.
Query-based attacks can be largely divided into score-based and decision-based according to the available type of the output of the target model (class-wise probabilities for score-based attacks and the top-1 class index for decision-based attacks).
On the other hand, query-based attacks can be categorized into optimization-based attacks and local search-based attacks.
Optimization-based methods optimize an adversary’s objective loss with estimated gradients of the loss with respect to ˆ x t .
In contrast, local search-based attacks repeatedly update an image according to how the model’s output changes after adding a small perturbation.
In the following, we brieﬂy introduce various query-based attacks used in this paper.
Bandit optimization with priors (Bandit-TD).
Ilyas et al.
(2018) observe that the image gradients in successive steps of an iterative attack have strong correlation.
In addition, they ﬁnd that the gradients of surrounding pixels also have strong correlation.
Bandit-TD exploits this information as priors for efﬁcient gradient estimation.
Simple Black-box Attack (SimBA & SimBA-DCT).
For each iteration, SimBA (Guo et al., 2019a) samples a vector q from a pre-deﬁned set Q and modify the current image ˆ x t with ˆ x t − q and ˆ x t + q and updates the image in the direction of decreasing y c 0 .
Inspired by the observation that lowfrequency components make a major contribution to misclassiﬁcation (Guo et al., 2018), SimBADCT exploits DCT basis in low-frequency components for query-efﬁciency.
Boundary Attack (BA).
BA (Brendel et al., 2018) updates ˆ x t on the decision-boundary so that the perturbation norm gradually decreases via random walks while misclassiﬁcation is maintained.
Sign-OPT.
Cheng et al.
(2019a) treat a decision-based attack as a continuous optimization problem of the nearest distance to the decision boundary.
They use the randomized gradient-free method (Nesterov & Spokoiny, 2017) for estimating the gradient of the distance.
Cheng et al.
(2019b)
propose SIGN-OPT which uses the expectation of the sign of gradient with random directions to efﬁciently estimate the gradients without exhaustive binary searches.
Hop Skip Jump Attack (HSJA).
Chen et al.
(2020) improve Boundary Attack with gradient estimation.
For each iteration of HSJA, it ﬁnds an image on the boundary with a binary search algorithm, and estimates the gradients, and calculates the step-size towards the decision boundary.
GeoDA.
Rahmati et al.
(2020) propose a geometry-based attack that exploits a geometric prior that the decision boundary of the neural network has a small curvature on average near data samples.
By linearizing the decision boundary in the vicinity of samples, it can efﬁciently estimate the normal vector of the boundary, which helps to reduce the number of required queries for the adversarial attack.
2. A DVERSARIAL D EFENSES
As Dong et al.
(2020) observe that randomization is important for effective defense against querybased attacks, we focus on randomization-based defenses among various defense methods.
In what follows, we brieﬂy explain three different randomization-based defenses along with PGDadversarial training.
Random Self-Ensemble (RSE).
RSE (Liu et al., 2018) adds Gaussian noise with σ inner = 0 .
1 to the input of each convolutional layer, except for the ﬁrst convolutional layer where σ init = 0 .
2 is used.
To stabilize the performance, they use an ensemble of multiple predictions for each image.
Parametric Noise Injection (PNI).
He et al.
(2019) propose a method to increase the robustness of neural networks by adding trainable Gaussian noise to activation or weight of each layer.
They introduce learnable scale factors of noise and allow them to be learned with adversarial training.
Random Resizing and Padding (R&P).
Xie et al.
(2018) propose a random input transform-based method.
In front of network inference, it applies random resizing and random padding to its input sequentially, making adversaries obtain noisy gradients.
It can be easily applied to a pre-trained model, but it increases total computational time due to the enlarged input image.
PGD-Adversarial Training (PGD-AT).
Madry et al.
(2018) propose PGD-adversarial training which trains a model with adversarial examples generated by PGD attack.
Unlike other defenses that are ineffective against adaptive attacks, it is well known that PGD-AT provides excellent defense against a variety of white-box attacks.
A NALYSIS 3. O UR APPROACH
To defend against query-based black-box attacks, we add Gaussian noise with a sufﬁciently small σ to the input as follows.
[Equation]
, where
[Equation]
(cid:28)
[Equation]
For an adversary, since the exact value of η is unknown, there can be multiple output values for any x , so f η is a random process.
In what follows, we explain how this transform introduces tremendous difﬁculty in both gradient estimation and local search in query-based black-box attacks.
3. D EFENSE A GAINST O PTIMIZATION - BASED A TTACKS
In this subsection, we will explain how small Gaussian input noise can disturb the gradient estimation in optimization-based attacks.
We ﬁrst look at defense against score-based attacks and then deal with decision-based attacks.
The core of optimization-based attacks is an accurate estimation of ∇ (cid:96) ( x ) , which needs to be approximated with ﬁnite difference because of the black-box setting.
For instance, the gradient can be estimated as ˜ g by Random Gradient-Free method (Nesterov & Spokoiny, 2017) as follows.
[Equation]
g i , where
[Equation]
Conceptually, by introducing small Gaussian noise into input, ˜ g can differ greatly from the true gradient ∇ (cid:96) as shown in Fig.
To illustrate it more formally, let us represent η by replacing it with η ( x ) to clarify η depends on both time and x .
Suppose f ∗ η ( x ) is a sample function of the random process
f η ( x ) ( x ) at some time.
Then, this function is very noisy because of η ( x ) with regard to x .
We also assume that (cid:96) ∗ is derived from f ∗ η ( x ) , then unless Var[ (cid:96) ∗ ( x + u )] is near zero, (cid:96) ∗ is discontinuous and nondifferentible and thus, ∇ (cid:96) ∗ does not exist.
Therefore, the gradient estimation using ﬁnite differences does not converge to the target gradient ∇ (cid:96) .
In decision-based attacks, ˆ x t is likely to be in the vicinity of the decision boundary.
Therefore even the small noise can move ˆ x t across the boundary so that the output is changed.
The estimated gradient through frequently erroneous predictions hinders the generation of adversarial examples.
We illustrate the above defense mechanism in Appendix A.
On the other hand, the error of the binary search algorithm, which is widely used to calculate the distance to the decision boundary, can be ampliﬁed due to η .
Therefore, algorithms such as HSJA, which assume that ˆ x is near the decision boundary, are likely to work incorrectly.
3. D EFENSE A GAINST L OCAL S EARCH - BASED A TTACKS
Almost all local search-based attacks do not take into account the uncertainty of the model output.
Suppose an adversary recognizes that the attack objective loss decreases for x + τ where τ is a perturbation, and updates x as x + τ .
However, since the actual evaluated input of f is x + τ + η where η ∼ N ( 0 , σ 2 I ) , the attack objective loss might be rather increases at the originally intended input x + τ .
This prediction error makes the attack algorithm stuck in the iterative process and prevents generating adversarial examples.
3. D IFFICULTY OF E VADING S MALL N OISE D EFENSE
Although the input of the function, x + η , is a Gaussian random process, but f ( x + η ) , the result of the nonlinear function f , is no longer a Gaussian random process.
This makes it very difﬁcult for query-based attacks to bypass SND.
One can approximate f ( x ) as E η [ f η ( x )] by taking the expectation through multiple queries using the fact that E ( η ) = 0 .
However, this attempt requires many queries for each iteration and greatly diminishes query efﬁciency.
In addition, even if a large amount of queries are used, E η [ f η ( x )] may be different from f ( x ) because of the nonlinearity of the deep neural networks.
With a simple example, we explain how the expectation value differs from the actual value when Gaussian noise is added to input of nonlinear functions.
Let F ( x ) =
x T x where F : R d → R , and F η ( x ) =
F ( x + η ) where η ∼ N ( 0 , σ 2 I ) .
Suppose we want to estimate F ( 0 ) with F η ( 0 ) .
Then, E [ F η ( 0 )] =
E [( 0 + η ) T ( 0 + η )]
=
E [ η T η ] = dσ 2 .
Therefore, E [ F η ( 0 )]
= dσ 2 (cid:54) =
= F ( 0 ) and if d is very large (e.g., for an image of size 224 × 224 × 3 , d =150 , 528 ), then the estimation error would be high.
Additionally, in the case of a simple network consisting of an afﬁne layer and ReLU activation,
F ( x ) = ReLU ( Wx + b ) and F : R −→ R .
Since F ( x ) is max (0 ,
Wx + b ) and E
[ F η ( x )] is Wx + b − Φ( − Wx + b | W | σ ) + | W | σφ ( − Wx + b | W | σ ) , the estimation error would exist (The detailed proof is in Appendix E). From the proof on the simple network, we can expect that the average of the output may have an error with the actual output even in a deep neural network.  EXPERIMENTS AND DISCUSSION 4. E XPERIMENTAL S ETTINGS
In this section, we evaluate the defense ability of SND against eight different query-based black-box attacks: BA, Sign-OPT, HSJA, GeoDA, SimBA, SimBA-DCT, Bandit-TD, Subspace Attack along with other defense methods: PNI, RSE, R&P, PGD-AT.
We use the CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) datasets for our experiments and following previous studies (Chen et al., 2020;
Guo et al., 2019a;
He et al., 2019), we use ResNet-20 for CIFAR-10 and ResNet50 (He et al., 2016) for ImageNet for target networks.
Following Brendel et al.
(2018), we randomly sampled 1,000 and 250 correctly classiﬁed images from the CIFAR-10 test set and the ImageNet
Validation set for evaluation.
We describe a detailed experimental setting in Appendix B.
For evaluation metrics, we ﬁrst deﬁne a successfully attacked image as an image from which an attack can ﬁnd an adversarial image within the perturbation budget (cid:15) and query budget Q .
With this deﬁnition, we use attack success rate , which is the percentage of the number of successfully attacked images over the total number of evaluated images.
Note that since we evaluate defense performance, a lower attack success rate is better .
We use (cid:15) = 1 . 0 for the CIFAR-10 dataset, and (cid:15)
= 5 . 0 for the ImageNet dataset.
In addition, we denote the q th query image as ˆ x q .
Note that ˆ x q and ˆ x t can be different.
4. E XPERIMENTAL R ESULTS
Evaluation of clean accuracy.
We ﬁrst evaluate the clean accuracy of models with defenses on the original test split (10k images) of the CIFAR-10 and validation split (50k images) of the ImageNet dataset.
As shown in Table 1, SND hardly reduces the clean accuracy compared to other methods.
The accuracy drop caused by SND is not signiﬁcant at σ ≤ 0 . 01 and becomes large at σ = 0 .
02 , which implies that sufﬁciently small σ is important for maintaining clean accuracy.
[Table]
Evaluation on the CIFAR-10 dataset.
We performed four different decision-based attacks against models with defenses
[Table]
and Table 2 shows the evaluated attack success rates.
SND shows competitive defense ability despite having more than 5% higher clean accuracy compared to other defenses.
Moreover, due to signiﬁcant performance drop, RSE and PNI cannot be applied to the models for large-scale image classiﬁcation with the ImageNet dataset [Table]
[Table]
Evaluation on the ImageNet dataset.
We performed six different query-based attacks against models with defenses, and Table 3 shows the evaluated attack success rates.
When the query budget Q is 20k, the average of the attack success rates over the attacks against the baseline is 87.9%, whereas
SND with σ = 0 .
01 signiﬁcantly reduces it to 10.0%.
SND with σ = 0 .
001 also signiﬁcantly reduces the average attack success rate to 29.5%, which is comparable to the second-best method, R&P (22.5%).
We also calculate the average (cid:96) 2 norm of adversarial perturbations || x 0 − ˆ x q || 2 at the predeﬁned query budget Q to show whether the perturbation norm diverges or not.
If an attack stops in the middle without requesting all Q queries, we use the last query image instead.
In decision-based attacks, it can be seen that randomization-based defenses, SND and R&P, signiﬁcantly increase the perturbation norm as q increases.
In SimBA and SimBA-DCT, the perturbation norm is minimal in
SND and R&P, which implies that the attacks have signiﬁcant difﬁculty in ﬁnding a perturbation which decreases y c 0 . [Table]
Empirical evidence for assumptions of SND.
To provide the supporting evidence for assumptions of SND in score-based attacks, we calculate ˆ σ = (cid:112) Var[ (cid:96) ( x + η )] where η ∼ N ( 0 , σ 2 I ) .
With σ = 0 .
and 1,000 test images of the CIFAR-10, we evaluate (cid:96) ( x + η ) for 100 iterations for each clean image and calculate the ˆ σ averaged over all images.
In our experiment, ˆ σ = 0 . 04 which is small but sufﬁcient to make (cid:96) ( x + η ) to be non-differentiable about x .
In decision-based attacks, our assumption in Section 3.2 is that if ˆ x is near the decision boundary, sufﬁciently small noise can easily move the image across the boundary.
We evaluate P mis :=
P ( h ( x ) (cid:54) =
h ( x + η )) through experiments.
We count the above mismatch case for all queries during the attack process.
With σ = 0 .
001 , 0 . 01 and the CIFAR-10 test images, the average P mis for all attacks is calculated as 0.22 and 0.25, respectively.
In constrast, on clean images, P mis is obtained as 0.002 and 0.021 respectively.
Therefore, the results shown in Table 5 support our argument.
Evaluation of an adaptive attack against SND.
As described in Section 3.4, we devise an adaptive attack against SND that takes the expectation of predictions for repetitive T queries.
In this experiment, we perform HSJA against SND with σ = 0 .
01 on the CIFAR-10 dataset.
Since HSJA is a decision-based attack, we regard the most predicted class in T queries as the expected class.
We measure the attack success rate and P mis according to the query budget, and the adaptive attack clearly shows a higher attack success rate than the baseline ( T = 1 ), as shown in Table 4.
On the same query budget, however, the adaptive attack shows a lower attack success rate (e.g., 22.7% ( T =1 ) >
18.2% ( T =5 ) at Q =10k and 29.3% ( T =5 ) > 27.3% ( T =10 ) at Q =50k).
Therefore, the expectation-based adaptive attack has limitations due to the restricted query budget.
Moreover, even if T increases, the P mis does not decrease and this reinforces our argument in Section 3.4.
We also apply the adaptive attack with T =10 to BA, SO, and GeoDA for comparison.
The results are shown in Table 6.
Varying σ for each inference.
So far, we have used a ﬁxed σ for SND.
Changing σ for each query may reduce clean accuracy while maintaining the defense ability.
From this motivation, we multiply η with k which is randomly determined between 0 and 1 using the beta distribution with three
[Table]
[Table] [Table] different ways: (1) Uniformly random (the same as α = β =1 ) (2) Sampling from a beta distribution with α = β {+=
+} whose probability density function (PDF) is ∩ -shaped.
(3) Sampling from a beta distribution with α
= β =0 .
5 whose PDF is ∪ -shaped.
We calculate clean accuracy and average (cid:96)norm of perturbations for each method.
Among the three ways, α = β =2 is better than the others, but SND with ﬁxed σ =0 .
01 is better than SND with variable σ , which implies large randomness in each query is crucial for a strong defense.
Detailed results can be found in Appendix C.
Defense against hybrid black-box attack.
Recently proposed Subspace attack (Guo et al., 2019b) exploits transferability-based priors, gradients from local substitute models trained on a small proxy dataset.
We use pre-trained ResNet-18 and ResNet-34 (He et al., 2016) as reference models for gradient priors.
For this attack, we perform an attack based on (cid:96) ∞ norm because the authors provide parameter settings only for (cid:96) ∞ norm.
Detailed results are described in Appendix D, but the result shows that SND alone cannot effectively defend against the hybrid attack with gradient priors.
However, when SND is combined with PGD-AT, it effectively protects the model and decreases the attack success rate from 100% to 42.4% at Q =20k and σ =0 .
To focus on the defensive ability against gradient estimation, we recalculate the attack success rate without initially misclassiﬁed images.
Then, the newly obtained attack success rate decreases from 100% to 16.4% at Q =20k.
This result implies that SND can be combined with other defenses against transfer-based attacks to achieve strong defense ability against all types of black-box attacks.
R ELATED WORK
History-based detection methods against query-based black-box attacks.
To the best of our knowledge, studies that mainly target defending against query-based black-box attacks have not yet been published.
However, history-based detection techniques for query-based attacks have been proposed recently (Chen et al., 2019; Li et al., 2020).
Considering that adversary requires many queries of similar images for ﬁnding an adversarial example, they store information about past query images to detect the unusual behavior of query-based attacks.
Certiﬁed defense with additive Gaussian noise. Li et al. (2019) analyze the connection between robustness of models against additive Gaussian noise and adversarial perturbations. They derive the certiﬁed bounds on the norm bounded adversarial perturbation and they propose a new training strategy to improve the certiﬁed robustness.
Simlilarly, randomized smoothing (Cohen et al., 2019) creates a smoothed classiﬁer that correctly classiﬁes when Gaussian noise is added to the classiﬁer’s input.
Cohen et al.
(2019) prove that this smoothed classiﬁer can have (cid:96) 2 certiﬁed robustness for an input.
Both SND and the above certiﬁed defenses add Gaussian noise to the input.
However, the purpose of the addition of noise in the certiﬁed defenses is to induce the classiﬁer to gain certiﬁed robustness.
Whereas, SND adds noise to disturb an accurate measurement of the output to defend against query-based black-box attacks at the inference.
In addition, the certiﬁed defenses use a much larger σ ( ≥ 0 . 25 ) than SND ( 0 . 01 ).
C ONCLUSION
In this paper, we highlight that even a small additive input noise can effectively neutralize querybased black-box attacks and name this approach Small Noise Defense (SND).
We demonstrate its effectiveness against eight different query-based attacks with CIFAR-10 and ImageNet datasets.
Our work suggests that query-based black-box attacks should consider the randomness of the target network as well.
SND is readily applicable to pre-trained models by adding only one code line without any structural changes and ﬁne-tuning.
Due to its simplicity and effectiveness, we hope that
SND will be used as a baseline of defense against query-based black-box attacks in the future.
We can list various exemplary future work as follows: (1) Finding a better type of noise.
Other types of noise, such as salt-and-pepper or image processing like random contrast, can be used instead of
Gaussian noise as long as it can disrupt query-based attacks while maintaining clean accuracy; (2)
Devising effective black-box attacks against SND while maintaining query efﬁciency.
The uncertainty of randomized predictions can be mitigated through expectation, but it requires many queries and hurts query-efﬁciency; (3) Rigorous proof of the defense ability of small input noise.
We explain how SND can defend against query-based attacks, but one can establish theorems like the lower bound of mean squared error of estimated gradient with small input noise.
A I LLUSTRATION OF DEFENSE AGAINST DECISION - BASED ATTACK WITH SMALL INPUT NOISE
B D ETAILED EXPERIMENTAL SETTING B. S ETTINGS OF ATTACK METHODS
BA: We adopt BA
provided
by Adversarial Robustness Toolbox ART) library (Nicolae et al., 2018) with default parameters.
Sign-OPT: We port the code 1 provided by the authors into our framework without changing the special parameters of the attack.
HSJA:
We adopt HSJA provided by Adversarial Robustness Toolbox (ART) library with default parameters except for increasing the maximum number of iterations to 64 to follow the authors’ code 2 .
GeoDA: We port the code 3 provided by the authors into our framework without changing the special parameters of the attack.
SimBA & SimBA-DCT:
We port the code 4 provided by the authors into our framework.
Following the authors, we use freq_dims =28, order =strided, and stride = for SimBA-DCT.
Bandit-TD: We port the code 5 provided by the authors into our framework with default parameters except for batch_size =1 and epsilon =4.9.
Subspace Attack: We port the code 6 provided by the authors into our framework with the original setting for (cid:96) ∞ norm untargeted attack for the ImageNet.
We use pre-trained ResNet-18 and ResNet34 trained on the imagenetv2-val dataset as reference models that are provided by the authors.
https://github.com/cmhcbb/attackbox
https://github.com/Jianbo-Lab/HSJA
https://github.com/thisisalirah/GeoDA
https://github.com/cg563/simple-blackbox-attack
https:
B. SETTINGS OF DEFENSE METHODS
Baseline: We train a ResNet-20 model on the CIFAR-10 dataset for 200 epochs and use this model for our experiments.
For the ImageNet dataset, we use the pre-trained ResNet-50 model provided by the Pytorch library.
PNI: We use the pre-trained ResNet-20 model with PNI-W (channel-wise) provided by the authors trained on the CIFAR-10 dataset.
PGD-AT:
We use the adversarilly trained ResNet-50 model for (cid:96) 2 norm with (cid:15) train = 3 provided from robustness library (Engstrom et al., 2019) with PGD on the ImageNet dataset for comparison.
RSE: We train a RSE-based ResNet-20 with σ init = 0 .
2 and σ inner = 0 .
Considering computational efﬁciency, we use 5 ensembles for each prediction of RSE.
R&P: R&P applies random resizing and random padding to its input sequentially.
It ﬁrst rescales the input image of size W × H × 3 with a scale factor s which is sampled from [ s min , s max ] , and places it in a random position within an empty image of size s max W × s max Y × 3 .
Following the authors, we set s min and s max as 310299 and 331299 respectively.
C E VALUATION OF VARYING σ FOR EACH INFERENCE
[Table]
D E VALUATION OF ATTACK SUCCESS RATES OF S UBSPACE A TTACK
[Table]
E T HE PROOF OF THE EXPECTATION ERROR ON A SIMPLE NETWORK
Let F ( x ) =
ReLU ( Wx + b ) where F : R −→ R and ReLU ( x ) = max (0 , x ) , and F η ( x ) =
F ( x + η ) , where η ∼ N (0 , σ 2 ) .
Suppose we want to estimate F ( x ) with E [ F η ( x )] .
Let W ( x + η ) + b be Y , then Y can be represented with µ y =
Wx + b and σ 2 y =
W 2 σ 2 as:
[Equation]
y
[Equation]
2 y
[Equation]
[Equation]
Then, F η ( x ) is max (0 , Y ) and E
[ F η ( x )] =
E [ max (0 , Y )] can be obtained by the law of total expectation.
[Equation]
Using the truncated normal distribution, we recall the fact as follows:
[Equation]
where φ ( x ) = 1 √ 2 π exp ( − 1 2 x 2 ) and Φ is the cumulative distribution function of the standard normal distribution.
Since Pr( Y > 0))
= 1 − Φ( µ y σ y ) , E [ F η ( x )] can be represented as:
[Equation]