L URING OF TRANSFERABLE ADVERSARIAL PERTURBATIONS IN THE BLACK - BOX PARADIGM
Anonymous authors
Paper under double-blind review
A BSTRACT
The growing interest for adversarial examples, i.e. maliciously modiﬁed examples which fool a classiﬁer, has resulted in many defenses intended to detect them, render them inoffensive or make the model more robust against them.
In this paper, we pave the way towards a new approach to improve the robustness of a model against black-box transfer attacks.
A removable additional neural network is included in the target model, and is designed to induce the luring effect , which tricks the adversary into choosing false directions to fool the target model.
Training the additional model is achieved thanks to a loss function acting on the logits sequence order.
Our deception-based method only needs to have access to the predictions of the target model and does not require a labeled data set.
We explain the luring effect thanks to the notion of robust and non-robust useful features and perform experiments on MNIST, SVHN and CIFAR10 to characterize and evaluate this phenomenon.
Additionally, we discuss two simple prediction schemes, and verify experimentally that our approach can be used as a defense to efﬁciently thwart an adversary using state-of-the-art attacks and allowed to perform large perturbations.
I NTRODUCTION
Neural networks based systems have been shown to be vulnerable to adversarial examples (Szegedy et al., 2014), i.e. maliciously modiﬁed inputs that fool a model at inference time.
Many directions have been explored to explain and characterize this phenomenon (Schmidt et al., 2018; Ford et al., 2019; Ilyas et al., 2019; Shafahi et al., 2019) that became a growing concern and a major brake on the deployment of Machine Learning (ML) models.
In response, many defenses have been proposed to protect the integrity of ML systems, predominantly focused on an adversary in the white-box setting (Madry et al., 2018; Zhang et al., 2019;
Cohen et al., 2019; Hendrycks et al., 2019;
Carmon et al., 2019).
In this work, we design an innovative way to limit the transferability of adversarial perturbation towards a model, opening a new direction for robustness in the realistic black-box setting (Papernot et al., 2017).
As ML-based online API are likely to become increasingly widespread, and regarding the massive deployment of edge models in a large variety of devices, several instances of a model may be deployed in systems with different environment and security properties.
Thus, the black-box paradigm needs to be extensively studied to efﬁciently protect systems in many critical domains.
Considering a target model M that a defender aims at protecting against adversarial examples, we propose a method which allows to build the model T , an augmented version of M , such that adversarial examples do not transfer from T to M .
Importantly, training T only requires to have access to M , meaning that no labeled data set is required, so that our approach can be implemented at a low cost for any already trained model.
T is built by augmenting M with an additional component
P (with T =
M ◦ P ) taking the form of a neural network trained with a speciﬁc loss function with logit-based constraints.
From the observation that transferability of adversarial perturbations between two models occurs because they rely on similar non-robust features (Ilyas et al., 2019), we design P such that (1) the augmented network exploits useful features of M and that (2) non-robust features of T and M are either different or require different perturbations to reach misclassiﬁcation towards the same class.
Our deception-based method is conceptually new as it does not aim at making M relying more on robust-features as with proactive schemes (Madry et al., 2018; Zhang et al.,
, nor tries to anticipate perturbations which directly target the non-robust features of M as with reactive strategies (Meng & Chen, 2017; Hwang et al., 2019).
Our contributions are as follows:
• We present an innovative approach to thwart transferability between two models, which we name the luring effect .
This phenomenon, as conceptually novel, opens a new direction for adversarial research.
• We propose an implementation of the luring effect which ﬁts any pre-trained model and does not require a label data set.
An additional neural network is pasted to the target model and trained with a speciﬁc loss function that acts on the logits sequence order.
• We experimentally characterize the luring effect and discuss its potentiality for black-box defense strategies on MNIST, SVHN and CIFAR10, and analyze the scalability on ImageNet (ILSVRC2012).
For reproducibility purposes, the code is available at https://anonymous.4open.
science/r/3c64e745-927d-4f51-b187-583e64586ff6/ .
L URING ADVERSARIAL PERTURBATIONS 2. N OTATIONS
We consider a classiﬁcation task where input-label pairs ( x
, y ) ∈
X × Y are sampled from a distribution D .
|Y| =
C is the cardinality of the labels space.
A neural network model M φ : X → Y , with parameters φ , classiﬁes an input x ∈ X to a label M ( x ) ∈ Y .
The pre-softmax output function of M φ (the logits) is denoted as h M : X → R C .
For the sake of readability, the model M φ is simply noted as M , except when necessary.
2. C ONTEXT : ADVERSARIAL EXAMPLES IN THE BLACK - BOX SETTING
Black-box settings are realistic use-cases since many models are deployed (in the cloud or embedded in mobile devices) within secure environments and accessible through open or restrictive API.
Contrary to the white-box paradigm where the adversary is allowed to use existing gradient-based attacks (Goodfellow et al., 2015; Carlini & Wagner, 2017; Chen et al., 2018;
Dong et al., 2018;
Madry et al., 2018; Wang et al., 2019), an attacker in a black-box setting only accesses the output label, conﬁdence scores or logits from the target model.
He can still take advantage of gradient-free methods (Uesato et al., 2018;
Guo et al., 2019;
Su et al., 2019; Brendel et al., 2018; Ilyas et al., 2018;
Chen et al., 2020) but, practically, the number of queries requires to mount the attack is prohibitive and may be ﬂagged as suspicious (Chen et al., 2019;
Li et al., 2020).
In that case, the adversary may take advantage of the transferability property (Papernot et al., 2017) by crafting adversarial examples on a substitute model and then transfering them to the target model.
2. O BJECTIVES AND DESIGN
Our objective is to ﬁnd a novel way to make models more robust against transferable black-box adversarial perturbation without expensive (and sometimes prohibitive) training cost required by many white-box defense methods.
Our main idea is based on classical deception-based approaches for network security (e.g. honeypots) and can be summarized as follow: rather than try to prevent an attack, let’s fool the attacker .
Our approach relies on a network P : X → X , pasted to the already trained target network M before the input layer, such as the resulting augmented model will answer T ( x ) =
M ◦ P ( x ) when fed with input x .
The additional component P is designed and trained to reach a twofold objective:
• Prediction neutrality: adding P does not alter the decision for a clean example x ,
i
e. T ( x ) =
M ◦ P ( x ) =
M ( x ) ; • Adversarial luring: according to an adversarial example x (cid:48) crafted to fool T , M does not output the same label as T (i.e. M ◦ P ( x (cid:48) )
(cid:54) =
M (
x (cid:48) ) ) and, in the best case, x (cid:48) is inefﬁcient (i.e. M ( x (cid:48) )
= y ).
To explain the intuition of our method, we follow the feature-based framework proposed in Ilyas et al.
(2019) where a feature f is a function from X to R , that M has learned to perform its predictions.
Considering a binary classiﬁcation task, f is said to be ρ -useful ( ρ > 0 ) if it satisﬁes
Equation 1a and is γ -useful robust if under the worst perturbation δ chosen in a predeﬁned set of allowed perturbations ∆ , f stays γ -useful under this perturbation (Equation 1b).
A ρ -useful feature f is said to be a non-robust feature if f is not robust for any γ ≥ 0 .
[Equation]
An adversary which aims at fooling a model will thus perform perturbations to the inputs to inﬂuence the useful features which are not robust with respect to the perturbation he is allowed to apply.
We denote F ∗ M the set of ρ -useful features learned by M .
We consider a set of allowed perturbations ∆ and γ > 0 such that we note F ∗ ,RM and F ∗ ,NRM respectively the set of γ -robust and non-robust features learned by M (relatively to ∆ ).
An adversary that aims at fooling M ◦ P will alter function compositions of the form f ◦ P with f ∈ F ∗ M .
These function compositions are the non-robust useful features of M ◦ P , whose set is denoted F ∗ ,NRM ◦ P .
Based on the observations that transferability of adversarial perturbations between two models occurs because these models rely on similar non-robust features (Ilyas et al., 2019), we consider f ◦ P ∈ F ∗ ,NRM ◦ P , and we derive two possibilities which ensure the lowest transferability between M ◦ P and M , regarding the robustness of f .
If f is robust, f ∈ F ∗ ,RM , that means the adversarial perturbations from ∆ is sufﬁcient to ﬂip the augmented feature f ◦ P but is not efﬁcient to directly impact f .
This case is the optimal one, since the adversarial example is unsuccessful on the target model ( M ◦ P ( x (cid:48) )
(cid:54) =
y and M ( x (cid:48) )
= y ).
On the other hand, if f ∈ F ∗ ,NRM (i.e. both f and f ◦ P are non-robust), restraining the transferability means that the additional model P impacts the way useful features vary with respect to input alterations so that the adversarial perturbation lead to two different labels.
We encompass these two cases (illustrated in Figure 1) within what we name the luring effect.
The adversary is tricked into modifying input values in some way to ﬂip useful and non-robust features of M ◦ P and these modiﬁcations are either without effect on the useful features of M , or ﬂip the non-robust features of M in a different way (and therefore are detectable, as presented in Section 4).
2. T RAINING THE LURING COMPONENT
To reach our two objectives ( prediction neutrality and adversarial luring ), we propose to train P with constraints based on the predicted labels order.
For x ∈ X , let α and β be the labels corresponding respectively to the ﬁrst and second highest conﬁdence score given to x by M .
The training of P is achieved with a new loss function that constraints α to (still) be the ﬁrst class predicted by
M ◦ P ( prediction neutrality ) and that makes the logits gap between α and β the highest as possible for M ◦ P ( adversarial luring ).
To understand the intuition behind this loss function, let’s formalize the concepts learned by M and
M ◦ P .
The prediction given by M corresponds to ”class α is predicted, class β is the second possible class”.
Once P has been trained following this loss function, the prediction given by M ◦ P corresponds to ”class α is predicted, the higher conﬁdence given to class α , the smaller conﬁdence given to class β ”.
Concepts learned by M and M ◦ P share the same goal of prediction, i.e. ”class α ” is predicted, but the relation between class α and class β is forced to be the most different as possible.
As learned concepts are essentially different, then useful features learned by M and M ◦ P to reach these concepts are necessarily different, and consequently display different types of sensitivity to the same input pixel modiﬁcations.
In other words, as the direction of conﬁdence towards classes is forced to be structurally different for M ◦ P and M , we hypothesize that useful features of the two classiﬁers should be different and behave differently to adversarial perturbations.
The luring loss , designed to induce this behavior, is given in Equation 2 and the complete training procedure is detailed in Algorithm 1.
The parameters of P are denoted by θ , x ∈ X is an input and M is the target model.
M has already been trained and its parameters are frozen during the process.
h M ( x ) and h M ◦ P ( x ) denote respectively the logits of M and M ◦ P for input x .
h Mi ( x ) and h M ◦ P
i
( x ) correspond respectively to the values of h M ( x ) and h M ◦ P ( x ) for class i .
The classes a and b correspond to the second maximum value of h M and h M ◦ P respectively.
[Equation]
The ﬁrst term of Equation 2 optimizes the gap between the logits of M ◦ P corresponding to the ﬁrst and second biggest unscaled conﬁdence score (logits) given by M (i.e. M ( x ) and a ).
This part formalizes the goal of changing the direction of conﬁdence between M ◦ P and M .
The second term is compulsory to reach a good classiﬁcation since the ﬁrst part alone does not ensure that h M ◦ P M ( x ) ( x ) is the highest logit value ( prediction neutrality ).
The parameter λ > 0 , called the luring coefﬁcient , allows to control the trade-off between ensuring good accuracy and shifting conﬁdence direction.
Input
[Table] [Table]
steps K , learning rate η , batch size B , luring coefﬁcient λ
Output luring component P parameters θ c
M ◦ P a ( x i ))
+ max (0 , h M ◦ P b ( x i ) − h M ◦ P M ( x i ) ( x i ))
P
C HARACTERIZATION OF THE LURING EFFECT 3. O BJECTIVE
Our ﬁrst experiments are dedicated to the characterization of the luring effect by (1) evaluating our objectives in term of transferability and (2) isolating it from other factors that may inﬂuence transferability.
For that purpose, we compare our approach ( Luring ) to the following approaches, which differ from ours in the training procedures and loss functions:
• Stack model : M ◦ P is retrained as a whole with the cross-entropy loss.
Stack serves as a ﬁrst baseline to measure the transferability between the two architectures of M ◦ P and M .
• Auto model : P is an auto-encoder trained separately with binary cross-entropy loss.
Auto serves as a second baseline of a component resulting in a neutral mapping from R d to R d . • C E model : P is trained with the cross-entropy loss between the conﬁdence score vectors M ◦ P ( x ) and
M ( x ) in order to mimic the decision of the target model M .
This model serves as a comparison between our loss and a loss function which does not aim at maximizing the gap between the conﬁdence scores.
We perform experiments on MNIST (Lecun et al., 1998), SVHN (Netzer et al., 2011) and CIFAR10 (Krizhevsky, 2009).
For MNIST, M has the same architecture as in Madry et al.
For SVHN and CIFAR10, we follow an architecture inspired from VGG (Simonyan & Zisserman, 2015).
Architectures and training setup for M and P are detailed in Appendix A and B. Tablein Appendix C gathers the test set accuracy and agreement rate (on the ground-truth label) between each augmented model and M .
We observe that our approach has a limited impact on the test accuracy with a relative decrease of 1.71%, 4.26% and 4.48% for MNIST, SVHN and CIFAR respectively.
3. A TTACK SETUP AND METRICS
For the characterization of the luring effect, we attack the model M ◦ P of the four approaches and transfer to M only the adversarial examples that are successful for these four models.
We deﬁne the disagreement rate , noted DR ( X (cid:48) ) , that represents the rate of successful adversarial examples crafted on M ◦ P for which
M and M ◦ P do not agree.
To measure the best case where the luring effect leads to unsuccessful adversarial examples when transferred to M , we note IAR ( X (cid:48) )
an inefﬁcient adversarial examples rate that represents the proportion of successful adversarial examples on M ◦ P but not on M .
For both metrics (see Equation 3), the higher, the better the luring effect to limit transferable attacks on the target model M .
[Equation]
We use the gradient-based attacks FGSM (Goodfellow et al., 2015), PGD (Madry et al., 2018), and
MIM (Dong et al., 2018) in its l ∞ (MIM) and l 2 (MIML2) versions.
We used three l ∞ perturbation budgets ( (cid:15) values): 0.2, 0.3 and 0.4 for MNIST, 0.03, 0.06 and 0.08 for SVHN, and 0.02, 0.03 and 0.04 for CIFAR10.
We detail the parameters used to run these attacks in Appendix D.1.
3. R ESULTS AND COMPLEMENTARY ANALYSIS
We report results for the DR and IAR in Figure 2.
The highest performances reached with our loss (for every (cid:15) ) show that our training method is efﬁcient at inducing the luring effect.
More precisely, we claim that the fact that both metrics decrease much slower as (cid:15) increases compared to the other architectures, brings additional conﬁrmation that non-robust features of M and M ◦ P tend to behave more differently than with the three other considered approaches.
As a complementary analysis, we investigate the magnitude of the adversarial perturbations.
Results show that our approach leads to similar l ∞ and l 2 distortion scales than the other methods.
This indicates that our approach truly impacts the underlying useful features and does not only imply different distortions scales needed to fool both M and M ◦ P .
The complete analysis is presented in Appendix D.2.
For MNIST, we notice that the l 0 distortion (i.e. the cardinality of the pixels impacted by the adversarial attack) is signiﬁcantly higher with our method, as presented in Figure 3 (left).
This points out that P leverages more different useful features, which are easily identiﬁable for MNIST because of the basic structure of the images.
This effect can also be demonstrated thanks to saliency maps (i.e. analyzing ∇ x P ( x ) ), as illustrated in Figure 3 (right): for the Luring model, P is sensitive to many additional useless background pixels compared to M .
Note that for Auto , modifying P ’s mapping consists almost exclusively on modifying pixels correctly correlated with the true label.
For more complex inputs from SVHN or CIFAR10, without a uniform background as for MNIST, we observe that the l 0 distortion is approximately the same for all the architectures (Figure 3 – middle).
However, for these two data sets, we analyze the inﬂuence of P ’s mapping thanks to the logits variations.
The detailed analysis in presented in Appendix D.3.
We ﬁnd that logits between
M and M ◦ P vary more differently with respect to input modiﬁcations when P is trained with our approach, which conﬁrms that our luring loss enables to reach Adversarial Luring.
U SING THE LURING EFFECT AS A DEFENSE
By fooling an adversary on the way to target a black-box model, the phenomenon that we characterized in the previous section can be seen as a defense mechanism on its own or as a complement of state-of-the-art approaches.
In this section we discuss the way to use our approach as a protection.
4. T HREAT MODEL
Attacker Our work sets in the black-box paradigm, i.e. we assume that the adversary has no access to the inner parameters and architecture neither of the target model M nor the augmented model M ◦ P .
This setting is classical when dealing with cloud-based API or edge neural networks in secure embedded systems where users have only access to input/output information with different querying abilities and output precision (i.e. scores or labels).
More precisely:
• The adversary goal is to craft (untargeted) adversarial examples on the model he has access to, i.e. the augmented model T =
M ◦ P , and rely on transferability in order to fool M .
• The adversarial knowledge corresponds to an adversary having a black-box access to a ML system S A containing the protected model T =
M ◦ P , while M stays completely hidden.
He can query T (without any limit on the number of queries) and we assume that he is able to get the logits outputs.
Moreover, for an even more strict evaluation we also
consider
stronger adversaries allowed to use SOTA transferability designed gradient-based methods to attack T (see 4.2).
• Classically, the adversarial capability is an upper bound (cid:15) of the perturbation (cid:107) x (cid:48) − x (cid:107) ∞ .
Defender For each query, the defender has access to two outputs: M ( x ) and M ◦ P ( x ) .
Therefore, two different inference schemes (illustrated in Appendix E) are viable according to the goals and characteristics of the system to defend, noted S D :
• (detection scenario)
If S D and S A are the same, the goal of the defender is to take advantage of the luring effect to detect an adversarial example by comparing the two inference outputs M ( x ) and M ◦ P ( x ) .
An appropriate metric is the rate of adversarial examples which are either detected or well-predicted by M , as expressed in Equation 4 and noted DAC for Detection Adversarial Accuracy .
[Equation]
• (transferability scenario)
If S D is a secure black-box system with limited access, that only contains the target model M , the goal of the defender is to thwart attacks crafted from S A .
For this speciﬁc scenario, the defender may only rely on the fact that the luring effect likely leads to unsuccessful adversarial examples ( M ( x (cid:48) )
= y ).
The appropriate metric is the classical adversarial accuracy ( AC ), which is the standard accuracy measured on the adversarial set X (cid:48) and noted AC MoP and AC M respectively for M ◦ P and M .
Related real-world transferability scenarios may be linked to the increasingly widespread deployment of models in a large variety of devices ( edge AI ) or services ( cloud AI ) as an integral part of complex systems with different security requirements.
For example, it is classically the case with the IoT domain where a model may be the part of a critical system and, simultaneously, be deployed and used in more mainstream connected objects with looser security requirements.
The model M is deployed for the critical system with strong security access, not accessible to an attacker, and the augmented protected model will be deployed in the others systems so that attacks that may be crafted more easily will not transfer to M .
4. A TTACKS
In order to evaluate our defense with respect to the threat model
[Table]
we attack M ◦ P with strong gradient-free attacks:
[Table]
0.86 MIM-W 0.
CIFAR
S TACK
C E
L URING (cid:15)
AC MoP
AC M
DAC
AC MoP
AC M
DAC
AC MoP
AC M
DAC SPSA 0.
0.50 ECO 0.
0.44 MIM-W 0.
[Table]
• SPSA attack (Uesato et al., 2018) ensures to consider the strongest adversary in our blackbox setting as it allows the adversary to get the logits of M ◦ P .
• Coherently with an adversary that has no querying limitation, ECO (Moon et al., 2019) is a strong score-based gradient-estimation free attack.
To perform an even more strict evaluation, and to anticipate future gradient-free attacks, we report the best results 1 obtained with the state-of-the-art transferability designed gradient-based attacks
MIM , DIM , MIM-TI and DIM-ti (Dong et al., 2018; Xie et al., 2018; Dong et al., 2019), under the name MIM-W .
The parameters used to run these attacks are presented in Appendix F.
4. R ESULTS
The results are presented in Table 1 for SVHN and CIFAR10 (results for MNIST are close to SVHN and are presented in Appendix G).
For CIFAR10, since no autoencoder we tried allows to reach a correct test set accuracy for the Auto approach, we do not consider it.
First, we notice that AC MOP almost always equal 0 with gradient-based iterative attacks 2 .
This is an indication that P does not induce a form of gradient masking (Athalye et al., 2018).
Remarkably on SVHN, for (cid:15) = 0 .
(the largest perturbation), and for the worst-case attacks, adversarial examples tuned to achieve the best transferability only reduce AC M to 0 .
48 against our approach, compared to almost 0 for the other architectures.
The robustness beneﬁts are more observable on SVHN and MNIST but the results on CIFAR10 are particularly promising in the scope of a defense scheme that only requires a pre-trained model.
Indeed, for the common l ∞ perturbation value of 0 .
03 , the worst DAC value for the C E and Luring approaches are respectively 0 .
19 (against
DIM attack) and 0 .
39 (against DIM-TI attack).
4. C OMPATIBILITY WITH I MAGE N ET AND ADVERSARIAL TRAINING
We scale our approach to ImageNet (ILSVRC2012).
For our experiments, the target model M consists of a MobileNetV2 model (Sandler et al., 2018), reaching 71 .
3% of accuracy on the validation set.
For a common l ∞ perturbation budget of 4 / 255 , the smaller AC M and DAC observed against the strong MIM-W attack with our approach equal 0 .
4 and 0 .
55 , while they equal 0 .
23 and 0 .
35 with the C E approach.
Following the results previously observed on the benchmarks used for characterization, these results validate the scalability of our approach to large-scale data sets.
Details on the ImageNet experiments and results are presented in Appendix H.
Even if very few efforts tackles the issue of thwarting adversarial perturbations transferred from a source model to a target model (transfer black-box attacks), there exists numerous detection and defense schemes intended to protect a target model in a white-box or gray-box setting against adversarial perturbations.
As the luring effect is conceptually new, it can be combined to any of these existing methods, to provide even more protection.
Indeed, our deceiving-based approach acts on the way M ◦ P performs inference relatively to M .
As the effort is focused on the design of P , another defense method, this time focusing on M and designed to protect M , can then be combined with our scheme.
We choose here to consider the combination with adversarial training (Madry et al., 2018), a state-of-the-art approach for robustness in the white-box setting.
The model M is already trained with adversarial training.
Interestingly, we note that the joint use of these defenses clearly improves the detection performance, with DAC values superior to 0.8 for the three data sets as well a strong improvement of the AC M metric for MNIST (0.97) and CIFAR10 (0.85).
The detailed experiments are presented in Appendix H.2.
D ISCUSSION AND RELATED WORK
A good practice in the ﬁeld of adversarial robustness is to consider an adaptive adversary (Carlini et al., 2019): an attacker using some knowledge of the defense method to bypass it more efﬁciently.
In our case, the only information that could be given to an adversary without breaking the black-box threat model is the supplementary knowledge that the augmented model has been trained with the luring loss.
Indeed, M has to stay hidden, otherwise the adversary could use a decision-based attack to thwart the defense (Brendel et al., 2018; Cheng et al., 2020).
Moreover, with a more permissive access to T , for example a white-box access to T , the adversary could simply extract M from the augmented model, and then defeat the defense.
The additional information of the luring loss, would not bring any advantage to the adversary in order to craft adversarial perturbations on T which transfer to M .
Indeed, it would imply for the attacker to inverse the effect of the luring loss on the prediction output vector of T , which appears as a prohibitive effort.
Therefore, respecting the threat model, an adaptive adversary would not be more efﬁcient than the adversary considered throughout this work.
Our characterization and ﬁrst results pave the way for a possible extension of the luring effect to more permissive threat models. For now, in a white-box setting, the defense scheme would be defeated since the adversary could use a gradient-based attack to fool both M and M ◦ P . Actually, the better solution to exploit the luring effect in a gray-box setting would be to cause the luring effect between two models M and M (cid:48) with different architectures. A model M (cid:48) trained with the luring loss would be publicly released.
The adversary could then be allowed to mount gradient-based attacks on M (cid:48) without being able to access M .
The possible efﬁciency of such future work is backed by results on the MIM-W attacks, which corresponds to T being attacked by gradient-based attacks.
Defenses in the black-box context are weakly covered in the literature as compared to the numerous approaches focused on white-box attacks.
More particularly, very few approaches deal with query-based black-box attacks, such as the recent BlackLight (Li et al., 2020).
To our knowledge, our work is the ﬁrst to exploit the idea of luring an adversary taking advantage of the transferability property.
However, a honeypot-based approach has been recently proposed (Shan et al., 2019) with a trapdoored model to detect adversarial examples.
Even if the threat model (white-box) is different from ours, this approach is conceptually close to our deception approach, and it is interestingly mentioned that transferability between the target and the trapdoor-protected model is very poor.
By ﬁtting the trapdoor approach in our black-box threat model we highlight complementary performances in terms of AC M and DAC metrics, meaning that an overall deception-based strategy is a promising direction for future work.
The detailed analysis is presented in Appendix I.
C ONCLUSION
We propose a conceptually innovative approach to improve the robustness of a model against transfer black-box adversarial perturbations, which basically relies on a deception strategy.
Inspired by the notion of robust and non-robust features, we derive and characterize the luring effect , which is implemented via a decoy network built upon the target model, and a loss designed to fool the adversary into targeting different non-robust features than the ones of the target model.
Importantly, this approach only relies on the logits of target model, does not require a labeled data set and therefore can by applied to any pre-trained model.
We show that our approach can be used as a defense and that a defender may exploit two prediction schemes to detect adversarial examples or enhance the adversarial robustness.
Experiments on
MNIST, SVHN, CIFAR10 and ImageNet demonstrate that exploiting the luring effect enables to successfully thwart an adversary using state-of-the-art optimized attacks even with large adversarial perturbations.
A S ETUP FOR BASE CLASSIFIERS
The architectures of the models trained on MNIST, SVHN and CIFAR10 are detailed respectively in tables 2, 3 and 4.
BN, MaxPool(u,v), UpSampling(u,v) and Conv(f,k,k) denote respectively batch normalization, max pooling with window size (u,v), upsampling with sampling factor (u,v) and 2D convolution with f ﬁlters and kernel of size (k,k).
For MNIST, we used 5 epochs, a batch size of 28 and the Adam optimizer with a learning rate of 0 . 01 .
For SVHN, we used 50 epochs, a batch size of 28 and the Adam optimizer with a learning rate of 0 . 01 .
For CIFAR10, we used 200 epochs, a batch size of 32 , and the Adam optimizer with a piecewise learning rate of 0 . 1 , 0 . 01 and 0 .
001 after respectively 80 and 120 epochs.
[Table]
[Figure]
[Table] [Table]
[Figure]
[Table] [Table]
B T RAINING SETUP FOR DEFENSE COMPONENTS
The architectures for the defense component P on MNIST, SVHN and CIFAR10 are detailed respectively in Tables 5, 6 and 7.
BN , MaxPool ( u, v ) , UpSampling ( u, v ) and Conv ( f, k, k ) denote respectively batch normalization, max pooling with window size ( u, v ) , upsampling with sampling factor ( u, v ) and 2D convolution with f ﬁlters and kernel of size ( k, k ) .
The detailed parameters used to perform training are also reported.
[Table]
• Stack Epochs: 5 .
Batch size: 28 .
Optimizer: Adam with learning rate 0 .•
Auto Epochs: 50 .
Batch size: 128 .
Optimizer: Adam with learning rate 0 .•
C E Epochs: 64 .
Batch size: 64 .
Optimizer: Adam with learning rate starting at 0 .
001 , decreasing to 0 . 0002 and 0 .
0004 respectively after 45 and 58 epochs • Luring Epochs: 64 .
Batch size: 64 .
Optimizer: Adam with learning rate starting at 0 .
001 , decreasing to 0 . 0002 and 0 .
0004 respectively after 45 and 58 epochs
[Table]
SVHN:
• Stack Epochs: 20 .
Batch size: 256 .
Optimizer: Adam with learning rate 0 .•
Auto Epochs: 5 .
Batch size: 128 .
Optimizer: Adam with learning rate 0 .•
C E Epochs: 210 .
Batch size: 256 .
Optimizer: Adam with learning rate starting at 0 . 0001 , decreasing to 0 .
00001 and 0 .
000008 respectively after 126 and 168 epochs • Luring Epochs: 210 .
Batch size: 256 .
Optimizer: Adam with learning rate starting at 0 . 0001 , decreasing to 0 .
00001 and 0 .
000008 respectively after 126 and 168 epochs.
Dropout is used
[Table]
CIFAR
[Table]
• Stack Epochs: 200 .
Batch size: 32 .
Optimizer: Adam with learning rate starting at 0 .
1 , decreasing to 0 .
01 and 0 .
001 respectively after 80 and 120 epochs • C E Epochs: 216 .
Batch size: 256 .
Optimizer: Adam with learning rate starting at 0 .
00001 , decreasing to 0 .
000005 and 0 .
0000008 respectively after 154 and 185 epochs.
Dropout is used • Luring Epochs: 216 .
Batch size: 256 .
Optimizer: Adam with learning rate starting at 0 .
00001 , decreasing to 0 .
000005 and 0 .
0000008 respectively after 154 and 185 epochs.
Dropout is used
C T EST SET ACCURACY AND AGREEMENT RATES
For the luring parameter, we set λ = 1 for MNIST and SVHN, and λ = 0 .
15 for CIFAR10.
For
CIFAR10, since no autoencoder we tried reaches correct test set accuracy, we exclude the Auto model.
[Table]
D L URING EFFECT ANALYSIS D. A TTACK PARAMETERS
For MIML2, we report results when adversarial examples are clipped to respect the threat model with regards to (cid:15) .
An illustration of a clean image and its adversarial counterpart for the maximum perturbation allowed is presented in Figure 4.
We note that the ground-truth label is still clearly recognizable.
For PGD, MIM and MIML2, the number of iterations is set to 1000 , the step size to 0 .
01 and µ to 1.0 (MIM and MIML2).
For MIML2, the l 2 bound is set to 30 on MNIST and 2 on
SVHN and CIFAR10.
D. A NALYSIS OF THE l 2 AND l ∞ DISTORTIONS
We investigate the impact of our additional component on the distance between a clean and an adversarial example, compared to the other approaches.
We measure the l 2 means of the adversarial perturbations needed to ﬁnd an adversarial example, and verify that our approach is globally at the same level than the other approaches.
For that purpose
[Table]
an appropriate and recommended attack is the Carlini & Wagner l 2 attack (hereafter, CWL2) (Carlini & Wagner, 2017) that searches for the closest adversarial examples in terms of the l 2 distortion.
We run CWL2 with the following parameters: 10 binary search steps, learning rate of 0 .
1 , initial constant of 0 . 5 and 500 iterations for MNIST, SVHN and CIFAR10.
These parameters have been chosen such that increasing the number of iterations does not lower the ﬁnal ldistortion.
For each data set, we run the attack on 1 , 000 test set examples correctly classiﬁed for the four approaches for both M and M ◦ P .
[Table]
The average l 2 and l ∞ distortions are reported in Table 9 and show that there is no signiﬁcant difference between our approach and the other approaches.
This is an additional observation that our luring loss allows to train an augmented model which causes the luring effect by predominantly targeting different useful non-robust features rather than artiﬁcially modifying the scale of the adversarial distortion needed to cause misclassiﬁcation.
D. L OGITS ANALYSIS
As a complementary analysis on the impact of the mapping induced by the additional compoment
P , we analyse how P acts on the variation of the logits with respect to the input features ( ∇ x i (cid:96) ( x ) ).
For that purpose, we ﬁrst pick 1000 test set examples correctly classiﬁed by all the models (the base model M and the augmented models of the four approaches: Auto , Stack , C E and Luring ).
Then, for each augmented model and each class c , we compute the number Ω c of pixels x d that lead to opposite effect on prediction between M and M ◦ P (see Equation 5):
[Equation]
Next, we look at the proportion of these examples for which Ω c is higher for the Luring approach than for the other approaches.
Results for each class are presented in Table 10.
For both SVHN and
CIFAR10, these proportions are strictly higher than 80% whatever the class.
This is a supplementary indication toward the fact that – with respect to the allowed adversarial perturbation altering x – our approach is more relevant than other approaches (that also perform a mapping within X ) to ﬂip useful and non-robust features of M ◦ P and M towards different labels.
[Table]
E T HREAT MODELS ILLUSTRATION
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
F A TTACK PARAMETERS
All the attacks (SPSA, ECO, MIM, DIM, MIM-TI, DIM-TI) are performed on 1000 correctly classiﬁed test set examples, with parameter values tuned for the highest transferability results.
More precisely, we searched for parameters leading to the lowest AC M and DAC values.
F. G RADIENT - FREE ATTACKS
For the SPSA attack, the learning rate is set to 0 .
1 , the number of iterations is set to 100 and the batch size to 128 .
For the ECO attack, for the three data sets, the number of queries is set to 20 , 000 .
We did not perform early-stopping as it results in less transferable adversarial examples.
The block size is set to 2 , 8 and 4 respectively for MNIST, SVHN and CIFAR10.
F. G RADIENT - BASED ATTACKS
For DIM and DIM-TI, the optimal p value was searched in { 0 . 1 , 0 . 2 , · · · , 1 . 0 } .
We obtained the optimal values of 1 . 0 , 1 . 0 and 0 .
6 respectively on MNIST, SVHN and CIFAR10.
For MIM and its variants, the number of iterations is set to 1000 , 500 and 100 respectively on MNIST, SVHN and
CIFAR10, and µ = 1 . 0 .
The optimal kernel size required by MIM-TI and DIM-TI was searched in { (3 , 3) , (5 , 5) , (10 , 10) , (15 , 15) } .
For the MIM-TI and DIM-TI, the size of the kernel resulting in the lowest AC M and DAC values reported in Section 4 are presented in tables 11, 12 andrespectively for MNIST, SVHN and CIFAR10.
[Table] [Table]
[Table]
G R ESULT FOR MNIST
[Table]
H I MAGE N ET AND ADVERSARIAL TRAINING H. R ESULTS ON I MAGE N ET
The ImageNet data set (Deng et al., 2009) is composed of color images, with 1 . 2 million images and 50 , 000 samples in the training and validation set respectively.
For our experiments, the target model
M consists of a MobileNetV2 model (Sandler et al., 2018), taking inputs of size 224 × 224 × 3 scales between − 1 and 1 .
The architecture for the defense component is described in Table 15.
We choose to present our approach along with the C E approach as it represents well the purpose of the method we present in this paper.
Indeed, the C E approach only requires training the additional component
P , which ﬁts into the scope of a defense which can be implemented on top of any already trained model M .
[Table]
For both the Luring and C E approaches, the additional component P is trained for 100 epochs
[Table]
with a bath size of 256 , using a momentum of 0 .
9 with learning rate starting at 0 .
01 , decreasing to 0 .after 80 epochs.
The λ parameter of the loss is set to 1 . 0 .
The accuracy of the augmented model as well as the agreement rate (the model M and the augmented model M ◦ P ) are presented in Table 16.
[Table]
The attacks are performed on 1 , 000 well-classiﬁed images from the validation set, and two common l ∞ perturbation budgets are considered : 4 / 255 , 5 / 255 and 6 / 255 .
Results are presented in Table 17.
The highest results observed in terms of both AC M and DAC values with our approach conﬁrm the scalability of our method on large-scale data sets.
[Table]
H. C OMPATIBILITY WITH ADVERSARIAL TRAINING
We present the performances of the luring approach when the target model M is trained with adversarial training (Madry et al., 2018) as a general state-of-the-art method for adversarial robustness.
The l ∞ bound constraint for PGD is set to 0 .
3 for MNIST and 0 .
03 for both SVHN and CIFAR10.
During training, the PGD is performed for 40 steps, and a step size of 0 . 1 for MNIST.
The PGD is performed for 10 steps with a step size of 0 .
008 for both SVHN and CIFAR10.
We report in
Table 18 the accuracy and agreement (augmented and target models agree on the ground-truth label) between the target model M and its augmented version T , whether M is trained classically or with adversarial training.
In Table 19 we report the AC M and DAC values corresponding to the same threat model as the one used throughout this paper, against the strong MIM-W attack.
Moroever, we add the accuracy in a white-box setting of the model M against the PGD attack with 40 iterations and a step size of 0 .
01 , denoted as AC M,wb .
Interestingly, we observe a productive interaction between the two defenses with DAC values that have greatly increased for the three data sets, as well as the AC M values for
MNIST and CIFAR10.
[Table] [Table]
I R ELATED WORK : COMBINING WITH TRAPDOORED MODELS
In Shan et al.
(2019), the authors introduce trapdoors to detect white-box adversarial examples attacks in order to trick the adversary’s attack into converging towards speciﬁc adversarial directions, which later enables a straightforward detection.
To this end, to defend a model against targeted adversarial examples towards label y t , the clean data set of input-label pairs ( x, y ) is augmented with trapdoored pairs ( x + ∆ , y t ) , where ∆ is a speciﬁc trigger.
Then, the target model is trained by minimizing the cross-entropy between the clean and trapdoored input-label pairs to obtain the trapdoored model .
An input x is detected as a targeted adversarial example thanks to a thresholdbased test, that compares a ” neuron activation signature ” of this input against that of the embedded trapdoors.
Note that the authors extend their approach to untargeted adversarial examples.
We refer to the article (Shan et al., 2019) for more details.
The authors mention that transferability between the target and the trapdoored model is very poor.
Based on the provided code 3 , we train trapdoored models with the architectures we used for M .
We observe that the transferability is hight from M to the trapdoor version but, on the contrary, the transferability is weak when adversarial examples are crafted on the trapdoored model and transferred to the target model.
For the latter scenario, we report in Table 20 the worst AC M and DAC values for FGSM, MIM, DIM, MIM-TI and DIM-TI along with those of our method.
The parameters used to run the attacks are the same as described in Section 4.
These results would correspond to a defense similar to ours where the public model is not the augmented model M ◦ P , but the trapdoored model trained from M .
The threat model stays the same, corresponding to an adversary which wants to transfer adversarial examples from the public model to the target model M .
Results in Table 20 on MNIST and CIFAR10 for DAC indicate that using trapdoors could allow for a more efﬁcient detection of adversarial examples, especially for large adversarial perturbations.
Results for the AC M values on the three data sets also corroborate the idea that both methods could beneﬁt from each other.
[Table]