Ensemble-based Adversarial Defense Using Diversified Distance Mapping
Anonymous authors
Paper under double-blind review
Abstract
We propose an ensemble-based defense against adversarial examples using distance map layers (DMLs).
Similar to fully connected layers, DMLs can be used to output logits for a multi-class classiﬁcation model.
We show in this paper how DMLs can be deployed to prevent transferability of attacks across ensemble members by adapting pairwise (almost) orthogonal covariance matrices.
We also illustrate how DMLs provide an eﬃcient way to regularize the Lipschitz constant of the ensemble’s member models, which further boosts the resulting robustness.
Through empirical evaluations across multiple datasets and attack models, we demonstrate that the ensembles based on DMLs can achieve high benign accuracy while exhibiting robustness against adversarial attacks using multiple white-box techniques along with AutoAttack.
Introduction
Ongoing research has provided defenses against adversarial examples, which are crafted from correctly classiﬁed inputs with imperceptible perturbations.
Despite the success of ensemble learning as a mechanism for reducing prediction errors and improving generalization by combining predictions of multiple models performing the same task (Russakovsky et al., 2015; Sagi & Rokach), early research has shown the ineﬀectiveness of mutiple ensemble-based defenses against adversarial examples and even has gone further to suggest that ensembles are only as robust as their weak components (He et al., 2017).
If this claim is true, it basically defeats the purpose of using an ensemble, which is building a strong model out of weaker ones.
Nevertheless, research continued to investigate the usage of ensembles as a defense mechanism (Pang et al., 2019; Verma & Swami;
Sen et al.).
However, recent attempts have been quickly shown ineﬀective (Tramèr et al.
; Croce &
Hein, 2020).
We believe that one of the primary reasons for the weakness of ensemble defenses is the inter-model attack transferability.
It was shown that even fundamentally diﬀerent models could exhibit high attack transferability rate (Papernot et al., 2017; Kurakin et al., 2018).
This phenomenon hindered the consideration of ensemble learning as a strong defense mechanism on its own.
The reason is that if the member models are not robust, and they exhibit high attack transferability rate, attacks generated from one model can attack the rest, and hence, attack the entire ensemble.
In this work, we show that it is possible to circumvent the problem above and instill diversity among ensemble members via the employment of specially initialized and optimized distance-map-layers (DMLs), and hence throttle the inter-model attack transferability.
Moreover, we demonstrate that
DMLs provide spontaneous regularization of the Lipschitz constant, and therefore further boost the robustness.
The rest of this paper is organized as follows: We ﬁrst discuss an overview of relevant recent works on ensemble defenses against adversarial examples and background information.
Then we introduce a distance map layer based on Mahalanobis distance, and also explain the threat models considered in this work.
We describe the creation of ensemble of DML-based individual models.
Afterwards, we introduce a randomized version of DML.
Finally, we evaluate the robustness of our ensemble model over MNIST, CIFAR-10 and RESISC-45 datasets.
Background and related work
Not long ago, researchers revealed the vulnerability of machine learning models, particularly deep neural networks (DNNs), against adversarial examples (Szegedy et al., 2014).
These models can provide incorrect predictions on examples that are slightly perturbed from correctly classiﬁed ones.
The process of generating adversarial examples from natural ones is called adversarial evasion attacks (Biggio et al.).
Evasion attacks can be categorized into black-box, gray-box, and white-box attacks.
In the black-box setting, the attacker does not have access to the model’s parameters and the potential defense of the model (Papernot et al., 2017; Brendel et al., 2018).
In the gray-box setting, although the attacker does not have access to the model’s parameter, it is aware of the defense applied in the model.
In the white-box evasion, the attacker has access to both the parameters and the defense applied in the model.
Therefore, the attacker can apply gradient-based attack techniques (Goodfellow et al., 2014; Carlini & Wagner, 2017; Madry et al., 2018), or it can even design a custom made adaptive attack based on its knowledge about the model and the defense (Tramèr et al.).
From the defense perspective, white-box attacks are certainly the most challenging type of evasion attacks.
Our proposed defense targets white-box attacks.
Speciﬁcally, we focus on 𝐿 𝑝 restricted perturbations of images, and, as it is typical in many research works, we consider the 𝐿 ∞ perturbations.
Since the discovery of adversarial attacks, a signiﬁcant research activity has been devoted to developing appropriate defenses (Madry et al., 2018; Zhang et al.).
However, most of the proposed defenses have been soon defeated (Tramèr et al.
; Croce & Hein, 2020).
An intuitive mechanism to defend against adversarial examples is training ensembles of other models in order to enhance their defenses (Pang et al., 2019; Verma & Swami; Sen et al.).
However, the eﬃcacy of existing ensemble approaches is often faced with skepticism due to the transferability phenomenon (Papernot et al., 2017), which makes the ensemble model believed to be at most as robust as its strongest constituent model (He et al., 2017).
In this paper, we promote the diversity among individual networks from a new perspective via distance map layers.
Our construction is based on the assumption that defending against transferred adversarial example is eﬀective, even if the individual models are vulnerable to direct attacks.
Our approach is orthogonal to the previous approaches and can be combined with other generic weak or strong defenses to further enhance the ensemble’s resistance to adversaries.
Although there is no strict illustration on what is intuitively deﬁned as diversity, in this work, we deﬁne the diversity as the highest dissimilarity between the learned features of diﬀerent classes across the ensemble members.
Diversity is greater when the prediction errors of individual members are highly uncorrelated (Liu & Yao, 1999a;b; Dietterich, 2000; Liu et al., 2019).
This property may lead an adversarial perturbation to fail to fool the majority of networks in the ensemble.
Topologically, diversity can be depicted as the variability in the shapes of the decision boundaries and inter-class neighborhood relationships in the embedding space.
Changing the shape of the decision boundaries in the embedding space implies diﬀerent loss shapes.
With diversity in the shape of the losses corresponding to diﬀerent models, the gradient from one model would not be a good approximation (transferable) to the other model.
Note that due to the high capacity of DNNs, changing the topology of the embedding space can have no or negligible eﬀect on the model’s accuracy.
Using this perspective on model diversity, we will illustrate that DML-based ensembles could be designed to achieve the requirement of diversity in prediction errors through injecting dissimilarities only between the DMLs of ensemble members.
Previous works increase the diversity over training data via promoting the diversity of prediction errors of ensemble members (Liu & Yao, 1999a;b; Liu et al., 2019).
Technical approach
In this section, we elaborate on our model with distance map layer (DML).
We start by recalling the deﬁnition of Mahalanobis distance between two point 𝑥 1 , 𝑥 2 ∈ 𝑅 𝑛
[Equation]
where 𝑀 is the inverse of the covariance matrix (referred to as the 𝑀 matrix for the rest of the paper), which is a positive semi-deﬁnite matrix.
Lemma 1.
The Mahalanobis distance is 𝑘 -Lipschitz continuous with 𝑘 = √ 2 (cid:107) 𝑃 (cid:107) 2 , where 𝑀 =
𝑃 𝑇 𝑃 , and 𝑃 is a triangular matrix.
A proof for this lemma can be found in (Zantedeschi et al., 2016).
We now introduce a mapping function, based on the Mahalanobis distance, that can be used as a diﬀerentiable layer in a neural network.
The function maps an input vector to an output vector by calculating Mahalanobis distances between the input and the learned centers of the function.
Equation 2 shows the element of the output vector corresponding to center 𝑐 for an input vector 𝑥 .
We call this function distance map layer (DML) .
[Equation]
In this formula, 𝑀 ∈ R 𝑁 × 𝑁 and 𝑐 ∈ R 𝑁 are the learnable parameters of the DML.
When the DML is used as the penultimate layer of a classiﬁcation neural network (i.e. the layer producing logits), each center will correspond to one class.
Essentially, the layer maps an input vector to a vector of distances to class centers.
Note that, in this usage, the negative of the distances has to be used to maintain the typical assumption of DNNs that the highest logit corresponds to the most likely class.
The classiﬁcation loss function will encourage the absolute distance to the target class to be small while the distances to all other classes to be large, which induces compactness in the embedding space.
This compactness has been used in prior work for classiﬁcation problems (Wan et al., 2018) and for adversarial robustness of individual models (Pang et al., 2020).
Our setting is composed of combining a classiﬁcation network 𝐺 with a DML ℎ to obtain a model 𝐹 characterized by 𝐹 ( 𝑥 )
=
ℎ ◦ 𝐺 ( 𝑥 ) where ℎ : R 𝑀 → R 𝑀 and 𝐺 : R 𝑁
→ R 𝑀 .
In other words, 𝐺 is used as a prior to map the input space to a representation space, and ℎ maps the representation space to distances from the corresponding class centers.
In the next section, we demonstrate how a distribution shift that 𝐹 shows with respect to 𝐺 could be eﬃciently utilized to form a diverse ensemble of models.
Before that, we want to highlight an important feature about DML.
If the network 𝐺 is 𝐿 𝐺 -Lipschitz continuous, and the DML ℎ is 𝐿 ℎ -Lipschitz continuous, then the network 𝐹 is Lipschitz continuous with constant upper-bounded by 𝐿 𝐺 · 𝐿 ℎ .
Thus, imposing small 𝐿 ℎ during the training process may lead to the regularization of Lipschitz constant for the network 𝐹 .
It is shown in the appendix that this can also largely improve the certiﬁcation bounds of network 𝐹 compared to the model 𝐺 .
3.1 Enhancing diversity between ensemble members
The main part of the proposed ensemble model is the deployment of DMLs.
The ensemble members stacked with DMLs are trained interactively to achieve the highest level of diversity.
In particular, a DML ensemble is trained to satisfy these properties: 1) the ensemble members have the highest accuracy over benign samples and diverse prediction errors over adversarial samples; 2) the attacks generated on one ensemble member are not transferable to other ensemble members; and 3) the training procedure maintains the Lipschitz constant of an ensemble member model small.
The ﬁrst objective is important as it indicates that the ensemble could provide high accuracy even with the low accuracy of the individual members.
The second objective guarantees a well-performing ensemble model under evasion attacks.
That is an adversarial example crafted on an individual ensemble member model would normally be misclassiﬁed in that individual member, but not necessarily in the other ensemble members.
This goal can be achieved in two folds: 1) by randomly choosing the centers and shuﬄing them to vary over classiﬁers in the formulation of DML; 2) by imposing the 𝑀 matrices to be dissimilar and possibly orthogonal between the ensemble members.
Increasing the cosine dissimilarity of DML’s covariance matrices of ensemble members would lead to nontransferable adversarial perturbations (Adam et al., 2019).
One approach to reduce the transferability across several models is to enforce various geometric relationship like orthogonality between the inputoutput gradient of the models.
For two distance map layers 𝑓 and 𝑔 with corresponding 𝑀 𝑓 and 𝑀 𝑔 inverse covariance matrices, we have shown in the appendix that the orthogonality of the 𝑀 matrices, i.e., 𝑀 𝑓 𝑀 𝑔 = 0 is a suﬃcient condition for the input-output gradients of 𝑓 and 𝑔 to be orthogonal.
Note that the explored suﬃcient condition is independent of the input 𝑥 .
Roughly speaking, the increase in the dissimilarity of the 𝑀 matrices reduces the cosine similarity of input-output gradients of the combined network by the chain rule formula.
Therefore, practically, minimizing the cosine similarity of the 𝑀 matrices across models enables reducing the rate of transferability of adversarial attacks across models in the ensemble.
3.2 DML parameter initialization and training
In this section, we describe the procedure of simultaneous and interactive training of ensemble members towards the goal of enhancing diversity.
The main building block of various ensemble members using DML is that the centers and the inverse covariance matrices of members are trained to have the highest level of dissimilarities between the ensemble members.
We aim to increase the failure independence by enhancing the disagreement diversity of diﬀerent networks.
We recall that these ensemble members are generated using a reference model by stacking a DML to the last hidden layer of the prior model.
The ensemble model consists of the created DML-based models.
To train the ensemble members, the parameters of each model learned to increase the dissimilarity of the 𝑀 matrices (e.g., by decreasing the dissimilarity loss function).
The DML’s centers in the ensemble are shuﬄed across its members.
In practice, we initialize the centers to be a permutation of the rows of an identity matrix.
Applying the shuﬄed normal vectors as the centers of the DML would avoid the scaling of the gradient of the network and can decrease the risk of gradient masking.
Shuﬄing the centers across the individual members may cause to diversity in adversarial directions for the members.
In this way, the ensemble framework can potentially be used to detect adversarial examples.
We discuss more about this mechanism in appendix.
Dissimilar 𝑀 matrices in the ensemble of models alternate the way in which each individual learner traverses the hypothesis space.
We also make the 𝑀 matrix partially learnable, i.e., only certain elements in 𝑀 are trainable and the rest are ﬁxed during the training.
This would make the classiﬁers have diﬀerent prediction errors on diﬀerent labels.
In the following, we elaborate on how we split the learnable parameters of the 𝑀 matrix across members to encourage diversity through diverse tasks.
3.3 Meta ensemble model
We choose the trainable parameters in 𝑀 such that there would not be any overlap between the indices of the learnable elements of diﬀerent ensemble members.
More speciﬁcally, let 𝐼 𝑗 denote the learnable parameters of the 𝑀 matrix of the DML in the 𝑗 𝑡ℎ ensemble member.
To promote the diversity in the topology of the DNNs, we select 𝐼 𝑗 such that 𝐼 𝑗 ∩ 𝐼 𝑘 = ∅ for 𝑗, 𝑘 ∈ [ 𝐾 ] .
By allowing a classiﬁer to learn speciﬁc elements in 𝑀 , it causes the model to be more accurate on the classes with labels corresponding to those elements, e.g., the 𝑘 𝑡ℎ ensemble model would show better benign accuracy on the labels in the set 𝐼 𝑘 .
On the other hand, during experiments, we realized that the classiﬁers are more vulnerable on those labels when exposed to the attackers.
Since the classiﬁers are more susceptible on those labels, the predictions from ensemble members could be aggregated while excluding the predictions of each member classiﬁer’s speciﬁc labels, e.g., 𝐼 𝑘 for classiﬁer 𝑘 .
Due to the fact that the sets 𝐼 𝑘 ’s for 𝑘 ∈ 𝐾 have no pairwise intersections, neglecting the prediction of models on those particular labels when aggregating the ﬁnal prediction would not impose any artiﬁcial bias on the ensemble.
We will refer to the aforementioned mechanism in the following as meta-model.
In summary, we are training models which are fundamentally diverse in their robustness to diﬀerent adversarial attacks, and therefore it increases the varieties in the shape of decision boundaries and consequently adversarial directions in the ensemble.
We use a tailored loss to interactively train the ensemble members.
Formally, the loss function is formulated as
[Equation]
where 𝐾 is the number of ensemble members, 𝐿 is the number of classes, L 𝑘𝐶𝐸 is the classiﬁcation loss (cross entropy) of 𝑘 𝑡ℎ ensemble member.
Minimizing the sum of losses from ensemble members increases the benign accuracy of the ensemble model, while the other terms in the loss formula (Equation (3)) manipulate the arrangement of the decision boundaries.
The second term encourages the dissimilarity of covariance matrices and thus advocates diverse hypothesis for classiﬁers.
The third term is a regularization parameter for the sum of 𝐿 1 norm of DML’s 𝑀 matrices to achieve a DML with a low Lipschitz constant.
The positive real numbers 𝛾 and 𝛽 balance these terms with the cross-entropy objective.
We choose these parameters via hyper-parameter tuning.
For training, one needs to initialize the centers and the 𝑀 matrices of the DML in each classiﬁer to have the most dissimilarity to the other ensemble members.
Towards having a DML with a low Lipschitz constant, we consider a diagonal 𝑀 matrix, where the diagonal terms are initialized randomly from the uniform distribution between 0 and 1.
It is possible to opt not to train the 𝑀 matrices if there exist suﬃcient dissimilarities between paired matrices (i.e., orthogonality) via appropriate initialization, given that networks trained from diﬀerent initialization can be far apart in their internal representations.
Finally, a consensus method over the predictions from individual classiﬁers is performed to obtain the output of the ensemble model.
The consensus method could be any operation on the prediction outputs, e.g., majority voting, mean, median, sum.
In this work, the ensemble output is based on the majority voting of 𝑘 -top predictions of the ensemble members.
Randomized DML and uncertainty in the DML-based DNN
Considering the distance map layer 𝑓 ( 𝑥 ) =
(cid:112) ( 𝑥 − 𝑐 ) 𝑇 𝑀
( 𝑥 − 𝑐 ) , where 𝑀 =
𝑃 𝑇 𝑃 and 𝑃 is a lower-triangular matrix, we construct a randomized version of DML 𝑔 from the base DML 𝑓 .
The randomized DML 𝑔 , is the expectation of the isotropic Gaussian perturbation of the 𝑀
[Equation]
(4) where 𝜎 is a hyper-parameter.
Here, we apply the uncertainty of a Gaussian process to identify the low-conﬁdence regions of the input space to the DML layer.
The Gaussian process assumes a set of priors on the set of all functions that can map the input space to the output space.
At training time, for any sample 𝑥 with target classiﬁcation 𝑦 , only the function of the center corresponding to 𝑦 will be considered.
At inference time, the expectation of all the output of these functions are used as the prediction.
As the 𝑀 matrix determines the shape of decision boundaries, uncertainty estimation from randomized smoothing of 𝑀 can be extracted during the inference time.
We sample i.i.d.
Gaussian samples 𝛿 1 , . .
. , 𝛿 𝑇 ∼ N ( 0 , 𝜎 2 𝐼 ) , and use the following Monte-Carlo estimator for the expectation of the randomized DML 𝑔
[Equation]
In the experiments, we show that the randomized DML provides better robust accuracy compared to
DML when used in an ensemble.
Experiments
In this section, we present our empirical studies to show the eﬀectiveness of our proposed method on decreasing the threat from adversarial attacks, while showing high benign accuracy.
We perform our experiments on MNIST, CIFAR-10 and RESISC-45 (Cheng et al., 2017) detasets.
Each of
MNIST and CIFAR-10 has 10 classes and RESISC-45 has 45 classes.
We implement the ensemble model based on the networks shown in Table 1.
In the experiments for each dataset, we consider an ensemble of ﬁve models.
Our baseline model is the ensemble of ﬁve models without DML trained with cross-entropy loss.
The DML-based ensemble models are jointly trained based on the loss in
Equation (3).
We set 𝛽 = 0 .
01 and 𝛾 = 0 .
01 in Equation (3) for training the ensemble models over all three datasets.
Following the illustration of the ensemble meta-model, for each classiﬁer, only two elements of the diagonal elements of the 𝑀 matrix are trainable and the rest of the diagonal elements are set to the ﬁxed value 0 .
For all the individual models in the ensemble and for all datasets, the center points of DMLs are set to the columns of an identity matrix with diﬀerent shuﬄing across members.
The centers are ﬁxed during the training procedure.
In Figure 1, we show the distribution of the positive CLEVER score values (Weng et al., 2018) for 1000 randomly selected images from the MNIST dataset for a model formed by summing the outputs of ensemble members.
We used the 𝐿 2 radius of 2 to calculate the CLEVER scores.
The results for the other datasets, which is similer to MNIST’s, are provided in the appendix.
The results shows that the number of samples with non-zero CLEVER score is an order of magnitude higher than the counterpart for the baseline model.
Moreover, for the baseline model, there exist no sample that has a
[Figure]
[Table]
[Figure]
score higher than 1 . 7 while for DML, the highest score is close to 2.
This shows that the DML-based ensemble has an intrinsic boost in robustness compared to the baseline.
In Tables 2-3, we show the benign and adversarial accuracy of ensemble models over the three datasets.
The white-box attacks that we consider are FGSM
(Goodfellow et al., 2014), BIM (Kurakin et al., 2017), PGD (Madry et al., 2018), and Carlini & Wagner (CW) (Carlini & Wagner, 2017), which are commonly used for evaluating defense models.
The number of iterations for BIM and
PGD are set to 10, while for CW, it is set to 200.
The 𝐿 ∞ budget 𝜖 for FGSM, PGD, and BIM; and the penalty (initial constant) parameters 𝛾 for CW are shown in the table.
The attacks we consider in this section are untargeted attacks unless otherwise mentioned.
In untargeted attacks adversaries are constructed to cause the classiﬁer to produce any incorrect label.
Unlike the previous research on ensemble-based defenses (Sharif et al., 2019; Liu et al., 2019) where the adversaries were crafted over a target model, in this paper, the adversarial examples are generated over the ensemble model derived by summing the predictions.
Crafting adversarial examples over the ensemble model would assess the true resistance of the ensemble as the generated adversaries contain the gradient information from all the individual members.
The predicted labels for the benign and adversarial samples in the ensemble model are obtained using the majority votes of top-1 predictions of the individual classiﬁers.
The results show a remarkable performance enhancement over the baseline model.
This could be explained by high non-transferability of adversaries and diverse ensemble representation across the ensemble members, which hinders the process of ﬁnding an optimal adversary that could fool the majority of ensemble members to have the correct prediction changed. [Table] [Table] [Table]
RESISCBenign 87. Untargeted 74. Targeted 75. We now examine if the adversaries generated on one ensemble member could be transferred to the other models in the ensemble.
For this purpose, in Table 4, we provide the results for both targeted and untargeted attacks when the crafted attacks on Model 1 are transferred to the other constituent ensemble members and the ensemble model.
It is observed that the crafted attacks exhibit low transferability rate.
This could be illustrated by the visualisation of the loss function around a sample from RESISC-45 dataset in Figure 2, which shows that the shapes of the loss surfaces of the constituent models are diverse and therefore the input-output gradients of a model do not provide a close approximation for the other models included in the ensemble.
In Table 5, we compare the distortion needed for the ensemble model versus the individual members for the targeted attacks using adaptive CW- 𝐿 2 attack (He et al., 2017) for 100 randomly selected samples from each dataset.
The results show that the perturbation required to generate adversaries on individual member is multiple times lower than the distortion required to create adversaries on the ensemble models.
Finally, in Table 6 we compared the robust accuracy for DML-ensemble and randomized DML ensemble over CIFAR-10 datasets.
The results for the other dataset could be found in appendix.
The results from randomized DML-ensemble shows outperform adversarial accuracy. [Figure] [Table] generating adversaries
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]  AutoAttack on the DML-based ensemble In this section, we evaluate our proposed model by the ensemble of parameter-free attacks, AutoAttack (Croce & Hein, 2020).
In (Croce & Hein, 2020), several recently introduced defenses, including the ensemble model in (Pang et al., 2019), were evaluated on AutoAttack, and most of them were shown to be, at best, less robust than they had been thought to be according to their original papers.
For instance for the ensemble defense proposed in (Pang et al., 2019), it was reported that the accuracy of the model dropped to zero under AutoAttack on CIFAR-10.
In Tables 7-8, we show the results for the evaluation of our proposed model under AutoAttack.
The results indicate that the DML-based ensemble models exhibit notable adversarial robustness.
Conclusion
In this paper, we introduces new Mahalanobis distance based ensemble models, in which the properties of the distance function are leveraged to increase the diversity of the ensemble predictions and reduce the attack transferability rate across the individual members.
The ensemble members initialized in a way that induces diversity, and are jointly trained while encouraging orthogonality among the inverse covariance matrices of their distance functions, which is shown to reduce transferability.
The proposed mechanism can be applied to pretrained networks as priors to form an ensemble with enhanced properties, which makes our model complementary to other existing defenses.
Our results show that the proposed ensemble-based defense achieves high benign accuracy and adversarial accuracy across multiple attack algorithms, including AutoAttack, without using adversarial training. [Table]
A Appendix A.1 Certifiable DML
Distance-map layer is a nonlinear map composed of square root of a quadratic function.
We consider the distance-map function ℎ
[Equation]
where 𝑀 ℎ is a semi-deﬁnite positive matrix with a Choleskey decomposition denoted by 𝑀 ℎ
= 𝐿 (cid:48)
𝑀 𝐿 𝑀 .
We let 𝜙 𝑟 − 1 denote the input to the layer ℎ .
Then, we obtain
[Equation]
On the other hand
[Equation]
Thus, the certiﬁable bound for ℎ is given by
[Equation]
− 𝐵 ℎ ≤ ℎ ( 𝜙 𝑟 − 1 ) ≤ (cid:13)(cid:13)
𝐴 ℎ 𝜙 𝑟 − 1 (cid:13)(cid:13) + 𝐵 ℎ
If 𝑀 ℎ is a diagonal matrix, with 𝛼 𝑀 = max 𝑖,𝑗 | 𝑀 𝑖𝑗 | and 𝛽 𝑀 =
min 𝑖,𝑗 | 𝑀 𝑖𝑗 | 𝛽 14 𝑀 (cid:13)(cid:13)
𝜙 𝑟 − 1 (cid:13)(cid:13) − 𝐵 ℎ ≤ ℎ ( 𝜙 𝑟 − 1 ) ≤ 𝛼 14 𝑀 (cid:13)(cid:13)
𝜙 𝑟
[Equation]
1 (cid:13)(cid:13) + 𝐵 ℎ
A.2 Orthogonality of input-output gradients of paired DMLs
One approach to reduce the transferability across several models is to enforce various geometric relationship like orthogonality between the input-output gradient of the models.
We consider two (squared) distance map layers 𝑓 , and 𝑔 deﬁned by 𝑓 ( 𝑥 ) =
( 𝑥 − 𝑐 𝑓 ) (cid:48)
𝑀 𝑓
( 𝑥 − 𝑐 𝑓 ) and 𝑔 ( 𝑥 ) =
( 𝑥 − 𝑐 𝑔 ) (cid:48)
𝑀 𝑔 ( 𝑥 − 𝑐 𝑔 ) , where 𝑀 𝑓 and 𝑀 𝑔 are the 𝑀 matrices, and 𝑐 𝑓 and 𝑐 𝑔 are centers, corresponding to 𝑓 and 𝑔 .
We have
[Equation]
and
[Equation]
Thus, we obtain
[Equation]
where we used the fact that 𝑀 𝑓 and 𝑀 𝑔 are symmetric matrices.
Given 𝑀 𝑓 𝑀 𝑔 = 0, we obtain
[Equation]
Thus, the orthogonality of the 𝑀 matrices for 𝑓 and 𝑔 is a suﬃcient condition for the input-output gradients of 𝑓 and 𝑔 to be orthogonal.
A.3 The threat model for the ensemble
To classify the input 𝑥 using an ensemble of classiﬁers, the output of each ensemble member on input 𝑥 is computed.
The ensemble prediction is performed by applying a consensus method to integrate the predictions form all individual members.
The consensus method could be like majority voting, mean, median, sum.
If all the members can predict with uncorrelated error, then a simple averaging may reduce the prediction error by the number of members.
However, when the errors are not independent, the reduction error would be smaller than this factor.
In this work, the ensemble output is obtained by using the majority voting of 𝑘 -top predictions of the ensemble members.
In the experiments, since the majority voting operator is non-diﬀerentiable, the white box adversaries are generated by the ensemble model which is obtained via summing of the predictions and attacking the aggregation layer of the ensemble.
Unlike the previous research (Liu et al., 2019) where the attacks are generated on a target model which is either a constituent ensemble member or a model not included in the ensemble, the white box attacks in this work are generated on the ensemble layer.
Generating the attacks on a target model might provide a false sense of robustness for the ensemble, while generating the adversaries by attacking the aggregation layer of the ensemble provides the strongest white-box attack having the gradient information from all the individual members.
Thus, it can measure the real robustness of the ensemble model.
We also investigate adaptive attacks (He et al., 2017) by crafting adversaries on all the component models of the ensemble.
We show that the perturbation which is required to generate adversaries on individual member is by order of magnitudes lower than the distortion required to create adversaries on the ensemble models.
A.4 DML-based ensemble for adversary detection
Given an input 𝑥 , DML-based ensemble might be utilized for detecting adversaries.
More particularly if the majority number of ensemble members which is determined by the threshold 𝑣 do agree on a class of the input class, the ensemble classiﬁes the input as the majority class.
Otherwise the input is identiﬁed as adversarial example.
To elaborate, we let 𝑓 𝑘 ( 𝑥 ) is the output of the 𝑘 𝑡ℎ classiﬁer on the input 𝑥 and 𝑙 = arg
max 𝑖 ∈
[ 𝐿 ] 𝑓 𝑘𝑖 ( 𝑥 ) .
Then the output of ensemble classiﬁer 𝑓 𝑒𝑛𝑠 on 𝑥 is given by
[Equation]
adversary
[Equation]
Evidently, increasing the threshold 𝑣 rises the probability of detecting adversaries, while the number of false positive cases might grows at the same time and therefore the benign accuracy of the ensemble model decreases.
Practically, the threshold 𝑣 in (8) is selected as (cid:98)
𝑛 2 (cid:99) + 1.
If 𝑣 = 𝐾 (i.e., the number of ensemble members), the majority framework is called unanimity framework were the ensemble classiﬁes the input 𝑥 only if all the classiﬁers agree on the class of input 𝑥 .
Unanimity framework poses a security risk if the adversarial examples cause the misclassiﬁcation to the same class over models.
Otherwise a defender can use the disagreement between models as a mechanism to detect adversarial examples.
An ensemble with diverse DML-based members is less likely to mislcassify the adversary input 𝑥 to the same class.
Thus, the detector becomes more powerful when the predictions from constituent members of ensemble are diverse.
Due to the shuﬄed representation of neighbour classes’ domain to the decision region of class 𝑥 for the DML-based models, the adversaries for the sample 𝑥 is misclassiﬁed diversely (i.e., diverse labels) across the ensemble members.
So the likelihood of two classiﬁers to misclassify the adversary input to the same class would decrease and it increases the probability to detect 𝑥 .
To insure the diversity in the centers form DML layer of the ensemble members, we shuﬄe the centers of classes per each individual classiﬁer.
Note that the diversity does not necessarily change the shape of the decision boundaries and only alter the adversarial directions.
In the other words, it could cause the classiﬁers to provide various prediction on a given input that could be leveraged to detect adversaries.
It is important to note that, unlike previous approaches on ensemble model (Sharif et al., 2019), imposing this diversity does not imply any upper bound in the number of constituent classiﬁers in the ensemble as there are inﬁnitely many ways to have the centers of classiﬁers shuﬄed.
In previous research (Sharif et al., 2019; Liu et al., 2019), the authors had to split the classiﬁers to plenty of ensembles in order to achieve diversity inside each ensemble.
A.4.1 Unanimity framework for adversarial detection
We might use the ensemble of DML-based networks to detect adversarial samples based on unanimity framework.
In this setting, the ensemble provides the prediction for label of input 𝑥 , only if all the individual members agree on the label of the 𝑥 .
Otherwise, the ensemble renders the sample as an adversarial example.
In Figure 3 we have shown the ROC plots and AUC scores for adversarial detection using unanimity framework over three datasets.
For each dataset we considered testset of correctly classiﬁed benign samples and the corresponding adversarial samples crafted with white-box attacks FGSM and PGD.
The parameters for the attacks used are similar to the ones in Tables 2-10.
The results show the remarkable performance of DML-based ensemble with unanimity framework as a detection metric.
[Figure]
A.5 Attack detection mechanism with 1-D projection
Shuﬄing the centers in high dimensional space does not necessarily lead to diverse adversarial directions as in high dimension the decision boundaries of one class to the other class could be numerous.
To circumvent this issue to promote the diversity of the adversarial direction by taking advantage of shuﬄed centers, we restrict the embedding space to a one dimensional centers, where the center location for each class per each model in the classiﬁer is just a shuﬄed representation of ﬁxed vector providing the center locations.
In Figure 4 it is observed that shuﬄing the centers in 1-D can signiﬁcantly increase the rate of adversarial detection.
[Figure]
A.5.1 Adversarial detection using uncertainty metric
We deﬁne the uncertainty distribution of the model at sample points as the variance of outputs of models across ensemble members and for multiple inferences.
The quantiﬁed uncertainty of the randomized-smoothing DML-based network is utilized to detect adversarial examples.
Since there is one output for each class in the network, the mean of the uncertainties for all classes indicates the uncertainty of the model at the sample 𝑥 .
In Figure 5 we have shown the uncertainty distribution of
FGSM and PGD adversarial examples versus normal examples.
Apparently, the distribution from adversarial examples is distinguishable from normal and noisy samples.
We use this intuition to train a logistic-regression model on the inputs of uncertainty of data to detect adversarial examples.
In Table 9 we show the adversarial accuracy of the same ensemble models we considered in Tables 2-3 for the randomized smoothing DML layer indicated in equation (4).
To compute the randomized smoothing DML layer with Monte-Carlo approximation formulation (5), we set 𝑇 =
and the noises are generated from the normal distribution N ( 0 , 𝜎 2 𝐼 ) with 𝜎 = 0 .
It is seen in Tablethat the adversarial accuracies of white box attacks for randomized smoothing DML outperform the corresponding results for ensemble of DML based models.
[Figure]
[Table]
[Figure]
[Table]
We could also detect adversaries using the uncertainty quantity measured using randomized DML.
As shown, the benign, noisy and adversarial data are representing diﬀerent uncertainty distribution.
Based on this observation, we train a logistic-regression classiﬁer from the uncertainties of a combined benign and noisy data as the negative class and the uncertainty from adversarial data as the positive class.
We use the testset of correctly classiﬁed samples for benign samples, and crafting noisy, and adversarial examples.
The adversaries are generated using from the FGSM and PGD attackers.
The trained model classiﬁes benign and adversarial examples.
We show the ROC plots and AUC scores of the classiﬁer in Figure 6.
The ROC plots demonstrate that the uncertainty from randomized DML might be used as an indicator to detect if a sample is adversary.
[Figure]
[Figure] [Figure]
A.6 CLEVER score
In Figures 7 and 8 we depicted the histogram of CLEVER score (Weng et al., 2018) of 1000 randomly selected images of the model derived by summing the prediction of individual constituent models in the ensemble for CIFAR-10 and RESISC-45 datasets.
The results is shown for samples that have higher CLEVER score than the threshold 0.2.
CLEVER score provides an upper bound for the radius of certiﬁed ball around each sample in 𝐿 2 norm.
The ﬁgures show that the scores from DML-based ensemble outperform of the counterpart of baseline model.
For instance for CIFAR-10 dataset the number of samples with the CLEVER score greater than 1.0 is less than 15 samples, however for
DML-based ensemble around 250 samples have the score of 2.0.
A.7 Orthogonality of input-outputs gradients
Now we demonstrate the eﬀect of pairwise orthogonality of the 𝑀 matrices for DMLs across the ensemble imposing by regularized 𝛾 in (3) on orthogonality of the input-output gradients of constituent models.
For this purpose we design an ablation study for a ensemble of two models.
Once we train the ensemble model with 𝛾 = 0 .
and once the ensemble model is trained with 𝛾 = 1 . 0.
The histograms for the cosine similarity of input-output gradients of the two models in the ensemble over three datasets is shown in Figures 9, 10 and 11.
For each dataset we have shown the cosine similarity of models’ gradients and also for the sign of gradients.
Iterative attacks like PGD apply the sign of gradients in order to craft the attacks.
The ﬁgures show that the gradients for the models were
[Figure]
[Figure]
trained with 𝛾 = 0 are more aligned compared to the ones with 𝛾 = 1.
The results for RESISCis not prominent as the cosine similarity of the gradients of ensemble models trained with 𝛾 = 0 .is
already close to zero.
The results from these ﬁgures conﬁrm that counting on the regularizer in the loss function to impose the orthogonality of the 𝑀 matrices of paired DMLs would signiﬁcantly reduce the cosine similarity between the gradients of loss function with respect to the input data for paired models in the ensemble.
A.8 Effectiveness of R-DML in adversarial training
In this section we illustrate empirically that R-DML can provide better benign and robust accuracy when compared to standard DML for adversarial training.
To better distinguish the performance of
R-DML from the randomness obtained by augmenting data with Gaussian noise, in addition we also perform experiments for DML where the input data is augmented with the isotropic Gaussian noise, i.e.,
[Equation]
We call this layer augmented DML (AUG-DML), as the input to DML augmented with random noise.
In Figure 12 we have shown the benign and robust validation accuracy.
The validation set is equal to the testset.
Figure 13 also counterparts which are obtained
[Figure] [Figure]
by combining DML with the models which are adversarialy trained.
In other words for the models in Figure 13 we use prior models which are adversarialy trained.
All the models we consider in this section were trained adversarialy with one-step FGSM attack with the stepsize 10 . 0 / 255 and without random initialization.
The results in Figure 12 show that R-DML marginally show better benign and robust accuracy, while the results from AUG-DML present similar performance as DML.
It shows that the noise injected to DML by convolving Gaussian distribution with the 𝑀 matrix is more eﬀective compared to the DML obtained by augmenting inputs to DML with random noise.
Also the results from Figure 13 for R-DML show a signiﬁcance improvement over DML and AUG-DML.
It is important to note that with the higher number of epochs in contrary to DML, R-DML avoids over-ﬁtting.
It is also seen that standard DML based network achieve the peak of benign and robust accuracy with only 2 epochs, although the results underperform the ones from R-DML after in the long run training.
It shows the DMLs can be applied eﬀectively with pretrained adversarialy trained priors to generate diversity across the model members.
A.9 Randomized smoothing and certified accuracy
In this section we examine the randomized smoothing model obtained by convolving the base model with the isotropic Gaussian noise (Cohen et al., 2019).
Our hypothesis is that the certiﬁed accuracy for smoothed classiﬁer with R-DML is providing tighter certiﬁed accuracy versus the standard DML.
The reason is that R-DML explores some points in the embedding space of the classiﬁer for certifying the samples which might be overlooked in the standard DML case when the input data is augmented
[Figure] [Figure]
with Gaussian noise.
The reason for that is since we apply DML as a last hidden layer, augmenting features with random noise could add more control on exploration of embedding space.
Given that, those embedding points may have diﬀerent classiﬁcation as the original sample.
Thus, R-DML results in a tighter bound for the certiﬁed accuracy by DML.
We also demonstrate that AUG-DML deﬁned in the last section does not lead to dramatic change in the certiﬁed accuracy compared to
R-DML, which supports the signiﬁcance of applying R-DML.
Similar ideas on inserting noise to the embedding space to explore overlooked regions based on data augmentation has been recently explored in for example (Chen et al., 2020).
However in this work, the noise is backpropagated in the network to ﬁnd the mapped location for the noise in the input space, which add additional computational burden to the training procedure.
However, in R-DML the noise in embedding space freely added by convolving it to the 𝑀 matrix.
In Figure 14 we have shown the certiﬁed accuracy for
R-DML versus DML and AUG-DML.
Note that these models are trained regularly and not through randomized smoothing training procedure.
The random noise for training R-DML and AuG-DML which is applied is isotropic Gaussian noise with standard deviation of 0 . 5.
The certiﬁed accuracy for models in this section is measured over 1000 randomly selected samples.
Figure 14 shows that
R-DML basically provides better certiﬁed accuracy compared to the other two DML based models.
In Figure 15 we have shown the certiﬁed accuracy of smoothed classiﬁer based on standard DML.
The ﬁgure also shows the certiﬁed accuracy of smoothed classiﬁer, when the DML in the smoothed classiﬁer is replaced by R-DML and AUG-DML.
This ﬁgure shows that R-DML provides tighter bound for the certiﬁed accuracy of smoothed classiﬁer by DML.
[Figure]
[Table]
In Figure 16 we have shown the certiﬁed accuracy for smoothed classiﬁer with three types of DML layer.
We then removed the DML layer and replaced it with R-DML to measure the certiﬁed accuracy.
It is seen that the model trained with AUG-DML does not not tolerate the noise from R-DML and it failed to provide certiﬁcation for any sample.
The smoothed classiﬁer trained by R-DML could provide high certiﬁed accuracy
[Table]
A.9.1 Effect of meta-model on promoting diversity
To show the eﬀectiveness of group-splitting mechanism discussed in earlier, we trained the RESISC45 dataset without this mechanism and only by scaling the 𝑀 matrix to impose diversity across members.
Figure 17 shows that under this experiment setting, no remarkable diversity in the shape of loss function across members are created.
The corresponding results for the robust accuracy and rate of transferability as presented in Tables 10, 11, 12 show lower performance compared to the model trained with meta-model description.
[Figure]
[Table]
A.9.2 Distortion-accuracy plots
In Figure 18 the distortion-accuracy plots are shown for FGSM, BIM and PGD white-box attacks and over all three datasets.
It is observed that increasing the distortion scale 𝜖 leads to almost 0% accuracy for the ensemble models which could be an indication that the DML-based models do not mask the gradient information, i.e., the attackers are applying interpretable gradients from the model.
In Figures 19-21 we visualize samples of original images and the corresponding adversarial images generated from Table 5 for all three datasets.
These ﬁgures show that the DML-based ensemble largely increases the required distortion such that the distortion is highly perceptible.
[Table] [Table]
[Figure] [Figure]