MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attack
Abstract
Adversarial attacks can deceive neural networks by adding tiny perturbations to their input data.
Ensemble defenses, which are trained to minimize attack transferability among sub-models, offer a promising research direction to improve robustness against such attacks while maintaining a high accuracy on natural inputs.
We discover, however, that recent state-of-the-art (SOTA) adversarial attack strategies cannot reliably evaluate ensemble defenses, sizeably overestimating their robustness.
This paper identifies the two factors that contribute to this behavior.
First, these defenses form ensembles that are notably difficult for existing gradientbased method to attack, due to gradient obfuscation.
Second, ensemble defenses diversify sub-model gradients, presenting a challenge to defeat all sub-models simultaneously, simply summing their contributions may counteract the overall attack objective; yet, we observed that ensemble may still be fooled despite most sub-models being correct.
We therefore introduce MORA, a model-reweighing attack to steer adversarial example synthesis by reweighing the importance of submodel gradients.
MORA finds that recent ensemble defenses all exhibit varying degrees of overestimated robustness.
Comparing it against recent SOTA white-box attacks, it can converge orders of magnitude faster while achieving higher attack success rates across all ensemble models examined with three different ensemble modes ( i.e. , ensembling by either softmax, voting or logits).
In particular, most ensemble defenses exhibit near or exactly 0% robustness against MORA with ℓ ∞ perturbation within 0 .
02 on CIFAR-10, and 0 .
01 on CIFAR-100.
We make
MORA open source with reproducible results and pre-trained models; and provide a leaderboard of ensemble defenses under various attack strategies 1 .
Introduction
Many safety-critical applications, such as autonomous robots [34], self-driving [8], search engines [24], etc. are becoming increasingly powerful and reliant on deep neural networks (DNNs).
Despite the monumental success of DNNs on these applications, they are highly susceptible to adversarial examples — an attacker can add tiny delibrate perturbations to the input data, misleading the model into giving incorrect results [23, 9].
Such adversarial attacks could pose a significant threat to the safety and reliability of deep learning applications.
To mitigate this threat, many defense strategies [17, 33, 4] based on adversarial training [17] have been proposed to improve model robustness.
Adversarial training, however, gains robustness at the expense of model accuracy on clean natural images [27].
Ensemble defenses [19, 14, 30, 31] have [Figure] [Figure] [Figure]
thus emerged to combine multiple predictions from independent sub-models The intuition is that an ensemble of models can often lead to higher accuracy transfer improve robustness against adversarial attacks.
This approach could potentially offer a promising research direction to improve model robustness while preserving high accuracy on natural inputs
Yet surprisingly, under the white-box threat model, existing state-of-the-art (SOTA) adversarial attacks with strong performance on conventional DNN models performed poorly on ensemble models, sizeably overestimating their robustness (Figures 1a and 1b).
This also suggests, to some extent, that ensemble defenses may rely on two potential design flaws below that cause obfuscated gradients [1], i.e. , they are either deliberately non-differentiable, or give no useful gradients, thus inducing overestimated robustness:
(a) Gradient obfuscation via ensemble-forming strategy.
They typically form ensembles by averaging probability vectors (softmax) of sub-models, and softmax operations can easily cause gradient obfuscation.
While the model’s actual robustness is pertinent to the strategy used to form an ensemble, this indicates that gradient-based attacks have to also leverage this effectively.
(b) Gradient diversification.
Motivated by the reasoning that a majority of sub-models may need to be fooled for successful attacks, they learn to reduce adversarial transferability among sub-models, often via gradient diversification.
This intuitively causes sub-models to counteract each other, averaging to a small or inaccurate overall gradient.
Attacking only the ensemble loss would fool most sub-models, but the ensemble may remain still correct; conversely, it is actually possible to fool an ensemble, despite the majority of its sub-models giving correct predictions (Figure 1c).
From the above observations, it is perceivable that the practical evaluation of ensemble robustness cannot be solely done by treating such models holistically.
To this end, this paper introduces MORA, model-reweighing attack, to adaptively adjust the importance of sub-models in attack iterations.
Sub-models are reweighed according to their respective “ease of attack”, which is in turn evaluated by the gradient of the difference of ensemble classification outputs w.r.t.
the ones of individual sub-models.
Pushing the limits of the current SOTA in ensemble robustness evaluation, it draws inspiration from recent effective attack tactics, e.g. , momentum [7, 5], step size schedule [5, 16], loss normalization [32], and multiple targets [5, 25].
We summarize our contributions:
• This paper presents the first extensive study on the robustness of ensemble defenses under multiple ensemble-forming strategies.
• By reweighing the importance weights of sub-models to steer adversarial example synthesis, we show that gradient-based attacks on ensemble defenses can often be orders of magnitude faster, while enjoying a higher success rate.
• Empirical results on a wide variety of different ensemble defenses show that MORA outperforms competing attacks in both performance and convergence rate.
Finally, this paper provides extensive ablation of its components and sensitivity analyses of hyperparameters.
To our best knowledge, MORA is currently the strongest attack against a wide range of ensemble defenses.
We make MORA open source with reproducible results and pre-trained models; moreover, we maintain a leaderboard of ensemble defenses under various attack strategies.
Preliminaries & Related Work 2.1 Adversarial Attacks
An adversarial example adds a small perturbation, typically bounded a small value with ℓ p norms, to the original image, such that the model under attack can be deceived into giving incorrect results.
The advent of adversarial attacks [23] has piqued the interest of deep learning practitioners, and revealed security concerns of deep learning [26, 21], improved GAN training [2] transfer learning [28, 6], and DNN interpretability [22], etc.
Formally, assuming a defending classifier f : I → R K , taking an input image x ∈ I =
[0 , 1] C × H × W as input, and arg max f ( x ) predicts the correct class label y ∈ C , then an attacker attempts to find an adversarial example ˆ x in the set:
[Equation]
Here, ˆ x ∈ A ϵ, x constrains the adversarial example ˆ x to be within both the input space I and a small ϵ -ball of ℓ p -distance from the original image x , or equivalently ∥ x − ˆ x ∥ p ≤ ϵ .
Satisfying the condition arg max f ( x ) ̸ =
y means that f ( x ) fails to give the expected correct classification y .
We focus on the ℓ ∞ white-box threat model commonly considered by the defenses examined in this paper, which grants the attacker completely access to the internals of the defender, including, for instance, its model architecture, parameters, training algorithms, etc.
One of the popular and effective white-box attacks used by many defenders to evaluate their robustness is projected gradient descent (PGD) [17], which finds adversarial examples by maximizing the classification loss with gradient descent:
[Equation]
where L is typically the softmax cross-entropy (SCE) loss used to train the model, α i is the step size, and we let the initial ˆ x 0 ≜ P ϵ, x ( x + µ ) .
The projection function P ϵ, x ( v ) constrains its input v to be within the feasible region
A ϵ, x , and finally µ ∼ U ( − ϵ, ϵ ) is a uniformly distributed noise bounded by [ − ϵ, ϵ ] .
Besides PGD, C&W [3] is also a gradient-based attack which, instead of projection, indirectly constrains the search space by regularization.
As PGD gains popularity, many defense mechanisms rely on it to evaluate their robustness.
Unfortunately, AutoAttack (AA) [5] finds that many of the defenses may inadvertently break PGD-based attacks, which result in drastic overestimation of their robustness, and proposes to combine an ensemble of diverse attacks to minimize robustness overestimation.
LAFEAT [32] learns to leverage intermediate layers of the DNN, and shows that attacking multiple layers can produce stronger attacks, but unfortunately it cannot be applied to ensemble defenses.
Adaptive Auto Attack ( A 3 ) [16] improves attack success rates by using the gradient directions to prescribe a more effective initial random perturbation.
As defenders may design mechanisms to circumvent existing attacks, Adaptive attacks [25] manually tailor specific attack strategies for an extensive set of defenses.
Finally,
Composite Adversarial Attacks (CAA) [18] further combine a large selection of attack methods, and use a genetic algorithm to learn an optimal attacking sequence.
In comparison, MORA is a unified approach which uses only one attack algorithm, does not require a compute-intensive learning procedure, and yet it still achieves fast and SOTA estimation of ensemble robustness.
2.2 Defending Against Adversarial Attacks
Defending against adversarial attacks can be defined as a saddle-point problem to minimize the training loss on adversarial examples [17] with samples ( x , y ) drawn from the training set:
[Equation]
where L is the training loss, e.g. , the SCE loss.
A direct optimization-based approach to approximately solving the above problem is adversarial training [17], i.e. , to train the DNN model with its own adversarial examples.
Training DNNs to be robust is, however, a challenging endeavor.
First, it may be much more compute intensive as training examples are typically generated with PGD [17], requiring a few forward/backward passes of the DNN.
Second, to avoid overfitting, it requires stopping training early, a much larger size of the training set [4], and using improved data augmentation [20] or generated data [10].
Thirdly, as noted by other literatures [5, 32], currently no other design choices can rival the robustness provided by adversarial training, and notably, many defense strategies are considered harmful to model robustness [25].
Finally, the resulting models often cannot achieve high clean accuracy [27].
2.3 Ensemble Defenses & Ensemble-forming Strategies
Ensemble-based defense techniques may pave an alternative path to address the challenges of adversarial robustness, as they could potentially work around the above limitations of adversarial training.
Adopting the theme of minimizing adversarial example transferability across sub-models, each ensemble defense proposed unique solutions.
ADP [19] increases the orthogonality of nonmaximal class logits among sub-models to encourage diversity.
GAL [14] minimizes a gradient alignment loss, which directly reduces the cosine-similarity between sub-models.
Building on top of this, TRS [31] further regularizes the smoothness of the loss function, as gradient orthogonality with smoothness may further diversify sub-models.
Dverge [30] instead uses the adversarial examples of a sub-model to train another sub-model, thus lowering transferability.
Ensemble defenses are also particularly interesting, as they are the last line of defense against even the strongest existing white-box attacks without resorting to adversarial training, showing a certain degree of robustness.
Besides the above mechanisms for training a successful ensemble defense, there exists different ways to combine sub-model predictions.
Let us assume that an ensemble defense trains M sub-models, f m : I → R K for m ∈ [1 : M ] , an ensemble f E : I → R K thus forms a final classification result by combining individual decisions from the sub-models, namely:
[Equation]
where ens is the ensemble-forming operator.
In this paper, we investigate ens ∈ { softmax , wta , id } , where the potential choices respectively denoting forming an ensemble from sub-model outputs f m ( x ) by either summing predicted probabilities (evaluated with softmax ), or majority votes (using wta , the winner-take-all operator), or simply summing the logits (with id , the identity operator).
Defending ensemble methods [19, 14, 30, 31] tested in this paper all employed the softmax -based strategy to report their robustness.
Methods that are exceptions to these options exist, for instance,
ECOC [29] allows sub-models to produce binary predictions, and use error correcting codes based on the Hamming distance to combine the predictions into classification outputs.
This approach is unfortunately not robust, and the added complexity is error-prone and may harm robustness [25].
Moreover, as the voting ( wta ) strategy is non-differentiable, an attacker can soften it approximately using a softmax operation with temperature τ , where we used τ = 0 .
universally:
[Equation]
τ
[Equation]
[Equation]
The Model-Reweighing Attack (MORA) 3.1 Problem Formulation & High-Level Overview
As discussed in Section 1, existing ensemble defenses may obfuscate gradients with the ensembleforming mode and gradient diversification, such that the final loss of the ensemble model can no longer provide effective signals for gradient descent.
It is therefore desirable to find an alternative
L to the original SCE loss L sce on the ensemble, such that for a given number of iterations I , the original L sce loss can be maximized:
[Equation]
, f [1: M ] ( ˆ x i ) , y )) , (6) and PGD( · ) denotes a PGD step along the gradient of loss function L ( f E (ˆ x i ) , f [1: M ] (ˆ x i ) , y ) , which not only takes the ensemble predictions f E (ˆ x i ) as input, but can further utilize sub-model predictions f [1: M ] to guide the PGD iterations.
The challenge at hand is, therefore, to find a suitable L which can generate attacks on ensemble defenses efficiently and effectively
[Table]
MORA +
[Table] Sub-model Reweighing <
[Table]
[Figure] MORA aims to provide a potential optimization route towards the above problem formulation.
Namely, in addition to the original output of the ensemble f E (ˆ x ) , we leverage the sub-model predictions f [1: M ] (ˆ x ) to facilitate the optimization.
By way of illustration, Figure 2 shows a highlevel overview of the model-reweighing attack, where we compliment the ensemble loss, with a newly added sub-model reweighing loss L mora β,τ , as an auxiliary attack vector alongside the original objective.
Not only can the new loss bypass the ensemble-forming strategy to work around its obfuscated gradients, but it further exploits information present in the individual sub-model and ensemble predictions to steer the direction of adversarial example synthesis.
3.2 Adaptive Sub-model Importance
Before we begin, assume that z [ m ] ≜ f m ( x ) represents the m th sub-model output, and let z [ m ] t denote the corresponding logit of label t .
We define the difference of logits (DL) [3] k
[ m ] ≜ z [ m ] y − z [ m ] ˆ y , which is the difference between the predictions of the ground truth z [ m ] y and the maximum of the remaining classes z
[ m ] ˆ y ≜ max i ∈C /y
z
[ m ] i , where C /y is the set of all class labels except y .
Similarly, we let z [E] ≜ f E ( x ) , and z [E] t and k [E] be the respective variants of the ensemble prediction.
It is notable that a successful attack happens when k [E] < 0 , and similarly k [ m ] < 0 means the m th sub-model is producing incorrect classification.
Ensemble defenses tend to diversify sub-model gradients, for instance, ADP [19] minimizes the cosine-similarity ⟨∇ ℓ ( z [ a ] ) , ∇ ℓ ( z [ b ] ) ⟩ among each loss function gradient pairs of sub-models ℓ ( z [ a ] ) and ℓ ( z [ b ] ) .
Their intuition is that it may lower transferability among these sub-models, such that attacks with the overall gradient of the ensemble, i.e. , ∇ ℓ ( z [E] ) = 1 M (cid:80) m ∈
[1: M ] ∇ ℓ ( z [ m ] ) , are becoming less effective in misleading all sub-models simultaneously, as individual gradients in ∇ ℓ ( z [ m ] ) are encouraged to be orthogonal to each other.
To this end, we propose to reweigh the importance of sub-models, by instead considering the modified gradient:
[Equation]
where λ [ m ] τ assigns weights to important sub-models to contribute more heavily to the attack gradient.
While the adversarial examples of the ensemble could present a challenge to discover, individual sub-models are weak defenders which can be easily defeated.
Based on this property, we propose to weigh sub-models importance with the rate of change in k [E] w.r.t.
that of k [ m ] , i.e. , sub-models would be given higher weights if attacking it would bring a significant change to the ensemble’s prediction.
Following this idea, for all ensemble-forming strategies (softmax, voting, logits), we rewrite k
[E] as a function of k [ m ] , where the term below can become a function of k [ m ] :
[Equation]
1: function MORA_ Attack ( f [1: M ] , x , y, β, τ, ν, ϵ, I ) 2: ˆ x 0 ← P ϵ, x ( x + u ) , where u ∼ U ( − ϵ, ϵ )
▷ Random init 3: µ 0 ← 0 4: for i ∈ [0 : I − 1] do 5: z
[ m ] ← f m (ˆ x i ) for all m ∈ [1 : M ] ▷ Sub-model predictions 6: z [E] ← 1 M (cid:80) m ∈ [1: M ] ens( z [ m ] )
▷ Ensemble prediction 7: k [E] ← z [E] y − z [E] ˆ y 8: if k [E] ≤ 0 then return ˆ x i ▷ Successful attack 9: g i +1 ← sign( ∇ ˆ x i L mora β,τ ( z [1: M ] , z [E] , y ))
▷ Sign-gradient with the MORA loss 10: α ← ϵ (1 + cos( iπ / I ))
▷ Cosine step-size schedule 11: µ i +1 ← P ϵ, x ( µ i
+ α
g i +1 )
▷ Iterative update 12: ˆ x i +1 ← P ϵ, x (ˆ x i + ν ( µ i +1 − ˆ x i ) + (1 − ν )
(ˆ x i − ˆ x
i − 1 ))
▷ ...with momentum 13: end for 14: return ˆ x I ▷ Give up after I iterations 15: end function
The weights are thus defined as follows:
[Equation]
While it is possible to compute the weights using gradient back-propagation, we can simply derive the following closed-form solution of the weights for each of the three ensemble-forming strategies.
For wta , we use the softened version of wta as defined in (5) and can derive the weights as follows:
[Equation]
where s = softmax (cid:0) z [ m ] [-> 0] is the indicator function that equals 
-]/ τ (cid:1) . [Equation]
Here 1[ k [ m ] > 0] is the indicator function that equals 1 if k [ m ] > 0 , or 0 otherwise, effectively stopping the attack on the m th sub-model upon success, and the detach operator admits no backward propagation to its input.
In the case of using sums of sub-model softmax outputs to form an ensemble decision, i.e. , ens = softmax , it is a special case of softwta τ where the temperature coefficient can be fixed at τ = 1 .
Finally, when ens = id , i.e. , forming ensembles by summing logits, λ [ m ] τ ( z [ m ] ) simply reduces to 1[ k [ m ] > 0] for the m th sub-model.
3.3 The MORA Loss
For reference, defenses mechanisms we examine in this paper aim to find ˆ x which maximizes the
SCE loss L sce ( z [E] , y ) , to evaluate the ensemble robustness.
The MORA loss improves this further by proposing two additional modifications to the untargeted loss function used to attack ensembles:
[Equation]
First, it additionally introduces a sum of the λ [ m ] τ -weighted variant of sub-model logits, in order to expose sub-model logits with adaptive reweighing described in Section 3.2.
Second, β interpolates the importance of the newly added auxiliary logits and the original ensemble logits.
Finally, inspired by the effective surrogate loss in [32], it further normalizes the logits by their respective DL using:
[Equation]
Finally, the targeted variant of the MORA loss simply replaces y with t where t is the intended target.
3.4 Improving the State-of-the-art
While the new L mora β,τ loss is highly effective against ensemble defenses we test in this paper, we strive for further advances in MORA’s ability to generate faster and better adversarial examples.
Inspired by recent publications, we borrow ideas from related adversarial attack tactics, which includes adopting a cosine step-size schedule [16], momentum [7, 5], random restarts [25] and multiple target attacks [5, 25].
We provide the overall algorithm in Algorithm 1, which computes an adversarial image ˆ x I as return, by taking as input the sub-models f [1: M ] , natural image x , ground truth label y , β to interpolate between the auxiliary logits and the original, τ controls the temperature, momentum µ = 0 .
75 following [32, 5], ϵ perturbation bound, and finally the maximum number of iterations I .
[Table]
Experimental Results
We compare MORA against SOTA attacks for a wide range of ensemble defenses under three ensemble-forming strategies (softmax, voting, and logits).
We use pre-trained ResNet-20 [12] models from ADP [19], Dverge [30], GAL [14], and reproduced TRS [31] using the same architecture with official source code, as pre-trained models were unavailable.
Our robustness evaluation considers the ℓ ∞ white-box attacks on the CIFAR-10 test set [15], with perturbation ϵ = 0 .
01 unless specified.
The full comparison results can be found in Table 1; larger ϵ comparisons, and similar results on
CIFAR-100 models are in Appendix A.
We provide our key observations below.
Traditional attacks may fail to break through gradient obfuscation.
We reproduce two traditional white-box attacks, i.e. , projected gradient descent (PGD) [17] and C&W [3] with 5 random restarts, each with a maximum of 100 iterations, giving a total of 500 iterations.
PGD uses a fixed step size of ϵ/ 4 .
For a fair comparison, MORA with 500 iterations sweeps β ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
75 , 1 } , with each β up to 100 iterations.
Even with a 500 iteration budget, it is clear that PGD and C&W may substantially overestimate robustness, especially when tested under the softmax and voting ensembleforming options, and MORA can work around this obstacle thanks to its attacks on sub-models.
Diversified gradients can hamper even integrated attacks with large arsenals.
Moreover, we test the defenses against recent integrated attacks with SOTA baselines on robustness evaluation, namely Adaptive Auto Attack ( A 3 ) [16], AutoAttack (AA) [5], and Composite Adversarial Attacks (CAA) [18], which comprise large arsenals of various attack strategies.
We reproduce CAA following [18] to search for the attack policy before evaluating the defending models.
Note that its computational complexity is thus much higher than the other attacks, but we only report its test-time complexity.
In particular, while they enjoyed much higher success rates than PGD and C&W, some defenses render their tactics ineffective.
We observe, e.g. , sizeable robustness overestimation on
ADP [19] under softmax and voting, which explicitly diversifies sub-model gradients.
As MORA can dynamically re-adjust sub-model importance w.r.t.
their “ease-of-attack”, it performs substantially better with much fewer iterations.
In addition to the earlier 500 iterations, the multi-targeted MORA mt targets the remaining 9 class labels with 100 iterations for each label and β fixed at 0 .
Others also use multi-targeted attacks along with respective tactics.
Robustness of most sub-models vs. robustness of ensemble.
We find that robustness of a majority of sub-models (fooling 3 / 8 for softmax and 2 / 8 for logits) usually do not translate to the overall robustness of the ensemble (Figures 1c and 3a).
As voting requires breaking 1 / 2 sub-models simultaneously (Figure 3b), it is perceivable that using voting may give rise to a higher overall robustness.
Yet surprisingly, for most defending ensembles, voting performs worse than softmax and logits.
Ensemble-forming strategies may give a false sense of security.
On one hand, softmax and voting strategies exhibit substantially larger overestimated robustness (up to 40%) than logits.
On the other hand, in stark contrast to the proposed use of softmax from [14, 19, 30, 31], we find summing by logits can form ensembles that are notably more robust than the other two (Figure 3c), while attackers only needs to successfully deceive a few sub-models (referring back to Figure 1c).
Up to 60 × faster convergence under 500 iterations.
Figure 4 compares the convergence speed of
MORA against AA losses, C&W, and PGD on defending ensembles.
MORA converges substantially faster than the other attacks, using only up to 31 steps to match AA losses with 500 iterations.
Ensemble defense mechanisms may be at odds with robustness.
In Table 2, we compare respective attacks on adversarially trained Dverge models.
To our surprise, forming larger ensembles is actually harmful to the robustness of ensemble.
[Figure]
[Figure]
[Table]
Additional results, ablation, and sensitivity analyses.
Due to the page limit, we provide full results of relevant figures in Appendix A, note that the above key observations still hold true for all ensemble defenses we test under different ensemble-forming strategies and ϵ perturbation bounds.
In addition, we provide extensive ablation study on the design choices we made, and sensitivity analysis on the temperature constant τ .
Conclusions
This paper identifies severe robustness overestimation in many ensemble defense techniques, and further investigates problem the robustness evaluation under three ensemble-forming strategies.
To efficiently and accurately evaluate the robustness of ensembles, we introduce MORA, a new attack technique which reweighs sub-model importance adaptively by their respective “ease-of-attack” during attack iterations.
MORA enjoys a much improved success rate and convergence rate compared with other SOTA attacks.
Moreover, we found several surprising observations related to ensemble defenses, for instance, (1) misleading a minority of sub-models is sufficient to fool the ensemble, (2) summing by logits is the simplest yet most robust way to form ensembles, (3) with adversarial training, ensemble defenses may actually harm robustness, etc.
We hope the above observations may help to guide future avenue on ensemble defenses, and provide a strong attack baseline for potential approaches.
Finally, MORA is open source with reproducible results and pre-trained models; and we continually update a leaderboard of ensemble defenses under various attack strategies.
Acknowledgements
This work is supported in part by National Key R&D Program of China (No. 2019YFB2102100), Key-Area Research and Development Program of Guangdong Province (No. 2020B010164003),
Science and Technology Development Fund of Macao S.A.R (FDCT) under No. 0015/2019/AKP, and Shenzhen Science and Technology Innovation Commission (No. JCYJ20190812160003719).
This work was carried out in part at SICC which is supported by SKL-IOTSC, University of Macau.
A Additional Results A.1 Larger Perturbations
Tables 3 and 4 compare the effectiveness of iterative methods [17, 3], learned attacks ( CAA [18]),
AutoAttack ( AA ) [5], MORA and MORA mt across various ensemble defense strategies with ResNet20 [12] sub-models under 3 ensemble-forming modes (softmax, voting and logits).
Here, Tableuses an ℓ ∞ perturbation bound ϵ = 0 . 02 , whereas Table 4 uses ϵ = 0 .
All results are re-runtimes and within ± 0 .
05% standard deviation.
Note that under ϵ = 0 .
02 most defense methods only retain up to 4% robust accuracies.
Dverge [30] with 8 sub-models however surprisingly has 23 .
31% robustness against the strongest MORA mt attack.
We believe this can be attributed to the fact that similarly to adversarial training, Dverge diversifies sub-models with adversarial examples from each other, rather than via explicit regularization as carried out by the other defenses.
It thus requires substantially higher training cost than the others ( ∼ 5 ×
ADP [19]).
Finally, Figure 5 compares the different ensemble-forming strategies under an increasing ℓ ∞ perturbation bound ϵ ∈ { 0 . 01 , 0 . 015 , 0 . 02 , 0 .
Note that across all configurations, logit-based ensembles are the most robust.
A.2 Convergence Speed Comparisons
Figures 6 and 7 compare the convergence speed of MORA against SOTA attacks for up toiterations.
As AA comprises multiple attack strategies, we extract its two gradient-based attacks (APGD-{CE,DLR}) to facilitate comparisons.
Each attack uses 5 restarts, each with up toiterations.
The horizontal axes show the iteration counts used, and the vertical axes denote the percentages of remaining unsuccessful images.
When forming ensembles with logits, MORA and “No reweighing” are both identical.
Note that MORA substantially outperforms most existing attacks under 500 iterations, and generally requires (up to 70 × ) fewer iterations to achieve the same accuracies of other attacks with 500 iterations.
A.3 Ablation and Sensitivity Analyses
In Table 5, we perform ablation of the individual components used in MORA.
We begin with the standard “PGD” attack [17] with 5 random restarts, each with 100 iterations, and a constant step size of ϵ/ 4 .
Each row then consecutively adds a new component.
“Momentum” introduces momentum ν = 0 .
75 as used in Algorithm 1, “Cosine Step Size” then replaces the constant step size with a cosine schedule α i = ϵ (1 + cos( iπ / I )) .
“Sub-model Logits” further exploits the sub-model logits directly following Section 3.1, and replaces random restarts with a β ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
75 , 1 } grid search to match the cost of 500 iterations.
“Logit Normalization” incorporates the normalization of logits as proposed in (12) of Section 3.3.
“Adaptive Reweighing” adaptively adjusts sub-model weights for attacking ensemble defenses ((10) of Section 3.2).
Finally, “Multiple Targets” [11] additionally uses 100 iterations of the targeted variant of L mora β,τ on each of the remaining 9 class labels (Section 3.3).
Increasing the temperature coefficient τ from 1 as used in (11) can affect the convergence speed of MORA.
To ensure a fair comparison in our results, we fix a constant τ = 5 for softmax-based ensembles, and τ = 10 for voting, as increasing τ may help improve convergence.
In addition, and
Figure 8 provides sensitivity analyses of τ on the three defending methods (ADP, GAL, Dverge).
Finally, Figure 9 shows the effect of varying β ∈ [0 , 1] , the interpolation between L mora β,τ and the ensemble’s original loss L sce .
Introducing L mora β,τ substantially improves the strength of attack.
Note that in our comparison results (Tables 1, 2, 3, 4 and 6), instead of 5 random restarts we use a β ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
75 , 1 } schedule to further improve the final attack success rate.
A.4 CIFAR-
Table 6 compares the attacks on PDD+DEG [13] defenses trained on CIFAR-100.
Similar to GAL [14] and TRS [31], PDD+DEG also diversifies sub-models gradients by minimizing cosine-similarities of gradients via regularization, and further diversifies feature selection with adaptive dropouts.
We report attacks on ensembles with three ensemble-forming methods (softmax, voting and logits).
Note that we only included PDD+DEG as other methods did not train on CIFAR-100.
[Table]
A.5 Failure Modes in Ensemble Defenses
Figure 10 provides a visualization of the loss surfaces of ADP [19] under 3 different ensembleforming strategies.
In this section, we continue the discussion of the two failure modes in ensemble defense that induce overestimated robustness as introduced in Section 1:
(a) Gradient obfuscation via ensemble-forming strategies.
It is evident that under PGD-10 attacks, both softmax- and voting-based ensembles (Figures 10a and 10b respectively) exhibit to some extent gradient obfuscation as they result in flatter loss surfaces in the adversarial direction g , whereas the logits-based variant does not (Figure 10c).
(b) Gradient diversification.
As sub-model gradients counteract, PGD attacks on softmax- and voting-based ensembles may result in an averaged gradient direction g that experience difficulty in
[Table]
Logits
ADP 92.
[Table]
[Table]
Dverge
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
GAL
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
TRS † 69.
increasing loss (Figures 10a and 10b respectively).
Adopting sub-model reweighing (bottom row in Figure 10) alleviates this difficulty, and allows the attack to succeed more reliably.
B Limitations and Potential Societal Impacts
The ℓ ∞ white-box threat model assumes the availability of the models’ gradients, which could present a challenge as such information may not be available to the attacker.
It is thus critical to evaluate white-box robustness accurately, as it provides the lower bounds on the robustness of ensemble defenses in practical scenarios.
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
We acknowledge that adversarial attacks may have the potential to be used by a malicious party, but we believe defending against such attacks is critically pertinent to the accurate evaluation of robustness.
We hope this paper furthers the understanding of ensemble robustness
[Figure]
and accurately evaluating adversarial robustness can help improve future defenses.
It is also noteworthy that the white-box threat model has applications in the context of advancements in deep learning
[Figure]
and can improve e
[Figure] g. , transfer learning [28, 6], GAN training [2], interpretability [22] [Figure] and etc
[Figure]
C Computational Resources
On NVIDIA Tesla V100 GPUs, MORA with 500 iterations uses up to 1 .
GPU-hours on each ensemble defense [Figure] and MORA mt uses up to 2 .
8 GPU-hours on the CIFAR-10 test set. The run time depends on the number of sub-models in an ensemble and its attack difficulty (Table 7).
[Figure]
D Licenses
Table 8 lists the relevant resources used in this paper and their respective licenses.
[Figure] [Figure]
Interpolation ( β )
A cc u r a c y ( % )
ADP 8 ADP
ADP(a)
ADP.
Interpolation ( β )
A cc u r a c y ( % )
GAL 8 GAL
GAL(b) GAL.
Interpolation ( β )
A cc u r a c y ( % )
Dverge 8 Dverge
Dverge(c) Dverge.
Defenses
[Table]
[Table] [Table] [Table]
[Figure]
[Table] [Table]