MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attack
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Adversarial attacks can deceive neural networks by adding tiny perturbationsto their input data.
Ensemble defenses, which are trained to minimize attacktransferability among sub-models, offer a promising research direction to improverobustness against such attacks while maintaining a high accuracy on natural inputs.
We discovered, however, that recent state-of-the-art (SOTA) adversarial attackstrategies cannot reliably evaluate ensemble defenses, sizeably overestimating theirrobustness.
This paper identiﬁes the two factors that contribute to this behavior.
First, these defenses form ensembles that are notably difﬁcult for existing gradientbased method to attack, due to gradient obfuscation.
Second, ensemble defensesdiversify sub-model gradients, presenting a challenge to defeat all sub-models simultaneously, simply summing their contributions may counteract the overallattack objective.
Yet, we observed that ensemble can still be fooled despite mostsub-models being correct.
We therefore introduce MORA, a model-reweighingattack to steer adversarial example synthesis by reweighing the importance ofsub-model gradients.
MORA discovered that recent ensemble defenses all exhibitvarying degrees of overestimated robustness.
Comparing it against recent SOTAwhite-box attacks, it can converge orders of magnitude faster while achieving higherattack success rates across all ensemble models examined with three differentensemble modes ( i.e. ensembling by either softmax, voting or logits).
In particular,most ensemble defenses exhibit near or exactly 0% robustness against MORA with(cid:96) ∞ perturbation within 0 .
02 on CIFAR-10, and 0 .
01 on CIFAR-100.
Introduction
Many safety-critical applications, such as autonomous robots [34], self-driving [8], search engines [24] etc. are becoming increasingly powerful and reliant on deep neural networks (DNNs).
Despite the monumental success of DNNs on these applications, they are highly susceptible toadversarial examples — an attacker can add tiny delibrate perturbations to the input data, misleadingthe model into giving incorrect results [23, 9].
Such adversarial attacks could pose a signiﬁcant threatto the safety and reliability of deep learning applications.
To mitigate this threat, many defense strategies [17, 33, 4] based on adversarial training [17] havebeen proposed to improve model robustness.
Adversarial training, however, gains robustness at theexpense of model accuracy on clean natural images [27].
Ensemble defenses [19, 14, 30, 31] havethus emerged to combine multiple predictions from independent sub-models.
The intuition is that anensemble of models can often lead to higher accuracy, while learning to stop adversarial exampletransfer among sub-models may improve robustness against adversarial attacks.
This approach couldpotentially offer a promising research direction to improve model robustness while preserving highaccuracy on natural inputs.
[Figure]
Yet surprisingly, under the white-box threat model, existing state-of-the-art (SOTA) adversarialattacks with strong performance on conventional DNN models performed poorly on ensemblemodels, sizeably overestimating their robustness (Figures 1a and 1b).
This also suggests, to someextent, that ensemble defenses may rely on two potential design ﬂaws below that cause obfuscatedgradients [1], i.e. they are either deliberately non-differentiable, or give no useful gradients, thusinducing overestimated robustness:
(a) Gradient obfuscation via ensemble-forming strategy.
They typically form ensembles by averagingprobability vectors (softmax) of sub-models, and softmax operations can easily cause gradientobfuscation.
While the model’s actual robustness is pertinent to the strategy used to form an ensemble,
this indicates that gradient-based attacks have to also leverage this effectively.
(b) Gradient diversiﬁcation.
Motivated by the reasoning that a majority of sub-models may need to be
fooled for successful attacks, they learn to reduce adversarial transferability among sub-models, oftenvia gradient diversiﬁcation.
This intuitively causes sub-models to counteract each other, averaging to
a small or inaccurate overall gradient.
Attacking only the ensemble loss would fool most sub-models,
but the ensemble may remain still correct; conversely, it is actually possible to fool an ensemble,
despite the majority of its sub-models giving correct predictions (Figure 1c).
From the above observations, it is perceivable that the practical evaluation of ensemble robustnesscannot be solely done by treating such models holistically.
To this end, this paper introduces MORA,model-reweighing attack, to adaptively adjust the importance of sub-models in attack iterations.
Sub-models are reweighed according to their respective “ease of attack”, which is in turn evaluatedby the gradient of the difference of ensemble classiﬁcation outputs w.r.t.
the ones of individualsub-models.
Pushing the limits of the current SOTA in ensemble robustness evaluation, it drawsinspiration from recent effective attack tactics, e.g. momentum [7, 5], step size schedule [5, 16], lossnormalization [32], and multiple targets [5, 25].
We summarize our contributions:
• This paper presents the ﬁrst extensive study on the robustness of ensemble defenses under
multiple ensemble-forming strategies.
• By reweighing the importance of sub-models that steers adversarial example synthesis,
we
show that gradient-based attacks on ensemble defenses can often be orders of magnitude
faster, while enjoying a higher success rate.
• Empirical results on a wide variety of different ensemble defenses show that MORA out-
performs competing attacks in both performance and convergence rate.
Finally, this paper
provides extensive ablation of its components and sensitivity analyses of hyperparameters.
To our best knowledge, MORA is currently the strongest attack against a wide range of ensembledefenses.
As existing ensemble defenses may rely on design decisions that stymie even the strongest competing attacks we tested, attacks on ensemble robustness should no longer treat the model underattack as a holistic loss function.
We make MORA open source with reproducible results and pretrained models; moreover, we maintain and continually update a leaderboard of ensemble defensesunder various attack strategies 1 .
Preliminaries & Related Work
2.1 Adversarial Attacks
An adversarial example adds a small perturbation, typically bounded a small value with (cid:96) p norms, tothe original image, such that the model under attack can be deceived into giving incorrect results.
Theadvent of adversarial attacks [23] has piqued the interest of deep learning practitioners, and revealedsecurity concerns of deep learning [26, 21], improved GAN training [2] transfer learning [28, 6],and DNN interpretability [22], etc.
Formally, assuming a defending classiﬁer f : I → R K , takingan input image x ∈ I =
[0 , 1] C × H × W as input, and arg max f ( x ) predicts the correct class labely ∈ C , then an attacker attempts to ﬁnd an adversarial example ˆ x in the set:
[Equation]
Here, ˆ x ∈ A (cid:15), x constrains the adversarial example ˆ x to be within both the input space
I and a 84 small (cid:15) -ball of (cid:96) p -distance from the original image x , or equivalently (cid:107) x − ˆ x (cid:107) p ≤ (cid:15) .
Satisfying the 85 condition arg max f ( x ) [-
-](cid:54) =
y means that f ( x ) fails to give the expected correct classiﬁcation y .
Wefocus on the (cid:96) ∞ white-box threat model commonly considered by the defenses examined in thispaper, which grants the attacker completely access to the internals of the defender, including, forinstance, its model architecture, parameters, training algorithms, etc.
One of the popular and effective white-box attacks used by many defenders to evaluate their robustnessis projected gradient descent (PGD) [17], which ﬁnds adversarial examples by maximizing theclassiﬁcation loss with gradient descent:
[Equation]
where L is typically the softmax cross-entropy (SCE) loss used to train the model, α i is the step size,and we let the initial ˆ x 0 (cid:44) P (cid:15), x ( x + µ ) .
The projection function P (cid:15), x ( v ) constrains its input v to be 94 within the feasible region
A (cid:15), x , and ﬁnally µ ∼ U ( − (cid:15), (cid:15) ) is a uniformly distributed noise bounded by 95 [ − (cid:15), (cid:15) ] .
Besides PGD, C&W [3] is also a gradient-based attack which, instead of projection, indirectlyconstrains the search space by regularization.
As PGD gains popularity, many defense mechanisms rely on it to evaluate their robustness.
Unfortunately, AutoAttack (AA) [5] ﬁnds that many of the defenses may inadvertently break PGD-basedattacks, which result in drastic overestimation of their robustness, and proposes to combine anensemble of diverse attacks to minimize robustness overestimation.
LAFEAT [32] learns to leverageintermediate layers of the DNN, and shows that attacking multiple layers can produce strongerattacks, but unfortunately it cannot be applied to ensemble defenses.
Adaptive Auto Attack ( A 3 )
[16]improves attack success rates by using the gradient directions to prescribe a more effective initialrandom perturbation.
As defenders may design mechanisms to circumvent existing attacks, Adaptive attacks [25] manually tailor speciﬁc attack strategies for an extensive set of defenses.
Finally,
Composite Adversarial Attacks (CAA) [18] further combine a large selection of attack methods,and use a genetic algorithm to learn an optimal attacking sequence.
In comparison, MORA is auniﬁed approach which uses only one attack algorithm, does not require a compute-intensive learningprocedure, and yet it still achieves fast and SOTA estimation of ensemble robustness.
2.2 Defending Against Adversarial Attacks
Defending against adversarial attacks can be deﬁned as a saddle-point problem to minimize thetraining loss on adversarial examples [17] with samples ( x , y ) drawn from the training set:
[Equation]
A direct optimization-based approach to approximately solving the above problem is adversarialtraining [17], i.e. to train the DNN model with its own adversarial examples.
Training DNNs to1 Anonymized link. be robust is, however, a challenging endeavor.
First, it may be much more compute intensive astraining examples are typically generated with PGD [17], requiring a few forward/backward passesof the DNN.
Second, to avoid overﬁtting, it requires stopping training early, a much larger sizeof the training set [4], and using improved data augmentation [20] or generated data [10].
Thirdly,as noted by other literatures [5, 32], currently no other design choices can rival the robustnessprovided by adversarial training, and notably, many defense strategies are considered harmful tomodel robustness [25].
Finally, the resulting models often cannot achieve high clean accuracy [27].
2.3 Ensemble Defenses & Ensemble-forming Strategies
Ensemble-based defense techniques may pave an alternative path to address the challenges ofadversarial robustness, as they could potentially work around the above limitations of adversarialtraining.
Adopting the theme of minimizing adversarial example transferability across sub-models,each ensemble defense proposed unique solutions.
ADP [19] increases the orthogonality of nonmaximal class logits among sub-models to encourage diversity.
GAL [14] minimizes a gradientalignment loss, which directly reduces the cosine-similarity between sub-models.
Building on top ofthis, TRS [31] further regularizes the smoothness of the loss function, as gradient orthogonality withsmoothness may further diversify sub-models.
Dverge [30] instead uses the adversarial examplesof a sub-model to train another sub-model, thus lowering transferability.
Ensemble defenses arealso particularly interesting, as they are the last line of defense against even the strongest existingwhite-box attacks without resorting to adversarial training, showing a certain degree of robustness.
Besides the above mechanisms for training a successful ensemble defense, there exists different waysto combine sub-model predictions.
Let us assume that an ensemble defense trains M sub-models,f m : I → R K for m ∈ [1 : M ] , an ensemble f E : I → R K thus forms a ﬁnal classiﬁcation result by 137 combining individual decisions from the sub-models, namely
[Equation]
[Equation]
where ens is the ensemble-forming operator.
In this paper, we investigate ens ∈ { softmax , wta , id } , 139 where the potential choices respectively denoting forming an ensemble from sub-model outputsf m ( x ) by either summing predicted probabilities (evaluated with softmax ), or majority votes (usingwta , the winner-take-all operator), or simply summing the logits (with id , the identity operator).
Defending ensemble methods [19, 14, 30, 31] tested in this paper all employed the softmax -basedstrategy to report their robustness.
Methods that are exceptions to these options exist, for instance,
ECOC [29] allows sub-models to produce binary predictions, and use error correcting codes basedon the Hamming distance to combine the predictions into classiﬁcation outputs.
This approach isunfortunately not robust, and the added complexity is error-prone and may harm robustness [25].
Moreover, as the voting ( wta ) strategy is non-differentiable, an attacker can soften it approximatelyusing a softmax operation with temperature τ , where we used τ = 0 .
universally:
[Equation]
The Model-Reweighing Attack (MORA)
3.1 Problem Formulation & High-Level Overview
As discussed in Section 1, existing ensemble defenses may obfuscate gradients with the ensembleforming mode and gradient diversiﬁcation, such that the ﬁnal loss of the ensemble model can nolonger provide effective signals for gradient descent.
It is therefore desirable to ﬁnd an alternative
L to the original SCE loss L sce on the ensemble, such that for a given number of iterations I , theoriginal L sce loss can be maximized: [Equation]
where
[Equation]
M ] (ˆ x i ) , y )) , (6) and PGD( · ) denotes a PGD step along the gradient of loss function L ( f E (ˆ x i ) , f [1: M ] (ˆ x i ) , y ) , whichnot only takes the ensemble predictions f E (ˆ x i ) as input, but can further utilize sub-model predictions
[Table]
+YuDXH7g2IolvsZ/yUEE3Fh3BAK3Ucg8CBdhjIPOr4i4ffrTKVaKhKFp5EHGEQxogZEXLLXsVbwg6TfwxKZMxrlvuV9BOWKZ4jEyCMU3fSzHMQaNgkhelIDM8BXYPXd60NAbFTZgPTyronlXatJNo+2KkQ/V3Rw7KmL6KbOVgZzPpDcT/vGaGnbMwF3GaIY/ZaFAnkxQTOsiHtoXmDGXfEmBa2F0p64EGhjbFkg3Bnzx5mtSOKv5Jxbs5Lp8fjeNYIjtkl+wTn5ySc3JJrkmVMPJInskreXOenBfn3fkYlc44455t8gfO5w/yEJ7W</latexit>
L mora ,⌧ f [1: M ] to guide the PGD iterations. The challenge at hand is, therefore, to ﬁnd a suitable L which cangenerate attacks on ensemble defenses efﬁciently and effectively.
MORA aims to provide a potential optimization route towards the above problem formulation.
Namely, in addition to the original output of the ensemble f E (ˆ x ) , we leverage the sub-modelpredictions f [1: M ] (ˆ x ) to facilitate the optimization.
By way of illustration, Figure 2 shows a highlevel overview of the model-reweighing attack, where we compliment the ensemble loss, with anewly added sub-model reweighing loss L mora β,τ , as an auxiliary attack vector alongside the originalobjective.
Not only can the new loss bypass the ensemble-forming strategy to work around itsobfuscated gradients, but it further exploits information present in the individual sub-model andensemble predictions to steer the direction of adversarial example synthesis.
3.2 Adaptive Sub-model Importance
Before we begin, assume that z [ m ] (cid:44) f m ( x ) represents the m th sub-model output, and let z [ m ] t denotethe corresponding logit of label t .
We deﬁne the difference of logits (DL) [3] k [ m ] (cid:44) z
[ m ] y − z [ m ] ˆ y ,which is the difference between the predictions of the ground truth z
[ m ] y and the maximum of theremaining classes
z [ m ] ˆ y (cid:44) max i ∈C /y
z
[ m ] i , where C /y is the set of all class labels except y .
Similarly,we let z [E] (cid:44) f E ( x ) , and z [E] t and k [E] be the respective variants of the ensemble prediction.
It isnotable that a successful attack happens when k [E] < 0 , and similarly k [ m ] < 0 means the m thsub-model is producing incorrect classiﬁcation.
Ensemble defenses tend to diversify sub-model gradients, for instance, ADP [19] minimizes thecosine-similarity (cid:104)∇ (cid:96) ( z [ a ] ) , ∇ (cid:96) ( z [ b ] ) (cid:105) among each loss function gradient pairs of sub-models (cid:96) ( z [ a ] )and (cid:96) ( z [ b ] ) .
Their intuition is that it may lower transferability among these sub-models, such thatattacks with the overall gradient of the ensemble, i.e. ∇ (cid:96) ( z [E] ) = 1 M (cid:80) m ∈
[1: M ] ∇ (cid:96) ( z [ m ] ) , are becoming less effective in misleading all sub-models simultaneously, as individual gradients in ∇ (cid:96) ( z [ m ] )are encouraged to be orthogonal to each other.
To this end, we propose to reweigh the importance ofsub-models, by instead considering the modiﬁed gradient:
[Equation]
where λ [ m ] τ assigns weights to important sub-models to contribute more heavily to the attack gradient.
While the adversarial examples of the ensemble could present a challenge to discover, individualsub-models are weak defenders which can be easily defeated.
Based on this property, we propose toweigh sub-models importance with the rate of change in k [E] w.r.t.
that of k [ m ] , i.e. sub-models wouldbe given higher weights if attacking it would bring a signiﬁcant change to the ensemble’s prediction.
Following this idea, for all ensemble-forming strategies (softmax, voting, logits), we rewrite k
[E] as afunction of k [ m ] , where the term below can become a function of k [ m ] :
[Equation]
1: function MORA_ Attack ( f [1: M ] , x , y, β, τ, ν, (cid:15), I ) 2: ˆ x 0 ← P (cid:15), x ( x + u ) , where u ∼ U ( − (cid:15), (cid:15) )
(cid:46) Random init 3: µ 0 ← 0 4: for i ∈ [0 : I − 1] do 5: z
[ m ] ← f m (ˆ x i ) for all m ∈ [1 : M ] (cid:46) Sub-model predictions 6: z [E] ← 1 M (cid:80) m ∈ [1: M ] ens( z [ m ] )
(cid:46) Ensemble prediction 7: k [E] ← z [E] y − z
[E] ˆ y 8: if k [E] ≤ 0 then return ˆ x i (cid:46) Successful attack 9: g i +1 ← sign( ∇ ˆ x i L mora β,τ ( z [1: M ] , z [E] , y )) (cid:46) Sign-gradient with the MORA loss 10: α ← (cid:15) (1 + cos( iπ / I ))
(cid:46) Cosine step-size schedule 11: µ i +1 ← P (cid:15), x ( µ i
+ α g i +1 )
(cid:46)
Iterative update 12: ˆ x i +1 ← P (cid:15), x (ˆ x i + ν ( µ i +1 − ˆ x i )
+ (1 − ν ) (ˆ x i − ˆ x i − 1 ))
(cid:46) ...with momentum 13: end for 14: return ˆ x I (cid:46) Give up after I iterations 15: end function
The weights are thus deﬁned as follows: [Equation] While it is possible to compute the weights using gradient back-propagation, we can simply derivethe following closed-form solution of the weights for each of the three ensemble-forming strategies.
For wta , we use the softened version of wta as deﬁned in (5) and can derive the weights as follows:
[Equation]
{+Here 1[ k [ m ] > 0] is the indicator function that equals 
+}
if k [ m ] > 0 , or 0 otherwise, effectivelystopping the attack on the m th sub-model upon success, and the detach operator admits no backwardpropagation to its input.
In the case of using sums of sub-model softmax outputs to form an ensembledecision, i.e. ens = softmax , it is a special case of softwta τ where the temperature coefﬁcient canbe ﬁxed at τ = 1 .
Finally, when ens = id , i.e. forming ensembles by summing logits, λ [ m ] τ ( z [ m ] )simply reduces to 1[ k [ m ] > 0] for the m th sub-model.
3.3 The MORA Loss
For reference, defenses mechanisms we examine in this paper aim to ﬁnd ˆ x which maximizes the
SCE loss L sce ( z [E] , y ) , to evaluate the ensemble robustness.
The MORA loss improves this furtherby proposing two additional modiﬁcations to the untargeted loss function used to attack ensembles:
[Equation]
First, it additionally introduces a sum of the λ [ m ] τ -weighted variant of sub-model logits, in order toexpose sub-model logits with adaptive reweighing described in Section 3.2.
Second, β interpolatesthe importance of the newly added auxiliary logits and the original ensemble logits.
Finally, inspiredby the effective surrogate loss in [32], it further normalizes the logits by their respective DL using:
[Equation]
Finally, the targeted variant of the MORA loss simply replaces y with t where t is the intended target.
3.4 Improving the State-of-the-art
While the new L mora β,τ loss is highly effective against ensemble defenses we test in this paper, we strivefor further advances in MORA’s ability to generate faster and better adversarial examples.
Inspiredby recent publications, we borrow ideas from related adversarial attack tactics, which includesadopting a cosine step-size schedule [16], momentum [7, 5], random restarts [25] and multiple targetattacks [5, 25].
We provide the overall algorithm in Algorithm 1, which computes an adversarialimage ˆ
x I as return, by taking as input the sub-models f [1: M ] , natural image x , ground truth label y ,β to interpolate between the auxiliary logits and the original, τ controls the temperature, momentumµ = 0 .
75 following [32, 5], (cid:15) perturbation bound, and ﬁnally the maximum number of iterations I .
[Table]
Experimental Results
We compare MORA against SOTA attacks for a wide range of ensemble defenses under threeensemble-forming strategies (softmax, voting, and logits).
We use pre-trained ResNet-20 [12] modelsfrom ADP [19], Dverge [30], GAL [14], and reproduced TRS [31] using the same architecture withofﬁcial source code, as pre-trained models were unavailable.
Our robustness evaluation considersthe (cid:96) ∞ white-box attacks on the CIFAR-10 test set [15], with perturbation (cid:15) = 0 .
01 unless speciﬁed.
The full comparison results can be found in Table 1; larger (cid:15) comparisons, and similar results on
CIFAR-100 models are in Appendix A.
We provide our key observations below.
Traditional attacks may fail to break through gradient obfuscation.
We reproduce two traditionalwhite-box attacks, i.e. projected gradient descent (PGD) [17] and C&W [3] with 5 random restarts,each with a maximum of 100 iterations, giving a total of 500 iterations.
PGD uses a ﬁxed step sizeof (cid:15)/ 4 .
For a fair comparison, MORA with 500 iterations sweeps β ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
75 , 1 } , witheach β up to 100 iterations.
Even with a 500 iteration budget, it is clear that PGD and C&W maysubstantially overestimate robustness, especially when tested under the softmax and voting ensembleforming options, and MORA can work around this obstacle thanks to its attacks on sub-models.
Diversiﬁed gradients can hamper even integrated attacks with large arsenals.
Moreover, wetest the defenses against recent integrated attacks with SOTA baselines on robustness evaluation,namely Adaptive Auto Attack ( A 3 ) [16], AutoAttack (AA) [5], and Composite Adversarial Attacks(CAA)
[18], which comprise large arsenals of various attack strategies.
We reproduce CAA following [18] to search for the attack policy before evaluating the defending models.
Note that itscomputational complexity is thus much higher than the other attacks, but we only report its test-timecomplexity.
In particular, while they enjoyed much higher success rates than PGD and C&W, somedefenses render their tactics ineffective.
We observe, e.g. , sizeable robustness overestimations on
ADP [19] under softmax and voting, which explicitly diversiﬁes sub-model gradients.
As MORA candynamically re-adjust sub-model importance w.r.t.
their “ease-of-attack”, it performs substantiallybetter with much fewer iterations.
In addition to the earlier 500 iterations, the multi-targeted MORA mttargets the remaining 9 class labels with 100 iterations for each label and β ﬁxed at 0 .
Others alsouse multi-targeted attacks along with respective tactics.
Robustness of most sub-models vs. robustness of ensemble.
We ﬁnd that robustness of a majority ofsub-models (fooling 3 / 8 for softmax and 2 / 8 for logits) usually do not translate to the overall robustnessof the ensemble (Figures 1c and 3a).
As voting requires breaking 1 / 2 sub-models simultaneously(Figure 3b), it is perceivable that using voting may give rise to a higher overall robustness.
Yetsurprisingly, for most defending ensembles, voting performs worse than softmax and logits.
Ensemble-forming strategies may give a false sense of security.
On one hand, softmax and votingstrategies exhibit substantially larger overestimated robustness (up to 40%) than logits.
On the otherhand, in stark contrast to the proposed use of softmax from [14, 19, 30, 31], we ﬁnd summing bylogits can form ensembles that are notably more robust than the other two (Figure 3c), while attackersonly needs to successfully deceive a few sub-models (refering back to Figure 1c).
Up to 60 × faster convergence under 500 iterations.
Figure 4 compares the convergence speed of MORA against AA losses, C&W, and PGD on defending ensembles.
MORA converges substantiallyfaster than the other attacks, using only up to 31 steps to match AA losses with 500 iterations.
Ensemble defense mechanisms may be at odds with robustness.
In Table 2, we compare respectiveattacks on adversarially trained Dverge models.
To our surprise, forming larger ensembles is actuallyharmful to the robustness of ensemble. [Figure] [Figure] [Figure]
[Table]
[Table] [Table]
Voting 83.
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
Logits [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
Additional results, ablation and sensitivity analyses.
Due to the page limit, we provide full resultsof relevant ﬁgures in Appendix A, note that the above key observations still hold true for all ensembledefenses we test under different ensemble-forming strategies and (cid:15) perturbation bounds.
In addition,we provide extensive ablation study on the design choices we made, and sensitivity analysis on thetemperature constant τ .
Conclusions
This paper identiﬁes severe robustness overestimation in many ensemble defense techniques, andfurther investigates problem the robustness evaluation under three ensemble-forming strategies.
Toefﬁciently and accurately evaluate the robustness of ensembles, we introduce MORA, a new attacktechnique which reweighs sub-model importance adaptively by their respective “ease-of-attack”during attack iterations.
MORA enjoys a much improved success rate and convergence rate comparedwith other SOTA attacks.
Moreover, we found several surprising observations related to ensembledefenses, for instance, (1) misleading a minority of sub-models is sufﬁcient to fool the ensemble,(2) summing by logits is the simplest yet most robust way to form ensembles, (3) with adversarialtraining, ensemble defenses may actually harm robustness, etc.
We hope the above observations mayhelp to guide future avenue on ensemble defenses, and provide a strong attack baseline for potentialapproaches.
Finally, MORA is open source with reproducible results and pre-trained models; and wecontinually update a leaderboard of ensemble defenses under various attack strategies.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] This paper presents an extensive study that identiﬁesexisting ensemble defenses may overestimate their robustness, as evinced by Figure 1,ﬁnds ensemble-forming strategies can affect its robustness (Table 1), and attackinga minority of sub-models is sufﬁcient to fool the ensemble (Figures 1c and 3a).
Itpresents model-reweighing attack (MORA), a new white-box (cid:96) ∞ attack which canprovide faster and more reliable robustness evaluation on ensemble defenses comparedto PGD [17], C&W [3], AutoAttack (AA) [5], Adaptive Auto Attack ( A 3 ) [16], and
Composite Adversarial Attacks (CAA) [18].
(b) Did you describe the limitations of your work?
[Yes] We provide a detailed description
of the limitations in Appendix B.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Adver-
sarial attacks may have the potential to be used by a malicious party, but this paper
speciﬁcally considers the (cid:96) ∞ white-box threat model, which assumes the availabil-
ity of the models’ gradients, which could present a challenge to execute maliciously.
In
addition, understanding and accurately evaluating adversarial robustness can help improve future defenses.
However, it is noteworthy that the white-box threat model has applications in the context of advancements in deep learning, and can improve e.
g. transfer learning , 6], GAN training 2], interpretability 22], and etc.
For more please refer to Appendix B.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include the
relevant materials in the supplemental material.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We provide the relevant details on the hyperparameters, and
a sensitivity analyses, and further provide an ablation study of the components in
Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Adversarial attacks do not generally have large variance
in the results. In Table 1, the results are re-run 5 times and within ± 0 .
05% standard
deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to Appendix C.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use source codefrom GAL [14], ADP [19], TRS [31] and Dverge [30] and their respective pre-trainedmodels.
We used the CIFAR-100 and CIFAR-10 datasets [15] for evaluation.
We citethe relevant authors in each case.
(b) Did you mention the license of the assets?
[Yes] We provide the licenses in Ap- pendix D.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
We provide only the source code and instructions for reproduction.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A] We use data under the conditions of the respective licenses.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] We do not include such data.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Additional Results
A.1 Larger Perturbations
Tables 3 and 4 compare the effectiveness of iterative methods [17, 3], learned attacks ( CAA [18]),
AutoAttack ( AA ) [5], MORA and MORA mt across various ensemble defense strategies with ResNet20 [12] sub-models under 3 ensemble-forming modes (softmax, voting and logits).
Here, Table 3uses an (cid:96) ∞ perturbation bound (cid:15) = 0 .
02 , whereas Table 4 uses (cid:15)
= 0 .
All results are re-run 5times and within ± 0 .
05% standard deviation.
Note that under (cid:15) = 0 .
02 most defense methods onlyretain up to 4% robust accuracies.
Dverge [30] with 8 sub-models however surprisingly has 23 .
31%robustness against the strongest MORA mt attack.
We believe this can be attributed to the fact thatsimilarly to adversarial training, Dverge diversiﬁes sub-models with adversarial examples from eachother, rather than via explicit regularization as carried out by the other defenses.
It thus requiressubstantially higher training cost than the others ( ∼ 5 ×
ADP
Finally, Figure 5 compares the different ensemble-forming strategies under an increasing (cid:96) ∞ perturbation bound (cid:15) ∈ { 0 . 01 , 0 . 015 , 0 . 02 , 0 . 025 , 0 . 03 } .
Note that across all conﬁgurations, logit-basedensembles are the most robust.
A.2 Convergence Speed Comparisons
Figures 6 and 7 compare the convergence speed of MORA against SOTA attacks for up to 500iterations.
As AA comprises multiple attack strategies, we extract its two gradient-based attacks(APGD-{CE,DLR}) to facilitate comparisons.
Each attack uses 5 restarts, each with up to 100iterations.
The horizontal axes show the iteration counts used, and the vertical axes denote thepercentages of remaining unsuccessful images.
When forming ensembles with logits, MORA and“No reweighing” are both identical.
Note that MORA substantially outperforms most existingattacks under 500 iterations, and generally requires (up to 70 × ) fewer iterations to achieve the sameaccuracies of other attacks with 500 iterations.
A.3 Ablation and Sensitivity Analyses
In Table 5, we perform ablation of the individual components used in MORA.
We begin with thestandard “PGD” attack [17] with 5 random restarts, each with 100 iterations, and a constant step sizeof (cid:15)/ 4 .
Each row then consecutively adds a new component.
“Momentum” introduces momentumν = 0 .
75 as used in Algorithm 1, “Cosine Step Size” then replaces the constant step size with a cosineschedule α i = (cid:15) (1 + cos( iπ / I )) .
“Sub-model Logits” further exploits the sub-model logits directlyfollowing Section 3.1, and replaces random restarts with a β ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
75 , 1 } grid search tomatch the cost of 500 iterations.
“Logit Normalization” incorporates the normalization of logits asproposed in (12) of Section 3.3.
“Adaptive Reweighing” adaptively adjusts sub-model weights forattacking ensemble defenses ((10) of Section 3.2).
Finally, “Multiple Targets” [11] additionally uses100 iterations of the targeted variant of L mora β,τ on each of the remaining 9 class labels (Section 3.3).
Increasing the temperature coefﬁcient τ from 1 as used in (11) can affect the convergence speedof MORA.
To ensure a fair comparison in our results, we ﬁx a constant τ = 5 for softmax-basedensembles, and τ = 10 for voting, as increasing τ may help improve convergence.
In addition, and
Figure 8 provides sensitivity analyses of τ on the three defending methods (ADP, GAL, Dverge).
Finally, Figure 9 shows the effect of varying β ∈ [0 , 1] , the interpolation between L mora β,τ and theensemble’s original loss
L sce .
Introducing L mora β,τ substantially improves the strength of attack.
Notethat in our comparison results (Tables 1, 2, 3, 4 and 6), instead of 5 random restarts we use aβ ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
75 , 1 } schedule to further improve the ﬁnal attack success rate.
A.4 CIFAR-100
Table 6 compares the attacks on PDD+DEG [13] defenses trained on CIFAR-100.
Similar to GAL [14]and TRS [31], PDD+DEG also diversiﬁes sub-models gradients by minimizing cosine-similaritiesof gradients via regularization, and further diversiﬁes feature selection with adaptive dropouts.
Wereport attacks on ensembles with three ensemble-forming methods (softmax, voting and logits).
Notethat we only included PDD+DEG as other methods did not train on CIFAR-100.
[Table]
[Table]
[Table]
[Table]
[Table]
Logits
ADP
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Dverge
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
GAL
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
TRS
[Table]
[Table]
[Table]
[Table]
[Table]
A.
Figure 10 provides a visualization of the loss surfaces of ADP [19] under 3 different ensembleforming strategies.
In this section, we continue the discussion of the two failure modes in ensembledefense that induce overestimated robustness as introduced in Section 1:
(a) Gradient obfuscation via ensemble-forming strategies.
It is evident that under PGD-10 attacks,both softmax- and voting-based ensembles (Figures 10a and 10b respectively) exhibit to some extentgradient obfuscation as they result in ﬂatter loss surfaces in the adversarial direction g , whereas thelogits-based variant does not (Figure 10c).(b)
Gradient diversiﬁcation.
As sub-model gradients counteract, PGD attacks on softmax- and
voting-based ensembles may result in an averaged gradient direction g that experience difﬁculty in
[Table]
increasing loss (Figures 10a and 10b respectively).
Adopting sub-model reweighing (bottom rowin Figure 10) alleviates this difﬁculty, and allows the attack to succeed more reliably.
B Limitations and Potential Societal Impacts
The (cid:96) ∞ white-box threat model assumes the availability of the models’ gradients, which could presenta challenge as such information may not be available to the attacker.
It is thus critical to evaluatewhite-box robustness accurately, as it provides the lower bounds on the robustness of ensembledefenses in practical scenarios.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
We acknowledge that adversarial attacks may have the potential to be used by a malicious party,but we believe defending against such attacks is critically pertinent to the accurate evaluation ofrobustness.
We hope this paper furthers the understanding of ensemble robustness, and accuratelyevaluating adversarial robustness can help improve future defenses.
It is also noteworthy that thewhite-box threat model has applications in the context of advancements in deep learning, and canimprove e.g. transfer learning [28, 6], GAN training [2], interpretability [22], and etc.
C Computational Resources
{+On NVIDIA Tesla V100 GPUs, MORA with 500 iterations uses up to 1 . 
+}
GPU-hours on each ensemble defense, and MORA mt uses up to 2 .
8 GPU-hours on the CIFAR-10 test set run time number sub-models ensemble its difﬁculty
[Figure]
D
Licenses
Table 8 lists the relevant resources used in this paper and their respective licenses.
[Figure]
[Figure]
Interpolation ( β )
A cc u r a c y ( % )
ADP 8 ADP
ADP(a)
ADP.
Interpolation ( β )
A cc u r a c y ( % )
GAL 8 GAL
GAL(b) GAL.
Interpolation ( β )
A cc u r a c y ( % )
Dverge 8 Dverge
Dverge(c) Dverge.
[Table]
[Table]
[Figure]
[Table]
[Table]