MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attack
Abstract
Adversarial attacks can deceive neural networks by adding tiny perturbations to their input data.
Ensemble defenses, which are trained to minimize attack transferability among sub-models, offer a promising research direction to improve robustness against such attacks while maintaining a high accuracy on natural inputs.
We discover, however, that recent state-of-the-art (SOTA) adversarial attack strategies cannot reliably evaluate ensemble defenses, sizeably overestimating their robustness.
This paper identiﬁes the two factors that contribute to this behavior.
First, these defenses form ensembles that are notably difﬁcult for existing gradientbased method to attack, due to gradient obfuscation.
Second, ensemble defenses diversify sub-model gradients, presenting a challenge to defeat all sub-models simultaneously, simply summing their contributions may counteract the overall attack objective; yet, we observe that ensemble may still be fooled despite most sub-models being correct.
We therefore introduce MORA, a model-reweighing attack to steer adversarial example synthesis by reweighing the importance of submodel gradients.
MORA ﬁnds that recent ensemble defenses all exhibit varying degrees of overestimated robustness.
Comparing it against recent SOTA white-box attacks, it can converge orders of magnitude faster while achieving higher attack success rates across all ensemble models examined with three different ensemble modes ( i.e. , ensembling by either softmax, voting or logits).
In particular, most ensemble defenses exhibit near or exactly 0% robustness against MORA with ` 1 perturbation within 0 .
02 on CIFAR-10, and 0 .
01 on CIFAR-100.
We make
MORA open source with reproducible results and pre-trained models; and provide a leaderboard of ensemble defenses under various attack strategies 1 .
Introduction
Many safety-critical applications, such as autonomous robots [34], self-driving [8], search engines [24], etc. are becoming increasingly powerful and reliant on deep neural networks (DNNs).
Despite the monumental success of DNNs on these applications, they are highly susceptible to adversarial examples — an attacker can add tiny delibrate perturbations to the input data, misleading the model into giving incorrect results [23, 9].
Such adversarial attacks could pose a signiﬁcant threat to the safety and reliability of deep learning applications.
To mitigate this threat, many defense strategies [17, 33, 4] based on adversarial training [17] have been proposed to improve model robustness.
Adversarial training, however, gains robustness at the
[Figure]
expense of model accuracy on clean natural images [27].
Ensemble defenses [19, 14, 30, 31] have thus emerged to combine multiple predictions from independent sub-models.
The intuition is that an ensemble of models can often lead to higher accuracy, while learning to stop adversarial example transfer among sub-models may improve robustness against adversarial attacks.
This approach could potentially offer a promising research direction to improve model robustness while preserving high accuracy on natural inputs.
Yet surprisingly, under the white-box threat model, existing state-of-the-art (SOTA) adversarial attacks with strong performance on conventional DNN models performed poorly on ensemble models, sizeably overestimating their robustness (Figures 1a and 1b).
This also suggests, to some extent, that ensemble defenses may rely on two potential design ﬂaws below that cause obfuscated gradients [1], i.e. , they are either deliberately non-differentiable, or give no useful gradients, thus inducing overestimated robustness:
(a) Gradient obfuscation via ensemble-forming strategy.
They typically form ensembles by averaging probability vectors (softmax) of sub-models, and softmax operations can easily cause gradient obfuscation.
While the model’s actual robustness is pertinent to the strategy used to form an ensemble, this indicates that gradient-based attacks have to also leverage this effectively.
(b) Gradient diversiﬁcation.
Motivated by the reasoning that a majority of sub-models may need to be fooled for successful attacks, they learn to reduce adversarial transferability among sub-models, often via gradient diversiﬁcation.
This intuitively causes sub-models to counteract each other, averaging to a small or inaccurate overall gradient.
Attacking only the ensemble loss would fool most sub-models, but the ensemble may remain still correct; conversely, it is actually possible to fool an ensemble, despite the majority of its sub-models giving correct predictions (Figure 1c)
From the above observations, it is perceivable that the practical evaluation of ensemble robustness cannot be solely done by treating such models holistically.
To this end, this paper introduces MORA, model-reweighing attack, to adaptively adjust the importance of sub-models in attack iterations.
Sub-models are reweighed according to their respective “ease of attack”, which is in turn evaluated by the gradient of the difference of ensemble classiﬁcation outputs w.r.t.
the ones of individual sub-models.
Pushing the limits of the current SOTA in ensemble robustness evaluation, it draws inspiration from recent effective attack tactics, e.g. , momentum [7, 5], step size schedule [5, 16], loss normalization [32], and multiple targets [5, 25].
We summarize our contributions:
• This paper presents the ﬁrst extensive study on the robustness of ensemble defenses under multiple ensemble-forming strategies.
• By reweighing the importance weights of sub-models to steer adversarial example synthesis, we show that gradient-based attacks on ensemble defenses can often be orders of magnitude faster, while enjoying a higher success rate.
• Empirical results on a wide variety of different ensemble defenses show that MORA outperforms competing attacks in both performance and convergence rate.
Finally, this paper provides extensive ablation of its components and sensitivity analyses of hyperparameters.
To our best knowledge, MORA is currently the strongest attack against a wide range of ensemble defenses.
We make MORA open source with reproducible results and pre-trained models; moreover, we maintain a leaderboard of ensemble defenses under various attack strategies.
Preliminaries & Related Work 2.1 Adversarial Attacks
An adversarial example adds a small perturbation, typically bounded a small value with ` p norms, to the original image, such that the model under attack can be deceived into giving incorrect results.
The advent of adversarial attacks [23] has piqued the interest of deep learning practitioners, and revealed security concerns of deep learning [26, 21], improved GAN training [2] transfer learning [28, 6], and DNN interpretability [22], etc.
Formally, assuming a defending classiﬁer f : I !
R K , taking an input image x 2 I = [0 , 1] C ⇥ H ⇥ W as input, and arg max f ( x ) predicts the correct class label y 2 C , then an attacker attempts to ﬁnd an adversarial example ˆ x in the set:
[Equation]
Here, ˆ x 2 A ✏ , x constrains the adversarial example ˆ x to be within both the input space I and a small ✏ -ball of ` p -distance from the original image x , or equivalently k x  ˆ x k p  ✏ .
Satisfying the condition arg max f ( x ) 6 =
y means that f ( x ) fails to give the expected correct classiﬁcation y .
We focus on the ` 1 white-box threat model commonly considered by the defenses examined in this paper, which grants the attacker completely access to the internals of the defender, including, for instance, its model architecture, parameters, training algorithms, etc.
One of the popular and effective white-box attacks used by many defenders to evaluate their robustness is projected gradient descent (PGD) [17], which ﬁnds adversarial examples by maximizing the classiﬁcation loss with gradient descent:
[Equation]
where L is typically the softmax cross-entropy (SCE) loss used to train the model, ↵ i is the step size, and we let the initial ˆ x 0 , P ✏ , x ( x + µ ) .
The projection function P ✏ , x ( v ) constrains its input v to be within the feasible region A ✏ , x , and ﬁnally µ
⇠ U (  ✏ , ✏ ) is a uniformly distributed noise bounded by [  ✏ , ✏ ] .
Besides PGD, C&W [3] is also a gradient-based attack which, instead of projection, indirectly constrains the search space by regularization.
As PGD gains popularity, many defense mechanisms rely on it to evaluate their robustness.
Unfortunately, AutoAttack (AA) [5] ﬁnds that many of the defenses may inadvertently break PGD-based attacks, which result in drastic overestimation of their robustness, and proposes to combine an ensemble of diverse attacks to minimize robustness overestimation.
LAFEAT [32] learns to leverage intermediate layers of the DNN, and shows that attacking multiple layers can produce stronger attacks, but unfortunately it cannot be applied to ensemble defenses.
Adaptive Auto Attack ( A 3 ) [16] improves attack success rates by using the gradient directions to prescribe a more effective initial random perturbation.
As defenders may design mechanisms to circumvent existing attacks, Adaptive attacks [25] manually tailor speciﬁc attack strategies for an extensive set of defenses.
Finally,
Composite Adversarial Attacks (CAA) [18] further combine a large selection of attack methods, and use a genetic algorithm to learn an optimal attacking sequence.
In comparison, MORA is a uniﬁed approach which uses only one attack algorithm, does not require a compute-intensive learning procedure, and yet it still achieves fast and SOTA estimation of ensemble robustness.
2.2 Defending Against Adversarial Attacks
Defending against adversarial attacks can be deﬁned as a saddle-point problem to minimize the training loss on adversarial examples [17] with samples ( x , y ) drawn from the training set:
[Equation]
where L is the training loss, e.g. , the SCE loss.
A direct optimization-based approach to approximately solving the above problem is adversarial training [17], i.e. , to train the DNN model with its own adversarial examples.
Training DNNs to be robust is, however, a challenging endeavor.
First, it may be much more compute intensive as training examples are typically generated with PGD [17], requiring a few forward/backward passes of the DNN.
Second, to avoid overﬁtting, it requires stopping training early, a much larger size of the training set [4], and using improved data augmentation [20] or generated data [10].
Thirdly, as noted by other literatures [5, 32], currently no other design choices can rival the robustness provided by adversarial training, and notably, many defense strategies are considered harmful to model robustness [25].
Finally, the resulting models often cannot achieve high clean accuracy [27].
2.3 Ensemble Defenses & Ensemble-forming Strategies
Ensemble-based defense techniques may pave an alternative path to address the challenges of adversarial robustness, as they could potentially work around the above limitations of adversarial training.
Adopting the theme of minimizing adversarial example transferability across sub-models, each ensemble defense proposed unique solutions.
ADP [19] increases the orthogonality of nonmaximal class logits among sub-models to encourage diversity.
GAL [14] minimizes a gradient alignment loss, which directly reduces the cosine-similarity between sub-models.
Building on top of this, TRS [31] further regularizes the smoothness of the loss function, as gradient orthogonality with smoothness may further diversify sub-models.
Dverge [30] instead uses the adversarial examples of a sub-model to train another sub-model, thus lowering transferability.
Ensemble defenses are also particularly interesting, as they are the last line of defense against even the strongest existing white-box attacks without resorting to adversarial training, showing a certain degree of robustness.
Besides the above mechanisms for training a successful ensemble defense, there exists different ways to combine sub-model predictions.
Let us assume that an ensemble defense trains M sub-models, f m : I ! R K for m 2 [1 : M ] , an ensemble f E : I ! R K thus forms a ﬁnal classiﬁcation result by combining individual decisions from the sub-models, namely:
[Equation]
where ens is the ensemble-forming operator.
In this paper, we investigate ens 2 { softmax , wta , id } , where the potential choices respectively denoting forming an ensemble from sub-model outputs f m ( x ) by either summing predicted probabilities (evaluated with softmax ), or majority votes (using wta , the winner-take-all operator), or simply summing the logits (with id , the identity operator).
Defending ensemble methods [19, 14, 30, 31] tested in this paper all employed the softmax -based strategy to report their robustness.
Methods that are exceptions to these options exist, for instance,
ECOC [29] allows sub-models to produce binary predictions, and use error correcting codes based on the Hamming distance to combine the predictions into classiﬁcation outputs.
This approach is unfortunately not robust, and the added complexity is error-prone and may harm robustness [25].
Moreover, as the voting ( wta ) strategy is non-differentiable, an attacker can soften it approximately using a softmax operation with temperature ⌧ , where we used ⌧ = 0 .
1 universally: softwta ⌧ ( x ) , softmax( x / ⌧ ) .
[Equation]
The Model-Reweighing Attack (MORA) 3.1 Problem Formulation & High-Level Overview
As discussed in Section 1, existing ensemble defenses may obfuscate gradients with the ensembleforming mode and gradient diversiﬁcation, such that the ﬁnal loss of the ensemble model can no longer provide effective signals for gradient descent.
It is therefore desirable to ﬁnd an alternative
L to the original SCE loss L sce on the ensemble, such that for a given number of iterations I , the original L sce loss can be maximized:
[Equation]
( L ( f E (
[Equation]
) , f [
[Equation]
[Equation] [Equation] ) , y )) , (6) and PGD( · ) denotes a PGD step along the gradient of loss function L ( f E (ˆ x i ) , f [1: M ] (ˆ x i ) , y ) , which not only takes the ensemble predictions f E (ˆ x i ) as input, but can further utilize sub-model predictions f [1: M ] to guide the PGD iterations.
The challenge at hand is, therefore, to ﬁnd a suitable L which can generate attacks on ensemble defenses efﬁciently and effectively. [Table] [Table] <
[Table]
=
[Table]
=
[Table]
qcx2VPNeJv7n9WIdXHgJFVGsicCzQUHMoA5hlgccUkmwZlNDEJbU7ArxGEmEtUmtYkJw509eJO16zT2rOTen1Ua9iKMMDsEROAEuOAcNcA2aoAUweATP4BW8WU
[Table] Wi/ [Table] /+
[Table]
=<
[Table]
z
[Table] < [Table] sha
[Table]
="FrmlLXT2xj6iBWdi9FidMT39phI="
[Table]
/BU3LhRx68qdf+P0sdDWAwNnzrmXe++JUikMet63MzM7N7+wuLRcWlldW99wN7dqJsk041WWyEQ3IjBciphXUaDkjVRzUJHk9ej+YuDXH7g2IolvsZ/yUEE3Fh3BAK3Ucg8CBdhjIPOr4i4ffrTKVaKhKFp5EHGEQxogZEXLLXsVbwg6TfwxKZMxrlvuV9BOWKZ4jEyCMU3fSzHMQaNgkhelIDM8BXYPXd60NAbFTZgPTyronlXatJNo+2KkQ/V3Rw7KmL6KbOVgZzPpDcT/vGaGnbMwF3GaIY/ZaFAnkxQTOsiHtoXmDGXfEmBa2F0p64EGhjbFkg3Bnzx5mtSOKv5Jxbs5Lp8fjeNYIjtkl+wTn5ySc3JJrkmVMPJInskreXOenBfn3fkYlc44455t8gfO5w
[Figure]
/latexit
[Figure]
MORA aims to provide a potential optimization route towards the above problem formulation.
Namely, in addition to the original output of the ensemble f E (ˆ x ) , we leverage the sub-model predictions f
[1: M ] (ˆ x ) to facilitate the optimization.
By way of illustration, Figure 2 shows a highlevel overview of the model-reweighing attack, where we compliment the ensemble loss, with a newly added sub-model reweighing loss L mora  , ⌧ , as an auxiliary attack vector alongside the original objective.
Not only can the new loss bypass the ensemble-forming strategy to work around its obfuscated gradients, but it further exploits information present in the individual sub-model and ensemble predictions to steer the direction of adversarial example synthesis.
3.2 Adaptive Sub-model Importance
Before we begin, assume that z [ m ] , f m ( x ) represents the m th sub-model output, and let z [ m ] t denote the corresponding logit of label t .
We deﬁne the difference of logits (DL) [3] k [ m ] , z [ m ] y  z
[ m ] ˆ y , which is the difference between the predictions of the ground truth z [ m ] y and the maximum of the remaining classes z
[ m ] ˆ y , max i 2 C /y
z [ m ] i , where C /y is the set of all class labels except y .
Similarly, we let z [E] , f E ( x ) , and z [E] t and k [E] be the respective variants of the ensemble prediction.
It is notable that a successful attack happens when k [E] < 0 , and similarly k [ m ] < 0 means the m th sub-model is producing incorrect classiﬁcation.
Ensemble defenses tend to diversify sub-model gradients, for instance, ADP [19] minimizes the cosine-similarity hr ` ( z [ a ] ) , r ` ( z [ b ] ) i among each loss function gradient pairs of sub-models ` ( z [ a ] ) and ` ( z [ b ] ) .
Their intuition is that it may lower transferability among these sub-models, such that attacks with the overall gradient of the ensemble, i.e. , r ` ( z [E] ) = 1 M
[Equation] ] r ` ( z [ m ] ) , are becoming less effective in misleading all sub-models simultaneously, as individual gradients in r ` ( z [ m ] ) are encouraged to be orthogonal to each other.
To this end, we propose to reweigh the importance of sub-models, by instead considering the modiﬁed gradient:
[Equation]
where  [ m ] ⌧ assigns weights to important sub-models to contribute more heavily to the attack gradient.
While the adversarial examples of the ensemble could present a challenge to discover, individual sub-models are weak defenders which can be easily defeated.
Based on this property, we propose to weigh sub-models importance with the rate of change in k [E] w.r.t.
that of k [ m ] , i.e. , sub-models would be given higher weights if attacking it would bring a signiﬁcant change to the ensemble’s prediction.
Following this idea, for all ensemble-forming strategies (softmax, voting, logits), we rewrite k
[E] as a function of k [ m ] , where the term below can become a function of k [ m ] :
[Equation]
1: function MORA_ Attack ( f [1: M ] , x , y,  , ⌧ , ⌫ , ✏ , I ) 2: ˆ x P ✏ , x ( x + u ) , where u ⇠ U (  ✏ , ✏ ) .
3: µ 4: for i 2 [0 : I  1] do 5: z
[ m ] f m (ˆ x i ) for all m 1 : M ] 6: z [E] 1 M
P m 2 [1: M ] ens( z [ m ] )
7: k [E] z [E] y  z [E] ˆ y 8: if k [E]  0 then return ˆ x i 9: g i + sign( r ˆ x i L mora  , ⌧ ( z [1: M ] , z [E] , y ))
10: ↵ ✏ (1 + cos( i ⇡ / I ))
11: µ i + P ✏ , x ( µ i + ↵ g i +1 )
12: ˆ x i + P ✏ , x (ˆ x i + ⌫
µ i +1
ˆ x i ) + (1  ⌫ ) (ˆ x i
ˆ x i  1 ))
13: end for 14: : end
The weights are thus deﬁned as follows:
[Equation]
While it is possible to compute the weights using gradient back-propagation, we can simply derive the following closed-form solution of the weights for each of the three ensemble-forming strategies.
For wta , we use the softened version of wta as deﬁned in (5) and can derive the weights as follows:
[Equation]
s [Equation] softmax
[Equation]
Here 1[ k [ m ] > 0] is the indicator function that equals 1 if k [ m ] > 0 , or 0 otherwise, effectively stopping the attack on the m th sub-model upon success, and the detach operator admits no backward propagation to its input.
In the case of using sums of sub-model softmax outputs to form an ensemble decision, i.e. , ens = softmax , it is a special case of softwta ⌧ where the temperature coefﬁcient can be ﬁxed at ⌧ = 1 .
Finally, when ens = id , i.e. , forming ensembles by summing logits,  [ m ] ⌧ ( z [ m ] ) simply reduces to 1[ k [ m ] > 0] for the m th sub-model.
3.3 The MORA Loss
For reference, defenses mechanisms we examine in this paper aim to ﬁnd ˆ x which maximizes the
SCE loss L sce ( z [E] , y ) , to evaluate the ensemble robustness.
The MORA loss improves this further by proposing two additional modiﬁcations to the untargeted loss function used to attack ensembles:
[Equation]
mora [Equation] [Equation] First, it additionally introduces a sum of the  [ m ] ⌧ -weighted variant of sub-model logits, in order to expose sub-model logits with adaptive reweighing described in Section 3.2.
Second,  interpolates the importance of the newly added auxiliary logits and the original ensemble logits.
Finally, inspired by the effective surrogate loss in [32], it further normalizes the logits by their respective DL using:
[Equation]
Finally, the targeted variant of the MORA loss simply replaces y with t where t is the intended target.
3.4 Improving the State-of-the-art
While the new L mora  , ⌧ loss is highly effective against ensemble defenses we test in this paper, we strive for further advances in MORA’s ability to generate faster and better adversarial examples.
Inspired by recent publications, we borrow ideas from related adversarial attack tactics, which includes adopting a cosine step-size schedule [16], momentum [7, 5], random restarts [25] and multiple target attacks [5, 25].
We provide the overall algorithm in Algorithm 1, which computes an adversarial image ˆ x I as return, by taking as input the sub-models f [1: M ] , natural image x , ground truth label y ,  to interpolate between the auxiliary logits and the original, ⌧ controls the temperature, momentum µ = 0 .
75 following [32, 5], ✏ perturbation bound, and ﬁnally the maximum number of iterations I .
[Table]
Experimental Results
We compare MORA against SOTA attacks for a wide range of ensemble defenses under three ensemble-forming strategies (softmax, voting, and logits).
We use pre-trained ResNet-20 [12] models from ADP [19], Dverge [30], GAL [14], and reproduced TRS [31] using the same architecture with ofﬁcial source code, as pre-trained models were unavailable.
Our robustness evaluation considers the ` 1 white-box attacks on the CIFAR-10 test set [15], with perturbation ✏ = 0 .
unless speciﬁed.
The full comparison results can be found in Table 1; larger ✏ comparisons, and similar results on
CIFAR-100 models are in Appendix A.
We provide our key observations below.
Traditional attacks may fail to break through gradient obfuscation.
We reproduce two traditional white-box attacks, i.e. , projected gradient descent (PGD) [17] and C&W [3] with 5 random restarts, each with a maximum of 100 iterations, giving a total of 500 iterations.
PGD uses a ﬁxed step size of ✏ / 4 .
For a fair comparison, MORA with 500 iterations sweeps  2 { 0 , 0 . 25 , 0 .
75 , 1 } , with each  up to 100 iterations.
Even with a 500 iteration budget, it is clear that PGD and C&W may substantially overestimate robustness, especially when tested under the softmax and voting ensembleforming options, and MORA can work around this obstacle thanks to its attacks on sub-models.
Diversiﬁed gradients can hamper even integrated attacks with large arsenals.
Moreover, we test the defenses against recent integrated attacks with SOTA baselines on robustness evaluation, namely Adaptive Auto Attack ( A 3 ) [16], AutoAttack (AA) [5], and Composite Adversarial Attacks (CAA) [18], which comprise large arsenals of various attack strategies.
We reproduce CAA following [18] to search for the attack policy before evaluating the defending models.
Note that its computational complexity is thus much higher than the other attacks, but we only report its test-time complexity.
In particular, while they enjoyed much higher success rates than PGD and C&W, some defenses render their tactics ineffective.
We observe, e.g. , sizeable robustness overestimation on
ADP [19] under softmax and voting, which explicitly diversiﬁes sub-model gradients.
As MORA can dynamically re-adjust sub-model importance w.r.t.
their “ease-of-attack”, it performs substantially better with much fewer iterations.
In addition to the earlier 500 iterations, the multi-targeted MORA mt targets the remaining 9 class labels with 100 iterations for each label and  ﬁxed at 0 .
Others also use multi-targeted attacks along with respective tactics.
Robustness of most sub-models vs. robustness of ensemble.
We ﬁnd that robustness of a majority of sub-models (fooling 3 / 8 for softmax and 2 / 8 for logits) usually do not translate to the overall robustness of the ensemble (Figures 1c and 3a).
As voting requires breaking 1 / 2 sub-models simultaneously (Figure 3b), it is perceivable that using voting may give rise to a higher overall robustness.
Yet surprisingly, for most defending ensembles, voting performs worse than softmax and logits.
Ensemble-forming strategies may give a false sense of security.
On one hand, softmax and voting strategies exhibit substantially larger overestimated robustness (up to 40%) than logits.
On the other hand, in stark contrast to the proposed use of softmax from [14, 19, 30, 31], we ﬁnd summing by logits can form ensembles that are notably more robust than the other two (Figure 3c), while attackers only needs to successfully deceive a few sub-models (referring back to Figure 1c).
Up to 60 ⇥ faster convergence under 500 iterations.
Figure 4 compares the convergence speed of
MORA against AA losses, C&W, and PGD on defending ensembles.
MORA converges substantially faster than the other attacks, using only up to 31 steps to match AA losses with 500 iterations.
Ensemble defense mechanisms may be at odds with robustness.
In Table 2, we compare respective attacks on adversarially trained Dverge models.
To our surprise, forming larger ensembles is actually harmful to the robustness of ensemble.
[Figure] [Figure]
[Table]
Additional results, ablation [Table] and sensitivity analyses.
Due to the page limit, we provide full results of relevant ﬁgures in Appendix A, note that the above key observations still hold true for all ensemble defenses we test under different ensemble-forming strategies and ✏ perturbation bounds.
In addition, we provide extensive ablation study on the design choices we made, and sensitivity analysis on the temperature constant ⌧ .
Conclusions
This paper identiﬁes severe robustness overestimation in many ensemble defense techniques, and further investigates problem the robustness evaluation under three ensemble-forming strategies.
To efﬁciently and accurately evaluate the robustness of ensembles, we introduce MORA, a new attack technique which reweighs sub-model importance adaptively by their respective “ease-of-attack” during attack iterations.
MORA enjoys a much improved success rate and convergence rate compared with other SOTA attacks.
Moreover, we found several surprising observations related to ensemble defenses, for instance, (1) misleading a minority of sub-models is sufﬁcient to fool the ensemble, (2) summing by logits is the simplest yet most robust way to form ensembles, (3) with adversarial training, ensemble defenses may actually harm robustness, etc.
We hope the above observations may help to guide future avenue on ensemble defenses, and provide a strong attack baseline for potential approaches.
Finally, MORA is open source with reproducible results and pre-trained models; and we continually update a leaderboard of ensemble defenses under various attack strategies.
Acknowledgements
This work is supported in part by National Key R&D Program of China (No. 2019YFB2102100), Key-Area Research and Development Program of Guangdong Province (No. 2020B010164003),
Science and Technology Development Fund of Macao S.A.R (FDCT) under No. 0015/2019/AKP, Shenzhen Science and Technology Innovation Commission (No. JCYJ20190812160003719), and
[-and etc [Figure] C Computational Resources On NVIDIA Tesla V100 GPUs, MORA with 500 iterations uses up to 1 . 
-]Shenzhen Industrial Application Projects of undertaking the National Key R&D Program of China (No.CJGJZD20210408091600002).
This work was carried out in part at SICC which is supported by
SKL-IOTSC, University of Macau.