Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints
Abstract
Evaluating adversarial robustness amounts to ﬁnding the minimum perturbation needed to have an input sample misclassiﬁed.
The inherent complexity of the underlying optimization requires current gradient-based attacks to be carefully tuned, initialized, and possibly executed for many computationally-demanding iterations, even if specialized to a given perturbation model.
In this work, we overcome these limitations by proposing a fast minimum-norm (FMN) attack that works with different (cid:96) p -norm perturbation models ( p = 0 , 1 , 2 , ∞ ), is robust to hyperparameter choices, does not require adversarial starting points, and converges within few lightweight steps.
It works by iteratively ﬁnding the sample misclassiﬁed with maximum conﬁdence within an (cid:96) p -norm constraint of size (cid:15) , while adapting (cid:15) to minimize the distance of the current sample to the decision boundary.
Extensive experiments show that FMN signiﬁcantly outperforms existing attacks in terms of convergence speed and computation time, while reporting comparable or even smaller perturbation sizes.
1. Introduction
Learning algorithms are vulnerable to adversarial examples, i e., intentionally-perturbed inputs aimed to mislead classiﬁcation at test time (Szegedy et al., 2014; Biggio et al., 2013).
While adversarial examples have received much attention, evaluating the robustness of deep networks against them remains a challenge.
Having an arsenal of diverse attacks that can be adapted to speciﬁc defenses is one of the most promising avenues for increasing conﬁdence in white-box robustness evaluations (Carlini et al., 2019; Tramer et al., 2020).
While it may seem that the number of attacks is already large, most of them are just small variations of the same technique, make similar underlying assumptions and thus tend to fail jointly.
In this work, we focus on minimum-norm attacks for evaluating adversarial robustness, i.e., attacks that aim to mislead classiﬁcation by ﬁnding the smallest input perturbation according to a given norm.
Within the class of gradientbased minimum-norm attacks, there are three main subcategories: (i) soft-constraint attacks, (ii) boundary attacks and (iii) projected-gradient attacks.
Soft-constraint attacks like CW (Carlini & Wagner, 2017) optimize a trade-off between conﬁdence of the misclassiﬁed samples and perturbation size.
CW needs a sample-wise tuning of the trade-off hyperparameter to ﬁnd the smallest possible perturbation, thus requiring many steps to converge.
Boundary attacks like BB (Brendel et al., 2019) or FAB (Croce & Hein, 2020b) move along the decision boundary towards the closest point to the input sample.
These attacks converge within relatively few steps, but may require an adversarial starting point and need to solve expensive optimization problems in each step.
Finally, recent minimum-norm projected-gradient attacks like DDN (Rony et al., 2019) perform a maximumconﬁdence attack in each step under a given perturbation budget (cid:15) , while iteratively adjusting (cid:15) to reduce the perturbation size, however, DDN is speciﬁc to the (cid:96) 2 norm and cannot be readily extended to other norms.
To overcome the aforementioned limitations, in this work we propose a novel, fast minimum-norm (FMN) attack (Sect. 2), which retains the main advantages of DDN while generalizing it to different (cid:96) p norms ( p = 0 , 1 , 2 , ∞ ).
We perform large-scale experiments (Sect. 3), showing that FMN is able to combine all desirable traits a good adversarial attack should have, providing an important step towards improving adversarial robustness evaluations.
We conclude by discussing limitations and future research directions (Sect. 4).
2. Minimum-Norm Adversarial Examples with Adaptive Projections
Problem formulation.
Given an input sample x ∈ [0 , 1] d , belonging to class y ∈ { 1 , . . .
, c } , the goal of an untargeted attack is to ﬁnd the minimum-norm perturbation δ (cid:63) such that the corresponding adversarial example x (cid:63) =
x + δ (cid:63) is
[Equation]
<
[Figure]
misclassiﬁed.
This problem can be formulated as:
[Equation]
where || · || p indicates the (cid:96) p -norm operator.
The loss L in the constraint in Eq.
(2) is deﬁned as:
[Equation]
where f j ( x , θ ) is the conﬁdence given by the model f for classifying x as class j , and θ is the set of its learned parameters.
Assuming that the classiﬁer assigns x to the class exhibiting the highest conﬁdence, i.e., y (cid:63) = arg max j ∈ 1 ,...,c f j ( x , θ ) , the loss function L ( x , y, θ ) takes on negative values only when x is misclassiﬁed 1 .
Finally, the box constraint in Eq.
(3) ensures that the perturbed sample x + δ lies in the feasible input space.
Solution algorithm.
To solve Problem (1)-(3), we reformulate it using an upper bound (cid:15) on (cid:107) δ (cid:107) p : min (cid:15),
[Equation]
and to the constraints in Eqs.
This allows us to derive an algorithm that works in two main steps, similarly to DDN (Rony et al., 2019), by updating the maximum perturbation size (cid:15) separately from the actual perturbation δ , as represented in Fig.
In particular, the constraint size (cid:15) is adapted to reduce the distance of the constraint to the boundary ( (cid:15) -step), while the perturbation δ is updated using a projected-gradient step to minimize the loss function L within the given (cid:15) -sized constraint ( δ -step).
The complete algorithm is given as Algorithm 1, while a more detailed explanation of the two aforementioned steps is given below.
Input: x , the input sample; t , a variable denoting whether the attack is targeted ( or untargeted ; y , the target ( ) class label if the attack is targeted and γ K , the and ﬁnal sizes; α 0 and α K , the initial and ﬁnal δ -step sizes; K , the number of iterations.
[-1: x 0 ← x , (cid:15) 
-]
, K do 3: g ← t · ∇ δ L ( x k − 1 + δ , y, θ ) //
loss gradient 4: γ k ← h ( γ 0 , γ K , k, K ) //
(cid:15) -step size decay 5: if L ( x k − 1 , y, θ ) ≥ 0 then 6: if adversarial not found yet then 7: (cid:15) k
= (cid:107) δ k − 1 (cid:107) p
+ L ( x k − 1 , y, θ ) /
(cid:107) g (cid:107) q 8: else 9: (cid:15) k
= (cid:15) k − 1 (1 + γ k ) 10: end if 11: else 12: if (cid:107) δ k − 1 (cid:107) p ≤ (cid:107) δ (cid:63) (cid:107) p then 13: δ (cid:63) ← δ k − 1 // update best min-norm solution 14: end if 15: (cid:15)
k
= min( (cid:15) k − 1 (1 − γ k ) , (cid:107) δ (cid:63) (cid:107) p ) 16: end if 17: α k ← h ( α 0 , α K , k, K ) //
δ -step size decay 18: δ k ← δ k − 1 + α k · g / (cid:107) g (cid:107) 2 19: δ k ← Π (cid:15) ( x 0 + δ k ) − x20:
δ k ← clip( x 0 + δ k ) − x21:
x k ← x 0 + δ k 22: end for 23: return x (cid:63) ← x 0 + δ (cid:63) (cid:15) -step.
This step updates the upper bound (cid:15) on the perturbation norm (lines 4-16 in Algorithm 1).
The underlying idea is to increase (cid:15) if the current sample is not adversarial (i.e., L ( x k − 1 , y, θ ) ≥ 0 ), and to decrease it otherwise, while reducing the step size to dampen oscillations around the boundary and reach convergence.
In the former case ( (cid:15) -increase ), the increment of (cid:15) depends on whether an adversarial example has been previously found or not.
If not, we estimate the distance to the boundary with a ﬁrst-order linear approximation, and set (cid:15)
k
= (cid:107) δ k − 1 (cid:107) p + L ( x k − 1 , y, θ ) /
||∇ L ( x k − 1 , y, θ ) || q , being q the dual norm of p .
This approximation allows the attack point to make faster progress towards the decision boundary.
Conversely, if an adversarial sample has been previously found, but the current sample is not adversarial, it is likely that the current estimate of (cid:15) is only slightly smaller than the minimum-norm solution.
We thus increase (cid:15) by a small fraction as (cid:15) k
= (cid:15) k − 1 (1 + γ k ) , being γ k a decaying step size.
In the latter case ( (cid:15) -decrease ), if the current sample is adversarial, i.e., L ( x k − 1 , y, θ ) < 0 , we decrease (cid:15) as
(cid:15) k
= (cid:15) k − 1 (1 − γ k ) , to check whether the current solution can be improved.
If the corresponding (cid:15) k value is larger than the optimal (cid:107) δ (cid:63) (cid:107) p found so far, we retain the best value and set (cid:15) k
= (cid:107) δ (cid:63) (cid:107) p .
To ensure convergence, the step size γ k is decayed with cosine annealing.
δ -step.
This step updates δ (lines 17-21 in Algorithm 1).
The goal is to ﬁnd the adversarial example that is misclassiﬁed with maximum conﬁdence (i.e., for which L is minimized) within the current (cid:15) -sized constraint
(Eq. 5) and bounds (Eq. 3).
This amounts to performing a projectedgradient step along the negative gradient of L .
We consider a normalized steepest descent with decaying step size α to overcome potential issues related to noisy gradients while ensuring convergence (line 18).
The step size α is decayed using cosine annealing.
Once δ is updated, we project it onto the given (cid:15) -sized (cid:96) p -norm constraint via a projection operator Π (cid:15) (line 19), to fulﬁll the constraint in Eq.
The projection is trivial for p = ∞ and p = 2 .
For p = 1 , we use the efﬁcient algorithm by Duchi et al.
For p = 0 , we retain only the ﬁrst (cid:15) components of δ exhibiting the largest absolute value.
We ﬁnally clip the components of δ that violate the bounds in Eq.
(3) (line 20).
Adversarial initialization.
Our attack can be initialized from the input sample x , or from a point x init belonging either to a different class (untargeted attacks) or to the target class (targeted attacks).
When initializing the attack from x init , we perform a 10 -step binary search between x and x init , to ﬁnd an adversarial point closer to the boundary.
Differences with DDN.
FMN applies substantial changes to both the algorithm and the formulation of DDN.
The main difference is that (i) DDN always rescales the perturbation to have size (cid:15) .
This operation is problematic when using other norms, especially sparse ones, as it hinders the ability of the attack to explore the neighboring space and ﬁnd a suitable descent direction; (ii) FMN uses the logit difference as the loss function L ; (iii) FMN does not need an initial value for (cid:15) , as (cid:15) is dynamically estimated; and (iv) γ is decayed to improve convergence around better minimumnorm solutions, by more effectively dampening oscillations around the boundary.
Finally, we include the possibility of (v) initializing the attack from an adversarial point, which can greatly increase the convergence speed.
3. Experiments
We report here an extensive experimental analysis involving several state-of-the-art defenses and minimum-norm attacks, covering (cid:96) 0 , (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ norms.
The goal is to empirically benchmark our attack and assess its effectiveness and efﬁciency as a tool for adversarial robustness evaluation.
3.1. Experimental Setup
Datasets and models.
We consider two commonly-used datasets for benchmarking adversarial robustness of deep neural networks, i.e., the MNIST handwritten digits and
CIFAR10.
Following the experimental setup in Brendel et al.
(2019), we use a subset of 1000 test samples to evaluate the considered attacks and defenses.
We use a diverse selection of models to thoroughly evaluate attacks under different conditions.
For MNIST, we consider the following four models: M1 , a 9-layer undefended ConvNet; M2 , the robust model by Madry et al.
(2017), trained on (cid:96) ∞ attacks; M3 , the robust model by Rony et al.
(2019), trained on (cid:96) 2 attacks; and M4 , the IBP Large Model by Zhang et al.
For
CIFAR10, we consider three state-of-the-art robust models from RobustBench (Croce et al., 2020): C1 , the robust model by Madry et al.
(2017), trained on (cid:96) ∞ attacks; C2 , the defended model by Carmon et al.
(2019), trained on (cid:96) ∞ attacks and additional unsupervised data; and C3 , the robust model by Rony et al.
(2019), trained on (cid:96) 2 attacks.
Attacks.
We compare our algorithm against different stateof-the-art attacks for ﬁnding minimum-norm adversarial perturbations across different norms: the Carlini & Wagner (CW) attack (Carlini & Wagner, 2017), the Decoupling
Direction and Norm (DDN) attack (Rony et al., 2019), the
Brendel & Bethge (BB) attack (Brendel et al., 2019), and the Fast Adaptive Boundary (FAB) attack (Croce & Hein, 2020b).
All these attacks are deﬁned on the (cid:96) 2 norm.
BB and FAB are also deﬁned on the (cid:96) 1 and (cid:96) ∞ norms, and only
BB is deﬁned on the (cid:96) 0 norm.
We consider untargeted and targeted attack scenarios, except for FAB, which is only evaluated in the untargeted case.
To ensure a fair comparison, we perform an extensive hyperparameter search for each of the considered attacks.
We consider dataset-level hyperparameter tuning, a scenario in which we choose the same hyperparameters for all samples, selecting the conﬁguration that yields the best attack performance on a smaller validation set.
The hyperparameter conﬁgurations considered for each attack are detailed in the appendix.
Evaluation criteria.
We evaluate the attacks along two different criteria (additional experiments can be found in the
Appendix): (i) perturbation size measured as the median (cid:107) δ
[Equation]
(cid:107)
p on the test set; (ii) execution time , measured as the average time spent per query (in milliseconds). [Table] [Table]
[Table]
74.03 Ours 5. (cid:96) FAB 10. - - - CW 4.
31.30 BB 4. 54.07 DDN 3. 29.52 Ours 4. (cid:96)
[Table]
FAB 10.
- - - BB 14.
Ours 4.
[Table]
3.2. Experimental Results
Perturbation size.
Table 1 reports the median value of (cid:107) δ (cid:63) (cid:107) at Q = 1000 queries, for all models, attacks and norms.
The values obtained conﬁrm that our attack can ﬁnd smaller or comparable perturbations with those found by the competing attacks, in most of the untargeted and targeted cases, and that the biggest margin is achieved in the (cid:96) 1 case.
FMN is only slightly worse than DDN and BB in a few cases, including (cid:96) 2 -DDN on M4 and (cid:96) ∞ -BB on M2.
The reason may be that these robust models exhibit noisy gradients and ﬂat regions around the clean input samples, hindering the initial optimization steps of the FMN attack.
Execution time.
The average runtime per query for each attack-model pair, measured on a workstation with an
NVIDIA GeForce RTX 2080 Ti GPU with 11GB of RAM, can be found in Table 2.
Our attack is up to 2 - 3 times faster, with the exception of DDN in the (cid:96) 2 case.
This is compensated by the fact that FMN ﬁnds better solutions.
4. Conclusions
This work introduces a novel minimum-norm attack that combines all desirable traits to help improve current adversarial evaluations: (i) ﬁnding smaller or comparable minimum-norm perturbations across a range of models and datasets; (ii) reducing runtime up to 3 times per query with respect to competing attacks.
FMN also works with different (cid:96) p norms ( p = 0 , 1 , 2 , ∞ ) and it does not necessarily require initialization from the target class.
We ﬁrmly believe that FMN will establish itself as a useful tool in the arsenal of robustness evaluation.
By facilitating more reliable robustness evaluations, we expect that FMN will foster advancements in the development of machinelearning models with improved robustness guarantees. Acknowledgements
This work has been partly supported by the PRINproject RexLearn (grant no. 2017TWNMH2), funded by the Italian Ministry of Education, University and Research; and by BMK, BMDW, and the Province of Upper Austria in the frame of the COMET Programme managed by FFG in the COMET Module S3AI.
WB acknowledges support from the German Federal Ministry of Education and Research (BMBF) through the Competence Center for Machine Learning (TUE.AI, FKZ 01IS18039A), from the German Science
{+Foundation (DFG) under grant no. BR 6382/1-
 (Emmy Noether Program) as well as support by Open Philantropy and the Good Ventures Foundation.+} Appendix A. Hyperparameters
We select the hyperparameters to be optimized for each attack as recommended by the corresponding authors (Brendel et al., 2019; Carlini & Wagner, 2017; Croce & Hein, 2020b; Rony et al., 2019).
For attacks that are claimed to be robust to hyperparameter changes, like BB and FAB, we follow the recommendation of using a larger number of random restarts rather than increasing the number of hyperparameter conﬁgurations to be tested.
In addition, as BB requires being initialized from an adversarial starting point, we initialize it by randomly selecting a sample either from a different class (in the untargeted case) or from the target class (in the targeted case).
Finally, as each attack performs operations with different levels of complexity within each iteration, possibly querying the model multiple times, we set the number of steps for each attack such that at least 1 ,forward passes (i.e., queries ) are performed.
This ensures a fairer comparison also in terms of the computational time and resources required to execute each attack.
CW.
This attack minimizes the soft-constraint version of our problem, i.e., min δ (cid:107) δ (cid:107) p + c · min
( L ( x + δ , y, θ ) , − κ ) .
The hyperparameters κ and c are used to tune the tradeoff between perturbation size and misclassiﬁcation conﬁdence.
To ﬁnd minimum-norm perturbations, CW requires setting κ = 0 , while the constant c is tuned via binary search (re-running the attack at each iteration).
We set the number of binary-search steps to 9, and the maximum number of iterations to 250, to ensure that at least 1 ,queries are performed.
We also set different values for c, η ∈ { 10 − 3 , 10 − 2 , 10 − 1 , 1 } .
DDN.
This attack, similarly to ours, maximizes the misclassiﬁcation conﬁdence within an (cid:15) -sized constraint, while adjusting (cid:15) to minimize the perturbation size.
We consider initial values of (cid:15) 0 ∈ { 0 . 03 , 0 . 1 , 0 .
3 , 1 , 3 } , and run the attack with a different number of iterations K ∈ { 200 , 1000 } , as this affects the size of each update on δ .
BB.
This attack starts from a randomly-drawn adversarial point, performs a 10-step binary search to ﬁnd a point which is closer to the decision boundary, and then updates the point to minimize its perturbation size by following the decision boundary.
In each iteration, BB computes the optimal update within a given trust region of radius ρ .
We consider different values for ρ ∈ { 10 − 3 , 10 − 2 , 10 − 1 , 1 } , while we ﬁx the number of steps to 1000 .
We run the attack 3 times by considering different initialization points, and eventually retain the best solution.
FAB.
This attack iteratively optimizes the attack point by linearly approximating its distance to the decision boundary.
It uses an adaptive step size bounded by α max and an extrapolation step η to facilitate ﬁnding adversarial points.
As suggested by Croce & Hein (2020b), we tune α max ∈ { 0 . 1 , 0 . 05 } and η ∈ { 1 . 05 , 1 , 3 } .
We considerdifferent random initialization points, and run the attack for 500 steps each time, eventually selecting the best solution.
FMN.
We run FMN for K = 1000 steps, using γ 0 ∈ { 0 . 05 , 0 . 3 } , γ K = 10 − 4 , and α K = 10 − 5 .
For (cid:96) 0 , (cid:96) 1 , and (cid:96) 2 , we set α 0 ∈ { 1 , 5 , 10 } .
For (cid:96) ∞ , we set α 0 ∈ { 10 1 , 10 2 , 10 3 } , as the normalized (cid:96) 2 step yields much smaller updates in the (cid:96) ∞ norm.
For each hyperparameter setting we run the attack twice, starting from (i) the input sample and (ii) an adversarial point.
B. Additional experiments
Evaluation criteria.
We evaluate the attacks along four different criteria:
(i) perturbation size and (ii) robustness to hyperparameter selection , measured as the median (cid:107) δ (cid:63) (cid:107) p on the test set (for a ﬁxed budget of Q queries and for sampleand dataset-level hyperparameter tuning); (iii) execution time , measured as the average time spent per query (in milliseconds); and (iv) convergence speed , measured as the average number of queries required to converge to a goodenough solution (within 10% of the best value found at
Q = 1000 ).
Sample-level vs. Dataset-level tuning.
In order to inspect the capabilities of our attack, we consider two main scenarios: tuning the hyperparameters at the sample-level and at the dataset-level .
In the sample-level scenario, we select the optimal hyperparameters separately for each input sample.
In the dataset-level scenario, we choose the same hyperparameters for all samples, selecting the conﬁguration that yields the best attack performance.
While sample-level tuning provides a fairer comparison across attacks, it is more computationally demanding and less practical than dataset-level tuning.
In addition, the comparison allows us to understand how robust attacks are to suboptimal hyperparameter choices.
Query-distortion (QD) curves.
To evaluate each attack in terms of perturbation size under the same query budget Q , we use the so-called QD curves introduced by Brendel et al.
These curves report, for each attack, the median value of δ (cid:63) as a function of the number of queries Q .
For each given Q value, the optimal δ (cid:63) for each point is selected among the different attack executions (i.e., using different hyperparameters and/or initialization points, as described in Sect. 3.1).
In Fig.
2, we report the QD curves for the
MNIST and CIFAR10 challenge models (i.e., M2 and C1) in the untargeted scenario.
It is worth noting that our attack attains comparable results in terms of perturbation size across all norms, while signiﬁcantly outperforming FAB and
BB in the (cid:96) 1 case.
It typically requires also less iterations
[Figure]
than the other attacks to converge.
While the QD curves show the complete behavior of each attack as Q increases, a more compact and thorough summary of our evaluation is reported in Table 3, according to the four evaluation criteria described.
The remaining curves, computed for all models, can be found in the end of the Appendix
[Figure]
Robustness to hyperparameter selection.
The values reported in the lower part of Table 3 show that, when using dataset-level hyperparameter tuning, FMN outperforms the other attacks in a much larger number of cases.
This shows that FMN is more robust to hyperparameter changes, while other attacks like (cid:96) 0 - and (cid:96) 1 -BB suffer when using the same hyperparameters for all samples.
Convergence speed.
To get an estimate of the convergence speed, we measure the number of queries required by each attack to reach a perturbation size that is within 10% of the value found at Q = 1000 queries (the lower the better).
Results are shown in Table 4.
Our attack converges on par with or faster than all other attacks for almost all models, often requiring only half or a ﬁfth as many queries as the state of the art.
Experiments on ImageNet.
We expand further our experiments by running an additional comparison between FMN and a widely-used maximum-conﬁdence attack, i.e., the Projected Gradient Descent (PGD) attack (Madry et al., 2017), on two pretrained ImageNet models (i.e., ResNet18 and
VGG16)
, considering (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ norms.
The hyperparameters are tuned at the dataset-level using 20 validation samples.
For FMN, we ﬁx the hyperparameters as discussed before, and only tune α 0 ∈ { 0 . 1 , 1 , 2 , 8 } , without using adversarial initialization.
For PGD, we tune the step size α ∈ { 0 . 001 , 0 . 01 , 0 .
We run both attacks for
Q = 1 , 000 queries on a separate set of 1 , 000 samples.
The success rates of both attacks at ﬁxed (cid:15) values are reported in
Table 5.
The results show that FMN outperforms or equals
PGD in all norms.
[Table]
C. Adversarial Examples
In Figs.
3-4, we report adversarial examples generated by all attacks against model M2 and C2, respectively, on MNIST and CIFAR10 datasets, in the untargeted scenario.
The clean samples and the original label are displayed in the ﬁrst row of each ﬁgure.
In the remaining rows we show the perturbed sample along with the predicted class and the corresponding norm of perturbation (cid:107) δ (cid:63) (cid:107) p .
It is worth noting that the output class for different untargeted attacks is not always the same, which might sometimes explain
[Table]
1.38 DDN 1.
1.39 Ours 1.
(cid:96) ∞ FAB .
- - - BB .
.037 Ours .
Dataset-level Hyperparameter Tuning (cid:96)
BB             33 Ours              (cid:96)
FAB 8.
- - - BB 10.
23.37 Ours 7.
(cid:96)
[Table] [Table] CIFAR
[Table] M
M M M
M M M M
C
C C C C
C
[Table]
differences in the perturbation sizes.
An example is given in
Fig. 4b, where the sample in the fourth column, labeled as “ship”, is perturbed by most of the attacks towards the class “airplane”, while in our case it outputs the class “dog” with a much smaller distance.
[Figure]
n
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
C l e a n label:label:label:
[Table]
[Table]
label:
[Table]
BB
[Table]
class
[Figure]
[Figure] [Figure]
[Figure] [Table]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
u r s
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
D. Additional query-distortion Curves
We already introduced in sect.
B the query-distortion curves as an efﬁciency evaluation metric for the attacks.
We report here the complete results for all models, in targeted and untargeted scenarios.
On the MNIST dataset, our attacks generally reach smaller norms with fewer queries, with the exception of M2 (Figs. 56), where it seems to reach convergence more slowly than
BB in (cid:96) 0 and (cid:96) ∞ .
In (cid:96) 2 , the CW attack is the slowest to converge, due to the need of carefully tuning the weighting term, as described in Sect.
On the CIFAR10 dataset (Figs.
7-8), our attack always rivals or outperforms the others, with the notable exception of DDN for the (cid:96) 2 norm, which sometimes ﬁnds smaller perturbations more quickly, as also shown in Table 4.
[Figure]
D i s t a n ce
[Figure]
D i s t a n ce
[Figure]
D i s t a
[Figure]
D i s t a n ce
[Figure]
D i s t a n ce
[Figure]
D i s t a n ce
[Figure]
[Figure]