Original: H O W
Revised: H O W

Original: T O
Revised: T O

Original: C O M PA R E
Revised: C O M PA R E

Original: A D V E R S
Revised: A D V E R S

Original: A R
Revised: A R

Original: I
Revised: I

Original: A L
Revised: A L

Original: R O B U S TN E S S
Revised: R O B U S TN E S S

Original: O F
Revised: O F

Original: C L A S S
Revised: C L A S S

Original: I F I E R S
Revised: I F I E R S

Original: F R O M
Revised: F R O M

Original: A
Revised: A

Original: G L O B A L
Revised: G L O B A L

Original: P E RS P E C T
Revised: P E RS P E C T

Original: I V E
Revised: I V E

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A B S T R AC T
Revised: A B S T R AC T

Original: Adversarial robustness of machine learning models has attracted considerable attention over recent years.
Revised: Adversarial robustness of machine learning models has attracted considerable attention over recent years.

Original: Adversarial attacks undermine the reliability of and trust in machine learning models, but the construction of more robust models hinges on a rigorous understanding of adversarial robustness as a property of a given model.
Revised: Adversarial attacks undermine the reliability of and trust in machine learning models, but the construction of more robust models hinges on a rigorous understanding of adversarial robustness as a property of a given model.

Original: Point-wise measures for speciﬁc threat models are currently the most popular tool for comparing the robustness of classiﬁers and are used in most recent publications on adversarial robustness.
Revised: Point-wise measures for speciﬁc threat models are currently the most popular tool for comparing the robustness of classiﬁers and are used in most recent publications on adversarial robustness.

Original: In this work, we userecently proposed robustness curves to show that point-wise measures fail to capture important global properties that are essential to reliably compare the robustness of different classiﬁers.
Revised: In this work, we use robustness curves to show that point-wise measures fail to capture important global properties that are essential to reliably compare the robustness of different classiﬁers.

Original: We introduce new ways in which robustness curves can be used to systematically uncover these properties and provide concrete recommendations for researchers and practitioners when assessing and comparing the robustness of trained models.
Revised: We introduce new ways in which robustness curves can be used to systematically uncover these properties and provide concrete recommendations for researchers and practitioners when assessing and comparing the robustness of trained models.

Original: Furthermore, we characterize scale as a way to distinguish small and large perturbations, and relate it to inherent properties of data sets, demonstrating that robustness thresholds must be chosen accordingly.
Revised: Furthermore, we characterize scale as a way to distinguish small and large perturbations, and relate it to inherent properties of data sets, demonstrating that robustness thresholds must be chosen accordingly.

Revised: We hope that our work contributes to a shift of focus away from point-wise measures of robustness and towards a discussion of the question what kind of robustness could and should reasonably be expected.

Original: We release code to reproduce all experiments presented in this paper, which includes a Python module to calculate robustness curves for arbitrary data sets and classiﬁers, supporting a number of frameworks, including TensorFlow, PyTorch and JAX.
Revised: We release code to reproduce all experiments presented in this paper, which includes a Python module to calculate robustness curves for arbitrary data sets and classiﬁers, supporting a number of frameworks, including TensorFlow, PyTorch and JAX.

Original:  I N T RO D U C T I O N
Revised:  I N T RO D U C T I O N

Original: Despite their astonishing success in a wide range of classiﬁcation tasks, deep neural networks can be lead to incorrectly classify inputs altered with specially crafted adversarial perturbations (Szegedy et al. 2014; Goodfellow et al. 2015).
Revised: Despite their astonishing success in a wide range of classiﬁcation tasks, deep neural networks can be lead to incorrectly classify inputs altered with specially crafted adversarial perturbations (Szegedy et al. 2014; Goodfellow et al. 2015).

Original: These perturbations can be so small that they remain almost imperceptible to human observers (J. P. Göpfert et al. 2020).
Revised: These perturbations can be so small that they remain almost imperceptible to human observers (J. P. Göpfert et al. 2020).

Original: Adversarial robustness describes a model’s ability to behave correctly under such small perturbations crafted with the intent to mislead the model.
Revised: Adversarial robustness describes a model’s ability to behave correctly under such small perturbations crafted with the intent to mislead the model.

Original: The study of adversarial robustness – with its deﬁnitions, their implications, attacks, and defenses – has attracted considerable research interest.
Revised: The study of adversarial robustness – with its deﬁnitions, their implications, attacks, and defenses – has attracted considerable research interest.

Original: This is due to both the practical importance of trustworthy models as well as the intellectual interest in the differences between decisions of machine learning models and our human perception.
Revised: This is due to both the practical importance of trustworthy models as well as the intellectual interest in the differences between decisions of machine learning models and our human perception.

Original: A crucial starting point for any such analysis is the deﬁnition of what exactly a small input perturbation is – requiring (a) the choice of a distance function to measure perturbation size, and (b) the choice of a particular scale to distinguish small and large perturbations.
Revised: A crucial starting point for any such analysis is the deﬁnition of what exactly a small input perturbation is – requiring (a) the choice of a distance function to measure perturbation size, and (b) the choice of a particular scale to distinguish small and large perturbations.

Original: Together, these two choices determine a threat model that deﬁnes exactly under which perturbations a model is required to be robust.
Revised: Together, these two choices determine a threat model that deﬁnes exactly under which perturbations a model is required to be robust.

Original: The most popular choice of distance function is the class of distances induced by (cid:96) p norms (Szegedy et al. 2014; Goodfellow et al. 2015; Carlini, Athalye, et al. 2019), in particular (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ , although other choices such as Wasserstein distance have been explored as well (Wong, Schmidt, et al. 2019).
Revised: The most popular choice of distance function is the class of distances induced by (cid:96) p norms (Szegedy et al. 2014; Goodfellow et al. 2015; Carlini, Athalye, et al. 2019), in particular (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ , although other choices such as Wasserstein distance have been explored as well (Wong, Schmidt, et al. 2019).

Original: Regarding scale, the current default is to pick some perturbation threshold ε without providing concrete reasons for the exact choice.
Revised: Regarding scale, the current default is to pick some perturbation threshold ε without providing concrete reasons for the exact choice.

Original: Analysis then focuses on the robust error of the model, the proportion of test inputs for which the model behaves incorrectly under some perturbation up to size ε .
Revised: Analysis then focuses on the robust error of the model, the proportion of test inputs for which the model behaves incorrectly under some perturbation up to size ε .

Original: This means that the scale is deﬁned as a binary distinction between small and large perturbations based on the perturbation threshold.
Revised: This means that the scale is deﬁned as a binary distinction between small and large perturbations based on the perturbation threshold.

Original: A set of canonical thresholds have emerged in the literature.
Revised: A set of canonical thresholds have emerged in the literature.

Original: For example, in the publications referenced in this section, the MNIST data set is typically evaluated at a perturbation threshold ε ∈ { 0 . 1 , 0 . 3 } for the (cid:96) ∞ norm, while CIFAR-  is evaluated at ε ∈ { 2 / 255 , 4 / 255 , 8 / 255 } , stemming from the three 8-bit color channels used to represent images.
Revised: For example, in the publications referenced in this section, the MNIST data set is typically evaluated at a perturbation threshold ε ∈ { 0 . 1 , 0 . 3 } for the (cid:96) ∞ norm, while CIFARis evaluated at ε ∈ { 2 / 255 , 4 / 255 , 8 / 255 } , stemming from the three 8-bit color channels used to represent images.

Original: Based on these established threat models, researchers have developed specialized methods to minimize the robust error during training, which results in more robust models.
Revised: Based on these established threat models, researchers have developed specialized methods to minimize the robust error during training, which results in more robust models.

Original: Popular approaches include speciﬁc data augmentation, sometimes used under the umbrella term adversarial training (Guo et al.
Revised: Popular approaches include speciﬁc data augmentation, sometimes used under the umbrella term adversarial training (Guo et al.

Original: 2017; Madry et al.
Revised: 2017; Madry et al.

Original: Carmon et al.
Revised: Carmon et al.

Original: Hendrycks et al. 2019), training under regularization that encourages large margins and smooth decision boundaries in the learned model (Hein and
Revised: Hendrycks et al. 2019), training under regularization that encourages large margins and smooth decision boundaries in the learned model (Hein and

Original: Andriushchenko 2017; Wong and Kolter 2018; Croce, Andriushchenko, et al. 2019; Croce and Hein 2020), and post-hoc processing or randomized smoothing of predictions in a learned model (Lecuyer et al. 2019;
Revised: Andriushchenko 2017; Wong and Kolter 2018; Croce, Andriushchenko, and Hein 2019; Croce and Hein 2020), and post-hoc processing or randomized smoothing of predictions in a learned model (Lecuyer et al. 2019;

Original: Cohen et al. 2019).
Revised: Cohen et al. 2019).

Original: In order to show the superiority of a new method, robust accuracies of differently trained models are typically compared for a handful of threat models and data sets, eg., (cid:96) ∞ ( ε = 0 . 1) and (cid:96) 2 ( ε = 0 . 3) for MNIST .
Revised: In order to show the superiority of a new method, robust accuracies of differently trained models are typically compared for a handful of threat models and data sets, eg., (cid:96) ∞ ( ε = 0 . 1) and (cid:96) 2 ( ε = 0 . 3) for MNIST .

Original: Out of 22 publications on adversarial robustness published at NeurIPS 2019, ICLR 2020, and ICML 2020, 12 publications contain results for only a single perturbation threshold.
Revised: Out of 22 publications on adversarial robustness published at NeurIPS 2019, ICLR 2020, and ICML 2020, 12 publications contain results for only a single perturbation threshold.

Original: In ﬁve publications, robust errors are calculated for at least two different perturbation thresholds, but still, only an arbitrary number of thresholds is considered.
Revised: In ﬁve publications, robust errors are calculated for at least two different perturbation thresholds, but still, only an arbitrary number of thresholds is considered.

Original: Only in ﬁve out of the total 22 publications do we ﬁnd extensive considerations of different perturbation thresholds and the respective robust errors.
Revised: Only in ﬁve out of the total 22 publications do we ﬁnd extensive considerations of different perturbation thresholds and the respective robust errors.

Original: Out of these ﬁve, three are analyses of randomized smoothing, which naturally gives rise to certiﬁcation radii (B. Li et al. 2019;
Revised: Out of these ﬁve, three are analyses of randomized smoothing, which naturally gives rise to certiﬁcation radii (B. Li et al. 2019;

Original: Carmon et al. 2019; Pinot et al. 2019).
Revised: Carmon et al. 2019; Pinot et al. 2019).

Original: Najaﬁ et al.
Revised: Najaﬁ et al.

Original: (2019) follow a learning-theoretical motivation, which results in an error bound as a function of the perturbation threshold.
Revised: (2019) follow a learning-theoretical motivation, which results in an error bound as a function of the perturbation threshold.

Original: Only Maini et al.
Revised: Only Maini et al.

Original: (2020) do not rely on randomization and still provide a complete, empirical analysis of robust error for varying perturbation thresholds 1 .
Revised: (2020) do not rely on randomization and still provide a complete, empirical analysis of robust error for varying perturbation thresholds 1 .

Original: Our contributions: In this work, we demonstrate that point-wise measures of (cid:96) p robustness are not sufﬁcient to reliably and meaningfully compare the robustness of different classiﬁers.
Revised: Our contributions: In this work, we demonstrate that point-wise measures of (cid:96) p robustness are not sufﬁcient to reliably and meaningfully compare the robustness of different classiﬁers.

Original: We show that, both in theory and practice, results of model comparisons based on point-wise measures may fail to generalize to threat models with even slightly larger or smaller ε and that robustness curves avoid this pitfall by design.
Revised: We show that, both in theory and practice, results of model comparisons based on point-wise measures may fail to generalize to threat models with even slightly larger or smaller ε and that robustness curves avoid this pitfall by design.

Original: Furthermore, we show that point-wise measures are insufﬁcient to meaningfully compare the efﬁcacy of different defense techniques when distance functions are varied, and that robustness curves, again, are able to reliably detect and visualize this property.
Revised: Furthermore, we show that point-wise measures are insufﬁcient to meaningfully compare the efﬁcacy of different defense techniques when distance functions are varied, and that robustness curves, again, are able to reliably detect and visualize this property.

Original: Finally, we analyze how scale depends on the underlying data space, choice of distance function, and distribution.
Revised: Finally, we analyze how scale depends on the underlying data space, choice of distance function, and distribution.

Original: It is our belief that the continued use of single perturbation thresholds in the adversarial robustness literature is due to a lack of awareness of the shortcomings of these measures.

Original: Based on our ﬁndings we suggest that robustness curves should become the standard tool when comparing adversarial robustness of classiﬁers, and that the perturbation threshold of threat models should be selected carefully in order to be meaningful, considering inherent characteristics of the data set.
Revised: Based on our ﬁndings we suggest that robustness curves should become the standard tool when comparing adversarial robustness of classiﬁers, and that the perturbation threshold of threat models should be selected carefully in order to be meaningful, considering inherent characteristics of the data set.

Original: We release code to reproduce all experiments presented in this paper 2 , which includes a Python module with an easily accessible interface (similar to Foolbox, Rauber et al.
Revised: We release code to reproduce all experiments presented in this paper 2 , which includes a Python module with an easily accessible interface (similar to Foolbox, Rauber et al.

Original: to calculate robustness curves for arbitrary data sets and classiﬁers.
Revised: to calculate robustness curves for arbitrary data sets and classiﬁers.

Original: The module supports classiﬁers written in most of the popular machine learning frameworks, such as TensorFlow, PyTorch and JAX.
Revised: The module supports classiﬁers written in most of the popular machine learning frameworks, such as TensorFlow, PyTorch and JAX.

Original:  M E T H O D S
Revised:  M E T H O D S

Original: An adversarial perturbation for a classiﬁer f and input-output pair ( x, y ) is a small perturbation δ with f ( x + δ ) (cid:54) =
Revised: An adversarial perturbation for a classiﬁer f and input-output pair ( x, y ) is a small perturbation δ with f ( x + δ ) (cid:54) =

Original: y .
Revised: y .

Original: Because the perturbation δ is small, it is assumed that the label y would still be the correct prediction for x + δ .
Revised: Because the perturbation δ is small, it is assumed that the label y would still be the correct prediction for x + δ .

Original: The resulting point x + δ is called an adversarial example.
Revised: The resulting point x + δ is called an adversarial example.

Original: The points vulnerable to adversarial perturbations are the points that are either already misclassiﬁed when unperturbed, or those that lie close to a decision boundary [Table] [Table]
Revised: The points vulnerable to adversarial perturbations are the points that are either already misclassiﬁed when unperturbed, or those that lie close to a decision boundary.

Original: [Table] [Table] [Table] [Table]
Revised: One tool to visualize and study the robustness behavior of a classiﬁer are robustness curves , ﬁrst used by Wong and Kolter (2018) and later formalized by C. Göpfert et al.

Original: [Table] [Table] [Table] [Table]
Revised: A robustness curve

Original: [Figure]
Revised: [Figure]

Original: One tool to visualize and study the robustness behavior of a classiﬁer are robustness curves (C.

Original: Göpfert et al. 2020).

Original: A robustness curve captures the distribution of shortest distances between a set of points and the decision boundaries of a classiﬁer:
Revised: captures the distribution of shortest distances between a set of points and the decision boundaries of a classiﬁer:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given an input space X and label set Y , distance function d on X × X , and classiﬁer f : X → Y .
Revised: Given an input space X and label set Y , distance function d on X × X , and classiﬁer f : X → Y .

Original: Assume ( x, y ) ∼ i.i.d.
Revised: Assume ( x, y ) ∼ i.i.d.

Original: P for some distribution P on X × Y .
Revised: P for some distribution P on X × Y .

Original: Then the d - robustness curve for f is the graph of the function
Revised: Then the d - robustness curve for f is the graph of the function

Original: [Equation]
Revised: [Equation]

Original: Since a model’s robustness curve shows how data points are distributed in relation to the decision boundaries of the model, it allows us to take a step back from robustness regarding a speciﬁc perturbation threshold, and insteadallows us to compare global robustness properties and their dependence on a given classiﬁer, distribution and distance function.
Revised: A model’s robustness curve shows how data points are distributed in relation to the decision boundaries of the model, essentially visualizing simultaneously an extremely large number of point-wise measures. This allows us to take a step back from robustness regarding a speciﬁc perturbation threshold and instead compare global robustness for different classiﬁers, distributions and distance functions.

Original: To see why this is relevant, consider Figure 1, which shows toy data along with two possible classiﬁers that perfectly separate the data.
Revised: To see why this is relevant, consider Figure 1, which shows toy data along with two possible classiﬁers that perfectly separate the data.

Original: For a perturbation threshold of ε , the blue classiﬁer has robust error 0 .
Revised: For a perturbation threshold of ε , the blue classiﬁer has robust error 0 .

Original: 5 , while the orange classiﬁer is perfectly robust.
Revised: 5 , while the orange classiﬁer is perfectly robust.

Original: However, for a perturbation threshold of 2 ε , the orange classiﬁer has robust error 1 , while the blue classiﬁer remains at 0 .
Revised: However, for a perturbation threshold of 2 ε , the orange classiﬁer has robust error 1 , while the blue classiﬁer remains at 0 .

Original: By freely choosing a single perturbation threshold for comparison, it is therefore possible to make either classiﬁer appear to be much better than the other, and no single threshold can capture the whole picture.
Revised: By freely choosing a single perturbation threshold for comparison, it is therefore possible to make either classiﬁer appear to be much better than the other, and no single threshold can capture the whole picture.

Original: In fact, for any two disjoint sets of perturbation thresholds, it is possible to construct a data distribution and two classiﬁers f , f (cid:48) , such that the robust error of f is lower than that of f (cid:48) for all perturbation thresholds in the ﬁrst set, and that of f (cid:48) is lower than that of f for all perturbation thresholds in the second set.
Revised: In fact, for any two disjoint sets of perturbation thresholds, it is possible to construct a data distribution and two classiﬁers f , f (cid:48) , such that the robust error of f is lower than that of f (cid:48) for all perturbation thresholds in the ﬁrst set, and that of f (cid:48) is lower than that of f for all perturbation thresholds in the second set.

Original: See Appendix A for a constructive proof.
Revised: See Appendix A for a constructive proof.

Revised: This shows that even computing multiple point-wise measures to compare two models may give misleading results.

Original:  E X P E R I M E N T S
Revised:  E X P E R I M E N T S

Original: In the following, we empirically evaluate the robustness of a number of recently published models, and demonstrate that the weaknesses of point-wise measures described above are not limited to toy examples, but occur for real-world data and models.
Revised: In the following, we empirically evaluate the robustness of a number of recently published models, and demonstrate that the weaknesses of point-wise measures described above are not limited to toy examples, but occur for real-world data and models.

Original: 3. E XPERIMENTAL S ETUP
Revised: 3. E XPERIMENTAL S ETUP

Original: We evaluate and compare the robustness of models obtained using the following training methods:
Revised: We evaluate and compare the robustness of models obtained using the following training methods:

Original: Standard training ( ST ), i. e. training without speciﬁc robustness considerations.
Revised: Standard training ( ST ), i. e., training without speciﬁc robustness considerations.

Original: Adversarial training ( AT ), see Madry et al.
Revised: Adversarial training ( AT ) (Madry et al.

Original: Training with robust loss ( KW ), see Wong and Kolter (2018).
Revised: Training with robust loss ( KW ) (Wong and Kolter 2018).

Original: Maximum margin regularization for a single (cid:96) p norm together with adversarial training ( MMR + AT ), see Croce, Andriushchenko, et al.
Revised: Maximum margin regularization for a single (cid:96) p norm together with adversarial training ( MMR + AT ) (Croce, Andriushchenko, and Hein 2019).

Original: Maximum margin regularization simultaneously for (cid:96) ∞ and (cid:96) 1 margins ( MMR-UNIV ), see Croce and Hein (2020).
Revised: Maximum margin regularization simultaneously for (cid:96) ∞ and (cid:96) 1 margins ( MMR-UNIV ) (Croce and Hein 2020).

Original: [Table]
Revised: [Table]

Original: Together with each training method, we state the threat model the trained model is optimized to defend against, eg., (cid:96) ∞ ( ε = 0 . 1) for perturbations in (cid:96) ∞ norm with perturbation threshold ε = 0 .
Revised: Together with each training method, we state the threat model the trained model is optimized to defend against, eg., (cid:96) ∞ ( ε = 0 . 1) for perturbations in (cid:96) ∞ norm with perturbation threshold ε = 0 .

Original: 1 , if any.
Revised: 1 , if any.

Original: The trained models are those made publicly available by Croce, Andriushchenko, et al.
Revised: The trained models are those made publicly available by Croce, Andriushchenko, and

Original:  (2019) 3 and Croce and Hein (2020) 4 .
Revised: Hein (2019) 3 and Croce and Hein (2020) 4 .

Original: The network architecture is a convolutional network with two convolutional layers, two fully connected layers and ReLU activation functions.
Revised: The network architecture is a convolutional network with two convolutional layers, two fully connected layers and ReLU activation functions.

Original: The evaluation is based on six real-world datasets: MNIST , Fashion-
Revised: The evaluation is based on six real-world datasets: MNIST , Fashion-

Original: MNIST ( FMNIST ) (Xiao et al. 2017), German Trafﬁc Signs ( GTS ) (Houben et al. 2013), CIFAR-10 (Krizhevsky 2009), Tiny-Imagenet-200 ( TINY-IMG ) (F.-F. Li et al. 2016), and Human Activity Recognition ( HAR ) (Anguita et al. 2013).
Revised: MNIST ( FMNIST ) (Xiao et al. 2017), German Trafﬁc Signs ( GTS ) (Houben et al. 2013), CIFAR-10 (Krizhevsky 2009), Tiny-Imagenet200 ( TINY-IMG ) (F.-F. Li et al. 2016), and Human Activity Recognition ( HAR ) (Anguita et al. 2013).

Original: For speciﬁcs on model training (hyperparameters, architecture details), refer to Appendix C. Models are 
Revised: For speciﬁcs on model training (hyperparameters, architecture details), refer to Appendix C. Models are generally trained on the full training set for the corresponding data set, and robustness curves evaluated on the full test set, unless stated otherwise.

Original: generally trained on

Original: the full

Original: training

Original: set for the corresponding data set

Original: and robustness curves evaluated on the full test set

Original: unless stated otherwise. For complex models, calculating the exact distance of a point to the closest decision boundary, and thus estimating the true robustness curve, is computationally very intensive, if not intractable.
Revised: For complex models, calculating the exact distance of a point to the closest decision boundary, and thus estimating the true robustness curve, is computationally very intensive, if not intractable.

Original: Therefore we bound the true robustness curve from below using strong adversarial attacks, which is consistent with the literature on empirical evaluation of adversarial robustness and also applicable to many different types of classiﬁers.
Revised: Therefore we bound the true robustness curve from below using strong adversarial attacks, which is consistent with the literature on empirical evaluation of adversarial robustness and also applicable to many different types of classiﬁers.

Original: We base our selection of attacks on the recommendations by
Revised: We base our selection of attacks on the recommendations by

Original: Carlini, Athalye, et al.
Revised: Carlini, Athalye, et al.

Original: Speciﬁcally, we use the (cid:96) 2 -attack proposed by (Carlini and Wagner 2017) for (cid:96) 2 robustness curves and PGD (Madry et al.
Revised: Speciﬁcally, we use the (cid:96) 2 -attack proposed by (Carlini and Wagner 2017) for (cid:96) 2 robustness curves and PGD (Madry et al.

Original: 2018) for (cid:96) ∞ robustness curves.
Revised: 2018) for (cid:96) ∞ robustness curves.

Original: For both attacks, we use the implementations of Foolbox (Rauber et al. 2017).
Revised: For both attacks, we use the implementations of Foolbox (Rauber et al. 2017).

Original: See Appendix C for information on adversarial attack hyperparameters.
Revised: See Appendix C for information on adversarial attack hyperparameters.

Original: In the following, “robustness curve” refers to this empirical approximation of the true robustness curve.
Revised: In the following, “robustness curve” refers to this empirical approximation of the true robustness curve.

Original: 3. T HE WEAKNESSES OF POINT - WISE MEASURES
Revised: 3. T HE WEAKNESSES OF POINT - WISE MEASURES

Original: Point-wise measures are used to quantify robustness of classiﬁers by measuring the robust test error for a speciﬁc distance function and a perturbation threshold (eg., (cid:96) ∞ ( ε = 4 / 255) ).
Revised: Point-wise measures are used to quantify robustness of classiﬁers by measuring the robust test error for a speciﬁc distance function and a perturbation threshold (eg., (cid:96) ∞ ( ε = 4 / 255) ).

Original: In Table 1 we show three point-wise measures to compare the robustness of ﬁve different classiﬁers on CIFAR-10 .
Revised: In Table 1 we show three point-wise measures to compare the robustness of ﬁve different classiﬁers on CIFAR-10 .

Original: If we compare the robustness of the four robust training methods (latter four columns of the table) based on the ﬁrst point-wise threat model (cid:96) ∞ ( ε = 1 / 255) (ﬁrst row of the table), we can see that the classiﬁer trained with AT is the most robust, followed by MMR + AT , followed by KW , and MMR-UNIV results in the least robust classiﬁer.
Revised: If we compare the robustness of the four robust training methods (latter four columns of the table) based on the ﬁrst point-wise threat model (cid:96) ∞ ( ε = 1 / 255) (ﬁrst row of the table), we can see that the classiﬁer trained with AT is the most robust, followed by MMR + AT , followed by KW , and MMR-UNIV results in the least robust classiﬁer.

Original: However, if we increase the ε of our threat model to ε = 4 /(second row of the table), KW is more robust than AT .
Revised: However, if we increase the ε of our threat model to ε = 4 /(second row of the table), KW is more robust than AT .

Original: For a even larger ε (third row of the table), we would conclude that MMR-UNIV is preferable over AT , and that AT results in the least robust classiﬁer.
Revised: For a even larger ε (third row of the table), we would conclude that MMR-UNIV is preferable over AT , and that AT results in the least robust classiﬁer.

Original: All three statements are true for the particular perturbation threshold ( ε ), and the magnitude of all perturbation thresholds is reasonable: publications on adversarial robustness typically evaluate
Revised: All three statements are true for the particular perturbation threshold ( ε ), and the magnitude of all perturbation thresholds is reasonable: publications on adversarial robustness typically evaluate

Original: CIFAR-10 on perturbation thresholds (cid:54) 10 / 255 for (cid:96) ∞ perturbations.
Revised: CIFAR-10 on perturbation thresholds (cid:54) 10 / 255 for (cid:96) ∞ perturbations.

Original: Meaningful conclusions on the robustness of the classiﬁers relative to each other can not be made without taking all possible ε into account.
Revised: Meaningful conclusions on the robustness of the classiﬁers relative to each other can not be made without taking all possible ε into account.

Original: In other words, a global perspective is needed.
Revised: In other words, a global perspective is needed.

Original: [Figure]
Revised: [Figure]

Original: 3.2. A GLOBAL PERSPECTIVE
Revised: 3.2. A GLOBAL PERSPECTIVE

Original: Figure 2 shows the robustness of different classiﬁers for the (cid:96) ∞ (right plot) and (cid:96) 2 (left plot) distance functions from a global perspective using robustness curves.
Revised: Figure 2 shows the robustness of different classiﬁers for the (cid:96) ∞ (right plot) and (cid:96) 2 (left plot) distance functions from a global perspective using robustness curves.

Original: The plot reveals why the three pointwise measures (marked by vertical black dashed lines in the left plot) lead to different results in the relative ranking of robustness of the classiﬁers.
Revised: The plot reveals why the three pointwise measures (marked by vertical black dashed lines in the left plot) lead to different results in the relative ranking of robustness of the classiﬁers.

Original: Both for the classiﬁers trained to be robust against attacks in (cid:96) ∞ distance (left plot) and (cid:96) 2 distance (right plot), we can observe multiple intersections of robustness curves, corresponding to changes in the relative ranking of the robustness of the compared classiﬁers.
Revised: Both for the classiﬁers trained to be robust against attacks in (cid:96) ∞ distance (left plot) and (cid:96) 2 distance (right plot), we can observe multiple intersections of robustness curves, corresponding to changes in the relative ranking of the robustness of the compared classiﬁers.

Original: The robustness curves allow us to reliably compare the robustness of classiﬁers for all possible perturbation thresholds.
Revised: The robustness curves allow us to reliably compare the robustness of classiﬁers for all possible perturbation thresholds.

Original: Furthermore, the curves clearly show the perturbation threshold intervals with strong and weak robustness for each classiﬁer, and are not biased by an arbitrarily chosen perturbation threshold.
Revised: Furthermore, the curves clearly show the perturbation threshold intervals with strong and weak robustness for each classiﬁer, and are not biased by an arbitrarily chosen perturbation threshold.

Original: 3.2. O VERFITTING TO SPECIFIC PERTURBATION THRESHOLDS
Revised: 3.2. O VERFITTING TO SPECIFIC PERTURBATION THRESHOLDS

Original: In addition to the problem of robustness curve intersection, relying on point-wise robustness measures to evaluate adversarial robustness is prone to overﬁtting when designing training procedures.
Revised: In addition to the problem of robustness curve intersection, relying on point-wise robustness measures to evaluate adversarial robustness is prone to overﬁtting when designing training procedures.

Original: Figure 3 shows (cid:96) ∞ robustness curves for MMR + AT with (cid:96) ∞ threat model as provided by Croce, Andriushchenko, et al.
Revised: Figure 3 shows (cid:96) ∞ robustness curves for MMR + AT with (cid:96) ∞ threat model as provided by Croce, Andriushchenko, and Hein (2019).

Original: The models trained on MNIST and FMNIST both show a change in slope, which could be a sign of overﬁtting to the speciﬁc threat models for which the classiﬁers were optimized for, since the change of slope occurs approximately at the chosen perturbation threshold ε .
Revised: The models trained on MNIST and FMNIST both show a change in slope, which could be a sign of overﬁtting to the speciﬁc threat models for which the classiﬁers were optimized for, since the change of slope occurs approximately at the chosen perturbation threshold ε .

Original: This showcases a potential problem with the use of point-wise measures during training.
Revised: This showcases a potential problem with the use of point-wise measures during training.

Original: The binary separation of “small” and “large” perturbations based on the perturbation threshold is not sufﬁcient to capture the intricacies of human perception under perturbations, but a simpliﬁcation based on the idea that perturbations below the perturbation threshold should almost certainly not lead to a change in classiﬁcation.
Revised: The binary separation of “small” and “large” perturbations based on the perturbation threshold is not sufﬁcient to capture the intricacies of human perception under perturbations, but a simpliﬁcation based on the idea that perturbations below the perturbation threshold should almost certainly not lead to a change in classiﬁcation.

Original: If a training procedure moves decision boundaries so that data points lie just beyond this threshold, it may achieve a low robust error, without furthering the actual goals of adversarial robustness research.
Revised: If a training procedure moves decision boundaries so that data points lie just beyond this threshold, it may achieve a low robust error, without furthering the actual goals of adversarial robustness research.

Original: Using robustness curves for evaluation cannot prevent this effect, but can be used to detect it.
Revised: Using robustness curves for evaluation cannot prevent this effect, but can be used to detect it.

Original: 3.2. T RANSFER OF ROBUSTNESS ACROSS DISTANCE FUNCTIONS
Revised: 3.2. T RANSFER OF ROBUSTNESS ACROSS DISTANCE FUNCTIONS

Original: In the following, we analyze to which extent properties of robustness curves transfer across different choices of distance functions.
Revised: In the following, we analyze to which extent properties of robustness curves transfer across different choices of distance functions.

Original: If properties transfer, it may not be necessary to individually analyze robustness for each distance function.
Revised: If properties transfer, it may not be necessary to individually analyze robustness for each distance function.

Original: In Figure 4 we compare the robustness of different models for the (cid:96) ∞ (left plot) and (cid:96) 2 (right plot) distance functions.
Revised: In Figure 4 we compare the robustness of different models for the (cid:96) ∞ (left plot) and (cid:96) 2 (right plot) distance functions.

Original: The difference to Figure 2 is that the models (indicated by colour) are the same models in the left plot and in the right plot.
Revised: The difference to Figure 2 is that the models (indicated by colour) are the same models in the left plot and in the right plot.

Original: We ﬁnd that for MMR + AT , the (cid:96) ∞ threat model leads to better robustness than the (cid:96) 2 threat model both for (cid:96) ∞ and (cid:96) 2 robustness curves.
Revised: We ﬁnd that for MMR + AT , the (cid:96) ∞ threat model leads to better robustness than the (cid:96) 2 threat model both for (cid:96) ∞ and (cid:96) 2 robustness curves.

Original: In fact, MMR + AT with the (cid:96) ∞ threat model even leads to better (cid:96) ∞ and (cid:96) 2 robustness curves than MMR-UNIV , which is speciﬁcally designed to improve robustness for all (cid:96) p norms.
Revised: In fact, MMR + AT with the (cid:96) ∞ threat model even leads to better (cid:96) ∞ and (cid:96) 2 robustness curves than MMR-UNIV , which is speciﬁcally designed to improve robustness for all (cid:96) p norms.

Original: Overall, the plots are visually similar.
Revised: Overall, the plots are visually similar.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: However, since both plots contain multiple robustness curve intersections, the ranking of methods remains sensitive to the choice of perturbation threshold.
Revised: However, since both plots contain multiple robustness curve intersections, the ranking of methods remains sensitive to the choice of perturbation threshold.

Original: For example, a perturbation threshold of ε = 3 / 255 (vertical black dashed line) for the (cid:96) ∞ distance function (left subplot) shows that the classiﬁer trained with MMR + AT ( (cid:96) 2 ( ε = 0 . 1) ) is approximately as robust as the classiﬁer trained with
Revised: For example, a perturbation threshold of ε = 3 / 255 (vertical black dashed line) for the (cid:96) ∞ distance function (left subplot) shows that the classiﬁer trained with MMR + AT ( (cid:96) 2 ( ε = 0 . 1) ) is approximately as robust as the classiﬁer trained with

Original: MMR-UNIV .
Revised: MMR-UNIV .

Original: The same perturbation threshold for the (cid:96) 2 distance function (right subplot) shows that the classiﬁer trained with MMR + AT is more robust than the classiﬁer trained with MMR-UNIV for (cid:96)threat models.
Revised: The same perturbation threshold for the (cid:96) 2 distance function (right subplot) shows that the classiﬁer trained with MMR + AT is more robust than the classiﬁer trained with MMR-UNIV for (cid:96)threat models.

Original: Using typical perturbation thresholds from the literature for each distance function does not alleviate this issue: At perturbation threshold ε = 2 / 255 for (cid:96) ∞ distance, the classiﬁer trained with MMR + AT ( (cid:96) 2 ( ε = 0 . 1) ) is more robust than the one trained with MMR-UNIV , while at perturbation threshold ε = 0 . 1 for (cid:96) 2 distance, the opposite is true.
Revised: Using typical perturbation thresholds from the literature for each distance function does not alleviate this issue: At perturbation threshold ε = 2 / 255 for (cid:96) ∞ distance, the classiﬁer trained with MMR + AT ( (cid:96) 2 ( ε = 0 . 1) ) is more robust than the one trained with MMR-UNIV , while at perturbation threshold ε = 0 . 1 for (cid:96) 2 distance, the opposite is true.

Original: This shows that even when robustness curves across various distance functions are qualitatively similar, this may be obscured by the choice of threat model(s) to compare on.
Revised: This shows that even when robustness curves across various distance functions are qualitatively similar, this may be obscured by the choice of threat model(s) to compare on.

Original: We also emphasize that in general, robustness curves across various distance functions may be qualitatively dis similar.
Revised: We also emphasize that in general, robustness curves across various distance functions may be qualitatively dis similar.

Original: In particular:
Revised: In particular:

Original: For linear classiﬁers, the shape of a robustness curve is identical for distances induced by different (cid:96) p norms.
Revised: For linear classiﬁers, the shape of a robustness curve is identical for distances induced by different (cid:96) p norms.

Original: This follows from Theorem 2 in Appendix B, which is an extension of a weaker result in C. Göpfert et al.
Revised: This follows from Theorem 2 in Appendix B, which is an extension of a weaker result in C. Göpfert et al.

Original: For non-linear classiﬁers, different (cid:96) p norms may induce different robustness curve shapes.
Revised: For non-linear classiﬁers, different (cid:96) p norms may induce different robustness curve shapes.

Original: See C. Göpfert et al.
Revised: See C. Göpfert et al.

Original: for an example.
Revised: for an example.

Original: Even for linear classiﬁers, robustness curve intersections do not transfer between distances induced by different (cid:96) p norms.
Revised: Even for linear classiﬁers, robustness curve intersections do not transfer between distances induced by different (cid:96) p norms.

Original: That is, for two linear classiﬁers, there may exist p, p (cid:48) such that the robustness curves for the (cid:96) p distance intersect, but not the robustness curves for the (cid:96) p (cid:48) distance.
Revised: That is, for two linear classiﬁers, there may exist p, p (cid:48) such that the robustness curves for the (cid:96) p distance intersect, but not the robustness curves for the (cid:96) p (cid:48) distance.

Original: See Appendix A for an example.
Revised: See Appendix A for an example.

Original: [Figure]
Revised: [Figure]

Original: 3. O N THE RELATIONSHIP BETWEEN SCALE AND DATA
Revised: 3. O N THE RELATIONSHIP BETWEEN SCALE AND DATA

Original: As the previous sections show, robustness curves can be used to reveal properties of robust models that may be obscured by point-wise measures.
Revised: As the previous sections show, robustness curves can be used to reveal properties of robust models that may be obscured by point-wise measures.

Original: However, some concept of scale, that is, some way to judge whether a perturbation is small or large, remains necessary.
Revised: However, some concept of scale, that is, some way to judge whether a perturbation is small or large, remains necessary.

Original: Especially when robustness curves intersect, it is crucial to be able to judge how critical it is for a model to be stable under the given perturbations.
Revised: Especially when robustness curves intersect, it is crucial to be able to judge how critical it is for a model to be stable under the given perturbations.

Original: For many pairs of distance function and data set, canonical perturbation thresholds have emerged in the literature, but to the best of our knowledge, no reasons for these choices are given.
Revised: For many pairs of distance function and data set, canonical perturbation thresholds have emerged in the literature, but to the best of our knowledge, no reasons for these choices are given.

Original: Since the assumption behind adversarial examples is that small perturbations should not affect classiﬁcation behavior, the question of scale cannot be answered independently of the data distribution.
Revised: Since the assumption behind adversarial examples is that small perturbations should not affect classiﬁcation behavior, the question of scale cannot be answered independently of the data distribution.

Original: In order to understand how to interpret different perturbation sizes, it can be helpful to understand how strongly the data point would need to be perturbed to actually change the correct classiﬁcation.
Revised: In order to understand how to interpret different perturbation sizes, it can be helpful to understand how strongly the data point would need to be perturbed to actually change the correct classiﬁcation.

Original: We call this the inter-class distance and analyze the distribution of inter-class distances for several popular data sets.
Revised: We call this the inter-class distance and analyze the distribution of inter-class distances for several popular data sets.

Original: In Figure 5 we compare the inter-class distance distributions in (cid:96) ∞ , (cid:96) 2 , and (cid:96) 1 norm for all data sets considered in this work.
Revised: In Figure 5 we compare the inter-class distance distributions in (cid:96) ∞ , (cid:96) 2 , and (cid:96) 1 norm for all data sets considered in this work.

Original: We observe that for the (cid:96) 1 and (cid:96) 2 norms, the shape of the curves is similar across data sets, but their extent is determined by the dimensionality of the data space.
Revised: We observe that for the (cid:96) 1 and (cid:96) 2 norms, the shape of the curves is similar across data sets, but their extent is determined by the dimensionality of the data space.

Original: In the (cid:96) ∞ norm, vastly different curves emerge for the different data sets.
Revised: In the (cid:96) ∞ norm, vastly different curves emerge for the different data sets.

Original: We hypothesize that, because the inter-class distance distributions vary more strongly for (cid:96) ∞ distances than for (cid:96) 1 distances, the results of robustifying a model w. r. t. (cid:96) ∞ distances may depend more strongly on the underlying data distribution than the results of robustifying w. r. t. (cid:96) 1 distances.
Revised: We hypothesize that, because the inter-class distance distributions vary more strongly for (cid:96) ∞ distances than for (cid:96) 1 distances, the results of robustifying a model w. r. t. (cid:96) ∞ distances may depend more strongly on the underlying data distribution than the results of robustifying w. r. t. (cid:96) 1 distances.

Original: This is an interesting avenue for future work.
Revised: This is an interesting avenue for future work.

Original: When we look at the smallest inter-class distances in the (cid:96) ∞ norm (where all distances lie in the interval [0 , 1] ), we can make several observations.
Revised: When we look at the smallest inter-class distances in the (cid:96) ∞ norm (where all distances lie in the interval [0 , 1] ), we can make several observations.

Original: Because the smallest inter-class distance for
Revised: Because the smallest inter-class distance for

Original: MNIST in the (cid:96) ∞ norm is around 0 .
Revised: MNIST in the (cid:96) ∞ norm is around 0 .

Original: 9 , we can see that transforming an input from one class to one from a different class almost always requires completely ﬂipping at least one pixel from almost-black to almost-white or vice versa.
Revised: 9 , we can see that transforming an input from one class to one from a different class almost always requires completely ﬂipping at least one pixel from almost-black to almost-white or vice versa.

Original: For the other datasets, the inter-class distance distributions are more spread out than the inter-class distance distribution of MNIST .
Revised: For the other datasets, the inter-class distance distributions are more spread out than the inter-class distance distribution of MNIST .

Original: We observe that for CIFAR-10 with (cid:96) ∞ perturbations of size (cid:62) 0 .
Revised: We observe that for CIFAR-10 with (cid:96) ∞ perturbations of size (cid:62) 0 .

Original: 25 , it becomes possible to transform samples from different classes into each other, so starting from this threshold, any classiﬁer must necessarily trade off between accuracy and robustness.
Revised: 25 , it becomes possible to transform samples from different classes into each other, so starting from this threshold, any classiﬁer must necessarily trade off between accuracy and robustness.

Original: The shapes of the curves and the threshold from which any classiﬁer must necessarily trade of between accuracy and robustness differ strongly between data sets – refer to Table 2 for exact values for the threshold.
Revised: The shapes of the curves and the threshold from which any classiﬁer must necessarily trade of between accuracy and robustness differ strongly between data sets – refer to Table 2 for exact values for the threshold.

Original: In Table 2, we summarize the smallest and largest inter-class distances in different norms together with additional information about the size, number of classes, and dimensionality of the all the data sets we consider in this work.
Revised: In Table 2, we summarize the smallest and largest inter-class distances in different norms together with additional information about the size, number of classes, and dimensionality of the all the data sets we consider in this work.

Original: The values correspond directly to Figure 5, but even in this simpliﬁed view, we can quickly make out key differences between the data sets.
Revised: The values correspond directly to Figure 5, but even in this simpliﬁed view, we can quickly make out key differences between the data sets.

Original: Compare, for example, MNIST and GTS : While it appears reasonable to expect (cid:96) ∞ robustness of 0 .
Revised: Compare, for example, MNIST and GTS : While it appears reasonable to expect (cid:96) ∞ robustness of 0 .

Original: 3 for MNIST , the same threshold for GTS is not possible.
Revised: 3 for MNIST , the same threshold for GTS is not possible.

Original: Relating Table 2 and Figure 3, we ﬁnd entirely plausible the strong robustness
Revised: Relating Table 2 and Figure 3, we ﬁnd entirely plausible the strong robustness

Original: [Table]
Revised: [Table]

Original: results for MNIST , and the small perturbation threshold for GTS .
Revised: results for MNIST , and the small perturbation threshold for GTS .

Original: Based on inter-class distances we also expect less (cid:96) ∞ robustness for CIFAR-10 than for FMNIST , but not as seen in Figure 3.
Revised: Based on inter-class distances we also expect less (cid:96) ∞ robustness for CIFAR-10 than for FMNIST , but not as seen in Figure 3.

Original: In any case, it is safe to say that, when judging the robustness of a model by a certain threshold, that number must be set with respect to the distribution the model operates on.
Revised: In any case, it is safe to say that, when judging the robustness of a model by a certain threshold, that number must be set with respect to the distribution the model operates on.

Original: Overall, the strong dependence of robustness curves on the data set and the chosen norm, emphasizes the necessity of informed and conscious decisions regarding robustness thresholds.
Revised: Overall, the strong dependence of robustness curves on the data set and the chosen norm, emphasizes the necessity of informed and conscious decisions regarding robustness thresholds.

Original: We provide an easily accessible reference in the form of Table 2, that should prove useful while judging scales in a threat model.
Revised: We provide an easily accessible reference in the form of Table 2, that should prove useful while judging scales in a threat model.

Original:  D I S C U S S I O N
Revised:  D I S C U S S I O N

Original: We have demonstrated that comparisons of robustness of different classiﬁers using point-wise measures can be heavily biased by the choice of perturbation threshold and distance function of the threat model, and that conclusions about rankings of classiﬁers with regards to their robustness based on point-wise measures therefore only provide a narrow view of the actual robustness behavior of the classiﬁers.
Revised: We have demonstrated that comparisons of robustness of different classiﬁers using point-wise measures can be heavily biased by the choice of perturbation threshold and distance function of the threat model, and that conclusions about rankings of classiﬁers with regards to their robustness based on point-wise measures therefore only provide a narrow view of the actual robustness behavior of the classiﬁers.

Original: Further, we have demonstrated different ways of using robustness curves to overcome the shortcomings of point-wise measures, and therefore recommend using them as the standard tool for comparing the robustness of classiﬁers.
Revised: Further, we have demonstrated different ways of using robustness curves to overcome the shortcomings of point-wise measures, and therefore recommend using them as the standard tool for comparing the robustness of classiﬁers.

Original: Finally, we have demonstrated how suitable perturbation thresholds necessarily depend on the data they pertain to.
Revised: Finally, we have demonstrated how suitable perturbation thresholds necessarily depend on the data they pertain to.

Original: It is our hope that practitioners and researchers alike will use the methodology proposed in this work, especially when developing and comparing adversarial defenses, and carefully motivate any concrete threat models they might choose, taking into account all available context.
Revised: It is our hope that practitioners and researchers alike will use the methodology proposed in this work, especially when developing and comparing adversarial defenses, and carefully motivate any concrete threat models they might choose, taking into account all available context.

Original: Limitations: Computing approximate robustness curves for state-of-the-art classiﬁers and large data sets is computationally very intensive, due to the need of computing approximate minimal adversarial perturbations with strong adversarial attacks.
Revised: Limitations: Computing approximate robustness curves for state-of-the-art classiﬁers and large data sets is computationally very intensive, due to the need of computing approximate minimal adversarial perturbations with strong adversarial attacks.

Original: Developing adversarial attacks which are both strong and fast is an ongoing challenge in the ﬁeld of adversarial robustness.
Revised: Developing adversarial attacks which are both strong and fast is an ongoing challenge in the ﬁeld of adversarial robustness.

Revised: One way to reduce the computational cost is to approximate the robustness curves by computing a set of point-wise measures.

Revised: However, since robustness curves may intersect at arbitrarily many points, this may give misleading results.

Revised: It would be interesting to investigate how closely robustness curves need to be approximated in order to estimate the number of intersections, if any, and their location, with high certainty.

Original: Another limitation of our work is the focus on a small group of distance functions (mainly (cid:96) ∞ and (cid:96) 2 norms).
Revised: Another limitation of our work is the focus on a small group of distance functions (mainly (cid:96) ∞ and (cid:96) 2 norms).

Original: Even though it does intuitively make sense that models should at least be robust against these types of perturbations, a more general evaluation able to consider more distance functions simultaneously could be advantageous. 
Revised: Even though it does intuitively make sense that models should at least be robust against these types of perturbations, a more general evaluation able to consider more distance functions simultaneously could be advantageous. A R O B U S T N E S S C U RV E S W I T H A R B I T R A RY I N T E R S E C T I O N S Theorem 1.

Revised: Let T 1 , T 2 ⊂ R > 0 be two disjoint ﬁnite sets.

Original: C. M ODEL TRAINING
Revised: Then there exists a distribution P on R × { 0 , 1 } and two classiﬁers c 1 , c 2 : R → { 0 , 1 } such that R c 1 |·| ( t ) < R c 2 |·| ( t ) for all t ∈ T 1 and R c 1 |·| ( t ) > R c 2 |·| ( t ) for all t ∈ T 2 .

Revised: Proof.

Revised: Without loss of generality, assume that T 1 = { t 1 , . . .

Revised: t n } and T 2 = { t (cid:48) 1 , . . .

Revised: , t (cid:48) n } with t i < t (cid:48)

Revised: i <

Revised: t

Revised: i +1 for i ∈ { 1 , . . . , n } .

Revised: We will construct c 1 , c 2 such that the robustness curves

Revised: R c 1 |·| ( · ) , R c 2 |·| ( · ) intersect at exactly the points ( t i + t (cid:48) i ) / 2 and ( t i + t (cid:48)

Revised: i +1 ) / 2 on the interval ( t 1 , t (cid:48) n ] .

Revised: Let d = t (cid:48) n and

Revised: P (cid:18) − d − t i + t (cid:48) i +  ,(cid:19) = P (cid:18) d + t i + t (cid:48) i ,(cid:19) =

Revised: 4 n +and

Revised: P (cid:18)

Revised: [Equation]

Revised: , 0 (cid:19) = 4 n + 1 .

Revised: Let c 1 ( x ) = 1 x (cid:62) − d and c 2 ( x ) = 1 x (cid:62) d .

Revised: Both classiﬁers have perfect accuracy on P , meaning that

Revised: R c i |·| (0) = 0 .

Revised: The closest point to the decision boundary of c 1 is − d − t 1 2 with weight 1 4 n +1 , so

Revised: R c 1 |·| ( t 1 2 )

Revised: = 1 4 n +1 .

Revised: The second-closest point is − d − t 1 + t (cid:48)

Revised: 2with weight 2 4 n +1 , so R c 1 |·| ( t 1 + t (cid:48) 2 2 ) = 3 4 n +1 , and so on.

Revised: Meanwhile, the closest point to the decision boundary of c 2 is d + t 1 + t (cid:48)

Revised: 1with weight 2 4 n +1 ,

Original: {+so R c 
+}
Revised: {+so R c 
+}

Revised: |·| ( t 1 + t (cid:48) 1 2 ) = 2 4 n +1 , the second-closest point is d t 2 + t (cid:48)

Revised: 2with weight 2 4 n +1 , so

Revised: R c

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: (cid:48)

Revised: [Equation]

Revised: = 4 4 n +1 , and so on.

Revised: Example 1.

Revised: To see that robustness curve intersections do not transfer between different (cid:96) p norms, consider the example in Figure 6.

Revised: The blue and orange linear classiﬁers both perfectly separate the displayed data.

Revised: The (cid:96) ∞ robustness curves of the classiﬁers do not intersect, meaning that the robust error of the blue classiﬁer is always better than that of the orange classiﬁer.

Revised: In (cid:96) 2 distance, the robustness curves intersect, so that there is a range of perturbation sizes where the orange classiﬁer has better robust error than the blue classiﬁer.

Revised: [Equation]

Revised: Theorem 2.

Revised: Let f ( x ) =

Revised: sgn

Revised: ( w T x + b ) be a linear classiﬁer.

Revised: Then the shape of the robustness curve for f regarding an (cid:96) p norm-induced distance does not depend on the choice of p .

Revised: It holds that

Revised: [Equation]

Revised: Lemma 1.

Revised: Let x ∈ R m with w T x + b (cid:54) = 0 .

Revised: Let p ∈

Revised: [1 , ∞ ] and q such that 1 p + 1 q = 1 , where we take 1 ∞ = 0 .

Revised: Then

Revised: [Equation]

Revised: and the minimum is attained by

Revised: [Equation]

Original: {+where x 1 ∞− 1 = x 0 = 
+}
Revised: {+where x 1 ∞− 1 = x 0 = 
+}

Revised: and e j is the j -th unit vector.

Revised: Proof of Theorem 2.

Revised: By Hölder’s inequality, for any δ ,

Revised: [Equation]

Revised: For δ such that sgn( w T ( x + δ ) + b ) (cid:54) = sgn

Revised: ( w T x + b ) it follows that

Revised: [Equation]

Revised: Using the identity q = pp − 1 , it is easy to check that for every p ∈ [1 , ∞ ] , with δ as deﬁned in

Revised: Equation (3),

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: , so that w T ( x +

Revised: [Equation]

Revised: b = 0 , and

Revised: [Equation]

Revised: Item 1 shows that δ is a feasible point, while Item 2 in combination with Equation (5) shows that (cid:107) δ (cid:107) p is minimal.

Revised: Using Lemma 1, we are ready to prove Theorem 2.

Revised: Proof.

Revised: By deﬁnition,

Revised: [Equation]

Revised: [Equation]

Revised: We can split R p 1 ( ε ) into the disjoint sets

Revised: [Equation]

Revised: [Equation]

Original: We use the same model architecture as Croce, Andriushchenko, et al.
Revised: [Equation] Choose q 1 , q 2 such that 1 p i + 1 q i = 1 . By Lemma 1, and using that f ( x ) = sgn( w T x + b ) , [Equation]

Revised: This shows that

Revised: [Equation]

Original:  and Wong and Kolter (2018).
Revised: [Equation] R I M E N TA L D E TA I L S C. M ODEL TRAINING We use the same model architecture as Croce, Andriushchenko, and Hein (2019) and Wong and Kolter (2018).

Original: Unless explicitly stated otherwise, the trained models are taken from Croce, Andriushchenko, et al.
Revised: Unless explicitly stated otherwise, the trained models are taken from Croce, Andriushchenko, and Hein (2019).

Original: The exact architecture of the model is: Convolutional layer (number of ﬁlters: 16, size: 4x4, stride: 2), ReLu activation function, convolutional layer (number of ﬁlters: 32, size: 4x4, stride: 2), ReLu activation function, fully connected layer (number of units: 100), ReLu activation function, output layer (number of units depends on the number of classes).
Revised: The exact architecture of the model is: Convolutional layer (number of ﬁlters: 16, size: 4x4, stride: 2), ReLu activation function, convolutional layer (number of ﬁlters: 32, size: 4x4, stride: 2), ReLu activation function, fully connected layer (number of units: 100), ReLu activation function, output layer (number of units depends on the number of classes).

Original: All models are trained with Adam Optimizer (Kingma and Ba 2014) for 100 epochs, with batch size 128 and a default learning rate of 0.001.
Revised: All models are trained with Adam Optimizer (Kingma and Ba 2014) for 100 epochs, with batch size 128 and a default learning rate of 0.001.

Original: More information on the training can be found in the experimental details section of the appendix of Croce, Andriushchenko, et al.
Revised: More information on the training can be found in the experimental details section of the appendix of Croce, Andriushchenko, and Hein (2019).

Original: The trained models are those made publicly available by Croce, Andriushchenko, et al. (2019) 5 and Croce and Hein (2020) 6 .
Revised: The trained models are those made publicly available by Croce, Andriushchenko, and Hein (2019) 5 and Croce and Hein (2020) 6 .

Original: C. A PPROXIMATED ROBUSTNESS CURVES
Revised: C. A PPROXIMATED ROBUSTNESS CURVES

Original: We use state-of-the-art adversarial attacks to approximate the true minimal distances of input datapoints to the decision boundary of a classiﬁer for our adversarial robustness curves (see Deﬁnition 1).
Revised: We use state-of-the-art adversarial attacks to approximate the true minimal distances of input datapoints to the decision boundary of a classiﬁer for our adversarial robustness curves (see Deﬁnition 1).

Original: We base our selection of attacks on the recommendations of Carlini, Athalye, et al.
Revised: We base our selection of attacks on the recommendations of Carlini, Athalye, et al.

Original: Specifically, we use the following attacks: For (cid:96) 2 robustness curves we use the (cid:96) 2 -attack proposed by Carlini and Wagner (2017) and for (cid:96) ∞ robustness curves we use PGD (Madry et al. 2018).
Revised: Speciﬁcally, we use the following attacks: For (cid:96) 2 robustness curves we use the (cid:96) 2 -attack proposed by Carlini and Wagner (2017) and for (cid:96) ∞ robustness curves we use PGD (Madry et al. 2018).

Original: For both attacks, we use the implementations of Foolbox  (Rauber et al. 2017).
Revised: For both attacks, we use the implementations of Foolbox (Version 2.4) (Rauber et al. 2017).

Original: For the (cid:96) ∞ attack,  we use the standard hyperparameters of the Foolbox implementation.
Revised: For the (cid:96) ∞ attack, the implementation of Foolbox automatically performs a hyperparameter search over different epsilon and uses the smallest resulting adversarial perturbation. For the rest of the hyperparameters, we use the standard values of the Foolbox implementation.

Original: For the (cid:96)1 and (cid:96) 2 attacks we increase the number of binary search steps that are used to ﬁnd the optimal tradeoff-constant between distance and conﬁdence from 5 to 10, which we found empirically to improve the results.
Revised: For the (cid:96) 2 attack, we increase the number of binary search steps that are used to ﬁnd the optimal tradeoff-constant between distance and conﬁdence from 5 to 10, which we found empirically to improve the results.

Original: For the rest of the hyperparameters, we again use the standard values of the Foolbox implementation.
Revised: For the rest of the hyperparameters, we again use the standard values of the Foolbox implementation.

Original: C. C OMPUTATIONAL ARCHITECTURE
Revised: C. C OMPUTATIONAL ARCHITECTURE

Original: We executed all programs on an architecture with 2 x 
Revised: We executed all programs on an architecture with 2 x Intel Xeon(R) CPU E5-2640 v4 @ 2.4 GHz,x Nvidia GeForce GTX 1080 TI 12G and 128 GB RAM. [Figure] [Figure] [Equation]

Revised: As we pointed out in Section

Original: [Table]
Revised: [Figure]

Revised: adversarial robustness of classiﬁers trained on CIFAR-10 is usually evaluated at a perturbation threshold ε ∈ { 2 / 255 , 4 / 255 , 8 / 255 } for the (cid:96) ∞ norm.

Revised: Robustness curves allow us to investigate robustness of classiﬁers for perturbation thresholds beyond those which are used in the literature.

Revised: It should not be necessary for the model to be invariant under large perturbations, if these perturbations are clearly perceptible or change the “correct” classiﬁcation of the input.

Revised: However, the thresholds that models are currently optimized for are small enough that even larger perturbations may not be perceptible.

Revised: Figure 7 shows four images of CIFAR-10 (top row), together with adversarial examples (bottom row).

Revised: With perturbation sizes ε ∈ { 17 / 255 , 18 / 255 } , the perturbations are more than two times larger than the biggest perturbation threshold used in the literature, and still almost imperceptible for untrained humans.

Revised: E R O B U S T N E S S C U RV E S F O R L A R G E R M O D E L S

Revised: In Section 3, we demonstrate the usefulness of robustness curves on a small convolutional network architecture used by Croce, Andriushchenko, and Hein (2019).

Revised: The choice of a small architecture allows us to compute robustness curves for a large number of different defensive strategies with limited computational resources.

Revised: Figure 8 shows approximate robustness curves for two state-of-theart robust models with a large network architecture (WideResNet-28-10), computed for a sample of

Revised: 1000 data points from CIFAR-10 .

Revised: Due to the small number of points used, the approximation may be rough, so the following observations should be taken with a grain of salt.

Revised: Both robust models are indeed much more robust than the model obtained by standard training even for perturbation thresholds that are signiﬁcantly larger than the threshold of 8 / 255 that the models are optimized for.

Revised: This observation may help decide whether it is worthwhile to stop using a conventionally trained model, sacriﬁcing accuracy for robustness.

Original: [Table]
Revised: Wu et al. (2020) has slightly worse accuracy than Sehwag et al. (2020) roughly up to perturbation size 1 / 255 . This is a trade-off for better accuracy between perturbation sizes 4 / 255 and 0 .

Revised: From perturbation size 0 .

Revised: 1 onward, Sehwag et al.

Revised: (2020) appears to have slightly better accuracy than Wu et al.

Revised: This observation may help decide which of the two robust models is preferable, based on the robustness requirements of a concrete application.

Revised: The gap between the performance of Wu et al.

Revised: (2020) and Sehwag et al.

Revised: (2020) is even wider at perturbation size 0 .

Revised: 04 than 8 / 255 , but

Revised: overall, the robustness curves of the robust models are quite similar.

Revised: This observation may help decide whether it is worthwhile to switch from one model to the other, if one of the models is already in use or preferable for other reasons.