Original: The Geometry of Adversarial Subspaces
Revised: The Geometry of Adversarial Subspaces

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Artiﬁcial neural networks (ANNs) are constructed using well-understood mathematical operations, and yet their high-dimensional, non-linear, and compositional nature has hindered our ability to provide an intuitive description of how and why they produce any particular output.
Revised: Artiﬁcial neural networks (ANNs) are constructed using well-understood mathematical operations, and yet their high-dimensional, non-linear, and compositional nature has hindered our ability to provide an intuitive description of how and why they produce any particular output.

Original: A striking example of this lack of understanding is our inability to design networks that are robust to adversarial input perturbations, which are often imperceptible to a human observer but cause signiﬁcant undesirable changes in the network’s response.
Revised: A striking example of this lack of understanding is our inability to design networks that are robust to adversarial input perturbations, which are often imperceptible to a human observer but cause signiﬁcant undesirable changes in the network’s response.

Original: The primary contribution of this work is to further our understanding of the decision boundary geometry of ANN classiﬁers by utilizing such adversarial perturbations.
Revised: The primary contribution of this work is to further our understanding of the decision boundary geometry of ANN classiﬁers by utilizing such adversarial perturbations.

Original: For this purpose, we deﬁne adversarial subspaces, which are spanned by orthogonal directions of minimal perturbation to the decision boundary from any given input sample.
Revised: For this purpose, we deﬁne adversarial subspaces, which are spanned by orthogonal directions of minimal perturbation to the decision boundary from any given input sample.

Original: We ﬁnd that the decision boundary lies close to input samples in a large subspace, where the distance to the boundary grows smoothly and sub-linearly as one increases the dimensionality of the subspace.
Revised: We ﬁnd that the decision boundary lies close to input samples in a large subspace, where the distance to the boundary grows smoothly and sub-linearly as one increases the dimensionality of the subspace.

Original: We undertake analysis to characterize the geometry of the boundary, which is more curved within the adversarial subspace than within a random subspace of equal dimensionality.
Revised: We undertake analysis to characterize the geometry of the boundary, which is more curved within the adversarial subspace than within a random subspace of equal dimensionality.

Original: To date, the most widely used defense against test-time adversarial attacks is adversarial training, where one incorporates adversarial attacks into the training procedure.
Revised: To date, the most widely used defense against test-time adversarial attacks is adversarial training, where one incorporates adversarial attacks into the training procedure.

Original: Using our analysis, we provide new insight into the consequences of adversarial training by quantifying the increase in boundary distance within adversarial subspaces, the redistribution of proximal class labels, and the decrease in boundary curvature.
Revised: Using our analysis, we provide new insight into the consequences of adversarial training by quantifying the increase in boundary distance within adversarial subspaces, the redistribution of proximal class labels, and the decrease in boundary curvature.

Original:  Introduction
Revised:  Introduction

Original: Artiﬁcial neural networks (ANNs) have been highly performant on common machine learning tasks, but their application space is limited by their susceptibility to adversarial attacks (Szegedy et al., 2014), which underscores a general lack of understanding for how and why they make their decisions.
Revised: Artiﬁcial neural networks (ANNs) have been highly performant on common machine learning tasks, but their application space is limited by their susceptibility to adversarial attacks (Szegedy et al., 2014), which underscores a general lack of understanding for how and why they make their decisions.

Original: Examples produced by adversarial attacks are a worst-case demonstration of an ANN’s inability to gracefully cope with identity-preserving shifts or distortions of its inputs.
Revised: Examples produced by adversarial attacks are a worst-case demonstration of an ANN’s inability to gracefully cope with identity-preserving shifts or distortions of its inputs.

Original: To construct one, an adversary must perturb the input in a small but speciﬁc way such that the output of the network changes signiﬁcantly.
Revised: To construct one, an adversary must perturb the input in a small but speciﬁc way such that the output of the network changes signiﬁcantly.

Original: Adversarial perturbations are deﬁned to have minimal length, resulting in inputs that are as close to the network’s decision boundary as possible.
Revised: Adversarial perturbations are deﬁned to have minimal length, resulting in inputs that are as close to the network’s decision boundary as possible.

Original: Here, we present a method that uses an untargeted adversarial attack objective to ﬁnd the multi-dimensional subspace where an ANN’s decision boundary is closest to any given input sample.
Revised: Here, we present a method that uses an untargeted adversarial attack objective to ﬁnd the multi-dimensional subspace where an ANN’s decision boundary is closest to any given input sample.

Original: The resulting adversarial subspace allows us to visualize and understand ANN decision boundaries in terms of their curvature, their shape, and their proximity to the input.
Revised: The resulting adversarial subspace allows us to visualize and understand ANN decision boundaries in terms of their curvature, their shape, and their proximity to the input.

Original: One existing technique to explain ANN classiﬁcations has relied on two-dimensional visualizations of the decision surface, although it suﬀers from a number of shortcomings, most notably high-dimensional points resolving to the same projected location, distortions of distances, and a lack of invertibility from the two-dimensional space back to the high-dimensional decision space (Féraud & Clérot, 2002; Rauber et al., 2016;
Revised: One existing technique to explain ANN classiﬁcations has relied on two-dimensional visualizations of the decision surface, although it suﬀers from a number of shortcomings, most notably high-dimensional points resolving to the same projected location, distortions of distances, and a lack of invertibility from the two-dimensional space back to the high-dimensional decision space (Féraud & Clérot, 2002; Rauber et al., 2016;

Original: Rodrigues et al., 2019).
Revised: Rodrigues et al., 2019).

Original: A popular alternative strategy is to investigate the decision boundary in the high-dimensional input space itself, early attempts of which (Golland, 2001; Baehrens et al., 2010) lead the way to the modern interest in adversarial attacks (Biggio et al., 2013; Szegedy et al., 2014).
Revised: A popular alternative strategy is to investigate the decision boundary in the high-dimensional input space itself, early attempts of which (Golland, 2001; Baehrens et al., 2010) lead the way to the modern interest in adversarial attacks (Biggio et al., 2013; Szegedy et al., 2014).

Original: For example, one can measure the distance to the decision boundary for random perturbation
Revised: For example, one can measure the distance to the decision boundary for random perturbation

Original: [Figure]
Revised: [Figure]

Original: + = + = ..
Revised: + = + = ..

Revised: predictedclass

Original: [Figure]
Revised: [Figure]

Original: There is an ongoing debate on the cause, prevalence, and uniformity of adversarial examples. Several studies have supported a hypothesis that most adversarial examples inhabit relatively dense regions of the input space (Goodfellow et al., 2014; Tabacof & Valle, 2016; MoosaviDezfooli et al., 2017; He et al., 2018).
Revised: from orthogonal directions which are proximal to initial adversarial example (Tramèr et Alternatively, much work has demonstrated the utility of visualizing two-dimensional cross-sections of the decision space between adversarial examples and random orthogonal directions (Warde-Farley & Goodfellow, 2016; Yu et al., 2019a) or orthogonal test set examples (Swirszcz et al., 2019).

Original: However, both early and recent works have suggested alternative explanations, including that adversarial examples are isolated pockets in the decision space (Szegedy et al., 2014), or are the result ofthin “spikes” of incorrect classiﬁcation regions in an otherwise correct classiﬁcation region (Yu et al., 2019). Here we provide quantitative evaluation supporting the hypothesis that decision boundariesare close to input samples ina large subspace, where the distance to the boundary grows sub-linearly as one increases the dimensionality of the subspace. Additional foundational work in this space has suggested that it is the linearity ofdeep neural networks that leads to adversarial examples (Goodfellow et al., 2014), although conﬂicting more recent work has suggested that increasing decision boundary linearity causes increased robustness (Li et al., 2019; Moosavi-Dezfooli et al., 2019; Terjék, 2020).
Revised: Unlike all of the existing approaches to characterize decision boundaries in the input space, which rely on at most one adversarial direction per input, we propose jointly deﬁning a subspace of adversarial directions.

Original: One of our major contributions tothis literature is a rigorous derivation and discussion of the analysis of  decision boundary curvature. We apply the analysis to determine the linearity of the boundary, conﬁrming that adversarial training linearizes the boundary. We further compare the boundary curvature in adversarial subspaces and random subspaces to show a increase in curvature in adversarial subspaces with and without adversarial training.
Revised: The resulting analysis leads to a general explanation of classiﬁer decision boundaries, as well as adversarial susceptibility, in the subspace where the boundary is closest to the input example.

Original: As a further demonstration of the utility of our approach, we investigate the most widely used technique for defending against adversarial attacks, so-called adversarial training.
Revised: As a further demonstration of the utility of our approach, we investigate the most widely used technique for defending against adversarial attacks, so-called adversarial training.

Original: The defense method requires including adversarial perturbations in the training process by either augmenting the dataset with pre-computed examples (Ilyas et al., 2019) or by incorporating an attack model into the training procedure (Goodfellow et al., 2014; Madry et al., 2018).
Revised: The defense method requires including adversarial perturbations in the training process by either augmenting the dataset with pre-computed examples (Ilyas et al., 2019) or by incorporating an attack model into the training procedure (Goodfellow et al., 2014; Madry et al., 2018).

Original: It has been recently demonstrated that adversarial training performs well on the CIFARmachine learning dataset when enough resources are dedicated to hyperparameter tuning (Gowal et al., 2020), although the performance is still far from that on unperturbed images.d i s t a n c e t o d e c i s i o n b n d . natural robust       dimension of adversarial space 1. d i s t a n c e c
Revised: It has been recently demonstrated that adversarial training performs well on the CIFARmachine learning dataset when enough resources are dedicated to hyperparameter tuning (Gowal et al., 2020), although the performance is still far from that on unperturbed images.

Original: oe ff i c i e n t 10. We provide a new perspective on this defense by observing that it increases the distance to the decision boundary in the entire adversarial subspace, even though the method utilizes only a single adversarial perturbation per input sample.
Revised: We provide a new perspective on this defense by observing that it increases the distance to the decision boundary in the entire adversarial subspace, even though the method utilizes only a single adversarial perturbation per input sample.

Original: We additionally ﬁnd that adversarial training increases the number of alternative label regions in adversarial subspaces, and changes the relative distribution of nearby alternate classes.
Revised: We additionally ﬁnd that adversarial training increases the number of alternative label regions in adversarial subspaces, and changes the relative distribution of nearby alternate classes.

Original: Herein, we will focus on deﬁning the method and summarizing the subspace with a continued application of understanding adversarial training.
Revised: Herein, we will focus on deﬁning the method and summarizing the subspace with a continued application of understanding adversarial training.

Original:  Methods
Revised:  Methods

Original: In the following we will outline the novel methods employed in this study.
Revised: In the following we will outline the novel methods employed in this study.

Original: Namely, the method for ﬁnding adversarial subspaces is ﬁrst described, and then we provide an overview of how we measure decision boundary curvature.
Revised: Namely, the method for ﬁnding adversarial subspaces is ﬁrst described, and then we provide an overview of how we measure decision boundary curvature (with additional derivations given in Appendix

Original:  Our primary investigation was conducted using the CIFAR-10 dataset (Krizhevsky & Hinton, 2009) and compared two identical
Revised: A). Our primary investigation was conducted using the CIFAR-10 dataset (Krizhevsky & d i s t a n c e t o d e c i s i o n b n d . natural robust       dimension of adversarial space 1. d i s t a n c e c

Original:  ResNet-50 ANNs (He et al., 2016) trained with and without adversarial examples.
Revised: oe ff i c i e n t 10. Observe that mean Hinton, 2009) and compared two identical ResNet-50 ANNs (He et al., 2016) trained with and without adversarial examples.

Revised: In Appendix G we provide results on the MNIST dataset, which demonstrates largely consistent trends across datasets and across weight initialization seeds.

Original: All distances measured are with respect to the ‘ 2 -norm.
Revised: All distances measured are with respect to the ‘ 2 -norm.

Original: Additional dataset and model details can be found in Appendix C.
Revised: Additional dataset and model details can be found in Appendix B.

Original: 2. Finding adversarial subspaces
Revised: 2. Finding adversarial subspaces

Original: We aim to ﬁnd the set of orthonormal vectors that span the subspace where an input sample has minimal distance to the decision boundary.
Revised: We aim to ﬁnd the set of orthonormal vectors that span the subspace where an input sample has minimal distance to the decision boundary.

Original: In order to achieve this, we follow a greedy optimization approach to span an adversarial subspace up to a desired number of dimensions.
Revised: In order to achieve this, we follow a greedy optimization approach to span an adversarial subspace up to a desired number of dimensions.

Original: More precisely, consider some neural network, f , that receives an n -pixel input image, x ∈ R n , and produces an output vector of probabilities associated with l label categories, ˆ y ∈ R l .
Revised: More precisely, consider some neural network, f , that receives an n -pixel input image, x ∈ R n , and produces an output vector of probabilities associated with l label categories, ˆ y ∈ R l .

Original: For a given input image, we wish to ﬁnd an ordered set of orthogonal vectors,  ∆ =
Revised: For a given input, we wish to ﬁnd an ordered set of orthogonal vectors, i.e. ∆ =

Original: [ δ 1 , δ 2 , . . .
Revised: [ δ 1 , δ 2 , . . .

Original: , δ m ] , m ≤ n , where for each vector we  solve the following optimization problem: minimize δ m [Equation] subject to
Revised: , δ m ] , m ≤ n , such that h δ k , δ m i = 0 , ∀ k < m . Thus for each vector, we want to solve an optimization objective that minimizes || δ m || p , where ||

Original: [Equation]

Original: Here, i is the correct class index and ||

Original: δ || p
Revised: δ || p

Original: =
Revised: =

Original: ( P i | δ | p ) 1 p is the l p norm.
Revised: ( P i | δ | p ) 1 p is the l p norm.

Revised: We expect the optimizer to explicitly and jointly consider the boundedness of the input domain, force the perturbations to be orthogonal, and produce perturbations that result in new (incorrect) classiﬁer outputs.

Original: The objectivein (1) is intractable to solve for deep neural networks, but is well approximated by the untargeted adversarial attack objective 1 , which modiﬁes the conditional to be subject to max j 

Revised: Such an objective is intractable to solve for deep neural networks, but is well approximated by the untargeted adversarial attack objective 1 , which modiﬁes the conditional to be subject to max j

Original: = i f j ( x + δ )
Revised: = i f j ( x + δ )

Original: − f i ( x + δ ) > 0 .
Revised: − f i ( x + δ ) > 0, where i is the correct class index.

Original: The length of these vectors is nearly synonymous with distances to the decision boundary, where the actual decision boundary would be an inﬁnitesimal step towards x from x + δ .
Revised: The length of these vectors is nearly synonymous with distances to the decision boundary, where the actual decision boundary would be an inﬁnitesimal step towards x from x + δ .

Original: We must further relax the objective to a diﬀerentiable approximation for ﬁnding a gradient based solution.
Revised: We must further relax the objective to a diﬀerentiable approximation for ﬁnding a gradient based solution.

Original: While there are many such approximations, we opted for aslight modiﬁcation of that which was used by Szegedy et al.
Revised: While there are many such approximations, we opted for a modiﬁcation of that which was used by Szegedy et al.

Original: (2014), which is to minimize the inverse of the cross entropy loss (the training objective of the model), weighed with a distance penalization term:
Revised: (2014) to ﬁnd singular adversarial directions, which is to minimize the inverse of the cross entropy loss (the training objective of the model), weighed with a distance penalization term. Speciﬁcally, ‘ =

Original: [Equation]
Revised: [Equation]

Original: with the cross entropy loss deﬁned as L CE ( x ) = − log
Revised: with the cross entropy loss deﬁned as L CE ( x ) = − log

Original: exp f i ( x ) exp

Original: [Equation]
Revised: [Equation]

Original: j f j ( x ) .
Revised: x ) .

Original: With the loss term from equation (2), we can now deﬁne our objective function as implemented: minimize
Revised: Using this loss term, we can now deﬁne our objective function as implemented: minimize

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this maximizes the model loss with respect to the true class label while keeping the perturbation as small as possible.
Revised: Intuitively, this maximizes the model loss with respect to the true class label while keeping the perturbation as small as possible.

Revised: The ﬁrst constraint in Equation 1 is to ensure that the image is adversarial 2 , the second keeps the perturbed images within the allowable pixel bounds, and the last one enforces orthogonality of all found adversarial subspace dimensions.

Original: We ﬁnd the optimal value of constant κ in equation (3) with a binary search performed during run time.
Revised: We found the optimal value of constant κ in equation (1) with a binary search performed during run time.

Original: The optimization problem (3) is solved using the greedy interior point optimizer implemented by Ipopt (Wächter & Biegler, 2006).
Revised: The optimization problem was solved using the greedy interior-point optimizer implemented by Ipopt (Wächter & Biegler, 2006).

Original: adv dir0. adv dir0. adv dir0.
Revised: Once the adversarial vectors are found, we conducted experiments to measure the distance to the decision boundary in the space spanned by the vectors (i.e. within the adversarial subspace).

Original: adv dir0. adv dir0. [Figure] adv dir0.
Revised: The experiment was repeated on 100 test images that were chosen to have equal label representation (10 images per label) and to be correctly classiﬁed by both networks.

Original: adv dir0. [Figure] adv dir0.
Revised: For each test image we found 50 orthonormal adversarial vectors, which allowed us to deﬁne a sequentially growing list of adversarial subspaces with dimensionality increasing from 1 to 50.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: For a given image and subspace we sampled directions via a linear combination of normally distributed variables, which has been previously proven to provide an even sampling (Muller, 1959).

Original: [Figure]
Revised: Finally, for each random direction we performed a binary search to ﬁnd the decision boundary.

Original: 2. Measuring decision boundary curvature
Revised: 2. Measuring decision boundary curvature

Original: As before, let f i , f j : R m → R be the functions corresponding to i and j -th logits of a neural network, f .
Revised: As before, let f i , f j : R m → R be the functions corresponding to i and j -th logits of a neural network, f .

Original: Let F ij def =
Revised: Let F ij def =

Original: f
Revised: f

Original: i − f j .
Revised: i − f j .

Original: Then the ( i, j ) -decision boundary is the set
Revised: Then the ( i, j ) -decision boundary is the set

Original: [Equation]
Revised: [Equation]

Original: The set B ij is deﬁned independently of the sign of F ij (that is, B ij =
Revised: The set B ij is deﬁned independently of the sign of F ij (that is, B ij =

Original: B ji ), although doing so will produce a change of sign in our calculations.
Revised: B ji ), although doing so will produce a change of sign in our calculations.

Original: Therefore we will continue to follow the above convention by identifying, at a point x , i with the correct label associated to x , and j with the incorrect label.
Revised: Therefore we will continue to follow the above convention by identifying, at a point x , i with the correct label associated to x , and j with the incorrect label.

Original: This convention (which agrees with Moosavi-Dezfooli et al., 2018), ensures that positive (resp.
Revised: This convention (which agrees with Moosavi-Dezfooli et al., 2018), ensures that positive (resp.

Original: negative) curvature of B ij near x corresponds to B ij bowing inward towards (resp.
Revised: negative) curvature of B ij near x corresponds to B ij bowing adv dir0. adv dir0. adv dir0. adv dir0.

Original: away from) x (see Figure 1 for an example of positive curvature).
Revised: adv dir0. [Figure] adv dir0. adv dir0.

Revised: [Figure]

Revised: [Figure]

Revised: adv dir0.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: inward towards (resp.

Revised: away from) x (see Figure 1 for an example of positive curvature).

Original: If the curvature along B ij is zero near x then B ij is aﬃne.
Revised: If the curvature along B ij is zero near x then B ij is aﬃne.

Original: The complete details of how our calculations were derived, and numerical considerations therein, are relegated to Appendix B, however we give a brief intuitive account below.
Revised: The complete details of how our calculations were derived, and numerical considerations therein, are relegated to Appendix A, however we give a brief intuitive account below.

Original: In our curvature analysis we use a method known as principal curvature decomposition to understand the degree of linearity of the decision boundary.
Revised: In our curvature analysis we use a method known as principal curvature decomposition to understand the degree of linearity of the decision boundary.

Original: Since the decomposition is performed on a level set of the decision space, which in general is of codimension 1, it yields n − 1 scalars and vectors, called the principal curvatures and principle curvature directions , respectively.
Revised: Since the decomposition is performed on a level-set of the decision space, which in general is of codimension 1, it yields n − 1 scalars and vectors, called the principal curvatures and principle curvature directions , respectively.

Original: The curvature directions point in the directions of maximal and minimal surface curvature, while the principal curvatures themselves indicate the signed magnitude of the curvature.
Revised: The curvature directions point in the directions of maximal and minimal surface curvature, while the principal curvatures themselves indicate the signed magnitude of the curvature.

Original: Similar to principal component analysis (PCA), which organizes the space as an orthogonal basis of directions of maximal variance, principal curvature analysis organizes the (tangential) space of the decision boundary into orthogonal directions of maximum curvature.
Revised: Similar to principal component analysis (PCA), which organizes the space as an orthogonal basis of directions of maximal variance, principal curvature analysis organizes the (tangential) space of the decision boundary into orthogonal directions of maximum curvature.

Original: The principle curvatures and associated directions are the eigen vectors and values of a linear operator s : R m → R m called the shape operator .
Revised: The principle curvatures and associated directions are the eigenvectors and eigenvalues of a linear operator s : R m → R m called the shape operator .

Original: In order to study the curvature over an adversarial  subspace we introduce the subspace pullback of s
Revised: In order to study the curvature over an adversarial (or random) subspace we introduce the subspace pullback of s

Original: [Equation]
Revised: [Equation]

Original: where P ∈ R m × n is an orthogonal projection, so that the image of P T spans the adversarial subspace of R m in question.
Revised: where P ∈ R m × n is an orthogonal projection, so that the image of P T spans the adversarial subspace of R m in question.

Original: This deﬁnition of (cid:0) P T (cid:1) ∗ s essentially restricts the shape operator to a linear subspace of the tangential space and is motivated by the principles of diﬀerential geometry, details of which can be found in Remark 1 and the surrounding material in
Revised: This deﬁnition of (cid:0) P T (cid:1) ∗ s essentially restricts the shape operator to a linear subspace of the tangential space and is motivated by the principles of diﬀerential geometry, details of which can be found in Remark 1 and the surrounding material in

Original: Appendix B.
Revised: Appendix A.

Original: The pullback shape operator allows us to compare the amount of curvature in the adversarial subspace to that of the entire response space.
Revised: The pullback shape operator allows us to compare the amount of curvature in the adversarial subspace to that of the entire response space.

Original: We note that while previous work has utilized similar analysis (viz.

Original: Moosavi-Dezfooli et al., 2018; 2019; Golden et al., 2019), the authors omit complete details of the formulas presented.

Original: Therefore one of our major contributions to this literature is a rigorous derivation and discussion of these quantities, which we provide in Appendix B.

Original:  Results
Revised:  Results

Original: By deﬁnition the distance to the decision boundary increases for each additional dimension of an adversarial subspace.
Revised: By deﬁnition, the distance to the decision boundary increases for each additional dimension of an adversarial subspace.

Original: The circles in the top left panel of Figure 2 indicate these distances, averaged across the test images.
Revised: The circles in the top left panel of Figure 2 indicate these distances, averaged across the test images, which are found by greedily minimizing Equation 1.

Original: The distance to the decision boundary grows sub-linearly as one increases the dimensionality of the subspace, with the fastest growth occurring in the ﬁrst couple of dimensions.
Revised: The distance to the decision boundary grows sub-linearly as one increases the dimensionality of the subspace, with the fastest growth occurring in the ﬁrst couple of dimensions.

Revised: The leftmost box plots of the top left panel in ﬁgure 2 as well as the Appendix Table 2 conﬁrms that the mean initial perturbation distances are comparable to more traditional adversarial attacks (Madry et al., 2018) and decision boundary distance estimates (Ding et al., 2020)).

Original: While this gives an idea of the lower-bound of distances to the decision boundary, it does not provide information about the behavior of the boundary elsewhere in the subspace.
Revised: While this gives an idea of the lower-bound of distances to the decision boundary, it does not provide information about the behavior of the boundary elsewhere in the subspace.

Original: To better understand this, we uniformly sampled 100 random directions within a given subspace and performed a binary search to ﬁnd the decision boundary along those directions.
Revised: To better understand this we uniformly sampled 100 random directions within a given subspace and performed a binary search to ﬁnd the decision boundary along those directions.

Original: We only report distances for decision boundary points that are within the ‘ inf allowable pixel box, which was never fewer than 40% of the subspace samples (see Appendix A Figure 7).
Revised: We only report distances for decision boundary points that are within the ‘ inf allowable pixel box, which was never fewer than 40% of the subspace samples (see Appendix C Figure 7).

Original: The box plots in the top left panel of Figureindicate the distribution of decision boundary distances across the sampled directions and test images.
Revised: The box plots in the top left panel of Figure 2 indicate the distribution of decision boundary distances across the sampled directions and test images.

Original: Using a small sample of images, we found that the perturbation was discernible at after an ‘ 2 -distance of 1.5, which occurs after about 20 dimensions for the adversarially trained network, and after greater than 50 dimensions for the naturally trained network.
Revised: Using a small sample of images, we found that the perturbation was discernible at after an ‘ 2 -distance of 1.5, which occurs after about 20 dimensions for the adversarially trained network, and after greater thandimensions for the naturally trained network.

Original: The bottom left panel of Figure 2 plots a distance coeﬃcient that is deﬁned as the maximum distance to the decision boundary within an adversarial subspace divided by the longest vector deﬁning the subspace (which is always associated with the highest dimension number).
Revised: The bottom left panel of Figure 2 plots a distance coeﬃcient that is deﬁned as the maximum distance to the decision boundary within an adversarial subspace divided by the longest vector deﬁning the subspace (which is always associated with the highest dimension number).

Original: For example, a value of 2 would indicate that the largest sampled distance is twice as far away as the largest found adversarial perturbation and a value of 1 would mean that the decision boundary is spherical in the subspace.
Revised: For example, a value of 2 would indicate that the largest sampled distance is twice as far away as the largest found adversarial perturbation.

Original: Finally, a complete description of the  distancesfound is conveyed as histograms in the rightmost panels, with dimensionality increasing from left to right, then top to bottom.
Revised: Finally, a complete description of the boundary distances is conveyed as histograms in the rightmost panels, with dimensionality increasing from left to right, then top to bottom.

Original: Together, this provides further evidence that the decision boundary grows consistently, without large variations in distance between the found adversarial subspace axes.
Revised: Together, this provides further evidence that the decision boundary grows consistently, without large variations in distance between the found adversarial subspace axes.

Original: In Figure 3 we visualize two-dimensional cross-sections of the decision surface.
Revised: In Figure 3 we visualize two-dimensional cross-sections of the decision surface.

Original: The horizontal axis of each cross-section corresponds to an adversarial subspace perturbation, the black arrow indicates the farthest decision boundary point, and the vertical axis is found by orthogonalizing the direction of the farthest decision boundary point with respect to the adversarial perturbation using one step of the Gram-Schmidt process.
Revised: The horizontal axis of each cross-section corresponds to an adversarial subspace perturbation, the black arrow indicates the farthest decision boundary point, and the vertical axis is found by orthogonalizing the direction of the farthest decision boundary point with respect to the adversarial perturbation using one step of the Gram-Schmidt process.

Original: The cross-sections are ordered according to the distance coeﬃcient, which is displayed in the upper-right corner.
Revised: The cross-sections are ordered according to the distance coeﬃcient, which is displayed in the upper-right corner.

Original: We emphasize that because these are cross-sections, they avoid the shortcomings of typical low-dimensional decision space projections noted in the introduction.
Revised: We emphasize that because these are cross-sections, they avoid the shortcomings of typical low-dimensional decision space projections noted in the introduction.

Original: Among the visualizations one can see examples of when the boundary increases smoothly from one axis to the next, and also the more rare (as indicated by Figure 2) cases when the boundary protrudes quite far from the origin when between the axes.
Revised: Among the visualizations one can see examples of when the boundary increases smoothly from one axis to the next, and also the more rare (as indicated by Figure 2) cases when the boundary protrudes quite far from the origin when between the axes.

Original: From this analysis we conclude that adversarial examples are not isolated or due to sharp protrusions into an otherwise homogeneously labeled region.
Revised: From this analysis we conclude that adversarial examples are not isolated or due to sharp protrusions into an otherwise homogeneously labeled region.

Original: Rather, there exists a smooth increase in distance to the decision boundary from the extreme case of an initially found adversarial example to semantically diﬀerentiated images.
Revised: Rather, there exists a smooth increase in distance to the decision boundary from the extreme case of an initially found adversarial example to semantically diﬀerentiated images.

Original: [Table] In Figure 4 we provide an analysis of boundary curvature for several experimental conditions that vary the type of training, the location of the boundary, and subspace deﬁning vectors.
Revised: In Figure 4 we provide an analysis of boundary curvature for several experimental conditions that vary the type of training, the location of the boundary, and subspace deﬁning vectors.

Original: This analysis reveals that there is a relationship between the boundary ﬂatness (i.e. linearity) and the existence of adversarial examples.
Revised: This analysis reveals that there is a relationship between the boundary ﬂatness (i.e. linearity) and the existence of adversarial examples.

Revised: Panels A and B report the mean curvature (i.e. the mean of the eigenvalues of the shape operator) per point on the decision boundary.

Original: Both model types have larger boundary curvature foradversarial boundary points  when compared to boundary points separating test set examples, as well as within adversarial subspaces.
Revised: Both model types have larger boundary curvature for boundary points separating clean images from adversarial images (adversarial boundary; Panel B) when compared to boundary points separating test set examples (test boundary; Panel A).

Original: However, we found that adversarial training tends to linearize the decision boundary, both between correctly labeled test images and near adversarial examples.
Revised: We additionally found that adversarial training tends to linearize the decision boundary, both between correctly labeled test images and near adversarial examples.

Original: We further demonstrated the relationship by showing that thecurvature within adversarial subspaces is notably larger than in  random  subspaces.
Revised: Panel C is a scatter plot of the principal curvatures (i.e. eigenvalues of the shape operator) in the high-dimensional input space for all of the adversarial boundary points. We consistently ﬁnd both positive and negative curvature at each boundary point and for both training types. Using the pullback deﬁned in Equation 3, we can measure the subspace principal curvatures for random (Panel D) and adversarial (Panel E) subspaces.

Original: When limited to these subspaces, the diﬀerence in curvature values between naturally and adversarially trained networks is less pronounced  – suggesting that much of the linearization occurring during adversarial training isnot along directions aligned withalternative adversarial examples.
Revised: The curvature within adversarial subspaces is notably larger than in random subspaces. The diﬀerence in curvature values between naturally and adversarially trained networks is less pronounced when limited to random subspaces – suggesting that much of the linearization occurring during adversarial training is along directions aligned with adversarial examples.

Original: This may be due to the fact that adversarial training utilizes only the ﬁrst few found adversarial directions.
Revised: [Table]

Original: In addition to the earlier comparisons between natural and adversarial network training, we measured the diversity of adversarial classes near test samples.
Revised: In addition to the earlier comparisons between natural and adversarial network training, we measured the diversity of adversarial classes near test samples.

Original: The leftmost subplot of Figure 5 gives the number of unique adversarial classes found in the ﬁrst 50 dimensions of the [Figure] U n i q u e a d v e r s a r i a l c l a ss e s
Revised: The leftmost subplot of Figure 5 gives the number of unique adversarial classes found in the ﬁrst 50 dimensions of the adversarial subspaces, averaged across a uniform sampling of correctly labeled test images.

Original: [Figure]

Original: bird cat

Original: [Figure]

Original: adversarial subspaces, averaged across a uniform sampling of correctly labeled test images.

Original: The other plots show the distribution of adversarial classes present in the subspaces with respect to the origin label.
Revised: The other plots show the distribution of adversarial classes present in the subspaces with respect to the origin label.

Original: These plots suggest that adversarial training results in a more even distribution of alternate classes near any given test sample, which we further quantify by measuring the discrete entropy of each distribution in Table 1.
Revised: These plots suggest that adversarial training results in a more even distribution of alternate classes near any given test sample, which we further quantify by measuring the discrete entropy of each distribution in Table 1.

Original: It is apparent from our analysis that adversarial training pushes the decision boundary away from the origin in all dimensions tested, even though few adversarial perturbations are provided per training sample.
Revised: It is apparent from our analysis that adversarial training pushes the decision boundary away from the origin in all dimensions tested, even though few adversarial perturbations are provided per training sample.

Original: In the top left panel of Figure 2, the dots indicate that the largest increase in boundary distance from one dimension to the next is after the ﬁrst, although the adversarial images corresponding to the ﬁrst several dimensions were often still imperceptibly changed from the original.
Revised: In the top left panel of Figure 2, the dots indicate that the largest increase in boundary distance from one dimension to the next is after the ﬁrst, although the adversarial images corresponding to the ﬁrst several dimensions were often still imperceptibly changed from the original.

Original: Furthermore, in the bottom left panel of Figure 2 one can see that the adversarial trained network consistently produces smaller coeﬃcients, indicating that the boundary size is less variable between the subspace axes.
Revised: Furthermore, in the bottom left panel of Figure 2 one can see that the adversarial trained network consistently produces smaller coeﬃcients, indicating that the boundary size is less variable between the subspace axes.

Original: In addition to (or more probably as a consequence of) increasing the distance to the boundary, adversarial training further modiﬁes the density of alternative class labels near test images. 
Revised: In addition to (or more probably as a consequence of) increasing the distance to the boundary, adversarial training further modiﬁes the density of alternative class labels near test images. This is evident in Figure 5, [Figure]

Original: This is evident in Figure 5, which demonstrates that adversarial training increases the diversity of alternative classes near test images on the left and modiﬁes the proximal class distribution on the right.
Revised: U n i q u e a d v e r s a r i a l c l a ss e s [Figure] which demonstrates that adversarial training increases the diversity of alternative classes near test images on the left and modiﬁes the proximal class distribution on the right.

Original: While some categories, like “car” and “truck” are intuitively adjacent, the distribution of classes generally failed to match any clear intuitions.
Revised: While some categories, like “car” and “truck” are intuitively adjacent, the distribution of classes generally failed to match any clear intuitions.

Original: For example, observe the lack of symmetry between cars and trucks, or that adversarial training switches the ”frog“ category’s most frequent alternate class from “bird” to “cat”.
Revised: For example, observe the lack of symmetry between cars and trucks, or that adversarial training switches the ”frog“ category’s most frequent alternate class from “bird” to “cat”.

Original: Indeed, Table 1 suggests that adversarial training spreads out the distribution among classes, suggesting that the decision surface is more evenly utilizing the high-dimensional space.
Revised: Indeed, Table 1 suggests that adversarial training spreads out the distribution among classes, suggesting that the decision surface is more evenly utilizing the high-dimensional space.

Original:  Related Work
Revised:  Related Work

Original: Related adversarial subspace studies: The work of Tramèr et al.
Revised: Related adversarial subspace studies: The work of Tramèr et al.

Original: (2017) is most similar to our adversarial decomposition, in that they reported the dimensionality of transferable adversarial examples.
Revised: (2017) is most similar to our adversarial decomposition, in that they reported the dimensionality of a subspace in the proximity of a given adversarial perturbation, although do not comment on the number or proximity of alternative adversarial examples.

Revised: Work from Ma et al.

Original: The authors estimated the  dimensionality  of a subspace  in  the proximity ofa given adversarial perturbation.
Revised: (2018) measured the local intrinsic dimensionality (LID) of adversarial subspaces, which indirectly estimates adversarial subspace dimensionality by measuring the distance from an adversarial example to a batch of nearest unperturbed dataset examples. This idea has been extended to algorithms for detecting adversarial examples (Ma et al., 2018; Mao et al., 2020), although Athalye et al. (2018) generated adversarial examples with indistinguishable LID scores from natural examples. In terms of understanding ANN classiﬁers, such an approach leaves open the questions of how proximal the decision boundary is from input samples in greater than one dimension or the actual dimensionality of adversarial subspaces.

Original: In contrast tothis local approach, we pursue a more global examination of the adversarial subspace surrounding a given input image by searching for orthogonaldirections of adversarial vulnerability.
Revised: Our approach is the ﬁrst to pursue a more global examination of the decision boundary where it is nearest to test samples by deﬁning adversarial subspaces using multiple orthogonal adversarial directions.

Original: Our emphasis on the subspace where the decision boundary is closest to test images is also unique when compared to work that characterizes the decision boundary in random directions, which will be considerably farther away from the origin (e.g.  Tabacof & Valle, 2016; Liu et al., 2017; He et al., 2018).
Revised: Our emphasis on the subspace where the decision boundary is closest to test images is also unique when compared to work that characterizes the decision boundary in random directions, which will be considerably farther away from the origin (e.g. Fawzi et al., 2016; Tabacof & Valle, 2016; Liu et al., 2017; He et al., 2018).

Original: Finally, all of these studies perform a clipping operation after deﬁning or ﬁnding the desired directions, which results in a non-orthogonal set (see Appendix E for a more detailed discussion).
Revised: Finally, most of these studies perform a clipping operation after deﬁning or ﬁnding the desired directions, which results in a non-orthogonal set (see Appendix E for a more detailed discussion).

Original: We overcome the problem of clipping pixel values outside the valid pixel range by including the valid range as inequality constraints in the optimization process.
Revised: We overcome the problem of clipping pixel values outside the valid pixel range by including the valid range as inequality constraints in the optimization process.

Revised: Our work is complementary to previous studies that investigate the relationship between adversarial robustness and input dimensionality (Amsaleg et al., 2017; Simon-Gabriel et al., 2019), since the comparative trends observed between robust and natural networks are relatively unchanged with small (MNIST) or large (CIFAR) dimensional inputs.

Original: Neuron response curvature: The use of response curvature to understand nonlinear functions has a long history in neuroscience (for a review, see Gollisch & Herz, 2012) as well as models of neural computation (e.g. Zetzsche & Krieger, 2001;
Revised: Neuron response curvature: The use of response curvature to understand nonlinear functions has a long history in neuroscience (for a review, see Gollisch & Herz, 2012) as well as models of neural computation (e.g. Zetzsche & Krieger, 2001;

Original: Rust et al., 2005; Golden et al.,2016).
Revised: Rust et al., 2005; Golden et al.,

Original: The principal curvature decomposition was previously employed in this context by Golden et al.
Revised: The principal curvature decomposition was previously employed in this context by Golden et al.

Original: (2019), who used it to understand invariance and selectivity of individual ANN neurons. Additionally, Paiton et al.
Revised: (2019), who used it to understand invariance and selectivity of individual ANN neurons, and by Paiton et al.

Original:  ﬁrst established a relationship between individual neuron response curvature and adversarial robustness.
Revised: (2020), who ﬁrst found a relationship between individual neuron response curvature and adversarial robustness.

Original: Our work is distinct from the aforementioned studies in that we investigate the curvature of the decision boundary, as opposed to that of individual neurons.
Revised: Our work is distinct from the aforementioned studies in that we investigate the curvature of the decision boundary, as opposed to that of individual neurons.

Original: However, the relationship between decision boundary curvature and adversarial robustness has been previously studied Moosavi-Dezfooli and colleagues, who provided a number of compelling results identifying “an increasing [adversarial] vulnerability with respect to the curvature of the decision boundary” (MoosaviDezfooli et al., 2018)(for a complete account, see the thesis of Moosavi-Dezfooli, 2019).
Revised: However, the relationship between decision boundary curvature and adversarial robustness has been previously studied Moosavi-Dezfooli and colleagues, who provided a number of compelling results identifying “an increasing [adversarial] vulnerability with respect to the curvature of the decision boundary” (Moosavi-Dezfooli et al., 2018)(for a complete account, see the thesis of Moosavi-Dezfooli, 2019).

Original: We provide additional insight by investigating curvature along the set of orthogonal directions that are most close to dataset samples, as opposed to directions that have shared adversarial susceptibility across networks or samples. This allows us to show support for hypotheses that go beyond the correlation between decision boundary curvature and network robustness, in that we measure boundary curvature in the adversarial subspace dataset samples. 
Revised: As we will discuss in the next Section, our analysis provides support for hypotheses that go beyond the correlation between decision boundary curvature and network robustness, in that we measure boundary curvature in the adversarial subspace dataset samples. We additionally note that while previous work has utilized Riemannian curvature analysis to understand neural networks (viz. Fawzi et al., 2016; Poole et al., 2016; Moosavi-Dezfooli et al., 2018; 2019; Golden et al., 2019), each study measures diﬀerent quantities without complete details of the derived formulas.

Revised: We demonstrate in Appendix F that these formulas do not ﬁnd the correct level-set curvature of simple manifolds.

Original: Finally, in Appendix B, we provide a what we believe is the ﬁrst detailed derivation of  the analysis required to characterize ANN decision boundary curvature.
Revised: This together with our rigorous derivation and discussion in Appendix A constitutes, to the best of our knowledge, the ﬁrst principled and analytically supported measurement of decision boundary curvature of trained nonlinear neural networks.

Original:  Conclusion
Revised:  Conclusion

Original: We have characterized adversarial subspaces for neural networks to better understand the geometry of their decision surfaces.
Revised: There is an ongoing debate on the cause, prevalence, and uniformity of adversarial examples. Several studies have supported a hypothesis that most adversarial examples inhabit relatively dense regions of the input space (Goodfellow et al., 2014; Tabacof & Valle, 2016; MoosaviDezfooli et al., 2017a; He et al., 2018). However, other works have suggested that adversarial examples live in isolated pockets or thin “spikes” of incorrect classiﬁcation regions (Szegedy et al., 2014; Yu et al., 2019b).

Original: The decomposition gives a succinct quantitative description of the nonlinear geometry of the network decision boundary in the subspace spanned by its most susceptible perturbation directions.
Revised: Here we provide quantitative evaluation supporting the hypothesis that decision boundaries are close to input samples in a large subspace. Additional work has suggested that it is the linearity of deep neural networks that leads to adversarial examples (Goodfellow et al., 2014), and that adversarial training leads to a less linear decision boundary (Madry et al., 2018). However, more recent work has proposed the alternative: that increasing decision boundary linearity causes increased robustness (Li et al., 2019b; Moosavi-Dezfooli et al., 2019; Terjék, 2020; Sarkar & Iyengar, 2020).

Original: We argue that adversarial subspaces assist in explaining the prevalence of adversarial examples as well as the eﬀects of adversarial training. Understanding this space better can improve neural network performance with semi-supervised relabeling (Benato et al., 2018), architecture design (Rauber et al., 2018), and  adversarial robustness (Tramèr et al., 2017; Moosavi-Dezfooli et al., 2019).
Revised: Our curvature analysis conﬁrms that adversarial training linearizes the boundary, which is also congruent with hypotheses about the robustness beneﬁts of smoothing the decision surface (Zhang et al., 2019; Wu et al., 2020). We further compare the boundary curvature in adversarial subspaces and random subspaces to show an increase in curvature in adversarial subspaces for both training methods.

Original: Importantly, we also provide a quantitative and falsiﬁable interpretability approach for understanding ANN decisions (Ribeiro et al., 2016; Vlassopoulos et al., 2020), which is a necessary prerequisite for the technology to be applied to many industries (Doshi-Velez & Kim, 2017; Leavitt & Morcos, 2020). Author Contributions Acknowledgments
Revised: Since we do not report the causal relationship between adversarial directions and boundary curvature, our results have no impact on hypotheses for adversarial susceptibility due to overﬁtting (Tanay & Griﬃn, 2016) or the perceptual quality of adversarial features (Ilyas et al., 2019), although we see this as an interesting future direction of study.

Original: Here we show additional analysis to the above described ﬁndings on CIFAR-10 data. As was described earlier, we uniformly randomly sampled directions within adversarial subspaces to measure the distance to the decision boundary.
Revised: Adversarial subspaces allow for a quantitative description of the nonlinear geometry and class composition of the network decision boundary in the subspace spanned by its most susceptible perturbation directions. They also assist in explaining the prevalence of adversarial examples as well as the eﬀects of adversarial training.

Original: Figure 7 shows how many of the samples pointed to decision boundary points that were outside of the allowed input pixel bounds.

Original: 9 17 25 33 41dimension of adversarial sapce 0. o u t o f b o un d s r a t i o natural robust
Revised: Understanding this space better can improve neural network performance with semi-supervised relabeling (Benato et al., 2018), architecture design (Rauber et al., 2018), and adversarial robustness (Tramèr et al., 2017;

Original: Figure 6 shows ﬁve adversarially perturbed images and their corresponding original image found for the two investigated models. We can see that that the perturbation length is increasing with dimension number. Interestingly, the adversarially trained model seems to have more semantically meaningful adversarials, e.g. the images of boats in the top row, misclassiﬁed as cars, are perturbed to have elements looking like wheels. We can also see that there is no clear, explicit ‘ 2 -length of an adversarial attack that makes an image ambiguous for a human observer.
Revised: Moosavi-Dezfooli et al., 2019).

Original: Adversarialof the top row with perturbation length of 1.12 could be mistaken as a car on a ﬁrst glance. In contrast, adversarial 1 of the fourth row isstill clearly identiﬁable as a horse with a larger perturbation length of 1.2. Thus, we note that while talking about adversarial subspaces, there could occur semantically meaningful adversarials in these spaces at diﬀerent distances to a given image sample. Figure 8 shows that perturbation lengths are increasing sublinearly and monotonically with increasing number of adversarial dimensions.
Revised: Importantly, we also provide a quantitative and falsiﬁable interpretability approach for understanding ANN decisions (Ribeiro et al., 2016; Vlassopoulos et al., 2020), which is a necessary prerequisite for the technology to be applied to many industries (Doshi-Velez & Kim, 2017; Leavitt & Morcos, 2020).

Original: We can also see that for the naturally trained model there is a much larger increase ofadversarial dimension with increasing distance to distance to decision boundary. Note that both models plateau at 50 dimensions because the optimizer stopped when reaching this point. a d v e r s a r i a l v e c t o r l e n g t h natural robust   distance to decision boundary 
Revised: We have identiﬁed an interesting avenue for future work in performing a large-scale analysis of the adversarial subspaces of various neural network architectures, as well as estimating decision boundary geometry of non-smooth (e.g. ReLU or stochastic networks, Li et al., 2019a; Dapello et al., 2020) neural networks (which requires adding a costly Hessian approximation step).

Original: d i m e n s i o n o f a d v e r s a r i

Original: a l s p a c e [Figure] We also provide additional, and more detailed information on the distributions of distances to the decision boundary of randomly sampled vectors in adversarial subspaces in Figure 9. We observe that the variance of the distances to the decision boundary of randomly sampled vectors is smoothly increasing with dimensionality.
Revised: Author Contributions Acknowledgments

Original: B Boundary curvature analysis
Revised: A Boundary curvature analysis

Original: The following deﬁnitions are common.
Revised: The following deﬁnitions are common.

Original: Our notation follows Lee (2013; 2018) for diﬀerential geometry, and Magnus & Neudecker (1999) for linear algebra.
Revised: Our notation follows Lee (2013; 2018) for diﬀerential geometry, and Magnus & Neudecker (1999) for linear algebra.

Original: B. Notation and concepts
Revised: A. Introduction

Original: In order to derive our main curvature expressions for a graph submanifold, we ﬁrst introduce some notation and recall some concepts.
Revised: In order to derive our main curvature expressions for a graph submanifold, we ﬁrst introduce some notation and recall some concepts.

Original: Tangent Spaces
Revised: Tangent Spaces

Original: If M is an n -dimensional smooth manifold then the tangent space at p ∈ M is a vector space T p M of derivations at p , which is isometric to R n .
Revised: If M is an n -dimensional smooth manifold then the tangent space at p ∈ M is a vector space T p M of derivations at p , which is isometric to R n .

Original: In particular, T p M is the set of directional derivative operators acting on the smooth real functions f : M → R .
Revised: In particular, T p M is the set of directional derivative operators acting on the smooth real functions f : M → R .

Original: The disjoint union of all of the tangent spaces is the tangent bundle T M def = {{ p } × T p M | p ∈ M } .
Revised: The disjoint union of all of the tangent spaces is the tangent bundle T M def = {{ p } × T p M | p ∈ M } .

Original: If we equip M with a smooth, bilinear, positive deﬁnite, symmetric form g : T M × T M → R then ( M, g ) is said to be a Riemannian manifold and g the
Revised: If we equip M with a smooth, bilinear, positive deﬁnite, symmetric form g : T M × T M → R then ( M, g ) is said to be a Riemannian manifold and g the

Original: Riemannian metric tensor .
Revised: Riemannian metric tensor .

Original: When it is unambiguous, denote by Γ( E ) the smooth mappings M → E (for some topological space E ).
Revised: When it is unambiguous, denote by Γ( E ) the smooth mappings M → E (for some topological space E ).

Original: Most often we will refer to X ( M ) def = Γ(T M ), and represent elements of X ( M ) with the letters X, Y, Z .
Revised: Most often we will refer to X ( M ) def = Γ(T M ), and represent elements of X ( M ) with the letters X, Y, Z .

Original: Diﬀerentials If M and N are manifolds and F : M → N is diﬀerentiable, then the diﬀerential of F at p ∈ M is the linear mapping denoted d F p : T p M → T p N , where the action on tangent vectors is deﬁned by
Revised: Diﬀerentials If M and N are manifolds and F : M → N is diﬀerentiable, then the diﬀerential of F at p ∈ M is the linear mapping denoted d F p : T p M → T p N , where the action on tangent vectors is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: That is, the rate of change of the function f ◦ F in the direction v ∈ T p M at the point p .
Revised: That is, the rate of change of the function f ◦ F in the direction v ∈ T p M at the point p .

Original: As is common practice, we often omit the p subscripts for clarity.
Revised: As is common practice, we often omit the p subscripts for clarity.

Original: When N in the above deﬁnition is R the linear operator d f is a linear functional, andso at each p∈
Revised: The linear operator d f is a linear functional when N in the above deﬁnition is R , and at each p

Original:  M there exists a vector grad f p ∈ T p .
Revised: ∈ M there exists a vector grad f p ∈ T p .

Original: So that
Revised: So that

Original: [Equation]
Revised: [Equation]

Original: ∀ v ∈ T p M : d f p 
Revised: T p M : d f p ( v ) =

Original: [Equation]

Original: v ) = g [Equation] grad f p , v ) , and grad f ∈ X ( M ) is the associated vector ﬁeld.
Revised: g (grad f p , v ) , and grad f ∈ X ( M ) is the associated vector ﬁeld.

Original: Connections A connection is an operator ∆ : X ( M ) × X ( M ) → X ( M ), linear in its ﬁrst argument, and satisfying a product rule in its second.
Revised: Connections A connection is an operator ∆ : X ( M ) × X ( M ) → X ( M ), linear in its ﬁrst argument, and satisfying a product rule in its second.

Original: Surprisingly, a connection uniquely deﬁnes a connection in each tensor bundle, 2 (which we overload with ∆ ) (Lee, 2018,
Revised: Surprisingly, a connection uniquely deﬁnes a connection in each tensor bundle, 3 (which we overload with ∆ ) (Lee, 2018,

Original: Prop. 4.15).
Revised: Prop. 4.15).

Original: Importantly for us, we have ∆ X f = Xf for all X ∈ X ( M ) (for any connection).
Revised: Importantly for us, we have ∆ X f = Xf for all X ∈ X ( M ) (for any connection).

Original: A Riemannian manifold always posesses a Levi–Cevita connection (Lee, 2018, §5), denoted by ∇ , which is the unique connection satisfying
Revised: A Riemannian manifold always posesses a Levi–Cevita connection (Lee, 2018, §5), denoted by ∇ , which is the unique connection satisfying

Original: [Equation]
Revised: [Equation]

Original: To f ∈ C 2 ( M ) we associate the Hessian operator ∇ 2 f , which is a symmetric (0 , 2)-tensor ﬁeld (Lee, 2018, Ex. 4.22) with ∇ 2 f
Revised: To f ∈ C 2 ( M ) we associate the Hessian operator ∇ 2 f , which is a symmetric (0 , 2)-tensor ﬁeld (Lee, 2018, Ex. 4.22) with ∇ 2 f

Original: ( X, Y ) def = ∇ X d f ( Y ) for X, Y ∈ X ( M ).
Revised: ( X, Y ) def = ∇ X d f ( Y ) for X, Y ∈ X ( M ).

Original: We have (via
Revised: We have (via

Original: Lee, 2018, Prop. 4.21)
Revised: Lee, 2018, Prop. 4.21)

Original: [Equation]
Revised: [Equation]

Original: Using the compatibility of ∇ with the metric we arrive at the following expression for the
Revised: Using the compatibility of ∇ with the metric we arrive at the following expression for the

Original: Hessian, which will be useful later
Revised: Hessian, which will be useful later

Original: [Equation]
Revised: [Equation]

Original: B.1. Submanifolds
Revised: A.1. Submanifolds

Original: If ( M, g ), ( ˜ M, ˜ g ) are two Riemanian manifolds with M ⊆ ˜ M , then an injection ι : M , → ˜ M is called an inclusion mapping .
Revised: If ( M, g ), ( ˜ M, ˜ g ) are two Riemanian manifolds with M ⊆ ˜ M , then an injection ι : M , → ˜ M is called an inclusion mapping .

Original: If the inclusion is a smooth isometry, that is
Revised: If the inclusion is a smooth isometry, that is

Original: [Equation]
Revised: [Equation]

Original: equivalently, g = ι ∗ ˜ g , then M is called a Riemanian submanifold of ˜ M .
Revised: equivalently, g = ι ∗ ˜ g , then M is called a Riemanian submanifold of ˜ M .

Original: Using the inclusion map diﬀerential, we embed T M in T ˜ M .
Revised: Using the inclusion map diﬀerential, we embed T M in T ˜ M .

Original: If M , ˜ M are of dimensions m < n , then T p M occupies an m dimensional subspace of T p ˜ M for each p ∈ M .
Revised: If M , ˜ M are of dimensions m < n , then T p M occupies an m dimensional subspace of T p ˜ M for each p ∈ M .

Original: The normal bundle  formed from the orthogonal compliment of T p M at each p ∈ M , and is deﬁned in a similar way to the tangent bundle, denoted N M and having a dimension of n − m .
Revised: The normal bundle is formed from the orthogonal compliment of T p M at each p ∈ M , and is deﬁned in a similar way to the tangent bundle, denoted N M and having a dimension of n − m .

Original: For vectors v ∈ T
Revised: For vectors v ∈ T

Original: ˜ M
Revised: ˜ M

Original: we have the decomposition v = v ⊥ + v ⊥ with v ⊥ ∈ T M and v ⊥ ∈ N M .
Revised: we have the decomposition v = v ⊥ + v ⊥ with v ⊥ ∈ T M and v ⊥ ∈ N M .

Original: Some care must be taken when taking covariant derivatives in T M with the ambient connection, formally
Revised: Some care must be taken when taking covariant derivatives in T M with the ambient connection, formally

Original: [Equation]
Revised: [Equation]

Original: where the overlines denote arbitrary smooth extensions of d ι ( X ) and d ι ( Y ) to open neighbourhoods of M in ˜ M .
Revised: where the overlines denote arbitrary smooth extensions of d ι ( X ) and d ι ( Y ) to open neighbourhoods of M in ˜ M .

Original: Second Fundamental Form Let M ⊆
Revised: Second Fundamental Form Let M ⊆

Original: ˜
Revised: ˜

Original: M be an embedded sumbanifold.
Revised: M be an embedded sumbanifold.

Original: The second fundamental form is the (bilinear) mapping II : X ( M ) × X
Revised: The second fundamental form is the (bilinear) mapping II : X ( M ) × X

Original: ( M ) → Γ(N M ) with II( X, Y ) def = (cid:0)
Revised: ( M ) → Γ(N M ) with II( X, Y ) def = (cid:0)

Original: ˜ ∇ X Y (cid:1) ⊥ .
Revised: ˜ ∇ X Y (cid:1) ⊥ .

Original: If we equip M with a normal ﬁeld N ∈ Γ(N M ), then the scalar second fundamental form is
Revised: If we equip M with a normal ﬁeld N ∈ Γ(N M ), then the scalar second fundamental form is

Original: [Equation]
Revised: [Equation]

Original: For the remainder we assume N is ﬁxed.
Revised: For the remainder we assume N is ﬁxed.

Original: The shape operator s : X ( M ) → X ( M ) is the endomorphism obtained from II N that satisﬁfes
Revised: The shape operator s : X ( M ) → X ( M ) is the endomorphism obtained from II N that satisﬁfes

Original: [Equation]
Revised: [Equation]

Original:  that is, by raising an index.
Revised: 6) that is, by raising an index.

Original: Curvature Let s be the shape operator associated to a normal ﬁeld on an embedded submanifold M of dimension n .
Revised: Curvature Let s be the shape operator associated to a normal ﬁeld on an embedded submanifold M of dimension n .

Original: Then at p ∈ M , the (real) eigenvalues of s are called principal curvatures and the eigenvectors are called principal directions at p .
Revised: Then at p ∈ M , the (real) eigenvalues of s are called principal curvatures and the eigenvectors are called principal directions at p .

Original: The Gaussian curvature is the product of the principal curvatures, det( s ), and the mean curvature is their average, 1 n tr( s ).
Revised: The Gaussian curvature is the product of the principal curvatures, det( s ), and the mean curvature is their average, 1 n tr( s ).

Original: Assume ( M, g ) and ( ˜ M, ˜ g ) are Riemannian manifolds with an inclusion mapping ι : M , → ˜ M and M has the submanifold structure.
Revised: Assume ( M, g ) and ( ˜ M, ˜ g ) are Riemannian manifolds with an inclusion mapping ι : M , → ˜ M and M has the submanifold structure.

Original: Suppose ˜ M posesses a shape operator (or any other linear endomorphism)
Revised: Suppose ˜ M posesses a shape operator (or any other linear endomorphism)

Original: ˜ s : T ˜ M → T ˜ M .
Revised: ˜ s : T ˜ M → T ˜ M .

Original: The submanifold structure then gives us an operator ι ∗ ˜ s : T M → T M , the pullback of s by ι , via
Revised: The submanifold structure then gives us an operator ι ∗ ˜ s : T M → T M , the pullback of s by ι , via

Original: [Equation]
Revised: [Equation]

Original: Remark 1 (Euclidean subspace) .
Revised: Remark 1 (Euclidean subspace) .

Original: We consider the special case of the Riemannian manifold
Revised: We consider the special case of the Riemannian manifold

Original: R m .
Revised: R m .

Original: Suppose P : R n → R m is orthogonal ( n ≤ m ) and let s : R m → R m be a linear operator.
Revised: Suppose P : R n → R m is orthogonal ( n ≤ m ) and let s : R m → R m be a linear operator.

Original: Then P T : R n , → R m is an inclusion map and the subspace pullback of s to R n is
Revised: Then P T : R n , → R m is an inclusion map and the subspace pullback of s to R n is

Original: [Equation]
Revised: [Equation]

Original:  PsP T 
Revised: = PsP T .

Original: We refer to the curvature quantities and directions of (cid:0) P T (cid:1) ∗ s with the suspace preﬁx.
Revised: We refer to the curvature quantities and directions of (cid:0) P T (cid:1) ∗ s with the suspace preﬁx.

Original: That is, subspace principal curvature , subspace Gaussian curvature , and so on.
Revised: That is, subspace principal curvature , subspace Gaussian curvature , and so on.

Original: B. Hypersurfaces
Revised: A. Hypersurfaces

Original: Let ( M, g ) be an embedded submanifold of ( ˜ M, ˜ g ).
Revised: Let ( M, g ) be an embedded submanifold of ( ˜ M, ˜ g ).

Original: Suppose F : U ⊆
Revised: Suppose F : U ⊆

Original: ˜
Revised: ˜

Original: M
Revised: M

Original: → R is a local deﬁning function for M , that is, M ∩ U = F − 1 ( { 0 } ) for an open U ⊆ M .
Revised: → R is a local deﬁning function for M , that is, M ∩ U = F − 1 ( { 0 } ) for an open U ⊆ M .

Original: Choose arbitrary
Revised: Choose arbitrary

Original: X, Y ∈ X ( M ).
Revised: X, Y ∈ X ( M ).

Original: From the connection product rule (Lee, 2018, p. 89)
Revised: From the connection product rule (Lee, 2018, p. 89)

Original: [Equation]
Revised: [Equation]

Original: Whence
Revised: Whence

Original: [Equation]
Revised: [Equation]

Original: The zero in the underbrace follows because grad F ∈ N M and Y ∈ T M .
Revised: The zero in the underbrace follows because grad F ∈ N M and Y ∈ T M .

Original: The Weingarten equation (Lee, 2018, Thm. 8.13(c)) yields sX = − ˜ ∇ X N , giving us
Revised: The Weingarten equation (Lee, 2018, Thm. 8.13(c)) yields sX = − ˜ ∇ X N , giving us

Original: [Equation]
Revised: [Equation]

Original: B.2. The Curvature of a Graph Manifold
Revised: A.2. The Curvature of a Graph Manifold

Original: The notation and conventions of this section closely follow (Lee, 2018, §8).
Revised: The notation and conventions of this section closely follow (Lee, 2018, §8).

Original: Let M be the graph of a smooth function f : R n → R , that is, M = { ( x, f ( x )) | x ∈ R n } .
Revised: Let M be the graph of a smooth function f : R n → R , that is, M = { ( x, f ( x )) | x ∈ R n } .

Original: Since f is smooth
Revised: Since f is smooth

Original: M is indeed a manifold (Lee, 2013, Ex. 1.30, p. 20).
Revised: M is indeed a manifold (Lee, 2013, Ex. 1.30, p. 20).

Original: We regard ( M, g ) as a Riemannian submanifold of ( R n +1 , h· , ·i ), with the inclusion map ι : M , → R n +1 .
Revised: We regard ( M, g ) as a Riemannian submanifold of ( R n +1 , h· , ·i ), with the inclusion map ι : M , → R n +1 .

Original: The function F : R n +

Revised: The function F : R n +

Original: → R is a global deﬁning function for M with F ( x, t ) def = f ( x ) − t .
Revised: → R is a global deﬁning function for M with F ( x, t ) def = f ( x ) − t .

Original: Denote by ∇ f the vector of partial derivatives of f , and by H f and H F , the matrices of second order partial derivatives of F and f , all with respect to usual coordinate frames, ( x i ), on R n and R n +1 .
Revised: Denote by ∇ f the vector of partial derivatives of f , and by H f and H F , the matrices of second order partial derivatives of F and f , all with respect to usual coordinate frames, ( x i ), on R n and R n +1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: At p ∈ M the diﬀerential d ι p has the Jacobian matrix
Revised: At p ∈ M the diﬀerential d ι p has the Jacobian matrix

Original: [Equation]
Revised: [Equation]

Original: We commit the mild sin of identifying d ι p with its Jacobian and X, Y ∈ T ˜ M with their parameterizations under the coordinate frame, so that we may use the notation of matrix multiplication.
Revised: We commit the mild sin of identifying d ι p with its Jacobian and X, Y ∈ T ˜ M with their parameterizations under the coordinate frame, so that we may use the notation of matrix multiplication.

Original: In dropping the p -subscripts, the left inverse, (d ι ) − 1 , is has the parameterization
Revised: In dropping the p -subscripts, the left inverse, (d ι ) − 1 , is has the parameterization

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: whence
Revised: whence

Original: [Equation]
Revised: [Equation]

Original: We can now compute the shape operator of M associated to the unit normal ﬁeld grad F/ | grad F | .
Revised: We can now compute the shape operator of M associated to the unit normal ﬁeld grad F/ | grad F | .

Original: Pick arbitrary X, Y ∈ T M .
Revised: Pick arbitrary X, Y ∈ T M .

Original: From (8) there is
Revised: From (8) there is

Original: [Equation]
Revised: [Equation]

Original: and, using the bilinearity of of g we have
Revised: and, using the bilinearity of of g we have

Original: [Equation]
Revised: [Equation]

Original: This, together with (9) yields the shape operator:
Revised: This, together with (9) yields the shape operator:

Original: [Equation]
Revised: [Equation]

Original: Whence the Gaussian
Revised: Whence the Gaussian

Original: and mean curvature are the scalar ﬁelds
Revised: and mean curvature are the scalar ﬁelds

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: B.2. The Curvature of a Level-Set Manifold
Revised: A.2. The Curvature of a Level-Set Manifold

Original: We now consider the case where M is the level set of a smooth function f : R n → R so that for some c ∈ R we have M = { x ∈ R n | f ( x )
Revised: We now consider the case where M is the level-set of a smooth function f : R n → R so that for some c ∈ R we have M = { x ∈ R n | f ( x )

Original: = c } .
Revised: = c } .

Original: The following theorem can be used to ensure the level sets of f do indeed posses a manifold structure, which is a simpliﬁcation of the constant rank level-set theorem (Lee, 2013, Thm. 5.12) for our setting.
Revised: The following theorem can be used to ensure the level-sets of f do indeed posses a manifold structure, which is a simpliﬁcation of the constant rank level-set theorem (Lee, 2013, Thm. 5.12) for our setting.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f : R n → R be a smooth function having constant rank r .
Revised: Let f : R n → R be a smooth function having constant rank r .

Original: Then the level sets of f are all smooth manifolds of codimension r .
Revised: Then the level-sets of f are all smooth manifolds of codimension r .

Original: We use the same construction as Lee (2013, Ex. 1.32, §5) to calculate the Gaussian and mean curvature at a point ( x 0 , y 0 ) ∈
Revised: We use the same construction as Lee (2013, Ex. 1.32, §5) to calculate the Gaussian and mean curvature at a point ( x 0 , y 0 ) ∈

Original: ( R n − 1 × R ) ∩ M .
Revised: ( R n − 1 × R ) ∩ M .

Original: From the implicit function theorem (Lee, 2013, Thm.
Revised: From the implicit function theorem (Lee, 2013, Thm.

Original: C.40) there are open connected neighbourhoods U 

Revised: C.40) there are open connected neighbourhoods U

Original: × V 0 ⊆
Revised: × V 0 ⊆

Original: R n − 1 × R with ( x 0 , y 0 ) ∈
Revised: R n − 1 × R with ( x 0 , y 0 ) ∈

Original: U 0 × V 0 and a smooth function g : U 0 → Vso that ( U 0 × V 0 )
Revised: U 0 × V 0 and a smooth function g : U 0 → Vso that ( U 0 × V 0 )

Original: ∩ M is the graph of g .
Revised: ∩ M is the graph of g .

Original: That is, ( U 0 × V 0 )
Revised: That is, ( U 0 × V 0 )

Original: ∩ M = (cid:8) ( x, g ( x )) (cid:12)(cid:12) x ∈ U 0 ⊆
Revised: ∩ M = (cid:8) ( x, g ( x )) (cid:12)(cid:12) x ∈ U 0 ⊆

Original: R n − 1 (cid:9) .
Revised: R n − 1 (cid:9) .

Original: Consequentially the gradient of g has the structure
Revised: Consequentially the gradient of g has the structure

Original: [Equation]
Revised: [Equation]

Original: Let ∂ i def = ∂ / ∂ x j and ∂ 2 ij def = ∂ 2 / ∂ x i ∂ x j .
Revised: Let ∂ i def = ∂ / ∂ x j and ∂ 2 ij def = ∂ 2 / ∂ x i ∂ x j .

Original: Then for i, j ∈ [ n − 1] and x ∈ U 
Revised: Then for i, j ∈ [ n − 1] and x ∈ U [Equation] i

Original: [Equation]
Revised: [Equation]

Original:  (14) and
Revised: , (14) and

Original: [Equation]
Revised: [Equation]

Original: (15) where every function on the right hand side is evaluated at either the point x or ( x, g ( x ))
Revised: (15) where every function on the right hand side is evaluated at either the point x or ( x, g ( x ))

Original: (depending on the function).
Revised: (depending on the function).

Original: 4 If we let ¯ ∇ def = ( ∂ / ∂ x 1 , . . . , ∂ / ∂ x n − 1 ) T , and ¯H be, similarly, the ( n − 1) × ( n − 1) matrix of cross-partial derivatives, there are the more compact expressions for (14) and (15):
Revised: 5 If we let ¯ ∇ def = ( ∂ / ∂ x 1 , . . . , ∂ / ∂ x n − 1 ) T , and ¯H be, similarly, the ( n − 1) × ( n − 1) matrix of cross-partial derivatives, there are the more compact expressions for (14) and (15):

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: .0% / 0.0% / 0.0% / 0.0% / 0.0% / 36.33% 54.5% 59.04% 61.76% 65.4% 69.03% 69.94% 70.85% 72.66% 73.57% 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 2.72% 5.45% 10.9% 12.72% 13.62% 14.53% 15.44% 15.44% 16.35% 17.26% and
Revised: .0% / 0.0% / 0.0% / 0.0% / 0.0% / 36.33% 54.5% 59.04% 61.76% 65.4% 69.03% 69.94% 70.85% 72.66% 73.57% 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 2.72% 5.45% 10.9% 12.72% 13.62% 14.53% 15.44% 15.44% 16.35% 17.26% and

Original: [Equation]
Revised: [Equation]

Original: where the sum ∇ g + ∇ g T is to be interpreted as the matrix of pairwise sums of the elements of ∇ g : (cid:0) ∇ g + ∇ g T (cid:1) ij def = ∂ g/ ∂ x i + ∂ g/ ∂ x j .
Revised: where the sum ∇ g + ∇ g T is to be interpreted as the matrix of pairwise sums of the elements of ∇ g : (cid:0) ∇ g + ∇ g T (cid:1) ij def = ∂ g/ ∂ x i + ∂ g/ ∂ x j .

Original: Then, as in Section B.2.1, F ( x ) def = g ( x 1 , . . .
Revised: Then, as in Section A.2.1, F ( x ) def = g ( x 1 , . . .

Original: , x n − 1 ) − x n is a local deﬁning function for M on ( U 0 × V 0 )
Revised: , x n − 1 ) − x n is a local deﬁning function for M on ( U 0 × V 0 )

Original: ∩ M .
Revised: ∩ M .

Original: The local graph parameterization provided by g is φ : U 

Revised: The local graph parameterization provided by g is φ : U

Original: → M with φ ( x ) def = ( x, g ( x )), and its diﬀerential has the parameterization d 
Revised: → M with φ ( x ) def = ( x, g ( x )), and its diﬀerential has the parameterization d φ

Original: [Equation]
Revised: = (Id ∇ g ) T .

Original: Finally, mutatis mutandis , the shape operator is given by (11) and the Gaussian and mean curvature can be computed as in (12).
Revised: Finally, mutatis mutandis , the shape operator is given by (11) and the Gaussian and mean curvature can be computed as in (12).

Original: Remark 2 (Numerical considerations) .
Revised: Remark 2 (Numerical considerations) .

Original: In order to compute the level-set curvature we needed to apply the implicit function theorem.
Revised: In order to compute the level-set curvature we needed to apply the implicit function theorem.

Original: However, it may be diﬃcult numerically to stabalise the inverse operation in (13).
Revised: However, it may be diﬃcult numerically to stabilize the inverse operation in (13).

Original: To remedy this problem, one may apply a change of basis before computing the quantities (14) and (15).
Revised: To remedy this problem, one may apply a change of basis before computing the quantities (14) and (15).

Original: Let B ∈ R n × n be an orthogonal change of basis operator.
Revised: Let B ∈ R n × n be an orthogonal change of basis operator.

Original: Then we apply all our operations to B T M def = (cid:8) B T x (cid:12)(cid:12) x ∈ M (cid:9) .
Revised: Then we apply all our operations to B T M def = (cid:8) B T x (cid:12)(cid:12) x ∈ M (cid:9) .

Original: Diﬀerentiating ˜ f
Revised: Diﬀerentiating ˜ f

Original: def = f ◦ B T yields the revised quantities ∀ x ∈ R n : ∇ ˜ f
Revised: def = f ◦ B T yields the revised quantities ∀ x ∈ R n : ∇ ˜ f

Original: ( x ) = B · ∇ f ( B T x ) and
Revised: ( x ) = B · ∇ f ( B T x ) and

Original: [Equation]
Revised: [Equation]

Original: C Dataset and models
Revised: B Dataset and models

Original: For the adversarial subspace analysis we use two identical ResNet-50 (He et al., 2016) architectures.
Revised: For the adversarial subspace analysis we use two identical ResNet-50 (He et al., 2016) architectures.

Original: The ﬁrst model was trained on the unmodiﬁed CIFAR-10 dataset (Krizhevsky & Hinton, 2009).
Revised: The ﬁrst model was trained on the unmodiﬁed CIFAR-10 dataset (Krizhevsky & Hinton, 2009).

Original: For the second model we additionally incorporated adversarial examples in the training process, widely known as adversarial training.
Revised: For the second model we additionally incorporated adversarial examples in the training process, widely known as adversarial training.

Original: A training perturbation size (cid:15) of 0.5 was used.
Revised: A training perturbation size (cid:15) of 0.5 was used.

Original: To perform the curvature analysis, we used twice-diﬀerentiable ELU activation functions (Clevert et al., 2015) for both models, which had no impact on the clean or adversarial accuracies.
Revised: To perform the curvature analysis, we used twice-diﬀerentiable ELU activation functions (Clevert et al., 2015) for both models, which had little impact on the clean or adversarial accuracies.

Original: In the analysis we include 100 CIFAR-10 test images, class-balanced, all unseen during training, and correctly classiﬁed by both models.
Revised: In the analysis we include 100 CIFAR-10 test images, class-balanced, all unseen during training, and correctly classiﬁed by both models.

Original: The optimization algorithm stopped after ﬁnding 50 orthogonal adversarial vectors.
Revised: The optimization algorithm stopped after ﬁnding 50 orthogonal adversarial vectors.

Original: For all test image we found 50 dimensions on both models.
Revised: For all test image we found 50 dimensions on both models.

Original: For the curvature analysis we used a subset of 20 class-balanced images andadversarial directions per image.
Revised: For the curvature analysis we used a subset of 50 class-balanced images andadversarial directions per image.

Original: Additionally, we only considered one seed per CIFAR-10 model because we found that the analysis on the MNIST trained models had little variation across seeds (see Appendix D). 
Revised: Additionally, we only considered one seed per CIFAR-10 model because we found that the analysis on the MNIST trained models had little variation across seeds (see Appendix D). [Figure]

Original: The accuracy of both models at diﬀerent pertubation lengths (cid:15) and dimensions is shown in
Revised: The accuracy of both models at diﬀerent perturbation lengths (cid:15) and dimensions is shown in

Original: Table 2.
Revised: Table 2.

Original: Here, the ﬁrst line of each row corresponds to the accuracy of the naturally trained model, whereas the second line is referring to the adversarially trained model.
Revised: Here, the ﬁrst line of each row corresponds to the accuracy of the naturally trained model, whereas the second line is referring to the adversarially trained model.

Original: We repeated many of the experiments on the MNIST dataset (LeCun et al., 1998) and found largely consistent results (see Appendix F).
Revised: We repeated many of the experiments on the MNIST dataset (LeCun et al., 1998) and found largely consistent results (see Appendix G). C Additional CIFAR analysis and supplementary figures [Figure]

Revised: Here we show additional analysis to complement the above described ﬁndings on CIFAR-10 data.

Original: D Seed consistency
Revised: As was described earlier, we uniformly randomly sampled directions within adversarial subspaces to measure the distance to the decision boundary. Figure 7 shows how many of the samples pointed to decision boundaries that were outside of the allowed input pixel bounds. Figure 6 shows ﬁve adversarially perturbed images and their corresponding original image found for the two investigated models. We can see that that the perturbation length is increasing with dimension number.

Revised: Interestingly, the adversarially trained model seems to have more semantically meaningful adversarials, e.g. the images of boats in the top row, misclassiﬁed as cars, are perturbed to have elements looking like wheels.

Revised: We can also see that there is no clear, explicit ‘ 2 -length of an adversarial attack that makes an image ambiguous for a human observer.

Revised: For example, the robust model example in the ﬁrst (ship) row of the adversarial 11 column pair has a

Revised: a d v e r s a r i a l v e c t

Revised: o r l e n g t h natural robust   distance to decision boundary

Revised: d i m e n s i o n o f a d v e r s a r i

Revised: a l s p a c e perturbation length of 1.12 and could be mistaken as a car on a ﬁrst glance.

Revised: In contrast, the robust model example in the fourth (horse) row of the adversarial 1 column pair is still clearly identiﬁable as a horse with a larger perturbation length of 1.2.

Revised: This illustrates that the direction of the perturbation is as important as the perturbation magnitude itself, as was previously reported by (Goodfellow et al., 2014; Ilyas et al., 2019), although assessing the “perceptual distance” is an open area of research.

Revised: Figure 8 shows that perturbation lengths are increasing sublinearly and monotonically with increasing number of adversarial dimensions.

Revised: For the naturally trained model there is also a much larger increase of adversarial dimension with increasing distance to distance to decision boundary.

Revised: Note that in the right panel of this ﬁgure both models plateau at 50 dimensions because we stopped the optimizer at this point.

Revised: We also provide additional, and more detailed information on the distributions of distances to the decision boundary of randomly sampled vectors in adversarial subspaces in Figure 9.

Revised: We observe that the variance of the distances to the decision boundary of randomly sampled vectors is smoothly increasing with dimensionality.

Original: We trained  natural and robust MNIST models on ﬁve diﬀerent seeds and extracted adversarial subspaces for each model to examine the inﬂuence of training seeds on our results.
Revised: We trained the natural and robust MNIST models with ﬁve diﬀerent seeds and extracted adversarial subspaces for each model to examine the inﬂuence of training seeds on our results.

Original: Figureshows that there is slight variation in number of adversarial subspace directions found by the optimizer.
Revised: Figure 10 shows that there is slight variation in number of adversarial subspace directions found by the optimizer.

Original: However, the overall course of bar heights is similar across seeds and there are no signiﬁcant outliers.
Revised: However, the overall trend of bar heights is similar across seeds and there are no signiﬁcant outliers.

Original: [Figure]
Revised: To further validate our results, we repeated the experiment for Figure 14 (measuring the length of the adversarial vector as one increases dimensionality on the MNIST trained networks) across ﬁve diﬀerent seeds per training type.

Original: [Figure]
Revised: The results are shown in ﬁgure

Original: seed
Revised: The variation across seeds for a single image is smaller than the variation across images for a single seed, which can be seen by comparing Figure 11 to Figure 14. In general, we also found that the result trends were consistent between MNIST and CIFAR networks (with diﬀerences noted in Appendix G).

Original: Thus, we considered one seed per CIFAR-10 model as suﬃciently representative. 
Revised: Thus, we considered one seed per CIFAR-10 model as suﬃciently representative. [Figure] [Figure]

Revised: seed         found dimensions

Revised: Robust models seed

Revised: [Figure]

Original: E Orthogonality and clipping
Revised: E Orthogonality and clipping

Original: One obstacle in ﬁnding proper adversarial subspaces spanned by orthogonal vectors is assuring that perturbations do not cause pixel values to be out of the valid input boundaries [0 , 1].
Revised: One obstacle in ﬁnding proper adversarial subspaces spanned by orthogonal vectors is assuring that perturbations do not cause pixel values to be out of the valid input boundaries [0 , 1].

Original: We noticed that many studies solve this problem by element wise pixel clipping.
Revised: We noticed that many studies solve this problem by element wise pixel clipping.

Original: Here, we argue that pixel clipping is problematic when talking about the dimensionality of subspaces.
Revised: Here, we show that pixel clipping is problematic when talking about the dimensionality of subspaces.

Original: Element wise pixel clipping is a non-linear operation performed on a vector and can therefore change its direction in the input space and its relative angle to other, ﬁxed vectors.
Revised: Element wise pixel clipping is a non-linear operation performed on a vector and can therefore change its direction in the input space and its relative angle to other, ﬁxed vectors.

Original: To show that clipping has a non-negligible eﬀect, we investigated relative angles of basis vectors that span (sub-) spaces for 100 MNIST test images relevant for two studies.
Revised: To show that clipping has a non-negligible eﬀect, we investigated relative angles of basis vectors that span (sub-) spaces for 100 MNIST test images relevant for two studies.

Original: First, in their work, Tramèr et al.
Revised: First, in their work, Tramèr et al.

Original: (2017) propose the GAAS method.
Revised: (2017) propose the GAAS method.

Original: GAAS estimates the dimensionality of adversarial subspaces by deﬁning an orthogonal set of vectors aligned with an attack vector to an input image.
Revised: GAAS estimates the dimensionality of adversarial subspaces by deﬁning an orthogonal set of vectors aligned with an attack vector to an input image.

Original: However, before passing the orthogonal basis vectors to the model, a clipping operation is performed.
Revised: However, before passing the orthogonal basis vectors to the model, a clipping operation is performed.

Original: Figure 11 (left) shows that this clipping operation causes the input vectors to be non-orthogonal - orthogonality would constitute an angle of 90° or π 2 .
Revised: Figure 12 (left) shows that this clipping operation causes the input vectors to be non-orthogonal - orthogonality would constitute an angle of 90° or π 2 .

Original: In fact, for the 100 sample images we tested, there is not a single vector pair that is completely orthogonal.
Revised: In fact, for the 100 sample images we tested, there is not a single vector pair that is completely orthogonal.

Original: The author’s conclusions regarding the dimensionality of adversarial subspaces and transferable subspaces are therefore grounded on ill-deﬁned subspaces, spanned by non-orthogonal vectors.
Revised: The author’s conclusions regarding the dimensionality of adversarial subspaces and transferable subspaces are therefore grounded on ill-deﬁned subspaces, spanned by non-orthogonal vectors.

Original: In another study, He et al.
Revised: In another study, He et al.

Original: (2018) estimate the distance to the decision boundary for the complete input space of MNIST.
Revised: (2018) estimate the distance to the decision boundary for the complete input space of MNIST.

Original: For that, a basis of 784 vectors, spanning the whole input space, is randomly generated.
Revised: For that, a basis of 784 vectors, spanning the whole input space, is randomly generated.

Original: Again, the inputs are clipped before they are passed to the model leading to the same problem as described earlier.
Revised: Again, the inputs are clipped before they are passed to the model leading to the same problem as described earlier.

Original: And as before, there are no orthogonal vector pairs, and angels are accumulated at much smaller values than π 2 as can be seen in Figure 11 (right).
Revised: And as before, there are no orthogonal vector pairs, and angels are accumulated at much smaller values than π 2 as can be seen in Figure 12 (right).

Original: We therefore note that it is not guaranteed that the input space is evenly covered and that edges of the 784-dimensional hypercube (in the case of MNIST) are arguably overrepresented.
Revised: We therefore note that it is not guaranteed that the input space is evenly covered and that edges of the 784-dimensional hypercube (in the case of MNIST) are arguably overrepresented.

Original: [Figure]
Revised: [Figure]

Revised: F Additional related work on decision boundary curvature

Revised: Several works from Moosavi-Dezfooli and colleagues explore decision boundary curvature and its relationship to adversarial robustness (Fawzi et al., 2016; Moosavi-Dezfooli et al., 2017b; 2018; 2019; Moosavi-Dezfooli, 2019).

Revised: Within these works they provide an inspired perspective on explaining adversarial robustness in terms of the curvature of decision boundaries.

Original: Tramèr et al. method
Revised: Almost all of the work is with respect to universal adversarial perturbations, which share a common subspace (Moosavi-Dezfooli et al., 2017a), although Fawzi et al. (2016) alternatively look at random subspaces. The universal perturbation subspaces diﬀer from our adversarial subspaces in that theirs are deﬁned by orthogonal directions that transfer well across inputs (in terms of proximity to the decision boundary) instead of those that point to the nearest decision boundary for a given input. Despite these diﬀerences, the stated quantity of interest, that is, the principle curvatures at a point on the decision boundary is the same in their work as well as ours.

Revised: However, each of these papers have a slightly diﬀerent analytic formula for estimating the curvature of the boundary within a given subspace (Fawzi et al., 2016; Moosavi-Dezfooli et al., 2017b), or in the high-dimensional input space (Moosavi-Dezfooli et al., 2018; 2019).

Revised: We found the most complete description to be given in Moosavi-Dezfooli (2019), which summarizes a majority of the earlier work and applies the decision boundary curvature analysis to both subspaces and the high-dimensional input space.

Revised: Therein, Moosavi-Dezfooli (2019, Section. 7.4) state the following formula, without derivation, for the shape operator (in the notation of Section 2.2):

Original: [Figure]
Revised: [Equation]

Revised: where H F ij is the Hessian of the boundary function, F ij ( x ) def = f i ( x ) − f j ( x ), P def =

Revised: Id −∇ F ij ( x ) ∇ F ij ( x ) T is described as a projection operator.

Original: He et al. method
Revised: This formulation is similar to that used by (Poole et al., 2016), with the only diﬀerence being that they use normalized gradients in the projection operator. Our formulation diﬀers from the above in several ways, the most important of which is that they claim to transform the Hessian by projecting it onto the tangent space (as written above), while we apply the implicit function theorem to parameterize the decision boundary as the graph of a function (see Appendix Section A.2.2, Equation 13). For computing the curvature of the decision boundary, exclusively projecting the Hessian is not going to result in [Table]

Revised: the correct calculations, since for the decision boundary (as deﬁned in Section 2.2, Equation 2) you need a diﬀerent normal than for the graph of the response function.

Revised: More speciﬁcally, the normal used for computing the shape operator of the graph surface is diﬀerent from that for computing the shape operator of the decision boundary.

Revised: The latter has to be exclusively in the pixel space, while the former always has a component in the direction of the function response.

Revised: Projecting the Hessian can restrict the computed curvatures in a certain subspace of directions, but it cannot change the computed curvatures in said subspace.

Revised: Another important diﬀerence is their projection operator is computed directly from the gradient

Revised: [Table]

Revised: and thus has a dimensionality equal to the input dimensionality.

Revised: To calculate the curvature of a decision boundary, one needs a level-set formulation, which should have one fewer dimensions than the input space.

Revised: The consequence of this diﬀerence can be made clear when looking at the curvature of an n -dimensional sphere, as deﬁned by the quadratic function:

Revised: [Equation]

Revised: where x is the input point vector and H is the Hessian (i.e. an identity matrix in R n × n ).

Revised: Given this function, the principal curvatures of the level-set at some radius, r , are all known to be (up to a sign change) 1 r .

Revised: However, we show in table 3 that the formulations provided by

Revised: Moosavi-Dezfooli (2019) and Poole et al.

Revised: (2016) produced curvatures that are often several orders of magnitude oﬀ from the correct values the case of high-dimensional inputs.

Revised: Due to the dimensionality issue above, we computed the average error across each principal curvature, regardless of the number of curvatures reported, as they should all equal 1 r .

Revised: We additionally tested a graph formulation (deﬁned in Section A.2.1), using a one-sheeted hyperboloid.

Revised: This formulation lives in the full space, so there is a principal curvature per input dimension.

Revised: However, we still found similar errors 4.

Revised: Note, that the formulation from (Golden et al., 2019) matched our own, although they did not provide an equation for mapping this to the level-set curvature.

Revised: Our own implementations were correct with high precision on these and other experiments with known functions, all of which can be replicated from our provided code.

Revised: Finally, while the subspace projection methods proposed by (Fawzi et al., 2016; MoosaviDezfooli et al., 2017b) could potentially provide metrics related to curvature in the subspaces they deﬁned, we chose to instead derive our pullback method described in Equation 3 from primary sources, as explained in Appendix A.

Revised: [Table]

Original: F MNIST results
Revised: G MNIST results

Original: [Figure]
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  We conducted the same experiments as described in Section 3 also for MNIST.
Revised: [Table] [Table] We conducted the same experiments as described in Section 3 on networks trained with the MNIST dataset.

Original: As a test set we used 500 images, unseen in training and correctly labeled by all investigated models. 
Revised: As a test set we used 500 images, unseen in training and correctly labeled by all investigated models. [Figure]

Original: The resulting ﬁgures are shown below. In Figure 15 we can see that perturbation lengths are much larger for MNIST than for CIFAR-10.
Revised: In Figure 16 we can see that perturbation lengths are much larger for MNIST than for CIFAR-10, although the relative increase in distances for the robust network is still present.

Revised: The optimization was in general more diﬃcult with MNIST, likely due to the fact that the images all live very close to the allowable input bounds.

Original: In fact, we stopped thesubspace algorithmat 30 found dimensions. In more than 95% of the test images, the algorithm found less than 30 dimensions as can be seen in Figure 10.
Revised: Speciﬁcally, the algorithm found less than 30 adversarial dimensions for more than 95% of the test images (Figure 10), to be compared with ﬁnding >50 dimensions for 100% of the CIFAR test images.

Original: That it is harder to fool the MNIST model can also be seen in
Revised: That it is harder to fool the MNIST model can also be seen in

Original: Table 3: the accuracies of both models are much higher for the same perturbation lengths than the accuracies of the CIFARmodels.
Revised: Table 5: the accuracies of both models are much higher for the same perturbation lengths than the accuracies of the CIFARmodels.

Original: For the curvature analysis, on the other hand, we found that the overall values to be larger on CIFAR.
Revised: For the curvature analysis, on the other hand, we found that the overall values to be larger on CIFAR.

Original: Additionally, the adversarially trained model had more decision boundary curvature within adversarial subspaces on MNIST than on CIFAR.
Revised: Additionally, the adversarially trained model had more decision boundary curvature within adversarial subspaces on MNIST than on CIFAR.

Original: We additionally noticed that most of the MNIST adversarials lie in a thin regions between the decision boundary and the valid pixel box boundary (compare Figure 14).
Revised: We additionally noticed that most of the MNIST adversarials lie in a thin regions between the decision boundary and the valid pixel box boundary (compare Figure 15).

Original: Figure 12 shows that this leads to large rates of out of bounds samples when measuring the distance to the decision boundary.
Revised: Figure 13 shows that this leads to large rates of out of bounds samples when measuring the distance to the decision boundary.

Original: For example 90% of the time the decision boundary is outside the valid pixel range in the 6 dimensional adversarial subspace of the naturally trained model.
Revised: For example 90% of the time the decision boundary is outside the valid pixel range in thedimensional adversarial subspace of the naturally trained model.

Original: [Table]
Revised: [Figure]

Original: 38.61% / 37.46% / 32.74% / 0.59% 9.46% 27.99% 46.52% 65.77% 76.32% 82.59% 83.94% 85.42% 80.16% 0.0% / 0.0% / 0.0% / 0.39% / 1.98% / 7.86% / 10.45% / 18.94% / 16.48% / 16.55% / 0.0% 0.0% 0.2% 7.69% 24.69% 37.51% 47.63% 48.86% 54.54% 40.74% 1 2 3 4 5 6 7     d i s t a n c e t o d e c

Original:  i s i o n b n d . natural robust 1 2 3 4 5 6 70.{+
+}
Revised: d i s t a n c e t o d e c i s i o n b n d . natural robust {+
+}

Revised: [Figure]

Original: d i s t a n c e c oe ffi
Revised: d i s t a n c e c oe ff

Original:  c i e n t   0.
Revised: i c i e n t   0.

Original: [Figure]
Revised: [Figure]

Original: adv dir0.

Original: adv dir0. adv dir0. adv dir0. adv dir0.
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: adversarial dimension a d v e r s a r i a l v e c t
Revised: a d v e r s a r i a l v e c t

Original: o r l e n g t 
Revised: o r l e n g t h

Original: h natural robust 2.
Revised: [Figure]

Original: distance to decision boundary a d v e r s a r i a l d i m e n s i o n
Revised: a d v e r s a r i a l d i m e n s i o n

Original: Original 6 (1.63)
Revised: Original 6 (1.63)

Original: Natural 3 (3.56)
Revised: Natural 3 (3.56)

Original: Robust 5 (1.67)
Revised: Robust 5 (1.67)

Original: Natural 8 (4.11)
Revised: Natural 8 (4.11)

Original: Robust 3 (1.92)
Revised: Robust 3 (1.92)

Original: Natural 2 (4.63)
Revised: Natural 2 (4.63)

Original: Robust 2 (3.02)
Revised: Robust 2 (3.02)

Original: Natural 4 (5.23)
Revised: Natural 4 (5.23)

Original: Robust 7 (3.52)
Revised: Robust 7 (3.52)

Original: Natural 0 (nan)
Revised: Natural 0 (nan)

Original: Robust 8 (0.95) 8 (1.85) 3 (1.96) 3 (1.90) 6 (2.31) 0 (2.49) 7 (2.50) 9 (2.94) 3 (3.12) 3 (3.21) 3 (0.86) 3 (2.28) 6 (1.21) 0 (2.51) 3 (1.54) 3 (4.18) 0 (1.55) 4 (4.58) 5 (2.18) 3 (7.63) 5 (1.36) 1 (2.12) 2 (1.50) 7 (2.38) 5 (2.30) 5 (3.18) 2 (2.82) 2 (3.42) 7 (3.39) 2 (4.13) 4 (0.88) 8 (3.33) 7 (1.48) 5 (3.77) 2 (1.48) 2 (3.84) 8 (1.96) 7 (4.44) 2 (2.73) 2 (4.98) 
Revised: Robust 8 (0.95) 8 (1.85) 3 (1.96) 3 (1.90) 6 (2.31) 0 (2.49) 7 (2.50) 9 (2.94) 3 (3.12) 3 (3.21) 3 (0.86) 3 (2.28) 6 (1.21) 0 (2.51) 3 (1.54) 3 (4.18) 0 (1.55) 4 (4.58) 5 (2.18) 3 (7.63) 5 (1.36) 1 (2.12) 2 (1.50) 7 (2.38) 5 (2.30) 5 (3.18) 2 (2.82) 2 (3.42) 7 (3.39) 2 (4.13) 4 (0.88) 8 (3.33) 7 (1.48) 5 (3.77) 2 (1.48) 2 (3.84) 8 (1.96) 7 (4.44) 2 (2.73) 2 (4.98) [Figure]

Original: Adv
Revised: Adv

Original: Adv
Revised: [Figure]

Original: Adv
Revised: Adv

Revised: [Figure]

Original: Adv
Revised: Adv

Revised: [Figure]

Original: Adv
Revised: Adv