Original: The Geometry of Adversarial Subspaces
Revised: The Geometry of Adversarial Subspaces

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Artiﬁcial neural networks (ANNs) are constructed using well-understood mathematical operations, and yet their high-dimensional, non-linear, and compositional nature has hindered our ability to provide an intuitive description of how and why they produce any particular output.
Revised: Artiﬁcial neural networks (ANNs) are constructed using well-understood mathematical operations, and yet their high-dimensional, non-linear, and compositional nature has hindered our ability to provide an intuitive description of how and why they produce any particular output.

Original: A striking example of this lack of understanding is our inability to design networks that are robust to adversarial input perturbations, which are often imperceptible to a human observer but cause signiﬁcant undesirable changes in the network’s response.
Revised: A striking example of this lack of understanding is our inability to design networks that are robust to adversarial input perturbations, which are often imperceptible to a human observer but cause signiﬁcant undesirable changes in the network’s response.

Original: The primary contribution of this work is to further our understanding of the decision boundary geometry of ANN classiﬁers by utilizing such adversarial perturbations.
Revised: The primary contribution of this work is to further our understanding of the decision boundary geometry of ANN classiﬁers by utilizing such adversarial perturbations.

Original: For this purpose, we deﬁne adversarial subspaces, which are spanned by orthogonal directions of minimal perturbation to the decision boundary from any given input sample.
Revised: For this purpose, we deﬁne adversarial subspaces, which are spanned by orthogonal directions of minimal perturbation to the decision boundary from any given input sample.

Original: We ﬁnd that the decision boundary lies close to input samples in a large subspace, where the distance to the boundary grows smoothly and sub-linearly as one increases the dimensionality of the subspace.
Revised: We ﬁnd that the decision boundary lies close to input samples in a large subspace, where the distance to the boundary grows smoothly and sub-linearly as one increases the dimensionality of the subspace.

Original: We undertake analysis to characterize the geometry of the boundary, which is more curved within the adversarial subspace than within a random subspace of equal dimensionality.
Revised: We undertake analysis to characterize the geometry of the boundary, which is more curved within the adversarial subspace than within a random subspace of equal dimensionality.

Original: To date, the most widely used defense against test-time adversarial attacks is adversarial training, where one incorporates adversarial attacks into the training procedure.
Revised: To date, the most widely used defense against test-time adversarial attacks is adversarial training, where one incorporates adversarial attacks into the training procedure.

Original: Using our analysis, we provide new insight into the consequences of adversarial training by quantifying the increase in boundary distance within adversarial subspaces, the redistribution of proximal class labels, and the decrease in boundary curvature.
Revised: Using our analysis, we provide new insight into the consequences of adversarial training by quantifying the increase in boundary distance within adversarial subspaces, the redistribution of proximal class labels, and the decrease in boundary curvature.

Original:  Introduction
Revised:  Introduction

Original: Artiﬁcial neural networks (ANNs) have been highly performant on common machine learning tasks, but their application space is limited by their susceptibility to adversarial attacks (Szegedy et al., 2014), which underscores a general lack of understanding for how and why they make their decisions.
Revised: Artiﬁcial neural networks (ANNs) have been highly performant on common machine learning tasks, but their application space is limited by their susceptibility to adversarial attacks (Szegedy et al., 2014), which underscores a general lack of understanding for how and why they make their decisions.

Original: Examples produced by adversarial attacks are a worst-case demonstration of an ANN’s inability to gracefully cope with identity-preserving shifts or distortions of its inputs.
Revised: Examples produced by adversarial attacks are a worst-case demonstration of an ANN’s inability to gracefully cope with identity-preserving shifts or distortions of its inputs.

Original: To construct one, an adversary must perturb the input in a small but speciﬁc way such that the output of the network changes signiﬁcantly.
Revised: To construct one, an adversary must perturb the input in a small but speciﬁc way such that the output of the network changes signiﬁcantly.

Original: Adversarial perturbations are deﬁned to have minimal length, resulting in inputs that are as close to the network’s decision boundary as possible.
Revised: Adversarial perturbations are deﬁned to have minimal length, resulting in inputs that are as close to the network’s decision boundary as possible.

Original: Here, we present a method that uses an untargeted adversarial attack objective to ﬁnd the multi-dimensional subspace where an ANN’s decision boundary is closest to any given input sample.
Revised: Here, we present a method that uses an untargeted adversarial attack objective to ﬁnd the multi-dimensional subspace where an ANN’s decision boundary is closest to any given input sample.

Original: The resulting adversarial subspace allows us to visualize and understand ANN decision boundaries in terms of their curvature, their shape, and their proximity to the input.
Revised: The resulting adversarial subspace allows us to visualize and understand ANN decision boundaries in terms of their curvature, their shape, and their proximity to the input.

Original: One existing technique to explain ANN classiﬁcations has relied on two-dimensional visualizations of the decision surface, although it suﬀers from a number of shortcomings, most notably high-dimensional points resolving to the same projected location, distortions of distances, and a lack of invertibility from the two-dimensional space back to the high-dimensional decision space (Féraud & Clérot, 2002; Rauber et al., 2016;
Revised: One existing technique to explain ANN classiﬁcations has relied on two-dimensional visualizations of the decision surface, although it suﬀers from a number of shortcomings, most notably high-dimensional points resolving to the same projected location, distortions of distances, and a lack of invertibility from the two-dimensional space back to the high-dimensional decision space (Féraud & Clérot, 2002; Rauber et al., 2016;

Original: Rodrigues et al., 2019).
Revised: Rodrigues et al., 2019).

Original: A popular alternative strategy is to investigate the decision boundary in the high-dimensional input space itself, early attempts of which (Golland, 2001; Baehrens et al., 2010) lead the way to the modern interest in adversarial attacks (Biggio et al., 2013; Szegedy et al., 2014).
Revised: A popular alternative strategy is to investigate the decision boundary in the high-dimensional input space itself, early attempts of which (Golland, 2001; Baehrens et al., 2010) lead the way to the modern interest in adversarial attacks (Biggio et al., 2013; Szegedy et al., 2014).

Original: For example, one can measure the distance to the decision boundary for random perturbation
Revised: For example, one can measure the distance to the decision boundary for random perturbation

Original: [Figure]
Revised: [Figure]

Original: + = + = ..
Revised: + = + = ..

Original: predictedclass

Original: [Figure]
Revised: [Figure]

Original:  from  orthogonal directions which are proximal to  initial adversarial example (Tramèr et 
Revised: directions away from adversarial examples (Tabacof & Valle, 2016; He et al., 2018), or for orthogonal directions which are proximal to an initial adversarial example (Tramèr et al., 2017).

Original: Alternatively, much work has demonstrated the utility of visualizing two-dimensional cross-sections of the decision space between adversarial examples and random orthogonal directions (Warde-Farley & Goodfellow, 2016; Yu et al., 2019a) or orthogonal test set examples (Swirszcz et al., 2019).
Revised: Alternatively, much work has demonstrated the utility of visualizing two-dimensional cross-sections of the decision space between adversarial examples and random orthogonal directions (Warde-Farley & Goodfellow, 2016; Yu et al., 2019a) or orthogonal test set examples (Swirszcz et al., 2019).

Original: Unlike all of the existing approaches to characterize decision boundaries in the input space, which rely on at most one adversarial direction per input, we propose jointly deﬁning a subspace of adversarial directions.
Revised: Unlike all of the existing approaches to characterize decision boundaries in the input space, which rely on at most one adversarial direction per input, we propose jointly deﬁning a subspace of adversarial directions.

Original: The resulting analysis leads to a general explanation of classiﬁer decision boundaries, as well as adversarial susceptibility, in the subspace where the boundary is closest to the input example.
Revised: The resulting analysis leads to a general explanation of classiﬁer decision boundaries, as well as adversarial susceptibility, in the subspace where the boundary is closest to the input example.

Original: As a further demonstration of the utility of our approach, we investigate the most widely used technique for defending against adversarial attacks, so-called adversarial training.
Revised: As a further demonstration of the utility of our approach, we investigate the most widely used technique for defending against adversarial attacks, so-called adversarial training.

Original: The defense method requires including adversarial perturbations in the training process by either augmenting the dataset with pre-computed examples (Ilyas et al., 2019) or by incorporating an attack model into the training procedure (Goodfellow et al., 2014; Madry et al., 2018).
Revised: The defense method requires including adversarial perturbations in the training process by either augmenting the dataset with pre-computed examples (Ilyas et al., 2019) or by incorporating an attack model into the training procedure (Goodfellow et al., 2014; Madry et al., 2018).

Original: It has been recently demonstrated that adversarial training performs well on the CIFARmachine learning dataset when enough resources are dedicated to hyperparameter tuning (Gowal et al., 2020), although the performance is still far from that on unperturbed images.
Revised: It has been recently demonstrated that adversarial training performs well on the CIFARmachine learning dataset when enough resources are dedicated to hyperparameter tuning (Gowal et al., 2020), although the performance is still far from that on unperturbed images.

Original: We provide a new perspective on this defense by observing that it increases the distance to the decision boundary in the entire adversarial subspace, even though the method utilizes only a single adversarial perturbation per input sample.
Revised: We provide a new perspective on this defense by observing that it increases the distance to the decision boundary in the entire adversarial subspace, even though the method utilizes only a single adversarial perturbation per input sample.

Original: We additionally ﬁnd that adversarial training increases the number of alternative label regions in adversarial subspaces, and changes the relative distribution of nearby alternate classes.
Revised: We additionally ﬁnd that adversarial training increases the number of alternative label regions in adversarial subspaces, and changes the relative distribution of nearby alternate classes.

Original: Herein, we will focus on deﬁning the method and summarizing the subspace with a continued application of understanding adversarial training.
Revised: Herein, we will focus on deﬁning the method and summarizing the subspace with a continued application of understanding adversarial training.

Original:  Methods
Revised:  Methods

Original: In the following we will outline the novel methods employed in this study.
Revised: In the following we will outline the novel methods employed in this study.

Original: Namely, the method for ﬁnding adversarial subspaces is ﬁrst described, and then we provide an overview of how we measure decision boundary curvature (with additional derivations given in Appendix
Revised: Namely, the method for ﬁnding adversarial subspaces is ﬁrst described, and then we provide an overview of how we measure decision boundary curvature (with a rigorous discussion given in Appendix

Original: A).
Revised: A).

Original: Our primary investigation was conducted using the CIFAR-10 dataset (Krizhevsky & d i s t a n c e t o d e c
Revised: Our primary investigation was conducted using the CIFAR-10 dataset (Krizhevsky & d i s t a n c e t o d e c

Original: i s i o n b n d . natural robust       dimension of adversarial space 1.
Revised: i s i o n b n d . natural robust       dimension of adversarial space 1.

Original: d i s t a n c e c
Revised: d i s t a n c e c

Original: oe ff i
Revised: oe ff i

Original: c i e n t 10.
Revised: c i e n t 10.

Original: Observe that mean 
Revised: Observe that mean smoothly

Original: Hinton, 2009) and compared two identical ResNet-50 ANNs (He et al., 2016) trained with and without adversarial examples.
Revised: Hinton, 2009) and compared two identical ResNet-50 ANNs (He et al., 2016) trained with and without adversarial examples.

Revised: The networks were modiﬁed and retrained using Cactivation functions for the curvature analysis (ELU with α = 1).

Revised: Further dataset and model details can be found in Appendix B.

Revised: In Appendix C, we provide additional results on the

Revised: CIFAR-10 dataset, including a comparison with the WideResNet-70 architecture with Swish activations from Gowal et al.

Original: In Appendix G we provide results on the MNIST dataset, which demonstrates largely consistent trends across datasets and across weight initialization seeds.
Revised: In Appendix D we provide results on the MNIST dataset, which demonstrates largely consistent trends across datasets. In Appendix E we include control tests to establish the consistency of our result for diﬀerent random seeds.

Original: Alldistances measured  are with respect to the ‘ 2 -norm.
Revised: All measured distances are with respect to the ‘ 2 -norm.

Original: Additional dataset and model details can be found in Appendix B.

Original: 2. Finding adversarial subspaces
Revised: 2. Finding adversarial subspaces

Original: We aim to ﬁnd the set of orthonormal vectors that span the subspace where an input sample has minimal distance to the decision boundary.
Revised: We aim to ﬁnd the set of orthonormal vectors that span the subspace where an input sample has minimal distance to the decision boundary.

Original: In order to achieve this, we follow a greedy optimization approach to span an adversarial subspace up to a desired number of dimensions.
Revised: In order to achieve this, we follow a greedy optimization approach to span an adversarial subspace up to a desired number of dimensions.

Original: More precisely, consider some neural network, f , that receives an n -pixel input image, x ∈ R n , and produces an output vector of probabilities associated with l label categories, ˆ y ∈ R l .
Revised: More precisely, consider some neural network, f , that receives an n -pixel input image, x ∈ R n , and produces an output vector of probabilities associated with l label categories, ˆ y ∈ R l .

Original: For a given input, we wish to ﬁnd an ordered set of orthogonal vectors, i.e. ∆ =
Revised: For a given input, we wish to ﬁnd an ordered set of orthogonal vectors, i.e. ∆ =

Original: [ δ 1 , δ 2 , . . .
Revised: [ δ 1 , δ 2 , . . .

Original: , δ m ] , m ≤ n , such that h δ k , δ m i = 0 , ∀ k < m .
Revised: , δ m ] , m ≤ n , such that h δ k , δ m i = 0 , ∀ k < m .

Original: Thus for each vector, we want to solve an optimization objective that minimizes || δ m || p , where ||
Revised: Thus for each vector, we want to solve an optimization objective that minimizes || δ m || p , where ||

Original: δ || p
Revised: δ || p

Original: =
Revised: =

Original: ( P i | δ | p ) 1 p is the l p norm.
Revised: ( P i | δ | p ) 1 p is the l p norm.

Original: We expect the optimizer to explicitly and jointly consider the boundedness of the input domain, force the perturbations to be orthogonal, and produce perturbations that result in new (incorrect) classiﬁer outputs.
Revised: We expect the optimizer to explicitly and jointly consider the boundedness of the input domain, force the perturbations to be orthogonal, and produce perturbations that result in new (incorrect) classiﬁer outputs.

Original: Such an objective is intractable to solve for deep neural networks, but is well approximated by the untargeted adversarial attack objective 1 , which modiﬁes the conditional to be subject to max j 

Revised: Such an objective is intractable to solve for deep neural networks, but is commonly approximated by the untargeted adversarial attack objective 1 , which modiﬁes the conditional to be subject to max j

Original: = i f j ( x + δ )
Revised: = i f j ( x + δ )

Original: − f i ( x + δ ) > 0, where i is the correct class index.
Revised: − f i ( x + δ ) > 0, where i is the correct class index.

Original: The length of these vectors is nearly synonymous with distances to the decision boundary, where the actual decision boundary would be an inﬁnitesimal step towards x from x + δ .
Revised: The length of these vectors is nearly synonymous with distances to the decision boundary, where the actual decision boundary would be an inﬁnitesimal step towards x from x + δ .

Original: We must further relax the objective to a diﬀerentiable approximation for ﬁnding a gradient based solution.
Revised: We must further relax the objective to a diﬀerentiable approximation for ﬁnding a gradient based solution.

Original: While there are many such approximations, we opted for a modiﬁcation of that which was used by Szegedy et al.
Revised: While there are many such approximations, we opted for a modiﬁcation of that which was used by Szegedy et al.

Original: (2014) to ﬁnd singular adversarial directions, which is to minimize the inverse of the cross entropy loss (the training objective of the model), weighed with a distance penalization term.
Revised: (2014) to ﬁnd singular adversarial directions, which is to minimize the inverse of the cross entropy loss (the training objective of the model), weighed with a distance penalization term.

Original: Speciﬁcally, ‘  =
Revised: Speciﬁcally, ‘ def =

Original: [Equation]
Revised: [Equation]

Original: with the cross entropy loss deﬁned as L CE ( x )  = − log
Revised: with the cross entropy loss deﬁned as L CE ( x ) def = − log

Original: [Equation]
Revised: [Equation]

Original: x ) .
Revised: x ) .

Original: Using this loss term, we can now deﬁne our objective function as implemented: minimize
Revised: Using this loss term, we can now deﬁne our objective function as implemented: minimize

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this maximizes the model loss with respect to the true class label while keeping the perturbation as small as possible.
Revised: Intuitively, this maximizes the model loss with respect to the true class label while keeping the perturbation as small as possible.

Original: The ﬁrst constraint in Equation 1 is to ensure that the image is adversarial 2 , the second keeps the perturbed images within the allowable pixel bounds, and the last one enforces orthogonality of all found adversarial subspace dimensions.
Revised: The ﬁrst constraint in Equation 1 is to ensure that the image is adversarial 2 , the second keeps the perturbed images within the allowable pixel bounds, and the last one enforces orthogonality of all found adversarial subspace dimensions.

Original: We found the optimal value of constant κ in equation (1) with a binary search performed during run time.
Revised: We found the optimal value of constant κ in equation (1) with a binary search performed during run time.

Original: The optimization problem was solved using the greedy interior-point optimizer implemented by Ipopt (Wächter & Biegler, 2006).
Revised: The optimization problem was solved using the greedy interior-point optimizer implemented by Ipopt (Wächter & Biegler, 2006).

Original: Once the adversarial vectors are found, we conducted experiments to measure the distance to the decision boundary in the space spanned by the vectors (i.e. within the adversarial subspace).
Revised: Once the adversarial vectors are found, we conducted experiments to measure the distance to the decision boundary in the space spanned by the vectors (i.e. within the adversarial subspace).

Original: The experiment was repeated on 100 test images that were chosen to have equal label representation (10 images per label) and to be correctly classiﬁed by both networks.
Revised: The experiment was repeated on 100 test images that were chosen to have equal label representation (10 images per label) and to be correctly classiﬁed by both networks.

Original: For each test image we found 50 orthonormal adversarial vectors, which allowed us to deﬁne a sequentially growing list of adversarial subspaces with dimensionality increasing from 1 to 50.
Revised: For each test image we found 50 orthogonal adversarial vectors, which allowed us to deﬁne a sequentially growing list of adversarial subspaces with dimensionality increasing from 1 to 50.

Original: For a given image and subspace we sampled  directions via a linear combination of normally distributed variables, which has been previously proven to provide an even sampling (Muller, 1959).
Revised: For a given image and subspace we sampled 100 directions via a linear combination of normally distributed variables, which has been previously proven to provide an even sampling (Muller, 1959).

Original: Finally, for each random direction we performed a binary search to ﬁnd the decision boundary.
Revised: Finally, for each random direction we performed a binary search to ﬁnd the decision boundary.

Original: 2. Measuring decision boundary curvature
Revised: 2. Measuring decision boundary curvature

Original: As before, let f i , f j : R m → R be the functions corresponding to i and j -th logits of a neural network, f .
Revised: As before, let f i , f j : R m → R be the functions corresponding to i and j -th logits of a neural network, f .

Original: Let F ij def =
Revised: Let F ij def =

Original: f
Revised: f

Original: i − f j .
Revised: i − f j .

Original: Then the ( i, j ) -decision boundary is the set
Revised: Then the ( i, j ) -decision boundary is the set

Original: [Equation]
Revised: [Equation]

Original: The set B ij is deﬁned independently of the sign of F ij (that is, B ij =
Revised: The set B ij is deﬁned independently of the sign of F ij (that is, B ij =

Original: B ji ), although doing so will produce a change of sign in our calculations.
Revised: B ji ), although doing so will produce a change of sign in our calculations.

Original: Therefore we will continue to follow the above convention by identifying, at a point x , i with the correct label associated to x , and j with the incorrect label.
Revised: Therefore we will continue to follow the above convention by identifying, at a point x , i with the correct label associated to x , and j adv dir0.

Original: This convention (which agrees with Moosavi-Dezfooli et al., 2018), ensures that positive (resp.

Original: negative) curvature of B ij near x corresponds to B ij bowing adv dir0.

Original: adv dir0. adv dir0.
Revised: adv dir0. adv dir0.

Original: adv dir0. adv dir0.
Revised: adv dir0. adv dir0.

Original: [Figure]
Revised: [Figure]

Original: adv dir0. adv dir0.
Revised: adv dir0. adv dir0.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: adv dir0.
Revised: adv dir0.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  inward towards (resp.
Revised: with the incorrect label. This convention (which agrees with Moosavi-Dezfooli et al., 2018), ensures that positive (resp. negative) curvature of B ij near x corresponds to B ij bowing inward towards (resp.

Original: away from) x (see Figure 1 for an example of positive curvature).
Revised: away from) x (see Figure 1 for an example of positive curvature).

Original: If the curvature along B ij is zero near x then B ij is aﬃne.
Revised: If the curvature along B ij is zero near x then B ij is aﬃne.

Original: The complete details of how our calculations were derived, and numerical considerations therein, are relegated to Appendix A, however we give a brief intuitive account below.
Revised: The complete details of how our calculations were derived, and numerical considerations therein, are relegated to Appendix A, however we give a brief intuitive account below.

Original: In our curvature analysis we use a method known as principal curvature decomposition to understand the degree of linearity of the decision boundary.
Revised: In our curvature analysis we use a method known as principal curvature decomposition to understand the degree of linearity of the decision boundary.

Original: Since the decomposition is performed on a level-set of the decision space, which in general is of codimension 1, it yields n − 1 scalars and vectors, called the principal curvatures and principle curvature directions , respectively.
Revised: Since the decomposition is performed on a level-set of the decision space, which in general is of codimension 1, it yields n − 1 scalars and vectors, called the principal curvatures and principle curvature directions , respectively.

Original: The curvature directions point in the directions of maximal and minimal surface curvature, while the principal curvatures themselves indicate the signed magnitude of the curvature.
Revised: The curvature directions point in the directions of maximal and minimal surface curvature, while the principal curvatures themselves indicate the signed magnitude of the curvature.

Original: Similar to principal component analysis (PCA), which organizes the space as an orthogonal basis of directions of maximal variance, principal curvature analysis organizes the (tangential) space of the decision boundary into orthogonal directions of maximum curvature.
Revised: Similar to principal component analysis, which organizes the space as an orthogonal basis of directions of maximal variance, principal curvature analysis organizes the (tangential) space of the decision boundary into orthogonal directions of maximum curvature.

Original: The principle curvatures and associated directions are the eigenvectors and eigenvalues of a linear operator s : R m → R m called the shape operator .
Revised: The principle curvatures and associated directions are the eigenvectors and eigenvalues of a linear operator s : R m → R m called the shape operator .

Original: In order to study the curvature over an adversarial (or random) subspace we introduce the subspace pullback of s
Revised: In order to study the curvature over an adversarial (or random) subspace we introduce the subspace pullback of s

Original: [Equation]
Revised: [Equation]

Original: where P ∈ R m × n is an orthogonal projection, so that the image of P T spans the adversarial subspace of R m in question.
Revised: where P ∈ R m × n is an orthogonal projection, so that the image of P T spans the adversarial subspace of R m in question.

Original: This deﬁnition of (cid:0) P T (cid:1) ∗ s essentially restricts the shape operator to a linear subspace of the tangential space and is motivated by the principles of diﬀerential geometry, details of which can be found in Remark 1 and the surrounding material in
Revised: This deﬁnition of (cid:0) P T (cid:1) ∗ s essentially restricts the shape operator to a linear subspace of the tangential space and is motivated by the principles of diﬀerential geometry, details of which can be found in Remark 1 and the surrounding material in

Original: Appendix A.
Revised: Appendix A.

Original: The pullback shape operator allows us to compare the amount of curvature in the adversarial subspace to that of the entire response space.
Revised: The pullback shape operator allows us to compare the amount of curvature in the adversarial subspace to that of the entire response space.

Original:  Results
Revised:  Results

Original: By deﬁnition, the distance to the decision boundary increases for each additional dimension of an adversarial subspace.
Revised: By deﬁnition, the distance to the decision boundary increases for each additional dimension of an adversarial subspace.

Original: The circles in the top left panel of Figure 2 indicate these distances, averaged across the test images, which are found by greedily minimizing Equation 1.
Revised: The circles in the top left panel of Figure 2 indicate these distances, averaged across the test images, which are found by greedily minimizing Equation 1.

Original: The distance to the decision boundary grows sub-linearly as one increases the dimensionality of the subspace, with the fastest growth occurring in the ﬁrst couple of dimensions.
Revised: The distance to the decision boundary grows sub-linearly as one increases the dimensionality of the subspace, with the fastest growth occurring in the ﬁrst couple of dimensions.

Original: The leftmost box plots of the top left panel in ﬁgure 2 as well as the Appendix Table 2 conﬁrms that the mean initial perturbation distances are comparable to more traditional adversarial attacks (Madry et al., 2018) and decision boundary distance estimates (Ding et al., 2020)).
Revised: The leftmost box plots of the top left panel in ﬁgure 2 as well as the Appendix Table 2 conﬁrms that the mean initial perturbation distances are comparable to more traditional adversarial attacks (Madry et al., 2018) and decision boundary distance estimates (Ding et al., 2020).

Original: While this gives an idea of the lower-bound of distances to the decision boundary, it does not provide information about the behavior of the boundary elsewhere in the subspace.
Revised: While this gives an idea of the lower-bound of distances to the decision boundary, it does not provide information about the behavior of the boundary elsewhere in the subspace.

Original: To better understand this we uniformly sampled 100 random directions within a given subspace and performed a binary search to ﬁnd the decision boundary along those directions.
Revised: To better understand this, we uniformly sampled 100 random directions within a given subspace and performed a binary search to ﬁnd the decision boundary along those directions.

Original: We only report distances for decision boundary points that are within the ‘ inf allowable pixel box, which was never fewer than 40% of the subspace samples (see Appendix C Figure 7).
Revised: We only report distances for decision boundary points that are within the ‘ inf allowable pixel box, which was never fewer than 40% of the subspace samples (see Appendix C Figure 7).

Original: The box plots in the top left panel of Figure 2 indicate the distribution of decision boundary distances across the sampled directions and test images.
Revised: The box plots in the top left panel of Figure 2 indicate the distribution of decision boundary distances across the sampled directions and test images.

Original: Using a small sample of images, we found that the perturbation was discernible at after an ‘ 2 -distance of 1.5, which occurs after about 20dimensions for the adversarially trained network, and after greater thandimensions for the naturally trained network.
Revised: There is no consistent ‘ 2 -length of an adversarial attack that makes an perturbation visible to a human observer, although in general the perturbations were visible after around dimension 20 for the robust network and beyond dimension 50 for the natural network (see Appendix C Figure 6 for examples).

Original: The bottom left panel of Figure 2 plots a distance coeﬃcient that is deﬁned as the maximum distance to the decision boundary within an adversarial subspace divided by the longest vector deﬁning the subspace (which is always associated with the highest dimension number).
Revised: The bottom left panel of Figure 2 plots a distance coeﬃcient that is deﬁned as the maximum distance to the decision boundary within an adversarial subspace divided by the longest vector deﬁning the subspace (which is always associated with the highest dimension number).

Original: For example, a value of 2 would indicate that the largest sampled distance is twice as far away as the largest found adversarial perturbation.
Revised: For example, a value of 2 would indicate that the largest sampled distance is twice as far away as the largest found adversarial perturbation.

Revised: The adversarial trained network consistently produces smaller coeﬃcients, indicating that the boundary size is less variable between the subspace axes.

Original: Finally, a complete description of the boundary distances is conveyed as histograms in the rightmost panels, with dimensionality increasing from left to right, then top to bottom.
Revised: Finally, a complete description of the boundary distances is conveyed as histograms in the rightmost panels, with dimensionality increasing from left to right, then top to bottom.

Original: Together, this provides further evidence that the decision boundary grows consistently, without large variations in distance between the found adversarial subspace axes.
Revised: Together, this provides further evidence that the decision boundary grows consistently, without large variations in distance between the found adversarial subspace axes.

Original: In Figure 3 we visualize two-dimensional cross-sections of the decision surface.
Revised: In Figure 3 we visualize two-dimensional cross-sections of the decision surface.

Original: The horizontal axis of each cross-section corresponds to an adversarial subspace perturbation, the black arrow indicates the farthest decision boundary point, and the vertical axis is found by orthogonalizing the direction of the farthest decision boundary point with respect to the adversarial perturbation using one step of the Gram-Schmidt process.
Revised: The horizontal axis of each cross-section corresponds to an adversarial subspace perturbation, the black arrow indicates the farthest decision boundary point, and the vertical axis is found by orthogonalizing the direction of the farthest decision boundary point with respect to the adversarial perturbation using one step of the Gram-Schmidt process.

Original: The cross-sections are ordered according to the distance coeﬃcient, which is displayed in the upper-right corner.
Revised: The cross-sections are ordered according to the distance coeﬃcient, which is displayed in the upper-right corner.

Original: We emphasize that because these are cross-sections, they avoid the shortcomings of typical low-dimensional decision space projections noted in the introduction.
Revised: We emphasize that because these are cross-sections, they avoid the shortcomings of typical low-dimensional decision space projections noted in the introduction.

Original: Among the visualizations one can see examples of when the boundary increases smoothly from one axis to the next, and also the more rare (as indicated by Figure 2) cases when the boundary protrudes quite far from the origin when between the axes.
Revised: Among the visualizations one can see examples of when the boundary increases smoothly from one axis to the next, and also the more rare (as indicated by Figure 2) cases when the boundary protrudes quite far from the origin when between the axes.

Original: From this analysis we conclude that adversarial examples are not isolated or due to sharp protrusions into an otherwise homogeneously labeled region.
Revised: From this analysis we conclude that adversarial examples are not isolated or due to sharp protrusions into an otherwise homogeneously labeled region.

Original: Rather, there exists a smooth increase in distance to the decision boundary from the extreme case of an initially found adversarial example to semantically diﬀerentiated images.
Revised: Rather, there exists a smooth increase in distance to the decision boundary from the extreme case of an initially found adversarial example to semantically diﬀerentiated images.

Original:  In Figure 4 we provide an analysis of boundary curvature for several experimental conditions that vary the type of training, the location of the boundary, and  subspace deﬁning vectors.
Revised: [Table] In Figure 4 we measure boundary curvature for several experimental conditions that vary the type of training, the location of the boundary, and the subspace deﬁning vectors.

Original: This analysis reveals that there is a relationship between the boundary ﬂatness (i.e. linearity) and the existence of adversarial examples.

Original: Panels A and B report the mean curvature (i.e. the mean of the eigenvalues of the shape operator) per point on the decision boundary.
Revised: Panels A and B report the mean curvature ( i [Table] e. the mean of the eigenvalues of the shape operator) per point on the decision boundary.

Original: Both model types have larger boundary curvature for boundary points separating clean images from adversarial images (adversarial boundary; Panel B) when compared to boundary points separating  test set examples (test boundary; Panel A).
Revised: Both model types have larger boundary curvature for boundary points separating clean images from adversarial images (adversarial boundary; Panel B) when compared to boundary points separating unperturbed test set examples (test boundary; Panel A).

Original: We additionally found that adversarial training tends to linearize the decision boundary, both between correctly labeled test images and near adversarial examples.
Revised: We additionally found that adversarial training tends to linearize the decision boundary, both between correctly labeled test images and near adversarial examples.

Original: Panel C is a scatter plot of the principal curvatures (i.e. eigenvalues of the shape operator) in the high-dimensional input space for all of the adversarial boundary points.
Revised: Panel C is a scatter plot of the principal curvatures (i.e. eigenvalues of the shape operator) in the high-dimensional input space for all of the adversarial boundary points.

Original: We consistently ﬁnd both positive and negative curvature at each boundary point and for both training types.
Revised: We consistently ﬁnd both positive and negative curvature at each boundary point and for both training types.

Original: Using the pullback deﬁned in Equation 3, we can measure the subspace principal curvatures for random (Panel D) and adversarial (Panel E) subspaces.
Revised: Using the pullback deﬁned in Equation 3, we can measure the subspace principal curvatures for random (Panel D) and adversarial (Panel E) subspaces.

Original: The curvature within adversarial subspaces is notably larger than in random subspaces.
Revised: The curvature within adversarial subspaces is notably larger than in random subspaces.

Original: The diﬀerence in curvature values between naturally and adversarially trained networks is less pronounced when limited to random subspaces – suggesting that much of the linearization occurring during adversarial training is along directions aligned with adversarial examples.
Revised: The diﬀerence in curvature values between naturally and adversarially trained networks is less pronounced when limited to random subspaces – suggesting that much of the linearization occurring during adversarial training is along directions aligned with adversarial examples.

Original: [Table]
Revised: The found boundary points can be the closest to the clean input even when there is positive curvature, as long as the radius of curvature is greater than the distance to the input. However, for naturally trained networks, adversarial boundary points tend to have more negative mean curvature, both in the full input space (Panel B) and in the adversarial subspace (Panel E).

Original: In addition to the earlier comparisons between natural and adversarial network training, we measured the diversity of adversarial classes near test samples.
Revised: In addition to the earlier comparisons between natural and adversarial network training, we measured the diversity of adversarial classes near test samples.

Original: The leftmost subplot of Figure 5 gives the number of unique adversarial classes found in the ﬁrst 50 dimensions of the adversarial subspaces, averaged acrossa uniform sampling of correctly labeled test images.
Revised: The leftmost subplot of Figure 5 gives the number of unique adversarial classes found in the ﬁrst 50 dimensions of the adversarial subspaces, averaged across correctly labeled test images.

Original: The other plots show the distribution of adversarial classes present in the subspaces with respect to the origin label.
Revised: The other plots show the distribution of adversarial classes present in the subspaces with respect to the origin label.

Original: These plots suggest that adversarial training results in a more even distribution of alternate classes near any given test sample, which we further quantify by measuring the discrete entropy of each distribution in Table 1.It is apparent from our analysis that adversarial training pushes the decision boundary away from the origin in all dimensions tested, even though few adversarial perturbations are provided per training sample. In the top left panel of Figure 2, the dots indicate that the largest increase in boundary distance from one dimension to the next is after the ﬁrst, although the adversarial images corresponding to the ﬁrst several dimensions were often still imperceptibly changed from the original.
Revised: These plots suggest that adversarial training results in an increased diversity of alternate classes near any given test sample, which we further quantify by measuring the discrete entropy of each distribution in Table 1.

Original: Furthermore, in the bottom left panel of Figure 2 one can see that the adversarial trained network consistently produces smaller coeﬃcients, indicating that the boundary size is less variable between the subspace axes.

Original: In addition to (or more probably as a consequence of) increasing the distance to the boundary, adversarial training further modiﬁes the density of alternative class labels near test images.
Revised: Thus, in addition to (or more probably as a consequence of) increasing the distance to the boundary, adversarial training further modiﬁes the density of alternative class labels near test images.

Original: This is evident in Figure 5,
Revised: While this could be considered a

Original: [Figure]
Revised: [Figure]

Original: U n i q u e a d v e r s a r i a l c l a ss e s
Revised: U n i q u e a d v e r s a r i a l c l a ss e s

Original: [Figure]
Revised: [Figure]

Original: which demonstrates thatadversarial training increases the diversity ofalternative classes near test images on the left and modiﬁes the proximal class distribution on the right.
Revised: airplane car bird cat deer [Figure] desired eﬀect, as it indicates that the robust network utilizes more of the high-dimensional space, it also leads to behavior that is less matched to human behavior (something that is often sought along with adversarial robustness).

Original: While some categories, like “car” and “truck” are intuitively adjacent, the distribution of classes generally failed to match any clear intuitions.
Revised: Indeed, while some categories, like “car” and “truck” are intuitively adjacent, the distribution of classes generally failed to match any clear intuitions.

Original: For example, observe the lack of symmetry between cars and trucks, or that adversarial training switches the ”frog“ category’s most frequent alternate class from “bird” to “cat”.
Revised: For example, observe the lack of symmetry between cars and trucks, or that adversarial training switches the ”frog“ category’s most frequent alternate class from “bird” to “cat”.

Original: Indeed, Table 1 suggests that adversarial training spreads out the distribution among classes, suggesting that the decision surface is more evenly utilizing the high-dimensional space.

Original:  Related Work
Revised:  Related Work

Original: Related adversarial subspace studies: The work of Tramèr et al.
Revised: Related adversarial subspace studies: The work of Tramèr et al.

Original: (2017) is most similar to our adversarial decomposition, in that they reported the dimensionality of a subspace in the proximity of a given adversarial perturbation, although  do not comment on the number or proximity of alternative adversarial examples.
Revised: (2017) is most similar to our adversarial decomposition, in that they reported the dimensionality of a subspace in the proximity of a given adversarial perturbation, although they do not comment on the class, number, or proximity of nearby orthogonal adversarial examples.

Original: Work from Ma et al.
Revised: Work from Ma et al.

Original: (2018) measured the local intrinsic dimensionality (LID) of adversarial subspaces, which indirectly estimates adversarial subspace dimensionality by measuring the distance from an adversarial example to a batch of nearest unperturbed dataset examples.
Revised: (2018) measured the local intrinsic dimensionality (LID) of adversarial subspaces, which indirectly estimates adversarial subspace dimensionality by measuring the distance from an adversarial example to a batch of nearest unperturbed dataset examples.

Original: This idea has been extended to algorithms for detecting adversarial examples (Ma et al., 2018; Mao et al., 2020), although Athalye et al.
Revised: This idea has been extended to algorithms for detecting adversarial examples (Ma et al., 2018; Mao et al., 2020), although Athalye et al.

Original: (2018) generated adversarial examples with indistinguishable LID scores from natural examples.
Revised: (2018) generated adversarial examples with indistinguishable LID scores from natural examples.

Original: In terms of understanding ANN classiﬁers, such an approach leaves open the questions of how proximal the decision boundary is from input samples in greater than one dimension or the actual dimensionality of adversarial subspaces.
Revised: In terms of understanding ANN classiﬁers, such an approach leaves open the questions of how proximal the decision boundary is from input samples in greater than one dimension or the actual dimensionality of adversarial subspaces.

Original: Our approach is the ﬁrst to pursue a more global examination of the decision boundary where it is nearest to test samples by deﬁning adversarial subspaces using multiple orthogonal adversarial directions.
Revised: Our approach is the ﬁrst to examine the decision boundary where it is nearest to test samples by deﬁning adversarial subspaces using multiple orthogonal adversarial directions.

Original: Our emphasis on the subspace where the decision boundary is closest to test images is also unique when compared to work that characterizes the decision boundary in random directions, which will be considerably farther away from the origin (e.g. Fawzi et al., 2016; 
Revised: Our emphasis on the subspace where the decision boundary is closest to test images is also unique when compared to work that characterizes the decision boundary in random directions, which will be considerably farther away from the origin (e.g. Fawzi et al., 2016; Tabacof & Valle, 2016;

Original: Tabacof & Valle, 2016; Liu et al., 2017; He et al., 2018).
Revised: Liu et al., 2017; He et al., 2018).

Original: Finally, most of these studies perform a clipping operation after deﬁning or ﬁnding the desired directions, which results in a non-orthogonal set (see Appendix E for a more detailed discussion).
Revised: Finally, most of these studies perform a clipping operation after deﬁning or ﬁnding the desired directions, which results in a non-orthogonal set (see Appendix F for a more detailed discussion).

Original: We overcome the problemof clipping pixel values outside the valid pixel range by including the valid  range as inequality constraints in the optimization process.
Revised: We overcome this problem by including the valid pixel range as inequality constraints in the optimization process.

Original: Our work is complementary to previous studies that investigate the relationship between adversarial robustness and input dimensionality (Amsaleg et al., 2017; Simon-Gabriel et al., 2019), since the comparative trends observed between robust and natural networks are relatively unchanged with small (MNIST) or large (CIFAR) dimensional inputs.
Revised: Our work is complementary to previous studies that investigate the relationship between adversarial robustness and input dimensionality (Amsaleg et al., 2017; Simon-Gabriel et al., 2019), since the comparative trends observed between robust and natural networks are relatively unchanged with small (MNIST) or large (CIFAR) dimensional inputs.

Original: Neuron response curvature: The use of response curvature to understand nonlinear functions has a long history in neuroscience (for a review, see Gollisch & Herz, 2012) as well as models of neural computation (e.g. Zetzsche & Krieger, 2001;
Revised: Neuron response curvature: The use of response curvature to understand nonlinear functions has a long history in neuroscience (for a review, see Gollisch & Herz, 2012) as well as models of neural computation (e.g. Zetzsche & Krieger, 2001;

Original: Rust et al., 2005; Golden et al., 
Revised: Rust et al., 2005; Golden et al., 2016;

Revised: Cohen et al., 2020).

Original: The principal curvature decomposition was previously employed in this context by Golden et al.
Revised: The principal curvature decomposition was previously employed in this context by Golden et al.

Original: (2019), who used it to understand invariance and selectivity of individual ANN neurons, and by Paiton et al.
Revised: (2019), who used it to understand invariance and selectivity of individual ANN neurons, and by Paiton et al.

Original: (2020), who ﬁrst found a relationship between individual neuron response curvature and adversarial robustness.
Revised: (2020), who ﬁrst found a relationship between individual neuron response curvature and adversarial robustness.

Original: Our work is distinct from the aforementioned studies in that we investigate the curvature of the decision boundary, as opposed to that of individual neurons.
Revised: Our work is distinct from the aforementioned studies in that we investigate the curvature of the decision boundary, as opposed to that of individual neurons.

Original: However, the relationship between decision boundary curvature and adversarial robustness has been previously studied  Moosavi-Dezfooli and colleagues, who provided a number of compelling results identifying “an increasing [adversarial] vulnerability with respect to the curvature of the decision boundary” (Moosavi-Dezfooli et al., 2018)(for a complete account, see the thesis of Moosavi-Dezfooli, 2019).
Revised: However, the relationship between decision boundary curvature and adversarial robustness has been previously studied by Moosavi-Dezfooli and colleagues, who provided a number of compelling results identifying “an increasing [adversarial] vulnerability with respect to the curvature of the decision boundary” (Moosavi-Dezfooli et al., 2018).

Original: As we will discuss in the next Section, our analysis provides support for hypotheses that go beyond the correlation between decision boundary curvature and network robustness, in that we measure boundary curvature inthe adversarial subspace dataset samples.
Revised: As we discussed in the previous section, our analysis provides support for hypotheses that go beyond the correlation between decision boundary curvature and network robustness, in that we observe diﬀerent boundary curvature in adversarial subspaces than random subspaces or the full input space.

Original: We additionally note that while previous work has utilized Riemannian curvature analysis to understand neural networks (viz.
Revised: We additionally note that while previous work has utilized Riemannian curvature analysis to understand neural networks (viz.

Original: Fawzi et al., 2016; Poole et al., 2016; Moosavi-Dezfooli et al., 2018; 2019; Golden et al., 2019), each study measures diﬀerent quantities without complete details of the derived formulas.
Revised: Fawzi et al., 2016; Poole et al., 2016; Moosavi-Dezfooli et al., 2018; 2019; Golden et al., 2019), each study measures diﬀerent quantities without complete details of the derived formulas.

Original: We demonstrate in Appendix F that these formulas do not ﬁnd the correct level-set curvature of simple manifolds.
Revised: We demonstrate in Appendix G that these formulas do not ﬁnd the correct level-set curvature of simple manifolds.

Original: This together with our rigorous derivation and discussion in Appendix
Revised: This together with our rigorous derivation and discussion in Appendix

Original: A constitutes, to the best of our knowledge, the ﬁrst principled and analytically supported measurement of decision boundary curvature of trained nonlinear neural networks.
Revised: A constitutes, to the best of our knowledge, the ﬁrst principled and analytically supported measurement of decision boundary curvature of trained nonlinear neural networks.

Original:  Conclusion
Revised:  Conclusion

Original: There is an ongoing debate on the cause, prevalence, and uniformity of adversarial examples.
Revised: There is an ongoing debate on the cause, prevalence, and uniformity of adversarial examples.

Original: Several studies have supported a hypothesis that most adversarial examples inhabit relatively dense regions of the input space (Goodfellow et al., 2014; Tabacof & Valle, 2016;
Revised: Several studies have supported a hypothesis that most adversarial examples inhabit relatively dense regions of the input space (Goodfellow et al., 2014; Tabacof & Valle, 2016;

Original: MoosaviDezfooli et al., 2017a; He et al., 2018).
Revised: MoosaviDezfooli et al., 2017a; He et al., 2018).

Original: However, other works have suggested that adversarial examples live in isolated pockets or thin “spikes” of incorrect classiﬁcation regions (Szegedy et al., 2014; Yu et al., 2019b).
Revised: However, other works have suggested that adversarial examples live in isolated pockets or thin “spikes” of incorrect classiﬁcation regions (Szegedy et al., 2014; Yu et al., 2019b).

Original: Here we provide quantitative evaluation supporting the hypothesis that decision boundaries are close to input samples in a large subspace.
Revised: Here we provide quantitative evaluation supporting the hypothesis that decision boundaries are close to input samples in a large subspace.

Original: Additional work has suggested that it is the linearity of deep neural networks that leads to adversarial examples (Goodfellow et al., 2014), and that adversarial training leads to a less linear decision boundary (Madry et al., 2018).
Revised: Additional work has suggested that it is the linearity of deep neural networks that leads to adversarial examples (Goodfellow et al., 2014), and that adversarial training leads to a less linear decision boundary (Madry et al., 2018).

Original: However, more recent work has proposed the alternative: that increasing decision boundary linearity causes increased robustness (Li et al., 2019b;
Revised: However, more recent work has proposed the alternative: that increasing decision boundary linearity causes increased robustness (Li et al., 2019b;

Original: Moosavi-Dezfooli et al., 2019; Terjék, 2020; Sarkar & Iyengar, 2020).
Revised: Moosavi-Dezfooli et al., 2019; Terjék, 2020; Sarkar & Iyengar, 2020).

Original: Our curvature analysis conﬁrms that adversarial training linearizes the boundary, which is also congruent with hypotheses about the robustness beneﬁts of smoothing the decision surface (Zhang et al., 2019;
Revised: Our curvature analysis conﬁrms that adversarial training linearizes the boundary, which is also congruent with hypotheses about the robustness beneﬁts of smoothing the decision surface (Zhang et al., 2019;

Original: Wu et al., 2020).
Revised: Wu et al., 2020).

Original: We further compare the boundary curvature in adversarial subspaces and random subspaces to show an increase in curvature in adversarial subspaces for both training methods.
Revised: We further compare the boundary curvature in adversarial subspaces and random subspaces to show an increase in curvature in adversarial subspaces for both training methods.

Original: Since we do not report the causal relationship between adversarial directions and boundary curvature, our results have no impact on hypotheses for adversarial susceptibility due to overﬁtting (Tanay & Griﬃn, 2016) or the perceptual quality of adversarial features (Ilyas et al., 2019), although we see this as an interesting future direction of study.
Revised: Since we do not report the causal relationship between adversarial directions and boundary curvature, our results have no impact on hypotheses for adversarial susceptibility due to overﬁtting (Tanay & Griﬃn, 2016) or the perceptual quality of adversarial features (Ilyas et al., 2019), although we see this as an interesting future direction of study.

Original: Adversarial subspaces allow for a quantitative description of the nonlinear geometry and class composition of the network decision boundary in the subspace spanned by its most susceptible perturbation directions.
Revised: Adversarial subspaces allow for a quantitative description of the nonlinear geometry and class composition of the network decision boundary in the subspace spanned by its most susceptible perturbation directions.

Original: They also assist in explaining the prevalence of adversarial examples as well as the eﬀects of adversarial training.
Revised: They also assist in explaining the prevalence of adversarial examples as well as the eﬀects of adversarial training.

Original: Understanding this space better can improve neural network performance with semi-supervised relabeling (Benato et al., 2018), architecture design (Rauber et al., 2018), and adversarial robustness (Tramèr et al., 2017;
Revised: Understanding this space better can improve neural network performance with semi-supervised relabeling (Benato et al., 2018), architecture design (Rauber et al., 2018), and adversarial robustness (Tramèr et al., 2017;

Original: Moosavi-Dezfooli et al., 2019).
Revised: Moosavi-Dezfooli et al., 2019).

Original: Importantly, we also provide a quantitative and falsiﬁable interpretability approach for understanding ANN decisions (Ribeiro et al., 2016; Vlassopoulos et al., 2020), which is a necessary prerequisite for the technology to be applied to many industries (Doshi-Velez & Kim, 2017; Leavitt & Morcos, 2020).
Revised: Importantly, we also provide a quantitative and falsiﬁable interpretability approach for understanding ANN decisions (Ribeiro et al., 2016; Vlassopoulos et al., 2020), which is a necessary prerequisite for the technology to be applied to many industries (Doshi-Velez & Kim, 2017; Leavitt & Morcos, 2020).

Original: We have identiﬁed an interesting avenue for future work in performing a large-scale analysis of the adversarial subspaces of various neural network architectures, as well as estimating decision boundary geometry of non-smooth (e.g. ReLU or stochastic networks, Li et al., 2019a; Dapello et al., 2020) neural networks (which requires adding a costly Hessian approximation step).
Revised: We have identiﬁed an interesting avenue for future work in performing a large-scale analysis of the adversarial subspaces of various neural network architectures, as well as estimating decision boundary geometry of non-smooth (e.g. ReLU or stochastic, Li et al., 2019a; Dapello et al., 2020) neural networks (which requires adding a costly Hessian approximation step (Moosavi-Dezfooli et al., 2019)).

Original: Author Contributions Acknowledgments
Revised: Author Contributions Acknowledgments

Original: A Boundary curvature analysis
Revised: A Boundary curvature analysis

Original: The following deﬁnitions are common.
Revised: The following deﬁnitions are common.

Original: Our notation follows Lee (2013; 2018) for diﬀerential geometry, and Magnus & Neudecker (1999) for linear algebra.
Revised: Our notation follows Lee (2013; 2018) for diﬀerential geometry, and Magnus & Neudecker (1999) for linear algebra.

Original: A. Introduction
Revised: A. Introduction

Original: In order to derive our main curvature expressions for a graph submanifold, we ﬁrst introduce some notation and recall some concepts.
Revised: In order to derive our main curvature expressions for a graph submanifold, we ﬁrst introduce some notation and recall some concepts.

Original: Tangent Spaces
Revised: Tangent Spaces

Original: If M is an n -dimensional smooth manifold then the tangent space at p ∈ M is a vector space T p M of derivations at p , which is isometric to R n .
Revised: If M is an n -dimensional smooth manifold then the tangent space at p ∈ M is a vector space T p M of derivations at p , which is isometric to R n .

Original: In particular, T p M is the set of directional derivative operators acting on the smooth real functions f : M → R .
Revised: In particular, T p M is the set of directional derivative operators acting on the smooth real functions f : M → R .

Original: The disjoint union of all of the tangent spaces is the tangent bundle T M def = {{ p } × T p M | p ∈ M } .
Revised: The disjoint union of all of the tangent spaces is the tangent bundle T M def = {{ p } × T p M | p ∈ M } .

Original: If we equip M with a smooth, bilinear, positive deﬁnite, symmetric form g : T M × T M → R then ( M, g ) is said to be a Riemannian manifold and g the
Revised: If we equip M with a smooth, bilinear, positive deﬁnite, symmetric form g : T M × T M → R then ( M, g ) is said to be a Riemannian manifold and g the

Original: Riemannian metric tensor .
Revised: Riemannian metric tensor .

Original: When it is unambiguous, denote by Γ( E ) the smooth mappings M → E (for some topological space E ).
Revised: When it is unambiguous, denote by Γ( E ) the smooth mappings M → E (for some topological space E ).

Original: Most often we will refer to X ( M ) def = Γ(T M ), and represent elements of X ( M ) with the letters X, Y, Z .
Revised: Most often we will refer to X ( M ) def = Γ(T M ), and represent elements of X ( M ) with the letters X, Y, Z .

Original: Diﬀerentials If M and N are manifolds and F : M → N is diﬀerentiable, then the diﬀerential of F at p ∈ M is the linear mapping denoted d F p : T p M → T p N , where the action on tangent vectors is deﬁned by
Revised: Diﬀerentials If M and N are manifolds and F : M → N is diﬀerentiable, then the diﬀerential of F at p ∈ M is the linear mapping denoted d F p : T p M → T p N , where the action on tangent vectors is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: That is, the rate of change of the function f ◦ F in the direction v ∈ T p M at the point p .
Revised: That is, the rate of change of the function f ◦ F in the direction v ∈ T p M at the point p .

Original: As is common practice, we often omit the p subscripts for clarity.
Revised: As is common practice, we often omit the p subscripts for clarity.

Original: The linear operator d f is a linear functional when N in the above deﬁnition is R , and at each p
Revised: The linear operator d f is a linear functional when N in the above deﬁnition is R , and at each p

Original: ∈
Revised: ∈

Original: M there exists a vector grad f p ∈ T p .
Revised: M there exists a vector grad f p ∈ T p .

Original: So that
Revised: So that

Original: [Equation]
Revised: [Equation]

Original: T p M : d f p ( v ) = g (grad f p , v ) , and grad f ∈ X ( M ) is the associated vector ﬁeld.
Revised: and grad f ∈ X ( M ) is the associated vector ﬁeld.

Original: Connections A connection is an operator ∆ : X ( M ) × X ( M ) → X ( M ), linear in its ﬁrst argument, and satisfying a product rule in its second.
Revised: Connections A connection is an operator ∆ : X ( M ) × X ( M ) → X ( M ), linear in its ﬁrst argument, and satisfying a product rule in its second.

Original: Surprisingly, a connection uniquely deﬁnes a connection in each tensor bundle, 3 (which we overload with ∆ ) (Lee, 2018,
Revised: Surprisingly, a connection uniquely deﬁnes a connection in each tensor bundle, 3 (which we overload with ∆ ) (Lee, 2018,

Original: Prop. 4.15).
Revised: Prop. 4.15).

Original: Importantly for us, we have ∆ X f = Xf for all X ∈ X ( M ) (for any connection).
Revised: Importantly for us, we have ∆ X f = Xf for all X ∈ X ( M ) (for any connection).

Original: A Riemannian manifold always posesses a Levi–Cevita connection (Lee, 2018, §5), denoted by ∇ , which is the unique connection satisfying
Revised: A Riemannian manifold always posesses a Levi–Cevita connection (Lee, 2018, §5), denoted by ∇ , which is the unique connection satisfying

Original: [Equation]
Revised: [Equation]

Original: To f ∈ C 2 ( M ) we associate the Hessian operator ∇ 2 f , which is a symmetric (0 , 2)-tensor ﬁeld (Lee, 2018, Ex. 4.22) with ∇ 2 f
Revised: To f ∈ C 2 ( M ) we associate the Hessian operator ∇ 2 f , which is a symmetric (0 , 2)-tensor ﬁeld (Lee, 2018, Ex. 4.22) with ∇ 2 f

Original: ( X, Y ) def = ∇ X d f ( Y ) for X, Y ∈ X ( M ).
Revised: ( X, Y ) def = ∇ X d f ( Y ) for X, Y ∈ X ( M ).

Original: We have (via
Revised: We have (via

Original: Lee, 2018, Prop. 4.21)
Revised: Lee, 2018, Prop. 4.21)

Original: [Equation]
Revised: [Equation]

Original: Using the compatibility of ∇ with the metric we arrive at the following expression for the
Revised: Using the compatibility of ∇ with the metric we arrive at the following expression for the

Original: Hessian, which will be useful later
Revised: Hessian, which will be useful later

Original: [Equation]
Revised: [Equation]

Original: A.1. Submanifolds
Revised: A.1. Submanifolds

Original: If ( M, g ), ( ˜ M, ˜ g ) are two Riemanian manifolds with M ⊆ ˜ M , then an injection ι : M , → ˜ M is called an inclusion mapping .
Revised: If ( M, g ), ( ˜ M, ˜ g ) are two Riemanian manifolds with M ⊆ ˜ M , then an injection ι : M , → ˜ M is called an inclusion mapping .

Original: If the inclusion is a smooth isometry, that is
Revised: If the inclusion is a smooth isometry, that is

Original: [Equation]
Revised: [Equation]

Original:  equivalently, g = ι ∗ ˜ g , then M is called a Riemanian submanifold of ˜ M .
Revised: [Equation] ι ( v )) , equivalently, g = ι ∗ ˜ g , then M is called a Riemanian submanifold of ˜ M .

Original: Using the inclusion map diﬀerential, we embed T M in T ˜ M .
Revised: Using the inclusion map diﬀerential, we embed T M in T ˜ M .

Original: If M , ˜ M are of dimensions m < n , then T p M occupies an m dimensional subspace of T p ˜ M for each p ∈ M .
Revised: If M , ˜ M are of dimensions m < n , then T p M occupies an m dimensional subspace of T p ˜ M for each p ∈ M .

Original: The normal bundle is formed from the orthogonal compliment of T p M at each p ∈ M , and is deﬁned in a similar way to the tangent bundle, denoted N M and having a dimension of n − m .
Revised: The normal bundle is formed from the orthogonal compliment of T p M at each p ∈ M , and is deﬁned in a similar way to the tangent bundle, denoted N M and having a dimension of n − m .

Original: For vectors v ∈ T
Revised: For vectors v ∈ T

Original: ˜ M
Revised: ˜ M

Original: we have the decomposition v = v ⊥ + v ⊥ with v ⊥ ∈ T M and v ⊥ ∈ N M .
Revised: we have the decomposition v = v ⊥ + v ⊥ with v ⊥ ∈ T M and v ⊥ ∈ N M .

Original: Some care must be taken when taking covariant derivatives in T M with the ambient connection, formally
Revised: Some care must be taken when taking covariant derivatives in T M with the ambient connection, formally

Original: [Equation]
Revised: [Equation]

Original: where the overlines denote arbitrary smooth extensions of d ι ( X ) and d ι ( Y ) to open neighbourhoods of M in ˜ M .
Revised: where the overlines denote arbitrary smooth extensions of d ι ( X ) and d ι ( Y ) to open neighbourhoods of M in ˜ M .

Original: Second Fundamental Form Let M ⊆
Revised: Second Fundamental Form Let M ⊆

Original: ˜
Revised: ˜

Original: M be an embedded sumbanifold.
Revised: M be an embedded sumbanifold.

Original: The second fundamental form is the (bilinear) mapping II : X ( M ) × X
Revised: The second fundamental form is the (bilinear) mapping II : X ( M ) × X

Original: ( M ) → Γ(N M ) with II( X, Y ) def = (cid:0)
Revised: ( M ) → Γ(N M ) with II( X, Y ) def = (cid:0)

Original: ˜ ∇ X Y (cid:1) ⊥ .
Revised: ˜ ∇ X Y (cid:1) ⊥ .

Original: If we equip M with a normal ﬁeld N ∈ Γ(N M ), then the scalar second fundamental form is
Revised: If we equip M with a normal ﬁeld N ∈ Γ(N M ), then the scalar second fundamental form is

Original: [Equation]
Revised: [Equation]

Original: For the remainder we assume N is ﬁxed.
Revised: For the remainder we assume N is ﬁxed.

Original: The shape operator s : X ( M ) → X ( M ) is the endomorphism obtained from II N that satisﬁfes
Revised: The shape operator s : X ( M ) → X ( M ) is the endomorphism obtained from II N that satisﬁfes

Original: [Equation]
Revised: [Equation]

Original: 6) that is, by raising an index.
Revised: (6) that is, by raising an index.

Original: Curvature Let s be the shape operator associated to a normal ﬁeld on an embedded submanifold M of dimension n .
Revised: Curvature Let s be the shape operator associated to a normal ﬁeld on an embedded submanifold M of dimension n .

Original: Then at p ∈ M , the (real) eigenvalues of s are called principal curvatures and the eigenvectors are called principal directions at p .
Revised: Then at p ∈ M , the (real) eigenvalues of s are called principal curvatures and the eigenvectors are called principal directions at p .

Original: The Gaussian curvature is the product of the principal curvatures, det( s ), and the mean curvature is their average, 1 n tr( s ).
Revised: The Gaussian curvature is the product of the principal curvatures, det( s ), and the mean curvature is their average, 1 n tr( s ).

Original: Assume ( M, g ) and ( ˜ M, ˜ g ) are Riemannian manifolds with an inclusion mapping ι : M , → ˜ M and M has the submanifold structure.
Revised: Assume ( M, g ) and ( ˜ M, ˜ g ) are Riemannian manifolds with an inclusion mapping ι : M , → ˜ M and M has the submanifold structure.

Original: Suppose ˜ M posesses a shape operator (or any other linear endomorphism)
Revised: Suppose ˜ M posesses a shape operator (or any other linear endomorphism)

Original: ˜ s : T ˜ M → T ˜ M .
Revised: ˜ s : T ˜ M → T ˜ M .

Original: The submanifold structure then gives us an operator ι ∗ ˜ s : T M → T M , the pullback of s by ι , via
Revised: The submanifold structure then gives us an operator ι ∗ ˜ s : T M → T M , the pullback of s by ι , via

Original: [Equation]
Revised: [Equation]

Original: Remark 1 (Euclidean subspace) .
Revised: Remark 1 (Euclidean subspace) .

Original: We consider the special case of the Riemannian manifold
Revised: We consider the special case of the Riemannian manifold

Original: R m .
Revised: R m .

Original: Suppose P : R n → R m is orthogonal ( n ≤ m ) and let s : R m → R m be a linear operator.
Revised: Suppose P : R n → R m is orthogonal ( n ≤ m ) and let s : R m → R m be a linear operator.

Original: Then P T : R n , → R m is an inclusion map and the subspace pullback of s to R n is 
Revised: Then P T : R n , → R m is an inclusion map and the subspace pullback of s to R n is (cid:0) P T

Original: [Equation]
Revised: [Equation]

Original:  = PsP T .
Revised: ∗ s = PsP T .

Original: We refer to the curvature quantities and directions of (cid:0) P T (cid:1) ∗ s with the suspace preﬁx.
Revised: We refer to the curvature quantities and directions of (cid:0) P T (cid:1) ∗ s with the suspace preﬁx.

Original: That is, subspace principal curvature , subspace Gaussian curvature , and so on.
Revised: That is, subspace principal curvature , subspace Gaussian curvature , and so on.

Original: A. Hypersurfaces
Revised: A. Hypersurfaces

Original: Let ( M, g ) be an embedded submanifold of ( ˜ M, ˜ g ).
Revised: Let ( M, g ) be an embedded submanifold of ( ˜ M, ˜ g ).

Original: Suppose F : U ⊆
Revised: Suppose F : U ⊆

Original: ˜
Revised: ˜

Original: M
Revised: M

Original: → R is a local deﬁning function for M , that is, M ∩ U = F − 1 ( { 0 } ) for an open U ⊆ M .
Revised: → R is a local deﬁning function for M , that is, M ∩ U = F − 1 ( { 0 } ) for an open U ⊆ M .

Original: Choose arbitrary
Revised: Choose arbitrary

Original: X, Y ∈ X ( M ).
Revised: X, Y ∈ X ( M ).

Original: From the connection product rule (Lee, 2018, p. 89)
Revised: From the connection product rule (Lee, 2018, p. 89)

Original: [Equation]
Revised: [Equation]

Original: Whence
Revised: Whence

Original: [Equation]
Revised: [Equation]

Original: The zero in the underbrace follows because grad F ∈ N M and Y ∈ T M .
Revised: The zero in the underbrace follows because grad F ∈ N M and Y ∈ T M .

Original: The Weingarten equation (Lee, 2018, Thm. 8.13(c)) yields sX = − ˜ ∇ X N , giving us
Revised: The Weingarten equation (Lee, 2018, Thm. 8.13(c)) yields sX = − ˜ ∇ X N , giving us

Original: [Equation]
Revised: [Equation]

Original: A.2. The Curvature of a Graph Manifold
Revised: A.2. The Curvature of a Graph Manifold

Original: The notation and conventions of this section closely follow (Lee, 2018, §8).
Revised: The notation and conventions of this section closely follow (Lee, 2018, §8).

Original: Let M be the graph of a smooth function f : R n → R , that is, M = { ( x, f ( x )) | x ∈ R n } .
Revised: Let M be the graph of a smooth function f : R n → R , that is, M = { ( x, f ( x )) | x ∈ R n } .

Original: Since f is smooth
Revised: Since f is smooth

Original: M is indeed a manifold (Lee, 2013, Ex. 1.30, p. 20).
Revised: M is indeed a manifold (Lee, 2013, Ex. 1.30, p. 20).

Original: We regard ( M, g ) as a Riemannian submanifold of ( R n +1 , h· , ·i ), with the inclusion map ι : M , → R n +1 .
Revised: We regard ( M, g ) as a Riemannian submanifold of ( R n +1 , h· , ·i ), with the inclusion map ι : M , → R n +1 .

Original: The function F : R n +

Revised: The function F : R n +

Original: → R is a global deﬁning function for M with F ( x, t ) def = f ( x ) − t .
Revised: → R is a global deﬁning function for M with F ( x, t ) def = f ( x ) − t .

Original: Denote by ∇ f the vector of partial derivatives of f , and by H f and H F , the matrices of second order partial derivatives of F and f , all with respect to usual coordinate frames, ( x i ), on R n and R n +1 .
Revised: Denote by ∇ f the vector of partial derivatives of f , and by H f and H F , the matrices of second order partial derivatives of F and f , all with respect to usual coordinate frames, ( x i ), on R n and R n +1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: At p ∈ M the diﬀerential d ι p has the Jacobian matrix
Revised: At p ∈ M the diﬀerential d ι p has the Jacobian matrix

Original: [Equation]
Revised: [Equation]

Revised: Id

Original:  We commit the mild sin of identifying d ι p with its Jacobian and X, Y ∈ T ˜ M with their parameterizations under the coordinate frame, so that we may use the notation of matrix multiplication.
Revised: [Equation] p [Equation] We commit the mild sin of identifying d ι p with its Jacobian and X, Y ∈ T ˜ M with their parameterizations under the coordinate frame, so that we may use the notation of matrix multiplication.

Original: In dropping the p -subscripts, the left inverse, (d ι ) − 1 , is has the parameterization
Revised: In dropping the p -subscripts, the left inverse, (d ι ) − 1 , is has the parameterization

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: whence
Revised: whence

Original: [Equation]
Revised: [Equation]

Original: We can now compute the shape operator of M associated to the unit normal ﬁeld grad F/ | grad F | .
Revised: We can now compute the shape operator of M associated to the unit normal ﬁeld grad F/ | grad F | .

Original: Pick arbitrary X, Y ∈ T M .
Revised: Pick arbitrary X, Y ∈ T M .

Original: From (8) there is
Revised: From (8) there is

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and, using the bilinearity of of g we have
Revised: and, using the bilinearity of of g we have

Original: [Equation]
Revised: [Equation]

Original: This, together with (9) yields the shape operator:
Revised: This, together with (9) yields the shape operator:

Original: [Equation]
Revised: [Equation]

Original: Whence the Gaussian
Revised: Whence the Gaussian

Original: and mean curvature are the scalar ﬁelds
Revised: and mean curvature are the scalar ﬁelds

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: A.2. The Curvature of a Level-Set Manifold
Revised: A.2. The Curvature of a Level-Set Manifold

Original: We now consider the case where M is the level-set of a smooth function f : R n → R so that for some c ∈ R we have M = { x ∈ R n | f ( x )
Revised: We now consider the case where M is the level-set of a smooth function f : R n → R so that for some c ∈ R we have M = { x ∈ R n | f ( x )

Original: = c } .
Revised: = c } .

Original: The following theorem can be used to ensure the level-sets of f do indeed posses a manifold structure, which is a simpliﬁcation of the constant rank level-set theorem (Lee, 2013, Thm. 5.12) for our setting.
Revised: The following theorem can be used to ensure the level-sets of f do indeed posses a manifold structure, which is a simpliﬁcation of the constant rank level-set theorem (Lee, 2013, Thm. 5.12) for our setting.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f : R n → R be a smooth function having constant rank r .
Revised: Let f : R n → R be a smooth function having constant rank r .

Original: Then the level-sets of f are all smooth manifolds of codimension r .
Revised: Then the level-sets of f are all smooth manifolds of codimension r .

Original: We use the same construction as Lee (2013, Ex. 1.32, §5) to calculate the Gaussian and mean curvature at a point ( x 0 , y 0 ) ∈
Revised: We use the same construction as Lee (2013, Ex. 1.32, §5) to calculate the Gaussian and mean curvature at a point ( x 0 , y 0 ) ∈

Original: ( R n − 1 × R ) ∩ M .
Revised: ( R n − 1 × R ) ∩ M .

Original: From the implicit function theorem (Lee, 2013, Thm.
Revised: From the implicit function theorem (Lee, 2013, Thm.

Original: C.40) there are open connected neighbourhoods U 

Revised: C.40) there are open connected neighbourhoods U

Original: × V 0 ⊆
Revised: × V 0 ⊆

Original: R n − 1 × R with ( x 0 , y 0 ) ∈
Revised: R n − 1 × R with ( x 0 , y 0 ) ∈

Original: U 0 × V 0 and a smooth function g : U 0 → Vso that ( U 0 × V 0 )
Revised: U 0 × V 0 and a smooth function g : U 0 → Vso that ( U 0 × V 0 )

Original: ∩ M is the graph of g .
Revised: ∩ M is the graph of g .

Original: That is, ( U 0 × V 0 )
Revised: That is, ( U 0 × V 0 )

Original: ∩ M = (cid:8) ( x, g ( x )) (cid:12)(cid:12) x ∈ U 0 ⊆
Revised: ∩ M = (cid:8) ( x, g ( x )) (cid:12)(cid:12) x ∈ U 0 ⊆

Original: R n − 1 (cid:9) .
Revised: R n − 1 (cid:9) .

Original: Consequentially the gradient of g has the structure
Revised: Consequentially the gradient of g has the structure

Original: [Equation]
Revised: [Equation]

Original: Let ∂ i def = ∂ / ∂ x j and ∂ 2 ij def = ∂ 2 / ∂ x i ∂ x j .
Revised: Let ∂ i def = ∂ / ∂ x j and ∂ 2 ij def = ∂ 2 / ∂ x i ∂ x j .

Original: Then for i, j ∈ [ n − 1] and x ∈ U
Revised: Then for i, j ∈ [ n − 1] and x ∈ U

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: , (14) and 
Revised: [Equation] and [Equation]

Original: [Equation]
Revised: [Equation]

Original: (15) where every function on the right hand side is evaluated at either the point x or ( x, g ( x ))
Revised: where every function on the right hand side is evaluated at either the point x or ( x, g ( x ))

Original: (depending on the function).
Revised: (depending on the function).

Original: 5 If we let ¯ ∇ def = ( ∂ / ∂ x 1 , . . . ,
Revised: 5 If we let ¯ ∇ def = ( ∂ / ∂ x 1 , . . . ,

Original: ∂ / ∂ x n − 1 ) T , and ¯H be, similarly, the ( n − 1) × ( n − 1) matrix of cross-partial derivatives, there are the more compact expressions for (14) and (15):
Revised: ∂ / ∂ x n − 1 ) T , and ¯H be, similarly, the ( n − 1) × ( n − 1) matrix of cross-partial derivatives, there are the more compact expressions for (14) and (15):

Original: [Equation]
Revised: [Equation]

Original: [Table] .0% / 0.0% / 0.0% / 0.0% / 0.0% / 36.33% 54.5% 59.04% 61.76% 65.4% 69.03% 69.94% 70.85% 72.66% 73.57% 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 0.0% / 2.72% 5.45% 10.9% 12.72% 13.62% 14.53% 15.44% 15.44% 16.35% 17.26% and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where the sum ∇ g + ∇ g T is to be interpreted as the matrix of pairwise sums of the elements of ∇ g : (cid:0) ∇ g + ∇ g T (cid:1) ij def = ∂ g/ ∂ x i + ∂ g/ ∂ x j .
Revised: where the sum ∇ g + ∇ g T is to be interpreted as the matrix of pairwise sums of the elements of ∇ g : (cid:0) ∇ g + ∇ g T (cid:1) ij def = ∂ g/ ∂ x i + ∂ g/ ∂ x j .

Original: Then, as in Section A.2.1, F ( x ) def = g ( x 1 , . . .
Revised: Then, as in Section A.2.1, F ( x ) def = g ( x 1 , . . .

Original: , x n − 1 ) − x n is a local deﬁning function for M on ( U 0 × V 0 )
Revised: , x n − 1 ) − x n is a local deﬁning function for M on ( U 0 × V 0 )

Original: ∩ M .
Revised: ∩ M .

Original: The local graph parameterization provided by g is φ : U 

Revised: The local graph parameterization provided by g is φ : U

Original: → M with φ ( x ) def = ( x, g ( x )), and its diﬀerential has the parameterizationd φ
Revised: → M with φ ( x ) def = ( x, g ( x )), and its diﬀerential has the parameterization

Original: = (Id ∇ g ) T .
Revised: [Equation]

Original: Finally, mutatis mutandis , the shape operator is given by (11) and the Gaussian and mean curvature can be computed as in (12).
Revised: Finally, mutatis mutandis , the shape operator is given by (11) and the Gaussian and mean curvature can be computed as in (12).

Original: Remark 2 (Numerical considerations) .
Revised: Remark 2 (Numerical considerations) .

Original: In order to compute the level-set curvature we needed to apply the implicit function theorem.
Revised: In order to compute the level-set curvature we needed to apply the implicit function theorem.

Original: However, it may be diﬃcult numerically to stabilize the inverse operation in (13).
Revised: However, it may be diﬃcult numerically to stabilize the inverse operation in (13).

Original: To remedy this problem, one may apply a change of basis before computing the quantities (14) and (15).
Revised: To remedy this problem, one may apply a change of basis before computing the quantities (14) and (15).

Original: Let B ∈ R n × n be an orthogonal change of basis operator.
Revised: Let B ∈ R n × n be an orthogonal change of basis operator.

Original: Then we apply all our operations to B T M def = (cid:8) B T x (cid:12)(cid:12) x ∈ M (cid:9) .
Revised: Then we apply all our operations to B T M def = (cid:8) B T x (cid:12) (cid:12) x ∈ M (cid:9) .

Original: Diﬀerentiating ˜ f
Revised: Diﬀerentiating ˜ f

Original: def = f ◦ B T yields the revised quantities ∀ x ∈ R n : ∇ ˜ f
Revised: def = f ◦ B T yields the revised quantities ∀ x ∈ R n : ∇ ˜ f

Original: ( x ) = B · ∇ f ( B T x ) and
Revised: ( x ) = B · ∇ f ( B T x ) and

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: B Dataset and models
Revised: B Dataset and models

Original: For the adversarial subspace analysis we use two identical ResNet-50  (He et al., 2016) architectures.
Revised: For the adversarial subspace analysis we used two identical ResNet-50 architectures (He et al., 2016) and two identical WideResNet-70 architectures (Gowal et al., 2020).

Original: The ﬁrst model was trained on the unmodiﬁed CIFAR-10 dataset (Krizhevsky & Hinton, 2009).
Revised: For each architecture, one model was trained on the unmodiﬁed CIFAR-10 dataset (Krizhevsky & Hinton, 2009).

Original: For the second model we additionally incorporated adversarial examples in the training process, widely known as adversarial training.
Revised: For the second model we additionally incorporated adversarial examples in the training process, widely known as adversarial training.

Original: A training perturbation size (cid:15) of 0.5 was used.
Revised: A training perturbation size (cid:15) of 0.5 was used.

Original: To perform the curvature analysis, we used twice-diﬀerentiable ELU activation functions (Clevert et al., 2015) for both models, which had little impact on the clean or adversarial accuracies.
Revised: To perform the curvature analysis, we used twice-diﬀerentiable ELU activation functions (Clevert et al., 2015) for both models, which had little impact on the clean or adversarial accuracies.

Original: In the analysis we include 100 CIFAR-10 test images, class-balanced, all unseen during training, and correctly classiﬁed by both models.
Revised: In the analysis we include 100 CIFAR-10 test images, class-balanced, all unseen during training, and correctly classiﬁed by both models.

Original: The optimization algorithm stopped after ﬁnding 50 orthogonal adversarial vectors.
Revised: The optimization algorithm stopped after ﬁnding 50 orthogonal adversarial vectors for all 100 test images.

Original: For all test image we found 50 dimensions on both models.

Original: For the curvature analysis we used a subset of 50 class-balanced images andadversarial directions per image.
Revised: For the curvature analysis we used a subset of 50 class-balanced images and 10 adversarial directions per image.

Original: Additionally, we only considered one seed per CIFAR-10 model because we found that the analysis on the MNIST trained models had little variation across seeds (see Appendix D). [Figure]
Revised: Additionally, we only considered one seed per CIFAR-10 model because we found that the analysis on the MNIST trained models had little variation across seeds (see Appendix E).

Original: The accuracy of both models at diﬀerent perturbation lengths (cid:15) and dimensions is shown in
Revised: The accuracy of both models at diﬀerent perturbation lengths (cid:15) and dimensions is shown in

Original: Table 2.
Revised: Table 2.

Original: Here, the ﬁrst line of each row corresponds to the accuracy of the naturally trained model, whereas the second line is referring to the adversarially trained model.
Revised: Here, the ﬁrst line of each row corresponds to the accuracy of the naturally trained model, whereas the second line is referring to the adversarially trained model.

Original: We repeated many of the experiments on the MNIST dataset (LeCun et al., 1998) and found largely consistent results (see Appendix G).
Revised: We repeated many of the experiments on the MNIST dataset (LeCun et al., 1998) and found largely consistent results (see Appendix D).

Original: C Additional CIFAR analysis and supplementary figures
Revised: C Additional CIFAR-10 results

Original: [Figure]

Original: Here we show additional analysis to complement the above described ﬁndings on  CIFAR-10 data.
Revised: Here we show additional analysis to complement our ﬁndings on the CIFAR-10 dataset.

Original: As was described earlier, we uniformly randomly sampled directions within adversarial subspaces to measure the distance to the decision boundary.

Original: Figure 7 shows how many of the samples pointed to decision boundaries that were outside of the allowed input pixel bounds.

Original: Figure 6 shows ﬁve adversarially perturbed images and their corresponding original image found for the two investigated models.
Revised: Figure 6 shows ﬁve adversarially perturbed images and their corresponding original image found for the two ResNet-50 models.

Original: We can see that that the perturbation length is increasing with dimension number.
Revised: We can see that that the perturbation length is increasing with dimension number.

Original: Interestingly,  the adversarially trained model seems to have more semantically meaningful adversarials, e.g. the images of boats in the top row, misclassiﬁed as cars, are perturbed to have elements looking like wheels.
Revised: Interestingly, while the adversarial categories may not be semantically adjacent to the clean classes, the adversarially trained model seems to require more semantically meaningful perturbations, e.g. the images of boats in the top row, misclassiﬁed as cars, are perturbed to have elements looking like wheels.

Original: We can also see that there is no clear, explicit ‘ 2 -length of an adversarial attack that makes an image ambiguous for a human observer.
Revised: We can also see that there is no clear, consistent ‘ 2 -length of an adversarial attack that makes an image ambiguous for a human observer.

Original: For example, the robust model example in the ﬁrst (ship) row of the adversarial 11 column pair has a  a d v e r s a r i a l v e c t o r l e n g t h natural robust   distance to decision boundary
Revised: For example, the robust model example in the ﬁrst (ship) row of the adversarial 11 column pair has a perturbation length of 1.12 and could be mistaken as a car on a ﬁrst glance.

Original: d i m e n s i o n o f a d v e r s a r i

Original: a l s p a c e perturbation length of 1.12 and could be mistaken as a car on a ﬁrst glance.

Original: In contrast, the robust model example in the fourth (horse) row of the adversarial 1 column pair is still clearly identiﬁable as a horse with a larger perturbation length of 1.2.
Revised: In contrast, the robust model example in the fourth (horse) row of the adversarial 1 column pair is still clearly identiﬁable as a horse with a larger perturbation length of 1.2.

Original: This illustrates that the direction of the perturbation is as important as the  perturbation magnitude itself, as was previously reported by (Goodfellow et al., 2014; Ilyas et al., 2019), although assessing the “perceptual distance” is an open area of research. 
Revised: This illustrates that the direction of the perturbation is as important as the [Figure] perturbation magnitude itself, as was previously reported by (Goodfellow et al., 2014; Ilyas et al., 2019), although assessing the “perceptual distance” is an open area of research. [Figure]

Revised: As was described earlier, we uniformly randomly sampled directions within adversarial subspaces to measure the distance to the decision boundary.

Revised: Figure 7 shows how many of the samples pointed to decision boundaries that were outside of the allowed input pixel bounds.

Original: Figure 8 shows that perturbation lengths are increasing sublinearly and monotonically with increasing number of adversarial dimensions.
Revised: Figure 8 shows that perturbation lengths are increasing sublinearly and monotonically with increasing number of adversarial dimensions.

Original: For the naturally trained model there is also a much larger increase of adversarial dimension with increasing distance to distance to decision boundary.
Revised: For the naturally trained model there is also a much larger increase of adversarial dimension with increasing distance to distance to decision boundary.

Revised: This ﬁgure is different from e.g. Figure 2 in that it displays the lengths of the adversarial subspace axes, not the lengths of the random perturbations within these subspaces.

Original: Note that in the right panel of this ﬁgure both models plateau at 50 dimensions because we stopped the optimizer at this point.
Revised: Note that in the right panel of this ﬁgure both models plateau at 50 dimensions because we stopped the optimizer at this point.

Original: We also provide additional, and more detailed information on the distributions of distances to the decision boundary of randomly sampled vectors in adversarial subspaces in Figure 9.
Revised: We also provide additional, and more detailed information on the distributions of distances to the decision boundary of randomly sampled vectors in adversarial subspaces in Figure 9.

Revised: Each subplot shows a subspace of increasing dimensionality, ascending left-to-right, then top-to-bottom.

Original: We observe that the variance of the distances to the decision boundary of randomly sampled vectors is smoothly increasing with dimensionality.       
Revised: We observe that the variance of the distances to the decision boundary of randomly sampled vectors is smoothly increasing with dimensionality. [Figure] a d v e r s a r i a l v e c t o r l e n g t h natural robust   distance to decision boundary

Original: D Seed consistency
Revised: d i m e n s i o n o f a d v e r s a r i a l s p a c e [Figure] d i s t a n c e t o d e c

Revised: i s i o n b n d .

Revised: ResNetWideResNet-70dimension of adversarial space 1.

Revised: d i s t a n c e c oe ff

Revised: i c i e n t 10.

Revised: Additional model comparison

Revised: To show that our ﬁndings hold for diﬀerent models, we present results for two diﬀerent architectures: a ResNet-50 (He et al., 2016) and a WideResNet-70-16, of which the latter is the state-of-the-art adversarially trained network for CIFAR-10 images (Gowal et al., 2020).

Revised: Both architectures were trained with and without adversarial perturbations, resulting inmodels overall.

Revised: Figures 10 through 14 repeat analysis to compare natural and adversarial trained models, but instead compare models of diﬀerent architectures and constant training procedure.

Revised: Figures 10, 11, and 14b show a comparison of the two naturally trained models.

Revised: These ﬁgures indicate that there is little diﬀerence between the two model architectures regarding distance to the decision boundary and perturbation length over dimensions.

Revised: Figures 12, 13, and 14a show a comparison of the two adversarially trained models.

Revised: Here, the ﬁgures show that perturbation lengths as well as distances to the decision boundary are larger for the

Revised: WideResNet-70-16 compared to the ResNet-50.

Revised: This is expected, as the WideResNet has higher adversarial accuracy and thus larger perturbations are needed to fool it.

Revised: At the level of these aggregate statistics, the hyperparameter diﬀerences in adversarial training (Figure 12) have more of an eﬀect than the model architecture diﬀerences themselves (Figure 10), although all diﬀerences are much less pronounced than the diﬀerences between adversarially and naturally trained networks (Figure 2).

Revised: The box plots depicting the distances to the decision boundary in Figures 10 and 12 show that, for both architectures, the adversarial subspace is large, continuous, and increases sublinearly in volume as the dimensionality is increased.

Revised: The observation that this ﬁnding is agnostic to model architecture is further supported by the similarity in distributions of samples measuring the distance to decision boundary that can be seen on the right hand side of the same ﬁgures.

Revised: Thus, the observed trends and conclusions drawn from the analysis provided in this study hold across architectures.

Revised: D Additional MNIST results

Revised: We conducted the same experiments described in Section 3 on networks trained with the

Revised: MNIST dataset.

Revised: As a test set we used 500 images (50 per class), unseen in training and correctly labeled by all investigated models.

Revised: [Figure]

Revised: a d v e r s a r i a l v e c t

Revised: o r l e n g t h

Revised: ResNet

Revised: [Figure]

Revised: [Figure]

Revised: d i m e n s i o n o f a d v e r s a r i a l s p a c e

Revised: [Figure]

Revised: d i s t a n c e t o d e c

Revised: i s i o n b n d .

Revised: [Figure]

Revised: t a n c e c oe ff i

Revised: c i e n t

Revised: [Figure]

Revised: In Figure 18 we can see that perturbation lengths are much larger for MNIST than for

Revised: CIFAR-10, although the relative increase in distances for the robust network is still present.

Revised: The optimization was in general more diﬃcult with MNIST, likely due to the fact that the images all live very close to the allowable input bounds.

Revised: Speciﬁcally, the algorithm found less than 30 adversarial dimensions for more than 95% of the test images (Figure 22), to be compared with ﬁnding >50 dimensions for 100% of the CIFAR-10 test images.

Revised: That it is harder to fool the MNIST model can also be seen in Table 3: the accuracies of both models are much higher for the same perturbation lengths than the accuracies of the

Revised: CIFAR-10 models.

Revised: For the curvature analysis, on the other hand, we found that the overall values were larger on CIFAR-10.

Revised: Additionally, the adversarially trained model had more decision boundary curvature within adversarial subspaces on MNIST than on CIFAR-10.

Revised: We noticed that most of the MNIST adversarials lie in a thin regions between the decision boundary and the valid pixel box boundary (compare Figure 17 against Figure 3).

Revised: Figureshows that this leads to large rates of out of bounds samples when measuring the distance to the decision boundary.

Revised: For example, 90% of the time the decision boundary is outside the valid pixel range in the 6 dimensional adversarial subspace of the naturally trained model.

Revised: nth adversarial perturbation

Revised: [Figure]

Revised: a d v e r s a r i a l v e c t

Revised: o r l e n g t h

Revised: ResNetWideResNet-70distance to decision boundary     d i m e n s i o n o f a d v e r s a r i

Revised: a l s p a c e

Revised: [Figure]

Revised: t o f b o un d s

Revised: [Figure]

Revised: a

Revised: [Figure]

Revised: i o

Revised: ResNet

Revised: [Figure]

Revised: WideResNet-

Revised: [Figure]

Revised: [Figure]

Revised: dimension of adversarial sapce 0.

Revised: o u t

Revised: o f

Revised: [Figure]

Revised: o un

Revised: [Figure]

Revised: a

Revised: [Figure]

Revised: i o

Revised: ResNetWideResNet-70-

Revised: [Figure]

Revised: Dimension (cid:15)           98.59% / 98.59% / 98.59% / 98.59% / 98.59% / 98.59% / 98.59% / 98.59% / 98.59% / 98.59% / 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 0.5 90.51% / 97.41% / 98.0% / 98.39% / 98.39% / 98.39% / 98.37% / 98.35% / 98.29% / 98.23% / 97.77% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 60.53% / 90.31% / 95.83% / 97.01% / 97.4% / 97.58% / 97.94% / 97.86% / 97.69% / 97.51% / 94.03% 98.36% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 98.56% 1.77% / 18.34% / 47.91% / 71.58% / 83.54% / 88.51% / 91.63% / 93.0% / 93.8% / 93.55% / 68.99% 89.3% 95.01% 96.59% 97.57% 97.69% 97.73% 97.72% 98.56% 98.56% 0.0% / 0.0% / 0.2% / 4.14% / 15.05% / 26.41% / 32.43% / 38.61% / 37.46% / 32.74% / 0.59% 9.46% 27.99% 46.52% 65.77% 76.32% 82.59% 83.94% 85.42% 80.16% 0.0% / 0.0% / 0.0% / 0.39% / 1.98% / 7.86% / 10.45% / 18.94% / 16.48% / 16.55% / 0.0% 0.0% 0.2% 7.69% 24.69% 37.51% 47.63% 48.86% 54.54% 40.74%

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: d i s t a n c e t o d e c

Revised: i s i o n b n d .

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  We trained the natural and robust MNIST models with ﬁve diﬀerent seeds and extracted adversarial subspaces for each model to examine the inﬂuence of training seeds on our results.
Revised: [Figure] [Figure] We trained the natural and robust MNIST models with ﬁve diﬀerent seeds and extracted adversarial subspaces for each model to examine the inﬂuence of training seeds on our results.

Original: Figure 10 shows that there is slight variation in number of adversarial subspace directions found by the optimizer.
Revised: Figure 22 shows that there is slight variation in number of adversarial subspace directions found by the optimizer.

Original: However, the overall trend of bar heights is similar across seeds and there are no signiﬁcant outliers.
Revised: However, the overall trend of bar heights is similar across seeds and there are no significant outliers.

Original: To further validate our results, we repeated the experiment for Figure 14 (measuring the length of the adversarial vector as one increases dimensionality on the MNIST trained networks) across ﬁve diﬀerent seeds per training type.
Revised: To further validate our results we repeated the experiment for Figure 16 (measuring the length of the adversarial vector as one increases dimensionality on the MNIST trained networks) across ﬁve diﬀerent seeds per training type.

Original: The results are shown inﬁgure
Revised: The results are shown in

Original:  The variation across seeds for a single image is smaller than the variation across images for a single seed, which can be seen by comparing Figure 11 to Figure 14.
Revised: Figure 23. The variation across seeds for a single image is smaller than the variation across images for a single seed, which can be seen by comparing Figure 23 to Figure 16.

Original: In general, we also found that the result trends were consistent between MNIST and CIFAR networks (with diﬀerences noted in Appendix G).
Revised: In general, we also found that the results and trends were consistent between MNIST and CIFAR-10 networks (with diﬀerences noted in Appendix D).

Original: Thus, we considered one seed per CIFAR-10 model as suﬃciently representative.
Revised: Thus, we considered one seed per CIFAR-10 model as suﬃciently representative.

Original: [Figure]
Revised: We also investigated the inﬂuence of random number generation relevant for sampling within adversarial subspaces . As an empirical test, we varied the number of samples and measured the change in mean distance to the decision boundary as one increases the number of samples. This was repeated for 100 diﬀerent 50-dimensional subspaces (i.e. subspaces for 100 diﬀerent test images) with the naturally trained CIFAR network. Figure 21 shows that the absolute change decreases steadily to an average l2 distance of 0.004.

Original: [Figure]
Revised: Thus, the mean variation from 100 samples is much smaller than the general distance scales we are measuring throughout adv dir0. adv dir0. adv dir0. adv dir0.

Revised: adv dir0.

Revised: adv dir0.

Revised: adv dir0.

Original: seed         found dimensions{+adv dir0. adv dir0. adv dirpredicted class 
 adversarial dimension   a d v e r s a r i a l v e c t+}
Revised: {+adv dir0. adv dir0. adv dirpredicted class 
 adversarial dimension   a d v e r s a r i a l v e c t+}

Original: Robust models seed
Revised: o r l e n g t h natural robust 2. distance to decision boundary     a d v e r s a r i a l d i m e n s i o n the paper. We also measured the variation in distances across random seeds for the sampling process.

Original: [Figure]
Revised: To do this, we used 10 diﬀerent seeds to generate 100 samples in a 50-dimensional subspace corresponding to one test image. For each seed, we compute a mean distance, averaged across 100 samples, giving us 10 values. The mean distance across seeds was 0.0933, and the standard deviation was 0.0007. The standard deviation is less than 1% of the mean and thus suﬃciently small to consider an estimate with 100 samples as representative.

Original: E Orthogonality and clipping
Revised: F Comparisons to related work: Orthogonality and clipping

Original: One obstacle in ﬁnding proper adversarial subspaces spanned by orthogonal vectors is assuring that perturbations do not cause pixel values to be out of the valid input boundaries [0 , 1].
Revised: One obstacle in ﬁnding proper adversarial subspaces spanned by orthogonal vectors is assuring that perturbations do not cause pixel values to be out of the valid input boundaries [0 , 1].

Original: We noticed that many studies solve this problem by element wise pixel clipping.
Revised: We noticed that many studies solve this problem by element wise pixel clipping.

Original: Here, we show that pixel clipping is problematic when talking about the dimensionality of subspaces.
Revised: Here, we show that pixel clipping is problematic when talking about the dimensionality of subspaces.

Original: Element wise pixel clipping is a non-linear operation performed on a vector and can therefore change its direction in the input space and its relative angle to other, ﬁxed vectors.
Revised: Element wise pixel clipping is a non-linear operation performed on a vector and can therefore change its direction in the input space and its relative angle to other, ﬁxed vectors.

Original: To show that clipping has a non-negligible eﬀect, we investigated relative angles of basis vectors that span (sub-) spaces for 100 MNIST test images relevant for two studies.
Revised: To show that clipping has a non-negligible eﬀect, we investigated relative angles of basis vectors that span (sub-) spaces for 100 MNIST test images relevant for two studies.

Original: First, in their work, Tramèr et al.
Revised: First, in their work, Tramèr et al.

Original: (2017) propose the GAAS method.
Revised: (2017) propose the GAAS method.

Original: GAAS estimates the dimensionality of adversarial subspaces by deﬁning an orthogonal set of vectors aligned with an attack vector to an input image.
Revised: GAAS estimates the dimensionality of adversarial subspaces by deﬁning an orthogonal set of vectors aligned with an attack vector to an input image.

Original: However, before passing the orthogonal basis vectors to the model, a clipping operation is performed.
Revised: However, before passing the orthogonal basis vectors to the model, a clipping operation is performed.

Original: Figure 12 (left) shows that this clipping operation causes the input vectors to be non-orthogonal - orthogonality would constitute an angle of 90° or π 2 .
Revised: Figure 24 (left) shows that this clipping operation causes the input vectors to be non-orthogonal – orthogonality would constitute an angle of 90° or π 2 .

Original: In fact, for the 100 sample images we tested, there is not a single vector pair that is completely orthogonal.
Revised: In fact, for the 100 sample images we tested, there is not a single vector pair that is completely orthogonal.

Original: The author’s conclusions regarding the dimensionality of adversarial subspaces and transferable subspaces are therefore grounded on ill-deﬁned subspaces, spanned by non-orthogonal vectors.
Revised: The author’s conclusions regarding the dimensionality Original 6 (1.63) Natural 3 (3.56) Robust 5 (1.67)

Revised: Natural 8 (4.11)

Revised: Robust 3 (1.92)

Revised: Natural 2 (4.63)

Revised: Robust 2 (3.02)

Revised: Natural 4 (5.23)

Revised: Robust 7 (3.52)

Revised: Natural 0 (nan)

Revised: Robust 8 (0.95) 8 (1.85) 3 (1.96) 3 (1.90) 6 (2.31) 0 (2.49) 7 (2.50) 9 (2.94) 3 (3.12) 3 (3.21) 3 (0.86) 3 (2.28) 6 (1.21) 0 (2.51) 3 (1.54) 3 (4.18) 0 (1.55) 4 (4.58) 5 (2.18) 3 (7.63) 5 (1.36) 1 (2.12) 2 (1.50) 7 (2.38) 5 (2.30) 5 (3.18) 2 (2.82) 2 (3.42) 7 (3.39) 2 (4.13) 4 (0.88) 8 (3.33) 7 (1.48) 5 (3.77) 2 (1.48) 2 (3.84) 8 (1.96) 7 (4.44) 2 (2.73) 2 (4.98)

Revised: [Figure]

Revised: Adv

Revised: Adv

Revised: Adv

Revised: Adv of adversarial subspaces and transferable subspaces are therefore grounded on ill-deﬁned subspaces, spanned by non-orthogonal vectors

Original: In another study, He et al.
Revised: In another study, He et al.

Original: (2018) estimate the distance to the decision boundary for the complete input space of MNIST.
Revised: (2018) estimate the distance to the decision boundary for the complete input space of MNIST.

Original: For that, a basis of 784 vectors, spanning the whole input space, is randomly generated.
Revised: For that, a basis of 784 vectors, spanning the whole input space, is randomly generated.

Original: Again, the inputs are clipped before they are passed to the model leading to the same problem as described earlier.
Revised: Again, the inputs are clipped before they are passed to the model leading to the same problem as described earlier.

Original: And as before, there are no orthogonal vector pairs, and angels are accumulated at much smaller values than π 2 as can be seen in Figure 12 (right).
Revised: And as before, there are no orthogonal vector pairs, and angels are accumulated at much smaller values than π 2 as can be seen in Figure 24 (right).

Original: We therefore note that it is not guaranteed that the input space is evenly covered and that edges of the 784-dimensional hypercube (in the case of MNIST) are arguably overrepresented. 
Revised: We therefore note that it is not guaranteed that the input space is evenly covered and that edges of the 784-dimensional hypercube (in the case of MNIST) are arguably overrepresented. found dimensions

Original: [Figure]
Revised: [Figure]

Revised: Natural models seed         found dimensions

Revised: Robust models seed

Revised: [Figure]

Revised: [Figure]

Revised: natural robust

Original: F Additional related work on decision boundary curvature
Revised: G Comparisons to related work: Decision boundary curvature

Revised: Reference Derivation

Revised: Generating matrix

Revised: P

Original: Several works from Moosavi-Dezfooli and colleagues explore decision boundary curvature and its relationship to adversarial robustness (Fawzi et al., 2016; Moosavi-Dezfooli et al., 2017b; 2018; 2019; Moosavi-Dezfooli, 2019).
Revised: Dimension Poole et al., None P H Fij P |∇ F ij |

Revised: Id − ∇ F ij ∇ F T ij |∇ F ij |n × n

Revised: Moosavi-Dezfooli et al.,

Revised: None

Revised: P H F ij P

Revised: Not given n × n Moosavi-Dezfooli et al.,

Revised: None

Revised: Crossenetropy Hessiann × n

Revised: Moosavi-Dezfooli, 2019, p.

Revised: None

Revised: P H Fij P |∇ F ij |

Revised: Id −∇ F ij ∇ F T ij n × n

Revised: This work

Revised: Appendix A (11) using (15) -

Revised: [Equation]

Revised: This section provides a detailed comparison of our analysis against previously published works on the principal curvature analysis applied to decision boundaries.

Revised: We include tablefor a quick summary.

Revised: The use of diﬀerential geometry to understand the non-linear response properties of model neurons has a long history in computational neuroscience, with early contributions made by

Revised: Zetzsche & Barth (1990).

Revised: More recently, several works have speciﬁcally used Reimannian geometry to measure the principal curvature of response manifolds (Fawzi et al., 2016;

Revised: Poole et al., 2016; Moosavi-Dezfooli et al., 2018; 2019; Golden et al., 2019).

Revised: As we discuss in

Revised: [Figure]

Revised: Tramèr et al.

Revised: method

Revised: [Figure]

Revised: He et al.

Revised: method

Revised: Appendix A, certain smoothness assumptions are required for such an analysis.

Revised: Our present study, as well Poole et al.

Revised: (2016) and Golden et al.

Revised: (2019), avoid this problem by using twice-diﬀerentiable activation functions.

Revised: In the case of Golden et al.

Revised: (2019), this was done without retraining the network, while Poole et al.

Revised: (2016) used nets with random weights.

Revised: We instead trained the twice-diﬀerentiable networks from scratch.

Revised: Moosavi-Dezfooli et al.

Revised: used a ﬁnite-diﬀerence approximation to avoid the problem (described in Moosavi-Dezfooli et al., 2019, , eq 1 and surrounding discussion).

Revised: Another high-level diﬀerence, discussed in detail in Appendix A, is the choice of measuring the response graph curvature (which is n dimensional for n dimensional inputs) or the level-set curvature (which is n − 1 dimensional for n dimensional inputs) of a given function.

Revised: This diﬀerence was discussed in (Golden et al., 2019, , at the end of Section 4 and in Figure 4), although their technical derivations are limited only to the graph formulation.

Revised: The other studies did not explicitly discuss this diﬀerence.

Revised: Importantly, though, the decision boundary is the level-set of the logit diﬀerence function, as deﬁned in Equation 2.

Revised: Several works from Moosavi-Dezfooli and colleagues explore decision boundary curvature and its relationship to adversarial robustness (Fawzi et al., 2016; Moosavi-Dezfooli et al., 2017b; 2018; 2019; Moosavi-Dezfooli, 2019).

Original: Within these works they provide an inspired perspective on explaining adversarial robustness in terms of the curvature of decision boundaries.
Revised: Within these works they provide an inspired perspective on explaining adversarial robustness in terms of the curvature of decision boundaries.

Original: Almost all of the work is with respect to universal adversarial perturbations, which share a common subspace (Moosavi-Dezfooli et al., 2017a), although Fawzi et al.
Revised: Almost all of the work is with respect to universal adversarial perturbations, which share a common subspace (Moosavi-Dezfooli et al., 2017a), although Fawzi et al.

Original: (2016) alternatively look at random subspaces.
Revised: (2016) alternatively look at random subspaces.

Original: The universal  perturbation subspaces diﬀer from our adversarial subspaces in that theirs are deﬁned by orthogonal directions that transfer well across inputs(in terms of proximity to the decision boundary) instead of those that point to the nearest decision boundary for a given input.
Revised: Their universal adversarial perturbation subspaces diﬀer from our adversarial subspaces in that theirs are deﬁned by orthogonal directions that transfer well across inputs instead of those that point to the nearest decision boundary for a given input.

Original: Despite these diﬀerences, the stated quantity of interest, that is, the principle curvatures at a point on the decision boundary is the same in their work as well as ours.
Revised: Despite these diﬀerences, the stated quantity of interest, that is, the principle curvatures at a point on the decision boundary is the same in their work as well as ours.

Original: However, each of these papers have a slightly diﬀerent analytic formula for estimating the curvature of the boundary within a given subspace (Fawzi et al., 2016; Moosavi-Dezfooli et al., 2017b), or in the high-dimensional input space (Moosavi-Dezfooli et al., 2018; 2019).
Revised: However, each of these papers have a slightly diﬀerent analytic formula for estimating the curvature of the boundary within a given subspace (Fawzi et al., 2016; Moosavi-Dezfooli et al., 2017b), or in the high-dimensional input space (Moosavi-Dezfooli et al., 2018; 2019).

Original: We found the most complete description to be given in Moosavi-Dezfooli (2019), which summarizes a majority of the earlier work and applies the decision boundary curvature analysis to both subspaces and the high-dimensional input space.
Revised: We found the most complete description to be given in Moosavi-Dezfooli (2019), which summarizes a majority of the earlier work and applies the decision boundary curvature analysis to both subspaces and the high-dimensional input space.

Original: Therein, Moosavi-Dezfooli (2019, Section. 7.4) state the following formula, without derivation, for the shape operator (in the notation of Section 2.2):
Revised: Therein, Moosavi-Dezfooli (2019, Section. 7.4) state the following formula, without derivation, for the shape operator (in the notation of Section 2.2):

Original: [Equation]
Revised: [Equation]

Original: where H F ij is the Hessian of the boundary function, F ij ( x ) def = f i ( x ) − f j ( x ),  P def =
Revised: where H F ij is the Hessian of the boundary function, F ij ( x ) def = f i ( x ) − f j ( x ), and P def =

Original: Id −∇ F ij ( x ) ∇ F ij ( x ) T is described as a projection operator.
Revised: Id −∇ F ij ( x ) ∇ F ij ( x ) T is described as a projection operator.

Original: This formulation is similar to that used by (Poole et al., 2016), with the only diﬀerence being that they use normalized gradients in the projection operator. 
Revised: This formulation is similar to that used by (Poole et al., 2016), with the only diﬀerence being that they use normalized gradients in the projection operator. [Table] [Table]

Original:  Our formulation diﬀers from the above in several ways, the most important of which is that they claim to transform the Hessian by projecting it onto the tangent space (as written above), while we apply the implicit function theorem to parameterize the decision boundary as the graph of a function (see Appendix Section A.2.2, Equation 13).
Revised: [Table] Our formulation diﬀers from the above in several ways, the most important of which is that they claim to transform the Hessian by projecting it onto the tangent space (as written above), while we apply the implicit function theorem to parameterize the decision boundary as the graph of a function (see Appendix Section A.2.2, Equation 13).

Original: For computing the curvature of the decision boundary, exclusively projecting the Hessian is not going to result in[Table] the correct calculations, since for the decision boundary (as deﬁned in Section 2.2, Equation 2) you need a diﬀerent normal than for the graph of the response function.
Revised: For computing the curvature of the decision boundary, exclusively projecting the Hessian is not going to result in the correct calculations, since for the decision boundary (as deﬁned in Section 2.2, Equation 2) you need a diﬀerent normal than for the graph of the response function.

Original: More speciﬁcally, the normal used for computing the shape operator of the graph surface is diﬀerent from that for computing the shape operator of the decision boundary.
Revised: More speciﬁcally, the normal used for computing the shape operator of the graph surface is diﬀerent from that for computing the shape operator of the decision boundary.

Original: The latter has to be exclusively in the pixel space, while the former always has a component in the direction of the function response.
Revised: The latter has to be exclusively in the pixel space, while the former always has a component in the direction of the function response.

Original: Projecting the Hessian can restrict the computed curvatures in a certain subspace of directions, but it cannot change the computed curvatures in said subspace.
Revised: Projecting the Hessian can restrict the computed curvatures in a certain subspace of directions, but it cannot change the computed curvatures in said subspace.

Original: Another important diﬀerence is their projection operator is computed directly from the gradient [Table] and thus has a dimensionality equal to the input dimensionality.
Revised: Another important diﬀerence is their projection operator is computed directly from the gradient, and thus has a dimensionality equal to the input dimensionality.

Original: To calculate the curvature of a decision boundary, one needs a level-set formulation, which should have one fewer dimensions than the input space.
Revised: To calculate the curvature of a decision boundary, one needs a level-set formulation, which should have one fewer dimensions than the input space.

Original: The consequence of this diﬀerence can be made clear when looking at the curvature of an n -dimensional sphere, as deﬁned by the quadratic function:
Revised: The consequence of this diﬀerence can be made clear when looking at the curvature of an n -dimensional sphere, as deﬁned by the quadratic function:

Original: [Equation]
Revised: [Equation]

Original: where x is the input point vector and H is the  Hessian (i.e. an identity matrix in R n × n ).
Revised: where x is the input point vector and H is the sphere’s Hessian (i.e. an identity matrix in R n × n ).

Original: Given this function, the principal curvatures of the level-set at some radius, r , are all known to be (up to a sign change) 1 r .
Revised: Given this function, the principal curvatures of the level-set at some radius, r , are all known to be (up to a sign change) 1 r .

Original: However, we show in table 3 that the formulations provided by Moosavi-Dezfooli (2019) and Poole et al.
Revised: However, we show in table 5 that the formulations provided by Moosavi-Dezfooli (2019) and Poole et al.

Original: (2016) produced curvatures that are often several orders of magnitude oﬀ from the correct values the case of high-dimensional inputs.
Revised: (2016) produced curvatures that are often several orders of magnitude oﬀ from the correct values the case of high-dimensional inputs.

Original: Due to the dimensionality issue above, we computed the average error across each principal curvature, regardless of the number of curvatures reported, as they should all equal 1 r .
Revised: Due to the dimensionality issue above, we computed the average error across each principal curvature, regardless of the number of curvatures reported, as they should all equalr .

Original: We additionally tested a graph formulation (deﬁned in Section A.2.1), using a one-sheeted hyperboloid.
Revised: We additionally tested a graph formulation (deﬁned in Section A.2.1), using a one-sheeted hyperboloid.

Original: This formulation lives in the full space, so there is a principal curvature per input dimension.
Revised: This formulation lives in the full space, so there is a principal curvature per input dimension.

Original: However, we still found similar errors 4.
Revised: However, we show similar errors in in Table 6.

Original: Note, that the formulation from (Golden et al., 2019) matched our own, although they did not provide an equation for mapping this to the level-set curvature.
Revised: Note, that the formulation from (Golden et al., 2019) matched our own, although they did not provide an equation for mapping this to the level-set curvature.

Original: Our own implementations were correct with high precision on these and other experiments with known functions, all of which can be replicated from our provided code.
Revised: Our own implementations were correct with high precision on these and other experiments with known functions, all of which can be replicated from our provided code.

Original: Finally, while the subspace projection methods proposed by (Fawzi et al., 2016; MoosaviDezfooli et al., 2017b)could potentially provide metrics related to curvature in the subspaces they deﬁned, we chose to instead derive our pullback method described in Equation 3 from primary sources, as explained in Appendix A.[Table] G MNIST results [Table]
Revised: Finally, while the subspace projection methods proposed by (Fawzi et al., 2016; Moosavi-Dezfooli et al., 2017b) potentially provide metrics related to curvature in the subspaces they deﬁned, we chose to instead derive our pullback method described in Equation 3 from primary sources, as explained in Appendix A.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: We conducted the same experiments as described in Section 3 on networks trained with the

Original: MNIST dataset.

Original: As a test set we used 500 images, unseen in training and correctly labeled by all investigated models.

Original: [Figure]

Original: In Figure 16 we can see that perturbation lengths are much larger for MNIST than for CIFAR-10, although the relative increase in distances for the robust network is still present.

Original: The optimization was in general more diﬃcult with MNIST, likely due to the fact that the images all live very close to the allowable input bounds.

Original: Speciﬁcally, the algorithm found less than 30 adversarial dimensions for more than 95% of the test images (Figure 10), to be compared with ﬁnding >50 dimensions for 100% of the CIFAR test images.

Original: That it is harder to fool the MNIST model can also be seen in

Original: Table 5: the accuracies of both models are much higher for the same perturbation lengths than the accuracies of the CIFARmodels.

Original: For the curvature analysis, on the other hand, we found that the overall values to be larger on CIFAR.

Original: Additionally, the adversarially trained model had more decision boundary curvature within adversarial subspaces on MNIST than on CIFAR.

Original: We additionally noticed that most of the MNIST adversarials lie in a thin regions between the decision boundary and the valid pixel box boundary (compare Figure 15).

Original: Figure 13 shows that this leads to large rates of out of bounds samples when measuring the distance to the decision boundary.

Original: For example 90% of the time the decision boundary is outside the valid pixel range in thedimensional adversarial subspace of the naturally trained model.

Original: [Figure]

Original: d i s t a n c e t o d e c

Original: [-i s i o n b n d . natural robust 
-]
Revised: [-i s i o n b n d . natural robust 
-]

Original: [Figure]

Original: d i s t a n c e c oe ff

Original: i c i e n t   0.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: a d v e r s a r i a l v e c t

Original: o r l e n g t h

Original: [Figure]

Original: a d v e r s a r i a l d i m e n s i o n

Original: Original 6 (1.63)

Original: Natural 3 (3.56)

Original: Robust 5 (1.67)

Original: Natural 8 (4.11)

Original: Robust 3 (1.92)

Original: Natural 2 (4.63)

Original: Robust 2 (3.02)

Original: Natural 4 (5.23)

Original: Robust 7 (3.52)

Original: Natural 0 (nan)

Original: Robust 8 (0.95) 8 (1.85) 3 (1.96) 3 (1.90) 6 (2.31) 0 (2.49) 7 (2.50) 9 (2.94) 3 (3.12) 3 (3.21) 3 (0.86) 3 (2.28) 6 (1.21) 0 (2.51) 3 (1.54) 3 (4.18) 0 (1.55) 4 (4.58) 5 (2.18) 3 (7.63) 5 (1.36) 1 (2.12) 2 (1.50) 7 (2.38) 5 (2.30) 5 (3.18) 2 (2.82) 2 (3.42) 7 (3.39) 2 (4.13) 4 (0.88) 8 (3.33) 7 (1.48) 5 (3.77) 2 (1.48) 2 (3.84) 8 (1.96) 7 (4.44) 2 (2.73) 2 (4.98)

Original: [Figure]

Original: Adv

Original: [Figure]

Original: Adv

Original: [Figure]

Original: Adv

Original: [Figure]

Original: Adv