Original: Deep Learning Generalization and the Convex Hull of Training Sets
Revised: Deep Learning Generalization and the Convex Hull of Training Sets

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In this work we study the generalization of deep learning functions in relation tothe convex hull of their training sets.
Revised: In this work, we study the generalization of deep learning functions in relation to the convex hull of their training sets.

Original: A trained image classiﬁer basically partitionsits domain via decision boundaries and assigns a class to each of those partitions.
Revised: A trained image classiﬁer basically partitions its domain via decision boundaries, and assigns a class to each of those partitions.

Original: The location of decision boundaries inside the convex hull of training set can beinvestigated in relation to the training samples.
Revised: The location of decision boundaries inside the convex hull of training set can be investigated in relation to the training samples.

Original: However, our analysis shows thatin standard image classiﬁcation datasets, a considerable portion of testing imagesare outside that convex hull.
Revised: However, our analysis shows that in standard image classiﬁcation datasets, a considerable portion of testing images are outside that convex hull.

Original: Therefore, the performance of a trained a modelpartially depends on how its decision boundaries are extended outside the convexhull of the training data.
Revised: Therefore, the performance of a trained model partially depends on how its decision boundaries are extended outside the convex hull of its training data.

Original: From this perspective, over-parameterization of deeplearning models is a necessity for shaping the extension of decision boundaries.
Revised: From this perspective, over-parameterization of deep learning models may be considered a necessity for shaping the extension of decision boundaries.

Revised: At the same time, over-parameterization should be accompanied by a speciﬁc training regime, in order to yield a model that not only ﬁts the training set, but also its decision boundaries extend desirably outside the convex hull.

Original: To illustrate this, we investigate the decision boundaries of a neural network, withvarious degrees of over-parameterization, inside and outside the convex hull ofits training set.
Revised: To illustrate this, we investigate the decision boundaries of a neural network, with various degrees of over-parameterization, inside and outside the convex hull of its training set.

Original: Moreover, we use a polynomial decision boundary to study thenecessity of over-parameterization  in shaping its extensions outside the convex hull.
Revised: Moreover, we use a polynomial decision boundary to study the necessity of over-parameterization and the inﬂuence of training regime in shaping its extensions outside the convex hull of training set.

Original:  Introduction
Revised:  Introduction

Original: A deep learning image classiﬁer is a mathematical function that maps images to classes, i e., a deeplearning function [Strang, 2019].
Revised: A deep learning image classiﬁer is a mathematical function that maps images to classes, i e., a deep learning function [Strang, 2019].

Original: These models/functions have shown to be exceptionally useful inreal-world applications.
Revised: These models/functions have shown to be exceptionally useful in real-world applications.

Original: However, generalization of these functions, is considered a mystery by deeplearning researchers [Arora et al., 2019].
Revised: However, generalization of these functions is considered a mystery by deep learning researchers [Arora et al., 2019].

Original: These models have orders of magnitude more parametersthan their training samples [Belkin et al., 2019, Neyshabur et al., 2019] and they can achieve perfectaccuracy on their training sets, even when the training images are randomly labeled, or the contentsof images are replaced with random noise [Zhang et al., 2017].
Revised: These models have orders of magnitude more parameters than their training samples [Belkin et al., 2019, Neyshabur et al., 2019] and they can achieve perfect accuracy on their training sets, even when the training images are randomly labeled, or the contents of images are replaced with random noise [Zhang et al., 2017].

Original: The training loss function of thesemodels has inﬁnite number of minimizers, where only a small subset of those minimizers generalizewell [Neyshabur et al., 2017a].
Revised: The training loss function of these models has inﬁnite number of minimizers, where only a small subset of those minimizers generalize well [Neyshabur et al., 2017a].

Original: If one succeeds in picking a good minimizer of training loss, themodel can classify the testing images correctly, nevertheless, for any correctly classiﬁed image,there are inﬁnite number of images that look the same, but models will classify them incorrectly(phenomenon known as adversarial vulnerability)
Revised: If one succeeds in picking a good minimizer of training loss, the model can classify the testing images correctly, nevertheless, for any correctly classiﬁed image, there are inﬁnite number of images that look the same, but models will classify them incorrectly (phenomenon known as adversarial vulnerability)

Original: [Papernot et al., 2016, Shafahi et al., 2019, Tsipraset al., 2019].
Revised: [Papernot et al., 2016, Shafahi et al., 2019, Tsipras et al., 2019].

Original: Here, we study some geometric properties of the training and testing sets to providenew insights about what a model can learn from its training data, and how it can generalize.
Revised: Here, we study some geometric properties of standard training and testing sets to provide new insights about what a model can learn from its training data, and how it can generalize.

Original: Speciﬁcally, we study the convex hulls of image classiﬁcation datasets (both in the pixel space and inthe wavelet space), and show that a considerable portion of testing images fall outside the convexhull of training sets.
Revised: Speciﬁcally, we study the convex hulls of image classiﬁcation datasets (both in the pixel space and in the wavelet space), and show that most of testing images fall outside the convex hull of training sets, with various distances from the hull.

Original: Therefore, the performance of a trained model partially depends on how well itcan extrapolate.
Revised: Therefore, the performance of a trained model partially depends on how well it can extrapolate.

Original: We further investigate this extrapolation in relation to the over-parameterization ofthe networks, using a polynomial decision boundary as an example.
Revised: We further investigate this extrapolation in relation to the over-parameterization of neural networks and the inﬂuence of training regimes in shaping the extension of decision boundaries.

Original:  Geometry of testing data w.r.t the convex hull of training sets
Revised:  Geometry of testing data w.r.t the convex hull of training sets

Original: First, we show that for standard datasets such as MNIST  and CIFAR-10, a considerable portion oftesting data are outside the convex hull of their training sets.
Revised: First, we show that for standard datasets: MNIST [LeCun et al., 1998] and CIFAR-10 [Krizhevsky, 2009], most of their testing data are outside the convex hull of their training sets.

Original: The convex hull of a training setisdenoted by H tr .
Revised: We denote the convex hull of a training set by H tr .

Original: To verify whether an image/data point is inside its corresponding H tr or not, we can simply try toﬁt a hyper-plane separating the point and the training set.
Revised: To verify whether an image/data point is inside its corresponding H tr or not, we can simply try to ﬁt a hyper-plane separating the point and the training set.

Original: If we ﬁnd such hyper-plane, the point isoutside the convex hull, and vice versa.
Revised: If we ﬁnd such hyper-plane, the point is outside the convex hull, and vice versa.

Original: This is basically a linear regression problem and there aremany efﬁcient and fast methods to perform it [Goldstein et al., 2015].
Revised: This is basically a linear regression problem and there are many efﬁcient and fast methods to perform it, e.g., [Goldstein et al., 2015].

Original: For the MNIST dataset, we seethat only about 26% of testing data are outside the H tr , in the pixel space. 
Revised: For the MNIST dataset, we see that about 95.1% of testing images are outside the H tr , in the pixel space. For CIFAR-10, that percentage is more than 99.9%.

Original: When we transform theimages with wavelets (an operation analogous to convolutional neural nets), 87% of testing data falloutside of the H tr .
Revised: When we transform the images with wavelets (an operation analogous to convolutional neural nets), these percentages almost remain the same.

Original: For CIFAR-10, however, more than 99.8% of testing images are outside the H tr ,while wavelet transformation increases that percentage to 99.96%.

Original: We can now investigate the testing data outside the H tr , to see how far they are located from it.
Revised: We can now investigate the testing data outside the H tr , to see how far they are located from it.

Original: To approximate the distance of a testing image to the H tr , we measure the distance between thatimage and the closest image to it in the training set.
Revised: One can measure the distance of a testing image to the H tr using high-dimensional geometry algorithms.

Original: In high-dimensional space, this approximation isaccurate enough for the purpose of our study.
Revised: There are also methods that aim to identify a coreset to approximate convex hulls of large datasets [Blum et al., 2019]. Here, we approximate the distance to convex hull by ﬁrst ﬁtting a linear Support Vector Machine (SVM) between the testing point and the H tr . Since an SVM maximizes its margin from its supports, the total margin of the resulting SVM will closely approximate the distance between the point and H tr 1 .

Original: Figure 1 shows the histogram of distance to H tr for thetesting images of the above datasets.
Revised: Figure 1 shows the histogram of distance to H tr for the testing images of the above datasets.

Original: [Figure]
Revised: [Figure]

Revised: for testing images that fall outside H tr .

Original:  Related work about geometry of data and deep learning
Revised: 2.1 Related work about geometry of data and deep learning

Original: Convex hulls of training sets are rarely considered in deep learning studies.
Revised: To the best of our knowledge, convex hulls of training sets are not commonly considered in deep learning studies, especially the ones focused on their generalization.

Original: Recently, Yousefzadehand Huang [2020] reported that in the wavelet space, distance ofthe testing images to the convex hullfor each training class can predict the label for more than 97% of MNIST testing data.
Revised: Recently, Yousefzadeh and Huang [2020] reported that in the wavelet space, distance of testing images to the convex hull for each training class can predict the label for more than 97% of MNIST testing data.

Original: Previously, Haffner [2002] considered the convex hull of MNIST data for Support Vector Machines.
Revised: Previously, Haffner [2002] considered the convex hull of MNIST data for Support Vector Machines.

Original: Similarly, Vincent and Bengio [2002] considered the convex hulls for K-Nearest Neighbor (KNN) algorithms.
Revised: Similarly, Vincent and Bengio [2002] considered the convex hulls for K-Nearest Neighbor (KNN) algorithms.

Original: However, those methods do not generalize to deep learning functions.
Revised: However, those methods do not generalize to deep learning functions.

Original: At the same time, some researchers have studied other geometrical aspects of deep learning models,e.g., [Cohen et al., 2020, Fawzi et al., 2018, Cooper, 2018, Kanbak et al., 2018, Neyshabur et al. ,2017b].
Revised: Some researchers have studied other geometrical aspects of deep learning models, e.g., [Cohen et al., 2020, Fawzi et al., 2018, Cooper, 2018, Kanbak et al., 2018, Neyshabur et al., 2017b].

Original: To our knowledge, those studies do not investigate the generalization of deep neural networksin relation to the convex hull of training sets. 
Revised: To our knowledge, those studies do not investigate the generalization of deep neural networks in relation to the convex hull of training sets. Most recently, Xu et al.

Revised: [2020] studied the extrapolation behavior of ReLU perceptrons and concluded that such models cannot extrapolate most non-linear tasks.

Revised: However, they do not connect their analysis to the fact that a considerable portion of testing samples of standard image datasets fall outside the convex hull of their training sets.

Original:  Learning outside the convex hull: A polynomial decision boundary
Revised:  Learning outside the convex hull: A polynomial decision boundary

Original: In the previous section, we showed that a considerable portion of  testing data  are outside the convexhull of  training set, while the distance to the H tr resembles a normal distribution.
Revised: In the previous section, we showed that most of the testing data of MNIST and almost all of the testing data of CIFAR-10 are outside the convex hull of their corresponding training sets, while the distance to the H tr has noticeable variations.

Original: Hence, a traineddeep learning model somehow manages to deﬁne its decision boundaries accurately enough outsidethe boundaries of what it has observed during training.
Revised: Hence, a trained deep learning model somehow manages to deﬁne its decision boundaries accurately enough outside the boundaries of what it has observed during training.

Original: But how does a model achieve that, or moreprecisely, how do we manage to train a model such that its decision boundaries have our desirableform outside the H tr ?
Revised: But how does a model achieve that, or more precisely, how do we manage to train a model such that its decision boundaries have the desirable form outside the H tr ?

Revised: Since we are interested in the generalization of image classiﬁers, and the pixel space is a bounded space, we consider the domain to be bounded, while the H tr occupies some portion of it.

Revised: Testing data can be inside and outside the H tr , but always inside the bounded domain.

Original: Let’s  use a polynomial ﬁtting problem as an example to gain some intuitive insights.
Revised: Let’s now use a polynomial decision boundary as an example to gain some intuitive insights.

Original: 1 Figure 2a shows two point sets colored in blue and red, each set belonging to a class.
Revised: Figure 4a shows two point sets colored in blue and red, each set belonging to a class.

Original: These sets are non-linearly separable, because they have no overlap.
Revised: These sets are non-linearly separable, because they have no overlap.

Original: If we use the polynomial y = 10 − 5 ( x + 20)( x + 17)( x + 10)( x + 5)( x )( x − 2)( x − 9) , as our decision boundary, wewill achieve perfect accuracy in separating these two sets, as shown in
Revised: If we use the polynomial [Equation] as our decision boundary, we achieve perfect accuracy in separating these two sets, as shown in

Original: Figure 2b.
Revised: Figure 4b.

Original: [Figure]
Revised: [Figure]

Original: Now that we have obtained this polynomial, i.e., decision boundary,  howdoes it generalize to unseendata?
Revised: Now that we have obtained this polynomial, i.e., decision boundary, we would be interested to know how it generalizes to unseen data.

Original:  Figure 3  shows the extension of this polynomial outside its H tr .
Revised: Let’s assume that our bounded domain is deﬁned by the limits shown in Figure 3 which also shows how our decision boundary generalizes outside the H tr .

Original: If our polynomial cancorrectly separate and label our testing data, we would say that our polynomial is generalizing well,and vice versa.
Revised: If our polynomial can correctly separate and label our testing data, we would say that our polynomial is generalizing well, and vice versa.

Original: But, what is reasonable to expect from the testing data?
Revised: But, what is reasonable to expect from the testing data?

Original: In what regions of the domainshould we be hopeful that our polynomial can generalize?
Revised: In what regions of the domain should we be hopeful that our polynomial can generalize?

Original: What if the domain is much larger than the H tr ? Is the extension of our polynomial on both sides reasonable enough?
Revised: What if the domain is much larger than the H tr ? Is the extension of our polynomial on both sides reasonable enough?

Original: Clearly, the answer to the above questions can be different inside and outside the H tr .
Revised: Clearly, the answer to the above questions can be different inside and outside the H tr .

Original: Inside the
Revised: Inside the

Original: H tr , if the unseen data has a similar label distribution as the training set, we can be hopeful that ourdecision boundary will generalize well.
Revised: H tr , if the unseen data has a similar label distribution as the training set, we can be hopeful that our decision boundary will generalize well.

Original: However, outside the H tr is uncharted territory and hence,there will be less hope/conﬁdence about the generalization of our decision boundary, especially whenwe go far outside the H tr .
Revised: However, outside the H tr is uncharted territory and hence, there will be less hope/conﬁdence about the generalization of our decision boundary, especially when we go far outside the H tr .

Original: Now in Figure 3, imagine that from some prior knowledge, we expect the classes to be separableoutside the convex hull, according to a horizontal line approximable by y = 0 .
Revised: Now, consider the decision boundary in Figure 3 and imagine that from some prior knowledge, we expect the classes to be separable outside the convex hull, according to the decision boundary shown in Figure 4.

Original: Then, our currentdecision boundary will generalize poorly outside the H tr , because it almost has a vertical shape.
Revised: In such case, the decision boundary, deﬁned by equation (1) and shown in Figure 3 will generalize poorly outside the H tr . [Figure]

Original: How can we incorporate that prior knowledge into our decision boundary  so that it can generalize wellboth inside and outside the H tr ?
Revised: How can we incorporate that prior knowledge into the decision boundary deﬁned by equation (1) and reshape it to the decision boundary in Figure 4, so that it can generalize well both inside and outside the H tr ?

Original: How can we change its shape  outside the H tr , while maintaining itscurrent shape inside the H tr ?
Revised: How can we change the shape of our polynomial outside the H tr , while maintaining its current shape inside the H tr ?

Original: Clearly, we should add to the degree of our polynomial, or in otherwords, we should over-parameterize it.
Revised: Clearly, we should add to the degree of our polynomial, or in other words, we should over-parameterize it.

Original: The necessity of over-parameterization for achieving that goalcan be rigorously shown using the orthogonal system of Legendre polynomials [Ascher and Greif,2011].
Revised: The necessity of over-parameterization for achieving that goal for our polynomial decision boundary can be rigorously shown using the orthogonal system of Legendre polynomials [Ascher and Greif, 2011].

Revised: From this perspective, over-parameterization is necessary, but it is not sufﬁcient for good generalization, because for an over-parameterized polynomial (i.e., decision boundary), there will be inﬁnite number of solutions that can ﬁt the training data, but each of them would have a different shape outside the H tr .

Revised: In fact, an over-parameterized polynomial can have the same shape as the polynomial in Figure 3.

Revised: But, how can we pick the decision boundary that ﬁts the data and also generalizes well outside the H tr ?

Revised: In the over-parameterized regime, the key to ﬁnding the desirable decision boundary is the optimization process, i.e., the training regime .

Original:   Output of deep learning functions outside their H tr
Revised: In other words, different training regimes would lead us to decision boundaries that all perfectly ﬁt the training set, but each has a different shape outside the H tr . This highlights that over-parameterization and training regime work in tandem to shape the extension of decision boundaries.  Output of deep learning functions outside their H tr

Original: In this section, we investigate a 2-layer neural network with ReLU activation.
Revised: In this section, we investigate a 2-layer neural network with ReLU activation.

Original: We train the model withvarious number of neurons on the data we investigate in previous section, depicted in Figure 2b.
Revised: We train the model with various number of neurons on the data we investigate in previous section, depicted in Figure 4b.

Original: Wethen investigate the output of the trained models inside and outside of the H tr , as shown in Figure 4. [Figure]
Revised: We then investigate the output of the trained models inside and outside of the H tr , as shown in Figure 5.

Original: We observe that when the model is under-parameterized (e.g., model with 2 neurons), the trainingregime does not have a signiﬁcant effect on the resulting model.
Revised: We observe that when the model is under-parameterized (e.g., model with 2 neurons), the training regime does not have a signiﬁcant effect on the resulting model.

Original: The model output also follows arelatively simple pattern inside and outside the H tr .
Revised: The model output also follows a relatively simple pattern inside and outside the H tr .

Original: For an under-parameterized model, the trainingloss function has limited number of minimizers, none of which lead to zero loss.
Revised: For an under-parameterized model, the training loss function has limited number of minimizers, none of which lead to zero loss.

Original: Finding thesame minimizer of training loss is equivalent to obtaining the same trained model, hence unlike theover-parameterized setting, the training regime is focused on ﬁnding the best shape for the decisionboundary inside the convex hull.
Revised: Finding the same minimizer of training loss is equivalent to obtaining the same trained model, hence unlike the over-parameterized setting, the training regime is focused on ﬁnding the best shape for the decision boundary inside the convex hull.

Original: When the model is over-parameterized, however, we have inﬁnite number of parameter conﬁgurationsthat minimize the training loss to zero, which is equivalent to developing a decision boundary thatperfectly separates our two sets of data points.
Revised: When the model is over-parameterized, however, we have inﬁnite number of parameter conﬁgurations that minimize the training loss to zero, which is equivalent to developing a decision boundary that perfectly separates our two sets of data points.

Original: So, forming the decision boundary inside the convexhull is easily achievable.
Revised: So, forming the decision boundary inside the convex hull is easily achievable.

Original: What is different about those inﬁnite number of models is the extension oftheir decision boundaries outside the convex hull of the data.
Revised: What is different about those inﬁnite number of models is the extension of their decision boundaries outside the convex hull of the data.

Original:  This seems to explain why we need over-parameterized models for deep learning and also explainwhy the generalization of deep learning models are so susceptible to different training regimes: 1generalization of these models relies on how they extrapolate,
Revised: [Figure] This seems to explain why we need over-parameterized models for deep learning and also explain why the generalization of deep learning models are so susceptible to different training regimes: Generalization of deep learning models depends on how they extrapolate; 2.

Original: 2- to desirably shape the extension ofdecision boundaries, we over-parameterize the models and then minimize their loss in a particularway.
Revised: In order to desirably shape the extension of decision boundaries, we over-parameterize the models and then minimize their loss using speciﬁc training regimes.

Original:  Conclusion and future work
Revised:  Conclusion and future work

Original: We show that a considerable portion of testing data for  standard image classiﬁcation models lieoutside the convex hull of training sets, both in pixel space and in wavelet space.
Revised: We showed that most of testing data for some standard image classiﬁcation models lie outside the convex hull of training sets, both in pixel space and in wavelet space.

Original: Therefore, thegeneralization of a deep network partially relies on its capability to extrapolate outside the boundariesof the data it has seen during training.
Revised: Therefore, the generalization of a deep network partially relies on its capability to extrapolate outside the boundaries of the data it has seen during training.

Original: Base on this observation, the signiﬁcant number of studies thatfocus on interpolation regimes in order to explain the generalization of deep networks seem to beinadequate to explain the extrapolation requirement.
Revised: Based on this observation, the signiﬁcant number of studies that focus on interpolation regimes seem to be insufﬁcient to explain the generalization of deep networks.

Original: From this perspective, over-parameterization of models is a necessity to desirably form the extensionof decision boundaries outside the convex hull of data.
Revised: From this perspective, over-parameterization of models may be considered a necessity to desirably form the extension of decision boundaries outside the convex hull of data.

Original: This can be proven for polynomial regressionmodels using the orthogonal system of Legendre polynomials.
Revised: This can be proven for polynomial regression models using the orthogonal system of Legendre polynomials.

Original: Moreover, we showed that the trainingregime can signiﬁcantly affect the shape of decision boundaries outside the convex hulls, affectingthe accuracy of a model in its extrapolation.
Revised: Moreover, we showed that the training regime can signiﬁcantly affect the shape of decision boundaries outside the convex hulls, affecting the accuracy of a model in its extrapolation.

Original: We investigated a  polynomial decision boundary todemonstrate these phenomena.
Revised: We investigated a 2-layer ReLU network and a polynomial decision boundary to demonstrate these ideas. In future work, we plan to more closely analyze the effect of over-parameterization and training regimes on the shape of decision boundaries outside the convex hulls, and investigate how that affects the generalization. We also plan to study how sensitive the classiﬁcations of standard trained models are w.r.t the minimum perturbations that would bring testing images inside the H tr . Studying the convex hulls of internal representations of the data in a trained network is another direction that can be pursued.

Revised: Such analysis can be performed, separately for each class in the dataset.

Revised: It has been speculated that a given image classiﬁcation dataset lies on a lower dimensional manifold and such manifold is what a deep learning model learns from the data.

Revised: Study of convex hulls might provide insights about such manifold and also about the distribution of training and testing sets.

Revised: Finally, measuring the volume of the convex hulls of training and testing sets, their overlap, and also the volume of the domain that remains unoccupied may be insightful.

Revised: The dimension of domain (number of pixels) relative to the number of samples may have a signiﬁcant effect on the portion of testing data that fall outside the H tr .

Revised: This relates to the limit theorems for the convex hull of random points in higher dimensions [Hueter, 1999] and also to studies on separability and distribution of random points [Fink et al., 2016].

Revised: Acknowledgments and Disclosure of Funding

Revised: R.Y. thanks Yaim Cooper for helpful discussion.

Revised: R.Y. was supported by a fellowship from the

Revised: Department of Veteran Affairs.

Revised: The views expressed in this manuscript are those of the author and do not necessarily reﬂect the position or policy of the Department of Veterans Affairs or the United

Revised: States government.