Effects of Data Geometry in Early Deep Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep neural networks can approximate functions on different types of data, fromimages to graphs, with varied underlying structure.
This underlying structure canbe viewed as the geometry of the data manifold.
By extending recent advancesin the theoretical understanding of neural networks, we study how a randomlyinitialized neural network with piecewise linear activation splits the data manifoldinto regions where the neural network behaves as a linear function.
We derivebounds on the number of linear regions and the distance to boundaries of theselinear regions on the data manifold.
This leads to insights into the expressivityof randomly initialized deep neural networks on non-Euclidean data sets.
Weempirically corroborate our theoretical results using a toy supervised learningproblem.
Our experiments demonstrate that number of linear regions varies acrossmanifolds and how our results hold when changing neural network architectures.
We further demonstrate how the complexity of linear regions changes on the lowdimensional manifold of images as training progresses, using the MetFaces dataset.
Introduction
The capacity of Deep Neural Networks (DNNs) to approximate arbitrary functions given sufficienttraining data in the supervised learning setting is well known [Cybenko, 1989, Hornik et al., 1989,
Anthony and Bartlett, 1999].
Several different theoretical approaches have emerged that study theeffectiveness and pitfalls of deep learning.
These studies vary in their treatment of neural networksand the aspects they study range from convergence [Allen-Zhu et al., 2019, Goodfellow and Vinyals,2015], generalization [Kawaguchi et al., 2017, Zhang et al., 2017, Jacot et al., 2018, Sagun et al.
,2018], function complexity [Mont ´ ufar et al., 2014, Mhaskar and Poggio, 2016], adversarial attacks[Szegedy et al., 2014, Goodfellow et al., 2015] to representation capacity [Arpit et al., 2017].
Somerecent theories have also been shown to closely match empirical observations [Poole et al., 2016,
Hanin and Rolnick, 2019b, Kunin et al., 2020].
One approach to studying DNNs is to examine how the underlying structure, or geometry, of the datainteracts with learning dynamics.
The manifold hypothesis states that high-dimensional real worlddata typically lies on a low dimensional manifold [Tenenbaum, 1997, Carlsson et al., 2007, Feffermanet al., 2013].
Empirical studies have shown that DNNs are highly effective in deciphering thisunderlying structure by learning intermediate latent representations [Poole et al., 2016].
The abilityof DNNs to “flatten” complex data manifolds, using composition of seemingly simple piece-wiselinear functions, appears to be unique [Brahma et al., 2016, Hauser and Ray, 2017].
DNNs with piecewise linear activations, such as ReLU [Nair and Hinton, 2010], divide the inputspace into linear regions, wherein the DNN behaves as a linear function [Mont ´ ufar et al., 2014].
Thedensity of these linear regions serves as a proxy for the DNN’s ability to interpolate a complex datalandscape and has been the subject of detailed studies [Mont´ufar et al., 2014, Telgarsky, 2015, Serraet al., 2018, Raghu et al., 2017].
The work by Hanin and Rolnick [2019a] on this topic stands outbecause they derive bounds on the average number of linear regions and verify the tightness of these bounds empirically for deep ReLU networks, instead of larger bounds that rarely materialize.
Haninand Rolnick [2019a] conjecture that the number of linear regions correlates to the expressive power ofrandomly initialized DNNs with piecewise linear activations.
However, they assume that the data isuniformly sampled from the Euclidean space R d , for some d .
By combining the manifold hypothesiswith insights from Hanin and Rolnick [2019a], we are able to go further in estimating the numberof linear regions and the average distance from linear boundaries .
We derive bounds on how thegeometry of the data manifold affects the aforementioned quantities.
To corroborate our theoretical bounds with empirical results, we design a toy problem where theinput data is sampled from two distinct manifolds that can be represented in a closed form.
We countthe exact number of linear regions on these two manifolds that a randomly initialized neural networksplits the two manifolds into. We also observe the average distance to the boundaries of linear regions.
We demonstrate how the number of linear regions varies for two distinct manifolds in our setting.
These results show that the number of linear regions on the manifold do not grow exponentially withthe dimension of input data.
Our experiments do not provide estimates for theoretical constants, as inmost deep learning theory, but demonstrate that the number of linear regions change as a consequenceof these constants.
We also study high dimensional data that lies on low dimensional manifolds withunknown structure and how the number of linear regions vary on and off this manifold, which is amore realistic setting.
To achieve this we present experiments performed on the manifold of naturalimages.
We sample data from the image manifold using a generative adversarial network (GAN)[Goodfellow et al., 2014] trained on the curated images of paintings.
Specifically, we generate imagesusing the pre-trained StyleGAN [Karras et al., 2019, 2020b] trained on the curated MetFaces dataset[Karras et al., 2020a].
We also assign random labels to the images in the dataset and train a deep
ReLU network in a supervised manner, a scenario in which it would overfit [Zhang et al., 2017].
We generate curves on the image manifold of faces, using StyleGAN, and show how overfitting isreflected in the density of linear regions of the aforementioned deep ReLU network.
Taken together,these results shed new light on the geometry of deep learning over structured data sets by taking adata intrinsic approach to understanding the expressive power of DNNs.
Preliminaries And Background
Our goal is to understand how the underlying structure of real world data matters for deep learning.
We first provide the mathematical background required to model this underlying structure as thegeometry of data.
We then provide a summary of previous work on understanding the approximationcapacity of deep ReLU networks via the complexity of linear regions.
For the details on how ourwork fits into one of the two main approaches within the theory of DNNs, from the expressive poweror from the learning dynamics perspective, we refer the reader to Appendix C.
2.1 Data Manifold and Definitions
We use the example of the MetFaces dataset [Karras et al., 2020a] to illustrate how data lies on a lowdimensional manifold.
The images in the dataset are 1028 × 1028 × 3 dimensional.
By contrast,the number of realistic dimensions along which they vary are limited, e.g. painting style, artist, sizeand shape of the nose, jaw and eyes, background, clothing style; in fact, very few 1028 × 1028 ×
3dimensional images correspond to realistic faces.
We illustrate how this affects the possible variationsin the data in Figure 1.
A manifold formalises the notion of limited variations in high dimensionaldata.
One can imagine that there exists an unknown function f : X → Y from a low dimensionalspace of variations, to a high dimensional space of the actual data points.
Such a function f : X → Y ,from one open subset X ⊂ R m , to another open subset Y ⊂ R k , is a diffeomorphism if f is bijective,and both f and f − 1 are differentiable, also referred to as smooth.
Therefore, a manifold is defined asfollows.
Definition 2.1.
Let k, m ∈ N 0 .
A subset M ⊂ R k is called a smooth m -dimensional submanifoldof R k (or m -manifold in R k ) iff every point x ∈ M has an open neighborhood U ⊂ R k such that
U ∩ M is diffeomorphic to an open subset Ω ⊂ R m .
A diffeomorphism (i e. differentiable mapping)
f : U ∩ M → Ω
For the MetFaces dataset example, suppose there are 10 dimensions along which the images vary.
Further assume that each variation can take a value continuously in some interval of R .
Then thesmooth parametrization would map f : Ω ∩ R 10 → M ∩ R 1028 × 1028 × 3 .
This parametrization and itsinverse are unknown in general and computationally very difficult to estimate in practice.
There are similarities in how geometric elements are defined for manifolds and Euclidean spaces.
A smooth curve, on a manifold M , γ : I → M is defined from an interval I to the manifold M asa function that is differentiable for all t ∈ I , just as for Euclidean spaces.
The shortest such curvebetween two points on a manifold is no longer a straight line, but is instead a geodesic .
One recurringgeometric element, which is unique to manifolds and stems from the definition of smooth curves, isthat of a tangent space , defined as follows.
Definition 2.2.
Let M be an m -manifold in R k and x ∈ M be a fixed point.
A vector v ∈ R k is calleda tangent vector of M at x if there exists a smooth curve γ : I → M
such that γ (0) =
x, ˙ γ (0) =
vwhere ˙ γ ( t ) is the derivative of γ at t .
The set T x M := { ˙ γ (0) | γ : R → M is smooth γ (0) =
x } , of tangent vectors of M at x is called the tangent space of M at x .
In simpler terms, the plane tangent to the manifold M at point x is called the tangent space anddenoted by by T x M .
Consider the upper half of a 2-sphere, S 2 ⊂ R 3 , which is a 2-manifold in R 3 .
The tangent space at a fixed point x ∈ S 2 is the 2D plane perpendicular to the vector x and tangentialto the surface of the sphere that contains the point x .
For additional background on manifolds werefer the reader to Appendix B.
2.2 Linear Regions of Deep ReLU Networks
The higher the density of these linear regions the more complex a function a DNN can approximate.
For example, a sin curve in the range [0 , 2 π ] is better approximated by 4 piecewise linear regions asopposed to 2.
To clarify this further, with the 4 “optimal” linear regions [0 , π/ 2) , [ π/ 2 , π ) , [ π, 3 π/ 2) ,and [3 π/ 2 , 2 π ] a function could approximate the sin curve better than any 2 linear regions.
In otherwords, higher density of linear regions allows a DNN to approximate the variation in the curve better.
We define the notion of boundary of a linear regions in this section and provide an overview ofprevious results.
We consider a neural network, F , which is a composition of activation functions.
Inputs at each layerare multiplied by a matrix, referred to as the weight matrix, with an additional bias vector that isadded to this product.
We limit our study to ReLU activation function [Nair and Hinton, 2010], whichis piece-wise linear and one of the most popular activation functions being applied to various learningtasks on different types of data like text, images, signals etc.
We further consider DNNs that map inputs, of dimension n in , to scalar values.
Therefore, F : R n in → R is defined as,
[Equation]
where W l ∈ M n l × n l − 1 is the weight matrix for the l th hidden layer, n l is the number of neurons inthe l th hidden layer, B l ∈ R n l is the vector of biases for the l th hidden layer, n 0 = n in and σ : R → Ris the activation function.
For a neuron z in the l th layer we denote the pre-activation of this neuron,for given input x ∈ R n in , as z l ( x ) .
For a neuron z in the layer l we havez ( x ) =
W l − 1 ,z σ ( ...σ ( B 1 + W 1 ,z x )) , for l > 1 (for the base case l = 1 we have z ( x ) =
W 1 ,z x ) where W l − 1 ,z is the row of weights, in theweight matrix of the l th layer, W l , corresponding to the neuron z .
We use W z to denote the weightvector for brevity, omitting the layer index l in the subscript.
We also use b z to denote the bias termfor the neuron z .
Neural networks with piecewise linear activations are piecewise linear on the input space [Mont´ufaret al., 2014].
Suppose for some fixed y ∈ R n in as x → y if we have z ( x ) → − b z then we observe adiscontinuity in the gradient ∇ x σ ( b z + W z z ( x )) at y .
Intuitively, this is because x is approachingthe boundary of the linear region of the function defined by the output of z .
Therefore, the boundaryof linear regions, for a feed forward neural network F , is defined as:
[Equation]
continuous [Equation] Hanin and Rolnick [2019a] argue that an important generalization for the approximation capacityof a neural network F is the ( n in − 1) − dimensional volume density of linear regions defined asvol n in − 1 ( B F ∩ K ) / vol n in ( K ) , for a bounded set K ⊂ R n in .
This quantity serves as a proxy fordensity of linear regions and therefore the expressive capacity of DNNs.
Intuitively, higher density oflinear boundaries means higher capacity of the DNN to approximate complex non-linear functions.
The quantity is applied to lower bound the distance between a point x ∈ K and the set B F , which isdistance ( x, B F ) = min neurons z | z ( x ) −
[Equation]
z | /
[Equation]
z ( x ) || , which measures the sensitivity over neurons at a given input.
The above quantity measures how “far”the input is from flipping any neuron from inactive to active or vice-versa.
Informally, Hanin and Rolnick [2019a] provide two main results for a randomly initialized DNN F ,with a reasonable initialisation.
Firstly, they show that
[Equation]
meaning the density of linear regions is bound above and below by some constant times the numberof neurons.
Secondly, for x ∈ [0 , 1] n in ,
[Equation]
where C > 0 depends on the distribution of biases and weights, in addition to other factors.
Inother words, the distance to the nearest boundary is bounded above and below by a constant timesthe inverse of the number of neurons.
These results stand in contrast to earlier worst case boundsthat are exponential in the number of neurons.
Hanin and Rolnick [2019a] also verify these resultsempirically to note that the constants lie in the vicinity of 1 throughout training.
Linear Regions on the Data Manifold
One important assumption in the results presented by Hanin and Rolnick [2019a] is that the input, x ,lies in a compact set K ⊂ R n in and that vol n in ( K ) is greater than 0.
Also, the theorem pertaining tothe lower bound on average distance of x to linear boundaries the input assumes the input uniformlydistributed in [0 , 1] n in .
As noted earlier, high-dimensional real world datasets, like images, lie onlow dimensional manifolds, therefore both these assumptions are false in practice.
This motivates usto study the case where the data lies on some m − dimensional submanifold of R n in , i.e. M ⊂ R n inwhere m ≪ n in .
We illustrate how this constraint effects the study of linear regions in Figure 2.
As introduced by Hanin and Rolnick [2019a], we denote the “ ( n in − k ) − dimensional piece” of
B F as B F,k .
More precisely, B F, 0 = ∅ and B F,k is recursively defined to be the set of pointsx ∈ B F \ {B F, 0 ∪ ... ∪ B F,k − 1 } with the added condition that in a neighbourhood of x the set B F,kcoincides with hyperplane of dimension n in − k .
In our setting, where the data lies on a manifold M ,we define B ′ F,k as B F,k ∩ M , and note that dim( B ′ F,k ) = m
− k
(Appendix E Proposition E.4).
Forexample, the transverse intersection (see Definition E.3) of a plane in 3D with the 2D manifold S 2is a 1D curve in S 2 and therefore has dimension 1 .
Therefore, B ′ F,k is a submanifold of dimension3 − 2 = 1 .
This imposes the restriction k ≤ m , for the intersection B F,k ∩ M to have a well definedvolume.
We first note that the definition of the determinant of the Jacobian, for a collection of neuronsz 1 , ..., z k , is different in the case when the data lies on a manifold M as opposed to in a compact setof dimension n in in R n in .
Since the determinant of the Jacobian is the quantity we utilise in our proofsand theorems repeatedly we will use the term Jacobian to refer to it for succinctness.
Intuitively,this follows from the Jacobian of a function being defined differently in the ambient space R n in asopposed to the manifold M .
In case of the former it is the volume of the paralellepiped determinedby the vectors corresponding to the directions with steepest ascent along each one of the n in axes.
Incase of the latter it is more complex and defined below.
Let H m be the m − dimensional
Hausdorffmeasure (we refer the reader to the Appendix B for background on Hausdorff measure).
The Jacobianof a function on manifold M , as defined by Krantz and Parks [2008] (Chapter 5), is as follows.
Definition 3.1.
The (determinant of) Jacobian of a function H :
M → R k , where k ≤ dim( M ) = m ,is defined as
[Equation]
P
[Equation]
a k -dimensional parallelepiped contained in T x M. (cid:111) , where D M : T x M → R k is the differential map (see Appendix B) and we use D M H ( P ) to denotethe mapping of the set P in T x M , which is a parallelepiped, to R k .
The supremum is taken over allparallelepipeds P .
We also say that neurons z 1 , ..., z k are good at x if there exists a path of neurons from z to the outputin the computational graph of F so that each neuron is activated along the path.
Our three mainresults that hold under the assumptions listed in Appendix A, each of which extend and improve uponthe theoretical results by Hanin and Rolnick [2019a], are:
Theorem 3.2.
Given F a feed-forward ReLU network with input dimension n in , output dimension 1 ,and random weights and biases.
Then for any bounded measurable submanifold M ⊂ R n in and anyk = 1 , ...., m the average ( m − k ) − dimensional volume of B F,k inside M , [Equation] [Figure]
[Equation]
where Y z 1 ,...,z k is J Mm,
H k
( x ) ρ b 1 ,...,
b k ( z 1 ( x ) , ..., z k ( x )) , times the indicator function of the event thatz j is good at x for each j = 1 , ..., k .
Here the function ρ b z 1 ,...,
b zk is the density of the joint distributionof the biases b z 1 , ..., b z k .
This change in the formula, from Theorem 3.4 by Hanin and Rolnick [2019a], is a result of thefact that z ( x ) has a different direction of steepest ascent when it is restricted to the data manifold
M , for any j .
The proof is presented in Appendix E. Formula 2 also makes explicit the fact thatthe data manifold has dimension m ≤ n in and therefore the m − k -dimensional volume is a morerepresentative measure of the linear boundaries.
Equipped with Theorem 3.2, we provide a result forthe density of boundary regions on manifold M .
Theorem 3.3.
For data sampled uniformly from a compact and measurable m dimensional manifold
M we have the following result for all k ≤ m :
[Equation]
where C grad depends on ||∇ z
( x ) || and the DNN’s architecture, C M depends on the geometry of M ,and C bias on the distribution of biases ρ b .
The constant C M is the supremum over the matrix norm of projection matrices onto the tangentspace, T x M , at any point x ∈ M .
For the Euclidean space C M is always equal to 1 and therefore theterm does not appear in the work by Hanin and Rolnick [2019a], but we cannot say the same for oursetting.
We refer the reader to Appendix F for the proof, further details, and interpretation.
Finally,under the added assumptions that the diameter of the manifold M is finite and M has polynomialvolume growth we provide a lower bound on the average distance to the linear boundary for pointson the manifold and how it depends on the geometry and dimensionality of the manifold.
Theorem 3.4.
For any point, x , chosen randomly from M , we have:
[Equation]
where C M,κ depends on the scalar curvature and dimensionality of the manifold M .
The functiondistance M is the distance on the manifold M .
This result gives us intuition on how the density of linear regions around a point depends on thegeometry of the manifold.
The constant C M,κ captures how volumes are distorted on the manifold
M as compared to the Euclidean space, for the exact definition we refer the reader to the proof in
Appendix G. For a manifold which has higher volume of a unit ball, on average, in comparison tothe Euclidean space the constant C M,κ is higher and lower when the volume of unit ball, on average,is lower than the volume of the Euclidean space.
For background on curvature of manifolds and aproof sketch we refer the reader to the Appendices B and D, respectively.
Note that the constant C Mis the same as in Theorem 3.3.
Another difference to note is that we derive a lower bound on thegeodesic distance on the manifold M and not the Euclidean distance in R k as done by Hanin and
Rolnick [2019a].
This distance better captures the distance between data points on a manifold whileincorporating the underlying structure.
In other words, this distance can be understood as how mucha data point should change to reach a linear boundary while ensuring that all the individual points onthe curve, tracing this change, are “valid” data points.
3.1 Intuition For Theoretical Results
One of the key ingredients of the proofs by Hanin and Rolnick [2019a] is the co-area formula[Krantz and Parks, 2008].
The co-area formula is applied to get a closed form representation of thek − dimensional volume of the region where any set of k neurons, z 1 , z 2 , ...,
z k is “good” in termsof the expectation over the Jacobian, in the Euclidean space.
Instead of the co-area formula we usethe smooth co-area formula [Krantz and Parks, 2008] to get a closed form representation of them − k − dimensional volume of the region intersected with manifold, M , in terms of the Jacobiandefined on a manifold (Definition 3.1).
The key difference between the two formulas is that in thesmooth co-area formula the Jacobian (of a function from the manifold M ) is restricted to the tangentplane.
While the determinant of the vanilla Jacobian measures the distortion of volume around apoint in Euclidean space the determinant of the Jacobian defined as above (Definition 3.1) measures [Figure] the distortion of volume on the manifold instead for the function with the same domain, the functionthat is 1 if the set of neurons are good and 0 otherwise.
The value of the Jacobian as defined in Definition 3.1 has the same volume as the projection of theparallelepiped defined by the gradients ∇ z ( x ) onto the tangent space (see Proposition F.1).
Thisintroduces the constant C M , defined above.
Essentially, the constant captures how the magnitudeof the gradients, ∇ z ( x ) , are modified upon being projected to the tangent plane.
Certain manifolds“shrink” vectors upon projection to the tangent plane more than others, on an average, which is afunction of their geometry.
We illustrate how two distinct manifolds “shrink” the gradients differentlyupon projection to the tangent plane as reflected in the number of linear regions on the manifolds (see Figure 11 in the appendix) for 1D manifolds.
We provide intuition for the curvature of a manifold in Appendix B, due to space constraints, which is used in the lower bound for the average distance in
Theorem 3.4.
Experiments4.1 Supervised Learning on Toy Dataset
To empirically corroborate our theoretical results, we calculate the number of linear regions andaverage distance to the linear boundary, bounds for which are presented in the theorems above, on aregression task for two different manifolds.
To achieve this we define two similar regression taskswhere the data is sampled from two different manifolds with different geometries.
We parameterizethe first task, a unit circle without its north and south poles, by ψ circle : ( − π, π ) → R 2 whereψ circle ( θ ) =
(cos θ, sin θ ) and θ is the angle made by the vector from the origin to the point withrespect to the x-axis.
We set the target function for regression task to be a periodic function in θ .
The target is defined as z ( θ ) = a sin( νθ ) where a is the amplitude and ν is the frequency (Figure 3).
DNNs have difficulty learning periodic functions [Ziyin et al., 2020].
The motivation behind this is topresent the DNN with a challenging task where it has to learn the underlying structure of the data.
Moreover the DNN will have to split the circle into linear regions.
For the second regression task, atractrix is parametrized by ψ tractrix : R 1 → R 2 where ψ tractrix ( y ) =
( t − tanh t, sech t ) (see Figure 3).
We assign a target function z ( t ) =
a sin( νt ) .
For the purposes of our study we restrict the domain ofψ tractrix to ( − 3 , 3) .
This allows us to observe effects of varying data geometry across the manifolds.
The results, averaged over 20 runs, are presented in Figures 4 and 5.
We note that C M is smallerfor Sphere (based on Figure 4) and the curvature is positive whilst C M is larger for tractrix and thecurvature is negative.
Both of these constants (curvature and C M ) contribute to the lower boundin Theorem 3.4.
Similarly, we show results of number of linear regions divided by the number ofneurons upon changing architectures, consequently the number of neurons, for the two manifolds in
Figure 8, averaged over 30 runs.
Note that this experiment observes the effect of C M × C grad , sincechanging the architecture also changes C grad and the variation in C grad is quite low in magnitude asobserved empirically by Hanin and Rolnick [2019a].
The empirical observations are consistent withour theoretical results.
We observe that the number of linear regions starts off close to # neurons andremains close throughout the training process for both the manifolds.
This supports our theoretical results (Theorem 3.3) that the constant C M , which is distinct across the two manifolds, affects thenumber of linear regions throughout training.
The tractrix has a higher value of C M and that isreflected in both Figures 4 and 5.
Note that its relationship is inverse to the average distance to theboundary region, as per Theorem 3.4, and it is reflected as training progresses in Figure 5.
This isdue to different “shrinking” of vectors upon being projected to the tangent space (Section 3.1).
4.2 Varying n in
To empirically corroborate the results of Theorems 2 and 3 we vary the dimension n in while keepingm constant.
We achieve this by counting the number of linear regions and the average distance toboundary region on the 1D circle as we vary the input dimension in steps of 5.
We draw samples of 1Dcircles in R n in by randomly choosing two perpendicular basis vectors.
We then train a neural networkwith the same architecture as in the previous section on the periodic target function ( a sin( νθ ) ) asdefined above.
The results in Figure 6 shows that the quantities stay proportional to # neurons , anddo not vary as n in is increased, as predicted by our theoretical results.
This stands in contrast to theresults by Hanin and Rolnick [2019a] where the upper and lower bounds both grow exponentiallywith n in for the number of linear regions in a compact set of R n in .
Further details are in Appendix H.
Our goal with this experiment is to study how overfitting relates to the number of linear regions ofdeep ReLU networks, in addition to observing the density of linear regions for very high dimensionalimage data that lies on a low dimensional manifold.
To discover latent low dimensional underlyingstructure of data we employ a GAN.
Adversarial training of GANs can be effectively applied to learna mapping from a low dimensional latent space to high dimensional data [Goodfellow et al., 2014].
The generator is a neural network that maps g : R k
→ R n in .
Recently, Karras et al.
[2019] introduceda new generator, StyleGAN, that interpolates better, meaning that it can disentangle the factors ofvariation in the dataset.
As a follow up, Karras et al.
[2020a] train the StyleGAN in a data efficientmanner on the MetFaces dataset.
We train a deep ReLU network on the MetFaces dataset withrandom labels (chosen from 0 , 1 ) with cross entropy loss.
As noted by Zhang et al.
[2017], trainingwith random labels can lead to the DNN memorizing the entire dataset with poor generalization.
We compare the log density of number of linear regions on a curve on the manifold with a straightline off the manifold (see Figure 9).
This leads to two observations: 1) the density of the linearregions decreases as training progresses, in case of overfitting, which is in contrast to the scenariowithout overfitting [Hanin and Rolnick, 2019a] and it ties the pathological behavior of deep ReLUnetworks to density of linear regions, 2) the density of linear regions is significantly lower on the datamanifold and devising methods to “concentrate” these linear regions on the manifold is a promisingresearch direction.
That could lead to increased expressivity for the same number of parameters.
Weprovide further experimental details in Appendix I.
Discussion and Conclusions
There is significant amounts of work in both supervised and unsupervised learning settings fornon-Euclidean data [Bronstein et al., 2017].
Despite these empirical results most theoretical analysisremains agnostic to data geometry, with a few prominent exceptions [Cloninger and Klock, 2020,
Shaham et al., 2015, Schmidt-Hieber, 2019].
We incorporate the idea of data geometry into measuringthe effective approximation capacity of DNNs. We derive average bounds on the density of boundaryregions and distance from the linear boundary under the added assumption that the data is sampledfrom a low dimensional manifold.
Our experimental results corroborate our theoretical results.
We also present insights into overfitting in high dimensional datasets where the data lies on alow dimensional manifold.
Estimating the geometry, dimensionality and curvature, of these imagemanifolds accurately is a problem that remains largely unsolved [Brehmer and Cranmer, 2020, PerraulJoncas and Meila, 2013], which limits our inferences on high dimensional dataset to observations thatguide future research.
We note that proving a lower bound on the number of linear regions, as doneby Hanin and Rolnick [2019a], for the manifold setting remains open.
Our work opens up avenuesfor further research that combines model geometry and data geometry and can lead to empiricalresearch geared towards developing DNN architectures for high dimensional datasets that lie on alow dimensional manifold.
Figure 6: We observe that as the dimension n in is increased, while keeping the manifold dimension constant, the number of linear regions remains proportional to number of neurons (26).
Figure 7: We observe that as the dimension n in is increased, while keeping the manifold dimension constant, the average distance varies very little.
Figure 8: The effects of changing the architecture on the number of linear regions.
We observe that the value of C M effects the number of linear regions proportionally.
The number of hidden units for three layer networks are in the legend along with the data manifold.
Figure 9: We observe that the log density of number of linear regions is lower on the manifold (blue) as compared to off the manifold (green).
As training progresses, in contrast to previous examples with generalization, the log density decreases.
This is for the MetFaces dataset.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work is primarily theoretical with few toy experiments we do not see its applicability (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix
A for a list
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix
J
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See experimental sections in the Appendix and main body
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Except for the cases where there are multiple graphs that
are overlapping (Figure 6,7, 8) because it would make interpreting them difficult.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Appendix J
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
[Table]