Effects of Data Geometry in Early Deep Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep neural networks can approximate functions on different types of data, fromimages to graphs, with varied underlying structure.
This underlying structure canbe viewed as the geometry of the data manifold.
By extending recent advancesin the theoretical understanding of neural networks, we study how a randomlyinitialized neural network with piece-wise linear activation splits the data manifoldinto regions where the neural network behaves as a linear function.
We derivebounds on the density of boundary of linear regions and the distance to theseboundaries on the data manifold.
This leads to insights into the expressivityof randomly initialized deep neural networks on non-Euclidean data sets.
Weempirically corroborate our theoretical results using a toy supervised learningproblem.
Our experiments demonstrate that number of linear regions varies acrossmanifolds and the results hold with changing neural network architectures.
Wefurther demonstrate how the complexity of linear regions is different on the lowdimensional manifold of images as compared to the Euclidean space, using the
MetFaces dataset.
Introduction
The capacity of Deep Neural Networks (DNNs) to approximate arbitrary functions given sufficienttraining data in the supervised learning setting is well known [Cybenko, 1989, Hornik et al., 1989,
Anthony and Bartlett, 1999].
Several different theoretical approaches have emerged that study theeffectiveness and pitfalls of deep learning.
These studies vary in their treatment of neural networksand the aspects they study range from convergence [Allen-Zhu et al., 2019, Goodfellow and Vinyals,2015], generalization [Kawaguchi et al., 2017, Zhang et al., 2017, Jacot et al., 2018, Sagun et al.
,2018], function complexity [Mont´ufar et al., 2014, Mhaskar and Poggio, 2016], adversarial attacks[Szegedy et al., 2014, Goodfellow et al., 2015] to representation capacity [Arpit et al., 2017].
Somerecent theories have also been shown to closely match empirical observations [Poole et al., 2016,
Hanin and Rolnick, 2019b, Kunin et al., 2020].
One approach to studying DNNs is to examine how the underlying structure, or geometry, of the datainteracts with learning dynamics.
The manifold hypothesis states that high-dimensional real worlddata typically lies on a low dimensional manifold [Tenenbaum, 1997, Carlsson et al., 2007, Feffermanet al., 2013].
Empirical studies have shown that DNNs are highly effective in deciphering thisunderlying structure by learning intermediate latent representations [Poole et al., 2016].
The abilityof DNNs to “flatten” complex data manifolds, using composition of seemingly simple piece-wiselinear functions, appears to be unique [Brahma et al., 2016, Hauser and Ray, 2017].
DNNs with piece-wise linear activations, such as ReLU [Nair and Hinton, 2010], divide the inputspace into linear regions, wherein the DNN behaves as a linear function [Mont´ufar et al., 2014].
Thedensity of these linear regions serves as a proxy for the DNN’s ability to interpolate a complex datalandscape and has been the subject of detailed studies [Mont´ufar et al., 2014, Telgarsky, 2015, Serraet al., 2018, Raghu et al., 2017].
The work by Hanin and Rolnick [2019a] on this topic stands out because they derive bounds on the average number of linear regions and verify the tightness of thesebounds empirically for deep ReLU networks, instead of larger bounds that rarely materialize.
Haninand Rolnick [2019a] conjecture that the number of linear regions correlates to the expressive powerof randomly initialized DNNs with piece-wise linear activations.
However, they assume that thedata is uniformly sampled from the Euclidean space R d , for some d .
By combining the manifoldhypothesis with insights from Hanin and Rolnick [2019a], we are able to go further in estimating thenumber of linear regions and the average distance from linear boundaries .
We derive bounds on howthe geometry of the data manifold affects the aforementioned quantities.
To corroborate our theoretical bounds with empirical results, we design a toy problem where theinput data is sampled from two distinct manifolds that can be represented in a closed form.
We countthe exact number of linear regions and the average distance to the boundaries of linear regions onthese two manifolds that a neural network divides the two manifolds into.
We demonstrate how thenumber of linear regions and average distance varies for these two distinct manifolds.
These resultsshow that the number of linear regions on the manifold do not grow exponentially with the dimensionof input data.
Our experiments do not provide estimates for theoretical constants, as in most deeplearning theory, but demonstrate that the number of linear regions change as a consequence of theseconstants.
We also study linear regions of deep ReLU networks for high dimensional data that lieson a low dimensional manifold with unknown structure and how the number of linear regions varyon and off this manifold, which is a more realistic setting.
To achieve this we present experimentsperformed on the manifold of natural face images.
We sample data from the image manifold usinga generative adversarial network (GAN)
[Goodfellow et al., 2014] trained on the curated imagesof paintings.
Specifically, we generate images using the pre-trained StyleGAN [Karras et al., 2019,2020b] trained on the curated MetFaces dataset [Karras et al., 2020a].
We generate curves on theimage manifold of faces, using StyleGAN, and report how the density of linear regions varies on andoff the manifold.
These results shed new light on the geometry of deep learning over structured datasets by taking a data intrinsic approach to understanding the expressive power of DNNs.
Preliminaries and Background
Our goal is to understand how the underlying structure of real world data matters for deep learning.
We first provide the mathematical background required to model this underlying structure as thegeometry of data.
We then provide a summary of previous work on understanding the approximationcapacity of deep ReLU networks via the complexity of linear regions.
For the details on how ourwork fits into one of the two main approaches within the theory of DNNs, from the expressive powerperspective or from the learning dynamics perspective, we refer the reader to Appendix C.
2.1 Data Manifold and Definitions
Suppose each point corresponds to an image of a face on this 2-torus.
We can chart two curves: one straight line cutting across the 3D space and another curve that stays on the torus.
Images corresponding to the points on the torus will have a smoother variation in style and shape whereas there will be images corresponding to points on the straight line that are not faces.
We use the example of the MetFaces dataset [Karras et al., 2020a] to illustrate how data lies on a lowdimensional manifold.
The images in the dataset are 1028 × 1028 × 3 dimensional.
By contrast,the number of realistic dimensions along which they vary are limited, e.g. painting style, artist, sizeand shape of the nose, jaw and eyes, background, clothing style; in fact, very few 1028 × 1028 ×
3dimensional images correspond to realistic faces.
We illustrate how this affects the possible variations in the data in Figure 1.
A manifold formalises the notion of limited variations in high dimensionaldata.
One can imagine that there exists an unknown function f : X → Y from a low dimensionalspace of variations, to a high dimensional space of the actual data points.
Such a function f : X → Y ,from one open subset X ⊂ R m , to another open subset Y ⊂ R k , is a diffeomorphism if f is bijective,and both f and f − 1 are differentiable (or smooth).
Therefore, a manifold is defined as follows.
Definition 2.1.
Let k, m ∈ N 0 .
A subset M ⊂ R k is called a smooth m -dimensional submanifoldof R k (or m -manifold in R k ) iff every point x ∈ M has an open neighborhood U ⊂ R k such that
U ∩ M is diffeomorphic to an open subset Ω ⊂ R m .
A diffeomorphism (i.e. differentiable mapping),
[Equation]
Ω is called a coordinate chart of M and the inverse,
[Equation]
is called a smooth parametrization of U ∩ M .
For the MetFaces dataset example, suppose there are 10 dimensions along which the images vary.
Further assume that each variation can take a value continuously in some interval of R .
Then thesmooth parametrization would map f : Ω ∩ R 10 → M ∩ R 1028 × 1028 × 3 .
This parametrization and itsinverse are unknown in general and computationally very difficult to estimate in practice.
There are similarities in how geometric elements are defined for manifolds and Euclidean spaces.
A smooth curve, on a manifold M , γ : I → M is defined from an interval I to the manifold M asa function that is differentiable for all t ∈ I , just as for Euclidean spaces.
The shortest such curvebetween two points on a manifold is no longer a straight line, but is instead a geodesic .
One recurringgeometric element, which is unique to manifolds and stems from the definition of smooth curves, isthat of a tangent space , defined as follows.
Definition 2.2.
Let M be an m -manifold in R k and x ∈ M be a fixed point.
A vector v ∈ R k is calleda tangent vector of M at x if there exists a smooth curve γ : I → M
such that γ (0) =
x, ˙ γ (0) =
vwhere ˙ γ ( t ) is the derivative of γ at t .
The set T x M := { ˙ γ (0) | γ : R → M is smooth γ (0) =
x } oftangent vectors of M at x is called the tangent space of M at x .
In simpler terms, the plane tangent to the manifold M at point x is called the tangent space anddenoted by by T x M .
Consider the upper half of a 2-sphere, S 2 ⊂ R 3 , which is a 2-manifold in R 3 .
The tangent space at a fixed point x ∈ S 2 is the 2D plane perpendicular to the vector x and tangentialto the surface of the sphere that contains the point x .
For additional background on manifolds werefer the reader to Appendix B.
2.2 Linear Regions of Deep ReLU Networks
The higher the density of these linear regions the more complex a function a DNN can approximate.
For example, a sin curve in the range [0 , 2 π ] is better approximated by 4 piece-wise linear regions asopposed to 2.
To clarify this further, with the 4 “optimal” linear regions [0 , π/ 2) , [ π/ 2 , π ) , [ π, 3 π/ 2) ,and [3 π/ 2 , 2 π ] a function could approximate the sin curve better than any 2 linear regions.
In otherwords, higher density of linear regions allows a DNN to approximate the variation in the curve better.
We define the notion of boundary of a linear regions in this section and provide an overview ofprevious results.
We consider a neural network, F , which is a composition of activation functions.
Inputs at each layerare multiplied by a matrix, referred to as the weight matrix, with an additional bias vector that isadded to this product.
We limit our study to ReLU activation function [Nair and Hinton, 2010], whichis piece-wise linear and one of the most popular activation functions being applied to various learningtasks on different types of data like text, images, signals etc.
We further consider DNNs that mapinputs, of dimension n in , to scalar values.
Therefore, F : R n in → R is defined as,
[Equation]
where W l ∈ M n l × n l − 1 is the weight matrix for the l th hidden layer, n l is the number of neurons inthe l th hidden layer, B l ∈ R n l is the vector of biases for the l th hidden layer, n 0 = n in and σ : R → Ris the activation function.
For a neuron z in the l th layer we denote the pre-activation of this neuron,for given input x ∈ R n in , as z l ( x ) .
For a neuron z in the layer l we have
[Equation]
for l > 1 (for the base case l = 1 we have z ( x ) =
W 1 ,z x ) where W l − 1 ,z is the row of weights, in theweight matrix of the l th layer, W l , corresponding to the neuron z .
We use W z to denote the
vector brevity, omitting the layer index l the
We also use to denote
Neural networks with piece-wise linear activations are piece-wise linear on the input space
[Mont ´ ufaret al., 2014].
Suppose for some fixed y ∈ R n in as x → y if we have z ( x ) → − b z then we observe adiscontinuity in the gradient ∇ x σ ( b z + W z z ( x )) at y .
Intuitively, this is because x is approachingthe boundary of the linear region of the function defined by the output of z .
Therefore, the boundaryof linear regions, for a feed forward neural network F , is defined as:
[Equation]
Hanin and Rolnick [2019a] argue that an important generalization for the approximation capacityof a neural network F is the ( n in − 1) − dimensional volume density of linear regions defined asvol n in − 1 ( B F ∩ K ) / vol n in ( K ) , for a bounded set K ⊂ R n in .
This quantity serves as a proxy fordensity of linear regions and therefore the expressive capacity of DNNs.
Intuitively, higher density oflinear boundaries means higher capacity of the DNN to approximate complex non-linear functions.
The quantity is applied to lower bound the distance between a point x ∈ K and the set B F , which is
[Equation] [Equation]
which measures the sensitivity over neurons at a given input.
The above quantity measures how “far”the input is from flipping any neuron from inactive to active or vice-versa.
Informally, Hanin and Rolnick [2019a] provide two main results for a randomly initialized DNN F ,with a reasonable initialisation.
Firstly, they show that
[Equation]
meaning the density of linear regions is bound above and below by some constant times the numberof neurons.
Secondly, for x ∈ [0 , 1] n in ,
[Equation]
where C > 0 depends on the distribution of biases and weights, in addition to other factors.
Inother words, the distance to the nearest boundary is bounded above and below by a constant timesthe inverse of the number of neurons.
These results stand in contrast to earlier worst case boundsthat are exponential in the number of neurons.
Hanin and Rolnick [2019a] also verify these resultsempirically to note that the constants lie in the vicinity of 1 throughout training.
Linear Regions on the Data Manifold
One important assumption in the results presented by Hanin and Rolnick [2019a] is that the input, x ,lies in a compact set K ⊂ R n in and that vol n in ( K ) is greater than 0.
Also, the theorem pertaining to
[Equation]
the lower bound on average distance of x to linear boundaries the input assumes the input uniformlydistributed in [0 , 1] n in .
As noted earlier, high-dimensional real world datasets, like images, lie onlow dimensional manifolds, therefore both these assumptions are false in practice.
This motivates usto study the case where the data lies on some m − dimensional submanifold of R n in , i.e. M ⊂ R n inwhere m ≪ n in .
We illustrate how this constraint effects the study of linear regions in Figure 2.
As introduced by Hanin and Rolnick [2019a], we denote the “ ( n in − k ) − dimensional piece” of
B F as B F,k .
More precisely, B F, 0 = ∅ and B F,k is recursively defined to be the set of pointsx ∈ B F \ {B F, 0 ∪ ... ∪ B F,k − 1 } with the added condition that in a neighbourhood of x the set B F,kcoincides with hyperplane of dimension n in − k .
We provide a detailed and formal definition for B F,kwith intuition in Appendix E.
In our setting, where the data lies on a manifold M , we define B ′ F,kas B F,k ∩ M , and note that dim( B ′ F,k ) = m
− k
(Appendix E Proposition E.4).
For example, thetransverse intersection (see Definition E.3) of a plane in 3D with the 2D manifold S 2 is a 1D curve in
S 2 and therefore has dimension 1 .
Therefore, B ′ F,k is a submanifold of dimension 3 − 2 = 1 .
Thisimposes the restriction k ≤ m , for the intersection B F,k ∩ M to have a well defined volume.
We first note that the definition of the determinant of the Jacobian, for a collection of neuronsz 1 , ..., z k , is different in the case when the data lies on a manifold M as opposed to in a compact setof dimension n in in R n in .
Since the determinant of the Jacobian is the quantity we utilise in our proofsand theorems repeatedly we will use the term Jacobian to refer to it for succinctness.
Intuitively,this follows from the Jacobian of a function being defined differently in the ambient space R n in asopposed to the manifold M .
In case of the former it is the volume of the paralellepiped determinedby the vectors corresponding to the directions with steepest ascent along each one of the n in axes.
Incase of the latter it is more complex and defined below.
Let H m be the m − dimensional
Hausdorffmeasure (we refer the reader to the Appendix B for background on Hausdorff measure).
The Jacobianof a function on manifold M , as defined by Krantz and Parks [2008] (Chapter 5), is as follows.
Definition 3.1.
The (determinant of) Jacobian of a function H :
M → R k , where k ≤ dim( M ) = m ,is defined as
[Equation]
contained in
[Equation]
where D M : T x M → R k is the differential map (see Appendix B) and we use D M H ( P ) to denotethe mapping of the set P in T x M , which is a parallelepiped, to R k .
The supremum is taken over allparallelepipeds P .
We also say that neurons z 1 , ..., z k are good at x if there exists a path of neurons from z to the outputin the computational graph of F so that each neuron is activated along the path.
Our three mainresults that hold under the assumptions listed in Appendix A, each of which extend and improve uponthe theoretical results by Hanin and Rolnick [2019a], are:
Theorem 3.2.
Given F a feed-forward ReLU network with input dimension n in , output dimension 1 ,and random weights and biases.
Then for any bounded measurable submanifold M ⊂ R n in and anyk = 1 , ...., m the average ( m − k ) − dimensional volume of B F,k inside M ,
[Equation]
where Y z 1 ,...,z k is J Mm,
H k
( x ) ρ b 1 ,...,
b k ( z 1 ( x ) , ..., z k ( x )) , times the indicator function of the event thatz j is good at x for each j = 1 , ..., k .
Here the function ρ b z 1 ,...,
b zk is the density of the joint distributionof the biases b z 1 , ..., b z k .
This change in the formula, from Theorem 3.4 by Hanin and Rolnick [2019a], is a result of thefact that z ( x ) has a different direction of steepest ascent when it is restricted to the data manifold
M , for any j .
The proof is presented in Appendix E. Formula 3 also makes explicit the fact thatthe data manifold has dimension m ≤ n in and therefore the m − k -dimensional volume is a morerepresentative measure of the linear boundaries.
Equipped with Theorem 3.2, we provide a result forthe density of boundary regions on manifold M .
Theorem 3.3.
For data sampled uniformly from a compact and measurable m dimensional manifold
M we have the following result for all k ≤ m :
[Equation]
where C grad depends on ||∇ z
( x ) || and the DNN’s architecture, C M depends on the geometry of M ,and C bias on the distribution of biases ρ b .
The constant C M is the supremum over the matrix norm of projection matrices onto the tangentspace, T x M , at any point x ∈ M .
For the Euclidean space C M is always equal to 1 and therefore theterm does not appear in the work by Hanin and Rolnick [2019a], but we cannot say the same for oursetting.
We refer the reader to Appendix F for the proof, further details, and interpretation.
Finally,under the added assumptions that the diameter of the manifold M is finite and M has polynomialvolume growth we provide a lower bound on the average distance to the linear boundary for pointson the manifold and how it depends on the geometry and dimensionality of the manifold.
Theorem 3.4.
For any point, x , chosen randomly from M , we have:
[Equation]
where C M,κ depends on the scalar curvature, the input dimension and the dimensionality of themanifold M .
The function distance M is the distance on the manifold M .
This result gives us intuition on how the density of linear regions around a point depends on thegeometry of the manifold.
The constant C M,κ captures how volumes are distorted on the manifold
M as compared to the Euclidean space, for the exact definition we refer the reader to the proof in
Appendix G. For a manifold which has higher volume of a unit ball, on average, in comparison tothe Euclidean space the constant C M,κ is higher and lower when the volume of unit ball, on average,is lower than the volume of the Euclidean space.
For background on curvature of manifolds and aproof sketch we refer the reader to the Appendices B and D, respectively.
Note that the constant C Mis the same as in Theorem 3.3.
Another difference to note is that we derive a lower bound on thegeodesic distance on the manifold M and not the Euclidean distance in R k as done by Hanin and
Rolnick [2019a].
This distance better captures the distance between data points on a manifold whileincorporating the underlying structure.
In other words, this distance can be understood as how mucha data point should change to reach a linear boundary while ensuring that all the individual points onthe curve, tracing this change, are “valid” data points.
3.1 Intuition For Theoretical Results
One of the key ingredients of the proofs by Hanin and Rolnick [2019a] is the co-area formula[Krantz and Parks, 2008].
The co-area formula is applied to get a closed form representation of thek − dimensional volume of the region where any set of k neurons, z 1 , z 2 , ...,
z k is “good” in termsof the expectation over the Jacobian, in the Euclidean space.
Instead of the co-area formula we usethe smooth co-area formula [Krantz and Parks, 2008] to get a closed form representation of them − k − dimensional volume of the region intersected with manifold, M , in terms of the Jacobiandefined on a manifold (Definition 3.1).
The key difference between the two formulas is that in thesmooth co-area formula the Jacobian (of a function from the manifold M ) is restricted to the tangentplane.
While the determinant of the “vanilla” Jacobian measures the distortion of volume around apoint in Euclidean space the determinant of the Jacobian defined as above (Definition 3.1) measuresthe distortion of volume on the manifold instead for the function with the same domain, the functionthat is 1 if the set of neurons are good and 0 otherwise.
The value of the Jacobian as defined in Definition 3.1 has the same volume as the projection ofthe parallelepiped defined by the gradients ∇ z ( x ) onto the tangent space (see Proposition F.1 in
Appendix).
This introduces the constant C M , defined above.
Essentially, the constant captureshow the magnitude of the gradients, ∇ z ( x ) , are modified upon being projected to the tangent plane.
Certain manifolds “shrink” vectors upon projection to the tangent plane more than others, on anaverage, which is a function of their geometry.
We illustrate how two distinct manifolds “shrink”the gradients differently upon projection to the tangent plane as reflected in the number of linearregions on the manifolds (see Figure 11 in the appendix) for 1D manifolds.
We provide intuitionfor the curvature of a manifold in Appendix B, due to space constraints, which is used in the lowerbound for the average distance in Theorem 3.4.
The constant C M,κ depends on the curvature as thesupremum of a polynomial whose coefficients depend on the curvature, with order at most n in and atleast n in − m .
Note that despite this dependence on the ambient dimension, there are other geometricconstants in this polynomial (see Appendix G).
Finally, we also provide a simple example as to howthis constant varies with n in and m , for a simple and contrived example, in Appendix G.1.
[Figure]
Experiments
Linear Regions on a 1D Curve To empirically corroborate our theoretical results, we calculate thenumber of linear regions and average distance to the linear boundary on 1D curves for regressiontasks in two settings.
The first is for 1D manifolds embedded in 2D and higher dimensions and thesecond is for the high-dimensional data using the MetFaces dataset.
We use the same algorithm, forthe toy problem and the high-dimensional dataset, to find linear regions on 1D curves.
We calculatethe exact number of linear regions for a 1D curve in the input space, x : I → R n in where I is aninterval in real numbers, by finding the points where z ( x ( t ))
=
b z for every neuron z .
The solutionsthus obtained gives us the boundaries for neurons on the curve x .
We obtain these solutions by usingthe programmatic activation of every neuron and using the sequential least squares programming(SLSQP) algorithm
[Kraft, 1988] to solve for | z ( x ( t )) − b z | = 0 for t ∈ I .
In order to obtain theprogrammatic activation of a neuron we construct a Deep ReLU network as defined in Equation 2.
We do so for all the neurons for a given DNN with fixed weights.
Supervised Learning on Toy Dataset We define two similar regression tasks where the data issampled from two different manifolds with different geometries.
We parameterize the first task, a unitcircle without its north and south poles, by ψ circle : ( − π, π ) →
R 2 where ψ circle ( θ ) =
(cos θ, sin θ )and θ is the angle made by the vector from the origin to the point with respect to the x-axis.
Weset the target function for regression task to be a periodic function in θ .
The target is defined asz ( θ ) =
a sin( νθ ) where a is the amplitude and ν is the frequency (Figure 3).
DNNs have difficultylearning periodic functions [Ziyin et al., 2020].
The motivation behind this is to present the DNN witha challenging task where it has to learn the underlying structure of the data.
Moreover the DNN willhave to split the circle into linear regions.
For the second regression task, a tractrix is parametrizedby ψ tractrix : R 1 → R 2 where ψ tractrix ( y ) =
( y − tanh y, sech y )
(see Figure 3).
We assign a targetfunction z ( t ) =
a sin( νt ) .
For the purposes of our study we restrict the domain of ψ tractrix to ( − 3 , 3) .
We choose ν
so as to ensure that the number of peaks and troughs, 6, in the periodic target functionare the same for both the manifolds.
This ensures that the domains of both the problems have lengthclose to 6.28.
Further experimental details are in Appendix H.
The results, averaged over 20 runs, are presented in Figures 4 and 5.
We note that C M is smallerfor Sphere (based on Figure 4) and the curvature is positive whilst C M is larger for tractrix and thecurvature is negative.
Both of these constants (curvature and C M ) contribute to the lower boundin Theorem 3.4.
Similarly, we show results of number of linear regions divided by the number ofneurons upon changing architectures, consequently the number of neurons, for the two manifolds in
Figure 8, averaged over 30 runs.
Note that this experiment observes the effect of C M × C grad , sincechanging the architecture also changes C grad and the variation in C grad is quite low in magnitude asobserved empirically by Hanin and Rolnick [2019a].
The empirical observations are consistent withour theoretical results.
We observe that the number of linear regions starts off close to # neurons andremains close throughout the training process for both the manifolds.
This supports our theoreticalresults (Theorem 3.3) that the constant C M , which is distinct across the two manifolds, affects thenumber of linear regions throughout training.
The tractrix has a higher value of C M and that isreflected in both Figures 4 and 5.
Note that its relationship is inverse to the average distance to the boundary region, as per Theorem 3.4, and it is reflected as training progresses in Figure 5.
This isdue to different “shrinking” of vectors upon being projected to the tangent space (Section 3.1).
Varying n in
To empirically corroborate the results of Theorems 2 and 3 we vary the dimension n inwhile keeping m constant.
We achieve this by counting the number of linear regions and the averagedistance to boundary region on the 1D circle as we vary the input dimension in steps of 5.
We drawsamples of 1D circles in R n in by randomly choosing two perpendicular basis vectors.
We then train anetwork with the same architecture as the previous section on the periodic target function ( a sin( νθ ) )as defined above.
The results in Figure 6 shows that the quantities stay proportional to # neurons ,and do not vary as n in is increased, as predicted by our theoretical results.
Our empirical studyasserts how the relevant upper and lower bounds, for the setting where data lies on a low-dimensionalmanifold, does not grow exponentially with n in for the density of linear regions in a compact set of 303 R n in but instead depend on the intrinsic dimension.
Further details are in Appendix H.
MetFaces: High Dimensional Dataset Our goal with this experiment is to study how the densityof linear regions varies across a low dimensional manifold and the input space.
To discover latentlow dimensional underlying structure of data we employ a GAN.
Adversarial training of GANs canbe effectively applied to learn a mapping from a low dimensional latent space to high dimensionaldata [Goodfellow et al., 2014].
The generator is a neural network that maps g : R k
→ R n in .
We traina deep ReLU network on the MetFaces dataset with random labels (chosen from 0 , 1 ) with crossentropy loss.
As noted by Zhang et al.
[2017], training with random labels can lead to the DNNmemorizing the entire dataset.
We compare the log density of number of linear regions on a curve on the manifold with a straight lineoff the manifold.
We generate these curves using the data sampled by the StyleGAN by [Karras et al.,2020a].
Specifically, for each curve we sample a random pair of latent vectors: z 1 , z 2 ∈ R k , thisgives us the start and end point of the curve using the generator g ( z 1 ) and g ( z 2 ) .
We then generate100 images to approximate a curve connecting the two images on the image manifold in a piece-wisemanner.
We do so by taking 100 points on the line connecting z 1 and z 2 in the latent space that areevenly spaced and generate an image from each one of them.
Therefore, the i th image is generated as:z ′ i
=
g
(((100 − i ) ×
z 1 + i × z 2 ) / 100) , using the StyleGAN generator g .
We qualitatively verify theimages to ensure that they lie on the manifold of images of faces.
The straight line, with two fixedpoints g ( z 1 ) and g ( z 2 ) , is defined as x ( t ) =
(1 − t ) g
( z 1 ) + tg ( z 2 ) with t ∈ [0 , 1] .
The approximatedcurve on the manifold is defined as x ′ ( t ) =
(1 − t ) g
( z ′ i )
+ tg
( z ′ i +1 ) where i = floor (100 t ) .
Wethen apply the method from Section 4 to obtain the number of linear regions on these curves.
The results are presented in Figure 9.
This leads us to the key observation: the density of linearregions is significantly lower on the data manifold and devising methods to “concentrate” these linearregions on the manifold is a promising research direction.
That could lead to increased expressivityfor the same number of parameters.
We provide further experimental details in Appendix I.
Discussion and Conclusions
There is significant work in both supervised and unsupervised learning settings for non-Euclideandata [Bronstein et al., 2017].
Despite these empirical results most theoretical analysis is agnosticto data geometry, with a few prominent exceptions [Cloninger and Klock, 2020, Shaham et al.
,2015, Schmidt-Hieber, 2019].
We incorporate the idea of data geometry into measuring the effectiveapproximation capacity of DNNs, deriving average bounds on the density of boundary regionsand distance from the boundary when the data is sampled from a low dimensional manifold.
Ourexperimental results corroborate our theoretical results.
We also present insights into expressivityof DNNs on low dimensional manfiolds for the case of high dimensional datasets.
Estimating thegeometry, dimensionality and curvature, of these image manifolds accurately is a problem thatremains largely unsolved [Brehmer and Cranmer, 2020, Perraul-Joncas and Meila, 2013], whichlimits our inferences on high dimensional dataset to observations that guide future research.
We notethat proving a lower bound on the number of linear regions, as done by Hanin and Rolnick [2019a],for the manifold setting remains open.
Our work opens up avenues for further research that combinesmodel geometry and data geometry and can lead to empirical research geared towards developing
DNN architectures for high dimensional datasets that lie on a low dimensional manifold.
Figure 6: We observe that as the dimension n in is increased, while keeping the manifold dimension constant, the number of linear regions remains proportional to number of neurons (26).
Figure 7: We observe that as the dimension n in is increased, while keeping the manifold dimension constant, the average distance varies very little.
Figure 8: The effects of changing the architecture on the number of linear regions.
We observe that the value of C M effects the number of linear regions proportionally.
The number of hidden units for three layer networks are in the legend along with the data manifold.
Figure 9: We observe that the log density of number of linear regions is lower
) as off
This is for the MetFaces dataset.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
Our workis primarily theoretical with few toy experiments we do not see its applicability(d) Have you read the ethics review guidelines and ensured that your paper conforms to
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix
A for a list
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix J
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See experimental sections in the Appendix and main body
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] Except for the cases where there are multiple graphs that
are overlapping (Figure 6,7, 8) because it would make interpreting them difficult.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix J
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]