On Deep Generative Models for Approximation and Estimation of Distributions on Manifolds
Abstract
Generative networks have experienced great empirical successes in distribution learning.
Many existing experiments have demonstrated that generative networks can generate high-dimensional complex data from a low-dimensional easy-tosample distribution.
However, this phenomenon can not be justified by existing theories.
The widely held manifold hypothesis speculates that real-world data sets, such as natural images and signals, exhibit low-dimensional geometric structures.
In this paper, we take such low-dimensional data structures into consideration by assuming that data distributions are supported on a low-dimensional manifold.
We prove statistical guarantees of generative networks under the Wasserstein-1 loss.
We show that the Wasserstein-1 loss converges to zero at a fast rate depending on the intrinsic dimension instead of the ambient data dimension.
Our theory leverages the low-dimensional geometric structures in data sets and justifies the practical power of generative networks.
We require no smoothness assumptions on the data distribution which is desirable in practice.
Introduction
Deep generative models, such as generative adversarial networks (GANs)
[Goodfellow et al., 2014,
Arjovsky et al., 2017] and variational autoencoder [Kingma and Welling, 2013, Mohamed and
Wierstra, 2014] utilize neural networks to generate new samples which follow the same distribution as the training data.
They have been successful in many applications including producing photorealistic images, improving astronomical images and modding video games [Reed et al., 2016, Ledig et al., 2017, Schawinski et al., 2017, Brock et al., 2018, Volz et al., 2018, Radford et al., 2015, Salimans et al., 2016].
To estimate a data distribution Q , generative models solve the following optimization problem
[Equation]
where ρ is an easy-to-sample distribution, G is a class of generating functions, discrepancy is some distance function between distributions, and p g θ q 7 ρ denotes the pushforward measure of ρ under g θ .
In particular, when we obtain a sample z from ρ , we let g
θ p z q be the generated sample, whose distribution follows p g θ q 7 ρ .
There are many choices of the discrepancy function in literature among which Wasserstein distance attracts much attention.
The so-called Wasserstein generative models [Arjovsky et al., 2017] consider the Wasserstein-1 distance defined as
[Equation]
where µ, ν are two distributions and Lip 1 p R D q consists of 1 -Lipschitz functions on R D .
The formulation in (2) is known as the Kantorovich-Rubinstein dual form of Wasserstein-1 distance and can be viewed as an integral probability metric
[Müller, 1997].
In deep generative models, the function class G is often parameterized by a deep neural network class
G NN .
Functions in G NN can be written in the following compositional form
[Equation]
where the W i ’s
and b i ’s are weight matrices and intercepts/biases of corresponding dimensions, respectively, and σ is ReLU activation applied entry-wise: σ p a q “ max p a, 0 q .
Here θ “ t W i , b i u Li “denotes the set of parameters.
Solving (1) is prohibitive in practice, as we only have access to a finite collection of samples,
X 1 , . . .
, X n i .
i .
d . „ Q .
Replacing Q by its empirical counterpart Q n “ 1 n ř ni “ 1 δ X i , we end up with
[Equation]
Note that (4) is also known as training deep generative models under the Wasserstein loss in existing deep learning literature [Frogner et al., 2015, Genevay et al., 2018].
It has exhibited remarkable ability in learning complex distributions in high dimensions, even though existing theories cannot fully explain such empirical successes.
In literature, statistical theories of deep generative models have been studied in Arora et al.
[2017], Zhang et al. [2017], Jiang et al.
[2018], Bai et al.
Liang [2017, 2018], Uppal et al.
[2019], Chen et al.
Lu and Lu [2020], Block et al.
Luise et al.
[2020], Schreuder et al.
Due to the well-known curse of dimensionality, the sample complexity in Liang [2017], Uppal et al.
[2019], Chen et al.
Lu and Lu [2020] grows exponentially with respect to underlying the data dimension.
For example, the CIFAR-10 dataset consists of 32 ˆ 32 RGB images.
Roughly speaking, to learn this data distribution with accuracy ϵ , the sample size is required to be ϵ ´ D where D “ 32 ˆ 32 ˆ 3 “ 3072 is the data dimension.
Setting ϵ “ 0 . 1 requires 10 3072 samples.
However, GANs have been successful with 60 , 000 training samples [Goodfellow et al., 2014].
A common belief to explain the aforementioned gap between theory and practice is that practical data sets exhibit low-dimensional intrinsic structures.
For example, many image patches are generated from the same pattern by some transformations, such as rotation, translation, and skeleton.
Such a generating mechanism induces a small number of intrinsic parameters.
It is plausible to model these data as samples near a low dimensional manifold [Tenenbaum et al., 2000, Roweis and Saul, 2000,
Peyré, 2009, Coifman et al., 2005].
To justify that deep generative models can adapt to low-dimensional structures in data sets, this paper focuses (from a theoretical perspective) on the following fundamental questions of both distribution approximation and estimation:
Q1: Can deep generative models approximate a distribution on a low-dimensional manifold by representing it as the pushforward measure of a low-dimensional easy-to-sample distribution?
Q2:
If the representation in Q1 can be learned by deep generative models, what is the statistical rate of convergence in terms of the sample size n ?
This paper provides positive answers to these questions.
We consider data distributions supported on a d -dimensional compact Riemannian manifold M isometrically embedded in R D .
The easy-to-sample distribution ρ is uniform on p 0 , 1 q d ` 1 .
To answer Q1 , our Theorem 1 proves that deep generative models are capable of approximating a transportation map which maps the low-dimensional uniform distribution ρ to a large class of data distributions on M .
To answer Q2 , our Theorem 2 shows that the Wasserstein-1 loss in distribution learning converges to zero at a fast rate depending on the intrinsic dimension d instead of the data dimension D .
In particular we prove that
[Equation]
for all δ ą 0 where C is a constant independent of n and D .
Our proof proceeds by constructing an oracle transportation map g ˚ such that g ˚7 ρ “ Q .
This construction crucially relies on a cover of the manifold by geodesic balls, such that the data distribution
Q is decomposed as the sum of local distributions supported on these geodesic balls.
Each local distribution is then transported onto lower dimensional sets in R d from which we can apply optimal transport theory.
We then argue that the oracle g ˚ can be efficiently approximated by deep neural networks.
We make minimal assumptions on the network, only requiring g θ belong to G NN with size depending on some accuracy ϵ .
Further we make minimial assumptions on the data distribution Q , only requiring it admit an upper and lower bounded density.
Standard technical assumptions are made on the manifold M .
Preliminaries
We establish some notation and preliminaries on Riemannian geometry and optimal transport theory before presenting our proof.
Notation.
For x P R d , } x } is the Euclidean norm, unless otherwise specified.
B X p 0 , r q is the open ball of radius r in metric space X .
If unspecified B p 0 , r q “ B R d p 0 , r q .
For a function f : R d Ñ R d and A Ď R d , f ´ 1 r A s denotes the pre-image of A under f .
B denotes the differential operator.
For 0 ă α ď 1 , we denote by C α the class of Hölder continuous functions with Hölder index α . } ¨ }denotes the 8 norm of a function, vector, or matrix (considered as a vector).
For any positive integer N P N , we denote by r N s the set t 1 , 2 , . . .
, N u .
2.1 Riemannian Geometry
Let p M , g q be a d -dimensional compact Riemannian manifold isometrically embedded in R D .
Roughly speaking a manifold is a set which is locally Euclidean i.e. there exists a function ϕ continuously mapping a small patch on M into Euclidean space.
This can be formalized with open sets and charts .
At each point x P M we have a tangent space T x M which, for a manifold embedded in R D , is the d -dimensional plane tangent to the manifold at x .
We say M is Riemannian because it is equipped with a smooth metric g x : T x M ˆ T x M Ñ R which can be thought of as a local inner product.
We can define the Riemannian distance d M : M ˆ M Ñ R on M as d M p x, y q “ inf t L p γ q| γ is a C 1 p M q curve such that γ p 0 q “ x, γ p 1 q “ y u , i.e. the length of the shortest path or geodesic connecting x and y .
An isometric embedding of the d -dimensional M in R D preserves the Riemannian distance between points on M in R D .
For more rigorous statements, see the classic reference Flaherty and do Carmo [2013].
We next define the exponential map at a point x
P M going from the tangent space to the manifold.
Definition 1 (Exponential map) .
Let x P M .
For all tangent vectors v P T x M , there is a unique geodesic γ that starts at x with initial tangent vector v , i.e. γ p 0 q “ x and γ 1 p 0 q “ v .
The exponential map centered at x is given by exp x p v q “ γ p 1 q , for all v P T x M .
The exponential map sends a point on the tangent space to manifold via travelling along a geodesic curve (see Figure 1 for an example).
It is well known that for all x P M , there exists a radius δ such that the exponential map restricted to B T x M p 0 , δ q is a diffeomorphism onto its image, i.e. it is a smooth map with smooth inverse.
As the sufficiently small δ -ball in the tangent space may vary for each x P M , we define the injectivity radius of M as the minimum δ over all x P M .
[Table]
/AMr/BmPVkv1rv1MR9dsbKdI/gD6/MH6M6Vcw==</latexit> v 2 T x M <latexit sha1_base64="bTkSkymD5w3gGuB4q77c6tx8b1c=">AAAB73icbVBNS8NAEJ34WetX1aOXxSJ4KkkR9Vjw4rGC/YA2lM1m0i7dZOPuRiihf8KLB0W8+ne8+W/ctjlo64OBx3szzMwLUsG1cd1vZ219Y3Nru7RT3t3bPzisHB23tcwUwxaTQqpuQDUKnmDLcCOwmyqkcSCwE4xvZ37nCZXmMnkwkxT9mA4THnFGjZW6Q5Qhas4Glapbc+cgq8QrSBUKNAeVr34oWRZjYpigWvc8NzV+TpXhTOC03M80ppSN6RB7liY0Ru3n83un5NwqIYmkspUYMld/T+Q01noSB7Yzpmakl72Z+J/Xy0x04+c8STODCVssijJBjCSz50nIFTIjJpZQpri9lbARVZQZG1HZhuAtv7xK2vWad1W7vK9XG80ijhKcwhlcgAfX0IA7aEILGAh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5AzaRkCI=</latexit> geodesic
Definition 2 (Injectivity radius) .
For x P M we define the injectivity radius at a point inj M p x q “ sup t δ ą 0 | exp x : B T x M p 0 , δ q Ď T x M Ñ M is a diffeomorphism u .
Then the injectivity radius of
M is
[Equation]
The injectivity radius defines the largest balls we can place around the origin in tangent space such that the exponential map is always injective for every base point x .
Within the injectivity radius the exponential map is a diffeomorphism between the tangent space and a patch of M , with exp ´ 1 the inverse.
Controlling a quantity called reach allows us to lower bound the manifold’s injectivity radius.
Definition 3 (Reach[Federer, 1959]) .
The reach τ of a manifold M is defined as the quantity τ “ inf t r ą 0 : D x ‰ y P M , v P R D such that r “ } x ´ v } “ } y ´ v } “ inf z P M } z ´ v }u .
Intuitively, if the distance of a point x to M is smaller than the reach, then there is a unique point in
M that is closest to x .
However, if the distance between x and M is larger than the reach, then there will no longer be a unique closest point to x in M .
For example, the reach of a sphere is its radius.
A manifold with large and small reach is illustrated in Figure 2.
The reach gives us control over the injectivity radius inj p M q ; in particular, we know inj p M q ě πτ (see Aamari and Levrard [2019] for proof).
2.2 Optimal Transport Theory
Let µ, ν be absolutely continuous measures on sets X, Y Ď R d .
We say a function f : X Ñ Y transports µ onto ν if f 7 µ “ ν .
In words, for all measurable sets A we have
[Equation]
where f ´ 1 p
A q is the pre-image of A under f .
Optimal transport studies the maps taking source measures µ on X to target measures ν on Y which also minimize a cost c : X ˆ Y Ñ R ` among all such transport maps.
However the results are largely restricted to transport between measures on the same dimensional Euclidean space.
In this paper, we will make use of the main theorem in Caffarelli [1992], in the form presented in Villani [2008].
Proposition 1.
Let c p x, y q “ } x ´ y } 2 in R d ˆ R d and let Ω 1 , Ω 2 be nonempty, connected, bounded, open subsets of R d .
Let f 1 , f 2 be probability densities on Ω 1 and Ω 2 respectively, with f 1 , f 2 bounded from above and below.
Assume further that Ω 2 is convex.
Then there exists a unique optimal transport map T :
Ω 1 Ñ Ω 2 for the associated probability measures µ p dx q “ f 1 p x q dx and ν p dy q “ f 2 p y q dy , and the cost c .
Furthermore, we have that T P C α p Ω 1 q for some α P p 0 , 1 q .
This proposition allows to produce Hölder transport maps which can be further approximated with neural networks with size depending on a given accuracy.
To connect optimal transport and Riemannian manifolds, we first define the volume measure on a manifold M and establish integration on M .
Definition 4 (Volume measure) .
Let M be a compact d -dimensional Riemannian manifold.
We define the volume measure µ M on M as the restriction of the d -dimensional Hausdorff measure H d .
A definition for the restriction of the Hausdorff measure can be found in Federer [1959].
We say that the distribution Q has density q if the Radon–Nikodym derivative of Q with respect to µ M is q .
According to Evans and Gariepy [1992]), for any continuous function f : M Ñ R supported within the image of the ball B T x M p 0 , ϵ q under the exponential map for ϵ ă inj p M q , we have ż
[Equation]
Here g xij p v q “ xB exp x p v qr e i s , B exp x p v
qr e j sy with p e 1 , ..., e d q an orthonormal basis of T x M .
This is the associated Riemannian metric of M .
Main Results
We will present our main results in this section, including an approximation theory for a large class of distributions on a Riemannian manifold (Theorem 1), and a statistical estimation theory of deep generative networks for distribution learning (Theorem 2).
We make some regularity assumptions on a manifold M and assume the target data distribution Q is supported on M .
The easy-to-sample distribution ρ is taken to be uniform on p 0 , 1 q d ` 1 .
Assumption 1.
M is a d -dimensional compact Riemannian manifold isometrically embedded in ambient dimension R D .
Via compactness, M is bounded: there exists M ą 0 such that } x } 8 ď M , @
x P M .
Further suppose M has a positive reach τ ą 0 .
Assumption 2.
Q is supported on M and has a density q with respect to the volume measure on M .
Further we assume boundedness of q i.e. there exists some constants c, C ą 0 such that c ď q ď C .
To justify the representation power of feedforward ReLU networks for learning the target distribution
Q , we explicitly construct a neural network generator class, such that a neural network function in this generator class can pushfoward ρ to a good approximation of Q .
Consider the following generator class G NN
[Equation]
at most L layers and max width p, while || W i || 8 ď κ, ||
b
i || 8 ď κ for all i P r L s , j P r D su , where } ¨ } 8 is the maximum magnitude in a matrix or vector.
Theorem 1 (Approximation Power of Deep Generative Models) .
Suppose M and Q satisfy Assumptions 1 and 2 respectively.
The easy-to-sample distribution ρ is taken to be uniform on p 0 , 1 q d ` 1 .
Then there exists a constant
ă α ă 1 (independent of D ) such that for any
ă ϵ ă 1 , there exists a g θ P G NN p L, p, κ q with parameters
[Equation]
that satisfies
[Equation]
Theorem 1 demonstrates the representation power of deep neural networks for distributions Q on M , which answers Question Q1 .
For a given accuracy ϵ , there exists a neural network g θ which pushes the uniform distribution on p 0 , 1 q d ` 1 forward to a good approximation of Q with accuracy ϵ .
The network size is exponential in the intrinsic dimension d .
We next present a statistical estimation theory to answer Question Q2 .
Theorem 2 (Statistical Guarantees of Deep Wasserstein Learning) .
Suppose M and Q satisfy
Assumption 1 and 2 respectively.
The easy-to-sample distribution ρ is taken to be uniform on p 0 , 1 q d ` 1 .
Let n be the number of samples of X i „ Q .
Choose any δ ą 0 .
Set ϵ
“ n ´ 1 d ` δ in Theorem 1 so that the network class G NN p L, p, κ q has parameters
[Equation]
Then the empirical risk minimizer ˆ g n given by (4) has rate
[Equation]
where C is a constant independent of n and D .
Additionally this result can be easily extended to the noisy case.
Suppose we are given n noisy i.i.d.
samples ˆ X 1 , ..., ˆ X n of the form ˆ
X
i “ X i ` ξ i , for X i „ Q and ξ i distributed according to some noise distribution.
The optimization in (4) is performed with the noisy empirical distribution ˆ Q n “ 1 n
[Equation]
ni
[Equation]
1 δ ˆ X i .
Then the minimizer ˆ g n satisfies
[Equation]
where V ξ “ E } ξ } 22 is the variance of the noise distribution.
Comparison to Related Works .
To justify the practical power of generative networks, lowdimensional data structures are considered in Luise et al.
[2020], Schreuder et al.
[2021], Block et al.
[2021], Chae et al.
These works consider the generative models in (1).
They assume that the high-dimensional data are parametrized by low-dimensional latent parameters.
Such assumptions correspond to the manifold model where the manifold is globally homeomorphic to Euclidean space, i.e. the manifold has a single chart.
In Luise et al.
[2020], the generative models are assumed to be continuously differentiable up to order s .
By jointly training of the generator and the latent distributions, they proved that the
Sinkhorn divergence between the generated distribution and data distribution converges, depending on data intrinsic dimension.
Chae et al.
[2021] and Schreuder et al.
[2021] assume the special case where the manifold has a single chart.
More recently, Block et al.
[2021] proposed to estimate the intrinsic dimension of data using the Hölder IPM between some empirical distributions of data.
This theory is based on the statistical convergence of the empirical distribution to the data distribution.
As an application to GANs, [Block et al., 2021, Theorem 23] gives the statistical error while the approximation error is not studied.
In these works, the single chart assumption is very strong while a general manifold can have multiple charts.
Recently, Yang et al.
[2022], Huang et al.
[2022] showed that GANs can approximate any data distribution (in any dimension) by transforming an absolutely continuous 1D distribution.
The analysis in Yang et al.
[2022], Huang et al.
[2022] can be applied to the general manifold model.
Their approach requires the GAN to memorize the empirical data distribution using ReLU networks.
Thus it is not clear how the designed generator is capable of generating new samples that are not a linear interpolation of the training data.
In contrast we explicitly construct an oracle transport map which transforms the low-dimensional easy-to-sample distribution to the data distribution.
Our work provides insights about how distributions on a manifold can be approximated by a neural network pushforward of a low-dimensional easy-to-sample distribution without exactly memorizing the data.
In comparison, the single-chart assumption in earlier works assumes that an oracle transport naturally exists.
Our work is novel in the construction of the oracle transport for a general manifold with multiple charts, and the approximation theory by deep neural networks.
Proof of Main Results 4.1 Proof of Approximation Theory in Theorem
To prove Theorem 1, we explicitly construct an oracle transport g ˚ pushing ρ onto Q , i.e. g ˚7 ρ “ Q .
Further this oracle will be piecewise α -Hölder continuous for some α P p 0 , 1 q .
Lemma 1.
Suppose M and Q satisfy Assumption 1 and 2 respectively.
The easy-to-sample distribution ρ is taken to be uniform on p 0 , 1 q d ` 1 .
Then there exists a function g ˚ : p 0 , 1 q d ` 1 Ñ M such that Q “ g ˚7 ρ where
[Equation]
(6) for some α -Hölder ( 0 ă α ă 1 ) continuous functions g ˚ 1 , . .
, g ˚ J and some constants
“ π 0 ă π 1 ă ¨ ¨ ¨ ă π J “ 1 .
Proof.
We construct a transport map g ˚ : p 0 , 1 q d ` 1 Ñ M that can be approximated by neural networks: First, we decompose the manifold into overlapping geodesic balls and map local distributions over the balls onto tangent planes.
Then once we have a distribution over a ball in the tangent space we can apply optimal transport theory to produce a map between the source distribution on p 0 , 1 q d to a ball on the manifold.
Finally, we glue together these local maps with indicators and a uniform random sample from p 0 , 1 q .
We proceed with the first step of decomposing the manifold.
Step 1: Overlapping ball decomposition.
Recall that M is a compact manifold with reach τ ą 0 .
Then the injectivity radius of M is greater than πτ (Aamari et al.
Set r “ πτ 2 .
For each c P M , define an open set
U c “ exp c p B T c M p 0 , r qq Ď M .
Since the collection t U c : x P M u forms an open cover of
M (in R D ), by the compactness of M we can extract a finite subcover which we denote as t
U c j u Jj “ 1 .
For convenience, we will write U j “ U c j .
Step 2: Defining local lower-dimensional distributions.
On each U j define a local distribution Q j with density q j via
[Equation]
Set K p x q “ ř Jj “ 1 1 x P U j as the number of balls U j containing x .
Note 1 ď K p x q ď J for all x P M .
Now define the distribution Q j with density q j given by
[Equation]
Write K j “ ş
U j 1 K p x q q j p x q d H as the normalizing constant.
Define ˜ q j p v q “ p q j ˝ exp c
j qp v q b det g c j kl p v q where g c j kl is the Riemannian at c j .
This quantity can be thought of as the Jacobian of the exponential map, denoted by | J exp cj p v q| in the following step.
Then ˜ q j is a density on ˜ U
j “ exp ´ 1 c
j p U j q , which is a ball of radius πτ 2 since
[Equation]
Let ˜ Q
j be the distribution in R d with density ˜ q j .
By construction, we can write
[Equation]
Step 3:
Constructing the local transport.
We have that exp ´ 1 c j is bi-lipschitz on U j and hence its
Jacobian is upper bounded.
Since | J exp cj p v q| “ 1 | J exp [Equation] 1 cj p x q| , we know that | J exp cj | lower bounded.
Since q j is lower bounded (away from 0 ), this means ˜ q j is also lower bounded.
Now the distribution ˜ ρ j supported on ˜ U
j “ B p 0 , τπ 2 q fulfills the requirements for our optimal transport result: (1) Its density ˜ p
j is lower and upper bounded; (2) The support B p 0 , τπ 2 q is convex.
Taking our cost to be c p x, y q “ 12 } x ´ y } 2 (i.e. squared Euclidean distance) then via Proposition 1 we can find an optimal transport map T j such that
[Equation]
where ρ d is uniformly distributed on p 0 , 1 q d .
Further T j P C α j for some α j P p 0 , 1 q .
Then we can construct a local transport onto U j via
[Equation]
which pushes ρ d forward to Q j .
Since g ˚ j is a composition of a Lipschitz map with an α j Hölder continuous maps, it is hence α j Hölder continuous.
Step 4:
Assembling the global transport.
It remains to patch together the local distributions Q j to form Q .
Define η j “ K j Q p U j q .
Notice
[Equation] [Equation] U
[Equation]
p
[Equation]
[Equation]
[Equation]
Hence it must be that ř Jj “ 1 η j “ 1 .
Set α “ min j Pr J s α j .
We can now define the oracle g ˚ .
Let x
P p 0 , 1 q d ` 1 .
Write
[Equation]
where x 1 is the first coordinate and x 2: d ` 1 are the remaining coordinates with π j “ ř j ´ 1 i “ 1 η i .
Let
Z „ ρ .
Then g p Z q „ Q .
We see this as follows.
For A Ď M we can compute
[Equation]
which completes the proof.
We have found an oracle g ˚ which is piecewise Hölder continuous such that g ˚7 ρ “ Q .
We can design a neural network g θ to approximate this oracle g ˚ .
Now in order to minimize W 1 pp g θ q 7 ρ, Q q “
W 1 pp g θ q 7 ρ, g ˚7 ρ q , we show it suffices to have g θ approximate g ˚ in L 1 p ρ q .
Lemma 2.
Let µ be an absolutely continuous probability distribution on a set Z Ď R d , and let f, g : Z Ñ R m be transport maps.
Then
[Equation]
for some C ą 0 .
The proof can be found in section B.1 in the appendix.
We now prove Theorem 1.
Proof of Theorem 1.
By
Lemma 1, there exists a transformation g ˚ p x q “ ř
Jj
[Equation]
p x
[Equation]
q g ˚ j p x 2: d ` 1 q such that g ˚7 ρ “ Q .
By Lemma 2, it suffices to approximate g ˚ with a neural network g θ P G NN p L, p, κ q in L 1 norm, with a given accuracy ϵ ą 0 .
Let p g ˚
q
p
i q denote the i th component of the vector valued function g ˚ .
Then it suffices to approximate
[Equation]
for each 1 ď i ď D , where p g ˚ j q p
i q denotes the i th component of the function g ˚ j .
We construct the approximation of p g ˚ q p i q by the function
[Equation]
where ˜ ˆ δ 2 is a ReLU network approximation to the multiplication operation with δ 2 accuracy, ˜ 1 δ 1 p π j ´ 1 ,π j q is a ReLU network approximation to the indicator function with δ 1 accuracy, and p g δ 3 j,θ q p i q is a ReLU network approximation to p g
˚ j q p i q with δ 3 accuracy.
We construct these using the approximation theory outlined in Appendix A.
First, we obtain ˜ 1 δ 1 p π j ´ 1 ,π j q via an application of Lemma 9.
Next, we obtain ˜ ˆ
δ 2 from an application of Lemma 7.
Finally, we discuss g δ 3 j,θ .
Let j P r J s .
To approximate the Hölder function g ˚ j , we use the following Lemma 3 that is proved in Appendix A. Similar approximation results can be found in
Shen et al.
[2022] and Ohn and Kim [2019]
as well.
In Lemma 3, our approximation error is in Lnorm and all weight parameters are upper bounded by a constant.
In comparison, the error in Ohn and Kim [2019] is in L 8 norm and the weight parameter increases as ϵ decreases.
Lemma 3.
Fix M ě 2 .
Suppose f P C α pr 0 , 1 s d q , α P p 0 , 1 s , with } f } L 8 ă M .
Let
ă ϵ ă 1 .
Then there exists a function Φ implementable by a ReLU network such that
[Equation]
The ReLU network has depth at most c 1 log `ϵ ˘ , width at most c 2 ϵ ´ dα , and weights bounded by M (where c 1 and c 2 are constants independent of ϵ ).
We can apply Lemma 3 to p g ˚ j q p i q for all 1 ď j ď J and 1 ď i ď D , since they are all elements of C α p 0 , 1 q d and elements of C α p 0 , 1 q d can be extended to C α r 0 , 1 s d .
Thus there exists a neural network p g δ 3 j,θ q p
i q P G NN p L, p, κ q with parameters given as above such that
[Equation]
The goal is now to show the L 1 distance between g θ (as defined in 11) and g ˚ is small.
We have
[Equation]
where δ 1 is the neural network approximation error of indicator functions in L 1 , δ 2 is the approximation error of multiplication, and δ 3 is the approximation error of our α -Hölder local transport maps.
We carefully argue in Appendix A.4 that each of these components can be approximated with the appropriately sized network.
To complete the proof we conclude g θ can be exactly represented by a neural network in G NN p L, p, κ q with parameters
[Equation]
4.2 Proof of Statistical Estimation Theory in Theorem
The proof of Theorem 2 is facilitated by the common bias-variance inequality, presented here as a lemma.
Lemma 4.
Under the same assumptions of Theorem 2, we have
[Equation]
where Q n is the clean empirical distribution.
Proof.
We compute recalling the definition of ˆ g n as the empirical risk minimizer.
[Equation]
where where we recall W 1 pp ˆ g n q 7 ρ, Q n q “ inf g θ P G NN W
pp g θ q 7 ρ, Q n q from (4).
The bias term can be controlled via Theorem 1.
To control convergence of the empirical distribution
Q n to Q we leverage the existing theory [Weed and Bach, 2019] to obtain the following lemma.
Lemma 5.
Under the same assumptions of Theorem 2, for all δ ą 0 , D C δ ą 0 such that
[Equation]
This follows directly from Theorem 1 from [Weed and Bach, 2019].
We attach a full proof in Section
B.2 of the appendix.
Finally we prove our statistical estimation result in Theorem 2.
Proof of Theorem 2.
Choose δ ą 0 .
Recall from Lemma 4 we have
[Equation]
The first term is the approximation error which can be controlled within an arbitrarily small accuracy ϵ .
Theorem 1 shows the existence of a neural network function g θ with O ` log `ϵ ˘˘ layers and
O p
Dϵ ´ d { α log p 1 ϵ qq neurons such that W 1 pp g θ q 7 ρ, Q q ď ϵ for any ϵ ą 0 .
We choose ϵ “ n ´ 1 d ` δ to optimally balance the approximation error and the statistical error.
The second term is the statistical error for which we recall from Lemma 5 that E
[Equation]
some constant C δ .
Thus we have
[Equation]
by setting C “ 1 ` 2 C δ .
This concludes the proof.
We remark the above proof proceeds similarly in the noisy case, which is presented in Section B.3 of the appendix.
Conclusion
We have established approximation and statistical estimation theories of deep generative models for estimating distributions on a low-dimensional manifold.
The statistical convergence rate in this paper depends on the intrinsic dimension of data.
In light of the manifold hypothesis, which suggests many natural datasets lie on low dimensional manifolds, our theory rigorously explains why deep generative models defy existing theoretical sample complexity estimates and the curse of dimensionality.
In fact, deep generative models are able to learn low-dimensional geometric structures of data, and allow for highly efficient sample complexity independent of the ambient dimension.
Meanwhile the size of the required network scales exponentially with the intrinsic dimension.
Our theory imposes very little assumption on the target density Q , requiring only that it admit a density q with respect to the volume measure and that q is upper and lower bounded.
In particular we make no smoothness assumptions on q .
This is practical, as we do not expect existing natural datasets to exhibit high degrees of smoothness.
In this work we assume access to computation of the W 1 distance.
However during GAN training a discriminator is trained for this purpose.
It would be of interest for future work to investigate the low-dimensional role of such discriminator networks which approximate the W 1 distance in practice
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See our conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We don’t foresee any negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] They are attached in the appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]