U NDERSTANDING GAN S : THE LQG S ETTING
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Adversarial Networks (GANs) have become a popular method to learn a probability model from data.
Many GAN architectures with different optimization metrics have been introduced recently.
Instead of proposing yet another architecture, this paper aims to provide an understanding of some of the basic issues surrounding GANs.
First, we propose a natural way of specifying the loss function for GANs by drawing a connection with supervised learning.
Second, we shed light on the generalization peformance of GANs through the analysis of a simple LQG setting: the generator is linear , the loss function is quadratic and the data is drawn from a Gaussian distribution.
We show that in this setting: 1) the optimal GAN solution converges to population Principal Component Analysis (PCA) as the number of training samples increases; 2) the number of samples required scales exponentially with the dimension of the data; 3) the number of samples scales almost linearly if the discriminator is constrained to be quadratic.
Thus, linear generators and quadratic discriminators provide a good balance for fast learning.
I NTRODUCTION
Learning a probability model from data is a fundamental problem in statistics and machine learning.
Building off the success of deep learning methods, Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have given this age-old problem a face-lift.
In contrast to traditional methods of parameter ﬁtting like maximum likelihood estimation, the GAN approach views the problem as a game between a generator whose goal is to generate fake samples that are close to the real data training samples and a discriminator whose goal is to distinguish between the real and fake samples.
The generator and the discriminators are typically implemented by deep neural networks.
GANs have achieved impressive performance in several domains (e.g., (Ledig et al., 2016;
Reed et al., 2016)).
Since (Goodfellow et al., 2014), many variations of GANs have been developed, including f -GAN (Nowozin et al., 2016),
MMD-GAN (Dziugaite et al., 2015; Li et al., 2015), WGAN (Arjovsky et al., 2017), improved WGAN (Gulrajani et al., 2017), relaxed WGAN (Guo et al., 2017), Least-Squares GAN (Mao et al., 2016), Boundary equilibrium GAN (Berthelot et al., 2017), etc.
These GANs use different metrics in the optimization for training the generator and discriminator networks (Liu et al., 2017).
The game theoretic formulation in GANs can be viewed as the dual of an optimization that minimizes a distance measure between the empirical distributions of the fake and real samples.
In the population limit where there are inﬁnite number of samples, this optimization minimizes the distance between the generated distribution and the true distribution from which the data is drawn.
In the original GAN framework, this distance measure is the Jenson Shannon divergence.
However,
Arjovsky et al (Arjovsky et al., 2017) noted that this distance does not depend on the generated distribution whenever its dimension is smaller than that of the true distribution.
In this typical case, the
Jenson Shannon divergence does not serve as a useful criterion in choosing the appropriate generated distribution.
To resolve this issue, (Arjovsky et al., 2017) proposed the Wasserstein GAN (WGAN) which uses the ﬁrst-order Wasserstein distance instead of Jensen-Shannon divergence.
This distance is meaningful even when the dimension of the generated distribution is less than the true distribution.
Nevertheless there are many other distance measures that satisfy this criterion and it is not clear how to choose among them.
This is responsible in part for the fact that there are so many different GAN architectures.
In fact, there is currently some confusion in the literature even on the basic issue of how to specify the loss function for GANs.
For example, while the ”Wasserstein” in Wasserstein
[Figure]
p o p u l a t i o n e m p i r i c a l
[Figure]
GAN refers to the use of Wasserstein distance in the distance measure between the generated and true distributions, the ”Least Squares” in Least-Squares GAN (Mao et al., 2016) refers to the use of squared error in the discriminator optimization objective .
These are two totally different types of objects.
The situation with GANs is in contrast to that in supervised learning, where how the loss function is speciﬁed in the formulation is clear and quite universally accepted.
A central issue in any learning problem is generalization : how close a model learnt from a ﬁnite amount of data is to the true distribution.
Or, in statistical terms, how fast is the rate of convergence of the learnt model to the true model as a function of number of samples?
Arora et al
(Arora et al., 2017) have recently studied this problem for GANs.
They observed that for Wasserstein GAN, if there are no constraints on the generator or the discriminator, the number of samples required to converge scales exponentially in the data dimension.
They then showed that if the discriminator is constrained to be in a parametric family, then one can achieve convergence almost linearly in the number of parameters in that family (Theorem 3.1 in (Arora et al., 2017)).
However, the convergence is no longer measured in the Wasserstein distance but in a new distance measure they deﬁned (the neural network distance).
The result is interesting as it highlights the role of the discriminator in generalization, but it is somewhat unsatisfactory in that the distance measure needs to be modiﬁed to tailor to the speciﬁc constraints on the discriminator.
Moreover, the result requires the invention of (yet) another family of distance measures for GANs.
In this paper, we ﬁrst argue that there is a natural way to specify the loss function (cid:96) for GANs, in an analogous way as in the supervised learning setup.
The resulting optimal GAN minimizes a generalized loss-function dependent Wasserstein distance between the generated distribution and the true distribution, and the dual formulation of this generalized Wasserstein distance leads to a loss-function dependent discriminator architecture.
To study the impact of the constraints on the generator and the discriminator on the generalization performance in this distance measure, we focus on the case when the true data distribution is Gaussian. In this case, a natural loss function to consider is quadratic, and a natural class of generators to consider is linear with a given rank k .
In this setting, the optimal GAN minimizes the second-order Wasserstein distance between the generated distribution and the empirical data distribution among all linear generators of a given rank.
We show the following results:
1 In the population limit as the number of data samples grow, the optimal generated distribution is the rank k Gaussian distribution retaining exactly the top k principal components of the true distribution,
i
e. GAN performs PCA in the population limit. The number of samples required for convergence in (second-order) Wasserstein distance however scales exponentially with the dimension of the data distribution.
3 Under a further constraint that the discriminator is quadratic , GAN converges to the same population-optimal PCA limit, but with the number of samples scaling almost linearly with the dimension.
The constrained GAN simply performs empirical PCA, and in the case when the rank k of the generator is the same as the dimension of the data distribution,
GAN is equivalent to maximum likelihood estimation of the underlying Gaussian model.
These results are summarized in Figure 1.
The GAN architecture with a linear generator and a quadratic discriminator is shown in Figure 4.
(Arora et al., 2017) observed that the number of samples required to generalize for GAN is exponential in the dimension of the data when there are no constraints on either the generator or the discriminator.
(They proved the result for ﬁrst-order Wasserstein distance but a similar result holds for second-order Wasserstein distance as well, see Lemma 2 in Section 3.)
Result 2 above says that even constraining the generator drastically to be linear cannot escape this exponential scaling.
Result 3 says that this exponential scaling is not due to statistical limitation, but much better inference can be obtained by constraining the discriminator appropriately.
Similar to Theorem 3.1 in (Arora et al., 2017), Result 3 points to the importance of constraining the discriminator.
But there are two key differences.
First, the convergence in Result 3 is with respect to the original (second-order)
Wasserstein distance, not another distance measure tailored to the constraint on the discriminator.
Thus, the original quadratic loss function is respected.
Second, the population limit is the same as the PCA limit achieved without constraints on the discriminator.
Thus, by imposing a discriminator constraint, the rate of convergence is drastically improved without sacriﬁcing the limiting performance.
There is no such guarantee in (Arora et al., 2017).
Our results also provide concrete evidence that an appropriate balance between the classes of generators and discriminators, i.e. linear generators and quadratic discriminators, can provide fast training.
The Linear-Quadratic-Gaussian (LQG) setting, dating back to at least Gauss, has been widely used across many ﬁelds, including statistics, machine learning, control, signal processing and communication.
It has resulted in celebrated successes such as linear regression, the Wiener ﬁlter, the Kalman ﬁlter, PCA, etc., and is often used to establish a baseline to understand more complex models.
We believe it serves a similar role here for GANs 1 .
Indeed it allows us to make a clear connection between GAN and PCA, perhaps the most basic of unsupervised learning methods.
Moreover, even in this simple setting, the generalization issues in GAN are non-trivial, and understanding them in this setting provides the foundation to tackle more complex data distributions and more complex generators and discriminators such as deep nets.
When the discriminator is not constrained, the number of samples scale exponentially with the dimension.
It turns out that a simple characterization of the performance of GAN in that regime can be obtained using rate-distortion theory (Cover & Thomas, 2012).
In the population limit, quadratic GAN minimizes the Wasserstein distance between the generated distribution and the true data distribution.
This distance is the solution of an unconstrained optimal transport problem.
It turns out that when there are a ﬁnite number of samples, the performance is again characterized by an optimal transport between the generated and true distributions, but with a constraint on the mutual information of the transport to be less than the logarithm of the number of samples.
This constraint reﬂects the fact that only empirical samples from the true distribution are seen and therefore the generated distribution cannot be too strongly coupled to the true distribution.
We discuss this story in the appendix.
The rest of the paper is organized as follows.
In Section 2, we discuss a formulation of the GAN problem for general loss functions.
In Section 3, we specialize to the LQG setting and analyze the generalization performance of GAN.
In Section 4, we analyze the performance of GAN under a quadratic constraint on the discriminator.
In Section 5, we present some experimental results.
A G ENERAL F ORMULATION FOR GAN S
Let { y i } n i =1 be n observed data points in R d drawn i.i.d.
from the distribution P Y .
Let Q n Y be the empirical distribution of these observed samples.
Moreover, let P X be a normal distribution
N ( 0 , I k ) .
GANs can be viewed as an optimization that minimizes a distance between the observed empirical distribution Q nY and the generated distribution P g ( X ) .
The population GAN optimization replaces Q nY with P Y .
The question we ask in this section is: what is a natural way of specifying a loss function (cid:96) for GANs and how it determines the distance?
2. WGAN R EVISITED
Let us start with the WGAN optimization (Arjovsky et al., 2017):
[Equation]
where G is the set of generator functions, and the p -th order Wasserstein distance between distributions P Z 1 and P Z 2 is deﬁned as (Villani, 2008)
[Equation]
where the minimization is over all joint distributions with marginals ﬁxed.
Replacing (2) in (1), the
WGAN optimization can be re-written as
[Equation]
or equivalently:
[Equation]
where the minimization is over all joint distributions P X,Y with ﬁxed marginals P X and P Y .
We now connect (4) to the supervised learning setup.
In supervised learning, the joint distribution P
X,Y is ﬁxed and the goal is to learn a relationship between the feature variable represented by
X ∈ R k , and the target variable represented by Y ∈ R d , according to the following optimization:
[Equation]
where (cid:96) is the loss function.
Assuming the marginal distribution of X is the same in both optimizations (4) and (5), we can connect the two optimization problems by setting (cid:96) ( y, y (cid:48) )
= (cid:107) y − y (cid:48)
(cid:107) in optimization (5).
Note that for every ﬁxed P X,Y , the solution of the supervised learning problem (5) yields a predictor g which is a feasible solution to the WGAN optimization problem (4).
Therefore, the WGAN optimization (3) can be re-interpreted as solving the easiest such supervised learning problem, over all possible joint distributions P X,Y with ﬁxed P X and P Y .
2. F ROM S UPERVISED TO U NSUPERVISED L EARNING
GAN is a solution to an unsupervised learning problem.
What we are establishing above is a general connection between supervised and unsupervised learning problems: a good predictor g learnt in a supervised learning problem can be used to generate samples of the target variable Y. Hence, to solve an unsupervised learning problem for Y with distribution P Y , one should solve the easiest supervised learning problem P X,Y with given marginal P Y (and P X , the randomness generating distribution).
This is in contrast to the traditional view of the unsupervised learning problem as observing the feature variable X without the label Y .
(Thus in this paper we break with tradition and use Y to denote data and X as randomness for the generator in stating the GAN problem.)
This connection between supervised and unsupervised learning leads to a natural way of specifying the loss function in GANs: we simply replace the (cid:96) 2 Euclidean norm in (3) with a general loss function (cid:96) :
[Equation]
The inner optimization is the optimal transport problem between distributions of g ( X ) and Y (Villani, 2008) with general cost (cid:96) .
This is a linear programming problem for general cost, so there is always a dual formulation (the Kantorovich dual (Villani, 2008)).
The dual formulation can be interpreted as a generalized discriminator optimization problem for the cost (cid:96) .
(For example, in the case of (cid:96) being the Euclidean norm, we get the WGAN architecture; see Figure 2(a).)
Hence, we propose (6) as a formulation of GANs for general loss functions.
g(.)
-Ψ(Y) Ψ(Y)
Yє R d Xє R k
Yє R d ^ ^ +
WGAN Ψ:1-Lip (b) (a) g(.)
-Ψ * (Y) -Ψ(Y)
Yє R d Xє R k
Yє R d ^ ^ +
Quadratic GAN Ψ:convex 2. Q UADRATIC L OSS The most widely used loss function in supervised learning is the quadratic loss: (cid:96) ( y, y (cid:48) ) = (cid:107) y − y (cid:48)
(cid:107)( squared Euclidean norm). Across many ﬁelds its use had led to many important discoveries. With the connection between supervised and unsupervised learning in mind, this loss function should be a prime choice to consider in GANs as well. This choice of the loss function leads to the quadratic
GAN optimization:
[Equation]
Since Wasserstein distances are weakly continuous measures in the probability space (Villani, 2008), similar to WGAN, the optimization of the quadratic GAN is well-deﬁned even if k < d .
The dual formulation (discriminator) for W 2 is shown in Figure 2(b).
Note that in this dual, the discriminator applies ψ to the real data and the convex conjugate ψ ∗ to the generated (fake) data.
The empirical quadratic GAN optimization can be formulated by replacing P Y with the empirical distribution Q nY of the data as follows:
[Equation]
Note that while in practice one generates fake samples from X , we will keep the notations simpler in this paper by assuming we can generate the exact distribution g ( X ) , i.e. we can generate as many fake samples as we wish.
Almost all our results can be extended to the case when we have ﬁnite number of samples from X comparable to the number of samples from Y .
For the rest of the paper, we will focus on the problem (8) for the particular case of Y Gaussian of dimension d , and g linear of rank k ≤ d .
This is the LQG setting for GANs.
GAN S UNDER THE LQG SETUP 3. T HE P OPULATION GAN O PTIMIZATION
First, we analyze the population GAN optimization under the LQG setup.
We have the following lemma:
Lemma 1 Let S be a k dimensional subspace in R d .
Let ˆ Y be a random variable whose support lies in S .
Then, ˆ Y ∗ , the optimal solution of the optimization
[Equation]
is the projection of Y to S .
Proof 1 See Appendix C.2.
This Lemma holds even if P Y is a non-Gaussian distribution.
However, P ˆ Y ∗ cannot be generated as g ( X ) when P X ∼ N ( 0 , I k ) and g ( . ) is restricted to be linear.
Using Lemma 1 and under the LQG setup, we show that the optimal solution for the population GAN optimization is the same as the PCA solution.
PCA is the most standard unsupervised learning approach (Jolliffe, 2002).
PCA computes an optimal linear mapping from Y to ˆ Y under the rank constraint on the covariance matrix of ˆ Y ( K ˆ Y ).
We say ˆ Y is the k -PCA solution of Y if K ˆ Y is a rank k matrix whose top k eigenvalues and eigenvectors are the same as top k eigenvalues and eigenvectors of the covariance matrix of Y ( K Y ).
Theorem 1 Let Y ∼ N ( 0 , K Y ) where K Y is full-rank.
Let X ∼ N ( 0 , I k ) where k ≤ d .
The optimal population GAN solution of optimization (7) under linear G is the k -PCA solution of Y .
Proof 2 See Appendix C.3.
Lemma 1 holds if we replace W 2 with W 1 .
However, the conclusion of Theorem 1 is tied to the Wdistance because the PCA optimization also considers the quadratic projection loss.
3. T HE E MPIRICAL GAN O PTIMIZATION
In reality, one solves the GAN optimization over the empirical distribution of the data Q nY , not the population distribution P Y .
Thus, it is important to analyze how close optimal empirical and population GAN solutions are in a given sample size n .
This notion is captured in the generalization error of the GAN optimization, deﬁned as follows:
Deﬁnition 1 (Generalization of GANs) Let n be the number of observed samples from Y .
Let ˆ g ( . ) and g ∗ ( . ) be the optimal generators for empirical and population GANs respectively.
Then,
[Equation]
is a random variable representing the excess error of ˆ g over g ∗ , evaluated on the true distribution.
d G ( P Y , Q nY ) can be viewed as a distance between P Y and Q nY which depends on G .
To have a proper generalization property, one needs to have d G ( P Y , Q nY ) → 0 quickly as n → ∞ .
Before analyzing the convergence rate of d G ( P Y , Q nY ) for linear G , we characterize this rate for an unconstrained
G .
For an unconstrained G , the second term of (10) is zero (this can be seen using a space ﬁlling generator function (Cannon & Thurston, 1987)).
Moreover, P ˆ g ( X ) can be arbitrarily close to Q nY .
Thus, we have
Lemma
If G is unconstrained, we have
[Equation]
) which goes to zero with high probability with the rate of O ( n − 2 /d ) .
The approach described for the unconstrained G corresponds to the memorization of the empirical distribution Q nY using the trained model.
Note that one can write
[Equation]
Thus, to have a small W 22 ( P Y , Q nY ) , the number of samples n should be exponentially large in d (Canas & Rosasco, 2012).
It is possible that for some distributions P Y , the convergence rate of
W 22 ( P Y , Q nY ) is much faster than O ( n − 2 /d ) .
For example, (Weed & Bach, 2017) shows that if P Y is clusterable (i.e., Y lies in a ﬁxed number of separate balls with ﬁxed radii), then the convergence of W 22 ( P Y , Q nY ) is fast.
However, even in that case, one optimal strategy would be to memorize observed samples, which does not capture what GANs do.
In supervised learning, constraining the predictor to be from a small family improves generalization.
A natural question is whether constraining the family of generator functions G can improve the generalization of GANs.
In the LQG setting, we are constraining the generators to be linear.
To simplify calculations, we assume that Y ∼ N ( 0 , I d ) and d = k .
Under these assumptions, the GAN optimization (8) can be re-written as
[Equation]
(12) where K is the covariance matrix with the eigen decomposition K = UΣU t .
The optimal population solution of this optimization is µ ∗ pop = 0 and K ∗ pop
= I , which provides a zero Wasserstein loss.
Theorem 2 Let µ ∗ n and K ∗ n be optimal solutions for optimization (12) .
Then, µ ∗ n → 0 with the rate of O ( d/n ) and Tr ( Σ ∗ n ) → d with the rate of O ( n − 2 /d ) .
Proof 3 See Appendix C.4.
It turns out that Tr ( Σ ∗ n ) , which is a random variable, is strongly concentrated around its expectation.
Thus, Theorem 2 indicates that there is a signiﬁcant bias in GAN’s estimation of the true distribution which translates to the slow convergence of the generalization error.
Note that in the Wasserstein space, the empirical distribution Q nY and the population distribution P Y are far from each other (the distance between them concentrates around n − 2 /d
(Canas & Rosasco, 2012)).
Thus, if there exists another Gaussian distribution within the sphere around Q nY with the radius of n − 2 /d , the Wasserstein-based learning method will converge to the wrong Gaussian distribution.
This phenomenon causes a bias in estimating the true distribution.
In Appendix A, we ﬁnd that in the regime of large d and n , the estimated covariance matrix is well approximated by: [Equation] i.e. a constant shrinkage in all dimensions. This result is obtained by drawing a connection between optimal transport and rate-distortion theories.
Theorem 2 considers the regime where k = d .
In practice, the dimension of the generated distribution is often much smaller than that of the true one (i.e., k (cid:28) d ).
In this case, GAN’s convergence rate can be increased from O ( n − 2 /d ) to O ( n − 2 /k ) .
However, this faster convergence comes at the expense of the increased bias term of the excess error (the second term of (10)).
The trade-off is favorable if Y is near low rank.
Nevertheless even the convergence rate of O ( n − 2 /k ) is still slow.
In practice, however, GANs have demonstrated impressive performance.
In the next section, we show that by suitably constraining the GAN optimization, the convergence rate can be improved exponentially.
GAN S WITH C ONSTRAINED D ISCRIMINATORS
In this section, ﬁrst we review the min-max (dual) formulation of WGAN (Arjovsky et al., 2017).
Then, we characterize the min-max formulation of the quadratic GAN.
Finally, we show that a properly constrained quadratic GAN achieves the empirical PCA solution, which converges to the population optimal with an exponentially faster rate of convergence compared to the case when the discriminator is unconstrained.
Using the Kantorovich duality (Villani, 2008), the ﬁrst-order Wasserstein distance W 1 ( P Y , P g ( X ) ) can be written as the following optimization (Arjovsky et al., 2017):
[Equation]
where the function ψ ( . ) is restricted to be 1-Lipschitz.
This dual formulation of W 1 is then used in optimization (1) to implement WGAN in a min-max architecture similar to the one of the original
GAN (Figure 2).
In this architecture, ψ ( . ) is implemented by deep neural networks.
In a similar way, one can write the second-order Wasserstein distance W 22 ( P Y , P g ( X ) )
as the following optimization (Villani, 2008):
[Equation]
[Equation]
where ψ ∗ (ˆ y ) :
= sup v ( v t ˆ y − ψ ( v )) is the convex-conjugate of the function ψ ( . ) .
Similarly, this dual formulation of W 22 can be used to implement the quadratic GAN optimization (7) in a min-max architecture which can be interpreted as a game between optimizing two functions g ( . ) and ψ ( . )
(Figure 2).
The following lemma characterizes the optimal solution of optimization (14) (Chernozhukov et al., 2017):
Lemma 3 Let P Y be absolutely continuous whose support contained in a convex set in R d .
For a ﬁxed g ( . ) ∈ G , let ψ opt be the optimal solution of optimization (14) .
This solution is unique.
Moreover, we have
[Equation]
where dist = means matching distributions.
In the LQG setup, since g ( X ) is Gaussian, (cid:53) ψ opt is a linear function.
Thus, without loss of generality, ψ ( . ) in the discriminator optimization can be constrained to ψ ( y ) =
y t
Ay / 2 where A is positive semideﬁnite.
Therefore, we have
[Equation]
where A † is the pseudo inverse of the matrix A .
Now let ˜ Y be a random variable whose distribution matches the empirical distribution Q nY . Similarly we can write:
[Equation]
For W 22 ( P ˜ Y , P g ( X ) ) , however, we cannot restrict ψ to convex quadratic functions because ˜ Y is a discrete variable while g ( X ) is Gaussian. Thus, Lemma 3 implies that (cid:53) ψ opt for (17) cannot be linear.
Nevertheless, by constraining to quadratic discriminators, we obtain a lower bound:
[Equation]
where ˆ K Y = E [ ˜ Y ˜ Y t ] (the empirical covariance matrix) and Z ∼ N ( 0 , ˆ K Y ) 2 .
Therefore, the empirical constrained quadratic GAN solves the following optimization:
[Equation]
Using Theorem 1, the optimal g ( X ) to this problem is the empirical PCA solution, i.e. keeping the top k principal components of the empirical covariance matrix .
Theorem 3 Under the LQG setup, the solution of the empirical constrained quadratic GAN optimization is equivalent to the empirical PCA.
Consider the case where d = k (the case k < d is similar).
The second term in the generalization distance d G ( P Y , Q nY ) (10) is zero.
Therefore, we have
[Equation]
[Equation]
The W 22 distance between two Gaussians depends only on the covariance matrices.
More speciﬁcally:
[Equation]
[Equation] [Figure] Hence, the convergence of this quantity only depends on the convergence of the empirical covariance to the population covariance, together with smoothness property of this function of the covariance matrices.
The convergence has been established to be at a quick rate of ˜ O
( (cid:112) d/n ) (Rippl et al., 2016).
Finally, note that if one has a ﬁnite number of X samples (replacing P g ( X ) with Q g ( X ) ), the constrained quadratic GAN would still have a fast convergence rate because only the empirical covariance matrix of g ( X ) plays a role in its optimization which converges quickly to the population covariance matrix.
E XPERIMENTAL R ESULTS
For experiments, we generate n i.i.d.
samples from P Y ∼ N ( 0 , I d ) , represented as Q nY .
We then ﬁt a d dimensional Gaussian distribution N ( µ, K ) to Q nY using two methods: Maximum Likelihood (ML) estimation, which computes the sample mean and the empirical covariance; and WGAN (Arjovsky et al., 2017) with an afﬁne generator function.
Note that according to Theorem 3, ML is equivalent to the constrained quadratic GAN (19).
Moreover, note that the WGAN implementation uses W 1 and not W 2 in its optimization.
Although analyzing GANs with W 2 is more tractable than that of W 1 , in practice we do not expect a signiﬁcant difference between their performance.
Considering this and owing to the lack of an implementation of GANs with W 2 , we perform numerical experiments using the WGAN implementation.
Details of the experiments can be found in
Appendix B.
Let ˆ µ and ˆ K be the estimated mean and the covariance.
For evaluation, we compute
[Equation]
22) which is the W 22 distance between N ( 0 , I d ) and N (cid:16) ˆ µ, ˆ K (cid:17) (see Lemma 4 in Appendix C).
Figure 3 demonstrates the estimation error of the ML (constrained quadratic GAN) and WGAN methods for d = 5 and d = 10 and in different sample sizes.
These ﬁgures are consistent with Theorem 2 and results of Section A which suggest that GAN’s convergence can be slow owing to a bias in its optimal empirical solution with respect to the population one.
Moreover, this ﬁgure shows that the convergence of the constrained quadratic GAN (ML) is fast.
Finally, in our experimental results of Figure 3, one should take into the consideration practical issues of the WGAN implementation such as the use of the stochastic gradient descent, convergence to bad locals, etc.  D ISCUSSIONS
From a broader perspective, the problem we addressed in this paper is that of ﬁnding a good generative model for data coming from a Gaussian ground-truth, Y ∼ N ( µ, K ) .
This is an age-old [Figure] real data Discriminator
Generator
[Figure]
problem in statistics, and the baseline solution is using maximum likelihood estimation: one uses the data to estimate the mean and covariance matrix of the Gaussian distribution, i.e. the empirical mean ˆ µ and empirical covariance matrix ˆ K , and obtain a generative model ˆ Y ∼ N (ˆ µ, ˆ K ) .
And when there is a desire to do dimensionality reduction, one can have a low-rank generative Gaussian model retaining the top k principal components of ˆ K .
This is the empirical PCA solution.
What we have shown in this paper is that there is a natural GAN architecture that can accomplish exactly these tasks (Figure 4).
While this is certainly a complicated way of performing maximum likelihood Gaussian estimation and PCA, we believe the result is interesting in several ways.
First, it is not at all obvious that there is a natural GAN architecture that can accomplish this task.
Since Gaussian modeling is a basic task, this is a good sanity check on GANs.
Second, arriving at this GAN architecture requires us to make several advances in our understanding of GANs.
We needed to ﬁnd a general way to specify the loss function for GANs, and then specialize to the quadratic loss function for the Gaussian problem at hand.
This led to the use of the second-order Wasserstein distance for the generator optimization, and to a general GAN architecture from the dual formulation of this optimization.
We then needed to ﬁnd a proper way to constrain the class of generators and the class of discriminators in a balanced way to achieve fast generalization.
Indeed our goal was not to recover the maximum likelihood solution but to overcome the slow generalization when there are no constraints on the generator and the discriminator.
That the ﬁnal architecture with balanced generators and discriminators is also the maximum likelihood solution gives this story a satisfactory ending.
Appendix A A R ATE -D ISTORTION V IEW OF GAN S
As (Arjovsky et al., 2017) suggested, there is a strong connection between GANs and the optimal transport problem (e.g., this connection is reﬂected in the use of the Wasserstein distance in
GAN’s optimization).
Moreover, the optimal transport theory and the optimal quantization theory are strongly tied to one other (e.g., (Villani, 2008; Canas & Rosasco, 2012)).
On the other hand, in the regime where the sample size n grows exponentially in d , Shannon has noted that the optimal quantization is strongly related to the rate-distortion optimization (Cover & Thomas, 2012).
In this section, we exploit these connections to better understand GANs through the rate-distortion analysis.
Note that in the light of Theorem 2 and without properly constraining the discriminator function, the regime where n is exponential in d provides an appropriate resolution to study GAN’s problem.
Consider the following optimization:
[Equation]
where I ( Y ; Z ) is the mutual information between Y and Z .
Let P ∗ Z be an optimal solution of this optimization.
Shannon has shown that in the regime where n = 2 Rd , generating n i.i.d.
samples from P ∗ Z is an optimal quantization strategy of P Y (Cover & Thomas, 2012).
We refer to this optimization as the rate-distortion optimization.
Using the rate-distortion theory, our goal is to understand W 2 and therefore GANs when n grows exponentially in d .
Consider the following optimization:
[Equation]
Note that unlike optimization (23) where P Z can be an arbitrary distribution, in optimization (24) P
Z is constrained to be equal to P Y .
This is to have n i.i.d.
samples from P Z = P Y (represented as Q nY ) be an optimal quantizer of P Y .
Theorem 4 Let D ( R ) be the optimal value of optimization (24) where P Y is absolutely continuous and L Lipschitz.
Then, for a given (cid:15) > 0 , we have
[Equation]
where .
≤ means inequality in the asymptotic regime of n = 2 dR .
Proof 4 See Appendix C.5.
Connecting W 2 to the rate-distortion optimization allows us to exploit standard rate-distortion results to better understand Wasserstein distances and in turn GANs.
For example,
Theorem 5 Let P Y ∼ N ( 0 , I d ) .
Then, the minimum value of optimization (24) is
[Equation]
Proof 5 See Appendix C.6.
For a large value of R , the ﬁrst-order tailor expansion of (26) leads to an expression of d 2 − 2 R which provides a 2 − 2 R = n − 2 /d distortion per dimension.
Next, we consider the empirical GAN optimization (8) when d = k .
This optimization can be written
[Equation]
where ˆ Y = g ( X ) .
Although Theorem 4 indicates a point-wise convergence of the objective of this optimization to a rate-distortion optimization, for an unconstrained P ˆ Y and an arbitrary P Y , a uniform convergence may not occur.
However, we conjecture that such a uniform convergence will occur under the LQG setup.
In such a case, the GAN optimization of (27) can be written as the following optimization:
[Equation]
Conjecture 1 Under the LQG setup, the empirical GAN optimization (27) and the rate-distortion optimization (28) are equivalent.
Note that unlike optimization (24) where both marginal distributions are ﬁxed (and thus, it is a constrained rate-distortion optimization), in optimization (28), one of the marginal distributions ( P ˆ Y ) is not ﬁxed due to the generator optimization.
Therefore, optimization (28) is the same as the rate-distortion optimization (23).
Next, we analyze optimization (28) when P Y ∼ N ( 0 , I d ) which is the rate distortion problem for an i.i.d.
Gaussian source (Cover & Thomas, 2012).
In this case, the optimal P ∗ ˆ Y of optimization (28) is
[Equation]
(29) where D ∗ = 2 − 2 R .
Given conjecture 1, this result has two main consequences:
- In the context of GANs, with X ∼ N ( 0 , I d ) and generator functions g ∈ G , this result suggests that linear generators are optimal when P Y is Gaussian (in the asymptotic regime).
This can be viewed as an unsupervised analogue of the fact that the Minimum Mean Squared Error (MMSE) estimator is linear for the Gaussian supervised learning problem.
The optimal generated distribution P ∗ ˆ Y which is the minimum W 2 distance ﬁtting distribution to n i.i.d.
samples of Y is always a biased estimate of the true distribution P Y ∼ N ( 0 , I d ) .
The bias is in fact simple: it is a shrinkage to the origin by the same amount on every dimension.
The amount of shrinkage is precisely D ∗ = 2 − 2 R .
So far in this section, we have considered the regime where k = d .
The analysis of k < d can be done similarly.
For example, consider the following GAN optimization where ˆ Y is restricted to be in a subspace S whose rank is k ≤ d :
[Equation]
One can decompose Y =
Y S (cid:48)
+ Y S where Y S is the projection of Y onto the subspace S .
Thus, we can re-write optimization (30) as
[Equation]
Similar to optimization (28) and given conjecture 1, we can re-write this optimization as
[Equation]
Let P Y ∼ N ( 0 , I d ) and P Y S ∼ N ( 0 , K S ) where the eigen decomposition of K S is K S =
U S Σ S U tS .
Moreover, suppose diagonal elements of Σ S are σ 2 i ( S ) .
Then, optimal P ∗ ˆ Y for optimization (32) is
[Equation]
where D ∗ i =
min i ( λ, σ 2 i ) and λ is chosen so that R = 1 k (cid:80) ki =1 12 log (cid:0) σ 2 i /D ∗ i (cid:1) (the reverse waterﬁlling solution (Cover & Thomas, 2012)).
Similar to the d = k case, even though the generator functions G were unconstrained, for Gaussian P Y , the optimal generator is linear.
Therefore, the objective function of optimization (32) can be re-written as
[Equation]
The GAN optimization aims to select a subspace to minimize this objective, or alternatively to solve
[Equation]
The ﬁrst term is the energy of the projected signal onto the subspace while the second term is the quantization cost (or, the optimal transport cost) in the subspace.
The larger the energy of the signal is, the larger the quantization cost is, and vice versa.
This tarde-off can lead to having an optimal solution of optimization (35) which is not equal to the PCA subspace.
However, for a low-rank distribution P Y , we show that optimization (35) always selects the PCA subspace:
Theorem 6 Let P Y ∼ N ( 0 , diag (1 , ..., 1 , 0 , ..., 0))
whose covariance has k non-zero eigenvalues.
Then, the PCA-subspace is the optimal solution of optimization (35) .
Proof 6 See Appendix C.7.
B D ETAILS OF E XPERIMENTS
The WGAN is implemented in pytorch.
Denote fully connected layer with the input dimension d in and the output dimension d out as FC ( d in , d out ) .
The generator can be represented as FC ( d, d ) ; and the discriminator can be represented as FC ( d, n f ) − ReLU − FC ( n f , n f ) − ReLU − FC ( n f , n f ) −
ReLU − FC ( n f , 1) .
The model is trained 100k iterations with batch size 128 with Adam optimizer.
The learning rate is set to 2 × 10 − 4 .
As for hyper parameters, n f is set to 128, the ratio of iterations between discriminator and generator is set to 5, and the weight clipping threshold is set to -0.
and 0.02.
Both ML and WGAN are repeated 10 times for each setting, and the mean and standard deviation is calculated and plotted ( 68 . 3% conﬁdence interval).
C P ROOFS C. N OTATION AND P RELIMINARY L EMMAS
For matrices we use bold-faced upper case letters, for vectors we use bold-faced lower case letters, and for scalars we use regular lower case letters.
For example, X represents a matrix, x represents a vector, and x represents a scalar number.
I n is the identity matrix of size n × n .
1 n 1 ,n 2 is the all one matrix of size n 1 × n 2 .
When no confusion arises, we drop the subscripts.
1 { x = y } is the indicator function which is equal to one if x = y , otherwise it is zero.
Tr ( X ) and X t represent the trace and the transpose of the matrix X , respectively.
(cid:107) x (cid:107) 2 = x t x is the second norm of the vector x .
When no confusion arises, we drop the subscript.
(cid:107) X (cid:107) is the operator (spectral) norm of the matrix X .
< x , y > is the inner product between vectors x and y .
A † is the pseudo inverse of the matrix A .
The eigen decomposition of the matrix A ∈ R n × n is denoted by A = (cid:80) ni
=
λ i ( A ) u
i ( A ) u i ( A ) t , where λ i ( A ) is the i -th largest eigenvalue of the matrix A corresponding to the eigenvector u
i
( A ) .
We have λ 1 ( A ) ≥ λ 2 ( A ) ≥ · · · .
N ( µ, K ) is the Gaussian distribution with mean µ and the covariance K .
KL ( P X , P Y ) is the Kullback Leibler divergence between two distributions P X and
P Y .
˜ O
( d ) is the same as O ( d log( d )) .
Lemma 4 Let Y ∼ N ( 0 , K Y ) and ˆ Y ∼ N ( 0 , K ˆ Y ) .
Then,
[Equation]
Proof 7 See reference (Givens et al., 1984).
C. P ROOF OF L EMMA
Let Y =
Y S (cid:48)
+ Y S where Y S represents the projection of Y onto the subspace S .
Since
[Equation]
(37) choosing ˆ Y =
Y S achieves the minimum of optimization (9).
C. P ROOF OF T HEOREM
Let S be a ﬁxed subspace of rank k where ˆ Y lies on.
According to Lemma 1, if ˆ Y is unconstrained, the optimal ˆ Y ∗ is the projection of Y onto S (i.e., ˆ Y ∗ =
Y S ).
Moreover, since Y is Gaussian, ˆ Y is also Gaussian.
Therefore, there exists a linear g ( . )
such that P ˆ Y ∗ =
P g ( X ) where X ∼ N ( 0 , I ) .
Thus, the problem simpliﬁes to choosing a subspace where E (cid:2) (cid:107) Y S
(cid:107) 2 (cid:3) is maximized, which is the same as the PCA optimization.
C. P ROOF OF T HEOREM
Let y 1 ,..., y n be n i.i.d.
samples of P Y .
Let ˆ µ be the sample mean.
Since P Y is absolutely continuous, the optimal W 2 coupling between Q nY and P Y is deterministic (Villani, 2008).
Thus, every point y i is coupled with an optimal transport vornoi region with the centroid c ( µ, K )
y i .
Therefore, we have
[Equation]
where
[Equation]
The ﬁrst step in (38) follows from the deﬁnition of W 2 , the second step follows from the optimal coupling between N ( µ, K ) and N ( 0 , I ) , and the third step follows from the matrix trace equalities.
Therefore,
[Equation]
[Equation]
which leads to µ ∗ n = ˆ µ .
Moreover, each component of the sample mean is distributed according to
N ( 0 , 1 /n ) .
Thus, (cid:107) µ ∗ N (cid:107) 2 ∼ χ 2 d
/n
which goes to zero with the rate of ˜ O
( d/n ) .
Let σ
i be the i -th diagonal element of Σ .
Moreover, deﬁne B = U t AU .
Therefore, we have
[Equation]
41) where b i,i is the i -th diagonal element of the matrix B .
Thus, σ ∗ i
=
b i,i and Tr ( Σ ∗ ) =
Tr ( B ) =
Tr ( A ) .
Furthermore, we have
[Equation]
which goes to zero with the rate of O ( n − 2 /d )
(Canas & Rosasco, 2012).
Since 1 N (cid:80) Ni =1 (cid:107)
y i (cid:107) 2 goes to d with the rate of O ( (cid:112) d/n ) (because it has a χ -squared distribution),
Tr ( A ) goes to d with a rate of O ( n − 2 /d ) .
Combining this result with Tr ( Σ ∗ ) =
Tr ( A ) completes the proof.
C. P ROOF OF T HEOREM
The lower-bound follows directly from the rate-distortion theorem (Cover & Thomas, 2012).
Thus, we focus on proving the upper bound.
To show this, we use the triangle inequality of the Wdistance: W 2 ( P Y , Q nY ) ≤ W 2 ( P Y , P Z )
+ W 2 ( P Z , Q nY ) , for some P Z that we will construct.
We ﬁrst introduce some notation: Let A
= { y 1 , ..., y n } be n i.i.d.
samples from P Y , with the empirical distribution Q nY .
Moreover, let B = { b 1 , ..., b n (cid:48) } be another n (cid:48)
= 2 d ( R − (cid:15) )
i.i.d.
samples from P Y .
Let V i be the voronoi region assigned to point b i .
Let
[Equation] [Equation]
Now, deﬁne A i as the subset of samples of A that fall in the voronoi region V i and let N i =
| A i | .
N i can be written as the sum of indicator functions:
[Equation]
Thus, N i is Binomial with the parameter p V i .
Deﬁne the probability measure P Z on the set
A such that for every y i
∈
V j , we have
[Equation] [Equation]
First, we show that W 2 ( P Z , Q nY ) goes to zero as n increases.
Let Z i :=
(cid:112) n/p V i .
Note that with high probability | Z i | < n 2 − d(cid:15) (cid:48) for 0 < (cid:15) (cid:48) < (cid:15) .
Thus, with high probability, | p i − 1 /n | < 2 − d(cid:15) (cid:48)
/n .
Therefore, with high probability, the total variation distance between P Z and Q nY can be bounded as (cid:107) P Z − Q nY (cid:107) TV ≤ O (2 − d(cid:15) (cid:48) ) .
Moreover, from Theorem 6.15 of (Villani, 2008), we have
[Equation] [Equation]
where r is the diameter of the space.
In our case, r = O ( d ) .
Thus,
[Equation]
47) which goes to zero as n increases.
Next, we show that W 22 ( P Y , P Z ) is less than R ( D − (cid:15) ) .
To show this, we will construct an admissible transport plan whose cost is less than R ( D − (cid:15) ) : in each region V i , we solve an optimal transport plan between P Y | Y ∈V i and A i points.
Combining these local transport plan provides an admissible transport plan between P Y and P Z .
Thus, W 22 ( P Y , P Z ) is smaller than or equal to the aggregate cost of local transport plans.
Recall that the region V i contains N i points of A .
As n → ∞ , N i increases as well.
Since P Y is absolutely continuous and Lipschitz, as n → ∞ , P Y in each region resembles a uniform distribution.
Since the number of points in the region increases, the optimal transport cost between P Y | Y ∈V i and points A i is smaller than the transport cost between P Y | Y ∈V i and its centroid.
Therefore, the overall transport cost between P Y and P Z (by combining local transports) is smaller than D ( R − (cid:15) ) .
This completes the proof.
[Equation]
P ROOF OF T HEOREM
Let us ﬁrst show that the claimed minimum value is achieved with a certain coupling dictated by the following virtual channel: Y = cZ + W where W follows N ( 0 , (1 − c 2 ) I d ) and is independent of Z .
Note that this coupling together with a choice P Z ∼ N ( 0 , I d ) still respects P Y
= P Z ∼ N ( 0 , I d ) .
Another constraint I ( Y ; Z ) ≤ Rd yields a condition on c : c ≤ √ 1 − 2 − 2 R .
Notice that
[Equation]
This together with the minimum value of c = √ 1 − 2 − 2 R then gives:
[Equation]
Now let us prove the converse: the optimal solution cannot go below the claimed bound.
Consider a constraint: dR ≥ I ( Y ; Z )
= h ( Y ) − h
( Y | Z ) .
Note that h ( Y | Z ) is maximized when ( Y, Z ) are jointly Gaussian.
So in an effort to ﬁnd a lower bound on I ( Y ; Z ) , we consider such a jointly
Gaussian setting, i.e., Y = cZ + W where W ∼ N ( 0 , (1 − c 2 ) I d ) , independent of Z .
This together with a simple calculation then yields: c ≤ √ 1 − 2 − 2 R .
Putting this to the objective function, we get:
[Equation]
This completes the proof.
C. P ROOF OF T HEOREM
The objective function of optimization (35) for the PCA subspace is k (1 − 2 − 2 R ) .
Now consider a subspace S
such that (cid:80) ki
=1 σ 2 i ( S ) = αk where 0 < α < 1 .
Below, we show that this subspace cannot be the optimal solution of optimization (35).
We have
[Equation]
where the ﬁrst step follows from the fact that σ 2 i ( S ) ≤ 1 and the second step follows from the log-sum inequality.
This inequality leads to
[Equation]
Thus, for this subspace, the objective function of optimization (35) is upper bounded by αk − αk 2 − 2 R/α , which for 0 < α < 1 is always less than k (1 − 2 − 2 R ) , the PCA-subspace objective function.
This completes the proof.