I WGAN: AN A UTOENCODER WGAN FOR I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Adversarial Networks (GANs) have been impactful on many problems and applications but suffer from unstable training.
Wasserstein GAN (WGAN) leverages the Wasserstein distance to avoid the caveats in the minmax two-player training of GANs but has other defects such as mode collapse and lack of metric to detect the convergence.
We introduce a novel inference WGAN (iWGAN) model, which is a principled framework to fuse auto-encoders and WGANs.
The iWGAN jointly learns an encoder network and a generative network using an iterative primal dual optimization process.
We establish the generalization error bound of iWGANs.
We further provide a rigorous probabilistic interpretation of our model under the framework of maximum likelihood estimation.
The iWGAN, with a clear stopping criteria, has many advantages over other autoencoder GANs.
The empirical experiments show that our model greatly mitigates the symptom of mode collapse, speeds up the convergence, and is able to provide a measurement of quality check for each individual sample.
We illustrate the ability of iWGANs by obtaining a competitive and stable performance with state-of-the-art for benchmark datasets.
I NTRODUCTION
One of the goals of generative modeling is to match the model distribution P θ ( x ) with parameters θ to the true data distribution P X for a random variable X ∈ X .
For latent variable models, the data point X is generated from a latent variable Z ∈ Z through a conditional distribution P ( X | Z ) .
Here X denotes the support for P X and Z denotes the support for P Z .
There has been a surge of research on deep generative networks in recent years and the literature is too vast to summarize here (Kingma & Welling, 2013; Goodfellow et al., 2014;
Li et al., 2015).
These models have provided a powerful framework for modeling complex high dimensional datasets.
We start introducing two main approaches for generative modeling.
The ﬁrst one is called variational auto-encoders (VAEs) (Kingma & Welling, 2013), which use variational inference to learn a model by maximizing the lower bound of the likelihood function.
VAEs have elegant theoretical foundations but the drawback is that they tend to produce blurry images.
The second approach is called generative adversarial networks (GANs) (Goodfellow et al., 2014), which learn a model by using a powerful discriminator to distinguish between real data points and generative data points.
GANs produce more visually realistic images but suffer from the unstable training and the mode collapse problem.
Although there are many variants of generative models trying to take advantages of both VAEs and GANs (Tolstikhin et al., 2017; Rosca et al., 2017), to the best of our knowledge, the model which provides a unifying framework combining the best of VAEs and GANs in a principled way is yet to be discovered.
1. R ELATED W ORK
GANs and WGANs:
The generative model is to learn a mapping, denoted by G , from Z to X to approximate the conditional distribution P G ( X | Z ) of the data point X ∈ X given latent code Z ∈ Z .
We consider the deterministic mapping G in this work.
Both GANs and Wasserstein GANs (WGANs)
(Arjovsky et al., 2017) can be viewed as minimizing certain divergence between the data distribution
P X and the generative model distribution P G ( Z ) .
For example, the Jensen-Shannon (JS) divergence is implicitly used in GANs (Goodfellow et al., 2014).
The 1 -Wasserstein distance between P X and
P G ( Z ) , denoted by W 1 ( P X , P G ( Z ) ) , is employed in WGANs.
Empirical experiments suggest that the Wasserstein distance is a more sensible measure to differentiate probability measures supported in low-dimensional manifold.
In terms of training, it turns out that it is hard or even impossible to compute these standard divergences in probability, especially when P X is unknown and P G ( Z ) is parameterized by deep neural networks (DNNs).
The training of GANs is converted into playing a game between two competing networks: the generator and the discriminator .
The generator is to fool the discriminator and the discriminator is to distinguish between true data samples and generated samples.
Instead, the training of WGANs is to study its dual problem because of the elegant form of Kantorovich-Rubinstein duality (Villani, 2008).
Analogous to GANs, the discriminator is now a real-valued 1 -Lipschitz function.
Many techniques such as weight clipping (Arjovsky et al., 2017) and gradient penalty (Gulrajani et al., 2017) are used to enforce the Lipschitz constraint.
More discussions about WGANs will be presented in Section 2.
Autoencoder GANs: Larsen et al.
ﬁrst introduced the VAE-GAN, which is a hybrid of VAEs and GANs.
The VAE-GAN uses a GAN discriminator to replace a VAE’s decoder to learn the loss function.
The motivation behind this modiﬁcation is that VAEs tend to produce blurry outputs during the reconstruction phase.
More recent VAE-GAN variants, such as Adversarial Generator Encoders (AGE) (Ulyanov et al., 2018) and Auto-encoding GANs ( α -GAN) (Rosca et al., 2017), use a separate encoder to stabilize GAN training.
The main difference with standard GANs is that, besides the generator G , there is an encoder Q : X → Z which maps the data points into the latent space.
This encoder is to approximate the conditional distribution Q ( Z | X ) of the latent variable
Z given the data point X .
Other encoder-decoder GANs are introduced in Adversarially Learned Inference (ALI) (Dumoulin et al., 2016) and Bidirectional Generative Adversarial Networks (BiGAN) (Donahue et al., 2016).
The objective of both ALI and BiGAN is to match two joint distributions under the framework of vanilla GANs, the joint distribution of ( X, Q ( X )) and the joint distribution of ( G ( Z ) , Z ) .
When the algorithm achieves equilibrium, these two joint distributions roughly match.
We are able to obtain more meaningful latent codes by Q ( X ) , and this should improve the quality of the generator as well.
Adversarial Variational Bayes (AVB) (Mescheder et al., 2017) presented a more ﬂexible latent distribution to train Variational Autoencoders.
Hu et al.
(2017) provided new interpretations of GANs and VAEs and revealed strong connections between them which are linked by the classic wake-sleep algorithm.
Duality in GANs: Regarding the optimization perspectives of GANs, (Chen et al., 2018; Zhao et al., 2018) studied duality-based methods for improving algorithm performance for training.
Primal-dual
Wasserstein GANs (PD-GANs) are introduced in (Gemici et al., 2018), which proposed a new penalty term whose evaluation samples are obtained from the encoder Q .
Farnia & Tse (2018) developed a convex duality framework to address the case when the discriminator is constrained into a smaller class.
Grnarova et al.
(2018) developed an evaluation metric to detect the non-convergence behavior of vanilla GANs, which is the duality gap deﬁned as the difference between the primal and the dual objective functions.
Husain et al.
(2019) investigated the close relationship between WAE (Tolstikhin et al., 2017) and f-GANs (Nowozin et al., 2016), and proved generalization results for autoencoder models.
1. O UR C ONTRIBUTIONS
Although there are many interesting works on autoencoder GANs, it remains unclear what the principles are underlying the fusion of auto-encoders and GANs.
For example, do there even exist these two mappings, the encoder Q and the decoder G , for any high-dimensional random variable X , such that Q ( X ) has the same distribution as Z and G ( Z ) has the same distribution as X ?
Is there any probabilistic interpretation such as the maximum likelihood principle on autoencoder GANs?
We introduce inference Wasserstein GANs (iWGANs), which provide satisfying answers for these questions.
We focus on the 1-Wasserstein distance, instead of the Kullback-Leibler divergence.
We borrow the strength from both the primal and the dual problems and demonstrate the synergistic effect between these two optimizations.
The encoder component tends out to be a natural consequence from our algorithm.
Furthermore, the iWGAN has a rigorous probabilistic interpretation under the maximum likelihood principle, and our learning algorithm is equivalent to the maximum likelihood estimation when our model is deﬁned as an energy-based model based on an autoencoder.
Our main contributions are listed as below:
We propose a novel framework, called iWGAN, to learn both an encoder and a decoder simultaneously.
We prove the existence of meaningful encoder and decoder, establish an equivalence between WGAN and iWGAN, and develop a generalization error bound for iWGAN.
We establish a rigorous probability interpretation for iWGANs and our training process is exactly the same as the maximum likelihood estimation.
As a byproduct, this interpretation allows us to perform the quality check at the individual sample level.
We demonstrate the natural use of the duality gap as a measure of convergence for iWGANs, and show its effectiveness for various numerical settings.
Our experiments do not experience any mode collapse problem.
I WGAN
The autoencoder generative model consists of two parts: an encoder Q and a generator G .
The encoder Q maps a data sample x ∈ X to a latent variable z ∈ Z , and the generator G takes a latent variable z ∈ Z to produce a sample G ( z ) .
The autoencoder generative model should satisfy the following three conditions simultaneously : (a) The generator can generate images which have a similar distribution with observed images; (b) The encoder can produce meaningful encodings in the latent space; (c) The reconstruction errors of this model based on these meaningful encodings are small.
The beneﬁt of using an autoencoder is to encourage the model to better represent all the data it is trained with, so that it discourages mode-collapse.
We ﬁrst show that, for any distribution residing on a smooth manifold, there always exists an encoder Q ∗ which guarantees meaningful encodings and exists a generator G ∗ which generates samples with the same distribution as data points by using these meaningful codes.
Theorem 2.1.
Consider a continuous random variable X ∈ X , where X is a d -dimensional smooth Riemannian manifold.
Then, there exist two mappings Q ∗ : X → R p and G ∗ : R p → X , with p = max { d ( d + 5) / 2 , d ( d + 3) / 2 + 5 } , such that Q ∗ ( X ) follows a multivariate normal distribution with zero mean and identity covariance matrix and G ∗ ◦ Q ∗ is an identity mapping, i.e.,
X =
G ∗ ( Q ∗ ( X )) .
Learning Q ∗ and G ∗ from the data points is a challenging task.
Recall that the 1 -Wasserstein distance between the data distribution P X and the generative model distribution P G ( Z ) is deﬁned as
[Equation]
where (cid:107) · (cid:107) represents the L 2 -norm and Π( P X , P Z ) is a set of all joint distributions of ( X, Z ) with marginal measures P X and P Z , respectively.
The main difﬁculty in (1) is to ﬁnd the optimal coupling π and this is a constrained optimization with P X ( x ) =
(cid:82) π ( x, z ) dz for x ∈ X and P Z ( z ) =
(cid:82) π ( x, z ) dx for z ∈ Z .
Based on the Kantorovich-Rubinstein duality, the WGAN studies the 1 -Wasserstein distance (1) from the dual format
[Equation]
where F is the set of all bounded 1 -Lipschitz functions.
Weight clipping (Arjovsky et al., 2017) and gradient penalty (Gulrajani et al., 2017) have been used to satisfy the constraint of Lipschitz continuity.
The experiment of (Arjovsky et al., 2017) showed that the WGAN can avoid the problem of gradient vanishment. However, the WGAN does not produce meaningful encodings and many experiments still display the problem of mode collapse (Arjovsky et al., 2017; Gulrajani et al., 2017).
On the other hand, the Wasserstein Autoencoder (WAE) (Tolstikhin et al., 2017), after introducing an encoder Q : X → Z to approximate the conditional distribution of Z given X , studies the reconstruction error
[Equation]
X − G ( Q ( X )) (cid:13)(cid:13) , where Q is a set of encoder mappings whose elements satisﬁes P Q ( X ) =
P Z . The penalty, such as D ( P Q ( X ) , P Z ) , is added to the objective to satisfy this constraint, where D is an arbitrary divergence between P Q ( X ) and P Z . The WAE can produce meaningful encodings and controlled reconstruction error. However, the WAE deﬁnes a generative model in an implicit way and does not model the generator through G ( Z ) with Z ∼ P Z directly.
To take the advantages of both WGAN and WAE, we propose a new autoencoder GAN model, called iWGAN, whose objective is
[Equation]
The term (cid:107) X − G ( Q ( X )) (cid:107) can be treated as the autoencoder reconstruction error as well as a loss to match the distributions between X and G ( Q ( X )) .
We note that the L 1 -norm (cid:107) · (cid:107) 1 has been used for the reconstruction term by α -GAN (Rosca et al., 2017) and CycleGAN (Zhu et al., 2017).
Another term E X ∼ P
X
f
( G ( Q ( X )))
− E Z ∼ P Z f ( G ( Z )) can be treated as a loss for the generator as well as a loss to match the distribution between G ( Q ( X )) and G ( Z ) .
We emphasize that this term is different with the objective function of the WGAN in (2).
Furthermore, it is challenging for practitioners to determine when to stop training GANs.
Most of the GAN algorithms do not provide any explicit standard for the convergence of the model.
However, the measure of convergence for iWGAN becomes very natural and we use the duality gap as the measure.
The duality gap can be deﬁned as
[Equation]
where L ( G, Q, f ) =
E X (cid:107) X − G ( Q ( X ))
(cid:107) + E X
[ f ( G ( Q ( X )))]
− E Z [ f ( G ( Z ))] .
Theorem 2.2.
The iWGAN objective (3) is equivalent to
[Equation]
Therefore, W 1 ( P X , P G ( Z ) ) ≤ W 1 ( P X , P G ( Z ) ) .
If there exists a Q ∗ ∈ Q such that Q ∗ ( X ) has the same distribution with Z , then W 1 ( P X , P G ( Z ) )
=
W 1 ( P X , P G ( Z ) ) .
Let ( (cid:101) Q, (cid:101) G, (cid:101) f ) be a ﬁxed solution and assume that the encoder, generator, and discriminator all have enough capacities.
Then the duality gap is larger than W 1 ( P X , P (cid:101) G (
(cid:101) Q ( X )) )
+ W 1 ( P (cid:101) G
( (cid:101) Q ( X )) , P (cid:101) G ( Z ) ) .
Moreover, if (cid:101) G outputs the same distribution as X and (cid:101) Q outputs the same distribution as Z , both the duality gap and
W 1 ( P X , P (cid:101) G ( Z ) ) are zeros and X = (cid:101) G ( (cid:101) Q ( X )) for X ∼ P X .
According to Theorem 2.2, the iWGAN objective is in general the upper bound of W 1 ( P X , P G ( Z ) ) .
However, this upper bound is tight.
When the space Q includes a special encoder Q ∗ such that
Q ∗ ( X ) has the same distribution as Z , the iWGAN objective is the exactly same as W 1 ( P X , P G ( Z ) ) .
Theorem 2.2 also provides an appealing property from a practical point of view.
The values of the duality gap and W 1 ( P X , P (cid:101) G ( Z ) ) give us a natural criteria to justify the algorithm convergence.
G ENERALIZATION E RROR B OUND AND T HE A LGORITHM
In practice, we minimize the empirical version, (cid:99) W 1 ( P X , P G ( Z ) ) , of W 1 ( P X , P G ( Z ) ) to learn both the encoder and the decoder.
Before we present the details of the algorithm, we ﬁrst develop the generalization error bound for iWGANs.
For discussions of generalization performance of classical
GANs, see Arora et al.
(2017) and Jiang et al.
Theorem 3.1.
Given a generator G ∈ G , and given n samples ( x 1 , . . .
, x n ) from X
= { x : (cid:107) x (cid:107) ≤ B } , with probability at least 1 − δ for any δ ∈ (0 , 1) , we have
[Equation]
where (cid:98) R n ( F ) = E (cid:15) (cid:2) sup f ∈F n − 1 (cid:80) ni
=1 (cid:15)
i f ( x i ) (cid:3) is the empirical Rademacher complexity of the 1-Lipschitz function set F , in which (cid:15) i is the Rademacher variable.
Theorem 3.1 indicates that the 1-Wasserstein distance between P X and P G ( Z ) can be dominantly upper bounded by the empirical (cid:99) W 1 ( P X , P G ( Z ) ) and Rademacher complexity of F .
Since (cid:99) W 1 ( P X , P G ( Z ) ) ≤ (cid:99) W 1 ( P X , P G ( Q ( X )) )
+ (cid:99) W 1 ( P G ( Q ( X )) , P G ( Z ) ) for any Q ∈ Q , the capacity of Q determines the value of (cid:99) W 1 ( P X , P G ( Z ) ) .
On the other hand, there are several existing results on the empirical Rademacher complexity of neural networks.
When F is a set of 1-Lipschitz neural networks, we can apply the conclusion from Bartlett et al.
(2017) to (cid:98) R n ( F ) , which produces an upper bound scaling as O ( B (cid:112) L 3 /n ) .
Here L denotes the depth of network f ∈ F .
Similar upper bound with an order of O ( B (cid:112)
Ld 2 /n ) can be obtained by utilizing the results from Li et al.
(2018), where d is the width of the network.
In practice, we adopt the gradient penalty deﬁned as GP ( f ) =
E X (cid:2) ( (cid:107)∇ X f ( X ) (cid:107) 2 − 1) 2 (cid:3) in (Gulrajani et al., 2017) to enforce the 1-Lipschitz constraint on f ∈ F .
In addition, we use the maximum mean discrepancy (MMD) penalty (Gretton et al., 2012), denoted by MMD k
( P Q ( X ) , P Z ) , to enforce
Q ( X ) to converge to P Z , where k is a kernel function.
The details of the algorithm are presented in
Algorithm 1.
Algorithm
1 while DualGap > (cid:15) 1 or L ( G i , Q i , f i ) > (cid:15) 2 do for t = 1 , ..., n critic do
Sample real data x i ∼ P X , latent variable z
i ∼ P Z and a random number (cid:15) ∼ U [0 , 1] ˆ x i ← (cid:15)x i
+ (1 − (cid:15) )
G i ( z i )
Calculate L i =
L ( G i , Q i , f i | x i , z i ) and gradient of − L i
Update f by Adam: f i +1 ← Adam ( −∇ f L i ) where for f i , −∇
[Equation]
= ∇
[Equation]
nk
[Equation] [Equation]
f
[Equation]
− f
[Equation]
+
[Equation]
f
[Equation]
ˆ x
[Equation]
(cid:107) 2 − 1)
[Equation]
(cid:17) end for t = 1 , ..., n critic do
Sample real data x (cid:48)
i ∼ P X , latent variable z (cid:48)
i ∼ P
Z
Calculate L (cid:48) i
=
L ( G i , Q i , f i +1 | x (cid:48)
i , z (cid:48) i ) and gradient of L (cid:48)
i
Update G , Q by Adam: G i +1 , Q
i +1 ← Adam ( ∇ G,Q L (cid:48) i )
where for G
i , Q i ,
[Equation]
i
[Equation]
) (cid:88) l (cid:54) =
j
[Equation] [Equation]
[Equation]
(cid:88) l,j k (
[Equation]
il , Q (
[Equation]
ij )) end 16 end
P ROBABILISTIC I NTERPRETATION AND THE MLE
The iWGAN has proposed an efﬁcient framework to stably and automatically estimate both the encoder and the generator.
In this section, we provide a probabilistic interpretation of the iWGAN under the framework of maximum likelihood estimation.
Maximum likelihood estimator (MLE) is a fundamental statistical framework for learning models from data.
However, for complex models, MLE can be computationally prohibitive due to the intractable normalization constant.
MCMC has been used to approximate the intractable likelihood function but do not work efﬁciently in practice.
The iWGAN can be treated as an adaptive method for MLE training, which not only provides computational advantages but also allows us to generate more realistic-looking images.
Furthermore, this probabilistic interpretation enables other novel applications such as image quality checking and outlier detection.
Let X denote the image.
Deﬁne the density of X by an energy-based model based on an autoencoder (Zhao et al., 2016; Berthelot et al., 2017):
[Equation]
where θ is the unknown parameter and V ( θ ) is the log normalization constant.
The major difﬁculty for the likelihood inference is due to the intractable function V ( θ ) .
Suppose that we have the observed data { x i : i = 1 , . . . , n } .
The log-likelihood function of θ is (cid:96) ( θ ) = n − 1 (cid:80) ni
=1 log p ( x i | θ ) , whose gradient is
[Equation]
where ˆ E obs [ · ] denotes the empirical average on the observed data { x i } and E θ [ · ] denotes the expectation under model p ( x | θ ) .
The key computational obstacle lies in the approximations of the model expectation E θ [ · ] .
To address this problem, we propose a novel dual approximation for this expectation.
By Theorem 5.10 of Villani (2008), there exists an optimal f ∗ such that
[Equation]
for the optimal coupling π .
Therefore, there exists a f ∗ such that f ∗ ( x ) − f ∗ ( G θ ( Q θ ( x )))
= (cid:107) x − G θ ( Q θ ( x ))
(cid:107) with probability one with respect to the distribution of x .
Since E θ in (7) is taken under the current estimated θ
and we also require G θ to be a good generator and the distributions of G θ ( z ) and G θ ( Q θ ( x )) to be close, we approximate (cid:107) x − G θ ( Q θ ( x ))
(cid:107) by f ∗ ( G θ ( z ))
− f ∗ ( G θ ( Q θ ( x ))) .
We replace (cid:107) x − G θ ( Q θ ( x ))
(cid:107) in the second term of (7) by f ∗ ( G θ ( z ))
− f ∗ ( G θ ( Q θ ( x ))) , yielding a gradient update for θ of form θ ← θ + (cid:15) ˆ ∇ θ (cid:96) ( θ ) , where
[Equation]
Here f ∗ needs to be learned and is solved by the corresponding dual problem at each iteration.
We approximate f ∗ by a network f η with an unknown parameter η , yielding a gradient update for η of form
[Equation]
The advantage of using expectations in (9) and (10) is that we can evaluate them by using only marginal distributions of z and x .
The above iterative updating process is exactly the same as in
Algorithm 1.
Therefore, the training of iWGAN is to seek the MLE.
This probabilistic interpretation provides a novel alternative method to tackle problems with the intractable normalization constant in latent variable models.
The MLE gradient update of p ( x | θ ) decreases the energy of the training data and increases the dual objective.
Compare with original GANs or WGANs, our method gives much faster convergence and simultaneously provides a higher quality generated images.
The probabilistic modeling opens a door for many interesting applications.
Next, we present a completely new approach for determining a highest density region (HDR) estimate for the distribution of X .
What makes HDR distinct from other statistical methods is that it ﬁnds the smallest region, denoted by U ( α ) , in the high dimensional space with a given probability coverage 1 − α , i.e.,
P ( X ∈
U
( α ))
= 1 − α .
We can use U ( α ) to assess each individual sample quality.
Note that FID or the Inception score are used to measure the whole sample quality, not at the individual sample level.
Let ˆ θ be the MLE.
The density ratio at x 1 and x 2 is
[Equation]
The smaller the reconstruction error is, the larger the density value is.
We can deﬁne the HDR for x through the HDR for the reconstruction error
e x := (cid:107) x − G ˆ θ ( Q ˆ θ ( x )) (cid:107) , which is simple because it is a one-dimensional problem.
Let ˜ U
( α ) be the HDR for e x .
Then, U ( α ) =
{ x : e x ∈ ˜
U
( α ) } .
Here
Q ˆ θ ( U ( α )) deﬁnes the corresponding region in the latent space, which can be used to generate better quality samples.
E XPERIMENTAL R ESULTS 5. M IXTURE OF G AUSSIANS
We train our iWGAN model on three toy datasets with an increasing difﬁculty shown on the right:
[Figure]
a).
RING: a mixture of 8 Gaussians, b).
SPIRAL: a mixture of 20 Gaussians and c).
GRID: a mixture of
Gaussians.
As the true data distributions are known, this setting allows for tracking of convergence and mode dropping.
Duality gap and convergence: We illustrate that as the duality gap converges to 0 , our model converges to the generated samples from the true distribution.
We keep track of the generated samples using G ( z ) and record the duality gap at each iteration to check the corresponding generated samples.
Figureshows the generated samples converge to the true distribution very fast without the mode collapse problem.
We compare our method with WGAN-GP in Figure 2.
Both methods adopt the same structure, learning rate, number of critical steps, and other hyper-parameters.
The iWGAN surpasses the performance of the WGAN-GP at very early stage and avoids the appearance of mode collapse.
Latent Space:
We choose the latent distribution to be a ﬁve dimensional standard normal distribution
Z ∼ N (0 , I 5 ) .
After training, the distribution of Q ( X ) is expected to be close to the distribution of
Z .
We plot the Q ( X ) i against Q ( X ) j for all i (cid:54) =
j in Figure 3.
We can tell that the joint distribution of any two dimensions of Q ( X ) is close to a bivariate normal distribution.
[Figure]
Individual sample quality check:
From the probability interpretation of iWGANs, we naturally adopt the reconstruction error (cid:107) X − G ( Q ( X ))
(cid:107) , or the quality score
[Equation]
as the metric of the quality of any individual sample.
The larger the quality score is, the better quality the sample has.
Figure 4 shows their quality scores for different samples.
The quality scores of samples near the modes of the true distribution are close to 1, and become smaller as the sample draw is away from the modes.
This indicates that the iWGAN does converge and learns the distribution well, and the quality score is a reliable metric for the individual sample quality.
5. MNIST & C ELEB A
We experimentally demonstrate our model’s ability on well-known datasets, MNIST and CelebA. The results by iWGAN on MNIST and
CelebA are shown in Figure 5 and Figure 6, respectively.
For latent space interpolations between
MNIST or CelebA validation set examples.
We sample pairs of validation set examples x 1 and xand project them into z 1 and z 2 by the encoder.
We then linearly interpolate between z 1 and z 2 and pass the intermediary points through the decoder to plot the input-space interpolations.
Figure displays images with high and low quality scores selected from CelebA. More experimental results, including reconstructed images, interpolations, and comparison with state-of-the-art results by
WGAN-GP, are shown in Appendix.
[Figure] [Figure]
C ONCLUSION
We have developed a novel iWGAN model, which fuses auto-encoders and GANs in a principle way.
We have established a generalization error bound for iWGAN.
We have provided a solid probabilistic interpretation on iWGAN using the maximum likelihood principle.
Our training algorithm with an iterative primal and dual optimization has demonstrated an efﬁcient and stable learning.
We have proposed a stopping criteria for our algorithm and a metric for individual sample quality checking.
The empirical results on both synthetic and benchmark datasets are state-of-the-art.
We now mention several future directions for research on iWGAN.
First, one might be interested in applying iWGAN into image-to-image translation, as the extension should be straightforward.
A second direction is to develop a formal hypothesis testing procedure to test whether the samples generated from iWAGN is the same as the data distribution.
A PPENDIX A PPENDIX A: P ROOF OF T HEOREM 2.
According to the Nash embedding theorem (Nash, 1956; Günther, 1991), every d -dimensional smooth Rieman- nian manifold X possesses a smooth isometric embedding into R p with p = max { d ( d +5) / 2 , d ( d +3) / 2+5 } .
Therefore, there exists an injective mapping u : X → R p which preserves the metric in the sense that the manifold metric on X is equal to the pullback of the usual Euclidean metric on R p by u .
The mapping u is injective so that we can deﬁne the inverse mapping u − 1 : u ( X ) → X .
Let ˜ X
= u ( X ) ∈ R p , and write ˜ X
= ( ˜ X 1 , . . .
, ˜ X p ) .
Let F i ( x )
=
P
˜ X i ≤ x ) , i = 1 , . .
. , p , be the marginal cdfs.
By applying the probability integral transformation to each component, the random vector (cid:0)
[Equation]
[Equation] [Equation]
:=
[Equation]
˜
[Equation] [Equation]
˜
[Equation]
, F
[Equation]
X
[Equation]
has uniformly distributed marginals.
Let C : [0 , 1] p
→
[0 , 1] be the copula of ˜ X , which is deﬁned as the joint cdf of ( U 1 , . .
, U p ) :
C ( u 1 , u 2 , . . . , u p ) =
P (cid:0) U 1 ≤ u 1 , U 2 ≤ u 2 , . .
, U p ≤ u p (cid:1) .
The copula C contains all information on the dependence structure among the components of ˜ X , while the marginal cumulative distribution functions F i contain all information on the marginal distributions.
Therefore, the joint cdf of ˜ X is
[Equation]
. . , F p (˜ x p ) (cid:1) .
Deﬁne, for i = 2 , . . .
, p ,
[Equation]
The conditional distribution of U k , given U 1 , . .
, U k − 1 , is given by Cherubini et al.
[Equation]
for k = 2 , . . .
, p .
We will construct Q ∗ as follows.
First, we obtain ˜ X ∈ R p by ˜ X = u ( X ) .
Second, we transform ˜ X into a random vector with uniformly distributed marginals ( U 1 , . . .
, U p ) by the marginal cdf F i .
Then, deﬁne ˜ U
1 =
U 1 and ˜ U
k = C k (cid:0) U k |
U 1 , . .
U k − 1 (cid:1) , k = 2 , . .
p.
Hence, ˜ U 1 , . .
˜ U p are independent uniform random variables.
Finally, let Z i =
Φ − 1 ( U i ) for i = 1 , . . .
, p .
This completes the transformation Q ∗ from X to Z =
( Z 1 , . . .
, Z p ) .
The above process can be inverted to obtain G ∗ .
First, we transform Z into independent uniform random variables by ˜ U
i =
Φ( Z i ) for i = 1 , . . .
, p .
Next, let U 1 = ˜
U 1 .
Deﬁne
[Equation]
where C − 1 k ( ·| u 1 , . . .
, u k ) is the inverse of C k and can be obtained by numerical root ﬁnding.
Finally, let ˜
X i =
F − 1 i ( U i ) for i = 1 , . . . , p and X = u − 1 ( ˜ X ) , where u − 1 : u ( X ) → X is the inverse mapping of u .
This completes the transformation G ∗ from Z to X .
A PPENDIX B: P ROOF OF T HEOREM 2.
By the iWGAN objective (3), (5) holds.
Since W 1 is a distance between two probability measures, W 1 ( P X , P G ( Z ) ) ≤ W 1 ( P X , P G ( Z ) ) .
If there exists a Q ∗ ∈ Q such that Q ∗ ( X ) has the same distribution as P
Z , we have
W
[Equation] [Equation]
( P
[Equation]
( Q
[Equation]
( X )
[Equation]
, P G ( Z ) )
=
W 1 ( P X , P G ( Z ) ) .
Hence, W 1 ( P X , P G ( Z ) )
=
W 1 ( P X , P G ( Z ) ) .
Observe that sup f L ( (cid:101) G, (cid:101) Q, f )
=
W 1 ( P X , P (cid:101) G ( (cid:101) Q ( X )) )
+
W 1 ( P (cid:101) G ( (cid:101) Q ( X )) , P (cid:101) G ( Z ) ) .
By Theorem 2.1, we have inf G,Q L ( G, Q, (cid:101) f ) ≤ L ( G ∗ , Q ∗ , (cid:101) f ) = 0 when G and
Q have enough capacity.
Therefore, the duality gap is larger than W 1 ( P X , P (cid:101) G (
(cid:101) Q ( X )) )
+ W 1 ( P (cid:101) G
( (cid:101) Q (
(cid:101) G (
A PPENDIX C: P ROOF OF T HEOREM 3.
We ﬁrst consider the difference between population W 1 ( P X , P G ( Z ) ) and empirical (cid:99) W 1 ( P X , P G ( Z ) ) given n samples S
= { x 1 , . . .
, x n } .
Let f 1 and f 2 be their witness function respectively.
Using the dual form of 1-Wassertein distance, we have
[Equation]
Given another sample set S (cid:48) =
{ x 1 , . . .
, x (cid:48)
i , . .
, x n } , it is clear that
[Equation]
where the second inequality is obtained since f is 1-Lipschitz continuous function.
Applying McDiamond’s Inequality, with probability at least 1 − δ/ 2 for any δ ∈ (0 , 1) , we have
[Equation]
δ (cid:19) .
By the standard technique of symmetrization in Mohri et al.
(2018), we have
[Equation]
It has been proved in Mohri et al.
that with probability at least 1 − δ/ 2 for any δ ∈ (0 , 1) ,
R n ( F ) ≤ (cid:98) R n ( F ) + B (cid:115)n [Equation] (cid:18)δ (cid:19) .
Combining Equation (11), Equation (12) and Equation (13), we have
[Equation]
By Theorem 2.2, we have (cid:99) W 1 ( P X , P G ( Z ) ) ≤ (cid:99) W 1 ( P X , P G ( Z ) ) . Thus,
[Equation]
δ
[Equation]
A PPENDIX D: E XTENSION TO f -GAN S
This framework can be easily extended to other types of GANs.
We consider f -GAN (Nowozin et al., 2016) here. Let h : R → ( −∞ , ∞ ] be a convex function with h (1) = 0 . The f -GAN minimizes the following objective for the generator G :
[Equation] where h ∗ ( x ) = sup y { x · y − h ( y ) } is the convex conjugate of h . If h ( x ) = x log( x ) − ( x +1) log( x +1) − 2 log 2 , f -GAN recovers the original GAN (Goodfellow et al., 2014). If h ( x ) = 0 when x = 1 and h ( x ) = ∞ otherwise, we have h ∗ ( x ) = x .
With the property that F is 1-Lipschitz function class, f -GAN turns to be WGAN.
Assume that F is the 1-Lipschitz function class.
We extent the iWGAN framework to the inference f-GAN (ifGAN) framework and deﬁne the new objective function as follows:
[Equation]
Following this deﬁnition, we have
W 1 ,h ( P X , P G ( Z ) )
=
inf Q ∈Q (cid:8) W 1 ( P X , P G ( Q ( X )) )
+ GAN h
( P G ( Q ( X )) , P G ( Z ) ) (cid:9) .
We show GAN h ( P X , P G ( Z ) )
≤ W 1 ,h ( P X , P G ( Z ) ) .
This is because
GAN h ( P X , P G ( Z ) ) = sup f ∈F E X [ f ( X )] − E Z [ h ∗ ( f ( G ( Z ))] ≤ inf Q ∈Q (cid:26) sup f ∈F
E X
[ f ( X )] − E X
[ f ( G ( Q ( X )))]
+ sup f ∈F E X
[ f ( G ( Q ( X )))] − E Z
[ h ∗ ( f ( G ( Z ))] (cid:27) =
W 1 ,h ( P X , P G ( Z ) ) .
This indicates that the ifGAN objective (14) is an upper bound of the f-GAN objective.
A PPENDIX E: A RCHITECTURES
E1:
M IXTURE OF G UASSIANS
Encoder architecture: x
[Equation]
[Equation]
{+FC 1024 → RELU → FC 
 →+} RELU → FC 256 → RELU → FC
→ RELU → FC
Generator architecture: z [-∈ R 5 → FC 512 → RELU → FC 
-][Equation]
{+FC 
+} → RELU → FC 512 → RELU → FC {+
 → RELU → FC+}
Discriminator architecture: x [Equation] → FC 512 → RELU → FC
→ RELU → FC 512 → RELU → FC
E2: MNIST
Encoder architecture: x ∈ R 28 × 28 → Conv 128 → RELU → Conv 256 → RELU → Conv 512 → RELU →
FC
Generator architecture: z ∈ R 8 → FC 4 × 4 × 512 → RELU → ConvTrans 256 → RELU → ConvTrans 128 → RELU → ConvTrans
Discriminator architecture: [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
Generator architecture: [Equation] ConvTrans [Equation]
BN
[Equation] → ConvTrans [-
-][Equation]
BN [Equation] RELU [Equation]
ConvTrans [Equation] BN → RELU → ConvTrans
Discriminator architecture: [Equation]
[Table]
A PPENDIX F: M ORE E XPERIMENTAL R ESULTS ON M IXTURE OF G AUSSIAN
We investigate the mode collapse problem for the iWGAN.
If we draw two random samples in the latent space z 1 , z 2 ∼ N (0 , I 5 ) , the interpolation, G ( λz 1 + (1 − λ ) z 2 ) , 0 ≤ λ ≤ 1 , should fall around the mode to represent a reasonable sample.
In Figure 8, we select λ ∈ { 0 , 0 .
95 , 1 . 0 } , and do interpolations on two random samples.
We repeat this procedure several times on 3 datasets as demonstrated in Figure 8.
No matter where the interpolations start and end, the interpolations would fall around the modes other than the locations where true distribution has a low density.
There may still be some samples that appears in the middle of two modes.
This may be because the generator G is not able to approximate a step function well.
A PPENDIX G: M ORE E XPERIMENTAL R ESULTS ON MNIST
[Equation]
Figure
G.2: G ENERATED S AMPLES
Figure 10 shows the comparison of random generated samples between WGAN-GP and iWGAN.
Figure 11 shows examples of interpolations of two random generated samples.
G.3: R ECONSTRUCTION
Figure 12 shows, based on the samples from validation dataset, the distribution of reconstruction error.
Figure 13 shows examples of reconstructed samples.
Figure 14 shows the best and worst samples based on quality scores from the validation dataset.
[Figure] [Figure]
A PPENDIX H: M ORE E XPERIMENTAL R ESULTS ON C ELEB A H.1 L ATENT S PACE
Figure 15 shows the ﬁrst 8 dimensions of the latent space calculated by Q ( x ) on CelebA.
H.2 R ANDOM G ENERATED S AMPLE
Figure 16, Figure 17, Figure 18 and Figure 19 display the random generated samples from Wasserstein GAN gradient penalty(WGAN-GP), iWGAN, Wasserstein Autoencoder(WAE), and Adversarial Learning Inference (ALI), respectively.
Table 1 shows numerical comparison among these four methods based on inception scores (IS), Fréchet inception distances (FID), reconstruction errors (RE), and maximum mean discrepancy (MMD) between encodings and standard normal random variables.
The IS and FID scores are calculated based on pre-trained inception models.
However, as discussed in Barratt & Sharma (2018), inception score is not a reliable metric for the wellness of generated samples.
This is also consistent with our experiments.
Although WAE delivers the best inception scores among four methods, WAE also has the worst FID scores. The generated samples (Figure 18) show that WAE is not the best generative model compared with other three methods. The reconstruction error (RE) is deﬁned as [Equation]
ˆ X i is the reconstructed sample for X i .
RE is used to measure if the method has generated meaningful latent encodings.
Smaller reconstruction errors indicate a more meaningful latent space which can be decoded into the original samples.
The maximum mean discrepancy (MMD) is deﬁned as
[Equation]
where k is a positive-deﬁnite reproducing kernel, z i s are drawn from prior distribution P Z , and ˜ z
i = Q ( x i ) are the latent encodings of real samples.
MMD is used to measure the difference between distribution of latent encodings and standard normal random variables.
Smaller MMD indicates that the distribution of encodings is close to the standard normal distribution. From Table 1, in terms of generative models, iWGAN and ALI are better models, where WGAN-GP comes after, but WAE is suffering from generating clear pictures. In terms of RE and MMD, iWGAN and WAE are better choices, where ALI cannot always reconstruct the sample to itself (Figure 21). In general, Table 1 shows that iWGAN has successfully produce both meaningful encodings and reliable generator simultaneously.
[Table] Figure 20 shows the interpolation comparison between iWGAN, WAE and ALI.
H.3: R ECONSTRUCTION
Figure 22 shows the distribution of reconstruction error of CelebA. Figure 21 shows the comparison between real images and reconstructed images among three methods, iWGAN, WAE and ALI.
Figure 23 shows samples with high and low quality scores in CelebA validation sets.
[Figure]