Original: W ASSERSTEIN A UTO -E NCODERS
Revised: W ASSERSTEIN A UTO -E NCODERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose the Wasserstein Auto-Encoder (WAE)—a new algorithm for building a generative model of the data distribution.
Revised: We propose the Wasserstein Auto-Encoder (WAE)—a new algorithm for building a generative model of the data distribution.

Original: WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational
Revised: WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational

Original: Auto-Encoder (VAE) (Kingma & Welling, 2014).
Revised: Auto-Encoder (VAE) (Kingma & Welling, 2014).

Original: This regularizer encourages the encoded training distribution to match the prior.
Revised: This regularizer encourages the encoded training distribution to match the prior.

Original: We compare our algorithm with several other techniques and show that it is a generalization of adversarial auto-encoders (AAE)
Revised: We compare our algorithm with several other techniques and show that it is a generalization of adversarial auto-encoders (AAE)

Original: (Makhzani et al., 2016).
Revised: (Makhzani et al., 2016).

Original: Our experiments show that WAE shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality, as measured by the FID score.
Revised: Our experiments show that WAE shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality, as measured by the FID score.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ﬁeld of representation learning was initially driven by supervised approaches, with impressive results using large labelled datasets.
Revised: The ﬁeld of representation learning was initially driven by supervised approaches, with impressive results using large labelled datasets.

Original: Unsupervised generative modeling, in contrast, used to be a domain governed by probabilistic approaches focusing on low-dimensional data.
Revised: Unsupervised generative modeling, in contrast, used to be a domain governed by probabilistic approaches focusing on low-dimensional data.

Original: Recent years have seen a convergence of those two approaches.
Revised: Recent years have seen a convergence of those two approaches.

Original: In the new ﬁeld that formed at the intersection, variational auto-encoders (VAEs) (Kingma & Welling, 2014) constitute one well-established approach, theoretically elegant yet with the drawback that they tend to generate blurry samples when applied to natural images.
Revised: In the new ﬁeld that formed at the intersection, variational auto-encoders (VAEs) (Kingma & Welling, 2014) constitute one well-established approach, theoretically elegant yet with the drawback that they tend to generate blurry samples when applied to natural images.

Original: In contrast, generative adversarial networks (GANs) (Goodfellow et al., 2014) turned out to be more impressive in terms of the visual quality of images sampled from the model, but come without an encoder, have been reported harder to train, and suffer from the “mode collapse” problem where the resulting model is unable to capture all the variability in the true data distribution.
Revised: In contrast, generative adversarial networks (GANs) (Goodfellow et al., 2014) turned out to be more impressive in terms of the visual quality of images sampled from the model, but come without an encoder, have been reported harder to train, and suffer from the “mode collapse” problem where the resulting model is unable to capture all the variability in the true data distribution.

Original: There has been a ﬂurry of activity in assaying numerous conﬁgurations of GANs as well as combinations of VAEs and GANs.
Revised: There has been a ﬂurry of activity in assaying numerous conﬁgurations of GANs as well as combinations of VAEs and GANs.

Original: A unifying framework combining the best of GANs and
Revised: A unifying framework combining the best of GANs and

Original: VAEs in a principled way is yet to be discovered.
Revised: VAEs in a principled way is yet to be discovered.

Original: Following Arjovsky et al.
Revised: Following Arjovsky et al.

Original: (2017), we approach generative modeling from the optimal transport (OT) point of view.
Revised: (2017), we approach generative modeling from the optimal transport (OT) point of view.

Original: The OT cost (Villani, 2003) is a way to measure a distance between probability distributions and provides a much weaker topology than many others, including f -divergences associated with the original GAN algorithms (Nowozin et al., 2016).
Revised: The OT cost (Villani, 2003) is a way to measure a distance between probability distributions and provides a much weaker topology than many others, including f -divergences associated with the original GAN algorithms (Nowozin et al., 2016).

Original: This is particularly important in applications, where data is usually supported on low dimensional manifolds in the input space X .
Revised: This is particularly important in applications, where data is usually supported on low dimensional manifolds in the input space X .

Original: As a result, stronger notions of distances (such as f -divergences, which capture the density ratio between distributions) often max out, providing no useful gradients for training.
Revised: As a result, stronger notions of distances (such as f -divergences, which capture the density ratio between distributions) often max out, providing no useful gradients for training.

Original: In contrast, OT was claimed to have a nicer behaviour (Arjovsky et al., 2017; Gulrajani et al., 2017) although it requires, in its
Revised: In contrast, OT was claimed to have a nicer behaviour (Arjovsky et al., 2017; Gulrajani et al., 2017) although it requires, in its

Original: GAN-like implementation, the addition of a constraint or a regularization term into the objective.
Revised: GAN-like implementation, the addition of a constraint or a regularization term into the objective.

Original: In this work we aim at minimizing OT W c
Revised: In this work we aim at minimizing OT W c

Original: ( P X , P G ) between the true (but unknown) data distribution
Revised: ( P X , P G ) between the true (but unknown) data distribution

Original: P X and a latent variable model P G speciﬁed by the prior distribution P Z of latent codes Z ∈ Z and the generative model P G ( X | Z ) of the data points X ∈ X given Z .
Revised: P X and a latent variable model P G speciﬁed by the prior distribution P Z of latent codes Z ∈ Z and the generative model P G ( X | Z ) of the data points X ∈ X given Z .

Original: Our main contributions are listed below (cf.
Revised: Our main contributions are listed below (cf.

Original: also Figure 1):
Revised: also Figure 1):

Original: • A new family of regularized auto-encoders (Algorithms 1, 2 and Eq. 4), which we call Wasserstein Auto-Encoders (WAE), that minimize the optimal transport W c ( P X , P G ) for any cost function c .
Revised: • A new family of regularized auto-encoders (Algorithms 1, 2 and Eq. 4), which we call Wasserstein Auto-Encoders (WAE), that minimize the optimal transport W c ( P X , P G ) for any cost function c .

Original: Similarly to VAE, the objective of WAE is composed of two terms: the c -reconstruction cost and a regularizer
Revised: Similarly to VAE, the objective of WAE is composed of two terms: the c -reconstruction cost and a regularizer

Original: D Z ( P Z , Q Z ) penalizing a discrepancy between two distributions in Z : P Z and a distribution of encoded data points,
Revised: D Z ( P Z , Q Z ) penalizing a discrepancy between two distributions in Z : P Z and a distribution of encoded data points,

Original: i
Revised: i

Original: e. Q Z := E P X
Revised: e. Q Z := E P X

Original: [ Q ( Z | X )] .
Revised: [ Q ( Z | X )] .

Original: [Figure]
Revised: [Figure]

Original: When c is the squared cost and D Z is the GAN objective, WAE coincides with adversarial auto-encoders of Makhzani et al.
Revised: When c is the squared cost and D Z is the GAN objective, WAE coincides with adversarial auto-encoders of Makhzani et al.

Original: • Empirical evaluation of WAE on MNIST and CelebA datasets with squared cost c ( x, y )
Revised: • Empirical evaluation of WAE on MNIST and CelebA datasets with squared cost c ( x, y )

Original: = (cid:107) x
Revised: = (cid:107) x

Original: − y (cid:107) 22 .
Revised: − y (cid:107) 22 .

Original: Our experiments show that WAE keeps the good properties of VAEs (stable training, encoder-decoder architecture, and a nice latent manifold structure) while generating samples of better quality , approaching those of GANs.
Revised: Our experiments show that WAE keeps the good properties of VAEs (stable training, encoder-decoder architecture, and a nice latent manifold structure) while generating samples of better quality , approaching those of GANs.

Original: • We propose and examine two different regularizers
Revised: • We propose and examine two different regularizers

Original: D Z ( P Z , Q Z ) .
Revised: D Z ( P Z , Q Z ) .

Original: One is based on GANs and adversarial training in the latent space Z .
Revised: One is based on GANs and adversarial training in the latent space Z .

Original: The other uses the maximum mean discrepancy, which is known to perform well when matching high-dimensional standard normal distributions P Z
Revised: The other uses the maximum mean discrepancy, which is known to perform well when matching high-dimensional standard normal distributions P Z

Original: (Gretton et al., 2012).
Revised: (Gretton et al., 2012).

Original: Importantly, the second option leads to a fully adversary-free min-min optimization problem.
Revised: Importantly, the second option leads to a fully adversary-free min-min optimization problem.

Original: Finally, the theoretical considerations used to derive the WAE objective might be interesting in their own right.
Revised: Finally, the theoretical considerations used to derive the WAE objective might be interesting in their own right.

Original: We prove in particular (Theorem 1) that in the case of generative models, the primal form of W c ( P X , P G ) is equivalent to a problem involving the optimization of a probabilistic encoder Q ( Z | X ) .
Revised: We prove in particular (Theorem 1) that in the case of generative models, the primal form of W c ( P X , P G ) is equivalent to a problem involving the optimization of a probabilistic encoder Q ( Z | X ) .

Original: The paper is structured as follows.
Revised: The paper is structured as follows.

Original: In Section 2 we derive a novel auto-encoder formulation for
Revised: In Section 2 we derive a novel auto-encoder formulation for

Original: OT between P X and the latent variable model P G .
Revised: OT between P X and the latent variable model P G .

Original: Relaxing the resulting constrained optimization problem we arrive at an objective of Wasserstein auto-encoders.
Revised: Relaxing the resulting constrained optimization problem we arrive at an objective of Wasserstein auto-encoders.

Original: We propose two different regularizers, leading to WAE-GAN and WAE-MMD algorithms.
Revised: We propose two different regularizers, leading to WAE-GAN and WAE-MMD algorithms.

Original: Section 3 discusses the related work.
Revised: Section 3 discusses the related work.

Original: We present the experimental results in Section 4 and conclude by pointing out some promising directions for future work.
Revised: We present the experimental results in Section 4 and conclude by pointing out some promising directions for future work.

Original:  P ROPOSED METHOD
Revised:  P ROPOSED METHOD

Original: Our new method minimizes the optimal transport cost W c ( P X , P G ) based on the novel auto-encoder formulation derived in Theorem 1.
Revised: Our new method minimizes the optimal transport cost W c ( P X , P G ) based on the novel auto-encoder formulation derived in Theorem 1.

Original: In the resulting optimization problem the decoder tries to accurately reconstruct the encoded training examples as measured by the cost function c .
Revised: In the resulting optimization problem the decoder tries to accurately reconstruct the encoded training examples as measured by the cost function c .

Original: The encoder tries to simultaneously achieve two conﬂicting goals: it tries to match the encoded distribution of training examples Q Z := E P X [ Q ( Z | X )] to the prior P Z as measured by any speciﬁed divergence D Z ( Q Z , P Z ) , while making sure that the latent codes provided to the decoder are informative enough to reconstruct the encoded training examples.
Revised: The encoder tries to simultaneously achieve two conﬂicting goals: it tries to match the encoded distribution of training examples Q Z := E P X [ Q ( Z | X )] to the prior P Z as measured by any speciﬁed divergence D Z ( Q Z , P Z ) , while making sure that the latent codes provided to the decoder are informative enough to reconstruct the encoded training examples.

Original: This is schematically depicted on Fig. 1.
Revised: This is schematically depicted on Fig. 1.

Original: 2. P RELIMINARIES AND NOTATIONS
Revised: 2. P RELIMINARIES AND NOTATIONS

Original: We use calligraphic letters (i.e. X ) for sets, capital letters (i.e. X ) for random variables, and lower case letters (i.e. x ) for their values.
Revised: We use calligraphic letters (i.e. X ) for sets, capital letters (i.e. X ) for random variables, and lower case letters (i.e. x ) for their values.

Original: We denote probability distributions with capital letters (i.e. P ( X ) ) and corresponding densities with lower case letters (i.e. p ( x ) ).
Revised: We denote probability distributions with capital letters (i.e. P ( X ) ) and corresponding densities with lower case letters (i.e. p ( x ) ).

Original: In this work we will consider several measures of discrepancy between probability distributions P X and P G .
Revised: In this work we will consider several measures of discrepancy between probability distributions P X and P G .

Original: The class of f -divergences (Liese & Miescke, 2008) is deﬁned by D f ( P X (cid:107) P G ) :
Revised: The class of f -divergences (Liese & Miescke, 2008) is deﬁned by D f ( P X (cid:107) P G ) :

Original: = (cid:82) f (cid:0)
Revised: = (cid:82) f (cid:0)

Original: p X ( x ) p G ( x ) (cid:1) p G ( x ) dx , where f : (0 , ∞ ) → R is any convex function satisfying f (1) = 0 .
Revised: p X ( x ) p G ( x ) (cid:1) p G ( x ) dx , where f : (0 , ∞ ) → R is any convex function satisfying f (1) = 0 .

Original: Classical examples include the
Revised: Classical examples include the

Original: Kullback-Leibler D KL and Jensen-Shannon D JS divergences.
Revised: Kullback-Leibler D KL and Jensen-Shannon D JS divergences.

Original: 2. O PTIMAL TRANSPORT AND ITS DUAL FORMULATIONS
Revised: 2. O PTIMAL TRANSPORT AND ITS DUAL FORMULATIONS

Original: A rich class of divergences between probability distributions is induced by the optimal transport (OT) problem (Villani, 2003).
Revised: A rich class of divergences between probability distributions is induced by the optimal transport (OT) problem (Villani, 2003).

Original: Kantorovich’s formulation of the problem is given by
Revised: Kantorovich’s formulation of the problem is given by

Original: [Equation]
Revised: [Equation]

Original: where c ( x, y ): X × X → R + is any measurable cost function and P ( X ∼ P X , Y ∼ P G ) is a set of all joint distributions of ( X, Y ) with marginals P X and P G respectively.
Revised: where c ( x, y ): X × X → R + is any measurable cost function and P ( X ∼ P X , Y ∼ P G ) is a set of all joint distributions of ( X, Y ) with marginals P X and P G respectively.

Original: A particularly interesting case is when ( X , d ) is a metric space and c ( x, y )
Revised: A particularly interesting case is when ( X , d ) is a metric space and c ( x, y )

Original: =
Revised: =

Original: d p ( x, y ) for p ≥ 1 .
Revised: d p ( x, y ) for p ≥ 1 .

Original: In this case W p , the p -th root of W c , is called the p -Wasserstein distance .
Revised: In this case W p , the p -th root of W c , is called the p -Wasserstein distance .

Original: When c ( x, y )
Revised: When c ( x, y )

Original: =
Revised: =

Original: d ( x, y ) the following Kantorovich-Rubinstein duality holds 1 :
Revised: d ( x, y ) the following Kantorovich-Rubinstein duality holds 1 :

Original: [Equation]
Revised: [Equation]

Original: where F L is the class of all bounded 1-Lipschitz functions on ( X , d ) .
Revised: where F L is the class of all bounded 1-Lipschitz functions on ( X , d ) .

Original: 2. A PPLICATION TO GENERATIVE MODELS : W ASSERSTEIN AUTO - ENCODERS
Revised: 2. A PPLICATION TO GENERATIVE MODELS : W ASSERSTEIN AUTO - ENCODERS

Original: One way to look at modern generative models like VAEs and GANs is to postulate that they are trying to minimize certain discrepancy measures between the data distribution P X and the model P G .
Revised: One way to look at modern generative models like VAEs and GANs is to postulate that they are trying to minimize certain discrepancy measures between the data distribution P X and the model P G .

Original: Unfortunately, most of the standard divergences known in the literature, including those listed above, are hard or even impossible to compute, especially when P X is unknown and P G is parametrized by deep neural networks.
Revised: Unfortunately, most of the standard divergences known in the literature, including those listed above, are hard or even impossible to compute, especially when P X is unknown and P G is parametrized by deep neural networks.

Original: Previous research provides several tricks to address this issue.
Revised: Previous research provides several tricks to address this issue.

Original: In case of minimizing the KL-divergence D KL ( P X , P G ) , or equivalently maximizing the marginal log-likelihood E P X [log p G ( X )] , the famous variational lower bound provides a theoretically grounded framework successfully employed by VAEs (Kingma & Welling, 2014; Mescheder et al., 2017).
Revised: In case of minimizing the KL-divergence D KL ( P X , P G ) , or equivalently maximizing the marginal log-likelihood E P X [log p G ( X )] , the famous variational lower bound provides a theoretically grounded framework successfully employed by VAEs (Kingma & Welling, 2014; Mescheder et al., 2017).

Original: More generally, if the goal is to minimize the f -divergence D f
Revised: More generally, if the goal is to minimize the f -divergence D f

Original: ( P X , P G ) (with one example being D KL ), one can resort to its dual formulation and make use of f -GANs and the adversarial training (Nowozin et al., 2016).
Revised: ( P X , P G ) (with one example being D KL ), one can resort to its dual formulation and make use of f -GANs and the adversarial training (Nowozin et al., 2016).

Original: Finally, OT cost W c ( P X , P G ) is yet another option, which can be, thanks to the celebrated Kantorovich-Rubinstein duality (2), expressed as an adversarial objective as implemented by the Wasserstein-GAN (Arjovsky et al., 2017).
Revised: Finally, OT cost W c ( P X , P G ) is yet another option, which can be, thanks to the celebrated Kantorovich-Rubinstein duality (2), expressed as an adversarial objective as implemented by the Wasserstein-GAN (Arjovsky et al., 2017).

Original: We include an extended review of all these methods in Supplementary A.
Revised: We include an extended review of all these methods in Supplementary A.

Original: In this work we will focus on latent variable models P G deﬁned by a two-step procedure, where ﬁrst a code Z is sampled from a ﬁxed distribution P Z on a latent space Z
Revised: In this work we will focus on latent variable models P G deﬁned by a two-step procedure, where ﬁrst a code Z is sampled from a ﬁxed distribution P Z on a latent space Z

Original: and then Z is mapped to the image X ∈ X
Revised: and then Z is mapped to the image X ∈ X

Original: = R d with a (possibly random) transformation.
Revised: = R d with a (possibly random) transformation.

Original: This results in a density of the form p G ( x ) :
Revised: This results in a density of the form p G ( x ) :

Original: [Equation]
Revised: [Equation]

Original: assuming all involved densities are properly deﬁned.
Revised: assuming all involved densities are properly deﬁned.

Original: For simplicity we will focus on non-random decoders, i.e. generative models P G ( X | Z ) deterministically mapping Z to X =
Revised: For simplicity we will focus on non-random decoders, i.e. generative models P G ( X | Z ) deterministically mapping Z to X =

Original: G ( Z ) for a given map
Revised: G ( Z ) for a given map

Original: G : Z → X .
Revised: G : Z → X .

Original: In Supplementary B we present similar results for random decoders.
Revised: In Supplementary B we present similar results for random decoders.

Original: It turns out that under this model, the OT cost takes a simpler form as the transportation plan factors through the map G : instead of ﬁnding a coupling Γ in (1) between two random variables living in the X space, one distributed according to P X and the other one according to P G , it is sufﬁcient to ﬁnd a conditional distribution Q ( Z | X ) such that its Z marginal Q Z ( Z ) :
Revised: It turns out that under this model, the OT cost takes a simpler form as the transportation plan factors through the map G : instead of ﬁnding a coupling Γ in (1) between two random variables living in the X space, one distributed according to P X and the other one according to P G , it is sufﬁcient to ﬁnd a conditional distribution Q ( Z | X ) such that its Z marginal Q Z ( Z ) :

Original: =
Revised: =

Original: E X ∼ P X
Revised: E X ∼ P X

Original: [ Q ( Z | X )] is identical to the prior distribution P Z .
Revised: [ Q ( Z | X )] is identical to the prior distribution P Z .

Original: This is the content of our main theorem below.
Revised: This is the content of our main theorem below.

Original: Theorem 1 For any function G : Z → X we have
Revised: Theorem 1 For any function G : Z → X we have

Original: [Equation]
Revised: [Equation]

Original: where Q Z is the marginal distribution of Z when X ∼ P X and Z ∼ Q ( Z | X ) .
Revised: where Q Z is the marginal distribution of Z when X ∼ P X and Z ∼ Q ( Z | X ) .

Original: Proof The proof is reported in Supplementary B.
Revised: Proof The proof is reported in Supplementary B.

Original: This result allows us to optimize over random encoders Q ( Z | X ) instead of optimizing over all couplings between X and Y .
Revised: This result allows us to optimize over random encoders Q ( Z | X ) instead of optimizing over all couplings between X and Y .

Original: Of course, both problems are still constrained.
Revised: Of course, both problems are still constrained.

Original: In order to implement a numerical solution we relax the constraints on Q Z by adding a penalty to the objective.
Revised: In order to implement a numerical solution we relax the constraints on Q Z by adding a penalty to the objective.

Original: This ﬁnally leads us to the WAE objective:
Revised: This ﬁnally leads us to the WAE objective:

Original: [Equation]
Revised: [Equation]

Original: where Q is any nonparametric set of probabilistic encoders, D Z is an arbitrary divergence between
Revised: where Q is any nonparametric set of probabilistic encoders, D Z is an arbitrary divergence between

Original: Q Z and P Z , and λ > 0 is a hyperparameter.
Revised: Q Z and P Z , and λ > 0 is a hyperparameter.

Original: Similarly to VAE, we propose to use deep neural networks to parametrize both encoders Q and decoders G .
Revised: Similarly to VAE, we propose to use deep neural networks to parametrize both encoders Q and decoders G .

Original: Note that as opposed to VAEs, the WAE formulation allows for non-random encoders deterministically mapping inputs to their latent codes.
Revised: Note that as opposed to VAEs, the WAE formulation allows for non-random encoders deterministically mapping inputs to their latent codes.

Original: We propose two different penalties D Z ( Q Z , P Z ) :
Revised: We propose two different penalties D Z ( Q Z , P Z ) :

Original: GAN-based D Z .
Revised: GAN-based D Z .

Original: The ﬁrst option is to choose D Z ( Q Z , P Z ) =
Revised: The ﬁrst option is to choose D Z ( Q Z , P Z ) =

Original: D JS ( Q Z , P Z ) and use the adversarial training to estimate it.
Revised: D JS ( Q Z , P Z ) and use the adversarial training to estimate it.

Original: Speciﬁcally, we introduce an adversary (discriminator) in the latent space
Revised: Speciﬁcally, we introduce an adversary (discriminator) in the latent space

Original: Z trying to separate 2 “true” points sampled from P Z and “fake” ones sampled from Q Z (Goodfellow et al., 2014).
Revised: Z trying to separate 2 “true” points sampled from P Z and “fake” ones sampled from Q Z (Goodfellow et al., 2014).

Original: This results in the WAE-GAN described in Algorithm 1.
Revised: This results in the WAE-GAN described in Algorithm 1.

Original: Even though WAE-GAN falls back to the min-max problem, we move the adversary from the input (pixel) space X to the latent space Z .
Revised: Even though WAE-GAN falls back to the min-max problem, we move the adversary from the input (pixel) space X to the latent space Z .

Original: On top of that, P Z may have a nice shape with a single mode (for a Gaussian prior), in which case the task should be easier than matching an unknown, complex, and possibly multi-modal distributions as usually done in GANs.
Revised: On top of that, P Z may have a nice shape with a single mode (for a Gaussian prior), in which case the task should be easier than matching an unknown, complex, and possibly multi-modal distributions as usually done in GANs.

Original: This is also a reason for our second penalty:
Revised: This is also a reason for our second penalty:

Original: MMD-based D Z .
Revised: MMD-based D Z .

Original: For a positive-deﬁnite reproducing kernel k : Z × Z → R the following expression is called the maximum mean discrepancy (MMD):
Revised: For a positive-deﬁnite reproducing kernel k : Z × Z → R the following expression is called the maximum mean discrepancy (MMD):

Original: [Equation]
Revised: [Equation]

Original: where H k is the RKHS of real-valued functions mapping Z to R .
Revised: where H k is the RKHS of real-valued functions mapping Z to R .

Original: If k is characteristic then MMD k deﬁnes a metric and can be used as a divergence measure.
Revised: If k is characteristic then MMD k deﬁnes a metric and can be used as a divergence measure.

Original: We propose to use D Z ( P Z , Q Z ) =
Revised: We propose to use D Z ( P Z , Q Z ) =

Original: MMD k ( P Z , Q Z ) .
Revised: MMD k ( P Z , Q Z ) .

Original: Fortunately, MMD has an unbiased U-statistic estimator, which can be used in conjunction with stochastic gradient descent (SGD) methods.
Revised: Fortunately, MMD has an unbiased U-statistic estimator, which can be used in conjunction with stochastic gradient descent (SGD) methods.

Original: This results in the WAE-MMD described in Algorithm 2.
Revised: This results in the WAE-MMD described in Algorithm 2.

Original: It is well known that the maximum mean discrepancy performs well when matching high-dimensional standard normal distributions (Gretton et al., 2012)
Revised: It is well known that the maximum mean discrepancy performs well when matching high-dimensional standard normal distributions (Gretton et al., 2012)

Original: so we expect this penalty to work especially well working with the Gaussian prior P Z .
Revised: so we expect this penalty to work especially well working with the Gaussian prior P Z .

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Literature on auto-encoders Classical unregularized auto-encoders minimize only the reconstruction cost.
Revised: Literature on auto-encoders Classical unregularized auto-encoders minimize only the reconstruction cost.

Original: This results in different training points being encoded into non-overlapping zones chaotically scattered all across the Z space with “holes” in between where the decoder mapping P G ( X | Z ) has never been trained.
Revised: This results in different training points being encoded into non-overlapping zones chaotically scattered all across the Z space with “holes” in between where the decoder mapping P G ( X | Z ) has never been trained.

Original: Overall, the encoder Q ( Z | X ) trained in this way does not provide a useful representation and sampling from the latent space Z becomes hard (Bengio et al., 2013).
Revised: Overall, the encoder Q ( Z | X ) trained in this way does not provide a useful representation and sampling from the latent space Z becomes hard (Bengio et al., 2013).

Original: Variational auto-encoders (Kingma & Welling, 2014) minimize a variational bound on the KL-divergence D KL ( P X , P G ) which is composed of the reconstruction cost plus 
Revised: Variational auto-encoders (Kingma & Welling, 2014) minimize a variational bound on the KLdivergence D KL ( P X , P G ) which is composed of the reconstruction cost plus the regularizer

Original: E P X [ D KL ( Q ( Z | X ) , P Z )] which captures how distinct the image by the encoder of each training example is from the prior P Z , which is not guaranteeing that the overall encoded distribution E
Revised: E P X [ D KL ( Q ( Z | X ) , P Z )] . The regularizer captures how distinct the image by the encoder of each training example is from the prior P Z , which is not guaranteeing that the overall encoded distribution E

Original: P X [ Q ( Z | X )] matches P Z like WAE does.
Revised: P X [ Q ( Z | X )] matches P Z like WAE does.

Original: Also, VAEs require non-degenerate Gaussian encoders and random decoders for which log p G ( x | z ) can be computed and differentiated with respect to the parameters.
Revised: Also, VAEs require non-degenerate Gaussian encoders and random decoders for which log p G ( x | z ) can be computed and differentiated with respect to the parameters.

Original: Later Mescheder et al.
Revised: Later Mescheder et al.

Original: (2017) proposed a way to use VAE with non-Gaussian encoders.
Revised: (2017) proposed a way to use VAE with non-Gaussian encoders.

Original: WAE minimizes OT W c ( P X , P G ) and allows both probabilistic and deterministic encoder-decoder pairs of any kind.
Revised: WAE minimizes OT W c ( P X , P G ) and allows both probabilistic and deterministic encoder-decoder pairs of any kind.

Original: A LGORITHM 1 Wasserstein Auto-Encoder
Revised: A LGORITHM 1 Wasserstein Auto-Encoder

Original: with GAN-based penalty (WAE-GAN).
Revised: with GAN-based penalty (WAE-GAN).

Original: Require: Regularization coefﬁcient λ > 0 .
Revised: Require: Regularization coefﬁcient λ > 0 .

Original: Initialize the parameters of the encoder Q φ , decoder G θ , and latent discriminator D γ .
Revised: Initialize the parameters of the encoder Q φ , decoder G θ , and latent discriminator D γ .

Original: while ( φ, θ ) not converged do
Revised: while ( φ, θ ) not converged do

Original: Sample { x 1 , . . .
Revised: Sample { x 1 , . . .

Original: , x n } from the training set
Revised: , x n } from the training set

Original: Sample { z 1 , . . .
Revised: Sample { z 1 , . . .

Original: , z n } from the prior P Z
Revised: , z n } from the prior P Z

Original: Sample ˜ z
Revised: Sample ˜ z

Original: i from Q φ ( Z | x i ) for i = 1 , . . . , n
Revised: i from Q φ ( Z | x i ) for i = 1 , . . . , n

Original: Update D γ by ascending: λ n n (cid:88)
Revised: Update D γ by ascending: λ n n (cid:88)

Original: i = log D
Revised: i = log D

Original: γ
Revised: γ

Original: ( z i ) + log (cid:0) 1 − D γ (˜ z i ) (cid:1)
Revised: ( z i ) + log (cid:0) 1 − D γ (˜ z i ) (cid:1)

Original: Update Q φ and G θ by descending: n n (cid:88) i
Revised: Update Q φ and G θ by descending: n n (cid:88) i

Original: =
Revised: =

Original: c (cid:0) x i , G θ (˜ z i ) (cid:1) − λ · log D γ (˜ z i ) end while 
Revised: c (cid:0) x i , G θ (˜ z i ) (cid:1) − λ · log D γ (˜ z i ) end while A

Original: A LGORITHM 2 Wasserstein Auto-Encoder with MMD-based penalty (WAE-MMD).
Revised: with MMD-based penalty (WAE-MMD).

Original: Require: Regularization coefﬁcient λ > 0 , characteristic positive-deﬁnite kernel k .
Revised: Require: Regularization coefﬁcient λ > 0 , characteristic positive-deﬁnite kernel k .

Original: Initialize the parameters of the encoder Q φ , decoder G θ , and latent discriminator D γ .
Revised: Initialize the parameters of the encoder Q φ , decoder G θ , and latent discriminator D γ .

Original: while ( φ, θ ) not converged do
Revised: while ( φ, θ ) not converged do

Original: Sample { x 1 , . . .
Revised: Sample { x 1 , . . .

Original: , x n } from the training set
Revised: , x n } from the training set

Original: Sample { z 1 , . . .
Revised: Sample { z 1 , . . .

Original: , z n } from the prior P Z
Revised: , z n } from the prior P Z

Original: Sample ˜ z
Revised: Sample ˜ z

Original: i from Q φ ( Z | x i ) for i = 1 , . . . , n
Revised: i from Q φ ( Z | x i ) for i = 1 , . . . , n

Original: Update Q φ and G θ by descending: n n (cid:88) i
Revised: Update Q φ and G θ by descending: n n (cid:88) i

Original: =
Revised: =

Original: c (cid:0) x i , G θ (˜ z i ) (cid:1)+ λ n ( n − 1) (cid:88) (cid:96) (cid:54) =
Revised: c (cid:0) [Equation] [Equation] (cid:1)

Revised: [Equation]

Revised: λ

Revised: [Equation]

Original:  j k ( z (cid:96) , z j ) + k
Revised: [Equation] 1) (cid:88) (cid:96) (cid:54) = j k ( z (cid:96) , z j ) + λ [Equation]

Revised: [Equation]

Original: (˜ z (cid:96) , ˜ z j) − 2 k ( z (cid:96) , ˜ z
Revised: 1) (cid:88) (cid:96) (cid:54) = j k (˜ z (cid:96) [Equation] ˜ z

Original: j ) end while
Revised: j ) [Equation]

Revised: 2 λ n 2 (cid:88) (cid:96),j k ( z (cid:96) , ˜ z j ) end while

Original: When used with c ( x, y ) = (cid:107) x − y (cid:107) 

Revised: When used with c ( x, y ) = (cid:107) x − y (cid:107)

Original: WAE-GAN is equivalent to adversarial auto-encoders (AAE) proposed by Makhzani et al.
Revised: WAE-GAN is equivalent to adversarial auto-encoders (AAE) proposed by Makhzani et al.

Original: Our theory thus suggests that AAEs minimize the 2-Wasserstein distance between P X and P G .
Revised: Our theory thus suggests that AAEs minimize the 2-Wasserstein distance between P X and P G .

Original: This provides the ﬁrst theoretical justiﬁcation for AAEs known to the authors.
Revised: This provides the ﬁrst theoretical justiﬁcation for AAEs known to the authors.

Original: WAE generalizes AAE in two ways: ﬁrst, it can use any cost function c in the input space X ; second, it can use any discrepancy measure D Z in the latent space Z (for instance MMD), not necessarily the adversarial one of WAE-GAN.
Revised: WAE generalizes AAE in two ways: ﬁrst, it can use any cost function c in the input space X ; second, it can use any discrepancy measure D Z in the latent space Z (for instance MMD), not necessarily the adversarial one of WAE-GAN.

Original: Literature on OT Genevay et al.
Revised: Literature on OT Genevay et al.

Original: (2016) address computing the OT cost in large scale using SGD and sampling.
Revised: (2016) address computing the OT cost in large scale using SGD and sampling.

Original: They approach this task either through the dual formulation, or via a regularized version of the primal.
Revised: They approach this task either through the dual formulation, or via a regularized version of the primal.

Original: They do not discuss any implications for generative modeling.
Revised: They do not discuss any implications for generative modeling.

Original: Our approach is based on the primal form of OT, we arrive at regularizers which are very different, and our main focus is on generative modeling.
Revised: Our approach is based on the primal form of OT, we arrive at regularizers which are very different, and our main focus is on generative modeling.

Original: The WGAN (Arjovsky et al., 2017) minimizes the 1-Wasserstein distance W 1 ( P X , P G ) for generative modeling.
Revised: The WGAN (Arjovsky et al., 2017) minimizes the 1-Wasserstein distance W 1 ( P X , P G ) for generative modeling.

Original: The authors approach this task from the dual form.
Revised: The authors approach this task from the dual form.

Original: Their algorithm comes without an encoder and can not be readily applied to any other cost W c , because the neat form of the
Revised: Their algorithm comes without an encoder and can not be readily applied to any other cost W c , because the neat form of the

Original: Kantorovich-Rubinstein duality (2) holds only for W 1 .
Revised: Kantorovich-Rubinstein duality (2) holds only for W 1 .

Original: WAE approaches the same problem from the primal form, can be applied for any cost function c , and comes naturally with an encoder.
Revised: WAE approaches the same problem from the primal form, can be applied for any cost function c , and comes naturally with an encoder.

Original: In order to compute the values (1) or (2) of OT we need to handle non-trivial constraints, either on the coupling distribution Γ or on the function f being considered.
Revised: In order to compute the values (1) or (2) of OT we need to handle non-trivial constraints, either on the coupling distribution Γ or on the function f being considered.

Original: Various approaches have been proposed in the literature to circumvent this difﬁculty.
Revised: Various approaches have been proposed in the literature to circumvent this difﬁculty.

Original: For W 1 Arjovsky et al.
Revised: For W 1 Arjovsky et al.

Original: (2017) tried to implement the constraint in the dual formulation (2) by clipping the weights of the neural network f .
Revised: (2017) tried to implement the constraint in the dual formulation (2) by clipping the weights of the neural network f .

Original: Later Gulrajani et al.
Revised: Later Gulrajani et al.

Original: (2017) proposed to relax the same constraint by penalizing the objective of (2) with a term λ · E ( (cid:107)∇ f ( X ) (cid:107) − 1) 2 which should not be greater than 1 if f ∈ F L .
Revised: (2017) proposed to relax the same constraint by penalizing the objective of (2) with a term λ · E ( (cid:107)∇ f ( X ) (cid:107) − 1) 2 which should not be greater than 1 if f ∈ F L .

Original: In a more general OT setting of W c Cuturi (2013) proposed to penalize the objective of (1) with the KLdivergence λ · D KL (Γ , P ⊗ Q ) between the coupling distribution and the product of marginals.
Revised: In a more general OT setting of W c Cuturi (2013) proposed to penalize the objective of (1) with the KLdivergence λ · D KL (Γ , P ⊗ Q ) between the coupling distribution and the product of marginals.

Original: Genevay et al.
Revised: Genevay et al.

Original: (2016) showed that this entropic regularization drops the constraints on functions in the dual formulation as opposed to (2).
Revised: (2016) showed that this entropic regularization drops the constraints on functions in the dual formulation as opposed to (2).

Original: Finally, in the context of unbalanced optimal transport it has been proposed to relax the constraint in (1) by regularizing the objective with λ · (cid:0) D f
Revised: Finally, in the context of unbalanced optimal transport it has been proposed to relax the constraint in (1) by regularizing the objective with λ · (cid:0) D f

Original: (Γ X , P )
Revised: (Γ X , P )

Original: +
Revised: +

Original: D f
Revised: D f

Original: (Γ Y , Q ) (cid:1) (Chizat et al., 2015;
Revised: (Γ Y , Q ) (cid:1) (Chizat et al., 2015;

Original: Liero et al., 2015), where Γ X and Γ Y are marginals of Γ .
Revised: Liero et al., 2015), where Γ X and Γ Y are marginals of Γ .

Original: In this paper we propose to relax OT in a way similar to the unbalanced optimal transport, i.e. by adding additional divergences to the objective.
Revised: In this paper we propose to relax OT in a way similar to the unbalanced optimal transport, i.e. by adding additional divergences to the objective.

Original: However, we show that in the particular context of generative modeling, only one extra divergence is necessary.
Revised: However, we show that in the particular context of generative modeling, only one extra divergence is necessary.

Original: T e s t i n t e r po l a ti on s
Revised: T e s t i n t e r po l a ti on s

Original: T e s t r ec on s t
Revised: T e s t r ec on s t

Original: r u c ti on s
Revised: r u c ti on s

Original: R a ndo m s a m p l e s
Revised: R a ndo m s a m p l e s

Original: [Figure]
Revised: [Figure]

Original: Literature on GANs Many of the GAN variations (including f -GAN and WGAN) come without an encoder.
Revised: Literature on GANs Many of the GAN variations (including f -GAN and WGAN) come without an encoder.

Original: Often it may be desirable to reconstruct the latent codes and use the learned manifold, in which cases these models are not applicable.
Revised: Often it may be desirable to reconstruct the latent codes and use the learned manifold, in which cases these models are not applicable.

Original: There have been many other approaches trying to blend the adversarial training of GANs with autoencoder architectures (Zhao et al., 2017; Dumoulin et al., 2017; Ulyanov et al., 2017; Berthelot et al., 2017).
Revised: There have been many other approaches trying to blend the adversarial training of GANs with autoencoder architectures (Zhao et al., 2017; Dumoulin et al., 2017; Ulyanov et al., 2017; Berthelot et al., 2017).

Original: The approach proposed by Ulyanov et al.
Revised: The approach proposed by Ulyanov et al.

Original: (2017) is perhaps the most relevant to our work.
Revised: (2017) is perhaps the most relevant to our work.

Original: The authors use the discrepancy between Q Z and the distribution E Z (cid:48) ∼ P Z [ Q (cid:0) Z | G ( Z (cid:48) )
Revised: The authors use the discrepancy between Q Z and the distribution E Z (cid:48) ∼ P Z [ Q (cid:0) Z | G ( Z (cid:48) )

Original: (cid:1) ] of auto-encoded noise vectors as the objective for the max-min game between the encoder and decoder respectively.
Revised: (cid:1) ] of auto-encoded noise vectors as the objective for the max-min game between the encoder and decoder respectively.

Original: While the authors showed that the saddle points correspond to P X
Revised: While the authors showed that the saddle points correspond to P X

Original: = P G , they admit that encoders and decoders trained in this way have no incentive to be reciprocal.
Revised: = P G , they admit that encoders and decoders trained in this way have no incentive to be reciprocal.

Original: As a workaround they propose to include an additional reconstruction term to the objective.
Revised: As a workaround they propose to include an additional reconstruction term to the objective.

Original: WAE does not necessarily lead to a min-max game, uses a different penalty, and has a clear theoretical foundation.
Revised: WAE does not necessarily lead to a min-max game, uses a different penalty, and has a clear theoretical foundation.

Original: Several works used reproducing kernels in context of GANs.
Revised: Several works used reproducing kernels in context of GANs.

Original: Li et al.
Revised: Li et al.

Original: ; Dziugaite et al.
Revised: ; Dziugaite et al.

Original: (2015) use MMD with a ﬁxed kernel k to match P X and P G directly in the input space X .
Revised: (2015) use MMD with a ﬁxed kernel k to match P X and P G directly in the input space X .

Original: These methods have been criticised to require larger mini-batches during training: estimating MMD k
Revised: These methods have been criticised to require larger mini-batches during training: estimating MMD k

Original: ( P X , P G ) requires number of samples roughly proportional to the dimensionality of the input space X
Revised: ( P X , P G ) requires number of samples roughly proportional to the dimensionality of the input space X

Original: (Reddi et al., 2015) which is typically larger than 10 3 .
Revised: (Reddi et al., 2015) which is typically larger than 10 3 .

Original: Li et al.
Revised: Li et al.

Original: (2017) take a similar approach but further train k adversarially so as to arrive at a meaningful loss function.
Revised: (2017) take a similar approach but further train k adversarially so as to arrive at a meaningful loss function.

Original: WAE-MMD uses MMD to match
Revised: WAE-MMD uses MMD to match

Original: Q Z to the prior P Z in the latent space Z .
Revised: Q Z to the prior P Z in the latent space Z .

Original: Typically Z has no more than 100 dimensions and P Z is
Revised: Typically Z has no more than 100 dimensions and P Z is

Original: Gaussian, which allows us to use regular mini-batch sizes to accurately estimate MMD.
Revised: Gaussian, which allows us to use regular mini-batch sizes to accurately estimate MMD.

Original: T e s t i n t e r po l a ti on s
Revised: T e s t i n t e r po l a ti on s

Original: T e s t r ec on s t
Revised: T e s t r ec on s t

Original: r u c ti on s
Revised: r u c ti on s

Original: R a ndo m s a m p l e s
Revised: R a ndo m s a m p l e s

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we empirically evaluate the proposed WAE model.
Revised: In this section we empirically evaluate the proposed WAE model.

Original: We would like to test if WAE can simultaneously achieve (i) accurate reconstructions of data points, (ii) reasonable geometry of the latent manifold, and (iii) random samples of good (visual) quality.
Revised: We would like to test if WAE can simultaneously achieve (i) accurate reconstructions of data points, (ii) reasonable geometry of the latent manifold, and (iii) random samples of good (visual) quality.

Original: Importantly, the model should generalize well: requirements (i) and (ii) should be met on both training and test data.
Revised: Importantly, the model should generalize well: requirements (i) and (ii) should be met on both training and test data.

Original: We trained
Revised: We trained

Original: WAE-GAN and WAE-MMD (Algorithms 1 and 2) on two real-world datasets: MNIST (LeCun et al., 1998) consisting of 70k images and CelebA (Liu et al., 2015) containing roughly 203k images.
Revised: WAE-GAN and WAE-MMD (Algorithms 1 and 2) on two real-world datasets: MNIST (LeCun et al., 1998) consisting of 70k images and CelebA (Liu et al., 2015) containing roughly 203k images.

Original: Experimental setup In all reported experiments we used Euclidian latent spaces Z = R d z for various d z depending on the complexity of the dataset, isotropic Gaussian prior distributions
Revised: Experimental setup In all reported experiments we used Euclidian latent spaces Z = R d z for various d z depending on the complexity of the dataset, isotropic Gaussian prior distributions

Original: P Z ( Z ) =
Revised: P Z ( Z ) =

Original: N ( Z ; 0 , σ 2 z · I d ) over Z , and a squared cost function c ( x, y ) =
Revised: N ( Z ; 0 , σ 2 z · I d ) over Z , and a squared cost function c ( x, y ) =

Original: (cid:107) x − y (cid:107) 2 2 for data points x, y ∈ X
Revised: (cid:107) x − y (cid:107) 2 2 for data points x, y ∈ X

Original: = R d x .
Revised: = R d x .

Original: We used deterministic encoder-decoder pairs, Adam (Kingma & Lei, 2014) with β 1 = 0 .
Revised: We used deterministic encoder-decoder pairs, Adam (Kingma & Lei, 2014) with β 1 = 0 .

Original: 5 , β 2 = 0 .
Revised: 5 , β 2 = 0 .

Original: 999 , and convolutional deep neural network architectures for encoder mapping
Revised: 999 , and convolutional deep neural network architectures for encoder mapping

Original: Q φ : X → Z and decoder mapping G θ : Z → X similar to the DCGAN ones reported by Radford et al.
Revised: Q φ : X → Z and decoder mapping G θ : Z → X similar to the DCGAN ones reported by Radford et al.

Original: (2016) with batch normalization (Ioffe & Szegedy, 2015).
Revised: (2016) with batch normalization (Ioffe & Szegedy, 2015).

Original: We tried various values of λ and noticed that λ
Revised: We tried various values of λ and noticed that λ

Original: = 10 seems to work good across all datasets we considered.
Revised: = 10 seems to work good across all datasets we considered.

Original: All reported experiments use this value.
Revised: All reported experiments use this value.

Original: Since we are using deterministic encoders, choosing d z larger than intrinsic dimensionality of the dataset would force the encoded distribution Q Z to live on a manifold in Z .
Revised: Since we are using deterministic encoders, choosing d z larger than intrinsic dimensionality of the dataset would force the encoded distribution Q Z to live on a manifold in Z .

Original: This would make matching Q Z to P Z impossible if P Z is Gaussian and may lead to numerical instabilities.
Revised: This would make matching Q Z to P Z impossible if P Z is Gaussian and may lead to numerical instabilities.

Original: We use d z = 8 for MNIST and d z = 64 for CelebA which seems to work reasonably well.
Revised: We use d z = 8 for MNIST and d z = 64 for CelebA which seems to work reasonably well.

Original: We also report results of VAEs.
Revised: We also report results of VAEs.

Original: VAEs used the same latent spaces as discussed above and standard
Revised: VAEs used the same latent spaces as discussed above and standard

Original: Gaussian priors P Z = N ( 0 , I d ) .
Revised: Gaussian priors P Z = N ( 0 , I d ) .

Original: We used Gaussian encoders Q ( Z | X ) =
Revised: We used Gaussian encoders Q ( Z | X ) =

Original: N (cid:0) Z ; Q φ ( X ) , Σ( X ) (cid:1) with mean Q φ and diagonal covariance Σ .
Revised: N (cid:0) Z ; Q φ ( X ) , Σ( X ) (cid:1) with mean Q φ and diagonal covariance Σ .

Original: For MNIST we used Bernoulli decoders parametrized by
Revised: For MNIST we used Bernoulli decoders parametrized by

Original: G θ and for CelebA the Gaussian decoders P G
Revised: G θ and for CelebA the Gaussian decoders P G

Original: ( X | Z ) =
Revised: ( X | Z ) =

Original: N (cid:0) Z ; G θ ( X ) , σ 2 G · I d (cid:1) with mean G θ ( Z ) .
Revised: N (cid:0) Z ; G θ ( X ) , σ 2 G · I d (cid:1) with mean G θ ( Z ) .

Original: Functions Q φ , Σ , and G θ were parametrized by deep nets of the same architectures as used in WAE.
Revised: Functions Q φ , Σ , and G θ were parametrized by deep nets of the same architectures as used in WAE.

Original: WAE-GAN and WAE-MMD speciﬁcs In WAE-GAN we used discriminator D composed of several fully connected layers with ReLu.
Revised: WAE-GAN and WAE-MMD speciﬁcs In WAE-GAN we used discriminator D composed of several fully connected layers with ReLu.

Original: We tried WAE-MMD with the RBF kernel but observed that it fails to penalize the outliers of Q Z because of the quick tail decay.
Revised: We tried WAE-MMD with the RBF kernel but observed that it fails to penalize the outliers of Q Z because of the quick tail decay.

Original: If the codes ˜ z
Revised: If the codes ˜ z

Original: = Q φ ( x ) for some of the training points x ∈ X end up far away from the support of P Z (which may happen in the early stages of training)
Revised: = Q φ ( x ) for some of the training points x ∈ X end up far away from the support of P Z (which may happen in the early stages of training)

Original: the corresponding terms in the U-statistic k ( z, ˜ z ) = e −(cid:107) ˜
Revised: the corresponding terms in the U-statistic k ( z, ˜ z ) = e −(cid:107) ˜

Original: z − z (cid:107) 

Revised: z − z (cid:107)

Original: /σ 2 k will quickly approach zero and provide no gradient for those outliers.
Revised: /σ 2 k will quickly approach zero and provide no gradient for those outliers.

Original: This could be avoided by choosing the kernel bandwidth σ 2 k in a data-dependent manner, however in this case per-minibatch U-statistic would not provide an unbiased estimate for the gradient.
Revised: This could be avoided by choosing the kernel bandwidth σ 2 k in a data-dependent manner, however in this case per-minibatch U-statistic would not provide an unbiased estimate for the gradient.

Original: Instead, we used the inverse multiquadratics kernel k ( x, y )
Revised: Instead, we used the inverse multiquadratics kernel k ( x, y )

Original: =
Revised: =

Original: C/ ( C + (cid:107) x
Revised: C/ ( C + (cid:107) x

Original: − y (cid:107) 22 ) which is also characteristic and has much heavier tails.
Revised: − y (cid:107) 22 ) which is also characteristic and has much heavier tails.

Original: In all experiments we used C = 

Revised: In all experiments we used C =

Original: d z σ 2 z , which is the expected squared distance between two multivariate Gaussian vectors drawn from P Z .
Revised: d z σ 2 z , which is the expected squared distance between two multivariate Gaussian vectors drawn from P Z .

Original: This signiﬁcantly improved the performance compared to the RBF kernel (even the one with σ 2 k = 

Revised: This signiﬁcantly improved the performance compared to the RBF kernel (even the one with σ 2 k =

Original: d z σ 2 z ).
Revised: d z σ 2 z ).

Original: Trained models are presented in Figures 2 and 3.
Revised: Trained models are presented in Figures 2 and 3.

Original: Further details are presented in Supplementary E.
Revised: Further details are presented in Supplementary E.

Original: Random samples are generated by sampling P Z and decoding the resulting noise vectors z into G θ ( z ) .
Revised: Random samples are generated by sampling P Z and decoding the resulting noise vectors z into G θ ( z ) .

Original: As expected, in our experiments we observed that for both WAE-GAN and
Revised: As expected, in our experiments we observed that for both WAE-GAN and

Original: WAE-MMD the quality of samples strongly depends on how accurately Q Z matches P Z .
Revised: WAE-MMD the quality of samples strongly depends on how accurately Q Z matches P Z .

Original: To see this, notice that while training the decoder function G θ is presented only with encoded versions Q φ ( X ) of the data points X ∼ P X .
Revised: To see this, notice that while training the decoder function G θ is presented only with encoded versions Q φ ( X ) of the data points X ∼ P X .

Original: Indeed, the decoder is trained on samples from Q Z and thus there is no reason to expect good results when feeding it with samples from P Z .
Revised: Indeed, the decoder is trained on samples from Q Z and thus there is no reason to expect good results when feeding it with samples from P Z .

Original: In our experiments we noticed that even slight differences between Q Z and
Revised: In our experiments we noticed that even slight differences between Q Z and

Original: P Z may affect the quality of samples.
Revised: P Z may affect the quality of samples.

Original: In some cases WAE-GAN seems to lead to a better matching and generates better samples than WAE-MMD.
Revised: In some cases WAE-GAN seems to lead to a better matching and generates better samples than WAE-MMD.

Original: However, due to adversarial training
Revised: However, due to adversarial training

Original: WAE-GAN is highly unstable, while WAE-MMD has a very stable training much like VAE.
Revised: WAE-GAN is highly unstable, while WAE-MMD has a very stable training much like VAE.

Original: [Table]
Revised: [Table]

Original: In order to quantitatively assess the quality of the generated images, we use the Fr´echet Inception Distance introduced by Heusel et al.
Revised: In order to quantitatively assess the quality of the generated images, we use the Fr´echet Inception Distance introduced by Heusel et al.

Original: and report the results on
Revised: and report the results on

Original: CelebA in Table 1.
Revised: CelebA in Table 1.

Original: These results conﬁrm that the sampled images from WAE are of better quality than from VAE, and WAE-GAN gets a slightly better score than WAEMMD, which correlates with visual inspection of the images.
Revised: These results conﬁrm that the sampled images from WAE are of better quality than from VAE, and WAE-GAN gets a slightly better score than WAEMMD, which correlates with visual inspection of the images.

Original: Test reconstructions and interpolations.
Revised: Test reconstructions and interpolations.

Original: We take random points x from the held out test set and report their auto-encoded versions G θ ( Q φ ( x )) .
Revised: We take random points x from the held out test set and report their auto-encoded versions G θ ( Q φ ( x )) .

Original: Next, pairs ( x, y ) of different data points are sampled randomly from the held out test set and encoded: z x = Q φ ( x ) , z y
Revised: Next, pairs ( x, y ) of different data points are sampled randomly from the held out test set and encoded: z x = Q φ ( x ) , z y

Original: =
Revised: =

Original: Q φ ( y ) .
Revised: Q φ ( y ) .

Original: We linearly interpolate between z x and z y with equally-sized steps in the latent space and show decoded images.
Revised: We linearly interpolate between z x and z y with equally-sized steps in the latent space and show decoded images.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Using the optimal transport cost, we have derived Wasserstein auto-encoders—a new family of algorithms for building generative models.
Revised: Using the optimal transport cost, we have derived Wasserstein auto-encoders—a new family of algorithms for building generative models.

Original: We discussed their relations to other probabilistic modeling techniques.
Revised: We discussed their relations to other probabilistic modeling techniques.

Original: We conducted experiments using two particular implementations of the proposed method, showing that in comparison to VAEs, the images sampled from the trained WAE models are of better quality, without compromising the stability of training and the quality of reconstruction.
Revised: We conducted experiments using two particular implementations of the proposed method, showing that in comparison to VAEs, the images sampled from the trained WAE models are of better quality, without compromising the stability of training and the quality of reconstruction.

Original: Future work will include further exploration of the criteria for matching the encoded distribution Q Z to the prior distribution P Z , assaying the possibility of adversarially training the cost function c in the input space X , and a theoretical analysis of the dual formulations for WAE-GAN and WAE-MMD.
Revised: Future work will include further exploration of the criteria for matching the encoded distribution Q Z to the prior distribution P Z , assaying the possibility of adversarially training the cost function c in the input space X , and a theoretical analysis of the dual formulations for WAE-GAN and WAE-MMD.

Original: A I MPLICIT GENERATIVE MODELS : A SHORT TOUR OF GAN S AND VAE S
Revised: A I MPLICIT GENERATIVE MODELS : A SHORT TOUR OF GAN S AND VAE S

Original: Even though GANs and VAEs are quite different—both in terms of the conceptual frameworks and empirical performance—they share important features: (a) both can be trained by sampling from the model P G without knowing an analytical form of its density and (b) both can be scaled up with
Revised: Even though GANs and VAEs are quite different—both in terms of the conceptual frameworks and empirical performance—they share important features: (a) both can be trained by sampling from the model P G without knowing an analytical form of its density and (b) both can be scaled up with

Original: SGD.
Revised: SGD.

Original: As a result, it becomes possible to use highly ﬂexible implicit models P G deﬁned by a twostep procedure, where ﬁrst a code Z is sampled from a ﬁxed distribution P Z on a latent space Z
Revised: As a result, it becomes possible to use highly ﬂexible implicit models P G deﬁned by a twostep procedure, where ﬁrst a code Z is sampled from a ﬁxed distribution P Z on a latent space Z

Original: and then Z is mapped to the image G ( Z ) ∈
Revised: and then Z is mapped to the image G ( Z ) ∈

Original: X
Revised: X

Original: = R d with a (possibly random) transformation
Revised: = R d with a (possibly random) transformation

Original: G : Z → X .
Revised: G : Z → X .

Original: This results in latent variable models P G of the form (3).
Revised: This results in latent variable models P G of the form (3).

Original: These models are indeed easy to sample and, provided G can be differentiated analytically with respect to its parameters, P G can be trained with SGD.
Revised: These models are indeed easy to sample and, provided G can be differentiated analytically with respect to its parameters, P G can be trained with SGD.

Original: The ﬁeld is growing rapidly and numerous variations of VAEs and GANs are available in the literature.
Revised: The ﬁeld is growing rapidly and numerous variations of VAEs and GANs are available in the literature.

Original: Next we introduce and compare several of them.
Revised: Next we introduce and compare several of them.

Original: The original generative adversarial network (GAN) Goodfellow et al.
Revised: The original generative adversarial network (GAN) Goodfellow et al.

Original: (2014) approach minimizes
Revised: (2014) approach minimizes

Original: [Equation]
Revised: [Equation]

Original:  with respect to a deterministic decoder G : Z → X , where T is any non-parametric class of choice.
Revised: (5) with respect to a deterministic decoder G : Z → X , where T is any non-parametric class of choice.

Original: It is known that D GAN ( P X , P G ) ≤ 2 · D JS ( P X , P G ) − log(4) and the inequality turns into identity in the nonparametric limit , that is when the class T becomes rich enough to represent all functions mapping X to (0 , 1) .
Revised: It is known that D GAN ( P X , P G ) ≤ 2 · D JS ( P X , P G ) − log(4) and the inequality turns into identity in the nonparametric limit , that is when the class T becomes rich enough to represent all functions mapping X to (0 , 1) .

Original: Hence, GANs are minimizing a lower bound on the JS-divergence.
Revised: Hence, GANs are minimizing a lower bound on the JS-divergence.

Original: However, GANs are not only linked to the JS-divergence: the f -GAN approach Nowozin et al.
Revised: However, GANs are not only linked to the JS-divergence: the f -GAN approach Nowozin et al.

Original: (2016) showed that a slight modiﬁcation D f , GAN of the objective (5) allows to lower bound any desired f -divergence in a similar way.
Revised: (2016) showed that a slight modiﬁcation D f , GAN of the objective (5) allows to lower bound any desired f -divergence in a similar way.

Original: In practice, both decoder G and discriminator T are trained in alternating SGD steps.
Revised: In practice, both decoder G and discriminator T are trained in alternating SGD steps.

Original: Stopping criteria as well as adequate evaluation of the trained GAN models remain open questions.
Revised: Stopping criteria as well as adequate evaluation of the trained GAN models remain open questions.

Original: Recently, the authors of Arjovsky et al.
Revised: Recently, the authors of Arjovsky et al.

Original: (2017) argued that the 1-Wasserstein distance W 1 , which is known to induce a much weaker topology than D JS , may be better suited for generative modeling.
Revised: (2017) argued that the 1-Wasserstein distance W 1 , which is known to induce a much weaker topology than D JS , may be better suited for generative modeling.

Original: When P X and P G are supported on largely disjoint low-dimensional manifolds (which may be the case in applications), D KL , D JS , and other strong distances between P X and P G max out and no longer provide useful gradients for P G .
Revised: When P X and P G are supported on largely disjoint low-dimensional manifolds (which may be the case in applications), D KL , D JS , and other strong distances between P X and P G max out and no longer provide useful gradients for P G .

Original: This “vanishing gradient” problem necessitates complicated scheduling between the G / T updates.
Revised: This “vanishing gradient” problem necessitates complicated scheduling between the G / T updates.

Original: In contrast, W 1 is still sensible in these cases and provides stable gradients.
Revised: In contrast, W 1 is still sensible in these cases and provides stable gradients.

Original: The Wasserstein GAN (WGAN) minimizes
Revised: The Wasserstein GAN (WGAN) minimizes

Original: [Equation]
Revised: [Equation]

Original: where W is any subset of 1-Lipschitz functions on X .
Revised: where W is any subset of 1-Lipschitz functions on X .

Original: It follows from (2)
Revised: It follows from (2)

Original: that D WGAN ( P X , P G ) ≤
Revised: that D WGAN ( P X , P G ) ≤

Original: W 1 ( P X , P G ) and thus WGAN is minimizing a lower bound on the 1-Wasserstein distance.
Revised: W 1 ( P X , P G ) and thus WGAN is minimizing a lower bound on the 1-Wasserstein distance.

Original: Variational auto-encoders (VAE) Kingma & Welling (2014) utilize models P G of the form (3) and minimize
Revised: Variational auto-encoders (VAE) Kingma & Welling (2014) utilize models P G of the form (3) and minimize

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with respect to a random decoder mapping P G
Revised: with respect to a random decoder mapping P G

Original: ( X | Z ) .
Revised: ( X | Z ) .

Original: The conditional distribution P G ( X | Z ) is often parametrized by a deep net G and can have any form as long as its density p G ( x | z ) can be computed and differentiated with respect to the parameters of G .
Revised: The conditional distribution P G ( X | Z ) is often parametrized by a deep net G and can have any form as long as its density p G ( x | z ) can be computed and differentiated with respect to the parameters of G .

Original: A typical choice is to use Gaussians P G ( X | Z ) =
Revised: A typical choice is to use Gaussians P G ( X | Z ) =

Original: N ( X ; G ( Z ) , σ 2 · I ) .
Revised: N ( X ; G ( Z ) , σ 2 · I ) .

Original: If Q is the set of all conditional probability distributions Q ( Z | X ) , the objective of VAE coincides with the negative marginal log-likelihood
Revised: If Q is the set of all conditional probability distributions Q ( Z | X ) , the objective of VAE coincides with the negative marginal log-likelihood

Original: D VAE ( P X , P G ) =
Revised: D VAE ( P X , P G ) =

Original: − E P X [log P G ( X )] .
Revised: − E P X [log P G ( X )] .

Original: However, in order to make the D KL term of (6) tractable in closed form, the original implementation of VAE uses a standard normal P Z and restricts Q to a class of Gaussian distributions Q ( Z | X ) =
Revised: However, in order to make the D KL term of (6) tractable in closed form, the original implementation of VAE uses a standard normal P Z and restricts Q to a class of Gaussian distributions Q ( Z | X ) =

Original: N (cid:0) Z ; µ ( X ) , Σ( X ) (cid:1) with mean µ and diagonal covariance Σ parametrized by deep nets.
Revised: N (cid:0) Z ; µ ( X ) , Σ( X ) (cid:1) with mean µ and diagonal covariance Σ parametrized by deep nets.

Original: As a consequence, VAE is minimizing an upper bound on the negative log-likelihood or, equivalently, on the KL-divergence D KL ( P X , P G ) .
Revised: As a consequence, VAE is minimizing an upper bound on the negative log-likelihood or, equivalently, on the KL-divergence D KL ( P X , P G ) .

Original: Further details can be found in Section D.
Revised: Further details can be found in Section D.

Original: One possible way to reduce the gap between the true negative log-likelihood and the upper bound provided by D VAE is to enlarge the class Q .
Revised: One possible way to reduce the gap between the true negative log-likelihood and the upper bound provided by D VAE is to enlarge the class Q .

Original: Adversarial variational Bayes (AVB) Mescheder et al.
Revised: Adversarial variational Bayes (AVB) Mescheder et al.

Original: (2017) follows this argument by employing the idea of GANs.
Revised: (2017) follows this argument by employing the idea of GANs.

Original: Given any point x ∈ X , a noise (cid:15) ∼ N (0 , 1) , and any ﬁxed transformation e : X × R → Z , a random variable e ( x, (cid:15) ) implicitly deﬁnes one particular conditional distribution Q e ( Z | X = x ) .
Revised: Given any point x ∈ X , a noise (cid:15) ∼ N (0 , 1) , and any ﬁxed transformation e : X × R → Z , a random variable e ( x, (cid:15) ) implicitly deﬁnes one particular conditional distribution Q e ( Z | X = x ) .

Original: AVB allows Q to contain all such distributions for different choices of e , replaces the intractable term D KL (cid:0) Q e ( Z | X ) , P Z (cid:1) in (6) by the adversarial approximation D f , GAN corresponding to the KL-divergence, and proposes to minimize
Revised: AVB allows Q to contain all such distributions for different choices of e , replaces the intractable term D KL (cid:0) Q e ( Z | X ) , P Z (cid:1) in (6) by the adversarial approximation D f , GAN corresponding to the KL-divergence, and proposes to minimize

Original: [Equation]
Revised: [Equation]

Original: The D KL term in (6) may be viewed as a regularizer.
Revised: The D KL term in (6) may be viewed as a regularizer.

Original: Indeed, VAE reduces to the classical unregularized auto-encoder if this term is dropped, minimizing the reconstruction cost of the encoder-decoder pair Q ( Z | X ) , P G ( X | Z ) .
Revised: Indeed, VAE reduces to the classical unregularized auto-encoder if this term is dropped, minimizing the reconstruction cost of the encoder-decoder pair Q ( Z | X ) , P G ( X | Z ) .

Original: This often results in different training points being encoded into nonoverlapping zones chaotically scattered all across the Z space with “holes” in between where the decoder mapping P G ( X | Z ) has never been trained.
Revised: This often results in different training points being encoded into nonoverlapping zones chaotically scattered all across the Z space with “holes” in between where the decoder mapping P G ( X | Z ) has never been trained.

Original: Overall, the encoder Q ( Z | X ) trained in this way does not provide a useful representation and sampling from the latent space Z becomes hard
Revised: Overall, the encoder Q ( Z | X ) trained in this way does not provide a useful representation and sampling from the latent space Z becomes hard

Original: Bengio et al.
Revised: Bengio et al.

Original: Adversarial auto-encoders (AAE) Makhzani et al.
Revised: Adversarial auto-encoders (AAE) Makhzani et al.

Original: (2016) replace the D KL term in (6) with another regularizer:
Revised: (2016) replace the D KL term in (6) with another regularizer:

Original: [Equation]
Revised: [Equation]

Original: where Q Z is the marginal distribution of Z when ﬁrst X is sampled from P X and then Z is sampled from Q ( Z | X ) , also known as the aggregated posterior Makhzani et al.
Revised: where Q Z is the marginal distribution of Z when ﬁrst X is sampled from P X and then Z is sampled from Q ( Z | X ) , also known as the aggregated posterior Makhzani et al.

Original: Similarly to AVB, there is no clear link to log-likelihood, as D AAE ≤ D AVB (see Appendix D).
Revised: Similarly to AVB, there is no clear link to log-likelihood, as D AAE ≤ D AVB (see Appendix D).

Original: The authors of Makhzani et al.
Revised: The authors of Makhzani et al.

Original: (2016) argue that matching Q Z to P Z in this way ensures that there are no “holes” left in the latent space Z and
Revised: (2016) argue that matching Q Z to P Z in this way ensures that there are no “holes” left in the latent space Z and

Original: P G ( X | Z ) generates reasonable samples whenever Z ∼ P Z .
Revised: P G ( X | Z ) generates reasonable samples whenever Z ∼ P Z .

Original: They also report an equally good performance of different types of conditional distributions Q ( Z | X ) , including Gaussians as used in VAEs, implicit models Q e as used in AVB, and deterministic encoder mappings, i.e. Q ( Z | X ) =
Revised: They also report an equally good performance of different types of conditional distributions Q ( Z | X ) , including Gaussians as used in VAEs, implicit models Q e as used in AVB, and deterministic encoder mappings, i.e. Q ( Z | X ) =

Original: δ µ ( X ) with µ : X → Z .
Revised: δ µ ( X ) with µ : X → Z .

Original: B P ROOF OF T HEOREM 1 AND FURTHER DETAILS
Revised: B P ROOF OF T HEOREM 1 AND FURTHER DETAILS

Original: We will consider certain sets of joint probability distributions of three random variables ( X, Y, Z ) ∈
Revised: We will consider certain sets of joint probability distributions of three random variables ( X, Y, Z ) ∈

Original: X × X × Z .
Revised: X × X × Z .

Original: The reader may wish to think of X as true images, Y as images sampled from the model, and Z as latent codes.
Revised: The reader may wish to think of X as true images, Y as images sampled from the model, and Z as latent codes.

Original: We denote by P G,Z ( Y, Z ) a joint distribution of a variable pair ( Y, Z ) , where Z is ﬁrst sampled from P Z and next Y from P G ( Y | Z ) .
Revised: We denote by P G,Z ( Y, Z ) a joint distribution of a variable pair ( Y, Z ) , where Z is ﬁrst sampled from P Z and next Y from P G ( Y | Z ) .

Original: Note that P G deﬁned in (3) and used throughout this work is the marginal distribution of Y when ( Y, Z ) ∼ P G,Z .
Revised: Note that P G deﬁned in (3) and used throughout this work is the marginal distribution of Y when ( Y, Z ) ∼ P G,Z .

Original: In the optimal transport problem (1), we consider joint distributions Γ( X, Y ) which are called couplings between values of X and Y .
Revised: In the optimal transport problem (1), we consider joint distributions Γ( X, Y ) which are called couplings between values of X and Y .

Original: Because of the marginal constraint, we can write Γ( X, Y ) =
Revised: Because of the marginal constraint, we can write Γ( X, Y ) =

Original: Γ( Y | X ) P X
Revised: Γ( Y | X ) P X

Original: ( X )
Revised: ( X )

Original: and we can consider Γ( Y | X ) as a non-deterministic mapping from X to Y .
Revised: and we can consider Γ( Y | X ) as a non-deterministic mapping from X to Y .

Original: Theorem 1. shows how to factor this mapping through Z , i.e., decompose it into an encoding distribution
Revised: Theorem 1. shows how to factor this mapping through Z , i.e., decompose it into an encoding distribution

Original: Q ( Z | X ) and the generating distribution P G ( Y | Z ) .
Revised: Q ( Z | X ) and the generating distribution P G ( Y | Z ) .

Original: As in Section 2.2, P ( X ∼ P X , Y ∼ P G ) denotes the set of all joint distributions of ( X, Y ) with marginals P X , P G , and likewise for P ( X ∼ P X , Z ∼ P Z ) .
Revised: As in Section 2.2, P ( X ∼ P X , Y ∼ P G ) denotes the set of all joint distributions of ( X, Y ) with marginals P X , P G , and likewise for P ( X ∼ P X , Z ∼ P Z ) .

Original: The set of all joint distributions of ( X, Y, Z ) such that X ∼ P X , ( Y, Z ) ∼ P G,Z , and ( Y ⊥⊥ X ) | Z will be denoted by P X,Y,Z .
Revised: The set of all joint distributions of ( X, Y, Z ) such that X ∼ P X , ( Y, Z ) ∼ P G,Z , and ( Y ⊥⊥ X ) | Z will be denoted by P X,Y,Z .

Original: Finally, we denote by P X,Y and P X,Z the sets of marginals on ( X, Y ) and ( X, Z ) (respectively) induced by distributions in P X,Y,Z .
Revised: Finally, we denote by P X,Y and P X,Z the sets of marginals on ( X, Y ) and ( X, Z ) (respectively) induced by distributions in P X,Y,Z .

Original: Note that P ( P X , P G ) , P X,Y,Z , and P X,Y depend on the choice of conditional distributions P G ( Y | Z ) , while P X,
Revised: Note that P ( P X , P G ) , P X,Y,Z , and P X,Y depend on the choice of conditional distributions P G ( Y | Z ) , while P X,

Original: Z does not.
Revised: Z does not.

Original: In fact, it is easy to check that P X,Z =
Revised: In fact, it is easy to check that P X,Z =

Original: P ( X ∼ P X , Z ∼ P Z ) .
Revised: P ( X ∼ P X , Z ∼ P Z ) .

Original: From the deﬁnitions it is clear that P X,Y ⊆ P ( P X , P G ) and we immediately get the following upper bound:
Revised: From the deﬁnitions it is clear that P X,Y ⊆ P ( P X , P G ) and we immediately get the following upper bound:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If P G ( Y | Z ) are Dirac measures (i.e., Y = G ( Z ) ), it turns out that P X,Y = P ( P X , P G ) :
Revised: If P G ( Y | Z ) are Dirac measures (i.e., Y = G ( Z ) ), it turns out that P X,Y = P ( P X , P G ) :

Original: Lemma 1 P X,Y ⊆ P ( P X , P G ) with identity if 4 P G ( Y | Z = z ) are Dirac for all z ∈ Z .
Revised: Lemma 1 P X,Y ⊆ P ( P X , P G ) with identity if 4 P G ( Y | Z = z ) are Dirac for all z ∈ Z .

Original: Proof The ﬁrst assertion is obvious.
Revised: Proof The ﬁrst assertion is obvious.

Original: To prove the identity, note that when Y is a deterministic function of Z , for any A in the sigma-algebra induced by Y we have E (cid:2) 

Revised: To prove the identity, note that when Y is a deterministic function of Z , for any A in the sigma-algebra induced by Y we have E (cid:2)

Original: [ Y ∈ A ] | X, Z (cid:3) =
Revised: [ Y ∈ A ] | X, Z (cid:3) =

Original: E (cid:2) 

Revised: E (cid:2)

Original: [ Y ∈ A ] | Z (cid:3) .
Revised: [ Y ∈ A ] | Z (cid:3) .

Original: This implies ( Y ⊥⊥ X ) | Z and concludes the proof.
Revised: This implies ( Y ⊥⊥ X ) | Z and concludes the proof.

Original: We are now in place to prove Theorem 1.
Revised: We are now in place to prove Theorem 1.

Original: Lemma 1 obviously leads to
Revised: Lemma 1 obviously leads to

Original: [Equation]
Revised: [Equation]

Original: The tower rule of expectation, and the conditional independence property of P X,Y,Z implies
Revised: The tower rule of expectation, and the conditional independence property of P X,Y,Z implies

Original: [Equation]
Revised: [Equation]

Original: It remains to notice that P X,Z = P ( X ∼ P X , Z ∼ P Z ) as stated earlier.
Revised: It remains to notice that P X,Z = P ( X ∼ P X , Z ∼ P Z ) as stated earlier.

Original: B. R ANDOM DECODERS P G ( Y | Z )
Revised: B. R ANDOM DECODERS P G ( Y | Z )

Original: If the decoders are non-deterministic, Lemma 1 provides only the inclusion of sets
Revised: If the decoders are non-deterministic, Lemma 1 provides only the inclusion of sets

Original: P X,Y ⊆ P ( P X , P G ) and we get the following upper bound on the OT:
Revised: P X,Y ⊆ P ( P X , P G ) and we get the following upper bound on the OT:

Original: Corollary 1 Let X =
Revised: Corollary 1 Let X =

Original: R d and assume the conditional distributions P G ( Y | Z = z ) have mean values G ( z ) ∈ R d and marginal variances σ 21 , . .
Revised: R d and assume the conditional distributions P G ( Y | Z = z ) have mean values G ( z ) ∈ R d and marginal variances σ 21 , . .

Original: , σ 2 d ≥ 0 for all z ∈ Z , where G : Z → X .
Revised: , σ 2 d ≥ 0 for all z ∈ Z , where G : Z → X .

Original: Take c ( x, y )
Revised: Take c ( x, y )

Original: = (cid:107) x − y (cid:107) 22 .
Revised: = (cid:107) x − y (cid:107) 22 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof First inequality follows from (9).
Revised: Proof First inequality follows from (9).

Original: For the identity we proceed similarly to the proof of
Revised: For the identity we proceed similarly to the proof of

Original: Theorem 1 and write
Revised: Theorem 1 and write

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Together with (11) and the fact that P X,Z = P ( X ∼ P X , Z ∼ P Z ) this concludes the proof.
Revised: Together with (11) and the fact that P X,Z = P ( X ∼ P X , Z ∼ P Z ) this concludes the proof.

Original: C R ELATIONS OF WAE TO AAE, VAE S , AND GAN S
Revised: C R ELATIONS OF WAE TO AAE, VAE S , AND GAN S

Original: In Section C.1, we compare minimizing the optimal transport cost W c , the upper bound W † c , and its relaxed version D WAE to VAE, AVB, and AAE in the special case when c ( x, y )
Revised: In Section C.1, we compare minimizing the optimal transport cost W c , the upper bound W † c , and its relaxed version D WAE to VAE, AVB, and AAE in the special case when c ( x, y )

Original: = (cid:107) x − y (cid:107) 2 and
Revised: = (cid:107) x − y (cid:107) 2 and

Original: P G ( Y | Z ) =
Revised: P G ( Y | Z ) =

Original: N ( Y ; G ( Z ) , σ 2 · I ) .
Revised: N ( Y ; G ( Z ) , σ 2 · I ) .

Original: We show that in this case the solutions of VAE and AVB both depend on σ 2 , while the minimizer G † of W † c ( P X , P G ) does not depend on σ 2 , and is the same as the minimizer of W c ( P X , P G ) for σ 2 = 0 .
Revised: We show that in this case the solutions of VAE and AVB both depend on σ 2 , while the minimizer G † of W † c ( P X , P G ) does not depend on σ 2 , and is the same as the minimizer of W c ( P X , P G ) for σ 2 = 0 .

Original: We also brieﬂy discuss the role of these conclusions in explaining the well-known blurriness of VAE outputs.
Revised: We also brieﬂy discuss the role of these conclusions in explaining the well-known blurriness of VAE outputs.

Original: Section C.2 shows that when c ( x, y ) = (cid:107) x − y (cid:107) , WAE and WGAN approach primal and dual forms respectively of the same optimization problem .
Revised: Section C.2 shows that when c ( x, y ) = (cid:107) x − y (cid:107) , WAE and WGAN approach primal and dual forms respectively of the same optimization problem .

Original: Finally, we discuss a difference in behaviour between the two algorithms caused by this duality.
Revised: Finally, we discuss a difference in behaviour between the two algorithms caused by this duality.

Original: We refer the reader to Supplementary A for the detailed overview of all these methods.
Revised: We refer the reader to Supplementary A for the detailed overview of all these methods.

Original: C. T HE 2-W ASSERSTEIN DISTANCE : RELATION TO VAE, AVB, AND AAE
Revised: C. T HE 2-W ASSERSTEIN DISTANCE : RELATION TO VAE, AVB, AND AAE

Original: Consider the squared Euclidean cost function c ( x, y )
Revised: Consider the squared Euclidean cost function c ( x, y )

Original: = (cid:107) x − y (cid:107) 2 , for which W c is the squared 2Wasserstein distance W 22 .
Revised: = (cid:107) x − y (cid:107) 2 , for which W c is the squared 2Wasserstein distance W 22 .

Original: The goal of this section is to compare the minimization of W 2 ( P X , P G ) to other generative modeling approaches.
Revised: The goal of this section is to compare the minimization of W 2 ( P X , P G ) to other generative modeling approaches.

Original: Let us focus our attention on generative distributions
Revised: Let us focus our attention on generative distributions

Original: P G ( Y | Z ) typically used in VAE, AVB, and AAE, i.e., Gaussians P G ( Y | Z ) =
Revised: P G ( Y | Z ) typically used in VAE, AVB, and AAE, i.e., Gaussians P G ( Y | Z ) =

Original: N (cid:0) Y ; G ( Z ) , σ 2 · I d (cid:1) .
Revised: N (cid:0) Y ; G ( Z ) , σ 2 · I d (cid:1) .

Original: In order to verify the differentiability of log p G ( x | z ) all three methods require σ 2 > 0 and have problems handling the case of deterministic decoders ( σ 2 = 0 ).
Revised: In order to verify the differentiability of log p G ( x | z ) all three methods require σ 2 > 0 and have problems handling the case of deterministic decoders ( σ 2 = 0 ).

Original: To emphasize the role of the variance σ 2 we will denote the resulting latent variable model P σG .
Revised: To emphasize the role of the variance σ 2 we will denote the resulting latent variable model P σG .

Original: Relation to VAE and AVB
Revised: Relation to VAE and AVB

Original: The analysis of Supplementary B shows that the value of W 2 ( P X , P σG ) is upper bounded by W † c ( P X , P σG ) of the form (10) and the two coincide when σ 2 = 0 .
Revised: The analysis of Supplementary B shows that the value of W 2 ( P X , P σG ) is upper bounded by W † c ( P X , P σG ) of the form (10) and the two coincide when σ 2 = 0 .

Original: Next we summarize properties of solutions G minimizing these two values W 2 and W † c :
Revised: Next we summarize properties of solutions G minimizing these two values W 2 and W † c :

Original: Proposition 1 Let X =
Revised: Proposition 1 Let X =

Original: R d and assume c ( x, y )
Revised: R d and assume c ( x, y )

Original: = (cid:107) x − y (cid:107) 2 , P G
Revised: = (cid:107) x − y (cid:107) 2 , P G

Original: ( Y | Z )
Revised: ( Y | Z )

Original: = N (cid:0) Y ; G ( Z ) ,
Revised: = N (cid:0) Y ; G ( Z ) ,

Original: σ 2 · I (cid:1) with any function G : X → R .
Revised: σ 2 · I (cid:1) with any function G : X → R .

Original: If σ 2 > 0 then the functions G ∗ σ and G † minimizing W c ( P X , P σG ) and
Revised: If σ 2 > 0 then the functions G ∗ σ and G † minimizing W c ( P X , P σG ) and

Original: W † c ( P X , P σG ) respectively are different: G ∗ σ depends on σ 2 , while G † does not.
Revised: W † c ( P X , P σG ) respectively are different: G ∗ σ depends on σ 2 , while G † does not.

Original: The function G † is also a minimizer of W c ( P X , P 0 G ) .
Revised: The function G † is also a minimizer of W c ( P X , P 0 G ) .

Original: In order to prove this proposition we will need the following simple result, which is basically saying that the variance of a sum of two independent random variables is a sum of the variances:
Revised: In order to prove this proposition we will need the following simple result, which is basically saying that the variance of a sum of two independent random variables is a sum of the variances:

Original: Lemma 2 Under conditions of Proposition 1, assume Y ∼ P σG .
Revised: Lemma 2 Under conditions of Proposition 1, assume Y ∼ P σG .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof First of all, using (3) we have
Revised: Proof First of all, using (3) we have

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Next we turn to the proof of Proposition 1:
Revised: Next we turn to the proof of Proposition 1:

Original: Proof Corollary 1 shows that G † does not depend on the variance σ 2 .
Revised: Proof Corollary 1 shows that G † does not depend on the variance σ 2 .

Original: When σ 2 = 0 the distribution
Revised: When σ 2 = 0 the distribution

Original: P G ( Y | Z ) turns into Dirac.
Revised: P G ( Y | Z ) turns into Dirac.

Original: In this case we combine Theorem 1 and Corollary 1 to conclude that
Revised: In this case we combine Theorem 1 and Corollary 1 to conclude that

Original: G † also minimizes W c ( P X , P 0 G ) .
Revised: G † also minimizes W c ( P X , P 0 G ) .

Original: Next we prove that when σ 2 > 0 the function G ∗ σ minimizing
Revised: Next we prove that when σ 2 > 0 the function G ∗ σ minimizing

Original: W c ( P X , P σG ) depends on σ 2 .
Revised: W c ( P X , P σG ) depends on σ 2 .

Original: The proof is based on the following example:
Revised: The proof is based on the following example:

Original: X = Z = R , P X = N (0 , 1) , P Z = N (0 , 1) , and 0 < σ 2 < 1 .
Revised: X = Z = R , P X = N (0 , 1) , P Z = N (0 , 1) , and 0 < σ 2 < 1 .

Original: Note that by setting G ( z )
Revised: Note that by setting G ( z )

Original: =
Revised: =

Original: c · z for any c > 0 we ensure that P σG is the Gaussian distribution, because a convolution of two Gaussians is also Gaussian.
Revised: c · z for any c > 0 we ensure that P σG is the Gaussian distribution, because a convolution of two Gaussians is also Gaussian.

Original: In particular if we take
Revised: In particular if we take

Original: G ∗ ( z ) = √ 1 − σ 2 · z .
Revised: G ∗ ( z ) = √ 1 − σ 2 · z .

Original: Lemma 2 implies that P σ G ∗ is the standard normal Gaussian N (0 , 1) .
Revised: Lemma 2 implies that P σ G ∗ is the standard normal Gaussian N (0 , 1) .

Original: In other words, we obtain the global minimum W c ( P X , P σG ∗ ) = 

Revised: In other words, we obtain the global minimum W c ( P X , P σG ∗ ) =

Original: and G ∗ clearly depends on σ 2 .
Revised: and G ∗ clearly depends on σ 2 .

Original: For the purpose of generative modeling, the noise σ 2 > 0 is often not desirable, and it is common practice to sample from the trained model G ∗ by simply returning G ∗ ( Z ) for Z ∼ P Z without adding noise to the output.
Revised: For the purpose of generative modeling, the noise σ 2 > 0 is often not desirable, and it is common practice to sample from the trained model G ∗ by simply returning G ∗ ( Z ) for Z ∼ P Z without adding noise to the output.

Original: This leads to a mismatch between inference and training.
Revised: This leads to a mismatch between inference and training.

Original: Furthermore,
Revised: Furthermore,

Original: VAE, AVB, and other similar variational methods implicitly use σ 2 as a factor to balance the (cid:96)reconstruction cost and the KL-regularizer.
Revised: VAE, AVB, and other similar variational methods implicitly use σ 2 as a factor to balance the (cid:96)reconstruction cost and the KL-regularizer.

Original: In contrast, Proposition 1 shows that for the same Gaussian models with any given σ 2 ≥ 0 we can minimize W † c ( P X , P σG ) and the solution G † will be indeed the one resulting in the smallest 2Wasserstein distance between P X and the noiseless implicit model G ( Z ) , Z ∼ P Z used in practice.
Revised: In contrast, Proposition 1 shows that for the same Gaussian models with any given σ 2 ≥ 0 we can minimize W † c ( P X , P σG ) and the solution G † will be indeed the one resulting in the smallest 2Wasserstein distance between P X and the noiseless implicit model G ( Z ) , Z ∼ P Z used in practice.

Original: Blurriness of VAE and AVB We next add to the discussion regarding the blurriness commonly attributed to VAE samples.
Revised: Blurriness of VAE and AVB We next add to the discussion regarding the blurriness commonly attributed to VAE samples.

Original: Our argument shows that VAE, AVB, and other methods based on the marginal log-likelihood necessarily lead to an averaging in the input space if P G ( Y | Z ) are Gaussian.
Revised: Our argument shows that VAE, AVB, and other methods based on the marginal log-likelihood necessarily lead to an averaging in the input space if P G ( Y | Z ) are Gaussian.

Original: First we notice that in the VAE and AVB objectives, for any ﬁxed encoder Q ( Z | X ) , the decoder is minimizing the expected (cid:96) 2 -reconstruction cost E P X E Q ( Z | X ) (cid:2) (cid:107) X
Revised: First we notice that in the VAE and AVB objectives, for any ﬁxed encoder Q ( Z | X ) , the decoder is minimizing the expected (cid:96) 2 -reconstruction cost E P X E Q ( Z | X ) (cid:2) (cid:107) X

Original: − G ( Z ) (cid:107) 2 (cid:3) with respect to G .
Revised: − G ( Z ) (cid:107) 2 (cid:3) with respect to G .

Original: The optimal solution G ∗ is of the form G ∗ ( z ) =
Revised: The optimal solution G ∗ is of the form G ∗ ( z ) =

Original: E P ∗ z
Revised: E P ∗ z

Original: [ X ] , where P ∗ z ( X ) ∝ P X ( X ) Q ( Z = z | X ) .
Revised: [ X ] , where P ∗ z ( X ) ∝ P X ( X ) Q ( Z = z | X ) .

Original: Hence, as soon as supp P ∗ z is non-singleton, the optimal decoder G ∗ will end up averaging points in the input space.
Revised: Hence, as soon as supp P ∗ z is non-singleton, the optimal decoder G ∗ will end up averaging points in the input space.

Original: In particular this will happen whenever there are two points x 1 , x 2 in supp P X such that supp Q ( Z | X = x 1 ) and supp Q ( Z | X = x 2 ) overlap.
Revised: In particular this will happen whenever there are two points x 1 , x 2 in supp P X such that supp Q ( Z | X = x 1 ) and supp Q ( Z | X = x 2 ) overlap.

Original: This overlap necessarily happens in VAEs, which use Gaussian encoders Q ( Z | X ) supported on the entire Z .
Revised: This overlap necessarily happens in VAEs, which use Gaussian encoders Q ( Z | X ) supported on the entire Z .

Original: When probabilistic encoders Q are allowed to be ﬂexible enough, as in AVB, for any ﬁxed
Revised: When probabilistic encoders Q are allowed to be ﬂexible enough, as in AVB, for any ﬁxed

Original: P G ( Y | Z ) the optimal Q ∗ will try to invert the decoder (see Appendix D) and take the form
Revised: P G ( Y | Z ) the optimal Q ∗ will try to invert the decoder (see Appendix D) and take the form

Original: [Equation]
Revised: [Equation]

Original: This approximation becomes exact in the nonparametric limit of Q .
Revised: This approximation becomes exact in the nonparametric limit of Q .

Original: When P G ( Y | Z ) is Gaussian we have p G ( y | z ) > 0 for all y ∈ X and z ∈ Z , showing that supp Q ∗ ( Z | X = x )
Revised: When P G ( Y | Z ) is Gaussian we have p G ( y | z ) > 0 for all y ∈ X and z ∈ Z , showing that supp Q ∗ ( Z | X = x )

Original: =
Revised: =

Original: supp P Z for all x ∈ X .
Revised: supp P Z for all x ∈ X .

Original: This will again lead to the overlap of encoders if supp P Z = Z .
Revised: This will again lead to the overlap of encoders if supp P Z = Z .

Original: In contrast, the optimal encoders of WAE do not necessarily overlap, as they are not inverting the decoders.
Revised: In contrast, the optimal encoders of WAE do not necessarily overlap, as they are not inverting the decoders.

Original: The common belief today is that the blurriness of VAEs is caused by the (cid:96) 2 reconstruction cost, or equivalently by the Gaussian form of decoders P G ( Y | Z ) .
Revised: The common belief today is that the blurriness of VAEs is caused by the (cid:96) 2 reconstruction cost, or equivalently by the Gaussian form of decoders P G ( Y | Z ) .

Original: We argue that it is instead caused by the combination of (a) Gaussian decoders and (b) the objective (KL-divergence) being minimized.
Revised: We argue that it is instead caused by the combination of (a) Gaussian decoders and (b) the objective (KL-divergence) being minimized.

Original: C. T HE 1-W ASSERSTEIN DISTANCE : RELATION TO WGAN
Revised: C. T HE 1-W ASSERSTEIN DISTANCE : RELATION TO WGAN

Original: We have shown that the D
Revised: We have shown that the D

Original: WAE criterion leads to a generalized version of the AAE algorithm and can be seen as a relaxation of the optimal transport cost W c .
Revised: WAE criterion leads to a generalized version of the AAE algorithm and can be seen as a relaxation of the optimal transport cost W c .

Original: In particular, if we choose c to be the
Revised: In particular, if we choose c to be the

Original: Euclidean distance c ( x, y )
Revised: Euclidean distance c ( x, y )

Original: = (cid:107) x − y (cid:107) , we get a primal formulation of W 1 .
Revised: = (cid:107) x − y (cid:107) , we get a primal formulation of W 1 .

Original: This is the same criterion that WGAN aims to minimize in the dual formulation (see Eq. 2).
Revised: This is the same criterion that WGAN aims to minimize in the dual formulation (see Eq. 2).

Original: As a result of Theorem 1, we have
Revised: As a result of Theorem 1, we have

Original: [Equation]
Revised: [Equation]

Original: This means we can now approach the problem of optimizing W 1 in two distinct ways, taking gradient steps either in the primal or in the dual forms.
Revised: This means we can now approach the problem of optimizing W 1 in two distinct ways, taking gradient steps either in the primal or in the dual forms.

Original: Denote by Q ∗ the optimal encoder in the primal and f ∗ the optimal witness function in the dual.
Revised: Denote by Q ∗ the optimal encoder in the primal and f ∗ the optimal witness function in the dual.

Original: By the envelope theorem, gradients of W 1 with respect to G can be computed by taking a gradient of the criteria evaluated at the optimal points Q ∗ or f ∗ .
Revised: By the envelope theorem, gradients of W 1 with respect to G can be computed by taking a gradient of the criteria evaluated at the optimal points Q ∗ or f ∗ .

Original: Despite the theoretical equivalence of both approaches, practical considerations lead to different behaviours and to potentially poor approximations of the real gradients.
Revised: Despite the theoretical equivalence of both approaches, practical considerations lead to different behaviours and to potentially poor approximations of the real gradients.

Original: For example, in the dual formulation, one usually restricts the witness functions to be smooth, while in the primal formulation, the constraint on Q is only approximately enforced.
Revised: For example, in the dual formulation, one usually restricts the witness functions to be smooth, while in the primal formulation, the constraint on Q is only approximately enforced.

Original: We will study the effect of these approximations.
Revised: We will study the effect of these approximations.

Original: Imperfect gradients in the dual (i.e., for WGAN)
Revised: Imperfect gradients in the dual (i.e., for WGAN)

Original: We show that (i) if the true optimum f ∗ is not reached exactly (no matter how close), the effect on the gradient in the dual formulation can be arbitrarily large, and (ii) this also holds when the optimization is performed only in a restricted class of smooth functions.
Revised: We show that (i) if the true optimum f ∗ is not reached exactly (no matter how close), the effect on the gradient in the dual formulation can be arbitrarily large, and (ii) this also holds when the optimization is performed only in a restricted class of smooth functions.

Original: We write the criterion to be optimized as J D ( f ) :=
Revised: We write the criterion to be optimized as J D ( f ) :=

Original: E P X [ f ( X )] − E
Revised: E P X [ f ( X )] − E

Original: P Z [ f ( G ( Z ))] and denote its gradient with respect to G by ∇ J D ( f ) .
Revised: P Z [ f ( G ( Z ))] and denote its gradient with respect to G by ∇ J D ( f ) .

Original: Let H be a subset of the 1 -Lipschitz functions F L on X containing smooth functions with bounded Hessian.
Revised: Let H be a subset of the 1 -Lipschitz functions F L on X containing smooth functions with bounded Hessian.

Original: Denote by f ∗H the minimizer of J D in H .
Revised: Denote by f ∗H the minimizer of J D in H .

Original: A ( f, f (cid:48) ) :
Revised: A ( f, f (cid:48) ) :

Original: = cos ( ∇ J D ( f ) , ∇ J D ( f (cid:48) )) will denote the cosine of the angle between the gradients of the criterion at different functions.
Revised: = cos ( ∇ J D ( f ) , ∇ J D ( f (cid:48) )) will denote the cosine of the angle between the gradients of the criterion at different functions.

Original: Proposition 2 There exists a constant C > 0 such that for any (cid:15) > 0 , one can construct distributions P X , P G and pick witness functions f (cid:15) ∈ F L and h (cid:15) ∈ H that are (cid:15) -optimal | J D ( f (cid:15) )
Revised: Proposition 2 There exists a constant C > 0 such that for any (cid:15) > 0 , one can construct distributions P X , P G and pick witness functions f (cid:15) ∈ F L and h (cid:15) ∈ H that are (cid:15) -optimal | J D ( f (cid:15) )

Original: − J D ( f ∗ ) | ≤ (cid:15) , | J D ( h (cid:15) )
Revised: − J D ( f ∗ ) | ≤ (cid:15) , | J D ( h (cid:15) )

Original: − J D ( h ∗ ) | ≤ (cid:15) , but which give (at some point z ∈ Z ) gradients whose direction is at least C
Revised: − J D ( h ∗ ) | ≤ (cid:15) , but which give (at some point z ∈ Z ) gradients whose direction is at least C

Original: -wrong
Revised: -wrong

Original: A ( f (cid:15) , f ∗ ) ≤ 1 − C , A ( h 0 , h ∗ ) ≤ 1 − C , and A ( h (cid:15) , h ∗ ) ≤ 0 .
Revised: A ( f (cid:15) , f ∗ ) ≤ 1 − C , A ( h 0 , h ∗ ) ≤ 1 − C , and A ( h (cid:15) , h ∗ ) ≤ 0 .

Original: Proof We give a sketch of the proof.
Revised: Proof We give a sketch of the proof.

Original: Consider discrete distributions P X supported on two points { x 0 , x 1 } , and P Z supported on { 0 , 1 } and let y 

Revised: Consider discrete distributions P X supported on two points { x 0 , x 1 } , and P Z supported on { 0 , 1 } and let y

Original: = G (0) , y 

Revised: = G (0) , y

Original: =
Revised: =

Original: G (1) ( y 0 (cid:54) =
Revised: G (1) ( y 0 (cid:54) =

Original: y 1 ).
Revised: y 1 ).

Original: Given an optimal f ∗ , one can modify locally it around y 0 without changing its Lipschitz constant such that the obtained f (cid:15) is an (cid:15) -approximation of f ∗ whose gradients at y 0 and y 1 point in directions arbitrarily different from those of f ∗ .
Revised: Given an optimal f ∗ , one can modify locally it around y 0 without changing its Lipschitz constant such that the obtained f (cid:15) is an (cid:15) -approximation of f ∗ whose gradients at y 0 and y 1 point in directions arbitrarily different from those of f ∗ .

Original: For smooth functions, by moving y 0 and y 1 away from the segment [ x 0 , x 1 ] but close to each other (cid:107) y 

Revised: For smooth functions, by moving y 0 and y 1 away from the segment [ x 0 , x 1 ] but close to each other (cid:107) y

Original: − y 1 (cid:107) ≤ K(cid:15) , the gradients of f ∗ will point in directions roughly opposite but the constraint on the Hessian will force the gradients of f F , 0 at y 0 and y 1 to be very close.
Revised: − y 1 (cid:107) ≤ K(cid:15) , the gradients of f ∗ will point in directions roughly opposite but the constraint on the Hessian will force the gradients of f F , 0 at y 0 and y 1 to be very close.

Original: Finally, putting y 0 , y 1 on the segment [ x 0 , x 1 ] , one can get an f ∗F whose gradients at y 0 and y 1 are exactly opposite, while taking f F ,(cid:15) ( y ) =
Revised: Finally, putting y 0 , y 1 on the segment [ x 0 , x 1 ] , one can get an f ∗F whose gradients at y 0 and y 1 are exactly opposite, while taking f F ,(cid:15) ( y ) =

Original: f
Revised: f

Original: ∗F ( y + (cid:15) ) , we can swap the direction at one of the points while changing the criterion by less than (cid:15) .
Revised: ∗F ( y + (cid:15) ) , we can swap the direction at one of the points while changing the criterion by less than (cid:15) .

Original: Imperfect posterior in the primal (i.e., for WAE)
Revised: Imperfect posterior in the primal (i.e., for WAE)

Original: In the primal formulation, when the constraint is violated, that is the aggregated posterior Q Z is not matching P Z , there can be two kinds of negative effects: (i) the gradient of the criterion is only computed on a (possibly small) subset of the latent space reached by Q Z ; (ii) several input points could be mapped by Q ( Z | X ) to the same latent code z , thus giving gradients that encourage G ( z ) to be the average/median of several inputs (hence encouraging a blurriness).
Revised: In the primal formulation, when the constraint is violated, that is the aggregated posterior Q Z is not matching P Z , there can be two kinds of negative effects: (i) the gradient of the criterion is only computed on a (possibly small) subset of the latent space reached by Q Z ; (ii) several input points could be mapped by Q ( Z | X ) to the same latent code z , thus giving gradients that encourage G ( z ) to be the average/median of several inputs (hence encouraging a blurriness).

Original: D F URTHER DETAILS ON VAE S AND GAN S
Revised: D F URTHER DETAILS ON VAE S AND GAN S

Original: VAE, KL-divergence and a marginal log-likelihood For models P G of the form (3) and any conditional distribution Q ( Z | X ) it can be easily veriﬁed that
Revised: VAE, KL-divergence and a marginal log-likelihood For models P G of the form (3) and any conditional distribution Q ( Z | X ) it can be easily veriﬁed that

Original: [Equation]
Revised: [Equation]

Original: Here the conditional distribution P G ( Z | X ) is induced by a joint distribution P G,Z ( X, Z ) , which is in turn speciﬁed by the 2-step latent variable procedure: (a) sample Z from P Z , (b) sample X from
Revised: Here the conditional distribution P G ( Z | X ) is induced by a joint distribution P G,Z ( X, Z ) , which is in turn speciﬁed by the 2-step latent variable procedure: (a) sample Z from P Z , (b) sample X from

Original: P G ( X | Z ) .
Revised: P G ( X | Z ) .

Original: Note that the ﬁrst term on the r.h.s.
Revised: Note that the ﬁrst term on the r.h.s.

Original: of (13) is always non-positive, while the l.h.s.
Revised: of (13) is always non-positive, while the l.h.s.

Original: does not depend on Q .
Revised: does not depend on Q .

Original: This shows that if conditional distributions Q are not restricted then
Revised: This shows that if conditional distributions Q are not restricted then

Original: [Equation]
Revised: [Equation]

Original: where the inﬁmum is achieved for Q ( Z | X )
Revised: where the inﬁmum is achieved for Q ( Z | X )

Original: =
Revised: =

Original: P G ( Z | X ) .
Revised: P G ( Z | X ) .

Original: However, for any restricted class Q of conditional distributions Q ( Z | X ) we only have
Revised: However, for any restricted class Q of conditional distributions Q ( Z | X ) we only have

Original: [Equation]
Revised: [Equation]

Original: where the inequality accounts for the fact that Q ( Z | X ) might be not ﬂexible enough to match
Revised: where the inequality accounts for the fact that Q ( Z | X ) might be not ﬂexible enough to match

Original: P ( Z | X ) for all values of X .
Revised: P ( Z | X ) for all values of X .

Original: Relation between AAE, AVB, and VAE
Revised: Relation between AAE, AVB, and VAE

Original: Proposition 3 For any distributions P X and P G :
Revised: Proposition 3 For any distributions P X and P G :

Original: [Equation]
Revised: [Equation]

Original: Proof By Jensen’s inequality and the joint convexity of D GAN we have
Revised: Proof By Jensen’s inequality and the joint convexity of D GAN we have

Original: [Equation]
Revised: [Equation]

Original: Under certain assumptions it is also possible to link D AAE to D VAE :
Revised: Under certain assumptions it is also possible to link D AAE to D VAE :

Original: Proposition 4 Assume D KL ( Q ( Z | X ) , P Z ) ≥ 1 / 4 for all Q ∈ Q with P X -probability 1.
Revised: Proposition 4 Assume D KL ( Q ( Z | X ) , P Z ) ≥ 1 / 4 for all Q ∈ Q with P X -probability 1.

Original: Then
Revised: Then

Original: D AAE ( P X , P G ) ≤ D VAE ( P X , P G ) .
Revised: D AAE ( P X , P G ) ≤ D VAE ( P X , P G ) .

Original: Proof We already mentioned that D GAN ( P, Q ) ≤ 2 · D JS ( P, Q ) − log(4) for any distributions
Revised: Proof We already mentioned that D GAN ( P, Q ) ≤ 2 · D JS ( P, Q ) − log(4) for any distributions

Original: P and Q .
Revised: P and Q .

Original: Furthermore, D JS ( P, Q ) ≤ 12 D TV ( P, Q )
Revised: Furthermore, D JS ( P, Q ) ≤ 12 D TV ( P, Q )

Original: (Lin, 1991, Theorem 3) and D TV ( P, Q ) ≤ (cid:112)
Revised: (Lin, 1991, Theorem 3) and D TV ( P, Q ) ≤ (cid:112)

Original: D KL ( P, Q ) (Tsybakov, 2008, Eq. 2.20), which leads to
Revised: D KL ( P, Q ) (Tsybakov, 2008, Eq. 2.20), which leads to

Original: [Equation]
Revised: [Equation]

Original: Together with the joint convexity of D JS and Jensen’s inequality this implies
Revised: Together with the joint convexity of D JS and Jensen’s inequality this implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) [log
Revised: ) [log

Original: [Equation]
Revised: [Equation]

Original: Z )]
Revised: Z )]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]

Original: P

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: E F URTHER DETAILS ON EXPERIMENTS
Revised: E F URTHER DETAILS ON EXPERIMENTS

Original: MNIST:
Revised: MNIST:

Original: We use mini-batches of size 100, σ 2 z = 1 , and 4x4 convolutional ﬁlters.
Revised: We use mini-batches of size 100, σ 2 z = 1 , and 4x4 convolutional ﬁlters.

Original: The reported models were trained for 100 epochs.
Revised: The reported models were trained for 100 epochs.

Original: We used α = 10 − 3 for Adam in the beginning, decreased it to 5 × 10 − 4 after 30 epochs, and to 10 − 4 after ﬁrst 50 epochs.
Revised: We used α = 10 − 3 for Adam in the beginning, decreased it to 5 × 10 − 4 after 30 epochs, and to 10 − 4 after ﬁrst 50 epochs.

Original: CelebA: We pre-processed CelebA images by ﬁrst taking a 140x140 center crops and then resizing to the 64x64 resolution.
Revised: CelebA: We pre-processed CelebA images by ﬁrst taking a 140x140 center crops and then resizing to the 64x64 resolution.

Original: We used mini-batches of size 100 and trained the models for various number of epochs (up to 250).
Revised: We used mini-batches of size 100 and trained the models for various number of epochs (up to 250).

Original: All reported WAE models were trained for 55 epochs and VAE for 35 epochs.
Revised: All reported WAE models were trained for 55 epochs and VAE for 68 epochs.

Original: Initial learning rate of Adam was set to α = 10 − 4 as often recommended in the literature, decreased it to 5 × 10 − 5 after 30 epochs, and to 10 − 5 after ﬁrst 50 epochs..
Revised: Initial learning rate of Adam was set to α = 10 − 4 as often recommended in the literature, decreased it to 5 × 10 − 5 after 30 epochs, and to 10 − 5 after ﬁrst 50 epochs..