W2GAN:
R ECOVERING
AN
O PTIMAL
T RANSPORT
M AP WITH A GAN
Anonymous authors
Paper under double-blind review
A BSTRACT
Understanding and improving Generative Adversarial Networks (GAN) using notions from Optimal Transport (OT) theory has been a successful area of study, originally established by the introduction of the Wasserstein GAN (WGAN).
An increasing number of GANs incorporate OT for improving their discriminators, but that is so far the sole way for the two domains to cross-fertilize.
In this work we address the converse question: is it possible to recover an optimal map in a GAN fashion?
To achieve this, we build a new model relying on the second Wasserstein distance.
This choice enables the use of many results from OT community.
In particular, we may completely describe the dynamics of the generator during training.
In addition, experiments show that practical uses of our model abide by the rule of evolution we describe.
As an application, our generator may be considered as a new way of computing an optimal transport map.
It is competitive in low-dimension with standard and deterministic ways to approach the same problem.
In high dimension, the fact it is a GAN-style method makes it more powerful than other methods.
I NTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are powerful probabilistic generative models which have attained state-of-the-art results, especially in high-dimensional image data.
Recently, Optimal Transport (OT) theory has contributed to the success of GANs, mainly in deﬁning more robust training objectives (Arjovsky et al., 2017;
Gulrajani et al., 2017; Miyato et al., 2018; Sanjabi et al., 2018).
In this work, we explore a contribution in the other direction: we show that an adversarially trained generative model can recover an optimal map , i.e. a map between two distributions with a minimal cost of “transport”.
Estimation of optimal maps constitutes a central problem in the OT community, especially for highdimensional continuous distributions (Peyr´e & Cuturi, 2018).
The well-known success of GANs in generative modelling of high-dimensional data suggests that such a generator might provide a good approximation of the optimal transport map in high-dimensional applications.
There has also been considerable recent interest in using GANs to learn unsupervised mappings across domains (Zhu et al., 2017; Almahairi et al., 2018;
Galanti et al., 2018).
However, to date, these methods generally rely on architectural features and heuristics to ensure meaningful mappings.
OT theory provides a means of formalizing learned mapping between distributions, hence placing this line of inquiry on stronger theoretical foundations (Courty et al., 2017a).
A fundamental problem with existing GAN frameworks is that we cannot theoretically characterize the training dynamics of the generator, nor the ﬁnal mapping that it learns.
As the generator’s training signal derives from the discriminator, any formal description of the generator’s training dynamics must rely on a theoretical characterization of the discriminator throughout training.
In the case of Wasserstein GANs (Arjovsky et al., 2017) and its extensions, there is no uniqueness of the optimal discriminator at each update step, and hence the generator may follow one of inﬁnitely many paths of evolution throughout training (Arjovsky & Bottou, 2017).
For traditional GANs relying on f -divergences, discriminators are updated only a few times at each iteration of the training procedure, and could be far from the optimal discriminator.
In such cases, characterizing the generator’s evolution in terms of the ideal discriminator is irrelevant.
On the other hand, the uniqueness of optima and the simplicity of geodesics in the case of the second Wasserstein metric (Villani, 2008) makes it a good candidate as a discriminator objective.
We introduce the W2GAN that minimizes the second Wasserstein distance.
Within the W2GAN framework, we can characterize the evolution of the generated distribution.
More speciﬁcally, we show that it follows the Wasserstein-2 geodesic between the generator’s initial distribution and the target distribution.
Crucially, the generator of W2GAN is not learning an arbitrary mapping onto the target distribution.
We show that, at convergence, it is reproducing the optimal transport map between the generator’s initial distribution and the target distribution.
Our contributions are the following:
• We introduce W2GAN, in which the generator recovers an optimal map in high-dimensional spaces.
• We show that the training dynamics of our generator is characterized as following a unique geodesic between the initial distribution and the target distribution.
Our analysis is based on interpreting the signal given by the discriminator at each update of the generator as a local update toward an optimal transport map.
While this can be applicable to other Wasserstein GAN models, with the W2GAN, this insight extends to a global description of the generator’s trajectory.
• We verify our proposed model on synthetic low-dimensional data, and show that it perform competitively on high-dimensional image data.
R ELATED W ORK
The intersection of GANs (Goodfellow et al., 2014) and OT theory was mainly initiated by the introduction of WGAN (Arjovsky et al., 2017), which proposed using the ﬁrst Wasserstein distance as an alternative to the f -divergence (Nowozin et al., 2016).
The objective of the discriminator in WGAN and its extensions (Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018) corresponds to a particular instance of the Kantorovitch problem, which is the main problem at stake in OT theory.
The Kantorovitch problem consists of ﬁnding an optimal plan for “transporting” one distribution onto another according to a predeﬁned point-wise cost function.
Borrowing these notions from OT theory has proved to be useful for improving both the understanding and performance of
GANs.
Salimans et al.
(2018) proposed replacing the discriminator with the Sinkhorn algorithm, which can compute any Wasserstein metric.
Alternatively, Sanjabi et al.
(2018) used the optimal transport map as a new way to train a GAN discriminator.
The optimal map has other important applications such as computing barycenters of distributions (Rabin et al., 2011; Bonneel et al., 2015).
Wasserstein barycenters may be used as a meaningful way to interpolate between probability measures (Cuturi & Doucet, 2014).
Nonetheless, computing the optimal map remains challenging in some situations.
Recently, regularized versions of the Kantorovitch problem were proposed to provide improved algorithms for computing the optimal map between (semi-)discrete distributions (Cuturi, 2013; Genevay et al., 2016).
Seguy et al.
(2018) developed a parametric approach to obtain a two-step procedure for computing an optimal map in the case of continuous distributions.
In all of these methods, dealing with high dimensional data is challenging.
One naturally suspects that a GAN-based approach of this problem would be efﬁcient, knowing the success of such models in high dimensions.
For learning an optimal map using GANs, Wasserstein metrics are of main interest, especially given the relationship between the discriminator and the optimal map (Lei et al., 2017).
In this context, however, the ﬁrst Wasserstein distance is not convenient as the gradient ﬂow of this metric is hard to solve (Ambrosio et al., 2008), and the relationship between the discriminator and the optimal map is neither unique, nor easily computed.
What misses is a unique and tangible characterization of the training dynamics of the generator.
Characterizing generator’s dynamics is a complex problem.
We need to understand the evolution of a probability measure through a signal given by a real function over distributions – the discriminator.
OT theory deals extensively with the characterization of paths in the space of probability measures.
Speciﬁcally, it allows one to relate probability measures and their evolution through gradient ﬂows (Villani, 2008; Ambrosio et al.
explored how to relate the generator’s evolution with paths in the space of probability measures described by the discriminator.
In the particular context of Wasserstein-based GANs, it is possible to link the generator’s evolution with the Wasserstein-1 geodesics, which unfortunately happen to be in a great number.
On the other hand, in the case of the second Wasserstein distance, the geodesics are unique and related to the optimal map in a simple way (Ambrosio et al., 2008).
B ACKGROUND
We state in this section some basic notions from OT theory (Villani, 2008; Santambrogio, 2010;
Ambrosio & Gigli, 2013) which will be necessary for the rest of the paper.
We provide a more comprehensive presentation in Appendix 8.4.
Optimal Transport problem and Wasserstein distances Computing Wasserstein distance and optimal transport map are two intrinsically related problems.
Given two probability measures µ
and ν on the euclidean space R d , an optimal map or a Monge map T is a function minimizing the Monge problem:
[Equation]
where A T is the set of all maps from R d to R d that respect the marginal ν , or more formally written as T # µ = ν 1 .
This objective involves a ﬁxed point-wise cost function c : R d × R d → R .
When the cost c is a power p of the euclidean distance, the value of (1) is called the p -th Wasserstein distance, and denoted W p ( µ, ν ) 2 .
We will assume this form for the cost function in the rest of the paper.
In this case, it can be shown that we recover the value of W p ( µ, ν ) with a relaxed formulation of (1), which is called the dual Kantorovich problem : [Equation] y
[Equation]
We will also denote as V ∗ ( φ, ψ ) the value (cid:82) x ∈X φ ( x ) dµ ( x ) +
(cid:82) y ∈Y ψ ( y ) dν
( y ) .
A pair ( φ, ψ ) maximizing 2 are called Kantorovitch potentials .
We will refer to the constraint in the deﬁnition of
A ∗ ( µ, ν ) as the c-inequality constraint or simply the inequality constraint .
In WGAN and its extensions, the discriminator corresponds to these Kantorovitch potentials computing W 1 .
More generally, it is possible to consider φ, ψ as discriminators computing any Wasserstein distance W p (Salimans et al., 2018;
Sanjabi et al., 2018).
On the other hand, the generator’s update rule is driven by the gradient of the discriminator, i.e. the Kantorovitch potentials.
In the particular case of of W 2 , we can derive a simple relationship between these potentials’ gradients and the Monge map T solving (1).
Relating Potentials to the Monge map An important property, which holds for W 2 , is that we can relate the dual Kantorovitch solution φ and the Monge map T (Brenier, 1991).
T is unique and determined by:
[Equation]
where Id is the identity.
A detailed and more generalized statement of this relationship is given by Proposition (2) in the Appendix.
In fact, there is also a converse statement: given a function
T : R d (cid:55)→ R d and a probability measure µ , T is a Monge map between µ and T # µ if T is the gradient of a strictly convex function.
That is, If T can be written as x − ∇ φ ( x ) for some real function φ one can rephrase this condition as (cid:107) x (cid:107) 22 2 − φ ( x ) being strictly convex.
Such a φ is said to be c -concave.
This relationship suggests that when the discriminator is optimizing the W 2 metric, the generator would be driven by the optimal map T .
Rigorously, optimizing (1) leads to W pp ( µ, ν )
W2GAN: THE MODEL
In this section, we introduce the W2GAN model in which the discriminator computes the second
Wasserstein distance W 2 .
Let us denote our discriminator ( φ, ψ ) , a target distribution P x and our generator G ( z ) where z ∼ P z .
Here z is not necessarily meant as the usual low-dimensional latent variable in GAN.
In all generality, z could have the same dimension as x (e.g. as in unsupervised domain translation applications).
In such a case, the next section will show that if G is initialized as the identity function, the training of our generator will compute a Monge map between P z and P x .
For optimization purposes, it is preferable to deal with a regularized version of (2).
We choose to use the L 2 regularization, but one could use other penalties (Cuturi, 2013;
Seguy et al., 2018; Blondel et al., 2017) 3 .
Hence our discriminator’s objective for approaching W 22 ( G # P z , P x ) is:
[Equation]
where ( . )
+ := max(0 , . )
and λ ineq is a scalar controlling the strength of the penalty.
We will refer to the penalty term by L ineq ( φ, ψ ) in the rest of the paper.
Our generator’s objective will then be
[Equation]
The training procedure of W2GAN is detailed in Algorithm 1.
In the next section we will prove that W2GAN’s generator can recover the Monge map between the initial generated distribution and the target distribution.
The experiment section will provide empirical evidence conﬁrming this and comparing it with other methods.
T HE GENERATOR RECOVERING A M ONGE MAP
The proof that our generator recovers a Monge map consists of characterizing the dynamic of the
W2GAN’s generator during training.
We observe that it follows a pre-determined trajectory: the unique W 2 geodesic between the initial distribution and the target distribution.
This result is stronger than only recovering the Monge map at the end of training, which will be a corollary to the analysis of the W 2 -geodesics detailed in the background section.
Hence, this extra piece of information is another interesting theoretical fact: one can uniquely characterize the dynamic of evolution of our generator during training.
The analysis of this evolution is divided in two part.
First we examine the direction of the update in the space of probability measure locally, for one generator update.
Second, we concatenate these local updates to obtain a global description of the trajectory.
The ﬁrst technical tool we need is the characterization of geodesics in the W 2 case.
Geodesics in the space of probability measures Recall that W 2 is a metric over distributions and thus makes the space over probability measures a metric space (Villani, 2008).
Studying properties of this space starts with the analysis of its geodesics.
That is, for any given distributions µ and ν , we look for paths described by µ t such that µ 0 = µ , µ
= ν and
[Equation]
It happens that such geodesics are unique (Villani, 2008).
Given the unique Monge map T between µ and ν , the only constant speed geodesic is given by µ t := T t # µ where T t = (1 − t ) Id + tT .
By (3), we can write T ( x ) = x − ∇ φ ( x ) for any Kantorovitch potential φ solving (2).
Hence
T t ( x ) = x − t ∇ φ ( x ) .
In particular, this guarantees that tφ ( x ) is a Kantorovitch potential solving (2) for µ and µ t =
T t # µ .
We now move to the analysis of our generator’s training dynamics.
We ﬁrst discuss the general update rule for any GAN model.
Local evolution of the generator of any GAN Let us consider the general objective of a GAN
[Equation]
We can also use the random variables notations G θ ( z ) ∼ G θ # P z =: µ θ and x ∼ P x .
Denoting θ t , θ t +1 the current parameter for the generator and the next one and α > 0 the hyper-parameter update, the above leads to the common update rule used by GAN methods:
[Equation]
where the last equality holds when one can control the norms of the gradients in order to invoke the Dominated Convergence theorem.
We reinterpret this update rule using the functional gradient analysis as in (Anonymous, 2019; Johnson & Zhang, 2018).
One may see that we obtain exactly the same update rule by differentiating the following objective at θ = θ t :
[Equation]
where F α ( x ) :
= x − α 2 ∇ x L 1 ( D, x ) .
Hence it is a completely equivalent formulation of the minimization objective for the generator.
Eqn.
(8) states that given the current generated distribution G θ t ( z ) , the next parameter θ will be chosen so that G θ ( z ) will be close to F α G θ t ( z ) .
Then if we want to locally understand how the generated distribution moves, we have to analyze this push-forward by F α .
Instead of looking at the space of parameters, we directly consider the space of probability measures.
Given the current µ θ t , we consider the multiple possible push-forwards of µ θ t , that is all the H # µ θ t for different functions H from R d to itself.
One may also see a push-forward as the random variable
H ( G θ t ( z )) .
As in (Anonymous, 2019), we associate a loss to any such function:
[Equation]
In particular, L ( Id ) is just the current loss of the generator.
We can take the derivative of L at identity to obtain the direction onto which the generated distribution should be updated, but now in the space of probability measures instead of the space of parameters.
This is done by computing
Gateaux differentials.
That is, we ﬁrst consider any direction h and compute the following directional derivative assuming bounds and regularity which allows us to switch the order of the expectation and the derivative:
[Equation]
Equation (10) is homogeneous in h and thus the gradient of L at identity is the function ∇L 1 ( D, x ) .
This is tantamount to saying that choosing h ( x ) to be −∇ x L 1 ( D, x ) will be locally the best choice for decreasing the loss of the generator.
We thus consider the update in the space of distributions that sends Id ( G θ t ( z ))
=
G θ t ( z ) onto H α
( G θ t ( z ))
where H α ( x ) := x − α 2 ∇ x L 1 ( D, x ) .
This is analogous to gradient descent in the space of probability measures: we considered the loss of the current distribution Id
( G θ t ( z )) and updated it in the most signiﬁcant direction given by H α .
Now, we can observe that H α
= F α .
Let us summarize what this means: given the current loss of the generated distribution µ θ t , the update rule in the ideal space of probability measures consists of taking the new distribution F α # µ θ t .
Then the update rule according to (8) in the space of parameters is only choosing θ t +1 so that the new generated distribution µ θ t +1 is as close to the ideal update F
α # µ θ t .
Links with a certain Monge map It is tempting to interpret the previous F α ( x ) = x − α 2 ∇ x L 1 ( D, x ) as an optimal transport map (or Monge map) in the Wasserstein-2 sense between the current distribution µ θ t and the distribution F α # µ θ t .
Indeed, the expression of F α is quite similar to the Monge map T given in proposition (3).
That is, T ( x ) = x − ∇ φ ( x ) where φ is a Kantorovitch potential maximizing (2).
However, there are still two apparent obstacles:
In the expression of F α the scalar α > 0 has no correspondence in the expression of T . 2.
How can we ensure that L 1 ( D, x ) is indeed a Kantorovitch potential,
i
e. is an optimum of (2) for µ = µ θ t and ν = F
α # µ θ t
The second question is answered in the background section: the requirement that L 1 ( D, . )
needs to be c -concave.
This does not hold for general GANs where the complexity of the discriminator function makes it impossible to decide whether (cid:107) x (cid:107) 22−
L 1 ( D, x ) is strictly convex.
In our case,
L 1 ( D, x ) =
φ ( x ) where φ is a Kantorovitch potential.
A Kantorovitch potential automatically satisﬁes such a condition, so the problem is solved.
For the ﬁrst concern, when α = 2 , F 2 is exactly the Monge map T from µ θ t to P x .
The characterization of W 2 geodesics in (5) shows that any F α for 0 (cid:54) α (cid:54) 2 is actually also a Monge map between µ θ t and F
α # µ θ t .
This solves the ﬁrst obstruction.
We also know that F α # µ θ t lies in the unique
Wasserstein 2 geodesic between µ θ t and F 2 # µ θ t =
P x .
We can summarize this by saying that an update of the current generated distribution µ θ t consists of taking a step of size α > 0 along the optimal transport map toward the distribution µ θ t − ∇L 1 ( D, G θ t )
# µ θ t = µ θ t − ∇ φ # µ θ t =
T # µ θ t =
P x .
In the experiment section, we empirically conﬁrm that the discriminator provides a signal along the optimal map.
From local to global dynamic of the generator We now want to extend the local description of the generator’s dynamics by showing that the subsequent generated distributions µ θ 0 , µ θ 1 , . . .
µ θ n all live on the W 2 geodesic joining µ θ 0 to P x .
Let us assume this is the case for a ﬁxed n > 0 .
As in the previous paragraph, we will assume the parametric update to match the update in the space of probability measures 4 .
From the previous paragraph, the update at time n in the space of probability measures leads to µ θ n +
= F α # µ θ n .
From the notations introduced when we discussed W 2 geodesics in (5), F α = T α 2 where T is a Monge map between µ θ n and P x .
By the description of such geodesics µ θ n +1 is on the geodesic between µ θ n and P x .
Now we use the uniqueness of displacement interpolation between geodesics (Villani, 2008) to obtain that this geodesic between µ θ n and P x is a restriction of the geodesic between µ θ 0 to P x .
As a result, the latter contains µ θ n +1 .
We conclude by induction that all the generated distributions progress toward P x on the expected
W 2 -geodesic.
In Figure 4 of the experiments section, we visualize our generator following this geodesic during training.
In particular, denoting G ∞ ( z ) the generator at convergence, we relate it to the optimal map T between µ θ 0 and P x :
[Equation]
For more information on the speed of convergence of the generated distribution, we conduct a similar analysis in the Appendix 8.2 relying on the notion of gradient ﬂows (Ambrosio et al., 2008).
Notice that our generator produces more than the Monge map – it provides a discrete version of the
W 2 -geodesic between the distributions as we observe it during training.
This is another problem of interest, see for instance the work of Seguy & Cuturi (2015).
In practice, our model is more complicated than WGAN and its extensions.
First because the discriminator in W2GAN involves two functions φ and ψ .
Second because the c -inequality constraint has a simpler formulation in the case of W 1 .
It is thus worth asking if we could have conducted the previous analysis in the Wasserstein-1 case.
The Wasserstein-1 case Our model shares a lot with GANs relying on the ﬁrst Wasserstein metric, and one might wonder why we do not use e.g WGAN, WGAN-GP or WGAN-LP as they achieve state-of-the art performance in generative modelling.
For the purpose of learning an optimal map, this section is meant to explain why we cannot theoretically rely on such models, at least by trying to adapt our analysis to the W 1 case.
Experimentally, however, as WGAN and its extensions rely on a similar objective as W2GAN, the seem to be following an optimal map – at least in some cases.
Let us enumerate what parts of the previous analysis does not apply to the case of W 1 :
The local analysis could actually be applied in a similar way.
Recall that local direction given to the generated distribution by the Wasserstein GAN discriminator is −∇ f ( . )
where f is a (non unique) Kantorovitch potential in the W 1 case.
It is true then that there exists an associated Monge map T from the current distribution toward the true distribution.
This relationship is T ( x ) (cid:107) T ( x ) (cid:107) =
−∇ f ( x ) (Brenier, 1991).
Hence, we can get the direction of an OT
map thanks to the knowledge of the dual optimal variable, but not its norm.
Thus, locally, the generator is updated toward the true distribution, and in the direction of the optimal map, but the magnitude of the update step is impossible to obtain.
• Going from a local to global analysis is much more difﬁcult in the case of W 1 .
Recall that in the case of W 2 , the main ingredient of the discussion was the uniqueness of geodesics and their nice analytic description.
In the case of W 1 , there are eventually inﬁnitely many geodesics.
A great description of this fact is available in (Bottou et al., 2018).
Hence, while locally the discriminator may be one among many Kantorovitch potentials, it is hard to decide whether globally the generator follows a geodesic.
Although this seems to be true as every local direction is happening on geodesics, it would remain impossible to decide which geodesic the generator is following.
It would be of interest to describe different kind of W 1 -geodesics and understand the ones that Wasserstein GANs’ generators are prone to follow.
• Essentially because of the previous discussion it is also really hard to solve the gradient ﬂow of the W 1 metric as we do in the Appendix 8.2 in the case of W 2 .
In conclusion, while we might have a locally nice description of GANs relying on W 1 , we cannot obtain a global description.
Low dimensional experiments still strongly suggest that their generators might recover a Monge map
at the end
of training
E XPERIMENTS 6. 2-D IMENSIONAL S YNTHETIC D ATA
We ﬁrst consider learning optimal maps between synthetic datasets in 2-dimensional space, in which we can visualize them easily 6 .
Each dataset is composed of two parts: X and Y , each composed of 2D data, and we verify that we can learn optimal maps from X to Y .
We consider three datasets (samples are shown in Figure 1-(a)).
(i) 4-Gaussians :
X and Y are mixtures of 4 Gaussians with equal mixture weights.
The mixture centers of Y is closer together than that of X .
(ii) Checkerboard :
X and Y are mixtures of uniform distributions over 2D squares, of size 4 and 5 respectively, with equal mixture weights.
The mixture centers of the two distributions form an alternating ”checkerboard” pattern.
(iii) 2-Spirals : X and Y are uniform distributions over spirals that are rotations of each other.
We also show in Figure 1-(b) optimal maps between data samples obtained by the optimal assignment algorithm, which we will refer to as Discrete-OT .
We use this as a benchmark for evaluating optimal maps obtained by various methods on these speciﬁc data samples.
We apply our proposed W2GAN model on these three datasets, where the generator takes input samples from X and and maps them to Y .
Note that we ﬁrst initialize the generator as an identity function 7 , in order to verify our theoretical analysis that the generator can recover globally an optimal map between its initial and target distributions.
As baselines, we compare with the following methods: (i) Barycentric-OT : a two-step algorithm for computing optimal maps between continuous or discrete distributions introduced by Seguy et al.
The algorithm is based on ﬁrst computing a regularized optimal transport plan, then estimating a parametric optimal map as a barycentric projection.
(ii) WGAN-GP (Gulrajani et al., 2017), and (iii) WGAN-LP (Petzka et al., 2018).
Both of these GAN-based models use the W 1 metric as the objective of the discriminator, but differ by the form of gradient penalty on the discriminator.
We similarly initialize their generators as identity functions.
We show in Figure 2 results of each of these models on the three 2D datasets.
When compared to the
Discrete-OT map in Figure 1-(b)
, we can see that our proposed W2GAN can successfully recover the Monge map in all three datasets.
In comparison, Barycentric-OT performs mostly very well, but we notice a clear collapse in the mapping in the case of 4-Gaussians.
WGAN-GP generally performs poorly, and we observe that training becomes especially unstable as G ( X ) approaches Y – sometimes diverging from a good solution.
This happens even with more training of the discriminator and smaller generator learning rates.
WGAN-LP, on the other hand, matches the performance of
[Figure]
The fact this optimal map seems match the Monge of the case is interesting according to our analysis in 5, which indeed indicates that locally the generator is driven on some optimal maps, but cannot predict on which W 1 geodesic the generator will globally evolve.
The experiments might suggest that WGAN and its extensions are more prone to choosing certain W 1 geodesics.
It clearly outperforms WGAN-GP, which is unstable according to the argument of Petzka et al.
(2018) that the gradient penalty in WGAN-GP can prevent the discriminator from converging.
We also verify our analysis 5 that the local direction given to the generator is Id − α 2 ∇ φ , which corresponds to a small step in the global direction Id − ∇ φ .
This direction is also supposed to be an optimal map T according to 3.
We can see in Figure 3 that the approximation of the Monge map by φ is quite reasonable.
This is especially interesting, given that φ , as well as ψ , are both real functions which are not explicitly trained to recover the Monge map.
Finally, Figure 4 shows the generator’s dynamics in checkerboard dataset.
We see that it evolves on the optimal map.
6. H IGH - DIMENSIONAL D ATA
Next, we move to the more challenging setting of learning optimal maps in high-dimensions.
We ﬁrst consider the task of mapping samples of a 28 × 28 multivariate Gaussian 8 to MNIST, which was originally proposed by Seguy et al.
(2018), and compare with their method.
We follow the exact experimental setting of Seguy et al.
We also use the same architecture for both W2GAN generator and their Barycentric-OT network for fair comparison 9 .
Figure 5 shows the generated samples (output of learned mappings) by our model and the Barycentric-OT model.
Qualitatively speaking, W2GAN seems to generate much better MNIST samples, which conﬁrms that it is a competitive method for estimating the Monge maps in high dimensions.
In this second set of experiments, we apply our model to the unsupervised domain adaptation task, which is a standard experimentation setting for evaluating large-scale OT maps (Courty et al., 2017b).
We also try to follow the same experimental settings of Seguy et al.
(2018), where the model learns to map across USPS and MNIST datasets.
In unsupervised domain adaptation, we have labels in one dataset (e.g. USPS) and our goal is to train a classiﬁer for the other dataset (e.g. MNIST) without having access to any labels in it.
An optimal map is assumed to learn a mapping in the image space that preserves, as much as possible, the digit identity.
Similar to Seguy et al.
(2018), we evaluate the accuracy the different models using a 1-nearest neighbour (1-NN) classiﬁer.
As a baseline, we train the classiﬁer using source (USPS/MNIST)
[Figure] [Figure]
images, and test it on target (MNIST/USPS) images (i e. identity mapping).
We compare this trivial mapping to the mapping obtained by both our W2GAN model and Barycentric-OT, where here the 1-NN classiﬁer is trained on domain transferred source digits.
We can see in Table 1 that our model achieves better results than Barycentric-OT.
We note that our baseline is slightly lower than that of Seguy et al.
(2018), possibly due to image-processing differences, and correspondingly, our implementation of Barycentric-OT also achieves slightly lower accuracy.
Even so, we can achieve higher accuracy with our model compared to the result reported in their paper.
C ONCLUSION A ND F UTURE W ORK
We believe this work offers a new perspective on GANs: a way to characterize the dynamics of the generator during training, and as a main application a way to compute a Monge map.
In W2GAN, the
[Figure] [Figure]
[Table]
generator recovers an optimal map between its initialized distribution and the target distribution.
To establish this, we connected the generator’s training procedure, and especially the signal it receives from the discriminator, with the Wasserstein-2 geodesics.
In the Appendix 8.1, we raise interesting questions about our model left for future works.
In particular, how to generalize this analysis of the generator dynamic during training to other GANs?
How to conﬁrm that asymptotically, when the term of regularization goes to 0 , the gradient of the regularized optimal potential φ for (4) does indeed recover the relationship 3 with the optimal map T solving (1)?
In the appendix 8.3 we explore the down-to-earth case of update in the space of parameters.
It would be also important to try relating the ideal geodesics in the space of probability measures with the actual projections in the space of parameters.
A PPENDIX 8. R EMAINING QUESTIONS AND FUTURE WORKS
In the following we detail questions raised by our model
• A crucial relationship used in our model is (22), relating an optimal map T solving the Monge problem (1) with the Kantorovitch potential φ solving the dual (2).
We recall that this link is T ( x ) = x −∇ φ ( x ) for any x in the support of the initial distribution.
In practice though, we get this potential by solving a regularized version of (2), for instance (4) or (27).
Assuming for simplicity we deal with (4), we face a dual problem which basically removes the hard constraint deﬁning (2) by including a penalty term L ineq in the main objective, the emphasis of which we control with the hyperparameter λ ineq .
Denoting φ λ ineq the corresponding solution -which may be proven to be unique up to translation- it seems to constitute a hard challenge to prove that φ λ ineq → φ as λ ineq → 0 .
The case of the entropy penalty, that is when we deﬁne the regularization L ineq ( φ, ψ ) :
= − E ( x,y ) ∼ µ × ν [exp( φ ( x )+ ψ ( y ) − c ( x,y ) λ ineq )] has been widely studied.
In a discrete setting, the convergence might be obtained as a result of Cominetti & San Mart ´ ın (1994), but their analysis does not straightforwardly scale to the case of absolutely continuous probability measures.
What we would ﬁnd more useful is to prove the convergence ∇ φ λ ineq → ∇ φ , as it would mean that our approximation of the optimal transport map T using relationship (3) is valid.
Notice that the entropy regularized dual also admits a primal formulation, which solution π λ ineq may be proven to converge in some sense to the unique transport map
π solving (19) (Carlier et al., 2017).
No similar results have been found about the dual variables for the moment.
Also, the case of other penalties as the L 2 one remain open problems.
• In a similar fashion, it would enrich our analysis to have bounds on the deviation from the ideal trajectory, the W 2 -geodesic, when we assume some errors of approximation for the generator and the discriminator.
In the same way, how far is the ﬁnal generated distribution from the Monge map in this more realistic case?
• Another crucial theoretical need for strengthening our analysis would be to deepen the parametric analysis 8.3 and try to understand the trajectory of the generated distribution in the parameter space more thoroughly.
Ideally, one would like to prove that this trajectory would be some projection of the W 2 -geodesic in the space of parameters.
• In practice, what are the new possibilities enabled by the analysis of the generator’s dynamic and the W2GAN model?
First, it would be of interest to characterize the dynamic of GANs relying of f -divergences in a similar manner.
In the Wasserstein-1 case, it is possible that knowing the generator is learning an optimal correspondence could be useful in a Domain Transfer situation, especially in high dimension where those models perform well.
In particular, listing the different W 1 -geodesics and characterizing the ones that are mainly followed by WGAN’s generators could be a great insight.
• On the other hand, as the W2GAN’s discriminator approximates the second Wasserstein distance, it enables tackling the challenge of computing Wasserstein barycenters of distributions (Cuturi & Doucet, 2014).
In fact, it seems likely that an adversarial and parametric method would perform well in the task of generating all Wasserstein interpolations between distributions in high-dimension, a difﬁcult problem addressed by Xie et al.
(2018) for instance.
8. C ONTINUOUS ANALYSIS OF THE GENERATOR ’ S EVOLUTION
We here strengthen the arguing that during training, the generated distribution µ θ is following the ”line” deﬁned the optimal transport map between its initialization µ θ 0 and the target distribution
P x .
As in the previous discussion, we assume the discriminator ( φ, ψ ) to compute exactly the squared second Wasserstein distance W 2 2 ( µ θ , P x ) , i.e we suppose it is trained inﬁnitely many times at each update of the generator, and we forget about the bias induced by the c-inequality constraint being encoded in the objective as a penalty (informally, we assume λ ineq = 0 ).
We again look at the evolution of the generated distribution in the case of ideal updates in the space of probability measures.
The difference in this paragraph is that we consider α → 0 , thus writing G ( t, Z ) as a time-dependent random variable where G is in the space of L 2 functions and t is a ﬁctive time variable.
In this case we may equivalently work on the corresponding generated probability distribution of interest, which we denote as µ t := µ θ t .
Then our generated distribution evolves according to the gradient ﬂow
[Equation]
One would need to introduce the deﬁnition of gradient ﬂow in the space of probability measures, in particular the notion of real functions’ gradient with respect to probability measures and velocities of time-dependent measures in order to fully express the meaning of the above.
We refer to Ambrosio et al.
for a comprehensive overview.
We can reﬁne (12) thanks to Ambrosio et al.
and obtain:
[Equation]
where T t is the unique optimal transport map, Solving (1) between µ t and P x .
According to the gradient ﬂow (13), locally, we recover that the generated distribution evolves towards P x by following the Wasserstein-2 geodesic.
In fact, this is a global behaviour, from Ambrosio et al.
Theorem 1.
Denote T the optimal transport map between µ 0 and P x .
Then we have that the gradient ﬂow solving (12) is uniquely determined:
[Equation]
[Equation]
Hence a consequence (also from Ambrosio et al.
(2008)) is that the generated distribution evolves exponentially fast towards P x :
Corollary 1.
[Equation]
The evolution in (14) suggests that the generated distribution follows the Wasserstein-2 geodesic between µ 0 and P x .
That means the training dynamic of the generator ”draws” the optimal transport map between µ 0 and P x .
At the end of training, the generator G ( ∞ , . ) provides a certain optimal transport map.
For each z , the ’arrows’ joining G (0 , z ) and G ( ∞ , z ) together constitute the optimal map.
Figure 7 in the Appendix helps visualizing this analysis.
8. T HE PARAMETRIC CASE
We now turn to the more realistic case of a parametrized generator G θ ( z ) , i.e we will analyse the update in the space of parameters.
Lui et al.
(2017) already made the conjecture that a generator trained with the second Wasserstein distance would have its parameters updated towards the direction of an optimal transport.
Let us prove their statement in the case of our model.
In the following, J f ( u ) denotes the Jacobian of a function f at point u .
Let G θ ( z ) be our parametrized generator, which we assume to admit derivatives w.r.t both the parameters θ and z .
Recall the notation µ θ := P G θ ( z ) =
G θ # P z where P z is the measure known beforehand on the latent variable z .
We consider the context of alternating gradient descent with learning rate α > 0 for the generator.
We naturally associate the ﬁctive time variable t, so that we consider the discrete update equation:
[Equation]
Proposition 1.
At each generator update, we assume the discriminator φ, ψ to achieve sup φ,ψ V ∗ ( φ, ψ, µ θ , P x ) =
W 22 ( µ θ , P x ) where V ∗ ( φ, ψ, µ θ , P x ) :=
(cid:82) R d φ ( x ) dP ( x ) + (cid:82)
R d ψ ( x ) dP x is the value function of the dual Kantorovitch problem (2).
Then (16) admits a tractable form as:
[Equation]
Moreover [Equation] the gradient ascent dynamics of the generator are linked to the optimal transport map T solving (1) between µ θ and P x as:
[Equation]
In the above we introduced the notation G ( θ, z ) :
= G θ ( z ) to make clear where the derivatives are taken.
Equation (17) is close to the one conjectured in Lui et al.
Following their approach, for the sake of clarity, we consider the case where the latent variable z is a constant.
Then we read (17) as
[Equation]
As similarly observed in Lui et al.
(2017), if we ignore the possibly signiﬁcant effect of
J G ( .,z ) ( θ t ) J TG ( .,z ) ( θ t ) , the term T ( G ( θ t , z ))
− G ( θ t , z ) enforces G to be updated on the optimal transport plan joining µ θ and P x .
One thus should be aware of the strong sensitivity of the model to any error of approximation.
The dynamics of the generator is easily perturbed by biased updates.
Let us denote the generated distribution at time t =
k by µ k .
Let us also introduce g k as the Wasserstein geodesics between µ k and P x .
Imagine, for instance, that there exists k > 0 , such that µ k +1 is not on the optimal path between µ k and P x .
Then, from now on, the generator will evolve ideally on the Optimal Transport map joining µ k +1 and P x , and thus will no longer evolve in the previous geodesic.
More succinctly, g k +1 (cid:54) =
g k .
In some sense, the learning dynamics of the generator is a Markov process in that it ”forgets” the previous updates.
A wrong update leads to a generator that no longer evolves exactly on the optimal map between the initial µ 0 and the target P x .
It would certainly be of interest to have a mathematical framework that accounts for this variance in the evolution of the generator, but this is beyond the scope of this paper.
8. R ESULTS IN O PTIMAL T RANSPORT THEORY
We develop a bit more the materials of the background section, introducing the same notions with more details in the same order.
Monge Problem Optimal Transport (OT) theory (Villani, 2008; Santambrogio, 2010; Ambrosio & Gigli, 2013) introduces a natural quantity to distinguish two probability measures.
Given two probability measures µ
and ν on the euclidean space R d , the original Monge problem is to ﬁnd a map T that “transports” the µ distribution on ν , and minimizes the cost of the transport, which is point-wise deﬁned by a ﬁxed cost function c : R d × R d → R .
The value c ( x, y ) can be seen as the cost for transporting a unit from x to y .
The problem is summarized by:
[Equation] [Equation]
where A T is the set of all maps from R d to R d that respect the marginal ν , more formally written as T # µ = ν .
The map T # µ is called the push-forward measure of µ and is deﬁned as T # µ ( A ) =
µ ( T − 1 ( A )) for any Borel set A ∈ R d .
Figure 6 in the Appendix provides an illustration of the shape of an optimal transport map T solving (1).
Kantorovitch relaxation Unfortunately, problem (1) often does not admit a solution as A T might be empty 10 .
To circumvent this issue, one considers the so-called Kantorovitch relaxation of this problem:
[Equation]
(19) where A ( µ, ν ) is the set of joint distributions whose ﬁrst and second marginals are equal to µ and ν respectively.
The joint π is called the transport plan between µ and ν .
If π is deterministic — speciﬁcally, for any x , there is a unique y such that π ( x, y ) > 0 — then it is also a transport map as deﬁned in the Monge problem.
It sufﬁces to deﬁne such a map T ( x ) by the only y respecting π ( x, y ) > 0 .
However, we could instead consider π to be a “one-to-many” transport plan: for each x , there might be several y such that π ( x, y ) > 0 .
While Monge’s development had the problem of non-existence of the transport map, the Kantorovitch relaxation A ( µ, ν ) is never empty.
In particular, it always contains the independent joint distribution µ × ν .
In many situations, under mild assumptions on the cost function c (semi-lower continuity and bounded from below), there always exists a minimizer of (19), so that the inﬁmum might be replaced by a minimum (Ambrosio & Gigli, 2013).
We distinguish between a transport map T achieving the minimum in (1) and a transport plan π minimizing (19).
Note that for most applications, we are more interested in obtaining an approximation of the optimal transport map than a transport plan.
Dual of Kantorovitch One could also work with the dual of problem (19), which is proven to lead to the same value V c ( µ, ν ) .
We write this dual as:
[Equation]
We will also denote as V ∗ ( φ, ψ ) the value (cid:82) x ∈X φ ( x ) dµ ( x ) +
(cid:82) y ∈Y ψ ( y ) dν
( y ) .
A pair ( φ, ψ ) maximizing V ∗ are called Kantorovitch potentials .
We will refer to the constraint in the deﬁnition of (2) which forces the sum of the potentials φ and ψ to be upper-bounded by the cost as the c-inequality constraint or simply the inequality constraint .
Wasserstein distances When the cost c is a distance, V c ( µ, ν ) exactly matches the famous
W 1 ( µ, ν ) , which is called the ﬁrst Wasserstein distance between µ and ν .
When the cost c is a distance to a power of some positive integer p , V c ( µ, ν ) 1 /p is denoted W p ( µ, ν ) and is commonly called the p th Wasserstein distance.
An important result is that these W p distances are actually respecting the axioms of a distance over distributions (Villani, 2008).
For the sake of simplicity, we will abusively extend the term “Wasserstein distance” to any V c , for any cost function c, even though not all choices of c lead to V c being a metric over probability distributions.
Computing Wasserstein distances The discriminator of the Wasserstein GAN (WGAN) (Arjovsky et al., 2017) computes the dual formulation (2), with ν being the “true” distribution X (deﬁned by training examples) and µ being the “fake” distribution G ( Z ) (deﬁned by samples drawn from the generator).
In practice, the inequality constraint on potentials φ, ψ may be enforced by the addition of a constraint-violation penalty term to the objective (Gulrajani et al., 2017).
Although introduced for other reasons, Cuturi (2013) proposed an efﬁcient penalized version of (19) whose resulting optimization problem is called the entropic-regularized optimal transport problem.
We shall only provide its dual form:
[Equation]
(21) where L ineq penalizes ( φ, ψ ) when they violate the inequality constraint:
[Equation]
Other penalties were theoretically explored by Blondel et al.
(2017), including the widely used
L 2 -penalty (Seguy et al., 2018):
[Equation]
where ( . )
+ := max(0 , . ) .
Objective (21) was shown to asymptotically recover the value of (2)
when λ ineq → 0 (Carlier et al., 2017).
Due to its suitability for training parametric models such as neural network, we exploit this optimization objective to compute Wasserstein distances W p .
As it is provable (Ambrosio & Gigli, 2013) that the ‘differential’ of W p ( ., ν ) with respect to its ﬁrst variable is − φ (where φ is a Kantorovitch potential), we may conclude that a GAN framework where the discriminator accurately computes (2) through (21) provides a means of training a generative model.
The details of such a framework are developed in the next section.
Links between the Kantorovitch potentials and the Monge map We end this section by exploring the relationship between optimal potentials and optimal transport plans and maps.
The goal is to recover an approximation of the solution of (1), namely an optimal transport map, given optimal solutions of (2) (i.e. Kantorovitch potentials).
Getting an approximate transport plan might be done by considering a speciﬁc relationship that holds between the solutions of the dual and the primal in regularized versions of the Optimal Transport (Cuturi, 2013; Xie et al., 2018).
As for an optimal transport map, Seguy et al.
(2018) provides a two-step procedure, that is, they ﬁrst compute a regularized transport plan, and then calculate an approximate transport map by doing a barycentric projection of the plan.
In our case, we begin by the observation that there is no distinction between an optimal plan and its corresponding map in the case of the Wasserstein distances W p with p > 1 .
In addition, the primal-dual relationship is straightforward.
Getting an optimal transport map solving (1) is a by-product of computing Kantorovitch potentials by solving (2).
We summarize this famous statement (Brenier, 1991):
Proposition 2.
Fix p (cid:62) 2 and the cost c ( x, y ) :
= (cid:107) x − y
[Equation]
p .
Then there is one unique optimal transport plan π solving (19) .
It is deterministic, that is it corresponds to an optimal transport map
T solving (1) : π = ( Id × T ) .
Also, the Kantorovitch potentials φ, ψ are unique up to a translation and the following relation holds:
[Equation]
[Equation]
Finally, we recall a useful result for the converse question: given a function T from R d to itself and a probability µ , is T a Monge map between µ and T # µ ?
In the case of the second Wasserstein distance, Brenier’s polarization theorem (Brenier, 1991) says this is true if T is the gradient of a strictly convex function.
If T may be written x − ∇ φ ( x ) for some real function φ one can rephrase this condition as (cid:107) x (cid:107) 22 2 − φ ( x ) is strictly convex.
Such a φ is said to be c-concave.
In particular, any
Kantorovitch potential φ is c-concave.
Geodesics in the space of probability measures Recall that W 2 is a metric over distributions and thus makes the space over probability measures a metric space.
Studying properties of this space starts with the analysis of its geodesics.
That is, for any given distributions µ and ν , we look for constant speed paths described by µ t such that µ 0 = µ , µ 1 = ν
and
[Equation]
It happens that such geodesics are unique (Villani, 2008).
Given a Monge map T between µ and ν , the only constant speed geodesic is given by µ t := T t # µ where T t = (1 − t ) Id + tT .
It is a remarkable fact, compared with the case of the ﬁrst Wasserstein distance, where the geodesics are eventually in inﬁnite number.
This is one obstruction to conduct directly our analysis in this case, and this is one reason we appeal to W 2 instead of W 1 .
By 3, we can write T ( x ) = x − ∇ φ ( x ) for any Kantorovitch potential φ solving (2).
Hence
T t ( x ) = x − t ∇ φ ( x ) .
In particular, this guarantees that tφ ( x ) is a Kantorovitch potential solving (2) for µ and µ t =
T t # µ .
8. P ROOFS
Proof of proposition 2.
This result is a particular case of a well-known correspondence between
Kantorovitch potentials and optimal transport map.
In fact, when the cost c is such that c ( x, y ) =
h ( x − y ) and h is strictly convex, one has the existence and uniqueness of an optimal Monge map T solving (1) and a speciﬁc relationship with the Kantorovith potential φ, ψ solving (2) Villani (2008);
Santambrogio (2017):
[Equation]
Fix p (cid:62) 2 and consider the case of the p-Wasserstein distance.
Then c ( x, y )
=
h ( x − y ) where h ( x ) :
= 1 p (cid:107) x
(cid:107) p .
A norm is always convex by triangle inequality, and any x → x p is strictly convex and increasing on R + , so the previous result provides the uniqueness of the optimal transport map T .
It only remains to invert the gradient of h. A quick calculation gives:
[Equation]
as we supposed the norm to be the euclidean one.
Plugging this into the ﬁrst expression, we obtain the desired result.
Proof of proposition 1.
Proving the ﬁrst part of proposition 2 is exactly similar as the way it is done for the Wasserstein-1 case in Arjovsky et al.
We recall that the main ingredients of the proof is ﬁrst to convey an envelop theorem to obtain that
[Equation]
and second to use a dominated convergence argument to invert the expectation and the gradient operator in the right-hand side of the above.
For the second part of the proof [Equation] we rewrite the gradient ascent equation (16) for the generator parameter θ :
[Equation]
Therefore, for a ﬁx latent variable z 0 ,
[Equation]
Then by ﬁrst order Taylor expansion:
[Equation]
We conclude using the hypothesis that φ is maximizing the dual of Kantorovitch and is related to an optimal map T through (3).
8. E XPLOITING A PRIMAL - DUAL RELATIONSHIP AND INTERPOLATING THE CONSTRAINT TO IMPROVE THE MODEL
In the following, we explain two important modiﬁcations of the objective functions for the discriminator inspired by optimal transport theory.
In order to unify notations, we consider the two random variables X
and Y to be discriminated by φ, ψ .
In the GAN context, one may think of X as the generated distribution G ( Z ) and Y as the true distributions.
In both cases the discriminator objective is divided into two parts.
First, a main objective L OT ( φ, ψ, X, Y ) :
=
E ( φ ( X ) + ψ ( Y )) which corresponds to the dual of the Kantorovitch problem.
Second, the inequality constraint
L ineq ( φ, ψ, X, Y ) :
= − λ ineq E [( φ ( X ) + ψ ( Y ) − (cid:107) X
[Equation]
Y (cid:107) [Equation] ) 2+ ] .
One could prefer an other choice of regularization such as the entropy penalty.
Hence the overall objective for the discriminator:
[Equation]
The ﬁrst idea is to take advantage of relation (3), which we know to be true between an optimal potential φ and an optimal map T .
In fact, Optimal transport theory asserts that the inequality constraint should be exactly saturated where there is some transport (Villani, 2008):
Theorem 2.
Consider any lower-semicontinuous cost function c and a optimal transport plan for (19) , and Kantorovitch potentials φ, ψ for (2) .
Then,
[Equation]
In our case, that is when the cost is a p
th power of the euclidean distance, we know from Propositionthan an optimal transport plan is actually an optimal transport map T , and we dispose of a relationship with the corresponding Kantorovitch potential φ .
Hence an immediate consequence of the above for the case of the square of the euclidean distance is:
Corollary 2.
For the Kantorovitch problem (1) related to c ( x, y ) :
=
[Equation]
, that is the computation of the second Wasserstein distance, given Kantorovitch potentials φ, ψ :
[Equation]
[Equation]
Thus we suggest enforcing our discriminator to abide by relation (24).
Notice that the previous corollary admits an exact symmetric relationship involving the gradient of ψ .
This is done by adding the following penalty during training:
[Equation]
We call it the c-equality penalty term as it tries to enforce the dual functions to saturate to c-inequality constraint at the right locations.
Hence the overall objective of the discriminator:
[Equation]
Remark 1.
It is interesting to bridge this objective with the one used in GAN relying on the ﬁrst
Wasserstein distance.
In WGAN-GP, the discriminator is asked to have gradient exactly equal to 1 .
Translating that into the Optimal Transport theory, it is tantamount to have potentials φ, ψ saturating the c-inequality.
That is, this gradient penalty term is similar to our L eq .
Importantly though, Petzka et al.
(2018) raised the fact that it is not a valid practice according to theory to ask the gradient norm of the discriminator to be 1 everywhere.
Instead, WGAN-LP is a model where the gradient’s norm is enforced to be less than 1, which is then translated into Optimal Transport perspective as potentials φ, ψ respecting the c-inequality.
Their penalty term is thus similar to our L ineq .
Our model takes advantage of the two forms.
The difference is that our L eq is completely justiﬁed -at least because it enforces a relationship which is true at optimum- thanks to the theory existing in the W 2 case.
A second modiﬁcation concerns the way we encode the penalty enforcing the inequality constraint to be respected by φ, ψ .
In fact, we modify L ineq in order to bring it closer to the theory.
Looking at the deﬁnition of A ∗ ( µ, ν ) , the set of constraint of the dual of Kantorovitch problem (2), we see that the c-inequality constraint should be respected point wise everywhere in the ambient space where the distributions are deﬁned.
Instead, the entropy regularization or the L 2 one only enforce it for pairs x, y that live in the supports of the two distributions.
We want to reduce this bias by somehow enforcing the inequality on a broader set.
It is sufﬁcient that the potentials respect the inequality constraint point wise on a convex compact set Ω containing the support of the two distributions.
Hence we suggest enforcing it on the convex envelop of the two supports: Ω :=
Conv(supp( µ ) (cid:83) supp( ν )).
To do so, we deﬁne two i.i.d random variables ˜ X and ˜ Y which follow the same law as (cid:15)X + (1 − (cid:15) )
Y where (cid:15) ∼ U ([0 , 1]) .
Hence the overall objective for φ, ψ :
[Equation]
Remark 2.
This interpolation idea has already been used in GANs relying on the ﬁrst Wasserstein distance, such as WGAN-GP and WGAN-LP in their gradient penalty.
This practice was more motivated by better results.
Here we provided a theoretical argument in favor of such practice.
On the other hand, models trying to broaden GANs to higher order Wasserstein distances and/or to compute
Optimal transport map (Sanjabi et al., 2018; Seguy et al., 2018; Salimans et al., 2018) in a similar manner only enforced the constraint on the support of the distributions.
Remark 3.
We motivated the deﬁnition of ˜ X :=
(cid:15)X + (1 − (cid:15) )
Y so that it basically takes value in the convex envelop of X and Y.
But in all generality, it is wrong that only convex interpolation between two points allows to recover all Conv(supp( µ ) (cid:83) supp( ν )).
By Carathodory theorem, one would need to interpolate at most between d+1 points to recover all of it.
Although not impossible at all, we wanted to keep the model simple and not depend on the dimension of the latent space R d .
8. P ARAMETERIZING φ AND ψ FOR STABILITY
An intuitive way of parameterizing φ and ψ is to simply replace φ and ψ with two neural networks of the same architecture.
A potential downside we found practically is that this parameterization tends to be unstable.
An alternative reparameterization is to replace ψ ( Y ) with − φ ( Y ) + (cid:15) ( Y ) where both φ and (cid:15) are neural networks.
This reparameterization and the property that
[Equation]
Algorithm 1 W2GAN with D = ( φ, ψ ) and L 2 -regularization.
Require: latent space Z and true distribution X from which we have an available sample procedure.
Require: λ eq , λ ineq , λ (cid:15) , n critic , bD, p .
Require:
Initial critic parameters w 0 , initial generator parameters θ 0 .
while θ has not converged do
Initialize generator and discriminator losses L D , L G to 0.
for t = 1 , ..., n critic do for i = 1 , ..., bD do
Sample real data x, x (cid:48) ∼ P X , latent variable y, y (cid:48) ∼ P G ( Z ) and (cid:15) 1 , (cid:15) 2 ∼ U ([0 , 1]) .
Update L D according to 28 and optionally 29.
end for
Update ( φ, ψ ) with respect to L D .
end for
L G ← L OT .
Update G with respect to L G .
end while yields an additional regularizer [Equation] 29) which we found particularly useful in the high dimensional setting.
8. A LGORITHMS
In the algorithm 1, one can choose to use the equality constraint as an additional constraint for the discriminator.
One can also uses the interpolation method, or any method of sampling in 8.6.
8. I MPLEMENTATION D ETAILS AND A RCHITECTURE
Below we describe the implementation details of our experiments.
8.9. 2D S ETTING
For the 2D synthetic data experiments, the learning rate used for Barycentric-OT is 0 .
005 (although we did not notice that the learning rate inﬂuenced the solution quality signiﬁcantly).
For the GAN experiments, learning rates were chosen from the set { 0 . 00001 , 0 . 0005 , 0 . 00005 , 0 . 00001 } .
For the
Wasserstein-based GANs, the number of discriminator updates per generator update is chosen from the set { 5 , 10 , 20 } .
This is set to 1 in the Jensen-Shannon based GAN by default.
λ gp for both
WGAN-LP and WGAN-LP is set to 10 , following their conventions.
For W2-OT and W2GAN, we set λ eq = λ ineq
= 200 .To enforce that G 0 ( z ) =
z , we parameterize G by G ( z )
=
H ( z ) + z , where
H ( z ) is initialized to be close to 0 .
H is parameterized by 4 fully connected hidden layers of size 128, with ReLU activations and batch norm in between the layers, and 1 fully connected ﬁnal layer.
φ and ψ are each parameterized by 2 fully connected layers with ReLU activations in between, andfully connected ﬁnal layer.
8.9. M ULTIVARIATE G AUSSIAN TO MNIST S ETTING
In this experiment, MNIST images are kept at their original size of 28 × 28 and pixel values are re-scaled to be in [ − 1 , 1] .
We note that in the high dimensional setting, better training stability and image quality is achieved by using both L eq and L (cid:15) which complement L ineq in enforcing the constraint.
For multivariate Gaussian to MNIST experiments, we use the same architecture as (Seguy et al., 2018) for Barycentric-OT.
We searched over λ ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 , 2 , 5 } , and use the
ADAM optimizer with learning rate = 0 . 0002 and β 1 = 0 .
5 , β 2 = 0 .
999 for both the dual variables and the mapping.
We run the experiment with a batch size of 64 for 200 , 000 iterations for each phase.
For W2GAN, to enforce that G 0 ( z ) is close to identity, and that G ( z ) ∈ [ − 1 , 1] we reparameterize
G by G ( z ) = 2 · H ( z ) + z clip where z clip is z clipped to be in [ − 1 , 1] and tanh is used as the ﬁnal activation layer of H .
we use the same architecture as barycentric-OT with the addition of
BatchNorm in between the layers of the generator.
Speciﬁcally, the architecture for our model is
FC( 28 ∗ 28 → 1024 )-BN-RELU-FC( 1024 → 1024 )-BN-RELU-FC( 1024 → 28 ∗ 28 )-Tanh for H in the generator, and FC( 28 ∗ 28 → 1024 )-RELU-FC( 1024 → 1024 )-RELU-FC( 1024 → 1 ) for both φ and (cid:15) in the discriminator.
We set λ ineq = λ eq = λ (cid:15) =
10 and use the ADAM optimizer with learning rate = 0 .
0001 and β 1 = 0 .
5 , β 2 = 0 .
999 for both the generator and the discriminator.
We ran the experiment for 100 , 000 iterations with a batch size of 64 .
8.9. U NSUPERVISED D OMAIN A DAPTATION S ETTING
The USPS dataset consists of 16 × 16 grayscale images of digits, with signiﬁcantly less training and testing data (7291 train and 2007 test images).
The MNIST digits are rescaled to 16 ×to match the USPS digits, and the grayscale pixels in both datasets are scaled to be in [ − 1 , 1] .
For this set of experiments, we use the same architecture as (Seguy et al., 2018) for BarycentricOT, and choose the best model between using entropy regularization and L2 regularization and λ ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 , 2 , 5 } .
We use the ADAM optimizer with learning rate = 0 . 0002 and β 1 = 0 .
5 , β 2 = 0 .
999 for both the dual variables and the mapping.
We ran the experiment for 20 ,iterations for each phase with a batch size of 1024 .
For W2GAN, to ensure that G 0 ( z ) is identity and that G ( z ) ∈ [ − 1 , 1] , we reparameterize G by
G ( z ) = 2 · H ( z ) + z where the last activation layer of G is tanh.
we use similar architecture as above for our model.
Speciﬁcally, the architecture is FC( 16 ∗ 16 → 200 )-BN-RELU-FC( 200 → 500 )-BNRELU-FC( 500 → 16 ∗ 16 )-Tanh for H in the generator, and FC( 16 ∗ 16 → 200 )-RELU-FC( 200 → 500 )-RELU-FC( 500 → 1 ) for both φ and (cid:15) in the discriminator.
We use the same optimizers and hyperparameters as the mutivariate Gaussian to MNIST experiment above.
8. A DDITIONAL FIGURES
[Figure]