Adversarial training for predictive tasks: theoretical analysis and limitations in the deterministic case
Abstract
To train a deep neural network to mimic the outcomes of processing sequences, a version of Conditional Generalized Adversarial Network (CGAN) can be used.
It has been observed by others that CGAN can help to improve the results even for deterministic sequences, where only one output is associated with the processing of an input.
Surprisingly, our CGAN-based tests on some deterministic geophysical processing sequences did not produce a real improvement compared to the use of an L p loss; we here propose a ﬁrst theoretical explanation why.
Our analysis goes from the non-deterministic case to the deterministic one.
It led us to develop an adversarial way to train a content loss that gave better results on our data.
Introduction
We consider the problem of mimicking a complicated processing sequence by learning some representation of the joint probability density function (pdf) that couples the outcomes of the sequence to its inputs.
In geophysics, our ﬁeld of application, processing sequences are usually based on workﬂows that represent a combination of algorithms and user-provided information to achieve a given task.
Wave equation and signal processing are classical components of the algorithms, and geological priors are often part of user-provided information [1].
Several geophysical processing sequences aim at removing some undesired very structured events in the geophysical data [1], like the “ghost” events illustrated in Fig.
Learning an efﬁcient representation that mimics such sequences can bring value, for example to take the best of various existing workﬂows, increase turnaround or obtain a processing guide.
Deep Neural Networks (DNNs) provide a ﬂexible tool to parameterize a function that predicts outcomes from inputs.
Many explorations have recently been done using DNNs to mimic geophysical processing sequences, see for instance Refs.
To train a DNN to predict outcomes from inputs, we may consider methods inspired from the
Generative Adversarial Network (GAN) framework [7], in particular Conditional GAN (CGAN) [8, 9].
Indeed, CGAN can deal with joint pdfs (contrary to the original GAN formulation that deals only with single parameter pdfs), the originality being that the discriminator becomes conditioned by the input data [10, 4, 3, 6].
However, in the common context of a deterministic processing sequence, where only one outcome is generated when the sequence is applied to an input [2, 3, 4, 5, 6], using a simple L p norm based loss for the training usually gives good results [11].
So, can CGAN be pertinent even in the deterministic case?
It has been observed that combining CGAN with an L p loss may help to improve the results further, see e.g. Ref.
[10] for natural image processing and Refs.
[4, 6] for geophysical processing.
Surprisingly, our Wasserstein CGAN-based trainings [12] on deterministic geophysical processing sequences, like the “deghosting” (or ghost removal) sequence [13], did not help to produce a real improvement in our tests compared to the use of an L p loss, see Fig.
In this paper, we propose a theoretical analysis of this aspect.
First, we remind why L p losses should perform well in the deterministic prediction case.
Then, we point out from the Wasserstein point of view what CGAN should bring compared to an L p loss, taking the opportunity to discuss the Wasserstein CGAN (W-CGAN) foundations.
Our analysis gives a ﬁrst explanation of why CGAN may perform more poorly than expected, and also leads to a proposal of an adversarial way to train a content loss that we call “Content CGAN” (C-CGAN); it gave better results on our data as illustrated in Fig.
For completeness, we start all our theoretical considerations from the non-deterministic prediction case, where multiple outcomes related to one input are possible and then take the deterministic limit.
difference between C-CGAN and L p becomes quite smaller.
Notations
X denotes the input data (image) space and Y the output data (image) space.
P Y,X = P X P Y
| X denotes the joint pdf associated to the (possibly non-deterministic) processing sequence we wish to mimic.
P X is the marginal pdf that describes the distribution of the input data; realizations of the random variable X ∼ P X are denoted by ˜ X ∈ X .
P Y
| X is the conditional pdf that describes the outcomes of the processing sequence related to a given input; realizations of the random variable Y ∼ P Y | X are denoted by ˜ Y ∈ Y .
G Zθ : X → Y represents a prediction function parameterized by a model θ , here a DNN, where the latent space random variable Z ∼ P Z gives the ﬂexibility to produce multiple outcomes related to a given input (for the non-deterministic prediction case).
θ is to be optimized so that the Z -realizations of G Zθ ( X ) tend to mimic the realizations of Y ∼ P Y | X .
E denotes the expectation over a speciﬁed random variable.
The deterministic prediction limit can be taken considering both:
• The “empirical” joint pdf, for instance, for P Y,X ( ˜ Y , ˜ X ) → 1 N D (cid:80) N D i =1 δ ( ˜ Y − ˜ Y i ) δ ( ˜ X − ˜ X i ) .
{ ˜ X i , ˜ Y i ; i = 1 ..N D } denotes a set of input and output data realizations.
• G Zθ independent of Z , so that a unique outcome is predicted by the DNN for each input.
Which processing sequences are suitable for the use of an L p loss?
L p -based losses are deﬁned for p ≥ 1 by
[Equation]
(1) where the output image space L p norm is deﬁned by
[Equation]
[Equation]
Y here represents the real L p (Ω) space (functions with integrable moments of order p ). Each ˜ Y ∈ Y represents an image indexed by the positions y in a “pixels space” Ω that is measurable for the measure µ ( y ) . Ω usually represents the (discrete) pixels grid space and µ the counting measure. However, our considerations generalize to continuous spaces Ω taking the Lebesgue measure for µ .
Training aims to minimize C p ( G Zθ ) , eq.
(1), with respect to θ .
As C p measures a similarity between one realization of Y and one realization of G Zθ ( X ) , the trained prediction function G Zθ will tend to become independent of Z and output some “average” of all outcomes related to one input data.
Indeed, we can easily compute the optimum for p = 2 : G Zθ ( X ) ≈ E Y ∼ P Y | X Y , and for p = 1 :
G Zθ ( X ) ≈ M Y ∼ P Y
| X Y where M denotes the median (see Ref. [11] section 6.2.1.2).
In the non-deterministic prediction case, if the multiple outcomes are related to structured events, training with an L p loss is obviously not suitable as it would tend to produce blurry predictions (due to the “averaging”).
However, if the multiple outcomes are related to zero-“average” noise, an L p loss is suitable and would even tend to produce denoised predictions.
Of course, an L p loss is also suitable to the deterministic prediction case; a denoising effect can still occur if each of the single outcomes are affected by zero-“average” noise.
Note that p = 1 (the median) is more robust to outliers than p = 2 but harder to train.
p = 1 . 5 has been chosen in Fig.
1, representing a current compromise in geophysics [1].
This being posed, what could CGAN bring compared to an L p loss in the deterministic case?
Let us ﬁrst discuss the CGAN foundations in the general non-deterministic case, from the Wasserstein point of view and complementarily to Ref.
[12], and then analyze the deterministic limit.
Wasserstein CGAN for processing sequences 4.1 Non-deterministic prediction case
For notational purposes, let us consider the “parameterized” conditional pdf P ( par ) Y | X whose realizations correspond to the ones of G Zθ ( X ) .
In other words, for any function D , P ( par ) Y | X is deﬁned so that
[Equation]
where we keep the superscript ( par ) to make explicit which random variable is related to the parameterized pdf.
Note that imposing a Gaussian parameterization to P ( par ) Y | X and using cross-entropy (XE) as a loss leads to eq.
(1), as recalled in Appendix A.
This allows us to understand from another point of view the conclusions of §3: L p losses are suited when the outcomes follow Gaussian statistics, and are not suited when the Gaussian assumption is too simplistic (as often with structured outcomes).
We now wish to deﬁne a similarity measure between the two joint pdfs P Y,X and P ( par ) Y,X = P X P ( par ) Y | X , without having to consider any parameterization like the Gaussian one.
XE is not adapted and
Wasserstein distances [14] seem like a natural choice.
We propose the following Wasserstein-based formulation suited to joint pdfs with same marginal ( p ≥ 1 and r ≥ 1 ):
[Equation]
P Y | X and P ( par ) Y
| X are considered as single parameter pdfs for each realization of X , and W L p represents a r -Wasserstein distance between P Y | X and P ( par ) Y | X for any L p -norm choice in the output image space [14].
The inﬁmum is taken over all joint pdfs Π XY,Y ( par ) with marginals P Y | X and P ( par ) Y | X , X being considered as a parameter.
Then, the expectation over all realizations of X is taken to obtain JW L p , representing a distance between the joint pdfs P Y,X and P ( par ) Y,X , as demonstrated in Appendix B. Switching to the dual formulation and taking r = 1 allows to simplify the second line of eq.
(4) into the Kantorovitch-Rubinstein (KR) formulation (see [14] section 1.2 and [15])
[Equation]
Note that the “discrimator” D X : Y → R is parameterized by the input data and is constrained to be 1-Lipschitz for the L p -norm, i.e. || D X ||
Lip Lp ≤ 1 . The corresponding “Lipschitz norm” is deﬁned by || D X
|| Lip Lp = sup ˜ Y (cid:54) =
˜ Y (2) | D X (˜ Y )
− D X (˜ Y (2) )
| ||
˜ Y − ˜ Y (2) ||
Lp , ∀ ( ˜ Y , ˜ Y (2) ) ∈
Y × Y [14, 15].
As demonstrated in Appendix C, if D X ( ˜ Y ) is differentiable with respect to ˜ Y , the Lipschitz norm simpliﬁes into
[Equation]
with 1 /p + 1 /q
= 1 .
Inserting eq.
(3) into eq.
(5) to come back to G Zθ , we ﬁnally obtain the following more tractable form
[Equation]
Eqs.
(6) and (7) provide an adversarial training framework for predictive tasks:
JW L p contains a supremum principle on D X , but the result is to be minimized with respect to θ during the training.
The discriminator’s parameterization by the input data ˜ X ∈ X establishes the relation with CGAN [8, 9] and its Wasserstein counterpart [12], that is known.
However, we underline some formal points that were not discussed in previous works to our knowledge:
• The discriminator’s dependency on the input data can possibly be strong and discontinuous, leading in the general case to one different discriminator per input data in eq.
Of course, this would be inefﬁcient numerically and is usually not necessary (especially when images lie in low dimensional manifolds, do not vary rapidly).
However, the considerations in this section lead to some clariﬁcation on the possibility of using a different discriminator architecture per group of input data with similar properties within W-CGAN if needed.
• Eq.
(6) provides the generalization to L p (cid:54)
=2 -norms to the derivative-based Lipschitz constraint of Ref.
• We established that JW L p represents a distance between two joint pdfs with same marginal.
• Appendix D establishes the link with the scheme obtained starting from a Wasserstein distance between joint pdfs without same marginals [17].
We mentioned in §3 that training with an L p loss, eq.
(1), compares one realization of P Y | X to one
Z -realization of G Z θ ( X ) , which leads to “averaging”.
Training with JW L p , eq.
(7), compares all realizations of P Y | X to all Z -realizations of G Zθ ( X ) , i.e. G Zθ can learn to mimic the realizations of P Y | X and no “averaging” occurs.
This fundamental difference would help to produce unblurred results in the non-deterministic case, when multiple outcomes are related to structured events.
In the deterministic prediction case, however, what JW L p would bring compared to an L p loss is unclear.
This is what we discuss now.
4.2 Advantages of Wasserstein CGAN in the deterministic case?
To take the deterministic prediction limit, we use the method mentioned in §2.
Eq.
(7) becomes
[Equation]
where ˜ X i and ˜ Y
i denote input and output data pairs.
We use a discriminator architecture form
[Equation]
where F ( ˜ X i , ˜ Y )
∈ Y is parameterized by a convolutional DNN without striding and an additional last “layer” simply represents a sum over the “pixels”.
The last layer is equivalent to a global average pooling [9, 18] and has the advantage to make the discriminator DNN model independent of the size of the data (i.e. the model can be used for any data size).
This architecture allows for an interpretation of what the discriminator learns since F ( ˜ X i , ˜ Y ) lies in the output image space.
Also, in our tests on geophysical processing tasks, it led to the highest Wasserstein distance estimates (or supremum values) when training the discriminator.
So, it is the architecture we choose.
Just to gain insight, we ﬁrst consider a linear parameterization F ( ˜ X i , ˜ Y )
( y ) =
α ( ˜ X i )( y ) ˜
Y ( y ) , where α is a function of ˜ X i parameterized by a DNN.
Inserting this in eqs.
(6) and (8), we obtain[-
-]
[Equation]
In the deterministic prediction limit, the L p -based loss deﬁned by eqs.
(1) and (2) becomes
[Equation]
Compared to C 1 , i.e. eq.
(11) with p = 1 , we observe that (cid:100)
JW L p adds learnt positive weights | α ( ˜ X i ) | with unit L q norm.
In other words, with a linear parameterization for F in the deterministic case, W-CGAN “only” adds automatic learning of optimal data-dependent variance-like weights compared to the L 1 loss.
In this simple case, these weights can be demonstrated to lead to 2 : (cid:100) JW L p =
( C p ) 1 /p , i.e. to a (cid:100) JW L p that is equivalent to the L p -based loss. The takaway is that W-CGAN, eq. (8), should “at least” learn an L p -based loss or reweighting.
What about more involved parameterizations for F in eq. (9), using for instance a convolutional DNN and non-linear activations?
This will produce more involved transformations of ˜ Y
i and G θ ( ˜ X i ) than a simple reweighting.
Indeed, ˜ Y
i
→ F ( ˜ X i , ˜ Y i ) and G θ ( ˜ X i ) → F ( ˜ X
i , G θ ( ˜ X i )) would then correspond to a postprocessing of the outputs.
The supremum principle in eq. (8) allows to learn the postprocessing that makes JW L p the most sensitive to the differences between the prediction and the output data, i.e. that should concentrate on the less matched events.
In conﬁgurations where adding such a postprocessing would not affect the relative “positions” of most of the minimums in the loss valley, the main effect should be to improve the training convergence and deal better with the amplitude and noise present in the output data.
These situations should tend to occur amongst others when the postprocessings do not dramatically affect the gross data amplitudes hierarchy.
This is a ﬁrst element to interpret when W-CGAN trainings on deterministic geophysical processing sequences may not help to produce a real improvement. The case of Fig.
2 possibly falls in this category, that will be further discussed in §4.4.
Another element is that the method contains free parameters.
Two of these are related to the Lipschitz constraint: q (or p ) in eq.
(6), for which q ≈ 1 represented a good value, and a weight to impose the constraint using for instance the method of Ref.
Also, like in Refs.
[10, 4, 6], we observed JW L p has to be combined to an L p (cid:48) loss to give correct results, thus an additional weight is needed.
We chose p (cid:48) = 1 . 5 and tuned the weight so that JW L p and L 1 . 5 losses contribute equally, to obtain the results in Fig.
As these hyper-parameters are data dependent, it may explain why W-CGAN did not produce a systematic improvement in our tests.
The question of tuning the parameters the best for any kind of data is important but goes beyond the scope of this paper and is left for a future study.
4.3 Content CGAN: An adversarial way to train a content loss
Another difﬁculty with W-CGAN is that it is not feasible to resolve exactly the supremum principle in JW L p , eq.
(8), at each iteration.
This can lead to slowness in the training and possibly sometimes to “localized” unstabilities, that may contribute to the explanation of some poor results.
We propose to tackle a part of this speciﬁc problem by a heuristic reformulation of eq.
Note that the linearized case result of §4.2 can equivalently be recovered by ﬁrstly imposing the following form to F
[Equation]
(12) and then do the linearized approximation (remember footnote 1 and beware that the argument of the sign does not depend on ˜ Y but on ˜ Y i , which is important for the Lipschitz norm, eq.
Keeping this form for any non-linear parameterization of F ( ˜ X i , ˜ Y ) and inserting eq.
(12) in eq.
(8), we obtain
[Equation]
where D ˜ X i is deﬁned through eqs.
(9) and (12).
Eq.
(13) looks like an L 1 -based content loss [10] that would adversarially be trained, simultaneously with the G θ training.
A good content loss should tend to maximize the differences between a prediction that has been “postprocessed” (through the DNN
F ) and the corresponding similarly postprocessed output data.
This is achieved by the supremum principle in eq.
(13), where the Lipschitz constraint provides robustness (to avoid singularities...).
This heuristical reasoning leads to our “Content CGAN” (C-CGAN) loss.
Among other advantages, the C-CGAN loss always remains positive, even if the supremum principle is not well resolved at some iteration, whereas the W-CGAN loss, eq.
(8), might not.
4.4 DNN architectures and results
In the results presented in Fig.
2, the DNN inputs a ghosted image ˜ X and predicts a deghosted image
G θ ( ˜ X ) .
The prediction function G θ architecture is Unet inspired [21].
The F function architecture, that deﬁnes the discriminator through eq.
(9), is Denet inspired [22].
For the ˜ X -dependency of F , we found it sufﬁcient to concatenate ˜ X to the ﬁrst Denet layer; however, this certainly would deserve a speciﬁc study for the reason underlined in §3.
As mentioned in §4.2, we used q ≈ 1 in the Lipschitz norm, eq.
(6), and combined W-CGAN (or C-CGAN) to the L 1 . 5 loss, so that both contribute equally.
Fig.
2 shows a result at 40 epochs, before full convergence (150 epochs).
C-CGAN gave better results than CGAN or L 1 . 5 only, on our data.
However, the main beneﬁt of C-CGAN is here to accelerate the training.
At full convergence we observed the differences between C-CGAN and L 1 . 5 become quite smaller, possibly for the reason outlined in §4.2.
Fig.
3 proposes a way to demystify what the discriminator has learnt and interpret the interest of
C-CGAN.
Firstly through F ( ˜ X i , ˜ Y i )( y ) , deﬁned in the output image space Y . The ﬁgure shows that
C-CGAN learns to concentrate on the most important events, i.e. around the ghosts; this is satisfying and contributes to explain why C-GCAN achieves better deghosting more rapidly.
Secondly, we consider so-called “adjoint-input” ∂LOSS ( G ) ∂G ( y ) (cid:12) (cid:12) G
= G θ ( ˜ X i ) , which is back-propagated in G θ to compute how to update θ [23].
The adjoint-input is also deﬁned in the Y space and allows for a visualization of the areas where the events should be better predicted after the update.
Fig.
3 shows that the L 1 . 5 loss adjoint-input tends to “put the weight” on all areas, regardless of their relative importance, thus not to concentrate more speciﬁcally on the ghost areas.
The W-CGAN adjoint-input also tends to concentrate on many areas, contributing to explain why it brought no improvement in
Fig.
The texture of the latter adjoint-input may seem atypical; we veriﬁed that the optimization of eq. (8) and of G θ converged without unstability, but further analysis regarding the hyper-parameters mentioned in §4.2 is on the way.
The C-CGAN adjoint-input, however, learned to concentrate more speciﬁcally around the ghost areas, contributing to explain why it converges more rapidly towards an acceptable solution in Fig.
Note, however, that the C-CGAN adjoint-input does not exhibit a very strong change in the amplitudes hierarchy compared to the L 1 . 5 adjoint-input, contributing to explain that the main effect of C-CGAN would here be to improve the training convergence (remind §4.2).
Conclusion and future work
We proposed a theoretical analysis of the CGAN framework for predictive tasks.
We took the opportunity to establish the CGAN foundations from the Wasserstein point of view, and pointed out what CGAN should bring compared to an L p loss in the deterministic prediction case.
We discussed that W-CGAN may perform more poorly than expected when the corresponding data-space “postprocessings” would not affect the relative “positions” of most of the minimums in the loss valley (for instance when they do not dramatically affect the gross data amplitudes hierarchy), or due to a difﬁculty with automatically tuning the corresponding hyper-parameters.
Another difﬁculty is that the W-CGAN loss represents a distance only if the supremum principle is perfectly resolved numerically; our C-CGAN formalism helps to keep positivity and gave better results on our data.
This ﬁrst analysis is still to be conﬁrmed by further studies.
It is certainly data dependent (geophysical data being speciﬁc, with very structured and continuous events).
Among others, understanding “physically” how to tune the CGAN hyperparameters for any kind of data is important in the deterministic as well as in the non-deterministic prediction cases; this will be a future study.
Broader Impact
Learning an efﬁcient representation that mimics involved processing sequences can bring value in a general industrial context, not only in geophysics.
The goal can be to take the best of various existing workﬂows, increase turnaround or obtain a processing guide. Acknowledgments and Disclosure of Funding
The authors are grateful to CGG and Lundin for the permission to publish this work.
The authors are indebted to Nicolas Salaun, Samuel Gray, Thibaut Allemand, Gilles Lambaré, Mathieu Chambefort and Stephan Clémençon for enlightening discussions and collaboration.
All funding of this work by CGG.
Appendix A Cross-entropy and Gaussian
Training can aim at optimizing a parameterized joint pdf P ( par ) Y,X = P X P ( par ) Y | X to mimic P Y,X the best from a given similarity measure or loss point of view.
Cross-entropy represents a common loss choice to minimize, Ref. [11] section 6.2.1.2:
[Equation]
Dealing with a very general parameterization for P ( par ) Y | X is unfeasible when Y has a large dimensionality.
However, a simple generalized Gaussian parameterization may be considered (the normalization factor is implicit), see Ref. [24] section 4.3:
[Equation]
When eq.
(15) is inserted into eq.
(14) and the terms that do not contribute to the G θ optimization are omitted, we obtain eq.
Note that “standard-deviation-like” weights σ ( y ) , with ∞ > 1 /σ ( y ) > 0 , can be added to the distance to control the width of the Gaussian, i.e. || . || L p → || 1 σ .
||
L p .
But they do not affect considerations on the maximum-likelihood of the Gaussian.
B Wasserstein distance between two joint pdfs with same marginal
We consider pdfs P iY,X (indexed by i ) with same marginal P X , i.e. P iY,X = P iY | X P X .
[Equation]
(16) where the indicator function constrains the support in X of P iY | X ( ˜ Y | ˜ X ) to be the same as the support of P X .
We here use the notations of §4.1.
W L p ( P 1 Y | X , P 2 Y | X ) , eq.
(4), represents a Wasserstein distance between the conditional pdfs P 1 Y | X and P 2 Y | X for each “parameter” X .
Knowing this, we check if
[Equation]
represents a distance between the joint pdfs P 1 Y,X and P 2 Y,X , i.e. if it satisﬁes the symmetry and separation properties, and the triangle inequality.
JW L p obviously satisﬁes the symmetry property: JW L p ( P 1 Y,X , P 2 Y,X ) =
JW L p ( P 2 Y,X , P 1 Y,X ) .
It also satisﬁes the separation property
[Equation]
The only subtlety lies in the ﬁrst implication of the ﬁrst line and the ﬁrst implication of the second line; they are true because P 1 Y,X , P 2 Y | X and P X all have the same support as mentioned above after eq.
Finally, let us check the triangle inequality.
As W L p represents a distance, we have
W L p ( P 1 Y | X , P 3 Y | X ) ≤ W L p ( P 1 Y | X , P 2 Y | X ) + W L p ( P 2 Y | X , P 3 Y | X ) .
Taking the P X expected value straightforwardly leads to
[Equation]
Thus JW L p deﬁnes a distance between two joint pdfs with same marginal.
C Lipschitz norm dual representation
We consider a real Banach space Y indexed by the positions y in a set Ω that is measurable for the measure µ ( y ) .
We denote by || .
||
d the corresponding norm, not necessarilly an L p norm. Considering a function D : Y → R , the Lipschitz semi-norm is deﬁned by
[Equation] Identifying constant functions, we obtain a norm; this is implicit in the article, hence the use of “Lipschitz norm” throughout the text. Imposing Lipschitzity through eq.
(18) is unpractical numerically, as a global property of D is considered.
We would like to turn this global property into a local neighborhood one. Taking ˜
Y (2) = ˜ Y + ˜Υ with ˜Υ ∈ Y and choosing a D ( ˜ Y ) that is derivable with respect to ˜ Y , the Lipschitz norm can be rewritten
[Equation]
where the differential is deﬁned through
[Equation]
In the following, Y (cid:48)
=
L ( Y , R ) denotes the topological dual of Y , i.e. the ensemble of the continuous linear functions from Y into R ( (cid:48) being the symbol of the topological dual).
Note that the differential takes the form in eq. (20) when the Riesz representation theorem ([19] chapter IV) can be invoked in Y (cid:48) . We introduce the function Φ ∂D
[Equation]
∈ Y (cid:48) that is “represented” by ∂D (˜ Y ) [Equation]
∂ ˜ Y , ˜Υ (cid:17) Ω . By deﬁnition of the “dual norm” ([19] chapter I, [20] chapter IV), we have [Equation]
d [Equation]
In other terms, if the Riesz representation theorem can be invoked in Y (cid:48) ([19] chapter IV), the dual norm can be “represented” by a norm || . || d (cid:48) on ∂D (˜ Y )
∂ ˜ Y .
Combining these results with eq. (19) gives: If a Riesz representation theorem exists for Y (cid:48) : || D || Lip d
[Equation]
sup ˜
[Equation] Eq. (21) is true for any norm in ﬁnite dimensional Banach spaces, that is the case when Ω represents a ﬁnite pixels grid and µ ( y ) the counting measure. Then, a straightforward choice for the norm d is the L p norm, eq.
(2), for which we have ([19] chapter IV)
[Equation] with 1 /p + 1 /q = 1 , p ≥ 1 .
For completeness, we introduced “standard-deviation-like” weights σ ( y ) , that must satisfy ∞ > 1 /σ ( y ) > (cid:15) where (cid:15) > 0 , even if in our applications σ ( y ) = 1 is taken. Inserting eq.
(22) in eq.
(21) leads to
[Equation]
with 1 /p
[Equation] [Equation]
q
[Equation]
p [Equation] [Equation] So, if Y represents a ﬁnite dimensional L p (Ω) space, the “naturally” associated Lipschitz norm is deﬁned by eq.
(23) (even if all norms are equivalent in ﬁnite dimensional spaces, the “naturally” associated dual norm may lead to better numerical convergence properties...).
What if Y represents an inﬁnite dimensional L p (Ω) space (that is the case for instance when Ω represents a continuous set and µ ( y ) the Lebesgue measure)?
Eqs.
(22)-(23) remain valid but only for ∞ > p ≥ 1 . Indeed, no Riesz representation theorem then exists for Y (cid:48) when Y = L ∞ (Ω) .
D General formulation of the Wasserstein distance between two joint pdfs
We here consider two joint pdfs P Y,X =
P X P Y | X and P (2) Y,
X = P (2) X P (2) Y
| X that do not necessarilly have the same marginals, i.e. P X (cid:54) =
P (2) X in the general case.
We consider the Y × X space to be a Banach space and denote the corresponding norm by || ( ˜ Y , ˜ X ) − ( ˜ Y (2) , ˜ X
|| , ∀ (( ˜ Y , ˜ X ) , ( ˜ Y (2) , ˜ X (2) )
) ∈ ( Y × X ) ×
( Y × X ) .
A r -Wasserstein distance between P Y,X and P (2) Y,X can be deﬁned by ( r ≥ 1 ) [17]
[Equation]
where the inﬁmum is taken over all joint pdfs Π ( Y,X ) , ( Y (2) ,X (2) ) with marginals P Y,X and P (2) Y,X .
Switching to the dual formulation for r = 1 , in the same spirit than in §4.1, leads to
JW
[Equation]
[Equation]
[Equation]
) where the discrimator D : Y × X → R is constrained to be 1-Lipschitz for the Lipschitz norm:
[Equation]
[Equation]
Now, we consider P Y,X and P (2) Y,X that have the same marginal, i.e. P X
= P (2) X .
Eq.
(25) becomes JW 2( P Y,X , P (2) Y,X ) → [Equation] In order to establish a relation between eq.
(7), JW L p , and eq.
(27), JW 2 , we specify a form for the norm on ( X, Y ) .
We consider Y to be the Banach space used throughout this article (i.e. equipped with the L p -norm) and X to be a Banach space (possibly indexed in another space than Ω ) equipped with a || 1 σ .
||
L s norm that includes a “standard-deviation-like” weight like in eq.
We pose
[Equation]
With a similar reasonment than in Appendix C and choosing a D ( ˜ Y , ˜ X ) that is derivable with respect to ˜ Y and ˜ X , one can prove that the Lipschitz norm can be represented by
[Equation]
with 1 /p
+ 1 /q = 1 and 1 /s + 1 /t
=
Taking the limit σ → 0 and denoting equivalently D
( ˜ Y , ˜ X ) or D ˜ X ( ˜ Y ) , eq.
(29) reduces to eq.
Thus, the JW L p of §4.1 can be considered as a limiting case of the JW 2 of this section.
Indeed, taking σ → 0 aims at imposing ˜ X
→ ˜ X (2) in eq.