D EMYSTIFYING MMD GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs.
As our main theoretical contribution, we show that the natural estimator for maximum mean discrepancies yields unbiased gradient estimates, which is essential in providing a good signal to the generator.
We discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramér GAN critic.
Being an intergal probability metric, the
MMD beneﬁts from training strategies recently developed for Wasserstein GANs.
In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance.
I NTRODUCTION
Generative Adversarial Networks (GANs; Goodfellow et al., 2014) provide a powerful method for general-purpose generative modeling of datasets.
Given examples from some distribution, a GAN attempts to learn a generator function, which maps from some ﬁxed noise distribution to samples that attempt to mimic a reference or target distribution.
The generator is trained to trick a discriminator , or critic , which tries to distinguish between generated and target samples.
This alternative to standard maximum likelihood approaches for training generative models has brought about a rush of interest over the past several years.
Likelihoods do not necessarily correspond well to sample quality (Theis et al., 2016), and GAN-type objectives focus much more on producing plausible samples, as illustrated particularly directly by Danihelka et al.
This class of models has recently led to many impressive examples of image generation (e.g. Huang et al., 2017a;b; Jin et al., 2017; Zhu et al., 2017).
GANs are, however, notoriously tricky to train (Salimans et al., 2016).
This might be understood in terms of the discriminator class: Goodfellow et al.
(2014) showed that, when the discriminator is trained to optimality among a rich enough function class, the generator network attempts to minimize the Jensen-Shannon divergence between the generator and target distributions.
This result has been extended to general f -divergences by Nowozin et al.
According to Arjovsky & Bottou (2017), however, it is likely that both the GAN and model probability measures are supported on manifolds within a larger space, as occurs for the set of images in the space of possible pixel values.
These manifolds might not intersect at all, or at best intersect on sets of measure zero.
In this case, the Jensen-Shannon divergence is constant, and the KL and reverse-KL divergences are inﬁnite, meaning they provide no useful gradient for the generator to follow.
This helps to explain some of the instability of GAN training.
The lack of sensitivity to distance, meaning that nearby but non-overlapping regions of high probability mass are not considered similar, is a long-recognised problem for KL divergence-based discrepancy measures (e.g. Gneiting & Raftery, 2007, Section 4.2).
It is natural to address this problem using Integral Probability Metrics (IPMs; Müller, 1997): these measure the distance between probaility measures via the largest discrepancy in expectation over a class of “well behaved” witness functions.
Thus, IPMs are able to signal proximity in the probability mass of the generator and reference distributions.
Arjovsky et al.
(2017) proposed to use the Wasserstein distance between distributions as the discriminator, which is an integral probability metric constructed from the witness class of 1-Lipschitz functions.
To implement the Wasserstein critic, weight clipping of the distriminator network was initially used to enforce k -Lipschitz smoothness of the discriminator.
Gulrajani et al.
(2017) improved on this result by directly constraining the gradient of the discriminator network at points between the generator and reference samples.
This new Wasserstein GAN implementation, called
WGAN-GP, is more stable and easier to train.
A second integral probability metric used in GANs is the maximum mean discrepancy (MMD), for which the witness function class is a unit ball in a reproducing kernel Hilbert space (RKHS).
Generative adversarial models based on minimizing the MMD were ﬁrst considered by Li et al.
and Dziugaite et al.
These works optimized a generator to minimize the MMD with a ﬁxed kernel, either using a generic kernel on image pixels or by modeling autoencoder representations instead of images directly.
Sutherland et al.
(2017) instead minimized the statistical power of an
MMD-based test with a ﬁxed kernel.
Such approaches struggle with complex natural images, where pixel distances are of little value, and ﬁxed representations can easily be tricked, as in the adversarial examples of Szegedy et al.
Adversarial training of the MMD loss is thus the obvious choice.
Here the kernel MMD is deﬁned on the output of a convolutional network, which is trained adversarially.
Recent notable work has made use of the IPM representation of the MMD to employ the same witness function regularisation strategies as Arjovsky et al.
Gulrajani et al.
(2017), effectively corresponding to an additional constraint on the MMD function class.
Without such constraints, the convolutional features are unstable and difﬁcult to train (Sutherland et al., 2017).
Li et al.
(2017b) used the weight clipping strategy of Arjovsky et al., with additional constraints to ensure that the kernel distribution embeddings remained injective.
In light of the observations by Gulrajani et al.
, however, we use a gradient constraint on the MMD witness function in the present work (see Sections 2.1 and 2.2).
Bellemare et al.
(2017)’s method, the Cramér GAN, also used the gradient constraint strategy of Gulrajani et al.
in their discriminator network.
As we discuss in Section 2.3, the Cramér GAN discriminator is related to the energy distance, which is an instance of the MMD (Sejdinovic et al., 2013), and which can therefore use a gradient constraint on the witness function.
Note, however, that there are important differences between the Cramér GAN critic and the energy distance, which make it more akin to the optimization of a scoring rule: we provide further details in Appendix A. Weight clipping and gradient constraints are not the only approaches possible: variance features (Mroueh et al., 2017) and constraints (Mroueh & Sercu, 2017) can work, as can other optimization strategies (Berthelot et al., 2017; Li et al., 2017a).
Given that both the Wasserstein distance and the MMD are integral probability metrics, it is of interest to consider how they differ when used in GAN training.
Bellemare et al.
(2017) pointed out a particular ﬂaw of the Wasserstein distance as an objective: that gradients of both the duality-based estimator and of the alternative estimator of Genevay et al.
(2017) are biased.
Optimization over biased gradients can lead to incorrect parameter values, even in expectation; Bellemare et al.
provide an explicit example where this occurs.
As our main theoretical contribution, we show in Sectionthat the natural maximum mean discrepancy estimator has unbiased gradients, and therefore does not suffer from this problem.
There are additional empirical advantages of the MMD-GAN over the Wasserstein GAN.
Certainly we would not expect the MMD on its own to perform well on raw image data, since these data lie on a low dimensional manifold embedded in a much higher dimensional pixel space.
Once the images are mapped through appropriately trained convolutional layers, however, they can follow a much simpler distribution with broader support across the mapped domain: a phenomenon also observed in autoencoders (Bengio et al., 2013).
In this setting, the MMD with characteristic kernels (Sriperumbudur et al., 2010) shows strong discriminative peformance between distributions.
To achieve comparable performance, a purely convolutional WGAN requires many more layers in the critic.
In our experiments (Section 5), we ﬁnd that MMD GANs achieve the same generator performance as WGAN-GPs with smaller discriminator networks, resulting in GANs with fewer parameters and easier training.
Thus, the MMD-GAN discriminator can be understood as a hybrid model that plays to the strengths of both the initial convolutional mappings and the kernel layer that sits on top.
L OSSES AND WITNESS FUNCTIONS
We begin with a review of the MMD and relate it to the loss functions used by other GAN variants.
Through its interpretation as an integral probability metric, we show that the gradient penalty of
Gulrajani et al.
(2017) applies to the MMD GAN.
2. M AXIMUM M EAN D ISCREPANCY AND W ITNESS F UNCTIONS
We consider a random variable X with probability measure P , which we associate with the generator, and a second random variable Y with probability measure Q , which we associate with the reference sample that we wish to learn.
Our goal is to measure the distance from P to Q using samples drawn independently from each distribution.
The maximum mean discrepancy is a metric on probability measures (Gretton et al., 2012), which falls within the family of integral probability metrics (Müller, 1997); this family includes the Wasserstein and Kolmogorov metrics, but not for instance the KL or χ 2 divergences.
Integral probability metrics make use of a class of witness functions to distinguish between P and Q , choosing the function with the largest discrepancy in expectation over P, Q ,
[Equation]
The particular witness function class F determines the probability metric: the Wasserstein-1 metric is deﬁned using the 1-Lipschitz functions, and the Kolmogorov metric from the functions of bounded variation 1 .
The loss functions of many other GAN variants also fall into this class (e.g. Mroueh et al., 2017; Mroueh & Sercu, 2017; Berthelot et al., 2017).
In this work, our witness function class F will be the unit ball in a reproducing kernel Hilbert space
H , with positive deﬁnite kernel k ( x, x (cid:48) ) .
The key property of a reproducing kernel Hilbert space is the reproducing property: for all f ∈ H , f ( x ) = (cid:104) f, k ( x, · ) (cid:105) H .
We deﬁne the mean embedding of the probability measure P as the element µ P ∈ H such that E P f
( X ) =
(cid:104) f, µ P (cid:105) H ; it is given by µ P = E X ∼ P k ( · , X ) .
The maximum mean discrepancy (MMD) is deﬁned as (1) with F the unit ball in H :
[Equation]
The witness function f ∗ that attains the supremum has a straightforward expression (Gretton et al., 2012, Section 2.3):
[Equation]
Given samples X
= { x i } mi =1 drawn i.i.d.
from P , and Y = { y j } nj =1 drawn i.i.d.
from Q , the empirical witness function is
[Equation]
and an unbiased estimator of the squared MMD is (Gretton et al., 2012, Lemma 6)
[Equation]
When the kernel is characteristic (Sriperumbudur et al., 2010; 2011), the embedding µ P is injective (i.e., associated uniquely with P ).
Perhaps the best-known characteristic kernel is the exponentiated quadratic, also known as the Gaussian RBF, kernel,
[Equation]
Both the kernel and its derivatives decay exponentially, however, causing signiﬁcant problems in high dimensions, and especially when used in gradient-based representation learning.
The rational quadratic kernel
[Equation]
with α > 0 corresponds to a scaled mixture of exponentiated quadratic kernels, with a Gamma( α, 1) prior on the inverse lengthscale (Rasmussen & Williams, 2006, Section 4.2).
This kernel will be the mainstay of our experiments, as its tail behaviour is much superior to that of the exponentiated quadratic kernel; it is also characteristic.
2. W ITNESS FUNCTION AND GRADIENT PENALTIES
The MMD has been a popular choice for the role of a critic in a GAN.
This idea was proposed simultaneously by Dziugaite et al.
(2015) and Li et al.
(2015), with numerous recent follow-up works (Sutherland et al., 2017; Liu, 2017;
Li et al., 2017b; Bellemare et al., 2017).
As a key strategy in these recent works, the MMD of (4) is not computed directly on the samples; rather, the samples ﬁrst pass through a mapping function h , generally a convolutional network.
Note that we can think of this either as the MMD with kernel k on features h ( x ) , or simply as the MMD with kernel κ ( x, y )
=
k ( h ( x ) , h ( y )) .
The challenge is to learn the features h so as to maximize the MMD, without causing the critic to collapse to a trivial answer early in training.
Bearing in mind that the MMD is an integral probability metric, strategies developed for training the Wasserstein GAN critic can be directly adopted for training the MMD critic.
Li et al.
(2017b) employed the weight clipping approach of Arjovsky et al.
(2017), though they motivated it from different considerations.
Gulrajani et al.
(2017) found a number of issues with weight clipping, however: it oversimpliﬁes the loss functions given standard architectures, the gradient decays exponentially as we move up the network, and it seems to require the use of slower optimizers such as
RMSProp rather than standard approaches such as Adam (Kingma & Ba, 2015).
It thus seems preferable to adopt Gulrajani et al.
’s proposal of regularising the critic witness (3) by constraining its gradient norm to be nearly 1 along randomly chosen convex combinations of generator and reference points, αx i + (1 − α )
y j for α ∼ Uniform(0 , 1) .
This was motivated by the observation that the Wasserstein witness satisﬁes this property (their Lemma 1), but perhaps its main beneﬁt is one of regularization: if the critic function becomes too ﬂat anywhere between the samples, the generator cannot easily follow its gradient.
We will thus follow this approach, as did
Bellemare et al.
(2017), whose model we describe next.
2. T HE ENERGY DISTANCE AND ASSOCIATED MMD
Liu (2017) and Bellemare et al.
(2017, Section 4) proposed to use the energy distance as the critic in an adversarial network.
The energy distance (Székely & Rizzo, 2004; Lyons, 2013) is a measure of divergence between two probability measures, deﬁned as
[Equation]
(7) where E P ρ ( X, X (cid:48) ) is an expectation over two independent samples from the generator P (likewise,
Y and Y (cid:48) are independent samples from the reference Q ), and ρ ( x, y ) is a semimetric of negative type.
2 We will focus on the case ρ ( x i , x j ) =
(cid:107) X − X (cid:48)
(cid:107) β for 0 < β ≤ 2 .
When β = 1 , ρ is a metric.
It has been shown (Sejdinovic et al., 2013, Lemma 12) that the energy distance is an instance of the maximum mean discrepancy, where the corresponding distance-induced kernel family for the distance (7) is
[Equation]
(8) for any choice of z 0 ∈ X (often z 0 = 0 is chosen to simplify notation, which corresponds to a fractional Brownian motion kernel; Sejdinovic et al., 2013, Example 15).
Note that the resulting kernel is not translation invariant.
It is characteristic (when β < 2 ), and the MMD is well deﬁned, for a class of distributions P that satisfy a moment condition (Sejdinovic et al., 2013, Remark 21), namely that there exists z 0 ∈ X such that
[Equation]
To apply the regularization strategy of Gulrajani et al.
(2017) in training the critic of an adversarial network, we need to compute the form taken by the witness function (2) given the kernel (8).
Bearing in mind that
[Equation]
where the second term of the above expression is constant, and substituting into (2), we have
[Equation]
This is in agreement with Bellemare et al.
’s function f ∗ (their page 5), albeit via a different argument (note that the function g ∗ in their footnote 4 differs from the above, and appears to be in error).
We now turn to the divergence implemented by the critic in Bellemare et al.’s
Algorithm 1, which is somewhat different from the energy distance (7).
The Cramér GAN witness function is deﬁned as
[Equation]
which is regularized using Gulrajani et al.
’s gradient constraint.
The expected surrogate loss associated with this witness function, and used for the Cramér critic, is
[Equation]
In brief, Y (cid:48) in (7) is replaced by the origin: it is explained that this is necessary for instance in the conditional case, where two independent reference samples Y, Y (cid:48) are not available.
Following this change, it becomes straightforward to deﬁne P and Q which are different, yet have
an expected
D c ( P, Q ) loss of zero.
For example, if P is a point mass at the origin in R , and Q is a point mass a distance t from the origin, then P (cid:54) =
Q , and yet D c ( P, Q ) =
because
[Equation]
Nevertheless, good empirical performance has been obtained in practice for the Cramér critic, both by Bellemare et al.
(2017) and in our Section 5 experiments.
Insight into this behaviour can be obtained by considering its relation to the score function associated with the energy distance.
We describe this relation in Appendix A.
U NBIASED GRADIENTS
One of the main criticisms of the Wasserstein GAN is that its gradient is biased (Bellemare et al., 2017, Section 3), which can cause problems when performing stochastic gradient descent.
By contrast, Bellemare et al.
claim that the energy distance used in the Cramér GAN critic does not suffer from biased gradients.
They do not, however, properly prove this claim: the essential step in the reasoning – the exchange in the order of the expectations and the derivatives – is simply assumed.
We prove here that under reasonable assumptions, the expectation and gradient can be interchanged for kernels on representations given by deep networks, and hence gradients of (4) are unbiased estimates of the gradient of the true MMD.
Theorem 1.
Let P , Q be distributions with moments of order α ≥ 1 (see Assumption A ), and let k be a continuously differentiable kernel subject to some growth conditions (Assumption B ).
Let φ θ ( X ) be a feedforward neural network with parameters θ as in Deﬁnition 2; in particular, convolutional networks with ReLU-like activations are covered.
The function θ (cid:55)→ E P , Q [ k ( φ θ ( X ) , φ θ ( Y ))] is well-deﬁned and differentiable for µ -almost all θ ∈ R m , where µ is the Lebesgue measure.
Moreover, the following formula holds:
[Equation]
The proof is given in Appendix B.
The main idea is to ﬁrst prove the result for differentiable activation functions, using the standard dominated convergence theorem.
We then consider a sequence of differentiable activation functions that converge to the activation function of interest, which is differentiable except at the origin: for example the ReLU activation.
The key point is to ﬁrst show that the set of parameters under which the expected gradient is undeﬁned has zero Lebesgue measure, regardless of the probability distributions P and Q .
We then prove that the expected gradients using the differentiable activations converge locally uniformly towards the expected gradient obtained using the singular activation function.
We then conclude using standard results of gradients of uniformly convergent sequences of functions.
E VALUATION METRICS
One problem with comparing GAN models is that quantitative comparisons are difﬁcult.
Some insight can be gained by visually examining samples, but we also consider the following approaches to evaluate GAN methods.
Inception score This metric, proposed by Salimans et al.
(2016), is based on the classiﬁcation output p ( y | x ) of the Inception model (Szegedy et al., 2016).
Deﬁned as exp ( E x KL( p ( y | x ) (cid:107) p ( y )))
, it is highest when each image’s predictive distribution has low entropy, but the marginal p ( y ) =
E x p ( y | x ) has high entropy.
This score correlates somewhat with human judgement of sample quality on natural images, but it has some issues, particularly when applied to domains which do not represent a variety of the types of classes in ImageNet.
In particular, it knows nothing about the desired distribution for the model.
FID The Fréchet Inception Distance , proposed by Heusel et al.
(2017), avoids some of the problems of Inception by measuring the similarity of the samples’ representations in the Inception architecture (at the pool3 layer) to those of samples from the target distribution.
The FID ﬁts a Gaussian distribution to the hidden activations for each distribution and then computes the Fréchet distance, also known as the Wasserstein-2 distance, between those Gaussians.
Heusel et al.
show that unlike the Inception score, the FID worsens monotonically as various types of artifacts are added to images.
Note, however, that the estimator of FID is biased.
KID We propose a metric similar to the FID, the Kernel Inception Distance , to be the squared
MMD between Inception representations.
We use a polynomial kernel, k ( x, y )
= (cid:0) 1 d x T y +
(cid:1)where d is the representation dimension, to avoid correlations with the objective of MMD GANs as well as to avoid tuning any kernel parameters.
5 Compared to the FID, the KID has several advantages.
First, it does not assume a parametric form for the distribution of activations; this is particularly sensible since the representations have ReLU activations, and therefore are not only never negative, but do not even have a density: about 2% of components in Inception representations are typically exactly zero.
With the cubic kernel we use here, the KID compares skewness as well as the mean and variance.
Also, unlike the FID, the KID has a simple unbiased estimator.
Figure 1 demonstrates the bias of the FID and the unbiasedness of the KID by comparing the CIFAR10 train and test sets.
The KID (Figure 1a) converges quickly to its presumed true value of 0; even for very small n , simple Monte Carlo estimates of the variance provide a reasonable measure of uncertainty.
By contrast, the FID estimate (Figure 1b) does not behave so nicely: at n = 2 000 , when the KID estimator is essentially always 0, the FID estimator is still quite large.
Even at n = 10 000 , the full size of the CIFAR test set, the FID still seems to be decreasing from its estimate of about 5.7 towards zero, showing the strong persistence of bias.
More importantly, Monte Carlo estimates of the variance are extremely small even when the estimate is very far from its asymptote, so it is difﬁcult to judge the reliability of an estimate.
For models on MNIST, we replace the Inception featurization with features from a LeNet-like convolutional classiﬁer 6 (LeCun et al., 1998), but otherwise compute the scores in the same way.
[Figure]
We also considered the diagnostic test of Arora & Zhang (2017), which estimates the approximate number of “distinct” images produced by a GAN.
The amount of subjectivity in what constitutes a duplicate image, however, makes it hard to reliably compare models based on this diagnostic.
Comparisons probably need to be performed both with a certain notion of duplication in mind and by a user who does not know which models are being compared, to avoid subconscious biases; we leave further exploration of this intriguing procedure to future work.
E XPERIMENTS
We compare the quality of samples generated by MMD-GAN with various kernels with the ones obtained by WGAN-GP (Gulrajani et al., 2017) and Cramér GAN (Bellemare et al., 2017) on two standard benchmark datasets: the MNIST dataset of 28 × 28 handwritten digits 7 and the CIFARdataset of 32 × 32 photos (Krizhevsky, 2009).
For all of our experiments we used the DCGAN architecture (Radford et al., 2016) for both generator and critic.
For MMD losses, we used only 16 top-layer neurons in the discriminator; more did not seem to improve performance.
For the generator we used the standard number of convolutional ﬁlters (64 in the second-to-last layer); for the critic, we compared networks with 16 and 64 ﬁlters in the ﬁrst convolutional later.
Networks with smaller critic run considerably faster: on our systems, the 16-ﬁlter networks typically ran at about twice the speed of the 64-ﬁlter ones.
Note that the critic size is far more important to training runtime than the generator size: we update the critic 5 times for each generator step, and moreover the critic network is run on two batches each time we use it, one from P and one from Q .
Given the same architecture, all models considered here run at about the same speed.
We evaluate several MMD GAN kernel functions in our experiments.
The simplest is the linear kernel: k dot ( x, y ) = (cid:104) x, y (cid:105) , whose MMD corresponds to the distance between means (this is somewhat similar to the feature matching idea of Salimans et al., 2016).
We also use the exponentiated quadratic (5) and rational quadratic (6) functions, with mixtures of lengthscales,
[Equation]
where Σ = { 2 , 5 , 10 , 20 , 40 , 80 } , A = { . 2 , .
Lastly we use the distance-induced kernel k distof (8), using the Euclidean distance, so that the MMD is the energy distance, and the Cramér critic in (10). [Figure]
[Table]
[Figure]
Each model was trained with a batch size of 128, and 5 discriminator updates per generator update.
For CIFAR-10 we trained for 150 000 generator updates, while for MNIST we used 50 000 .
MNIST After training for 50 000 generator iterations, all variants we tried achieved reasonable results.
Table 1 shows the quantitative measures, computed on the basis of a LeNet model.
All have achieved KIDs of essentially zero, and FIDs around the same as that of the test set, with Inception scores slightly lower.
Figure 4, in Appendix C, shows samples.
Figure 2, however, shows the evolution of our quantitative criteria throughout the training process for several models.
This shows that the linear kernel dot is clearly worse than the other models for most of the training process, but it makes a sharp improvement around 40 000 iterations and basically catches up to the other models.
There is also some evidence that rbf , and perhaps WGAN-GP, converge more slowly than rq and
Cramér GAN.
Examining samples during training, we observed that rbf more frequently produces extremely “blurry” outputs, which can persist for a substantial amount of time before eventually resolving.
This makes sense, given the very fast gradient decay of the rbf kernel: when generator samples are extremely far away from the reference samples, slight improvements yield very little reward for the generator, and so bad samples can stay bad for a long time.
Given their otherwise similar properties, we thus recommend the use of rq kernels over rbf in MMD GANs.
CIFAR-10 Scores for various models trained on CIFAR-10 are shown in Table 2.
The scores for rq with a small critic network approximately match those of WGAN-GP with a large critic network, at substantially reduced computational cost.
Here the distance kernel, with a large critic, seemed to do somewhat worse.
With a small critic, WGAN-GP and Cramér GAN both performed very poorly.
These results illustrate the beneﬁts of using the MMD on deep convolutional feaures as a GAN critic.
In this hybrid system, the initial convolutional layers map the generator and reference image distributions to a simpler representation, which is well suited to comparison via the MMD.
The
MMD in turn employs an inﬁnite dimensional feature space to compare the outputs of these convolutional layers.
By comparison, WGAN-GP requires a larger discriminator network to achieve similar performance.
It is interesting to consider the question of kernel choice: the distance kernel and RQ kernel are both characteristic (Sriperumbudur et al., 2010), and neither suffers from the fast decay of the exponentiated quadratic kernel, yet the RQ kernel performs slightly better in our experiments.
The relative merits of different kernel families for GAN training will be an interesting topic for further study.
[Table]
[Figure]
[Table]
[Figure]
WGAN-GP samples with critic size 16 are quite bad, as well as being under-diverse. Cramér GAN samples with critic size 16 are more appealing to the eye but seem to be under are
A S CORE FUNCTIONS , DIVERGENCES , AND THE C RAMÉR GAN
It is not immediately obvious how to interpret the surrogate loss (10).
An insight comes from considering the score function associated with the energy distance, which we now brieﬂy review (Gneiting & Raftery, 2007).
A scoring rule is a function S ( P, y ) , which is the loss incurred when a forecaster makes prediction P , and the event y is observed.
The expected score is the expectation under Q of the score,
[Equation]
If a score is proper , then the expected score obtained when P = Q is greater or equal than the expected score for P (cid:54) =
Q ,
[Equation]
A strictly proper scoring rule shows an equality only when P and Q agree.
We can deﬁne a divergence measure based on this score,
[Equation]
Bearing in mind the deﬁnition of the divergence (11), it is easy to see (Gneiting & Raftery, 2007, eq. 22) that the energy distance (7) arises from the score function
[Equation]
The interpretation is straightforward: the score of a reference sample y is determined by comparing its average distance to a generator sample with the average distance among independent generator samples, E P ρ ( X, X (cid:48) ) .
As discussed earlier, the Cramér GAN critic does not use the energy distance (7) directly on the samples, but ﬁrst maps the samples through a function h , for instance a convolutional network; this should be chosen to maximize the discriminative performance of the critic.
Writing this mapping as h , we break the energy distance down as D e ( P, Q ) =
S ( Q, Q ) − S ( P, Q ) , where
[Equation]
and
[Equation]
When training the discriminator, the goal is to maximize the divergence by learning h , and so both (12) and (13) change: in other words, divergence maximization is not possible without two independent samples Y, Y (cid:48) from the reference distribution
Q.
An alternative objective in light of the score interpretation, however, is to simply optimize the average score (13).
In other words, we would ﬁnd features h that make the average distance from generator to reference samples much larger than the average distance between pairs of generator samples.
We no longer control the term encoding the “variability” due to Q , E Q ρ ( h ( Y ) , h ( Y (cid:48) )) , which might therefore explode: for instance, h might cause h ( Y ) to disperse broadly, and far from the support of P , assuming sufﬁcient ﬂexibility to keep E P ρ ( h ( X ) , h ( X (cid:48) )) under control.
We can mitigate this by controlling the expected norm E Q ρ ( h ( Y ) , 0) , which has the advantage of only requiring a single sample to compute.
For example, we could maximize
[Equation]
This resembles the Cramér GAN critic (10), but the generator-to-generator distance is scaled differently, and there is an additional term: E P ρ ( h ( X ) , 0) is being maximized, which is more difﬁcult to interpret.
An argument has been made (in personal communication with Bellemare et al.)
that this last term is required if the function f c in (9) is to be a witness of an integral probability metric (1), although the asymmetry of this witness in P vs Q needs to be analyzed further.
B P ROOF OF UNBIASED GRADIENTS
In this section we prove the interversion of the expectation and the gradient in Theorem 1.
We ﬁrst consider a class of "well-behaved" activation functions:
Deﬁnition 1.
An activation function will be said to be well behaved if it satisﬁes the following:
• σ (0) =
• σ is differentiable on R ∗ •
The derivatives of σ are bounded | dσdx | ≤ L • There exist a family of activation functions ( σ t ) t> 0 that are all differentiable on R and such that • The family σ t converges uniformly towards σ on R .
• The family of derivatives σ (cid:48)
t converges locally uniformly towards σ (cid:48) on R ∗ .
• σ (cid:48) t (0) converges towards a ﬁnite value c
Given such conditions, the family σ t can be extended at the value 0 by σ 0 = σ . By convention we denote σ (cid:48)
=
c , although it is not the derivative of σ 0 .
Deﬁnition 2.
Given an input X in R d , we deﬁne the feedforward neural network φ θ with K hidden layers by the following:
[Equation]
Where σ is a well-behaved activation function as deﬁned in Deﬁnition 1.
b k is a vector in R d k and W k is a matrix of weights.
The functions g k are introduced to account for convolutions and are known linear transformations.
the only requirement is that g k ( W k ) is a d k by d k − 1 matrix with ( d 0 = d ).
We will denote by θ the concatenation of all the parameters of the network θ =
( b k , W k , )
Kk =
and we will denote the last layer h K by φ θ ( X ) , θ ∈ R m .
For simplicity we will drop the subscript for functions g .
We will ﬁrst consider the case where σ is differentiable at 0 .
The case where σ is non differentiable will then be obtained by considering a parametric family of differentiable activation functions σ α such that σ α converges point-wise towards σ when α goes to 0 .
We will then extend the results by taking the limit when α approches 0 .
We have the following results:
Theorem 2.
If the activation function σ is differentiable on R and has bounded derivatives, then θ (cid:55)→ φ θ ( X ) is differentiable for all X in R d .
Moreover, there exist continuous real valued functions deﬁned on R m α ( θ ) , β ( θ ) , a ( θ ) and b ( θ ) such that:
[Equation]
for all X in R d .
Proof.
First note that σ is L -Lipchitz, therefore : (cid:107) σ ( X ) (cid:107) ≤ L (cid:107) X (cid:107) for any ﬁnite dimensional vector, where σ is applied point-wise and the norm used is the euclidian norm.
Let now k be in { 1 , ..., K } , we have the following:
[Equation]
A simple recursion then gives:
[Equation]
With a k ( β )
=
L k (cid:81) kj =1 (cid:107) g ( W j ) (cid:107) Op and b k ( θ ) =
L k (cid:80) ni =1 (cid:107) b
i (cid:107) (cid:81) nj
= i +1 (cid:107) g
( W j ) (cid:107) Op
By setting b bound.
For the gradient, we will proceed by recursion.
For k = 0 we trivially have (cid:107) ∂ θ h 0 (cid:107) = 0 .
Now for some k in { 1 , ...K } , we assume that (cid:107) ∂ θ h k − 1 (cid:107) ≤ β k − 1 ( θ ) + α k − 1 ( θ ) (cid:107) X
(cid:107) .
For the parameters ¯ θ of layers higher than k we have (cid:107) ∂ ¯ θ h k (cid:107) 2 = 0 .
Now for the parameters (cid:101) θ of layers lower than k we get:
[Equation]
h k (cid:107) [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] By recursion we get (cid:107) ∂ (cid:101) θ h k (cid:107) ≤ L (cid:107) g ( W k ) (cid:107) ( β k
[Equation]
1 ( θ ) + α k − 1 ( θ ) (cid:107) X
(cid:107) )
Now for the parameters at layer k θ k = ( b k , W k ) we also have: [Equation] X
[Equation]
Using Minkowski inequality we ﬁnally get:
[Equation]
) with α k = La k ( θ ) + L (cid:107) g ( W k ) (cid:107) α k − 1 ( θ ) and β k
=
L (1 + b k θ ) + L (cid:107) g
( W k ) (cid:107) β
k − 1 ( θ ) .
The results follows by taking k = K .
Let now P and Q be two probability distribution over the input space R d and let X and Y be two random variables following P and Q respectively.
We also consider a kernel K deﬁned on the feature space.
We make the following assumptions on P , Q and the kernel K for a given α ≥ 1 :
[Equation]
B
The kernel is differentiable and satisﬁes the growth conditions:
[Equation]
C
The gradient of the kernel is M -Lipchitz for some positive M , i.e:
[Equation]
Here C is a positive constant.
We have the following result:
Theorem 3.
If the activation function σ is differentiable then θ (cid:55)→ E P , Q [ K ( φ θ ( X ) , φ θ ( Y ))] is welldeﬁned and differentiable for all θ ∈ R m .
Moreover, the following formula holds:
[Equation]
Proof.
The integrability of E P , Q [ K ( φ θ ( X ) , φ θ ( Y ))] is straightforward using the bound on the network and on the kernel.
For the differentiability, we will use the dominated convergence theorem .
Indeed, by the chain rule one has:
[Equation]
Taking the norm for each operator respectively, one gets:
[Equation]
We will use now the bounds on φ θ ( X ) and ∂ θ φ θ ( X ) .
First recall that the functions α , β , a and b are continuous on R m and considering a closed ball B ( θ 0 , µ ) for some θ 0 in R m and µ > 0 , each of those functions attain a maximum value on that ball.
We simply denote by b ∗ , a ∗ , β ∗ and α ∗ these maximum values and use them directly in the bounds.
We also introduce a threshold R > 0 on the norms (cid:107) X (cid:107) and (cid:107) Y (cid:107) .
Let G θ ( X, Y ) :
= (cid:107) ∂
θ K ( φ θ ( X ) , φ θ ( Y ))
(cid:107) for convenience, we then getdifferent cases:
• If (cid:107) X (cid:107) ≤ R and (cid:107) Y (cid:107) ≤ R , then: G θ ( X, Y ) ≤ F 1 ( X, Y ) :
= 4 (cid:104)
b ∗ + a ∗ R (cid:105)
α − 1 (cid:104) β ∗ + α ∗ R (cid:105) •
If (cid:107) X (cid:107) > R and (cid:107) Y (cid:107) >
R , then: G θ ( X, Y ) ≤ F 2 ( X, Y ) with F 2 ( X, Y ) [Equation]
= (cid:104)
[Equation]
a [Equation] (cid:105)
[Equation] − 1 (cid:0) (cid:107) X (cid:107) α − [Equation] (cid:107) Y (cid:107) α − 1 (cid:1) (2 β ∗ + α ∗ ( (cid:107) X {+(cid:
+}
+ (cid:107) Y (cid:107) ))
If (cid:107) X (cid:107) > R and (cid:107) Y (cid:107) ≤ R , then: G θ ( X, Y ) ≤ F 3 ( X, Y ) with
[Equation] (cid:1)
[Equation] (cid:107) X (cid:107) α − 1 + (cid:0) [Equation] a
[Equation] (cid:1) α − 1 (cid:1) ( [Equation] α ∗ ( (cid:107) X (cid:107) + R ))
• If (cid:107) Y (cid:107) >
R and (cid:107) X (cid:107) ≤ R , then [Equation] [Equation]
X, Y ) with
[Equation] It is clear from Assumption A that the functions F 1 , F 3 and F 4 are integrable.
F 2 involves terms of the form (cid:107) X (cid:107) α − 1 (cid:107) Y (cid:107) which can be bounded in expectation using Holder inequality:
[Equation]
Therefore F 2 is also integrable.
Now denoting by:
[Equation]
we have that G θ ( X, Y ) is upper bounded by F uniformly on the parameter ball B ( θ 0 , η ) .
The dominated convergence applies then and we get the interversion result for all θ 0 in R m .
Now we would like to consider the case when the activation function σ is not differentiable in 0 .
We will consider a family of activation functions σ t that are differentiable on R and that converges point-wise towards σ as t goes to 0 .
To keep the proof simple, we consider the case of the RELU activation function σ ( x ) = max(0 , x ) and the following family of approximating functions:
[Equation]
for t > 0 .
It is easy to see why these functions converge towards σ .
Moreover, the derivative of σ t is given by:
[Equation]
It is also clear that dσ t ( x ) dx converge towards dσ ( x ) dx for all x ∈ R ∗ .
For x = 0 , dσ t ( x ) dx → 12 , which is a sub-gradient of σ in x = 0 .
And dσ t ( x ) dx is uniformly bounded in x and t .
A key property of this family is that it is uniformly convergent and the derivatives dσ t ( x ) dx are locally uniformly convergent on R ∗ .
In other words if C is a compact set included in R ∗ then:
[Equation]
where norm used is the supremum norm over the set C .
We will extend the family to t = 0 by setting σ 0 = σ
and we also denote by σ (cid:48)
= lim t → 0 σ (cid:48) t .
Note that σ (cid:48)
coincides with the derivative of σ on R ∗ and has a deﬁned value in 0 .
Let’s deﬁne the family of functions f t ( θ ) :=
E P , Q [ K ( φ ( t ) θ ( X ) , φ ( t ) θ ( Y ))] and g t ( θ ) :=
E P , Q [ ∂ θ K ( φ ( t ) θ ( X ) , φ ( t ) θ ( Y ))] .
Here we denoted by φ ( t ) θ ( X ) the network with activation function σ t .
We have the following proposition:
Lemma 1.
For a ﬁxed θ , X and Y , then the following point-wise convergence hold:
[Equation]
Moreover, f t ( θ ) → t → 0 f 0 ( θ ) and g t ( θ ) → t → 0 g 0 ( θ )
Proof.
The point-wise convergence is obtained directly by taking the limit.
The limits in expectation are obtained using the dominated convergence theorem and Assumptions A and B as well the growth properties of the network Theorem 2.
Now we would like to prove that ∂ θ f 0 ( θ ) =
g 0 ( θ ) for µ -almost all θ .
For this to hold we need to prove ﬁrst that g t ( θ ) converges locally uniformly towards g 0 ( θ ) as t goes to 0 .
Let’s ﬁrst introduce some notations.
For a given θ 0 ∈ R d and η > 0 we deﬁne the set N
[Equation]
R d (cid:12)(cid:12)
[Equation]
θ ∈ B ( θ 0 , η ) , k ∈ { 1 , ..., K } , i ∈ { 1 , ..., d k } :
b ki + g ( W k )
[Equation]
h k − 1 = 0 (cid:111)
N ( θ 0 , η ) is the set of X where at least one linear form in the network vanishes.
This allows to introduce the set of "well-behaved" parameters:
[Equation]
Note that N ( θ 0 , η ) is always a closed set.
We have the following uniform convergence result:
Lemma 2.
For all k ∈ { 0 , ..., K } , the family of hidden layers ( h kt ) t ≥ 0 obtained using the activations functions σ t converge uniformly on B ( θ 0 , η ) × R d towards h k 0 for every θ 0 ∈ R m and η > 0 .
Proof.
We proceed by recursion on the layers.
For k =
the statement trivially holds.
Assume now that for k − 1 , ( h k − 1 t ) t ≥ 0 converges uniformly on a ball B ( θ 0 , η ) × R d .
We have the following:
[Equation]
For the ﬁrst term we use the fact that σ t is uniformly convergent on R , for (cid:15) > 0 there exists η > such that for all t <
η we have that (cid:107) σ t ( b k + g ( W k )
h k − 1 t
[Equation]
[Equation] ≤ (cid:15) .
For the second term we use the Lipchitz continuity of σ 0 :
[Equation]
Since (cid:107) g ( W k ) (cid:107) is bounded on the ball B ( θ 0 , η ) , one gets the desired result by recursion.
Lemma 3.
Under assumption Assumption C , and for all θ ∈ Θ P ∩ Θ Q , g t ( θ ) converges locally uniformly towards g 0 ( θ ) .
Proof.
Let θ
be in Θ P ∩ Θ Q and let η > 0 .
For simplicity we drop the subscripts of the expectations.
Using Assumption C one has:
[Equation]
( Y ) − ∂
[Equation]
[Equation]
Y ) (cid:107) ]
All the expectations that do not involve a difference between two terms can be upper bounded using Assumption B and the growth properties of the network in Theorem 2 over all parameters in
B ( θ 0 , η ) .
We will ﬁrst control the difference E
[ (cid:107)K ( φ ( t ) θ ( X ) , φ ( t ) θ ( Y ))
− K ( φ (0) θ ( X ) , φ (0) θ ( Y )) (cid:107) ] .
It can be uniformly controlled using the the uniform convergence of φ ( t ) towards φ 0 and continuity of the gradient ∇K .
Let A n be a partition of the set R d × R d , such that A n are compact.
The exist
N 0 >
such that (cid:80) ∞
i = N 0 +1 E P , Q ( A n ) < (cid:15)
On one hand we have that E [ (cid:107)K ( φ ( t ) θ ( X ) , φ ( t ) θ ( Y ))
−
K ( φ (0) θ ( X ) , φ (0) θ ( Y )) (cid:107) ] is bounded uniformly on the ball B ( θ 0 , η ) by some constant S .
This comes by virtue of the growth properties of the network in Theorem 2 and of the gradient of the kernel in
Assumption B .
It comes that:
[Equation]
On the other hand by uniform convergence of φ ( t ) θ towards φ (0) θ () , there exists α > 0 such that [Equation] φ ( t ) θ ( X ) − φ (0) θ ( X ) (cid:107) ≤ β and (cid:107) φ ( t ) θ ( X ) − φ (0) θ ( X ) (cid:107) ≤ β for all
( X, Y ) ∈
A n for a given β .
Moreover, The image U of B ( θ 0 , η ) ×
A n by ( θ, X, Y ) (cid:55)→ ( φ ( t ) θ ( X ) , φ ( t ) θ ( Y )) is included in a compact set C .
By continuity of the gradient ∇K , one deduces that it is uniformly continuous on the compact set C , therefore provided that β is small enough, one can get that:
[Equation]
Now we will control the error:
[Equation]
Here we will only provide a proof for the variable X , the same holds for Y .
We proceed by recursion over the layers.
Indeed for k = 0 , h k = X
and ∂ θ h k = 0 which deﬁne a constant family in t that is uniformly convergent.
Assume now that for layer k − 1 for every (cid:15) > 0 , there exists α k − 1 < α such that E [ (cid:107) ∂ θ h k − 1 t − ∂ θ h k − 1(cid:107) ] ≤ (cid:15) for all θ ∈ B ( θ, η ) .
Note that h k − 1 depends on θ although it is not explicitly shown by the notation.
We have the following decomposition:
[Equation]
We get three terms, in the second and third terms one can add and subtract a control term that allows the use of the recursion assumption for h k − 1 t and ∂ θ h k − 1 t .
These terms can therefore be bounded by (cid:15) uniformly on the ball B ( θ 0 , η ) provided that t ≤ α k − 1 .
The remaining terms are proportional to:
[Equation]
which can further be upper-bounded by:
[Equation]
We will control now the errors:
[Equation]
Let’s consider the set N ( θ 0 , η ) , Note that its complementary N c ( θ 0 , η ) is an open set.
Let D n be a partition of this open set: N c ( θ 0 , η ) = ∪ ∞ n =1 D n , we assume that D n are compact.
If we restrict the parameters θ to a closed ball ¯ B ( θ 0 , (cid:101) η ) then ¯ B ( θ 0 , (cid:101) η ) × D n is a compact set.
We denote by V the image of ¯ B ( θ 0 , (cid:101) η ) × D n by the mapping R : ( θ, X ) (
cid:55)→ b k
+ g ( W k )
h k − 1.
One can easily see that there exists a ball B (0 , r ) centered on 0 with some radius r such that its intersection with V is empty.
This comes from the fact that V is continuous on the compact set ¯ B ( θ 0 , (cid:101) η ) × D n .
It follows then from the uniform convergence of h k − 1 t towards h k − 1that there exists an α 1 > 0 such that for all t ≤ α 1 V t the image of ¯ B ( θ 0 , (cid:101) η ) × D n by the application R t : ( θ, X ) (
cid:55)→ b k
+ g ( W k ) h k − 1 t has an empty intersection with a smaller ball B (0 , r 2 ) .
One can then safely use the fact that σ (cid:48) t is locally Lipchitz on R \ B (0 , r 2 ) with some constant L r .
Note that the Lipchitz constant L r depends on r which in turn depends on the set D n .
Therefore, L r might diverge as D n gets closer to the critical set
N ( θ 0 , η ) .
To avoid this issue, we will make sure to cut off the contribution of sets D n that are too small anyways.
Indeed there exists N 0 >
such that (cid:80) ∞ N 0 +1 P ( D n ) ≤ (cid:15) .
One can therefore use the following decomposition for A 1 :
[Equation]
The third term is bounded by 2 L(cid:15) while the second term is uniformly controlled by C ( (cid:80) N 0 n =1 L D n ) (cid:15) .
Remains the ﬁrst term.
since θ 0 belongs to Θ , provided that η is small enough then P ( N ( θ 0 , η )) ≤ (cid:15) , the result follows using the fact that σ (cid:48) t is bounded.
The term A 2 can be controlled in exactly the same way:
[Equation]
Here the ﬁrst and third terms are controlled by the sizes of the sets D n and N ( θ (cid:48) , η ) , while for the second term one can simply use the locally uniform convergence of σ (cid:48) t towards σ (cid:48) 0 since the images of the compact sets ¯ B ( θ 0 , (cid:101) η ) × D n by the mapping R : ( θ, X ) (
cid:55)→ b k
+ g ( W k ) h k − 1are far away from 0 by at least a positive number r .
We have shown that for (cid:15) > 0 there exists η > 0 and α > 0 such that:
[Equation]
for all θ ∈ B ( θ 0 , η ) and all t ≤ α , which exactly means that g t converges locally uniformly to g 0 .
Lemma 4.
The set Θ c P has 0 Lebesgue measure.
Proof.
The key idea here is to prove that for any θ ∈ Θ c P , and along some coordinates, there cannot be more than a countable number of neighbors of θ that are also in Θ c P .
One should ﬁrst notice that:
[Equation]
Indeed, if θ ∈ Θ c P then there exists (cid:15) > 0 such that for all η > 0 we have:
[Equation]
We also have that (cid:49)
X ∈ θ,η → (cid:49)
X ∈ θ, (cid:48) from the continuity of the network.
Therefore by the dominated convergence theorem one gets that P ( N ( B ( θ, 0)))
=
lim η → 0 P ( N ( B ( θ, η ))) > (cid:15) .
The converse is trivial.
Let θ 0 be in Θ c P , then by deﬁnition there exists (cid:15) > 0 such that P ( N ( θ 0 , 0)) > (cid:15) .
This also means that there is a layer k and a coordinate
i in the layer k : i ∈ { 1 , ..., d k } such that the set { X ∈ R d | b
i + g ( W k ) i
h k − 1 = 0 } has a non zero measure.
Without loss of generality, one can assume that N ( θ 0 , 0) =
{ X ∈ R d | b i + g ( W k ) i h
k − 1 = 0 } .
Consider now the family of parameters ( θ h ) h ∈ R such that all the coordinates of θ h match those of θ 0 except at layer k and for the node i where b hi
=
b i + h .
It is clear that the family of sets N ( θ h , 0) are jointly disjoint.
That is, if h, h (cid:48) ≥ are such that h (cid:54) =
h (cid:48)
then N ( θ h , 0) ∩ N ( θ (cid:48) h , 0)
= ∅ .
We will prove now that the intersection A between this family ( θ h ) 0 ≤ h and Θ c P can be at most countable.
Indeed if A was uncountable, then this would deﬁne an uncountable family of sets ( N ( θ, 0))
θ ∈ A with positive measures.
Let’s now introduce the following family of sets:
[Equation]
Since A is uncountable, then there exists n such that A n is inﬁnite.
Now consider the set B obtained by taking the union of all the disjoint sets N ( θ, 0 with θ ∈
A n and if ( θ m ) ∞ m =1 is an injective sequence in A n then P ( B ) ≥ (cid:80)
∞
m =1 P ( N ( θ m , 0) ≥ (cid:80) ∞ m =1 1 n = + ∞ .
As this can’t happen, it means that the intersection set A is at most countable.
So far we have shown that while keeping all the other parameters ﬁxed for θ , only a countable possible values of the coordinate b ki can be taken in Θ c P .
One can then easily conclude using Fubini theorem:
[Equation]
The integral over b ki is equal to zero which leads to the result.
We are now ready to prove the main result.
Proof of Theorem 1.
By Theorem 3 we have that ∂ θ f t ( θ ) =
g t ( θ ) for all t > 0 and θ ∈ R m .
We also have by Lemma 1 that f t ( θ ) converges point-wise towards f 0 ( θ ) , and by Lemma 3, we know that g t ( θ ) converges uniformly towards g 0 ( θ ) for all θ be in Θ P ∩ Θ Q .
This allows to conclude that f 0 ( θ ) is differentiable on Θ P ∩ Θ Q and its gradient is given by g 0 ( θ )
=
E P , Q [ ∂ θ K ( φ θ ( X ) , φ θ ( Y ))] .
Moreover, (Θ P ∩ Θ Q ) c
= Θ c P ∪ Θ c Q has zero Lebesgue measure as it its the union of two Lebesgue 0 measure sets by Lemma 4. The conclusion follows.
C M ODEL SAMPLES
[Figure]