T OWARDS A TESTABLE NOTION OF GENERALIZATION FOR GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data.
We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is.
Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution.
We use this procedure to assess the performance of several modern generative adversarial network architectures.
We ﬁnd that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance.
We ﬁnally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and provide a concrete formulation for doing so.
I NTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have recently attracted signiﬁcant interest as a means for performing unsupervised learning.
However, recently concerns have been raised about their ability to generalise from training data and their capacity to overﬁt (Arora &
Zhang, 2017; Arora et al., 2017).
Moreover, techniques for evaluating the quality of GAN output are either ad hoc, lack theoretical rigor, or are not suitably objective – often times “visual inspection” of samples is the main tool of choice for practitioner.
More fundamentally, it is sometimes unclear exactly what we want a GAN to do: what is the unsupervised learning task that we are trying to achieve?
In this paper, we provide a simple formulation of the GAN training framework, which consists of using a ﬁnite dataset to estimate an underlying true distribution.
The quality of GAN output is measured quantitatively in terms of a statistical distance D between the estimated and true distribution.
Within this context, we propose an intuitive notion of what it means for a GAN to generalise.
We also show how our notion of performance can be measured empirically for any given GAN, when D is chosen to be a Wasserstein distance.
Unlike other methods (such as the inception score (Salimans et al., 2016)), we show that this choice of D requires us to make no density assumptions about the underlying true distribution.
We investigate this choice of D empirically, showing that its performance is heavily dependent on the choice of ground metric underlying the Wasserstein distribution.
We suggest a novel choice of ground metric that we show achieves good performance, and discuss how we might use this observation to improve the design of Wasserstein GANs (WGANs)
(Arjovsky et al., 2017).
T HE OBJECTIVE OF UNSUPERVISED LEARNING
In order to assess the effectiveness of GANs, it is essential that we deﬁne what we mean for a GAN to work well.
We do so in this section by providing a simple but general formulation of the goal of unsupervised learning in this context.
Our formulation is compatible with most of what is considered in the literature and what is done in practice, but we hope it will allow us to clarify certain details about the task.
To begin with, we assume some distribution π on a set X .
For instance, X may be the set of 32x colour images, and π the distribution from which the CIFAR-10 dataset was sampled.
Informally, the goal will be to learn π in a particular sense.
We assume that π is almost completely intractable: we do not know its density (or even if it has one), and we do not have a procedure to draw new samples from it.
However, we do suppose that we have a ﬁxed dataset X := { x 1 , · · · , x n } , where x i iid ∼ π .
Equivalently, we have the empirical distribution
[Equation]
where δ denotes the Dirac mass.
Denote by P ( X ) the set of distributions on X .
We wish to use X to produce a distribution α ( X ) ∈
P ( X ) that is as “close” as possible to π .
We measure closeness measured in terms of a function
D : P ( X ) × P ( X ) → R .
Usually D is chosen to be a statistical divergence , which means that
D ( P, Q ) ≥ 0 for all P and Q , with equality iff P = Q .
Stated compactly, then, our goal as designers of unsupervised learning algorithms in this context is to choose α such that D ( α ( X ) , π ) is usually small for the sorts of X and π
we see in practice.
Now, we can immediately see that one possibility is simply to choose
[Equation]
Moreover, in the case that D is a metric for the weak topology on P ( X ) , we have that
[Equation]
almost surely as n → ∞ , so that, assuming X is large enough, we can already expect D ( ˆ X, π ) to be small.
This suggests a natural notion of generalisation in this context: we can say that a choice of α generalises for a given X if
[Equation]
In other words, using α here actually achieved something: in a sense, it has injected additional information about π into ˆ X (perhaps through some sort of smoothing or regularisation), and brought us closer to π than we already were a priori .
G ENERALISATION IN GAN S
How do GANs ﬁt into this framework?
Most GAN algorithms in widespread use adhere to the following template.
They take as input a distribution P , from which we assume we can sample, and compute (or approximate)
[Equation]
for some choices Q ⊆ P ( X ) and D Γ : P ( X ) × P ( X ) → R .
In other words, a GAN takes a distribution P from which only sampling is possible, and (in the ideal case) outputs the D Γprojection of P onto Q .
In practice, Q denotes corresponds to the set of pushforward measures G ( ν ) obtained from a ﬁxed noise distribution ν on a noise space Z and some set G of functions
G : Z → X .
That is,
[Equation]
Usually G corresponds to the set of functions realisable by some neural network architecture, and ν is some multivariate uniform or Gaussian distribution.
For example, the original GAN formulation (Goodfellow et al., 2014) took D Γ to be the JensonShannon divergence, whereas the f -GAN (Nowozin et al., 2016) generalised this to arbitrary fdivergences, and the Wasserstein GAN (Arjovsky et al., 2017) advocated the Wasserstein distance.
These works also made various recommendations about the choice G through various neural network architectural considerations.
Many of the results proved in these papers involve showing that
(usually under some assumptions, such as sufﬁcient discriminator capacity) a proposed objective for
G is in fact equivalent to D Γ ( P, G ◦ ν ) .
For instance, the original GAN paper showed that
In terms of our framework in the previous section, then, using a GAN Γ amounts to choosing
[Equation]
It is a crucial point that, in most cases (e.g. (Radford et al., 2015)), Γ receives ˆ X as input rather than π itself: we only have access to ﬁnitely many CIFAR-10 samples, for example.
Moreover, training
GANs usually involves making many passes over the same dataset, so that, in effect, sampling from
P returns many duplicate points over the run.
We would not expect this to occur with nonzero probability if P = π for most π of interest.
The observation that P is ˆ X rather than π itself was recently made by Arora et al.
The authors argue that this introduces a problem for the ability of GANs to generalise, since, if D Γ is a divergence (which is almost always the case), and if Q is too big (in particular, if it is big enough that ˆ X ∈ Q ), then we have trivially that Γ( ˆ X ) =
ˆ X .
In other words, the GAN objective appears to actively encourage Γ( ˆ X ) to memorise the dataset X , and never to produce novel samples from outside it.
The authors’ proposed remedy involves trying to ﬁnd a better choice of D Γ .
The problem, they argue, is that popular choices of D Γ do not allow the approximation D Γ ( π, Q ) ≈ D Γ ( ˆ X, Q ) .
They claim a good D Γ is one that satisﬁes the condition (at least with high probability):
[Equation]
small whenever
[Equation]
small
[Equation]
They point out that this is certainly violated when D Γ is the Jensen-Shannon divergence, since
[Equation]
when one of P and Q is discrete and the other continuous.
They give a similar result for the Wasserstein distance in the case that π is Gaussian.
As a solution, they introduce the neural network distance D NN , deﬁned by
[Equation]
for some choice of a class of functions F .
They show that, assuming some smoothness conditions on the members of F , the choice D Γ = D NN satisﬁes (2) with high probability given a fairly modest sample size n .
However, (2) does not seem to capture an appealing notion of generalisation for GANs.
Ultimately what matters is not that D Γ generalises, but that Γ( ˆ X ) does.
Indeed, ﬁxating solely on (2) invites choosing D Γ so as to bring Q artiﬁcially close to π when Q is close to ˆ π (with “closeness” measured in terms of d ).
This does not seem intuitively desirable; we see, for instance, that the degenerate choice D 0 ( P, Q ) :
= 0 trivially satisﬁes (2).
Moreover, we also note that, just like D NN , D 0 is a pseudometric, and it is therefore unclear what mathematical properties of D NN render it more suitable for estimating π than the obviously undesirable D 0 .
The authors do somewhat acknowledge this shortcoming of D NN , pointing out that, for instance, D NN ( P, Q ) can be small even if P and Q are “quite different” (in some sense).
The issues raised in (Arora et al., 2017) apply only in the case that Q is too large.
In practice, however, Q is heavily restricted (since G is restricted via a choice of neural network architecture), so that a priori we do not know whether Γ( ˆ X )
=
ˆ X is even possible.
As such, we do not see the choice of α ( X ) =
Γ( ˆ X ) necessarily as a bad idea.
In fact, this choice falls perfectly within the framework of minimum distance estimation (MDE)
(Wolfowitz, 1957; Basu et al., 2011), which involves estimating an underlying distribution by minimising a distance measure to a given empirical distribution.
We believe therefore that it is an open question as to how well GANs actually work.
In particular, we propose to consider whether GANs work in the sense described in the previous section.
In other words, what is the value of D (Γ( ˆ X ) , π ) ?
We turn to this question in the next section.
T ESTING GAN S
Our goal in this section is to compute D ( α ( X ) , π ) for α ( X ) =
Γ( ˆ X ) .
This raises some difﬁculties, given that π is intractable.
Our approach is to take D to be the ﬁrst Wasserstein distance W d deﬁned by
[Equation]
where d is a metric on X referred to as the ground metric , and Π( P, Q ) denotes the set of joint distributions on the product space
X × X with marginals P and Q .
Before proceeding, we point out that it still makes sense to choose D = W d even if D Γ (cid:54) =
W d .
In our setup, D deﬁnes our end goal, whereas D Γ simply deﬁnes a means to (possibly) achieve that goal, namely by allowing us to project onto Q .
If our goal is to minimise W d ( α ( X ) , π ) , it is still conceivable that the original GAN formulation might be useful, even though in this case D Γ is the
Jenson-Shannon divergence – perhaps the choice of Q regularises ˆ X in such a way that brings us closer in W d sense also.
Our choice of the Wasserstein distance means that D is sensitive to the topology of the underlying set X , which we control by our choice of d .
Moreover, W d metricizes weak convergence for the
Wasserstein space P d ( X ) deﬁned by
[Equation]
(Villani, 2008).
Consequently, if we denote by A a set of samples a 1 , · · · , a m iid ∼ α ( X ) , and by
Y a set of samples (separate from X ) y 1 , · · · , y k iid ∼ π , with ˆ A
and ˆ Y the corresponding empirical distributions, then, provided
[Equation]
we have that D ( ˆ A, ˆ Y )
→ D ( α ( X ) , π ) almost surely as min { m, k } → ∞ .
Note that condition (3) holds automatically in the case that ( X , d ) is compact, since then P d ( X )
=
P ( X ) .
As such, to estimate D ( α ( X ) , π ) , for D = W d , we propose the following.
Before training, we divide all the samples from π that we have into a training set X and a testing set Y .
We train our GAN on X , obtaining α ( X ) .
We then take samples A from α ( X ) , and obtain an estimate of D ( α ( X ) , π ) by computing D ( ˆ A, ˆ Y ) , which can be done exactly by solving a linear program (Villani, 2003).
We can also use the same methodology to estimate D ( ˆ X, π ) by D ( ˆ X, ˆ Y ) , which allows us to check whether 1 holds by checking whether
[Equation]
4. R ESULTS
We used our methodology to test two popular GAN varieties – the DCGAN (Radford et al., 2015) and the Improved Wasserstein GAN (I-WGAN)
(Gulrajani et al., 2017) – on the MNIST and CIFAR10 datasets.
In all cases, A and Y consisted of 10000 samples.
4.1. L 2 AS GROUND METRIC
Initially, we took our ground metric d to be the L 2 distance, which has the appealing property of making ( X , d ) compact when X is a space of RGB or greyscale images, therefore ensuring that (3) holds.
We found that this choice produced nice results for the I-WGAN.
Figure 1 shows the value of W L 2 ( ˆ G, ˆ Y ) over multiple training runs for MNIST and CIFAR-10.
In both cases, this curve decays towards an asymptote that nicely corresponds to the visual quality of the samples produced, examples of which are shown in Figures 6 and 7 in the appendix.
Moreover, we see that W L 2 ( ˆ G, ˆ Y ) is much closer to W L 2 ( ˆ X, ˆ Y ) towards the tail end of training for MNIST than for CIFAR-10.
This seems to reﬂect the fact that, visually, the eventual MNIST samples produced by the I-WGAN do seem to match more closely the true MNIST distribution than the I-WGAN CIFAR-10 samples, which are easily identiﬁed as “fake”.
[Figure] [Figure]
However, we did not obtain such a nice result when we ran the same experiment on CIFAR-10 samples produced by a DCGAN.
In fact, the W L 2 ( ˆ G, ˆ Y ) curves produced, shown in Figure 2 apparently show that ˆ G is closer to ˆ Y than ˆ X very early on in training, at around batch 500 .
As observed in
Figure 8 in the appendix, the visual quality of samples at this stage is very poor.
This is both intriguing and concerning: our metric considers 10000 samples whose appearance resembles the heavily blurred Figure 8b to be closer to ˆ Y than 10000 actual π samples sourced from X .
Motivated by the visual blurriness of the samples in Figure 8b
[Figure]
we explored the effect on W L p ( ˆ X, ˆ Y ) of blurring the CIFAR-10 training set X .
In particular, we let X and Y each consist of 10000 distinct CIFAR-10 samples in X .
We then independently convolved each channel of each image with a Gaussian kernel having standard deviation σ , obtaining a blurred dataset β σ ( X ) and corresponding empirical distribution β σ ( ˆ X ) .
The visual effect of this procedure is shown in Figure 9 in the appendix.
We then computed W L p ( β σ ( ˆ X ) , ˆ Y ) with σ ranging between 0 and 10 for a variety of values of p .
The results of this experiment in the case p = 2 are shown in Figure 3, and similar results were observed for other values of p : in all cases, we found that
[Equation]
whenever σ > 0 .
That is, blurring X by any amount brings ˆ X closer to ˆ Y in W L p than not.
This occurs even though X is distributed identically to Y (both being drawn from π ), while β σ ( X ) (presumably) is not when σ > 0 .
[Figure]
L 2 ( (
X ) , Y )
W L 2 ( ( X ) , Y )
4.1. E MBEDDED L 2 AS GROUND METRIC
To remedy these issues, we sought to replace L 2 with a choice of d that is more naturally suited to the space of images in question.
Our idea involves mapping X through a ﬁxed pre-trained neural network η into a feature space Y , and then computing distances using some metric d Y on Y , rather than in X directly.
It is easily seen that, provided η is injective, d Y ◦ η : X × X → R deﬁned by
[Equation]
is a metric.
It also holds that, when η is ( d, d Y ) -continuous, ( X , d Y ◦ η ) is compact when ( X , d ) is: given a sequence x i ⊆
X , there exists a subsequence x i (cid:48) that converges in d to some x (by compactness), so that d Y ◦ η ( x i (cid:48) , x ) =
d Y ( η ( x i (cid:48) ) , η ( x ))
→ 0 by continuity of η .
Neither of these conditions on η – injectivity and ( d, d Y ) -continuity – are unreasonable to expect from a typical neural network trained using stochastic gradient descent (at least, when d Y is a typical metric such as an
L p distance).
Consequently, W d Y ◦ η constitutes a valid metric on P d Y ◦ η ( X ) , and (3) is automatically satisﬁed.
To test the performance of W d Y ◦ η , we repeated the blurring experiment described above.
We took η ( x ) to be the result of scaling x ∈ X to size 224x224, mapping the result through a DenseNet- (Huang et al., 2016) pre-trained on ImageNet (Deng et al., 2009), and extracting features immediately before the linear output layer.
Under the same experimental setup as above otherwise, we obtained the plot of W L 2 ◦ η ( β σ ( ˆ X ) , ˆ Y ) shown in Figure 3.
Happily, we now see that this curve increases monotonically as σ grows in accordance with the declining visual quality of β σ ( X ) shown in Figure 9.
Next, we reran our earlier experiments, this time computing W L 2 ◦ η ( ˆ A, ˆ Y ) over the course of training.
In all cases, we observed that the value of this quantity reﬂected very well the visual quality of the training samples.
For the I-WGAN, we used the same η as above, and obtained the results on MNIST and CIFAR-10 shown in Figure 4.
(Note that on MNIST we modiﬁed ﬁrst duplicated each input across three channels for shape compatibility with the DenseNet.)
For the DCGAN on
CIFAR-10, we obtained the curve shown in Figure 5.
In all cases we observe that W L 2 ◦ η
( ˆ A, ˆ Y ) decreases monotonically towards an asymptote in a way that accurately summarises the visual quality of the samples throughout the training run.
Moreover, in all cases there is a large gap between the eventual value of W L 2 ◦ η
( ˆ A, ˆ Y ) and W L 2 ◦ η
( ˆ X, ˆ Y ) , which reﬂects the fact that the GAN samples produced are ultimately distinguishable from real samples from π .
In particular, we see an improvement in this respect for the I-WGAN on MNIST: previously, the ﬁnal value of W L 2 ( ˆ A, ˆ Y ) was essentially the same as W L 2 ( ˆ X, ˆ Y ) , despite the fact that it is still quite easy to distinguish between real and generated samples (see e.g. the various mistakes that are discernible throughout Figure 6)
D ISCUSSION AND FUTURE WORK
We believe our work reveals two promising avenues of future inquiry.
First, we suggest that W L p ◦ η is an appealing choice of D , both due to its nice theoretical properties – it metricizing weak con-
[Figure] [Figure]
vergence, and does not require us to make any density assumptions about π – and due to its sound empirical performance demonstrated above.
It would be very interesting to use this D to produce a systematic and objective comparison of the performance of all current major GAN implementations, and indeed to use this as a metric for assessing future GAN designs.
We also view the test (4) as potentially useful for determining whether our algorithms are overﬁtting.
This would be particularly so if applied via a cross-validation procedure: if we consistently observe that (4) holds when training a GAN according to many different X and Y partitions of our total π samples, then it seems reasonable to infer that α ( X ) has indeed learnt something useful about π .
We also believe that the empirical inadequacy of W L 2 that we observed suggests a path towards a better WGAN architecture.
At present, WGAN implementations implicitly use W L 2 for their choice of D Γ .
We suspect that altering this to our suggested W L 2 ◦ η may yield better quality samples.
We brieﬂy give here one possible way to do so that is largely compatible with existing WGAN setups.
In particular, following Arjovsky et al.
(2017), we take
[Equation]
for a class F of functions f : X → R that are all ( L 2 ◦ η, d R ) -Lipschitz for some ﬁxed Lipschitz constant K .
Here d R denotes the usual distance on R .
To construct such an F in practice, we can simply consider a space F (cid:48) of ( d Y , d R ) -Lipschitz functions with ﬁxed Lipschitz constant L , which may be done for instance via weight-clipping like Arjovsky et al.
(2017), or via a gradient penalty like Gulrajani et al.
Provided η is itself ( d, d Y ) -Lipschitz with some Lipschitz constant M (which, for a neural network with ﬁxed weights, can easily be checked), we then obtain a KMLipschitz function f : X → R by taking
[Equation]
Moreover, for our suggested η , this still enables us to compute the gradient of f via the chain rule, since η is a neural network and therefore assumed to be differentiable.
We have begun some experimentation in this area but leave a more detailed empirical inquiry to future work.
It is also clearly important to establish better theoretical guarantees for our method.
At present, we have no guarantee that the number of samples in A and Y are enough to ensure that
[Equation]
(perhaps with some ﬁxed bias that is fairly independent of α , so that it is valid to use the value of
D ( ˆ A, ˆ Y ) to compare different choices of α ), or that (4) entails (1) with high probability.
We do however note that some recent theoretical work on the convergence rate of empirical Wasserstein estimations (Weed & Bach, 2017) does suggest that it is plausible to hope for fast convergence of
D ( ˆ A, ˆ Y ) to D ( α ( X ) , ˆ Y ) .
We also believe that the convincing empirical behaviour of W L 2 ◦ η does suggest that it is possible to say something more substantial about our approach, which we leave to future work.
R ELATED W ORK
The maximum mean discrepancy (MMD) is another well-known notion of distance on probability distributions, which has been used for testing whether two distributions are the same or not (Gretton et al., 2012) and alo for learning generative models in the style of GAN (Li et al., 2015; Dziugaite et al., 2015;
Sutherland et al., 2016; Li et al., 2017).
It is parameterized by a characteristic kernel k , and deﬁnes the distance between probability distributions by means of the distance of k ’s reproducing kernel Hilbert space (RKHS).
The MMD induces the same weak topology on distributions as the one of the Wasserstein distance.
Under a mild condition, the MMD between distributions P and Q under a kernel k can be understood as the outcome of the following two-step calculation.
First, we push-forward P and Q from their original space X to a Hilbert space H (isomorphic to k ’s RKHS) using a feature function φ : X → H induced by the kernel k .
Typically, H is an inﬁnite -dimensional space, such as the set (cid:96) 2 of sqaure-summable sequences in R ∞ as in Mercer’s theorem.
Let P (cid:48) and Q (cid:48) be the resulting distributions on the feature space.
Second, we compute the supremum of E
P (cid:48)
( X ) [ f ( X )] − E Q (cid:48)
( X ) [ f ( X )] over all linear 1 -Lipschitz functions f : H → R .
The result of this calculation is the MMD between P and Q .
This two-step calculation shows the key difference between MMD and our use of Wasserstein distance and neural embedding η .
While the co-domain of the feature function φ is an inﬁnitedimensional space (e.g. (cid:96) 2 ) in most cases, its counterpart η in our setting uses a ﬁnite -dimensional space as co-domain.
This means that the MMD possibly uses a richer feature space than our approach.
On the other hand, the MMD takes the supremum over only linear functions f among all 1 -Lipschitz functions, whereas the Wasserstein distance considers all these 1 -Lipschitz functions.
These difference balancing acts between the expressiveness of features and that of functions taken in the supremum affect the learning and testing of various GAN approaches as observed experimentally in the literature.
One interesting future direction is to carry out systematic study on the theoretical and practical implications of these differences.
A E XAMPLE I-WGAN SAMPLES
[Figure]
[Figure] [Figure]
E XAMPLE DCGAN SAMPLES
[Figure]
C E FFECT OF BLURRING CIFAR-
[Figure]