R EAL OR N OT R EAL , THAT IS THE Q UESTION
Anonymous authors
Paper under double-blind review
A BSTRACT
While generative adversarial networks (GAN) have been widely adopted in various topics, in this paper we generalize the standard GAN to a new perspective by treating realness as a random variable that can be estimated from multiple angles.
In this generalized framework, referred to as RealnessGAN, the discriminator outputs a distribution as the measure of realness.
While RealnessGAN shares similar theoretical guarantees with the standard GAN, it provides more insights on adversarial learning.
More importantly, compared to multiple baselines, RealnessGAN provides stronger guidance for the generator, achieving improvements on both synthetic and real-world datasets.
Moreover, it enables the basic DCGAN (Radford et al., 2015) architecture to generate realistic images at 1024*1024 resolution when trained from scratch.
I NTRODUCTION
The development of generative adversarial network (GAN) (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017) is one of the most important topics in computer vision since its ﬁrst appearance in (Goodfellow et al., 2014).
It learns a discriminator along with the target generator in an adversarial manner, where the discriminator distinguishes generated samples from real ones.
Due to its ﬂexibility when dealing with high dimensional data, GAN has obtained remarkable progresses on realistic image generation (Brock et al., 2019).
In the standard formulation (Goodfellow et al., 2014), the realness of an input sample is estimated by the discriminator using a single scalar .
However, take images for instance, we naturally perceive images from more than one angles and deduce whether it is life-like based on multiple criteria.
As shown in Fig.
1, when a portrait is given, one might focus on its facial structure, skin tint, hair texture and even details like iris and teeth if allowed, each of which indicates a different degree of realness.
Based on this observation, the single scalar could be viewed as an abstract or a summarization of multiple measures, which together reﬂect the overall realisticity of an image.
Such a concise measurement may convey insufﬁcient information to guide the generator, potentially leading to wellknown issues such as mode-collapse and gradient vanishing.
In this paper, we propose to generalize the standard framework (Goodfellow et al., 2014) by treating realness as a distribution rather than a single scalar, which we refer to as RealnessGAN.
The learning process of RealnessGAN abide by the standard setting, but in a distributional form.
While the standard GAN can be viewed as a special case of RealnessGAN, RealnessGAN as well guarantees converging to a Nash-equilibrium where the generator and the discriminator reaches optimalities.
[Figure]
Moreover, by expanding the scalar realness score into a distributional one, the discriminator D naturally provides stronger guidance to the generator G where G needs to match not only the overall realness (as in the standard GAN), but the underlying realness distribution as well.
Consequently,
RealnessGAN facilitate G to better approximate the real data manifold while generating decent samples.
As shown in the experiments, based on a rather simple DCGAN architecture, RealnessGAN could successfully learn from scratch to generate realistic images at 1024*1024 resolution.
R EALNESS GAN 2. G ENERATIVE A DVERSARIAL N ETWORKS
Generative adversarial network jointly learns a generator G and a discriminator D , where G attempts to generate samples that are indistinguishable from the real ones, and D classiﬁes generated and real samples.
In the original work of (Goodfellow et al., 2014), the learning process of D and G follows a minimax game with value function V ( G, D ) :
[Equation]
where the generator’s distribution p g over data x is deﬁned by a prior p z on input latent variables and G .
As proved by Goodfellow et al.
(2014), under such a learning objective, the optimal D satisﬁes D ∗ G ( x ) = p data [Equation]
x [Equation] p data ( x )+ p g ( x ) for a ﬁxed G .
Fixing D at its optimal, the optimal G satisﬁes p g = p data .
The theoretical guarantees provide strong supports for GAN’s success in many applications (Radford et al., 2015; Yu et al., 2017; Zhu et al., 2017), and inspired multiple variants (Arjovsky et al., 2017; Mao et al., 2017; Zhao et al., 2017; Berthelot et al., 2017) to improve the original design.
Nevertheless, a single scalar is constantly adopted as the measure to distinguish generated samples from real ones, while the concept of realness is essentially a random variable covering multiple factors, such as texture and overall conﬁguration in the case of images.
In this work, we intend to follow this observation, enforcing the discriminator D to learn a realness distribution.
2. A D ISTRIBUTIONAL V IEW ON R EALNESS
We start by substituting the scalar output of a discriminator D with a distribution p realness , so
that for an input sample x , D ( x ) = { p realness ( x , u ); u ∈ Ω } , where Ω is the set of outcomes of p realness .
While 0 and 1 in equation 1 are used as two virtual ground-truth scalars that respectively represent the realness of real and fake images, we also need two virtual ground-truth distributions to respectively stand for the realness distributions of real and fake images. We refer to these two distributions as A 1 (real) and A 0 (fake), which are also deﬁned on Ω . As in the standard GAN where we can replace 0 and 1 with other scalars such as − 1 and 1 , there are various choices for A 1 and A 0 .
And we will discuss later what factors lead to a good pair of A 1 and A 0 .
In addition, the difference between two scalars is replaced with the difference between two distributions, i.e. the Kullback-Leibler (KL) divergence.
Subsequently, the minimax game between a generator G and a distributional discriminator D becomes
[Equation]
An immediate observation is that if we let p realness be a discrete distribution with two outcomes { u 0 , u 1 } , and set A 0 ( u 0 )
=
A 1 ( u 1 ) =
and A 0 ( u 1 ) =
A 1 ( u 0 ) = 0 , the updated objective in equation 3 can be explicitly converted to the original objective in equation 1, suggesting RealnessGAN is a generalized version of the original GAN.
Following this observation, we then extend the standard theoretical analysis to the case of RealnessGAN.
Similar to Goodfellow et al.
(2014), our analysis concerns the space of probability density functions, where D and G are assumed to have inﬁnite capacity.
We start from ﬁnding the optimal realness discriminator D for any given generator G .
Theorem 1.
When G is ﬁxed, for any outcome u and input sample x , the optimal discriminator D satisﬁes
[Equation]
Proof.
The objective of D , given a ﬁxed G , is to minimize V ( G, D ) , which is equivalent to:
[Equation]
where C is a term irrelevant to D .
For any x ∈ Supp ( p data ) ∪ Supp ( p g ) , when the KL divergence achieves its minimum, D obtains its optimal, which concludes the proof.
A full proof is available in the appendix.
Next, we move on to the conditions for G to reach its optimal when D = D
(cid:63)G .
Theorem 2.
When D = D (cid:63)G , and there exists an outcome u ∈ Ω such that A 1 ( u ) (cid:54)
=
A 0 ( u ) , the maximum of V ( G, D (cid:63)G ) is achieved if and only if p g = p data .
Proof.
When p g = p data , D (cid:63)G ( x , u ) =
A 1 ( u )+ A 0 ( u ), we have:
[Equation]
Subtracting V (cid:63) ( G, D
(cid:63)G ) from V ( G, D
(cid:63)G ) gives:
[Equation]
Since V (cid:63) ( G
[Equation]
D (cid:63)G ) is a constant with respect to G
[Equation]
maximizing V ( G, D
(cid:63)G ) is equivalent to maximizing V (cid:48)
( G, D (cid:63)G ) .
The optimal V (cid:48) ( G, D
(cid:63)G ) is achieved if and only if the KL divergence reaches its minimum
[Equation]
where:
[Equation]
for any valid x and u .
Hence, as long as there exists a valid u that A 1 ( u ) (cid:54) =
A 0 ( u )
[Equation]
we have p data = p g for any valid x .
2. D ISCUSSION
The theoretical analysis gives us more insights on RealnessGAN.
Number of outcomes: according to equation 12, each u ∈ Ω with A 0 ( u ) (cid:54) =
A 1 ( u ) may work as a constraint, pushing p g towards p data .
In the case of discrete distributions, along with the increment of the number of outcomes, the constraints imposed on G accordingly become more rigorous and can cost G more effort to learn.
This is due to the fact that using more outcomes suggests the shape of the realness distribution is more ﬁne-grained for G to match.
In experiment, we veriﬁed that it is beneﬁcial to update G an increasing number of times before D ’s update as the number of outcomes grows.
Effectiveness of anchors: view equation 12 as a cost function to minimize, when p data (cid:54) =
p g , for some u ∈ Ω , the larger the difference between A 1 ( u ) and A 0 ( u ) is, the stronger the constraint on G becomes.
Intuitively, RealnessGAN can be more efﬁciently trained if we choose A
and A 1 to be adequately different.
Objective of G : according to equation 3, the best way to fool D is to increase the KL divergence between D ( x ) and the anchor distribution A 0 of fake samples, rather than decreasing the KL divergence between D ( x ) and the anchor distribution A 1 of real samples.
It’s worth noting that these two objectives are equivalent in the original work (Goodfellow et al., 2014).
An intuitive explanation is that, in the distributional view of realness, real samples do not necessarily share the same realness distribution.
It is possible that each of them corresponds to a distinct one.
While A 1 only serves as an anchor, it is ineffective to drag all generated samples towards the same target.
Flexibility of RealnessGAN: as a generalization of the standard framework, it is straightforward to integrate RealnessGAN with different GAN architectures, such as progressive GANs (Karras et al., 2018; 2019) and conditional GANs (Zhu et al., 2017; Ledig et al., 2017).
Moreover, one may also combine the perspective of RealnessGAN with other reformulations of the standard GAN, such as replacing the KL divergence in equation 3 with the Earth Mover’s Distance.
2. I MPLEMENTATION
In our implementation, the realness distribution p realness is characterized as a discrete distribution over N outcomes Ω = { u 0 , u 1 , ..., u N − 1 } .
Given an input sample x , the discriminator D returns N probabilities on these outcomes, following:
[Equation]
(13) where ψ = ( ψ 0 , ψ 1 , ..., ψ N − 1 ) are the parameters of D .
Similarly, A 1 and A 0 are discrete distributions deﬁned on Ω .
As shown in the theoretical analysis, the ideal objective for G is maximizing the KL divergence between D ( x ) of generated samples and A 0 : [Equation] However, as the discriminator D is not always at its optimal, especially in the early stage, directly applying this objective in practice could only lead to a generator with limited generative power.
Consequently, a regularizer is needed to improve G . There are several choices for the regularizer, such as the relativistic term introduced in (Jolicoeur-Martineau, 2019) that minimizes the KL divergence between D ( x ) of generated samples and random real samples, or the term that minimizes the KL divergence between A 1 and D ( x ) of generated samples, and they lead to different updated objectives:
[Equation]
We have compared these objectives in Appendix A, where these two regularizers could lead to similar results. In the experiment section, the objective in equation 15 is adopted.
Feature resampling.
In practice, especially in the context of images, we are learning from a limited number of discrete samples coming from a continuous data manifold.
We may encounter issues caused by insufﬁcient data coverage during the training process.
Inspired by conditioning augmentation mentioned in (Zhang et al., 2016), we introduce a resampling technique performed on the realness output to augment data variance.
Given a mini-batch { x 0 , ..., x M − 1 } of size M , a Gaussian distribution N ( µ i , σ i ) is ﬁtted on { ψ i ( x 0 ) , ψ i ( x 1 ) , ..., ψ i ( x M − 1 ) } , which are logits computed by D on i -th outcome.
We then resample M new logits { ψ (cid:48)
[Equation]
} for i -th outcome and use them succeedingly.
The randomness introduced by resampling beneﬁts the training of RealnessGAN in two aspects.
First of all, it augments data by probing instances around the limited training samples, leading to more robust models.
Secondly, the resampling approach implicitly demand instances of ψ i ( x ) to be homologous throughout the mini-batch, such that each outcome reﬂects realness from a consistent angle across samples.
We empirically found the learning curve of RealnessGAN is more stable if feature resampling is utilized, especially in the latter stage, where models are prone to overﬁt.
R ELATED W ORK
Generative adversarial network (GAN) was ﬁrst proposed in (Goodfellow et al., 2014), which jointly learns a discriminator D and a generator G in an adversarial manner.
Due to its outstanding learning ability, GANs have been adopted in various generative tasks (Radford et al., 2015; Yu et al., 2017;
Zhu et al., 2017), among which Deep Convolutional GAN (DCGAN)
(Radford et al., 2015) has shown promising results in image generation.
Although remarkable progress has been made.
GAN is known to suffer from gradient diminishing and mode collapse.
Variants of GAN have been proposed targeting these issues.
Speciﬁcally,
Wasserstein GAN (WGAN) Arjovsky et al.
(2017) replaces JS-divergence with Earth-Mover’s Distance, and Least-Square GAN (LSGAN) (Mao et al., 2017) transforms the objective of G to Pearson divergence.
Energy-based GAN (EBGAN) (Zhao et al., 2017) and Boundary Equilibrium GAN (BEGAN) (Berthelot et al., 2017) employ a pre-trained auto-encoder as the discriminator, learning to distinguish between real and generated samples via reconstruction.
Besides adjusting the objective of GAN, alternative approaches include more sophisticated architectures and training paradigms.
Generally, ProgressiveGAN (Karras et al., 2018) and StyleGAN (Karras et al., 2019) propose a progressive paradigm, which starts from a shallow model focusing on low resolution level, and gradually grows into a deeper model to incorporate more details as resolution grows.
On the other hand,
COCO-GAN (Lin et al., 2019) tackle high resolution image generation in a divide-and-conquer strategy.
It learns to produce decent patches at corresponding sub-regions, and splices the patches to produce a higher resolution image.
It’s worth noting that many works on generative adversarial networks have discussed ‘distributions’ (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017), which usually refers to the underlying distribution of samples.
Some of the existing works aim to improve the original objective using different metrics to measure the divergence between the learned distribution p g and the real distribution p data .
Nevertheless, a single scalar is constantly adopted to represent the concept of realness.
In this paper, we propose a complementary modiﬁcation that models realness as a random variable follows the distribution p realness .
In the future work, we may study the combination of realness discriminator and other GAN variants to enhance the effectiveness and stability of adversarial learning.
E XPERIMENTS
In this section we study RealnessGAN from multiple aspects.
Speciﬁcally, 1) we ﬁrstly focus on
RealnessGAN’s mode coverage ability on a synthetic dataset. 2)
Then we evaluate RealnessGAN on
CIFAR10 (32*32) (Krizhevsky, 2009) and CelebA (256*256) (Liu et al., 2015) datasets qualitatively and quantitatively.
Finally we explore RealnessGAN on high-resolution image generation task, which is known to be challenging for unconditional non-progressive architectures.
Surprisingly, on the FFHQ dataset (Karras et al., 2019), RealnessGAN managed to generate images at the 1024* resolution based on a non-progressive architecture.
We compare RealnessGAN to other popular objectives in generative adversarial learning, including the standard GAN ( Std-GAN )
(Radford et al., 2015),
WGAN-GP
(Arjovsky et al., 2017), HingeGAN (Zhao et al., 2017) and LSGAN (Mao et al., 2017).
For experiments on synthetic dataset, we use a generator with four fully-connected hidden layers, each of which has 400 units, followed by batch normalization and ReLU activation.
The discriminator has three fully-connected hidden layers, with 200 units each layer.
LinearMaxout with 5 maxout pieces are adopted and no batch normalization is used in the discriminator.
The latent input z is a
On real-world datasets, the network architecture is identical to the DCGAN architecture in Radford et al.
(2015), with the prior p z ( z ) a 128-dimensional Gaussian distribution N ( 0 , I ) .
Models are trained using Adam (Kingma & Ba, 2015) for 520 k iterations.
To guarantee training stability, we adopt settings that are proved to be effective for baseline methods.
Batch normalization (Ioffe &
Szegedy, 2015) is used in G , and spectral normalization (Miyato et al., 2018) is used in D .
For
WGAN-GP we use lr = 1 e − 4 , β 1 = 0 .
5 , β 2 = 0 .
9 , updating D for 5 times per G ’s update (Gulrajani et al., 2017); for the remaining models, we use lr = 2 e − 4 , β 1 = 0 .
5 , β 2 = 0 .
999 , updating D for one time per G ’s update (Radford et al., 2015).
Fr´echet Inception Distance (FID) (Heusel et al., 2017) and Sliced Wasserstein Distance (SWD) (Karras et al., 2018) are reported as the evaluation metrics.
Unless otherwise stated, A 1 and A 0 are chosen to resemble the shapes of two normal distributions with a positive skew and a negative skew, respectively.
4. S YNTHETIC D ATASET
Since p data is usually intractable on real datasets, we use a toy dataset to compare the learned distribution p g and the data distribution p data .
The toy dataset consists of 100 , 000 2D points sampled from a mixture of 9 isotropic Gaussian distributions whose means are arranged in a 3 by 3 grid, with variances equal to 0 .
As shown in Fig.2, the data distribution p data contains 9 welly separated modes, making it a difﬁcult task despite its low-dimensional nature.
To evaluate p g , we draw 10 , 000 samples and measure their quality and diversity.
As suggested in (Dumoulin et al., 2016), we regard a sample as of high quality if it is within 4 σ from the µ of its nearest Gaussian.
When a Gaussian is assigned with more than 100 high quality samples, we consider this mode of p data is recovered in p g .
Fig.2 visualizes the sampled points of different methods, where LSGAN and HingeGAN suffer from signiﬁcant mode collapse, recovering only a single mode.
Points sampled by WGAN-GP are overly disperse, and only 0 .
03% of them are of high quality.
While Std-GAN recovers 4 modes in p data with 32 . 4% high quality samples, 7 modes are recovered by RealnessGAN with 60 . 2% high quality samples.
The average σ s of these high quality samples in Std-GAN and RealnessGAN are respectively 0 .
083 and 0 .
The results suggest that treating realness as a random variable rather than a single scalar leads to a more strict discriminator that criticizes generated samples from various aspects, which provides more informative guidance.
Consequently, p g learned by RealnessGAN is more diverse and compact.
We further study the effect of adjusting the number of outcomes in the realness distribution p realness on this dataset.
To start with, we ﬁx k G and k D to be 1 , which are the number of updates for G and D in one iteration, and adjust the number of outcomes used in p realness , A 0 and A 1 .
As shown in the ﬁrst row of Fig.3, it can be observed that in general G recovers less modes as the number of outcomes grows, which is a direct result of D becoming increasingly powerful and imposing more constraints on G .
When D is too powerful for G , we could increase k G to allow G to catch up with current D .
The second row of Fig.3 shows the converged cases when k G have been adjusted to suitable settings, which suggests the effectiveness of RealnessGAN when G is given sufﬁcient learning space.
The ratio of high quality samples r HQ and the number of recovered modes n mode in these cases are plotted in Fig.3.
The two curves imply that besides k G , r HQ and n mode are all positively related to the number of outcomes, validating that measuring realness from more aspects leads to a better generator.
4. R EAL - WORLD D ATASETS
As GAN has shown promising results when modeling complex data such as natural images, we evaluate RealnessGAN on real-world datasets, namely CelebA,CIFAR10 and FFHQ, which respectively contains images at the 32*32, 256*256 and 1024*1024 resolutions.
The training curves of baseline methods and RealnessGAN on CelebA and CIFAR10 are showed in Fig.4.
The qualitative results measured in FID and SWD are listed in Tab.1.
We report the minimum, the maximum, the mean and the standard deviation computed along the training process.
On both datasets, compared to baselines, RealnessGAN obtains better scores in both metrics.
Meantime, the learning process of
RealnessGAN is smoother and steadier (see SD in Tab.1 and curves in Fig.4).
Samples of generated images on both datasets are included in Fig.5. [Figure]
r e c o v e r e d m o d e s
[Figure] On FFHQ, we push the resolution of generated images to 1024*1024, which is known to be challenging especially for a non-progressive architecture.
As shown in Fig.5, despite build on a relatively simple DCGAN architecture, RealnessGAN is able to produce realistic samples from scratch at such a high resolution. Quantitatively, applying RealnessGAN to DCGAN obtains a FID of 17 . 18 . For reference, our re-implemented StyleGAN
(Karras et al., 2019) with a similar setting receives a FID of 16 .
These results strongly support the effectiveness of RealnessGAN, as StyleGAN is one of the most advanced GAN architectures.
The implementation of RealnessGAN offers several choices that also worth studying.
On synthetic dataset, we explored the relationship between the number of outcomes and G ’s update frequency.
On
CelebA dataset, apart from evaluating RealnessGAN as a whole, we also studied the effectiveness of feature resampling and the affect of using different settings of A 0 and A 1 .
Speciﬁcally, Fig.6 shows the training curves of RealnessGAN with and without feature resampling.
It can be noticed that [Table] [Table] resampling signiﬁcantly stabilizes the training process especially in the latter stage.
Tab. reports the results of exploiting different anchor distributions by varying the KL divergence between
A 0 and A 1 .
FID score indicates that, as the KL divergence between A 0 and A 1 increases, RealnessGAN tends to perform better, which veriﬁes our discussion in Sec.2.3 that a larger difference between anchor distributions gives rise to stronger constraints on G . [Table]
[Figure]
C ONCLUSION
In this paper we extend the view of realness in generative adversarial networks under a distributional perspective.
In our proposed extension, RealnessGAN, we represent the concept of realness as a realness distribution rather than a single scalar.
so that the corresponding discriminator estimates realness from multiple angles, providing more informative guidance to the generator.
We prove
RealnessGAN has theoretical guarantees on the optimality of the generator and the discriminator.
On both synthetic and real-world datasets, RealnessGAN also demonstrates the ability of effectively and steadily capturing the underlying data distribution
A O BJECTIVE OF G
As mentioned in Sec.2. in theoretical the objective of G is max G E x ∼ p g
[ D KL ( A 0 (cid:107) D ( x ))] .
However, in practice since D is not always optimal, to train with such an objective we need either an additional constraint or a pair of A 0 and A 1 that has a large KL divergence. We show in Fig. that when the KL divergence between A 0 and A 1 is sufﬁciently large on CelebA we could obtain a generator with limited generative power. As mentioned in Sec.2.
we provide two alternative choices for the additional constraint, resulting in equation 15 and equation 16.
In Fig.7 we show the training curves of RealnessGAN on CelebA using these two alternative objectives.
Moreover, in
Tab.3 we include the results of RealnessGAN on CIFAR10 using these two objectives as well as the ideal objective. On both datasets, we can see these two constraints lead to comparable results [Table] [Figure]
[Table] [Figure] [Table] [Figure]
B F ULL PROOF ON T HEOREM
Proof.
Given a ﬁxed G , the objective of D is:
[Equation]
where h ( A 1 ) and h ( A 0 ) are their entropies, and the ﬁrst term in equation 19 is irrelevant to D , marked as C 1 .
The objective thus is equivalent to:
[Equation]
where p x ( u ) = p data ( x ) A 1 ( u )+ p g ( x ) A 0 ( u ) p data ( x )+ p g ( x ) is a distribution deﬁned on Ω u .
Consequently, let C 2 =
p data ( x ) + p g ( x ) , we have
[Equation]
From equation 23 we can see, for any x ∈ Supp ( p data ) ∪ Supp ( p g ) , when D KL ( p x (cid:107) D ( x )) achieves its minimum, D obtains its optimal D (cid:63) .
And at that time, we have D (cid:63) ( x ) =
p x , which concludes the proof.
C I NTERPRETATION ON E STIMATED R EALNESS D ISTRIBUTION
In Fig.9, we include some generated images and their estimated realness distributions.
Intuitively, images with similar distributions tend to have something in common, such as unrealistic foregrounds (left) and unrealistic backgrounds (right).
D FFHQ S AMPLES