R EAL OR N OT R EAL , THAT IS THE Q UESTION
A BSTRACT
While generative adversarial networks (GAN) have been widely adopted in various topics, in this paper we generalize the standard GAN to a new perspective by treating realness as a random variable that can be estimated from multiple angles.
In this generalized framework, referred to as RealnessGAN 1 , the discriminator outputs a distribution as the measure of realness.
While RealnessGAN shares similar theoretical guarantees with the standard GAN, it provides more insights on adversarial learning.
Compared to multiple baselines, RealnessGAN provides stronger guidance for the generator, achieving improvements on both synthetic and real-world datasets.
Moreover, it enables the basic DCGAN (Radford et al., 2015) architecture to generate realistic images at 1024*1024 resolution when trained from scratch.
I NTRODUCTION
The development of generative adversarial network (GAN) (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017) is one of the most important topics in machine learning since its ﬁrst appearance in (Goodfellow et al., 2014).
It learns a discriminator along with the target generator in an adversarial manner, where the discriminator distinguishes generated samples from real ones.
Due to its ﬂexibility when dealing with high dimensional data, GAN has obtained remarkable progresses on realistic image generation (Brock et al., 2019).
In the standard formulation (Goodfellow et al., 2014), the realness of an input sample is estimated by the discriminator using a single scalar .
However, for high dimensional data such as images, we naturally perceive them from more than one angles and deduce whether it is life-like based on multiple criteria.
As shown in Fig.1, when a portrait is given, one might focus on its facial structure, skin tint, hair texture and even details like iris and teeth if allowed, each of which indicates a different aspect of realness.
Based on this observation, the single scalar could be viewed as an abstract or a summarization of multiple measures, which together reﬂect the overall realness of an image.
Such a concise measurement may convey insufﬁcient information to guide the generator, potentially leading to well-known issues such as mode-collapse and gradient vanishing.
In this paper, we propose to generalize the standard framework (Goodfellow et al., 2014) by treating realness as a random variable, represented as a distribution rather than a single scalar. We refer to [Figure] such a generalization as RealnessGAN.
The learning process of RealnessGAN abide by the standard setting, but in a distributional form.
While the standard GAN can be viewed as a special case of RealnessGAN, RealnessGAN and the standard GAN share similar theoretical guarantees. i.e. RealnessGAN converges to a Nash-equilibrium where the generator and the discriminator reach their optimalities.
Moreover, by expanding the scalar realness score into a distributional one, the discriminator D naturally provides stronger guidance to the generator G where G needs to match not only the overall realness (as in the standard GAN), but the underlying realness distribution as well.
Consequently, RealnessGAN facilitates G to better approximate the data manifold while generating decent samples.
As shown in the experiments, based on a rather simple DCGAN architecture, RealnessGAN could successfully learn from scratch to generate realistic images at 1024*1024 resolution.
R EALNESS GAN 2. G ENERATIVE A DVERSARIAL N ETWORKS
Generative adversarial network jointly learns a generator G and a discriminator D , where G attempts to generate samples that are indistinguishable from the real ones, and D classiﬁes generated and real samples.
In the original work of (Goodfellow et al., 2014), the learning process of D and G follows a minimax game with value function V ( G, D ) : [Equation]
[Equation]
where the approximated data distribution p g is deﬁned by a prior p z on input latent variables and G .
As proved by Goodfellow et al.
, under such a learning objective, the optimal D satisﬁes D ∗ G ( x ) =
[Equation]
) for a ﬁxed G .
Fixing D at its optimal, the optimal G satisﬁes p g = p
data . The theoretical guarantees provide strong supports for GAN’s success in many applications (Radford et al., 2015; Yu et al., 2017; Zhu et al., 2017; Dai et al., 2017), and inspired multiple variants (Arjovsky et al., 2017; Mao et al., 2017; Zhao et al., 2017; Berthelot et al., 2017) to improve the original design.
Nevertheless, a single scalar is constantly adopted as the measure of realness, while the concept of realness is essentially a random variable covering multiple factors, e.g. texture and overall conﬁguration in the case of images.
In this work, we intend to follow this observation, encouraging the discriminator D to learn a realness distribution.
2. A D ISTRIBUTIONAL V IEW ON R EALNESS
We start by substituting the scalar output of a discriminator D with a distribution p realness , so
that for an input sample x , D ( x ) = { p realness ( x , u ); u ∈ Ω } , where Ω is the set of outcomes of p realness .
Each outcome u can be viewed as a potential realness measure, estimated via some criteria.
While 0 and 1 in equation 2 are used as two virtual ground-truth scalars that respectively represent the realness of real and fake images, we also need two virtual ground-truth distributions to stand for the realness distributions of real and fake images.
We refer to these two distributions as A 1 (real) and A 0 (fake), which are also deﬁned on Ω .
As in the standard GAN where 0 and 1 can be replaced with other scalars such as − 1 and 1 , there are various choices for A 1 and A 0 .
Factors lead to a good pair of Aand A 0 will be discussed later.
Accordingly, the difference between two scalars is replaced with the Kullback-Leibler (KL) divergence.
The minimax game between a generator G and a distributional discriminator D thus becomes
[Equation]
An immediate observation is that if we let p realness be a discrete distribution with two outcomes { u 0 , u 1 } , and set A 0 ( u 0 )
=
A 1 ( u 1 ) =
and A 0 ( u 1 ) =
A 1 ( u 0 ) = 0 , the updated objective in equation 3 can be explicitly converted to the original objective in equation 2, suggesting RealnessGAN is a generalized version of the original GAN.
Following this observation, we then extend the theoretical analysis in Goodfellow et al. (2014) to the case of RealnessGAN.
Similar to Goodfellow et al.
(2014), our analysis concerns the space of probability density functions, where D and G are assumed to have inﬁnite capacities.
We start from ﬁnding the optimal realness discriminator D for any given generator G [Equation]
Theorem 1.
When G is ﬁxed
[Equation]
for any outcome u and input sample x [Equation] the optimal discriminator D satisﬁes
[Equation]
Proof.
Given a ﬁxed G , the objective of D is:
[Equation]
where h ( A 1 ) and h ( A 0 ) are their entropies. Marking the ﬁrst term in equation 7 as C 1 since it is irrelevant to D , the objective thus is equivalent to: [Equation]
where p x ( u ) = p data ( x ) A 1 ( u )+ p g ( x ) A 0 ( u ) p data ( x )+ p g ( x ) is a distribution deﬁned on Ω . Let C 2 = p data ( x ) + p g ( x ) , we then have
[Equation] Observing equation 10, one can see that for any u ∈ Ω , when D KL ( p x (cid:107) D ( x )) achieves its minimum, D obtains its optimal D (cid:63) , leading to D (cid:63) ( x ) = p x , which concludes the proof.
Next, we move on to the conditions for G to reach its optimal when D = D
(cid:63)G .
Theorem 2.
When D = D (cid:63)G , and there exists an outcome u ∈ Ω such that A 1 ( u ) (cid:54)
=
A 0 ( u ) , the maximum of V ( G, D (cid:63)G ) is achieved if and only if p g = p data .
Proof.
When p [Equation] = p [Equation]
[Equation] , we have:
[Equation]
Subtracting V (cid:63) ( G, D
(cid:63)G ) from V ( G, D
(cid:63)G ) gives:
[Equation]
Since V (cid:63) ( G, D (cid:63)G ) is a constant with respect to G , maximizing V ( G, D
(cid:63)G ) is equivalent to maximizing V (cid:48)
( G, D (cid:63)G ) .
The optimal V (cid:48) ( G, D
(cid:63)G ) is achieved if and only if the KL divergence reaches its minimum, where:
[Equation]
for any valid x and u .
Hence, as long as there exists a valid u that A 1 ( u ) (cid:54) =
A 0 ( u ) , we have p data =
p g for any valid x .
2. D ISCUSSION
The theoretical analysis gives us more insights on RealnessGAN.
Number of outcomes: according to equation 16, each u ∈ Ω with A 0 ( u ) (cid:54) =
A 1 ( u ) may work as a constraint, pushing p g towards p data .
In the case of discrete distributions, along with the increment of the number of outcomes, the constraints imposed on G accordingly become more rigorous and can cost G more effort to learn.
This is due to the fact that having more outcomes suggests a more ﬁnegrained shape of the realness distribution for G to match.
In Sec.4, we veriﬁed that it is beneﬁcial to update G an increasing number of times before D ’s update as the number of outcomes grows.
Effectiveness of anchors: view equation 16 as a cost function to minimize, when p data (cid:54) =
p g , for some u ∈ Ω , the larger the difference between A 1 ( u ) and A 0 ( u ) is, the stronger the constraint on G becomes.
Intuitively, RealnessGAN can be more efﬁciently trained if we choose A
and A 1 to be adequately different.
Objective of G : according to equation 3, the best way to fool D is to increase the KL divergence between D ( x ) and the anchor distribution A 0 of fake samples, rather than decreasing the KL divergence between D ( x ) and the anchor distribution A 1 of real samples.
It’s worth noting that these two objectives are equivalent in the original work (Goodfellow et al., 2014).
An intuitive explanation is that, in the distributional view of realness, realness distributions of real samples are not necessarily identical.
It is possible that each of them corresponds to a distinct one.
While A 1 only serves as an anchor, it is ineffective to drag all generated samples towards the same target.
Flexibility of RealnessGAN: as a generalization of the standard framework, it is straightforward to integrate RealnessGAN with different GAN architectures, such as progressive GANs (Karras et al., 2018; 2019) and conditional GANs (Zhu et al., 2017; Ledig et al., 2017).
Moreover, one may also combine the perspective of RealnessGAN with other reformulations of the standard GAN, such as replacing the KL divergence in equation 3 with the Earth Mover’s Distance.
2. I MPLEMENTATION
In our implementation, the realness distribution p realness is characterized as a discrete distribution over N outcomes Ω
= { u 0 , u 1 , ..., u N − 1 } .
Given an input sample x , the discriminator D returns N probabilities on these outcomes, following:
[Equation]
where ψ = ( ψ 0 , ψ 1 , ..., ψ N − 1 ) are the parameters of D .
Similarly, A 1 and A 0 are discrete distributions deﬁned on Ω .
As shown in the theoretical analysis, the ideal objective for G is maximizing the KL divergence between D ( x ) of generated samples and A 0 :
[Equation]
However, as the discriminator D is not always at its optimal, especially in the early stage, directly applying this objective in practice could only lead to a generator with limited generative power.
Consequently, a regularizer is needed to improve G .
There are several choices for the regularizer, such as the relativistic term introduced in (Jolicoeur-Martineau, 2019) that minimizes the KL divergence between D ( x ) of generated samples and random real samples, or the term that minimizes the KL divergence between A 1 and D ( x ) of generated samples, each of which leads to a different objective:
[Equation]
In Sec.4, these objectives are compared.
And the objective in equation 19 is adopted as the default choice.
Feature resampling.
In practice, especially in the context of images, we are learning from a limited number of discrete samples coming from a continuous data manifold.
We may encounter issues caused by insufﬁcient data coverage during the training process.
Inspired by conditioning augmentation mentioned in (Zhang et al., 2016), we introduce a resampling technique performed on the realness output to augment data variance.
Given a mini-batch { x 0 , ..., x M − 1 } of size M , a Gaussian distribution N ( µ i , σ i ) is ﬁtted on { ψ i ( x 0 ) , ψ i ( x 1 ) , ..., ψ i ( x M − 1 ) } , which are logits computed by D on i -th outcome.
We then resample M new logits { ψ (cid:48) i ( x 0 ) , ..., ψ
[Equation]
i ( x M − 1 ); ψ (cid:48) i ∼ N ( µ i , σ i ) } for i -th outcome and use them succeedingly.
The randomness introduced by resampling beneﬁts the training of RealnessGAN in two aspects.
First of all, it augments data by probing instances around the limited training samples, leading to more robust models.
Secondly, the resampling approach implicitly demands instances of ψ i ( x ) to be homologous throughout the mini-batch, such that each outcome reﬂects realness consistently across samples.
We empirically found the learning curve of RealnessGAN is more stable if feature resampling is utilized, especially in the latter stage, where models are prone to overﬁt.
R ELATED W ORK
Generative adversarial network (GAN) was ﬁrst proposed in (Goodfellow et al., 2014), which jointly learns a discriminator D and a generator G in an adversarial manner.
Due to its outstanding learning ability, GANs have been adopted in various generative tasks (Radford et al., 2015; Yu et al., 2017;
Zhu et al., 2017), among which Deep Convolutional GAN (DCGAN)
(Radford et al., 2015) has shown promising results in image generation.
Although remarkable progress has been made.
GAN is known to suffer from gradient diminishing and mode collapse.
Variants of GAN have been proposed targeting these issues.
Speciﬁcally,
Wasserstein GAN (WGAN) Arjovsky et al.
(2017) replaces JS-divergence with Earth-Mover’s Distance, and Least-Square GAN (LSGAN) (Mao et al., 2017) transforms the objective of G to Pearson divergence.
Energy-based GAN (EBGAN) (Zhao et al., 2017) and Boundary Equilibrium GAN (BEGAN) (Berthelot et al., 2017) employ a pre-trained auto-encoder as the discriminator, learning to distinguish between real and generated samples via reconstruction.
Besides adjusting the objective of GAN, alternative approaches include more sophisticated architectures and training paradigms.
Generally, ProgressiveGAN (Karras et al., 2018) and StyleGAN (Karras et al., 2019) propose a progressive paradigm, which starts from a shallow model focusing on a low resolution, and gradually grows into a deeper model to incorporate more details as resolution grows.
On the other hand,
COCO-GAN (Lin et al., 2019) tackles high resolution image generation in a divide-and-conquer strategy.
It learns to produce decent patches at corresponding sub-regions, and splices the patches to produce a higher resolution image.
It’s worth noting that many works on generative adversarial networks have discussed ‘distributions’ (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017), which usually refers to the underlying distribution of samples.
Some of the existing works aim to improve the original objective using different metrics to measure the divergence between the learned distribution p g and the real distribution p data .
Nevertheless, a single scalar is constantly adopted to represent the concept of realness.
In this paper, we propose a complementary modiﬁcation that models realness as a random variable follows the distribution p realness .
In the future work, we may study the combination of realness discriminator and other GAN variants to enhance the effectiveness and stability of adversarial learning.
E XPERIMENTS
In this section we study RealnessGAN from multiple aspects.
Speciﬁcally, 1) we ﬁrstly focus on
RealnessGAN’s mode coverage ability on a synthetic dataset. 2)
Then we evaluate RealnessGAN on
CIFAR10 (32*32) (Krizhevsky, 2009) and CelebA (256*256) (Liu et al., 2015) datasets qualitatively and quantitatively.
Finally we explore RealnessGAN on high-resolution image generation task, which is known to be challenging for unconditional non-progressive architectures.
Surprisingly, on the FFHQ dataset (Karras et al., 2019), RealnessGAN managed to generate images at the 1024* resolution based on a non-progressive architecture.
We compare RealnessGAN to other popular objectives in generative adversarial learning, including the standard GAN ( Std-GAN )
(Radford et al., 2015),
WGAN-GP
(Arjovsky et al., 2017), HingeGAN (Zhao et al., 2017) and LSGAN (Mao et al., 2017).
For experiments on synthetic dataset, we use a generator with four fully-connected hidden layers, each of which has 400 units, followed by batch normalization and ReLU activation.
The discriminator has three fully-connected hidden layers, with 200 units each layer.
LinearMaxout with 5 maxout pieces are adopted and no batch normalization is used in the discriminator.
The latent input z is a 32 -dimensional vector sampled from a Gaussian distribution N ( 0 , I ) . All models are trained using Adam (Kingma & Ba, 2015) for 500 iterations.
On real-world datasets, the network architecture is identical to the DCGAN architecture in Radford et al.
(2015), with the prior p z ( z ) a 128-dimensional Gaussian distribution N ( 0 , I ) .
Models are trained using Adam (Kingma & Ba, 2015) for 520 k iterations.
To guarantee training stability, we adopt settings that are proved to be effective for baseline methods.
Batch normalization (Ioffe &
Szegedy, 2015) is used in G , and spectral normalization (Miyato et al., 2018) is used in D .
For
WGAN-GP we use lr = 1 e − 4 , β 1 = 0 .
5 , β 2 = 0 .
9 , updating D for 5 times per G ’s update (Gulrajani et al., 2017); for the remaining models, we use lr = 2 e − 4 , β 1 = 0 .
5 , β 2 = 0 .
999 , updating D for one time per G ’s update (Radford et al., 2015).
Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) and Sliced Wasserstein Distance (SWD) (Karras et al., 2018) are reported as the evaluation metrics.
Unless otherwise stated, A 1 and A 0 are chosen to resemble the shapes of two normal distributions with a positive skewness and a negative skewness, respectively.
In particular, the number of outcomes are empirically set to 51 for CelebA and FFHQ datasets, and 3 for CIFAR dataset.
4. S YNTHETIC D ATASET
Since p data is usually intractable on real datasets, we use a toy dataset to compare the learned distribution p g and the data distribution p data .
The toy dataset consists of 100 , 000 2D points sampled from a mixture of 9 isotropic Gaussian distributions whose means are arranged in a 3 by 3 grid, with variances equal to 0 .
As shown in Fig.2, the data distribution p data contains 9 welly separated modes, making it a difﬁcult task despite its low-dimensional nature.
To evaluate p g , we draw 10 , 000 samples and measure their quality and diversity.
As suggested in (Dumoulin et al., 2016), we regard a sample as of high quality if it is within 4 σ from the µ of its nearest Gaussian.
When a Gaussian is assigned with more than 100 high quality samples, we consider this mode of p data is recovered in p g .
Fig.2 visualizes the sampled points of different methods, where LSGAN and HingeGAN suffer from signiﬁcant mode collapse, recovering only a single mode.
Points sampled by WGAN-GP are overly disperse, and only 0 .
03% of them are of high quality.
While Std-GAN recovers 4 modes in p data with 32 . 4% high quality samples, 8 modes are recovered by RealnessGAN with 60 . 2% high quality samples.
The average σ s of these high quality samples in Std-GAN and RealnessGAN are respectively 0 .
083 and 0 .
The results suggest that treating realness as a random variable rather than a single scalar leads to a more strict discriminator [Figure] that criticizes generated samples from various aspects, which provides more informative guidance.
Consequently, p g learned by RealnessGAN is more diverse and compact.
We further study the effect of adjusting the number of outcomes in the realness distribution p realness on this dataset.
To start with, we ﬁx k G and k D to be 1 , which are the number of updates for G and D in one iteration, and adjust the number of outcomes of p realness , A 0 and A 1 .
As shown in the ﬁrst row of Fig.3, it can be observed that in general G recovers less modes as the number of outcomes grows, which is a direct result of D becoming increasingly rigorous and imposing more constraints on G .
An intuitive solution is to increase k G such that G is able to catch up with current D .
The second row of Fig.3 demonstrates the converged cases achieved with suitable k G s, suggesting RealnessGAN is effective when sufﬁcient learning capacity is granted to G .
The ratio of high quality samples r HQ and the number of recovered modes n mode in these cases are plotted in Fig.3.
The two curves imply that besides k G , r HQ and n mode are all positively related to the number of outcomes, validating that measuring realness from more aspects leads to a better generator.
4. R EAL - WORLD D ATASETS
As GAN has shown promising results when modeling complex data such as natural images, we evaluate RealnessGAN on real-world datasets, namely CIFAR10, CelebA and FFHQ, which respectively contains images at 32*32, 256*256 and 1024*1024 resolutions.
The training curves of baseline methods and RealnessGAN on CelebA and CIFAR10 are shown in Fig.4.
The qualitative results measured in FID and SWD are listed in Tab.1.
We report the minimum, the maximum, the mean and the standard deviation computed along the training process.
On both datasets, compared to baselines, RealnessGAN obtains better scores in both metrics.
Meantime, the learning process of
RealnessGAN is smoother and steadier (see SD in Tab.1 and curves in Fig.4).
Samples of generated images on both datasets are included in Fig.8.
On FFHQ, we push the resolution of generated images to 1024*1024, which is known to be challenging especially for a non-progressive architecture.
As shown in Fig.8, despite building on a relatively simple DCGAN architecture, RealnessGAN is able to produce realistic samples from scratch at such a high resolution.
Quantitatively, RealnessGAN obtains an FID score of 17 . 18 .
For reference, our [Figure] [Table]
re-implemented StyleGAN (Karras et al., 2019) trained under a similar setting receives an FID score of 16 .
These results strongly support the effectiveness of RealnessGAN , as StyleGAN is one of the most advanced GAN architectures so far.
4. A BLATION S TUDY
The implementation of RealnessGAN offers several choices that also worth digging into.
On synthetic dataset, we explored the relationship between the number of outcomes and G ’s update frequency.
On real-world dataset, apart from evaluating RealnessGAN as a whole, we also studied the affect of feature resampling, different settings of A 0 and A 1 and choices of G ’s objective. [Figure] [Equation]
[Figure] [Table] [Figure]
[Table] [Figure]
Feature Resampling.
Fig.5 shows the training curves of RealnessGAN with and without feature resampling.
It can be noticed that despite the results are similar, feature resampling stabilizes the training process especially in the latter stage.
Effectiveness of Anchors. Tab.2 reports the results of varying the KL divergence between {+anchor distributions A 
+}
and A 1 .
The FID score indicates that, as the KL divergence between A 0 and Aincreases, RealnessGAN tends to perform better, which veriﬁes our discussion in Sec.2.3 that a larger difference between anchor distributions imposes stronger constraints on G .
To further testify, two different pairs of anchors with similar KL divergences ( 11 . 95 and 11 . 67 ) are exploited and they yield comparable FID scores ( 23 . 98 and 24 . 22 ).
Objective of G. As mentioned in Sec.2.3, theoretically, the objective of G is max G E x ∼ p g
[ D KL ( A 0 (cid:107) D ( x ))] . However, in practice, since D is not always optimal, we need either a pair of A 0 and A 1 that are drastically different, or an additional constraint to aid this objective. Fig.6 shows that, with the ideal objective alone, when the KL divergence between A 0 and A 1 is sufﬁciently large, on CelebA we could obtain a generator with limited generative power.
On the other hand, by applying constraints as discussed in Sec.2.4, G can learn to produce more realistic samples as demonstrated in Fig.8. Similar results are observed on CIFAR10, where RealnessGAN obtains comparable FID scores with and without constraints, as shown in Tab.3. Fig.7 also provides the training curves of RealnessGAN on CelebA using these two alternative objectives.
C ONCLUSION
In this paper, we extend the view of realness in generative adversarial networks under a distributional perspective.
In our proposed extension, RealnessGAN, we represent the concept of realness as a realness distribution rather than a single scalar.
so that the corresponding discriminator estimates realness from multiple angles, providing more informative guidance to the generator.
We prove
RealnessGAN has theoretical guarantees on the optimality of the generator and the discriminator.
On both synthetic and real-world datasets, RealnessGAN also demonstrates the ability of effectively and steadily capturing the underlying data distribution.
Acknowledgement We thank Zhizhong Li for helpful discussion on the theoretical analysis.
This work is partially supported by the Collaborative Research Grant of ”Large-scale
Multi-modality
Analytics” from SenseTime (CUHK Agreement No. TS1712093), the General Research Funds (GRF) of Hong Kong (No. 14209217 and No. 14205719),
Singapore MOE AcRF Tier 1, NTU
SUG, and NTU NAP.