T HE RELATIVISTIC DISCRIMINATOR : A KEY ELEMENT MISSING FROM STANDARD GAN
A BSTRACT
In standard generative adversarial network (SGAN), the discriminator D estimates the probability that the input data is real.
The generator G is trained to increase the probability that fake data is real.
We argue that it should also simultaneously decrease the probability that real data is real because 1) this would account for a priori knowledge that half of the data in the mini-batch is fake, 2) this would be observed with divergence minimization, and 3) SGAN would be more similar to integral probability metric (IPM) GANs.
We show that this property can be induced by using a “relativistic discriminator” which estimate the probability that the given real data is more realistic than a randomly sampled fake data.
We also present a variant in which the discriminator estimate the probability that the given real data is more realistic than fake data, on average.
We generalize both approaches to non-standard GAN loss functions and we refer to them respectively as Relativistic GANs (RGANs) and Relativistic average GANs (RaGANs).
We show that IPM-based GANs are a subset of RGANs which use the identity function.
Empirically, we observe that 1) RGANs and RaGANs are signiﬁcantly more stable and generate higher quality data samples than their non-relativistic counterparts, 2)
Standard RaGAN with gradient penalty generate data of better quality than
WGAN-GP while only requiring a single discriminator update per generator update (reducing the time taken for reaching the state-of-the-art by 400%), and 3) RaGANs are able to generate plausible high resolutions images (256x256) from a very small sample (N=2011), while GAN and LSGAN cannot; these images are of signiﬁcantly better quality than the ones generated by WGAN-GP and SGAN with spectral normalization. The code is freely available on https://github.com/AlexiaJM/RelativisticGAN.
I NTRODUCTION
Generative adversarial networks (GANs) (Hong et al., 2017) form a broad class of generative models in which a game is played between two competing neural networks, the discriminator D and the generator G .
D is trained to discriminate real from fake data, while G is trained to generate fake data that D will mistakenly recognize as real.
In the original GAN by Goodfellow et al.
(2014), which we refer to as Standard GAN (SGAN)
, D is a classiﬁer, thus it is predicting the probability that the input data is real.
When D is optimal, the loss function of SGAN is approximately equal to the
Jensen–Shannon divergence (JSD) (Goodfellow et al., 2014).
SGAN has two variants for the generator loss functions: saturating and non-saturating.
In practice, the former has been found to be very unstable, while the latter has been found to more stable (Goodfellow et al., 2014).
Under certain conditions, Arjovsky & Bottou (2017) proved that, if real and fake data are perfectly classiﬁed, the saturating loss has zero gradient and the non-saturating loss has non-zero, but volatile gradient.
In practice, this means that the discriminator in SGAN often cannot be trained to optimality or with a too high learning rate; otherwise, gradients may vanish and, if so, training will stop.
This problem is generally more noticeable in high-dimensional setting (e.g., high resolution images and discriminator architectures with high expressive power) given that there are enough degrees of freedom available to perfectly classify the training set.
To improve on SGAN, many GAN variants have been suggested using different loss functions and discriminators that are not classiﬁers (e.g., LSGAN (Mao et al., 2017), WGAN (Arjovsky et al., 2017)).
Although these approaches have partially succeeded in improving stability and data quality, the large-scale study by Lucic et al.
(2017) suggests that these approaches do not consistently improve on SGAN.
Additionally, some of the most successful approaches, such as WGAN-GP (Gulrajani et al., 2017), are much more computationally demanding than SGAN.
Many of the recent successful GANs variants have been based on Integral probability metrics (IPMs) (Müller, 1997)
(e.g., WGAN , WGAN-GP, Fisher GAN (Mroueh & Sercu, 2017), Sobolev GAN (Mroueh et al., 2017)).
In IPM-based GANs, the discriminator is real-valued and constrained to a speciﬁc class of function which regularize the discriminator.
See Mroueh et al.
(2017) for a review of the different IPMs.
These IPM constraints have been shown to be beneﬁcial even in non-IPM based GANs.
Spectral normalization (Miyato et al., 2018) improves the stability of various GANs and it consists in making the discriminator Lipschitz-1, which is the constraint of WGAN.
Similarly, the gradient penalty of
WGAN-GP also provides improve the stability of SGAN (Fedus et al., 2017).
Although this shows that certain IPM constraints improve the stability of GANs, it does not explain why IPM-based GANs generally provide increased stability over other metrics/divergences in GANs (e.g., JSD for SGAN, f -divergences for f -GANs
(Nowozin et al., 2016)).
Note that although powerful, IPM-based GANs tend to more computationally demanding than other
GANs.
Certain IPM-based GANs use a gradient penalty (e.g. WGAN-GP, Sobolev GAN) which is very computationally costly and most IPM-based GANs need more than one discriminator update per generator update (WGAN-GP requires at least 5 (Gulrajani et al., 2017)).
Assuming equal training time for D and G , every additional discriminator update increase training time by a signiﬁcant 50%.
In this paper, we argue that non-IPM-based GANs are missing a key ingredient, a relativistic discriminator, which IPM-based GANs already possess.
We show that a relativistic discriminator is necessary to make GANs analogous to divergence minimization and produce sensible predictions based on the a priori knowledge that half of the samples in the mini-batch are fake.
We provide empirical evidence showing that GANs with a relativistic discriminator are more stable and produce data of higher quality.
B ACKGROUND 2. G ENERATIVE ADVERSARIAL NETWORKS
GANs can be deﬁned very generally in terms of the discriminator in the following way:
[Equation]
[Equation]
and
[Equation]
[Equation] ) where ˜ f 1 , ˜ f 2 , ˜ g 1 , ˜ g 2 are scalar-to-scalar functions, P is the distribution of real data, P z is generally a multivariate normal distribution centered at 0 with variance 1, D ( x ) is the discriminator evaluated at x , G ( z ) is the generator evaluated at z ( Q is the distribution of fake data, thus of G ( z ) ).
Note that, through the paper, we refer to real data as x r and fake data as x f .
Without loss of generality, we assume that both L D and L G are loss functions to be minimized.
Most GANs can be separated into two classes: non-saturating and saturating loss functions.
GANs with the saturating loss are such that ˜ g 1 = − ˜ f
1 and ˜ g
= − ˜ f 2 , while GANs with the non-saturating loss are such that ˜ g
= ˜ f 2 and ˜ g 2 = ˜ f 1 .
Saturating GANs are intuitive as they can be interpreted as alternating between maximizing and minimizing the same loss function.
After training D to optimality, the loss function is generally an approximation of a divergence (e.g., Jensen–Shannon divergence (JSD) for
SGAN (Goodfellow et al., 2014), f -divergences for F-GANs (Nowozin et al., 2016), and Wassertein distance for WGAN (Arjovsky et al., 2017)).
Thus, training G to minimize L G can be roughly interpreted as minimizing the approximated divergence (although this is not technically true; see
Jolicoeur-Martineau (2018)).
On the other hand, non-saturating GANs can be thought as optimizing the same loss function, but swapping real data with fake data (and vice-versa).
In this article, unless otherwise speciﬁed, we assume a non-saturating loss for all GANs.
SGAN assumes a cross-entropy loss, i.e., ˜ f 1 ( D ( x ))
= − log
( D ( x )) and ˜ f 2 ( D ( x )) =
− log(1 −
D ( x )) , where D ( x ) = sigmoid ( C ( x )) , and C ( x ) is the non-transformed discriminator output (which we call the critic as per Arjovsky et al.
In most GANs, C ( x ) can be interpreted as how realistic the input data is ; a negative number means that the input data looks fake (e.g., in SGAN,
D ( x ) = sigmoid ( − 5) = 0 ), while a positive number means that the input data looks real (e.g., in
SGAN, D ( x ) = sigmoid (5) = 1 ).
2. I NTEGRAL PROBABILITY METRICS
IPMs are statistical divergences represented mathematically as:
[Equation]
where F is a class of real-valued functions.
IPM-based GANs can be deﬁned using equation 1 and 2 where ˜ f 1 ( D ( x ))
=
˜ g 2 ( D ( x ))
= − D ( x ) and ˜ f
( D ( x ))
=
˜ g 1 ( D ( x ))
=
D ( x ) , where D ( x ) =
C ( x ) (i.e., no transformation is applied).
It can be observed that both discriminator and generator loss functions are unbounded and would diverge to −∞ if optimized directly.
However, IPMs assume that the discriminator is of a certain class of function that does not grow too quickly which prevent the loss functions from diverging.
Each IPM applies a different constraint to the discriminator (e.g., WGAN assumes a Lipschitz D , WGAN-GP assumes that D has a gradient norm equal to 1 around real and fake data).
M ISSING PROPERTY OF SGAN
We argue that the key missing property of SGAN is that the probability of real data being real ( D ( x r ) ) should decrease as the probability of fake data being real ( D ( x f ) ) increase.
We provide three arguments suggesting that SGAN should have this property.
3. P RIOR KNOWLEDGE ARGUMENT
Assuming a rational human was shown half real data and half fake data.
If they perceived all samples shown as equally real ( C ( x f ) ≈ C ( x r ) for most x r and x f ), they would assume that each sample has probability .50 of being real.
However, this is not the case for the discriminator in SGAN.
If all samples looked real ( C ( x f ) ≈ C ( x r ) ≥ 3 ), D would assume incorrectly that they are indeed all real ( D ( x ) ≈ 1 for all x ).
Of course, once trained with the labels, D would decrease D ( x f ) and thus would obtain more reasonable estimates.
However, if D ( x r ) decreased as D ( x f ) increased, we would have had that D ( x ) ≈ .
50 for all x before even retraining D .
A rational human would not require retraining.
IPM-based GANs implicitly account for the fact that some of the samples must be fake because they compare how realistic real data is compared to fake data.
This is the behavior that we would want.
3. D IVERGENCE MINIMIZATION ARGUMENT
In SGAN, when optimized, we have that the discriminator loss function is equal to the
Jensen–Shannon divergence (JSD) (Goodfellow et al., 2014).
The JSD is minimized ( JSD ( P || Q ) = 0 ) when D ( x r )
=
D ( x f ) =
12 for all x r ∈ P and x f ∈ Q and maximized ( JSD ( P || Q ) = log(2) )
when D ( x r ) = 1 , D ( x f ) = 0 for all x r ∈ P and x f ∈ Q .
Thus, if we were directly minimizing the divergence from maximum to minimum, we would expect D ( x r ) to smoothly decrease from 1 to .50 for most x r and D ( x f ) to smoothly increase from 0 to .50 for most x f (Figure 1a).
However, when minimizing the saturating loss in SGAN, we are only increasing D ( x f ) , we are not decreasing
D ( x r ) (Figure 1b).
Furthermore, we are bringing D ( x f ) closer to 1 rather than .50.
This means that SGAN dynamics are very different from the minimization of the JSD.
To bring
SGAN closer to divergence minimization, training the generator should not only increase D ( x f ) but
(A) Divergence minimization
D i sc r
i m i na t o r ou t pu t
D ( x ) 0.
Beginning
End (B) Actual generator training 0.
Beginning
End (C) Ideal generator training 0.
Beginning
End
Real data
Fake data also decrease D ( x r ) (Figure 1c). Note that although speciﬁc to the JSD, similar dynamics are true for other divergences; when the divergence is maximal, D ( x r ) and D ( x f ) are very far from one another, but they converge to the same value as the divergence approach zero. Thus, this argument applies to other divergences.
3. G RADIENT ARGUMENT
We compare the gradients of standard GAN and IPM-based GANs for further insight.
It can be shown that the gradients of the discriminator and generator in non-saturating SGAN are respectively:
[Equation]
where J is the Jacobian.
It can be shown that the gradients of the discriminator and generator in IPM-based GANs are respectively:
[Equation]
[Equation]
where C ( x ) ∈ F (the class of functions assigned by the IPM).
From these equations, it can be observed that SGAN leads to the same dynamics as IPM-based GANs when we have that:
D ( x r ) = 0 , D ( x f ) = 1 in the discriminator step of SGAN 2.
D ( x f ) = 0 in the generator step of SGAN.
C ( x ) ∈ F
Assuming that the discriminator and generator are trained to optimality in each step (which we sometimes do for D , but never for G ) and that it is possible to perfectly distinguish real from the fake data (strong assumption, but generally true early in training); we would have that D ( x r ) = 1 ,
D ( x f ) = 0 in the generator step and that D ( x r ) = 1 , D ( x f ) = 1 in the discriminator step for most x r and x f (Figure 1b).
Thus, the only missing assumption would be that D ( x r ) = 0 in the discriminator step.
Although the above scenario is not realistic (because we never train G to optimality), if all the assumptions were respected and the generator could indirectly inﬂuence D ( x r ) , we would have that D ( x r ) = 0 , D ( x f )
= 1 .
Thus, SGAN would have the same gradients as IPM-based GANs.
We conjecture that making SGAN more similar to IPM-based GANs could potentially improve its stability (our results will show that it does in fact improves stability).
M ETHOD 4. R ELATIVISTIC STANDARD GAN
In standard GAN, the discriminator can be deﬁned, in term of the non-transformed layer C ( x ) , as
D ( x ) = sigmoid ( C ( x )) .
A simple way to make discriminator relativistic (i.e., having the output of
D depends on both real and fake data) is to sample from real/fake data pairs ˜ x
= ( x r , x f ) and deﬁne it as D (˜ x )
= sigmoid ( C ( x r ) − C ( x f )) .
We can interpret this modiﬁcation in the following way: the discriminator estimates the probability that the given real data is more realistic than a randomly sampled fake data .
Similarly, we can deﬁne D rev (˜ x )
= sigmoid ( C ( x f ) − C ( x r )) as the probability that the given fake data is more realistic than a randomly sampled real data.
An interesting property of this discriminator is that we do not need to include D rev in the loss function through log(1 − D rev (˜ x )) because we have that 1 − D rev (˜ x ) = 1 − sigmoid ( C ( x f ) − C ( x r ))
=
sigmoid ( C ( x r ) − C ( x f ))
=
D (˜ x ) ; thus, log( D (˜ x ))
= log(1 − D rev (˜ x )) .
The discriminator and generator (non-saturating) loss functions of the Relativistic Standard GAN (RSGAN) can be written as:
[Equation]
4. R ELATIVISTIC GAN S
More generally, we consider any discriminator deﬁned as a ( C ( x r ) − C ( x f )) , where a is the activation function, to be relativistic.
This means that almost any GAN can have a relativistic discriminator.
This forms a new class of models which we call Relativistic GANs (RGANs).
Most GANs can be parametrized very generally in terms of the critic:
[Equation]
and
[Equation]
where f 1 , f 2 , g 1 , g 2 are scalar-to-scalar functions.
If we use a relativistic discriminator, these GANs now have the following form:
[Equation]
and
[Equation]
If one use the identity function (i.e., f 1 ( y ) =
g 2 ( y )
= − y , f 2 ( y ) =
g 1 ( y ) =
y ), this results in a degenerate case since there is no supremum/maximum.
However, if one adds a constraint so that
C ( x r ) − C ( x f ) is bounded, then there is a supremum and one arrives at IPM-based GANs.
Thus, although different, IPM-based GANs share a very similar loss function focused on the difference in critics.
Importantly, g 1 is normally ignored in GANs because its gradient is zero since the generator does not inﬂuence it.
However, in RGANs, g 1 is inﬂuenced by fake data, thus by the generator.
4. R ELATIVISTIC AVERAGE GAN S
The discriminator has a very different interpretation in SGAN compared to RSGAN.
In SGAN, D ( x ) estimates the probability that x is real, while in RGANs, D ( x r , x f ) estimates the probability that x r is more realistic than x f .
As a middle ground, we developed an alternative to the Relativistic
Discriminator, which retains approximately the same interpretation as the discriminator in SGAN while still being relativistic.
We propose the Relativistic average Discriminator (RaD) which compares the critic of the input data to the average critic of samples of the opposite type.
The discriminator loss function for this approach can be formulated as:
[Equation]
) where
[Equation]
RaD has a more similar interpretation to the standard discriminator than the relativistic discriminator.
With RaD, the discriminator estimates the probability that the given real data is more realistic than fake data, on average .
As before, we can generalize this approach to work with any GAN loss function using the following formulation:
[Equation]
We call this general approach Relativistic average GAN (RaGAN).
E XPERIMENTS
Experiments were conducted on the CIFAR-10 dataset (Krizhevsky, 2009) and the CAT dataset (Zhang et al., 2008).
Code was written in Pytorch (Paszke et al., 2017) and models were trained using the Adam optimizer (Kingma & Ba, 2014) for 100K generator iterations with seed 1 (which shows that we did not ﬁsh for the best seed, instead, we selected the seed a priori ).
We report the Fréchet
Inception Distance (FID) (Heusel et al., 2017), a measure that is generally better correlated with data quality than the Inception Distance (Salimans et al., 2016)
(Borji, 2018); lower FID means that the generated images are of better quality.
For the models architectures, we used the standard CNN described by Miyato et al.
(2018) on CIFAR10 and a relatively standard DCGAN architecture (Radford et al., 2015) on CAT (see Appendix).
We also provide the source code required to replicate all analyses presented in this paper (See our repository: [Anonymous until peer review is ﬁnished]).
5. CIFAR-
In these analyses, we compared standard GAN (SGAN), least-squares GAN (LSGAN), Wassertein
GAN improved (WGAN-GP), Hinge-loss GAN (HingeGAN) (Miyato et al., 2018), Relativistic
SGAN (RSGAN), Relativistic average SGAN (RaSGAN), Relativistic average LSGAN (RaLSGAN), and Relativistic average HingeGAN (RaHingeGAN) using the standard CNN architecture on stable setups (See Appendix for details on the loss functions used).
Additionally, we tested RSGAN and
RaSGAN with the same gradient-penalty as WGAN-GP (named RSGAN-GP and RaSGAN-GP respectively).
We used the following two known stable setups: (DCGAN setup)
lr = .
0002 , n D = 1 , β 1 = .and
β 2 = .
999 (Radford et al., 2015), and (WGAN-GP setup) lr
= .
0001 , n D = 5 , β 1 = .
50 and β 2 = .
9 (Gulrajani et al., 2017), where lr is the learning rate, n D is the number of discriminator updates per generator update, and β 1 , β 2 are the ADAM momentum parameters.
For optimal stability, we used batch norm (Ioffe & Szegedy, 2015) in G and spectral norm (Miyato et al., 2018) in D .
Results are presented in Table 1.
We observe that RSGAN and RaSGAN generally performed better than SGAN.
Similarly, RaHingeGAN performed better than HingeGAN.
RaLSGAN performed on par with LSGAN, albeit sightly worse.
WGAN-GP performed poorly in the DCGAN setup, but very well in the WGAN-GP setup.
RasGAN-GP performed poorly; however, RSGAN-GP performed better than all other loss functions using only one discriminator update per generator update.
Importantly, the resulting FID of 25.60 is on par with the lowest FID obtained for this architecture using spectral normalization, as reported by Miyato et al.
Overall, these results show that using a relativistic discriminator generally improve data generation quality and that RSGAN works very well in conjunction with gradient penalty to obtain state-of-the-art results.
[Table]
5. CAT
CAT is a dataset containing around 10k pictures of cats with annotations.
We cropped the pictures to the faces of the cats using those annotations.
After removing outliers (hidden faces, blurriness, etc.), the CAT dataset contained 9304 images ≥ 64x64, 6645 images ≥ 128x128, and 2011 images ≥ 256x256.
The CAT dataset is particularly challenging due to its small sample size and high-resolution images; this makes it perfect for testing the stability of different GAN loss functions.
We trained different GAN loss functions on 64x64, 128x128, 256x256 images.
For 256x images, we compared RaGANs to known stable approaches: SpectralSGAN (SGAN with spectral normalization in D ) and WGAN-GP.
Although some approaches were able to train on 256x images, they did so with signiﬁcant mode collapse.
To alleviate this problem, for 256x256 images, we packed the discriminator (Lin et al., 2017)
(i.e., D took a concatenated pair of images instead of a single image).
We looked at the minimum, maximum, mean and standard deviation (SD) of the FID at 20k, 30k, ..., 100k generator iterations; results are presented in Table 2.
Overall, we observe lower minimum FID, maximum FID, mean and standard deviation (sd) for
RGANs and RaGANs than their non-relativistic counterparts (SGAN, LSGAN, RaLSGAN).
In 64x64 resolution, both SGAN and LSGAN generated images with low FID, but they did so in a very unstable matter.
For example, SGAN went from a FID of 17.50 at 30k iterations, to 310.56 at 40k iterations, and back to 27.72 at 50k iterations.
Similarly, LSGAN went from a FID of 20.27 at 20k iterations, to 224.97 at 30k iterations, and back to 51.98 at 40k iterations.
On the other hand,
RaGANs were much more stable (lower max and SD) while also resulting in lower minimum FID.
Using gradient-penalty did not improve data quality; however, it reduced the SD lower than without gradient penalty, thus increasing stability further.
SGAN was unable to converge on 128x128 or bigger images and LSGAN was unable to converge on 256x256 images.
Meanwhile, RaGANs were able to generate plausible images with low FID in all resolutions.
Although SpectralSGAN and WGAN-GP were able to generate 256x256 images of cats, the samples they generated were of poor quality (high FID).
Thus, in this very difﬁcult setting, relativism provided a greater improvement in quality than gradient penalty or spectral normalization.
C ONCLUSION AND FUTURE WORK
In this paper, we proposed the relativistic discriminator as a way to ﬁx and improve on standard GAN.
We further generalized this approach to any GAN loss and introduced a generally more stable variant called RaD. Our results suggest that relativism signiﬁcantly improve data quality and stability of [Table] [Table] LSGAN
[Table] [Table]
GANs at no computational cost.
Furthermore, using a relativistic discriminator with other tools of the trade (spectral norm, gradient penalty, etc.) may lead to better state-of-the-art.
Future research is needed to fully understand the mathematical implications of adding relativism to
GANs.
Furthermore, our experiments were limited to certain loss functions using only one seed, due to computational constraints.
More experiments are required to determine which relativistic
GAN loss function is best over a wide-range of datasets and hyper-parameters.
We greatly encourage researchers and machine learning enthusiasts with greater computing power to experiment further with our approach.
A PPENDICES A I NTUITIVE AND MEMEFUL VISUAL REPRESENTATION OF R A GAN S
[Table] (Standard
[Table]
) (Relativistic average Standard GAN)
Real image looks real and fake images look fake
C ( x r ) =
C ( x f ) = −
P ( x r is bread ) =
P ( x r is bread | C ( x f )) =
Real image looks real but fake images look similarly real on average
C ( x r ) =
C ( x f ) =
P ( x r is bread ) =
P ( x r is bread | C ( x f ))
= .
Real image looks fake but fake images look more fake on average
C ( x r ) = −
C ( x f ) = −
P ( x r is bread )
= .
P ( x r is bread | C ( x f ))
= .
B I NTUITION BEHIND R A GAN S
Although the relative discriminator provide the missing property that we want in GANs (i.e., G inﬂuencing D ( x r ) ), its interpretation is different from the standard discriminator.
Rather than measuring “the probability that the input data is real”, it is now measuring “the probability that the input data is more realistic than a randomly sampled data of the opposing type (fake if the input is real or real if the input is fake)”.
To make the relativistic discriminator act more globally, as in its original deﬁnition, our initial idea was the following: average the relativistic discriminator over random samples of data of the opposing type.
This can be conceptualized in the following way:
[Equation]
where D ( x r , x f ) = sigmoid ( C ( x r ) − C ( x f )) .
Then, the following loss function for D could be applied:
[Equation]
The main problem with this idea is that it would require looking at all possible combinations of real and fake data in the mini-batch.
This would transform the problem from O ( m ) to O ( m 2 ) complexity, where m is the batch size.
This is problematic; therefore, we do not use this approach.
Instead, we propose to use the Relativistic average Discriminator (RaD) which compares the critic of the input data to the average critic of samples of the opposite type (See section 4.3).
This approach has O ( m ) complexity.
C G RADIENTS C.
[Equation]
∇ w L GAND =
−∇ w E x r ∼ P [log D ( x r )] − ∇ w E x f ∼ Q
θ [log(1 − D ( x f ))]
= −∇ w E x r ∼ P (cid:20) log (cid:18) e C ( x r ) e C ( x r ) +(cid:19)(cid:21) − ∇ w E x f ∼ Q θ (cid:20) log (cid:18) 1 − e C ( x f ) e C ( x f ) +(cid:19)(cid:21) =
−∇ w E x r ∼ P (cid:104) C ( x r ) − log (cid:16) e C ( x r ) + 1 (cid:17)(cid:105) − ∇ w E x f ∼ Q θ (cid:104) log(1) − log (cid:16) e C ( x f ) + 1 (cid:17)(cid:105) =
− E x r ∼ P [ ∇ w C ( x r )]
+ E x r ∼ P (cid:20) e C ( x r ) e C ( x r ) + 1 ∇ w C
( x r ) (cid:21) + E x f ∼ Q
θ (cid:20) e C ( x f ) e C ( x f ) + 1 ∇ w C
( x f ) (cid:21) =
− E x r ∼ P [ ∇ w C ( x r )]
+ E x r ∼ P
[ D ( x r ) ∇ w C ( x r )]
+ E x f ∼ Q
θ [ D ( x f ) ∇ w C ( x f )]
= − E x r ∼ P [(1 − D ( x r )) ∇ w C ( x r )]
+ E x f ∼ Q
θ [ D ( x f ) ∇ w C ( x f )] ∇ θ L GANG =
−∇ θ E z ∼ P
z
[log D ( G ( z ))]
= −∇ θ E z ∼ P z (cid:20) log (cid:18) e C ( G ( z ))
e C ( G ( z ))
+(cid:19)(cid:21) =
[Equation]
θ
[Equation]
∼ P
[Equation]
z ))
[Equation]
[Equation]
[Equation]
=
[Equation]
e
[Equation]
( z ))
[Equation]
=
[Equation]
P
[Equation]
[Equation]
[Equation]
[Equation]
D S IMPLIFIED FORM OF RELATIVISTIC SATURATING AND NON - SATURATING GAN S
The formulation of RGANs can be simpliﬁed when we have the following two properties: (1) f 2 ( − y ) =
f 1 ( y ) and (2) the generator assumes a non-saturating loss ( g 1 ( y ) =
f 2 ( y ) and g 2 ( y ) =
f 1 ( y ) ).
These two properties are observed in standard GAN, LSGAN using symmetric labels (e.g., - and 1), IPM-based GANs, etc.
With these two properties, RGANs with non-saturating loss can be formulated simply as:
[Equation]
) and
[Equation]
Assuming f 2 ( − y )
=
f 1 ( y ) , we have that
[Equation]
If
[Equation]
and g 2 ( y ) = − f 2 ( y ) (saturating GAN), we have that
L RGAN − S G
[Equation]
C
[Equation]
If g 1 ( y ) = f 2 ( y ) and g 2 ( y ) =
f 1 ( y ) (non-saturating GAN), we have that
L RGAN
[Equation]
NS G
[Equation]
x f ) ∼ ( P , Q ) [ f 1 ( C ( x f ) − C ( x r ))] .
E T ESTING THE GRADIENT ARGUMENT
Previously, we argued that SGAN could be equivalent to IPM-GANs under very strict conditions and assumptions.
We mentioned that although most assumptions are reasonable, the assumption that the generator is trained to optimality is unrealistic.
In which case, SGAN would not be equivalent to
IPM-based GANs since D ( x r ) would not reach 0.
As an experiment, we calculated the mini-batch average of D ( x r ) in the ﬁrst 100 iterations of the training for the CAT dataset in 256x256.
Note that SGAN becomes stuck at around 200 iterations and can never go beyond generating noise.
Thus, a difference in the distribution of D ( x r ) could reveal something meaningful about why Relativistic GANs can converge while their non-relativistic counterparts cannot.
In those 100 iterations, we have that the distance between P and Q is maximal since G only generate noise.
Thus, we can perfectly distinguish real from fake data, which is one of the assumptions.
The remaining assumptions were that D and G would be trained to optimality.
Although we did not train
D more than once, after the discriminator step, we generally had that D ( x r )
≈ 1 .
What we wanted to verify is whether D ( x r ) ≈ 0 after the generator step in Relativistic GANs, even though we did not train G enough to reach optimality.
[Figure]
Testing gradient argument (Giters = 1)
[Figure]
[Figure]
[Figure]
RSGAN
RaSGAN
Testing gradient argument (Giters = 2)
Results are shown in Figure 2.
We observe that with only one generator update per discriminator update ( n G = 1 ), RSGAN and RaSGAN never reach an average D ( x r ) of 0 but the distribution is much less concentrated around 1 than with SGAN.
With n G = 2 , RSGAN and RaSGAN sometimes reach an average D ( x r ) of 0 and they form an almost uniform distribution around [0 , 1] .
This suggests that with the missing property (i.e., using Relativistic GANs), SGAN can be made more similar to
IPM-based GANs, but never equivalent.
Thus, Relativistic Standard GANs can be seen as having a dynamic in-between SGAN and IPM-based GANs.
F CIFAR-10 H ARD / UNSTABLE EXPERIMENTS
In these analyses, we compared SGAN, LSGAN, WGAN-GP, RSGAN, RaSGAN, RaLSGAN, and
RaHingeGAN with the standard CNN architecture on unstable setups in CIFAR-10.
Unless otherwise speciﬁed, we used lr = .
0002 , β 1 = .
5 , β 2 = .
999 , n D = 1 , and batch norm (Ioffe & Szegedy, 2015) in G and D .
We tested the following four unstable setups: (1) lr = .
001 , (2) β 1 = .
9 , β 2 = .
9 , (3) no batch norm in G or D , and (4) all activation functions replaced with Tanh in both G and D (except for the output activation function of D ).
Results are presented in Table 4.
We observe that RaLSGAN performed better than LSGAN in all setups.
RaHingeGAN performed slightly worse than HingeGAN in most setups.
RSGAN and
RaSGAN performed better than SGAN in two out of four setups, although differences were small.
WGAN-GP generally performed poorly which we suspect is due to the single discriminator update per generator update.
Overall, this provide good support for the improved stability of using the relative discriminator with LSGAN, but not with HingeGAN and SGAN.
Although results are worse for the relativistic discriminator in some settings, differences are minimal and probably reﬂect natural variations.
It is surprising to observe low FID for SGAN without batch normalization considering its well-known difﬁculty with this setting (Arjovsky et al., 2017).
Given these results, we suspected that CIFARmay be too easy to fully observe the stabilizing effects of using the relative discriminator.
Therefore, in the manuscript, we focused on the more difﬁcult CAT dataset with high resolution pictures.
G L OSS FUNCTIONS USED IN EXPERIMENTS G. SGAN ( NON - SATURATING )
[Equation] [Equation]
G. RSGAN
[Equation]
[Table]
[Table]
[Table] [Table]
G. R A SGAN
[Table] [Table]
[Table]
(cid:0) C ( x r ) − E x f ∼ Q C ( x f ) (cid:1) ˜ D ( x f ) = sigmoid ( C ( x f ) − E x r ∼ P C ( x r ))
G. LSGAN
L LSGAND =
E x r ∼ P (cid:2) ( C ( x r ) − 0) 2 (cid:3) + E x f ∼ Q (cid:2) ( C ( x f ) − 1) 2 (cid:3) (26)
L LSGANG =
E x f ∼ Q (cid:2) ( C ( x f ) − 0) 2 (cid:3) (27)
G. R A LSGAN
L
RaLSGAND
=
E x r ∼ P (cid:2) ( C ( x r ) − E x f ∼ Q C ( x f ) − 1) 2 (cid:3) + E x f ∼ Q (cid:2) ( C ( x f ) − E x r ∼ P C ( x r ) + 1) 2 (cid:3) (28)
L
RaLSGANG
=
E x f ∼ P (cid:2) ( C ( x f ) − E x r ∼ P C ( x r ) − 1) 2 (cid:3) + E x r ∼ P (cid:2) ( C ( x r ) − E x f ∼ Q C ( x f ) + 1) 2 (cid:3) (29)
G. H INGE GAN
L HingeGAND =
E x r ∼ P [max(0 , 1 − C ( x r ))]
+ E x f ∼ Q [max(0 , 1 + C ( x f ))] (30)
L HingeGANG = − E x f ∼ Q
[ C ( x f )] (31)
G. R A H INGE GAN
L
HingeGAN D
=
E x r ∼ P (cid:104) max(0 , 1 − ˜ D ( x r ))
(cid:105)
+ E x f ∼ Q (cid:104) max(0 ,
+ ˜ D
( x f )) (cid:105) (32)
L HingeGANG =
E x f ∼
[Table]
[Table]
[Table]
(33) ˜ D
( x r ) =
C ( x r ) − E x f ∼ Q C
( x f ) ˜ D ( x f )
=
C ( x f ) − E x r ∼ P C ( x r )
G. WGAN-GP
[Equation]
[Equation]
P ˆ x is the distribution of ˆ x = (cid:15)x r
+ (1 − (cid:15) )
x f , where x r ∼ P , x f ∼ Q , (cid:15) ∼ U [0 , 1] .
G. RSGAN-GP
L RSGAND = − [Equation] ( P , [Equation]
) [log( sigmoid
[Equation]
r [Equation] [Equation] f )))]
[Equation]
ˆ x
[Equation]
P ˆ x (cid:104) (
[Equation] C (
[Equation]
|
[Equation]
[Equation]
(cid:105) (36) L RSGANG = − E ( x r ,x f ) ∼ ( P , Q ) [log( sigmoid ( C ( x f ) − C ( x r )))] (37)
P ˆ x is the distribution of ˆ x = (cid:15)x r
+ (1 − (cid:15) )
x f , where x r ∼ P , x f ∼ Q , (cid:15) ∼ U [0 , 1] .
G. R A SGAN-GP
L RaSGAND =
− E x r ∼ P (cid:104) log (cid:16) ˜ D ( x r ) (cid:17)(cid:105)
− E x f ∼ Q (cid:104) log (cid:16) 1 − ˜ D
( x f ) (cid:17)(cid:105) + λ E ˆ x ∼ P ˆ x (cid:104) ( ||∇ ˆ x C (ˆ x ) || 2 − 1) 2 (cid:105) (38)
L RaSGANG = − E x f ∼ Q (cid:104) log (cid:16) ˜ D ( x f ) (cid:17)(cid:105) − E x r ∼ P (cid:104) log (cid:16) 1 − ˜ D
( x r ) (cid:17)(cid:105) (39) ˜ D
( x r ) = sigmoid (cid:0) C ( x r ) − E x f ∼ Q C ( x f ) (cid:1)
˜ D
( x f ) = sigmoid ( C ( x f ) − E x r ∼ P C ( x r ))
P ˆ x is the distribution of ˆ x = (cid:15)x r
+ (1 − (cid:15) )
x f , where x r ∼ P , x f ∼ Q , (cid:15) ∼ U [0 , 1] .
H
A LGORITHMS
Require:
The number of D iterations n D ( n D = 1 unless one seeks to train D to optimality), batch size m , and functions f which determine the objective function of the discriminator ( f is f 1 from equation 10 assuming that f 2 ( − y )
=
f 1 ( y ) , which is true for many GANs).
while θ has not converged do for t = 1 , . .
. , n D do
Sample { x ( i ) } mi =1 ∼ P
Sample { z ( i ) } mi =1 ∼ P z
Update w using SGD by ascending with ∇ w 1 m (cid:80) m i =1 (cid:2)
f ( C w ( x ( i ) )
− C w ( G θ ( z ( i ) )))
(cid:3) end for
Sample { x ( i ) } mi =1 ∼ P
Sample { z ( i ) } mi =1 ∼ P z
Update θ using SGD by ascending with ∇ θ 1 m (cid:80) mi =1 (cid:2)
f ( C w ( G θ ( z ( i ) ))
− C w ( x ( i ) ))
(cid:3) end while
Algorithm
Require:
The number of D iterations n D ( n D = 1 unless one seek to train D to optimality), batch size m , and functions f 1 and f 2 which determine the objective function of the discriminator (see equation 10).
while θ has not converged do for t = 1 , . .
. , n D do
Sample { x ( i ) } mi =1 ∼ P
Sample { z ( i ) } mi =1 ∼ P z
Let C w ( x r ) = 1 m (cid:80) mi =1 C w ( x ( i ) )
Let C w ( x f ) =
1 m (cid:80) mi =1 C w ( G θ ( z ( i ) ))
Update w using SGD by ascending with ∇ w 1 m (cid:80) mi =1 (cid:104) f 1 ( C w ( x ( i ) )
− C w ( x f ))
+ f
( C w ( G θ ( z ( i ) ))
− C w ( x r )) (cid:105) end for
Sample { x ( i ) } mi =1 ∼ P
Sample { z ( i ) } mi =1 ∼ P z
Let C w ( x r ) = 1 m (cid:80) mi =1 C w ( x ( i ) )
Let C w ( x f ) =
1 m (cid:80) mi =1 C w ( G θ ( z ( i ) ))
Update θ using SGD by ascending with ∇ θ 1 m (cid:80) mi =1 (cid:104) f
( C w ( G θ ( z ( i ) ))
− C w ( x r ))
+ f
( C w ( x ( i ) )
− C w ( x f )) (cid:105) end while
S CNN
Generator z ∈ R 128 ∼ N (0 , I ) linear, 128 -> 512*4*
Reshape, 512*4*4 -> 512 x 4 x
ConvTranspose2d 4x4, stride 2, pad 1, 512->
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, 256->
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, 128->
BN and ReLU
ConvTranspose2d 3x3, stride 1, pad 1, 64->
Tanh
Discriminator x ∈ R 3x32x
Conv2d 3x3, stride 1, pad 1,
3->
LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, 64->
LeakyReLU 0.
Conv2d 3x3, stride 1, pad 1, 64->
LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, 128->
LeakyReLU 0.
Conv2d 3x3, stride 1, pad 1, 128->
LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, 256->
LeakyReLU 0.
Conv2d 3x3, stride 1, pad 1, 256->
Reshape, 512 x 4 x 4 -> 512*4* linear, 512*4*4 ->
[Table]
Discriminator
[Table]
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 64->
Tanh
Discriminator x ∈ R 3x128x
Conv2d 4x4, stride 2, pad 1, no bias,
3->
LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias,
64->
BN and LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, 128->
BN and LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, 256->
BN and LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, 512->
BN and LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, 1024->
[Table]
Discriminator (PACGAN2 (Lin et al., 2017))
[Table]
Concatenate
[Table]
[Table]
Conv2d 4x4, stride 2, pad 1, no bias,
[Table]
LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, [Table] >
LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias,
[Table] >
BN and LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, [Table] >
BN and LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, 256->
BN and LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, 512->
BN and LeakyReLU 0.
Conv2d 4x4, stride 2, pad 1, no bias, 1024->
J S AMPLES
This shows a selection of cats from certain models.
Images shown are from the lowest FID registered at every 10k generator iterations.
Given space constraint, with cats in high resolution, we show some of the nicer looking cats for each approach, there are evidently some worse looking cats (See https://github .
com/AlexiaJM/RelativisticGAN/tree/ master/images/full_minibatch for all cats of the mini-batch).
[Figure]