A DVERSARIAL P ROBLEMS FOR G ENERATIVE N ET -
WORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We are interested in the design of generative networks.
The training of these mathematical structures is mostly performed with the help of adversarial (min-max) optimization problems.
We propose a simple methodology for constructing such problems assuring, at the same time, consistency of the corresponding solution.
We give characteristic examples developed by our method, some of which can be recognized from other applications and some are introduced here for the ﬁrst time.
We compare various possibilities by applying them to well known datasets using neural networks of different conﬁgurations and sizes.
I NTRODUCTION
The problem we are interested in, can be summarized as follows:
We are given two collections of training data { z j } and { x i } .
In the ﬁrst set the samples follow the origin probability density h ( z ) and in the second the target density f ( x ) .
The target density f ( x ) is considered unknown while h ( z ) can either be known with the possibility to produce samples
z j every time it is necessary or unknown in which case we have a second ﬁxed training set { z j } .
Our goal is to design a deterministic transformation G ( z ) so that the data { y j } produced by applying the transformation y =
G ( Z ) onto { z j } follow the target density f ( y ) .
Of course one may wonder whether the proposed problem enjoys any solution, namely, whether there indeed exists a transformation G ( z ) capable of transforming z into y with the former following the origin density h ( z ) and the latter the target density f
( y ) .
The problem of transforming random vectors has been analyzed in Box & Cox (1964) where existence is shown under general conditions.
Computing, however, the actual transformation is a completely different challenge with one of the possible solutions relying on adversarial approaches applied to neural networks.
The most well known usage of this result is the possibility to generate synthetic data that follow the unknown target density f ( x ) .
In this case h ( z ) is selected to be simple (e.g. i.i.d.
standard Gaussian or i.i.d.
uniform) so that generating realizations from h ( z ) is straightforward.
As mentioned, the adversarial approach can be applied even if the origin density h ( z ) is unknown provided that we have a dataset { z j } with data following the origin density.
It was Goodfellow et al.
(2014) that ﬁrst introduced the idea of adversarial (min-max) optimization and demonstrated that it results in the determination of the desired transformation G ( z ) (consistency).
Alternative adversarial approaches were subsequently suggested by Martin Arjovsky &
Bottou (2017); Bi´nkowski et al.
and shown to also deliver the correct transformation G ( z ) .
We must mention the work of Nowozin et al.
(2016) in which a class of min-max optimizations, f-GANs, was deﬁned to design generator/discriminator pairs.
Then, Liu et al.
(2017) deﬁned the adversarial divergences class of objective function which further combined f-GANs, MMD-GAN (Li et al., 2017), WGAN, WGAN-GP (Gulrajani et al., 2017), and entropic regularized optimal transport problems.
Also, they investigated under what conditions the discriminator’s class has the effect of matching generalized moments.
Next, the work of Song & Ermon (2019) connected fGANs and Wasserstein GANs (WGANs)
(Martin Arjovsky & Bottou, 2017), and later Birrell et al.
(2020) generalized the results by introducing the ( f, Γ) − divergencies which allowed to bridge fdivergencies and integral probability metrics.
Our class of generative adversarial problems establishes a one-to-one correspondence with f-gans under the ideal (non data-driven) setup.
However, we believe that our approach enjoys certain signif- icant advantages: First, the deﬁnition of the two functions φ ( z ) , ψ ( z ) in Equ. equation 8 is straightforward while Nowozin et al.
(2016) requires to solve an additional optimization problem for the derivation of each GAN loss.
An additional beneﬁt of our approach is the complete control over the result of the maximization problem that deﬁnes the discriminator.
In other words, we can decide what function, the discriminator must estimate.
In Nowozin et al.
(2016) such ﬂexibility does not exist.
This is important because we can select the approximation function properly so that we avoid the need to impose difﬁcult constraints on the discriminator output (e.g. positivity) since such constraints tend to seriously affect the approximation quality of the corresponding neural network.
Further, there is no need for the discriminator to be a Lipschitz function, as WGAN or WGAN-GP something that needs extra operations to ensure that the discriminator is Lipschitz.
Furthermore, we will show that the function the discriminator tries to approximate is a transformation of the likelihood ratio r ( x ) =
g ( x ) /f
( x ) and there are important applications in Statistics where one is interested in estimating only the transformation of the likelihood ratio with the most common cases being the likelihood ratio itself, its logarithm (log-likelihood ratio), or the ratio r ( x ) 1+ r ( x ) which plays the role of the posterior probability between two densities.
In other words, there are applications where one is interested only in the “max” part of the min-max problem.
Finally, because we know what transformation of the likelihood function the discriminator tries to approximate, it is possible to compare the different GANs on how closely they reach the optimal value of the likelihood ratio r ( x ) =
meaning f ( x )
=
g ( x ) .
As in Nowozin et al.
, we will show that our methods provides an abundance of adversarial problems that are capable of identifying the appropriate transformation G ( z ) .
Furthermore, we will also provide a simple recipe as to how we can successfully construct such problems.
Arguing along the same lines of the existing min-max formulations: We would like to optimally specify a vector transformation G ( z ) , the generator , and a scalar function D ( x ) , the discriminator .
To achieve this, for each combination { G ( z ) , D ( x ) }
we deﬁne the cost function
[Equation]
where φ ( z ) , ψ ( z ) are two scalar functions of the scalar z and E x ∼ f [ · ] , E z ∼ h
[ · ] denote expectation with respect to the density f ( x ) , h ( z ) respectively.
The optimum combination generator/discriminator is then identiﬁed by solving the following min-max problem
[Equation]
We must point out that our goal is not to solve equation 2, but rather ﬁnd a class of functions φ ( z ) , ψ ( z ) so that the transformation G ( z ) that will come out of the solution of equation 2 is such that y =
G ( z ) follows the target density f ( y ) when z follows the origin density
h ( z ) .
If z is random following h ( z ) then
y =
G ( z ) is also random
and we denote with g ( y ) its corresponding probability density.
Clearly, there exists a correspondence between transformations G ( z ) and densities g ( y ) when the density h
( z ) of z is ﬁxed.
Since we can write
[Equation]
this allows us to argue that the min-max problem in equation 2 is equivalent to
[Equation]
It is now possible to combine the two expectations by applying a change of measure and a change of variables and equivalently write equation 3 as follows:
[Equation]
where r ( x ) =
g ( x ) /f
( x ) denotes the corresponding likelihood ratio.
Since f ( x ) is also ﬁxed
[Equation]
there is again a correspondence between r ( x ) and g ( x ) , hence the previous min-max problem becomes equivalent to
[Equation]
[Equation]
Here L f denotes the class of all likelihood ratios r ( x ) with respect to the density f ( x ) , namely, all the functions r ( x ) that satisfy
[Equation]
Using these deﬁnitions, let us deﬁne the cost
[Equation]
and, according to equation 4, we are interested in the following min-max problem
[Equation]
As mentioned, our actual goal is not to solve the adversarial problem.
Instead, we would like to properly identify pairs of functions { φ ( z ) , ψ ( z ) } so that equation 7 accepts as solution the function r ( X ) = 1 .
Indeed, if r ( X ) = 1 is the solution to equation 7, this means that g ( x ) =
f ( x ) is the solution to equation 3 and, ﬁnally, that the optimum G ( x ) obtained from equation 1 is such that y
=
G ( x ) follows g ( y )
=
f ( y ) which, of course, is our original objective.
Even though the minmax problem in equation 1 is what we attempt to solve, it is through equation 7 that we understand what its solution entails.
In the next section we focus on equation 6, equation 7 and propose a simple design method (recipe) for the two functions φ ( z ) , ψ ( z ) that assures that the solution of equationis indeed r ( x ) = 1 .
Before we discuss the details of our work, we would like to summarize this paper’s contribution.
• We design a family of GANs problems using a likelihood ratio approach.
In this class, all optimization problems have the desired property that the generator output follows the target distribution of the random vector of interest, x , in other words, that the likelihood ratio of the two distributions is equal to one.
• We propose a straightforward recipe to explore the GANs family.
With this methodology, we were able to identify subgroups characterized by speciﬁc transformations of the likelihood ratio.
In these subgroups, we discovered novel objective functions and classiﬁed to them previously introduced GANs (such as Wasserstein, Cross-Entropy GANs).
• We propose a new online metric for evaluating the performance of our generative model during training.
• Our experiments provide insights for the behavior of the different GANs objective functions, with some of the novel objective functions performing better than the already known GANs.
A C LASS OF F UNCTIONS φ ( z ) , ψ ( z )
Suppose that ω ( r ) is a strictly increasing and (left and right) differentiable scalar function of the nonnegative scalar r , i.e. r ∈ [0 , ∞ ) .
Denote with J ω = ω (cid:0) [0 , ∞ ) (cid:1) the range of values of ω ( r ) and let ω − 1 ( z ) be the inverse function of ω ( r ) which exists and is deﬁned for z ∈ J ω .
Let ρ ( z ) > 0 be a positive scalar function also deﬁned for z ∈ J ω then, using ω ( r ) and ρ ( z ) , we propose the following pair φ ( z ) , ψ ( z )
[Equation]
(8) where “ (cid:48) ” denotes derivative.
Since ω ( r ) and ρ ( z ) are arbitrary (provided they satisfy the strict increase and positivity constraint respectively), the class of pairs deﬁned by equation 8 is very rich allowing for a multitude of choices.
We show next that any such pair { φ ( z ) , ψ ( z ) } gives rise to a min-max problem, as in equation 7, that accepts r ( x ) = 1 as its unique solution.
We prove this claim in two steps.
The ﬁrst, involves a theorem where we consider a simpliﬁed version of the min-max problem.
Theorem 1 Let ω ( r ) , φ ( z ) , ψ ( z ) and J ω be deﬁned as above with the additional constraint ψ (cid:0) ω (1) (cid:1)
= 0 .
Fix r ≥ 0 and consider φ ( D ) + rψ ( D ) as a function of the scalar D .
Then, for any D ∈ J ω , we have that
[Equation]
with equality if and only if D = ω ( r ) .
Consider next the minimization with respect to r of the maximal value in equation 9.
It is then true that
[Equation]
with equality if and only if r = 1 .
A consequence of Theorem 1 is the next corollary, which constitutes the second and ﬁnal step in proving that the adversarial problem deﬁned in equation 7 has as unique solution the function r ( X ) = 1 .
Corollary
If the functions φ ( z ) , ψ ( z ) satisfy equation 8 and ω ( r ) is strictly increasing and left and right differentiable, then in the adversarial problem deﬁned in equation 7 the maximizer is
D
[Equation]
X ) =
ω
[Equation]
r ( X ) (cid:1) and the minimizer is r ( X )
= 1 , while the resulting min-max value is equal to
[Equation] [Equation]
2. S UBCLASSSES OF THE GANS FAMILY
Let us now present some of the subclasses of the GANs family, where each subclass is characterized by the type of transformation of the likelihood ratio, ω ( r ) .
Once we ﬁx ω ( r ) and give pairs { φ ( z ) , ψ ( z ) } that satisfy equation 8, we are able to “pick” objective functions laying in the ω ( r ) subregion.
Subclass A : ω ( r ) =
r α
The ﬁrst examined subclass is the simplest one, consisting of just powers of the likelihood ratio.
To the best of our knowledge, this is the ﬁrst work proposing objective functions from this class.
To ﬁnd the pairs { φ ( z ) , ψ ( z ) } we proceed as follows.
We have that ω − 1 ( z ) =
z 1 α and J ω =
[0 , ∞ ) .
According to equation 8, for z ∈ [0 , ∞ ) we must deﬁne φ (cid:48)
( z ) = − z 1 α ρ ( z ) , ψ (cid:48)
( z ) =
ρ ( z ) .
Also r = D − a .
Some examples of this subclass are presented in Table 1.
For the particular selection ω ( r ) =
r (corresponding to α = 1 ) we can show that the resulting cost is equivalent to the Bregman cost (Bregman, 1967).
In fact there is a one-to-one correspondence between our ρ ( z ) function and the function that deﬁnes the Bregman cost.
This correspondence however is lost once we switch to a different α or a different ω ( r ) function, suggesting that the proposed class of pairs { φ ( z ) , ψ ( z ) } , is far richer than the class induced by the Bregman cost.
[Table]
Subclass B : ω ( r ) =
α − 1 log r
This subclass considers one of the most popular transformations of the likelihood ratio, the log-likelihood ratio.
As with subclass A for the ﬁrst time, the next examples are presented.
They can be used either under a min-max setting, for the determination of the generator/discriminator pair, or under a pure maximization setting for the direct estimation of the log-likelihood ratio function log r ( x ) .
We have ω − 1 ( z ) =
e αz and J ω
= R .
As before ρ ( z ) must be strictly positive and, according to equation 8, for all real z we must deﬁne φ (cid:48)
( z ) =
− e αz ρ ( z ) , ψ (cid:48) ( z )
=
ρ ( z ) .
And the likelihood ratio is r = e aD .
In Table 2 we can see some examples of this subclass.
Subclass C : ω ( r ) =
rr +
As we already mentioned, this is another important transform of the likelihood ratio.
Interestingly, in this subclass belongs the ﬁrst introduced GAN (Goodfellow et al., 2014)
the Cross Entropy GAN.
[Table]
−
[Table]
+
[Table]
− log(1 + e −
[Table]
R
When ω ( r ) =
rr +
we have ω − 1 ( z ) =
z 1 − z and J ω =
For ρ ( z ) > 0 , z ∈ [0 , 1] we must deﬁne the functions φ ( z ) , ψ ( z ) according to equation 8 φ (cid:48)
( z ) = − z
−
z ρ ( z ) , ψ
[Table]
( z ) =
ρ ( z ) .
In this case the likelihood ratio is r =
D 1 − D .
In Table 3 we see the cross entropy GAN and C2 which is presented here for the ﬁrst time.
[Table]
Subclass D : ω ( r ) =
sign(log r )
This is a special case of ω ( r ) with the corresponding function not being strictly increasing.
It turns out that we can still come up with optimization problems, two of which are known and used in practice, by considering ω ( r ) as a limit of a sequence of strictly increasing functions.
Monotone Loss: As a ﬁrst approximation we propose sign( z )
≈ tanh( c 2 z )
where c > 0 a parameter.
We note that lim c →∞ tanh( c 2 z )
= sign( z ) .
Using this approximation we can write
[Equation]
As we mentioned, we have exact equality for c → ∞ .
Let us perform our analysis by assuming that c is ﬁnite.
We note that ω − 1 ( z ) =
( 1+ z 1 − z ) 1 c and J ω =
[ − 1 , 1] .
Consequently, if ρ ( z ) > 0 for z ∈ [ − 1 , 1] , we must deﬁne φ (cid:48)
( z ) = − (cid:16) 1+ z 1 − z (cid:17) 1 c ρ ( z ) , ψ (cid:48)
( z ) =
ρ ( z ) .
D1)
If we let c → ∞ in order to converge to the desired sign function, this yields φ (cid:48)
( z ) =
− ρ ( z ) and ψ (cid:48)
( z ) =
ρ ( z ) .
This suggests that φ ( z ) =
− (cid:82)
z ρ ( x ) dx is decreasing and ψ ( z )
=
(cid:82)
z ρ ( x ) dx = − φ
( z ) is increasing.
In fact any strictly increasing function ψ ( z ) can be adopted provided we select φ ( z )
=
− ψ ( z ) .
There is a popular combination that falls under Case D1).
In particular, the selection ψ ( z ) =
z = − φ ( z ) reminds us of Wasserstein GAN Martin Arjovsky & Bottou (2017), with two differences, in our case z should lie in [ − 1 , 1] and the discriminator is not constrained to be a Lipschitz function.
Hinge Loss: As a second approximation we use the expression sign( z )
≈ sign( z ) | z | 1 c , c > 0 , which is strictly increasing, continuous and converges to sgn( z ) as c → ∞ .
This suggests that
[Equation]
(13) and ω − 1 ( z ) =
e z c .
Since ω ( r ) can assume any real value we conclude that J ω =
R which, clearly, differs from the previous approximation where we had J ω
=
[ − 1 , 1] .
If ρ ( z ) > 0 , z ∈ R then, according to equation 8 we must deﬁne φ (cid:48)
( z ) =
− e z c ρ ( z ) , ψ (cid:48) ( z ) =
ρ ( z ) .
We present the following case that leads to a very well known pair from a completely different application.
D2)
If we select ψ (cid:48) ( z ) =
ρ ( z ) =
{ e −| z | 1 c + 1 z < − 1 } > 0 then φ (cid:48)
( z ) =
− e z 1 c { e −| z | 1 c + 1 z < − 1 } .
If we now let c → ∞ , we obtain the limiting form for the derivatives which become ψ (cid:48)
( z ) =
− 1 z < 1 and φ (cid:48)
( z ) = 1 z > − 1 .
By integrating we arrive at φ ( z ) = − max { 1+ z, 0 } and ψ ( z ) = − max { 1 − z, 0 } .
we notice that since c → ∞ we cannot ﬁnd the likelihood ratio in terms of the discriminator.
The cost based on this particular pair is called the hinge loss Tang (2013) and it is very popular in binary classiﬁcation where one is interested only in the maximization problem.
The corresponding method is known to exhibit an overall performance which in practice is considered among the best Rosasco et al.
Janocha & Czarnecki (2017).
Here, as in Zhao et al.
, we propose the hinge loss as a means to perform adversarial optimization for the design of the generator G ( x ) .
[Table]
Wasserstein z −
[Table]
R
In Table 4 we can see Hinge and Wasserstein GANs optimization problems.
The detailed derivation of the above optimization problems and some more examples can be found in Appendix A.2.
This completes our presentation of examples.
However, we must emphasize, that these are only a few illustrations of possible pairs { φ ( z ) , ψ ( z ) } one can construct.
Indeed combining, as dictated by equation 8, any strictly increasing function ω
( r ) with any positive function ρ ( z ) generates a legitimate pair { φ ( z ) , ψ ( z ) } and a corresponding min-max problem equation 7 that enjoys the desired solution r ( x ) = 1 .
Remark
Given the numerous choices we have in deﬁning adversarial optimizations for solving the same problem, one may wonder whether there exists a means to rank these methods and identify the most efﬁcient, at least for classes of data.
This is an extremely challenging question which, unfortunately, ﬁnds no answer even in simpler problems.
For example, in binary classiﬁcation there are also classes of optimization problems that are used to train neural networks Bartlett et al.
Masnadi-Shirazi & Vasconcelos (2009).
However, until now, no theoretical analysis exists that can order them and designate the most efﬁcient one.
This is possible only through experience with countless simulations.
D ATA -D RIVEN S ETUP AND N EURAL N ETWORKS
Let us now consider the data-driven version of the problem.
As mentioned, the target density f ( x ) is unknown.
Instead we are given a collection of realizations { x i } that follow f ( x ) and a second collection { z j } that follows the origin density h
( z ) .
These data constitute our training set .
Regarding the second set { z j } it can either become available “on the ﬂy” when h ( z ) is known by generating realizations every time they are needed, or it can be considered ﬁxed from the start exactly as { x i } , if h ( z ) is also unknown.
As we pointed out in Section 1, we are interested in designing a generator
G ( z ) so that when we apply it onto the data z j , that is, y j = G ( z j ) the resulting y j will follow a density that matches the target density f ( x ) .
Since we are now considering the data-driven version of the problem, we are going to limit G ( z ) , D ( x ) to be the outputs of corresponding neural networks.
Therefore the generator is replaced by G ( z , θ ) while the discriminator by D ( x , ϑ ) where θ, ϑ summarize the parameters of the two neural networks.
Of course instead of neural networks one could use any other parametric family, as SVMs, capable of efﬁciently approximating any nonlinear function.
Once we have selected our favorite ω ( r ) and ρ ( z ) functions we can compute from equation 8 the functions φ ( z ) , ψ ( z ) that enter into the min-max problem deﬁned in equation 2.
This problem, after limiting the generator and discriminator to neural networks, can be rewritten as follows
[Equation]
If θ o , ϑ o are the corresponding optimum parameter values, and the structure of the two networks is sufﬁciently rich, we expect that G ( z , θ o ) , D ( x , ϑ o ) will approximate the optimum functions
D ( x ) , G ( z ) of the ideal problem in equation 2 respectively.
In particular for θ o , the generator
G ( z , θ o ) , whenever applied onto any z j that follows h ( z ) , it will result in a y j =
G ( z j , θ o ) that follows a density which is expected to be close to the target density f
( y ) .
Remark 2 When replacing D ( x ) , G ( z ) with neural networks we must take special care of the corresponding outputs.
Basically, we must guarantee that they are of the correct form.
This is particularly important in the case of the scalar output D ( x , ϑ ) of the discriminator.
We recall that the optimum discriminator is D ( x ) = ω (cid:0) r
( x ) (cid:1) .
This implies that we need to assure that D ( x , ϑ ) takes values in J ω (the range of ω ( r ) ).
Consequently, we must apply the proper nonlinearity in the output of the discriminator that will guarantee this fact.
E XPERIMENTS
In this section, we want to examine the performance of the GANs objectives presented in Tablefor different datasets.
For that reason we tested their performance on four different datasets, namely
MNIST (LeCun et al., 1998), CelebA (Liu et al., 2015), CIFAR-10 datasets (Krizhevsky et al., 2009), and Stanford Cars (Krause et al., 2013).
We recall that GANs are notorious for their nonrobust behavior Bengio (2012);
Creswell et al.
; Mescheder et al.
For the stabilization of the training process, we used the maximum gradient-penalty methodology which was generalized to a class of Lipschitz GANs in Zhou et al.
(implementation details in Appendix A.3).
In this section we present our results for the regularization parameter λ
= 10 , in the Appendix we included results for different values of λ .
In Tables 5, 6 we present the ﬁnal attained Frechet
Inception Distances (FID)
Heusel et al.
(2017) and Kernel Inception Distances (KID) Bi´nkowski et al.
(2018) scores after training.
In Table 7 we computed the absolute difference of the discriminator estimated likelihood ratio from the optimal likelihood ratio, which is equal to one and the variance around the optimal value.
For Hinge and
Wasserstein GANs we cannot compute the likelihood ratio related metrics, as we mentioned in section 2.1.
Also, in Figure 4 we show the evolution of FID (second row) and KID (third row) during training.
In the ﬁrst row we see the value of the distance d ( x , y ) =
| E x ∼ f
[ D ( x )] − E y ∼ g
[ D ( y )] | .
For the expectations estimation, we used 64 examples for MNIST and 128 for Stanford Cars, CelebA, and CIFAR-10.
We believe that this is an insightful quantity since its value is an indicator of how accurate the discriminator can distinguish samples from the target distribution and the generator output.
Finally, in Appendix A.3, some generated samples of our trained generative models are included.
[Table] [Table]
[Figure]
GAN
CARS
CELEBA
CIFAR
MNIST × 1 , × 10 −× 1 , × 10 −× 10 − 2 , × 10 −× 10 − 1 , × 10 −
[Table]
For the MNIST dataset, the different GANs have very similar performance, something that is reasonable since this is a simple dataset.
The d ( x , y ) curves are quickly converging indicating that the discriminator has stabilized.
Interestingly, the FID, KID curves for GANs with similar d ( x , y ) have also close FID, KID scores.
In particular, Subclass C GANs (Cross-Entropy with C2); Subclass D (Hinge) with Wasserstein; B2, B1a and A3; B1b with Exponential; A1b with A2; and A1a with
MSE.
For the CIFAR-10 dataset, the FID and KID scores for different objective functions tend to have similar behavior, but some differences exist.
In particular, the objectives A1a, A1b, A2, MSE,
B1a, B1b in the initial iterations create a steep valley for the metric d ( x , y ) ; and at the same time in the FID score, these objectives attain smaller FID scores faster than the other GANs.
Also, the further improvement of the scores during training seems to be related to the behavior of d ( x , y ) .
For instance, the d ( x , y ) of A1b, A2 is still increasing during the last iterations, and the generated image quality (FID score) is improving.
Lastly, similar to MNIST, GANs that have very close d ( x , y ) tend to have close FID, KID curves.
For Stanford Cars and CelebA, we observe some performance gaps between different subclasses.
Speciﬁcally for CelebA, the Subclass D and Subclass B objective functions start to diverge after some training steps, something that is evident from the FID and KID curves where the scores increase dramatically after approximately the ﬁrst half of iterations.
Also, this is noticeable for the d ( x, y ) , where a sudden drop is evident around the same period.
Interestingly the variance around the optimal likelihood ratio value is increased for the GANS who started to diverge.
For Stanford cars, the Subclass C has the worst performance, with the recorded MMD score being an order larger than the other objectives (expect Subclass B –B2, Exponential objectives–).
Furthermore, for this dataset, the distances d ( x , y ) are “noisy” (strong ﬂuctuations) and with approximately ten times larger magnitude value when compared to the corresponding curves in the other datasets.
It is reasonable since this dataset is a complex one, where we have natural scenes with cars, and, importantly, signiﬁcantly smaller ( ∼ 8000 training samples, when MNIST, CIFAR10 has 60000 and CelebA 200000).
In summary, our simulations indicate that the Subclass A objectives A1a, A1b, A2, MSE have the best performance both in terms of the computed metrics (hence image generation quality) and stability during training.
They might not give the exact best score in all datasets, but are very close to the best recorded, and, most importantly, they have an overall more stable behavior when compared to objectives from the other subclasses.
Furthermore, the discriminator convergence to the optimal likelihood ratio between the dataset and the generator output can be an ad-hoc measure of the behavior/stability of our generative models than can be used during training to provide online, useful insights of the current training condition.
C ONCLUSION
In this paper, we provided and demonstrated a straightforward methodology to determine loss functions that solve the generative adversarial problem.
Our results suggest that there is not a single loss function that achieves the best performance in terms of the examined metrics for all different datasets.
This performance variation among loss functions becomes evident as the increasing complexity of the datasets that convolutes the generation task is better addressed by some loss functions that clearly outperform others.
Speciﬁcally, in simpler datasets, such as MNIST, the evaluated loss functions yield very similar performance, whereas, in more intricate datasets like CelebA, CIFAR10, and Stanford Cars, performance ”gaps” between the different loss functions, and different subclasses, emerges.
Our ﬁndings also propose that in every generation task unexplored loss functions outperformed the previously proposed ones.
Consequently, this function class is worth-exploring to identify new loss functions that can be used and evaluated in different applications.
Our method provides a versatile tool that can be exploited in that direction.
A. T HE SUBCLASSES OF THE GAN S F AMILY
Subclass A : ω ( r ) =
r α
The ﬁrst examined subclass is the simplest one, consisting of just powers of the likelihood ratio.
To the best of our knowledge, this is the ﬁrst work proposing objective functions from this class.
To ﬁnd the pairs { φ ( z ) , ψ ( z ) } we proceed as follows.
We have that ω − 1 ( z ) =
z 1 α and J ω =
[0 , ∞ ) .
According to equation 8, for z ∈ [0 , ∞ ) we must deﬁne φ (cid:48)
( z ) = − z 1 α ρ ( z ) , ψ (cid:48)
( z ) =
ρ ( z ) .
The following examples can be shown to satisfy these equations.
A1)
If we select ρ ( z ) =
z β , with β (cid:54) = − 1 , − 1 − 1 α , this yields φ ( z ) =
− z 1+ 1 α + β 1+ 1 α + β and ψ ( z ) =
z 1+ β 1+ β .
For β = − 1 , ρ ( z ) =
z − 1 , φ ( z ) =
− αz 1 α , ψ ( z ) =
log z .
For β = − 1 − 1 α , ρ ( z ) =
z − 1 − 1 α
φ ( z ) =
− log z , ψ ( z ) =
− αz − 1 α . A2)
If we select α = 1 , ρ ( z ) = 1 (1+ z ) then, φ ( z ) =
− (1 + z ) and ψ ( z ) = − (1 + z − 1 ) .
A3)
If we select α = 1 , ρ ( z ) = 1 (1+ z )
z then, φ ( z ) =
− log(
+ z ) and ψ ( z ) =
− log(1 + z − 1 ) .
For the particular selection ω ( r ) =
r (corresponding to α = 1 ) we can show that the resulting cost is equivalent to the Bregman cost (Bregman, 1967).
In fact there is a one-to-one correspondence between our ρ ( z ) function and the function that deﬁnes the Bregman cost.
This correspondence however is lost once we switch to a different α or a different ω ( r ) function, suggesting that the proposed class of pairs { φ ( z ) , ψ ( z ) } , is far richer than the class induced by the Bregman cost.
We should mention that in A1)
the selection α = 1 , β = 0 is known as the mean square error criterion and if we apply only the maximization problem then this corresponds to a likelihood ratio estimation technique proposed in the literature by Sugiyama et al.
We will refer to this case as the MSE GAN.
Subclass B : ω ( r ) =
α − 1 log r
This subclass considers one of the most popular transformations of the likelihood ratio, the log-likelihood ratio.
As in the ﬁrst subclass, for the ﬁrst time, the next examples are presented.
They can be used either under a min-max setting, for the determination of the generator/discriminator pair, or under a pure maximization setting for the direct estimation of the log-likelihood ratio function log r ( x ) .
We have ω − 1 ( z ) =
e αz and J ω
= R .
As before ρ ( z ) must be strictly positive and, according to equation 8, for all real z we must deﬁne φ (cid:48)
( z ) =
− e αz ρ ( z ) , ψ (cid:48) ( z )
=
ρ ( z ) .
The following examples satisfy these equations.
B1)
If ρ ( z ) =
e − βz with β (cid:54) = 0 , α , this produces φ ( z ) =
− e ( α − β )
z α − β , ψ ( z ) = − e − βz β .
If β = 0 then ρ ( z ) = 1 , φ ( z ) =
− e αz α , ψ ( z ) =
z .
If β = α then ρ ( z ) =
e − αz , φ ( z ) = − z and ψ ( z ) =
− e − αz α .
We call the α = β = 0 .
5 case the Exponential GAN.
B2)
If α = 1 , ρ ( z ) = 1 1+ e z then, φ ( z ) = − log(1 + e z ) and ψ ( z ) =
− log(1 +
e − z ) .
Subclass C : ω ( r ) =
rr +
As we already mentioned, this is another important transform of the likelihood ratio.
Interestingly, in this subclass belongs the ﬁrst introduced GAN (Goodfellow et al., 2014)
the Cross Entropy GAN.
When ω ( r ) =
rr +
we have ω − 1 ( z ) =
z 1 − z and J ω =
For ρ ( z ) > 0 , z ∈ [0 , 1] we must deﬁne the functions φ ( z ) , ψ ( z ) according to equation 8 φ (cid:48)
( z ) = − z
−
z ρ ( z ) , ψ (cid:48) ( z ) =
ρ ( z ) .
The next set of examples can be seen to satisfy these equations.
C1)
If we select ρ ( z ) = 1 z , this yields φ ( z )
=
log(1 − z ) and ψ ( z ) = log z .
C2)
Selecting ρ ( z ) =
(1 − z )
α , with α (cid:54) = 0 , − 1 , yields φ ( z )
= − 1 1+ α (1 − z )
α +1 + 1 α
(1 − z ) α and ψ ( z )
= − 1 1+ α (1 − z ) 1+ α .
For α = 0 , we have ρ ( z ) = 1 and φ (
z ) =
z + log(1 − z ) , ψ ( z ) =
z , while for α = − 1 we have ρ ( z )
=
11 − z and φ ( z ) =
− log(1 − z ) − 1 1 − z , ψ ( z ) =
− log(1 − z ) .
In C1)
we recognize the functions used in the original article by Goodfellow et al.
C2) appears for the ﬁrst time.
Subclass D : ω ( r ) =
sign(log r )
This is a special case of ω ( r ) with the corresponding function not being strictly increasing.
It turns out that we can still come up with optimization problems, two of which are known and used in practice, by considering ω ( r ) as a limit of a sequence of strictly increasing functions.
Monotone Loss: As a ﬁrst approximation we propose sign( z )
≈ tanh( c 2 z )
where c > 0 a parameter.
We note that lim c →∞ tanh( c 2 z )
= sign( z ) .
Using this approximation we can write
[Equation]
As we mentioned, we have exact equality for c → ∞ .
Let us perform our analysis by assuming that c is ﬁnite.
We note that ω − 1 ( z ) =
( 1+ z 1 − z ) 1 c and J ω =
[ − 1 , 1] .
Consequently, if ρ ( z ) > 0 for z ∈ [ − 1 , 1] , we must deﬁne φ (cid:48)
( z ) = − (cid:16) 1+ z 1 − z (cid:17) 1 c ρ ( z ) , ψ (cid:48)
( z ) =
ρ ( z ) .
D1)
If we let c → ∞ in order to converge to the desired sign function, this yields φ (cid:48)
( z ) =
− ρ ( z ) and ψ (cid:48)
( z ) =
ρ ( z ) .
This suggests that φ ( z ) =
− (cid:82)
z ρ ( x ) dx is decreasing and ψ ( z )
=
(cid:82)
z ρ ( x ) dx = − φ
( z ) is increasing.
In fact any strictly increasing function ψ ( z ) can be adopted provided we select φ ( z )
=
− ψ ( z ) .
There is a popular combination that falls under Case D1).
In particular, the selection ψ ( z ) =
z = − φ ( z ) reminds us of Wasserstein GAN Martin Arjovsky & Bottou (2017), with two differences, in our case z should lie in [ − 1 , 1] and the discriminator is not constrained to be a Lipschitz function.
Hinge Loss: As a second approximation we use the expression sign( z )
≈ sign( z ) | z | 1 c , c > 0 , which is strictly increasing, continuous and converges to sgn( z ) as c → ∞ .
This suggests that
[Equation]
and ω − 1 ( z ) =
e z c .
Since ω ( r ) can assume any real value we conclude that J ω =
R which, clearly, differs from the previous approximation where we had J ω
=
[ − 1 , 1] .
If ρ ( z ) > 0 , z ∈ R then, according to equation 8 we must deﬁne φ (cid:48)
( z ) =
− e z c ρ ( z ) , ψ (cid:48) ( z ) =
ρ ( z ) .
We present the following case that leads to a very well known pair from a completely different application.
D2)
If we select ψ (cid:48) ( z ) =
ρ ( z ) =
{ e −| z | 1 c + 1 z < − 1 } > 0 then φ (cid:48)
( z ) =
− e z 1 c { e −| z | 1 c + 1 z < − 1 } .
If we now let c → ∞ , we obtain the limiting form for the derivatives which become ψ (cid:48)
( z ) =
− 1 z < 1 and φ (cid:48)
( z ) = 1 z > − 1 .
By integrating we arrive at φ ( z ) = − max { 1+ z, 0 } and ψ ( z ) = − max { 1 − z, 0 } .
The cost based on this particular pair is called the hinge loss Tang (2013) and it is very popular in binary classiﬁcation where one is interested only in the maximization problem.
The corresponding method is known to exhibit an overall performance which in practice is considered among the best
Rosasco et al.
Janocha & Czarnecki (2017).
Here, as in Zhao et al.
, we propose the hinge loss as a means to perform adversarial optimization for the design of the generator G ( x ) .
This completes our presentation of examples.
However, we must emphasize, that these are only a few illustrations of possible pairs { φ ( z ) , ψ ( z ) } one can construct.
Indeed combining, as dictated by equation 8, any strictly increasing function ω
( r ) with any positive function ρ ( z ) generates a legitimate pair { φ ( z ) , ψ ( z ) } and a corresponding min-max problem equation 7 that enjoys the desired solution r ( x ) = 1 .
A. E XPERIMENTS
In our experiments we employed the two neural networks the generator and the discriminator.
For the generator, we used a four-layer neural network where the ﬁrst layer is linear and the remaining deconvolutional; with ReLU activation functions between the layers except the ﬁnal layer where we used a sigmoid function since the output is an image with pixel values in the range [0 , 1] .
The generator input is a standard i.i.d.
normal vector with dimension 64 for MNIST and 128 for Stanford
Cars, CelebA and CIFAR-10.
The output of the generator is a 784 × 1 vector for the MNIST dataset, whereas for the other datasets it is a 3072 × 1 vector.
For the discriminator, we used a four-layer neural network with three convolutional layers followed by a linear layer.
We applied Leaky ReLUs between the layers except for the ﬁnal layer where we adopted proper functions based on the range J ω .
For the training of the two neural networks we applied the Adam algorithm Kingma & Ba (2014) with β 1 = 0 .
5 , β 2 = 0 .
9 , learning rate 10 − 4 and batch size 50 for MNIST and 128 for Stanford Cars, CelebA and CIFAR-10.
For all datasets, the training lasted 180000 iterations.
A. CELEBA
In Figures 2, 3, 4, 5 we can see the estimation of likelihood ratio from the discriminator neural network and the FID, KID scores of the generated images during training.
For every optimization problem, we tested its performance for different values of the regularization parameter λ of the maximum gradient penalty.
The different values of λ where { 0 . 01 , 0 . 1 , 1 , 10 } as in Zhou et al.
We must mention that in the cases where some λ value is missing means that the algorithm diverged during training.
From our simulations, we notice that, especially in the case of the KID score, the likelihood ratio approximation accuracy of the discriminator coincides with the quality of the synthetic images produced by the generator.
For instance, in Figure 2 for A3, λ
=
10 at ∼ 150000 iterations the likelihood ratio (which should converge to 1) starts to be more variant around the optimal value, then we notice the KID, FID scores increase indicating that the generated images quality drops.
It is apparent in Figure 7 where we see some blurry, almost faceless, images.
Also, for A1b, B1a, B1b,
Exponential, B2 GANs, we notice the same behavior.
Moreover, in all cases, we notice a faster convergence for larger values of λ .
Interestingly, there is a similar behavior, in terms of convergence, between the different λ ’s.
For example, in B2, for λ = 10 , we notice that the variance around one is increasing.
Similarly for λ = 1 , λ = 0 .
1 but more slowly in terms of iterations.
Therefore, it seems that the regularization parameter cannot ﬁx the divergence problems of a GAN, but they can inﬂuence the period they will become evident.
As we already discussed, in the case of Hinge and Wasserstein GANs, the likelihood ratio value is not computable.
For this reason in Figure 5 we present the discriminator output for dataset ( D ( X ) ) and for synthetic ( D ( G ( Z )) )
samples.
In the Hinge GAN, the discriminator output gradually increases its range of values for λ = 10 around 130000 iteration and for λ = 1 arounditeration.
Accordingly, the synthetic images produced by the generator start to look less similar to the ones from the dataset.
Similarly, Wasserstein GANs have the same patterns.
But in this case, the discriminator output tends to have large values.
Our simulations argue, that a way to reduce those large values is to increase the regularization parameter λ , in other words to ”force” the discriminator being Lipschitz.
[Figure]
A. S YNTHETIC E XAMPLES
In the Figures 6, 7 and 8 we present some random samples of the trained generators for the datasets
Stanford cars, CelebA
[Figure]
and CIFAR-10 for λ = 10 .
[Figure]
[Figure]
[Figure]
A1a
A1b
A A
MSE
B1a
B1b
Exponential
B
Cross Entropy
C
Hinge
Wasserstein
A1a
A1b
A A
MSE
B1a
B1b
Exponential
B
Cross Entropy
C
Hinge
Wasserstein
A1a
A1b
A A
MSE
B1a
B1b
Exponential
B
Cross Entropy
C
Hinge
Wasserstein