A N O NLINE L EARNING A PPROACH TO G ENERATIVE A DVERSARIAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the problem of training generative models with a Generative Adversarial Network (GAN).
Although GANs can accurately model complex distributions, they are known to be difﬁcult to train due to instabilities caused by a difﬁcult minimax optimization problem.
In this paper, we view the problem of training
GANs as ﬁnding a mixed strategy in a zero-sum game.
Building on ideas from online learning we propose a novel training method named C HEKHOV GAN 1 .
On the theory side, we show that our method provably converges to an equilibrium for semi-shallow GAN architectures, i.e. architectures where the discriminator is a one-layer network and the generator is arbitrary.
On the practical side, we develop an efﬁcient heuristic guided by our theoretical results, which we apply to commonly used deep GAN architectures.
On several real-world tasks our approach exhibits improved stability and performance compared to standard GAN training.
I NTRODUCTION
A recent trend in generative models is to use a deep neural network as a generator.
Two notable approaches are variational auto-encoders (VAE) (Kingma & Welling, 2013; Rezende et al., 2014) as well as Generative Adversarial Networks (GAN) (Goodfellow et al., 2014).
Unlike VAEs, the GAN approach offers a way to circumvent log-likelihood-based estimation and it also typically produces visually sharper samples (Goodfellow et al., 2014).
The goal of the generator network is to generate samples that are indistinguishable from real samples, where indistinguishability is measured by an additional discriminative model.
This creates an adversarial game setting where one pits a generator against a discriminator.
Let us denote the data distribution by p data ( x ) and the model distribution by p u ( x ) .
A probabilistic discriminator is denoted by h v : x → [0; 1] and a generator by G u : z → x .
The GAN objective is:
[Equation]
Each of the two players (generator/discriminator) tries to optimize their own objective, which is exactly balanced by the loss of the other player, thus yielding a two-player zero-sum minimax game.
Standard GAN approaches aim at ﬁnding a pure Nash Equilibrium by using traditional gradient-based techniques to minimize each player’s cost in an alternating fashion.
However, an update made by one player can repeatedly undo the progress made by the other one, without ever converging.
In general, alternating gradient descent fails to converge even for very simple games as shown by Salimans et al.
In the setting of GANs, one of the central open issues is this nonconvergence problem, which in practice leads to oscillations between different kinds of generated samples (Metz et al., 2016).
While standard GAN methods seek to ﬁnd pure minimax strategies, we propose to consider mixed strategies, which allows us to leverage online learning algorithms for mixed strategies in large games.
Building on the approach of Freund & Schapire (1999), we propose a novel training algorithm for
GANs that we call C HEKHOV GAN .
[Figure]
The standard GAN training method is not guaranteed to converge for general GAN architectures.
Nevertheless, it does converge for shallow ones 2 , i.e. for GAN architectures which consist of a single layer network as a discriminator, and a generator with one hidden layer (see Fig. 1(a)).
Unfortunately, shallow GANs are very different from the deep GANs (Fig. 1(c)) which are used in practice, and ideally one would hope to understand models which are more similar to deep architectures.
In this paper we make a step forward by considering semi-shallow GANs, an intermediate architecture where the generator is any arbitrary network and the discriminator consists of a single layer (Fig. 1(b)).
Our contributions are:
(1) We show that ﬁnding a Mixed Nash Equilibrium (MNE or simply equilibrium) gives rise to useful generators and discriminators.
(2) We provide an algorithm that provably converges to an equilibrium for semi-shallow architectures.
(3) Guided by our theoretical results we devise a new GAN training algorithm that is applicable to standard deep GAN architectures.
We ﬁrst discuss the beneﬁts of pursuing a mixed equilibrium.
Based on results from game theory we show that by reaching an equilibrium we obtain useful (mixed) generator and discriminator.
In the context of GANs, "usefulness" means that the mixed generator provided by the equilibrium solution will “fool" any adversary at least as good as any single generator (similar results hold for the discriminator).
On the theory side, we show that GANs with semi-shallow architectures induce semi-concave games, i.e., games which are concave with respect to the max player, but need not have a special structure with respect to the min player.
Then we show that in such games players may efﬁciently invoke regret minimization procedures in order to ﬁnd an equilibrium; this in turn gives rise to a way of ﬁnding an equilibrium in semi-shallow GANs.
To the best of our knowledge, this result is novel in the context of GANs and might also ﬁnd uses in other scenarios where such structure may arise.
We would like to emphasize that this is a signiﬁcant step from a theoretical point of view as the standard approach to training GANs is only known to be theoretically sound for convex-concave games, which correspond to shallow-networks (though it is still used heuristically to train deep GAN architectures).
On the practical side, we develop an efﬁcient heuristic guided by our theoretical results, which we apply to commonly used deep GAN architectures.
We provide experimental results demonstrating that our approach exhibits better empirical stability compared to the vanilla GAN and generates more diverse samples, while retaining the same level of visual quality.
In Section 2, we brieﬂy review necessary notions from online learning and zero-sum games.
We then present our approach and its theoretical guarantees in Section 3, and our practical algorithm is presented in Section 4.
Lastly, we present empirical results on standard benchmarks in Section 5.
B ACKGROUND & R ELATED W ORK 2. GAN S
GAN Objectives: The classical way to learn a generative model consists of minimizing a divergence function between a parametrized model distribution p u ( x ) and the true data distribution p data ( x ) .
The original GAN approach by Goodfellow et al.
(2014) is for example known to be related to optimizing the Jensen-Shannon divergence.
This was later generalized by Nowozin et al.
(2016) who described a broader family of GAN objectives stemming from f -divergences.
A different popular type of GAN objectives is the family of Integral Probability Metrics (Müller, 1997), such as the kernel MMD (Gretton et al., 2012;
Li et al., 2015) or the Wasserstein metric (Arjovsky & Bottou, 2017).
All of these divergence measures yield a minimax objective.
Training methods for GANs:
In order to solve the minimax objective in Eq. 1, Goodfellow et al.
(2014) suggested an approach that alternatively minimizes over u and v using mini-batch stochastic gradient descent.
This approach can be shown to converge only when the updates are made in function space.
In practice, this condition is not met - since this procedure works in the parameter space - and many issues arise during training (Arjovsky & Bottou, 2017; Radford et al., 2015), thus requiring careful initialization and proper regularization as well as other tricks (Metz et al., 2016;
Pfau & Vinyals, 2016; Radford et al., 2015; Salimans et al., 2016).
Even so, several problems are still commonly observed including a phenomena where the generator oscillates, without ever converging to a ﬁxed point, or mode collapse when the generator maps many latent codes z to the same point, thus failing to produce diverse samples.
The closest work related to our approach is that of Arora et al.
(2017) who showed the existence of an approximate mixed equilibrium with certain generalization properties; yet without providing a constructive way to ﬁnd such equilibria.
Instead, they advocate the use of mixed strategies, and suggest to do so by using the exponentiated gradient algorithm of Kivinen & Warmuth (1997).
The work of Tolstikhin et al.
(2017) also uses a similar mixture approach based on boosting.
Other works have studied the problem of equilibrium and stabilization of GANs, often relying on the use of an auto-encoder as discriminator (Berthelot et al., 2017) or jointly with the GAN models (Che et al., 2016).
In this work, we focus on providing convergence guarantees to a mixed equilibrium (deﬁnition in Section 3.2) using a technique from online optimization that relies on the players’ past actions.
2. O NLINE L EARNING
Online learning is a sequential decision making framework in which a player aims at minimizing a cumulative loss function revealed to her sequentially.
The source of the loss functions may be arbitrary or even adversarial, and the player seeks to provide worst case guarantees on her performance.
Formally, this framework can be described as a repeated game of T rounds between a player P 1 and an adversary P 2 .
At each round t ∈ [ T ] :
The player ( P 1 ) chooses a point u t ∈ K according to some algorithm
A 2.
The adversary ( P 2 ) chooses a loss function f t ∈ F 3.
The player ( P 1 ) suffers a loss f t ( u t ) , and the loss function f t ( · ) is revealed to her.
The adversary is usually limited to choosing losses from a structured class of objectives F , most commonly linear/convex losses.
Also, the decision set K is often assumed to be convex.
The performance of the player’s strategy is measured by the regret , deﬁned as,
[Equation]
Thus, the regret measures the cumulative loss of the player compared to the loss of the best ﬁxed decision in hindsight .
A player aims at minimizing her regret, and we are interested in no-regret strategies for which players ensure regret which is sublinear in T for any loss sequence 3 .
While there are several no-regret strategies, many of them may be seen as instantiations of the
Follow-the-Regularized-Leader (FTRL) algorithm
where
[Equation]
FTRL takes the accumulated loss observed up to time t and then chooses the point in K that minimizes the accumulated loss plus a regularization term η − 1 t R ( u ) .
The regularization term prevents the player from abruptly changing her decisions between consecutive rounds 4 .
This property is often crucial to obtaining no-regret guarantees.
Note that FTRL is not always guaranteed to yield no-regret, and is mainly known to provide such guarantees in the setting where losses are linear/convex (Hazan et al., 2016; Shalev-Shwartz et al., 2012).
2. Z ERO - SUM G AMES
Consider two players, P 1 , P 2 , which may choose pure decisions among the sets K 1 and K 2 , respectively.
A zero-sum game is deﬁned by a function M : K 1 × K 2 (cid:55)→ R which sets the utilities of the players.
Concretely, upon choosing a pure strategy ( u , v ) ∈ K 1 × K 2 the utility of P 1 is − M ( u , v ) , while the utility of P 2 is M ( u , v ) .
The goal of either P 1 / P 2 is to maximize their worst case utilities; thus,
[Equation]
This deﬁnition of a game makes sense if there exists a point ( u ∗ , v ∗ ) , such that neither P 1 nor P 2 may increase their utility by unilateral deviation.
Such a point ( u ∗ , v ∗ ) is called a Pure Nash Equilibrium , which is formally deﬁned as a point which satisﬁes the following conditions:
[Equation]
While a pure Nash equilibrium does not always exist, the pioneering work of Nash et al.
(1950) established that there always exists a Mixed Nash Equilibrium (MNE or simply equilibrium), i.e., there always exist two distributions D 1 , D 2 such that,
[Equation]
Finding an exact MNE might be computationally hard, and we are usually satisﬁed with ﬁnding an approximate MNE.
This is deﬁned below,
Deﬁnition 1.
Let ε > 0 .
Two distributions D 1 , D 2 are called ε -MNE if the following holds,
[Equation]
Terminology:
In the sequel when we discuss zero-sum games, we shall sometimes use the GAN terminology, relating the min player P 1 as the generator , and the max player P 2 , as the discriminator .
No-Regret & Zero-sum Games: In zero-sum games, no-regret algorithms may be used to ﬁnd an approximate MNE.
Unfortunately, computationally tractable no-regret algorithms do not always exist.
An exception is the setting when M is convex-concave.
In this case, the players may invoke the powerful no-regret methods from online convex optimization to (approximately) solve the game.
This seminal idea was introduced in Freund & Schapire (1999), where it was demonstrated how to invoke no-regret algorithms during T rounds to obtain an approximation guarantee of ε
= O (1 / √ T ) in zero-sum matrix games.
This was later improved by Daskalakis et al.
(2015), and Rakhlin &
Sridharan (2013), demonstrating a guarantee of ε =
O (log T/T ) .
The result that we are about to present builds on the scheme of Freund & Schapire (1999).
F INDING E QUILIBRIUM IN GAN S
Why Mixed Equilibrium?
In this work, our ultimate goal is to efﬁciently ﬁnd an approximate
MNE for the game.
However, in GANs, we are usually interested in designing good generators, and one might ask whether ﬁnding an equilibrium serves this cause better than solving the minimax problem, i.e., ﬁnding u ∈ argmin
u ∈K 1 max v ∈K 2 M ( u , v ) .
Interestingly, the minimax value of the equilibrium generator is always smaller than the minimax value of any pure strategy.
Actually, the equilibrium strategy of the generator might be much better.
This beneﬁt of ﬁnding an equilibrium can be demonstrated on the following simple zero-sum game.
Consider the following paper-rock-scissors game, i.e. a zero-sum game with the minimax objective
[Equation]
; where M =
[Equation] [Equation]
[Equation]
Solving for the minimax objective yields a pure strategy with a minimax value of 1 ; conversely, the equilibrium strategy of the min player is a uniform distribution over actions; and its minimax value is 0 .
Thus, ﬁnding an equilibrium by allowing mixed strategies implies a smaller minimax value, and as we show in the Section 3.3 this is true in general.
In the context of GANs, this result means that the mixed generator provided by the equilibrium solution will “fool" any adversary at least as good as any single generator.
Similarly, the mixed discriminator provided by the equilibrium solution will discern any generator at least as good as any single discriminator.
The rest of this section presents a method that efﬁciently ﬁnds an equilibrium for semi-shallow GANs (see Fig. 1(b)).
Such architectures do not induce a convex-concave game, and therefore the result of Freund & Schapire (1999) does not directly apply.
Nevertheless, we show that semi-shallow GANs imply a game structure which gives rise to an efﬁcient procedure for ﬁnding an equilibrium.
In
Sec.
3.1 we show that semi-shallow GANs deﬁne games with a property that we denote as semiconcave.
Later, Sec. 3.2 provides an algorithm with provable guarantees for such games.
Finally, in
Section 3.3 we show that the minimax objective of the generator’s equilibrium strategy is optimal with respect to the minimax objective.
3. S EMI - SHALLOW GAN S
Semi-shallow GANs do not lead to a convex-concave game.
Nonetheless, here we show that for an appropriate choice of the activation function they induce a game that is concave with respect to the discriminator.
Later, in Sec. 3.2, we show that this property allows to efﬁciently ﬁnd an equilibrium.
Proposition 1.
Consider the GAN objective in Eq.
(1) and assume that the discriminator is a singlelayer network with a sigmoid activation function, meaning h v ( x )
= 1 / (1 + exp( − v (cid:62) x )) , where v ∈ R n .
Then the GAN objective is concave in v .
Note that the above is not restricted to the sigmoid activation function, and it also holds for other choices of the activation function 5 .
3. S EMI - CONCAVE Z ERO - SUM G AMES
Here we discuss the setting of zero-sum games (see Eq.
(4)) which are semi-concave .
Formally a game, M , is semi-concave if for any ﬁxed u 0 ∈ K 1 the function g ( v ) :
= M ( u 0 , v ) is concave in v .
Algorithm 1 presents our method for semi-concave games.
This algorithm is an instantiation of the scheme derived by Freund & Schapire (1999), with speciﬁc choices of the online algorithms
A 1 , A 2 , used by the players.
Note that both A 1 , A 2 are two different instances of the FTRL approach presented in Eq.
Let us discuss Algorithm 1 and then present its guarantees.
First note that each player calculates a sequence of T points based on an online algorithm A 1 / A 2 .
Interestingly, the sequence of (loss/reward) functions given to the online algorithm is based on the game objective M , and
Algorithm
Input : #steps T , Game objective M ( · , · ) for t = 1 . . .
T do
Calculate:
[Equation]
t
[Equation]
√
[Equation]
(cid:107)
[Equation]
Update: f
[Equation]
& g
[Equation]
M ( u
[Equation]
, · ) end for
Output mixed strategies : D 1 ∼ Uni { u 1 , . .
, u T } , D 2 ∼ Uni { v 1 , . . .
, v T } .
also on the decisions made by the other player.
For example, the loss sequence that P 1 receives is { f t ( u ) :=
M ( u , v t ) } t ∈ [ T ] .
After T rounds we end up with two mixed strategies D 1 , D 2 , each being a uniform distribution over the respective online decisions { u t } t ∈ [ T ] , { v t } t ∈ [ T ] .
Note that the ﬁrst decision points u 1 , v 1 are set by A 1 , A 2 before encountering any (loss/reward) function, and the dummy functions f 0 ( u ) = 0 , g 0 ( v ) = 0 are only introduced in order to simplify the exposition.
Since
P 1 ’s goal is to minimize, it is natural to think of the f t ’s as loss functions, and measure the guarantees of A 1 according to the regret as deﬁned in Equation (2).
Analogously, since P 2 ’s goal is to maximize, it is natural to think of the g t ’s as reward functions, and measure the guarantees of A 2 according to the following appropriate deﬁnition of regret, Regret
[Equation]
( v t ) .
The following theorem presents our guarantees for semi-concave games:
Theorem 1.
Let K 2 be a convex set.
Also, let M be a semi-concave zero-sum game, and assume M is L -Lipschitz continuous.
Then upon invoking Alg.
1 for T steps it outputs mixed strategies ( D 1 , D 2 ) that are ε -MNE, where ε =
O (1 / √ T ) .
The most important point to note is that the accuracy of the approximation ε improves as the number of iterations T grows.
This lets us obtain an arbitrarily good approximation for a large enough T .
As mentioned before, both A 1 , A 2 are two different instances of the FTRL approach presented in
Eq.
Concretely, Alg.
A 1 is in fact follow-the-leader (FTL), i.e., FTRL without regularization.
Alg.
A 2 also uses the FTRL scheme.
Yet, instead of the original reward functions, g t ( · ) , it utilizes linear approximations ˜
g t ( v ) = ∇ g t ( v t ) (cid:62) v .
Also note the use of the (minus) square (cid:96) 2 norm as regularization 6 .
The η 0 parameter depends on the Lipschitz constant of M as well as on the diameter of K 2 deﬁned as, d 2 := max v 1 , v 2 ∈K 2 (cid:107) v 1 − v 2 (cid:107) .
Concretely, η 0 = d 2 / √ 2 L .
Next we provide a short proof sketch for Thm. 1.
The full proof appears in Appendix A.
Proof sketch.
The proof makes use of a theorem due to Freund & Schapire (1999) which shows that if both A 1 and A 2 ensure no-regret then it implies convergence to an approximate MNE.
Since the game is concave with respect to P 2 , it is well known that the FTRL version A 2 appearing in Thm.is a no-regret strategy (see e.g. Hazan et al.
The challenge is therefore to show that A 1 is also a no-regret strategy.
This is non-trivial, especially for semi-concave games that do not necessarily have any special structure with respect to the generator 7 .
However, the loss sequence received by the generator is not arbitrary, but rather it follows a special sequence based on the choices of the discriminator, { f t ( · ) =
M ( · , v t ) } t .
In the case of semi-concave games, the sequence of discriminator decisions, { v t } t has a special property which “stabilizes" the loss sequence { f t } t , which in turn enables us to establish no-regret for A 1 .
Remark: Note that Alg.
A 1 in Thm.
1 assumes the availability of an oracle that can efﬁciently ﬁnd a global minimum for the FTL objective, (cid:80) t − 1 τ =
f τ ( u ) .
This involves a minimization over a sum of generative networks.
Therefore, our result may be seen as a reduction from the problem of ﬁnding an equilibrium to an ofﬂine optimization problem.
This reduction is not trivial, especially in light of the negative results of Hazan & Koren (2016), which imply that in the general case ﬁnding an equilibrium is hard, even with such an efﬁcient ofﬂine optimization oracle at hand.
Thus, our result enables to take advantage of progress made in supervised deep learning in order to efﬁciently ﬁnd an equilibrium for GANs.
3. M INIMAX VALUE OF E QUILIBRIUM S TRATEGY
In GANs we are mainly interested in ensuring the performance of the generator (resp.
discriminator)
with respect to the minimax (resp. maximin ) objective.
Let ( D 1 , D 2 ) be the pair of mixed strategies that Algorithm 1 outputs.
Note that the minimax value of D 1 might be considerably smaller than the pure minimax value, as is shown in the example regarding the paper-rock-scissors game (see Sec. 3).
The next lemma shows that the mixed strategy D 1 is always (approximately) better with respect to the pure minimax value (see proof in appendix B.2)
Lemma 1.
The mixed strategy D 1 that Algorithm 1 outputs is ε -optimal with respect to the minimax value, i.e.,
[Equation]
where ε here is equal to the one deﬁned in Thm.
Analogous result hold for D 2 with respect to the pure maximin objective.
P RACTICAL C HEKHOV GAN A LGORITHM FOR D EEP A RCHITECTURES Algorithm
Input : #steps T , Game objective M ( · , · ) , number of past states K , spacing m
Initialize : Set loss/reward f 0 ( · ) = 0 , g 0 ( · ) = 0 , initialize queues Q 1 .insert( f 0 ), Q 2 .insert( g 0 ) for t = 1 . . .
T do Update generator and discriminator based on a mini-batch of noise samples and data samples: u t +1 ← u t − η t · ∇ u t   1 |Q 1 | (cid:88)
f ∈Q 1 f ( u ) + C √ t (cid:107) u (cid:107)  & v t +1 ← v t − η t · ∇ v t (cid:32)|Q 2 | (cid:88)
g ∈Qg ( v ) − C √ t (cid:107) v (cid:107)(cid:33)
Calculate: f t ( · ) =
M ( · , v t ) & g t ( · ) =
M ( u t , · )
Update Q 1 and Q 2 (see main text or Algorithm 3 in the appendix) end for
Output mixed strategies : D 1 ∼ Uni { u 1 , . .
, u K ∈ Q 1 } , D 2 ∼ Uni { v 1 , . .
. , v K ∈ Q 2 } .
In Section 3 we described a method (Alg. 1) which provably reaches an equilibrium for semi-shallow
GANs.
This method considers the whole history of generators and discriminators in making a decision at each round, which contrasts with the standard GAN training method that only considers the last generator and discriminator.
Another difference is that our method outputs a mixed model (i.e., generator and discriminator) rather than a single model.
Building on the theoretical approach introduced in Section 3, we now present a practical method (Alg. 2) which can be efﬁciently applied to train common deep GAN architectures.
Algorithmcombines the ideas of (a) considering the history of generators and discriminators at each update, and (b) outputting a mixed strategy, while only requiring an access to gradient information which can be efﬁciently obtained by running back-propagation.
Next we discuss Alg. 2 in more details and highlight the differences compared to the theoretical approach:
(i)
We use the FTRL objective (Eq.
(3)) for both players.
Note that Alg.
A 1 appearing in Thm 1 uses FTRL with linear approximations, which is only appropriate for semi-concave games.
(ii)
As calculating the global minimizer of the FTRL objective is impractical, we instead update the weights
based on the gradients of the FTRL objective.
This can be done by using traditional optimization techniques such as SGD or Adam.
Thus the update at each round depends on the gradients of the past generators and discriminators.
This differs from the standard GAN training which only employs the gradient of the last generator and discriminator.
(iii)
The full FTRL algorithm requires saving the entire history of past generators/discriminators, which is computationally intractable.
We ﬁnd it sufﬁcient to maintain a summary of the history using a small number of representative models.
In order to capture a diverse subset of the history, we keep a queue Q containing K := |Q| states (models).
The spacing between consecutive models is determined by the following heuristic: every m update steps we remove the oldest model in the queue and add the current one.
The number of steps between switches, m , can be set as a constant, but our experiments revealed it is more effective to keep m small at the beginning and increase its value as the number of rounds increases.
We hypothesize that as the training progresses and the individual models become more discriminative, we should switch the models at a lower rate, keeping them more spaced out.
The pseudo-code and a detailed description of the algorithm appears in the Appendix.
Intuition.
In practice GANs commonly exhibit a non-convergent behavior.
As a consequence, the generator oscillates between generating different modes from the target distribution.
This is hypothesized to be due to the differences of the minimax and maximin solutions of the game (Goodfellow, 2016).
If the order of the min and max operations switch, the minimization with respect to the generator’s parameters is performed in the inner loop.
This causes the generator to map every latent code to one or very few points for which the discriminator believes are likely.
As simultaneous gradient descent updates do not clearly prioritize any speciﬁc ordering of minimax or maximin, in practice we often obtain results that resemble the latter.
In contrast, C HEKHOV GAN takes advantage of the history of the player’s actions which yields better gradient information.
Intuitively, the generator is updated such that it fools the past discriminators.
In order to do so, the generator has to spread its mass more fairly according to the true data distribution.
The discriminator can no longer simply learn to put low probability on the few modes of generated samples, which causes oscillations.
The mode collapse problems of GANs is also closely related to the phenomenon of catastrophic forgetting (Seff et al., 2017).
When GANs are trained sequentially on samples coming from different modes, the discriminator tends to forget the previous modes it has learned about.
This leads to having generated samples that focus only on the last or most prominent modes.
By introducing a history of samples from previous generators, the discriminator is less likely to forget the part of the space that it has already learned.
Fig.
2 illustrates the mode collapse problem.
The data consists of a mixture of 7 Gaussians with different sampling probabilities whose centers are aligned in a circle.
As two modes have higher probabilities and are seen more frequently, they attract the gradients towards them and cause mode collapse and forgetting.
Chekhov GAN manages to recover the true data distribution in this case as well, unlike vanilla GANs.
[Figure]
E XPERIMENTAL RESULTS
We now demonstrate that C HEKHOV GAN yields improved stability and sample diversity.
To do so, we use a comparable number of datasets and baselines as standard GAN approaches, e.g. Metz et al.
; Arjovsky & Bottou (2017).
We test our method on models where the traditional GAN training has difﬁculties converging and engages in a behavior of mode collapse.
We also perform experiments on harder tasks using the DCGAN architecture (Radford et al., 2015) which is commonly used in the literature.
Note that the DCGAN architecture, when trained using standard techniques, still suffers from instabilities and mode collapse (Nagarajan & Kolter, 2017; Roth et al., 2017).
We here demonstrate that C HEKHOV GAN reduces mode dropping while retaining high visual sample quality.
For all of the experiments, we generate from the newest generator only.
Experimental details and comparisons to additional baselines, as well as a set of recommended hyperparameters are available in Appendix D and Appendix C, respectively.
5. N ON - CONVERGENCE AND M ODE D ROPPING 5.1. A UGMENTED MNIST
We ﬁrst evaluate the ability of our approach to avoid mode collapse on real image data coming from an augmented version of the MNIST dataset.
Similarly to (Metz et al., 2016; Che et al., 2016), we combine three randomly selected MNIST digits to form 3-channel images, resulting in a dataset with 1000 different classes, one for each of the possible combinations of the ten MNIST digits.
We train a simpliﬁed DCGAN architecture (see details in Appendix D) with both GAN and C HEKHOV
GAN with a different number of saved past states.
The evaluation of each model is done as follows.
We generate a ﬁxed amount of samples (25,600) from each model and classify them using a pretrained MNIST classiﬁer with an accuracy of 99 . 99% .
The models that exhibit less mode collapse are expected to generate samples from most of the 1000 modes.
We report two different evaluation metrics in Table 1: i) the number of classes for which a model generated at least one sample, and ii) the reverse KL divergence.
The reverse KL divergence between the model and the target data distribution is computed by considering that the data distribution is a uniform distribution over all classes.
[Table]
5. I MAGE M ODELING
We turn to the evaluation of our model for the task of generating rich image data for which the modes of the data distribution are unknown.
In the following, we perform experiments that indirectly measure mode coverage through metrics based on the sample diversity and quality
. 5.2. I NFERENCE VIA O PTIMIZATION ON CIFAR
We train a DCGAN architecture on CIFAR10 (Krizhevsky & Hinton, 2009) and evaluate the performance of each model using the inference via optimization technique introduced in (Metz et al., 2016) and explained in Appendix D.3.3.
The average MSE over 10 rounds using different seeds is reported in Table 2.
Using C HEKHOV
GAN with as few as 5 past states results in a signiﬁcant gain which can be further improved by increasing the number of past states to 10 and 25.
In addition, the training procedure becomes more stable as indicated by the decrease in the standard deviation.
The percentage of mini-batches that achieve the lowest reconstruction loss with the different models is given in Table 2.
This can also be visualized by comparing the closest images x closest from each model to real target images x target as shown in Figure 3.
The images are randomly selected images from the batch which has the largest absolute difference in MSE between GAN and C HEKHOV GAN with 25 states.
The samples obtained by the original GAN are often blurry while samples from C HEKHOV GAN are both sharper and exhibit more variety, suggesting a better coverage of the true data distribution.
[Table]
[Table]
Note that the numbers quoted in our paper can directly be compared to the ones reported in unrolled
GAN (Metz et al., 2016) since we have used the same architecture and choice of hyper-parameters.
We include a comparison in the appendix.
5.2. E STIMATION OF M ISSING M ODES ON C ELEB A
We estimate the number of missing modes on the CelebA dataset (Liu et al., 2015) by using an auxiliary discriminator as performed in (Che et al., 2016).
The experiment consists of two phases.
In the ﬁrst phase we train GAN and C HEKHOV GAN models and generate a ﬁxed number of images.
In the second phase we independently train a noisy discriminator using the DCGAN architecture where the training data is the previously generated data from each of the models, respectively.
The noisy discriminator is then used as a mode estimator.
The test images from CelebA that are classiﬁed as fake by the mode estimator are considered as images belonging to a missing mode.
Table 4 shows the number of missed modes for the two models.
Generated samples from each model are given in the Appendix.
[Table]
Interestingly, even with small number of past states (K=5), C HEKHOV GAN manages to stabilize the training and generate more diverse samples on all the datasets.
In terms of computational complexity, our algorithm scales linearly with K .
However, all the elements in the sum are independent and can be computed efﬁciently in a parallel manner. [Table]
5. I NCEPTION S CORE AND F RÉCHET I NCEPTION D ISTANCE
We compare our algorithm against several state-of-the-art GAN methods using the Inception Score (Salimans et al., 2016) as metric, as well as its improved version, the Fréchet Inception Distance (Heusel et al., 2017).
All the models used in this experiment rely on the same architectures for both the generator and the discriminator.
We generated 50,000 samples for the calculation of the two metrics. Table 5 shows the results on Cifar10 when all models are trained in an unsupervised fashion. We apply our algorithm on top of GAN and WGAN (Arjovsky et al., 2017), as both deﬁne a minimax game. Both Chekhov GAN and Chekhov WGAN are trained using 5 past states.
For fair comparison,
MIX+GAN and MIX+WGAN (Arora et al., 2017) use a mixture of 5 generators and discriminators and the number of unrolling steps for Unrolled GAN (Metz et al., 2016) is set to 5.
The number of parameters for MIX+(GAN/WGAN) is 5 times the number of parameters for all the other baselines.
We ﬁnd that by applying our algorithm on top of WGAN, we get an improvement of 5.13 vs. 4.
(inception score) and 58.82 vs. 66.59 (FID).
Throughout the training, Chekhov WGAN consistently outperforms WGAN by achieving higher scores and lower distances, as shown in Figure 3.
Across the WGAN-based variants, Chekhov WGAN consistently outperforms MIX+WGAN as well, while
WGAN GP (Gulrajani et al., 2017) achieves the best scores within the group.
However, as shown in
Figure 4, Chekhov WGAN improves upon WGAN and reaches almost the same level of performance as WGAN GP.
In terms of the GAN-based variants, Chekhov GAN consistently outperforms Unrolled GAN by a large margin across all epochs.
MIX+GAN, GAN and Chekhov GAN achieve comparable scores in terms of both metrics, but Chekhov GAN does so while reducing the variance signiﬁcantly (see
Figure 4).
C ONCLUSION
We have presented a principled approach to training GANs, which is guaranteed to reach convergence to a mixed equilibrium for semi-shallow architectures.
Empirically, our approach presents several advantages when applied to commonly used GAN architectures, such as improved stability or reduction in mode dropping.
Our results open an avenue for the use of online-learning and gametheoretic techniques in the context of training GANs.
One question that remains open is whether the theoretical guarantees can be extended to more complex architectures
A A NALYSIS
Here we provide the proof of Thm. 1.
Proof.
We make a use of a theorem due to Freund & Schapire (1999) which shows that if both A 1 , Aensure no regret implies approximate MNE.
For completeness we provide its proof in Sec.
A.2.
Theorem 2.
The mixed strategies ( D 1 , D 2 ) that Algorithm 1 outputs are ε -MNE, where
[Equation]
here B A 1 T , B A 2 T are bounds on the regret of A 1 , A 2 .
According to Thm. 2, it is sufﬁcient to show that both A 1 , and A 2 ensure a regret bound of O ( √ T ) .
Guarantees for A 2 : this FTRL version is well known in online learning, and its regret guarantees can be found in the literature, (e.g, Theorem 5.1 in Hazan et al.
The following lemma provides its guarantees,
Lemma 2.
Let d 2 be the diameter of K 2 .
Invoking A 2 with η 0 = d 2 / √ 2 L , ensures the following regret bound over the sequence of concave functions {
g t } t ∈
[ T ] ,
[Equation]
Moreover, the following applies for the sequence { v t } t ∈
[ T ] generated by A 2 ,
[Equation]
Note that the proof heavily relies on the concavity of the g t ( · ) ’s, which is due to the concavity of the game with respect to the discriminator.
For completeness we provide a proof of the second part of the lemma in Sec.
B.3.
Guarantees for A 1 : By Lemma 2, the sequence generated by the discriminator not only ensures low regret but is also stable in the sense that consecutive decision points are close by.
This is the key property which will enable us to establish a regret bound for algorithm A 1 .
Next we state the guarantees of A 1 ,
Lemma 3.
Let C := max u ∈K 1 , v ∈K 2 | M ( u , v ) | .
Consider the loss sequence appearing in Alg 1, { f t ( · ) :
=
M ( · , v t ) } t ∈ [ T ] .
Then algorithm A 1 ensures the following regret bound over this sequence,
[Equation]
Combining the regret bounds of Lemmas 2, 3 and Theorem 2 concludes the proof of Thm. 1.
A. P ROOF OF L EMMA
Proof.
We use the following regret bound regarding the FTL (follow-the-leader) decision rule, derived in Kalai & Vempala (2005) (see also Shalev-Shwartz et al.
Lemma 4.
For any sequence of loss functions { f t } t ∈ [ T ] , the regret of FTL is bounded as follows,
[Equation]
Since A 1 is FTL, the above bound applies.
Thus, using the above bound together with the stability of the { v t } t ∈ [ T ] sequence we obtain,
[Equation]
where the fourth line uses f t ( · ) :
= M ( · , v t ) , the ﬁfth line uses the Lipschitz continuity of M .
And the sixth line used the stability of the v t ’s due to Lemma 2.
Finally, we use | f t ( u ) | = | M ( u, v t ) | ≤
C . A. P ROOF OF T HEOREM
Proof.
Writing explicitly f t ( u ) :
=
M ( u , v t ) and g t ( v ) :
= M ( u t , v ) , and plugging these into the regret guarantees of A 1 , A 2 , we have,
[Equation] [Equation]
By deﬁnition, min u ∈K 1 (cid:80) Tt =1 M ( u , v t )
≤ E u ∼D 1 [ (cid:80) Tt =1 M ( u , v t )] .
Using this together with Equation (5), we get,
T (cid:88)
[Equation]
Summing Equations (6),(7), and dividing by T , we get,
[Equation]
[Equation]
Recalling that D 1 ∼ Uni { u 1 , . .
. , u t } , D 2 ∼ Uni { v 1 , . .
v t } , and denoting ε :=
B A 1 T T + B A 2 T T
, we conclude that,
[Equation]
We can similarly show that,
[Equation]
which concludes the proof
[Equation]
B R EMAINING P ROOFS B. P ROOF OF P ROPOSITION
Proof.
Look at the ﬁrst term in the GAN objective, E p data log h v ( x ) .
For a ﬁxed x we have,
[Equation]
and it can be shown that the above expression is always concave in v 8 .
Since an expectation over concave functions is also concave, this implies the concavity of the ﬁrst term in H .
Similarly, look at the second term in the GAN objective, E z ∈ p
z log(1 − h v ( G u ( z ))) .
For a ﬁxed
G u ( z ) we have,
[Equation]
and it can be shown that the above expression is always concave in v .
Since an expectation over concave functions is also concave, this implies the concavity of the second term in H .
Thus H is a sum of two concave terms and is therefore concave in v .
B. P ROOF OF L EMMA
Proof.
Writing explicitly f t ( u ) :
=
M ( u , v t ) and g t ( v ) :
= M ( u t , v ) , and plugging these into the regret guarantees of A 1 , A 2 , we have,
[Equation]
Summing the above equations and dividing by T , we get,
[Equation]
Next we show that the second term above is always smaller than the minimax value,
[Equation]
Plugging the above into Equation (9), and recalling D 1 ∼ Uni { u 1 , . .
. , u t } , we get,
[Equation]
which concludes the proof.
B. P ROOF OF THE SECOND PART OF L EMMA 2 (S TABILITY OF FTRL SEQUENCE IN CONCAVE CASE )
Proof.
Here we establish the stability of the FTRL decision rule, A 2 , depicted in Theorem 1.
Note that the following applies to this FTRL objective,
[Equation] [Equation]
Where C is a constant independent of v .
Let us denote by Π K 2 the projection operator onto K 2 ⊂ R n , meaning,
[Equation]
By Equation (10) the FTRL rule, A 2 , can be written as follows,
[Equation]
The projection operator is a contraction (see e.g, Hazan et al.
(2016)), using this together with the above implies,
[Equation]
where we used (cid:107)∇ g t ( v t ) (cid:107)
≤
L which is due to the Lipschitz continuity of M .
We also used η 0 = d 2 / √ 2 L .
C P RACTICAL C HEKHOV GAN A LGORITHM
The pseudo-code of algorithms A
and A 2 is given in Algorithm 3.
The algorithm is symmetric for both players and consists as follows.
At every step t if we are currently in the switching mode (i.e. t mod m == 0 ) and the queue is full, we remove a model from the end of the queue, which is the oldest one.
Otherwise, we do not remove any model from the queue, but instead just override the head (ﬁrst) element with the current update.
Input : Current step t , m >if ( t mod m =
= 0 and |Q| ) =
= K ) then
Q .remove_last
Q .insert( f t ) m
= m
+ inc else
Q .replace_ﬁrst( f t ) end if
We set the initial spacing, m , to NK , where N is the number of update steps per epoch, and K is the number of past states we keep.
The number of updates per epoch is just the number of the data points divided by the size of the minibatches we use.
The default value of inc is 10.
Depending on the dataset and number of total update steps, for higher values of K , this is the only parameter that needs to be tuned.
We ﬁnd that our model is not sensitive to the regularization hyperparameters.
For symmetric architectures of the generator and the discriminator (such as DCGAN), for practitioners, we recommend using the same regularization for both players.
For our experiments we set the default regularization to 0.1.
D E XPERIMENTS D. T OY D ATASET : M IXTURE OF G AUSSIANS
We perform several experiments on a toy dataset where we varied the architecture size and the sampling probabilities.
The toy dataset consists of a mixture of 7 Gaussians with a standard deviation
[Figure]
The architecture for the generator consists in two fully connected layers (of size 128) and a linear projection to the dimensionality of the data (i.e. 2).
The activation functions for the fully connected layers are tanh.
The discriminator is symmetric and hence, composed of two fully connected layers (of size 128) followed by a linear layer of size 1.
The activation functions for the fully connected layers are tanh, whereas the ﬁnal layer uses sigmoid as an activation function.
Following Metz et al.
, we intialize the weights for both networks to be orthogonal with scaling of 0.8.
AdamKingma & Ba (2014) was used as an optimizer for both the discriminator and the generator, with a learning rate of 1 e − 4 and β 1 = 0 .
The discriminator and generator respectively minimize and maximize the objective
[Equation] [Equation]
The setup is the same for both models.
For C HEKHOV GAN we use K = 5 past states with L regularization on the network weights using an initial regularization parameter of 0.01.
Effect of the latent dimension.
We ﬁnd that for the case where z ∼ N (0 , I 256 ) , GANs with the traditional updates can fail to cover all modes by either rotating around the modes (as shown in Metz et al.
or converge to only a subset of the modes.
However, if we sample the latent code from a lower dimensional space, e.g. z ∼ N (0 , I 2 ) , such that it matches the data dimensionality, the generator needs to learn a simpler mapping.
We then observe that both GAN and C HEKHOV
GAN are able to recover the true data distribution in this case (see Figure 5).
Additionally, tuning the momentum of the optimiser can also stabilize the training of GANs, resulting in a generator that learns all the modes from the data distribution.
Experiments with concave discriminator.
As out theoretical guarantees hold for a general (i.e. deep) generator and a concave discriminator, we perform experiments using a concave discriminator as well.
We use the same architecture as described previously.
In order to make the discriminator concave, we keep the parameters of the ﬁrst two layers of the discriminator ﬁxed and update only the parameters of the last layer.
We generally ﬁnd this much harder to stabilize due to the great imbalance between the number of trainable parameters of the discriminator and generator.
Increasing the size of the last layer of the discriminator signiﬁcantly improves the performance.
Figure 6 shows that
C HEKHOV GAN can learn the true data distribution.
The ﬁnal layer is increased by 10 times and the number of past states, K , is 10.
Mode collapse.
We run an additional experiment directly targeted at testing for mode collapse.
We sample points x from the data distribution p data with different probabilities for each mode.
We perform an experiment with 5 Gaussian mixtures, again of standard deviation 0.01 arranged in a circle.
The probabilities to sample points from each of the modes are [0.35, 0.35, 0.1, 0.1, 0.1].
In this case two modes have higher probability and could potentially attract the gradients towards them and cause mode collapse.
In order to make things harder, the size of the hidden layers is 16 instead of 128.
C HEKHOV GAN with K = 10 manages to recover the true data distribution in this case as well, unlike vanilla GANs.
Using the history helps C HEKHOV GAN to spread its mass and cover even the modes with low sampling probability.
Unrolled GAN Metz et al.
(2016) instead uses the way the discriminator would react to the current changes when updating the generator.
A comparison to
Unrolled GANs and vanilla GANs is given in Figure 7.
[Figure]
Mixed Nash Equilibrium.
For all the experiments we generate from the most current generator only.
We empirically ﬁnd that generating such that we ﬁrst sample uniformly at random one of the generators from the past, and then generating from it (which corresponds to the MNE) does not lead to any signiﬁcant improvement (Figure 8).
We hypothesize that this is due to a single generator being powerful enough to learn the distribution when playing against the experts from the past.
[Figure]
D. A UGMENTED MNIST
We here detail the experiment on the Stacked MNIST dataset.
The dataset is created by stacking three randomly selected MNIST images in the color channels, resulting in a 3-channel image that belongs to one out of 1000 possible classes.
The architectures of the generator and discriminator are given in
Table 6 and Table 7, respectively.
[Table] [Table]
We use a simpliﬁed version of the DCGAN architecture as suggested by Metz et al.
It contains "deconvolutional layers" which are implemented as transposed convolutions.
All convolutions and deconvolutions use kernel size of 3 × 3 with a stride of 2.
The weights are initialized using the Xavier initialization Glorot & Bengio (2010).
The activation units for the discriminator are leaky ReLUs with a leak of 0.3, whereas the generator uses standard ReLUs.
We train all models for 20 epochs with a batch size of 32, using the RMSProp optimizer with batch normalization.
The optimal learning rate for GAN is 0.001, and for C HEKHOV GAN is 0.01.
For all C HEKHOV GAN models we use regularization of 0.1 for the discriminator and 0.0001 for the generator.
The regularization is L regularization only on the fully connected layers.
For K = 5 , the increase parameter inc is set to 50.
For K=10, inc is 120.
D. CIFAR10 / C ELEB A
We use the full DCGAN architecture Radford et al.
(2015) for the experiments on CIFAR
and
CelebA, detailed in Table 8 and Table 9.
[Table] [Table]
As for MNIST, we apply batch normalization.
The activation functions for the generator are ReLUs, whereas the discriminator uses leaky ReLUs with a leak of 0.3.
The learning rate for all the models is 0.0002 for both the generator and the discriminator and the updates are performed using the Adam optimizer.
The regularization for C HEKHOV GAN is 0.1 and the increase parameter inc is 10.
D.3. R ESULTS ON CIFAR
We train for 30 epochs, which we ﬁnd to be the optimal number of training steps for vanilla GAN in terms of MSE on images from the validation set.
Table 10 includes comparison to other baselines.
The ﬁrst set of baselines (given with purple color) consist of GAN where the updates in the inner loop (for the discriminator), the outer loop (for the generator), or both are performed 25 times.
The baselines shown with green color are regularized versions of GANs, where we apply the same regularization as in our C HEKHOV GAN in order to show that the gain is not due to the regularization only.
Figurepresents two randomly sampled batches from the generator trained with GAN and C HEKHOV GAN .
Table 11 gives comparison between C HEKHOV GAN and Unrolled GAN for different numbers of past states or unrolling steps.
[Table]
D.3. C ELEB A
All models are trained for 10 epochs.
Randomly generated batches of images are shown in Figure 10.
D.3. D ETAILS ABOUT I NFERENCE VIA O PTIMIZATION ON CIFAR
This approach consists in ﬁnding a noise vector z closest that when used as input to the generator would produce an image that is the closest to a target image in terms of mean squared error (MSE): z closest = argmin
z
[Equation]
We report the MSE in image space between x closest and x target .
This measures the ability of the generator to generate samples that look like real images.
A model engaging in mode collapse would fail to generate (approximate) images from the real data.
Conversely, if a model covers the true data distribution it should be able to generate any speciﬁc image from it.
D. I NCEPTION S CORE AND FID ON C IFAR
Additional plots that showcase inception scores for Chekhov WGAN and the WGAN-based baselines (WGAN, WGAN GP, MIX+WGAN) and Chekhov GAN and the GAN-based baselines (GAN,
Unrolled GAN, MIX+GAN) are given in Figure 11.
For Chekhov WGAN we set the regularization to 0.3 and inc = 1 , whereas for Chekhov GAN we set the regularization to 0.005 and inc to 50.
All the baselines were run using their recommended set of hyperparameters.