B AYESIAN M ODELLING AND M ONTE C ARLO I NFERENCE FOR GAN
Anonymous authors
Paper under double-blind review
A BSTRACT
Bayesian modelling is a principal framework to perform model aggregation, which has been a primary mechanism to combat mode collapsing in the context of
Generative Adversarial Networks (GANs).
In this paper, we propose a novel
Bayesian modelling framework for GANs, which iteratively learns a distribution over generators with a carefully crafted prior.
Learning is efﬁciently triggered by a tailored stochastic gradient Hamiltonian Monte Carlo with novel gradient approximation to perform Bayesian inference.
Our theoretical analysis further reveals that our treatment is the ﬁrst Bayesian modelling framework that yields an equilibrium where generator distributions are faithful to the data distribution.
Empirical evidence on synthetic high-dimensional multi-modal data and the natural image database CIFAR-10 demonstrates the superiority of our method over both start-of-the-art multi-generator GANs and other Bayesian treatment for GANs.
I NTRODUCTION
Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) is a popular method to learn a distribution on complex data such as natural images, videos and texts.
However, it is notoriously hard to train and suffers from mode collapse .
There has been a series of works addressing these issues.
One noticeable thread focuses on objective design, which improves the original JensenShannon divergence with more stable pseudo-metrics such as f -divergence (Nowozin et al., 2016), χ 2 -divergence
(Mao et al., 2017), and Wasserstein distance (Arjovsky et al., 2017).
However, such treatment is inherently limited when a single generator does not include enough model capacity to capture the granularity in data distribution in practice.
Clearly, such generator can never produce accurate samples regardless of the choice of objectives.
An alternative remedy is to learn multiple generators instead of a single generator.
This type of methods (Hoang et al., 2018; Tolstikhin et al., 2017; Wang et al., 2016) is motivated by a straightforward intuition that multiple generators can better model multi-modal distributions since each generator only needs to capture a subset of the modes.
To entail model aggregation, Bayesian modelling is a natural and principle framework to articulate the aggregation process in a probabilistic way.
Recently, Saatci & Wilson (2017) proposes Bayesian GAN, a probabilistic framework for GAN under Bayesian inference.
It shows that modelling the distribution of generator helps alleviate mode collapse and motivates interpretability of the learned generators.
The probabilistic model is built upon conditional distributions of generator and discriminator, whose maximum likelihood estimation can be realized as a metaphor of typical GAN objective.
Empirical study shows that Bayesian GAN provides state-of-the-art quantitative results for semi-supervised image classiﬁcation task.
However, a critical theoretical question of this framework is still not well-understood: does it really converge to the generator distribution that produces the real data distribution?
Actually, our empirical study shows that Bayesian GAN suffers from severe performance loss due to this lack of convergence guarantee.
With this observation, we follow the prior work to exploit Bayesian modelling as a principled way to realize model aggregation, but approach this problem from a theoretical perspective; we try to analyze the developed treatment (including choice of priors, approximate inference, and its convergence property) and simultaneously propose a new Bayesian modelling framework with the desirable convergence guarantee and consequently superior empirical performance. [Table] D [Table]
D
[Table]
L EAST -S QUARES GAN ( D −
[Table] N [Table] Our main contributions are:
• We theoretically establish, to our best knowledge, the ﬁrst Bayesian treatment of GANs such that any generator distribution faithful to the data distribution is an equilibrium.
• We prove the previous Bayesian method (Saatci & Wilson, 2017) for any minimax GAN objective induces incompatibility of its deﬁned conditional distributions.
• We propose two special Monte Carlo inference algorithms for our Bayesian model which efﬁciently approximate the gradient of a non-differentiable criterion.
• Empirical studies on synthetic high-dimensional multi-modal data and CIFAR-10 demonstrate the superiority of the proposed framework over the state-of-the-art GAN methods.
R ELATED W ORK
Generative Adversarial Networks is a powerful class of methods to learn a generative model for any complex target data distribution.
There is a game between a generator and a discriminator.
Both of them adapt their strategies to maximize their own objective function involving the other:
[Equation]
Eqn. 1 gives a general mathematical form where p data is real data distribution and p gen ( · ; θ g ) are generated data distribution with generator parameter θ g .
The objective functions φ 1 , φ 2 , φ(termed as GAN objective in this paper) are elaborately chosen such that at the equilibrium, the generator generates the target data distribution.
Table 1 summarizes several widely used GAN objectives, including the original min-max version, non-saturating version of original GAN (Goodfellow (2016)), LSGAN (Mao et al.
(2017)), and WGAN (Arjovsky et al.
As reported in
Table 1, some GAN objectives which satisfy φ 3 ( · )
= − φ 2 ( · ) actually represent a min-max game, i.e. min θ g max θ g J d ( θ d ; θ g ) .
Training GAN with multiple generators is considered in several recent works to mitigate the mode collapse problem.
In the spirit of boosting algorithm, Wang et al.
(2016) propose to progressively train new generator using a subset of training data that are not well captured by previous generators, while Tolstikhin et al.
(2017) further propose a more robust mechanism to reweight samples in the training set for a new generator.
From the perspective of game theory, MIX-GAN (Arora et al., 2017) extends the game between a single generator and discriminator to the multiple-player setting.
Other works resort to third party classiﬁers to help multiple generators and discriminators achieve better equilibrium, such as MAD-GAN (Ghosh et al., 2017) and the recent state-of-art method,
MGAN (Hoang et al., 2018).
Bayesian GAN proposed by Saatci & Wilson (2017) adapts a different approach which models generator and discriminator distributions by deﬁning the conditional posteriors (Eqn. 8).
The posterior likelihood is specially designed such that maximizing it exactly corresponds to optimizing GAN objectives.
The authors argue that compare to point mass ML estimation, learning the generator distribution which is multi-modal itself offers better ability to ﬁt a multi-modal data distribution.
To facilitate discussion, we categorize GAN frameworks into the following taxonomy: optimizationbased methods and probabilistic methods .
Optimization-based methods set up an explicit mini-max game between the generator and discriminator, where an ideal equilibrium typically characterize a generator faithful to data distribution.
In probabilistic methods, generators and discriminators evolve as particles of underlying distributions, where an equilibrium is searched from a stochastic exploration in the distribution space (of the generators and discriminators).
M ETHODOLOGY
We ﬁrst summarize the notations.
Second, we elaborate our Bayesian modelling for GAN and develop constituent prior and likelihood formulations.
Finally, we develop posterior inference algorithms for our Bayesian modelling. A detailed discussion of the motivation of our modelling and the comparison with Bayesian GAN is delayed to Section 4.
3. N OTATIONS
p data ( x ) over a sample space X is the target data distribution we want to learn.
Our generator and discriminator are parameterized by θ g ∈ Θ g and θ d ∈ Θ d .
A generator with parameter θ g deﬁnes a mapping from a random noise vector z ∼ p
z to a random vector G ( z ; θ g ) .
The induced probability density of G ( z ; θ g ) is denoted as p gen ( x ; θ g ) .
A discriminator is a function mapping data to a real-valued score, i.e. D ( x ; θ d ) :
X → [0 , 1] (or X → R in some settings).
Further, we use q g ( θ g ) ∈
P Θ g , q d ( θ d ) ∈
P Θ d to denote the distribution over generators and discriminators respectively.
The total data distribution generated by generator following the density q g ( θ g ) is naturally a mixture of data distribution given by every single generator, p model ( x ; q g ) =
E θ g ∼ q g ( θ g ) [ p gen ( x ; θ g )] .
Our goal is to ﬁnd a generator distribution q ∗ g ( θ g ) such that the total generated data distribution matches our target, i.e. p model ( x ; q ∗ g ) (cid:39) p data ( x ) .
J g ( θ g ; θ d ) and J d ( θ d ; θ g ) denote objective functions of generator and discriminator as introduced in Eqn.
The common choices 1 are listed in Table 1.
With a slight abuse of the notation, we extend the notation J g ( θ g ; D ∗ ) by replacing D ( x ; θ d ) in equation 1 with any score function D ∗ .
Then J g ( θ g ; θ d ) can be viewed as an abbreviation of J g ( θ g ; D ( · ; θ d )) .
Likewise, J d ( θ d ; p gen ( · )) represents discriminator objective given a virtual generator that generates data with density p gen ( · ) .
3. B AYESIAN M ODELLING FOR GAN
The same as Bayesian GAN, we iteratively update the generator and discriminator.
We denote q ( t ) g and q ( t ) d are their distributions at time step t .
Likelihood.
The likelihood function indicates a preference for generators and discriminators given current distributions of generator and discriminator, thus should encode the information that distributionally reﬂect the objective of generators J g and discriminators J d .
An ideal solution is to form a distribution that is proportional to quantities that evaluates such ﬁtness:
[Equation]
where p ( t ) model ( x ) =
E θ g ∼ q ( t ) g [ p gen ( x ; θ g )] is the mixed data distribution under current generator distribution q ( t ) g and D ( t ) ( · ) =
E θ d ∼ q ( t ) d [ D ( · ; θ d )] is the averaged discriminating score function under current discriminator distribution q ( t ) d .
We emphasize, although sharing the same spirit of reﬂecting the GAN objectives in likelihood, there is a crucial difference between our likelihood model and that of Bayesian GAN.
We will revisit it in the later theory section.
Prior.
While the likelihood design is rather straightforward, an ideal prior is more involved.
When the generated data distribution is increasingly close to the real data distribution, there will be less information for discriminator to distinguish between them; consequently, the discriminator tends to assign equal scores to all data samples, resulting in equal likelihoods for all generators.
At that stage, a good strategy is to keep the generator distribution the same as the previous time step, since it already generates the desired data distribution perfectly.
Hence, we use the generator distribution in the previous time step as a prior for the next time step.
Such dynamically evolving prior for generator turns out to be crucial.
In Section 4.2, we show the Bayesian model in previous work suffers from bad convergence due to its ﬁxed and weakly informative prior.
In contrast, we set a uniform improper prior on the discriminator to pursuit unrestricted adaptability to evolving generators.
Posterior.
Integrating the prior mentioned above and the likelihood in Eqn. 2, we formulate our iterative Bayesian model as
[Equation]
This iterative process for updating q ( t ) g and q ( t ) d can be universally applied to any common GAN objectives with a guarantee of effectiveness (i.e., any desired generator distribution is a convergence point of the iteration) which will be formally stated in Section 4.1.
3. I NFERENCE A LGORITHM
So far we have introduced our Bayesian modelling for GAN.
In this section we develop novel inference algorithms to compute the posterior.
Similar to most advanced Bayesian methods, exact calculation of the posterior is intractable .
Following the strategy in Saatci & Wilson (2017), we adopt Stochastic Gradient Hamiltonian Monte Carlo to generate samples from the posterior.
In each iteration, M samples { θ ( t ) g,m } Mm =1 are used to approximate the generator distribution q ( t ) g .
[Equation]
Empowered by the adapted SGHMC (algorithm 1 in appendix), we are able to sample from q ( t +1) g and q ( t +1)
d based on gradients in Eqn.
4 and Eqn. 5.
The gradients come from two sides: the GAN objective J g , J d and the prior q ( t ) g .
Getting GAN objective’s gradient is easy while computing the prior’s gradient, ∇ θ g log q ( t ) g
( θ g ) , is actually non-trivial since we have no exact analytic form of
q ( t ) g
( θ g ) .
To address this challenge, we propose the following two methods to approximate ∇ θ g log q ( t ) g
( θ g ) , leading to two practical inference algorithms.
Gaussian Mixture Approximation (GMA) .
Although the analytic form of the distribution q ( t ) g ( θ g ) is unknown, we have M g Monte Carlo samples { θ ( t ) g,m } M g m =1 which enables us to directly approximate the distribution as a Mixture of Gaussian in left side of Eqn. 6, where σ is a hyper-parameter and C is the normalization constant.
Then we derive the prior gradient approximation as shown in the right side of Eqn. 6.
[Equation]
Partial Summation Approximation (PSA) .
From Eqn. 5, actually we can make an interesting observation that the prior gradient can be recursively unfolded as a summation over all historical
GAN objective gradients, shown as:
[Equation]
Therefore if we store all historical discriminator samples { θ ( i ) d,m } t,M d i =1 ,m =1 , the prior gradient can be computed accurately via simple summation.
Practically, computing gradients with all discriminator samples costs huge amount of storage and computational time, which is unaffordable.
Hence we propose to maintain a subset of discriminators by subsampling the whole sequence of discriminators.
T HEORY
In this section, we ﬁrst present the good convergence property of our Bayesian model.
Second, we theoretically analyze the distribution evolution of Bayesian GAN (which will be referred as BGAN in the rest of paper) and make a comparison with our model.
All proofs are delayed to the appendix (Section A) due to limited page.
4. C ONVERGENCE P ROPERTY OF O UR M ODEL
We call a generator distribution is ideal if the generator following this distribution produces target data distribution.
Theorem 1 states that any the ideal generator distribution is an equilibrium of the dynamics deﬁned in Eqn.
Although its mathematical proof evolves more elaboration, the idea behind is quite simple.
When the generator distribution is ideal, discriminator should not be able to distinguish the synthetic data from real.
Thus averaged discriminator function will degenerate to a constant function.
Then generator distribution will remain unchanged since discriminator essentially puts no preference between generators.
Theorem 1.
Assume the GAN objective and the discriminator space are symmetry.
For any ideal generator distribution q ∗ g ( θ g ) satisfying p ∗ model (cid:44) E θ g ∼ q ∗
g
[ p gen ( · ; θ g )]
= p data , there exists a discriminator distribution
q ∗ d such that D ∗ ( · ) (cid:44) E θ d ∼ q ∗ d D ( · ; θ d ) ≡ c 2 .
Moreover, q ∗ g and q ∗ d is an equilibrium of the dynamic deﬁned in Eqn. 3.
4. C OMPARISON WITH B AYESIAN GAN
In this paragraph, we ﬁrst analyze the distribution evolution behind the BGAN algorithm which is uncovered in the original paper.
Corollary 1 states our derivation where p ( θ g | α g ) , p ( θ d | α d ) are the predeﬁned priors.
In practice, BGAN use weakly informative Gaussian prior.
Corollary 1.
The Bayesian GAN algorithm actually performs distribution dynamics in Eqn. 8.
[Equation]
Between our model (Eqn. 3) to BGAN (Eqn.
8), there are essentially two differences, the choice of prior and likelihood.
Using weakly informative prior as BGAN do, actually causes problems.
As we mentioned in the last paragraph, intuitively, better the generator distribution is, less informative the discriminator is.
A non-informative discriminator distribution plus a weakly informative prior will provide bad guidance to the generator.
In terms of likelihood, BGAN uses expectation of objective values (e.g. E θ g ∼ q ( t ) g J d ( θ d ; θ g ) ), while our model adopts objective value of expectation (e.g. J d ( θ d ; E θ g ∼ q ( t ) g
[ p gen ( · ; θ g ))] ).
We argue that computing the objective value of expectation makes more sense since our goal is to match the expectation (over generators) of generated data distribution to the target data distribution.
Compatibility Issue .
We further show BGAN’s choice of likelihood and prior leads to theoretical issues.
Speciﬁcally, BGAN is not suitable for any minimax-style GAN objective due to the incompatibility of its conditional posteriors.
This problem may limit the usage of BGAN since many widely used GAN objective is in min-max fashion, such as the original GAN and WGAN.
Consider a simple case where we use only one Monte Carlo sample for the distributions q ( t ) g and q ( t ) d .
Then the distribution evolution in Eqn.
8 will degenerate to a Gibbs sampling process.
[Equation]
Thus implicitly θ ( t ) g and θ ( t ) d are sampled from a joint distribution of θ d and θ g deﬁned by the conditionals p ( θ g | θ d ) =
exp {J g ( θ g ; θ d ) } p ( θ g | α g ) and p ( θ d | θ g ) =
exp {J d ( θ d ; θ g ) } p ( θ d | α d ) .
However, our theoretical analysis shows that such a presumed joint distribution does not exist when J d ( θ d ; θ g )
=
−J g ( θ g ; θ d ) .
Speciﬁcally, Lemma 1 shows the existence of a joint distribution satisfying the conditionals in Eqn.
9 requires the GAN objective to be decomposable, i.e.
∃ φ g , φ d , s.t.
J d ( θ d ; θ g )
=
φ g ( θ g ) + φ d ( θ d ) .
Apparently, no GAN objective is decomposable.
Therefore, conditionals in Eqn.
9 are actually incompatible.
Sampling with incompatible conditional distribution is problematic and leads to unpredictable behavior (Arnold & Press, 1989).
Lemma 1. Consider a joint distribution p ( x, y ) of variable X and Y .
Its conditional distributions can be represented in the forms of p ( x | y )
∝ exp { L ( x, y ) } q x ( x ) and p ( y | x ) ∝ exp {− L ( x, y ) } q y ( y ) only if X and Y are independent, i.e., p ( x, y )
=
p ( x ) p ( y ) and L ( x, y ) is decomposable, i.e., ∃ L x and L y , L
( x, y )
=
L x ( x ) + L y ( y ) .
E XPERIMENTS
In this section, we evaluate our model with two inference algorithms proposed in Section 3.3 (denoted as ours-GMA and ours-PSA ).
We compare with three baselines: 1) GAN (or DCGAN 2 ) : naively trained multiple generators in the vanilla GAN framework; 2) MGAN :
Mixture GAN (Hoang et al., 2018) which is the start-of-art method to train GAN with multiple generators; 3) BGAN : Bayesian
GAN (Saatci & Wilson, 2017).
For each model, we conduct thorough experiments under the four different GAN objectives introduced in Table 1 referred as GAN-MM, GAN-NS, WGAN and LSGAN here.
For a fair comparison, each model has the same number of generators with the same architecture.
Discriminator architectures are also the same except for that of MGAN which has an additional branch of the classiﬁer.
To facilitate reproducibility, we report implementation and experiment details in Section C of the appendix.
5. H IGH - DIMENSIONAL M ULTI - MODAL S YNTHETIC D ATASET
Dataset.
Consider learning a data distribution in a high dimensional space X
=
R D , which is a uniform mixture of n modes.
Each mode lies on a d -dimensional sub-space of X .
We call this d -dimensional sub-space as mode-space of the i -th mode.
Speciﬁcally, the data of the i -th mode is generated by the following process,
[Equation]
In our experiment, n , D , and d are set to
Metric.
We deﬁne projection distance (cid:15) p for generated data sample x as the minimum of Euclidean distance from x to any of mode-spaces i.e. (cid:15) p ( x ) =
min i (cid:15) i ( x ) (cid:44) (cid:107) x − A i ( A Ti A i ) − 1 A Ti x (cid:107) 2 .
Then we set a threshold 3 η to test the belonging of x , i.e. the data samples whose Euclidean distance to the mode-space is below η are considered as belonging to that mode.
The trained models are evaluated by the samples { x k } Kk =1 ∼ p model it generates.
We deﬁne hit set H i (cid:44) {
x k | (cid:15) i ( x k ) <
η } to indicate the samples belong to each mode.
We further deﬁne projected hit set , PH i (cid:44) { ( A Ti A i ) −
A Ti x − b i | x ∈ H i } by projecting data in each hit set back to the canonical low dimensional space.
Now we introduce three evaluation metrics: hit ratio , hit distance , and cover error .
Hit ratio
H r (cid:44) (cid:80) ni =1 |H i | /K is the percentage of generated data belonging to any of the modes of real data.
Hit distance H d (cid:44) (cid:80) ni =1 (cid:80) x ∈H i (cid:15) i ( x ) / (cid:80) ni
=1 |H
i | is the averaged projection distance over all data in the hit set.
Lastly, cover error C e evaluates how well the generated data covers each mode.
Essentially it computes the KL-divergence between the estimated distribution of samples in PH i and the uniform distribution over [ − 1 , 1] d .
Formally, it is deﬁned as the averaged KL-divergence on n modes i.e. C e (cid:44) 1 n (cid:80) ni
=1 KL (ˆ p ( · ; PH i ) (cid:107)U [ − 1 , 1] d ) .
The intuition is that if data generated is close to the ground truth distribution, they should be uniformly distributed in the square area of each mode.
[Table]
Optimization-Based v.s. Probabilistic.
The left part of Table 2 summarizes the results in terms of hit ratio and hit distance .
Probabilistic methods including our algorithms and BGAN always achieve a hit ratio of 1 , which means every data point generated from these models is very close to one mode of the target distribution.
On the other hand, optimization based methods, both GAN and MGAN, consistently have a signiﬁcantly larger hit error [Table] and sometimes may even generate data samples that do not belong to any mode.
Moreover, the data distribution generated by the optimization-based methods ﬁts the target uniform distribution much worse than its probabilistic counterparts, which is quantitatively reﬂected by the cover error showed in the right side of Tableand visually demonstrated by the projected hit sets in Figure 1.
According to the visualization, data generated by GAN or MGAN tend to be under dispersed and hardly cover the whole square region of the true mode, while data generated by probabilistic methods align much better with the ground truth distribution.
We attribute this superiority to stronger exploration power in the generator space coming from the randomness in probabilistic methods.
Bayesian GAN v.s. Our Methods.
The incompatibility issue of BGAN with minimax-style GAN objectives theoretically derived in Section 4.2 is empirically veriﬁed in our experiments.
As visualized in Figure 1, with the GAN-MM objective, BGAN is trapped in a local equilibrium and fails in capturing one mode of the true data.
Besides, as shown in Table 2, BGAN with the WGAN objective achieves much poorer coverage than with other GAN objectives, while our model is much more robust to the choice of GAN objectives (consistently lower cover errors).
A qualitative comparison is made in Figure 6 (in the appendix) which shows the data distribution generated by BGAN trained with WGAN objective tends to shrink.
More visual illustrations under different GAN objectives are placed in Section D of the appendix.
5. N ATURAL I MAGE D ATASET
Datasets.
CIFAR-10 is a well-understood benchmark dataset (Krizhevsky et al.
with 50k training and 10k test 32x32 RGB images.
It contains 10 classes of images: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
In our experiments, we use all 50k training samples to train and evaluate models.
Evaluation Protocols.
We make a thorough comparison for all methods with different GAN objectives under two widely adopted metrics: Inception Score (Salimans et al.
and Fr´echet
Inception Distance (Heusel et al.
Inception Score computes exp( E x [ KL ( p ( y | x ) (cid:107) p ( y ))]) where p ( y | x ) standards is predicted label distribution by a pretrained Inception model (Szegedy et al.
While Inception Score (IS) reﬂects the ﬁdelity and diversity of images and is known to be well-correlated with human judgment (Salimans et al.
, it does not measure the closeness between real data distribution and synthetic data.
Therefore, we employ Fr´echet Inception Distance (FID).
To compute it, we embed real and synthetic images to the last hidden layer of Inception model and evaluate the Fr´echet distance between two distributions in the embedding space.
We delay details of computing these metrics into the appendix.
Model Architectures.
Inspired by Hoang et al.
(2018), we adapted the parameter sharing technique for both generators and discriminators.
Speciﬁcally, generators (of one model) are not disjoint neural networks.
They only differ at the ﬁrst layer and sharing all parameters at the following layers.
This tied parameter design reduced the model complexity and boost the performance.
Especially for the probabilistic model such as Bayesian GAN, we observe a signiﬁcant enhancement of generated image qualities compare to the original result (Figure 7 in Saatci & Wilson (2017)).
More implementation details such as hyper-parameters of network layers are delayed to Section C in the appendix.
Quantitative Results.
We train every model 500 epochs.
We denote the epoch delivering highest
IS − 0 .
1 FID as the best epoch and report the scores of the model at this epoch.
The results are [Table] [Figure] summarized in Table 3.
Overall our proposed model outperforms all baselines under all GAN objectives.
Generally, probabilistic methods achieve better scores then optimization based methods which indicate injecting stochasticity into GAN training helps to generate more multi-modal images.
Moreover, we would highlight that Bayesian GAN is observed to have a visible performance drop when accompanying with min-max style GAN objectives, which provides another empirical evidence for our theory analysis in Section 4.2.
While our Bayesian method ﬁts any GAN objectives and constantly performs well.
Qualitative Results.
Figure 2 display the samples randomly generated by baselines and our model.
Every row in the ﬁgure represents samples of one learned generator.
As Figure 2 and Figure 7 shows, all three baselines suffer from mode collapses problem.
Hoang et al.
(2018) already notice that mode collapse in one of the generators could happen after a roughly long training procedure (aroundepochs).
Our experiment shows Bayesian GAN also have this problem.
However, our novel Bayesian model is robust to mode collapse.
Speciﬁcally, mode collapse occurs to our model only once in a total of over ten trials.
In contrast, mode collapse happens ﬁve times more frequently in baseline methods.
Section E in appendix contains more images generated by all models at their best epochs.
Actually, for the baseline models, their best epoch always appear before epoch 250.
In another word, training too many epochs causes performance drop which indicates an instability near the equilibrium.
D ISCUSSION
In this paper, we propose a novel Bayesian modelling framework for GAN, with a likelihood function establishing a connection to existing GAN models and a novel prior stabilizing the inference process.
We propose scalable Bayesian inference algorithms which are asymptotically correct.
As future work, we plan to extend the proposed framework to non-parametric Bayesian modelling and investigate more theoretical properties of GANs in the Bayesian context.
Developing Bayesian generalization for deep learning models is not a recent idea and happens in many ﬁelds other than generative models.
By this work, we emphasize the importance of going beyond the intuition and understanding the theoretical behavior of the Bayesian model.
We hope that our work helps inspire continued exploration into Bayesian deep learning from a more rigorous perspective.
A O MITTED PROOFS
Theorem 1.
This theorem is general and holds when the GAN objective and the discriminator space have symmetry.
The symmetry of GAN objective means its functions φ 1 and φ 2 satisfy that ∃ c ∈ R , ∀ x ∈ R , φ 1 ( x ) ≡ φ 2 ( c − x ) .
While the symmetry of discriminator space Θ d indicates that for any θ d ∈ Θ d , there is a θ (cid:48)
d ∈ Θ d such that D ( x ; θ d ) ≡ c − D ( x ; θ (cid:48) d ) .
Note that the symmetry condition are very weak, ﬁrst it holds for all the common choices of GAN objectives such as those listed in Table 1.
Second, it holds for neural network which is the most common parameterization for discriminator in practice.
Proof.
[Equation]
Eqn. 11 and Eqn.
12 prove that q ∗ d ( θ d ) ∝ exp( J d ( θ d ; p ∗ model )) satisﬁes E θ d ∼ q ∗ d D ( · ; θ d ) ≡ c 2 .
Note that, in the above equations, S ( θ d ) denotes the symmetric discriminator of θ d satisfying
D ( x ; S ( θ d ))
≡ c − D ( x ; θ d ) .
Hence according Eqn. 13, we know J g
( θ g ; D ∗ ) is a constant.
[Equation]
Thus the generator distribution will not change based on the dynamics in Eqn.
3 since q ∗ g ( θ g )
∝ exp {J g ( θ g ; D ∗ ) } q ∗ g ( θ g ) .
Corollary
Proof.
[Equation] [Equation]
[Equation]
Algorithm 1 from the original paper (Saatci & Wilson, 2017) implies that Eqn.
14 where θ k d ∼ q ( t ) d ( θ d ) are Monte Carlo samples of discriminator and every z ( i ) is a mini-batch containing n g noise samples.
Be deﬁnition p
( θ g | z
( i ) , θ kd )
∝ (cid:81) n g j =1 D ( G ( z ( i ) j ; θ g ); θ kd ) p ( θ g | α g )
(Eqn. 1 in the original paper), each term 4 log p ( θ g | z
( i ) , θ kd )
= (cid:80) n g j =
φ
( D ( G ( z ( i ) j ; θ g ); θ kd ))
+log p ( θ g | α g ) .
Hence, the total summation is a Monte Carlo approximation of the expectation in the right side of Eqn. 15.
The same derivation can be done to q ( t +1) d .
Together, we get Corollary 1.
Lemma
Proof.
[Equation]
where α ( y ) =
( (cid:82) exp { L ( x, y ) } q x ( x ) dx ) − 1 and β ( x )
=
( (cid:82) exp {− L ( x, y ) } q y ( y ) dy ) − 1 .
B I NFERENCE ALGORITHM
Stochastic Gradient Hamiltonian Monte Carlo (Chen et al., 2014) is a gradient based MCMC sampling method.
It use noise gradient estimation of potential function to generate sample from a given distribution.
Our inference algorithm (algorithm 1) is based on this brilliant algorithm.
Input for for do θ d,m ← θ ( t ) d,m do
[Equation]
+
[Equation]
( θ d,m ) + n ; n ∼ N (0 , 2 αηI ) θ d,m ← θ d,m + v end for θ ( t +1)
d,m ← θ d,m end for for m = 1 to M g do θ g,m ← θ ( t ) g,m for l = 1 to L do v ← (1 − α ) v + η ∇ θ g log q ( t +1) g
( θ g,m ) + n ; n ∼ N (0 , 2 αηI ) θ g,m ← θ g,m + v end for θ ( t +1)
g,m ← θ g,m end for end for
C E XPERIMENT DETAILS
We implement our model and baselines and conduct experiments in PyTorch (Paszke et al., 2017) except for the MGAN model in CIFAR-10 experiment where we adapt its ofﬁcial tensorﬂow implementation 5 .
C. H IGH - DIMENSIONAL MULTI - MODAL SYNTHETIC DATASET
Model Architecture : Each generator or discriminator is a three layer perceptron.
For the generator, the dimensions of input, hidden layer and output are 10, 1000, and 100 respectively.
For the discriminator, the dimensions of input, hidden, output layers are 100, 1000, and 1.
All activation functions are leaky ReLU(Maas et al., 2013).
Training hyperparameters : All models are optimized by Adam (Kingma & Ba, 2014) with a learning rate of 10 − 4 .
For probabilistic methods, the SGHMC noise factor ( α in algorithm 1) is set as 10 − 1 .
C. CIFAR-
Implementation of inception score .
As Barratt & Sharma (2018) pointed out, Inception Score is sensitive to the inception model used and number of data splits.
We follow the convention of using open-sourced tensorﬂow inception model 6 and setting split number to ten.
Model architecture :
Generator architecture has four deconvolution layers (kernel size 4, stride 2) with the following input, hidden feature-maps, output size: 100x1x1 → 512x4x4 → 256x8x8 → 128x16x16 → 3x32x32.
Every deconvolution layer is followed by batch-normalization layer and
Relu activation except for the last deconvolution layer who is followed by Tanh activation.
Discriminator architecture has four convolution layers (kernel size 5, stride 2) with the following input, hidden feature-maps, output size: 3x32x32 → 128x16x16 → 256x8x8 → 512x4x
→ 1x1x1.
Batch-normalization is applied to each layer except the last one.
Activation are leaky-ReLU.
In our experiments, each model are trained with 10 generators.
As for discriminator, DCGAN and
MGAN has one discriminator while probabilistic models (BGAN and ours) have 4 discriminators (i.e. 4 Monte Carlo samples from discriminator distribution).
Training hyperparameters : All models are optimized by Adam(Kingma & Ba, 2014) with a learning rate of 2 × 10 4 .
For probabilistic methods, the SGHMC noise factor is set as 3 × 10 2 .
Following the conﬁguration in MGAN, the batch size of generators and discriminators are 120 and 64.
Note that, since probabilistic model has 10 generator Monte Carlo samples and 4 discriminators, indeed batch size for every generator and discriminator is 12 and 16 respectively.
D A FULL VISUALIZATION OF PROJECTED HIT SETS
In this section we shows projected hit sets for all models under different GAN objectives.
[Figure] [Figure]
As we can see, training the WGAN objective leads to much worse performance both
, our are
[Figure]
Three probabilistic models performs perfectly in this
, while miss although employs an additional to by different generators to be disjoint, it still suffers from collapsing problem.
This is because in MGAN, generators may still disjoint samples and
[Figure]
Speciﬁcally, three probabilistic models generate data almost perfectly covering the ground-truth ‘squares’ while the optimization-based methods have difﬁculty covering the whole ‘squares’ and tend to yield under-dispersed data distributions.
Note that since the GAN-NS objective is not in a min-max style, the success of BGAN is expected.
E M ORE CIFAR-10 GENERATED IMAGE RESULTS
In this section we shows images generated by all models trained on CIFAR-10 under different GAN objectives.
F T OY EXPERIMENT ON CATEGORICAL DISTRIBUTION
Setup.
In this toy example, we consider the case where X , Θ g , and Θ d are all ﬁnite sets, speciﬁcally
X = { x 1 , · · · , x N } , Θ g = { θ 1 g , · · · , θ N g g } , Θ d = { θ 1 d , · · · , θ N d d } .
The target data distribution is a categorical distribution Cat ( λ 1: N ) with λ i
= p data ( x i ) as the probability of generating data x i .
The data distribution generated by generator i is another categorical distribution p data ( x ; θ ig ) =
Cat ( α i 1: N ) .
Similarly, the distributions of generator and discriminator can also be parameterized as categorical distributions, i.e. q g ( θ g ) =
Cat ( β 1: N g ) and q d ( θ d ) =
Cat ( γ 1: N d ) .
In practice, we set N = 10 , N g = 20 , N d = 100 .
The parameters λ 1: N of the categorical distribution are obtained by ﬁrstly sampling i.i.d { ˜ λ j } Nj =1 from the uniform distribution U [0 , 1] and then normalizing λ j = ˜ λ
j (cid:80) Nj =1 ˜ λ j .
Other categorical distribution parameters
α i 1: N , β 1: N g and γ 1: N d are also initialized in a similar way.
For the discriminators, their function values are randomly generated from { D ( x i ; θ jd ) } N,N d i =1 ,j =1 ∼ U [0 , 1] .
Metric.
We employ l 1 distance for evaluation which can be directly computed on categorical distributions as follows.
[Equation]
Evaluation.
Our Bayesian model (Eqn. 3) and the BGAN (Eqn.
8) are evaluated with three different
GAN objectives: min-max, non-saturating and LSGAN.
which are listed in Table 1.
In each iteration step, the distributions of generator and discriminator are updated in a closed form since the normalization constant of the categorical posterior can be easily calculated via summation.
Result.
Figure 24 shows the curves of l 1 distance as a function of number of iterations.
For each curve, 20 random trials are averaged.
Our model converges to the optimum no matter which objective is employed, showing the robustness of our method.
However, the BGAN model is quickly trapped in a bad equilibrium.
This might be attributed to the ﬁxed prior of BGAN model which restricts the model space that is explored, while our model maintains a dynamic prior for each iteration which encourages the model to explore towards the optimum.
[Figure]
L
D i s t a n c e ( l o w e
r
i s b e tt e r )
[Figure]