B AYESIAN M ODELLING AND M ONTE C ARLO I NFERENCE FOR GAN
Anonymous authors
Paper under double-blind review
A BSTRACT
Bayesian modelling is a principal framework to perform model aggregation, which has been a primary mechanism to combat mode collapsing in the context of
Generative Adversarial Networks (GANs).
In this paper, we propose a novel
Bayesian modelling framework for GANs, which iteratively learns a distribution over generators with a carefully crafted prior.
Learning is efﬁciently triggered by a tailored stochastic gradient Hamiltonian Monte Carlo with novel gradient approximation to perform Bayesian inference.
Our theoretical analysis further reveals that our treatment is the ﬁrst Bayesian modelling framework that yields an equilibrium where generator distributions are faithful to the data distribution.
Empirical evidence on synthetic high-dimensional multi-modal data and image databases (CIFAR-10, STL-10, and ImageNet) demonstrates the superiority of our method over both start-of-the-art multi-generator GANs and other Bayesian treatment for GANs.
I NTRODUCTION
Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) is a popular method to learn a distribution on complex data such as natural images, videos, and texts.
However, it is notoriously hard to train and suffers from mode collapse .
There has been a series of works addressing these issues.
One noticeable thread focuses on objective design, which improves the original JensenShannon divergence with more stable pseudo-metrics such as f -divergence (Nowozin et al., 2016), χ 2 -divergence
(Mao et al., 2017), and Wasserstein distance (Arjovsky et al., 2017).
However, such treatment is inherently limited when a single generator does not include enough model capacity to capture the granularity in data distribution in practice.
Clearly, such generator can never produce accurate samples regardless of the choice of objectives.
An alternative remedy is to learn multiple generators instead of a single generator.
This type of methods (Hoang et al., 2018; Tolstikhin et al., 2017; Wang et al., 2016) is motivated by a straightforward intuition that multiple generators can better model multi-modal distributions since each generator only needs to capture a subset of the modes.
To entail model aggregation, Bayesian modelling is a natural and principled framework to articulate the aggregation process in a probabilistic way.
Recently, Saatci & Wilson (2017) propose Bayesian GAN, a probabilistic framework for GAN under Bayesian inference.
It shows that modelling the distribution of generator helps alleviate mode collapse and motivates interpretability of the learned generators.
The probabilistic model is built upon Bayesian models for generator and discriminator, whose maximum likelihood estimation can be realized as a metaphor of typical GAN objective.
Empirical study shows that Bayesian GAN provides state-of-the-art results for the semi-supervised image classiﬁcation task.
However, a critical theoretical question of this framework is still not well-understood: does it really converge to the generator distribution that produces the real data distribution?
Actually, our empirical study shows that Bayesian GAN suffers from severe performance loss due to this lack of convergence guarantee.
With this observation, we follow the prior work to exploit Bayesian modelling as a principled way to realize model aggregation, but approach this problem from a theoretical perspective; we analyze the developed treatment (including choice of priors, approximate inference, and its convergence property) φ 1 ( D ) φ 2 ( D ) φ 3 ( D )
M IN - MAX S TYLE
GAN (
[Table]
MAX
[Table]
log( D ) log(1 − D ) − log(1 − D )
Y ES GAN ( NON
[Table]
SATURATING ) log( D ) log(1 − D ) log( D )
N O W ASSERSTEIN GAN
D − D
D
Y ES
L EAST -S QUARES GAN ( D − 1)
D( D − 1)
N O and simultaneously propose a new Bayesian modelling framework with the desirable convergence guarantee and consequently superior empirical performance.
Our main contributions are:
• We theoretically establish, to our best knowledge, the ﬁrst Bayesian treatment of GANs such that any generator distribution faithful to the data distribution is an equilibrium.
• We prove the previous Bayesian method (Saatci & Wilson, 2017) for any minimax GAN objective induces incompatibility of its deﬁned conditional distributions.
• We propose two special Monte Carlo inference algorithms for our Bayesian model which efﬁciently approximate the gradient of a non-differentiable criterion.
• Empirical studies on synthetic high-dimensional multi-modal data and benchmark image datasets, CIFAR-10, STL-10, and ImageNet, demonstrate the superiority of the proposed framework over the state-of-the-art GAN methods.
R ELATED W ORK
Generative Adversarial Networks is a powerful class of methods to learn a generative model for any complex target data distribution.
There is a game between a generator and a discriminator.
Both of them adapt their strategies to maximize their own objective function involving the other:
[Equation]
Eqn. 1 gives a general mathematical form where p data is real data distribution and p gen ( · ; θ g ) are generated data distribution with generator parameter θ g .
The objective functions φ 1 , φ 2 , φ(termed as GAN objective in this paper) are elaborately chosen such that at the equilibrium, the generator generates the target data distribution.
Table 1 summarizes several widely used GAN objectives, including the original min-max version, non-saturating version of original GAN (Goodfellow (2016)), LSGAN (Mao et al.
(2017)), and WGAN (Arjovsky et al.
As reported in
Table 1, some GAN objectives which satisfy φ 3 ( · )
= − φ 2 ( · ) actually represent a min-max game, i.e. min θ g max θ g J d ( θ d ; θ g ) .
Training GAN with multiple generators is considered in several recent works to mitigate the mode collapse problem.
In the spirit of boosting algorithm, Wang et al.
(2016) propose to progressively train new generator using a subset of training data that are not well captured by previous generators, while Tolstikhin et al.
(2017) further propose a more robust mechanism to reweight samples in the training set for a new generator.
From the perspective of game theory, MIX-GAN (Arora et al., 2017) extends the game between a single generator and discriminator to the multiple-player setting.
Other works resort to third-party classiﬁers to help multiple generators and discriminators achieve better equilibrium, such as MAD-GAN (Ghosh et al., 2017) and the recent state-of-art method,
MGAN (Hoang et al., 2018).
Bayesian GAN proposed by Saatci & Wilson (2017) adopts a different approach which models generator and discriminator distributions by deﬁning the conditional posteriors (Eqn. 8).
The posterior likelihood is specially designed such that maximizing it exactly corresponds to optimizing GAN objectives.
The authors argue that compare to point mass ML estimation, learning the generator distribution which is multi-modal itself offers better ability to ﬁt a multi-modal data distribution.
To facilitate discussion, we categorize GAN frameworks into the following taxonomy: optimizationbased methods and probabilistic methods .
Optimization-based methods set up an explicit mini-max game between the generator and discriminator, where an ideal equilibrium typically characterize a generator faithful to data distribution.
In probabilistic methods, generators and discriminators evolve as particles of underlying distributions, where an equilibrium is searched from a stochastic exploration in the distribution space (of the generators and discriminators).
M ETHODOLOGY
We ﬁrst summarize the notations.
Second, we elaborate our Bayesian modelling for GAN and develop constituent prior and likelihood formulations.
Finally, we develop posterior inference algorithms for our Bayesian modelling.
A detailed discussion of the motivation of our modelling and the comparison with Bayesian GAN is delayed to Section 4.
3. N OTATIONS
p data ( x ) over a sample space X is the target data distribution we want to learn.
Our generator and discriminator are parameterized by θ g ∈ Θ g and θ d ∈ Θ d .
A generator with parameter θ g deﬁnes a mapping from a random noise vector z ∼ p
z to a random vector G ( z ; θ g ) .
The induced probability density of G ( z ; θ g ) is denoted as p gen ( x ; θ g ) .
A discriminator is a function that maps data to a real-valued score, i.e. D ( x ; θ d ) :
X → [0 , 1] (or X → R in some settings).
Further, we use q g ( θ g ) ∈
P Θ g , q d ( θ d ) ∈
P Θ d to denote the distribution over generators and discriminators respectively.
The total data distribution generated by generator following the density q g ( θ g ) is naturally a mixture of data distribution given by every single generator, p model ( x ; q g ) =
E θ g ∼ q g ( θ g ) [ p gen ( x ; θ g )] .
Our goal is to ﬁnd a generator distribution q ∗ g ( θ g ) such that the total generated data distribution matches our target, i.e. p model ( x ; q ∗ g ) (cid:39) p data ( x ) .
J g ( θ g ; θ d ) and J d ( θ d ; θ g ) denote objective functions of generator and discriminator as introduced in Eqn.
The common choices 1 are listed in Table 1.
With a slight abuse of the notation, we extend the notation J g ( θ g ; D ∗ ) by replacing D ( x ; θ d ) in equation 1 with any score function D ∗ .
Then J g ( θ g ; θ d ) can be viewed as an abbreviation of J g ( θ g ; D ( · ; θ d )) .
Likewise, J d ( θ d ; p gen ( · )) represents discriminator objective given a virtual generator that generates data with density p gen ( · ) .
3. B AYESIAN M ODELLING FOR GAN
The same as Bayesian GAN, we iteratively update the generator and discriminator.
We denote q ( t ) g and q ( t ) d are their distributions at time step t .
Likelihood.
The likelihood function indicates a preference for generators and discriminators, given current distributions of generator and discriminator.
Therefore, it should encode the information that distributionally reﬂect the objective of generators J g and discriminators J d .
An ideal solution is to form a distribution that is proportional to quantities that evaluate such ﬁtness:
[Equation]
where p ( t ) model ( x ) =
E θ g ∼ q ( t ) g [ p gen ( x ; θ g )] is the mixed data distribution under current generator distribution q ( t ) g and D ( t ) ( · ) =
E θ d ∼ q ( t ) d [ D ( · ; θ d )] is the averaged discriminating score function under current discriminator distribution q ( t ) d .
We emphasize, although sharing the same spirit of reﬂecting the GAN objectives in likelihood, there is a crucial difference between our likelihood model and that of Bayesian GAN.
We will revisit it in the later theory section.
Prior.
While the likelihood design is rather straightforward, an ideal prior is more involved.
When the generated data distribution is increasingly close to the real data distribution, there will be less information for discriminator to distinguish between them; consequently, the discriminator tends to assign equal scores to all data samples, resulting in equal likelihoods for all generators.
At that stage, a good strategy is to keep the generator distribution the same as the previous time step, since it already generates the desired data distribution perfectly.
Hence, we use the generator distribution in the previous time step as a prior for the next time step.
Such dynamically evolving prior for generator turns out to be crucial.
In Section 4.2, we show the Bayesian model in previous work suffers from bad convergence due to its ﬁxed and weakly informative prior.
In contrast, we set a uniform improper prior on the discriminator to pursuit unrestricted adaptability to evolving generators.
Posterior.
Integrating the prior mentioned above and the likelihood in Eqn. 2, we formulate our iterative Bayesian model as
[Equation]
This iterative process for updating q ( t ) g and q ( t ) d can be universally applied to any common GAN objectives with a guarantee of effectiveness (i.e., any desired generator distribution is a convergence point of the iteration) which will be formally stated in Section 4.1.
3. I NFERENCE A LGORITHM
So far we have introduced our Bayesian modelling for GAN.
In this section we develop novel inference algorithms to compute the posterior.
Similar to most advanced Bayesian methods, exact calculation of the posterior is intractable .
Following the strategy in Saatci & Wilson (2017), we adopt Stochastic Gradient Hamiltonian Monte Carlo to generate samples from the posterior.
In each iteration, M samples { θ ( t ) g,m } Mm =1 are used to approximate the generator distribution q ( t ) g .
[Equation]
Empowered by the adapted SGHMC (algorithm 1 in the appendix), we are able to sample from q ( t +1) g and q ( t +1)
d based on gradients in Eqn.
4 and Eqn. 5.
The gradients come from two sides: the
GAN objective J g , J d and the prior q ( t ) g .
Getting GAN objective’s gradient is easy while computing the prior’s gradient, ∇ θ g log q ( t ) g
( θ g ) , is actually non-trivial since we have no exact analytic form of
q ( t ) g
( θ g ) .
To address this challenge, we propose the following two methods to approximate ∇ θ g log q ( t ) g
( θ g ) , leading to two practical inference algorithms.
Gaussian Mixture Approximation (GMA) .
Although the analytic form of the distribution q ( t ) g ( θ g ) is unknown, we have M g Monte Carlo samples { θ ( t ) g,m } M g m =1 which enables us to directly approximate the distribution as a Mixture of Gaussian in the left side of Eqn. 6, where σ is a hyper-parameter and
C is the normalization constant.
Then we derive the prior gradient approximation as shown in the right side of Eqn. 6.
[Equation]
Partial Summation Approximation (PSA) .
From Eqn. 5, actually we can make an interesting observation that the prior gradient can be recursively unfolded as a summation over all historical
GAN objective gradients, shown as:
[Equation]
Therefore if we store all historical discriminator samples { θ ( i ) d,m } t,M d i =1 ,m =1 , the prior gradient can be computed accurately via simple summation.
Practically, computing gradients with all discriminator samples costs huge amount of storage and computational time, which is unaffordable.
Hence we propose to maintain a subset of discriminators by subsampling the whole sequence of discriminators.
T HEORY
In this section, we ﬁrst present the good convergence property of our Bayesian model.
Second, we theoretically analyze the distribution evolution of Bayesian GAN (which will be referred to as BGAN in the rest of the paper) and compare BGAN with our model.
All proofs are included in the appendix (Section A) due to the limited space.
[Figure]
4. C ONVERGENCE P ROPERTY OF O UR M ODEL
We say a generator distribution is ideal if the generator following this distribution produces the target data distribution.
Theorem 1 shows that any ideal generator distribution is an equilibrium of the dynamics deﬁned in Eqn. 3.
Although its mathematical proof involves more elaboration, the idea behind is quite simple.
When the generator distribution is ideal, the discriminator is not able to distinguish the synthetic data from real.
Thus the averaged discriminator function will degenerate to a constant function.
Afterwards, the generator distribution will remain unchanged since the discriminator essentially puts no preference over generators.
Here, we note that the discriminator is only involved in the likelihood.
The prior still needs to be carefully designed so that the model can converge to an equilibrium where the generator is ideal.
Simply choosing a weakly informative prior as Bayesian GAN did will not give Theorem 1.
Theorem 1.
Assume the GAN objective and the discriminator space are symmetry.
For any ideal generator distribution q ∗ g ( θ g ) satisfying ± ∗ (cid:44) E θ g ∼ q ∗
g
[ p gen ( · ; θ g )]
= p data , there exists a discriminator distribution
q ∗ d such that D ∗ ( · ) (cid:44) E θ d ∼ q ∗ d D ( · ; θ d ) ≡ c 2 .
Moreover, q ∗ g and q ∗ d is an equilibrium of the dynamic deﬁned in Eqn. 3.
4. C OMPARISON WITH B AYESIAN GAN
This section presents analyses of the BGAN algorithm, where we ﬁnd a theoretical issue in its convergence and highlight the importance of our renovation of the prior and likelihood.
Corollary 1 states our derivation of posterior modelling in BGAN, where p g ( θ g ; α g ) , p d ( θ d ; α d ) are the predeﬁned priors.
In practice, BGAN use weakly informative Gaussian prior.
Corollary 1.
The Bayesian GAN algorithm actually performs distribution dynamics in Eqn. 8.
[Equation]
Difference in Likelihood.
The subtle adjustment of our likelihood term lies in the order of taking expectation.
As shown in Eqn. 2, our choice of likelihood yields a concrete physical meaning.
Our discriminator likelihood explicitly evaluates the discriminator ability of distinguishing real data distribution and total data distribution generated by all generators.
Hence, our approach matches the target data distribution with mixed data distribution produced by generators, while BGAN may not.
Difference in Prior.
The choice of prior also plays an important role in convergence.
A weakly/noninformative prior, as adopted in BGAN, prevents the generators from convergence even if they already produce the data distribution faithfully.
The phenomenon arises from the fact that the information provided by discriminator vanishes when the generators are ideal and the resulting generator posterior will degenerate to the prior.
To remedy this issue, our solution is to take generator distribution at previous time step as the prior.
Then whenever the discriminator degenerate to a constant, the generator distribution will stay unchanged because the prior is itself.
We demonstrate the superior convergence property of our model on a categorical distribution, where analytic posterior computation of various choices of likelihood and prior are feasible; we compute the exact equilibrium of Bayesian GAN under all the four combinations of the priors and likelihoods we proposed and in the BGAN paper.
The experiment details are in the appendix (section D).
Figureis an example of the data distributions generated by each model after it converges.
Among the combinations, our model is the only formulation that yields proper convergence , which validates our theoretical analysis.
Compatibility Issue .
We further show BGAN’s choice of likelihood and prior leads to theoretical issues.
Speciﬁcally, BGAN is not suitable for any minimax-style GAN objective due to the incompat- ibility of its conditional posteriors.
This problem may limit the usage of BGAN since many widely used GAN objective is in min-max fashion, such as the original GAN and WGAN.
Consider a simple case where we use only one Monte Carlo sample for the distributions q ( t ) g and q ( t ) d .
Then the distribution evolution in Eqn.
8 will degenerate to a Gibbs sampling process.
[Equation]
Thus θ ( t ) g and θ ( t ) d are implicitly sampled from a joint distribution of θ d and θ g deﬁned by the conditionals p ( θ g | θ d ) =
exp {J g ( θ g ; θ d ) } p ( θ g | α g ) and p ( θ d | θ g ) =
exp {J d ( θ d ; θ g ) } p ( θ d | α d ) .
However, our theoretical analysis shows that such a presumed joint distribution does not exist when J d ( θ d ; θ g )
=
−J g ( θ g ; θ d ) .
Speciﬁcally, Lemma 1 shows the existence of a joint distribution satisfying the conditionals in Eqn.
9 requires the GAN objective to be decomposable, i.e.
∃ φ g , φ d , s.t.
J d ( θ d ; θ g )
=
φ g ( θ g ) + φ d ( θ d ) .
Apparently, no GAN objective is decomposable.
Therefore, conditionals in Eqn.
9 are actually incompatible.
Sampling with incompatible conditional distribution is problematic and leads to unpredictable behavior (Arnold & Press, 1989).
Lemma 1. Consider a joint distribution p ( x, y ) of variable X and Y .
Its conditional distributions can be represented in the forms of p ( x | y )
∝ exp { L ( x, y ) } q x ( x ) and p ( y | x ) ∝ exp {− L ( x, y ) } q y ( y ) only if X and Y are independent, i.e., p ( x, y )
=
p ( x ) p ( y ) and L ( x, y ) is decomposable, i.e., ∃ L x and L y , L
( x, y )
=
L x ( x ) + L y ( y ) .
E XPERIMENTS
In this section, we evaluate our model with two inference algorithms proposed in Section 3.3 (denoted as ours-GMA and ours-PSA ).
We compare with three baselines: 1) GAN (or DCGAN 2 ) : naively trained multiple generators in the vanilla GAN framework; 2) MGAN :
Mixture GAN (Hoang et al., 2018) which is the start-of-art method to train GAN with multiple generators; 3) BGAN : Bayesian
GAN (Saatci & Wilson, 2017).
For each model, we conduct thorough experiments with the four different GAN objectives introduced in Table 1, which are referred to as GAN-MM, GAN-NS, WGAN and LSGAN here.
For a fair comparison, each model has the same number of generators with the same architecture.
Discriminator architectures are also the same except for that of MGAN which has an additional branch of the classiﬁer.
To facilitate reproducibility, we report implementation and experiment details in Section B of the appendix.
5. H IGH - DIMENSIONAL M ULTI - MODAL S YNTHETIC D ATASET
Dataset.
Consider learning a data distribution in a high dimensional space X
=
R D , which is a uniform mixture of n modes.
Each mode lies on a d -dimensional sub-space of X .
We call this d -dimensional sub-space as mode-space of the i -th mode.
Speciﬁcally, the data of the i -th mode is generated by the following process,
[Table]
[Equation]
In our experiment
[Table] n , D [Table] and d are set to 10 , 100 , and 2 .
Hyper-parameters for A and b are set to be σ A = σ b = 5 .
Each model train ten generators.
Metric.
We deﬁne projection distance (cid:15) p for generated data sample x as the minimum of Euclidean distance from x to any of mode-spaces i.e. (cid:15) p ( x ) =
min i (cid:15) i ( x ) (cid:44) (cid:107) x − A i ( A Ti A i ) − 1 A Ti x (cid:107) 2 .
Then we set a threshold 3 η to test the belonging of x , i.e. the data samples whose Euclidean distance to the mode-space is below η are considered as belonging to that mode.
The trained models are evaluated by the samples { x k } Kk =1 ∼ p model it generates.
We deﬁne hit set H i (cid:44) {
x k | (cid:15) i ( x k ) <
η } to indicate the samples belong to each mode.
We further deﬁne projected hit set , PH i (cid:44) { ( A T i A i ) −[-
-]
1 A T i x − b i | x ∈
H i } by projecting data in each hit set back to the canonical low dimensional space.
Now we introduce three evaluation metrics: hit ratio , hit distance , and cover error .
Hit ratio
H r (cid:44) (cid:80) ni =1 |H i | /K is the percentage of generated data belonging to any of the modes of real data.
Hit distance H d (cid:44) (cid:80) ni =1 (cid:80) x ∈H i (cid:15) i ( x ) / (cid:80) ni
=1 |H
i | is the averaged projection distance over all data in the hit set.
Lastly, cover error C e evaluates how well the generated data covers each mode.
Essentially it computes the KL-divergence between the estimated distribution of samples in PH i and the uniform distribution over [ − 1 , 1] d .
Formally, it is deﬁned as the averaged KL-divergence on n modes i.e. C e (cid:44) 1 n (cid:80) ni
=1 KL (ˆ p ( · ; PH i ) (cid:107)U [ − 1 , 1] d ) .
The intuition is that if data generated is close to the ground truth distribution, they should be uniformly distributed in the
Optimization-Based v.s. Probabilistic.
The left part of Table 2 summarizes the results in terms of hit ratio and hit distance .
Probabilistic methods including our algorithms and BGAN always achieve a hit ratio of 1 , which means every data point generated from these models is very close to one mode of the target distribution.
On the other hand, optimization based methods, both GAN and MGAN, consistently have a signiﬁcantly larger hit error, and sometimes may even generate data samples that do not belong to any mode.
Moreover, the data distribution generated by the optimization-based methods ﬁts the target uniform distribution much worse than its probabilistic counterparts, which is quantitatively reﬂected by the cover error showed in the right side of Tableand visually demonstrated by the projected hit sets in Figure 2.
According to the visualization, data generated by GAN or MGAN tend to be under dispersed and hardly cover the whole square region of the true mode, while data generated by probabilistic methods align much better with the ground truth distribution.
We attribute this superiority to stronger exploration power in the generator space coming from the randomness in probabilistic methods.
Bayesian GAN v.s. Our Methods.
The incompatibility issue of BGAN with minimax-style GAN objectives theoretically derived in Section 4.2 is empirically veriﬁed in our experiments.
As visualized in Figure 2, with the GAN-MM objective, BGAN is trapped in a local equilibrium and fails in capturing one mode of the true data.
Besides, as shown in Table 2, BGAN with the WGAN objective achieves much poorer coverage than with other GAN objectives, while our model is much more robust to the choice of GAN objectives (consistently lower cover errors).
A qualitative comparison is made in Figure 9 (in the appendix) which shows the data distribution generated by BGAN trained with WGAN objective tends to shrink.
More visual illustrations under different GAN objectives are placed in Section E of the appendix.
5. N ATURAL I MAGE D ATASET
Datasets.
We evaluate our method on 3 widely-adopted datasets: CIFAR-10 (Krizhevsky et al., 2010), STL-10 (Coates et al., 2011) and ImageNet (Deng et al., 2009).
CIFAR-10 has 50k training
[Table] [Table]
[Figure]
The tenth generator of MGAN (Figure 3(a)) and the ﬁrst of BGAN (Figure 3(b) while generators of our method all work well.
DCGAN (Figure 10 in the appendix) also presents ’single generator collapse’ issue.
Note that, mode collapse also happens when baseline models trained with other GAN objectives.
and 10k test 32x32 RGB images from 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. STL-10, containing 100k 96x96 RGB images, is a more diverse dataset than CIFAR-10. ImageNet has over 1.2 million images from 1,000 classes and presents the most diverse dataset.
For a fair comparison with baselines, we use the same settings as MGAN.
We resize the
STL-10 and ImageNet images down to 48x48 and 32x32 respectively.
Evaluation Protocols.
We employ two common image generation metrics: Inception Score (Salimans et al.
and Fr ´ echet Inception Distance (Heusel et al. (2017)).
Inception Score computes exp( E x [ KL ( p ( y | x ) (cid:107) p ( y ))]) where p ( y | x ) standards the predicted label distribution by a pre-trained
Inception model (Szegedy et al., 2015) and p ( y ) is the average of p ( y | x ) over all images in the dataset.
Inception Score (IS) reﬂects the ﬁdelity and diversity of images and is well-correlated with human judgment (Salimans et al., 2016).
However, it does not measure the similarity between the real data and the synthetic data.
Therefore, as a complementary, we use Fr´echet Inception Distance (FID).
Speciﬁcally, it measures the Fr´echet distance between two image distributions in the feature embedding space conduct by the Inception model.
More details are included in the appendix
Model Architectures.
Inspired by Hoang et al.
(2018), we adapted the parameter sharing technique for both generators and discriminators.
Speciﬁcally, generators (of one model) are not disjoint neural networks.
They only differ at the ﬁrst layer and sharing all parameters at the following layers.
This tied parameter design reduced the model complexity and boost the performance.
Especially for the probabilistic model such as Bayesian GAN, we observe a signiﬁcant enhancement of generated image qualities compare to the original result (Figure 7 in Saatci & Wilson (2017)).
More implementation details such as hyper-parameters of network layers are reported in the appendix (Section B). [Figure]
Quantitative Results.
Table 3 and Table 4 summarize the Inception scores and FIDs on the three benchmark datasets for our model and baselines.
For each model, we denote the epoch delivering highest ’IS − 0 .
1 FID’ as the best epoch and report the scores of the model at this epoch.
We design this ’IS − 0 .
1 FID’ principle to neutralize the discrepancy between Inception score and FID.
Because
Inception score and FID may not be consistent with each other, it is possible that the Inception score improves but the FID gets worse when we evaluate the model at a different checkpoint.
Hence, we need a rule to pick one point of the Pareto frontier of the model performance.
Overall, our proposed model outperforms the baselines on all three benchmark image datasets.
Furthermore, according to results on CIFAR-10, our model achieves the best performance under all GAN objectives.
Generally, probabilistic methods achieve better scores than optimization based methods, which indicates that injecting stochasticity into GAN training helps generate more multi-modal images.
Note that the performance gap increases as the dataset gets more diverse.
Speciﬁcally, when using GAN-NS objective, our model improves FID by 4 .
on CIFAR-10 (Table 3) in comparison to MGAN.
While the FID improvements are 4 . 82 and 18 . 06 on STL-10 and ImageNet (Table 4), respectively.
Besides, we note that Bayesian GAN has a signiﬁcant performance drop when accompanied by min-max style GAN objectives, which provides another empirical evidence for our theory analysis in
Section 4.2.
By contrast, our Bayesian method ﬁts any GAN objectives and constantly performs well.
Qualitative Results.
Figure 3 display the samples randomly generated by the baselines and our model.
Each row in the ﬁgure contains samples of one learned generator.
In Figure 3 and Figure(in the appendix), all the three baselines noticeably suffer from mode collapse.
Indeed, almost in every training trial, one or two generators of the baseline models degenerate during the training.
Hoang et al.
(2018) already notice that mode collapse in one of the generators could happen after a long training procedure (around 250 epochs).
Our experiment shows Bayesian GAN also have this issue.
However, our novel Bayesian model is robust to mode collapse.
Visual results for the entire ablation study on CIFAR-10 are included in Section F of the appendix.
Figure 4(a) and Figure 4(b) display images generated by our model trained on ImageNet and STL-10.
Each row in these two ﬁgures contains generated images from one generator.
The results show that our method is robust to ’single generator mode collapse’ in both STL-10 and ImageNet.
We also include cherry-picked results on STL-10 to exhibit the capability of our model to generate visually appealing images with complex details while improving the robustness against mode collapse.
D ISCUSSION
In this paper, we propose a novel Bayesian modelling framework for GAN, with a likelihood function establishing a connection to existing GAN models and a novel prior stabilizing the inference process.
We propose scalable Bayesian inference algorithms which are asymptotically correct.
As future work, we plan to extend the proposed framework to non-parametric Bayesian modelling and investigate more theoretical properties of GANs in the Bayesian context.
Developing Bayesian generalization for deep learning models is not a recent idea and happens in many ﬁelds other than generative models.
By this work, we emphasize the importance of going beyond the intuition and understanding the theoretical behavior of the Bayesian model.
We hope that our work helps inspire continued exploration into Bayesian deep learning from a more rigorous perspective.
A O MITTED PROOFS
Theorem 1.
This theorem is general and holds when the GAN objective and the discriminator space have symmetry.
The symmetry of GAN objective means its functions φ 1 and φ 2 satisfy that ∃ c ∈ R , ∀ x ∈ R , φ 1 ( x ) ≡ φ 2 ( c − x ) .
While the symmetry of discriminator space Θ d indicates that for any θ d ∈ Θ
d there is a θ (cid:48)
d ∈ Θ d such that D ( x ; θ d ) ≡ c − D ( x ; θ (cid:48) d ) .
Note that the symmetry condition are very weak, ﬁrst it holds for all the common choices of GAN objectives such as those listed in Table 1.
Second, it holds for neural network which is the most common parameterization for discriminator in practice.
Proof.
[Equation] [Equation]
Eqn. 11 and Eqn.
12 prove that q ∗ d ( θ d ) ∝ exp( J d ( θ d ; p ∗ model )) satisﬁes E θ d ∼ q ∗ d D ( · ; θ d ) ≡ c 2 .
Note that, in the above equations, S ( θ d ) denotes the symmetric discriminator of θ d satisfying
D ( x ; S ( θ d ))
≡ c − D ( x ; θ d ) .
Hence according Eqn. 13, we know J g
( θ g ; D ∗ ) is a constant.
[Equation] [Equation]
Thus the generator distribution will not change based on the dynamics in Eqn.
3 since q ∗ g ( θ g )
∝ exp {J g ( θ g ; D ∗ ) } q ∗ g ( θ g ) .
Corollary
Proof.
[Equation]
Algorithm 1 from the original paper (Saatci & Wilson, 2017) implies that Eqn.
14 where θ kd ∼ q ( t ) d ( θ d ) are Monte Carlo samples of discriminator and every z ( i ) is a mini-batch containing n g noise samples.
Be deﬁnition p
( θ g | z
( i ) , θ kd )
∝ (cid:81) n g j =1 D ( G ( z ( i ) j ; θ g ); θ kd ) p ( θ g | α g )
(Eqn. 1 in the original paper), each term 4 log p ( θ g | z
( i ) , θ kd )
= (cid:80) n g j =
φ
( D ( G ( z ( i ) j ; θ g ); θ kd ))
+log p ( θ g | α g ) .
Hence, the total summation is a Monte Carlo approximation of the expectation in the right side of Eqn. 15.
The same derivation can be done to q ( t +1) d .
Together, we get Corollary 1.
Lemma
Proof.
[Equation]
⇒ X
[Equation]
Y are independent .
= ⇒ p ( x ) = p ( x | y )
=
⇒ L ( x, y )
= log p ( x ) − log q x ( x ) −
log α
( y ) =
⇒ L ( x, y ) is decomposable.
(16) where α ( y ) =
( (cid:82) exp { L ( x, y ) } q x ( x ) dx ) − 1 and β ( x )
=
( (cid:82) exp {− L ( x, y ) } q y ( y ) dy ) − 1 .
B E XPERIMENT DETAILS B. N ATURAL I MAGE D ATASET
For MGAN results, we adopt the ofﬁcial Tensorﬂow implementation 5 .
While, for our model and other baselines, we implement them in PyTorch (Paszke et al., 2017).
Remark on Inception score and FID .
Barratt & Sharma (2018) point out that Inception score is sensitive to the inception model used and the number of data splits in the computation.
This is also true for Fr ´ echet Inception Distance (FID).
We ﬁnd that the FID computed by a PyTorch Inception model 6 is much lower than the FID given by a Tensorﬂow model 7 .
In our experiments, to facilitate a fair comparison with prior work, we compute Inception score and
FID using the Tensorﬂow Inception model.
We adopt the ofﬁcial Tensorﬂow implementation for FID to compute both Inception score and FID.
We will release our evaluation code soon.
Model architecture :
In our experiments, each model is trained with 10 generators.
As for discriminator, DCGAN and MGAN have one discriminator while probabilistic models (BGAN and ours) have 4 discriminators (i.e. 4 Monte Carlo samples from discriminator distribution).
The neural network structures are the same as MGAN.
As reported in Table 4,5,6 in Section C.2 of the original MGAN paper.
Brieﬂy, the structures are the following.
Generator architecture has four (or ﬁve) deconvolution layers (kernel size 4, stride 2) with the following input, hidden feature-maps, output size: 100x1x1 → 512x4x
[Table]
→ 3x32x32.
Every deconvolution layer is followed by batch-normalization layer and Relu activation except for the last deconvolution layer who is followed by Tanh activation.
Discriminator architecture has four (or ﬁve) convolution layers (kernel size 5, stride 2) with the following input, hidden feature-maps, output size: 3x32x32 → 128x16x16 → 256x8x8 → 512x4x4 → 1x1x1.
Batch-normalization is applied to each layer except the last one.
Activations are leaky-ReLU.
Training hyperparameters : All models are optimized by Adam(Kingma & Ba, 2014) with a learning rate of 2 × 10 4 .
For probabilistic methods, the SGHMC noise factor is set as 3 × 10 2 .
Following the conﬁguration in MGAN, the batch size of generators and discriminators are 120 and 64.
Note that, since probabilistic model has 10 generator Monte Carlo samples and 4 discriminators, indeed batch size for every generator and discriminator is 12 and 16 respectively.
B. H IGH - DIMENSIONAL MULTI - MODAL SYNTHETIC DATASET
Model Architecture : Each generator or discriminator is a three layer perceptron.
For the generator, the dimensions of input, hidden layer and output are 10, 1000, and 100 respectively.
For the discriminator, the dimensions of input, hidden, output layers are 100, 1000, and 1.
All activation functions are leaky ReLU(Maas et al., 2013).
Training hyperparameters : All models are optimized by Adam (Kingma & Ba, 2014) with a learning rate of 10 − 4 .
For probabilistic methods, the SGHMC noise factor ( α in algorithm 1) is set as 10 − 1 .
C I NFERENCE ALGORITHM
Stochastic Gradient Hamiltonian Monte Carlo (Chen et al., 2014) is a gradient based MCMC sampling method.
It use noise gradient estimation of potential function to generate sample from a given distribution.
Our inference algorithm (algorithm 1) is based on this brilliant algorithm.
Input: Initial Monte Carlo samples of { θ (0)
d,m } M d
m
=
and { θ (0) g,m } M g m =1 , learning rate η , SGHMC noise factor α , number of updates in SGHMC procedure L , number of updating iterations T .
for do for m do θ d,m ← θ ( t ) d,m for l = 1 to L do v ← (1 − α ) v + η ∇ θ d log q ( t +1)
d ( θ d,m ) + n ; n ∼ N (0 , 2 αηI ) θ d,m ← θ d,m + v end for θ ( t +1)
d,m ← θ d,m end for for m
1 to M g do θ g,m ← θ ( t ) g,m for l
1 to L do v ← (1 − α ) v + η ∇ θ g log q
( t +1)
g ( θ g,m ) + n ; n ∼ N (0 , 2 αηI ) θ g,m ← θ g,m + v end for θ ( t +1) g,m ← θ g,m end for end for
D T OY EXPERIMENT ON CATEGORICAL DISTRIBUTION
Setup.
In this toy example, we consider the case where X , Θ g , and Θ d are all ﬁnite sets, speciﬁcally
X = { x 1 , · · · , x N } , Θ g = { θ 1 g , · · · , θ N g g } , Θ d = { θ 1 d , · · · , θ N d d } .
The target data distribution is
[Figure]
[Table]
a categorical distribution Cat ( λ 1: N ) where λ i = p data ( x i ) is the probability of generating data x i .
Generator G i generates data following the categorical distribution p data ( x ; θ ig ) =
Cat ( α i 1: N ) .
Further, the probability distributions of generator and discriminator are categorical distributions q g ( θ g )
=
Cat ( β 1: N g ) and q d ( θ d ) =
Cat ( γ 1: N d ) .
In practice, we set N = 10 , N g = 20 , N d = 100 .
In each experiment trial, target data distribution λ 1: N and data distributions for each generator α i 1: N are ﬁxed and initialized by the following categorical distribution generating procedure.
We ﬁrst independently sample from the uniform distribution U [0 , 1] to get { ˜ λ j } Nj =1 and then normalize them to get a categorical distribution λ j = ˜
λ
j (cid:80) Nj =1 ˜ λ j .
For the discriminators, their function values are randomly generated from a uniform distribution, i.e.
{ D ( x i ; θ j d ) } N,N d i =1 ,j =1 ∼ U [0 , 1] .
Models in comparison.
As listed in Table 5, we compare four models with different pairs of likelihood and prior.
There are two choices of likelihood, expectation of objective value and objective value of expectation .
Mathematically, expectation of objective value likelihood has the following formula:
• Generator likelihood l g ( θ g ) of observing discriminator distribution q d ( θ d ) is proportion to exp { E θ d ∼ q d
( θ d ) J g ( θ g ; θ d ) } .
• Discriminator likelihood l d
θ d ) of observing generator distribution q g ( θ g ) is proportion to exp { E θ g ∼ q g ( θ g )
J d ( θ d ; θ g ) } .
While the objective value of expectation likelihood is different as follows:
• Generator likelihood l g ( θ g ) of observing discriminator distribution q d ( θ d ) is proportion to exp {J g ( θ g ;
E θ d ∼ q d
θ d ) D ( · ; θ d )) } .
• Discriminator likelihood l d
θ d ) of observing generator distribution q g ( θ g ) is proportion to exp {J d ( θ d ; E θ d ∼ q d
( θ d ) p gen ( · ; θ g )) } .
Thus model A indeed is the design of Bayesian GAN, while model D is our proposed model.
Model
B and C are introduced to conduct the ablation study.
Metric.
We employ l 1 distance for evaluation which can be directly computed on categorical distributions as follows.
[Equation]
Evaluation.
In the categorical distribution settings, all the likelihood, prior and posterior computing can be done analytically.
For each model, we update the generator and discriminator distributions iteratively and monitor the distance between the target data distribution and the data distribution generated by the model.
We experiment with two different choices of GAN objectives, GAN-MM and GAN-NS.
Note GAN-MM is a minimax-style objective.
Result.
Figure 5 shows how l 1 distance changes as the number of updating iterations increases.
As we can see, model A and model B are easily and quickly trapped into bad local minima, indicating the convergence issue caused by non-informative prior.
Interestingly, model C presents bifurcate results when accompanied by different GAN objectives.
Its abnormal behavior in the setting of using
GAN-MM objective indicates that the expectation of objective value likelihood used in Bayesian
GAN does not ﬁt the minimax-style GAN objectives.
Finally, our proposed Bayesian model converges the fastest (towards the global optima) and is robust to all GAN objectives.
E A FULL VISUALIZATION OF PROJECTED HIT SETS
In this section we shows projected hit sets for all models under different GAN objectives.
[Figure] [Figure]
As we can see, training the WGAN objective leads to much worse performance both
, our methods are
[Figure]
probabilistic performs perfectly employs an to
This
[Figure]
Speciﬁcally, three probabilistic models generate data almost perfectly covering the ground-truth ‘squares’ while the optimization-based methods have difﬁculty covering the whole ‘squares’ and tend to yield under-dispersed data distributions.
Note that since the GAN-NS objective is not in a min-max style, the success of BGAN is expected.
F M ORE CIFAR-10 GENERATED IMAGE RESULTS
In this section we shows images generated by all models trained on CIFAR-10 under different GAN objectives.