F OLDED
H AMILTONIAN
M ONTE
C ARLO
FOR
B AYESIAN G ENERATIVE A DVERSARIAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Adversarial Networks (GANs) can learn complex distributions over images, audio, and data that are difﬁcult to model.
We deploy a Bayesian formulation for unsupervised and semi-supervised GAN learning.
We propose Folded
Hamiltonian Monte Carlo (F-HMC) within this framework to marginalise the weights of the generators and discriminators.
The resulting approach improves the performance by having suitable entropy in generated candidates for generator and discriminators’ weights.
Our proposed model efﬁciently approximates the high dimensional data due to its parallel composition, increases the accuracy of generated samples and generates interpretable and diverse candidate samples.
We have presented the analytical formulation as well as the mathematical proof of the
F-HMC.
The performance of our model in terms of autocorrelation of generated samples on converging to a high dimensional multi-modal dataset exhibits the effectiveness of the proposed solution.
Experimental results on high-dimensional synthetic multi-modal data and natural image benchmarks, including CIFAR-10,
SVHN and ImageNet, show that F-HMC outperforms the state-of-the-art methods in terms of test error rates, runtimes per epoch, inception score and Frechet
Inception Distance scores.
I NTRODUCTION
Generative adversarial networks (GANs)
[Goodfellow et al.
(2014)] have received traction in the ﬁeld of deep generative models.
The development of GANs covers a wide range from multi-layer perceptrons to the BigGAN framework [Brock et al.
with residual blocks and self-attention layers (Zhang et al.
to synthesise realistic images.
Despite GAN’s effectiveness in generating realistic images, it experiences mode collapse, which occurs when the generator over-optimises for a particular discriminator and the discriminator never learns how to escape the trap.
Recent work has focused on alternative metrics such as f-diversities [Nowozin et al.
(2016)] or Wasserstein divergences [Arjovsky et al.
(2017)] to substitute the Jensen-Shannon divergence inherent in traditional
GAN training to alleviating several practical issues.
Saatci & Wilson (2017) recently proposed Bayesian GAN, a probabilistic framework for GANs based on Bayesian inference.
It demonstrates how modelling the distribution of generators alleviates mode collapse and motivates the interpretability of learned generators.
GAN training measures the full posterior distribution across network weights in a single-mode based on mini-max optimisation.
Even if the generator does not recall training instances, samples from the generator are expected to be excessively compact compared with data distribution samples.
In fact, as demonstrated by Bayesian
GAN, a posterior distribution over the generators’ parameters can be vast and highly multi-modal to model the real data distribution by fully reﬂecting the posterior distribution over the generator and discriminator parameters.
In addition, He et al.
(2019) proposed the probGAN, which is similar to the Bayesian GAN, iteratively learns a distribution over generators but with a carefully crafted prior.
Learning is triggered by a tailored Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) to perform Bayesian inference.
In both Bayesian GAN and ProbGAN, SGHMC is deployed to marginalised the parameters of the generator and discriminators.
The foundation of our idea in this paper is to utilise different samples for generator parameters in order to mitigate GAN collapse mode and more efﬁciently produce data samples with an adequate degree of entropy, especially with high-dimensional and highly correlated data.
To do so, we propose a Folded Hamiltonian Monte Carlo (F-HMC) to replace the SGHMC part of the Bayesian framework.
This proposal shares several desired properties with SGHMC, such as
1) being experimentally well-adjusted in training the GAN due to its Hamiltonian dynamics and 2) directly importing parameters such as the learning rate from gradient descent into the sampler.
Furthermore, it beneﬁts from the following advantages:
F-HMC explores more accurately the target density, especially in the scenario with highdimensional and highly correlated data.
F-HMC converges faster to the target density in terms of lag number.
F-HMC provides the practical advantage to the Bayesian GAN method by exploring a rich multi-modal distribution over the weight parameters of generators at an acceptable entropy level.
More importantly, because of the parallel composition, F-HMC has an efﬁcient run time in high-dimensional data approximation.
Our main contributions are listed below.
We will return to them in the experiment section to highlight each one of them.
We propose F-HMC to sample parameters of generators to create candidates from the multimodal high-dimensional distribution.
We have mathematically veriﬁed the functionality of F-HMC.
Empirical results on a high-dimensional multi-modal synthetic dataset show that the generated samples from our proposed method cover target distribution with more similarity to the target density concerning entropy value.
We have shown that using our proposed model, the semi-supervised learning algorithms on natural image datasets (ImageNet, SVHN
and CIFAR10) outperform the state-of-the-art in terms of inception scores (IS) and Frechet Inception Distance scores (FID).
The structure of the paper is as follow: We begin with the problem formulation in Section 2 and enhance the framework by introducing the proposed F-HMC in Section 4.
We also have provided the theoretical analysis of F-HMC to generate samples from the desired target.
Then, we have demonstrated the performance of our proposed model in Section 5.
Finally, Section 6 concludes the paper and discusses future work.
P ROBLEM F ORMULATION
Suppose having observed data D
= { x (cid:48) i , y (cid:48) i } Ni =1 from an unknown probability distribution p data where x (cid:48) describes the input and y (cid:48) denotes the corresponding label.
We would like to estimate p data which is a high-dimensional multi-modal distribution.
Bayesian GAN [Saatci & Wilson (2017)] investigates the distributions over the weight parameters of the generators and creates distributions over an inﬁnity space of generators and discriminators, corresponding to every conceivable setting of these weight vectors.
We build upon the problem formulation in Bayesian GAN [Saatci & Wilson (2017)] and formulate the posterior as p ( y (cid:48) | f ( x (cid:48) , α )) where f ( x (cid:48) , α ) =
Gen ( z, ˆ α g ) .
Here z represents white noise derived from p ( z ) , and ˆ α g represents distribution over generator parameters.
We denote that parameter set α consisting of two sub parameter set ˆ α g related to the generator and ˆ α d associated with the discriminators.
We require the weight candidates of generators and discriminators to create candidates for posterior estimation.
In this regard, we need to estimate posterior over ˆ α g , ˆ α d .
Since generators and discriminators are performing min-max optimisation, the parameter over ˆ α g and ˆ α d are interdependent.
First, generator weights ˆ α g are sampled from a prior p (ˆ α g | β g ) , and a particular generative neural network is constructed conditioning on these samples.
Then, white noise z derived from p ( z ) is transformed through the network Gen ( z ; ˆ α g ) to generate candidate data samples.
Conversely, discriminator conditioned on its weights Disc ( . ; ˆ α d ) produces the probability that these candidate samples are generated from the data distribution.
This process can be stated as follow considering
L as the likelihood:
[Equation]
g ) (1)
From the discriminator side, we need to form classiﬁcation likelihood that classiﬁes actual data from the generated samples and can be formulated as:
[Equation]
Here p (ˆ α d | β d ) indicates prior for ˆ α d .
By marginalising z from Equations 1 and 2, the equations get updated to p (ˆ α g | ˆ α d ) = (cid:82) p
(ˆ α g , z | ˆ α d ) dz and p (ˆ α d | ˆ α g )
= (cid:82) p (ˆ α d | z, X, ˆ α d ) dz .
We can approximate the posterior over ˆ α g and ˆ α g by iteratively sampling from p (ˆ α g | ˆ α d ) and p (ˆ α d | ˆ α g ) .
Therefore we can have the corresponding generators and discriminators to generate candidate samples from the multimodal high dimensional distribution ( p data ).
This paper proposes F-HMC as an efﬁcient sampling strategy, especially when the target is high-dimensional and highly correlated.
B ACKGROUND
This section brieﬂy reviews data sampling basics, such as the Hamiltonian Monte Carlo sampler and
Stochastic Hamiltonian Monte Carlo derived from [Chen et al.
Suppose one wants to sample from the posterior distribution of X given a set of independent observations x ∈ D :
[Equation]
(3) where the potential energy function U is given by
[Equation] [Equation]
Hamiltonian Monte Carlo (HMC) is a method for efﬁciently exploring the state space by proposing samples of X in a Metropolis-Hastings (MH) framework.
By incorporating auxiliary momentum variables, V , these suggestions are generated from a Hamiltonian system.
HMC creates samples from a joint distribution of ( X, V ) to sample from p ( X | D ) :
[Equation]
The samples of X have a marginal distribution p ( X | D ) if the resultant samples of V are simply discarded.
Here M is a mass matrix that, coupled with V , deﬁnes a kinetic energy term.
H ( X, V ) =
U ( X ) − 12 V T M − 1 V deﬁnes the Hamiltonian function.
H intuitively calculates the total energy of a physical system by utilising position variables X and momentum variables V .
To propose samples,
HMC simulates Hamiltonian dynamics.
[Equation]
Furthermore, SGHMC is based on the idea of combining stochastic optimisation with a ﬁrst-order
Langevin dynamic MCMC technique, demonstrating that adding the ”right amount” of noise to stochastic gradient ascent iterates results in samples from the target posterior as the step size is annealed.
SGHMC accomplishes this by including a “friction” term in the momentum update:
[Equation]
F-HMC MODEL
This section proposes F-HMC as an efﬁcient and scalable sampling strategy, particularly when the target is multi-modal and highly correlated (Contribution 1).
Additionally, we have presented the mathematical analysis of F-HMC (Contribution 2) in section 4.2 to prove that the F-HMC samples from the equivalent target distribution speciﬁed in Equation 7.
4. M ETHODOLOGY
In the scenario where the gradient elements are on dramatically different scales and highly correlated, the SGHMC cannot efﬁciently explore the target density.
As a pragmatic approach, we propose F-HMC, a parallelised algorithm that, instead of ﬁnding a single chain that samples from the whole distribution, combines samples from several chains.
Chains are running in parallel, each exploring a different region of the state space (e.g., a few modes only).
F-HMC assumes that a Gaussian mixture approximates the target density pi . In that case, each mixture component represents a reliable proposal in a speciﬁc region of the sample space S ⊂ R d . Assume pi has J modes and consider its approximation via the mixture model as follows: [Equation]
where N d ( x ; µ j , σ j ) is the probability density of a d-variate Gaussian distribution with µ mean and σ covariance matrix. We deﬁne the partition ˆ S = ∪ Jk =1 S j using the mixture representation in 8, so that pi is more similar to N d ( x ; µ j , σ j ) on each set S ( j ) than any other distribution entering 8, which is formulated as follow: [Equation] The approximation 8 along with the regions deﬁned in 9 enable us to determine the proposal distribution of F-HMC.
One may consider proposal distribution as: [Equation] In other words, we would utilise the dominating component of the mixture as a proposed distribution in each area S ( j ) n . While such a concept may have acceptable local properties, it may not ensure optimal ﬂow across various regions.
Therefore, we are using a second or (fold) HMC on top of these samplers concerning the cross-correlation of partitions to allow a ﬂow between different regions. Thus the F-HMC proposal distribution models are as follow: [Equation] where ˆ σ is calculated with respect to cross-correlation between all components.
Figure 1 demonstrates the functionality of the F-HMC; since the ﬁrst part is running on parallel and the second fold is running on a reduced number of dimensions, the overall execution time, especially on high dimensional setup, improves.
Furthermore, because two HMC samplers thoroughly examine the data, the system’s entropy will be satisfactory.
Algorithm 1 shows one iteration of the Bayesian learning for the generator parameters using our proposed model.
Here θ is the friction term for HMC, and η is the learning rate.
K shows the number of iterations in Bayesian GAN, and I shows the number of F-HMC iterations, and S indicates the number of components in the F-HMC.
4. T HEORETICAL ANALYSIS
The purpose of this section is to demonstrate that the probability density of data created using FHMC is similar to the target distribution of SGHMC.
As a result, we can conﬁrm that F-HMC samples are mathematically valid (Contribution 2).
Therefore, we can replace the F-HMC as a valid sampler with SGHMC in the Bayesian GAN design.
In the Experiment section, we illustrated how this replacement improved the outcomes.
Considering earlier setup outlined in Background section 3, V in Equation 7 , converges to a stationary distribution [Chen et al.
given by the following formula:
[Equation]
By taking the F-HMC methodology into account, the V in Equation 7 can be decomposed as the following matrix:
[Equation]
Theorem: π ( X, V ) ∝ exp ( − H
( X, V )) is the unique stationary distribution of the dynamics described in Equation 7 .
Proof : let W = (cid:20)− I
I (cid:21) and R = (cid:20) {+
+}
B (cid:21) , Equation 7 along with the proposed decomposition in
Equation 13 can be written in the following form:
[Equation]
We employ the Fokker-Planck Equation (FPE) (Risken (1992)) to describe the temporal evolution of the probability density function in relation with a Stochastic Differential Equation (SDE) that deﬁnes the development of the distribution on the random variable under speciﬁed stochastic dynamics.
Under Hamiltonian dynamics, the random variables in our situation are position variable X and momentum variable
V. Consider the following SDE:
[Equation]
15) ρ t ( ω ) is the distribution of ω governed by Eq. 15.
we also Consider J i ( . )
as ∂ i ( . )
by using FPE the equation evolves under the following formula:
[Equation]
We can rewrite Equation 16 in the following compact form:
[Equation]
) where ∇ T [ g ( ω ) ρ [Equation]
( ω
[Equation]
[Equation]
and ∇
[Equation]
[Equation]
From Equation 17 and considering the variable ω =
( X, V ) and g ( ω )
= − [ R + W ] ∇ H ( X, V ) and
R ( ω ) =
R ( X, V ) =
R = (cid:20)[-
-]
[Equation] (cid:21) , the distribution evolution under dynamic system in Equationcan be written as follow:
[Equation]
Since π ( X, V ) ∝ exp ( − H ( X, V )) , we can verify that the π ( X, V ) is invariant under Equation 18 by calculating [ e − H ( X,V ) ∇ H ( X, V ) + ∇ e − H ( X,V ) ]
= 0 .
Therefore, we can conﬁrm that the dynamics given in Equation 13 have similar invariance properties to that of the original Hamiltonian dynamics of Equation 7.
E XPERIMENTS
We implemented the proposed model using Pymc3 1 and report its performance on generating samples from complex distributions in section 5.1.
We have examined the model’s performance in marginalising the generators’ parameters on synthetic and natural image datasets such as SVHN [Netzer et al.
(2011)], CIFAR 10 [Krizhevsky (2009)] and ImageNet [Deng et al.
(2009)] in section 5.2 and 5.3, respectively.
We have compared our results with WDCGAN [Arjovsky et al.
DCGAN, 10DCGAN (which is a fully supervised convolutional neural network composed of ten
DCGANs constructed by ten random subsets with 80% of the size of the training set, [Radford et al.
(2016)]), Bayesian GAN [Saatci & Wilson (2017)] and ProbGAN [He et al.
(2019)] on supervised and semi-supervised tasks with four different numbers of labelled examples.
For a fair comparison, each model has the same number of generators and discriminator with the same architecture.
5. F-HMC PERFORMANCE
To evaluate the performance of F-HMC and compare it against SGHMC, we have designed a set of experiments.
First, we use Normalising Flows [Kobyzev et al.
Rezende & Mohamed (2016)] as a rich family of distributions to examine F-HMC and SGHMC’s abilities to explore complex distribution.
Figure 2 shows the potential energy of the rich target distribution and the generated candidates using SGHMC and F-HMC.
F-HMC successfully covers the target distribution (Advantage 1).
Second, we have measured auto-correlation between the samples generated in each sampler as a metric to show the power of the sampler in exploring the target distribution.
The more precise the sampler, the faster the auto-correlation reaches zero.
Figure 3 shows auto-correlation in
F-HMC drops faster to zero than SGHMC in terms of lag number (Advantage 2).
H IGH -D IMENSIONAL M ULTI M ODAL S YNTHETIC D ATASET
We present experiments on a multi-modal synthetic dataset to test inferring a multi-modal posterior p (ˆ α g | D ) .
This experiment shows F-HMC’s ability to explore a set of generators’ parameters with proper entropy and different complementary properties to encapsulate a rich data distribution.
We ﬁt a regular GAN, Bayesian GAN, and our proposed model to a dataset with D = 100 and 500.
The generator for all models is a two-layer neural network: 10-1000-100, fully connected, with ReLU activation.
The red samples in Figure 4 depict the target data, whereas the green samples depict the corresponding generated data.
The experiments on D = 100 are shown in the ﬁrst two rows, while the results on D = 500 are shown on two lower rows.
The name of generating sampler is displayed on the right side of each row.
We can ensure that both strategies (BGAN and F-HMC GAN) cover the intended distribution simply by comparing them visually.
Even the visual comparison is insufﬁcient to detect more outstanding performance in D = 100.
Still, it is evident in D = 500 that F-HMC provides a more desirable match to the target distribution as the dimension increases (Advantageand Contribution 3).
Figure 5 shows the comparison of the performance of GAN, F-HMC GAN, and Bayesian GAN in terms of Jensen-Shannon divergence.
The experiment estimates the similarity of the probability distribution of generated data to the original data and the level of entropy and conﬁrms that the F-HMC exceeds other models (Advantage 3 and Contribution 3).
5. N ATURAL I MAGE D ATASET
We used a 5-layer network architecture for GAN’s generator in all experiments on the natural images’ datasets.
The corresponding discriminator for supervised GAN is a 5-layer 2-class DCGAN, and we have used a 5-layer, K + 1 class DCGAN for a semi-supervised GAN performing classiﬁca- tion over K classes (see Saatci & Wilson (2017) for further details about Bayesian GAN structure)).
To evaluate the performance of our proposed model, we have employed experiments in three measurement levels: 1-performance metric in supervised and semi-supervised learning using test error rate.
2-run time per epoch in minutes by running all the models on a single GPU.
3-quality of generated images in terms of IS and FID scores.
FID [Heusel et al.
(2017)] is a measure that calculates the distance between vectors derived for real and synthetic images.
IS [Salimans et al.
(2016)] is an objective measure for assessing the quality and diversity of generated images. [Table]
[Table]
Table 1 demonstrates supervised and semi-supervised learning results for all image benchmarks.
Our proposed model mainly outperforms BayesGAN, probGAN, W-DCGAN, and 10-DCGAN in terms of test error rate (Contribution 4).
F-HMC shows its substantial impact when running on higher-dimensional data (ImageNet) due to the parallel composition of F-HMC; it can efﬁciently explore higher dimension data.
Table 2 shows the generated images’ quality and the run time of the models.
The quality of images increases by using F-HMC.
It is observed that the run time enhances when running the model on a higher dimension.
We perceive that in lower-dimensional data (CIFAR, SVHN), the Bayesian GAN and probGAN’s run time is more satisfying.
Once we run the models on higher-dimensional data (ImageNet), the run time improves in F-HMC.
The parallel composition of F-HMC makes the over epoch run time less than exploring the whole dimensions at once in Bayesian GAN and probGAN (Advantage 4).
C ONCLUSION
Folded Hamiltonian Monte Carlo is presented in this paper as a scalable strategy in sampling high-dimensional, highly correlated data to improve Bayesian GAN in producing synthetic images/generating data by marginalising the weights of the generators and discriminators.
We demonstrated that F-HMC converges faster and adapts to higher-dimensional inputs with more signiﬁcant similarity to the target data.
The theoretical and mathematical analysis of F-HMC is presented which conﬁrms its functionality.
F-HMC outperforms the state-of-the-art in terms of test error rates, runtimes per epoch, IS, and FID scores when evaluated on synthetic high-dimensional multi-modal data and natural image benchmarks, such as CIFAR-10, SVHN, and ImageNet.
Despite the notable improvement that F-HMC can bring in sampling target density, its hyperparameters affect its enforcement.
Nested sampling methods [Betancourt et al.
(2011)] sample from the likelihood space instead of sample space and are more likely to land on a better set of parameters.
Future directions include the adoption of a nested sampling method, e.g. restricted Hamiltonian Monte Carlo methods, to help the improvement of the outcomes.
R EPRODUCIBILITY
The experiments presented in the paper are designed to be reproducible and easy to extend.
The notebook provides instructions on how to install the packages as well as the code required to run the experiments and the link to download the necessary datasets.
A A PPENDIX