C ONJUGATE E NERGY -B ASED M ODELS
A BSTRACT
In this paper, we propose conjugate energy-based models (CEBMs), a new class of energy-based models that deﬁne a joint density over data and latent variables.
The joint density of a CEBM decomposes into an intractable distribution over data and a tractable posterior over latent variables.
CEBMs have similar use cases as variational autoencoders, in the sense that they learn an unsupervised mapping from data to latent variables.
However, these models omit a generator network, which allows them to learn more ﬂexible notions of similarity between data points.
Our experiments demonstrate that conjugate EBMs achieve competitive results in terms of image modelling, predictive power of latent space, and out-of-domain detection on a variety of datasets.
I NTRODUCTION
Deep generative models approximate a data distribution by combining a prior over latent variables with a neural generator, which maps latent variables to points on a data manifold.
It is common to evaluate these models in terms of their ability to generate realistic examples, or their estimated densities for unseen data.
However, an arguably more practical use case for these models is unsupervised representation learning.
If a generator can faithfully represent the data in terms of a lower-dimensional set of latent variables, then we hope that these variables will encode a set of meaningful factors of variation that will be relevant to a broad range of downstream tasks.
Guiding a model towards a semantically meaningful representation requires some form of inductive bias.
A large body of work on variational autoencoders (VAEs, Kingma & Welling (2013); Rezende et al.
(2014)) has explored the use of priors as inductive biases.
Relatively mild biases in the form of conditional independence are common in the literature on disentangled representations (Higgins et al., 2016;
Kim & Mnih, 2018; Chen et al., 2018; Esmaeili et al., 2019).
More generally, recent work has shown that deﬁning priors that reﬂect the structure of the underlying data will lead to representations that are easier to interpret and generalize better.
Examples include priors that represent objects in an image (Eslami et al., 2016; Lin et al., 2020b; Engelcke et al., 2019; Crawford & Pineau, 2019b), or moving objects in video (Crawford & Pineau, 2019a; Kosiorek et al., 2018; Wu et al., 2020; Lin et al., 2020a).
Despite steady progress, work on disentangled representations and structured VAEs still predominantly considers synthetic data sets.
The likelihood model in VAEs is typically a neural generator that is optimized to accurately reconstruct all the examples in the training set.
For complex natural scenes, learning a model that can produce pixel-perfect reconstructions poses fundamental challenges, given the combinatorial explosion of possible inputs.
This is not only a problem for gen- ∗ Equal contribution.
[Figure]
eration, but also from the perspective of the learned representation; a VAE must encode all factors of variation that give rise to large deviations in pixel space, regardless of whether these factors are semantically meaningful (e.g. presence and locations of objects) or not (e.g. shadows of objects in the background of the image).
In this paper, we consider energy-based models (EBMs) as an alternative to VAEs for learning representations in an unsupervised manner.
The general idea of using EBMs for this purpose is by no means new; it has a long history in the context of restricted Boltzmann machines (RBMs) and related models (Smolensky, 1986; Hinton, 2002; Welling et al., 2004).
Our motivation in the present work is to design a class of EBMs that retain the desirable features of VAEs, but employ a discriminative energy function to model data at an intermediate level of representation that does not necessarily encode all features of an image at the pixel level.
Concretely, we propose conjugate EBMs (CEBMs), a new family of energy-based models in which the energy function deﬁnes a neural exponential family.
While the normalizer of CEBMs is intractable, we can nonetheless compute the posterior in closed form when we pair the likelihood with an appropriate conjugate bias term.
Thus, the neural sufﬁcient statistics in a CEBM fully determine the marginal likelihood and the encoder, thereby side-stepping the need for a generator (Figure 1).
We evaluate CEBMs in experiments that test how well learned representations agree with class labels (which are not used during training).
We show that CEBMs learn a latent space in which neighbors are more likely to belong to the same class, which translates to increased performance in few-label downstream classiﬁcation tasks.
B ACKGROUND 2. C ONJUGATE E XPONENTIAL F AMILIES
An exponential family is a set of distributions whose probability density can be expressed as
[Equation]
where h : X → R + is a base measure, η ∈ H ⊆
R K is a vector of natural parameters, t : X → R K is a vector of sufﬁcient statistics, and A : H → R is the log normalizer (or cumulant function),
[Equation]
If a likelihood belongs to an exponential family, then there exists a conjugate prior that is itself an exponential family
[Equation]
The convenient property of conjugate exponential families is that both the marginal p ( x | λ, ν ) and the posterior p ( η | x, λ, ν ) are tractable.
If we deﬁne ˜ λ ( x ) = λ + t ( x ) , ˜ ν
= ν
+ 1 , then the posterior and marginal are
[Equation] [Equation]
2. L EGENDRE D UALITY IN E XPONENTIAL F AMILIES
Two convex functions A : H → R + and A ∗ : M → R + on spaces H ⊆
R K and M ⊆
R K are conjugate duals when
[Equation]
When A is a function of Legendre type (see Rockafellar (1970)), the gradients of these functions deﬁne a bijection between conjugate spaces by mapping points to their corresponding suprema
[Equation]
such that we can express A ∗ ( µ ) at the supremum as
[Equation]
The log normalizer A ( η ) of an exponential family is of Legendre type when the family is regular and minimal ( H is an open set and sufﬁcient statistics t ( x ) are linearly independent; see Wainwright & Jordan (2008) for details).
We refer to M as the mean parameter space, since we can express any µ ∈ M as the expected value of the sufﬁcient statistics µ ( η ) =
E p ( x | η ) [ t ( x )] .
2. B REGMAN D IVERGENCES AND E XPONENTIAL F AMILIES
A Bregman divergence for a function F : M
→ R that is continuously-differentiable and strictly convex on a closed set M has the form
[Equation]
Well-known special cases of Bregman divergences include the squared distance ( F ( µ ) =
(cid:104) µ, µ (cid:105) ) and the Kullback-Leiber (KL) divergence ( F ( µ ) =
(cid:80) k
µ k log µ k ).
Any Bregman divergence can be associated with an exponential family and vice versa, where
F ( µ ) =
A ∗ ( µ ) is the conjugate dual of A ( η ) (see Banerjee et al.
To see this, we re-express the log density of a (regular and minimal) exponential family using the substitution µ = ∇ A ( η ) 1 ,
[Equation]
In other words, the log density of an exponential family can be expressed in terms of a bias term
A ∗ ( t ( x )) , and a notion of agreement in the form of a Bregman divergence D A ∗ ( t ( x ) , µ ) between the sufﬁcient statistics t ( x ) and the mean parameters µ .
We will make use of this property to provide an interpretation of both CEBMs and VAEs in terms of Bregman divergences.
C ONJUGATE E NERGY -B ASED M ODELS
In this paper we are interested in learning a probabilistic model that deﬁnes a joint distribution p θ,λ ( x, z ) over high-dimensional data x ∈ R D and a lower-dimensional set of latent variables z ∈
R K .
The intuition that guides our work is that we would like to measure agreement between latent variables and data at a high level of representation, rather than at the level of individual pixels, where it may be more difﬁcult to distinguish informative features from noise.
To this end, we will explore energy-based models as an alternative to VAEs.
Concretely
[Equation]
we propose to consider models of the form
[Equation]
In this energy function, θ are the weights of a network t θ : R D
→ R H , which plays the role of an encoder by mapping high-dimensional data to a lower-dimensional vector of neural sufﬁcient statistics.
The function η : R K → R H maps latent variables to a vector of natural parameters in the same space as the neural sufﬁcient statistics.
The function E λ : R K → R serves as an inductive bias, with hyperparameters λ , that plays a role analogous to the prior.
We will consider a bias E λ ( z ) in form of a tractable exponential family with sufﬁcient statistics η ( z )
[Equation]
We can then express the energy function as
[Equation]
This form of the energy function has a convenient property: It corresponds to a model p
[Equation]
λ
[Equation]
x
[Equation]
z ) in which the posterior p θ,λ ( z | x ) is tractable.
To see this, we make a substitution ˜
λ θ ( x ) = λ + t θ ( x ) analogous to the one in Equation 4, which allows us to express the energy as
E
[Equation]
We see that we can factorize the corresponding density p θ,λ ( x, z ) =
p θ,λ ( x ) p θ,λ ( z | x ) , which yields a posterior and marginal that are analogous the distributions in Equation
[Equation]
In other words, the joint density of this model factorizes into a tractable posterior p θ,λ ( z | x ) and an intractable energy-based marginal likelihood p θ,λ ( x | λ ) .
This posterior is conjugate, in the sense that it is in the same exponential family as the bias.
For this reason, we refer to this class of models as conjugate energy-based models (CEBMs).
R ELATIONSHIP TO VAE S
CEBMs differ from VAEs in that they lack a generator network.
Instead, the density is fully speciﬁed by the encoder network t θ ( x ) , which deﬁnes a notion of agreement (cid:104) ˜
λ θ ( x ) , η ( z ) (cid:105) between data and latent variables in the latent space.
As with other exponential families, we can make this notion of agreement explicit by expressing the conjugate posterior in terms of a Bregman divergence using the decomposition in Equation
[Equation]
Here B ∗ ( µ ) is the conjugate dual of the the log normalizer B ( λ ) , and we use ˜ µ θ ( x ) = µ
[Equation]
˜
λ θ ( x )) as a shorthand for the mean-space posterior parameters.
We see that maximizing the density corresponds to minimizing a Bregman divergence in the space of sufﬁcient statistics of the bias.
In Figure 1, we compare CEBMs to VAE in terms of the energy function for the log density of the generative model.
In making this comparison, we have to keep in mind that these models are trained using different methods, and that VAEs have a tractable density p θ ( x, z ) .
That said, the objectives in both models maximize the marginal likelihood, so we believe that it is instructive to write down the corresponding Bregman divergence in the VAE likelihood.
This likelihood is typically a Gaussian with known variance, or a Bernoulli distribution (when modeling binarized images).
Both distributions have sufﬁcient statistics t ( x ) = x .
Once again omitting the base measure h ( x ) for expediency, we can express the log density of a VAE as an energy
E
[Equation] [Equation]
Here A ∗ ( x ) is the conjugate dual of the log normalizer A ( η ) , and we use η θ ( z ) and µ θ ( z ) to refer to the output of the generator network in the natural-parameter and the mean-parameter space respectively.
To reduce clutter and accommodate the case where a base measure h ( x ) is needed (e.g. that of a Gaussian likelihood with known variance), we will introduce the additional shorthands
[Equation]
Then the energy function of a VAE has the form E θ,λ ( x, z ) =
D A ∗ ( x, µ θ ( z ))
+ E ( x ) + E λ ( z ) .
Like that of a CEBM, the energy function of a VAE contains a Bregman divergence, as well as two terms that depend only on x and z .
However, whereas the Bregman divergence in CEBM is deﬁned in the mean-parameter space of the latent variables, that of a VAE is computed in the data space.
I NDUCTIVE B IASES
CEBMs have a property that is somewhat counter-intuitive.
While the posterior p θ,λ ( z | x ) in this class of models is tractable, the prior is in general not tractable.
In particular, although the bias − E λ ( z ) is the logarithm of a tractable exponential family, it is not the case that p θ,λ ( z ) =
exp {− E λ ( z ) } .
Rather the prior p θ,λ ( z ) has the form,
[Equation]
In other words, E λ ( z ) deﬁnes an inductive bias, but this bias is different from the bias in a VAE (where the prior is always tractable by construction 2 ), in the sense that it imposes only a soft constraint on the geometry of the latent space.
In principle the bias in a CEBM can take the form of any exponential family distribution.
Since products of exponential families are also in the exponential family, this covers a broad range of possible biases.
For purposes of evaluation in this paper, we will constrain ourselves to two cases: a spherical Gaussian and a Mixture of Gaussians.
We provide comprehensive derivation in Appendix C.
R ELATED W ORK
Energy-Based Latent Variable Models.
The idea of using EBMs to jointly model data and latent variables has a long history in the machine learning literature.
Examples of this class of models include restricted Boltzmann machines (Smolensky, 1986; Hinton, 2002), deep belief nets (DBNs, Hinton et al.
(2006)), and deep Boltzmann machines (Salakhutdinov & Hinton, 2009).
The idea of extending RBMs in exponential families and exploiting conjugacy to yield a tractable posterior is also not new and has been explored in Exponential Family Harmoniums (Welling et al., 2004).
The models differ from CEBMs in that the they employ a bilinear interaction term x (cid:62) Wz , which ensures that both the likelihood p ( x | z ) and p ( z | x ) are tractable.
In CEBMs, the corresponding term t θ ( x ) (cid:62) z is nonlinear, which means that the posterior is tractable, but the likelihood is not.
We provide a detailed discussion regarding the connection to this class of models in Appendix A.
EBMs for Image Modelling.
Recent work has shown that EBMs with convolutional energy functions can model distributions over images, given their competitive results in image generation (Xie et al., 2016; Nijkamp et al., 2019a;b; Du & Mordatch, 2019).
This line of work typically focuses on generation and not on unsupervised representation learning as we do here.
A line of work, which is similar to ours in spirit, employs EBMs as priors on the latent space of deep generative models (Pang et al., 2020; Aneja et al., 2020).
These approaches, unlike our work, require a generator.
Grathwohl et al.
; Liu & Abbeel (2020) have proposed to interpret a classiﬁer as an EBM that deﬁnes a joint energy function on the data and labels.
CEBMs with a discrete bias can interpreted as the unsupervised variant of this model class.
Che et al.
(2020) interpret a GAN as an EBM deﬁned by both the generator and discriminator.
Training EBMs.
While we here employ PCD for training, there is a large literature on alternative methods for training unnormalized models.
An alternative class of methods is Noise Contrastive
Estimation (NCE, (Gutmann & Hyv¨arinen, 2010)) in which we deﬁne a noise model and learn by discriminating between data and noise.
An example of this is Gao et al.
(2020), where the authors pretrain a ﬂow that acts as the noise model for training EBMs with NCE.
Another popular approach is score matching (SM, Hyv¨arinen & Dayan (2005); Vincent (2011); Song et al.
Bao et al.
(2020)), which matches the gradients of log probability density of the model and data distribution.
Lastly, there are adversarial methods for avoiding MCMC sampling during MLE, where we introduce a variational distribution q φ ( x ) that is trained alongside the model with a maximin objective (Grathwohl et al., 2021).
We refer the readers to Song & Kingma (2021) for a more comprehensive study of training methods for EBMs.
Note that because CEBMs can marginalize over z to get the marginal, all of the discussed methods are available for training, and therefore could be applied in the context of this work.
[Figure]
E XPERIMENTS
Our experiments evaluate to what extent CEBMs can learn representations that encode meaningful factors of variation, whilst discarding details about the input that we would consider noise.
We begin with a qualitative evaluation by visualizing samples and then demonstrate that learned representations align with class structure, in the sense that nearest neighbors in the latent space are more likely to belong to the same class (Section 7.1).
Next, we pre-train the models in an unsupervised manner, and use the learned representations for a few-label classiﬁcation task (Section 7.2) and an out-of-distribution task (Appendix F).
We train CEBMs using presistent contarstive divergence (Tieleman, 2008; Du & Mordatch, 2019) where we sample from the models using Stochastic Gradient Langevin Dynamics (SGLD, (Welling & Teh, 2011)).
We train our models using 60 SGLD steps, 90k gradient steps, batch size 128, Adam optimizer with learning rate 1e-4.
We refer the readers to Appendix E & I for more training details.
7. S AMPLES AND L ATENT S PACE
We begin with a qualitative evaluation by visualizing samples from the model (see Figure 2).
While generation is not our intended use case in this paper, such samples do serve as a useful diagnostic, in the sense that they allow us to visually inspect what characteristics of the data are captured by the learned representation.
We initialize the samples with uniform noise and run 500 SGLD steps.
We observe that the distribution over images is diverse and captures the main characteristics of the dataset.
Sample quality is roughly on par with samples from other EBMs Nijkamp et al.
(2019a), although it is possible to generate samples with higher visual quality using class-conditional EBMs Du & Mordatch (2019)
; Grathwohl et al.
; Liu & Abbeel (2020) (which assume access to labels).
To assess to what extent the representation in CEBMs aligns with classes in each dataset, we look at the agreement between the label for each data point and the label of its nearest neighbor in the latent space.
In Figure 3, we show samples from CIFAR-10, along with the images that correspond to the nearest neighbors in pixel space, the latent space of a VAE, and the latent space of a CEBM.
The distance in pixel space is a poor measure of similarity in this dataset, whereas proximity in the latent space is more likely to agree with class labels in both VAEs and CEBMs. [Table]
In Figure 3 (right), we quantify this agreement by computing the fraction of neighbors in each class conditioned on the class of the original image.
We see a stronger alignment between classes and the latent representation in CEBMs, which is reﬂected in higher numbers on the diagonal of the matrix.
On average, a fraction of 0.38 of the nearest neighbors are in the same class in the VAE, whereas 0.45 of the neighbors are in the same class in the CEBM.
This suggests that the representation in
CEBMs should lead to higher performance in downstream tasks.
7. F EW - LABEL C LASSIFICATION
To evaluate performance in settings where few labels are available, we train logistic classiﬁers using the learned latent features (trained in an unsupervised manner) with 1 , 10 , 100 training examples per class, as well as the full training set.
We evaluate our model for both of the inductive biases discussed in Section 5 and Appendix C: the spherical Gaussian (CEBM) and the mixture of Gaussian (GMMCEBM).
We compare our models against the IGEBM Du & Mordatch (2019), a standard VAE with the spherical Gaussian prior, GMM-VAE (Tomczak & Welling, 2018) where the prior is a mixture of Gaussians (GMM), and BIGAN Donahue et al. (2016).
IGEBM does not explicitly have latent representations.
To resolve this, we extract the features from the last layer of the encoder for IGEBM.
We report the classiﬁcation accuracy on the test set in Table 1.
Overall CEBMs achieve a higher accuracy compared to VAEs in particular for CIFAR-10 and SVHN where the pixel distance is not good measure for similarity.
Moreover, we observe that CEBMs outperform IGEBMs, which suggests that the inductive bias in a CEBM can increase performance for downstream tasks.
The prformance of BIGANs is competitive, and we suspect that the reason is that, silimiar to CEBMs,
BIGANs also do not deﬁne a likelihood that measure similarity at the pixel level.
We also observe that the CEBM with the GMM inductive bias does not always ourtperform the one with standard
Gaussainn inductive bias, wich we supect is due to GMM-VAE being more diffuclt to converge.
C ONCLUSION
We introduced CEBMs; a new family of energy-based models that deﬁne a joint energy function over both the data and latent variables.
This joint distribution factorizes into an intractable energy-based marginal likelihood, which can be trained using standard methods for EBMS, and a tractable posterior, which serves to maps input data to a low-dimensional latent representation.
This factorization allows us to directly optimize the marginal likelihood of the data, while at the same time imposing an inductive bias on the latent space.
Experimental results for this class of models are encouraging; we observe a closer agreement between unsupervised representations and class labels, which translates into improvements in downstream classiﬁcation tasks relative to VAE-based baselines. A CKNOWLEDGMENTS
We would like to thank our reviewers for their thoughtful comments.
This work was supported by the Intel Corporation, the 3M Corporation, NSF award 1835309, startup funds from Northeastern
University, the Air Force Research Laboratory (AFRL), and DARPA.
A C ONNECTION TO E XPONENTIAL F AMILY H ARMONIUMS
As mentioned in Section 6, there is a long history of incorporating latent variables in EBMs, particularly in the context of restricted Boltzmann machines (RBMs) (Smolensky, 1986; Hinton, 2002), deep belief nets (Hinton et al., 2006), and deep Boltzmann machines (Salakhutdinov & Hinton, 2009).
Moreover, the idea of formulating EBMs into the exponential family is also not new; Welling et al.
(2004) proposed a new class of models called Exponential Family Harmoniums (EFHs) by extending RBMs into the exponential family.
In this Section, we discuss the connection between our approach to these models.
Concretely, we show that EFHs can be recovered a special case of
CEBMs.
For observed variable x and latent variable z , the energy of an RBM is deﬁned as
[Equation]
where θ x ∈ R D , θ z ∈ R K , and θ xz ∈ R D × K .
In RBMs, the conditional distributions p θ ( x | z ) and p θ ( z | x ) are both tractable which means that during contrastive divergence, we can sample x ∼ p θ ( x ) using Gibbs sampling.
EFHs extend these models into the exponential family by incorporating the sufﬁcient statistics of x and z in the energy,
[Equation]
where t x ( · ) and t z ( · ) are the sufﬁcient statistics for variables x and z respectively.
Welling et al.
(2004) show that this energy function yields the following conditional distributions:
Likelihood
[Equation]
Posterior
[Equation]
where ˜ θ x and ˜ θ z are the canonical parameters, and A ( · ) and B ( · ) are the log normalizer of the models p θ ( x | z ) and
p θ ( z | x ) respectively.
Given that both conditional distributions are tractable,
EFHs have the same advantage as RBMs: We can use a Gibbs sampler for sampling x ∼ p θ ( x ) .
CEBMs can be considered an extension of EFHs.
In Equation 10, we recover the energy function for an EFH by setting
[Equation]
Perhaps the most crucial difference between CEBMs and EFHs (and other RBM-based models) is the non-linearity relationship between the observed and latent variables.
The non-linearity in t θ ( · ) has the beneﬁt of providing the ﬂexibility to learn more complex structures in the data.
This modelling choice however comes with a cost.
In CEBMs, while the posterior is still tractable, the likelihood model is not.
As a consequence, we lose the ability to use Gibbs sampling to sample x ∼ p θ ( x ) .
However, given that our motivation here is not to generate high quality samples at test time but to learn good representations representations, we believe giving up the ability to easily sample x in order to learn more complex structures while keeping the posterior tractable is an appropriate trade-off
[Equation]
B D ERIVATION OF P RIOR AND L IKELIHOOD IN A CEBM B. P RIOR
[Equation] [Equation]
[Table]
DNN outputs a scalar value f θ ( x ) : R D → R .
In EBMs deﬁned on the data space as well as labels (type E θ ( x, y ) ) such as JEM, the DNN outputs a vector of length L corresponding to the number of classes f θ ( x ) : R D → R L .
In GAN, D θ ( x ) refers to the discriminator.
B. L IKELIHOOD
[Equation]
C T WO S PECIAL C ASES FOR I NDUCTIVE B IASES
In this work we will consider two special cases for inductive bias:
Spherical Gaussian.
As a bias that is analogous to the standard prior in VAEs, we consider a spherical Gaussian with ﬁxed hyperparameters ( µ, σ ) = (0 , 1) for each dimension of z ∈ R K ,
[Equation]
Each term has
sufﬁcient
statistics η ( z k ) =
( z k , z 2 k ) and natural parameters
[Equation]
The marginal likelihood of the CEBM is then
[Equation]
where ˜ λ θ,k
( x ) = λ + t θ,k
( x ) and the log normalizer is
[Equation]
Mixture of Gaussians.
In our experiments, we will consider datasets that are normally used for classiﬁcation.
These datasets, by design, exhibit multimodal structure that we would like to see reﬂected in the learned representation.
In order to design a model that is amenable to uncovering this structure, we will extend the energy function in Equation 10 to contain a mixture component y
[Equation]
As an inductive bias, we will consider a bias in the form of a mixture of L Gaussians,
[Equation]
Here z ∈ R K is a vector of features and y ∈ { 1 , . . .
, L } is a categorical assignment variable.
The bias for each component l is a spherical Gaussian with hyperparameters λ l,k for each dimension k .
Again, using the notation ˜ λ θ,l,k = λ l,k + t θ,l,k ( x ) to refer to the posterior parameters, then we obtain an energy
[Equation]
We then deﬁne a joint probability over data x and the assignment y in terms the log normalizer B ( · ) ,
[Equation]
which then allows us to compute the marginal p θ,λ by summing over y .
We optimize this marginal with respect hyperaparameters λ l,k as well as the weights θ .
D B ACKGROUND ON E NERGY -B ASED M ODELS
An EBM (LeCun et al., 2006) deﬁnes a probability density for x ∈ R D via the Gibbs-Boltzmann distribution
[Equation]
The function E θ : R D −→ R is called the energy function which maps each conﬁguration to a scalar value, the energy of the conﬁguration.
This type of model is widely used in statistical physics, for example in Ising models.
The distribution can only be evaluated up to an unknown constant of proportionality, since computing the normalizing constant Z θ (also known as the partition function) requires an intractable integral with respect to all possible inputs x .
Our goal is to learn a model p θ ( x ) that is close to the true data distribution p data ( x ) .
A common approach for achieving this objective is to minimize the Kullback-Leibler divergence between the data distribution and the model, which is equivalent to maximizing the expected log-likelihood
[Equation]
The key difﬁculty when performing maximum likelihood estimation is that computing the gradient of log
Z θ is intractable.
A common strategy is to express this gradient as an expectation with respect to p θ ( x ) ,
[Equation]
(32) which means that the gradient of L ( θ ) has the form:
[Equation]
This corresponds to maximizing the probability of samples x ∼ p data ( x ) from the data distribution and minimizing the probability of samples x (cid:48) ∼ p θ ( x (cid:48) ) from the learned model.
Contrastive divergence methods Hinton (2002) compute a Monte Carlo estimate of this gradient.
Estimating this gradient however requires samples from the model x (cid:48) ∼ p θ ( x (cid:48) ) , whereas this density is intractable.
A common method for generating samples from EBMs is Stochastic Gradient Langevin
Dynamics (SGLD, Welling & Teh (2011)), which initializes a sample x (cid:48) 0 ∼ p 0 ( x
[Equation]
) and performs a sequence of gradient updates with additional injected noise (cid:15) ,
[Equation]
SGLD is motivated as a discretization of a stochastic differential equation whose stationary distribution is equal to the posterior.
It is correct in the limit i → ∞ and α → 0 , but in practice will have a bias.
The initialization x (cid:48) 0 is crucial because it determines the number of steps needed to converge to a high-quality sample.
For this reason, EBMs are commonly trained Nijkamp et al. (2019a);
Du &
Mordatch (2019); Grathwohl et al.
(2019) using persistent contrastive divergence (PCD, Tieleman (2008)), which initializes some samples from a replay buffer B of previously generated samples.
E T RAINING CEBM S
Hyperparameters.
In CEBMs and VAEs, we choose the dimension of latent variables to be 128.
For CEBMS, We found that the optimization becomes difﬁcult with smaller dimensions.
We train our models using 60 SGLD steps where we initialize samples from the replay buffer with 0.
probability, and initialize from uniform noise with 0.05 probability.
We train all the models with 90k gradient steps, batch size 128, Adam optimizer with learning rate 1e-4.
When doing PCD, we used a reply buffer of size 5000.
We set the α in the SGLD teps to be 0.075.
Similar to Du & Mordatch (2019), we found it useful to add some noise to the image before encoding.
In our experiments, we used Gaussian noise with σ 2 = 0 .
We used 50 GMM components for GMM-VAE and 10 GMM components for GMM-
CEBM
Training Stability.
As observed in previous work Du & Mordatch (2019)
; Grathwohl et al.
(2019), training EBMs is challenging and often requires a thorough hyperparameters search.
We found that the choices of activation function, learning rate, number of SGLD steps, and L2 regularization (proposed by Du & Mordatch (2019)) will all affect training stability.
Models regularly diverge during training, and it is difﬁcult to perform diagnostics given that log p θ,λ ( x ) cannot be computed.
As suggested by Nijkamp et al. (2019a), we found checking the difference in energy between data and model samples can help to verify training stability.
In general we also observed a trade-off between sample quality and the predictive power of latent variables in our experiments.
We leave investigation of the source of this trade-off to future work, but we suspect that this is because SGLD has more difﬁculty converging when the latent space is more disjoint.
Nijkamp et al.
(2019a;b) performed a comprehensive analysis of convergence of PCD in recent
EBMs for images, where they study a variety of factors such as MCMC chain initialization, network, and optimizer.
They identify that a key factor for diagnosing these models is the difference between the energy of positive and negative samples.
Many of these ﬁndings were helpful during the training and evaluation of EBMs in our work.
F O UT - OF -D ISTRIBUTION D ETECTION
EBMs have formed the basis for encouraging results in out-of-distribution (OOD) detection Du &
Mordatch (2019); Grathwohl et al.
While not our focus in this paper, OOD detection is an additional benchmark that helps evaluate whether a learned model accurately characterizes the data distribution.
In Table 3, we report results in terms of two metrics.
The ﬁrst is the area under the receiver-operator curve (AUROC) when thresholding the log marginal log p θ,λ ( x ) .
The second is the gradient-based score function proposed by Grathwohl et al.
CEBMs results for OOD detection in most cases
G V ISULAIZATION OF L ATENT S PACE
[Table]
H C ONFUSION M ATRICES FOR 1-NN C LASSIFICATION
We perform 1-nearest-neighbor classiﬁcation task for MNIST, Fashion-MNIST, SVHN, CIFAR10.
We compute the L2 distance in the latent space of VAE, IGEBM and CEBM, and also in pixel space.
We visualize the confusion matrices
[Table]
CEBM
I M ODEL A RCHITECTURES
CEBMs employ an encoder network t θ ( x ) in the form of 4-layer CNN (which is proposed by Nijkamp et al. (2019a)), followed by an MLP output layer.
We choose the dimension of latent variables to be 128 to be in CEBMs.
We use the same architecture for the encoder of a VAE and the encoder of a BIGAN.
Table 4, Table 5, Table 6, and Table 7 show the architectures used for CEBM, IGEBM,
VAE and BIGAN, respectively.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
padding 1.
ReLU.
FC.
128 ReLU.
FC
[Table] [Table]
3 × 32 ReLU.
[Table]
4 conv.
[Table]
padding 1.
ReLU.
[Table]
4 upconv.
[Table] [Table]
[Table] [Table]
4 conv.
32 stride 2.
padding 1.
ReLU.
4 × 4 upconv.
[Table]
padding 1.
ReLU.
4 × 4 conv
[Table] [Table]
padding 1.
ReLU
[Table]
4 × 4 upconv
[Table] [Table]
padding 0.
ReLU
[Table]
128 ReLU.
FC
[Table] [Table]
[Table]
padding 1.
[Table] [Table]
FC
[Table] [Table]
padding 1.
ReLU
[Table]
4 upconv
[Table] [Table]
padding
[Table]
4 conv
[Table]
256 stride 2.
padding 1.
ReLU
[Table]
4 × 4 upconv
[Table] [Table]
padding 1.
ReLU
[Table]
× 4 conv
[Table] [Table]
padding 1.
ReLU
[Table]
3 × 3 upconv
[Table] [Table]
padding 1.
ReLU.
FC
[Table]
1024 ReLU.
FC
[Table]
2 ×
[Table] [Table]
padding
[Table] [Table]