Learning Probabilistic Models from Generator Latent Spaces with Hat EBM
Anonymous Author(s)
AffiliationAddress
Abstract
This work proposes a method for using any generator network as the foundationof an Energy-Based Model (EBM).
Our formulation posits that observed imagesare the sum of unobserved latent variables passed through the generator networkand a residual random variable that spans the gap between the generator output andthe image manifold.
One can then define a hat EBM that includes the generatoras part of its forward pass.
The model can be trained without inferring the latentvariables of the observed data or dealing with the generator Jacobian determinant.
This enables explicit probabilistic modeling of the output distribution of any typeof generator network.
Experiments show strong performance of the proposedmethod on (1) unconditional ImageNet synthesis at 128 × 128 resolution, (2) refining the output of existing generators, and (3) learning EBMs that incorporatenon-probabilistic generators.
Introduction
Generator networks [16, 8] which transform a latent distribution to a complex observed distribution(e.g., images, videos) are found across different deep generative models.
One limitation of generatornetworks is the difficulty of obtaining an explicit representation of the probability distribution definedby the output of the network after transformation of the latent space.
For generators from Generative
Adversarial Networks (GANs) [8, 26] and Variational Autoencoders (VAEs) [16, 27] where thelatent states corresponding to realistic images follow a trivial distribution (e.g., isotropic Gaussian),the difficulty of obtaining image space probabilities lies in calculating the log determinant of the
Jacobian of the generator transform for density change-of-variables.
For other generator models suchas a deterministic autoencoder, there might not be a way to generate probabilistic samples from thelatent space that correspond to realistic images.
When the latent space has smaller dimension thanthe image space, there is further difficulty in describing the generator image distribution since thedistribution measure must be confined to the manifold of the generator output.
This work proposes a method for using a generator network as the foundation for an Energy-Based
Model (EBM).
The generator network is concatenated with a hat network that takes an image inputand outputs a scalar.
Before the generator output is fed into the hat network, the generated image isadjusted by adding a residual image that spans the gap between the generator output and the imagemanifold.
The total function, including the generator, addition of the residual image, and the hatnetwork, is called the Hat EBM .
This allows us to define an EBM which can incorporate the generatorlatent space as part of its MCMC sampling process.
Figure 1 displays a diagram of the Hat EBMand selected Hat EBM samples for unconditional ImageNet 128 × 128 .
Our method is general andapplies to any generator model, in contrast to existing methods for converting generator networks to
EBMs which only apply to a specific generator model like a GAN [3] or VAE [30].
Additionally, our learning method does not require any inference of latent states of the observed data as used in relatedworks [16, 31, 11,
Our Hat EBM formulation enables applications with pretrained generator networks such as learning an
EBM to refine samples from generators with probabilistic latent spaces (e.g. GANs) and synthesizingsamples using non-probabilistic generators (e.g. deterministic autoencoders).
Further, we propose aself-contained learning method that extends cooperative learning of EBM and generator networks[31] to achieve high-quality synthesis.
Our main contributions are summarized below.
• We introduce a method for defining a Hat EBM that incorporates a generator network as part of itsforward pass.
This EBM includes the generator latent space as part of Langevin sampling.•
We show that our method can refine samples from pretrained GAN generators and sample from the
latent space of deterministic autoencoders which are originally incompatible with sampling.
• We propose a self-contained Hat EBM learning method that trains both a generator and energy
network from scratch.
This enables us to achieve an FID score of 29.2 on unconditional ImageNetat resolution 128 × 128, demonstrating the EBMs are competitive with state-of-the-art generative
models on complex and high resolution datasets.
Related Work
EBM.
An EBM defines an unnormalized density or a Gibbs distribution.
The prototypes includeexponential family distributions, Boltzmann machines [1, 28], and the FRAME model [36].
Recentwork has introduced the EBM with a ConvNet potential [35, 34].
This dramatically increasesthe model capacity and demonstrates strong image synthesis performance [23, 6] and adversarialrobustness [12].
Several works investigate training and EBM in tandem with an auxiliary model.
Thework [15] jointly trains an EBM and generator without MCMC by using samples from the generatoras direct approximations of the EBM density and training the generator using a variational objective.
A similar approach is explored by [10].
Cooperative learning [33] trains the EBM and generatorby using the generator to initialize samples needed to train the EBM and uses reconstruction lossbetween generator and EBM samples to learn the generator.
The work [7] learns an EBM using Noise
Contrastive Estimation with an auxiliary flow model.
Latent Space EBM.
EBMs in the data space can be highly multi-modal, and MCMC sampling canbe difficult [32, 23, 6].
Recent works [24, 25] explore learning an EBM in latent space, which is then mapped to the data space with a learned generator.
The energy landscape in the latent spaceis smoother and less multi-modal because it is in a much lower dimensional space and stands onan expressive generator.
These works define a prior EBM in the latent space as a correction ofthe non-informative uniform prior or isotropic Gaussian prior.
To learn the model, one needs toinfer the posterior of the latent variables.
Posterior inference given such a complicated model isnon-trivial.
One needs to either design a sophisticated amortized inference network or run expensive
MCMC.
Our model also defines an EBM in the latent space, while its learning does not need posteriorinference, making the learning much simpler and more scalable.
[3] leverages a pretrained GAN todefine an EBM in the latent space of the generator with a correction based on the discriminator, andshows improved synthesis quality over the pretrained GAN.
The work [30] uses a pretrained VAE tofacilitate EBM learning.
Our model is similar in that it can be utilized to improve pretrained GAN or VAE generators.
Our method is however more general since it can be used to improve any pretrainedgenerator, even the non-probabilistic ones like deterministic autoencoders.
Formulation of Hat EBM
This section presents the formulation of the Hat EBM energy function and the proposed learningprocedure.
We first review the fundamental equations of EBM learning.
Then we introduce twovariants of the Hat EBM: one for sampling from the latent space directly, and one for sampling aresidual image conditional on the generator output.
Finally, we propose a method for learning thehat network and generator network of a Hat EBM simultaneously so that our model can be used forself-contained image generation without the need for a pretrained generator.
3.1 Review of EBM Learning
We briefly review the main components of EBM learning following the standard method derivedfrom works such as [13, 36, 32].
A deep EBM has the form
[Equation]
(1) where U ( x ; θ ) is a deep neural network with weights θ and Z ( θ ) is the intractable normalizingconstant.
Given a true but unknown data density q ( x ) , Maximum Likelihood (ML) learning uses theobjective arg min θ D KL ( q ( x ) || p ( x ; θ )) , which can be minimized using the stochastic gradient
[Equation]
) where the positive samples { X + i } ni =1 are a set of data samples and the negative samples { X − i } ni =1 91 are samples from the current model p ( x ; θ ) .
To obtain the negative samples for a deep EBM, it iscommon to use MCMC sampling with the K steps of Langevin equation
[Equation]
[Equation]
where ε is the step size and V k ∼ N (0 , I ) .
The Langevin trajectories are initialized from a set ofstates { X − i, 0 } ni =1 obtained from a certain initialization strategy.
3.2 Hat EBM: Joint Distribution of Latent and Residual Image
This section presents our method for adapting a fixed generator network G ( Z ) into an EBM, whichwe call the Hat EBM .
The Hat EBM defines the joint distribution of the random variable Z ∈ R min the m -dimensional latent space of the generator network and a random variable Y ∈ R d in thed -dimensional image space.
The joint energy has the form
U
[Equation]
Y
[Equation]
(4) where H ( x ; θ ) is a neural network that takes an image x ∈ R d as input and returns a scalar output.
The weights of H are given by θ .
We call H the hat network because it sits atop the generator G toincorporate the generator latent space directly into the probabilistic model.
The random variable Y is meant to accommodate the gap between the output of G ( Z ) and the imagemanifold, since in general we expect that G ( Z ) contains an approximate but imperfect representationof the image distribution which can be refined by the residual state Y .
In practice, we find that themajority of the appearance of G ( Z )+ Y comes from the generator network and not the residual image Y , indicating that the majority of the sampling dynamics of our model are determined by movementin the latent space of G ( Z ) .
We could further encourage movement in the latent as opposed toresidual space by including a prior term such as ∥ Y ∥ 22 in the energy (4) for an appropriate σ .
An appealing aspect of the Hat EBM formulation is that we can learn the model without eithercalculating the log determinant of G ( Z ) as would be required for an energy of the form U ( G ( Z ); θ ) ,or inferring the Z latent vectors associated with observed images X as in existing work on learninglatent prior EBMs [24].
This is possible because we define the distribution of observed images X by X =
G ( Z )
+ Y where the pair ( Y, Z ) is drawn from a true but unknown density q ( y, z ) .
We canthen use the Maximum Likelihood framework in Section 3.1 to learn the weights θ of the hat network H ( x ; θ ) by minimizing the objective arg min θ D KL ( q ( y, z ) || p ( y, z ; θ ))
where
[Equation]
One can obtain negative samples using alternating Langevin updates
[Equation] [Equation]
which switch off between updates with respect to z and updates with respect to y .
The algorithmis essentially Metropolis-within-Gibbs since the Langevin update can be written as a MetropolisHastings step with Gaussian proposal.
This shows that (6) and (7) define a valid sampler for p ( y, z ; θ ) .
Finally, updating θ can be accomplished using
[Equation]
where X + i are observed samples and the pairs (
Y − i , Z − i ) are obtained via MCMC.
In our formulationthe observed data are sufficient statistics for H ( x ; θ ) so that there is no need to infer the ( Y, Z ) pairsfor the positive samples when learning the weights of the hat network.
3.3 Conditional Hat EBM: Residual Image Conditional on Latent Sample
Next we present a conditional variant of the Hat EBM.
While the previous version of the Hat EBM isapplicable to any generator network G ( z ) , including autoencoder networks which cannot typically besampled from, the conditional version of the Hat EBM is tailored towards generator networks whichmap a trivial latent distribution to complex signals like images.
For these kinds of generators, one canuse the latent sampling distribution as an ancestral distribution and learn a conditional distribution onthe residual image given the latent sample.
Suppose we use a trivial marginal distribution p 0 ( z ) for Z and a generator trained using this latentdistribution.
We can now define a conditional Hat EBM p ( y | z ; θ ) =
[Equation]
Z
[Equation]
{− H
[Equation]
G ( z ) + y ; θ ) }and a joint density
[Equation]
In this case, we posit that observed images X are generated according to X
= G ( Z ) + Y forsome distribution q ( y, z )
=
p 0 ( z ) q ( y | z ) .
Obtaining the negative samples ( Z − i , Y − i ) is done by firstdrawing Z − i from p 0 ( z ) and then obtaining Y − i | Z − i by using Langevin updates on the conditionalprobability p ( y | z ; θ ) .
Note that p ( y | z ; θ )
= p ( y | G ( z ; ϕ ); θ ) because of the form of (9).
We update θusing the same equation (8) as the joint Hat EBM because X
+ i is still a sufficient statistic for learning
H ( x ; θ ) and because we do not need to infer the Z + i for X
+ i since the prior p
( z ) does not containany model parameters.
3.4 Learning the Hat Network and Generator in Tandem for Conditional Hat EBM
Both of the formulations of the Hat EBM above assume that a pretrained generator network G ( z )is available as the basis for learning the Hat EBM.
In order to use the Hat EBM as a self-containedlearning process for image generation, we now propose a method to learn the weights ϕ of a generatornetwork G ( z ; ϕ ) and the weights θ of a hat network H ( x ; θ ) simultaneously.
Our method is based onthe cooperative learning [31] strategy.
We first briefly review the cooperative learning formulation,and then present the learning formulation for the Hat EBM.
A key difference between the derivationsis that the original cooperative learning method requires MCMC inference of the latent variable ˆ Zassociated with an MCMC sample X to train G ( z ; ϕ ) , while in our formulation the latent variable
Z is explicitly defined and does not need to be inferred.
This enables major computational savingsbecause we can bypass the expensive MCMC inference of ˆ Z .
In the original cooperative learning [31], the generator output G ( z ; ϕ ) is trained to match theappearance of a Langevin chain X K sampled from the potential U ( X ; θ ) and initialized from thestate X
= G ( Z 0 ; ϕ ) where Z 0 ∼ N (0 , I ) .
The model defines the conditional density of an images
X given latents Z ∼ N (0 , I ) as X | Z ∼ N ( G ( Z ; ϕ ) , τ 2 I ) for some sufficiently small τ .
Given asampled state X t , updating G ( z ; ϕ ) requires inferring Z | X K using the latent Langevin equation
[Equation]
before updating ϕ using the Maximum Likelihood objective function
[Equation]
[Equation]
where the i index denotes a member i of a batch with size n .
We note that in the code released withthe cooperative learning paper, the latent variable is not inferred at all and Z 0 ∼ N (0 , I ) is used inplace of Z K in the objective (11).
In accordance with this approach, we find that inferring Z K oftenhurts model performance and leads to additional complication.
The difficulty of inferring Z | X K andthe omission of this step in practice leave an unresolved gap in the rigor of the cooperative learningformulation.
The Hat EBM update of the generator allows us to bypass the Langevin update for Zwithout theoretical complications.
To learn a Hat EBM generator, we propose initialize Langevin sampling from X
=
G ( Z ; ϕ ) + Y 0where
Y
= 0 and Z ∼ N (0 , I ) to draw residual sample Y K .
The Langevin update is only used toupdate Y k while Z remains fixed, as in the method from Section 3.3.
We once more define our modelas X |
Z ∼ N ( G ( Z ; ϕ ) , τ 2 I ) except we further define the data distribution of ( X, Z ) as Z ∼ N (0 , I )and
X =
Y + G ( Z ; ϕ t ) where Y | Z is drawn from the Hat EBM density (9) using the generator
G ( Z ; ϕ t ) .
Then the G ( z ; ϕ ) can be learned using the Maximum Likelihood objective
[Equation]
Conceptually, this loss function should allow G ( Z ; ϕ ) to closely match the appearance of samples
X =
Y + G ( Z ; ϕ t ) for a fixed generator
G ( Z ; ϕ t ) and the current hat network H ( x ; θ t ) .
Rigoroustraining of H and G involves using the gradient of (12) until convergence is reached, then updatingthe hat network according to (8), and continuing this cycle until EBM convergence.
In practice, weimplement (12) to obtain ϕ t +1 using only one update initialized from ϕ =
ϕ t rather than training untilfull convergence.
This is done to increase training efficiency and to avoid the need of maintaining aseparate copy of generator weights for the fixed network G ( z ; ϕ t ) .
In our experiments, we observe that the objective (12) has some limitations because the generatoroutput can become too closely tethered to biases of the current hat network.
We find the sameproblems with the original cooperative learning objective (11).
See the supplementary material foran illustration.
To overcome these problems, we choose to train G ( z ; ϕ ) at time t + 1 to matchthe historical distribution of hat H ( x ; θ t ℓ ) and generator G ( z ; ϕ t ℓ ) for a selection of past epochst 1 , t 2 , . . .
t L ≤ t instead of training the generator to match the distribution of the current networks
H ( x ; θ t ) and generator G ( z ; ϕ t ) .
This simply involves redefining the data distribution ( X, Z ) byfirst sampling t ℓ from { t 1 , . . .
, t L } and then generating Z and X K =
Y K + G
( Z ; ϕ t ℓ ) where Y | Z 187 follows the energy H ( G ( Z ; ϕ t ℓ ) + Y ; θ t ℓ ) .
In this case, the gradient (12), replacing ϕ t with ϕ t ℓ , isa stochastic approximation of the Maximum Likelihood gradient defined by the joint distribution( t ℓ , Z, X K ) .
See the supplementary material for details.
In practice, we implement this procedureby keeping a persistent bank of 10,000 pairs ( X, Z ) created from past hat network updates.
Whenupdating the generator, we draw n = 128 pairs from the bank and replace it with a newly generatedbatch of n = 128 pairs from the current hat EBM.
This ensures that the selection { t 1 , . . .
, t L } of pastepochs always remains within a certain range of the current epoch t .
Saving the generated images Xat each EBM update allows us to learn from past generator weights ϕ t ℓ without maintaining a copyof the weights.
See Figure 2 for an illustration.
Experiments
In the subsequent empirical evaluations, we will address the following questions:
Refinement:
To what extent can our method refine samples from a pretrained generator modelwith known prior distribution?
Retrofit: Is our method capable of turning a generator model pretrained as a deterministicauto-encoder into a generative model for which samples resemble realistic images?3.
Synthesis:
Can our method learn a generator from scratch with competitive quality of synthesis oncommon image datasets?
Can our method be scaled up to challenging datasets such as ImageNetwith competitive synthesis for unconditional sampling?4.
Out-of-Distribution: Can the Hat EBM be used for Out-of-Distribution (OOD) detection to distinguish between samples from the training distribution and samples from dissimilar distributions?
Figure 3 visualizes some representative sampling paths for the models trained for questions 1, 2,and 3 on the CIFAR-10 dataset.
The supplementary appendix contains details such as pseudocode,training settings, and model architectures.
4.1 Refinement
In this section, we examine the situation explored in [3] from the perspective of the joint Hat EBMtrained according to the method in Section 3.2.
A pretrained SNGAN with generator G is usedas a baseline network.
We learn a Hat EBM that incorporates the generator to refine the initialgenerator samples.
The hat network has the exact same structure as the SNGAN discriminator exceptthat we remove spectral norm layers.
We use fixed batch norm statistics for the generator and ourenergy is deterministic.
The experiment is performed for the CIFAR-10 32 × 32 and CelebA 64 × 64datasets.
To evaluate our method, we compare with Discriminator Driven Latent Sampling (DDLS)[3], which takes the pretrained discriminator D learned with G and samples from the potential
U ( z ) =
D ( G ( z )) + 12 ∥ z ∥ 22 .
We train D and G using the Mimicry [17] repository for reproducible
GAN experiments to obtain stronger baselines for G ( z ) than those used in [3].
Table 1 shows our results.
The Hat EBM can learn a hat network capable of refining an image moresuccessfully than DDLS on both CIFAR-10 and CelebA. We use the joint version of the Hat EBM
Reﬁnement
Langevin [Figure]
X
G ( Z )
Y
X
G ( Z )
Y
X
G ( Z )
Y
Retroﬁt
Langevin Steps →
X
G ( Z )
Y
X
G ( Z )
Y
X
G ( Z )
Y
Synthesis
Langevin Steps →
X
G ( Z )
Y
X
G ( Z )
Y
X
G ( Z )
Y
[Table]
We observe that the majority of the refinement is occurring in the latent space, and that the residualimage Y is essentially imperceptible.
See Figure 3.
We notice that training quickly becomes unstablewhen Y is removed from the Hat EBM.
In our experience, it is essential to incorporate the residualfor stable learning.
A possible reason for this phenomenon is that the hat network can learn todiscriminate between generator images and images not from the generator, whether they are realisticor not.
If so, the hat network can assign increasingly high energy to generator samples in the absenceof the residual Y .
Even an imperceptible Y appears enough to prevent the hat network from easilydistinguishing positive and negative samples so that learning becomes stable.
4.2 Retrofit
In this section, we incorporate a non-probabilistic generator network G ( z ) into a probabilistic Hat
EBM model.
This essentially allows us to sample from the latent space of G ( z ) to find latent sampleswhose mapping corresponds to a realistic image.
Similarly to results in Section 4.1, the residualimage has small norm in the image space and most of the appearance of the sampled images comesfrom G ( z ) .
This happens naturally without the need to coerce Y to be close to 0 by including a priorterm such as p 0 ( y ) =
∥
y ∥ 22 , although including a prior term can further limit growth of Y .
The autoencoder generator G ( z ) is pre-trained as the second half of a standard inference network andgenerator network pairing.
An image X is fed into the inference network I and converted into a latentstate
Z
= I ( X ) , which is then decoded with ˆ X =
G ( Z ) .
The inference network and generator arelearned jointly using the MSE loss ∥ ˆ X − X ∥ 22 .
To keep the latent space mapping of I ( X ) numericallystable, we project the raw output of I ( X ) to the annulus around the origin with radius √ d so that
[Table]
∥
I ( X ) ∥ 2 = √ d .
More sophisticated methods such as perceptual and adversarial loss could have beenused to train the autoencoder, but we use MSE loss to keep our implementation minimal.
We observethat when Z is a vector-shaped latent state, it can be extremely difficult to achieve reconstructions ˆ Xwith sharp appearance even for simple datasets like CIFAR-10.
To obtain better reconstructions andtherefore a latent space with more realistic mappings to the image space, we use image-shaped latentstates Z .
The details of our autoencoder networks can be found in the appendix.
When Z is an imageshaped latent, we treat it exactly the same as a vector latent in the learning and sampling algorithms.
We experiment with assimilating an autoencoder into a Hat EBM potential for the CIFAR-10 dataset.
Our results are presented in Figure 3 with additional results in the supplementary appendix.
Wetrain the Hat EBM using shortrun learning in the latent and image space by initializing Y 0 = 0 and
Z 0 ∼ N (0 , I ) and using K = 100 MCMC steps of (6) and (7) from initialization during both trainingand testing evaluation to generate samples.
During the Langevin dynamics, image appearance isrefined mostly in the latent space.
Our best model achieves a solid FID score of 26.01 ± 0.09.
Thisdemonstrates that Hat EBM can learn a probabilistic model over a non-probabilistic latent space.
4.3 Synthesis
In this section, we use the conditional Hat EBM formulation from Section 3.4 to learn a hat networkand generator network from scratch for self-contained synthesis.
We explore synthesis for CIFAR-10,
CelebA at resolution 64 × 64, and unconditional ImageNet at resolution 128 × 128.
While recentgenerative models show promising results for class-label conditional sampling, unconditional sampling with high quality synthesis remains a significant challenge.
We find especially strong resultsfor ImageNet synthesis using the conditional Hat EBM.
This demonstrates strong potential for oursynthesis method for learning with highly diverse unstructured datasets.
We use SNGAN architectures for sizes 32 × 32 , 64 × 64 , and 128 × 128 , where the discriminator 269 architecture is used for the hat network.
During learning, we keep the generator batch norm parametersfixed to mean 0 and variance 1.
We remove all spectral norm layers from the hat network.
Trainingparameters can be found in the supplementary material.
For ImageNet models, we found thatannealing the generator and hat network learning rate by a factor of 10 after 250K weight updates foreach network further improved the FID score significantly.
Our results are shown in Table 2.
See thesupplementary material for uncurated samples from each model.
Results show strong performance compared to a selection of representative generative modelsacross all datasets, with an especially strong performance for ImageNet.
Our method significantlyoutperforms other EBM learning methods on CIFAR-10.
The Hat EBM synthesis results are onpar with the SNGAN baseline for CIFAR-10 and CelebA. The Hat EBM results for ImageNetsignificantly outperform SNGAN.
At the budget of ~8 GPUs, our Hat EBM achieves a score of 40.0,outperforming the small-scale SSGAN.
To our knowledge, the current state-of-the-art score for unconditional ImageNet 128 × 128 is the
SSGAN [4] with a score of 23.4 trained using a BigGAN network and 128-core TPUv3 pods.
Toscale up our Hat EBM, we doubled the number of channel dimensions for both the hat network and
[Table]
generator network from the original SNGAN architecture and trained on 32-core TPU-v3 pods.
Ourbest FID score for unconditional ImageNet 128 × 128 was 29.2, which comes within a competitiverange of state-of-the-art.
We believe that further scaling in future work could enable Hat EBM tomatch or surpass state-of-the-art.
Our results decisively demonstrate the potential of EBM learningwell beyond the scale investigated in any prior EBM work.
4.4 OOD
We assess our model performance on Out-Of-Distribution detection.
We use the conditional Hat EBMmodel trained on CIFAR-10 from Section 4.3 and calculate the energy H ( X ; θ ) on in-distributionimages from the CIFAR-10 test set and the OOD datasets which include CIFAR-100, CelebA, and
SVHN.
We follow standard OOD evaluation from works such as [22] which use the AUROC metric.
This metric measures the ability of the Hat EBM to distinguish between in-distribution samples notseen during training and OOD samples.
Following [9, 30], we expect that the energy of the OODdatasets be higher than the energy of in-distribution test images.
Our results are shown in Table 3.
The Hat EBM shows strong performance as an OOD detectionmethod.
Among methods that are fully unsupervised, our model has the top performance across allthree OOD datasets.
Our method approaches the results of methods that are trained with labelled datasuch as HDGE [19] and the fine-tuned OOD EBM [20], although we do not yet match these scores.
Overall, there is strong evidence at the Hat EBM is naturally an effective method for OOD detection,especially when supervised label information is unavailable.
Conclusion
The maximum-likelihood based learning of EBMs poses a significant challenge: drawing negativesamples from the current density model, which is often highly multi-modal.
Prior art addresses thischallenge by recruiting approximations of the EBM in the form of an ancestral sampling from agenerator model, truncated Langevin chains, flow-based models, or lifting the EBM into the inducedlatent space of generator models.
In contrast, our work proposes a method for absorbing any generatoras a backbone of an EBM.
The formulation assumes that observed images are the sum of unobservedlatent variables pushed forward through the generator and a residual random variable which closes thegap between generator samples and image manifold.
As such, the Hat EBM sits atop the generator.
The generator allows for efficient sampling but may only capture the coarse structure of the images,while the residuals can capture fine-grained or even imperceptible details.
The Hat EBM formulation is presented in three variations: (1) joint learning of latent and residualimage for adapting any fixed generator, (2) conditional learning for generators with known priordistribution, (3) self-contained learning of both EBM and generator from scratch.
Notably, thetraining requires neither the log determinant of the generator Jacobian or inference of latent variables,rendering the learning much simpler and more scalable.
Empirical evaluations demonstrate the various capabilities of the Hat EBM: (1) Strong performancefor the ImageNet synthesis at 128 × 128 resolution with self-contained learning, (2) Significantrefinement of the quality of synthesis of pre-trained generators on CIFAR-10 and CelebA withconditional learning, (3) Retrofitting pre-trained auto-encoder
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]