L EARNING E NERGY -B ASED G ENERATIVE M ODELS VIA C OARSE - TO -F INE E XPANDING AND S AMPLING
A BSTRACT
Energy-based models (EBMs) parametrized by neural networks can be trained by Markov chain Monte Carlo (MCMC) sampling-based maximum likelihood estimation.
Despite the recent signiﬁcant success of EBMs in data generation, the current approaches to train EBMs can be unstable and sometimes may have difﬁculty synthesizing diverse and high-ﬁdelity images.
In this paper, we propose to train EBMs via a multistage coarse-to-ﬁne expanding and sampling strategy, namely CF-EBM.
To improve the learning procedure, we propose an effective net architecture and advocate applying smooth activations.
The resulting approach is computationally efﬁcient and achieves the best performance on image generation amongst EBMs and the spectral normalization GAN.
Furthermore, we provide a recipe for being the ﬁrst successful EBM to synthesize 512 × 512 -pixel images and also improve out-of-distribution detection.
In the end, we effortlessly generalize CFEBM to the one-sided unsupervised image-to-image translation and beat baseline methods with the model size and the training budget largely reduced.
In parallel, we present a gradient-based discriminative saliency method to interpret the translation dynamics which align with human behavior explicitly.
I NTRODUCTION
[Figure]
image-to-image translation (256 2 ) interpreted by the discriminative saliency (cid:80) ∇ X E ( X ) .
Recently, energy-based models (EBMs) (Zhu et al., 1998; LeCun et al., 2006) parameterized by modern neural networks have drawn much attention from the deep learning communities.
Successful applications with EBMs include generations of images (Xie et al., 2016; 2018b;
Du & Mordatch, 2019), videos (Xie et al., 2017; 2019), 3D volumetric shapes (Xie et al., 2018c; 2020), unordered point clouds (Xie et al., 2021a), texts (Deng et al., 2020), molecules (Ingraham et al., 2018;
Du et al., 2019), etc., as well as image-to-image translation (Xie et al., 2021b;c), out-of-distribution detection (Liu et al., 2020) and inverse optimal control (Xu et al., 2019).
EBMs are characterized by ( i ) Simplicity:
The maximum likelihood learning of EBMs uniﬁes representation and generation in a single model, and ( ii ) Explicitness:
EBMs provide an explicit density distribution of data by training an energy function that assigns lower values to observed data and higher values to unobserved ones.
However, it is still difﬁcult to train an EBM to synthesize diverse and high-ﬁdelity images.
The maximum likelihood estimation (MLE) of EBMs requires the Markov Chain Monte Carlo (MCMC) (Liu, 2008; Barbu & Zhu, 2020) to sample from the model and then updates the model parameters according to the difference between those samples and the observed data.
Such an “analysis by synthesis” (Grenander et al., 2007) learning scheme is challenging because the sampling step is inefﬁcient, unstable, and hard to traverse modes of the learned model, especially when the energy function is multimodal due to highly varied or high resolution training data.
Fortunately, it is common knowledge that the manifold residing in a downsampled low-dimensional image space is smoother than that in the original high-dimensional counterpart.
Thus, learning an EBM from low-dimensional data is much stabler and faster in terms of convergence (Odena et al., 2017; Gao et al., 2018).
Inspired by the above, we propose to train EBMs via a multistage coarse-to-ﬁne expanding and sampling strategy (CF-EBM).
As shown in Figure 1(a), the approach starts with learning a coarselevel EBM on low resolution images and then smoothly transits to learn the ﬁner-level EBM by adding new layers that take into account the higher resolution information as the learning progresses.
The gradient-based short-run MCMC (Nijkamp et al., 2019), e.g. , Langevin dynamics (Neal et al., 2011), is used for sampling.
From the modeling aspect, the coarse-level training can be useful for exploring the global structure of image, while the ﬁne-level training will then gradually reﬁne the image details.
Recent works have demonstrated the advantages of this incremental learning (Karras et al., 2018;
Wang et al., 2018).
However, there have been no works focusing on the incremental learning of EBMs that incorporates bottom-up modeling and top-down sampling in a single net.
Besides, as shown in Figure 1(a), the top-down gradient information for synthesis ﬂows through coarse-level layers towards ﬁne-level layers.
Thus, during the coarse-to-ﬁne expanding, we can use the coarse-level synthesis to help the ﬁne-level synthesis to stabilize the sampling.
Such a coarse-to-ﬁne expanding and sampling scheme is useful for high-ﬁdelity synthesis in several vision tasks.
See Figure 1(b).
Furthermore, we propose a one-sided energy-based unsupervised image-to-image translation method that can scale up to high resolution.
The approach is immediately available with the FC-EBM by using iterative Langevin dynamic without an explicit cycle consistency (Zhu et al., 2017) or geometry constraints (Fu et al., 2019).
Meanwhile, it works as a saliency method (Simonyan et al., 2014; Adebayo et al., 2018) by using gradients to explain essential aspects, namely the discriminative saliency, in translating from the source domain X to the target domain Y as shown in Figure 1(c).
To accomplish this, EBM learns a scalar energy function E : Y → R and applies MCMC to sample
Y starting from X , instead of learning a block-box domain mapping network F : X → Y in prior works (Zhu et al., 2017; Huang et al., 2018; Park et al., 2020).
The contributions of our paper can be summarized as:
• To the best of our knowledge, this is the ﬁrst work that trains EBMs under the “analysis by synthesis” scheme via a multistage coarse-to-ﬁne expanding and sampling strategy.
Besides, we propose eliminating the additive noise to the observed data while experimenting with several essential techniques for improving EBM, e.g. smooth activations.
In particular, CF-EBM is the ﬁrst pure EBM approach applied to model images as large as 512 × 512 .
• We propose a novel energy-based unsupervised image translation approach, which is essentially different from all other existing GAN-based approaches.
We demonstrate noticeable results in terms of both translation quality and efﬁciency of time and memory.
• We conduct extensive experiments to validate our approach, including image generation, denoising, inpainting, out-of-distribution detection and unsupervised image-to-image translation.
Strong results are shown to outperform or be competitive with the prior art mostly.
The rest of the paper is organized as follows.
Section 2 summarizes related works and how our proposal is differentiated from those.
Section 3 introduces the proposed methodology in detail.
Section 4 presents extensive experiments to test our method.
Section 5 concludes our paper and discuss some future research directions.
R ELATED WORK 2. E NERGY - BASED GENERATIVE MODELS
The main challenge to train EBMs via MLE lies in drawing fair samples from the model, especially when the energy function is parameterized by a highly non-linear ConvNet.
The contrastive divergence (CD) (Hinton, 2002; Tieleman, 2008), with MCMC chains initialized from data distribution, is an efﬁcient but biased way to train EBMs.
Another direction is to adopt the idea of energy-based correction of a more tractable model to train EBMs.
Noise contrastive estimation (NCE) (Gutmann &
Hyv¨arinen, 2010; Gao et al., 2020) and introspective neural networks (INN) (Lazarow et al., 2017;
Jin et al., 2017; Lee et al., 2018b) belong to this theme.
Generative cooperative networks (CoopNets) (Xie et al., 2018b; 2021d;b) train an EBM with a generator or a variational auto-encoder (VAE) (Kingma & Welling, 2014) as amortized sampler by MCMC teaching (Xie et al., 2018a).
Triangle divergence (Han et al., 2019) trains an EBM without MCMC by amortizing the MCMC via a VAE.
However, these frameworks still struggle to scale up and model multimodal data.
There have been several strategies to improve the EBM training.
Gao et al.
(2018) adopts a multi-grid method that trains multiple EBMs at different grids simultaneously, where the EBM at coarser grid is used to initialize the image generation by EBM at ﬁner grid.
However, optimizing and sampling from multiple
EBMs will result in low efﬁciency of both time and memory.
To stabilize the training, Nijkamp et al.
Grathwohl et al.
(2020) add the Gaussian white noise to the observed data, resulting in noisy synthesized images.
In contrast, our paper proposes to train a single EBM via a coarse-to-ﬁne growing strategy, along with some improved techniques.
With smooth parameter training and image sampling, our model can preserve EBM’s compatibility and synthesize high-ﬁdelity images.
2. U NSUPERVISED IMAGE - TO - IMAGE TRANSLATION
Unsupervised image-to-image translation aims at learning two directions of mappings between two unpaired domains.
Recent successes are all based on adversarial learning, e.g. , CycleGAN (Zhu et al., 2017), UNIT (Liu et al., 2017), MUNIT (Huang et al., 2018), DRIT (Lee et al., 2018a) and U-GATIT
(Kim et al., 2020).
These methods typically train two GANs with two levels of learning objectives: ( i ) Distribution level: Two adversarial losses are used to capture style discrepancy between source and target domain; ( ii ) Instance level: To tackle the difﬁculty of unpaired setting, they adopt a cycle consistency loss for content preservation.
This loss enables an instance-level supervision to regularize the training of two mappings by enforcing them to be a bijective function between two domains.
Except for works about two-sided translation, efforts on research about one-sided unsupervised image translation have also been made, e.g. , DistanceGAN (Benaim & Wolf, 2017), GcGAN (Fu et al., 2019) and CUT (Park et al., 2020), which apply geometric or contrastive constraints.
We solve this problem from the prospective of EBM, which is different from GAN-based methods.
The proposed concise EBM solution only relies on its built-in objective, which is a distribution-level statistics matching, to accomplish the one-sided image translation.
It transfers the style and preserves the source content by MCMC without using the cycle-consistency loss.
The model demonstrates better performances with less time and memory.
Another distinction between our method and GAN-based methods is the natural interpretability of Langevin dynamics.
It provides a gradient-based saliency map to visualize those key regions that make the two domains distinct, as illustrated in Figure 1(c).
M ETHOD
In this section, we ﬁrst present the EBM learning framework, and then the proposed CF-EBM approach.
After that, we generalize our model to the unsupervised image-to-image translation.
3. MCMC- BASED M AXIMUM L IKELIHOOD L EARNING OF E NERGY - BASED M ODEL
Let x ∈ R D be the observed example, e.g. an image.
An energy-based model is deﬁned as follows:
[Equation]
where E θ ( x ) : R D −→ R is the energy function deﬁned by a bottom-up ConvNet parameterized by θ .
Z ( θ ) =
(cid:82) exp( − E θ ( x ))
d x is the intractable normalizing constant or the partition function.
Given N observed examples { x i } Ni =1 ∼ p data ( x ) , where p data ( x ) denotes the unknown data distribution, the model can be trained by maximizing the log-likelihood L ( θ )
=
1 N (cid:80) Ni
=1 log p θ ( x i ) ≈
E [Equation] p
[Equation]
log( p θ ( x )) .
The derivative of the negative log-likelihood is given by
[Equation]
where the second expectation term under p θ ( x ) is intractable and can be approximated via MCMC.
Given that, the EBM is updated by gradient descent.
To sample ˜ x ∼ p θ ( x ) via MCMC, we rely on gradient-based Langevin dynamics that recursively computes the following step
[Equation]
where η t is the step size of Langevin step and also the variance of Gaussian noise (cid:15) t .
Theoretically, to ensure convergence, the MCMC is typically performed with inﬁnite steps and an inﬁnitesimal stepsize (Welling & Teh, 2011).
However, it is impractical for training EBMs.
In this paper, we follow Nijkamp et al.
(2019) to use short-run MCMC, which always starts from a ﬁxed distribution and runs a ﬁxed number T of Langevin steps in both training and testing stages.
The training with a short-run MCMC might result in a biased estimation of EBM but the learned short-run MCMC is still a valid generator, which enables us to synthesize realistic images and efﬁciently train the model, as seen in most well-established EBM works (Nijkamp et al., 2019; Grathwohl et al., 2020; Pang et al., 2020).
In this paper, we keep the step size constant and linearly decay the noise variance till 0.
3. CF-EBM: C OARSE - TO - FINE E XPANDING AND S AMPLING OF E NERGY -B ASED M ODEL
[Table] data { x ( s ) i , i = 1 ,...,N ; s = 1 ,...,S } , numbers of Langevin steps { T ( s ) } Ss =1 , and batch sizes { n ( s ) }
Ss = Output: Energy function E ( S ) E (0) ← ∅ for s = 1 : S do m = 0 , β = Expand model: E ( s ) ← E ( s − 1) + Expand ( · ) while ( m ≤ N ) do # Draw real data and initial MCMC samples
Draw observed { x ( s ) i } n ( s ) i =1 ∼ p data ( x )
Draw initial { ˜ x ( s ) i } n ( s ) i
=1 ∼ U ( s ) ( − 1 , 1) # Smooth MCMC sampling if ( β < 1 and s > 1 ) then
Draw initial { ˜ x
( s − 1) i } n ( s ) i =1 ∼ U ( s − 1) ( − 1 , 1)
Update { ˜ x
( s − 1) i } n ( s ) i =1 for T ( s − 1) (1 − β ) steps of
Eq.
(3) with E ( s − 1) ˜
x
( s ) i =
(cid:112) 1 − β 2 Upsample (˜ x
( s − 1) i ) + β U ( s ) ( − 1 , 1) end
Update { ˜ x
( s ) i } n ( s ) i =1 for T ( s ) steps of Eq.
(3) with E ( s )
# Optimize energy function Update E ( s ) with gradient descent in Eq.
(2) m ← m + n ( s ) , β = max(1 , 2 N/n ( s ) ) end end
Our primary contribution is the multistage coarse- to-ﬁne expanding and sampling methodology for training EBMs.
The key idea is to incrementally grow the EBM from a low resolution (coarse model) to a high resolution (ﬁne model) by gradually adding new layers to the energy function.
In this way, both stability and time efﬁciency in training EBMs beneﬁt.
And eventually, we only keep the highestresolution EBM for image generation using the short-run MCMC sampling.
Figure 2 (a) illustrates the proposed multistage coarse-to-ﬁne EBM training strategy, which is accompanied by the pseudocode in Algorithm 1.
Let
S denote the total number of training stages and ( x ( s ) , s = 1 , ..., S ) denote the multi-resolution versions of an image x , with x (0) being the minimal resolution version of x , and x ( S ) = x . For each x ( s ) , we can easily generate x ( s − 1) by average pooling with a pool- [Figure] ing size.
At stage s , we associate the model with an energy function E ( s ) to learn from x ( s ) .
The model starts from an initial minimal resolution at stage 1.
When the training proceeds to the next stage s + 1 , we will add randomly initialized layers at the bottom to expand the resolution, and obtain a new energy function E ( s +1) .
For example, in Figure 2 (a), when the training stage switches from stages 2 to 3, we expand E (2) , which is an energy function of resolution 16 × 16 , by adding a bottom block accounting for resolution 32 × 32 to get E (3) .
After that, we only need to train E (3) at stage 3.
Compared with PGAN (Karras et al., 2018) which proposes a progressive growing strategy to train
GAN, our CF-EBM integrates both the learning (Eq. and the sampling (Eq. (3)) into a single neural network E θ ( x ) , which is essentially different from PGAN that applies a discriminator to guide the learning and a generator to produce samples.
Therefore, training an EBM via a multistage growing strategy is more challenging and non-trivial.
With the newly expanded energy function E ( s +1) and the enlarged training examples x ( s +1) , the stage transition must be taken care of to avoid instability issue. The CF-EBM deals with this issue from the following two aspects: i
Smooth EBM Learning: Let Expand ( · ) denote an expanding block. To expand the resolution of the EBM at each stage, we add an Expand ( · ) to the bottom of the energy function to double its resolution.
We fade in the block smoothly to prevent sudden shocks to the well-trained coarse-level energy function.
Speciﬁcally, Expand ( · ) is a composition of a primal block Primal ( · ) and a fading block Fade ( · ) .
We deﬁne Expand ( x ) =
β Primal ( x ) + (1 − β ) Fade ( x ) , where β is a fading factor that gradually increases from 0 to 1 as more data are consumed in the training.
As a result, the model will ﬁrst rely on the well-trained coarse-level layers and gradually move the attention to the newly added ﬁne-level layers.
The auxiliary fading block will be abandoned when β reaches 1.
In this paper, Primal ( · ) consists of two convolution layers and Fade ( · ) is a convolution layer followed by a 2 × 2 average pooling to link the previous coarse-level layers.
(ii) Smooth MCMC Sampling:
The smooth sampling is realized both implicitly and explicitly.
( a ) Implicitly : Considering the resolution transition from stage s − 1 to stage s , we expand the energy function by E ( s ) ( x ( s ) )
=
E ( s − 1) ( Expand ( x ( s ) )) .
The gradient ∇ x ( s ) E ( s ) ( x ( s ) ) for Langevin sampling in Eq.
(3) can be unfolded as:
[Equation]
Initially, because β is small, the newly added primal block Primal ( · ) , whose weights are randomly initialized, would not largely affect the sampling immediately. Only the coarse-level function E ( s − 1) and the auxiliary fading block Fade ( · ) greatly contribute to the synthesis.
As β increases, the Primal ( x ( s ) ) becomes increasingly well-trained. Meanwhile, the term ∇ x ( s ) Fade ( x ( s ) ) fades away and ∇ x ( s ) Primal ( x ( s ) ) gradually takes the lead in the sampling, thus the synthesized images would become sharper and sharper.
( b ) Explicitly : To mitigate the impact of the sudden model expansion on sampling, we utilize the well-trained coarse model E ( s − 1) to initialize the MCMC sampling of the newly expanded model E ( s ) .
Speciﬁcally, at stage s , the algorithm ﬁrst generates low-resolution samples from E ( s − 1) by running T ( s − 1) (1 − β ) Langevin steps.
Those samples are then upsampled ( 2 × ) by Upsample ( · ) and mixed with a uniform noise to initialize the MCMC sampling of E ( s ) .
As β increases, such an MCMC initialization assistance from the coarse model would fade away.
Eventually, when β reaches 1, we can directly sample from E ( s ) with a purely noise-initialized MCMC.
Figure 2 (b) illustrates the smooth sampling process at stage 3.
We use different numbers of Langevin steps at different stages.
For example, we only run MCMC with T (1) =
15 steps at stage 1, and gradually increase the number of steps as the model grows.
The maximal number of steps is 60 in our paper.
[Table]
After training, we obtain the target EBM with E ( S ) .
The sampling procedure by the learned model is presented in Algorithm 2. The
MCMC starts from a uniform distribution and runs T ( S ) steps of Langevin updates to generate samples.
Note that, both the initial distribution and the number of Langevin steps are the same in both training and sampling.
If we run MCMC with more steps than that used in training, the oversaturation occurs (Nijkamp et al., 2019).
3. E NERGY - BASED U NSUPERVISED I MAGE - TO - IMAGE T RANSLATION
We then generalize the proposed approach to the task of unpaired image-to-image translation and scale it up to high resolution image datasets.
Given two image domains, X and Y , endowing two ground-truth distributions p ( x ) : x ∈ X and p ( y ) : y ∈ Y , the goal is to learn two energy functions
F ( x ) : X → R and G ( y ) :
Y → R .
Here, we only explain how to learn G as the vice versa should be straightforward.
The target distribution p G ( y )
∝ exp( − G ( y )) .
Hence, the image translation process is given by:
[Equation]
The underlying assumption of applying
EBM for image translation is that the two domains ( X , Y ) share the same ambient space where each data can be decomposed into a content code c and a domain speciﬁc style code s X of domain X or s Y of domain Y respectively.
This assumption is somewhat similar to the partially shared latent space assumption in Huang et al. ; Choi et al. (2020); however, it is directly deﬁned in the ambient space and is more efﬁcient without resorting to complementary models.
Consequently, as shown in Figure 2(c) and Figure 3, the above dynamic aims to evolve the style code s X towards s Y , where the gradient descent steps into a lower energy area with content c unchanged as much as possible.
Discriminative Saliency Map:
With the proposed method, we visualize the image translation dynamic in Figure 3.
The three datasets are selected for demonstration because each contains different kinds of discriminative saliency, e.g. color and texture.
But how are we supposed to know this?
We adopt the gradient-based saliency maps (Simonyan et al., 2014) to highlight the discriminative aspect that leads to the translation, which we call discriminative saliency map .
It is computed as the aggregated scores in Eq.
(4), i.e. , (cid:80) t ∇ y t G ( y t ) .
It quantiﬁes how much a change of each pixel would contribute to the image translation.
From Figure 4, we can conclude all types of discriminative saliency in different datasets, e.g. , color and shape in orange → apple , and nose and ear in cat → dog .
[Figure]
3. S MOOTH A CTIVATIONS I MPROVES T RAINING
We further claim that smooth activations potentially strengthen EBM learning.
Given a simplest
EBM model E ( x ) =
Act ( Block ( x )) with a single Block ( · ) activated by Act ( · ) , the score function in Eq.
(3) is ∇ x E ( x ) =
Act (cid:48)
( Block ( x )) Block (cid:48)
( x ) .
When the data x is continuous, the smoothness of Act (cid:48) ( · ) will substantially affect the iterative sampling process, thus applying an activation with continuous gradients everywhere that enforces a smooth Langevin dynamics is potentially useful.
In this regard, we consider common activations for demonstration, including ReLU, LeakyReLU,
GELU, CELU, and Swish.
See Appendix A.4 for their functional forms and visualizations.
In Table 8, we observe that non-smooth activations ReLU and LeakyReLU tend to diverge, whereas others don’t.
Note that most well-established EBM generative models add Gaussian noise to smooth data and make it easier for model convergence (Nijkamp et al., 2019; Grathwohl et al., 2020).
Otherwise, the model will diverge quickly.
Similarly, score-based models (Song & Ermon inject multiple level noises to improve score estimations.
However, the additive noisy information will be leaked to the sampler and result in foggy synthesized samples (Grathwohl et al., 2020; Song & Ermon, 2020).
In this papaer, we integrate smooth activations with the proposed model such that we can alleviate the instability issue effortlessly while preserving original data structures.
See Appendix A.3.1 for more discussions and Figure 12 for a visualization of perturbed samples.
E XPERIMENTS
In the main part, we showcase the results on various tasks, including image generation, image restoration (inpainting and denoising), out-of-distribution (OOD) detection and unpaired image-toimage translation.
These are implemented on the same base neural architecture and TITAN X (12GB) platform mostly.
Detailed experimental settings are provided in Appendix A.2.
We adopt Fr´echet
Inception Distance (FID) (Heusel et al., 2017) and Kernel Inception Distance (KID) (Bi´nkowski et al., 2018) for quantitative evaluation , and Amazon Turker Platform (AMT) for perceptual human evaluation (detailed in Appendix A.1).
KID score is proved more reliable when there are fewer testing images available, e.g. in the image translation.
In Appendix A.4, we conduct a detailed ablation study, including activation functions, normalization strategies, and layer connections.
Finally, in
Appendix A.6, we scale the model up to 512 2 -pixel images (e.g., CelebA-HQ) when all intermediate
CF-EBMs learned in Algorithm 1 are incorporated in the learning and sampling process.
4. I MAGE GENERATION
Datasets ( i ) CIFAR-10 (Krizhevsky, 2009) is a dataset containing 60k images ( 32 × 32 ) ofclasses; ( ii ) CelebA
(Liu et al., 2015) is a celebrity facial dataset containing over 200k images.
For fair comparisons with former EBM works, the 64 × 64 resolution is used for quantitative evaluation; ( iii ) CelebA-HQ 512 × 512 contains 30K high resolution facial images (Karras et al., 2018).
[Table]
[Figure] [Figure]
[Table]
[Figure] [Table] [Figure] Quantitative Results: The quantitative unconditional image generation results on CIFAR-10 and
CelebA are given in Table 1 and Table 2.
In Table 2, † works are experimented on the center- crop( 140 × 140 )-resized
CelebA whereas others are learned on the resized CelebA.
In all cases, the proposed CF-EBM is leading by a wide margin over other approaches.
Du & Mordatch (2019) claims that large models are necessary to learn multimodal data.
To examine this, in Table 3, we compare the number of parameters and relative evaluation time of representative open-sourced EBMs and score matching models since both kinds rely on Langevin dynamics for sampling.
We use 50 steps for sampling (cross-validated) while all other three methods require either 60, 100 or even more steps.
It is observed that our model consistently stands out in terms of efﬁciency while keeping the lowest
FID on CIFAR-10.
[Table]
Model Analysis: EBM generative models, once learned, can be directly applied to image restoration tasks.
As in Du & Mordatch (2019), we conduct an experiment on the image restoration to demonstrate the model’s ability of mode coverage and generalization to the unseen data.
( i ) Image denoising
: We randomly
pick up images from the testing dataset of CIFAR-10.
Then, we pollute each image x with an additive Gaussian noise (cid:15) ∼
N
Figure 5 shows the restored result ˜
x
on all ten classes of CIFAR-10, implying a good generalization ability.
( ii ) Image inpainting : it can be viewed as associative memory (Hopﬁeld, 1982).
As shown in
Figure 6, the recovered images don’t mimic the ground truth with 25% image areas occluded.
We also provide diverse image inpaintings in Figure 11 in Appendix A.3.
We demonstrate the EBM characterized by MLE indeed learns meaningful representations We additionally provide nearest neighbor search results in Figure 10 to validate that our model doesn’t suffer from over-ﬁtting.
Qualitative Samples: We have presented qualitative samples of Celeba-HQ 256 × 256 in Figure 1.
Please refer to Appendix and see Figure 8 for CIFAR-10 samples, Figure 15 for how the noise level affects the samples’ visual effect in the evaluation and Figures 16, 17 for more high resolution samples.
4. O UT - OF - DISTRIBUTION DETECTION
Out-of-distribution (OOD) detection is a binary classiﬁcation problem.
Given input examples, the model must produce a score that we expect to be higher for in-distribution examples and lower for out-distribution examples.
Du & Mordatch (2019) has shown that the likelihoods from EBMs can be a reliable OOD detector.
According to this principle, we construct the OOD detection experiments and compare our model with the other three generative models, Glow (Kingma & Dhariwal, 2018), conditional EBM (Du & Mordatch, 2019) and JEM (Grathwohl et al., 2020).
All models are trained on CIFAR-10.
We use SVHN (Netzer et al., 2011), a uniform distribution (Uniform), a constant distribution (Constant), CIFAR
Interpolation (Interp) and CIFAR100 (Krizhevsky, 2009) as OOD distributions.
We apply the area under the receiver-operating curve (AUROC) (Hendrycks & Gimpel, 2016a) as a metric for evaluation.
As shown in Table 4, our model shows mostly better or competitive results, even though it doesn’t incorporate any label information at the training stage (Grathwohl et al., 2020).
[Table]
4. U NSUPERVISED I MAGE - TO -I MAGE TRANSLATION
We mainly use four unpaired image-to-image translation datasets for evaluation, including cat2dog,
Yosemite summer2winter, vangogh2photo and apple2orange.
All images are resized to 256 ×resolutions.
More details are provided in Appendix A.7.1.
We compare our approach against
CycleGAN (Zhu et al., 2017), CUT (Park et al., 2020) and two latest state-of-the-art methods UGAT-IT (Kim et al., 2020) and FQ-GAN (Zhao et al., 2020).
CycleGAN is a popular GAN-based image translation framework and we use it to demonstrate a basic comparison between two kinds of models, GAN-based model with cycle constraint and the proposed non-GAN-based model without cycle constraint .
CUT (Park et al., 2020) is the state-of-the-art one-sided unpaired image translation model which applies the contrastive loss to preserve the content.
U-GAT-IT and FQ-GAN are current leading models on cat2dog and vangogh2photo.
We present the baselines’ results mostly from the public results or checkpoints.
Quantitative Results: Results are shown in Table 5.
In most cases, our model obtains better KID scores and preference scores in the human evaluation.
Meanwhile, the training time and model size are incredibly low.
We postulate that this efﬁciency comes from two aspects: model design and optimization process .
( i ) Model design : The energy-based model is an image-to-scalar mapping instead of an image-toimage mapping as in GANs.
It can also implicitly transfer styles and preserve contents without cycle constraint.
Therefore, the model size will be smaller than GAN-based models where it applies two pairs of GANs.
( ii ) Optimization process : GANs apply the adversarial loss to match styles and the cycle loss to preserve the content.
In practice, the two losses are partially decoupled and optimized alternatively.
In contrast, our approach only applies MLE on one neural network (Eq. (2)) and practically, we ﬁnd that the training takes less iterations to uncover the discrepancy between two domains.
Although we need
to run additional Langevin sampling
, the
overall optimization cost is still low
[Table]
Qualitative Results: As can be seen in Figure 7, our model can generate sharper images with superior visual quality.
The source content is better preserved while focusing on evolving the source style to the target domain.
We put the comparison between our model and one-sided translation model CUT (Park et al., 2020) in Appendix A.7.2.
More uncurated results on photo ↔ V angogh and Yosemite summer ↔ winter can be found in Appendix A.7.3.
C ONCLUSION
To tackle the training instability and the multimodal MCMC sampling difﬁculty of EBM generative models, we propose a coarse-to-ﬁne expanding and sampling strategy, coined CF-EBM.
Extensive experiments demonstrate the superior performance compared with various generative models, in sample quality and computational efﬁciency.
This success is due to our newly proposed architecture, smoothing activations, and the ﬂexibility of EBM.
To the best of our knowledge, CF-EBM is the ﬁrst pure EBM generative model that can produce large high-ﬁdelity images and be competent in unsupervised image-to-image translation.
The future work includes investigating EBM’s intrinsic mechanics, experimenting on large architectures, and applications on other types of learning scenarios.
A A PPENDIX A. E VALUATION METRICS
We consider two commonly used metric for evaluating image synthesis quality.
( i ) FID (Heusel et al., 2017) compares the statistics (mean and variances of Gaussian distributions) between the generated samples and real samples.
FID is consistent with increasing disturbances and human judgment.
Lower scores indicate that the model can generate higher quality images.
We follow the dataset split protocol in Zhang et al.
(2020) to compute FID.
( ii ) KID (Bi´nkowski et al., 2018) improves FID as an unbiased estimator, making it more reliable when there are fewer available test images.
We use generated images translated from all test images in the source domain vs. test images in the target domain to compute KID.
Lower KID values indicate that images are better translated.
We follow Kim et al.
(2020) to compute KID.
For AMT evaluation, we follow the setting in FQ-GAN.
Each testing image is judged by six users, who are asked to select the best-translated image to target domain where two standards should be considered, visual image quality, and the extent of background information is kept.
We inform the participants of the name of the target domain and six example images of the target domain as a visual illustration. Design
[Table]
FromRGB 3 −→ ch/ch/
[Table] BlockDown ch/ 2 −→ ch ch
[Table] FromRGB 3 −→ ch ch ×
[Table]
BlockDown ch −→ 2 ch 2 ch × 64 ×
BlockDown 2 ch −→ 4 ch 4 ch × 32 ×
[Table]
ch −→ [Table] ch 4 ch × 16 × BlockDown
[Table] ch −→ [Table] ch 4 ch ×
[Table] × [Table] ch 4 ch
[Table] × [Table] ch
[Table] Flatten & Linear resolution from 128 to 256 , the gray-shadowed FromRGB is abandoned and two new blocks above that are added. And so on for other resolution transitions.
A. A RCHITECTURES AND SETTINGS
The architecture detail up to 256 × 256 resolution is given in Table 6.
Each BlockDown contains the 2-layer convolution operations where the ﬁrst layer won’t change feature size whereas the second layer comes with an average pooling 2 × 2 for down-sampling.
The kernel size is 3 × 3 with stride 1.
The last Block (without downsampling) ﬁrst uses a 3 × 3 kernel and then uses a 4 × 4 kernel.
The residual connection used in BlockDown is a single convolution and an average pooling of size 2 × 2 .
We don’t apply the residual connection in the last Block .
The stride is set to be 1 in all convolutions.
FromRGB is a block for transforming RGB images to feature maps.
Thus, in Section 3, the Primal ( · ) consists of a FromRGB and a BlockDown while the Fade ( · ) consists of an average pooling operation and a FromRGB block.
We train the network using Adam optimizer with β 1 = 0 .
5 , β 2 = 0 .
999 and (cid:15) =
10 − 8 .
We set the learning rate schedule as α = { 8 : 0 .
0012 , 128 : 0 . 0015 } and the data feeding schedule as N = { 8 : 50K , 16 : 75K , 32 : 100K , 64 : 125K , 128 : 150K } .
In terms of the sampling hyperparameters, we set a schedule regrading the number of steps T = { 8 : 15 , 16 : 30 , 32 : 50 , 64 : 50 , 128 : 60 } , step size 1 . 0 and the standard deviation of an additive noise η (cid:15)t = 2 e − 2 − 2 e − 2 / ( T − t + 1) in most experiments.
As most experiments were implemented on the less powerful TITAN X (12 GB), we don’t sweep over these values and hyper-parameters are very likely not to be optimal due to limited computational resources.
Nonetheless, the presentation of experimental results is adequate to validate the proposed approach.
CelebA experiment We preprocess CelebA to produce CelebA-64 in two settings.
Each image is ﬁrstly center-cropped into 140 × 140 and then resized to 64 × 64 .
In Table 2, three baselines
NCSN (Song & Ermon, 2019), mr-Langevin (Block et al., 2020), and WGAN-GP (Lee et al., 2018b) applied this setting.
The FID computation protocol is also based on NCSN where the score is calculated between 10k samples and all test images.
Each image is directly resized to 64 × 64 .
FID is computed on 40k generated samples.
In Table 2, VAE, DCGAN, and short-run EBM exactly follow this setting.
A. M ORE IMAGE SYNTHESIS RESULTS
Figure 8 shows the generated images of which the EBM is modeled on CIFAR-10 and ImageNet-1K ( 32 × 32 ).
Table 7 describes the quantitative results of image generation on ImageNet-1k from which
CF-EBM is observed to achieve competitive results with a much smaller model.
In Figure 9, we visualize the short-run MCMC initialized from uniform noises for CelebA. Figureshows the generated samples and their nearest neighbors from the training data.
As is shown, the synthesized images are not present in the training data.
Figure 11 illustrates the diverse image completions on CelebA- 64 2 test data.
It demonstrates that our model can generalize well to unseen test data modes.
It also learns meaningful representations which captures different facial details and expressions, e [Table] g. eyes, smiles, eyebrows and beard, which allows to produce diverse images completions.
[Table]
A.3. T HE EFFECT OF DATA PERTURBATION
As almost all EBM related generative models add Gaussian noise N (0 , σ 2 I ) to perturb the data for the purpose of training stabilization, it is necessary to investigate the effect of such perturbation.
Nijkamp et al.
(2019) has already reported that a larger perturbation leads to lower FID score where the minimum noise standard deviation is σ = 0 .
Also, the work Song & Ermon (2019) applies a sequence of decayed perturbation from σ L = 1 . 0 to σ 1 = 0 .
We claim that the model will memorize the leaked noisy information even if the ﬁnal noise is small.
As a result, the synthesized samples are noisy and foggy.
Figure 12 shows the visualization effect at different levels of perturbation on the observed data.
Note that, even the noise as small as σ = 0 .
01 can cause apparent visualization difference.
Moreover, the noise term for sampling is usually not decayed to 0 such that each sample is noisy in the continuous image space.
From Figure 2 in Nijkamp et al.
(2019), Figure 1 in Song &
Ermon (2020) and Figure 2 in Grathwohl et al.
(2020), we observe the perturbation will be ﬁnally reﬂected in the generated samples.
Therefore, the data perturbation should be taken as the last trick for generative modeling, or be discarded.
A. A BLATIONS
We mainly examine the activation functions, normalization and layer connections to see the model performance differences.
A.4. A CTIVATIONS
We show formulas of the following ﬁve activations and also give their derivatives:
• Rectiﬁed Linear Unit (ReLU) (Nair & Hinton, 2010): f ( x ) = max(0 , x ) , and the derivative: ∂f
[Equation]
• Leaky Rectiﬁed Linear Unit (LeakyReLU) (Maas et al., 2013): f ( x ) = max(0 , x ) + α · min(0 , x ) , and the derivative:
[Equation]
where α = 0 .
2 is the common setting.
• Continuously Differential Exponential Linear Unit (CELU) (Barron, 2017): f ( x ) = max(0 , x ) + min(0 , α · (exp( x/α )
[Equation]
1) , and the derivative:
[Equation]
• Swish
(Ramachandran et al., 2016):
f ( x ) = x · Sigmoid ( x ) , and the derivative is:
[Equation]
• Gaussian Error Linear Unit (GELU
[Equation]
Hendrycks & Gimpel
[Equation]
x
[Equation]
where Φ( · ) is the cumulative distribution function for Gaussian distribution.
It is successfully applied in natural language processing like BERT.
The derivative can be approximated as:
[Equation]
Figure 13 visualizes above activations and their derivatives.
Obviously, both ReLU and LeakyReLU are non-smooth around 0.
Image synthesis results on CIFAR-10 are shown in Table 8.
We ﬁnd that the use of LeakyReLU and ReLU diverge at an early time.
These phenomenons are also observed on
CelebA experiment.
Therefore, smooth activations indeed improve the training stability and sample quality of EBM.
[Table]
FID
ReLU
[Equation]
LeakyReLU
[Table]
GELU
[Table]
[Equation]
[Table]
A.4. N ORMALIZATION
Since the objective in Eq.
(2) is similar to that in Wasserstein GAN, and the EBM acts as a discriminator, we thus naturally only consider spectral normalization (Miyato et al., 2018) to improve the performance and stability.
SN constrains the Lipschitz constant of the learnable neural network parameters, which is widely used to stabilize the training of the discriminator network in GAN.
However, batch normalization is not appropriate here.
Because each update of Langevin dynamics will rely on the running mean/variance of x ∼ p data .
Even if the statistic is not updated in Eq.
(3), the distribution of p θ ( x t ) is changing all the time along a chain.
As a result, the pre-computed statistics of batch normalization don’t ﬁt each x t .
Other normalizations, like instance normalization and layer normalization, are not appropriate in a network of which the target is to discriminate two distributions.
A.4. A BLATE THE COMPONENTS
We ﬁrst examine how each component affects the model performances, including progressive growing, residual-connections, and spectral normalization.
Results are shown in Table 9.
From (a)-(c), we ﬁnd that spectral normalization and residual connection improved the sample quality.
(d) Besides, we also compare the time efﬁciency regarding the training with and without coarse-to-ﬁne settings.
It is observed that the model without coarse-to-ﬁne learning requires approximately 3 × more time than the counterpart.
(e) When the coarse EBM is ﬁxed during training higher resolution, the FID is worse than that of a single ﬁne EBM.
Conﬁguration
FID
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Table]
[Equation]
[Table] [Table]
[Equation]
[Table] [Table]
A. E VALUATIONS ABOUT LIKELIHOOD
We implement an experiment on the continuous MNIST following the setting in (Du & Mordatch, 2019).
Its open source implementation is at https://github.com/openai/ebm_code_ release .
It applies Annealed Importance Sampling (AIS)
(Neal, 2001) to obtain a bound of partition function.
Figure 14 illustrates the generated samples on MNIST.
As can be seen, our model can produce more realistic digits.
Table 11 shows the log likelihood comparison across three models.
[Table] [Table]
A. S CALE U P CF-EBM S
To scale up the EBM generative model to be capable of sampling higher resolution images, we integrate intermediate CF-EBMs that have been learned in Algorithm 1.
When the training enters into modeling higher resolution, we ﬁx the coarse-level CF-EBM, which only produces initial points for ﬁne-level CF-EBM samplers.
In this scenario, the number of Langevin steps can be reduced to 15 to expedite sampling at each scale, meaning that generating a 256 × 256 images requires 90 steps.
On a single TITAN X GPU (12GB), the total training time (200k iterations) of CelebA-HQ 128 × 128 and 256 × 256 are about 120 hours and 235 hours, respectively.
On a single TITAN V100 GPU, these costs can be reduced to 55 hours and 100 hours (we tested it once).
In this experiment, we ﬁx the initial noise variance η (cid:15)
= 0 .
We observe that, in the testing phase, choosing a different value of η (cid:15) 0 largely affects the synthetic results, as shown in Figure 15.
A larger η (cid:15) 0 tends to add details to look more realistic but have more noisy images.
Figure 16 shows more generated samples on CelebA-HQ 256 × 256 .
In this experiment, we set the channel multiplier atand train it for 5 days on a single GPU.
Figure 17 shows generated samples on CelebA-HQ 512 ×where we set the channel multiplier at 48 and the model is trained for 10 days on a single GPU.
Table 10 compares the FID of different models on CelebA-HQ 128 × 128 .
Note that our approach obtains competitive results without any regularization term.
A. U NPAIRED I MAGE - TO -I MAGE TRANSLATION A.7. D ATASET
In this paper, we experiment on ﬁve popular image-to-image translation datasets: selﬁe2anime It is ﬁrst introduced in Kim et al.
The selﬁe and anime datasets each contains 3400 training images and 100 testing images.
photo2vangogh and apple2orange These datasets are used in Zhu et al.
The training dataset size of each class: 6,287 (photo), 400 (vangogh), 995 (apple) and 1019 (navel orange).
The test datasets consist of 751 (photo), 400 (vangogh), 266 (apple) and 248 (navel orange).
Yosemite summer2winter This dataset is used in Zhu et al.
The training size of each class: 1273 (summer) and 854 ( winter).
The testing size of each class: 309 (summer) and 238 (winter).
cat2dog These datasets are used in DRIT (Lee et al., 2018a).
The numbers of data for each class are 871 (cat) and 1,364 (dog).
Follow (Kim et al., 2020), we use 100 (cat) and 100 (dog) as test data.
Experimental settings For the unpaired image-to-image translation task, we only made two changes compared with the settings used in image generation: ( i ) the channel multiplier is ch = 16 ; ( ii ) the learning rate is multiplied by 10 only for dog → cat .
A.7. C OMPARE WITH CUT
We implement CUT based on its open-source code https://github.com/taesungp/ contrastive-unpaired-translation .
We then pick up the most realistic translation results from CUT on cat → dog.
In Figure 18, we compare them with the proposed CF-EBM.
As can be seen, CUT sometimes hallucinates a tongue, resulting in unfaithful translations.
This observation is also mentioned in the original paper (Park et al., 2020).
In contrast, CF-EBM returns better and more faithful translations mostly.
[Figure]