U NIFYING D IFFUSION M ODELS ’ L ATENT S PACE , WITH A PPLICATIONS TO C YCLE D IFFUSION AND G UIDANCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Diffusion models have achieved unprecedented performance in generative modeling.
The commonly-adopted formulation of the latent code of diffusion models is a sequence of gradually denoised samples, as opposed to the simpler (e.g., Gaussian) latent space of GANs, VAEs, and normalizing ﬂows.
This paper provides an alternative, Gaussian formulation of the latent space of diffusion models, as well as a reconstructable DPM-Encoder that maps images into the latent space.
While our formulation is purely based on the deﬁnition of diffusion models, we demonstrate several intriguing consequences.
(1) Empirically, we observe that a common latent space emerges from two diffusion models trained independently on related domains.
In light of this ﬁnding, we propose CycleDiffusion , which uses DPM-Encoder for unpaired image-to-image translation.
Furthermore, applying CycleDiffusion to text-to-image diffusion models, we show that large-scale text-to-image diffusion models can be used as zero-shot image-to-image editors.
(2) One can guide pretrained diffusion models and GANs by controlling the latent codes in a uniﬁed, plug-and-play formulation based on energy-based models.
Using the CLIP model and a face recognition model as guidance, we demonstrate that diffusion models have better coverage of low-density sub-populations and individuals than GANs.
I NTRODUCTION
Diffusion models (Song & Ermon, 2019; Ho et al., 2020) have achieved unprecedented results in generative modeling and are instrumental to text-to-image models such as DALL · E 2 (Ramesh et al., 2022).
Unlike GANs (Goodfellow et al., 2014), VAEs (Kingma & Welling, 2014), and normalizing ﬂows (Dinh et al., 2015), which have a simple (e.g., Gaussian) latent space, the commonly-adopted formulation of the “latent code” of diffusion models is a sequence of gradually denoised images.
This formulation makes the prior distribution of the “latent code” data-dependent, deviating from the idea that generative models are mappings from simple noises to data (Goodfellow et al., 2014).
This paper provides a uniﬁed view of generative models of images by reformulating various diffusion models as deterministic maps from a Gaussian latent code z to an image x (Figure 1, Section 3.1).
A question that follows is encoding : how to map an image x to a latent code z .
Encoding has been studied for many generative models.
For instance, VAEs and normalizing ﬂows have encoders by design, GAN inversion (Xia et al., 2021) builds post hoc encoders for GANs, and deterministic diffusion probabilistic models (DPMs) (Song et al., 2021a;b) build encoders with forward ODEs.
However, it is still unclear how to build an encoder for stochastic DPMs such as DDPM (Ho et al., 2020), non-deterministic DDIM (Song et al., 2021a), and latent diffusion models (Rombach et al., 2022). We propose DPM-Encoder (Section 3.2), a reconstructable encoder for stochastic DPMs.
We show that some intriguing consequences emerge from our deﬁnition of the latent space of diffusion models and our DPM-Encoder.
First, observations have been made that, given two diffusion models, a ﬁxed “random seed” produces similar images (Nichol et al., 2022).
Under our formulation, we formalize “similar images” via an upper bound of image distances.
Since the deﬁned latent code contains all randomness during sampling, DPM-Encoder is similar-in-spirit to inferring the “random seed” from real images.
Based on this intuition and the upper bound of image distances, we propose CycleDiffusion (Section 3.3), a method for unpaired image-to-image translation using our DPM-Encoder.
Like the GAN-based UNIT method (Liu et al., 2017), CycleDiffusion encodes and decodes images using the common latent space.
Our experiments show that CycleDiffusion outperforms previous methods based on GANs or diffusion models (Section 4.1).
Furthermore, by applying large-scale text-to-image diffusion models (e.g., Stable Diffusion; Rombach et al., 2022) to CycleDiffusion, we obtain zero-shot image-to-image editors (Section 4.2).
With a simple latent prior, generative models can be guided in a plug-and-play manner by means of energy-based models (Nguyen et al., 2017; Nie et al., 2021; Wu et al., 2022).
Thus, our uniﬁcation allows uniﬁed, plug-and-play guidance for various diffusion models and GANs (Section 3.4), which avoids ﬁnetuning the guidance model on noisy images for diffusion models (Dhariwal & Nichol, 2021; Liu et al., 2021).
With the CLIP model and a face recognition model as guidance, we show that diffusion models have broader coverage of low-density sub-populations and individuals (Section 4.3).
R ELATED W ORK
Recent years have witnessed a great progress in generative models, such as GANs (Goodfellow et al., 2014), diffusion models (Song & Ermon, 2019; Ho et al., 2020; Dhariwal & Nichol, 2021), VAEs (Kingma & Welling, 2014), normalizing ﬂows (Dinh et al., 2015), and their hybrid extensions (Sinha et al., 2021; Vahdat et al., 2021; Zhang & Chen, 2021; Kim et al., 2022a).
Previous works have shown that their training objectives are related, e.g., diffusion models as VAEs (Ho et al., 2020; Kingma et al., 2021; Huang et al., 2021); GANs and VAEs as KL divergences (Hu et al., 2018) or mutual information with consistency constraints (Zhao et al., 2018); a recent attempt (Zhang et al., 2022b) has been made to unify several generative models as GFlowNets (Bengio et al., 2021).
In contrast, this paper uniﬁes generative models as deterministic mappings from Gaussian noises to data ( aka implicit models) once they are trained.
Generative models with non-Gaussian randomness (Davidson et al., 2018; Nachmani et al., 2021) can be uniﬁed as deterministic mappings in similar ways.
One of the most fundamental challenges in generative modeling is to design an encoder that is both computationally efﬁcient and invertible.
GAN inversion trains an encoder after GANs are pre-trained (Xia et al., 2021).
VAEs and normalizing ﬂows have their encoders by design.
Song et al.
(2021a;b) studied encoding for ODE-based deterministic diffusion probabilistic models (DPMs).
However, it remains unclear how to encode for general stochastic DPMs, and DPM-Encoder ﬁlls this gap.
Also,
CycleDiffusion can be seen as an extension of Su et al.
(2022)’s DDIB approach to stochastic DPMs.
Previous works have formulated plug-and-play guidance of generative models as latent-space energybased models (EBMs) (Nguyen et al., 2017; Nie et al., 2021; Wu et al., 2022), and our uniﬁcation makes it applicable to various diffusion models, which are effective for modeling images, audio (Kong et al., 2021), videos (Ho et al., 2022; Hoppe et al., 2022), molecules (Xu et al., 2022), 3D objects (Luo & Hu, 2021), and text (Li et al., 2022).
This plug-and-play guidance can provide principled, ﬁne-grained model comparisons of coverage of sub-populations and individuals on the same dataset.
A concurrent work observed that ﬁxing both (1) the random seed and (2) the cross-attention map in
Transformer-based text-to-image diffusion models results in images with minimal changes (Hertz et al., 2022).
The idea of ﬁxing the cross-attention map is named Cross Attention Control (CAC) in that work, which can be used to edit model-generated images when the random seed is known . For real images with stochastic DPMs, they generate masks based on the attention map because the random seed is unknown for real images. In Section 4.2, we show that CycleDiffusion and CAC can be combined to improve the structural preservation of image editing
[Table] [Table]
M ETHOD 3. G AUSSIAN L ATENT S PACE FOR D IFFUSION M ODELS
Generative models such as GANs, VAEs
[Table]
and normalizing ﬂows can be seen as a family of implicit models , meaning that they are deterministic maps G : R d !
X from latent codes z to images x .
At inference, sampling from the image prior x ⇠ p x ( x ) is implicitly deﬁned as z ⇠ p z ( z ) , x = G ( z ) .
The latent prior p z ( z ) is commonly chosen to be the isometric Gaussian distribution.
In this section, we show how to unify diffusion models into this family.
Overview is shown in Figure 1 and Table 1.
Stochastic DPMs: Stochastic DPMs (Ho et al., 2020; Song & Ermon, 2019; Song et al., 2021b;a; Watson et al., 2022) generate images with a Markov chain.
Given the mean estimator µ T (see Appendix A) and x T ⇠ N ( 0 , I ) , the image x := x 0 is generated through x t  1 ⇠ N ( µ T ( x t , t ) , diag(  2 t )) .
Using the reparameterization trick, we deﬁne the latent code z and the mapping G recursively as
[Equation]
where  is concatenation.
Here, z has dimension d =
d I ⇥ ( T +1) , where d I is the image dimension.
Deterministic DPMs: Deterministic DPMs (Song et al., 2021a;b; Salimans & Ho, 2022; Liu et al., 2022; Lu et al., 2022; Karras et al., 2022; Zhang & Chen, 2022) generate images with the ODE formulation.
Given the mean estimator µ T , deterministic DPMs generate x := x 0 via
[Equation]
Since backpropagation through Eq.
(2) is costly, we use fewer discretization steps T g when computing gradients.
Given the mean estimator µ T g with number of steps T g , the image x := x 0 is generated as (with gradients
[Equation]
Latent diffusion models (LDMs): An LDM (Rombach et al., 2022) ﬁrst uses a diffusion model
G latent to compute a “latent code” z 0 =
G latent ( z ) , 2 which is then decoded as x = D ( z 0 ) .
Note that
G latent is an abstraction of the diffusion models that are already uniﬁed above.
Diffusion autoencoder (DiffAE): DiffAE (Preechakul et al., 2022)
ﬁrst uses a deterministic DDIM to generate a “latent code” z 0 , 2 which is used as condition for an image-space deterministic DDIM:
[Equation]
DDGAN: DDGAN (Xiao et al., 2022) models each reverse time step t as a GAN conditioned on the output of the previous step.
We deﬁne the latent code z and generation process G of DDGAN as
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
In this section, we investigate the encoding problem, i.e., z ⇠ Enc( z | x , G ) .
The encoding problem has been studied for many generative models, and our contribution is DPM-Encoder , an encoder for stochastic DPMs. DPM-Encoder is deﬁned as follows.
For each image x := x 0 , stochastic DPMs deﬁne a posterior distribution q ( x 1: T | x 0 )
(Ho et al., 2020; Song et al., 2021a).
Based on q ( x 1: T | x 0 ) and Eq.
(1), we can directly derive z ⇠ DPMEnc( z | x , G ) as (see details in Appendices A and B)
[Equation]
A property of DPM-Encoder is [-achieved by ﬁrst using the LDM encoder E to encode z 
-]perfect reconstruction, meaning that we have x = G ( z ) for every z ⇠ Enc( z | x , G ) .
A proof by induction is provided in Appendix B.
3. C YCLE D IFFUSION : I MAGE - TO -I MAGE T RANSLATION WITH DPM-E NCODER
Given two stochastic DPMs G 1 and G 2 that model two distributions D 1 and D 2 , several researchers and practitioners have found that sampling with the same “random seed” leads to similar images (Nichol et al., 2022).
To formalize “similar images”, we provide an upper bound of image distances based on assumptions about the trained DPMs, shown at the end of this subsection.
Based on this ﬁnding, we propose a simple unpaired image-to-image translation method, CycleDiffusion.
Given a source image x 2 D 1 , we use DPM-Encoder to encode it as z and then decode it as ˆ x = G 2 ( z ) :
[Equation]
We can also apply CycleDiffusion to text-to-image diffusion models by deﬁning D 1 and D 2 as image distributions conditioned on two texts.
Let G t be a text-to-image diffusion model conditioned on text t .
Given a source image x , the user writes two texts: a source text t describing the source image x and a target text ˆ t describing the target image ˆ x to be generated.
We can then perform zero-shot image-to-image editing via (zero-shot means that the model has never been trained on image editing)
[Equation]
Inspired by the realism-faithfulness tradeoff in SDEdit (Meng et al., 2022), we can truncate z towards a speciﬁed encoding step T es  T .
The algorithm of CycleDiffusion is shown in Algorithm 1.
Why does a ﬁxed z leads to similar images?
We illustrate with text-to-image diffusion models.
Suppose the text-to-image model has the following two properties:
Conditioned on the same text, similar noisy images lead to similar enough mean predictions.
Formally, µ T ( x
t , t | t ) is K t -Lipschitz, i.e., k µ T
[Equation]
x
[Equation]
[Equation]
Given the same image, the two texts lead to similar predictions.
Formally, k µ T ( x t , t | t )  µ T ( x t , t | ˆ t ) k  S t .
Intuitively, a smaller difference between t and ˆ t gives us a smaller S t .
Let B t be the upper bound of k x t  ˆ x t k 2 at time step t when the same latent code z is used for sampling (i.e., x 0 = G t ( z ) and ˆ x 0 = G ˆ t ( z ) ). We have B sT = 0 because k x T  ˆ x t k 2 = 0 , and
B s 0 is the upper bound for the generated images k x  ˆ x k 2 .
The upper bound B t can be propagated through time, from T to 0 .
Speciﬁcally, by combining the above two properties, we have
[Equation]
3. U NIFIED P LUG - AND -P LAY G UIDANCE FOR G ENERATIVE M ODELS
Prior works showed that guidance for generative models can be achieved in the latent space (Nguyen et al., 2017; Nie et al., 2021; Wu et al., 2022).
Speciﬁcally, given a condition C , one can deﬁne the guided image distribution as an energy-based model (EBM): p ( x |C ) / p x ( x ) e   C E ( x |C ) .
Sampling for x ⇠ p ( x |C ) is equivalent to z ⇠ p z ( z |C ) , x = G ( z ) , where p ( z |C ) / p
z ( z ) e   C E ( G ( z ) |C ) .
Examples of the energy function E ( x |C ) are provided in Section 4.3.
To sample z ⇠ p ( z |C ) , one can use any model-agnostic samplers.
For example, Langevin dynamics (Welling & Teh, 2011) starts from z h
i ⇠ N ( 0 , I ) and samples z :=
z h n i iteratively through z
[Equation]
CycleDiffusion
[Table]
E XPERIMENTS
This section provides experimental validation of the proposed work.
Section 4.1 shows how CycleDiffusion achieves competitive results on unpaired image-to-image translation benchmarks.
Section 4. provides a protocol for what we call zero-shot image-to-image translation; CycleDiffusion outperforms several image-to-image translation baselines that we re-purposed for this new task.
Section 4.
shows how diffusion models and GANs can be guided in a uniﬁed, plug-and-play formulation.
4. C YCLE D IFFUSION FOR U NPAIRED I MAGE - TO -I MAGE T RANSLATION
Given two unaligned image domains, unpaired image-to-image translation aims at mapping images in one domain to the other.
We follow setups from previous works whenever possible, as detailed below.
Following previous work (Park et al., 2020; Zhao et al., 2022), we conducted experiments on the test set of AFHQ (Choi et al., 2020) with resolution 256 ⇥ 256 for Cat !
Dog and Wild !
Dog.
For each source image, each method should generate a target image with minimal changes. Since
CycleDiffusion sometimes generates noisy outputs, we used T sdedit steps of SDEdit for denoising.
When T = 1000 , we set T sdedit = 100 for Cat !
Dog and T sdedit = 125 for Wild ! Dog.
Metrics: To evaluate realism, we reported Frechet Inception Distance (FID; Heusel et al., 2017) and
Kernel Inception Distance (KID; Bi´nkowski et al., 2018) between the generated and target images.
To evaluate faithfulness, we reported Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity
Index Measure (SSIM; Wang et al., 2004) between each generated image and its source image.
Baselines: We compared CycleDiffusion with previous state-of-the-art unpaired image-to-image translation methods: CUT (Park et al., 2020), ILVR (Choi et al., 2021), SDEdit (Meng et al., 2022), and EGSDE (Zhao et al., 2022).
CUT is based on GAN, and the others use diffusion models.
Pre-trained diffusion models: ILVR, SDEdit, and EGSDE only need the diffusion model trained on the target domain, and we followed them to use the pre-trained model from Choi et al.
(2021) for
Dog.
CycleDiffusion needs diffusion models on both domains, so we trained them on Cat and Wild.
Seen in Table 2 are the results.
CycleDiffusion has the best realism (i.e., FID and KID).
There is a mismatch between the faithfulness metrics (i.e., PSNR and SSIM), and note that SSIM is much better correlated with human perception than PSNR (Wang et al., 2004).
Among all diffusion model-based methods, CycleDiffusion achieves the highest SSIM.
Figure 2 displays some image samples from
CycleDiffusion, showing that our method can change the domain while preserving local details such as the background, lighting, pose, and overall color the animal.
[Figure]
This section provides experiments for zero-shot image editing.
We curated a set of 150 tuples ( x , t , ˆ t ) for this task, where x is the source image, t is the source text (e.g., “an aerial view of autumn scene.”
in Figure 3 second row on the right)
, and ˆ t is the target text (e.g., “an aerial view of winter scene.”).
The generated image is denoted as ˆ x .
We also demonstrate that CycleDiffusion can be combined with the Cross Attention Control (Hertz et al., 2022) to further preserve the image structure.
Metrics: To evaluate faithfulness to the source image, we reported PSNR and SSIM.
To evaluate authenticity to the target text, we reported the CLIP score S CLIP (ˆ x | ˆ t ) =
cos ⌦
CLIP img (ˆ x ) , CLIP text (ˆ t ) ↵ , where the CLIP embeddings are normalized.
We note a trade-off between PSNR/SSIM and S CLIP : by copying the source image we get high PSNR/SSIM but low S CLIP , and by ignoring the source image (e.g., by directly generating images conditioned on the target text) we get high S CLIP but low
PSNR/SSIM.
To address this trade-off, we also reported the directional CLIP score (Patashnik et al., 2021)
(both CLIP embeddings and the embedding differences are normalized):
[Equation]
Baselines: The baselines include SDEdit (Meng et al., 2022) and DDIB (Su et al., 2022).
We used the same hyperparameters for the baselines and CycleDiffusion whenever possible (e.g., the number of diffusion steps, the strength of classiﬁer-free guidance; see Appendix C).
Pre-trained text-to-image diffusion models: To investigate how the zero-shot performance changes with data size, data quality, and training details, we used the following models: (1) LDM-400M, a 1.45B-parameter model trained on LAION-400M (Schuhmann et al., 2021), (2) SD-v1-1, a 0.98Bparameter Stable Diffusion model trained on LAION-5B (Schuhmann et al., 2022), (3) SD-v1-4, ﬁnetuned from SD-v1-1 for improved aesthetics and classiﬁer-free guidance.
{+Results: Table 
+}
shows the results for zero-shot image-to-image translation.
CycleDiffusion excels at being faithful to the source image (i.e., PSNR and SSIM); by contrast, SDEdit and DDIB have comparable authenticity to the target text (i.e., S CLIP ), but their outputs are much less faithful.
For all methods, we ﬁnd that the pre-trained weights SD-v1-1 and SD-v1-4 have better faithfulness than LDM-400M .
Figure 3 provides samples from CycleDiffusion, demonstrating that CycleDiffusion achieves meaningful edits that span (1) replacing objects, (2) adding objects, (3) changing styles, and (4) modifying attributes.
See Figure 7 (Appendix E) for qualitative comparisons with the baselines.
CycleDiffusion + Cross Attention Control:
Besides ﬁxing the random seed, Hertz et al.
(2022) shows that ﬁxing the cross attention map (i.e., Cross Attention Control, or CAC) further improves the similarity between synthesized images.
CAC is applicable to CycleDiffusion: in Algorithm 1, we can apply the attention map of µ T ( x t , t | t ) to µ T ( x t , t | ˆ t ) . However, we cannot apply it to all samples because CAC puts requirements on the difference between t and ˆ t . Figure 4 shows that CAC helps CycleDiffusion when the intended structural change is small.
For instance, when the intended change is color but not shape (left), CAC helps CycleDiffusion preserve the background; when the intended change is horse !
elephant, CAC makes the generated elephant to look more like a horse in shape.
[Table]
CycleDiffusion when the intended structural change is small.
For instance, when the intended change is color but not shape (left), CAC helps CycleDiffusion preserve the background; when the intended change is horse !
elephant, CAC makes the generated elephant look more like a horse in shape.
[Figure]
[Figure]
[Figure]
[Figure] 4. U NIFIED P LUG - AND -P LAY G UIDANCE FOR D IFFUSION M ODELS AND GAN S Previous methods for conditional sampling from ( aka guiding) diffusion models require training the guidance model on noisy images (Dhariwal & Nichol, 2021; Liu et al., 2021), which deviates from the idea of plug-and-play guidance by leveraging the simple latent prior of generative models (Nguyen et al., 2017).
In contrast, our deﬁnition of the Gaussian latent space of different diffusion models allows for uniﬁed plug-and-play guidance of diffusion models and GANs.
It facilitates principled comparisons over sub-populations and individuals when models are trained on the same dataset.
We used the text t to specify sub-population.
For instance, a photo of baby represents the baby subpopulation in the domain of human faces.
We instantiate the energy in Section 3.4 as E CLIP ( x | t ) =
[Equation]
, CLIP text
[Equation] ↵ ⌘ , where DiffAug l stands for differentiable augmentation (Zhao et al., 2020) that mitigates the adversarial effect, and we sample from the energybased distribution using Langevin dynamics in Eq.
(10) with n = 200 ,  = 0 .
We enumerated the guidance strength (i.e., the coefﬁcient  C in Section 3.4)  CLIP 2 { 100 , 300 , 500 , 700 , 1000 } .
For evaluation, we reported (1  E CLIP ( x | t )) averaged over 256 samples.
This metric quantiﬁes whether the sampled images are consistent with the speciﬁed text t .
Figure 5 plots models with pre-trained weights on FFHQ (Karras et al., 2019) (citations in Table 5, Appendix G).
In Figure 6, we visualize samples for SN-DDPM and DDGAN trained on CelebA.
We ﬁnd that diffusion models outperform 2D/3D GANs for complex text, and different models represent the same sub-population differently.
Broad coverage of individuals is an important aspect of the personalized use of generative models.
To analyze this coverage, we guide different models to generate images that are close to a reference x r in the identity (ID) space modeled by the IR-SE50 face embedding model (Deng et al., 2019), denoted as R .
Given an ID reference image x r , we instantiated the energy deﬁned in Section 3.
as E ID ( x | x r )
= 1  cos ⌦
R ( x ) , R ( x r ) ↵ with strength  ID =
2500 (i.e.,  C in Section 3.4).
For sampling, we used Langevin dynamics detailed in Eq.
(10) with n = 200 and  = 0 .
To measure
ID similarity to the reference image x r , we reported cos ⌦
R ( x ) , R ( x r ) ↵ , averaged over 256 samples.
In Table 4, we report the performance of StyleGAN2, StyleGAN-XL, GIRAFFE-HD, EG3D, LDMDDIM, DDGAN, and DiffAE.
DDGAN is trained on CelebAHQ, while others are trained on FFHQ.
We ﬁnd that diffusion models have much better coverage of individuals than 2D/3D GANs.
Among diffusion models, deterministic LDM-DDIM ( ⌘ = 0 ) achieves the best identity guidance performance.
We provide image samples of identity guidance in Figure 9 (Appendix G).
[Figure] [Figure]
[Table]
C ONCLUSIONS AND D ISCUSSION
This paper provides a uniﬁed view of pre-trained generative models by reformulating the latent space of diffusion models.
While this reformulation is purely deﬁnitional, we show that it allows us to use diffusion models in similar ways as CycleGANs (Zhu et al., 2017) and GANs.
Our CycleDiffusion achieves impressive performance on unpaired image-to-image translation (with two diffusion models trained on two domains independently) and zero-shot image-to-image translation (with text-to-image diffusion models).
Our deﬁnition of latent code also allows diffusion models to be guided in the same way as GANs (i.e., plug-and-play, without ﬁnetuning on noisy images), and results show that diffusion models have broader coverage of sub-populations and individuals than GANs.
Besides the interesting results, it is worth noting that this paper raised more questions than provided answers.
We have provided a formal analysis of the common latent space of stochastic DPMs via the bounded distance between images (Section 3.3), but it still needs further study.
Notably, Khrulkov & Oseledets (2022) and Su et al.
(2022) studied deterministic DPMs based on optimal transport.
Furthermore, efﬁcient plug-and-play guidance for stochastic DPMs on high-resolution images with many diffusion steps still remains open.
These topics can explored future studies.