Making Text-to-Image Diffusion Models Zero-Shot Image-to-Image Editors by Inferring “Random Seeds”
Abstract
Recent text-to-image diffusion models trained on large-scale data achieve remarkable performance on text-conditioned image synthesis (e.g., GLIDE, DALL · E-2,
Imagen, Stable Diffusion).
This paper introduces a simple method to use stochastic text-to-image diffusion models as zero-shot image editors.
Our method, CycleDiffusion , is based on the ﬁnding that when all random variables (or “random seed”) are ﬁxed, two similar text prompts will produce similar images.
The core of our idea is to infer the random variables that are likely to generate a source image conditioned on a source text.
With the inferred random variables, the text-to-image diffusion model then generates a target image conditioned a target text.
Our experiments show that CycleDiffusion outperforms SDEdit and the ODE-based DDIB method, and it can be further improved by Cross Attention Control.
Introduction
It has been observed that given a ﬁxed random seed, a text-to-image diffusion model will generate similar images when conditioned on two similar text prompts.
In this paper, we show how to exploit this ﬁnding to make stochastic text-to-image diffusion models zero-shot image-to-image editors, without any assumptions on the model architecture
We ﬁrst formalize “random seeds” by deﬁning the Gaussian latent codes z of stochastic diffusion models (as opposed to the ODE-based deterministic ones) as the concatenation of all Gaussian noises in the denoising process.
We then propose DPM-Encoder to infer z from a real image x .
Finally, we propose CycleDiffusion, a method for zero-shot image editing.
Given a pretrained text-to-image diffusion model, CycleDiffusion requires no ﬁnetuning to achieve this task.
It ﬁrst encodes the source image x as z using DPM-Encoder, conditioned on the source text t ; then it decodes z into the target image ˆ x with the text-to-image diffusion model conditioned on the target text ˆ t .
Our experiments compared CycleDiffusion with SDEdit [6] and the ODE-based DDIB [16].
We simulated hyperparameter search and random trials for each sample , which is quite common for text-to-image diffusion models, and used the directional CLIP score as the criterion.
Results show that CycleDiffusion outperforms SDEdit and DDIB.
We also demonstrate that CycleDiffusion can be combined with the Cross Attention Control [2] to further preserve the image structure.
Method 2.1 Formalizing “Random Seeds” of Stochastic Diffusion Models
We ﬁrst formalize the “random seeds” of stochastic diffusion probabilistic models (DPMs), such as
[3], non-deterministic DDIMs [14], and score-based SDEs [15].
These models generate
Output images with a Markov chain structure.
Given x T ∼ N ( 0 , I ) , the image x := x 0 is generated through x t − 1 ∼ N ( µ T ( x t , t ) , diag( σ 2 t )) .
We can deﬁne the latent code z and the mapping x = G ( z ) as
[Equation]
where ⊕ is concatenation.
Here, z has dimension d = d I × ( T +1) , where d I is the image dimension.
2.2 DPM-Encoder: an Encoder for Diffusion Models
To edit real images, we propose DPM-Encoder to infer z from the image x .
For each image x := x 0 , a stochastic DPM have a predeﬁned posterior q ( x 1: T | x 0 ) [3, 14].
Based on q ( x 1: T | x 0 ) and Eq.
(1), we can directly derive z := (cid:0) x T ⊕ (cid:15) T ⊕ · · · ⊕ (cid:15) 2 ⊕ (cid:15) 1 (cid:1) ∼ DPMEnc( z | x , G ) as x 1 , . . .
, x T
[Equation] [Equation]
T ∼
[Equation]
2.3 CycleDiffusion: Image-to-Image Translation with DPM-Encoder
Since DPM-Encoder can infer z from a given real image x , we use it to build our zero-shot image-toimage translation method CycleDiffusion.
Let G t be a text-to-image diffusion model conditioned on text t .
The task input is a source image x , a source text t describing the source image x , and a target text ˆ t describing the target image ˆ x to be generated.
Like the GAN-based UNIT method [5],
CycleDiffusion encodes x as z with DPM-Encoder and decodes it as ˆ x = G ˆ t ( z ) .
Formally, we have
[Equation]
Inspired by the realism-faithfulness tradeoff in SDEdit [6], we can truncate z towards a speciﬁed encoding step T es ≤ T .
The full algorithm of CycleDiffusion with truncation is shown in Algorithm 1.
An analysis for image similarity with ﬁxed z .
This part analyzes how the ﬁxed z helps bound image distances.
Suppose the text-to-image model has the following two properties:
Conditioned on the same text, similar noisy images lead to similar enough mean predictions.
Formally, µ T ( x t , t | t ) is K t -Lipschitz, i.e., (cid:107) µ T ( x t , t | t ) − µ T (ˆ x t , t | t ) (cid:107) ≤ K t (cid:107) x
t − ˆ x t (cid:107) .
Given the same image, the two texts lead to similar predictions.
Formally, (cid:107) µ T
(ˆ x t , t | t ) − µ T (ˆ x t , t | ˆ t ) (cid:107) ≤ S t .
Intuitively, a smaller difference between t and ˆ t gives us a smaller S t .
Let B t be the upper bound of (cid:107) x t − ˆ x t (cid:107) 2 at time step t when the same latent code z is used for sampling (i.e., x 0 =
G t ( z ) and ˆ x 0 =
G ˆ t ( z ) ).
We have B sT = 0 because (cid:107) x
T − ˆ x t (cid:107) 2 = 0 , and
B s 0 is the upper bound for the generated images (cid:107) x − ˆ x (cid:107) 2 .
The upper bound B t can be propagated through time, from T to 0 .
Speciﬁcally, by combining the above two properties, we have
[Equation]
Experiments
This section provides experiments for zero-shot image-to-image translation.
Following Section 2.3, we curated a set of 150 triplets ( x , t , ˆ t ) for this task, where x is the source image, t is the source text (e.g., “an aerial view of autumn scene.”
in Figure 1), and ˆ t is the target text (e.g., “an aerial view of winter scene.”).
The target image to be generated is denoted as ˆ x .
Metrics: To evaluate the faithfulness to source images, we reported PSNR and SSIM.
To evaluate the authenticity to the target text, we used the CLIP score, i.e., the cosine similarity between CLIP embeddings: S CLIP (ˆ x | ˆ t ) = cos (cid:10) CLIP img (ˆ x ) , CLIP text (ˆ t ) (cid:11) .
We also reported directional CLIP score [8], i.e., the cosine similarity between the differences of CLIP embeddings:
[Equation]
Baselines: The baselines include SDEdit [6] and DDIB [16].
We used the same hyperparameters (and hyperparameter trials) for the baselines and CycleDiffusion whenever possible (see Appendix A).
Pre-trained text-to- diffusion models: We experimented
45B
Diffusion
[Table]
↑
S D
[Table]
CLIP ↑
PSNR ↑
SSIM ↑
[Table]
[Table]
DDIB [16] 0. CycleDiffusion w/ DDIM ( η = 0 . 1 ; ours)
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table]
w/ DDIM ( η = 0 . 1 ; ours) 0.
Results: Table
shows the results for zero-shot image-to-image translation.
CycleDiffusion excels at being faithful to the source image (i.e., PSNR and SSIM); by contrast, SDEdit and DDIB have comparable authenticity to the target text (i.e., S CLIP ), but their outputs are much less faithful.
For all methods, we ﬁnd that the pre-trained weights SD-v1-1 and SD-v1-4 have better faithfulness than LDM-400M .
Figure 1 provides samples from CycleDiffusion, demonstrating that CycleDiffusion achieves meaningful edits that span (1) replacing objects, (2) adding objects, (3) changing styles, and (4) modifying attributes.
See Figure 3 (Appendix B) for qualitative comparisons with the baselines.
CycleDiffusion + Cross Attention Control:
Besides ﬁxing the random seed, [2] shows that ﬁxing the cross attention map (this operation is called Cross Attention Control, or CAC) also improves the similarity between synthesized images.
CAC is applicable to CycleDiffusion: in Algorithm 1, we can apply the attention map of µ T ( x t , t | t ) to µ T (ˆ x t , t | ˆ t ) .
However, we cannot apply it to all samples because CAC puts requirements on t and ˆ t (i.e., the target text is a subsequence of the source one, or the two texts differ in only one token [2]).
Figure 2 shows that CAC helps CycleDiffusion when the intended structural change is small.
For instance, when the intended change is color but not shape (left), CAC helps CycleDiffusion preserve the background; when the intended change is horse → elephant, CAC makes the generated elephant to look more like a horse in shape.
Related Work
Several recent methods for text-to-image synthesis are built upon diffusion probabilistic models (DPMs), such as GLIDE [7], DALL · E 2 [9], Imagen [11], Stable Diffusion [10].
It has been observed that when using the same random seed, text-to-image DPMs tend to generate similar images given two similar text prompts.
It holds for both stochastic DPMs [3, 14] and deterministic DPMs [14, 15].
Based on the ﬁnding for deterministic DPMs, [16] proposed dual diffusion implicit bridge (DDIB) for unpaired image translation, and our CycleDiffusion is an extension of it to stochastic DPMs.
Besides random seeds, a concurrent work [2] found that ﬁxing the cross-attention map in Transformerbased text-to-image DPMs further improves the similarity between images.
To ﬁx the cross-attention map for two text prompts, they proposed Cross Attention Control (CAC).
To edit real images, they combined CAC with DDIB [16] or mask heuristics because “how to infer random seeds” for stochastic
DPMs is non-trivial.
In our experiments, we show that CAC can be applied to CycleDiffusion to improve the structural preservation of the image.
Two concurrent works, Imagic [4] and UniTune [17], showed that zero-shot image editing can also be achieved by ﬁnetuning the text-to-image DPM on the source image to be edited.
Different from these two works, our CycleDiffusion is an optimization-free method that does not need ﬁnetuning the large text-to-image DPMs.
Similar optimization-free methods include the Cross Attention Control [2] discussed above and DiffEdit [1], which automatically infers a mask before editing.
Conclusions
This paper proposes CycleDiffusion, a simple method for zero-shot image editing with pretrained text-to-image DPMs without optimization.
Different from several works that use deterministic DPMs for this task, CycleDiffusion shows that stochastic DPMs can also be used for zero-shot image editing.
Besides the promising results, we also analyze how the ﬁxed z helps bound image distances.
However, an analysis on needs further exploration.
A Experimental Details
CycleDiffusion and the baselines for zero-shot image-to-image translation have some shared hyperparameters; each method also has its own unique hyperparameters.
For both CycleDiffusion and the baselines, we can enumerate some of these parameters and select the best per sample based on a certain criterion.
Moreover, for stochastic methods such as SDEdit and our CycleDiffusion, one may run several trials.
In this section, we provide these details.
Per sample selection criterion: For each test sample, we allow each method to enumerate some combinations of hyperparameters (detailed below).
To select the best combination for each sample, we used the directional CLIP score S D-CLIP as the criterion (higher is better).
DDIB: Since DDIB only applies to deterministic DPMs, we used the deterministic DDIM sampler with 100 steps.
We set the classiﬁer-free guidance of the encoding step as 1 ; we enumerated the classiﬁer-free guidance of the decoding step as { 1 , 1 . 5 , 2 , 3 , 4 , 5 } .
SDEdit: For SDEdit, we used the DDIM sampler ( η = 0 . 1 ) with 100 steps.
We enumerated the classiﬁer-free guidance of the decoding step as { 1 , 1 . 5 , 2 , 3 , 4 , 5 } ; we enumerated the SDEdit step as { 15 , 20 , 25 , 30 , 40 , 50 } ; we ran 15 trials for each hyperparameter combination.
CycleDiffusion:
For our CycleDiffusion, we used the DDIM sampler ( η = 0 . 1 ) with 100 steps.
We set the classiﬁer-free guidance of the encoding step as 1 ; we enumerated the classiﬁer-free guidance of the decoding step as { 1 , 1 . 5 , 2 , 3 , 4 , 5 } ; we enumerated the early stopping step T es as { 15 , 20 , 25 , 30 , 40 , 50 } ; we ran 15 trials for each hyperparameter combination.
B Additional Results for Zero-Shot Image-to-Image Translation
Figure 3 provides a qualitative comparison for zero-shot image-to-image translation.
Compared with
SDEdit and the ODE-based DDIB, CycleDiffusion improves the faithfulness to the source image.