ST-DDPM: E XPLORE C LASS C LUSTERING FOR C ONDITIONAL D IFFUSION P ROBABILISTIC M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Score-based generative models involve sequentially corrupting the data distribution with noise and then learns to recover the data distribution based on score matching.
In this paper, for the diffusion probabilistic models, we ﬁrst delve into the changes of data distribution during the forward process of the Markov chain and explore the class clustering phenomenon.
Inspired by the class clustering phenomenon, we devise a novel conditional diffusion probabilistic model by explicitly modeling the class center in the forward and reverse process and make an elegant modiﬁcation to the original formulation, which enables controllable generation and gets interpretability.
We also provide another direction for faster sampling and more analysis of our method.
To verify the effectiveness of the formulated framework, we conduct extensive experiments on multiple tasks and achieve competitive results compared with the state-of-the-art methods (conditional image generation on CIFAR-10 with an inception score of 9.58 and FID score of 3.05).
I NTRODUCTION
Deep generative models such as generative adversarial networks (GANs) Goodfellow et al.
Zhu et al.
; Brock et al.
, ﬂows Rezende & Mohamed (2015); Kingma & Dhariwal (2018);
Ho et al.
(2019), variational autoencoders (VAEs) Kingma & Welling (2013);
Maaløe et al.
(2019), score-based generative models Song & Ermon (2019);
Ho et al.
Song et al.
(2020b)
and
autoregressive models Van Oord et al.
; Oord et al.
; Salimans et al.
(2017) have been proposed to generate high quality samples in wide variety of data modalities.
The score-based generative models involve sequentially corruptig the data distribution with noise and then learns to recover the data distribution based on score matching (e.g. the gradient of the log probility density Song & Ermon (2019); Song et al.
(2020b), the noise corruption Ho et al.
Given the Markov chain that gradually corrupts the data with noise, in this paper, we ﬁrst delve into the changes of data distribution during the forward process and visualize these behaviours.
Speciﬁcally, we measure the class clustering by considering the intra-class to inter-class variance ratio at each time step.
Low values of the variance ratio demonstrate better class separation since the samples are concentrated around their corresponding class mean.
We notice that in the early stage of forward process, the data distribution maintains stable intra-class variance and inter-class variance, leading to a balanced variance ratio.
As the noise gradually increases, the class separation decreases sharply and the data distribution is ultimately converted into the noise distribution, as shown in Figure 1 (Left).
The class clustering phenomenon reveals the process of diffusion where the corruption noise ﬁrst corrupts the datapoints without increasing the variance ratio, and then greatly decreases the class separation and mixes the datapoints from different classes, which generates the noise distribution.
Inspired by the class clustering phenomenon, we formulate a novel conditional diffusion probabilistic model by explicitly modeling the class center in the forward and reverse process, which enables controllable generation.
Concretely, as shown in Figure 1 (Right), in the early stage of forward process, samples are gradually clustered to their corresponding class center while all the samples are ﬁnally gathered into the noise distribution as the diffusion progresses.
The reverse process is then modulated by the class center during generation:
T =
T =
T =
T =𝑨 " 𝑨 # 𝑩 " 𝑩 #
[Figure]
[Figure]
Noise By explicitly introducing the class center, we make an elegant modiﬁcation to the original formulation Ho et al.
and further decouple the module for conditional information encoding and the module for noise prediction.
The formulation that exploits the class center is much simpler since we don’t need to train a separate model or apply heuristics and domain knowledge to estimate the gradient of log likelihood Song et al.
(2020b).
The learned class centers, which get interpretability, also reﬂect the common characteristics of datapoints from the same class (e g. template face, number).
With the explicitly-modeled class centers, we can naturally speed up the reverse process with earlier starting, which is different from strided sampling Song et al.
(2020a).
Due to the guided “shift” from sample to its corresponding class center during both the forward and reverse process, we name the conditional DDPM as ST-DDPM.
To verify the effectiveness of the formulated framework ST-DDPM, we conduct extensive experiments on the task of conditional image generation, free-form image inpainting Yu et al.
Liu et al.
(2018), attribute-to-image synthesis Yan et al.
and text-to-image synthesis Reed et al.
; Xu et al.
; Zhang et al.
(2018), and achieve competitive results compared with the original formulation.
We also provide another direction for faster sampling and further explore the relationship between the guided class centers and other score-based methods.
R ELATED W ORK
Score-based Generative Models.
Score-based generative models Song & Ermon (2019)
; Ho et al.
(2020); Song et al.
(2020b)
; Tae et al. ; Kim et al. ; Tashiro et al. (2021) involve sequentially corruptig the data distribution with noise using Markov chains and then learns to convert the noise distribution to the data distribution based on score matching.
Score matching with Langevin dynamics (SMLD) Song & Ermon (2019) learns to estimate the gradient of the log probility density with respect to data at each time step and samples from a sequence of decreasing noise scales for generation.
Denoising diffusion probabilistic model (DDPM)
Ho et al.
(2020) trains the model with denoising score matching to predict the noise corruption at each step and then reverse the noise distribution during generation.
Song et al. Song et al.
(2020b) further propose a uniﬁed framework to generalize prior works by introducing the stochastic differential equations (SDEs).
There are other methods to learn the reverse process of Markov chains including infusion training Bordes et al.
(2017), variational walkback Goyal et al.
(2017), generative stochastic networks Alain et al.
and others Salimans et al.
; Song et al.
; Levy et al.
; Nijkamp et al.
Lawson et al.
D ENOISING D IFFUSION P ROBABILISTIC M ODELS
The DDPM Ho et al.
(2020) employs a forward diffusion Markov chain to step-by-step convert a real data distribution into a standard Gaussian distribution, and then build a parameterized Markov chain to model the reverse process.
Speciﬁcally, the forward process starts from x 0 which comes from the real data distribution q ( x 0 ) , and performs T steps of diffusion to generate x 1 , x 2 , · · · , x T sequentially.
The forward trajectory is given by
[Equation]
where β 1 , β 2 , · · · , β T correspond to aforementioned diffusion rate of each forward step and are ﬁxed as a variance schedule.
The reverse process is conducted by parameterized operators p θ ( x t
[Equation]
1 | x t ) : p
[Equation] [Equation]
where p ( x T ) =
N ( x T ; 0 , I ) .
The DDPM is trained to minimize a variational bound of negative log likelihood E q [ − log p θ ( x 0 )] , denoted by:
[Equation]
Rewrite L (3) by Bayes rule establishes a connection between p θ ( x t − 1 | x t ) and q ( x t − 1 | x t , x 0 ) for backward operators:
[Equation]
where the forward process posterior conditioned upon x 0 is related to the forward process distribution via Bayes rule, denoted by:
[Equation]
Fortunately, all right items have closed form expressions, which can exactly derive a Gaussian distribution, denoted by:
[Equation]
where α t = 1 − β t and ¯ α t = (cid:81) ti =
1 α i .
Consequently, the KL divergences between Gaussians in
L (4) can be computed analytically.
Further simplications and reparameterizations Ho et al.
(2020) come to the following variant of the variational bound:
[Equation]
M EASURING C LASS C LUSTERING IN RGB AND F EATURE S PACES
We ﬁrst delve into the changes of data distribution during the forward process and visualize these behaviours.
Concretely, we measure the class clustering in RGB and feature spaces by considering the intra-class to inter-class variance ratio at each time step, given by
[Equation]
[Figure] [Figure]
where C is the number of classes, N is the number of data points, x i,j is the j -th sample in class i , µ i is the mean of samples of class
i and µ is the mean across all the samples.
Low values of the variance ratio demonstrate better class separation.
We compute the variance ratio and visualize the data distribution on MNIST and CIFAR-10 dataset.
Note that for better visualization, the feature vectors for CIFAR-10 dataset are computed based on pre-trained ResNet18 backbone.
As shown in Figure 2 and Figure 3 given by t-SNE visualization, in the early stage of forward process, the data distribution maintain stable intra-class variance and inter-class variance, leading to a balanced class clustering (T approximately from 0 to 800).
As the noise gradually increases, the class separation decreases sharply and the data distribution is ultimately converted into the noise distribution (T = 1000).
The class clustering phenomenon inspires us to explicitly introduce the class center or class template during forward and reverse process for controllable generation.
T HE F ORMULATION OF ST-DDPM
Inspired by the class clustering phenomenon, we formulate the conditional diffusion probabilistic model ST-DDPM by explicitly modeling the condition as the class center in the forward and reverse process.
Concretely, we introduce the condition encoding network to obtain the class center or class template as u = u φ ( c ) , where c is the given condition (e.g. corrupted image, class label or attributes), and then rebuild the diffusion operators and modify the original recurrence relation Eq.
(1), given by:
q
[Equation]
x
[Equation]
u
[Equation]
x
[Equation]
14 t
[Equation] t [Equation] where we add a condition-related item for Gaussian mean to make the forward process has a “shift” toward the class center.
With the recurrence relation, we can derive the general x t sampling formula:
[Equation]
repeat
[Equation]
[Table]
x T ∼ N ( 0 , I ) for t =
T to 1 do
[Equation]
∼
[Equation] [Equation]
if
[Equation]
2 , else
[Equation]
=
[Equation]
return x
See Appendix A.1 for detailed derivation.
An intuitive explanation of the proposed diffusion operators and the weight-scheduling of u can be given as follows: Since ¯ α t is closer to 1 in the early stage, the class-guided forward process ﬁrst corrupts the sample while maintains the speciﬁcity of samples.
As the diffusion procedures, the samples from the same class gather at the class center.
As ¯ α t gets smaller and closer to 0 , the data distribution is ultimately converted into noise distribution.
In total, inspired by the class clustering, the well-designed weights ﬁrstly allow the speciﬁcity of datapoints, then gather datapoints from the same class for universality, and ﬁnally turn the distribution into pure noise by eliminating the impact of class means.
For convenience, we denote ¯ α 14 t (1 − α 14 t ) u
= m t and ¯ α 14 t (1 − ¯ α 14 t ) u
= n t .
Then the forward process posterior distribution q ( x t − 1 | x t , x 0 , u ) can be derived from probability density:
[Equation]
where
[Equation]
See Appendix A.2 for detailed derivation.
Then we can directly model parameterized µ θ ( x t , u , t ) to predict µ t ( x t , x 0 , u ) .
Similar to Ho et al.
(2020), we further reparameterize Eq.
(7) for (cid:15) t ∼ N ( 0 , I ) , then the forward process posterior mean becomes:
[Equation]
We then optimize the simpliﬁed loss to predict the corruption noise
[Equation] [Equation]
Since we explicitly model the class center during the forward process, from Eq.
(7), we have
[Equation]
where we can extract u out of parameters and use (cid:15) θ ( x t , t ) to predict x t −√ ¯ α t x 0 √ 1 − ¯ α t , and decouple the condition encoding module u φ and the denoising module (cid:15) θ , making the ﬁnal simpliﬁed loss
L simple ( φ, θ ) becomes:
[Equation]
P l a n e
C a r
B i r d
C a t
D ee r
D o g
F r o g
H
o r s e
S h i p
T
[Figure]
Algorithm 1 and Algorithm 2 separately display the complete training procedure with this simpliﬁed objective and sampling procedure with the proposed ST-DDPM.
E XPERIMENTS
To verify the effectiveness of the formulated framework ST-DDPM, we conduct extensive experiments on the task of conditional image generation, free-form image inpainting Yu et al.
Liu et al.
(2018) and text-to-image synthesis Reed et al.
; Xu et al.
; Zhang et al.
Following prior work Ho et al.
, we set T = 1000 and set the diffusion rate increasing linearly from β 1 = 10 − 4 to β T = 0 .
02 for all experiments.
For the denoising network (cid:15) θ , we use the same architecture as Ho et al.
(2020), which is a U-Net based on a Wide ResNet since we focus on the impact of our formulation.
More experimental details can be found in the Appendix B.
6. C LASS -C ONDITIONAL I MAGE G ENERATION
[Table]
For class-conditional image generation, the condition refers to an one-hot vector corresponding to a speciﬁc class label.
To consturct the condition encoding network u φ , we map the one-hot vector to an initialized embedding and employ stacked convolution and upsample layers to predict the class center.
Figure 4 (Left) presents the class-conditional synthesis results and the learned class centers on the 28 × 28 MNIST dataset.
As shown in the last row, the learned class centers also exhibit the shape of corresponding numbers, further indicating the class clustering phenomenon mentioned in the previous section.
Figure 4 (Right) presents the conditional synthesis results on the 32 × 32 CIFAR-10 Krizhevsky et al.
(2009) dataset, also verifying the capability and effectiveness of our formulated method for controllable generation.
The results of negative log likelihoods are presented in Appendix C and more examples can be found in the Appendix E.
To explore the sample quality, we compute the Inception scores and FID scores on CIFAR-10 dataset, as shown in Table 1.
For comparison, following Song et al.
(2020b), we introduce a time-dependent classiﬁer and compute the gradient for class-conditional sampling (grad. DDPM).
Also, we build a simple conditional DDPM (cond.
DDPM)
by directly injecting a class embedding along with the p l a n e h
o r s e d o g p l a n e c a r f r
o g s h
i
p b i r d h
o r s e
Image 𝜇 " Image 𝜇 "
Sample Sample
Sample
Sample 𝜇 " Image
Sample timestep embedding.
Details are given in the Appendix B. With the same denoising architecture as the unconditional DDPM Ho et al.
(2020), the proposed ST-DDPM even obtains better IS score of 9.58 and FID score of 3.05.
For class-conditional generation, our model achieves competitive results with the state-of-the-art methods, and better sample quality than most methods in the literature, indicating the effectiveness of the proposed method.
As shown in Figure 5, we further visualize the progressive diffusion and decoding of ST-DDPM modulated by different class centers.
The ST-DDPM explicitly models the class center during the forward and reverse process, making the class clustering more signiﬁcant.
Figure 5 (Left) presents the progressive diffusion where the class clustering phenomenon is marked by the red bounding box.
Figure 5 (Right) shows the progressive decoding toward different classes with the same initialized noise, qualitatively verifying the capability of ST-DDPM for controllable generation.
6. I MAGE I NPAINTING
Image inpainting Yu et al.
Liu et al.
(2018) aims to synthesize contents for the missing regions of a given image such that the result is visually realistic and semantically correct.
Different from class-conditional generation where the conditional encoding network is conﬁned to a limited and discrete space (i.e. the number of class labels), the condition space of image inpainting is continuous and non-enumerable.
To further prove the effectiveness of ST-DDPM that explicitly models the class center, we consider different images for imputation as different class and build a continuous conditional encoding network to generate the class center.
Concretely, we denote the corrupted image as the input condition and apply a U-Net based architecture to predict its corresponding class center, making the class centers non-enumerable.
The reverse process also starts from noise and is modulated by the predicted center.
Figure 7 separately presents the inpainting results on the 256 × 256 Place2 Zhou et al.
(2017), 256 ×
CelebA-HQ Liu et al.
(2015) and 256 × 256 LSUN church Yu et al.
(2015) dataset, which indicates the effectiveness and scalability of the ST-DDPM method.
More examples can be found in the
Appendix E. We also visualize the progressive diffusion decoding and the predicted class center in
Figure 6.
Notice that the decoupled condition encoding network has learned to predict the template face (i.e. u φ ) based on uncorrupted region, and the decoding process is modulated by the class center for inpainting, further demonstrating the effectiveness and interpretability of our proposed ST-DDPM.
The further evaluation results can be found in Appendix C.
[Figure]
6. A TTRIBUTE - TO -I MAGE S YNTHESIS
[Figure]
Attribute-to-image synthesis Yan et al.
(2016) requires to generate object image from high-level visual attributes (e.g. age, gender, lighting), which are also continuous values.
Similarly, we map the attribute vectors to class center (i.e. template face) by condition encoding network and leverage the proposed ST-DDPM formulation.
By interpolating between the minimum and maximum attribute value to modify the value progressively, we can further explore the capabilities of ST-DDPM for image synthesis in a continuous condition space.
Figure 8 presents the attribute-to-image synthesis results on the 64 × 64 LFW Huang et al.
(2008) dataset.
As shown in Figure 8 (Left), samples generated by progressive condition are visually consistent with attribute description, indicating that our method can effectively explore the continuous condition space.
Figure 8 (Right) visualizes the class centers predicted by the condition encoding network and the u φ also shows the shape of template face, demonstrating the interpretability of our method.
Since we decouple the condition encoding network and denoising network, we can further explore the condition space by interpretation of two different conditions, given by λ · u φ ( c 1 ) + (1 − λ ) · u φ ( c 2 ) .
Figure 9 shows the samples generated by interpolations on the condition space.
The interpretation smoothly introduces attributes from two different conditions, which can be extended to multi-condition generation in the future.
6. T EXT - TO -I MAGE S YNTHESIS
Text-to-image synthesis Reed et al.
; Xu et al.
; Zhang et al.
(2018) aims to generate realistic and text-consistent images according to the given natural language descriptions.
We extend the ST-DDPM to the more challenging cross-modal generation by mapping the encoded word
[Figure]
sequences to class centers (details in the appendix B).
Also, we obtain an Inception score of 4. and FID score of 18.18, demonstrating the effectiveness of our ST-DDPM method.
More generated examples and visualization results of class centers can be found in the Appendix E.
6. F ASTER D IFFUSION D ENOISING
Different from strided sampling Song et al.
(2020a), since we decouple the denoising network and condition encoding network and the diffusion process is modulated by the class center and datapoints from the same class gather earlier, as shown in Figure 5 (Left), we can devise a way for faster diffusion denoising that starts from earlier starting points x T = n T + (cid:15) , where (cid:15) ∼ N ( 0 , I ) (details in the appendix B).
By reducing the number of reverse steps, we enable faster sampling without harming quality as much as possible.
Figure 10 presents the speedup and generated examples on the 32 × 32 CIFAR-10 dataset with different time steps.
With T = 700 , our method can achieve a speedup of 1 .
43 ×
and the sample quality is slightly decreased.
With T = 900 , our method can achieve a speedup of 1 .
12 × without harming the sample quality.
Also, our technique of earlier starting (ES) can be combined with strided sampling (SS) and achieve a speedup of 11 .
1 × , and detailed derivation and more results are given in the Appendix D.
6. S CORE - BASED M ETHODS FOR CONTROLLABLE GENERATION
[Figure]
With bayes rule, Song et al.
(2020b) implement controllable generation by introducing gradient guidance, which is given by extra time-dependent classiﬁer or domain knowledge.
Computationally, an advantage of the proposed approach over the guidance strategy in Song et al.
(2020b) is that no backpropagation through a classiﬁer or other estimation is needed.
The class centers, the guidance of ST-DDPM, are predicted in a feed-forward manner and could be cached.
To further explore the relationship, we separately visualize the normalized guidance during the reverse process.
As shown in Figure 11, the learned class centers with weight-scheduling, as a guidance without extra estimation, show a similar trend with the gradient of classiﬁer during the reverse process.
Notice that the learned class centers also have better class separation compared with the unstable gradient guidance.
C ONCLUSION
In this paper, for the diffusion probabilistic models, we delve into the changes of data distribution during the forward process of the Markov chain and explore the class clustering phenomenon.
To enable controllable generation, inspired by the class clustering phenomenon, we devise a novel conditional diffusion probabilistic model ST-DDPM by explicitly modeling the class center in the forward and reverse process, and make an elegant modiﬁcation to the original formulation.
The learned class centers also reﬂect the common characteristics of datapoints from the same class (e.g. template face).
Extensive experiments verify the effectiveness and scalability of our method.
A E XTENDED D ERIVATIONS A. F ORWARD PROCESS GENERAL FORMULA DERIVATION
[Equation]
A.
[Equation]
For the reason that all these multivariate Gaussian distributions have diagonal covariance matrix
[Equation]
we can treat x and u as univariable
[Equation]
i
[Equation]
e. [Equation] each element of x follows following derivation.
By denoting ¯ α 14 t (1 − α 14 t ) u
= m t and ¯ α 14 t (1 − ¯ α 14 t ) u
= n t , we can derive q ( x t − 1 | x t , x 0 , u ) as follow:
[Equation]
The coefﬁcient of x 2 t − 1 as 1 − ¯ α t − 1 1 − ¯ α t α t + β t 1 − ¯ α t = α
[Equation]
− ¯ α t +1 − α t 1 − ¯ α t = 1 , so we try to transform E to the form of [ x t − 1 + ( a x t
+ b x 0 + c )] 2 .
We extract the coefﬁcient of x 2 t as a 2 :
[Equation]
We extract the coefﬁcient of x 20 as b 2 :
[Equation]
[Equation]
[Equation]
[Equation]
The coefﬁcient of x t − 1 x t is − 2 √
α t (1 − ¯
α t − 1 ) 1 − ¯ α t = 2 a , which means that a = − √ α t (1 − ¯
α t − 1 ) 1 − ¯ α t .
The coefﬁcient of x t − 1 x 0 is − 2 β t √ ¯ α t − 1 1 − ¯ α t = 2 b , which means that b = − β t √ ¯ α t − 1 1 − ¯
α t .
The coefﬁcient of x t x 0 is 2 √ ¯ α t β t (1 − ¯ α t − 1 ) (1 − ¯ α t )
[Equation]
t
[Equation]
, which is equal to 2 ab .
Note that m t = n t − √ α t n t − 1 :
[Equation]
The coefﬁcient of x t − 1 should be equal to 2 c :
[Equation] [Equation]
The coefﬁcient of x t should be 2 ac :
[Equation] [Equation]
[Equation] [Equation]
[Equation]
[Equation]
[Equation]
[Equation]
The coefﬁcient of x 0 should be 2 bc :
[Equation]
The remaining items should be equal to c 2 :
[Equation]
Finally we have a = − √ α t (1 − ¯ α t −
[Equation]
1 − ¯ α t ,
[Equation]
t n t − n
[Equation]
1 and
[Equation] [Equation]
Now we have the forward process posterior distribution as follows:
[Equation]
which is right a probability density function of some Gaussian distribution.
Then we have:
[Equation] [Equation]
B I MPLEMENTATION D ETAILS
Follows the backbone of PixelCNN++ Salimans et al.
(2017) and DDPM Ho et al.
(2020), our employ a U-Net architecture based on a Wide ResNet for noise prediction (cid:15) θ and replace the weight normalization Salimans & Kingma (2016) with a group normalization Wu & He (2018) to make the implementation simpler.
We use four feature map resolutions for 32 × 32 models, and six resolutions for 64 × 64 and 256 × 256 models.
All models have two convolutional residual blocks per resolution level and self-attention blocks at 16 × 16 resolution between convolutional blocks.
The time embedding is then speciﬁed by adding the Transformer Vaswani et al.
(2017) sinusoidal position embedding into each residual block.
To predict the class centers µ φ , we stack convolutional residual blocks and upsample layers to map feature vectors to three-channels map for CIFAR-10 and LFW dataset.
For image inpainting on
CelebA-HQ, LSUN-church, Place2 datasets, we also employ a U-Net architecture for pixel-to-pixel prediction.
For text-to-image synthesis on CUB bird dataset, we stack convolutional residual blocks and upsample layers with attention mechanism from the pre-trained word embeddings to predict the class center.
For image inpainting, we use the Irregular Mask Dataset collected by Liu et al.
(2018), which contains 55,116 irregular raw masks for training and 24,866 for testing.
During training, for each image in the batch, we ﬁrst randomly sample a mask from 55,116 training masks, then perform some random augmentations on the mask, ﬁnally we use it to mask the image and get our class center for training.
So the training masks are different all the time. The mask is irregular and may be 100During testing, we use 12,000 test masks sampled and augmented from 24,866 raw testing masks. These 12, masks are categorized by hole size according to hole-to-image area ratios (0-20%, 20-40%, 40-60%). Follow the training setting of DDPM Ho et al.
, we set T = 1000 and set the diffusion rate increasing linearly from β 1 = 10 − 4 to β T = 0 .
02 for all experiments.
The dropout rate is set to 0.1.
We use the Adam optimizer, and set the learning rate to 2 × 10 − 4 for 32 × 32 images and 2 × 10 −for higher resolution without any sweeping.
The batch size is set to 128 for CIFAR-10, 64 for LFW, and 24 for high-resolution Places2, LSUN-church, CUB-bird and Celea-HQ datasets.
We use EMA on model parameters with a decay factor of 0.9999.
We train the 32 × 32 models with four Nvidia RTX 2080Ti GPUs and higher resolution models with four A6000 GPUs. [Table] [Table]
For comparison [Table] following Song et al.
(2020b) ; Dhariwal & Nichol (2021) [Table] we introduce a timedependent classiﬁer and compute the gradient for class-conditional sampling (grad. DDPM), given by:
[Equation]
[Equation]
where φ is the parameters of the pretrained classiﬁer [Table] y is the class label.
Also, we build the simple conditional DDPM (cond.
DDPM)
by directly injecting a class embedding along with the timestep embedding into the denoising network, given by:
[Equation]
[Equation]
where c is the given condition (e
[Table]
g. class label).
C M ORE E VALUATION R ESULTS
We measure the negative log likelihoods (lossless codelengths) on CIFAR-10 dataset, given in Table 2.
Further, to evaluate the
, we
Yu
(2018) by reporting the FID score on Place2 dataset.
Table 3 shows the comparison results.
The
ST-DDPM achieves competitive results comparable to prior GAN-based methods for the FID score, showing the great potential of our method [Table] [Table]
D F ASTER S AMPLING WITH S TRIDED S AMPLING
Given the Eq.(7), we can get a transfer relation as follows:
[Equation]
As t increases, √ ¯ α t get closer to zero, leading to an approximate transfer relation as follows:
[Equation]
Therefore, we provide another thought for faster sampling that is to select a earlier starting timestep.
Also, inspired by DDIM Song et al.
(2020a) with strided sampling, we adapt our formulations into similar non-Markovian inference processes, which enables new and fast generative processes.
Speciﬁcally, we introduce a family of inference distribution indexed by σ ∈ R T ≥ 0 :
[Equation]
where
[Equation]
and for all t >
[Equation]
They are designed to ensure that q σ ( x t | x 0 ) = N ( x t
[Equation]
√ ¯ α [Equation] x
[Equation]
+
[Equation]
) for all t .
Then we employ a trainable generative process p θ ( x 0: T ) where each p θ ( x t − 1 | x t ) models the reverse conditional distribution q σ ( x t − 1 | x t , x 0 ) .
Speciﬁcally, for some x 0 ∼ q ( x 0 ) and (cid:15) t ∼ N ( 0 , I ) , we can obtain x t using Eq.
(7) and train a model (cid:15) θ to predict (cid:15) t from x t without x 0 .
We can then predict the denoised observation f θ ( x t , t ) with ˆ (cid:15) t :
[Equation]
Then we can deﬁne the generative process with a ﬁxed prior p θ ( x T ) =
N ( x T ; 0 , I ) and p θ ( x t
[Equation]
| x
[Equation]
= (cid:26) N
[Equation]
x
[Equation]
otherwise
[Equation]
[Equation]
from which one can generate a sample x t − 1 from x t via:
[Equation]
We do these modiﬁcations to ensure the optimal solution for ST-DDIM is also the same as that for
ST-DDPM, which enables us to adopt other forward processes with smaller steps for accelerated sampling without having to retrain the model.
We take the same settings with DDIM for τ and σ , where τ is a sub-sequences of [1 , · · · , T ] and σ τ i ( η ) =
η (cid:112) (1 − ¯
α τ i − 1 )(1 − ¯ α τ i ) (cid:113) 1 − ¯ α τi ¯ α τi − 1 .
η is a hyperparameter that we can directly control.
The evaluation results of FID on the CIFAR-10 dataset with strided sampling (SS) and earlier starting points (ES) with different η are presented in Table 5.
With earlier starting and strided sampling,our method can achieve a speedup of 11 .
1 × without signiﬁcant decreasing in sample quality.
Compared with existing works Watson et al.
; Song et al.
(2020a), our method can also achieve competitive evaluation results with the same number of steps.
Also, more generated examples with different η and sampling steps T are presented in Figure 12. [Table] η [Table]
η
[Table]
[Table] η =1. [Table] T=
T=
T= T= E A DDITIONAL S AMPLES Figure 13, 15, 16, 17, 18 present more generated examples separately on the CIFAR-10 Krizhevsky et al.
(2009), CelebA-HQ Liu et al.
, LSUN-church Yu et al.
, Place2 Zhou et al.
(2017) and LFW Huang et al.
(2008) dataset.
We also visualize the class center in the second column in
Figure 15, 16, 17.
Figure 14 shows the generated examples and class centers on the widely-used 256 × 256 CUB bird Wah et al.
(2011) dataset.
Notice that the class centers are also bird-like templates, demonstrating the effectiveness of our method.
[Figure]
This bird is blue with black and has a long, pointy beak.
This bird has a brown crown, a yellow belly, and a pointed bill.
This bird is completely black with a short blunt bill.
A bird with a large beak, brown wings and a white neck and belly.
A small sized bird that is mostly yellow that has a black chest marking
The small bird has a white belly, breast and through along with a short beak.
A very large bird with a yellow curved beak, and gray wings.
G r o und t r u t h
S TDD P
M
A small bird with a grey breast, short black bill and brownish red crown.
G r o und t r u t h
S TDD P
M
G r o und t r u t h
C e n t e r 𝜇 "
A
[Table]
and
[Table]
back and feathers
[Table]
bird has a long narrow beak.
The bird has a bright
[Table]
color on it's breast, belly, abdomen.
This bird has a yellow belly with brown wings and a brown crown.
This colorful bird has bright blue wings and tail along with a long, pointed beak.
𝜆 = 0.
𝜆 = 0. 𝜆 = 0.
𝜆 = 0. 𝜆 = 0.
𝜆 = 0. 𝜆 = 0.
𝜆 = 0.
𝜆 = 0.9 𝜇 / (𝑐 2 ) 𝜇 / (𝑐 4 )