Original: T RUNCATED D IFFUSION P ROBABILISTIC M ODELS AND D IFFUSION - BASED A DVERSARIAL A UTO -E NCODERS
Revised: T RUNCATED D IFFUSION P ROBABILISTIC M ODELS AND D IFFUSION - BASED A DVERSARIAL A UTO -E NCODERS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Employing a forward diffusion chain to gradually map the data to a noise distribution, diffusion-based generative models learn how to generate the data by inferring a reverse diffusion chain.
Revised: Employing a forward diffusion chain to gradually map the data to a noise distribution, diffusion-based generative models learn how to generate the data by inferring a reverse diffusion chain.

Original: However, this approach is slow and costly because it needs many forward and reverse steps.
Revised: However, this approach is slow and costly because it needs many forward and reverse steps.

Original: We propose a faster and cheaper approach that adds noise not until the data become pure random noise, but until they reach a hidden noisy-data distribution that we can confidently learn.
Revised: We propose a faster and cheaper approach that adds noise not until the data become pure random noise, but until they reach a hidden noisy-data distribution that we can confidently learn.

Original: Then, we use fewer reverse steps to generate data by starting from this hidden distribution that is made similar to the noisy data.
Revised: Then, we use fewer reverse steps to generate data by starting from this hidden distribution that is made similar to the noisy data.

Original: We reveal that the proposed model can be cast as an adversarial auto-encoder empowered by both the diffusion process and a learnable implicit prior.
Revised: We reveal that the proposed model can be cast as an adversarial auto-encoder empowered by both the diffusion process and a learnable implicit prior.

Original: Experimental results show even with a significantly smaller number of reverse diffusion steps, the proposed truncated diffusion probabilistic models can provide consistent improvements over the non-truncated ones in terms of performance in both unconditional and text-guided image generations
Revised: Experimental results show even with a significantly smaller number of reverse diffusion steps, the proposed truncated diffusion probabilistic models can provide consistent improvements over the non-truncated ones in terms of performance in both unconditional and text-guided image generations

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generating photo-realistic images with probabilistic models is a challenging and important task in machine learning and computer vision, with many potential applications in data augmentation, image editing, style transfer, etc .
Revised: Generating photo-realistic images with probabilistic models is a challenging and important task in machine learning and computer vision, with many potential applications in data augmentation, image editing, style transfer, etc .

Original: Recently, a new class of image generative models based on diffusion processes (Sohl-Dickstein et al., 2015) has achieved remarkable results on various commonly used image generation benchmarks (Song & Ermon, 2019; Ho et al., 2020; Song & Ermon, 2020; Song et al., 2021b; Dhariwal & Nichol, 2021), surpassing many existing deep generative models, such as autoregressive models (van den Oord et al., 2016), variational auto-encoders (VAEs) (Kingma &
Revised: Recently, a new class of image generative models based on diffusion processes (Sohl-Dickstein et al., 2015) has achieved remarkable results on various commonly used image generation benchmarks (Song & Ermon, 2019; Ho et al., 2020; Song & Ermon, 2020; Song et al., 2021b; Dhariwal & Nichol, 2021), surpassing many existing deep generative models, such as autoregressive models (van den Oord et al., 2016), variational auto-encoders (VAEs) (Kingma &

Original: Welling, 2013; Rezende et al., 2014; van den Oord et al., 2017; Razavi et al., 2019), and generative adversarial networks (GANs) (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017;
Revised: Welling, 2013; Rezende et al., 2014; van den Oord et al., 2017; Razavi et al., 2019), and generative adversarial networks (GANs) (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017;

Original: Miyato et al., 2018;
Revised: Miyato et al., 2018;

Original: Brock et al., 2019; Karras et al., 2019; 2020b).
Revised: Brock et al., 2019; Karras et al., 2019; 2020b).

Original: This new modeling class, which includes both score-based and diffusion-based generative models, uses noise injection to gradually corrupt the data distribution into a simple noise distribution that can be easily sampled from, and then uses a denoising network to reverse the noise injection to generate photo-realistic images.
Revised: This new modeling class, which includes both score-based and diffusion-based generative models, uses noise injection to gradually corrupt the data distribution into a simple noise distribution that can be easily sampled from, and then uses a denoising network to reverse the noise injection to generate photo-realistic images.

Original: From the perspective of score matching (Hyvärinen & Dayan, 2005;
Revised: From the perspective of score matching (Hyvärinen & Dayan, 2005;

Original: Vincent, 2011) and Langevin dynamics (Neal, 2011; Welling & Teh, 2011), the denoising network is trained by matching the score function, which is the gradient of the log-density of the data, of the corrupted data distribution and that of the generator distribution at different noise levels (Song & Ermon, 2019).
Revised: Vincent, 2011) and Langevin dynamics (Neal, 2011; Welling & Teh, 2011), the denoising network is trained by matching the score function, which is the gradient of the log-density of the data, of the corrupted data distribution and that of the generator distribution at different noise levels (Song & Ermon, 2019).

Original: This training objective can also be formulated under diffusion-based generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020).
Revised: This training objective can also be formulated under diffusion-based generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020).

Original: These two types of models have been further unified by Song et al.
Revised: These two types of models have been further unified by Song et al.

Original: (2021b)
Revised: (2021b)

Original: under the framework of discretized stochastic differential equations.
Revised: under the framework of discretized stochastic differential equations.

Original: Despite their impressive performance, diffusion-based (or score-based) generative models suffer from high computational cost, both in training and sampling.
Revised: Despite their impressive performance, diffusion-based (or score-based) generative models suffer from high computational costs, both in training and sampling.

Original: This is because they need to perform a large number of diffusion steps, typically hundreds or thousands, to ensure that the noise injection is small enough at each step to make the assumption that both  diffusion and denoising process have same Gaussian form hold in the limit of small diffusion rate (Feller, 1949; Sohl-Dickstein et al., 2015).
Revised: This is because they need to perform a large number of diffusion steps, typically hundreds or thousands, to ensure that the noise injection is small enough at each step to make the assumption that both the diffusion and denoising processes have the Gaussian form hold in the limit of small diffusion rate (Feller, 1949; Sohl-Dickstein et al., 2015).

Original: In other words, when the number of diffusion steps is small or when the rate is large, the Gaussian assumption may not hold well, and the model may not be able to capture the true score functionof the data.
Revised: In other words, when the number of diffusion steps is small or when the rate is large, the Gaussian assumption may not hold well, and the model may not be able to capture the true score function

Original: Therefore, previous works have tried to reduce the number of diffusion steps by using non-Markovian reverse processes (Song et al., 2020; Kong & Ping, 2021), adaptive noise scheduling

Original: [Figure]
Revised: [Figure]

Original:  (San-Roman et al., 2021; Kingma et al., 2021), knowledge distillation (Luhman & Luhman, 2021;
Revised: of the data. Therefore, previous works have tried to reduce the number of diffusion steps by using non-Markovian reverse processes (Song et al., 2020; Kong & Ping, 2021), adaptive noise scheduling (San-Roman et al., 2021; Kingma et al., 2021), knowledge distillation (Luhman & Luhman, 2021;

Original: Salimans & Ho, 2022), diffusing in a lower-dimension latent space (Rombach et al., 2022), etc. , but they still cannot achieve significant speedup without sacrificing the generation quality.
Revised: Salimans & Ho, 2022), diffusing in a lower-dimension latent space (Rombach et al., 2022), etc. , but they still cannot achieve significant speedup without sacrificing the generation quality.

Original: In this paper, we propose a novel way to shorten the diffusion trajectory by learning an implicit distribution to start the reverse diffusion process, instead of relying on a tractable noise distribution.
Revised: In this paper, we propose a novel way to shorten the diffusion trajectory by learning an implicit distribution to start the reverse diffusion process, instead of relying on a tractable noise distribution.

Original: We call our method truncated diffusion probabilistic modeling (TDPM), which is based on the idea of truncating the forward diffusion chain of an existing diffusion model, such as the denoising diffusion probabilistic model (DDPM) of Ho et al.
Revised: We call our method truncated diffusion probabilistic modeling (TDPM), which is based on the idea of truncating the forward diffusion chain of an existing diffusion model, such as the denoising diffusion probabilistic model (DDPM) of Ho et al.

Original: To significantly accelerate diffusion-based text-to-image generation, we also introduce the truncated latent diffusion model (TLDM), which truncates the diffusion chain of the latent diffusion model (LDM) of Rombach et al.
Revised: To significantly accelerate diffusion-based text-to-image generation, we also introduce the truncated latent diffusion model (TLDM), which truncates the diffusion chain of the latent diffusion model (LDM) of Rombach et al.

Original: We note
Revised: We note

Original: LDM is the latent text-to-image diffusion model behind Stable Diffusion , an open-source project that provides state-of-the-art performance in generating photo-realistic images given text input.
Revised: LDM is the latent text-to-image diffusion model behind Stable Diffusion , an open-source project that provides state-of-the-art performance in generating photo-realistic images given text input.

Original: By truncating the chain, we can reduce the number of diffusion steps to an arbitrary level, but at the same time, we also lose the tractability of the distribution at the end of the chain.
Revised: By truncating the chain, we can reduce the number of diffusion steps to an arbitrary level, but at the same time, we also lose the tractability of the distribution at the end of the chain.

Original: Therefore, we need to learn an implicit generative distribution that can approximate this distribution and provide the initial samples for the reverse diffusion process.
Revised: Therefore, we need to learn an implicit generative distribution that can approximate this distribution and provide the initial samples for the reverse diffusion process.

Original: We show that this implicit generative distribution can be implemented in different ways, such as using a separate generator network or reusing the denoising network.
Revised: We show that this implicit generative distribution can be implemented in different ways, such as using a separate generator network or reusing the denoising network.

Original: The former option has more flexibility and can improve the generation quality, while the latter option has no additional parameters and can achieve comparable results.
Revised: The former option has more flexibility and can improve the generation quality, while the latter option has no additional parameters and can achieve comparable results.

Original: We reveal that DDPM and VAE have a similar relationship as TDPM and adversarial auto-encoder (AAE, Makhzani et al.
Revised: We reveal that DDPM and VAE have a similar relationship as TDPM and adversarial auto-encoder (AAE, Makhzani et al.

Original: Specifically, DDPM is like a VAE with a fixed encoder and a learnable decoder that use a diffusion process, and a predefined prior.
Revised: Specifically, DDPM is like a VAE with a fixed encoder and a learnable decoder that use a diffusion process, and a predefined prior.

Original: TDPM is like an AAE with a fixed encoder and a learnable decoder that use a truncated diffusion process, and a learnable implicit prior.
Revised: TDPM is like an AAE with a fixed encoder and a learnable decoder that use a truncated diffusion process, and a learnable implicit prior.

Original: Our truncation method has several advantages when we use it to modify DDPM for generating images without text guidance or LDM for generating images with text guidance.
Revised: Our truncation method has several advantages when we use it to modify DDPM for generating images without text guidance or LDM for generating images with text guidance.

Original: First, it can generate samples much faster by using fewer diffusion steps, without sacrificing or even enhancing the generation quality.
Revised: First, it can generate samples much faster by using fewer diffusion steps, without sacrificing or even enhancing the generation quality.

Original: Second, it can exploit the cooperation between the implicit model and the diffusion model, as the diffusion model helps the implicit model train by providing noisy data samples, and the implicit model helps the diffusion model reverse by providing better initial samples.
Revised: Second, it can exploit the cooperation between the implicit model and the diffusion model, as the diffusion model helps the implicit model train by providing noisy data samples, and the implicit model helps the diffusion model reverse by providing better initial samples.

Original: Third, it can adapt the truncation level to balance the generation quality and efficiency, depending on the data complexity and the computational resources.
Revised: Third, it can adapt the truncation level to balance the generation quality and efficiency, depending on the data complexity and the computational resources.

Original: For generating images with text guidance, our method can speed up the generation significantly and make it suitable for real-time processing: while LDM takes the time to generate one photo-realistic image, our TLDM can generate more than 50 such images.
Revised: For generating images with text guidance, our method can speed up the generation significantly and make it suitable for real-time processing: while LDM takes the time to generate one photo-realistic image, our TLDM can generate more than 50 such images.

Original: The main contributions of our paper are as follows:
Revised: The main contributions of our paper are as follows:

Original: • We introduce TDPM, a new diffusion-based generative model that can shorten the diffusion trajectory by learning an implicit distribution to start the reverse diffusion process, and demonstrate that the learning of the implicit distribution can be achieved in various ways.
Revised: • We introduce TDPM, a new diffusion-based generative model that can shorten the diffusion trajectory by learning an implicit distribution to start the reverse diffusion process, and demonstrate that the learning of the implicit distribution can be achieved in various ways.

Original: We further introduce TLDM to significantly accelerate diffusion-based text-to-image generation.
Revised: We further introduce TLDM to significantly accelerate diffusion-based text-to-image generation.

Original: • We show TDPM can be formulated as a diffusion-based AAE.
Revised: • We show TDPM can be formulated as a diffusion-based AAE.

Original: • We show that the implicit distribution can be realized by reusing the denoising network for the reverse diffusion process, which can reduce the reverse diffusion steps by orders of magnitude without adding any extra parameters and with comparable generation quality.
Revised: • We show that the implicit distribution can be realized by reusing the denoising network for the reverse diffusion process, which can reduce the reverse diffusion steps by orders of magnitude without adding any extra parameters and with comparable generation quality.

Original: • We reveal the synergy between the implicit model and the diffusion model, as the diffusion process can simplify the training of the implicit model like GANs, and the implicit model can speed up the reverse diffusion process of the diffusion model.
Revised: • We reveal the synergy between the implicit model and the diffusion model, as the diffusion process can simplify the training of the implicit model like GANs, and the implicit model can speed up the reverse diffusion process of the diffusion model.

Original: • We show that both TDPM and TLDM can adapt the truncation level according to the data complexity and the computational resources, to achieve a good balance between the generation quality and the efficiency.
Revised: • We show that both TDPM and TLDM can adapt the truncation level, according to the data complexity and the computational resources, to achieve a good balance between the generation quality and the efficiency.

Original:  P RELIMINARIES ON DIFFUSION MODELS
Revised:  P RELIMINARIES ON DIFFUSION MODELS

Original: In Gaussian diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020), starting from the data distribution x 0 ∼ q ( x 0 ) , a pre-defined forward diffusion process q t produces auxiliary variables x t =1: T by gradually adding Gaussian noise, with variance β t ∈ (0 , 1) at time t , as follows:
Revised: In Gaussian diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020), starting from the data distribution x 0 ∼ q ( x 0 ) , a pre-defined forward diffusion process q t produces auxiliary variables x t =1: T by gradually adding Gaussian noise, with variance β t ∈ (0 , 1) at time t , as follows:

Original: [Equation]
Revised: [Equation]

Original: With the limit of small diffusion rate ( i.e. , β t is kept sufficiently small), the reverse distribution q ( x t − 1 | x t ) also follows a Gaussian distribution (Feller, 1949;
Revised: With the limit of small diffusion rate ( i.e. , β t is kept sufficiently small), the reverse distribution q ( x t − 1 | x t ) also follows a Gaussian distribution (Feller, 1949;

Original: Sohl-Dickstein et al., 2015) and can be approximated using a neural network parameterized Gaussian distribution p θ as:
Revised: Sohl-Dickstein et al., 2015) and can be approximated using a neural network parameterized Gaussian distribution p θ as:

Original: [Equation]
Revised: [Equation]

Original: Moreover, with a sufficiently large T , the outcome of the diffusion chain x T will follow an isotropic
Revised: Moreover, with a sufficiently large T , the outcome of the diffusion chain x T will follow an isotropic

Original: Gaussian distribution.
Revised: Gaussian distribution.

Original: Thus, with the pre-defined forward (inference) diffusion process and the learned reverse (generative) diffusion process, we can sample from x T ∼ N ( 0 , I ) and run the diffusion process in reverse to get a sample from the data distribution q ( x 0 ) .
Revised: Thus, with the pre-defined forward (inference) diffusion process and the learned reverse (generative) diffusion process, we can sample from x T ∼ N ( 0 , I ) and run the diffusion process in reverse to get a sample from the data distribution q ( x 0 ) .

Original: Under the variational inference (Kingma & Welling, 2013; Blei et al., 2017) framework, viewing q ( x 1 , ..., x T | x 0 ) in (1) as the inference network, we can use the evidence lower bound (ELBO) as our learning objective.
Revised: Under the variational inference (Kingma & Welling, 2013; Blei et al., 2017) framework, viewing q ( x 1 , ..., x T | x 0 ) in (1) as the inference network, we can use the evidence lower bound (ELBO) as our learning objective.

Original: Following previous works (Sohl-Dickstein et al., 2015; Ho et al., 2020), the negative ELBO of a diffusion probabilistic model, parameterized by θ , can be expressed as
Revised: Following previous works (Sohl-Dickstein et al., 2015; Ho et al., 2020), the negative ELBO of a diffusion probabilistic model, parameterized by θ , can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where D KL ( q || p ) =
Revised: where D KL ( q || p ) =

Original: E q [log q − log p ] denotes the Kullback–Leibler (KL) divergence from distributions p to q .
Revised: E q [log q − log p ] denotes the Kullback–Leibler (KL) divergence from distributions p to q .

Original: Generally speaking, diffusion probabilistic models assume the number of diffusion steps T to be sufficiently large to satisfy two conditions: 1) the reverse distribution at each denoising step can be fitted with a Gaussian denoising generator p θ ( x t − 1 | x t ) ; 2) with a sufficiently small diffusion rate β t , the long forward diffusion process will successfully corrupt the data, making q ( x T | x 0 )
Revised: Generally speaking, diffusion probabilistic models assume the number of diffusion steps T to be sufficiently large to satisfy two conditions: 1) the reverse distribution at each denoising step can be fitted with a Gaussian denoising generator p θ ( x t − 1 | x t ) ; 2) with a sufficiently small diffusion rate β t , the long forward diffusion process will successfully corrupt the data, making q ( x T | x 0 )

Original: ≈ N ( 0 , I ) , and hence approximately L T becomes zero and depends on neither x 0 nor θ .
Revised: ≈ N ( 0 , I ) , and hence approximately L T becomes zero and depends on neither x 0 nor θ .

Original: What happens if T is insufficiently large?
Revised: What happens if T is insufficiently large?

Original: Given a non-Gaussian data distribution q ( x 0 ) , when the number of denoising steps is reduced, the true posterior q ( x t − 1 | x t ) is not Gaussian and usually intractable (Feller, 1949), resulting in new challenges to current diffusion models.
Revised: Given a non-Gaussian data distribution q ( x 0 ) , when the number of denoising steps is reduced, the true posterior q ( x t − 1 | x t ) is not Gaussian and usually intractable (Feller, 1949), resulting in new challenges to current diffusion models.

Original: As noted in Xiao et al.
Revised: As noted in Xiao et al.

Original: (2022), when β t is not sufficiently small, the diffusion step becomes larger and the denoising distribution can be multi-modal and hence too complex to be well fitted by Gaussian.
Revised: (2022), when β t is not sufficiently small, the diffusion step becomes larger and the denoising distribution can be multi-modal and hence too complex to be well fitted by Gaussian.

Original: The authors propose to define p θ ( x t − 1 | x t ) with an implicit generator and substitute the ELBO with
Revised: The authors propose to define p θ ( x t − 1 | x t ) with an implicit generator and substitute the ELBO with

Original: [Equation]
Revised: [Equation]

Original: where D adv represents a statistical distance that relies on an adversarial training setup.
Revised: where D adv represents a statistical distance that relies on an adversarial training setup.

Original: This modified objective can be minimized by leveraging the power of conditional GANs in fitting implicit multimodal distributions (Arjovsky et al., 2017; Goodfellow et al., 2014;
Revised: This modified objective can be minimized by leveraging the power of conditional GANs in fitting implicit multimodal distributions (Arjovsky et al., 2017; Goodfellow et al., 2014;

Original: Nowozin et al., 2016).
Revised: Nowozin et al., 2016).

Original: While the concept of diffusion has been used, the proposed models in Xiao et al.
Revised: While the concept of diffusion has been used, the proposed models in Xiao et al.

Original: (2022) are shown to work the best only when the number of diffusion steps is limited to be as few as four, and start to exhibit deteriorated performance when further increasing that number.
Revised: (2022) are shown to work the best only when the number of diffusion steps is limited to be as few as four, and start to exhibit deteriorated performance when further increasing that number.

Original:  T RUNCATED DIFFUSION AND ADVERSARIAL AUTO - ENCODING
Revised:  T RUNCATED DIFFUSION AND ADVERSARIAL AUTO - ENCODING

Original: We first introduce the idea of accelerating both the training and generation of diffusion models by truncating the diffusion chains and describe the technical challenges.
Revised: We first introduce the idea of accelerating both the training and generation of diffusion models by truncating the diffusion chains and describe the technical challenges.

Original: We then develop the objective function and training algorithm for TDPM.
Revised: We then develop the objective function and training algorithm for TDPM.

Original: We further reveal TDPM can also be formulated as an
Revised: We further reveal TDPM can also be formulated as an

Original: AAE (Makhzani et al., 2015)) empowered by diffusion models.
Revised: AAE (Makhzani et al., 2015)) empowered by diffusion models.

Original: While DDPM can be considered as a hierarchical version of a variational auto-encoder (VAE) with a fixed multi-stochastic-layer encoder, our derivation shows that TDPM can be considered as a hierarchical version of an AAE with a fixed multi-stochastic-layer encoder but a learnable implicit prior.
Revised: While DDPM can be considered as a hierarchical version of a variational auto-encoder (VAE) with a fixed multi-stochastic-layer encoder, our derivation shows that TDPM can be considered as a hierarchical version of an AAE with a fixed multi-stochastic-layer encoder but a learnable implicit prior.

Original: 3. M OTIVATION AND TECHNICAL CHALLENGES
Revised: 3. M OTIVATION AND TECHNICAL CHALLENGES

Original: We propose a novel method called TDPM to speed up the diffusion process and the generative model.
Revised: We propose a novel method called TDPM to speed up the diffusion process and the generative model.

Original: The main idea is to shorten the forward diffusion chain that transforms the data into Gaussian noise, and use a learned implicit distribution to sample the starting point of the reverse diffusion chain that reconstructs the data.
Revised: The main idea is to shorten the forward diffusion chain that transforms the data into Gaussian noise, and use a learned implicit distribution to sample the starting point of the reverse diffusion chain that reconstructs the data.

Original: To be more precise, we adopt the DDPM framework that defines a variance schedule { β 1 , β 2 , ..., β T } , which controls the amount of noise added at each step of the forward diffusion process.
Revised: To be more precise, we adopt the DDPM framework that defines a variance schedule { β 1 , β 2 , ..., β T } , which controls the amount of noise added at each step of the forward diffusion process.

Original: The forward process has a simple analytical form as a Gaussian distribution:
Revised: The forward process has a simple analytical form as a Gaussian distribution:

Original: [Equation]
Revised: [Equation]

Original: Here, x t is the noisy version of the data x 0 at step t , and ¯ α t is the cumulative product of the diffusion coefficients α i .
Revised: Here, x t is the noisy version of the data x 0 at step t , and ¯ α t is the cumulative product of the diffusion coefficients α i .

Original: The forward chain of length T is designed to be long enough to make the data distribution indistinguishable from Gaussian noise N ( 0 , I ) .
Revised: The forward chain of length T is designed to be long enough to make the data distribution indistinguishable from Gaussian noise N ( 0 , I ) .

Original: However, a long forward chain also implies a high computational cost for the reverse process, which uses a learned neural network to predict the conditional distribution of the clean data given the noisy one at each step.
Revised: However, a long forward chain also implies a high computational cost for the reverse process, which uses a learned neural network to predict the conditional distribution of the clean data given the noisy one at each step.

Original: The proposed TDPM cuts off the last part of the forward chain and only keeps the first T trunc steps { β 1 , β 2 , ..., β T trunc } ⊂ { β 1 , β 2 , ..., β T } .
Revised: The proposed TDPM cuts off the last part of the forward chain and only keeps the first T trunc steps { β 1 , β 2 , ..., β T trunc } ⊂ { β 1 , β 2 , ..., β T } .

Original: We choose T trunc to be much smaller than T so that we can save a lot of computation time in generation.
Revised: We choose T trunc to be much smaller than T so that we can save a lot of computation time in generation.

Original: The benefit of this truncation is illustrated in Figure 1, where the bottom row shows the truncated diffusion chain.
Revised: The benefit of this truncation is illustrated in Figure 1, where the bottom row shows the truncated diffusion chain.

Original: We can see that the data are only partially corrupted by noise and still retain some features of the original data.
Revised: We can see that the data are only partially corrupted by noise and still retain some features of the original data.

Original: This means that we can recover the data more easily and accurately by applying a few Gaussian denoising steps from the corrupted data.
Revised: This means that we can recover the data more easily and accurately by applying a few Gaussian denoising steps from the corrupted data.

Original: Moreover, we do not change the diffusion rates β t for the first T trunc steps, so we do not compromise the quality of the forward and reverse processes between time 0 and T trunc .
Revised: Moreover, we do not change the diffusion rates β t for the first T trunc steps, so we do not compromise the quality of the forward and reverse processes between time 0 and T trunc .

Original: However, truncating the forward chain also introduces a new challenge for the reverse process.
Revised: However, truncating the forward chain also introduces a new challenge for the reverse process.

Original: Unlike the original chain, where the starting point of the reverse process is x T ∼ N ( 0 , I ) , the truncated chain has an unknown distribution of the corrupted data at step T trunc .
Revised: Unlike the original chain, where the starting point of the reverse process is x T ∼ N ( 0 , I ) , the truncated chain has an unknown distribution of the corrupted data at step T trunc .

Original: This makes it difficult to sample from this distribution and initiate the reverse process.
Revised: This makes it difficult to sample from this distribution and initiate the reverse process.

Original: To overcome this challenge, we introduce an implicit generative model that approximates the distribution of the corrupted data by minimizing a divergence measure between the implicit and the true noisy distributions at step T trunc .
Revised: To overcome this challenge, we introduce an implicit generative model that approximates the distribution of the corrupted data by minimizing a divergence measure between the implicit and the true noisy distributions at step T trunc .

Original: This way, we can use the implicit model to sample the starting point of the reverse process and then apply the learned denoising network to generate the data.
Revised: This way, we can use the implicit model to sample the starting point of the reverse process and then apply the learned denoising network to generate the data.

Original: 3. H AND - CRAFTED TDPM OBJECTIVE FUNCTION
Revised: 3. H AND - CRAFTED TDPM OBJECTIVE FUNCTION

Original: Mathematically, recall that the DDPM loss in (3) consists of three terms: L 0 , (cid:80) Tt =
Revised: Mathematically, recall that the DDPM loss in (3) consists of three terms: L 0 , (cid:80) Tt =

Original: 2 L t − 1 , and L T .
Revised: 2 L t − 1 , and L T .

Original: The training objective of a conventional diffusion model focuses on terms (cid:80)
Revised: The training objective of a conventional diffusion model focuses on terms (cid:80)

Original: Tt =
Revised: Tt =

Original: 2 L t − 1 and L 0 .
Revised: 2 L t − 1 and L 0 .

Original: It assumes L T does not depend on any parameter and will be close to zero by carefully pre-defining the forward noising process such that q ( x T | x 0 )
Revised: It assumes L T does not depend on any parameter and will be close to zero by carefully pre-defining the forward noising process such that q ( x T | x 0 )

Original: ≈ p ( x T ) =
Revised: ≈ p ( x T ) =

Original: N ( 0 , I ) .
Revised: N ( 0 , I ) .

Original: When the diffusion chains are truncated at time T trunc ≪ T , the forward diffusion ends at time T trunc , where the marginal distribution of the forward diffusion-corrupted data can be expressed as
Revised: When the diffusion chains are truncated at time T trunc ≪ T , the forward diffusion ends at time T trunc , where the marginal distribution of the forward diffusion-corrupted data can be expressed as

Original: [Equation]
Revised: [Equation]

Original: As p ( x 0 ) is unknown and hence q ( x T trunc ) is often intractable, to reverse this truncated forward diffusion chain, we can no longer start the reverse diffusion chain from a known distribution such as N ( 0 , I ) .
Revised: which takes a semi-implicit form (Yin & Zhou, 2018) whose density function is often intractable. To reverse this truncated forward diffusion chain, we can no longer start the reverse diffusion chain from a known distribution such as N ( 0 , I ) .

Original: To this end, we propose TDPM that starts the reverse chain at time T trunc from p ψ ( x T trunc ) , an implicit distribution parameterized by ψ .
Revised: To this end, we propose TDPM that starts the reverse chain at time T trunc from p ψ ( x T trunc ) , an implicit distribution parameterized by ψ .

Original: We match p ψ ( x T trunc ) to q ( x T trunc ) via a loss term as ˜ L T trunc :
Revised: We match p ψ ( x T trunc ) to q ( x T trunc ) via a loss term as ˜ L T trunc :

Original: = D ( q ( x T trunc )
Revised: = D ( q ( x T trunc )

Original: || p ψ ( x T trunc )) , where D ( q || p ) is a statistical distance between distributions q and p , such as the Jensen–Shannon divergence and Wasserstein distance.
Revised: || p ψ ( x T trunc )) , where D ( q || p ) is a statistical distance between distributions q and p , such as the Jensen–Shannon divergence and Wasserstein distance.

Original: As we keep all the diffusion steps before time T trunc in TDPM the same as those in DDPM, we combine ˜ L T trunc with all the loss terms of DDPM before time T trunc in (3) to define the TDPM loss as
Revised: As we keep all the diffusion steps before time T trunc in TDPM the same as those in DDPM, we combine ˜ L T trunc with all the loss terms of DDPM before time T trunc in (3) to define the TDPM loss as

Original: [Equation]
Revised: [Equation]

Original: We note while in general p ψ ( x T trunc ) in TDPM is intractable, we can employ a deep neural networkbased generator G ψ to generate a random sample in a single step viax T [Equation] = G [Equation]
Revised: We note while in general p ψ ( x T trunc ) in TDPM is intractable, we can employ a deep neural networkbased generator G ψ to generate a random sample in a single step via

Original: [Equation]

Original: , z ∼ N ( 0 , I ) .

Original: [Equation]
Revised: [Equation]

Original: We will discuss later that we may simply let ψ = θ to avoid adding more parameters.
Revised: We will discuss later that we may simply let ψ = θ to avoid adding more parameters.

Original: TDPMAS
Revised: TDPM

Revised: AS DIFFUSION -

Original: Following the terminology of AAE, let us define the prior as p ψ ( x T trunc ) , the decoder (likelihood) as
Revised: Following the terminology of AAE, let us define the prior as p ψ ( x T trunc ) , the decoder (likelihood) as

Original: [Equation]
Revised: [Equation]

Original: which is empowered by a reverse diffusion chain of length T trunc , and the encoder (variational posterior) as q ( x T trunc | x 0 ) .
Revised: which is empowered by a reverse diffusion chain of length T trunc , and the encoder (variational posterior) as q ( x T trunc | x 0 ) .

Original: Thus we can view q ( x T trunc ) defined in (7) as the aggregated posterior (Hoffman & Johnson, 2016; Tomczak & Welling, 2018).
Revised: Thus we can view q ( x T trunc ) defined in (7) as the aggregated posterior (Hoffman & Johnson, 2016; Tomczak & Welling, 2018).

Original: In addition to imposing an auto-encoding data-reconstruction loss, the key idea of the AAE (Makhzani et al., 2015) is to also match the aggregated posterior to a fixed prior.
Revised: In addition to imposing an auto-encoding data-reconstruction loss, the key idea of the AAE (Makhzani et al., 2015) is to also match the aggregated posterior to a fixed prior.

Original: This idea differs AAE from a VAE that regularizes the autoencoder by matching the variational posterior to a fixed prior under the KL divergence.
Revised: This idea differs AAE from a VAE that regularizes the autoencoder by matching the variational posterior to a fixed prior under the KL divergence.

Original: To this end, we introduce a diffusion-based AAE (Diffusion-AAE), whose loss function is defined as
Revised: To this end, we introduce a diffusion-based AAE (Diffusion-AAE), whose loss function is defined as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Diffusion-AAE has two notable differences from a vanilla AAE: 1) its encoder is fixed and has no learnable parameters, while its prior is not fixed and is optimized to match the aggregated posterior, and 2) its decoder is a reverse diffusion chain, with T trunc stochastic layers all parameterized by θ .
Revised: Diffusion-AAE has two notable differences from a vanilla AAE: 1) its encoder is fixed and has no learnable parameters, while its prior is not fixed and is optimized to match the aggregated posterior, and 2) its decoder is a reverse diffusion chain, with T trunc stochastic layers all parameterized by θ .

Original: Note in general as the likelihood in (10) is intractable, the first loss term in (11) is intractable.
Revised: Note in general as the likelihood in (10) is intractable, the first loss term in (11) is intractable.

Original: However, the loss of Diffusion-AAE is upper bounded by the loss of TDPM, as described below.
Revised: However, the loss of Diffusion-AAE is upper bounded by the loss of TDPM, as described below.

Original: Theorem 1.
Revised: Theorem 1.

Original: The Diffusion-AAE loss in (11) is upper bounded by the TDPM loss in (8):
Revised: The Diffusion-AAE loss in (11) is upper bounded by the TDPM loss in (8):

Original: L Diffusion-AAE ≤ L TDPM .
Revised: L Diffusion-AAE ≤ L TDPM .

Original: 3. M ATCHING THE PRIOR TO AGGREGATED POSTERIOR
Revised: 3. M ATCHING THE PRIOR TO AGGREGATED POSTERIOR

Original: Via the loss term ˜
Revised: Via the loss term ˜

Original: L T trunc :
Revised: L T trunc :

Original: = D ( q ( x T trunc )
Revised: = D ( q ( x T trunc )

Original: || p ψ ( x T trunc )) in (8), we aim to match the prior p ψ ( x T trunc ) to the aggregated posterior q ( x T trunc ) in TDPM.
Revised: || p ψ ( x T trunc )) in (8), we aim to match the prior p ψ ( x T trunc ) to the aggregated posterior q ( x T trunc ) in TDPM.

Original: While we have an analytic density function for neither p nor q , we can easily draw random samples from both of them.
Revised: While we have an analytic density function for neither p nor q , we can easily draw random samples from both of them.

Original: Thus, we explore the use of two different types of statistical distances that can be estimated from samples of both q and p .
Revised: Thus, we explore the use of two different types of statistical distances that can be estimated from samples of both q and p .

Original: We empirically show that TDPM can achieve good performance regardless of which distance is used for optimization.
Revised: We empirically show that TDPM can achieve good performance regardless of which distance is used for optimization.

Original: One possible statistical distance is based on the idea of GANs (Goodfellow et al., 2014; Arjovsky et al., 2017; Bi´nkowski et al., 2018), which are widely used to learn implicit distributions from empirical data.
Revised: One possible statistical distance is based on the idea of GANs (Goodfellow et al., 2014; Arjovsky et al., 2017; Bi´nkowski et al., 2018), which are widely used to learn implicit distributions from empirical data.

Original: In this setting, we use a generator G ψ ( · ) :
Revised: In this setting, we use a generator G ψ ( · ) :

Original: R d → R d to transform samples from an isotropic Gaussian p ( z ) into samples that approximate the corrupted data, and a discriminator
Revised: R d → R d to transform samples from an isotropic Gaussian p ( z ) into samples that approximate the corrupted data, and a discriminator

Original: D ϕ ( · ) : R d → [0 , 1] to distinguish between the samples from the corrupted data distribution q ( x T trunc | x 0 ) and the implicit generative distribution p ψ ( x T trunc ) .
Revised: D ϕ ( · ) : R d → [0 , 1] to distinguish between the samples from the corrupted data distribution q ( x T trunc | x 0 ) and the implicit generative distribution p ψ ( x T trunc ) .

Original: The generator and the discriminator are trained by the following objective L GAN T trunc :
Revised: The generator and the discriminator are trained by the following objective L GAN T trunc :

Original: [Equation]
Revised: [Equation]

Original: 3. T RAINING ALGORITHM
Revised: 3. T RAINING ALGORITHM

Original: As the objective in Equation 8 is a sum of different terms, following DDPM (Ho et al., 2020) to fix the terms Σ θ ( x t , t )
Revised: As the objective in Equation 8 is a sum of different terms, following DDPM (Ho et al., 2020) to fix the terms Σ θ ( x t , t )

Original: =
Revised: =

Original: σ 2 t I , we can simplify 1 T trunc (cid:80) T trunc t =1 L t − 1 as an expectation defined as
Revised: σ 2 t I , we can simplify 1 T trunc (cid:80) T trunc t =1 L t − 1 as an expectation defined as

Original: [Equation]
Revised: [Equation]

Original: where ϵ t is an injected noise at a uniformly sampled timestep index t , x t = √ ¯ α t x 0 + √ 1 − ¯ α t ϵ t is a noisy image at time t , and ϵ θ is a denoising U-Net that predicts the noise in order to refine the noisy image x t .
Revised: where ϵ t is an injected noise at a uniformly sampled timestep index t , x t = √ ¯ α t x 0 + √ 1 − ¯ α t ϵ t is a noisy image at time t , and ϵ θ is a denoising U-Net that predicts the noise in order to refine the noisy image x t .

Original: Therefore the final simplified version of (8) is constructed as
Revised: Therefore the final simplified version of (8) is constructed as

Original: [Equation]
Revised: [Equation]

Original: While λ , the weight of L T trunc , can be tuned, we fix it as one for simplicity.
Revised: While λ , the weight of L T trunc , can be tuned, we fix it as one for simplicity.

Original: Here the TDPM objective consists of two parts: the denoising part ϵ θ is focused on denoising the truncated chain, getting updated from L simple_trunc , while the implicit part G ψ is focused on minimizing E q [ D ( q ( x T trunc )
Revised: Here the TDPM objective consists of two parts: the denoising part ϵ θ is focused on denoising the truncated chain, getting updated from L simple_trunc , while the implicit part G ψ is focused on minimizing E q [ D ( q ( x T trunc )

Original: || p ψ ( x T trunc ))] , getting updated from L GAN T trunc .
Revised: || p ψ ( x T trunc ))] , getting updated from L GAN T trunc .

Original: An interesting finding of this paper is that we do not necessarily need to introduce a separate set of parameters ψ for the generator G ψ , as we can simply reuse the same parameters θ of the reverse diffusion model ( i.e. , let ψ = θ ) without clearly hurting the empirical performance.
Revised: An interesting finding of this paper is that we do not necessarily need to introduce a separate set of parameters ψ for the generator G ψ , as we can simply reuse the same parameters θ of the reverse diffusion model ( i.e. , let ψ = θ ) without clearly hurting the empirical performance.

Original: This suggests that the reverse diffusion process from T to T trunc could be effectively approximated by a single step using the same network architecture and parameters as the reverse diffusion steps from T trunc to 0 .
Revised: This suggests that the reverse diffusion process from T to T trunc could be effectively approximated by a single step using the same network architecture and parameters as the reverse diffusion steps from T trunc to 0 .

Original: Therefore, we provide two configurations to parameterize the implicit distributions. 1)
Revised: Therefore, we provide two configurations to parameterize the implicit distributions. 1)

Original: To save parameters, we let the implicit generator and denoising model share the same U-Net parameters but using different time step indices.
Revised: To save parameters, we let the implicit generator and denoising model share the same U-Net parameters but using different time step indices.

Original: Specifically, we first use x T trunc =
Revised: Specifically, we first use x T trunc =

Original: G ψ ( x T )=
Revised: G ψ ( x T )=

Original: ϵ θ ( x T , t = T trunc +1) , where x T ∼ N ( 0 , I ) , to generate a noisy image at time T trunc .
Revised: ϵ θ ( x T , t = T trunc +1) , where x T ∼ N ( 0 , I ) , to generate a noisy image at time T trunc .

Original: We further explore employing a different model, e.g. , StyleGAN2 (Karras et al., 2020a), for the implicit generator, which provides better performance but increases the model size to get x T Trunc .
Revised: We further explore employing a different model, e.g. , StyleGAN2 (Karras et al., 2020a), for the implicit generator, which provides better performance but increases the model size to get x T Trunc .

Original: Then for t = T trunc , . . .
Revised: Then for t = T trunc , . . .

Original: , 1 , we iteratively refine it as x t − 1 = 1 √ α t ( x t − 1 − α t √ 1 − ¯ α t ϵ θ ( x t , t ))
Revised: , 1 , we iteratively refine it as x t − 1 = 1 √ α t ( x t − 1 − α t √ 1 − ¯ α t ϵ θ ( x t , t ))

Original: + β t z t , where z t ∼ N ( 0 , I ) when t > 1 and z 1 = 0 .
Revised: + β t z t , where z t ∼ N ( 0 , I ) when t > 1 and z 1 = 0 .

Original: This process is depicted in Algorithms 1 and 2 in the Appendix.
Revised: This process is depicted in Algorithms 1 and 2 in the Appendix.

Revised: For the implementation details, please refer to Appendix D.6 and our code at https://github.com/JegZheng/ truncated-diffusion-probabilistic-models .

Original: 3. R ELATED WORK
Revised: 3. R ELATED WORK

Original: In our previous discussions, we have related TDPM to several existing works such as DDPM and
Revised: In our previous discussions, we have related TDPM to several existing works such as DDPM and

Original: AAE.
Revised: AAE.

Original: A detailed discussion on other related works is provided in Appendix B.
Revised: A detailed discussion on other related works is provided in Appendix B.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We aim to demonstrate that TDPM can generate good samples faster by using fewer steps of reverse diffusion.
Revised: We aim to demonstrate that TDPM can generate good samples faster by using fewer steps of reverse diffusion.

Original: We use different image datasets to test our method and follow the same setting as other diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021; Rombach et al., 2022) for our backbones.
Revised: We use different image datasets to test our method and follow the same setting as other diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021; Rombach et al., 2022) for our backbones.

Original: We also have two ways to set up the implicit generator that starts the reverse diffusion.
Revised: We also have two ways to set up the implicit generator that starts the reverse diffusion.

Original: One way is to reuse the denoising network, and the other way is to use a separate network.
Revised: One way is to reuse the denoising network, and the other way is to use a separate network.

Original: We try both ways for generating images without any labels.
Revised: We try both ways for generating images without any labels.

Original: For generating images from text, we use the first way with the LDM backbone.
Revised: For generating images from text, we use the first way with the LDM backbone.

Original: We provide comprehensive details, toy examples, and additional experimental results in Appendices D.4-D.8.
Revised: We provide comprehensive details, toy examples, and additional experimental results in Appendices D.4-D.8.

Original: We use FID (lower is better) and Recall (higher is better) to measure the fidelity and diversity, respectively, of the generated images.
Revised: We use FID (lower is better) and Recall (higher is better) to measure the fidelity and diversity, respectively, of the generated images.

Original: We use CIFAR-10 (Krizhevsky et al., 2009), LSUN-bedroom, and LSUN-Church (Yu et al., 2015) datasets in unconditional experiments, and CUB-200 (Welinder et al., 2010) and MS-COCO (Lin et al., 2014) for text-to-image experiments.
Revised: We use CIFAR-10 (Krizhevsky et al., 2009), LSUN-bedroom, and LSUN-Church (Yu et al., 2015) datasets in unconditional experiments, and CUB-200 (Welinder et al., 2010) and MS-COCO (Lin et al., 2014) for text-to-image experiments.

Original: The images consist of 32 × 32 pixels for CIFAR-10 and 256 × 256 pixels for the other datasets.
Revised: The images consist of 32 × 32 pixels for CIFAR-10 and 256 × 256 pixels for the other datasets.

Original: 4. E FFICIENCY IN BOTH TRAINING AND SAMPLING
Revised: 4. E FFICIENCY IN BOTH TRAINING AND SAMPLING

Original: We first look at the results on CIFAR-10.
Revised: We first look at the results on CIFAR-10.

Original: We use DDPM
Revised: We use DDPM

Original: (Ho et al., 2020) or improved DDPM (Nichol & Dhariwal, 2021) as our backbones.
Revised: (Ho et al., 2020) or improved DDPM (Nichol & Dhariwal, 2021) as our backbones.

Original: We use 4, 49, or 99 steps of reverse diffusion, which corresponds to 5, 50, or 100 number of function evaluations (NFE).
Revised: We use 4, 49, or 99 steps of reverse diffusion, which correspond

Original: For the implicit generator, we either reuse [Table](resolution 256 × 256 ). Similar to Table 1, TDPM
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: T

Original: [Table] [Table]
Revised: [Table] [Figure]

Original: uses DDPM backbone for

Original: [Table]
Revised: [Figure]

Original: [Table]
Revised: to 5, 50, or 100 number of function evaluations (NFE).

Original: [Figure] [Figure] the denoising U-Net or use a StyleGAN2 network (respectively, we call them TDPM and TDPM+).
Revised: For the implicit generator, we either reuse the denoising U-Net or use a StyleGAN2 network (respectively, we call them TDPM and TDPM+).

Original: For comparison, we also include DDIM (Song et al., 2020) and DDGAN (Xiao et al., 2022).
Revised: For comparison, we also include DDIM (Song et al., 2020) and DDGAN (Xiao et al., 2022).

Original: The comparison with a more diverse set of baselines can be found in Table 9 in Appendix D.7.
Revised: The comparison with a more diverse set of baselines can be found in Table 9 in Appendix D.7.

Original: Table 1 shows that our TDPM can get good FID with fewer NFE.
Revised: Table 1 shows that our TDPM can get good FID with fewer NFE.

Original: TDPM+ can get even better FID, and it is the best when NFE=100.
Revised: TDPM+ can get even better FID, and it is the best when NFE=100.

Original: Compared with TDPM with 0 steps of reverse diffusion (a GAN with DDPM’s U-Net as generator and StyleGAN2 as discriminator) and StyleGAN2, TDPM with more than 0 steps of reverse diffusion has better recall and the FID is as good as StyleGAN2-ADA (a GAN with data augmentation for better training).
Revised: Compared with TDPM with 0 steps of reverse diffusion (a GAN with DDPM’s U-Net as generator and StyleGAN2 as discriminator) and StyleGAN2, TDPM with more than 0 steps of reverse diffusion has better recall and the FID is as good as StyleGAN2-ADA (a GAN with data augmentation for better training).

Original: This means TDPM can largely avoid the mode missing problem in GANs.
Revised: This means TDPM can largely avoid the mode missing problem in GANs.

Original: We show some examples of generated images on CIFAR-10 in Figure 13.
Revised: We show some examples of generated images on CIFAR-10 in Figure 13.

Original: We also check how fast TDPM can train and sample.
Revised: We also check how fast TDPM can train and sample.

Original: In training, we count how many images TDPM needs to well fit the truncated diffusion chain and the implicit prior.
Revised: In training, we count how many images TDPM needs to well fit the truncated diffusion chain and the implicit prior.

Original: Figure 3 shows that when we use fewer steps of reverse diffusion, the diffusion part needs less time to train.
Revised: Figure 3 shows that when we use fewer steps of reverse diffusion, the diffusion part needs less time to train.

Original: But the implicit prior needs more time to train because it has to model a harder distribution, e.g. , fitting the implicit prior with 4 diffusion steps needs similar time to directly fit it on the data.
Revised: But the implicit prior needs more time to train because it has to model a harder distribution, e.g. , fitting the implicit prior with 4 diffusion steps needs similar time to directly fit it on the data.

Original: When we use 99 steps of reverse diffusion, the diffusion chain and the implicit prior need similar time to train, and the whole model trains faster than both GAN and DDPM.
Revised: When we use 99 steps of reverse diffusion, the diffusion chain and the implicit prior need similar time to train, and the whole model trains faster than both GAN and DDPM.

Original: In sampling, we compare TDPM with 0, 1, 4, 49, or 99 steps of reverse diffusion.
Revised: In sampling, we compare TDPM with 0, 1, 4, 49, or 99 steps of reverse diffusion.

Original: We report both FID and the sampling time (s/image) on one NVIDIA V
Revised: We report both FID and the sampling time (s/image) on one NVIDIA V

Original: GPU in Figure 4.
Revised: GPU in Figure 4.

Original: When we use 4 steps of reverse diffusion, the FID is much lower than 0 steps, and the sampling time is slightly longer.
Revised: When we use 4 steps of reverse diffusion, the FID is much lower than 0 steps, and the sampling time is slightly longer.

Original: When we use more steps of reverse diffusion, the FID goes down slowly, but the sampling time goes up linearly.

Original: When we use 99 steps of reverse diffusion, the FID 
Revised: When we use more steps of reverse diffusion, the FID goes down

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: slowly, but the sampling time goes up linearly.

Original:  of TDPM is better than DDPM with 1000 steps.
Revised: When we use 99 steps of reverse diffusion, the FID of TDPM is better than DDPM with 1000 steps.

Original: Because the FID does not change much when we use more steps of reverse diffusion, we suggest using a small number of steps, such as 4 or more, to balance the quality and speed of generation.
Revised: Because the FID does not change much when we use more steps of reverse diffusion, we suggest using a small number of steps, such as 4 or more, to balance the quality and speed of generation.

Original: 4. R ESULTS ON HIGHER - RESOLUTION AND MORE DIVERSE IMAGE DATASETS
Revised: 4. R ESULTS ON HIGHER - RESOLUTION AND MORE DIVERSE IMAGE DATASETS

Original: To test the performance of the proposed truncation method on high-resolution images, we train TDPM using two different diffusion models, DDPM (Ho et al., 2020) and ADM (Dhariwal & Nichol, 2021), as backbones on two datasets of 256 × 256 resolution, LSUN-Church and LSUN-Bedroom (Yu et al., 2015).
Revised: To test the performance of the proposed truncation method on high-resolution images, we train TDPM using two different diffusion models, DDPM (Ho et al., 2020) and ADM (Dhariwal & Nichol, 2021), as backbones on two datasets of 256 × 256 resolution, LSUN-Church and LSUN-Bedroom (Yu et al., 2015).

Original: We compare the FIDs of TDPM with those of the backbone models and some state-of-the-art
Revised: We compare the FIDs of TDPM with those of the backbone models and some state-of-the-art

Original: GANs in Tables 2.
Revised: GANs in Tables 2.

Original: The results show that TDPM can generate images of similar quality with much smaller truncation steps T trunc , which means that it can produce images significantly faster than the backbone models.
Revised: The results show that TDPM can generate images of similar quality with much smaller truncation steps T trunc , which means that it can produce images significantly faster than the backbone models.

Original: We also visualize the samples from the implicit distribution x T trunc ∼ p θ ( x T trunc ) that TDPM generates and the corresponding x 0 that it finishes at the end of reverse chain in Figure 5.
Revised: We also visualize the samples from the implicit distribution x T trunc ∼ p θ ( x T trunc ) that TDPM generates and the corresponding x 0 that it finishes at the end of reverse chain in Figure 5.

Original: We further evaluate TDPM on ImageNet-1K (with resolution 64 × 64) that exhibits high diversity.
Revised: We further evaluate TDPM on ImageNet-1K (with resolution 64 × 64) that exhibits high diversity.

Original: Here we adopt the TDPM+ configuration, where we use a pre-trained ADM (Dhariwal & Nichol, 2021) checkpoint for t < T Trunc and train a StyleGAN-XL (Sauer et al., 2022) based implicit model at t =
Revised: Here we adopt the TDPM+ configuration, where we use a pre-trained ADM (Dhariwal & Nichol, 2021) checkpoint for t < T Trunc and train a StyleGAN-XL (Sauer et al., 2022) based implicit model at t =

Original: T Trunc (for simplicity, we choose to not use the progressive growing pipeline of StyleGAN-XL;
Revised: T Trunc (for simplicity, we choose to not use the progressive growing pipeline of StyleGAN-XL;

Original: See Appendix D.6 for more details).
Revised: See Appendix D.6 for more details).

Original: We compare both FID and Recall with our backbone models in
Revised: We compare both FID and Recall with our backbone models in

Original: Table 3 and show example generations in Figure 2.
Revised: Table 3 and show example generations in Figure 2.

Original: Similar to our observations in Table 1, TDPM has good generation quality with small truncation steps T trunc .
Revised: Similar to our observations in Table 1, TDPM has good generation quality with small truncation steps T trunc .

Original: Moreover, properly training an implicit model at T trunc can further improve the performance of the backbone.
Revised: Moreover, properly training an implicit model at T trunc can further improve the performance of the backbone.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4. T EXT - TO - IMAGE GENERATION
Revised: 4. T EXT - TO - IMAGE GENERATION

Original: Besides unconditional generation tasks, we develop for text-to-image generation the TLDM, a conditional version of TDPM that leverages as the backbone the LDM of Rombach et al.
Revised: Besides unconditional generation tasks, we develop for text-to-image generation the TLDM, a conditional version of TDPM that leverages as the backbone the LDM of Rombach et al.

Original: (2022), which is a state-of-the-art publicly released model with 1.45B parameters pre-trained on LAION400M (Schuhmann et al., 2021).
Revised: (2022), which is a state-of-the-art publicly released model with 1.45B parameters pre-trained on LAION400M (Schuhmann et al., 2021).

Original: LDM consists of a fixed auto-encoder for pixel generation and a latent-diffusion module to connect text and image embeddings.
Revised: LDM consists of a fixed auto-encoder for pixel generation and a latent-diffusion module to connect text and image embeddings.

Original: Here we fine-tune its latent-diffusion part on CUB-200 and MS-COCO datasets with 25K and 100K steps as the baseline.
Revised: Here we fine-tune its latent-diffusion part on CUB-200 and MS-COCO datasets with 25K and 100K steps as the baseline.

Original: Similar to the unconditional case, we fine-tune with the LDM loss for t <
Revised: Similar to the unconditional case, we fine-tune with the LDM loss for t <

Original: T Trunc and GAN loss for t = T Trunc .
Revised: T Trunc and GAN loss for t = T Trunc .

Original: More details about the setting can be found in Appendix D.6.
Revised: More details about the setting can be found in Appendix D.6.

Original: The results of LDM with different DDIM sampling steps and TLDM with different truncated steps are summarized in Figure 6 and Table 4.
Revised: The results of LDM with different DDIM sampling steps and TLDM with different truncated steps are summarized in Figure 6 and Table 4.

Original: Similar to applying diffusion directly on the original image-pixel space, when the diffusion chain is applied in the latent space, we observe TLDM can achieve comparable or better performance than LDM even though it has shortened the diffusion chain of LDM to have much fewer reverse diffusion steps.
Revised: Similar to applying diffusion directly on the original image-pixel space, when the diffusion chain is applied in the latent space, we observe TLDM can achieve comparable or better performance than LDM even though it has shortened the diffusion chain of LDM to have much fewer reverse diffusion steps.

Original: For the case that NFE is as small as 5, we note although the FID of TLDM has become higher due to using fewer diffusion steps, the generated image using TLDM at NFE=5 is still visually appealing, as shown in Figure 7.
Revised: For the case that NFE is as small as 5, we note although the FID of TLDM has become higher due to using fewer diffusion steps, the generated image using TLDM at NFE=5 is still visually appealing, as shown in Figure 7.

Original: Compared with 50 and 250 steps using LDM, the sampling speed of TLDM using 5 steps is 10 and 50 times faster, respectively, while largely preserving generation quality.
Revised: Compared with 50 and 250 steps using LDM, the sampling speed of TLDM using 5 steps is 10 and 50 times faster, respectively, while largely preserving generation quality.

Original: We provide additional text-to-image generation results of TLDM in Appendix D.8.
Revised: We provide additional text-to-image generation results of TLDM in Appendix D.8.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we investigate how to reduce the trajectory length of the diffusion chain to achieve efficient sampling without loss of generation quality.
Revised: In this paper, we investigate how to reduce the trajectory length of the diffusion chain to achieve efficient sampling without loss of generation quality.

Original: We propose truncated diffusion probabilistic modeling (TDPM) that truncates the length of a diffusion chain.
Revised: We propose truncated diffusion probabilistic modeling (TDPM) that truncates the length of a diffusion chain.

Original: In this way, TDPM can use a much shorter diffusion chain, while being required to start the reverse denoising process from an intractable distribution.
Revised: In this way, TDPM can use a much shorter diffusion chain, while being required to start the reverse denoising process from an intractable distribution.

Original: We propose to learn such a distribution with an implicit generative model powered by the same U-Net used for denoising diffusion, and validate with multiple ways to learn the implicit distribution to ensure the robustness of the proposed TDPM.
Revised: We propose to learn such a distribution with an implicit generative model powered by the same U-Net used for denoising diffusion, and validate with multiple ways to learn the implicit distribution to ensure the robustness of the proposed TDPM.

Original: We reveal that TDPM can be cast as an adversarial auto-encoder with a learnable implicit prior.
Revised: We reveal that TDPM can be cast as an adversarial auto-encoder with a learnable implicit prior.

Original: We conduct extensive experiments on both synthetic and real image data to demonstrate the effectiveness of TDPM in terms of both sample quality and efficiency, where the diffusion chain can be shortened to have only a few steps. 
Revised: We conduct extensive experiments on both synthetic and real image data to demonstrate the effectiveness of TDPM in terms of both sample quality and efficiency, where the diffusion chain can be shortened to have only a few steps. A CKNOWLEDGMENTS

Revised: H. Zheng and M. Zhou acknowledge the support of NSF-IIS 2212418 and IFML.

Original: A P ROOF
Revised: A P ROOF

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: As the last terms in both losses are the same
Revised: As the last terms in both losses are the same

Original: [Equation]
Revised: [Equation]

Original: we only need to show that the first term in (11) is smaller than or equal to L 0 + (cid:80) T trunc t =2 L t − 1 in (8).
Revised: we only need to show that the first term in (11) is smaller than or equal to L 0 + (cid:80) T trunc t =2 L t − 1 in (8).

Original: Using Jensen’s inequality
Revised: Using Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original:  where the second to last equality follows the same derivation of the ELBO in Ho et al.
Revised: 15) where the second to last equality follows the same derivation of the ELBO in Ho et al.

Original: B R ELATED WORK
Revised: B R ELATED WORK

Original: Diffusion probabilistic models (Sohl-Dickstein et al., 2015; Ho et al., 2020) employ a forward Markov chain to diffuse the data to noise and learn the reversal of such a diffusion process.
Revised: Diffusion probabilistic models (Sohl-Dickstein et al., 2015; Ho et al., 2020) employ a forward Markov chain to diffuse the data to noise and learn the reversal of such a diffusion process.

Original: With the idea of exploiting the Markov operations (Goyal et al., 2017; Alain et al., 2016; Bordes et al., 2017), diffusion models achieve great success and inspire a variety of tasks including image generation and audio generation (Kong et al., 2020;
Revised: With the idea of exploiting the Markov operations (Goyal et al., 2017; Alain et al., 2016; Bordes et al., 2017), diffusion models achieve great success and inspire a variety of tasks including image generation and audio generation (Kong et al., 2020;

Original: Chen et al., 2020; Jolicoeur-Martineau et al., 2020; Vahdat et al., 2021).
Revised: Chen et al., 2020; Jolicoeur-Martineau et al., 2020; Vahdat et al., 2021).

Original: Recently, plenty of studies have been proposed to generalize diffusion model to continuous time diffusion and improve the diffusion models in likelihood estimation (Vincent, 2011; Song &
Revised: Recently, plenty of studies have been proposed to generalize diffusion model to continuous time diffusion and improve the diffusion models in likelihood estimation (Vincent, 2011; Song &

Original: Ermon, 2020; 2019; Nichol & Dhariwal, 2021; Song et al., 2021b;a; Kingma et al., 2021).
Revised: Ermon, 2020; 2019; Nichol & Dhariwal, 2021; Song et al., 2021b;a; Kingma et al., 2021).

Original: Another mainstream is to improve the sampling efficiency of diffusion models, which are known for their enormous number of sampling steps.
Revised: Another mainstream is to improve the sampling efficiency of diffusion models, which are known for their enormous number of sampling steps.

Original: Luhman & Luhman (2021) improve diffusion processes with knowledge distillation and San-Roman et al.
Revised: Luhman & Luhman (2021) improve diffusion processes with knowledge distillation and San-Roman et al.

Original: (2021) propose a learnable adaptive noise schedule.
Revised: (2021) propose a learnable adaptive noise schedule.

Original: Song et al.
Revised: Song et al.

Original: and
Revised: and

Original: Kong & Ping (2021) exploit non-Markovian diffusion processes and shorten the denoising segments.
Revised: Kong & Ping (2021) exploit non-Markovian diffusion processes and shorten the denoising segments.

Original: Jolicoeur-Martineau et al.
Revised: Jolicoeur-Martineau et al.

Original: and Huang et al.
Revised: and Huang et al.

Original: (2021) use better SDE solvers for continuous-time models.
Revised: (2021) use better SDE solvers for continuous-time models.

Original: Aside from these works, recently other types of generative models such as VAEs (Kingma & Welling, 2013), GANs (Goodfellow et al., 2014), and autoregressive models (van den Oord et al., 2016) have been incorporated to diffusion models.
Revised: Aside from these works, recently other types of generative models such as VAEs (Kingma & Welling, 2013), GANs (Goodfellow et al., 2014), and autoregressive models (van den Oord et al., 2016) have been incorporated to diffusion models.

Original: They are shown to benefit each other (Xiao et al., 2022; Pandey et al., 2022;
Revised: They are shown to benefit each other (Xiao et al., 2022; Pandey et al., 2022;

Original: Meng et al., 2021) and have  closer relation to our work.
Revised: Meng et al., 2021) and have a closer relation to our work.

Original: Xiao et al.
Revised: Xiao et al.

Original: (2022) consider the use of implicit models (Huszár, 2017; Mohamed & Lakshminarayanan, 2016; Tran et al., 2017; Yin & Zhou, 2018; Li & Malik, 2018) to boost the efficiency of diffusion models, where they deploy implicit models in each denoising step, which has higher difficulty in the training as the number of diffusion steps increases.
Revised: (2022) consider the use of implicit models (Huszár, 2017; Mohamed & Lakshminarayanan, 2016; Tran et al., 2017; Yin & Zhou, 2018; Li & Malik, 2018) to boost the efficiency of diffusion models, where they deploy implicit models in each denoising step, which has higher difficulty in the training as the number of diffusion steps increases.

Original: Pandey et al.
Revised: Pandey et al.

Original: (2022) build diffusion models on top of the output of VAEs for refinement.
Revised: (2022) build diffusion models on top of the output of VAEs for refinement.

Original: Our work is also related if viewing TDPM as a diffusion model on top of an implicit model, where the implicit model can be parameterized with the U-Net or a separate network.
Revised: Our work is also related if viewing TDPM as a diffusion model on top of an implicit model, where the implicit model can be parameterized with the U-Net or a separate network.

Original: C D ISCUSSION
Revised: C D ISCUSSION

Original: Potential societal impacts : This paper proposes truncated diffusion probabilistic model as a novel type of diffusion-based generative model.
Revised: Potential societal impacts : This paper proposes truncated diffusion probabilistic model as a novel type of diffusion-based generative model.

Original: The truncated part can be trained as implicit generative models such as GANs jointly or independently with the diffusion part.
Revised: The truncated part can be trained as implicit generative models such as GANs jointly or independently with the diffusion part.

Original: The capacities of truncated diffusion probabilistic models are competitive to existing diffusion-based ones and efficiency is largely improved.
Revised: The capacities of truncated diffusion probabilistic models are competitive to existing diffusion-based ones and efficiency is largely improved.

Original: On the contrary of these positive effects, some negative perspectives could also be seen, depending on how the models are used.
Revised: On the contrary of these positive effects, some negative perspectives could also be seen, depending on how the models are used.

Original: One major concern is the truncated diffusion technique proposed in this paper could potentially be a way to hack the existing diffusion models if the implicit models are maliciously used to fit the intermediate steps.
Revised: One major concern is the truncated diffusion technique proposed in this paper could potentially be a way to hack the existing diffusion models if the implicit models are maliciously used to fit the intermediate steps.

Original: For example, for some existing diffusion models, for safety concerns, the models capacity to generate private data needs to be locked by hiding the diffusion ending point into an unknown distribution.
Revised: For example, for some existing diffusion models, for safety concerns, the model’s capacity to generate private data needs to be locked by hiding the diffusion ending point into an unknown distribution.

Original: The technique of TDPM could be used to crack these existing online diffusion models by providing intermediate noisy images or fine-tuning the first few steps with TDPM to unlock the capacity.
Revised: The technique of TDPM could be used to crack these existing online diffusion models by providing intermediate noisy images or fine-tuning the first few steps with TDPM to unlock the capacity.

Original: Besides, the capacity of generating good images can also be misused to generate ill-intentioned images at a much lower cost.
Revised: Besides, the capacity of generating good images can also be misused to generate ill-intentioned images at a much lower cost.

Original: Discussions : In this work we mainly focus on reducing the length of the diffusion chain of a finitetime diffusion model.
Revised: Discussions : In this work, we mainly focus on reducing the length of the diffusion chain of a finite-time diffusion model.

Original: Our model has shown its effectiveness in improving finite-time diffusion models and it is non-trivial to further explore our model on continuous-time diffusion models (Song et al., 2021b).
Revised: Our model has shown its effectiveness in improving finite-time diffusion models and it is non-trivial to further explore our model on continuous-time diffusion models (Song et al., 2021b).

Original: Moreover, while in this paper DDPM is the primary baseline, TDPM can also be built on other recent diffusion models.
Revised: Moreover, while in this paper DDPM is the primary baseline, TDPM can also be built on other recent diffusion models.

Original: While p θ ( x T trunc ) is parameterized as an implicit distribution, it can also be formulated as a semi-implicit distribution (Yin & Zhou, 2018), which allows it to be approximated with a Gaussian generator.
Revised: While p θ ( x T trunc ) is parameterized as an implicit distribution, it can also be formulated as a semi-implicit distribution (Yin & Zhou, 2018), which allows it to be approximated with a Gaussian generator.

Original: Xiao et al.
Revised: Xiao et al.

Original: (2022) also present a closely related work.
Revised: (2022) also present a closely related work.

Original: While we share the same spirit to reduce the length of the diffusion chain, these two strategies are not conflicting with each other.
Revised: While we share the same spirit to reduce the length of the diffusion chain, these two strategies are not conflicting with each other.

Original: Inthe future work we will look into the integration of these different strategies.
Revised: In future work we will look into the integration of these different strategies.

Original: There also exists plenty of options in approximating p θ ( x T trunc ) .
Revised: There also exists plenty of options in approximating p θ ( x T trunc ) .

Original: When truncating the diffusion chain to be short, the implicit distribution still faces multi-modal and need to fit with different methods depending upon the properties that we need.
Revised: When truncating the diffusion chain to be short, the implicit distribution still faces multi-modal and needs to fit with different methods depending upon the properties that we need.

Original: For example, in order to capture all modes, a VAE would be preferred, like done in Pandey et al.
Revised: For example, in order to capture all modes, a VAE would be preferred, like done in Pandey et al.

Original: Below we provide an alternative method proposed in Zheng & Zhou (2021) to fit the truncated distribution.
Revised: Below we provide an alternative method proposed in Zheng & Zhou (2021) to fit the truncated distribution.

Original: Besides the training, it’s also an open question whether TDPM can be incorporated into more advanced architectures to have further improvements and we leave this exploration for future work.
Revised: Besides the training, it’s also an open question whether TDPM can be incorporated into more advanced architectures to have further improvements and we leave this exploration for future work.

Original: D A LGORITHM DETAILS AND COMPLEMENTARY R ESULTS
Revised: D A LGORITHM DETAILS AND COMPLEMENTARY R ESULTS

Original: Below we provide additional algorithm details and complementary experimental results.
Revised: Below we provide additional algorithm details and complementary experimental results.

Original: D. A DDITIONAL ANALYSIS ON THE PARAMETERIZATION OF THE IMPLICIT GENERATOR
Revised: D. A DDITIONAL ANALYSIS ON THE PARAMETERIZATION OF THE IMPLICIT GENERATOR

Original: As shown in Section 3, in general the objective of TDPM consists of the training of the diffusion model
Revised: As shown in Section 3, in general, the objective of TDPM consists of the training of the diffusion model

Original: ϵ θ (a U-Net architecture (Ronneberger et al., 2015)) with simple loss of DDPM L simple and the training of an implicit prior model G ψ with objective L GAN T trunc .
Revised: ϵ θ (a U-Net architecture (Ronneberger et al., 2015)) with simple loss of DDPM L simple and the training of an implicit prior model G ψ with objective L GAN T trunc .

Original: Without loss of generality, in our main paper, we show two configurationsof to parameterize the implicit part for t =
Revised: Without loss of generality, in our main paper, we show two configurations to parameterize the implicit part for t =

Original: T trunc : 1) the implicit generator shares the same U-Net architecture used for 0 < t < T trunc ; 2) the implicit generator is instantiated with a separate network.
Revised: T trunc : 1) the implicit generator shares the same U-Net architecture used for 0 < t < T trunc ; 2) the implicit generator is instantiated with a separate network.

Original: Below we explain this two configurations (denoted as TDPM+ in the main paper).
Revised: Below we explain this two configurations (denoted as TDPM+ in the main paper).

Original: Configuration 1) :
Revised: Configuration 1) :

Original: At t =
Revised: At t =

Original: T trunc , the Unet generates the noisy image at the truncated step: x T trunc =
Revised: T trunc , the Unet generates the noisy image at the truncated step: x T trunc =

Original: ϵ θ ( x T trunc +1 , t =
Revised: ϵ θ ( x T trunc +1 , t =

Original: T trunc + 1) , where x T trunc +1 ∼ N (0 , I ) is the pure noise image whose pixels are iid sampled from standard normal.
Revised: T trunc + 1) , where x T trunc +1 ∼ N (0 , I ) is the pure noise image whose pixels are iid sampled from standard normal.

Original: For t =
Revised: For t =

Original: T trunc , T trunc − 1 , . .
Revised: T trunc , T trunc − 1 , . .

Original: , 1 , the same Unet iteratively refines the noisy images by letting x t − 1 = 1 √ ¯ α t ( x t − 1 − α t √ 1 − ¯ α t ϵ t − 1 ) + β t z t ; z t> 1 ∼ N (0 , I ) , z 1 = 0 , where ϵ t − 1 =
Revised: , 1 , the same Unet iteratively refines the noisy images by letting x t − 1 = 1 √ ¯ α t ( x t − 1 − α t √ 1 − ¯ α t ϵ t − 1 ) + β t z t ; z t> 1 ∼ N (0 , I ) , z 1 = 0 , where ϵ t − 1 =

Original: ϵ θ ( x t , t ) is the predicted noise by the Unet.
Revised: ϵ θ ( x t , t ) is the predicted noise by the Unet.

Original: Under this setting, the Unet-based generator plays two roles at the same time and the training will be more challenging than using two different generators here.
Revised: Under this setting, the Unet-based generator plays two roles at the same time and the training will be more challenging than using two different generators here.

Original: However, we can also see as T trunc gets larger, the distribution of p ( x T trunc ) will become more similar to a noise distribution, and generating the noisy images will be more like generating noises.
Revised: However, we can also see as T trunc gets larger, the distribution of p ( x T trunc ) will become more similar to a noise distribution, and generating the noisy images will be more like generating noises.

Original: In this case, being able to generate both noisy images and predicting noise becomes easier for the generator.
Revised: In this case, being able to generate both noisy images and predicting noise becomes easier for the generator.

Original: Configuration 2) (TDPM+) :
Revised: Configuration 2) (TDPM+) :

Original: Unlike previous configuration, where the implicit generator at step t =
Revised: Unlike previous configuration, where the implicit generator at step t =

Original: T shares the same U-Net architecture with t < T trunc .
Revised: T shares the same U-Net architecture with t < T trunc .

Original: Another way is to parameterize G ψ with a separate generator.
Revised: Another way is to parameterize G ψ with a separate generator.

Original: Although this configuration increases the total parameter of the generative model, it allows the model has better flexibility in the training stage.
Revised: Although this configuration increases the total parameter of the generative model, it allows the model has better flexibility in the training stage.

Original: For example, these two networks can be trained in parallel or leverage a pre-trained model.
Revised: For example, these two networks can be trained in parallel or leverage a pre-trained model.

Original: In our paper, we conduct the experiments by using
Revised: In our paper, we conduct the experiments by using

Original: Stylegan2 generator architecture Karras et al.
Revised: Stylegan2 generator architecture Karras et al.

Original: (2020b)
Revised: (2020b)

Original: for t = T trunc , resulting in an increase of 19M and 28M for the generator parameters when handling 32 × 32 and 256 × 256 images.
Revised: for t = T trunc , resulting in an increase of 19M and 28M for the generator parameters when handling 32 × 32 and 256 × 256 images.

Original: The process of training and sampling of these configurations are summarized in Algorithm 1 and 2.Algorithm
Revised: The process of training and sampling of these configurations are summarized in Algorithm 1 and 2.

Original: [Table]
Revised: [Table]

Original: D.A LTERNATIVES OF LEARNING THE IMPLICIT DISTRIBUTION
Revised: D.

Revised: A LTERNATIVES OF LEARNING THE IMPLICIT DISTRIBUTION

Original: Another possible statistical distance is based on conditional transport (Zheng & Zhou, 2021), which is proposed to balance the model-seeking and mode-covering behaviors when fitting an empirical data distribution.
Revised: Another possible statistical distance is based on conditional transport (Zheng & Zhou, 2021), which is proposed to balance the model-seeking and mode-covering behaviors when fitting an empirical data distribution.

Original: In this setting, we use the same generator G ψ as before, but instead of a discriminator, we use a conditional distribution π η parameterized by η to find an optimized mapping between the samples of p and q , and a critic ϕ to measure the point-to-point cost c ϕ in the feature space.
Revised: In this setting, we use the same generator G ψ as before, but instead of a discriminator, we use a conditional distribution π η parameterized by η to find an optimized mapping between the samples of p and q , and a critic ϕ to measure the point-to-point cost c ϕ in the feature space.

Original: The generator, the conditional distribution, and the critic are trained by the following objective L CT T trunc :
Revised: The generator, the conditional distribution, and the critic are trained by the following objective L CT T trunc :

Original: [Equation]
Revised: [Equation]

Original: Similar to (14)
Revised: Similar to (14)

Original: [Equation]
Revised: [Equation]

Original: we fit TDPM-CT with following loss
Revised: we fit TDPM-CT with following loss

Original: [Equation]
Revised: [Equation]

Original: We empirically find out this objective has no significant difference than using GAN objective shwon in Equation 14 in performance-wise as long as the generator is well trained
Revised: We empirically find out this objective has no significant difference than using GAN objective shown in Equation 14 in performance-wise as long as the generator is well trained

Original: [Equation]
Revised: [Equation]

Original: D. C ONDITIONAL TRUNCATED DIFFUSION PROBABILISTIC MODELS
Revised: D. C ONDITIONAL TRUNCATED DIFFUSION PROBABILISTIC MODELS

Original: For conditional generation
Revised: For conditional generation

Original: [Equation]
Revised: [Equation]

Original: we extend (14) and derive a conditional version of TDPM:
Revised: we extend (14) and derive a conditional version of TDPM:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where L c simple_trunc aims to train the conditional diffusion model with 
Revised: where L c simple_trunc aims to train the conditional diffusion model with L

Revised: [Equation]

Revised: [Equation]

Revised: ∼

Original: [Equation]
Revised: [Equation]

Original: 19) and the truncated distribution L c T trunc can be fitted with GAN or CT:
Revised: 19) and the truncated distribution L c T trunc can be fitted with GAN or CT:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: D. A NALYSIS ON TOY EXPERIMENTS
Revised: D. A NALYSIS ON TOY EXPERIMENTS

Original: Although we present image experiments in the main paper, our studies were firstly justified our method on synthetic toy data as a proof of concept.
Revised: Although we present image experiments in the main paper, our studies were firstly justified our method on synthetic toy data as a proof of concept.

Original: We adopt representative 2D synthetic datasets used in 
Revised: We adopt representative 2D synthetic datasets used in prior works (Gulrajani et al., 2017; Zheng & Zhou, 2021), including Swiss Roll, Double Moons, 8-modal, and 25-modal Gaussian mixtures with equal component weights. We use an empirical

Original: [Figure]
Revised: [Figure]

Original: prior works (Gulrajani et al., 2017; Zheng & Zhou, 2021), including Swiss Roll, Double Moons, 8-modal, and 25-modal Gaussian mixtures with equal component weights.

Original: We use an empirical sample set X , consisting of |X| = 2 , 000 samples and illustrate the generated samples aftertraining epochs.
Revised: sample set X , consisting of |X| = 2 , 000 samples and illustrate the generated samples aftertraining epochs.

Original: We take 20 grids in the range [ − 10 , 10] for both the x and y axes to approximate the empirical distribution of ˆ p θ and ˆ q , and report the corresponding forward KL D KL (ˆ q || ˆ p θ ) as the quantitative evaluation metric.
Revised: We take 20 grids in the range [ − 10 , 10] for both the x and y axes to approximate the empirical distribution of ˆ p θ and ˆ q , and report the corresponding forward KL D KL (ˆ q || ˆ p θ ) as the quantitative evaluation metric.

Original: Figure 8 shows the results on the Swiss Roll data.
Revised: Figure 8 shows the results on the Swiss Roll data.

Original: We present a short chain with T = 2 and a longer chain with T = 5 to show the impacts of the number of diffusion steps.
Revised: We present a short chain with T = 2 and a longer chain with T = 5 to show the impacts of the number of diffusion steps.

Original: The first row shows that the data distribution is diffused with accumulated noise, and with more steps the diffused distribution will be closer to an isotropic Gaussian distribution.
Revised: The first row shows that the data distribution is diffused with accumulated noise, and with more steps the diffused distribution will be closer to an isotropic Gaussian distribution.

Original: As one can see, truncating the diffusion chain to a short length will result in a clear gap between q ( x T trunc ) and N ( 0 , I ) .
Revised: As one can see, truncating the diffusion chain to a short length will result in a clear gap between q ( x T trunc ) and N ( 0 , I ) .

Original: When DDPM (shown in the second row) samples from the isotropic Gaussian distribution, it becomes hard to recover the original data distribution from pure noise with only a few steps.
Revised: When DDPM (shown in the second row) samples from the isotropic Gaussian distribution, it becomes hard to recover the original data distribution from pure noise with only a few steps.

Original: Although we can see DDPM can get slightly improved with a few more steps ( T = 5 ), as long as q ( x T ) is not close to Gaussian, DDPM can hardly recover the data distribution.
Revised: Although we can see DDPM can get slightly improved with a few more steps ( T = 5 ), as long as q ( x T ) is not close to Gaussian, DDPM can hardly recover the data distribution.

Original: By contrast, as shown in the third and fourth rows, TDPM successfully approximates the non-Gaussian q ( x T trunc ) with its implicit generator, and we can see the remaining part of the truncated chain is gradually recovered by the denoising steps.
Revised: By contrast, as shown in the third and fourth rows, TDPM successfully approximates the non-Gaussian q ( x T trunc ) with its implicit generator, and we can see the remaining part of the truncated chain is gradually recovered by the denoising steps.

Original: From both visualizations and D KL (ˆ q
Revised: From both visualizations and D KL (ˆ q

Original: ||
Revised: ||

Original: ˆ p θ ) , we can see that TDPM is able to fit every step in such short chains.
Revised: ˆ p θ ) , we can see that TDPM is able to fit every step in such short chains.

Original: TDPM-GAN and TDPM-CT both succeed in fitting p θ ( x T trunc )
Revised: TDPM-GAN and TDPM-CT both succeed in fitting p θ ( x T trunc )

Original: but the latter one fits slightly better when the diffusion length is 2.
Revised: but the latter one fits slightly better when the diffusion length is 2.

Original: When the length increases to 5, fitting the implicit distribution with
Revised: When the length increases to 5, fitting the implicit distribution with

Original: GAN becomes easier.
Revised: GAN becomes easier.

Original: This observation demonstrate a benefit of combining the diffusion models and GANs.
Revised: This observation demonstrate a benefit of combining the diffusion models and GANs.

Original: If the implicit generator is sufficiently powerful to model q ( x T trunc ) , then the number of steps in need can be compressed to a small number.
Revised: If the implicit generator is sufficiently powerful to model q ( x T trunc ) , then the number of steps in need can be compressed to a small number.

Original: On the contrary, if the implicit generator cannot capture the distribution, we need more steps to facilitate the fitting of the data distribution.
Revised: On the contrary, if the implicit generator cannot capture the distribution, we need more steps to facilitate the fitting of the data distribution.

Original: Shown in Figure 9-Figure 11, we can see 8-modal Gaussian is more similar to an isotropic Gaussian after getting diffused, thus DDPM can recover a distribution similar to data with 5 steps.
Revised: Shown in Figure 9-Figure 11, we can see 8-modal Gaussian is more similar to an isotropic Gaussian after getting diffused, thus DDPM can recover a distribution similar to data with 5 steps.

Original: On 25-Gaussians, we can observe GAN does not suffer from mode-collapse and provide a better approximation than CT, which results in better data distribution recovery in the final step.
Revised: On 25-Gaussians, we can observe GAN does not suffer from mode-collapse and provide a better approximation than CT, which results in better data distribution recovery in the final step.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D. A DDITIONAL ABLATION STUDIES
Revised: D. A DDITIONAL ABLATION STUDIES

Original: Using Pre-trained diffusion backbones:
Revised: Using Pre-trained diffusion backbones:

Original: Different from the default setting
Revised: Different from the default setting

Original: [Table]
Revised: [Table]

Original: here we put the implicit model of TDPM+ trained at t =
Revised: here we put the implicit model of TDPM+ trained at t =

Original: T trunc and a pre-trained DDPM model 1 in the same pipeline of sampling.
Revised: T trunc and a pre-trained DDPM model 1 in the same pipeline of sampling.

Original: In this case we do not need to spend any time on pretraining the DDPM model, and only need to train the implicit model for t = T trunc .
Revised: In this case we do not need to spend any time on pretraining the DDPM model, and only need to train the implicit model for t = T trunc .

Original: As shown in Table 5, when combined with a pre-trained DDPM for t < T trunc , the generation performance of TDPM trained under this two-step procedure is comparable to TDPM trained end-to-end.
Revised: As shown in Table 5, when combined with a pre-trained DDPM for t < T trunc , the generation performance of TDPM trained under this two-step procedure is comparable to TDPM trained end-to-end.

Original: [Table]
Revised: [Table]

Original: Sensitivity to noise schedule : Nichol & Dhariwal (2021) show the noise schedule affects the training of DDPM.
Revised: Sensitivity to noise schedule : Nichol & Dhariwal (2021) show the noise schedule affects the training of DDPM.

Original: Here we examine if TDPM is sensitive to the choice of noise schedule.
Revised: Here we examine if TDPM is sensitive to the choice of noise schedule.

Original: We compare the linear schedule with cosine schedule, which adds noise in a milder manner.
Revised: We compare the linear schedule with cosine schedule, which adds noise in a milder manner.

Original: The results on CIFARare reported in Table 6, which suggest that TDPM is not sensitive to the choice between these two schedules. 
Revised: The results on CIFARare reported in Table 6, which suggest that TDPM is not sensitive to the choice between these two schedules. [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: On the choice of truncated step : As the diffused distribution could facilitate the learning of the implicit generator G ψ (Arjovsky & Bottou, 2017), where we could observe by increasing the number of diffusion steps, the FID of TDPM consistently gets better.
Revised: On the choice of truncated step : As the diffused distribution could facilitate the learning of the implicit generator G ψ (Arjovsky & Bottou, 2017), where we could observe by increasing the number of diffusion steps, the FID of TDPM consistently gets better.

Original: A natural question is on which step should we truncate the diffusion chain.
Revised: A natural question is on which step should we truncate the diffusion chain.

Original: We study the signal-to-noise ratio (SNR) of different diffusion step.
Revised: We study the signal-to-noise ratio (SNR) of different diffusion step.

Original: Based on q ( x t | x 0 )
Revised: Based on q ( x t | x 0 )

Original: =
Revised: =

Original: N ( √ ¯ α t x 0 , 1 − ¯ α t I ) , we calculate SNR as
Revised: N ( √ ¯ α t x 0 , 1 − ¯ α t I ) , we calculate SNR as

Original: [Equation]
Revised: [Equation]

Original: We visualize the SNR evolution across time step t > 0 in Figure 12, where we can observe the SNR rapidly decays in the first 100 steps.
Revised: We visualize the SNR evolution across time step t > 0 in Figure 12, where we can observe the SNR rapidly decays in the first 100 steps.

Original: According to previous studies in Arjovsky & Bottou (2017), injecting noise into the data distribution could smoothen the data distribution support and facilitate the
Revised: According to previous studies in Arjovsky & Bottou (2017), injecting noise into the data distribution could smoothen the data distribution support and facilitate the

Original: GAN training.
Revised: GAN training.

Original: The SNR change in this interval indicates injecting noise in the level of t ∈ (cid:74) 1 , 100 (cid:75) could bring in more significant improvement for the GAN training.
Revised: The SNR change in this interval indicates injecting noise in the level of t ∈ (cid:74) 1 , 100 (cid:75) could bring in more significant improvement for the GAN training.

Original: When the step is greater than 200, the SNR is change is no longer significant and close to zero, which indicates the implicit model  might not be too informative, though it is easier to train.
Revised: When the step is greater than 200, the SNR is change is no longer significant and close to zero, which indicates the implicit model diffusion might not be too informative, though it is easier to train.

Original: Our experimental observations in Figurealso justify this conclusion: when training a GAN at T Trunc = 4 , the required number of iterations is similar to training it on clean data; by training the GAN model at T Trunc = 99 , the training of GAN is significantly facilitated.
Revised: Our experimental observations in Figurealso justify this conclusion: when training a GAN at T Trunc = 4 , the required number of iterations is similar to training it on clean data; by training the GAN model at T Trunc = 99 , the training of GAN is significantly facilitated.

Original: For T Trunc > 100 , we empirically examine to train a GAN and find it would converge faster than training the diffusion model for t < T Trunc .
Revised: For T Trunc > 100 , we empirically examine to train a GAN and find it would converge faster than training the diffusion model for t < T Trunc .

Original: Comparison of model efficiency : In complement of the results in Table 1-2, we provide detailed model size and generation time on v100 GPU.
Revised: Comparison of model efficiency : In complement of the results in Table 1-2, we provide detailed model size and generation time on v100 GPU.

Original: The results are summarized in Table 7.
Revised: The results are summarized in Table 7.

Original: We can see
Revised: We can see

Original: TDPM has an increasing in the total number of parameter, as it involves a discriminator to help train the implicit model, while its sampling efficiency is also obvious.
Revised: TDPM has an increasing in the total number of parameter, as it involves a discriminator to help train the implicit model, while its sampling efficiency is also obvious.

Original: [Table]
Revised: [Table]

Original: D. E XPERIMENTAL S ETTINGS D.6. M ODEL ARCHITECTURE
Revised: D. E XPERIMENTAL S ETTINGS D.6. M ODEL ARCHITECTURE

Original: Generator: Our generator structure strictly follows the U-Net structure (Ronneberger et al., 2015) used in DDPM, improved DDPM and ADM (Ho et al., 2020; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021), which consists of multiple ResNet blocks (He et al., 2016) with Attention blocks (Vaswani et al., 2017) injected in the bottleneck.
Revised: Generator: Our generator structure strictly follows the U-Net structure (Ronneberger et al., 2015) used in DDPM, improved DDPM, and ADM (Ho et al., 2020; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021), which consists of multiple ResNet blocks (He et al., 2016) with Attention blocks (Vaswani et al., 2017) injected in the bottleneck.

Original: Please refer to these paper for more details on the architecture.
Revised: Please refer to these paper for more details on the architecture.

Original: A key difference between our model and previous diffusion models is that our model also train such U-Net as an extra implicit generator G θ that takes a latent variable z ∼ N ( 0 , I ) and a fixed time index t
Revised: A key difference between our model and previous diffusion models is that our model also train such U-Net as an extra implicit generator G θ that takes a latent variable z ∼ N ( 0 , I ) and a fixed time index t

Original: =
Revised: =

Original: T trunc + 1 as input.
Revised: T trunc + 1 as input.

Original: However, this does not result in a difference in the generator architecture.
Revised: However, this does not result in a difference in the generator architecture.

Original: We parameterize G θ with the same U-Net architecture for simplicity and the time embedding t =
Revised: We parameterize G θ with the same U-Net architecture for simplicity and the time embedding t =

Original: T trunc + 1 is specified to be trained with the implicit loss shown in (12) and (16).
Revised: T trunc + 1 is specified to be trained with the implicit loss shown in (12) and (16).

Original: We have also tested to use all zero time embedding for t =
Revised: We have also tested to use all zero time embedding for t =

Original: T trunc + 1 and found no clear differences.
Revised: T trunc + 1 and found no clear differences.

Original: For our results of TDPM+, the generator G ψ specifically takes a StyleGAN2 architecture Karras et al.
Revised: For our results of TDPM+, the generator G ψ specifically takes a StyleGAN2 architecture Karras et al.

Original: (2020b)
Revised: (2020b)

Original: and there is no time-embedding in G ψ .
Revised: and there is no time-embedding in G ψ .

Original: An increase of generator parameter appears caused by separating the implicit model and denoising U-Net.
Revised: An increase of generator parameter appears caused by separating the implicit model and denoising U-Net.

Original: Note that the generator is trained with GAN loss and without specially designed adaptive augmentation in Karras et al.
Revised: Note that the generator is trained with GAN loss and without specially designed adaptive augmentation in Karras et al.

Original: (2020a).
Revised: (2020a).

Original: For the detailed model architecture please refer to the corresponding paper or their Github repository: https://github.com/NVlabs/stylegan2-ada-pytorch .
Revised: For the detailed model architecture please refer to the corresponding paper or their Github repository: https://github.com/NVlabs/stylegan2-ada-pytorch .

Original: Discriminator: Similar to Xiao et al.
Revised: Discriminator: Similar to Xiao et al.

Original: , we adopt the discriminator architecture used in Karras et al. (2020b), but without the time step input.
Revised: , we adopt the discriminator architecture used in Karras et al. (2020b), but without the time step input.

Original: The discriminator discriminate x T trunc is from the diffused distribution q ( x T trunc ) or implicit generative distribution p θ ( x T trunc ) .
Revised: The discriminator discriminate x T trunc is from the diffused distribution q ( x T trunc ) or implicit generative distribution p θ ( x T trunc ) .

Original: Please refer toAppendix
Revised: Please refer to

Original:  C of Xiao et al.
Revised: Appendix C of Xiao et al.

Original: for the detailed design.
Revised: for the detailed design.

Original: Navigator: Training with L CT T trunc involves an extra module named navigator (Zheng & Zhou, 2021).
Revised: Navigator: Training with L CT T trunc involves an extra module named navigator (Zheng & Zhou, 2021).

Original: We strictly follow the architecture used in Zheng & Zhou (2021), where the navigator is an MLP taking the pairwise feature distance as inputs.
Revised: We strictly follow the architecture used in Zheng & Zhou (2021), where the navigator is an MLP taking the pairwise feature distance as inputs.

Original: There is no time embedding used in the navigator as it is only used for the training at t =
Revised: There is no time embedding used in the navigator as it is only used for the training at t =

Original: T Trunc .
Revised: T Trunc .

Original: The feature is extracted from the layer before the final scalar output.
Revised: The feature is extracted from the layer before the final scalar output.

Original: Please refer to their Appendix D for detailed information.
Revised: Please refer to their Appendix D for detailed information.

Original: Architecture for text-to-image experiments: We adopt the 1.45B LDM model (Rombach et al., 2022) that is pretrained on LAION-400M dataset (Schuhmann et al., 2021).
Revised: Architecture for text-to-image experiments: We adopt the 1.45B LDM model (Rombach et al., 2022) that is pretrained on LAION-400M dataset (Schuhmann et al., 2021).

Original: The LDM model consists of a U-Net KL-regularized autoencoder with downsampling-factor 8 (resolution 256 -> 32), a U-Net in the latent space, and a BERT (Devlin et al., 2018) text encoder transform raw text to a sequence of 1280-dimension embeddings.
Revised: The LDM model consists of a U-Net KL-regularized autoencoder with downsampling-factor 8 (resolution 256 -> 32), a U-Net in the latent space, and a BERT (Devlin et al., 2018) text encoder transform raw text to a sequence of 1280-dimension embeddings.

Original: We only fine-tune the latent model in our experiments.
Revised: We only fine-tune the latent model in our experiments.

Original: In the training of the truncated part, the discriminator takes the first-half of the U-Net (downsampling backbone) with a linear predicting head on top of it.
Revised: In the training of the truncated part, the discriminator takes the first-half of the U-Net (downsampling backbone) with a linear predicting head on top of it.

Original: Architecture for toy experiments: The generator we use an architecture stacked with 4 linear layers with 128 hidden units.
Revised: Architecture for toy experiments: The generator uses an architecture stacked with 4 linear layers with 128 hidden units.

Original: Each intermediate layer is equipped with a time-embedding layer and follows softplus activation.
Revised: Each intermediate layer is equipped with a time-embedding layer and follows softplus activation.

Original: The discriminator and navigator have the same architecture, without time-embedding layers, and using leakyReLU as the activation function.
Revised: The discriminator and navigator have the same architecture, without time-embedding layers, and using leakyReLU as the activation function.

Original: D.6. T RAINING CONFIGURATIONS
Revised: D.6. T RAINING CONFIGURATIONS

Original: Datasets: We use CIFAR-10 (Krizhevsky et al., 2009), LSUN-bedroom and LSUN-Church (Yu et al., 2015) datasets for unconditional generation in the main experiments.
Revised: Datasets: We use CIFAR-10 (Krizhevsky et al., 2009), LSUN-bedroom, and LSUN-Church (Yu et al., 2015) datasets for unconditional generation in the main experiments.

Original: Additionally, we apply
Revised: Additionally, we apply

Original: CelebA(Liu et al., 2015) and CelebA-HQ (Lee et al., 2020) for complementary justification.
Revised: CelebA(Liu et al., 2015) and CelebA-HQ (Lee et al., 2020) for complementary justification.

Original: For text-to-image experiments, we use CUB-200 (Welinder et al., 2010) and MS-COCO
Revised: For text-to-image experiments, we use CUB-200 (Welinder et al., 2010) and MS-COCO

Original: (Lin et al., 2014).
Revised: (Lin et al., 2014).

Original: The images consist of 32 × 32 pixels for CIFAR-10.
Revised: The images consist of 32 × 32 pixels for CIFAR-10.

Original: For the other datasets, we apply center-crop along the short edge and resize to the target resolution ( 64 × 64 for CelebA; 256 × 256 for the others).
Revised: For the other datasets, we apply center-crop along the short edge and resize to the target resolution ( 64 × 64 for CelebA; 256 × 256 for the others).

Original: Diffusion schedule: For all datasets, we strictly follow the diffusion process used in our backbone models, and instantiate the truncated diffusion schedule by obtaining the first T Trunc diffusion rates { β 1 , ..., β T Trunc } .
Revised: Diffusion schedule: For all datasets, we strictly follow the diffusion process used in our backbone models, and instantiate the truncated diffusion schedule by obtaining the first T Trunc diffusion rates { β 1 , ..., β T Trunc } .

Original: For example, if our goal is to fit a model with NFE=50, to truncate the diffusion process used in Ho et al.
Revised: For example, if our goal is to fit a model with NFE=50, to truncate the diffusion process used in Ho et al.

Original: (2020) ( β 1 = 10 − 4 , β T = 0 .
Revised: (2020) ( β 1 = 10 − 4 , β T = 0 .

Original: 02 , T=1000), we first initialize β 1 , β 2 , ... β 1000 , and then taking the first 49 steps to complete the truncation.
Revised: 02 , T=1000), we first initialize β 1 , β 2 , ... β 1000 , and then taking the first 49 steps to complete the truncation.

Original: Optimization: We train our models using the Adam optimizer (Kingma & Ba, 2015), where most of the hyperparameters match the setting in Xiao et al.
Revised: Optimization: We train our models using the Adam optimizer (Kingma & Ba, 2015), where most of the hyperparameters match the setting in Xiao et al.

Original: , and we slightly modify the generator learning rate to match the setting in Ho et al.
Revised: , and we slightly modify the generator learning rate to match the setting in Ho et al.

Original: (2020), as shown in Table 8.
Revised: (2020), as shown in Table 8.

Original: We train our models using V100 GPUs, with CUDA 10.1, PyTorch 1.7.1.
Revised: We train our models using V100 GPUs, with CUDA 10.1, PyTorch 1.7.1.

Original: The training takes approximately 2 days on CIFAR-10 with 4 GPUs, and a week on CelebA-HQ and LSUN-Church with 8 GPUs.
Revised: The training takes approximately 2 days on CIFAR-10 with 4 GPUs, and a week on CelebA-HQ and LSUN-Church with 8 GPUs.

Original: [Table]
Revised: [Table]

Original: For TDPM+, where we use StyleGAN2 generator as G ψ , we directly use their original training hyper-parameters and train the model in parallel with the diffusion model.
Revised: For TDPM+, where we use StyleGAN2 generator as G ψ , we directly use their original training hyper-parameters and train the model in parallel with the diffusion model.

Original: For TLDM, we set the base learning rate as 10 − 5 and the mini-batch size is set to 64.
Revised: For TLDM, we set the base learning rate as 10 − 5 and the mini-batch size is set to 64.

Original: For the ImageNet1K-64 × 64 experiments, we use StyleGAN-XL generator as G ψ and strictly follow all the default training hyper-parameters.
Revised: For the ImageNet1K-64 × 64 experiments, we use StyleGAN-XL generator as G ψ and strictly follow all the default training hyper-parameters.

Original: To simplify the implementation and save computation, instead of applying the default progressive growing pipeline 16 × 16 → 32 × 32 → 64 × 64 , we directly train the implicit model on 64 × images corrupted at T Trunc .
Revised: To simplify the implementation and save computation [Table] instead of applying the default progressive growing pipeline 16 × 16 → 32 × 32 → 64 × 64 , we directly train the implicit model on 64 × images corrupted at T Trunc .

Original: Without using the progressive growing pipeline, the result of StyleGANXL shown in Table 2 is clearly worse than the progressive one reported in their paper (FID 1.51).
Revised: Without using the progressive growing pipeline, the result of StyleGANXL shown in Table 2 is clearly worse than the progressive one reported in their paper (FID 1.51).

Original: However, when used as the implicit model of TDPM, the final performance of TDPM becomes competitive with this result.
Revised: However, when used as the implicit model of TDPM, the final performance of TDPM becomes competitive with this result.

Original: Evaluation: When evaluating the sampling time, we use models trained on CIFAR-10 and generate a batch of 128 samples.
Revised: Evaluation: When evaluating the sampling time, we use models trained on CIFAR-10 and generate a batch of 128 samples.

Original: When evaluating the FID, and recall score, following the convention, we use 50k generated samples for CIFAR-10, LSUN-bedroom and LSUN-church, 30k samples for CelebAHQ (since the CelebA HQ dataset contains only 30k samples), 30k samples for the text-to-image datasets.
Revised: When evaluating the FID, and recall score, following the convention, we use 50k generated samples for CIFAR-10, LSUN-bedroom and LSUN-church, 30k samples for CelebAHQ (since the CelebA HQ dataset contains only 30k samples), 30k samples for the text-to-image datasets.

Original: The recall scores are calculated with the recipe in Kynkäänniemi et al.
Revised: The recall scores are calculated with the recipe in Kynkäänniemi et al.

Original: In the sampling stage, we follow our backbone to apply the same guidance in the diffusion part ( t < T Trunc ) if applicable.
Revised: In the sampling stage, we follow our backbone to apply the same guidance in the diffusion part ( t < T Trunc ) if applicable.

Original: Specifically, for LDM backbone, we use classifier-free guidance (Ho & Salimans, 2022) with scale 1.5 and there is no DDIM steps for TDLM.
Revised: Specifically, for LDM backbone, we use classifier-free guidance (Ho & Salimans, 2022) with scale 1.5 and there are no DDIM steps for TDLM.

Original: D. A DDITIONAL RESULTS ON UNCONDITIONAL GENERATION
Revised: D. A DDITIONAL RESULTS ON UNCONDITIONAL GENERATION

Original: [Table]
Revised: [Table]

Original: =4 (ours) 3.
Revised: =4 (ours) 3.

Original: 0.55 TDPM+, T trunc =4 (ours) 3.
Revised: 0.55 TDPM+, T trunc =4 (ours) 3.

Original: 0.57 Progressive distiallation (Salimans & Ho, 2022) 4.
Revised: 0.57 Progressive distiallation (Salimans & Ho, 2022) 4.

Original: - Denoising Diffusion GAN (Xiao et al., 2022), T=  4.
Revised: - Denoising Diffusion GAN (Xiao et al., 2022), T=  4.

Original: 0.54 TDPM, T trunc =1 (ours) 4.
Revised: 0.54 TDPM, T trunc =1 (ours) 4.

Original: 0.53 TDPM+, T trunc =1 (ours) 3.
Revised: 0.53 TDPM+, T trunc =1 (ours) 3.

Original: 0.56 DDPM Distillation (Luhman & Luhman, 2021) 9.
Revised: 0.56 DDPM Distillation (Luhman & Luhman, 2021) 9.

Original: 0.51 SNGAN (Miyato et al., 2018) 21.
Revised: 0.51 SNGAN (Miyato et al., 2018) 21.

Original: 0.44 AutoGAN (Gong et al., 2019) 12.
Revised: 0.44 AutoGAN (Gong et al., 2019) 12.

Original: 0.46 TransGAN (Jiang et al., 2021) 9.
Revised: 0.46 TransGAN (Jiang et al., 2021) 9.

Original: - StyleGAN2 w/o ADA (Karras et al., 2020a) 8.
Revised: - StyleGAN2 w/o ADA (Karras et al., 2020a) 8.

Original: 0.41 StyleGAN2 w/ ADA (Karras et al., 2020a) 2.
Revised: 0.41 StyleGAN2 w/ ADA (Karras et al., 2020a) 2.

Original: 0.46 StyleGAN2 w/ Diffaug (Zhao et al., 2020) 5.
Revised: 0.46 StyleGAN2 w/ Diffaug (Zhao et al., 2020) 5.

Original: 0.42 Progressive distiallation (Salimans & Ho, 2022) 9.
Revised: 0.42 Progressive distiallation (Salimans & Ho, 2022) 9.

Original: - Denoising Diffusion GAN (Xiao et al., 2022), T=  14.
Revised: - Denoising Diffusion GAN (Xiao et al., 2022), T=  14.

Original: 0.19 TDPM, T trunc =

Revised: 0.19 TDPM, T trunc =

Original: (ours) 7.
Revised: (ours) 7.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Equation]
Revised: [Equation]

Original:  [Equation] 
Revised: [Figure] [Equation] [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Equation] [Figure]

Original: D.
Revised: [Equation] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original:  A DDITIONAL RESULTS ON TEXT - TO - IMAGE GENERATION
Revised: D. A DDITIONAL RESULTS ON TEXT - TO - IMAGE GENERATION

Original: A white and gray bird with black wings.
Revised: A white and gray bird with black wings.

Original: An airplan flying over a body of water.
Revised: An airplan flying over a body of water.

Original: A sign reads “TDPM”.
Revised: A sign reads “TDPM”.

Original: Busy city street at dusk with sun setting.
Revised: Busy city street at dusk with sun setting.

Original: A clock tower near the river.
Revised: A clock tower near the river.

Original: A cluster of flower on the wooden table.
Revised: A cluster of flower on the wooden table.

Original: The bathroom has a big mirror.
Revised: The bathroom has a big mirror.

Original: The bagel is put in a squre plate.
Revised: The bagel is put in a squre plate.