Deep Equilibrium Approaches to Diffusion Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Diffusion-based generative models have shown to be extremely effective in generating high-quality images, with generated samples often surpassing the quality ofthose produced by other models under several metrics.
One distinguishing featureof these models, however, is that they typically require long sampling chains inorder to produce high-ﬁdelity images.
This presents a challenge not only from thelenses of sampling time, but also from the inherent difﬁculty in backpropagatingthrough these chains in order to accomplish tasks such as model inversion, i.e., approximately ﬁnding latent states that generate known images.
In this paper, we lookat diffusion models through a different perspective, that of a (deep) equilibrium(DEQ) ﬁxed point model.
Speciﬁcally, we extend the recent denoising diffusionimplicit model (DDIM) [59], and model the entire sampling chain as a joint, multivariate ﬁxed point system.
This setup provides an elegant uniﬁcation of diffusionand equilibrium models, and shows beneﬁts in 1) single image sampling, as it replaces the fully-serial typical sampling process with a parallel one; and 2) modelinversion, where we can leverage fast gradients in the DEQ setting to much morequickly ﬁnd the noise that generates a given image.
The approach is also orthogonal and thus complementary to other methods used to reduce the sampling time, orimprove model inversion.
We demonstrate our method’s strong performance acrossseveral datasets, including CIFAR10, CelebA, and LSUN Bedroom and Churches.
Introduction
Diffusion models have emerged as a promising class of generative models that can generate highquality images [60, 59, 50], outperforming GANs on perceptual quality metrics [16] and likelihoodbased models on density estimation [37].
One of the limitations of these models, however, is thefact that they require a long diffusion chain (many repeated applications of an denoising process), inorder to generate high-ﬁdelity samples.
Several recent papers have focused on tackling this limitation,e.g., by shortening the length of diffusion process through an alternative parameterization [59, 39], orthrough progressive distillation of a sampler with large diffusion chain into a smaller one [47, 56].
However, all of these methods still rely on a fundamentally sequential sampling process, imposingchallenges on accelerating the sampling and for other applications like differentiating through theentire generation process.
In this paper, we propose an alternative approach that also begins to address such challenges from adifferent perspective.
Speciﬁcally, we propose to model the generative process of a speciﬁc class ofdiffusion model, the denoising diffusion implicit model (DDIM) [59], as a deep equilibrium (DEQ)model.
Deep equilibrium (DEQ) models are networks that aim to ﬁnd the ﬁxed point of the underlyingsystem in the forward pass and differentiate implicitly through this ﬁxed point in the backward pass.
To apply DEQ to diffusion models, we ﬁrst formulate this process as an equilibrium system consistingof all T joint sampling steps, and then simultaneously solve the ﬁxed point of all the T steps.
This approach has several beneﬁts: First , the DEQ sampling process can be solved in parallelover multiple GPUs by scattering the workload.
This is particularly beneﬁcial in the case of singleimage ( i.e., batch-size-one) generation, where the serial sampling nature of diffusion models hasinevitably made them unable to maximize GPU computation.
Second , solving for the joint equilibriasimultaneously leads to faster overall convergence as we have better estimates of the intermediatestates in fewer steps.
Speciﬁcally, the formulation naturally lends itself to an augmented diffusionchain that each state is generated according to all others.
Third , the DEQ formulation allows us toleverage a faster differentiation through the chain.
This enables us to much more effectively solveproblems that require us to differentiate through the generative process, useful for tasks such as modelinversion that seeks to ﬁnd the noise that leads to a particular instance of an image.
We demonstrate the advantages of this formulation on two applications: single image generationand model inversion.
Both single image generation and model inversion are widely applied in realworld image manipulation tasks like image editing and restoration [67, 54, 30, 1, 43, 48].
On CIFAR10 [41] and CelebA [45], DEQ achieves up to 2 ⇥ speedup over the sequential sampling processes of DDIM [59], while maintaining a comparable perceptual quality of images.
In the model inversion,the loss converges much faster when trained with DEQs than with sequential sampling.
Moreover,optimizing the sequential sampling process can be computationally expensive.
Leveraging modernautograd packages is infeasible as they require storing the entire computational graph for all T states.
Some recent works like Nie et al.
[51] achieve this through use of SDE solvers.
In contrast, with DEQs we can use the implicit differentiation of O (1) memory complexity.
Empirically, the initialhidden state recovered by DEQ more accurately regenerates the original image while capturing itsﬁner details.
To summarize, our main contributions are as follows:
• We formulate the generative process of an augmented type of DDIM as a deep equilibriummodel that allows the use of black-box solvers to efﬁciently compute the ﬁxed point andgenerate images.•
The DEQ formulation parallelizes the sampling process of DDIM, and as a result, it can berun on multiple GPUs instead of a single GPU.
This alternate sampling process convergesfaster compared to the original process.•
We demonstrate the advantages of our formulation on single image generation and modelinversion.
We ﬁnd that optimizing the sampling process via DEQs consistently outperformsnaive sequential sampling.
Preliminaries
Diffusion Models Denoising diffusion probabilistic models (DDPM)
[58, 29] are generative modelsthat can convert the data distribution to a simple distribution, ( e.g., a standard Gaussian, N ( 0 , I ) ),through a diffusion process.
Speciﬁcally, given samples from a target distribution x 0 ⇠ q ( x 0 ) , thediffusion process is a Markov chain that adds Gaussian noise to the data to generate latent statesx 1 , ..., x T in the same sample space as x 0 .
The inference distribution of diffusion process is given by:
[Equation]
[Equation]
To learn the parameters ✓ that characterize a distribution p ✓ ( x 0 ) =
R p ✓ ( x 0: T ) d x 1: T as an approximation of q ( x 0 ) , a surrogate variational lower bound [58] was proposed to train this model:
[Equation]
After training, samples can be generated by a reverse Markov chain, i
As noted in [58, 59], the length T of a diffusion process is usually large (e.g. T = 1000 [29]) as itcontributes to a better approximation of Gaussian conditional distributions in the generative process.
However, because of the large value of T , sampling from diffusion models can be visibly slowercompared to other deep generative models like GANs [25].
One feasible acceleration is to rewrite the forward process into a non-Markovian one that leads to a“shorter” and deterministic generative process, i.e., denoising diffusion implicit model [59] (DDIM).
DDIM can be trained similarly to DDPM, using the variational lower bound shown in Eq.
Essentially, DDIM constructs a nearly non-stochastic scheme that can quickly sample from thelearned data distribution without introducing additional noises.
Speciﬁcally, the scheme to generate asample x t  1 given x t is:
[Equation]
3) where ↵ 1 , ..., ↵ T 2 (0 , 1] , ✏ t ⇠ N ( 0 , I ) , and ✏ ( t ) ✓ ( x t ) is an estimator trained to predictthe noise given a noisy state x t .
Different values of  t deﬁne different generative processes.
For a variance schedule  1 . . .
T , we use the notation ↵ t =
Q t s =1 (1   s ) .
When  t =p (1  ↵ t  1 ) / (1  ↵ t ) p 1  ↵ t / ↵ t  1 for all t , the generative process represents a DDPM.
Setting t = 0 for all t gives rise to a DDIM, which results in a deterministic generating process except theinitial sampling x T ⇠ p ( x T ) .
Deep Equilibrium Models Deep equilibrium models are a recently-proposed class of deep networks that, in their forward pass, seek to ﬁnd a ﬁxed point of a single layer applied repeatedly to ahidden state.
Speciﬁcally, consider a deep feedforward model with L layers:
[Equation]
where x is the input variable,
z
[ i ] is the hidden state of i
th layer, and f [ i ] ✓ is a layer that deﬁnes thefeature transformation.
Assuming the above model is weight-tied, i.e., f
[ i ] ✓ = f ✓ , 8 i , then in the limitof inﬁnite depth, the output z
[ i ] of this network converges to a ﬁxed point z ⇤ .
[Equation]
Inspired from the neural convergence phenomenon, Deep equilibrium (DEQ) models [5] are proposedto directly compute this ﬁxed point
z ⇤ as the output, i.e.,
[Equation]
The equilibrium state z ⇤ can be solved by black-box solvers like Broyden’s method [11], or Andersonacceleration [4].
To train this ﬁxed-point system, Bai et al.
[5] leverage implicit differentiation todirectly backpropagate through the equilibrium state
z ⇤ using O (1) memory complexity.
DEQ isknown as a principled framework for characterizing convergence and energy minimization in deeplearning.
We leave a detailed discussion in Sec. 6.
A Deep Equilibrium Approach to DDIMs
In this section, we present the main modeling contribution of the paper, a formulation of diffusionprocesses under the DEQ framework.
Although diffusion models may seem to be a natural ﬁt for
DEQ modeling (after all, we typically do not care about intermediate states in the denoising chain, butonly the ﬁnal clean image), there are several reasons why setting up the diffusion chain “naively” as a
DEQ ( i.e., making f ✓ be a single sampling step) does not ultimately lead to a functional algorithm.
Most fundamentally, the diffusion process is not time-invariant ( i.e., not “weight-tied” in the DEQsense), and the ﬁnal generated image is practically-speaking independent of the noise used to generateit ( i.e., not truly based upon “input injection” either).
Thus, at a high level, our approach to building a DEQ version of the DDIM involves representing allthe states x 0: T simultaneously within the DEQ state.
The advantage of this approach is that 1) we canexactly capture the typical diffusion inference chain; and 2) we can create a more expressive reverseprocess where the state x t is updated based upon all later states x t +1: T , improving the inferenceprocess.
A notable downside of this representation is that 1) we need to store all states simultaneously(but only the images, and not the intermediate network states, so that this is not overly constraining);2) we can execute all steps of the inference chain in parallel rather than solely in sequence as istypically required in diffusion models; and 3) we can use common DEQ acceleration methods, suchas the Anderson solver [4] to ﬁnd the ﬁxed point, which makes the sampling process converge faster.
A DEQ formulation of DDIMs.
The generative process of DDIM is given by:
[Equation]
This process also lets us generate a sample using a subset of latent states { x ⌧ 1 , . . .
, x ⌧ S } , where{ ⌧ 1 , . . .
, ⌧ S } ✓ T .
While this helps in accelerating the overall generative process, there is a tradeoffbetween sampling quality and computational efﬁciency.
As noted in Song et al.
[59], larger T leadto lower FID scores of the generated images but need more compute time; smaller T are faster tosample from, but the resulting images have worse FID scores.
Reformulating this sampling process as a DEQ addresses multiple concerns raised above.
We candeﬁne a DEQ, with a sequence of latent states x 1: T as its internal state, that simultaneously solves forthe equilibrium points at all the timesteps.
The global convergence of this process is upper boundedby T steps, by deﬁnition.
To derive the DEQ formulation of the generative process, ﬁrst we rearrangethe terms in Eq.
[Equation]
[Equation]
Let c ( t ) 1 = p 1  ↵ t  1
[Equation]
↵ t
[Equation]
1  ↵
[Equation]
↵ t .
Then we can write
[Equation] [Equation]
By induction, we can rewrite the above equation as:
[Equation]
This deﬁnes a “fully-upper-triangular” inference process, where the update of x t depends on thenoise prediction network ✏ ✓ applied to all subsequent states x t +1:
T ; in contrast to the traditionaldiffusion process, which updates x t based only on x t +1 .
Speciﬁcally, let h ( · ) represent the functionthat performs the operations in the equations (10) for a latent x t at timestep t , and let ˜ h ( · ) representthe function that performs the same set of operations across all the timesteps simultaneously.
We canwrite the above set of equations as a ﬁxed point system:
[Equation]
or,
[Equation]
x
[Equation]
The above system of equations represent a DEQ with x T as input injection.
We can simultaneouslysolve for the roots of this system of equations through black-box solvers like Anderson acceleration [4].
Let g ( x 0: T  1 ; x T ) =
˜ h
( x 0: T  1 ; x T )  x 0: T  1 , then we have
[Equation]
This DEQ formulation has multiple beneﬁts.
Solving for all the equilibria simultaneously leads to abetter estimation of the intermediate latent states x t in a fewer number of steps ( i.e.,  t steps for x t ).
This leads to faster convergence of the sampling process as the ﬁnal sample x 0 , which is dependenton the latent states of all the previous time steps, has a better estimate of these intermediate latentstates.
Note that by the same reasoning, the intermediate latent states x t converge faster too.
Thus,we can get images with perceptual quality comparable to DDIM in a signiﬁcantly fewer numberof steps.
Of course, we also note that the computational requirements of each individual step hassigniﬁcantly increased, but this is at least largely offset by the fact that the steps can be executed asmini-batched in parallel over each state.
Empirically, in fact, we often notice signiﬁcant speedupusing this approach on tasks like single image generation.
Efﬁcient Inversion of DDIM
One of the primary strengths of DEQs is their constant memory consumption, for both forward passand backward pass, regardless of their ‘effective depth’.
This leads to an interesting application of
DEQs in inverting DDIMs that fully leverages this advantage along with the other beneﬁts discussedin the previous section.
Algorithm[-
-]
Input: A target image x 0 ⇠ D , ✏ ✓ ( x t , t ) a trained denoising diffusion model, N the total number of epochs .
f denotes the sampling process in Eq (7)
Initialize ˆ x T ⇠ N ( 0 , I ) for epochs from 1 to N do for t =
T, ..., 1 do
Sample ˆ x t  1 = f
(ˆ x t ; ✏ ✓ (ˆ x t , t )) end for
Take a gradient descent step on r
[Equation]
end for
Output: ˆ x
Algorithm
DDIM
Input: A target image x 0 ⇠ D , ✏ ✓ ( x t , t ) a trained denoising diffusion model, N the total number of epochs .
g is the function in Eq.
Initialize ˆ x 0: T ⇠ N ( 0 , I ) for epochs from 1 to N do .
Disable gradient computation x ⇤ 0: T = RootSolver ( g ( x 0: T  1 ); x T ) .
Enable gradient computation
Compute Loss L ( x 0 , x ⇤ 0 )
Use the 1-step grad to compute @
L @ x T
Take a gradient descent step using above end for
Output: x ⇤
4.1 Problem Setup
Given an arbitrary image x 0 ⇠ D , and a denoising diffusion model ✏ ✓ ( x t , t ) trained on a dataset D ,model inversion seeks to determine the latent ˆ x T ⇠ N ( 0 , I ) that can generate an image ˆ x 0 identicalto the original image x 0 through the generative process for DDIM described in Eq.
For an inputimage x 0 , and a generated image ˆ x 0 , this task needs to minimize the squared-Frobenius distancebetween these images:
[Equation]
4.2 Inverting DDIM: A Naive Way
A relatively straightforward way to invert DDIM is to randomly sample x T ⇠ N ( 0 , I ) , and update itvia gradient descent by ﬁrst estimating x 0 using the generative process in Eq.
(7) and backpropagatingthrough this process after computing the loss objective in (13).
The overall process has beensummarized in Algorithm 1.
This process has a large computational overhead.
Every training epoch requires a sequential sampling for all T timesteps.
Optimizing through this generative processwould require the creation of a large computational graph for storing relevant intermediate variablesnecessary for the backward pass.
Sequential sampling further slows down the entire process.
4.3 Efﬁcient Inversion of DDIM with DEQs
Alternatively, we can use the DEQ formulation to develop a much more efﬁcient inversion method.
We provide a high-level overview of this approach in Algorithm 2.
We can apply implicit functiontheorem (IFT) to the ﬁxed point, i.e., (12) to compute gradients of the loss L ( x 0 , x ⇤ 0 ) in (13) w.r.t.
[Equation]
where ( · ) could be any of the latent states x 1 , ..., x T , and J  1 g ✓  x ⇤ 0: T is the inverse Jacobian ofg ( x 0: T  1 ; x T ) evaluated at x ⇤ 0: T .
Refer to [5] for a detailed proof.
Computing the inverse of
Jacobian matrix can become computationally intractable, especially when the latent states x t arehigh dimensional.
Further, prior works [5, 7, 24] have reported growing instability of DEQs duringtraining due to the ill-conditioning of Jacobian.
Recent works [23, 22, 24, 8] suggest that we do notneed an exact gradient to train DEQs.
We can instead use an approximation to Eq.
(14), i.e.,
[Equation]
(15) where M is an approximation of J  1 g ✓  x ⇤ 0: T .
For example, [23, 22, 24] show that setting M
= I ,i.e., 1-step gradient, works well.
In this work, we follow Geng et al.
[24] to further add a dampingfactor to the 1-step gradient.
The forward pass is given by:
[Equation]
[Equation] [Equation]
The gradients for the backward pass can be computed through standard autograd packages.
Weprovide the PyTorch-style pseudocode of our approach in the Appendix.
Using inexact gradients forthe backward pass has several beneﬁts: 1) It remarkably improves the training stability of DEQs;2)
Our backward pass consists of a single step and is ultra-cheap to compute.
It reduces the totaltraining time by a signiﬁcant amount.
Experiments
We consider four datasets that have images of different resolutions for our experiments: CIFAR10(32 ⇥ 32) [41], CelebA (64 ⇥ 64) [45], LSUN Bedroom (256 ⇥ 256) and LSUN Outdoor Church(256 ⇥ 256) [66].
For all the experiments, we use Anderson acceleration as the default ﬁxed pointsolver.
We use the pretrained denoising diffusion models from Ho et al.
[29] for CIFAR10, LSUN
Bedroom, and LSUN Outdoor Church, and from Song et al.
[59] for CelebA.
While training DEQsfor model inversion, we use the 1-step gradient Eq.
(17) to compute the backward pass.
The dampingfactor ⌧ for 1-step gradient is set to 0 .
All the experiments have been performed on NVIDIA RTX
A6000 GPUs.
We provide additional experiment details in the Appendix.
Code will be released alongwith the paper.
5.1 Convergence of DDIM with DEQs
We verify that DEQ converges to a ﬁxed point by plotting the values of k ˜ h ( x 0: T )  x 0: T k 2 over
Anderson solver steps.
As seen in Fig.
1, DEQ converges to a ﬁxed point for generative processesof different lengths.
It is easier to reach simultaneous equilibria on smaller sequence lengths thanlarger sequence lengths.
However, this does not affect the quality of images generated.
We visualizethe latent states of DEQ in Fig.
Our experiments demonstrate that DEQ is able to generate highquality images in as few as 15 Anderson solver steps on diffusion chains that were trained on a much larger number of steps T .
One might note that DEQs converge to a limit cycle for diffusionprocesses with larger sequence lengths.
This is not a limitation as we only want the latent states atthe last few timesteps to converge well, which happens in practice as demonstrated in Fig.
Further,these residuals can be driven down by using more powerful solvers like quasi-Newton methods,e.g., Broyden’s method.
Dataset
[Table]
DEQ
[Table]
Time
[Table]
Time
[Table]
Time
CIFAR
[Table] [Table]
16s
[Table] [Table]
91s
[Table] [Table]
95s
[Table] [Table]
31s
[Table] [Table]
12s
LSUN Bedroom
[Table] [Table]
72s
[Table] [Table]
19s
[Table] [Table]
82s
LSUN Church
[Table] [Table]
77s [Table]
1.68s
[Table] [Table]
99s
.2 Sample quality of images generated with DEQ
We verify that DEQs can generate images of comparable quality to DDIM by reporting Fréchet
Inception Distance (FID) [28] in Table 1.
For the forward pass of DEQ, we run Anderson solver fora maximum of 15 steps for each image.
We report FID scores on 50,000 images, and average timeto generate an image (including GPU time) on 500 images.
We note signiﬁcant gains in wall-clocktime on single-shot image generation with DEQs on images with smaller resolutions.
Speciﬁcally,
DEQs can generate images almost 2 ⇥ faster than the sequential sampling of DDIM on CIFAR-10(32 ⇥ 32) and CelebA (64 ⇥ 64).
We note that these gains vanish on sequences of shorter lengths, and on images with larger resolutions as seen in case of LSUN Bedrooms, and Outdoor Churches(256 ⇥ 256).
We also ﬁnd that on full-batch inference with larger batches, sequential sampling mightoutperform DEQs, as DEQs would have larger memory requirements in this case, i
[Table]
e., processingsmaller batches of size B might be faster than processing larger batches of size BT .
5.3 Model Inversion of DDIM with DEQs
We report the minimum values of squared Frobenius norm between the recovered and target imagesaveraged from 100 different runs in Table 2.
DEQ outperforms the baseline method on all the datasetsby a signiﬁcant margin.
We also plot the training loss curves of DEQ and the baseline in Fig.
Weobserve that DEQ converges faster and has much lower loss values than the baseline method inducedby DDIM.
We also visualize the images generated with the recovered latent states in Fig.
It isworth noting that images generated with DEQs capture more vivid details of the original images, liketextures of foliage, crevices, and other ﬁner details than the baseline.
[Table]
have been reported on 100 images.
See Appendix for detailed training settings.
Related Work
[Figure]
Implicit Deep Learning Implicit deep learning is an emerging ﬁeld that introduces structuredmethods to construct modern neural networks.
Different from prior explicit counterparts deﬁnedby hierarchy or layer stacking, implicit models take advantage of dynamical systems [38, 20, 2],e.g.
, optimization [3, 63, 17, 23, 18], differential equation [13, 19, 61, 26], or ﬁxed-point system [5,6, 27].
For instance, Neural ODE [13] describes a continuous time-dependent system, while Deep
Equilibrium (DEQ) model [5], which is actually path-independent, is a new type of implicit modelsthat outputs the equilibrium states of the underlying system, e.g., z ⇤ from z ⇤ =
f ✓ ( z ⇤ , x ) given theinput x .
This ﬁxed-point system can be solved by black-box solvers [4, 11], and further acceleratedby the neural solver [9] in the inference.
An active research topic is the stability [7, 24, 8] of such asystem as it will gradually deteriorate during training, albeit having strong performance [13, 5, 7].
DEQ has achieved SOTA results on a wide-range of tasks like language modeling [5], semanticsegmentation [6], graph modeling [27, 44], object detection [64], optical ﬂow estimation [8], andgenerative models like normalizing ﬂow [46], with theoretical guarantees [33, 65].
Diffusion Models Diffusion models [58, 29, 59], or score-based generative models [60, 62], arenewly developed generative models that utilize an iterative denoising process to progressively samplefrom a learned data distribution, which actually is the reverse of a forward diffusion process.
Theyhave demonstrated impressive ﬁdelity for text-conditioned image generation [53] and outperformedstate-of-the-art GANs on ImageNet [16].
Despite the superior practical results, diffusion modelssuffer from a plodding sampling speed, e.g., over hours to generate 50k CIFAR-sized images [59].
Toaccelerate the sampling from diffusion models, researchers propose to skip a part of the samplingsteps by reframing the reverse chain [59, 40, 39], or distill the trained diffusion model into a fasterone [47, 56].
Plus, the forward and backward processes in diffusion models can be formulated asstochastic differential equations [62], bridging diffusion models with Neural ODEs [13] in implicitdeep learning.
However, the community still lacks insights into the connection between DEQ anddiffusion models, where we build our work to investigate this.
Model inversion Model inversion gives insights into the latent space of a generative model, as aninability of a generative model to correctly reconstruct an image from its latent code is indicativeof its inability to model all the attributes of image correctly.
Further, the ability to manipulate thelatent codes to edit high-level attributes of images ﬁnds applications in many tasks like semanticimage manipulation [67, 1], super resolution [12, 42], in-painting [15], compressed sensing [10],etc.
For diffusion models like DDPM [29], the generative process is stochastic, which can make modelinversion very challenging.
Many existing works based on diffusion models [48, 15, 62, 31, 35] editimages or solve inverse problems without requiring full model inversion.
Instead, they do so byutilizing existing understanding of diffusion models as presented in some recent works [62, 32, 34].
Diffusion models have been widely applied to conditional image generation [14, 15, 50, 55, 31, 57, 35,49].
Chung et al.
[15] propose a method to reduce the number of steps in reverse conditional diffusionprocess through better initialization, based on the idea of contraction theory of stochastic differentialequations.
Our proposed method is orthogonal to this work; we explicitly model DDIM as a joint,multi-variate ﬁxed point system, and leverage black-box root solvers to solve for the ﬁxed point.
Conclusion
We propose an approach to elegantly unify diffusion models and deep equilibrium models.
Wemodel the entire sampling chain of the denoising diffusion implicit model (DDIM) as a joint, multivariate (deep) equilibrium model.
This setup replaces the traditional sequential sampling processwith a parallel one, thereby enabling us to enjoy speedup obtained from multiple GPUs.
Further,we can leverage inexact gradients to optimize the entire sampling chain quickly, which results insigniﬁcant gains in model inversion.
We demonstrate the beneﬁts of this approach on 1) single-shotimage generation, where we were able to obtain FID scores on par with or slightly better than thoseof DDIM; and 2) model inversion, where we achieved faster convergence.
It is possible to furtherspeedup the sampling process by training a DEQ model to predict the noise at a particular timestep ofthe diffusion chain.
We can jointly optimize the noise prediction network, and the latent variables ofthe diffusion chain, which we leave as future work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] • Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]