Diffusion models as plug-and-play priors
Abstract
We consider the problem of inferring high-dimensional data x in a model that consists of a prior p ( x ) and an auxiliary differentiable constraint c ( x , y ) on x given some additional information y .
In this paper, the prior is an independently trained denoising diffusion generative model.
The auxiliary constraint is expected to have a differentiable form, but can come from diverse sources.
The possibility of such inference turns diffusion models into plug-and-play modules, thereby allowing a range of potential applications in adapting models to new domains and tasks, such as conditional generation or image segmentation.
The structure of diffusion models allows us to perform approximate inference by iterating differentiation through the ﬁ xed denoising network enriched with different amounts of noise at each step.
Considering many noised versions of x in evaluation of its ﬁ tness is a novel search mechanism that may lead to new algorithms for solving combinatorial optimization problems.
The code is available at https://github.com/AlexGraikos/diffusion_priors .
Introduction
Deep generative models, such as denoising diffusion probabilistic models [DDPMs; 39, 13] can capture the details of very complex distributions over high-dimensional continuous data p ( x ) [30, 7, 1, 38, 43, 15].
The immense effective depth of DDPMs, sometimes with thousands of deep network evaluations in the generation process, is an apparent limitation on their use as off-the-shelf modules in hierarchical generative models, where models can be mixed and one model may serve as a prior for another conditional model.
In this paper, we show that DDPMs trained on image data can be directly used as priors in systems that involve other differentiable constraints.
In our main problem setting, we assume that we have a prior p ( x ) over high-dimensional data x and we wish to perform inference in a model that involves this prior and a constraint c ( x , y ) on x given some additional information y .
That is, we want to ﬁ nd an approximation to the posterior distribution p ( x | y )
∝ p ( x ) c ( x , y ) .
In this paper, p ( x = x 0 , h = { x T , ..., x 1 } ) is provided in the form of an independently trained DDPM over x T , . . .
, x 0 (§2.2), making the DDPM a ‘plug-and-play’ prior.
Although the recent community interest in DDPMs has spurred progress in training algorithms and fast generation schedules [30, 37, 45], the possibility of their use as plug-and-play modules has not been explored.
Furthermore, as opposed to existing work on plug-and-play models (starting from [29]), the algorithms we propose do not require additional training or ﬁ netuning of model components or inference networks.
One obvious application of plug-and-play priors is conditional image generation (§3.1, §3.2).
For example, a denoising diffusion model trained on MNIST digit images might de ﬁ ne p ( x ) , while the constraint c ( x , y ) may be be the probability of digit class y under an off-the-shelf classi ﬁ er.
However, changing the semantics of x , we can also use such models for inference tasks where neural networks struggle with domain adaptation, such as image segmentation: c ( x , y ) constrains the segmentation x to match an appearance or a weak labeling y (§4).
Finally, we describe a path towards using DDPM priors to solve continuous relaxations of combinatorial search problems by treating y as a latent variable with combinatorial structure that is deterministically encoded in x (§5).
1.1 Related work
Conditioning DDPMs.
DDPMs have previously been used for conditional generation and image segmentation [36, 42, 1].
With few exceptions – such as [3], which uses a pretrained DDPM as a feature extractor – these algorithms assume access to paired data and conditioning information during training of the DDPM model.
In [7], a classi ﬁ er p ( y | x t ) that guides the denoising model towards the desired subset of images with the attribute y is trained in parallel with the denoiser.
In [5], generation is conditioned on an auxiliary image by guiding the denoising process through correction steps that match the low-frequency components of the generated and conditioning images.
In contrast, we aim to build models that combine an independently trained DDPM with an auxiliary constraint.
Our approach is also related to work on adversarial examples.
Adversarial samples are produced by optimizing an image x to satisfy a desired constraint c – a classi ﬁ er p ( y | x ) – without reference to the prior over data.
As supervised learning algorithms can ignore the structure in data x , focusing only on the conditional distribution, it is possible to optimize for input x that provides the desired classi ﬁ cation in various surprising ways [41].
In [31], a diffusion model is used to defend from adversarial samples by making images more likely under a DDPM p ( x ) .
We are instead interested in inference , where we seek samples x that satisfy both the classi ﬁ er and the prior.
(Our work may, however, have consequences for adversarial generation.)
Conditional generation from unconditional models.
Works that preceded the recent popularity of DDPMs [29, 9] show how an unconditional generative model, such as a generative adversarial network [GAN; 11] or variational autoencoder [VAE; 21], can be combined with a constraint model to generate conditional samples.
Regarding generative diffusion models, recent literature has focused on utilizing unconditional, pretrained DDPMs as priors to solve linear inverse imaging problems.
Both in [40] and [20], the authors modify the DDPM sampling algorithm, with knowledge of the linear degradation operator, to reconstruct an image consistent with the learned prior and given measurements.
A generalization of these methods in [18] shows how any pretrained denoising network can be used as the prior for solving linear inverse problems.
We also clarify that although the term ‘plug-and-play’ is widely used in the inverse imaging literature we refer to it in the scope of in-domain generation under differentiable constraints, in the same sense as [29].
Latent vectors in DDPMs.
Modeling the latent prior distribution in VAE-like models using a DDPM has been studied in [38, 43].
On the other hand, in §5, we perform inference in the low-dimensional latent space under a pretrained DDPM on a high-dimensional data space.
Our approach to semantic segmentation (§4) is also related to [34], where a prior p ( z ) over latents is used to tune a posterior network
q ( z | x ) .
There, the priors are of relatively simple structure and are sample-speci ﬁ c, rather than global diffusion priors like in this paper.
Method 2.1 Problem setting
Recall that we want to ﬁ nd an approximation to the posterior distribution p ( x | y )
∝ p ( x ) c ( x , y ) , where p ( x ) is a ﬁ xed prior distribution.
Fixing y and introducing an approximate variational posterior q ( x ) , the free energy
[Equation]
is minimized when q ( x ) is closest to the true posterior
, i.e., when KL( q ( x ) ∥ p ( x | y )) is minimized.
When q ( x ) , and the learning algorithm used to ﬁ t it, are expressive enough to capture the true posterior, this minimization yields the exact posterior p ( x | y ) .
Otherwise, q will capture a ‘modeseeking’ approximation to the true posterior [27]; in particular, if q ( y ) is a Dirac delta, it is optimal to concentrate q at the mode of p ( x | y ) .
When the prior involves latent variables h (i.e., p ( x ) = R h p ( x | h ) p ( h ) d h ), the free energy is
[Equation]
We are, in particular, interested in a general procedure for minimizing F with respect to an approximate posterior q ( x ) for any differentiable c when p is a DDPM (§2.2).
A free energy of the same structure was also studied in [43], where a DDPM p ( z ) over a latent space is hybridized as a parent to a decoder p ( x | z ) , with an additional inference model q ( z | x ) trained jointly with both of these models.
On the other hand, we aim to work with independently trained components that operate directly in the pixel space, e.g., an off-the-shelf diffusion model p ( x ) trained on images of faces and an off-the-shelf face classi ﬁ er p ( y | x ) , without training or ﬁ netuning them jointly (§3.2).
2.2 Denoising diffusion probabilistic models as priors
Denoising diffusion probabilistic models (DDPMs) [39, 13] generate samples x 0 by reversing a (Gaussian) noising process.
DDPMs are deep directed stochastic networks:
[Equation]
[Equation] where µ θ and Σ θ are neural networks with learned parameters (often, as in this paper, Σ θ is ﬁ xed to a scalar diagonal matrix depending on t ).
The model starts with a sample from a unit Gaussian x T and successively transforms it with a nonlinear network µ θ ( x t , t ) adding a small Gaussian innovation signal at each step according to a noise schedule.
After T steps, the sample x = x 0 is obtained.
In general, using such a model as a prior over x would require an intractable integration over latent variables
h
= ( x T , ..., x 1 ) :
[Equation]
However, DDPMs are trained under the assumption that the posterior q ( x t | x t − 1 ) is a simple diffusion process that successively adds Gaussian noise according to a prede ﬁ ned schedule β t :
[Equation]
Therefore, if p ( x ) is the likelihood (5) of x under a DDPM, then in the ﬁ rst expectation of (2) we should use q ( h = { x T , ..., x 1 }| x 0 = x ) =
Q
Tt =1 q ( x t | x t − 1 ) .
The simplest approximation to the posterior over x = x 0 is a point estimate:
[Equation]
where by δ we denote the Dirac delta function.
Thus, we can sample x t at any arbitrary time step using the forward noising process as
[Equation]
(8) where α t = 1 − β t and ¯ α t = Q ti =1 α t .
Analogously to [13], we can also extract a conditional
Gaussian q ( x t − 1 | x t , η ) and express the ﬁ rst expectation in (2) as
[Equation]
which after reparametrization [13] leads to
[Equation]
[Equation]
[Equation]
where the stage t noise reconstruction ϵ θ ( x t , t ) is a linear transformation of the model’s expectation µ θ ( x t , t ) :
[Equation]
[Equation]
The weighting w t ( β ) is generally a function of the noise schedule, but in most pretrained diffusion models it is set to 1.
Thus, the free energy in (2) reduces to
[Equation]
[Equation]
The ﬁ rst term is the cost usually used to learn the parameters θ of the diffusion model.
To perform inference under an already trained model ϵ θ , we instead minimize F with respect to η through sampling ϵ in the summands over t .
A similar derivation applies if a Gaussian approximation to the posterior q ( x ) is used (see §A).
Such an approximation allows to model not only a mode of the posterior, but the uncertainty in its vicinity.
We summarize the algorithm for a point estimate q ( x ) as Algorithm 1.
Variations on this algorithm are possible.
Depending on how close to a good mode we can initialize η , this optimization may involve summing only over t ≤ t max < T ; different time step schedules can be considered depending on the desired diversity in the estimated x .
Note that optimization is stochastic and each time it is run it can produce different point estimates of x which are are both likely under the diffusion prior and satisfy the constraint as much as possible.
We observed that optimizing simultaneously for all t makes it dif ﬁ cult to guide the sample towards a mode in image generation applications; therefore, we anneal t from high to low values.
Intuitively, the ﬁ rst few iterations of gradient descent should coarsely explore the search space, while later iterations gradually reduce the temperature to steadily reach a nearby local maximum of p ( x | y ) .
Examples of annealing schedules designed for the tasks demonstrated in §3, 4, 5 are presented in the Appendix (Fig.
B.1).
Another interesting case is when x is parametrized through a latent variable (this can be seen as a case of a hard, non-differentiable constraint: if x is a deterministic function of y , x = f ( y ) , then c ( x , y ) is supported on the corresponding manifold).
Then the procedure in Algorithm 1 can be performed with gradient descent steps with respect to y on
[Equation]
(13) instead of steps 4 and 5.
(For some semantics of the latent representation, one may wish to make the prior on x the pushforward by f of a known prior on the latent y .
In this case, (13) must be weighted by the Jacobian of f at y .)
Experiments: Conditional image generation 3.1 Simple illustration on MNIST
We ﬁ rst explore the idea of generating conditional samples from an unconditional diffusion model on
MNIST.
We train the DDPM model of [7] on MNIST digits and experiment with different sets of
[Figure]
constraints log c ( x , y ) to generate samples with speci ﬁ c attributes.
The examples in Fig.
1 showcase such generated samples.
For the digit in (a) we set the constraint log c to be the unnormalized score of ‘thin’ digits, computed as negative of the average image intensity, whereas in (b) we invert that and generate a ‘thick’ digit with high mean intensity.
Similarly, in (c) and (d) we hand-craft a score that penalizes the vertical and horizontal symmetry respectively, by computing the L 2 distance between the two folds (vertical/horizontal) of the digit x , which leads to the generation of skewed, non-symmetric samples.
We also showcase how the auxiliary constraint c ( x , y ) can be modeled by a different, independently trained network.
The digit in Fig.
1 (e) is generated by constraining the DDPM with a classi ﬁ er network that is separately trained to distinguish between the digit class y = 3 and all other digits.
The auxiliary constraint in this case is the likelihood of the inferred digit, as it is estimated by the classi
ﬁ er. Finally, for (f) we multiply horizontal symmetry and digit classi ﬁ er constraints, prompting the inference procedure to generate a perfectly centered and symmetric digit.
Details of model training and inference can be found in the Appendix (§B.1).
3.2 Using off-the-shelf components for conditional generation of faces
We consider the generation of natural images with a pretrained DDPM prior and a learned constraint.
We utilize the pretrained DDPM network on FFHQ-256 [19] from [3] and a pretrained ResNetface attribute classi ﬁ er on CelebA [25].
The attribute classi ﬁ er computes the likelihood of presence of various facial features y in a given image x , as they are de ﬁ ned by the CelebA dataset.
Examples of such features are no beard , smiling , blond hair and male .
To generate a conditional sample from the unconditional DDPM network we select a subset of these and enforce their presence or absence using the classi ﬁ er predicted likelihoods as our constraint c .
If y is a set of attributes we wish to be present, the constraint log c ( x , y ) can be expressed as
[Equation]
We only strictly enforce a small subset of facial attributes and therefore x is allowed to converge towards different modes that correspond to samples that exhibit, in varying levels, the desired features.
In Fig.
2 we demonstrate our ability to infer conditional samples x with desired attributes y , using only the unconditional diffusion model and the classi ﬁ er p ( y | x ) .
In the ﬁ rst row, we show the results of the optimization procedure of Algorithm 1 for various attributes.
The classi ﬁ er objective c ( x , y ) manipulates the image with the goal of making the classi ﬁ er network produce the desired attribute predictions, whereas the diffusion objective attempts to pull the sample x towards the learned distribution p ( x ) .
If we ignored the denoising loss, the result would be some adversarial noise that fools the classi ﬁ er network.
The DDPM prior, however, is strong enough to guide the process towards realistic-looking images that simultaneously satisfy the classi ﬁ er constraint set.
We notice that the generated samples x , although having converged towards a correct mode of p ( x ) , still exhibit a noticeable amount of noise related to the optimization of classi ﬁ er objective.
To address that, inspired by [31], we simply denoise the image using the DDPM model alone, starting from the low noise level t = 200 so as to retain the overall structure.
The results of this denoising are shown in the second row of Fig.
In Fig.
3 we showcase the intermediate steps of the optimization process for inference with the conditions blond hair + smiling + not male , thus solving a problem like that studied in [8] using only independently trained attribute classi ﬁ ers and an unconditional generative model of faces.
The sample x is initialized with Gaussian noise N ( 0 , I ) , and as we perform gradient steps with decreasing values of t , we observe facial features being added in a coarse-to- ﬁ ne manner.
[Figure] [Figure]
In the Appendix (§B.2) we provide additional samples and further discuss the sample quality in comparison to unconditional generation.
We also present results on inference with con ﬂ icting attributes as well as common failure cases.
Experiments: Semantic image segmentation
We test the applicability of diffusion priors in discrete tasks, such as inferring semantic segmentations from images.
For this purpose, we use the EnviroAtlas dataset [32] which is composed of 5-class, 1m-resolution land cover labels from four geographically diverse cities across the US; Pittsburgh,
PA, Durham, NC, Austin, TX and Phoenix, AZ.
We only have access to the high resolution labels from Pittsburgh, and the task is to infer the land cover labels in the other three cities, given only probabilistic weak labels ℓ weak derived from coarse auxiliary data [34].
We use Algorithm 1 to perform an inference procedure that does not directly take imagery as input, but uses constraints derived from unsupervised color clustering.
We use only cluster indices in inference, making the algorithm dependent on image structure, but not color.
Local cluster indices as a representation have a promise of extreme domain transferability, but they require a form of a combinatorial search which matches local cluster indices to semantic labels so that the created shapes resemble previously observed land cover, as captured by a denoising diffusion model of semantic segmentations.
DDPM on semantic pixel labels.
We train a DDPM model on the 14 -resolution one-hot representations of the land cover labels, using the U-Net diffusion model architecture from [7].
To convert the one-hot diffusion samples to probabilities we follow [15] and assume that for any pixel i in the inferred sample x , the distribution over the label ℓ is, p ( ℓ i ) ∝ R 1 . 5 0 .
5 N ( x ℓ i | η i , σ ) , where σ is user-de ﬁ ned a parameter.
We chose this approach for its simplicity and ease to apply in our inference setting of Algorithm 1.
Alternatively, we could use diffusion models for categorical data [14] with the appropriate modi ﬁ cations to our inference procedure.
Samples drawn from the learned distribution are presented in Fig 4.
Inferring semantic segmentations.
In order to infer the segmentation of a single image, under the diffusion prior, we directly apply Algorithm 1 with a hand-crafted constraint c which provides structural and label guidance.
To construct c , we ﬁ rst compute a local color clustering z of input the image (§B.3 in the Appendix).
In addition, we utilize the available weak labels ℓ weak [34] and
[Figure]
[Figure]
force the predicted segments’ distribution to match the weak label distribution when averaged in non-overlapping blocks.
We combine the two objectives in a single constraint c ( x , z , ℓ weak ) by (i) computing the mutual information between the color clustering z and the predicted labels x , transformed into a valid probability distribution from the inferred one-hot vectors, in overlapping image patches and (ii) computing the negative KL divergence between the average predicted distribution and the distribution given by the weak labels in non-overlapping blocks
[Equation]
Empirically, we ﬁ nd that we can reduce the number of optimization steps needed to perform inference by initializing the sample x with the weak labels ℓ weak instead of random noise, allowing us to start from a smaller t i .
Examples of images and their inferred segmentations are shown in Fig.
Domain transfer with inferred samples.
The above inference procedure is agnostic to colors by design, and we expect it to have a greater ability to perform in new areas than the approach in [34], which still ﬁ netunes networks that take raw images as input.
We also investigate domain transfer approaches where patches segmented using the the diffusion prior are used to train neural networks for fast inference.
We pretrain a standard U-Net inference network p ( x | I ) solely on 20k batches of 16 randomly sampled 64 × 64 image patches in PA.
We randomly sample 640 images in each of the other geographies and generate semantic segmentations using our inference procedure, then ﬁ netune the inference network on these segmentations.
This network is then evaluated on the entire target geography.
The results in Table 1 demonstrate that this approach to domain transfer is comparable with the state-of-the-art work of [34] for weakly-supervised training.
The naïve approach of training a U-Net only on the available high-resolution PA data (PA supervised) fails to generalize to the geographically
[Table]
different location of Phoenix, AZ.
Similarly, the model of [33], which is a US-wide high-resolution land cover model trained on imagery and labels [Table] and multi-resolution auxiliary data over the entire contiguous US also suffers.
When the weak labels are provided as input (PA supervised + weak) the results can improve signi ﬁ cantly.
Experiments: Continuous relaxation of combinatorial problems
So far [Table] we have considered inference under a DDPM prior and a differentiable constraint c ( x , y ) .
We consider the case of a ‘hard’ constraint [Table] where y is a latent vector deterministically encoded in an image x ( x = f ( y ) ) and we have a DDPM prior over images p DDPM ( x ) .
We will use the variation of Algorithm 1 described at the end of §2.2 to obtain a point estimate of the distribution over y , p ( y )
∝ p DDPM ( f ( y )) .
We illustrate this in the setting of a well-known combinatorial problem, the traveling salesman problem (TSP).
Recall that a Euclidean traveling salesman problem on the plane is described by
N points v 1 , . . .
, v N ∈ R 2 , which form the vertex set of a complete weighted graph G , where the weight of the edge from v i to v j is the Euclidean distance ∥ v i − v j ∥ .
A tour of G is a connected subgraph in which every vertex has degree 2.
The TSP is the optimization problem of ﬁ nding the tour with minimal total weight of the edges, or, equivalently, a permutation σ of { 1 , 2 , . . .
, N } that minimizes
[Equation]
Although the general form of the TSP is NP-hard, a polynomial-time approximation scheme is known to exist in the Euclidean case [2, 28] and can yield proofs of tour optimality for small problems.
Humans have been shown to have a natural propensity for solving the Euclidean TSP (see [26] for a survey).
Humans construct a tour by processing an image representation of the points v 1 , . . .
, v N through their visual system.
However, the optimization algorithms in common use for solving the
TSP do not use a vision inductive bias, instead falling into two broad categories:
• Discrete combinatorial optimization algorithms and ef ﬁ cient integer programming solvers, studied for decades in the optimization literature
]; • More recently, there has been work on neural nets, trained by reinforcement learning or imitation learning, that build tours sequentially or learn heuristics for their (discrete) iterative re ﬁ nement.
Successful recent approaches [6, 23, 16, 17, 4] have used Transformer [44] and graph neural network [22] architectures.
The algorithm we propose using DDPMs is a hybrid of these categories: it reasons over a continuous relaxation of the problem, but exploits the learning of generalizable structure in example solutions by a neural model.
In addition, ours is the ﬁ rst TSP algorithm to mimic the convolutional inductive bias of the visual system.
Encoding function.
Fix a set of points v 1 , . .
. , v N ∈ [0 , 1] × [0 , 1] .
We encode an symmetric N × N matrix with 0 diagonal A as a 64 × 64 greyscale image f
( A ) by superimposing: (i) raster images of line segments from v i to v j with intensity value A ij for every pair ( i, j ) , and (ii) raster images matrix w.r.t.
denoising model
Recover tour
Input t =t =t =t =
t =
Extracted + 2-opt
Oracle of small black dots placed at v i for each i .
For example, if A is the adjacency matrix of a tour, then f ( A ) is a visualization of this tour as a 64 × 64 image.
Diffusion model training.
We use a dataset of Euclidean TSPs, with ground truth tours obtained by a state-of-the-art TSP solver [10], from [23] (we consider two variants of the dataset, each with ∼ 1.5m training graphs: with 50 vertices in each graph and with a varying number from 20 tovertices in each graph).
Each training tour is represented via its adjacency matrix A and encoded as an image f
( A ) .
We then train a DDPM with the U-Net architecture from [7] on all of such encoded image.
Model and training details can be found in the Appendix (§B.4).
Some unconditional samples from the trained DDPM are shown in Fig.
6; most samples indeed resemble image representations of tours.
Solving new TSPs.
Suppose we are given a new set of points v 1 , . .
. , v N .
Solving the TSP requires ﬁ nding the adjacency matrix A of a tour of minimal length.
As a differentiable relaxation, we set A = S + S ⊤ , where S is a stochastic
N × N matrix with zero diagonal (parametrized via softmax of a matrix of parameters over rows).
We run the inference procedure using the trained
DDPM p
DDPM ( f ( A )) as a prior to estimate A .
The hyperparameters and noise schedule are described in §B.4.
Examples of the optimization are shown in Fig.
Although the inferred A is usually sharp (i.e., all entries close to 0 or 1), rounding A to 0 or 1 does not always give the adjacency matrix of a tour (see, for example, the top row of Fig. 7; other common incorrect outputs include pairs of disjoint tours).
To extract a tour from the inferred A , we greedily insert edges to form an initial proposal, then re ﬁ ne it using a standard and lightweight combinatorial procedure, the 2-opt heuristic [24] (amounting to iteratively uncrossing pairs of edges that intersect).
The entire procedure is shown in Fig.
7, and full details can be found in the Appendix (§B.4).
Results.
We evaluate the trained models on test sets of 1280 graphs each with N = 50 and N =vertices.
We report the average length of the inferred tour and the gap (discrepancy from the length of the ground truth tour) in Table 2 (left), from which we make several observations.
• The right side of Table 2 shows the number of 2-opt (edge uncrossing) steps performed in the re ﬁ nement step of the algorithm when the inference algorithm is run for varying numbers of steps.
Running the inference with more steps results in extracted tours that are closer to local minima with respect to the 2-opt neighbourhood, indicating that the DDPM encodes meaningful information about the shape of tours.
[Table]
• The DDPM inference is competitive with recent baseline algorithms that do not use beam search in generation of the tour (those shown in the table).
These baseline algorithms improve when beam search decoding with very large beam size is used, but encounter diminishing returns as the computation cost grows.
Our performance on the 100-vertex problems is similar to [23] with the largest beam size they report (5000), which has 2.18% gap, while having similar computation time.
• The model trained on problems with 50 nodes performs almost identically to the model trained on problems with 50 or fewer nodes, and both models generalize better than baseline methods from 50-node problems to the out-of-distribution 100-node problems.
We emphasize a unique feature of our algorithm: all ‘reasoning’ in our inference procedure happens via the image space.
This property also leads to sublinear computation cost scaling with increasing size of the graph – as long as it can reasonably be represented in a 64 × 64 image – since most of the computation cost of inference is borne by running the denoiser on images of a ﬁ xed size.
In the
Appendix (§B.4)
we explore the generalization of the model trained on 20- to 50-node TSP instances to problems with 200 nodes and discuss potential extensions.
Conclusion
We have shown how inference in denoising diffusion models can be performed under constraints in a variety of settings.
Imposing constraints that arise from pretrained classi ﬁ ers enables conditional generation, while common-sense conditions, such as mutual information with a clustering or divergence from weak labels, can lead to models that are less sensitive to domain shift in the distribution of conditioning data.
A notable limitation of DDPMs, which is inherited by our algorithms, is the high cost of inference, requiring a large number of passes through the denoising network to generate a sample.
We expect that with further research on DDPMs for which inference procedures converge in fewer steps [37, 45], plug-and-play use of DDPMs will become more appealing in various applications.
Finally, our results on the traveling salesman problem illustrate the ability of DDPMs to reason over uncertain hypotheses in a manner that can mimic human ‘puzzle-solving’ behavior.
These results open the door to future research on using DDPMs to ef ﬁ ciently generate candidates in combinatorial search problems.
Acknowledgments
The authors thank the anonymous NeurIPS 2022 reviewers for their comments.
All authors are funded by their primary institutions.
Partial support was provided by the NASA
Biodiversity program (Award 80NSSC21K1027), NSF grants IIS-2123920 and IIS-2212046, and the
Partner University Fund 4D Vision award.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately re ﬂ ect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the conclusion and discussion throughout the paper.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Although no immediate negative societal impacts are expected, researchers should bear in mind the risks of ﬂ exible conditional generation of images, e.g., for creating ‘deep fakes’.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] For most experiments; see the Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the Appendix and relevant experiment sections.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Main experiments were run one time.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See the relevant experiment sections.
(b) Did you mention the license of the assets?
[No] But all datasets used are free to use for research purposes; see the relevant citations.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identi ﬁ able information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[-17) where α t = 1 − β t and ¯ α t = (cid:81) ti =
-]
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]