Paper: Closed-Form Diffusion Models (ICLR-2024)

Reviewer comment (Theory):
The proposed smoothing method essentially makes an assumption on the underlying data distribution (or the inductive biases), namely that the barycenters of empirical data points also reside within the supports of the true data distribution. This assumption shares similarities with the mixing of training data points.

Author response:
**Inductive bias of our method.**

As the reviewer points out, our method assumes that the barycenters of nearby training points lie within the support of the data distribution. This is a reasonable prior if the data distribution is supported on a manifold whose curvature isn’t too large relative to the number of training samples. As a classical example, Roweis and Saul’s “locally linear embedding” imposes a very similar prior on its inputs.

While this assumption often isn’t true of the image manifold in pixel space (barycenters of images typically don’t look like natural images), by mapping our training data to a latent space with an appropriately-regularized autoencoder, we obtain a latent data manifold whose structure is more favorable to our method. This explains why our method is able to generate plausible CIFAR samples when operating in the latent space of an autoencoder with a tight bottleneck. We have updated our discussion in Section 6.4 to clarify this point.

For future work, we will explore training an autoencoder that enforces local linearity in the latent space; see e.g. Section 4.2 in “Learning Signal-Agnostic Manifolds of Neural Fields” by Du et al. (2021). We hope this strategy will yield a latent data manifold with a locally-linear structure that is highly favorable to our method and produce natural-looking decoded samples. We have updated our conclusion to include this future direction.

Reviewer comment (Experiments):
An alternative approach to the proposed diffusion process would be to directly sample the mixup, which could render the diffusion process unnecessary. For example, a generative model could initially sample \( M \) training data points and then return their barycenter (with probabilities related to the variance) as the generated sample, or use a weighted average by sampling weighting parameters from the \((M-1)\)-simplex. A comparative analysis with these alternative methods should be provided to offer a more comprehensive evaluation (e.g., FID score and visualization).

Author response:
**Alternative sampling processes.**

Naively-implemented “generative mixup” would draw pairs of training points uniformly from the training set and then output some data-independent convex combination of these points. This amounts to sampling from the convex hull of the training set, and is not a suitable inductive bias for most target distributions. One exception is when the target distribution is log-concave; in this case, convex combinations of pairs of target samples have log-likelihood at least as large as one of the endpoint samples. For example, all Gaussian distributions are log-concave. Our fitted Gaussian baseline in Section 6.4 (which performs very poorly) demonstrates that this log-concavity prior is unsuitable for image generation.

On the other hand, directly sampling tuples of $M$ training points with probability depending on their variance and returning their barycenters is challenging, because there are $N^M$ such tuples for training sets of size $N$.

Instead, we have implemented and experimented with a new sampler in Appendix D.4 of the revision, which implements the one-step sampling procedure described in Proposition B.1. This is an extreme case of our existing strategy for reducing the number of sampling steps using a warm start from an unsmoothed CFDM, and uses Gumbel perturbations $u_{i,m}$ to the distance weights in Equation (4) for analytical convenience. Proposition B.1 shows that this method is equivalent to drawing a batch of query points $z_{S-1}$ lying near the support of the data distribution $\rho_1$, drawing $M$ training points $x_i$ with probabilities  $\pi^i_\sigma = \textrm{softmax}\left(-\frac{1}{\sigma} \|z_{S-1} - x_i \|^2 \right)$, and then returning their barycenters.

As expected, this method achieves significantly higher sample throughput than our existing approach ($\sim 4$ times faster for CIFAR generation in latent space). While the decoded samples are reasonable, the speedup comes at a cost to sample quality: Many of our decoded samples retain the appearance of superpositions of training samples.

Reviewer comment (Experiments):
How does \( M \) affect the behavior of \( \sigma \)-CFDM?

Author response:
**Effect of $M$ on CFDM samples.**

We have included an experiment in Appendix E of the revised manuscript demonstrating the impact of $M$ on our method’s samples.

Reviewer comment (Experiments):
Can this method be extended to conditional generation?

Author response:
**Extension to conditional generation.**

In future work, we intend to explore using our method for conditional generation by using Dhariwal and Nichol (2021)'s classifier guidance, which would amount to augmenting our velocity field (7) with the gradient of a pretrained classifier. We have updated our conclusion to include this future direction.

Reviewer comment (Theory):
If my understanding is correct, the motivation is to have a diffusion model with good generalization capacity. Although the smooth diffusion model performs as well as DDPM, it is still unclear how it is connected to the generalization of diffusion models.

Author response:
**Relationship to the generalization of diffusion models.**

Our goal is to design a diffusion model that generalizes controllably while requiring no training and being efficient to sample. While our method is indeed inspired by recent work studying the generalization of diffusion models, our contribution is not a rigorous study of the generalization of existing (neural) diffusion models, but rather introducing a new class of diffusion models that do not require trained approximations to the score function.

In particular: Pidstrigach (2022) shows that a score-based generative model (SGM) can generalize only if the model score incurs unbounded approximation error relative to the true score; otherwise, the model memorizes its training data. We combine this strong result with the observation that neural networks tend to learn overly smooth approximations to their target function and propose a simple, training-free method to induce the error that Pidstrigach (2022) shows to be necessary, in a form similar to that of neural SGMs.

We have updated our introduction and our lead-in to Section 4 to clarify these points.

Reviewer comment (Theory):
Why do we have a closed-form score function as in Section 3? Could you give some references or provide derivations in the appendix?

Author response:
**References for closed-form score formula in Section 3.**

We have updated our discussion in Section 3 to include several references for this formula. In particular, Appendix B.3 of “Elucidating the Design Space of Diffusion-Based Generative Models” by Karras et al. (2022) includes a full derivation of the closed-form score formula; their “ideal denoiser” corresponds to our function $k_t$.

Reviewer comment (Experiments):
Seemingly, you only provide a comparison to DDPM in terms of training time and LPIPS. Could you provide more comprehensive experiments compared to other benchmarks and additional metrics?

Author response:
**Additional benchmarks and metrics.**

In our revised manuscript (Table 1), we have also reported our image generation results in terms of the kernel inception distance (KID). Furthermore, in Appendix D.4 of the revision, we implement and experiment with an alternative sampler. This sampler implements the one-step procedure described in Proposition B.1 and is an extreme case of our existing strategy for reducing the number of sampling steps, using a warm start from an unsmoothed CFDM and Gumbel perturbations to the distance weights in Equation (4) for analytical convenience. Proposition B.1 shows that this equivalent to drawing a batch of query points $z_{S-1}$ lying near the support of the data distribution $\rho_1$, drawing $M$ training points $x_i$ with probabilities  $\pi^i_\sigma = \textrm{softmax}\left(-\frac{1}{\sigma} \|z_{S-1} - x_i \|^2 \right)$, and then returning their barycenters. 

As expected, this method achieves significantly higher sample throughput than our existing approach ($\sim 4$ times faster for CIFAR generation in latent space). While the resulting outputs are reasonable, the speedup comes at a cost to sample quality: Many of our decoded samples retain the appearance of superpositions of training samples.

Reviewer comment (Evaluation):
The presented results are not sufficient and unclear. They are unclear, because even in the Table 1, the proposed method is better on one dataset, whereas being worse on another. It would be helpful to include also other metrics than LPIPS (e.g., FID, SSIM).

Author response:
In our revised manuscript, we have also reported our image generation results in terms of the kernel inception distance (KID).

Reviewer comment (Writing):
The definition of the notation \hat{c}_k (baycenters) is missing.

Author response:
**Definition of notation $\bar{c}_k$ for barycenters.**

We thank the reviewer for pointing out this oversight and have defined this notation in Section 4.2 of the revised manuscript.

Reviewer comment (Theory):
The exponential dependence of the sampling error on T is concerning. Although empirical evidence is provided to justify that this error bound is pessimistic, it also renders the bound unnecessary. Meanwhile, it's unclear if the conclusion that under sigma < 0.4, a large starting T is harmless will generalize to other datasets.

Author response:
**Exponential dependence of the sampling error on $T$.**

As the reviewer notes, the error bound that in Theorem 5.2 is pessimistic in its dependence on $T$. Given that our experiments in Section 6.2 show little accuracy loss for practical values of $\sigma$, we believe this bound to be loose as a consequence of the elementary techniques used to prove it. We would be interested in alternative methods that lead to an error bound which reflects the small error that we observe in practice.

However, the bound in Theorem 5.2 also indicates that the error incurred by starting at $T>0$ scales linearly with $\sigma$. When $\sigma=0$, the smoothed and unsmoothed CFDMs are identical, and one can sample the unsmoothed CFDM (i.e. mixture of Gaussians) at any time without incurring error. For $\sigma>0$, this is no longer true – but it is not obvious how the error incurred by sampling the unsmoothed CFDM at $T>0$ depends on $\sigma$. Our result confirms that this dependence is linear, so small values of $\sigma$ lead to small errors. 

We have updated our discussion in Section 5.2 to clarify this point, namely that the interest of the theorem is the $\sigma$ dependence rather than the $T$ dependence.

**Generality of conclusion that under $\sigma<0.4$, a large starting $T$ is harmless.**

This was true across all datasets considered in this paper—not just the checkerboard dataset used in Section 6.2. 

For example, in Sections 6.3 and 6.4, we used a start time of $T=0.98$ for all image generation experiments with little degradation in sample quality. The fact that $[0,0.4]$ is the appropriate range for $\sigma$ is partially a consequence of having rescaled the training images/latents to lie within the unit ball before sampling and then reversed this rescaling on the generated samples. We have updated our discussion in Section 6.2 to clarify this point. 

Subject to time constraints within the discussion period, we would be happy to run any further experiments the reviewer has in mind to further confirm this observation.

Reviewer comment (Evaluation):
The 3D point cloud experiment is interesting but I don't understand why the proposed method fills the gap there. Could the authors elaborate on this?

Author response:
**Filling in gaps between sparse manifold samples in the 3D point cloud experiment.**

We have updated our discussion of this experiment in Section 6.1 to clarify why our method fills in gaps between sparse samples from the surface.

Reviewer comment (Novelty):
The practical utility of the proposed closed-form models is also unclear. Given that the model can only sample from baycenters of data point tuples, is there a clear case where we would prefer such a model over a trained score model?

Author response:
**Practical utility of CFDMs.**

The benefits of our method over a neural SGM are twofold:

1. Computationally: Our method requires no training and can be sampled very quickly while running on consumer-grade CPUs.
2. Theoretically: Unlike with neural SGMs, we are able to characterize the support of our model’s samples (Theorem 5.1) and can read off the weights of the training points that contributed to each generated sample.

As the reviewer highlights, our model samples barycenters of training points. In light of this fact, sampling in an appropriately-structured latent space (one where the data manifold is “locally linear”) is likely the most realistic path towards obtaining sample quality comparable to neural SGMs on high-resolution image generation tasks. Our CIFAR results in Section 6.4 indicate that this is a promising direction, and we continue to work toward scaling our method up to larger and higher-resolution datasets.