Paper: Interpreting and improving diffusion models using the Euclidean distance function (ICLR-2024)

Reviewer comment (Theory):
Theorem 3.1 rests on a strong assumption that is violated in the case of image manifolds. Up to this theorem, the math is not novel and is borrowed from other sources. The contribution of this work is in Theorem 3.1, where the existing math about non-linear projection is used to interpret backward diffusion/denoising. This theorem is the foundation for the rest of the paper, and the derivations about the bounds sit on top of this theorem.

Author response:
We thank the reviewer for the critical review. The main objection raised in this review is the dependence on reach in Theorem 3.1. The reviewer claims that the reach assumption is unrealistic for image manifolds. 

Our main theoretical contribution is Theorem 4.2, which depends only on the existence of a unique projection and Assumption 2, which states that the denoiser is an approximate projection in a relative sense. In Sections 3 and 4 we outlined the reasons for why we believe this is a reasonable assumption, using Theorem 3.1 when the noise level is small and arguing that for large noise levels both the denoising and projection vectors approximately point in the same direction.

Note that Assumption 2 can be tested empirically (see the experiments in Appendix E), and Theorem 4.2 does not require any condition on the reach of the manifold. Theorem 3.1, which requires a condition on reach, is only used to motivate the definition of Assumption 2.

Reviewer comment (Experiments):
The proposed schedules and samplers are tested on a limited number of small-scale datasets.

Author response:
We thank the reviewer for the helpful comments and references. Regarding the comment on only testing the schedules and samplers on small-scale datasets, we also conducted experiments on high-resolution latent-diffusion models such as Stable diffusion. The text-to-image FID scores on 30k samples using the MS-COCO dataset are stated in Fig. 1, with more details in Section 6.2.

Reviewer comment (Theory):
Thm4.2: In real settings, is final sample error (dist_K(x_0)) bounded?

Author response:
* Thm 4.2: Under our relative-error Assumption 2, Theorem 4.2 characterizes the noise schedules that would enable convergence of the DDIM algorithm, as measured by the distance to the data manifold. Intuitively, if the noise schedule has too large steps, the premise of Assumption 2 would be violated during the sampling process and convergence may not be guaranteed. This motivates our definition of admissible schedules and the theorem justifies the need for multiple steps of denoising in diffusion models. 
Regarding bounds of $\text{dist}(x_0)$ in practice, this is possible when the error parameters ($\nu$, $\eta$) can be computed.  For this, one can use ground truth projections (if available). It may also be possible to bound ($\nu$, $\eta$) using denoiser training error, but we defer this to future research.

Reviewer comment (Theory):
Thm4.3: Why does beta star suddenly appear?

Author response:
* Thm 4.3: We apologize if the theorem as stated is unclear.  $\beta^*$ is defined in the statement of Theorem 4.3.  As the theorem establishes, it equals the largest admissible step-size for constant sigma schedules.  Equivalently, it defines the fastest log-linear decrease of sigma.

Reviewer comment (Theory):
Thm4.4: Can we adopt beta for achieving convergence to zero? And how does it relate to the schedule showcased in Section 6?

Author response:
* Thm 4.4: This is an excellent question. Unfortunately, we must defer its answer to future research given the short rebuttal period.

Reviewer comment (Experiments):
The results presented in Table 2 appear to be incomplete, with certain experiments seemingly omitted. It would be beneficial to provide a more comprehensive set of results to ensure a thorough evaluation.

Author response:
The results in the second half of the table are taken from the respective papers and thus some values are missing because they are not originally reported.