Paper: The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing (ICLR-2024)

Reviewer comment (Experiments):
While the paper shows that SDE outperforms ODE baselines in inpainting and image-to-image translation, it lacks a comparison with the latest methods in these tasks.

Author response:
Thank you for your valuable comment. The preliminary goal of the inpainting and image-to-image translation experiments is to show the versatility of SDE in general image editing, instead of improving the latest methods. Our dragging experiment against the latest baselines is able to show the effectiveness of the SDE formulation. Besides, based on the results, we believe that the SDE formulation is orthogonal to other improvements for a specific task and applies to the latest ODE-based methods.

Moreover, in our submission, we conduct experiments on recent work including SD (CVPR 2022), DiffEdit (ICLR 2023), DDIB (ICLR 2023), and DragDiffusion (arXiv 2023.06). Nevertheless, if you would like to point out specific papers, we are happy to discuss the relation and will try to do the experiments.

Reviewer comment (Evaluation):
The paper provides the cost time for the dragging task but does not provide similar information for inpainting and image-to-image translation.

Author response:
Thank you for your valuable question. In the last paragraph (on page 9) of Sec. 5.4 in our submission, we introduce that the SDE counterpart takes nearly the same time as the direct ODE baseline in all experiments. To more clearly demonstrate the time efficiency, we list the cost time per image for all inpainting and image-to-image translation experiments here and we will add it to the revision as soon as possible.


|       | Inpainting | DiffEdit   | DDIB                   |
|-------|------------|------------|------------------------|
| SDE   | 3.06s      | 4.82s      | 299.20s                |
| ODE   | 3.11s      | 4.86s      | 300.18s                |
| Device| NVIDIA A100 | NVIDIA A100 | NVIDIA GeForce RTX 3090|



From the table, we can see that the time efficiency of both SDE and ODE mainly depends on the number of function evaluations, which are the same using the same sampling strategy. Therefore, when applied to the latest ODE baselines, SDE-based methods would have a similar time cost. Also see the discussion on comparison with the latest methods in our response to your weakness1.

Reviewer comment (Experiments):
It is suggested to compare the performance of SDE in inpainting and image-to-image translation with the latest methods.

Author response:
Thank you for your valuable comment. The preliminary goal of the inpainting and image-to-image translation experiments is to show the versatility of SDE in general image editing, instead of improving the latest methods. Our dragging experiment against the latest baselines is able to show the effectiveness of the SDE formulation. Besides, based on the results, we believe that the SDE formulation is orthogonal to other improvements for a specific task and applies to the latest ODE-based methods.

Moreover, in our submission, we conduct experiments on recent work including SD (CVPR 2022), DiffEdit (ICLR 2023), DDIB (ICLR 2023), and DragDiffusion (arXiv 2023.06). Nevertheless, if you would like to point out specific papers, we are happy to discuss the relation and will try to do the experiments.

Reviewer comment (Evaluation):
Provide the running time of their proposed methods for inpainting and image-to-image translation tasks, and include the running time of the latest methods in these tasks to provide a more complete picture of the computational efficiency.

Author response:
Thank you for your valuable question. In the last paragraph (on page 9) of Sec. 5.4 in our submission, we introduce that the SDE counterpart takes nearly the same time as the direct ODE baseline in all experiments. To more clearly demonstrate the time efficiency, we list the cost time per image for all inpainting and image-to-image translation experiments here and we will add it to the revision as soon as possible.


|       | Inpainting | DiffEdit   | DDIB                   |
|-------|------------|------------|------------------------|
| SDE   | 3.06s      | 4.82s      | 299.20s                |
| ODE   | 3.11s      | 4.86s      | 300.18s                |
| Device| NVIDIA A100 | NVIDIA A100 | NVIDIA GeForce RTX 3090|



From the table, we can see that the time efficiency of both SDE and ODE mainly depends on the number of function evaluations, which are the same using the same sampling strategy. Therefore, when applied to the latest ODE baselines, SDE-based methods would have a similar time cost. Also see the discussion on comparison with the latest methods in our response to your weakness1.

Reviewer comment (Novelty):
The core idea that SDE beats ODE in image editing seems similar to CycleDiffusion [6]. This article seems to just generalize this phenomenon to multiple tasks and verify them. Overall, the innovative contribution seems insufficient.

Author response:
Thank you for your valuable question. we emphasize our contributions and explain the differences and links between our work and CycleDiffusion in detail:

1. We introduce a unified perspective for multiple diffusion-based image editing tasks (I2I, inpainting, dragging) that encompasses various editing techniques. In contrast, CycleDiffusion focuses on I2I.  **Given the fact that existing research in such tasks is mainly independent, we believe the unified perspective is a nontrivial and valuable contribution.**
2. **To our knowledge, we present the FIRST theoretical analysis (which is one of our core contributions) on the superiority of SDE over ODE in image editing, while all existing methods including Cycle-Diffusion are purely empirical.**
3. We present a new algorithm (SDE-Drag) as well as a new benchmark for open-set image dragging. Besides the SDE formulation, **the core contribution of SDE-Drag is a simple copy&paste manipulation strategy for the latent variables**, which significantly outperforms existing optimization-based methods. Such contributions are unique in our paper compared to existing work including Cycle-Diffusion.
4. **The core idea of Cycle-Diffusion is to construct an invertible SDE algorithm**, inspired by the ODE methods. **It is mainly compared with non-invertible SDE baselines**. However, our core idea is that **the general SDE formulation (including Cycle-Diffusion and others) is better than ODE in editing**. Our theory applies to other SDE formulations and we apply other SDE-based methods in the inpainting task other than Cycle-Diffusion.

We believe the above contributions make our paper distinct from Cycle-Diffusion.

Reviewer comment (Novelty):
The authors claim to propose a unified probabilistic formulation for diffusion-based image editing. However, many related image editing/I2I methods are not mentioned, e.g., RePaint [1], DDNM [2], DPS [3], T2I-Adapter [4], and ControlNet [5].

Author response:
Thank you for your insightful suggestion and for pointing out the related works, as briefly discussed below.

Our framework analyzes the difference between SDE and ODE when the prior distribution mismatch is due to manipulation or domain transformation during inference. Therefore, it also applies to RePaint, DDNM, and DPS. However, training-based methods like T2I-Adapter and ControlNet without mismatch, are excluded from our framework.

We will add the above discussion in our revision.

Reviewer comment (Evaluation):
The problematic of the paper (end of Section 3.1) is never clearly answered. Actually, I do not think that the paper properly gives the answer to this question. This is linked to the following point.

Author response:
Thank you for your meticulous and insightful comment. We agree that the final objective of image editing is to analyze $\tilde p_0$ and $q_0$ under certain divergence. Below, we explain **why making $\tilde p_0$ and $p_0$ close (i.e., the contribution of this paper) is meaningful both theoretically and empirically**.

Theoretically, we indeed assume that the diffusion model characterizes the true score functions, namely $KL(p_0 || q_0)=0$ $(p_0=q_0)$ in the submission (See Line 3, Paragraph 1 in Sec. 3.2). Then our theory on $KL(\tilde p_0|| p_0)$ holds for $KL(\tilde p_0||q_0)$ as desired. **Intuitively, the assumption can be relaxed to a bounded approximation error of the score function**, i.e., $\mathbb E_{q_t}[ \|s_{\mathbf \theta}(\mathbf x_t, t) - \nabla_{\mathbf x_t} \ln q_t \|^2]  < \epsilon$ based on the latest theoretical work [a, b, c]. In particular, if $\epsilon \rightarrow 0$, then the total variation distance (TV) for $q_0$ and $p_0$ tends to zero, namely, $TV(q_0, p_0) \rightarrow 0$, making $p_0$ an accurate approximation for $q_0$.

Empirically, **experimental results suggest that $p_0$ is a good surrogate for $q_0$, especially compared to $\tilde p_0$**. For instance, Stable Diffusion [d] can generate high fidelity images in human perception. Besides, Tab. 5（on page 26）in our submission shows that **the FID for sampling (namely $p_0$) is significantly lower than the FID for editing (ODE baseline, namely $\tilde p_0$)**, suggesting that $p_0$ is much closer to $q_0$ than $\tilde p_0$, making it meaningful to minimize $KL(\tilde p_0|| p_0)$.

We sincerely appreciate the valuable comments. We will revise the paper to clarify that the final objective of image editing is to analyze $\tilde p_0$ and $q_0$ and present the above discussion on why minimizing $KL(\tilde p_0 || p_0)$ is meaningful. If you have any further concerns, please feel free to post more comments.


[a] Chen, Sitan, et al. "Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions." *arXiv preprint arXiv:2209.11215* (2022).

[b] Lee, Holden, Jianfeng Lu, and Yixin Tan. "Convergence of score-based generative modeling for general data distributions." *International Conference on Algorithmic Learning Theory*. PMLR, 2023.

[c] Chen, Sitan, Giannis Daras, and Alex Dimakis. "Restoration-degradation beyond linear diffusions: A non-asymptotic analysis for ddim-type samplers." International Conference on Machine Learning. PMLR, 2023.

[d] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Reviewer comment (Theory):
In the paragraph "Samples" from Section 2, the term "equivalent" is not true. Sampling is not "equivalent" to discretization. Moreover, discretizing (4) or (5) does not enable sampling from \(q_0\) if the score is not perfectly matched with \(\epsilon_\theta\).

Author response:
Thank you very much for pointing out our mistakes. We will change our expression: **Samples can be generated from the diffusion model by discretizing the reverse SDE (Eq.(4)) or ODE (Eq.(5)) from $T$ to $0$**.

Reviewer comment (Writing):
The ODE inversion process explanation should be clarified. The links and differences between deterministic ODE inversion and random Gaussian noise should be explained.

Author response:
Thank you for your insightful suggestion. We are not sure about the "random Gaussian noise" in your comment. We guess you mean the $\bar{w}_s$ in Eq. (6) in the submission. **Note that when using ODE inversion, we set $\eta=0$, which implies that $\sigma_s = 0$. Namely, the coefficient of the random Gaussian noise in Eq. (6) is zero** and it is a deterministic process. If we misunderstand your comment, please feel free to post further questions.

In case you are not familiar with ODE inversion, it generally aims to obtain $\mathbf x_T$ given $\mathbf x_0$ based on the invertibility of the ODE. According to Eq. (5) in the submission, the general formulation of ODE inversion is defined as $\mathbf x_T = \mathbf x_0 + \int_0^T \left[f(t) \mathbf x_t + \frac{g^2(t)}{2 \sqrt{1 - \alpha_t}}  \mathbf \epsilon_{\mathbf \theta}(\mathbf x_t, t)\right] dt$. This implies that for each given data point $\mathbf x_0$, there exists a uniquely determined $\mathbf x_T$ associated with it through ODE inversion. 

As for the specific DDIM sampler, one step is Eq. (6) in our submission. We rewrite it here for clarity (only ODE formulation, i.e., $\eta=0$): $\mathbf x_t = \sqrt{\alpha_{t}} \left( \frac{\mathbf x_s - \sqrt{1 - \alpha_s} \mathbf \epsilon_{\mathbf \theta}(\mathbf x_s, s)}{\sqrt{\alpha_s}} \right) + \sqrt{1 - \alpha_{t}} \mathbf \epsilon_{\mathbf \theta}(\mathbf x_s, s)$. When $t>s$, it is a step of DDIM inversion. 

We will provide the details of ODE inversion in the Appendix in the revision.

Reviewer comment (Theory):
The "mild assumption" should be detailed, at least in the Appendix.

Author response:
Thank you for your insightful suggestion. For completeness, we will list the assumption in Appendix D, which is the same as Assumption A.1. of [h].

[h] Lu, Cheng, et al. "Maximum likelihood training for score-based diffusion odes by high order denoising score matching." *International Conference on Machine Learning*. PMLR, 2022.

Reviewer comment (Theory):
If the noise is fixed, is CycleSDE still an SDE?

Author response:
Thank you for your question. We interpret "noise is fixed" to mean that the noises in the backward process of Cycle-SDE are solved by the forward process. Actually, given $\mathbf x_{t_0}$, the noises in the backward process are not iid standard Gaussian noise like the origin SDE, but instead a combination of noises in the forward process.  Consequently, the reverse process is stochastic, and thus Cycle-SDE is still an SDE.

Reviewer comment (Theory):
Is the log-Sobolev inequality likely to be verified in practice?

Author response:
Thank you for your question. The log-Sobolev inequality is a **classical and standard assumption in theoretical analyses of Langevin-type algorithms [i, j], which cannot be verified in large-scale and high-dimensional real data** like LAION [e]. Note that without the assumption, although it is nontrivial to obtain the strong linear convergence result, our main results (i.e., Theorem 1,2&3) still hold. 

[i] Chewi, Sinho, et al. "Analysis of Langevin Monte Carlo from Poincar\'e to Log-Sobolev." arXiv preprint arXiv:2112.12662 (2021).

[j] Vempala, Santosh, and Andre Wibisono. "Rapid convergence of the unadjusted Langevin algorithm: Isoperimetry suffices." Advances in neural information processing systems 32 (2019).