Paper: The Convergence of Variance Exploding Diffusion Models under the Manifold Hypothesis (ICLR-2024)

Reviewer comment (Theory):
It was claimed in the introduction that the underlying algorithm converges at rate $\exp(-T)$. However, upon looking at Theorem 2, there is a term which is linear in $T$ (under the Wasserstein distance). The authors may comment and explain this "discrepancy" -- otherwise it seems to be that the abstract is exaggerated.

Author response:
**Weakness 1. The discussion on the faster forward convergence $\exp{(-T)}$ and $\operatorname{Poly}(T)\\bar{\gamma}_K$ term.** 

When discussing the faster convergence rate of our new forward process, we focus on the reverse beginning error term, which corresponds to the forward process, instead of the discretization error term. As shown in Section 5, a faster forward convergence guarantee is important to balance different error sources and achieve great sample complexity.

For the $\operatorname{Poly}(T)\bar{\gamma}_K$ term, which corresponds to the discretization scheme and smallest stepsize $\bar{\gamma}_K$, we can choose small enough stepsize to eliminate the influence of $\operatorname{Poly}(T)$. Furthermore, $\operatorname{Poly}(T)\bar{\gamma}_K$ term for the discretization terms is widely appeared in the literature (Theorem 1 of [1] under the  Wasserstein distance; Theorem 2 of [2] under the TV distance). We note that under the choice of $T$ and $\bar{\gamma}_K$ in Corollary 2 of [1], the first discretization term of Theorem 1 in [1] depends on $T^2 \sqrt{\bar{\gamma}_K}$.

We have now made the definition of the faster convergence rate clearer in the abstract in the revision of our paper (highlighted in green).

Reviewer comment (Theory):
In most theorems/corollaries, the authors distinguish "aggressive" $\beta_t = t^2$, and "conservative" $\beta_t$'s (e.g. $\beta_t = t$). I wonder if there is some "phase transition" at exponent $2$ (i.e. $\beta_t = t^\alpha$ with $\alpha < 2$ and $\alpha = 2$). The authors may comment on this point.

Author response:
**Weakness 3. Transition situation $\beta_t = t^{\alpha_1}, \alpha_1\in [1,2]$ between conservative $t$ and the most aggressive $t^2$.** 

We use $\tau  = T^2$ as an example and discuss the influence of different $\beta_t =t^{\alpha_1}, \alpha_1\in [1,2]$ for the forward and reverse process. For the forward process, the "phase transition" appears at $1+\ln(T-\ln(T))/\ln(T)$. When $\alpha_1 \geq 1+\ln(T-\ln(T))/\ln(T)$ is aggressive, the forward convergence rate is faster than $\exp(-T)$, which means the influence of $T$ is logarithmic factors and achieve great balance in reverse SDE (Section 5). For the analysis of the unified tangent-based framework, as discussed in section 6, whether $\alpha_1$ can be aggressive depends on the form of the reverse process.

We have now added more details and discussion about the different phases of $\alpha_1$ in Section 4 in the revision of our paper (highlighted in green).

Reviewer comment (Experiments):
While the paper introduces new concepts and theoretical results, it would have been beneficial to have more specific experiments on datasets.

Author response:
**Weakness 1: The additional synthetic experiments.** 

We use pure VESDE as a baseline and add more synthetic experiments in the main text to show the advantage of our new forward process with aggressive and conservative drift terms. More specifically, For aggressive drift VESDE, we show that the new process balances the different error sources. For conservative drift VESDE, we further consider the approximated score function and show that the conservative drift VESDE can improve the quality of generated distribution without training. More specifically, the conservative drift VESDE directly uses the models training by pure VESDE and reduces the influence of the large data mean and variance to obtain better images. We verify this phenomenon in different sampling methods (Euler-Maruyama method for reverse SDE and RK45 for PFODE), different $T$, and different synthetic datasets. We also calculate the KL divergence between the generated distribution and the original distribution to support our augmentation.

We have now added more experiment results and discussion about the advantages of our new process in Section 7 and Appendix F in the revision of our paper (highlighted in green).

Reviewer comment (Evaluation):
While it's understood that the introduced VESDE exhibits a faster convergence rate than VPSDE, there's no guarantee that it offers better fidelity.

Author response:
**Weakness 3: The better fidelity with drift VESDE.** 

The above discussion shows that drift VESDE generated distributions closer to the original distribution in KL divergence on the synthetic experiments. Then, we explain why drift VESDE has better fidelity than pure VESDE from the theoretical perspective.

Before comparing drift VESDE and previous pure VESDE, we update Corollary 1 by more careful calculation on the time discretization error (Lemma 17) and the early stopping parameter (Lemma 23). With this result, we show that the faster forward convergence rate allows balance in three error sources and finally leads to better sample complexity compared to the previous guarantee for pure VESDE [4]. The results show that the drift VESDE can use smaller steps to achieve the same accuracy, which means this model can use the same steps to achieve better fidelity compared to pure VESDE.

We have now added a detailed comparison and discussion on the sample complexity of pure VESDE in Remark 1 in the revision of our paper (highlighted in green).

Reviewer comment (Theory):
I find it puzzling whether the variance-exploding property has any utility beyond qualitative convergence guarantees.

Author response:
**Weakness 2: The utility of the variance-exploding property.** 

As shown in Figure 3 of [1], the VESDE (Eq (3) in our paper) with $\sigma_t^2 = t^2$ has great PFODE curvature compared to VPSDE and VESDE with $\sigma_t^2 = t$, which relies have on the large variance term $\sigma_t^2 =t^2$ and $q_T = \mathcal{N}(0, T^2 \mathbf{I})$. Based on this PFODE curvature, VESDE with $\sigma_t^2 = t^2$ performs well in multi-step [2] and one-step [3] image generation. Furthermore, for our new drift VESDE, we show that it performs better than pure $\sigma_t^2=t^2$ or  $\sigma_t^2=t$ on the synthetic experiments since it can balance different error sources.

Reviewer comment (Novelty):
The forward process the paper proposes is just a small modification of the variance preserving SDE with a parameter $\tau$. When $\tau=1$, it is exactly the well-studied variance preserving SDE in the literature. Because of that, it is not clear what the advantage is of proposing this new VESDE given that it seems to be quite similar to the VPSDE.

Author response:
**Weakness 1: The advantage of the new drift VESDE in the application.** 

We note that pure VESDE (eq (3) in our paper) achieves SOTA performance in multi-step [1] and one-step image generation [2]. Hence, we use pure VESDE as a baseline and add more synthetic experiments in the main text to show the advantage of our new forward process with aggressive and conservative drift terms. More specifically, For aggressive drift VESDE, we show that the new process balances the different error sources. For conservative drift VESDE, we further consider the approximated score function and show that the conservative drift VESDE can improve the quality of generated distribution without training. More specifically, the conservative drift VESDE directly uses the models training by pure VESDE and reduces the influence of the large data mean and variance to obtain better images. We verify this phenomenon in different sampling methods (Euler-Maruyama method for reverse SDE and RK45 for PFODE), different $T$, and different synthetic datasets. We also calculate the KL divergence between the generated distribution and the original distribution to support our augmentation.

We have now added more experiment results and discussion about the advantages of our new process in Section 7 and Appendix F in the revision of our paper (highlighted in green).

Reviewer comment (Writing):
On page 1, in the last paragraph, “the these models” is a typo.

Author response:
**Weakness 4: The choice of $T$ in the VESDE setting.** 

We note that for the VESDE setting, previous works usually choose large $T$ to obtain large variance $\mathcal{N}(0, \sigma_T^2)$ and achieve good performance. For example, [1] choose $T=80$ for pure VESDE. Furthermore, we also show that with larger $T$, our drift VESDE performs better on the synthetic experiments (Figure 3 and Table 1 in Appendix F). 

Thanks for pointing out the typos in the question part. We have corrected them in the revision paper (highlighted in green).

Reviewer comment (Evaluation):
Is there a reason for using different metrics in Sections 5 (total variation) and 6 (Wasserstein)?

Author response:
**Question 1. The discussion on the TV+Wasserstein and pure Wasserstein guarantee.** 

As discussed in Section 3.1 of [4], since the manifold hypothesis encompasses distributions that admit a continuous density on a lower dimensional manifold, which will lower bound the TV or KL divergence by a constant, studying TV distance or KL divergence bound will vacuous results. 

Hence, there are two types of guarantee for VESDE with reverse SDE. The first one is the TV+Wasserstein guarantee, which is used by VPSDE \[2\]\[3\]. This guarantee relies on the early stopping technique and divides the guarantee into two phases. In the first phase $t\in [0, T-\delta]$, the score function is well-defined, and we use Girsanov's Theorem to obtain the TV guarantee in this phase. For the second phase $t\in [T-\delta, T]$, we choose the appropriate early stopping parameter $\delta$ to obtain a $W_2$ close guarantee instead of using the exploding score, which is not well defined when $t=T$. The second one is the pure Wasserstein guarantee, which is used by VPSDE with tangent-based analysis [4].

For reverse PFODE, since the reverse process lacks randomness, we can not use Girsanov's theorem (the above first guarantee). Hence, we provide a pure Wasserstein guarantee for reverse PFODE. Recently, some works \[5\]\[6\]\[7\] achieve TV + Wasserstein guarantee for VPSDE with reverse PFODE. However, they add additional corrector components [5] or have additional assumptions on the discretization process \[6\]\[7\]. We leave the TV+Wasserstein guarantee for VESDE with reverse PFODE as an interesting future work.