Paper: SciRE-Solver: Accelerating  Diffusion Models Sampling by Score-integrand Solver with Recursive Difference (ICLR-2024)

Reviewer comment (Writing):
After reading the main paper, I have difficulties in understanding why the proposed recursive difference works better. The authors claim "This method recursively extracts the hidden lower-order derivative information of the higher-order derivative terms in the Taylor expansion of the score-integrand at the required point, as illustrated in Figure 2." The recursive difference trick is a key contribution of the work, authors should consider rewriting the above high-density sentence into an easy-to-understand paragraph and highlight why it works, presenting more analysis.

Author response:
Thanks for the detailed review and valuable suggestions. We highly value your suggestions and have rephrased the relevant portion in main paper to provide a detailed explanation and context. For example, we indicate that in algorithms based on the exponential integrator (such as DPM-Solver [1] and DEIS [2]), in order to achieve fast sampling, the original coefficient $\frac{e^h-h-1}{h^2}$ of the difference term is replaced by $\frac{e^h-1}{h}$. Although this substitution is based on the concept of equivalent infinitesimals, it simultaneously reveals that after the replacement of the original coefficients, the acceleration effect becomes more pronounced. Therefore, we speculate that utilizing the conventional FD method directly to evaluate the derivative of the score function may be a suboptimal choice. Then, based on Taylor expansion, we recursively applied finite differences to handle higher-order terms and derived a new equivalent infinitesimal. We refer to this new equivalent infinitesimal as the recursive difference technique. For specific details, please refer to pages 4-5 of the main paper and Appendix E.

[1] Lu et al., Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, NeurIPS 2022.

[2] Zhang et al., Fast sampling of diffusion models with exponential integrator, ICLR 2023.

Reviewer comment (Evaluation):
Can the author show some comparison in terms of numerical accuracy (MSE against ground truth solution) besides FID? How fast do various methods converge to the ground truth solution?

Author response:
Thanks for your response.

We have conducted a comparison of MSE on the bedroom dataset under the same basic settings. 
Specifically, we used the uniform-time trajectory with the termination time set at $1e-3$.  
Due to time constraints, we have included these results in the attachment. Please refer to the report file "report_MSE.txt".  We sincerely hope that our endeavors garner your support.

Thanks again.

Reviewer comment (Theory):
After reading the main paper and appendix, it is unclear to me why the chosen coefficient C6 is better than C5. Besides the empirical experiments, do authors have more principled math analysis for them?

Author response:
Thanks for the thoughtful questions. In fact, these issues may be equivalent. In fact, the main characteristic of the RD method is that this novel estimation incorporates low-order derivative information hidden in the higher-order derivative terms of the Taylor expansion. Refer to the demonstration in Eq. $(3.5)$ and the summary in Figure 2 for details. Compared to FD method, the RD method incorporates additional information $\frac{1-\phi_1(m)}{\phi_1(m)}\frac{\Gamma (\tau_{t_{i-1}}) - \Gamma (\tau_{t_i})}{h_{t_i}}$ from other higher-order derivative terms. Naturally, we have reason to believe that this additional term may counterbalance to a certain level with these higher-order terms from the Taylor expansion. This explanation may address your concerns. The details are included in the newly added Appendix E, especially in E2.

In E2, to address the concerns you raised, we have analyze RD method from three perspectives as comprehensively as possible.

- Firstly, we indicate that a commonality among the coefficients of RD method and the coefficients used in the exponential integrator is the amplification of the original coefficients of FD.
  
- Secondly, we compare the coefficients of EI and RD method from the perspective of analyzing these three different coefficient functions (FD, EI, RD).
  
- Finally, we attempt to theoretically substantiate our viewpoint regarding the offsetting of higher-order terms. We demonstrate that under certain conditions, RD methods for derivative estimation can achieve better truncation errors. We feel that these conditions may be relaxed, but after several days of derivation, we have not yet found a suitable approach. Nevertheless, these conditions all affirm the observed phenomenon.

Reviewer comment (Experiments):
Similar to Q2, can authors present evidence that the proposed method can better estimate score gradients?

Author response:
Thanks for the thoughtful questions. In fact, these issues may be equivalent. In fact, the main characteristic of the RD method is that this novel estimation incorporates low-order derivative information hidden in the higher-order derivative terms of the Taylor expansion. Refer to the demonstration in Eq. $(3.5)$ and the summary in Figure 2 for details. Compared to FD method, the RD method incorporates additional information $\frac{1-\phi_1(m)}{\phi_1(m)}\frac{\Gamma (\tau_{t_{i-1}}) - \Gamma (\tau_{t_i})}{h_{t_i}}$ from other higher-order derivative terms. Naturally, we have reason to believe that this additional term may counterbalance to a certain level with these higher-order terms from the Taylor expansion. This explanation may address your concerns. The details are included in the newly added Appendix E, especially in E2.

In E2, to address the concerns you raised, we have analyze RD method from three perspectives as comprehensively as possible.

- Firstly, we indicate that a commonality among the coefficients of RD method and the coefficients used in the exponential integrator is the amplification of the original coefficients of FD.
  
- Secondly, we compare the coefficients of EI and RD method from the perspective of analyzing these three different coefficient functions (FD, EI, RD).
  
- Finally, we attempt to theoretically substantiate our viewpoint regarding the offsetting of higher-order terms. We demonstrate that under certain conditions, RD methods for derivative estimation can achieve better truncation errors. We feel that these conditions may be relaxed, but after several days of derivation, we have not yet found a suitable approach. Nevertheless, these conditions all affirm the observed phenomenon.

Reviewer comment (Novelty):
A recent work investigates a similar problem and shares a similar algorithm. Can authors comment on the connection and difference [1]?

[1] Zhang et al. Improved order analysis and design of exponential integrator for diffusion models sampling

Author response:
Thanks for the interesting question. In fact, we believe that the exponential integrator fundamentally utilizes an equivalent infinitesimal substitution, amplifying the ratio of difference terms (as indicated in the previous answer). Upon your prompt, we noticed that paper [3] is also under review at ICLR 2024. Wishing them good luck. Based on my reading and learning, it appears that paper [3] also involves the adjustment of coefficients for difference terms. The difference lies in the fact that in reference [3], a method similar to the Butcher tableau is employed to adjust the coefficients of the original EI within the context of EI.

[3] Zhang et al., Improved order analysis and design of exponential integrator for diffusion models sampling.

Reviewer comment (Writing):
The paragraph above Figure 2 needs revision for clarity. The score approximation error is inevitable. How can the proposed sampling method mitigate this issue? The third sentence is also vague without explaining what the "additional variables" means. It is not clear how these considerations lead to the hypothesis either.

Author response:
Thanks for the valuable feedback. Taking into full consideration this feedback, as well as the inevitability of approximation errors, we have revised the expression of this paragraph from the perspective of the characteristics of existing literature. Specifically, in order to accelerate sampling, existing literature has replaced the coefficients of finite difference terms with the concept of equivalent infinitesimals during algorithm design. We believe that such replacement fundamentally alters the way derivatives are estimated, and it also potentially indicates that using finite difference to estimate derivatives may not lead to further acceleration effects. This analysis supports our hypothesis.

Reviewer comment (Experiments):
In Table 1, SciRE-solver uses pre-trained model from EDM. But the results of the original EDM sampler is missing from the comparison.

Author response:
Thanks for the detailed review. We have included the experiments for EDM in Table 1.  When invoking the Heun iteration algorithm for EDM, we utilized the trajectory with the EDM type.

Reviewer comment (Writing):
Minor: in the abstract, "Experiments demonstrate also that demonstrate that" -> "Experiments also demonstrate that".

Author response:
Thanks for the detailed review. We have thoroughly examined our paper and made the necessary revisions. Many Thanks.

Reviewer comment (Theory):
Is there any acceleration algorithm for diffusion SDE as well? If yes, I would love to see the authors providing a discussion. If no, could the authors elaborate a bit on why training-free acceleration is mostly for diffusion ODE?

Author response:
Thanks for providing such interesting topic. We observed that 'dpm-solver++' has introduced a dedicated accelerated iterative scheme specifically designed for diffusion SDE.

The most significant difference between diffusion SDE and diffusion ODE lies in the presence of stochastic term. In diffusion SDE, once the coefficient function of the stochastic term is determined, the remaining integral term is equivalent to solving an ODE, because the integration of stochastic term can be estimated using the mean formula and variance formula of Itô integration. Therefore, there are two key aspects for diffusion SDEs. 
- The first involves finding a balance between the coefficient function of the stochastic term and the coefficient function of the Score function term. 
- The second aspect pertains to devising an algorithm capable of accommodating diffusion ODEs with deterministic variance perturbations.

[1] Lu et al., DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models.

Reviewer comment (Writing):
In Eq (3.1), $h(r)$ should be $h_1(r)$.

Author response:
Thanks for the very detailed review. We carefully examined the entire text and corrected it.

Reviewer comment (Theory):
I understand that $NSR$ is monotone, but why is it strictly monotone? Namely, how to guaratee the existence of its inverse function.

Author response:
Under the variance-preserving setting, $\sigma_t^2 +\alpha_t^2 = 1$. Since $\sigma_t$ is strictly monotonic, $NSR(t)=\frac{\sigma_t}{\alpha_t} $ is also strictly monotonic. The inverse of $NSR(t)=\frac{\sigma_t}{\alpha_t}$ is provided in Appendix G3.

Reviewer comment (Evaluation):
Why the authors say in Figure 1 that the proposed algorithm outperforms DDIM and DPM solver?

Author response:
Thanks for the detailed question. For a single image, we often lack a reasonable method to compare its generation quality, as each algorithm's generated image may not necessarily be the optimal image under that specific noise. In such cases, we can only evaluate based on the details and clarity of the generated images. Therefore, in Figure 1, we can assess based on the details and clarity of the generated samples. For instance, the samples produced by our algorithm at 15 steps are clearer than those generated by DDIM at 15 and 20 steps, and are even comparable to those produced by DDIM at 50 steps.

Reviewer comment (Theory):
Maybe this is a dumb question. Why can we assume the neural network is differentiable? I would imagine this is not the case when the activation function is ReLU.

Author response:
Thanks for the thoughtful question. The differentiability allows us to compute the gradient of the loss function with respect to the network parameters, enabling parameter updates. ReLU is indeed non-differentiable. We have provided a review study that we hope will be helpful to you for more details.

[4] Dubey et al., Activation functions in deep learning: A comprehensive survey and benchmark, Neurocomputing (2022).

Reviewer comment (Writing):
This sentence is hard to parse. "This method recursively extracts the hidden lower-order derivative information of the higher-order derivative terms in the Taylor expansion of the score-integrand at the required point, as illustrated in Figure 2". I would suggest the authors present their most general form algorithm in the format of Algorithm 1 and 2.

Author response:
Thanks for the valuable suggestion. In order to facilitate reader comprehension, we have rephrased the sentence into a paragraph and provided a detailed explanation. In the revised version, we illustrate this with an example using Eq. (3.7), where the colors annotated in the equation correspond to the colors in Figure 2. For the general form of the algorithm, we showcase it in the attached scire-solver.py.

Reviewer comment (Theory):
In Theorem 3.2, $m$ has to be larger than 2 or 3?

Author response:
In fact, $m-1$ represents the number of recursive operations performed on how many higher-order derivatives. We recommend performing at least two recursive operations, so m should be greater than 2.

Reviewer comment (Presentation):
The legend in Figure 3 is a bit misleading. I assume the first four ones are for dashed lines, but it is not apparent at first glance.

Author response:
Thanks for the detailed review. We have made appropriate adjustments to make it look more apparent.