Paper: Improved order analysis and design of exponential integrator for diffusion models sampling (ICLR-2024)

Reviewer comment (Novelty):
The paper's main contribution, as I understand it—prescribing order conditions for exponential integrators—already exists in prior work.

Author response:
Indeed, our work draws upon well-established theories and analyses from the numerical literature. However, our application of these mathematical tools to diffusion model sampling involves significant efforts and contributions. To the best of our knowledge, these are presented explicitly for the first time in the context of accelerating diffusion models:
1. We revisit existing, popular diffusion ODE parameterizations and explicitly detail the transformations that convert probability flow ODEs into canonical semi-linear ODE forms. This includes models for both noise prediction and data prediction.
2. Building upon these connections, we utilize analyses and mathematical tools from exponential integrators to conduct error analysis and design superior numerical algorithms.
3. We conduct comprehensive experiments to show that our new algorithm, which fully incorporates stratified order conditions, outperforms existing methods in diffusion sampling. The advantage is more obvious when time scheduling is not optimal, see figure 2. Those experiments highlight the importance of designing order-stratified methods. 
4. Furthermore, extensions based on order-stratified numerical schemes, including both stochastic and multistep methods, also demonstrate superior performance over current methodologies.

Reviewer comment (Novelty):
The contributions between Equation (9) and the end of Section 3, along with the corresponding proofs in the Appendix, reproduce existing derivations without crediting the original work, mainly by Hochbruck and Ostermann (2005).

Author response:
* Firstly, we acknowledge that key mathematical analysis tools in our study are based on well-established theories in the numerical methods literature. We have updated our revision to emphasize this point more clearly.
* Secondly, the application of these numerical solvers to diffusion models represents a novel approach. For instance, the utilization of order-satisfying techniques plays a crucial role in accelerating diffusion sampling. Unlike existing works that rely on sophisticated and varied derivations, such as those found in DPM-Solver methods, our approach directly leverages conclusions and numerical methods from the numerical literature. This provides a more unified and mathematically robust explanation and framework.

Reviewer comment (Writing):
If my understanding is correct that the main contribution is deriving a set of exponential integrators satisfying specific order conditions to make them provably efficient for diffusion models, then this should be stated more clearly if it is not the case.

Author response:
The central contribution of our paper is the acceleration of the diffusion sampling algorithm, which is built upon a well-established, order-satisfied numerical scheme. We acknowledge that our analysis framework, grounded in the referenced mathematical numeric analysis, may have limited contribution and novelty from the perspective of numerical analysis. However, we emphasize our significant contributions to the field of diffusion models, especially in accelerating visual content generation.
Notably, our work introduces high-order, order-satisfied diffusion sampling algorithms, previously nonexistent in this domain. We respectfully disagree with the reviewer’s perspective that pointing out prevalent flaws in popular, but order-unsatisfied methods and achieving state-of-the-art diffusion sampling performance is trivial.
We hope the reviewer can value our contribution to the accelerating diffusion model. Celebrated works such as DDIM and DPM-Solver, while adopting different ways of presenting proposed algorithms, essentially represent special cases that utilize the suboptimal Runge-Kutta tables found in the existing EI literature. Our work situates algorithm development within a more mathematically rigorous framework and conducts a broader range of experiments to demonstrate its effectiveness.

Reviewer comment (Novelty):
The Butcher Tableaus of the "refined exponential solver" and the theoretical derivation surrounding the order conditions (Lemma 1, Theorems 1 and 2, and Proposition 1) are already known from Hochbruck and Ostermann (2005).

Author response:
* Firstly, we acknowledge that key mathematical analysis tools in our study are based on well-established theories in the numerical methods literature. We have updated our revision to emphasize this point more clearly.
* Secondly, the application of these numerical solvers to diffusion models represents a novel approach. For instance, the utilization of order-satisfying techniques plays a crucial role in accelerating diffusion sampling. Unlike existing works that rely on sophisticated and varied derivations, such as those found in DPM-Solver methods, our approach directly leverages conclusions and numerical methods from the numerical literature. This provides a more unified and mathematically robust explanation and framework.

Reviewer comment (Novelty):
Lemma 1 is equivalent to Equations 4.9 to 4.12 in Hochbruck and Ostermann (2005); the "proof" in Appendix B.1 is almost a verbatim reproduction of Section 4.1 in that paper, without marking it as such.

Author response:
* Firstly, we acknowledge that key mathematical analysis tools in our study are based on well-established theories in the numerical methods literature. We have updated our revision to emphasize this point more clearly.
* Secondly, the application of these numerical solvers to diffusion models represents a novel approach. For instance, the utilization of order-satisfying techniques plays a crucial role in accelerating diffusion sampling. Unlike existing works that rely on sophisticated and varied derivations, such as those found in DPM-Solver methods, our approach directly leverages conclusions and numerical methods from the numerical literature. This provides a more unified and mathematically robust explanation and framework.

Reviewer comment (Novelty):
Theorem 1 is a version of Theorem 4.2 by Hochbruck and Ostermann (2005); Theorem 2 is a version of Theorem 4.3 by Hochbruck and Ostermann (2005).

Author response:
* Firstly, we acknowledge that key mathematical analysis tools in our study are based on well-established theories in the numerical methods literature. We have updated our revision to emphasize this point more clearly.
* Secondly, the application of these numerical solvers to diffusion models represents a novel approach. For instance, the utilization of order-satisfying techniques plays a crucial role in accelerating diffusion sampling. Unlike existing works that rely on sophisticated and varied derivations, such as those found in DPM-Solver methods, our approach directly leverages conclusions and numerical methods from the numerical literature. This provides a more unified and mathematically robust explanation and framework.

Reviewer comment (Novelty):
Proposition 1 paraphrases Theorem 4.7 by Hochbruck and Ostermann (2005).

Author response:
* Firstly, we acknowledge that key mathematical analysis tools in our study are based on well-established theories in the numerical methods literature. We have updated our revision to emphasize this point more clearly.
* Secondly, the application of these numerical solvers to diffusion models represents a novel approach. For instance, the utilization of order-satisfying techniques plays a crucial role in accelerating diffusion sampling. Unlike existing works that rely on sophisticated and varied derivations, such as those found in DPM-Solver methods, our approach directly leverages conclusions and numerical methods from the numerical literature. This provides a more unified and mathematically robust explanation and framework.

Reviewer comment (Novelty):
The observation that the algorithm called RES outperforms the DPM-solver++ has also been made and mathematically underpinned by Hochbruck and Ostermann (2005; Figure 6.3).

Author response:
We note that the experimental setups, which are low-dimensional synthetical problems, in Hochbruck and Ostermann (2005; Figure 6.3) differ markedly from ours. Besides ODE acceleration, the robustness against various time-scheduling, boosting stochastic, and multistep sampling methods for various diffusion models are never discovered or mentioned to the best of our knowledge.

Reviewer comment (Novelty):
The fact that the analysis already exists reduces the paper's contribution to a point where it should be rejected.

Author response:
We strongly contest the reviewer's argument for rejection. Our reasons are as follows:
* As highlighted in Q1, only one of our four major steps or contributions is based on the referenced mathematical literature. The remaining three steps represent novel approaches, or at least approaches not yet explored by researchers in diffusion models to the best of our knowledge.
* It is important to recognize that works whose main contribution appears to align with algorithms in mathematical references can still offer substantial contributions. Specifically in the context of diffusion models, generative diffusion models, interpreted as a pair of time-reversible forward and backward SDEs[1], are acknowledged in sources like [2,3]. DDIM[4] and DPM-Solvers[5], despite their differing presentations and derivations, essentially distill to the first-order exponential Euler method and the second-order exponential integrator[6] with order defects.
* Lastly, the empirical improvements we present are significant. Reducing inference cost is crucial for diffusion models. While some authors or researchers might view our conclusions as predictable, it is the first time that order artifacts in popular and widely used sampling algorithms have been identified, discussed, and shown to substantially accelerate diffusion models through improved order-conditioned algorithms. We do not such results are “unsurprising” to generative AI researcher.

Reviewer comment (Novelty):
It is problematic that the analysis is presented as new instead of attributing the results to the original works, even though the authors seem familiar with Hochbruck and Ostermann (2005), as they cite it on page 18.

Author response:
* Firstly, we acknowledge that key mathematical analysis tools in our study are based on well-established theories in the numerical methods literature. We have updated our revision to emphasize this point more clearly.
* Secondly, the application of these numerical solvers to diffusion models represents a novel approach. For instance, the utilization of order-satisfying techniques plays a crucial role in accelerating diffusion sampling. Unlike existing works that rely on sophisticated and varied derivations, such as those found in DPM-Solver methods, our approach directly leverages conclusions and numerical methods from the numerical literature. This provides a more unified and mathematically robust explanation and framework.

Reviewer comment (Presentation):
The performance in the secondary experiments (in Figure 4) do not really have a clear message, and in my opinion can be relegated to the appendix.

Author response:
Figure 4 is divided into two parts, each highlighting the significance of implementing order-satisfied numerical schemes in various contexts, including stochastic sampling, text-to-image diffusion, and image super-resolution diffusion:
- Fig 4(a) presents a comparison between the RES-based stochastic sampling algorithm and the widely-used EDM(Heun)-based stochastic sampling algorithm. This comparison demonstrates that order-satisfied numerical schemes can enhance the efficiency of stochastic sampling.
- Fig 4(b) explores large-scale text-to-image settings, covering both base text-to-image models and image super-resolution tasks. Our results show that, within the same computational budget, our approaches achieve superior image quality, as indicated by lower FID scores, and better text-image alignment, as evidenced by higher CLIP scores, compared to existing methods. Thanks to great interest in AIGC with text-to-image, we believe RES can accelerate various diffusion applications.

Reviewer comment (Evaluation):
The performance in the stochastic setting is not so clear cut, and therefore I am inclined to conclude that the primary performance benefits are constrained to the deterministic setting.

Author response:
Recent studies, including EDM [1], have demonstrated that stochastic sampling often surpasses deterministic methods in performance. Drawing inspiration from stochastic EDM, we adapted our algorithm for a stochastic environment. This involves adding extra stochasticity after each Runge-Kutta (RK) step, which corresponds to a nonzero value of $\eta$  in Algorithm 2. Our approach utilizes the RES single-step update scheme, in contrast to the Heun update-step scheme used by the baseline EDM.

In our study, we explored two key aspects:

- The impact of the number of function evaluations (NFE) on image quality, as measured by the FID score, is shown in the left panel of Figure 4(a).
- The relationship between NFE and the magnitude of stochasticity (indicated by the magnitude of $\eta$) is presented in the right panel of the same figure.

We have included more discussion on these findings in the revised version of our paper.

Reviewer comment (Presentation):
In Fig 1b, I am not really clear on what the left part of (b) is trying to convey. In my opinion, it is just cluttering up the image and it might be good to remove it.

Author response:
We respectfully disagree with the suggestion to move Figure 1b to the appendix. This figure is central to our analysis, as it illustrates the core methodology for error propagation and the derivation of the order condition. We believe that including this diagram within the main body of the text, particularly in conjunction with Section 3.2, aids readers in grasping the intuition and derivation process more easily.

Reviewer comment (Presentation):
What is the dashed line in Figure 3?

Author response:
To match the performance of DPM-Solver ++ (S) at 99 NFE, our RES single-step method requires only 59 NFE. This comparison, illustrated in the figure, underscores our method's acceleration capabilities, particularly in the paragraph of "Time-scheduling robustness".

Reviewer comment (Experiments):
Can the authors provide more details on the stochastic sampling setting? What is the problem being considered and how can we interpret the results?

Author response:
Recent studies, including EDM [1], have demonstrated that stochastic sampling often surpasses deterministic methods in performance. Drawing inspiration from stochastic EDM, we adapted our algorithm for a stochastic environment. This involves adding extra stochasticity after each Runge-Kutta (RK) step, which corresponds to a nonzero value of $\eta$  in Algorithm 2. Our approach utilizes the RES single-step update scheme, in contrast to the Heun update-step scheme used by the baseline EDM.

In our study, we explored two key aspects:

- The impact of the number of function evaluations (NFE) on image quality, as measured by the FID score, is shown in the left panel of Figure 4(a).
- The relationship between NFE and the magnitude of stochasticity (indicated by the magnitude of $\eta$) is presented in the right panel of the same figure.

We have included more discussion on these findings in the revised version of our paper.

Reviewer comment (Experiments):
Some experiments are worth comparison. For example, with the same NFE, how does it compare to step-distillation method [1].

Author response:
* Our work primarily concentrates on training-free methods for pre-trained diffusion models. The majority of our experiments and discussions are centered around methodologies within this category. We have included a discussion of suggested methods in our related work section.
* The methods you've suggested, such as step-distillation, demand additional training. As for the consistency model, it is indeed a novel approach that offers faster sampling speeds. However, it has not yet been extensively validated in large-scale tasks, such as text-to-image settings, and currently shows a significant performance gap when compared to diffusion models.
* It's also worth noting that the suggested works, including progressive distillation and consistency distillation, rely on ODE solvers as a foundational element or 'teacher'. Therefore, our method has the potential to accelerate these processes as well.

Reviewer comment (Experiments):
With the same NFE, how does it compare to consistency model [2].

Author response:
* Our work primarily concentrates on training-free methods for pre-trained diffusion models. The majority of our experiments and discussions are centered around methodologies within this category. We have included a discussion of suggested methods in our related work section.
* The methods you've suggested, such as step-distillation, demand additional training. As for the consistency model, it is indeed a novel approach that offers faster sampling speeds. However, it has not yet been extensively validated in large-scale tasks, such as text-to-image settings, and currently shows a significant performance gap when compared to diffusion models.
* It's also worth noting that the suggested works, including progressive distillation and consistency distillation, rely on ODE solvers as a foundational element or 'teacher'. Therefore, our method has the potential to accelerate these processes as well.

Reviewer comment (Evaluation):
In Figure 2, why is the multistep setting even worse than the single step?

Author response:
We believe this question pertains to the third subplot in Figure 3, where various methods are tested under a suboptimal time schedule. Both multistep and single-step methods approximate the nonlinear function using a polynomial. However, unlike single-step methods that query new function evaluations, multistep methods attempt to save on evaluations by reusing previous ones, which may become outdated. In a suboptimal time schedule, the disadvantage of relying on these outdated evaluations could negate the benefit of conserving new evaluations. This results in poorer performance compared to single-step methods, which, despite incurring extra evaluations, obtain up-to-date information for nonlinear function approximation.