From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper5563/Area_Chair_UJPv
metareview: The paper is very well-written, provides a very useful recipe for training diffusion models faster and combines that with extensive experiments. Good work!
recommendation: Accept
confidence: Certain
award: No

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Sincerely looking forward to your reply
comment: We'd like to thank reviewer DD9K for the valuable response and the strong acceptance of our work. If our response has addressed the concerns and brings new insights to the reviewer, we will highly appreciate it if the reviewer considers raising the score.

In addition, please let us know if you have any other questions, comments, or concerns. Thank you very much!

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Thank you for the valuable review
comment: Thanks again for your valuable review. We believe that the proposed DPM-Solver can greatly advance the application of diffusion models because of its advantages in both fast sampling and fast convergence. Please let us know if you have any other questions, comments, or concerns. Thank you very much!

From: NeurIPS.cc/2022/Conference/Paper5563/Reviewer_EaE9
title: Thank you for the reply
comment: I would like to thank the authors for their detailed reply and acknowledge that I have read it. I appreciate the answers to my questions, which have been addressed well, and the additional experiments. Overall, my impression of the paper hasn't significantly changed (hence, I maintain my score) and I still think that this is a solid work that should be accepted.

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Response to Reviewer Wm2X (cont.)
comment: 
## Q2: Originality beyond directly using the variation-of-constants

Thanks for the valuable comments. The variation-of-constants is a classical idea for solving the semi-linear ODE, so it is natural to combine it with diffusion ODEs which have a semi-linear structure. However, to the best of our knowledge, the connection between the variation-of-constants and DPMs has not been revealed in the prior literature, so the combination of each other is also our contribution. Further, compared to directly using the variation-of-constants, we emphasize that the change-of-variable for $\lambda$ is highly nontrivial and can bring much more insights. We discuss the difference between directly using the variation-of-constants and the proposed DPM-Solver below to demonstrate that **the change-of-variable for $\lambda$ is also critical and is our novel contribution**.

For the original diffusion ODE, the exact solution can be formulated by the variation of constants:
$$
x_t = e^{\int_s^t f(\tau)\mathrm{d}\tau}x_s + \int_s^t \left(e^{\int_\tau^t f(r)\mathrm{d} r}\frac{g^2(\tau)}{2\sigma_\tau} \epsilon_\theta(x_\tau,\tau)\right)\mathrm{d}\tau.
$$
If we directly approximate the above integral, we need to discretize and approximate the coefficients related to $f(t)$ and $g(t)$ (i.e., the noise schedule). Such computation is complicated and may involve other discretization errors. Instead, we propose the following equivalent but much simpler formulation:
$$
x_t = \frac{\alpha_t}{\alpha_s}x_s - \alpha_t \int_{\lambda_s}^{\lambda_t} e^{-\lambda} \hat\epsilon_\theta(\hat x_\lambda,\lambda)\mathrm{d}\lambda.
$$
What we want to emphasize is that, the above integral is **invariant to the noise schedule** between $\lambda_s$ and $\lambda_t$ because the integrand contains no functions about $f(t)$ and $g(t)$ and only related to $\epsilon_\theta$. Such formulation can unify the sampling of diffusion models for different types of noise schedules. Therefore, the proposed DPM-Solver is also invariant to the noise schedule and is a general solver for all types of diffusion models. We have updated Appendix A in the revision to discuss more such invariance properties. Therefore, DPM-Solver is a customized solver for DPMs, and its derivation is more than directly using the variation-of-constants. We hope the above explanation can address the concern about the originality.

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Response to Reviewer EaE9 (cont.)
comment: ## Q4: Extend the limitations

Thanks for the valuable suggestion. We have updated the conclusion section to discuss the proposed method's limitations.

## Q5: Comparison with the concurrent work DEIS

The concurrent work DEIS also uses the variation-of-constants to design a fast sampler of diffusion models. However, there are many differences between DPM-Solver and DEIS:

### 1. Proposition 3.1 in our work is a novel formulation, which is **invariant to the noise schedule**.

In DEIS, their solver is based on the following formulation of the exact solution:
$$
x_t = e^{\int_s^t f(\tau)\mathrm{d}\tau}x_s + \int_s^t \left(e^{\int_\tau^t f(r)\mathrm{d} r}\frac{g^2(\tau)}{2\sigma_\tau} \epsilon_\theta(x_\tau,\tau)\right)\mathrm{d}\tau.
$$
Furthermore, DEIS further approximates each $\epsilon_\theta(x_\tau, \tau)$ by the Lagrange interpolation of the function values at previous time steps. Then the remaining coefficients are computed by a 1-D numerical integrating solver, which needs more computations (Although the costs are small, the calling of further numerical solvers makes the problem more complicated).

Instead, DPM-Solver proves that we can approximate the following equivalent formulation:
$$
x_t = \frac{\alpha_t}{\alpha_s}x_s - \alpha_t \int_{\lambda_s}^{\lambda_t} e^{-\lambda} \hat\epsilon_\theta(\hat x_\lambda,\lambda)\mathrm{d}\lambda.
$$
What we want to emphasize is that the above integral is **invariant to the noise schedule** between $\lambda_s$ and $\lambda_t$ because the integrand contains no functions about $f(t)$ and $g(t)$ and is only related to $\epsilon_\theta$. Such formulation can unify the sampling of diffusion models for different types of noise schedules. Therefore, the proposed DPM-Solver is also invariant to the noise schedule and is a general solver for all types of diffusion models. We have updated Appendix A in the revision to discuss more such invariance properties. Moreover, the coefficients of DPM-Solver can be **analytically** computed without any further approximations or numerical solvers. Therefore, DPM-Solver is derived from a more general perspective and avoids any other approximations.

### 2. DPM-Solver is a single-step Runge-Kutta method, while DEIS is a multi-step method.

To approximate the solution, DPM-Solver uses a Runge-Kutta-like method which needs only a single value of the previous time step. In contrast, DEIS uses a multi-step method that needs multiple values from previous time steps.

- Single-step method (DPM-Solver) needs only one initial value to guarantee the convergence order. In contrast, the multi-step method (DEIS) needs at least $k$ initial value for a $k$-step method to guarantee the convergence order. For sampling from DPMs, it is hard to get $k$ initial values because we can only sample from $p_T=\mathcal{N}(0,I)$. Therefore, DPM-Solver theoretically has the convergence order guarantee, but DEIS may have no such property, which may cause it to converge slower than DPM-Solver.

### 3. The choice of time steps for DPM-Solver is general and suitable for all pre-trained models, while DEIS needs extra tuning for the time steps and depends on the specific noise schedule.

DPM-Solver chooses linear $\lambda$ (logSNR), while DEIS chooses time steps for the time $t$. As mentioned in the first point, DPM-Solver is invariant to the specific noise schedule, so the choice of intermediate $\lambda$ is invariant to the detailed noise schedule $f(t)$ and $g(t)$. However, DEIS manually designs the time steps for $t$, which also depends on the specific noise schedule. Therefore, given a dataset, we may train DPMs by different noise schedules and get different noise prediction models. For DPM-Solver, the choices of time steps for sampling from these models are the same, without any further tuning of the time steps. However, for DEIS, the choices of time steps for the sampling from these models need extra tuning costs and are not general.  Moreover, the proposed DPM-Solver also develops an adaptive step size method (as proposed in Sec. 3.3 and Appendix C), which does not need any ad-hoc time step tuning. 

=============================

In summary, we believe that DPM-Solver is a more general solver for DPMs than DEIS, which is invariant to the noise schedule and has a convergence order guarantee. Nevertheless, we may combine DPM-Solver and DEIS to develop a more powerful solver, and we leave it for future work.

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Response to Reviewer EaE9
comment: 
We thank reviewer EaE9 for acknowledging our significant contributions and giving many insightful comments and valuable suggestions for improving our work. Below we provide a point-to-point response to all comments.

## Q1: Comparison with other relevant baselines

Thanks for the valuable suggestion. We have updated Sec. 5 and Fig. 2 in the revision to include the mentioned PNDM, FastDPM, and Itô-Taylor methods. As the proposed DPM-Solver and these relevant baselines all use the same discrete-time pre-trained models, we can directly compare the sample quality between DPM-Solver and these methods. Still, the experimental results show that DPM-Solver with 12 NFEs can achieve $\sim$4$\times$ speedup compared with all of the baselines.

## Q2: Related literature about accelerating sampling of DPMs

Sorry for the missing citations, and thanks for the valuable suggestions. We have updated Sec.4.3 in the revision to discuss other methods for designing novel types of DPMs that support faster sampling.

## Q3: Applying DPM-Solver to higher-resolution image datasets

Thanks for the insightful suggestion. We further evaluate DPM-Solver, DDIM, and DDPM on two high-resolution datasets: ImageNet 128x128 and LSUN bedroom 256x256. For both experiments, we use the pre-trained model in [1]. For ImageNet 128x128, we use the best settings for DDIM in [1], i.e., classifier guidance with a scale of 1.25. For LSUN bedroom 256x256, we use their released unconditional diffusion model.

We have updated Sec.5, Fig.2, and Appendix E.5 for each experiment's detailed results and settings. Below we list the detailed sample quality evaluated by FID, varying the NFE. Due to the time constraints of the rebuttal, we only evaluate DDPM for 10, 50, 250 NFE, DDIM for 10, 12, 15, 20, 50, 100 NFE, and DPM-Solver for 10, 12, 15, 20, 50 NFEs. Nevertheless, the results are sufficient to show the effectiveness of DPM-Solver.

For instance, for ImageNet 128x128, DPM-Solver can achieve 4.08 FID with 12 NFE, while DDIM needs 100 NFE for 4.18 FID; and DPM-Solver can achieve 3.52 FID with 20 NFE, which outperforms DDIM with 250 NFE (3.54 FID). For LSUN bedroom 256x256, DPM-Solver can achieve 2.94 FID with 15 NFE, while DDIM needs 50 NFE for 2.92 FID; and DPM-Solver can achieve 2.01 FID with 50 NFE, which even outperforms DDPM/DDIM with 250 NFE (2.02 FID, which is the best results in [1]). 

Therefore, **DPM-Solver is still suitable for high-resolution image datasets**. Besides, we will finish the intermediate NFE experiments (the"$\backslash$" in the table) for completeness in the camera-ready version. In addition, as DDIM can accelerate the sampling of the audio data, we believe that DPM-Solver is also suitable for the audio data because DDIM is the first-order case of DPM-Solver, and we leave it for future work.

[1] Dhariwal, Prafulla, and Alexander Nichol. "Diffusion models beat gans on image synthesis." Advances in Neural Information Processing Systems 34 (2021): 8780-8794.

**Table 1**. Sample quality of DPMs measured by FID $\downarrow$ on ImageNet 128$\times$128 with classifier guidance and on LSUN bedroom 256$\times$256, varying the number of function evaluations (NFE).

| Sampling method $\backslash$ NFE | 10       | 12           | 15           | 20           | 50       | 100          | 250          |
| -------------------------------- | -------- | ------------ | ------------ | ------------ | -------- | ------------ | ------------ |
| **ImageNet 128$\times$128**      |          |              |              |              |          |              |              |
| DDPM                             | 199.56   | $\backslash$ | $\backslash$ | $\backslash$ | 49.38    | $\backslash$ | **2.97**     |
| DDIM                             | 11.12    | 9.38         | 8.22         | 7.15         | 5.05     | 4.18         | 3.54         |
| DPM-Solver                       | **7.32** | **4.08**     | **3.60**     | **3.52**     | **3.13** | $\backslash$ | $\backslash$ |
| **LSUN bedroom 256$\times$256**  |          |              |              |              |          |              |              |
| DDPM                             | 274.67   | $\backslash$ | $\backslash$ | $\backslash$ | 82.70    | $\backslash$ | 2.02    |
| DDIM                             | 10.05    | 7.51         | 5.90         | 4.98         | 2.92     | 2.30         | 2.02 |
| DPM-Solver                       | **6.10** | **4.29**     | **2.94**     | **2.60**     | **2.01** | $\backslash$ | $\backslash$ |

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Response to Reviewer Wm2X
comment: We thank reviewer Wm2X for the acknowledgment of our significant contributions and the insightful comments. We have updated our paper by adding a discussion about the runtime comparison between the proposed DPM-Solver and other methods in Appendix E.7. Below, we provide a point-to-point response to all comments.

## Q1: The comparison of runtime for sampling between DPM-Solver and other methods

Thanks for the insightful suggestion. Theoretically, for the same NFE, the runtime of DPM-Solver and DDIM are **almost the same** (linear to NFE) because the main computation costs are the **serial evaluations** of the large neural network $\epsilon_\theta$ and the other coefficients are **analytically** computed with ignorable costs.

Below in Table 1, we evaluate the runtime of DPM-Solver and DDIM on a single NVIDIA A40, varying different datasets and NFE. We use `torch.cuda.Event` and `torch.cuda.synchronize` for accurately computing the runtime. We use the discrete-time pre-trained diffusion models for each dataset. We evaluate the runtime for 8 batches and compute the mean and std of the runtime. We use batch size 64 for LSUN bedroom 256x256 due to the GPU memory limitation and batch size 128 for other datasets.


**Table 1.** The runtime of a single batch (second / batch, $\pm$std) on a single NVIDIA A40 of DDIM and DPM-Solver for sampling by discrete-time diffusion models, varying the number of functions evaluations (NFE).


| Sampling method $\backslash$ NFE                             | 10                 | 20                 | 50                  | 100                 |
| ------------------------------------------------------------ | ------------------ | ------------------ | ------------------- | ------------------- |
| **CIFAR-10 32$\times$32 (batch size = 128)**                 |                    |                    |                     |                     |
| DDIM                                                         | 0.956($\pm$0.011)  | 1.924($\pm$0.016)  | 4.838($\pm$0.024)   | 9.668($\pm$0.013)   |
| DPM-Solver                                                   | 0.923($\pm$0.006)  | 1.833($\pm$0.004)  | 4.580($\pm$0.005)   | 9.204($\pm$0.011)   |
| **CelebA 64$\times$64 (batch size = 128)**                   |                    |                    |                     |                     |
| DDIM                                                         | 3.253($\pm$0.015)  | 6.438($\pm$0.029)  | 16.132($\pm$0.050)  | 32.255($\pm$0.044)  |
| DPM-Solver                                                   | 3.126($\pm$0.003)  | 6.272($\pm$0.006)  | 15.676($\pm$0.008)  | 31.269($\pm$0.012)  |
| **ImageNet 64$\times$64 (batch size = 128)**                 |                    |                    |                     |                     |
| DDIM                                                         | 5.084($\pm$0.018)  | 10.194($\pm$0.022) | 25.440($\pm$0.044)  | 50.926($\pm$0.042)  |
| DPM-Solver                                                   | 4.992($\pm$0.004)  | 9.991($\pm$0.003)  | 24.948($\pm$0.007)  | 49.835($\pm$0.028)  |
| **ImageNet 128$\times$128 (batch size = 128, with classifier guidance)** |                    |                    |                     |                     |
| DDIM                                                         | 29.082($\pm$0.015) | 58.159($\pm$0.012) | 145.427($\pm$0.011) | 290.874($\pm$0.134) |
| DPM-Solver                                                   | 28.865($\pm$0.011) | 57.645($\pm$0.008) | 144.124($\pm$0.035) | 288.157($\pm$0.022) |
| **LSUN bedroom 256$\times$256 (batch size = 64)**            |                    |                    |                     |                     |
| DDIM                                                         | 37.700($\pm$0.005) | 75.316($\pm$0.013) | 188.275($\pm$0.172) | 378.790($\pm$0.105) |
| DPM-Solver                                                   | 36.996($\pm$0.039) | 73.873($\pm$0.023) | 184.590($\pm$0.010) | 369.090($\pm$0.076) |




For DDIM, we use the official implementation in [1]. We find that our implementation of DPM-Solver reduces some repetitive computation of the coefficients, so under the same NFE, DPM-Solver is slightly faster than DDIM of their implementation. Nevertheless, the runtime evaluation results show that the runtime of DPM-Solver and DDIM are almost the same for the same NFE, and the runtime is approximately linear to the NFE. Therefore, **the speedup for the NFE is almost the actual speedup of the runtime**, so the proposed DPM-Solver can significantly speed up the sampling of DPMs.


[1] https://github.com/ermongroup/ddim

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Response to Reviewer DD9K
comment: We thank reviewer DD9K for the interest and strong acknowledgment of our novel contributions and the insightful questions. Below we respond to the questions.

## Q1: Documentation of the code

Thanks for the valuable suggestion. We will release our official code with complete documentation in the camera-ready version.

## Q2: More expressions about $\lambda(t)$ and its relationship with time $t$.

In fact, choosing time steps for $\lambda(t)$ is **invariant to the specific noise schedule** (the choice of $f(t)$ and $g(t)$ of the forward process). We have updated a detailed discussion in Appendix A about such invariance property. Below we give a brief introduction.

Our Proposition 3.1 shows that the exact solution of diffusion ODEs has the following formulation:
$$
x_t = \frac{\alpha_t}{\alpha_s}x_s - \alpha_t \int_{\lambda_s}^{\lambda_t} e^{-\lambda} \hat\epsilon_\theta(\hat x_\lambda,\lambda)\mathrm{d}\lambda.
$$
What we want to emphasize is that, the above integral is **invariant to the noise schedule** between $\lambda_s$ and $\lambda_t$ because the integrand contains no functions about $f(t)$ and $g(t)$ and only related to $\epsilon_\theta$. Such formulation can unify the sampling of diffusion models for different types of noise schedules. Therefore, choosing time steps for $\lambda(t)$ is also invariant to the specific noise schedules, and a simple but efficient way is to uniformly split $\lambda(t)$, as used in our experiments. Moreover, we believe there is a more accurate and efficient way of choosing time steps for a given noise prediction model $\epsilon_\theta$, and we leave it for future work.

In addition, when we uniformly pick $\lambda(t)$, the detailed time $t$ depends on the noise schedule $f(t)$ and $g(t)$ and may be rather different. However, when we uniformly pick $\lambda(t)$ for DPM-Solver, the solution is invariant to the noise schedule $f(t)$ and $g(t)$.


## Q3: Insights and relationship with training DPMs.

Thanks for the insightful question. We have updated Appendix A to discuss the relationship between our invariance sampling and the maximum likelihood training of DPMs. In fact, both the sampling and the maximum likelihood training of diffusion models can be converted to an integral w.r.t. $\lambda$, such that the formulation is invariant to the specific noise schedules, and we summarize it in the following table. Such invariance property for training and sampling brings new insights for understanding diffusion models. For instance, we can directly define the noise prediction model $\epsilon_\theta$ w.r.t. the (half-)logSNR $\lambda$ instead of the time $t$, then the training and sampling for diffusion models can be done **without further choosing any ad-hoc noise schedules**. Such findings may unify the different ways of training and inference of diffusion models, and we leave it for future study.

|           Method            |                    Invariance Formulation                    |
| :-------------------------: | :----------------------------------------------------------: |
| Maximum likelihood training |    $\int_{\lambda_T}^{\lambda_0} \mathbb{E}[ \|\epsilon_\theta(\hat x_{\lambda},\lambda)-\epsilon \|_2^2 ]\mathrm{d}\lambda$ |
| Sampling by diffusion ODEs  | $\hat x_{\lambda_t} = \frac{\hat\alpha_{\lambda_t}}{\hat\alpha_{\lambda_s}}\hat x_{\lambda_s} - \hat\alpha_{\lambda_t} \int_{\lambda_s}^{\lambda_t} e^{-\lambda} \hat\epsilon_\theta(\hat x_\lambda,\lambda)\mathbb{d}\lambda$ |

[1] Kingma, Diederik, et al. "Variational diffusion models." Advances in neural information processing systems 34 (2021): 21696-21707.

[2] Song, Yang, et al. "Maximum likelihood training of score-based diffusion models." Advances in Neural Information Processing Systems 34 (2021): 1415-1428.

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Summary of the revision (cont.)
comment: 
## 3. Add relevant baselines, related work, and more discussions about the limitations.

We have updated Sec.5 and Fig.2 in the revision to include the mentioned PNDM, FastDPM, and Itô-Taylor methods. As the proposed DPM-Solver and these relevant baselines all use the same discrete-time pre-trained models, we can directly compare the sample quality between DPM-Solver and these methods. Still, the experimental results show that DPM-Solver with 12 NFEs can achieve $\sim$4$\times$ speedup compared with all of the baselines.

In addition, we have updated Sec. 4.3 and Sec. 6 to discuss more the related methods and limitations.

From: NeurIPS.cc/2022/Conference/Paper5563/Authors
title: Summary of the revision
comment: We sincerely thank all the reviewers for the valuable comments, which help to improve the quality of our work. We summarize the revision in the updated version as follows:

## 1. High-resolution dataset experiments.

We further evaluate DPM-Solver, DDIM, and DDPM on two high-resolution datasets: ImageNet 128x128 and LSUN bedroom 256x256. For both experiments, we use the pre-trained model in [1]. For ImageNet 128x128, we use the best settings for DDIM in [1], i.e., classifier guidance with a scale of 1.25. For LSUN bedroom 256x256, we use their released unconditional diffusion model.

We have updated Sec.5, Fig.2, and Appendix E.5 for each experiment's detailed results and settings. Below we list the detailed sample quality evaluated by FID, varying the NFE. Due to the time constraints of the rebuttal, we only evaluate DDPM for 10, 50, 250 NFE, DDIM for 10, 12, 15, 20, 50, 100 NFE, and DPM-Solver for 10, 12, 15, 20, 50 NFEs. Nevertheless, the results are sufficient to show the effectiveness of DPM-Solver.

For instance, for ImageNet 128x128, DPM-Solver can achieve 4.08 FID with 12 NFE, while DDIM needs 100 NFE for 4.18 FID; and DPM-Solver can achieve 3.52 FID with 20 NFE, which outperforms DDIM with 250 NFE (3.54 FID). For LSUN bedroom 256x256, DPM-Solver can achieve 2.94 FID with 15 NFE, while DDIM needs 50 NFE for 2.92 FID; and DPM-Solver can achieve 2.01 FID with 50 NFE, which even outperforms DDPM/DDIM with 250 NFE (2.02 FID, which is the best results in [1]). 

Therefore, **DPM-Solver is still suitable for high-resolution image datasets**. Besides, we will finish the intermediate NFE experiments (the"$\backslash$" in the table) for completeness in the camera-ready version. In addition, as DDIM can accelerate the sampling of the audio data, we believe that DPM-Solver is also suitable for the audio data because DDIM is the first-order case of DPM-Solver, and we leave it for future work.

[1] Dhariwal, Prafulla, and Alexander Nichol. "Diffusion models beat gans on image synthesis." Advances in Neural Information Processing Systems 34 (2021): 8780-8794.

**Table 1.** Sample quality of DPMs measured by FID $\downarrow$ on ImageNet 128$\times$128 with classifier guidance and on LSUN bedroom 256$\times$256, varying the number of function evaluations (NFE).

| Sampling method $\backslash$ NFE | 10       | 12           | 15           | 20           | 50       | 100          | 250          |
| -------------------------------- | -------- | ------------ | ------------ | ------------ | -------- | ------------ | ------------ |
| **ImageNet 128$\times$128**      |          |              |              |              |          |              |              |
| DDPM                             | 199.56   | $\backslash$ | $\backslash$ | $\backslash$ | 49.38    | $\backslash$ | **2.97**     |
| DDIM                             | 11.12    | 9.38         | 8.22         | 7.15         | 5.05     | 4.18         | 3.54         |
| DPM-Solver                       | **7.32** | **4.08**     | **3.60**     | **3.52**     | **3.13** | $\backslash$ | $\backslash$ |
| **LSUN bedroom 256$\times$256**  |          |              |              |              |          |              |              |
| DDPM                             | 274.67   | $\backslash$ | $\backslash$ | $\backslash$ | 82.70    | $\backslash$ | 2.02     |
| DDIM                             | 10.05    | 7.51         | 5.90         | 4.98         | 2.92     | 2.30         | 2.02 |
| DPM-Solver                       | **6.10** | **4.29**     | **2.94**     | **2.60**     | **2.01** | $\backslash$ | $\backslash$ |


## 2. Invariance property of DPM-Solver and relationship with maximum likelihood training of DPMs.

We have updated Appendix A to show that DPM-Solver has a significant property that it is invariant to the specific noise schedule ($f(t)$ and $g(t)$). Moreover, choosing time steps for $\lambda(t)$ instead of $t$ is also invariant to the noise schedule. In addition, our proposed invariant sampling method is highly related to the invariance property of maximum likelihood training of DPMs in [2]. Such invariance property for training and sampling brings new insights for understanding diffusion models. For instance, we can directly define the noise prediction model $\epsilon_\theta$ w.r.t. the (half-)logSNR $\lambda$ instead of the time $t$, then the training and sampling for diffusion models can be done **without further choosing any ad-hoc noise schedules**. Such findings may unify the different ways of training and inference of diffusion models, and we leave it for future study.

[2] Kingma, Diederik, et al. "Variational diffusion models." Advances in neural information processing systems 34 (2021): 21696-21707.

From: NeurIPS.cc/2022/Conference/Paper5563/Reviewer_DD9K
rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: It has been showed recently that diffusion probabilistic models can be understood as ODE, which is useful to devise efficient prediction methods. This made it possible to strongly accelerate inference by several orders of magnitude compared to original SDE-based methods.

In this line of research, this paper shows that even more progress can be made by leveraging a decomposition of the solution as a sum of a linear and a non-linear part. The method can be used on off-the-shelf and already trained DPM, and doesn't require fine-tuning. 
strengths_and_weaknesses: The paper is both a challenging but an exciting read. It is challenging because it is pretty rigorous and the math often goes beyond my comfort zone. Still, it is exciting because it is self contained and really brings an interesting idea on the table, to devise a specific solver that allows effective inference with DPM.

Regarding the weaknesses, maybe I should say that the code provided in the supplementary is not really advertised in the paper and I think it could really benefit from some work. As such it is not documented at all so that it is really difficult to get into it. I think it would be worth it to spend some effort to make that better.
questions: * it would help me if you could provide some expression for \lambda(t). I guess this is really important in practice, and this is something that is missing. What does it mean for t when \lambda is uniformly picked ?
* I understand that your contribution is mostly relative to having faster inference, but do you think the insights you bring could actually be useful for training ?
limitations: nothing special
ethics_flag: No
soundness: 4 excellent
presentation: 4 excellent
contribution: 4 excellent

From: NeurIPS.cc/2022/Conference/Paper5563/Reviewer_Wm2X
rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: This paper proposes a training-free few-step sampler of diffusion probabilistic models (DPMs) by leveraging semi-linear structure in the diffusion ODEs to approximate the solutions of the equation. The proposed method can accelerate the sampling of DPMs without sacrificing the sampling quality.

strengths_and_weaknesses: Strength

- (Quality) The authors effectively provide a theoretical derivation and error analysis of the proposed method and the experiments' details.
- (Clarity) The paper is well-written and clear to understand. The reviewer could understand the contents without difficulty.
- (Significance) Numerical experiments verify that DPM-solver significantly outperforms previous sampling methods in terms of quality with fewer function evaluations. The reviewer considers this paper deserves to be accepted by NeurIPS if the authors include the runtime comparison between the proposed and other methods.

Weakness

- (Originality) The proposed method shows excellent performance; however, its theoretical background is based on the variation of constants which is a classical idea in the literature of numerical differential equations, so one can think the originality is somewhat marginal.

***Updated Review***

The reviewer appreciates the authors' sincere answers. I have updated my score and hope the paper is accepted to the NeurIPS conference.
questions: Would you please include the comparison of runtime for sampling between the proposed method and other methods? The reviewer is concerned that the computational time would be a critical weak point in the proposed method.

limitations: The authors mentioned the limations of their work properly.
ethics_flag: No
soundness: 4 excellent
presentation: 4 excellent
contribution: 4 excellent

From: NeurIPS.cc/2022/Conference/Paper5563/Reviewer_EaE9
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: This paper tackles the slow-sampling problem in denoising diffusion generative models. Diffusion models have demonstrated outstanding synthesis quality, together with good sample diversity and mode coverage, but they are slow during sampling, since many neural networks calls are required to iteratively solve a generative differential equation and generate a single batch of data. To this end, the paper proposes a novel ODE solver, called *DPM-Solver*, that can very quickly solve the generative ODE of diffusion models. The generative ODE, also known as probability flow ODE, is of semi-linear form, consisting of both a linear and a non-linear (neural network) term. The paper proposes an ODE solver that solves the linear component analytically and exactly, while only requiring a numerical approximation for the non-linear term. The non-linear term, formally a time integral over the learnt score function with an exponential weighting, can be solved with different orders of accuracy (implying different accuracy-compute trade-offs). Overall, the framework can be interpreted as an exponential integrator approach tailored to diffusion models, with a Runge-Kutta-like technique applied to only the non-linear term. Experimentally, the paper demonstrates significantly accelerated diffusion model sampling for 3 low-resolution image generation tasks (CIFAR-10, CelebA 64x64, ImageNet 64x64), comparing to various well-established baselines.
strengths_and_weaknesses: **Stengths:**
- The paper proposes a rigorously derived and technically solid ODE integrator for fast sampling from diffusion models. The proposed method, *DPM-Solver*, is well-motivated based on techniques from the ODE literature (exponential integrators).
- The experimental results on low-resolution image synthesis successfully validate the method and show a clear advantage over the baselines.
- Overall, the paper is easy to follow and clearly explained (the English could be better, but that's okay; maybe some proofreading from a native speaker might help). The paper nicely discusses its differences compared to the most commonly used ODE solvers in diffusion models, this is, Runge-Kutta methods and DDIM [1].
- I think the contribution is sufficiently significant. The tackled problem, accelerating sampling from diffusion models, is very specific. However, this is a very relevant problem, considering that slow sampling currently is diffusion models' main weakness.

**Weaknesses:**

I do not think that the paper has major flaws. However, there are some points of criticism:
- The paper misses relevant baselines in its experimental evaluations: In particular, it would be very appropriate to quantitatively compare to "Pseudo Numerical Methods for Diffusion Models on Manifolds" [2], which is a published method that also accelerates diffusion model sampling with higher-order ODE methods. Furthermore, "FastDDIM" [3] would another appropriate baseline. Potentially, also Tachibana et al. [4] could be included. I actually think the proposed solver will outperform all these baselines. Nevertheless, I think a more rigorous evaluation with all possible baselines would strengthen the paper significantly, considering that accelerated sampling is the paper's main and only contribution.
- Beyond fast ODE samplers, the literature on diffusion models includes also many other ideas for accelerated sampling. For instance, sampling can be accelerated by working in a smooth latent space [5], by using GAN-based large-step denoising [6], of by devising tailored diffusion processes that can be denoised efficiently [7]. These works are not even cited. So, considering that this paper tackles accelerated diffusion model sampling, I think a more thorough discussion on the relevant literature is missing.
- Experimentally, the paper only uses low-resolution image datasets. The paper could be improved by also applying the method on other problems, such as high-resolution image synthesis or audio synthesis, where diffusion models have also been widely applied.

**Conclusion:**
Overall, I think this is a solid paper that deserves to be accepted. It successfully tackles a specific but relevant problem in the diffusion model literature in a convincing and technically solid way. It might be worth noting that there is highly related concurrent work that proposed a very similar method [8]. However, I don't think that this should have any negative impact here. I think exponential integration techniques are very natural to use in diffusion models, and having multiple works concurrently analyzing and demonstrating that independently in different ways is good. All that said, it also needs to be mentioned that the paper could still be significantly improved in various ways. In particular, more thorough experiments could be run and additional baselines could be considered (see weaknesses and questions below).

[1] Song et al., "Denoising Diffusion Implicit Models", ICLR 2021

[2] Liu et al., "Pseudo Numerical Methods for Diffusion Models on Manifolds", ICLR 2022

[3] Kong and Ping, "On Fast Sampling of Diffusion Probabilistic Models", arXiv 2021

[4] Tachibana et al., "Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives", arXiv 2021

[5] Vahdat et al., "Score-based Generative Modeling in Latent Space", NeurIPS 2021

[6] Xiao et al., "Tackling the Generative Learning Trilemma with Denoising Diffusion GANs", ICLR 2022

[7] Dockhorn et al., "Score-Based Generative Modeling with Critically-Damped Langevin Diffusion", ICLR 2022

[8] Zhang and Chen, "Fast Sampling of Diffusion Models with Exponential Integrator", arXiv 2022
questions: I also have some minor additional questions and suggestions:

- In the concurrent work [8], a very similar scheme, *DEIS*, is proposed. I think it would be interesting to compare and discuss the detailed differences between the method proposed here, DPM-sampler, and DEIS and also run detailed quantitative comparisons. My understanding is that DPM-Solver primarily leverages Runge-Kutta-like techniques to solve the non-linear term, while DEIS also explored muiltistepping methods that rely on past function evaluations. Furthermore, it seems the papers use different time-reparametrizations. I'd be curious how the $\rho$-parametrization in DEIS (Eq. (18) in their paper) and the $\lambda_t$-parametrization here are related. To this end, I am also wondering whether maybe the ideal future solver is based on a combination of ideas from both of these papers? As mentioned, since this is concurrent work, I don't consider this as a criterion for recommending acceptance here. But it would be nice and insightful for the community. I'd be curious about the authors' thoughts on this.

[8] Zhang and Chen, "Fast Sampling of Diffusion Models with Exponential Integrator", arXiv 2022
limitations: Limitations have been discussed only very briefly (the proposed ODE solver may not be suitable for also accelerating likelihood calcuation). It seems fine to me, although this could be extended a bit, ideally. In particular, the main limitation is that the sampler is still not fast enough to generate model samples so quickly that diffusion models can be used in real-time, interactive applications.

Potential negative societal impacts have been briefly mentioned, in a satisfactory manner.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good