Paper: Efficient Integrators for Diffusion Generative Models (ICLR-2024)

Reviewer comment (Experiments):
There are too many proposed integrators, and it is difficult to figure out which one is / ones are the "main" contribution of the paper. More specifically, it is difficult to see in which situation we should prefer one integrator over the other. The authors appear to take a very unclear stance on their "main" sampler: they propose a whole arsenal of samplers and pick results which happened to beat some baselines under very specific dataset+sampler+hyper-parameter combinations.

Author response:
Response: We address this question in different parts

>There are too many proposed integrators, and it is difficult to figure out which one is / ones are the "main" contribution of the paper. 

As highlighted in Section 4 (Notation), the main samplers that we recommend using are the Conjugate-Splitting based PSLD samplers (CSPS) and Splitting based PSLD samplers (SPS). We understand that the samplers specified in Table 2 might suggest a lot of contributions. However, we would like to emphasize that Table 2 lists our ablation samplers, the discussion of which is necessary to construct our main samplers, which build on top of these samplers discussed in Table 2.

>More specifically, it is difficult to see in which situation we should prefer one integrator over the other

From Figure 5, we make the following observations. For deterministic sampling, our CSPS samplers always perform better than SPS samplers. For stochastic sampling, except CIFAR-10, the CSPS integrators always perform better than the corresponding SPS samplers. Therefore, for most practical applications, the proposed CSPS samplers should be preferred.

> the authors are proposing a whole arsenal of samplers, and picking results which happened to beat some baselines results under very specific dataset+sampler+hyper-parameter combinations.

We would like to point out that we do not perform any type of cherry-picking when comparing different sampler baselines and our proposed samplers. On the contrary, we perform an extensive comparison across multiple sampling budgets for different datasets in Figure 5, which clearly illustrates when our deterministic and stochastic samplers perform comparably or outperform different baselines. Moreover, our choice of datasets is standard in the diffusion model literature [1,2,3,4]. Regarding hyperparameters, we tune the value of \lambda for CSPS samplers during inference since Theorem 2 suggests that a specific value of \lambda is unlikely to work across multiple integration step sizes h (i.e., for different sampling budgets in terms of NFEs) and is thus a core requirement for CSPS samplers. Moreover, since we evaluate other baselines based on the scores provided in their respective papers, it is safe to assume that our comparison baselines have also been optimized for best performance.

Reviewer comment (Theory):
The rationale behind choosing $B_t = \lambda I$ and $B_t = \lambda 1$ is unclear. It is also unclear when DDIM-I should be preferred over DDIM-II and vice versa. While there is a theorem on DDIM-I, there is no theorem on DDIM-II, which could clarify differences between DDIM-I and DDIM-II.

Author response:
Response:  We thank the reviewer for this question. The main rationale behind choosing $B_t$ as in $\lambda$-DDIM is mostly simplicity. Our current choice of $B_t =\lambda I$ assumes that the value of $\lambda$ is independent of time $t$, which might be overly simplistic since Theorem 2 suggests that at any time $t$, the eigenvalues $\tilde{\lambda}$ must be bounded for a stable conjugate integrator, therefore, implying a time-varying $\lambda$ might be a better choice. However, we make this simplistic choice because a time-independent lambda is more straightforward to tune during inference. We had the same rationale when choosing $B_t = \lambda 1$.


>While there is a theorem on DDIM-I, I don't see any theorem on DDIM-II, which could clarify differences between DDIM-I and DDIM-II.


We would like to point out that for the choice of $B_t = \lambda I$, the result in theorem 2 simplifies, resulting in Corollary-1. However, similar simplifications are not apparent for other choices of $B_t = \lambda 1$, and Theorem 2 holds in general for other choices of $B_t$. We acknowledge that further theoretical investigation of different choices of $B_t$ could clarify differences between the two formulations and could be an interesting future direction. We already state this point as an interesting direction for future work in the conclusion.

Reviewer comment (Evaluation):
Readers might expect conjugate+splitting integrators to outperform conjugate/splitting integrators, as they combine the best of both worlds. However, Table 2 shows Reduced OBA outperforming Conjugate OBA. The authors hypothesize "this might be due to a sub-optimal choice of $B_t$" — this kind of explanation may confuse readers, as it does not provide any guide on when to use or avoid conjugate splitting.

Author response:
While we see this trend for CIFAR-10 in Table 2, we would also like to highlight that for the CelebA64 and the AFHQv2 datasets, CSPS-S (a.k.a Conjugate OBA) outperforms SPS-S (a.k.a Reduced OBA) (see Figure 5 for comparisons). 


>The authors hypothesize "this might be due to a sub-optimal choice of Bt" -- I think this kind of explanation only confuses the readers, as it does not provide any guide on when we should and should not use conjugate splitting.


We would like to point out that the performance of CSPS (and related conjugate integrators) depends significantly on the choice of $B_t$. The choice of optimal $B_t$ is unlikely to be the same for all practical datasets, which is the point in the main text. We will elaborate on this fact in more details in our final revision. In practice, we find that our two considered choices perform well across most of our experimental datasets. Therefore, in general, for both deterministic and stochastic samplers, we recommend using Conjugate Splitting based samplers.

Reviewer comment (Theory):
One factor contributing to confusion is the lack of theoretical results — the authors mostly rely on numerical results to judge integrator performance. A theorem comparing, for example, truncation error of proposed samplers would greatly strengthen the paper.

Author response:
We would like to highlight that we provide an extensive truncation error analysis for deterministic naive and reduced splitting integrators in Theorem 3 (with corresponding proofs in Appendix C.2.3). In fact, we arrive at the design choices for the reduced splitting integrators based on truncation error analysis for both deterministic and stochastic samplers.

Reviewer comment (Evaluation):
The authors claim that if $B_t = 0$, the conjugate integrator is equivalent to DDIM. However, Figure 2(a) shows that FID for DDIM ($\lambda$-DDIM with $B_t = 0$) is too poor compared to results in the original DDIM paper. For example, in the original DDIM paper, DDIM achieves FID 13.36 with NFE=10, while in Figure 2(a), FID > 50.

Author response:
Response: We think there is a misunderstanding here. The results in Fig. 2(a) are for DDIM (which is equivalent to our conjugate integrator with $B_t=0$) applied to the PSLD ODE, which is an augmented diffusion model (i.e., data space coupled with some auxiliary variables). In contrast, in the original DDIM paper, the sampler is developed for DDPM, which is a non-augmented diffusion model and is different in design than PSLD. This result also demonstrates an important aspect of DDIM (and related solvers like DEIS and DPM-Solver) that solvers developed for non-augmented diffusion models like VP-SDE/DDPM may not generalize directly to other types of diffusion models. Therefore one of our main contributions in this work is to develop a framework for designing solvers for a broader class of diffusion models like PSLD.

Reviewer comment (Experiments):
Results for EDM + pre-conditioning are missing in Figure 5. The authors add pre-conditioning for their methods but not for EDM. It is unclear if this is because EDM + pre-conditioning outperforms the proposed integrators. For example, EDM + pre-conditioning achieves 1.97 FID with NFE=35, which beats the best result in this paper, 2.11 FID with NFE=100.

Author response:
As can be observed, the FID score of 1.97 in NFE=35 for unconditional CIFAR-10 is achieved after using a number of tricks during training, including extensive hyperparameter tuning, preconditioning, customized loss functions, and an extensive data augmentation pipeline. We emphasize that we do not explore any of these choices (except preconditioning) for improved PSLD training for the following reasons:



1. Firstly, our comparison baselines like DEIS, DPM-Solver, etc., were evaluated on a vanilla VP-SDE model, and hence tuning such choices for our method would likely result in unfair comparisons.
2. Secondly, tuning for all these design choices is computationally expensive.

    Therefore, the claim that EDM + preconditioning achieves a FID score of 1.97 which beats the best result in our paper, is inaccurate since the comparison is not fair in the first place.


    Result for EDM + pre-conditioning is missing in Figure 5. Why do the authors add pre-conditioning for their methods, but not for EDM? Is it because EDM + pre-conditioning out-performs the proposed integrators?


    To incorporate preconditioning in EDM while maintaining fair comparison with other methods, we trained an unconditional VP-SDE baseline model (Config A in the table above) + EDM’s preconditioning on CIFAR-10 (with 1/C_out as the loss weighting) using their EDM’s official implementation (the hyperparameters during training were set to the baseline parameters in Table 7 in the EDM paper). During sampling, we use the recommended settings in the EDM sampler, i.e., their discretization, choice of scaling and schedule, and Heun’s solver since these are core aspects of their sampler. We compare the sample quality results for different solver budgets below:


|  Model                 | 30       | 50       | 70       | 100      |
|------------------------|----------|----------|----------|----------|
| EDM (w/o precond.)     | 3.18     | 3.08     | 3.06     | 3.06     |
| EDM (+precond)         | 5.26     | 5.12     | 5.13     | 5.12     |

**Table 3**: Comparison between EDM (Config A) with and without EDM specific preconditioning on the CIFAR-10 dataset. All FID scores are reported on 50k samples using the optimal deterministic sampler proposed in EDM. 


Interestingly, training VP-SDE with the preconditioning proposed in EDM leads to worse performance than without preconditioning. In any case, our proposed samplers outperform both these baselines for CIFAR-10.

Reviewer comment (Experiments):
Results on higher-dimensional data are missing. Additional results on image resolutions $\geq 512$ are desired.

Author response:
Response: Since we work with a broader and more recent class of diffusion models like PSLD, the pre-trained models available for these augmented diffusion models are primarily trained on small-scale datasets like CIFAR-10 and CelebA-64. Therefore for additional results on >=512 resolution images, we would need to train a new model from scratch. We would like to emphasize that training such high-resolution augmented models is difficult for two reasons:

1. Firstly, training such high-res diffusion models is another active area of research (e.g., DALL-Ev2, Imagen, StableDiffusion) which we don’t consider in this work.
2. Secondly, training such large-scale models is extremely compute-intensive, which is prohibitive for us. For instance, for training on ImageNet at 64x64 resolution, the authors in EDM train the diffusion model for two weeks on a setup with 32 GPUs which is computationally expensive. Training models with resolutions >512 scales proportionately.

Reviewer comment (Experiments):
Results on conditional generation are missing. Additional experiments on class-conditional and text-conditional generation would be useful.

Author response:
Response: As mentioned in our previous response, training text conditional models is computationally prohibitive. We acknowledge that investigating the efficacy of proposed solvers for these tasks could be an interesting direction for future work in the context of augmented diffusion models. However, most of these tasks are beyond the scope of this work.

Reviewer comment (Evaluation):
The relative behavior of stochastic (SPS-S, CSPS-S) and deterministic samplers (CSPS-D) is unintuitive. Common knowledge is that deterministic samplers outperform stochastic samplers in the low-NFE regime, and vice versa in the high-NFE regime. However, Figure 5 bottom shows a reverse trend. Clarification is needed on why this happens.

Author response:
We thank the reviewer for this question. In the context of non-augmented diffusion models (like VP-SDE, VE-SDE, etc.), indeed, deterministic samplers outperform stochastic samplers in the low-NFE regime and vice-versa in the high-NFE regime. However, for PSLD (which is an augmented diffusion model), this trend does not seem to hold (at least in the context of samplers presented in this work). We hypothesize that this might be an effect of noise injection from two sources during stochastic sampling. Firstly, we explicitly inject stochasticity in the position space controlled with the parameter \lambda_s (as part of our design choices adopted for reduced splitting integrators). Secondly, noise is also injected into the data space due to coupling with momentum variables during sampling. This might lead the sampler to escape regions of local minima during sampling. Note that such noise injection does not happen for deterministic sampling, which might lead to slower convergence to good samples (especially pronounced in higher resolution datasets like CelebA-64 and AFHQv2.) However, a detailed theoretical investigation of this observation would be an interesting direction for future work.

Reviewer comment (Experiments):
Combinations between datasets and methods are not consistent. For example in Figure 5, EDM is not tested on CelebA-64. Furthermore, these methods must be tested in more complex distributions such as ImageNet.

Author response:
**Combinations between datasets and methods are not consistent. For example in Figure 5, EDM is not tested on CelebA-64.**


Response: In Figure 5 (in the main text), we report FID scores for different baseline samplers based on the results in their respective papers or by using pre-trained checkpoints from their official code implementations. Consequently, for a given dataset, we do not include results for a baseline if the corresponding pre-trained diffusion checkpoint is missing or the paper lacks results for that dataset.  


**Furthermore, these methods must be tested in more complex distributions such as ImageNet.**


Since we demonstrate the proposed samplers in the context of PSLD (a recently proposed state-of-the-art diffusion model), due to the absence of pre-trained models for this type of diffusion model, we would need to re-train PSLD models on large datasets like ImageNet. Training large-scale diffusion models is typically computationally prohibitive, and thus in this work, we only consider small-scale datasets for which training fits within our compute limits or datasets for which pre-trained PSLD checkpoints are available. For instance, for training on ImageNet at 64x64 resolution, the authors in [3], train the diffusion model for two weeks on a setup with 32 GPUs which is computationally expensive.

Reviewer comment (Reproducibility):
The training of each pretrained model and the size of the model is not specified in the main paper. Are these models identical and trained for the same amount of time?

Author response:
Response: We thank the reviewer for pointing this out. For most competitive baseline results presented in Figure 5 (like DEIS, DPM-Solver, etc.), which use the VP-SDE diffusion baseline, the size of the pre-trained diffusion model is around 108M parameters, while the size of the pre-trained PSLD model used to evaluate our proposed sampler is 97M parameters. Both models are trained for approximately the same number of steps for CIFAR-10 (around 800k-1M steps). Therefore, our comparisons are fair in terms of sample quality and time for each score function evaluation. We have added these details in Appendix E.2.

Reviewer comment (Theory):
In Collorary 1, the authors show a connection between stability and the parameter $\lambda$ (which should evolve with time?), however it is not clear how this hyperparrameter is chosen. Furthermore, Corollary 1 does not explain the behaviour of $\lambda$-DDIM-II, which performs best.

Author response:
Response: We would like to point out that the main motivation behind including Theorem 2 is to give intuition about when projecting to the space $\hat{z}_t$ (which is parameterized by the design choice $B_t$) is a good idea. Furthermore, since we employ simplistic choices of $B_t = \lambda I$, we include Corollary 1 to provide further intuition since the expression for $\bar{\Lambda}$ simplifies. With this motivation in mind, we address the reviewer’s question in two parts:


**In Collorary 1, the authors show a connection between stability and the parameter λ (which should evolve with time?), however it is not clear how this hyperparrameter is chosen**


In this work, the parameter $\lambda$ is chosen at inference using grid search. Specifically, given a step size h, we tune $\lambda$ using grid search to optimize for best sample quality. In principle, while $\lambda$ can be time-dependent, we use a rather simplistic choice by setting $\lambda$ to be a constant scalar across all timesteps. While this might seem overly simplistic (due to the result in Corollary-1, which suggests that $\lambda$ would be rather time-dependent), it works well empirically and significantly reduces the effort required in tuning $\lambda$ during inference.


**Furthermore, Corollary 1 does not explain the behaviour of λ-DDIM-II, which performs best.**


As stated above, choosing $B_t = \lambda I$ simplifies the expression for $\bar{\Lambda}$. However, the same is not true for the choice of $B_t = \lambda 1$, and thus it is difficult to argue about the better performance of $\lambda$-DDIM-II over $\lambda$-DDIM-I. Therefore, further investigation into different theoretical aspects of Conjugate Integrators remains an exciting direction for future work. We already make this point more explicit in our future directions section at the end of the main text.

Reviewer comment (Evaluation):
Do the authors have an explanation why some methods outperform CSPS-D for very small numbers of NFE? (Figure 5, up-left plot).

Author response:
Response: We thank the reviewer for this question. For a very small number of NFEs, baselines like DPM-Solver, and DEIS outperform our proposed solvers. To intuitively understand this observation, we highlight that we evaluate our proposed samplers on the PSLD diffusion trained with the epsilon prediction objective. As noted in [1], for augmented space models like CLD, the dynamics in the epsilon space are oscillatory in nature, which prohibits taking larger step sizes with these models. On the other hand, baseline samplers in Figure 5, like DPM-Solver and DEIS, are evaluated on VP-SDE, which does not suffer from this problem and thus allows for larger step size. To quantify this intuition, we generalize DEIS and apply it to the PSLD diffusion ODE. The results in Table 1 below show that DEIS applied to PSLD performs significantly worse in comparison to the proposed samplers in this work across all timesteps.

Reviewer comment (Evaluation):
Have the authors measured the total integration time? If the generation speed is our main objective, then such results should be provided as well.

Author response:
Response: We assume the reviewer means the integration time for computing the coefficients required for ODE integration. Given a set of sampling timepoints ${t_i}$, since coefficients are shared between all samples, we only need to perform this process once at the start of sampling. Empirically, for our largest budget of 100 time points, numerical integration for computing coefficients takes around 20s on our setup. We have added these results in Appendix B.6

Reviewer comment (Experiments):
I don't know whether I deviate too much from the author's experiences. In my experience, in most cases, 25-step DPM-solver sampling could produce good enough sampling results for large models like StableDiffusions on large diverse image distribution; and less than 15-step DPM-solver sampling could produce good enough results for small diffusion models in small data domain. Please tell me if you observe a different phenomenon, and I will be happy to test it. Getting back to the subject, are the 100 and 50 NFEs considered by this paper a bit redundant?

Author response:
Response: For non-augmented diffusion models (i.e., without auxiliary variables) like Stable Diffusion, indeed, DPM-Solver can produce plausible samples in 15-25 steps. However,

1. Firstly, Stable Diffusion [1] is position space only, i.e., there are no auxiliary variables in its design. In this work, we consider the sampler design for a broader class of diffusion models, and it's not apparent whether DPM-Solver works effectively in such cases. In this work, we propose Conjugate Integrators, which allow us to bring samplers like DEIS [6], DPM-Solver [7] (see our response to your point 5), and DDIM [8] in a common framework. This further allows us to extend these samplers in a principled way to a more broad class of diffusion models like PSLD. This is illustrated in Table 1 below, where we first extend DEIS (which is an exponential-based integrator like DPM-Solver) to PSLD and show that it does not generalize well to diffusions in augmented spaces like PSLD even for NFE=50 and NFE=100, which justifies our use of 100 and 50 NFEs in this work.
2. Secondly, for data-space-only diffusions like VP-SDE, it is commonly observed that state-of-the-art ODE-based samplers like DPM-Solver and DEIS converge quickly in fewer steps but saturate to a sub-optimal FID for even smaller datasets like CIFAR-10 (see the Top-left subfigure in Figure 5) and therefore might not be suitable for applications which might allow for a larger sampling budget. On the contrary, though applied to the PSLD diffusion, our proposed sampler in Figure 5 converges to a better FID score of 2.11 on CIFAR-10 (vs. 2.59 for DPM-Solver). Therefore, when evaluating diffusion models, we think it is more reasonable to evaluate sampler quality on a range of sampling budgets (visualized as sampling speed in NFE vs sample quality in FID). 

Overall, we think the sampling budgets of 50 and 100 NFEs are **not** redundant for the results presented in this paper.

Reviewer comment (Theory):
The stability analysis seems to avoid the key points? I would like to know a two-phase conclusion: first, when will matrix A_t stability have an inverse, this is the key to the stable conjuncture; second, under the condition that A_t is inversible and stable, when will eq 7 and 8 as ODE solvers be stable? Theorem 2 seems like a naive application of ODE stability conditions, and gets rid of the key part. A_t is an integral of multiple components, so we may not directly assume it to be stably invertible. But you also do not assume it stable anyway (minimum eigenvalue larger than some positive constant). It would also be helpful if the authors could explain in what reality settings those conditions can be satisfied.

Author response:
Response: We thank the reviewer for pointing this issue out. We address the question in two parts.


**When will A_t be invertible:**


Theoretically, since the expression for $A_t$ in Eqn. 6 consists of multiple terms in the integral with the matrices $B_t$, $F_t$, and $C_\text{skip}(t)$ having no specific properties, it's non-trivial to conclude if arbitrary choices of these matrices can ensure invertibility on $A_t$. An alternate (but less elegant) fix would be to update our mapping instead to ensure an invertible transformation from $z_t$ to $\hat{z}_t$ by setting $\hat{z}_t = (A_t + \delta I) z_t$ where $\delta > 0$ is a small constant to ensure non zero eigenvalues at any time t. 


However, empirically, when using the PSLD diffusion model (as considered in this work), we do not notice any instabilities because of this issue across our experiments with multiple datasets. We have highlighted this aspect in Section 3.1 and Appendix B.5 in the paper.


**When will the ODE solvers derived in Eqn. 8 be stable:**

Theorem 2 states when the ODE integrator in Eqn. 8 is stable, provided the mapping $A_t$ is invertible. We have updated Theorem 2 to reflect the assumption on the invertibility of $A_t$.

Reviewer comment (Theory):
Theorem 3 gives error analysis based on h, I have concerns about its error with respect to the original t. After the projection is to solve the original problem, measuring the error in a projected space could be less meaningful. This is the same problem in the DPM-Solver paper, they only analyze errors when lambda is stably transformed from t, but in reality is often not the case. But DPM-Solver performs great generally so I am not criticizing it. Just point out that error analysis with respect to t variable is preferred for typical ode numerical analysis.

Author response:
Response: We think there might be some misunderstanding here. 


Firstly, we would like to clarify that Theorem 3 presents the error analysis for Naive Splitting Integrators (specifically for the Naive Velocity Verlet (NVV) sampler). As presented in Section 3.2, Splitting Integrators are independent of the Conjugate Integrators presented in Section 3.1 and, thus, do not involve a projection step. Therefore, the truncation error in Theorem 3 is measured in the original rather than the projected space. Furthermore, we don’t combine Conjugate Integrators with Splitting Integrators until Section 3.3. To clarify further, in Section 3.3, we numerically solve individual splitting updates using conjugate integrators.


Secondly, the error analysis in Theorem 3 is based on $h$, which is the step size during numerical integration. We would like to point out that error analysis based on the integration step size is commonplace in the numerical analysis of ODEs. We hope this clarifies any misunderstandings about the proposed samplers.

Reviewer comment (Experiments):
This work also reminds me of the DPM-Solvers. DPM-Solver could be viewed as a special case, it also projects the integral to another space, the lambda space in fact. It uses a prediction-correction (PC) method to achieve much higher accuracy in the projected space. So I care about two things: first how the dpm-solver will formulate under your settings and when will your method outperform it? Second, can your method benefit from PC? While this work does not compare with DPM-Solver, considering the huge influences of and similarity with DPM-Solver, it could be better if the authors could compare with it.

Author response:
Response: We address this question in multiple parts.


**DPM-Solver as a particular case of Conjugate Integrators**


We thank the reviewer for their intuitive insight. During the response period, we established a theoretical connection of Conjugate Integrators with DPM-Solver [1]. More specifically, we find that the DPM Solver sampler for the position space-only diffusion models considered in [1] is a particular case of conjugate integrators with the choice of matrix $B_t = 0$. We have updated Proposition 2 in the main text with proof in Appendix B.3 to reflect the same. 


**When does our method outperform DPM-Solver?**


Our method can outperform DPM-Solver (or, more broadly, other exponential integrator-based methods like DEIS) in the following use cases.

1. **Exponential-based integrators do not generalize automatically to diffusion models in augmented spaces:** Firstly, with the perspective presented in DPM Solver [1] and other exponential-based integrators like DEIS [5], it is not apparent how to generalize these samplers to a broader class of diffusion models i.e. augmented diffusion models (where diffusion is performed in the data space concatenated with some auxiliary variables like in CLD [2] or PSLD [3]) as considered in this work. Theoretically, one of our main contributions is a framework that generalizes ideas in fast diffusion sampling like DDIM [4], DPM-Solver [1], and DEIS [5] under a single unified framework (see Propositions 1,2 in the main text). The benefit of such a framework is that it allows the construction of analogous samplers for different types of diffusion models like CLD and PSLD (which could be helpful in different contexts) and not just position space-only diffusion models (like Stable Diffusion). Moreover, a principled framework like ours allows us to extend beyond exponential integrator-based samplers. The resulting samplers can generalize better to other types of diffusion models. 

    To illustrate our point, empirically, we compare the FID scores of DEIS (generalized to PSLD by setting $B_t=0$) with “\lambda-DEIS” (obtained by setting $B_t=\lambda 1$) applied to the PSLD reverse diffusion in Table 1 (in this response) for the CIFAR-10 dataset. Our extension of DEIS i.e. $\lambda$-DEIS, outperforms the former by a large margin. We expect similar results for DPM-Solver since DEIS and DPM-Solver are exponential integrator-based methods and exhibit similar empirical performance in practice [5]. These results demonstrate that the exponential integrator-based framework does not generalize to novel diffusions automatically, therefore justifying the need for a broader framework as proposed in this work._

2. **DPM Solver saturates in sample quality for higher budgets.** When comparing with DPM Solver (applied to VP-SDE, which is a non-augmented diffusion model), we find that exponential integrator-based methods converge faster but to a sub-optimal FID. In contrast, our proposed sampler (though applied to PSLD) eventually converges to better overall FID scores for slightly larger budgets (FID=2.11 for our sampler vs 2.59 for DPM-Solver). Please refer to the top-left subfigure in Figure 5 in the main text, where we compare with DPM-Solver and other sampler baselines. Therefore, our method is better suited for applications with a slightly larger computing budget at inference.

**Can the proposed samplers be combined with PC?**


Yes, our samplers can be extended to incorporate PC-based sampling as well since special cases like DPM-Solver and DEIS can automatically incorporate correction steps.

Reviewer comment (Experiments):
Improvements seem not to be good enough, especially considering speed. Large datasets and models are also not considered.

Author response:
Response: We address this comment in two parts:

**Improvements seem not to be good enough, especially considering speed**

We respectfully disagree and would like to re-highlight some important empirical contributions:

In terms of absolute comparisons with state-of-the-art samplers proposed in this work, for sampling budgets between 50 and 100 steps, we outperform existing samplers like DEIS and DPM-Solver across all datasets considered in this work. We find that our proposed sampler for PSLD achieves a FID score of 2.11 in 100 steps (compared to 2.57 for the best-performing baseline DEIS) for CIFAR-10 (See Table 1 and Fig. 5 in the main text). We observe similar results for other datasets like CelebA and AFHQv2.
For Stochastic Samplers proposed in this work, we again outperform state-of-the-art solvers like EDM [6], where our best stochastic sampler achieves a FID score of 2.36 in 100 function evaluations (as compared to 2.63 for the baseline).

In terms of generalizing existing samplers like DPM-Solver, DEIS, etc., and applying them to the PSLD diffusion, our proposed sampler (CSPS-D) outperforms these methods by a large margin, as illustrated in Table 1 (below in the response).

**Large datasets and models are also not considered**

Since we demonstrate the proposed samplers in the context of PSLD (a recently proposed state-of-the-art diffusion model), due to the absence of pre-trained models for this type of diffusion model, we would need to re-train PSLD models on large datasets like ImageNet. Training large-scale diffusion models is typically computationally prohibitive, and thus in this work, we only consider small-scale datasets for which training fits within our compute budget or datasets for which pre-trained PSLD checkpoints are available. For instance, for training on ImageNet at 64x64 resolution, the authors in [3] train the diffusion model for two weeks on a setup with 32 GPUs which is computationally expensive. Moreover, we believe that the advantages of our sampling framework are already well-demonstrated on smaller-scale datasets considered in this work.

Minor Comments: We thank the reviewer for pointing out these issues, which we will fix in our final revision.

**Questions:**

>how the dpm-solver will formulate under your settings and when will your method outperform it?

We already provide a response to this question in point 5 of our response above.

**Additional Results**
...
CSPS-D (Our proposed deterministic sampler in this work) outperforms DEIS by a large margin for all sampling budgets.

Reviewer comment (Experiments):
How the dpm-solver will formulate under your settings and when will your method outperform it?

Author response:
Response: We address this question in multiple parts.


**DPM-Solver as a particular case of Conjugate Integrators**


We thank the reviewer for their intuitive insight. During the response period, we established a theoretical connection of Conjugate Integrators with DPM-Solver [1]. More specifically, we find that the DPM Solver sampler for the position space-only diffusion models considered in [1] is a particular case of conjugate integrators with the choice of matrix $B_t = 0$. We have updated Proposition 2 in the main text with proof in Appendix B.3 to reflect the same. 


**When does our method outperform DPM-Solver?**


Our method can outperform DPM-Solver (or, more broadly, other exponential integrator-based methods like DEIS) in the following use cases.

1. **Exponential-based integrators do not generalize automatically to diffusion models in augmented spaces:** Firstly, with the perspective presented in DPM Solver [1] and other exponential-based integrators like DEIS [5], it is not apparent how to generalize these samplers to a broader class of diffusion models i.e. augmented diffusion models (where diffusion is performed in the data space concatenated with some auxiliary variables like in CLD [2] or PSLD [3]) as considered in this work. Theoretically, one of our main contributions is a framework that generalizes ideas in fast diffusion sampling like DDIM [4], DPM-Solver [1], and DEIS [5] under a single unified framework (see Propositions 1,2 in the main text). The benefit of such a framework is that it allows the construction of analogous samplers for different types of diffusion models like CLD and PSLD (which could be helpful in different contexts) and not just position space-only diffusion models (like Stable Diffusion). Moreover, a principled framework like ours allows us to extend beyond exponential integrator-based samplers. The resulting samplers can generalize better to other types of diffusion models. 

    To illustrate our point, empirically, we compare the FID scores of DEIS (generalized to PSLD by setting $B_t=0$) with “\lambda-DEIS” (obtained by setting $B_t=\lambda 1$) applied to the PSLD reverse diffusion in Table 1 (in this response) for the CIFAR-10 dataset. Our extension of DEIS i.e. $\lambda$-DEIS, outperforms the former by a large margin. We expect similar results for DPM-Solver since DEIS and DPM-Solver are exponential integrator-based methods and exhibit similar empirical performance in practice [5]. These results demonstrate that the exponential integrator-based framework does not generalize to novel diffusions automatically, therefore justifying the need for a broader framework as proposed in this work._

2. **DPM Solver saturates in sample quality for higher budgets.** When comparing with DPM Solver (applied to VP-SDE, which is a non-augmented diffusion model), we find that exponential integrator-based methods converge faster but to a sub-optimal FID. In contrast, our proposed sampler (though applied to PSLD) eventually converges to better overall FID scores for slightly larger budgets (FID=2.11 for our sampler vs 2.59 for DPM-Solver). Please refer to the top-left subfigure in Figure 5 in the main text, where we compare with DPM-Solver and other sampler baselines. Therefore, our method is better suited for applications with a slightly larger computing budget at inference.

**Can the proposed samplers be combined with PC?**


Yes, our samplers can be extended to incorporate PC-based sampling as well since special cases like DPM-Solver and DEIS can automatically incorporate correction steps.