Paper: Conditional Variational Diffusion Models (ICLR-2024)

Reviewer comment (Novelty):
The paper could do a clearer job differentiating itself from VDM.

Author response:
Thanks for pointing this out and for suggesting the use of a bullet list. We have answered this specific concern for other reviews too, but we also include an answer here to keep reviews and relevant comments organized.

We agree with your suggestion. We have included a bullet list in the Introduction (Section 1) explicitly stating the contributions of our work. Also, in Related Work (Section 2), we have included a summary of Variational Diffusion Models (VDMs, Kingma et al. 2023) and explained some of the differences with our method. Moreover, throughout the paper, we have extended the discussions about our technical contributions and their importance for the success of the method. See especially Sections 3.1, 3.2 and 3.4. We also include here a relevant summary.

VDMs (Kingma et al. 2023) constitute the main framework described in the literature for learning the variance schedule.  VDMs formulate a diffusion process for unconditioned distribution sampling, where the latent variables are indexed in continuous time, and the schedule is a learnable function that must satisfy a few minimal conditions. The parameters of the schedule are defined as a monotonic network, and the model is trained by minimizing a weighted version of the noise prediction loss. Their work also uses Fourier features to improve the prediction of high-frequency details. We introduce the Conditional Variational Diffusion Model (CVDM), which includes important generalizations and novelties with respect to VDMs:

- Similarly to Kingma et al. (2023), we learn the schedule as part of the training. However, we extend their approach to the conditioned case. This turns out to be a highly non-trivial task, as detailed below. We also allow for learning a different schedule for each element in the output (e.g., a pixel-wise schedule for images). These extensions require several technical novelties, including a separation-of-variables approach when defining the schedule (details in Section 3.2).
- We prove that the rate of convergence of the discrete-time diffusion loss to the continuous-time case depends strongly on the schedule functions. Kingma et al. (2023) show that diffusion models are invariant to the schedule selection in the continuous case. Nevertheless, any discretization used for model sampling will introduce the dependency shown in Appendix E.2 on the schedule's derivatives. Our finding motivates the introduction of a novel regularization term that proves to be critical for the performance of our method (details in Section 3.4).
- We replace the architecture in Kingma et al. (2023) with two networks, one required to be positive (for the conditioning variable) and one monotonic-convolutional network (for the time variable). This allows us to test our model with inputs of different resolutions without retraining. Moreover, by incorporating some schedule properties in the loss function, our method does not need the post-processing of the schedule that Kingma et al. 2023 uses, nor the preprocessing of the input. This results in a cleaner and more straightforward method.

Reviewer comment (Presentation):
A bullet-pointed list of contributions would have been appreciated.

Author response:
Thanks for pointing this out and for suggesting the use of a bullet list. We have answered this specific concern for other reviews too, but we also include an answer here to keep reviews and relevant comments organized.

We agree with your suggestion. We have included a bullet list in the Introduction (Section 1) explicitly stating the contributions of our work. Also, in Related Work (Section 2), we have included a summary of Variational Diffusion Models (VDMs, Kingma et al. 2023) and explained some of the differences with our method. Moreover, throughout the paper, we have extended the discussions about our technical contributions and their importance for the success of the method. See especially Sections 3.1, 3.2 and 3.4. We also include here a relevant summary.

VDMs (Kingma et al. 2023) constitute the main framework described in the literature for learning the variance schedule.  VDMs formulate a diffusion process for unconditioned distribution sampling, where the latent variables are indexed in continuous time, and the schedule is a learnable function that must satisfy a few minimal conditions. The parameters of the schedule are defined as a monotonic network, and the model is trained by minimizing a weighted version of the noise prediction loss. Their work also uses Fourier features to improve the prediction of high-frequency details. We introduce the Conditional Variational Diffusion Model (CVDM), which includes important generalizations and novelties with respect to VDMs:

- Similarly to Kingma et al. (2023), we learn the schedule as part of the training. However, we extend their approach to the conditioned case. This turns out to be a highly non-trivial task, as detailed below. We also allow for learning a different schedule for each element in the output (e.g., a pixel-wise schedule for images). These extensions require several technical novelties, including a separation-of-variables approach when defining the schedule (details in Section 3.2).
- We prove that the rate of convergence of the discrete-time diffusion loss to the continuous-time case depends strongly on the schedule functions. Kingma et al. (2023) show that diffusion models are invariant to the schedule selection in the continuous case. Nevertheless, any discretization used for model sampling will introduce the dependency shown in Appendix E.2 on the schedule's derivatives. Our finding motivates the introduction of a novel regularization term that proves to be critical for the performance of our method (details in Section 3.4).
- We replace the architecture in Kingma et al. (2023) with two networks, one required to be positive (for the conditioning variable) and one monotonic-convolutional network (for the time variable). This allows us to test our model with inputs of different resolutions without retraining. Moreover, by incorporating some schedule properties in the loss function, our method does not need the post-processing of the schedule that Kingma et al. 2023 uses, nor the preprocessing of the input. This results in a cleaner and more straightforward method.

Reviewer comment (Writing):
The typical convention in imaging inverse problems is y=A(x), rather than x=A(y).

Author response:
Thanks for pointing this out. We tried to present our method in the context of general inverse problems, so we used the $A(\mathbf{y}) = \mathbf{x}$ notation throughout. It is true that all the applications presented involve images, so the notation you suggest could have been used. However, we think it is not worth it to change it now. It could end up being more confusing for the reviewers, who already read the paper with the original notation.

Reviewer comment (Evaluation):
One of the main motivations of the paper is the ability to report uncertainty in order to point out artifacts in the solutions. In Figure 2, the authors show error images between the reconstructions and the ground truth. Do the uncertainty estimates correspond well to the regions of the image with more error?

Author response:
Thanks for the question. It is indeed relevant to explore the connection between uncertainty and reconstruction error. As a general matter, the answer to your question is that, yes, regions with higher uncertainty exhibit larger reconstruction errors. In the new Appendix M, you can find extended material about uncertainty. We discuss the relationship between uncertainty estimates and the schedule and offer a clear illustration of the correlation between uncertainty and reconstruction error. We have also improved the discussion about uncertainty in Section 5. In summary, even though uncertainty estimation is not the primary aspect of the paper, we agree it is important, so we have extended and improved the parts related to this topic, including a discussion of the point you raise here.

Reviewer comment (Experiments):
The performance is not significantly better than other methods for the BioSR super-resolution dataset. If one of the major goals is to demonstrate the flexibility and robustness of this approach, it would be nice to see comparisons against more conditional generative model datasets, especially those in the CDDPM paper (though there is a focus on inverse problems).

Author response:
Thanks for the suggestion. We agree that the method's robustness is emphasized in the paper as a feature of CVDM, and more experiments for conditioned sampling would further support this claim. To better showcase the flexibility of our method, we have conducted additional experiments. In particular, we have used our method for an image super-resolution task using ImageNet 1K (Deng et al. 2009). We compare our results against two diffusion-based methods, SR3 (Saharia et al. 2021) and DDRM (Kawar et al. 2022), using two different metrics (SSIM and PSNR). Using a learned schedule (and no fine-tuning), our method yields comparable results to these two methods, providing further evidence of the flexibility and robustness of CVDM.

Reviewer comment (Experiments):
When applied to super-resolution, this method does not require information about the PSF. This means that the model is learning the PSF implicitly from the data, but also means that the model would need to be retrained in order to get accurate results on data from a new PSF, which might be impractical if too much data is required for training. Since the flexibility and robustness of this method is highlighted, it would be nice to see some experiments or discussion addressing this, e.g. comparison against other blind deconvolution methods on datasets with different PSFs.

Author response:
The BioSR dataset for super-resolution microscopy is actually a good benchmark in this regard. It contains images acquired under varied conditions (i.e., different hardware, wavelength, etc.), and each variation corresponds to a different point spread function (PSF). The aim of the dataset is to help in training models that can be generalized to any fluorescence microscopy system. Therefore, since the model was trained on a wide array of PSFs, certain generalization capabilities are expected. In other words, we would expect the model to perform well in the general domain of fluorescence microscopy. The results obtained in our experiments show that the model was able to learn these different PSFs simultaneously. However, if the problem settings change too much with respect to the training dataset, there is in fact a need for retraining. Even though this is not something desirable, the lack of hyperparameter fine-tuning means that the method can be readily trained on new data, which shows the flexibility of our schedule-learning approach.

Reviewer comment (Experiments):
The schedules demonstrated for structured versus background regions appear to be linear with different slopes for the different regions. Would other/baseline methods perform better using an estimated linear pixelwise schedule with a higher slope on "structured" regions?

Author response:
Thanks for mentioning this idea. It is indeed an interesting concept. That type of implementation, if done carefully, would probably improve the performance of a method like CDDPM. However, there would still be some limitations. First, we find cases where the schedule is nonlinear, which suggests that linear schedules may not be a good model for all problems or all instances. Second, and more importantly, there would be a need for fine-tuning the slopes of the linear schedules and the exact boundary of the "structured" regions. Since our model is capable of learning different schedules by itself, we think it is better to avoid that type of work and adopt a learned-schedule approach.

Reviewer comment (Experiments):
The original VDM paper showed that a significant amount of the performance was due to the use of Fourier features. It would be helpful to know whether anything like that is the case here.

Author response:
Thanks for pointing this out. Conceivably, the use of Fourier features could help with performance. However, one of the aims of the paper is to simplify the training process of conditioned diffusion models as much as possible, so we deliberately omit this type of preprocessing. In that sense, we are more interested in developing a robust, flexible, and easy-to-use framework to learn the schedule in conditioned problems than obtaining the best possible results for any specific task. In this light, getting comparable-to-better results to the state of the art in several tasks, with a general method that needs no fine-tuning, is strong evidence for our hypothesis that setting the schedule as a hyperparameter is an unnecessary effort.

Reviewer comment (Novelty):
It would be helpful to have an explicit list or section summarizing the differences/extensions from VDM.

Author response:
Thanks for the advice. We have addressed this suggestion in several ways. In Section 2 (Related Work), we have included a summary of Variational Diffusion Models (VDMs, Kingma et al. (2023)) to provide more background before starting the Methods section. Also, in Section 2, we have explained some of the differences between VDMs and our method, and we have included a bullet list in Section 1 (Introduction) to state the specific contributions of our work more clearly. Moreover, throughout the paper, we have extended the discussions about our technical contributions and their impact on the performance of the method. See especially Sections 3.1, 3.2, 3.4 and 4.3.

Reviewer comment (Evaluation):
Does the pattern of schedules for structured versus background regions shown in Figure 2 hold true across the whole sample and for all samples?

Author response:
The pattern does hold in general. For instance, for the super-resolution microscopy problem, we observe higher values of $\beta$ for structure than background pixels. However, the exact shapes and slopes can vary between pixels according to the difficulty of the local inverse problem.

Reviewer comment (Evaluation):
Are there significant differences in the inference time of this model versus the methods being compared against?

Author response:
Inference time is the same for our method and CDDPM.  We have clarified this in Sections 4.1 and 4.2, adding a corresponding text. In that sense, this method could probably benefit from consistency models to reduce the number of function evaluations. DFCAN is faster as it is not a diffusion model but rather a regression model.

Reviewer comment (Novelty):
The biggest weakness is the paper's technical novelty. The proposed approach seems like a straightforward conditional extension to the of variational diffusion models. In addition, the paradigm of turning an unconditional model to a conditional version has been largely explored and established, e.g. [1]. I can see the decomposition of the learnable variance schedule, and the regularized learning are novel. Besides that, can the authors comment on the technical non-triviality of this extension?

Author response:
We can comment about the technical non-triviality of our contributions in more detail. In general, our hypothesis is that the schedule can be learned, eliminating the need for fine-tuning the schedule while exhibiting better results. Kingma et al. (2023) explored this idea in [6], but only for the unconditioned case. We are interested in the case of conditioned sampling, which opens up many more opportunities for application, but a straightforward application of Saharia et al. (2021) does not work at all in this case (the schedule does not converge to anything useful or meaningful).

So, in fact, learning the schedule for conditioned sampling is a non-trivial challenge and is far from a direct combination of [1] and [6]. As you mention, two of the key ideas that address this problem are the decomposition/factorization of the variance schedule (Section 3.2) and the regularization strategy (Section 3.4), both of which are essential:

- It is non-trivial how to properly condition the schedule on the data $\mathbf{x}$, because the schedule function needs to be monotonic in $t$, but not necessarily on $\mathbf{x}$. Our decomposition/factorization of the function $\beta$ is in our opinion a clever way of solving this problem and is absolutely key for our method. See comments in Sections 3.2 and 4.3.
- We prove that the rate of convergence of the discrete-time diffusion loss to the continuous-time case depends strongly on the schedule functions. Kingma et al. (2023) show that diffusion models are invariant under schedule selection in the continuous case. Nevertheless, any discretization used for model sampling will introduce the dependency shown in Appendix E.2 on the schedule's derivatives. Our finding motivates the introduction of a novel regularization term that proves to be critical for the performance of our method (details in Section 3.4 and 4.3, and Appendix K).

Regarding implementation details, we replace the architecture in Kingma et al. (2023) with two networks, one required to be positive (for the conditioning variable) and one monotonic-convolutional network (for the time variable). This allows us to test our model with inputs of different resolutions without retraining. Moreover, by incorporating schedule properties in the loss function, our method does not need the post-processing of the schedule that Kingma et al. 2023 uses, nor the preprocessing of the input (addition of Fourier features). These design decisions are arguably less novel, but they result in a cleaner and more straightforward method compared to Kingma et al. (2023).

Reviewer comment (Experiments):
Another weakness is lack of ablation study. For example, how does the method perform without using the regularized learning approach in Section 3.4?

Author response:
Thanks for this suggestion. We agree that an ablation study would help clarify our technical contributions' impact on the performance. We have answered something similar for a different reviewer, but we also respond here in order to keep the reviews and specific comments organized.

We have clarified the role played by our specific contributions in the new version of the paper (see Sections 3.2, 3.4 and 4.3). We have also performed an ablation study. Some important points:

1. The regularization term described in Section 3.4, which you mention as an example, is critical to keep the schedule from converging to an uninformative solution that trivially minimizes the diffusion loss. In this regard, this technical novelty is essential for the method. See details in Section 3.4. We have also ran experiments without using this regularization term, with results described in Section 4.3.
2. Our factorization of the function $\beta$ is also crucial for our method. Recall that the schedule function $\gamma$ needs to be monotonic in $t$, but not necessarily on $\mathbf{x}$, so it is non-trivial how to properly condition the schedule on the data $\mathbf{x}$. See details in Section 3.2. While trying these changes out, the method simply did not converge.
3. We performed an ablation study on the effect of having a global unique schedule instead of a pixel-wise dedicated schedule. In our ablation study (Appendix K), we compare global vs. pixel-wise schedules (CVDM-simple). The latter outperforms CDDPM but does not match CVDM's results, highlighting the value of pixel-wise schedules in addressing unique image challenges. We also include an ablation study on the impact of our regularization strategy, which shows the importance of regularized learning for CVDM.

Reviewer comment (Writing):
There is room for improvement in terms of clarity, especially in the experiment section. See my detailed comments in the below Questions section.

Author response:
Thanks a lot for going into so much detail regarding the experimental section (and the rest of the paper). We include an answer to all your questions below.

Reviewer comment (Theory):
The condition \( A(x) = y \) is brought up in the introduction, but is not used in the method section. Does this condition specification matter?

Author response:
This condition specification does not matter in Methods section (our method is general and agnostic about the specific mapping $A$). It is only mentioned in the Introduction to further clarify what is meant by an Inverse Problem, and to introduce the relation between $\mathbf{x}$ and $\mathbf{y}$, which is used throughout the paper.

Reviewer comment (Novelty):
Missing a background of Variational Diffusion Models (VDM) before starting the method section. And I would appreciate a more straightforward discussion on how this version extends beyond and distinguishes from VDM.

Author response:
Thanks for the suggestion. We have answered similar concerns for other reviewers, too. We also include an answer here. We have addressed your suggestion in several ways. In Related Work (Section 2), we have included a summary of Variational Diffusion Models (VDMs, [6]), to provide more background before starting the Methods section, as you suggest. In Section 2, we have also explained some of the differences with our method, and we have included a bullet list in the Introduction (Section 1) to explain the contributions of our work more clearly. Moreover, throughout the paper, we have extended the discussions about our technical contributions and their importance for the success of the method. See especially Sections 3.1, 3.2, 3.4 and 4.3. We also include here a relevant summary.
VDMs (Kingma et al. 2023) constitute the main framework described in the literature for learning the variance schedule. VDMs formulate a diffusion process for unconditioned distribution sampling, where the latent variables are indexed in continuous time, and the schedule is a learnable function that must satisfy a few minimal conditions. The parameters of the schedule are defined as a monotonic network, and the model is trained by minimizing a weighted version of the noise prediction loss. Their work also uses Fourier features to improve the prediction of high-frequency details. We introduce the Conditional Variational Diffusion Model (CVDM), which includes important generalizations and novelties with respect to VDMs, as described in detail above (see our answers to Weaknesses).

Reviewer comment (Writing):
The notation \( SNR'' \) on page 5 is not clear enough; it might be worth writing \( SNR''(t,x) \) when it is first introduced.

Author response:
We have followed the suggestion of using SNR$''(t,\mathbf{x})$ when introducing the term in Section 3.4. Also, we have slightly rewritten that paragraph to further clarify the notation.

Reviewer comment (Experiments):
Missing (the reference to) the details on the base model architecture and other training details for the competing methods. This question is important to understand whether the empirical comparison is a fair one.

Author response:
Thanks for this insight. The architecture details for our method are provided in Appendix G. Since one of our aims is to prove that learning a schedule can improve the results over fine-tuning it, we base both our architecture and that of CDDPM on [1], and reduce the number of parameters to avoid overfitting. To fine-tune CDDPM, we choose a linear schedule as in [1] and perform a grid search to find the best schedule. In that sense, both diffusion models share the same base architecture, with the addition of the schedule network for our method. For non-diffusion models, DFCAN is implemented as in [5] (referenced in Section 4.1) and US-TIE is implemented as in [4] (referenced in Section 4.2).

Reviewer comment (Novelty):
Missing a description of competing methods, i.e. DFCAN, CDDPM, and a discussion on how the proposed method is different from them.

Author response:
Thanks for pointing this out. We have added a brief description of these methods in Section 4. Specifically, DFCAN is a regression-based method that introduces the Fourier attention channel mechanism, currently state of the art in super-resolution microscopy. We implemented DFCAN as in [5], and CDDPM is the conditioned DDPM implemented and trained as described in [1].

Reviewer comment (Experiments):
For the CDDPM, did you fine-tune the hyperparameters, namely the variance schedule?

Author response:
Yes, we fine-tuned them. We changed our text to emphasize better that the CDDPM schedule was fine-tuned (see Sections 4.1 and 4.2).

Reviewer comment (Experiments):
Limited methods used for comparison. There are definitely other qualified competing methods in the diffusion space, e.g. [2], [3], [4]

Author response:
Both DFCAN and US-TIE were chosen for being the state of the art for super-resolution microscopy and QPI, respectively. Regarding diffusion models, there are certainly a number of methods available. The main reason we chose CDDPM is that we wanted to test the effect of learning the schedule, as opposed to fine-tuning it, while controlling for many of the implementation details of the methods. In that sense, CDDPM is sufficiently similar to CVDM to allow us to assess better the impact of learning the schedule.

    To a considerable extent, we have verified our hypothesis that treating the schedule as a hyperparameter is unnecessary because it takes effort and may lead to suboptimal performance. This is of special importance to the users of the method since reducing the number of hyperparameters provides a way of applying this problem to a larger array of problems without the overhead of fine-tuning for the specific case. We find that learning the schedule in this way is straightforward and leads to comparable-to-better performance. Aditionally, we also wanted a way to quantify uncertainty for these methods. 

    To provide a more comprehensive understanding of our method's performance, we have conducted further experiments for an image super-resolution task using ImageNet. The results are described in Section 4.3 and Appendix L. For this task, we compare against [1] and one of the methods referenced in your question (Denoising Diffusion Restoration Models, or DDRMs). We get comparable results to [1] and slighlty improve on DDRMs. In any case, we want to clarify that we are not claiming a superiority in results for all tasks and compared to all other diffusion-based methods. To our mind, one of the great strengths of CVDM is that it provides a method for learning the schedule that is robust, provides a way to quantify the uncertainty of the reconstruction, requires no fine-tuning, yields good results (comparable or superior to state-of-the-art methods for the problems we tried), and which can be used to enhance other architectures.

Reviewer comment (Presentation):
Table 1: Why is there only one underlined result?

Author response:
Thanks for noticing this, it was an error on our part. We had missed a few results that needed to be underlined, we have now fixed Table 1.

Reviewer comment (Presentation):
Table 1: Why do some rows not have underlines and bolded results?

Author response:
For those rows, there are no statistically significant differences between methods.

Reviewer comment (Evaluation):
Table 1: How do you define statistically significant? Do you repeat the experiments multiple times and compute the average results with standard errors? If so, please describe this detail in the table or the text.

Author response:
The idea is the following. Take a testing dataset with N samples. For each method, we calculate the mean error over the dataset using 1 reconstruction for each input. Then, we compare the sample error means between two different methods, by performing a hypothesis test to determine if the two population means are statistically significant. We have included a clarification in the caption of Table 1 to explain this.

Reviewer comment (Presentation):
Table 2a: I would suggest adding the "synthetic" description to the HCOCO dataset in the caption.

Author response:
We have followed your suggestion and added the "synthetic" description for the HCOCO dataset.

Reviewer comment (Writing):
Section 4.2.2: What is the "US-TIE method"?

Author response:
Thanks for this comment. We have clarified in the text (Section 4.2) that US-TIE corresponds to the method described in [4] for solving the transport intensity equation.

Reviewer comment (Evaluation):
Figure 3a: Where does the learned uncertainty reflect in the figure, or is it not? I was thinking of a figure like Figure 4a in [1]. And how do you draw the conclusion on "We briefly point out that our uncertainty estimation is consistent with the values of β as described above" if there is no comparison between the uncertainty estimation and the values of \( \beta \)?

Author response:
Thanks for pointing this out. We have added an illustrative Figure in Appendix M. The Figure shows several reconstructions for the same input, and it illustrates how different reconstructions can vary in the regions with higher sample variance. It also shows the relationship between sample variance and $\beta$, and how regions of higher sample variance generally exhibit higher reconstruction error.

The uncertainty (i.e., higher variance) in the reconstruction is theoretically tied to $\beta$. As described in [3], the reverse diffusion process can be characterized by a stochastic differential equation with a diffusion coefficient of $\sqrt{\beta(t,\mathbf{x})}$ (in the variance-preserving case). Hence, higher values of $\beta$ introduce more diffusion into the reverse process over time, leading to more varied reconstructions. We have added this explanation at the end of Section 5, and in the new Figure 16 (Appendix M) it is also possible to appreciate that $\beta$ is correlated to the sample variance and the reconstruction error.

Reviewer comment (Evaluation):
How can the uncertainty quantification around the reconstruction be useful?

Author response:
In applications such as microscopy, relevant information may sometimes be contained in single pixels (for instance, cell interaction). Therefore, a single hallucination in the reconstruction method can lead to totally misleading conclusions. The same is the case for MRI, where a few pixels can determine the difference between the presence or absence of a tumor. In fact, this is one of the challenges for the adoption of accelerated MRI in clinical environments, for instance, in [2]. That's why an estimation of certainty for different parts of the reconstruction can be a valuable tool.

Reviewer comment (Writing):
Appendix A, second paragraph, first sentence, should be "Ay = x" instead of "Ax = y"?

Author response:
Thanks for pointing out this error. We have fixed it in the text.

Reviewer comment (Novelty):
While this yields good results in practice, I am not sure if this is enough novelty for a conference like ICLR.

Author response:
The differences with [1] go far beyond the replacement of the SNR term. Their framework is for unconditioned sampling only, which we extend in two ways. First, we allow for conditioned sampling, which opens up many more opportunities for application, including the case of inverse problems with which we motivate our work in Section 1. Second, we allow for learning a pixel-wise variance schedule, which further improves performance (see Ablations in Section 4.3).

Moreover, these extensions are highly non-trivial. Learning the schedule for conditioned sampling is a difficult challenge and is far from a direct application of [1]. There are at least two new ideas we use to make it work: the decomposition/factorization of the variance schedule (Section 3.2) and the regularization strategy (Section 3.4), both of which are essential:

- It is non-trivial how to properly condition the schedule on the data $\mathbf{x}$, because the schedule function needs to be monotonic in $t$, but not necessarily on $\mathbf{x}$. Our decomposition/factorization of the function $\beta$ is, in our opinion, a clever way of solving this problem and is absolutely key for our method. See comments in Sections 3.2 and 4.3.
- We prove that the rate of convergence of the discrete-time diffusion loss to the continuous-time case depends strongly on the schedule functions. [1] show that diffusion models are invariant under schedule selection in the continuous case. Nevertheless, any discretization used for model sampling will introduce the dependency shown in Appendix E.2 on the schedule's derivatives. Our finding motivates the introduction of a novel regularization term that proves to be critical for the performance of our method (details in Section 3.4 and 4.3, and Appendix K).

Regarding implementation details, we replace the architecture in [1] with two networks, one required to be positive (for the conditioning variable) and one monotonic-convolutional network (for the time variable). This allows us to test our model with inputs of different resolutions without retraining. Moreover, by incorporating schedule properties in the loss function, our method does not need the post-processing of the schedule that [1] uses, nor the preprocessing of the input (addition of Fourier features). These two last design decisions are arguably less novel, but they result in a cleaner and more straightforward method compared to [1].

So we think it is clear that our work has substantial differences with [1], and there is a good deal of technical novelty. The "replacement of the unstable SNR term" that you mention is indeed a part of our work, but it is not the most novel or important part. The revised version of the paper makes this more clear. In Section 2 (Related Work), we have included a summary of Variational Diffusion Models (VDMs) [1] to provide more background before starting the Methods section. Also, in Section 2, we have explained some of the differences between VDMs and our method, and we have included a bullet list in Section 1 (Introduction) to state the specific contributions of our work more clearly. Moreover, throughout the paper, we have extended the discussions about our technical contributions and their impact on the performance of the method. See especially Sections 3.1, 3.2, 3.4 and 4.3.