Paper: Rethinking the Noise Schedule of Diffusion-Based Generative Models (ICLR-2024)

Reviewer comment (Evaluation):
The idea of data-driven ODE noise schedule is interesting, and the authors show that the proposed method is able to improve the sampling quality. Is the newly proposed sampling strategy highly related to the WSNR metric, can we adjust the sampling schedule based on other metrics such as PSNR? The improvement is mainly due to the data-driven framework or the newly proposed metric.

Author response:
We are very grateful to the reviewer for raising such insightful questions, which have sparked our thinking about how to use WSNR to uniformly design both training and sampling noise schedules.

Our proposed WSNR metric can be used to uniformly represent noise levels across different scales, meaning that a data-driven sampling schedule can be designed and expressed through WSNR. We found that for the FFHQ-64 and FFHQ-256 datasets, the ODE integration intervals should be [0.002, 98.96] and [0.008, 394.62], respectively. When represented in terms of WSNR, these intervals become [117166875, 0.0478] and [117166875, 0.0481], showing that FFHQ 64 and 256 are similar in terms of the WSNR metric.

We have not explored replacing WSNR with PSNR, because WSNR is a promising metric. We will leave the exploration regarding PSNR for future work.

Reviewer comment (Novelty):
The noise scheduling is discussed in previous works from different perspectives. The concurrent work [1] also discusses the noise schedule from the spectra view. The authors are encouraged to discuss the differences. [1] Relay diffusion: Unifying diffusion process across resolutions for image synthesis

Author response:
Q1: The concurrent work [1] also discusses the noise schedule from the spectra view. The authors are encouraged to discuss the differences. 

A1:   Thank you for bringing concurrent work [1] to our attention. We acknowledge that [1] also explores the noise level from a spectral perspective in RGB space. 

**However, our work and [1] have several distinct differences**:
1. **Different Approaches**: Our work is dedicated to designing a robust training noise schedule applicable across different resolutions and latent spaces. [1] mainly focuses on the relay connection within their proposed RDM, enabling a cascading sampling process from 64x64 to 256x256 in a single pass. It seems that they do not contribute to the training noise schedule. If my understanding of [1] is incorrect, please kindly correct me.
2. **Different Analysis Metrics**: We propose WSNR as a consistent quantization metric, ensuring consistent quantization results at the same noise level across different resolutions. In contrast, [1] uses the SNR quantization metric, which is not consistent in different resolutions.
3. **Different Analytical Perspectives**: We analyze the average power of images and Gaussian noise from a power spectrum perspective, highlighting that the average power of Gaussian noise is uniform across all frequency components. This conclusion is not stated in their work.

Additionally, it's worth noting that we have conducted theoretical analysis on the sampling noise schedule and proposed improvements.
Moreover, [1] is also a submission to ICRL 2024. We believe these are two independent pieces of research work, and this should not be considered a weakness or even a reason for rejection.



Q2: The noise schedule is discussed in previous works from different perspectives. 

A2:  We appreciate the reviewer's comment regarding the discussion of noise scheduling in prior works. 
Since our work is titled "Rethinking the Noise Schedule of Diffusion-Based Generative Models", we have acknowledged that some previous works have explored noise schedule design. However, our paper has **unique contributions**:
- We propose a novel WSNR metric to consistently quantify the noise level across multiple resolutions in **RGB space and latent space** for the forward process of diffusion models.
- We significantly improve the performance of diffusion models, via our novel WSNR-Equivalent training noise scheduler. Specifically, our method decreases the FID score by 3.6 on FFHQ-256x256 dataset, and achieves the same FID scores on ImageNet-256 and 512 for UViT-M (#Params: 287M)  as those of UViT-L (#Params: 287M).  
- We analyze that the **ODE sampling noise schedule** should be data-driven. Consequently, we propose estimating the integration interval based on the average data distance.
- We explore the relationship between the number of evaluation points and generation quality. Our findings lead us to develop a **NFE-guided sampling noise schedule**.
- Our sampling schedule refines the FID of pre-trained CIFAR-10 and FFHQ-64 models from 1.92 and 2.45 to 1.89 and 2.25, respectively, utilizing 35 network evaluations per image.


As the reviewer did not provide further explanation and references about the "previous works" and "different perspectives", we will discuss the recent representative research works [2,3,4,5,6], which proposed handcrafted training noise schedules to improve the performance of diffusion models. However, the proposed schedules are based on the SNR metric, which is neither consistent in different resolutions, nor consistent in latent space. 

That means their schedules **fail to** achieve desired performance consistently on various across multiple resolutions in RGB space and latent space under a single set of hyperparameters. In contrast, **ours can achieve superior results** consistently with just one set of hyperparameters.

Besides, our analysis and improvement on ODE sampling noise schedule, an aspect not addressed by the other methods. This unique focus not only demonstrates our method's innovative approach but also highlights its ability to fill a critical gap in the current research landscape. By delving into areas previously unexplored, our work contributes significantly more to the advancement of the field, offering novel insights and practical improvements that set a new standard for future research in this area.

Reviewer comment (Writing):
The motivation or justification for rethinking the noise schedule of diffusion-based generative models is not clearly explained in the introduction.

Author response:
We appreciate the reviewer’s feedback regarding the clarity of our paper’s motivation and the role of Fig 1. We acknowledge an error in our initial submission regarding the referencing of figures. This analysis is actually presented in Fig 2. We apologize for any confusion this may have caused and have corrected this reference in our revised manuscript. Fig 1 is intended to illustrate the training noise schedule and sampling noise schedule.

We have revised the introduction section to explain the motivation clearly.
Here, we summarize our motivation briefly:

1. As shown in Fig. 2, significant variation in noise levels across images of different resolutions when exposed to Gaussian noise with identical standard deviations, leads to a discrepancy in the noise levels focused on by manually designed training noise schedules in different RGB space resolutions and latent space. Motivated by this observation, we propose a novel metric WSNR to quantify the noise level consistently across multiple resolutions in RGB space and latent space. We further propose a novel training noise schedule based on the WSNR metric.

2. As illustrated in Fig. 4 and elaborately explained in Sec. 5, the diversity of the generated data is jointly influenced by the initial Gaussian distribution at the start point and the Euclidean distance from the data points in the dataset. Therefore, we propose a data-driven sampling noise schedule to determine the integration interval of the diffusion ODE.

3. As demonstrated in Tab. 4, an increased number of evaluation points (the unique time steps at which model makes predictions) leads to better results when the step size is relatively large. Conversely, with smaller step sizes, the advantage of adding more evaluation points becomes less significant. Drawing on these findings, we propose a strategy for dynamically selecting numerical methods according to computational constraints, aiming to optimize generation quality.

Reviewer comment (Presentation):
I could not understand how Figure 1 is related to the main motivation of the paper. Figure 2 was more intuitive.

Author response:
We appreciate the reviewer’s feedback regarding the clarity of our paper’s motivation and the role of Fig 1. We acknowledge an error in our initial submission regarding the referencing of figures. This analysis is actually presented in Fig 2. We apologize for any confusion this may have caused and have corrected this reference in our revised manuscript. Fig 1 is intended to illustrate the training noise schedule and sampling noise schedule.

We have revised the introduction section to explain the motivation clearly.
Here, we summarize our motivation briefly:

1. As shown in Fig. 2, significant variation in noise levels across images of different resolutions when exposed to Gaussian noise with identical standard deviations, leads to a discrepancy in the noise levels focused on by manually designed training noise schedules in different RGB space resolutions and latent space. Motivated by this observation, we propose a novel metric WSNR to quantify the noise level consistently across multiple resolutions in RGB space and latent space. We further propose a novel training noise schedule based on the WSNR metric.

2. As illustrated in Fig. 4 and elaborately explained in Sec. 5, the diversity of the generated data is jointly influenced by the initial Gaussian distribution at the start point and the Euclidean distance from the data points in the dataset. Therefore, we propose a data-driven sampling noise schedule to determine the integration interval of the diffusion ODE.

3. As demonstrated in Tab. 4, an increased number of evaluation points (the unique time steps at which model makes predictions) leads to better results when the step size is relatively large. Conversely, with smaller step sizes, the advantage of adding more evaluation points becomes less significant. Drawing on these findings, we propose a strategy for dynamically selecting numerical methods according to computational constraints, aiming to optimize generation quality.

Reviewer comment (Theory):
From Eq. (1), it seems to implicitly assume the variance exploding (VE) case, but it is not clear what happens in the variance preserving (VP) case.

Author response:
Our proposed methods are applicable to VP case. As shown in Tab.3, our data-driven sampling noise schedule works when it is applied to DPM-Solver, which is based on VP ODE.

Besides, we conduct experiments on VP cases to evaluate our WSNR-Equivalent training noise schedule. The FID score on FFHQ dataset in the following table show that our method further improves performance on VP cases.

| FFHQ | 64 | 128 | 256 |
| --- | --- | --- | --- |
| $p(\text{WSNR})$ | 3.80 | 6.23 | 7.99 |
| $p(\sigma)$ | 3.80 | 7.41 | 11.78 |


Actually, the denoiser D_theta (x) in Eq. 1 can be used to explain VP, VE and EDM cases. 
As shown in Eq. 7 in [1], we can write the $D_\theta(x; \sigma)$ in the following form:

$$D_\theta(x; \sigma) = c_{\text{skip}}(\sigma) x + c_{\text{out}}(\sigma) F_\theta(c_{\text{in}}(\sigma) x; c_{\text{noise}}(\sigma))$$
For VP, $c_{\text{skip}}(\sigma)=1, c_{\text{out}}(\sigma)=-\sigma, c_{\text{in}}(\sigma)=\frac{1}{\sqrt{\sigma^2 + 1}}, c_{\text{noise}}(\sigma) = (M-1)\sigma^{-1}(\sigma)$. 
This implies that our Eq.1 is compatible with the VP case and our sampling noise schedule can be evaluated with  the pre-trained VP model.


As for the training process, we follow the training loss in [1], setting the training target as:

$E_{\sigma, y, n} \left[ \overbrace{\lambda(\sigma) c_{\text{out}}(\sigma)^2}^{\text{effective weight}} \left\| \overbrace{F_\theta \left( c_{\text{in}}(\sigma) \cdot (y + n); c_{\text{noise}}(\sigma) \right)}^{\text{network output}} - \overbrace{\frac{1}{c_{\text{out}}(\sigma)} (y - c_{\text{skip}}(\sigma) \cdot (y + n))}^{\text{effective training target}} \right\|^2 \right]$

Therefore, we believe that our training process is compatible with interpreting the VP case.

[1] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in Neural Information Processing Systems 35 (2022).

Reviewer comment (Writing):
Overall, writing should be improved. In the current form, motivation is not clearly explained in the introduction, and it is not until Figure 2 in Section 4 that the motivation is understood.

Author response:
We appreciate the reviewer’s feedback regarding the clarity of our paper’s motivation and the role of Fig 1. We acknowledge an error in our initial submission regarding the referencing of figures. This analysis is actually presented in Fig 2. We apologize for any confusion this may have caused and have corrected this reference in our revised manuscript. Fig 1 is intended to illustrate the training noise schedule and sampling noise schedule.

We have revised the introduction section to explain the motivation clearly.
Here, we summarize our motivation briefly:

1. As shown in Fig. 2, significant variation in noise levels across images of different resolutions when exposed to Gaussian noise with identical standard deviations, leads to a discrepancy in the noise levels focused on by manually designed training noise schedules in different RGB space resolutions and latent space. Motivated by this observation, we propose a novel metric WSNR to quantify the noise level consistently across multiple resolutions in RGB space and latent space. We further propose a novel training noise schedule based on the WSNR metric.

2. As illustrated in Fig. 4 and elaborately explained in Sec. 5, the diversity of the generated data is jointly influenced by the initial Gaussian distribution at the start point and the Euclidean distance from the data points in the dataset. Therefore, we propose a data-driven sampling noise schedule to determine the integration interval of the diffusion ODE.

3. As demonstrated in Tab. 4, an increased number of evaluation points (the unique time steps at which model makes predictions) leads to better results when the step size is relatively large. Conversely, with smaller step sizes, the advantage of adding more evaluation points becomes less significant. Drawing on these findings, we propose a strategy for dynamically selecting numerical methods according to computational constraints, aiming to optimize generation quality.

Reviewer comment (Novelty):
The proposed WSNR-Equivalent training noise schedule and data-driven sampling noise schedule seem to be independent of each other, which weakns the focus of this paper.

Author response:
We respectively disagree with this viewpoint.  
Firstly, the training and sampling processes of diffusion models are strongly interrelated. Diffusion models are trained independently for each noise level. The inference stage can be seen as refining the generated images multiple times at various timesteps using the network. We believe that optimizing both the training and sampling stages simultaneously **highlights our contribution**.

Secondly, exploring the training and sampling processes of diffusion models within a single work is **not a weakness**. In the **NIPS 2022 Outstanding Paper** [1], the authors **similarly improved the training and sampling processes of diffusion models**, which was not considered a weakness by the reviewers, area chairs, or senior area chairs.

Finally, combining our methods will **further enhance the performance**.  We evaluate our trained model on FFHQ-256 using our proposed sampling method. The table shows that our sampling method consistently lowers the FID score of the trained model, indicating improved performance since a lower FID score is desirable.

|  | Heun (NFE=35) | Ours (NFE=35) |
| --- | --- | --- |
| p(WSNR) | 7.89 | **7.40** |
| p($\sigma$) | 11.49 | **11.02** |


[1] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in Neural Information Processing Systems 35 (2022): 26565-26577.

Reviewer comment (Experiments):
Experiments in Table 1 and Table 2 compare with only EDM training noise schedule. The authors are suggested to compare with more training noise schedules to further verify the effectiveness of training noise schedule.

Author response:
We appreciate your suggestion to include comparisons with more training noise schedules to further validate the effectiveness of our approach. 

Due to the limited time for rebuttal and our computational resources, we conduct experiments on VP cases to evaluate our WSNR-Equivalent training noise schedule. The results in the following table show that our method further improves performance on VP cases.

| FFHQ | 64 | 128 | 256 |
| --- | --- | --- | --- |
| p(WSNR) | 3.80 | **6.23** | **7.99** |
| p($\sigma$) | 3.80 | 7.41 | 11.78 |



We understand the importance of ensuring that our paper meets the high standards of ICLR and would greatly appreciate it if you could confirm whether the revisions made align with your expectations and satisfactorily resolve the issues previously highlighted.

Your further guidance and feedback will be important to us and we are committed to making any additional improvements as needed.

Thank you very much for your time and consideration. We look forward to your response.

Reviewer comment (Presentation):
As illustrated in Fig. 1, we observed substantial disparities in noise levels across images of varying resolutions under the same noise schedule. Figure 1 has nothing to do with resolutions.

Author response:
Thanks for pointing it out, we acknowledge an error in our initial submission regarding the referencing of figures. Specifically, the discussion of resolutions is visualized in Fig 2 instead of Fig 1. We apologize for any confusion this may have caused and have corrected this reference in our revised manuscript. We appreciate the reviewers bringing this to our attention and ensuring the accuracy of our work.

Reviewer comment (Writing):
P·,c(u,v) is the power of the frequency component at (u,v) within the c-th channel. Are u and v in the frequency domain?

Author response:
Yes, u and v are in the frequency domain. Because $P$ represents the power spectrum.
The power spectrum is defined as 
$$P(u,v) = |F(u,v)|^2 $$
where $F(u,v)$ is the 2D discrete Fourier transform (DFT). Please check the Eq. (4-67) and (4-89) in [2], which is a textbook widely adopted by university computer vision courses. 

[2] Gonzalez, Rafael C. Digital image processing 4th Edition.

Reviewer comment (Reproducibility):
P·,c(u,v) is the power of the frequency component at (u,v) within the c-th channel. What technique is used to convert the images into frequency domain?

Author response:
Yes, u and v are in the frequency domain. Because $P$ represents the power spectrum.
The power spectrum is defined as 
$$P(u,v) = |F(u,v)|^2 $$
where $F(u,v)$ is the 2D discrete Fourier transform (DFT). Please check the Eq. (4-67) and (4-89) in [2], which is a textbook widely adopted by university computer vision courses. 

[2] Gonzalez, Rafael C. Digital image processing 4th Edition.

Reviewer comment (Theory):
Given a finite dataset, an ideal solution for the denoiser D(xt) can be found as the weighted sum of all clean data in the dataset. This statement does not have support.

Author response:
We appreciate the reviewer's point regarding the lack of a formal proof for our assertion that an ideal solution for the denoiser can be represented as a weighted sum of all clean data in the dataset. 

Here, we provide proof to make it clear:
Given the standard deviation of Gaussian noise $\sigma$ and the clean data probability distribution $p_{\text{data}}(x) = \frac{1}{Y} \sum_{i=1}^{Y} \delta(x - y_i)$, the probability of the noisy data $p(x; \sigma)$ can be expressed as: 

$$
\begin{align}
p(x; \sigma) & = p_{\text{data}} * \mathcal{N}(0, \sigma^2 \mathbf{I}) \\
& = \int_{\mathbb{R}^d} p_{\text{data}}(x_0) \mathcal{N}(x; x_0, \sigma^2 \mathbf{I}) dx_0 \\
& = \int_{\mathbb{R}^d} \left[ \frac{1}{Y} \sum_{i=1}^{Y} \delta(x_0 - y_i) \right] \mathcal{N}(x; x_0, \sigma^2 \mathbf{I}) dx_0 \\
& = \frac{1}{Y} \sum_{i=1}^{Y} \int_{\mathbb{R}^d} \mathcal{N}(x; y_i, \sigma^2 \mathbf{I}) \delta(x_0 - y_i) dx_0 \\
& = \frac{1}{Y} \sum_{i=1}^{Y} \mathcal{N}(x; y_i, \sigma^2 \mathbf{I}) \\
\end{align}
$$


The loss of the Denoiser is:

$$
\mathcal{L}(D; \sigma) = E_{y \sim p_\text{data}} \mathbb{E}_{x \sim p(x, \sigma)}  \| D(x; \sigma) - y \|_2^2
$$

$$\mathcal{L}(D; \sigma) = \int_{\mathbb{R}^d} \frac{1}{Y} \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) \| D(x; \sigma) - y_i \|_2^2  dx$$

The solution is obtained by setting the gradient w.r.t $D(x; \sigma)$ to zero:

$$
0 = \nabla_{D(x;\sigma)} \mathcal{L}(D; x, \sigma) 
$$

$$
0 = \nabla_{D(x;\sigma)} \left[ \frac{1}{Y} \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) \| D(x; \sigma) - y_i \|_2^2 \right]
$$

$$
0 = \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) \nabla_{D(x;\sigma)} \left[ \| D(x; \sigma) - y_i \|_2^2 \right]
$$


$$
0  = \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) \left[ 2 D(x; \sigma) - 2 y_i \right] 
$$

$$
0 = \left[ \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) D(x; \sigma) \right] - \left[ \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) y_i \right] 
$$

$$
D(x; \sigma) = \frac{\sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) y_i}{\sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I})} 
$$

$$
D(x; \sigma) = \frac{\sum_{i=1}^Y \mathrm{exp}(\frac{||x-y_i||^2}{-2\sigma^2})  y_i}{\sum_{i=1}^Y \mathrm{exp}(\frac{||x-y_i||^2}{-2\sigma^2})}
$$

$$
D(x; \sigma) = \sum_{i=1}^Y \mathrm{softmax}(\frac{||x-y_i||^2}{-2\sigma^2}) y_i
$$

As shown in the last Equation in this comment, the ideal solution for the denoiser $D(x)$ can be found as the weighted sum of all clean data in the dataset.

Reviewer comment (Theory):
Eq. 4 describes the ideal solution for the denoiser D(xt) but it is not proved.

Author response:
We appreciate the reviewer's point regarding the lack of a formal proof for our assertion that an ideal solution for the denoiser can be represented as a weighted sum of all clean data in the dataset. 

Here, we provide proof to make it clear:
Given the standard deviation of Gaussian noise $\sigma$ and the clean data probability distribution $p_{\text{data}}(x) = \frac{1}{Y} \sum_{i=1}^{Y} \delta(x - y_i)$, the probability of the noisy data $p(x; \sigma)$ can be expressed as: 

$$
\begin{align}
p(x; \sigma) & = p_{\text{data}} * \mathcal{N}(0, \sigma^2 \mathbf{I}) \\
& = \int_{\mathbb{R}^d} p_{\text{data}}(x_0) \mathcal{N}(x; x_0, \sigma^2 \mathbf{I}) dx_0 \\
& = \int_{\mathbb{R}^d} \left[ \frac{1}{Y} \sum_{i=1}^{Y} \delta(x_0 - y_i) \right] \mathcal{N}(x; x_0, \sigma^2 \mathbf{I}) dx_0 \\
& = \frac{1}{Y} \sum_{i=1}^{Y} \int_{\mathbb{R}^d} \mathcal{N}(x; y_i, \sigma^2 \mathbf{I}) \delta(x_0 - y_i) dx_0 \\
& = \frac{1}{Y} \sum_{i=1}^{Y} \mathcal{N}(x; y_i, \sigma^2 \mathbf{I}) \\
\end{align}
$$


The loss of the Denoiser is:

$$
\mathcal{L}(D; \sigma) = E_{y \sim p_\text{data}} \mathbb{E}_{x \sim p(x, \sigma)}  \| D(x; \sigma) - y \|_2^2
$$

$$\mathcal{L}(D; \sigma) = \int_{\mathbb{R}^d} \frac{1}{Y} \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) \| D(x; \sigma) - y_i \|_2^2  dx$$

The solution is obtained by setting the gradient w.r.t $D(x; \sigma)$ to zero:

$$
0 = \nabla_{D(x;\sigma)} \mathcal{L}(D; x, \sigma) 
$$

$$
0 = \nabla_{D(x;\sigma)} \left[ \frac{1}{Y} \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) \| D(x; \sigma) - y_i \|_2^2 \right]
$$

$$
0 = \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) \nabla_{D(x;\sigma)} \left[ \| D(x; \sigma) - y_i \|_2^2 \right]
$$


$$
0  = \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) \left[ 2 D(x; \sigma) - 2 y_i \right] 
$$

$$
0 = \left[ \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) D(x; \sigma) \right] - \left[ \sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) y_i \right] 
$$

$$
D(x; \sigma) = \frac{\sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I}) y_i}{\sum_{i=1}^Y \mathcal{N}(x; y_i, \sigma^2 \mathrm{I})} 
$$

$$
D(x; \sigma) = \frac{\sum_{i=1}^Y \mathrm{exp}(\frac{||x-y_i||^2}{-2\sigma^2})  y_i}{\sum_{i=1}^Y \mathrm{exp}(\frac{||x-y_i||^2}{-2\sigma^2})}
$$

$$
D(x; \sigma) = \sum_{i=1}^Y \mathrm{softmax}(\frac{||x-y_i||^2}{-2\sigma^2}) y_i
$$

As shown in the last Equation in this comment, the ideal solution for the denoiser $D(x)$ can be found as the weighted sum of all clean data in the dataset.

Reviewer comment (Reproducibility):
The proposed method is hardly reproducible.

Author response:
We will release code to reproduce the results.

Reviewer comment (Writing):
Writing should be improved. It is hard to follow due to poor connection between consecutive sentences, especially in the Introduction.

Author response:
In the revised version of the draft, we fix typos and improve the connection between sentences. Besides, we  make the algorithm and figures more readable. We will keep polishing our draft for the final version. Thanks for your suggestion.

Reviewer comment (Writing):
Typos: "Eq. 7 implies that the proportion of data points whose square distance exceeds α times the standard deviation from the mean is? no more than 1/α^2."

Author response:
In the revised version of the draft, we fix typos and improve the connection between sentences. Besides, we  make the algorithm and figures more readable. We will keep polishing our draft for the final version. Thanks for your suggestion.

Reviewer comment (Writing):
Please use one-letter variables in the algorithms for readability.

Author response:
In the revised version of the draft, we fix typos and improve the connection between sentences. Besides, we  make the algorithm and figures more readable. We will keep polishing our draft for the final version. Thanks for your suggestion.

Reviewer comment (Presentation):
Please put titles on the axes in the figures for readability.

Author response:
In the revised version of the draft, we fix typos and improve the connection between sentences. Besides, we  make the algorithm and figures more readable. We will keep polishing our draft for the final version. Thanks for your suggestion.

Reviewer comment (Experiments):
This paper proposes a training method and a sampling method. How do they affect the performance when applied together?

Author response:
To prove the compatibility of our training and sampling methods, we evaluate our trained model on FFHQ-256 using our proposed sampling method. The table shows that our sampling method consistently lowers the FID score of the trained model, indicating improved performance since a lower FID score is desirable.

|  | Heun (NFE=35) | Ours (NFE=35) |
| --- | --- | --- |
| p(WSNR) | 7.89 | **7.40** |
| p($\sigma$) | 11.49 | **11.02** |

Reviewer comment (Experiments):
What is the number of evaluation points?

Author response:
Evaluation points are the **unique timesteps** at which the diffusion model makes predictions. The 2nd Heun's adopts the start point and end point to estimate the 2nd derivative. Therefore, the end point of the current timestep is identical to the start point of the next timestep. For example, when employing the 2nd Heun method for sampling, which uses 18 timesteps, the Number of Function Evaluations (NFE) is calculated as 2x18 - 1 = 35. Please refer to [this code segment](https://github.com/NVlabs/edm/blob/main/generate.py#L25) for more details.  However, the model actually performs evaluations at only 18 unique timesteps. We refer to these 18 points as evaluation points.  In the case of the midpoint method, all NFEs are performed at unique timesteps, so NFE equals the Number of Evaluation Points (NEP).