Paper: Elucidating the Exposure Bias in Diffusion Models (ICLR-2024)

Reviewer comment (Experiments):
The proposed method of Epsilon scaling is not tuning-free as it is sensitive to the choice of hyper parameter of the schedule $\lambda_t = kt+ b$ and thus requires extensive hyper parameter tuning. The choice of optimal hyperparameters varies for each dataset as well as the number of sampling steps T. As per Table 10, Table 11 and Table 12, values of k and b have been tuned up to 4th or even 5th decimal place. For instance, there are values like k=0.00022 , b=1.00291 in Table 10. This degree of hyperparameter tuning seems a bit extreme. The paper is also missing relevant sensitivity analysis for hyperparameters k and b, and it would be useful to include it (both over large strides and small strides).

Author response:
**A1**. Thank you for pointing this out, Reviewer gcKq also asked a similar question. Overall, the parameters k,b are insensitive to the FID result, please refer to our answer A2 to Reviewer gcKq for details. Since the linear schedule $\lambda_t = kt+b$ is determined from its optimal uniform schedule $\lambda_t=b^*$, b of the linear schedule is actually computed rather than brute-force searched. Thus in the linear schedule case, only k is searched with several trials. The sensitivity analysis is of great interest and helpful for the reader to understand the easiness of our method, for example, we present a curve that shows the FID w.r.t the full range of $b$. We decided to include it in the paper later on.

Thank you for the response, for a better clarification of our k selection, we elaborate the complete procedure and full search range for k in a linear $\lambda_t$ case. Taking the LSUN 64x64 as an example, we found that the optimal uniform $\lambda_t^*=1.011$ for NFE=20, then to decide the linear schedule $\lambda_t=kt+b$, we search the optimal $k$  on a stride $0.0002$ and b is calculated by retaining $\sum \lambda_t = \sum \lambda_t^*$, the full range of $k$ we implemented is shown in Table 13.  More experiments on FFHQ128x128, ImageNet 64 are presented in Table 14, 15. Overall, we highlight the ease and low cost of finding the schedule $\lambda_t$. Although different opinions might exist regarding the definitionof sensitivity (how much is sensitive, how much is insensitive), the goal of the sensitivity analysis or people’s expectation is that the schedule $\lambda_t$ is easy to find in practice, which we have shown.  

Table 13. FID with linear schedule $\lambda_t=kt+b$ on LSUN 64x64 dataset under NFE=20

| k | 0 (uniform) | 0.0002 | 0.0004 | 0.0006 | 0.0008 | 0.0010 |
| --- | --- | --- | --- | --- | --- | --- |
| ADM-ES | 8.22 | 8.10 | 8.01 | 7.75 | 7.60 | 7.69 |

Table 14. FID with linear schedule $\lambda_t=kt+b$ on FFHQ 128x128 dataset under NFE=20

| k | 0 (uniform) | 0.0003 | 0.0004 | 0.0005 | 0.0006 | 0.0007 | 0.0008 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ADM-ES | 26.14 | 24.50 | 24.18 | 24.35 | 24.41 | 24.73 | 24.83 |

Table 15. FID with linear schedule $\lambda_t=kt+b$ on ImageNet 64x64 dataset under NFE=20 (FID computed by 10k samples)

| k | 0 (uniform) | 0.0001 | 0.0002 | 0.0003 | 0.0004 | 0.0005 |
| --- | --- | --- | --- | --- | --- | --- |
| ADM-ES | 10.23 | 10.19 | 9.98 | 10.22 | 10.27 | 10.31 |

Reviewer comment (Experiments):
The process of finding optimal hyperparameters for Epsilon scaling seems to be slow and can be computationally intensive. Usually, generating 10K sampling to calculate FID takes several minutes for small sampling steps and several hours for large sampling steps. Tuning hyper-parameters for each dataset and each choice of sampling length T, can thus require a lot of GPU compute and time. While the method itself is simple, tuning this method for optimal performance can be tricky.

Author response:
**A2**. As we mentioned above, the determination of $\lambda_t$ is practically easy and not expensive. For each dataset and network, we find the optimal $\lambda_t$ by always using a single A100 GPU, costing from several minutes to three hours. We hope this tackles your concerns about the computational cost of our method.

Reviewer comment (Evaluation):
In Table 3, for CIFAR-10, values of EDM for VP and VE should be reported for NFE=35 which gets FID = 1.97 (VP) and 1.98 (VE) on CIFAR-10, respectively. Underreporting FID values of EDM is a bit misleading.

Author response:
**A3**. Thank you for the question, to make sure we measure the true improvement of our method, we re-run the sampling and FID measurement for EDM baseline in the same machine and environment. We report FID=1.97 for EDM (VP, uncond) and FID=1.82 for EDM (VP, cond) in the paper, the latter is different from FID=1.79 reported in EDM, which is possibly due to the machine difference. To avoid misunderstandings, we will add a sidenote in the table to mention the result is reported by re-running the baseline.

Reviewer comment (Presentation):
Could the authors include the number of sampling steps in qualitative results in Figures 11-15?

Author response:
**A5**. That is a good suggestion, we will include it in the next version paper.

Reviewer comment (Evaluation):
Results for the proposed epsilon-scaling mechanism with DDIM solver aren't impressive. There's only a marginal change in the FID score, compared to the DDIM sampler which is used as a popular framework for many diffusion models. Here, the gains are marginal even for cases with reasonable timesteps (~50 or more).

Author response:
**A1**. Although the FID gain of Epsilon Scaling in DDIM is less than the gain in DDPM, our method is still useful in DDIM sampler as a simple plug-in sampling solution. Except for the outlier in CelebA 20-step sampling, the FID improvement is also non-trivial and can be significant from 4.06 to 3.38 in CIFAR-10. In the case that DDIM-ES gains little benefit, one can easily get a better sample quality by increasing the value of $\eta$ from 0 to 0.2, 0.5, etc.

Reviewer comment (Experiments):
There is little information provided regarding how the scaling factor $k$ and $b$ are selected, and how many hyperparameters were searched for optimal $k$ and $b$. It would also be good to quantify sensitivity to these hyperparameters.

Author response:
**A2**.  Thank you for the suggestions. First, the FID gain can be achieved in a wide range of k,b, which indicates the insensitivity of the hyperparameters. For example, in Tables 11, 12, we present the FID and parameter b across ADM and EDM baselines. We are going to add a more complete sensitivity graph and analysis in the paper.

Table 11. FID on ADM baseline and CIFAR-10 under different parameters b (b=1 represents the baseline)  
| b | 1 (baseline) | 1.015 | 1.016 | 1.017 | 1.018 | 1.019 |  
| --- | --- | --- | --- | --- | --- | --- |  
| NFE=100 | 3.37 | 2.20 | 2.18 | 2.17 | 2.21 | 2.31 |  
| NFE=50 | 4.43 | 2.53 | 2.51 | 2.49 | 2.53 | 2.55 |  

Table 12. FID on EDM baseline and CIFAR-10 under different parameters b (b=1 represents the baseline)  
| b | 1 (baseline) | 1.0005 | 1.0006 | 1.0007 | 1.0008 |  
| --- | --- | --- | --- | --- | --- |  
| NFE=35 | 1.97 | 1.948 | 1.947 | 1.949 | 1.953 |  
|  |  |  |  |  |  |  
| b | 1 (baseline) | 1.004 | 1.005 | 1.006 | 1.007 |  
| NFE=13 | 7.16 | 6.60 | 6.55 | 6.54 | 6.55 |  

Thanks to the insensitivity of parameters $k$, $b$, the parameter searching is effortless, searching for a constant $b$ takes no more than 10 trials for a specific network, dataset and sampling step. We usually take 2 rounds of searching, first doing the coarse search with a stride of 0.005, then doing a fine search with a stride of 0.001. When using linear schedule $\lambda_t=kt+b$, we first find its optimal uniform schedule $\lambda_t=b^*$, then search for the optimal slope $k$ while maintaining $\sum\lambda_t$ unchanged (i.e. $\sum\lambda_t=NFE \times b^* $), in this way, $b$ of the linear schedule is calculated, we got only $k$ is searched. There are already descriptions of the selection of $k,b$ in Appendix A.8, but we will accordingly add more details to the appendix.

Reviewer comment (Experiments):
Does the method perform worse than Heun solvers with suboptimal $k$ and $b$ selection?

Author response:
**A3**. Due to the wide range of near-optimal solutions, our method still works well under suboptimal k and b, see Table 12 where we use Heun solver.

Reviewer comment (Theory):
The main concern lies in the assumption at the start of the derivation. In part 3.2, the authors assume that the reconstructed image x_{\theta}^t at the sampling process follows the Gaussian distribution, where the mean is the GT image x_0, and the variance is the Gaussian noise. This conflicts with some intuitive observations. For example, the reconstructed image x_{\theta}^t is often a degraded version of the GT image, and the mean of x_{\theta}^t is different from x_0.

Author response:
**A1**. The generated sample is the result of multi-step sampling, where the Gaussian approximation of $x_0$ prediction error holds in a single step (please see our answer A1 to reviewer nv6a). For multi-step sampling, this Gaussian might be loose and exhibit the mean drift.

Reviewer comment (Writing):
Some formulations, notations and explanations in part 3.1 and 3.1 are redundant. It is highly recommended to improve the organization and the writing of this section.

Author response:
**A2**. Thanks for your suggestions, we will simplify section 3.1 since the DDPM formulation is already well-known in the field.

Reviewer comment (Evaluation):
It is necessary to discuss the further meaning and defects of exposure bias correction. For instance, the necessity of matching the forward and sampling process. Reducing such bias forces the generated images to be more similar to the training images in distribution, i.e, more data repetition in the generated images. The authors are encouraged to give more discussions.

Author response:
**A3**. From the sampling trajectory perspective, exposure bias correction indeed tries to match the training and sampling trace. The main benefit of this match is to avoid the intermediate sample $\hat{x_t}$ falling into the problematic vector field that has never been trained in the training phase. From the sample quality perspective, the exposure bias correction corresponds to the correction of overexposure, underexposure, and detail defects. 

We empirically observed that the generated image samples before and after applying Epsilon Scaling share the same semantic mode. Thus, the exposure bias correction does not affect the diversity of the synthesised samples. Our experiments of recall further show that the correction of exposure bias results in even slightly better sample diversity (see Table 10 below). Thus, reducing exposure bias does not lead to data repetition. We plan to add this discussion in section 5.6 


Table 10. Recall of ADM and ADM-ES using 100-step sampling

|  | CIFAR-10 | LSUN 64x64 | FFHQ 128x128 | ImageNet 64x64 | ImageNet 128x128 |
| --- | --- | --- | --- | --- | --- |
| ADM | 0.591 | 0.605 | 0.497 | 0.621 | 0.586 |
| ADM-ES | 0.613 | 0.606 | 0.545 | 0.632 | 0.592 |

Reviewer comment (Experiments):
Several prior works have observed and identified the "exposure bias" problem studied in the current paper. It would be helpful to discuss them in the paper: (1) Section 4 (practical considerations) and Fig 13 in EDM [1] points out that the neural network tends to remove slightly too much noise. Hence they use an inflated noise to counteract it. I think adding $S_{noise}$ into ODE or SDE samplers is a valid baseline for the current paper. (2) Section 4.2 in PFGM [2] / Section 5, Fig 4.b in PFGM++ [3] further dig into the exposure bias problem and show that the strong norm-t relation in the diffusion model is the cause.

Author response:
**A1**. We think our Epsilon Scaling is consistent with EDM’s statement: the neural network tends to remove slightly too much noise, because our solution scales down the predicted noise, thus avoiding removing too much noise. We will update the SDE sampler experiments once we get the results. As for PFGM++ discussion, please refer to our answers A2 and A3.

Reviewer comment (Experiments):
The current paper demonstrates the discrepancy of the prediction norms (Fig 2) without further examining the reason. [2] seems to offer a plausible explanation for this phenomenon. If their argument is true, this "exposure bias" mainly occurs in large $t$ (norm-t is strongly correlated). I guess one simple experiment to validate the hypothesis is to use true score up to a certain time $t$ during sampling and study the effects (like current Fig.2, but the green and red curve would be the same in [t,T]).

Author response:
**A2**. First of all, we reckon the strong norm-t opinion does not contradict our claim that the prediction error of $x_0$ is the cause of exposure bias. We are convinced that PFGM++ is robust to prediction error compared with diffusion models. So they are two concepts and can be concurrent. Also, we argue that exposure bias exists in the whole sampling chain of diffusion models. The experiment in paper [1] supports our opinion, where they measure the FID by leaking the ground truth $x_t$, $t \sim(1, T)$ and $T=1000$, to the network at the beginning of sampling, then the sampling starts at $x_t$. We replicate their results on ADM and ImageNet 32 in Table 6. We see that the FID is still far from near-zero even though the sampling starts at $x_{300}$ and $x_{100}$ which jump over the large t area (norm-t is strongly correlated).

Table 6. FID on ImageNet 32
| t | 100 | 300 | 500 | 700 | 1000 |
| --- | --- | --- | --- | --- |
| ADM | 0.98 | 1.81 | 2.59 | 3.11 | 3.54 |

[1] Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. ICML, 2023

Reviewer comment (Experiments):
It's encouraging to see that the method improves over EDM, especially in the small NFE regime. Could the authors try to apply the method on PFGM++, which is claimed to be more robust to the "exposure bias"? It could also verify if or not the exposure bias is due to the strong norm-t correlation pertaining to diffusion models.

Author response:
**A3**. We think PFGM++ is an interesting baseline to apply our method. We present the results on PFGM++ (D=128) and PFGM++ (D=2048) under unconditional sampling (see Table 7, 8). We found that Epsilon Scaling (ES) is compatible with PFGM++ and even enjoys the extra benefit: under very small NEF regimes, the FID gain of ES is still significant. We believe the robustness to prediction error of PFGM++ is the reason. We would like to include the full experiments on PFGM++ in the main paper and discuss the strong norm-t relation in the paper as well.

 Table 7. FID on PFGM++ baseline (D=128) and CIFAR-10
| NFE | 9 | 13 | 21 | 35 |
| --- | --- | --- | --- | --- |
| PFGM++ | 37.82 | 7.55 | 2.34 | 1.92 |
| PFGM++ with ES  | 17.91 | 4.51 | 2.31 | 1.91 |

Table 8. FID on PFGM++ baseline (D=2048) and CIFAR-10
| NFE | 9 | 13 | 21 | 35 |
| --- | --- | --- | --- | --- |
| PFGM++ | 37.16 | 7.34 | 2.31 | 1.91 |
| PFGM++ with ES  | 17.27 | 4.88 | 2.20 | 1.90 |

Regarding the small NFE regime on EDM baseline, we present the results of using the Euler solver below, since we found that, when NFE is smaller than 10 in EDM, the Heun solver performs much worse than the Euler solver mainly due to the twice-larger stepsize and severe truncation error.

Table 9. FID on EDM baseline under small NFE regimes (CIFAR-10 and unconditional)
| NFE | 5 | 7 | 9 | 13 |
| --- | --- | --- | --- | --- |
| EDM | 68.69 | 36.58 | 22.97 | 12.28 |
| EDM-ES | 55.26 | 27.29 | 16.34 | 8.39 |

Reviewer comment (Theory):
For Q4, if we deliberately scale the network prediction $\epsilon_\theta$ by $\lambda_t$ (i.e., the network prediction error studied in the paper), the resulting solution seems not related to the $\Delta N(t)=\int_t^T \lambda_t$. Concretely, the new ODE is $dx_t /dt = \lambda_t /t x_t $, and the solution is related to the time integration of $t\lambda_t$. So the argument in the paper appears to be inconsistent with this simple example.

Author response:
Dear reviewer FqVz, Thanks for your question **Q4**, we have some doubts on the case you provided. If we are not mistaken, the forward ODE you defined is $x_t = -t^2 \epsilon$ where $\epsilon \sim N(0, I)$, according to the probability ODE $dx = -\dot\sigma(t) \sigma(t) \nabla_x log p(x) dt$ (equation 1 in EDM paper), you get the ODE $dx/dt = -x_t/t$, soving this ODE, you have  $x_t = \frac{T}{t}x_T$, based on which, we found $x_0$ is infinite. In a similar way, we tried with a similar forword ODE $x_t = t \epsilon$ where $\epsilon\sim N(0, I)$, we derive the reverse ODE $dx/dt =x_t/t$, soving it out leads us to $x_t = \frac{t}{T}x_T$, we now have $x_0=0$. May we ask is this the case you actually mean? Also, it would be appreciated if you could elaborate the details of "when we scale $\epsilon$ by $\lambda_t$, the resulting $x_t $ relates to $\Delta N(t)= \int \lambda_t/t dt$", just to make sure we are on the same page for further discussions. Thank you!

Reviewer comment (Theory):
I don't immediately see why the quotient of $\Delta N(t) = |\epsilon^s|/|\epsilon^t|$ can be translated into the scaling factor at $\epsilon(x_t,t)$ during sampling. Because $\Delta N(t) = |\epsilon^s|/|\epsilon^t|$ and $\Delta N(t)=\int_t^T \lambda_t$ only implies that $\frac{d |\epsilon^s(x_t,t)|/|\epsilon^t(x_t,t)|}{dt}=\lambda_t$. I tried the simplest data distribution I can come up with --- $p(x)=\delta(x-0)$. The argument even failed in this case. (I use the notation in EDM and assume $t=\sigma$). The true $\epsilon$ at $x_t$ is $\epsilon(x_t,t) = -\frac{x_t}{t^2}$. By solving the diffusion ODE $dx/dt = -\frac{x_t}{t}$ we get $x_t = \frac{Tx_T}{t}$ (assume the start point at time $T$ is $x_T$). However, when we scale $\epsilon$ by $\lambda_t$, the resulting $x_t$ does not directly relate to $\Delta N(t)=\int_t^T \lambda_t$, but a interval over $\lambda_t/t$. I suggest rethinking the principled way of scaling by working out the toy example first.

Author response:
Dear reviewer FqVz, Thanks for your question **Q4**, we have some doubts on the case you provided. If we are not mistaken, the forward ODE you defined is $x_t = -t^2 \epsilon$ where $\epsilon \sim N(0, I)$, according to the probability ODE $dx = -\dot\sigma(t) \sigma(t) \nabla_x log p(x) dt$ (equation 1 in EDM paper), you get the ODE $dx/dt = -x_t/t$, soving this ODE, you have  $x_t = \frac{T}{t}x_T$, based on which, we found $x_0$ is infinite. In a similar way, we tried with a similar forword ODE $x_t = t \epsilon$ where $\epsilon\sim N(0, I)$, we derive the reverse ODE $dx/dt =x_t/t$, soving it out leads us to $x_t = \frac{t}{T}x_T$, we now have $x_0=0$. May we ask is this the case you actually mean? Also, it would be appreciated if you could elaborate the details of "when we scale $\epsilon$ by $\lambda_t$, the resulting $x_t $ relates to $\Delta N(t)= \int \lambda_t/t dt$", just to make sure we are on the same page for further discussions. Thank you!

Reviewer comment (Theory):
The assumption that the model value $x_\theta^t$ is averaged to the true $x_0$ should be more verified.

Author response:
**Q1. The assumption that the model value $x_{\theta}^t$ is averaged to the true $x_0$ should be more verified.**

**A1**. The Gaussian assumption of the single step $x_0$ prediction error was first proposed at Analytic-DPM [1] and later on verified at [2] in their section 5.3 and appendix A.5, where they use Shapiro–Wilk statistical test to verify that $x_{\theta}^t-x_0$ is nearly an isotropic Gaussian, we will also mention this in the paper.

[1] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. ICLR, 2022b

[2] Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. ICML, 2023

Reviewer comment (Experiments):
Analytic-DPM is also a simulation-free (only calculates the optimal variance over the timesteps). This paper proposed that the limit of the Analytic-DPM method is that this is not advantageous in the low-NFE regime, but this paper reproduces well even in low-NFE regime (NFE=10), if the variance clipping in high-SNR (last or second last sampling step) timesteps is held. The fair comparison in various NFE regime should be done, which can affect the scoring of this paper.

Author response:
**Q2. Analytic-DPM is also a simulation-free (only calculates the optimal variance over the timesteps) This paper proposed that the limit of the Analytic-DPM method is that this is not advantageous in the low-NFE regime, but this paper reproduces well even in low-NFE regime (NFE=10), if the variance clipping in high-SNR (last or second last sampling step) timesteps is held. The fair comparison in various NFE regime should be done, which can affect the scoring of this paper..**

**A2**. Our observation for Analytic-DPM is that their optimal sampling schedule $\beta_{t}^*$ underperforms the DDPM baseline in the large NFE regime (400~1000), and $\beta_{t}^*$ indeed performs well in the mid and low NFE range. We appreciate your suggestions of verifying our method in various NFE regimes and we add the results in A3

Reviewer comment (Experiments):
What about the case that the NFE is less than 20 or greater than 100?

Author response:
**Q3. What about the case that the NFE is less than 20 or greater than 100?**

**A3**. Thanks for mentioning this, our method also works well when NFE<20 and NFE>100. For example, in the ADM baseline, our ADM-ES consistently gains FID improvements on different datasets using NFE=10 and constant Epsilon Scaling (see Table 1). We will roll it out to all baselines we used in our paper after the short discussion period.

Table1. FID under NFE=10 on ADM baseline
|  | CIFAR-10 | LSUN 64x64 | ImageNet 64x64 (cond) |
| --- | --- | --- | --- |
| ADM | 23.37 | 45.35 | 27.90 |
| ADM-ES (ours) | 16.89 | 23.66 | 20.94 |

As for large NFE regions, we show the full results on Analytic-DDPM baseline (Table 2) and ADM baseline (Table 3), where we use constant $\lambda_t$ for Epsilon Scaling (ES). However, we highlight that it is unnecessary to use very high NFE in practice since NFE=100 achieves the near-optimal FID in most cases while being computationally efficient. Many works in the literature have also noticed this phenomenon [3][4][5].

Table 2. FID on CIFAR-10 using Analytic-DDPM baseline (linear noise schedule)
| NFE | 20 | 50 | 100 | 200 | 400 | 1000 |
| --- | --- | --- | --- | --- | --- | --- |
| Analytic-DDPM | 14.61 | 7.25 | 5.40 | 4.01 | 3.62 | 4.03 |
| Analytic-DDPM with ES | 11.02 | 5.03 | 4.09 | 3.39 | 3.14 | 3.42 |


Table 3. FID on CIFAR-10 using ADM baseline
| NFE | 200 | 300 | 1000 |
| --- | --- | --- | --- |
| ADM | 3.04 | 2.95 | 3.01 |
| ADM-ES | 2.15 | 2.14 | 2.21 |


[3] Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML, 2021

[5] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., & Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 2022

[4] Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. NeurIPS, 2023

Reviewer comment (Theory):
In the line below Equation 7, does $q ( x_t|x_{t+1},x_\theta^{t+1})$ have the same distribution as $q(x_{t+1}|x_t,x_\theta^t)$?

Author response:
**Q6. In the line below Equation 7, does $q(x_t | x_{t+1}, x_{\theta}^{t+1})$ have the same distribution as $q(x_{t+1} | x_t, x_{\theta}^t)$ ?**

**A6**. Thank you for pointing this out, we intended to say that $q(\hat{x_t}| x_{t+1}, x_{\theta}^{t+1})$ and $q(x_{t-1}| x_{t}, x_{\theta}^{t})$ are in the same function format and they are just the difference in notation. Clear clarification would be made, such as, 'we now compute compute $q(\hat{x_t}| x_{t+1}, x_{\theta}^{t+1})$, which share the same function format as $q(x_{t-1}| x_{t}, x_{\theta}^{t})$, by substituting with the index $t+1$ and using $\hat{x_t}$ to highlight that it is generated in the sampling stage'

Reviewer comment (Writing):
The exposure bias term $\delta_t$ should be more precisely denoted in the main section. This proposed metric is only introduced in detail in the appendix section.

Author response:
**Q7. The exposure bias term $\delta_t$ should be more precisely denoted in the main section. This proposed metric is only introduced in detail in the appendix section.**

**A7**. Thanks for the suggestion, we also realized that the exposure bias $\delta_t$ is important for the readers to understand the metric and we will move it to the main paper.

Reviewer comment (Experiments):
It will be helpful if the Analytic-DPM and the existing exposure bias methods are also considered as the benchmark, even though this does not use the baseline reverse diffusion process by modifying.

Author response:
**Q8. It will be helpful if the Analytic-DPM and the existing exposure bias methods are also considered as the benchmark, even though this does not use the baseline reverse diffusion process by modifying.**

**A8**. We tested the performance of our Epsilon Scaling on Analytic-DPM, PFGM++ even though they are not the initial target baselines of our method (see Table 2 for the results of Analytic-DPM). Regarding the exposure bias solutions, we also directly compare Epsilon Scaling with Input Perturbation [2] (a training regularization to reduce exposure bias) on the baseline ADM. The results shown in Table 4,5 demonstrate the effectiveness and generalization of Epsilon Scaling. We use constant $\lambda_t$ for Epsilon Scaling (ES) below.

Table 4. FID on CIFAR-10 (2nd Heun solver and D=128)
| NFE | 9 | 13 | 21 | 35 |
| --- | --- | --- | --- | --- |
| PFGM++ | 37.82 | 7.55 | 2.34 | 1.92 |
| PFGM++ with ES | 17.91 | 4.51 | 2.31 | 1.91 |

Table 5. FID on CIFAR-10
| NFE | 10 | 20 | 50 | 100 |
| --- | --- | --- | --- | --- |
| ADM (baseline) | 23.37 | 10.36 | 4.43 | 3.37 |
| ADM-IP | 20.93 | 6.96 | 2.91 | 2.38 |
| ADM-ES (ours) | 16.89 | 5.15 | 2.49 | 2.17 |

Reviewer comment (Reproducibility):
It will be also helpful also to be mentioned if the Analytic-DPM requires the heavy clipping of the noise variance $\beta_t$ in the near-signal part of the sampling, which causes.

Author response:
**Q5. It will more support the method, if the bias of the expected noise $\epsilon_{\theta}^s$ or $\epsilon_{\theta}^t$. This helps the reader to understand that by simply scaling down the expected noise calibrates the sampling steps, without taking bias (i.e. translating) the noise**

**It will be also helpful also to be mentioned if the Analytic-DPM requires the heavy clipping of the noise variance $\beta_t$ in the near-signal part of the sampling, which causes**

**A5**. It looks like these two questions are sort of incomplete, would you mind re-stating your questions?