Paper: S\(^{2}\)-DMs: Skip-Step Diffusion Models (ICLR-2024)

Reviewer comment (Presentation):
I cannot understand what does Figure 1 means in the paper. Does it represent for the trajectory of an individual sample $x_t$? What does *real* mean? What is the dashed line? What is the solid line? Is it a two-dimensional dataset? What is the x-axis and what is the y-axis?

Author response:
Thank you for pointing out the ambiguity in our illustration.  We apologize for the oversight.  To clarify, the 'real' refers to the actual transition trajectories between two distributions. The left side of the figure depicts the sampling trajectories of different algorithms, where the dashed lines represent the trajectories of other algorithms, and the solid lines denote those of our proposed algorithm.  The $x_T$  denotes the normal distribution, and $x_0$ represents the data distribution.  The process involves sampling from the normal distribution and restoring it to the $x_0$ data distribution via various algorithmic trajectories.  We have now added a detailed explanation of this in our manuscript and resubmitted it for your review.

Reviewer comment (Experiments):
The additional training objective is scaled by *0.01*. I strongly doubt the effectiveness of the novel part of this paper.

Author response:
We assure you that all our results are fully reproducible.        
Regarding the $L_{skip}$ weight being set to 0.01, this was a deliberate choice based on our experimental observations. We found that the value of $L_{skip}$ was approximately 80-100 times that of $L_0$. To maintain a balanced numerical relationship between $L_0$ and $L_skip$, a weight of 0.01 was selected. This weight is effective within this range. Setting the weight too high disrupts this balance and adversely affects training performance.
we have submitted all our codes as supplementary materials, along with a detailed README for guidance. Should there be any doubts or questions, we encourage experimental replication using these resources.

Reviewer comment (Experiments):
Many other fast sampling techniques [1,2,3] have been proposed in the last years. The author should consider to compare with them.

Author response:
We chose to compare our method with PNDM because it already demonstrates excellent performance and is a very recent work (ICLR22).  Moreover, it shows the best performance compared to the other paper mentioned.

Reviewer comment (Experiments):
Can authors compare with the DDIM which is only trained on the sampling time steps $t_i \in [t_0, t_1,\cdots, t_N]$. I am not convinced by this additional loss $L_{skip}$.

Author response:
In our experiments, DDIM refers to the original training method of DDIM, while S2-DMs (DDIM) represents our own training approach with $L_{skip}$. After the training was completed, our proposed training method achieved better results across different sampling step numbers.

Reviewer comment (Evaluation):
Why does the performance get worse when the step increases as shown in Figure 3?

Author response:
We posit that the introduction of $L_skip$ is intended to bridge the gap between training and sampling with respect to step skipping.  This means when a large skip value, such as 50 as illustrated in Figure 3, is set, the model is more inclined to perform optimally with a skip of 50.  This principle applies similarly for smaller skip settings like 10 or 2.  To illustrate, a model without Lskip tends to perform best with a skip of 1, and its performance diminishes as the skip value increases.

Reviewer comment (Theory):
It was not clear to me how aligning the *forward* processes $q_\theta(x_t | x_{t - skip})$ and $q(x_t | x_{t-1})$ would result in the model incorporating the "skip" information as claimed. For instance, these conditional densities rely on some *fixed* $x_{t-1}$ and $x_{t-skip}$ -- which themselves are random variables depending on an actual datapoint $x_0$, and so may be significantly different from one another (e.g. in Equation 8 and Equation 9).

Author response:
We aim to ensure that the model's output from $x_{t-1}$ to $x_{t}$ aligns with the output from $x_{t-1}$ to $x_{t-skip}$, thus conforming to the model's behavior during sampling from $x_{t-1}$ to $x_{t-skip}$. This serves as a guiding principle during training, enabling the model to better adapt to the skip-step pattern prevalent in sampling.

Reviewer comment (Theory):
The derivation of the loss (Equation 10) is not sound. The original DDPM loss (Equation 7) is *not* obtained by matching the forward transitions and matching terms as claimed, but rather by minimizing the KL between the reverse transitions (see Equation 5 in the DDPM paper). Moreover, the authors take their proposed derivation and modify it with another heuristic in Equation 12.

Author response:
In the derivation process, compromises were made to stabilize the loss function, and heuristic methods were also introduced for the sake of training stability. However, the effectiveness of the experiments indicates that these compromises were effective. This also forms part of our future work: exploring how to model more effectively to minimize such compromises.

Reviewer comment (Evaluation):
I am quite skeptical of the results in Table 1 and Table 2, given that the proposed change is such a small modification of the standard DDIM training procedure. These results would be significantly more convincing if the authors presented error bars with their FID scores.

Author response:
We believe that while our work may appear to involve minimal modifications, the results are significantly effective and easy to follow. This is because we identified the asymmetry problem between training and sampling in diffusion models and modeled it separately. This approach is the first of its kind and but also has the potential to inspire future researchers on how to model efficiently, further mitigating this issue. Hence, we view our work as both simple and effective.

Reviewer comment (Evaluation):
The authors obtained significantly lower FID scores in Table 1 and Table 2 for DDIM than the original DDIM paper [2] (see Table 1 in [2]). If one compares the results for S2DM in the submission and the FID scores from Table 1 in [2], DDIM consistently outperforms the reported S2DM scores.

Author response:
Firstly, we are very confident in the correctness of our experimental results and have provided the relevant code and a README for guidance to facilitate replication of the experiments.  
Furthermore, as mentioned in our paper, we have conducted our experiments inheriting the official DDIM code repository, and all default hyperparameters and random seeds remain unaltered (as can be verified in the code we submitted). The reason why our experiments on CIFAR10 did not achieve the results reported in the original paper is due to DDIM employing additional tricks and non-default parameters specifically for CIFAR10 sampling, as detailed here: https://github.com/ermongroup/ddim/issues/3. Others have obtained similar results to ours under the same default parameters. In our experiments on CELEBA, the performance of the DDIM we trained is notably better than that reported in the original paper. For example, at 100 sampling steps, our DDIM achieved an FID of 4.63 compared to the original paper's 6.53, and at 10 steps, our DDIM reached an FID of 13.12 versus the original's 17.33. Therefore, we do not agree with your skepticism about our experimental results, as we have not modified any hyperparameters.

Reviewer comment (Presentation):
The reference to Watson et al. (2021) appears twice at the start of Section 2.

Author response:
Thank you for pointing out the textual and formulaic errors. We have made the necessary corrections and uploaded the revised version.

Reviewer comment (Writing):
Section 5, "Relate Work" should be "Related Work".

Author response:
Thank you for pointing out the textual and formulaic errors. We have made the necessary corrections and uploaded the revised version.

Reviewer comment (Novelty):
Section 5 lists several relevant pieces of work "other innovative methods have been introduced to further refine DDPMs sampling, such as reverse SDEs with unique coefficients, ”corrector” steps, and probability flow ODEs" but does not include a citation for these methods.

Author response:
Thank you for pointing out the textual and formulaic errors. We have made the necessary corrections and uploaded the revised version.

Reviewer comment (Writing):
Equation 3 should use "\log".

Author response:
Thank you for pointing out the textual and formulaic errors. We have made the necessary corrections and uploaded the revised version.

Reviewer comment (Writing):
Figure 3 "stpe" should be "step".

Author response:
Thank you for pointing out the textual and formulaic errors. We have made the necessary corrections and uploaded the revised version.

Reviewer comment (Theory):
In (9), why is \( q(x_t | x_{t - 1}) = \sqrt{\alpha_t} x_{t - 1} + \sqrt{1 - \bar{\alpha}_t} \epsilon \)? The original process for adding noise should have \( \sqrt{1 - \alpha_t} \) instead of \( \sqrt{1 - \bar{\alpha}_t} \) for the noise term?

Author response:
Upon reevaluation, we acknowledge the error you pointed out in our initial submission. We appreciate this correction and will revise our manuscript to include the correct formula $\sqrt{1-\alpha_{t}}$ as suggested.

Reviewer comment (Experiments):
How does the balance between \( L_0 \) and \( L_{skip} \) (the value of \( \tau \)) affect the performance?

Author response:
In addressing the balance between $L_0$ and $L_{skip}$, we first wish to correct a textual mistake regarding the value of $\tau$. It was erroneously stated in our initial submission. In reality, the value of Lskip is approximately 80-100 times greater than that of $L_0$. To balance their magnitudes effectively, we set $\tau$ to 0.99. This setting not only maintains the balance but also proves to be effective in a range around this value. However, setting $\tau$ too low, such as 0.5 or 0.1, disrupts this balance, leading to a significant decline in performance.

Reviewer comment (Experiments):
With the regularized term, is the training cost larger than training the original DDPM? I am curious whether the trained score also performs better when sampling using DDPM.

Author response:
Yes, the training cost is greater than DDPM, which is also mentioned in the appendix.

    | **Dataset/Model** | CIFAR/DDIMs | CelebA/DDIMs | CIFAR/$S^2$-DMs | CelebA/$S^2$-DMs |
    | ----------------- | ----------- | ------------ | ------------- | -------------- |
    | Time per iter (s) | 0.0025      | 0.0048       | 0.0031        | 0.0056         |
    | Memory per GPU (G)| 4.83        | 15.55        | 4.83          | 15.56          |

Reviewer comment (Theory):
The average value of \( L_0 \) is approximately 80–100 times larger than that of \( L_{skip} \), however in (13) when setting \( \tau = 0.99 \), \( L_0 \) would be 10000 larger than \( (1 - \tau)L_{skip} \).

Author response:
R1 from Rebuttal 2: Regarding the values of L0 and Lskip, their average values during training were: $L_0$=0.02 and $L{skip}$=1.2. Therefore, we chose 0.99 and 0.01 as weight values to balance their relationship.

Reviewer comment (Writing):
Caption in Figure 2, line 3, contains an additional "the" in "the the".

Author response:
We have corrected the typographical errors and resubmitted the revised manuscript.

Reviewer comment (Writing):
Caption in Figure 3 contains the typo 'stpe'.

Author response:
We have corrected the typographical errors and resubmitted the revised manuscript.