Paper: Lipschitz Singularities in Diffusion Models (ICLR-2024)

Reviewer comment (Presentation):
I believe 1 of the methods from the appendix is not mentioned in the main text - namely the Remp method (D.3.3).

Author response:
**Q1: "Remap" is not mentioned in the main text.**  
**A1:** Yes, thank you very much for reminding us. We didn't mention "Remap" in our main text because of space constraints. we have added a brief discussion of Remap in the "Comparison with some alternative methods" part of **Section 4**. This discussion aims to introduce the concept of Remap and highlight its limitations. Our intention is to draw the readers' attention to this intriguing alternative approach.

Reviewer comment (Experiments):
It would be nice to see an expanded discussion of these with some small experiment to show the quantitative difference between the proposed method and these other methods. I appreciate the space limitation, but I think this is really an interesting point.

Author response:
**Q2: It would be nice to see an expanded discussion of these with some small experiment to show the quantitative difference between the proposed method and these other methods.**  
**A2:** Thank you for your advice, we have incorporated a new figure (**Figure 3** of the updated PDF) in the "Comparison with some alternative methods" part of **Section 4**, to visually represent the quantitative evaluation of these alternative methods. By including this visual representation, we aim to facilitate a rapid understanding of the performance of these alternative methods for the readers.

Reviewer comment (Writing):
Could the authors highlight better which Lipschitz constant it is that is important, and why we care about it? While I understand which and why it is cared about, it is perhaps not the clearest from reading the paper. The sentence in the abstract "they frequently exhibit the infinite Lipschitz near the zero point of timesteps" is a good example of this - it does not specify what function has high Lipschitz constant, or why indeed that matters. From reading the paper in depth, the authors care about the Lipschitz constant of the quantity $-\sigma_t \nabla \log q_t(x)$ as a) neural networks find it difficult to learn high Lipschitz constant functions, and this is the function we are asking the score net to learn, and b) because this quantity is involved in the reverse rollouts, having a term with high Lipschitz constant makes discretising the SDE challenging to do accurately, but this should be apparent from the abstract/introduction.

Author response:
**Q3: Could the authors highlight better which lipshitz constant is important, and why we care about it?**  
**A3:** Thank you for your valuable suggestion. We have made the necessary adjustments in the **abstract** and **introduction** by explicitly mentioning which specific Lipschitz constants we are concerned with. Additionally, we have included a concise discussion in the **introduction** to elucidate the importance of these Lipschitz constants. Our aim with these modifications is to enhance the readers' comprehension of our work and minimize any potential confusion.

Reviewer comment (Reproducibility):
The only drawback is that no open-source code is available to experiment with their approach.

Author response:
We greatly appreciate your kind words and recognition of our work. We are highly encouraged by your kind words "The authors' contribution is excellent for the community, as reducing the instabilities in the generative process, such as diffusion models, has important practical consequences." We have anonymously submitted the core part of our code as supplementary material and will make it publicly available once the paper is accepted. Thank you once again for your support.

Reviewer comment (Theory):
It may be worth showing that directly learning $\nabla \log q_{x}(t)$ with the least square is prohibitive.

Author response:
**Q1: Directly learning $\nabla \log q_t(x)$ with least square and with weighted least square.**
**A1:** In accordance with your analysis, directly learning $\nabla \log q_t(x)$ with least square presents challenges, especially with a small $\sigma_t$. We have implemented this approach and present a quantitative comparison in the table below. The experimental results affirm that directly learning $\nabla \log q_t(x)$ using the least square method is not feasible.

Moreover, we have also attempted to learn $\nabla \log q_t(x)$ using a weighted least square approach. Specifically, we aimed to learn $\theta^* = \arg\min_ {\theta} \mathbb{E}_ t \left[ \sigma_t^2\mathbb{E}_ {x}\left[ \Vert s_ \theta(x, t) - \nabla \log q_ {0t}(x|x_0) \Vert_2^2\right] \right] $. Intuitively, the weighted least square method should outperform the original least square method, as it mitigates the influence of problematic intervals. As depicted in the subsequent table, learning $\nabla \log q_t(x)$ with the weighted least square method yields significantly superior results compared to direct learning with the least square method.

**Table1. Quantitative comparison between directly learning $\nabla \log q_t(x)$ with the least square and the baseline on FFHQ $256\times 256$.**
| Method | FID-10k |
| --- | --- |
| Learning $\nabla \log q_t(x)$ with LS | 97.96|
| Learning $\nabla \log q_t(x)$ with weighted LS | 13.87 |
| Baseline | 9.50 |

Reviewer comment (Experiments):
Learning $\nabla \log q_{x}(t)$ directly with weighted least square: can we reduce the weight of the least square when $\sigma_t$ is small, e.g., learn $E \sigma_{t}^2\|\nabla \log q_{x}(t) - \frac{1}{\sigma_{t}}I\|^2$?

Author response:
**Q1: Directly learning $\nabla \log q_t(x)$ with least square and with weighted least square.**
**A1:** In accordance with your analysis, directly learning $\nabla \log q_t(x)$ with least square presents challenges, especially with a small $\sigma_t$. We have implemented this approach and present a quantitative comparison in the table below. The experimental results affirm that directly learning $\nabla \log q_t(x)$ using the least square method is not feasible.

Moreover, we have also attempted to learn $\nabla \log q_t(x)$ using a weighted least square approach. Specifically, we aimed to learn $\theta^* = \arg\min_ {\theta} \mathbb{E}_ t \left[ \sigma_t^2\mathbb{E}_ {x}\left[ \Vert s_ \theta(x, t) - \nabla \log q_ {0t}(x|x_0) \Vert_2^2\right] \right] $. Intuitively, the weighted least square method should outperform the original least square method, as it mitigates the influence of problematic intervals. As depicted in the subsequent table, learning $\nabla \log q_t(x)$ with the weighted least square method yields significantly superior results compared to direct learning with the least square method.

**Table1. Quantitative comparison between directly learning $\nabla \log q_t(x)$ with the least square and the baseline on FFHQ $256\times 256$.**
| Method | FID-10k |
| --- | --- |
| Learning $\nabla \log q_t(x)$ with LS | 97.96|
| Learning $\nabla \log q_t(x)$ with weighted LS | 13.87 |
| Baseline | 9.50 |

Reviewer comment (Theory):
For Eq (9), what if we only learn $\epsilon_{\theta}(\alpha_{f_T(t)}x_0 + \sigma_{f_T(t)}\epsilon, f_{T}(t))$, i.e., only learn the score function for time $f_{T}(t)$ and only use those time steps to do sampling?

Author response:
**Q2: What if we only learn the score function for time $f_T(t)$ and only use those time steps to do sampling.**
**A2:** This method exhibits inferior performance compared to the baseline. The experimental results for FFHQ $256\times 256$ are presented in the table below. Specifically, when $t \ge 100$, we train the score function for $t=100, 101, \dots, T-1$, and when $t < 100$, we only train the score function for $t = 0, 20, 40, 60, 80$. During inference, we exclusively employ the timesteps of $f_T(t)$ for sampling. Likewise, the baseline method also restricts sampling to the timesteps of $f_T(t)$. The experimental results, depicted in the subsequent table, demonstrate the inferior performance of this method compared to the baseline.

Although sampling in our method is limited to the timesteps of $f_T(t)$ ($t = 0, 20, 40, 60, 80$ when $t < 100$), training on additional timesteps can provide valuable information and enhance the network's denoising capabilities. By solely learning from the timesteps of $f_T(t)$, the network struggles to capture information from other timesteps, resulting in subpar performance. On the other hand, if we adopt our proposed method to train on all timesteps but limit sampling to the timesteps of $f_T(t)$ , then it becomes the situation of DDIM sampling. The outcomes of DDIM are presented in **Table 3** of our paper. However, if we extend the sampling to all timesteps, it will further enhance the performance.

**Table2. Quantitative comparison between only training for time $f_T(t)$ and the baseline on FFHQ $256\times 256$.**
| Method | FID-10k |
| --- | --- |
| Training on all $f_T(t)$, sampling on $f_T(t)$ | 48.15|
| Training on all $t$, sampling on $f_T(t)$ (Baseline) | 21.75 |

Reviewer comment (Experiments):
Is that $\sigma_{t}$ an input of the neural network: what if we learn $\epsilon_{\theta}(x, \sigma_{t})$ (the intuition is that $\sigma_{t}$ will help adjust the network Lipschize automatically).

Author response:
**Q3: What if we learn $\epsilon_\theta(x, \sigma_t)$.**
**A3:** Based on our understanding, this method aims to replace the network's conditional input $t$ with its corresponding $\sigma_t$. If there are any misunderstandings, please let us know.

**1) This method can be classified as "Remap" mentioned in our paper.**
In reality, this method can be classified as one of the alternative methods mentioned in our paper, *i.e.*, Remap (Please refer to the "Comparison with some alternative methods" part in **Section 4** and **Section D.3.3** in the appendix). The core idea behind Remap is to design a remap function $\lambda=f(t)$ as the network's conditional input instead of directly utilizing $t$, *i.e.*, $\epsilon_\theta(x_t, \lambda)$. In this particular case, we set $\lambda = \sigma_t$.

**2) Analysis of Remap, where the remap function is $\lambda = \sigma_t$**
The performance of Remap relies heavily on whether sampling is performed uniformly based on $t$ or $\lambda$ during both training and inference. **1) Uniformly sampling $t$:** As discussed in our paper, if the sampling strategy remains consistent (uniformly sampling $t$) during both training and inference, Remap has no impact on $\frac{\partial \epsilon_\theta(x_t, t)}{\partial t}$. As a result, Remap may yield similar performance to the baseline. **2) Uniformly sampling $\lambda:$** However, if there is a change in the sampling strategy (uniformly sampling $\lambda=f(t)$, i.e. $\sigma_t$ here) during training **or** inference, the inference results may deteriorate, similarly to the examples shown in our paper. This occurs because the equivalent schedule may compel the network to focus on specific stages of the entire process. For instance, taking $\lambda=\sigma_t$ as an example, where $\sigma_{999} = 1.0000$, $\sigma_{782}= 0.9990$, and $\sigma_{0}= 0.01$. If we uniformly sample $\sigma_t$ during training, timesteps falling within the range $t\in(782, 999)$ will rarely undergo training. We provide additional examples of remap functions, with a quantitative evaluation provided in **Figure 3**, along with a detailed analysis in the "Comparison with some alternative methods" section in **Section 4** and **Section D.3.3** in the appendix.

Besides, it is important to note that $\lambda=\sigma_t$ is a special remap function. The difference of $\sigma_{t}$ and $\sigma_{t-1}$ gets extremely small near $t=T$. This characteristic may hinder the network's training process, as it is hard for the network to distinguish adjacent timesteps with extremely similar conditional inputs. Consequently, learning $\epsilon_\theta(x, \sigma_t)$ may yield poorer performance than the baseline, as confirmed by the experimental results presented in the following table.

**Table3. Quantitative comparison between using $\sigma_t$ as conditions and the baseline on FFHQ $256\times 256$.**
| Method | FID-10k |
| --- | --- |
| $\epsilon_\theta(x, \sigma_t)$ | 16.41|
| Baseline | 9.50 |

Reviewer comment (Writing):
Eq 10, are $\beta_{t}$ and $\eta_{t}$ defined?

Author response:
**Q5: Eq10, are $\beta_t$ and $\eta_t$ defined?**
**A5:** Yes, thank you for pointing out this oversight. we have included their definition ($\beta_t = 1 - \frac{\alpha_t}{\alpha_{t-1}}$, and $\eta_t^2 = \beta_t$.) after Equation (10).