Paper: The Emergence of Reproducibility and Consistency in Diffusion Models (ICLR-2024)

Reviewer comment (Novelty):
Lack of discussion about text-to-image diffusion models.

Author response:
We thank the reviewer for the valuable suggestions to study the phenomenon more comprehensively. During the revision, we conducted more experiments considering two extra cases: (i) the Conditional Diffusion Model on the ImageNet Dataset [1], and (ii) Stable Diffusion [2] on the LAION-5B dataset [3]. The results can be found in Appendix A.1 and A.2 of the revised paper. As such, our new experiments not only covered larger datasets in (i, ii), but also text-to-image diffusion models in (ii). Specifically, we find the following:

1. Conditional Diffusion Model on the ImageNet Dataset: We evaluated the reproducibility of EDM [4] and ADM [5], as shown in the following table:

	| $\text{RP} \text{ Score}$| EDM  | ADM  |
	|----------|------|------|
	| **EDM**      | 1.0  | 0.81 |
	| **ADM**      | 0.81 | 1.0  |

	Detailed experimental settings and visual results are presented in Appendix A.1. We find that the model reproducibility still exists on ImageNet.

2. Stable Diffusion on the LAION-5B Dataset: We evaluated the reproducibility of stable diffusion models with versions v1-1 to v1-4, with their respective reproducibility scores shown in the table below:

	| $\text{RP} \text{ Score}$ | V1-1 | V1-2 | V1-3 | V1-4 |
	|----------|------|------|------|------|
	| **V1-1**     | 1.0  | 0.10 | 0.03 | 0.03 |
	| **V1-2**     | 0.10 | 1.0  | 0.21 | 0.20 |
	| **V1-3**     | 0.03 | 0.21 | 1.0  | **0.63** |
	| **V1-4**     | 0.03 | 0.20 | **0.63** | 1.0  |
	
	The relationship between V1-1 to V1-4 could be summarized as 
	- Versions v1-1, v1-2, and v1-3 each are trained on different subsets of the LAION-5B dataset.
	- Versions v1-3 and v1-4 share the same training subset from LAION-5B.
	- Version v1-2 is resumed from v1-1, while v1-3 and v1-4 are resumed from v1-2.
	
	Notably, only versions v1-3 and v1-4 were trained on the exact same dataset, resulting in the highest reproducibility scores (0.63). Lesser but noticeable reproducibility scores (below 0.21) are observed among v1-1, v1-2, and v1-3, this can be attributed to their sequential training and overlapping datasets. There is no model reproducibility between v1-1 and others, because the training datasets are nonoverlapping and different. Further details on the experimental setup and visual results can be found in Appendix A.2.

In conclusion, our findings suggest that this phenomenon persists for larger diffusion models trained on large datasets.

[1]Deng, Jia, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. "Imagenet: A large-scale hierarchical image database." In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.

[2]Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695. 2022.

[3] Schuhmann, Christoph, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35 (2022): 25278-25294.

[4] Karras, Tero, Miika Aittala, Timo Aila, and Samuli Laine. "Elucidating the design space of diffusion-based generative models." _Advances in Neural Information Processing Systems_ 35 (2022): 26565-26577.

[5] Dhariwal, Prafulla, and Alexander Nichol. "Diffusion models beat gans on image synthesis." _Advances in neural information processing systems_ 34 (2021): 8780-8794.

Reviewer comment (Evaluation):
Why are the reproducibility scores between transformer-based models and unet-based models low as shown in Figure 7(b)? According to Theorem 1, the reproducibility scores should be high. Could you please provide more explanation?

Author response:
**A2:** Thanks the reviewer for this very good question. When solving inverse problems with diffusion models, our experiments in Figure 7 showed that the model reproducibility between different type of network architectures are very low. We use Diffusion Posterior Sampling (DPS) [1] to solve the inpainting problem, with details provided in Appendix F. We conjecture that the lack of reproducibility across network architectures is due to the following reasons:
1. Here, DPS introduces the gradient term $\dfrac{\partial \epsilon_{\theta} ( x_t, t)}{\partial x_t}$ during the sampling, and this extra term might break the reproducibility for different type of architectures. In contrast, for the unconditional diffusion model, the reproducibility holds as long as the denoiser function $\epsilon_{\theta} ( x_t, t)$ is reproducible.
2. Second, for image inpainting or inverse problem solving in general, the data $ x_t$ passed into the denoiser  $\epsilon_{\theta}( x_t, t)$ is out-of-distribution. As such, the reproducibility between different types of architectures might not hold for out-of-distribution data generation.
    
We will incorporate this discussion into the revised paper in Section 4 (when the extra page is provided).

[1] Chung, Hyungjin, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. "Diffusion posterior sampling for general noisy inverse problems." arXiv preprint arXiv:2209.14687 (2022).

Reviewer comment (Experiments):
The present study is mainly on CIFAR-10, and it would be interesting to see how do the results transfer to larger data sets, like ImageNet. Especially when moving to the very large data sets used for text-conditional generation, it doesn't seem obvious that similar results would hold. It would be interesting to see how far can this be extended.

Author response:
We very much appreciate the reviewer for acknowledging the quality of our presentation, experiments, and the timeliness of our results. We also thank the reviewer for carefully reading our paper and detailed comments. We value the constructive feedback and encouragement that the reviewer provided. In the following, we carefully respond to each of the concerns and suggestions that the reviewer raised.

___

> **Q1**: “The present study is mainly on CIFAR-10, and it would be interesting to see how do the results transfer to larger data sets, like ImageNet. Especially when moving to the very large data sets used for text-conditional generation.”

**A1**:
We thank the reviewer for the valuable suggestions to study the phenomenon more comprehensively. During the revision, we conducted more experiments considering two extra cases: (i) the Conditional Diffusion Model on the ImageNet Dataset [1], and (ii) Stable Diffusion [2] on the LAION-5B dataset [3]. The results can be found in Appendix A.1 and A.2 of the revised paper. As such, our new experiments not only covered larger datasets in (i, ii), but also text-to-image diffusion models in (ii). Specifically, we find the following:

1. Conditional Diffusion Model on the ImageNet Dataset: We evaluated the reproducibility of EDM [4] and ADM [5], as shown in the following table:

	|$\text{RP} \text{Score}$|EDM|ADM|
	|------|------|------|
	| **EDM**      | 1.0  | 0.81 |
	| **ADM**      | 0.81 | 1.0  |

	Detailed experimental settings and visual results are presented in Appendix A.1. We find that the model reproducibility still exists on ImageNet.

2. Stable Diffusion on the LAION-5B Dataset: We evaluated the reproducibility of stable diffusion models with versions v1-1 to v1-4, with their respective reproducibility scores shown in the table below:

	| $\text{RP} \text{ Score}$ | V1-1 | V1-2 | V1-3 | V1-4 |
	|----------|------|------|------|------|
	| **V1-1**     | 1.0  | 0.10 | 0.03 | 0.03 |
	| **V1-2**     | 0.10 | 1.0  | 0.21 | 0.20 |
	| **V1-3**     | 0.03 | 0.21 | 1.0  | **0.63** |
	| **V1-4**     | 0.03 | 0.20 | **0.63** | 1.0  |
	
	The relationship between V1-1 to V1-4 could be summarized as 
	- Versions v1-1, v1-2, and v1-3 each are trained on different subsets of the LAION-5B dataset.
	- Versions v1-3 and v1-4 share the same training subset from LAION-5B.
	- Version v1-2 is resumed from v1-1, while v1-3 and v1-4 are resumed from v1-2.
	
	Notably, only versions v1-3 and v1-4 were trained on the exact same dataset, resulting in the highest reproducibility scores (0.63). Lesser but noticeable reproducibility scores (below 0.21) are observed among v1-1, v1-2, and v1-3, this can be attributed to their sequential training and overlapping datasets. There is no model reproducibility between v1-1 and others, because the training datasets are nonoverlapping and different. Further details on the experimental setup and visual results can be found in Appendix  A.2.

In conclusion, our findings suggest that this phenomenon persists for larger diffusion models trained on large datasets.

[1]Deng, Jia, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. "Imagenet: A large-scale hierarchical image database." In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.

[2]Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695. 2022.

[3] Schuhmann, Christoph, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35 (2022): 25278-25294.

[4] Karras, Tero, Miika Aittala, Timo Aila, and Samuli Laine. "Elucidating the design space of diffusion-based generative models." _Advances in Neural Information Processing Systems_ 35 (2022): 26565-26577.

[5] Dhariwal, Prafulla, and Alexander Nichol. "Diffusion models beat gans on image synthesis." _Advances in neural information processing systems_ 34 (2021): 8780-8794.

Reviewer comment (Experiments):
When studying the encoding from the noise hyperplane to the image manifold, would it be more appropriate to interpolate the noises using spherical linear interpolation instead of direct linear interpolation? From what I understand, the issue with linear interpolation is that because two randomly sampled high-dimensional noise vectors are, with a high probability, orthogonal to each other, the magnitude of a linearly interpolated vector decreases halfway by a factor of sqrt(2).

Author response:
> **Q2**: ”When studying the encoding from the noise hyperplane to the image manifold, would it be more appropriate to interpolate the noises using spherical linear interpolation instead of direct linear interpolation?”

 **A2**:
Thanks for your very insightful suggestion. Given the spherical linear interpolation is confined to a single dimension and time limit, we opted not to visualize the manifold reproducibility in a figure. Instead, we developed a manifold reproducibility score to measure the manifold reproducibility. The metric is defined as follows:

The metric begins by selecting two initial noise vectors, $(\mathbf{\epsilon}^{(0)}, \mathbf{\epsilon}^{(1)})$, from the noise space $\mathcal E$. These vectors are then processed through two distinct diffusion model architectures, resulting in pairs of clear images: ($\mathbf x^{(0)}_1$, $\mathbf x^{(1)}_1$) from the first model and ($\mathbf x^{(0)}_2$, $\mathbf x^{(1)}_2$) from the second. The manifold reproducibility score for the two unconditional diffusion models is defined as:


$RP_{manifold} Score :=  \mathbb{P}( \mathcal M_{SSCD} ( x^{(\alpha)}_1,  x^{(\alpha)}_2)>0.6 \mid \alpha \in [0,1] )$

where $\mathbf x^{(\alpha)}_i$ are generated from spherical linear interpolation based on ($\mathbf x^{(0)}_i$, $\mathbf x^{(1)}_i$):
$$
\begin{align*}
    & \mathbf x^{(\alpha)}_i = \frac{\text{sin}((1 - \alpha) \theta)}{\text{sin}(\theta) } \mathbf x^{(0)}_i + \frac{\text{sin}(\alpha \theta)}{\theta} \mathbf x^{(1)}_i, \ \ i \in \{1, 2\} \\
    & \theta = \text{arccos} \left(\frac{ \left(x^{(0)}_i\right )^T x^{(1)}_i}{||x^{(0)}_i|| ||x^{(1)}_i||} \right)
\end{align*}
$$

We applied this score to compare the manifold reproducibility across various unconditional diffusion models.  Results are shown below. Detailed descriptions of the experimental settings can be found in Appendix A.3. In this case, our observation of model reproducibility is still consistent with our current findings.

| $\text{RP}_{manifold} \text{ Score}$   | DDPMv6 | EDMv1 | Multistagev1 |
|--------------|--------|-------|--------------|
| **DDPMv6**       | 1.0    | 0.90  | 1.0          |
| **EDMv1**        | 0.90   | 1.0   | 0.93         |
| **Multistagev1** | 1.0    | 0.93  | 1.0          |