Paper: Latent Consistency Models: Synthesizing High-Resolution Images with Few-step Inference (ICLR-2024)

Reviewer comment (Novelty):
My major concern is about the technical novelty and contribution of this work. The paper argues there are two main contributions: (a) usage of augmented ODE and (b) skipped timestep (e.g., $k=20$) for better distillation. For (a), it seems quite straightforward to consider augmented ODE since the Stable Diffusion uses cfg. For (b), it also seems straightforward since Stable Diffusion uses a large timestep ($T=1000$), unlike EDM formulation in the original consistency model paper, and thus, one can easily expect using consecutive timesteps for distillation is inefficient. In these respects, I think both (a) and (b) looks too straightforward and marginal technical contribution to be accepted at ICLR.

Author response:
We understand the concerns raised about the perceived straightforwardness of our contribution. Still, we politely ask for a reevaluation of our paper contribution. To clarify:

1) A key difference between the original consistency models [1] and LCM lies in the transformation of the **empirical PF-ODE** (Eq.24) from the original consistency model to a **more general PF-ODE** (Eq.8). This was not addressed in the original model [1]. We also adapted the Euler/Heun solver under EDM schedule [2] to a more compatible solver for Stable Diffusion (see Eq.18 and Appendix E, including DDIM, DPM, DPM++ solvers).  Our approach expands the original model's $x$-prediction (Eq.24) to include $\epsilon$-prediction and $v$-prediction for Stable Diffusion, **offering broader parameterizations as shown in Eq. 9 and Appendix D**. These transformations and adaptations are **non-trivial and are critical** for LCM's success. However, we have weakened these difference in the main paper for readers much easier to read. 

2) Our method successfully combines latent consistency distillation (LCD), one-stage guided consistency distillation (CFG), and skipping-step techniques to achieve a highly efficient **single-stage** guided distillation for text-to-image tasks. This represents a significant advancement over previous two-stage methods like Guided-Distill [3]. With only 32 A100 GPU hours, we achieve impressive 2~4-step inference results, **demonstrating both efficiency and effectiveness**.

**Simplicity and effectiveness**, far from being limitations, are vital principles in AI research. Our work not only identifies critical components for effective LDM distillation but also achieves remarkable few-step inference results, paving the way for real-time generative AI applications. **We strongly require a reevaluation of our paper, and believe firmly that LCM have made enough contribution to this field**.

Reviewer comment (Novelty):
Title -- "Latent" Consistency model? I expected the authors to consider unique aspects of the "latent" diffusion model in incorporating the concept of consistency models; for instance, the original consistency model paper argues using perceptual metrics is efficient for better distillation. However, it seems the proposed method does not depend on whether the pretrained model is a latent diffusion model or not; it can be applied to any diffusion model that uses a large timestep (e.g., $T=1000$); thus, I think including "latent" in the title is unnecessary.

Author response:
We thank the reviewer for their attention to the choice of terminology in our paper’s title. The title ``Latent Consistency Models" was chosen deliberately to acknowledge two prior seminal papers titled “Latent Diffusion Models” (Rombach et al.) and “Consistency Models” (Song et al.) and we believe our title is an accurate reflection of the essence of our work. The choice to include “latent” in the title emphasizes the context in which the consistency model is applied. In particular, our research is primarily based on the most widely used open-sourced latent diffusion models (i.e., Stable Diffusion). We utilize the same autoencoder framework as these models, which is central to our method. Furthermore, our experiments demonstrate that our models not only produce generation quality comparable to existing latent diffusion models but also significantly expedite the sampling process. While it is true that some of the methods we proposed (e.g., the skipping-step technique) could be adapted to other diffusion models, the effectiveness and efficiency of our approach are particularly pronounced in the latent space, as demonstrated by our experiments. Moreover, we think the inclusion of the word “latent” helps delineate our work from the original Consistency Model by Song et al, thereby avoiding potential confusion. In conclusion, we think our title is carefully chosen to honor prior works while accurately representing the core innovation of our work. We hope this clarifies our rationale and justifies the choice of our title’s wording.

Reviewer comment (Presentation):
For figures for qualitative illustrations, it's better to provide text prompts to show the image-text alignment as well.

Author response:
Thanks for pointing out. We have updated the main paper. The text prompts of Figure 1 are provided in **Appendix.M**. Also we provide the LCM SD-XL [4] results in **Appendix.L (Figure.12)**.

Reviewer comment (Evaluation):
Which metric does LCD use for distillation? The original consistency model paper mainly uses the LPIPS score, but it seems such a metric is not applicable since this method deals with latent diffusion models.

Author response:
Thank you for pointing out this issue. Since we have already employed an autoencoder to encode the original images into the latent space, the LPIPS metric used in the original Consistency Models paper [1] is not appropriate in the latent space. We used the L2 distance in the latent space as our distance metric. We have updated our paper to make this information more explicitly.

Reviewer comment (Novelty):
Insufficient content for related work.

Author response:
Thanks for pointing out. We have added more related work content in the updated paper.

Reviewer comment (Experiments):
Lack of ablation studies. The authors should provide qualitative results of the ablation studies on the ODE solvers & skipping-step schedule as well as qualitative and quantitative results on guided consistency distillation.

Author response:
> 3.1 Qualitative comparison of different ODE-solvers and skipping steps

Thanks. **Figure 3 clearly demonstrates the effects of using different ODE-solvers and skipping-step schedules**, making further visualization in the main paper somewhat redundant. Nonetheless, we've included updated qualitative ablation study results on these aspects in **Appendix.K (Figure 11)**, supporting conclusions similar to those in Section 5.2 and Figure 3, particularly the importance of step-skipping for faster training and the drawbacks of excessively large skipping steps.

> 3.2 Qualitative and quantitative results on guided consistency distillation.

Thanks. Our qualitative and quantitative results on guided consistency distillation **are already presented in Figures 4 and 5**. It is important to note that using a very small Classifier-free guidance (CFG) scale is essentially equivalent to disabling CFG. From Figure 4, it is evident that a smaller CFG scale ($\omega=2$) results in a lower CLIP Score compared to a larger scale. Moreover, an increased CFG scale significantly enhances image quality, as shown by the increasing CLIP Score in Figure 4 and the visualization results in Figure 5. In conclusion, our experimental results have clearly demonstrated the effectiveness of one-stage guided consistency distillation.

Reviewer comment (Presentation):
What's the overall pipeline of the proposed method? It seems that the authors do not describe or show the pipeline in detail.

Author response:
Thanks for asking. LCM's pipeline, mainly outlined in Algorithm 1, consists of three main steps: (1) Start with any pretrained LDM; (2) Apply the latent consistency distillation (LCD) method from Algorithm 1 to convert the LDM into an LCM; (3) Employ the multi-step inference method in Algorithm 3 for fast sampling from the LCM.

Reviewer comment (Evaluation):
It is better to show the comparisons between training time and memory.

Author response:
Thank you for your suggestion. In terms of training time, both Guided Distill [1] and our LCM method are almost the same, each trained for the same number of iterations (100K, as detailed in Section 5.1) and use same batch sizes. Regarding memory usage, both methods exhibit similar requirements, with Guided Distill consuming approximately 71.1GB and LCM also using 71.3GB of memory for global batchsize=$72$ with 8 A100s. We would like to mention that LCM has potential for further training optimizations, such as mixed precision training and gradient checkpointing techniques. These considerations are beyond the current scope of our paper and left as future improvement.

References:

[1] Meng, et al. "On distillation of guided diffusion models" (2023)

Reviewer comment (Experiments):
The authors should benchmark their approach against the single-step diffusion model [InstaFlow](https://github.com/gnobitab/InstaFlow) [1], and also include results from the original 50-step Stable Diffusion as a baseline. It's currently unclear how their method's speed gains affect performance.

Author response:
### 1. About benchmarking.

> 1.1 should benchmark their approach against the single-step diffusion model InstaFlow.

Thank you for pointing out the InstaFlow paper. We acknowledge the significance of InstaFlow [1], which was released on September 12th, as indicated at https://arxiv.org/abs/2309.06380, just two weeks prior to the ICLR submission deadline. We became aware of this paper even a few days later. Due to such a short timeframe and the absence of publicly available code and pretrained models for InstaFlow, reproducing their work for a detailed comparison was not feasible within our project timeline.
Additionally, as per the guidelines provided in the ICLR FAQ section (https://iclr.cc/Conferences/2024/ReviewerGuide#FAQ), recent works published **within four months of the submission deadline are not required as baseline algorithms for comparison**. Given that InstaFlow appears to be submitted to the same ICLR24 conference, we consider it appropriate to regard it as concurrent work, aligning with standard academic practices.
We appreciate your understanding in this matter and are open to future comparisons with InstaFlow as part of our ongoing research, should the necessary resources become available.

> 1.2 Include results from the original 50-step Stable Diffusion as a baseline.

Thanks for your suggestion. We include additional **50-Step DDIM Sampler** Stable Diffusion results here and we have also added it in the caption of Table 1 and 2.

(512$\times$512 Resolution) FID: 10.74 ,  CLIP-Score: 30.34  

(768$\times$768 Resolution) FID: 12.74 ,  CLIP-Score: 30.82

> 1.3: Aboust speed gains comparison

Thanks for your suggestion. We also include the inference time in **Appendix.H (Figure.7)** for your reference. It can be noticed that LCM achieve at least **$4\times$ acceleration** copmared with DPM-Solver++.

Reviewer comment (Evaluation):
The proposed latent consistency fine-tuning seems not working, as shown in Figure 6. On the Simpsons dataset, the quality of 30K finetuning is worse than the original LCM.

Author response:
### 2. About latent consistency fine-tuning.

We appreciate your interest in our approach to latent consistency fine-tuning (LCF). Upon reflection, we recognize that the Simpsons dataset may not have been the optimal choice for demonstrating fine-tuning capabilities. This is primarily because the original Stable Diffusion (SD) model already exhibits a strong proficiency in generating images in the distinctive style characteristic of the Simpsons. To provide a more comprehensive evaluation, we have supplemented our fine-tuning results **with two additional datasets**, the details of which can be found in **Section 5.3 (Figure.6)** and **Appendix.I (Figure.8)**. Moreover, it's important to highlight that our primary contribution lies in introducing LCF as a **viable alternative for fine-tuning LCM directly** instead of teacher diffusion model, showcasing its potential effectiveness. While there is certainly room for further refinement and improvements in LCF, we believe that these aspects represent exciting avenues for future research. Our current work lays the groundwork for such explorations.

Reviewer comment (Experiments):
The paper lacks results on realistic photo generation, featuring only artistic illustrations. Including realistic photo results would strengthen the evaluation.

Author response:
### 3. Lacks results of realistic photo generation.

Thank you for your valuable observation regarding the lack of results for realistic photo generation in our paper. We agree that this aspect is important for a comprehensive evaluation of our approach.

To clarify, the style of generative images in our study is significantly influenced by the choice of the teacher diffusion model. In Figure 1, we employed Dreamshaper-V7, a popular variant of Stable Diffusion that is fine-tuned specifically for artistic styles. Additionally, our testing utilized text prompts from LAION-5B-Aesthetic6.5+, a dataset that predominantly features aesthetically oriented prompts, resulting in a natural inclination toward artistic style generation in our results.

We acknowledge that Dreamshaper-V7, being fine-tuned on artistic images, is not the ideal model for generating realistic photos. To address this, we have included results of LCM applied to **SD-XL** [2], which represents a more challenging LDM, to demonstrate our method's capability in realistic image generation. These photo-realistic generation results with 4-step inference, underscore the effectiveness of LCM. You can find these additional results in **Appendix.L (Figure.12)**. We hope this addition adequately addresses your concern and highlights the broader applicability of LCM, including in scenarios requiring high-quality, photo realistic image generation.

Reviewer comment (Novelty):
The paper's novelty appears limited. It primarily adapts consistency models to latent space and uses guided distillation in Meng et al. [2] to support Classifier-Free Guidance. While skipping time steps shows efficacy, it doesn't substantially elevate the paper's technical novelty.

Author response:
### 4. About paper novelty.
> 4.1. It primarily adapts consistency models ... elevate paper's technical novelty.

We understand the concerns raised about the perceived straightforwardness of our contributions. We acknowledge (in fact, happily) the conceptual simplicity of our work and **firmly believe that such simplicity is a strength, rather than a shortcoming**, in this important line of research with a vast and fast-growing literature and a multitude of possible technical paths that may be taken to accelerate inference. A simple and effective method offers clear advantages over more complex ones, including easier implementation, optimization and deployment. In fact, during our research, we explored several more complex techniques and methods. However, those seemingly more “fancy” approaches either yield only marginal performance gains or require substantial training (hence computationally prohibitive). Therefore, we opted for simplicity, choosing to exclude those methods in favor of a direct, efficient, and effective approach. Despite the simplicity, we believe LCM offers clear and significant contributions. **LCM is a highly efficient method that first achieves single-stage guided distillation on high resolution text-to-image generation, improving prior work by a large margin in the 2-4 step regime.** Also perhaps because of its simplicity and effectiveness, LCM has already been adopted extensively in a variety of AIGC contexts by the open-sourced community, enabling users to achieve near real-time generations (Due to the ICLR anonymity policy, we do not provide the link here).
**In conclusion, we argue that the conceptually simplicity of our contributions should not be seen as a lack of novelty, but rather a virtue.**

Reviewer comment (Theory):
Are there some challenges of applying consistency models to latent space compared to pixel space?

Author response:
> 4.2. Are there some challenges of applying ... compared to pixel space.

Thanks for your question. Although our approach is conceptually simple, we argue that it is **non-trivial**, especially given a large volume of recent works and many methods that have been proposed to accelerate inference. We have also explored
several more complex (and seemingly more novel) techniques and methods but those yield only marginal performance gains. Nevertheless, let us highlight some of the technical challenges: 

(1). We **transform** the empirical PF-ODE (as per Equation 24) and its corresponding solver (as per Equation 7) in the original consistency model paper [1] to a **more general form** that can be made compatible with Stable Diffusion (PF-ODE, as illustrated in Equation 8, and its corresponding solver, as in Equation 18). We also **reformulate the parameterization** shown in Eq. 9 and Appendix D. These transformation and reformulation are **non-trivial and a crucial factor** for the successful implementation of LCM. 

(2). The introduction of latent consistency distillation (LCD), single-stage guided distillation (Classifier-Free Guidance or CFG), and the skipping-step techniques are also **critical elements for the success of LCM**. The single-stage guided distillation has a significant impact for improving image quality (see Figure 4, 5). While the skipping-step techniques accelerate the convergence for at least $6\times$, which is shown in Figure.3. 

(3). Last but not least, we would like to mention that the engineering and system endeavors are also nontrivial. The task of integration latent consistency models and Stable Diffusion required in-depth understanding and engineering with the Stable Diffusion framework. Our careful engineering has results in a robust system that not only achieves state-of-the-art performance in research, while also delivering the practical effectiveness immediately ready for real-world applications.

In sum, we had to overcome several technical challenges to deliver the impressive experimental results and efficiency achieved by LCM. **We believe that our efforts are nontrivial and our contributions are significant and noteworthy.**

Reviewer comment (Reproducibility):
No corresponding text prompts in the visual results (e.g., Figure 1).

Author response:
> 5.1. No corresponding text prompts in the visual results (e.g., Figure 1)

Thanks for pointing out. We have updated the main paper. The text prompts of Figure 1 are provided in **Appendix.M**. Also we provide the LCM SD-XL [2] results in **Appendix.L (Figure.12)**.

Reviewer comment (Writing):
Typo. Section 3 Preliminaries -- Diffusion Models: "origin data distribution" -> "original data distribution"

Author response:
> 5.2. Typo

Thanks. We have fixed the typo in the updated main paper.

Reviewer comment (Experiments):
Although the authors claim several contributions, it is not clear which ones have the most significant impact on efficiency and quality.

Author response:
Thanks for your question. Our skipping-step technique is the most critical component for speeding up the convergence (i.e., efficiency) and the one-stage guided consistency distillation algorithm has a significant impact on the quality of the generated images. The detailed ablation ablation study of skipping-step technique and one-stage guided consistency distillation can be found in Section 5.2. We briefly summarize the impact of skipping-step technique and one-stage guided consistency distillation below:

**Skipping-Step Technique**: This method notably enhances convergence speed, a crucial aspect of efficiency. As quantitatively demonstrated in Figure 3 (DDIM sampler), the model's convergence speed with k=1 is significantly slower compared to other settings from k=5 to k=20. In specific, it shows that at 4000 training steps, the model with k=1 has a slower convergence, evidenced by a FID of 26.8, compared to k=20 with a significantly better FID of 13.3. Moreover, at k=20, the model reaches a FID of 14.4 in just 2000 steps, matching k=1 performance at 12000 steps, achieving at least **$6\times$** convergence speed, strongly demonstrating the effectiveness of the skipping-step method.

**One-stage guided consistency distillation**: Incorporating Classifier-Free Guidance (CFG) into the LCM significantly enhances image quality. Figures 4 and 5 illustrate that a larger CFG scale produces more realistic and higher-quality images. However, a very small CFG scale (which is equivalent to not using guided distillation), particularly at 2, results in lower quality generation. This enhancement in quality is validated by the CLIP scores in Figure 4 and the visibly superior image quality showcased in Figure 5.

Reviewer comment (Theory):
What is the computational complexity of solving augmented PF-ODE?

Author response:
Thanks. We would like to clarify that we first view the guided reverse diffusion process as solving an augmented probability flow ODE (Augmented PF-ODE), then  LCM is designed to effectively to solve this PF-ODE during inference stage. The computational complexity of solving this Augmented PF-ODE is essentially \textbf{equivalent} to (Number of inference steps, also named the number of function evaluation (NFE)) $\times$ (Computational cost for each step). Table 1 and 2 show the superiority of LCM on solving the augmented PF-ODE in a few inference steps (1$\sim$8). Additionally, since the classifier-free guidance (CFG) has already been distilled into LCM, we do not need to calculate the unconditional term $\epsilon_\theta(z_t,t,\varnothing)$ during each inference steps, which further reduce the computational cost in each step, resulting in significantly faster speeds and reduced GPU memory usage during inference. **We also include the inference time in **Appendix. H (Figure.7)** for your reference**. We achieve at least **$4\times$** acceleration compared with DPM-Solver++.

Reviewer comment (Evaluation):
The experiments show that the proposed method reduces the inference steps, however, how much faster is the inference time exactly compared with other methods?

Author response:
Thanks for your suggestion. We report the inference speed/FID performance in **Appendix. H (Figure.7)**. It can be seen that we achieve at least **4$\times$ acceleration** compared with DPM-Solver++.

Reviewer comment (Experiments):
What about the performance when the proposed LCMs are applied to other LDMs besides Stable Diffusion?

Author response:
Thanks for raising this point. We would like to emphasize that Stable Diffusion, recognized as one of the most advanced and extensively used LDMs in the AI researchers and artistic communities, serves as a challenging benchmark. The successful application of LCM to Stable Diffusion has already demonstrated our method's effectiveness and potential. Having established its performance on such challenging task, we believe additional validation on simpler LDM is not essential. To further highlight LCM's promise and superiority, we also undertook a more challenging experiment applying LCM to the **SD-XL** [1], which has three times as many parameters and a resolution of 1024x1024, compared to the original SD model. Detailed generation results in **Appendix.L (Figure. 12)** further demonstrate LCM's potential and impressive performance.

Reference: 
[1] Podel, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis". arXiv preprint arXiv:2307.01952 (2023).

Reviewer comment (Experiments):
It is not clear which contributions have the most significant impact on efficiency and quality.

Author response:
Thanks for your question. Our skipping-step technique is the most critical component for speeding up the convergence (i.e., efficiency) and the one-stage guided consistency distillation algorithm has a significant impact on the quality of the generated images. The detailed ablation ablation study of skipping-step technique and one-stage guided consistency distillation can be found in Section 5.2. We briefly summarize the impact of skipping-step technique and one-stage guided consistency distillation below:

**Skipping-Step Technique**: This method notably enhances convergence speed, a crucial aspect of efficiency. As quantitatively demonstrated in Figure 3 (DDIM sampler), the model's convergence speed with k=1 is significantly slower compared to other settings from k=5 to k=20. In specific, it shows that at 4000 training steps, the model with k=1 has a slower convergence, evidenced by a FID of 26.8, compared to k=20 with a significantly better FID of 13.3. Moreover, at k=20, the model reaches a FID of 14.4 in just 2000 steps, matching k=1 performance at 12000 steps, achieving at least **$6\times$** convergence speed, strongly demonstrating the effectiveness of the skipping-step method.

**One-stage guided consistency distillation**: Incorporating Classifier-Free Guidance (CFG) into the LCM significantly enhances image quality. Figures 4 and 5 illustrate that a larger CFG scale produces more realistic and higher-quality images. However, a very small CFG scale (which is equivalent to not using guided distillation), particularly at 2, results in lower quality generation. This enhancement in quality is validated by the CLIP scores in Figure 4 and the visibly superior image quality showcased in Figure 5.

Reviewer comment (Theory):
What is the computational complexity of solving augmented PF-ODE?

Author response:
Thanks. We would like to clarify that we first view the guided reverse diffusion process as solving an augmented probability flow ODE (Augmented PF-ODE), then  LCM is designed to effectively to solve this PF-ODE during inference stage. The computational complexity of solving this Augmented PF-ODE is essentially \textbf{equivalent} to (Number of inference steps, also named the number of function evaluation (NFE)) $\times$ (Computational cost for each step). Table 1 and 2 show the superiority of LCM on solving the augmented PF-ODE in a few inference steps (1$\sim$8). Additionally, since the classifier-free guidance (CFG) has already been distilled into LCM, we do not need to calculate the unconditional term $\epsilon_\theta(z_t,t,\varnothing)$ during each inference steps, which further reduce the computational cost in each step, resulting in significantly faster speeds and reduced GPU memory usage during inference. **We also include the inference time in **Appendix. H (Figure.7)** for your reference**. We achieve at least **$4\times$** acceleration compared with DPM-Solver++.

Reviewer comment (Evaluation):
How much faster is the inference time exactly compared with other methods?

Author response:
Thanks for your suggestion. We report the inference speed/FID performance in **Appendix. H (Figure.7)**. It can be seen that we achieve at least **4$\times$ acceleration** compared with DPM-Solver++.

Reviewer comment (Experiments):
What about the performance when the proposed LCMs are applied to other LDMs besides Stable Diffusion?

Author response:
Thanks for raising this point. We would like to emphasize that Stable Diffusion, recognized as one of the most advanced and extensively used LDMs in the AI researchers and artistic communities, serves as a challenging benchmark. The successful application of LCM to Stable Diffusion has already demonstrated our method's effectiveness and potential. Having established its performance on such challenging task, we believe additional validation on simpler LDM is not essential. To further highlight LCM's promise and superiority, we also undertook a more challenging experiment applying LCM to the **SD-XL** [1], which has three times as many parameters and a resolution of 1024x1024, compared to the original SD model. Detailed generation results in **Appendix.L (Figure. 12)** further demonstrate LCM's potential and impressive performance.

Reference: 
[1] Podel, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis". arXiv preprint arXiv:2307.01952 (2023).