Paper: A Unified Framework for Consistency Generative Modeling (ICLR-2024)

Reviewer comment (Presentation):
The final algorithm is not very clear to me by reading the main paper, there are several tricks proposed. It would be great to have the algorithm written out in the main paper rather than in the appendix.

Author response:
**A:** We appreciate your thoughtful feedback and recognize the importance of providing a clear understanding of our proposed algorithm. Considering the constraints of page limitations, we regret that we are unable to present the complete training and sampling algorithm in the main text.

To address this concern and enhance the accessibility of our work, we will incorporate a jump link in the main paper, directing readers to the corresponding pseudocode in the appendix. This approach will enable readers to seamlessly navigate between the main text and the detailed algorithmic representations for each model, ensuring a more comprehensive understanding of the proposed methodology.

We look forward to your continued insights and hope that this adjustment enhances the clarity and accessibility of our work.

Reviewer comment (Writing):
the notation is a little confusing, which makes the paper hard to follow.

Author response:
**A:** I apologize for any confusion caused by our current notation choices. In the revised version, we are actively working to enhance the use of symbols for a clearer and more easily understandable presentation. If you have specific suggestions or examples where the notation can be improved, please feel free to provide them, as your input is invaluable in refining our paper for better comprehension.

Reviewer comment (Presentation):
The reverse diffusion process with gaussian noise can reconstruct the original clear image from the noisy version step by step, it would be better to show some figures using PCMs and CCMs.

Author response:
**A:** Thank you for raising this point. While our consistency modeling framework shares similarities with the diffusion model, it distinguishes itself by directly recovering the image from noise through a single-step mapping, bypassing intermediate stages.

Reviewer comment (Novelty):
**Question mark on novelty:** the idea of writing continuity equation and probability path is not new, and many derivations from the generalized CM in this paper are followed exactly from [Lipman et al 2023] on flow matching generative models. The formulation of Couping Consistency Model (CCM) -- section 3.4, which is this work's most well-performed framework empirically, the authors again borrowed heavily from previous work of [Lipman et al. 2023] with the linear path $x_t = tx_1 + (1-t)x_0$. Equation (20) and the idea of CCM-OT on learning probability path with joint distribution to reduce training loss variance is straightforward taken from Multisample Flow matching paper [Pooladian et al. 2023, Section 3], but surprisingly there was no mention of this citation around this equation.

Author response:
**Q: Many derivations from the generalized CM in this paper are followed exactly from [1] on flow matching generative models. The formulation of Couping Consistency Model (CCM) -- section 3.4, which is this work's most well-performed framework empirically, the authors again borrowed heavily from previous work of [1] with the linear path .**

**A:** Thank you for highlighting this observation. In Section 3.1, we explicitly acknowledge that our approach to constructing probabilistic paths draws inspiration from recent advances in Flow-based models [1]. It is essential to recognize, however, that our primary focus diverges significantly.

Firstly, our major contribution lies in unveiling the intrinsic relationship between probabilistic paths and consistency models (CMs). This unique perspective enables us to approach consistency training from a foundational angle, potentially catalyzing the development of innovative single-step generative models. The introduction of the linear path into consistency models facilitates a distinct single-step Im2Im translation. In contrast, [1] typically employs around 100~200 steps, underscoring the non-trivial nature of our contribution.

Secondly, it's imperative to emphasize that the conclusions in [1] are limited to Gaussian paths. In our work, we generalize these conclusions to a broader form, a crucial step for incorporating Poisson Consistency Models (PCMs) into our proposed framework. This generalization significantly enhances the adaptability and applicability of our approach beyond the constraints of Gaussian paths established by [1].
******************************************************

**Q:Equation (20) and the idea of CCM-OT on learning probability path with joint distribution to reduce training loss variance is straightforward taken from Multisample Flow matching paper [2]**

**A:** We appreciate your insightful observation and acknowledge the connection between our CCM-OT and optimal transport flow matching in [2]. In our related section (Section 4), we have cited and expounded upon the relevance of their work, underscoring the differences in our motivation.

While [2] demonstrates the efficacy of coupling distributions in reducing the variance of the flow matching target, we place a specific emphasis on refining the estimation of vector fields within our proposed framework, constituting a unique contribution in the realm of CMs. To the best of our knowledge, there exists no prior work proposing the link of optimal transport and CMs. Consequently, our work fills a significant gap in this domain, introducing a novel application of optimal transport principles to elevate the performance of CMs.

We are grateful for your suggestion, and as you recommended, we will refer back to the work [2] in the corresponding section to ensure that readers gain a comprehensive understanding of our contributions.
***********************************************

*[1] Lipman Y, Chen R T Q, Ben-Hamu H, et al. Flow matching for generative modeling[J]. arXiv preprint arXiv:2210.02747, 2022.*


*[2] Pooladian A A, Ben-Hamu H, Domingo-Enrich C, et al. Multisample flow matching: Straightening flows with minibatch couplings[J]. arXiv preprint arXiv:2304.14772, 2023.*

Reviewer comment (Evaluation):
**Question mark on the results of empirical evaluation:** although the results provided with 1 NFE sampling for CIFAR10 in table 2 and 3 showed that CCM-OT outperformed original CM (denoted DCM in this paper) of [Song et al. 2023], I do not understand where the authors got the FID score of 18.4 for DCM to begin with. In [Song et al. 2023], it is clearly stated that their model reached 1-NFE FID 3.55 & 8.70 with distillation and without distillation, respectively. Therefore, I urge the authors provide a clear explanation on the discrepancy between the reported numbers of the baseline. Moreover, DCM also provided more extensive experiments with ImageNet 64x64 and LSUN Bedroom 256x256, and although I understand the lack of computational resources, I advise the authors to provide results on more datasets for better understanding the gains of their proposed methods compared with baselines, which should also included distillation techniques such as Progressive distillation [Salisman & Ho 2022].

Author response:
**Q:I urge the authors provide a clear explanation on the discrepancy between the reported numbers of the baseline.**

**A:** Thank you for bringing attention to the differences in our reported metrics compared to [1]. Specifically, we meticulously replicated the work of [1], maintaining consistency in **time discretization, network design, learning rate, moving average rate, loss computation, and other key aspects**. However, due to computational constraints, we made necessary adjustments, including reducing the batch size from 512 to 256, and the number of iterations from 800K to 400K. Additionally, we implemented mixed-precision training for further acceleration. 

Despite these modifications, we want to emphasize our comparisons are fair. All models were configured with identical experimental hyperparameters. We believe that the primary source of the observed discrepancies may be attributed to the use of a smaller batchsize in our implementation. We will further elucidate our experimental setup and discrepancy compared with [1] in the revised version.
****************************************************

**Q:I advise the authors to provide results on more datasets for better understanding the gains of their proposed methods compared with baselines.**

**A:** We appreciate your valuable feedback and fully acknowledge the merit of conducting evaluations on a broader set of datasets. However, it's worth noting that datasets like Imagnet and LSUN Bedroom, used by [1], involved training a consistency model on 64 A100 GPUs clusters—such computational resources are beyond our current capacity.

Despite these constraints, we have extended our evaluation to include another widely recognized image-generated benchmark dataset, CelebA. All methods were trained with a batch size of 64 and updated for 200K iterations and the results are as follows,

| method | | FID | |
|  ----  | ----  | ----  | ----  |
| | NFE=1|NFE=2|NFE=5|
| DCM | 41.0 | 27.3 | 25.3 |
| DCM-MS | 38.3 | 22.2 | 18.2|
| PCM | 32.9 | **17.8** | **16.1** |
| CCM-OT | **30.8** | 28.0| 25.5|

We observed consistent improvements with our proposed method, reinforcing its efficacy. We understand the importance of comprehensive evaluations and hope that these additional results contribute to a better understanding of the strengths of our approach.
*****************************************************************
**Q:Comparison with distillation technology such as Progressive distillation [2]**

**A:** We appreciate the suggestion to compare our approach with distillation technologies like Progressive Distillation [4]. However, it's important to note that our primary focus in this paper is on the consistency training introduced by [1]. Our objective is to achieve a single-step generative model directly, without relying on pre-trained score networks.

Unlike distillation approaches based on score networks, our consistency framework can be positioned as an independent family of generative models that are computationally resource-friendly. 
****************************************************
*[1] Song Y, Dhariwal P, Chen M, et al. Consistency models[J]. 2023.*

*[2] Salimans T, Ho J. Progressive distillation for fast sampling of diffusion models[J]. arXiv preprint arXiv:2202.00512, 2022.*

Reviewer comment (Experiments):
The evaluation is weak. The paper only conducts quantitative comparisons on a toy dataset and CIFAR-10.

Author response:
**A:** We appreciate your pointer and acknowledge the need for a more evaluation. In response, we will include another benchmark dataset, Celeba64 $ \times $64, in the revised version. All methods were trained with a batch size of 64 and updated for 200K iterations, yielding the following results:

| method | | FID | |
|  ----  | ----  | ----  | ----  |
| | NFE=1|NFE=2|NFE=5|
| DCM | 41.0 | 27.3 | 25.3 |
| DCM-MS | 38.3 | 22.2 | 18.2|
| PCM | 32.9 | **17.8** | **16.1** |
| CCM-OT | **30.8** | 28.0| 25.5|

The result further highlights the advantages of our proposed algorithm over the original DCM, affirming the effectiveness of our framework. We commit to promptly include these tables and visualizations in the revised version.

It is crucial to emphasize that our work extends the modeling capabilities of consistency models (CMs) [1]. In addition to unconditional generation, we introduced a unique single-step unpaired im2im experiment on AFHQ, addressing limitations in previous CMs related to the diffusion process.

Reviewer comment (Experiments):
As in the Consistency Models paper, the consistency distillation method generally performs better than consistency training. However, the paper does not provide any results or comparisons for this.

Author response:
**A:** Thank you for bringing up this point. Our framework is designed to establish a self-sufficient and self-reliant approach for consistency generative modeling, without dependence on pre-trained score networks.

While we do mention consistency distillation in our work, it was not the primary focus for several reasons:

1) Incorporating consistency distillation necessitates an additional pre-training step, introducing computational overhead and somewhat deviating from our original design philosophy for this framework. 

2) The effectiveness of consistency distillation is inherently depend on the quality of the pre-trained model used. Concurrent research [2] has shown that independent consistency modeling can achieve significant improvements through various advanced training tricks, potentially surpassing the performance of consistency distillation. We believe our framework can also benefit from these strategies.

Reviewer comment (Novelty):
The key contribution that goes beyond the gaussian distribution by replacing the gaussian distribution with either poisson distribution or a joint distribution seems to not specific to the training of consistency models but also the vanilla diffusion models. Could the authors clarify more for on this?

Author response:
**A:** I appreciate the reviewer's thoughtful question. Our work indeed is inspired by prior efforts that have explored modifications to the Gaussian distribution in vanilla diffusion models, such as the Poisson flow [3] and Flow matching [4]. However, it is crucial to highlight that our contribution extends beyond a simple replacement of distributions.

Our distinctive contribution lies in establishing a profound connection between the self-consistency property inherent in Consistency Models (CMs) and the probabilistic paths. This profound connection not only enhances our understanding of CMs but also facilitates the design of more effective single-step generative models.