From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The paper proposed a novel conditional contrastive diffusion approach for better learning the statistical relationships between two modalities via the maximal mutual information. A contrastive regularizer is inspired by a contrastive loss where the numerator measures the mutual information between the condition and latent embeddings. The paper presented experimental results on diverse setups: dance-to-music generation, text-to-image synthesis and class-conditioned image synthesis, showing improvements over the baselines.

Overall, all the reviewers are positive about the paper, acknowledging the novelty of introducing contrastive learning in diffusion models. However, each reviewer shared concerns. R-uFmy mentioned the range of $\lambda$, whether negative samples would affect the training time, and Stable Diffusion should be benchmarked against. The author’s response has mostly addressed R-uFmy’s concerns, while R-uFmy still wants the authors to compare against Stable Diffusion because of the common nature of latent space, and has doubts whether the proposed method would require fewer steps. Consequently, R-uFmy raised the score to 6.

R-MzmQ raised the concerns of lack of evidence to support the claim that incorporating prior into the variational lower bound can lead to the loss of the cross-modal correspondence (an excellent outer-alignment question), how to balance the variational loss and contrastive loss (they seem to contradict), and other questions on the results and comparisons. Most of the issues were addressed in the rebuttal, and R-MzmQ also raised the score to 6.

R-3LEn raised the question of comparing the proposed CDCD with DALLE-2 and Imagen. However, it’s a bit unfair for a paper to compete against large-scale foundation models trained with a lot more resources.

Overall, the AC feels that the proposed idea has novelty and is backed up by solid experiments. The diverse experiments suggest that the proposed contrastive diffusion regularizers may have broader impact on the community. 

justification_for_why_not_higher_score: While it's difficult to conduct comparisons with much larger-scale models, it's unclear whether the proposed ideas would have the same impact at much larger scale. 
justification_for_why_not_lower_score: All the reviewers are positive about the paper, and two of them are very positive. R-6GAW said "I didn't see any weaknesses" yet only gave 6. 
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper1428/Reviewer_uFmy
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper presents a regularizer that enforces that the condition embedding and the latent embedding of the input datum share the maximal mutual information between them. This regularizer then helps the diffusion process because the condition is rich with information about the latent embeddings, and thus helping the denoising process. The regularizer term is inspired by a contrastive loss in which the numerator indirectly measures the mutual information between the condition and latent embeddings, and the denominator includes information from conditions and embeddings that should not be related to each other (i.e., mimicking negative samples). The paper also discusses two ways of applying this regularizer: step-size parallel diffusion and sample-wise auxiliary diffusion. The former allows parallel evaluations of the denoising diffusion process while the latter does not. The paper presents experiments on dance-to-music generation, text-to-image synthesis, and class-conditioned image synthesis showing improvements over the included baselines.
strength_and_weaknesses: Strengths:

I find the idea of using negative samples as part of the contrastive-loss-based regularizer to maximize the mutual information between the condition and latent interesting and novel. I think using these negative samples can indeed inform the model about what not to generate and constraint better the data generation.

Weaknesses:
1. While I find the idea novel, I think the method is quite elaborate and can imply more computational resources. 
- First, including negative samples as part of the loss will increase computation making the computational cost even more expensive for a denoising diffusion process. 
- Second, while the proposed stepwise diffusion allows parallelization, it still requires more resources that can increase the cost of an already expensive denoising diffusion process.

2. Insufficient experiments:
- The paper lacks an ablation study about the parameter $\lambda$ which controls the contribution to the total loss of the proposed regularizer. According to Section 4.1, $\lambda$ was set to 5e-5, which I find the value too low. It is unclear how to set this parameter from the experiments. More importantly, what the impact of increasing the value of $\lambda$ and thus enforcing the regularizer stronger on performance is not clear.
- The paper also misses an ablation study about the latent encoder. What is the effect of not even using one? Wouldn’t a latent encoder likely reduce the information (in the information theoretical sense) from the original input. Can the proposed methods work on raw signals, i.e., latents are the input signals directly.
- The experiments in paragraph “Results and Discussion” and Fig. 4 state that because the proposed method requires fewer steps to converge the proposed method is faster to converge. While the experiments show a reduction in steps, it is unclear about the cost of each step in terms of time in the proposed method. I think having a paper demonstrating that the proposed method indeed reduces the time of convergence is more important than the number of steps.
- The experiment in Table 3 is missing a more appropriate baseline: Stable Diffusion. I think using Stable Diffusion instead of DALLE makes more sense because Stable Diffusion also uses a latent representation while DALLE does not.

3. From the theoretical perspective, is there a proof showing that the proposed regularizer combined w/ the variational-bound-based loss still preserves the Langevin dynamics in some way? I think discussing the theoretical guarantees can be informative.

================= Post-Discussion =================

After engaging with the authors in the discussion, I still think the paper can benefit from reporting wall-clock time of the training phase, add more extensive ablation studies, and add Stable Diffusion as a baseline. For the most part, most of my concerns about clarity were addressed. Nevertheless, because I think there are missing experiments, I cannot champion the paper fully as I think the paper can benefit from another revision. I will slightly increase my rating to 6 - marginally above the acceptance threshold.
clarity,_quality,_novelty_and_reproducibility: Clarity concerns:
- While I find the idea of using negative samples in a contrastive-based loss to improve a diffusion process interesting, the narrative is missing intuitive explanations of the “Step-Wise Parallel Diffusion” and “Sample-Wise Auxiliary Diffusion”. It is unclear from the paper the justification of the two different designs. 
- Also, I think the proposed regularizer discussed in 3.1 requires more discussion. Is it fair to say that the regularizer enforces the diffusion process to also generate negative samples? From Fig. 2, I understand that the negative samples go through a denoising diffusion process too. Thus, is it fair to say that the regularizer thus makes the model learn two distributions: one distribution that generates the expected data given a condition, while the other one, is a distribution generating what it is not expected?

Reproducibility concerns:
- Given the current state of the paper, I find it hard to implement and reproduce the results. The architecture of the conditioning encoders is not discussed, and it is unclear if their parameters are optimized as part of the regularizer.
- Overall, the equations do not reveal what parameters are learned and are a bit convoluted. For example, in Sec. 3.1, the pdfs involved in the mutual information all seem to be parameterized by \theta. Are they really sharing the same parameters? Is f() the neural network to learn? If so, what is the architecture?

summary_of_the_review: Overall, I think the idea of using negative samples by means of a contrastive-based loss  is interesting. However, there are practical concerns that make me doubt the proposed method may be useful to speed up the diffusion processes. This is because while the proposed method indeed requires fewer steps, it is unclear if the step became more costly in terms of time or even computational resources. Second, the reproducibility given the state of the narrative falls short.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: Yes, Other reasons (please specify below)
details_of_ethics_concerns: Given that the proposed method generates data, it may be used to generate fake data and used for harmful purposes.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1428/Reviewer_MzmQ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Existing diffusion-based cross-modal generation methods mainly establish the cross-modal relationships by incorporating the cross-modal prior model into the variational lower bound of the diffusion model. However, the authors claim that this method may lead to the loss of the cross-modal correspondence in the denoising process. To overcome this, the authors propose Conditional Discrete Contrastive Diffusion (CDCD) loss, which enhances the cross-modal relationships by constructing negative samples and introducing contrastive loss in the training.
strength_and_weaknesses: Strength:
1. The idea of introducing contrastive loss to enhance the cross-modal relationship is reasonable. The proposed pipeline is well demonstrated in Figure2.
2. The proposed approach is clearly described, which makes the manuscript easy to follow.

Weaknesses:
1. The authors should provide more evidence to support the claim that incorporating prior into the variational lower bound can lead to the loss of the cross-modal correspondence.
2. Would the objective of enhancing cross-modal relationships contradict to increase the sample quality? How would the authors balance the variational loss and contrastive loss?
3. In the construction of inter-negative samples, the authors take all the images x’ other than x as negative samples. In this way, similar images may also be considered negative samples. How would the authors address this?
4. In the text-to-image generation task, the authors use the VQ-diffusion-S as the baseline. The results of the proposed approach slightly outperform the VQ-diffusion-S while falling behind the VQ-diffusion-B greatly. The authors should verify the effectiveness of the proposed approach on larger models.
5. In Table3, the performance of the proposed approach falls behind the DF-GAN.
clarity,_quality,_novelty_and_reproducibility: The idea that introducing the contrastive loss into the diffusion model is reasonable and novel which demonstrates the contribution of this paper. The proposed approach is clearly described in the paper.
summary_of_the_review: The idea of this paper makes sense. However, the experimental results are not satisfactory. There are some issues with the proposed approach. Therefore, I believe that the paper is below the acceptance bar.

=======================================================================================================

Thank the authors for addressing my concerns. After reading the authors’ responses and peer reviews, I decide to change my score to 6. My main concern is the experiment results fall behind the VQ-D-B. The new experiment results show that the contrastive loss improves the FID of VQ-D-B from 19.7 to 18.44, which addresses my main concern.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1428/Reviewer_3LEn
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a so-called Discrete Contrastive Diffusion model for cross-modal music and image generation. The main idea is to explicitly enhance input-output connections by maximizing their mutual information instead of implicitly learning such relationships. Specifically, the authors try to combine diffusion training and contrastive learning via the conventional variational objectives. Experiments on three different tasks (i.e., dance-to-music generation, text-to-image synthesis, and class-conditioned image synthesis) verify the effectiveness of the proposed method by showing higher synthesis quality and faster inference speed compared to other existing diffusion models. 


strength_and_weaknesses: Strengths:
To the best of my knowledge, this paper is the first to combine diffusion training and contrastive learning to design an effective generative model. The authors propose the CDCD loss and design two contrastive diffusion mechanisms to achieve this goal. The enhanced diffusion model is designed in a reasonable manner. The paper is well-written and well-organized. The proposed method is clearly described with all necessary analyses and mathematical details. Experiments have been conducted to demonstrate the effectiveness of the proposed method by showing higher synthesis quality and faster inference speed compared to other existing diffusion models.

Weaknesses:
By zooming in the synthesized images shown in Fig. 6 and Fig. 7, it seems to me that they are not as good as some SOTA results obtained by DALLE-V2, Imagen, etc. The authors need to explain the reasons and show/compare more qualitative results on the tasks of text-to-image synthesis and class-conditioned image synthesis in their supplemental materials. 

clarity,_quality,_novelty_and_reproducibility: Good. 
summary_of_the_review: Due to enough novelty and academic contributions as analyzed above, I suggest accepting this paper after some minor revisions.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1428/Reviewer_6GAW
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper introduces a Conditional Discrete Contrastive Diffusion (CDCD) loss to enhance input-output connections by maximizing their mutual information. The author also designs two contrastive diffusion mechanisms to  incorporate  $L_{CDCD}$  into the denoising process.  Diverse multimodal conditional synthesis tasks have been evaluated and the proposed methods achieve higher or competitive general synthesis quality. 

strength_and_weaknesses: ###### Strength：

· This work first combines diffusion training and contrastive learning, which provides a new perspective on the research of conditional DPMs.

· The proposed object function and contrastive diffusion mechanisms are shown promising quantitative and visual results. 

· The frame work improves the convergence of diffusion models, reducing the number of required diffusion steps by more than 35% on two benchmarks.

· Proper INTRA- AND INTER-NEGATIVE SAMPLING methods are adopted and relevant experiments revealed their effectiveness. 

###### Weakness：

I did't see any weaknesses
clarity,_quality,_novelty_and_reproducibility: ###### Quality & Novelty

· The process of proof and experiment are sufficient.  Some people in the community will be very interested in this work.

###### Clarity:

· The author's writing is good and ideas were clarified well.

###### Reproducibility:

· The author provided implementation details  and already released their codes and pre-trained models.  I think this work is reproducible. 
summary_of_the_review: 
This paper introduces a simple yet effective method to improve the input-output connections via maximized mutual information. And both proposed DPMs are proven effective and achieve state-of-the-art scores in our evaluations. Compare to previous methods, the proposed methods show better performance. Overall, I think this paper is good.


correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: Yes, Privacy, security and safety
recommendation: 6: marginally above the acceptance threshold