Paper: Democratized Diffusion Language Model (ICLR-2024)

Reviewer comment (Novelty):
The innovation of the paper is insufficient. The main contribution is to reproduce the CDCD structure and analyze the existing DLMs, without proposing new models or methods.

Author response:
The main aim of our paper is to understand the generation process of DLMs. With novel methodology in this direction, we observed that existing DLMs allow performing early exiting. We proposed several methods for performing early exiting (See Sections 4.2, 4.3). This work proposes faster sampling algorithms for DLMs and also lightens future research directions to further understanding and analysis of DLMs, which could lead to more new methods.

Reviewer comment (Experiments):
The length of the trained model is limited to 64, and it is not clear whether there will be different conclusions for longer lengths. The length of 64 is still a bit far from actual application.

Author response:
We will add experiments with longer sequences to the rebuttal revision soon. Please consider these new experiments when reevaluating your score.

Reviewer comment (Experiments):
We still care about the performance of pre-trained models on downstream tasks, and the paper did not select some downstream tasks for evaluation.

Author response:
Please note that we do not propose new DLMs with our paper. Instead, we studied sampling algorithms for existing models. These sampling algorithms are not connected to downstream task performance.

Reviewer comment (Evaluation):
It is recommended to add the model parameter quantity to the comparison in Table 1.

Author response:
Thank you for this proposal; we have added the number of model parameters in the latest revision.

Reviewer comment (Reproducibility):
Does the model have the ability to output the 
 token? The sentences in Appendix D are all truncated results.

Author response:
Sentences in the Appendix D section were sampled with a sequence length of 64. Thus, they are truncated. All used in our experiments model could produce 
 tokens. For example, it is possible to set it as a condition at any specific point within the sequence, or the model itself could generate it for long enough sequences.

Reviewer comment (Theory):
It is good to see analysis of sampling between different diffusion models, however, no further explanation about the deep reason to cause these differences.

Author response:
Differences between different diffusion models are specific to different training objectives and could be caused by noise scheduling. Experiments with varying noise scales ground this hypothesis. However, we did not dig into this and focused on proposing and understanding sampling methods for different DLMs and establishing an experimental setup for such research direction. Understanding various ways to perform an early exit for various models first could provide more evidence and grounding for further research on this topic rather than making solid assumptions without proper evaluation.

Reviewer comment (Experiments):
The advantage of DDLM is to early exit and speedup the generation process. However, compared with some faster ODE solvers (e.g. DPM-solver[1]), early exit of DDLM maybe not superior than them.

Author response:
First, let us clarify that DPM-solver is an second-order method, which implies two model’s forward passes during one step. Also note that DPM-solver is very similar to Heun sampler from [1]. In our experiments this type of samplers do not outperform Euler sampler with similar number of forward passes. 

[1] https://arxiv.org/abs/2206.00364

Reviewer comment (Experiments):
Why after your findings that using a noise scale of 0.9 is optimal, you still use a scale of 1.0 in later experiments.

Author response:
The purpose of experiments with noise scaling is to understand the underlying behavior of DLMs, leading to emerging capabilities to perform an early exiting. However, we observed that performing generation with a slightly reduced noise scale without hurting AR-NLL and Dist values, such a setup is not conventional. In our case, this finding could provide insights on early exiting for DLMs for future works, but performing experiments in such configuration for other models is not conventional.

Reviewer comment (Evaluation):
In table 2, it is weird AR-NLL=0.44 and dist_1=0 with noise=0, do you have generation examples?

Author response:
This is caused by AR-NLL metric issues since it is evaluated with external LM. It is conventional to obtain small values for repetitive samplings (e.g., one word repeated 64 times). Because of the issues with this metric, we also incorporated other metrics, such as a number of distinct tokens and self-bleu. E.g. Table 2 shows that the Dist value reached 0, indicating that AR-NLL does not provide helpful information.

Reviewer comment (Evaluation):
The GPT-Score is a relative value, with step-1000 as the reference text. However, the reference text may not be fluent. Is it possible to obtain the absolute value?

Author response:
It is possible, although we observed that absolute value without side-by-side comparison with GPT-4 had a large variation. Performing a side-by-side comparison is established by other works (e.g., in Direct Preference Optimization), and to the best of our knowledge, no works used absolute evaluation with GPT-4 (doing so is an exciting research idea itself). In our case, we used it to understand the convergence of sampling algorithms. Even though the final sampling could not be fluent, this sampling is the one that is obtained with the original models as is. So, this experiment shows that early exiting samples after a specific step do not differ from the final sampling (see Appendix Section D).

Reviewer comment (Experiments):
Can you also compare with discrete diffusion models?

Author response:
We are unaware of discrete diffusion models capable of generating unconditional text. Existing unconditional general-purpose text generation models are continuous, so we performed experiments with them.

Reviewer comment (Evaluation):
Why in Fig5, the NLL of Plaid (c) (<3.6) is lower than the DDLM (a) (>3.68)? This contradicts to the main table.

Author response:
For Figure 5, we used 1000 samples to evaluate the models, while the main table used 5000 samples. This led to a slight variation in AR-NLL values. We added an explanation to the latest revision.