Paper: Fast Sampling via De-randomization for Discrete Diffusion Models (ICLR-2024)

Reviewer comment (Theory):
I'm not convinced that the accelerated sampling algorithm that the authors propose is actually correct from a probabilistic standpoint. The authors motivate it by proving (in Theorem 3.1) that the transition times in the forward process are independent and follow a particular distribution based on the noise schedule. However, this is only proven for the forward process. In order for the proposed sampling algorithm to make sense, this property would have to also be true for the reverse process: it must be the case that, after conditioning on the current value of \(x_t\), the transition times remain independent and have a tractable distribution.

Author response:
**A1-2.** We believe there is a misunderstanding here by the reviewer.  Theorem 3.1 indeed holds for both the forward and reverse processes. More specifically, the independence of transition times $\tau_n$ follow from our definition of forward process in Eq. (4) and are not conditioned on $\mathbf{x}\_0$ or any other variables. In the forward process, we have $\mathbf{x}\_{t,n} = b_{t,n}\mathbf{x}\_{t-1,n} + (1-b_{t,n}) \mathbf{w}\_n $, and $\tau_{n} = t$ is equivalent to $b_{0,n} =1, \ldots, b_{t-1,n} = 1$ and $b_{t,n} = 0$. Since $\\{ b_{t,n} \\}\_{t=0}^T$ is independent for different $n$ by definition, each $\tau_n$ is also independent. Therefore, the corresponding reverse process also has independent transition time. We believe the reviewer's misunderstanding may stem from the pre-assumption that the transition of different tokens is dependent. However, we want to clarify that these transitions are, in fact, independent. This underscores the reason why transition time remains independent in both the forward and reverse processes.

Reviewer comment (Theory):
The independence of transition times in the reverse process does not seem obvious. Note that the values of the tokens get noise added independently during the forward process when you condition on \(x_0\), but are not independent when you condition on their values in \(x_t\). This is why discrete diffusion methods are typically justified using a variational ELBO argument; approximating the tokens as independent creates a bias that must be corrected for through iterative refinement. The same might be true of the transition times.

Author response:
**Answer**: Yes, we sample the transition times $\tau$ first rather than inferring them from a partially denoised sequence, which is shown both in Line 3 of Algorithm 1, and the ELBO we added in Appendix B. Therefore, $\tau$’s are always i.i.d. and probabilistically sound. We hope this will totally remove your concern.

Reviewer comment (Theory):
The authors note that their reverse sampler decodes predictions using "different decoding strategies including greedy decoding, top-k decoding and temperature decoding" and mentions that argmax decoding shows strong performance. This does not seem to make sense under a probabilistic view of the process; it feels like a heuristic to improve results. There is apparently some precedent for this, since the RDM model also uses heuristic decoding methods.

Author response:
**Answer**: For the decoding strategy in Section 3, it is a part of $f_{\theta}(\boldsymbol{x}\_{t,1:N}, t)$, similar to the output layer of the neural network. Instead of being associated with the diffusion model, it is more pertinent to the NLP task. In essence, it is orthogonal to all discrete diffusion models. The argmax decoding mentioned in “decoding type” paragraph in Section 5 also refers to the decoding strategy related to the $\hat{x}_{0, n}$ network and is therefore irrelevant to our method. The argmax decoding strategy has also been introduced in Section 5.3 and detailed in Appendix E.4 of Zheng et al. (2023).

For the top-k-selection related to our diffusion model, in Appendix E, we have attributed this to RDM (Zheng et al. 2023), specifically the “improved decoding” in Section 5.3 and detailed in Appendix E.4 of Zheng et al. (2023). This is to ensure a fair comparison with RDM.

Reviewer comment (Experiments):
The authors motivate their approach based on accelerating generation speed, but I'm not convinced the gains are actually significant when compared to just using an autoregressive model. The authors seem to focus on accelerating sampling when there are more denoising steps than there are tokens (e.g. the 1000-step and \(\infty\)-step conditions for WMT, the 1000-step and 4000-step conditions for unconditional generation). But if you have more denoising steps than tokens, it seems like you may as well just use an ordinary autoregressive model. Indeed, in the "\(\infty\)-step" diffusion, the continuous-time reverse sampler requires one network evaluation per token generated, which is the same number of computation steps as an autoregressive model would use.

Author response:
**A6.** We respectfully disagree that our focus is on accelerating sampling when there are more denoising steps $T$ than the number of tokens $N$. While it is true that our method can get more significant improvement when T is larger, we care about the result for all sampling steps, including those much smaller than the number of tokens. Below is an example of our result shown in Table 5 of our revision.

| Dataset | Steps | RDM-Multi |  | DNDM-Multi |  | RDM- $k$-Multi |  | DNDM- $k$-Multi |  |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  |  | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE |
| IWSLT14 | 25 | **31.26** | 25 | 30.95 | **9.03** | **32.82** | 25 | 32.30 | **9.03** |
|  | 50 | **31.50** | 50 | 31.45 | **14.07** | **32.82** | 50 | 32.80 | **14.07** |
|  | 1000 | 31.69 | 1000 | **31.82** | **30.33** | 32.64 | 1000 | **33.15** | **30.33** |
|  | $\infty$ | - | - | **31.89** | **32.73** | - | - | **33.44** | **32.73** |
| WMT14 | 25 | **25.25** | 25 | 25.01 | **13.52** | **26.03** | 25 | 25.98 | **13.52** |
|  | 50 | **25.75** | 50 | 25.33 | **20.58** | 26.14 | 50 | **26.37** | **20.58** |
|  | 1000 | 25.66 | 1000 | **25.71** | **38.94** | 25.82 | 1000 | **26.88** | **38.94** |
|  | $\infty$ | - | - | **24.79** | **40.67** | - | - | **26.39** | **40.67** |
| WMT16 | 25 | **32.29** | 25 | 31.97 | **8.5** | **33.12** | 25 | 32.94 | **8.5** |
|  | 50 | **32.53** | 50 | 32.50 | **14.73** | **33.41** | 50 | 33.26 | **14.73** |
|  | 1000 | 32.63 | 1000 | **32.86** | **38.45** | 33.67 | 1000 | **33.79** | **38.45** |
|  | $\infty$ | - | - | **32.91** | **41.64** | - | - | **33.86** | **41.64** |



As we can see from the table, our algorithm is 2-3 times faster even for small sampling steps (25). We have also proved that our algorithm can accelerate the discrete diffusion for all $N, T$ and included more discussion in Appendix D. 

In addition, we agree that our infinite limit has the same NFE as the autoregressive diffusion. However, our method also supports finite step sampling, which is much faster than the autoregression diffusion in terms of NFE. Moreover, our primary objective and contribution revolve around expediting multinomial diffusion-type discrete diffusions. The comparison with autoregressive models does not contribute to substantiating our assertion regarding the acceleration of sampling speed.

Reviewer comment (Experiments):
I expect that an autoregressive model would have a better tradeoff between generation time and quality than the method proposed here.

Author response:
**A6.** We respectfully disagree that our focus is on accelerating sampling when there are more denoising steps $T$ than the number of tokens $N$. While it is true that our method can get more significant improvement when T is larger, we care about the result for all sampling steps, including those much smaller than the number of tokens. Below is an example of our result shown in Table 5 of our revision.

| Dataset | Steps | RDM-Multi |  | DNDM-Multi |  | RDM- $k$-Multi |  | DNDM- $k$-Multi |  |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  |  | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE |
| IWSLT14 | 25 | **31.26** | 25 | 30.95 | **9.03** | **32.82** | 25 | 32.30 | **9.03** |
|  | 50 | **31.50** | 50 | 31.45 | **14.07** | **32.82** | 50 | 32.80 | **14.07** |
|  | 1000 | 31.69 | 1000 | **31.82** | **30.33** | 32.64 | 1000 | **33.15** | **30.33** |
|  | $\infty$ | - | - | **31.89** | **32.73** | - | - | **33.44** | **32.73** |
| WMT14 | 25 | **25.25** | 25 | 25.01 | **13.52** | **26.03** | 25 | 25.98 | **13.52** |
|  | 50 | **25.75** | 50 | 25.33 | **20.58** | 26.14 | 50 | **26.37** | **20.58** |
|  | 1000 | 25.66 | 1000 | **25.71** | **38.94** | 25.82 | 1000 | **26.88** | **38.94** |
|  | $\infty$ | - | - | **24.79** | **40.67** | - | - | **26.39** | **40.67** |
| WMT16 | 25 | **32.29** | 25 | 31.97 | **8.5** | **33.12** | 25 | 32.94 | **8.5** |
|  | 50 | **32.53** | 50 | 32.50 | **14.73** | **33.41** | 50 | 33.26 | **14.73** |
|  | 1000 | 32.63 | 1000 | **32.86** | **38.45** | 33.67 | 1000 | **33.79** | **38.45** |
|  | $\infty$ | - | - | **32.91** | **41.64** | - | - | **33.86** | **41.64** |



As we can see from the table, our algorithm is 2-3 times faster even for small sampling steps (25). We have also proved that our algorithm can accelerate the discrete diffusion for all $N, T$ and included more discussion in Appendix D. 

In addition, we agree that our infinite limit has the same NFE as the autoregressive diffusion. However, our method also supports finite step sampling, which is much faster than the autoregression diffusion in terms of NFE. Moreover, our primary objective and contribution revolve around expediting multinomial diffusion-type discrete diffusions. The comparison with autoregressive models does not contribute to substantiating our assertion regarding the acceleration of sampling speed.

Reviewer comment (Evaluation):
Unconditional generation samples (in the appendix) remain unimpressive relative to autoregressive model generation quality. This applies to previous work as well, of course, but I don't think the improvements in Table 3 translate to a noticeable increase in overall sample quality.

Author response:
**A8.** Again, our contribution is on the sampling speed instead of the sample quality. Since our method is a training-free accelerated sampling algorithm, the performance of our method in terms of sample quality is contingent on the pre-trained models, which may not always be optimized for sample quality.

Reviewer comment (Novelty):
The proposed approach only applies to discrete diffusion models that have the form in Equation (1), but this is only a subset of the discrete diffusion models that have been considered in the literature. Austin et al. (2021) discusses a few alternatives that are not of this form (e.g. discreteized Gaussian, token-embedding-distance transitions, or hybrids of absorbing and other diffusion processes as discussed in the appendix). Another recent example is Blackout Diffusion (Santos et al. 2023).

Author response:
**A9.** Eq. (1) has already included the most two widely used discrete diffusion models:  multinomial diffusion and absorbing diffusions. Note that the discretized Gaussian diffusion and Blackout diffusion only outperform other discrete diffusion models for continuous data, while we focus on the generation of discrete data such as text. We leave the acceleration of other “discrete” diffusion models as a future work.

Reviewer comment (Evaluation):
I think that the use of the phrase “sampling iterations” in the paper can be somewhat misleading, for instance on page 5, where it is indicated that the generation time increases sub-linearly with sampling iterations (as opposed to RDM). While technically true from the diffusion perspective, this also overlooks the fact that the relevant metrics are actually the amount of function evaluations (or the time taken for sampling). Similarly, comparisons with the same step counts to RDM in Table 1 & 2 seem suboptimal to me, and instead it should be (approximately) the same amount of function evaluations / wall clock time.

Author response:
**A1**.  Thank you for your suggestion. We have used “sampling steps” to replace “sampling iterations”  in the revision. While the number of sampling steps is also a widely used metric as used in [1] and [2], we have also incorporated the NFE metric into our experiments to provide a more comprehensive comparison in our revision.  Please refer to Tables 5 and 6 in Appendix D, where we list the average NFEs of both RDM and our method, under the same sampling steps. Below is an example of our result shown in Table 6 of our revision. We can see that the BLEU score achieved by our method with about 40 NFE outperforms the BLEU score of RDM with 50 NFE. 


| Dataset | | RDM-Absorb |  | DNDM-Absorb |  | RDM- $k$-Absorb |  | DNDM- $k$-Absorb |  |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  |  Steps | BLEU | Steps | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE |
| IWSLT14 | 25 | 31.58 | 25 | **32.43** | **13.81** | **34.50** | 25 | 34.14 | **13.81** |
|  | 50 | 31.80 | 50 | **32.63** | **19.24** | **34.58** | 50 | 34.34 | **19.24** |
|  | 1000 | 31.91 | 1000 | **32.93** | **31.08** | **34.60** | 1000 | 34.56 | **31.08** |
|  | &infin; | - | - | **33.03** | **32.07** | - | - | **34.65** | **32.07** |
| WMT14 | 25 | 24.97 | 25 | **25.79** | **15.09** | **27.50** | 25 | 27.18 | **15.09** |
|  | 50 | 24.95 | 50 | **26.10** | **22.45** | **27.73** | 50 | 27.66 | **22.45** |
|  | 1000 | 25.22 | 1000 | **26.43** | **38.76** | 27.75 | 1000 | **27.82** | **38.76** |
|  | &infin; | - | - | **26.50** | **40.39** | - | - | **27.50** | **40.39** |
| WMT16 | 25 | 32.86 | 25 | **33.20** | **13.91** | 33.92 | 25 | **33.96** | **13.91** |
|  | 50 | 32.93 | 50 | **33.30** | **20.95** | 34.10 | 50 | **34.20** | **20.95** |
|  | 1000 | 33.25 | 1000 | **33.60** | **38.27** | **34.44** | 1000 | 34.38 | **38.27** |
|  | &infin; | - | - | **33.42** | **41.59** | - | - | **34.41** | **41.59** |

Reviewer comment (Experiments):
It would also be good to elaborate where exactly does the method provide an improvement over RDM, since they focused more on step counts <25, whereas the comparisons here are for step counts >25. In this sense, is the model even a direct competitor for RDM, or could the two methods be combined?

Author response:
**A2**. Our model is not intended as a direct competitor to RDM but rather as a training-free accelerator in terms of sampling speed, which can be integrated with various discrete diffusion models in the form of (1), including RDM. Our approach addresses a key inefficiency inherent in existing sampling algorithms of discrete diffusion models. Specifically, in traditional diffusion models, including RDM, increasing the number of sampling steps leads to better sampling quality but at the cost of linearly increasing computational time, often with only marginal performance gains. In contrast, our method achieves a more favorable tradeoff between sampling time and quality. The improvement in efficiency is because our model significantly reduces the number of function evaluations (NFE), as our method. We have added Tables 5 and 6 in Appendix D with NFE as a performance metric.  We can see that even for 1000 sampling steps, the effective number of sampling steps (NFE) is around 30-40. Therefore, with smaller NFE (30-40), our method can consistently achieve a higher BLEU score than RDM with 50 NFE. Please see the answer to A1 and Tables 5 and 6 in our revision for the detailed comparison in terms of NFE.

Reviewer comment (Novelty):
I think the paper may require more work on clarifying what are the new contributions of the proposed model and designing the experiments such that they clearly showcase these contributions (see also questions on ARDM). E.g., under what conditions exactly can we expect improvements in sampling speed?

Author response:
**A3**. As we emphasized in the introduction, our main contribution is proposing a new sampling method for discrete diffusion models that reduces the generation time (i.e., NFE) while preserving the sample quality.  Please see the table in A1 for details. 

From the theoretical point of view, for discrete diffusion models, our method is guaranteed to achieve a reduction in NFE compared with the number of sampling steps. This is proved in Theorem D.1 in the newly added Appendix D. More specifically, Theorem D.1 shows that the NFE is reduced from $T$ to $(1- C_{D_{\tau}, T, N})T$, where $C_{D_{\tau}, T, N}$ is a constant with range $(0,1)$ that depend on $D_{\tau}, T, N$.  The details of the discussion can be found in Appendix D.

Reviewer comment (Novelty):
It seems to me that there are quite a bit of (potentially subtle) similarities with the previous work [1], where the authors note a connection between absorbing diffusion models and order-agnostic autoregressive models. In particular, as pointed out in their Appendix C, the continuous-time limit of an absorbing diffusion model is effectively equivalent to to their order-agnostic ARDM, which seems to be also equivalent to the infinite-time limit for absorbing diffusion presented in this paper. Would the authors of the submission agree, or maybe are there differences that I have overlooked? I do see that in the non-infinite-time case, the models do not exactly match, and the model is formulated from a different point of view, potentially allowing more flexibility in designing the generative process. But it would be good to give more elaboration on the differences to ARDM and maybe also provide direct numerical comparisons in the paper.

Author response:
**A4**. Thank you for bringing our attention to the ARDM. We would like to emphasize that our method is a training-free fast sampling method for various discrete diffusion processes, including absorbing diffusion and multinomial diffusion. Such a fast sampling method enables us to tackle discrete diffusion models with a large number of sampling steps, or even infinite steps. While ARDM can be seen as the infinite-time limit of absorbing diffusion, it remains orthogonal to our training-free fast sampling method.

Reviewer comment (Theory):
One downside of the paper as of now is that a method to estimate the data likelihood is not included. Perhaps it would be possible to create a lower bound by not having having intermediate latents \(x_t\) at all, and considering the transition times \(\tau\) as latent variables? For instance, setting the inference distribution to \(q(\tau^{1:D}) \prod_k^D q(x_T^k | x_0^k)\), where \(\tau^k\) is distributed according to Uniform({1,…,T}), and the corresponding generative process defined such that we first sample from the priors \(p(x_T)\) and \(p(\tau)\), and then generate the different dimensions according to the ordering of \(\tau\). If I am not mistaken, this would be similar to what was done in the ARDM paper as well, and one could form an ELBO similar to theirs.

Author response:
**A5**. Thank you for your valuable advice. We have provided the derivation of ELBO in Appendix B.3 which can estimate the data likelihood. The ARDM obtains their ELBO by decomposing with respect to the data dimension, which depends on the autoregressive nature of their model. Therefore, their method doesn’t apply to our model. Instead, we take the approach of classic diffusion models, especially the non-markovian continuous time diffusion model DDIM [3], and derive the ELBO for our model.  Our ELBO is based on both intermediate latent $x_t$ and the transition time $\tau$.

Reviewer comment (Evaluation):
How is the evaluation with the BERT model done for the unconditional generation experiments, exactly?

Author response:
**A6**. Thanks for pointing this out. We realized that BERT may not be a good model for evaluating the perplexity of unconditionally generated texts. We have reevaluated the perplexity scores (PPL) of the generated samples using the GPT2-large model. Our new evaluation results are as follows: the average perplexity score for the text samples generated from multinomial diffusion trained on enwik8 using vanilla sampling methods is 801.7817. Additionally, the average perplexity score for the text samples generated from multinomial diffusion trained on the text8 dataset via DNDM is 556.7812.

Reviewer comment (Experiments):
The experiments are promising, but the scope is a bit limited. For example, the claims of the paper could be strengthened with results on sequence-to-sequence generation tasks such as question generation and paraphrasing (1, 2).

Author response:
**A1**. Thank you for your valuable suggestion. We have added Theorem D.1, which shows that the NFE of our method is provably reduced from $T$ to $(1- C_{D_{\tau}, T, N})T$. Therefore, our method is guaranteed to achieve a reduction in NFE compared with the number of sampling steps. We're currently in the middle of conducting experiments on sequence-to-sequence generation tasks. We will add these additional experiments to the final version.

Reviewer comment (Experiments):
An ablation of the choice to use Beta distribution to approximate the true transition time distribution (induced by the $\alpha_t$ schedule during training) would be clarifying. This is especially true in the case of the continuous time evaluation (Tables 1, 2), where it's slightly unclear whether the boosts to performance are due to the continuous time formulation or the choice of Beta distribution.

Author response:
**A2**. Thank you for your valuable suggestion. We have conducted an ablation study on different choices of Beta distributions (Tables 7, 8, 9 in Appendix F.1). Specifically, when utilizing the same Beta distribution as those employed in continuous time, the discrete (finite step) version of DNDM does not achieve comparable performance as DNDM-C. Please refer to the following table and Table 9 in Appendix F.1. 

BLEU scores of different sampling steps with the same transition time distribution $Beta(100, 4)$ on WMT16:

| Steps | DNDM-k-multi | DNDM-k-absorb | DNDM-multi | DNDM-absorb |
| :---: | :--- | :--- | :--- | :--- |
| 50 | 31.60 | 31.74 | 30.39 | 29.69 |
| 1000 | 33.59 | 34.37 | 32.87 | 33.52 |
| $\infty$ | 33.86 | 34.41 | 32.91 | 33.42 |

We note that, in the last column, although the performance of 1000 steps is slightly better than infinite steps, this aligns with the result of DNDM-absorb in Table 2, and therefore does not contradict our conclusion that the difference in Beta-distribution used does not significantly impact the performance comparison between 1000 steps and infinite steps. This suggests that the performance improvements are mainly attributed to the continuous time formulation itself.

Reviewer comment (Experiments):
How well does the model compare to RDM when the Beta distribution transition times are not used?

Author response:
**A3**. We appreciate your insightful question. As detailed in the ablation study in Appendix C, we conducted experiments using various transition time distributions. The effect of different distributions can be seen as different schedules for selecting $\alpha_t$ in discrete diffusion models. In particular, In Table 4, $cosine^2$ schedule corresponds to the $\alpha_t$ schedule utilized in the RDM results. Overall, we observed a slight decrease in performance when not employing the Beta distribution. However, our approach still demonstrates superior performance compared to both RDM-absorbing and RDM-multinomial with comparable amounts of NFEs. Please refer to the following table and Table 4 in Appendix C for more details.

| Datasets | Schedules | DNDM-multi |  | DNDM-absorb |  | DNDM-k-multi |  | DNDM-k-absorb |  |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|  |  | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE |
| IWSLT14 | Cosine | 31.72 | 31.71 | 32.71 | 31.21 | 32.91 | 31.71 | 34.50 | 31.21 |
|  | Cosine $^2$ | 31.78 | 31.74 | **32.93** | 31.21 | 32.78 | 31.74 | 34.53 | 31.21 |
|  | Linear $\alpha$ | 31.77 | 31.82 | 32.65 | 31.33 | 32.83 | 31.82 | 34.53 | 31.33 |
|  | Beta (reported) | **31.82** | **30.33** | **32.93** | **31.08** | **33.15** | **30.33** | **34.56** | **31.08** |
| WMT14 | Cosine | **25.80** | 39.61 | **26.54** | 39.18 | 26.63 | 39.61 | 27.81 | 39.18 |
|  | Cosine ${ }^2$ | 25.52 | 39.48 | 26.53 | 39.18 | 25.01 | 39.48 | **27.95** | 39.18 |
|  | Linear $\alpha$ | 25.58 | 39.97 | 26.33 | 39.82 | 25.47 | 39.97 | 27.63 | 39.82 |
|  | Beta (reported) | 25.71 | **38.94** | 26.43 | **38.76** | **26.88** | **38.94** | 27.82 | **38.76** |
| WMT16  | Cosine | 32.71 | 40.50 | 33.56 | 40.45 | 33.46 | 40.50 | 34.37 | 40.45 |
|  | Cosine ${ }^2$ | 32.73 | 40.50 | 33.51 | 40.45 | 33.44 | 40.50 | 34.24 | 40.45 |
|  | Linear $\alpha$ | 32.85 | 40.36 | 33.46 | 40.36 | 33.47 | 40.36 | 33.88 | 40.36 |
|  | Beta (reported) | **32.86** | **38.46** | **33.60** | **38.27** | **33.79** | **38.45** | **34.38** | **38.27** |

Reviewer comment (Theory):
Regarding the WMT14 results, the authors mention the weaker performance relative to the other datasets could be a sign that WMT14 "violates our assumption that the pretrained score model is uniformly accurate for all time steps" -- could you clarify what this means?

Author response:
**A5**. Thank you for pointing this out. We apologize for any confusion caused by our initial argument. 
In our paper, we employed a pretrained model checkpoint trained on 50 steps. This means the network was only familiarized with 50 distinct time steps during its training phase, and the timesteps during reverse sampling might have been mostly unseen, especially for timesteps 1000 or infinite timesteps. For example, the model trained on $t = 1, \ldots, 50$ might be asked to output at $t = 2.71$ during sampling. Therefore, we conjecture that the network can provide accurate estimates even for those steps which it was not explicitly trained on. However, for WMT14, the pretrained model checkpoint may not be able to generalize to unseen time steps effectively. We have now relocated this argument from the main text to an extended explanation in Appendix F.1 for clarity.

Reviewer comment (Experiments):
This paper is not complete. First of all, it lacks ablation studies to demonstrate the influence of distributions of the transition time. With respect, I do not agree with that, all instances of transition times are equal in effect.

Author response:
Thank you for your suggestion. We have added several ablation studies regarding this matter. Specifically, in Table 4 of our revision, we present a comparison of the effects by different schedules of transition time on our algorithm's performance. For reference, we also provide the table of our ablation study here. We can see that our proposed Beta schedule outperforms previous schedules. 

| Datasets | Schedules | DNDM-multi |  | DNDM-absorb |  | DNDM-k-multi |  | DNDM-k-absorb |  |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|  |  | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE | BLEU | Avg NFE |
| IWSLT14 | Cosine | 31.72 | 31.71 | 32.71 | 31.21 | 32.91 | 31.71 | 34.50 | 31.21 |
|  | Cosine $^2$ | 31.78 | 31.74 | **32.93** | 31.21 | 32.78 | 31.74 | 34.53 | 31.21 |
|  | Linear $\alpha$ | 31.77 | 31.82 | 32.65 | 31.33 | 32.83 | 31.82 | 34.53 | 31.33 |
|  | Beta (reported) | **31.82** | **30.33** | **32.93** | **31.08** | **33.15** | **30.33** | **34.56** | **31.08** |
| WMT14 | Cosine | **25.80** | 39.61 | **26.54** | 39.18 | 26.63 | 39.61 | 27.81 | 39.18 |
|  | Cosine ${ }^2$ | 25.52 | 39.48 | 26.53 | 39.18 | 25.01 | 39.48 | **27.95** | 39.18 |
|  | Linear $\alpha$ | 25.58 | 39.97 | 26.33 | 39.82 | 25.47 | 39.97 | 27.63 | 39.82 |
|  | Beta (reported) | 25.71 | **38.94** | 26.43 | **38.76** | **26.88** | **38.94** | 27.82 | **38.76** |
| WMT16 | Cosine | 32.71 | 40.50 | 33.56 | 40.45 | 33.46 | 40.50 | 34.37 | 40.45 |
|  | Cosine ${ }^2$ | 32.73 | 40.50 | 33.51 | 40.45 | 33.44 | 40.50 | 34.24 | 40.45 |
|  | Linear $\alpha$ | 32.85 | 40.36 | 33.46 | 40.36 | 33.47 | 40.36 | 33.88 | 40.36 |
|  | Beta (reported) | **32.86** | **38.46** | **33.60** | **38.27** | **33.79** | **38.45** | **34.38** | **38.27** |

Additionally, Tables 7, 8, and 9 in our revision are dedicated to comparing the performance under various parameter choices in the Beta distribution.

Reviewer comment (Experiments):
Following up on the above point, why not the author study which instantiation of the transition time is the best for sampling, in the sense of quality and speed? Stopping at the distribution stage is not a good thought, I believe.

Author response:
If we understand your question correctly, it refers to a specific set of choices of transition times. The exploration of different transition times presents several practical challenges. First of all, the number of transition times equals N, corresponds to the length of a sentence, which is not fixed. Secondly, for long sequences with N=30, the total number of possible transition times is $T^{30} \geq 2^{30}$ . It is thus impractical to enumerate every possible instantiation. Therefore, we have focused our efforts on comparing the performance impacts of different distributions of transition times. Please see A1 for more details.

Reviewer comment (Presentation):
Another thing is the visualization of transition times, the x_t at those times, and other related values. Letting me see that x_t gets still after the transition time will make me more convinced of your method and creativity.

Author response:
In Figure 5 in Appendix G.3.1, we have included an entire generation trajectory of $x_t$ with respect to the transition time of each token. This trajectory shows how $x_t$ evolves over time following their respective transition times. In particular, $x_t$ gets still after $t=39$. We also provide a simplified trajectory in Figure 2(b).

Reviewer comment (Experiments):
Apart from that, the authors did not implement this work in image generation. I guess they may have a good reason for that, as diffusion is the dominating and most influential method in image generation. A method to improve diffusion sampling but not implemented on the image is strange. Are there anything barriers to the method to apply in images?

Author response:
Our paper primarily focuses on the acceleration of sampling for discrete diffusion models on discrete space, especially on models in the form of (1) in our paper, which includes multinomial diffusion [1] and absorbing diffusion [2]. Image generation, on the contrary, typically falls into the realm of continuous diffusion models due to the continuous nature of the image data. According to [1] and [3], the performance of discrete diffusion models for image generation, even the more sophisticated discretized Gaussian [2], or blackout models [3], is worse than the continuous diffusion models such as DDPM [4]. Therefore, we choose to prioritize evaluation of our models on discrete data generation tasks, as we don’t think discrete diffusion models can easily beat DDPM on image generation.

Reviewer comment (Evaluation):
The proposed method does not outperform competitors in all shown cases. Sometimes the improvements are marginal.

Author response:
We agree. However, we want to clarify that the primary contribution of our method lies in its accelerated sampling speed. We use the term *outperform* to specifically refer to the enhanced sampling speed achieved by our method, as opposed to improvement in the overall generation performance (e.g., BLEU score). Instead, we show that under the same number of sampling steps, our approach attains a competitive BLEU score as RDM, with substantially faster generation. For example, for 1000 steps, RDM on WMT16 with k-absorb is slightly better than our 1000 steps, but is 20 times slower than our approach.

Reviewer comment (Theory):
As a theoretical work, I appreciate the idea a lot, but the theoretical analysis is not that deep so I can safely vote for acceptance with no objection from the other reviewers and the AC. Deeper theoretical analysis related to sampling NFEs, approximation error with the original process, and techniques to further improve it could be much more appreciated.

Author response:
Thank you for your suggestion. In the appendix, we have added an analysis on the number of function evaluations (NFE). Specifically, in Theorem D.1,  we have theoretically established that the NFE in our method will not exceed $(1 - (1 - \frac{1}{T})^{N})\cdot T$, as opposed to $T$  NFE for the original sampling method. This suggests that our method is more efficient in terms of NFE. Detailed proofs and discussions regarding NFE can be found in Appendix D.

Regarding the approximation error to the original process, we would like to clarify that there is no approximation error.  In Appendix B.1,  we have shown that our new process is equivalent to the original process in Eq.(1).

Reviewer comment (Experiments):
Algorithms like DPM-Solvers do not work in the language case? Please kindly tell me.

Author response:
The DPM-Solver [5] is exclusively designed for continuous diffusion applications. At the core of DPM-Solver is a specialized high-order ordinary differential equation (ODE) solver tailored for the semi-linear structure of continuous diffusion models (e.g., DDPM, VP-SDE and VE-SDE), employing methods such as Taylor expansions. It is unclear if DPM-Solver can be applied to discrete diffusion.

On the other hand, a lot of recent studies [6,7,8] have shown that continuous diffusion models do not work as well as GPT or discrete diffusion models for language generation.