Paper: Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning (ICLR-2024)

Reviewer comment (Experiments):
Discussion is limited on diffusion adaptation, a core method contribution of the paper. When using an existing MLM checkpoint instead of extensive pre-training, the performance of diffusion LM hinges on the effectiveness of diffusion adaptation. However, the paper does not provide enough performance metrics to validate the effectiveness of diffusion adaptation. For example, how does it compare to training from scratch, and how does the number of adaptation steps affect downstream performance? Without these results, it is unclear if diffusion adaptation is an effective approach to building diffusion LMs.

Author response:
Thank you for your question! 
We would like to clarify that our study does not aim to find the best practice for building diffusion language models. Instead, we hope to conduct a pioneering investigation into the scalability of diffusion language models without consuming huge computational resources. Our positive findings support this goal and can stimulate more future efforts on building large diffusion language models, either by pretraining from scratch or diffusive adaption.

Reviewer comment (Evaluation):
While showing diffusion LM can be promising and shows similar capabilities as auto-regressive LM, it is hard to get a concrete understanding of the performance of diffusion LM from the current paper.

Author response:
Conducting a strict comparison across the recipes of different pretrained models can be challenging, especially given the expenses involved and the abundance of technical details. As we respond in Q1, we are not aiming to beat other language models in this study, but to validate the scalability of diffusion language models and encourage further research efforts on this. And our results in Table 1, Figure 3, and Table 2 can support that our models have decent performance compared to models at similar scales, and demonstrate generalist ability that can follow instructions to solve various tasks.

Reviewer comment (Experiments):
No effective performance comparison between diffusion LM and other kinds of LMs. While it is convenient to adapt an existing MLM to diffusion LMs and evaluate it, as the authors pointed out, the current available MLM checkpoints are out-of-date, and diffusion adaptation inevitably loses performance compared to training from scratch with diffusion objective. This makes the current result potentially considerably inferior to what diffusion LM can achieve in theory (with a state-of-the-art training recipe and trained from scratch). As a result, it is hard to appreciate the exact capabilities of diffusion LM from the current results. This could also make some discussions ineffective, for instance, it cannot be ruled out that the deficiencies in reasoning are due to insufficient training.

Author response:
Conducting a strict comparison across the recipes of different pretrained models can be challenging, especially given the expenses involved and the abundance of technical details. As we respond in Q1, we are not aiming to beat other language models in this study, but to validate the scalability of diffusion language models and encourage further research efforts on this. And our results in Table 1, Figure 3, and Table 2 can support that our models have decent performance compared to models at similar scales, and demonstrate generalist ability that can follow instructions to solve various tasks.

Reviewer comment (Experiments):
Discussion on in-context learning and reasoning abilities is vague/ineffective. From Figure 5, it is hard to observe performance improvement due to in-context learning. For auto-regressive LM, in-context learning can be evaluated on raw pre-trained LM prior to instruction-finetuning. Maybe a similar evaluation on diffusion LMs could better visualize their in-context learning abilities. Also, as the authors pointed out, the limited context length of the original model limits the evaluation of few-shot performance. One possible solution without training from scratch could be using interpolation of position embeddings (e.g. [1]).

Author response:
Observation in Figure 5 is actually consistent with the findings of the FLAN series paper [1,2], which shows that model performance is not sensitive to the number of demonstrations after instruction tuning, and we have mentioned this phenomenon in Section 4.2.2. Note that the most important message here is that instruction tuning helps our Diffusion LMs obtain in-context learning ability, without which we found that a raw pretrained MLM cannot learn from demonstrations. 

--

[1] Scaling Instruction-Finetuned Language Models

[2] The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

Reviewer comment (Evaluation):
There is no concrete conclusion from the discussion on reasoning abilities due to insufficient reasoning performance. One interesting discussion is on the generation order of diffusion LM and its difference from auto-regressive LMs, but a concrete conclusion would probably require a benchmark or a probe dataset for a systematic evaluation, showing the advantage of the generation order employed by diffusion LMs.

Author response:
Thank you for your interest and valuable suggestions for a more in-depth study on causal generation order. In our paper, we mention two aspects showing promise in eliciting reasoning ability. 
- Qualitatively, we showed the causal generation order of the models, which indicates correct dependency in reasoning. 
- Quantitatively, in Section 4.5, we found the accuracy of our models on GSM8K can attain a non-trivial improvement from 5.6% to 12.8% after tuning on chain-of-thought data [1] of GSM8K distilled from code-davinci-002. This suggests the inability to perform reasoning should be attributed to insufficient model capability instead of the diffusion language modeling paradigm itself.

We believe our findings encourage effort in further scaling and improving the capability of diffusion language models, which facilitates exploring the concrete benefits of causal generation order with the support of sufficiently strong reasoning and planning ability.

[1] Fu et al. Specializing smaller language models towards multi-step reasoning. ICML 2023.

Reviewer comment (Experiments):
How does the performance of diffusion LM compare to other non-autoregressive language models?

Author response:
Most non-autoregressive models struggle to achieve comparable performance as AR.  For example, RDM is among the state-of-the-art non-AR approaches that perform close to AR LM models for supervised tasks like machine translation, whilist our Diffusion LMs perform generally better than other non-AR LMs with the aid of large-scale pretraining. Moreover, we also demonstrate that our large-scale Diffusion LMs can also offer general-purpose generative abilities like AR LMs, which have not been well examined and verified for previous non-autoregressive approaches.

Reviewer comment (Experiments):
What about the inference computation cost of diffusion LMs, compared to auto-regressive LMs? How does the number of denoising steps affect the performance-computation tradeoff, as it is a crucial parameter in image diffusion models?

Author response:
**(1) Inference computation cost:**
Here are the results of the average latency of generating a sample with `XLM-R-BASE` sized autoregressive and diffusion language models on IWSLT14. The latency of diffusion language models depends on the number of iterations during generation. 
|                      | Ave. latency (s) ↓ | BLEU  |
|----------------------|------------------------------:|-------|
| Diffusion (4 iters)  |                          0.05 | 32.20 |
| Diffusion (10 iters) |                          0.10 | 35.10 |
| Diffusion (25 iters) |                          0.23 | 35.71 |
| Autoregressive       |                          0.33 | 33.30 |
| Diffusion (50 iters) |                          0.45 | 36.25 |


**(2) Performance-computation tradeoff:** Here are the results using different denoising steps of our BASE models finetuned on IWSLT14. Using more sampling steps consistently brings performance improvement while the relative gains diminish. 
The results indicate that **we can use much fewer sampling steps than 50 to maintain most of the performance**.
| # Steps | 1 | 2 |  4 | 10 | 16 | 25 | 35 | 50 | 100 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLEU | 19.65 | 27.95 | 32.20 | 35.10 | 35.37 | 35.71 | 35.48 | 36.25 | 36.22 |

Reviewer comment (Experiments):
The performance comparison in Table 1 is a bit confusing: 1) Are the RDM models trained by the authors? If so, why not evaluate on all three datasets? 2) "The performance of finetuned XLM-R models is competitive or superior to the common encoder-decoder Transformer baseline." may not be so safe to say as the decoder-only models are larger than encoder-decoder models. 3) How is XLM-R (autoregressive) evaluated on IWSLT14, as it is an encoder-only model?

Author response:
Thank you for your questions and sorry for the confusion. We answer these questions respectfully as follows.

1. We trained these RDM baseline models following previous practice in machine translation and summarization using `Transformer-BASE` architecture on WMT and Gigaword, and using `Transformer-BASE-IWSLT` on IWSLT14. the only difference between `Transformer-BASE` and `Transformer-Base-IWSLT` is the intermediate dimension of FFN (2048 vs 1024). 
For the missing Gigaword performance with RDM (`Transformer-BASE`), it has a Rouge-L score of 16.25, which is again competitive to an autoregressive counterpart (v.s. 10.42).  We summarize them together as follows. 
|  | IWSLT14 | WMT14 | Gigaword-10K |
| --- | --- | --- | --- |
| AR (`Transformer-BASE`) | 33.30  |  26.85 | 10.42 |
| RDM (`Transformer-BASE`)  |  32.14 |  26.54 | 16.25 |

2. As you suggested, we conduct a fairer comparison with similar model sizes and configurations. The IWSLT14 performance of the encoder-decoder with the same width and depth as `XLM-R-BASE` is 33.56, similar to the result of 33.30 for `Transformer-BASE-IWSLT14` we reported in Tab. 1. As a result, our conclusion should still hold.

3. The model has the same architecture (i.e., depth, dimension of hidden states, dimension of feedforward layer, embedding size, and number of attention heads) as XLM-R, but the attention mask is the causal mask.

Reviewer comment (Experiments):
In Table 2, why not use zero-shot AR as a reference but use supervised AR?

Author response:
Thank you for your suggestion! The supervised AR is a commonly adopted baseline in previous research, with a performance that satisfies practical needs. We include it as a strong reference to provide an intuition that the BLEU score instruction-tuned models achieve is decent. Again, we are not aiming to beat supervised AR in this paper. 

To compare with zero-shot AR LMs as you suggested, we evaluate the zero-shot performance of Flan-T5 and Flan-XLM-R of different sizes on IWSLT14. Both series of models are instruction-tuned on FLAN2022 collections. The results demonstrate that diffusion LMs are also comparable to instruction-tuned autoregressive models like Flan-T5. We will include these results in our next version. 
| Diffusion LMs | Flan-XLM-R-Base (86M) | Flan-XLM-R-Large (304M) |  | Flan-XLM-R-XL (2.84B) | Flan-XLM-R-XXL (9.7B) |
| --- | --- | --- | --- | --- | --- |
| BLEU | 21.26 | 25.24 |  | 28.13 | 29.59 |
| **AR LMs** | **Flan-T5-Small (44M)** | **Flan-T5-BASE (198M)** | **Flan-T5-Large (705M)** | **Flan-T5-XL (3B)** | **Flan-T5-XXL (11B)** |
| BLEU | 10.80 | 18.52 | 26.05 | 31.12 | 34.41 |

Reviewer comment (Reproducibility):
Do the AR models used in the paper use beam search during decoding?

Author response:
For task-specific finetuning, we follow standard practice and use beam search with beam size=5. As for instruction tuning, we directly cite the results of Flan-T5 from [1], which does not mention the use of beam search. 

[1] Scaling Instruction-Finetuned Language Models.

Reviewer comment (Evaluation):
Given that diffusion LMs use a different receptive field and generation order than auto-regressive LMs, could there be specific types of tasks or domains where diffusion LMs particularly excel or fall short? That could be a valuable discussion (but probably beyond the scope of the paper).

Author response:
Great question! We would like to share some of our insights regarding this. 

We suggest that data that do not conform to a left-to-right generation order will probably benefit from diffusion language models. Typical examples include images, codes, molecules, and protein sequences. In particular, discrete diffusion language models have recently been shown to "beat" typical continuous diffusion models in generating images and videos [1]. Besides, LM-Design [2] has validated the superiority of iterative refinement-based non-autoregressive models in generating protein amino acid sequences given protein structures (a.k.a., protein inverse folding). Given this, diffusion language models should have strong potential in modeling across various modalities beyond natural languages.

-- 

[1] Language Model Beats Diffusion - Tokenizer is key to visual generation. 2023 (under review at ICLR 2024)
> (Note that here in this paper the "language model" is essentially masked LM operating on discrete visual tokens and generating in a mask-predict denoising way, which is conceptually identical to the discrete diffusion LM in our paper)

[2] Zheng et al. Structure-informed Language Models Are Protein Designers. ICML 2023

Reviewer comment (Novelty):
The main method in this paper is similar to existing works: The idea of leveraging pretrained masked models as a knowledge base and starting point for tuning discrete diffusion language models (named as "diffusive adaption") is introduced in [1]. The main methods in this paper (tuning pretrained MLMs with diffusion objective) is also mainly adhering to the work RDM-Diffusion [2]. The "prompt-response" format (partially diffusion on target tokens) essentially adopts the setting of DiffuSeq [3].

Author response:
Thank you for your question. In this work, we are not only presenting a method; more importantly, we are providing a pioneering exploration into the scalability of diffusion language models, or more broadly, non-autoregressive language models, without the inherent risk of consuming substantial computational resources. We would like to emphasize our additional contributions that go beyond the methods outlined in previous works.
- DiffusionBERT[1] empirically finds it feasible to tune masked language models into generative models. We further establish the formal connection between reparameterized diffusion models (RDM) and masked language models in terms of training objectives.
- Furthermore, the application of RDM to advance their methodologies [1,2] results in a commendable boost in competitive performance for this technical routine. This improvement not only facilitates our groundbreaking exploration into scaling diffusion language models but also extends to the broader realm of non-autoregressive language models, all achieved at a significantly lower cost. Our results can inspire further exploration in scaling up non-autoregressive language models, mitigating the previously perceived risk associated with the substantial demand for computational resources.
- For the "prompt-response" format, we consider it a common practice to apply decoder-only language models for various tasks and do not imply it as a key contribution of our paper.

--

[1] He et al. "Diffusionbert: Improving generative masked language models with diffusion models." 2022

[2] Zheng et al. "Structure-informed Language Models Are Protein Designers". In ICML 2023

Reviewer comment (Experiments):
While referring BERT-like diffusion language models as 'decoder-only' models, (GPT-like) AR decoder models (which is currently most popular) is not compared or discussed throughout this study. Besides, the authors need further clarifications on their definitions of 'decoder' models, since we do not generally refer BERT-like models as decoder-only models, which might cause some confusions.

Author response:
Thank you for your questions! 

**(1) Clarifications on the definition of the "decoder-only" model.**
We presented our clarification on the concept of "decoder-only" models in our original submission, that in this paper, the decoder-only architecture, as a counterpart of encoder-decoder architecture, refers to the language model architecture that does not comprise a separate encoder stack to encode contexts/conditions. Under this definition, masked language models (e.g., BERT and XLM-R) are treated as decoder-only models. We will make more clarifications on this in the next version as you suggested. 
 
**(2) Why we used encoder-decoder models (T5) instead of GPT-like decoder-only models?**
- As for comparisons in Table 1, conventional supervised models on machine translation and summarization are encoder-decoder Transformers, which typically outperform decoder-only Transformers. Table 1 supports this that, on IWSLT14, encoder-decoder AR or Diffusion LMs are consistently much better than decoder-only AR or Diffusion LMs. respectively. As such, we listed them there as strong references to demonstrate how well our model's performance is in comparison to commonly-used supervised baselines.

- Similar reasons go with the comparisons in subsequent experiments regarding instruction-following. We consider T5 as a strong reference model because its representativeness outweighs the mismatch resulting from the architectural differences. As previous work suggests, encoder-decoder models like T5 outperform decoder-only models at moderate scales [1][2][3], and the gap diminishes as models scale up. Besides, the T5 series has both open-sourced pretrained models and instruction-tuned models on the FLAN instruction-tuning dataset at different scales similar to XLM-R. So we believe T5 can serve as a valid and strong baseline for the performance of autoregressive models.

--

[1] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

[2] UL2: Unifying Language Learning Paradigms

[3] Examining Scaling and Transfer of Language Model Architectures for Machine Translation

[4] Scaling Instruction-Finetuned Language Models

[5] Larger-Scale Transformers for Multilingual Masked Language Modeling

Reviewer comment (Experiments):
The paper does not fully explore the configurations of Diffusion LMs, such as the number of diffusion steps, the noise level, and the length predictor. It would be valuable to explore how these factors affect the quality and diversity of the generated texts, especially when the inference latency due to the demand on multiple diffusion steps would limit the scaling of diffusion models. For example, would larger models remedy the demand for more steps & generalize better to predicted lengths or opposite?

Author response:
Thank you for your suggestions. We conduct further analyses on these as you suggested. 

**(1) Number of diffusion steps.**
Here are the results using different denoising steps of our models finetuned on IWSLT14. Using more sampling steps consistently brings performance improvement while the relative gains diminish.  The results indicate that we can use much fewer sampling steps than 50 to maintain most of the performance. Additionally, to reach similar performance, larger models require fewer steps (e.g., 10-step LARGE model surpasses the 50-step BASE models). 

| # Steps | 1 | 2 |  4 | 10 | 16 | 25 | 35 | 50 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BASE | 19.65 | 27.95 | 32.20 | 35.10 | 35.37 | 35.71 | 35.48 | 36.25 |
| LARGE | 21.39 | 29.76 | 34.31 | 37.18 | 38.12 | 38.45 | 38.68 | 38.84 |
| XL | 19.55 | 28.85 | 34.09 | 37.95 | 38.99 | 39.60 | 38.88 | 40.08 |

**(2) About length predictor.**
We have tried two different parameterizations for length predictor in our preliminary study, either predicting the absolute lengths or predicting the relative length between target responses and their input prompts. We empirically found for machine translation both strategies performed similarly, while for other tasks, absolute length prediction is generally superior. As a result, we decided to use absolute length prediction for all later experiments. Please note that in this study, our focus is not to seek the best configurations but to demonstrate the effectiveness and possibilities of the Diffusion LMs for general-purpose language learning.

Reviewer comment (Experiments):
The paper lacks some important baselines and comparisons in Table 1. For example, why are there dashes on most AR transformer baselines? Besides, only small AR models (<100M) are compared against the 9.7B diffusion model. Other important baselines (e.g., GPT-structured decoder models, vanilla AR transformers at similar scale to diffusion variants (~80M), and pretrained + AR fine-tune) should be incorporated to compare against the performance of scaled diffusion models.

Author response:
Thank you for your feedback and suggestions!

**(1) About the dashes on most AR Transformers.**
Sorry for the confusion. We trained these RDM baseline models following previous practice in machine translation and summarization using Transformer-BASE architecture on WMT and Gigaword, and using Transformer-BASE-IWSLT on IWSLT14. 

**(2) About other important baselines you suggested.**
In Table 1, we have compared a XLM-R-BASE model trained from scratch in both RDM and autoregressive ways. We also conduct vanilla AR Transformer with similar model size with XLM-R-BASE as you suggested. 
To make it clearer, we summarize the results as follows. 
| Model | achitecture | pretrained | IWSLT14 (BLEU) |
| --- | --- | --- | --- |
| AR  | vanilla encoder-decoder (~86M) | no | 33.56 |
| AR (GPT-like) | XLM-R-BASE (decoder-only, ~86M) | no | 26.07 |
| AR (GPT-like) | XLM-R-BASE (decoder-only, ~86M) | yes | 34.16 |
| Diffusion | XLM-R-BASE (decoder-only, ~86M) | no | 28.79 |
| Diffusion | XLM-R-BASE (decoder-only, ~86M) | yes | 34.10 |

As shown in the table, Diffusion LM attains similar performance with encoder-decoder AR LM but largely outperforms decoder-only AR LM with exact same architecture. A notable difference between the models in these two settings lies in the receptive field. Diffusion LM always has a global receptive field on the conditioning input, whereas AR can only perceive the condition with unidirectional attention if not equipped with an encoder. This supports our motivation to build diffusion language models for their favorable global receptive field.

--

[1] Tay et al. "Ul2: Unifying language learning paradigms." In ICLR 2022.

[2] Zhang et al. "Examining scaling and transfer of language model architectures for machine translation." In ICML 2022.

Reviewer comment (Evaluation):
Why are the performance gaps of using oracle length and predicted length opposite in the two MT tasks in Figure 3? Are there any further investigations or explanations?

Author response:
Generally speaking, results with oracle length are better as more information is provided. But sometimes the predicted length leads to better performance in terms of BLEU score, which is based on n-gram precision, and is biased towards short sentences.

Reviewer comment (Experiments):
On Table 2, what is the vanilla performance of XLM-based diffusion models before fine-tuning on FLAN? Since XLM models already adopt a multilingual pretraining objective, it seems natural that it elicits German knowledge and scales better, since larger models undertake greater-scale multilingual pretraining and have more parameters to store this knowledge.

Author response:
Before finetuning, the XLM-R models are unable to generate a good full sentence, we suggest this is due to only a constant 15% masking ratio during typical MLM pretraining, which is not compatible with generative denoising sampling where operating on various noise/mask levels is required.

We conducted preliminary experiments on this with the 2-shot performances of XLM-R on IWSLT14 De->En before finetuning. The results are shown as follows.
|   | base | large | xl | xxl |
| --- | --- | --- | --- | --- |
| IWSLT14 |  ~0 | 12.12 | 8.91 | 7.78 |

Qualitatively, the BASE model fails to follow instructions and generate German while XL and XXL models tend to repeatedly generate irrelevant tokens like emojis. This suggests that the generative ability of masked language models is elicited by our diffusive adaptation.

Reviewer comment (Evaluation):
Why BBH-NLP seems not following the scaling trend in Figure 4?

Author response:
This is because many questions in BBH-NLP require reasoning to solve. Under the setting of answer-only evaluation, the model performance does not always improve with sizes as demonstrated in [1]. 

--

[1] Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Reviewer comment (Novelty):
The finding that pre-training helps an adapted diffusion model, and that pre-trained model size influences the adapted model’s generalization, is perhaps not surprising. In particular, (a) it is already known XLM-R performance generally improves with scale on several downstream tasks, so the benefit of scale likely comes from the pre-training procedure, rather than the diffusion adaptation and (b) given that pre-training helps over no pre-training, it is not very surprising that _better_ pre-trained models help more.

Author response:
Thank you for your comment! We would like to make clarifications on this as follows:
- **MLM Pretraining does not permit generative capabilities.** Despite pretraining masked LMs, such as XLM-R, improving downstream tasks with scale, these tasks are generally predictive tasks (regression/classification for NLU) as MLMs are typically thought of as encoders. Whether these pretrained MLMs can solve more tasks in general in a generative manner like causal LMs/GPTs, however, remains under-explored and non-trivial to achieve. Without diffusive adaptation as a generative surgery, only MLM pretraining at scale does not help with generative capabilities on its own.
- Although it is common to expect that models will perform better with model scaling and pretraining, **it is imperative not to assume this as a given without the backing of empirical results**. 
In particular, previous work on continuous diffusion language models has found it challenging to fit large-scale datasets [1,2,3]. We suggest that our success in scaling diffusion language models is attributed to both adopting the formalization of discrete diffusion models that fit language tokens well and utilizing pretrained masked language models which are well-tested to be scalable. 
- **Scaling is not only about performance improvement but, more importantly, is about eliciting general-purpose ability.** The major contribution of our study is therefore to show that the Diffusion LMs can also possess promising generative capabilities that can be elicited from large-scale pretrained MLMs by diffusive adaptation, not only for each individual specialized purpose as in DiffusionBERT, but also generalist abilities like instruction following, in-context learning and promise of reasoning, which are nowadays what the community really cares and excited about LLMs. And this is exactly what we think of something new and surprising, and what we are thrilled to share with the community.

[1] Yuan et al. "Seqdiffuseq: Text diffusion with encoder-decoder transformers." 2022

[2] Gao et al. "Difformer: Empowering diffusion model on embedding space for text generation." 2022

[3] Ye et al. "Dinoiser: Diffused conditional sequence learning by manipulating noises." 2023

Reviewer comment (Experiments):
The paper claims to be interested in how diffusion LMs scale with respect to data, but that is not tested in this work. What is tested is how diffusion LM capabilities scale with respect to pre-trained model size and how diffusion LMs respond to instruction-tuning.

Author response:
Our purpose is to explore whether pretraining on large-scale data can also empower Diffusion LMs, like their autoregressive counterparts, we mainly focused on diffusive adaptation from pretrained masked LMs (i.e., XLM-Roberta) rather than building Diffusion LMs from scratch for the purpose of highly efficient proof-of-concept. As we have demonstrated in Section 4.2 and Table 1, pretraining on large-scale data gives rise to significant gains for RDM-based Diffusion LMs over the same models without pretraining (e.g., 34.10 vs 28.79 on IWSLT14). We will leave the analysis of the effects of pretraining on different data scales from scratch in the next version.

Reviewer comment (Experiments):
For Table 1, there is a number of results missing, particularly for XLM-R-BASE (AR), which is perhaps the most comparable portion of the table as it directly compares autoregressive generation to RDM on the exact same model. It is also not clear what the value of having the Encoder-Decoder models there is, since the models are pre-trained differently, have different sizes, and different architectures; the take-away from this comparison is very unclear.

Author response:
**(1) About autoregressive decoder-only baselines.**

Thank you for your suggestion. We actually did these comparisons in Table 1 and explained the findings in footnote 7. However, there may have been some confusion caused by our naming.   In order to make things clearer, we rearrange the results as follows:
| models | architecture | IWSLT14 (BLEU) | Gigaword-10K (Rouge-L) |
| --- | --- | --- | --- |
| AR LM (decoder-only) | XLM-R-BASE | 26.07 | 4.7 |
| Diffusion LM | XLM-R-BASE | 28.79 | 10.01 |

As shown in the table, Diffusion LM sufficiently outperforms AR LM in the decoder-only setting with exact same architecture. A notable difference between the models in these two settings lies in the receptive field. Diffusion LM always has a global receptive field on the conditioning input, whereas AR can only perceive the condition with unidirectional attention if not equipped with an encoder. This supports our motivation to build diffusion language models for their favorable global receptive field.

**(2) About encoder-decoder baselines.**

As for comparisons in Table 1, conventional supervised models on machine translation and summarization are encoder-decoder Transformers, which typically outperform decoder-only Transformers. Table 1 supports this that, on IWSLT14, encoder-decoder AR or Diffusion LMs are consistently much better than decoder-only AR or Diffusion LMs. respectively. As such, we listed them there as strong references to demonstrate that how well our model's performance is in comparison to commonly-used supervised baselines.

Similar reasons go with the comparisons in subsequent experiments regarding instruction-following. We consider T5 as a strong reference model because its representativeness outweighs the mismatch resulting from the architectural differences. As previous work suggests, encoder-decoder models like T5 outperform decoder-only models at moderate scales [1][2][3], and the gap diminishes as models scale up. Besides, the T5 series has both open-sourced pretrained models and instruction-tuned models on the FLAN instruction-tuning dataset at different scales similar to XLM-R. So we believe T5 can serve as a valid and strong baseline for the performance of autoregressive models.

--

[1] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

[2] UL2: Unifying Language Learning Paradigms

[3] Examining Scaling and Transfer of Language Model Architectures for Machine Translation

[4] Scaling Instruction-Finetuned Language Models

[5] Larger-Scale Transformers for Multilingual Masked Language Modeling

Reviewer comment (Experiments):
It is surprising that the number of examples used in in-context learning has little effect on model performance (and sometimes has a very negative effect). It would be very useful to have these plotted against a model that is known to perform well at in-context learning, to see if the same trends hold. As it stands, the results suggest that the diffusion LMs presented are good zero-shot learners, but poor in-context learners.

Author response:
This is actually consistent with the findings of the FLAN series paper [1], which shows that model performance is not sensitive to the number of demonstrations after instruction tuning, and we have mentioned this phenomenon in Section 4.2.2. Note that the most important message here is that instruction tuning helps our Diffusion LMs obtain in-context learning ability, without which we found that a raw pretrained MLM cannot learn from demonstrations. 

As you suggested, we present the few-shot performance of XL sized Flan-T5 and Flan-XLM-R on IWSLT14 here, and will include results on more datasets in our next version. Similar to what we have observed in our manuscript, the performance of both instruction-tuned models changes little even though the number of in-context examples vary.
|  | 0-shot | 1-shot | 2-shot |
| --- | --- | --- | --- |
| Flan-T5-XL (3B) | 31.12 | 31.54 | 31.70 |
| Flan-XLM-R-XL (2.89B) | 28.13 | 27.51 | 29.19 |

--

[1] Scaling Instruction-Finetuned Language Models

[2] The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

Reviewer comment (Presentation):
What is the reason for missing fields in Table 1? Are these rows results obtained from other papers? If so, that should be made clearer.

Author response:
We are sorry for the confusion. We trained encoder-decoder baseline models following previous practice in machine translation and summarization using Transformer-BASE architecture on WMT and Gigaword, and using Transformer-BASE-IWSLT on IWSLT14. The blankets are because the dataset and architecture mismatches in common practice, although the only difference is the intermediate dimension of FFN (2048 vs 1024)