Paper: Are Bert Family Good Instruction Followers?  A Study on Their Potential And Limitations (ICLR-2024)

Reviewer comment (Novelty):
I'm not exactly sure what's the core contribution of the paper - it reads too much like an experiment report in this current draft. My biggest takeaway is that you can also do instruction tuning on encoder-only models. But why is that particularly "surprising"? To be fair, I'd be surprised it actually works much better than decoder-only or encoder-decoder models because then it means that the popular approach is wrong and they should go back to encoder-only models instead. But based on your experiments, it's not the case, decoder-only and encoder-decoder models still seem better at similar sizes.

Author response:
**To Concern 1:** "What's the core contribution..."  
**Response:** We agree with you that this version merely pointed out and preliminarily analyzed some of the potential and limitations of existing Encoder-only LMs in completing various generation tasks in an instruction-following manner. There is still much work to do in the future (as another reviewer said "It can potentially lead to another series of research.'').

Though preliminary, this paper still provides the following observations to the community.

- It is a common impression that Decoder-only and Encoder-Decoder models are good at following instructions in generating the expected output for various tasks, including understanding, extraction, generation, etc. We demonstrate that Encoder-only also has the potential to follow instructions and complete various tasks in a generation manner.
- The BERT family plays a vital role in the development of pre-trained LMs and are still strong competitors in many scenarios, e.g., understanding and information extraction. It is might not surprising that the BERT family can complete simple classification and understanding tasks with zero-shot prompt learning, given proper task format and subtle label name search strategy. However, no existing works provide suggestions and observations on whether the BERT family can follow instructions and complete various tasks in a generation manner. This paper fills this blank.
- As discussed in Question 1, Encoder-only models cannot directly complete various tasks in a generation manner. We provide a basic method with details that allow for such an exploration.
- We present the key factors to make Encoder-only models competitive with the other two variants and disclose their limitations, which might promote further study.
- From the aspect of non-autoregressive generation, this is the first work that builds an instruction following models that can generalize across tasks and languages based on pre-trained LMs.

We also agree that "It would be truly surprising ...". But we also believe the future success of any possible alternatives of existing methods/models can start from "potential", .e.g., better than mT0 on XCOPA tasks, 3.1x faster than AR models.

Last, we have removed "surprisingly" from our paper.

Reviewer comment (Experiments):
The experiments are not exactly controlled in the sense that, your baselines - BLOOMZ-3B and mT0-3.7B, are not even based on the same backbone model as Instruct-XMLR. This means that many confounders (e.g., pretraining data / steps) exist for making any scientific conclusions based on the experiments.

Author response:
**To Concern 2:**"The experiments..."  
**Response:** We agree that "many confounders exist ...". The trends of close-source and soaring computation costs of LLMs make it difficult (sometimes impossible) to exactly control variables in experiments. To avoid any misunderstanding in our observations, we use"potential, risks, possible, etc" throughout the whole paper if necessary. On the other side, empirical observations and suggestions without fine-grained and strict control have also promoted the development of LLMs in the past year.

Reviewer comment (Experiments):
Why don't you also include an XMLR without instruction-tuning baseline so that we can directly see the relative improvement coming from instruction tuning?

Author response:
**To Question 1:** "Why don't...".  
**Response:** We agree that providing the results of XMLR without instruction-tuning can clarify whether the relative improvement comes from instruction tuning. However, it is difficult to collect the evaluation results of XMLR-w/o Instruction tuning owing to its poor performance. We provide several examples below to give an intuitive explanation (we italicize the content of the prompt).
You can find more examples in ***Appendix A.6*** in the revised paper. 

We find that XML-R fails to complete these tasks without instruction tuning, i.e., it generates the special tokens [PAD] in XNLI and unrelated texts in machine translation.

| | |
|:- | :-:|
|XNLI||
|Source|*Suppose* "And he said, Mama, I'm home." *Can we infer that* "He didn't say a word."? |
|Target|No|
|Instruct-XMLR|No|
|XML-R|[PAD]|
|Machine translation|
|Source|*Translate the following text from German to English:* Auch Fast Food ist erlaubt.|
|Target|Fast food is also permitted.|
|Instruct-XMLR|Fast food is acceptable.|
|XML-R|Translate this page:|
||

Reviewer comment (Writing):
Some minor issues regarding citation formats (e.g. "Following the previous work Muennighoff et al. (2022), we evaluate the model’s ability of task" is not directly readable, may need to include parenthesis)

Author response:
Thanks a lot for pointing out our mistakes. We have fixed the citation format issues in the latest paper version.

Reviewer comment (Theory):
Would the authors agree that the benefit in text generation mainly comes from the next-token prediction formulation itself, and not necessarily the decoder architecture? One such example may be [1]: RWKV: Reinventing RNNs for the Transformer Era.

Author response:
Thank you for acknowledging the value of this paper!  
We are not sure how much benefit the transformer decoder architecture brings. However, we do have some empirical observations about the effects of model architecture and the decoding paradigm.  
- Though the community of decoder-only structures is much larger than other architectures (owing to their merits of sample-efficiency, structure simplicity, ease of monitoring during pre-training, etc.), other architectures like transformer Encoder-Decoder (e.g., FlanT5-UL2 [1], AlexaTM [2]), RWKV [3], Convolutional Seq2Seq [4] can also achieve very promising generation performance with the next-token prediction formulation.  
- The next-token prediction (auto-regressive) decoding paradigm is one of the most effective strategies in various generation tasks, which is usually better than other paradigms, such as semi-autoregressive and non-autoregressive.  

We firmly believe there exist alternative architectures. There is no theoretical guarantee to prove that the existing decoder-only structure is optimal or a good sub-optimal.  

We think the most three import factors in self-supervised pre-training are data, model capacity, and objectives, where 1) data decides the patterns to be discovered and composed, 2) model capacity corresponds to the capability to memorize, manipulate, and generalize existing patterns, 3) the objectives correlate the learning efficiency for different patterns. In other words, the model capacity might be more important for LLMs than the structure.  

BTW, we also believe many valuable techniques can be started from “competitive” (like RWKV) or “potential”.

Reviewer comment (Experiments):
The method is not tested on longer sequence generation. The classification tasks don't need a longer output sequence. And machine translation fits the non-auto-regressive methods. Need to test it on the tasks like dialogue, summarization, etc.

Author response:
**To Comment 1:** "Longer sequence generation tasks like dialogue, summarization, etc."  
**Response:** Exactly! It is essential to discuss the performance of the introduced method on longer sequence generation tasks. 
Besides machine translation, *we further provide evaluation results on **5** additional sequence generation tasks with **6** datasets* (with longer output sequences than most previously reported tasks).  We summarize these tasks and datasets below. *More details of these tasks and datasets can be found in **Appendix A.4*** of our revised paper.
| Tasks     |      Datasets            |
| - | - |
| Text Summarization     | XSUM, Gigaword |
| Text Simplification        | WIKI-AUTO |
| Paraphrase Generation    | QQP |
| Controlable Generation | COMMONGEN|
| Dialogue Generation          | PersonaChat |
|  |  |

We present the experimental results below, along with ***preliminary analysis***. We give the variance (this number denotes the performance range, i.e., the performance boundary) of ***5 different prompts*** in the bracket. Since length prediction plays a key role in language generation tasks in non-autoregressive models, we also report the results of Instruct-XMLR decoded ***with ground truth length (w/ GL)*** to explore the effects of length prediction.

R-1/2: ROUGE-1/2, B-2: BLEU-2, D-2: Distinct-2
| Dataset|XSUM |Gigaword|WIKI-AUTO|QQP|COMMONGEN|PersonaChat|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|Score| R-1/R-L| R-1/R-L| B-2/D-2|B-2/D-2|B-2/D-2| B-2/D-2|
|BLOOMZ-3B | 35.12/27.97| 31.45/28.94 |41.90/0.57|38.24/0.54|35.20/0.44|16.39/0.39|
|mT0-3.7B| **37.85**/**30.28**| **34.31**/**31.54** | 55.92/0.64|51.93/0.63| 31.86/0.36|9.17/0.39|
|Instruct-XMLR-XL|35.62/27.90|30.64/28.52|47.91/0.65|43.50/0.67|21.68/**0.52**|2.88/0.52|
|Instruct-XMLR-XL w/ GL| 36.00/27.84|32.60/29.85|**56.93**/**0.76**|**60.02**/**0.69**|**35.44**/0.48|**17.30**/**0.53**|
| | | | ||||

In general, we can find that:
- The results in most tasks are consistent with what we mentioned in the original paper, i.e.,  Instruct-XMLR-XL outperforms BLOOMZ-3B on 4 out of 6 tasks (with the left 2 tasks comparable) and underperforms mT0-3.7B without target length. 
- Length prediction significantly impacts some specific tasks where there is no strong alignment relationship between source length and target length (e.g., COMMONGEN and PersonaChat). Since the BLEU score is sensitive to length, length prediction leads to a serious decline compared to decoding with ground truth length.
- Instruct-XMLR can generate more diverse texts for all text generation tasks due to the non-autoregressive generation manner, i.e., unlikely to be trapped by common n-gram patterns.

Reviewer comment (Evaluation):
It would be better to analyze how stable the method is. How do the prompt templates affect the performance. Need to report the variance or significant test.

Author response:
**To Comment 2:** "To analyze how stable the method is. Need to report the variance or significant test."  
**Response:** We are sorry to miss it in the previous version. In our revised PDF, we supplement the difference of various prompts and the correlated analysis in ***Appendix A.3***.

Reviewer comment (Experiments):
The proposed method cannot handle fewshot prompt learning.

Author response:
**To Comment 3:** "The proposed method cannot handle fewshot prompt learning."  
**Response:** We have tried to make up for this flaw by including some few-shot instruction data for training rather than the only zero-shot instruction data we adopted in our paper. Unfortunately, this doesn't bring any profit. Part of the results are given below, where more details can be seen in ***Appendix A.5***.  
" ***w/zs** denotes the model trained on zero-shot instruction data, **w/fs** denotes the model trained on mixed zero/few-shot instruction data, **w/ICL4** denotes we adopt four examples in the demonstration for few-shot prompt learning.* ''
- Results for seen languages.
  | Model|XCOPA|XNLI|XWinograd|
  |:-|:-:|:-:|:-:| 
  | Instruct-XMLR w/zs|0.646|0.518|0.582 |
  | Instruct-XMLR w/zs w/ICL4 | 0.548 | 0.414 | 0.547 |
  | Instruct-XMLR w/fs | 0.642 | 0.489 | 0.565 |
  | Instruct-XMLR w/fs w/ICL4 | 0.571 | 0.442 | 0.553 |
  ||||

Reviewer comment (Experiments):
Have you tried to use a larger model?

Author response:
**To Question:**  
We are now experimenting with the biggest version of the BERT Family, the 11B XMLR-XXL model, with only around 1/2
data of the original Instruct-XMLR-XL model being processed till now. We present the intermediate results below to give intuitive analysis.

- Results for unseen languages.
  | Model  |  XCOPA | XNLI | XWinograd |
  |:-|:-:|:-:|:-:|
  | Instruct-XMLR-XL | 0.599 | 0.466 | 0.539 |
  | Instruct-XMLR-XXL| 0.593 | 0.523 | 0.562 |
  ||||

We can find that: 
- With only 1/2 instruction data, Instruct-XMLR-XXL already outperforms Instruct-XMLR-XL on XNLI and XWinograd, indicating the positive effects of larger model size.  
- Compared with Instruct-XMLR-XL, Instruct-XMLR-XXL seems more effective for language generalization.