Paper: Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR-2024)

Reviewer comment (Experiments):
**Limited Evaluation;** While the experimental results include diverse evaluation tasks, the fine-tuning datasets are limited to just two tasks: NLI and Trnaslation.

Author response:
We thank the reviewer for valuable suggestions and comments. We address your questions below in detail.

**Regarding experiments on finetuning on more tasks.** Sorry for the confusion. In Table 10 of the appendix, we have shown experimental results on two additional finetuning datasets including openbookQA, which is a QA dataset and SNLI as another NLI dataset. Due to space limit, we put these results in Appendix,. The results are consistent with our main experiments in the paper. Also based on our gradient analysis given in section 3.2-3.3, we can see that format specialization happens for finetuning tasks with any well-defined output syntax. The reason is that the gradient associated with such syntax is present consistently in all finetuning data points, resulting in a strong gradient signal. Lots of  fine tuning tasks in the NLP community today contain such syntax elements.

We would deeply appreciate it if the reviewer finds that these clarifications are helpful and strengthens our manuscript, and raise the score accordingly.

Reviewer comment (Experiments):
Why did the authors limit their experimentation to only these two tasks? Is it implied that format specialization does not occur in other tasks, such as question answering and summarization?

Author response:
We thank the reviewer for valuable suggestions and comments. We address your questions below in detail.

**Regarding experiments on finetuning on more tasks.** Sorry for the confusion. In Table 10 of the appendix, we have shown experimental results on two additional finetuning datasets including openbookQA, which is a QA dataset and SNLI as another NLI dataset. Due to space limit, we put these results in Appendix,. The results are consistent with our main experiments in the paper. Also based on our gradient analysis given in section 3.2-3.3, we can see that format specialization happens for finetuning tasks with any well-defined output syntax. The reason is that the gradient associated with such syntax is present consistently in all finetuning data points, resulting in a strong gradient signal. Lots of  fine tuning tasks in the NLP community today contain such syntax elements.

We would deeply appreciate it if the reviewer finds that these clarifications are helpful and strengthens our manuscript, and raise the score accordingly.

Reviewer comment (Experiments):
I couldn’t find a satisfactory explanation in the paper regarding the absence of experimental results from fine-tuning on other tasks.

Author response:
We thank the reviewer for valuable suggestions and comments. We address your questions below in detail.

**Regarding experiments on finetuning on more tasks.** Sorry for the confusion. In Table 10 of the appendix, we have shown experimental results on two additional finetuning datasets including openbookQA, which is a QA dataset and SNLI as another NLI dataset. Due to space limit, we put these results in Appendix,. The results are consistent with our main experiments in the paper. Also based on our gradient analysis given in section 3.2-3.3, we can see that format specialization happens for finetuning tasks with any well-defined output syntax. The reason is that the gradient associated with such syntax is present consistently in all finetuning data points, resulting in a strong gradient signal. Lots of  fine tuning tasks in the NLP community today contain such syntax elements.

We would deeply appreciate it if the reviewer finds that these clarifications are helpful and strengthens our manuscript, and raise the score accordingly.

Reviewer comment (Experiments):
Why do the authors use only two datasets for fine-tuning?

Author response:
We thank the reviewer for valuable suggestions and comments. We address your questions below in detail.

**Regarding experiments on finetuning on more tasks.** Sorry for the confusion. In Table 10 of the appendix, we have shown experimental results on two additional finetuning datasets including openbookQA, which is a QA dataset and SNLI as another NLI dataset. Due to space limit, we put these results in Appendix,. The results are consistent with our main experiments in the paper. Also based on our gradient analysis given in section 3.2-3.3, we can see that format specialization happens for finetuning tasks with any well-defined output syntax. The reason is that the gradient associated with such syntax is present consistently in all finetuning data points, resulting in a strong gradient signal. Lots of  fine tuning tasks in the NLP community today contain such syntax elements.

We would deeply appreciate it if the reviewer finds that these clarifications are helpful and strengthens our manuscript, and raise the score accordingly.

Reviewer comment (Experiments):
The limitation of the paper is only evaluated on encoder-decoder models. It would be interesting to also evaluate the approach on decoder-only models.

Author response:
We thank the reviewer for valuable suggestions and comments. We address your questions below in detail.

**Regarding experiments on more models (different base models including decoder-only model).** We have done additional experiments on different base models beyond mT5 shown in the main paper, including T5XXL, T5XL and PaLM-8B. Due to limited space in the main paper, we put these results in Table 11, 12, 13 in Appendix. The PaLM-8B model we used in the additional single-task experiments is a decoder-only model. We report the results in Table 13 in the Appendix. As we can see from these results, the additional results are consistent with our main experiments on the mT5 XXL model, where ProMoT outperforms prompt-tuning in-domain and outperforms standard finetuning out-of-domain. In particular, as an improved finetuning method, ProMoT shows significantly less forgetting of in-context capabilities compared to standard finetuning and may even improve over out-of-domain but semantically related tasks compared to the pretrained model.

Reviewer comment (Experiments):
Do you have any difference trends on different base models? and also different model types (e.g., decoder-only models)?

Author response:
We thank the reviewer for valuable suggestions and comments. We address your questions below in detail.

**Regarding experiments on more models (different base models including decoder-only model).** We have done additional experiments on different base models beyond mT5 shown in the main paper, including T5XXL, T5XL and PaLM-8B. Due to limited space in the main paper, we put these results in Table 11, 12, 13 in Appendix. The PaLM-8B model we used in the additional single-task experiments is a decoder-only model. We report the results in Table 13 in the Appendix. As we can see from these results, the additional results are consistent with our main experiments on the mT5 XXL model, where ProMoT outperforms prompt-tuning in-domain and outperforms standard finetuning out-of-domain. In particular, as an improved finetuning method, ProMoT shows significantly less forgetting of in-context capabilities compared to standard finetuning and may even improve over out-of-domain but semantically related tasks compared to the pretrained model.

Reviewer comment (Experiments):
Only on one model.

Author response:
**Regarding experiments on more models.** Sorry for the confusion. We have done additional experiments on other models beyond mT5, including T5XXL, T5XL and PaLM-8B. Due to limited space in the main paper, we put these results in Table 11, 12, 13 in Appendix. As we can see from these tables, results on these additional models are consistent with our main experiments on the mT5 XXL model, where ProMoT outperforms prompt-tuning in-domain and outperforms standard finetuning out-of-domain. In particular, as an improved finetuning method, ProMoT shows significantly less forgetting of in-context capabilities compared to standard finetuning and may even improve over out-of-domain but semantically related tasks compared to the pretrained model.

Reviewer comment (Experiments):
Finetuned on only two datasets.

Author response:
**Regarding experiments on more datasets.** We have included experiments on two additional finetuning datasets (SNLI and OpenBookQA) in Appendix Table 10 for the mT5 XXL model. The results are consistent with our main experiments in the paper.

Reviewer comment (Presentation):
In Table 3,4, It may be worthwhile to put more highlights on the most relevant datasets, e.g. CB and WiC for RTE, WMT16 en>de, en>ro for WMT14 en>fr.

Author response:
**Regarding highlighting similarity tasks.** Thanks for the suggestions. We have put some texts in the tables’ captions to highlight the improvement over the similar tasks.

Reviewer comment (Experiments):
Prefix + Parameter tuning baseline seems to be missing. What happens if you do both prefix and parameter tuning but don't use the prefix embedding for other tasks. This should be done in both single task and multi-task setting.

Author response:
**Regarding the Prefix + Parameter tuning baseline.** Thanks for the comments. We have included a ‘joint fine-tuning’ setting in the ablation study (Table 6), which trains the soft prompt and the main model jointly exactly as the comments suggested. In evaluation, we only use the main model and remove the soft prompt on evaluation tasks with different formats. As we can see from the table, the prefix + parameter tuning joint training still suffers from significant loss of general in-context learning performance, as the format learning can not be separated into the prompt if we train the two parts jointly.

Reviewer comment (Experiments):
It would be good to include a prefix-tuning column. This would have some performance for the source task and same performance as the pre-trained model on other tasks.

Author response:
**Regarding the prefix-tuning column.** Thank you for your suggestion on the table organization. For the source tasks, we report the in-domain performance of prompt-tuning in Table 2, showing that prompt-tuning is indeed inferior compared to standard finetuning or ProMoT in-domain. For evaluation on other tasks, exactly as the reviewer mentioned, since prompt tuning does not touch the parameters of the pretrained model, it has the same few-shot performance on evaluation tasks as the pretrain model where we remove the soft prompt. We have also made this clear to the audience in the footnotes of Table 3 and 4. We hope these would clarify the prompt tuning results well.

Reviewer comment (Experiments):
It is hoped that the prefix-tuning stage learns the format. It would be good to empirically test this hypothesis. One simple experiment is to check if the prefix embedding outputs the source task format on other tasks.

Author response:
**Regarding evaluating prefix-tuning on other tasks.** Thanks for the suggestions. As requested by the reviewer, we evaluated the performance of using the prompt trained from the English to French translation task WMT14 En-Fr using prompt tuning on other translation tasks with different language pairs. We show some examples below, we add a 1-shot prompt before the actual input in evaluation denoted by [1-shot prompt].

As we can see from the table, when we use the prompt from WMT14 En-Fr to test on En-De and En-Ro translation tasks, the predictions are still in French, which implies that the format information of outputting in French is learned by the prompt.

| Input                                                                                                                                                                                                                      | Target                                                                                                                                                                            | Prediction with the soft prompt trained from En-Fr            |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|
| [1-shot example omitted…]   Translate English to German: The first series of The Lenny Henry Show aired in 1984, and in the Nineties he was known as, among other things, chef Gareth Blacklock in the comedy series Chef! | Die erste Reihe der Lenny Henry Show ging 1984 auf Sendung, und in den Neunzigern war er unter anderem als der K\u00fcchenchef Gareth Blacklock in der Comedyserie Chef! bekannt. | Nous souhaitons aider nos clients \u00e0 r\u00e9ussir.        |
| [1-shot example omitted…]  Translate English to German: I know Georgy won't go.                                                                                                                                            | Ich wei\u00df, dass Georgij nicht hingehen wird.                                                                                                                                  | Cela ne signifie pas une Europe minimaliste \u2013 et je veux |


| Input                                                                                                                             | Target                                                                                                                      | Prediction with the soft prompt trained from En-Fr                                                    |
|-----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [1-shot example omitted…]  Translate English to Romanian: People using ad-blocking software are likely to run into a few hitches. |  Persoanele care folosesc programe software de blocare a reclamelor \u00eent\u00e2mpin\u0103 c\u00e2teva constr\u00e2ngeri. | Les personnes qui utilisent des logiciels de blocage des annonces |
| [1-shot example omitted…] Translate English to Romanian: Brent futures were up 61 cents Wednesday at $48.36 per barrel.           | Cot\u0103rile Brent au crescut miercuri cu 61 de cen\u021bi, ajung\u00e2nd la 48,36 USD per baril.                          | Les futures sur le Brent ont gagn\u00e9 61 centimes mercredi      |

Reviewer comment (Evaluation):
Results on Palm 8B in the appendix don't look as promising and they should not be in the appendix but in the main paper. The approach seems to show worse performance than pre-trained model on other tasks after using the proposed approach. Would also be good to include 4-shot results here.

Author response:
**Regarding the performance on PaLM.** Thanks for the comments. We would like to highlight that ProMoT, as an improved finetuning method, still vastly outperformed prompt-tuning in-domain and significantly out-performed standard finetuning out-of-domain, providing a very attractive trade-off point between the two extremes. However, we acknowledge that our results on PaLM models are somewhat more exploratory, which is why they are placed in the appendix. Factors such as the effectiveness of prompt tuning on these models will affect their ability to absorb the format part of the finetuning task. This is an interesting topic that warrants a dedicated examination in a future work. 

We would deeply appreciate it if the reviewer finds that these additional experiments and clarification are helpful and strengthens our manuscript, and raise the score accordingly.