Paper: Understanding Catastrophic Forgetting in Language Models via Implicit Inference (ICLR-2024)

Reviewer comment (Theory):
The paper's main weakness is the main assumption being restrictive and artificial, therefore failing to reflect the behavior of language models in most realistic settings. The main assumption of the paper (Equation 7) is that the input space of different tasks overlaps, so the model has to do probabilistic inference on the task identity. Both the proposed synthetic regression task and the ICL examples have this overlap. The ICL examples are purposefully designed to make it confusing by including two different instructions. Such examples are adversarial in nature and unlikely to exist in real-world applications of language models, where the task is usually clearly specified in the prompt. For example, the input of the summarization task "Please summarize the following passage: [text]" is very unlikely to be confused with the input of a translation task "Please translate the following passage to English: [text]" given a random passage in place of [text].

Author response:
1. We would like to clarify that while the input space overlaps, the optimal mixture regression has close to zero “task ambiguity”---the optimal regressor can identify continuous vs discrete with near perfect accuracy even for a small number of exemplars. (Appendix C.5). However, the fine-tuned transformer is not optimal and performs imperfect task inference, which manifests as task ambiguity that is resolved via conjugate prompting. 
2. We would also like to clarify that we prepend an instruction such as “Repeat the input” or “Capitalize every character” in our ICL experiments, so again, there is no inherent ambiguity. However, we do agree that this setup is somewhat artificial, and was intended as a sanity check that these phenomena do appear in large-scale real world LLMs as well.
3. We have added a new experiment that is in a more natural and commonly studied setting of forgetting. We consider Code LLaMa (LLaMa-2 fine-tuned on code) and see that there is a drop in performance on MNLI when compared to LLaMa-2, which can be recovered via conjugate prompting. We have more details below, as well as in Appendix D.

Reviewer comment (Evaluation):
It can also be argued that when the input space of different tasks overlaps, the problem itself is ill-posed as there is not a single correct answer for inputs from the intersection. The failure of finetuned models thus may come from the ambiguity of the task itself rather than from catastrophic forgetting.

Author response:
Even if the problem is ill-posed, we note that the pre-trained model has a consistent preference for performing the task of interest on the input space in our three setups (ridge regression, ICL, harmful generation). Conjugate prompting hints that this drop comes from changing the model’s task inference rather than degrading the capability. Regardless of initial task ambiguity, the fine-tuned model faces even worse ambiguity from its changed task inference.

Reviewer comment (Experiments):
The synthetic experiments are interesting, but they do relatively little as evidence that this is what should be happening in large language models. I think they are useful, but given the fact that somewhat limited experiments were conducted on true LLMs, and claims are focused in that direction, it feels a little bit lopsided in terms of focus.

Author response:
We want to thank the reviewer for the suggestion to try less artificial tasks. In light of this, we validate conjugate prompting on XNLI, a multi-lingual variant of the MNLI task from GLUE. We observe that Code-LLaMa, a code-based fine-tune of LLaMA-2, displays catastrophic forgetting with an >8% accuracy drop on the English variant of this benchmark. However, with conjugate prompting into French, Spanish, and German, we see a less than 2% change in model performance, supporting our claim. In fact, we even see a slight increase in French and Spanish, possibly due to increased reasoning capabilities from code that are not suppressed by incorrect task inference [Ma et al, 2023](https://huggingface.co/papers/2309.16298). In this scenario, using Code LLaMa in Spanish gets higher performance than using LLaMa 2 in English. We discuss these new results in detail in Appendix D.

Reviewer comment (Novelty):
“As noted in prior work” on page 3, should really cite other work.

Author response:
We appreciate the reviewer’s suggestions to improve the clarity of our paper and have made the following modifications.
- We have now cited the prior work alluded to on Page 3

Reviewer comment (Writing):
You never identify IF as “Instruction Following”, while this is guessable from context, it took me a few minutes to figure this out and I recommend you introduce it properly.

Author response:
We appreciate the reviewer’s suggestions to improve the clarity of our paper and have made the following modifications.
- We now identify IF as instruction following at the start of Section 4.1

Reviewer comment (Experiments):
In Section 2, the authors mentioned that the trade-off is less significant as the dataset size increases. Is this phenomenon consistent with the further expansion of the dataset, or is it merely an incidental occurrence specific to some certain smaller models?

Author response:
We believe our findings are consistent across data, model scale, and various tasks as evidenced by our experiments across linear regression and natural language (with 7B LLaMa and ChatGPT 3.5). To better demonstrate this, we provide extra experiments for larger models (Appendix C.6) and less/more pretraining data (Appendix C.7). In all three settings, we find that catastrophic forgetting is still a problem and conjugate prompting still improves performance on the continuous distribution. When we scale models, the trends stay roughly the same, though the base performance of the larger pretrained model is slightly stronger on the discrete distribution. We encourage you to look at these new sections for more details addressing these concerns!

Reviewer comment (Experiments):
As the authors pointed out in the limitations section, this method would benefit from further validation and application in various domains and tasks involved.

Author response:
We want to thank the reviewer for the suggestion. In light of this, we validate conjugate prompting on XNLI, a multi-lingual variant of the MNLI task from GLUE. We observe that Code-LLaMa, a code-based fine-tune of LLaMA-2, displays catastrophic forgetting with an >8% accuracy drop on the English variant of this benchmark. However, with conjugate prompting into French, Spanish, and German, we see a less than 2% change in model performance, supporting our claim. In fact, we even see a slight increase in French and Spanish, possibly due to increased reasoning capabilities from code that are not suppressed by incorrect task inference [Ma et al, 2023](https://huggingface.co/papers/2309.16298). In this scenario, using Code LLaMa in Spanish gets higher performance than using LLaMa 2 in English. We discuss these new results in detail in Appendix D.

Reviewer comment (Experiments):
In Section 2, the authors fine-tuned a GPT-2 model and explored the phenomenon of trade-offs. However, it remains unclear whether this phenomenon is consistent across larger-scale language models or exhibits significant variations in performance improvements for different tasks, especially during the fine-tuning process. Are these differences in trade-offs related to the complexity or similarity of the tasks involved?

Author response:
We believe our findings are consistent across data, model scale, and various tasks as evidenced by our experiments across linear regression and natural language (with 7B LLaMa and ChatGPT 3.5). To better demonstrate this, we provide extra experiments for larger models (Appendix C.6) and less/more pretraining data (Appendix C.7). In all three settings, we find that catastrophic forgetting is still a problem and conjugate prompting still improves performance on the continuous distribution. When we scale models, the trends stay roughly the same, though the base performance of the larger pretrained model is slightly stronger on the discrete distribution. We encourage you to look at these new sections for more details addressing these concerns!

Though our hypothesis explains the behavior across all settings we consider, the exact trends will depend on confounding variables such as the complexity and similarity of tasks. We believe there is exciting future work in understanding the precise relationship between the fine-tuning data and the resulting model’s performance building off the viewpoint proposed in this work.