Revisiting Catastrophic Forgetting in Large Language Model Tuning (2024)

Abstract: Catastrophic Forgetting (CF) means models forgetting previously acquired knowledge when learning new data. It compromises the effectiveness of large language models (LLMs) during fine-tuning, yet the underlying causes have not been thoroughly investigated. This paper takes the first step to reveal the direct link between the flatness of the model loss landscape and the extent of CF in the field of LLMs. Based on this, we introduce the sharpness-aware minimization to mitigate CF by flattening the loss landscape. Experiments on three widely-used fine-tuning datasets, spanning different model scales, demonstrate the effectiveness of our method in alleviating CF. Analyses show that we nicely complement the existing anti-forgetting strategies, further enhancing the resistance of LLMs to CF.

## introduction instruction fine-tuning is key to improving the capabilities and controllability of large language models (llms) @xcite , which have already demonstrated strong performance in various tasks @xcite @xcite . one major obstacle to tuning llms is catastrophic forgetting (cf, @xcite , which means llms forget prior knowledge when learning new data. recent works have provided substantial evidence confirming the negative impact of cf on llms, e.g., @xcite empirically show that the fine-tuned model is even worse than their foundation counterpart on several tasks, and @xcite reveal the dropped general performance of llms after developing their agent capabilities. understanding the effectiveness of llms during fine-tuning is important for downstream tasks, however, the underlying causes of cf remain largely unexplored. there are two types of work addressing * corresponding author. cf, from data and model perspectives, respectively. scialom et al. (2022) propose continual learning with rehearsal in instruction tuning, though its effectiveness varies with task selection. @xcite indicate that both continual learning and weight averaging (wise-ft) effectively preserve generality. while the above techniques are somewhat successful, they require expensive extra dataconstructing and training costs, and are even sometimes impractical in llms, because a) the data cards of many pretrained models are unclear @xcite , making rehearsal unfeasible, and b) anti-forgetting training brings an unstable and expensive training process @xcite . in this work, we turn to finding a cheap, stable, and orthogonal solution to alleviate the cf in tuning llms. in particular, we ❶ reveal the high correlation between the extent of cf and the flatness of the loss landscape @xcite , ❷ mitigate the cf in llms by flattening the lls from optimization perspective, ❸ show the complementarity between our method and existing anti-forgetting works. specifically, we designed three probing analyses to achieve ❶, and found that a flatter lls could reduce the severity of cf. based on our observation, we introduced "sharpness-aware minimization" (sam, @xcite to flatten the model lls to approach ❷. for ❸, we found that our introduced optimizer nicely complements a series of anti-forgetting methods, including rehearsal @xcite and wise-ft @xcite . our contributions are as follows: we investigate the relationship between cf and the flatness of lls through three analyses: i) loss landscape visualization, ii) flatness degree of lls, and iii) general task performance, where the visualized comparison of i) and results of ii) & iii) are discussed in § 2.1 and § 2.2, respectively, to intuitively and quantitatively reveal their correlation. we simulate continue-learning scenarios with three models: llama2-7b @xcite tuned on 1) "alpaca @xcite )", 2) "@xmath0, and 3) "@xmath1, where "→" means continue-train on the following dataset and the data gaps are gradually increased. for the experimental details, please refer to the appendix a.1.1. ## improving cf with sharpness-aware optimization motivated by the above findings, we naturally consider introducing the sharpness-aware minimization (sam, @xcite method to flatten the model loss landscape, thereby mitigating the cf problem. given model weights as @xmath2, loss function as f , and a training dataset @xmath3, y i )} n @xmath4.i.d. sampled from the distribution d, the working mechanism of sam can be formulated as follows: further utilizing taylor series expansion, ϵ can be approximate as ρ∇ w f (w)/∥∇ w f (w)∥ 2 . in case, eq. 1 can be simplified as: eq. 2 could be solved through a two-step gradient descent process. firstly, the perturbation ϵ is computed by ρ∇ w f (w)/∥∇ w f (w)∥ 2 . following this, the second phase of gradient descent performs the actual update of the weights. ## experimental setting dataset to validate the capability of sam on diverse datasets, we conduct experiments on four widely-used instruction fine-tuning datasets: alpaca @xcite , @xcite , metamathqa @xcite , and open-platypus @xcite . for detailed introduction, please refer to the appendix a.1.1. ## experimental results we demonstrate the superiority of our proposed method through four experimental analyses: for different cf settings (alpaca as the baseline, difficulty rank: @xmath5, their performance on evaluation tasks experiences a significant decline (∆ 1 cf: sharegpt: 11.71%; open-platypus: 12.90%; meta-math: 13.10%), confirming that different degrees of cf occur in the fine-tuning phase. with the introduction of sam, the cf is markedly mitigated and the performance is comparable to baseline. notably, the tab. 2 shows the results in balanced training cost, i.e., (w/o) @xmath6. given the need for a fair comparison and to balance computational cost of sam, the number of epochs for training differs between sam (w/o) and (w/). it is crucial to present the sam (w/o) @xmath7. 3 to uphold our experimental rigor. results show sam (w/o) @xmath8. remarkably, sam (w/) @xmath9, highlighting our method's superiority. another important concern is preserving capability within the domain. our approach effectively alleviates cf while minimally impacting the acquisition of new knowledge, which is supported on the open-platypus dataset. as shown in tab. 4, it is clear that sam mitigates cf with little impact of specific domain performance in llms (e.g., ∆ avg: sam (w/o): 10.7%; sam (w/): 11.2%). ## related work in continual learning, approaches to mitigate cf have been extensively explored, including a) replaybased methods for knowledge retention @xcite @xcite ; b) regularization strategies to safeguard existing information @xcite ; and c) architectural modifications to facilitate new learning @xcite . while traditional techniques have achieved some success with small-scale models, their effectiveness diminishes when applied to llms due to the high predictability and unpredictability of llms @xcite . llms encounter a more severe cf problem in continue learning. research on cf in llms primarily follows two approaches. the first is data-centric, utilizing replay techniques to incorporate past data or generate synthetic experiences @xcite , which helps in retaining old knowledge. the success of this method heavily depends on the quality of the selected data, as poor choices can undermine its effectiveness. the second approach is model-centric, employing integration strategies to combine fine-tuned and original models @xcite , enhancing generalization and preventing forgetting. however, the effectiveness is constrained by the model's architecture, capacity, and the quality of initial training. ## conclusion we reexamine the phenomenon of catastrophic forgetting (cf) and reveal the direct link between the flatness of the model loss landscape (lls) and the extent of cf in llms. based on our findings, we introduce sharpness-aware optimization to flatten the lls, thereby mitigating cf. experiments show its effectiveness, especially in larger models, suggesting our method may become a standard strategy to mitigate cf during llms tuning. in future work, we would like to explore more strategies to orthogonally address the forgetting problem in llm tuning, such as advanced optimizers @xcite @xcite , proper anti-forgetting learning curricula @xcite @xcite , and model merging techniques @xcite . and cf. while this finding makes us introduce the sharpness-aware minimization technique to flatten the loss landscape and thereby mitigate cf, it also means that other potentially influential factors in cf have not been comprehensively explored. on the other hand, our research primarily addresses cf during the fine-tuning phase of llms, but it does not directly tackle cf that may occur during other stages. for instance, cf can also be a significant challenge when models are updated with new data post-deployment. therefore, the scope of our findings and the applicability of our solution may be limited to specific phases of the llms lifecycle.