Pattern Shifting or Knowledge Losing? A Forgetting Perspective for Understanding the Effect of Instruction Fine-Tuning (2024)

Abstract: “Instruction Fine-Tuning(IFT) emerges as an essential step of training large language models torobustly carry out tasks of interest. However, there lacks a systematic investigation about theunderlying mechanisms of instruction fine-tuning, particularly on the forgetting phenomenonafter IFT, known as alignment tax. Therefore, to understand the mechanism of IFT from theforgetting perspective, we investigate the alternation of the text pattern and knowledge withinmodels throughout the entire IFT process. Specifically, we restore fine-tuned models to their baseversion by training them on the data sharing a similar distribution with the pre-training corpusand compare their results Our experiment indicates that there is a stage transition of forgettingduring IFT process: (1) Pseudo Forgetting: in this stage, models mainly shift their familiar textpattern away from pre-training data format while the world knowledge is preserved. Consequently,models will recover to their original performance when they are restored to the base version. (2)Actual Forgetting: in this stage, models forget the acquired knowledge as well. Therefore, theyfail to reach the original performance even if they are restored to the base version.”

## introduction instruction fine-tuning (ift) has emerged as an indispensable process during the development of large language models (llms) @xcite @xcite . by training llms in formatted input-output pairs, ift enables them to demonstrate extraordinary capabilities on unseen tasks even in zero-shot settings @xcite . since ift requires fewer data compared with the pre-training stage and delivers superior outcomes @xcite , it has become an essential approach to enhance llms. however, previous studies @xcite find that fine-tuned models fail to solve the problems which their base versions could solve, resulting in performance degradation in specific tasks. such a seemingly catastrophic forgetting @xcite ) phenomenon after ift is reported as alignment tax @xcite . unfortunately, the underlying mechanism of such forgetting remains an open problem and receive distinct opinions. on the one hand, some researchers @xcite @xcite indicate that llms merely learn the superficial pattern of ift data and forgetting originates from suppressing llms performance in areas out of distribution with the ift data. on the other hand, other researchers @xcite @xcite suggest that ift enables llms to express the knowledge gained during pre-training in the format of instruction-output pairs, where forgetting originates from trading base models' generality for such speciality. the absence of such a performance fine-tuning procedure original lm performance fine-tuned lm restored lm actual forgetting process pseudo forgetting process turning point original lm performance degradation figure 1 : the illustration of theoretical instruction tuning process, during the initial stage, fine-tuned llm is better than the original. subsequently, llm begins to fall behind the original base llm while restoring it to the original text pattern can bring back the performance. in the late stage, even if llm is restored to its original pattern, the performance is still worse than the original systematic and unanimous analysis on the alignment tax limits our potential to develop a better ift training strategy or thoroughly evaluate llms, which consequently delays improvements and further application of llms. therefore, a comprehensive analysis of the underlying core factors behind forgetting after ift is significant. to this end, this paper aims to reveal the mechanism of ift from the forgetting perspective. specifically, this paper will explore two specific research questions: for rq1, we propose to restore fine-tuned llms to base llms in order to investigate whether the world knowledge @xcite is retained in llms after ift. specifically, we collect data from the corpus base llms are pre-trained on, which shares a similar data distribution with the pre-training data. we then sample them to form the restoring data for training, which aims in restoring the llms familiar text pattern from instruction text to plain text, and probing data to measure whether the familiar text pattern is recovered to plain text. subsequently, we apply several benchmarks to compare the performance between base, ift and restored llms. we find that during the early ift process, llms forget the original text pattern and is more familiar with the text pattern of fine-tuning data, we call such an alternation of text pattern as pattern shifting. after restoring the text pattern to the original plain text format, their recovered performance is comparable with base models. such results indicates that the world knowledge still preserves and the drop originates from the forget of the original text pattern, which we refer to as pseudo forgetting. as for the late training process, even if the familiar text pattern is restored, the clear gap between recovered models and base models reveals that both text pattern and knowledge are forgotten, which we refer to as actual forgetting. as for rq2, we further investigate the probability distribution of the answer to the benchmark to further reveal the characteristic of forgetting pattern. for the pseudo forgetting stage, we analysis the probability and entropy of the answer across the whole vocabulary and find that the overall desired output generation probability across the whole vocabulary of llms begins to converge at high outcome during the early process of pseudo forgetting. we call such phenomenons as format alignment. after reaching a high generation probability, the entropy of the next generated tokens keeps altering, indicating that models are aligning their preserved knowledge to the desired output format, refered to as knowledge ## stage transition of forgetting during instruction fine-tuning this section proposes results and findings of the forgetting phenomenon. experiment results lead to following conclusions that forgetting in instruction tuning has two stages, pseudo forgetting and actual forgetting further analysis based on these conclusions are presented in section 4.2. ## characteristic of forgetting stages in this section, we verify the characteristic of stages during the pseudo forgetting process, we focus more on the trajectory of actual generation probability as well as entropy over the whole vocabulary distribution. we introduce the preliminaries in section 5.1 and the corresponding finding in section 5.2. our result indicates that format alignment completes during the pseudo forgetting stage and begins knowledge alignment. models begin to be over confident as the training proceeds which might cause the actual forgetting. ## forgetting across models and data this section expands the forgetting phenomenon to a larger llms and other ift data to investigate whether such a pattern of forgetting preserves across models. experiment results implies a similar stage transition of forgetting, which helps to settle down the finding of this paper. the comparison with the smaller model reveals that larger model are more likely to face actual forgetting. ## conclusion in our research, we have delved into the phenomenon of catastrophic forgetting within the context of iterative fine-tuning (ift) with the objective of elucidating its underlying mechanisms. through our investigations, we have identified two distinct stages of forgetting: pseudo forgetting and actual forgetting. during the initial phase of ift, we observe what we term as pseudo forgetting, where large language models (llms) like llama undergo a pattern shift rather than actual knowledge loss. the models adapt their existing patterns to new instructions while retaining the knowledge acquired during pre-training. this stage is characterized by a performance trajectory that initially oscillates before trending downward. in contrast, actual forgetting emerges in the later stages of ift, signifying a genuine loss of knowledge within the llms. our experiments demonstrate that models saved at later checkpoints exhibit a significant performance disparity when compared to their original pre-trained state, indicating that knowledge has indeed been forgotten during the instruction tuning process. our study also indicates the characteristic of these forgetting stages, llms are undergoing format alignment during the initial training process since the generation probability of correct format answer reaches a high point in just a few steps. after the completion of format alignment, the generation probability persist while the entropy across these correct format answer fluctuate, indicating the knowledge alignment. during the late stage of training, the entropy declines continuously, implying that models are over confident at their answer, which may cause the actual forgetting. finally, by calculating the em of llms on instruction data, we find that llms are memorizing ift data throughout the instruction tuning process and may cause the actual forgetting. in conclusion, our research provides significant insights into the stages of catastrophic forgetting in llms during ift and establishes a foundation for future work aimed at mitigating such forgetting in large language models.