Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal (2024)

Abstract: Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model’s ability, which may not be feasible in real-world applications. When conducting continual learning based on a publicly-released LLM checkpoint, the availability of the original training data may be non-existent. To address this challenge, we propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal. Concretely, we first employ the base LLM for in-context learning to generate synthetic instances. Subsequently, we utilize the latest LLM to refine the instance outputs based on the synthetic inputs, preserving its acquired ability. Finally, we select diverse high-quality synthetic instances for rehearsal in future stages. Experimental results demonstrate that SSR achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. Besides, SSR effectively preserves the generalization capabilities of LLMs in general domains.

## introduction large language models (llms) have demonstrated remarkable performance across various natural language processing (nlp) tasks @xcite . in real-world applications, llms are often updated in a continual learning (cl) manner (de masson d' @xcite , where new instruction tuning data is incrementally introduced over time. however, a significant issue that limits the effectiveness of llms is catastrophic forgetting, which refers to the llm's tendency to forget previously acquired knowledge when learning new instances @xcite @xcite . to mitigate catastrophic forgetting, a line of work focuses on rehearsing previous training instances (de masson d' @xcite @xcite . these rehearsal-based methods maintain the model's ability by training on real data from previous training stages. however, the real data may not always be desirable in practical applications. for instance, when conducting continual learning based on a publicly-released llm checkpoint (e.g. llama-2chat), the availability of the original training data may be non-existent. this raises an interesting research question: can we maintain the llm's ability during continual learning without using real data in previous training stages? we propose the self-synthesized rehearsal (ssr) framework to mitigate catastrophic forgetting in continual learning. as shown in figure 1 , unlike standard rehearsal-based continual learning that samples training instances from previous stages as rehearsal data, ssr framework uses the llm to generate synthetic instances for rehearsal. specifically, we first use the base llm to generate synthetic instances, conducting in-context learning (icl) with few-shot demonstrations. these demonstrations can be collected from the previous data or human-constructed containing similar knowledge to the previous data. then, the latest llm is used to refine the outputs of synthetic instances to retain the latest llm's ability. finally, we select diverse high-quality synthetic instances for rehearsal in the future stages. extensive experiments on the task sequences derived from the superni dataset @xcite demonstrate that ssr has superior or comparable performance compared to the conventional rehearsal-based approaches, with higher data utilization efficiency. besides, experiments on al-pacaeval and mmlu @xcite show that ssr can also effectively preserve the generalization capabilities of llms in general domains. we release our code and data at https: //github.com/deeplearnxmu/ssr . ## related work learning a sequence of datasets continually while preserving past knowledge and skills is a crucial aspect of achieving human-level intelligence. existing approaches to continual learning can be broadly categorized into three main categories: (i) regularization-based, (ii) architecture-based, and (iii) rehearsal-based methods. regularization techniques @xcite @xcite control the extent of parameter updates during the learning process, preventing interference with previously learned tasks. nonetheless, these methods typically rely on hyperparameters that need to be carefully tuned for optimal performance. architecturebased approaches @xcite @xcite often take a different approach by learning separate sets of parameters dedicated to individual tasks. this enables the model to specialize and adapt its parameters for each task, avoiding interference between tasks and preserving task-specific knowledge. however, these approaches will introduce additional training parameters, which may not be very flexible and feasible for various llms. therefore, we focus on rehearsal-based methods (de masson d' @xcite , which are also called replay-based methods. these methods typically involve the storage of a subset of data from previous tasks. these stored data are used for future rehearsal through techniques such as experience replay @xcite and representation consolidation @xcite . prior rehearsal-based approaches for language models mainly focus on using a little bit of precedent data @xcite @xcite . however, these approaches often ignore discussion on real-world applications where previous data may be limited or unavailable. although data-free knowledge distillation methods @xcite introduce auxiliary generative models for data construction, they are primarily designed for classification tasks, which may not be effective in llms, where a wide range of nlp tasks are involved. additionally, similar to introducing teacher models @xcite @xcite , it can be challenging and time-consuming to train additional generative models. self-distillation methods @xcite ) may be useful, but catastrophic forgetting of the latest llms and the knowledge discrepancy among llms from distinct stages are still inevitable challenges. in this work, we propose a rehearsal-based continual learning framework in which llms can be trained on self-synthesized data to retain the knowledge of the previous stages, with several demonstrations used during data construction. unlike other approaches, our framework does not depend on additional generative models for data construction or require previous real data for rehearsal. this offers advantages in terms of data efficiency and application flexibility. ## rehearsal-based continual learning in continual learning, the llm is sequentially updated for t stages, with each stage t having its corresponding instruction data d (t) . to mitigate catastrophic forgetting, in each stage t, rehearsalbased methods @xcite sample some training instances of previous stages to expand the training data in the current stage. formally, the augmented training data d (t) can be formulated as follows: input: find the square of 7 output: the square of 7 is 32 ## our framework in this section, we detail the proposed self-synthesized rehearsal (ssr) framework, which involves three main steps: 1) in-context learning based instance synthesis, 2) synthetic output refinement, and 3) rehearsal with selected synthetic instances, as illustrated in figure 2 . ## conclusion in this work, we propose self-synthesized rehearsal (ssr), a continual learning framework for mitigating catastrophic forgetting in llms, to effectively preserve knowledge without relying on real data during rehearsal. through extensive experiments, ssr demonstrates its data efficiency and superior performance to conventional rehearsal-based approaches. besides, it preserves llm's generalization capability both in specific and general domains, with flexibility and robustness in real-world scenarios. overall, ssr presents a promising solution for continual learning of llms in real-world settings, with implications for maintaining the acquired abilities of llms.