I ns CL : A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions (2024)

Abstract: Instruction tuning effectively optimizes Large Language Models (LLMs) for downstream tasks. Due to the changing environment in real-life applications, LLMs necessitate continual task-specific adaptation without catastrophic forgetting. Considering the heavy computational cost, replay-based Continual Learning (CL) methods are the simplest and most widely used for LLMs to address the forgetting issue. However, traditional replay-based methods do not fully utilize instructions to customize the replay strategy. In this work, we propose a novel paradigm called Instruction-based Continual Learning (InsCL). InsCL dynamically replays previous data based on task similarity, calculated by Wasserstein Distance with instructions. Moreover, we further introduce an Instruction Information Metric (InsInfo) to quantify the complexity and diversity of instructions. According to InsInfo, InsCL guides the replay process more inclined to high-quality data. We conduct extensive experiments over 16 tasks with different training orders, observing consistent performance improvements of InsCL. When all tasks have been trained, InsCL achieves performance gains of 3.0 Relative Gain compared with Random Replay, and 27.96 Relative Gain compared with No Replay.

## introduction large language models (llms) show remarkable capabilities from a wide range of natural language processing (nlp) tasks @xcite @xcite , demonstrating large potential in handling various taskspecific settings. to complete realistic downstream tasks, recent works suggest that instruction tuning is an incredible method for unleashing the power of llms @xcite @xcite . however, in real-life applications, the consistent emergence of new corpora and knowledge changes task schemas frequently, necessitating continual task-specific adaptation for llms @xcite . accordingly, continual learning (cl) is proposed to learn a sequence of tasks incrementally, updating models for the changing environment without catastrophic forgetting @xcite . considering the heavy burden on computing time and gpu memory of tuning llms, replaybased methods are the simplest and most effective among all traditional cl methods. despite several replay-based methods that have been wellstudied @xcite @xcite , some traditional strategies cannot achieve optimal performance in continual instruction tuning due to the unique data composition. to address this issue, we propose a data-efficient paradigm called instruction-based continual learning (inscl), applied to continual fine-tuning llms with natural language instructions. inscl effectively utilizes instructions as high-quality task descriptions, designing a dynamic instruction-information-based replay method. as shown in figure 1 , when the new task d i comes, in-scl will sample replay data r from all the previous tasks (here we list two previous tasks in figure 1 ). inscl dynamically replays α * data from previous tasks based on their similarity with the current task. we draw on the application of optimal transport @xcite in comparing different distributions and adopt wasserstein distance @xcite as a similarity measure. since instructions naturally contain high-quality task-related descriptions, we use instructions to calculate wasserstein distance instead of using the full amount of data, significantly reducing the computational cost @xcite . for the previous tasks that are more different from the current task, inscl allocates a larger replay scale (larger bar width in figure 1 ). after determining the sample size based on task similarity, inscl leverages instruction information to guide the sampling process more inclined to high-quality data. prior works have shown that the performance with less but high-quality data can be comparable with full data @xcite @xcite . for instruction tuning scenarios, early attempts @xcite @xcite affirm that llms' performance can be improved by increasing the training template complexity and diversity. inspired by this, we propose an instruction information metric (insinfo) to quantify the complexity and diversity of instructions. with insinfoguided sampling, inscl replays more high-quality data (longer bar length in figure 1 ). we empirically demonstrate that replaying more data with high insinfo helps to alleviate the forgetting issue. the main contributions of this paper include: (1) we propose inscl, a novel replay-based cl paradigm for instruction tuning. inscl allocates replay size based on task similarity, dynamically replaying high-quality data with high insinfo. (2) experiments are conducted over 16 tasks with different training orders, demonstrating the effectiveness of inscl. (3) we further analyze the forgetting phenomenon in continual instruction tuning. without replaying, we found that complex reasoning tasks suffer from a higher forgetting rate, where forgetting instances are mainly instruction-unrelated. ## method continual learning of llms focuses on adapting an instruction-tuned model to handle a sequence of tasks in a specific application scenario. this approach accounts for consistently emerging materials while processing the tasks simultaneously. we define n tasks to be learned as a sequence @xmath0, . . . , d n }. when llms are tuned with i-th task, we form a replay dataset r α j by sampling examples from d j , where @xmath1, i -1]. formally, the training data augmented with replay data is defined as: ## experimental setup data collection. to facilitate our research, we mainly utilize the superni dataset @xcite , a comprehensive benchmark focusing on specific nlp tasks distilled from real-world demands. superni is annotated by nlp practitioners from github and nlp courses, ensuring that each instance is coupled with respective natural language instructions. at the most comprehensive level, we integrate 765 english tasks from superni into 16 categories, as shown in figure 2 . and we demonstrate details of the data composition in appendix a.2. following the setting of prior cl studies @xcite , we randomly hold out 20% instances on each task to evaluate llms on different training stages. model and training details. our work is most related to the continual instruction tuning setting as continual-t0 @xcite . we conduct our task-incremental experiments with the popular llama-7b @xcite , training each task for 2 epochs with a batch size of 64. we use the adam optimizer (kingma and ba, 2014) with a learning rate of 2e-5 and utilize the standard language modeling objective: here we utilize rouge-l @xcite to calculate r i j and the upper bound. ## experiments we leverage llama-7b to calculate sentence embeddings and compare our inscl with the follow-ing strategies: for each previous task, we set the cluster number as the amount of instructions. we sort the data in descending order according to cosine distance from the corresponding center and take the top-α as replay data. for fairness of comparison among different methods, we note m @xmath2. here we set α to 200. ## conclusions in this paper, we mainly discuss the efficient adaptation of llms to continual downstream tasks with instructions. replay-based cl methods do not require additional modifications to llms and fully utilize previous data, mitigating catastrophic forgetting effectively. we proposed inscl, an effective data-efficient method to mitigate catastrophic forgetting for llms instruction tuning. inscl is a model-agnostic and training-free method, indicating strong transferability. different from existing replay-based methods, we fully utilize instructions as representative task descriptions to design the replay strategy. inscl leverages instruction embeddings and distributions to calculate wasserstein distance for task similarity, adjusting the replay ratio dynamically. then, with our insinfo-guided sampling, inscl selects more high-quality data with complex and diverse instructions. we conduct extensive experiments over 16 tasks with different training orders, observing consistent performance improvements of inscl. additionally, we further analyze the forgetting rate and forgetting category, aiming to provide a guideline for future work.