SEE : Continual Fine-tuning with Sequential Ensemble of Experts (2025)

Abstract: Continual fine-tuning of large language models (LLMs) suffers from catastrophic forgetting. Rehearsal-based methods mitigate this problem by retaining a small set of old data. Nevertheless, they still suffer inevitable performance loss. Although training separate experts for each task can help prevent forgetting, effectively assembling them remains a challenge. Some approaches use routers to assign tasks to experts, but in continual learning, they often require retraining for optimal performance. To address these challenges, we introduce the Sequential Ensemble of Experts (SEE) framework. SEE removes the need for an additional router, allowing each expert to independently decide whether a query should be handled. The framework employs distributed routing, and during continual fine-tuning, SEE only requires the training of new experts for incoming tasks, rather than retraining the entire system. Experiments reveal that the SEE outperforms prior approaches, including multi-task learning, in continual fine-tuning. It also demonstrates remarkable generalization ability, as the expert can effectively identify out-of-distribution queries, which can then be directed to a more generalized model for resolution. This work highlights the promising potential of integrating routing and response mechanisms within each expert, paving the way for the future of distributed model ensembling.

## introduction large language models (llms) have demonstrated impressive performance across various scenarios @xcite @xcite @xcite . however, they often require further refinement or the ability to learn new tasks for real-world applications. in these scenarios, llms are typically ## preliminaries continual fine-tuning let m denote the llm. the model undergoes continual fine-tuning across n stages, with each stage i involving updates based on the instruction dataset d (i) corresponding to task t i @xcite . formally, the update process at each stage is defined as: ## methods in contrast to previous research, our work integrates the concept of moe with rehearsal-based methods. for each task t i , we assign a specific expert e i to handle it. these experts are able to assess whether a query falls within their scope of responsibility. if so, they proceed to generate the response. see integrates the experts and the base model through a process of sequential routing. an overview of see is shown in figure 2 , and it can be divided into three main steps: (1) task reconstruction, (2) expert training, and (3) inference. task reconstruction at stage i, we reconstruct each task dataset d i to enable the experts to clarify their responsibilities. initially, each sample x in d i can be represented as a tuple (q, r), where q is the query and r is the corresponding response. we transform each x into x, which consists of the tuple (q, o pos , r), by adding a special positive indicator o pos . additionally, we sample τ % of the data from previous tasks, as in rehearsal-based methods, but ignore the response r and add a negative indicator o neg . the final constructed di consists of positive samples {(q, o pos , r) j } n @xmath0, o neg ) j } m @xmath1: given a query, the expert first determines whether the question falls under its responsibility and decides whether to provide an answer. by sequentially ensembling all experts, we can handle various tasks through sequential routing. inference after continuing fine-tuning on n tasks, we obtain n experts, denoted as (e (m θ ,l 1 ) , . . . , e (m θ ,l n ) ), where the i-th expert specializes in task t i . when a question is posed, the expert e (m θ ,l n ) generates an indicator to decide whether to continue generating a response. if the indicator is o neg , the expert e (m θ ,l n ) stop generating and routes the query to the expert e (m θ ,l n -1 ) we iteratively repeat this process until an expert generates o pos . once this happens, the expert is chosen to generate the final response. if an expert fails to produce an indicator, we route the query to the base model m 0 and let it to answer the question. this entire routing process is referred to as sequential routing. by applying sequential routing, we combine all individual experts into a more powerful model. ## analysis generalization while the see model has demonstrated outstanding performance in tasks during continuous fine-tuning, it is also essential to assess its generalization. in our design, the generalization ability of the see depends on two key factors: the average generalization ability of the experts and the accuracy with which the see routes out-of-distribution (ood) queries to the base model. the former refers to the scenario where an ood query is incorrectly routed to an expert, in which case see's generalization ability is determined by the average generalization ability of the experts. the latter reflects see's ability to correctly identify ood tasks. to assess this, we evaluate see's average expert performance on mmlu @xcite and the standard performance of see. we compare their performance against that of the base model, mtl, and randsel. the experiment is based on llama-2-7b and the results are presented in figure 4 . as shown in figure 4 , the base model m θ achieves the highest accuracy. the average performance of the experts in see is similar to that of randsel. however, it is important to consider the ood routing accuracy when evaluating the actual performance of see. specifically, 99.72% of instances are routed to the base model in see(10%), while this percentage is 99.52% in see(1%). as a result, the actual performance of the see framework significantly exceeds that of other methods, demonstrating its superior generalization ability. perplexity across 10 superni tasks treating see as a unified model, we compute the perplexity for each instance (q, r). first, the instance is transformed into (q, o pos , r), where o pos represents a positive indicator. this transformed instance is then routed to the relevant expert within see. then, the perplexity is computed by the selected expert. we evaluate the perplexity of see(10%) across 10 superni tasks and compare the results with those of mtl, which, in other hand, computes the perplexity using the original (q, r) pairs. figure 5 illustrates that see achieves lower perplexity across the 10 superni tasks, highlighting the effectiveness of the see framework. ## related work continue learning existing approaches to continue learning can be divided into regularizationbased, architecture-based, and rehearsal-based methods. regularization-based methods constrain the inner distribution of llms to remain close to the original state by adding auxiliary loss or controlling parameter updates @xcite @xcite . however, these methods require manually tuned hyperparameters, limiting their applicability. rehearsal-based methods maintain a history buffer with a subset of previous datasets (de @xcite or synthetic instances @xcite , replayed during future training. while effective at mitigating catastrophic forgetting, they cannot fully preserve prior knowledge. architecture-based methods introduce new parameters for each dataset and learn them independently, potentially avoiding knowledge loss @xcite @xcite . however, managing these additional parameters remains challenging. in our framework, we use rehearsal with queries from previous tasks and organize additional parameters as expert weights, selecting them via sequential routing. ## conclusion in this work, we propose the sequential ensemble of experts (see), which is specifically designed to adapt to continuous fine-tuning. see resolves the separation between the router's decision-making and the experts' execution by integrating routing and response mechanisms within each expert. it employs a distributed routing method called sequential routing, improving the system's scalability. our experiments demonstrate see's effectiveness in handling both continual learning tasks and outof-distribution instances, paving the way for future advancements in distributed model ensembling.