SLIM : Let LLM Learn More and Forget Less with Soft L o RA and Identity Mixture (2025)

Abstract: Despite the recent efforts from the NLP community, balancing the training budget, downstream performance, and general capabilities of large language models (LLM) remains a challenge in many applications. Training the entire model for downstream tasks is expensive, and could easily result in catastrophic forgetting. Parameter-efficient fine-tuning (PEFT) could reduce the training cost, but it still suffers from forgetting, and limits the learning on the downstream tasks. To address the aforementioned issues, we propose a novel mixture of expert (MoE) framework based on Soft LoRA and Identity Mixture (SLIM). SLIM allows dynamic routing between LoRA adapters and identity layers, thus enabling the bypass of LoRA adapters to suppress forgetting of general capacity. We adopt weight yielding with sliding clustering for better out-of-domain distinguish to enhance the routing. We also convert the mixture of LoRA adapters to the model merging formulation and introduce dynamic merging with its fast implementation for LoRA adapters to keep the general capabilities. Extensive experiments demonstrate that the proposed SLIM is comparable to the state-of-the-art PEFT approaches on the downstream tasks while achieving the leading performance in mitigating catastrophic forgetting. We plan to open-source the code upon publication.

## introduction in recent years, large language models (llm) have demonstrated extraordinary general capabilities, and are widely used for question-answering, code generation, mathematics, and other fields. however, the increasing utilization of llm in specialized domains needs to support tasks with a mixture of world and specific knowledge. meanwhile, since the scale of the llm is large, fine-tuning the entire model could be expensive, and thus may not be affordable for many users. furthermore, after finetuning the downstream tasks, the llm may lose the general capability, due to catastrophic forgetting. how to reduce the training cost while mitigating catastrophic forgetting without loss of learning capacity, is an essential problem in the utilization of llm, especially when utilized in a scenario where both common and professional capabilities are needed. an intuitive solution is to train only a small bunch of parameters. for example, low-rank adaptation (lora) @xcite , a widely used peft approach, is validated to forget fewer pieces of knowledge compared with fine-tuning the full weight of the base model, with limitations of learning capacity @xcite . @xcite propose to optimize the model in the orthogonal subspace with low-rank adaptation to alleviate forgetting. @xcite propose to insert calibrating modules to the base model for learning the downstream tasks. @xcite utilize learnable mask prompts to fine-tune the clip without loss of text-vision alignment. these approaches attempt to introduce minimal changes to the base model to keep its raw capacity. some approaches utilize data replay to reduce forgetting. for example, @xcite utilize a 60% data replay for continual training. @xcite utilize a 30% replay rate in the continual pre-training for the coding model. despite the fact that data replay could effectively reduce the influence of forgetting, the large amount of training data is still a burden, and the pre-training data may not be accessible for downstream fine-tuning. in this work, we propose a novel moe architecture with soft lora and identity mixture (slim), a mixture of lora adapters and identity layers. by introducing identity layers as experts, the model is enabled to activate these layers as information "highways" to skip the lora adapters, thus avoiding the influence of downstream tasks. we further propose weight yielding with sliding clustering to estimate the distribution of the samples from the downstream tasks and modify the routing weight according to the consistency between the input sample and the distribution. this mechanism encourages the model to dynamically route the samples from the downstream tasks to the lora adapters while assigning the samples out of the downstream distribution to the identity layers to alleviate forgetting. inspired by dare @xcite , we convert the low-rank adaptation to model merging formulation and propose a dynamic merging approach and its fast implementation to mix lora adapters to the base model for forgetting alleviation. we conduct experiments on various downstream datasets while validating the model performance on multiple general tasks. extensive experiments show that the proposed slim could achieve sota peft performance, and significantly mitigate catastrophic forgetting, as demonstrated in fig. 1 . our main contributions could be summarized as follows: ## conclusion in this work, we propose slim, an moe parameterefficient fine-tuning approach. in addition to implementing the lora adapters, slim adopts identity layers as experts which could be activated as information "highways" to bypass the lora adapters for general capacity preservation. we propose weight yielding with sliding clustering to modify the routing weight according to the consistency between the input sample and the downstream distribution to take advantage of these "highways". we further introduce dynamic merging that formulates peft to model merging, which effectively combines the downstream capabilities with the general capabilities of the base model. slim achieves comparable performance to state-of-the-art peft methods, with much less general capability loss.