Mixture of L o RA Experts for Continual Information Extraction with LLM s (2025)

Abstract: We study continual information extraction (IE), which aims to extract emerging information across diverse IE tasks incessantly while avoiding forgetting. Existing approaches are either task-specialized for a single IE task or suffer from catastrophic forgetting and insufficient knowledge transfer in continual IE. This paper proposes a new continual IE model using token-level mixture of LoRA experts with LLMs. We leverage a LoRA router to route each token to the most relevant LoRA experts, facilitating effective knowledge transfer among IE tasks. We guide task experts’ selection by task keys to retain the IE task-specific knowledge and mitigate catastrophic forgetting. We design a gate reflection method based on knowledge distillation to address forgetting in the LoRA router and task keys. The experimental results show that our model achieves state-of-the-art performance, effectively mitigating catastrophic forgetting and enhancing knowledge transfer in continual IE.

## introduction large language models (llms) have demonstrated remarkable capabilities as a powerful expert tool across diverse applications @xcite , and knowledge plays a crucial role in shaping their effectiveness and future potential @xcite ). among such knowledge-intensive applications, information extraction (ie) @xcite serves as a fundamental task, aims to identify and structure specific information from unstructured natural language text. it encompasses diverse tasks, e.g., named entity recognition (ner), relation extraction (re), and event detection (ed), based on the different extraction targets @xcite . in practice, new types and tasks are emerging continually across diverse ie scenarios. studying continual learning (ring, 1994) within * corresponding author ie scenarios is crucial for building adaptive and scalable ie models. previous continual ie models @xcite @xcite @xcite are task-specialized, resulting in separate application scenarios and isolated models for different ie tasks. real-world applications demand a unified model that can continuously extract user-specified information across diverse ie tasks, rather than being limited to a single task. the task-specific design hinders model reuse and increases deployment complexity, limiting the scalability of continual ie. moreover, the different ie tasks are not completely independent of each other. there is shared knowledge among different ie tasks that a model can leverage. the isolated models significantly limit the ability to share knowledge between related tasks and settings. therefore, these task-specialized models hinder the development of ie in terms of cross-task adaptation and effective knowledge sharing. toward this issue, we define a new problem called continual ie to address. compared to task-specialized continual learning, continual ie expects the model to extract different information of different ie tasks from a continuous data sequence, not only extracting new information but remembering all learned information so far. existing continual learning methods for ie focus on alleviating catastrophic forgetting @xcite in their single tasks using knowledge transfer and experience replay. these task-specialized models fall short in knowledge transfer among ie tasks and cannot adapt to continual ie. recent continual learning studies @xcite @xcite @xcite for llms with parameter-efficient finetuning (peft) can be applied to continual ie by learning and selecting independent peft blocks such as soft prompt @xcite or low-rank adaptation (lora) @xcite for each task. however, they are still struggled with catastrophic forgetting and insufficient knowledge transfer. on one hand, the selection modules in these methods are crucial, as they need correct knowledge for prediction. however, these methods only use concatenation or attention weights at the sentence level for selecting the correct peft blocks. these sentencelevel selection modules are insufficiently discriminative in continual ie, where the semantic features of sentences in different ie tasks are similar. thus, these methods easily make incorrect choices and use wrong knowledge, leading to catastrophic forgetting. on the other hand, these methods only update the peft blocks for the current task during training, while freezing those that have already been learned. this prevents them from the backward transfer of useful knowledge from new tasks to old ones. moreover, their limited use of previous knowledge in current training hinders the forward transfer from old tasks to new ones. to address these issues in continual ie, we propose a novel continual ie model, mole-cie, through the token-level mixture of lora experts with llms. specifically, we route each token to the most relevant lora experts by a lora router to facilitate knowledge transfer among ie tasks. we allocate task experts and a shared ie expert to retain the task-specific knowledge of ie. we design a gate reflection method using knowledge distillation to mitigate the forgetting of the lora router and task keys. our main contributions are threefold: ## task definition ie aims to extract structured information from unstructured natural language text. in continual ie, a sequence of k tasks {t 1 , t 2 , . . . , t k } is presented to the model in a sequential manner. each task is an independent ie sub-task with its own training set d train i , development set d dev i , test set d test i , and type set r i . r i of each task t i is disjoint with other tasks. at the i-th stage, a continual ie model is trained on d train i and evaluated on current and all previous test sets dtest @xmath0. in other words, the model must handle the emerging types within the same extraction task and shift to different extraction tasks over time. the model will not be informed of the specific extraction task during inference. ## methodology figure 1 shows the framework of our continual ie model mole-cie. we add the mixture of lora experts (mole) module to the attention layer of the pre-trained transformer blocks. the attention from the pre-trained model is combined with the attention produced from the mole module to form a new attention, which is used in the subsequent layers. during training, we freeze the parameters of the pre-trained model and only update the parameters of the mole module. in the mole module, the model first routes each token to the lora experts that are most relevant to it. then, the model selects task experts by calculating the similarity between the task keys and the input tokens. finally, the model fixes an ie expert and aggregates the output of each lora expert by the corresponding weight. for continual training, we design the gate reflection module to distill past knowledge in the lora router and task keys. we pick and store a few instances of new types for the next task. ## experiments and results in this section, we conduct experiments to evaluate our model and report the results. the source code is attached as supplementary materials. please refer to the appendix for more experimental analyses. the source code is accessible online. foot_0 ## conclusion in this paper, we define continual ie and design a novel model mole-cie. we propose the mixture of lora experts and gate reflection to mitigate catastrophic forgetting and facilitate knowledge transfer. experiment results on three benchmark ie tasks show that mole-cie achieves superior accuracy. the results also verify its effectiveness in knowledge reservation and transfer. for maven @xcite , we follow @xcite to take the original development set as the test set and re-split the training set. for the relation extraction task, we follow @xcite to use 80 relations and each with 700 samples for few-rel @xcite . we allocate 420 samples to the training set, 140 samples to the development set, and 140 samples to the test set. for tacred, we follow @xcite to remove the samples of "no_relation". for the named entity recognition task, we use the origin dataset for ontonotes-5.0 @xcite and i2b2 @xcite . for the partition of each dataset, we ensure that the label and instance counts for each split are as consistent as possible. after partitioning, we merge the split of each dataset based on the task type. the statistics of the datasets after merging are presented in