LEM o E : Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models (2024)

Abstract: Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts, prompting the formulation of lifelong model editing task. While recent years have witnessed the development of various techniques for single and batch editing, these methods either fail to apply or perform sub-optimally when faced with lifelong editing. In this paper, we introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong model editing. We first analyze the factors influencing the effectiveness of conventional MoE adaptor in lifelong editing, including catastrophic forgetting, inconsistent routing and order sensitivity. Based on these insights, we propose a tailored module insertion method to achieve lifelong editing, incorporating a novel KV anchor routing to enhance routing consistency between training and inference stage, along with a concise yet effective clustering-based editing order planning. Experimental results demonstrate the effectiveness of our method in lifelong editing, surpassing previous model editing techniques while maintaining outstanding performance in batch editing task. Our code will be available.

## introduction large language models @xcite @xcite encode a vast amount of world knowledge during pre-training, which can be accessed and utilized through natural language prompts @xcite . however, the dynamic nature of the real world necessitates regular and continual updates to these models to correct outdated information or integrate new knowledge @xcite . also, retraining or finetuning of llms is often resource-intensive and time-consuming @xcite , making it impractical for lifelong growing knowledge. therefore, lifelong model editing @xcite has been proposed to remedy the continual knowledge updates and injections for llms in a cheap and timely manner @xcite . in recent years, there has been a proliferation of effective model editing techniques proposed for single or batch editing, such as mend @xcite , rome @xcite , memit @xcite , and memoe @xcite . however, these methods often prove inapplicable or exhibit suboptimal performance when faced with lifelong editing task @xcite . in this paper, we introduce lemoe, an advanced mixture of experts (moe) adaptor, to address the challenges inherent in lifelong editing. initially, we analyze the factors that influence the effectiveness of conventional moe adaptor in lifelong editing, including catastrophic forgetting, inconsistent routing, and order sensitivity. in the catastrophic forgetting analysis ( §3.1), we evaluate the performance of conventional moe adaptor at different positions within the editing sequence to quantify the impact of subsequent edits on preced-ing ones. we observe that the conventional moe adaptor exhibits significant catastrophic forgetting, where earlier edits are more prone to errors. in the routing consistency analysis ( §3.2), we compare the expert networks selected by routing strategy during the training and inference stages when faced with the same input. this comparison reveals a routing inconsistency in the conventional routing strategy, where identical inputs are routed to different experts at different stages. finally, in the order sensitivity analysis, we highlight that editing order profoundly impacts model performance ( §3.3). through varying the sequence order of the same dataset during lifelong editing, we observe performance variations of up to 20 points, surpassing the improvement of some optimization algorithms. based on these insights, we propose a tailored module insertion method to achieve lifelong editing ( §4.1). as illustrated in figure 1 , we align the expert networks in the moe architecture with the data batches in the sequential editing process. when conducting current editing, we freeze the expert network corresponding to the previous data, thereby mitigating the adverse effects of current data editing on previous edits and alleviating catastrophic forgetting from a model mechanism perspective. secondly, we introduce a novel key-value (kv) anchor routing ( §4.2), wherein each expert is assigned a key vector and the input instance-level embedding serves as the corresponding value. based on these key-value pairs, we align the routing computation processes during both training and inference stage. this ensures that identical inputs undergo same routing computation to reach the same expert across all stages, thereby enhancing routing consistency and further mitigating catastrophic forgetting. finally, leveraging the consistency between the moe preferences of editing order and the objectives of clustering algorithm, we employ a concise yet effective clustering-based order planning to enhance the overall performance of lemoe ( §4.3). we conduct experiments on the llama-7b and mistral-7b models using the zsre @xcite and selfcheckgpt @xcite datasets to evaluate the performance of lemoe. experimental results show that our approach surpasses previous model editing methods, while maintaining excellent performance in batch editing. the main contributions of our work can be summarized as follows: ## preliminaries of model editing based on previous research @xcite @xcite , model editing involves the process of transforming an initial base model f θ (where θ denotes the model's parameters) into an edited model f θ ′ . the goal is to modify the model's outputs for a specific set of editing instances, while maintaining consistent behavior for all other instances @xcite . the target editing instance can be described as (x e i , y e i ), with the condition that f θ (x e i ) ̸ = y e i . the set of this instances is termed the editing scope i edit , whereas the out-of-scope set o edit comprises inputs not associated with the editing examples. formally, the criteria for a successful edit can be described as: we divide model editing tasks into two categories: batch editing and lifelong editing: 1) batch editing refers to the simultaneous modification of model f θ using multiple input instances in one editing operation: ## analysis of influencing factors in this section, we analyze the factors that influence the effectiveness of conventional moe adaptor in lifelong editing, including catastrophic forgetting analysis ( §3.1), routing consistency analysis ( §3.2), and order sensitivity analysis ( §3.3). ## methods based on the above insights, in this section, we provide a detailed introduction to lemoe an advanced moe adaptor with new module inserting, kv anchor routing and clustering-based order planning. ## conclusion in this paper, we propose lemoe, an advanced moe adaptor for lifelong model editing. we analyze three factors influencing the effectiveness of moe adaptor in lifelong model editing. then, we propose three optimization modules. these modules align the routing computation processes between training and testing phases, ensuring the same inputs are routed to the same experts. experimental results validate the effectiveness of lemoe across multiple models and datasets.