Continual Learning for Multilingual Neural Machine Translation via Dual Importance-based Model Division (2023)

Abstract: A persistent goal of multilingual neural machine translation (MNMT) is to continually adapt the model to support new language pairs or improve some current language pairs without accessing the previous training data. To achieve this, the existing methods primarily focus on preventing catastrophic forgetting by making compromises between the original and new language pairs, leading to sub-optimal performance on both translation tasks. To mitigate this problem, we propose a dual importance-based model division method to divide the model parameters into two parts and separately model the translation of the original and new tasks. Specifically, we first remove the parameters that are negligible to the original tasks but essential to the new tasks to obtain a pruned model, which is responsible for the original translation tasks. Then we expand the pruned model with external parameters and fine-tune the newly added parameters with new training data. The whole fine-tuned model will be used for the new translation tasks. Experimental results show that our method can efficiently adapt the original model to various new translation tasks while retaining the performance of the original tasks. Further analyses demonstrate that our method consistently outperforms several strong baselines under different incremental translation scenarios.

## introduction multilingual neural machine translation (mnmt) @xcite @xcite handles multiple translation directions in a single model and recent largescale mnmt models, such as mbart50-nn @xcite , m2m-100 @xcite and nllb @xcite have demonstrated promising translation performance. however, a figure 1 : the bleu decline of different mnmt models on the en↔xx under different pruning ratios. we perform weight pruning with the general magnitudebased method @xcite . the biggest challenge for continual learning is catastrophic forgetting @xcite , which results in severe performance decline on the previous tasks when adapting to new tasks. to alleviate this problem, some works attempt to balance the performance between the original and incremental tasks, such as the replay-based methods @xcite @xcite and regularization methods @xcite @xcite . however, as the whole parameter spaces are shared across different translation tasks, those methods are vulnerable to inter-task interference, especially when the incremental tasks are largely different from the original ones. other works @xcite employ additional task-specific parameters for new task adaptation while keeping the parameters in the original model fixed, which can naturally avoid the forgetting problem. however, those methods increase the model size and the structure of the task-specific modules requires specialized manual design. previous studies @xcite @xcite show that some parameters in the network are not important to the original tasks so that they can be pruned without causing obvious performance degradation. as shown in figure 1 , the current mnmt models also have such unimportant parameters, which provides the possibility of using those negligible parameters to learn the knowledge of new tasks instead of introducing additional parameters. however, this model pruning strategy focuses more on the importance of parameters to the original tasks while neglecting their roles in new task adaptation, which restricts the model's ability to learn the specific knowledge of the incremental languages. in this work, considering the importance of parameters on the original and new translation tasks, we propose a dual importance-based model division method that can efficiently adapt the original mnmt model to various incremental language pairs. first, we fine-tune the original mnmt model with the new training data and propose two methods to evaluate the importance of parameters in different tasks. then we find the parameters that are unimportant to the original tasks but important to the new tasks and remove them from the original model to obtain a pruned model. finally, we expand the pruned model to its original size and fine-tune the newly added parameters with new training data. in this way, we can better capture the specific knowledge of incremental translation tasks and avoid catastrophic forgetting on the original translation tasks. our main contributions are summarized as follows: replay-based methods. the replay-based methods utilize the training data or pseudo data of the previous tasks to replay the old knowledge while training on new tasks @xcite @xcite @xcite . however, those methods will bring more training consumption, especially for the large-scale pre-trained mnmt model. moreover, the previous training data are sometimes unavailable due to data privacy or storage limitations @xcite . the noise in pseudo data will also hurt the performance of the previous and new tasks. compared to those methods, our method does not need to access the previous data, and thus it is more flexible and efficient for incremental learning. regularization-based methods. in addition to the original training objective, the regularizationbased methods usually introduce additional penalty terms to balance the performance on the previous and new tasks. @xcite and @xcite employ regularization terms to constrain the change of all the parameters from their original values. @xcite utilizes the original model as the teacher to prevent catastrophic forgetting via knowledge distillation. @xcite updates the parameters within the low forgetting risk regions with a hard constraint. by contrast, our method only prunes the parameters that are not essential to the previous tasks so that it can prevent the forgetting problem. it also allows our method to focus on learning the new tasks without making compromises with the previous tasks. ## method in this work, we aim to prevent the catastrophic performance degradation on the previous translation tasks without accessing their training data while adapting the model to new translation tasks efficiently. the main idea of our method is that the parameters have varying degrees of importance for the previous and new translation tasks. based on this, we utilize the parameters that are not important to the previous tasks but essential to the new tasks to perform new task adaptation. specifically, our method involves the following steps as shown in figure 2 . first, we fine-tune the original mnmt model on the new training data to obtain a taskspecific model, and then evaluate the importance of parameters in the original mnmt model and the task-specific model separately. second, we remove the parameters that are unimportant to the original tasks but important to the new tasks from the original mnmt model to obtain a pruned model, and we make it responsible for the translation of the original tasks. the pruned model can retain the performance of the original tasks since the removed parameters are not necessary. last, we expand the pruned model to its original size and fine-tune the newly added parameters to learn the new translation tasks. as those new parameters are essential to the new tasks, they can learn the knowledge of the new tasks more efficiently. ## conclusion in this paper, we propose a dual importance-based model division method in continual learning for mnmt, which is based on the importance of parameters to the original and incremental translation tasks. we search for the parameters that are unimportant to the previous tasks but essential to the incremental tasks and utilize those parameters for new task adaptation. experimental results show that our method can efficiently adapt the original mnmt model to various incremental translation tasks and consistently outperforms several previous continual learning methods. further analyses demonstrate that our method can also be applied to the pre-trained mnmt model and benefit the zero-shot translation.