ICL : Iterative Continual Learning for Multi-domain Neural Machine Translation (2024)

Abstract: In a practical scenario, multi-domain neural machine translation (MDNMT) aims to continuously acquire knowledge from new domain data while retaining old knowledge. Previous work separately learns each new domain knowledge based on parameter isolation methods, which effectively capture the new knowledge. However, task-specific parameters lead to isolation between models, which hinders the mutual transfer of knowledge between new domains. Given the scarcity of domain-specific corpora, we consider making full use of the data from multiple new domains. Therefore, our work aims to leverage previously acquired domain knowledge when modeling subsequent domains. To this end, we propose an Iterative Continual Learning (ICL) framework for multi-domain neural machine translation. Specifically, when each new domain arrives, (1) we first build a pluggable incremental learning model, (2) then we design an iterative updating algorithm to continuously update the original model, which can be used flexibly for constructing subsequent domain models. Furthermore, we design a domain knowledge transfer mechanism to enhance the fine-grained domain-specific representation, thereby solving the word ambiguity caused by mixing domain data. Experimental results on the UM-Corpus and OPUS multi-domain datasets show the superior performance of our proposed model compared to representative baselines.

## introduction multi-domain neural machine translation (md-nmt) aims to train a single model with mixeddomain data and has shown great performance in recent years @xcite @xcite . traditional work @xcite fine-tunes a model for each domain based on an original model. how- † yujie zhang is the corresponding author. ever, fine-tuning models individually for each domain increases maintenance costs and limits the ability to utilize diverse knowledge. therefore, a smarter and more practical approach is to continuously update the original translation model with new domain data, rather than fine-tuning multiple domain-specific models. in this scenario, the key challenge is how to continuously and sequentially learn new domain knowledge while avoiding catastrophic forgetting, which is a challenge in the field of continual @xcite . currently, existing studies can be divided into three lines: (i) replay-based methods: these methods retain part or all training data from previous tasks (de masson d' @xcite @xcite @xcite . (ii) regularizationbased methods: these methods aim to approximate the loss incurred on previous tasks and are usually in quadratic form @xcite @xcite . these methods effectively learn the knowledge between different domains. however, they cannot completely avoid the problem of catastrophic forgetting. to address this limitation, researchers propose (iii) parameter isolationbased methods: these methods design separate pluggable modules and freeze all original parameters to completely retain the performance on previous tasks @xcite @xcite @xcite . these methods are also called "plug and play" @xcite @xcite . however, the parameters introduced for each domain in these methods are learned independently, so subsequent domains cannot leverage knowledge from previous domains. to deal with this challenge, we propose an iterative continual learning (icl) method for multi-domain neural machine translation based on a parameter isolation framework. in this work, we aim to address the challenge of utilizing knowledge across multiple new domains. specifically, our approach consists of three stages: first, we train original and incremental models for the original and new domains, respectively. then we attach parameters to the original model to create pluggable modules via domain knowledge transfer, including the word embeddings layer and ffn layers. finally, as new domains arrive, the pluggable modules from the previous domain are continuously integrated into the original model for the new domain. compared to previous work, our approach not only prevents catastrophic forgetting but also makes full use of the knowledge across domains. moreover, comprehensive experimental results and analyses on multiple language pairs demonstrate that our proposed model improves across all baselines. in conclusion, our contribution can be summarized as follows: ## related work recent work on continual learning of nmt can be divided into three categories: replay-based methods. the first category of methods requires retaining part or all of the training data from previous tasks @xcite @xcite @xcite . however, these methods result in higher training costs, particularly when applied to large-scale pre-trained multi-lingual neural machine translation (mnmt) models. furthermore, previous training data may be inaccessible due to privacy concerns or storage limitations. in contrast, our approach does not require additional data and offers greater flexibility for continual learning. regularization-based methods. the second category of works alleviates catastrophic forgetting by adding penalty terms to the learning objective, balancing performance between previous and new tasks @xcite @xcite . in this scenario, @xcite uses a hard constraint to update parameters in regions with a low risk of forgetting. @xcite introduce an online knowledge distillation approach, where previous models assist in training the current model. in contrast to these methods, our framework naturally prevents catastrophic forgetting. ## method in this work, our goal is to address the challenge of how to learn knowledge across multiple new domains. as shown in figure 1 (a), the original model needs to build a single pluggable module for each new domain, and the knowledge between multiple new domains cannot learn from each other within the framework of pluggable incremental learning (pil) @xcite . however, in real-world scenarios, the domain-specific corpus is typically low-resource, and new domains often arrive in stages over time. consequently, we aim to effectively utilize knowledge from multiple new domains. specifically, we focus on learning knowledge from different domains by iteratively updating the original model, as shown in figure 1 ( @xcite . compared with pil, our approach differs in two key aspects: domain knowledge transfer strategy ( § 3.2) and iterative continual learning framework ( § 3.3). ## experiments we conduct experiments for our framework (icl) to explore the following questions: (i) can icl learn more knowledge between new domains under low-resource domain scene? ( § 4.3) (ii) how is the performance of icl on original domains compared with previous work? ( § 4.3) (iii) which factors affect the performance of icl? ( § 4.5, § 4.6) ## conclusion in this work, we propose an icl method for multidomain neural machine translation, this method can makes full use of knowledge between multiple new domains. specifically, our framework constantly updated original model to obtain the new knowledge. moreover, we design the domain knowledge transfer mechanism to enhance the domain-specific features of word. experimental results demonstrate the effectiveness of our approach.