Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation (2023)

Abstract: In the real-world scenario, a longstanding goal of multilingual neural machine translation (MNMT) is that a single model can incrementally adapt to new language pairs without accessing previous training data. In this scenario, previous studies concentrate on overcoming catastrophic forgetting while lacking encouragement to learn new knowledge from incremental language pairs, especially when the incremental language is not related to the set of original languages. To better acquire new knowledge, we propose a knowledge transfer method that can efficiently adapt original MNMT models to diverse incremental language pairs. The method flexibly introduces the knowledge from an external model into original models, which encourages the models to learn new language pairs, completing the procedure of knowledge transfer. Moreover, all original parameters are frozen to ensure that translation qualities on original language pairs are not degraded. Experimental results show that our method can learn new knowledge from diverse language pairs incrementally meanwhile maintaining performance on original language pairs, outperforming various strong baselines in incremental learning for MNMT.

## introduction multilingual neural machine translation (mnmt) aims at handling multiple translation directions in a single model and achieves great success in recent years @xcite @xcite . however, the powerful mnmt models need to be retrained from scratch using a mixture of original and incremental training data when new language pairs arrive @xcite . considering that the original training data of mnmt models is often large-scale @xcite , and thus the method that utilizes original data to train incrementally is time-consuming and cumbersome @xcite . therefore, a practical scenario is that these models can continually support new language pairs while preserving the previously learned knowledge without accessing previous training data, which belongs to incremental learning, and has drawn much attention recently @xcite @xcite . in this scenario, existing studies attempt to overcome the issue of catastrophic forgetting @xcite on original language pairs, such as replaybased methods @xcite and regularization-based methods @xcite . however, the methods primarily focus on balancing performance between old and new translation directions and use only incremental data to acquire new knowledge, restricting the development of new language pairs @xcite . as shown in fortunately, the development of transformerbased language models unveils that feed-forward networks (ffn) might be a core component that stores the knowledge @xcite @xcite . in these efforts, external knowledge is injected into ffn layers to enhance the performance of pre-trained language models. more importantly, it opens the door to leveraging the knowledge from neural models in adapting mnmt models to incremental language pairs. in this work, considering the knowledge in neural networks, we propose a knowledge transfer (kt) method that can efficiently adapt mnmt models to diverse incremental language pairs. first, we convert incremental training data into continuous representation by additional parameters, forming pluggable modules. then the pluggable modules can be flexibly introduced in the embedding layer and ffn layers of original mnmt models, respectively. the two stages are regarded as a process of knowledge transfer and equip original models with knowledge of unseen languages before adaptation, alleviating the representation gap between original models and introduced parameters. and the knowledge transfer method can further provide better optimization than training from scratch for the introduced parameters. moreover, except for the pluggable modules, all the parameters of the original model are frozen. therefore, our architecture can also retain previously learned knowledge from the original translation model to completely maintain the translation qualities on original language pairs. to sum up, our contributions are as follows: replay-based methods. the first branch of works utilizes previous training data or create pseudo data that is essentially a replay on old tasks (de masson d' @xcite @xcite . specifically, previous data sometimes cannot be accessed due to data protection and security @xcite . in this scenario, @xcite replay pseudo samples of previous tasks, which can avoid forgetting previously learned knowledge. however, the pseudo data with noise significantly hurts the performance for both old and new tasks, and the data generation procedure requires additional time costs, restricting the efficiency of incremental learning @xcite . in contrast to these methods, our approach does not require extra data and is more flexible in the real-world scenario. regularization-based methods. the second branch of works introduces additional penalty terms to the learning objective on the parameters, alleviating the issue of catastrophic forgetting @xcite @xcite . in particular, @xcite propose a complementary online knowledge distillation method that utilizes previous models (teachers) to supplement current model (student) training. in contrast to these methods, our architecture can naturally avoid forgetting previously learned knowledge and retain the performance of old tasks. it allows our method to focus solely on learning new knowledge better instead of preserving old knowledge. ## method in this work, we aim to completely maintain the performance of previous translation tasks without original training data. as shown in figure 1 , we introduce additional components for new language pairs and adopt a strategy that does not disturb the parameters of the original model. we hope to minimize the impact on the original model during the incremental learning process. as a result, we exclusively concentrate on how to handle the situation of learning new language pairs. furthermore, the additional components are transferred from parameters in another pre-trained translation model, in a similar way to a pluggable module, rather than randomly initialized, as shown in figure 2 . it can also reduce the cost of learning new language pairs during the training stage, enhancing the practicability and efficiency of incremental learning methods in the real-world scenario. ## conclusion in this work, we propose a knowledge transfer method in incremental learning for mnmt, which leverages the knowledge from neural models. it can encourage original models to learn new knowledge from updated training data while naturally mitigating the issue of degradation on previous translation directions. moreover, it is more efficient to utilize the knowledge transfer scheme than introducing randomly initialized parameters in incremental learning. experimental results demonstrate that the proposed method outperforms several strong baselines in the comprehensive language consideration.