Continual Learning in Multilingual NMT via Language-Specific Embeddings (2021)

Abstract: This paper proposes a technique for adding a new source or target language to an existing multilingual NMT model without re-training it on the initial set of languages. It consists in replacing the shared vocabulary with a small language-specific vocabulary and fine-tuning the new embeddings on the new language’s parallel data. Some additional language-specific components may be trained to improve performance (e.g., Transformer layers or adapter modules). Because the parameters of the original model are not modified, its performance on the initial languages does not degrade. We show on two sets of experiments (small-scale on TED Talks, and large-scale on ParaCrawl) that this approach performs as well or better as the more costly alternatives; and that it has excellent zero-shot performance: training on English-centric data is enough to translate between the new language and any of the initial languages.

## introduction multilingual neural machine translation models are trained on multilingual data to translate from and/or into multiple languages @xcite . multilingual nmt is a compelling approach in production, as one only needs to train, deploy and maintain one model (instead of 2 × n ones, where n is the number of languages). it has also been shown to improve mt quality for low-resource languages (at the cost of a slight degradation for high-resource languages) and it can allow translation between languages that have no aligned data ("zero-shot translation"). however, such models can be costly to train, as they usually involve larger architectures and large datasets. moreover, because they are trained jointly on all the languages, they require to know in advance the full set of languages. adding a new language to an existing model usually means retraining the model on the full multilingual dataset. naively fine-tuning the original model on the new language's data is not an option because of vocabulary mismatch (the shared vocabulary needs to be modified to include the new language's tokens) and catastrophic forgetting (the model will quickly forget how to translate in the other languages). in this paper, we study the problem of multilingual nmt incremental training or continual learning and propose a novel way to efficiently add a new source or target language. some desirable properties of an incremental training method are: we propose a novel technique for incrementally adding a new source or target language, which consists in substituting the shared embedding matrix with a language-specific embedding matrix, which is fine-tuned on the new language's data only while freezing the other parameters of the model. in some cases (e.g., when the new language is on the target size), a small number of additional parameters (e.g., adapter modules) have to be trained to match the performance of the re-training baseline. we perform two sets of experiments, with a 20-language transformer base trained on ted talks, and a 20language transformer big (with deep encoder and shallow decoder) trained on paracrawl; and show that this approach is fast and parameter-efficient and that it performs as well or better as the more costly alternatives. ## related work some previous works study how to adapt a multilingual mt model to unseen low-resource languages, but without seeking to maintain good performance in the initial languages @xcite . @xcite introduce a "vocabulary substitution" approach for adding new languages to a multilingual nmt model. they create a new shared bpe vocabulary that includes the new language and initialize the embeddings of the overlapping tokens with their previous values. then they fine-tune the entire model on the initial model's training data combined with parallel or monolingual data in the new language. contrary to ours, their approach assumes access to the initial model's training data and results in a small performance drop in the existing languages. @xcite and @xcite @xcite propose multi-decoder / multi-encoder architectures which they show to be compatible with incremental training. to add a new target (resp. source) language, one just has to freeze the model's encoder (resp. decoder) and train a new languagespecific decoder (resp. encoder). however, this results in an enormous number of parameters. @xcite ; @xcite incrementally train language-specific embeddings for cross-lingual transfer of bert classification models. this approach consists of four stages: 1) train a monolingual bert on language l 1 ; 2) train embeddings on language l 2 using the masked lm objective while freezing the other parameters; 3) fine-tune the l 1 bert model on the desired classification task using labeled data in language l 1 ; 4) substitute the l 1 embeddings with the l 2 embeddings in the classification model and use it for l 2language classification. @xcite also combine their approach with l 2 -specific adapter layers and position embeddings. while this algorithm is close to ours, it is used on encoder-only transformers for classification tasks. our work extends this algorithm to encoder-decoder transformers for multilingual mt. also similar to our technique, @xcite do domain adaptation by freezing most of the nmt parameters and only fine-tuning one component (e.g., the source embeddings). @xcite show that adapter modules can be used to adapt an english-centric multilingual model to unseen language pairs, but whose source and target languages are known. we wanted to go further and use adapter layers to adapt a multilingual model to unseen languages. however, we obtained the surprising result that adapting the embedding matrix is sometimes enough. in the other cases, adapter modules can be used sparingly to match baseline performance. @xcite introduce "denoising adapters" which they show can be used to incrementally adapt a multilingual mt model to new languages using monolingual data only. the initial model is a many-to-many model with a shared vocabulary and source-side language codes (to indicate the target language). ## ted talks experiments we adapt a 20-language model trained on ted talks to greek (el), either on the source side or target side. we pick greek as the new language as it is from an unseen language family and uses an unseen alphabet. we also do experiments with ukrainian (uk), indonesian (id), or swedish (sv) as the new language, foot_0 which are shown in appendix. ## paracrawl experiments in this section, we test our approach in a more realistic, large-scale setting: a 20-language transformer big initial model trained on paracrawl @xcite . the incremental training experiments are done in three languages: arabic (ar), russian (ru), and chinese (zh). arabic and chinese are both from unseen language families and use unseen scripts. russian is close to a known language (bulgarian) and uses the same script. for training on those languages, we use unpc @xcite . ## conclusion we propose a new technique for incrementally training multilingual nmt models on a new source or target language. it consists in creating a new monolingual bpe vocabulary for that language, substituting the shared embedding matrix with language-specific embeddings, and training those while freezing the other model parameters. at inference, translating in any of the initial languages is done by using the initial shared embeddings, and translating in the new language is done by using the newly trained embeddings. this approach does not change performance on the initial languages as the initial parameters are kept aside and not modified. for new source languages, it can achieve close performance to the more costly and less flexible bilingual and re-training baselines. for new target languages, this technique can be combined with language-specific parameters (finetuned transformer layers or adapter modules) to match baseline performance at a small parameter cost. we validate this technique on two sets of experiments: small-scale on ted talks and largescale on paracrawl; and show that it is compatible with two architectures: transformer base 6-6 and big 12-2. we also show that incremental training on data aligned with english is enough to learn to translate between the new language and any of the initial languages. translation between a new source and a new target language is also possible by combining their respective parameters at inference. finally, we provide supplementary material to facilitate reproducibility. 11 a appendix