Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages (2023)

Abstract: With multilingual machine translation (MMT) models continuing to grow in size and number of supported languages, it is natural to reuse and upgrade existing models to save computation as data becomes available in more languages. However, adding new languages requires updating the vocabulary, which complicates the reuse of embeddings. The question of how to reuse existing models while also making architectural changes to provide capacity for both old and new languages has also not been closely studied. In this work, we introduce three techniques that help speed up the effective learning of new languages and alleviate catastrophic forgetting despite vocabulary and architecture mismatches. Our results show that by (1) carefully initializing the network, (2) applying learning rate scaling, and (3) performing data up-sampling, it is possible to exceed the performance of a same-sized baseline model with 30% computation and recover the performance of a larger model trained from scratch with over 50% reduction in computation. Furthermore, our analysis reveals that the introduced techniques help learn new directions more effectively and alleviate catastrophic forgetting at the same time. We hope our work will guide research into more efficient approaches to growing languages for these MMT models and ultimately maximize the reuse of existing models.

## introduction research into multilingual machine translation (mmt) @xcite has shifted from a relatively small number of translation directions @xcite @xcite to much larger scale, recently reaching up to tens of thousands of translation directions @xcite . despite the notable increase in the number of supported languages, these models still need to be upgraded as increasing amount of data in new languages are becoming available. the process of adding new languages to existing models is an instance of continual learning @xcite , in which the models need to effectively learn new tasks (new translation directions) while not catastrophically forgetting @xcite ) the knowledge about tasks from the previous training stage (original translation directions). unlike the most conventional continual learning setup where the model remains the same as the initial learning stage, growing languages for mmt models needs to deal with new parameters. adding new languages to existing training data shifts subword token distribution (e.g., tokens from unseen scripts are added), hence the need to re-train the tokenizer, which adds new embedding parameters to the mmt model. previous studies have shown the effectiveness of adapting embedding parameters to retain performance on old translation directions @xcite @xcite . this is usually done by reusing the embeddings for tokens that overlap between the old and new vocabularies. one aspect that has not been extensively addressed in previous research is how to deal with other architecture mismatches that may arise during the continual learning phase. when growing mmt models to support many additional languages, it also makes sense to increase the model size overall. this extra capacity can be used to learn not only the new directions well but also the old directions better. it is not obvious, however, how to reuse the parameters from existing models (i.e., to engage in continual learning) given such architectural changes. thus, in addition to dealing with different vocabularies, we also investigate how to make use of previously trained models when scaling up model size in order to train much more efficiently than training from scratch. left: the hidden dimension in the feed-forward layers is doubled (d ′ @xmath0. right: additional layers are inserted at the bottom of encoder and the top of decoder so that the model becomes "deeper". both architectural changes increase capacity and are not well-addressed by previous works. in this work, we introduce a recipe that helps significantly reduce the amount of required computation for continually learning new languages in mmt models. the recipe involves training the models with a combination of three techniques: it is our hope that this work will help save computation for future research into large-scale multilingual machine translation, guide more efficient reuse of existing models for continual learning, and allow people to efficiently adapt large publicly-released mmt models for new languages and datasets. ## method adding new languages to existing models, especially languages in new scripts, leads to different subword tokenization, thus different vocabularies, which precludes simple fine-tuning with the exact same model on additional data. during the continual learning stage, we may also want to increase the model size overall to have extra capacity to learn the new languages and improve old languages at the same time. therefore, we also investigate two typical architectural changes commonly done to increase the network capacity: (1) make the model "wider" by expanding the hidden dimension of the feed-forward layers, and (2) make the model "deeper" by inserting new layers to both encoder and decoder. in this section, we delineate three techniques that we found most effective in achieving computation reduction for the continual learning of mmt models. proper initialization. while we can copy weights from the old model foot_0 to the new model, it is not immediately clear how the new parameters (e.g., new token embeddings, new feed-forward weights, new layers) should be initialized and coadapted with the old weights such that maximal knowledge about the old directions is retained. instead of initializing the new parameters randomly, we find that initializing the new embeddings with that @xmath1. when ## experiments languages. we conduct our experiments on 25 languages covering ten language families, four resource levels (high, mid, low, and very low) and four scripts foot_1 . to grow the languages atop existing learned languages, we train the seed model on 20 languages (40 english-centric directions) and add 5 new languages (10 english-centric directions) during the continual learning stage. mimicking the common scenario where newly added languages are mostly low-resource, we select five low and very-low resource languages covering four language families and four scripts as the new languages to add to the seed model. to verify the validity of our approach, we also experiment with two other 20/5 groupings and one 12/13 division of old/new languages in section 5.1 foot_2 . ## effect of each technique 4.1 ablation study to understand the effect of each introduced technique, we conduct an ablation study where each model is trained on the same configuration except for one essential element (i.e., proper initialization, learning rate scaling, data up-sampling). as a naive baseline, while having scaled learning rate and data upsampling, we include the models where no weights from the seed m20 model are used ("random init all") to compare with a less naive baseline "random init new" where the m20 weights are reused and only the newly added parameters (i.e., new token embeddings in all three models, new fully-connected layer weights in mt25 wide and weights of new layers in mt25 deep ) are randomly initialized. to summarize, each configuration in random init all : all parameters are initialized randomly while model is trained with data up-sampling and learning rate scaling. random init new : newly added parameters are initialized randomly while weights of m20 are copied to the new model. model is trained with data up-sampling and learning rate scaling. no up-sampling : model weights are properly initialized and their learning rates are scaled during training. no language pair is up-sampled. no lr scaling : model weights are properly initialized and low-resource pairs are up-sampled whereas no learning rate scaling is applied. results in ## related work our work is closely related to prior research on adapting existing mmt models @xcite to new languages @xcite . @xcite add lowresource languages to multilingual models by finetuning on low-resource data while regularizing with related high-resource data. @xcite introduce simple vocabulary substitution for adapting mmt models to new languages without any architectural changes. another line of research employs modular approaches, which include training lightweight adapters @xcite , language-specific encoder-decoders @xcite , language specific embeddings @xcite for learning new languages. while sometimes escaping the need to train on old examples, growing the model in a modular fashion @xcite requires non-trivial changes to standard architectures. in contrast, our work relies on rehearsal mechanism (i.e., also train on old examples) but does not need to modify network structures @xcite . our approach is also related to works that focus on continual learning of mt models for adapting multiple domains. @xcite ; @xcite adopt a method derived from elastic weight consolidation @xcite to alleviate catastrophic forgetting. while most prior works only investigate two-stage continual learning, @xcite propose a new framework that extends to multi-stage training to mitigate catastrophic forgetting @xcite . the initialization of the new parameters and embeddings in our technique is also related to that in @xcite , which accommodate multilingual models to unseen scripts via matrix factorization. our focus on the architectural differences between initial and continual learning phase is also relevant to recent discoveries that wider networks forget less catastrophically @xcite . ## conclusion we show in this work that it is possible to efficiently bootstrap from existing models and recover the baseline performance with much less computation while vocabularies and architectures can be different in the continue learning stage. we highlight the importance of (1) reusing the existing model weights and carefully initializing the new parameters, (2) applying learning rate scaling, and (3) performing data up-sampling. analyses reveal that scaling down the learning rate for old parameters helps alleviate catastrophic forgetting, and that data up-sampling is vital to achieving good performance on the new directions. we hope our work can help save computation for research into large-scale multilingual mt models, and more generally, will help spur research into continual multitask learning in the presence of architectural changes.