Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021)

Abstract: We propose a straightforward vocabulary adaptation scheme to extend the language capacity of multilingual machine translation models, paving the way towards efficient continual learning for multilingual machine translation. Our approach is suitable for large-scale datasets, applies to distant languages with unseen scripts, incurs only minor degradation on the translation performance for the original language pairs and provides competitive performance even in the case where we only possess monolingual data for the new languages.

## introduction the longstanding goal of multilingual machine translation @xcite @xcite has been to develop a universal translation model, capable of providing high-quality translations between any pair of languages. due to limitations on the data available, however, current approaches rely on first selecting a set of languages for which we have data and training an initial translation model on this data jointly for all languages in a multi-task setup. in an ideal setting, one would continually update the model once data for new language pairs arrives. this setting, dubbed in the literature as continual learning @xcite @xcite , introduces new challenges not found in the traditional multi-task setup, most famously catastrophic forgetting @xcite , in which the model may lose its previously-learned knowledge as it learns new language pairs. this situation is further complicated by the training procedures of standard tokenizers, such as byte-pair encoding (bpe) @xcite or sentencepiece @xcite , which necessitate access to monolingual data for all the languages considered before producing the vocabulary. failing to comply with these requirements, one risks suboptimal segmentation rules which in the worst case could result in strings of @xmath0. in this work, we investigate how vocabularies derived from bpe transform if they are rebuilt with the same settings but with additional data from a new language. we show in section 3.1 that there is a large token overlap between the original and updated vocabularies. this large overlap allows us to retain the performance of a translation model after replacing its vocabulary with the updated vocabulary that additionally supports a new language. past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts @xcite @xcite . these works usually focus solely on learning the new language pair, with no consideration for catastrophic forgetting. moreover, these works only examine the setting where the new language pair comes with parallel data, despite the reality that for a variety of low-resource languages, we may only possess high-quality monolingual data with no access to parallel data. finally, unlike our approach, these approaches do not recover the vocabulary one would have built if one had access to the data for the new language from the very beginning. having alleviated the vocabulary issues, we study whether we are able to learn the new language pair rapidly and accurately, matching the performance of a model which had access to this data at the beginning of training. we propose a simple adaptation scheme that allows our translation model to attain competitive performance with strong bilingual and multilingual baselines in a small amount of additional gradient steps. moreover, our model retains most of the translation quality on the original language pairs it was trained on, exhibiting no signs of catastrophic forgetting. ## continual learning via vocabulary substitution related works adapting translation models to new languages has been studied in the past. neubig and hu (2018) showed that a large multilingual translation model trained on a subset of languages of the ted dataset @xcite could perform translation on the remaining (related) languages. tang et al. (2020) was able to extend the multilingual translation model mbart @xcite ) from 25 to 50 languages by exploiting the fact that mbart's vocabulary already supported those additional 25 languages. @xcite was able to add new languages to machine translation models by training language-specific encoders and decoders. other works @xcite @xcite have studied repurposing translation models as initializations for bilingual models for a target low-resource language pair. most recently @xcite examined reusing language models for highresource languages as initializations for unsupervised translation models for a related low-resource language through the following recipe: build vocabulary v x and a language model for high-resource language x; once data for low-resource language y arrives, build a joint vocabulary v x,y and let v y |x be the tokens from y that appear in v x,y ; substitute the vocabulary for the language model with the one given by v x y v y |x and use the language model as the initialization for the translation model. our approach in this work, we are not only interested in the performance of our multilingual translation models on new language pairs, we also require that our models retain the performance on the multiple language pairs that they were initially trained on. we will also be interested in how the performance of these models compares with those obtained in the oracle setup where we have all the data available from the start. the approaches discussed above generate vocabularies that are likely different (both in selection and number of tokens) from the vocabulary one would obtain if one had a priori access to the missing data, due to the special attention given to the new language. this architectural divergence will only grow as we continually add new languages, which inhibits the comparisons to the oracle setup. we eliminate this mismatch by first building a vocabulary v n on the n languages available, then once the new language arrives, build a new vocabulary v n `1 as we would have if we had possessed the data from the beginning and replace v n with v n `1. we then reuse the embeddings for tokens in the intersection foot_0 and continue training. the success of our approach relies on the fact for large n (i.e. the multilingual setting), v n and v n `1 are mostly equivalent, which allows the model to retain its performance after we substitute vocabularies. we verify this in the following section. ## experiments in this section, we outline the set of experiments we conducted in this work. we first discuss the languages and data sources we use for our experiments. we then provide the training details for how we trained our initial translation models. next, we compute the token overlap between various vocabularies derived from bpe before and after we include data for a new language and empirically verify that this overlap is large if the vocabulary already suppots a large amount of languages. we then examine the amount of knowledge retained after vocabulary substitution by measuring the degradation of the translation performance on the original language pairs from replacing the original vocabulary with an updated one. finally, we examine the speed and quality of the adaptation to new languages under various settings. languages considered our initial model will have to access to data coming from 24 languages foot_1 . our monolingual data comes primarily from the newscrawl datasets foot_2 and wikipedia, while the parallel data comes wmt training sets and paracrawl. we will adapt our model to the following four languages: kazakh, which is not related linguistically to any of the original 24 languages, but does share scripts with russian and bulgarian; bengali, which is related to the other indo-aryan languages but possesses a distinct script; polish, which is related to (and shares scripts with) many of the slavic languages in our original set; pashto, which is not closely related foot_3 to any of the languages in our original set and has a distinct script. we provide an in-depth account of the data available for each language in the appendix. ## conclusion we present an approach for adding new languages to multilingual translation models. our approach allows for rapid adaptation to new languages with distinct scripts with only a minor degradation in performance on the original language pairs.