Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders (2021)

Abstract: State-of-the-art multilingual machine translation relies on a universal encoder-decoder, which requires retraining the entire system to add new languages. In this paper, we propose an alternative approach that is based on language-specific encoder-decoders, and can thus be more easily extended to new languages by learning their corresponding modules. So as to encourage a common interlingua representation, we simultaneously train the N initial languages. Our experiments show that the proposed approach outperforms the universal encoder-decoder by 3.28 BLEU points on average, while allowing to add new languages without the need to retrain the rest of the modules. All in all, our work closes the gap between shared and language-specific encoderdecoders, advancing toward modular multilingual machine translation systems that can be flexibly extended in lifelong learning settings.

## introduction multilingual machine translation is the ability to generate translations automatically across a (large) number of languages. research in this area has attracted a lot of attention in recent times both from the scientific and industrial community. under the neural machine translation paradigm @xcite , the opportunities for improving this area have dramatically expanded. thanks to the encoder-decoder architecture, there are viable alternatives to expensive pairwise translation based on classic paradigms foot_0 . the main proposal in this direction is the universal encoder-decoder @xcite with massive multilingual enhancements @xcite . while this approach enables zero-shot translation and is beneficial for low-resource languages, it has multiple drawbacks: (i) the entire system has to be retrained when adding new languages or data; (ii) the quality of translation drops when adding too many languages or for those with the most resources @xcite ; and (iii) the shared vocabulary grows dramatically when adding a large number of languages (especially when they do not share alphabets). other limitations include the incompatibility of adding multiple modalities such as image or speech. in this paper, we propose a new framework that can be incrementally extended to new languages without the aforementioned limitations ( §3). our proposal is based on language-specific encoders and decoders that rely on a common intermediate representation space. for that purpose, we simultaneously train the initial n languages in all translation directions. new languages are naturally added to the system by training a new module coupled with any of the existing ones, while new data can be easily added by retraining only the module for the corresponding language. we evaluate our proposal on three experimental configurations: translation for the initial languages, translation when adding a new language, and zero-shot translation ( §4). our results show that the proposed method is better in the first configurations by improving the universal system: by 0.40 bleu points on average in the initial training and by 3.28 bleu points on average when adding new languages. however, our proposed system is still lagging behind universal encoder-decoder in zero-shot translation. ## related work multilingual neural machine translation can refer to translating from one-to-many languages @xcite , from many-to-one @xcite and many-to-many @xcite . within the many-to-many paradigm, existing approaches can be further divided into shared or language-specific encoder-decoders. shared encoder-decoder. joh (2017) feed a single encoder and decoder with multiple input and output languages. given a set of languages, a shared architecture has a universal encoder and a universal decoder that are trained on all initial language pairs at once. the model shares parameters, vocabulary and tokenization among languages to ensure that no additional ambiguity is introduced in the representation. this architecture provides a simple framework to develop multilingual systems because it does not require modifications of a standard neural machine translation model, and information is easily shared among the different languages through common parameters. despite the model's advantages in transfer learning, the use of a shared vocabulary and embedding representation forces the model to employ a vocabulary that includes tokens from all the alphabets used. additionally, recent work @xcite , that imposes representational invariance across language, shows increasing the number of languages varies the quality of the languages already in the system (generally enhancing low-resource pairs but being detrimental for highresource pairs). language-specific encoder-decoders which may or may not share parameters at some point. @xcite proposed extending the bilingual recurrent neural machine translation architecture @xcite to the multilingual case @xcite by designing a shared attention-based mechanism between the languagespecific encoders and decoders to create a language independent representation. as the language specific components rely on the shared modules, modifying those components to add a new language or add further data to the system would require retraining the whole system (similarly to the previous shared approach). @xcite proposes a model based on the addition of new languages to an already trained system by vocabulary adaptation and transfer learning. while limited, it requires some retraining to adapt the model to the new task. ## proposed method our proposed approach trains a separate encoder and decoder for each of the n languages available without requiring multi-parallel corpus. we do not share any parameter across these modules, which allows to add new languages incrementally without retraining the entire system. ## experiments in multilingual machine translation in this section we report machine translation experiments in different settings. since the main difference between the shared and the languagespecific encoders-decoders lies in whether they retrain the entire system when adding new languages, we accordingly design our experiments to compare the systems under this condition. ## conclusions in this paper, we present a novel method to train language-specific encoders-decoders without sharing any parameters at all. more relevantly, our system allows to incrementally add new languages into the system without having to retrain it and without varying the translation quality of initial languages in the system. when adding a new language, the language-specific encoder-decoders outperform the shared ones by 3.28 bleu on average and, most importantly, the training of this new language was done in only one day, as opposed to the week taken by the shared system.