Multilingual Neural Machine Translation With the Right Amount of Sharing (2022)

Abstract: Large multilingual Transformer-based machine translation models have had a pivotal role in making translation systems available for hundreds of languages with good zero-shot translation performance. One such example is the universal model with shared encoder-decoder architecture. Additionally, jointly trained language-specific encoder-decoder systems have been proposed for multilingual neural machine translation (NMT) models. This work investigates various knowledge-sharing approaches on the encoder side while keeping the decoder language- or language-group-specific. We propose a novel approach, where we use universal, language-group-specific and language-specific modules to solve the shortcomings of both the universal models and models with language-specific encoders-decoders. Experiments on a multilingual dataset set up to model real-world scenarios, including zero-shot and low-resource translation, show that our proposed models achieve higher translation quality compared to purely universal and language-specific approaches.

## introduction multilingual neural machine translation has been a fundamental topic in recent years, especially for zero-and few-shot translation scenarios. traditionally, universal nmt models (see fig. 1a ) have been used to produce zero-shot or low-resource translations @xcite . however, previous research has established that universal nmt models with shared encoder-decoder architecture have some disadvantages: (1) high-resource language pairs tend to suffer loss in translation quality @xcite ; (2) the vocabulary of the model increases greatly, especially for languages that do not share an alphabet such as english and japanese; (3) the need to retrain from scratch when a new language does not share the model's vocabulary. recently, there has been renewed interest in multilingual systems, which have jointly trained language-specific encoders-decoders (see fig. 1c ) which we call the modular architecture @xcite . the goal of these models has been to achieve a better overall translation quality compared to universal or uni-directional nmt models. however, there is a disadvantage: lower zero-shot translation quality compared to universal models. to combat this problem, shared encoder/decoder layers (also called interlingua layers) have been proposed @xcite . in this paper, we focus on improving the overall translation quality by using different knowledgeand layer-sharing methods. more specifically, we investigate the effect of sharing encoder layers to improve the generalizability and quality of nmt models. secondly, we present novel language group based models that are inspired by the universal and modular systems. we propose (1) various degrees of granularity (or specificity) of modules (illustrated in fig. 1 ); (2) layer sharing, including combining layers of various granularities into a tiered architecture (illustrated by fig. 2 ). our methods show better translation quality in all testing scenarions compared to the universal model without increasing training or inference time by having variable degrees of modularity or sharing in the encoder. our research looks beyond zero-shot and highresource nmt performance -we set up our experiments to investigate model performance for many data scenarios like zero-shot and low-to high-resource settings. we use a combination of europarl @xcite , emea @xcite , and jrc-acquis @xcite datasets for training and evaluation and six languages grouped into two language groups: germanic (german, english, danish) and romance (french, spanish, portuguese). the results show that our approaches can provide an improvement to universal models in all data scenarios. furthermore, our approaches improve the zero-shot and low-resource translation quality of the modular architecture without harming the high-resource language translation quality. the main contributions of our paper are: ## related works multilingual neural machine translation models follow the encoder-decoder architecture and approaches following this architecture can vary in the amount of parameter sharing @xcite . the most straightforward approach with no parameter sharing would be having a system of unidirectional models. while it is feasible with a small amount of high-resource languages, it becomes problematic in scenarios with low-resource languages or a large number of languages. firstly, the number of uni-directional models in the system grows quadratically with the number of languages, harming maintainability. secondly, there is no transfer learning between language pairs due to separate models, which means that low-resource languages generally have low translation quality. these issues are addressed by pivoting with some success, however, it does not come without tradeoffs @xcite . the main problem with pivoting is that it is not possible to fully utilize all the training data since we only use training data that contains the pivot language. furthermore, due to multiple models being potentially used for a translation, the translation is slower, and there is a chance of error propagation and loss of information. the most widely used approach in multilingual nmt uses a fully shared (universal) model, which has a single encoder and decoder shared between all the languages and uses a token added to the input sentence to indicate the target language @xcite . @xcite identified that the universal model suffers from the capacity bottleneck: with many languages in the model, the translation quality begins to deteriorate. this especially harms the translation quality of high-resource language pairs. zhang et al. (2020) further confirmed this and suggested deeper and language-aware models as an improvement. still, the problem of low maintainability remains, since adding the languages to the model is not possible without retraining the whole model. furthermore, adding languages with different scripts likely results in lower translation quality since the vocabulary can not be altered. @xcite suggested a proof-ofconcept model with language-specific encoders and decoders that started bilingual and was incrementally trained to include other languages. @xcite further improved on it and proposed a joint training procedure that produced a model that outperformed the universal model in translation quality. furthermore, their proposed model is expandable by incrementally adding new languages without affecting the existing languages' translation quality. @xcite investigated the performance of the modular model from the industry perspective. they found that the modular model often outperforms single direction models thanks to transfer learning while being a competitor to the universal model as well due to the additional capacity of languagespecific modules. modular models can contain shared modules as well. @xcite set out to improve the zeroshot performance of modular models, which is often worse than the zero-shot performance of universal models. they achieve this by sharing upper layers of language-specific encoders between all languages. the current paper is an extension of that work. while @xcite used englishcentric training data and denoising autoencoder task to achieve universal interlingua, in this paper we are not using an autoencoder task, since our data is not one language centric. introducing language-specific modules into a universal model can be a good way to increase the capacity of the model without significantly increasing training or inference time. an example of a system that utilizes this is described in @xcite . they use language-specific and language group layers in the decoder of the model following the universal architecture model to provide more capacity. they also note that language-specific layers are more effective when applied to the decoder. @xcite also found that sharing in decoder is not beneficial when there are shared layers in the encoder. these are also the main motivations for focusing on sharing encoder layers in this paper. the dataset is composed of english, german, danish, french, spanish, and portuguese. for creating the dataset and defining models, these are divided into germanic (english, german, danish) and romance (french, spanish, portugese) language groups. we define high-resource (english, german, french), medium-resource (spanish), and low-resource (danish, portuguese) languages that produce high-resource (1,000,000 lines), higher medium resource (500,000 lines), lower medium resource (100,000 lines), lowresource (10,000 lines), and zero-shot (0 lines) language pairs when combined according to the rules in the dataset size is quite small compared to data used for training state-of-the-art models mainly due to limited computational resources. however, we believe that it still allows us to draw conclusions that can be applied at larger scales. ## conclusion in this paper, we propose multiple ways of improving universal models and models with languagespecific encoders-decoders by combining features of both. we experimented with language-and language-group-specific modules and sharing layers of the encoders between all languages, groups of languages, or combining them into a tiered architecture. we found that having some layers universally shared (between all languages) benefits the zero-shot and low-resource translation quality of the modular architectures while not hurting the translation quality of high-resource directions. the modular models with some universally shared layers outperform the universal models in all language-resource types (from zero to high). our best model outperforms the baseline language modular model by 1.04 bleu points and the universal model by 1.19 bleu points on average.