Language Branch Gated Multilingual Neural Machine Translation (2022)

Abstract: Knowledge transfer across languages is crucial for multilingual neural machine translation. In this paper, we propose language branch (LB) gated multilingual neural machine translation that encourages knowledge transfer within the same language branch with a LB-gated module that is integrated into both the encoder and decoder. The LB-gated module distinguishes LB-specific parameters from global parameters shared by all languages and routes languages from the same LB to the corresponding LB-specific network. Comprehensive experiments on the OPUS-100 dataset show that the proposed approach substantially improves translation quality on both middle- and low-resource languages over previous methods. Further analysis demonstrates its ability in learning similarities between language branches.

## introduction recent years have witnessed a growing interest in multilingual neural machine translation (nmt), which supports translation among multiple languages with one single model @xcite @xcite @xcite @xcite . as the parameters of multilingual nmt are fully or partially shared by multiple languages, knowledge transfer across languages improves translation quality of low-resource languages. despite these advantages, there still exist challenges in multilingual nmt. as previous studies have found, multilingual nmt usually underperforms its bilingual counterparts on high-resource languages @xcite . a way to alleviate this issue is to use language-aware modules, which could be more computationally efficient than simply enlarging * corresponding author. model capacity with deeper/wider models @xcite @xcite . as for lowresource languages, positive transfer is more pronounced among related languages than distant languages @xcite . additionally, explicit language clustering benefits multilingual nmt @xcite although itself can learn linguistic typology during training @xcite . inspired by these, we propose language branch (subfamily) gated multilingual neural machine translation, which fuses language branch information into multilingual nmt by a language branch gated module (lbgm). a language branch is a subfamily of a language family. take the indo-european language family as an example. it can be further divided into subfamilies like germanic, slavic, celtic, etc. the reason why we use language branches rather than language families is that the latter are relatively coarse-grained. languages within a language branch are more closely related to each other than those in a language family. a token that indicates the language branch for the current sentence is fed into lbgm, in addition to the input from other layers in multilingual nmt. with the language branch token, lbgm distinguishes language-branch-specific parameters from global parameters shared by all languages and uses a gate to aggregate these two parts as the output of the module. we conduct experiments on the opus-100 dataset @xcite with a large number of different languages. our main findings can be summarized as follows: ## related work research of multilingual nmt mainly focuses on partial or full parameter sharing in nmt modules @xcite @xcite . @xcite propose prefixing sentences with a language token in a joint set of parallel corpora, using a single nmt model to enable multilingual translation. there is a trade-off between boosting the performance of low-resource languages and sacrificing the performance of highresource languages @xcite . language-specific parameters previous works have been trying to use language-specific parameters to alleviate the trade-off issue in multilingual nmt, such as adding adaptation layers into pretrained models for each language @xcite @xcite . @xcite propose language-aware layer normalization to relax normalization constraint for target languages. @xcite produce masks for different language pairs and use them to select subnetworks for language pairs, in order to counter parameter interference. the closest work to ours is done by @xcite , who introduce a module called clsr into the transformer model. the clsr module adopts a gating function, which is trained with injected zero-mean gaussian noise and discretized at inference time, to choose whether to share the parameters for all languages or not. however, the clsr does not take the relationship between languages into account, since it uses language identity as the router. language clustering @xcite have found that full parameter sharing improves translation quality mainly for related languages that are from the same language group. tan et al. ## methodology we adopt the transformer model as the backbone network @xcite . following @xcite , we extend the transformer to multilingual transformer by prefixing a language token to the source and target side respectively. we cluster languages into their language branches. more specifically, for indo-european languages, we cluster them into 10 branches, including baltic, celtic, east slavic, germanic, hellenic, indo-aryan, iranian, romance, south slavic and west slavic. but for afro-asiatic languages, as there are only five languages in the dataset we use, we just group them into one language branch. for isolated languages, e.g., esperanto, japanese, we keep them in the isolated language group, expecting them to benefit from the positive transfer from all languages. we adapt the multilingual transformer to integrate language branch information with the proposed lbgm, which is illustrated in figure 1 . particularly, we use an additional token, i.e., lb token, to indicate the language branch for each corresponding sentence. lbgm contains a gating function and two feedforward networks, namely, lbspecific ffn and global ffn. the lb-specific ffn is exclusively used for sentences from the corresponding language branch while the global ffn is fully shared across all languages. the purpose of the fully-shared global ffn is to capture global linguistic information from all language pairs, so as to enable knowledge transfer across all languages, especially for the isolated language branch which only contains one or a few languages. the lbspecific ffn is to capture linguistic features for languages from the same language branch and hence enabling intra-language-branch knowledge transfer. lbgm can be formulated as follows: the lb token is used to route information from previous layer into the corresponding lb-specific ffn, which is acting as an "expert" for that language branch. in contrast to the previous clsr method zhang et al. (2021), we do not use the discrete gating function and additional loss component. in our preliminary experiments, we have found that these two components could not bring improvements to our lbgm module and the discretized gating function is even harmful to the lbgm. instead, we use an individual gating function per lbgm sub-module (not shared by the whole model like clsr). this is because, at different positions, the importance of the two types of ffn may be different, which is to be determined by the corresponding gating function in our model. ## experiments we conducted experiments with a massive number of languages to examine the effectiveness of the proposed lbgm. ## analysis ideally, the closer two language branches are to each other, the more similar the parameters of the corresponding lb-specific ffns are to each other in lbgm, especially for language branches which are from the same language family. motivated by this, we calculated the pairwise cosine similarities of lb-specific ffns of any two different language branches learned in lbgm, trying to use these similarities to measure the relationship between language branches. specifically, the cosine similarity is computed with the weight matrix of lb-specific ffn (reshaped into a vector via row-wise concatenation) as follows: figure 2 shows the cosine similarity matrix of different language branches (26 in total contained in the opus-100 dataset) learned by our lbgm. the deeper the color is, the more similar the two language branches are. it is clear to see that b a c e g e ia ie ir r o e s s s w s s e m p a u d r c o n l i ja k o s t k a u r n c t k o g k a l k i ba ce ge ia ie ir ro es ss ws se mp au dr con li ja ko st ka ur nc tk og kal ki 0.0 0.2 0.4 0.6 0.8 1.0 ge (germanic) is more similar to ba (baltic), ws (west slavic), ce (celtic), etc., which are from the same language family, than other language branches, indicated by deeper color in figure 2 . similar results can be observed among ja (japonic), ko (koreanic) and st (sino-tibetan) language branches, among og (oghuz), kal (karluk) and ki (kipchak) language branches, etc. this suggests that the proposed lbgm is able to learn similarities between different language branches. ## conclusions in this paper, we have presented lbgm that uses a lb-specific ffn and a global ffn shared across all languages to enhance knowledge transfer within the same language branch for multilingual neural machine translation. experiments on the opus-100 dataset have shown that lbgm can significantly improve translation quality on both middleand low-resource languages, over the baseline and clsr @xcite . further analysis on lbspecific ffn discloses that the proposed lbgm is able to capture language branch relations.