Multilingual Neural Machine Translation: Can Linguistic Hierarchies Help? (2021)

Abstract: Multilingual Neural Machine Translation (MNMT) trains a single NMT model that supports translation between multiple languages, rather than training separate models for different languages. Learning a single model can enhance the low-resource translation by leveraging data from multiple languages. However, the performance of an MNMT model is highly dependent on the type of languages used in training, as transferring knowledge from a diverse set of languages degrades the translation performance due to negative transfer. In this paper, we propose a Hierarchical Knowledge Distillation (HKD) approach for MNMT which capitalises on language groups generated according to typological features and phylogeny of languages to overcome the issue of negative transfer. HKD generates a set of multilingual teacher-assistant models via a selective knowledge distillation mechanism based on the language groups, and then distills the ultimate multilingual model from those assistants in an adaptive way. Experimental results derived from the TED dataset with 53 languages demonstrate the effectiveness of our approach in avoiding the negative transfer effect in MNMT, leading to an improved translation performance (about 1 BLEU score in average) compared to strong baselines.

## introduction the surge over the past few decades in the number of languages used in electronic texts for international communications has promoted machine translation (mt) systems to shift towards multilingualism. however, most successful mt applications, i.e., neural machine translation (nmt) systems, usually rely on supervised deep learning, which is notoriously data-hungry @xcite . despite decades of research, highquality annotated mt resources are only available ⇤ corresponding author for a subset of the world's thousands of languages @xcite . hence, data scarcity is one of the significant challenges which comes along with the language diversity and multilingualism in mt. one of the most widely-researched approaches to tackle this problem is unsupervised learning which takes advantage of available unlabeled data in multiple languages @xcite @xcite . however, unsupervised approaches have relatively lower performance compared to their supervised counterparts @xcite . nevertheless, the performance of the supervised mnmt models is highly dependent on the types of languages used to train the model @xcite . if languages are from very distant language families, they can lead to negative transfer @xcite , causing lower translation quality compared to the individual bilingual counterparts. to address this problem, some improvements have been achieved recently with solutions that employ some sort of supervision to guide mnmt using linguistic typology @xcite @xcite . the linguistic typology provides this supervision by treating the world's languages based on their functional and structural characteristics @xcite . taking advantage of this property, which explains both language similarity and language diversity, we aim in our approach to combine two solutions for training an mnmt model: (a) creating a universal, languageindependent mnmt model @xcite ; (b) systematically designing the possible variations of language-dependent mnmt models based on the language relations @xcite . our approach to preventing negative transfer in mnmt is to group models which behave similarly in separate language clusters. then, we perform a knowledge distillation (kd) @xcite approach by selectively distilling the bilingual teacher models' knowledge in the same language cluster to a multilingual teacher-assistant model. the intermediate teacher-assistant models are representative of their own language cluster. we further adaptively distill knowledge from the multilingual teacher-assistant models to the ultimate multilingual student. in summary, our main contributions are as follows: ## related work the majority of works on mnmt mainly focus on different architectural choices varying in the degree of parameter sharing in the multilingual setting. for example, the works based on the idea of minimal parameter sharing share either encoder, decoder, or attention module @xcite , and those with complete parameter sharing tend to share entire models @xcite . in general, these techniques implicitly assume that a set of languages is pre-given without considering the positive or negative effect of language transfer between the languages shared in one model. hence, they can usually achieve comparable results with individual models (trained with individual language pairs) only when the languages are less diverse or the number of languages is small. when several diverse language pairs are involved in training an mnmt system, the negative transfer @xcite usually happens between more distant languages, resulting in degraded translation accuracy in the multilingual setting. to address this problem, @xcite suggested a clustering approach using either prior knowledge of language families or using language embedding. they obtained the language embedding by retrieving the representation of a language tag which is added to the input of an encoder in a universal mnmt model. later, @xcite introduced another clustering technique using the multi-view language representation. they fused language embeddings learned in an mnmt model with syntactic features of a linguistic knowledge base @xcite . @xcite proposed a knowledge distillation approach which transfers knowledge from bilingual teachers to a multilingual student when the accuracy of teachers are higher than the student. their approach eliminates the accuracy gap between the bilingual and multilingual nmt models. however, we argue that distilling knowledge from a diverse set of parent models into a student model can be sub-optimal, as the parents may compete instead of collaborating with each other, resulting in negative transfer. ## hierarchical knowledge distillation we address the problem of data scarcity and negative transfer in mnmt with a hierarchical knowledge distillation (hkd) approach. the hierarchy in hkd is constructed in such a way that the node structure captures the similarity structure and the relatedness of the languages. specifically, in an inverse pyramidal structure as shown in figure 1 , the root node corresponds to the ultimate mnmt model that we aim to train, the leave nodes correspond to each individual bilingual nmt models, figure 1 : hkd approach: in the first phase of knowledge distillation, aka "selective kd", the knowledge is transferred from bilingual teacher models per clusters (orange circles) to the multilingual teacher-assistant models (green circles). for example t l1 , t l2 , t l4 , and t l6 are belonged to one cluster and distilled to teacher-assistant model t c1 . in the second kd phase, aka "adaptive kd", knowledge is transferred from ensemble of intermediate related teacher-assistant models to the ultimate student (red circle) adaptively. and the non-terminal nodes represent the language clusters. our hypothesis is that leveraging common characteristics of languages in the same language group, which is formed using clustering algorithms based on the typological properties of languages (o'horan et al., 2016), the hkd method can train a high quality mnmt model by distilling knowledge from related languages, rather than diverse ones. our hkd approach consists of two knowledge distillation mechanisms, providing two levels of supervision for training the ultimate mnmt model (illustrated in figure 1 ), including: (i) selective distillation of knowledge from individual bilingual teachers to the multilingual intermediate teacherassistants, each of which corresponds to one language group; and (ii) adaptive distillation of knowledge from all related cluster-wise teacherassistants to the super-multilingual ultimate student model in each mini-batch of training per language pair. note that we do not utilize multilingual adaptive kd in both distillation phases as we need to have the predictions of all the relevant experts in adaptive kd. using adaptive kd for both stages is particularly impractical when there is a huge set of diverse teachers as in the first phase. hence, in the first distillation phase, we aim to generate the cluster-wise teacher assistants using selective kd as the pre-requisites for the adaptive kd phase. the main steps of hkd are elaborated as follows: clustering: clustering can be conducted using different language vectors such as: i) sparse language vectors from typological knowledge base (kb) databases, ii) dense learned language embedding vectors from multilingual nlp tasks, and iii) the combination of kb and task-learned language vec-tors. the implicit causal relationships between languages are usually learned from translation tasks; the genetic, the geographical, and the structural similarities between languages are extracted from typlogical kbs @xcite . thus, the language groups created by different language vectors can contribute differently to the translation and it is not quite clear which types of language features are more helpful in mnmt systems @xcite . for example, "greek" can be clustered with "arabic" and "hebrew" based on the mix of kb and task-learned language vectors. meanwhile, it can be clustered with "macedonian" and "bulgarian" based on nmt-learned language vectors. therefor, we cluster the languages based on all types of language representations and propose to explore a mixture of linguistic features by utilizing all clusters in training the ultimate mnmt student. so, given a training dataset consisting of l languages and k clustering approaches, where each clustering approach creates n clusters, we are interested in training a many-to-one mnmt model (ultimate student) by hierarchically distilling knowledge from all m clusters to the ultimate student, where m := p k @xmath0. ## experiment in this section we explain our experiment settings as well as our experimental findings. data: we conducted extensive experiments on a parallel corpus (53 languages!english) from ted talks transcripts 2 created and tokenized by @xcite . this corpus has 26% of language pairs having less than or equal to 10k sentences (extremely low-resource), and 33% of language pairs having less than 20k sentences (low-resource). all the sentences were segmented with bpe segmentation @xcite with a learned bpe model with 32k merge operations on all languages. we 2 https://github.com/neulab/word-embeddings-for-nmt kept the output vocabulary of the teacher and student models the same to make the knowledge distillation feasible. details about the size of training data, language codes, and preprocessing steps are described in section a.2. clustering: we clustered all the languages based on the three different types of representations discussed in section 3 in order to take advantage of a mixture of linguistic features while training the ultimate student. following oncevay et al. (2020), we adopted their multi-view language representation approach that uses singular vector canonical correlation analysis -svcca @xcite to fuse the one-hot encoded kb representation obtained from syntactic features of wals @xcite and a dense nmt-learned view obtained from mnmt @xcite . specifically, svcca-53 uses 53 languages of ted dataset to build the language representations and generates 10 clusters, the languages within each of which usually have the same phylogenetic or geographical features. svcca-23 instead uses 23 languages of wit-23 @xcite to compute the shared space. we also generated language clusters based on either kb-based representation using syntax features of wals @xcite and nmt-learned representation alone. tables 1-4 show the generated language clusters. i e /b a lt o -s la v ic i e /i t a li c i e /i n d o -i r a n ia n i e /g e r m a n ic t u r k ic u r a li c a f r o a s ia t ic s in o -t ib e t a n a u s t r o n e s ia n koreanic japonic austroasiatic ie/hellenic kra-dai ie/albanian ie/armenian kartvelian mongolic dravidian isolate (basque) be, bs, sl, mk, lt, sk cs, uk, hr, sr, bg, pl, ru bn, ur, ku hi, fa, mr gl, pt, ro fr, es, it nb, da sv, de, nl kk, az, tr et, fi, hu he, ar my, zh ms, id ko ja vi el th sq hy ka mn ta eu training configuration: all models were trained with transformer architecture @xcite . the individual baseline models were trained with the model hidden size of 256, feed-forward hidden size of 1024, and 2 layers. all multilingual models were trained with the model hidden size of 512, feed-forward hidden size of 1024, and 6 layers. we used selective multilingual knowledge distillation approach for training our cluster-based mnmt models as selective multilingual kd perform better than universal mnmt without kd @xcite . we did not carry out intense parameter tuning to search for the best parameter settings of each model for the sake of simplicity. it is noteworthy that the purpose of our experiments is rather to demonstrate the benefit of considering different type of language clusters through a unified hierarchical knowledge distillation. details about the training configuration is provided in section a.2. ## conclusion we presented a hierarchical knowledge distillation (hkd) approach to mitigate the negative transfer effect in mnmt when having a diverse set of languages in training. we put together all languages which behave similarly in the first phase of distillation process and generated the expert teacher-assistants for each group of languages. as