Evaluating Multiway Multilingual NMT in the T urkic Languages (2021)

Abstract: Despite the increasing number of large and comprehensive machine translation (MT) systems, evaluation of these methods in various languages has been restrained by the lack of high-quality parallel corpora as well as engagement with the people that speak these languages. In this study, we present an evaluation of state-of-the-art approaches to training and evaluating MT systems in 22 languages from the Turkic language family, most of which being extremely under-explored. First, we adopt the TIL Corpus with a few key improvements to the training and the evaluation sets. Then, we train 26 bilingual baselines as well as a multi-way neural MT (MNMT) model using the corpus and perform an extensive analysis using automatic metrics as well as human evaluations. We find that the MNMT model outperforms almost all bilingual baselines in the out-of-domain test sets and finetuning the model on a downstream task of a single pair also results in a huge performance boost in both low- and high-resource scenarios. Our attentive analysis of evaluation criteria for MT models in Turkic languages also points to the necessity for further research in this direction. We release the corpus splits, test sets as well as models to the public.

## introduction the last few years have seen encouraging advances in low-resource mt development with the increasing availability of public multilingual corpora @xcite @xcite @xcite @xcite and more inclusive multilingual mt models @xcite @xcite . in this study, we take the 1 https://github.com/ turkic-interlingua/til-mt turkic language family into focus, which has not been studied at large in mt research (detailed review in section 2). most recently, in a wide evaluation of translation between hundreds of languages with a multilingual model (m2m-124) trained on large web-mined parallel data, translation into, from, and between turkic languages was shown to be very challenging compared to other language families @xcite . with the promise of strong transfer capabilities of multilingual models especially for related languages, we hope that the inclusion of a wider set of turkic languages into a joint model can unlock automatic translation even for the very low-resourced turkic languages where no prior translation models exist @xcite @xcite @xcite . to this aim, we adopt the til corpus @xcite speakers, amount of available data (in sentences) in our corpus. the column mt? indicates if there are currently available online machine translation systems for the language. k: thousand, m: million.) lingua 2 community @xcite including x-wmt test sets with a few key improvements (section 3). we train a multi-way nmt (mnmt) model on the entire parallel corpus, which constitutes the first large-scale multilingual translation model specifically for turkic languages (section 4). we perform an extensive analysis of the strengths and weaknesses of this model, comparing it to the bilingual baselines and evaluating it under a domain shift. we find that the mnmt model outperforms almost all bilingual baselines in the out-of-domain tests while it performs comparably or underperforms in the in-domain tests. we further analyze its capacity for transfer learning by fine-tuning the model on several language pairs all of which experience gains, both in-and out-of-domain scenarios. in addition, we complement the automatic evaluation with a human evaluation study for multiple languages (section 5), gaining insights into types of common mistakes that the model makes and the suitability of different automatic metrics for tur-2 https://turkicinterlingua.org/ kic languages. we plan on releasing the improved corpus, evaluation sets, and all the models to the public. this work will not only enrich the landscape of languages currently considered in mt research and spur future research on nlp for turkic languages but will hopefully also inspire the building of new translation engines and derived technologies for populations with millions of native speakers ( ## related work this section discusses the previous work on mt of these languages including the available corpora and languages resources. the 19 turkic languages covered in the study are: altai, azerbaijani, bashkir, crimean tatar, chuvash, gagauz, karachay-balkar, karakalpak, khakas, kazakh, kumyk, kyrgyz, sakha, turkmen, turkish, tatar, tuvan, uyghur, and uzbek. there are several other widely spoken languages that are left out from our study such as shor, salar, urum, nogai, khorasani turkic, qashqai, and khalaj, due to the lack (or very limited amount) of any available parallel corpora. future work will focus on extending the corpus to these languages as well. ## til corpus as we adopt the til corpus as the training data, we perform a few key modifications to better the quality of the datasets. first, we notice that the alignments for the bible foot_1 and tedtalks foot_2 datasets were not optimal as most "sentences" were actually comprised of multiple sentences in order to preserve the quality of the alignment with target sequence. for example, in the case of tedtalks, the original speech utterance may have been 2-3 sentences in text but the translation of that speech may end up differing by 1 or even more sentences depending on the translator. common practice in this situation, as seen through multiple corpora across opus 6 , is to leave the entire utterance as is to preserve the quality of the alignment even if the number of sentences do not match. instead, we drop the examples where the total number of sentences do not match and split (and realign) the cases where they do. this naturally increased the overall number of sentence alignments in both the bible and tedtalks corpora for all language pairs. second, we perform a corpus-wide length and length-ration filtering where we drop sentence pairs that are single words as well as the entries where source and target ratio is over 2. third, we re-curate the in-domain evaluation sets following the improvements to the corpus. details on the evaluation sets are described further in section 3.1. ## the promise of mnmt: cross-lingual knowledge transfer one of the biggest advantages of a large mnmt model is its capacity for transfer learning as can be accomplished through fine-tuning. since we plan on releasing the model to the public, we believe many understudied and underperforming language pairs could benefit from cross-lingual knowledge transfer. this phenomenon is well-known in the broader nlp community as well as in mt research. to test this hypothesis, we fine-tune our mnmt model on 4 language pairs ranging from high(er)resource to extremely low-resource in training data available. ## future work and conclusion in this work, we train and evaluate the first largescale mnmt model for the turkic language family which consists of many underexplored languages. among many results, we find it very promising to train and finetune a mnmt model with a language family corpus as it boosts the cross-lingual knowledge transfer between the related languages and consistently improves over the strong bilingual baselines in out-of-domain scenarios. our analysis also shows that chrf and bleu do not correlate in the same when the target language group if different: bleu underestimates the translations for the turkic languages. in the future work, we hope to include more of the underrepresented turkic language pairs in the study and explore the potential of transfer learning into the translation of unseen languages and language pairs ("zero-shot").