A Large-Scale Study of Machine Translation in T urkic Languages (2021)

Abstract: Recent advances in neural machine translation (NMT) have pushed the quality of machine translation systems to the point where they are becoming widely adopted to build competitive systems. However, there is still a large number of languages that are yet to reap the benefits of NMT. In this paper, we provide the first large-scale case study of the practical application of MT in the Turkic language family in order to realize the gains of NMT for Turkic languages under high-resource to extremely low-resource scenarios. In addition to presenting an extensive analysis that identifies the bottlenecks towards building competitive systems to ameliorate data scarcity, our study has several key contributions, including, i) a large parallel corpus covering 22 Turkic languages consisting of common public datasets in combination with new datasets of approximately 1.4 million parallel sentences, ii) bilingual baselines for 26 language pairs, iii) novel high-quality test sets in three different translation domains and iv) human evaluation scores. All models, scripts, and data will be released to the public.

## introduction having been studied widely over the last few decades, machine translation (mt) evaluation has traditionally focused on european languages, due to limitations of the available technology as well as resources. although low-resource mt has recently started to gain more attention and new evaluation benchmarks are becoming available @xcite @xcite , there are still a large amount of underrepresented languages excluded from mt evaluation. in addition to the cost of preparing such labor-intensive annotations, the lack of training resources also limits the evaluation of mt models in in order to extend the evaluation of the state-ofthe-art methods in mt @xcite and ultimately aid in designing methods with wider range of applicability, in this paper, we present a largescale case study of mt methods in a very challenging case of the turkic language family. the turkic language family consists of around 35 languages spoken by communities across eurasia by around 200 million people. of this number, around 20 are official languages of a state, or sub-national entity, with the remaining being minority languages. the languages are distinct in their highly complex use of morphology, and thus create extremely sparse vocabularies, presenting a challenging case of evaluation of statistical models, in particular mt systems @xcite and n-gram language models @xcite . our study includes the preparation of novel public resources covering many languages in the turkic family, most of which included for the first time in parallel corpora. we also present new benchmarks for mt which could be used for assessing different factors determining the limits of mt methods in various languages, such as data size, evaluation metrics, translation domain, linguistic typology, relatedness, and the writing system. we test the use of our resources in mt and present the first evaluation results for many turkic languages. our novel resources consist of i) a large-scale multicentric parallel corpus of 75m+ sentence pairs in 22 turkic languages and their translations into english, russian, as well as in-family languages, covering over 400 translation directions, ii) 3 new test sets for each translation direction curated from our corpus in 3 different translation domains, iii) bilingual baselines in 26 different language pairs. our baselines are evaluated using automatic metrics as well as human assessments against commercial or open-source systems where applicable. we release our parallel corpora, test sets, and baseline systems publicly to encourage future research in turkic languages. ## turkic languages & mt this section gives a brief overview of turkic languages from a linguistic perspective as well as presenting the previous work on mt of these languages. in our study, we include 22 turkic languages: altai, azerbaijani, bashkir, crimean tatar, chuvash, gagauz, karachay-balkar, karakalpak, khakas, kazakh, kumyk, kyrgyz, sakha, salar, shor, turkmen, turkish, tatar, tuvan, uyghur, urum, and uzbek. there are several other widely spoken languages that were left out from our study such as nogai, khorasani turkic, qashqai, and khalaj, due to the lack of any available parallel corpora. future work will focus on extending the corpus to these languages as well. ## til corpus our parallel corpus is collected through unifying publicly available datasets and additional parallel data we prepare by crawling public domain resources. ## bilingual baselines we train bilingual baselines for 26 language pairs in three different resource categories: high (¿5m), medium (100k-5m) and low (¡100k). the choice of pairs to train was based on multiple factors such as the availability of test sets, native speakers (for human evaluation), and other comparable mt systems. ## evaluation automatic evaluation metrics are very commonplace in mt research, and there has been a recent line of work exploring better metrics that capture translation quality beyond the syntactic and lexical features @xcite @xcite . methods relying on contextual embeddings to capture the semantic similarity between the hypothesis and references fall short in terms of their language coverage. this is largely due to the pretraining of these evaluation models that require a significant of monolingual data which most of the low-resource languages lack. in this study, we evaluate our systems using both automatic metrics and human evaluation of translations. ## results & discussion the upper section of baseline results for the mid-and low-resource pairs are in the lower part of another notable aspect is the importance of scripts in the performance of the models. language pairs with more than one script consistently underperform (both in automatic and human evaluations) the ones where both the source and target language use the same script. in fact, the best 6 models on the x-wmt test sets all have latin scripts in both the source and target language. a suboptimal performance in the face of a script disparity is a known phenomenon @xcite @xcite , where techniques such as transliteration show to improve performance. this is mostly attributable to model's inability to represent both languages in a shared space effectively when they do not share the same script, which can be damaging for the downstream performance. ## conclusion & future work in this paper, we introduce a large parallel corpus covering 22 turkic languages along with in-domain and out-of-domain evaluation sets. we also train the first baseline models for several language pairs and take the initial steps to address the challenges associated with machine translation in the turkic languages. this study was carried out as in a participatory research setting by a diverse community of researchers, engineers, language specialists, and native speakers of turkic languages. future work will focus on studies of methods for effective crosslingual transfer, extending of the coverage of the corpus to more languages and domains, and increasing the size of the test sets to provide more comprehensive benchmarks.