The Only Chance to Understand: Machine Translation of the Severely Endangered Low-resource Languages of Eurasia (2022)

Abstract: Numerous machine translation systems have been proposed since the appearance of this task. Nowadays, new large language model-based algorithms show results that sometimes overcome human ones on the rich-resource languages. Nevertheless, it is still not the case for the low-resource languages, for which all these algorithms did not show equally impressive results. In this work, we want to compare 3 generations of machine translation models on 7 low-resource languages and make a step further by proposing a new way of automatic parallel data augmentation using the state-of-the-art generative model.

## introduction being one of the oldest tasks of natural language processing (nlp), machine translation changed many different state-of-the-art approaches over the past 70 years. starting with old dictionarybased systems, then going forward with statistical algorithms, switching to neural approaches with sequence-to-sequence methods, currently, the best mt systems use language models (lm) with transformer architecture inside. all these new language models rely on huge data corpora from which they are able to extract general patterns about any language including grammar, vocabulary, discourse characteristics, etc. their results are especially remarkable on the translation tasks from one rich-resource language to another, where they achieve results sometimes indistinguishable from the human ones. however, when it comes to the low-resource languages, not all models can perform well. recently, new large lms were developed especially for fewshot learning, however, they are still evaluated on the datasets containing several tens of thousands of samples. in this work, we want to evaluate the performance of the algorithms coming from 3 differ-ent generations of models: statistical, sequenceto-sequence and transformer-based. additionally, we want to propose a new fully automatic parallel data augmentation method based on gpt model and compare the quality of the models fine-tuned with the generated data. for our purposes, we will take 7 extremely lowresource languages of eurasia coming from 4 different language families. all these languages will be the source languages of translation, while russian will be the target one. these languages are mainly spoken in russia, so linguists have already collected small corpora of linguistic data for these languages including the sentences translations to russian. we extracted sentences from these corpora and composed 7 datasets of parallel sentences. the minimum size of the corpus used is 586 training pairs and the maximum size is equal to 8619. in the following, we will start by presenting the related work that has been carried out so far (section 2), then the languages used for this study (section 3). after this, we will describe the experiments (section 4) and analyse the obtained results (section 5). all the contributions made in this paper will be summed up in the conclusion (section 6) as well as the direction of the future work. ## related work throughout the history of machine translation, numerous models have been proposed. during the era of statistical machine translation, one of the first models were word-based models such as ibm ones @xcite . these models were then followed by the phrase-based systems @xcite which became widely used for several years. the next decade was marked by the appearance of neural network-based machine translation algorithms starting with a sequence-to-sequence model with lstm layers @xcite which was then modified with cnn @xcite and different types of attention mechanism @xcite . in the recent years, the transformer architecture @xcite appeared and it showed groundbreaking results in many nlp tasks. for machine translation, firstly, the original transformer paper showed new state-of-the-art scores and then the metrics were improved by the t5 model @xcite , the multilingual mbart-25 @xcite and mbart-50 @xcite models followed by the other transformer-based architectures. these three generations of models have also been used in the low-resource settings. we can find adaptions of all kinds of algorithms for the under-resourced conditions. for example, the phrase-based statistical models have been used for the translation of the low-resource arabic dialects @xcite . as for the sequence-to-sequence models, an interesting approach to data processing and further seq2seq model training and tuning was shown in the paper by @xcite . the authors transliterated all low-resource languages that they had into the common alphabet shared with a richresource language coming from the same language group. then they pre-trained a sequence-tosequence model using the corpus of a rich-resource language and fine-tuned it with small corpora of the low-resource languages. another example of the successful application of the seq2seq model to the low-resource machine translation is the multitask training using the translation task from and to several dialects at the same time @xcite . the transformer-based models have also been tested in the low-resource conditions. for example, @xcite proposed a new 3-stage training approach with no data for the low-resource languages. the authors trained the transformer model using the corpora of the close rich-resource languages. additionally, they used the so-called synthetic corpora which contained the translations of the sentences from all zero-source languages which they generated using the model obtained after the first stage of training. as we can see, the main approach that is used to improve the quality of the translation is transferring some knowledge from the languages that are coming from the same language group or family. moreover, these language families have many daughter languages that are popular in the world. however, in our work, some of the languages either are the only remaining living languages of their family or come from the families which are not widely known, so we will try to exploit some approaches that do not rely on the languages similarities. ## results in this section, we will show and discuss the performance of all machine translation models that we implemented. the figures with the comparison of all results for every language are presented in the appendix a. ## conclusion in this study, we have presented our work on the machine translation for 7 low-resource languages of eurasia. we have compared the phrase-based statistical model, the seq2seq model, the mbart model and the rugpt-3 model. we have shown that the statistical model achieves the highest quality for the majority of the languages and mbart model shows the best quality for the remaining ones. we have also proposed the new way of augmenting the dataset with parallel sentences generated by the gpt-model fine-tuned for the translation task. the study has shown that this method allows to increase the quality of the model starting from a certain size of the training dataset, otherwise the quality decreases as the gpt model is not able to generate coherent examples. our future directions of research include training other transformer-based architectures like m2m100 and using multi-task learning during the fine-tuning stage. by this work, we would like to bring attention to the low-resource languages of eurasia and encourage other researchers to continue our work. every language is the part of the world's treasure and it is important to do our best trying to preserve them. model translation english translation target eta ženŝina ne smogla razžeč svoj ogon this woman did not manage to start her fire moses ta ženŝina ne smogla razžeč togoi this woman did not manage to start her fire seq2seq eta ženŝina ne mogla i ogon ogon this woman was not able and fire fire mbart ta ženŝina ne smogla razžeč ogon that woman did not manage to start the fire rugpt-3 ženŝinoj i ne smog razžeč ogon woman and did not manage to start the fire mbart+ ta ženŝina ne smogla razžeč ogon that woman did not manage to start the fire