The Effect of Domain and Diacritics in Y oruba– E nglish Neural Machine Translation (2021)

Abstract: Massively multilingual machine translation (MT) has shown impressive capabilities and including zero and few-shot translation between low-resource language pairs. However and these models are often evaluated on high-resource languages with the assumption that they generalize to low-resource ones. The difficulty of evaluating MT models on low-resource pairs is often due to lack of standardized evaluation datasets. In this paper and we present MENYO-20k and the first multi-domain parallel corpus with a especially curated orthography for Yoruba–English with standardized train-test splits for benchmarking. We provide several neural MT benchmarks and compare them to the performance of popular pre-trained (massively multilingual) MT models both for the heterogeneous test set and its subdomains. Since these pre-trained models use huge amounts of data with uncertain quality and we also analyze the effect of diacritics and a major characteristic of Yoruba and in the training data. We investigate how and when this training condition affects the final quality of a translation and its understandability.

Our models outperform massively multilingual models such as Google ( +8.7 BLEU) and Facebook M2M ( +9.1 ) when translating to Yoruba and setting a high quality benchmark for future research.

## introduction neural machine translation (nmt) achieves high quality performance when large amounts of parallel sentences are available @xcite . large and freely-available parallel corpora do exist for a small number of high-resource pairs and domains. however, for low-resource languages such as yorùbá (yo), one can only find few thousands of parallel sentences online foot_0 . in the best-case scenario, i.e. some amount of parallel data exists, one can use the biblethe bible is the most available resource for low-resource languages @xcite -and jw300 @xcite . notice that both corpora belong to the religious domain and they do not generalize well to popular domains such as news and daily conversations. in this paper, we address this problem for the yorùbá-english (yo-en) language pair by creating a multi-domain parallel dataset, menyo-20k, which we make publicly available foot_1 with cc by-nc 4.0 licence. it is a heterogeneous dataset that comprises texts obtained from news articles, ted talks, movie and radio transcripts, science and technology texts, and other short articles curated from the web and translated by professional translators. based on the resulting train-development-test split, we provide a benchmark for the yo-en translation task for future research on this language pair. this allows us to properly evaluate the generalization of mt models trained on jw300 and the bible on new domains. we further explore transfer learning approaches that can make use of a few thousand sentence pairs for domain adaptation. finally, we analyze the effect of yorùbá diacritics on the translation quality of pre-trained mt models, discussing in details how this affects the understanding of the translated text especially in the en-yo direction. we show the benefit of automatic diacritic restoration in addressing the problem of noisy diacritics. ## the yor ùbá language the yorùbá language is the third most spoken language in africa, and it is native to southwestern nigeria and the republic of benin. it is one of the national languages in nigeria, benin and togo, and spoken across the west african regions. the language belongs to the niger-congo family, and it is spoken by over 40 million native speakers @xcite . yorùbá has 25 letters without the latin characters c, q, v, x and z, and with additional characters e . , gb, s . , o . . yorùbá is a tonal language with three tones: low, middle and high. these tones are represented by the grave (e.g. "à "), optional macron (e.g. "ā") and acute (e.g. "á") accents respectively. these tones are applied on vowels and syllabic nasals, but the mid tone is usually ignored in writings. the tone information and underdots are important for the correct pronunciation of words. often, articles written online, including news articles such as bbc foot_2 ignore diacritics. ignoring diacritics makes it difficult to identify or pronounce words except when they are embedded in context. for example, èdè (language), edé (crayfish), e . de . (a town in nigeria), è . de . (trap) and è . dè . (balcony) will be mapped to ede without diacritics. machine translation might be able to learn to disambiguate the meaning of words and generate correct english even with un-diacriticized yorùbá. however, one cannot generate correct yorùbá if the training data is un-diacriticized. one of the purposes of our work is to build a corpus with correct and complete diacritization in several domains. ## menyo-20k the dataset collection was motivated by the inability of machine translation models trained on jw300 to generalize to new domains @xcite for yorùbá with surprisingly high bleu scores, the evaluation was done on very few examples from the covid-19 and ted talks domains with 39 and 80 sentences respectively. inspired by the flores dataset for nepali and sinhala @xcite , we create a high quality test set for yorùbá-english with few thousands of sentences in different domains to check the quality of industry mt models, pre-trained mt models, and mt models based on popular corpora such as jw300 and the bible. ## related work in order to make mt available for a broader range of linguistic communities, recent years have seen an effort in creating new parallel corpora for low-resource language pairs. recently, @xcite provided novel supervised, semi-supervised and unsupervised benchmarks for indo-aryan languages {sinhala,nepali}-english on an evaluation set of professionally translated sentences sourced from the sinhala, nepali and english wikipedias. novel parallel corpora focusing on african languages cover south african languages ({afrikaans, isizulu, northern sotho, setswana, xitsonga}-english) @xcite with mt benchmarks evaluated in @xcite , as well as multidomain (news, wikipedia, twitter, conversational) amharic-english @xcite and multidomain (government, wikipedia, news etc.) igbo-english @xcite . further, the lorelei project @xcite has created parallel corpora for a variety of lowresource language pairs, including a number of niger-congo languages such as {isizulu, twi, wolof, yorùbá }-english. however, these are not open-access. on the contrary, masakhane @xcite ) is an ongoing participatory project focusing on creating new freely-available parallel corpora and mt benchmark models for a large variety of african languages. while creating parallel resources for low-resource language pairs is one approach to increase the number of linguistic communities covered by mt, this does not scale to the sheer amount of possible language combinations. another research line focuses on low-resource mt from the modeling side, developing methods which allow a mt system to learn the translation task with smaller amounts of supervisory signals. this is done by exploiting the weaker supervisory signals in larger amounts of available monolingual data, e.g. by identifying additional parallel data in monolingual corpora @xcite @xcite , comparable corpora @xcite , or by including auto-encoding @xcite or language modeling tasks @xcite during training. low-resource language pairs can benefit from high-resource languages through transfer learning @xcite , e.g. in a zero-shot setting @xcite , by using pre-trained language models @xcite , or finding an optimal path of pivoting through related languages @xcite . by adapting the model hyperparameters to the low-resource scenario, @xcite were able to achieve impressive improvements over a standard nmt system. ## conclusion we present menyo-20k, a novel en-yo multi-domain parallel corpus for machine translation and domain adaptation. by defining a standardized train-development-test split of this corpus, we provide several nmt benchmarks for future research on the en-yo mt task. further, we analyze the domain differences on the menyo-20k corpus and the translation performance of nmt models trained on religion corpora, such as jw300 and bible, across the different domains. we show that, despite consisting of only 10k parallel sentences, adding the menyo-20k corpus train split to jw300 and bible largely improves the translation performance over all domains. further, we train a variety of supervised, semi-supervised and fine-tuned mt benchmarks on available en-yo corpora, creating a high quality baseline that outperforms current massively multilingual models, e.g. google mnmt by bleu +18.8 (en-yo). this shows the positive impact of using smaller amounts of high-quality data (e.g. c4+transfer, bleu 12.4) that takes into account language-specific characteristics, i.e. diacritics, over massive amounts of noisy data (facebook m2m-100, bleu 3.3). apart from having low bleu scores, our human evaluation reveals that models trained on low-quality diacritics (google mnmt) suffer especially in fluency, while still being intelligible to the reader. while correctly diacritized data is vital for translating en-yo, it only has an impact on the quality of yo-en translation quality when there is a domain mismatch between training and testing data.