The SADID Evaluation Datasets for Low-Resource Spoken Language Machine Translation of A rabic Dialects (2020)

Abstract: Low-resource Machine Translation recently gained a lot of popularity, and for certain languages, it has made great strides. However, it is still difficult to track progress in other languages for which there is no publicly available evaluation data. In this paper, we introduce benchmark datasets for Arabic and its dialects. We describe our design process and motivations and analyze the datasets to understand their resulting properties. Numerous successful attempts use large monolingual corpora to augment low-resource pairs. We try to approach augmentation differently and investigate whether it is possible to improve MT models without any external sources of data. We accomplish this by bootstrapping existing parallel sentences and complement this with multilingual training to achieve strong baselines.

## introduction machine translation (mt) models have achieved state-of-the-art results on several high-resource language pairs. this success is mostly due to advances in modeling and the availability of clean parallel corpora. yet, advances in mt on low-resource language pairs are still lagging behind mainly due to the scarcity of parallel training data. augmenting with back-translated large monolingual corpora @xcite can partly offset the effects of small amounts of training data, but it is not applicable for pairs where the target language doesn't have large monolingual corpora. on the other hand, freely available benchmark datasets across different tasks have historically provided reference points for researchers to drive their fields forward. notable examples include the general language understanding evaluation (glue) @xcite for nlu and the conference on machine translation (wmt) datasets for mt. still, benchmark datasets for most low-resource language pairs remain a much-needed resource. arabic dialects, like most low-resource languages, lack freely available benchmark datasets that can be used to evaluate models. this makes previous research results difficult to track and reproduce. in this paper, we introduce benchmark datasets for evaluation on mt tasks between arabic dialects, modern standard arabic (msa) and english. we describe the design considerations and data collection guidelines we adopt. we provide an analysis and an empirical evaluation of state-of-the-art mt models on our benchmark datasets. we explore optimal unsupervised segmentation parameters and introduce a novel data augmentation method which we call bootstrapping. the goal of the project is to provide datasets that are reliable public evaluation benchmarks to track progress in the translation quality across different dialects. we put a specific emphasis on creating datasets that are not only able to assess the performance of mt systems but also test their robustness on two levels: the domain, and the dialectal diversity. ## related work machine translation resources for arabic are mostly focused on msa. nonetheless, there are a number of efforts dedicated to dialects. the bolt (broad operational language translation) 1 program this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . on the modeling side, several efforts to improve mt systems have been proved to work well for the low-resource task. sennrich et al. (2016a) made use of monolingual data by augmenting the parallel text with back-translated sentences. lample et al. (2018) leveraged back-translation and the denoising effect of a language model in order to generate synthetic parallel data while having access to monolingual data only. additionally, zoph et al. (2016) made use of transfer learning by training on a related high-resource pair and fine-tuning on the low-resource pair. in a similar approach, liu et al. (2020) demonstrated that multilingual denoising pretraining produces significant performance gains in low-resource tasks. fadaee et al. (2017) proposed an approach that augments the training data by generating new sentence pairs with synthetically created contexts targeting low-frequency words. ## arabic and its dialects arabic is an example of the linguistic phenomenon diglossia @xcite , where its written format, modern standard arabic (msa), differs greatly from the regional spoken dialects. mutual intelligibility between dialects is limited. for example, most arabic speakers understand egyptian due to the popularity of egyptian movies and television shows across the arab world. though, the tunisian dialect is quite difficult to understand for an egyptian speaker. at a high level, @xcite divides arabic dialects into 5 regional clusters: maghrebi arabic (mauritania, morocco, algeria, tunisia, and libya), egyptian arabic (egypt and sudan), levantine arabic (lebanon, syria, jordan, and palestine), gulf arabic (qatar, kuwait, saudi arabia, united arab emirates, iraq, and bahrain) and yemeni arabic. however, dialects differ from country to country and even from city to city. msa refers to the formal arabic that developed in the 19th century from classical arabic, the language of the quran and the literature starting from the 7th century. differently, each arabic dialect is the product of the pre-arabization language of the corresponding geographical location, and other historical factors like colonization. for example, arabic dialects in north africa use a number of french loanwords and still hold some words from the native languages of the amazigh aboriginals. msa is the only variety that is standardized, taught in schools, and used in the media and in official documents. the dialects are not taught in schools, and they are mainly used for day-to-day conversations. moreover, until the advent of social media, they have remained rather absent from written formats. this, in addition to the msa's prevalence in written form, explains why almost all arabic datasets have predominantly msa content. unlike msa, dialects don't have an established orthography. however, it is possible to write arabic dialect text by using the spelling rules of msa, which are mostly phonetic. in essence, arabic dialects not only introduce a low-resource challenge, but they also present the challenge of a spoken language that has phonological, morphological, syntactic, lexical, and orthographical variability with every speaker even within the same geographical location. these factors need to be accounted for in order to create a reliable evaluation benchmark for arabic dialects. ## dataset creation arabic dialects are spoken languages and they are mainly used for day-to-day communication. when it comes to less ordinary topics, people tend to use loanwords from another language (e.g. msa, english, etc.) or switch to another language. we wanted the domain of our datasets to be an accurate representation of the topics people discuss when speaking their dialects. therefore, we conducted a survey where people use their dialects to discuss diverse topics touching upon daily life, politics to culture. topics, where dialects are typically not used, mainly relate to highly technical concepts. we summarize our design motivations in (i) accurately reflecting the topics that people discuss when speaking their dialects, and (ii) capturing the dialectal diversity within the same dialect. we create our dataset in three stages: selecting and curating source passages, translating the resulting passages, and performing quality checks. ## resulting dataset we obtain a 4-way dataset between english, msa, egyptian, and levantine that can be used in several different mt and dialect identification experiments. for each language, the dataset sizes for the dev, devtest, and test sets are 2,997, 2,997, and 2,994 sentences respectively. to understand the properties of our datasets and quantify the similarities between the languages, we compute the jaccard similarity coefficient between pairs of the three arabic languages. we see that there's a considerable 0.33 similarity between the two dialects. we find that msa is slightly closer to levantine (0.30) than it is to egyptian (0.29), and that even an intersection over the three languages still shows high lexical similarity (0.18). this hints that a multilingual setting could be helpful. further statistics about the different splits are included in tables 5 and 6 in appendix c. ## experiments 6.1 data to the best of our knowledge, the only data publicly available for egyptian (egy) and levantine (lev) is distributed through the linguistic data consortium foot_0 . for egyptian, we use data from catalogs ldc2012t09 @xcite , ldc2019t01, ldc2019t18, and ldc2020t05. for levantine, we use ldc2012t09. these catalogs are part of the bolt project and they've been translated from dialectal text retrieved from forums, transcribed phone conversations, sms, and chats. for our msa and multilingual experiments, we also use a selection of parallel msa data from news and web blogs translated as part of the gale (global autonomous language exploitation) program @xcite . we learn a joint source and target byte-pair-encoding @xcite which will be discussed more in detail in section 6.3. our total data amounts to 402k sentences and 3.80m tokens for egyptian, 138k sentences and 1.27m tokens for levantine, and 1.49m sentences and 24.45m tokens for msa. ## results in this section, we analyze our results from