A ra B ench: Benchmarking Dialectal A rabic- E nglish Machine Translation (2020)

Abstract: Low-resource machine translation suffers from the scarcity of training data and the unavailability of standard evaluation sets. While a number of research efforts target the former, the unavailability of evaluation benchmarks remain a major hindrance in tracking the progress in low-resource machine translation. In this paper, we introduce AraBench, an evaluation suite for dialectal Arabic to English machine translation. Compared to Modern Standard Arabic, Arabic dialects are challenging due to their spoken nature, non-standard orthography, and a large variation in dialectness. To this end, we pool together already available Dialectal Arabic-English resources and additionally build novel test sets. AraBench offers 4 coarse, 15 fine-grained and 25 city-level dialect categories, belonging to diverse genres, such as media, chat, religion and travel with varying level of dialectness. We report strong baselines using several training settings: fine-tuning, back-translation and data augmentation. The evaluation suite opens a wide range of research frontiers to push efforts in low-resource machine translation, particularly Arabic dialect translation. The evaluation suite and the dialectal system are publicly available for research purposes.

## introduction modern standard arabic (msa) is the lingua franca of the arab world. however, spoken language, aka dialects, are mainly used in daily interactions and social media, whereas msa is prominently used in news, education and print media. dialects differ widely from msa in terms of lexical choice, morphology and syntax. while a lot of work has been done on msa-to-english machine translation (mt), little effort has been put in translating arabic dialects to english. even the existing few attempts in translating arabic dialects to english @xcite are limited to a small number of dialects. moreover, the results of these studies are not comparable because of the lack of standard evaluation sets. training data and evaluation benchmarks are two essential ingredients to achieve a high quality translation system @xcite . creating such resources for dialectal arabic is difficult and poses a unique set of challenges: their spoken nature, non-standard orthography, variation in the level of dialectness and the fuzzy boundaries between various dialects makes it extremely difficult to create a dataset that covers all these diverse aspects. these issues have contributed towards the slow progress in dialectal arabic mt. recently, several works have addressed the problem of limited training data for mt @xcite . however, the unavailability of standard evaluation sets remains a major barrier to track the progress of translation systems. @xcite highlighted the importance of evaluation benchmarks in pushing efforts on low-resource mt and prepared evaluation sets of nepali-english and sinhala-english. in this paper, we aim to further this cause by providing an evaluation benchmark for another low-resource language set, dialectal arabic. to this end, we pool together already available dialectal arabic-english resources and we additionally build novel testsets. we consolidate the collected data into an evaluation suite covering 4 broad, 15 fine-grained and 25 city-level dialect categories, belonging to diverse genres, such as media, chat, religious, travel, and varying dialectness level, thus broadening the scope of research in low-resource mt, particularly, arabic dialect translation. we aim to increase the evaluation sets further in the future. we carried out an extensive evaluation around our test suite by training dialectal systems from scratch and by adapting an industrial-scale msa-to-english system towards dialectal arabic. more concretely, we explored fine-tuning, back-translation, and data augmentation as our training strategies. a few notable findings include: i) fine-tuning an msa-english system with a small amount of dialectal data achieves significantly better results; ii) mixing of available dialectal and msa training data enables zero-shot translation of dialects with no training data; iii) a single general system can be built that translates a large variety of dialects and msa effectively. the contribution of our work are as follows: ## data curation in this section, we describe the datasets that we consolidated to form an evaluation suite. we gather resources from previously published work as well as curate our own evaluation sets. arabic-dialect/english parallel text (apt) @xcite foot_2 provided egyptian-english and levantine-english parallel corpus consisting of ≈3.5 million tokens of arabic dialects -38k egyptian-english and 138k levantine-english sentences. the data was collected from dialectal arabic weblogs and online user groups, and the translation was carried out by arabic annotators using amazon mechanical turk. several research works @xcite @xcite have used this data to build dialectal arabic to english mt systems. however, all the mentioned work used different splits, due to which the results are not comparable. we refer to this data as apt hereafter. multi-dialectal parallel corpus of arabic (mdc) @xcite selected 2k parallel egyptian-english sentences from the corpus built by @xcite and tasked native speakers of palestinian, syrian, jordanian and tunisian to translate these egyptian sentences into their own dialect. they additionally translated 8000 more sentences from egyptian-english data into syrian only. the dataset serves as an interesting resource comparing identical sentences translated into different dialects. @xcite selected 2k english sentences from the btec corpus, originally a japanese/english bank for parallel phrases, and recruited native speakers from 26 arab cities to translate english phrases into their own dialect. additionally for five selected cities: doha, beirut, cairo, tunis, and rabat, they translated 10k more sentences. the madar corpus is unique in terms of the variety of dialects covered and the travel domain. ## data preparation data processing we ran the following preprocessing pipeline: filtered out sentences with length above 100 words, removed diacritization and normalized arabic characters such as replacing different forms of alif and hamzah ( o, i, m) are replaced with a bare alif a. similarly for y to y, and p to h, and converted indian digits to arabic numerals 0-9 @xcite . ## evaluation in this section, we describe experimental setup and present our results using the evaluation suite. ## discussion and analysis translation quality vs. dialectness level we studied the relationship between translation quality and dl of a test set. we hypothesized that a high dl may result in low translation quality when using an msa system ( lexical choice errors although the inclusion of msa during fine-tuning significantly improved the translation quality, it also caused lexical ambiguity in some cases. we found that the system erred when translating a few words that have differing semantic meaning between a dialect and msa. for example, the tunisian word "hzny" ("carry me") and moroccan word "dyny" ("take me") got translated into their msa variants, "shake me" and "my debt" respectively. we conjecture that this happeneds because of two reasons. first, msa has a significant influence during training because of its relatively larger size. second, inadequate context in the input text increases the ambiguity for the system. we found this phenomenon to be frequent in the madar testset where average sentence length is only 7 words. machine translation machine translation of arabic dialects got attention for a short while due to bolt project. subsequent efforts were carried to improve msa-to-english systems by appending dialectto-msa module as pre-processing step @xcite @xcite @xcite or adapting the msa-to-english systems towards in-domain dialectal data @xcite . @xcite studied the use of sentence level dialect identification in optimizing mt system selection in mixed dialectal scenario. more recently @xcite used multi-task learning in neural mt with individual encoders for msa and dialects and a shared decoder. despite the number of efforts in translating arabic dialects to msa, they are limited to a few dialects and the results among various studies are not comparable due to the difference of evaluation sets. in this paper, we provide the first dialectal arabic-english evaluation suite based on a large number of dialects, covering various genre and varying amount of dialectness level. ## conclusion and future directions we presented arabench, an evaluation suite covering 4 coarse and 15 fine-grained, and 25 city-level dialect categories. the evaluation suite is first of its kind that put together a large number of dialects, covering several domains, and with varying level of dialectness. we adapted an industrial scale msa-english system to train very strong baselines based on fine-tuning, back-translation and data augmentation and did a large scale evaluation on arabench. we showed that a single general system can be effectively trained to translate both msa and a large variety of dialects. the evaluation suite enables numerous future research directions in low-resource mt and dialect translation. for example, adapting unsupervised mt methods towards learning to translate related languages and testing the generalization capabilities of existing methods in mt against varying genre and level of dialectness are a few interesting directions to explore.