Multi-Parallel Corpus of N orth L evantine A rabic (2023)

Abstract: Low-resource Machine Translation (MT) is characterized by the scarce availability of training data and/or standardized evaluation benchmarks. In the context of Dialectal Arabic, recent works introduced several evaluation benchmarks covering both Modern Standard Arabic (MSA) and dialects, mapping, however, mostly to a single Indo-European language - English. In this work, we introduce a multi-lingual corpus consisting of 120,600 multi-parallel sentences in English, French, German, Greek, Spanish, and MSA selected from the OpenSubtitles corpus, which were manually translated into the North Levantine Arabic. By conducting a series of training and fine-tuning experiments, we explore how this novel resource can contribute to the research on Arabic MT.

## introduction levantine arabic is considered one of the core units within the arabic dialectal continuum. it can be divided into at least three dialectal regions @xcite but the most notable division within this group lies between south levantine (palestinian) and north levantine (based on the urban speech of mainly beirut and damascus) with clear differences between the two @xcite . at the same time, north levantine arabic (also called syrian or shami) is perceived as a clearly established linguistic unit with a positive evaluation and perception @xcite . in the field of natural language processing, north levantine arabic is, similarly to other arabic dialects, considered a low-resource language. it is mainly used for daily speech, and written resources are very scarce. formal texts are almost exclusively written in modern standard arabic (msa). recently, written north levantine arabic started apc arb eng my passport is there, along with some papers. who's eating my dumplings? wer isst meine klöße? to appear in texts posted to social networks that became a useful resource of monolingual datasets for several dialects of arabic @xcite . parallel datasets are even scarcer. in this paper, we introduce a novel multi-parallel corpus where north levantine arabic is paired with msa and several indo-european languages (english, french, german, greek, and spanish). the corpus contains roughly 1 million words on the english side. by targeting the subset of the multiparallel opensubtitles-v2018 @xcite ) dataset, we ensure that with a single round of translation, we can achieve the desired multi-lingual, multi-parallel mapping between msa, dialectal arabic and several indo-european languages. considering that the opensubtitles dataset consists of lines from movie subtitles 1 , it should well represent the "everyday dialogue" domain, where the arabic dialects are most commonly used. ## related work in their pioneer work, @xcite introduced a parallel levantine-english corpus of 138k sentences suitable for training mt systems. the levantine sentences were extracted from arabic weblogs and online user groups and translated into english. in follow-up work, @xcite translated 2,000 sentences from the egyptian-english corpus introduced by @xcite into several arabic dialects (including north levantine arabic), creating the first multi-parallel corpus of multi-dialectal arabic. the multi-parallel aspects were further explored (e.g., @xcite and the data were compiled into standardized benchmarks (e.g., @xcite @xcite . arab-acquis @xcite matched multi-parallel corpus of 22 european languages with human translations into msa -dialectical aspects were not considered. the exploitation of the opensubtitles corpus in the context of arabic mt was previously explored by nagoudi et al. (2022), who used it to sample training/testing data for translation from four languages (english, french, german, and russian) into msa and alhafni et al. (2022) who sampled english-msa sentence pairs for the extended arabic parallel gender corpus (apgc v2.0). ## data preparation as a first step, we filtered the opensubtitles-v2018 corpus by identifying lines that are available in all of the desired languages (msa, english, french, german, greek, and spanish), obtaining 3,661,627 sentences. subsequently, a number of additional filters (for convenience, we applied filters to the english side) were applied: the goal was not to translate similar sentences like good morning and good morning! or i was born in 1961 and i was born in 1983. language iso 639-3 code #words north levantine arabic apc 738,812 modern standard arabic arb 802,313 english eng 999,193 french fra 956,208 german deu 940,234 greek ell 869,543 spanish spa 920,922 after those filtering steps, we ended up with 120,771 sentences. before the translation, an additional corpus-wise filtering step was applied by removing multi-parallel lines where: english characters appear in the arabic sentence, arabic characters appear in the english sentence, or arabic characters appear in a particular sentence for all of the indo-european languages. the final size of the corpus is equal to 120,600 lines that were manually translated into the north levantine arabic dialect. the translation was performed by native speakers of the dialect through a professional translation company without using any mt or cat tool. considering the lack of official spelling standards for levantine, we did not provide the translators with specific orthographic guidelines @xcite , but rather relayed on their expertise, asking only for internal consistency. first, a sample of 1,000 sentences was translated independently from english and from msa. no difference in translation quality was observed (assessed by authors of the paper -speakers of north levantine arabic). therefore, all the remaining sentences were translated from msa (this direction was less costly). the translation was done in batches of 5,000 sentences, and the quality of the translation was checked after each batch (again by the authors of the paper -speakers of the dialect). in order to quantitatively measure the impact of the source language, we computed the overlap coefficient (oc) @xcite for the samples of 1,000 sentences that were used initially foot_1 . the oc value measures the percentage of lexical overlap between the vocabularies of two languages (dialects). the oc similarity between the msa source translated into apc target equals 35.95, and the one between the (parallel) msa and the target apc when translating from english equals 26.85. to put those numbers into context, the oc value between the 1,000 sentences in msa and syrian that were independently translated from egyptian by @xcite equals 39.85. those results indicate that the variety in the apc output may have been slightly reduced by translating from msa. however, it should be mentioned that we compare disjoint sets of sentences, and there is not enough data to say how this affects the downstream tasks, such as mt. sentence samples (multi-parallel lines) are presented in ## mt experiments in order to demonstrate the validity of the corpus, we conducted a number of mt experiments and evaluations. baselines and metrics we report the performance of two well-established baselines: a multilingual nllb model @xcite , using the facebook/nllb-200-distilled-600m variant (600m parameters) from the transformers @xcite package, and uni-directional models (depending on the language pair, between 76m and 240m parameters) provided by the helsinki-nlp group @xcite . to indicate to what extent msa can be used when the dialectal system is not available, we translate into both arb (e.g., opus arb ) and apc, always using the apc files as reference. we measure the output quality by reporting the surface-level chrf++ foot_2 metric @xcite , and the trainable, estimatorbased comet foot_3 metric @xcite . ## results automatic metrics the base ml system trained from scratch achieves the lowest scores on both testsets. on average, the larger, multi-lingual nllb model achieves better scores than the opus models. translating into arb gives consistently higher scores for sentences from the flores-200 test-set, but lower ones for sentences from madar. we attribute this to the vastly different nature of those test-sets. sentences in flores-200 are long, with the effects of fine-tuning on the corpus that we introduce highlight the difficulties of low-resource mt. on the madar test-set, coming from a similar domain as the resource introduced in this work, significant improvements can be observed when translating into apc -both for opus (26.32→38.50) and nllb (33.04→37.77) models. similar behavior can be observed for the opus model when translating into english (38.53→51.09). however, that is not the case for the nllb model. it is possible that a comparable amount of dialectal arabic (mixed with msa) has already been seen on the source side during training, and more sophisticated fine-tuning schemas are required. on the flores-200 test-set (different domain), minor improvements can be observed for the nllb model (on average, +1.97 chrf when translating into apc ), with inconsistent results for the opus models (37.54→39.24 when translating from deu but 34.77→33.56 when translating from spa). ## conclusions in this work, a novel, multi-parallel corpus of north levantine arabic, based on the opensubtitles-v2018 dataset, is introduced. by fine-tuning wellestablished baseline mt models, we show that the dialectal aspects of language are partially orthogonal to the domain-specific properties -a dialectspecific model fine-tuned on data from a particular domain may perform worse than a more generic model if a domain shift occurs during testing. however, human evaluation confirms that the dialectspecific aspects of the output are still ranked higher and more appreciated by the final users of the mt system.