Benchmarking Dialectal A rabic- T urkish Machine Translation (2023)

Abstract: Due to the significant influx of Syrian refugees in Turkey in recent years, the Syrian Arabic dialect has become increasingly prevalent in certain regions of Turkey. Developing a machine translation system between Turkish and Syrian Arabic would be crucial in facilitating communication between the Turkish and Syrian communities in these regions, which can have a positive impact on various domains such as politics, trade, and humanitarian aid. Such a system would also contribute positively to the growing Arab-focused tourism industry in Turkey. In this paper, we present the first research effort exploring translation between Syrian Arabic and Turkish. We use a set of 2,000 parallel sentences from the MADAR corpus containing 25 different city dialects from different cities across the Arab world, in addition to Modern Standard Arabic (MSA), English, and French. Additionally, we explore the translation performance into Turkish from other Arabic dialects and compare the results to the performance achieved when translating from Syrian Arabic. We build our MADAR-Turk data set by manually translating the set of 2,000 sentences from the Damascus dialect of Syria to Turkish with the help of two native Arabic speakers from Syria who are also highly fluent in Turkish. We evaluate the quality of the translations and report the results achieved. We make this first-of-a-kind data set publicly available to support research in machine translation between these important but less studied language pairs.

## introduction the rapid advancements in machine translation technology have significantly helped to break down language barriers and facilitate cross-cultural communication using distant languages, including turkish and arabic. given that syria and iraq border turkey to the south, there is cultural overlap and close ties between those nations and turkey, making arabic and turkish two of the most widely spoken languages in the middle east. despite this, there has been no significant research or machine translation effort that specifically addresses the translation of 1 the madar-turk data set is available from http://resources.camel-lab.com/ . efforts to develop neural machine translation between several language pairs including turkish @xcite and arabic @xcite have yielded promising results, improving translation quality and reducing the need for extensive linguistic knowledge. building such systems requires a large amount of data, which currently does not exist for the turkish and arabic language pair -for modern standard arabic (msa), and more so for the dialects. focusing on benchmarking, we present the first research effort exploring translation between syrian arabic and turkish using a set of 2,000 parallel sentences from the madar corpus containing 25 different city dialects from various cities across the arab world, in addition to msa, english, and french. additionally, we explore the translation performance into turkish from other arabic dialects and compare the results to the performance achieved when translating from syrian arabic. we make the data set publicly available. 1 the paper is structured as follows. section 2 presents some related work in turkish and arabic machine translation, and section 3 discusses the linguistic challenges of arabic-turkish translation. section 4 details the madar-turk data set creation process. sections 5 and 6 present our benchmarking results and error analysis, respectively. we conclude and describe our future work in section 7. ## challenges of arabic-turkish translation while ottoman turkish used to be written in arabic script, modern turkish uses the roman script, which adds to the many linguistic differences between turkish and arabic and its dialects in terms of morphology, syntax, and lexicon. ## benchmarking dialect arabic to turkish mt we translated the various sentences from the madar data set into turkish using google translate. 3 to evaluate the quality of the automatic translations, we compared them against the reference translations produced by the translators. we measure the translation quality using bleu @xcite . we use the sacrebleu implementation @xcite for evaluating automatic translations against the reference translations (@xmath0, @xmath1. the results are shown in clearly, a lot more effort has to be done to aid turkish translation from all these different languages, especially from arabic. erated as yeezy instead of turkish yeterli. semantic errors refer to cases where a word is translated with a different meaning than intended. for instance, the damascus arabic word (with ambiguous diacritization as nus∼ 'half' or nas∼ 'text') is incorrectly translated in the context of the phrase ns qnynh 'half bottle' as s ¸is ¸e metni 'bottle text' as opposed to the correct translation yarım s ¸is ¸e 'half bottle'. and morphology errors refer to cases where a word is translated with errors in morphological features. for example, the tunisian arabic verb nhb 'i want' (turkish reference istiyorum) is mistranslated as seviyoruz 'we love' (i.e. plural instead of singular morphology). this is most likely a result of confusion with the msa reading of the arabic word which also means 'we love'. the summary of our results is given in ## conclusion and future work we introduced madar-turk, a set of 2,000 sentences from the madar corpus, translated from the damascus dialect into turkish. to the best of our knowledge, this is a first-of-a-kind human reference set for dialectal arabic-turkish. our study provides the first-ever benchmarking results on translation performance from arabic dialects to turkish. by producing this data set and making it publicly available, we hope to support ongoing efforts to improve translation and language access for individuals who speak arabic dialects in the turkish context. in the future, we plan to continue expanding the human reference set to improve machine translation in the context of this resource-scarce language pair. we also plan to use this data set as part of developing improved methods for machine translation for low-resource language pairs.