A rabic Train at NADI 2024 shared task: LLM s’ Ability to Translate A rabic Dialects into M odern S tandard A rabic (2024)

Abstract: Navigating the intricacies of machine translation (MT) involves tackling the nuanced disparities between Arabic dialects and Modern Standard Arabic (MSA), presenting a formidable obstacle. In this study, we delve into Subtask 3 of the NADI shared task (CITATION), focusing on the translation of sentences from four distinct Arabic dialects into MSA. Our investigation explores the efficacy of various models, including Jais, NLLB, GPT-3.5, and GPT-4, in this dialect-to-MSA translation endeavor. Our findings reveal that Jais surpasses all other models, boasting an average BLEU score of 19.48 in the combination of zero- and few-shot setting, whereas NLLB exhibits the least favorable performance, garnering a BLEU score of 8.77.

## introduction the diverse array of arabic dialects poses a significant challenge for machine translation (mt) systems aiming to connect the different spoken varieties with modern standard arabic, the standardized written form of the language. in recent years, there has been a notable surge of interest in developing effective mt systems capable of accurately translating arabic dialects into msa. such advancements not only foster smoother communication across diverse arabic-speaking communities but also expand access to a wide array of arabic content, enriching cultural exchange and understanding. in our endeavor to confront this challenge and foster progress in arabic dialect mt, we embarked on an exploration within the framework of the nadi shared task. this task specifically focuses on the sentence-level translation of four major arabic dialects -egyptian, emirati, jordanian, and palestinian -into msa. our primary objective lies in evaluating and comparing the performance of three distinct ap-proaches, gpt, jais, and nllb, in the task of translating arabic dialects into msa at the sentence level. in addition, google translate assesses the performance of translating dialectal text into english and french, and subsequently translating it back into arabic (msa). to facilitate this evaluation, we meticulously curated our own datasets for model refinement. our approach involved google translate translations to generate parallel datasets along with the gold msa translations. this dataset shifts the translation task to a translation-correction task. subsequently, all models underwent a rigorous evaluation using the test data set, which includes samples drawn from the four specified dialects. our observations yielded noteworthy insights, with the performance analysis revealing that jais exhibited superior translation proficiency compared to gpt, nllb, and google translate as evidenced by higher bleu scores. these findings underscore the importance of continued exploration and innovation in the realm of arabic dialect mt, as we strive to develop robust systems that not only bridge linguistic divides, but also foster cross-cultural exchange and understanding. through collaborative efforts and thoughtful refinement, we aim to chart a path towards more inclusive and accessible language technologies for arabic speakers worldwide ## related work the landscape of arabic dialect machine translation (mt) has undergone significant evolution, transitioning from earlier rule-based and statistical methods to more sophisticated neural network models. @xcite laid the groundwork by exploring these traditional approaches together. however, with the advent of deep learning, neural network-based methods have taken center stage. recent studies have delved into the capabilities of large language models (llms) in translating arabic dialects. @xcite conducted an extensive investigation into the translation proficiency of instruction-tuned llms across various arabic varieties. their research illuminated the challenges faced by llms like bard and chat-gpt when dealing with dialects lacking substantial public datasets. nevertheless, these models demonstrated remarkable performance, surpassing existing commercial systems and affirming their effectiveness as dialect translators. conversely, farhan et al. (2020) pioneered unsupervised dialectal neural machine translation (nmt), presenting two systems for this novel challenge. the dialectal to standard language translation (d2slt) system leverages dialectal similarities via unique strategies, achieving promising results. notably, the highest bleu score in the unsupervised setting reached 32.14, at bar with the best score obtained in the supervised setting, which stands at 48.25. many proposed methods depend on accurately identifying the dialect before translation, posing several challenges in the process. @xcite address the limitations of current automatic arabic dialect identification (adi) systems in distinguishing between arabic micro-dialects. they critique the prevailing approach of treating adi as a single-label classification problem, arguing that it contributes to system failures. through manual error analysis conducted by seven native arabic speakers, approximately 66% of identified errors were found to be inaccuracies. consequently, the paper proposes reframing adi as a multi-label classification task and offers recommendations for enhancing future adi datasets. this paper focuses on exploring the capabilities of the most recent models, such as jais, gpt-3.5, and gpt-4, for dialect to msa translation. ## results for our results' analysis, translations were assessed using two main criteria: (i) the performance of mt systems on each dialect separately, and (ii) the average score across all four dialects. this dual evaluation strategy enables a thorough evaluation of system performance, taking into account both dialect-specific intricacies and overall translation quality. as we can see in ## discussion in machine translation (mt) across arabic dialects and modern standard arabic (msa), our study within the nadi shared task's subtask 3 illuminates the challenges and advancements in this domain. the final results for the test set ( ## conclusion subtask 3 of the nadi shared task is a crucial avenue for advancing arabic dialect machine translation, encouraging exploration and comparison of diverse translation techniques. by fostering the development of novel datasets and evaluating system efficacy across varied dialects, we've unearthed robust and universally applicable machine translation approaches, poised to enhance communication for arabic speakers worldwide. our findings underscore the profound influence of training data on model proficiency, elucidating the superior performance of models such as jais in contrast to those not primarily tailored for arabic, such as gpt and nllb. as an additional achievement of our project, we took second place among all participants in the competition.