ANLP - RG at NADI 2023 shared task: Machine Translation of A rabic Dialects: A Comparative Study of Transformer Models (2023)

Abstract: In this paper, we present our findings within the context of the NADI-2023 Shared Task (Subtask 2). Our task involves developing a translation model from the Palestinian, Jordanian, Emirati, and Egyptian dialects to Modern Standard Arabic (MSA) using the MADAR parallel corpus, even though it lacks a parallel subset for the Emirati dialect. To address this challenge, we conducted a comparative analysis, evaluating the fine-tuning results of various transformer models using the MADAR corpus as a learning resource. Additionally, we assessed the effectiveness of existing translation tools in achieving our translation objectives. The best model achieved a BLEU score of 11.14% on the dev set and 10.02 on the test set.

## introduction arabic dialects (ad) represent a diverse range of informal languages spoken throughout arab countries. the rise of social media has greatly amplified the growth of these dialects, which have become an integral part of everyday communication. platforms such as twitter, facebook, and instagram often feature user-generated content written in these dialects. arabic dialects can indeed vary significantly from one region to another, and these vast regional differences make them challenging to understand and interpret. this linguistic diversity can be so pronounced that even within a single country, identical words might bear different meanings. as a result, due to the variation among these arabic dialects (ads), it becomes exceedingly challenging to create tools capable of accurately processing arabic social media content. it can also be difficult to employ standard tools designed for modern standard arabic (msa), which serves as the mother language for these dialects. one solution to overcome this problem involves leveraging the richness of the msa language by translating the dialect variants into it. currently, there is a significant amount of work focused on translating dialects to msa. however, most approaches treat each dialect separately, as seen in studies like @xcite for tunisian and al-ibrahim and duwairi (2020) for the jordan dialect. yet, it's important to acknowledge that these dialectal variations coexist in social networks. therefore, it is important to develop models capable of handling the processing of all these dialects collectively. this work fits into this context by involving the development of a machine translation (mt) model to translate a subset of arabic dialects, namely palestinian and levantine, into modern standard arabic (msa). as part of the competition offered by the nadi shared task, the challenge involves the development of a translation model for four dialects based solely on the madar corpus. it's worth noting that madar lacks a parallel corpus for the emirati dialect. in this paper, we outline the experiments to build a dialect translation model. specifically, we have compared the results of two methods: the first involves fine-tuning the arat5 transformer model @xcite utilizing various corpus configurations from madar, while the second entails refining existing tools and employing the back-translation method. the rest of this paper is structured as follows: section 2 outlines related works. section 3 describes the dataset used. the fine-tuning of arat5 models is presented in section 4. we assess the benefits of leveraging tools to improve the translation process in section 5. in section 6, we discuss the results obtained. finally, section 7 provides a conclusive summary. ## related works in the field of neural machine translation for arabic dialects, the primary focus has been on translating these dialects into modern standard arabic (msa). however, most of these works typically concentrate on a single dialect, resulting in a lack of models that address the full spectrum of dialects. for instance, al-ibrahim and duwairi (2020) conducted a study that focused on translating the jordanian arabic dialect into msa using deep learning techniques and implemented an rnn encoder-decoder model. however, the size of the corpus limited their progress. similarly, @xcite tackled the challenge of translating levantine dialects, including jordanian, syrian, and palestinian, into msa. they worked with a relatively small dataset of about 20,000 parallel sentences from the ad applications and resources (madar) and parallel arabic dialect corpus (padic) corpora. their approach introduced an rnn-based multitask learning model in which the decoder was shared across language pairs, with each source language having its own encoder. a transductive transfer learning approach, introduced by @xcite , emerged in the context of low-resource neural machine translation for the algerian arabic dialect. this approach employed fine-tuned transfer learning to transfer knowledge from a parent model to a child model. the evaluation was carried out using the madar and padic corpus. this study applied the transductive transfer learning strategy with two nmt models: seq2seq and attentional-seq2seq. moreover, nagoudi et al. (2022) introduced turjuman, a versatile neural toolbox that can translate 20 languages into msa. the turju-man toolbox uses the power of the arat5 model, renowned for its ability to decode arabic. notably, turjuman allowed for flexibility in decoding methods, facilitating the creation of paraphrases for msa translations. the tool was trained to use semantic similarity to collect publicly available parallel data samples to ensure data quality. this initiative resulted in the development and launch of araopus-20, which establishes a new benchmark for machine translation. it encompasses a benchmark dataset (araopus-20) and the translation toolkit (turjuman). another contribution comes from @xcite , who introduced a hybrid approach to building a translation model for the tunisian dialect. they proposed different augmentation methods to create a large corpus. using this corpus, the authors tested different nmt models. the best model was obtained using joeynmt, achieving a bleu score of 69.22 ## nadi-202shared task subtask 2: datasets in subtask 2, we had access to three primary datasets: training (train), development (dev), and testing (test), as outlined in ## fine-tuning arat5 models in light of the impressive performance showcased by the transformer architecture in neural machine translation (nmt) of arabic dialects, as highlighted by @xcite and nagoudi et al. ## leveraging existing tools to elevate the bleu scores in our translation task, we pursued enhancements through the utilization of existing tools. our approach unfolded in two key steps: firstly, we showcased the efficacy of these tools in translating dialects into modern standard arabic (msa). secondly, taking advantage of the ## discussion the results we obtained exhibit a range of variations, encompassing both positive and negative outcomes. our performance curve for the implemented strategies demonstrates fluctuations, highlighting the complexity of the translation task. it's important to note that the table of bleu scores does not include certain results, such as those for arat5-tweet-base, which received a nadi bleu score of 0 and an overall bleu score of 0.28. in contrast, the highest bleu score of 11.14 on the dev set was achieved through the fine-tuning of arat5v2-base-1024. tis con moreover, upon analyzing the use of deepl translate and dl-translate 0.3.0, we observed that the models from the deepl translate library outperformed those from the dl-translate library. these models exhibited the potential to contribute to enhancing our corpus, resulting in higher msa scores. in contrast, the results obtained from the dl-translate library were significantly lower. moreover, our experiments with data augmentation yielded limited benefits, potentially due to the extensive diversity of arabic written forms. ## conclusion our work constitutes an integral contribution to the nadi2023 shared task, which centers around the machine translation of arabic dialects (ad) into modern standard arabic (msa) using the madar corpus. throughout our research, we explored a multitude of methods and strategies aimed at tackling this complex and challenging task. our efforts yielded two notable strengths. firstly, fine-tuning models, with a particular focus on the "arat5v2-base-1024" model, emerged as an effective approach for enhancing translation quality. additionally, we achieved commendable results by leveraging existing translation tools, especially the google translator model, coupled with backtranslation methods. these outcomes underscore the relevance and practicality of these approaches for translating arabic dialects. in fact, these results open up the possibility of utilizing these methods to automate the parallel corpus construction process. furthermore, we are dedicated to furthering our research efforts by delving into additional fine-tuning techniques for transformer models.