Alson at NADI 2024 shared task: Alson - A fine-tuned model for A rabic Dialect Translation (2024)

Abstract: DA-MSA Machine Translation is a recentchallenge due to the multitude of Arabic dialects and their variations. In this paper, we present our results within the context of Subtask 3 of the NADI-2024 Shared Task(Abdul-Mageed et al., 2024) that is DA-MSA Machine Translation . We utilized the DIALECTS008MSA MADAR corpus (Bouamor et al., 2018),the Emi-NADI corpus for the Emirati dialect (Khered et al., 2023), and we augmented thePalestinian and Jordanian datasets based onNADI 2021. Our approach involves develop013ing sentence-level machine translations fromPalestinian, Jordanian, Emirati, and Egyptiandialects to Modern Standard Arabic (MSA).To016 address this challenge, we fine-tuned models such as (Nagoudi et al., 2022)AraT5v2-msa-small, AraT5v2-msa-base, and (Elmadanyet al., 2023)AraT5v2-base-1024 to comparetheir performance. Among these, the AraT5v2-base-1024 model achieved the best accuracy, with a BLEU score of 0.1650 on the develop023ment set and 0.1746 on the test set.

## introduction the arabic language includes two main variants: modern standard arabic (msa), which is a formal language used in newspapers, news broadcasts, and literature, and dialects arabic (da), which contains a diverse array of informal languages spoken across arab countries, each distinguished by its own unique characteristics @xcite . da-msa machine translation is a relatively recent development in machine translation (mt), largely due to the spread of social media and the widespread use of dialects in daily interactions and online communication. given the multitude of arabic dialects and their variations. creating accurate tools to process arabic social media content presents significant challenges @xcite . therefore, there is a need to unify these dialects under the umbrella of msa, which serves as a standard form of arabic. da-msa translation not only facilitates communication between speakers of different arabic dialects but also enhances the comprehension of written or spoken content for those who primarily use standard arabic. despite its importance, the availability of datasets for da-msa translation is limited compared to english datasets. therefore, translation in english has preceded arabic. in our research, we focus on translating four arabic dialects (egyptian, emirati, jordanian, and palestinian) into msa using the madar parallel dataset and emi-nadi for emirati dialect. we augmented this dataset by extracting data from the nadi2021 dataset for jordanian and palestinian dialects. additionally, we utilized chatgpt to translate these dialects into msa, demonstrating the capabilities of large language models. throughout our experiments, we fine-tuned model's such as arat5v2-msa-small, arat5v2-msa-base arat5v2base-1024 to achieve accurate msa translations. this paper is organized as follows: in section 2, we review related work. in section 3, we present our methodology and dataset. in section 4, we evaluate the models accuracy and present the results. in section 5, we provide a discussion of the results. finally, we conclude with an analysis of errors observed during the experimentation. ## related work some researchers have concentrated on translating a single dialect into modern standard arabic. for instance, the architecture of an rnn sequence-to-sequence learning model (encoder-decoder model) was adopted in @xcite to translate the jordanian dialect to msa. the researchers trained this model to accurately learn the translation probability of jordanian words/phrases to their corresponding standard counterparts, manually collecting the dataset for training. in another study, the focus was on the tunisian dialect (sghaier and zrigui, 2020). the objective was to develop a system that translates text written in tunisian dialect (td) into modern standard arabic (msa) using a rule-based approach. this study @xcite ) also focused on one dialect translating egyptian dialects to modern standard arabic (msa) by combining supervised and unsupervised learning techniques in neural machine translation to improve accuracy and reliability. on the other hand, efforts have been directed towards translating multiple dialects. @xcite focused on machine translation of arabic dialects to modern standard arabic (msa) using the dialect-msa madar corpus. the researchers employed pre-training of the arat5 transformer model and compared its performance with existing transformer models. they also utilized the back-translation method, which involves using english as an intermediary language between dialectal arabic and msa, leading to improved translation quality. they achieved a bleu score of 11.14 % on the dev set and 10.02 % on the test set. in another study, @xcite the emphasis was on machine translation of text from four arabic dialects to modern standard arabic. the researchers fine-tuned three types of t5 models, namely, arat5, mt5, and mt0. additionally, they developed a new dataset, emi-nadi, which contains msa translations of sentences written in emirati. in our work, we emulate the second approach and focus on translating multiple arabic dialects into modern standard arabic. ## results all models for subtask 3 were evaluated using the bilingual evaluation understudy (bleu), a metric for automatically machine translated text @xcite . we obtained two sets of results: one set for the 4 dialects + jor-pla-nadi dataset, and another set for the madar + jor-pla-nadi dataset. the results for the 4 dialects + jor-pla-nadi dataset outperformed those for the madar + jor-pla-nadi dataset in terms of accuracy. we utilized three variants from the arat5v2 series: arat5v2-msa-small, arat5v2msa-base, and arat5v2-base-1024. our initial experiments were conducted using arat5v2-msasmall, which produced low accuracy, with the highest achieved accuracy being 0.024 at epoch 30. subsequently, we employed arat5v2-msa-base, which although better and more complex than the first model, still resulted in low accuracy, albeit slightly improved with the highest achieved accuracy of 0.056 at epoch 10. finally, we proceeded with arat5v2-base-1024 as the last variant, which proved to be the most effective among them. the results were based on experiments conducted on two datasets with different epochs. we observed that in all experiments, the 4 dialects + jor-pla-nadi dataset consistently outperformed the madar + jor-pla-nadi dataset across randomly different epochs. this comparison was conducted to determine which dataset yielded better performance, as illustrated in figure 2 . and the highest result we achieved on the development dataset, trained on the 4 dialects + jor-pla-nadi dataset, was 0.1650. additionally, the highest result we achieved on the test dataset, also trained on the 4 dialects + jor-pla-nadi dataset, was 0.1746. these results were obtained with a learning rate of 7e-5, a batch size of 16, and an epoch of 2. see ## discussion in our study, we observed that the arat5v2-base-1024 model, which was trained on four dialects + jor-pla-nadi, outperforms other models. this is due to its training on a large and diverse arabic dataset. this suggests that pre-trained models on extensive and diverse arabic language data achieve better performance in translating dialects to modern standard arabic (msa). we also noted that the dataset with the four dialects + jor-pla-nadi yields better results than the madar + jor-pla-nadi dataset because focusing on a specific number of dialects helps the model learn more effectively. conversely, training on a large number of dialects may weaken the model's predictive ability and overall performance. additionally, the similarity between dialects and the variation in word meanings across dialects may additionally compromise the model's effectiveness. on adjusting the model parameters, such as batch size and max target input, we experimented with values but noticed only marginal performance differences. similarly, varying the learning rate yielded high accuracy with values of 5e-5 and 7e-5. increasing the epoch count to 20 or 15 did not significantly impact accuracy but consumed more time and resources. in fact, it sometimes even resulted in a decrease in accuracy. furthermore, our model achieved low bleu scores when trained on arat5v2-msa-small and arat5v2-msa-base models. at the level of each dialect separately, we noticed that the egyptian (egy) dialect, despite having a high number of rows in the training data, did not achieve highest accuracy. this could be due to the influence of modern standard arabic (msa). on the other hand, the emirati (emi) dialect showed lowest accuracy, possibly due to foreign dialects. the jordanian (jor) and palestinian (pal) dialects achieved high accuracy, with minimal differences between them, likely because of their high similarity to each other and to msa. this similarity enhances the model's performance in translating these dialects to msa. ## error analysis in our research, we conducted a comprehensive error analysis of our language model, which was specifically trained to translate four distinct arabic dialects egyptian, emirati, jordanian, and palestinian into msa. following our error analysis, we observed that the model faces considerable challenges with dialect-specific words and culturally nuanced expressions, particularly in the egyptian and emirati dialects. for instance, in the egyptian dialect, the letter is often pronounced as and in the emirati dialect, is frequently altered to . these phonetic changes are unique to each dialect and often lead to mistranslations or the model failing to translate the words. this difficulty arises because these words far significantly from their msa counterparts. consequently, our findings suggest that the closer a word or phrase is to msa, the more effectively the model can translate it. conversely, as the words become more distinct from msa, the model's ability to translate them declines. furthermore, the model sometimes fails to translate verbs into their correct tenses or accurately translate repetitive structures in dialects. this indicates a need for the model to be more finely attuned to the subtleties and nuances of each dialect. fine-tuning with specialized datasets that capture these dialect-specific characteristics is essential for enhancing the model's translation accuracy. improving the model's ability to manage these phonetic and structural variations is crucial for better performance. as illustrated in figure 3 , the current model output shows several instances of poor translation for each dialect. 7 conclusion, future work, and