D ial2 MSA -Verified: A Multi-Dialect A rabic Social Media Dataset for Neural Machine Translation to M odern S tandard A rabic (2025)

Abstract: Social media has become an essential focus for Natural Language Processing (NLP) research due to its widespread use and unique linguistic characteristics. Normalising social media content, especially for morphologically rich languages like Arabic, remains a complex task due to limited parallel corpora. Arabic encompasses Modern Standard Arabic (MSA) and various regional dialects, collectively termed Dialectal Arabic (DA), which complicates NLP efforts due to their informal nature and variability. This paper presents Dial2MSA-Verified, an extension of the Dial2MSA dataset that includes verified translations for Gulf, Egyptian, Levantine, and Maghrebi dialects. We evaluate the performance of Seq2Seq models on this dataset, highlighting the effectiveness of state-of-the-art models in translating local Arabic dialects. We also provide insights through error analysis and outline future directions for enhancing Seq2Seq models and dataset development. The Dial2MSA-Verified dataset is publicly available to support further research.

## introduction the rapid growth in social media users has established it as an area of interest for natural language processing (nlp) research. normalising social media texts' content is transforming informal text into a more standardised form that aligns with established linguistic conventions. this process is a challenging nlp task for morphologically rich languages such as arabic, especially when parallel corpora for arabic social media and their corresponding standard forms are limited @xcite . arabic, a widely spoken global language, exists in two primary forms: modern standard arabic (msa) and various regional dialects, collectively known as dialectal arabic (da). msa, the standardised form of the arabic language, is utilised in formal contexts such as education, media, literature, and official documentation. as a linguistic bridge across the arab world, msa promotes a shared understanding and cultural cohesion among diverse arab communities. in terms of grammar and vocabulary, msa follows strict standardised rules, ensuring consistency in formal communication. conversely, da is the language of daily interaction, prevalent in informal settings and deeply reflective of the cultural and social identities unique to each region and community @xcite . the significant variation between arabic dialects further complicates nlp tasks, as models trained on msa alone may struggle with the language used on social media. arabic users on these platforms tend to use their local informal dialect. a single arabic word may indicate different interpretations based on the context of the sentence, between two dialects or between a dialect and msa @xcite , which shows why it is important to normalise text used in social media. such a text often combines msa, dialects, non-arabic words, and unconventional spelling and may include slang, abbreviations, shortened or compound words, perhaps with grammar or spelling mistakes @xcite . figure 1 demonstrates the issues in arabic social media text and their correct format. the increase of unstructured text from various sources, including social media, has highlighted the need for effective preprocessing and normalisation to enhance data quality and usability. while basic preprocessing methods can still handle some issues in figure 1 , others, such as dialectal variations, syntax mistakes, ambiguity, and polysemy, require advanced techniques. to address these issues and the normalisation task, we adopted a sequence to sequence (seq2seq) technique, specifically using neural machine translation (nmt) architectures. seq2seq models have shown promising results in handling translations across different language pairs, including from and to msa. however, nmt models require large amounts of parallel data (i.e., pairs of sentences in two languages) for effective training. the limited availability of da-msa datasets poses an obstacle @xcite . moreover, in the context of social media, the dial2msa dataset @xcite is currently the only publicly available resource that covers multiple regional dialects, and it has not been fully verified. in this paper, we present dial2msa-verified, which is built upon dial2msa @xcite , a seq2seq dataset from social media that encompasses four dialects: egyptian (egy), maghrebi (mgr), levantine (lev), and gulf (glf). our contributions to this dataset are two-fold: for da-msa, early research used rule-based mt @xcite @xcite , statistical machine translation (smt) @xcite @xcite and hybrid approaches @xcite . later systems adapted nmt by either translating one dialect to msa or multiple dialects to msa. in single-da to msa translation, @xcite employed an rnn seq2seq encoder-decoder model to translate the jordanian dialect into msa. slim et al. (2022) applied a transductive transfer learning (tl) approach for translating the algerian dialect to msa using seq2seq models. faheem et al. (2024) combined supervised and unsupervised nmt methods to enhance the translation from the egy dialect to msa. in multi-da translation, shapiro and duh (2019) conducted training on transformer-based models across different arabic varieties, including egy and lev dialects and msa. their findings indicated that leveraging multi-da datasets can improve the translation quality for other unencountered dialects. additionally, baniata et al. (2021) investigated the translation between multiple dialects and msa by employing a word-piece model to generate sub-word units for input features in the nmt transformer model. recently, three shared-tasks were created for da-msa translation: the fourth and fifth nadi shared-tasks @xcite and osact da-msa mt shared-task @xcite . participants were allowed to use any available dataset and encouraged to create new datasets to train their models. as a result, some teams used a large language model (llm) such as chatgpt from openai to augment the training dataset @xcite . participants experimented with various nmt models, such as fine-tuning transformerbased pre-trained in arabic models. ## evaluation and results in this section, we evaluate our proposed models using the bilingual evaluation understudy (bleu) @xcite and chrf++ @xcite metrics. during training, we validated the models using the bleu metric on the development set, selecting the checkpoint that achieved the highest bleu score. for these optimal checkpoints, we report both bleu and chrf++ scores on the testing set using the sacrebleu implementation @xcite . this implementation supports multireference evaluation for both metrics, providing a comprehensive assessment of model performance. we present results for two configurations: joint regional (j-r) and independent regional (i-r). for the i-r configuration, dialect classification is used to select the appropriate translation model. this classification model was evaluated on the testing set of the dial2msa-verified dataset using accuracy and macro-average f1 metrics. additionally, we compare the results obtained when using a single reference file versus multiple reference files for the dial2msa-verified-test dataset to highlight the impact of reference diversity on evaluation. the overall comparison between the two configurations shows that the j-r configuration outperforms the i-r configuration. this result is due to two reasons. the j-r configuration may benefit from leveraging shared linguistic patterns among similar dialects during training. moreover, the i-r configuration depends on a dialect classification model to choose the appropriate translation model for each input. despite the promising results of the dialect identification model (results in appendix b), it still does not achieve perfect accuracy. furthermore. the use of multiple reference translations significantly enhances the evaluation of the model's performance. a single da sentence can be translated into msa in multiple forms due to the rich nature of arabic morphology and syntax. each translation can preserve the core meaning while using different vocabulary choices and sentence structures. for example, the levantine dialectal phrase (i want to go), which can be translated into msa as , , or , all conveying the same essential meaning. with more reference translations, there is a higher likelihood that the model's output will align with at least one reference, leading to a more accurate assessment. the highest bleu and chrf++ metrics scores are achieved when evaluating the proposed models on all available references. as shown in we provide a comprehensive example table showcasing the original tweets, their gold standard translations, and the corresponding arat5 translations in appendix c. our analysis revealed notable issues in normalising arabic in social media into msa, particularly due to oov tokens, many of which stem from non-arabic origins, as well as unique expressions tied to specific dialects. in the egy dialect, for example, several english loanwords have adapted meanings that differ from the literal sense of arabic. the term , as seen in , implies "too much" or "over-the-top" in english. however, the model interpreted it as "more available" (its arabic literal meaning), resulting in translations like . similarly, (derived from "sketch") was misinterpreted as (your silence) due to morphological resemblance, while (from "shoes," but translated to or "car") posed similar challenges. the j-r-arat5 model, however, showed some accuracy in translating certain local dialect words. for instance, was correctly rendered as additionally, the model showed an ability to handle other arabised english terms frequently seen on social media. words such as were appropriately translated to (followers), and to (facebook). it also successfully translated as (retweet), demonstrating its adaptability to social media language. mgr dialect posed a different set of challenges, particularly due to lexical and conjugation differences from other dialects. for example, , meaning "insulting," was often misinterpreted as something related to work ( ), while (meaning "what is up" in algerian dialect) was incorrectly translated to (name). the term (referring to the french bem exam) also created difficulties, as it appears in arabic script but is inherently non-arabic. lev and egy dialects featured unique dialectal words that the model struggled to translate accurately, even though direct msa equivalents exist. words like (doodles), (yesterday), and (laziness) were challenging for the model, perhaps due to morphological or contextual ambiguities that made it difficult for the model to identify the correct translations. ## conclusion and future work this work introduced dial2msa-verified, an extension of the dial2msa dataset that involves the verification of previously unverified dialects. we enriched the training data by incorporating seq2seq datasets from various domains. we conducted a comprehensive model evaluation using multi-reference evaluation, demonstrating improved performance compared to single-reference evaluations. our findings indicate that models trained in the j-r configuration outperformed those in the i-r configuration. this improvement is due to the inherent similarities between dialects, allowing dialects to be learned from one another. additionally, the i-r configuration relied on dialect identification for model selection, which affected translation performance. overall, arat5 outperformed other models, achieving an average bleu score of 41.12 and a chrf++ score of 62.05. in future work, we plan to expand the training data, focusing on the social media domain, as the limited availability remains an obstacle. additionally, we plan to explore the possibility of improving the normalisation performance by leveraging more advanced models, data augmentation techniques and transfer learning techniques.