ASOS at NADI 2024 shared task: Bridging Dialectness Estimation and MSA Machine Translation for A rabic Language Enhancement (2024)

Abstract: This study undertakes a comprehensive investigation of transformer-based models to advance Arabic language processing, focusing on two pivotal aspects: the estimation of Arabic Level of Dialectness and dialectal sentence-level machine translation into Modern Standard Arabic. We conducted various evaluations of different sentence transformers across a proposed regression model, showing that the MARBERT transformer-based proposed regression model achieved the best root mean square error of 0.1403 for Arabic Level of Dialectness estimation. In parallel, we developed bi-directional translation models between Modern Standard Arabic and four specific Arabic dialects—Egyptian, Emirati, Jordanian, and Palestinian—by fine-tuning and evaluating different sequence-to-sequence transformers. This approach significantly improved translation quality, achieving a BLEU score of 0.1713. We also enhanced our evaluation capabilities by integrating MSA predictions from the machine translation model into our Arabic Level of Dialectness estimation framework, forming a comprehensive pipeline that not only demonstrates the effectiveness of our methodologies but also establishes a new benchmark in the deployment of advanced Arabic NLP technologies.

## introduction dialectal arabic (da) is essential in daily interactions for over 422 million arabic speakers, significantly influencing casual conversations and digital communications, including social media @xcite . despite modern standard arabic (msa) being used formally in education, official, and media contexts, da's regional variations and frequent code-switching with msa present challenges for text analysis and computational processing, highlighting da's cultural and linguistic importance @xcite . the rise of digital communication has increased the use of da, driving significant nlp research to better understand and process it, thus bridging the gap between msa's formal structure and da's nuances @xcite . this paper addresses two crucial aspects of arabic nlp: the estimation of the level of dialectness (aldi) and the translation of arabic dialect to msa. building on the shared tasks provided by @xcite and previous years' tasks @xcite , we explore innovative methodologies to enhance the computational handling of arabic's dual nature. integrating aldi estimation with dialect-to-msa translation not only advances our understanding of arabic's linguistic duality but also improves practical applications in processing its informal variants. recent advancements in machine translation focus on translating arabic dialects to msa. however, much work, such as @xcite , has centered on single dialects using limited datasets, limiting their effectiveness across the arabic dialect spectrum. notably, @xcite enhanced nmt for the algerian dialect using transductive transfer learning. additionally, @xcite developed turjuman, leveraging the arat5 model for multi-dialectal translation, demonstrating the benefits of scalable frameworks. in parallel, the conceptualization of aldi marks a significant shift in addressing dialect variation in arabic text. previous works have focused on dialect estimation at both the sentence level @xcite and token level @xcite , often viewing the distinction between msa and da as binary. earlier studies @xcite recognized the spectrum of dialectness and emphasized standard annotation guidelines for identifying dialect switching. recently, @xcite advanced this understanding by releasing a dataset of 127,835 arabic comments with aldi labels and proposing a method for estimating aldi that can generalize to other dialect corpora. building on these foundations, this study introduces a transformative approach that bridges aldi estimation with da-to-msa translation. our contributions are threefold: aldi estimation model, introducing a transformer-based regression model that sets new benchmarks in aldi estimation, offering a sophisticated tool for analyzing dialectal nuances in arabic text. dialect-specific translation models, developing advanced bi-directional translation models (msa2dialect and dialect2msa), fine-tuned to handle the specificities of individual arabic dialects, enhancing translation accuracy. evaluation pipeline, establishing a novel evaluation pipeline that utilizes machine translation outputs to refine aldi estimations, illustrating the potential of integrating these two critical aspects of arabic nlp. the remainder of this paper details our work, starting with the dataset and our methodological approach, followed by the system setup, results with discussion, and finally a conclusion reflecting on the impact with future directions of our research. ## dataset for the first subtask focused on the aldi estimation, the dataset utilized consisted solely of the training set provided by nadi 2024 @xcite . this dataset consists of 100k sentences and it is specifically tailored to evaluate the aldi across various arabic texts. a distribution plot of the aldi scores from this training dataset is included in figure 2 , which illustrates the frequency of scores across different intervals. the distribution of aldi scores, as depicted in figure 2 , shows a significant concentration at lower scores with less frequent occurrences across higher values, indicating a prevalence of text closely aligned with msa. for the dialect to msa subtask, a more involved dataset preparation strategy was employed. to create a robust and dialect-specific training environment, 20,000 sentences from a monolingual msa dataset provided by nadi 2024 (abdul-mageed et al., 2023) were generated for each of the target dialects: egyptian, emirati, jordanian, and palestinian by using dialect-specific trained machine translation models. additionally, the madar dataset @xcite was utilized selectively for each dialect to include region-specific variations. this ensured that each dialect's dataset was finely tuned for optimal translation performance. the composition of these datasets is summarized in ## system our framework consists of an integrated pipeline developed to advance the machine translation of arabic dialects to msa and to estimate the level of dialectness. the system is described in figure 2 showing the integration of the machine translation model with the aldi estimation regression model. as shown in figure 2 , the initial phase of our pipeline involves translating msa to specific arabic dialects using the monolingual msa dataset, which contains 20,000 sentences for each target dialect. this is part of a data augmentation strategy intended to enhance the training datasets for subsequent translation tasks. following this, we integrate dialect-specific datasets from the madar corpus @xcite , which provides additional contextual diversity and linguistic features pertinent to regional arabic dialects like egyptian (cairo, alexandria, aswan), jordanian (amman, salt), and palestinian (jerusalem). each of these enriched datasets is then used to fine-tune separate da-msa models for each dialect using the arat5-v2 (elmadany et al., 2022) model, thereby generating msa predictions. the output from our machine translation models, termed 'msa predictions' feeds into the aldi estimation model as a test phase. the regression model is a marbert transformer based model @xcite , where the architecture pretrained specifically for the arabic language. the proposed regression model as shown in figure 2 is designed to estimate the level of dialectness as a single percentage value, reflecting how closely a sentence aligns with msa. the pipeline enhances arabic dialect understanding by automating parallel corpora construction and serving as a critical evaluation framework for machine translation models and aldi estimation approaches. ## result this section demonstrates the results of our comprehensive experiments conducted to evaluate the effectiveness of our machine translation models from various arabic dialects to msa and vice versa, as well as the performance of our aldi estimation model. msa to dialect models, the msa to dialect (msa2dia) models were trained using a monolingual dataset to increase the volume of the training data for the dialect to msa training. the validation bleu scores for each dialect are shown in figure 3 . the results depicted in figure 3 showcase the performance of the msa to dialect (msa2dia) models across various dialects. the emirati dialect model achieved the highest validation bleu score, closely followed by the palestinian and jordanian models, with the egyptian model trailing. the models' strong bleu scores indicate that each dialect's training data was well-represented, which enabled the models to provide dialect-specific translations from modern standard arabic (msa). dialect to msa models, the dialect to msa (dia2msa) translation models discussed here represent an effort to convert various arabic dialects back to modern standard arabic (msa). this process involved using a generated monolingual dataset combined with existing datasets as shown in figure 2 in order to fine-tune different sequenceto-sequence models, leading to the results captured in the tables 2 and 3 which present the bleu scores for three different models: arat5 v2, turjman, and mt5 for the development phase and test phase of the shared task respectively. as shown in tables 2 and 3 , three models, arat5 v2 foot_0 , turjman foot_1 , and mt5 foot_2 , were tested for their ability to translate arabic dialects back to modern standard arabic. arat5 v2 showed the highest proficiency, particularly with the emirati dialect, with overall bleu scores of 28.74 and 17.13 on dev and test sets, respectively. turjman and mt5 models had lower scores, with mt5 struggling across all dialects, indicating limitations in adaptability and generalization capabilities. aldi estimation, we evaluated various transformer-based models on the development dataset using a proposed regression model. the development phase results, as depicted in the test phase as shown in msa predictions for aldi estimation, we proposed an evaluation framework designed to assess the effectiveness of dialect to msa (dia2msa) models by leveraging outputs from these models as input for the proposed aldi estimation model. figure 4 shows the histogram of the distribution of the aldi scores for the msa predictions produced by our best model in the distribution depicted in figure 4 illustrates that the majority of translations by the dia2msa models are closely aligned with msa, as indicated by the distribution primarily clustering around lower aldi scores, with a significant peak at 0.1. this skew towards lower scores suggests a high level of effectiveness in the dia2msa models' ability to produce outputs that mirror the characteristics of msa. error analysis, an error analysis of the aldi score estimation model which quantifies dialectal content in arabic text is implemented. the model was evaluated using rmse across various segments determined by the data percentiles, revealing how model accuracy varied with different levels of dialectal content. the rmse distribution is shown in figure 5, illustrating the model's performance across segmented score ranges. the model excels in regions with moderate dialectal content with rmse of 0.08669 but struggles at the extremes, a challenge likely increased by the lack of representation of higher dialectal scores in the training set. to enhance model performance, we will work on enriching the training dataset with more examples of medium to high dialectal content, retraining the model for better generalization across the dialect spectrum, and employing advanced techniques like transfer learning for improved handling of dialectal nuances. ## conclusion our work makes a significant contribution to the nadi2024 shared task by employing and evaluating transformer-based models to tackle two critical aspects: the estimation of aldi and the mt of dialectal sentences into msa. we rigorously tested various sentence transformers and found that the marbert transformer-based regression model performed exceptionally well, achieving an rmse of 0.1403 in aldi estimation on unlabeled test data. additionally, we developed and refined bidirectional translation models between msa and four specific arabic dialects. this not only expanded our dataset but also significantly improved translation quality, as evidenced by a bleu score of 0.1713. integrating msa predictions from the mt model into our aldi estimation framework established a comprehensive pipeline that effectively captured the spectrum of dialectal variation within arabic text. these advancements set a new benchmark for deploying advanced arabic nlp technologies, highlighting their potential in nuanced linguistic analyses.