A lex UNLP - STM at NADI 2024 shared task: Quantifying the A rabic Dialect Spectrum with Contrastive Learning, Weighted Sampling, and BERT -based Regression Ensemble (2024)

Abstract: Recognizing the nuanced spectrum of dialectness in Arabic text poses a significant challenge for natural language processing (NLP) tasks. Traditional dialect identification (DI) methods treat the task as binary, overlooking the continuum of dialect variation present in Arabic speech and text. In this paper, we describe our submission to the NADI shared Task of ArabicNLP 2024. We participated in Subtask 2 - ALDi Estimation, which focuses on estimating the Arabic Level of Dialectness (ALDi) for Arabic text, indicating how much it deviates from Modern Standard Arabic (MSA) on a scale from 0 to 1, where 0 means MSA and 1 means high divergence from MSA. We explore diverse training approaches, including contrastive learning, applying a random weighted sampler along with fine-tuning a regression task based on the AraBERT model, after adding a linear and non-linear layer on top of its pooled output. Finally, performing a brute force ensemble strategy increases the performance of our system. Our proposed solution achieved a Root Mean Squared Error (RMSE) of 0.1406, ranking second on the leaderboard.

## introduction the arabic language exhibits a duality of forms: modern standard arabic (msa) serves as the standardized variety taught in schools and used in formal settings and communication across the arab world. in contrast, numerous regional varieties of dialectal arabic (da) dominate everyday spoken communication, including interactions on social media. these dialects significantly differ from msa in phonology, grammar, and vocabulary @xcite . this bifurcation of the language, coupled with frequent code-switching between msa and da, presents significant challenges for arabic nlp systems @xcite . researchers have developed numerous systems to address dialect iden-tification (di), often at the sentence level @xcite @xcite and occasionally at the token level to identify instances of code-switching @xcite . however, these approaches often adopt a binary perspective, categorizing a sentence or token as either msa or da. to tackle these challenges, @xcite ) introduced the aoc-aldi dataset, which contains 127,835 sentences labeled for their level of dialectal variation, derived from the aoc dataset @xcite . the aoc dataset includes user comments from three newspapers, representing egyptian, levantine, and gulf dialects. in this paper, we explore the use of pre-trained transformer-based language models. given that the aoc-aldi dataset is imbalanced, with approximately 50% consisting of msa samples labeled as 0, transformer models often overfit in this context. therefore, we fine-tuned the linear and non-linear layers on top of these pre-trained language models and explored different training strategies, including contrastive learning and the use of a weighted random sampler in the data loader. combining these approaches resulted in further improvements in system performance. finally, employing a bruteforce ensemble strategy with the collaboration of 5 checkpoints from different training paradigms further increased the effectiveness of our system. throughout this paper, we use the following abbreviations: weighted random sampler (wrs), contrastive loss (cl), temperature parameter for nt-xent loss (t), remove non-arabic letters and links (rmnall), and drop 43,000 samples from majority label (ds). ## related work early research on arabic dialect identification (di) focused on both binary msa-da classification and multi-class problems involving various da variants (althobaiti, 2020; @xcite . arabic di has garnered significant attention, leading to multiple shared tasks @xcite and datasets @xcite @xcite . research has been conducted at both the sentence and token levels, but existing methods often struggle to differentiate between sentences that share similar dialectal cues but exhibit varying levels of dialectness. to address these issues, @xcite introduced the aoc-aldi dataset, which includes 127,835 sentences labeled for dialectness, derived from user comments on articles from three newspapers representing egyptian, levantine, and gulf dialects @xcite . they employed a bertbased regression model, fine-tuning a regression head atop marbert @xcite , and conducted multiple rounds of fine-tuning with different random seeds to ensure consistency. there has been an interest in contrastive learning @xcite in recent years due to its significant advancements in various nlp tasks. however, its application within the arabic context has been relatively limited. this trend has shifted as demonstrated by @xcite , who showcased the efficacy of contrastive learning in arabic hate speech detection, resulting in substantial improvements over baseline methods. recent studies in computer vision @xcite have shown that incorporating a classification loss alongside a regression loss enhances the performance of deep regression models. they found that this approach is particularly beneficial for regression tasks with imbalanced data distributions. ## data for our experiments, we used the training aoc-aldi dataset provided by the shared task. we conducted an extensive analysis of the dataset to gain insights that could aid us in effectively addressing the problem. the dataset comprises 102,886 data samples with 68 unique labels ranging from 0 to 1 where 0 represents msa, and 1 signifies high divergence from msa. the distribution of the dataset is illustrated in figure 1 . we observed a significant imbalance in the dataset, where the msa label predominates over other labels. specifically, there are 53,844 data samples labeled as 0 (msa), accounting for approximately 52.3% of the total dataset. to mitigate this imbalance, we employed wrs in the data loader. additionally, we explored undersampling the majority class by removing around 43,000 samples from the msa data samples. we also performed simple data preprocessing by removing non-arabic letters from the sentences, but it decreased the system's performance. finally, we split this dataset into an 85% training set and a 15% dev set. our dev set serves to monitor the training process and detect issues such as overfitting or underfitting and to keep the best checkpoints of the training procedure. for the shared task provided dev set, we found that it has 107 data samples, which have a slightly different distribution from the provided training dataset. the distribution of the shared task dev set can be found in figure 2 for this circumstance, we saved the top 4 checkpoints of any training paradigm on our dev set and then tested them all on the subtask 2 dev set to choose the best one. ## results we saved the best 4 checkpoints on our dev set, which represents 15% from the subtask 2 provided training data, then tested those 4 checkpoints in the shared task dev set to get the best checkpoint. the results on the subtask 2 dev set and test set are given in ## qualitative results in this section, we provide practical examples drawn from the dev set of the shared task. subsequently, we conduct a comparative analysis between the outputs generated by our best system based on the ensemble and the actual ground truth, as detailed in ## discussion in experiment 4, we cleaned the dataset by rm-nall. surprisingly, this led to a decrease in performance. this decline might be due to inconsistencies among the dataset annotators, who could have considered these elements when scoring the samples. as a result of the reduced performance, we chose not to implement these preprocessing steps. considering the imbalanced distribution of the given training set, we sought techniques to mitigate bias towards the majority class. finally adding the wrs and combining the rmse with contrastive loss (experiments 6,7, and 9) led to a significant improvement in the results. experiments 1, 2, 5, and 6 showed that adding wrs led to an increase in the performance of the arabertv2-twitter base rather than the large version. this is because wrs led to overfitting to the minority samples, so we needed to reduce the complexity of the model. the noticeable difference between the results of the shared task dev set and test set (leaderboard results) in experiment 9, 13 in ## conclusion in this paper, we address the challenge of recognizing the nuanced spectrum of dialectness in arabic text. we developed a sophisticated approach that surpasses binary classification by acknowledging the continuum of dialect variation. by using wrs, cl, and a robust ensemble strategy, our solution achieved a noteworthy rmse of 0.1406, ranking second on the leaderboard.