SANA at NADI 2023 shared task: Ensemble of Layer-Wise BERT -based models for Dialectal A rabic Identification (2023)

Abstract: Our system, submitted to the Nuanced Arabic Dialect Identification (NADI-23), tackles the first sub-task: Closed Country-level dialect identification. In this work, we propose a model that is based on an ensemble of layer-wise fine-tuned BERT-based models. The proposed model ranked fourth out of sixteen submissions, with an F1-macro score of 85.43.

## introduction arabic is the national language of 25 countries spoken by more than 372 million speakers foot_0 . while modern standard arabic (msa) is the formal written language and is used in speech in a formal contexts such as in academia, official communications, and education (althobaiti, 2020), each country has its own culturallybased dialect that is used in daily communication and informal situations @xcite . nowadays, within the evolution of virtual communication technologies and the intense popularity of social media platforms, dialectal arabic has replaced msa as the primary written form of arabic to generate online informal content. for example, users on social media share news, comment on political and social events, and express opinions concerning various aspects of life using their own dialect. accordingly, social media is an invaluable resource for harvesting huge amounts of dialectal arabic data which can be utilized in numerous computational linguistics and natural language processing (nlp) applications. due to variations between dialects in term of vocabulary usage, meaning, and sense of given words or phrase, automatic identification between unique dialects is a crucial component for improving several downstream applications such as sentiment analysis, speech recognition, and machine translation. in order to increase the efficiency of arabic nlp, the nuanced arabic dialect identification (nadi) shared task series are dedicated to developing solutions for arabic dialects identification and other related dialectal processes @xcite @xcite . the majority of the works submitted to the nadi-22 employed pre-trained bert-based models that are specifically trained on arabic corpus, such as marbert @xcite , @xcite , and aragpt2 (antoun et al., 2021) using various tuning and data augmentation techniques (abdel-salam, 2022; @xcite . other researchers, such as (alshenaifi and azmi, 2022) and @xcite , used classical machine learning algorithms with tf-idf and word embeddings. in this paper, following the first line of work, we present our system submitted to the nadi-2023 shared task @xcite . specifically, to address the first shared sub-task, our approach is based on an ensemble of layer-wise bert-based models. each model is trained independently by accessing hidden states from a designated bert layer and averaging them to generate the final text embeddings. this paper is organized as follows: section 2 presents the dataset utilized in our work, section 3 introduces the proposed system for arabic dialect identification, section 4 provides details experimental results and evaluation, section 5 discusses the model ## dataset the nadi-2023 shared task provided the twt-23 dataset for the arabic dialects identification task. the dataset contained a total of 23,400 tweets that included 18 arabic dialects. the dataset was categorized into 18k tweets for training, 1800 tweets for development, and 3600 samples for testing. the training set contained 1000 samples for each dialect class, and the development set included 100 samples for each target class. ## system description interpretability of pre-trained language models is an outstanding and active research area in nlp. ## results we evaluated the performance of our proposed method on the dialects identification task through a set of experiments to investigate the impact of each layer on classifying the 18 dialects. ## error analysis and discussion figure 1 shows a heat-map of confusion matrix for the development set to further analyze the margin of error in the model's predictions. in general, with minor exceptions, the model seems to perform well for most of the classes. for example, the model preforms well at predicting instances for egypt, morocco, and sudan classes, with true positive exceeding 95 instances. conversely, the number of true positives are as low as 79 instances or less when predicting instances for the palestine, uae, and qatar classes. to further analyse, missing of diacritics: in arabic, while different arabic dialects share common linguistic features, differences remain in the usage of the vocabulary and its meaning. diacritics plays a crucial role in disambiguate the senses, meanings, and semantics of arabic language @xcite @xcite . we hypothesize that adding diacritics may improve the model's performance in predicting the dialect of a given text. to illustrate more, the first two examples in sahabayi", where in the ksa dialect it would be pronounced with the following diacritics " ‫ص‬ ‫ا‬ ‫ِح‬ ‫ب‬ َ ‫ي‬ ِ sahibayi". including diacritics may also resolve the ambiguity in the second example, where the words " ‫ر‬ ‫ا‬ ‫ج‬ ‫ل‬ rajl and ‫ي‬ ‫ن‬ ‫و‬ ‫ر‬ ynwr" in the example, which translate respectively to "man" and "lights up", pronounced differently in both egyptian and libyan dialects. regional varieties: among the 18 dialects classes, the ksa class has the largest variety of dialects due to the geographical diversity and historical migration of people from different linguistic backgrounds. thus, the east region of ksa tends to share a lot of linguistic similarities with egypt, while the southern region share similarities with yemen, the northern region is similar to the levantine dialect (this includes: syria, jordan, palestine, and lebanon), and the middle and western regions congruent with rest of gulf countries @xcite . also, according to @xcite , the majority of most active twitter users are from ksa. hence, we believe that these factors affected the performance of our model, as the majority of the fp predictions were a result of flawed prediction where other classes were categorized as ksa, examples 3 5 in ## conclusion this work describes our proposed system to automatically identifying dialectal arabic, which has been submitted to the nadi-2023 shared task. the proposed system leveraged the intermediate layers of the pre-trained marbertv2 in identifying the arabic dialects instead of relying on the final layer for text representation. the proposed layer-wise bert-based models demonstrate a strong overall performance in distinguishing 18 arabic dialects, achieving an f1 score of 87% on the development set and 85% on the test set. furthermore, we analyzed the performance of our model and discuss the factors that caused fp and fn predictions. hence, further elaboration could be followed to study the impact of using diacritics on model performance.