IUNADI at NADI 2023 shared task: Country-level A rabic Dialect Classification in Tweets for the Shared Task NADI 2023 (2023)

Abstract: In this paper, we describe our participation in the NADI2023 shared task for the classification of Arabic dialects in tweets. For training, evaluation, and testing purposes, a primary dataset comprising tweets from 18 Arab countries is provided, along with three older datasets. The main objective is to develop a model capable of classifying tweets from these 18 countries. We outline our approach, which leverages various machine learning models. Our experiments demonstrate that large language models, particularly Arabertv2-Large, Arabertv2-Base, and CAMeLBERT-Mix DID MADAR, consistently outperform traditional methods such as SVM, XGBOOST, Multinomial Naive Bayes, AdaBoost, and Random Forests.

## introduction officially spoken in more than 20 countries, and in a myriad of regional variations, the arabic language has consistently piqued the curiosity of researchers across various disciplines. this is because of arabic's historical significance and pivotal role in shaping the cultural, religious, social, and political fabric of the arab world. historically, arabic has often been typologically classified into three distinct categories: classical arabic, modern standard arabic (msa), and dialectal arabic (da). classical arabic refers to the language used in the holy qur'an and pre-islamic poetry, while msa pertains to the language of newspapers, literature, education, official documents, and formal media and news broadcasts. da, which is the primary focus of this paper, is more concerned with the language used in daily communication by speakers of arabic. these dialects are often classified into: egyptian, levantine, gulf, and maghrebi arabic. within each of these distinct communities, an array of subdialects can be found in different geographical regions @xcite @xcite . for the most part, msa was the predominant variation used in written arabic. but the advent of online forums and social media platforms, such as twitter, gave the variations of da a space to grow their written content presence. these dialects differ phonologically, morphologically, syntactically, and semantically. yet, it is noteworthy to mention that there can still be some degree of overlap between da and msa. this is due to the fact that arabic is a root-based language, which means that many words share common roots consisting of three or four letters. this unprecedented massive increase in digital content in da has propelled the development of nlp tools that can read, manipulate, and potentially generate this content. while developing such tools to handle text in msa has posed many challenges, this task has been even more arduous to do for text written in da, e.g., tweets. arab users of twitter mainly use no standardized orthographic variation (e.g., , , in this article, we outline our system, which we entered in task 1 of the nadi2023 shared task focusing on arabic dialects classification @xcite . as with the three preceding nadi shared tasks @xcite , @xcite , and @xcite , the primary objective of this task is to develop models capable of categorizing tweets originating from 18 distinct arab countries. 1 665 2 methodology 2.1 data for the purposes of this task, a twitter dataset of 23.4k tweets, covering 18 different dialects from 18 countries, is provided. this dataset is divided into 3 smaller sets: 18k tweets for training, 1.8k tweets for development, and 3.6k tweets for testing. additionally, datasets from the previous two nadi tasks @xcite , and madar @xcite were provided. participants in this task were not allowed to use any other datasets. ## evaluation for subtask 1, the evaluation metrics will include precision, recall, f-score, and accuracy. macroaveraged f-score will be the official metric; hence we report our results using this metric along with all the evaluation metrics. we decided which models to submit based on the model's performance on the development dataset provided by the organizers. ## results as shown in in addition to the officially submitted systems, we performed a more extensive evaluation of the development set. we trained and evaluated 8 different classifiers. the results of these experiments are shown in our pre-processing pipeline had a positive effect on the random forests model, improving the f1 score to 0.43, compared to 0.39 without preprocessing. in contrast, it had a detrimental impact on the naive bayes model, reducing f1 to 0.41 from 0.43 without pre-processing. the pipeline had no impact on the results of xgboost, svc, and adaboost. it is important to note that pre-trained models already incorporate their own internal preprocessing pipelines. even though the pipeline did not achieve significant results, we still believe it was necessary to eliminate redundancy and reduce data size. ## discussion the arabic dialect identification task, as explored in this research, addresses a crucial challenge in natural language processing, particularly for applications involving arabic text. we observed promising results during the evaluation phase, demonstrating the system's ability to correctly identify arabic dialects with a high degree of accuracy. however, it is essential to recognize that the task itself presents inherent challenges due to the nuances and variations present within arabic dialects. arabic speakers often code-switch between dialects and standard arabic, which affects the performance of models. given an additional three months to work on this task, several avenues for improvement and further development can be pursued: fine-tuning strategies: experimenting with 3 advanced fine-tuning techniques, such as domain adaptation or multi-task learning, may help the model handle ambiguous phrases and codeswitching more effectively. post-processing techniques: implementing postprocessing techniques, such as dialect consistency checks, to ensure that the identified dialect remains consistent within a given text could mitigate errors caused by code-switching. ## conclusion in this paper, we have detailed our contributions to the nadi 2023 shared task on arabic tweet classification across 18 arab countries. our experiments have revealed that employing arabertv2large yields the most promising results. our system achieved a ranking of 13th out of 16 participating teams. looking ahead, our future research will explore the potential benefits of employing ensemblebased approaches with transformer-based models. additionally, we are keen to investigate the potential advantages of incorporating tokenization, stop word removal or splitting, and stemming into our pre-processing pipeline.