Domain-Adapted BERT -based Models for Nuanced A rabic Dialect Identification and Tweet Sentiment Analysis (2022)

Abstract: This paper summarizes the solution of the Nuanced Arabic Dialect Identification (NADI) 2022 shared task. It consists of two subtasks: a country-level Arabic Dialect Identification (ADID) and an Arabic Sentiment Analysis (ASA). Our work shows the importance of using domain-adapted models and language-specific pre-processing in NLP task solutions. We implement a simple but strong baseline technique to increase the stability of fine-tuning settings to obtain a good generalization of models. Our best model for the Dialect Identification subtask achieves a Macro F-1 score of 25.54% as an average of both Test-A (33.89%) and Test-B (19.19%) F-1 scores. We also obtained a Macro F-1 score of 74.29% of positive and negative sentiments only, in the Sentiment Analysis task.

## introduction the arabic language is one of the rich languages in the world, spoken in large geographical regions. it is officially spoken by people from the middle east and north africa (mena) countries, covering a population of approximately 400 million people. it's a culturally and grammatically rich language, with a complex morphological structure. arabic is one of the semitic languages and has a widely varying collection of more than 30 different dialects (according to the summer institute of linguistics a.k.a. sil international). these dialects are affected by geopolitical and religious influence. the question of how to classify the different varieties of spoken arabic is a long-standing problem in the fields of arabic and semitic linguistics. researchers still develop tools and systems to keep the language in the race of natural language processing (nlp) tasks on both modern standard arabic (msa) and its dialects (da). a dialect of the arabic language can have a different meaning of a word or a vernacular dialect can differ syntactically, morphological, and orthographically in the choice of vocabulary and pronunciation. each of these variations of dialects is distinct enough to make users resort to formal arabic to understand each other. this prompts the need to develop a system that can automatically detect the source, region, and/or specific dialect of a given sequence of tokens or text segments. the nadi shared task series .... 2022 ) is one of the prominent competitions that provides datasets and modeling opportunities for researchers to improve nlp work in arabic. social media provides an environment for the use of both formal and informal language. this makes it more difficult when arabic is used on social media since both dialects and the formality of the language will be taken into consideration when processing text data from social media like twitter. this variety of dialects can be classified and used for more semantic and linguistic findings and work using machine learning and deep learning models. language models (lm) have evolved over the years from the birth of the nlp domain, starting with simple n-gram lms, with many computational and performance limitations. after the introduction of deep learning (dl), language modeling switched to language modeling using recurrent neural networks (rnn), gated recurrent unit (gru), and long-short term memory (lstm) with chronologically better deployability than the earlier methods. the drastic improvement was after introducing the transformer architecture for language modeling @xcite using the self-attention mechanism. transformer-based lm like bidirectional encoder representations from transformers (bert) @xcite , are currently widely used in the nlp field to achieve state-of-the-art (sota) results in various tasks. bert and its variations (roberta, distilbert, albert, etc.) are outstanding models, and they are close to becoming a de facto baseline for almost all nlp tasks, especially for natural language understanding (nlu) downstream tasks. this is because of the capability of these general models to be fine-tuned on narrower tasks in different domains with high accuracy and low cost. in this paper, we develop a system for the classification of arabic dialects at the country level. arabic dialect identification (adid) problem is challenging because adjacent countries influence each other, with the present intermediate dialects @xcite . our system also provides arabic sentiment analysis (asa) of given tweet texts. we improve both adid and asa tasks using arabert (antoun et al., 2020) and marbert (abdul-mageed et al., 2020a), arabic language-specific pre-trained bert models. the rest of this paper is organized as follows. in section 2, we provide a detailed explanation of the problem and datasets provided by @xcite . section 3 talks about the methodology and general system development. we provide the results in section 4 and discuss the results and model limitations in section 5. the paper is concluded in section 6. ## data the nadi-2022 shared task provides two problem definitions of country-level dialect identification and sentiment analysis in arabic, posted as subtask 1 and 2 respectively. the geographical distribution of the countries covered in the dataset of subtask 1 is shown in the map in fig 1 . dialect distributions in the training and development sets vary based on the countries. in the datasets, for each country, we present the count of tweets included in both training and development sets, as seen in fig 2 . in the general collections of the tweets, there was no msa taken into consideration in both datasets provided, rather just spoken dialects in the various countries as used in @xcite . in subtask 1, the test set is divided into test-a which includes 18 dialects on the country level, and test-b which covers k country-level dialects, where k is kept unknown. in subtask 2 there's only one set for the test as shown in ## system development in recent advancements of nlp, models with stateof-the-art (sota) results like smart-roberta large @xcite have shown that using transformer models, it is reasonable to expect sota performance in tasks such as sentiment analysis @xcite ) and question answering @xcite . the sota leaderboard of sst-2 dataset @xcite shows clearly that transformer models are currently the best for text classification with almost the top 50 models using transformer architecture 2 . we use the same approach in solving both subtask 1 and subtask 2 of the shared task. we used pre-trained transformer models in all experiments. domain-specific transfer learning and finetuning of transformer models is proven to be more robust by issifu et al. @xcite and bayrak et al. @xcite , @xcite . they fine-tuned transformer models on biomedical and turkish law datasets respectively to achieve results better than their original general transformer models. better performance obtained in these works are accredited to for a more robust performance of the system, we adopt arabic language domain-specific pretrained transformer models, arabert and mar-bert. these pre-trained models gained sota in sa on ajgt,hard,labr (2-class, unbalanced) datasets. ## results to evaluate the results, we use the official metrics defined by the shared task: macro-averaged fscore for dialect id (subtask 1) and macro-f1-pn score -neglecting the neutral class-over the positive and negative for the sentiment classification (subtask 2). in as giyaseddin team. in the same table, we show the macro f-1 score with the accuracy for the experimented models against each of the dev, test-a, and test-b set provided by the shared-task for subtask 1. similarly, the test results of subtask 2 are presented in ## discussion and future work according to our experiments, we see that the preprocessing we used has a positive impact on dialect identification, unlike sentiment analysis. initial results say that tokens and expressions that identify a dialect are not correlated with the processed (replaced or removed tokens like emojis, repetitions, etc.) so we see better results with them processed. in asa, on the other hand, we see that they have an opposite effect on the classification. in general, marbert performs better on both subtasks even though its 2nd version performs better on the ar-lue benchmark @xcite . in fig 5 we see the kernel density estimation (kde) line of failing predictions of the model on adid lies slightly to the left of the line of the successful predictions. this means that the probability to make correct predictions is higher when the sentence is longer. text label prediction #words comment 0 egypt iraq 3 mislabelled 1 syria ksa 7 mislabelled 2 tunisia oman 3 unclear / both 3 ksa iraq 4 unclear / both 4 syria oman 3 mislabelled 5 palestine egypt 10 mislabelled in our analysis, we focus more on adid, in which we still have to face the challenge of the highly correlated dialects such as palestinian with jordanian, or saudi arabian with emirati or omani. combining msa with the dialects makes the problem harder and it is out of the scope of subtask 1. moreover, labeling such a dataset is hard to achieve without any confusion in the labels, even a humanlevel baseline might not be purely reliable. we present some of the examples that are either mislabelled or unclear in ## conclusion this study is focused on two main tasks: arabic dialect identification and arabic sentiment analysis based only on the text of the tweets. we demonstrate the nuanced variations between the models before and after applying language-specific preprocessing, besides using domain-adapted models pre-trained on arabic corpus. understanding these variations requires knowledge of the nature of different data collections that should be considered. we conclude that it is important to choose the set of hyper-parameters of fine-tuning carefully to obtain a more stable and better generalization. finally, we found that marbert outperforms other models in the generalization capability in both subtasks.