A rabic Dialect Identification Using BERT -Based Domain Adaptation (2020)

Abstract: Arabic is one of the most important and growing languages in the world. With the rise of the social media giants like Twitter, Arabic spoken dialects have become more in use. In this paper we describe our effort and simple approach on the NADI Shared Task 1 that requires us to build a system to differentiate between different 21 Arabic dialects, we introduce a deep learning semisupervised fashion approach along with pre-processing that was reported on NADI shared Task 1 Corpus. Our system ranks 4th in NADI’s shared task competition achieving 23.09% F1 macro average score with a very simple yet an efficient approach on differentiating between 21 Arabic Dialects given tweets.

## introduction arabic dialect classification is a task of identifying the dialect of the writer given an input text. this task has been an active field of research the past few years due to the rise of arabic corpora which are made available @xcite . however, nadi's arabic dialect corpus @xcite have been quite challenging and intriguing. nadi's corpora has 21 different country dialects from which some of them are quite similar to each other in terms of morphology. some of the data provided had some english words, others had quran verses generated from third party apps which makes it extremely difficult for not just models, but also human cannot differentiate between the dialects if the tweet is a quranic verse, since quran compromises classical arabic not dialectal arabic. this is noise in the dataset that a model can not fix. also, the imbalance of training data introduced other challenges and difficulties. previous work in this task involved the use of traditional ml algorithms, rnn with their variants, and hybrid approaches like in @xcite . however, we were inspired by self-attention technique @xcite . due to the huge success of transformers in many classification tasks, we opted for approach of using pretrained bert @xcite in semi-supervised deep learning as it is proved by @xcite that fine-tuning pretrained model to specific domain is an efficient solution. we fine-tuned pretrained bert transformer on "arabert" @xcite ) arabic text, which is trained on 23b gb of data, using masked language modelling with huggingface interface @xcite . this process was done after doing data pre-processing and data augmentation in order to alleviate the problems of having class imbalance and lack of training data. transfer learning has proven to be an efficient approach in classification tasks, especially when we do not have enough training data. we leveraged the unlabeled data provided by nadi @xcite through twitter api's, for fine-tuning using language modelling to achieve domain adaptation. where we adapt pretrained arabert to the domain of tweets. that is because of similarity of morphological and semantic features that both the unlabeled data and target data share. pretrained bert model "arabert" was fine-tuned on nadi's unlabeled corpus using masked language model to let it learn better features. these features are going to be used to classify target labeled training examples provided by nadi, which have the same distribution as the unlabeled ones. our results confirm that such a technique have improved our baseline performance significantly by 3% f1 macro average score. the rest of the paper is organized as follows. section 2 presents the data that were supplied for training our model. section 3 is a description of our system in detail and what infrastructure we used to produce the results. in section 4, we present our results using various techniques. section 5 is a discussion about task and common errors that occured. section 6 concludes the work we have done as well as suggested future work. ## data the dataset was one of the factors that made this problem quite challenging. nadi shared task organizers (abdul-mageed et al., 2020) provided a corpus of arabic tweets from twitter platform having 21 class labels corresponding to countries {egypt, iraq, saudi arabia, algeria, oman, emirates, libya, syria, morocco, yemen, tunisia, lebanon, jordan, kuwait, palestine, qatar, bahrain, djibouti, mauritania, somalia and sudan}. the corpus is divided into training set, dev set and test set to report our final results on. the number of examples in the 3 sets is, 21,000 tweets, 4,957 tweets and 5,000 tweets respectively. to aid in the training and model building processes, the organizers also provided additional 10 million unlabeled tweets ids from the same distribution of the labeled tweets, to be obtained using a provided python script. ## results we present our experiments results on the development set provided by organizers which consisted of 4,957 examples. in a second experiment which was not submitted was concluded based on hierarchical model of arabert with naive-bayes. however, this approach had an f1 macro average of 22.3%, which is still 2% f1 macro score less than a single transformer model. ## discussion it was observed in our experiments, through analysis of data, that some of the tweets had noticeable noise that cannot be solved. some users used to tweet through third party apps. we noticed that these tweets were mainly prayers or quran versus coming from various apps like {http ://d3waapp.org , http://knzmuslim.com , http://gharedly.com and http://du3a.org} . the main problem is that these tweets are given different labels according to the nationality of the user, and in fact they are not even differential by humans, which makes it very hard for a trained model. another noticeable noise was that some tweets are actually a retweets. where the retweeted content refers to a dialect, but it was given another label because the actual user retweeting is having a different dialect. in future work, we plan on exploiting hierarchical models having transformer model, as it's backbone as we believe that regrouping our classes into different sets, would improve the results. also another promising idea was to build a byte pair encoding tokenizer trained on extracting frequent sub words, which may prove to be helpful especially in dialect classification. we believe that the main differences between dialects are specific sub words that occurs in certain dialect more than others. we will also experiment ensemble techniques and how they can fit into our system. ## conclusion we introduce a simple efficient single deep neural network model to classify 21 arabic dialects, which was based on the idea of fine-tuning approach using a pre-trained transformer model on similar domain after various cleaning and augmentation methods. we were able to achieve 4th best score in nadi shared task 1 competition with f1 macro average score of 22.03%.