Multi-dialect A rabic BERT for Country-level Dialect Identification (2020)

Abstract: Arabic dialect identification is a complex problem for a number of inherent properties of the language itself. In this paper, we present the experiments conducted, and the models developed by our competing team, Mawdoo3 AI, along the way to achieving our winning solution to subtask 1 of the Nuanced Arabic Dialect Identification (NADI) shared task. The dialect identification subtask provides 21,000 country-level labeled tweets covering all 21 Arab countries. An unlabeled corpus of 10M tweets from the same domain is also presented by the competition organizers for optional use. Our winning solution itself came in the form of an ensemble of different training iterations of our pre-trained BERT model, which achieved a micro-averaged F1-score of 26.78% on the subtask at hand. We publicly release the pre-trained language model component of our winning solution under the name of Multi-dialect-Arabic-BERT model, for any interested researcher out there.

## introduction the term arabic language is better thought of as an umbrella term, under which it is possible to list hundreds of varieties of the language, some of which are not even mutually comprehensible. nonetheless, such varieties can be grouped together with varying levels of granularity, all of which correspond to the various ways the geographical extent of the arab world can be divided, albeit loosely. despite such diversity, up until recently, such varieties were strictly confined to the spoken domains, with modern standard arabic (msa) dominating the written forms of communication all over the arab world. however, with the advent of social media, an explosion of written content in said varieties have flooded the internet, attracting the attention and interest of the wide arabic nlp research community in the process. this is evident in the number of held workshops dedicated to the topic in the last few years. in this paper we present and discuss the strategies and experiments we conducted to achieve the first place in the nuanced arabic dialect identification (nadi) shared task 1 @xcite , which is dedicated to dialect identification at the country level. in section 2 we discuss related work. this is followed by section 3 in which we discuss the data used to develop our model. section 4 discusses the most significant models we tested and tried in our experiments. the details and results of said experiments can be found in section 5. the analysis and discussion of the results can be obtained in section 6 followed by our conclusions in section 7. ## related work the task of arabic dialect identification is challenging. this can be attributed to a number of reasons, including: a paucity of corpora dedicated to the topic, the lack of a standard orthography between and across the various dialects, and the nature of the language itself (e.g. its morphological richness among other peculiarities). to tackle these challenges, the arabic nlp community has come up with a number of responses. one response was the development of annotated corpora that focus primarily on dialectical data, such as the arabic on-line commentary dataset (zaidan and callison-burch, 2014), the madar arabic dialect corpus and lexicon @xcite , the arap-tweet corpus @xcite , in addition to a city-level dataset of arabic dialects that was curated by @xcite . another popular form of response is the organization of nlp workshops and shared tasks, which are solely dedicated to developing approaches and models that can detect and classify the use of arabic dialects in written text. one example is the madar shared task @xcite , which focuses on dialect detection at the level of arab countries and cities. the aforementioned efforts by the arabic nlp community, have resulted in a number of publications that explore the application of a variety of machine learning (ml) tools to the problem of dialect identification, with varying emphasis on feature engineering, ensemble methods, and the level of supervision involved @xcite @xcite . the past few years have also witnessed a number of published papers that explore the potential of deep learning (dl) models for dialect detection, starting with @xcite , who show the enhanced performance that can be brought about through the use of lstms and cnns, all the way to @xcite , who highlight the potential of pre-trained language models to achieve state of the art performance on the task of dialect detection. ## dataset the novel dataset of nadi shared task consists of around 31,000 labeled tweets covering the entirety of the 21 arab countries. additionally, the task presents an unlabeled corpus of 10m tweets. the labeled dataset is split into 21,000 examples for training, with the rest of the tweets, i.e., 10,000, distributed equally between the development and test sets. each tweet is annotated with a single country only. in figure 1 we can see the distribution of tweets per country in which egypt and bahrain has the highest and lowest tweet frequencies, respectively. we also note that the ratio of the development to train examples is generally similar across the various dialects, except for the ones with lowest frequencies. the unlabeled dataset is provided in the form a twitter crawling script, and the ids of 10m tweets, which in combination can be used to retrieve the text of these tweets. we are able to retrieve 97.7% of them, as the rest seem to be unavailable (possibly deleted since then or made private). this dataset can be beneficial in multiple ways, including building embedding models (e.g., word2vec, fasttext) @xcite , pre-training language models, or in semi-supervised learning and data augmentation techniques. this dataset can also be used to further pre-train an already existing language model, which can positively affect its performance on tasks derived from a domain similar to that of the 10m tweets, as we show in our results in section 6. ## system description in this section, we present the various approaches employed in our experiments, starting with the winning approach of our multi-dialect-arabic-bert model, followed by the rest of them. the results of our experiments are presented in ## discussion and analysis figure 2 : the confusion matrix of our best model on nadi development set to aid us with the analysis of the strengths and weaknesses of our winning model, we provide the confusion matrix for its performance on the nadi development set in figure 2 . the matrix highlights a number of issues stemming from the training dataset itself. for instance, it can be clearly seen that the model is biased to the countries with more training data such as egypt, iraq and saudi arabia; for these countries, the model achieves better results, while achieving much worse f1-scores for the ones with the least training data available. it can also be seen that the model suffers when trying to differentiate between geographically nearby countries. for example, 50% of the development samples from bahrain are labeled as uae and 22% from sudan are labeled as egypt. this is expected, given the similarities in dialects between neighbouring countries. some of the results shown in the confusion matrix have also led us to further investigate the datasets themselves. this resulted in finding that our model does in fact predict the correct class for certain tweets, which were somehow originally mislabeled. some of these examples can be seen in ## conclusion in this paper we describe our first place solution for the nadi competition, task 1. this was achieved via three stages: firstly, we further pre-trained a publicly released bert model (i.e., arabic-bert) on the 10 millions tweets supplied by the nadi competition organizers. secondly, we trained the resultant model on the nadi labelled data for task 1, multiple times, independently, with each of these iterations using a different mixture of maximum sentence length and learning rate. thirdly, we selected the 4 best performing iterations (based on their performance on the development dataset), and aggregated their softmax predictions via a simple element wise averaging function, to produce the final prediction for a given tweet. for future work, we would like to investigate other advanced pre-training methods, such as xlnet @xcite , and electra @xcite , which we believe might hold the key to better performance on this task.