Comparison between Voting Classifier and Deep Learning methods for A rabic Dialect Identification (2020)

Abstract: In this paper, we present three methods developed for the NADI shared task on Arabic Dialect Identification for tweets. The first and the second method use respectively a machine learning model based on a Voting Classifier with words and character level features and a deep learning model at word level. The third method uses only character-level features. We explored different text representation such as Tf-idf (first model) and word embeddings (second model). The Voting Classifier was the most powerful prediction model, achieving the best macro-average F1 score of 18.8% and an accuracy of 36.54% on the official test. Our model ranked 9 on the challenge and in conclusion we propose some ideas to improve its results.

## introduction the arabic language is one of the most widely spoken language in the world, currently considered as the fifth language @xcite with more than 330 million arabic speakers. it is the official language of more than 22 countries. in its written form, commonly referred as literary arabic, it usually is divided into two categories: classical arabic and modern standard arabic (msa). however, in their daily life arabic speakers mostly use dialects which are a linguistic variant of classical arabic with their own features, varying with respect to the country or the region. one can say that msa is only used for written and official communication while dialects are used for oral communication as well as for many device mediated communication forms: email, sms, chat or blogs. therefore, arabic dialect identification (did) has become a very important data preparation step that attracts many attention in nlp research. indeed, the knowledge about the dialect of an input text is useful in several nlp tasks such as sentiment analysis @xcite , machine translation or document classification. this paper describes our submission to the nadi shared task on arabic fine-grained dialect identification cover a total of 100 provinces from all 21 arab countries @xcite . two subtasks were proposed: our submission is dealing with the first subtask. in section 2, we describe the dataset and in section 3 we present our three methods. in section 4, we detail our results and give future directions in section 5. ## dataset arabic has a widely varying collection of dialects. many of these dialects remain understudied due to rarity of resources. the dataset used in this work is made up of a collection of tweets shared by users in different arab countries. this dataset is collected from 21 countries which are algeria, bahrain, djibouti, egypt, iraq, jordan, kuwait, lebanon, libya, mauritania, morocco, oman, palestine, qatar, saudi-arabia, somalia, sudan, syria, tunisia, united-arab-emirates, yemen. the dataset for the shared task contains 25,957 tweets divided in training, development and test set as shown in the distribution of tweets among countries is very unbalanced. the egyptian dialect is best represented in te dataset as shown in figure 1 . around 21% (5543 tweets) of the tweets belong to egypt while only 0.8% belong to bahrain (218 tweets) and djibouti ( 220 tweets 3 data preparation and classification methods ## results and discussion the results obtained by our different models for the test set with and without pre-processing are presented in there are dialects which are not easy to detect even by an arabic speakers as they do not contain any clue words, like mauritania and djibouti. these tweets are often written in standard arabic. figure 3 shows that our models are not able to correctly classify these tweets. bahrain, djibouti dialects were the most difficult to detect : we had no true positive (tp). finally, we think that the class imbalance in our data set had a direct influence on the prediction of the less presented class in train data with a tp rate close to zero and fn rate close to 100% as shown in the figure 3 ("sudan", "somalia", "qatar", etc). ## conclusion in this paper, we described our system submitted to nadi shared task on country level dialect identification from twitter data. we presented three methods for this shared task. experimental results show that voting classifier was the most powerful prediction model, achieving the best macro-average f1 score than other machine learning and deep learning models. despite the fact that there was a substantial amount of incorrectly annotated tweets in the dataset, we were still able to achieve an f1 score of 17.69% on the development set, 16.06% on the test set and 18.8% on the test set trained on the train and the development set. it would be interesting to correct the errors in training data in order to see the impact on the results. another perspective would be to evaluate our model with other tokenizers like "madamira" @xcite and to perform a deeper analysis of classification errors.