A rabic Dialects Identification for All A rabic countries (2020)

Abstract: Arabic dialects are among of three main variant of Arabic language (Classical Arabic, modern standard Arabic and dialectal Arabic). It has many variants according to the country, city (provinces) or town. In this paper, several techniques with multiple algorithms are applied for Arabic dialects identification starting from removing noise till classification task using all Arabic countries as 21 classes. Three types of classifiers (Naïve Bayes, Logistic Regression, and Decision Tree) are combined using voting with two different methodologies. Also clustering technique is used for decreasing the noise that result from the existing of MSA tweets in the data set for training phase. The results of f-measure were 27.17, 41.34 and 52.38 for first methodology without clustering, second methodology without clustering, and second methodology with clustering, the used data set is NADI shared task data set.

## introduction arabic dialects is one of three variations of arabic language (classical arabic ca, modern standard arabic msa and dialectal arabic da). it is a native language of arabic people used in the communication among them and in the social media @xcite . each country of 21 countries, in the arab world, has its own dialect, sometimes they are written in same script with different pronunciation. recently, dialect identification (di) is interesting field in natural language processing (nlp) and conducted by number of researchers because it is increasing rapidly on the web and social media. there are nine distinct dialectal categories in arab world: egyptian, gulf, iraqi, levantine, maghrebi @xcite , yemeni, somali, sudanese and mauritania. each one of them has many varieties according to the city and town. it is clear that there are four levels of arabic dialectal identification (adi): (1) identification of dialectal arabic from msa and ca, it is very easy task similar to identification of arabic language among other languages, (2) identification of the main category of dialectal arabic of( nine or five categories), it is more difficult than previous point, (3) identification of country level out of 21 countries, it more difficult than the previous two points, (4) identification of city level or town level which is subfield of country level, it is the most difficult among all these levels. country native word country native word country native word iraq ‫-ﻛﺮوﯾﺖ‬ krwyt ‫-ﻗﻨﻔﺔ‬ qnfħ kuwait ‫-ﻛﻨﺒﮫ‬ knbh ‫-ﻗﻨﻔﮫ‬ qnfħ saudi arabia ‫-ﺑﺎطﺮﻣﮫ‬ batrmh ‫-ﻛﻨﺒﮫ‬ knbh bahrain ‫-ﻗﻨﻔﺔ‬ qnfħ lebanon ‫-ﻛﻨﺒﺎﯾﮫ‬ knbayh syria ‫-ﻛﻨﺒﺎﯾﮫ‬ knbayh egypt ‫-ﻛﻨﺒﮫ‬ knbh oman ‫-ﻗﻨﻔﺔ‬ qnfħ morocco & tunisia ‫-ﻓﻮطﻮي‬ fwtwy algeria ‫-ﻓﻮطﻮي‬ fwtwy palestine ‫-ﻛﻨﺒﺎﯾﮫ‬ knbayh yemen ‫-ﻛﻨﺒﮫ‬ knbh jordan ‫-ﻛﻨﺒﺎﯾﮫ‬ knbayh qatar ‫-ﻛﻨﺒﮫ‬ knbh united arab emirates ‫-اﻧﺘﺮﯾﮫ‬ antryh, ‫-ﻗﻨﻔﺔ‬ qnfħ ‫-ﻛﻨﺒﮫ‬ knbh there are many challenges in adi over identification of other types of arabic language such as: 1. in case of the levels 2,3 and 4, all the countries and cities use msa for writing in some times, therefore these will be noise in the dialectal identification. 2. existing of a city in a specific country use dialect that very close to other country more than its country such as basrah in iraq use dialect very close to kuwait dialect. 3. some countries are much similar in most words and different in little words therefore identification of their dialects are much difficult. this paper is a part of nadi 2020 shared task (subtask1) where the tweets in arabic dialect has been classified into the country belong it by voting among three classifiers (naïve bayes nb, logistic regression lr, and decision tree dt) in different methodologies to make final decision. also a preprocessing phase is done, before implementation of the classifier, such as noisy redundant, removing stopwords, feature extraction and feature selection. ## related work there are many works for identification of arabic dialects in all the levels. we chose some of the close works to our work. @xcite worked on nine dialects (tunisia, algeria, syria, lebanon, yemen, egypt, golf's countries, morocco and iraq) using the platform alize and gaussian mixture models (gmm). they showed the complexity of the automatic identification of arabic dialects. @xcite used a supervised approach for identification of arabic dialects. they got an accuracy of 85.5% on an arabic online-commentary. @xcite presented a multi-dialect, multi-genre, human annotated corpus of dialectal arabic with data obtained from both online commentary on the newspaper and twitter. they used five arabic dialects ( egyptian, levantine, gulf, maghrebi and iraqi). @xcite presented a set of experiments of letter-based (n-gram) markov language model and nb classifiers on social media. experimental results showed that nb classifier using character-level bigram model can identify the 18 different arabic dialects with a considerable accuracy. @xcite described a system to identify of four regional arabic dialects (egyptian, levantine, gulf, north african) and modern standard arabic (msa) in a transcribed speech corpus as a dsl shared task. they used ensemble classifier of set of linear models as base classifiers and they achieved a score of 0.51 in the closed training track. @xcite presented subtractive bivalency profiling (sbp) for identification of four arabic dialects( egyptian, levant , gulf , and north african) as well as msa where the accuracy were 76%. @xcite explored the use of different features (char, word n-gram, language model probabilities, etc) on different classifiers for arabic dialects identification. the work is part of multi arabic dialect applications and resources (madar) shared task @xcite in wanlp 2019 on arabic fine-grained dialect identification. they showed that traditional machine learning classifier tends to perform better when compared to neural network models in a low resource setting. @xcite presented a fine-grained dialect classification task covering 25 specific cities from across the arab world, in addition to standard arabic. they used several classification systems with large space of features. their results show that the exact city of a speaker can be identified at an accuracy of 67.9%. adi, in our work, is achieved by (i) identifying arabic language from other similar languages, (ii) identifying dialects from msa & ca, and (iii) identifying dialects among 21 arabic dialects. the final step is achieved by voting among three well-known and very different classifiers (nb, lr and dt) in two different methodologies. also, because the used data is none golden standard, little steps of noisy removal are done. ## data set the used data set is nadi tweeter data set @xcite . it consists of three parts training part of 21,000 tweets, development part of 4957 tweets and test set of 5,000 tweets. it is not golden standard corpus and has different noise levels such as existing of 405 non-arabic tweets (kurdish and persian). also, this data set is mixed of da, msa and ca. therefore a noisy removal should be taken as preprocessing. the data sets are labeled in two levels; first level (country level) of 21 coun-tries and second level (provinces level) of 100 provinces. ## our system our system consist of five phases: (i) preprocessing, (ii) noisy tweets removal, (iii) formal clitics and stop words removal, (iv) features extraction and selection, and (v) the classification. figure 1 explained the proposed method. they will be explained in the next few sections. ## discussion in this paper, adi is implemented and applied on nadi shared task dataset of very close 21 classes. the proposed system starts from removing noise till the classification process for 21 classes of all arabic countries. three classifiers were combined and used in two methodologies (one classifier for classification of 21 classes or using twenty one binary classifiers). selecting the classes iraq, egypt and egypt classes for foreign, msa and unknown tweets respectively dropped down the macro-average score but really foreign and msa are noise. the results of classification were very low for many reasons: (i) existing of 504 non-arabic tweets in training set, (ii) existing of msa tweets which is used in all the arabic countries result in high noise in learning phase, (iii) some dialects are close to each other's, (iv) existing of ambiguous tweets where it written as msa but can be pronounced in many dialects, (v) existing of a city in a country that used dialect close to other country more than its country, (vi) the tweets were classified, in the data set, according to user location but not user dialect which produce errors in the classes of training set and hence the wrong learning. the result of using twenty two binary classifiers with clustering (removing msa from training data) gave us the best results because the system will focus on the dialect of each country in the training process without noise tweets. ## conclusions and future works arabic dialects are native languages for each country or city in arab world where sometimes the writing of dialects is same but the pronunciations are different. we can see four levels of arabic dialectal identification (adi) from the easiest task to the most difficult task. the first level is identification of dialectal arabic from the other two arabic language varieties. the second level is identification of the main category of dialectal arabic of (nine or five categories). the third level is identification of country level out of 21 countries. the fourth level is identification of city dialects or town dialects. there are many challenges in adi according to the level number. simply we can conclude that adi is a hard task for many reasons as was mentioned in discussion section. the task need to golden standard corpus and a dictionary for almost all words used in each dialect. the learning from low noise data set gives a good results but the task is still need to special learning techniques and huge dataset (golden standard).