Simple vs Oversampling-based Classification Methods for Fine Grained A rabic Dialect Identification in T witter (2020)

Abstract: In this paper, we present a description of our experiments on country-level Arabic dialect identification. A comparison study between a set of classifiers has been carried out. The best results were achieved using the Linear Support Vector Classification (LSVC) model by applying a Random Over Sampling (ROS) process yielding an F1-score of 18.74% in the post-evaluation phase. In the evaluation phase, our best submitted system has achieved an F1-score of 18.27%, very close to the average F1-score (18.80%) obtained for all the submitted systems.

## introduction fine grained arabic dialect identification is a very challenging topic due, in one hand, to the rarity of dialectal resources, and in the other hand, to the inter-closeness of arabic dialects spoken by twenty two countries, as well as the intra-closeness of each sub-dialect spoken in many provinces of a country. one of the most recent scientific events dedicated to arabic fine grained dialect identification was organized in the framework of the fourth workshop for arabic natural language processing, madar'2019 @xcite , where a couple of works have been proposed. indeed, in @xcite , the authors proposed a system based on extracting and applying a feature union process on a set of features (tf-idf word n-grams, tf-idf character n-grams, tf-idf character with boundary n-grams and tf-idf skip n-grams) multiplied by a transformation weight. in fact, they applied empirical experiments to find the best values of "n" (n-grams) as well as the transformation weights. other features have been added, like sentence length ratio for every sentence in the data. to classify these features, the authors used an ensemble hard voting classifier with three ml algorithms (linear support vector classifier "lsvc", multinomial naive bayes "mnb" and bernoulli naive bayes "bnb"). they obtained a macro f-score of 67.32%. based on the work done by @xcite , @xcite have suggested an ensemble of 5 classifiers (mnb, svc, bnb, k-nearest-neighbours "knn" and a weak dummy classifier based on prior probabilities of each dialect), in addition to 96 language models on word and char-level trained using kenlm @xcite from moses, with default parameters. this setup has given an f1-score of 66.7%. in a more refined experiment, @xcite presented a very large scale dataset covering 319 cities from all 21 arab countries. they also introduced a hierarchical attention multi-task learning (ha-mtl) approach for dialect identification exploiting the data at the city, state, and country levels. they also evaluated the use of bert on the three tasks, while comparing it to the mtl approach. more recently, @xcite built a new corpus and called it qadi (18 countries). they experimented the arabert model @xcite on both qadi and madar corpus (25 cities + msa). the score they achieved was 60.6% and 29% on qadi and madar, respectively. in this paper, we present our contribution based on our system presented in @xcite and @xcite to solve the challenge introduced in the first subtask, i.e. identifying 21 arabic countries' dialects. in section 2, a description of the used dataset is presented. the applied cleaning steps and preprocessing are explained in section 3. in sections 4 and 5, we exposed the two adopted approaches (both simple and oversampling-based classification), respectively. finally, the findings and discussion are presented in section 6. ## description of the dataset in this section, we describe the dataset used in our current (i.e., nadi) shared task @xcite . this dataset is collected from 21 arab countries from the twitter domain, covering a total of 100 provinces. the dataset is divided to three sets: train, dev and test, for which we addressed, in the minimum number of words and characters per sentence in the training and development sets are 1 and 3, respectively. this information was useful to determine the number "n" of grams to be selected in the features extraction phase. in figure 1, we note that the numbers of words for the 21 dialects are differential in training dataset. as an example, we mention the case of saudi arabia versus sudan or iraq versus libya. we also note that the ratio of the numbers of words to the number of sentences is more considerable for some dialects as egyptian, iraqi and algeria. ## data cleaning and preprocessing dealing with texts collected from twitter necessitates a cleaning process before doing analysis and processing. hence we have applied two simple cleaning steps: for preprocessing, we explored the following steps: punctuation removal, normalization of arabic letters, stop words removal, lemmatization, stemming, and part of speech tagging. for the lemmatization process we used farasa foot_0 developed by @xcite , whereas for stemming and part of speech tagging, we used isri arabic stemmer and postagger of nltk foot_1 @xcite . to investigate the impact of these six text functionalities, we combined them in different ways (see tables 2 and 3). we used different combinations of n-grams as features which resulted in 200 experiments. nevertheless, we kept those which yielded the best performance, namely: 3-grams word, 5-grams word, 3-grams char, 5-grams char, 3-grams char wb, 5-grams char wb (see tables 2 and 3). these features have been transformed using the sklearn tfidf vectorizer @xcite as presented in @xcite . ## simple classification model this classification model is simply based on three linear classifiers, namely: linear support vector classification (lsvc), ridge classifier (rdg) and stochastic gradient descent (sgd). the parameters we used for these classifiers are as follows: linear svc (@xmath0.01, @xmath1, @xmath2, ridge classifier (@xmath3, @xmath4.0001, max @xmath5, @xmath6. so as to avoid overfitting, we have applied a k-fold cross validation (@xmath7. corpus preprocessing features extraction combination simple classification oversampling based classification lsvc rdg sgd label lsvc rdg sgd ensemble ensemble figure 2: classification system architecture of arabic dialects. ## oversampling based classification since the dataset used in this task is imbalanced, as can be noticed clearly in figure 1 , we used oversampling techniques to solve this problem @xcite . in the following, we briefly describe three different techniques that we used for oversampling: random over-sampler (ros): this can be achieved by simply duplicating random examples from the minority class in the training dataset prior to fitting a model. this can balance the class distribution but does not provide any additional information to the model @xcite . ## results and discussion after conducting more than 200 empirical experiments, in which we used different combinations of n-grams, we have reported the outputs of our 8 best configurations, as mentioned in tables 2 and 3, performed by the lsvc, rdg and sgd classifiers, in addition to the ensemble classifier. note that the oversampling-based method outperformed the simple classification model (f1-@xmath8.74%). the impact of applying cross validation procedure k-cv (@xmath9. we should ## conclusion in this paper, we presented a description of our system for identifying arabic dialects from twitter collected from 21 arabic countries. we have performed an empirical comparison study where we conducted 200 experiments in which we used several combinations of features and preprocessing steps, as stemming, lemmatization and part of speech tagging. we used a simple classification model comprising three classifiers: lsvc, rdg and sgd, while we tried to resolve the problem of imbalanced data using the oversampling methods. as expected, using a small and imbalanced dataset didn't help to get high scores, however, this task can be seen as the first try to identify high number of dialects with few resources.