LTG - ST at NADI Shared Task 1: A rabic Dialect Identification using a Stacking Classifier (2020)

Abstract: This paper presents our results for the Nuanced Arabic Dialect Identification (NADI) shared task of the Fifth Workshop for Arabic Natural Language Processing (WANLP 2020). We participated in the first sub-task for country-level Arabic dialect identification covering 21 Arab countries. Our contribution is based on a stacking classifier using Multinomial Naive Bayes, Linear SVC, and Logistic Regression classifiers as estimators; followed by a Logistic Regression as final estimator. Despite the fact that the results on the test set were low, with a macro F1 of 17.71, we were able to show that a simple approach can achieve comparable results to more sophisticated solutions. Moreover, the insights of our error analysis, and of the corpus content in general, can be used to develop and improve future systems.

## introduction most resources for arabic have been developed for modern standard arabic (msa) since it is the official language in most arabic speaking countries. msa is used in media coverage, politics, books, and even online. however, in each individual arabic country, the predominant language used for everyday conversation (in real life and online) is a specific dialect for that region or country @xcite . arabic dialects are not standardized. there are no formal grammar rules nor formalism to guide the speakers @xcite . there are some efforts on creating standards for automatically processing such dialects @xcite , but the language use remains non-standardized. within one same city, people can pronounce and write the same word differently. this aspect accentuates the difficulty of automatically processing such languages, and automatically distinguishing them from each other is as challenging, since no clear structure exists. nevertheless, many attempts have been made for arabic dialect identification. despite the difficulty of the task, there are still syntactic and morphological aspects of the languages that can be exploited to differentiate them from each other. most works rely on machine learning approaches, and span various levels of accuracy depending on the dataset used and dialects being processed. the nearer (geographically) the countries are to each other, the more similar the spoken language, and therefore the more difficult it is to automatically distinguish the dialects @xcite . the multi arabic dialect applications and resources (madar) corpus @xcite is an important resource for arabic dialect identification. the corpus covers parallel sentences written in 25 arabic city dialects from the travel domain. this corpus has been used in a shared task @xcite for both fine-grained (26 dialects) and coarse-grained (6 dialects) arabic dialect identification where various machine learning approaches have been used. a simple mulitnomial naive bayes (mnb) has shown to be very powerful in the identification of the exact city of 26 dialects from the madar corpus with an accuracy of 67.9%, using as features character and word 5-grams language models and the output of the coarse-grained classifier @xcite . other approaches also focused on the use of machine learning and ensemble methods, using as features word counts, language models, and embeddings @xcite @xcite . in this paper, we present our participation in the nuanced arabic dialect identification (nadi) shared task @xcite . we participated in the first sub-task aiming at country-level arabic dialect identification from tweets covering 21 arab countries. our contribution is based on a stacking classifier using multinomial naive bayes, linear svc, and logistic regression classifiers as estimators; and logistic regression as final estimator. we have experimented with various architectures, from traditional machine learning approaches as the approach presented in this paper and clustering, but also with more recent approaches as bi-lstms and cnns. we have also experimented using unlabeled tweets (provided by the shared task organizers) to train embeddings and language models. however, none of these architectures gave satisfying results. in section 2 we describe the nadi tweet corpus. we present our proposed model in section 3, and describe our results and give an overall discussion of the results in section 4. finally, we conclude in section 5 and discuss possible future work. ## data we used the nadi corpus provided by the shared task organizers @xcite . the corpus comprises 21,000 tweets covering 21 arab countries for the first sub-task, and 100 provinces from the same arab countries (as sub-task 1) in the second sub-task. in the following, we will only focus on the data set for the country-level classification. each tweet in the corpus is annotated with its associated country (i.e. dialect) label. the dialects of the following countries are included in the nadi corpus: algeria, bahrain, djibouti, egypt, iraq, jordan, kuwait, lebanon, libya, mauritania, morocco, oman, palestine, qatar, saudi arabia, somalia, sudan, syria, tunisia, united arab emirates, and yemen. despite the relatively large amount of tweets used for training, the corpus is extremely unbalanced, which we believe has made this task even more difficult. as can be seen in figure 1 , the egyptian dialect on it's own represents more than 21% of the training data, while the dialects from djibouti, bahrain, sudan, mauritania, and somalia only represent 1% of the training data. a further analysis of the corpus also showed that for all dialects except egypt and iraq, most of the word types were not representative of the language. most vocabulary items are shared between dialects, which might confuse any classification system. in figure 2 we show that the lightest (at the bottom) color represents the normalized number of unique words for each dialect. the color in the middle represents the normalized number of words shared between each dialect and up to four other dialects. the darkest color (at the top) represents the normalized number of words shared between each dialect with more than four other dialects. from figure 2 it is evident that 58% of the word types in the egyptian and iraqi tweets are unique to these dialects. however, for the remaining 19 dialects, over 50% of the word types were shared with other dialects. our preliminary investigations have shown that many of these shared vocabulary items were actually msa words. it is important to note that many dialectal words are shared with msa. for example the words respectively enters, joy, your age, the club, more beautiful, madrid, i believe, feelings in english, were shared by more than 10 dialects. another example is the word (there) which in the corpus is uniquely present in tweets labeled as libyan, actually exists also in other forms in the corpus. when written , it is only present in tweets from iraq and morocco, while when written in its most basic form without vowels, , it is present in tweets labeled with the 17 dialects from algeria, djibouti, egypt, iraq, jordan, lebanon, libya, morocco, oman, qatar, saudi arabia, somalia, sudan, syria, tunisia, united arab emirates, and yemen. this also plays a role in the difficulty of the task. as most dialects share their vocabulary, the words distinguishing them from each others might actually not be that different from common shared words. therefore distinguishing the dialects from each other gets more challenging. ## system our model, as shown in figure 3 , is based on a stacking classifier. we first train three different models on the nadi train split, and use their respective predictions to train a final estimator to get our final predictions. stacking allows us to use the strengths of each individual classifier by using their output as input of a final estimator. all of the experiments are implemented using python and the scikitlearn library @xcite . our model classifies tweets into the nadi 21 dialects. it uses a combination of word and character ngrams and skipgrams concatenated using feature union estimator in sklearn. we give different weights to each feature vector, which were selected after a thorough analysis by experimenting with various possible weights (and combination of weights) using grid search. we used the following features: figure 3 : model architecture. our model is a stacking classifier. first, three different classifiers are trained on the nadi train split. their respective predictions are thereafter used with a final estimator to get final predictions. ## results and discussion we report in we believe that the main issue is the unbalanced nature of the dataset. we have experimented with various approaches to boost the performance. we used oversampling and under-sampling, as well as balanced sampling, but none of these gave satisfying results. as can be seen in figure 4 our model achieves high scores when predicting the egyptian, saudi arabian, algerian, and iraqi dialects. from the confusion matrix it is also apparent that most dialects were miss-classified as these four dialects. we believe that these are partly due to the amount of tweets of each dialect present in the training set, as these represent the top four most frequent dialects in the train set (see figure 1 ). to support this hypothesis, we trained our model only on tweets from these four dialects and achieved an f1-score of over 60%. further analysis of the output have shown that our model performs very poorly on the less frequent dialects. our model is not able to correctly predict the dialects qatar, bahrain, djibouti, and kuwait. once again, we believe that this is due to the skewed nature of the data. but also, the difficulty to differentiate between the dialects in general. the individual precision and recall scores (see a better performing system would have performed on the entire data set as our system performs on the egyptian dialect: with high precision and high recall. a model that returns multiple predictions, with most predictions correctly labeled. we think that such a system might require a more diversified data set, with less overlap in the vocabulary. moreover, a close analysis of the tweets themselves revealed that many tweets labeled as dialects were actually written in (mostly) msa, which we believe could have further skewed the classifications. as an example, consider the sentences (1) and ( 2 ). these are labelled as algerian in the training data, despite being both written in msa. we did not carry an exhaustive analysis on the amount of tweets that actually were written in msa despite being labeled as algerian, but from our preliminary analysis, it seems that many were either completely written in msa, or mostly used msa words. this might be partly due to which region the person tweeting is from, as people from different regions and backgrounds might prefer to write in msa, or prefer to reach a broader international audience. admittedly, these sentences are generic and could in theory be present in all dialects. but it is exactly these type of sentences that can complicate the classification task. social media platforms can be a gold mine for arabic dialect identification, and having these type of sentences in a corpus is inevitable. we nevertheless believe that filtering these out, or even classifying them as msa, might increase the performance of dialect identification systems. ## conclusion we proposed a model for automatically classifying 21 arabic dialects using a corpus of tweets. our system uses a stacking classifier relying on tf-idf word and character features, and the three classifiers multinomial naive bayes, linear svc, and logistic regression. our model did not achieve high scores on all dialects, but we believe that it shows that simple approaches can achieve satisfying results even when using heavily unbalanced data. we have experimented with more advanced deep learning approaches, but were not able to achieve satisfying results. we also believe that the poor results achieved reflect the data we have used. the heavy unbalance in the data has added an extra layer of difficulties to the task of dialect identification and we think that more balanced data can help us develop better models that can achieve higher accuracy. however, the insights we have gained during the analysis of the train and dev data sets and our results, have enabled us to understand more what are the typical issues encountered during arabic dialect identification, and we believe that these insights can be used to develop and improve future systems. we think that a more systematic approach based on contextualized word embeddings as bert @xcite combined with sequence labeling approaches can give better results and achieve higher scores, and aim to go in this direction in future work.