Team A lexa at NADI Shared Task (2020)

Abstract: In this paper, we discuss our team’s work on the NADI Shared Task. The task requires classifying Arabic tweets among 21 dialects. We tested out different approaches, and the best one was the simplest one. Our best submission was using Multinational Naive Bayes (MNB) classifier (Small and Hsiao, 1985) with n-grams as features. Despite its simplicity, this classifier shows better results than complicated models such as BERT. Our best submitted score was 17% F1-score and 35% accuracy.

## introduction the interest of the research community concerning the arabic natural language processing (nlp) currently is focused on dialect identification at several levels, region level, country level, and provinces. most previous works focused on modern standard arabic (msa) @xcite because it is commonly used in formal writing between arab countries. many previous work on arabic dialect classification used a combination of n-gram both on word and char level with multinomial naive bayes such as @xcite @xcite . @xcite successfully applied svm. @xcite proposed a semi-supervised model with bert and obtained the top rank for madar twitter user dialect identification subtask in the madar shared task @xcite . madar corpus was the first large-scale resource built for arabic dialects @xcite . the nuanced arabic dialect identification (nadi) @xcite provided a labeled dataset consisting of arabic tweets with two subtasks: the first subtask is based on country-level dialect identification and the second subtask is province-level dialect identification. the dataset was challenging and the same dataset was used for both subtasks, even the algorithms that achieved high results in similar tasks did not gain satisfying results. in this paper, we focused on the first subtask. alexa model was built using a weighted ensemble model with n-grams features in the word and character levels. the ensemble model consists of the onevsrest classifier with mnb, mnb, and logistic regression. our model obtained a 17% f1-score and 35% accuracy; it ranks twelve based on f1 out of 18 participants. the remainder of the paper is organized as follows: data analysis will be shown in section 2. section 3 proposes a description of the alexa model. the results for subtask 1 are discussed in sections 4, and 5 respectively, followed by the conclusion in section 6. ## data the nadi shared task contains two subtasks; both use the same training and development data but differ in labels. the number of training, development, and testing datasets are shown in this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . ## system in this section, we propose our system (alexa model) that consists of multiple steps. in feature extraction level, we extract features from the tweets such as language models (n-grams) @xcite that assigns probabilities to a specific number of sequences of words or characters. there are many experiments done by combining different sets of language models on word level and character level @xcite and @xcite . on the word level, unigrams and bigrams (1,2) were the best, and on character level we end up with n grams range from 1 to 5 character with two types of tfidf vectorizers as shown in the figure 1 "char" and "char wb." "char wb" characters with the word boundaries. the features were weighted as follows: word-level unigram and bigram features weighted as 0.8, character-level tfidf vectorizer "char wb" weighted as 1.1 and tfidf vectorizer "char" weighted as 1.0. then, the extracted features were concatenated to train the ensemble model; the ensemble model is composed of one vs rest strategy with mnb @xcite , mnb, and logistic regression @xcite . the prediction form the three classifiers are summed to produce the one label for each tweet. the parameters used to train each classifier: when the assigned parameter in multi class is 'ovr', then the algorithm uses one-vs-rest (ovr) scheme. figure 1 : alexa model architecture ## results it is worth mentioning that using different pre-trained embedding models with this dataset did not perform well. we used bert @xcite multilingual model and aravec model @xcite to generate embeddings for the dataset. both of them achieved low results when using them as main features or as extra features. on the other hand, using these models for training outperformed the best submission on the task. we will talk about the model and the results in the post-evaluation section. ## discussion the provided dataset did not include modern standard arabic (msa) label in the training and development dataset, see figure 3 which contains wrongly classified tweets, the tweets in figure 3 are written in msa. figure 3 : examples of wrongly classified tweets our team investigated this problem; we used a model that was trained on detecting msa text to extract the wrongly labeled tweets in our dataset. to train this model, we used madar corpus @xcite because it contains msa tweets and other dialects. our used model achieves an accuracy score higher than the top submitted score on the madar shared task. such that, we assume that this model can accurately classify msa tweets. we also used different arabic datasets to see if our claim is true or not and we got similar results. based on our system, more than 20% of the training and development tweets were msa tweets. in ## conclusion in this paper, we tested different approaches in an arabic dialect classification task, nadi shared task. the best submitted score on the test phase was using an ensemble model that contains multinational naive bayes, onevsrest mnb, and logistic regression. for features, we used both countvectorizer and tfidfvectorizer features. our best submission achieved 17% f1-score. however, in the post-evaluation phase, we used an ensemble model which outperformed the best submitted score on the task. our ensemble model contained weighted concatenation between bert's probabilities and mnb probabilities. this model achieved 27.73% f1-score and 40.90% accuracy. we also noticed that many tweets were wrongly labeled because there were tweets written in msa, and there was no msa label. also, there are six countries with less than 240 tweets; this leads the model to never predict these dialects. for future work, we would like to overcome these issues by finding a way to deal with the imbalanced dataset and a way to overcome the msa problem.