Weighted combination of BERT and N- GRAM features for Nuanced A rabic Dialect Identification (2020)

Abstract: Around the Arab world, different Arabic dialects are spoken by more than 300M persons, and are increasingly popular in social media texts. However, Arabic dialects are considered to be low-resource languages, limiting the development of machine-learning based systems for these dialects. In this paper, we investigate the Arabic dialect identification task, from two perspectives: country-level dialect identification from 21 Arab countries, and province-level dialect identification from 100 provinces. We introduce an unified pipeline of state-of-the-art models, that can handle the two subtasks. Our experimental studies applied to the NADI shared task, show promising results both at the country-level (F1-score of 25.99%) and the province-level (F1-score of 6.39%), and thus allow us to be ranked 2nd for the country-level subtask, and 1st in the province-level subtask.

## introduction language identification is considered an important task in natural language processing (nlp), as it helps personalizing applications, automatically detecting the source variety of a given text or speech segment, and collecting/tagging the data @xcite . in the case of arabic language, the official language of over 20 countries, and with more than 360 million native speakers, this task becomes very challenging due to the different language variations (dialectal arabic), and the complex taxonomy of arabic language @xcite . in this paper, we consider the arabic dialect identification task from two perspectives: country-level dialect identification and province-level dialect identification. in the case of the previous multi arabic dialect applications and resources (madar) shared task @xcite , several approaches have been proposed @xcite . @xcite developed country-level identification models based on bidirectional encoder representations from transformers (bert) @xcite , and gated recurrent units @xcite . they ranked 1st in the subtask aiming at classifying tweets with 71.70% macro f1-score and 77.40% accuracy. @xcite investigated various feature extraction methods, such as term frequency-inverse document frequency (tf-idf) and word embedding, in order to improve the model performances. the best results, 69.86% f1-score and 76.20% accuracy, were obtained using a simple linear support vector classification (linearsvc) classifier with a user voting mechanism. the main contribution of this paper is the introduction of a novel approach based on a pipeline of stateof-the-art models for both sub-tasks of nadi shared task @xcite . for countrylevel identification, we build a system based on raw tweets. the core of this system is an ensemble model that applies a weighted voting technique on two classifiers, namely m_ngram and m_bert (will be used as labels for our models on the rest of this paper, the m stands for model). m_ngram uses tf-idf with word and character n-grams to represent tweets. a stochastic gradient descent (sgd) classifier is then trained to optimize the huber loss @xcite . m_bert fine-tunes arabert weights (antoun et al., 2020) with a softmax classifier trained to optimize the multi-class entropy loss. for province-level identification, we build a hierarchical classification, by first performing the country-level identification, and then fine-tuning for each identified country a m_bert model to predict its provinces. figure 1 illustrates the overall architecture of the proposed solution. the proposed system generates f1-scores of 25.99% and 6.39% for the country-level identification and province-level identification, respectively. the rest of this paper is organized as follows. section 2 describes the nadi shared task dataset. section 3 describes the proposed approaches, models and our data preparation. section 4 presents experimental results, error analysis and discussion. finally, the conclusion is given in section 5. ## dataset presentation the nuanced arabic dialect identification (nadi) shared task is the first task focusing on naturallyoccurring fine-grained dialect. it has been divided into two subtasks: 1) the country-level identification subtask and 2) the province-level identification subtask. the organizers of the shared task provide four sets of collected tweets: the train set (21k), the development set (4,957), the test set (5,000), and the unlabeled tweets set (10m). as we can see in figure 2 , the nadi task is quite challenging due to the unbalanced distribution of tweets (as example a very low frequency of tweets for djibouti (dj) and bahrain (bh) while the number of egypt (eg) tweets is very high) and the nuance between arabic dialects. ## methods in this section, we review the data preparation pipeline and the proposed models, namely m_ngram and m_bert, used to build our ensemble model. ## experimental results, error analysis and discussion in this section, we conduct experiments to evaluate the performance of the proposed models m_bert, m_ngram, and ensemble model on the development set. ## conclusion in this paper, we described our submission to the nadi shared task. we built a system composed of two classifiers: ensemble model for country-level identification, and a hierarchical classifier for provincelevel identification. the quote "alone we are strong, together we are stronger." has been verified: our ensemble model in subtask 1 increased significantly our f1-score to 27.24% on development set and 25.99% on the test set, allowing us to rank second in the competition. in subtask 2 the hierarchical classifier achieved 6.39% f1-score and ranked 1st. this work has shown that the combination of neural network-based features (bert) with statistical features (tf-idf) might increase the performance in other nlp tasks.