Country-level A rabic Dialect Identification Using Small Datasets with Integrated Machine Learning Techniques and Deep Learning Models (2021)

Abstract: Arabic is characterised by a considerable number of varieties including spoken dialects. In this paper, we presented our models developed to participate in the NADI subtask 1.2 that requires building a system to distinguish between 21 country-level dialects. We investigated several classical machine learning approaches and deep learning models using small datasets. We examined an integration technique between two machine learning approaches. Additionally, we created dictionaries automatically based on Pointwise Mutual Information and labelled datasets, which enriched the feature space when training models. A semi-supervised learning approach was also examined and compared to other methods that exploit large unlabelled datasets, such as building pre-trained word embeddings. Our winning model was the Support Vector Machine with dictionary-based features and Pointwise Mutual Information values, achieving an 18.94% macros-average F1-score.

## introduction arabic dialect identification (adi) no doubt caught the attention of researchers @xcite . recently, many events and campaigns have been organised to handle adi and all associated challenges @xcite @xcite @xcite . dialectal arabic (da) started to appear online in text form with the rise of web 2.0, which has allowed users to generate online content, such as social media posts, blogs, discussion forums, and emails. however, the adi problem has become challenging due to a lack of data to represent a wide spectrum of arabic dialects and their complex taxonomy. in this paper, we examined the possibility of exploiting small datasets in order to build an adi system with solid performance. we experimented with an integration technique between two classical machine learning (ml) approaches. we also automatically created dictionaries based on pointwise mutual information (pmi) and labelled datasets. we exploited the automatically created dictionaries to enrich the feature space when training the models. a semi-supervised learning paradigm, namely, co-training was also designated to exploit the noisy and unlabelled data available hugely online. we also investigated another approach to leverage large unlabelled data, such as building pre-trained word embeddings to be used later in the training process. two deep learning (dl) models, the gated recurrent unit (gru) and the bidirectional long-short term memory (bilstm), were investigated along with a co-training method and word embeddings (either pre-trained, or learned during training). ## datasets the dataset used in this paper is the nadi corpus, consisting of 31,000 labelled tweets and encompassing 21 country-level arabic dialects. the dataset is divided into three parts: 68% for training (21,000 tweets), 16% for the development set (5,000 tweets), and 16% for the test set (5,000 tweets). the nadi shared task organisers also shared with participating teams around 10m unlabelled tweets. the text pre-processing steps are essential in any natural language processing (nlp) application and may affect positively or negatively the final outputs. the pre-processing pipeline we followed in our study before training our proposed models includes tokenising, the removal of stop words and one-character words, and cleaning the texts. the cleaning step involves the removal of diacritical marks, non-alphanumeric characters including emojis and excessive white spaces. ## feature extraction in this section, we illustrate the extracted features and the text representation of the input data before feeding them to the proposed models. ## system description this section presents the approaches we employed in our experiments to examine various traditional ml and dl models when working on country-level adi problem, covering 21 countries, with small datasets and various features. ## results we conducted several experiments in which we examined various classical ml and dl models to perform multi-way adi using small datasets. we also examined various features, as explained in section 3 and section 4. we started with character (2-5)grams and word (1-3)-grams as features to develop an svm model. we created dictionaries automatically based on pointwise mutual information and labelled datasets. including the automatically generated dictionaries in the feature space to train the svm model boosted the model's performance especially in terms of recall measure. the use of dictionaries alone as binary features increased the performance of the svm model, but adding the tweet's pmi values to the features set yielded the better svm performance. furthermore, we experimented with an integration approach between svm and lr, as well as set of features including dictionarybased features and pmi values. this approach outperformed the svm model in terms of precision, as expected, but fell behind slightly in terms of recall and f1-score. we also experimented with the ensemble method by combining the decisions of three individual classifiers: lr, svm, and rf. the ensemble classifier with dictionary-based features and tweet's pmi values surpassed the performance of all models in terms of precision, a predictable outcome as the ensemble classifier relied on hard voting. moreover, the bilstm and gru-based models fell short in comparison to classical ml techniques. the small dataset of 21,000 tweets may significantly affect the overall performance of the dl models. ## discussion and analysis we analysed the outputs of various models on the development dataset. we noticed that many mislabelled tweets are short in length and do not contain normalisation is one of the preprocessing steps when building nlp applications. for dialectal arabic normalisation is a vital step that should be taken into consideration according to the arabic nlp application's needs. in order to reduce the data sparsity and increase its accuracy, one should take normalisation of some arabic letters into account, especially when building pre-trained word embeddings or learning them during the training process. ## conclusion in this paper, we investigated the possibility of exploiting small datasets in order to build a strongly performing adi system, by integrating various machine learning approaches. we also created dictionaries automatically based on pointwise mutual information and labelled datasets, which enriched the feature space when training models. a semisupervised learning paradigm, namely co-training was also examined as a way to leverage the large and noisy unlabelled dialectal arabic datasets. another approach examined in this study to exploit large unlabelled datasets was building pre-trained word embeddings model to be used later in the training process. both approaches to leverage large and noisy unlabelled data showed similar results and performed on par. our winning model was the support vector machine with dictionary-based features and pmi values, achieving 18.94% macrosaverage f1-score on the nadi subtask 1.2 test set. in terms of precision, the integration method that merged two classical machine learning algorithms surpassed all models with precision equal to 22.70%. we also concluded that pre-processing text is a vital step before building any dialectal arabic models and needs to be further investigated to understand its side effects and benefits.