A rabic Dialect Identification and Sentiment Classification using Transformer-based Models (2022)

Abstract: In this paper, we present two deep learning approaches that are based on AraBERT, submitted to the Nuanced Arabic Dialect Identification (NADI) shared task of the Seventh Workshop for Arabic Natural Language Processing (WANLP 2022). NADI consists of two main sub-tasks, mainly country-level dialect and sentiment identification for dialectical Arabic. We present one system per sub-task. The first system is a multi-task learning model that consists of a shared AraBERT encoder with three task-specific classification layers. This model is trained to jointly learn the country-level dialect of the tweet as well as the region-level and area-level dialects. The second system is a distilled model of an ensemble of models trained using K-fold cross-validation. Each model in the ensemble consists of an AraBERT model and a classifier, fine-tuned on (K-1) folds of the training set. Our team Pythoneers achieved rank 6 on the first test set of the first sub-task, rank 9 on the second test set of the first sub-task, and rank 4 on the test set of the second sub-task.

## introduction arabic is the official language of 22 countries, recognized as the 4th most used language on the internet @xcite . arabic can be classified into three types @xcite , mainly classical arabic (ca), modern standard arabic (msa), and arabic dialects (ad). unlike both ca and msa, arabic dialects lack a standardized representation and data that cover their complex taxonomy. several initiatives were made to advance the research in this field. one of the most prominent work has been carried out through the nuanced arabic dialect identification (nadi) shared tasks. the first two nadi shared tasks @xcite comprised country-level and province-level dialect identification. many participants presented their systems to the nadi shared tasks. most of the systems submitted rely on the bidirectional encoder representation from transformers (bert) @xcite models. for instance, @xcite pretrained a multilingual bert model on unlabeled arabic tweets, then fine-tuned the model for the dialect classification task. furthermore, @xcite fine-tuned the transformer-based model for arabic language understanding arabert (antoun et al.) on an extended corpus constructed using a reverse translation of the given arabic nadi dataset. @xcite employed arabic-bert @xcite alongside semisupervised learning and ensembling methods in their system. @xcite introduced an ensemble that applies a weighted voting technique on two classifiers, the first based on tf-idf with word and character n-grams and the second based on arabert. @xcite proposed a multi-task model that leverages marbert's contextualized word embedding @xcite with two task-specific attention layers, aggregated to predict both the province and the country of a given arabic tweet. the nadi 2022 shared task (abdul-mageed et al., 2022) provides two sub-tasks, mainly country-level dialect identification and sentiment analysis. inspired by the previous submissions, we fine-tune arabert for each sub-task. the system for the first sub-task is a multi-task model that performs dialect identification by predicting the region, area, and country of the tweet. the system for the second sub-task is a distilled model from an ensemble of k models that were trained using k-fold cross-validation for sentiment classification. this paper is structured as follows: section 2 describes the data used. section 3 gives an overview of fine-tuning bert models. section 4 presents the systems submitted to subtasks 1 and 2 respectively. we show the results on the nadi subtasks 1 and 2 and discuss them in sections 5 and 6. we conclude with section 7. 2 data ## fine-tuning bert as mentioned in the previous sections, the two subtasks fall under the category of text classification. an intuitive solution would be to fine-tune a pretrained language model on each sub-task by adding an output layer to the encoder and training the parameters of the network to predict the classes for the subtask. fine-tuning is a form of transfer learning, as it tailors the knowledge encoded in the model to the downstream task. therefore, it is crucial to find an appropriate model to fine-tune. after investigating multiple bert variants, we choose to use an arabic pre-trained language model called arabert @xcite . arabert is trained on a huge corpus of arabic text from a collection of publicly available large-scale raw arabic text. the specific model employed in both subtasks is the bert-large-arabertv02-twitter. it is based on arabertv0.2-large, and it is pre-trained using the masked language modeling task on 60m multi-dialect tweets. however, fine-tuning a bert variant might not be sufficient to reach the desired performance on the sub-tasks. therefore, our contribution lies in employing multi-task learning for the first sub-task, and knowledge distillation from an ensemble of models for the second sub-task. all models have been trained on nvidia tesla volta v100. ## results we evaluate our systems on the validation set provided by the organizers. ## discussion as we can notice, the simple task of predicting whether the dialect is western or eastern is challenging by itself. this clearly confirms that the task of dialect identification is not an easy task. furthermore, we notice that the model has trouble distinguishing between the levantine dialect and the peninsular gulf dialect. this is expected as these dialects are the most similar among all four families (area). moreover, we notice that the confusion between dialects within the same area is higher compared to dialects from different areas (highlighted in figure 5 by the clusters of values in red and green). this is expected as the training process injected knowledge that helps the model distinguish between the dialect classes (i.e., regions and areas). therefore, a more fine-grained region-level and area-level classification should result in an improvement to the country-level dialect identification task. we can also note the discrepancy in the performance of the model between test-a and test-b. in fact, test-a tests the model's performance on all the dialects, while test-b tests the performance on a subset of k dialects. test-b does not reflect the model's performance on all dialects, as the model might be tested on country-level dialects that are more difficult to predict. as for subtask 2, we can see that the macro-averaged f1 score reported on the test set is higher than the score reported on the development set. this implies that distilling an ensemble of k models trained on different partitions of the training set helped the model generalize well on unseen data. ## conclusion in this paper, we introduced two arabert-based systems to tackle dialect and sentiment classification. we conclude by confirming that dealing with arabic dialect data is quite challenging. in future work, we propose to vary the training approach for every individual model in the ensemble, by changing the sequence length used, or even the training batch size per model. we also propose to build an ensemble of k multi-task models for subtask 1.