Building an Ensemble of Transformer Models for A rabic Dialect Classification and Sentiment Analysis (2022)

Abstract: In this paper, we describe the approaches we developed for the Nuanced Arabic Dialect Identification (NADI) 2022 shared task, which consists of two subtasks: the identification of country-level Arabic dialects and sentiment analysis. Our team, UniManc, developed approaches to the two subtasks which are underpinned by the same model: a pre-trained MARBERT language model. For Subtask 1, we applied undersampling to create versions of the training data with a balanced distribution across classes. For Subtask 2, we further trained the original MARBERT model for the masked language modelling objective using a NADI-provided dataset of unlabelled Arabic tweets. For each of the subtasks, a MARBERT model was fine-tuned for sequence classification, using different values for hyperparameters such as seed and learning rate. This resulted in multiple model variants, which formed the basis of an ensemble model for each subtask. Based on the official NADI evaluation, our ensemble model obtained a macro-F1-score of 26.863, ranking second overall in the first subtask. In the second subtask, our ensemble model also ranked second, obtaining a macro-F1-PN score (macro-averaged F1-score over the Positive and Negative classes) of 73.544.

## introduction there are approximately 400 million arabic speakers worldwide, spread geographically in 22 countries around the world @xcite . with early manifestations of arabic dating back to the 8 th century bce, the arabic language has been redefined and refined over many decades across different continents. many scholars struggled to define arabic as a single language, with many considering classical arabic (ca)-the language of the quran-as the ideal archetype. in modern times, modern standard arabic (msa) has been used in most official publications, broadcasts, political speeches, and written texts. however, most people use spoken varieties of arabic in their daily lives. some of these spoken varieties differ from each other significantly and are almost mutually unintelligible, whilst others bear strong similarities. these spoken variations of arabic are commonly referred to as dialectical arabic (da). thus far, the majority of the research in arabic natural language processing (nlp) has overlooked the variations across the different arabic dialects @xcite , largely due to the lack of datasets that take the different da types into consideration. the goal of the nuanced arabic dialect identification (nadi) shared task series is to diminish this research gap, by providing datasets where examples are organised according to dialects @xcite @xcite . as part of the nadi 2022 shared task, organisers made available datasets that support two sub-tasks, namely, dialect identification (subtask 1) and sentiment analysis of country-level dialectical arabic (subtask 2). recent advancements in nlp research have led to the development of transformer-based language models which learn contextual embedding representations of sequences, and which have been shown to obtain state-of-the-art performance on many nlp tasks @xcite @xcite . marbert @xcite ) is a language model that was pre-trained specifically on da, and formed the basis of our approach to the nadi 2022 shared tasks. the organisers prepared a dataset of labelled tweets covering 18 arab countries for the dialect identification subtask. it was split into training, development, and two test sets. whilst the first test set (test-a) covers 18 country-level dialects (as the training and development sets do), the second one (test-b) includes an unknown number of dialects. the distribution of examples across the different classes of interest for each of the subtasks is shown in figure 1 . as one can observe in figure 1-a , the distribution across the 18 dialects is unbalanced, with eqypt being the most frequently occurring label in the dataset for subtask 1. for the sentiment analysis subtask, the organisers provided a dataset of tweets labelled as any one of three classes: positive, negative and neutral. it was divided into training, development and test sets. as shown in figure 1-b , the positive and negative classes have an almost equal distribution between them, but the neutral class has a slightly lower number of training samples. the datasets for both subtasks were preprocessed whereby urls were replaced with the token 'url', and twitter usernames were replaced with the token 'user', in order to normalise them. ## methodology our approaches to the two subtasks are both underpinned by the first version of marbert, a language model that had been trained on a 128gb dataset containing both msa and da tweets @xcite . it is worth noting that we built our own version of the marbert model by continuing to train it for the masked language modelling (mlm) objective @xcite ; we describe this model in detail in section 3.2 below. however, our experiments showed that using our own marbert model led to performance improvement only for sentiment analysis and not for dialect identification. therefore this model formed the basis of our solution for subtask 2 but not for subtask 1. ## evaluation and results the performance of our ensemble models for the dialect identification subtask is summarised in with regard to the second subtask, we present the performance of our ensemble models for sentiment analysis in ## discussion to allow us to draw some insights on the class-level performance of our best-performing dialect identification model, we provide the confusion matrix based on the development set, in figure 2 . as one can observe in the confusion matrix, the majority of the true samples from dialects such as egypt, ksa, and iraq, have been correctly predicted by our model. this can be explained by the fact that such classes are over-represented in the training data. however, the over-representation of such classes is likely to have also led to a detrimental effect, i.e., the model being biased towards such dominant dialects, as can be observed in the columns of the confusion matrix, where many samples tend to be wrongly predicted as egypt or ksa, for instance. meanwhile, as expected, the model obtained poor performance with respect to the least represented dialects such as bahrain and qatar. also, our model tends to be confused by dialects which correspond to regions which are geographically close to each other and hence share certain dialects, e.g., oman vs ksa, lebanon vs egypt. as for our best-performing sentiment analysis model, the confusion matrix in figure 3 shows that the model performs almost equally well on the positive and negative classes. unsurprisingly, it does not perform as well for the neutral class, which has a slightly lower number of training samples. tweet text english translation gold pred. 1 may god protect him and protect us iraq oman 2 may god keep them for you libya oman 3 no, she is egyptian ksa egypt 4 but the series is nice jordan iraq 5 i love you ksa iraq 6 :( back from qatar :( ksa uae 7 may allah protect him and prolong his age ksa oman 8 god bless you, amen ksa oman we also investigated some samples from neighbouring countries such as ksa, oman and uae (united arab emirates), which are all gulf countries. as shown in ## conclusion and future work in this paper, we presented our ensemble-based approaches to the nadi 2022 subtasks: dialect identification and sentiment analysis. our results demonstrate that an ensemble model consisting of a combination of marbert models fine-tuned in different ways, for each of the subtasks, obtains topranking performance. a potential future direction is the exploration of multi-task learning for jointly training a model on the two subtasks.