Ahmed and Khalil at NADI 2022: Transfer Learning and Addressing Class Imbalance for A rabic Dialect Identification and Sentiment Analysis (2022)

Abstract: In this paper, we present our findings in the two subtasks of the 2022 NADI shared task. First, in the Arabic dialect identification subtask, we find that there is heavy class imbalance, and propose to address this issue using focal loss. Our experiments with the focusing hyperparameter confirm that focal loss improves performance. Second, in the Arabic tweet sentiment analysis subtask, we deal with a smaller dataset, where text includes both Arabic dialects and Modern Standard Arabic. We propose to use transfer learning from both pre-trained MSA language models and our own model from the first subtask. Our system ranks in the 5th and 7th best spots of the leaderboards of first and second subtasks respectively.

## introduction the 2022 nuanced arabic dialect identification (nadi) shared task @xcite is comprised of two subtasks: arabic dialect identification, and sentiment analysis for arabic dialects. the aim of the shared task is to alleviate the lack of resources in nlp for arabic dialects, amid growing interest in arabic dialect language models @xcite @xcite . the 2022 edition is the third nadi shared task. the 2021 @xcite and 2020 nadi shared tasks @xcite focus on country-and province-level arabic (sub-)dialect identification. these two editions also tackled tweets in arabic dialects, gathering dialects from 100 provinces in 21 arab countries. in this paper, we tackle both subtasks, using both transfer learning from pre-trained language models, and transfer learning from one subtask to the other, as well as loss functions adapted to the class imbalance in the dataset. the first subtask tackles country-level arabic dialect identification in tweets. we first analyse the data, and find that there is a high class imbalance between the 18 countries represented in the tweets. we find that the largest class has nearly 20 times as many samples as the smallest one. we try multiple pre-trained arabic language models, and find that the highest-performing model is marbert @xcite . we try different loss functions, and find that focal loss @xcite performs the best, as it applies a modulating term to the cross-entropy loss, enabling the training process to focus on wrongly classified samples. we fine-tune the focusing hyperparameter γ, and observe how performance fluctuates accordingly. the second subtask deals with sentiment analysis for tweets in various arabic dialects, as well as in modern standard arabic. there are three classes: positive, negative, and neutral sentiment. in our data analysis, we find that there is less class imbalance in the second subtask, especially between the positive and negative classes. however, this second subtask has a much smaller training set, and therefore needs a supplement of knowledge from other sources. given that external labeled data is not allowed, we decide to employ transfer learning, by fine-tuning the best model from the first subtask on this second one. as the dataset of the second subtask contains both arabic dialects and modern standard arabic, we hypothesize that performance will benefit from language models trained on modern standard arabic, as well as from data in arabic dialects. finally, we show that our system ranks in the 5th and 7th best spots of the leaderboards in the first and second subtasks respectively, and propose suggestions for improving performance. ## data in this section, we describe the data used for training our system in both subtasks. the first subtask deals with arabic dialect identification. the training data contains 18 classes. each class corresponds to the national vernacular we perform a similar analysis for validation data, and find that the distribution is similar, as shown in figure 2 . the validation dataset has 4,871 samples. the class with the most samples is again egypt with 1,041 datapoints, whereas the smallest ones are qatar and bahrain with 52 samples each. the second subtask is sentiment analysis over tweets in various arabic dialects. this is a threeway classification problem, where the goal is to predict whether a tweet -regardless of the arabic dialect -has positive, neutral or negative sentiment. this subtask has fewer datapoints than the first one. the training set contains 1,500 samples, whereas the validation set contains 500 samples. there is roughly the same distribution over the sentiment classes between the two sets, as shown in figures 3 and 4 . ## system description for both subtasks, we investigate the potentials of transfer learning for different arabic bert-based models. specifically, we compared the following pre-trained bert-based models: marbert (abdul-mageed et al., 2021a), camelbert (inoue et al., 2021) and arabert (antoun et al., 2020 our experiments consist of fine-tuning a pretrained bert model, plus one or more fully connected layers. it turns out that the best performance is achieved using only the pre-trained model plus a classification layer. for all experiments, we use the following hyperparameters: a learning rate of 4 * 10 -5 , 10 training epochs, an adam optimizer with weight decay regularization. the batch size is set to 32 for the first subtask, and 8 for the second subtask. we implement our models using pytorch. for the loss functions, we experiment with self-adjusting dice loss (selfadjdiceloss) @xcite , negative log-likelihood loss (nllloss), cross-entropy loss (crossentropyloss) with and without weighted classes, and focal loss (focalloss) @xcite . the latter has shown the best performance for both sub tasks. this could be due to the fact that the first subtask's dataset is imbalanced, and focal loss is designed to alleviate class imbalance. in order to focus on hard, wrongly classified samples, focal loss applies a modulating term to the cross-entropy loss. given the crossentropy loss formula: participants of the shared task were not allowed to use external labeled data for training. however, the second subtask has a substantially smaller training set than the first one. we decide to leverage the knowledge learned by the model during the first subtask, and fine-tune the model on the training set of the second subtask. ## results and discussion for the first subtask, we experiment with the γ hyperparameter of the focal loss. we try the following values: 5, 10, 20 (default value), and 25. we show the results on the validation and training sets in figure 5 . we see that the lowest validation loss is achieved with γ = 25, but the highest accuracy and f1 scores are achieved with γ = 20. so we use γ = 20 for the remainder of the experiments. this confirms that performance is higher when class imbalance is addressed during training. the accuracy and f1 scores seem to peak for the validation set at the second epoch, whereas they increase for the training set as the training epochs progress. this indicates that overfitting occurs after the second epoch, in particular for γ = 20. we show our dev set results in for the test set results, our system scores in the 7th best spot in the first subtask, out of 19 participants, and the 5th best spot out of 10 participants in the second subtask. the leaderboards and test results for the first and second subtasks are shown in figures 7 and 6 respectively. for the first subtask, there are two test subsets: test-a is a subset containing all 18 classes, whereas test-b is a subset containing k classes, where k is unknown. the results shown in figure 7 are the average of the macro-f1 scores between both test subsets. we notice that the results of the second through fifth rows in the leaderboard of the first subtask are close. for the second subtask, the shared task organizers evaluate using "macro-f1-pn", which is a macro-f1 score computed for the positive and negative classes, ignoring the neutral cases. if we had more time, we would investigate domain adversarial learning and multi-task learning. as transfer learning proved useful in the second subtask, this suggests that a multi-task learning setting could benefit both subtasks. moreover, in the second subtask, there are tweets from different countries, but it is a feature that does not matter in the sentiment analysis task. the model would benefit from learning to not distinguish between arabic dialects, as it would learn dialect-agnostic sentiment features that enable easy knowledge transfer between tweets in different arabic dialects. ## conclusions in this paper, we presented our team's approach to the two subtasks of the 2022 nadi shared task. we first analysed the data, and find that there is class imbalance between the 18 classes of the arabic dialect identification subtask. in the arabic tweet sentiment analysis subtask, we find that classes are relatively more balanced, but there are fewer datapoints to train on. we propose to train on marbert, and we find that focal loss is the loss function that performs best, as it addresses class imbalance. our experiments with the γ focusing hyperparameter show that we need a large γ value for high f1 scores, confirming that focal loss alleviates class imbalance. finally, our system scores favorably in the leaderboards in both subtasks. we suggest for the second subtask that domain-adversarial training could benefit performance, as it would make the model learn dialect-agnostic features about the sentiment classes.