Benchmarking transfer learning approaches for sentiment analysis of A rabic dialect (2022)

Abstract: Arabic has a widely varying collection of dialects. With the explosion of the use of social networks, the volume of written texts has remarkably increased. Most users express themselves using their own dialect. Unfortunately, many of these dialects remain under-studied due to the scarcity of resources. Researchers and industry practitioners are increasingly interested in analyzing users’ sentiments. In this context, several approaches have been proposed, namely: traditional machine learning, deep learning transfer learning and more recently few-shot learning approaches. In this work, we compare their efficiency as part of the NADI competition to develop a country-level sentiment analysis model. Three models were beneficial for this sub-task: The first based on Sentence Transformer (ST) and achieve 43.23% on DEV set and 42.33% on TEST set, the second based on CAMeLBERT and achieve 47.85% on DEV set and 41.72% on TEST set and the third based on multi-dialect BERT model and achieve 66.72% on DEV set and 39.69% on TEST set.

## introduction digital connectivity among arab population has remarkably grown in the last few years. apart from technological progress, the covid-19 pandemic has been a factor for the increase of the penetration rate and consequently the increase in dialectal textual content in social networks. the dialect forms that differ from one region to another, have been considered for a long time to oral conversations of everyday life. they have neither standard nor sufficient resources for computational processing, unlike the mother language: msa. as a result, there is a growing interest in dealing with this type of content. in this work we focus on developing a sentiment analysis model in the framework of shared task: sentiment analysis of country-level arabic. several approaches have been proposed in the literature to build sentiment analysis models for poorly endowed languages. deep learning has been proved as a very effective paradigm to classify sentiments in large data sets. however, this approach was not effective on small data sets and most of the time traditional machine learning algorithms get better scores. in recent time transfer learning approaches has been shown to be beneficial to train a small data set and this by fine tuning a neural network model trained on a large data-set. bert model @xcite based on transformer architecture is one of the effective transfer learning model. indeed, @xcite have used it to classify if an algerian tweets is positive, negative or neutral. the model achieved an accuracy of 68%. also, (abdul-mageed et al., 2020) have proposed another variant of bert model baptized marbert that focused on both dialectal arabic (da) and modern standard arabic (msa). the sentiment analysis model achieved an f-score of 71.50%. using the same model, @xcite have explored the effectiveness of augmenting data techniques proposed by @xcite to analyse the sentiments among a tweets corpus, the authors obtained an f1score of 86%. moreover, the few-shot learning (fsl) approach has also been exploited in sentiment analysis. it is a sub-area of transfer learning which allows to classify new data when there is only a few training samples with supervised information which is the case in the present work. fsl is adapted with some success to nlp tasks, such text classification: @xcite have proposed a meta-learning based method by using distributional signatures for few-shot text classification. @xcite have presented few-shot text classification system upgraded by label semantic augmented meta-learner (lasaml) uses of label semantics. in order to build our system for nadi shared task: country-level sentiment analysis, we opted for transfer learning approach. in fact, we compared the effectiveness of three transfer learning models on different configurations of the corpus proposed by the organizers, this paper is structured as follows: section 2 describes the nadi shared task's data set. section 3 details the pre-processing and normalisation applied to the data set. section 4 describes the data augmentation. section 5 presents our proposed sentiment analysis model for country-level arabic. finally, the conclusion is given in section 6. ## data three labeled data sets have been provided by the organizers @xcite to build a sentiment analysis model: train set for model training: it contains 1500 tweets, 23889 words and 10422 different words (including 207 emojis). dev set to adjust the model parameters: it contains 500 tweets, 7893 words and 4290 different words (including 102 emojis). test set for evaluation: it contains 3000 tweets, 47293 words and 17926 different words (including 346 emojis). more statistics on word distribution in each class are given in ## data pre-processing and normalisation in order to enhance the quality of the tweets corpus before feeding them as an input to the classification models, we apply the following pre-processing treatments: normalisation: we used in this phase the set of camel tools for arabic language processing @xcite ) that allows to apply the following alterations: normalization of few arabic characters and spelling errors in order to unify them into one form. in fact, there are some letters in arabic that can be described as confusing in some cases. we normalize words containing such letters having one representative letter. for example, the different representations of hamza ( , , and ) ## data augmentation different data augmentation techniques were proved to be useful to efficiently augment the corpus. thus, due to the small size of the competition corpus, we propose to augment the corpus with other versions of the corpus generated by different augmentation method and test their effectiveness to improve the quality of the sentiment classification model. ## conclusion in this paper, we presented our submitted method to the third nadi shared task. we proposed a transformer models for sentiment analysis of country level arabic. the experimental results shows that camelbert and setfit models achieved an f1-score of 43.11% and 42.33% respectively on testing data set better than multi-dialect bert model, while the latter achieved the best f1-score of 66.72% on development data-set.