Foul at S em E val-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in A lgerian A rabic (2023)

Abstract: This paper describes the system we designed for our participation to SemEval2023 Task 12 Track 6 about Algerian dialect sentiment analysis. We propose a transformer language model approach combined with a lexicon mixing terms and emojis which is used in a post-processing filtering stage. The Algerian sentiment lexicons was extracted manually from tweets. We report on our experiments on Algerian dialect, where we compare the performance of marbert to the one of arabicbert and camelbert on the training and development datasets of Task 12. We also analyse the contribution of our post processing lexical filtering for sentiment analysis. Our system obtained a F1 score equal to 70\% , ranking 9th among 30 participants.

## introduction afrisenti-semeval shared task (task 12) @xcite aims at sentiment analysis on african dialectal languages of different african countries. it covers three main tasks and twelve languages with a corpus made of messages collected from the micro-blogging service twitter @xcite . our participation is for the first task (task a): monolingual opinion detection, in the specific case of the algerian dialect. in case a tweet conveys both positive and negative sentiment expression, the strongest one must be selected for the whole message. algerian dialect is considered a less-resourced language for sentiment analysis as few works have addressed the task in the past despite the relatively large number of speakers of the language 1 unlike other languages like english (belbachir and 1 algerian arabic is spoken by the majority of algerian population, counting around 32m speakers of the dialect. source: https://www.visualcapitalist. com/100-most-spoken-languages/ boughanem, 2018; @xcite . various works on opinion detection have addressed arabic in the past @xcite , but most of them concern classical arabic language the exception of the work of @xcite . however everyday life communication uses the dialect. each arabic country is characterized by its dialect different from the others by a large number of aspects: phonology, orthography, morphology, lexicon,... @xcite the importance of social media in everyday communication and the fact that algerian arabic is used by a majority of the population warrants our interest in developing an opinion-mining algorithm able to handle the dialect. to do so we need to solve the problems specific to the dialect, augmented by the peculiarities of the form communication takes in a microblogging service like twitter for instance: for our approach, we chose transformers @xcite because of their ability to leverage context for semantic analysis and also because they have already been investigated for arabic dialect sentiment analysis @xcite . but the lack of algerian dialect-specific resources needed by this type of approach for training leads us to consider an approach mixing transformers and specific lexicons. since the pioneering work of @xcite , many have provided proof of the utility of emoticons for tweets sentiment analysis in arabic @xcite and @xcite . this is why we complemented our lexicon of algerian-specific subjective terms (relatively rare but with a strong subjective value) with a second lexicon based on emoticons. this paper is organized as follows: section 2 presents methods existing for sentiment analysis and describes the corpus data. section 3 describes our system and explains our different experiments. sections 4 and 5 present our experiments and results. section 6 discusses negative results. ## system description to detect positive, negative, or neutral sentiments, we developed an approach that uses the contextual language model called marbert @xcite . this model is arabic-focuses language model (lm) exploiting large and massive diverse data. note that it has been adapted for arabic dialect detection task (alkhamissi et al., 2021) on short utterances but not for language understanding. to complement the performance of the language model which we found during our experiments to be under-performing on some algerian dialect sentiment terms, we constructed two lexicons: one for subjective terms and another for emoticons, both regrouping highly opinionated lexical items identified by hand. the lexicon was used in a post-processing stage of the language model as described in figure 1 . in this section, we detail the three modules of our pipeline: a) filtering, b) language model, and c) lexical post-processing. ## results on official sets for our system, we obtain an f1 score equal to 0.70 on test data and ranking 9 th among 30 participants. we analyze in more detail the behavior of our model on the positives, negatives, and neutrals tweets classes. for this, we use the train and development collection which is annotated for all our experiments as explained in subsection 4.1. ## negative results we use different methods to determine sentiment tweets which did not yield better results than the pre-training language models on arabic. we used the svm classifier with a vectorization based on tf-idf. this reports 2 points drop on the f1 measure. we use the long-short-term memory model (lstm) which yielded an accuracy of around 60%. we can conclude that the language models trained in arabic more specifically in dialectal arabic report better results. ## conclusion in this article, we describe our approach based on a language model and sentiment lexicons that we build manually to detect positive, negative, and neutral documents. our approach ranks among the top 10 for the algerian dialect sentiment analysis task. we show that the marbert language model performs better than arabic-bertor camelbert. we prove that the introduction of lexical filtering with both algerian dialectal terms and emoticons as a post-processing step to analysis with marbert increases the performance of our system. we believe that this system can be improved by adding new terms or emoticons to lexicons used in complementing analysis with a bert-based transformer and by using other combination strategies between the different models. for future work, we plan to improve our system by using other subjective lexicons of different languages. muhammad abdul-mageed, abdelrahim elmadany, and el moatez billah nagoudi. 2021. arbert & marbert: deep bidirectional transformers for arabic. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 7088-7105, online. association for computational linguistics. hind alamro, manal alshehri, basma alharbi, zuhair khayyat, manal m. kalkatawi, inji ibrahim jaber, and xiangliang zhang. 2021. overview of the arabic sentiment analysis 2021 competition at kaust. this paper provides an overview of the arabic sentiment analysis challenge organized by king abdullah university of science and technology (kaust). the task in this challenge is to develop machine learning models to classify a given tweet into one of the three categories positive, negative, or neutral. from our recently released asad dataset, we provide the competitors with 55k tweets for training, and 20k tweets for validation, based on which the performance of participating teams are ranked on a leaderboard, https://www.kaggle.com/c/arabicsentiment-analysis-2021-kaust. the competition received in total 1247 submissions from 74 teams (99 team members). the final winners are determined by another private set of 20k tweets that have the same distribution as the training and validation set. in this paper, we present the main findings in the competition and summarize the methods and tools used by the top ranked teams. the full dataset of 100k labeled tweets is also released for public usage, at https://www.kaggle.com/c/arabic-sentimentanalysis-2021-kaust/data. randah alharbi, walid magdy, kareem darwish, ahmed abdelali, and hamdy mubarak. 2018. partof-speech tagging for arabic gulf dialect using bi-lstm. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018), miyazaki, japan. european language resources association (elra). badr alkhamissi, mohamed gabr, muhammad el-nokrashy, and khaled essam. 2021. adapting mar-bert for improved arabic dialect identification: submission to the nadi 2021 shared task. in proceedings of the sixth arabic natural language processing workshop, pages 260-264, kyiv, ukraine (virtual). association for computational linguistics. stefano baccianella, andrea esuli, and fabrizio sebastiani. 2010. sentiwordnet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. in proceedings of the seventh international conference on language resources and evaluation (lrec'10), valletta, malta. european language resources association (elra). faiza belbachir and mohand boughanem. 2018. using language models to improve opinion detection. information processing and management, 54:958-968. chia-ping chen, tzu-hsuan tseng, and tzu-hsuan yang. 2018. sentiment analysis on social network: using emoticon characteristics for twitter polarity classification. in international journal of computational linguistics & chinese language processing, volume 23, number 1, june 2018, taipei, taiwan. association for computational linguistics and chinese language processing. jianpeng cheng, li dong, and mirella lapata. 2016. long short-term memory-networks for machine reading. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 551-561, austin, texas. association for computational linguistics. nurendra choudhary, rajat singh, vijjini anvesh rao, and manish shrivastava. 2018. twitter corpus of resource-scarce languages for sentiment analysis and multilingual emoji prediction. in proceedings of the 27th international conference on computational linguistics, pages 1570-1577, santa fe, new mexico, usa. association for computational linguistics. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. abdelkader el mahdaouy, eric gaussier, and saïd ouatik el alaoui. 2017. arabic text classification based on word and document embeddings. in proceedings of the international conference on advanced intelligent systems and informatics 2016, pages 32-41, cham. springer international publishing. bjarke felbo, alan mislove, anders søgaard, iyad rahwan, and sune lehmann. 2017. using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 1615-1625, copenhagen, denmark. association for computational linguistics. emna fsih, sameh kchaou, rahma boujelbane, and lamia hadrich-belguith. 2022. benchmarking transfer learning approaches for sentiment analysis of arabic dialect. in proceedings of the the seventh arabic natural language processing workshop (wanlp), pages 431-435, abu dhabi, united arab emirates (hybrid). association for computational linguistics.