Assessing the Performance of C hat GPT -4, Fine-tuned BERT and Traditional ML Models on M oroccan A rabic Sentiment Analysis (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks across different languages. However, their performance in low-resource languages and dialects, such as Moroccan Arabic (MA), requires further investigation. This study evaluates the performance of ChatGPT-4, different fine-tuned BERT models, FastText as text representation, and traditional machine learning models on MA sentiment analysis. Experiments were done on two open source MA datasets: an X(Twitter) Moroccan Arabic corpus (MAC) and a Moroccan Arabic YouTube corpus (MYC) datasets to assess their capabilities on sentiment text classification. We compare the performance of fully fine-tuned and pre-trained Arabic BERT-based models with ChatGPT-4 in zero-shot settings.

## introduction the field of sentiment analysis (sa) has made remarkable advances, enabling the extraction and analysis of human sentiments from textual data for a variety of purposes. this technology has diverse applications, ranging from social media monitoring and market research to political discourse analysis. however, sa faces several challenges, such as the phenomena of data and concept drift @xcite , particularly pronounced in the everevolving landscape of social media. data drift refers to changes in the statistical properties of the input data over time, while concept drift indicates a deeper shift in the underlying meaning or interpretation of the data that the model aims to predict. these drifts can lead to a decrease in the accuracy of sa models if not addressed. for instance, the way people express sentiments online can evolve rapidly, making previously trained models less effective. this necessitates ongoing monitoring and adaptation of sa methodologies, including updating rule-based systems and dictionaries, as well as retraining machine learning models to ensure they remain aligned with the shifting linguistic and cultural contexts of web-based communication. while sa has seen substantial progress in major languages, its application to dialectal languages, such as ma, a regional variant of modern standard arabic (msa), has not received much attention. ma is the main medium of communication among moroccans. the unique linguistic features of ma, including regional variations, colloquialism, borrowed words from other languages, coupled with the use of multiple scripts (arabic and latin), present significant challenges for sa. previous sa research @xcite has predominantly focused on corpora written in arabic script despite the increasing prevalence of latin script usage in online communication, particularly on social media platforms. recent advancements in ai, particularly the emergence of llms, such as gpt-4 (openai, 2023), palm 2 @xcite and falcon @xcite offer potential solutions to the challenges posed by data and concept drift in sa. these models, trained on vast and diverse datasets and fine-tuned for various tasks, have demonstrated promising capabilities in sa @xcite @xcite . while some research has explored the potential of llms for standard arabic sentiment analysis @xcite , no study has so far evaluated the performance of llms on ma. this work conducts the first-ever evaluation of chatgpt's performance on ma sa, offering valuable insights into the applicability of llms in analyzing sentiment in arabic dialects. ## related work the availability of sentiment data from social media platforms has greatly increased interest in arabic sentiment analysis (sa) research over the last ten years. speakers of arabic dialects were historically limited to using their dialects only when speaking. however, the emergence of social media has given arabic speakers the ability and space to express themselves in writing as well @xcite . this has resulted in an abundance of informal, dialectal textual material, as opposed to msa formality. a multitude of datasets spanning multiple genres-mostly tweets-have been created for arabic sa, including egyptian @xcite , levantine @xcite , maghrebi @xcite , as well as the saudi dialect @xcite . other datasets @xcite include several arabic dialects in addition to msa. arabic sentiment analysis has traditionally concentrated on rule-based techniques, much like other languages @xcite . the main goal of these techniques was to create sentiment lexicons. arabic sentiment analysis has seen a rise in interest in applying machine learning techniques in recent years. these techniques are less vulnerable to the drawbacks of lexicon-based techniques and are capable of identifying sentiment patterns from a big corpus of text. to implement morphological and syntactic features, popular machine learning techniques have been employed, such as naïve bayes (nb), support vector machines (svms), and k-nearest neighbor (knn) classifiers @xcite @xcite . transformer-based models, such as bert @xcite , albert @xcite , xl-net @xcite , and roberta @xcite , have been introduced and proved successful in several natural language processing (nlp) applications. bert and bert-like models achieved state-of-the-art performance on many nlp tasks, including sentiment analysis in many languages @xcite . abdul-mageed et al. (2020); antoun et al. developed two models, arbert and marbert, pretrained on a large collection of datasets in msa and several arabic dialects (levantine, moroccan arabic, etc.). they reported new state-of-the-art results on the majority of the datasets in their fine-tuning benchmark. in addition to discriminative models, such as bert, generative models have recently gained prominence in nlp research. these models, such as gpt @xcite , t5 @xcite , and bloom @xcite , are designed to create new text samples. multilingual and language-specific versions of these models have been developed. for example, arat5 @xcite and aragpt-2 (antoun et al.) are tailored for arabic. generative models have demonstrated potential in tasks, such as text completion, translation, summarization, and even sentiment analysis, where they can generate text that aligns with specific sentiments @xcite . in this work, we evaluate the performance of chatgpt-4 and transformer-based models on sa of ma using the aforementioned open source datasets, namely mac @xcite and myc @xcite . to our knowledge, this is the first-ever attempt to compare the performance of these models on the ma sa task. the structure of the rest of the paper is as follows: section 3 presents the experimental setup and the various experiments we conducted, datasets, and model architectures. section 4 presents the experiments' results and analysis. section 5 provides concluding remarks and future work. ## experimental setup our research objective is to evaluate the capability of the chatgpt model, some existing pre-trained bert models, and fasttext @xcite as sentiment analyzers for ma. to assess this, we utilize the two datasets, mac and myc, designed for sentiment analysis. on each dataset, we evaluate chatgpt, fine-tuned bert-based, and fasttext models. furthermore, we compared the fine-tuned and pre-trained bert-based models with the chat-gpt results. the results of our experiments are compared with other related work. we use gpt-4-turbo model by openai foot_0 for both mac and myc datasets as a sentiment analyzer for ma. we asked the model to predict the class of the given input tokens. for gpt-4, we utilize the chatopenai wrapper provided by langchain framework foot_1 to send prompts and receive responses. for bert-based models, we fine-tune (full network or freezing the model's backbone) various existing models pretrained on a large corpus in a variety of languages. ## conclusion and future work this work examined sentiment analysis on mac and myc datasets. we gained insights into the performance of different models and architectures in capturing sentiment nuances present in ma in different contexts and in both arabic and latin script. although fine-tuned models performed well, the results obtained with chatgpt show the latter's great potential for sa. the results have also shown that the performance of all these models on ma is less than that on msa. this difference in performance can be attributed to several factors, such as language uniformity, and consistency in msa grammar and vocabulary, which makes it easier for models to learn and predict accurately. on the other hand, ma varies across regions. lack of standardization of ma makes it challenging for models to perform consistently. additionally, the availability of data influence model performance. models, such as gpt-4 are trained on large corpora that include a substantial amount of msa texts, given its prevalence in written and formal contexts. future work requires the creation of large ma datasets and the development of new strategies to deal with the inconsistency in the ma data. another research direction would be to leverage the complementary nature of fasttext and bert embeddings by employing an attention mechanism to combine them effectively. by integrating the context-aware representations from bert with the morphological and semantic information captured by fasttext embeddings.