Evaluating C hat GPT and Bard AI on A rabic Sentiment Analysis (2023)

Abstract: Large Language Models (LLMs) such as ChatGPT and Bard AI have gained much attention due to their outstanding performance on a range of NLP tasks. These models have demonstrated remarkable proficiency across various languages without the necessity for full supervision. Nevertheless, their performance in low-resource languages and dialects, like Arabic dialects in comparison to English, remains to be investigated. In this paper, we conduct a comprehensive evaluation of three LLMs for Dialectal Arabic Sentiment Analysis: namely, ChatGPT based on GPT-3.5 and GPT-4, and Bard AI. We use a Saudi dialect Twitter dataset to assess their capability in sentiment text classification and generation. For classification, we compare the performance of fully fine-tuned Arabic BERT-based models with the LLMs in few-shot settings. For data generation, we evaluate the quality of the generated new sentiment samples using human and automatic evaluation methods. The experiments reveal that GPT-4 outperforms GPT-3.5 and Bard AI in sentiment analysis classification, rivaling the top-performing fully supervised BERT-based language model. However, in terms of data generation, compared to manually annotated authentic data, these generative models often fall short in producing high-quality Dialectal Arabic text suitable for sentiment analysis.

## introduction sentiment analysis is the task of determining the emotional tone of a piece of text, such as whether it is positive, negative, or neutral. it is a challenging task for many languages, including arabic, due to the complex morphology and syntax of the language. various approaches have been used to tackle this challenge, including rulebased and dictionary-based methods @xcite @xcite , classical machine learning algorithms @xcite @xcite @xcite , deep learning @xcite , and pre-trained language models such as bert @xcite . however, sentiment analysis faces a critical challenge, particularly in the context of social media, which is data drift and concept drift @xcite . this challenge necessitates continuous monitoring of sentiment analysis models and updating rule-based systems and dictionaries, if utilized, as well as retraining machine learning models with new data. recent advancements in nlp, particularly the emergence of large language models (llm) such as @xcite and palm 2 @xcite , and their utilization in chatgpt and bard ai, respectively, show potential in countering the issues of data and concept drift in sentiment analysis. these llms are trained on large and diverse datasets and fine-tuned or prompted for various tasks, including sentiment analysis @xcite @xcite , and have proven their capabilities for this task in english. although few research efforts have been made to test the ability of llms for arabic sentiment analysis, focusing on single language models like arat5 @xcite or multiple models, including @xcite , to the best of our knowledge, no study has been conducted to evaluate both of bard ai and chatgpt llms for arabic sentiment analysis. in particular, this is the first attempt to evaluate bard ai on arabic sentiment analysis. this paper aims to evaluate three generative large language models, namely generative pretrained transformers gpt-3.5 and gpt-4 through chatgpt by openai, and the pathways language model (palm) through bard by google on a sentiment analysis dataset comprising in the saudi di-alect, the saudi dialect twitter corpus (sdtc) @xcite , comprising 5,400 tweets classified into five classes: positive, negative, neutral, spam, and "i do not know" class, to reveal the capabilities of llms in tackling the arabic sentiment analysis challenge. the contribution of this paper is twofold. first, it includes the evaluation of google bard ai for the first time in this type of analysis. second, it evaluates these models for arabic sentiment analysis from different and novel perspectives, as illustrated in the experiment's design (section 3). mainly, we address the following research questions (rqs): the structure of the rest of the paper is as follows: section 2 presents previous work on sentiment analysis and using generative models for natural understanding tasks. section 3 shows the experiment design and the dataset used to evaluate various models. then, in section 4, we present the results of four comprehensive experiments and analysis. we conclude the paper in section 5. ## experiments design and data the primary objective of these experiments is to assess the capabilities of generative models for arabic sentiment analysis and the potential of data augmentation and generation for this task. we evaluate three models: for gpt-3.5 and gpt-4 we utilize the chatgpt api to send prompts and receive responses. for palm 2, prompts are manually sent to bard ai via its web interface, from which we extract the relevant responses. also, we utilize these models to generate new samples and systematically evaluate the generated examples. we have four main experiments: the model that shows the best performance will serve as our baseline model. for the abovementioned experiments, we use the saudi dialect twitter corpus (sdtc) @xcite . sdtc comprises 5,400 tweets distributed across five classes: positive, negative, neutral, spam, information, and difficult to classify. in our experiments, we focused on the first three classes: positive, negative, and neutral tweets, amounting to 558, 1,632, and 500, respectively. the total number of tweets in our experiments is 2,690. we randomly split the sdtc dataset into 75% for training sdtc train and 25% for testing sdtc test , obeying the class distribution. also, we selected 30 tweets from each class (90 tweets overall) from sdtc train to evaluate the output of each proposed prompt. we use sdtc dev to refer to these 90 tweets. sdtc train is used to fine-tune the language models in exp 1 and exp 4. the sdtc test set is used for evaluating the fine-tuned language models (exp 1 and exp 4) and for the predictions of the generative models in exp 2. experiment 3 involves human judgment, and the outputs of the generative models will be used to fine-tune the language models in exp 4. we use sdtc dev for evaluating the output of different prompts in exp 2 and exp 3. we make sdtc dev balanced in classes because of its relatively small size, due to budget and time constraints, as we couldn't evaluate all prompts and different numbers of shots on a larger scale. however, we evaluate the best settings in terms of prompts and number of shots on the whole test set sdtc test . these prompts were inspired or adapted from previously published research @xcite . ## experiments and results in this section, we show and discuss the results of the experiments described in section 3. to assess the performance of the models, we employed the accuracy (acc) metric, along with the microaverages of precision (p), recall (r), and f-1 score (f) values. when evaluating the models' performance, we focus on the f1 measure as the primary metric of comparison. ## conclusion in this paper, we presented various experiments using three generative models for arabic sentiment analysis in the saudi dialect: gpt 3.5, gpt-4, and bard ai (palm 2). we compare their performance with fully supervised bert-based models. we also evaluate the quality of generated examples by these llms using manual and automatic methods. the experiments show that the generative large language models, with little or no training data in few-shot settings, perform relatively well on arabic sentiment analysis compared to fully fine-tuned models. for sentiment analysis text classification, the experiments show that gpt-4 outperforms most of the bert-based model and is on a par with the second-best bert-based model. bard ai comes next with a performance comparable to fully finetuned models, while gpt3.5 significantly underperforms the two models and is lower than the bert-based models. for sentiment generation, all models struggle to generate high-quality text for sentiment analysis in the saudi dialect, especially for neutral text. interestingly, chatgpt (both gpt-3.5 and gpt-4) tends to generate nonsense text in the saudi dialect more than bard ai. also, implementing safeguards to prevent the generation of harmful or toxic content is crucial for responsible and safe utilization. however, these restrictions can sometimes act as barriers when generating representative text that has a negative sentiment, especially in applications that require a comprehensive representation of human emotions and viewpoints. future research should consider comparing the generative models performance among different arabic dialects and datasets. also, another direction for future work is analyzing the performance of generative models pre-trained on dialectal arabic text (such as arat5) and fully fine-tuned on generating tweets.