GPTA ra E val: A Comprehensive Evaluation of C hat GPT on A rabic NLP (2023)

Abstract: ChatGPT’s emergence heralds a transformative phase in NLP, particularly demonstrated through its excellent performance on many English benchmarks. However, the model’s efficacy across diverse linguistic contexts remains largely uncharted territory. This work aims to bridge this knowledge gap, with a primary focus on assessing ChatGPT’s capabilities on Arabic languages and dialectal varieties. Our comprehensive study conducts a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. To our knowledge, this marks the first extensive performance analysis of ChatGPT’s deployment in Arabic NLP. Our findings indicate that, despite its remarkable performance in English, ChatGPT is consistently surpassed by smaller models that have undergone finetuning on Arabic. We further undertake a meticulous comparison of ChatGPT and GPT-4’s Modern Standard Arabic (MSA) and Dialectal Arabic (DA), unveiling the relative shortcomings of both models in handling Arabic dialects compared to MSA. Although we further explore and confirm the utility of employing GPT-4 as a potential alternative for human evaluation, our work adds to a growing body of research underscoring the limitations of ChatGPT.

## introduction large language models (llms) pretrained on next token prediction brought significant progress to nlp. these models can be finetuned to follow human instructions, allowing users to steer model output @xcite @xcite @xcite . chatgpt 1 is the most prominent among these models and has recently received significant 1 https://openai.com/blog/chatgpt attention due to its remarkable abilities. 2 the underlying model behind chatgpt is believed to be pretrained on large datasets from multiple languages, which could enable the model to work well in multilingual settings. there is, however, much hype about chatgpt's abilities. in fact, some observations around chatgpt remain anecdotal, calling for a rigorous evaluation of its abilities in different languages. while multiple investigations of chatgpt performance on english are accumulating fast @xcite @xcite , evaluation on other languages remains largely speculative. this lack of methodic evaluation inhibits our understanding of the capabilities and limitations of llms beyond english. this motivates our work for evaluating chatgpt on the wide collection of languages and language varieties commonly referred to as arabic. arabic has rich morphologies and syntactic structures. it also has a vast native speaker population of over 450 million people, making investigations of it necessary for technological inclusion. combined with this large population, the complexity of arabic requires models to have a deep understanding of expansive sociolinguistic contexts. this also makes the evaluation on arabic all the more important to our understanding of how large multilingual models behave, with implications that go well beyond arabic. concretely, our objective is to systematically evaluate chatgpt on a wide range of arabic natural language understanding (nlu) and natural language generation (nlg) tasks, identifying existing gaps and ultimately possibly guiding the development of more effective arabic and multilingual applications. through our comprehensive benchmarking on 44 diverse tasks comprising over 60 datasets, we observe that chatgpt exhibits inferior performance compared to much smaller finetuned models (section 6 and section 7). when we consider the performance of chatgpt and gpt-4 on country-level dialectal arabic as compared to modern standard arabic (msa), the modern-day variety used in formal settings, we reveal even more inferior dialectal performance (section 8). to the best of our knowledge, this is the first work to study the comparison of chatgpt and gpt-4 on arabic dialectal variation. the present study unambiguously demonstrates that, notwithstanding its noteworthy performance on standardized english benchmark datasets @xcite , the efficacy of chatgpt as a universally applicable solution appears to be unsubstantiated when considering languages characterized by extensive linguistic variations, particularly in the context of arabic. succinctly put, it is evident that considerable enhancements can still be made pertaining to arabic and other linguistically analogous languages. our contributions can be summarized as follows: ## related work we provide a brief overview of the previous works that evaluate chatgpt on nlp tasks here. we offer a detailed walkthrough in appendix a. performance on english. in machine translation (mt), while chatgpt equates to commercial tools like google translate @xcite , it falls short in domain-specific translation and low-resource languages. however, strategies like pivot prompting @xcite and additional information-infused prompts @xcite can enhance performance. in question-answering (qa), while chatgpt shows potential @xcite @xcite , it falters on complex opendomain questions and tasks requiring reasoning @xcite and is vulnerable to adversarial examples and perturbations @xcite . for text classification, chatgpt performs well in zero-shot and few-shot settings @xcite @xcite , sometimes matching or surpassing fully supervised models. performance on multilingual tasks. in multilingual tasks, chatgpt is found to struggle with generating non-latin scripts @xcite , but strategies such as prompting task descriptions in a high-resource language such as english can improve results @xcite . in line with this research, we find chatgpt to work better with english prompts than arabic prompts. lai et al. (2023) evaluate chatgpt in zero-shot setting on seven tasks covering 37 languages including arabic. the authors show chatgpt performs generally well for high-resource languages such as english, russian, and german compared to mediumresource languages such as arabic. huang et al. (2023) introduce cross-lingual thought and assess the multilingual capability of chatgpt on 7 benchmarks. the authors specifically evaluate chatgpt on arabic for natural language inference task in few-shot settings. compared to these concurrent efforts, our work conducts chatgpt evaluation on arabic at a much larger scale with 44 diverse tasks, as well as a comprehensive analysis of chatgpt and gpt-4's performance on dialectal varieties. wu et al. (2023a) find that chatgpt fails to surpass the passing score in chinese medical licensing exam. similarly, @xcite show that chatgpt fails to pass the japanese medical licensing exam. in our work, we conclusively show that chatgpt is inferior to supervised models across a wide range of domains on several varieties of arabic. ## datasets we evaluate chatgpt on both arabic nlu and nlg tasks. for nlu, we use a total of 48 datasets representing 20 different tasks from the orca benchmark @xcite . the diverse coverage of these datasets lends our evaluation broad representativeness and applicability to many real-world scenarios. for nlg, we compile 23 publicly available datasets from across 13 task clusters. we briefly introduce our evaluation datasets below and provide a detailed description of them in appendix b. nlu tasks. in nlu, we include 20 tasks spanning across arabic-nli (1), 3 claim prediction (1), dialect identification (3), machine-generated text detection (1), paraphrase detection (1), sentiment analysis and emotion detection (2), social meaning detection (9), stance detection (1), and word sense disambiguation (1). task-specific details with full citation of all datasets are in appendix b.1. nlg tasks. we include 24 nlg tasks covering 13 task clusters: code-switched translation (2), diacritization (1), dialect translation (6), grammatical error correction (1), mt (4), news title generation (1), open-domain dialectal generation (3), paraphrasing (1), qa (1), question generation ( 1 ), text rewriting (1), transliteration (1), and summarization (1). again, we provide task-specific details with appropriate references in appendix b.2. ## prompt design a prompt is a set of instructions provided to an llm that programs the model by enhancing its purpose and capabilities @xcite . a 3 number of tasks in each cluster are in parentheses. you are an annotatorgpt. your task is to do task. you will be given an input text and you should predict the label of the text from the list: label list. now predict the label of the following input: input: test input output: you are an annotatorgpt. your task is to do task. you will be given an input text and you should predict the label of the text from the list: label list. here, we provide you k examples of the task: example 1: ... example 2: ... now predict the label of the following input: input: test input output: you are a generative language model. your task is to do task. you will be given an input text and you should generate the task outcome. now generate the output of the following input: input: test input output: nlu tasks nlg tasks 0-shot k-shot you are a generative language model. your task is to do task. you will be given an input text and you should generate the task outcome. here, we provide you k examples of the task: example 1: ... example 2: ... now predict the label of the following input: input: test input output: prompt can influence subsequent interactions with the model as well as its generated outputs by defining a set of specific rules. therefore, in order to acquire a desired outcome, it is important to clearly define the set of rules and intents to the model. in our initial experiments, we experimented with a diverse set of prompt templates in both english and arabic. specifically, we evaluated on dialect identification, machine-generated text detection, and toxicity detection for nlu and machine translation tasks for nlg with both english and arabic prompts, observing that the english prompt outperforms its arabic counterpart. hence, we design a universal prompt template in english as follows: ## experiments we run experiments under 0-shot, 3-shot, 5-shot, and 10-shot settings. for the training data of the few-shot experiments, we randomly sample from each respective training dataset. 4 for a k-shot setting, we make sure that if a training sample is selected then it will also be selected for n-shot settings where @xmath0. for evaluation, we randomly sample a set of 200 examples from the test set of each dataset, to keep the cost man-ageable. we evaluate chatgpt (gpt-3.5-turbo), 5 which is an optimized version of gpt-3.5 series. we set the temperature to 0.0 while generating responses from chatgpt. we compare this model with bloomz (7.1b parameters), foot_3 which is finetuned on a diverse set of tasks on 46 languages including arabic @xcite . we choose bloomz since it is claimed to achieve impressive generalization on unseen tasks. following chung et al. (2022a), we use free-form generation for both chatgpt and bloomz along with simple post-processing, e.g. removing leading or trailing whitespace. additionally, to further situate performance of chatgpt, we finetune marbert v2 (abdul-mageed et al., 2021a) and arat5 model (nagoudi et al., 2022b). again, we choose these models as they are reported to achieve sota on nlu and nlg in abdul-mageed et al. (2021a) and nagoudi et al. (2022b), respectively. for both marbert v2 and arat5, we identify the best model on the respective development split (dev). we train marbert v2 for 25 epochs with a patience of 5 and arat5 for 10 epochs. for both models, we set the learning rate to 5e-5 across all the tasks. we present our overall experimental setup in figure 1 . in addition to reporting the performance of finetuned models on the same 200 set as chatgpt and bloomz, we report the 3-run average of the models' performances on the respective full test sets (reported in gray). for nlu tasks, we use macro-f 1 scores and for nlg tasks we use the appropriate metric suited to each task ( we provide examples of nlg model responses in appendix i. ## evaluation on nlu tasks overview. we present our evaluation on nlu tasks in emotion and sentiment. chatgpt achieves 17.33% f 1 on emotion detection with 5-shot learning, while bloomz achieves 17.13 with 10-shot. both of these models, however, perform much lower than finetuned marbert v2 (f 1 =68.74). for sentiment, chatgpt (64.96) foot_4 outperforms bloomz (50.44) by a margin of 14.52 f 1 scores. again, marbert v2 (f 1 =68.92) outperforms both models. dialect. on binary-level dialect classification, chatgpt achieves 77.00 f 1 with 3-shot learning, outperforming bloomz (49.68 f 1 ) by 27.32 points. meanwhile, marbert v2 (88.00) significantly outperforms both models. on region-level dialect, chatgpt achieves 30.69 f 1 , outperforming bloomz (30.19). marbert v2 outperforms both models (89.69). on country-level dialect, bloomz exhibits lower performance (14.61) compared to chatgpt (19.74) and marbert v2 (33.88). similar to the two other dialect tasks, chatgpt falls behind the much smaller model on country-level dialect id. chatgpt performs slightly better for mixed @xmath1, while bloomz performs better in mixed @xmath2. interestingly, both chatgpt and bloomz show highly superior performance than the finetuned arat5. we suspect that both models have been pretrained with a lot of data from highresource languages like english and french. this pretraining step helps them to perform better than the arabic-dedicated smaller model. ## human evaluation we also perform a set of human evaluations, motivated by the potential of human evaluation to capture subtleties of language that automated metrics may overlook. we carry out our evaluation on eight nlg tasks: code-switched translation (2), dialogue generation (3), machine translation (1), paraphrase (1), and summarization (1). we particularly pick these tasks as they represent diverse generation task categories and chatgpt performs poorly on some of them in our automatic evaluation setting ( evaluation setup. we build our human evaluation framework on top of @xcite , @xcite and implement a four-level (a, b, c, d) rating system that we adapt to language generation. (refer to appendix g.1 for details). we prepare the framework with examples and ask three pairs of native arabic speakers (@xmath3, based on how effectively a sample has fulfilled the task described in the input prompt. results. we aggregate scores from each pair of annotators and use cohen's kappa metric to measure the inter-annotator reliability. we find the interannotator agreement more than 0.6 for the majority of the tasks. we report the result in figure 5 (refer to appendix l for more detailed results). from human evaluation, we find that outputs produced by gpt-4 and chatgpt, for all tasks, are rated as mostly of fine quality. we also find that chatgpt is rated higher than bloomz for summarization in the human evaluation than it is in the automated setting, perhaps reflecting that, although useful, rouge is not an ideal metric for summarization as it captures only token-level overlap rather than deep semantics. in addition, chatgpt is rated higher than bloomz in human evaluation than in automated evaluation for paraphrasing. this, again, reflects bleu not being ideal for paraphrase detection for the same reasons as in the case of summarization. human analysis on cst. as evident from we observe that chatgpt produces fluent translations that are also semantically close (faithful) to the source text. however, chatgpt struggles to produce code-switched text (i.e., it generates mostly in arabic script). interestingly, this issue is more prevalent for msa than egyptian and moroccan. we hypothesize that this discrepancy cuts across several linguistic categories and involves topics such as translation of endearment expressions; multi-word expressions, idioms, and proverbs; negation; sub-token-level code-switching; and dialectal variants of msa lexica. we further suspect that the tokens in the source english text are very common in msa. as a result, the model does not seem able to code-switch the words into english. bloomz is rated 'd' on all of the tasks. upon inspecting the explanations from gpt-4, we find that this poor rating of bloomz is mostly due to it just copying the input samples rather than following the input instruction properly. does gpt-4 eval correlate with human eval? @xcite and @xcite show a positive correlation between human and gpt-4 evaluation, which we also test for arabic nlp in this work. hence, we compute the percentage of evaluation agreement between humans and gpt-4 on the models' responses. more specifically, for each task, we calculate at how many occurrences human evaluators agree with gpt-4 evaluation. as figure 6 shows, at least one human evaluator agrees with gpt-4 evaluation 71.5% of the time on average. this suggests that it may be possible to use gpt-4 to automate the time-consuming and costly process of using humans for assessing model performance. this is at least true for the tasks we consider. ## conclusion we presented a comprehensive evaluation of chatgpt on 44 diverse arabic nlp tasks. our evaluation covers both nlu and nlg under k-shot settings. comparing results from chatgpt to bloomz and finetuned models, we show that chatgpt, in spite of its large size, can struggle on multiple arabic tasks compared to much smaller finetuned models. we conduct an extensive quality assessment using both human and gpt-4 evaluation of model responses and find a positive alignment between human and gpt-4 judgment. we further investigate the performance of chatgpt and gpt-4 on msa vs da tasks, revealing inherent limitations of both models, particularly when applied to dialects. our findings accentuate the inherent multilingual limitations observed in both chatgpt and gpt-4, thereby indicating a substantial scope for improving these models. we intend to expand our evaluations to encompass additional low-resource languages.