A Systematic Study and Comprehensive Evaluation of C hat GPT on Benchmark Datasets (2023)

Abstract: The development of large language models (LLMs) such as ChatGPT has brought a lot of attention recently. However, their evaluation in the benchmark academic datasets remains under-explored due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. In this paper, we aim to present a thorough evaluation of ChatGPT’s performance on diverse academic datasets, covering tasks like question-answering, text summarization, code generation, commonsense reasoning, mathematical problem-solving, machine translation, bias detection, and ethical considerations. Specifically, we evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in these datasets. This makes our work the largest evaluation of ChatGPT in NLP benchmarks. In short, our study aims to validate the strengths and weaknesses of ChatGPT in various tasks and provide insights for future research using LLMs. We also report a new emergent ability to follow multi-query instructions that we mostly found in ChatGPT and other instruction-tuned models. Our extensive evaluation shows that even though ChatGPT is capable of performing a wide variety of tasks, and may obtain impressive performance in several benchmark datasets, it is still far from achieving the ability to reliably solve many challenging tasks. By providing a thorough assessment of ChatGPT’s performance across diverse NLP tasks, this paper sets the stage for a targeted deployment of ChatGPT-like LLMs in real-world applications.

## introduction in recent years, the introduction of transformerbased @xcite language models (lms), such as bert @xcite , @xcite , t5 @xcite , etc. have led to significant advancements in nlp @xcite @xcite @xcite . the effectiveness of these models was evaluated by finetuning them on benchmark datasets @xcite , achieving state-of-the-art (sota) performance across various tasks. recently, large language models (llms) such as @xcite have demonstrated in-context-learning capability without requiring any fine-tuning on taskspecific data. the impressive performance of @xcite @xcite @xcite in fewshot learning scenarios is a major finding as this helps llms to be more efficient, making it possible to use lm-as-a-service @xcite to empower a set of new real-world applications. intuitively, in-context learning works by learning through analogies drawn from the given demonstration examples @xcite . after a largescale pre-training with a self-supervision objective, llms can identify task-level prior patterns from the given prompt and generate a relevant continuation. large-scale pretraining also helps them to acquire emergent capabilities like chain of thought @xcite . however, training only with self-supervision lacks grounding to real-world concepts and may not align well with its inference-time use cases resulting in unhelpful, hallucinated and sometimes toxic output @xcite . thus, instead of learning meta-tasks in an implicit way from raw texts, recent approaches @xcite @xcite @xcite proposed learning tasks in an explicit way with a large scale prompted (supervised) meta-pretraining (a.k.a., instructional tuning) to follow instructions. in addition to that, @xcite proposed to use proximal policy optimization (ppo) to finetune the llm policy with human feedback in a reinforcement learning (rl) framework, introduc-ing gpt-3.5 (text-davinci-003) foot_0 . chatgpt is the latest addition in this series that additionally uses dialog-based instructional data in the supervised and rl-based meta-training stages. chatgpt has shown the ability to solve numerous tasks (e.g., question answering, text summarization, code generation, etc.) as a single model, instigating the question of "is chatgpt turing complete?". despite its impressive capability in performing a wide range of challenging tasks, there remain some major concerns foot_1 about using llms like chatgpt to solve real-world problems @xcite . putting aside their high computational cost, which can be prohibitive in many practical scenarios, a primary concern is that they can fail on simple tasks involving reasoning and commonsense @xcite . second, they can perpetuate biases present in the training data, leading to unfair or prejudiced results. another concern is their ability to be used for malicious purposes, such as generating fake or misleading text. this can be a problem when it comes to misinformation or propaganda generation that could have real-world negative impacts. while many researchers and practitioners have raised such concerns regarding chatgpt, a systematic study evaluating chatgpt's performance on nlp benchmarks is still missing (as of 20 jan, 2023, when the paper was submitted to acl-2023 for reviewing). in this regard, this paper aims to conduct a comprehensive evaluation 4 of chatgpt on benchmark datasets to investigate its effectiveness and limitations in various scenarios, such as language understanding and generation capability, commonsense reasoning, open domain knowledge, the existence of new capabilities, along with studying its potential limitations, such as biases, misinformation generation, ethical concerns and etc. meanwhile, we discover a unique capability that was not reported and analyzed for any llms before. we observe that chatgpt can answer multiple arbitrary (unrelated) knowledge-based queries from a single input prompt (sec. 4). we also report several limitations found in existing datasets while evaluating chat-gpt. in short, we conduct an extensive evaluation by analyzing 255k chatgpt generated responses across 140 benchmark nlp datasets. tasks: we use several benchmark datasets and tasks for a zero-shot evaluation of chatgpt. we categorize our evaluation into two groups: (i) leaderboard-based evaluation, and (ii) task-based evaluation. figure 1 shows the list of all tasks that we used for evaluation in this paper. more details about the tasks and the datasets that we evaluate can be found in appendix c, evaluation: since chatgpt is a conversational language model that gives human-like responses, for most of the tasks (e.g., usually discriminative classification tasks like sentiment analysis), we require human intervention to validate its responses. while for some other tasks (e.g., generative tasks like summarization or machine translation), we only use the available automatic metrics for evaluation. during the initial phase of our evaluation when the chatgpt api was not available, a human annotator went to https://chat.openai.com/ and provided the input prompt. afterward, the chatgpt-generated responses were manually evaluated by at least two annotators against the gold labels. if there was a disagreement, another annotator chimed in and we considered the majority voting. when the api became available, we used the gpt-3.5-turbo model to generate the responses for different datasets. below we describe our evaluation procedure for different types of tasks. for discriminative tasks, after providing an input sample to chatgpt, the generated response is compared against the gold label. though most of the responses generated by chatgpt are evaluated by human annotators, it was challenging to assess all generative responses solely through human annotators in scenarios when the size of the datasets was large. in such cases, we design an evaluation script for the respective dataset to first parse the results and then compare the parsed results with the gold labels. subsequently, any samples where the script could not parse the result properly were manually reviewed by the human annotators. we denote this evaluation approach as evaluation script + human-in-the-loop (see appendix d for details). for generative tasks, such as summarization or machine translation where automatic evaluation metrics like rouge @xcite or bleu @xcite are available, we solely evaluate the performance of chatgpt using these automatic metrics instead of any human intervention. ## polyquery synthesis in this section, we present a unique capability of chatgpt that we discover in the course of our study. specifically, it can identify multiple queries (potentially for different objectives) in a single prompt and retrieve responses for all these queries from the latent representation of the model. retrieving a set of arbitrary information in this way makes it an impressive feature, paving the way to use the chatgpt api in real-world limited-budget scenarios by solving multiple tasks at once based on a single input prompt. to our best knowledge, no prior work investigated this feature of llms. we name this capability as polyquery synthesis. to do a systematic evaluation, we create a small dataset from the efficientqa dev split @xcite and web-questions @xcite test split. for each dataset, we combine 5 different samples into a single sample and create a prompted and non-prompted (non-instructional) input. in total, we use 100 samples from each dataset for evaluation. we also show an example in figure 4 . we generate responses for 13 different models from openai foot_3 ; see ## conclusions and future work this paper evaluates the effectiveness and limitations of chatgpt in standard academic datasets. to our best knowledge, this is the first work that conducts an extensive evaluation of chatgpt in benchmark nlp datasets. we observe that even though chatgpt obtains impressive zero-shot performance across various tasks, it is still far from reaching human-level performance in many tasks. moreover, potential biases and ethical concerns, as well as misinformation generation risks of chat-gpt are discussed. in addition, a unique capability of chatgpt has been studied. though there may have numerous other capabilities of chatgpt that go unnoticed in this paper, future work should nonetheless investigate the capability of chatgpt on more tasks. we will make all our prompts and chatgpt-generated responses publicly available.