Can we trust the evaluation on C hat GPT ? (2023)

Abstract: ChatGPT, the first large language model with mass adoption, has demonstrated remarkableperformance in numerous natural language tasks. Despite its evident usefulness, evaluatingChatGPT’s performance in diverse problem domains remains challenging due to the closednature of the model and its continuous updates via Reinforcement Learning from HumanFeedback (RLHF). We highlight the issue of data contamination in ChatGPT evaluations, with a case study in stance detection. We discuss the challenge of preventing data contamination and ensuring fair model evaluation in the age of closed and continuously trained models.

## introduction chatgpt @xcite has become the most prominent and widely-adopted pre-trained large language model (llm) thanks to its impressive capabilities to perform a plethora of natural language tasks and its public accessibility. although significant concerns regarding llms, particularly their tendency to "hallucinate" (or "making things up") and generation of biased or harmful content in scale have been raised @xcite , chatgpt is becoming a common tool not only for everyday tasks such as essay writing, translation, and summarization @xcite , but also for more sophisticated tasks such as code generation, debugging @xcite , and mathematical problem-solving @xcite . with more than 100 million users within two months after its launch @xcite and its abilities pass hard exams like bar exam @xcite and medical licensing exam @xcite , chat-gpt has stirred public perception of ai and has been touted as the paradigm for the next-generation search engine and writing assistant, which is already being tested by microsoft's bing search and office products @xcite . beyond commer-cial interests, llms are also being tested for assisting scientific research @xcite @xcite . although openai-the creators of chatgptperformed internal tests, they do not cover all problem domains. although the excellent general performance of chatgpt is evident, it is still important to quantitatively characterize its performance on specific tasks to better understand and contextualize the model. note that, given that it is currently not possible for a user to fine-tune chatgpt, one can only evaluate it with a few-shot/zero-shot setting-a highly desirable setting that requires close to no annotated data. a recent study showed that although chatgpt performs generally well in many tasks, it has different strengths and weaknesses for different tasks and does not tend to beat the sota models @xcite . however, given that the chatgpt is a closed model without information about its training dataset and how it is currently being trained, there is a large loxodonta mammal in the room: how can we know whether chatgpt has not been contaminated with the evaluation datasets? preventing data leakage (training-test contamination) is one of the most fundamental principles of machine learning because such leakage makes evaluation results unreliable. it has been shown that llms can also be significantly affected by data leakage, both by the leakage of labels and even by the leakage of dataset without labels @xcite @xcite . given that the chatgpt's training datasets are unknown and that chatgpt is constantly updated, partly based on human inputs from more than 100 million users via reinforcement learning from human feedback (rlhf) @xcite , it is impossible to ascertain the lack of data leakage, especially for the datasets that have been on the internet. as far as it has been known, chatgpt is trained in a three-step process. first, an initial llm (gpt 3/3.5) is fine-tuned in a supervised manner on a dataset curated by asking hired human annotators to write what they think is the desired output to prompts submitted to the openai api. 1 next, a set of prompts is sampled from a larger collection of prompts submitted to the openai api. for each prompt in this set, the llm produces multiple responses, which are then ranked by human annotators who are asked to indicate their preferred response. the second step then trains a reward model (rm) on this dataset of response-ranking pairs to mimic the human ranking. this step keeps the llm frozen and solely trains the rm. finally, the llm is made to generate responses to a set of prompts, which were not included in the previous steps, but submitted to the openai api nevertheless. the now-frozen rm is used as a reward function, and the llm is further fine-tuned to maximize this reward using the proximal policy optimization (ppo) algorithm @xcite . thus, if openai continuously updates its models, by using queries submitted by researchers who wanted to evaluate chatgpt's performance on various natural language processing (nlp) tasks, it is likely that chatgpt is already contaminated with the test datasets of many nlp tasks, which can lead to performance overestimation in nlp tasks. such contamination has been documented in the training data of other language models @xcite @xcite . 2 it is important to highlight a distinction between two kinds of contamination acknowledged in literature @xcite : (1) the case where both the task input and labels are leaked to the model via training versus (2) the case where just the input is exposed. the latter is surely a smaller concern. however, even without the correct labels, exposure to the text in the same domain has been documented to increase the performance of the model to the corresponding nlp task @xcite . although we do not have any documented evidence that the ground-truth output answers/labels of the nlp tasks were submitted to the platform and the chatgpt model has been trained with such data, we cannot exclude such possibility either. the annotator-generated responses to queries submitted to openai during the rlhf step 1 additional labeler-written prompts are included too. 2 https://archive.is/44rra could potentially match the input text with output labels of the right kind; it is not possible to ensure no one has exposed certain input-label pairs to the model, for instance, via a few-shot learning experiment. given that language models show competitive performance in classification tasks despite poorly labeled data @xcite , we cannot discard the possibility that the rlhf pipeline might essentially be a weaker variant of type (1) contamination. here, we use a case study of a stance detection problem @xcite to raise awareness on this issue of data leakage and ask a question about how we should approach the evaluation of closed models. stance detection is a fundamental computational tool that is widely used across many disciplines, including political science and communication studies. it refers to the task of extracting the standpoint (e.g., favor, against, or neither) towards a target from a given text. the task becomes more challenging when the texts are from social media like twitter because of the presence of abbreviations, hashtags, urls, spelling errors, and the incoherent nature of tweets. recent studies have claimed that chatgpt outperforms most of the previous models proposed for this task @xcite on a few existing evaluation datasets, such as the semeval 2016 task6 dataset @xcite and p-stance @xcite , even in a zero-shot setting where the model was not fine-tuned on the task-specific training data. can this result be due to the data leakage and contamination of the model? could this study itself have contaminated the chatgpt model? although it is not possible to definitely answer these questions, it is also impossible to rule out the possibility of contamination without the model owners' in-depth analysis. following its release on nov 30 th 2022, on dec 15 th 2022, jan 9 th , jan 30 th , feb 9 th , and feb 13 th 2023, chatgpt has been updated multiple times. 3 while most of these releases updated the model itself, it is our understanding that the february releases were about handling more users to the platform, optimizing for speed, and the offering of chatgpt plus-a subscription plan which provides priority access to new features, and faster response times. 4 given that there has been at least one study that evaluated chatgpt's performance on stance detection tasks @xcite , and that newer versions of chatgpt are more likely to be exposed to past queries to the platform, an opportunity arises to test whether the performance of the newer versions of chatgpt on stance detection has been substantially improved after the study by @xcite . as we will present below, we do see an overall improvement in the performance before and after the publication of the stance detection evaluation paper @xcite . of course, there is an alternative explanation that the model simply got better. however, we would also like to note that openai has been updating the model primarily to address the model's problematic behaviors by making it more restricted, which led to the observation, although largely anecdotal, that the model has become 'less impressive.' ## methods given that zhang et al., 2022 was released on arxiv on december 30, 2022, and chatgpt was launched on november 30, 2022, we assume zhang et al., 2022 used either the november 30 or december 15 version of chatgpt (henceforth called v1) to obtain their results (fig. 1 ). following their work, we used the test sets of semeval 2016 task 6 @xcite and pstance @xcite to perform our experiments. the semeval 2016 task 6 dataset consists of relevant tweets in english with stance annotations towards six targets-'hillary clinton,' 'feminist movement,' 'legalization of abortion,' 'climate change is a real concern,' 'atheism,' and 'donald trump.' similarly, the p-stance dataset contains english tweets with stance annotations towards three targets-'donald trump,' 'joe biden,' and 'bernie sanders.' we also used the same prompt. specifically for semeval 2016 task 6, for instance, given the input: "rt gunnjessica: because i want young american women to be able to be proud of the 1st woman president #semst", the input to chatgpt is: "what's the attitude of the sentence: 'rt gun-njessica: because i want young american women to be able to be proud of the 1st woman president #semst' to the target 'hillary clinton'. select from "favor, against or neutral". similarly, since the p-stance dataset does not have a neutral stance, the prompt is slightly modified to "what's the attitude of the sentence: 'air borne illnesses will only become more common with climate change. we need to immediately address this and fight for medicare for all or this could be the new normal. #berniesanders' to the target 'bernie sander'. select from "favor, or against". 5 since chatgpt did not provide an api to collect data at the time of the experiment, we first manually collected the responses of jan 30th chatgpt for 860 tweets from the test data of semeval 2016 task 6, pertaining to the targets, 'hillary clinton (hc),' 'feminist movement (fm),' and 'legalization of abortion (la)' and extract the stance label from them. while the test set contains tweets pertaining to other targets ('atheism,' 'donald trump,' 'climate change is a real concern'), we sampled the 860 tweets pertaining to the targets used in the previous work @xcite . after manual inspection of the preliminary results of the 860 tweets, we decided to collect and include the responses for the 2,157 tweets in the p-stance test dataset in our analysis, but the jan 30th chatgpt version was no longer available by then. nevertheless, we use an open-source api foot_3 to automate the collection of responses from the feb 13th chatgpt plus for both the p-stance and semeval 2016 task 6 datasets. then we manually go through these (often verbose) responses to extract the stance labels from them when explicitly mentioned. in sum, we were only able to use the feb 13th chatgpt plus version for the p-stance dataset and the jan 30th chatgpt and feb 13th chatgpt plus version for the semeval 2016 task 6 dataset because openai (1) does not provide access to its older models after newer models are released, (2) imposes an upper bound on the number of requests which can be submitted to the platform in an hour, and, at the time of this experiment, (3) lacked a public api which in turn hindered the speed and efficiency of data collection. ## evaluation metric and results the macro-f and micro-f scores are shown for different versions of chatgpt in a zero-shot setting on semeval 2016 task 6 and p-stance datasets in overall, we see an improvement in performance, measured using the micro-f and macro-f scores, in recent versions of chatgpt compared to v1. in particular, we see an average of 12.46 and 8.6 point improvement in the micro and macro-f scores, respectively, when comparing jan 30 chatgpt to v1 on the semeval task. we see a smaller but nonnegligible improvement-6.1 point on the micro-f and 1.89 point on the macro-f-when comparing feb 13 chatgpt plus to v1 on the same task. fig. 2 also shows the temporal evolution of zero-shot performances of various models on selected targets of semeval. the macro-f scores of the models are taken from the previous work @xcite . although it is still difficult to conclude with only a few data points, we see a significant jump in the zero-shot capability of chatgpt when compared to previous models. given that chatgpt is based on instructgpt3 in which some nlp dataset contamination was already documented @xcite , this raises further concerns if v1 too may have been contaminated. a similar plot for the micro-f scores is not shown here due to our pending uncertainties of scores indicated in the previous work @xcite (see appendix a.1) and the general unavailability of micro-f scores by other models. on the p-stance dataset, we observe a 0.74-point improvement in the micro-f scores and a 0.26 point in the macro-f scores when comparing feb 13 chatgpt plus to v1. in sum, the improvement is greater for semeval than for the p-stance dataset. on the semeval dataset, we also observe a performance drop by feb 13 chatgpt plus relative to jan 30 chatgpt. even though the performance has dropped, it is still quite an improvement compared to v1. ## discussion and conclusion in this article, we discuss the reasons why we cannot trust the evaluation of chatgpt models at its face value due to the possibility of data leakage. first, the closed nature of the model makes it impossible to verify whether any existing dataset was used or not. second, with a constant training loop, it is also impossible to verify that no researchers or users have leaked a particular dataset to the model, especially given the sheer scale of availability of the model (more than 100 million users foot_4 at the time of writing). any evaluation attempt using chatgpt may expose the very evaluation dataset to chatgpt, potentially making all subsequent evaluations unreliable. note that even the mere exposure of the input may make evaluation unreliable @xcite . therefore, unless the evaluation is completely novel, it is difficult to ensure the lack of data leakage to the model. given that data leakage likely leads to a boost in estimated performance, we did a case study where there could have been potential contamination, with documented evidence that researchers performed an evaluation of chatgpt with an existing test dataset. in other words, the stance detection task figure 2 : evolution of zero-shot performance, measured using the macro-f score, on the semeval 2016 task 6a by various models. scores of the previous models are taken from @xcite . this work sheds light on a bigger problem when it comes to using chatgpt and similar large language models on nlp benchmarks. given these models are trained on large chunks of the entire web, care must be taken to ensure that the pretraining and fine-tuning data of these models are not contaminated by the very benchmarks their performance is often tested on. given the results showing that even benign contamination can lead to measurable differences, making claims about these models' zero-shot or few-shot inference capabilities require a more careful inspection of the training datasets of these models. for example, the bigbench dataset @xcite attempts to address this issue by accompanying the benchmark data with a special string ("canary" string). the purpose of this string is to allow researchers to better filter big-bench tasks out of the training data for large language models. this string also makes it possible to probe whether a language model was trained on big-bench tasks, by evaluating whether the model assigns anomalously high or low probabilities to the string. 8 yet, checking for data contamination is becoming increasingly challenging because the most prominent language models, like 8 big-bench canary string: https://archive.is/cbgl2 chatgpt and the recently released gpt-4, foot_5 are closed and more models are following the practice. while our work is not without limitation (see 'limitations' section), we would like to underline that our primary goal of this article is to highlight the ample possibility of data leakage and the impossibility of verifying the lack of data leakage with a closed model. as long as the trend of closed models and continuous training loop continues, it will become more challenging to prevent data leakage (training-test data contamination) and ensure fair evaluation of models. therefore, in order to ensure the fair evaluability of the models, we argue that the model creators should (1) pay closer attention to the training datasets and document potential data contamination, (2) create mechanisms through which the training datasets and models can be scrutinized regarding data leakage, and (3) build systems that can prevent data contamination from user inputs.