Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLM s (2024)

Abstract: Natural Language Processing (NLP) research is increasingly focusing on the use of Large Language Models (LLMs), with some of the most popular ones being either fully or partially closed-source. The lack of access to model details, especially regarding training data, has repeatedly raised concerns about data contamination among researchers. Several attempts have been made to address this issue, but they are limited to anecdotal evidence and trial and error. Additionally, they overlook the problem of indirect data leaking, where modelsare iteratively improved by using data coming from users. In this work, we conduct the first systematic analysis of work using OpenAI’s GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination. By analysing 255 papers and considering OpenAI’s data usage policy, we extensively document the amount of data leaked to these models during the first year after the model’s release. We report that these models have been globally exposed to ∼4.7M samples from 263 benchmarks. At the same time, we document a number of evaluation malpractices emerging in the reviewed papers, such as unfair or missing baseline comparisons and reproducibility issues. We release our results as a collaborative project on https://leak-llm.github.io/, where other researchers can contribute to our efforts.

## introduction the recent emergence of large language models (llms), that show remarkable performance on a wide range of tasks, has led not only to a dramatic increase in their use in research but also to a growing number of companies joining the race for the biggest and most powerful models. in pursuing a competitive advantage, many popular llms today are locked behind api access and their details are unknown @xcite @xcite . this includes model weights @xcite , training data @xcite , or infrastructural details to assess model carbon footprint @xcite . in particular, the lack of information on training data raises important questions about the credibility of llms performance evaluation. the data from which these models learn, typically collected automatically by scraping documents from the web, may contain training, validation, and -most critically -test sets coming from nlp benchmarks. because of this, researchers and stakeholders may later inadvertently evaluate llms on the same data they were trained on. this phenomenon, known as data contamination, may not be an issue in the general use of commercial llms, where adherence to research principles is not mandatory, but it becomes a serious problem when these models are widely used and evaluated in research. unfortunately, many proprietary models are locked behind inference-only apis, making it hard to inspect data contamination. because of this, existing work on the matter mostly focuses on detecting extreme forms of overfitting and memorization, such as the model's ability to generate benchmarks verbatim. these approaches are not only limited but also neglect that recent proprietary llms get iteratively improved from user interactions. if such interactions involve benchmark data (for example when researchers evaluate llms against baselines), the model may, in fact, become contaminated even if it was contamination-free during its initial training. we refer to this phenomenon as indirect data leaking. in this paper, we address the issue of indirect data contamination in closed-source 1 llms by conducting a systematic literature review. we review 255 papers and carefully detail data leakage emerging from them. we focus primarily on the models accessible through openai's chatgpt, foot_1 (gpt-3.5 and gpt-4 foot_2 ) as these are the most frequently used commercial llms in nlp research. by considering openai's data usage policy, we assess how much data was reported to be sent to the models in a way that it could be used for further training, hence giving the models an unfair advantage during evaluation. we also report a series of emergent evaluation malpractices, including lack of comparison with other approaches, differences in the evaluation scale (e.g., evaluating open models on entire benchmarks while comparing to proprietary llms evaluated on samples only), lack of code and data access, or data leakage even in situations where it could be avoided. to our knowledge, this work is the most comprehensive and extensive quantification of the data leakage issue in llms to date. in short, our contributions are as follows: we believe that our work can contribute to ongoing efforts on quantifying llm data contamination by pointing out which datasets are worthy of further investigation. we release our survey results as a collaborative repository, in the form of a webpage at https://leak-llm.github.io/ . it features a list of datasets, detailing the extend of data leakage for each of them. we invite other researchers to contribute any additional known leaks to the list. ## prior work on llm data contamination work on llms data contamination traces back to openai's @xcite , one of the first models with apionly access and limited training data disclosure. despite results hinting at the presence of significant data contamination @xcite , the model has been used extensively in research and the issue was rarely taken into account when interpreting its performance. with the release of chatgpt and following closed-source models to general public, foot_3 the data contamination topic became an even more pressing issue. when a model is closed-source, it becomes implicitly complex to assess data contamination from known benchmarks. therefore, only few practical approaches have been proposed to investigate the issue. one notable example is the lm contamination index, foot_4 featuring a regularly updated estimate of contamination for a list of both open and proprietary models. this approach works by zero-shot prompting the model to generate instances from specific datasets, providing details on the required split and format @xcite . the premise is that no model should be able to replicate specific benchmark formats without having seen them first. more applied approaches have been proposed recently (golchin and surdeanu, 2023), where llms are prompted to complete a given sentence coming from a known benchmark. the completion is then compared with the original reference through text overlap metrics and a statistical test is used to assess if the model is contaminated. although these preliminary works are promising, they cannot be fully trusted and have some limitations. most importantly, they are based on an assessment of the model's ability to generate an example from the benchmark. the recall of such methods can be affected by two issues: ## the issue of indirect data leaking the related work presented in section 2 approaches the issue of data contamination mainly by backtracking models' training data. it is commonly assumed that using benchmarks available only to authorised parties, or datasets being constructed after the chatgpt release, is a guarantee that they have not been leaked. this ignores the fact that models using reinforcement learning from human feedback @xcite , such as those used by chatgpt, are subject to repeated updates @xcite with training data also coming from user interactions. this process leads to a previously overlooked phenomenon, where new data are leaked to the model just through using it. we refer to this problem as indirect data leaking and consider it a new development of the issue for two main reasons: the issue (1) is particularly complex to trace, even with a conscious and targeted effort by the llm vendor. when evaluating a closed-source llm, users often feed the model with test-set samples (with or without labels) surrounded by additional text, such as instructions in the form of prompts. in some cases, especially when evaluating the llm robustness, the test-set samples are perturbed and hence no longer an exact match of their original version. therefore, it is unlikely that llm vendors could effectively exclude leaked benchmarks from further model fine-tuning, especially at scale. for (2), it would be necessary to understand how the llm vendor uses the data to improve the model. a very likely scenario is continued pretraining, where the data leaked by users is treated as an in-domain corpus (and thus given more influence than pretraining data). this procedure is known to improve models' performances in the leaked domains @xcite . notably, @xcite find that fine-tuning a model on in-domain text enriched by textual instructions leads to an increase in the model performance even if gold labels are not shown to the model. this setup perfectly matches the kind of data shown to chat llms when evaluated by researchers. this means that closed-source llms such as gpt-3.5 and gpt-4 can make use of these gold standard examples from widely used nlp benchmarks to gain an unfair advantage over other models. we also point out that recent work @xcite showed that after model updates, chat-gpt performance improved on benchmarks to which it was previously exposed @xcite . with these motivations, we conduct a systematic review to quantify how much of such data the models powering chatgpt could have obtained. ## methodology following the standard systematic review protocol from the medical domain @xcite , we analyse the existing work on llms evaluation to inspect the issue of indirect data contamination and other evaluation malpractices. we focus on openai's gpt-3.5 and gpt-4 models, as they are the most prominently used in recent nlp research. we organize our work into five macro-steps, corresponding to the following subsections. ## results following our methodology, in the first step we identified 255 research papers, 212 of which were found relevant foot_13 during the initial screening (see sec. 4.2). among the relevant papers, 70 (∼ 32%) were peer-reviewed, while the remainder (142) consisted of pre-prints. 15 we subsequently analysed the retrieved papers to examine the problem of data contamination and the adopted evaluation practices. ## suggested practices in closed-source llm evaluation our survey revealed both a significant amount of data leakage in chatgpt and many worrying trends in its evaluation. in light of this, we list a series of suggested practices that we believe could help mitigate the issues. we believe that researchers looking to objectively evaluate llms today should: access the model in a way that does not leak data the first step when planning proprietary llms evaluation should be reading their most upto-date data policies, and access models accordingly (e.g. api instead of web interface for ope-nai's llms). we also acknowledge that in some cases this might not be viable due to budget limits, or an overly steep learning curve for the use of apis by researchers outside of computer science. 21 interpret performance with caution the lack of system specifications and training details can make proprietary llms look like incredibly powerful tools with impressive zero-shot performance. this can often be explained by data contamination @xcite . in our review, we documented that over 4 million samples across more than 200 nlp datasets have been leaked to these models. the performance of closed-source llms should always be interpreted while keeping these results in mind. when possible, avoid using closed-source models we strongly encourage using the available open-source llms. while there has been discussion in the research community about proprietary models being consistently better than open-source ones, we note that (1) this is often driven by hype, while there is evidence of the opposite @xcite , (2) research done solely on closed llms limits scientific progress, bringing benefits mainly to the llm vendors and (3) llm vendors can arbitrarily make changes to the models, e.g., making previous versions unavailable, changing their behaviour in a way that may not be visible to the user @xcite or changing the data treatment policy. adopt a fair and objective comparison evaluating closed-source llms is tied to comparing them with pre-existing approaches. evaluating proprietary models on a limited number of samples while evaluating open ones on dramatically larger sets is scientifically dubious at best. when sampling is required (for example because of budgetary restrictions), it should be applied to all the considered approaches. we also discourage taking state-of-theart values directly from previous work and suggest to re-run all approaches on the considered data only. ## conclusion and future work in this work, we present our findings based on the analysis of 255 papers evaluating the performance of gpt-3.5 and gpt-4. we investigate the problem of indirect data contamination and report that 4.7m samples coming from 263 distinct datasets have been exposed to the models in such a way that this data could be used for training by openai. we also report concerning research practices with respect to reproducibility and fairness. finally, informed by our analysis, we detailed some suggested practices for the evaluation of closed-source llms. future work in our future work, we aim to run experiments via the openai api to see the impact of leaked test data on the performance of gpt-3.5 and gpt-4 on the leaked datasets and the tasks in general. furthermore, we consider investigating indirect data leakage in other closed-source models, namely from anthropic or cohere, which appeared in a small number of papers reviewed in this work.