Paper: To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination (ICLR-2024)

Reviewer comment (Reproducibility):
The methodology is only applied to GPT-3.5/GPT-4, where training details are unknown. In particular, as noted in footnote 1, OpenAI has admitted to using a small amount of data beyond the cutoff date. While I understand the choice of the GPT family as a commonly used model, it would have been better to verify the approach with fully open models where more training details are available (and more trustworthy).

Author response:
**1. Additional LLMs.**
Thank you for the suggestion! We originally focused on GPT-3.5 and GPT-4 for a few critical reasons: (i) relevance to the large community, in and out of research, that rely on these models, (ii) these models’ high performance on code generation tasks. With respect to this second point, we clarify that when a model’s pass rate on these problems is prohibitively low, we are effectively unable to observe non-trivial trends or extract longitudinal insights–and we have found this to be the case for some additional models we have tested. For example, GPT-3.5 achieved 27% pass rate before the cutoff and 13% after,  while GPT-4 achieved 37% pass rate before and 16% after. In contrast, the pass rates we obtained for Text-Davinci-002 with a comparable prompting strategy are less than 1% before *and* after–yielding it impossible to conduct a substantive analysis featuring functional correctness. 

We also are actively collecting data from some other models, including a modern open-source LLM, and plan on providing these results when the results are available soon (we aim to do so before the end of this discussion window). This discussion is also repeated in the new Appendix B.6.

Reviewer comment (Experiments):
The methodology requires underlying datasets that are longitudinal in nature, i.e. release problems/individual tasks over time; this limits the applicability to sources other than Project Euler / Codeforces.

Author response:
**2. Longitudinal datasets.**

We agree that our analysis requires longitudinal datasets; however, we anticipate that the popularity of such benchmarks will continue to rise, as releasing datasets over time is potentially the only foolproof method for avoiding data contamination. For example, creators of recent benchmarks such as [BIG-Bench](https://arxiv.org/abs/2206.04615) are consciously updating their benchmarks over time. Even other non-LLM benchmarks such as [OpenML Benchmarking Suites](https://arxiv.org/abs/1708.03731) are being updated over time, since there is a related push to ensure that the community [does not overfit](https://arxiv.org/abs/2112.01716) to any single set of benchmarks. We are optimistic that as staggered release of benchmark datasets becomes increasingly common and/or an accepted best practice, that the framework we present here can be more widely applied. 

Furthermore, we note that while our analysis can only be conducted on longitudinal datasets, its implications extend to any dataset which can be contained in a webscale training dataset; it leverages longitudinal structure in a test benchmark to reveal the massive contamination effect which can be observed on “seen” examples of any dataset.

Reviewer comment (Presentation):
Particularly in section 2, some citations are formatted differently, with the author names outside the parentheses; in sequences of different citations, readability could be improved by using the same citation format as in section 1.

Author response:
**3. Minor comments.**
Thank you for catching these formatting issues; they have now been fixed in the updated pdf.

Reviewer comment (Presentation):
Stacking log(Github Presence) and log(Difficulty) at the bottom of Figure 1 makes reading the figure tricky. It is suggested to move log(Difficulty) to the right side to improve readability.

Author response:
**W1+2: Fig 1 minor changes.**
Thank you for pointing out these stylistic changes. We have now moved the legend and updated “Github” -> “GitHub”.

Reviewer comment (Presentation):
The proper stylization is "GitHub", not "Github", and this should be corrected in the figure.

Author response:
**W1+2: Fig 1 minor changes.**
Thank you for pointing out these stylistic changes. We have now moved the legend and updated “Github” -> “GitHub”.

Reviewer comment (Presentation):
There is no equivalent plot for Project Euler visible in the main text or appendix. The reviewer asks where it is, as they cannot find it.

Author response:
**W3: Fig 1 for Project Euler.**
Thank you; this plot can be found in Figure 17; the associated plot for GPT3.5 can be found in Figure 19. Here you’ll see how the performance of these systems on Project Euler is much decreased on problems released before the training cutoffs, and entirely wiped out for problems released after the cutoff — neither GPT-4 or GPT-3.5 yield any pass rates above 0 for problems after the training cutoff.

Reviewer comment (Evaluation):
The pass rate is significantly lower for easy and medium problems, even when log(Github Presence) = 0. Since GitHub Presence is used as a proxy for contamination, log(GitHub Presence) = 0 should be the best guess for "low or no contamination," yet there remains a 10–20% decrease in pass rate. Possible explanations could include: (a) GPT-4 genuinely becomes much worse after the knowledge cutoff; (b) GitHub presence is inadequate and/or misleading; (c) the distribution of Codeforce problems changed after GPT-4 was finished pretraining; or (d) something changed in how the pass rate is calculated on generated outputs. The reviewer asks whether the authors can investigate the cause of this shift further.

Author response:
**W4: Pass rate when log(Github Presence)=0.**
This is a great insight! We believe that the main factor at play is b) - GitHub Presence is a strong indicator of availability in the training set, but it still underestimates the true availability in the training set. For example, *all* Codeforces problems show up on the internet in several places (e.g., [here](https://cf.kira924age.com/#/table/), [here in pdf](https://github.com/AliOsm/PDF-CodeForces-Problems)). Therefore, we believe this is a major reason for why the completion rate is higher before the cutoff even for problems with log(GitHub Presence)=0.  We comment on your other hypotheses as well.
1. In terms of the models themselves, we used the exact same models when evaluating problems released before and after the cutoff date (in fact, the evaluations were performed in one large batch and later separated by date).
2. As we state above, we believe that this is the main factor. We agree that “GitHub presence” may under-estimate the total presence of a Codeforces problem in the GPT training data. For example, *all* Codeforces problems show up on the internet in several places (e.g., [here](https://cf.kira924age.com/#/table/), [here in pdf](https://github.com/AliOsm/PDF-CodeForces-Problems)).
3. It is challenging to fully rule out this possibility, but qualitatively, we cannot spot any change over time in the type of Codeforces problems released. It is possible that a difference in problems does cause a small change in pass rate after the cutoff, but we believe that the majority of the observed difference in pass rate can be attributed to (2).
    1. To investigate, we conduct a set of additional experiments to assess whether the distribution over tags (only available for Codeforces) and/or difficulty level (available for Codeforces and Project Euler) changed in a statistically significant way during the post-cutoff period, relative to the pre-cutoff period. We present this analysis in its entirety, consisting of qualitative plots and  $\chi^2$ tests, in Appendix B.7 of our updated submission pdf, and summarize key findings below:
        - For **Codeforces**, we do not find any statistically significant difference in the distribution of normalized counts over problem tags during the pre-vs. post- period. We also do not find any statistically significant difference in the distribution over discretized difficulty scores during the pre-vs. Post-period. 
        - For **Project Euler**, problem tags are not publicly available, so we cannot perform tag analysis. We do not find any statistically significant difference in the distribution over discretized difficulty scores during the pre-vs. Post-period.
        - These findings help to mitigate concerns that the drop-off in performance we observe might be attributable to significant changes in the distribution over tags (for Codeforces) and/or over difficulty levels during the post-cutoff period. 
			
4. We ran the same evaluation script for all problems, and only then separated by date: https://anonymous.4open.science/r/to-the-cutoff-review-253A/eval/chronological_evaluation/chronological_dataset.py (line 162).

Reviewer comment (Evaluation):
The reviewer was hoping for either a qualitative or quantitative analysis of GPT-4’s outputs on Codeforces problems released after the knowledge cutoff. They note there is not even a single example of raw generated outputs. They request that the authors provide some manual examples, even in the appendix, to convincingly demonstrate that GPT-4 is indeed outputting worse code, and to rule out trivial explanations such as formatting changes affecting results.

Author response:
**W5: GPT-generated outputs on problems released after the cutoff.**

To address this question, we would like to: (a) make a minor methodological clarification; and (b) note that we have added a new section to the appendix (B.8) that contains numerous examples of GPT-4 and GPT-3.5.-Turbo output (i.e., generated code) for Codeforces problems released before and after the GPT training cutoff. (We restrict our attention to Codeforces because our Project Euler analysis focused on the correctness of generated numerical solutions, rather than evaluating the outputs of generated code). 

With respect to (a), we note for clarity that what we are trying to control for in our natural experiment are factors with the potential to influence an LLM’s ability to produce a functionally or numerically correct solution to a problem (i.e., the problem’s difficulty), and/or the likelihood that the LLM would have seen this problem during training (i.e., GitHub presence, since scraped GitHub repositories are part of the GPT training corpus). Thus, GPT need not produce *worse* code when we evaluate its ability to produce functionally correct solutions for problems released after the cutoff. Indeed, given that we perform all of our evaluations at a single point in time, the code-generating ability of each GPT instance is “fixed”--what is (potentially) changing is the artificially inflated performance benefit (or lack thereof) the model may demonstrate when it is evaluated on examples it *has* seen during training versus those it has not. 

With respect to (b)---i.e., our new appendix section—because the full description and generated output associated with a given problem can both be quite lengthy, we have constructed this new section by partitioning the Codeforces problems and along two dimensions: (1) release date pre- vs. post GPT cutoff; (2) discretized LLM functional correctness. With respect to (2), for a given LLM and problem, functional correctness is computed as the ratio of test cases that the LLM’s generated code passes (see Section 4 for details). Thus, functional correctness takes values in [0,1.0]. We discretize via the following mapping:

$\lambda: x \mapsto 0 | (x \leq Q1) \lor 1 | (Q1 < x \leq Q2) \lor 2 | (Q2 < x \leq Q3) \lor 3 | x > Q3$

where $x$ represents a given problem's raw difficulty score, and Q1, Q2, and Q3 correspond to the first, second, and third quartiles, respectively. 

We thus consider 8 subgroups per model (i.e., {pre, post} x {0,1,2,3}), and draw two examples uniformly at random from each subgroup. We present each generation along with corresponding problem title, ID, difficulty score, url, and the LLM’s functional correctness score. Our intent here is to avoid cherry-picking of results, while facilitating the visualization of a representative set of generations. We also note that the generated code produced by each model for *every* Codeforces problem is available in the results file that we provide as part of our supplementary material. We view additional/comparative static and behavioral analyses of these outputs as a promising direction for future work.

Reviewer comment (Presentation):
The reviewer finds Tables less effective at communicating results than Figures. Since Tables 1, 2, and those in the appendix are regression tables, they suggest switching to Forest plots of regression coefficients. They provide example links and describe a concrete visualization suggestion: X-axis as estimated parameters with confidence intervals, Y-axis as covariates (Difficulty & GitHub Presence), Hue indicating Before Cutoff or After Cutoff, and two side-by-side axes for GPT-4 and GPT-3.5.

Author response:
**Table 1.**
Thanks once again for this suggestion! We have now added these figures. We replaced Table 1 and Table 2 with their corresponding forest plots. Additionally, we have added the forest plots into the appendix for every regression coefficient table. These figures give new visual insights so the reader can quickly understand the relative relationship between the regression coefficients. For example, we can easily see that the coefficients for both Difficulty and GitHub presence get closer to 1 (the null effect) after the cutoff. 

Thank you once again, for your excellent points. Please let us know if you have any follow-up questions or further suggestions. We are still conducting new analyses and are excited to update you with their results. We look forward to continuing the discussion.

Reviewer comment (Novelty):
The results are interesting but not that surprising. Many blogs or discussion in the community about Data Contamination has involved similar results.

Author response:
**1. results interesting but not surprising.**
Thank you, we agree that the results are interesting. While we understand that the results may not be surprising to you, they are to other reviewers; XbnJ described “the findings are surprising, revealing that people should deal with the ability of LLMs more carefully”. More importantly, our work is a non-trivial contribution to the community that presents the first scientifically rigorous confirmation of the phenomena that social media posts have speculated about via small-scale/ad hoc analyses. While many informal statements have been made about GPT-4 memorization, we show the first statistically significant differences in performance on problems released before and after the cutoff date. Furthermore, our work lays the groundwork for rigorous analysis of contamination and best practices in the age of LLMs trained on webscale data. Our methodology is becoming increasingly relevant as the community acknowledges the limitations of static benchmarks for LLM evaluation, and shifts toward dynamic/longitudinal benchmarks such as the ones we construct, open-source, and analyze here.

Reviewer comment (Evaluation):
There is lack of depth analysis about how implicit contamination is possible. If some real examples can be extracted to show how this could happen, it will be much better.

Author response:
**2. On how implicit contamination is possible.**
To begin, we’d like to clarify our intention for including the words “implicit” and “explicit” in our abstract, specifically in reference to how a given problem might come to be included in a model’s training corpus. Our intention here was to acknowledge that the process of collecting webscale data can lead to the inclusion (in the training set) of examples that were never *intentionally* selected for training–for example, we consider the unintentional inclusion of BIG-bench examples in the GPT-4 training corpus ([OpenAI, 2023](https://arxiv.org/abs/2303.08774)). Through re-posting, removal of watermark information, indirect scraping, or other means, information intended to be omitted from LLM scraping is rarely truly safe. For clarity, we have modified our abstract text to refer to examples that are “intentionally” (explicitly) or “unintentionally” (implicitly) included in the training data. 

On a separate note, we have newly added Appendix B.8 which includes a number of samples of generated outputs from GPT-4 and GPT-3.5-Turbo. These can be used to qualitatively examine their outputs across a range of pass rates on both datasets.

Reviewer comment (Novelty):
The contribution of this paper is not enough to be accepted by ICLR, for its limited scope and technical novelty.

Author response:
**3. On scope and novelty.**
Thank you for your comments. We focus on code generation for a few key reasons: (i) it is a very popular use-case; (ii) most recent LLMs have code as a major part of their training data; (iii) unlike general online natural language text, GitHub has a uniform interface and is easily scrapable and cleanable, making it simultaneously easy for GitHub solutions to be added to train datasets as well as for us to assess GitHub presence; and (iv) code generation and problem solving datasets have objective correctness metrics (test cases) producing objective evaluations of open-ended generations that don’t require another model or human in the loop. 

We also note that dataset desiderata for the methodological approach we propose include: (i) problems must have been released over a sufficiently long time-horizon, such that it is possible to partition examples into pre- and post- GPT training cutoff subsets based on problem release date.  This restrictions precludes some other popular benchmarks which have been released in a single time-step (e.g. [HumanEval](https://github.com/openai/human-eval) or [MBPP](https://arxiv.org/pdf/2108.07732.pdf)). (ii) problems should consist of high-quality questions requiring non-trivial solution generation that admit objective evaluation functions/correctness measures.
We have thus focused our efforts on the non-trivial scraping and processing required to analyze Project Euler and Codeforces datasets in the way that we have. We also emphasize that our work introduces tools and paves the way for further rigorous study related to data contamination, which gives our work the potential to have high impact in the community.

We’d also like to highlight that our paper should also be viewed as a methodological one. Specifically, we believe our work takes a novel view on data contamination estimation by employing a natural experiment – a concept borrowed from the economic literature. We use this concept effectively here and argue for its further use, particularly in experimenting, evaluating, and discovering the intricacies of LLMs. We argue that this evaluation methodology should be used in particular on closed-source models which refuse to reveal critical and important details about their development/training. We have updated our manuscript to highlight this point further.

Reviewer comment (Experiments):
The scope is limited to Code datasets.

Author response:
**3. On scope and novelty.**
Thank you for your comments. We focus on code generation for a few key reasons: (i) it is a very popular use-case; (ii) most recent LLMs have code as a major part of their training data; (iii) unlike general online natural language text, GitHub has a uniform interface and is easily scrapable and cleanable, making it simultaneously easy for GitHub solutions to be added to train datasets as well as for us to assess GitHub presence; and (iv) code generation and problem solving datasets have objective correctness metrics (test cases) producing objective evaluations of open-ended generations that don’t require another model or human in the loop. 

We also note that dataset desiderata for the methodological approach we propose include: (i) problems must have been released over a sufficiently long time-horizon, such that it is possible to partition examples into pre- and post- GPT training cutoff subsets based on problem release date.  This restrictions precludes some other popular benchmarks which have been released in a single time-step (e.g. [HumanEval](https://github.com/openai/human-eval) or [MBPP](https://arxiv.org/pdf/2108.07732.pdf)). (ii) problems should consist of high-quality questions requiring non-trivial solution generation that admit objective evaluation functions/correctness measures.
We have thus focused our efforts on the non-trivial scraping and processing required to analyze Project Euler and Codeforces datasets in the way that we have. We also emphasize that our work introduces tools and paves the way for further rigorous study related to data contamination, which gives our work the potential to have high impact in the community.

We’d also like to highlight that our paper should also be viewed as a methodological one. Specifically, we believe our work takes a novel view on data contamination estimation by employing a natural experiment – a concept borrowed from the economic literature. We use this concept effectively here and argue for its further use, particularly in experimenting, evaluating, and discovering the intricacies of LLMs. We argue that this evaluation methodology should be used in particular on closed-source models which refuse to reveal critical and important details about their development/training. We have updated our manuscript to highlight this point further.

Reviewer comment (Novelty):
The only novelty is how to split the "train" and "test" set.

Author response:
**3. On scope and novelty.**
Thank you for your comments. We focus on code generation for a few key reasons: (i) it is a very popular use-case; (ii) most recent LLMs have code as a major part of their training data; (iii) unlike general online natural language text, GitHub has a uniform interface and is easily scrapable and cleanable, making it simultaneously easy for GitHub solutions to be added to train datasets as well as for us to assess GitHub presence; and (iv) code generation and problem solving datasets have objective correctness metrics (test cases) producing objective evaluations of open-ended generations that don’t require another model or human in the loop. 

We also note that dataset desiderata for the methodological approach we propose include: (i) problems must have been released over a sufficiently long time-horizon, such that it is possible to partition examples into pre- and post- GPT training cutoff subsets based on problem release date.  This restrictions precludes some other popular benchmarks which have been released in a single time-step (e.g. [HumanEval](https://github.com/openai/human-eval) or [MBPP](https://arxiv.org/pdf/2108.07732.pdf)). (ii) problems should consist of high-quality questions requiring non-trivial solution generation that admit objective evaluation functions/correctness measures.
We have thus focused our efforts on the non-trivial scraping and processing required to analyze Project Euler and Codeforces datasets in the way that we have. We also emphasize that our work introduces tools and paves the way for further rigorous study related to data contamination, which gives our work the potential to have high impact in the community.

We’d also like to highlight that our paper should also be viewed as a methodological one. Specifically, we believe our work takes a novel view on data contamination estimation by employing a natural experiment – a concept borrowed from the economic literature. We use this concept effectively here and argue for its further use, particularly in experimenting, evaluating, and discovering the intricacies of LLMs. We argue that this evaluation methodology should be used in particular on closed-source models which refuse to reveal critical and important details about their development/training. We have updated our manuscript to highlight this point further.