Repository: stanford-crfm/helm
Issue: Metadata on models (known train-test contamination, reference scores to compare against)
Root cause: Unimplemented Feature Gap

For each model we benchmark, drawn from the list below, I will:
**Model list:**
[BLOOM	T0++	GPT-J	GPT-NeoX	T5	UL2	OPT 66	OPT 175	YaLM	MT-NLG	Anthropic-52	AI21 178	AI21 17	AI21 7.5	GPT-3 175	GPT-3 6.7	GPT-3 1.3	InstructGPT-3 175	InstructGPT-3 6.7	InstructGPT-3 1.3	Codex]


- [x] **Contamination.** Identify any scenarios that are explicitly disclosed as contaminated in their work. As the operational contamination condition, I will take any mention of training on the scenario as a sign of contamination (i.e. I won't distinguish only training on the training set vs. not, or only training on inputs vs inputs + labels, etc.). I will also include some metadata if its not too cumbersome about the nature of the overlap, but the top line item will be a binary marker of overlap
- [x] **Overlap in results.** Identify scenarios in the overlap between their evaluation and our benchmark. Provide reference results for these. With these in hand, we can then verify our results match (I won't do this for right now). Since the evaluation conditions in prior work likely deviate from ours, I will also ideally provide some annotation if it proves to be not too cumbersome.

A corollary is I plan to the second thing in full thoroughness (i.e. identify the full overlap, not just 2 or so datasets), because this will mean we can explicitly say what the preceding coverage is of these datasets (and hence what was missing). I.e. we will go from a sparse matrix of scenarios x models where we had results to a complete matrix (and of course this is only for accuracy since none of these works will have looked at any other metric for these scenarios). So we will get a concrete quantitative measure of what has changed, which will be a nice top-level thing in the paper.  

It would be good to have this in a structured yaml file in the repo.  Do you have thoughts on what this would look like?

For contamination, the structure you gave in #716 seems good to me. To annotate, I probably will do in a spreadsheet and then convert to that structure. 

For reference scores, not sure (I also don't really think they need to be in the repo as they are more of a paper thing imo, so I would be inclined to just have a spreadsheet that we eventually populate with our accuracies on the same scenarios)

Update: Reference scores and contamination information is provided here.
https://docs.google.com/spreadsheets/d/1jpm0Cy0r5Yk_l9o9kwVpxbSfdKrx6ZcQ7iQfs7QYPIc/edit#gid=358703398

Two questions before I convert the contamination information into a yaml file:
1. For the Pile, I assume by construction these models are not trained on the validation/test set. So, in the sense of having seen the particular data in the validation/test set, I expect the models that train on the Pile do not see it. But in general I would like to not have to maintain this nuance, because namely it is a level of granularity not consistently reported in papers and I would like to just have a binary "contaminated vs. not" that we can determine just by whether the model is trained on the scenario's data, independent of whether it was the train or val/test data. So, what do folks think we should do for these annotations?
2. For the remaining contamination, it is for T0++ or GPT-3 and its derivatives. In part, this is because I am not sure which specific Anthropic model we have and whether they did any tuning on task-related data; InstructGPT similarly doesn't mention contamination in relation to their user data. Similarly if AI21-17B is similarly tuned, I am not sure. And for GPT-3, they do their contamination analysis, which I have noted in parentheticals. Do we want all of these to be indicated as contamination (e.g. when the fraction of clean data with no N-gram overlaps based on their filter is >95%)? 

I suggest that for each model, we send the authors the list of models and our guess at what's contaminated and ask for corrections.

I'm not sure about 1 - if only the training split was used, not the val or test split, this seems fine to me...

Just to clarify the latter point, I think it is about the principle we want and whether we want to treat LM datasets as exception. 

Potential principles would be:
1. Declare model `M` to be contaminated on dataset `X` if `M` was trained on any of `X[train] + X[dev] + X[test]`.
2. Declare `M` to be contaminated if trained on any of `X[dev] + X[test]`. 
3. Declare `M` to be contaminated if trained on any of `X[dev][labels] + X[test][labels]`.

As I would see it, 3 is the most cut-and-dry case of contamination and a clear violation of the sanctity of the evaluation. Which matters because it could strongly compromise the generalizability of the findings. 2 w/o 3 (i.e training on the inputs, or passages in QA, or so on) could also inflate things, and it is not obvious to me whether the most sensible action is to call it contamination or not. 1 is clearly too lax of a condition, but the benefit is some works are not terribly clear, so if we are annotating based on papers, it is more conservative (since it flags more things as contaminated necessarily). 

Of course, we could also implement back-off, if that's what is you have in mind. I.e. use condition 1 unless we have more information. The challenge is we need to assume people are reasonably: multiple works say they train on the Pile. I assume this means they train on the Pile training set, but they don't say further, so I cannot verify this strictly.