Repository: stanford-crfm/helm
Issue: conf file that specifies scenarios included in training data
Root cause: Unimplemented Feature Gap

For each model, we need to record the set of known / suspected scenarios that are included in the model's training data.  These should be highlighted when we present results (with an asterisk or grayed out numbers).  Should also include a description / justification of how we know.  These numbers are still useful for getting upper bounds on performance.

Proposed design:
```
models:
- together/t0
- together/t5pp
scenarios:
- commonsenseqa
description: See Table 5 of https://arxiv.org/pdf/2110.08207.pdf
```
What is excluded is the cross product of models and scenarios.

Note that the scenarios are actually part of the run name along with the arguments (e.g., mmlu:subject=philosophy), so we might have to think a bit more carefully about the semantics of that string.

Would it be too restricting to achieve this at the "group" level? (e.g. `mmlu` as a whole)

It's a good thought.  I worry that it might not be enough?  For example, `mmlu` contains many disparate sources, and it might be the case that one particular subject could be contaminated, but others are okay.  Of course, if one is run, then the whole group is affected, but at least we'll have more resolution.

Sounds good! (In case it helps / is useful, for some scenarios, we also have groups for the "subgroups" (e.g. `NaturalQuestions (open-book)`.)

To begin, it is worth noting the various nuances (of which these are just some quick ones):
(i) One could theoretically discuss overlap at the level of batches/documents. This is too fine-grained for this to be sensible for a researcher-annotated process, though something automated could operate at this level.
(ii) One could discuss overlaps at the level of distinguishable sources as described in the paper introducing the dataset. This most directly comes up in meta-benchmarks (e.g The Pile, RAFT, MMLU, ICE), which themselves are clear aggregations of sources. This is the level of abstraction I see as fairly reasonable. This also implies it is binary whether a model has been contaminated or not, which I think is quite desirable even if reductive. 
(iii) Even when sources are attributed, we will not necessarily be able to resolve underspecified overlaps. For example, are models trained on Wikipedia contaminated with WikiText-103 (assuming it is not specified further). Obviously it is unclear. 
(iv) Of course, even in the case of "contamination", we can have many degrees. For example, the model has seen the training data (doesn't seem to be an issue), the model has seen the test inputs (?), the model has seen the test inputs with outputs (clearly an issue), the model has seen the test contexts (E.g. in open-book QA). Once again, this compromises the simple binary of "contaminated" vs. "not contaminated". 

With all of this said, what I will annotate is cases where the creators of model X have explicitly acknowledged it is contaminated wrt benchmark Y (independent of what precisely that means). Of course this is neither complete nor precise, but it is actionable, coherent, and is what I see as a sensible tradeoff of cost (i.e. my time) vs value. 

Superseded by #746 - I state the concrete condition I am going to use for resolving train-test (notably it is at the scenario level, not the run level, in spite of the theoretical loss of precision implied that is discussed above). 
Practically, I don't think there will be a discrepancy. 

Most importantly, I will fill out the binary contamination matrix of `scenarios x models` with the binary contamination condition I give there (which relates to my message above). 
@dilarasoylu @percyliang Let me know if you think we should use a different condition; I will probably start annotating later today. 

(Note: If I find an example where scenarios vs. runs yield different notions of contamination, I will switch to the finer-grained thing. And we can always implement the finer-grained thing from the coarser-grained annotations if the coarser and finer-grained annotations would yield the same results.)

Hiya Rishi, sorry just getting back on this, the scenario x model matrix sounds good to me.

An example we may want to be have more fine grained annotations is "CommonsenseQA". Perhaps we can address as needed.

Sounds good. By scenario, i mean something that is conceptually a scenario (or has its own line(s) in the conf). I.e. HellaSwag and OpenbookQA are scenario albeit being implemented as special cases of the commonsenseqa scenario (similarly for xsum and cnn/dm under summarization)