Repository: stanford-crfm/helm
Issue: ensure aggregate tables are meaningful
Root cause: Architectural Constraint

At the moment, we aggregate stuff within a scenario group into the top table for that group. In some cases (e.g. contrast sets), this doesn't make sense, and in some cases it is of debatable value (e.g. ice, twitterAAE), and in some cases there is weird redundancy (E.g. we run bAbI and CivilComments on "all").  

Low priority, but should be not hard to specify somewhere and turn of aggregation accordingly. 

I iterated over the list of scenario groups:
[BoolQ](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=boolq) - We aggregate contrast sets and no contrast sets. **This does not make much sense.**
[NarrativeQA](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=narrative_qa) - No aggregation
[NaturalQuestions](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=natural_qa) - I don't think aggregation formats makes much sense.
[NaturalQuestions (closed-book)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=natural_qa_closedbook) - No agg
[NaturalQuestions (open-book)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=natural_qa_openbook_longans) - No agg
[QuAC (Question Answering in Context)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=quac) - no agg
[HellaSwag](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=hellaswag) - No agg
[OpenbookQA](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=openbookqa) - No agg
[TruthfulQA](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=truthful_qa) - No agg
[MMLU (Massive Multitask Language Understanding)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=mmlu) - Agg makes sense
[CNN/DailyMail](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=summarization_cnndm) - No agg
[XSUM](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=summarization_xsum) - No agg
[IMDB](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=imdb) - We aggregate contrast sets and no contrast sets. **This does not make much sense.**
[RAFT (Real-world Annotated Few-Shot)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=raft) - Agg makes sense
[CivilComments](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=civil_comments) - Agg is a little weird because we do "all" but not unreasonable. 
[ICE (International Corpus of English)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=ice) - Agg is a little weird because we stratify by nationality (e.g. Hong Kong) but also by gender (e.g. male), so its a little funny but probably fine.
[The Pile](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=the_pile) - Agg is fine
[TwitterAAE](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=twitter_aae) - We aggregate and we could say we don't care and only care about the performance disparity, but I think aggregating is fine
[BLiMP (The Benchmark of Linguistic Minimal Pairs for English)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=blimp) - Agg makes sense
[WikiFact](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=wikifact) - Agg makes sense
[bAbI](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=babi_qa) - Agg is a bit weird because of all but is reasonable
[Dyck](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=dyck_language) - No agg
[Synthetic reasoning (abstract symbols)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=synthetic_reasoning) - Agg is a bit strange because we do different modes (e.g. pattern_match) but also difficulties (e.g. easy/hard). A bit weird but seems ok.
[Synthetic reasoning (natural language)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=synthetic_reasoning_natural) - Agg is fine
[GSM8K (Grade school math word problems)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=gsm) - No agg
[MATH](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=math_regular) - Agg is fine
[MATH (chain-of-thoughts)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=math_chain_of_thought) - Agg is fine
[APPS (Code)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=code_apps) - no agg
[HumanEval (Code)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=code_humaneval) - no agg
[LegalSupport](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=legal_support) - no agg
[LSAT](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=lsat_qa) - no  agg
[Data imputation](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=entity_data_imputation) - agg is fine
[Entity matching](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=entity_matching)  -agg is fine
[Copyright](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=copyright) - **Agg is fine, but we merge code copyright with text copyright. This should be fixed #906**
[Disinformation (reiteration)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=disinformation_reiteration) - agg is fine
[Disinformation (wedging)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=disinformation_wedging) - no agg
[BBQ (Bias Benchmark for Question Answering)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=bbq) - no current agg; we include all but its fine
[BOLD (Bias in Open-Ended Language Generation Dataset)](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=bold) - no current agg; we include all but its fine
[RealToxicityPrompts](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=real_toxicity_prompts) - no agg
[Synthetic efficiency](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&group=synthetic_efficiency) - N/A

**Takeaways.**

1. We definitely should fix the copyright issue: #906 (@lxuechen)
2. We should fix the natural questions issue. I will submit a PR.
3. We should fix the contrast sets issue - @dtsip
4. We could make some changes for CivilComments, ICE, bAbI, synthetic reasoning (abstract), BBQ, and BOLD. I don't think is critical; if anything, I think the ICE/synthetic reasoning (abstract) are potentially the more strange ones and I don't have any serious issue with what we have (i.e. the aggregate is not ridiculous as a quantity, just a bit awkward).

So, I will close this issue with the PR for 2, make a P2 issue for 3 that I will assign to Dimitris for now, and make a p3 issue for 4.