zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Create script to verify that Scenarios are reproducible between runs
Root cause: Unimplemented Feature Gap

Script will run

```
benchmark-present  --suite run1 --max-eval-instances 100 --dry-run --conf-path  src/benchmark/presentation/run_specs.conf
benchmark-present  --suite run2 --max-eval-instances 100 --dry-run --conf-path  src/benchmark/presentation/run_specs.conf
```

Output files and prompts (e.g., scenario.json) in `run1/` vs. `run2` should be identical.

We should do this now. A handful of issues came up regarding reproducibility.
Repository: stanford-crfm/helm
Issue: Disinformation reiteration scenario names do not include topic.
Root cause: Algorithmic Error

The names of the scenarios for the disinformation reiteration scenarios do not include the topic ("covid" or "climate"). This means that on the benchmarking website, they are not differentiated---both link to the same generations (for "climate").

<img width="1223" alt="image" src="https://user-images.githubusercontent.com/10811150/170879265-5b39382a-04bb-40fb-b865-1afe4f1cb3b8.png">


Resolved by https://github.com/stanford-crfm/benchmarking/pull/474
Repository: stanford-crfm/helm
Issue: CommonsenseQA metrics can be made general purpose
Root cause: Architectural Constraint

The metrics used for CommonsenseQA scenario seem like they could be refactored into multi-purpose metrics that other scenarios where more than one instance is produced per data-point can make use of. 

What exactly is this referring to?  I hope with #550 , we won't need CommonsenseQA metrics.

Oh, got it, if that will replace all the CommonsenseQA metrics, then there is no need for this issue.

@percyliang, I was originally considering:

* Switching `CommonsenseQA` to use the `MultipleRequestInstance` (defined [here](https://github.com/stanford-crfm/benchmarking/blob/7c88f0c3b43eb2482817c75b663e1d64e921aba4/src/benchmark/scenario.py#L104)) as opposed to the vanilla `Instance`.
* Moving the metrics to `multiple_request_metrics.py` ([here](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/multiple_request_metrics.py)), so that we re-use code for all the scenarios using multiple requests.


Closing, since I expect #550 will address this in the way we want.
Repository: stanford-crfm/helm
Issue: Perturb references in Individual Fairness Perturbations
Root cause: Validation Gap


@dilarasoylu 

More broadly, can you look through the results (and if we don't have enough results, can you coordinate with Tony L. to get results for these even if that means doing some runs beyond the big runs) to verify:

(a) We are good to go wrt all hyperparams for all the perturbation methods. E.g. you mentioned something about swapping things non-deterministically with some probabilities for the Diyi AAE stuff that should be resolved.
(b) We are good to go wrt decisions beyond hyperparams. E.g. I remember there was some weird with tokenizers/detokenizers when we looked at the results, and we should also make sure things like perturbing references are done in the right ways (which presumably should match w.e. robustness is doing, but we should look at the results ourselves). 

If you need help, we can get robustness folks to look at the results with you or I can find some time too, but it is quite important we really have meaningful notions of fairness since this is the scalable type of fairness in our benchmark (whereas the performance disparities stuff is very restricted to ICE and perhaps CivilComments).
Repository: stanford-crfm/helm
Issue: Missing entries in `training_efficiency.json` for the AI21 models
Root cause: Unimplemented Feature Gap

AI21 models:

- ai21/j1-jumbo
- ai21/j1-grande
- ai21/j1-large

We're getting lots of warnings in the logs:

```
WARNING: tried to estimate training CO2 emissions for model ai21/j1-jumbo that is not in training_efficiency_dict
WARNING: tried to estimate training CO2 emissions for model ai21/j1-jumbo that is not in training_efficiency_dict
WARNING: tried to estimate training CO2 emissions for model ai21/j1-jumbo that is not in training_efficiency_dict
WARNING: tried to estimate training CO2 emissions for model ai21/j1-jumbo that is not in training_efficiency_dict
```

The AI21 folks haven't released any details on how they trained their models, so not sure what to do here.

> The AI21 folks haven't released any details on how they trained their models, so not sure what to do here.

If you have a list of questions to ask, maybe @percyliang and I can reach out to them.

Percy already asked them earlier, and they said they couldn't reveal anything that is not in their paper. Maybe they are more willing now though?

Not fully in-the-loop on this, so forgive the naive questions:

1. @deepakn94 Do we have the questions we would want to send to AI21 (namely do Percy and Tony L. have them?)?
2. @percyliang @teetone Not sure when you last asked, but seems like we may as well ask or decide we are not going to ask. Regardless, let's decide and resolve this. 

We would like estimates of how many GPUs (and what type) they trained their model on, how long it took, and ideally also what region those GPUs were in if they were in the cloud.

Great, then let's resolve whether we are emailing and close this! (Can address during meeting.)

Closing for now
Repository: stanford-crfm/helm
Issue: Add entries for `gooseai/gpt-neo-20b` and `gooseai/gpt-j-6b` for Efficiency
Root cause: Unimplemented Feature Gap

To the static files at `src/benchmark/static`.

@deepakn94 We're currently using a free GooseAI account for https://crfm-models.stanford.edu, but I think it might be enough for now. I set the total GooseAI quota to 500K for your account.

This should already be handled for inference: https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/static/inference_efficiency.json#L322.

Looks like I need to handle it for training.
Repository: stanford-crfm/helm
Issue: Decide on the metrics for each scenario
Root cause: Unimplemented Feature Gap

## Purpose

We need to decide on the exact metrics (and perturbations) we will run on each scenario (CC @dtsip, @percyliang, @teetone, @rishibommasani), and update `run_specs.py` / `run_specs.conf` accordingly (CC @dilarasoylu @rishibommasani).

For the generic scenarios (we can also address team-specific but let's not get distracted by that to start), where by accuracy I mean the appropriate metric (e.g. ppl, accuracy, F1), here is my proposal:

Language modelling (i.e. WikiText, Pile, ICE, possibly TwitterAAE) - Acc, Eff
QA and RC (i.e. NaturalQuestions, CommonsenseQA {HellaSwag, OpenBookQA, ...}, MMLU, BoolQ, NewsQA, NarrativeQA, QuAC) - Acc, Robustness, Fairness, Eff
IR (TREC, MS MARCO) - Acc, Eff
Summarization (XSUM, CNN/DM) - Acc, Bias, Toxicity, Eff
Text Classification (RAFT, IMDB) - Acc, Robustness, Fairness, Eff

**Analysis.**
The principle is Acc, Eff for everything (which we already are actively measuring),  Robustness/Fairness for anything but where it is messy, which is often generative (i.e. language modelling/summarization) and likely not for IR since IR already is quite token-intensive due to multiplicative factors and in itself is already new in using LMs directly in this way, and Bias/Toxicity for anything generative. 

**TL;DR** This implies that for Acc and Eff we hit everything, for bias/toxicity we hit everything that type-checks, and for robustness/fairness we hit everything that type-checks (I think the stuff we exclude is very sensible/it is not particularly clear to me that there is anything that is more "right" that could be scaled to a large collection of tasks in a sensible fashion). So, as far as I am concerned, this is optimal and I don't see a reason that inhibits this beyond token expenditure if we broaden the amount of stuff we do perturbations on (which I have been mentioning for quite a while as likely the most concerning thing wrt token usage due to the multiplicative factor implied by augmentations). 




Update: we have resolved this for all scenarios (as described above except we also do robustness/fairness for IR). 

The remaining action item is we need to add toxicity and bias to the summarization scenarios. 
1. We need to merge #414 ASAP (@ryanachi).
2. Once merged, we need to add toxicity and bias to the list of metrics for the summarization scenarios (@fladhak).
Repository: stanford-crfm/helm
Issue: Add stats with individual instances
Root cause: Unimplemented Feature Gap

Add statistics with the individual instances too and serialize them out. See metric.py.

This would be extremely useful for the efficiency team as we are trying to figure out how accurate our latency estimates are. What is the desired output here? I can help implement this...

Note that metrics can be an arbitrary function on the set of request states.  But in most cases, the metric is computed per instance and then defined as the average over instances.  So I think what we'd need to do is to:
1. Define a new class `MetricResult`, which has `stats: List[Stat]` and `instance_stats: Dict[Instance, List[Stat]]`.
2. Then have all the `evaluate_*` functions in `Metric` return a `MetricResult`, which would be serialized as normal.
Would you have the bandwidth to implement this?

Let me take a stab at it today.

Started something here: https://github.com/stanford-crfm/benchmarking/pull/234.

I'm not sure we need `instance_stats` to be a `Dict` though...each `Stat` already bundles statistics for multiple instances, and has the individual values as well: https://github.com/stanford-crfm/benchmarking/blob/master/src/common/statistic.py#L29. Not sure what to do across trials either...currently, I made <instance, trial_id> the key in the `instance_stats` dict, but could do something different.
Repository: stanford-crfm/helm
Issue: Make output_path available for perturbations and metrics.
Root cause: Architectural Constraint

# Purpose
The information retrieval metrics require access to a `qrels` file downloaded as part of the caller scenario (discussed in #473). This value is currently hard coded since there doesn't seem to be a way to get this path in `run_specs.py`. We need to make this path available to the metrics that require it.

Another alternative is for the information retrieval metric to retrieve the exact same qrels file, but this introduces quite a few more dependencies.

@dilarasoylu I think I follow this, but what exactly do you think needs to be improved? 

It seems like the current version is fine, though perhaps there is something a bit more elegant. If anything, I wonder if the hard-coding of `benchmark_output` is an issue/too restrictive. 

It seems that the `output_path` is a command line parameter. @teetone has been using the `benchmark_output` as the default output path so far, but it is possible to supply a different output path. This path is propagated into scenarios (thought `self.output_path` parameter), but not to do the metrics and perturbations.

The goal is to propagate the command line specified `output_path` to the metrics and perturbations similar to how we do it for the scenarios.

Ah got it, that makes sense. Kk will look into 

I don't think we need this anymore based on how the new information retrieval scenario is defined?

Closing for now, @dilarasoylu let me know if this is still needed.

Thanks Rishi!

This is still needed for perturbations. It won't change any practical functionality.

We are currently hardcoding "benchmark_output".




Ok, I am going to re-open this but make p3. Maybe @dtsip can handle if its related to perturbations (iirc we simply need to snake this path through to get it to perturbations)? 

I added `get_benchmark_output_path()` in #2109, which returns the value set through the flag.

The remaining work would be find all instances where "benchmark_output" is hardcoded in metrics and perturbations, and replace them with calling this method.

Related: #1918

Working on this!

@AnshKhurana Are you still working on this?
Repository: stanford-crfm/helm
Issue: Improve README for public release 
GitHub labels: documentation

p2

General

- Do another pass over README
- Move out Stanford-specific documentation from README (separate private repo?)

In theory, we should add another doc/README for adding new models (we already have for scenarios, metrics, perturbations). 

And in that doc, we should remember to acknowledge contamination: https://github.com/stanford-crfm/benchmarking/pull/767#discussion_r951818899

README has already been broken up.  Still could use a final pass though.

Yup, all of the before release issue wrt documentation are things i will do after we launch final run in period of sept 15-25

I will do a pass as well, but I think the current version is fine for release. 

Transfering to @yifanmai since I will have other stuff to prioritize for next few days, and will come back to this later if needed, but Yifan will probably know better.
Repository: stanford-crfm/helm
Issue: quac:model=openai/text-curie-001 failed
Root cause: Validation Gap

Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 118, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 69, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 75, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 99, in run_one
    scenario_state = self.executor.execute(scenario_state)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/executor.py", line 84, in execute
    request_states = list(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/executor.py", line 77, in process
    result: RequestResult = self.remote_service.make_request(self.execution_spec.auth, state.request)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/remote_service.py", line 47, in make_request
    RemoteService._check_response(response)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/remote_service.py", line 29, in _check_response
    raise RemoteServiceError(response["error"])
proxy.remote_service.RemoteServiceError: Failed to make request to openai after retrying 5 times. Error: OpenAI error: This model's maximum context length is 2049 tokens, however you requested 2050 tokens (1950 in your prompt; 100 for the completion). Please reduce your prompt; or completion length.


It seems like an issue with the GPT-2 tokenizer. I created an issue: https://github.com/huggingface/transformers/issues/17682
Repository: stanford-crfm/helm
Issue: Out of memory when calculating Code metrics
Root cause: Resource Mishandling

`benchmark-present --max-eval-instances 1000 --conf-path src/benchmark/presentation/run_specs_code.conf --num-threads 5 &> openai-codex.log`

run_specs_code.conf:

```
"code:model=code,dataset=HumanEval": {status: "READY"}
"code:model=code,dataset=APPS": {status: "READY"}

"copyright:model=code,datatag=prompt_num_line_1-min_lines_20.json": {status: "READY"}
"copyright:model=code,datatag=prompt_num_line_5-min_lines_20.json": {status: "READY"}
"copyright:model=code,datatag=prompt_num_line_10-min_lines_20.json": {status: "READY"}
```

Killed


@lxuechen Can you take a look? It didn't work even on a machine with 64 GB of RAM. I have a copy of the logs at `/juice/scr/nlp/crfm/benchmarking/openai-codex.log`.


Ok. I can work on it tonight. Thanks for the heads up. 

I think I found the issue. One of the generated programs is doing weird stuff -- it's generating a very long string, and as a result the log has this large string which occupies ~1 gig. 

I think the issue is only with `"code:model=code,dataset=APPS": {status: "READY"}`, since this is the only command that runs generated code. So if this is blocking everything else, I think it's okay to omit the particular line for APPs at the moment. 

I am trying to think of ways to fix this, but I need some time. 

For context, the default evaluation code for HumanEval written by OpenAI doesn't have this problem, since they explicitly set memory usage limits for the Python process when running model generated code. 
https://github.com/openai/human-eval/blob/312c5e5532f0e0470bf47f77a6243e02a61da530/human_eval/execution.py#L171

I copied the evaluation code for APPs (written by Dan Hendrycks and others) for our benchmarking experiments, which doesn't have these guards. I can send in a patch but would there be a good way to test it aside from re-running on the full 1k-instance set? I'm still not too sure on which example things start to break.
Repository: stanford-crfm/helm
Issue: Resume downloading from partially downloaded file
Root cause: Unimplemented Feature Gap

See general.py: ensure_file_download.

```
wget:
-c,  --continue                  resume getting a partially-downloaded file
```

but,  wget -c doesn't work for some URLs

I'm not sure there's much you can do here.

Should we close then?

Yeah, let's close.
Repository: stanford-crfm/helm
Issue: Decide on a name for the bias/erasure metric
Root cause: Documentation Deficiency

As discussed [here](https://github.com/stanford-crfm/benchmarking/pull/414#discussion_r889834182) by @rishibommasani, we need to decide on a name for the metric that's currently called `BiasErasureMetric`:

> Note: Still not sure what we want to call this; I am guessing erasure should not be the canonical name for the metric as much as the qualitative characterization for when there is a large disparity. Some might call this bias, or more specifically a frequency bias (e.g. work that shows men are more often discussed in news articles or wikipedia entries than women). It could also be called representation, though that word is highly polysemous with very related but not exactly the same senses. So not quite sure, but we should have some reasonable name for it.
> 
> For the overall metric, might be fine to call BiasMetric but then the modes should be clearer (we could also call occurrence and co-occurrence in the code, which is more precise)
Repository: stanford-crfm/helm
Issue: bAbI task 19
Root cause: Configuration Error

Hi, I checked the performance for babi task 19 and it shows 0 for all the models:
https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*babi_qa%3Atask%3D19.*
The reason for that is that all answers to that task should be two words, but the max_token reported on the website is set to 1.
The strange thing is that on the specs I have set it to 2 for task 19: 
https://github.com/stanford-crfm/benchmarking/blob/11f344de17b9e2146e6d3f92d67d88f7ece17b12/src/benchmark/run_specs.py#L777

When I tested it locally it worked fine and gave non-zero performance:
https://github.com/stanford-crfm/benchmarking/pull/137

In the reasoning meeting people said it might have to do with overriding that is taking place. Do you know what could be the source for that? Thanks!
Repository: stanford-crfm/helm
Issue: Turn DEFAULT_CONFIGS in accounts.py into a configuration file
Root cause: Architectural Constraint

Move this to a configuration file (`default_quotas.conf` alongside `accounts.jsonl`):

```
DEFAULT_QUOTAS = {
    # model group -> {granularity -> quota}
    "gpt3": {"daily": 10000},
    "codex": {"daily": 10000},
    "jurassic": {"daily": 10000},
}
```

This seems unnecessary since no external users are running their own crfm-models playground server deployments.
Repository: stanford-crfm/helm
Issue: GPT-2 Tokenizer truncation: try out suggested logic
Root cause: Algorithmic Error

Suggested: https://github.com/huggingface/transformers/issues/17682

Old workaround: https://github.com/stanford-crfm/benchmarking/pull/515

Another method: https://stackoverflow.com/questions/58979779/berttokenizer-when-encoding-and-decoding-sequences-extra-spaces-appear
Repository: stanford-crfm/helm
Issue: Solidify all decisions for commonsense-related question answering
Root cause: Unimplemented Feature Gap

- [x] We should look over the results and verify we want to include all 5 commonsenseQA scenarios
- [x] We should finalize whatever prompting decisions, as the performance for all models except text-davinci-002 seem much worse (fro **HellaSwag**, compare https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*dataset%3Dhella.* vs.  https://paperswithcode.com/sota/sentence-completion-on-hellaswag). 
- [x] The naming of the configurations should be more explicit. Calling them all commonsenseqa when there is also a commonsenseQA dataset is confusing.

@michiyasunaga could you look at this sometime tomorrow (Wednesday) since we need to solidify before the run. 

- 1. dataset inclusion: I feel that we could perhaps use the initial two datasets, hellaswag and openbookqa? Looking at all the results, these two datasets seem sufficient to tell the overall trend of models
- 2. prompting method: yes our knowledge team has been observing this trend where using CausalLM scoring works better than MMLU-style MCQA for these commonsense QA datasets. Our discussion with Percy so far is that MMLU-style MCQA prompting is cleaner/more principled, and in theory can handle comparison among answer choices (e.g. handling “none of the above” choice) as well, so we have been leaning toward using it for all datasets. We could confirm with Percy one more time.
- 3. naming: yeah.. the original motivation discussed with Percy was to group these datasets together because they all have the multiple choice QA format and test commonsense. I guess we do still want to have some group name.. What about we just call this group “commonsense” instead of “commonsenseqa”? (then at least the name would not conflict exactly with the commonsenseqa dataset). 

i. Those two seem good to me.
ii. Yes, this is what we discussed, but I didn't realize that the CausalLM scoring method is way better for HellaSwag (like a 50 point difference!).  I think we should move the CLM code into the adapter (which we discussed before), and use that for any LM-like multiple-choice task, which would be useful for BLIMP as well.
iii. Let's rename to "commonsense" then.



For ii, tracking in #550 

Closing, with remainder handled in #550