zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Support tokenizers for other models
Root cause: Unimplemented Feature Gap

TODO

branch: `token_estimation`

What exactly needs to be done?  Can we not just use the GPT2 tokenizer for all models?  (This is just for truncation, right?)

> What exactly needs to be done? Can we not just use the GPT2 tokenizer for all models? (This is just for truncation, right?)

It's for prompt construction and metric calculation. To make a fair comparison, I think we should use those tokenizers. I think most, or all of the model's tokenizers are available in HuggingFace.
Repository: stanford-crfm/helm
Issue: OpenAI will support counting the number of tokens for us
Root cause: Unimplemented Feature Gap

From https://community.openai.com/t/how-do-i-calculate-the-pricing-for-generation-of-text/11662/5 , "we’re planning to implement a tokenizer into the client to make it easier. And adding an easy cost estimator into the api would be a great addition too."

We now use tiktoken for token counting in #1483.
Repository: stanford-crfm/helm
Issue: Decide on the main data imputation metric
Root cause: Unimplemented Feature Gap

Should it be exact match or quasi exact match?

Should be QEM as far as I can tell (e.g. see Instance 17 in https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=entity_data_imputation%3Adataset%3DBuy%2Cmodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical), @lorr1 to confirm. 

QEM should be fine. Casing doesn't matter. As long as there is no sub-word correctness.

I.e., New York != New York City

but

New York == new york

Sounds great, thanks for the reply @lorr1 @rishibommasani! I have updated our lists and closing this issue accordingly.
Repository: stanford-crfm/helm
Issue: Handle `match_case` for fairness perturbations
Root cause: Documentation Deficiency

https://github.com/stanford-crfm/benchmarking/pull/589#discussion_r916254751

I have double checked that the input text isn't passed to `match_case` function unless there is a substitution for all of the fairness perturbations. Closing the issue as no code change is needed.
Repository: stanford-crfm/helm
Issue: `metric.py` code cleanup
Root cause: Others

Follow up to this https://github.com/stanford-crfm/benchmarking/pull/598, clean up and refactor metric computation.

Related issue #597 

I feel like we are at a pretty decent state after https://github.com/stanford-crfm/benchmarking/pull/680, so imho we can close this.
Repository: stanford-crfm/helm
Issue: support any Hugging Face model
Root cause: Unimplemented Feature Gap

We want to be able to run queries against GPT-2 and GPT-J.

Two options:

1. Write a separate server that allows us to spin up Hugging Face servers locally.
2. Try to use Hugging Face servers directly (https://api-inference.huggingface.co/docs/python/html/quicktour.html#api-options-and-parameters)

Let us start with 2 since it's simpler (we might want to do 1 later).  Either way, need to write a subclass of `Client` to support the separate server with the HF model.

Note: the existing implementation `HuggingFaceClient` is broken and tries to load the model directly, which is not a good idea.

@percyliang I was going looking at the HuggingFace documentation, and it seems like the Inference API just returns the text completion and not the tokens and their probabilities.

From https://api-inference.huggingface.co/docs/python/html/detailed_parameters.html#text-generation-task:

```python
headers = {"Authorization": f"Bearer {API_TOKEN}"}
API_URL = "https://api-inference.huggingface.co/models/gpt2"

def query(payload):
    data = json.dumps(payload)
    response = requests.request("POST", API_URL, headers=headers, data=data)
    return json.loads(response.content.decode("utf-8"))

query({"inputs": "The answer to the universe is"})
```

Response:

```
    {
        "generated_text": 'The answer to the universe is that we are the creation of the entire universe," says Fitch.\n\nAs of the 1960s, six times as many Americans still make fewer than six bucks ($17) per year on their way to retirement.'
    }
```

Hmm, so we could maybe ask Hugging Face (maybe get Sidd to do it) if they have plans to provide a deeper API?

If that's not possible, then we can host the models ourselves too - I'd make a Hugging Face model server (maybe this already exists) that can hotswap various HF models...  The only downside with that is that it eats up a GPU and will probably have poor utilization.
Repository: stanford-crfm/helm
Issue: Microsoft will support log probs fully for MT-NLG models  (around 7/15/2022)
Root cause: Unimplemented Feature Gap

- `MicrosoftClient` should already handle log probs. Test out with `echo=True` and `echo=False`.
- Test `echo=True` and `max_tokens=0`
- Add validation in `do_it`, so we don't cache responses without log probs
- Give `microsoft/TNLGv2_530B` and `microsoft/TNLGv2_7B` `FULL_FUNCTIONALITY_TEXT_MODEL_TAG` tag
- Test with language modeling scenario
Repository: stanford-crfm/helm
Issue: Move `ai21_perplexity.py` and `ai21_random_window.py` and add README to `scripts/train_test_contamination` 
GitHub labels: p2


@Tiiiger Could you add a README to document what the scripts are doing?

Hi yes! Will do this and push other code within this week

> Hi yes! Will do this and push other code within this week

Thank you!

@Tiiiger Reminder about this, since we are getting close to release. If you can finish by 9/8, that would be fantastic, thanks!

@Tiiiger Reminder about this
Repository: stanford-crfm/helm
Issue: Inspect scenario parameters to ensure all the settings are covered in the run spec groups
Root cause: Validation Gap


@dilarasoylu What does this mean? Happy to do this, but not clear what needs to be done. 

This is mostly a placeholder to remember - what needs to be done is relatively quick: For each scenario that is captured in more than one scenario group (e.g. `Disinformation`), we want to ensure that the `scenario groups` we defined in `run_specs.py` cover all the possible settings for the scenario. For example, for `Disinformation`, there are currently two modes. We want to go in to `Disinformation` and ensure that this is actually true. If, for example, there is a 3rd mode, we should ensure that its group is correctly assigned.

This is fine - it is addressed by the run_specs.conf that has been checked multiple times and we won't be changing (beyond #681).
Repository: stanford-crfm/helm
Issue: Add landing page
GitHub labels: p1

Visualization


Don't think this needs to be a P1/there should be other higher priorities wrt visualization. 

Similar to ecosystem graphs/CRFM website, if you make a page, I can populate the content/etc. We can also look at various other benchmarks for examples (e.g. SuperGLUE, SQuAD), though I think we will need something different here since there is more conceptual work about the top-down philosophy around being holistic, systematic, and modular. 

Plan:
- [ ] @dilarasoylu - make an empty landing page with a back-pointer to it from each of the other pages (exactly as in ecosystem graphs).
- [ ] @rishibommasani - populate with content.
Repository: stanford-crfm/helm
Issue: Support `TogetherClient` (offline evaluation) and generating queries.jsonl file
Root cause: Unimplemented Feature Gap

Workflow (as of 7/14/2022):

1. Do a dry run for the external models: `benchmark_present --dry-run` (@percyliang I modified this part. I think it bloats `benchmark-present` and `benchmark-run` with extra logic if we make them generate `queries.jsonl`. I think a separate script is cleaner.)
2. Use the `together_export_request` script to generate `queries.jsonl` file using the dry run output: `python3 scripts/together/together_export_request.py benchmark_output/runs/dryrun`. Each line of `queries.jsonl` represents a single request.
3. Send  `queries.jsonl` to Jue/Ce for offline evaluation.
4. Jue/Ce sends back a new jsonl file (`results.jsonl`) with the responses where each line in the file represents the response to the request in `queries.jsonl`.
5. Use the `together_import_results` script to upload results (`python3 scripts/together/together_import_results.py results.jsonl prod_env/cache/together.sqlite`). The script uploads request/response pairs to the cache (`together.sqlite`).
6. Next time we run `benchmark-present`, `TogetherClient` should see a response for the request. The client returns an error if it doesn't see an entry in the cache for a request.
Repository: stanford-crfm/helm
Issue: Side-by-side comparison view page is broken when missing `metrics.json` file
Root cause: Validation Gap

Example: https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*narra.*

Could we exclude results for failed runs (all of them are GooseAI for this past run) that are missing `metrics.json`?

![Screen Shot 2022-07-15 at 3 35 37 PM](https://user-images.githubusercontent.com/16793796/179320760-818afc69-8449-4c11-9f5f-48fc4385e44c.png)


Moving to p3.  We should not have missing stats.json files.
Repository: stanford-crfm/helm
Issue: Reduce the size of `runs.json` to speed up loading
Root cause: Others


@percyliang we currently load `runs.json` once the `benchmarking` page is opened, so if we are happy with the current load times right now, we don't have to split this file.

At the very least, this should not be a priority for @dilarasoylu. I am going to unassign her, and if we think stuff is too slow we can address later. 

Remark: Stuff is much faster now, though not sure if this is the sole source.

I've changed the frontend code (though I don't think it has been deployed) to not load runs.json at all, so this is irelevant now.
Repository: stanford-crfm/helm
Issue: Add units when visualizing metrics
Root cause: Unimplemented Feature Gap

4 thoughts looking at the visualization that will appear on the website (see https://crfm-models.stanford.edu/static/benchmarking.html?scenarioGroup=NarrativeQA). 

- [x] Are we happy with these 3 metrics (i.e. inference {raw}, inference {idealized}, train) being the core metrics depicted for each scenario? I think this make senses (and tracks the paper), and other stuff can always be seen in the finer-grained run-level view, but just checking. 
- [x] Should we rename `idealized`? The name makes sense to me, but it might be misinterpreted/confusing for a random viewer of the website, especially since `idealized` is not necessarily less than `raw`. 
- [x] We should add `(raw)` or something to the `raw` variant. 
- [ ] We should add units somewhere (most critically because training and inference shouldn't be compared/are not even of the same type as units of time vs units of CO2). This may be a more general thing, though it only really applies to efficiency afaik, since everything else will be stuff like accuracy, F1, robustness, bias, where there are no canonical units in this sense

Let me know what you think @deepakn94 @santhnm2 

- I think those 3 metrics are fine...the `train` one is `CO2 cost` right now.
- I think `idealized` is ok as a name, especially if we can have some sort of legend that explains the metrics in more detail.
- I am fine adding `(raw)` or `actual` or something similar. Again, I think the legend would be helpful here.
- Legend could be a good place to add units. Even with accuracy, things like `percent` could be useful.

Sounds good. Action items:
1. Add (raw) or (actual) to the non-idealized inference time. Can you take this?
2. Add units and descriptions for all metrics. I will handle this in #384. 

Is this what you wanted? #647 

Sounds good, will address units in a different PR, thanks!

@dilarasoylu Do not prioritize this until other visualization stuff is done, but this should be a cosmetic fix. 

Sounds good, units are quick add ons once we merge the new visualization PR
Repository: stanford-crfm/helm
Issue: Support controlled randomness in requests
Root cause: Unimplemented Feature Gap

Right now, queries are deterministic because of caching. 

Add a `random: <string>` field in request, so that we can get the best of both worlds.  This string would have to be propagated down into the cache, which currently is on the raw request, which we can't change.

Also, we would ideally support this feature in a backward compatible way so we don't blow away the existing cache.  Not sure what the best way to do this is.

One option is to cache using both the raw request as well as `{request: ..., random: ...}`.  If random is specified, use the latter.  If not, use the former.  (It'd be nicer to have a more uniform way of dealing with this, but maybe it's okay for now.)

This logic would be repeated both in openai_client and ai21_client for now...

@percyliang What's an example of a value for `random` in `{request: ..., random: ...}`?

Is the setup something like this:

```
import random
random.seed(args.seed)

for request in requests:
   random_id = random.randint(some range?)
   response = cache({request: request, random: random_id})
```

The idea is that there would be a field `random: str` in `Request`, and that would used to key the cache.  There's no actual randomness in our code - it's just being able to call the API again with the same request.

> The idea is that there would be a field `random: str` in `Request`, and that would used to key the cache. There's no actual randomness in our code - it's just being able to call the API again with the same request.

So, the user will generate a random string when making a request? We will have to take in a seed value and generate the random strings on the benchmarking side still, right?

There's really no randomness.  I think people are just going to set `random=i for i in range(5)`.
Repository: stanford-crfm/helm
Issue: Quac + perturbations possibly not reproducible
Root cause: Algorithmic Error

Run: `"quac:model=text,data_augmentation=canonical": {status: "READY", priority: 1}`


Example:
![Screen Shot 2022-07-17 at 11 01 29 PM](https://user-images.githubusercontent.com/16793796/179456767-20526c56-9fb6-4959-a855-d6b6646b6551.png)



Reference order for `Quac` instances is not fixed, leading to the discrepancy. Specifically, following [line](https://github.com/stanford-crfm/benchmarking/blob/89c0e9113c18aef67612b544a8ed9ee3356c519a/src/benchmark/quac_scenario.py#L130) in `Quac` is causing the change in the reference order:
```
answers = list(set(answers))  # deduplicate
```
The order of the references affect the perturbation results because the perturbations set the random seed once for each instance (and re-use the same random object for all the references).

Since the issue only seem to happen with `Quac`, the quickest solution would be to modify the above line to keep the list order. If you think the random seed should be set again with each reference, we need to make a change to the base perturbation class as well as the `perturb` function of all the perturbations. This will cause a one time change to the other scenario prompts as well.
Repository: stanford-crfm/helm
Issue: Inspect the Codebase for TODOs
GitHub labels: p2

## Purpose

There are TODO statements left in various places of the code-base. This is a placeholder issue to remember to go through them all.

I have surveyed the codebase, clearing some TODOs. Of those that remain, they can be categorized as follows:

1. TODOs with associated issues mentioned (in the documentation near that TODO). These will be resolved as the corresponding issues are resolved.
2. TODOs related to `window_services` (e.g. `end_of_text_token`). Will be addressed by @YianZhang with #671.
3. TODOs related to visualization/front-end. This is being actively worked on by @dilarasoylu, and I think some of these are minor/can be overlooked. 
4. TODOs related to `schema.yaml`. I will address these as I address #384. 
5. TODOs related to information retrieval - @okhat @dilarasoylu we have discussed this, but can we resolve the TODO in `run_specs.conf` regarding running stuff on AI21. And can we address the path existing issue in `multiple_requests_metrics.py as we make changes?
6. TODOS related to NewsQA (handling data since cannot share) and Dialogue (reflecting that it was not integrated into codebase). These are fine. 

So, overall, no critical TODOs except those already tracked by active work and issues.
Repository: stanford-crfm/helm
Issue: Add ability to compute metrics from scenario files
Root cause: Unimplemented Feature Gap

## Purpose
As we are making changes to the metrics, it would be great if we added the functionality to read the scenario files and output stats, without running the benchmark, similar to the `Summarizer`.

I think deserialization of scenarios in general is a bit tricky...for debugging, I would use `--max-eval-instances 10` to make the iteration time shorter.
Repository: stanford-crfm/helm
Issue: Acommodate multiple scenarios in runSpec page information box 
Root cause: Unimplemented Feature Gap

# Purpose
Since we are allowing RegExes for the `runSpec` page, it is possible to open up multiple runs that belong to different scenarios at once. In these cases, we should either not render the scenario information box at the top of the page or include information from all the scenarios in question.

# Example
If we have the RegEx `runSpec=.*_qa.*model%3Dai21_j1-jumbo.*`, the header we get is as follows:
<img width="267" alt="Screen Shot 2022-07-20 at 8 16 42 PM" src="https://user-images.githubusercontent.com/21346670/180122727-f2935e9b-c0ac-46bb-b145-bc190ae1bce7.png">



Closing, since we don't want to open up multiple runs corresponding to different scenarios.  Clicking on groups seems to work.
Repository: stanford-crfm/helm
Issue: Add the ability to generate "clean" cache for public release
Root cause: Unimplemented Feature Gap

While doing the final run, we should build a "clean" cache for the final release

In particular, we should have a mode where in `Cache`, we use the cache path as usual, but we in addition write out all results into a separate file `cache_path + "new.sqlite"` or something.  Then we make sure if we only have the new cache, we don't have any cache misses.

This could be something that @yifanmai could take on.

Makes sense; essentially this would mean using a "cache" as a log of requests from a run. I can take a look.

From Percy's comment [here](https://github.com/stanford-crfm/benchmarking/pull/988#pullrequestreview-1133485332), we might try optimizing the follower cache to check that the key exists and only writing when it does not exist, rather than writing every time.

We should be able to figure out if the optimisation is net positive based on the cache hit rate in CacheStats.

Moving this to p3, since we can release the cache later.

Providing public dumps of the cache is not on the roadmap.
Repository: stanford-crfm/helm
Issue: Change path of the URL for the `SyntheticEfficiencyScenario`
Root cause: Configuration Error

Tokenizers have been renamed in https://github.com/stanford-crfm/benchmarking/pull/633/files#diff-d51d49420311f844f65c0edf5760f86d4f17a862c5601c223bc0dcce14be3940, from "huggingface/gpt2_tokenizer_fast" and "ai21"  to "huggingface/gpt2" and "ai21/j1".

We need to update the path of the files stored at https://raw.githubusercontent.com/stanford-crfm/benchmarking_efficiency/main/synthetic_efficiency_instances/

Conf file update: https://github.com/stanford-crfm/benchmarking/pull/633/files#diff-52664b27fded1037798f95738805b8d59cca9619ded13ad937d2dce9f390a554R513

Resolved by https://github.com/stanford-crfm/benchmarking_efficiency/commit/e8e2892517e137bcdb0638c0f63ceb4c11ac3319
Repository: stanford-crfm/helm
Issue: `NaturalQAScenario` is not reproducible
Root cause: Algorithmic Error

  Comparing `ScenarioState`s for natural_qa:mode=closedbook,model=openai_davinci,data_augmentation=canonical {
    ERROR: Not reproducible - content of benchmark_output/runs/dryrun_results1/natural_qa:mode=closedbook,model=openai_davinci,data_augmentation=canonical/scenario_state.json and benchmark_output/runs/dryrun_results2/natural_qa:mode=closedbook,model=openai_davinci,data_augmentation=canonical/scenario_state.json are different. Line 380:
    --- scenario_state.json (1):             "output": "4 September 1936",

    +++ scenario_state.json (2):             "output": "September 1936",



  Comparing `ScenarioState`s for natural_qa:mode=openbook-longans,model=openai_davinci,data_augmentation=canonical {
    ERROR: Not reproducible - content of benchmark_output/runs/dryrun_results1/natural_qa:mode=openbook-longans,model=openai_davinci,data_augmentation=canonical/scenario_state.json and benchmark_output/runs/dryrun_results2/natural_qa:mode=openbook-longans,model=openai_davinci,data_augmentation=canonical/scenario_state.json are different. Line 529:
    --- scenario_state.json (1):             "output": "4 September 1936",

    +++ scenario_state.json (2):             "output": "September 1936",
Repository: stanford-crfm/helm
Issue: A lot of the `07-21-2022` runs failed with OOM error (NLP cluster)
Root cause: Resource Mishandling

Example:

```
line 21: 24082 Killed                  benchmark-present --models-to-run microsoft/TNLGv2_530B --max-eval-instances 1000 --num-threads 1 --priority 2 --local &> microsoft-TNLGv2_530B.log
[1]+  Killed                  benchmark-present --models-to-run openai/text-curie-001 --suite 07-21-2022 --max-eval-instances 1000 --num-threads 1 --priority 2 --local &> openai-text-curie-001.log
[1]+  Killed                  benchmark-present --models-to-run openai/text-curie-001 --suite 07-21-2022 --max-eval-instances 1000 --num-threads 1 --priority 2 --local &> openai-text-curie-001.log
```

I suspect it's because we were running on many more models in parallel this week. 

Update README.

Running with a john seems to fix this issue. I updated the README: https://github.com/stanford-crfm/benchmarking/commit/8f6c26095fdf85f205520c5e3935700a3a9cae0d
Repository: stanford-crfm/helm
Issue: Support tokenizers for T5/T0/UL2 
Root cause: Unimplemented Feature Gap

Another thing is T0 has multiple version: T0 T0p T0pp.

T5
UL2
T0pp

Note: Since it came up here, we should resolve some nuances on T5 and T0.

1. For T5, there is the T5-11B from Raffel et al. (2019). But in the T0 paper, they compare to T5+LM-11B from Lester et al. (2021). I am not sure whether the latter is public (I assume it is, but didn't find it easily). So we should be clear what we are evaluating when we say T5. 
2. In terms of which models to run, I think the list of candidates would be `T5 (3B), T5 (11B), T0 (3B),  T0 (11B), T0+(11B), T0++(11B)`. Of these, I think if we have budget to do 2, we should do `T5 (11B)` and `T0++ (11B)`, i.e. the biggest models where ++ is the best of the BigScience models and T5 is the original approach that hasn't been prompt-specialized. If we have budget for 3, I would add `T0 (11B)`. Of course if we can do 11Bs, adding the 3Bs might be negligible further cost but its less clear we are making scale-centric comparisons here as compared to other settings where we have larger scale gaps (e.g. the GPT-3 series). Finally, T0+ seems like its awkwardly in the middle and not too critical, but could include if we have budget or want to really stress-test the nuances of prompt specialization in the sense of the T0 paper.
Repository: stanford-crfm/helm
Issue: Support WindowService and model for OPT models
Root cause: Unimplemented Feature Gap

Models:
   OPT (66B)
   OPT (175B)

From https://arxiv.org/pdf/2205.01068.pdf:
OpenPre-trained Transformers (OPT) is a decoder-only pre-trained transformer fully and responsibly shared with interested researchers.
Repository: stanford-crfm/helm
Issue: Set `prefix_token` and `end_of_text_token` for newer tokenizers for language modeling
Root cause: Unimplemented Feature Gap

https://github.com/stanford-crfm/benchmarking/pull/661#discussion_r930463123

Also, double check usage of `encode` in Adapter and `max_request_length` vs `max_sequence_length`.

We're only using `prefix_token` at the moment. If the models return the log prob for the first token, then an empty string is fine. If it doesn't, use the model's end-of-text token.