zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: update model names
Root cause: Configuration Error

https://crfm.stanford.edu/helm/v0.2.0/?models still says InstructGPT.  Not sure if this is because the latest changes haven't been deployed?

I am thinking this should be just a matter of re-deploying, since I have now changed both `models.py` (#1242) and `schema.yaml` (#1262), but maybe @teetone can confirm?

This needs re-deploying because schema.yaml lives in the website repo rather than in GCS. I'll do the re-deploy.

The model names have been updated. There's still one more mention of InstructGPT that should be changed; the ChatGPT description says "Sibling model to InstructGPT..." 

> The model names have been updated. There's still one more mention of InstructGPT that should be changed; the ChatGPT description says "Sibling model to InstructGPT..."

The description is from OpenAI's [blog](https://openai.com/blog/chatgpt):

"ChatGPT is a sibling model to [InstructGPT](https://openai.com/blog/instruction-following/), which is trained to follow an instruction in a prompt and provide a detailed response."

That sounds reasonable. Let's keep that as is then.
Repository: stanford-crfm/helm
Issue: Many raw run pages are very slow to load and crash the browser
Root cause: Algorithmic Error

https://crfm.stanford.edu/helm/v0.2.0/?group=msmarco_regular&subgroup=track%3A%20regular%2C%20valid_topk%3A%2030&runSpecs=%5B%22msmarco%3Atrack%3Dregular%2Cvalid_topk%3D30%2Cmodel%3Dopenai_text-davinci-002%2Cdata_augmentation%3Dcanonical%22%5D

Should check all the other prediction pages too.

If we have a display JSON that shows the predictions without the requests and we chunk up the requests, is this still a problem?

> If we have a display JSON that shows the predictions without the requests and we chunk up the requests, is this still a problem?

This is not enough; we already removed the requests from the prediction JSON. The problem is that there are too many HTML elements. We need to reduce the number of HTML elements.

Removing MS Marco from issue title because this affects a lot more runs than just MS Marco

See #1924 for some additional discussion of this issue. cc @brianwgoldman @farzaank 

The new React frontend no longer crashes on the raw predictions page! 🎉
Repository: stanford-crfm/helm
Issue: move v1.0 to v0.1.0
Root cause: Configuration Error

- Move v1.0 to v0.1.0 (need to update the URL and various contents of JSON files that refer to v1.0).
- Need to update the arxiv paper to point to v0.1.0.
- Redirect https://crfm.stanford.edu/helm/v1.0 to https://crfm.stanford.edu/helm/v0.1.0 (which is weird but hopefully people will stop using v1.0 for now).

@yifanmai, do you mind taking this one since you have access to the website's repo?

Sure.

Done. The v1.0 to v0.1.0 rewrites the URL e.g. https://crfm.stanford.edu/helm/v1.0/?runs=1&runSpec=boolq%3Amodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical becomes https://crfm.stanford.edu/helm/v0.1.0/?runs=1&runSpec=boolq%3Amodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical
Repository: stanford-crfm/helm
Issue: check if scenarios end up in training data of models
Root cause: Unimplemented Feature Gap

For example, OpenAI API requests are used to improve the their APIs:
https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance

Minimally, we should make sure we have a process for updating the train-test contamination information (by explicitly checking with the model providers).

In addition, we might want to designate some datasets as off limits (but this will in general be difficult to do) given that we want to expand the scope of HELM.

Can we fold this into a periodic (idk what the cadence should be; e.g. quarterly might make sense given other much more formal reporting requirements {e.g. taxes} are quarterly but might be too slow in some ways for models) update report, where we try to get each model provider to do updates for this and for ecosystem graphs? Seems like that could be a good compromise of sorts.

We opted out the CRFM accounts out of OpenAI data collection. However, OpenAI will probably still get our data because other HELM users can run HELM on their own OpenAI organizations.
Repository: stanford-crfm/helm
Issue: Make prediction loading faster
Root cause: Algorithmic Error

Currently, the browser rendering is still blocked for 2s-3s on render.

So jQuery is creating thousands of objects and calling `append` thousands of times. This requires multiple GCs during the rendering. We should switch to a real templating library...

A temporary fix is that instead of rendering the instances and then the predictions (two layout phases), we can just do them at the same time (one layout phase). This saves about 30% time.

Can we close?

It's fast _enough_, but it could be made even faster. :)

I created #1304 to track moving appropriate things to Handlebars, so we can close this.
Repository: stanford-crfm/helm
Issue: AI21 Jurassic only allows 1 to 16 number of completions
Root cause: Architectural Constraint


What should we do about this?  Is this a problem?

Closing this since I don't believe this is an issue. To the best of my knowledge, this at best only comes up for toxicity stuff, where RealToxicityPrompts did 25 completions in their paper and we do 16 to standardize. Seems fine/25 was arbitrary anyways in theirs and we have recorded this change already.
Repository: stanford-crfm/helm
Issue: helm-summarize can leak local paths in run.json
Root cause: Validation Gap

`run_path` in `runs.json` is prefixed by the flag value `--output-path`, if the user sets `--output-path` to an relative or absolute file path on their local machine, it will appear in `run_path`.

Doesn't really have any security implications, but this probably shouldn't happen anyway.
Repository: stanford-crfm/helm
Issue: Friendly interface for selecting models in the playground
Root cause: Unimplemented Feature Gap

For both settings and environment, users enter YAML (e.g., model: openai/davinci).  Have an autocomplete system that can show the possible arguments (e.g., set of models).  This maintains the flexibility of something more programmatic with something more user-friendly than having to remember all the model names or look them up.

[Monaco](https://microsoft.github.io/monaco-editor/) has a [third-party YAML plugin](https://github.com/remcohaszing/monaco-yaml). Need to investigate if there's a first-party way to do this. Also investigate CodeMirror, TinyMCE, Ace et al.

Another easier option is to just have a dropdown box (on the side?) like any other playground and that just updates the settings box when you select something.  This might be better since typing `model: ...` isn't super discoverable).

Playground is in maintenance only mode; closing.
Repository: stanford-crfm/helm
Issue: add RealToxicityPrompts scenarios/metrics
Root cause: Unimplemented Feature Gap

Paper: https://arxiv.org/pdf/2009.11462.pdf

![Screen Shot 2022-01-14 at 2 32 58 PM](https://user-images.githubusercontent.com/16793796/149593881-07d82235-a71b-45ce-afde-3e5725885846.png)
Repository: stanford-crfm/helm
Issue: Questions on hyperparameter settings
GitHub labels: user question

p2

Hello, we are working on a study related to the hyperparameters in GPT-3. We observe that the recommended hyperparameters in HELM are different across tasks (e.g., temperature): https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_specs.py

Would you mind sharing how you tuned the hyperparameters in each task? Thanks in advance for your time. 


@teetone @rishibommasani Could you comment on the hyperparameter selection?

Just checking to see if you have updates on this thread, thanks

Hi!

1. The hyperparameters for many scenarios are largely similar (with the logic described in the appendix of the paper).
2. In the code for `run_specs.py`, we have annotated any scenario-specific decisions on hyperparameters with a justification. Let me know if we missed something/there is something you want justified. 

On the topic of temperature, when the scenario entails decoding a short sequence where there is a correct answer, we decode with temperature 0 to sample the (token-wise argmax), which is the sequence-level argmax for 1 token and probably fairly close for other small sequences. We did not explore other temperatures, but this is fairly standard (noting that things like nucleus sampling or so on are not able to implemented for all models given the APIs exposed, so we only can change the temperature). For longer-sequence generation (e.g. summarization), the temperature is discussed both in the paper + in the comments. Again, if there is some specific temperature value you want explained/justified, happy to provide it.

@rishibommasani  Thank you so much for your response. Your answer has solved my problem. Please close the issue.
Repository: stanford-crfm/helm
Issue: Demo commands in Tutorial is not runnable
Root cause: Documentation Deficiency

Hello, when I try to run the demo commands in [https://crfm-helm.readthedocs.io/en/latest/tutorial/](url)

`helm-run --run-specs run_spec.conf --local --suite v1 --max-eval-instances 10`

I got an error message:

> helm-run: error: unrecognized arguments: --run-specs run_spec.conf

I also tried to replace "--run-specs" with "--conf-paths", but it shows another error:

> ValueError: No model with the name: huggingface_gpt-2

There must be some mismatch between the tutorial and the code?

Thanks for catching this; I just opened PR #1316 to fix this.
Repository: stanford-crfm/helm
Issue: Run helm-summarize based on downloaded JSON output files
Root cause: Documentation Deficiency

Hello, I am wondering if it is supported to download model JSON output files from the website "raw run" first [https://crfm.stanford.edu/helm/latest/?runs=1](url) and run `helm-summarize --suite v1` locally (without running `helm-run`)? Thanks!

Hi, this isn't currently doable in v0.2.0 due to a JSON schema issue: the [v0.2.0](https://crfm.stanford.edu/helm/v0.2.0/) version of the JSON files is incompatible with the schemas in the v0.2.0 PyPI package release due to the breaking schema change in #1280. However, there will be a v0.2.1 release in a week or two that will fix this issue. I'll add documentation on how to download the files and run summarize after that release.

Hello, are there any news on when v0.2.1 is going to be released? Thank you!

Hi @wenlinyao, we've recently added a tutorial for downloading a zip file of existing results and running `helm-summarize` [here](https://crfm-helm.readthedocs.io/en/latest/get_helm_rank/). Let me know if you have more questions.
Repository: stanford-crfm/helm
Issue: Macro-F1 and MCC needed for overall evaluation
Root cause: Unimplemented Feature Gap

Hi guys,

For our text classification datasets, we often evaluate with macro-f1 or mcc overall. So far, only accuracy is supported as far as I see this. 


<img width="677" alt="Screenshot 2023-01-17 at 16 49 16" src="https://user-images.githubusercontent.com/3775944/213053719-8a41b93b-0492-4c8f-96da-caefcfc99cfc.png">


Opened #1368 for F1.

I'm not sure how MCC should work: it's possible for the model to output invalid outputs that are not any specific label; to compute the confusion matrix, we'd have to treat "invalid output" as its own class, and that seems confusing.
Repository: stanford-crfm/helm
Issue: Get biomedical scenarios working
Root cause: Unimplemented Feature Gap

Support things like MedQA.

One suggestion would be to wrap the BigBIO datasets ([GitHub](https://github.com/bigscience-workshop/biomedical), [HuggingFace](https://huggingface.co/bigbio)). It doesn't have MedQA though.
Repository: stanford-crfm/helm
Issue: benchmark Cohere command model
Root cause: Algorithmic Error


The Command model is currently generating a lot of empty responses for a weird reason: it _really_ wants to start a question answer with a `\n`, and since `\n` is a stop sequence character, it immediately terminates the generation. I had to set stop sequence to empty to get any non-empty generations. This happens _even_ if you strip the `Answer:` prefix and remove all `\n` from the prompt. E.g.

Prompt: `"Elmendorf Air Force Base (IATA: EDF, ICAO: PAED, FAA LID: EDF) is a United States military facility in Anchorage, the largest city in Alaska. Originally known as Elmendorf Field, it became Elmendorf Air Force Base after World War II, and in 2010 it merged with nearby Fort Richardson to form Joint Base Elmendorf-Richardson. Is there an air force base in anchorage alaska?"`

Response: `"\n\nYes, there is an air force base in Anchorage, Alaska. It is known as Elmendorf Air Force Base and is located in the city's northern suburbs."`

Prompt: `"Passage: Elmendorf Air Force Base (IATA: EDF, ICAO: PAED, FAA LID: EDF) is a United States military facility in Anchorage, the largest city in Alaska. Originally known as Elmendorf Field, it became Elmendorf Air Force Base after World War II, and in 2010 it merged with nearby Fort Richardson to form Joint Base Elmendorf-Richardson.\nQuestion: Is there an air force base in anchorage alaska?\nAnswer:"`

Response: `"\nYes, there is."`

Prompt: `"Is there an air force base in anchorage alaska?"`

Response: `"\nYes, there is an air force base in Anchorage, Alaska. It is known as Joint Base Elmendorf-Richardson and is home to the 4th Air Force and the Pacific Air Forces."`

I know we've been reluctant to have model-specific `AdapterSpec`s, but we kind of have to do that in this case to set stop sequence to empty.
Repository: stanford-crfm/helm
Issue: Evaluation instances include both valid and test splits
Root cause: Architectural Constraint

Hi,

I noticed that for few-shot learning, the `eval_instances` include both validation and test splits.

https://github.com/stanford-crfm/helm/blob/4384c61607a991d3e972ab0c822b779c71fccf9d/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L38-L39

https://github.com/stanford-crfm/helm/blob/4384c61607a991d3e972ab0c822b779c71fccf9d/src/helm/benchmark/scenarios/scenario.py#L15

I assume the reason for this decision is to help datasets with unlabeled test sets? However, for datasets with both validation and test sets, numbers from helm would not be comparable to numbers in the literature as it's standard practice to compare numbers of the test set. I might be missing something obvious. Thanks!

@rishibommasani could you comment on this experiment design decision?

@rishibommasani ping?

Hi @shijie-wu! 


Sorry for the delay in answering. Indeed, as you noted, part of the reason is for (i) datasets where test in unlabelled and valid is labelled but also (ii) when only one of these two partitions exist in the original dataset {which happens quite a bit}. Aside: if the dataset does not define a valid/test partition or we created it, the details (e.g. splitting ratio) are in the paper (and of course in the code as well). 

We compute metrics for each partition. For example, bAbI includes both a valid and test split. In this case, as you can see here for `text-davinci-003`, we report the results for both partitions separately: https://crfm.stanford.edu/helm/latest/?group=babi_qa&subgroup=task%3A%2015&runSpecs=%5B%22babi_qa%3Atask%3D15%2Cmodel%3Dopenai_text-davinci-003%22%5D#metrics

For the purpose of reporting numbers in papers, if one wants to be precise, I would point to the underlying run files as the purest results. The ones we render in the top-level tables involve several aggregation decisions (e.g. aggregating across valid + test, aggregating across 3 random seeds, macro-average across different subsplits like domains within MMLU) that we thought were the best defaults for general consumption of the metrics (which goes beyond comparisons in papers, where one might more purely care about comparability with prior work). 

As an aside, I would question whether the practice in the field is to use test set numbers. I agree that perhaps is the ideal and/or something we may hope to be the norm, but in what I have seen, I think the actual practice tends to be much more uneven and valid, test, and various other things all get used (often with poor disclosure).
Repository: stanford-crfm/helm
Issue: Compute Average metric for MMLU
Root cause: Documentation Deficiency

It seems that whenever run experiments on MMLU, we need to choose one senario "mmlu:subject=XXX". Are there any convenient ways of run evaluation on all the 57 subjects of mmlu and compute average metric, similar to the implementation from https://github.com/hendrycks/test/blob/master/evaluate.py .

This isn't currently supported but should be fairly easy to add. We would need to add an special "all" subject that uses all the subjects.

Looking at the implementation, it uses [macro-averaged accuracies across subjects](https://github.com/hendrycks/test/blob/4450500f923c49f1fb1dd3d99108a0bd9717b660/evaluate.py#L133), not micro-averaged. So actually it is already possible to replicate this today:

- Create a `mmlu_run_spec.conf` with entries for [all subjects in mmlu](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_specs.conf#L32-L88)
- Do `helm-run` using this `mmlu_run_specs.conf`, setting `--max-eval-instances 1000000` to use all of `mmlu` instead of subsampling
- Run `helm-summarize`

The summarized accuracy should be the same as the one computed above.

If we want micro-averaged, then we do need the special "all" subject.



Closing this for now; please reopen if you have additional questions about mmlu or if you still need the special "all" subject.
Repository: stanford-crfm/helm
Issue: Support human evaluation 
Root cause: Unimplemented Feature Gap

More generally, support offline evaluation. After sending requests to the LLM and getting responses, generate requests for some offline step (e.g. submitting to human evaluators and retrieving results). Later on, retrieve the responses and compute the metrics.

This requires designing an API in Metrics to support this, and a way to run the steps for sending requests and retrieving responses to the offline step.

We also support Scale AI in #1609.

MTurk, Surge AI and Scale AI should be enough integrations for now.
Repository: stanford-crfm/helm
Issue: Evaluate BioMedLM
Root cause: Unimplemented Feature Gap

Once @yifanmai's PR https://github.com/stanford-crfm/helm/pull/1323 is in, we can support and evaluate https://huggingface.co/stanford-crfm/BioMedLM with the biomedical tasks here: https://github.com/stanford-crfm/helm/pull/1332.

Closing because there currently isn't a need to evaluate BioMedLM.
Repository: stanford-crfm/helm
Issue: Published MMLU results only cover 5/57 subjects?
Root cause: Documentation Deficiency

[The MMLU results](https://crfm.stanford.edu/helm/v0.2.0/?group=mmlu) page appears to only cover 5/57 subjects.

This does not match what I understand to be the [core benchmarking run_specs.conf](https://github.com/stanford-crfm/helm/blob/3c43cde06edbe061c94ee07857f63020fb20c77e/src/helm/benchmark/presentation/run_specs.conf#L33-L89).  Nor does it match the 10 subjects mentioned in [the docstring for that section of the run_specs.conf](https://github.com/stanford-crfm/helm/blob/3c43cde06edbe061c94ee07857f63020fb20c77e/src/helm/benchmark/presentation/run_specs.conf#L32).

Is this expected?

Thanks!


@teetone would you know how the MMLU subjects subset was chosen?

Hi @tgunter, the comment in run_specs.conf is misleading. We will remove it. For the paper's main results, we ran the RunSpecs with a priority of 1 or 2. Ideally, we would also run priority 3 and 4 RunSpecs, but we're constrained on API credits and compute.

> Hi @tgunter, the comment in run_specs.conf is misleading. We will remove it. For the paper's main results, we ran the RunSpecs with a priority of 1 or 2. Ideally, we would also run priority 3 and 4 RunSpecs, but we're constrained on API credits and compute.

@teetone @yifanmai you may want to clarify that on the [MMLU results page](https://crfm.stanford.edu/helm/v0.2.0/?group=mmlu), I think most readers would assume that all 57 tasks were included (especially as the paper indicates that they are).  I'm also curious as to how the subset of 5 tasks was chosen?
Repository: stanford-crfm/helm
Issue: Installation on MacOS still doesn't work
Root cause: Configuration Error

It seems we were trying to fix this through https://github.com/stanford-crfm/helm/pull/1224 but it still didn't work when I ran `./pre-commit-venv.sh` on my macbook. See error message below.

`ERROR: Could not find a version that satisfies the requirement torch==1.12.1+cu113 (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1)
ERROR: No matching distribution found for torch==1.12.1+cu113`

I commented out "torch==1.12.1+cu113; torchvision==0.13.1+cu113" in `requirement-freeze.txt` and then it worked. Maybe we need a better solution than this.

@yifanmai might know how to fix this?

I don't think there's a clean solution for this. I don't think we should maintain separate requirement files for different OS-es. However, I do think it's a good idea to switch the torch / torchvision in the requirements to the non-CUDA versions. Developers who want CUDA should set that up themselves manually (because we can't know what CUDA version the developers have on their machine anyway).

Apparently pip allows [platform specific entries in requirements.txt](https://stackoverflow.com/questions/29222269/is-there-a-way-to-have-a-conditional-requirements-txt-file-for-my-python-applica) so we can use that.
Repository: stanford-crfm/helm
Issue: The output of convert_ids_to_tokens cannot be assembled into the output of batch_decode
Root cause: Algorithmic Error

Hi，when I run helm on other data sets, I find that the output of `convert_ids_to_tokens`(`all_tokens`)cannot be assembled into the output of `batch_decode`(`all_decoded_text `). Is this normal?
And what does this TD above mean and is it related to this？
https://github.com/stanford-crfm/helm/blob/ad0ac3ee6ca8e66cb318f2936a1191a964687384/src/helm/proxy/clients/huggingface_client.py#L90-L92
These are the values of some relevant variables：
`sequences：[tensor([198], device='cuda:0')]`
`all_decoded_text：['\n']`
`all_tokens：[['Ċ']]`
This is a multiple-choice scenario, maybe the model wants the output to be C?
Thank you！

Sorry, I may have misunderstood, the fact that the token looks like a C may just be a coincidence.
Repository: stanford-crfm/helm
Issue: Inconsistent HELM dependencies?
Root cause: External Dependency Breakage

I tries installing `crfm-helm` and received the following warning:

```
crfm-helm 0.2.0 requires protobuf~=3.20.2, but you have protobuf 3.19.6 which is incompatible.
```

No problem! I'll just upgrade `protobuf`

```
tensorflow 2.11.0 requires protobuf<3.20,>=3.9.2, but you have protobuf 4.21.12 which is incompatible.
tensorflow-metadata 1.12.0 requires protobuf<4,>=3.13, but you have protobuf 4.21.12 which is incompatible.
tensorboard 2.11.2 requires protobuf<4,>=3.9.2, but you have protobuf 4.21.12 which is incompatible.
crfm-helm 0.2.0 requires protobuf~=3.20.2, but you have protobuf 4.21.12 which is incompatible.
```

So it seems `crfm-helm` requires `protobuf` to be >=3.20, but also requires tensorflow, which itself requires `protobuf` to be <3.20

Do I understand this correctly?

`crfm-helm` does not have a [direct](https://github.com/stanford-crfm/helm/blob/v0.2.0/requirements.txt) or [transitive](https://github.com/stanford-crfm/helm/blob/v0.2.0/requirements-freeze.txt) dependency on TensorFlow, so your TensorFlow dependency is probably coming from another package.

That said, we should probably upgrade the version of `protobuf`; I filed #1339 for this

Hi Rylan, does this answer your question, and can this issue be closed?

Yes - thank you!