zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Missing Dependency chardet
Root cause: Environment Incompatibility

It seems that `chardet` is missing from the dependency list in the main branch when I tried to execute `helm-run`.

```bash
Traceback (most recent call last):
  File "/data/chenhui_zhang/anaconda3/envs/crfm-helm/bin/helm-run", line 5, in <module>
    from helm.benchmark.run import main
  File "/data/chenhui_zhang/helm/src/helm/benchmark/run.py", line 13, in <module>
    from helm.proxy.services.remote_service import create_authentication, add_service_args
  File "/data/chenhui_zhang/helm/src/helm/proxy/services/remote_service.py", line 3, in <module>
    import requests
  File "/data/chenhui_zhang/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/requests/__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "/data/chenhui_zhang/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/requests/exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "/data/chenhui_zhang/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/requests/compat.py", line 13, in <module>
    import charset_normalizer as chardet
  File "/data/chenhui_zhang/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/charset_normalizer/__init__.py", line 24, in <module>
    from .api import from_bytes, from_fp, from_path, is_binary
  File "/data/chenhui_zhang/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/charset_normalizer/api.py", line 5, in <module>
    from .cd import (
  File "/data/chenhui_zhang/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/charset_normalizer/cd.py", line 14, in <module>
    from .md import is_suspiciously_successive_range
  File "charset_normalizer/md.py", line 10, in <module>
ImportError: cannot import name 'is_ascii' from 'charset_normalizer.utils' (/data/chenhui_zhang/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/charset_normalizer/utils.py)
```

Similar issue for `cchardet`

`requests` does not use `charset`- instead it uses `charset-normalizer` ([notes](https://requests.readthedocs.io/en/latest/community/updates/#id9)), which is [included](https://github.com/stanford-crfm/helm/blob/5f6be0bbb3c91624afb3b69044bfad39fed9b8ba/requirements.txt#L34) in crfm-helm's `requirements.txt`.

This is probably a issue with your pip environment and can be resolved by reinstalling `requests` and/or `charset-normalizer`.
Repository: stanford-crfm/helm
Issue: Frontend z-index bugs on the leaderboard page
Root cause: Algorithmic Error

The dropdown menu is placed below the "models" cell, whereas it should be above.

![Screenshot 2024-05-06 174636](https://github.com/stanford-crfm/helm/assets/185227/a29a94ed-5466-4234-b7ec-fb350416292c)

When scrolling right, the link icons are placed above the frozen first column, whereas it should be placed under

![Screenshot 2024-05-06 174740](https://github.com/stanford-crfm/helm/assets/185227/6bb6de56-0bdc-4d96-a91c-c16f9923ff7b)


taking a look


The z-index for the header row also needs to be increased. (or alternatively, the icon z-index needs to be the default value)

![image](https://github.com/stanford-crfm/helm/assets/185227/e4cddcc3-657c-4655-a8bc-7261640f2e13)




re-closing via #2651
Repository: stanford-crfm/helm
Issue: Fix instance level metrics being way too far to the right
Root cause: Algorithmic Error

@teetone flagged this 

"it's hard to read the instance-level numbers for some pages. We have to scroll all the way to the right to see them:"

need to see why the x-axis is so weird here

TBH I think we should just change the value column from right justify to left justify.

> TBH I think we should just change the value column from right justify to left justify.

This seems like a reasonable solution
Repository: stanford-crfm/helm
Issue: NarrativeQA: should we be generating only one instance per story?
Root cause: Architectural Constraint

NarrativeQA has a set of passages and several questions (with different answers for each). We are currently generating one instance per question which means that some passages are duplicated. Do we want to pick a single question per passage at random?

Resolved in https://github.com/stanford-crfm/benchmarking/pull/302.
Repository: stanford-crfm/helm
Issue: Duplication of fields across `*_schema.yaml` files
Root cause: Architectural Constraint

Many fields are duplicated across `*_schema.yaml` files. Here is how to deal with some of them.

- [x] `adapter`: This should be auto-inferred from the docstrings of `AdapterSpec`. Done in #2611.
- [ ] `metrics`: This could be moved to a `metric_metadata.yaml` file that is parallel to `model_metadata.yaml`. Or it could be inferred from a `metadata()` method on metrics objects. Note: It may not be possible to deduplicate this because there can be different `Metric` objects that use the same metric names - we don't enforce that the names are unique in a global namespace.
- [ ] `perturbations`: This should be moved to a `perturbation_metadata.yaml` file that is parallel to `model_metadata.yaml`. Doesn't seem to be a big problem because perturbations are rarely used.
- [ ] `metric_groups`: stay in `schema.yaml`.
- [ ] `run_groups`: There is an argument that "leaf" run groups i.e. scenarios should go in some `scenario_metadata.yaml` file. But it is possible that users will want to change which metrics groups are used for scenarios, so it should stay in `schema.yaml`.
Repository: stanford-crfm/helm
Issue: Minor bug in URL routing

When you visit the main [HELM](https://crfm.stanford.edu/helm/lite/latest/) page it looks like nothing has been updated since Dec 2023.
 
Thats because the path `/helm/lite/latest` maps to v1.0.0 (2023-12-19)
It should likely map to the “latest” v1.3.0 (2024-05-07)

Resolved (visit https://crfm.stanford.edu/helm/lite/latest/ and find what you mentioned).

If you still see this issue feel free to reopen.
Repository: stanford-crfm/helm
Issue: max_train_instances should accept integer arguments
Root cause: Documentation Deficiency

Currently we need `max_train_instances=zero` which is silly.

Actually, `max_train_instances=0` already works because of [this logic](https://github.com/stanford-crfm/helm/blob/v0.5.2/src/helm/benchmark/run_expander.py#L52-L55).
Repository: stanford-crfm/helm
Issue: list of the questions used for each of the MMLU subsets
Root cause: Documentation Deficiency

Hi, 

How did you choose the 5 questions used in the context window? I see they're different for each subset, and was wondering if there's a json file somewhere I can use. 

Regards,
Surya

Hi Surya, The 5 questions are sampled from the train split using the code [here](https://github.com/stanford-crfm/helm/blob/v0.5.1/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L122). We don't have a JSON file for the 5 examples, but you can extract them from the raw text from the MMLU leaderboards, or run the code yourself and export the examples.

Thanks!
Repository: stanford-crfm/helm
Issue: Change `gsutil rsync` to `gcloud storage rsync` in documentation
Root cause: Documentation Deficiency

`gsutil rsync` is deprecated and users should use `gcloud storage rsync` instead.

Valentin also reports:

> Quick comment on the downloading instructions: `gsutil -m rsync -r $GCS_BENCHMARK_OUTPUT_PATH $LOCAL_BENCHMARK_OUTPUT_PATH` doesn't work for Windows machines since the folder names contain colons (e.g., `commonsense:dataset=openbookqa,method=multiple_choice_joint,model=google_gemini-1.0-pro-001`), which is not allowed on Windows. A simple work-around is to use `gcloud storage rsync -r $GCS_BENCHMARK_OUTPUT_PATH $LOCAL_BENCHMARK_OUTPUT_PATH` instead, which automatically renames folders on Windows.
Repository: stanford-crfm/helm
Issue: Unable to download HELM leaderboard results (v1.3.0)
Root cause: Documentation Deficiency

**Bug**
Hi, I am unable to follow the instructions here: https://crfm-helm.readthedocs.io/en/v0.5.1/get_helm_rank/#download-helm-leaderboard-results to download HELM leaderboard results, for the latest version v1.3.0.

This works without issue for the version described in the documentation (v0.3.0), but does not work for any of the later versions.

**Error**
(venv) ➜  ~ export LEADERBOARD_VERSION=v1.3.0
(venv) ➜  ~ curl -O https://storage.googleapis.com/crfm-helm-public/benchmark_output/archives/$LEADERBOARD_VERSION/run_stats.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   225  100   225    0     0    299      0 --:--:-- --:--:-- --:--:--   300
(venv) ➜  ~ mkdir -p benchmark_output/runs/$LEADERBOARD_VERSION && unzip run_stats.zip -d benchmark_output/runs/$LEADERBOARD_VERSION
Archive:  run_stats.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of run_stats.zip or
        run_stats.zip.zip, and cannot find run_stats.zip.ZIP, period.

Any support or advice with this issue is much appreciated.

Thanks.


Hi, we no longer support zip files for recent versions of HELM.

Hi @yifanmai, thanks for your reply. Is there a supported file format for recent HELM versions? If so, is there a command I can use to download recent sets of results (v1.3.0 preferably)? Thanks in advance.

Hey we are also interested in getting the results for v1.3 and unable to download the results.

Hi @jasonwright38 and @mayankjobanputra, you can download the raw results by following [these instructions in the documentation](https://crfm-helm.readthedocs.io/en/latest/downloading_raw_results/). Let me know if you have any more questions.

Thanks for getting back to me @yifanmai, was able to follow those instructions and import v1.3.0 results. Much appreciated!

Me too! Thanks @yifanmai 

@yifanmai for me when I downloaded the `lite` version, a lot of models are missing which are still present on the website somehow.

Am I doing something wrong? I followed the steps below:

``` bash
export GCS_BENCHMARK_OUTPUT_PATH=gs://crfm-helm-public/lite/benchmark_output

gsutil -m rsync -r $GCS_BENCHMARK_OUTPUT_PATH/ $LOCAL_BENCHMARK_OUTPUT_PATH/
```

It downloaded about 14GB of data.

```
<>:/fast_ssd/HELM$ ls
releases  runs

<>:/fast_ssd/HELM$ cd runs/

<>:/fast_ssd/HELM/runs$ ls
v1.0.0  v1.1.0  v1.2.0  v1.3.0  v1.3.0-canary  v1.4.0  v1.4.0-canary

```

Hi @mayankjobanputra, which models are missing from your download?

Closing due to staleness; please reopen if you still have further questions.
Repository: stanford-crfm/helm
Issue: Standardize multiple choice
Root cause: Architectural Constraint

Should binary tasks (e.g. BoolQ, toxicity detection) be multiple choice with answers A, B or use raw answers (i.e. Yes/No, True/False)?

Decision:
- For tasks where the label set is closed (e.g. Y/N), predict by directly generating the label text (after all the label is semantically meaningful). This behavior should be defined in the multiple-choice adapter.
- For open label sets, predict A/B/C/D.


There are two parts to this issue:
1. Make sure the overall prompting strategy is correct (i.e., the prompts we generate follow this logic).
2. Refactor all the tasks to move this behavior into the adapter.

For the first, we should make sure we are consistent across scenarios before the next run. (We already are for scenarios I have visibility on and a quick skim tells me that most other things are also correct.)

For the second, I am not entirely sure I can do it before the big run, but it might not be a priority at the moment?


Is this done?

Not yet. We moved this to p2 since it does not affect the actual queries we are making.

Also, this wants to wait on https://github.com/stanford-crfm/benchmarking/issues/225.

Thinking about it a bit more, I am not sure there is much to do here.

If we interpret the adaptation method `multiple_choice` to be a/b/c/d we are fine. That is, there is no reason to mark scenarios like IMDB as multiple choice since generation is a more natural strategy.
Repository: stanford-crfm/helm
Issue: Support for Hugging Face Local Models
Root cause: Documentation Deficiency

One question regarding support for Hugging Face local model, I found this:

https://github.com/stanford-crfm/helm/blob/7ead1d6e41e91683ff6b06a94bd9e2ed7b629d0a/docs/huggingface_models.md?plain=1#L13

saying "Models at local file paths are not supported." 
Also, I have found this link: https://github.com/stanford-crfm/helm/pull/1505, which I am not able to find mentioned files. Could you please confirm the first link is the most updated and accurate one?
Many thanks!

The docs are outdated - we do support Hugging Face models at local file paths now.

Hi @NoushNabi, I have added documentation for using Hugging Face models at local paths at the [Adding New Models](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/) and [Hugging Face Model Hub Integration](https://crfm-helm.readthedocs.io/en/latest/huggingface_models/) pages.
Repository: stanford-crfm/helm
Issue: Potential Inclusion of the Phi-3 model family
Root cause: Unimplemented Feature Gap

Awesome work, would it be possible to also include the Phi-3 models from Microsoft? Thanks!

Thanks for the suggestion! I'll look into adding Phi-3 to the leaderboard.

We've just added Phi-3 to the [HELM Lite](https://crfm.stanford.edu/helm/lite/) and [MMLU](https://crfm.stanford.edu/helm/mmlu/) leaderboards.
Repository: stanford-crfm/helm
Issue: Using TogetherChatClient instead of TogetherClient for the Llama-3 Chat models
Root cause: Configuration Error

When I tried to run the `together/llama-3-8b-chat` and `together/llama-3-70b-chat` models with the default setup in `src/helm/config/model_deployments.yaml`, both models have `client_spec` set to `"helm.clients.together_client.TogetherClient"`. However, this leads to many strange behaviors at generation time, such as repetition of instructions or outputting "..." as the answer. 

This behavior stopped after I updated `src/helm/config/model_deployments.yaml` to reset the `client_spec` to `"helm.clients.together_client.TogetherChatClient"`. I wonder whether this is a better client to handle these chat models and perhaps should be the default?

Thank!

Thanks for the bug report - I can indeed reproduce this and am investigating.

Thank you very much!

Fixed in main branch by #2703 and #2701.
Repository: stanford-crfm/helm
Issue: Upgrade `transformers~=4.41`
Root cause: External Dependency Breakage

Bumping transformers version makes the GHA pipeline hang: https://github.com/stanford-crfm/helm/actions/runs/9293563927/job/25633789280

GitHub actions is using `transformers==4.42.4` now ([link](https://github.com/stanford-crfm/helm/blob/a248e43ce98125899ac5f2ae9fb27256c8559f8e/requirements.txt#L312C1-L313C1)) so I think this problem has probably been resolved
Repository: stanford-crfm/helm
Issue: Annotator issues
Root cause: Architectural Constraint

- [ ] `RunSpec.annotators` should be named `RunSpec.annotator_specs`
- [ ] Annotators should support concurrency. `AnnotatorFactory.get_annotator()` should not enforce singletons. The image2structure annotators should be using locks instead. This is a correctness issue - currently, if multiple annotators with different arguments are instantiated, the run could get an annotator with the wrong arguments.
- [ ] Annotator cache directory should be based on the `name` field, rather than the class name. For instance, the folder for `LatexCompilerAnnotator` should be `latex_compiler`, not `latexcompiler`.
- [ ] Annotator frontend display should support most "simple" JSON objects, but currently only supports lists of dicts (and single dicts after #2700)
Repository: stanford-crfm/helm
Issue: create `RemoteService` to allow for programmatic access
Root cause: Unimplemented Feature Gap

Create a `RemoteService` class that has the same signature as `Service` (for the functions used by`Server`).  This can be used **client-side**.  Then `demo.py` should be:

```
auth = Authentication(api_key="crfm")
service = RemoteService("http://crfm-models.stanford.edu")

# Make a request
request = Request(prompt="Life is like a box of")
print(service.make_request(auth, request))

# Modify account
account = service.get_account()
account.description = "Updated"
service.update_account(auth, account)
```

Behind the hood, it will make all the necessary REST calls.
Repository: stanford-crfm/helm
Issue: Is the paper button not responding upon clicking?
Root cause: Documentation Deficiency

Check https://crfm.stanford.edu/helm/air-bench/latest:
![image](https://github.com/stanford-crfm/helm/assets/8592144/a879fbaf-b202-41cb-beb5-7d557feb5e8e)
Does it mean this paper is still in progress?

Yes, "TBD" means the paper is still in progress.

Found the paper already on July 11th: https://arxiv.org/abs/2407.17436. It is indeed gets updated recently :) Thanks!
<img width="1401" alt="image" src="https://github.com/user-attachments/assets/4522381e-7908-4b66-a3c3-deaa6ca48d90">
Repository: stanford-crfm/helm
Issue: Difference between HEIM and VHELM?


HEIM is for text-to-image models and VHELM is for vision-language models.

> HEIM is for text-to-image models and VHELM is for vision-language models.

Thanks for your answer. The vision-language model can do both text-to-image and image-to-text tasks, right?


No - of the [current models on listed VHELM v1.0.0](https://crfm.stanford.edu/helm/vhelm/v1.0.0/#/models), only GPT-4o supports image generation. Additionally, GPT-4o's image generation is not generally available on OpenAI's API yet.
Repository: stanford-crfm/helm
Issue: How can I run mmlu with local model?
Root cause: Documentation Deficiency

I got a model finetune from Llama-3-8B, but I add the special_tokens in the new model "Llama-3-8B", the path which stores the model is `data/Llama-3-8B_ft`,  how can i run mmlu on this local model?

What info should I add in the config (model_deployments.yaml, model_metadata.yaml, tokenizer_configs.yaml...)? If you can provide a 
 tutorial about this that would be so nice~



If you host the model weights on Huggingface, you can use this drafted (work in progress) documentation to get an idea of what you can do: 

https://github.com/stanford-crfm/helm/pull/2463/files?short_path=b030132#diff-b030132068ab1309f5cbce7eeba7219d8a33664a250718a2d8286706d0c74232

For your use case you would use the existing tokenizers we have for Llama 3 8b

Here's a complete example. In your case, if you have added new tokens to the tokenizer, you need to configure a new tokenizer rather than using the existing Llama 3 tokenizer, as shown below.

`prod_env/model_deployments.yaml`:

```
model_deployments:
  - name: huggingface/your-model-name
    model_name: your-org-name/your-model-name
    tokenizer_name: your-org-name/your-model-name
    max_sequence_length: 8192
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        pretrained_model_name_or_path: /path/to/your/Llama-3-8B_ft
```

`prod_env/model_deployments.yaml`:
```
tokenizer_configs:
  - name: your-org-name/your-model-name
    tokenizer_spec:
      class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
      args:
        pretrained_model_name_or_path: /path/to/your/Llama-3-8B_ft
    prefix_token: "
"
    end_of_text_token: "
"
```

`prod_env/model_metadata.yaml`:
```
models:
  - name: your-org-name/your-model-name
    display_name: Your Model Name
    description: Your Model Description
    creator_organization_name: Your Org Name
    access: open
    release_date: 2024-01-01
    tags: [TEXT_MODEL_TAG, PARTIAL_FUNCTIONALITY_TEXT_MODEL_TAG]
```

We should probably do an end-to-end tutorial for Hugging Face at some point.

After doing so, you can run MMLU following [this doc](https://huggingface.co/microsoft/Phi-3-small-128k-instruct), setting `export MODELS_TO_RUN=your-org-name/your-model-name` as appropriate

Closing due to staleness; feel free to reopen if you have more questions.
Repository: stanford-crfm/helm
Issue: How can I get efficiency metric for my huggingface local model? (llama2)
Root cause: Documentation Deficiency

I have tested the standard usage process of HELM with the original llama-2. On the helm-server generated website, I noticed that there are no efficiency metrics recorded in the leaderboard presented by HELM.
![image](https://github.com/pan-x-c/EE-LLM/assets/67233215/3d39a00e-dd1f-47f9-93fb-81bec1c23567)

However, I found the `Observed inference runtime (s)` in the Predictions section for the corresponding dataset (cnn_dailymail as shown below).
![image](https://github.com/pan-x-c/EE-LLM/assets/67233215/995716a3-448f-4f35-a76e-588d6653e7f2)

Could you please clarify how I can obtain the efficiency metrics if the former is correct? Alternatively, why is there only one count in the Predictions section where I set the `--max-eval-instances=100` if the latter is correct?

In general, we no longer support computing denoised runtime. You can still used observed inference runtime though.

> why is there only one count in the Predictions section

The count is from the number of "train trials" i.e. the number of trials the evaluation is run with a different selection of in context learning examples using a different random seed. This can be set using `--num-train-trials` and defaults to 1.  The value you see is the mean for 100 instances in your single trial.

Thanks for your helpful relpy!
Repository: stanford-crfm/helm
Issue: Chart runs out of colors when there are too many organizations
Root cause: Architectural Constraint

At the bottom of https://crfm.stanford.edu/helm/mmlu/latest/#/models

![Screenshot 2024-06-21 084446](https://github.com/stanford-crfm/helm/assets/185227/844a2ceb-7703-4259-9574-45ae7e882e03)


wontfix: Daisy only allows using the baked-in color names in this pie chart, so we can't specify our own hexadecimal colors, which means we're limited to as many colors as Daisy allows.
Repository: stanford-crfm/helm
Issue: Issue with ICE
Root cause: Validation Gap

Encountered this error when running ICE:

![image](https://user-images.githubusercontent.com/47439426/162945072-59746d6b-e1fe-4b05-9b0b-8f080dcf51ff.png)


Also seems to happen for WikiText

![image](https://user-images.githubusercontent.com/47439426/162949415-10aa1731-2ddc-43a3-9e53-966e8acc10fb.png)


Will look into this!

Please verify the following works:

```
# "ice:subset=CAN": {status: "READY"}
# TODO: @Nathan - Add East Africa.
# "ice:subset=HK": {status: "READY"}
# "ice:subset=IND": {status: "READY"}
# "ice:subset=JA": {status: "READY"}
# "ice:subset=PHI": {status: "READY"}
# "ice:subset=SIN": {status: "READY"}
# "ice:subset=USA": {status: "READY"}

# "ice:split=spoken": {status: "READY"}
# "ice:split=written": {status: "READY"}

# TODO: @Nathan - Fix issue with gender on cluster; file extensions break.
# "ice:gender=F": {status: "READY"}
# "ice:gender=M": {status: "READY"}
```

You can test with a small value for `max_eval_instances`.

@YianZhang Where are we in terms of having this fixed? The goal would be to be finished by Monday for the next big run.

@teetone @nathankim7 I think the issue happens rarely (or at least that was Nathan's hypothesis), since it doesn't seem to happen in Nathan's testing, so it probably has to do with a particular token being escaped/some other rare nuisance with tokenization (since, e.g. the Pile works in the big run iirc). So I think testing with a small value of instances won't work unless we can independently isolate what causes the error. 

@rishibommasani I think your hypothesis is correct. I am currently focused on another high-priority PR (LM for AI21 models), but I will try to fix this by Monday. The simplest solution is just to remove that assertion. It will have negligible impact on the results since it is a rare case and it doesn't make the task easier/harder for the model.

Great, sounds good and makes sense wrt prioritization. Thanks!