zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: blanc requires torch<2.0
Root cause: External Dependency Breakage

We want to use PyTorch 2 because it is required for various VLM things. Unfortunately, we have a transitive dependency on `blanc` via `summ-eval`, and `blanc` requires `torch<2.0`, which causes a version conflict. For now, the workaround is to exclude `crfm-helm[summarize]` from `crfm-helm[all]`.

We can undo this workaround if / when `blanc` supports PyTorch 2.

I am not one to bump on issues, but modern-day (mid-2025) `vllm` requires `torch` v2. I want to use HELM, but am blocked on the `torch` v1 range requirement.

https://github.com/PrimerAI/blanc has not had a commit in two years, perhaps `blanc` should be dropped as a required version in HELM to pull in `torch` v2.

It looks like `blanc` was removed from the dependency list at some point. HELM currently [allows any version of PyTorch 2.x](https://github.com/stanford-crfm/helm/blob/add5163ff6f5c091d4344aeac11a4c7f34644468/setup.cfg#L61). Are you still seeing a dependency conflict for PyTorch versions?

Oh wow, I see. I think I was just reading in `setup.cfg` last night and hallucinated a `2.0` limit after seeing this issue.

Nevermind, ignore my false positive comment :)

Great, thanks for the update!
Repository: stanford-crfm/helm
Issue: Cleva translation metric not registered for `helm-summarize`
Root cause: Validation Gap

Despite #2283, if we run `{description: "cleva:model=text,task=translation,subtask=zh2en,prompt_id=0,version=v1", priority: 1}`, we cannot find any metric executing  `helm-summarize`.
 
![image](https://github.com/stanford-crfm/helm/assets/139481960/fe1a9fc0-183d-49d2-be75-3c4a1fdad60a)


Looks like the CLEVA translation scenario is broken in multiple ways:

- The metric name in `cleva_translation_metrics` is `bleu` but should be `cleva_machine_translation_bleu` to match [this](https://github.com/stanford-crfm/helm/blob/5c8c19460eb2ea3a4da4cd51a67a304ca002f138/src/helm/benchmark/metrics/machine_translation_metrics.py#L89)
- CLEVA should override `Metric` and implement `evaluate_generation()` so that we can get per-instance stats.

Unfortunately the table still didn't work after addressing these changes. I think this scenario should be considered broken. If anyone needs to use this / wants to fix this, feel free to comment on this issue.

Closing due to staleness. Please feel free to reopen if you would like to use this scenario.
Repository: stanford-crfm/helm
Issue: perspective_api_client's use of spanAnnotations
Root cause: Algorithmic Error

In the perspective_api_client, the score returned comes from [the first element of the spanScores array](https://github.com/stanford-crfm/helm/blob/fbffc14c0abd329f778f2166819096faf62ec68c/src/helm/proxy/clients/perspective_api_client.py#L53). Based on my read of the API documentation, this is an unexpected choice given that:

* `spanScores` are "currently done at per-sentence level". This means if the model outputs twenty sentences, only the first one gets scored.
  *  As an example, if I send in "Hello world. You suck!", the first span TOXICITY is 0.017 but the second span is 0.902.
* The docs state that "Some attributes may not return spanScores at all"
* There is a field `summaryScore` which is the "summary score for the entire comment".

This appears to have been the case since its introduction in PR #81. @teetone do you recall why it is implemented this way?
Repository: stanford-crfm/helm
Issue: Percy’s feedback on new HELM frontend
GitHub labels: enhancement

p2

frontend

### Results tables pages

- [x] When mousing over a model name, should show more information about the model, either as a tooltip (preferred) or via a link to the models page (need a href anchor).
- [x] We still write ‘Model/adapter’ - should just be ‘Model’ for simplicity
- [x] When mouseover heading (e.g., F1), should tooltip show the description based on the schema
- [x] We still show ⚠; would just drop all the train-test contamination stuff from the frontend since we’re not consistently updating this

### Predictions pages
- [x] Make it clearer that we have a scenario and a model.  Perhaps the two should be the same size?  Perhaps they should be horizontally arranged side-by-side.
- [ ] Right after the adapter specification, we should have a panel to display the key metrics (the same ones that are displayed on a results table - need to pull that out of the schema).  This is the most important.
- [x] When mouseover any key in adapter or metrics (e.g., num_train_trials), show a tooltip with the description of that key.
- [x] There should be at most two instances in a column.  Right now, the input is often too long and requires scrolling in a very small window.
- [x] In general, strong dislike for scrollbars, and I think the input box should be a lot bigger (scroll when we hit nearly a screen full of content).
- [x] When click ‘Requests’, could show all the information in a popup to save a click because right now you need to click expand to actually see the prompts.
- [x] Rename ‘Requests’ to ‘Prompt’
- [x] ‘Details’ doesn’t show anything, so can just get rid of it.  Basically, we would just have a button ‘Prompts’ that showed the prompts + the decoding parameters.


- [x] More discoverability for clickable table cells (e.g. link icon)
- [x] More discoverability for mouse hover table cells (e.g. dotted underline)
- [x] Bold the top value per column in leaderboard tables
- [x] Uniform column widths
- [x] Navigation between pages e.g. raw predictions should link to scenario and model
- [ ] Some kind of model page
- [ ] Model page could link to ecosystems graph
- [x] Delete the prompt pop-up button, which doesn't do anything
- [x] Include number of characters in parenthesis for input
- [x] On the raw prediction page, "Input" is in grey but "prediction" is in black - make the same
- [x] Don't use red for prompt details button
- [x] Global HELM landing page that links Classic, Lite, etc.
Repository: stanford-crfm/helm
Issue: Preserve legacy quota groups
Root cause: Architectural Constraint

In the current version of HELM (v0.4.0), all OpenAI models have the same quota group (`openai`) but previously the models were in the quota groups `gpt3`, `gpt4` or `codex`. We need to make the server backwards compatible with these quota groups so that people can continue to use the server.
Repository: stanford-crfm/helm
Issue: English-specific quirks of F1 overlap score
Root cause: Validation Gap

Created from @brianwgoldman's [comment](https://github.com/stanford-crfm/helm/issues/2252#issuecomment-1898732062) in #2252:

> **Summary**
>
> Take a look at what values are getting passed in [here](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L107), and what they are after applying `normalize_text`.
>
> **Details**
>
> Based on [these lines](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/static/schema_lite.yaml#L218-L219), I believe the F1 you are seeing in that table is computed by [this method](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L107). That ends up getting called [here](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L305). If I'm reading the code right, the `f1_score` function is being called with `pred` equal to the [highest probability completion](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L349) and each [is_correct reference's text](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L344) as `gold`.
> 
> If that is all true, then f1_score is computed on a per-request basis, such that:
> * precision is roughly "words in common between prediction and reference" / "words in prediction"
> * recall is roughly "words in common between prediction and reference" / "words in reference"
> 
> An important note is that before doing that computation, it [does normalization](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/benchmark/metrics/evaluate_reference_metrics.py#L41) which among other things removes articles like "a". So I *think* if your `is_correct` reference only contains "A" or if your model's prediction is "A", your f1_score will be zero.

This also mean `f1_score` is only valid for English, because it applies to English articles only.
Repository: stanford-crfm/helm
Issue: Instruction following prompt can be strange
Root cause: Documentation Deficiency

Reported by @JosselinSomervilleRoberts 

```
Human: Here are some input-output examples. Read the examples carefully to figure out the mapping. The output of the last example is not given, and your job is to figure out what it is.

The following are multiple choice questions (with answers) about common sense.

[truncated]

Question: A person wants to start saving money so that they can afford a nice vacation at the end of the year. After looking over their budget and expenses, they decide the best way to save money is to
A. make more phone calls
B. quit eating lunch out
C. buy less with monopoly money
D. have lunch with friends
Answer:

Please provide the output to this last example. It is critical to follow the format of the preceding outputs!

Assistant: Answer:
```

@JosselinSomervilleRoberts points out it's not obvious that "Answer: [...] Assistant: Answer:" is the best prompt - should there be only the first "Answer:", only the second, or both? This could even be model specific.

We changed things so that "It is critical to follow the format of the preceding outputs!" the prompt is no longer the default for instruction following models in #2550. It now must be explicitly set using the `follow_format_instructions` run expander.
Repository: stanford-crfm/helm
Issue: output more information when benchmarking
Root cause: Unimplemented Feature Gap

When benchmarking, print out
- the `Instance`s of a `Scenario`
- the `RequestState`s that are produced by the `Adapter`
- the `Metric`s and the resulting `Stat`s.

Output:
- Summary statistics to stdout using `htrack` and `hlog` (this is basically fine)
- Human readable to a text file.
- Serializable information to a json file.

Format for text file:
```
Scenario ${scenario.name}
${scenario.description}
Tags: ${scenario.tags}
${n} instances

------- Instance ${i}/${total}: ${instance.tags}
Input: <input>
Reference(tags): ${reference.text}
...
Reference(tags): ${reference.text}
```

Where to store these files:
- Each `RunSpec` should be given a `path` (e.g., `reasoning/mmlu/chemistry`) which a new field of `RunSpec`.
- Output to `scenario.{txt,json}`, `scenario_state.{txt,json}`, `metrics.{txt,json}` in that `path`.
Repository: stanford-crfm/helm
Issue: Runs use both test and eval splits but summarized metric does not
Root cause: Configuration Error

When selecting evaluation instances for runs, we always sample from both test and eval splits.

However, `helm-summarize` only computes the top level accuracy metric from only test or eval (e.g. [MMLU only uses test](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/static/schema_lite.yaml#L732)), so many of the instances are wasted.

We now support the `eval_split=test`  run expander (example run entry: `mmlu:model=text,eval_split=test,subject=anatomy`). Users still have to remember to manually set this in the run entry to match the split provided to `helm-summarize`'s `schema.yaml`
Repository: stanford-crfm/helm
Issue: HELM summary tables do not enforce that metrics come from the same runs across different models
Root cause: Validation Gap

For instance, if we run model A on only MMLU abstract algebra, and model B MMLU chemistry, the main MMLU group will happily show the numbers from model A and model B side-by-side even though the numbers are not comparable.

This means that in HELM classic, some tables are bogus like the [copyright one](https://crfm.stanford.edu/helm/v0.3.0/?group=copyright_text).
Repository: stanford-crfm/helm
Issue: Missing CLEVA file
Root cause: External Dependency Breakage

CLEVA depends on a file at the following URL, but the URL returns a 404 Not Found error:

http://39.108.215.175/data/v1/dialogue_generation.zip

@Jianqiao-Zhao is there an updated URL for the `dialogue_generation.zip` file?

Hi @yifanmai , the file was accidentally deleted on our file server. I am working on it to fix this problem as soon as possible. Sorry for the inconvenience!

Hi @yifanmai , I have fixed this problem. Please try again to see if it persists. Thank you very much!

Looks good, thanks!
Repository: stanford-crfm/helm
Issue: Proxy server does not charge for Anthropic model usage
Root cause: Validation Gap

Current, Anthropic model usage is [charged using `FreeTokenCounter`](https://github.com/stanford-crfm/helm/blob/v0.4.0/src/helm/proxy/token_counters/auto_token_counter.py#L32-L33), which means that users with any Anthropic quota can use unlimited amounts of Anthropic quota.
Repository: stanford-crfm/helm
Issue: How to summarize runs with HEIM?
Root cause: Documentation Deficiency

Hi,

Thank you for coming up with this amazing benchmark for text-to-image models!

I am trying to use the HEIM to reproduce some of the results of the paper, and am having troubles summarizing the raw runs into tables. 

For HELM, we can get the results with this command:
```
# Summarize benchmark results
helm-summarize --suite v1
```


I tried the same with HEIM, replacing the `--suite v1` argument with `--suite heim_v1`, and I cannot get any summary table.

The reason probably is the schema_classic.yaml file doesn't contain any HEIM related information: https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/static/schema_classic.yaml

Could the authors help me confirm if that is the case? And if so, could you provide some insights of how to perform the summary with HEIM?

Thank you very much!



HEIM is currently not supported by `helm-summarize`, but there are plans to add it soon. cc @teetone

Hi @gaojl19, you can now run `helm-summarize` for HEIM by downloading `schema_heim.yaml` from [here](https://github.com/stanford-crfm/helm/blob/a186baeb00660710758a72b900f6645fa02e7beb/src/helm/benchmark/static/schema_heim.yaml) and running:

```sh
helm-summarize --schema-path path/to/schema_heim.yaml --suite your-suite
```

You can also refer to the [Reproducing Leaderboards](https://crfm-helm.readthedocs.io/en/latest/reproducing_leaderboards/) documentation for instructions.
Repository: stanford-crfm/helm
Issue: Language modeling scenarios do not work with OpenAI models
Root cause: External Dependency Breakage

It looks like _all_ OpenAI models, including the legacy text completion models (`davinci-002` and `babbage-002`) no longer support setting `max_tokens` to 0.

If you attempt to send a request with `max_tokens` set to 0 to `openai/davinci-002`, you will get the cryptic error message: "The server had an error while processing your request. Sorry about that!"

If you attempt to send the same request to `openai/gpt-3.5-turbo-0301`, you will get the less cryptic error message: "0 is less than the minimum of 1 - 'max_tokens'"

This means that language modelling scenarios will no longer work with OpenAI at all, because they set `max_tokens` to 0.

OpenAI models now support logprobs, but _only_ on generated text, not prompt / echoed text. I don't foresee this changing any time soon. There is nothing that can be done here.
Repository: stanford-crfm/helm
Issue: How to download the leaderboards as csv or json format?
Root cause: Documentation Deficiency

I found there is no way to systematically investigate those results without having them as json files.

You can download a zip file for the leaderboards for HELM Classic v0.3.0 [here](https://storage.googleapis.com/crfm-helm-public/benchmark_output/archives/v0.3.0/run_stats.zip). Also refer to this [doc](https://crfm-helm.readthedocs.io/en/latest/get_helm_rank/).

> You can download a zip file for the leaderboards for HELM Classic v0.3.0 [here](https://storage.googleapis.com/crfm-helm-public/benchmark_output/archives/v0.3.0/run_stats.zip). Also refer to this [doc](https://crfm-helm.readthedocs.io/en/latest/get_helm_rank/).

Thanks for your reply. However, it is still hard to interpret the run results and correspond them to the current version of scenarios one by one directly without any documentation...

BTW, are there any downloadable files for HEIM or HELM (lite)? @yifanmai 

There are no archives; you have to download from GCS directly (instructions [here](https://github.com/stanford-crfm/helm/blob/cace4fbdea241e47549eced910d3072c6714eb4b/docs/downloading_raw_results.md)).
Repository: stanford-crfm/helm
Issue: What is `prompt_num_line` parameter?
GitHub labels: user question

![image](https://github.com/stanford-crfm/helm/assets/8592144/961b8680-c216-485a-b5cf-649e86046b2d)
I fail to find any related specification on this parameter:
![image](https://github.com/stanford-crfm/helm/assets/8592144/46858f40-4fff-44cf-8a25-159807b70576)



There is no additional documentation for this parameter. These files were used for experiments for the copyright aspect by using samples of Linux source code.

> There is no additional documentation for this parameter. These files were used for experiments for the copyright aspect by using samples of Linux source code.

Does it indicate the line of code used for the prompt?
Repository: stanford-crfm/helm
Issue: The two leaderboards are the same as each other.
Root cause: Architectural Constraint

![image](https://github.com/stanford-crfm/helm/assets/8592144/a9383da9-de97-4a32-95bc-86e56449698f)
I cannot notice any difference between them...

This is due to a [hack](https://github.com/stanford-crfm/helm/blob/5c8c19460eb2ea3a4da4cd51a67a304ca002f138/src/helm/benchmark/presentation/summarize.py#L1006) for computing results for the original HELM paper.

> This is due to a [hack](https://github.com/stanford-crfm/helm/blob/5c8c19460eb2ea3a4da4cd51a67a304ca002f138/src/helm/benchmark/presentation/summarize.py#L1006) for computing results for the original HELM paper.

Any plan to reformat it?

This is not planned.

> This is not planned.

Thanks!
Repository: stanford-crfm/helm
Issue: What is `Vary multiple-choice strategy`?
GitHub labels: user question

![image](https://github.com/stanford-crfm/helm/assets/8592144/3057f670-32e4-436f-b40c-64c503674fb0)
I checked the original paper and documentation but failed to find any clue.


This refers to the ablation study in which different multiple choice adaptation methods (joint, separate and separate calibrated) and used with the same models and scenarios. Please refer to Section 8.2 _Prompting analysis_ on page 56 to 59 of the [HELM paper](https://arxiv.org/abs/2211.09110) for more information, especially Figure 33.
Repository: stanford-crfm/helm
Issue: What is `Vary prompting`?
GitHub labels: user question

![image](https://github.com/stanford-crfm/helm/assets/8592144/3efce431-b4a1-4d89-aeb3-47e2dab00620)
Any further elaboration on this? I cannot find any specifications on the original paper or this repo...


Please refer to Section 8.2 _Prompting analysis_ on page 56 of the [HELM paper](https://arxiv.org/abs/2211.09110) for more information.
Repository: stanford-crfm/helm
Issue: Some scenarios are missing from the documentation
Root cause: Documentation Deficiency

Such as `Vary multiple-choice strategy`, `vary prompting`, and `vary number of in-context examples` could not be found in the [scenario documentation](https://crfm.stanford.edu/helm/classic/latest/#/scenarios)
![image](https://github.com/stanford-crfm/helm/assets/8592144/88b749b4-612b-4916-b4d6-629b362d286e)


These are not scenarios, and therefore are not listed in the list of scenarios.

> These are not scenarios, and therefore are not listed in the list of scenarios.

Thanks. But, weirdly, I fail to find a definition of the scenario in your HELM paper. What is exactly regarded as a valid scenario in the leaderboards?  The current layout and documentation seem confusing and misleading in those aspects... @yifanmai
Repository: stanford-crfm/helm
Issue: Perspective API key expiring 4/15/2022
Root cause: Configuration Error

https://github.com/stanford-crfm/benchmarking#perspective-api

Request submitted on 4/6/22 for `hai-gcp-models` GCP account.

Google graciously increased our query limit for PerspectiveAPI from 100 to 200. Expires in July 2022.
Repository: stanford-crfm/helm
Issue: Unable to reproduce HELM Lite leaderboard
Root cause: Documentation Deficiency

Dear HELM developers,

I downloaded HELM Lite raw data and came trying to reproduce the scores in the leaderboard. For each scenario, I am averaging the average correctness across sub-scenarios (I am considering that scenarios with no subscenarios 'have' one subscenario). The numbers match for all scenarios except 'med_qa', 'mmlu', 'narrative_qa', and 'wmt_14'. Is there anything different with these specific scenarios that I should know? What is the proper way of reproducing the leaderboard numbers from correctness data?

Thank you for your help!

Thank you for your work on reproducing the scores! For these scenarios, the issue is probably that the evaluation runs contain both "test" and "valid" splits, but the reported number on the leaderboard is computed over the "test" split only. This issue is tracked in #2300.

Thank you so much, Yifan! 

It's working perfectly now!
Repository: stanford-crfm/helm
Issue: Evaluation results not available in json files
Root cause: Documentation Deficiency

Hi,
I found the results for different benchmarks and models(such as [this](https://crfm.stanford.edu/heim/latest/?runs=1&runSpec=parti_prompts%3Acategory%3DIndoor%2Cmodel%3Dhuggingface_promptist-stable-diffusion-v1-4%2Cmax_eval_instances%3D100)) quite useful. However, in the [json](https://nlp.stanford.edu/helm/v1.1.0/benchmark_output/runs/latest/parti_prompts:category=Indoor,model=huggingface_promptist-stable-diffusion-v1-4,max_eval_instances=100/scenario_state.json), the results such as  Expected/Maximum CLIP score and Expected/Maximum aesthetics score aren't available. 
Is there a script I can use to extract this information? Otherwise, can the authors release these results in an appropriate format?

Hi @adhirajghosh, you can get the scores by changing `scenario_state.json` to `per_instance_stats.json` in the URL. Here is a [link](https://nlp.stanford.edu/helm/v1.1.0/benchmark_output/runs/latest/parti_prompts:category=Indoor,model=huggingface_promptist-stable-diffusion-v1-4,max_eval_instances=100/per_instance_stats.json) to an example file.

There's a tracking issue to add a direct link to this file: #1705

Thank you! This was quite helpful. Any possibility of being able to access/download the images as well?

I am not sure how to download image files for HEIM. @teetone could you advise?

Hi @adhirajghosh , for HEIM, the images are not actually stored in GCS as image files. Instead, you should access the `display_predictions.json` file in the same folder, which will contain a field called `base64_images`, which is a string array in which each string is a base64 encoded image.

cc @teetone
Repository: stanford-crfm/helm
Issue: Are the listed models in grey deprecated yet?
GitHub labels: user question

![image](https://github.com/stanford-crfm/helm/assets/8592144/b678261a-e971-480d-8d02-69d2f1aaaec8)
Check https://crfm.stanford.edu/helm/classic/latest/


These models have been deprecated, but their results will not be removed from earlier Classic versions.

> These models have been deprecated, but their results will not be removed from earlier Classic versions.

Thanks, it is better to clarify this fact in the documentation or new user would make the same question :)