zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Add a flag for disabling caching
Root cause: Unimplemented Feature Gap

Use cases

- Sqlite doesn't work on Azure ML for unknown reasons (#1914)
- When developing models, different models or checkpoints might be rapidly swapped in and out without changing the model name
    - NeurIPS Efficiency Challenge's client [disabled caching for this reason](https://github.com/stanford-crfm/helm/blob/main/src/helm/proxy/clients/http_model_client.py#L28)
    - DataComp wants to disable the cache as well

Sketch:

We currently have a [`--mongo-uri` flag](https://github.com/stanford-crfm/helm/blob/v0.3.0/src/helm/benchmark/run.py#L174-L179), which is ugly because (1) it's MongoDB specific, but caches might eventually use other databases and (2) no one else other than us uses it. So we should replace this flag.

1. Make this an `ObjectSpec` for a `Cache` (or a `KeyValueStore`). This would be more aligned with what we do for clients / tokenizers, but a bit cumbersome because it means putting JSON in a CLI flag, which we have not done before.
2. Same as idea 1, but put the `ObjectSpec` in a `prod_env/cache_config.yaml`.
3. Make this a `--cache-uri` - set this to empty string or "null:" or "blackhole:" to disable the cache, "file:..." (or "sqlite:.." - this is not standard) for SQLite, or "mongodb:..." for MongoDB. Drawbacks - not every cache backing store will have a URI scheme; some URI schemes might be ambiguous (maybe some other database technology also wants to use "file:...")

Additional thought: Should we disable caching by default? 

Also need to make a `BlackHoleKeyValueStore` / `BlackHoleCache` that can be passed to the clients to keep them happy.

Not sure we want to disable caching by default (or at least we should do it cautiously since if people are depending on it, they could accidentally burn through a bunch of credits).

What about a simple `--skip-cache` flag?  I could imagine development where you do use a cache, but you want to temporarily turn it off.

I'm fine with adding a `--skip-cache` flag as well. It does make the internals a bit messier (because you're passing both `mongo_uri` and `skip_cache` through a bunch of functions)

Can we convert `mongo_uri` into a `Optional[str]` when you're passing through?

Nope - `mongo_uri is None` means that you want the Sqlite cache, not that you want to disable the cache.
Repository: stanford-crfm/helm
Issue: Parallelize tokenization
Root cause: Architectural Constraint

Currently tokenization is not actually parallel, because all threads use a single tokenizer and the underlying tokenizer is thread-hostile i.e. can only be used in one thread a time - see #1421 and the upstream issue in tokenizers.

When setting `--num-threads=N`, we should actually spawn N fast tokenizers, one for each of N threads, so that we actually get N-fold parallelism.

Currently tokenizers are singletons, and it would be a fairly big architectural change to make them not singletons, so I'm closing this as won't fix.
Repository: stanford-crfm/helm
Issue: Dockerize helm for deployment
Root cause: Unimplemented Feature Gap

Hi 
I am facing challenges in deploying helm code which have cloned to our server
1.First i tried with app service but it was getting stuck 

2.Please help me in how to dockerize this so that i can deploy on aws ec2 instance 


Hi @kapilmayank, we are currently not planning on publishing Docker images for HELM.

Dockerizing this should be similar to dockerizing any other Python package. You can refer to the [documentation](https://crfm-helm.readthedocs.io/) for installing and using HELM. You can also consider using a tool like [pipx](https://github.com/pypa/pipx).

In order to ensure reproducibility I think publishing images is necessary. The python lockfile requirements are not enough, especially consider that they change depending on the benchmarks that you want to run. Docker gives you the ability to verify the entire environment (modulo hardware) with a single hash. I think adding some container support would greatly improve the usability of the project. 

I think it might be in scope for me to work on this. If you're interested in adding a dockerfile to the repo I will push it up as a PR.
Repository: stanford-crfm/helm
Issue: HELM image visualization
GitHub labels: p2

frontend

HEIM (Text2Image)

Does the HELM frontend visualization support the display of image content?

Unfortunately the frontend code in the current main branch does not support displaying image content, but we are working on it.

The [HEIM website](https://crfm.stanford.edu/heim/latest/) uses a uncommitted branch to display images.

cc @teetone
Repository: stanford-crfm/helm
Issue: Some metrics could not be accessed...
Root cause: Documentation Deficiency

https://crfm.stanford.edu/helm/latest/#/groups/cleva_paraphrase_generation_metrics
![image](https://github.com/stanford-crfm/helm/assets/8592144/f77d9cbb-c3da-4ace-9921-9785cdc6671a)


The results are not accessible because CLEVA results have not yet been published.

Closing as duplicate of #2008.
Repository: stanford-crfm/helm
Issue: How to download the whole evaluation results for the new version of leaderboard?
Root cause: Unimplemented Feature Gap

@JosselinSomervilleRoberts @dlwh @yifanmai Any update?

The previous file download shortcut vanished...

Any update? @JosselinSomervilleRoberts 

You can download v0.3.0 results following the instructions [here](https://crfm-helm.readthedocs.io/en/latest/get_helm_rank/).

We don't have a corresponding zip file for v0.4.0 or lite yet.
Repository: stanford-crfm/helm
Issue: The pipeline break up for local model
Root cause: Validation Gap

I ran HELM (mainline) with my local model 
`helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 1000 --enable-local-huggingface-models ${MODEL_DIR}`
The inference can finish successfully. However, I keep getting errors when I run `helm-summarize --suite v1`. I found the line of code throwing the error:
https://github.com/stanford-crfm/helm/blob/2ff3803ab17e8515672b72694e3aac9c78836984/src/helm/benchmark/model_deployment_registry.py#L136 

I did not see this issue with the commit earlier (maybe the one in mid Oct).

@JosselinSomervilleRoberts could you take a look?

@zluw1117 I created #2064, can you check if that fixes your problem pls? Thanks.
Repository: stanford-crfm/helm
Issue: How to count the number of scenarios?
Root cause: Documentation Deficiency

https://crfm.stanford.edu/helm/classic/latest
![image](https://github.com/stanford-crfm/helm/assets/8592144/9d15dc61-1c29-49ca-bb11-84ee7d0f8e57)
versus
https://crfm.stanford.edu/helm/classic/latest/#/scenarios
![image](https://github.com/stanford-crfm/helm/assets/8592144/8588056e-39a9-4ee5-b5b6-c844da36b7c3)

The count depends on whether you are counting:

1. the number of scenarios that are actually run on the HELM leaderboard, versus the number of scenarios that are implemented in the HELM library
2. the number of unique scenarios, versus the number of unique combinations of scenarios with adapters

> The count depends on whether you are counting:
> 
> 1. the number of scenarios that are actually run on the HELM leaderboard, versus the number of scenarios that are implemented in the HELM library
> 2. the number of unique scenarios, versus the number of unique combinations of scenarios with adapters

Thanks, however, there are still inconsistencies between the number of counts. Right now, I have little idea about which page uses what criteria for scenario counting, and this makes me very confused as an outsider. 

@yifanmai but I think this criterion is better to be mentioned somewhere in the documentation...
Repository: stanford-crfm/helm
Issue: Is HEIM an expansion of HELM or more independent?
GitHub labels: user question

I am curious about the structure of the two holistic evaluation frameworks. Since text-to-image also involves language modeling, does it mean that HEIM can be regarded as an expansion of HELM?
![image](https://github.com/stanford-crfm/helm/assets/8592144/ea047348-7254-471f-93f3-2a4764fece45)

HEIM was a fork of HELM, we are now in the process of merging it in HELM
Repository: stanford-crfm/helm
Issue: How to count the number of models?
GitHub labels: user question

![image](https://github.com/stanford-crfm/helm/assets/8592144/f44eb112-e4fe-4407-b9d6-ffcacf2244f3)
I count the number of models and it is 81 rather than 119. @dlwh @yifanmai @percyliang @chrisdonahue

119 is the total number of listed models. 81 is the number of models with results. The count will be updated to only models with results in the next version as part of #2195.
Repository: stanford-crfm/helm
Issue: These two links seem not work in https://crfm.stanford.edu/heim/latest/
Root cause: Configuration Error

![image](https://github.com/stanford-crfm/helm/assets/8592144/9128de72-63f9-448c-a763-1897138a966d)


Works for me. What is it redirecting you to?

Paper: https://arxiv.org/abs/2211.09110
GItHub: https://github.com/stanford-crfm/helm
Are these expected?
That does make sense when HEIM is not a subset of HELM. 

> Works for me. What is it redirecting you to?

If this holds, then https://github.com/stanford-crfm/helm/issues/2029#issuecomment-1832944851 makes more sense:
So eventually HELM will include HEIM, right? Because I cannot raise any issues regarding HEIM in its repo, which is very weird. @JosselinSomervilleRoberts 

These two links are currently working, and go to the HEIM paper and this repository respectively.

The code for HEIM has been merged as part of #1939.

> These two links are currently working, and go to the HEIM paper and this repository respectively.
> 
> The code for HEIM has been merged as part of #1939.

It seems the first one still not working, would you like to do a quick check?
It will direct me to https://arxiv.org/abs/2211.09110, is that expected?
@yifanmai 

Thanks for catching; it should go to the HEIM paper instead. I will update the link.

I have fixed the link. Thanks again for finding this.
Repository: stanford-crfm/helm
Issue: How to count the number of scenarios of HEIM?
GitHub labels: user question

HEIM (Text2Image)

![image](https://github.com/stanford-crfm/helm/assets/8592144/5e05109e-6428-4bc6-afee-fc46fe7bd686)
I count the number of scenarios and it is 42 rather than 29. 

cc @teetone 

The counts should be fixed for future versions, thanks to #2154. However, the specific website version you linked to will not be fixed, because we generally do not back-patch old website versions.

I would suggest looking at the [HEIM paper](https://arxiv.org/abs/2311.04287) for the list of scenarios and their descriptions.
Repository: stanford-crfm/helm
Issue: There is no way to raise issues on https://github.com/stanford-crfm/heim
GitHub labels: user question

HEIM (Text2Image)

![image](https://github.com/stanford-crfm/helm/assets/8592144/a8936578-6a06-4eb7-8fdf-5dc0ec7b7db8)
@teetone 

The HEIM repository has been removed. HEIM issues should be filed here. cc @teetone
Repository: stanford-crfm/helm
Issue: How to count the number of metrics?
Root cause: Algorithmic Error

![image](https://github.com/stanford-crfm/helm/assets/8592144/7bed58bc-fddf-43b7-8f33-fa987eac9535)
I count the number of metrics and it is 54 rather than 110. @dlwh @yifanmai @percyliang @chrisdonahue

The metrics count was fixed by #2152 and removed by #2166.
Repository: stanford-crfm/helm
Issue: The json file is not accessible in HEIM
Root cause: Configuration Error

https://crfm.stanford.edu/heim/latest/benchmark_output/runs/v1.1.0/groups/json/mscoco_base_mscoco_base_.json
![image](https://github.com/stanford-crfm/helm/assets/8592144/15936eea-e8cd-45fa-b031-b7b1da720c8d)


cc @teetone 

This seems to be fixed. The file is located at https://nlp.stanford.edu/helm/v1.1.0/benchmark_output/runs/latest/groups/mscoco_base.json
Repository: stanford-crfm/helm
Issue: Why some metrics are unavailable?
Root cause: Unimplemented Feature Gap

You mention you evaluate Chinese metrics:
![image](https://github.com/stanford-crfm/helm/assets/8592144/ada2d2a9-eb2b-400b-90c6-e780dc7ea58c)
But none of the above metrics show up in the leaderboards since those benchmark results are not available...
![image](https://github.com/stanford-crfm/helm/assets/8592144/cd78d1ff-c28a-4950-b838-8c94d2f4be21)
@dlwh @yifanmai @percyliang @chrisdonahue @unnawut 


I think the scenarios have been added but we did not run them yet. cc @yifanmai 

Duplicate of #2025
Repository: stanford-crfm/helm
Issue: Empty leaderboard
Root cause: Documentation Deficiency

I checked IMDB (only_contrast:True) but found the whole leaderboard is empty.
![image](https://github.com/stanford-crfm/helm/assets/8592144/1e086432-f940-4903-93db-58bae0eed25f)
I also checked IMDB and IMDB (only_contrast:False) but found they are the same:
![image](https://github.com/stanford-crfm/helm/assets/8592144/e8e9b79b-ec1e-4d65-9d4c-1945abe7368f)
![image](https://github.com/stanford-crfm/helm/assets/8592144/b5f11349-672a-479c-998e-3bbc80600d84)
What is the meaning of having three leaderboards? @yifanmai 

The same thing happens for BoolQ.

For the contrast sets on IMDB and BoolQ, the "only_contrast: True" results are only accessible via the ["Robustness to contrast sets" group](https://crfm.stanford.edu/helm/v0.3.0/?group=robustness_contrast_sets).
Repository: stanford-crfm/helm
Issue: Duplicate leaderboards
Root cause: Algorithmic Error

![image](https://github.com/stanford-crfm/helm/assets/8592144/47652e66-f56f-4bbb-93d2-4c634d1c7ad5)
and
![image](https://github.com/stanford-crfm/helm/assets/8592144/455c52b1-7a32-4171-b5bc-b5a5cc03ee86)
and
![image](https://github.com/stanford-crfm/helm/assets/8592144/88f7769e-2d6f-498f-b588-7a93cc060061)


cc @farzaank 

This is intended - there are multiple groups that will contain the same sub-group table.

Are those evaluation results on the first leaderboard the average of the last two? @yifanmai
Repository: stanford-crfm/helm
Issue: The website is suddenly broken when I successively click on the groups
Root cause: Resource Mishandling

![image](https://github.com/stanford-crfm/helm/assets/8592144/bc457972-3c62-4d14-b2d5-a5c89853c2e2)
![image](https://github.com/stanford-crfm/helm/assets/8592144/29389aee-fe93-4eed-ba23-07bf5099289c) 
The page is suddenly crashed. You can reproduce it by clicking on the groups back and forth about 10 times successively.


I was indeed able to reproduce it I only a few clicks. cc @farzaank 

I am looking into the root cause of this but it likely will not be addressed until the new year. 

Fixed, closed!
Repository: stanford-crfm/helm
Issue: Leaderboard with no name?
Root cause: Validation Gap

![image](https://github.com/stanford-crfm/helm/assets/8592144/ab00044a-a3d2-48f3-aec3-7fc0e6ea4af1)


cc @farzaank 

Closing this because HELM Classic is a deprecated leaderboard and will not be updated.
Repository: stanford-crfm/helm
Issue: [enhancement] can we make this set of leaderboard into a control panel?
Root cause: Unimplemented Feature Gap

![image](https://github.com/stanford-crfm/helm/assets/8592144/7a19f478-c3ce-40c6-9f1f-99b4fc23cba6)
I found `synthetic efficiency` has **60** sub-leaderboards based on the num_prompt_tokens, num_instances, and tokenizer.
For NLP practitioners, it is super hard to locate the needed leaderboard at first glance. I wonder if you can make this table into a control panel so they can retrieve the needed information quickly.

Another manner is to provide a manner to retrieve all the leaderboards in a programmatical manner so they can check any of those interested: https://github.com/stanford-crfm/helm/issues/2026


cc @farzaank 

This is a quirk in `summarize.py` - we tried to include only the relevant adapter attributes, but it doesn't actually know which attributes are relevant and has to heuristically guess.

If you'd like a more user-friendly UI, you could look at the [v0.3.0 version of this page](https://crfm.stanford.edu/helm/classic/v0.3.0/?group=synthetic_efficiency) on the old frontend.

> This is a quirk in `summarize.py` - we tried to include only the relevant adapter attributes, but it doesn't actually know which attributes are relevant and has to heuristically guess.
> 
> If you'd like a more user-friendly UI, you could look at the [v0.3.0 version of this page](https://crfm.stanford.edu/helm/classic/v0.3.0/?group=synthetic_efficiency) on the old frontend.

Thanks, but since `num_instances=10` is the default setting for all 60 leaderboards, why bother to include it in the name of each leaderboard?

Classic is archived; won't fix.
Repository: stanford-crfm/helm
Issue: What does the white-out models mean in HEIM?
GitHub labels: user question

HEIM (Text2Image)

![image](https://github.com/stanford-crfm/helm/assets/8592144/ec3e407f-a578-47f7-9490-3b174c11995f)


We hope to evaluate these models in the future but have yet to include the results.
Repository: stanford-crfm/helm
Issue: What is the last update date of HELM?
GitHub labels: enhancement

user question

p2

frontend

Since the latest update of HELM, I could not find the last update date as similarly shown in HEIM:
![image](https://github.com/stanford-crfm/helm/assets/8592144/ffcc166f-60e6-47c1-903e-ebb859e5bfaa)


November first as shown on Pypi: https://pypi.org/project/crfm-helm/

@yifanmai , should we add this to the frontend?

> November first as shown on Pypi: https://pypi.org/project/crfm-helm/

That is way too implicit...

Agreed that we should add the release date back to the new frontend

cc @farzaank @percyliang What do you think?

Yes, let's add it back.
Repository: stanford-crfm/helm
Issue: Missing link to LegalSupport paper
GitHub labels: documentation

user question

![image](https://github.com/stanford-crfm/helm/assets/8592144/150c5493-47ee-4887-af75-0a7d1e68ab95)


![image](https://github.com/stanford-crfm/helm/assets/8592144/08821f65-f206-4c4f-9b0f-b27bd4f05f28)
While it is a bit confusing since you do not mention it on the webpage.

@neelguha Has LegalSupport been released yet? Could you add links to the paper if so?

I'm so sorry for the delay–we're working on finishing a writeup and should have it released in the next month or so.

Thanks for the update, Neel!

@neelguha do you have any updates on the LegalSupport release?
Repository: stanford-crfm/helm
Issue: [enhancement] add a link to synthetic efficiency
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/8592144/5fa0eb94-084c-4470-bb29-18a77cbe56f8)


Found it finally. I suggest to have a link to the source: https://worksheets.codalab.org/bundles/0x17a361bc066b4b0e87d968069759d361

Classic is archived; won't fix.
Repository: stanford-crfm/helm
Issue: Where is the inference runtime performance?
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/8592144/b1959ac9-a0dd-4e6f-885e-3f3655db3b40)
I failed to find any of those related information here.


Refer to the HELM paper for information on synthetic efficiency. This was from HELM Classic, which is now deprecated; we do not currently support computing synthetic efficiency on any of the currently supported leaderboards.
Repository: stanford-crfm/helm
Issue: [enhancement] add link to Demographic Stereotypes in HEIM
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/8592144/c3f9cd75-be49-427b-9a65-242986d78a29)


This table is explained further in the [HEIM paper](https://arxiv.org/abs/2311.04287). In general, most of the tables in the HEIM website are intended to be interpreted in the context of the paper.
Repository: stanford-crfm/helm
Issue: [enhancement] add link to DrawBench in HEIM
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/8592144/0a8e8d17-9d12-40fc-969e-90c632e65f0e)


I attempted to look it up in the repo but...
![image](https://github.com/stanford-crfm/helm/assets/8592144/a2746c91-379e-4c9a-8a65-35135cbc1624)
![image](https://github.com/stanford-crfm/helm/assets/8592144/99df9394-4e73-4828-b908-cd40d676d710)
Still have no idea where the prompt datasets are... @teetone 


I found it, so it is better to mention it in the webpage: https://docs.google.com/spreadsheets/d/1y7nAbmR4FREi6npB1u-Bo3GFdwdOPYJc617rBOxIRHY/edit#gid=0
Repository: stanford-crfm/helm
Issue: It seems this leaderboard is the same across scenarios in HEIM, any explanation?
Root cause: Algorithmic Error

![image](https://github.com/stanford-crfm/helm/assets/8592144/f2b12d5c-157c-474f-97be-70ae187b265c)
I do not see any differences between those two scenarios, why not combine them together?


Also for [dailydall.e](https://crfm.stanford.edu/heim/latest/?group=daily_dalle)
[Logos](https://crfm.stanford.edu/heim/latest/?group=logos)
[Magazine Cover Photos](https://crfm.stanford.edu/heim/latest/?group=magazine_cover)
![image](https://github.com/stanford-crfm/helm/assets/8592144/105d96e1-5707-4745-b30b-003c3bca5617)
Repository: stanford-crfm/helm
Issue: [enhancement] add link to Mental Disorders in HEIM
Root cause: Documentation Deficiency

It is hard to find the original prompts in https://crfm.stanford.edu/heim/latest/?group=mental_disorders
I need to hack into the original repo and find it in the script.

This table is explained further in the [HEIM paper](https://arxiv.org/abs/2311.04287). In general, most of the tables in the HEIM website are intended to be interpreted in the context of the paper.
Repository: stanford-crfm/helm
Issue: PartiPrompts scenario description should link to the original paper
GitHub labels: enhancement

user question

p3

HEIM (Text2Image)

from this paper: https://arxiv.org/abs/2206.10789

@teetone

Currently the PartiPrompts description is missing a link to the original paper. This should be added in the HEIM schema (which I don't have access to). See:

![Screenshot 2024-01-05 114105](https://github.com/stanford-crfm/helm/assets/185227/36abfcfa-fbd3-43c5-be61-92813e11493e)
Repository: stanford-crfm/helm
Issue: "--enable-local-huggingface-models" also seems broken
Root cause: Configuration Error

Seemingly in a different manner from #2056 

I've tried a bunch of permutations, but I can't get it to pick up a local model. Basing things on #1505 

run_specs.conf:

```
entries: [
  {description: "gsm:model=huggingface/llama2_gsm8k_lora", priority: 1},
  #{description: "gsm:model=huggingface/meta-llama_Llama-2-7b-hf", priority: 1},
  #{description: "gsm:model=stanford-crfm/BioMedLM@main", priority: 1},
]
```

commands: 

```
helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10  --enable-huggingface-models /scr/biggest/dlwh/llama2_gsm8k_lora/ --enable-local-huggingface-models /scr/biggest/dlwh/llama2_gsm8k_lora/

helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10 --enable-local-huggingface-models /scr/biggest/dlwh/llama2_gsm8k_lora/
```

both fail with     

```
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm/lib/python3.8/site-packages/helm/proxy/models.py", line 923, in get_model
    raise ValueError(f"No model with name: {model_name}")
ValueError: No model with name: huggingface/llama2_gsm8k_lora
```

cc @julian-q 

@yifanmai fwiw i tried this with v0.2.3 (referenced at https://github.com/stanford-crfm/helm/pull/1505#issuecomment-1648352798) and it doesn't work there either. could be i'm misconfiguring?

This should be working now on v0.4.0.