zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Just follow the Quick Start to do, but got Error: No model with name
Root cause: Documentation Deficiency

[Quick Start](https://crfm-helm.readthedocs.io/en/latest/quick_start/)

`ValueError: No model with name: huggingface/gpt2`


The model got renamed recently. Could you try replacing `huggingface/gpt2` with `openai/gpt2`? We should update the docs to reflect this. cc @JosselinSomervilleRoberts 

@prchann I just had a look at the quick start and it looks already fixed so I am going to close this issue
Repository: stanford-crfm/helm
Issue: Metrics not getting displayed on custom scenario
Root cause: Configuration Error

I had created custom scenario on my own dataset
helm run ,helm summarize is working but there are no metrics

![image](https://github.com/stanford-crfm/helm/assets/51744576/0a84fd05-ef59-4952-a9b1-dd8284d69aac)

Also i saw some warnings
 WARNING: run spec codeprogress:mode=classify,model=lambda_FLAN_T5 does not have any stat matched by MetricNameMatcher(name='f1_score', split='valid', sub_split=None, perturbation_name=None), 0 near misses matching just the name
  WARNING: run spec codeprogress:mode=classify,model=lambda_FLAN_T5 does not have any stat matched by MetricNameMatcher(name='ece_10_bin', split='valid', sub_split=None,


Please help me  resolve this asap
and reply on kapilmayank6@gmail.com


You should modify your `schema.yaml` or `schema_classic.yaml` file by modifying the `run_specs` section to include a new run group with name `codeprogress`. You can duplicate an existing run group such as `imdb` or `raft`, change its name to `codeprogress` and modify it as needed.

Closing due to staleness; feel free to reopen if you have follow-up questions.
Repository: stanford-crfm/helm
Issue: The metrics for BOLD are currently unavailable
Root cause: Configuration Error

I evaluated BOLD (Bias in Open-Ended Language Generation Dataset) with the run_spec of `bold:subject=all,model=MY_HF_MODEL`.

After I ran `helm-server` and went to the `raw_runs` tab, there was generation for each question.
However, there were no metrics, and there was only "**Metrics are currently unavailable. Please try again later.**" at the end of the page.

Could you check this bug? Thanks.

### 2024.01.05
![image](https://github.com/stanford-crfm/helm/assets/50437946/b409e9e7-bef0-45b4-ab22-4ee3729a80cd)

I found that `metric_specs` of `run_specs.json` is empty. It should be a bug I believe.

https://github.com/stanford-crfm/helm/blob/49e3c83c523c63a4bcd468cb6afa1df6f9010d42/src/helm/benchmark/run_specs.py#L543

I found that the reason was due to this line.
I believe this was intended, but I'm not sure this line is necessary enough because I was embarrassed at first and needed to read code deeply to understand why I didn't have any metrics.

I think the UI should tell us why it omits some metrics.....
Repository: stanford-crfm/helm
Issue: opencv-python 4.9.0.80 breaks type checking
Root cause: External Dependency Breakage

`mypy` type checking succeeds with `opencv-python==4.8.1.78` but breaks with `opencv-python==4.9.0.80` fails with the following error:

```
src/helm/benchmark/metrics/image_generation/fractal_dimension/fractal_dimension_util.py:61: error: Unsupported operand types for / ("ndarray[Any, dtype[generic]]" and "float")  [operator]
```

cc @teetone FYI. Probably not urgent to fix though.
Repository: stanford-crfm/helm
Issue: helm-frontend should not be under src
Root cause: Architectural Constraint

helm-frontend installs some Node modules that contain Python code. mypy complains about that code, since it typechecks _all_ code under `src`, including `src/helm-frontend/node_modules`. This results in pre-commit producing errors like:

```
src/helm-frontend/node_modules/flatted/python/test.py:3:1: E302 expected 2 blank lines, found 1
```
Repository: stanford-crfm/helm
Issue: Should we pick the eval instances at random?
Root cause: Algorithmic Error

Currently, when `max_eval_instances` is set, we pick the first that-many instances (https://github.com/stanford-crfm/benchmarking/blob/master/src/benchmark/adapter.py#L273). However, not all scenarios are necessarily shuffled. Should we set the seed to something and pick a subset at random?

Fixed in https://github.com/stanford-crfm/benchmarking/pull/260.
Repository: stanford-crfm/helm
Issue: Toxic fraction for QuAC is empty.
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/50437946/0d18e01e-2f8b-4d59-b3da-9feea3fbf3cd)


helm-run \
    --run-specs quac:model=huggingface/7B-FT,max_train_instances=1 \
      --enable-local-huggingface-models /workspace/llama/7B-FT \
      --suite v1 \
    --max-eval-instance 1000000

I used this command to evaluate a local Llama model with QuAC.
I think the results should have stereotypes and toxic fraction for the metric, but they are empty.
Could you help me figure out the reason why some metrics are missing? Thanks.

https://github.com/stanford-crfm/helm/blob/49e3c83c523c63a4bcd468cb6afa1df6f9010d42/src/helm/benchmark/run_specs.py#L542
I found that the reason was this line.
I suggest that you specify in the UI that some metrics are omitted in the lite version.

Thank you.

This is correct - we no longer compute the toxic fraction in QuaC for HELM Lite, because we found that in HELM Classic, the toxic fraction for QuaC is extremely small, as you can see [here](https://crfm.stanford.edu/helm/classic/latest/#/groups/quac).
Repository: stanford-crfm/helm
Issue: Deprecated URL in VicunaScenario
Root cause: External Dependency Breakage

The list of questions used in `VicunaScenario` has been deprecated and removed from the repository. The URL below is no longer valid.

https://github.com/stanford-crfm/helm/blob/2a112cb98cd391d743cb52ab1d63c1b54f7c6a0d/src/helm/benchmark/scenarios/vicuna_scenario.py#L29-L30

Below is the new URL of the original dataset used in `VicunaScenario`.
``https://raw.githubusercontent.com/lm-sys/vicuna-blog-eval/main/eval/table/question.jsonl``

[This README file](https://github.com/lm-sys/vicuna-blog-eval/blob/main/README.md) has more information about the deprecation.

Thanks; I've opened a pull request to fix this.
Repository: stanford-crfm/helm
Issue: All "synthetic efficiency" scenario use `num_instances=10`, why still list this in the leaderboard names?
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/8592144/f27cff73-1069-435f-adc0-e6d009db58a1)
Right now, the 60 leaderboards clutter and make it super hard to retrieve the right one.
I suggest removing redundant information since it is the same across all leaderboards in that scenario.

Closing as we do not intend to revise this version of the Classic leaderboard.
Repository: stanford-crfm/helm
Issue: Any documentation on the metrics?
Root cause: Documentation Deficiency

I attempt to understand the meaning of each metric, particularly for the general information such as `# eval`, `# train`, `truncated`, `# prompt tokens`, `# output tokens`, and `# trials`.
However, these metrics are not mentioned anywhere in the original paper, GitHub repository, or benchmark documentation. Any idea?

You can find the documentation for the metrics in `schema_classic.yaml` [here](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/static/schema_classic.yaml#L63) by searching for those strings.
Repository: stanford-crfm/helm
Issue: HELM logo scaled incorrectly on Safari
GitHub labels: bug

p2

frontend

The logo is scaled or cropped incorrectly on Safari. Probably a CSS issue.

![helm-safari](https://github.com/stanford-crfm/helm/assets/185227/708a4361-e342-4b5c-bc84-6618c4a55d66)


@dlwh also says: "if you remove the w-full class from the logo it looks fine".

Fixed by #2241.
Repository: stanford-crfm/helm
Issue: Sum all tokens used in benchmark-present
Root cause: Unimplemented Feature Gap

Currently, `benchmark-present` (and probably other things) reports per-scenario token usage. It would be great to get the aggregate usage 
More generally, could be good to have some other aggregation - e.g. it could be good to have an ordered list of the form 
[scenario_name (sum of tokens used by scenario)], so it is clear where the tokens are being allocated. 

Not super critical since we have enough tokens for this month.

@teetone Is this resolved based on the computations you did for token usage last week? 

I think this is resolved from Tony's estimates, so will make a P3 but I think we can just close from the estimation scripts. 

@teetone Is this addressed from your scripts (e.g. https://github.com/stanford-crfm/benchmarking/pull/448)? 

Let me know if not, since I can take this o/w.

We should do this on the website, where for each group, we print out the number of tokens and cost.

@teetone I think you did something related to this recently?

We can close this: https://github.com/stanford-crfm/helm/pull/1049
Repository: stanford-crfm/helm
Issue: quac implementation
Root cause: Validation Gap

When I try to evaluate davinci on quac for 10 instances, it seems like the model predicts nothing. It turns out this is even worse, because then the logprob computation yields a division by zero error. I post a screenshot below. 

<img width="1863" alt="image" src="https://user-images.githubusercontent.com/47439426/160533244-f699d8d5-6490-40e3-9e0b-91299078ca46.png">


I get this from running `venv/bin/benchmark-present --max-eval-instances 10 --conf src/benchmark/presentation/run_specs.conf` when I get o QuAC

Hm, this is weird. It ran normally when I tested it and when running on the [website](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=quac).

I am not sure I will get a chance to debug it tonight, but one thing you could try is removing the trailing space in [this line](https://github.com/stanford-crfm/benchmarking/blob/master/src/benchmark/quac_scenario.py#L124).

It was the trailing space. This is fixed in https://github.com/stanford-crfm/benchmarking/pull/257/  and davinci gets 0.519 F1 score.
Repository: stanford-crfm/helm
Issue: Better `sample_examples` strategy for multiple choice
Root cause: Algorithmic Error

When using `MultipleChoiceJointAdapter`, the in context training examples are selected by `sample_examples` defined in InContextLearningAdapter. That method attempts to balance the labels used by training examples [here](https://github.com/stanford-crfm/helm/blob/883877aefcaac43d0a92dfccddb18a4f2f3ad858/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L163). However, this is balancing based on the full text of the correct reference, not the letter answer the model should provide. This can result in a situation where no example uses a specific option, even if there are enough examples to demonstrate all options.

I assume this behavior isn't the intent of `sample_examples`, so opening a feature request to improve it. I'll open a PR separately to demonstrate the current undesirable behavior.

Demonstrating behavior in #2225.

There are other significant bugs in the sampling algorithm...

It says:

> 1. Sort the class labels (i.e., correct References) by the number of Instances that belong to the class so more common labels are included in the in-context examples. Break ties by shuffling.
> 2. Keep sampling one train Instance from each class in the order established in step 1, until there are k examples.
> 3. If we run out of examples to sample, sample the rest from the Instances that do not have class labels.

But in practice, we start sampling from instances with no labels once any class runs out of examples.

If we try to sample five examples from examples with labels `["a", "a", "a", "b", None, None]`, you would get `["a", "a", "b", None, None]`

If we try to sample five examples from examples with labels `["a", "a", "a", "b", None]`, I believe you would actually get `ValueError: Sample larger than population or is negative`.
Repository: stanford-crfm/helm
Issue: HellaSwag evaluation
Root cause: Validation Gap

When evaluating davinci on HellaSwag with `commonsense_qa:dataset=hellaswag,method=clm`, I get an error in the metrics. 
Screenshot:

<img width="1902" alt="image" src="https://user-images.githubusercontent.com/47439426/160534715-2f6389fe-5c97-4b2e-863f-e5e0e3af76ba.png">

I arrive at this error by running `venv/bin/benchmark-present --max-eval-instances 10 --conf src/benchmark/presentation/run_specs.conf` once I get to this scenario. 

@rishibommasani `--max-eval-instances` must be an integer multiple of 4 because each HellaSwag question has 4 choices :)

ah got it :), thanks! 

Can we throw a more descriptive error about this somewhere?

what is the LCM of all the commonsense scenarios, so I can make sure its always a multiple of this

Sure, could you first verify changing to 12 will not lead to the error?

I guess LCM is 60 since we have some 2, 3, 4, 5?

`{"hellaswag": 4, "openbookqa": 4, "commonsenseqa": 5, "piqa": 2, "siqa": 3,}`

Yes :)

Wait, each instance should correspond to a question, even if it has multiple choices, right?

I think this scenario is something that we need to come back to (because the adaptation procedure should be generalized).  Or we can just stick with the MMLU adaptation procedure which would require only one request per instance.

Could see various arguments, but one approach that seems reasonable to me is:
(a) For all multiple choice, we have a single query as the **default**. This is the thing in the final benchmark 
(b) We have some analyses for specific datasets on how the adaptation procedure changes things and, in particular, whether it changes the relative ranking of models.



Using a multiple resolved the issue for the CLM-style adaptation, so that is good. 

Did introduce a new issue for PIQA:
<img width="825" alt="image" src="https://user-images.githubusercontent.com/47439426/160538498-cce97398-210f-4012-bbc6-77b657f438fa.png">


@percyliang Yeah there are multiple ways to do this:

1. As @rishibommasani suggested, use MMLU style as default and say we also implement other methods and compare with MMLU style for some MCQA (only commonsense QA at this time?) 

Other possible solutions is to refactor Adapter, as it says "Note that an `Instance` could produce many `Request`s (e.g., one for each `Reference`)." in the class comment, but this feature seems still not to be implemented yet.  

2. Shallow refactor: in the `Adapter` class, if we use CLM style for MCQA, we multiply the instance by number of choices, so the outer interface aligns with others.

3. Deep refactor: support multiple requests per instance in the `Adapter` class.

 

 

@rishibommasani Could you provide the command that triggers the new bug?

I just ran the following commands but didn't get the problem:

```sh
venv/bin/benchmark-run -r commonsense_qa:dataset=piqa,method=mcqa
venv/bin/benchmark-run -r commonsense_qa:dataset=piqa,method=clm
```

Can we close this then?

yup, this is resolved
Repository: stanford-crfm/helm
Issue: Link to (regex) filtered results page
Root cause: Unimplemented Feature Gap

It would nice if I could go to e.g. https://nlp.stanford.edu/helm/v1.1.0-canary/#/runs?q=mmlu.*gemini&regex=true and see the results.

And ideally the page would automatically update the url in response to edits in the box...

https://github.com/stanford-crfm/helm/pull/2155 handles this- once it's merged I'll close

This works now!
Repository: stanford-crfm/helm
Issue: Switching to efficiency in Leaderboard messes up the ordering
Root cause: Algorithmic Error

When I click on efficiency in the leaderboard, the order is weird (intuitively I think it should be mean win rate).
You have to click twice on mean win rate to bring the expected order.

![image](https://github.com/stanford-crfm/helm/assets/73412738/11a8f14f-d00d-4dc7-9d7d-7fa2f0628525)


This is due to a high level issue in the leaderboard implementation, I will fix this in the next week or two.

Fixed by #2845
Repository: stanford-crfm/helm
Issue: Where are the leaderboards about aspirational scenarios?
Root cause: Unimplemented Feature Gap

Those options are greyed out and inaccessible:
![image](https://github.com/stanford-crfm/helm/assets/8592144/f59441b7-b7bd-4163-ba91-f2ddc878b8b7)


Duplicate of #2010.

"Aspirational" means that we would like to add scenarios that cover these categories, but we do not have any yet.
Repository: stanford-crfm/helm
Issue: Where are the supported scenarios such as WikiText-103 and NewsQA?
Root cause: Documentation Deficiency

The HELM papers claim their support but now they are found in the documentation.
![image](https://github.com/stanford-crfm/helm/assets/8592144/2b1dbd9d-8dca-4b9e-a243-e7e3de908fbc)
![image](https://github.com/stanford-crfm/helm/assets/8592144/d2af8808-f6e1-4744-80fe-683ba4a7438c)
![image](https://github.com/stanford-crfm/helm/assets/8592144/937ae0aa-dd98-4beb-ac3d-6b2b65194f8f)
![image](https://github.com/stanford-crfm/helm/assets/8592144/e8860f10-c618-4917-9a6d-fb3ff46b32b8)


These two scenarios are supported by the HELM framework; however, we do not publish any results for these two scenarios.
Repository: stanford-crfm/helm
Issue: HuggingFace warning: Setting `pad_token_id` to `eos_token_id`
Root cause: Validation Gap

When running with the example run_spec: `mmlu:subject=anatomy,model=openai/gpt2` and no caching, the HuggingFace client outputs the following warning on every call:

```
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation
```

This is annoying, but I'm not sure if it hurts anything. I think a fix could be to pass `pad_token_id` to `generate` with the value being either:

* eos_token_id specified [here](https://github.com/stanford-crfm/helm/blob/44647d7cc076dc69bfe561b375a9736f7a530218/src/helm/proxy/clients/huggingface_client.py#L86)
* If that isn't set, default it to `tokenizer.eos_token_id` as [recommended in this stackoverflow](https://stackoverflow.com/questions/69609401/suppress-huggingface-logging-warning-setting-pad-token-id-to-eos-token-id).

However, I'm not sure what impacts, if any, this could have, so I'm going to leave it to someone else to make it.

More info on [Stack Overflow](https://stackoverflow.com/questions/69609401/suppress-huggingface-logging-warning-setting-pad-token-id-to-eos-token-id). I think we could fetch `pad_token_id` from the tokenizer and pass it to the `generate()` call.
Repository: stanford-crfm/helm
Issue: Padding makes it look like prompt is cut in frontend
Root cause: Documentation Deficiency

When you zoom on a prompt there is a margin on the left but not the right which makes it look like it has been cut.

Also I think having everything on one line is not ideal as you have to scroll, an auto-return line would be nice IMO.

![image](https://github.com/stanford-crfm/helm/assets/73412738/7942746a-c816-42ea-b8f9-8be2864381d0)


Updating the style for this is in my backlog, thanks for creating an issue for it.

Also renaming to be an improvement rather than a bugfix, as everything works, this is just a UI redesign.
Repository: stanford-crfm/helm
Issue: Add documentation on how to use MKDocs
GitHub labels: documentation

p2

We should add some documentation on how to contribute documentation (i.e. how to build docs). Roughly speaking, it should look something like the below.

The docs are in `docs/`. We use MKDocs to build the documentation. Roughly speaking, the steps are, from the repository root folder:

```
pip install -r docs/requirements.txt
mkdocs build
```

But there are some gotchas involving plugins.
Repository: stanford-crfm/helm
Issue: Add documentation on how to run Lite
GitHub labels: documentation

p1

I.e. which conf file to use, parameters, etc.

Roughly speaking, it should look something like this:

```
helm-run \
--conf-paths run_specs_dec2023.conf \
--suite my-lite \
--num-train-trials 1 \
--max-eval-instances 1000 \
--priority 2 \
--skip-completed-runs \
--runner-class-name helm.benchmark.slurm_runner.SlurmRunner \
--models mistralai/mistral-7b-v0.1
```

replacing `mistralai/mistral-7b-v0.1` with your model.

The run specs file is [run_specs_dec2023.conf](https://github.com/stanford-crfm/helm/blob/fda003f328a4301476e583f25ce69c6c5420d057/src/helm/benchmark/presentation/run_specs_dec2023.conf) but I will likely rename this soon.

Fixed by #2523. Doc is [here](https://crfm-helm.readthedocs.io/en/latest/reproducing_leaderboards/).
Repository: stanford-crfm/helm
Issue: OpinionsQA has no metrics
Root cause: Unimplemented Feature Gap

@ShibaniSanturkar - the current version of OpinionsQA has no metrics in HELM. This leads to several questions

1. Is there a script that does the metric computation? If so, can it be converted to a HELM metric?
2. In #2226, @percyliang mentions that OpinionsQA requires `top_logprobs` - is this true or can it be removed?

Closing due to staleness.
Repository: stanford-crfm/helm
Issue: Do you have any plan to add SDXL or up-to-date model to HEIM?
Root cause: Unimplemented Feature Gap

Thank you for contributing and sharing to this amazing project. It is a valuable resource for much research.

I'd like to request considering the addition of SDXL or open up-to-date models support to the HEIM project.

Is there any plan to add these models?  I'd love to hear your thoughts on this matter.

Thanks in advance.

cc @teetone for HEIM

Hi @hyoje42, thanks for reaching out, we're currently working on adding more models. We should have an update soon!
Repository: stanford-crfm/helm
Issue: crfm-models server returns error if client's HELM version is too recent
Root cause: Interface Contract Mismatch

#1903 made a breaking schema change to `Request` by adding a `model_deployments` field.

For now, the solution is to downgrade your local HELM version to v0.2.4.

If installed from PyPI:

```
pip install crfm-helm==v0.2.4
```

If installed from git in editable mode:

```
git checkout v0.2.4
```

We can't simply upgrade the server to fix this because it would break the server for people on _old_ client versions, which is what most people are actually using (since most people have not complained yet). In particular, a request from an old client to a new server results in an empty `model_deployment`, which causes `AutoClient` to fail when looking up the correct client. We need to fix `AutoClient` to handle this case correctly first.

Actually we need v0.2.4. v0.3.0 will result in the following error:

```
helm.proxy.services.remote_service.RemoteServiceError: EXCEPTION: __init__() got an unexpected keyword argument 'multimodal_prompt'
```
Repository: stanford-crfm/helm
Issue: `multimodal_prompt` and `image_generation_parameters` in Request not supported by crfm-models
Root cause: Validation Gap

`multimodal_prompt` and `image_generation_parameters` are nested dataclasses, but the proxy server simply uses `Request(**json.loads(args["request"]))` [here](https://github.com/stanford-crfm/helm/blob/3a040113efa3658b9181a7793c195fdf16e715f9/src/helm/proxy/server.py#L177), which will produce nested dicts rather than nested dataclasses.

I suppose we will just not support image generation in the server...