zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Are we able to feed HELM a table of LLM input/output instead of connecting to a model?
Root cause: Unimplemented Feature Gap

Hello,
I have spreadsheets of input/output from language models. Is there any relatively straightforward way to run HELM metrics on data in this format? (That is, instead of connecting HELM to a language model and grabbing the output via API.) I was not able to find any mentions of this sort of task in any of the documentation or previous issues.
Thank you.

This use case is currently not officially supported, unfortunately.

You consider try one of these approaches, which are not officially supported:

- Write a Python script that imports your spreadsheet inputs and outputs into the results cache, and then run the full HELM pipeline, which should serve results from the cache. There is an [old script](https://github.com/stanford-crfm/helm/blob/4e0d5edd93facebfce81ab99885f9251ee880d09/scripts/offline_eval/import_results.py) you can refer to that does something similar.
- Write a Python script that runs a partial HELM pipeline that constructs a list of `RequestState` and a `ScenarioState` from the inputs and outputs from your spreadsheets, and then runs the rest of the metric pipeline. You can use the [HELM pipeline code](https://github.com/stanford-crfm/helm/blob/4e0d5edd93facebfce81ab99885f9251ee880d09/src/helm/benchmark/runner.py#L227) as a reference.

I will try out these approaches. Thank you for your help!! @yifanmai
Repository: stanford-crfm/helm
Issue: The link to Image2Struct paper is not working...
GitHub labels: user question

Check https://crfm.stanford.edu/helm/image2struct/latest and click on `PAPER` button:
![image](https://github.com/stanford-crfm/helm/assets/8592144/28f09411-b75f-42cb-a8f0-10a4bf13feed)
but find:
<img width="928" alt="image" src="https://github.com/stanford-crfm/helm/assets/8592144/adeaf29f-54eb-468c-855f-610a539d4787">



The Image2Struct paper has not been published yet. I will update this link when it is ready, which should be in a few weeks.

Fixed by #3118

> Fixed by #3118

Thanks for update!
Repository: stanford-crfm/helm
Issue: UnicodeDecodeError while running benchmark
Root cause: Environment Incompatibility

Hi, Thank you for developing this amazing benchmark!

I created a virtual environment using anaconda and successfully installed crfm-helm (I followed the instructions from the official document [https://crfm-helm.readthedocs.io/en/latest/installation/](url)).

When I tried to run benchmarking with the example given in the document [https://crfm-helm.readthedocs.io/en/latest/quick_start/](url), I followed exactly the instructions given. However, when I run "helm-run --conf-paths run_entries.conf --suite v1 --max-eval-instances 10", I got this error: "UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5580: character maps to undefined". I attached some screenshots of traceback below.

Could anyone help me with this issue? 

Thank you very much!

![helm_ss1](https://github.com/stanford-crfm/helm/assets/118117017/e8472ee9-22ca-4562-9c2c-7b58641eb412)
![helm_ss2](https://github.com/stanford-crfm/helm/assets/118117017/53b5e7c4-ef0a-473c-a474-cd136ab8a550)
![helm_ss3](https://github.com/stanford-crfm/helm/assets/118117017/45a0ee6c-3ce1-4344-9532-f797b8cb8865)


Hi @carina-ding, it looks like you have a malformed `model_metadata.yaml` file. The YAML parser is complaining that it is invalid YAML. You can check this by running this:

```python
import dacite
import yaml

from helm.benchmark.model_metadata_registry import ModelMetadataList


path = "path/to/model_metadata.yaml"

with open(path, "r") as f:
    raw = yaml.safe_load(f)

model_metadata_list = dacite.from_dict(ModelMetadataList, raw)
```

Hi @yifanmai , thank you very much for your reply!

Sorry for the slight delay, as I was away last week.

I tried running the codes that you provided, it seems that it's producing the same error while running the codes.

I have attached some screenshots below, please let me know if I did anything wrong.

Thank you!

![helm_ss4](https://github.com/user-attachments/assets/b35eebb6-0291-4e6f-8d42-85df6975e0f1)
![helm_ss5](https://github.com/user-attachments/assets/b62d5d71-8984-48eb-9201-02ea79fe5659)
![helm_ss6](https://github.com/user-attachments/assets/80dbe0fb-85c0-4022-bd7c-c8d984ca56bd)


It looks like the problem is because you're running on Windows, which is not officially supported yet. We're planning to add support in the next couple of months. In the meantime, you could consider running it on Linux using WSL2.

Hi @yifanmai , thank you very much for your reply and suggestions! I will try to run it using WSL2.
Thank you!

Closing this issue due to staleness; feel free to reopen if you have further questions.

I opened a new issue to track adding Windows support. Feel free to subscribe to it for updates: #2907
Repository: stanford-crfm/helm
Issue: How can I use HELM to evaluate local model?
Root cause: Documentation Deficiency

I modified the Huggingface's model stucture, and I wanted to use HELM to evaluate my model's performance. But I met some problems. I found I could create the files model_deployments.yaml, model_metadata.yaml and tokenizer_configs.yaml, but I cannot write my local model's path.
How can I use my local model after I created the files model_deployments.yaml, model_metadata.yaml and tokenizer_configs.yaml in prod_env/ ? 
Must I upload my model to Huggingface?  
Need I create a file like '/helm/src/helm/clients/huggingface_client.py'?

Many thanks!

Hi @Zoey384 - if your Hugging Face model checkpoint is on local disk, you can set `pretrained_model_name_or_path: /path/to/your/checkpoint` inside `client_spec` in your `model_deployments.yaml` as below. You do not need to upload your model to Hugging Face Hub.

https://github.com/stanford-crfm/helm/blob/fb9eb8941603c7b9d9dc5d421d94009dce711407/src/helm/config/model_deployments.yaml#L894-L897



Closing due to staleness. Feel free to reopen if you have more questions.
Repository: stanford-crfm/helm
Issue: HELM Leaderboard Submission Guidance Inquiry
Root cause: Documentation Deficiency

Hello HELM Team,

I have been following this project and was very impressed by the comprehensive framework it provides for benchmarking language models. Our team is eager to submit our model to the HELM leaderboard website to evaluate its performance relative to other state-of-the-art models.

Despite thoroughly reviewing the GitHub repository, including the issues and docs file, we are still unclear on the exact steps required to make a submission to the leaderboard website.

Our model is compatible with OpenAI's API format, could you kindly provide some guidance on how we can proceed with our submission? 
Any additional details on the process or requirements would be greatly appreciated.

Thank you for your time and assistance.

Hi @gothiswaysir, thanks for reaching out. I'll follow up via email regarding submitting to the leaderboard.

Closing this thread because we followed up by email.
Repository: stanford-crfm/helm
Issue: Benchmark Runner Error
Root cause: Environment Incompatibility

Hi, Thank you for developing this amazing benchmark!

I created a virtual environment using anaconda and successfully installed crfm-helm (I followed the instructions from the official document [https://crfm-helm.readthedocs.io/en/latest/installation/](url).

When I tried to run benchmarking with the example given in the document [https://crfm-helm.readthedocs.io/en/latest/quick_start/](url), I followed exactly the instructions given. However, when I run "helm-run --conf-paths run_entries.conf --suite v1 --max-eval-instances 10", I got "NotADirectoryError" and "helm.benchmark.runner.RunnerError". I attached some screenshots of traceback below.

Could anyone help me with this issue?

Thank you very much!

![helm_ss7](https://github.com/user-attachments/assets/8a4bd61d-7fcc-4f17-88e5-8f2b6e56a22c)
![helm_ss8](https://github.com/user-attachments/assets/9992581a-46bc-4fd1-b98f-7d3b56094874)
![helm_ss9](https://github.com/user-attachments/assets/80601c48-b90f-469b-8fe0-dbb3a5bc1946)


Hi, it looks like you're trying to run `helm-run` on Windows, which is not currently supported. We're planning to add support in the next couple of months. Alternatively, you can run `helm-run` on Linux instead using WSL2.

Hi @yifanmai , thank you very much for your reply! I will try on Linux using WSL2.
Thank you!

Closing due to staleness; please reopen if you have further questions.
Repository: stanford-crfm/helm
Issue: Fix issue with clicking first col in leaderboards
Root cause: Algorithmic Error

doesnt take you to the right groups

This issue seems to be resolved, may have been due to an outdated front-end.

Edit: not true, this behavior happens when you use the dropdown to navigate between scenarios but not if you visit directly via the scenario page. My PR decoupling these two will make it so that this issue is more egregious.

Not resolved, but only affects certain tables.

For instance, for the LegalBench group on https://crfm.stanford.edu/helm/lite/v1.5.0/#/leaderboard

- In the first tab:
    - The first column is not clickable (incorrect behavior)
    - The second column is clickable and goes to the all LegalBench runs for the model (correct behavior)
- In the second tab:
    - The first column is clickable and goes to a single LegalBench run for the model (correct behavior)
    - The second column is clickable and goes to all the runs for the model, not just LegalBench (incorrect behavior)

Also note that the change needs to be made to `helm-summarize`, not the React frontend.

Yes that's right. I found that out shortly after closing, but opted not to re-open as there is a duplicate issue (https://github.com/stanford-crfm/helm/issues/2434) which I commented on and left open afterwards. 

I should have left a comment here as well. 

Now, I'll close the other as a duplicate since you added more details here. 

Great, thanks!
Repository: stanford-crfm/helm
Issue: Metric names for data augmentation do not render well on the website
Root cause: Documentation Deficiency

example: https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=natural_qa%3Amode%3Dclosedbook%2Cdata_augmentation%3Dall

## Diagnostics

Possible point person: @dtsip 

I have two solution proposals, one addressing the solution in the UI and the other one standardizing the perturbation field in `metrics.json`.  Consider the following three cases for the perturbation field:
```
# Case 1

"perturbation": {
  "name": "TyposPerturbation",
  "prob": 0.1
}
```
```
# Case 2

"perturbation": {
  "name": "expansion"
}
```
```
# Case 3

"perturbation": "worst"
```
## Proposals
### 1: Standardize the Perturbation Field
One solution is to standardize the perturbation field to always be an object with at least the `name` field in it.

### 2: Account for Both the String and the Object Case in the UI
The second solution is to account for both of these cases in the UI.

## Questions
1. Which approach do you prefer?
2. Consider `Case 1`. How would we want it to show up in the UI?
```
# Current Version
f1_score@1 on valid with [object Object]
```
```
# A Possible Option
f1_score@1 on valid with TyposPerturbation with prob=0.1
```


In terms of describing each perturbation, @teetone has already implemented a `perturbation.description()` method that returns [this dataclass](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/augmentations/perturbation_description.py#L5) that is meant to encapsulate all relevant information about the perturbation (each perturbation we implement augments this description to include things like typo probability). So we should  just stick with this and render the description into a string exposing all the relevant info.

The case where the perturbation is just a string is an artifact of how I implemented the worst case metric [here](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/metric.py#L79). (The idea is to assign "worst" as a perturbation description so that all perturbed versions are grouped together and we return the worst performance across them.) Not sure how we want to handle this, we could create a dummy perturbation instance, but technically this is not really a new perturbation. For now we could just support both in the frontend (e.g., `description = instance.perturbation if isinstance(instance.perturbation, str) else str(instance.perturbation.description()`).
Repository: stanford-crfm/helm
Issue: Eval instances seem to be shuffled across different models on the website
Root cause: Algorithmic Error

Compare the following:

https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=babi_qa%3Atask%3D1%2Cmodel%3Dopenai_davinci
https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=babi_qa%3Atask%3D1%2Cmodel%3Dai21_j1-jumbo
https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=babi_qa%3Atask%3D1%2Cmodel%3Dhuggingface_gpt2
Repository: stanford-crfm/helm
Issue: GPT-J is slow even with an A100. We have to look into speeding it up.
Root cause: Algorithmic Error

2.5 tokens / second

A request with 800 tokens will take ~300 seconds.

Hmm, that doesn't sound right...it's slow, but not that slow.

Initially, I thought it was slow because we were rerouting the requests to the GCP instance, but it was still even slow even if I query directly on the instance. 

@deepakn94 Do you mind taking a look at this one? We're using a single A100. What was the spec of the machine you tried, and how slow was it for you?

Hmm yeah, I don't think I tried anything that large in terms of prompt size. I was using a V100 GPU...I can run some experiments and get back to you.

For a prompt with 1200 tokens and 500 generated tokens, I am seeing a runtime of about 30 seconds on a V100 GPU.

I think this might be fixed if we move over to the GooseAI API...

> I think this might be fixed if we move over to the GooseAI API...

I think we can close this issue for now and see how it goes with GooseAI's API.
Repository: stanford-crfm/helm
Issue: Official Llama 3.1 Evals
Root cause: Unimplemented Feature Gap

Hey, 

Is anyone currently working on adding the official Meta llama 3.1 eval dataset/tasks to helm?
https://huggingface.co/datasets/meta-llama/Meta-Llama-3.1-405B-Instruct-evals

Thanks!



Thanks for the interest! We are working on the Llama 3.1 Instruct evaluations currently and will have them soon.

Llama 3.1 Instruct Turbo evaluations have been released! ([Lite](https://crfm.stanford.edu/helm/lite/latest/#/leaderboard), [MMLU](https://crfm.stanford.edu/helm/mmlu/latest/#/leaderboard))
Repository: stanford-crfm/helm
Issue: pip needs to be temporarily downgraded to 24.1.2
Root cause: External Dependency Breakage

Using pip 24.2 with `pip check` results in the error `ninja 1.11.1.1 is not supported on this platform` when running `pip check`, so we have to temporarily downgrade to pip 24.1.2. We should remove the pinned version once this is resolved.

Upstream issue: https://github.com/scikit-build/ninja-python-distributions/issues/256


This has been fixed upstream, so we should just need to bump the `ninja` version to 1.11.1.2.
Repository: stanford-crfm/helm
Issue: HEIMHumanEvalScenario requires permissions to download data from codalab
Root cause: External Dependency Breakage

Hi folks, thanks for the work here! I'm trying to use the `HEIMHumanEvalScenario` class but run into an error after `get_instances()` is called. Specifically, `https://worksheets.codalab.org/rest/bundles/0x502d646c366c4f1d8c4a2ccf163b958f/contents/blob/` returns `ERROR 403: Forbidden` when using `wget` to download data during the `HEIMHumanEvalScenario.get_instances()` call: https://github.com/stanford-crfm/helm/blob/60a58658b4648d7b027869a6b22fec1af2eaa855/src/helm/benchmark/scenarios/vision_language/heim_human_eval_scenario.py#L62

Visiting the URL manually shows a forbidden error specifying that read permissions are needed. Is this intended behavior?

@teetone could you answer regarding HEIM?
Repository: stanford-crfm/helm
Issue: Efficiency warnings
GitHub labels: p1

@deepakn94 I saw this warning a lot when running everything. I just wanted to double check to see if it's okay. Feel free to close this issue if it is.

```
WARNING: prompt with 1624 tokens is larger than the largest prompt size in inference_efficiency_dict["openai/davinci"]["runtime_for_input_tokens"]
```

Yeah, needs to be fixed. Will get to it soon...

Should be fixed (at least for the most part) in the same PR where #295 will be addressed.
Repository: stanford-crfm/helm
Issue: VHELM images do not scale when window is resized
Root cause: Algorithmic Error

Some images are giant

Can't reproduce, seems fine with small window width.
![Screenshot 2024-10-28 205429](https://github.com/user-attachments/assets/8d9e5561-0101-45b5-b945-0c24d0e35108)
Repository: stanford-crfm/helm
Issue: Add metadata field to HELM predictions
Root cause: Unimplemented Feature Gap

Replaces need for abusing references

Note: need to go back and fix all of the hacky ways we avoided this in the past: i.e. see HELM Safety scenarios+metrics for example

IMHO we should just do the right things going forwards, but we shouldn't change existing things because that risks breaking backwards compatibility.

Fixed by #3094
Repository: stanford-crfm/helm
Issue: stop sequences don't seem to work
Root cause: Interface Contract Mismatch

When querying GPT-3, stop sequences doesn't seem to be respected.

http://crfm-models.stanford.edu/static/index.html?prompt=Answer%20the%20following%20question%20about%20geography.%0A%0AQuestion%3A%20What%20is%20the%20longest%20river%3F%0AAnswer%3A%20Nile%20%23%23%0A%0AQuestion%3A%20What%20is%20the%20tallest%20mountain%3F%0AAnswer%3A&settings=temperature%3A%200%0Astop_sequences%3A%20%5B%22%23%23%22%5D%0Atop_k_per_token%3A%205&environments=

@percyliang  For stop sequence:`##` and  prompt:

```
Question: What is the longest river?
Answer: Nile##

Question: What is the tallest mountain?
Answer:
```

We get back `text:  Mount Everest` in the response (which is correct), but we also get these extraneous tokens:

```
        "tokens": [
          " Mount",
          " Everest",
          "##",
          "\n",
          "\n",
          "Question",
          ":",
          " What"
        ],
```

According to the OpenAI documentation, "the API will stop generating further tokens. The returned text will not contain the stop sequence." Weirdly, we see the stop sequence and extra tokens after in the list of tokens. I'm going to ask about this in the OpenAI discussion forum.
Repository: stanford-crfm/helm
Issue: `huggingface/gpt2` failed on multiple scenarios
Root cause: Validation Gap

commonsense_qa:dataset=piqa,method=mcqa,model=huggingface_gpt2
Traceback (most recent call last):
 …
    score_1 = max(score_func(gold.output, preds[0]) for gold in golds)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/basic_metrics.py", line 53, in exact_match
    return 1 if gold.strip() == pred.strip() else 0
AttributeError: 'NoneType' object has no attribute 'strip'
```

GPT-2 also failed on "math:subject=number_theory,level=1"



It also did not produce generations for one of the two disinformation scenarios (@bnewm0609) 

**No** predictions for **disinformation-reiteration** for **GPT-2**: https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=disinfo%3Atype%3Dreiteration%2Cmodel%3Dhuggingface_gpt2
Predictions for **disinformation-reiteration** for **GPT-3**: https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=disinfo%3Atype%3Dreiteration%2Cmodel%3Dopenai_davinci
Predictions for **disinformation-wedging** for **GPT-2**: https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=disinfo%3Atype%3Dwedging%2Cmodel%3Dhuggingface_gpt2

GPT-2 failed on "math:subject=number_theory,level=1"

the MATH dataset has its own metric, `is_equiv` [run_specs.py#L101-L103](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/run_specs.py#L101-L103)

```py
def get_math_metrics() -> List[MetricSpec]:
    metric_names = {"names": ["math_equiv"]}
    return [MetricSpec(class_name="benchmark.basic_metrics.BasicMetric", args=metric_names)]
```
defined in [math_scenario.py#L271-L291](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/math_scenario.py#L271-L291). 

Yuhuai (TonyW) let me know I should look into this issue. What was the error, do you know @teetone? Or are you handling it and I'll leave it to you?
Repository: stanford-crfm/helm
Issue: Breaking change in nltk 3.8.2
Root cause: External Dependency Breakage

Upstream issue: https://github.com/nltk/nltk/issues/3293

We can work around by pinning the version to 3.8.1.

This causes errors in the tests such as:

```
E       LookupError: 
E       **********************************************************************
E         Resource punkt_tab not found.
E         Please use the NLTK Downloader to obtain the resource:
E       
E         >>> import nltk
E         >>> nltk.download('punkt_tab')
E         
E         For more information see: https://www.nltk.org/data.html
E       
E         Attempted to load tokenizers/punkt_tab/english/
E       
E         Searched in:
E           - '/home/runner/nltk_data'
E           - '/opt/hostedtoolcache/Python/3.9.19/x64/nltk_data'
E           - '/opt/hostedtoolcache/Python/3.9.19/x64/share/nltk_data'
E           - '/opt/hostedtoolcache/Python/3.9.19/x64/lib/nltk_data'
E           - '/usr/share/nltk_data'
E           - '/usr/local/share/nltk_data'
E           - '/usr/lib/nltk_data'
E           - '/usr/local/lib/nltk_data'
E           - 'benchmark_output/perturbations/synonym'
E       **********************************************************************
```

Example stack trace:

```
src/helm/benchmark/metrics/test_bias_metrics.py:16: in check_test_cases
    bias_score = bias_func(test_case.texts)
src/helm/benchmark/metrics/bias_metrics.py:157: in evaluate_stereotypical_associations
    tokens = word_tokenize(text.lower())
/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/nltk/tokenize/__init__.py:129: in word_tokenize
    sentences = [text] if preserve_line else sent_tokenize(text, language)
/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/nltk/tokenize/__init__.py:106: in sent_tokenize
    tokenizer = PunktTokenizer(language)
/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/nltk/tokenize/punkt.py:1744: in __init__
    self.load_lang(lang)
/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/nltk/tokenize/punkt.py:1749: in load_lang
    lang_dir = find(f"tokenizers/punkt_tab/{lang}
```

Fixed by #3070.
Repository: stanford-crfm/helm
Issue: Add wrapper with consistent padding for MiniLeaderboards

We should create a wrapper component for mini leaderboards.

This will include padding and the see more button by default.

Currently there is an inconsistent way of displaying mini leaderboards in HELM front pages. Some of them have "See More" while others don't. Handling padding is similar. 

This functionality will standardize these practices and also prevent issues like the padding mismatch in https://crfm.stanford.edu/helm/safety/latest/ which happens by default when using the lite layout. We should add guardrails for this so anyone making their own leaderboard can use the lite layout properly.

This is also related to the [Lite](https://crfm.stanford.edu/helm/lite/latest/) leaderboard overall having this bug since we updated the number of models to show in the mini leaderboards.

Fixing the Lite and Classic mini leaderboard visual appearances in #3114 but not creating a new component since the additional wrapper code is only a few lines.