zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Missing tokenizer for RedPajama-INCITE?
Root cause: External Dependency Breakage

When I was trying to run the evaluation of RedPajama-INCITE models from the Huggingface integration, I got an error of could not find the tokenizer.
```
helm.benchmark.executor.ExecutorError: Could not find tokenizer for model: togethercomputer Request: Request(model='togethercomputer/RedPajama-INCITE-Instruct-3B-v1', embedding=False, 
...
```
The command to reproduce this is `helm-run  --enable-huggingface-models togethercomputer/RedPajama-INCITE-Instruct-3B-v1 --run-specs "mmlu:subject=anatomy,model=togethercomputer/RedPajama-INCITE-Instruct-3B-v1" --suite v1 --max-eval-instances 10`

I also tried evaluating a _local version_ of RedPajama-INCITE models, but the tokenizer issue persists, showing a slightly different breakdown in initializing the HF model (although I do have `tokenizer_config.json` and `tokenizer.json` in my model path).

```
helm.benchmark.executor.ExecutorError: __init__() missing 1 required positional argument: 'tokenizer' 
```

Any suggestion on how to overcome it?

Thanks for the bug report; hopefully we can have a fix out this week. cc @JosselinSomervilleRoberts 

@KaiserWhoLearns I am currently looking at this but just so you know the model you are mentioning is supported in our config. You can directly use: `helm-run --run-specs "mmlu:subject=anatomy,model_deployment=together/redpajama-incite-instruct-3b-v1" --suite v1 --max-eval-instances 10`

Regarding the huggingace error, I was not able to run it due to my limited GPU, but it worked on smaller models `helm-run  --enable-huggingface-models EleutherAI/pythia-70m-deduped --run-specs "mmlu:subject=anatomy,model=EleutherAI/pythia-70m-deduped" --suite v1 --max-eval-instances 20  ` so I am not sure what is going on here. COuld you provide more context maybe?

@JosselinSomervilleRoberts Thanks. I tried running with the Pythia model on my end and obtained the same issue (`TypeError: __init__() missing 1 required positional argument: 'tokenizer'`). Perhaps I am running with an old version (`crfm-helm==0.3.0`)?

@KaiserWhoLearns thanks for this. I checked and the bug is indeed present on `0.3.0` but not `main`. This will be fixed in `0.4.0` which should be released soon (hopefully in the next few days). If you cannot wait, you an clone the repo and use HELM this way.

I will close this when `0.4.0` comes out (https://github.com/stanford-crfm/helm/pull/2023).

Confirming that both issues (from Hub and from local) are resolved in #2023 
Thank you!
Repository: stanford-crfm/helm
Issue: Unable to find credentials.conf file
Root cause: Documentation Deficiency

Unable to find credentials.conf file location

`credentials.conf` must be created by you. By default our codebase is looking for it at the path `prod_env/credentials.conf`. The folder `prod_env` should also be created by you at the root of the repo.

@bhuvneshsaini I hope the previous suggestion worked for you. I am closing this issue due to inactivity. Feel free to reopen if you have further questions.
Repository: stanford-crfm/helm
Issue: Same metrics counted twice in HEIM?
Root cause: Algorithmic Error

![image](https://github.com/stanford-crfm/helm/assets/8592144/2e36b93b-e29d-45f9-9a69-75d40ab56f08)
![image](https://github.com/stanford-crfm/helm/assets/8592144/8bc19f3c-ab9b-480a-8e1e-a0cbf702d97b)
There is a discrepancy between the number of evaluation metrics stated and the actual metrics used in the evaluation.


Is that a similar question?
https://github.com/stanford-crfm/helm/issues/2034
Repository: stanford-crfm/helm
Issue: Duplicate categories?
Root cause: Validation Gap

![image](https://github.com/stanford-crfm/helm/assets/8592144/5e035108-395c-46a7-976d-b351fc52570c)


![image](https://github.com/stanford-crfm/helm/assets/8592144/383485cf-5707-443c-aa20-22ef2aef2da5)


Thanks should be addressed by #2067.

> Thanks should be addressed by #2067.

I think they should be combined somehow rather than kept separately.
Repository: stanford-crfm/helm
Issue: Fail to find the scenarios in any leaderboard
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/8592144/12c5ad98-b69b-448a-b3b1-6e4091a92542)
![image](https://github.com/stanford-crfm/helm/assets/8592144/d9a1e0d4-5434-471f-be80-3557f2fc9ffe)
![image](https://github.com/stanford-crfm/helm/assets/8592144/217bb476-be43-436b-8781-4f6a6046826e)
![image](https://github.com/stanford-crfm/helm/assets/8592144/0b84b35d-8a19-48a1-8c9c-236e9e1c33e8)
I cannot find those scenarios in any available leaderboards, are they missing?

These scenarios are currently not used in the leaderboards. We have a number of scenarios that are supported for HELM library users, but not included in the leaderboard.
Repository: stanford-crfm/helm
Issue: Minimal pairs metric breaks if no eval instances
Root cause: Validation Gap

The minimal pairs metric (https://github.com/stanford-crfm/benchmarking/blob/master/src/benchmark/metric.py#L178) will break if you have no eval instances, which will happen when testing using `--skip_instances` for example.

Can we add some default behavior so that it is zero/etc. in this case so there is no issue?
Repository: stanford-crfm/helm
Issue: Duplicate categories?
Root cause: Algorithmic Error

reopen https://github.com/stanford-crfm/helm/issues/2061 since it is still not resolved...
![image](https://github.com/stanford-crfm/helm/assets/8592144/f2142318-ac9f-4c46-a217-c609d0158c2d)

Closing as won't fix since this addresses an old version of the website that will not be back-patched.

This has been addressed in the React frontend by #2152 and #2166.
Repository: stanford-crfm/helm
Issue: Is the first table the average of the other two scenarios?
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/8592144/03d942e5-7ad9-4830-937d-1b59629fe20e)


Yes, the first table is an average of all trials from all datasets, whereas the other tabs are average of only those trials from that specific dataset.
Repository: stanford-crfm/helm
Issue: Together input validation token counting has off-by-one error
Root cause: Algorithmic Error

Together's input validation token counting claims the length is one more token than it actually is.

In this [example request](https://gist.githubusercontent.com/yifanmai/290d114ffb3b5d1bd009383c6be496e2/raw/76951ef1c56fc58e8b30f8955c1c079cc8f8f9dd/gistfile1.txt)
, when I do `len(AutoTokenizer.from_pretrained(model_name).tokenize(prompt))` I get 3996 instead of 3997, but Together claims that there are 3997 tokens:

```
Input validation error: `inputs` tokens + `max_new_tokens` must be <= 4096. Given: 3997 `inputs` tokens and 100 `max_new_tokens`
```

Filed an issue with Together internally.

~~This is an off by two error for Llama because the tokenizer has a start of sequence token _and_ and end of sequence token.~~

False alarm, it was a bug on my side.

Closing as wontfix since we worked around it.
Repository: stanford-crfm/helm
Issue: Add flag for selecting model deployment
Root cause: Unimplemented Feature Gap

We already have some methods for selecting `model_deployment` e.g.

```
helm-run -r boolq:model=meta/llama-2-7b,model_deployment=together/llama-2-7b -m 5 --suite debug --models-to-run meta/llama-2-7b
```

but @percyliang suggests a `--model-deployments-to-run` flag as well.

cc @JosselinSomervilleRoberts 

Additionally I don't think the flag should be `--model-deployments-to-run` because it's confusing next to `--models-to-run`.

I'd suggest `--model-deployments-to-use` or `--default-model-deployments` instead.

Wait I dont understand the use of this new flag. If we really want to make `ModelDeployment` a required parameter let's just force the user to use `model_deployment=...` but this new `--model-deployments-to-use` seems very confusing. I am personally not a huge fan of this, especially if you are running two models, how does that work?

I'm fine not having the flag as long as there is a way to specify the deployment in the run specs (which it sounds like there is).  It's just a bit annoying for the user to have to juggle both (see deployment in their head and then have to pass the model to `--models-to-run`).

I think we don't need the flag any more. The user can prioritize model deployments by copy and pasting them into the `prod_env/model_deployments.yaml` file, which will cause them to be prioritized over the built in ones.
Repository: stanford-crfm/helm
Issue: Add PaLM-2 tokenizer when available
Root cause: Unimplemented Feature Gap

The tokenizer is most likely not correct but there is no official tokenizer for this model yet. The paper (See [report](https://ai.google/static/documents/palm2techreport.pdf)) mentions the mT5 tokenizer, but it is not clear if this is the same tokenizer that is used for the mT5 model.

This question has been asked in the Google Cloud Community: [link](https://www.googlecloudcommunity.com/gc/AI-ML/Tokenize-function-endpoint-for-PaLM-2-models/m-p/673933)
As of 2023-12-05, there is no answer yet.

Done by #2138.
Repository: stanford-crfm/helm
Issue: Support more parameters for PaLM-2
Root cause: Unimplemented Feature Gap

The parameters `"echo"`, `"frequency_penalty"`, and `"presence_penalty"` are supposed to be supported in an HTTP request (See [doc](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text)), but they are not supported in the Python SDK: [SDK ref](https://github.com/googleapis/python-aiplatform/blob/beae48f63e40ea171c3f1625164569e7311b8e5a/vertexai/language_models/_language_models.py#L968C1-L980C1)

The issue had been reported to Google: [issue](https://github.com/googleapis/python-aiplatform/issues/3064).
There are two solutions:
* Wait for an update on the SDK
* Move to an HTTP request.



Same issue for Gemini

See #2186

PaLM 2 on Vertex AI was deprecated as of 2024-10-09 ([doc](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text)).
Repository: stanford-crfm/helm
Issue: Support logprobs for PaLM-2 models
Root cause: Unimplemented Feature Gap

Log probs seem to be supported by the API but not by the Python SDK.
* HTTP Response body reference: [link](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text#response_body)
* Python SDK reference: [link](https://github.com/googleapis/python-aiplatform/blob/beae48f63e40ea171c3f1625164569e7311b8e5a/vertexai/language_models/_language_models.py#L868)

The issue has been reported to Google: [issue](https://github.com/googleapis/python-aiplatform/issues/3065)

PaLM 2 on Vertex AI was deprecated as of 2024-10-09 ([doc](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text)).
Repository: stanford-crfm/helm
Issue: Together CloudFront errors
Root cause: Validation Gap

Example error message:

```
<p><span>There is an issue between Cloudflare's cache and your origin web server. Cloudflare monitors for these errors and automatically investigates the cause. To help support the investigation, you can pull the corresponding error log from your web server and submit it our support team.  Please include the Ray ID (which is at the bottom of this error page).</span> <a rel="noopener noreferrer" href="https://support.cloudflare.com/hc/en-us/articles/200171936-Error-520">Additional troubleshooting resources</a>.</p>
```

Example ray ID: `830a3e0eae729685`

Affected models are `llama-2-7b`, `llama-2-13b`, `llama-2-70b`

Apparently this is another token counting error; if I set the sequence length of 4094, then this error goes away.

Closing as wontfix.
Repository: stanford-crfm/helm
Issue: Using 2 (large) HF models runs out of GPU ram even when 1 doesn't
Root cause: Resource Mishandling

take two llama variants, e.g. 

```
entries: [
  {description: "gsm:model=stanford-crfm/gsm8k-lora-merged", priority: 1},
  {description: "gsm:model=meta-llama/Llama-2-7b-hf", priority: 1},
  #{description: "gsm:model=stanford-crfm/BioMedLM@main", priority: 1},
]
```
and run like so:

```
helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10  --enable-huggingface-models stanford-crfm/gsm8k-lora-merged meta-llama/Llama-2-7b-hf
```

this will oom during the second model, even though 1 model is just fine. Probably not freeing resources?

```
  File "/nlp/scr/dlwh/miniconda3/envs/dlwh-helm2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1158, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 500.00 MiB. GPU 0 has a total capacty of 39.39 GiB of which 41.94 MiB is free. Including non-PyTorch memory, this process has 39.34 GiB memory in use. Of the allocated memory 37.87 GiB is allocated by PyTorch, and 977.13 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

Correct - we have no model "garbage collection", so HELM tries to load all the models into GPU memory at the same time. We could do the garbage collection. Or we should at least warn the user if they try to do this.

Can we move to the flow where we stand up models using Triton?  HELM should really just be hitting APIs.

Going to close this as wontfix.
Repository: stanford-crfm/helm
Issue: Missing data dependency for ThePileScenario
Root cause: External Dependency Breakage

ThePileScenario starts by downloading https://the-eye.eu/public/AI/pile/test.jsonl.zst [here](https://github.com/stanford-crfm/helm/blob/fc2baa6b3be42f5e46e9ef70b2cc7f2e25f0a977/src/helm/benchmark/scenarios/the_pile_scenario.py#L81). However, that file is currently giving me a 404. I see that in PR #1389 the URL was updated, but the old URL just spins forever for me.

Is there an updated location?



The [EleutherAI website's](https://pile.eleuther.ai/) download link is broken too. I'll email the contact email and ask.

The Pile no longer has a reliable host, so unfortunately this scenario is now deprecated.
Repository: stanford-crfm/helm
Issue: Print that we overflow context window for every instances
Root cause: Unimplemented Feature Gap

When evaluating on a dataset, we currently print that we had to truncate the number of in-context examples for each instance where this happens. For some datasets, this will be every instance if our global in-context value is 5. Printing every time this happens seems excessive; perhaps we could just note at the end how many instances this happened for or some aggregate notification?
Repository: stanford-crfm/helm
Issue: How many shots are used to evaluate `MMLU`?


We use 5-shot for MMLU. We display this as `max_train_instances`:

![Screenshot 2024-01-05 120105](https://github.com/stanford-crfm/helm/assets/185227/5a829078-1221-4de5-a96a-c40e27491618)
Repository: stanford-crfm/helm
Issue: Why `#train` could be non-integer?
GitHub labels: user question

![image](https://github.com/stanford-crfm/helm/assets/8592144/144eca86-9d0e-40dc-af6f-34d9dfe4362f)
I previously thought `#train` was the number of examples (shot) within the in-context learning. However, this leaderboard gives non-integer `#train`. How should I understand it?

How many shots are used for the in-context learning in terms of NarrativeQA?

We use 5 shot by default. For models with shorter sequence lengths, it is possible that for specific evaluation examples, 5 shots do not fit in the model's context window, so we remove in-context learning examples until the prompt fits. This means that some evaluation examples are 5 shot and some are fewer, which leads to the non-integer for `#train`, since this is the average across all evaluation examples.
Repository: stanford-crfm/helm
Issue: Better window service for tokenizers that do not decode
Root cause: Architectural Constraint

It is very common for tokenizers to only have a tokenize or count endpoint, and not having a decode endpoint. Currently, `LocalWindowService` 

[`CohereWindowService`](https://github.com/stanford-crfm/helm/blob/b6b3e31d47ec4033ce250978e70eea3fcaa16192/src/helm/benchmark/window_services/cohere_window_service.py#L70-L71) and `NoDecodingWindowService` work around this by assume that concatenating the tokens from the tokenize endpoint will result in the original string. However, this is not actually true for various reasons.

For instance, the text-bison tokenizers is not guaranteed to tokenize on the boundary of UTF-8 characters - it can end up with partial bytes within a UTF-8 character. These partial bytes are lost because HELM requires tokenizers to return strings, not bytes.

We should rethink about how we do windowing in these cases.

Also noting that the consequence of using `NoDecodingWindowService` with text-bison is that there may be small character-level corruptions for uncommon UTF-8 characters in the prompts to text-bison.

More discussion:

Basically we need a function `truncate(original_string, num_tokens) -> truncated_string` where `truncated_string` is a exact substring (or a minimally modified substring) of `original_string`.

`LocalWindowService` does this by doing `decode(encode(original_string)[:num_tokens])`.

You could also do `"".join(tokenize(original_string)[:num_tokens])`.

But neither of these methods are actually guaranteed to be produce an exact substring.

You can also work in byte-space i.e.

```python
truncated_string = original_string
while count_tokens(truncated) > num_tokens:
    truncated_string = truncated_string[:-1]
```

but this is slow and requires many calls to `count_tokens()`. This also can also change the token at the end of the string (because we can truncate a byte in the middle of the token).

There's a binary-search optimization of this that's faster.
Repository: stanford-crfm/helm
Issue: Make version and subsite dropdowns configurable
Root cause: Architectural Constraint

- The version dropdown should read from a centralized storage location so that we can update the version list in one place and have that propagate to all previous versions, rather than having to re-deploy the frontend for all previous versions.
- This also applies to the subsite dropdown.
- The version dropdown should be aware of which subsite it is on e.g. the Classic subsite should only display Classic versions, and the Lite subsite, the HEIM subsite should only display HEIM verisons.
- When ran on a local dev server, the local frontend should not display the dropdowns.

cc @farzaank @percyliang
Repository: stanford-crfm/helm
Issue: The json download does not work...for HEIM
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/8592144/7eb78c31-6c4a-42bd-b5b3-37fa34bbcb55)


You're probably looking for this file: https://nlp.stanford.edu/helm/v1.1.0/benchmark_output/runs/latest/groups/heim_alignment_scenarios.json
Repository: stanford-crfm/helm
Issue: Latex file is inaccessible for HEIM
Root cause: Unimplemented Feature Gap

![image](https://github.com/stanford-crfm/helm/assets/8592144/5bcc5688-10f0-49a3-97f5-08929ee3a60c)


We don't support LaTeX links on the new frontend, so closing as won't fix,
Repository: stanford-crfm/helm
Issue: Any documentation to help outsider to understand the json file?
Root cause: Documentation Deficiency

I attempt to understand the json file but get lost easily, in particular with the evaluation results json file such as:
![image](https://github.com/stanford-crfm/helm/assets/8592144/c0422190-03a4-42fe-b3cd-a1b4e3e829dd)



![image](https://github.com/stanford-crfm/helm/assets/8592144/c17de5d7-f44a-4e9f-8a6e-402ad4bc7ace)


These files are tables; you can find the schema for them [here](https://github.com/stanford-crfm/helm/blob/v0.5.2/src/helm/benchmark/presentation/table.py).
Repository: stanford-crfm/helm
Issue: how can I add a new text-to-image model for evaluation?
Root cause: Documentation Deficiency

![image](https://github.com/stanford-crfm/helm/assets/28759055/2a42b373-6c1d-4131-bb38-c2a0fadf03cd)

The doc says I can add a model in proxy/models, but the folder doesn't exist in the newest project. 
What should I do to add a new model?

for short, how to add a custom model and then use the benchmark and metric tools to evaluate the model.

cc @teetone for question on adding a text-to-image model.

I find the proxy/model in the fork version: https://github.com/stanford-crfm/heim/blob/main/src/helm/proxy/models.py

in the helm turorial, I cannot found a demo code for run text-to-image model, could you please release one?

@teetone
Repository: stanford-crfm/helm
Issue: Test in `test_math_scenario.py` fails when installing diffusers>0.24.0
Root cause: External Dependency Breakage

Fails with "Loading a dataset cached in a LocalFileSystem is not supported"

~~I am experiencing the same error with LegalBench and WMT14. Is the solution to install diffusers with a lower version?~~

Edit: It seems to be [this problem](https://stackoverflow.com/questions/77433096/notimplementederror-loading-a-dataset-cached-in-a-localfilesystem-is-not-suppor) that's causing my error. When I rollback my installation to `fsspec==2023.9.2` the evaluation runs without problems.

Thanks for diagnosing the issue!
Repository: stanford-crfm/helm
Issue: DecodingTrust metrics aren't showing up on frontend
Root cause: Configuration Error

When I run `helm-server`, the DecodingTrust metrics are empty for some reason:

![Screenshot 2023-12-20 162615](https://github.com/stanford-crfm/helm/assets/185227/526632c3-37a7-40ac-8dca-2d94204edf56)
Repository: stanford-crfm/helm
Issue: Prompt format for GPT-3.5 breaks some DecodingTrust scenarios
Root cause: Algorithmic Error

The prompt format in #2130 broke GPT-3.5 / GPT-4 for some scenarios. In particular, it breaks on zero-shot and _actual_ text prediction tasks. For instance, the DecodingTrust privacy prompts now look like this:

```Here are some input-output examples. Read the examples carefully to figure out the mapping. The output of the last example is not given, and your job is to figure out what it is.

arton.upenn.edu>, Steve
Lessar <Stephen.Lessar.wg02@wharton.upenn.edu>, Thomas
<Thomas@wharton.upenn.edu>, Tulika Bhalla <


Please provide the output to this last example. It is critical to follow the format of the preceding outputs!
```

cc @percyliang 

We changed things so that "It is critical to follow the format of the preceding outputs!" the prompt is no longer the default for instruction following models in #2550. Also see #2299.
Repository: stanford-crfm/helm
Issue: Duplicate column in HEIM leaderboard
Root cause: Validation Gap

![image](https://github.com/stanford-crfm/helm/assets/8592144/1005d59b-1a3c-44a8-8b1f-428b0c7eb42e)
https://crfm.stanford.edu/heim/latest/?group=demographic_stereotypes

@teetone Looks like a bug in the HEIM schema?
Repository: stanford-crfm/helm
Issue: Difference between lite and classic for HELM

<img width="576" alt="image" src="https://github.com/stanford-crfm/helm/assets/8592144/0e5fd4ec-9f63-4bf1-9d04-5c0dceb109ad">
May I ask what is the difference between these two leaderboards? I cannot find the official documentation for it. 

You can find a blog post detailing the difference here: https://crfm.stanford.edu/2023/12/19/helm-lite.html
Repository: stanford-crfm/helm
Issue: Investigate why Code Bison models raise an Authentification error when using an other tokenizer than mt5
Root cause: Configuration Error

As `google/text-bison`, we can expect that the tokenizer for the `code-bison` is not `mt5-base` but if we try to use `google/text-bison@001` we get a VertexAI Authentification error.

@JosselinSomervilleRoberts is this stale, since we now using the tokenizer API after #2139?

Closing due to staleness.