zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: When tags list is passed in Scenario args , it fails at ObjectSpec.__hash__()
Root cause: Validation Gap

When tags is passed as args parameter into any Scenario object. It fails to to create hash because of List object.

Possible Fix:

object_spec.py
class ObjectSpec:
......
.......
    def __hash__(self):
        # bug : Scenario object has tags as a list which is not hashable
        return hash((self.class_name, tuple((k, self.args[k]) if not isinstance(self.args[k], list) else (k, ",".join(self.args[k])) for k in sorted(self.args.keys()))))



Reference : https://github.com/stanford-crfm/helm/blob/0f1433d144400c716d82f700bdf3d539823908b2/src/helm/common/object_spec.py#L16C61-L16C61

Thanks for the bug report; feel free to open a PR for this.

I do wish Python had a way of making immutable lists for fields in frozen dataclasses...

@yifanmai Please review the PR.
Repository: stanford-crfm/helm
Issue: Extend the existing presentation framework and run_spec.conf to specify the matrix of what we want to run
Root cause: Unimplemented Feature Gap

https://docs.google.com/spreadsheets/d/1jpm0Cy0r5Yk_l9o9kwVpxbSfdKrx6ZcQ7iQfs7QYPIc/edit#gid=0

Dup of #191
Repository: stanford-crfm/helm
Issue: bert-score 0.3.11 has a non-standard dependency specifier
Root cause: Configuration Error

When running pip install, I get the following message:

```
DEPRECATION: bert-score 0.3.11 has a non-standard dependency specifier transformers>=3.0.0numpy. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of bert-score or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
```
Repository: stanford-crfm/helm
Issue: Adding new scenario to HELM
Root cause: Documentation Deficiency

Hi, HELM team!
I'm trying to add text-to-sql dataset as a scenario.
I followed https://crfm-helm.readthedocs.io/en/latest/code/#adding-new-scenarios and checked this [PR](https://github.com/stanford-crfm/helm/issues/1723).

But in the result, after helm run, I got result data in raw runs, but the Results section has got nothing. 
Could help to fix it?
What should I add/do to have integrated metrics along the scenario?
Maybe I should check smth?

Hi @somal! if you have a PR, branch or fork with your code, I could look at it and give more specific advice.

The most likely causes of the error are:

- the `RunSpec`s in `run_specs.py` do not have the right `MetricSpec`s
- the metrics / metric groups aren't correctly added to the scenario in `schema.yaml`
- `helm-summarize` was not run after `helm-run`

In my experience, you can check as followings:
+ check if the split field of the instance to be evaluated in the get_instance function of the scenari matches the main_split in the corresponding position of the schema.yaml file.

+ If a custom metric is added, please confirm if the MetricName of the metric is consistent with the main_name in the corresponding position of the schema.yaml file.

+ If a custom metric is added, please ensure that the MetricName named num_instances is added in the corresponding class of the metric .
Repository: stanford-crfm/helm
Issue: LLama 2 stop sequence tokenization is incorrect
Root cause: Validation Gap

When running local hugging face model Llama-2-7b-hf, I got this HuggingFace error in some tasks (e.g. synthetic_reasoning) but not in MMLU. 

The error information is:
HuggingFace error: Total number of tokens in each stop word should be 1.
        Request failed. Retrying (attempt #2) in 64 seconds... (See above for error details)

The error output:
<img width="1082" alt="image" src="https://github.com/stanford-crfm/helm/assets/65808395/141bdbe0-59b5-49f0-a098-17d5918d9559">



Hi, there's a couple problems here:

1. We need to switch `HuggingFaceClient` to use `StoppingCriteria` instead of `eos_token_id`, which would allow us to use stop sequences of multiple tokens - see #1501
2. The Llama 2 `AutoTokenizer` is doing something weird in that it is outputting the start of sequence special token when asked to tokenize "\n", and we need it to not do that. Possibly we're not using the API correctly, or we need to strip the tokens ourselves.

Closing due to staleness, since Llama 2 has been superseded by Llama 3.
Repository: stanford-crfm/helm
Issue: Lit-GPT model integration
Root cause: Unimplemented Feature Gap

Hello HELM team, would you be interested in accepting a PR to integrate Lightning AI's [Lit-GPT](https://github.com/Lightning-AI/lit-gpt)? If yes, I would love to contribute here.

Hi @aniketmaurya, this is something that we'd definitely like to have. HELM is also supporting the NeurIPS efficiency challenge, so it would be nice if users could run HELM on the Lit-GPT checkpoints directly.

Here's some related documentation:

- [development setup](https://crfm-helm.readthedocs.io/en/latest/developer_setup/)
- [adding new models](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/)

Would you like to take a shot at an initial PR? Otherwise I can take a look sometime next week and draft a PR.

Also, if this would be faster with sync communication, feel free to email me and set up a time for a video call. I can be reached at my email address in the git commit log.

One blocker is the Lit-GPT has a lot of configuration parameters; I'm currently working on configuration files for models over the next couple of weeks, which would solve this, but in the meantime, we might have to read configuration from environment variables or something.

hi @yifanmai, thank you for your reply! I have a initial PR [here](https://github.com/stanford-crfm/helm/pull/1792).

Yeah, I realized that for configuration I need to depend on environment variable. Would be supercool to have a configuration file. I will drop you an email to setup a time for call.

I think we should migrate to use the new LitGPT API. We could also consider using supporting LitServe.
Repository: stanford-crfm/helm
Issue:  HuggingFace error: The size of tensor a must match the size of tensor b
Root cause: Configuration Error

Thanks for this library.

I'm exploring the package with a small base model (EleutherAI/Pythia-70m). For a couple of tasks, I see:

```
Setting `pad_token_id` to `eos_token_id`:187 for open-end generation.
        HuggingFace error: The size of tensor a (2048) must match the size of tensor b (5845) at non-singleton dimension 3
        Request failed. Retrying (attempt #4) in 30 seconds... (See above for error details)
```

This happens on the following runs:

helm.benchmark.runner.RunnerError: Failed runs: ["natural_qa:mode=openbook_longans,model=huggingface_model,data_augmentation=canonical", "imdb:model=huggingface_model,data_augmentation=canonical", "raft:subset=one_stop_english,model=huggingface_model,data_augmentation=canonical", "raft:subset=tai_safety_research,model=huggingface_model,data_augmentation=canonical"]


I'm wondering how I should deal with this using your framework. Or is this an unsupported model (or unsupported scenario for the model) given that its max. output length is 2K tokens whereas the scenario expects a larger output length?

Thanks in advance!

This is because HELM tries to auto-infer the model length based on the Hugging Face Hub `AutoTokenizer`, and in the case of `EleutherAI/Pythia-70m`, the `model_max_length` is set incorrectly:

```python
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/Pythia-70m")
print(tokenizer.model_max_length)
# Result: 1000000000000000019884624838656
```

Ideally this should be fixed upstream by the EleutherAI folks.

As a workaround, HELM should allow manually overriding this value via a configuration file, especially since `model_max_length` is frequently set incorrectly. I'm planning to add something like this in the next few of weeks.

Great. Thanks for your reply! Looking forward to the update.

We now support configuring the `max_sequence_length` in `model_deployments.yaml`. Here is [an example](https://github.com/stanford-crfm/helm/blob/v0.5.2/src/helm/config/model_deployments.yaml#L766-L771). This allows using Hugging Face models that do not have `model_max_length` set correctly. Additionally, HELM now raises an error if you are trying to use `--enable-huggingface-models` with a model that does not specify `model_max_length` (#2179).
Repository: stanford-crfm/helm
Issue: gsm8k Request error
Root cause: Configuration Error

When I use the following conf it is normal entries: [{description: "mmlu:model=neurips/local,subject=college_computer_science", priority: 4}] But when I use entries: [{description: "gsm:model=neurips /local", priority: 4}], there will be the following prompt
![image](https://github.com/stanford-crfm/helm/assets/43968255/016e29e9-0b6b-47a1-a63a-b294b95d3fd6)


Hi, `neurips/local` requires you to have a currently running HTTP server at `localhost:8080` that serves your model, and that error message says that HELM was not able to find that server. Did you have the server running at that time?

@yifanmai hello! of course, my service has been started, when I change to entries: [{description: "mmlu:model=neurips/local,subject=college_computer_science", priority: 4}] it will be normal

I see, this is probably because your model is unable to complete processing the entire request before the HTTP server times out. This happens on GSM8K but not MMLU because the requests for GSM8K are longer and therefore take a longer time to process. If you're able to adjust the HTTP server timeout, you should try adjusting that to a few minutes.

@yifanmai LGTM! How should I adjust it, which part of the code needs to be changed.

On the HELM client side, I opened #1807 to increase the timeout.

On the HTTP server side, how to adjust the timeout would depend on which server you're using. For instance, for Gunicorn, the configuration is found [here](https://docs.gunicorn.org/en/stable/settings.html#timeout).

@MrJungle1 I increased the client timeout. Could you try again (using HELM at the head of main)? This means `git pull` from main if you have a checked out repo, or `pip install crfm-helm@git+https://github.com/stanford-crfm/helm.git@main` if you're installing via `pip install`.

@yifanmai Yes, I also located here, and manually changed it to 100 in my current branch, and now it works normally, thanks.

@yifanmai Hello, but I encountered a new problem on the main branch. When I was in the scene of summarization_cnndm, I reported an error ModuleNotFoundError: No module named 'summ_eval', I followed his prompt pip install and added [all] but he replaced my original torch version Dropped back to 1.13.1 from 2.0.1, it works fine, but I need torch2.0.1
![image](https://github.com/stanford-crfm/helm/assets/43968255/0067448b-0016-47a3-996e-f6cb63eb4f6d)


Hi, you'll need to manually install the desired version of PyTorch if you need a different version. You can run:

```
pip install torch==2.0.1
```

and then repeat for each other torch dependency that you need a specific version of.

Alternatively, you can `pip uninstall torch torchvision` and then follow the [official instructions on the PyTorch website](https://pytorch.org/get-started/locally/) to get the current latest version of PyTorch.
Repository: stanford-crfm/helm
Issue: Together Vicuna and Alpaca does not strip 
 EOS token
Root cause: External Dependency Breakage

Affected models on Together: `lmsys/vicuna-7b-v1.3`, `lmsys/vicuna-13b-v1.3`, `togethercomputer/alpaca-7b`

Together does not strip the `
` EOS token from the end of the generated text. This results in incorrect results when computing exact match metrics.

Example request: https://api.together.xyz/jobs/job/2b678c6280de6971251aac77ad3231734a9d6a86975b9299fab2d620996


This still seems to be happening, and is affecting Alpaca as well.

This is still a known bug, but stop sequences was also not been respected for some other models and we just did postprocessing on our side in general to handle this - does that not catch it?

The current postprocessing does not catch it. Together suggested adding the EOS token as a stop sequence, which seems to work as a workaround.

Closing this as we have a workaround, though the bug still exists upstream in Together.
Repository: stanford-crfm/helm
Issue: Together Alpaca uses weight diffs without applying them to base weights
Root cause: Algorithmic Error

Affected models on Together: `tatsu-lab/alpaca-7b-wdiff`

Together uses the weight diffs without applying them to the base weights. This results in garbage generations.

![together-alpaca-7b-wdiff](https://github.com/stanford-crfm/helm/assets/185227/a458ac56-634a-4764-83a1-a43755cad5a8)


Seems fixed now on Together.
Repository: stanford-crfm/helm
Issue: Together MPT HalfTensor type error
Root cause: Interface Contract Mismatch

Affected models: `togethercomputer/mpt-30b-instruct`, `togethercomputer/mpt-30b`

Error `Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.HalfTensor instead`

https://api.together.xyz/jobs/job/83af28ef89f447290af38b4cac2eaf4252fe3a292d8f8adf2eff2fc73255cf

Together seems to not be returning this error any more.
Repository: stanford-crfm/helm
Issue: Together Falcon list index out of range
Root cause: Validation Gap

Affected models: `togethercomputer/falcon-40b-instruct`, `togethercomputer/falcon-40b`, `togethercomputer/falcon-7b-instruct`, `togethercomputer/falcon-7b`

When attempting to use Falcon on Together, it produces the error `list index out of range`

Request: https://api.together.xyz/jobs/job/7c4e3432ab9887a6cac8c8d91baa2ac4e0a8a03f061a57982d285560376db101

Falcon seems to not be returning this error any more.
Repository: stanford-crfm/helm
Issue: Where can I find the prompt for each evaluation scenario?
Root cause: Documentation Deficiency

Could you tell me where I can find the prompt templates for each scenario?

I search over various scenario files  in [here](https://github.com/stanford-crfm/helm/tree/main/src/helm/benchmark/scenarios), but can't find any prompts.


Prompts are downloaded and processed from 3rd party source. E.g. for MMLU...

Download the raw data:

https://github.com/stanford-crfm/helm/blob/4810bd56e752b952bf1e85fb2963504e40f9e527/src/helm/benchmark/scenarios/mmlu_scenario.py#L66-L72

Processing the raw data into prompts:

https://github.com/stanford-crfm/helm/blob/4810bd56e752b952bf1e85fb2963504e40f9e527/src/helm/benchmark/scenarios/mmlu_scenario.py#L74-L94

Thanks for the explanation @unnawut !

To give more info on prompt templates aka "adapters":

Each scenario has one or more possible `AdapterSpec`, which defines the prompt template. You can look for the corresponding `AdapterSpec` for the scenario by going to `run_specs.py` e.g. [here](https://github.com/stanford-crfm/helm/blob/4810bd56e752b952bf1e85fb2963504e40f9e527/src/helm/benchmark/run_specs.py#L1117) is the `AdapterSpec` for `boolq`.

You can also get the `AdapterSpec` by going to go to [Raw Runs](https://crfm.stanford.edu/helm/latest/?runs=1), clicking a relevant run, and then looking at "Adapter specification" at the top of the page.

You can look at the adapter implementation to see how the `AdapterSpec` produces the prompt template e.g. [`MultipleChoiceJointAdapter`](https://github.com/stanford-crfm/helm/blob/4810bd56e752b952bf1e85fb2963504e40f9e527/src/helm/benchmark/adaptation/adapters/multiple_choice_joint_adapter.py#L9)

To see the raw prompt text we used, go to [Raw Runs](https://crfm.stanford.edu/helm/latest/?runs=1) on the results page, click a relevant run, and then click "Prediction" - note that doing this will download a large amount of data (potentially >100 MB), so do not do this on a low-bandwidth connection.
Repository: stanford-crfm/helm
Issue: Falcon model returns a multiple-choice letter instead of the full answer. How to correctly calculate Exact Match and other metrics
Root cause: Validation Gap

Falcon model returns this:
Instance id12 [split: valid]
Input:
Emissary veins connect the intracranial venous sinuses to
References:
- veins draining the scalp.
- veins draining the eye.
- the pterygoid venous plexus.
- All of the above areas. [correct]
**[ Exact match: 0 |** # train: 5 | truncated: 0 | # prompt tokens: 0 | # output tokens: 3619 | # trials: 1 ]
Prediction{trial 0}: **D**

While the answer is correct (D), Exact match is '0'.

How do I adjust Helm config to recognize "D" as the correct answer?


Some notes:

- The expected output from the model is one of A, B, C or D.
- The frontend is confusing - if the prediction is exactly one of A, B, C or D, then the Prediction value in the web frontend will be updated to the text of the corresponding reference (i.e. in this case it will _not_ show the raw text of the output).
- In this case, there is likely extra text in the output that prevents it from being an exact match to A, B, C or D - usually the end of sequence string `
`, which is not rendered in the frontend.
- You might need to modify the `Client` to strip end of sequence token, though ideally this should be done upstream by the model itself.
- You can check this by going to `scenario_state.json` and checking the `result` field. Here's an example of a bad `result` for boolq, where the expected outputs are Yes and No:

```

      "result": {
        "success": true,
        "embedding": [],
        "completions": [
          {
            "text": "Yes
",
            "logprob": -0.666473388671875,
            "tokens": [
              {
                "text": " Yes",
                "logprob": -0.011688232421875,
                "top_logprobs": {
                  "\u2581Yes": -0.011688232421875
                }
              },
              {
                "text": "
",
                "logprob": -0.65478515625,
                "top_logprobs": {
                  "
": -0.65478515625
                }
              }
            ]
          }
        ],
        "cached": false,
        "request_time": 5.627094745635986
      },
```
Repository: stanford-crfm/helm
Issue: Together Falcon and MPT have incorrect sequence length
Root cause: Configuration Error

Both Falcon and MPT have a default sequence length of 2048. However, Together does not support requests with more than 1024 tokens.

```
helm.proxy.clients.together_client.TogetherClientError: Together request (job_id=c6b3f185e4f83a0d26801a5618fe0e171086b5e25c6fbe5ef525cbffc131c2dd) failed with error: Input validation error: `inputs` must have less than 1024 tokens. Given: 1244
```

This might be fixed; will try again.

This is working now.
Repository: stanford-crfm/helm
Issue: Draft : Optimizing Project Environments with Docker
Root cause: Documentation Deficiency

Wave off the difficulties of using virtualenv and Anaconda – switch to Docker for smoother and more controlled environment management.

https://github.com/stanford-crfm/helm/pull/1825

https://github.com/stanford-crfm/helm/pull/1825

I think providing a Docker image is a good idea.

By the way, I would also suggest trying [pipx](https://pypa.github.io/pipx/) as an alterative way to avoid environment management.

Closing this - I would generally recommend using `pipx` for users who don't want to deal with environments. We should probably add that to the documentation.
Repository: stanford-crfm/helm
Issue: Running llama model
Root cause: Documentation Deficiency

Is it possible to test the llama model with the crfm-helm=0.2.3 version, and if so, would it be convenient to provide your run configuration for running the llama model?

Hi, v0.2.3 doesn't work with LLaMA yet. Normally I would suggest using the Hugging Face integration, but for some reason, the LLaMA and Llama 2 tokenizers tokenize the newline character `\n` as two tokens (because a special token is being prepended) which causes an error as in #1782. I'm hoping to look into this issue soon.

v0.2.4, which will be released soon, will support LLaMA on [Together](https://api.together.xyz/) (an inference platform) as an alternative to the Hugging Face version.

> Hi, v0.2.3 doesn't work with LLaMA yet. Normally I would suggest using the Hugging Face integration, but for some reason, the LLaMA and Llama 2 tokenizers tokenize the newline character as two tokens (because a special token is being prepended) which causes an error as in #1782. I'm hoping to look into this issue soon.`\n`
> 
> v0.2.4, which will be released soon, will support LLaMA on [Together](https://api.together.xyz/) (an inference platform) as an alternative to the Hugging Face version.

Bug report：Thank you very much for your response. However, I'm currently running the demonstration program "mmlu:subject=philosophy,model=huggingface/llama-7b". I encountered a different issue from #1782 and would appreciate your assistance with it!
![image](https://github.com/stanford-crfm/helm/assets/71212131/316a49ef-6989-4980-a103-63013da148e9)


`huggingface/llama-7b` requires you to have a local copy of LLaMA located at the local path `./huggingface_models/llama-7b` that has been converted to the Hugging Face format using [these instructions](https://huggingface.co/docs/transformers/main/model_doc/llama). The error message says that it could not find the model at that path.

Alternatively, if you don't have a local copy of LLaMA, or you don't want the hassle of running the conversion script, you can try using [huggyllama](https://huggingface.co/huggyllama/llama-7b) on Hugging Face Model Hub. Here are the [instructions for the HELM integration with Hugging Face Model Hub]

Example: (run 10 instances of `boolq`):

```bash
helm-run \
    --run-specs boolq:model=huggyllama/llama-7b \
    --enable-huggingface-models huggyllama/llama-7b \
    --suite v1 \
    --max-eval-instances 10
```

Thank you for the solution, I am also facing the same issue.

For future reference: Inference using LLaMA is challenging; I would suggest using a version of Llama from Llama 2 onwards, which are supported by `HuggingFaceClient`.
Repository: stanford-crfm/helm
Issue: Together not returning logprobs for `llama-65b` and `llama-2-70b`
Root cause: Interface Contract Mismatch

Together suggested setting `details: true` as a workaround, but even with this flag, Together only returns logprobs with the value of 0.

It turns out that Together is returning the logprobs, but the response JSON schema is changed for some reason, which causes HELM to not be able to parse the response.

`togethercomputer/llama-2-7b` (i.e. what HELM expects):
```
  response: {
    result_type: 'language-model-inference',
    choices: [
      {
        finish_reason: 'length',
        index: 0,
        logprobs: {
          token_logprobs: [
            -0.6357421875,
            -0.0012521743774414062,
            -0.03607177734375,
            -0.0880126953125,
            -0.013671875
          ],
          tokens: [ '▁w', 'olf', '<0x0A>', '<0x0A>', 'Pass' ],
          top_logprobs: [
            { '▁w': -0.6357421875 },
            { olf: -0.0012521743774414062 },
            { '<0x0A>': -0.03607177734375 },
            { '<0x0A>': -0.0880126953125 },
            { Pass: -0.013671875 }
          ]
        },
        text: 'wolf'
      }
    ],
    request_time: 0.7430341243743896,
    request_datetime: 1689955682
  }
```

`togethercomputer/llama-2-70b` (i.e. the changed JSON schema):
```
  response: {
    choices: [
      {
        finish_reason: 'stop_sequence',
        text: 'south east\n',
        token_logprobs: [ -1.0234375, -0.47021484, -0.09625244 ],
        tokens: [ ' south', ' east', '\n' ]
      }
    ],
    request_time: 5.549241542816162,
    request_datetime: 1694554450
  }
```

In general, we're not able to compute calibration for most models, since many model providers are no longer giving us logprobs through their APIs.