zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Support few-shot chain-of-thought in GPQA / MMLU
Root cause: Unimplemented Feature Gap

Related: #3017 and #3018

Both GPQA and MMLU have a similar way of doing few-shot chain-of-thought so we should reuse the common infrastructure. We should coordinate to not duplicate work.

### Scenario
1. When constructing instances, set `instance.extra_data["chain_of_thought"]` to the chain of thought in the dataset instance.

### Run spec function
1. Update the run spec function to take in an boolean parameter `use_chain_of_thought`
2. Update the function to use the new adapter and metric if and only if `use_chain_of_thought` is true.

### Adapter
1. Create a subclass of `MultipleChoiceJointAdapter` that includes the chain of thought in the output.
2. Add a new enum value (similar to `ADAPT_MULTIPLE_CHOICE_JOINT`) for that subclass and add the enum value and the subclass to `AdapterFactory`.
4. Set the `AdapterSpec` to use this enum value when constructing the run spec for in the run spec functions.

### Metric
1. Create a new subclass of `Metric` in a new file `chain_of_thought_metrics.py`
2. Override `evaluate_generation()` to parse the model generated output to look for something like `(A)` and output a `Stat` named `chain_of_thought_exact_match`
Repository: stanford-crfm/helm
Issue: Is there any way to load GGUF models?
Root cause: Unimplemented Feature Gap

Title says everything :)

HI @InAnYan, we don't have official support for GGUF and we don't support running local in-process inference on GGUF models, but you can try one of these ways:

- vLLM has [limited GGUF support](https://docs.vllm.ai/en/v0.6.2/quantization/gguf.html). You can try running your GGUF model on vLLM and then use the [HELM vLLM integration](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/#vllm).
- [LLaMA.cpp HTTP Server](https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md) and [Ollama server](https://github.com/ollama/ollama/blob/main/docs/openai.md) both have partial OpenAI API compatibility, so it may be possible to use it with HELM's `OpenAIClient` by setting `base_url` to your server's URL ([example](https://github.com/stanford-crfm/helm/blob/0b5ead0cf20f7aa4d05e313814de874bd990ec3c/src/helm/config/model_deployments.yaml#L2525-L2528))

I've modified the llama-server code on my [llama.cpp fork](https://github.com/Nero7991/llama.cpp/tree/oai_compat) to add more OpenAI API support so the JSON response is compatible with the OpenAI client used by helm-run. I was able to run benchmarks on the Qwen's QwQ - qwq-32b-preview-q4_k_m.gguf 32B 4 bit quantized GGUF model after this. Created a pull request [here](https://github.com/ggerganov/llama.cpp/pull/10627)

To use the modified llama.cpp server with HELM, I used the following model configurations by creating a folder with model_deployments.yaml and model_metadata.yaml files.

In the `model_deployments.yaml ` file:
```
model_deployments:
  - name: vllm/qwq-32b-preview-q4_k_m
    model_name: vllm/qwq-32b-preview-q4_k_m
    tokenizer_name: qwen/qwen1.5-7b
    max_sequence_length: 32767
    client_spec:
      class_name: "helm.clients.vllm_client.VLLMClient"
      args:
        base_url: http://127.0.0.1:8080/v1/
```

Important to point correctly to the `llama-server` URL. 
```
./llama-server -m models/qwq-32b-preview-q4_k_m.gguf -ngl 65 --port 8080
```

`model_metadata.yaml` file:
```
models: 
  - name: vllm/qwq-32b-preview-q4_k_m
    display_name: QwQ Preview 4 bit GGUF (32B)
    description: QwQ Preview 4 bit GGUF (32B), proposed by Aibaba Cloud. Qwen is a family of transformer models with SwiGLU activation, RoPE, and multi-head attention. The 32B version also includes grouped query attention (GQA). ([blog](https://qwenlm.github.io/blog/qwen1.5-32b/))
    creator_organization_name: Qwen
    access: open
    release_date: 2024-04-02
    tags: [TEXT_MODEL_TAG, LIMITED_FUNCTIONALITY_TEXT_MODEL_TAG, INSTRUCTION_FOLLOWING_MODEL_TAG]
```

Then, need to install OpenAI HELM client:
```
pip install crfm-helm[openai]
```

I started the benchmark using:
```
helm-run --run-entries mmlu:subject=philosophy,model=vllm/qwq-32b-preview-q4_k_m --suite my-suite --max-eval-instances 10 --local-path ~/GitHub/helm-config/
```


Thanks @Nero7991! This looks great. I will follow the pull request to llama.cpp.
Repository: stanford-crfm/helm
Issue: Double check everywhere we fix the random seed
Root cause: Validation Gap

Ensure results are reproducible

## Proposal

We can have every scenario add the following line to their `__init__()` method.

```
        self.random: random.Random = random.Random(SEED)
```

## Scenario Checklist

Scenarios where the randomness is checked to be self-contained.

- [x] `The Pile`
- [x] `ICE`
- [x] `TwitterAAE`
- [x] `WikiText-103`
- [x] `NaturalQuestions`
- [x] `HellaSwag`
- [x] `OpenBookQA`
- [x] `MMLU`
- [x] `BoolQ`
- [x] `NewsQA`
- [x] `NarrativeQA`
- [X] `MS MARCO Passage Ranking`
- [ ] `DROP`
- [x] `QuAC`
- [x] `Summarization, XSUM`
- [x] `Summarization, DailyMail`
- [x] `RAFT`
- [x] `IMDB`
- [x] `BLIMP`
- [ ] `CoLA`
- [x] `WikidataFact`
- [x] `GSM8K`
- [x] `HumanEval`
- [x] `APPS`
- [x] `bAbI`
- [x] `LSAT`
- [x] `MATH`
- [x] `pattern induction`
- [x] `synthetic matching`
- [x] `synthetic substitution`
- [x] `synthetic fact matching`
- [x] `synthetic fact deduction`
- [x] `Dyck-n language`
- [x] `number relationship induction`
- [x] `RealToxicityPrompts`
- [x] `BOLD`
- [x] `CivilComments`
- [x] `Copyright`
- [x] `BBQ`
- [x] `TruthfulQA`
- [x] `Disinformation-Reiteration`
- [x] `Disinformation-Wedging`
- [x] `Dialogue` (subset of Common Sense Dialogues, Empathetic Dialogues, and Persona Chat)



I went over all the scenarios listed above as well as all the files matching `src/benchmark/*_scenario*`.

There were a few things that were not robust so I submitted https://github.com/stanford-crfm/benchmarking/pull/351 to fix them. The issue was that randomness was set in `__init__()` but used in `get_instances()` and we have no control over what happens in between. Still, this has not been an issue since in our current codebase (as of https://github.com/stanford-crfm/benchmarking/commit/8b94ab1e59310bc80c38f5852f66ba35c42d9a86) nothing happens between these (see https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/runner.py#L67).

Feel free to close when 351 merges.
Repository: stanford-crfm/helm
Issue: Installation Guide: Required Dependencies for Supported Operating Systems and Distributions
Root cause: Documentation Deficiency

The installation guide currently lacks detailed information on the dependencies required for various supported operating systems and distributions. To help users set up the project smoothly, we should add a section in `docs/installation.md` that outlines the dependencies for each supported OS and distro (e.g., Ubuntu, Debian, macOS, etc.). This will ensure that users have a clear understanding of the prerequisites and can avoid common setup issues.

Are you running into an issue with a specific distro? I believe HELM should work on Ubuntu and MacOS; Debian hasn't been tested but I don't see why it wouldn't work.

There is a known issue that HELM doesn't work on Windows; see #2907.

Oh, sorry it's for Developer setup, so it should be `docs/developer_setup.md`

For Ubuntu, it's quite trivial such as `build-essential` and `unzip`. 
Even though they are trivial, some open-source projects include `git` in their setup dependencies if a distro needs it.

This issue should be a very low priority I think.

On the other hand, do you think that the setup of `cuda` should be mentioned in both `docs/installation.md` and `docs/developer_setup.md`?



That's a good point. Do you have an example of documentation from another open source project that has done something similar?

Yes, CUDA setup is quite complicated, and it's worth discussing this in the documentation.

Sorry, I should have given some examples. Here are three.
- [Electron](https://www.electronjs.org/docs/latest/development/build-instructions-linux). There are `build-essential curl` for Ubuntu. They also have different setup prerequisites for other distros.
- [Android](https://source.android.com/docs/setup/start/requirements). There are `build-essential zip unzip curl` for Ubuntu
- [OpenWrt](https://openwrt.org/docs/guide-developer/toolchain/install-buildsystem).  There are `build-essential git unzip wget` for Debian/Ubuntu/Mint. They also have different setup prerequisites for other distros.

Thanks for your time!

Thanks @shakatoday! I will take a look.
Repository: stanford-crfm/helm
Issue: Reproducing WMT14
Root cause: Documentation Deficiency

I want to reproduce and then adjust the data of the WMT2014 Benchmark. Therefore I cant use helm directly (if I understand it correctly). Therefore i want to use the dataset and the way how the prompt is constructed in my own pipeline. (same applies to the evaluation code)

Unfortunately I cant understand and find my way through the code. 

I believe the PromptRunExpander is responsible for creating the prompt but i cant figure out what it exactly does and what values are used and where they are obtained from.

I would happy for any explenation on how the flow of information is and how the final prompt is constructed

Hi @MaxHahnbueck, here's some pointers to the code:

The main flow for the prompt generation happens in `Runner.run_one` [here](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/runner.py#L235-L288). Specifically, this calls `WMT14Scenario.get_instances` ([link](https://github.com/stanford-crfm/helm/blob/ff9c7c9a7dd5af8a62d6de2b1a36c70c8f60b690/src/helm/benchmark/scenarios/wmt_14_scenario.py)) to download and load instances into memory, and it calls `GenerationAdapter.generate_requests` ([link](https://github.com/stanford-crfm/helm/blob/ff9c7c9a7dd5af8a62d6de2b1a36c70c8f60b690/src/helm/benchmark/adaptation/adapters/generation_adapter.py#L34-L62)) to turn them into prompt strings, which are placed in `RequestState.request.input.text`.

You may also want to check out the [documentation](https://crfm-helm.readthedocs.io/) if you haven't already. Hope this helps.

Thanks. I think i figured out most stuff now. 

If I understand correctly annotators in scenario_state [here](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/runner.py#L287-L294) describe how postprocessing is done. WMT14 (and others such as med_qa) do not have these annotators.

But if I want to use BLEU-4 as the metric I believe I need some kind of preprocessing as my output looks like this:

```
It seems like there are two sentences to translate. Here are the translations:

1` German: Wiederaufnahme der Sitzungsperiode
   English: Resumption of the session

2. German: Sie stehen keine 100 Meter voneinander entfernt: Am Dienstag ist in Gutach die neue B 33-Fußgängerampel am Dorfparkplatz in Betrieb genommen worden - in Sichtweite der älteren Rathausampel.
   English: They are not 100 meters apart from each other: On Dienstag, in Gutach, the new B 33 pedestrian traffic light at Dorfparkplatz has been put into operation - in sight of the older town hall traffic light.
```
Maybe thats just not clean output by the model, but I think I just need the last part after the "english:"

Where can I find the postprecessing steps?
Repository: stanford-crfm/helm
Issue: VHELM setup
Root cause: Documentation Deficiency

I've followed all the steps for VHELM setup and got the error:

(crfm-helm) smsarov@MacBook-Air-Alexander diploma % helm-run --run-entries mmmu:subject=Accounting,model=openai/gpt-4o-mini-2024-07-18 --suite my-vhelm-suite --max-eval-instances 10

main {
  Reading tokenizer configs from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/config/tokenizer_configs.yaml...
  Reading model deployments from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/config/model_deployments.yaml...
} [0.717s]
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/helm-run", line 8, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/benchmark/run.py", line 332, in main
    run_specs = run_entries_to_run_specs(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/benchmark/run.py", line 43, in run_entries_to_run_specs
    for run_spec in construct_run_specs(parse_object_spec(entry.description)):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/benchmark/run_spec_factory.py", line 64, in construct_run_specs
    run_specs: List[RunSpec] = [run_spec_function(**args)]
TypeError: get_mmmu_spec() missing 1 required positional argument: 'question_type'

I tried running quick start for usual helm and it worked as expected



Thanks for your interest in using the framework! There are two question types in MMMU: `open` and `multiple-choice`, you have to pass `question_type=multiple-choice` when testing on the task. 
You can refer to the `run_entries_vhelm.conf` for other args in vhelm tasks.
https://github.com/stanford-crfm/helm/blob/ee10b8fd5a0f46949c98cd7a0a79cb7e1b163073/src/helm/benchmark/presentation/run_entries_vhelm.conf#L82

I'm also getting this issue when I run the quick start for vhelm: https://github.com/stanford-crfm/helm/blob/main/docs/vhelm.md

By updating the run instruction to:

```
helm-run --run-entries mmmu:subject=Accounting,model=openai/gpt-4o-mini-2024-07-18,question_type=multiple-choice --suite my-vhelm-suite --max-eval-instances 10
```

I was able to get it to move past that error (a better error message would be helpful), but then I run into an API error because I don't have a `OPENAI_API_KEY`. That error is more straight forward, but it would be good if the doc mentioned that it was necessary or instead defaulted to a small model that is feasible to run locally.
Repository: stanford-crfm/helm
Issue: Issue with Model Execution on the HumanEval Benchmark
Root cause: Configuration Error

I’m experiencing an issue while running some models on the HumanEval benchmark. Most of the predictions generated by the model are empty, with no results at all. However, when testing the same model on Colab, it successfully predicted for a specific input that wasn’t working on HELM. Additionally, other models are running successfully on HELM, generating correct predictions for the given inputs. I’ve tried adjusting several parameters, but I still couldn’t figure out the cause. Does anyone know why this might be happening?

Hi @lucas-s-p, does this issue also show up when you try to evaluate the same models on other scenarios (e.g. `mmlu`), or does this only happen for HumanEval?

Hi @yifanmai! I’ve tested the model on another scenario in HELM, the APPS benchmark, and it works correctly, generating proper predictions. The issue with empty predictions occurs on the HumanEval benchmark. I’m using the scenario added in this PR: [add scenarios HumanEval and APPS](https://github.com/stanford-crfm/helm/pull/142)

Unfortunately most of the authors of the HumanEval integration in HELM are no longer active in this repository, so there isn't much in the way of maintenance or support for this scenario...

cc @lxuechen @q-hwang @teetone in case one of you can help.
Repository: stanford-crfm/helm
Issue: Switch AnthropicTokenizer to use client.beta.messages.count_tokens()
Root cause: External Dependency Breakage

When `anthropic` is upgraded to 0.39.0 or newer, the following error happens:

```
src/helm/tokenizers/anthropic_tokenizer.py:26: error: "Anthropic" has no attribute "get_tokenizer"  [attr-defined]
```

This is because of [this breaking change](https://github.com/anthropics/anthropic-sdk-python/blob/93cbbbde964e244f02bf1bd2b579c5fabce4e267/CHANGELOG.md?plain=1#L97-L100):

```
### ⚠ BREAKING CHANGES

* **client:** remove legacy `client.count_tokens()` & `client.get_tokenizer()` methods ([#726](https://github.com/anthropics/anthropic-sdk-python/issues/726))
  * This functionality has been replaced by the `client.beta.messages.count_tokens()` API which supports newer models and all content functionality, such as images and PDFs.
```

The temporary workaround is to limit `anthropic` to versions before 0.39.0. The long term change is to limit `anthropic` to after 0.39.0 and switch `AnthropicTokenizer` to use `client.beta.messages.count_tokens()`.

Looks like this is infeasible because the new beta API does not support encoding and decoding. The much longer solution is to change the tokenizer framework allow tokenizers to only support length counting.

`# error log :
File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/helm/tokenizers/anthropic_tokenizer.py", line 26, in _init_
tokenizer_object=anthropic.Anthropic().get_tokenizer()
File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/anthropic/client.py", line 122, in __init_
super()._init_(
File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/anthropic/base_client.py", line 856, in __init_
self._client = http_client or SyncHttpxClientWrapper(
File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/anthropic/base_client.py", line 754, in __init_
super()._init_(**kwargs)
TypeError: _init_() got an unexpected keyword argument 'proxies'`

I got the above error and upgraded to anthropic 0.40.0.
But then I got the issue on this page. 
`# error log:
  File "/home/anaconda3/envs/crfm-helm/lib/python3.9/site-packages/helm/tokenizers/anthropic_tokenizer.py", line 26, in __init__
    tokenizer_object=anthropic.Anthropic().get_tokenizer()
AttributeError: 'Anthropic' object has no attribute 'get_tokenizer'`

So I downgraded to 0.39.0 again, but then I got the first issue again.
Is there a solution for this problem?

You need a version of `httpx` before the `proxies` argument was deleted. According to the [httpx CHANGELOG](https://github.com/encode/httpx/blob/9e8ab40369bd3ec2cc8bff37ab79bf5769c8b00f/CHANGELOG.md#0280-28th-november-2024) version 0.27.2 and before should work. Could you try running:

```python
pip install httpx==0.27.2
```

@yifanmai wow! Thank you so much. It solved the problem I was struggling with all day.
Repository: stanford-crfm/helm
Issue: Missing score_util in live_qa_annotator
Root cause: Validation Gap

Forwarding a user's bug report:

> Hi Yifan, there seems to be code error in the helm main repo. I cant really run the live_qa  scenarios because there is no score_util file in the codebase although it is being imported here: https://github.com/stanford-crfm/helm/blob/318c3d9f0a8ba6cb31f700b84ea2320a051977ac/src/helm/benchmark/annotation/live_qa_annotator.py#L8C32-L8C42

@farzaank could you take a look?
Repository: stanford-crfm/helm
Issue: Support AI21's new `j1-grande` 17B parameters model (new medium-sized model)
Root cause: Unimplemented Feature Gap

"The new model is a mid-point in terms of size, cost and performance between Jumbo and Large. We also implemented a few tweaks to its training process. Internal benchmarks suggest it can really help the unit economics on your end compared to Jumbo, without compromising too much on quality."

TODO: Figure out the number of parameters. Not documented yet.

Get quota increased: https://github.com/stanford-crfm/benchmarking/issues/353
Repository: stanford-crfm/helm
Issue: Running `"boolq:data_augmentation=all": {status: "READY"}` errors
Root cause: Architectural Constraint

Error: 

```
  File "/Users/tonyhlee/research/mercury/benchmarking/src/benchmark/augmentations/contrast_sets_perturbation.py", line 66, in apply
    assert should_perturb_references  # contast sets do not make sense otherwise
AssertionError
```

Fixed as part of https://github.com/stanford-crfm/benchmarking/pull/304/ (just removing contrast sets from the "all" perturbations and adding a separate run). 

Hm, but this is actually a bigger issue where `should_perturb_references` is defined once for the data_augmenter and cannot take different values between perturbations.


@dtsip What's the status on this?

The error is fixed but we want to refactor things a little bit eventually (p2). Closing this and tracking in https://github.com/stanford-crfm/benchmarking/issues/457

Fantastic, thanks!
Repository: stanford-crfm/helm
Issue: Frontend needs to be able to display per-instance metrics
Root cause: Unimplemented Feature Gap


Made P2 since more important things to address for now.

@deepakn94 I have forgotten - what are the per-instance metrics that we need to show? Is it the inference time/is it fine if we just visualize the aggregate info?

I think this issue was mainly intended to provide a link on the frontend to the `per_instance_metrics.json` file that is already generated by the benchmark script (similar to `metrics.json`).

Handled in #825
Repository: stanford-crfm/helm
Issue: Verification about support for Granite models in the HELM leaderboard
Root cause: Unimplemented Feature Gap

Hello, team.

I recently made two contributions to add support for the IBM Granite 3.0 and 3.1 models. After these changes, I would like to know if there's anything additional I need to do to have these models appear in the HELM leaderboard, or if they will be automatically included in the next release of the framework.

Thank you in advance for your help!

Best regards,
Ronaldd Matias.

Hi, generally evaluation results need to be manually run internally. We are hoping to use the watsonx version of the Granite models (because we generally consider the first party API version to be the "official" version, and there are slight differences in how the chat template is applied between the API version and the Hugging Face version), but this will take a while to set up. I'm planning to have Granite results in a few weeks.
Repository: stanford-crfm/helm
Issue: Incorporating Deepseek R1
Root cause: Unimplemented Feature Gap

Would you have any plans to add Deepseek R1? Thanks!

Hi, I've added DeepSeek R1 to the AIR-Bench leaderboard this week, and I'm planning to add it to other leaderboards in a couple of weeks.

DeepSeek R1 has been added to Capabilities and Safety leaderboards as well.
Repository: stanford-crfm/helm
Issue: unicode isn't being displayed properly in web interface
Root cause: Interface Contract Mismatch

Unicode contained in the completions - for example, see:
http://crfm-models.stanford.edu/static/index.html?prompt=A%20%24%7Boccupation%7D%20is%20someone%20who&settings=temperature%3A%200.5%0Amax_tokens%3A%20100%0Amodel%3A%20%24%7Bmodel%7D&environments=occupation%3A%20%5Bmathematician%2C%20lawyer%2C%20doctor%2C%20programmer%2C%20president%5D%0Amodel%3A%20%5Bai21%2Fj1-jumbo%2C%20openai%2Fdavinci%5D


![single_quote_two_tokens](https://user-images.githubusercontent.com/16793796/147907563-96017b6d-1ed7-4868-ae5d-8fdd76b8ad5d.png)

@percyliang It looks like OpenAI breaks up the UTF-8 bytes for a single character into separate tokens.

For `’ =  \xe2\x80\x99`,

OpenAI gives back tokens:

```
      {
        "bytes:\\xe2\\x80": -0.28217113
      },
      {
        "bytes:\\x99": -0.0014966717
      }
```

We treat each token as a separate entity on the front end so users can hover over each token and see the log probability.

Again, similar to #29, we see a possible issue with the tokens:

```
          {
            "bytes:\\xe2\\x80": -0.024369814
          },
          {
            "bytes:\\x99": -4.477904e-05
          }
```

but `text` in the response is correct:

```
"text": " passes as an engineer to the uninitiated,\u201d he says, \u201cbut is really a designer.\u201d\n\nThat\u2019s a bit of a stretch, but it\u2019s true that programmers are engineers, too.\n\n\u201cI think it\u2019s a bit of a stretch, but it\u2019s true that programmers are engineers, too,\u201d says Jeff Dean, a distinguished engineer at Google who has been involved in the company\u2019s"
```