zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Add Support for GZIP
Root cause: Unimplemented Feature Gap

## Purpose

We can unpack files that have a `.tar` or `.tar.gz` extension, but we also want to accommodate files compressed without a directory structure.

Is this needed still?

I actually have the code that performs this in the MS MARCO scenario, moving it to the appropriate general purpose function that can be called by anyone would be nice, which is what this issue can address.
Repository: stanford-crfm/helm
Issue: Adaptive evaluation by customized data loader
Root cause: Unimplemented Feature Gap

Hi @yifanmai, 

We briefly discussed earlier the implementation of a data loader that allows for the adaptive loading of a batch of informative questions to enhance evaluation efficiency. Therefore, I am opening this issue and am happy to help create a PR for it as well. 

Cheers,
Sang

Great, I think the first step is to identify a couple of tasks / data augmentations to port over to HELM. Would you have pointers to those? Also, do you have any deadline constraints?
Repository: stanford-crfm/helm
Issue: Request to Include MMLU and Winogrande human-translated into 11 African languages in HELM Leaderboard Frontend
Root cause: Unimplemented Feature Gap

Hello HELM authors,

I am one of the authors of the benchmark "MMLU and Winogrande human-translated into 11 African languages" added to the HELM repo about a month ago via this PR: [https://github.com/stanford-crfm/helm/pull/3237](https://github.com/stanford-crfm/helm/pull/3237) and providing a benchmark to test LLMs on low-resource languages. However, when I was checking the leaderboard website at this link ([https://crfm.stanford.edu/helm/](https://crfm.stanford.edu/helm/)), I noticed that our benchmark was not present. As such, I was wondering what the process was for getting added to the website (e.g. with a page like this one: [https://crfm.stanford.edu/helm/cleva/latest/](https://crfm.stanford.edu/helm/cleva/latest/))? We would be glad to contribute any code, code modifications, or assets to be added; please just inform us about the steps we need to take and we can follow them promptly. Thank you very much!

Links to the benchmark information are provided below again for reference.

Repository
[https://github.com/InstituteforDiseaseModeling/Bridging-the-Gap-Low-Resource-African-Languages](https://github.com/InstituteforDiseaseModeling/Bridging-the-Gap-Low-Resource-African-Languages)
Dataset
[https://huggingface.co/datasets/Institute-Disease-Modeling/mmlu-winogrande-afr](https://huggingface.co/datasets/Institute-Disease-Modeling/mmlu-winogrande-afr)
Paper
[https://arxiv.org/abs/2412.12417](https://arxiv.org/abs/2412.12417)

Hi @Adam-Kasumovic, could you contact me via email at my git commit email regarding requesting a leaderboard? Generally adding a new leaderboard requires manual setup interally and does not automatically happen. I have a few questions before we set this up.

Hello @yifanmai, I sent the email. Thank you!

Leaderboard is now up here: https://crfm.stanford.edu/helm/mmlu-winogrande-afr/latest/
Repository: stanford-crfm/helm
Issue: Unable to run benchmarks with conf-paths and with local huggingface model
Root cause: Documentation Deficiency

Hi, I love this project, but I am not sure if I am doing something wrong or I missing something.  (crfm-helm==0.5.4 via pip)

I can run this:
`helm-run --enable-local-huggingface-models /my_absolute_local_path/Meta-Llama-3.1-8B-Instruct  --run-entries mmlu:subject=philosophy,model=huggingface/Meta-Llama-3.1-8B-Instruct --suite my-suite --max-eval-instances 10`

But I am unable to do it with conf-paths, like this:

```
export SCHEMA_PATH=src/helm/benchmark/static/schema_lite.yaml
export RUN_ENTRIES_CONF_PATH=src/helm/benchmark/presentation/run_entries_lite_20240424.conf
export NUM_TRAIN_TRIALS=1
export MAX_EVAL_INSTANCES=1000
export PRIORITY=2

export SUITE_NAME=my-suite
export MODELS_TO_RUN=huggingface/Meta-Llama-3.1-8B-Instruct
export HF_MODEL_PATH=/my_absolute_local_path/Meta-Llama-3.1-8B-Instruct

helm-run --conf-paths $RUN_ENTRIES_CONF_PATH --num-train-trials $NUM_TRAIN_TRIALS --max-eval-instances $MAX_EVAL_INSTANCES --priority $PRIORITY --suite $SUITE_NAME --models-to-run $MODELS_TO_RUN --enable-local-huggingface-models $HF_MODEL_PATH
```

My metadata and deployment:
prod_env/model_metadata.yaml
```
models:
  - name: meta-llama/Meta-Llama-3.1-8B-Instruct
    display_name: Llama 3.1 8B inst 
    description: Stock model 
    creator_organization_name: meta-llama 
    access: Limited
    num_parameters: 8043892736
    release_date: 2024-08-13
    tags: [TEXT_MODEL_TAG, PARTIAL_FUNCTIONALITY_TEXT_MODEL_TAG]
```
prod_env/model_deployments.yaml
```
model_deployments:
  - name: huggingface/Meta-Llama-3.1-8B-Instruct
    model_name: meta-llama/Meta-Llama-3.1-8B-Instruct
    tokenizer_name: meta-llama/Meta-Llama-3.1-8B-Instruct
    max_sequence_length: 2048
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        pretrained_model_name_or_path: /my_absolute_local_path/Meta-Llama-3.1-8B-Instruct
        device_map: auto
```

I tried multiple combinations and nothing worked for me. When I manually replace `model=text,` with `model=huggingface/Meta-Llama-3.1-8B-Instruct,` then I am able to run it with conf-paths but it is unsustainable to generate conf file for each model evaluation. 

Thank you in advance

Could you provide the error log for this command?

Could you also try:

```
export MODELS_TO_RUN=meta-llama/Meta-Llama-3.1-8B-Instruct

helm-run --conf-paths $RUN_ENTRIES_CONF_PATH --num-train-trials $NUM_TRAIN_TRIALS --max-eval-instances $MAX_EVAL_INSTANCES --priority $PRIORITY --suite $SUITE_NAME --models-to-run $MODELS_TO_RUN
```

`--models-to-run` should be the model name `meta-llama/Meta-Llama-3.1-8B-Instruct`, not the model deployment name `huggingface/Meta-Llama-3.1-8B-Instruct`

Also you should omit the `--enable-local-huggingface-models` flag (because the model is already defined in the conf files).

Additionally `max_sequence_length` should be set to 131072 for Llama 3.1 8B.

Hi @yifanmai ,

thank you for fast reply it works now.
Repository: stanford-crfm/helm
Issue: BBQ Scenario failing + other necessary fixes
Root cause: Validation Gap

Error: 

```
File "benchmarking/src/benchmark/bbq_scenario.py", line 101, in get_instances
with open(prompts_path, "r") as f:
IsADirectoryError: [Errno 21] Is a directory: 'benchmark_output/scenarios/bbq/data'
```

TODO: fix stereotype tags and answers
Repository: stanford-crfm/helm
Issue: How to submit the test results on the air-bench leaderboard of HELM by our closed-source LLM
Root cause: Documentation Deficiency

Hi there, We have a closed-source, self-developed large language model, and we would like to submit our test results on the air-bench leaderboard of HELM. Could you please advise on how to proceed with this?

Hi @carlWM, could you email me at yifan@cs.stanford.edu with more information about your closed-source model?
Repository: stanford-crfm/helm
Issue: make execution multithreaded
Root cause: Unimplemented Feature Gap

Create a thread pool with some parallelism in `Executor`, so that we can process the examples in parallel.

```
        # TODO: make a thread pool to process all of these in parallel up to a certain number
        def render_instance(instance: Instance, pred_output: Optional[str]) -> str:
            tags_str = ",".join(instance.tags)
            gold_output: Optional[str] = None
            if instance.first_correct_reference is not None:
                gold_output = instance.first_correct_reference.output

            if request_state.output_mapping is not None and pred_output is not None:
                pred_output = request_state.output_mapping.get(pred_output.strip())
            correct_str = "CORRECT" if gold_output == pred_output else "WRONG"
            return (
                f'[{tags_str}] "{instance.input[:100]}" => "{gold_output}", predicted "{pred_output}" [{correct_str}]'
            )

```
Repository: stanford-crfm/helm
Issue: MedHELM Runs on GCS
Root cause: Documentation Deficiency

We want to use the MedHELM model runs and outputs for our research. On the GCS bucket, [crfm-helm-public](https://console.cloud.google.com/storage/browser/crfm-helm-public?inv=1&invt=AbrrpQ) I could not find the medHELM runs. 

Do we know when this data will be uploaded? 

Thank you for the help, this dataset would be really helpful for our research.

Hi @anubrata, thanks for the inquiry. The data should be available in GCS some time next week - I'll update this issue to let you know.

Hi, @yifanmai. I just wanted to follow up on this. Could you please share the GCS link for the Medhelm runs? 

Hi @anubrata, thanks for your patience. The MedHELM results have now been added to `medhelm/benchmark_output` in the `crfm-helm-public` bucket. You can find additional information on downloading the data in the [documentation](https://crfm-helm.readthedocs.io/en/latest/downloading_raw_results/).
Repository: stanford-crfm/helm
Issue: Train-test contamination
Root cause: Unimplemented Feature Gap

Pilot:
- Fix GPT-3
- Look at scenarios whose inputs fall in and out of training data
- Look at the distribution of log probabilities


Assigning to Percy as reminder/placeholder until we find someone else to take on. 

@Tiiiger Maybe you could post updates here, so we can update this as needed. Will keep Percy as the assignee for now, but can switch over at some point if it makes sense.

@teetone @percyliang @teetone 
Status: 

Try to evaluate the perplexity of common benchmark test/validation set of AI21 models.

A lot of test sets are too big to evaluate directly so I am looking into the convergence of subsampling.

@Tiiiger Just to track the status so everyone's on board:
1. Initial experiments with AI21 models seem promising
2. Since there are various bottlenecks for further testing with OpenAI/Microsoft/Anthropic/EleutherAI (via Goose) models, next step is testing with public models on HF like GPT-2

Is this right/anything else to add (incl. the experiments/figures whenever you get a chance)?

(Note: Moving this to P2 for now.)

transferring to @fladhak 

also i just realized i was a complete idiot pushing these to `main` directly. Should have created a separate branch and do PR.  sorry for this. 

Closing because contamination tracking is deprecated.
Repository: stanford-crfm/helm
Issue: Issue with link to instances beyond first page
Root cause: Algorithmic Error

There seems to be a bug on helm frontend - link to instances beyond the first page does not automatically bring you to that page. 

E.g.,https://crfm.stanford.edu/helm/lite/latest/#/runs/narrative_qa:model=openai_gpt-4o-2024-05-13?instancesPage=2&instance=id1118, clicking on it, the frontend goes to page 1 instead of page 2.

However, if you click on the bottom navigation button to reach page 2, it automatically jumps to that instance.
Repository: stanford-crfm/helm
Issue: Proposal: Integrating Vietnamese LLM evaluation
Root cause: Unimplemented Feature Gap

# Background
We have talked about the integration of Vietnamese LLM evaluation into HELM as there was no reliable evaluation toolkit before. The implementation is mainly based on our NAACL 2024 paper: Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models (https://aclanthology.org/2024.findings-naacl.182).

# Proposal
The evaluation includes ten scenarios, each targeting a key aspect of LLM capability:

- Summarization: Evaluates the model's ability to condense large texts while retaining essential information.
- Question-Answering: Assesses comprehension and accurate extraction of answers from provided contexts.
- Knowledge: Tests the model's ability to recall and apply information across different domains.
- Sentiment Analysis: Measures the ability to detect and classify emotional tones in text.
- Text Classification: Evaluates accuracy in categorizing text into predefined labels.
- Toxic Detection: Identifies the model's capacity to flag harmful or biased language.
- Language Modeling: Tests fluency and coherence in generating contextually appropriate text.
- Reasoning: Measures logical deduction and understanding of complex relationships.
- Math: Assesses competency in solving mathematical problems in text form.
- Information Retrieval: Tests effectiveness in searching, retrieving, and synthesizing relevant information.
- Translation: Translate text from English to Vietnamese and vice versa

Our evaluation also includes tools to ensure the ethical deployment of LLMs:

- Bias Assessment: Identifies and mitigates potential biases in model outputs.
- Toxicity Assessment: Measures and controls the generation of harmful or offensive language.
- Fairness Evaluation: Ensures equitable performance across demographic groups and languages.
- Robustness Analysis: Examines resilience to noisy inputs and adversarial attacks, ensuring reliable performance in real-world scenarios.



Great, I think the first step is to open a pull request for a scenario. I'd recommend starting with the "easiest" scenarios, and doing one pull request per scenario. Let me know if you need any additional advice on how to do this.
Repository: stanford-crfm/helm
Issue: Broken links of scenario and metric data
Root cause: Architectural Constraint

Hi,

I recently noticed that there are some hard-coded links no longer valid and also some new dataset changes. The benchmark process will break if data files were not cached previously.

- All links pointing to https://worksheets.codalab.org seems no longer valid.
  - e.g. `wget` the link in [disinformation_metrics.py](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/metrics/disinformation_metrics.py) returns 404 error code.
- `TruthfulQA` recently updated the their dataset by adding a new column `Best Incorrect Answer` with the link coded in [truthful_qa_scenario.py](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/truthful_qa_scenario.py). They moved the original dataset to a different link: https://github.com/sylinrl/TruthfulQA/blob/main/data/v1/TruthfulQA.csv.

Thank you and have a great day!

Thanks for the report - I'll take a look.

CodaLab had an outage and has recovered - the links to CodaLab should be working again.

@gaduan8 The URL for TruthfulQA has been fixed. Thanks again for the bug report!
Repository: stanford-crfm/helm
Issue: Issues with running claude-3-7-sonnet via the remote service
Root cause: Environment Incompatibility

Hi @yifanmai,  
I encountered an issue while attempting to run the latest claude-3.7-sonnet model (anthropic/claude-3-7-sonnet-20250219) via the remote service.  

```
helm.proxy.services.remote_service.RemoteServiceError: EXCEPTION: Unknown model 'anthropic/claude-3-7-sonnet-20250219'
  File "/home/benchmarking/src/helm/proxy/server.py", line 67, in safe_call
    result = func(params)

  File "/home/benchmarking/src/helm/proxy/server.py", line 196, in perform
    raise ValueError(f"Unknown model '{request.model}'")
 Request: {"model_deployment": "", "model": "anthropic/claude-3-7-sonnet-20250219", "embedding": false, "prompt": "Life is like a box of", "temperature": 1.0, "num_completions": 1, "top_k_per_token": 1, "max_tokens": 100, "stop_sequences": [], "echo_prompt": true, "top_p": 1, "presence_penalty": 0, "frequency_penalty": 0, "random": null, "messages": null, "multimodal_prompt": null, "image_generation_parameters": null}
```

The model is added to the model_deployment config file in PR #3366 , so I believe the problem stems from the remote service still running the old release (v0.5.4), which doesn’t support the newer models.  
Would it be possible for you to update the remote service to support the latest models? Thanks!

Thanks for opening an issue; I'll redeploy the service today.

Redeployed; it should be working now.
Repository: stanford-crfm/helm
Issue: Where did https://crfm.stanford.edu/helm/latest go?
Root cause: Documentation Deficiency

There are references to https://crfm.stanford.edu/helm/latest in the docs, but it returns 404 as of now. What would be new proper address?

Thanks for the bug report! The new updated URL is https://crfm.stanford.edu/helm/classic/latest/

I've updated the docs to refer to this new URL instead of the old one. Additionally, I've added a redirect form the old URL to the new one.
Repository: stanford-crfm/helm
Issue: Urgent help needed on max_tokens
Root cause: Documentation Deficiency

I am running a test for a model trained for reasoning, which maxes out the token limit. How can I increase the limit which is now set to 512 tokens?


Hi @rankofootball, you can use the `increase_max_tokens` to increase the number of max tokens for reasoning models. For instance, to run simple_safety_tests on o1, we use the run entry `simple_safety_tests:increase_max_tokens=4096,model=text`.

If you're running this on MCQA scenarios, you may also need `output_format_instructions=mcqa` . For instance, to run bbq on o1, we use `bbq:subject=all,method=multiple_choice_joint,output_format_instructions=mcqa,max_train_instances=0,increase_max_tokens=4096,model=text`

Closing due to staleness. Feel to reopen if you have additional questions.
Repository: stanford-crfm/helm
Issue: numeracy scenario gets stuck at `trial 0: evaluate 54 (total 200)`
Root cause: Resource Mishandling

Run `benchmark-present --max-eval-instances 200  --conf src/benchmark/presentation/run_specs_copy.conf`

where `run_specs_copy.conf` has the following line

```
"numeracy:model=openai/davinci,relation_type=paraboloid,mode=function": {status: "READY"}
```

@friedeggs Is this resolved now by turning off the solver for the parabol- relations?

If so, feel free to close, thanks!

It's deferred! I'll close since it's superseded by #452. Thanks!
Repository: stanford-crfm/helm
Issue: Improve gdown for the copyright scenario
Root cause: Unimplemented Feature Gap

Throw an error when the user didn't install `gdown`. 
Also, it seems like we download every time we run the scenario, even if the file is already downloaded.

Thanks for the note. Let me address this after dinner. 

@lxuechen I did a test run, and it works, so no rush. It's just something we should improve.

@lxuechen Could you also move the gdown download logic to `general.py` so future Scenarios can use it?
Repository: stanford-crfm/helm
Issue: Proposal: Integration of Speech Pathology Scenarios into HELM
Root cause: Unimplemented Feature Gap

### Background

With nearly [3.4 million children](https://www.buffalo.edu/ai4exceptionaled.html) requiring speech and language services, there is a growing need for tools that assess the ability of speech language models (Speech LLMs) to support clinical applications, particularly in the early detection of speech disorders in children. There has been some work ([example](https://arxiv.org/pdf/2501.0003939)) in developing better speech models for these sorts of use cases, but there isn't a robust evaluation framework for these models, and to compare these models. This proposal introduces new evaluation scenarios in HELM explicitly designed for speech pathology, enabling benchmarking of models on relevant and sensitive tasks.
The initial evaluation will use the following datasets to run evaluations: 

- [UltraSuite](https://ultrasuite.github.io/) - Pediatric speech recordings from both typically developing children and those with diagnosed speech disorders
- [Dysarthia Dataset](https://www.kaggle.com/datasets/poojag718/dysarthria-and-nondysarthria-speech-dataset) - Audio data for patients with and without dysarthia
- [Czech Dataset](https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-1597) -  Audio data for children with and without speech language impairment
- [Gilliam Dataset](https://www.kaggle.com/datasets/dgokeeffe/specific-language-impairment) - SLI/Typically Developing speech data

We are actively seeking additional datasets covering other languages and disorder types to ensure broader coverage and generalizability.

### Core Scenarios

1. Binary Disorder Classification - Determine whether a given speech sample exhibits signs of a speech disorder.
 Metrics: Accuracy, F1-score, Area Under the Curve (AUC)

2. Disorder Type Classification - Classify the specific type of speech disorder (e.g., articulation/phonological, etc).
 Metrics: Macro-F1, Confusion Matrix

3. Symptom Localization - Identify specific symptoms of speech disorder in audio (e.g., stuttering, omission, substitution, etc).
 Metrics: F1, Intersection-over-Union (IoU)

4. Speech Transcription - Transcribe audio for patients with speech disorders
 Metrics: WER

5. Classification via transcription (baseline metrics): Use transcription to determine disorder classification
Metrics: Accuracy, F1-score, Area Under the Curve (AUC)


Hi @patelfagun1998, thanks for the helpful suggestion and references. I think these scenarios are very valuable, and we currently have a few audio / speech models in HELM that can be evaluated on these scenarios. Are you interested in contributing these scenarios to HELM? If so, feel free to open a pull requests for them.

Thanks for getting back to me! Created the [PR](https://github.com/stanford-crfm/helm/pull/3533) for the classification scenario. Will structure the other ones based on any guidance I get for this one.
Repository: stanford-crfm/helm
Issue: `max_tokens` restriction for Anthropic models
Root cause: Validation Gap

When using the latest Anthropic models (e.g., claude-3-7-sonnet-20250219) through the HELM server, I encountered the following error:
```
EXCEPTION: Request.max_tokens must be <= 4096
```

I believe the error originates from the line here: https://github.com/stanford-crfm/helm/blob/bbe78de791bf67f6bf3b37872635cbeda36a893d/src/helm/clients/anthropic_client.py#L246 . The error would occur when we set `max_tokens` above 4096.

According to the official Anthropic documentation (https://docs.anthropic.com/en/docs/about-claude/models/all-models), newer models like Claude 3.7 Sonnet support a maximum output token limit of 64k, well above the current 4096-token constraint.

I'm wondering if it is possible to remove or adjust the restriction on `max_tokens` for Anthropic models?


This makes sense. Could you send me a pull request for this?
Repository: stanford-crfm/helm
Issue: Tracking for the IR scenarios 
Root cause: Unimplemented Feature Gap

## In-context instance sampling

- [ ] In `MSMARCO`, we choose 4 in-context instances, 2 with the positive answers and 2 with the answers. There is no guarantee that the chosen instances would contain the positive and the negative example using the same passage. We think that adding this constraint could help us achieve a higher MRR score, so this is something we want to experiment with.
- [ ] We want to experiment with adding "recall_cut" measure.
- [ ] The benchmarking project is missing the training efficiency values for J-1 Jumbo / Grande models, which is leading to a long wait time when using the `BasicMetrics`. We have commented out the BasicMetrics for the time being to be able to run our specs in reasonable amount of time. We want to put these back in when the efficiency computations get faster / use caching.

## Things to mention in the paper

- [x] Handling utf8 issues using `ir-datasets`