zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Keeping track of nuances for ICE
GitHub labels: harms

Paper

@nathankim7 Can you add any nuances we need to keep in mind when writing the paper to discuss about ICE. 

Things that come to mind to start:
0. Describe the structure of the data (e.g. different types of text) + how it was collected (especially since ICE will not be familiar). Bring attention to importance of having such coverage and recognize the value provided by this linguistic fieldwork. 
1. Why we chose certain regions, both due to availability of metadata and due to coverage arguments
2. That we consider metadata for spoken vs. written and for gender. We choose to ignore age because messy/couldn't figure out clear way to include nicely.
3. That we handle the tags in spoken/written data and specify how we do the preprocessing (noting challenges for East Africa in particular). Note that these also appear in written data because the annotators corrected misspellings and we use the correct spellings. Importantly, note the annotators were regional as well, as this seems especially good.
4. Remember to include the distribution of different ethnicities from Nigeria (e.g. Igbo, Yoruba). Similarly include for other datasets if available. 
5. Remember to include the gender distribution for each region. We can also include the age distribution per region which then means they don't need to be standardized across regions




1. Covered in the Data section
2. Covered in last paragraph of Data section
3. Covered in Pre-processing section; mention of annotators in Data section
4. We are no longer covering any datasets where ethnicity is explicitly notated. However of the datasets that do have readable metadata (CAN, HK, IND, USA), there is some variety in birthplace/nationality/languages spoken which suggests some ethnic variety. This would be better addressed, though, in the analysis along with a deeper discussion of point 1. that references existing lit on World Englishes (I am getting that ready).
5. The above datasets have 50:50 parity on gender.

Great, excellent work!!
Repository: stanford-crfm/helm
Issue: Download OPT-175B Weights
Root cause: Unimplemented Feature Gap

I have received the URL for downloading the weights, we need to decide where they need to go.

Instructions to download are provided here: https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/download_opt175b.md

License terms are provided here:
https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/MODEL_LICENSE.md

Download to Azure, spin up 16 A100s to serve it.  But don't keep it running all the time.

Note: Rohan downloaded to `/nlp/scr2/nlp/opt-175b-lm/model_weights` in case that is helpful. My link will expire in 2-3 days.

Should already be on cluster thanks to Tatsu's folks
Repository: stanford-crfm/helm
Issue: Duplicate entries in med_dialog dataset
Root cause: Validation Gap

I was looking at the `med_dialog` dataset in MedHELM and noticed that there are a large number of duplicate questions, especially in the `icliniq` split.

Looking at the HELM source code here:
https://github.com/stanford-crfm/helm/blob/79909a2fdbb83953912d9b73b5cc1e86d25ab8f0/src/helm/benchmark/scenarios/med_dialog_scenario.py#L118

I downloaded the "original" dataset from
https://worksheets.codalab.org/rest/bundles/0x82f0c47f6d3e4462ae9ef8ea39eebe64/

and confirmed the duplicate entries are present there as well:
```
>>> import json
... 
... for subset in ['icliniq', 'healthcaremagic']:
...     for split in ['train', 'valid', 'test']:
...         x = json.load(open(f'{subset}/{split}.json'))
... 
...         n_total = len([xx['src'] for xx in x['data']])
...         n_unique = len(set([xx['src'] for xx in x['data']]))
...         print(f'{subset} {split}: {n_total} total, {n_unique} unique')
```
gives
```
icliniq train: 24851 total, 16573 unique # dups
icliniq valid: 3105 total, 2087 unique # dups
icliniq test: 3108 total, 2069 unique # dups
healthcaremagic train: 181122 total, 181112 unique # dups
healthcaremagic valid: 22641 total, 22641 unique
healthcaremagic test: 22642 total, 22642 unique
```

(I'm not sure what the data hosted at `codalab` is / where it comes from.  Possibly from the BioBART preprocessing?  Unfortunately the original MedDialog link is dead: https://github.com/UCSD-AI4H/Medical-Dialogue-System)



Hi @bkj, thanks for the investigation and the reproduction instructions. I've confirmed that the duplication is indeed a problem in the dataset based on your code. Perhaps we can publish a new version of this scenario where we do the deduplication inside HELM. cc @MiguelAFH
Repository: stanford-crfm/helm
Issue: Feature Request: Batch Processing Support for LLM API Integration
Root cause: Unimplemented Feature Gap

# Feature Request: Batch Processing Support for LLM API Integration with Medhelm

## Overview

We would like to request the addition of robust batch processing capabilities to your Large Language Model (LLM) API, specifically to facilitate seamless integration with Medhelm. The goal is to enable efficient, scalable processing of dataset inputs in batches, evaluation of responses, and structured result storage, while supporting optional intermediate measurements for tracking progress. This enhancement should be designed to work without requiring any modifications to the Medhelm repository.

## User Story

As a developer integrating your LLM API with the Medhelm platform,  I want to process dataset inputs in batches,  
evaluate LLM responses using predefined metrics,  
and store the results in a structured format,  
so that I can efficiently monitor and benchmark LLM performance  
without making any changes to the Medhelm repository.

## Acceptance Criteria

- The API supports processing of dataset inputs in batches for LLM tasks.
- Batch and final results are written with relevant metadata (e.g., batch ID, timestamp).
- The system gracefully handles errors, logging issues without halting the overall process.
- Key configuration parameters, including batch size and evaluation metrics

---

**Thank you for considering this feature request. We believe this enhancement will streamline LLM evaluation workflows and improve large-scale task management within Medhelm.**

Hi @blidiselalin, thanks for the suggestion. Batch support would be very useful and has been previously requested a few times (see #2336 and #1341). However, it would be tricky to implement as it requires architectural changes. Therefore, we may revisit this some time in the future but it is not on the near-term roadmap.

If you're interested in tackling this problem, please first post a brief design document here, so that we can have a discussion on design decisions before you proceed with implementation.
Repository: stanford-crfm/helm
Issue: boolq dataset on alpaca evaluation issue
Root cause: Documentation Deficiency

Alpaca maximum sequence length is 512 
Boolq  maximum sequence length is 828

How was this solved during the evaluation?

When the prompt is longer than the sequence length of the model, we truncate the prompt to fit. This is described in the original [HELM paper](https://arxiv.org/abs/2211.09110) (search for "truncation"). You can also refer to the code for [`_make_prompt_fit`](https://github.com/stanford-crfm/helm/blob/v0.5.6/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L287), [`fits_within_context_window`](https://github.com/stanford-crfm/helm/blob/v0.5.6/src/helm/benchmark/window_services/local_window_service.py#L89) and [`truncate_from_right`](https://github.com/stanford-crfm/helm/blob/v0.5.6/src/helm/benchmark/window_services/local_window_service.py#L96) for more information about how the truncation is implemented.

Thanks
Repository: stanford-crfm/helm
Issue: cannot import name 'cached_download' from 'huggingface_hub'
Root cause: External Dependency Breakage

**​​Python Version​​:** 3.9
**​​Virtual Environment​​:** Created using uv (uv venv --python 3.9 .venv)
**Command Executed​​:**
`helm-run     --run-entries mscoco:model=huggingface/stable-diffusion-v1-4     --suite my-heim-suite     --max-eval-instances 10 `
**​​Error Message​​:**
`ImportError: cannot import name 'cached_download' from 'huggingface_hub' (/data3/zhangdeming/DiffusionModelsEval/.venv/lib/python3.9/site-packages/huggingface_hub/__init__.py)`



This seems to be related to this [StackOverflow answer](https://stackoverflow.com/a/79377102), which says that `cached_download` has been removed from new versions of `huggingface_hub` and recommends installing an old version of `huggingface_hub` as a workaround (`pip install huggingface-hub==0.25.2`). Could you try it out and see if it works for you?

The long term fix is to remove calls to `cached_download` and bump the required version of `huggingface_hub` in our dependencies.
Repository: stanford-crfm/helm
Issue: improve representation of instance in per_instance_metrics.json
Root cause: Architectural Constraint

Currently, we are using the Python-stringified version of an Instance as a key:
```
{
  "(Instance(input='{The country\\'s consumer watchdog has taken Apple to court for false advertising because the tablet computer does not work on Australia\\'s 4G network. 
Apple\\'s lawyers said they were willing to publish a clarification. However the company does not accept that it misled customers. The Australian Competition and Consumer Co
mmission (ACCC) said on Tuesday: \"Apple\\'s recent promotion of the new \\'iPad with wi-fi + 4G\\' is misleading because it represents to Australian consumers that the prod
uct can, with a sim card, connect to a 4G mobile data network in Australia, when this is not the case.\" The watchdog then lodged a complaint at the Federal Court in Melbour
ne. At a preliminary hearing, Apple lawyer Paul Anastassiou said Apple had never claimed the device would work fully on the current 4G network operated by Telstra. Apple say
s the new iPad works on what is globally accepted to be a 4G network. The matter will go to a full trial on 2 May. The Apple iPad\\'s third version went on sale earlier this
 month, with Australia the first country where it was available. Shoppers lined up by the hundreds at Apple stores on opening day and the company said it had been its strong
est iPad launch to date. The ACCC said it was seeking an injunction on sales as well as a financial penalty against Apple, corrective advertising and refunds to consumers. O
n its website, Apple does state that 4G LTE is only supported on selected networks in the US and Canada.}', references=[Reference(output='US technology firm Apple has offere
d to refund Australian customers who felt misled about the 4G capabilities of the new iPad.}', tags=['correct'])], split='valid', sub_split=None, id='id7528', perturbation=N
one, contrast_inputs=None, contrast_references=None), 0)": [
```

Instead, if each instance already has a unique ID, we should use that as the key.  Otherwise, we can just have the list of metrics without the instances as a parallel array to the list of instances in `scenario.json` (slightly dangerous, but we need to save space since these are just going to be all loaded in the browser).
Repository: stanford-crfm/helm
Issue: ModuleNotFoundError and model not recognized
Root cause: Documentation Deficiency

Hello HELM team,

I’m experiencing two persistent issues when trying to run HELM evaluations, both in Google Colab and in a clean local environment (PyCharm, Python 3.10), despite installing the library directly from source using:

git clone https://github.com/stanford-crfm/helm.git
cd helm
pip install -e .[all]

1. ModuleNotFoundError: No module named 'helm.benchmark.simple_scenarios'
When attempting to import:

from helm.benchmark.simple_scenarios import MultipleChoiceScenario

I receive:

ModuleNotFoundError: No module named 'helm.benchmark.simple_scenarios'

This happens even after verifying that other modules (like helm.benchmark.metrics.basic_metrics) are present. The simple_scenarios.py module seems to be missing entirely from the installed package.

2. Model not recognized by helm-run

In a separate attempt to run a scenario with:

--models-to-run stanfordhealthcare/llama-4-maverick-17b-128e-instruct-fp8

I encounter the following:

Exception: Unknown model 'stanfordhealthcare/llama-4-maverick-17b-128e-instruct-fp8'

The model is correctly defined in the YAML configuration file, but helm-run fails to recognize it.

Questions:
	•	Has simple_scenarios.py been removed or renamed in recent versions?
	•	What is the correct way to define and evaluate multiple-choice tasks now?
	•	How can we ensure that public models are correctly recognized by helm-run?

Any guidance would be greatly appreciated. 

Thanks in advance for your help!

Hi @burakverdi-hue,

In the latest PyPI release v0.5.7 and the current main branch, the `simple_scenarios` module is defined in [this file](https://github.com/stanford-crfm/helm/blob/v0.5.6/src/helm/benchmark/scenarios/simple_scenarios.py). The full module path is `helm.benchmark.scenarios.simple_scenarios`. In addition, there is no `MultipleChoiceScenario` in this file - you are probably looking for `SimpleMCQAScenario`. Putting these together, the correct command should be:

```python
from helm.benchmark.scenarios.simple_scenarios import SimpleMCQAScenario
```

The arguments to `model-to-run` should be the values in the `model_name` field in `model_deployments.yaml`, _not_ the `name` field. For instance, you should use `--models-to-run meta/llama-4-maverick-17b-128e-instruct-fp8` instead of `--models-to-run together/llama-4-maverick-17b-128e-instruct-fp8`

Note that by default, HELM will use Together AI for Llama 4, so you need to have a Together AI API key. If you do not have one, you can run the Hugging Face model locally by following [these instructions](https://crfm-helm.readthedocs.io/en/latest/huggingface_models/).

Also note that the model deployments that start with `stanfordhealthcare/llama-4-maverick-17b-128e-instruct-fp8` are for Stanford Healthcare internal use only, and does not do anything useful for external users.
Repository: stanford-crfm/helm
Issue: Need Support for Schema-Less Benchmarks
Root cause: Unimplemented Feature Gap

Description:
Currently, leaderboard schemas must be manually modified for a new benchmark to appear in leaderboards. This manual step increases the effort and friction required when adding benchmarks, making the process more error-prone and less scalable.

Proposed Improvement:
A RunSpec should be sufficient to provide all the information needed for a leaderboard schema. If the RunSpec fully specifies the benchmark details, the system should be able to automatically generate or update the corresponding leaderboard entry without requiring manual schema edits.

Benefits:
* Reduces manual intervention when adding benchmarks.
* Ensures consistency between RunSpec definitions and leaderboard schemas.
* Lowers maintenance overhead for leaderboard management.
* Makes it easier for contributors to add new benchmarks.
Repository: stanford-crfm/helm
Issue: Populate `schemas.yaml`
Root cause: Documentation Deficiency

Make sure all metrics have descriptions.

Add description of MMR@10 into schema.yaml [Issue 384; Rishi] (bug bash)

Can we distribute this to people who own different metrics?

I think it is probably just faster for me to do, but sure we can try. Just need to get an hour or so to handle this, so will resolve either way by Sunday as a concrete deadline. 

Also add units. 

I'm getting:
WARNING: Missing an entry for the following metrics in src/proxy/static/schema.yaml: 
        num_output_tokens: mmlu 
        inference_runtime_discrepancy: mmlu 
        finish_reason_length: mmlu 
        finish_reason_stop: mmlu 
        finish_reason_endoftext: mmlu 
        finish_reason_unknown: mmlu 
        num_in_context_examples: mmlu 
        num_instances: mmlu 
        perplexity: mmlu 
        bits_per_byte: mmlu 
        logprob_per_byte: mmlu 
        estimated_num_tokens_cost: mmlu 
        num_prompt_tokens: mmlu 
        max_num_output_tokens: mmlu 
        num_requests: mmlu
Repository: stanford-crfm/helm
Issue: Duplication of main metrics
Root cause: Architectural Constraint

The HELM leaderboards have used the convention where each scenario has a "main metric" which is usually an accuracy metric. If the main metric are also added to another metric group such as "accuracy", this leads to odd behavior where the metric will show up twice on the scenario tables. The workaround has been to not add main metrics to a separate metric group, which also feels rather unnatural.
Repository: stanford-crfm/helm
Issue: Request for release of clinician ratings and model outputs from Section 2.3.2
Root cause: Unimplemented Feature Gap

Hi authors,

Thanks for the great work on MedHELM. I wanted to ask whether you have plans to release the evaluation data used in **Section 2.3.2 Evaluation of Open-Ended Benchmarks** — specifically,

- the 31 instances from ACI-Bench and 25 instances from MEDIQA-QA that were evaluated, and
- the independent clinician ratings on those outputs.

We’d like to experiment with other judger models and measure their intraclass correlation coefficients (ICC) against clinician ratings.

Thanks!

Hi,

We are going through a peer-reviewed submission and have plans to  release the evaluation data once the paper gets accepted. 

Thanks
Repository: stanford-crfm/helm
Issue: standardize multiple choice scenarios
Root cause: Architectural Constraint

Some scenarios have Yes and No listed explicitly as references
https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=msmarco%3Atask%3Dpassage%2Ctrack%3Dregular%2Ctopk%3D20%2Cnum_eval_queries%3D10%2Cnum_train_queries%3D1000%2Cmodel%3Dopenai_davinci
and some don't:
https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=boolq%2Cmodel%3Dopenai_davinci

We should standardize. (cc @dtsip @dilarasoylu @rishibommasani)

We agreed that we _don't_ want to list the references as A/B since it is less natural---the labels already have meaning as words so we don't want to make the model figure out the multiple-choice format in-context.
Repository: stanford-crfm/helm
Issue: Batch download of predictions data from leaderboards web site?
Root cause: Documentation Deficiency

Hi @yifanmai ,

Apologies for using the Github issues tracker for this but I could not find another contact for you.

Is there a way to batch download all of the Full JSONs of predictions for a specific leaderboard?

I would be interested in MMLU, i.e. the JSONs for all 46 pages from here: https://crfm.stanford.edu/helm/mmlu/latest/#/runs

Thanks (we are working on comparing alternative ways to rank models across multi-task benchmarks)

Hi, you can find instructions for downloading the JSON files [here](https://crfm-helm.readthedocs.io/en/latest/downloading_raw_results/).

tldr: all the data for that leaderboard is stored in Google Cloud Storage under `gs://crfm-helm-public/mmlu/benchmark_output`, so you can download everything using `gcloud rsync`.

Great!! That is exactly what I need, thanks for the quick reply.
Repository: stanford-crfm/helm
Issue: msmarco prompts have two "questions"
Root cause: Documentation Deficiency

https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=msmarco%3Atask%3Dpassage%2Ctrack%3Dregular%2Ctopk%3D20%2Cnum_eval_queries%3D10%2Cnum_train_queries%3D1000%2Cmodel%3Dopenai_davinci

Question: Does the passage above answer the question what causes an inflamed bunion and what can be done about it?

The double "question" might be confusing, and "what causes ..." should be quoted or something?  Also, could we move more of the prompt hacking logic to the adapter?
Repository: stanford-crfm/helm
Issue: Thoughts on adding image-to-image (i2i) benchmarks?
Root cause: Unimplemented Feature Gap

From what I understand HELM contains image-to-text (VHELM, Image2Struct) and text-to-image (HEIM) benchmarks, but as far as I can tell there aren't any image-to-image or image editing benchmarks. I've seen a few repos that do handle this: 

https://github.com/ali-vilab/ICE-Bench

https://github.com/PhoenixZ810/RISEBench

I think the input / output data structures both can handle having an attached media object (like an image), but there currently just isn't a set of metrics that would produce anything meaningful. 

I'm wondering if there are any plans or any interest in adding image-to-image benchmarks to HELM. Our meta-benchmarking system - which is currently using HELM - will need this capability and I'm wondering: is there interest in this feature being contributed upstream? If so, do maintainers have any thoughts on the subject or are there any anticipated pain points?

@yifanmai Any thoughts on this?

Both ICE-Bench and RISEBench look reasonable. I'm not planning to implement these myself, but I would be happy to review pull requests if you plan to add these benchmarks in the library.

There are a number of general reasons why we have not done more benchmarking of models with image outputs, and these issues also apply to both benchmarks:

1. There are not many notable end-to-end multimodal models that output images yet, as can be seen from the small number of models in both papers.
2. In general, VLMs and some automated metrics are a poor judge of image quality, and don't have good alignment with human ratings.
3. Many image processing libraries used for computing image metrics are logistically difficult to install. Some require manual compilation of binaries, or have conflicts with other dependencies of HELM.

> Many image processing libraries used for computing image metrics are logistically difficult to install. Some require manual compilation of binaries, or have conflicts with other dependencies of HELM.

Yeah, I hear that. Welcome to my life. I'm so enveloped in it all the time, I forget the overhead that comes from bringing this into a new project.

However, there is a chance I will want to integrate these benchmarks with HELM, so I want to think carefully about how it might be done and make a plan before I make a push. One idea for a path forward is to build specific docker images for each flavor of dependencies a user might want to run, and a base HELM instance could dispatch jobs to appropriate backend images. Perhaps it's not even HELM itself doing it, but something that can produce a benchmark compatible with what helm-summarize expects. That might free up design space to address other issues as we discussed in: https://github.com/stanford-crfm/helm/issues/3680

In any case, this answer is helpful and I'll consider this resolved from HELM's perspective.