zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Proposal: Adding Vietnamese LLMs into HELM
Root cause: Unimplemented Feature Gap

# **Motivation**  
Currently, many Vietnamese-specific large language models (LLMs) are not officially supported by HELM. As a result, users who wish to evaluate or benchmark these models must manually define configuration files, which can be time-consuming and error-prone. Given that Vietnamese is among the top 10 most widely spoken languages globally, expanding HELM’s support to include Vietnamese LLMs would be a significant step toward promoting multilingual equity and advancing language technology for underrepresented languages. Official integration would lower the barrier for researchers and practitioners working on Vietnamese NLP and foster broader adoption and development of LLMs for Vietnamese.

# **Proposal**  
We propose to incorporate support for several prominent Vietnamese LLMs into HELM. These include:
- **URA-LLaMa**: https://huggingface.co/collections/ura-hcmut/ura-llama-65d6fba390f11951bcfce7f4
- **VinaLLaMa**: https://huggingface.co/collections/vilm/vinallama-654a099308775ce78e630a6f
- **PhoGPT**: https://huggingface.co/vinai/PhoGPT-7B5-Instruct
- **GemSUra**: https://huggingface.co/collections/ura-hcmut/gemsura-65da96cd27be2e8c65f17131
- **MixSUra**: https://huggingface.co/collections/ura-hcmut/mixsura-65d6fae4fc2da9f5bdf59a79
- **Vistral**: https://huggingface.co/Viet-Mistral/Vistral-7B-Chat
- **Vietcuna**: https://huggingface.co/vilm/vietcuna-7b-v3

We are happy to contribute the necessary configuration files and model specifications to facilitate this integration.

Sounds reasonable. Note that HELM currently does not support applying the chat template for Hugging Face models, which we should probably fix in the future.
Repository: stanford-crfm/helm
Issue: How to Configure with Custom Model Deployments in ANNOTATOR_MODELS
Root cause: Unimplemented Feature Gap

**📝 Description:**  
Need guidance on configuring `ANNOTATOR_MODELS` to use self-hosted or custom-deployed models instead of pre-defined ones like `gpt`, `llama`, or `claude`. 

Please provide the steps or structure required to integrate a custom model deployment under the `AnnotatorModelInfo` schema.

https://github.com/stanford-crfm/helm/blob/21a7531f375abcdeec454ea3f93403fcb80e3abf/src/helm/benchmark/annotation/aci_bench_annotator.py#L68-L81

@yifanmai, Could you please help me with how to configure the annotator models?

Sorry, but this is currently not supported by HELM.

I would like to add custom deployments for annotator models in the medium term, but I'm not actively working on this currently due to limited bandwidth.

@yifanmai, Can I work on this issue?
Repository: stanford-crfm/helm
Issue: Running MedHELM benchmarks on local
Root cause: Documentation Deficiency

Hi helm team!

I'm looking to run the MedHELM benchmarks locally, however noticed that several oof the benchmarks use data from a share drive with paths such as '/share/pi/nigam/data/medhelm/mimiciv_billing_codes/mimiciv_icd10.feather'.

I was wondering if there was a way to pull this data and run these benchmarks, sepcifically the MIMIC-IV billing code to ICD-10 classification benchmark.

Thanks!

cc @MiguelAFH

@MattLiCodes - Do you already have access to the MIMIC-IV dataset? MIMIC-IV is not available to the general public, thus HELM does not support downloading it and running the benchmark out-of-the-box. Please refer to the [MIMIC-IV paper](https://www.nature.com/articles/s41597-022-01899-x) for details.

Hey @yifanmai! I have access to the dataset from PhysioNet...how do I go about setting up the dataset for the benchmark?

@MiguelAFH Is there a paper that you guys are basing the benchmark on? I'm trying to figure out which fields in the MIMIC-IV dataset to use for evaluation. Thanks!

Never mind! I found the source. Thanks!

We used this repo to derive the dataset used for the paper: https://github.com/JoakimEdin/medical-coding-reproducibility
Repository: stanford-crfm/helm
Issue: Upstream error when using bert_score through unitxt
Root cause: External Dependency Breakage

When running a unitxt scenario that uses bert-score e.g. `unitxt:recipe=recipes.torr.qtsumm.concat.no_augmentation_5_demos,model=openai_gpt-4.1-nano-2025-04-14`, on some environments this happens:

```
  File "/path/to/site-packages/bert_score/scorer.py", line 98, in __init__
    self._model = get_model(self.model_type, self.num_layers, self.all_layers)
  File "/path/to/site-packages/bert_score/utils.py", line 255, in get_model
    model = AutoModel.from_pretrained(model_type)
  File "/path/to/site-packages/transformers/models/auto/auto_factory.py", line 543, in from_pretrained
    has_local_code = type(config) in cls._model_mapping.keys()
  File "/path/to/site-packages/transformers/models/auto/auto_factory.py", line 780, in keys
    mapping_keys = [
  File "/path/to/site-packages/transformers/models/auto/auto_factory.py", line 781, in <listcomp>
    self._load_attr_from_module(key, name)
  File "/path/to/site-packages/transformers/models/auto/auto_factory.py", line 777, in _load_attr_from_module
    return getattribute_from_module(self._modules[module_name], attr)
  File "/path/to/site-packages/transformers/models/auto/auto_factory.py", line 693, in getattribute_from_module
    if hasattr(module, attr):
  File "/path/to/site-packages/transformers/utils/import_utils.py", line 1851, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/path/to/site-packages/transformers/utils/import_utils.py", line 1865, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.timm_wrapper.configuration_timm_wrapper because of the following error (look up to see its traceback):
cannot import name 'ImageNetInfo' from 'timm.data' (/path/to/site-packages/timm/data/__init__.py)
```
Repository: stanford-crfm/helm
Issue: EMS metric fails when the inputs are identical homogenous images files
Root cause: Validation Gap

Reproducing:

```python
from helm.benchmark.metrics.vision_language.image_metrics import AnnotatedImageMetrics
from typing import Dict, Callable, Generator, Mapping, Tuple
from PIL import Image

class Cache(object):
    def get(self, request: Mapping, compute: Callable[[], Dict]) -> Tuple[Dict, bool]:
        response = compute()
        return response, False

metrics = AnnotatedImageMetrics(generation_type="webpage", metric_names=['ems'])
metrics._cache = Cache()

blank_image = Image.open('blank.png')
blank_image2 = Image.open('blank.png')

print(metrics.ems(blank_image, blank_image2))

```

```
% python reproduce-bug.py
Traceback (most recent call last):
  File "/Users/tcz/Dropbox/Reverse Browser/HelmTest/reproduce-bug.py", line 16, in <module>
    print(metrics.ems(blank_image, blank_image2))
  File "/Users/tcz/Dropbox/Reverse Browser/HelmTest/venv/lib/python3.9/site-packages/helm/benchmark/metrics/vision_language/image_metrics.py", line 478, in ems
    emd_raw, _ = self._cache.get(cache_key_numerator, compute_numerator)
  File "/Users/tcz/Dropbox/Reverse Browser/HelmTest/reproduce-bug.py", line 7, in get
    response = compute()
  File "/Users/tcz/Dropbox/Reverse Browser/HelmTest/venv/lib/python3.9/site-packages/helm/benchmark/metrics/vision_language/image_metrics.py", line 439, in compute_numerator
    return self.compute_block_emd_raw_wrapper(
  File "/Users/tcz/Dropbox/Reverse Browser/HelmTest/venv/lib/python3.9/site-packages/helm/benchmark/metrics/vision_language/image_metrics.py", line 528, in compute_block_emd_raw_wrapper
    emd_value = compute_emd_recursive(
  File "/Users/tcz/Dropbox/Reverse Browser/HelmTest/venv/lib/python3.9/site-packages/helm/benchmark/metrics/vision_language/emd_utils.py", line 337, in compute_emd_recursive
    cost = compute_cost_matrix_on_sig(
  File "/Users/tcz/Dropbox/Reverse Browser/HelmTest/venv/lib/python3.9/site-packages/helm/benchmark/metrics/vision_language/emd_utils.py", line 223, in compute_cost_matrix_on_sig
    sig1_reshaped = reshape_sub_sig_batch(
  File "/Users/tcz/Dropbox/Reverse Browser/HelmTest/venv/lib/python3.9/site-packages/helm/benchmark/metrics/vision_language/emd_utils.py", line 165, in reshape_sub_sig_batch
    num_channels: int = int(round(sub_sigs.shape[0] * sub_sigs.shape[1] / (num_patches * flat_patch_size)))
ZeroDivisionError: division by zero

```
Repository: stanford-crfm/helm
Issue: schema_medhelm.yaml generated with incorrect metrics (uses Accuracy instead of BERTScore as on public leaderboard)
Root cause: Configuration Error

**Description:**
When running the MedHelm schema generation, the resulting `schema_medhelm.yaml` includes **local** metrics (Accuracy) for benchmarks like ACI-Bench and MIMIC-RRS. However, according to the [[public MedHelm leaderboard](https://crfm.stanford.edu/helm/medhelm/latest/#/leaderboard)](https://crfm.stanford.edu/helm/medhelm/latest/#/leaderboard), these tasks should report BERTScore (F1), not Accuracy.

**Steps to Reproduce:**

1. Clone the MedHelm repo.
2. Run the schema generation script (e.g., `helm-summarize --suite <..> --schema-path ./schema_medhelm.yaml`).
3. Inspect the contents of `schema_medhelm.yaml`.


**Current `schema_medhelm.yaml` snapshot (excerpt):**

https://github.com/stanford-crfm/helm/blob/6d986dcdabaa96035dd55430db249ed0ad4f34cb/src/helm/benchmark/static/schema_medhelm.yaml#L225-L240

![Image](https://github.com/user-attachments/assets/07c9a8ac-11d7-45e2-b831-b54a14cc6191)

**Expected Behavior:**

![Image](https://github.com/user-attachments/assets/d53decad-7df7-4b8c-a5d6-bc403cf7381c)


@yifanmai 

Thanks for the report. The schema file in the repository has changed over time. We should publish a frozen schema snapshot that corresponds to the version used for v1.0.0.
Repository: stanford-crfm/helm
Issue: Use Python `logging` over `print` in hlog

Hi Team,

We're building some tools around HELM for internal use, and noticed the [`HierarchicalLogger`](https://github.com/stanford-crfm/helm/blob/df44dec56db0e69e712d2b063274c9d6bbe5164c/src/helm/common/hierarchical_logger.py#L6) uses regular `print` over python `logging`.

I think it might be useful to switch to using python `logging`, in case external tools call HELM and want to use their own log format. For example, for us, we have a custom override for python `logging`, that we need to ensure traceability & interpretation by our logging stack - currently we are forced to monkey patch HELM's HierarchicalLogger which is always something to be avoided where possible.

Happy to contribute a PR for this - let me know if you are interested! 


Yes I think that would be a great idea. Please open a PR for this, thanks!

Awesome - PR created at #3598 

Merged!

I made some minor tweaks to the log format in #3606; feel free to take a look and comment on that PR.

I'm also running into a separate issue. For some reason, the logger sometimes double-logs the same message (see example below). Any idea why this might be happening? 

```
2025-05-22 10:32:38,249 INFO           BasicMetric() {
2025-05-22 10:32:38,249 INFO             Parallelizing computation on 5 items over 4 threads {
2025-05-22 10:32:38,277 INFO             } [0.028s]                                                                                                                                                                                   
INFO:helm:        } [0.028s]
2025-05-22 10:32:38,278 INFO             Skipping computing calibration metrics because logprobs were not available.
INFO:helm:        Skipping computing calibration metrics because logprobs were not available.
2025-05-22 10:32:38,281 INFO           } [0.032s]
```

This behavior seems to start midway through a run. My hypothesis is that it has something to do with threading or forking, or some interaction with tqdm progress bars.

I'm using crfm-helm githash `04a93a622da61e504db3386efde05e243321b777`, Python 3.10.13, bash 5.0.17. I get this from running something like `helm-run --suite debug -r mmlu_pro:subject=all --model openai/gpt-4o-mini-2024-07-18 -m 5`

Ok that's strange - it implies that there are two handlers being added to `logging` that do printing; a fix to this in setupLogging might be to _clear_ the handlers first before calling `logger.addHandler()`? Something is slipping the default log handlers into the main logger...

Turns out the issue is that `RougeScorer` is imported and instantiated during metric computation, and `RougeScorer` uses absl logging [here](https://github.com/mim-solutions/rouge_score/blob/7f3310fabf5c4f310c592ac946f10f4a1c23087b/rouge_scorer.py#L40), and absl logging interferes with Python logging: https://github.com/abseil/abseil-py/issues/99

The workaround is to disable absl logging, as described in the upstream issue.
Repository: stanford-crfm/helm
Issue: test_gptneox_window_service is flaky
Root cause: External Dependency Breakage

Example failing test:

https://github.com/stanford-crfm/helm/actions/runs/15216281789/job/42802441863

example error:

```
FAILED src/helm/tokenizers/test_huggingface_tokenizer.py::TestHuggingFaceTokenizer::test_get_tokenizer_gptneox - ValueError: Converting from Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 'Qwen2Tokenizer', 'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer', 'RobertaTokenizer', 'RoFormerTokenizer', 'SeamlessM4TTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 'UdopTokenizer', 'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer', 'CodeLlamaTokenizer', 'GemmaTokenizer', 'Phi3Tokenizer']
```
Repository: stanford-crfm/helm
Issue: Proposal: Evaluation of LLM Simulated Student Coding
Root cause: Unimplemented Feature Gap

### Background

There has been a growing use of LLMs to simulate student behaviors in education, where simulated students have been shown effectiveness in assessment designing, teacher training and teaching-by-learning. With a LLM that can intentionally make mistakes on questions based on their given personas, their response on feeded question items would provide instructors some image of how they should redesign their items or instructions so that it is suitable for a given set of student characteristics. Although some studies have developed methods to elicit student-like responses by prompting or fine-tuning LLMs, LLMs’ capability to simulate students has not been formally evaluated, especially in computer science education. With simulated student coders, instructors can leverage the tool for themselves to train to teach students, and identify how they should design the programming questions to suit the level of their learners. This proposal aims to introduce novel evaluation metrics from HELM and benchmarks models based on response, mistake, and difficulty alignment.

### Dataset

The following dataset will be used in the evaluation

- [CodeInsights](https://huggingface.co/datasets/CodeInsightTeam/code_insights_csv) - Items and student response data from a foundational programming course, including submission timestamps, code responses, and question texts

### Main Evaluators

- Score Correlation: Compute the per-item accuracy (fraction correct) for simulated vs real students and measure their agreement
-- Metrics: Spearman Correlation
- Score Difference: Compute RMSE between simulated vs real per-item scores for each student
-- Metrics: RMSE
- Code Embedding Similarity: Generate code embeddings using CodeBERT and compute cosine similarity between simulated and real student code
-- Metrics: Cosine Similarity
- Semantic/Structural Similarity: Compute Abstract Syntax Tree (AST) edit distance to compare submitted code with simulated-student-generated code (RHUBARB)
-- Metrics: AST Edit Distance
- Edit Improvement Similarity: Measure the average AST edit-distance change between successive attempts (students re-attempting) for real and simulated students
-- Metrics: RMSE
- Mistake Distribution Similarity: Categorize errors using Mantyla et al.’s (2008) taxonomy of functional defects, then compare error distributions of real vs simulated students
-- Metrics: Kullback-Leibler Distribution
- Item Difficulty: Fit a Rasch model to both real-student and simulated data to estimate item difficulties, and compare estimates
-- Metrics: Spearman Correlation


Could you clarify what the task for the LLM is - is the task to solve these questions, or to grade the student transcripts, or to offer teaching suggestions to students, or something else?

The task is for LLMs to simulate student code - which means, we are testing if LLMs can imitate the style of students' codes. We have sample codes from different students, and we want to see if LLMs can grasp their style
Repository: stanford-crfm/helm
Issue: Proposal: Integrate Psychometric-Based Question Validity Tools into HELM
Root cause: Unimplemented Feature Gap

# Motivation
When evaluating language models, ensuring the validity and reliability of benchmark questions is crucial. Questions with flawed design, ambiguous wording, incorrect reference answers, or grading issues can distort evaluation outcomes. Applying psychometric tools to analyze model response patterns provides a systematic approach to identifying such problematic items. By flagging problematic questions—such as those with low discrimination, poor cross–item fit—we can improve the quality of benchmarks and make model comparisons more trustworthy.

# Implemented techniques
We implement four tools that encode structural assumptions about question behavior:

- Tetrachoric Correlation
Measures how often two questions are answered correctly by the same models. Unusually low or negative values flag items that do not align with a common underlying trait.

- Scalability Coefficient
Assesses how consistently each question ranks models by ability. Questions that violate the expected order, where stronger models should outperform weaker ones, are identified as problematic.

- Cronbach’s Alpha
Quantifies the overall internal consistency among all questions. By observing how consistency changes when a question is removed, we can pinpoint items that undermine this consistency.

- IRT Discrimination Parameter
Captures how sharply a question separates stronger from weaker models. Low or negative discrimination values indicate items that fail to distinguish model proficiency.

# Modification
Add several statistics algorithms as a standalone component of HELM. It can be a script in the scripts folder, a separate HELM command, or a tool in a separate repository / package that takes a dependency on HELM.
Repository: stanford-crfm/helm
Issue: Ensure that the case of a perturbed word matches the case of the original word
Root cause: Algorithmic Error

## Purpose

Some of the perturbations use dictionaries to substitute words. It would be good to ensure that the case of the word to be substituted matches the case of the original word.

@dtsip Should robustness also be making use of this for any perturbations/what do you think? There is a simple implementation in #364 from Dilara for this currently. 

Yes, good catch! I think some folks took care of it but let me take a proper pass.


Thanks @dtsip!
Repository: stanford-crfm/helm
Issue: Decide mode for NaturalQA and RAFT
Root cause: Unimplemented Feature Gap

For naturalQA (@ShibaniSanturkar) I think we want to focus on closedbook and openbook-longans. The wiki version will not fit in the context anyway. Right?

For RAFT (@bobbyy98) we might want to take a look at the results first and see what makes sense. In the end, we have 50 inputs per task so might have run everything already.

What's the status of this?

We discussed this a bit in person, and decided that we want to focus on openbook-longans and closedbook as Dimitris said. The wiki version has long wikipedia chunks which do not fit in the context. And it seems like currently, the codebase truncate inputs to the context length, causing us to lose even the question in the wiki version.

Trying to decide which subsets of RAFT to keep with @dtsip.

Fixed in https://github.com/stanford-crfm/benchmarking/pull/456 .

@dtsip closing this as a result
Repository: stanford-crfm/helm
Issue: Jury Score Configuration for Custom Models in HELM Suite
Root cause: Documentation Deficiency

I am using HELM to test custom models and see that Stanford's main page shows jury scores along with regular scores for some models.
I understand what jury scores are but need help turning on this feature for my custom models. Is there a setting or option I need to change to get jury scores in HELM?

In HELM, the jury is specified by the scenario implementer. You can refer to the implementation for Omni-MATH ([annotator](https://github.com/stanford-crfm/helm/blob/e115c06bb72bcbda44b8d37f25c6b80d4e97d521/src/helm/benchmark/annotation/omni_math_annotator.py), [metrics](https://github.com/stanford-crfm/helm/blob/e115c06bb72bcbda44b8d37f25c6b80d4e97d521/src/helm/benchmark/metrics/omni_math_metrics.py)) as an example of how to implement an LLM judge jury in code.

thanks
Repository: stanford-crfm/helm
Issue: Long passages are silently truncated
Root cause: Validation Gap

When the context is too long, it looks like we just drop what runs over (e.g., input 1 from [this run](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=natural_qa%3Amode%3Dopenbook-wiki)). This does not make a lot of sense for most tasks (e.g., for QA we are throwing away the question itself) and we should probably not query these points at all.

Tracking in https://github.com/stanford-crfm/benchmarking/issues/399
Repository: stanford-crfm/helm
Issue: MedHELM mimiciv_billing_code Data Access
Root cause: Documentation Deficiency

Hi HELM team!

I have access to the MIMIC-IV dataset and have checked out the original benchmark for MIMIC-IV Billing Code selection. I was wondering how you guys distilled the test set from the original benchmark to the 1000 examples used for the MedHELM benchmark.

Would be great to get some clarity on this and also was wondering if I could get access to the feather data file used for this benchmark. Thanks!

@MiguelAFH - could you assist here?

We used this [repo](https://github.com/JoakimEdin/medical-coding-reproducibility) to create the MIMIC-IV Billing Code dataset. As for the limit, we pass the entire dataset and specify to HELM that only the first 1000 entries will be included.