zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: fix formatting for language modeling
Root cause: Documentation Deficiency

https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=twitter_aae%3Ademographic%3Daa
Can we remove the 
 and also fix the unicode characters?

@percyliang 
1. I don't see why we should remove 
. Keeping it more faithfully reflects our task formulation - it is prepended to the first chunk of each document.
2. The unicode characters are from the original dataset. We use the dataset as is. This is not a decoding error. 
3. We should remove this scenario if we don't hear back from Brendan.

1. I think of this is as part of adaptation, not part of the actual scenario.
2. Presumably these Unicode characters can actually be rendered.
3. Let's ping him again then.

1. I think there are some discrepancies on this page. Input on this page is not actually input but raw instance text. Prediction is the actual prompt we send to (and receive from) the API.
2. I am not sure we want to decode those unicode characters in preprocessing because I don't find any evidence that previous work has done this. For the web interface, it may be doable.
3. Shall I write to him, or do you mean you want to follow up yourself?

Sent a comment over Slack - we should make a decision wrt TwitterAAE's inclusion. 

Once resolved, we can then address this, but I think both items 1 and 2 are pretty negligible at the any rate, so inclined to make this a P2. 

If 
 is inserted during adaptation, it shouldn't show up as part of the instance.

I currently cannot see anything on this page... I can't remember what it used to look like, either.

More generally, for language modeling scenarios, show log probabilities of tokens rather than the prediction.

Handled by #867
Repository: stanford-crfm/helm
Issue: Refactoring the Contamination Scenario
Root cause: Unimplemented Feature Gap

Hi, @yifanmai!

Sorry for the delayed response, but getting back to the discussion [[pull request](https://github.com/stanford-crfm/helm/pull/3627)], we would like to follow your suggestion to make the implementation more efficient and better integrated into HELM.


# Proposal

We are planning to refactor our solution and turn the contamination index computation into a scenario, instead of treating it as a nested HELM run. The proposed command for execution will be:

```
helm-run --run-entries contamination:dataset=bluex,model=ibm/granite-3.3-8b-instruct,strategy=ts_guessing_question_multichoice,language=pt --suite my-suite --max-eval-instances 1500
```

In this new format:

- contamination indicates that the model contamination will be computed for a given scenario;
- dataset  specifies the scenario being used;
- strategy defines the contamination method applied;
- model indicates the model to be evaluated;
- language specifies the prompt language.

# Impact

No existing files in HELM will need to be modified — only new files will be added, following the same pattern used for adding new scenarios. The following files will be included:

- contamination_run_specs.py
- contamination_scenario.py
- test_contamination_scenario.py

However, since the contamination scenario needs to access one of HELM’s datasets to perform word/option masking, it will also require adding a few auxiliary files:

- contamination_utils.py
- contamination_base.py
- prompt_translations.py
- ts_guessing_question_based.py
- ts_guessing_question_multichoice.py

These additional files ensure the full functionality of the contamination computation.
Thus, according to the proposal, contamination becomes a scenario, and the specified dataset acts as a meta-scenario.

The new file structure is organized as follows:

```
src/
└── helm/
    ├── benchmark/
        ├── run_specs/
            contamination_run_specs.py
        └── scenarios/
            └──⭐ contamination/
                contamination_scenario.py
                contamination_utils.py
                contamination_base.py
                prompt_translations.py
                ts_guessing_question_based.py
                ts_guessing_question_multichoice.py
                test_contamination_scenario.py
 ```
We would like to confirm whether this proposal is aligned with HELM’s architecture and whether you think we can proceed with refactoring the module in this direction.


Thanks. At a high level, this proposal looks good to me.
Repository: stanford-crfm/helm
Issue: MMLU evaluation fails with Qwen3
Root cause: Interface Contract Mismatch


I'm trying to evaluate Qwen/Qwen3-30B-A3 on MMLU. I have my model deployed on vLLM on my local machine.


When I run the task normally like:
yaml{description: "mmlu:model=Qwen/Qwen3-VL-30B-A3B-Instruct,eval_split=test,subject=clinical_knowledge", groups: ["mmlu_clinical_knowledge"], priority: 1}

It either gives:

KeyError: 'choices'
File "/helm/clients/openai_client.py", line 354, in _make_chat_request
    for raw_completion in response["choices"]:
KeyError: 'choices'
Or it completes the evaluation but does not show anything in raw predicted text .


So I added a parameter:
yaml{description: "mmlu:model=Qwen/Qwen3-VL-30B-A3B-Instruct,eval_split=test,subject=clinical_knowledge,increase_max_tokens=2048", groups: ["mmlu_clinical_knowledge"], priority: 4}
And ran the evals. It did run the evaluations and the answer is present in the generated text, but it was not able to detect it.


Do you have any solution to make it work?

these below are the inference with the latter config 

<img width="1512" height="782" alt="Image" src="https://github.com/user-attachments/assets/9914fdcd-5f6e-409e-9c29-3d877dc9c04c" />
<img width="1512" height="702" alt="Image" src="https://github.com/user-attachments/assets/345c403b-9bcd-43ca-9ba6-c257a47b057a" />
Repository: stanford-crfm/helm
Issue: Stop sequences are not visualized when new line
Root cause: Validation Gap

I have not checked if there are other errors, but I assume this is the same issue @dilarasoylu saw previously when html is rendered. As a result, most adapters show a stop_sequences field that looks blank. At least for this, I think we definitely want to be able to indicate the real stop sequence is a newline character (which is shown in the json).
Repository: stanford-crfm/helm
Issue: Highlight the standard metrics for scenarios in run view as well as SOTA numbers 
Root cause: Unimplemented Feature Gap

This was an issue raised due to the 5/10 benchmarking bug bash. Resolving it should be part of the broader effort to improve the visualization. 

Moving forward, we may also want to add numbers for the scenario referring to standard results (i.e. Should we also include reference values for the reference metrics from papers etc. so that we can see what to expect?). These could also include human performance if known, though probably we want caveating. 

We should add these to the docstring of the scenario: what are the prior art, and what are the differences between prompting strategies.

@dilarasoylu So we now have the standard metric for every scenario and this is already clearly addressed in the tables.

So I think what would remain is:
1. In the view of a run, highlight this metric. Options would be to put it at the top of the metrics list or highlight with some kind of color.
2. Add numbers related to prior work/etc. I am not quite sure this should go in the scenario file as Percy suggests, since it won't be timeless (of course we could timestamp when we added it) and likely will quickly change (and I doubt we will have the energy to maintain this/there are things like paperswithcode that are designed to do this anyways), as opposed to somewhere else that relates to the frontend (as opposed to the scenarios/deeper in the core benchmarking code). We should get these numbers of the paper as the higher priority and find some means to put on website if we already gather them.

As a refresher, the TODOs are:

- [ ] In the run view, highlight the standard metric. We also have this information in `schema.yaml`, so the TODO is to add something to identify it in the metric list. I think it tends to be near the top currently anyways.
- [ ] Visualize the SOTA numbers/other comparisons. I am not really sure where on the website this should go visually, but I will 
collect this information as part of #637 in addition to the fact we will need it anyways in the paper.  

Yeah, I think we can leave off the webpage.  It would be good to have a link to the original paper from the scenario though (we can put that in schema).  Closing this for now.
Repository: stanford-crfm/helm
Issue: Much higher WER for gpt4o-mini, compared with published leaderboard
Root cause: External Dependency Breakage

This is intended as a question/request for advice rather than an issue. Thanks 🙏 

I am trying to use HELM as a  benchmark for audio models, and I've been trying to reproduce the HELM [leaderboard](https://crfm.stanford.edu/helm/audio/latest/#/leaderboard) results for the `librispeech-WER` scenario with `GPT-4o mini Audio (Preview 2024-12-17)`

Running HELM locally with the same dataset, number of samples etc I consistently see WER ~ 0.36 compared with 0.163 shown in the published leaderboard.

My current theory is that openai safety filters have changed, so that even using the same checkpoint we now get many more refusals ( things like "I'm unable to assist with that"), and this pushes up the mean WER per sample a lot.

**So questions are: am I right that this metric has gone up for the same checkpoint? And does my theory re filter changes make sense?**

For reference, this is how I run the eval to reproduce the leaderboard for this scenario + model

```
#!/usr/bin/env bash
set -euo pipefail

export SUITE_NAME=audio_leaderboard
export MODELS_TO_RUN=openai/gpt-4o-mini-audio-preview-2024-12-17
export RUN_ENTRIES_CONF_PATH=src/helm/benchmark/presentation/run_entries_audio.conf
export SCHEMA_PATH=src/helm/benchmark/static/schema_audio.yaml
export NUM_TRAIN_TRIALS=1
export MAX_EVAL_INSTANCES=1000
export PRIORITY=1

helm-run \
  --conf-paths "$RUN_ENTRIES_CONF_PATH" \
  --num-train-trials "$NUM_TRAIN_TRIALS" \
  --max-eval-instances "$MAX_EVAL_INSTANCES" \
  --priority "$PRIORITY" \
  --suite "$SUITE_NAME" \
  --models-to-run "$MODELS_TO_RUN" \
  --groups-to-run librispeech

helm-summarize --schema "$SCHEMA_PATH" --suite "$SUITE_NAME"

helm-server --suite "$SUITE_NAME"
```

Note that we'd need to run this with `disable-cache` if running it multiple times, otherwise you'd see the cached results the next time. 

## Addendum: evidence for increased number of refusals

The reason for my theory is that I see a much higher rate of refusals in the results:

- Leaderboard: 38 refusals; avg length of refusal 7.89 words; median 7; min 7; max 18.
- Local run: 62 refusals; avg length 15.0 words; median 21; min 7; max 24.

So the number of refusals has increased, which would push up mean WER, and the no. of words in  the refusal text itself has increased, which would compound the same effect.
Repository: stanford-crfm/helm
Issue: Visualizing prefixes for instances
Root cause: Unimplemented Feature Gap

For all scenarios, the interface should include any prompting (e.g. prefixes like "Question:" ) so the user knows this is a prefix, possibly with different formatting (e.g. highlighting).

For multiple choice adapter, it should be indicated the model only needed to produce the correct letter (i.e. it does not generate the whole sequence). It might be better to do:

```
Question: Who was sitting outside?

Answer: 
A. Percy
B. Arthur
C. Henry

{Prediction from model} 
A.
```

The exact prompt can be examined if you click on the prediction.  The prompt isn't available on the page listing the instances because it's too involved.
Repository: stanford-crfm/helm
Issue: estimate the number of tokens accurately for the quota
Root cause: Unimplemented Feature Gap

We need to know the number of tokens in `Service::make_request` to keep track of the quotas accurately.  Need to figure out how this is actually computed since each service presumably uses a different tokenizer (Jurassic has larger tokens for example).

Pricing documentation:

https://openai.com/api/pricing/#faq-completions-pricing
https://studio.ai21.com/docs/calculating-usage/

Tokenizer:

https://community.openai.com/t/how-do-i-calculate-the-pricing-for-generation-of-text/11662/5
https://api.ai21.com/studio/v1/tokenize - request: {text: "some text"} payload: [tokens]

From https://community.openai.com/t/how-do-i-calculate-the-pricing-for-generation-of-text/11662/5 , "we’re planning to implement a tokenizer into the client to make it easier. And adding an easy cost estimator into the api would be a great addition too."
Repository: stanford-crfm/helm
Issue: if show all the msmarco datasets, MRR@10 shows up many times
Root cause: Documentation Deficiency

Have not looked, but my guess is this is just the general behavior when many things are displayed at the same time? If so, feel free to close. 


@okhat 

@rishibommasani I don't actually understand this, sorry. Could you rephrase?

@rishibommasani I think this happens with other scenarios as well, seems to be a UI issue. I can take a look with other p2 issues.

Closing for now since seems more general; may need a different visualization issue not addressed by recent visualization PR.
Repository: stanford-crfm/helm
Issue: Metrics for anthropic models
Root cause: Documentation Deficiency

1. Estimated number of tokens for Anthropic is 0 -> I assume this is because we don't have to pay for tokens, but we should clarify this (maybe by changing the name of the metric). If we need to change the metric name/description, please add a note in https://github.com/stanford-crfm/benchmarking/issues/384. 
2. Log probs are 0 for Anthropic -> I assume this is because we don't have access to probabilities, but we should clarify this (maybe by outputting N/A or something)

Probably should just make these `None` and hope this doesn't complicate the downstream handling of this code.

@rishibommasani Is this still an issue?  We don't want things on the website to be misleading.

1. We changed the metric name to mention it is about cost. Currently no description in schema (i only added descriptions for things we needed). Let me add a P2 issue for myself to add descriptions for every metric, and then beyond that I think this is fine.
See https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=boolq%3Amodel%3Danthropic_stanford-online-all-v4-s3%2Cdata_augmentation%3Dcanonical#metrics for an example.
2. Addressed

Superseded by #831.
Repository: stanford-crfm/helm
Issue: Training costs should be standardized when unknown
Root cause: Validation Gap

If we don't know the training cost, we should consistently output the same thing (seems like we currently sometimes put 0 and sometimes ?

<img width="2126" alt="image" src="https://user-images.githubusercontent.com/47439426/167918847-06fa285f-4a2e-4f76-98a5-d162efa7c36e.png">
 

Seems like a UI thing? The logs have this as `None` uniformly when unknown: not sure where the 0 or ? come from.

For example: https://crfm-models.stanford.edu/static/benchmark_output/runs/raft:subset=banking_77,model=openai_curie/metrics.json.

@deepakn94 Is this resolved (and can you close if so)? 

No, it's not. No idea why the UI reports a 0 here: https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=raft%3Asubset%3Dade_corpus_v2%2Cmodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical.

Not seeing this issue any more (i.e. just seeing empty cell when missing), so I think this is good (note: only spot checked in a few places).
Repository: stanford-crfm/helm
Issue: Show confidence intervals and variance over random seeds
Root cause: Unimplemented Feature Gap

Probably can do as part of the broader visualization effort. In the 3 tensor world, we actually end up storing the mean and std dev. across runs

@percyliang I think we should just show this as a (+/- ...) in small font for every value in the tabular value, with no changes in the runs view? 
And then have this also in the paper (@dtsip we will need to add support for this in the _website table -> LaTeX table code_)?

Since I do think it is really worth making the persistent point that if model performance is highly unstable as a function of randomness, this is a serious problem that we should be devoting attention as the field becomes increasingly enamored with few-shot learning. 

I'm so sorry I lost track of this. I'll write things up this week and send it to you, and then we can quickly implement it. If there's an urgent deadline please let me know and I can drop other things and work on this! Sorry again!

Almost done writing things up… sorry for the delay.

On Mon, Jan 9, 2023 at 11:06 PM Percy Liang ***@***.***>
wrote:

> Assigned #403 <https://github.com/stanford-crfm/helm/issues/403> to
> @AnanyaKumar <https://github.com/AnanyaKumar>.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/stanford-crfm/helm/issues/403#event-8188753945>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAOGZQUTNYHUU27LAR5RZN3WRUC5RANCNFSM5VVZKX2A>
> .
> You are receiving this because you were assigned.Message ID:
> ***@***.***>
>
Repository: stanford-crfm/helm
Issue: Tracking for summarization scenarios
Root cause: Documentation Deficiency

Here are some action items from the bug bash on 5/10
- [x] Figure out which temperatures we are using and document why in this thread.
- [x] The models sometimes terminate generation too quickly, possibly because of the "}". What do we want to do?
- [x] "The ground truths are unreliable, so automated eval may not suffice." For now, I think we should stick with it but we can discuss if we need something else. 
- [x] We may want to compute abstractivity/extractivity metrics (e.g. the stuff from Max's Newsroom paper with fragments, which I also use in my paper/has been reused a bunch).

**In general, we should document anything we want to appear in the paper here**

@esdurmus @fladhak 

Here are some additional items (@fladhak @esdurmus):

**CNN/DM**
- [ ] There is a stray  “}” after every answer and “{“, “}” in the input -> **Should be in a new output_suffix**
- [ ] **Should there be <start> <end> tags?**
- [ ] How are we accounting for the brackets while computing exact match?
- [ ] AI-21 model always (and some other models sometimes) ends in the middle of a sentence, possibly because it never generates the stop sequence until it reaches max length
- [ ] The first prompt example is a non-article: `Download PDF maps related to today's show …`

**XSum:**
- [ ] The context seems to be truncated for some examples. Does the answer lie in the span in each case?
- [ ] The typical eval doc length for cnndm is much longer than the prompt examples. Is that a cause for concern?
- [ ] Second in-context example has an empty summary.
- [ ] Is it possible to only include “good” passage / summary pairs in the training set? For examples, those with non-empty bodies, those of representative length etc.
- [ ] Should the length of the in context examples matter? For example, if the test input is long, should the in context examples also be long?
- [ ] logprob on test is much higher for the *-001 models
- [ ] max tokens should be set higher than 128, and we should expect the model to stop with }
- [ ] DaVinci loves saying “All images are copyrighted”
- [ ] temperature doesn’t seem to be doing much - should we still keep it?
- [ ] There are quite a few inputs where the models just output “All images are copyrighted.” as the answer, regardless of the context. This seems to be the answer to the latest in context example, which seems to be copied over.


@fladhak can you verify the above is addressed

Friendly ping!

I think the core concerns are addressed; there are some minor points, but I think we are good wrt the key things about lengths, formatting, valid summaries, etc. 

Yes, these are addressed for the most part. I just forgot to mark it in this issue. Sorry!
Repository: stanford-crfm/helm
Issue: Some MMLU instances are underspecified
Root cause: Documentation Deficiency

Some instances involve graphs, etc. (e.g. economics). I am not sure if we want to do anything, but just noting for now. 

Hi! 
I just double checked the original MMLU data/code from Hendrycks, and they use the data as it is for evaluating GPT3, etc. So I guess we don't need to do anything with it for now?

Sounds good
Repository: stanford-crfm/helm
Issue: ICE FileNotFoundError
Root cause: Environment Incompatibility

"ice:gender=F": {status: "READY"}
"ice:gender=M": {status: "READY"}


FileNotFoundError: [Errno 2] No such file or directory: 'benchmark_output/scenarios/ice/ICE-CAN/Corpus/W2B-037.txt'

Is there a full log for this error? This seems to be a case of a mismatched file extension (the named file is stored as a .TXT (with capital letters)), though the code has long been updated to handle this case. Dry-runs on a local machine for these two runs did not run into any problems either.

I think this is fixed now. Thanks @nathankim7!
Repository: stanford-crfm/helm
Issue: Don't include all the dev packages in setup.py
Root cause: Architectural Constraint

```
def get_requirements(path: str):
    # TODO: don't include all the dev packages
    requirements = []
    for line in open(path):
        if not line.startswith('-r'):
            requirements.append(line.strip())
    return requirements
```

Verify package builds correctly.

@dilarasoylu Could you handle this alongside #356? No hurry/both should be fairly low priority, but seem reasonable to bundle together. 

It made sense to separate out the dev packages when we only had a few dependencies in requirements.txt at the beginning of the year. We now have many more dependencies, and the dev packages do not take long to install (5 seconds) compared to installing everything (1 minute 5 seconds).
Repository: stanford-crfm/helm
Issue: Sort the predicted outputs
Root cause: Algorithmic Error

In basic_metrics.py:

```
        # TODO: Sort the predictions, or take them from the top tokens of the first completion
        preds = [completion.text.strip() for completion in request_state.result.completions]
```

@teetone Not sure I understand what this meant, but can we close? Will make a P3 for now. 

This affects accuracy@k metrics, since by default the completions might not be sorted.  We should look into this to make sure our metrics aren't wrong.

@michiyasunaga Can you look into this, as one of the few places we use such a metric is for WikiFact (though we are not currently doing this afaik; https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/run_specs.py#L385)? 

@dilarasoylu may also be able to support if this is related to IR metrics. 

Hi folks! I have looked into this issue. 

Indeed, it seems that the default list of predictions returned here is not sorted https://github.com/stanford-crfm/benchmarking/blob/29b8896596f56a4ae68c836e7153e1d97734c926/src/benchmark/metrics/basic_metrics.py#L449
For example, in the "Gordon Freedman is an instance of" instance of this WikiFact scenario (https://crfm-models.stanford.edu/static/benchmark_output/runs/latest/wikifact:k=5,subject=P31,model=openai_text-curie-001/scenario_state.json), the returned items in result.completions ["music critic", "figure in sports", "person", "human", "author"] are not sorted. The current code takes "music critic" (logprob=-8.4) as top 1 prediction, but I think it should be "person" (logprob=-0.91) according to the logprob score.

To resolve this, I have added a line to sort predictions by logprob in basic_metrics.py and made a PR here: https://github.com/stanford-crfm/benchmarking/pull/834

@michiyasunaga closing this, but please confirm that's right. Thanks once again!