zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Remaining run spec cleanup
Root cause: Architectural Constraint

Capturing remaining feedback from #2321:

- [ ] @percyliang regarding `common_metric_specs.py`: Some of these are really specific to classic HELM, and we'd ideally put as much stuff into `classic_metric_specs.py`.  I think `common_metrics_specs.py` should be the bare minimum that we need.
- [ ] Split out some run specs from `classic_run_specs.py` into `extra_run_specs.py`, so that `classic_run_specs.py` only contains things actually used in HELM Classic.
Repository: stanford-crfm/helm
Issue: Front-end should provide warning after failing
Root cause: Unimplemented Feature Gap

The loading icon just goes infinitely. We should probably have some set amount of retries after which we surface an error message to users.

I have spent a few times accidentally staring at it waiting for it to work 

Agreed... we should make sure the error handlers work correctly.

Closing because this seems to work for me - please reopen if there is a specific page that this breaks on.

@yifanmai to be more specific, this involves the error where fetching schema or something else in benchmark_output fails for some reason (i.e. the URL being wrong or something else). The front end doesn't say anything, and you have to go to developer console to see if something is wrong. In these cases, most pages just show a loading sign for a while. 

I think this is fine for now, as developers should be able to figure out the issue, but it would be nice to show users/developers if something went wrong in fetching data.
Repository: stanford-crfm/helm
Issue: GitHub Action workers running out of disk
Root cause: Configuration Error

Getting errors `System.IO.IOException: No space left on device : '/home/runner/runners/2.313.0/_diag/Worker_20240226-203428-utc.log'` on most runs, like [this one](https://github.com/stanford-crfm/helm/actions/runs/8055245658).

Seems to be working now after we switched to `setup-python` for cache management.
Repository: stanford-crfm/helm
Issue: Standardize max_tokens vs stop sequence for all scenarios
Root cause: Unimplemented Feature Gap

See: https://docs.google.com/document/d/1gOMf86Jo5TUOliMVnWY-BNTZd2XStdI3N8p-ZkZoX8Y/edit

did that for robustness-related ones in https://github.com/stanford-crfm/benchmarking/pull/257/

TODO: one more pass over all scenarios

Closing since we have done 2 passes
Repository: stanford-crfm/helm
Issue: Standardize the wording and formatting of text in prompts/queries
Root cause: Unimplemented Feature Gap

1. You will need to look at `run_specs` for most of this. For example, "Answer: " vs "Answer:" vs "Answer" vs "answer"
2. You will need to look at the actual scenario files for standardizing this in some settings. For example, every QA dataset should have a standardized means for combining the context with the question (if there is both context and question) for every instance. 

See: https://docs.google.com/document/d/1gOMf86Jo5TUOliMVnWY-BNTZd2XStdI3N8p-ZkZoX8Y/edit

took a pass over the robustness scenarios and standardized that in https://github.com/stanford-crfm/benchmarking/pull/257/ 

Non-robustness scenarios are covered in #267.
Repository: stanford-crfm/helm
Issue: Should RealToxicityPrompts use `include_generative_harms_metrics`?
Root cause: Configuration Error

I understand the Core Scenarios do not do this because you want to streamline, but for `real_toxicity_prompts` should the function call of `get_generative_harms_metric_specs` also include `include_generative_harms_metrics=True` . 

https://github.com/stanford-crfm/helm/blob/481d12e0360366ac949018242a8f4141fb283887/src/helm/benchmark/run_specs/classic_run_specs.py#L349-L355

Currently,  the toxicity metrics are not run for `real_toxicity_promps` scenario. If the `include_generative_harms_metrics` flag should stay at the default of false, what is the best way for a user to run with `include_generative_harms_metrics` from the CLI.

Thanks for catching this. I opened a pull request #2530 to fix this.
Repository: stanford-crfm/helm
Issue: Fix linking to raw runs for sub-group
Root cause: Algorithmic Error

When in the leaderboard and viewing a specific scenario, clicking on a key metric for that scenario should take you to it's raw runs. Instead it only sends you to all the raw runs for the model, not necessarily filtered by the scenario as well.

This only occurs when using the dropdown to navigate through scenarios

Closing as https://github.com/stanford-crfm/helm/issues/2806 is a more detailed version of this issue
Repository: stanford-crfm/helm
Issue: Refactor out Request validation to the Request class
Root cause: Architectural Constraint

We re-implement Request validation in various places e.g. [here](https://github.com/stanford-crfm/helm/blob/6124c4a2c4a24b79b297f0fa6f0c3d936bdd9123/src/helm/clients/vision_language/open_flamingo_client.py#L90-L91) and [here](https://github.com/stanford-crfm/helm/blob/6124c4a2c4a24b79b297f0fa6f0c3d936bdd9123/src/helm/clients/openai_client.py#L137-L138). We should refactor it out and move it to the Request class.

Things to validate:

- Exactly one of `Request.messages`, `Request.prompt` or `Request.multimodel_prompt` should be set
- Every `MediaObject` of type "text" has the `text` field set
- Every `MediaObject` of type "image" has the `location` field set

I have taken a stab at it [here](https://github.com/stanford-crfm/helm/pull/2686)
Repository: stanford-crfm/helm
Issue: Set decoding parameters for every scenario
Root cause: Configuration Error

Temperature:
0 - Any scenario that is trying to get argmax (e.g. all QA scenarios) where diversity is undesirable. (Since many of these belong to robustness, knowledge, or reasoning teams - @dtsip @michiyasunaga @tonywu95 please verify these are right/have your teams check.)
0.5/1 - Any scenario that has nontrivial generation (summarization, prompt-based toxicity eval; @fladhak, @ryanachi)
TBD - Anything for interaction

No usage of parameters we have for some models but not others (e.g. frequency penalty - I don't believe this is an issue). 

took a pass over robustness and made sure everything is 0.0 with https://github.com/stanford-crfm/benchmarking/pull/257/

Keeping this issue open so we can double check and do another pass.

Closing since we have done 2 passes
Repository: stanford-crfm/helm
Issue: Some scenarios are mismatched between the documentation and blog post of HELM (Instruct)
Root cause: Documentation Deficiency

I check the [blog post](https://crfm.stanford.edu/2024/02/18/helm-instruct.html) and found there are 7 supported scenarios：
<img width="881" alt="1709753879390" src="https://github.com/stanford-crfm/helm/assets/8592144/80ea0324-025b-4ae0-bba0-f96d047ded3a">
However, `Anthropic Red Team` is not found in the [leaderboard documentation](https://crfm.stanford.edu/helm/instruct/latest/#/scenarios) in the meanwhile:
<img width="1850" alt="1709753565646" src="https://github.com/stanford-crfm/helm/assets/8592144/1ec596b8-cb7c-4385-ba0f-053cf60c522d">
Actually, this scenario is merged into the [Anthropic RLHF scenario](https://crfm.stanford.edu/helm/instruct/latest/#/groups/anthropic_hh_rlhf), as shown in:
<img width="977" alt="1709823732158" src="https://github.com/stanford-crfm/helm/assets/8592144/8b3f443a-a3c4-45bb-862b-62d62e07f838">




This does indeed seem to be an error in the HELM Instruct blog post. @YianZhang What do you tink?

I agree that red team should have been in its own scenario, but I don't think we intend to revise the released HELM tables, so I will close this issue.
Repository: stanford-crfm/helm
Issue: logprobs were not available
Root cause: Unimplemented Feature Gap

The result do not have the logprobs, then i can not computing it in my case. like this https://github.com/tatsu-lab/opinions_qa



My understanding is that OpinionsQA in HELM is currently unmaintained, and that it is not possible to reproduce the paper with HELM easily. In any case, most recent models in HELM do not provide logprobs, which are required for computing OpinionsQA's metrics.

cc @ShibaniSanturkar for OpinionsQA

Thank you for your response, I will check the original chatgpt API parameter.

Best,
JF
________________________________
From: Yifan Mai ***@***.***>
Sent: Friday, March 29, 2024 16:47
To: stanford-crfm/helm ***@***.***>
Cc: Zhu, Jianfeng ***@***.***>; Author ***@***.***>
Subject: EXT: Re: [stanford-crfm/helm] logprobs were not available (Issue #2444)


My understanding is that OpinionsQA in HELM is currently unmaintained, and that it is not possible to reproduce the paper with HELM easily. In any case, most recent models in HELM do not provide logprobs, which are required for computing OpinionsQA's metrics.

cc @ShibaniSanturkar<https://github.com/ShibaniSanturkar> for OpinionsQA

—
Reply to this email directly, view it on GitHub<https://github.com/stanford-crfm/helm/issues/2444#issuecomment-2027721486>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHHGS7CDTYOKQ7ATTXB3ZZLY2XHPFAVCNFSM6AAAAABEMAYN2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRXG4ZDCNBYGY>.
You are receiving this because you authored the thread.Message ID: ***@***.***>

CAUTION: EXTERNAL SENDER Do not click any links, open any attachments, or REPLY to the message unless you trust the sender and know the content is safe.
Repository: stanford-crfm/helm
Issue: Upgrading openai breaks the sqlitedict cache
Root cause: External Dependency Breakage

```
Traceback (most recent call last):
  File "/home/yifanmai/oss/helm/venv/lib/python3.8/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/yifanmai/oss/helm/src/helm/clients/auto_client.py", line 111, in make_request_with_retry
    return client.make_request(request)
  File "/home/yifanmai/oss/helm/src/helm/clients/openai_client.py", line 300, in make_request
    return self._make_chat_request(request)
  File "/home/yifanmai/oss/helm/src/helm/clients/openai_client.py", line 172, in _make_chat_request
    response, cached = self.cache.get(cache_key, wrap_request_time(do_it))
  File "/home/yifanmai/oss/helm/src/helm/common/cache.py", line 209, in get
    response = key_value_store.get(request)
  File "/home/yifanmai/oss/helm/src/helm/common/key_value_store.py", line 61, in get
    result = self._sqlite_dict.get(key_string)
  File "/usr/lib/python3.8/_collections_abc.py", line 660, in get
    return self[key]
  File "/home/yifanmai/oss/helm/venv/lib/python3.8/site-packages/sqlitedict.py", line 245, in __getitem__
    return self.decode(item[0])
  File "/home/yifanmai/oss/helm/venv/lib/python3.8/site-packages/sqlitedict.py", line 102, in decode
    return loads(bytes(obj))
ModuleNotFoundError: No module named 'openai.openai_object'
```

Unfortunately there isn't a way to work around this besides deleting the cache and starting over.
Repository: stanford-crfm/helm
Issue: Do a run with ~200 instances, all scenarios, many models
Root cause: Unimplemented Feature Gap

Blockers:
1. Resolve P1 issues
2. Make sure the conf file is fully up to date. 

@teetone I can do the run/handle this, but I think some small details are:
(a) We will need to settle on where we want to store things like NewsQA and ICE, i.e. things we cannot just download and will need a local copy for. 
(b) I think we can always do this runs from the NLP cluster in the mercury region, so we should make sure all the data is loaded there so downloading is not the bottleneck and it is just the cost of getting the predictions from the various models. 

So, maybe you can let me know how (a)/(b) are going, and then can un-assign after? Or we could make separate issues, but idk if these are problems in the first place (I basically just wanted you to sanity-check me)

Run completed April 13, 2022: https://crfm-models.stanford.edu/static/benchmarking.html
Repository: stanford-crfm/helm
Issue: HumanEval Code Scenario- From where do we get input_output.json
Root cause: Documentation Deficiency

Thanks in advance
Please reply on kapilmayank6@gmail.com

HumanEval does not appear to have a `input_output.json`. Are you referring to [APPS's `input_output.json` here](https://github.com/stanford-crfm/helm/blob/5c8c19460eb2ea3a4da4cd51a67a304ca002f138/src/helm/benchmark/scenarios/code_scenario.py#L158)?

> Please reply on [kapilmayank6@gmail.com](mailto:kapilmayank6@gmail.com)

We prefer to keep discussions on GitHub rather than email, so that there is a public record that can be referred to by other users.

Closing due to staleness. Feel free to reopen if you have more questions.
Repository: stanford-crfm/helm
Issue: Multiple choice joint adapter should not strip trailing whitespace from the prompt
Root cause: Algorithmic Error

This leads to prompts like:

```
Question: What is 1 + 2?
A. 3
B. 4
Answer:
```

where there is no space after `Answer:` This causes most models to generate a space before the actual answer, which is not desirable.

I remember that back in the day, trailing spaces caused openai/davinci to do weird stuff because of differences in tokenization due to having the space or not.  Is this changed with the new models?  Why change this now - is something broken?

The lack of trailing space at the end of the prompt causes models that use Llama 2 as a tokenizer to break. The Llama 2 tokenizer considers " A" (with a leading space) to be two tokens:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokens = tokenizer.tokenize(" A")  # ['▁', '▁A']
```

which means the model will get 0 on scenarios that use `get_multiple_choice_adapter_spec()` such as MMLU, because they have `max_tokens = 1`, which means the model will return only whitespace, and the letter gets truncated.
Repository: stanford-crfm/helm
Issue: train_trial_index
Root cause: Architectural Constraint

@rishibommasani and I were discussing earlier today (in the context of line 40 of https://github.com/stanford-crfm/benchmarking/blob/bbq_bold_civil/src/benchmark/bbq_metrics.py):

@teetone should we iterate once per `train_trial_index` in custom `Metric` files? For instance, see line 28 of https://github.com/stanford-crfm/benchmarking/blob/master/src/benchmark/commonsense_qa_metrics.py. Or should the logic for this be implemented elsewhere?

Is this still relevant or can we close this?

I'm assuming this is not currently relevant, but I'll defer to @rishibommasani @teetone.

What's the status of this?
Repository: stanford-crfm/helm
Issue: non temperature-based decoding
Root cause: Unimplemented Feature Gap

@teetone — another discussion @rishibommasani and I had this morning:

How do we want to handle cases where the authors evaluated a (generation) dataset using a top-p_decoding strategy, rather than a temperature-based one? For instance, see Table 8 (page 17) of [RealToxicityPrompts](https://arxiv.org/pdf/2009.11462.pdf)

@ryanachi Let's not bother with this; please ask someone like Lisa or John Hewitt for guidance on Slack (and include me), so we can find a good temperature choice rather than nucleus sampling, since we definitely won't be able to do nucleus efficiently for all models. This should be resolved by Monday and you should be responsible for updating the adapter_spec with the temperature we decide on. 

(just asked on Slack)
Repository: stanford-crfm/helm
Issue: Leaderboard downloadability
Root cause: Unimplemented Feature Gap

Continue on https://github.com/stanford-crfm/helm/issues/2322:
I have a few questions on the downloadable format of the existing leaderboards:

1. This zip files only contains all the evaluation results that are not the one shown in the leaderboards. Is there any manner to download the leaderboard data only in json or csv format?
2. How about HELM (Lite), HELM (Instruct), and HEIM?

With respect to the tables, for now you can copy the tables values into a Google Sheet or Excel document (by dragging across the table, copying and pasting). This also preserves the links to corresponding runs/predictions. If there is significant demand we can prioritize adding back JSON downloading in the newest leaderboards. 

You can also currently download the full JSON for any run/prediction set by going to the predictions page. 

> With respect to the tables, for now you can copy the tables values into a Google Sheet or Excel document (by dragging across the table, copying and pasting). This also preserves the links to corresponding runs/predictions. If there is significant demand we can prioritize adding back JSON downloading in the newest leaderboards.
> 
> You can also currently download the full JSON for any run/prediction set by going to the predictions page.

Thanks, does the prediction JSON download include HEIM, HELM (Instruct), and HELM (Lite)?

Currently, the only supported data downloads are:

1. Downloading the zip, as discussed in #2322 - this _only_ applies to Classic v0.3.0
2. Downloading raw data from the GCS bucket (instructions [here](https://github.com/stanford-crfm/helm/blob/cace4fbdea241e47549eced910d3072c6714eb4b/docs/downloading_raw_results.md)) - this applies to any version of Classic, Lite, HEIM or Instruct

If your use case does not fit into either of these, you can write a your own script to download the web tables or the JSON files.

> Currently, the only supported data downloads are:
> 
> 1. Downloading the zip, as discussed in [How to download the leaderboards as csv or json format? #2322](https://github.com/stanford-crfm/helm/issues/2322) - this _only_ applies to Classic v0.3.0
> 2. Downloading raw data from the GCS bucket (instructions [here](https://github.com/stanford-crfm/helm/blob/cace4fbdea241e47549eced910d3072c6714eb4b/docs/downloading_raw_results.md)) - this applies to any version of Classic, Lite, HEIM or Instruct
> 
> If your use case does not fit into either of these, you can write a your own script to download the web tables or the JSON files.

Thanks!
Repository: stanford-crfm/helm
Issue: Local model example
Root cause: Documentation Deficiency

It would be beneficial if there was documentation on how to run the scenarios against a local model (i.e. Llama 2).

Everything i've tried... doesn't work, and the examples on https://crfm-helm.readthedocs.io/en/latest/huggingface_models/ doesn't work with `unrecognized arguments: --run-entries boolq:model=stanford-crfm/BioMedLM@main`


Could you using `-r` instead of `--run-entries`? We recently renamed this argument to `--run-entries` but the PyPI package version is old and still uses the old argument name.

For Llama 2, you can try
```
helm-run \
    --run-entries boolq:model=meta-llama/Llama-2-7b-hf \
    --enable-huggingface-models meta-llama/Llama-2-7b-hf \
    --suite v1 \
    --max-eval-instances 10
```

Note that this requires you to follow the Hugging Face documentation for [Accessing gated models as a user](https://huggingface.co/docs/hub/en/models-gated#access-gated-models-as-a-user) and accept the license agreement on the [Llama 2 model page](https://huggingface.co/meta-llama/Llama-2-7b-hf).

Closing issue due to staleness. Please reopen if you're still running into this problem.
Repository: stanford-crfm/helm
Issue: Allow specifying MongoDB password via alternate means
Root cause: Unimplemented Feature Gap

Currently users have to specify the password in plaintext in a command line flag, which is very insecure. There was a first attempt here #1658 but it was slightly incorrect.

In particular, we should definitely support loading from `credentials.conf` and possibly from an environment variable.

Ideally this should not break reverse compatibility.

We also should not print the MongoDB password to output logs.
Repository: stanford-crfm/helm
Issue: Add validation for schema YAML files
Root cause: Unimplemented Feature Gap

Tasks:

- [ ] Add validation function for schema YAML files
- [ ] Call validation function in `summarize.py` i.e. `helm-summarize`
- [ ] Add test that calls the validation function on all schema files included in the HELM package

Things to validate:

- [ ] For each group name in `subgroups` of a `run_group`, there is a `run_group` defined with that name.
- [ ] For each entry in `metric_groups` of a `run_group`, there is a `metric_group` defined with that name.
- [ ] For each entry in `metrics` of `metric_groups`, there is either a metric with that name, or it matches the format `${var_name}` (where `var_name` can be any name).
- [ ] For each split in `metrics` of `metric_groups`, split is either `test`, `valid`, `__all__`, or matches the format `${var_name}` (where `var_name` can be any name).
- [ ] (maybe?) Each entry in `run_groups` is either a parent node (contains `category` and `subgroups`) or a child node (contains `metric_groups`) but not both.
- [ ] Each child node (contains `metric_groups`) is contained in at least one parent node.
Repository: stanford-crfm/helm
Issue: Bug cuasing run_spec to always be set to singleton(GoogleRunExpander().expand(run_spec)).
Root cause: Validation Gap

Bug causes run_spec to always be set to singleton(GoogleRunExpander().expand(run_spec)).

"or GOOGLE_GEMINI_MODEL_TAG" should be changed to "or GOOGLE_GEMINI_MODEL_TAG in model.tags"

https://github.com/stanford-crfm/helm/blob/4d416b712d21d5fe6b2c311a2d0ebd29b694b5bd/src/helm/benchmark/run_spec_factory.py#L142

Thanks for catching this!
Repository: stanford-crfm/helm
Issue: RunExpander instantiation error output from `pre-commit.sh`
Root cause: Validation Gap

`pre-commit.sh` causes the following error, currently present in the `master` branch.

```
src/benchmark/run_specs.py:1018: error: Cannot instantiate abstract class 'RunExpander' with abstract attribute 'expand'
```

@teetone Has this been resolved/can we close this?
Repository: stanford-crfm/helm
Issue: helm-summarize gives extra weight to runs in multiple groups
Root cause: Algorithmic Error

Suppose you have scenario `foo` with a parameter `param` and it has two runs like so:

```
{description: "foo:model=text,param=a", priority: 2}
{description: "foo:model=text,param=b", groups: ["x", "y"], priority: 3}
```

When aggregating the stats, `helm-summarize` will triple-count `foo:param=b` and single-count `foo:param=a` because `foo:param=b` gets counted one additional time per group. This is because of how [group_runs](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/summarize.py#L408) works.

This is also causing the MMLU leaderboard table cells to go to the run page with a single run, rather than going to the run directly.
Repository: stanford-crfm/helm
Issue: Please include gpt-4-0125-preview
Root cause: Unimplemented Feature Gap

From various sources and self-performed benchmarks gpt-4-1106-preview appears to outperform gpt-4-0125-preview in every aspect -- responses, JSON objects, even "laziness" which was supposed to be the primary improvement.

Possibly even include gpt-4-turbo-2024-04-09 since it was released today. Although, this appears to really be geared towards new vision efficiencies.

1106 introduced the much needed update to support JSON objects (I used helper functions to assist before this release) which was great. But, it appears OpenAI models degrade with each iteration. It makes sense since they are further fine-tuning the data set and only until the next major release will you see real gains in actual responses (3 vs 3.5 vs 4...). 

Hoping gpt-5 is released soon (rumored to be released this summer)!

Tldr; please add GPT-4 Turbo (0125 preview) to the benchmark.

Thanks for the suggestion. I'm currently working through evaluating a backlog of other models, but I agree that we should evaluate the gpt-4-turbo-2024-04-09 snapshot (which has already been added to the codebase)

gpt-4-turbo-2024-04-09 has been added to Lite v1.4.0.