zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Incorrect scoring due to answer format mismatch in MMLU evaluation
Root cause: Validation Gap

![image](https://github.com/user-attachments/assets/e60141be-2a1c-4945-8906-802df744aa9e)

## Description
During the MMLU evaluation of our LLM, we encountered an issue where correct answers are being marked as incorrect due to format mismatches. Specifically, when the model outputs the correct numerical answer but includes a preceding letter (e.g., "C. 12" instead of just "12"), the scoring system fails to recognize it as correct.

## Current Behavior
The scoring system marks answers as incorrect if they don't exactly match the reference answer format, even if the numerical value is correct.

## Expected Behavior
The scoring system should be able to correctly identify and score answers that are numerically correct, regardless of minor formatting differences such as preceding letters.



Hi @DerryChan, unfortunately this is something we don't plan to support for the default built-in MMLU scenario.

Some suggestions that you could try for your use case:

- Your could change your model to respect the `max_tokens` parameter, which is set 1 for to MMLU. This will usually cause the model to only output the letter.
- If your model is instruction-tuned, you can try adding an additional prompt that tells the model to only respond with a single letter. In particular, adding `output_format_instructions=mmlu` to your run entry (e.g. `mmlu:output_format_instructions=mmlu,model=text`) will add `"Answer with only a single letter."` to the prompt.
- You could implement your own MMLU variant that uses a modified metric that performs the additional post-processing necessary to interpret your model's output.
Repository: stanford-crfm/helm
Issue: Displaying perturbations on the web
Root cause: Unimplemented Feature Gap

Inspecting runs is a bit painful at the moment. We want to implement at the very least
- [x] Group inputs with the same id (at least make sure they appear next to each other
- [x] Display the perturbation used to produce each input (it is an field in `Instance`)
- [x] Render perturbed metrics propeply at the top

Things that would be nice to have
- [ ] Highlight what changed from the original input

For highlighting, seems like we need:
1. Some alignment algorithm, presumably pretty simple/naive and ideally off-the-shelf. Since we will have the original instance and its perturbed counterpart, this should tell us the parts to highlight and not highlight in the perturbed counterpart.
2. To use the same highlighting (as the green highlighting we do for the right answer), with a different color, on the parts identified in 1.

I think it is worth having, both to facilitate the robustness team's work, and, eventually, for folks who will view the results on the web interface (in the spirit of it being a highly interactive benchmark). Obviously will require someone to have the time though...

Moving to P3 since this isn't much of an issue given things are already grouped and Percy obviously has higher priority stuff.
Repository: stanford-crfm/helm
Issue: Perturbations are incorrectly displayed in frontend
Root cause: Unimplemented Feature Gap

The new UI does not do perturbation grouping, so it incorrectly shows a very large number of trials. The old UI groups requests by perturbations, then by trials.

[New UI](https://crfm.stanford.edu/helm/classic/v0.4.0/#/runs/boolq:model=openai_davinci,data_augmentation=canonical)
![Screenshot 2024-08-20 085758](https://github.com/user-attachments/assets/466f7e6b-f0e1-432f-8a26-ec644de4563d)

[Old UI](https://crfm.stanford.edu/helm/classic/v0.3.0/?runs=1&runSpec=boolq%3Amodel%3Dopenai_text-davinci-003%2Cdata_augmentation%3Dcanonical)
![Screenshot 2024-08-20 085744](https://github.com/user-attachments/assets/55e9144f-dfcd-421e-885f-0f984b12118d)
Repository: stanford-crfm/helm
Issue: run_specs returns null when using expand
Root cause: Configuration Error

Why does the RunExpander class in run_dexpander. py not have a specific implementation of expand, and returns empty when running “run_specs = [child_run_spec for parent_run_spec in run_specs for child_run_spec in expander.expand(parent_run_spec)] ”

when debugging i found：
class RunExpander(ABC):
    """
    A `RunExpander` takes a `RunSpec` and returns a list of `RunSpec`s.
    For example, it might fill out the `model` field with a variety of different models.
    """

    name: str

    @abstractmethod
    def expand(self, run_spec: RunSpec) -> List[RunSpec]:
        pass

Hi @xvanQ, `RunExpander` is an Abstract Base Class, which means it an abstract interface that is intended to be implemented by subclasses and not intended to be instantiated directly. For more information, refer to the [Python documentation on Abstract Base Classes](https://docs.python.org/3/library/abc.html) and [PEP 3119](https://peps.python.org/pep-3119/).

> Hi @xvanQ, `RunExpander` is an Abstract Base Class, which means it an abstract interface that is intended to be implemented by subclasses and not intended to be instantiated directly. For more information, refer to the [Python documentation on Abstract Base Classes](https://docs.python.org/3/library/abc.html) and [PEP 3119](https://peps.python.org/pep-3119/).

Thank you for your answer. I found that the problem is that if I use model='text ', the value in the corresponding values_dict is empty, which causes run_specs to be empty after expand, resulting in an index error when calling run_specs [0]

My command line parameters are as follows. Could you please take a look and see if there are any issues
--description wikifact:model=text,k=5,subject=plaintiff --model facebook/opt-iml-30b

https://github.com/stanford-crfm/helm/pull/2142#issue-2040546258
I think I have identified the problem, as not registering resulted in ALL_MODEL_METADATA being empty.
add：
register_builtin_configs_from_helm_package()
register_configs_from_directory(args.local_path)
Repository: stanford-crfm/helm
Issue: Question - Latest Version of results

How do I get the latest version of results?
Document mentions a dashboard version https://crfm-helm.readthedocs.io/en/latest/get_helm_rank/
Is that the latest? 

Found info here  https://crfm-helm.readthedocs.io/en/latest/downloading_raw_results/

You can find the latest results here: https://crfm.stanford.edu/helm/

In particular, refer to:

- HELM Lite: https://crfm.stanford.edu/helm/lite/
- MMLU: https://crfm.stanford.edu/helm/mmlu/
Repository: stanford-crfm/helm
Issue: Add amazon bedrock 3p models to the model configuration.
Root cause: Unimplemented Feature Gap

I want to add bedrock 3rd party models to the list of models available . Will use the Bedrock Client API's from the model providers  if they have them((Anthropic, Cohere and A121 ), Otherwise create one like the one for Titan. What are your thoughts?
 

Yes, I think it would be good to add the Bedrock models. In general, you would subclass the respective client in HELM (e.g. `AnthropicClient`) and modify it to connect to Bedrock. See here for the respective docs:

- [Anthropic](https://docs.anthropic.com/en/api/claude-on-amazon-bedrock)
- [AI21](https://docs.ai21.com/docs/aws-bedrock)
- [Cohere](https://docs.cohere.com/docs/amazon-bedrock)

Also refer to #1713 for some previous notes.

I added Mistral and LLAMA3 models on bedrock and submitted a PR. I will add the other 3 models  next week. Thank you
Repository: stanford-crfm/helm
Issue: Address the limitations of the MT-NLG API
Root cause: External Dependency Breakage

Original PR: https://github.com/stanford-crfm/benchmarking/pull/293

```
Supported parameters:
engine
prompt
temperature
max_tokens
stop ("Only a single "stop" value (str) is currently supported.")
top_p
echo

Not supported parameters:
n (to get multiple completions)
best_of
logprobs
presence_penalty
frequency_penalty

Not supported feature:
"logprobs" is always null in the response.
```

@teetone making this a P3 since idk when this will be done (my impression is this is a Microsoft-side issue)?

They support logprobs, and we handle n locally so we're okay.
Repository: stanford-crfm/helm
Issue: Verify and add entries for BBQ, BOLD, and CivilComments scenarios to `run_specs.conf`
Root cause: Unimplemented Feature Gap

Follow up to https://github.com/stanford-crfm/benchmarking/pull/235:

Verify you can run the scenarios and add them to `run_specs.conf`(see the TODO in the file)
Repository: stanford-crfm/helm
Issue: Add metric tooltip to Predictions
Root cause: Documentation Deficiency

The tooltip that describes metrics on hover should be in Predictions like it is in Leaderboard in the front-end

This seems to already be working.

![Screenshot 2024-10-30 142006](https://github.com/user-attachments/assets/ef482c0e-9509-4230-b38d-644ebee385a5)
Repository: stanford-crfm/helm
Issue: Help me understand the HELM Classic Leaderboard's missing results
Root cause: Validation Gap

![Screenshot from 2024-09-16 14-16-46](https://github.com/user-attachments/assets/d6c5216c-e19b-4d69-a637-e84d1e8e30c4)

Why were many models not evaluated on tasks like HellaSwag, OpenBookQA, MS MARCO, and summarization tasks like XSUM and CNN/Daily Mail? Is it because they are not suitable for these tasks?

Yes - many scenarios on HELM Classic require logprobs from the model, because they use an adapter that requires logprobs. Many recent model APIs do not provide logprobs.

Closing this issue due to staleness. Feel free to reopen if you have further questions.
Repository: stanford-crfm/helm
Issue: Add GPQA scenario
Root cause: Unimplemented Feature Gap

Paper: https://arxiv.org/abs/2311.12022
It is easiest to use the Hugging Face version: https://huggingface.co/datasets/Idavidrein/gpqa

Should be similar to original MMLU: see `mmlu_scenario.py` for the original MMLU and `air_bench_scenario.py` for how to use `load_dataset()` with Hugging Face datasets.

Edit: Also look at `simple_scenarios.py` and `test_simple_scenarios.py` for an example of MCQA.
Repository: stanford-crfm/helm
Issue: Add MMLU-Pro
Root cause: Unimplemented Feature Gap

https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro

Should be similar to original MMLU: see `mmlu_scenario.py` for the original MMLU and `air_bench_scenario.py` for how to use `load_dataset()` with Hugging Face datasets.

Edit: Also look at `simple_scenarios.py` and `test_simple_scenarios.py` for an example of MCQA.

Edit 2: Also see [this doc](https://crfm-helm.readthedocs.io/en/latest/adding_new_scenarios/).

Edit 3: To create the run spec function, take this function in `lite_run_specs.py`:

```python
@run_spec_function("mmlu")
def get_mmlu_spec(subject: str, method: str = ADAPT_MULTIPLE_CHOICE_JOINT) -> RunSpec:
```

and modify it so mmlu becomes mmlu-pro, then you should be able to do `helm-run`.
Repository: stanford-crfm/helm
Issue: o1 series models cannot take system prompt
Root cause: Unimplemented Feature Gap

It seems that the current o1 series models `o1-mini-2024-09-12` and `o1-preview-2024-09-12` are still not able to take system prompts. Would it be a good idea to set them to concatenate system prompts with user prompts? Thanks!

I don't think so, because the model API should reflect the actual OpenAI API as closely as possible. Rewriting the request internally can lead to unexpected results for other researchers.

Got it, that makes sense!
Repository: stanford-crfm/helm
Issue: Add Gemini-1.5-Pro-002 and Gemini-1.5-Flash-002
Root cause: Unimplemented Feature Gap

Blog: https://developers.googleblog.com/en/updated-production-ready-gemini-models-reduced-15-pro-pricing-increased-rate-limits-and-more/

Gemini-1.5-Pro-002 and Gemini-1.5-Flash-002 should work exactly the same as the older snapshots `google/gemini-1.5-pro-001` and `google/gemini-1.5-flash-001`, so it's a matter of copy and pasting the model metadata /deployment and updating the strings.


I am working on this issue.
Repository: stanford-crfm/helm
Issue: Add Llama 3 Instruct Lite / Turbo family
Root cause: Unimplemented Feature Gap

Blog: https://www.together.ai/blog/together-inference-engine-2

See [here](https://docs.together.ai/docs/chat-models) for the Together model strings.

Should be very similar to the existing Llama 3 Instruct models. They should have _new_ model metadata entries because we don't consider the Lite / Turbo versions to be the same as the original versions.

I am working on this issue.
Repository: stanford-crfm/helm
Issue: HatefulMemesScenario get_instances returning error
Root cause: External Dependency Breakage

Calling get_instances for HatefulMemesScenario is getting me the following:
Exception: Failed with exit code 8: wget 'https://huggingface.co/datasets/neuralcatcher/hateful_memes/resolve/main/img/23058.png?download=true' -O output/benchmarks/downloaded/hatefulmemes/img/23058.png.tmp

Specifically, the traceback is:
- line 66 in common/general.py, in shell: raise Exception(f"Failed with exit code {exit_code}: {cmd}")
- line 94 in common/general.py, in ensure_file_downloaded: shell([downloader_executable, source_url, "-O", tmp_path])

Could you provide the complete error logs, including the exception message itself?

cc @teetone 


![Screenshot 2024-10-14 at 11 11 15 AM](https://github.com/user-attachments/assets/bdfb5690-fc2f-4055-90d9-fa6c121cf9e7)


OK, I can replicate this locally. There is indeed a row for image `23058.png` which is missing from this [Hugging Face mirror](https://huggingface.co/datasets/neuralcatcher/hateful_memes).

@teetone How are you running this in production? Are you using the [canonical version](https://github.com/facebookresearch/mmf/tree/main/projects/hateful_memes) of the dataset instead?
Repository: stanford-crfm/helm
Issue: Proposal: Add free-form data fields to Instance

## Background

The current schema for `Instance` is very limited - it allows a single `Input` and a `List[Reference]`. This is appropriate for most question answering scenarios and traditional NLP tasks, but other scenarios contain additional structured data for each instance.

Examples:

- AIR-Bench, HarmBench, SimpleSafetyTests needs to store the taxonomy of categories. It currently puts these in references.
- BBQ needs to store whether the question is ambiguous or not. It currently puts this in the tags of every reference (the same "ambiguous" / "non-ambiguous" tag is duplicated across references).
- XSTest needs to store whether the question is safe or not. It currently uses the last reference for this.
- MedAlign in #3038 needs to store the EHR context separately from the question, because it needs to truncate the EHR context in a special way before it is interpolated into the prompt.
- MTBench, which is not yet implemented, needs to store the category of the instance, which is used to determine the prompt for the LLM-as-judge.

## Proposal 

Add the field `data: Dict[str, Any]` to `Instance`. The value of `data` can contain nested dictionaries and arrays, but only contains leaf `str` and `number` values, and should be serializable to JSON.

Scenarios can use `data` however they wish, e.g. setting data to `{"category": ""}` or `{"ehr": "...", "question": "..."}`

The frontend can traverse the data object and render a list of key value pairs, where the keys are the JSON path.

## Open Questions

- Should there be two separate fields, `data` and `metadata`?
- Should be the field be named `extra_data` instead of `data`?

## Alternatives Considered

- Allow subclassing `Input`. The subclass can define new fields needed by the scenario. This allows better type-checking than the proposal. However, this doesn't work because subclasses of `Input` are not preserved after round-trip to JSON using `dacite` or `cattrs`. The extra fields defined in the subclass will be removed after the round trip. We need to perform this round trip in various places e.g. `helm-summarize`.
- Make `Instance` itself a free-form dict. We have too many existing `Instance` field accesses, so this would require too many changes and potentially break other things.

MMLU and GPQA both need this because they both need to store an extra chain-of-thought annotation. See: #3088

Fixed by #3094.
Repository: stanford-crfm/helm
Issue: Proposal: User-provided custom adapters
Root cause: Unimplemented Feature Gap

## Background

There are a number of scenarios that require user-provided custom adapters. Currently, HELM only supports built-in adapters.

Example use cases:

- MedAlign in #3038 needs to truncate the EHR context in a special way before it is interpolated into the prompt.
- MTBench, which is not yet implemented, needs to generate multi-turn requests in which the model's previous response is incorporated into the message history.
- Minimal-pairs scenarios like BLIMP and EWOK may want to send two requests per instance and take the instance with lower logprobs.
- Model-as-examiner scenarios like AutoBencher may want to query a model to generate instances dynamically, rather than using predefined instances from a static dataset.
- Stand-alone projects (like HEIM and VHELM) and external users of HELM may want to store their project-specific adapters in a separate repository outside of HELM, especially if the adapters contain private or proprietary information.

The custom adapters subclass might also require additional arguments to be passed in, besides existing arguments in `AdapterSpec`.

## Proposal 

Currently, adapters are instantiated inside `AdapterFactory.get_adapter()` ([code](https://github.com/stanford-crfm/helm/blob/572a1bb141c60112a2c9ff8e9563222484e5d8a2/src/helm/benchmark/adaptation/adapters/adapter_factory.py#L28)) which expects `adapter_spec.method` to be one of several canned enum values. It instantiates an adapter from one of several built-in classes based on this enum value.

We could allow users to set `method` to either an enum value or a class name. If `method` is an enum, we revert to the default behavior, otherwise, we instantiate an instance of the provided class and pass in `adapter_spec` and `tokenizer_service`.

We could also add an `args: Dict[str, Any]` field to `AdapterSpec` which can be used to passing additional arguments to `AdapterSpec`.

## Alternatives Considered

We could make `class_name: str` a separate field from method. A valid `AdapterSpec` will have exactly one of `method` and `class_name` set. The semantics are otherwise the same as the main proposal.

We could make `AdapterSpec` an `ObjectSpec`. This would mirror `ScenarioSpec` and `MetricSpec`, which are also subclasses `ObjectSpec`. This would also allow users to pass arbitrary extra parameters to the `Adapter`. However, this doesn't work because there are a massive number of dependencies on specific fields in `AdapterSpec` - various `RunExpander`s modify specific fields in ObjectSpec, and `Runner` directly accesses several fields including `max_eval_instances` and `eval_splits`.

We could put in an `custom_adapter_spec: Object` field inside `AdapterSpec`. If set, we instantiate the adapter using the usual `ObjectSpec` semantics, additionally passing in `adapter_spec` and `tokenizer_service` as additional parameters.
Repository: stanford-crfm/helm
Issue: Proposal: Multi-turn support
Root cause: Unimplemented Feature Gap

## Background

Currently, HELM only supports single-turn scenarios. The main evaluation flow in `Runner.run_one()` works at follows:

1. Get instances from the scenario
2. Get prompts for the instances from the adapter
3. Run prompts through the model to get outputs
4. Score outputs on metrics

However, some scenarios need to look at outputs from the model, and then send more prompts to the model. Examples:

- In MT-Bench, the model is given an initial prompt, and then the it is given a follow up prompt with the context of the first interaction.
- In SOTOPIA, two models converse for several turns, and a third model grades the conversation.

## Proposal 

Add a loop to `Runner.run_one()` around steps 2 and 3. After running prompts through the model, call the adapter again to get additional prompts, and then run those through the model. Repeat until the adapter stops producing new prompts.

Concretely, the Adapter will have a new method for generating more request states:

```python
class Adapter(ABC):
    def adapt_next_turn(self, instances: List[Instance], scenario_state: ScenarioState, parallelism: int) -> List[RequestState]:
        """
        Takes `Instance`s and all previous `RequestState`s, and returns new `RequestState`s.
        """
        return []
```

In the frontend, all requests will be grouped under the same instance and displayed.

### Pros

Because the next turn for all instances are generated and processed simultaneously in each instance of the loop, we get thread-level parallelism when processing requests to the models.

### Cons

The API is somewhat unnatural if the user is thinking in terms of the chronology of a single conversation, since the method requires operating on all conversations at once. The user thus needs to perform some internal bookeeping. We could alleviate this by providing bookkeeping utilities in utility classes or functions.

This doesn't address the high storage costs of multi-turn conversations, which is O(N^2), since we keep all N requests, and the length of the Nth request grows with N since it includes all previous context.

Closing this issue as not planned. In general, [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai/) is a better fit for Sotopia-style interactive evaluation.
Repository: stanford-crfm/helm
Issue: Not able to install install-heim-extras.sh for heim leaderboard
Root cause: Environment Incompatibility

Hi All,

I am try to create replicate leaderboard to text to image models(HEIM) in colab but not able but getting the following error when trying to read the "install-heim-extras.sh" file:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
chex 0.1.87 requires jax>=0.4.27, but you have jax 0.3.25 which is incompatible.
chex 0.1.87 requires jaxlib>=0.4.27, but you have jaxlib 0.3.25+cuda11.cudnn805 which is incompatible.
flax 0.6.11 requires jax>=0.4.2, but you have jax 0.3.25 which is incompatible.
optax 0.2.3 requires jax>=0.4.27, but you have jax 0.3.25 which is incompatible.
optax 0.2.3 requires jaxlib>=0.4.27, but you have jaxlib 0.3.25+cuda11.cudnn805 which is incompatible.
orbax-checkpoint 0.6.4 requires jax>=0.4.26, but you have jax 0.3.25 which is incompatible.

Could you try running the following and see if it fixes things for you?

```sh
pip install --upgrade jaxlib==0.4.30
```

Closing due to staleness. Feel to reopen if you have additional questions.
Repository: stanford-crfm/helm
Issue: How to use this package when I have the pompts and images across models
Root cause: Documentation Deficiency


Hi @snehith3195, could you provide more information about your use case? Please refer to the documentation [here](https://crfm-helm.readthedocs.io/en/latest/) for general instructions.

Closing due to staleness. Feel to reopen if you have additional questions.