zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Include the number of in-context examples as a Metric
Root cause: Unimplemented Feature Gap

Necessary, especially if the number of in-context examples was reduced to fit within the context window.

Presumably, this will require modifying `adapter.construct_prompt` (https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/adapter.py#L451) to record the number of in-context examples used for each instance. When we do this, can we also record and display whether the instance was truncated or not?

(currently we just return `self.tokenizer.truncate_from_right(prompt, self.adapter_spec.max_tokens)`)

@dtsip Can you resolve this while you are addressing truncation? Took the liberty of assigning you, but let me know if that isn't ok. 

@dtsip What's the status of this?

I haven't really looked into it since it requires changes that are not really relevant to my immediate todos.

The major change required is to have `construct_prompt` return additional info (whether the input is truncated and how many in-context examples we use) and add these to `Request` (?) so that we can later display them.

Maybe it's worth creating a `Prompt` dataclass to record the extra info.

Ok, took a closer look and I don't feel super confident implementing this given my familiarity of this part of the codebase. It requires keeping track of some per-instance quantity from prompt (which is currently a string), to request, to metric.

Resolved by https://github.com/stanford-crfm/benchmarking/pull/650
Repository: stanford-crfm/helm
Issue: AI2 reasoning challenge
Root cause: Unimplemented Feature Gap

[ARC](https://arxiv.org/pdf/1803.05457.pdf) is a challenging QA dataset. I was wondering if there are any reasons why it wasn't included as part of the available datasets. Are there any plans to add it in the future?

We don't have any plans to add AI2 at the moment, but you are welcome to open a pull request to add it!

We do have other scenarios for question answering using commonsense reasoning, including [OpenBookQA and CommonSenseQA](https://github.com/stanford-crfm/helm/blob/e2ab6c2d8749cc1712e7c1dd45c78dcec87e2068/src/helm/benchmark/scenarios/commonsense_scenario.py#L24-L37).
Repository: stanford-crfm/helm
Issue: CI Test Automation for Models
Root cause: Unimplemented Feature Gap

- [ ] Add models as a pytest mark
- [ ] Skip models pytest mark by default
- [ ] Add GitHub action that only runs model tests and is scheduled weekly
- [ ] Add GitHub action for running on PRs that affect models
- [ ] Add documentation for how to add a test for a model

We have unit tests for some models, but we decided not to run them in CI for security and cost reasons.
Repository: stanford-crfm/helm
Issue: Rewrite JSON / LaTeX / code URLs in frontend
Root cause: Architectural Constraint

There are a number of incorrect JSON / LaTeX / code links. Unfortunately, these are baked into the JSON release files, and I would prefer not to modify them. A workaround would be to have the frontend rewrite these paths and URLs to have the correct prefixes.

Instances of this:

- #1442
- #1591
Repository: stanford-crfm/helm
Issue: Add Python API tutorial in the form of a Python notebook
Root cause: Documentation Deficiency

A executable Python notebook (e.g. Colab) would be a good demonstration of the Python API and its internals. Levanter and [DSP](https://github.com/stanfordnlp/dsp) both have good notebooks.

Things to document:

- [ ] How to use `AutoClient` and `Service`, and how to provide the credentials and cache
- [ ] Step-by-step internals of what `Runner` does during a single evaluation run.
- [ ] How to add your own model
- [ ] How to add your own scenario (might be tricky)


Noting that the API needs to be improved; the current one is not great for notebooks to begin with.
Repository: stanford-crfm/helm
Issue: Add link to per_instance_stats.json on frontend
Root cause: Unimplemented Feature Gap

We currently have a link to `stats.json` but not `per_instance_stats.json`.

Is it open to work? 

Sure! Unfortunately, we actually have two different frontends currently (the old one that will be deprecated eventually and the new React one that's not ready yet), so there are two places that need to be change. Feel free to open a PR for just the old frontend, a PR for just the new React frontend, or both. I'd suggest starting with the old frontend first because it's easier to setup. The new React frontend requires Node to build (setup instructions [here](https://github.com/stanford-crfm/helm/tree/main/src/helm-frontend)).

In both cases, you want to:
1. find the function that gives the the URL to `scenario_state.json`, and add a new function next to it that gives the URL to `per_instance_stats.json` (which is in the same folder).
2. Find the link to `scenario_state.json` and add a link next to it to `per_instance_stats.json`.

Old frontend:
1. URL function is [here](https://github.com/stanford-crfm/helm/blob/v0.3.0/src/helm/benchmark/static/json-urls.js#L43-L45)
2. Link is [here](https://github.com/stanford-crfm/helm/blob/v0.3.0/src/helm/benchmark/static/benchmarking.js#L877-L881)

New frontend:
1. URL function is [here](https://github.com/stanford-crfm/helm/blob/v0.3.0/src/helm-frontend/src/services/getScenarioStateByName.ts#L4)
2. Link is [here](https://github.com/stanford-crfm/helm/blob/v0.3.0/src/helm-frontend/src/routes/Run.tsx#L183-L190)

cc @farzaank for frontend changes

Note that the old frontend doesn't exist any more. So this change only needs to be made in one place.
Repository: stanford-crfm/helm
Issue: Run evaluation against an isolated local model API
Root cause: Unimplemented Feature Gap

Hi,

I would like to evaluate a local [AWQ](https://github.com/mit-han-lab/llm-awq/) quantized model. The model implements all the HuggingFace `AutoModelForCausalLM` APIs - however since it's a quantized model - it has a few custom layers (it implements zero point quantization under the hood, so all the Linear layers are replaced by a custom layer, etc.).

I also want to run this model in an isolated environment because it has conflicting dependencies with those of helm (for e.g. different PyTorch versions, etc.).

Is there a way to run my custom LLM locally as a REST service / API and obtain helm results?

Thanks!

There currently isn't an easy way to run a model server and connect it to HELM, but I think that would be a good thing to add to HELM.

Could you let me know which dependency conflicts you are running into? I'm planning to expand the set of allowable versions to avoid conflicts. I am also planning to upgrade the PyTorch version in HELM.

In the meantime, my recommendation would be to, in a virtual environment, install HELM, and then manually uninstall PyTorch and reinstall the PyTorch version that you need. This is a rather hacky workaround though.

Also, we just added support for evaluating Hugging Face models on local disk without needing to push them to Hub in #1505 using `--enable-local-huggingface-models`. This is on the main branch but not yet released on PyPI. You might find this useful!
Repository: stanford-crfm/helm
Issue: Reproduction of bigcode results on HELM 
Root cause: Documentation Deficiency

Per https://github.com/bigcode-project/bigcode-evaluation-harness/issues/110#issuecomment-1619906664, 

I wonder can you release your bash commands to run `bigcode/starcoder` on HELM MATH task for the community to rigorously reproduce the results in starcoder paper.

Actually I'm trying to find your prompt template for MATH as well.

The command is:

```
helm-run \
--conf-paths path/to/run_specs.conf \
--priority 2 \
--num-train-trials 3 \
--max-eval-instances 1000 \
--suite starcoder \
--models huggingface/starcoder
```

where `path/to/run_specs.conf` is the path to your local copy of [run_specs.conf](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_specs.conf)

To run only the MATH scenario, append `--groups math_regular` or `--groups math_chain_of_thought`.
Repository: stanford-crfm/helm
Issue: Using HELM benchmarking on own dataset
Root cause: Documentation Deficiency

Hi,
I want to use HELM specific to Question answering task on my own dataset and Finetuned model. I want to compare answers generated by my model and Original answers on the HELM metrices.
I have some hint from the other github issues related to my question like creating scenario but still facing some challenges.
1. How  can we upload our own dataset to create scenarios.
2.What should be the input format for the dataset.
3.In which file can we send our queries.
4.How to understand the output created in JSON files(I have run to quick start from the documentation) there are multiple JSON and I want to understand where is the value of metrics(evaluation part) for specific query.
5.I want to specifically use metrics 
Accuracy-F1, calibration-10-bin expected calibration error,Robustness-F1,Fairness-F1,Bias-Stereotypes(race,gender),Demographic representation gender, race, Toxicity-Toxic fraction, Efficiency-denoised inference runtime.
How to evaluate specifically on these metrics.


Thanks

1. Currently, you have to fork the HELM repository and make code modifications. Refer to the following links: [Developer setup](https://crfm-helm.readthedocs.io/en/latest/developer_setup/), [Adding a new scenario](https://crfm-helm.readthedocs.io/en/latest/code/#adding-new-scenarios), [Example PR that adds a scenario](https://github.com/stanford-crfm/helm/pull/1643)
2. This is up to you, but the most convenient formats are CSV, JSONL and Hugging Face Datasets.
3. The queries are generated by applying the specified adapter to the instances returned by `get_instances()` of your `Scenario`.
4. `per_instance_stats.json` output file will contain the metrics for each instance. Refer to the [schema docs](https://crfm-helm.readthedocs.io/en/latest/schemas/#helm.benchmark.metrics.metric.PerInstanceStats) for the schema of this file.
5. The metrics are specified by passing `MetricSpec`s to the `metrics` parameter of the `RunSpec`. ([example](https://github.com/stanford-crfm/helm/blob/abfdcd8acd23b5ef3ec7ec987f5c90fb9de81406/src/helm/benchmark/run_specs.py#L1116))

@yifanmai Why sometimes I don't have Stereotypes(race) this value?
Repository: stanford-crfm/helm
Issue: GenderPerturbation do not perturb words at the beginning of the text
Root cause: Algorithmic Error

Is there a reason why GenderPerturbation apply the perturbation only on words surrounded by non-alphanumeric characters, and not for example, at the beginning of the sentence?

```python
>>> gender_pertubation = GenderPerturbation(mode="pronouns", prob=1.0, source_class="male", target_class="female", bidirectional=False)
>>> print(gender_pertubation.perturb("he went to the market, and there he had a soup.", RNG))
he went to the market, and there she had a soup.
```

This happens because of the regex:
https://github.com/stanford-crfm/helm/blob/daa165aae07e575bdcb3b5cca699403c82c759dc/src/helm/benchmark/augmentations/gender_perturbation.py#L195-L198

Perhaps a better option would be using the regex word boundary `\b`? Something like that:
`pattern = fr"\b({re.escape(word)})\b"`


@shlomihod You're right; this looks like a bug. We should fix this for future releases. But I also need to think about reverse compatibility (i.e. unfortunately we can't evaluate the fixed perturbation for old models that have been deprecated).

@shlomihod Thanks again for catching this subtle bug!

cc @dilarasoylu regarding `GenderPerturbation` bug that we should probably fix

On related topic: I think there is a principled issue regarding the perturbation: depends on the text, they might have no effect.

Dialect and robustness would probably transform many words, but gender might have no impact on a text (e.g., reviews written in first person). I think there should be another metric to assess the proportions of datapoints that were changed, something like [manipulation check](https://en.wikipedia.org/wiki/Manipulation_check) for experiment design. Only if the manipulation is substantial (not sure what is the right way, but here is a simple heuristic for start: > 50% of the examples changed), then we are allowed to interpret the effect of the metric.
Repository: stanford-crfm/helm
Issue: Remove `__all__` split and perturbation matchers in metric names
Root cause: Architectural Constraint

The `__all__` matcher was a hack used to get certain HEIM aggregations to work where the stats context was not set correctly. We should strongly consider reducing tech debt by removing it.

Going to mark this as wontfix. I'm okay with keeping some hacks for HEIM. We should strongly discourage future use of the `__all__` matcher, though.
Repository: stanford-crfm/helm
Issue: How to use RedPajama-INCITE-Base-v1 (7B)
Root cause: Documentation Deficiency

I want to use RedPajama-INCITE-Base-v1 (7B) (together/redpajama-incite-base-7b-v1) in HELM.

It is written in
- the model list ( https://crfm.stanford.edu/helm/latest/?models )

and
- [src/helm/benchmark/static/schema.yaml](https://github.com/stanford-crfm/helm/blob/daa165aae07e575bdcb3b5cca699403c82c759dc/src/helm/benchmark/static/schema.yaml#L707)
		

However, I can't find it in
- [src/helm/proxy/models.py](https://github.com/stanford-crfm/helm/blob/daa165aae07e575bdcb3b5cca699403c82c759dc/src/helm/proxy/models.py#L669)

(Also, the text of the model on the top page ( https://crfm.stanford.edu/helm/latest/ ) is gray.)
![image](https://github.com/stanford-crfm/helm/assets/11292872/bccd3268-d460-4423-a20b-8be88b646e06)

When I want to use RedPajama-INCITE-Base-v1 (7B),
should I modify the code with the reference to the following page?
https://crfm-helm.readthedocs.io/en/latest/adding_new_models/


Sorry, in my case, I want to use Hugging Face Model, togethercomputer/RedPajama-INCITE-7B-Base.
I thought it would be good to refer to the following document.
https://crfm-helm.readthedocs.io/en/latest/huggingface_models/

So I would like to close this issue. Sorry for the trouble.

Hi @takitsuba, the Hugging Face integration will indeed work. Additionally, the [built-in versions of RedPajama (e.g. `together/redpajama-incite-base-3b-v1`)](https://github.com/stanford-crfm/helm/blob/abfdcd8acd23b5ef3ec7ec987f5c90fb9de81406/src/helm/proxy/models.py#L684-L693) use Together's API, which requires [signing up for a Together account](https://api.together.xyz/). I can provide more details on how to use Together's API if you are interested.

Thank you for your kindness, @yifanmai !
When the time comes for me to use it, please let me consult with you again!

I filed #1755 for an easier way to use RedPajama etc without Together.
Repository: stanford-crfm/helm
Issue: Add integration with OpenAI Evals
Root cause: Unimplemented Feature Gap

We want to write a generic converter that allows us to import an arbitrary OpenAI evaluation task and run it as a HELM scenario. This would be similar to what we do with BIG-Bench currently.

This isn't feasible because OpenAI doesn't have a Scenario / Adapter / Metrics decomposition that we do, but we could re-implement some targeted tasks from OpenAI Evals if we wanted to.
Repository: stanford-crfm/helm
Issue: Architecture and data schema review

Write up review of HELM architecture and data schema, especially things that will be hard to change later.

Would also be good to get a few outside opinions from a few other senior engs.

Hi everyone!
I have few suggestions:
* Make default Runner implementation "replaceable", like SlurmRunner, but via some configuration, not just CLI flags. For example, we are heavy AWS users, there are services that can be used to manage workloads units like "jobs" - ECS tasks or SageMaker jobs, and so on. I guess, many HELM users have other preferences in other clouds or K8s...
* Introduce some interface for the persistence layer - responsible for storing benchmark run outputs. Right now it writes into "local" filesystem (`benchmark_output` folder) using standard Python IO means. It'd be nice to have support of other storages - like S3 or GCS. May be this persistence layer can be handled completely by a runner implementation.
* Overall, it'd be nice to separate core parts of HELM (run specs schema, output data schema, adapters, metrics) from underlying infrastructure and UI-related components - I guess, it will facilitate contribution into these different aspects and will make merging from different forks easier.

Overall, this is a great framework, thank you so much for your efforts! We use it as a kind of standard / specification for making internal benchmarks for different projects or integrating with publicly available ones.

- *Make default Runner implementation "replaceable"* - Agreed re alternate runners, especially k8s. I'd suggest opening a new issue if you have a high priority need; I'm curious about how much demand there is for this in the community.
- *Introduce some interface for the persistence layer* - Agreed. S3 and GCS support might be doable if we use `fsspec` (the main concern there is that there are weird interactions with `fsspec` and `os.path.join()`-style path manipulation). But I would also like to support database storage as well.
- *Modularity* - The repository as is is somewhat of a mono-repo. There are components that can be used separately but it's not obvious where the boundaries are. Additionally, installing HELM requires installing a massive list of dependencies (#1327). These are all things I'd like to improve.
Repository: stanford-crfm/helm
Issue: PubMedQA JSON Decoder Error
Root cause: External Dependency Breakage

![image](https://github.com/stanford-crfm/helm/assets/39891180/d3317d2f-ef0a-4210-a2bf-5714095beaf2)
![image](https://github.com/stanford-crfm/helm/assets/39891180/9933ea91-0a82-410a-9531-1ffabef2902e)
![image](https://github.com/stanford-crfm/helm/assets/39891180/8a7681a6-a5c6-47c6-8cdb-4ba18d75f80e)
The first pic is the error I got. The second pic is the run_specs I used. The third pic is the command I used. If you need more information, please let me know.

This is likely because the https://worksheets.codalab.org/ was down for the last few days. Could you try again and let me know if this still fails?

See: https://github.com/codalab/codalab-worksheets/issues/4511

`pubmed_qa` is working again now that CodaLab is back up.
Repository: stanford-crfm/helm
Issue: Add HumorSet scenario to HELM?
Root cause: Unimplemented Feature Gap

Before creating a PR, I'd like to get some feedback on whether it's a good idea to add [HumorSet](https://github.com/zmxv/humorset) to HELM.
HumorSet is an open-ended question collection that assesses LLM's ability to explain jokes.
humorset1 contains 128 short jokes in English and currently relies on human ratings.
It has been tested on GPT-4 and Claude 2.

Looks reasonable to me.

We have a human rating integration that you can use. Take a look at [summarization_critique_metrics](https://github.com/stanford-crfm/helm/blob/87ee885c69ea34bd736839079f6ff2911b65f873/src/helm/benchmark/metrics/summarization_critique_metrics.py) for an example. You can also use GPT-4 as a simulation of a human rater (see #1627).

@zmxv are you still working on this?

Closing due to staleness; feel free to reopen if needed.
Repository: stanford-crfm/helm
Issue: Allow Hugging Face models to be loaded with bf16
Root cause: Unimplemented Feature Gap

This would significantly increase inference speed.

The main issue is figuring out where this configuration (and model configuration in general) would live. We could use a flag or environment variable for now, but ideally we should have a universal config file that takes care of other kinds of configuration (e.g. implementation type, host URL, etc.) Ideally we do this at the same time as the model architecture refactor.

Nothing that this was suggested by Chenhui Zhang (working with Bo Li).

Fixed by #1912.
Repository: stanford-crfm/helm
Issue: Update Anthropic Client
Root cause: External Dependency Breakage

Anthropic changed their python sdk - making this code line outdated.

https://github.com/stanford-crfm/helm/blob/64ff40be6ac594857ae883d5d3b36e571d0f0d66/src/helm/proxy/clients/anthropic_client.py#L67

Would love to know if this might help - https://github.com/BerriAI/litellm

~100 lines of code, that standardizes all the llm api calls to the OpenAI call

```
from litellm import completion

## set ENV variables
# ENV variables can be set in .env file, too. Example in .env.example
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["ANTHROPIC_API_KEY"] = "anthropic key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# anthropic call
response = completion("claude-v-2", messages)

```


cc: @teetone 

Anthropic legacy code - there's a bunch of stuff that isn't used any more, but we keep it for backwards compatibility so that we can used our database of previous requests and responses for the old models.

Re litellm - it basically reimplements what we already do in HELM, which is to standardize everything to a single format, though in HELM's case, HELM uses its own standard request and response. I can see litellm being useful in other use cases though. In particular, it's hard for users to install HELM _just_ to use our standardization code, because HELM is so heavyweight.



Hey @yifanmai thanks for the feedback. Curious - why not use something like `lite` to expand the models HELM can support? I don't see the conflict necessarily (HELM seems to support a wide range of models including open-source models like Bloom) 

any reason why this isn't useful here (e.g. expanding support to Cohere, Replicate, etc.)

We currently support a superset (with the exception of Azure, see #1415) of the models in lite, so we wouldn't gain any additional coverage. See the model list [here](https://crfm.stanford.edu/helm/latest/?models=1).
Repository: stanford-crfm/helm
Issue: Targeted Evaluations and Harms pages are broken
Root cause: Validation Gap

https://crfm.stanford.edu/helm/latest/?group=targeted_evaluations
https://crfm.stanford.edu/helm/latest/?group=harms

The breaking tables are and Disinformation (Wedging) its ancestor tables Harm, Targeted Evaluations.

The reason it breaks is rather subtle:

1. Palmyra API has a content moderation filter and will return an error if the prompt or generation contains moderated content.
2. To account for this, in `PalmyraClient` we [treat the generation as an empty string](https://github.com/stanford-crfm/helm/blob/v0.2.3/src/helm/proxy/clients/palmyra_client.py#L103-L114) if the content moderation filter is triggered.
3. When computing entropy, if the generation is an empty string, the [value of the entropy stat is `NaN`](https://github.com/stanford-crfm/helm/blob/v0.2.3/src/helm/benchmark/metrics/disinformation_metrics.py#L58-L60).
4. In `helm-summarize`, we generate JSON for various group tables. In the cells with `NaN values, it gets rendered as follows: 
```javascript
{
    "value": NaN,
    "description": "min=NaN, mean=NaN, max=NaN, sum=NaN (4)",
    "style": {},
    "markdown": false
}
```
5. This value is not valid JSON. Attempting to parse in Chrome results in the error `Unexpected token 'N', ..."{"value": NaN,"descr"... is not valid JSON`. This is because `NaN` is not considered a valid literal in JSON, yet the Python stdlib outputs `NaN`. This is a known [wontfix issue](https://bugs.python.org/issue40633) and is documented [here](https://docs.python.org/3.8/library/json.html#infinite-and-nan-number-values).

Possible fixes:

- When generating JSON that the browser will read, replace the `NaN`s with `null`s.
- In the disinformation computation code, don't compute entropy if the generation is empty.

The tables are working again in v0.2.4.