zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Upgrade mypy
Root cause: External Dependency Breakage

We use 0.982, and the current version is 1.5.1.

This will have to be done incrementally because 1.5.1 raises hundreds of additional errors on the current repository.

mypy now gives the note:

```
By default the bodies of untyped functions are not checked, consider using --check-untyped-defs
```

If I set `--check-untyped-defs`, this results in 209 errors in 34 files.

I will set `check_untyped_defs = False` for now.
Repository: stanford-crfm/helm
Issue: Investigate logprobs of LLaMA and Llama 2 from Together
Root cause: External Dependency Breakage

LLaMA and Llama 2 get anomalous calibration scores when inference is run on Together.

For Llama 2, Together gave us a mix of responses with and without logprobs. So that means that Llama 2 rows on the calibration table have sporadic empty and filled cells. And I'm not sure the filled cells are correct (because there might be mixed filled / empty within the same run).

We should also make sure that the logprobs are replicated when running on Hugging Face.

In general, we're not able to compute calibration for most models, since many model providers are no longer giving us logprobs through their APIs.
Repository: stanford-crfm/helm
Issue: BPB is 0 for huggingface models
Root cause: Unimplemented Feature Gap

When I used hugging face `gpt-j-6b` model to run on `twitter_aae` dataset, everything was fine except for the final stat gives BPB=0. I tried to use other datasets, but the problem persisted.

Command to run the experiment:
`helm-run --conf-paths run_specs.conf --enable-huggingface-models EleutherAI/gpt-j-6b --suite v1 --max-eval-instances 1000`

`run_specs.conf` content:
```
entries: [
      {description: "twitter_aae:demographic=aa,model=EleutherAI/gpt-j-6b", priority: 1},
 ]

```

`bits_per_byte` result in `stats.json`:
```
{
    "name": {
      "name": "bits_per_byte",
      "split": "test"
    },
    "count": 1,
    "sum": 0.0,
    "sum_squared": 0.0,
    "min": 0.0,
    "max": 0.0,
    "mean": 0.0,
    "variance": 0.0,
    "stddev": 0.0
  },
```

Thank you in advance for your help!

This is because `HuggingFaceClient` currently does not return logprobs. This was discussed in #1469 and #1497.

I think this is important and we do want to fix this soon.

Closing as duplicate.
Repository: stanford-crfm/helm
Issue: Unused `stop_sequences` in `MultipleChoiceJointAdapter`
Root cause: Documentation Deficiency

https://github.com/stanford-crfm/helm/blob/242a43dda181be6050319a62ab5de64a902224dd/src/helm/benchmark/adaptation/adapters/multiple_choice_joint_adapter.py#L61

Hi, it seems that `multiple_choice_joint` adapter does not use `stop_sequences`, which could be useful for some non-instruction-tuned models since they are likely to synthesize new examples after prediction. May I know if this is a bug or intended?

The reason is because we expect set `max_tokens` to 1 i.e. we expect "A" through "Z" to each be a single token, and we're telling the model to stop outputting any other tokens afterwards.

I see. It makes sense to me. Thanks for your reply!
Repository: stanford-crfm/helm
Issue: Lit-GPT follow up
Root cause: Interface Contract Mismatch

- [ ] Locking for multi threaded evaluation - @aniketmaurya
- [ ] implement log probs - @aniketmaurya
- [ ] make the model configuration work - @yifanmai 
- [ ] add instruction for installing lit-gpt - @yifanmai 

Not sure if this already covered, but it appears LitGPTTokenizer._tokenize_do_it also needs to be updated in order to report a `'tokens'` key in the dictionary. Specifically, CachingTokenizer calls that method [here](https://github.com/stanford-crfm/helm/blob/100b1be3f0c21803994346cfb88a9e13cd4cc752/src/helm/proxy/tokenizers/caching_tokenizer.py#L95) and the in the next call expects the response dictionary to have an `'tokens'` key [here](https://github.com/stanford-crfm/helm/blob/100b1be3f0c21803994346cfb88a9e13cd4cc752/src/helm/proxy/tokenizers/caching_tokenizer.py#L51)

As an alternative, it looks like LitGPTTokenizer could override `_tokenization_raw_response_to_tokens`.

cc @JosselinSomervilleRoberts for recent changes to LitGPTTokenizer

Closing this because we should migrate to use the new LitGPT API, and also the new HELM `model_deployments.yaml` format. See #1783 for details.
Repository: stanford-crfm/helm
Issue: How to evaluate huggingface models , it is not very clear to me from the documentation 
Root cause: Documentation Deficiency

I have gone through the documentation provided here :
https://crfm-helm.readthedocs.io/en/latest/huggingface_models/

getting the following error when running the command :
 helm-run     --run-specs boolq:model=togethercomputer/RedPajama-INCITE-Chat-3B-v1     --enable-huggingface-models togethercomputer/RedPajama-INCITE-Chat-3B-v1     --suite v1     --max-eval-instances 10

error:
helm.benchmark.runner.RunnerError: Failed runs: ["boolq:model=togethercomputer_RedPajama-INCITE-Chat-3B-v1"]


Hi, could you send me the traceback of the error that caused the run to fail? You can find this in the console output; it will start with:

```
  Error when running boolq:model=togethercomputer_RedPajama-INCITE-Chat-3B-v1:
Traceback (most recent call last):
```

Hey , thank you for responding , although i got a fix by running :
helm-run --conf-paths run_specs.conf --enable-huggingface-models togethercomputer/RedPajama-INCITE-Chat-3B-v1 --suite v1 --max-eval-instances 10
Repository: stanford-crfm/helm
Issue: Support Hugging Face hosted inference API
Root cause: Unimplemented Feature Gap

https://huggingface.co/inference-api

I'm aware of at least two users asking for this, one reached out via email and the other reached out indirectly via Slack.

Note: "Text Generation Inference (TGI)" is a self-hosted version of this. https://github.com/huggingface/text-generation-inference
Repository: stanford-crfm/helm
Issue: StrategyQA and some ideas
GitHub labels: user question

scenarios

additions

Hello. I am interested in the evaluation of AI systems and LLMs both in general and specifically with respect to reading comprehension, story comprehension (e.g., [NarrativeQA](https://crfm.stanford.edu/helm/latest/?group=narrative_qa)), question-answering, and question-answering strategies.

With respect to AI evaluation, in general, I find interesting the forefront R&D topics of:
1. [automatic item generation](https://en.wikipedia.org/wiki/Automatic_Item_Generation), e.g., using templates, LLMs [1][2], or both (e.g., [Guidance](https://github.com/guidance-ai/guidance)),
2. [item response theory](https://en.wikipedia.org/wiki/Item_response_theory) and other means of measuring, comparing, and evaluating items,
3. [adaptivity](https://en.wikipedia.org/wiki/Computerized_adaptive_testing) and parallelism; while evaluating multiple instances of an AI system, items could be selected based upon the performances of those instances on previous items.

With respect to question-answering strategies, in particular, has the [StrategyQA](https://github.com/eladsegal/strategyqa) dataset been considered for HELM? Thank you.

#### References

[1] Laverghetta Jr, Antonio, and John Licato. "Generating better items for cognitive assessments using large language models." (2023).

[2] Olney, Andrew M. "Generating multiple choice questions from a textbook: LLMs match human performance on most metrics." In AIED Workshops. 2023.

Thanks for the suggestions! **StrategyQA** is very relevant to HELM for assessing implicit reasoning so I think adding it would be a good idea.

1. automatic item generation: We aren't working on this in HELM, but another researcher in the Stanford NLP group is working on a paper on something similar.
2. item response theory: We don't have any scenarios that have item difficulty annotations, though if we did, we should be able to support it in HELM (i.e. as additional per-item metadata) and use it for computing metrics. There is also perhaps an idea of using LLMs to do item difficulty annotations.
3. adaptivity: (copy and pasted from our discussion elsewhere) I can see this being useful, especially for making the evaluations more efficient by reducing the number of instances that you would have to present to the LLM to evaluate it. However, this is tricky because (1) it would require a benchmark with difficulty annotations on the instances, and (2) HELM's architecture does not support this because we sample the evaluation instances randomly before the evaluation, so there is no way to pick an evaluation instance based on previous results.

Closing this issue due to staleness - feel free to reopen if you're planning to work on StrategyQA further.
Repository: stanford-crfm/helm
Issue: Problem in `helm-summarize`, KeyError `non_english`
Root cause: Validation Gap

So, I was running HELM with this spec

```python
entries: [{
description: "raft:subset=banking_77,model=neurips/local,data_augmentation=canonical", priority: 2}]
```

And I was using Lightning AI's implementation for the server side of things. I ran the benchmark with this command

```bash
helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 1000
```
Till then, things worked great, but when I was trying to do 

```bash
helm-summarize --suite v1
```

I was getting this error

```bash
Traceback (most recent call last):
  File "/home/ec2-user/.conda/envs/crfm-helm/bin/helm-summarize", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/.conda/envs/crfm-helm/lib/python3.8/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/home/ec2-user/.conda/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 1338, in main
    summarizer.write_groups()
  File "/home/ec2-user/.conda/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 1127, in write_groups
    json.dumps(list(map(asdict_without_nones, self.create_index_tables())), indent=2),
  File "/home/ec2-user/.conda/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 658, in create_index_tables
    for subgroup in self.expand_subgroups(group):
  File "/home/ec2-user/.conda/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 612, in expand_subgroups
    return expand_subgroups_(group, visited=set())
  File "/home/ec2-user/.conda/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 606, in expand_subgroups_
    return [group] + [
  File "/home/ec2-user/.conda/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 609, in <listcomp>
    for subsubgroup in expand_subgroups_(self.schema.name_to_run_group[subgroup], visited)
KeyError: 'non_english'
```

Thanks for reporting this; this is indeed a bug. I'm making the fix in #1872.

Awesome @yifanmai, can you please tell me, any time when this feature would be integrated?



Thanks @yifanmai for fixing the bug. 

Fixed; thanks again @Anindyadeep for reporting this.
Repository: stanford-crfm/helm
Issue: Possible limit on number of requests to AI21
Root cause: Unimplemented Feature Gap

Sent > 50 requests within a minute and got back `AI21 error: 50 per 1 minute`

In general, we should implement some sort of automatic backoff for all queries...

We have automatic backoff now. I will close this issue for now and reopen if I see it again.
Repository: stanford-crfm/helm
Issue: helm-critique breaking change
Root cause: External Dependency Breakage

## Repro

Gonna pin to an older version to avoid this but I noticed there were indeed some recent changes made here that disapper if i go to an older version (the one with the decode implemented for neurips client)

From here https://github.com/llm-efficiency-challenge/neurips_llm_efficiency_challenge

```
pip install git+https://github.com/stanford-crfm/helm.git
helm-run --conf-paths run_specs_full_coarse_600_budget.conf --suite v1
```

## Logs

```
Traceback (most recent call last):
  File "/opt/conda/envs/helm/bin/helm-run", line 5, in <module>
    from helm.benchmark.run import main
  File "/opt/conda/envs/helm/lib/python3.10/site-packages/helm/benchmark/run.py", line 19, in <module>
    from .executor import ExecutionSpec
  File "/opt/conda/envs/helm/lib/python3.10/site-packages/helm/benchmark/executor.py", line 9, in <module>
    from helm.proxy.services.server_service import ServerService
  File "/opt/conda/envs/helm/lib/python3.10/site-packages/helm/proxy/services/server_service.py", line 22, in <module>
    from helm.proxy.clients.auto_client import AutoClient
  File "/opt/conda/envs/helm/lib/python3.10/site-packages/helm/proxy/clients/auto_client.py", line 21, in <module>
    from helm.proxy.critique.critique_client import CritiqueClient
ModuleNotFoundError: No module named 'helm.proxy.critique'
```

Thanks for the report.

@JosselinSomervilleRoberts could you take a look at this?

Hey @msaroufim, could you please see if #1888 solved it? I was not able to reproduce the error on my side. Thanks for reporting the bug!
Repository: stanford-crfm/helm
Issue: Update language modeling to work with AI21Tokenizer
Root cause: Interface Contract Mismatch

The [AI21Tokenizer](https://github.com/stanford-crfm/benchmarking/blob/master/src/proxy/tokenizer/ai21_tokenizer.py#L33) currently doesn't support `encode` and `decode` since we only have API access to the tokenizer. `tokenize` and a few other methods are supported.

We need to update the language modeling adapter logic to work with the Jurassic tokenizer:
https://github.com/stanford-crfm/benchmarking/blob/master/src/benchmark/adapter.py#L474

Confused by AI21's implementation. Opened an [issue](https://github.com/AI21Labs/lm-evaluation/issues/3).

@YianZhang is working on this now.

@YianZhang How is this going?
Repository: stanford-crfm/helm
Issue: Error when following display message to install all additional dependencies for HELM
Root cause: Documentation Deficiency

For some cases, HELM requires adding of additional packages and hence gives a message to install with `[all]`. Hence When I did that, it shows me this error

```
pip install helm-crfm[all]
ERROR: Could not find a version that satisfies the requirement helm-crfm[all] (from versions: none)
ERROR: No matching distribution found for helm-crfm[all]
```

Thanks for the fix!
Repository: stanford-crfm/helm
Issue: Running HELM for a local HuggingFace model
Root cause: Documentation Deficiency

Hi. Do you have some example/docs how to use HELM for a HF model loaded from a local directory (checkpoint + tokenizer)? I wonder if this is supported out of the box, I couldn't find relevant info on that. Thanks.

Hi, we have an experimental way to do this, which is described in #1501.

To summarize, if your model is at some path `/path/to/my-model`, then you can add the 
`--enable-local-huggingface-models /path/to/my-model` flag to `helm-run` and the model will become available under the model name `huggingface/my-model` (where `my-model` is taken from the final folder name of the path). So you can do something like:

```
helm-run --enable-local-huggingface-models /path/to/my-model --run-specs boolq:model=huggingface/my-model --suite v1 --max-eval-instances 10
```

Note that this is experimental and may change in future versions.

That's great, or even better because it works, thank you!

I might have another question, but will follow up in another issue if needed.
Repository: stanford-crfm/helm
Issue: Code cleanup: In run_specs.py and runner.py
GitHub labels: p2

Rename `scenario` variable to `scenario_spec` everywhere.

In `runner.py`:

- Rename `RunSpec.scenario` => `RunSpec.scenario_spec`
- Rename `RunSpec.metrics` => `RunSpec.metric_specs`

@dilarasoylu Can you make sure this is standardized; no rush, but should do at some point.
Repository: stanford-crfm/helm
Issue: Error when installing in editable mode
Root cause: Documentation Deficiency

Hello, I am following [the developer setup guidelines](https://crfm-helm.readthedocs.io/en/latest/developer_setup/) for installing in editable mode, as I want to use the library for running additional evaluations. I encountered the following issue:

```
> pip3 install --editable .
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/<my_path>/helm
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)
```

I then found [this PR](https://github.com/stanford-crfm/helm/pull/1535) saying that `setup.py` was removed as the newer pip version does not support it anymore. I then upgraded my pip and the installation worked fine.  Maybe you can update `./install_dev.sh` to force it to upgrade pip to the latest version and specify in the documentation that a recent pip version is needed?

Noted, thanks for the PR!
Repository: stanford-crfm/helm
Issue: Make sure run_specs.conf has all the scenarios listed
Root cause: Configuration Error

Currently, they are missing some that are in `run_specs.py`

Assigning this to @rishibommasani, who can make sure this represents the pilot that we want to run.

Updated the scenarios in the `pilot_conf` branch (see https://github.com/stanford-crfm/benchmarking/commit/7cf4c1244df3e067555ebd679b1520e4e3127c5c).

Currently does not bear mention to scenarios that are not in the main codebase (e.g. ICE, MS MARCO, BLiMP, BBQ, BOLD, CivilComments, Disinformation). 


Will add the augmentations + metrics, but need to figure out what gets handled through RunExpanders vs. direct specification first. Let me know if you have thoughts on how to do.
Repository: stanford-crfm/helm
Issue: which run_specs_*.conf file corresponds to core scenarios?
Root cause: Documentation Deficiency

I would like to see how my model performs against "core scenarios" published here: https://crfm.stanford.edu/helm/latest/?group=core_scenarios

which one of the run_spec*.conf file do I run to execute the same benchmarks?
The *.conf files in question are here: src/helm/benchmark/presentation


The current recommendation is to use [run_specs_lite.conf](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/presentation/run_specs_lite.conf) with `--max-eval-instances 1000` (or you can use 100 instead which is faster but lower confidence). If you only want core scenarios, you should delete all the entries after the "Knowledge" section.
Repository: stanford-crfm/helm
Issue: Data structure holding Instances by split
GitHub labels: enhancement

p3

framework

In tracing the code, it looks like we repeatedly separate TRAIN and EVAL Instances. I think we could get some simplification if we do the split once and store it in an object. Maybe something named SplitInstances. Some code pointers:

* In [get_run_instances](https://github.com/stanford-crfm/helm/blob/45e144d954f55dc4655ee4f939ab7a94dd8d2da8/src/helm/benchmark/adaptation/adapters/adapter.py#L50) we separate in order to downsample eval, then concatenate together.
* In [DataPreprocessor](https://github.com/stanford-crfm/helm/blob/45e144d954f55dc4655ee4f939ab7a94dd8d2da8/src/helm/benchmark/data_preprocessor.py#L24) we separate to conditionally augment train/eval separately, then concatenate together.
* In most Adapter.adapt calls we split again ([example](https://github.com/stanford-crfm/helm/blob/45e144d954f55dc4655ee4f939ab7a94dd8d2da8/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L31))

Probably not a big win, but I figured I'd raise it as a possible improvement.

Generally agreed, though I'd want to think about the downstream consequences.

Many scenarios also have each split in a different source data file, so this refactor would fit that data model more naturally.
Repository: stanford-crfm/helm
Issue: helm-run saves output to "benchmark_output" regardless of the --output-path flag
Root cause: Architectural Constraint

Even when specifying a different `--output-path` via the CLI flag, the benchmark runs still produce some data under the folder `benchmark_output`. The folders produced are `benchmark_output/dialect` and `benchmark_output/person_name`. This occurs with the following `run_specs.conf` file:
```
# Minimal example where we are running something non-trivial.
entries: [
  {description: "mmlu:subject=philosophy,data_augmentation=canonical,model=neurips/local", priority: 1}
]
```

Is there another flag that needs to be specified to have all run data written to the `--output-path` location, or is this a potential bug?

Edit: The command that I am running to reproduce this issue is as follows: `helm-run --conf-paths run_specs.conf --max-eval-instances 10 --suite v1 --output-path modified_benchmark_output`

Unfortunately this is a known issue - there are a number of places in the codebase in which `./benchmark_output/` is hardcoded. The specific instance syou pointed out are in [`DialectPerturbation`](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/augmentations/dialect_perturbation.py#L23) and [`PersonNamePerturbation`](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/augmentations/person_name_perturbation.py#L37C1-L37C1), there are many other instances of this, such as [`BIGBenchScenario`](https://github.com/stanford-crfm/helm/blob/2e07a0ba22196ba3a126ad04d1cc2add1c49dce3/src/helm/benchmark/run_specs.py#L1844) and [`CLEVAScenario`](https://github.com/stanford-crfm/helm/blob/2e07a0ba22196ba3a126ad04d1cc2add1c49dce3/src/helm/benchmark/scenarios/cleva_scenario.py#L20). We should clean up all of these eventually.
Repository: stanford-crfm/helm
Issue: using labelled data from LLMs
Root cause: Documentation Deficiency

I want to store the answers from multiple models on multiple tasks to do Knowledge Distillation, but I'm broke and I can't afford to run them. I was thinking of using data from this project.
Can we access full json files via an API/dataset? E.g., if we wanted to get the responses of openai/text-d
avinci-003 on babi_qa, is there a way to do it other than [looking for the run on the project site](https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=babi_qa%3Atask%3D15%2Cmodel%3Dopenai_text-davinci-003) and then downloading the full json?

Right now, the raw runs JSON is the best way to get this data. You can access it through "Full JSON" (the `scenario_state.json` file). The schema for this file is defined in this [Python dataclass](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/adaptation/scenario_state.py#L11); you could write a Python script to drill down to the specific data you need.
Repository: stanford-crfm/helm
Issue: Give truncated / paginated view of large raw results
Root cause: Unimplemented Feature Gap

Reference: https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=msmarco%3Atrack%3Dregular%2Cvalid_topk%3D30%2Cmodel%3Dai21_j1-grande%2Cdata_augmentation%3Dcanonical

I attempted to open the above page to better understand the prompts produced by msmarco. However, the page repeatedly crashes. It looks like the "Full JSON" is 4.4 GB. In cases where there is so much data, I think a good experience would be one of the following:

* Don't even try to load the data, instead require download.
* Load the first K requests, and require download for more.
* Load the first K requests and support a "Next K" button.

This is related to #1277. @farzaank has spent some time thinking about this as well.

We don't have a real backend database, so our options are:

- Use a real database, or
- Pre-split the large JSON files into smaller paged JSON

Ah. Given that #1277 already existed, I'll close this as a duplicate.