zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Do dry run on all scenarios and estimate OpenAI token usage
Root cause: Unimplemented Feature Gap

TODO: figure out default parameters like `max_eval_instances` and number of in-context examples

Here is the token usage estimated with a dry run over most scenarios (excluding scenarios broken at the time) with `max_eval_instances=3`. 

Will run again with `max_eval_instances=1000`.

[status.txt](https://github.com/stanford-crfm/benchmarking/files/8366987/status.txt)


Some observations from https://github.com/stanford-crfm/benchmarking/files/8366987/status.txt (they may be expected, but confused me):

1. The Pile and WikiText103 override the max_eval from 3 to 10 and 6 respectively.
2. The same scenarios also have more than 2048 tokens used on average per instance. How is this possible? Is this just a miscalculation/unneeded overestimation, or is something else happening/I thought these would get truncated, but maybe that isn't true/necessary?

Here is the status for dry run with 1k max eval
[status.txt](https://github.com/stanford-crfm/benchmarking/files/8367151/status.txt)
Repository: stanford-crfm/helm
Issue: helm-summarize should ignore ref*_rank metrics
Root cause: Validation Gap

Currently there is a ton of log spam of the form: `metric name ref500_rank undefined in schema.yaml but appears in 27 run specs, including msmarco:track=trec,valid_topk=30,model=ai21_j2-grande,data_augmentation=canonical`

This isn't a "real" metric; it's a hack for the frontend to be able to display rankings of MSMARCO references. We should probably reconsider how we do this.
Repository: stanford-crfm/helm
Issue: helm-summarize fails with cannot convert float NaN to integer
Root cause: Validation Gap

```
Traceback (most recent call last):
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/bin/helm-summarize", line 8, in <module>
    sys.exit(main())
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 997, in main
    summarizer.write_groups()
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 919, in write_groups
    tables: List[Table] = self.create_group_tables_by_metric_group(group)
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 812, in create_group_tables_by_metric_group
    table = self.create_group_table(
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 719, in create_group_table
    self.create_cell(
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/presentation/summarize.py", line 540, in create_cell
    description = aggregate_stat.bare_str()
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/metrics/statistic.py", line 59, in bare_str
    f"min={process(self.min)}, "
  File "/u/scr/maiyifan/miniconda3/envs/helm-2023-04-29/lib/python3.8/site-packages/helm/benchmark/metrics/statistic.py", line 53, in process
    if int(x) == x:
ValueError: cannot convert float NaN to integer
```

Fixed by #1715.
Repository: stanford-crfm/helm
Issue: Switch to using gsutil for uploads
Root cause: Unimplemented Feature Gap

We have a homebrew script, but we should just switch to GCS

The `gsutil rsync` command is:

```
gsutil -o "GSUtil:max_upload_compression_buffer_size=16G" -m rsync -r -j json,text -x "eval_cache" /path/to/benchmark_output/runs/suite-name gs://bucket-name/path/to/benchmark_output/runs/suite-name
```

Unfortunately, this only compresses over the wire and does not compress at rest ([compressing at rest breaks file diff detection[(https://stackoverflow.com/questions/31170100/gsutil-rsync-with-gzip-compression#comment50450228_31170521)). `gsutil cp -z` supports compression at rest but not change detection.

This is the advantage of our homebrew script - it supports both compression at rest and change detection, and also supports multi-node parallelization. But I would still prefer to use `gsutil rsync` for simplicity and maintainability.
Repository: stanford-crfm/helm
Issue: Add documentation for model request and response API
GitHub labels: documentation

user question

p2

I found [this demo file](https://github.com/stanford-crfm/helm/blob/main/demo.py), but I did not find any way to understand the full python documentation.

What's the best way to go about this?

cc: @yifanmai 

Could you please explain what you meant by "the full python API"? What is the exact thing that you did not understand?

You are recommended to reading this [documentation](https://crfm-helm.readthedocs.io/en/latest/) unless you are trying to hack the package or use some low-level functions.

"Full Python API" is kind of vague. Spoke with Ayush offline, for now we want request and response calls.

For now, we should add the following to the docs:

- [ ] The [Request](https://github.com/stanford-crfm/helm/blob/main/src/helm/common/request.py#L9) format and the [Response](https://github.com/stanford-crfm/helm/blob/main/src/helm/common/request.py#L150) format
- [ ] How to configure and use `RemoteService`
- [ ] How to use `AutoClient` directly (if making requests using local credentials)

Eventually I would like to have a more Python notebook-friendly API.

Yes, this is what would be very helpful for now.

Further, some working python examples (i think Yifan means that by notebooks) where I don't use the command line at all, will be very helpful as well. For eg; I think I want to use ranking_metrics [here](https://crfm-helm.readthedocs.io/en/latest/metrics/#helm.benchmark.metrics.ranking_metrics) but I don't know how to do that right now. Maybe there is a way to understand this without examples, but its really hard right now.

@yifanmai @YianZhang 

Filed a new issue to track this and other documentation improvements: #1704
Repository: stanford-crfm/helm
Issue: Clean up protobuf --no-binary workaround
Root cause: External Dependency Breakage

We have a [weird hack](https://github.com/stanford-crfm/helm/blob/33ca6e62ecfe46c5d1a9e1073b863bc979f1e657/pre-commit.sh#L19-L20) for installing protobuf with `--no-binary`. It will probably cause issues for PyPI installation so we should get rid of it.

Example stack track:

```

src/helm/proxy/clients/auto_client.py:29: in <module>
    from .ice_tokenizer_client import ICETokenizerClient
src/helm/proxy/clients/ice_tokenizer_client.py:3: in <module>
    from icetk import icetk as tokenizer
../../anaconda3/lib/python3.10/site-packages/icetk/__init__.py:1: in <module>
    from .ice_tokenizer import IceTokenizer
../../anaconda3/lib/python3.10/site-packages/icetk/ice_tokenizer.py:14: in <module>
    from .text_tokenizer import TextTokenizer
../../anaconda3/lib/python3.10/site-packages/icetk/text_tokenizer.py:18: in <module>
    from . import sentencepiece_model_pb2 as model
../../anaconda3/lib/python3.10/site-packages/icetk/sentencepiece_model_pb2.py:34: in <module>
    _descriptor.EnumValueDescriptor(
../../anaconda3/lib/python3.10/site-packages/google/protobuf/descriptor.py:796: in __new__
    _message.Message._CheckCalledFromGeneratedFile()
E   TypeError: Descriptors cannot not be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E   
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
========================= short test summary info ==========================
ERROR src/helm/benchmark/contamination/test_compute_contamination_metrics.py - TypeError: Descriptors cannot not be created directly.
!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!
============================= 1 error in 1.32s =============================
```
Repository: stanford-crfm/helm
Issue: Documentation & Website not loading
GitHub labels: bug

user question

p1

Hey HELM Team,

Just a heads up that this documentation page is showing an error: https://crfm-helm.readthedocs.io/en/latest/models/

Also, I noticed that no data is loading on these pages, but I figure you're aware :) 

https://crfm.stanford.edu/helm/latest/?runs=1

Thanks!

Thanks for the heads up! I'm looking into it.
Repository: stanford-crfm/helm
Issue: `summarization_metrics` in `summarization_xsum` may be flaky
Root cause: Validation Gap

Saw a failed run for `summarization_xsum:temperature=0.3,device=cpu,model=together_llama-7b` with stack trace:

```
  File "/u/scr/maiyifan/miniconda3/envs/helm-together-async/lib/python3.8/site-packages/helm/benchmark/metrics/metric.py", line 77, in process
    self.metric.evaluate_generation(
  File "/u/scr/maiyifan/miniconda3/envs/helm-together-async/lib/python3.8/site-packages/helm/benchmark/metrics/summarization_metrics.py", line 177, in evaluate_generation
    self.humaneval = self._load_humaneval(eval_cache_path)
  File "/u/scr/maiyifan/miniconda3/envs/helm-together-async/lib/python3.8/site-packages/helm/benchmark/metrics/summarization_metrics.py", line 99, in _load_humaneval
    score_analyzer = SummarizationHumanEvalAnalyzer(dataset, eval_cache_path, shots=shots)
  File "/u/scr/maiyifan/miniconda3/envs/helm-together-async/lib/python3.8/site-packages/helm/benchmark/metrics/summarization_metrics.py", line 261, in __init__
    self.load_humaneval_data()
  File "/u/scr/maiyifan/miniconda3/envs/helm-together-async/lib/python3.8/site-packages/helm/benchmark/metrics/summarization_metrics.py", line 273, in load_humaneval_data
    tasks_by_id[row.HITId].append(row)
  File "/sailhome/maiyifan/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 5902, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'Series' object has no attribute 'HITId'
```

Probably caused by #1361

I am still having the same error message, with the mainline code now. Can you reopen this issue? Thanks.
Repository: stanford-crfm/helm
Issue: Weird code in code_scenario.py
Root cause: Validation Gap

This line of code  looks weird and raises an error when run with run_specs:

```
{description: "code:model=openai/davinci,dataset=humaneval", priority: 1}
{description: "code:model=openai/davinci,dataset=apps,timeout=3", priority: 1}
```

https://github.com/stanford-crfm/helm/blob/e8d29cf2c54d11b0e7a849f1c65bc7643f586d01/src/helm/benchmark/scenarios/code_scenario.py#L142

It worked when I changed it to
https://github.com/stanford-crfm/helm/blob/9cd131428c1618574280fb3d6df262b5195096aa/src/helm/benchmark/scenarios/code_scenario.py#L142

@yifanmai
Repository: stanford-crfm/helm
Issue: Better documentation and onboarding for using external models with API keys
Root cause: Documentation Deficiency

We should:

1. Restructure `AutoClient` and the clients to be more helpful about telling you which key is missing. We should have a uniform "please fill in key X in `credentials.conf`" exception message.
2. Clarify in ReadTheDocs how to use credentials:
    - where `credentials.conf` is
    - the expected format of the file (i.e. yaml)
    - the path to is (it's assumed to be under the `prod_env`, which is configurable using the `--local_path` flag)
    - that it is only used during local mode (which is controlled by `--local` in <v0.2.2 and `--server-url` >=v0.2.3)
3. Add an example in the tutorial that involves adding a key to the `credentials.conf` for an external model (we can use OpenAI for this)

Further to this I want to signal that the flag `--api-key-path` seems not to do anything
Repository: stanford-crfm/helm
Issue: Tensor dimension error on `tiiuae/falcon-7b`
Root cause: Interface Contract Mismatch

```
python -m helm.benchmark.run \
    --enable-huggingface-models tiiaue/falcon-7b \
    --run-specs mmlu:subject=anatomy,method=multiple_choice_joint,model=tiiaue/falcon-7b \
    --local --max-eval-instances 5 --suite tiiaue_falcon-7b \
    --local-path ./envs/tiiaue_falcon-7b \
    2>&1 | tee $tiiaue_falcon-7b_mmlu
```
results in the following [tensor error](https://gist.github.com/julian-q/7d099195955a002f00e96b4c0d969b63)

It would be nice to add the falcon models to the set of evaluations.

I've also had issues with it: https://github.com/huggingface/peft/issues/685 if I get it to work I will ping y'all. 

it also has eos = pad token issue which might be an issue if you fine tune the model e.g., it (un) learns to predict eos: https://stackoverflow.com/questions/76633368/why-does-the-falcon-qlora-tutorial-code-use-eos-token-as-pad-token

We've add Falcon using Together's inference API.
Repository: stanford-crfm/helm
Issue: Add model `text-embedding-ada-002`
Root cause: Unimplemented Feature Gap

From conversation with @yifanmai , requesting the addition of this embedding model.

It is the best embedding model and is low cost - [reference](https://openai.com/blog/new-and-improved-embedding-model).
The file that needs to be changed is most likely [here](https://github.com/stanford-crfm/helm/blob/5a8b8ceca3cd7db804171a06cff07b84199fccea/src/helm/proxy/models.py#L550)

@kanodiaayush We now have `text-embedding-ada-002` on the CRFM playground API. Feel free to try it out!

Thank you very much @yifanmai !! I will be using it!
Repository: stanford-crfm/helm
Issue: Merge HEIM into HELM
Root cause: Architectural Constraint

The following things would be difficult to integrate into HELM:

- Some metrics do not conform to HELM conventions (e.g. collapsing stats into `expected_` and `max_` metrics instead of providing complete stats)
- Some metrics have incorrect contexts (e.g. missing split and perturbation name)
- The way perturbations are done do not conform to HELM conventions (e.g. having one run that has the base instances and all the perturbations, and all instances are evaluated; instead, we have multiple runs of the same scenario with different perturbations)
- Sub scenario grouping does not conform to HELM conventions (e.g. the `mscoco_fid` scenario is not part of the `mscoco` group)

I think we want to add most of HEIM's functionality (e.g. models, scenarios, metrics) to HELM, but we need to modify them to follow HELM conventions. Those modifications will make their implementations diverge from this specific HEIM data release, so I don't think it'll be possible to reproduce this specific HEIM release using main HELM. We could consider doing a future updated HEIM release with all of the HELM-ified abstractions, which should reproduce most of the numbers, but the specific aggregations and schema structure may be different.

Also, reproducing the run will require re-running all the automated image metrics, which are not cached (but fortunately the MTurk evaluations won't need to be redone).

I think we should consider freezing the HEIM paper code as a fork / tag because of this.

Most of these are minor and fixable. I think it's more powerful to have a single codebase. 

Also, by the way, we cache all the slow automated image metric computation.

This seems more or less done. The bullet points in the initial comment have either been addressed, or decided as wontfix as they are sufficiently minor. We still have the schema and frontend left to merge, but that can be tracked elsewhere.
Repository: stanford-crfm/helm
Issue: Configuration file for adding models to HELM
Root cause: Unimplemented Feature Gap

Currently we provide some flags for adding models to HELM without modifying code e.g. `--enable-huggingface-model` and `--enable-local-huggingface-model` (in #1505). This becomes unwieldy if we have lots of models. We should have HELM read these models from a configuration file instead.

Ideally the config file format should be extensible to be generalizable this to other model providers e.g. it should allow users to add OpenAI models in Azure for #1415

Fixed by #1761, #1903 and #2142.
Repository: stanford-crfm/helm
Issue: Add fsspec support for loading Hugging Face models
Root cause: Unimplemented Feature Gap

#1505 allows evaluating Hugging Face models stored on local paths. We should add fsspec support, which would allow evaluating Hugging Face models stored in cloud storage.

The path is passed directly `AutoModelForCasualLM.from_pretrained()`, and I don't think there's a way to make it support fsspec. So if we wanted to support this, we would have to use fsspec to download the files to a local path, and then load from there. But perhaps the user should just do this themselves?

I think it is better for the user to do this themselves.
Repository: stanford-crfm/helm
Issue: Any hints on how to utilize the package to benchmark a user provided dataset and user provided model?
Root cause: Documentation Deficiency

Basically, I would like to utilize the code to benchmarking a local model against a personal test dataset. Is it possible? Didn't find much information in the doc. Thanks!

For running your own model, the easiest way to do this currently is by using the [Hugging Face Hub integration](https://crfm-helm.readthedocs.io/en/latest/huggingface_models/).

For adding your own data, or for non-Hugging Face models, I would recommend forking the repository and then adding the model and scenario in the code. Here's some references:

- [Developer setup](https://crfm-helm.readthedocs.io/en/latest/developer_setup/)
- [Adding a new scenario](https://crfm-helm.readthedocs.io/en/latest/code/#adding-new-scenarios)
- [Example PR for adding a new scenario](https://github.com/stanford-crfm/helm/pull/1643)
- [Adding a new model](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/)
- [Example PR for adding a new model](https://github.com/stanford-crfm/helm/pull/1491)
Repository: stanford-crfm/helm
Issue: Specifying device for models
Root cause: Unimplemented Feature Gap

Thank you for the amazing work!

Is it possible to specify what device models will use when running specs? How do we use a specific device when we have multiple GPUs or if we want to use CPUs?




Hi, we currently don't have a way of specifying the device. If you're using Hugging Face models (which is currently the only supported integration for local inference), and you are using a PyTorch model, the model will automatically be placed on device `cuda:0` if available, or CPU otherwise.

[Code link](https://github.com/stanford-crfm/helm/blob/2bb4d785f1d2f5c359f534babfae9dc3b0a24133/src/helm/proxy/clients/huggingface_client.py#L25)

We currently don't support using CPU, or specifying the GPU. A fairly easy change would be to use a shell environment variable to store this device e.g. `export HELM_HUGGINGFACE_DEVICE='cuda:1'`, and then assign this to `self.device` in the code linked above.

We also currently don't support any kind of more sophisticated model device placement (e.g. utilizing multiple GPUs by placing a copy of the model on each GPU, sharding models across multiple GPUs, etc). This would require more complicated changes.

We now support setting the device by passing the `device` parameter within the `args` for `HuggingFaceClient` in the model deployment configuration. See [this doc](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/) for more details.
Repository: stanford-crfm/helm
Issue: `ConnectionError` when using GPT-4 through the CRFM API proxy
Root cause: External Dependency Breakage

Hypothesis is that there is an upstream GPT-4 error, and the proxy server is retrying (which takes up to 15 minutes), which causes the proxy server to drop the connection instead of waiting.

I tried making a direct request to GPT-4, and it also errors out. So this is an upstream bug for OpenAI.
Repository: stanford-crfm/helm
Issue: Running against local model service
Root cause: Unimplemented Feature Gap

For the neurips competetion we woud like to be able to run Helm against a model running local to the machine.  I.e run helm in a container and have it send requests to another container. I started implementing a new local http client  following the "adding a model" readme.

However I see that there already appears to be something pretty similiar here: https://github.com/stanford-crfm/helm/blob/05ce7e8e6e8cc072bf1047d374490e0fbc7e1915/src/helm/proxy/services/server_service.py#L38

Do you think this would fit our needs? How exactly does one use this with helm-run?
I would imagine it to be something like:
```Bash
RUN echo 'entries: [{description: "mmlu:subject=philosophy,model=local_model", priority: 1}]' > run_specs.conf

helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10 --local-path="http://localhost"
```


cc @yifanmai - do you mind if we get label permissions only so we can tag features as `competition`

Granted permissions.

`ServerService` is probably too heavyweight. It basically runs a full "playground API" server (which we also call a "proxy" server elsewhere in the code), which has a lot of extra functionality that you don't need (e.g. user authentication, user quotas).

I think you would need to create a new kind of server and client that's basically subset of the full playground API:

- Make a new HTTP server library with `make_request()`, `tokenize()` and `decode()` endpoints similarly to the [full playground server](https://github.com/stanford-crfm/helm/blob/e2ab6c2d8749cc1712e7c1dd45c78dcec87e2068/src/helm/proxy/server.py#L155-L182).
- Make a new `Client` subclass for which the endpoint URL(s) is configurable, either by flag or environment variable, that only implements `make_request()`, `tokenize()` and `decode()` similarly to the [full playground client](https://github.com/stanford-crfm/helm/blob/05ce7e8e6e8cc072bf1047d374490e0fbc7e1915/src/helm/proxy/services/remote_service.py#L57-L85)

I also think that the current abstractions would make it difficult to reuse code, so it might be better to make these separate implementations.

I'm not sure what the right names of these clients and servers would be... possibly `CompetitionClient`, or `NativeClient` (because it uses "native" CRFM JSON schemas).

this feels like a feature that should be integrated into helm. I think most people will want to set up their own local models to run helm evaluation.

This was landed in #1693 there are more changes incoming to make this solution more generic 

vLLM is now the recommended way to evaluate a local model service (see [documentation](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/#vllm))