zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Upgrade protobuf and clean up installation
Root cause: External Dependency Breakage

The protobuf dependency is weird because 

- it is manually installed [here](https://github.com/stanford-crfm/helm/blob/ad0ac3ee6ca8e66cb318f2936a1191a964687384/pre-commit.sh#L25) with `--no-binary`
- it is using an [old version of 3.20](https://github.com/stanford-crfm/helm/blob/ad0ac3ee6ca8e66cb318f2936a1191a964687384/requirements.txt#L41); the current version is 4.21



Duplicate of #1613
Repository: stanford-crfm/helm
Issue: Support request batching
Root cause: Unimplemented Feature Gap

OpenAI's API supports [request batching](https://platform.openai.com/docs/guides/rate-limits/batching-requests). N users (as of 2023-03-07, N = 3) have expressed interest in batching on the CRFM proxy API to OpenAI.

Edit: users want to batch requests to other models such as Megatron too (as of 2022-05-03, N = 2)

Feedback:

> That would be super useful. I've been using the OpenAI API previously. And its batching allows for much higher throughput.

N +=1 please!

This could help with Hugging Face local model inference performance by batching requests to the node.

@yifanmai @RylanSchaeffer 

I'm the maintainer of LiteLLM we provide an Open source proxy for load balancing Azure + OpenAI + Any LiteLLM supported LLM
**It can process (500+ requests/second)**

From this thread it looks like you're trying to load balance between OpenAI instance - I hope our solution makes it easier for you. **(i'd love feedback if you're trying to do this)**

## Here's the quick start:
Doc: https://docs.litellm.ai/docs/simple_proxy#load-balancing---multiple-instances-of-1-model

## Step 1 Create a Config.yaml
```python
model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_version: "2023-05-15"
      api_key: 
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: 
      api_base: https://openai-gpt-4-test-v-2.openai.azure.com/
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: 
      api_base: https://openai-gpt-4-test-v-2.openai.azure.com/
```

## Step 2: Start the litellm proxy:
```
litellm --config /path/to/config.yaml
```

## Step3 Make Request to LiteLLM proxy:
```
curl --location 'http://0.0.0.0:8000/chat/completions' \
--header 'Content-Type: application/json' \
--data ' {
      "model": "gpt-4",
      "messages": [
        {
          "role": "user",
          "content": "what llm are you"
        }
      ],
    }
'
```

Thanks @ishaan-jaff for the tips and doc references - I think we do want to integrate with LiteLLM at some point, so this is very useful.

what's missing in LiteLLM to integrated today @yifanmai ? 

@ishaan-jaff If you have spare cycles, it would be good to get HELM working end-to-end with LiteLLM, and document (or open a pull request for) the changes that need to be made. Also, there needs to be user documentation for this - your earlier comment is a good starting point.

I expect that the following may be issues: 

- The Azure OpenAI API format is [subtly different](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/switching-endpoints) from the default OpenAI API format.
- We still use the [deprecated `engine` field instead of `model`](https://github.com/stanford-crfm/helm/blob/bb43b916267bd4bbd783ad2cea22801b17a115dd/src/helm/proxy/clients/openai_client.py#L89), which will probably not work with LiteLLM.
Repository: stanford-crfm/helm
Issue: Long retry times may lead to blocked server threads
Root cause: Architectural Constraint

Currently, the [retry loop](https://github.com/stanford-crfm/helm/blob/ed812cf708cddc3e2a9c93bbc69c82354cf7e857/src/helm/proxy/retry.py#L55) for a request can take several minutes. For the proxy server, this might mean that the request handler thread gets blocked for several minutes.

It's possible that if the server is handling N requests that have entered a retry loop, where N is the number of request handling threads, it might stop responding to new requests entirely.

I'm planning to try this out empirically (on a local instance) to see if this is indeed the case.

We have enough threads on the proxy and our QPS is low. Also we're deprioritizing work on the proxy. So I will close this for now.
Repository: stanford-crfm/helm
Issue: Remove stale Cohere models
Root cause: Unimplemented Feature Gap

> Effective December 2, 2022, we will be consolidating our generative models and only serving our Medium (focused on speed) and X-Large (focused on quality). We will also be discontinuing support for our Medium embedding model. ([source](https://docs.cohere.ai/changelog/model-sizing-update-improvements))

We currently have the following [models](https://github.com/stanford-crfm/helm/blob/v0.2.0/src/helm/proxy/models.py):

- `cohere/small-20220720`
- `cohere/medium-20220720`
- `cohere/medium-20221108`
- `cohere/large-20220720`
- `cohere/xlarge-20220609`
- `cohere/xlarge-20221108`

We should archive or delete the `cohere/small-20220720` and `cohere/large-20220720`. Also we should check if `cohere/medium-20220720` and `cohere/xlarge-20220609` are still supported.

Closing this, but we should think about how to archive old models in general.
Repository: stanford-crfm/helm
Issue: imdb instance sampling is non-deterministic
Root cause: Algorithmic Error

The reason for this is that the instance order is dependent on the order of files returned by os.listdir() [here](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/imdb_scenario.py#L62), which is [arbitrary](https://docs.python.org/3/library/os.html#os.listdir). This bug already existed in [v0.1.0](https://github.com/stanford-crfm/helm/blob/v0.1.0/src/helm/benchmark/scenarios/imdb_scenario.py#L56).

This is a subtle bug because running multiple times on the same download folder usually results in the same order due to scenario download caching (the order seems to be based on some OS-dependent order). But deleting the scenario download folder and re-running it causes the order to change.

The long term fix is to use `sorted(os.listdir())` instead.

`code` has the [same problem](https://github.com/stanford-crfm/helm/blob/caba3d986b4a0e61ffe9e8498072562708ab32c2/src/helm/benchmark/scenarios/code_scenario.py#L138). We should audit the other scenarios as well.

`ice` has the same problem.

Unfortunately we recently migrated the scenarios between disks and this caused the `os.listdir` to be shuffled, so now we don't have access to the original `os.listdir` order. I might be able to reconstruct the original order based on the published evaluations, but it'll be tricky.
Repository: stanford-crfm/helm
Issue: Race condition for scenario downloads
Root cause: Resource Mishandling

We run parallel runs across multiple workers that share a common `benchmark_output` directory on NFS. Scenarios are downloaded to this directory. The same scenario can be run in more than one worker at once, which means that there can be race conditions between scenario file operations. Usually this causes a run to fail, but silent data corruption is also theoretically possible (but I have not seen any evidence of it happening yet).

Workarounds:
1. Pre-download scenarios to the NFS folder
2. Don't run a scenario in more than one worker at one time

Possible solutions:
1. Use file locks
2. Don't share `benchmark_output` between workers
Repository: stanford-crfm/helm
Issue: Excessive truncate_sequence log spam
Root cause: Validation Gap

The production logs contain too many of these warnings:
`WARNING: truncate_sequence needs to strip "\n"`

This can be done by using the `warnings`  module from the Python standard library, which will "print the first occurrence of matching warnings for each location (module + line number) where the warning is issued" by default ([link](https://docs.python.org/3/library/warnings.html#warning-filter)).
Repository: stanford-crfm/helm
Issue: Cache-only mode
Root cause: Unimplemented Feature Gap

Sometimes we want to run `helm-run` with spending disabled i.e. ensure that requests are only loaded from cache. This can be done by setting the value in the in `credentials.conf` to an empty or dummy string, but this requires the client to not use the credentials in `__init__`. Note that deleting the KV pair completely does not work (it results in a `KeyError`). We could also add a flag to switch to this mode.

We should decide what we want the use to do to enter cache-only mode, and add it to the documentation.

Easiest way to do this is to comment out the keys in `credentials.conf`.
Repository: stanford-crfm/helm
Issue: xlrd dependency
Root cause: Environment Incompatibility

Running `ice` results in `ImportError: Missing optional dependency 'xlrd'. Install xlrd >= 1.0.0 for Excel support Use pip or conda to install xlrd`. We should consider adding this as an dependency.

Additionally: `ImportError: Pandas requires version '2.0.1' or newer of 'xlrd' (version '1.0.0' currently installed).`
Repository: stanford-crfm/helm
Issue: `disinformation` scenario mutated since last version
Root cause: External Dependency Breakage

The file at the [reiteration URL](https://drive.google.com/uc?export=download&id=1uVJbsgPCHFAvH43I6SVvU3Ayo8dh-y_N) has been updated, as seen in the following diff:

```
51c51
< {"thesis": "Climate activists act like members of a religious cults", "headlines": ["More Proof Eco-activism is Becoming a Religion: NBC Has Online Booth Where You Can Confess Your Climate Change Sins"], "split": "valid", "topic": "climate"}
---
> {"thesis": "Climate activists act like members of a religious cult.", "headlines": ["More Proof Eco-activism is Becoming a Religion: NBC Has Online Booth Where You Can Confess Your Climate Change Sins"], "split": "valid", "topic": "climate"}
```

where "cults" has been changed to `cult.`

why/how did it change?

Closing since we're no longer using this dataset. Something like #1363 would prevent this from happening in the future.
Repository: stanford-crfm/helm
Issue: Max_Length becomes negative
Root cause: Algorithmic Error


```
uch profiling. Where the personal data are collected from the data subject, the data subject should also be informed whether he or she is obliged to provide the personal data and of the consequences, where he or she does not provide such data. That information may be provided in combination with standardised icons in order to give in an easily visible, intelligible and clearly legible manner,\n\nSummarize the above article.", tokenizer='huggingface/gpt2', encode=True, truncation=True, max_length=-1023)
  0%|                                                                                                                                                                                            | 0/10 [00:40<?, ?it/s]
          HuggingFace error: can't convert negative int to unsigned
          Request failed. Retrying (attempt #4) in 30 seconds...
```

When running with `helm-run --conf-paths run_specs_test.conf --local --suite test --max-eval-instances 10 -d`
The max-length somehow becomes negative: -1023
I assume this happens because of very long input lengths. So my question would be: how should I handle such long input lengths? Is there a maximum length I should set for now?

This is related to https://github.com/stanford-crfm/helm/pull/1369.

cc @rishibommasani @teetone have you seen this before? I'm guessing the issue is within Hugging Face.

seems like an HF issue, @YianZhang may also have encountered in his work on tokenization/truncation. But nope, can't say i have encountered

It only occurs for eurlexsum, but not for billsum and multilexsum. But when inspecting the actual text to be tokenized, I didn't find anything suspicious.  

Closing due to staleness. If anyone has a good repro, please reopen with repro instructions and logs.
Repository: stanford-crfm/helm
Issue: Tracker for LEXTREME
Root cause: Unimplemented Feature Gap

- [x] Multi-class classification metrics for single label classification
- [ ] Multi-label classification metrics for eval for multi-label classification
- [ ] Multi-label generation adapter for multi-label classification
- [ ] Seqeval metrics for named entity recognition
- [ ] Deal with instances getting truncated

cc @JoelNiklaus

Also:

- #1401
- #1323

Issues with `greek_legal_code_subject`:

- The label set is too large to include in the prompt; sequence length is not long enough
- Because we never tell the model what the label set is, it has trouble producing the right labels
- Also issue with F1; if some label is never included in test set and model never predicts it, then F1 is trivially 1 and pushes up macro-F1
  - Should we always include all label classes, or (existing behavior) only include label classes in the test set when computing macro-F1?


Maybe remove Greek Legal Code Subject and Chapter, and Multi Eurlex Levels 2 and 3 because they have more than 50 labels: https://arxiv.org/pdf/2301.13126.pdf (Table 2)

[Existing example](https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=entity_data_imputation%3Adataset%3DBuy%2Cmodel%3Dopenai_text-davinci-003) of classification task with 70+ labels (mostly works, but the model doesn't do very well).

Possible idea to try: ask model to say "n/a" for no label and to join multiple labels with "," _in the prompt_. It might help somewhat.

@JoelNiklaus I was curious, for the Greek etc. benchmarks, have you tried using a Greek prompt, etc?

Interesting point. I haven't tried that so far just for simplicity and consistency. Are instruction tuned models expected to perform better then? I thought they are likely to perform better with English prompts since most of the prompts in the instruction tuning datasets are in English.
Repository: stanford-crfm/helm
Issue: YALM tokenizer missing sentencepiece file
Root cause: Validation Gap

```
  File "/nlp/u/maiyifan/miniconda3/envs/helm-v0.2.1-rc1/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 167, in get_tokenizer_client
    client = YaLMTokenizerClient(cache_config=cache_config)
  File "/nlp/u/maiyifan/miniconda3/envs/helm-v0.2.1-rc1/lib/python3.8/site-packages/helm/proxy/clients/yalm_tokenizer_client.py", line 24, in __init__
    self.tokenizer = YaLMTokenizer()
  File "/nlp/u/maiyifan/miniconda3/envs/helm-v0.2.1-rc1/lib/python3.8/site-packages/helm/proxy/clients/yalm_tokenizer/yalm_tokenizer.py", line 33, in __init__
    self._tokenizer = spm.SentencePieceProcessor(model_file=vocab_file)
  File "/nlp/u/maiyifan/miniconda3/envs/helm-v0.2.1-rc1/lib/python3.8/site-packages/sentencepiece/__init__.py", line 447, in Init
    self.Load(model_file=model_file, model_proto=model_proto)
  File "/nlp/u/maiyifan/miniconda3/envs/helm-v0.2.1-rc1/lib/python3.8/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
  File "/nlp/u/maiyifan/miniconda3/envs/helm-v0.2.1-rc1/lib/python3.8/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
OSError: Not found: "src/helm/proxy/clients/yalm_tokenizer/voc_100b.sp": No such file or directory
```

We should either check in this file or auto-download it.

Also, we need to decide if this file is a packaged resource, or if it should be saved into `benchmark_output`.

The file is checked in: https://github.com/stanford-crfm/helm/blob/main/src/helm/proxy/clients/yalm_tokenizer/voc_100b.sp. That's strange - I haven't run into this error before.

Oh right, then we just need to add it to the [manifest](https://github.com/stanford-crfm/helm/blob/913895386ff08034443bf9b768999be01a55e5ee/MANIFEST.in#L1) and use [importlib resources to resolve the file path](https://github.com/stanford-crfm/helm/blob/106113fc7fbd63e50404965ffdec1ed52a45586c/src/helm/benchmark/presentation/contamination.py#L82). I can send you a PR for that.
Repository: stanford-crfm/helm
Issue: `entity_matching_scenario` does not set random seed
Root cause: Validation Gap

It uses [`DataFrame.sample`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html) [here](https://github.com/stanford-crfm/helm/blob/913895386ff08034443bf9b768999be01a55e5ee/src/helm/benchmark/scenarios/entity_matching_scenario.py#L93) but does not set `random_state`. This means that the sampled instances depend on what was run before it.

So far, we seem to be getting lucky in getting the exact same random state because when we use `run_specs.conf`, the same stuff runs before it (and some of that stuff sets the seed) but I get a different random state when the run is run in isolation (which I would like to do).

Using the following `run_specs.conf` with `--num-threads=1` seems to work (i.e. generate the same instances as what we used before):

```
entries: [
  {description: "legal_support:model=text_code", priority: 2}
  {description: "lsat_qa:model=text_code,task=all", priority: 2}
  {description: "entity_matching:model=text,dataset=Beer", priority: 2}
  {description: "entity_matching:model=text,dataset=Abt_Buy", priority: 2}
  {description: "entity_matching:model=text,dataset=Dirty_iTunes_Amazon", priority: 2}
]
```

My preferred fix would be to store the random state at the start of the scenario and persist it somewhere...
Repository: stanford-crfm/helm
Issue: `APPSMetric` in `code` scenario is extremely slow
Root cause: Algorithmic Error

It takes about 10 hours just to run `code:model=code,dataset=apps,timeout=3`.

I have encountered the same problem. Is there any solution now? @yifanmai 

Sorry, there are no updates on this issue.

I'm also not sure to what extent this can be optimized and how much is overhead. `APPSMetric` is basically attempting to execute all the generated code, which inherently takes a lot of time.
Repository: stanford-crfm/helm
Issue: Add you.com YouChat
Root cause: Unimplemented Feature Gap

References:

- https://betterapi.net/about/
- https://demo.betterapi.net/
- https://github.com/You-OpenSource/You-Python

Youchat 完全无法用了么

You.com doesn't seem to have a raw language / chat model available via API - the current APIs are RAG or search products.
Repository: stanford-crfm/helm
Issue: Helm documentation is not accessible now
GitHub labels: user question

Hi,

Helm documentation (https://crfm-helm.readthedocs.io/) seems to be not accessible now.

Could you please help to fix it?

The document is currently up. Do you have access to readthedocs.io in general, e.g. can you view the [Requests docs](https://requests.readthedocs.io/en/latest/)?

You can also generate the documentation locally. To do so, in your Python virtual environment, clone the repository, then run:

```
./pre-commit.sh
pip install -r docs/requirements.txt
mkdoc serve
```

Then go to http://localhost:8000/ in your browser.


OK thanks.
Repository: stanford-crfm/helm
Issue: enable setting `logit_bias` for `Request`
Root cause: Unimplemented Feature Gap

OpenAI completion API allows setting [`logit_bias`](https://platform.openai.com/docs/api-reference/completions/create) during generation. It would be helpful if we support this for CRFM API as well. 

Not planned because:

- This is model provider specific
- We don't need this for any scenarios (yet)
Repository: stanford-crfm/helm
Issue: Handle new versions of models deployed to the same model name
Root cause: Architectural Constraint

If a model provider publishes a new version of model to a new model name, all is well. If a model provider publishes a new version of a model to an existing model, then:

- Comparisons is annoying; users have to go to an old HELM data release version and compare it with the current one, unless we want to add version to the "model name / key" so that we can support multiple versions in a single HELM data release version.
- Caching is annoying for the same reason, because the cache key uses only the model name and not the version, meaning that a user that wants fresh version responses might get stale version responses. Possible fixes:
  - Add version to the cache key (use a "made-up" version if needed)
  - Add a way for a user to specify a cache "invalidation" time e.g. "Basically, the experimenter can say I'd like to run with timestamp Mar 2023. This adds new entries if old entries are prior to Mar 2023. The reproducer can say Feb 2023 (falls back to old models' cache) or, say, Apr 2023 (uses the Mar 2023 runs)." (@okhat's suggestion)

I think we should convince FM developers to just not do this.

IMHO, too much hassle to deal with.
Repository: stanford-crfm/helm
Issue: Make requests cache available externally
Root cause: Unimplemented Feature Gap

Some models have been deprecated e.g. #1345 and are not reachable via API. We still have the cache of the raw requests and results used to generated the HELM results. We may want to make this cache available to researchers outside CRFM.

Options:
- Publish the data dump of the request cache
- Make the CRFM proxy API available to users outside CRFM in a cache-read-only, no-spend mode

This is not planned because no external users have asked for this yet.
Repository: stanford-crfm/helm
Issue: Add LLaMA / Alpaca models
Root cause: Unimplemented Feature Gap


Good! Exciting to use it with FlexGen.

@yifanmai @wcy1231  expectations to increase LLaMA、Alpaca etc. 7B models

@julian-q For adding the Hugging Face implementation: PR #1506 is very similar. The main thing is that we need to load the weights from a path. We can abuse the `credentials.conf` for this for now by adding a key called `llamaPath` / `alpacaPath` that points to the path to the weights.

We do want #1505 eventually, but for LLaMA / Alpaca, we do actually want it in `models.py` and `schema.yaml`. That is, we want it to be a built-in model

We now have LLaMA / Alpaca

Together: #1821, #1566
Hugging Face: #1505
Repository: stanford-crfm/helm
Issue: Support Azure OpenAI in client
Root cause: Unimplemented Feature Gap

The Azure deployment of openai models uses nearly the same API for `completion`, just the setup and authentication is slightly different.
It could be nice if helm could support both.

For information:

the python openai library supports both:
https://github.com/openai/openai-python

Thanks for the suggestion. One possible way to implement this: allow user to set `openaiApiEndpoint` in `credentials.conf`, then have `AutoClient` [pass this](https://github.com/stanford-crfm/helm/blob/main/src/helm/proxy/clients/auto_client.py#L79) to `OpenAIClient`, falling back to the default endpoint if not provided.

it needs a bit more of configuration:

```
import openai
openai.api_type="azure"
openai.api_version='2022-12-01'
```

```
 raw_request = {
                "engine": "testtextdavanci003",
```
In Azure OpenAI evey model gets a user assigned name, which need to go into `engine`.

I hardcoded all this now, because I want to use as well the "helm-run" tool  and I could not figure out, if the run-spec allows to configure this.
(but this is maybe an other issue)

But know it works with some local modificatons, so the existing code is not far from have it working.

But maybe a different client is the better option, which could share most code with the openai client.


I looked at the [doc](https://learn.microsoft.com/en-us/azure/ai-services/openai/quickstart?tabs=bash&pivots=programming-language-python) and https://github.com/openai/openai-python/issues/411 might be a blocker for this. `openai.api_base` is currently only configurable as a global.