zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Regarding prompting in sentiment analysis scenario
Root cause: Documentation Deficiency

Hi,
Thanks for the great project.
I noticed in the code that the IMDB movie review sentiment analysis scenario has a prompt structure with a field of target completion. I did not find its description in the paper. Can someone please explain it to me? I think it is the actual sentiment of the review, that is provided in the prompt. Does that constitute a fair evaluation of the models if we provide the ground truth sentiment as well? (I am not sure if my understanding is correct here, would appreciate any clarifications.)
Moreover, the paper mentions a 5-shot prompting approach. I can't find that in the code for sentiment analysis. Why has the specific prompt format been selected? 
Thanks!

Hi @ishcha, you can view the prompt structure by going to a raw run page (e.g. [this run for imdb on openai/text-davinci-003](https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=imdb%3Amodel%3Dopenai_text-davinci-003%2Cdata_augmentation%3Dcanonical)) and clicking the "Prediction{trial 0}" link, which will show you the exact prompt that was sent to the model. For IMDB, it is as follows (note that I have truncated the passages here for brevity, and the model gets the full passages):

```
Passage: A romanticised and thoroughly false vision of unemployment from a middle class "artist" with a comfortable upbringing... [truncated]
Sentiment: Negative

Passage: 1939 is universally accepted as the greatest year in Hollywood history, [truncated]
Sentiment: Positive

Passage: I love killer Insects movies they are great fun to watch, [truncated]
Sentiment: Negative

Passage: Since starting to read the book this movie is based on, [truncated]
Sentiment: Positive

Passage: Ooof! This one was a stinker. [truncated]
Sentiment: Negative

Passage: Lemuel Gulliver (Ted Danson) is a doctor [truncated]
Sentiment:
```

Note that the ground truth sentiment is only provided for the training examples.

The code for this can be found in [`generation_adapter`](https://github.com/stanford-crfm/helm/blob/fa3a2f854cfae6129e865d6d98bc2a739019df15/src/helm/benchmark/adaptation/adapters/generation_adapter.py), [`in_context_learning_adapter`](https://github.com/stanford-crfm/helm/blob/fa3a2f854cfae6129e865d6d98bc2a739019df15/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py) and [`get_imdb_spec`](https://github.com/stanford-crfm/helm/blob/fa3a2f854cfae6129e865d6d98bc2a739019df15/src/helm/benchmark/run_specs.py#L1136).

The choice of five-shot is a trade-off; the number needs to be large enough to get good accuracy (e.g. [Zhao et al (2021)](https://arxiv.org/pdf/2102.09690.pdf)), but small enough so that the examples are likely to fit within the sequence length of the models.

cc @rishibommasani for questions about the paper.

Closing this for now; feel free to reopen if you have follow-up questions.
Repository: stanford-crfm/helm
Issue: how to add my new dataset
Root cause: Documentation Deficiency

hi, I want to evaluate LLM on my new dataset. How to use new dataset?Thank you!

Could you specify a bit more about what you want to do?
Are you adding a new dataset for an existing scenario or are you adding a new dataset that comes with a new scenario?

If it is a new scenario, you will need to create it in `src/helm/benchmark/scenarios`. Then in `run_specs.py`, you will need to add a function `get_<YOUR-SCENARIO>_spec()` to create a `RunSpec` and register it in `CANONICAL_RUN_SPEC_FUNCS`.

This should be a good starting point. Feel free to send me a PR (if your data is public of course) as we might be interested in adding this for everyone in HELM.

Also, make sure to ass your scenario in `schema.yaml`

Please also refer to the documentation for [setting up your development environment](https://crfm-helm.readthedocs.io/en/latest/developer_setup/) and [adding new scenarios](https://crfm-helm.readthedocs.io/en/latest/code/#adding-new-scenarios).

Closing this question as it has been answered; feel free to reopen if you have follow-up questions.
Repository: stanford-crfm/helm
Issue: Evaluate BigCode 15B
GitHub labels: p1

models

They will release the model on April 24th but have an intermediate checkpoint.

https://huggingface.co/bigcode/large-model

v0.2.4 will be released for the last week of April; do you think this will make it in, or should we plan to release this in v0.2.5 in May?

> v0.2.4 will be released for the last week of April; do you think this will make it in, or should we plan to release this in v0.2.5 in May?

I think we can make it! I was able to run HELM with their intermediate checkpoint.

OK, I'll do a canary run this week, and if everything looks good, I'll do a full run for v0.2.4.
Repository: stanford-crfm/helm
Issue: how to test model on my own dataset
Root cause: Documentation Deficiency

i want to test some models on my own dataset, but i can't find the way to add dataset. 

To add your dataset to the benchmark, create `your_scenario.py` file in the `benchmark/scenarios` directory and implement your RunSpec in the `benchmark/run_specs.py` file. This approach works for me.

how did this go? Can we close this? Is this a scenario you would like to add to HELM? In that case, could you create a PR?

Please also refer to the documentation for [setting up your development environment](https://crfm-helm.readthedocs.io/en/latest/developer_setup/) and [adding new scenarios](https://crfm-helm.readthedocs.io/en/latest/code/#adding-new-scenarios).

Closing this for now. Feel free to reopen if you have follow-up questions.
Repository: stanford-crfm/helm
Issue: Eval local HuggingFace models
Root cause: Unimplemented Feature Gap

Currently, the HuggingFace model registry + client expects HF models to be on the hub, i.e. specified by `<organization>/<model>`. Ideally, we would want to be able to eval HF models that are stored locally as well, in case they are still in development / otherwise not on the hub.
Repository: stanford-crfm/helm
Issue: The role of `num_train_trials` when evaluate
Root cause: Documentation Deficiency

If `num_train_trials>1`, I get multiple results for every input, whether this result is averaged or voted when calculating metric? It looks like that results averaged over instances and trial?

> If `num_train_trials>1`, I get multiple results for every input

`num_train_trials` is the number of trials, where each trial samples a different set of in-context examples. For example, using [this run for imdb on openai/text-davinci-003](https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=imdb%3Amodel%3Dopenai_text-davinci-003%2Cdata_augmentation%3Dcanonical)), the first trial has the prompt:

```
Passage: A romanticised and thoroughly false vision of unemployment from a middle class "artist" with a comfortable upbringing... [truncated]
Sentiment: Negative

Passage: 1939 is universally accepted as the greatest year in Hollywood history, [truncated]
Sentiment: Positive

Passage: I love killer Insects movies they are great fun to watch, [truncated]
Sentiment: Negative

Passage: Since starting to read the book this movie is based on, [truncated]
Sentiment: Positive

Passage: Ooof! This one was a stinker. [truncated]
Sentiment: Negative

Passage: Lemuel Gulliver (Ted Danson) is a doctor [truncated]
Sentiment:
```

and the second trial has the prompt:

```
Passage: Welcome to a bad ghost story and someone's nightmare. [truncated]
Sentiment: Negative

Passage: This movie was a really great flick about something that affects us all. [truncated]
Sentiment: Positive

Passage: This movie was so terrible it was almost good... almost. [truncated]
Sentiment: Negative

Passage: An enthralling, wonderful look at the films that inspired the excellent Martin Scorsese. [truncated]
Sentiment: Positive

Passage: :Spoilers:<br /><br />I was very disappointed in Love's Abiding Joy. [truncated]
Sentiment: Negative

Passage: Lemuel Gulliver (Ted Danson) is a doctor [truncated]
Sentiment:
```

Note that I have truncated the passages here for brevity, and the actual prompts contain the full passages.

> It looks like that results averaged over instances and trial?

Yes, the per-instance metrics are averaged over trials, and the scenario metrics are averaged over instances and trials.

Closing this for now; feel free to reopen if you have follow-up questions.
Repository: stanford-crfm/helm
Issue: eval local models
Root cause: Unimplemented Feature Gap

I'd like to use this framework to evaluate a local personal model, yet it's not supported now. How much work is that if I'd like to make changes to this repo? Is it possible to be done within one day by a data scientist (not CS background)?

Hi! I'm working on adding support for local HuggingFace `transformers` models this weekend. Is it a local HuggingFace model, or just a plain PyTorch model?

yes, local model based on a pretrained hugging face model. 

Great! This is my top priority. Closing in favor of #1486, should be done in the next few days.
Repository: stanford-crfm/helm
Issue: Why does the tokenizer in the HuggingfaceClient class come from HuggingfaceTokenizers, rather than HuggingfaceServer?
GitHub labels: user question

p2

models

cleanup

`tokenize` method in HuggingfaceClient class: 
```
    def tokenize(self, request: TokenizationRequest) -> TokenizationRequestResult:
        tokenizer = HuggingFaceTokenizers.get_tokenizer(request.tokenizer)
        cache_key = asdict(request)
        ...
```

another tokenizer in HuggingfaceServer class: 
```
    with htrack_block(f"Loading Hugging Face tokenizer model for config {model_config}"):
            self.tokenizer = AutoTokenizer.from_pretrained(model_config.model_id, **model_kwargs)
```

This is mostly a historical artifact: for the production runs, we use the Hugging Face tokenizer standalone for counting tokens for other models (the first code path), but we don't use any Hugging Face models directly (the second code path). I agree with you that it would be good to do a refactor to unify the two code paths.

Closing due to inactivity.
Repository: stanford-crfm/helm
Issue: How to display a new scenario to server?
Root cause: Documentation Deficiency

I tried to add the new scenario in helm and it has been working successfully for evaluation, but it doesn't show up in the "Results" in the website, we can only observe the results through "raw_runs".
We have now tried to add it in helm/benchmark/static/schema.yaml, but it still doesn't work.

The steps in https://crfm-helm.readthedocs.io/en/latest/code/ might be out-of-date. 

Could you please tell me how to fix it?

Thanks.



do u add your new scenario in schema.yaml? 

Thanks. I have fixed it，I forget to fill the group keyword in run_specs.py.



---- Replied Message ----
| From | ***@***.***> |
| Date | 05/02/2023 19:35 |
| To | ***@***.***> |
| Cc | ***@***.***>***@***.***> |
| Subject | Re: [stanford-crfm/helm] How to display a new scenario to server? (Issue #1500) |

do u add your new scenario in schema.yaml?

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>

Can we close this?
Repository: stanford-crfm/helm
Issue: Improve support of stop sequences HuggingFaceClient
Root cause: Unimplemented Feature Gap

Hi. I'm trying to run evaluation for a model hosted on Hugging Face this way:

```
helm-run \
    --run-specs boolq:model=chainyo/alpaca-lora-7b@main  \
    --enable-huggingface-models chainyo/alpaca-lora-7b@main \
    --local \
    --suite v1 \
    --max-eval-instances 10
```

but I'm getting a HuggingFace error:

```
        Loading Hugging Face tokenizer model for config chainyo/alpaca-lora-7b {████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [00:14<00:00,  3.05it/s]
        } [3m10.426s]
        HuggingFace error:
        Request failed. Retrying (attempt #2) in 254 seconds...
        HuggingFace error:
        Request failed. Retrying (attempt #3) in 264 seconds...
        HuggingFace error:
        Request failed. Retrying (attempt #4) in 284 seconds...
        HuggingFace error:
        Request failed. Retrying (attempt #5) in 324 seconds...
        HuggingFace error:
        Request failed. Retrying (attempt #6) in 404 seconds...
```

I'm trying to run it on A100 with 40GB so there should be more than enough GPU memory but I see the it feels GPU:0 almost to the maximum (32201MiB) before showing these errors.

Same problem with `databricks/dolly-v2-12b`.

I had a bit of debugging and the problem isn't in the connection but in the format of `stop_sequence_id`. See the screenshot.

![image](https://user-images.githubusercontent.com/2318534/234051995-1273f762-9423-4c15-8419-e780e2ef178b.png)


Currently, `HuggingFaceClient` only supports exactly one stop sequence, and the stop sequence must have exactly one token. I am planning to investigate [StoppingCriteria](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.StoppingCriteria) to support multiple stop sequences, which can each have multiple tokens. If this works, I will open a pull request in the next couple of weeks.

I haven't had time to do this, but StoppingCriteria / StoppingCriteriaList should work for this.
Repository: stanford-crfm/helm
Issue: Support messages argument in openai chat models
Root cause: Unimplemented Feature Gap

Hi all,

Currently, the messages argument for openai chat model is overridden with prompt: https://github.com/stanford-crfm/helm/blob/main/src/helm/proxy/clients/openai_client.py#L59. Can we make it not overridden when "messages" is given as part of the request?

I drafted a quick PR for this (#1546). Could you please check it out and see if it answers your needs>
Thanks
Repository: stanford-crfm/helm
Issue: HELM for instruction-following models
Root cause: Unimplemented Feature Gap

HELM was built during the era of few-shot in-context learning. The field is moving towards instruction-tuned models intended to be used in a zero-shot manner. We should update to HELM to support this.

[Proposal Google doc](https://docs.google.com/document/d/1AQMCq_EuVO736qIkGqMtAs1h8ea2tQeryATtHYbIh5M/edit#)

Done by [HELM instruct](https://crfm.stanford.edu/helm/instruct/), which evaluates instruction-following capability, and #2670, which adds instructions for instruction-following models on the current evaluations.
Repository: stanford-crfm/helm
Issue: Cannot Run Raft
Root cause: Validation Gap

Running `venv/bin/benchmark-run -r raft:subset=ade_corpus_v2` triggers the error below:

`FileNotFoundError: [Errno 2] No such file or directory: 'benchmark_output/scenarios/raft/data/prompt_construction_settings.json'`

I fixed this: https://github.com/stanford-crfm/benchmarking/pull/155/commits/a9e1c43c8f5356d09f86aa21adbdbb065bd3a4a8
Repository: stanford-crfm/helm
Issue: Support newer versions of Python
Root cause: Unimplemented Feature Gap

We use Python 3.8 in production, and it has been tested on Python 3.9, but we should also make sure it works on Python 3.10 and Python 3.11 for future-proofing.

+1 for Python 3.9

Concretely, this would mean adding 3.9 and 3.10 to the [Python version matrix](https://github.com/stanford-crfm/helm/blob/main/.github/workflows/test.yml#L13) in GitHub actions.

cc @andyzorigin If you run into any issues when using Python 3.10, feel free to add them here.
Repository: stanford-crfm/helm
Issue: Palmyra/Anthropic post-processing
Root cause: Validation Gap

In `src/helm/proxy/clients/palmyra_client.py`, sometimes the completion API returns an answer that starts with a space. This breaks some scenarios such as MMLU (It expects a single letter like `"A"` and does not work with something like `" A"`).

Same problem in `src/helm/proxy/clients/anthropic_client.py`. Sometimes the completions start with multiple `\n` `:` or ` `. So we query more tokens and remove them.

Right now I have added some post-processing inside the client to fix this but this is hacky as we are not returning faithfully the completions.
Ways to solve this:
- Prompt engineering
- Make scenarios more flexible (maybe this post-processing should live inside a scenario?)
- Add a new concept of `PostRunExpander` similar to a `RunExpander`.
Repository: stanford-crfm/helm
Issue: Import models and scenarios from generative search engine verifiability paper
Root cause: Unimplemented Feature Gap

We should import over the relevant models and scenarios from https://github.com/nelson-liu/evaluating-verifiability-in-generative-search-engines

cc @nelson-liu

I looked at the paper and it seems that the models are:
- Bing Chat
- NeevaAI
- perplexity.ai
- YouChat

Do we have API keys for those models?
EDIT: Nevermind, those models don't seem to have APIs

For Bing Chat, it seems like there is no official repo or API. I found this repo though: https://github.com/acheong08/EdgeGPT/tree/master/docs

For NeevaAI, I did not find anything, and they announced that they have no API: https://help.neeva.com/hc/en-us/articles/4404882342803-Developer-API

For YouChat, I found this: https://github.com/ettiebot/chatapi/tree/main

Not planned.
Repository: stanford-crfm/helm
Issue: Palmyra random_seed not used
Root cause: Configuration Error

In `src/helm/proxy/clients/palmyra_client.py`, the `random_seed` is disabled in the request as it seems like it is not doing anything on Writer's side for now. Once they change it, reenable it.

Writer has disabled their server-side caching, so we don't need to send `random` to them now (it is correctly handled inside HELM).
Repository: stanford-crfm/helm
Issue: Text normalization needs to handle Ġ
Root cause: Validation Gap

For models where the output has to be manually tokenized using a GPT-2 tokenizer, the tokens will contain a leading Ġ because of https://github.com/openai/gpt-2/issues/80. We should figure out how to deal with this; either we post-process these tokens to remove leading Ġ for these models, or we include Ġ as a whitespace character to strip in our string canonicalization.

In case it helps:

```
>>> t(" antidisestablish")['input_ids']
[1885, 29207, 40037]
>>> ids = t(" antidisestablish")['input_ids']
>>> t.convert_ids_to_tokens(ids)
['Ġant', 'idis', 'establish']
>>> [t.convert_tokens_to_string([i]) for i in t.convert_ids_to_tokens(ids)]
[' ant', 'idis', 'establish']
```

That's exactly it, thank you!

This problem is the same as the "▁" character for some Google models.
Repository: stanford-crfm/helm
Issue: Metrics for Summarization Scenario
Root cause: Documentation Deficiency

This might be my lack of understanding, but how can I access the ROUGE-2 scores for the instances on [this](https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=summarization_cnndm%3Atemperature%3D0.3%2Cdevice%3Dcpu%2Cmodel%3Danthropic_stanford-online-all-v4-s3) page? I thought I did it properly by running a `helm-run` command with the proper scenario, but it prompted me for an Anthropic API key. When I download the `Full JSON` file at the top, the ROUGE-2 and accuracy metrics aren't included in all the instances. Just the log-probabilities. Are all the instances for the scenario on the page? And if so, how can I get the metrics for them?

Thank you!

Could you share your command?
Also, I saw that you were referring to `anthropic/stanford-online-all-v4-s3`. this is Deprecated (it was an early version provided by Anthropic for us to test). We now have an official `AnthropicClient` querying their official endpoint where you can access `anthropic/claude-v1.3` and `anthropic/claude-instant-v1.0`. Make sure to check out #1484. You will need an Antropic API key though.

Regarding the missing metrics I will try to dig into this once I have your command. Feel free to also share any files that you think might be useful, like your `run_specs`.


The per-instance ROUGE scores for the run you linked to are here:

https://storage.googleapis.com/crfm-helm-public/benchmark_output/runs/v0.2.2/summarization_cnndm:temperature=0.3,device=cpu,model=anthropic_stanford-online-all-v4-s3/per_instance_stats.json

This URL is hidden at the moment. To get this URL, scroll to the bottom of the run page, copy the URL of the last JSON link, and change `stats.json` to `per_instance_stats.json`. I filed an issue to add a link from the frontend here: #1705

You will probably want to use `curl` / `wget` to get fetch this page as it may be too big to open in a browser.

Hi @kushkhosla96, I hope the previous suggestion helped. I am closing this issue due to inactivity; feel free to reopen if you have further questions.
Repository: stanford-crfm/helm
Issue: Prompt engineering for chat model that are too verbose
Root cause: Architectural Constraint

Chat models tend to be verbose which is a problem with some of our scenarios.
For example with the MMLU scenario, we expect one single letter as the answer. However some models like `anthropic/claude-v1.3` like to justify themselves first (like ChatGPT). therefore instead of answering `A` they will answer something like `Given the context, [...], the answer is A`. Once truncated, all that is left here is `Given` which is not a valid response.

For Anthropic, the way we deal with this right now is by adding a `PROMPT_ANSWER_START`.
Here is what a question looks like with `PROMPT_ANSWER_START = "The answer is":
```
Human: The following are multiple-choice questions (with answers) about anatomy.

Question: What is the embryological origin of the hyoid bone?
A. The first pharyngeal arch
B. The first and second pharyngeal arches
C. The second pharyngeal arch
D. The second and third pharyngeal arches
Answer: D

[...]

Question: The mucosa of the hard palate is
A. keratinised and has submucosa and minor salivary glands posterolaterally.
B. non-keratinised and has submucosa and minor salivary glands posteromedially.
C. keratinised and lacks submucosa and minor salivary glands.
D. non-keratinised and lacks submucosa and minor salivary glands.
Answer:

Assistant: The answer is
```

This needs to be tested. One problem is that the choice of `PROMPT_ANSWER_START` is tricky and mostly depends on the scenario. For example the coice of "The answer is" does not work well for summarization tasks. Also, the model can be quite sensitive, fro example "The answer is" gives quite good results but "The answer is:" gives terrible results.

I feel like as we are gonna have to deal with more and more chat models we should have a general way to deal with this.


I think we should take a step back and rethink adapters.  The current `AdapterSpec` was designed for in-context learning, but if we're doing zero-shot instruction following, I think we might want a completely different set of fields.

In particular, we probably want a general way to construct a prompt with explicit instructions on the output format (e.g., The answer is one of ...).  We could ask for the answer in an `<answer>...</answer>` tag.  Sometimes people use JSON format to structure their prompts too.

I would say that if there is not a reasonable way to get these instruction following models to be performant, then we should just just skip them for the old HELM and work on the new HELM.

This is more or less solved by the additional instructions in #2670.
Repository: stanford-crfm/helm
Issue: Cohere and AI21 model names are changing
Root cause: External Dependency Breakage

https://docs.cohere.com/changelog/model-names-are-changing

We should migrate cached results to reflect the new names, and then rename in them in the codebase.

The old names will work for a while, so this is not urgent.

AI21 model names have also changed: https://docs.ai21.com/changelog/jurassic-2-model-renaming-no-action-needed-thanks-to-automatic-rerouting

The affected Cohere and AI21 models are deprecated or will be deprecated soon, so this is considered wontfix.