zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Custom model integration
Root cause: Documentation Deficiency

What is the way today to integrate a new custom model running on non-public endpoint ?

Could you specify? Are you running your model locally or simply querying an endpoint?

If you are simply querying an endpoint, make sure to check out #1542. You will basically need to write your own `Client` (If your endpoint is non-public can you just specify an `apiKey` and `endpoint` in your config file?)

if you are running your model locally, then if it has a HugginFace interface, you will very soon be able to run it thanks to #1505, otherwise, we do not have a way right now to run custom local models.

It would be great if you could provide more information on your use case (is it a standard endpoint like Hugging Face Inference, or a custom API?)

We have general instructions for adding a model to a fork of the repo: refer to the docs on [development setup](https://crfm-helm.readthedocs.io/en/latest/developer_setup/) and [adding new models](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/).

Closing this issue as it has been answered; feel free to re-open if you have follow-up questions.

Also noting that the `--enable-huggingface-models` is currently the most common way to use custom models; refer to the [documentation](https://crfm-helm.readthedocs.io/en/latest/huggingface_models/) for how to use this flag.
Repository: stanford-crfm/helm
Issue: Error when len(prompt) + tokens_to_generate exceeds maximum sequence length
Root cause: Validation Gap

For prompts that exceed the maximum sequence length for a model, HELM currently detects this and skips examples as needs. I get the following message:

`The original constructed prompt exceeded the max context length. Removed 4 in-context examples to fit it within the context window.`

However, some servers (Megatron-LM, Anthropic) count the length of the prompt towards the maximum sequence length. HELM does not detect when the sum of these terms exceeds the sequence length and we currently get an error response from the server (with Megatron):

`Length of prompt + tokens_to_generate longer than allowed`



This is similar to #1520.

@tgale96 do you know what is this limit for Megatron?

It turns out that this issue with Megatron was caused by the +1 in the max_request_length of the GPT2WindowService, which I fixed as part of the [PR](https://github.com/stanford-crfm/helm/pull/1530/commits/a415e3f1fa1c0da55aad1d57c03425e0e42561ae). 

@yifanmai repoted this same issue with some Anthropic models, I believe. He asked me to file this issue and can probably elaborate on where he saw it.

I think Anthropic got fixed by #1553, but Palmyra is still broken: see #1571.
Repository: stanford-crfm/helm
Issue: Rename bigcode `large-model` to `starcoder`. Update entries in the cache.
Root cause: Configuration Error

https://huggingface.co/bigcode

Will you have time to do the update or would you like help?

> Will you have time to do the update or would you like help?

I can make the code change. Do you mind just updating the Mongo cache after? We should already have `large-model` aka `starcoder` results for most scenarios except maybe just a few.

Also, just need to run this one time before a big run https://huggingface.co/bigcode/starcoder#generation to download and cache the model weights.

Sure, I can update the cache.
Repository: stanford-crfm/helm
Issue: Is the data set of human evaluations for summarisation available?
Root cause: Documentation Deficiency

And if it is, where can it be found?

with kind regards

it is not available yet

We are currently working on it but it's not our highest priority right now.

I will try to come back to this once it's done!

The raw Mechanical Turk CSV exports for the human evaluations are available [here](https://worksheets.codalab.org/bundles/0x6cde27162e37430cae65af85433139a7). The code for interpreting these CSV exports can be found [here](https://github.com/stanford-crfm/helm/blob/86ffdf665ed6fcd326bbf2b15390b4c84fbddf79/src/helm/benchmark/metrics/summarization_metrics.py).

Unfortunately we don't have this data in a more user-friendly format. If there's some specific data or aggregate that you're looking for, let me know and I can provide more information.

Thank you very much for sharing!

Hi! I am trying to use the CodaLAB link but it does not seem to load -- is it still up or has it been deleted?

Thanks for the bug report - it seems to be a CodaLab issue so I forwarded this to the CodaLab team.

@oawiles Sorry for the slow response - the link should be working now.
Repository: stanford-crfm/helm
Issue: As models are added, is there a URL to view the evaluations results?
Root cause: Documentation Deficiency

This is a question/suggestion.  I assume that new models are being added and evaluated on an ongoing basis.  Can we create a webpage to view the summary evaluation results for each model?  It doesn't have to include the comprehensive comparisons that were part of the HELM paper; just a table of results for each model would be helpful.

Is this what you are looking for? https://crfm.stanford.edu/helm/latest/?group=core_scenarios

Thanks @JosselinSomervilleRoberts ; yes, this is.  There seem to be 37 models evaluated here (7 more than the 30 in the HELM paper), but I don't see, for example, GPT-4.  Do you know if that is being worked on?  Also, do you know what the procedure is for when new evaluations are done, after support for a model has been added?  

GPT-4's results has not been released yet. We are currently in the process of adding a lot of models (GPT-4 has been added to HELM) but we have not evaluated all of them and so have not published the results. For GPT-4, we are currently thinking of waiting to have proper ways to evaluate instruction models instead of publishing numbers that might be misleading because the models would not be queried properly.

What do you mean by your last questions? Could you specify, please?

@JosselinSomervilleRoberts: sorry, that question was missing the word "know" (fixed now).  I was wondering what the procedure is for determining when a model gets evaluated, and also, does the "core_scenarios" webpage get updated as part of the evaluation, or is that a separate consideration/process.  

When new models are evaluated, they will appear on the webpage. Right now we prefer to do releases and so we don't update the website for every new model but when we consider that we have enough content so that it is relevant to do a release.

If this answered all your questions, please remember to close this issue! thanks!

Great, thanks for answers!
Repository: stanford-crfm/helm
Issue: Inference_runtime does not show on Efficiency Metrics of Web Browser
Root cause: Documentation Deficiency

After running several inference case, the output folder under stats.json which collected 3 infererence_runtime successfully. But it does not on the browser http://localhost:8000/ under Efficiency Metrics of Results Page. Can anyone advise? Thanks. 

Could you paste the commands you ran?
To get results to show up you need to run:
```bash
helm-run --suite <YOUR_SUITE> <THE_REST_OF_YOUR_ARGS>
helm-summarize --suite <YOUR_SUITE>
helm-server
```
Can you confirm you followed these steps?
Thanks

Thanks for your reply. Here is the command I run:

                helm-run \
                    --run-specs boolq:model=databricks/dolly-v2-3b \
                    --enable-huggingface-models databricks/dolly-v2-3b \
                    --local \
                    --suite v1 \
                    --max-eval-instances 3

But stats.json did not show denoised inference runtime, so there is no information showed on Efficiency page as below: 

<img width="373" alt="image" src="https://github.com/stanford-crfm/helm/assets/89594861/1bfc623f-ffab-499a-9376-3552c677cad4">

Thanks for helping!

The following steps are : 

helm-summarize --suite v1
helm-server



Can anyone shed some lights on this?

This is because we don't have performance data for those models. These need to be populated and added to the JSON files here (for idealized runtimes, denoised runtimes has a similar file in the same directory): https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/efficiency_data/inference_idealized_runtimes.json.

Thanks for your reply deepakn94. I wonder how to run these efficiency data runtime.json to get denoised or idealist inference runtime? e.g. this file: https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/efficiency_data/inference_denoised_runtimes.json

The high-level procedure is described here: https://arxiv.org/pdf/2305.02440.pdf.

Code is here: https://github.com/stanford-crfm/helm-efficiency. You first need to run relevant profiling code like this: https://github.com/stanford-crfm/helm-efficiency/blob/main/scripts/gpt2.sh. This should dump a logfile into `logs/`; you can then use the code in https://github.com/stanford-crfm/helm-efficiency/blob/main/notebooks/fit_runtimes.ipynb to find the right parameters that are dumped into the relevant JSON file. 

Hi @yidinghabana, I hope the previous suggestion helped. I am closing this issue due to inactivity; feel free to reopen if you have further questions.
Repository: stanford-crfm/helm
Issue: The built-in MMLU prompt adapter doesn't reproduce paper performance 
Root cause: Configuration Error

Hi, I found the built-in MMLU prompt adapter has much lower performance, I need to adjust the default prompt to reproduce the llama paper performance. 

The built-in MMLU prompt adapter is 
```
The following are multiple choice questions (with answers) about college medicine.

Question: Which of the following is not a true statement?
A. Muscle glycogen is broken down enzymatically to glucose-1-phosphate
B. Elite endurance runners have a high proportion of Type I fibres in their leg muscles
C. Liver glycogen is important in the maintenance of the blood glucose concentration
D. Insulin promotes glucose uptake by all tissues in the body
Answer: 
```
With this default prompt, the performance is much lower and doesn't match the paper performance of llama-13b. 

The prompt used in another git is without the "Question: " prefix, which is like
```
The following are multiple choice questions (with answers) about college medicine.

Which of the following is not a true statement?
A. Muscle glycogen is broken down enzymatically to glucose-1-phosphate
B. Elite endurance runners have a high proportion of Type I fibres in their leg muscles
C. Liver glycogen is important in the maintenance of the blood glucose concentration
D. Insulin promotes glucose uptake by all tissues in the body
Answer: 
```
This prompt can exactly reproduce the paper performance of llama-13b.

Should we adjust the MMLU default prompt adapter?

Thanks for the question.  When designing HELM, we discussed quite a bit about what prompts should be used, since we know that the prompts definitely affect performance.  In the end, because we had so many scenarios and models, it seemed more principled to choose fairly simple/default prompts and we prioritized consistency, as long as the prompt was reasonable.  In this case, putting "Question: " is something that's reasonable, and in some sense, you could say that it's a failing of the LM for not being able to handle that properly.  So I think we'd prefer to keep the default prompt in this case with the knowledge that there are many things you could do to get better accuracy out of any particular LM if you tuned the prompt.

Closing this issue; please re-open if you have follow up questions.
Repository: stanford-crfm/helm
Issue: For raw runs hosted on the HELM website, why is max_eval_instance always equal to 1000?
Root cause: Documentation Deficiency

First of all, thank you so much for HELM it's a great project!

I assume the evaluations were run on more than 1000 instances. Is there a way for me to access more than the 1000 instances displayed on the raw runs HELM pages?

Thanks again,
Michael

Some reasons:

- Efficiency: Some scenarios have tens of thousands of instances, so evaluating a subset saves time and money.
- Consistency across scenarios: Since different scenarios have different number of instances, limiting to 1000 instances per scenario makes things more consistent across scenarios. This means that we would expect metrics to have similar variance across metrics.

You can run HELM if you want to evaluate on more than 1000 instances. For most scenarios, HELM will let you evaluate on the entire dataset (though note that some scenarios are sub-sampled and do not provide the whole dataset).


Hi @mahaddad, I am closing this issue due to inactivity; feel free to reopen if you have further questions.
Repository: stanford-crfm/helm
Issue: Add LLaMA, Alpaca and other Together models
Root cause: Unimplemented Feature Gap

List of models:

- llama-7b
- alpaca-7b
- vicuna-13b
- pythia-7b
- redpajama-incite-base-3b-v1
- mpt-7b

Thanks for the amazing work, Hi, is there any income plan for llama-65B? with regards.

Yes, we are planning to add llama-65B.

Closed by #1566.
Repository: stanford-crfm/helm
Issue: Decice QA behavior for no_answer questions
Root cause: Unimplemented Feature Gap

NewsQA and QuAC contain questions where the answer is not in the passage. This is nice because we test whether the model hallucinates answers. But it is tricky to expect the model to learn a specific format for abstaining (e.g., the string 'CANNOTANSWER'). Possible solutions:

- Skip these questions (not great, we are missing an opportunity and deviating from the dataset)
- Manually add text like "No answer", "not enough info" as correct references (feels somewhat arbitrary)
- Post-hoc abstaining by setting a logprob threshold (feels tricky to do it properly in the context of a single scenario, might be better suited for a calibration study)


Implemented this in #257 .
Repository: stanford-crfm/helm
Issue: `anthropic.get_tokenizer` is thread hostile
Root cause: External Dependency Breakage

See https://github.com/anthropics/anthropic-sdk-python/issues/33

Until this is fixed, we should hold a lock when calling `anthropic.get_tokenizer`.

This has been fixed upstream, so we should eventually upgrade the version of the Anthropic library and remove our lock.
Repository: stanford-crfm/helm
Issue: `palmyra-large` sequence length failures
Root cause: Validation Gap

Despite #1553, there are still failures: e.g. on `natural_qa:mode=openbook_longans,model=writer_palmyra-large,data_augmentation=canonical`

"Total number of tokens (prompt+generation) exceeded the service limit of 2048 tokens. Please make the prompt shorter or reduce the number of tokens to generate."

Related: #1558 would help us measure the actual max length. If that doens't work, perhaps we should just cap the length to something smaller than the advertised max length.

@JosselinSomervilleRoberts The token counting script could be used to debug this.

I believe #1580 should fix this issue
Repository: stanford-crfm/helm
Issue: How to handle long prompts which is over model can take? 
Root cause: Unimplemented Feature Gap

For some evaluate instances from a few scenarios, e.g. natural_qa, it has very long prompts as eval instance to send into the model.generate. How to handle such cases? Since it shows errors that prompts has been over the max prompts that model can take. I used truncation function, but it does not take the whole piece of eval instance, only cut into pieces. The first piece was entered into the model. 

Please advise. Thanks. 

Hi @yidinghabana, sorry to bother you in your issue reply thread. I have some fundamental questions regarding how to run HELM. I find you might be an expert for this package. Would you mind take a look? You may find my detailed questions in [https://github.com/stanford-crfm/helm/issues/1577](url). Really appreciate your help.

@yidinghabana We truncate the prompts in the adapter and window service. In the few-shot setting, we try to remove train instances from the prompt is needed. If even a single test instance is too long for the context length, there is nothing we can do at the moment besides naively truncating that instance. This will cause shorter sequence length models to perform badly on such scenarios. In general, HELM is more appropriate for models with longer context lengths (at least 2048 is desired for reasonable results).

Hi @yidinghabana, I am closing this issue due to inactivity. Feel free to reopen the issue if you have further questions.
Repository: stanford-crfm/helm
Issue: Error on helm-run following quick-start instructions
Root cause: Documentation Deficiency

Hi team,

I'm trying to use HELM to test performance on bunch of open-source LLMs (Cerebras, GLM, etc.) on different scenarios (e.g., code generation, common sense reasoning). I'm building the test using SageMaker notebook instance.

I'm new to HELM so I start with pip install my model in the Sagemaker environment and try following code 
`echo 'entries: [{description: "mmlu:subject=philosophy,model=huggingface/gpt2", priority: 1}]' > run_specs.conf`

`helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10`

Firstly, it returns error saying `'[Errno 2] No such file or directory: 'proxy_api_key.txt'`

Then we create a proxy_api_key.txt thru `vim proxy_api_key.txt` and insert with API from Hugging Face User Access Tokens. However, we get error `AuthenticationError(f"Invalid API key {auth.api_key}")`.

Then we replace the API key with HuggingFace GPT-2 inference API from https://huggingface.co/gpt2 and still get the same error.

We wonder how can we fix this issue. Thanks for your help!



This should be fixed with #1584, thanks for reporting the bug!

The v0.2.3 fixed this. 
Please, add the following to https://crfm-helm.readthedocs.io/en/latest/tutorial/:
For MacBook before running '$run-helm', install 'wget' via
$brew install wget 

'$pip install wget' does not work on MacBook, at least not on OS Ventura 13.4
Repository: stanford-crfm/helm
Issue: Installation instructions
Root cause: Documentation Deficiency

The installation instructions mention installing a python package (https://crfm-helm.readthedocs.io/en/latest/installation/) but all subsequent tutorial pages seems to assume a command line setup. Is it possible to update the installation instructions? 

The installation instructions are correct; the installation will add command-line entry points to your Python PATH ([see here](https://github.com/stanford-crfm/helm/blob/86ffdf665ed6fcd326bbf2b15390b4c84fbddf79/setup.cfg#L101-L108)).

If you only want the command line, you can also install HELM using [pipx](https://pypa.github.io/pipx/) instead of the pip.

If you don't need the command-line and only want to use the Python API, that is possible as well. Right now, the Python API is not well documented, but we're planning to improve this: see #1704

Hi @nishkalavallabhi, I am closing this issue due to inactivity. Feel free to reopen the issue if you have further questions.
Repository: stanford-crfm/helm
Issue: Proxy API key
Root cause: Documentation Deficiency

Thank you for the great work, and I have a possibly dumb question.

I ran `helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10` following the tutorial.

However, it shows an error, `FileNotFoundError: [Errno 2] No such file or directory: 'proxy_api_key.txt'`
I guess that this file is only available for Stanford CRFM members and affiliates. 
Am I right or are there any other ways to get a key?

Should I try `--local` instead if I can't get a key?

Thank you.

Thank you for the question! Yes, `proxy_api_key.txt` is generally for internal use. For external users,  you can use `--local`. In fact, if you pull the latest commit, local model is the default so you won't have to manually specify it.
Repository: stanford-crfm/helm
Issue: What does it mean to have non-integer number of tokens in your Results page
GitHub labels: user question

In the [Results tab](https://crfm.stanford.edu/helm/latest/?groups=1) of your page, maximums and sums for # of tokens or instances are often float. What does this mean in practice?

Could you please give a concrete example so that we will know which number you are referring to? If it is "# of tokens" for a scenario, most likely it is an average over all instances.

I see, thanks.
Repository: stanford-crfm/helm
Issue: Documentation link is dead
GitHub labels: user question

In [Tutorial](https://crfm-helm.readthedocs.io/en/latest/tutorial/) page:

> As an example, the main run_specs.conf file used for the HELM benchmarking paper can be found [here](https://github.com/stanford-crfm/helm/blob/main/src/benchmark/presentation/run_specs.conf).
Repository: stanford-crfm/helm
Issue: Safety Evaluation
Root cause: Unimplemented Feature Gap

Evaluate  safety issue  of LLMs from various aspects 

(To be continued, open by JingFeng, Hongye, ...) 

We don't have "safety" as its own category in HELM right now, but some aspects of safety (fairness, bias, toxicity) are already included in HELM. Suggestions for new safety evals are welcome!

We now have Safety and AIR-Bench scenarios and leaderboards, which covers this use case.
Repository: stanford-crfm/helm
Issue: Are metrics for all runs reported somewhere?
Root cause: Documentation Deficiency

Foe example here we have 6 runs and json reports min, max and mean. Can we get the metrics for all 6 runs?

![image](https://github.com/stanford-crfm/helm/assets/8182738/b1f19465-c8fc-48ca-843a-e1abc966bacd)


Hi, the 6 data points come from 3 train trials each in 2 runs ([here](https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=summarization_cnndm%3Atemperature%3D0.3%2Cdevice%3Dcpu%2Cmodel%3Dcohere_command-xlarge-beta) and [here](https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=summarization_cnndm%3Atemperature%3D0.3%2Cdevice%3Dcuda%2Cmodel%3Dcohere_command-xlarge-beta)) for a total of 6 trials.

Unfortunately, each run page provides the per-run statistics that are already aggregated over the 3 train trials in that run. So we don't have the trial statistics specifically.

If you really need to, it is possible to recompute the per-trial statistics using the raw per instance stats (e.g. [here](https://storage.googleapis.com/crfm-helm-public/benchmark_output/runs/v0.2.2/summarization_cnndm:temperature=0.3,device=cpu,model=cohere_command-xlarge-beta/per_instance_stats.json) and [here](https://storage.googleapis.com/crfm-helm-public/benchmark_output/runs/v0.2.2/summarization_cnndm:temperature=0.3,device=cuda,model=cohere_command-xlarge-beta/per_instance_stats.json)).

To see the raw per instance stats for a run, go to the bottom of the page, copy the "JSON" link URL under the "All Metrics" section, then change `stats.json` to `per_instance_stats.json`. These are very large files, so I'd recommend using `wget` or `curl` instead of trying to open them in a browser.

Thanks a lot for clarification.
Repository: stanford-crfm/helm
Issue: Support arXiV and PubMed datasets for summarization
Root cause: Unimplemented Feature Gap

- Datasets released with [this paper](https://aclanthology.org/N18-2097/)

- With much longer average document and summary length to test the models on:
![image](https://github.com/stanford-crfm/helm/assets/8182738/fcf2eb7d-2c2b-4418-bcf6-df29ee089836)


Thanks for the suggestion! If you're interested, you're welcome to open a PR for this - see the instructions for [developer setup](https://crfm-helm.readthedocs.io/en/latest/developer_setup/) and [adding a scenario](https://crfm-helm.readthedocs.io/en/latest/code/#adding-new-scenarios).

Closing due to staleness; feel free to reopen if you are interested in adding these datasets.