zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: dcoument better how to create run spec configurations
Root cause: Documentation Deficiency

It took me very long time and looking into the code to figure out how to create a run spec conf for:
"summarisation"
```
entries: [{description: "summarization_xsum:temperature=0.3,model=openai/text-davinci-003", priority: 1}]
```

Concrete missing docu is in my view:
1. this list is not mentioned: https://github.com/stanford-crfm/helm/blob/6791ae77930a43daf7dce224ad366343155f6c27/src/helm/benchmark/run_specs.py#L1806
  in the scenario docu here: 
  https://crfm-helm.readthedocs.io/en/latest/scenarios/
  
  So to figure out that the scenario_name is indeed "summarization_xsum" required reading the code

2. The run_spec requires at least one arguments, even though I do not want to change the temperature. "not passing any argument" does not work

3. The arguments of the class SummarizationScenario do not match (at all) with the arguments of function "get_xsum_summarization_spec"

Thanks for the feedback. Agreed that this is complicated. There are several improvements that I would like to have:

- Add better documentation of how `run_specs.py` and run expanders work and how to construct these strings (I have a draft of this)
- Put all documentation of the arguments for each function `run_specs.py` and run expanders to ReadTheDocs
- Better error messages
- Consider making the `run_specs.py` functions match the scenario names; this may break reverse compatibility though

hello, I was wondering if there was an update on this?

I'm planning to write up some documentation this week.
Repository: stanford-crfm/helm
Issue: allow disabling helm.benchmark.toxicity_metrics.ToxicityMetric in `helm-run`
Root cause: Unimplemented Feature Gap

The need for an API key for the service is quite a hurdle

Agreed; we should consider just skipping the metric if the API key is not set.

Should be fixed after https://github.com/stanford-crfm/helm/pull/1554 is merged
Repository: stanford-crfm/helm
Issue: special handling of "content validation errors"
Root cause: Validation Gap

GPT-3 (and probbaly other models) can fail on "inappropriate content".

```
OpenAI error: The response was filtered due to the prompt triggering Azure OpenAI’s content management policy
```
Probably most APIs allow to detect this specifically. (openai API does so).
In this case, it does not work to "retry" the request, and ideally it should "fail fast".


"fail-fast" is probably not the correct behavior . content validation errors should not end a full benchmark run, rather "ignore" the current instance (idealy without retrying it first during a long time )


Agreed that it should not retry the request.

We currently don't have partial run functionality (i.e. ignore some instances); if we did add partial run functionality, it should probably be user-configurable.

Writer does this as well on some models: 

```
Invalid response: {'tpe': 'fail.input', 'errors': [{'description': 'We detected unsafe content in your text inputs. Remove any unsafe content and try again.', 'key': 'fail.content.moderation.failed', 'extras': None}], 'extras': None}
```

We did something like this for the Writer models [here](https://github.com/stanford-crfm/helm/blame/e2ab6c2d8749cc1712e7c1dd45c78dcec87e2068/src/helm/proxy/clients/palmyra_client.py#L139-L150) for their content filter in #1533. It should be straightforward to do a similar thing of the Azure OpenAI models also.
Repository: stanford-crfm/helm
Issue: how to use helm evaluate our model?  
Root cause: Documentation Deficiency

Hi,

Thanks for your cool work.
I am wondering how to use the helm'code  to evaluate our model?  and Which " .py"  file I should look at first?

Thank you very much.

To run HELM on your model, clone the repository, set up a [developer environment](https://crfm-helm.readthedocs.io/en/latest/developer_setup/), and then follow [these steps to add a new model](https://github.com/stanford-crfm/helm/blob/76d4a7fe0050d4b40c23f38c0367d58952019ee0/docs/adding_new_models.md).

Then follow [these steps](https://crfm-helm.readthedocs.io/en/latest/quick_start/) to run HELM, replacing `huggingface/gpt2` with the name of your model.

> these steps

how to run all scenario in our model, can you provide a complete scenario configure script？

@yifanmai hi, I also want to run all scenaro in my lora model. can you provide a complete scenario configure script？ 

Hi, I posted an answer to the follow-up question in #1432.

Should we close this ?
Repository: stanford-crfm/helm
Issue: Why BoolQ runs 5 times of max_eval_instances?
Root cause: Documentation Deficiency

When I run this command:
```
# entries: [{description: "boolq:model="huggingface/gpt2",data_augmentation=canonical", priority: 2}]
helm-run --conf-paths run_specs.conf --local --suite v1 --max-eval-instances 5 --num-threads 1
```
The model runs a total of 25 instances.

Firstly, you probably want `description` to be `"boolq:model=huggingface/gpt2,data_augmentation=canonical"` instead of `"boolq:model="huggingface/gpt2",data_augmentation=canonical"`.

Secondly, this is because of `data_augmentation=canonical`, which adds four extra perturbations of each instance, for a total of 5 original instance + 4 * 5 perturbed instances = 25 instances. Specifically, the [canonical perturbations consists of both the the robustness and fairness perturbations](https://github.com/stanford-crfm/helm/blob/ee55451c3fc0c544a0565593ad83c367b909e2a7/src/helm/benchmark/run_expander.py#L621), and there is [1 robustness perturbation and 3 fairness perturbations](https://github.com/stanford-crfm/helm/blob/ee55451c3fc0c544a0565593ad83c367b909e2a7/src/helm/benchmark/run_expander.py#L481-L493).

You can remove `data_augmentation=canonical` if you don't want this behavior. That will run on the original instances only.

Should we close this?
Repository: stanford-crfm/helm
Issue: Adding a new language scenario (German)
Root cause: Documentation Deficiency

Hello, I want to work on adding a new scenario with the language as German. The description in docs for adding a new scenario was not clear enough about the addition of a new language. Could you please let me know where I could start? 

To add a new scenario, clone the repository, set up a [developer environment](https://crfm-helm.readthedocs.io/en/latest/developer_setup/), and then follow [these steps to add a new scenario](https://crfm-helm.readthedocs.io/en/latest/code/#adding-new-scenarios). You might also find it useful to refer to one of the pull requests that implement a scenario, such as #1332.

Adding a German scenario should be exactly the same as adding an English scenario; the framework is not aware of the language that is used in the dataset. Note that most English-only models will likely perform poorly on your dataset, so you probably want to try to run using a multi-lingual model such as text-davinci-003.



Thank you for the quick response!
Repository: stanford-crfm/helm
Issue: max_request_length v.s. max_sequence_length in window_services. 
Root cause: Documentation Deficiency

Hi,

Thanks for your work. I am wondering what the difference between `max_sequence_lenth` and `max_request_length` in windows_services. Specifically, I found that `max_request_length` is 1025 which is larger than the `max_sequence_length` (1024) in GPT2. Could you please explain why?

https://github.com/stanford-crfm/helm/blob/a1c5e4293d6b475aca567abaadf4eae8c978bd4e/src/helm/benchmark/window_services/gpt2_window_service.py#L17

I believe this is because the extra end of text token gets appended to the sequence.

@teetone could you confirm?

There's a longer explanation in the documentation [here](https://github.com/stanford-crfm/helm/blob/86ffdf665ed6fcd326bbf2b15390b4c84fbddf79/src/helm/benchmark/adaptation/adapters/language_modeling_adapter.py#L49-L67) and [here](https://github.com/stanford-crfm/helm/blob/86ffdf665ed6fcd326bbf2b15390b4c84fbddf79/src/helm/benchmark/window_services/window_service.py#L32-L45). We should probably document this more clearly, though.
Repository: stanford-crfm/helm
Issue: can you provide a complete scenario configure script？
Root cause: Documentation Deficiency

how to run all scenario in our model, can you provide a complete scenario configure script？

For the "official" HELM runs on core scenarios, we use [run_specs.conf](https://github.com/stanford-crfm/helm/blob/b344db17ee6bf3787cfbe90598c0dfbb39ffa208/src/helm/benchmark/presentation/run_specs.conf) and [run_specs_gpu.conf](https://github.com/stanford-crfm/helm/blob/b344db17ee6bf3787cfbe90598c0dfbb39ffa208/src/helm/benchmark/presentation/run_specs_gpu.conf) (these need a GPU). You would run them like this:

```
# TODO: Change these
export MODEL=your-org/your-model
export CONF_BASE_PATH=/path/to/confs

helm-run --num-train-trials 3 --max-eval-instances 1000 --priority 2 --local --models-to-run $MODEL --conf-paths $CONF_BASE_PATH/run_specs.conf
helm-run --num-train-trials 3 --max-eval-instances 1000 --priority 2 --local --models-to-run $MODEL --conf-paths $CONF_BASE_PATH/run_specs_gpu.conf
```

Note that `--models-to-run` will run only your model rather than all models.

Alternatively, you can create your own run specs conf file based on run_specs.conf with only the runs you want to run.
Repository: stanford-crfm/helm
Issue: num of eval lines in OpenBook QA
Root cause: Configuration Error


Hi! I want to ask for the number of instances in OpenBook QA eval dataset is 500, but when we test it, the result shows it runs 455 lines. We don't know why the remaining 5 instnces are missing?

Hi, could you provide the command you are running that produces this behavior? I tried ` helm-run --run-specs commonsense:dataset=openbookqa,method=multiple_choice_joint,model=simple/model1 --local --max-eval-instances 500 --suite openbookqa`  and it indeed produces 500 instances. The scenario has more than 1000 test instances and the [official HELM run](https://crfm.stanford.edu/helm/latest/?runs=1&runSpec=commonsense%3Adataset%3Dopenbookqa%2Cmethod%3Dmultiple_choice_separate_calibrated%2Cmodel%3Dopenai_text-davinci-003%2Cdata_augmentation%3Dcanonical) uses 1000 instances.

Hi @jesseLiu2000, are you still running into this issue with OpenBook QA?

We have solved it, thank you so much !
Repository: stanford-crfm/helm
Issue: scenarios and metric correspondence
Root cause: Documentation Deficiency

hi，Can each scenario test 7 metrics? I don't understand data issues related to datasets, how to measure bias、fairness and toxicity？

Hi,

Please refer to the following sections in the [paper](https://arxiv.org/pdf/2211.09110.pdf):

- Fairness: Section 4.6, C.4
- Bias: Section 4.7, 5.6, C.5, 
- Toxicity: Sections 4.8, 5.7, C.7

You can see the results here:

- Fairness: [core evaluations](https://crfm.stanford.edu/helm/latest/?group=core_scenarios#Fairness), [targeted evaluations](https://crfm.stanford.edu/helm/latest/?group=targeted_evaluations#Fairness)
- Bias: [core evaluations](https://crfm.stanford.edu/helm/latest/?group=core_scenarios#Bias), [targeted evaluations](https://crfm.stanford.edu/helm/latest/?group=targeted_evaluations#Bias)
- Toxicity: [core evaluations](https://crfm.stanford.edu/helm/latest/?group=core_scenarios#Toxicity), [targeted evaluations](https://crfm.stanford.edu/helm/latest/?group=targeted_evaluations#Toxicity)

The mapping of scenarios to fairness / bias / toxicity metrics and perturbations can be found [in the code here](https://github.com/stanford-crfm/helm/blob/b344db17ee6bf3787cfbe90598c0dfbb39ffa208/src/helm/benchmark/run_specs.py).

Let me know if you have more questions.
Repository: stanford-crfm/helm
Issue: Path error 
GitHub labels: bug

user question

framework

![website](https://user-images.githubusercontent.com/62088997/229088832-01c00a01-de7d-443c-aca9-9d05a5c55e6a.png)

Clicking on code for a specific raw run scenario on the [website results page](https://crfm.stanford.edu/helm/v0.1.0/?runs=1&runSpec=boolq%3Amodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical#metrics)), doesn't route to the github repo due to a possible error in path

![error](https://user-images.githubusercontent.com/62088997/229087980-f841fd5c-40bd-4659-9429-2c27f1c38675.png)


Hi, unfortunately this error exists in a previous archived version, and this has been fixed in subsequent versions. Could you use the latest version instead? https://crfm.stanford.edu/helm/latest/

Hello, unfortunately, the problem exists (image below) even if I use the latest version :( 

![image](https://user-images.githubusercontent.com/62088997/229556055-37a26741-3ad9-4421-b9f5-9a9a3fc0d798.png)

The url: https://crfm.stanford.edu/nlp/u/maiyifan/miniconda3/envs/helm-v0.2.1-rc1/lib/python3.8/site-packages/helm/benchmark/scenarios/bbq_scenario.py


This is indeed a bug: the code should link to the GitHub URL. Thanks for reporting!

In the meantime, you can workaround as follows: take the suffix `helm/benchmark/scenarios/bbq_scenario.py` and prefix it with `https://github.com/stanford-crfm/helm/blob/main/src/` e.g. the correct location for this file is https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/bbq_scenario.py

Closing as this was an issue with the old frontend.
Repository: stanford-crfm/helm
Issue: "LocalWindowService.truncate_from_right" may not always ensure feasible text length
Root cause: Validation Gap

Certain datasets can trigger the assertion in https://github.com/stanford-crfm/helm/blob/368fba919c0de6759d9cce6b6d051e4b077d73a9/src/helm/benchmark/window_services/local_window_service.py#L96-L97

In my local run, two tests in `lextreme` failed due to this line:

```text
{description: "lextreme:subset=multi_eurlex_level_2,model=EleutherAI/gpt-j-6B", priority: 1},
{description: "lextreme:subset=multi_eurlex_level_3,model=EleutherAI/gpt-j-6B", priority: 1}
```

Basically, `Encode(Decode(Encode_with_Truncation(...)))` may not have the same amount of tokens as the original truncated tokens.

Here's the way to reproduce the  error with sample from `multi_eurlex`:

```python3
!wget https://gist.githubusercontent.com/sxjscience/23d99b806c389e9435c95a33760311b9/raw/65a13e4db93e0e849ffa4bcc78c7c033dabe7472/truncation_encode_decode.json -O truncation_encode_decode.json

from transformers import AutoTokenizer

with open("truncation_encode_decode.json") as f:
    dat = json.load(f)

tokenizer = AutoTokenizer.from_pretrained("gpt2")

truncated_tokens = tokenizer.encode(dat['text'], truncation=True, max_length=2039, add_special_tokens=False)
tokens_in_decoded = tokenizer.encode(tokenizer.decode(truncated_tokens), add_special_tokens=False)

assert len(tokens_in_decoded) <= len(truncated_tokens), f"In Decoded: {len(tokens_in_decoded)}, Truncated: {len(truncated_tokens)}"
```

Output:

```bash
---> 12 assert len(tokens_in_decoded) <= len(truncated_tokens), f"In Decoded: {len(tokens_in_decoded)}, Truncated: {len(truncated_tokens)}"

AssertionError: In Decoded: 2040, Truncated: 2039
```

I noticed that other window services handled this corner case: https://github.com/stanford-crfm/helm/blob/368fba919c0de6759d9cce6b6d051e4b077d73a9/src/helm/benchmark/window_services/cohere_window_service.py#L140-L141

Thus, the error can be fixed via the same approach.
Repository: stanford-crfm/helm
Issue: Support using LLMs to respond to CritiqueRequests
Root cause: Unimplemented Feature Gap

Currently, we use human workers on a labeler platform such as Surge AI to respond to `CritiqueRequests`. It would be good if we could also support asking LLMs for critiques.

We would also need to come up with some way for users to configure which backend they want to use for critiques.

Fixed in #1627.
Repository: stanford-crfm/helm
Issue: Refactor WindowService and clean up context window length tags
Root cause: Architectural Constraint

The context window length system is too complicated (it requires setting a tag on the model, defining a new window service with the window context length, and instantiating the window service for the model). The sequence length could possibly be set on the model itself.

General plan:

1. Merge a unit test that pins the constant mapping of model name to sequence length.
2. For each model in `models.py`, add the sequence length.
3. Change [`fits_within_context_window`](https://github.com/stanford-crfm/helm/blob/d69a6e696e156ec175a4b6d33e18faf4e2686a7e/src/helm/benchmark/window_services/window_service.py#L86) and [`truncate_from_right`](https://github.com/stanford-crfm/helm/blob/d69a6e696e156ec175a4b6d33e18faf4e2686a7e/src/helm/benchmark/window_services/window_service.py#L94) to take in the sequence length as a parameter.
4. Change [`InContextLearningAdapter`](https://github.com/stanford-crfm/helm/blob/d69a6e696e156ec175a4b6d33e18faf4e2686a7e/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py) to get the sequence length from the model and pass in the sequence length into `fits_within_context_window` and `truncate_from_right`. The model can be found using the model name in `self.adapter_spec.model`.
5. Fix up all other unit tests.
6. Delete all the affected model tags and the associated if-else branches in `WindowServiceFactory`.
7. Remove the unit test that we added in 1.

@JosselinSomervilleRoberts this is the model sequence length refactor that I was suggesting last week. 

cc @percyliang 

This sounds great!

This is done - the sequence length and tokenizer are now set in `model_deployments.yaml`
Repository: stanford-crfm/helm
Issue: Add Cerebras-GPT
Root cause: Unimplemented Feature Gap

https://huggingface.co/cerebras/Cerebras-GPT-13B

This model probably won't be used on any leaderboards, but might be useful for local debugging because the smallest size is only [111M](https://huggingface.co/cerebras/Cerebras-GPT-111M) which should be pretty fast for CPU inference.

Adding it should be pretty straightforward and only require modifications to `model_metadata.yaml`, `model_deployments.yaml` and `tokenizer_configs.yaml` - copy and paste the configs for the Pythia models and modify as necessary.

Closing due to staleness. This model is probably too old.
Repository: stanford-crfm/helm
Issue: Prettier repr for `Instance`
GitHub labels: p2

frontend

Structured printing with good indentation as for `json.dump(string, indent=2)`. 

@lxuechen What's the context?  `scenario.json` seems to have reasonable output?

This would be useful mostly when debugging. 

But where do you want it to be printed out?

I think this is totally an optional thing. The original idea came up when I was printing out intermediate results when debugging through setting checkpoints. 


@lxuechen Can I close this?

Closing for now; we can revisit if needed but seems non-critical.
Repository: stanford-crfm/helm
Issue: helm-run: error: unrecognized arguments: --enable-huggingface-models
Root cause: Environment Incompatibility

I run the Example，but there is an error，it says：“helm-run: error: unrecognized arguments: --enable-huggingface-models stanford-crfm/BioMedLM”. THen how to run it?
# Run boolq on stanford-crfm/BioMedLM at the default main revision
helm-run \
    --run-specs boolq:model=stanford-crfm/BioMedLM \
    --enable-huggingface-models stanford-crfm/BioMedLM \
    --local \
    --suite v1 \
    --max-eval-instances 10
![image](https://user-images.githubusercontent.com/55521338/230286595-ec38800e-d879-4138-93f6-6124ee5740b3.png)


You need to install a more recent version of HELM: `pip install crfm-helm==0.2.2`
Repository: stanford-crfm/helm
Issue: question about setting of official implementation
Root cause: Documentation Deficiency

What is the setting of the official implementation,  I find the #1432 

- I see that sometimes `num_train_trials=1` in the website, and sometimes `num_train_trials=3`，but #1432 set `num_train_trials=3`.
- No method is specified in `.conf`, and the code [run_spec.py#L654](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_specs.py#L654) defaults to ` ADAPT_MULTIPLE_CHOICE_JOINT`, but there are multiple on the website page `mmlu` `multiple_choice_separate_calibrated, multiple_choice_separate_original, multiple_choice_joint`, but the json on website is also `multiple_choice_joint`


- We currently use `num_train_trials=3` for the published evaluation results.
- The `mmlu` alternative methods are found in the `run_specs_extra.conf` [here](https://github.com/stanford-crfm/helm/blob/daaccdc414e9567d612e7841e60ffe302afb0222/src/helm/benchmark/presentation/run_specs_extra.conf#L36). We only run these on a limited number of models as part of the multiple choice ablation analysis.

> * We currently use `num_train_trials=3` for the published evaluation results.
> * The `mmlu` alternative methods are found in the `run_specs_extra.conf` [here](https://github.com/stanford-crfm/helm/blob/daaccdc414e9567d612e7841e60ffe302afb0222/src/helm/benchmark/presentation/run_specs_extra.conf#L36). We only run these on a limited number of models as part of the multiple choice ablation analysis.

Thanks for your reply, I understand better.
In response to the second point, what metrics does the MMLU alternative method affect on the web page? My understanding it will not affect: https://crfm.stanford.edu/helm/latest/?group=mmlu, but will affect https://crfm.stanford.edu/helm/latest/?group=ablation_multiple_choice? Is that right？

That is correct; it does not affect the main `mmlu` group, but it affects `ablation_multiple_choice`.

If you're curious about `ablation_multiple_choice`, this was for investigating the effect of presenting multiple choice questions in different ways. The explanation and results are in [the paper](https://arxiv.org/abs/2211.09110) under 8.2 Prompting analysis > Formulation of multiple choice scenarios.

Thank you again for your reply
Repository: stanford-crfm/helm
Issue: Use tiktoken to tokenize gpt-4 and gpt-3.5-turbo
Root cause: Algorithmic Error

gpt-4 and gpt-3.5-turbo do _not_ use the same tokenizer as the gpt-3 models, [according to tiktoken](https://github.com/openai/tiktoken/blob/46287bfa493f8ccca4d927386d7ea9cc20487525/tiktoken/model.py#L15-L16) which is the official OpenAI library.

I believe this is done with #1483