Repository: stanford-crfm/helm
Issue: Understanding Prompt template for different scenarios
Root cause: Documentation Deficiency

I am looking at [HELM json file for sentiment analysis](https://storage.googleapis.com/crfm-helm-public/benchmark_output/runs/v0.2.4/babi_qa:task=15,model=meta_llama-2-7b/scenario_state.json). It uses 5 (input+label) examples along test input as prompt. 

Looking at the [scenario file for imdb](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/imdb_scenario.py), it doesn't mention that it is a few shot.

How can I find the prompt structure (zero-shot/few-shot) for different scenarios while running inference? Above .py file doesn't seem to reflect that. 

Hi, the choice of zero-shot versus five-shot is made in the `AdapterSpec` i.e. the configuration for the adapter. In the case of imdb, the `AdapterSpec` is constructed [here](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/run_specs.py#L1156) using `get_generation_adapter_spec()`, which has a default value for the parameter `max_train_instances` of 5 [here](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/run_specs.py#L267). For instance, you can run imdb in one-shot or zero-shot by instead setting `max_train_instances` to 1 or 0 respectively.

Hi @yifanmai, thank you for pointing me the source. I want to understand how the evaluation metrics like `Mean win rate` is computed here for various models: https://crfm.stanford.edu/helm/latest/?group=core_scenarios. For example, if we take llama-2-7B model, `Mean win rate` is 0.633:

Is `Mean win rate` metric computation based on zero-shot inference? If it is few-shot, what is the mechanism behind selecting few-shot examples (input+label) during inference? Looking at [HELM json file for sentiment analysis](https://storage.googleapis.com/crfm-helm-public/benchmark_output/runs/v0.2.4/babi_qa:task=15,model=meta_llama-2-7b/scenario_state.json) file, it seems like it is few-shot.

I am trying to understand, if [NeurIPS Large Language Model Efficiency Challenge](https://llm-efficiency-challenge.github.io/) runs inference on IMDB dataset (or on any other dataset for inference) using llama-2-7B model, what would be the input to model? Would it be just input review or a few examples of input+review along with input review? If it is latter, as a participant, do I need to provide those examples (input + label) or will HELM take care of finding few-shot examples and passing it to model?

Hi @yifanmai could you help clarify above questions?

For tables with a "mean win rate" column, the values of "mean win rate" for a given model is computed as the fraction of head-to-head comparisons between the given model and all other models, across all the columns of the table, where the given model has a "better" value than the other model (where "better" can be greater or lesser depending on the metric). 

> Is Mean win rate metric computation based on zero-shot inference?

We currently use zero-shot for almost all scenarios, with a few exceptions (e.g. language modeling scenarios).

> If it is few-shot, what is the mechanism behind selecting few-shot examples (input+label) during inference?

We sample randomly from the examples in the train split. The sampling is not uniformly random; we try to pick training examples in a way that makes example labels more "balanced".

> Would it be just input review or a few examples of input+review along with input review?

It is the latter.

> If it is latter, as a participant, do I need to provide those examples (input + label) or will HELM take care of finding few-shot examples and passing it to model?

HELM will take care of adding the few-shot examples ot the prompt.

Thanks @yifanmai for the kind response!