Repository: EleutherAI/lm-evaluation-harness
Issue: Custom Task with MMLU don't accept `num_fewshot`?
Root cause: Documentation Deficiency

Here is my task configuration

```yaml
group: open_llm_leaderboard
task:
  - task: mmlu
    task_alias: mmlu 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: arc_challenge
    task_alias: arc 25 shot
    num_fewshot: 25
    metric_list:
      - metric: acc_norm
  - task: hellaswag
    task_alias: hellaswag 10 shot
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: hellaswag_id
    task_alias: hellaswag_id 10 shot
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: truthfulqa_mc2
    task_alias: truthfulqa 0 shot
    dataset_path: truthful_qa
    dataset_name: multiple_choice
    output_type: multiple_choice
    metric_list:
      - metric: acc
  - task: winogrande
    task_alias: winogrande 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: gsm8k
    task_alias: gsm8k 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
```

When running this task, usually for every task there is this log

```bash
2024-02-02:11:02:13,650 WARNING  [task.py:792] Task 'arc_challenge': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
2024-02-02:11:02:13,650 WARNING  [task.py:792] Task 'arc_challenge': num_fewshot > 0 but fewshot_split is None. using preconfigured rule.
```

But, for MMLU, there is no log about `num_fewshot`. Is there a way to add num fewshot to MMLU?

It should still work. The log is meant to tell you that that `num_fewshot` is set > 0 but you didn't set a specific split for the task to use as the source for few-shot samples. The default `fewshot_split` for mmlu is `dev`

There will be a log that tells you `num_fewshot` being changed if you use `--num_fewshot` through the cli.

What is `fewshot_split`? Does it mean the split from datasets library like `train`, `test`, etc?

Yep, it’s the dataset split fewshot examples will be drawn from. 

If not set, it defaults to train if available, else dev, else test (but not overlapping the docs chosen as few-shot with the doc being tested.)

After evaluation, the `num_fewshot` from mmlu is correctly follow config. I will close this.

Hey @fahadh4ilyas were you able to get a script for reproducing results from open_llm_leaderboard? if so can you send it?

@YannDubs Here is my script

```bash
MODEL_PATH=mistralai/Mistral-7B-Instruct-v0.2

NUMEXPR_MAX_THREADS=48 lm_eval --model hf \
    --model_args "pretrained=$MODEL_PATH,dtype=bfloat16,use_flash_attention_2=True" \
    --tasks open_llm_leaderboard --include_path ./open-llm-leaderboard \
    --device cuda:0 -o ./results/mistral -s -b auto:4 --verbosity DEBUG
```

And here is what inside `./open-llm-leaderboard` directory

`./open-llm-leaderboard/open-llm-leaderboard.yaml`
```yaml
group: open_llm_leaderboard
task:
  - task: mmlu
    task_alias: mmlu 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: arc_challenge
    task_alias: arc 25 shot
    num_fewshot: 25
    metric_list:
      - metric: acc_norm
  - task: hellaswag
    task_alias: hellaswag 10 shot
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: hellaswag_id
    task_alias: hellaswag_id 10 shot
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: truthfulqa_mc2
    task_alias: truthfulqa 6 shot
    dataset_path: truthful_qa
    dataset_name: multiple_choice
    output_type: multiple_choice
    metric_list:
      - metric: acc
  - task: winogrande
    task_alias: winogrande 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: gsm8k
    task_alias: gsm8k 5 shot
    num_fewshot: 5
```

@YannDubs @fahadh4ilyas We ended up including this in lm-eval. The task name is `openllm`

(Note that the averaging performed by open LLM leaderboard (over MMLU subtasks and over the different tasks) is unweighted.)

Right, will have to finish the unweighting config soon.

@lintangsutawika does `openllm` task config is equal to my open llm config? If it's equal, then I don't see the problem. I see the evaluation relatively and not absolute.