Repository: EleutherAI/lm-evaluation-harness
Issue: Can not add `mmlu` task inside benchmark?
Root cause: Validation Gap

Here is my config

```yaml
group: open_llm_leaderboard
task:
  - task: arc_challenge
    task_alias: arc 25 shot
    num_fewshot: 25
    metric_list:
      - metric: acc_norm
  - task: hellaswag
    task_alias: hellaswag 10 shot
    process_docs: !function hellaswag_utils.process_docs
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: hellaswag_id
    task_alias: hellaswag_id 10 shot
    process_docs: !function hellaswag_utils.process_docs
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: truthfulqa_mc2
    task_alias: truthfulqa 0 shot
    dataset_path: truthful_qa
    dataset_name: multiple_choice
    output_type: multiple_choice
    process_results: !function truthfulqa_utils.process_results_mc2
    metric_list:
      - metric: acc
  - task: mmlu
    task_alias: mmlu 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: winogrande
    task_alias: winogrande 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: gsm8k
    task_alias: gsm8k 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
```

Somehow because `mmlu` is not found, `lm_eval` skip all task after `mmlu` (including `mmlu` itself). But, if I do

```bash
lm_eval --tasks mmlu
```

It is working fine.

@lintangsutawika do you know what might be going on here?

`mmlu` is a group, we don't support changing attributes to groups which currently works only for a single task (and not a collection of task). An alternative ways is to make a copy of `mmlu` with the adjusted parameters. (Use `group_alias` instead of `task_alias` and set to "mmlu 5 shot")




> `mmlu` is a group, we don't support changing attributes to groups which currently works only for a single task (and not a collection of task). An alternative ways is to make a copy of `mmlu` with the adjusted parameters. (Use `group_alias` instead of `task_alias` and set to "mmlu 5 shot")
> 
> 
> 

Could you give an example? is it like this?

```yaml
  - group: mmlu
    group_alias: mmlu 5 shot
```

Or like this

```yaml
  - group: mmlu
    group_alias: mmlu 5 shot
    task:
      - mmlu_stem
      - etc.
```

> mmlu is a group, we don't support changing attributes to groups which currently works only for a single task (and not a collection of task). An alternative ways is to make a copy of mmlu with the adjusted parameters. (Use group_alias instead of task_alias and set to "mmlu 5 shot")

Could we add a more descriptive failure message in this scenario then, rather than just failing to add it to the grouping?

@fahadh4ilyas the `num_fewshot: 5` would need to be set to each mmlu task unfortunately. Might need to copy the mmlu dir in `tasks/mmlu` and change the `num_fewshot`.

@haileyschoelkopf That would be appropriate. I'll make an issue for this. But also from the usecase here, I think it's also valid to make some changes so that the above yaml config would be possible.

@lintangsutawika thanks for the explanation on mmlu, I going through similar issue but one step before. I am trying to use the lm_eval in another repo, using similar config file and trying the following [used this doc for interface ref](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage)

```
 lm_eval.tasks.include_path("/data/home/external-repo/path-to-config/")
tasks_list = [
       "arc_challenge",
    ]
    # Evaluate
    results = evaluator.simple_evaluate(
        model=args.model,
        model_args= "pretrained=EleutherAI/pythia-160m",
        tasks=tasks_list, 
        limit=args.limit,  # Limit the number of examples per task
        
    )
```
running into couple of issues:
1- it complains about `KeyError: 'Missing task arc_challenge'` somehow doesn't pick it up. if I "tasks.initialize_tasks()" first then it does not pick up `num_fewshot` and reports n-shot 0

2- it also complains about not finding `'/data/home/external-repo/path-to-config/hellaswag_utils.py'`.

I would appreciate your help on what is the correct usage/ how should I use the open-llm config in an external repo.

Hey @HamidShojanazeri 
1. The correct way is in fact to run `tasks.initialize_tasks()` first so that it registers all tasks in the eval harness. As for the `num_fewshot`, If you're using the `evaluator.simple_evaluate` I think you should be able to use the `num_fewshot` arg in it. I haven't tried myself, but it's added there when the cli is used.
2. Is `hellaswag_utils.py` part of a custom task? The utils by itself can't be included like that of a python module, but rather used by a yaml task. The `include_path` is meant to register tasks outside of the eval harness.

thanks @lintangsutawika for the reply, 
re 1: , I was interested to have a single config as shared in this issue at the top to aggregate all the tasks I want to run where each task would have a different `num_fewshot` (as above). Would it be possible to use such a config file to pass different tasks with different `num_fewshots`?

I experimented to add a new config file for each task something like adding following to `tasks/arc/arc_challenge_fewshot.yaml`
it works fine there, so I was wondering  if the single config file like above can work. 

```
include: arc_challenge.yaml
task: arc_challenge_few_shot
task_alias: arc 25 shot
num_fewshot: 25
metric_list:
  - metric: acc_norm
 ```

So more specifically we should set `num_fewshot` to some number to activate it then it reads out from config for each specific task like [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L134-L145)?

```
results = evaluator.simple_evaluate(
        model=args.model,
        model_args= "pretrained=EleutherAI/pythia-160m",
        tasks=tasks_list, 
        num_fewshot=5,
    )
```
 
 re 2: yes, it is coming from ` process_docs: !function hellaswag_utils.process_docs` in the config file
 
 ```
group: open_llm_leaderboard
task:
  - task: arc_challenge
    task_alias: arc 25 shot
    num_fewshot: 25
    metric_list:
      - metric: acc_norm
  - task: hellaswag
    task_alias: hellaswag 10 shot
    process_docs: !function hellaswag_utils.process_docs
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
```

> thanks @lintangsutawika for the reply, re 1: , I was interested to have a single config as shared in this issue at the top to aggregate all the tasks I want to run where each task would have a different `num_fewshot` (as above). Would it be possible to use such a config file to pass different tasks with different `num_fewshots`?
> 
> I experimented to add a new config file for each task something like adding following to `tasks/arc/arc_challenge_fewshot.yaml` it works fine there, so I was wondering if the single config file like above can work.
> 
> ```
> include: arc_challenge.yaml
> task: arc_challenge_few_shot
> task_alias: arc 25 shot
> num_fewshot: 25
> metric_list:
>   - metric: acc_norm
> ```
> 
> So more specifically we should set `num_fewshot` to some number to activate it then it reads out from config for each specific task like [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L134-L145)?
> 
> ```
> results = evaluator.simple_evaluate(
>         model=args.model,
>         model_args= "pretrained=EleutherAI/pythia-160m",
>         tasks=tasks_list, 
>         num_fewshot=5,
>     )
> ```
> 
> re 2: yes, it is coming from ` process_docs: !function hellaswag_utils.process_docs` in the config file
> 
> ```
> group: open_llm_leaderboard
> task:
>  - task: arc_challenge
>    task_alias: arc 25 shot
>    num_fewshot: 25
>    metric_list:
>      - metric: acc_norm
>  - task: hellaswag
>    task_alias: hellaswag 10 shot
>    process_docs: !function hellaswag_utils.process_docs
>    num_fewshot: 10
>    metric_list:
>      - metric: acc_norm
> ```

my solution is adding myself `hellaswag_utils.py` file. I got it from [lm_eval/tasks/hellaswag/utils.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/10488d0d2cfbfd2d73416e92eb30c4a74ac3920b/lm_eval/tasks/hellaswag/utils.py)

> @fahadh4ilyas the `num_fewshot: 5` would need to be set to each mmlu task unfortunately. Might need to copy the mmlu dir in `tasks/mmlu` and change the `num_fewshot`.
> 
> @haileyschoelkopf That would be appropriate. I'll make an issue for this. But also from the usecase here, I think it's also valid to make some changes so that the above yaml config would be possible.

I tried to do this

```yaml
group: open_llm_leaderboard
task:
  - group: mmlu
    group_alias: mmlu 5 shot
    task:
      - mmlu_stem
      - mmlu_other
      - mmlu_social_sciences
      - mmlu_humanities
    num_fewshot: 5
    metric_list:
      - metric: acc
```

and I got this error. Could you please give me an example?

```python
2024-01-18:12:50:41,971 INFO     [config.py:58] PyTorch version 2.1.2 available.
/home/fahadh/anaconda3/envs/merging/lib/python3.10/site-packages/torch/cuda/__init__.py:611: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
2024-01-18:12:50:46,266 INFO     [__main__.py:156] Verbosity set to DEBUG
2024-01-18:12:50:54,640 WARNING  [__init__.py:185] Unexpected error loading config in
                                 /home/fahadh/lm-evaluation-harness/lm_eval/tasks/benchmarks/open_llm_leaderboard.yaml
                                 Config will not be added to registry
                                 Error: unhashable type: 'list'
                                 Traceback: Traceback (most recent call last):
  File "/home/fahadh/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 173, in include_task_folder
    register_configurable_group(config, yaml_path)
  File "/home/fahadh/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 69, in register_configurable_group
    if task_name in ALL_TASKS:
TypeError: unhashable type: 'list'
```

> > thanks @lintangsutawika for the reply, re 1: , I was interested to have a single config as shared in this issue at the top to aggregate all the tasks I want to run where each task would have a different `num_fewshot` (as above). Would it be possible to use such a config file to pass different tasks with different `num_fewshots`?
> > I experimented to add a new config file for each task something like adding following to `tasks/arc/arc_challenge_fewshot.yaml` it works fine there, so I was wondering if the single config file like above can work.
> > ```
> > include: arc_challenge.yaml
> > task: arc_challenge_few_shot
> > task_alias: arc 25 shot
> > num_fewshot: 25
> > metric_list:
> >   - metric: acc_norm
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > So more specifically we should set `num_fewshot` to some number to activate it then it reads out from config for each specific task like [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L134-L145)?
> > ```
> > results = evaluator.simple_evaluate(
> >         model=args.model,
> >         model_args= "pretrained=EleutherAI/pythia-160m",
> >         tasks=tasks_list, 
> >         num_fewshot=5,
> >     )
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > re 2: yes, it is coming from ` process_docs: !function hellaswag_utils.process_docs` in the config file
> > ```
> > group: open_llm_leaderboard
> > task:
> >  - task: arc_challenge
> >    task_alias: arc 25 shot
> >    num_fewshot: 25
> >    metric_list:
> >      - metric: acc_norm
> >  - task: hellaswag
> >    task_alias: hellaswag 10 shot
> >    process_docs: !function hellaswag_utils.process_docs
> >    num_fewshot: 10
> >    metric_list:
> >      - metric: acc_norm
> > ```
> 
> my solution is adding myself `hellaswag_utils.py` file. I got it from [lm_eval/tasks/hellaswag/utils.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/10488d0d2cfbfd2d73416e92eb30c4a74ac3920b/lm_eval/tasks/hellaswag/utils.py)

Thanks @fahadh4ilyas for clarification. So @lintangsutawika this clarifies 2 for my question. 

@HamidShojanazeri I may have misunderstood your context. If you're trying to list all tasks you're interested in within 1 config file, you can differentiate each num_fewshot. There was a bug earlier regarding the parameters that were set with`!function` that has been fixed. You can remove those lines from the config file.

@fahadh4ilyas will take a look

@fahadh4ilyas checkout #1321 it should allow the above yaml to work. It's still a WIP since there are a few things that still need to be tested. But for your usecase, it should work.