Repository: EleutherAI/lm-evaluation-harness
Issue: Missing `include_task_folder`
Root cause: Documentation Deficiency

Hi all, I had a folder with custom YAML task definitions, which I was importing via `include_task_folder`. Now this method is not available anymore: how should I include my YAML files?

Founds the relevant docs! https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md

It's not really clear to me how to use `TaskManager` tho -- I'm getting the following error:

```
  File "[..]./cli/eval-cli.py", line 71, in <module>
    main()
  File "[..]./cli/eval-cli.py", line 52, in main
    task_manager.initialize_tasks(include_path="src/backend/tasks/")
  File "[..]/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 54, in initialize_tasks
    tasks = self._get_task_and_group(task_dir)
  File "[..]/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 300, in _get_task_and_group
    tasks_and_groups[config["group"]] = {
KeyError: 'group'
```

How are we supposed to replace `include_task_folder` ?

Oh I see, if a task doesn't belong to a group this will automatically raise an error.

I assigned each of my custom tasks to a group, and now I'm getting another error:

```
{'group': 'mmlu_flan_n_shot_loglikelihood', 'task': ['mmlu_flan_n_shot_loglikelihood_stem', 'mmlu_flan_n_shot_loglikelihood_other', 'mmlu_flan_n_shot_loglikelihood_social_sciences', 'mmlu_flan_n_shot_loglikelihood_humanities']}
Traceback (most recent call last):
  File "[..]./cli/eval-cli.py", line 71, in <module>
    main()
  File "[..]./cli/eval-cli.py", line 51, in main
    task_manager = TaskManager(include_path="src/backend/tasks/")
  File "[..]/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 30, in __init__
    self._task_index = self.initialize_tasks(
  File "[..]/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 54, in initialize_tasks
    tasks = self._get_task_and_group(task_dir)
  File "[..]/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 341, in _get_task_and_group
    tasks_and_groups[group]["task"].append(task)
KeyError: 'task'
```

Ok this last error seems to happen when there is a clash between `task` and `group` names

Oh I see it now, `task_manager` needs to be passed to `simple_evaluate`

> Oh I see, if a task doesn't belong to a group this will automatically raise an error.

This should definitely not be the case--we'll push a fix if so! Looking through this method, it looks like this branch only executes if `_config_is_group` is true, which only happens when `task` is a list (and so should only happen on a benchmark's config, as far as I'm aware?)

Not also including a deprecation warning for `include_task_folder` is a mistake on our part, we added a warning in `initialize_tasks()` and `include_path()` that I hope helps. We'll try to be more clear about what components define our public interface in future--wanted to get this PR in because it should speed up startup times pretty massively, now that we're getting so many language variants / tasks with many subtasks added to the library.

cc @lintangsutawika might have an easy fix for this, if the issue persists after passing `task_manager`?

Ok in case anyone else runs into this -- the following:

```python
task_manager = TaskManager(include_path="./src/my/custom/tasks/")
results = evaluator.simple_evaluate([..], task_manager=task_manager)
```

seems to run like a charm! :) I'm babysitting some runs in case there are any issues!