Repository: EleutherAI/lm-evaluation-harness
Issue: flan_held_in task is broken starting with commit #1741
Root cause: Algorithmic Error

Hello!

when launching the following
`lm-eval --model hf --model_args pretrained=HuggingFaceTB/SmolLM-135M-Instruct --task flan_held_in --batch_size 32` 

the evaluation won't start. 

The problem start with commit 517aadc4d05507e2fea7b7753e91e6ad7dd76847. 

If I checkout  to  a previous commit e.g. 5a7ed3eee00eca378f157abbe9baf4d406f063f2  everything is fine.


Below the error stacktrace I am getting:

```
2024-07-31:12:17:58,977 INFO     [__main__.py:272] Verbosity set to INFO
2024-07-31:12:17:59,010 INFO     [__init__.py:394] `group` and `group_alias` keys in tasks' configs will no longer be used in the next release of lm-eval. `tag` will be used to allow to call a collection of tasks just like `group`. `group` will be removed in order to not cause confusion with the new ConfigurableGroup which will be the offical way to create groups with addition of group-wide configuations.
2024-07-31:12:18:01,504 WARNING  [__main__.py:312]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2024-07-31:12:18:01,504 INFO     [__main__.py:369] Selected Tasks: ['flan_held_in']
2024-07-31:12:18:01,505 INFO     [evaluator.py:158] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-07-31:12:18:01,505 INFO     [evaluator.py:195] Initializing hf model, with arguments: {'pretrained': 'HuggingFaceTB/SmolLM-135M-Instruct'}
2024-07-31:12:18:01,512 INFO     [huggingface.py:170] Using device 'mps'
Traceback (most recent call last):
  File "/Users/mick/.pyenv/versions/llm-evalharness/bin/lm-eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/__main__.py", line 375, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/utils.py", line 395, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/evaluator.py", line 227, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 519, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 313, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 297, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 297, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 284, in _load_individual_task_or_group
    return _load_task(task_config, task=name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 163, in _load_task
    **utils.load_yaml_config(
      ^^^^^^^^^^^^^^^^^^^^^^^
  File /Users/mick/prj/lm-evaluation-harness/lm_eval/utils.py", line 434, in load_yaml_config
    yaml_dir = os.path.dirname(yaml_path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 152, in dirname
```