Repository: EleutherAI/lm-evaluation-harness
Issue: IndexError: index out of bounds for multiple tasks of mmlu_pro 
Root cause: Validation Gap

I got "IndexError: Invalid key: 0 is out of bounds for size 0" when I ran multiple tasks of mmlu_pro.

```
 accelerate launch --multi_gpu --num_processe 4 -m lm_eval --model hf --tasks mmlu_pro_biology,mmlu_pro_business,mmlu_pro_chemistry,mmlu_pro_computer_science,mmlu_pro_economics,mmlu_pro_engineering,mmlu_pro_health,mmlu_pro_history,mmlu_pro_law,mmlu_pro_math,mmlu_pro_other,mmlu_pro_philosophy,mmlu_pro_physics,mmlu_pro_psychology pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,parallelize=True --batch_size auto --limit 20 --output_path output --write_out --log_samples --apply_chat_template

[rank1]: Traceback (most recent call last):
[rank1]:   File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank1]:     return _run_code(code, main_globals, None,
[rank1]:   File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code
[rank1]:     exec(code, run_globals)
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 461, in <module>
[rank1]:     cli_evaluate()
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
[rank1]:     results = evaluator.simple_evaluate(
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
[rank1]:     return fn(*args, **kwargs)
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 227, in simple_evaluate
[rank1]:     task_dict = get_task_dict(tasks, task_manager)
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 616, in get_task_dict
[rank1]:     task_name_from_string_dict = task_manager.load_task_or_group(
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 410, in load_task_or_group
[rank1]:     collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 310, in _load_individual_task_or_group
[rank1]:     return _load_task(task_config, task=name_or_config)
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 276, in _load_task
[rank1]:     task_object = ConfigurableTask(config=config)
[rank1]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/api/task.py", line 881, in __init__
[rank1]:     test_doc = self.task_docs[0]
[rank1]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2866, in __getitem__
[rank1]:     return self._getitem(key)
[rank1]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2850, in _getitem
[rank1]:     pa_subtable = query_table(self._data, key, indices=self._indices)
[rank1]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 587, in query_table
[rank1]:     _check_valid_index_key(key, size)
[rank1]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 527, in _check_valid_index_key
[rank1]:     raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
[rank1]: IndexError: Invalid key: 0 is out of bounds for size 0
[rank2]: Traceback (most recent call last):
[rank2]:   File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank2]:     return _run_code(code, main_globals, None,
[rank2]:   File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code
[rank2]:     exec(code, run_globals)
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 461, in <module>
[rank2]:     cli_evaluate()
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
[rank2]:     results = evaluator.simple_evaluate(
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
[rank2]:     return fn(*args, **kwargs)
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 227, in simple_evaluate
[rank2]:     task_dict = get_task_dict(tasks, task_manager)
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 616, in get_task_dict
[rank2]:     task_name_from_string_dict = task_manager.load_task_or_group(
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 410, in load_task_or_group
[rank2]:     collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 310, in _load_individual_task_or_group
[rank2]:     return _load_task(task_config, task=name_or_config)
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 276, in _load_task
[rank2]:     task_object = ConfigurableTask(config=config)
[rank2]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/api/task.py", line 881, in __init__
[rank2]:     test_doc = self.task_docs[0]
[rank2]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2866, in __getitem__
[rank2]:     return self._getitem(key)
[rank2]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2850, in _getitem
[rank2]:     pa_subtable = query_table(self._data, key, indices=self._indices)
[rank2]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 587, in query_table
[rank2]:     _check_valid_index_key(key, size)
[rank2]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 527, in _check_valid_index_key
[rank2]:     raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
[rank2]: IndexError: Invalid key: 0 is out of bounds for size 0
[rank0]: Traceback (most recent call last):
[rank0]:   File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 461, in <module>
[rank0]:     cli_evaluate()
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
[rank0]:     results = evaluator.simple_evaluate(
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 227, in simple_evaluate
[rank0]:     task_dict = get_task_dict(tasks, task_manager)
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 616, in get_task_dict
[rank0]:     task_name_from_string_dict = task_manager.load_task_or_group(
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 410, in load_task_or_group
[rank0]:     collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 310, in _load_individual_task_or_group
[rank0]:     return _load_task(task_config, task=name_or_config)
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 276, in _load_task
[rank0]:     task_object = ConfigurableTask(config=config)
[rank0]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/api/task.py", line 881, in __init__
[rank0]:     test_doc = self.task_docs[0]
[rank0]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2866, in __getitem__
[rank0]:     return self._getitem(key)
[rank0]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2850, in _getitem
[rank0]:     pa_subtable = query_table(self._data, key, indices=self._indices)
[rank0]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 587, in query_table
[rank0]:     _check_valid_index_key(key, size)
[rank0]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 527, in _check_valid_index_key
[rank0]:     raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
[rank0]: IndexError: Invalid key: 0 is out of bounds for size 0
[rank3]: Traceback (most recent call last):
[rank3]:   File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank3]:     return _run_code(code, main_globals, None,
[rank3]:   File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code
[rank3]:     exec(code, run_globals)
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 461, in <module>
[rank3]:     cli_evaluate()
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
[rank3]:     results = evaluator.simple_evaluate(
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
[rank3]:     return fn(*args, **kwargs)
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 227, in simple_evaluate
[rank3]:     task_dict = get_task_dict(tasks, task_manager)
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 616, in get_task_dict
[rank3]:     task_name_from_string_dict = task_manager.load_task_or_group(
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 410, in load_task_or_group
[rank3]:     collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 310, in _load_individual_task_or_group
[rank3]:     return _load_task(task_config, task=name_or_config)
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 276, in _load_task
[rank3]:     task_object = ConfigurableTask(config=config)
[rank3]:   File "/home/ubuntu/lm-evaluation-harness/lm_eval/api/task.py", line 881, in __init__
[rank3]:     test_doc = self.task_docs[0]
[rank3]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2866, in __getitem__
[rank3]:     return self._getitem(key)
[rank3]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2850, in _getitem
[rank3]:     pa_subtable = query_table(self._data, key, indices=self._indices)
[rank3]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 587, in query_table
[rank3]:     _check_valid_index_key(key, size)
[rank3]:   File "/opt/conda/envs/py310/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 527, in _check_valid_index_key
[rank3]:     raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
[rank3]: IndexError: Invalid key: 0 is out of bounds for size 0
```