Repository: EleutherAI/lm-evaluation-harness
Issue: Cannot load local `mmlu` dataset
Root cause: Documentation Deficiency

I want to load the local `mmlu` dataset, I have already load it from [hails/[mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train)，
![image](https://github.com/user-attachments/assets/3b81e7d6-b601-47c5-b32a-68098ff8c1f2)



then I change the `_default_template_yaml` as follows:
```
dataset_path: parquet
dataset_kwargs:
  data_dir: autodl-tmp/mmlu_no_train/all
test_split: test
...
```

I run 
```
export HF_DATASETS_OFFLINE=1
```
and
```
lm_eval --model vllm \
    --model_args pretrained=${model_name},tensor_parallel_size=${GPUs_per_model},dtype=${dtype},gpu_memory_utilization=0.9,data_parallel_size=0 \
    --tasks mmlu \
    --apply_chat_template \
    --batch_size ${batch_size}
```

got
![image](https://github.com/user-attachments/assets/f2fbd0eb-d607-4739-bf00-f2e4e4cdcefc)


You need to make sure you have exactly one [dataset_kwargs](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mmlu/default/_default_template_yaml#L16) in your yaml file. Otherwise the last `dataset_kwargs` will overwrite the first `dataset_kwargs` so the `data_dir` will be None.  I spent hours to debug this error message `Parquet magic bytes not found in footer`  and found out this in a painful way..  

> You need to make sure you have exactly one [dataset_kwargs](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mmlu/default/_default_template_yaml#L16) in your yaml file. Otherwise the last `dataset_kwargs` will overwrite the first `dataset_kwargs` so the `data_dir` will be None. I spent hours to debug this error message `Parquet magic bytes not found in footer` and found out this in a painful way..

@wukaixingxp Thank you so much!! I have solved this problem, the correst yaml should be:
```
dataset_path: autodl-tmp/mmlu_no_train
dataset_kwargs:
  data_dir: all
...
```
Thank you so much really!!!!!!

Hi 
The above changes for local datasets can't work on my environment, and will trigger the following two issues:
a) one issue is can't find dev: I need to remove the line "fewshot_split: dev" from the default template YAML, then the issue could pass.
```
  File "huggingface_transformers/benckmark/lm-evaluation-harness-main/lm_eval/api/task.py", line 845, in __init__
    if self.fewshot_docs() is not None:
       ^^^^^^^^^^^^^^^^^^^
  File "huggingface_transformers/benckmark/lm-evaluation-harness-main/lm_eval/api/task.py", line 974, in fewshot_docs
    return self.dataset[self.config.fewshot_split]
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "anaconda/anaconda202402_work/envs/transinfer_cuda12/lib/python3.12/site-packages/datasets/dataset_dict.py", line 72, in __getitem__
    return super().__getitem__(k)
           ^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'dev'
``` 
b) After change with a, I encounter another issue with the dataset name. Then I tried to apply the below patch to bypass this check, and the mmlu_human_aging task can run successfully.
```
File "anaconda/anaconda202402_work/envs/transinfer_cuda12/lib/python3.12/site-packages/datasets/builder.py", line 342, in __init__
    self.config, self.config_id = self._create_builder_config(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "anaconda/anaconda202402_work/envs/transinfer_cuda12/lib/python3.12/site-packages/datasets/builder.py", line 569, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'human_aging' not found. Available: ['default']
```
```
--- a/lm_eval/api/task.py
+++ b/lm_eval/api/task.py
@@ -265,7 +265,7 @@ class Task(abc.ABC):
         """
         self.dataset = datasets.load_dataset(
             path=self.DATASET_PATH,
-            name=self.DATASET_NAME,
+            # name=self.DATASET_NAME,
             data_dir=data_dir,
             cache_dir=cache_dir,
             download_mode=download_mode,
@@ -922,9 +922,9 @@ class ConfigurableTask(Task):
                     )

     def download(self, dataset_kwargs: Optional[Dict[str, Any]] = None) -> None:
+        print(f"dataset_path: {self.DATASET_PATH}, dataset_kwargs: {dataset_kwargs}")
         self.dataset = datasets.load_dataset(
             path=self.DATASET_PATH,
-            name=self.DATASET_NAME,
             **dataset_kwargs if dataset_kwargs is not None else {},
         )
````

From the below run log, I found this mmlu_human_aging task will load all 14042 prompts to run the benchmark. That means if I specift the "--tasks mmlu", per task will run those 14042 prompts. Is this expected? Thanks
````
2024-11-06:21:50:17,047 DEBUG    [cache.py:33] requests-mmlu_human_aging-3shot-rank0-world_size1-tokenizer is not cached, generating...
2024-11-06:21:50:17,047 INFO     [task.py:414] Building contexts for mmlu_human_aging on rank 0...
100%|██████████████████████████████████████████████████████████████████████| 14042/14042 [00:51<00:00, 271.75it/s]
2024-11-06:21:51:09,203 DEBUG    [evaluator.py:460] Task: mmlu_human_aging; number of requests on this rank: 56168
2024-11-06:21:51:09,207 INFO     [evaluator.py:489] Running loglikelihood requests
Running loglikelihood requests:   0%|                                                   | 0/56168 [00:00<?, ?it/s]Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
Running loglikelihood requests:   1%|▎                                      | 537/56168 [00:40<1:01:28, 15.08it/s]
```

> Hi The above changes for local datasets can't work on my environment, and will trigger the following two issues: a) one issue is can't find dev: I need to remove the line "fewshot_split: dev" from the default template YAML, then the issue could pass.
> 
> ```
>   File "huggingface_transformers/benckmark/lm-evaluation-harness-main/lm_eval/api/task.py", line 845, in __init__
>     if self.fewshot_docs() is not None:
>        ^^^^^^^^^^^^^^^^^^^
>   File "huggingface_transformers/benckmark/lm-evaluation-harness-main/lm_eval/api/task.py", line 974, in fewshot_docs
>     return self.dataset[self.config.fewshot_split]
>            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "anaconda/anaconda202402_work/envs/transinfer_cuda12/lib/python3.12/site-packages/datasets/dataset_dict.py", line 72, in __getitem__
>     return super().__getitem__(k)
>            ^^^^^^^^^^^^^^^^^^^^^^
> KeyError: 'dev'
> ```
> 
> b) After change with a, I encounter another issue with the dataset name. Then I tried to apply the below patch to bypass this check, and the mmlu_human_aging task can run successfully.
> 
> ```
> File "anaconda/anaconda202402_work/envs/transinfer_cuda12/lib/python3.12/site-packages/datasets/builder.py", line 342, in __init__
>     self.config, self.config_id = self._create_builder_config(
>                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "anaconda/anaconda202402_work/envs/transinfer_cuda12/lib/python3.12/site-packages/datasets/builder.py", line 569, in _create_builder_config
>     raise ValueError(
> ValueError: BuilderConfig 'human_aging' not found. Available: ['default']
> ```
> 
> ```
> --- a/lm_eval/api/task.py
> +++ b/lm_eval/api/task.py
> @@ -265,7 +265,7 @@ class Task(abc.ABC):
>          """
>          self.dataset = datasets.load_dataset(
>              path=self.DATASET_PATH,
> -            name=self.DATASET_NAME,
> +            # name=self.DATASET_NAME,
>              data_dir=data_dir,
>              cache_dir=cache_dir,
>              download_mode=download_mode,
> @@ -922,9 +922,9 @@ class ConfigurableTask(Task):
>                      )
> 
>      def download(self, dataset_kwargs: Optional[Dict[str, Any]] = None) -> None:
> +        print(f"dataset_path: {self.DATASET_PATH}, dataset_kwargs: {dataset_kwargs}")
>          self.dataset = datasets.load_dataset(
>              path=self.DATASET_PATH,
> -            name=self.DATASET_NAME,
>              **dataset_kwargs if dataset_kwargs is not None else {},
>          )
> ```
> 
> From the below run log, I found this mmlu_human_aging task will load all 14042 prompts to run the benchmark. That means if I specift the "--tasks mmlu", per task will run those 14042 prompts. Is this expected? Thanks
> 
> ```
> 2024-11-06:21:50:17,047 DEBUG    [cache.py:33] requests-mmlu_human_aging-3shot-rank0-world_size1-tokenizer is not cached, generating...
> 2024-11-06:21:50:17,047 INFO     [task.py:414] Building contexts for mmlu_human_aging on rank 0...
> 100%|██████████████████████████████████████████████████████████████████████| 14042/14042 [00:51<00:00, 271.75it/s]
> 2024-11-06:21:51:09,203 DEBUG    [evaluator.py:460] Task: mmlu_human_aging; number of requests on this rank: 56168
> 2024-11-06:21:51:09,207 INFO     [evaluator.py:489] Running loglikelihood requests
> Running loglikelihood requests:   0%|                                                   | 0/56168 [00:00<?, ?it/s]Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
> Running loglikelihood requests:   1%|▎                                      | 537/56168 [00:40<1:01:28, 15.08it/s]
> ```
> ```

hello, I meet the same error. Have you solve the problem?